V2 Full rollout complete.

A few weeks ago we rolled back the V2 release. Today all Pipes are now running on the V2 engine!

Leave a comment

Full V2 rollout update

Yesterday we migrated all Pipes to the V2 engine. We experienced some minor delays in the completion of the full migration.

We expect to migrate all Pipes to the V2 engine sometime over the next week or two. V1 Pipes will still run as V1 for the time being. Please take this opportunity to further test your V1 Pipes as V2 by appending &_v2=true to any V1 Pipe URL.

Thanks! -Pipes Team

Posted in Uncategorized | Leave a comment

V2 Engine Migration Complete

It’s been a little over a year since we first announced the V2 engine. After many months of work and help from the Pipes developer community, all Pipes now are running off our new V2 engine! Special thanks to hapdaniel and libra_sun for reporting issues and making V2 a better engine.

We also improved several things with our Date Modules.

In the Date Formatter:

- Added new format token: “%K”, produces RFC 822 compliant date (shortcut for “%a’,’ %d %b %G %H’:'%M’:'%S ‘GMT’”)

- Added new Timezone input field – which allows you to specify a target time zone. This must be one of the time zone IDs supported by the Java platform, or a custom time zone ID using the format defined in http://download.oracle.com/javase/6/docs/api/java/util/TimeZone.html

In the Date Builder:

- Added new output fields: day_name, day_ordinal_suffix, and month_name

We are continually updating the V2 engine and will be adding new features. If your (previous) V1 Pipes still do not work, please report the issue and include your Pipe ID in your post. Community members and staff will look at the issue. Please also read about the new V2 limits.

(if you get a blank page on the Pipe Editor please clear your browsers cache)

Leave a comment

The Pipes V2 Engine: Limits and More

Here at Pipes, we’ve been working hard to prepare the V2 Engine for full deployment in August 2011. We realize that this is a huge change for most of you and your Pipes, so we are trying to make the transition as seamless as possible by giving you incremental updates and opportunities to test your Pipes with V2 functionality. Throughout this process, you’ve been giving us your feedback, and we’ve been listening. So, we’d like to take a moment to both reassure you about the V2 Pipes Engine and provide the previously-lacking information regarding limitations and how to best mitigate pain points in the migration process.

Many of you have been asking why V2? Why can’t we just keep the Pipes V1 Engine? Well, simply put, change is intrinsic to software. Products need to be maintained and updated on a regular basis to fix bugs, make use of improved technologies, and provide cool, new features to you — our awesome users. The Pipes V1 engine is nearly 4 years old and was originally built quickly to provide a proof-of-concept. No one could have predicted how hugely successful Pipes would become! And we are grateful to all our devoted users for years of commitment to the product. But now to prove our commitment to you, we are moving to the V2 Engine to provide an even greater Pipes product.

Now on the nitty gritty…

The V2 engine isn’t vastly “buggy” or “broken”. It’s just different. It uses the same engine as the YQL product. We wouldn’t be releasing V2 if we didn’t have high confidence in the engine. After comparing thousands of Pipes running in V1 versus V2, we’ve found only minor incompatibilities and have been working diligently to update V2 where it makes sense. But no process is perfect, and so we’ve been relying on you to report issues and findings.

There are however some explicit differences with the V2 Engine. As many of you have probably already noticed, we’ve imposed a few limitations on Pipes that previously did not exist in V1. Why? Primarily to reduce abuse, improve performance, and to scale. These limitations are in no way intended to prevent you from building feature-full Pipes. Rather, we just ask you to think differently in certain cases about how you create your Pipe. Anything that you were able to do with V1 you should be able to do in a slightly different way with V2.

Here’s an outline of the new V2 limitations in detail:

* 3 2 Subpipes (nesting level of 3): Pipe nesting using subpipes has been limited to a nesting depth of 3. For a single pipe, you can drag as many subpipe modules as you like on the editor. However, opening a subpipe and including another subpipe module can only be done up to a depth of 3. That is, we only allow pipe -> subpipe1 -> subpipe2. Subpipes are meant to be used just like functions in programming. If you keep nesting functions, the trace becomes so long and complex that it essentially defeats the purpose of modularizing functionality. With subpipes, nesting too deep actually negatively impacts the performance of your Pipe. This limit will affect both existing and new Pipes. If you get this error when running your pipe, you’ll need to edit it and remove any/all subpipe modules at level 3 (the third tab in the editor).

* 10 Module Inputs: Many modules have dynamic inputs. Previously you could add as many inputs as you would like for feeds, parameters, etc. Now, we limit it to 10 fields per input parameter, with the exception of the URL Builder Module. This change is independent of the engine and was a feature we wanted to include with V1 but never got around to. Simply put, having too many input fields makes a module appear unruly. If you want to fetch many feeds for example, you can use multiple Fetch Feed modules and Union them together. Existing Pipes if left untouched will not be affected by this limit. However, if you begin editing a module with more than 10 inputs, the restriction will be enforced.

* 30sec Execution Timeout: All Pipes must run within a 30 second execution timeout. This is a restriction put in place by the underlying YQL engine. In the current implementation, we allot 27 seconds to fetching your external feeds and sources, and 3 seconds for the V2 engine to perform the Pipe processing on top of those feeds. Any feeds that take longer than 27 seconds will be cut off. Because the V2 engine is so powerful, 3 seconds is plenty of processing time. So if your V2 Pipes are hitting the timeout limit, chances are there are one ore more sources that are slow to fetch. In situations like those, you may want to take into consideration the quality of the source itself. For example, if a feed takes longer than 27 seconds to fetch, it’s likely a poor feed and should perhaps be replaced with a different source if possible.

* 1.5MB Feed Size: YQL enforces a maximum response size of 1.5MB, so Pipes is subject to this as well. That is, we can currently only fetch 1.5MB of data per feed. We will continue to investigate configuring this value and potentially adjusting it for Pipes.

So there you have it. We know it’s late in the game, but we hope we have helped clarify some core aspects of the Pipes V2 Engine. After the full migration to V2 in August, we will continue with regular, incremental updates to provide further improvements and features. If you have any more questions, concerns, encounter problems as you are upgrading your Pipes, or have feedback on the limitations, please let us know. All we ask is that you help maintain the integrity and relevance of this forum by only posting specific Pipes V2 related questions and issues. And as always, please include your pipe id so we can help you more quickly!

- The Pipes Team

Posted in Uncategorized | Leave a comment

Phase 1 starting (for V2) on June 29th.

We are starting phase 1 for the Pipes V2 engine migration this Wednesday June 29th.

Phase 1 – All newly created Pipes and clones will use the V2 engine and you will no longer be able to revert to V1.

Existing V1 Pipes will still continue to run as V1 until August 1st. On August 1st all V1 Pipes will be auto upgraded to V2.

Please post any issues to the V2 message board here: http://tech.groups.yahoo.com/group/pipes-engine2users/

Thanks – Pipes Team

Leave a comment

Pipes V2 engine timeline.

The V2 engine will be phased in 2 steps.

June 28, 2011 – All new Pipes will use the V2 engine and you will no longer be able to revert to V1. Existing V1 Pipes will still continue to run as V1.

August 1, 2011 – *All* Pipes will be upgraded to the new V2 engine.

Please post any V2 issues here: http://tech.groups.yahoo.com/group/pipes-engine2users/post

And please read this previous post about the V2 engine: http://blog.pipes.yahoo.net/2010/06/09/yahoo-pipes-v2-engine/

Thanks! – The Pipes Team

Leave a comment

Pipes now has Cross-Origin Resource Sharing (CORS) support.

If you use a modern browser (FF3.5+,Chrome 3+,Safari 4+,IE8+) you can now access Pipes data using AJAX from a different domain.

You can use YUI3 IO (use: ‘native’) to make the request for you or you can hand craft your own.

Some sample code:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html>
    <head>
        <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
        <title>Pipes CORS test</title>
        <script src="https://github.com/douglascrockford/JSON-js/raw/master/json2.js">
        </script>
    </head>
    <body>
        <div id="output">
        </div>
        <script>
            function createCORSRequest(method, url){
                var xhr = new XMLHttpRequest();
                if ("withCredentials" in xhr) {
                    xhr.open(method, url, true);
                } else
                    if (typeof XDomainRequest != "undefined") {
                        xhr = new XDomainRequest();
                        xhr.open(method, url);
                    } else {
                        xhr = null;
                    }
                return xhr;
            }

            var request = createCORSRequest("get", "http://pipes.yahoo.com/pipes/pipe.run?_id=ZKJobpaj3BGZOew9G8evXg&_render=json&ticker=YHOO,GOOG");

            if (request) {
                request.onload = function(){
                    var r = JSON.parse(request.responseText);
                    var holder = "";
                    for (var x = 0, rl = r.value.items.length; x < rl; x++) {
                        holder += r.value.items[x].title + "<br>";
                        holder += r.value.items[x].description;
                        holder += "<hr>";
                    }
                    document.getElementById("output").innerHTML = holder;
                };
                request.send();
            }
        </script>
    </body>
</html>
Leave a comment

Yahoo! Pipes V2 engine

Some background: The current Pipes engine is nearly 3 years old and was created quite quickly to see if the Pipes idea would be successful. It was. Wildly so. Its been a good stretch for that software and the system design, but the time has come to put something else in its place. During the last year the Pipes team has been developing a second generation engine, which is used in the YQL product (developer.yahoo.com/yql). Its much more stable, performs far better that the current engine, and we’ve been working on migrating the old Pipes-specific functionality out of the v1 engine to this new engine. We’re at a “beta” stage now, where most of the big bugs are out, and we’re shaking down the system to see what we’ve missed. Some things to note:

  • strict equality between the output of Pipes 1 and Pipes 2 is not a requirement. However, the Pipe should run and give output that makes sense.
  • there will be bugs, this isn’t ready for primetime.
  • the mechanism to run Pipes on the new engine is given to you in confidence.
  • the Pipes 2 engine is a new “baseline” for new functionality, not the end goal. It gives us a scalable platform to build features and capabilities which we’ve been unable to add to the old one, including more sophisticated quota management, Pipe stat tracking, caching and enhanced modules (like scripting).

What we’d like to get from you:

  • general feedback. It works the same, it doesn’t, etc
  • specific bugs. What doesn’t work. Please give us the Pipe ID and if you can pinpoint where things are going awry the better. Once you find a bug (for example with a specific module) it will probably be common to many Pipes so please try and avoid duplication.

Please post your feedback here.

So how do you get going?

  • Its easy – First login to Pipes, then use the upgrade links on the pipe.info pages or in the Pipes editor.
  • You can only upgrade your own Pipes.
  • If you see a Pipe you want running in V2 but do not own it, you can clone it and then upgrade it.

What’s next:

  • Auto-upgrade all Pipes that work the same in both engines.
  • After a number of months we’ll retire the old engine

Thanks for trying out the V2 engine!

The Pipes Team

Leave a comment

YQL execute to power the Pipes webservice module update.

If anyone has been using YQL execute to power the Pipes webservice module, in your execute statement JSON.stringify is no longer needed.

Our previous post about using YQL to power the Pipes webservice module has been updated: http://blog.pipes.yahoo.net/2009/06/01/using-yql-execute-to-power-the-pipes-webservice-module/

Leave a comment

Craigslist and Yahoo! Pipes – resolved.

We’ve been working with Craigslist over the last week to see how both products can better work with each other. After providing more information about how to track and ban Pipes that violate their TOS (or are considered generally abusive), they have opened Craigslist RSS feeds back to us.

Leave a comment