Yahoo! Pipes V2 engine

Some background: The current Pipes engine is nearly 3 years old and was created quite quickly to see if the Pipes idea would be successful. It was. Wildly so. Its been a good stretch for that software and the system design, but the time has come to put something else in its place. During the last year the Pipes team has been developing a second generation engine, which is used in the YQL product ( Its much more stable, performs far better that the current engine, and we’ve been working on migrating the old Pipes-specific functionality out of the v1 engine to this new engine. We’re at a “beta” stage now, where most of the big bugs are out, and we’re shaking down the system to see what we’ve missed. Some things to note:

  • strict equality between the output of Pipes 1 and Pipes 2 is not a requirement. However, the Pipe should run and give output that makes sense.
  • there will be bugs, this isn’t ready for primetime.
  • the mechanism to run Pipes on the new engine is given to you in confidence.
  • the Pipes 2 engine is a new “baseline” for new functionality, not the end goal. It gives us a scalable platform to build features and capabilities which we’ve been unable to add to the old one, including more sophisticated quota management, Pipe stat tracking, caching and enhanced modules (like scripting).

What we’d like to get from you:

  • general feedback. It works the same, it doesn’t, etc
  • specific bugs. What doesn’t work. Please give us the Pipe ID and if you can pinpoint where things are going awry the better. Once you find a bug (for example with a specific module) it will probably be common to many Pipes so please try and avoid duplication.

Please post your feedback here.

So how do you get going?

  • Its easy – First login to Pipes, then use the upgrade links on the pages or in the Pipes editor.
  • You can only upgrade your own Pipes.
  • If you see a Pipe you want running in V2 but do not own it, you can clone it and then upgrade it.

What’s next:

  • Auto-upgrade all Pipes that work the same in both engines.
  • After a number of months we’ll retire the old engine

Thanks for trying out the V2 engine!

The Pipes Team

This entry was posted in Uncategorized. Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>