Better builds with Hudson, hardware and help

Some might say that the build is one of the engines drive a project.  The Eclipse and Equinox build needs a tune up.

(Image © Paul Gorbould, http://www.flickr.com/photos/gorbould/3531940727/in/set-72157607916475025/, licensed under Creative Commons by-nc-sa 2.0)

A simplified summary of our build process today is as follows.  (Many of these processes occur in parallel.)

1. Checkout code from eclipse.org to an IBM build server.
2. Generate build scripts, compile code and create a master feature of all the bundles used in the build.
3. Copy master feature to eclipse.org for signing, copy back to IBM server when complete.
4. Run the p2 director to provision products and use repository tooling to slice out zipped repositories.
5. Run JUnit and performance tests.

Here are some of the fundamental ways this process can be improved:


Hudson


Problem: The build process takes too long to complete code checkout, compilation, signing and packaging. It's also also too monolithic.

Solution: Take advantage of the local access to the eclipse.org filesystem by running the build on the Hudson install at at the foundation.  Now that we have hardware donations there will be new Hudson slaves for more build cycles.  Also, breaking the build up into smaller builds and chaining them together will let us identify problems earlier. See bug 302436 for details. We run test builds on Hudson today and they work very well.


Hardware

Problem:  There aren't enough test machines to run our tests in a reasonable timeframe.  Committers aren't able to rerun the tests on the same hardware that was used in the build.

Solution: New test  hardware at the Eclipse foundation is a start. We'll probably need more but it's a good beginning. Thank you to all the companies who have donated hardware to the foundation recently.


Help

Problem

Today we run our JUnit tests  on Windows machines by invoking them via rsh.  This allows us to manipulate the display while running ui tests.

Solution

This is where you come in.

I'm not sure how to do this on Hudson.  Sonatype has a series of articles on running tests in a multiple OS environment and they state that this is still a problem for them.  If anyone has any pointers to articles on how to do this it would be appreciated.   Please update bug 305213 with your suggestions.

Also, we need some rack mounted Macs to run JUnit tests on this important platform.  So once we get the new hardware integrated, some more hardware donations would be welcome :-)