Eclipsing the build

A plan item for 3.7 is to transition the Eclipse and Equinox build process to run on eclipse.org hardware.  Why may you ask, isn't this the case today? Two reasons:

1.  Until recently, the Eclipse foundation didn't have hardware to support our requirements of running tests on multiple platforms (Windows, Mac, Linux).
2.  There wasn't a plan item to transition our build to eclipse.org until 3.7.   You may think that we open source developers are happy hippies who have the freedom to work on the bugs that interest them :-). The reality is that I prioritize my bugs as follows:

• Fix bugs that cause the build to fail.
• Fix bugs that block other committers from moving forward with their work or are necessary for legal reasons.  
• Plan items
• Everything else

I was going to write a post describing the progress migrating to eclipse.org hardware but I thought it would be a good idea to give you some background regarding how the Eclipse and Equinox project's build works today. 

The Eclipse build creates SDKs for 15 platforms.

A graphical representation of the 15 platforms we build  (os,ws,arch)

We use fragments to define platform specific code that is associated with a host bundle.  In some of these fragments, there's C code that needs to be compiled on the platform specific hardware to create binary libraries.  For instance, the SWT and equinox launcher fragments are examples of  these artifacts. We have about seven machines to compile the libraries for these 15 platforms, as some are cross compiled.  The SWT and Equinox team have hudson instances, which connect to all these internal IBM servers, compile the C code, and commit the resulting libraries in binary form to CVS at eclipse.org. This part of the build isn't transitioning to eclipse.org.

A simplified description of the existing Eclipse build that produces the artifacts that are available for download on eclipse.org is shown below.   Essentially, once a build is launched from cron, the build scripts check out all the code from eclipse.org to the IBM build machine which compiles the bundles. These bundles are copied to eclipse.org to be signed, then copied back to the IBM build machine to be assembled into a p2 repo, from which we create SDKs to run the tests. The zips, repos and test results are rsynced to eclipse.org.

Once we get the issues worked out with the new Hudson slaves to run tests, all the JUnit tests will run on eclipse.org hardware.  Eventually,  we hope to run the performance tests there once we have sufficient hardware.

Advantages of moving the build to hudson.eclipse.org

1. The build process will be more open and other Eclipse and Equinox committers will be able to start a build.  Today, I manage the crontab of the build machine.
2. Fewer issues with network timeouts due to the fact that the eclipse.org filesystem is local to Hudson.
3. Hudson has a rich feature set and many plugins to extend it. 

Disadvantages

1. Like kindergarten, we'll have to share the filesystem and machines with others.  Today the build runs on dedicated machines and thus we don't have resource constraint issues from other teams.  (The build process itself isn't faster at eclipse.org because of these resource constraints, but the tests are because at the moment we are the only team using the Windows and Mac test machines.)
2. Can't hack and slash the build on the the command line to rerun a single test suite or restart the build manually.

In a future post, I'll describe the process to transition our builds to run on eclipse.org and the obstacles that we have overcome.

How can you help?  Consider donating to Friends of Eclipse. 

References

Build at Eclipse.org

Graphics thanks to open source clip art

Servers
Database
iMac
Server cabinet
Tower
Database

Read more...