Releng of the Nerds: November 2010

The Mozilla foundation is similar to the Eclipse foundation in many ways. However, it has a wholly owned non-profit subsidiary, mozilla.com, which has a sizeable engineering and marketing staff
that work on the Mozilla projects themselves. The Eclipse foundation's revenue comes from membership dues from member companies. Much of Mozilla's revenue stream arises from search functionally embedded in Firefox that generates cash flow from companies such as Google, Amazon, Yahoo and eBay. Of course, Mozilla projects provide more general desktop tools, than the specialized tools we provide at Eclipse, so they have a larger user base.

A while ago, I watched a webinar regarding the Mozilla build process.

It has some interesting numbers. Today, Mozilla has

12 release engineers
17 platforms to support
12 branches to build from
1200 build and test machines
3 colos

Each of their builds consumes 12.40 hours of build time, 54.48 hours of test time and 2.79 days of CPU time. Because of the enormous resources that the build consumes, they run a lot of tests in parallel. With 1200 machines you can do that :-)

They also run many performance tests with each build. Since the hardware must be identical to compare performance results across machines, 350 of 1200 machines are Mac minis. It's the lowest common denominator for hardware that can run on Mac, Windows and Linux operating systems. In addition, they have a small footprint in a colo, and are relatively inexpensive.

Mozilla's many Mac minis

At Eclipse we have,

15 platforms to support
2 branches to build from (Helios and Indigo)
5 eclipse.org build and test machines

The Eclipse project builds for 15 different platforms but we only run JUnit tests on three. We don't have the hardware to run tests on all of them. Windows, Mac and Linux are represent the vast majority of users and many of the platforms have low download numbers. The SWT team tests all 15 platforms a few times a milestone and ensure that they start up and work as expected.

The build scheduling engine they use at Mozilla is Buildbot, where we use Hudson on eclipse.org servers. Their build process used to take ten days to make a release available. This wasn't really acceptable. Mozilla have what they call "zero day" or "chem-spill releases" where they must prepare a release in a single day to address a critical issue. This is really an admirable feat. During the Callisto release, I recall having to stay up all night to ensure a build completed to ensure a critical fix was available for the release train the next day. I'm not sure that with the size of the Eclipse release train today, a zero day release would be possible if a lower level project had to be rebuilt.

Mozilla also run tests on a number mobile devices. To reduce interference between these devices, they are stored in an IKEA shoe closet in the colo.