Kill Build: Vol. 1

Here are some of the ways I've seen a build killed during my years as an Eclipse committter.


1.  Lightning. Ottawa has tremendous thunderstorms in the summer.  What also happens during the early summer? Our coordinated release.  One summer, a transformer next to our office was hit by lightning.  Boom.  No power for about 12 hours. Bye bye build.

Image © ViaMoi, http://www.flickr.com/photos/viamoi/3338093351/ licensed under Creative Commons by-nc-sa 2.0  

2.  Ice.  The air conditioner in our lab filled with ice and ceased to function. The machines in our build lab then overheated and shut down.

3.  Flood: Our lab's air conditioner leaked the resulting water and flooded the lab. The build was relegated to a watery grave.

Image © thirsk, http://www.flickr.com/photos/thirsk/840616475/ licensed under Creative Commons by-nc-sa 2.0   

4. Random loss of power.  A committer plugged in a space heater in her office which tripped a circuit breaker and killed the power in our build lab. Or the total and intermittent loss of power from the Ottawa Hydro.  In the end, the results is the same: a dead build.

Image © ViaMoi, http://www.flickr.com/photos/viamoi/3339707547  licensed under Creative Commons by-nc-sa 2.0   


5.  Hardware self-destruction.  The eclipse foundation's power supply for a disk array sparked and brought down the switching gear. Dying switches, UPS's and drives have also wrought havoc.

6.  Permission issues: Permissions problems on the filesystem caused committing to the repository to fail. Or the signing process to time out.  Or both.

7. Network timeouts aka no network love.

Image © zoso_tc, http://www.flickr.com/photos/zoso_tc/3024964999/  licensed under Creative Commons by-nc-sa 2.0    

8. Human Error: The usual suspects compile errors, uncoordinated changes across projects, problems in builder itself, corrupted jars etc. have sent many builds to an early death.

9. Upgrades invoke the unexpected.  Changing a ssl certificate unleashed shenanigans.  An simple upgrade of a test machine caused failures.

Image © waferboard, http://www.flickr.com/photos/waferboard/5321533361/  licensed under Creative Commons by-nc-sa 2.0    


10.  Not respecting resource constraints.  Consuming all the disk space, CPU, and bandwidth available is a fine way to finish off a build.

How your builds failed?  Any exciting ways I've missed?