Kill Build: Vol. 1

>> Friday, June 03, 2011

Here are some of the ways I've seen a build killed during my years as an Eclipse committter.

1.  Lightning. Ottawa has tremendous thunderstorms in the summer.  What also happens during the early summer? Our coordinated release.  One summer, a transformer next to our office was hit by lightning.  Boom.  No power for about 12 hours. Bye bye build.

Image © ViaMoi, licensed under Creative Commons by-nc-sa 2.0  

2.  Ice.  The air conditioner in our lab filled with ice and ceased to function. The machines in our build lab then overheated and shut down.

3.  Flood: Our lab's air conditioner leaked the resulting water and flooded the lab. The build was relegated to a watery grave.

Image © thirsk, licensed under Creative Commons by-nc-sa 2.0   

4. Random loss of power.  A committer plugged in a space heater in her office which tripped a circuit breaker and killed the power in our build lab. Or the total and intermittent loss of power from the Ottawa Hydro.  In the end, the results is the same: a dead build.

Image © ViaMoi,  licensed under Creative Commons by-nc-sa 2.0   

5.  Hardware self-destruction.  The eclipse foundation's power supply for a disk array sparked and brought down the switching gear. Dying switches, UPS's and drives have also wrought havoc.

6.  Permission issues: Permissions problems on the filesystem caused committing to the repository to fail. Or the signing process to time out.  Or both.

7. Network timeouts aka no network love.

Image © zoso_tc,  licensed under Creative Commons by-nc-sa 2.0    

8. Human Error: The usual suspects compile errors, uncoordinated changes across projects, problems in builder itself, corrupted jars etc. have sent many builds to an early death.

9. Upgrades invoke the unexpected Changing a ssl certificate unleashed shenanigans.  An simple upgrade of a test machine caused failures.

Image © waferboard,  licensed under Creative Commons by-nc-sa 2.0    

10.  Not respecting resource constraints.  Consuming all the disk space, CPU, and bandwidth available is a fine way to finish off a build.

How your builds failed?  Any exciting ways I've missed?


Max 1:08 AM  

Security finding Hudson too unsecure to have access to svn. Killed the build.

Hudson being taken down for maintenance without warning. Killed the build. mirrors being corrupt. Killed the build.

Post a Comment

  © Blogger template Simple n' Sweet by 2009

Back to TOP