Kill Build: Vol. 1
>> Friday, June 03, 2011
Here are some of the ways I've seen a build killed during my years as an Eclipse committter.
1. Lightning. Ottawa has tremendous thunderstorms in the summer. What also happens during the early summer? Our coordinated release. One summer, a transformer next to our office was hit by lightning. Boom. No power for about 12 hours. Bye bye build.
Image © ViaMoi, http://www.flickr.com/photos/viamoi/3338093351/ licensed under Creative Commons by-nc-sa 2.0
2. Ice. The air conditioner in our lab filled with ice and ceased to function. The machines in our build lab then overheated and shut down.
3. Flood: Our lab's air conditioner leaked the resulting water and flooded the lab. The build was relegated to a watery grave.
Image © thirsk, http://www.flickr.com/photos/thirsk/840616475/ licensed under Creative Commons by-nc-sa 2.0
4. Random loss of power. A committer plugged in a space heater in her office which tripped a circuit breaker and killed the power in our build lab. Or the total and intermittent loss of power from the Ottawa Hydro. In the end, the results is the same: a dead build.
Image © ViaMoi, http://www.flickr.com/photos/viamoi/3339707547 licensed under Creative Commons by-nc-sa 2.0
5. Hardware self-destruction. The eclipse foundation's power supply for a disk array sparked and brought down the switching gear. Dying switches, UPS's and drives have also wrought havoc.
6. Permission issues: Permissions problems on the filesystem caused committing to the repository to fail. Or the signing process to time out. Or both.
7. Network timeouts aka no network love.
Image © zoso_tc, http://www.flickr.com/photos/zoso_tc/3024964999/ licensed under Creative Commons by-nc-sa 2.0
8. Human Error: The usual suspects compile errors, uncoordinated changes across projects, problems in builder itself, corrupted jars etc. have sent many builds to an early death.
9. Upgrades invoke the unexpected. Changing a ssl certificate unleashed shenanigans. An simple upgrade of a test machine caused failures.
Image © waferboard, http://www.flickr.com/photos/waferboard/5321533361/ licensed under Creative Commons by-nc-sa 2.0
10. Not respecting resource constraints. Consuming all the disk space, CPU, and bandwidth available is a fine way to finish off a build.
How your builds failed? Any exciting ways I've missed? Read more...
