New platform tests: OS X Mountain Lion

Late Monday night, we moved a new testing platform into production.  We have 80+ shiny new Mountain Lion slaves running tests for desktop builds, with several reserved for staging tests.  There are a few issues with failing tests that are being tracked in bug 786084.

Image ©ekai,  http://www.flickr.com/photos/ekai/457004988/ licensed under Creative Commons by-nc-sa 2.0

This was one of the larger projects I've worked on since I started at Mozilla a few months ago.  Many of the existing test and builds slaves that we run are configured with an old installation of Puppet (0.24.8) that we call "pre-historic puppet".  We have a newer installation of Puppet running 2.7.19 nicknamed PuppetAgain that serves as the Puppet Master for these new Mountain Lion slaves.   The PuppetAgain installation used to just support CentOS slaves, so we added modules to support the Mac slaves and accommodate all the quirky Apple specific configuration needs.   
 
Given my experience as an Eclipse committer, I installed the Geppetto project as my Puppet manifest editor and it's a great tool.  Thanks to the folks at Cloudsmith for developing it, it has been very useful. 

In addition to becoming familiar with Puppet, I also had the opportunity to learn about the releng buildbot configs, adding new buildbot masters, updating graphing databases to store Talos results, running staging tests and all the steps to add a new platform. Since the process wasn't documented, I wrote up some notes to help others if they need to do the same.

Thanks to everyone on the release engineering and release engineering operations teams for all their help, especially answering my many questions and reviewing patches.  Its great to see something that you worked on in production, and finally generating green builds.