Android Panda tests in production

Two weeks ago the Mozilla release engineering team had a work week in Toronto.  Toronto in January?  Balmy, according to one of our of colleagues from California.
 

Hal wears sandals in the snow.  Photo: John Hopkins

Over the past few months,  Ateam, IT and release engineering have been wrestling panda boards into submission and deploying them into production as part of our test infrastructure for Firefox for Android.  (There are pandas running B2G tests as well. )   Firefox is also a colloquial name for the red panda who apparently likes snow too.

Image ©maiac,  http://www.flickr.com/photos/maiac/5442715157/licensed under Creative Commons by-nc-sa 2.0

As Dustin described in his post on Mozpool, the panda boards are development boards that sit in specialized chassis that were developed for Mozilla.  During the work week, I finished putting the  last of the 400 panda boards in production running tests on the cedar, mozilla-central, try and mozilla-inbound branches.   From a releng perspective, this involved setting up 32 foopies and 5 new buildbot masters and updating our configs.   For those readers uninitiated with our release engineering configuration, we use buildbot as our continuous integration system to schedule builds and tests.  Foopies are are dedicated servers that handle the connections of the mobile devices to the buildbot masters.

Some of the issues that we encountered while we were rolling the pandas into production were quite difficult to overcome.  The Ateam, especially Joel spent many days debugging them, including this this particular nefarious one where the panda boards would spontaneously reboot while running tests

Bug 811444 - android panda boards magically reboot in the middle of the test

Joel's solution the solution was to slow down the running of the tests so the panda boards wouldn't reboot and allowed us to move forward.

Many thanks to Joel, Callek, Armen, Amy, Dustin and Jake for all their help putting these into production.

Next step: Manage the Android  Panda boards using Mozpool

Further reading
Dustin Mitchell's post on MozPool, including pictures of the awesome custom ruby red Mozilla chassis
Bug 802667 - configure new buildbot masters for use with android on pandas
Bug 803248 - buildbot config changes to support panda_android*
Bug 805658 - add all panda boards to slavealloc, disabled  
Bug 811723 - change android reftests to run with --ignore-window-size for panda only 
Bug 825984 - Turn on Android 4.0 test jobs on try and mozilla-inbound
Bug 829181 - put remaining pandas into production for Android 4.0 tests

Read more...