Mozpool and Mozharness for Android Panda tests

Over the last few months, I migrated the test infrastructure that we run for Android 4.0 on Panda devices to use mozharness and mozpool. Before I go into a description of what this entailed, some background for those not familiar with our release engineering infrastructure.  We use Buildbot for our continuous integration engine. We have 50+ Buildbot masters are designated specific purposes such as scheduling, building, try servers and testing.  Test masters are further subdivided into ones specifically allocated to running tests on Mac, Linux, Windows, Tegras (Android 2.2 devices) and Pandas (Android 4.0 devices).  We have many devices that are allocated to each master to handle the volume of tests and builds we run each day. 

Why Mozpool?

We have over 800 panda devices in production that are used to run unit tests and talos (performance) tests for Firefox for Android 4.0.    Panda devices are rack mounted in a chassis.  One of the issues with these devices is that they are development boards and inherently quite unstable when running a large volume of tests 24/7.  Not many organizations run the volume of automated tests on mobile devices that we do.  Dealing with their issues in a non-automated fashion does not scale.

Pandas fall down

Mozpool is software that is used to mitigate that inherent stability and ensure that the devices that are available to run tests are in a verified state and have the correct Android image.  If Mozpool determines that the device has problems that makes it unsuitable to run tests, it changes its state so it won't be in the pool of devices running tests.  For instance, if it doesn't boot correctly or the requisite image cannot be applied. This reduces the number of jobs that fail due to infrastructure issues.  Mozpool also provides logging of the actions on devices and and a web page for looking at the state of your devices.  There is an API implemented vi REST and HTTP for simple actions on your devices, such as requesting a device, and returning it to the pool.  Another advantage of Mozpool is that it's easy to reimage the devices with a new image, either via the mozpool ui or simply specifying a new image in your mozpool client code.

Why Mozharness?
There are four main projects that are used to manage our infrastructure buildbotcustom, buildbot-configs, tools and mozharness. From the mozharness FAQ in Aki's words "Mozharness is a configuration-driven python script harness with full logging that allows production infrastructure and individual developers to use the same scripts."  Implementing mozharness scripts allows developers who don't have access to our infrastructure to run the same scripts we do on production hardware.   

Traditionally, the scripts that define the buildbot actions for our tests have been defined in the buildbotcustom project.  The code is very convoluted and it is difficult to new people to parse what bits of code apply to each platform.   Also,  every time we want to deploy changes for them we have to do a reconfig.  A reconfig is an operation where we upgrade our buildbot masters to the latest version of the code in the buildbotcustom, buildbot-configs and tools repositories.  A reconfig is usually initiated by the releng person on buildduty only a few times a week.  When you use mozharness, the buildbot config scripts point to the production branch of the mozharness repo. New changes can be deployed to production with a simpler merge to the production branch of mozharness.  No reconfig.

Closeup of zip line harness aka visual representation of some buildbotcustom code

 Image by Image ©timcaynes, http://www.flickr.com/photos/timcaynes/1588430260/under Creative Commons by-nc-sa 2.0

With mozharness scripts, you have a discrete script and config file that applies to each platform.  For instance, for panda android unittests, the config file is here, and the corresponding script is here. 

There are actions in mozharness that serve as boilerplate code that can be reused to common actions when running tests, such as installing Python packages into a virtual environment, closing repositories, downloading and extracting files and so on.  For instance, here are the default actions when running the android unit tests on Pandas:

These actions correspond to methods is classes that that your mozharness scripts will inherit. For instance, the PandaTest class in the mozharness script in inherits VirtualEnvMixin which has a method create_virtualenv to install the appropriate python modules for the test run.

 

The request-device action uses the mozpool client code within mozharness to interact with the mozpool and verify that the device is in a usable state to run tests. You can also override these methods with ones in your own mozharness script if you have changes that are unique to your platform.

If you are have questions regarding mozharness, please join the mozharness channel on irc.mozilla.org :-)

Further reading
Dustin Mitchell's description of Mozpool
Mozpool API documentation
Aki Sasaki's Mozharness FAQ 
Aki Sasaki's first blog post on why mozharness and more

Read more...

Android Panda tests in production

Two weeks ago the Mozilla release engineering team had a work week in Toronto.  Toronto in January?  Balmy, according to one of our of colleagues from California.
 

Hal wears sandals in the snow.  Photo: John Hopkins

Over the past few months,  Ateam, IT and release engineering have been wrestling panda boards into submission and deploying them into production as part of our test infrastructure for Firefox for Android.  (There are pandas running B2G tests as well. )   Firefox is also a colloquial name for the red panda who apparently likes snow too.

Image ©maiac,  http://www.flickr.com/photos/maiac/5442715157/licensed under Creative Commons by-nc-sa 2.0

As Dustin described in his post on Mozpool, the panda boards are development boards that sit in specialized chassis that were developed for Mozilla.  During the work week, I finished putting the  last of the 400 panda boards in production running tests on the cedar, mozilla-central, try and mozilla-inbound branches.   From a releng perspective, this involved setting up 32 foopies and 5 new buildbot masters and updating our configs.   For those readers uninitiated with our release engineering configuration, we use buildbot as our continuous integration system to schedule builds and tests.  Foopies are are dedicated servers that handle the connections of the mobile devices to the buildbot masters.

Some of the issues that we encountered while we were rolling the pandas into production were quite difficult to overcome.  The Ateam, especially Joel spent many days debugging them, including this this particular nefarious one where the panda boards would spontaneously reboot while running tests

Bug 811444 - android panda boards magically reboot in the middle of the test

Joel's solution the solution was to slow down the running of the tests so the panda boards wouldn't reboot and allowed us to move forward.

Many thanks to Joel, Callek, Armen, Amy, Dustin and Jake for all their help putting these into production.

Next step: Manage the Android  Panda boards using Mozpool

Further reading
Dustin Mitchell's post on MozPool, including pictures of the awesome custom ruby red Mozilla chassis
Bug 802667 - configure new buildbot masters for use with android on pandas
Bug 803248 - buildbot config changes to support panda_android*
Bug 805658 - add all panda boards to slavealloc, disabled  
Bug 811723 - change android reftests to run with --ignore-window-size for panda only 
Bug 825984 - Turn on Android 4.0 test jobs on try and mozilla-inbound
Bug 829181 - put remaining pandas into production for Android 4.0 tests

Read more...