Mozpool and Mozharness for Android Panda tests

Over the last few months, I migrated the test infrastructure that we run for Android 4.0 on Panda devices to use mozharness and mozpool. Before I go into a description of what this entailed, some background for those not familiar with our release engineering infrastructure.  We use Buildbot for our continuous integration engine. We have 50+ Buildbot masters are designated specific purposes such as scheduling, building, try servers and testing.  Test masters are further subdivided into ones specifically allocated to running tests on Mac, Linux, Windows, Tegras (Android 2.2 devices) and Pandas (Android 4.0 devices).  We have many devices that are allocated to each master to handle the volume of tests and builds we run each day. 

Why Mozpool?

We have over 800 panda devices in production that are used to run unit tests and talos (performance) tests for Firefox for Android 4.0.    Panda devices are rack mounted in a chassis.  One of the issues with these devices is that they are development boards and inherently quite unstable when running a large volume of tests 24/7.  Not many organizations run the volume of automated tests on mobile devices that we do.  Dealing with their issues in a non-automated fashion does not scale.

Pandas fall down

Mozpool is software that is used to mitigate that inherent stability and ensure that the devices that are available to run tests are in a verified state and have the correct Android image.  If Mozpool determines that the device has problems that makes it unsuitable to run tests, it changes its state so it won't be in the pool of devices running tests.  For instance, if it doesn't boot correctly or the requisite image cannot be applied. This reduces the number of jobs that fail due to infrastructure issues.  Mozpool also provides logging of the actions on devices and and a web page for looking at the state of your devices.  There is an API implemented vi REST and HTTP for simple actions on your devices, such as requesting a device, and returning it to the pool.  Another advantage of Mozpool is that it's easy to reimage the devices with a new image, either via the mozpool ui or simply specifying a new image in your mozpool client code.

Why Mozharness?
There are four main projects that are used to manage our infrastructure buildbotcustom, buildbot-configs, tools and mozharness. From the mozharness FAQ in Aki's words "Mozharness is a configuration-driven python script harness with full logging that allows production infrastructure and individual developers to use the same scripts."  Implementing mozharness scripts allows developers who don't have access to our infrastructure to run the same scripts we do on production hardware.   

Traditionally, the scripts that define the buildbot actions for our tests have been defined in the buildbotcustom project.  The code is very convoluted and it is difficult to new people to parse what bits of code apply to each platform.   Also,  every time we want to deploy changes for them we have to do a reconfig.  A reconfig is an operation where we upgrade our buildbot masters to the latest version of the code in the buildbotcustom, buildbot-configs and tools repositories.  A reconfig is usually initiated by the releng person on buildduty only a few times a week.  When you use mozharness, the buildbot config scripts point to the production branch of the mozharness repo. New changes can be deployed to production with a simpler merge to the production branch of mozharness.  No reconfig.

Closeup of zip line harness aka visual representation of some buildbotcustom code

 Image by Image ©timcaynes, http://www.flickr.com/photos/timcaynes/1588430260/under Creative Commons by-nc-sa 2.0

With mozharness scripts, you have a discrete script and config file that applies to each platform.  For instance, for panda android unittests, the config file is here, and the corresponding script is here. 

There are actions in mozharness that serve as boilerplate code that can be reused to common actions when running tests, such as installing Python packages into a virtual environment, closing repositories, downloading and extracting files and so on.  For instance, here are the default actions when running the android unit tests on Pandas:

These actions correspond to methods is classes that that your mozharness scripts will inherit. For instance, the PandaTest class in the mozharness script in inherits VirtualEnvMixin which has a method create_virtualenv to install the appropriate python modules for the test run.

 

The request-device action uses the mozpool client code within mozharness to interact with the mozpool and verify that the device is in a usable state to run tests. You can also override these methods with ones in your own mozharness script if you have changes that are unique to your platform.

If you are have questions regarding mozharness, please join the mozharness channel on irc.mozilla.org :-)

Further reading
Dustin Mitchell's description of Mozpool
Mozpool API documentation
Aki Sasaki's Mozharness FAQ 
Aki Sasaki's first blog post on why mozharness and more

Read more...

Mozilla Summit thoughts

The first weekend of October, I attended the Mozilla Summit in Toronto.  This was the first time I attended a summit since they only occur every few years.  The summit is a gathering of the Mozilla community, both paid contributors and volunteers.  Unlike other open source conferences I've attended,  in that the focus wasn't solely on the amazing new products and initiatives we are working on.  The major purpose of the summit was on the how to improve the open web and community as a whole, attract new contributors, and plan for the future. It was wonderful so talk to so many people in person that I only normally communicate with via IRC, as well as meet new people.

I really enjoyed the conference and would like to thank the organizers for all the hard work that went into it.  Conference organizing is an all consuming task and I'd like to thank those that make the summit happen, and run so smoothly.   Many people have already written about the amazing technical achievements that were unveiled, here's a good summary.  Coop blogged about the summit from the perspective of release engineering.  Many things struck me about the conference from a community standpoint.

There were many more diverse faces then I've seen at other open source conferences.  Lots of young people, even some teenagers who were excited to be contributing to the Mozilla community.  People from South America, Asia and Africa.  There were also a larger percentage of women than I've ever seen at an open source event.  I always joke that one of the good things about being a woman in open source is that there never is a lineup for the bathroom at break times.  For this conference, I had to wait in line once,  behind a colleague wearing an Ada Initiative shirt.  I had meals where I wasn't the only woman at the table, this was very refreshing.  I'm the only woman out of 18 people on the Mozilla release engineering team.  My colleagues are all great but at the same time it was amazing to meet and speak with so many talented women who are passionate about open source.

There were keynotes every morning and the remainder of the slots were working sessions where people could come together around a common topic and work together to formulate ideas to make things better for the future.  For instance, I attended workshops on topics such as organizing your project for contribution.  One of the interesting points that David Eaves and Mike Hoye brought up in this session is that the number one predictor that a contributor will continue to make contributions is if there initial bug is triaged into to the correct bucket and they receive feedback.  The faster they receive feedback, even if the feedback is that the bug won't be looked at for two weeks, the more likely it is that they will make another contribution. I also attended a session on how to scale our community to larger numbers which included discussions regarding the research of volunteer organizations scale, educational theory and how to maintain volunteer engagement.   Very different from my usual work topics of how to scale infrastructure and build capacity but very interesting the same.

Liz Henry set up a table where you could make jewellery with your favourite bug number or your irc nick.  What a fantastic idea, thank you Liz.

Saturday night had a lot of fun social events including a hockey (shinny) game at Maple Leaf Gardens where Potch served as announcer.  Thank you Lawrence Mandel and Lukas Blakk for organizing, it was amazing.  A couple of the people in the game were pretty new to skating and had never played hockey before but they did fine! In the end it was a pretty close game.

On the way back to the hotel I had the chance to explore the art of Nuit Blanche with some colleagues.  It was beautiful.

Ai Weiwei's Toronto installation of Forever Bicycles in Nathan Phillips Square.

I've never been to a conference where the focus is so much on "what can we do next" instead of "what we are doing now".  I wonder if other open source communities do the same. What did you enjoy most about the Mozilla Summit?

Read more...

Meeting little Releng

Seven years ago,  Mr. Releng and I filled out the paperwork for international adoption.  As anyone who has pursued adoption is aware, this path is often very long and fraught with unforseen obstacles, no matter the country where you apply.   So for a very long time, it seemed that everything that could go wrong, did go wrong and made for a very long wait.  But this summer, we travelled to Bangkok, Thailand and something went very right.

Our son aka little Releng

I may be biased, but he's pretty awesome.  I didn't realize how amazing it is to have a child in your life, it is such a gift.  In Canada, you can take up to nine months parental leave for adoption. Mr. Releng is taking the first half, while I'll take the second half. 

As an aside, we have never visited any countries in Asia before and so our visit to Thailand was quite an adventure.  We spent most of the time in Bangkok, so I can't comment on the rest of the country.  But it's quite an extraordinary city with beautiful sights, very friendly people and delicious food.  Even though it's literally half way around the world from here,  I would recommend as a fascinating place to visit.  Bangkok feels like it is in the future, with it's gleaming tall buildings,

yet at the same time it has an amazing past to explore as well.

Statue at Grand Palace

Read more...