Enabling tests that ride the trains

At Mozilla, we have a train schedule for releases.  Changes are first landed on a trunk branches, then are uplifted to other branches for stabilization.  Johnathan Nightingale has a very good blog post that explains this concept.   For instance, the usual route for a change to land from a trunk branch such as mozilla-central would be to uplift to mozilla-aurora, mozilla-beta, and then finally mozilla-release where it  would be available to users in the next release. 

Merge day, which occurs every six weeks is when changes are uplifted to the next stable branch. Here's a picture from a talk John O'Duinn gave last year that shows an example of how changes move between branches¹.

Picture from John O'Duinn's Release Engineering as a Force Multiplier talk at Releng 2013

 
For the release engineering team, each merge day we would update code in our buildbot configs to reflect changes that needed to be made after uplift.  For instance, we often deprecate platforms or add new tests that only apply to certain branches.  We used to have to to specify the branches that these applied to in our configs and update them every merge day.  It was tricky to get this right.   Last fall, Steve Fink fixed a bug that would allow us to base config versions on the version of gecko that rides the trains.  So each merge do we update the version of gecko in our configs on a per branch basis, and then have code like this so that the tests are only enabled for branches where the gecko version applies

......
Enable jittests for desktop where gecko is 31 or more

Load jetpack where gecko is at least 21

To test these changes, you can set up your buildbot test master and run builder_list.py against your master.   The builder_list.py script will output the list of build/test jobs (builders) that are enabled on your master.  Then apply your patch against the master and diff the resulting builder files to ensure that the tests are enabled in the branches you want.  As a side note, if you are enabling tests on platforms that would be on different test masters, you'll have to configure your master for mac, linux and windows test masters and diff the builders for each platform. If you are enabling tests on trunk trees for the first time, your diff should not reveal any new builders on mozilla-aurora, mozilla-beta, mozilla-release but just on mozilla-central, mozilla-inbound and the associated twig branches.
 
I recently fixed a few bugs where there was a request enable tests on trunk branches and ride the trains, so I thought I'd write about if others had to implement a similar request.

Train-related graffiti in Antwerp (Belgium), near Antwerpen-Centraal train station by  ©vitalyzator, https://flic.kr/p/6tQ3H Creative Commons by-nc-sa 2.0

Further reading and notes
¹ This applies to Firefox Desktop and mobile only, Firefox OS is on a different cadence and there are different branches involved
Release Management's rapid release calender
Release Engineering Merge duty
Release Engineering Testing Techniques

Read more...