Enabling tests that ride the trains

>> Wednesday, April 02, 2014

At Mozilla, we have a train schedule for releases.  Changes are first landed on a trunk branches, then are uplifted to other branches for stabilization.  Johnathan Nightingale has a very good blog post that explains this concept.   For instance, the usual route for a change to land from a trunk branch such as mozilla-central would be to uplift to mozilla-aurora, mozilla-beta, and then finally mozilla-release where it  would be available to users in the next release. 

Merge day, which occurs every six weeks is when changes are uplifted to the next stable branch. Here's a picture from a talk John O'Duinn gave last year that shows an example of how changes move between branches1.

Picture from John O'Duinn's Release Engineering as a Force Multiplier talk at Releng 2013

For the release engineering team, each merge day we would update code in our buildbot configs to reflect changes that needed to be made after uplift.  For instance, we often deprecate platforms or add new tests that only apply to certain branches.  We used to have to to specify the branches that these applied to in our configs and update them every merge day.  It was tricky to get this right.   Last fall, Steve Fink fixed a bug that would allow us to base config versions on the version of gecko that rides the trains.  So each merge do we update the version of gecko in our configs on a per branch basis, and then have code like this so that the tests are only enabled for branches where the gecko version applies

Enable jittests for desktop where gecko is 31 or more

Load jetpack where gecko is at least 21

To test these changes, you can set up your buildbot test master and run builder_list.py against your master.   The builder_list.py script will output the list of build/test jobs (builders) that are enabled on your master.  Then apply your patch against the master and diff the resulting builder files to ensure that the tests are enabled in the branches you want.  As a side note, if you are enabling tests on platforms that would be on different test masters, you'll have to configure your master for mac, linux and windows test masters and diff the builders for each platform. If you are enabling tests on trunk trees for the first time, your diff should not reveal any new builders on mozilla-aurora, mozilla-beta, mozilla-release but just on mozilla-central, mozilla-inbound and the associated twig branches.
I recently fixed a few bugs where there was a request enable tests on trunk branches and ride the trains, so I thought I'd write about if others had to implement a similar request.

Train-related graffiti in Antwerp (Belgium), near Antwerpen-Centraal train station by  ©vitalyzator, https://flic.kr/p/6tQ3H Creative Commons by-nc-sa 2.0

Further reading and notes
1 This applies to Firefox Desktop and mobile only, Firefox OS is on a different cadence and there are different branches involved
Release Management's rapid release calender
Release Engineering Merge duty
Release Engineering Testing Techniques


Schooling yourself in release engineering

>> Monday, March 31, 2014

Traditionally, there haven't been many courses offered in colleges or universities that cover the fundamentals of release engineering.  This means that students don't get exposed to the potential that a career in release engineering has to offer.  Conversely, it also doesn't provide students who become employed in more traditional developer roles the background regarding the complexity and challenges that arise within the scope of  release engineering.  However, this is beginning to change which is fantastic!  For example:

Release Engineering as a Discipline,  Center of Computer Science, RWTH Aachen University in Aachen Germany

Overview of the Build and Release Process, (updated link) Seneca College, Toronto

Release Engineering -- Applications of Mining Software Repositories, École Polytechnique, Montréal

Software Release Planning, University of Calgary

Seneca College Library Image ©moqubhttps://flic.kr/p/9PyVVm Creative Commons by-nc-sa 2.0

If anyone knows of other courses that are offered, I'd love to hear about them.  Maybe someday I won't have to explain to new people I meet what a release engineer does all day.  Just kidding, this will still happen :-)


Upcoming Release Engineering events

>> Wednesday, March 26, 2014

University of Waterloo engineering students in 1964 Image ©Kitchener Waterloo Record, http://www.flickr.com/photos/48169267@N08/4967256177/in/photolist-8yWucT-eqaQdV-epd6Lw-c5ywEJ-c5yvwd/under Creative Commons by-nc-sa 2.0
There are quite a few upcoming events related to release engineering so I thought I'd list them here.

The CFP for LISA is open, submissions due April 14.  Release engineering topics are welcome. LISA is November 9–14, 2014, in Seattle, WA.

There is also a Release engineering workshop as part of Usenix Federated Conferences week, on June 20, 2014, in Philadelphia, PA.  The CFP has closed, but I think you can still register.

The Releng 2014 workshop is April 11, at Google in Mountain View.  We were really overwhelmed by the number of people were interested in registering for this workshop, and the event is now sold out.  A few of the talks will be recorded, so if you couldn't get a ticket, they will be available online after the event. 

Finally, there's the first IEEE Special Issue on Release Engineering.   The deadline for submission of a paper is August 1, 2014.  Not an event, but a great place to get a paper on your area of expertise published.

Any other events I missed?


EclipseCon 2014 trip report

>> Monday, March 24, 2014

I attended EclipseCon for a couple of days last week.  It was great to see friends and colleagues again and learn what they were working on. 

Some highlights for me:

My favourite presentation was by Tamar Cohen who presented on the Verve software that is used to provide 3D visualization of remote environments using NASA robots.  NASA + robots = I could listen all day.  The next day, I had to opportunity to sit at lunch with Tamar.  I mentioned to her that I think she says that she has one of the best jobs in the world, building software for robots some of which will leave our planet, and she agreed :-)  She also made an interesting point from a maintenance perspective, in that most of the software that her team writes for experiments by astronauts is used for the duration of the experiment and then discarded. So there aren't really any long term release engineering requirements for most of their software.

  Tamar Cohen

I really enjoyed the keynote by Catarina Moto on open hardware materials.  Here's a link to many of the videos she used during her talk.  I was especially impressed by the discussion that showed the robotic hand made by open hardware components (more info here).  Such a fine example of how technology can be a force for good in the world.

Catarina Moto talking about open hardware materials

I also really liked Andrew Low's talk on Porting NodeJS to Linux for PPC.  He's a great speaker and  I like porting stories :-)  Nick Stinemates talk on Docker was really interesting for me too, because we are currently investigating using it in some capacity.

On Wednesday, I was happy to present an overview of Mozilla release engineering from both a technical and human perspective.  The slides are here.

The pdf on the EclipseCon website seems to be clearer.  I think slideshare does some compression that causes the images to be a bit fuzzy.

In any case, I received some good feedback on the talk after I spoke, and via twitter and email.  Many people were impressed by both the scale of builds and tests we run, and the tooling we use to manage it.  So big kudos to the Mozilla Releng team and all the others we work with like ATeam and IT that allowed me to have great stories to tell.  There were about 40 people who attended the talk.  As an interesting anecdote, I asked how many people developed in Python in the talk and two hands went up.  As release engineers at Mozilla, we spend most of our days immersed in developing Python code, so this was interesting.  By the number of people in the Java talks at EclipseCon, it is clear that this the Eclipse community continues to be very Java focused.

At the end of my talk, Ian Bull asked an interesting question.  He said something along the lines of "If you had to do it all over again, how would you change things at Eclipse to make it better from a  release engineering standpoint?" (I'm paraphrasing, I'm jetlagged and this was a few days ago).

Ian Bull always asks great questions

I responded that it didn't really matter what I thought, the Eclipse community doesn't make release engineering a priority and allocate resources to it so it wouldn't change, no matter what I thought or did.  Every open source community makes different decisions based on their priorities.  If they want changes, they have to allocate resources, whether they be people, or money or both, to make these priorities happen.  No resources, nothing gets done.  In much the same vein that I'm always surprised that conference organizers reach out to try to get a more diverse speakers along gender/POC/geographic/sexuality etc lines when the CFP is announced, but the rest of the year nobody in the community is championing diversity.  Again, no priorities, no resources, no change.

Thanks to everyone who made EclipseCon happen, it was an interesting conference!


Built to Scale talk at EclipseCon

>> Monday, March 10, 2014

I'm honoured to be giving a talk at EclipseCon next week entitled Built to Scale: The Mozilla Release Engineering Toolbox.  To give you some context, here are some numbers about the scale of build and test jobs we run.

We run about 6000 build jobs and 50,000 test jobs every week day.  Each test job has many actual test suites within it of course.  We have 1800+ devices to build on, plus 3900+ for tests.  Some devices reside in our data centres, some reside in AWS.  When a developer lands (commits) a change, our goal is to have the relevant job start within 15 minutes of being added to the scheduler database.

My talk will discuss we manage this scale of continuous integration in terms of hardware and software.  Also, I'll touch on how we manage this from a human perspective, because that isn't easy either.  I'll also discuss some of the lessons along the way as we have moved many of our infrastructure to AWS.  And I'll also describe how we manage our 1000+ mobile devices that we run tests on as part of our CI farm.

Image ©ardonik, http://www.flickr.com/photos/ardonik/3954691105/sizes/l/ under Creative Commons by-nc-sa 2.0 Release engineering at this scale has lots of pieces to fit together.

In preparing this talk, I have been thinking a lot about the audience.  The audience will be people in the Eclipse community, who don't have a lot of context about how we do things at Mozilla. I recently read the book Resonate by Nancy Duarte which describes how to create great visual presentations and story arcs as a speaker.  One of the ideas in the book is that the most important thing that you can do as a speaker is think about your audience, what they know, and how to engage them. 

I use the Presentation Zen approach when preparing a talk which means that I write out all the topics on index cards, arrange them, rearrange them, and discard non-essential content.  Before touching a computer.  When I was initially preparing the talk, I had an entire index card of Mozilla specific words that I would have to explain.  It was ridiculous.  Nobody would ever remember the context of those terms from one slide to the next. I put that card in the shredder.

Last week, I thought of a new approach to present my talk.  I think it will really work.    I want to make the talk as interesting and relevant to the Eclipse community as it would be as if I gave it to a room full of Mozillians who have more context.

So this is what I know about the audience for my talk
You are Eclipse community members
Like all developers, you have known the pain of slow builds and test results.
You'd like to know how to scale large amounts of hardware and software.
And how things can get better.
So you can work on optimizing your product, and not be frustrated by your build and release process.

If you have specific questions you'd like me to address in the talk, please let me know in the comments or via twitter (@kmoir).  Looking forward to seeing you all at EclipseCon!

1 I also recently read Why Don't Students Like School: A Cognitive Scientist Answers Questions About How the Mind Works and What It Means for the Classroom which is an excellent book and extremely applicable to people teaching programming languages and other abstract concepts.  One of the topics that stuck with me from reading this book is that our brains need to have a lot of simple concepts memorized to understand more complex concepts. For instance, if you don't have your multiplication tables memorized, simple algebra will be difficult because you will have to stop and think what the value of 3x is when x is 7 instead of just pulling that from memory.  So this is why when people complain that schools teach a lot of memorization and not more abstract thinking, it's not really a valid argument.  You need a lot of concepts memorized before you can do more abstract thinking.  Highly recommended book.

John O'Duinn gave a talk at the Releng 2013 workshop last year and later as a Google Tech Talk that gives a great overview of why release engineering is a high priority at Mozilla.   Well worth watching.


Early Registration for Releng 2014 now open

>> Monday, February 17, 2014

We are pleased to announce that early registration for the Releng 2014 workshop is now open.  We are also very excited to announce that we have two great invited speakers lined up - release engineers Dinah McNutt of Google and Chuck Rossi of Facebook.  Register now!  We also encourage you to submit a talk, paper or poster of your own! The CFP closes February 28


Releng 2014 CFP now open

>> Friday, January 24, 2014

Last year I was on the organizing committee for a one day workshop on release engineering in San Francisco as part of ICSE.  We had so much positive feedback that we have organized another one this year.  This year's workshop will be held on April 11, at Google in Mountain View, California.  If you are interested in submitting a talk or paper, the deadline for the CFP is February 28.  More details on the website

Here's what people liked about last year's workshop

"Brought release engineers together in the same room! So awesome. Great talks and interaction between researchers and practitioners."

"The information sharing and the candidness of the speakers with their organizational challenges."

"Overall, I loved this opportunity and you all did a wonderful job putting this on. THANK YOU!"

"Specific stories of how some companies improved their release engineering. Being able to network and ask questions of presenters and people working on the same problem domain."
"The huge variation in scale that things are being done across the industry. "
"Informal structure led to lots of hallway conversations"

Image ©stuckincustoms, http://www.flickr.com/photos/stuckincustoms/351212669/sizes/m/under Creative Commons by-nc-sa 2.0 This picture of the international airport in Bangkok has nothing to do with release engineering, but has interesting architecture.

As was the case last year, there will be both release engineers and researchers in attendance.  This should generate a lot of interesting discussions as we talk about how we have succeeded and failed in building, deploying and testing software often at a large scale and what further research is needed. But if you work at a small company too we would love to hear your stories too.  One note of feedback that we received last year was that people would like to hear from a variety of companies, not just large ones.

In parallel to the workshop, a call will be launched for the first ever IEEE Software Special Issue on Release Engineering.  So submit a talk or paper, polish it with feedback from the workshop, and submit it to the special issue for possible publication.

I'm happy to help if you have any questions regarding the submission process or the workshop in general. Please feel free to drop me a line (I'm kmoir and I work at mozilla dot com).

More information regarding from last year's workshop:

Release Engineering as a Force Multiplier - John O'Duinn 
Releng 2013 recap - Kim Moir

As an aside, there is another release engineering event as part of Usenix on June 20 in Philadelphia with an open CFP too.   As Dinah McNutt, the organizer of this event remarked to me "It's going to be a great year to be a release engineer".


  © Blogger template Simple n' Sweet by Ourblogtemplates.com 2009

Back to TOP