Releng 2014 invited talks available

On April 11,  there was a Releng workshop held at Google in Mountain View. The two keynote talks and panel at the end of the day were recorded and made available on the Talks at Google channel on YouTube.  Thank you Google!


Moving to mobile: The challenges of moving from web to mobile releases, Chuck Rossi, Facebook https://www.youtube.com/watch?v=Nffzkkdq7GM

Some interesting notes from Chuck's talk:

• Facebook has more users on their beta apps than most companies have on their release apps.  Number one app for IOS, number one non-Google app for Android. They have more beta users on Android than most people have users.
• There are three things we look act as software developers: features, quality, schedule.  As release engineers we focus on quality and schedule.  Because we ship on time.  New features can go in the next mobile release which is only four weeks in the future.
• Developers have karma, if you release to many bad changes you lose karma.  Developers start out with four Karma stars.  You can only see you own karma.  The release engineering team can reduce your karma if you release too many breaking changes.
• The release engineering team has a lot of respect within Facebook, they are the ones who determine if your feature gets in a release.   He notes that you should make sure that you work for a organization that respects you for your judgement and skillset as as release engineer. As an aside, it seems the release engineering at Facebook has responsibility that corresponds to release engineering + release management at Mozilla.
• Compute power and storage are infinite at Facebook for their build and test infrastructure. It's not a contraint. 
• They run 28 - 30,000 builds a day.  Wow.
• They have a ui for developers to look at their build and test results called shrubbery. It looks like a fork of Mozilla's tbpl (Facebook's release engineering team has Mozilla alumni :-) They also run Buildbot as their continuous integration engine too.
• Like Mozilla they have sheriffs to monitor builds but this role is rotated among the developer teams.  If you put developers on call, they will make it less likely that they will get paged by being more careful when releasing changes because they share your operational burden.
• They have 17 apps to upload to the Google Play store.  In the presentation, he asked Google to provide and API to be able to do so. Right now they have to a person interact with the UI and press buttons for all these apps they need to upload. Very painful.  
• 15 minutes to deploy a new (web) facebook.com. Of course, this doesn't happen all at once.  They deploy to a certain percentage of their users, and evaluate the data and then deploy to more users.

 
The 10 Commandments of Release Engineering,  Dinah McNutt, Google

https://www.youtube.com/watch?v=RNMjYV_UsQ8

Some notes from Dinah's talk. 

• Release engineering is accelerating the path for development to operations
• You want to be able to reproduce your build environment and source code management system if you have to recreate a very old build
• Configuration management and release engineering as disciplines will probably merge as the next few years
• Reproducibility is a virtue. Binaries don't belong in SCMs.  However, it's important to be able to reproduce binaries.  If you do need a repo with binaries, put them in a separate repo. 
• Use the right tool for the job, you will have multiple tools.  Both commercial and open source.
• View the job of a release engineer and making a developers job easier.  By setting up tooling and best practices.
• Package management.  Provides auditing, upgrading, installation, removals. Tars and jars are not package managers.
• You need to think about your upgrade process before you release 1.0.
• Customers find problems we cannot find ourselves. Even if we're dogfooding.
• As release engineers, step back and look at the big picture.  Look and see how we can make things better from a cost perspective so we have the resources we need to do our jobs.
• It's a great year to be a release engineer. Dinah is the organizing committee for Release Engineering Summit June 20 in Philadelphia as part of USENIX. There is also one as a part of LISA in Seattle in November.  Overwhelming interest for a first time summit in terms of submissions!

Closing discussion panel:
https://www.youtube.com/watch?v=D4jLlWdsrWg

Stephanie Bellomo, one of my colleagues from the organizing committee for this workshop moderated the panel.  Really interesting discussions, well worth a listen.  I like that the first question of "What is your worst operational nightmare and how did you recover from it?"  I love war stories.

As an aside, we charged $50 per attendee for this workshop.  We talked to other people who had organized similar events and they suggested this would be an appropriate fee.  I've read that if you don't charge a fee for an event, you have more no-shows on the day of the event because psychologically,  they attach a lesser value to the event since they didn't pay for it.  However, we didn't have many expenses to pay for the workshop other than speaker gifts, eventbrite fees and badges.  Google provided the venue and lunch, again, thank you for the sponsorship. So we donated $1,531.00 USD to each of the following organizations from the remaining proceeds.

YearUp is an organization that in their words "Year Up empowers low-income young adults to go from poverty to professional careers in a single year."  I know Mozilla has partnered with YearUp to provide mentoring opportunities within the IT group and it was an amazing experience for all involved. 
 
The second organization we donated to is the Tanzania Education Fund.  This organization was one that Stephany mentioned since she had colleagues that were involved with for many years.  They provide pre-school, elementary, secondary and high school education for students in Tanzania.   Secondary education is not publicly funded in Tanzania.  In addition, 50% of their students are girls, in an area where education for girls is given low priority.  Education is so important to empower people.

Thanks to all those that attended and spoke at the workshop!

Read more...

Remote work in review

I recently read two books on remote work.  Scott Berkun's Year without Pants and Remote: Office Not Required by Jason Fried and David Heinemeier Hansson.  The first book describes Scott Berkun's year as a remote manager at Automattic, which runs Wordpress.com.  The second book describes the authors' experiences at the fully distributed company 37signals, which has since renamed itself to Basecamp to reflect the importance of its flagship product.  Both books were interesting reflections on the nature of remote work in the context of their respective companies.  They were not books that addressed the nature of remote work in general, but described the approach they felt was successful within their companies. Of the two books, I would really recommend reading the Year Without Pants.  Remote isn't as compelling but it's a short read.

Some notes from "The Year without Pants" 


Communication
"To work at a remote company demanded great communication skills and everyone had them"

Culture
Chapter 4 on culture always wins is  is a fantastic read, it's available for free on his website. 

"Trust is everything". 

"1. Hire great people
 2. Set good priorities
 3. Remove distractions
 4. Stay out of the way"

In other words, treat people like grown ups and they will do good work.
 
"In every meeting in every organization around the world where bad behavior is happening, there is someone with the most power in the room who can do something about it.  What that person does shapes the culture.  If the most powerful person is silent, this signals passive acceptance of whatever is going on."

Wow, this is very significant.  If you are the most powerful person in the room, speak up and call out bad behaviour.  The people with less power are often hesitant to speak up because there may be consequences for them and they feel they lack authority.

 Hiring
"Hire self sufficient, passionate people"

I often get questions from people who don't work at home how I don't get distracted and goof off all day since I work from home.  It's simple.  I love my job.  It's a lot of fun.  I want to be shipping, not slacking.

Shipping
Shipping every day gives people a sense of accomplishment.  Many bug fixes are deployed to Wordpress.com every day. No gatekeepers to deployment but the people deploying the change are expected to watch the site after they deploy for a few hours to ensure there aren't unexpected problems.


Some notes from "Remote: Office Not Required"

Communication
The book suggests asking managers or employees to work from home a few days a week to level the playing field with respect to communications between employees who work in an office and those who are remote.   This will ensure they appreciate what hinders communication and take steps for improvement.  And it will reduce the tendency to treat remote workers as second class citizens and cut them out of essential conversations.

Culture
"...coming  into the office just means that people have to put on pants.  There is not guarantee of productivity."

"If you view those who work under you as capable adults who will push themselves to excel even when they're not breathing down their necks, they'll delight you in return"

Again, trust people and have high expectations of them and you'll be rewarded with excellence. 

Hiring
The authors note that international exposure is good as a selling point with clients.  Hiring around the world increases the talent pool available, but is not without tax or legal complications.  Also, given the degree of written communication with remote work, it's best to hire people with the language skills that can thrive in this situation.  

Productivity
The book stresses that workflow tools need to be available to all team members at all times in order to be productive. i.e. recording the state of a project in a wiki, bug tracker and recording meetings.  If a team member is working in a timezone offset from the majority of team members and doesn't have this in place, it can be a productivity drain.

 "Would-be remote workers and managers have a lot to learn from how the open source software movement has conquered the commercial giants over the past few decades. Open source is a triumph of asynchronous collaboration and communication like few the world has ever seen."

Absolutely.  I learned so much working in open source for so many years.

The authors also mention that you'll have to worry about your employees overworking, not underworking.  Because they office is physically in your home, it's easy to get sucked in at all hours to just work on one little thing that takes longer than you expect.


My thoughts on remote work
If you had asked me five years ago  if I ever thought I'd work full time from my home my answer would have been a definitive no.  But I wanted to work for Mozilla, and wasn't interested in moving to an city where they had a physical office. Here are my personal suggestions for working successfully on a remote team, given my two years of experience being a part of one:

• Work for a company with strong culture, tools and process that support remote work.  Mozilla is a great example, there are many others.  It's not optimal to be the only remote worker in a company that has no experience or culture that supports distributed collaboration.
• Get up, shower, eat breakfast and put on clothes you wouldn't be ashamed to seen in outside the house.  Despite the title of the book I reviewed above, this means wearing pants.
• I think it's helpful that I have people on my timezone that I collaborate with on regular basis.  I think I would find my job much more difficult if this was not the case.
• Do stuff to get out of the house on a regular basis - visiting friends, family or regularly scheduled activities such as sports or hobbies.  Otherwise, you will become, as Laura Thomson so eloquently put it a "lonely code hermit".  This isn't healthy.
• Talk about stuff not directly related to code with your coworkers so you have some rapport for them as people outside the context of work.
• Meet up with your team in person on a regular basis.  At Mozilla releng, we do this three or four times a year.  As much as irc, bugzilla, wikis, video conferencing and other tools are great for collaboration, there isn't anything that can substitute for talking in person.  Also, it's great for getting to know them more as people.  And when you know them as people, you are more willing to help each other and feel more cohesive as a team.
• Ship.  If you are worried that you don't appear to be productive because you aren't physically in the office, ship awesome stuff.  Then nobody cares where you're sitting and you get to have a 30 second commute.
• If you work in software, you aren't constrained by geography to determine the kind of job you can have.  If you don't want to work for the software companies that have offices in the town you live in, you have a choice to choose a better job.  Life is too short to get up every day and work at a job you don't like.   There's a lot  opportunity for those who work in many fields to work remotely.

As aside, I was supposed to be at a Mozilla work week in Portland this past week.  But I didn't fly there because I came down with a bad cold.  Despite this, I could connect to the same room the they were in and see all the talks they were giving and also give a presentation.  This was so excellent.  Since we are so used to being a distributed team, having one person remote when we were supposed to be all together wasn't a problem.  We already had the culture and tools in place to accommodate this. Thank you Mozilla releng for being such an amazing team to work with.

My idea for another book on remote work would be to have one with the format where there were about 10+ chapters, each written by an employee from a different company about how they approach it, what tools they use and so on. I think this could be a very interesting read.

I'll close with some thoughts from Scott Berkun's book on whether remote work is for everyone 

"For me,  I know that for any important relationship I'd want to be physically around that person as much as possible. If I started a rock band or a company, I'd want to share the same physical space often.  The upsides outweigh the downsides.  However, if the people I wanted to work with were only available remotely, I'm confident that we could do great work from thousands of miles away."

What do you think are the keys to successfully working as a remote employee?

Further reading
John O'Duinn's We are all Remoties Talk

Read more...

Enabling tests that ride the trains

At Mozilla, we have a train schedule for releases.  Changes are first landed on a trunk branches, then are uplifted to other branches for stabilization.  Johnathan Nightingale has a very good blog post that explains this concept.   For instance, the usual route for a change to land from a trunk branch such as mozilla-central would be to uplift to mozilla-aurora, mozilla-beta, and then finally mozilla-release where it  would be available to users in the next release. 

Merge day, which occurs every six weeks is when changes are uplifted to the next stable branch. Here's a picture from a talk John O'Duinn gave last year that shows an example of how changes move between branches¹.

Picture from John O'Duinn's Release Engineering as a Force Multiplier talk at Releng 2013

 
For the release engineering team, each merge day we would update code in our buildbot configs to reflect changes that needed to be made after uplift.  For instance, we often deprecate platforms or add new tests that only apply to certain branches.  We used to have to to specify the branches that these applied to in our configs and update them every merge day.  It was tricky to get this right.   Last fall, Steve Fink fixed a bug that would allow us to base config versions on the version of gecko that rides the trains.  So each merge do we update the version of gecko in our configs on a per branch basis, and then have code like this so that the tests are only enabled for branches where the gecko version applies

......
Enable jittests for desktop where gecko is 31 or more

Load jetpack where gecko is at least 21

To test these changes, you can set up your buildbot test master and run builder_list.py against your master.   The builder_list.py script will output the list of build/test jobs (builders) that are enabled on your master.  Then apply your patch against the master and diff the resulting builder files to ensure that the tests are enabled in the branches you want.  As a side note, if you are enabling tests on platforms that would be on different test masters, you'll have to configure your master for mac, linux and windows test masters and diff the builders for each platform. If you are enabling tests on trunk trees for the first time, your diff should not reveal any new builders on mozilla-aurora, mozilla-beta, mozilla-release but just on mozilla-central, mozilla-inbound and the associated twig branches.
 
I recently fixed a few bugs where there was a request enable tests on trunk branches and ride the trains, so I thought I'd write about if others had to implement a similar request.

Train-related graffiti in Antwerp (Belgium), near Antwerpen-Centraal train station by  ©vitalyzator, https://flic.kr/p/6tQ3H Creative Commons by-nc-sa 2.0

Further reading and notes
¹ This applies to Firefox Desktop and mobile only, Firefox OS is on a different cadence and there are different branches involved
Release Management's rapid release calender
Release Engineering Merge duty
Release Engineering Testing Techniques

Read more...

Schooling yourself in release engineering

Traditionally, there haven't been many courses offered in colleges or universities that cover the fundamentals of release engineering.  This means that students don't get exposed to the potential that a career in release engineering has to offer.  Conversely, it also doesn't provide students who become employed in more traditional developer roles the background regarding the complexity and challenges that arise within the scope of  release engineering.  However, this is beginning to change which is fantastic!  For example:

Release Engineering as a Discipline,  Center of Computer Science, RWTH Aachen University in Aachen Germany

Overview of the Build and Release Process, (updated link) Seneca College, Toronto


Release Engineering -- Applications of Mining Software Repositories, École Polytechnique, Montréal

Software Release Planning, University of Calgary

Seneca College Library Image ©moqub,  https://flic.kr/p/9PyVVm Creative Commons by-nc-sa 2.0

If anyone knows of other courses that are offered, I'd love to hear about them.  Maybe someday I won't have to explain to new people I meet what a release engineer does all day.  Just kidding, this will still happen :-)

Read more...

Upcoming Release Engineering events

University of Waterloo engineering students in 1964 Image ©Kitchener Waterloo Record, http://www.flickr.com/photos/48169267@N08/4967256177/in/photolist-8yWucT-eqaQdV-epd6Lw-c5ywEJ-c5yvwd/under Creative Commons by-nc-sa 2.0

 
There are quite a few upcoming events related to release engineering so I thought I'd list them here.

The CFP for LISA is open, submissions due April 14.  Release engineering topics are welcome. LISA is November 9–14, 2014, in Seattle, WA.

There is also a Release engineering workshop as part of Usenix Federated Conferences week, on June 20, 2014, in Philadelphia, PA.  The CFP has closed, but I think you can still register.

The Releng 2014 workshop is April 11, at Google in Mountain View.  We were really overwhelmed by the number of people were interested in registering for this workshop, and the event is now sold out.  A few of the talks will be recorded, so if you couldn't get a ticket, they will be available online after the event. 

Finally, there's the first IEEE Special Issue on Release Engineering.   The deadline for submission of a paper is August 1, 2014.  Not an event, but a great place to get a paper on your area of expertise published.

Any other events I missed?

Read more...

EclipseCon 2014 trip report

I attended EclipseCon for a couple of days last week.  It was great to see friends and colleagues again and learn what they were working on. 

Some highlights for me:

My favourite presentation was by Tamar Cohen who presented on the Verve software that is used to provide 3D visualization of remote environments using NASA robots.  NASA + robots = I could listen all day.  The next day, I had to opportunity to sit at lunch with Tamar.  I mentioned to her that I think she says that she has one of the best jobs in the world, building software for robots some of which will leave our planet, and she agreed :-)  She also made an interesting point from a maintenance perspective, in that most of the software that her team writes for experiments by astronauts is used for the duration of the experiment and then discarded. So there aren't really any long term release engineering requirements for most of their software.

  Tamar Cohen

I really enjoyed the keynote by Catarina Moto on open hardware materials.  Here's a link to many of the videos she used during her talk.  I was especially impressed by the discussion that showed the robotic hand made by open hardware components (more info here).  Such a fine example of how technology can be a force for good in the world.

 
Catarina Moto talking about open hardware materials

I also really liked Andrew Low's talk on Porting NodeJS to Linux for PPC.  He's a great speaker and  I like porting stories :-)  Nick Stinemates talk on Docker was really interesting for me too, because we are currently investigating using it in some capacity.


On Wednesday, I was happy to present an overview of Mozilla release engineering from both a technical and human perspective.  The slides are here.

Built to Scale: The Mozilla Release Engineering toolbox from Kim Moir

The pdf on the EclipseCon website seems to be clearer.  I think slideshare does some compression that causes the images to be a bit fuzzy.

In any case, I received some good feedback on the talk after I spoke, and via twitter and email.  Many people were impressed by both the scale of builds and tests we run, and the tooling we use to manage it.  So big kudos to the Mozilla Releng team and all the others we work with like ATeam and IT that allowed me to have great stories to tell.  There were about 40 people who attended the talk.  As an interesting anecdote, I asked how many people developed in Python in the talk and two hands went up.  As release engineers at Mozilla, we spend most of our days immersed in developing Python code, so this was interesting.  By the number of people in the Java talks at EclipseCon, it is clear that this the Eclipse community continues to be very Java focused.

At the end of my talk, Ian Bull asked an interesting question.  He said something along the lines of "If you had to do it all over again, how would you change things at Eclipse to make it better from a  release engineering standpoint?" (I'm paraphrasing, I'm jetlagged and this was a few days ago).



Ian Bull always asks great questions

I responded that it didn't really matter what I thought, the Eclipse community doesn't make release engineering a priority and allocate resources to it so it wouldn't change, no matter what I thought or did.  Every open source community makes different decisions based on their priorities.  If they want changes, they have to allocate resources, whether they be people, or money or both, to make these priorities happen.  No resources, nothing gets done.  In much the same vein that I'm always surprised that conference organizers reach out to try to get a more diverse speakers along gender/POC/geographic/sexuality etc lines when the CFP is announced, but the rest of the year nobody in the community is championing diversity.  Again, no priorities, no resources, no change.

Thanks to everyone who made EclipseCon happen, it was an interesting conference!

Read more...

Built to Scale talk at EclipseCon

I'm honoured to be giving a talk at EclipseCon next week entitled Built to Scale: The Mozilla Release Engineering Toolbox.  To give you some context, here are some numbers about the scale of build and test jobs we run.

We run about 6000 build jobs and 50,000 test jobs every week day.  Each test job has many actual test suites within it of course.  We have 1800+ devices to build on, plus 3900+ for tests.  Some devices reside in our data centres, some reside in AWS.  When a developer lands (commits) a change, our goal is to have the relevant job start within 15 minutes of being added to the scheduler database.

My talk will discuss we manage this scale of continuous integration in terms of hardware and software.  Also, I'll touch on how we manage this from a human perspective, because that isn't easy either.  I'll also discuss some of the lessons along the way as we have moved many of our infrastructure to AWS.  And I'll also describe how we manage our 1000+ mobile devices that we run tests on as part of our CI farm.

Image ©ardonik, http://www.flickr.com/photos/ardonik/3954691105/sizes/l/ under Creative Commons by-nc-sa 2.0 Release engineering at this scale has lots of pieces to fit together.

In preparing this talk, I have been thinking a lot about the audience.  The audience will be people in the Eclipse community, who don't have a lot of context about how we do things at Mozilla. I recently read the book Resonate by Nancy Duarte which describes how to create great visual presentations and story arcs as a speaker.  One of the ideas in the book is that the most important thing that you can do as a speaker is think about your audience, what they know, and how to engage them. 

I use the Presentation Zen approach when preparing a talk which means that I write out all the topics on index cards, arrange them, rearrange them, and discard non-essential content.  Before touching a computer.  When I was initially preparing the talk, I had an entire index card of Mozilla specific words that I would have to explain.  It was ridiculous.  Nobody would ever remember the context of those terms from one slide to the next.¹  I put that card in the shredder.

Last week, I thought of a new approach to present my talk.  I think it will really work.    I want to make the talk as interesting and relevant to the Eclipse community as it would be as if I gave it to a room full of Mozillians who have more context.

So this is what I know about the audience for my talk
You are Eclipse community members
Like all developers, you have known the pain of slow builds and test results.
You'd like to know how to scale large amounts of hardware and software.
And how things can get better.
So you can work on optimizing your product, and not be frustrated by your build and release process.


If you have specific questions you'd like me to address in the talk, please let me know in the comments or via twitter (@kmoir).  Looking forward to seeing you all at EclipseCon!



Notes:
¹ I also recently read Why Don't Students Like School: A Cognitive Scientist Answers Questions About How the Mind Works and What It Means for the Classroom which is an excellent book and extremely applicable to people teaching programming languages and other abstract concepts.  One of the topics that stuck with me from reading this book is that our brains need to have a lot of simple concepts memorized to understand more complex concepts. For instance, if you don't have your multiplication tables memorized, simple algebra will be difficult because you will have to stop and think what the value of 3x is when x is 7 instead of just pulling that from memory.  So this is why when people complain that schools teach a lot of memorization and not more abstract thinking, it's not really a valid argument.  You need a lot of concepts memorized before you can do more abstract thinking.  Highly recommended book.

John O'Duinn gave a talk at the Releng 2013 workshop last year and later as a Google Tech Talk that gives a great overview of why release engineering is a high priority at Mozilla.   Well worth watching.

Read more...

Early Registration for Releng 2014 now open

We are pleased to announce that early registration for the Releng 2014 workshop is now open.  We are also very excited to announce that we have two great invited speakers lined up - release engineers Dinah McNutt of Google and Chuck Rossi of Facebook.  Register now!  We also encourage you to submit a talk, paper or poster of your own! The CFP closes February 28. 

Read more...

Releng 2014 CFP now open

Last year I was on the organizing committee for a one day workshop on release engineering in San Francisco as part of ICSE.  We had so much positive feedback that we have organized another one this year.  This year's workshop will be held on April 11, at Google in Mountain View, California.  If you are interested in submitting a talk or paper, the deadline for the CFP is February 28.  More details on the website. 

Here's what people liked about last year's workshop

"Brought release engineers together in the same room! So awesome. Great talks and interaction between researchers and practitioners."


"The information sharing and the candidness of the speakers with their organizational challenges."


"Overall, I loved this opportunity and you all did a wonderful job putting this on. THANK YOU!"

"Specific stories of how some companies improved their release engineering. Being able to network and ask questions of presenters and people working on the same problem domain."

"The huge variation in scale that things are being done across the industry. "

"Informal structure led to lots of hallway conversations"

Image ©stuckincustoms, http://www.flickr.com/photos/stuckincustoms/351212669/sizes/m/under Creative Commons by-nc-sa 2.0 This picture of the international airport in Bangkok has nothing to do with release engineering, but has interesting architecture.

As was the case last year, there will be both release engineers and researchers in attendance.  This should generate a lot of interesting discussions as we talk about how we have succeeded and failed in building, deploying and testing software often at a large scale and what further research is needed. But if you work at a small company too we would love to hear your stories too.  One note of feedback that we received last year was that people would like to hear from a variety of companies, not just large ones.


In parallel to the workshop, a call will be launched for the first ever IEEE Software Special Issue on Release Engineering.  So submit a talk or paper, polish it with feedback from the workshop, and submit it to the special issue for possible publication.

I'm happy to help if you have any questions regarding the submission process or the workshop in general. Please feel free to drop me a line (I'm kmoir and I work at mozilla dot com).


More information regarding from last year's workshop:


Release Engineering as a Force Multiplier - John O'Duinn 
Releng 2013 recap - Kim Moir

As an aside, there is another release engineering event as part of Usenix on June 20 in Philadelphia with an open CFP too.   As Dinah McNutt, the organizer of this event remarked to me "It's going to be a great year to be a release engineer".

Read more...

Mozpool and Mozharness for Android Panda tests

Over the last few months, I migrated the test infrastructure that we run for Android 4.0 on Panda devices to use mozharness and mozpool. Before I go into a description of what this entailed, some background for those not familiar with our release engineering infrastructure.  We use Buildbot for our continuous integration engine. We have 50+ Buildbot masters are designated specific purposes such as scheduling, building, try servers and testing.  Test masters are further subdivided into ones specifically allocated to running tests on Mac, Linux, Windows, Tegras (Android 2.2 devices) and Pandas (Android 4.0 devices).  We have many devices that are allocated to each master to handle the volume of tests and builds we run each day. 

Why Mozpool?

We have over 800 panda devices in production that are used to run unit tests and talos (performance) tests for Firefox for Android 4.0.    Panda devices are rack mounted in a chassis.  One of the issues with these devices is that they are development boards and inherently quite unstable when running a large volume of tests 24/7.  Not many organizations run the volume of automated tests on mobile devices that we do.  Dealing with their issues in a non-automated fashion does not scale.

Pandas fall down

Mozpool is software that is used to mitigate that inherent stability and ensure that the devices that are available to run tests are in a verified state and have the correct Android image.  If Mozpool determines that the device has problems that makes it unsuitable to run tests, it changes its state so it won't be in the pool of devices running tests.  For instance, if it doesn't boot correctly or the requisite image cannot be applied. This reduces the number of jobs that fail due to infrastructure issues.  Mozpool also provides logging of the actions on devices and and a web page for looking at the state of your devices.  There is an API implemented vi REST and HTTP for simple actions on your devices, such as requesting a device, and returning it to the pool.  Another advantage of Mozpool is that it's easy to reimage the devices with a new image, either via the mozpool ui or simply specifying a new image in your mozpool client code.

Why Mozharness?
There are four main projects that are used to manage our infrastructure buildbotcustom, buildbot-configs, tools and mozharness. From the mozharness FAQ in Aki's words "Mozharness is a configuration-driven python script harness with full logging that allows production infrastructure and individual developers to use the same scripts."  Implementing mozharness scripts allows developers who don't have access to our infrastructure to run the same scripts we do on production hardware.   

Traditionally, the scripts that define the buildbot actions for our tests have been defined in the buildbotcustom project.  The code is very convoluted and it is difficult to new people to parse what bits of code apply to each platform.   Also,  every time we want to deploy changes for them we have to do a reconfig.  A reconfig is an operation where we upgrade our buildbot masters to the latest version of the code in the buildbotcustom, buildbot-configs and tools repositories.  A reconfig is usually initiated by the releng person on buildduty only a few times a week.  When you use mozharness, the buildbot config scripts point to the production branch of the mozharness repo. New changes can be deployed to production with a simpler merge to the production branch of mozharness.  No reconfig.

Closeup of zip line harness aka visual representation of some buildbotcustom code

 Image by Image ©timcaynes, http://www.flickr.com/photos/timcaynes/1588430260/under Creative Commons by-nc-sa 2.0

With mozharness scripts, you have a discrete script and config file that applies to each platform.  For instance, for panda android unittests, the config file is here, and the corresponding script is here. 

There are actions in mozharness that serve as boilerplate code that can be reused to common actions when running tests, such as installing Python packages into a virtual environment, closing repositories, downloading and extracting files and so on.  For instance, here are the default actions when running the android unit tests on Pandas:

These actions correspond to methods is classes that that your mozharness scripts will inherit. For instance, the PandaTest class in the mozharness script in inherits VirtualEnvMixin which has a method create_virtualenv to install the appropriate python modules for the test run.

 

The request-device action uses the mozpool client code within mozharness to interact with the mozpool and verify that the device is in a usable state to run tests. You can also override these methods with ones in your own mozharness script if you have changes that are unique to your platform.

If you are have questions regarding mozharness, please join the mozharness channel on irc.mozilla.org :-)

Further reading
Dustin Mitchell's description of Mozpool
Mozpool API documentation
Aki Sasaki's Mozharness FAQ 
Aki Sasaki's first blog post on why mozharness and more

Read more...

Mozilla Summit thoughts

The first weekend of October, I attended the Mozilla Summit in Toronto.  This was the first time I attended a summit since they only occur every few years.  The summit is a gathering of the Mozilla community, both paid contributors and volunteers.  Unlike other open source conferences I've attended,  in that the focus wasn't solely on the amazing new products and initiatives we are working on.  The major purpose of the summit was on the how to improve the open web and community as a whole, attract new contributors, and plan for the future. It was wonderful so talk to so many people in person that I only normally communicate with via IRC, as well as meet new people.

I really enjoyed the conference and would like to thank the organizers for all the hard work that went into it.  Conference organizing is an all consuming task and I'd like to thank those that make the summit happen, and run so smoothly.   Many people have already written about the amazing technical achievements that were unveiled, here's a good summary.  Coop blogged about the summit from the perspective of release engineering.  Many things struck me about the conference from a community standpoint.

There were many more diverse faces then I've seen at other open source conferences.  Lots of young people, even some teenagers who were excited to be contributing to the Mozilla community.  People from South America, Asia and Africa.  There were also a larger percentage of women than I've ever seen at an open source event.  I always joke that one of the good things about being a woman in open source is that there never is a lineup for the bathroom at break times.  For this conference, I had to wait in line once,  behind a colleague wearing an Ada Initiative shirt.  I had meals where I wasn't the only woman at the table, this was very refreshing.  I'm the only woman out of 18 people on the Mozilla release engineering team.  My colleagues are all great but at the same time it was amazing to meet and speak with so many talented women who are passionate about open source.

There were keynotes every morning and the remainder of the slots were working sessions where people could come together around a common topic and work together to formulate ideas to make things better for the future.  For instance, I attended workshops on topics such as organizing your project for contribution.  One of the interesting points that David Eaves and Mike Hoye brought up in this session is that the number one predictor that a contributor will continue to make contributions is if there initial bug is triaged into to the correct bucket and they receive feedback.  The faster they receive feedback, even if the feedback is that the bug won't be looked at for two weeks, the more likely it is that they will make another contribution. I also attended a session on how to scale our community to larger numbers which included discussions regarding the research of volunteer organizations scale, educational theory and how to maintain volunteer engagement.   Very different from my usual work topics of how to scale infrastructure and build capacity but very interesting the same.

Liz Henry set up a table where you could make jewellery with your favourite bug number or your irc nick.  What a fantastic idea, thank you Liz.

Saturday night had a lot of fun social events including a hockey (shinny) game at Maple Leaf Gardens where Potch served as announcer.  Thank you Lawrence Mandel and Lukas Blakk for organizing, it was amazing.  A couple of the people in the game were pretty new to skating and had never played hockey before but they did fine! In the end it was a pretty close game.

On the way back to the hotel I had the chance to explore the art of Nuit Blanche with some colleagues.  It was beautiful.

Ai Weiwei's Toronto installation of Forever Bicycles in Nathan Phillips Square.

I've never been to a conference where the focus is so much on "what can we do next" instead of "what we are doing now".  I wonder if other open source communities do the same. What did you enjoy most about the Mozilla Summit?

Read more...

Meeting little Releng

Seven years ago,  Mr. Releng and I filled out the paperwork for international adoption.  As anyone who has pursued adoption is aware, this path is often very long and fraught with unforseen obstacles, no matter the country where you apply.   So for a very long time, it seemed that everything that could go wrong, did go wrong and made for a very long wait.  But this summer, we travelled to Bangkok, Thailand and something went very right.

Our son aka little Releng

I may be biased, but he's pretty awesome.  I didn't realize how amazing it is to have a child in your life, it is such a gift.  In Canada, you can take up to nine months parental leave for adoption. Mr. Releng is taking the first half, while I'll take the second half. 

As an aside, we have never visited any countries in Asia before and so our visit to Thailand was quite an adventure.  We spent most of the time in Bangkok, so I can't comment on the rest of the country.  But it's quite an extraordinary city with beautiful sights, very friendly people and delicious food.  Even though it's literally half way around the world from here,  I would recommend as a fascinating place to visit.  Bangkok feels like it is in the future, with it's gleaming tall buildings,

yet at the same time it has an amazing past to explore as well.

Statue at Grand Palace

Read more...

Resources for learning Python

When I started working at Mozilla, I didn't know Python very well.  All of the code that drives our continuous integration is written in it so I've had a spent a lot of time becoming more proficient.  I've taken a number of free self-study classes/tutorials that others might find useful getting up to speed with Python.

Google Python class  - Two day class that describes how to manipulate strings, dictionaries, lists. File I/O, regular expressions and sorting algorithms.  Class lectures that were recorded at a Google training event are available on youtube, and the exercises are pretty straightforward with a solution key to check your work.  All the exercises are done using the Python installation on your local machine.

Code academy has a Python class.  I did the exercises for the topics that weren't covered in the Google class, including list comprehensions, lambda expressions and OO in Python.  The exercises are completed and validated on the website.

The best course I've taken so far has been the nine week Interactive Programming in Python class offered by Coursera. Coursera is a company that offers university courses for free.  You receive a grade and certification of completion,  but not an actual credit at the institution. The Python course I took is in conjunction with Rice University.  For this course, you have to register for the course when it's run and complete the work each week.   There are an hour or so of video lectures, a quiz or two and an assignment to complete for around 8-10 hours of work each week.  After you submit your assignment, you're asked to review the work of five other students.  This is how to scale marking assignments for the thousands of students enrolled in the class :-).  The code for your assignments is written in the browser,  on the CodeSkulptor website.

Syllabus for coursera course

Each week the assignment is to implement a different game which was a lot of fun.   We also learned how to implement the calculations for velocity, reflection and collisions to make the objects in our games appear as a realistic game.   The final project is to implement a version of the Asteroids game complete with missiles and exploding rocks.  I've also never done much with a the UI aspects of programming in the past, so it was interesting to learn how to incorporate these elements.

Screenshot of final asteroids game

This iteration of the course is almost over and the next one starts in October.  Essentially the course provides lots of time to practice writing code, which is the only path to getting better.  I found the course to be really well organized, and the video lectures were succinct in getting to the point of the lecture.  No profs going off on tangents about random subjects :-)

Mr. Releng also took this course at the same time I did as he is also using Python at work.  I said to him one day that I certainly wouldn't wanted to have attended all my university courses online as I always liked to ask questions in class.  With the pre-recorded lectures, the only way to ask questions is on the associated forums.  He agreed, but said while this was probably not the future of university, it was certainly great for continuing education.

Other Python resources I looked at  include http://www.learnpython.org/.  The pyvideo website includes talks from many of the PyCon and related conferences here which are interesting to watch.

What resources did you use to learn Python?

Read more...

Lessons learned organizing a technical event

I was on the organizing committee for Releng 2013.  This was the first time I've been involved in organizing a technical event.  I thought my experiences might be useful to others in the same situation thus this blog entry:-)

Reaching out to potential speakers
Three of the people on our organizing committee were researchers.  I was the only release engineer.  My goal was to get as many release engineers as possible to submit talks and attend the workshop.  I asked for suggestions from my coworkers and LinkedIn contacts to try to reach people that might be interested in submitting a talk.  I also wrote to multiple release engineering oriented mailing lists and Google+ groups.  I had a query for #relengcon and @relengcon mentions on Twitter.  If people mentioned that the relengcon sounded interesting, I replied to them and suggested that they submit a talk.  I also posted information about the workshop on several release engineering LinkedIn and meetup.com groups.  It's often a good idea to contact the organizer of a group and ask them to send text crafted by the organizers to their members.  Receiving a message from someone they already know lends some validity to the message, versus just another person adding to their inbox noise.

Carl and Gareth from Netflix talk about self-service build and delivery

Academic vs. Industry
Releng 2013 was a workshop under the larger ICSE academic conference.   The submission process was much more formal than the industry conferences I've attended in the past.   There were two types of submissions: a formal paper or an abstract for a talk.  There were strict submission guidelines with respect to the format of the talk abstract.  In contrast, the submission process for most industry conferences seems to be "submit a paragraph summarizing your talk into a form on web page".  Very easy.  The different approach stems from the fact that proceedings from academic conferences are usually published.  Academics often need the promise of publication to secure funding to attend a conference.  

Funding
This workshop had effectively zero budget.  ICSE workshops don't get any funding from the fees paid to the larger ISCE conference.   Attendees had to pay registration fees to attend the workshop whether they had a talk accepted or not.  Many industry conferences offer a reduced registration fee for speakers.  Since this was a first time event, I really concentrated on encouraging people to submit talks and register for the event.  Now that the event was a success, I would feel comfortable approaching a company and asking them to sponsor it so we could reduce speaker fees.  We had some funding from a research institution for one of our keynote speakers.  I reached out to Mozilla developer relations who provided some t-shirts,  mugs and stickers.  In the future, I'd also reach out to other companies for swag donations :-)

Recording of sessions
The cost of hiring a company to record the talks was prohibitive given our lack of budget.  Again, now that the first workshop was a success, this could be another item that could be sponsored. 

Post-workshop dinner organization
The OpenTable website is a great resource to see the availability of seating at  of various restaurants in several US cities.  It also links to their respective ratings.  This was easy way to make reservations at restaurants for dinner, especially since none of the organizers live in the Bay area.

Image ©photographus, http://www.flickr.com/photos/misspixels/8480711076/in/photostream/ under Creative Commons by-nc-sa 2.0

 
Panel
I had never moderated a panel before.  I found this article was quite helpful on how to successfully moderate a panel.  I spent quite a bit of time before the event thinking of panel questions and we also sought input from our attendees in a survey.   As for participants on the panel, I reached out to release engineers who didn't present a talk but looked like they had a wealth of release engineering experience on their LinkedIn profiles.  Sometimes diversity is just asking different people!

Speaker gifts
We wanted to give speaker gifts to our fine keynote speakers.  I was unsure what a typical speaker gift comprised so I asked on Twitter.   The answers ranged from fancy pens, laptop bags, local bottles of wine and local cookbook.  Another person suggested that the gift be something carried on the plane if the speaker wasn't local.  In fact, they usually personalized the gift by presenter and shipped it to their home to avoid dealing with airline hassles regarding gifts.  Release engineering is about building things.  Bram suggested Lego as a speaker gift.  I liked this idea a lot and added it as one of the items in the speaker gift bag.

In the end the workshop went pretty well.  The most important thing was to get people who are passionate about release engineering in the same room.  And that's where the interesting conversations begin and you make connections with people.  That's what really matters in the end.  

If you've been involved in organizing a technical event what advice do you have?

Read more...

Releng 2013 Recap

On May 20, we held the Releng 2013 workshop in San Francisco.  By all accounts it was a success!  I was unsure what to expect given that this was a first time for this workshop.   It seemed to be a topic that generated a lot of interest.  We had around 80 attendees which to my understanding was the highest attendance of any workshop at ICSE this year.   It was great to have so many people interested in release engineering and research in the room.  It was also fantastic to meet my co-organizers Bram, Foutse and Christian in person after working to organize this event since last fall.

The night before the workshop, I went out to dinner with the other organizers and some of their professor and post-doc colleagues.  I was struck by how enthusiastic they were about release engineering topics and interacting with industry.  One of them said  "We want to make sure that we're working on relevant problems." 

There were two keynotes at the workshop.  The opening keynote was Release Engineering as Force Multiplier by John O'Duinn of Mozilla.  The afternoon one was by Roman Scheiter of LinkedIn and entitled Against All Odds – Completely Overhauling Linkedin's Release Process.  Both were fantastic accounts of how the build, test and release pipelines at these companies were improved to make the organizations as a whole more effective.  It was an interesting contrast in that LinkedIn
moved from branch based development model to trunk based continuous integration while Mozilla moved from trunk based continuous integration to branch based development model.

One of the researchers commented that if you have negative results, you don't publish a paper.  So they were somewhat surprised to see the openness from those from industry on the things we did wrong.  My understanding is that's not that common in academia given the push to be the first to publish a new result.  A different culture.

Some highlights for me from the other talks:

• The software delivery model at Netflix by Curt Patrick and Gareth Bowles (development islands, no release engineers, their team provides tooling for self-serve builds)
• Hal Wine from Mozilla on using Hg and Git on the same codebase (many asked why - that sounds crazy!)
• Jim Buffenbarger from Boise State University gave a talk on amake which includes automatic dependency processing
• Akos Frohner and Boris Debic from Google gave a talk about the continuous release process at Google and which included some incredible numbers (100 million unit tests run a day!).  Boris also had some great things to say about the value of release engineering such as  "Release engineering should taught in business school, not in computer science classes.  It has real business value, and developers can learn it later. " "Startups should hire release enginners early, otherwise they will have to drain the swamp later".  "Companies that don't do release engineering well don't compete well in the marketplace."  Definitely words of wisdom.
• Dustin Mitchell from Mozilla gave a talk on the Buildbot continuous integration framework.  Later that afternoon, Moses Mendoza and Matthaus Owens from Puppet Labs presented how they build their packages for their consumers  and mentioned that they hoped to collaborate with Dustin and learn more about Buildbot.  Which was great - so glad that the workshop encouraged this collaboration.
• Ryan Hardt, a post-doc from University of Wisconsin (Milwaukee) gave a talk about Formiga, an Eclipse plug-in for refactoring Ant code which looks very promising. 
• Peter Rigby  from Concordia University described  DCVS systems facilitate different workflows between developers as opposed to more centralized systems.  Very interesting!

At the end of the day, I had the honour of moderating the discussion panel. The panel started out with professors Mike Godrey from Waterloo and Gunther Ruhe from the university of Calgary, as well as practitioners Jason Newblanc from Salesforce and Curt Patrick from Netflix.  Discussions included what topics needed more research (dependency analysis), to  a discussion of the skills do you look for when hiring a good release engineer (communication skills above all).  Given that one of the main goals of the workshop was to bring together researchers and release engineers it was very gratifying to hear from the researchers on the panel how happy they were that they had the opportunity to learn from the presentations earlier in the day.  After a while the panel shifted to include new members with new topics, and then we wrapped up to head to post-workshop dinners where the discussions continued.

Several attendees mentioned to me that they were happy to attend the workshop because it validated the release engineering work that they do is important.   I'm very fortunate to work at Mozilla on a large release engineering team where there are many people who love to talk about build optimization and automation.  But many release engineers work in organizations where they are the only ones who are interested in this subject.   To bring all these people together was a great experience!

We would like to thank everyone who attended the workshop.  Thank you to our session chairs who timed the talks and ensure that we stayed on schedule.   Thank you to Juliana Saraiva, our student volunteer, who helped with setup and throughout the day with audio and visual issues. 
A special thanks to all of our speakers.  Having been a speaker in the past, I understand all the work that goes into preparing talks on top of your regular day jobs and truly appreciate the effort that made this workshop successful.  At the end of the day, there was a resounding show of hands from attendees that they would like to attend another workshop.  In fact, several companies volunteered to  their space to host events which is very generous.   Please follow @relengcon on twitter for more information on upcoming events.  Many of the papers and presentations are available on the web site.  If your slides are missing from the web site, please email them to Bram.

Also thank you to my colleagues at Mozilla for arranging our releng work week to be same week so many of us could attend, despite it being a Canadian holiday weekend.  You're all amazing :-)

Read more...