Releng of the Nerds

This week in Mozilla Releng - June 20, 2014

>> Friday, June 20, 2014

Ben is away for the next few Fridays, so I'll be covering this blog post for the next couple of weeks.

Major highlights:

Aki fixed an issue which was causing the sheriffs grief because the specific version of an external repository was not specified in b2g manifests.
Ben completed work on to support update balrog blob schema to support multiple partials. This is important because sometimes we need to have multiple partials in a beta or release.
John is back for his second internship. He completed his work providing an easy setup of some of our releng applications in a local test environment using Docker and blogged about it here. This should reduce the barriers for new people to contribute.
Jordan, wearing the buildduty shield this week, fixed backlogs on our AWS spot instances that were causing high pending counts.

Completed work (resolution is 'FIXED'):

Balrog: Backend

Buildduty

General Automation

Loan Requests

Other

Platform Support

Release Automation

Releases

Releases: Custom Builds

Repos and Hooks

New git repositories to mirror for Flame

Tools

In progress work (unresolved and not assigned to nobody):

Balrog: Backend

Buildduty

General Automation

Loan Requests

Other

Platform Support

Release Automation

Repos and Hooks

Allow pushing Instantbird-only changes on a closed c-c- tree

Tools

Talking about speaking up

>> Monday, June 09, 2014

We all interpret life through the lens of our previous experiences. It's difficult to understand what each day is like for someone who has had a life fundamentally different from your own because you simply haven't had those experiences. I don't understand what it's like to transition from male to female while involved in an open source community. I don't know the steps taken to become an astrophysicist. To embark to a new country as an immigrant. I haven't lived struggled to survive on the streets as homeless person. Or a person who has been battered by domestic abuse. To understand the experiences of others, all we can do is listen and learn from others, with empathy.

There have been many news stories recently about women or other underrepresented groups in technology. I won't repeat them because frankly, they're quite depressing. They go something like this:
1. Incident of harassment/sexism either online/at a company/in a community/at a conference
2. People call out this behaviour online and ask the organization to apologize and take steps to prevent this in the future.
3. People from underrepresented groups who speak up about behaviour are told that their feelings are not valid or they are overreacting. Even worse, they are harassed online with hateful statements telling them they don't belong in tech or are threatened with sexual assault or other acts of violence.
4. Company/community/conference apologizes and issue written statement. Or not.
5. Goto 1

I watched an extraordinary talk the other day that really provided a vivid perspective about the challenges that women in technology face and what people can do to help. Brianna Wu is head of development at Giant Spacekat, a game development company. She gave the talk "Nine ways to stop hurting and start helping women in tech" at AltConf last week. She is brutally honest with the problems that exist in our companies and communities, and the steps forward to make it better.

She talks about how she is threatened and harassed online. She also discusses how random people threatening you on the internet is not a just theoretical, but really frightening because she knows it could result in actual physical violence. The same thing applies to street harassment.

Here's the thing about being a woman. I'm a physically strong person. I can run. But I'm keenly aware that men are almost always bigger than me, and by basic tenets of physiology, stronger than me. So if a man tried to physically attack me, chances are I'd lose that fight. So when someone threatens you, online or not, it is profoundly frightening because you fear for your physical safety. And to have that happen over and over again, like many women in our industry experience, apart from being terrifying, is exhausting and has a huge emotional toll.

I was going to summarize the points she brings up in her talk but she speaks so powerfully that all I can do is encourage you to watch the talk.

One of her final points really drives home the need for change in our industry when she says to the audience "This is not a problem that women can solve on their own....If you talk to your male friends out there, you guys have a tremendous amount of power as peers. To talk to them and say, look dude this isn't okay. You can't do this, you can't talk this way. You need to think about this behaviour. You guys need to make a difference in a way that I can't." Because when she talks about this behaviour to men, it often goes in one ear and out the next. To be a ally in any sense of the word, you need to speak up.

THIS 1000x THIS.

Thank you Brianna for giving this talk. I hope that when others see it they will gain some insight and feel some empathy on the challenges that women, and other underrepresented groups in the technology industry face. And that you will all speak up too.

Further reading
Ashe Dryden's The 101-Level Reader: Books to Help You Better Understand Your Biases and the Lived Experiences of People
Ashe Dryden Our most wicked problem

Mozilla pushes - May 2014

>> Monday, June 02, 2014

Here's May's monthly analysis of the pushes to our Mozilla development trees. You can load the data as an HTML page or as a json file

Trends
This was a record breaking month where we overcame our previous record of 8100+ pushes with a record of 11000+ pushes this month. Gaia-try, just created in April has become a popular branch with 29% of pushes.

Highlights

11711 pushes

New record

378 pushes/day (average)

New record

Highest number of pushes/day: 613 pushes on May 29, 2014

New record

22 pushes/hour (average)

New record

General Remarks

The introduction of Gaia-try in April has been very popular and comprised around 29% of pushes in May. The Try branch itself consisted of around 38% of pushes.

The three integration repositories (fx-team, mozilla-inbound and b2g-inbound) account around 22% of all the pushes, compared to 30% in the previous month.

Records
May 2014 was the month with most pushes (11711 pushes)
May 2014 has the highest pushes/day average with 378 pushes/day
May 2014 has the highest average of "pushes-per-hour" is 22 pushes/hour
May 29th, 2014 had the highest number of pushes in one day with 613 pushes

May 2014 is a record setting month, 11711 pushes!

Note that Gaia-try was added in April and has quickly become a high volume branch

I changed the format of this pie chart this month. It seemed to be previously based on several months data, but not all data from the previous year. So I changed it to be only based on the data from the current month which seemed more logical.

20 years on the web

>> Friday, May 16, 2014

Note: I started writing this a long time ago as part of #mynerdstory but never got around to finishing it until recently. So I changed it a bit when I noticed it had been over 20 years since I first used the internet.

I found this picture the other day. It's me on graduation day at Acadia, twenty years ago this month. A lot has changed since then.

In the picture, I'm in Carnegie Hall, where the Computer Science department had their labs, classrooms and offices. I'm sitting in front of a Sun workstation, which ran a early version of Mosaic. I recall the first time I saw a web browser display a web page, I was awestruck. I think it was NASA's web page. My immediate reaction was that I wanted to work on that, to be on the web.

As a I've mentioned before, my Dad was a manager at a software and services firm in Halifax. He brought home our first computer when I was 9. Dad was always upgrading the computers or fixing them and I'd watch him and asked lots of questions about how the components connected together. In junior high, I taught myself BASIC from the manual, wrote a bunch of simple programs, and played so many computer games that my dreams at night became pixelated. When I was 16, I started working at my Dad's office doing clerical work during the school break. One of my tasks was to run a series of commands to connect to BITNET via an accoustic coupler using Kermit and download support questions from their university customers. I thought it was so magical that these computers that were so physically distant could connect and communicate.

In high school, I took computer science in grade 12 and we wrote programs in Pascal on Apple IIs. My computer science teacher was very enthusiastic and welcoming. He taught us sorting algorithms, and binary trees, and other advanced topics that weren't on the curriculum. Since he had such an interest he taught a lot of extra material. Thanks Mr. B.

When it was time to apply to university, I didn't apply to computer science. I don't know why, my grades were fine and I certainly had the background. I really lacked self confidence that I could do it. In retrospect, I would have been fine. I enrolled at Acadia in their Bachelor of Business Administration program, probably because I liked reading the Globe and Mail.

I arrived on campus with a PC to write papers and do my accounting assignments. The reason I had access to a computer was that the company my Dad worked for allowed their employees borrow a computer for home use for a year at a time, then return it. Otherwise, they were prohibitively expensive at the time. My third year of university I decided that I was better suited to computer science than business so started taking all my elective courses from the computer science faculty. I still wanted to graduate on in four years so I didn't switch majors. It was such a struggle to scrape together the money from part-time jobs and student loans to pay for four years of university, let alone six.

One of my part-time jobs was helping people in the university computer labs with questions and fixing problems. Everything was very text based back then. We used Archie to search for files, read books transcribed by the Gutenberg project and use uudecode to assemble pictures posted to Usenet groups. I applied for a Unix account on the Sun system that only the Computer Science students had access to. It was called dragon and the head sysadmin had a sig that said "don't flame me, I'm on dragon". I loved learning all the obscure yet useful Unix commands.

My third year I had a 386 portable running Windows 3.1. I carried this computer all over campus, plugging it in at the the student union centre and working on finance projects with my business school colleagues. By my fourth year, they had installed Sun workstations in the Computer Science labs with Mosaic installed. This was my first view of the world wide web. It was beautiful. The web held such promise.

I applied for 40 different jobs before I graduated from Acadia and was offered a job in Ottawa working for the IT department of Revenue Canada. A ticket out of rural Nova Scotia! I didn't like my first job there that much but they paid for networking and operating system courses that I took at night. I was able to move to a new job in a year and started being a sysadmin for their email servers that served 30,000 users. It was a lot of fun and I learned a tremendous amount about networking, mail related protocols and operating systems. I also spent a lot of time in various server rooms across Canada installing servers. Always bring a sweater.

I left after a few years to work at Nortel as technical support for a telephony switch that offloaded internet traffic from voice switches to a dedicated switch. Most internet traffic back then was via modem which were longer duration calls than most voice calls and caused traffic issues. I took a lot of courses on telephony protocols, various Unix variants and networking. I traveled to several telco customers to help configure systems and demonstrate product features. More time in cold server rooms.

Shortly after Mr. Releng and I got married we moved to Athens, Georgia where he was completing his postdoc. I found a great job as a sysadmin for the UGA's computer systems division. The group provided FTP, electronic courseware and email services to the campus. We also secured a lot of hacked Linux servers set up by unknowing graduate students in various departments. When I started, I didn't know Linux very well so my manager just advised me to install Red Hat about 30 times and change the options every time, learn how to compile custom kernels and so on. So that's what I did. At that time you also had to compile Apache from source to include any modules such as ssl support, or different databases so I also had fun doing that.

We used to do maintenance on the computer systems between 5 and 7am once a week. Apparently not many students are awake at that hour. I'd get up at 4am and drive in to the university in the early morning, the air heavy with the scent of Georgia pine and the ubiquitous humidity. My manager M, always made a list the night before of what we had to do, how long it would take, and how long it would take to back the changes out. His attention to detail and reluctance to ever go over the maintenance window has stayed with me over time. In fact, I'm still kind of a maintenance nerd, always figuring out how to conduct system maintenance in the least disruptive way to users. The server room at UGA was huge and had been in operation since the 1960s. The layers of cable under the tiles were an archeological record of the progress of cabling within the past forty years. M typed on a DVORAK keyboard, and was one of the most knowledgeable people about all the Unix variants, and how they differed. If he found a bug in Emacs or any other open source software, he would just write a patch and submit it to their mailing list. I thought that was very cool.

After Mr. Releng finished his postdoc, we moved back to Ottawa. I got a job at a company called OTI as a sysadmin. Shortly after joining, my colleague J said "We are going to release an open source project called Eclipse, are you interested in installing some servers for it?" So I set up Bugzilla, CVS, mailman, nntp servers etc. It was a lot of fun and the project became very popular and generated a lot of traffic. A couple years later the Eclipse consortium became the Eclipse Foundation and all the infrastructure management moved there.

I moved to the release engineering team at IBM and started working with S who taught me the fundamentals of release engineering. We would spent many hours testing and implementing new features in the build, and test environment, and working with the development team to implement new functionality, since we used Eclipse bundles to build Eclipse. I have written a lot about that before on my blog so I won't reiterate. Needless to say, being paid to work full time in an open source community was a dream come true.

A couple of years ago, I moved to work at Mozilla. And the 20 year old who looked Mosaic for the first time and saw the beauty and promise of the web, couldn't believe where she ended up almost 20 years later.

Many people didn't grow up with the privilege that I have, with access to computers at such a young age, and encouragement to pursue it as a career. I thank all of you who I have worked with and learned so much from. Lots still to learn and do!

Release Engineering Special Issue

A different type of mobile farm ©Suzie Tremmel, https://flic.kr/p/6tQ3H Creative Commons by-nc-sa 2.0

Are you a release engineer with a great story to share? Perhaps the ingenious way that you optimized your build scripts to reduce end to end build time? Or how you optimized your cloud infrastructure to reduce your IT costs significantly? How you integrated mobile testing into your continuous integration farm? Or are you a researcher who would like to publish their latest research in a area related to release engineering?

If so, please consider submitting a report or paper to the first IEEE Release Engineering special issue. Deadline for submissions is August 1, 2014 and the special issue will be published in the Spring of 2015.

IEEE Release Engineering Special Issue

If you have any questions about the process or the special issue in general, please reach out to any of the guest editors. We're happy to help!

We're also conducting a roundtable interview with several people from the release engineering community in the issue. This should raise some interesting insights given the different perspectives that people from organizations with large scale release engineering efforts bring to the table.

Mozilla pushes - April 2014

>> Monday, May 12, 2014

Here's April's monthly analysis of the pushes to our Mozilla development trees. You can load the data as an HTML page or as a json file.

Trends

April (as March did) has the highest number of pushes ever.
We will soon have 8,000 pushes/month as our norm, this seems to be the case this month.

Highlights

8109 pushes

NEW RECORD

270 pushes/day (average)

NEW RECORD

Highest number of pushes/day: 414 pushes on April 14, 2014

14.8 pushes/hour (average)

General Remarks

Try continues to have around 50% of all the pushes
The three integration repositories (fx-team, mozilla-inbound and b2g-inbound) account for around 30% of all the pushes

Records

April 2014 was the month with most pushes (8109 pushes)
April 2014 has the highest pushes/day average with 270 pushes/day
February 2014 has the highest average of "pushes-per-hour" is 16.57 pushes/hour
March 4th, 2014 had the highest number of pushes in one day with 435 pushes

Disclaimer

The data collected prior to 2014 could be slightly off since different data collection methods were used.

Releng 2014 invited talks available

>> Tuesday, May 06, 2014

On April 11, there was a Releng workshop held at Google in Mountain View. The two keynote talks and panel at the end of the day were recorded and made available on the Talks at Google channel on YouTube. Thank you Google!

Moving to mobile: The challenges of moving from web to mobile releases, Chuck Rossi, Facebook https://www.youtube.com/watch?v=Nffzkkdq7GM

Some interesting notes from Chuck's talk:

Facebook has more users on their beta apps than most companies have on their release apps. Number one app for IOS, number one non-Google app for Android. They have more beta users on Android than most people have users.
There are three things we look act as software developers: features, quality, schedule. As release engineers we focus on quality and schedule. Because we ship on time. New features can go in the next mobile release which is only four weeks in the future.
Developers have karma, if you release to many bad changes you lose karma. Developers start out with four Karma stars. You can only see you own karma. The release engineering team can reduce your karma if you release too many breaking changes.
The release engineering team has a lot of respect within Facebook, they are the ones who determine if your feature gets in a release. He notes that you should make sure that you work for a organization that respects you for your judgement and skillset as as release engineer. As an aside, it seems the release engineering at Facebook has responsibility that corresponds to release engineering + release management at Mozilla.
Compute power and storage are infinite at Facebook for their build and test infrastructure. It's not a contraint.
They run 28 - 30,000 builds a day. Wow.
They have a ui for developers to look at their build and test results called shrubbery. It looks like a fork of Mozilla's tbpl (Facebook's release engineering team has Mozilla alumni :-) They also run Buildbot as their continuous integration engine too.
Like Mozilla they have sheriffs to monitor builds but this role is rotated among the developer teams. If you put developers on call, they will make it less likely that they will get paged by being more careful when releasing changes because they share your operational burden.
They have 17 apps to upload to the Google Play store. In the presentation, he asked Google to provide and API to be able to do so. Right now they have to a person interact with the UI and press buttons for all these apps they need to upload. Very painful.
15 minutes to deploy a new (web) facebook.com. Of course, this doesn't happen all at once. They deploy to a certain percentage of their users, and evaluate the data and then deploy to more users.

The 10 Commandments of Release Engineering, Dinah McNutt, Google

https://www.youtube.com/watch?v=RNMjYV_UsQ8

Some notes from Dinah's talk.

Release engineering is accelerating the path for development to operations
You want to be able to reproduce your build environment and source code management system if you have to recreate a very old build
Configuration management and release engineering as disciplines will probably merge as the next few years
Reproducibility is a virtue. Binaries don't belong in SCMs. However, it's important to be able to reproduce binaries. If you do need a repo with binaries, put them in a separate repo.
Use the right tool for the job, you will have multiple tools. Both commercial and open source.
View the job of a release engineer and making a developers job easier. By setting up tooling and best practices.
Package management. Provides auditing, upgrading, installation, removals. Tars and jars are not package managers.
You need to think about your upgrade process before you release 1.0.
Customers find problems we cannot find ourselves. Even if we're dogfooding.
As release engineers, step back and look at the big picture. Look and see how we can make things better from a cost perspective so we have the resources we need to do our jobs.
It's a great year to be a release engineer. Dinah is the organizing committee for Release Engineering Summit June 20 in Philadelphia as part of USENIX. There is also one as a part of LISA in Seattle in November. Overwhelming interest for a first time summit in terms of submissions!

Closing discussion panel:
https://www.youtube.com/watch?v=D4jLlWdsrWg

Stephanie Bellomo, one of my colleagues from the organizing committee for this workshop moderated the panel. Really interesting discussions, well worth a listen. I like that the first question of "What is your worst operational nightmare and how did you recover from it?" I love war stories.

As an aside, we charged $50 per attendee for this workshop. We talked to other people who had organized similar events and they suggested this would be an appropriate fee. I've read that if you don't charge a fee for an event, you have more no-shows on the day of the event because psychologically, they attach a lesser value to the event since they didn't pay for it. However, we didn't have many expenses to pay for the workshop other than speaker gifts, eventbrite fees and badges. Google provided the venue and lunch, again, thank you for the sponsorship. So we donated $1,531.00 USD to each of the following organizations from the remaining proceeds.

YearUp is an organization that in their words "Year Up empowers low-income young adults to go from poverty to professional careers in a single year." I know Mozilla has partnered with YearUp to provide mentoring opportunities within the IT group and it was an amazing experience for all involved.

The second organization we donated to is the Tanzania Education Fund. This organization was one that Stephany mentioned since she had colleagues that were involved with for many years. They provide pre-school, elementary, secondary and high school education for students in Tanzania. Secondary education is not publicly funded in Tanzania. In addition, 50% of their students are girls, in an area where education for girls is given low priority. Education is so important to empower people.

Thanks to all those that attended and spoke at the workshop!

Remote work in review

>> Monday, May 05, 2014

I recently read two books on remote work. Scott Berkun's Year without Pants and Remote: Office Not Required by Jason Fried and David Heinemeier Hansson. The first book describes Scott Berkun's year as a remote manager at Automattic, which runs Wordpress.com. The second book describes the authors' experiences at the fully distributed company 37signals, which has since renamed itself to Basecamp to reflect the importance of its flagship product. Both books were interesting reflections on the nature of remote work in the context of their respective companies. They were not books that addressed the nature of remote work in general, but described the approach they felt was successful within their companies. Of the two books, I would really recommend reading the Year Without Pants. Remote isn't as compelling but it's a short read.

Some notes from "The Year without Pants"

Communication
"To work at a remote company demanded great communication skills and everyone had them"

Culture
Chapter 4 on culture always wins is is a fantastic read, it's available for free on his website.

"Trust is everything".

"1. Hire great people
2. Set good priorities
3. Remove distractions
4. Stay out of the way"

In other words, treat people like grown ups and they will do good work.

"In every meeting in every organization around the world where bad behavior is happening, there is someone with the most power in the room who can do something about it. What that person does shapes the culture. If the most powerful person is silent, this signals passive acceptance of whatever is going on."

Wow, this is very significant. If you are the most powerful person in the room, speak up and call out bad behaviour. The people with less power are often hesitant to speak up because there may be consequences for them and they feel they lack authority.

Hiring
"Hire self sufficient, passionate people"

I often get questions from people who don't work at home how I don't get distracted and goof off all day since I work from home. It's simple. I love my job. It's a lot of fun. I want to be shipping, not slacking.

Shipping
Shipping every day gives people a sense of accomplishment. Many bug fixes are deployed to Wordpress.com every day. No gatekeepers to deployment but the people deploying the change are expected to watch the site after they deploy for a few hours to ensure there aren't unexpected problems.

Some notes from "Remote: Office Not Required"

Communication
The book suggests asking managers or employees to work from home a few days a week to level the playing field with respect to communications between employees who work in an office and those who are remote. This will ensure they appreciate what hinders communication and take steps for improvement. And it will reduce the tendency to treat remote workers as second class citizens and cut them out of essential conversations.

Culture
"...coming into the office just means that people have to put on pants. There is not guarantee of productivity."

"If you view those who work under you as capable adults who will push themselves to excel even when they're not breathing down their necks, they'll delight you in return"

Again, trust people and have high expectations of them and you'll be rewarded with excellence.

Hiring
The authors note that international exposure is good as a selling point with clients. Hiring around the world increases the talent pool available, but is not without tax or legal complications. Also, given the degree of written communication with remote work, it's best to hire people with the language skills that can thrive in this situation.

Productivity
The book stresses that workflow tools need to be available to all team members at all times in order to be productive. i.e. recording the state of a project in a wiki, bug tracker and recording meetings. If a team member is working in a timezone offset from the majority of team members and doesn't have this in place, it can be a productivity drain.

"Would-be remote workers and managers have a lot to learn from how the open source software movement has conquered the commercial giants over the past few decades. Open source is a triumph of asynchronous collaboration and communication like few the world has ever seen."

Absolutely. I learned so much working in open source for so many years.

The authors also mention that you'll have to worry about your employees overworking, not underworking. Because they office is physically in your home, it's easy to get sucked in at all hours to just work on one little thing that takes longer than you expect.

My thoughts on remote work
If you had asked me five years ago if I ever thought I'd work full time from my home my answer would have been a definitive no. But I wanted to work for Mozilla, and wasn't interested in moving to an city where they had a physical office. Here are my personal suggestions for working successfully on a remote team, given my two years of experience being a part of one:

Work for a company with strong culture, tools and process that support remote work. Mozilla is a great example, there are many others. It's not optimal to be the only remote worker in a company that has no experience or culture that supports distributed collaboration.
Get up, shower, eat breakfast and put on clothes you wouldn't be ashamed to seen in outside the house. Despite the title of the book I reviewed above, this means wearing pants.
I think it's helpful that I have people on my timezone that I collaborate with on regular basis. I think I would find my job much more difficult if this was not the case.
Do stuff to get out of the house on a regular basis - visiting friends, family or regularly scheduled activities such as sports or hobbies. Otherwise, you will become, as Laura Thomson so eloquently put it a "lonely code hermit". This isn't healthy.
Talk about stuff not directly related to code with your coworkers so you have some rapport for them as people outside the context of work.
Meet up with your team in person on a regular basis. At Mozilla releng, we do this three or four times a year. As much as irc, bugzilla, wikis, video conferencing and other tools are great for collaboration, there isn't anything that can substitute for talking in person. Also, it's great for getting to know them more as people. And when you know them as people, you are more willing to help each other and feel more cohesive as a team.
Ship. If you are worried that you don't appear to be productive because you aren't physically in the office, ship awesome stuff. Then nobody cares where you're sitting and you get to have a 30 second commute.
If you work in software, you aren't constrained by geography to determine the kind of job you can have. If you don't want to work for the software companies that have offices in the town you live in, you have a choice to choose a better job. Life is too short to get up every day and work at a job you don't like. There's a lot opportunity for those who work in many fields to work remotely.

As aside, I was supposed to be at a Mozilla work week in Portland this past week. But I didn't fly there because I came down with a bad cold. Despite this, I could connect to the same room the they were in and see all the talks they were giving and also give a presentation. This was so excellent. Since we are so used to being a distributed team, having one person remote when we were supposed to be all together wasn't a problem. We already had the culture and tools in place to accommodate this. Thank you Mozilla releng for being such an amazing team to work with.

My idea for another book on remote work would be to have one with the format where there were about 10+ chapters, each written by an employee from a different company about how they approach it, what tools they use and so on. I think this could be a very interesting read.

I'll close with some thoughts from Scott Berkun's book on whether remote work is for everyone

"For me, I know that for any important relationship I'd want to be physically around that person as much as possible. If I started a rock band or a company, I'd want to share the same physical space often. The upsides outweigh the downsides. However, if the people I wanted to work with were only available remotely, I'm confident that we could do great work from thousands of miles away."

What do you think are the keys to successfully working as a remote employee?

Further reading
John O'Duinn's We are all Remoties Talk

Enabling tests that ride the trains

>> Wednesday, April 02, 2014

At Mozilla, we have a train schedule for releases. Changes are first landed on a trunk branches, then are uplifted to other branches for stabilization. Johnathan Nightingale has a very good blog post that explains this concept. For instance, the usual route for a change to land from a trunk branch such as mozilla-central would be to uplift to mozilla-aurora, mozilla-beta, and then finally mozilla-release where it would be available to users in the next release.

Merge day, which occurs every six weeks is when changes are uplifted to the next stable branch. Here's a picture from a talk John O'Duinn gave last year that shows an example of how changes move between branches¹.

Picture from John O'Duinn's Release Engineering as a Force Multiplier talk at Releng 2013

For the release engineering team, each merge day we would update code in our buildbot configs to reflect changes that needed to be made after uplift. For instance, we often deprecate platforms or add new tests that only apply to certain branches. We used to have to to specify the branches that these applied to in our configs and update them every merge day. It was tricky to get this right. Last fall, Steve Fink fixed a bug that would allow us to base config versions on the version of gecko that rides the trains. So each merge do we update the version of gecko in our configs on a per branch basis, and then have code like this so that the tests are only enabled for branches where the gecko version applies

......
Enable jittests for desktop where gecko is 31 or more

Load jetpack where gecko is at least 21

To test these changes, you can set up your buildbot test master and run builder_list.py against your master. The builder_list.py script will output the list of build/test jobs (builders) that are enabled on your master. Then apply your patch against the master and diff the resulting builder files to ensure that the tests are enabled in the branches you want. As a side note, if you are enabling tests on platforms that would be on different test masters, you'll have to configure your master for mac, linux and windows test masters and diff the builders for each platform. If you are enabling tests on trunk trees for the first time, your diff should not reveal any new builders on mozilla-aurora, mozilla-beta, mozilla-release but just on mozilla-central, mozilla-inbound and the associated twig branches.

I recently fixed a few bugs where there was a request enable tests on trunk branches and ride the trains, so I thought I'd write about if others had to implement a similar request.

Train-related graffiti in Antwerp (Belgium), near Antwerpen-Centraal train station by ©vitalyzator, https://flic.kr/p/6tQ3H Creative Commons by-nc-sa 2.0

Further reading and notes
¹ This applies to Firefox Desktop and mobile only, Firefox OS is on a different cadence and there are different branches involved
Release Management's rapid release calender
Release Engineering Merge duty
Release Engineering Testing Techniques

Schooling yourself in release engineering

>> Monday, March 31, 2014

Traditionally, there haven't been many courses offered in colleges or universities that cover the fundamentals of release engineering. This means that students don't get exposed to the potential that a career in release engineering has to offer. Conversely, it also doesn't provide students who become employed in more traditional developer roles the background regarding the complexity and challenges that arise within the scope of release engineering. However, this is beginning to change which is fantastic! For example:

Release Engineering as a Discipline, Center of Computer Science, RWTH Aachen University in Aachen Germany

Overview of the Build and Release Process, (updated link) Seneca College, Toronto

Release Engineering -- Applications of Mining Software Repositories, École Polytechnique, Montréal

Software Release Planning, University of Calgary

If anyone knows of other courses that are offered, I'd love to hear about them. Maybe someday I won't have to explain to new people I meet what a release engineer does all day. Just kidding, this will still happen :-)

Upcoming Release Engineering events

>> Wednesday, March 26, 2014

University of Waterloo engineering students in 1964 Image ©Kitchener Waterloo Record, http://www.flickr.com/photos/48169267@N08/4967256177/in/photolist-8yWucT-eqaQdV-epd6Lw-c5ywEJ-c5yvwd/under Creative Commons by-nc-sa 2.0

There are quite a few upcoming events related to release engineering so I thought I'd list them here.

The CFP for LISA is open, submissions due April 14. Release engineering topics are welcome. LISA is November 9–14, 2014, in Seattle, WA.

There is also a Release engineering workshop as part of Usenix Federated Conferences week, on June 20, 2014, in Philadelphia, PA. The CFP has closed, but I think you can still register.

The Releng 2014 workshop is April 11, at Google in Mountain View. We were really overwhelmed by the number of people were interested in registering for this workshop, and the event is now sold out. A few of the talks will be recorded, so if you couldn't get a ticket, they will be available online after the event.

Finally, there's the first IEEE Special Issue on Release Engineering. The deadline for submission of a paper is August 1, 2014. Not an event, but a great place to get a paper on your area of expertise published.

Any other events I missed?

EclipseCon 2014 trip report

>> Monday, March 24, 2014

I attended EclipseCon for a couple of days last week. It was great to see friends and colleagues again and learn what they were working on.

Some highlights for me:

My favourite presentation was by Tamar Cohen who presented on the Verve software that is used to provide 3D visualization of remote environments using NASA robots. NASA + robots = I could listen all day. The next day, I had to opportunity to sit at lunch with Tamar. I mentioned to her that I think she says that she has one of the best jobs in the world, building software for robots some of which will leave our planet, and she agreed :-) She also made an interesting point from a maintenance perspective, in that most of the software that her team writes for experiments by astronauts is used for the duration of the experiment and then discarded. So there aren't really any long term release engineering requirements for most of their software.

Tamar Cohen

I really enjoyed the keynote by Catarina Moto on open hardware materials. Here's a link to many of the videos she used during her talk. I was especially impressed by the discussion that showed the robotic hand made by open hardware components (more info here). Such a fine example of how technology can be a force for good in the world.

Catarina Moto talking about open hardware materials

I also really liked Andrew Low's talk on Porting NodeJS to Linux for PPC. He's a great speaker and I like porting stories :-) Nick Stinemates talk on Docker was really interesting for me too, because we are currently investigating using it in some capacity.

On Wednesday, I was happy to present an overview of Mozilla release engineering from both a technical and human perspective. The slides are here.

Built to Scale: The Mozilla Release Engineering toolbox from Kim Moir

The pdf on the EclipseCon website seems to be clearer. I think slideshare does some compression that causes the images to be a bit fuzzy.

In any case, I received some good feedback on the talk after I spoke, and via twitter and email. Many people were impressed by both the scale of builds and tests we run, and the tooling we use to manage it. So big kudos to the Mozilla Releng team and all the others we work with like ATeam and IT that allowed me to have great stories to tell. There were about 40 people who attended the talk. As an interesting anecdote, I asked how many people developed in Python in the talk and two hands went up. As release engineers at Mozilla, we spend most of our days immersed in developing Python code, so this was interesting. By the number of people in the Java talks at EclipseCon, it is clear that this the Eclipse community continues to be very Java focused.

At the end of my talk, Ian Bull asked an interesting question. He said something along the lines of "If you had to do it all over again, how would you change things at Eclipse to make it better from a release engineering standpoint?" (I'm paraphrasing, I'm jetlagged and this was a few days ago).

Ian Bull always asks great questions

I responded that it didn't really matter what I thought, the Eclipse community doesn't make release engineering a priority and allocate resources to it so it wouldn't change, no matter what I thought or did. Every open source community makes different decisions based on their priorities. If they want changes, they have to allocate resources, whether they be people, or money or both, to make these priorities happen. No resources, nothing gets done. In much the same vein that I'm always surprised that conference organizers reach out to try to get a more diverse speakers along gender/POC/geographic/sexuality etc lines when the CFP is announced, but the rest of the year nobody in the community is championing diversity. Again, no priorities, no resources, no change.

Thanks to everyone who made EclipseCon happen, it was an interesting conference!

Built to Scale talk at EclipseCon

>> Monday, March 10, 2014

I'm honoured to be giving a talk at EclipseCon next week entitled Built to Scale: The Mozilla Release Engineering Toolbox. To give you some context, here are some numbers about the scale of build and test jobs we run.

We run about 6000 build jobs and 50,000 test jobs every week day. Each test job has many actual test suites within it of course. We have 1800+ devices to build on, plus 3900+ for tests. Some devices reside in our data centres, some reside in AWS. When a developer lands (commits) a change, our goal is to have the relevant job start within 15 minutes of being added to the scheduler database.

My talk will discuss we manage this scale of continuous integration in terms of hardware and software. Also, I'll touch on how we manage this from a human perspective, because that isn't easy either. I'll also discuss some of the lessons along the way as we have moved many of our infrastructure to AWS. And I'll also describe how we manage our 1000+ mobile devices that we run tests on as part of our CI farm.

Image ©ardonik, http://www.flickr.com/photos/ardonik/3954691105/sizes/l/ under Creative Commons by-nc-sa 2.0 Release engineering at this scale has lots of pieces to fit together.

In preparing this talk, I have been thinking a lot about the audience. The audience will be people in the Eclipse community, who don't have a lot of context about how we do things at Mozilla. I recently read the book Resonate by Nancy Duarte which describes how to create great visual presentations and story arcs as a speaker. One of the ideas in the book is that the most important thing that you can do as a speaker is think about your audience, what they know, and how to engage them.

I use the Presentation Zen approach when preparing a talk which means that I write out all the topics on index cards, arrange them, rearrange them, and discard non-essential content. Before touching a computer. When I was initially preparing the talk, I had an entire index card of Mozilla specific words that I would have to explain. It was ridiculous. Nobody would ever remember the context of those terms from one slide to the next.¹ I put that card in the shredder.

Last week, I thought of a new approach to present my talk. I think it will really work. I want to make the talk as interesting and relevant to the Eclipse community as it would be as if I gave it to a room full of Mozillians who have more context.

So this is what I know about the audience for my talk
You are Eclipse community members
Like all developers, you have known the pain of slow builds and test results.
You'd like to know how to scale large amounts of hardware and software.
And how things can get better.
So you can work on optimizing your product, and not be frustrated by your build and release process.

If you have specific questions you'd like me to address in the talk, please let me know in the comments or via twitter (@kmoir). Looking forward to seeing you all at EclipseCon!

Notes:
¹ I also recently read Why Don't Students Like School: A Cognitive Scientist Answers Questions About How the Mind Works and What It Means for the Classroom which is an excellent book and extremely applicable to people teaching programming languages and other abstract concepts. One of the topics that stuck with me from reading this book is that our brains need to have a lot of simple concepts memorized to understand more complex concepts. For instance, if you don't have your multiplication tables memorized, simple algebra will be difficult because you will have to stop and think what the value of 3x is when x is 7 instead of just pulling that from memory. So this is why when people complain that schools teach a lot of memorization and not more abstract thinking, it's not really a valid argument. You need a lot of concepts memorized before you can do more abstract thinking. Highly recommended book.

John O'Duinn gave a talk at the Releng 2013 workshop last year and later as a Google Tech Talk that gives a great overview of why release engineering is a high priority at Mozilla. Well worth watching.

Early Registration for Releng 2014 now open

>> Monday, February 17, 2014

We are pleased to announce that early registration for the Releng 2014 workshop is now open. We are also very excited to announce that we have two great invited speakers lined up - release engineers Dinah McNutt of Google and Chuck Rossi of Facebook. Register now! We also encourage you to submit a talk, paper or poster of your own! The CFP closes February 28.

Releng 2014 CFP now open

>> Friday, January 24, 2014

Last year I was on the organizing committee for a one day workshop on release engineering in San Francisco as part of ICSE. We had so much positive feedback that we have organized another one this year. This year's workshop will be held on April 11, at Google in Mountain View, California. If you are interested in submitting a talk or paper, the deadline for the CFP is February 28. More details on the website.

Here's what people liked about last year's workshop

"Brought release engineers together in the same room! So awesome. Great talks and interaction between researchers and practitioners."

"The information sharing and the candidness of the speakers with their organizational challenges."

"Overall, I loved this opportunity and you all did a wonderful job putting this on. THANK YOU!"

"Specific stories of how some companies improved their release engineering. Being able to network and ask questions of presenters and people working on the same problem domain."

"The huge variation in scale that things are being done across the industry. "

"Informal structure led to lots of hallway conversations"

Image ©stuckincustoms, http://www.flickr.com/photos/stuckincustoms/351212669/sizes/m/under Creative Commons by-nc-sa 2.0 This picture of the international airport in Bangkok has nothing to do with release engineering, but has interesting architecture.

As was the case last year, there will be both release engineers and researchers in attendance. This should generate a lot of interesting discussions as we talk about how we have succeeded and failed in building, deploying and testing software often at a large scale and what further research is needed. But if you work at a small company too we would love to hear your stories too. One note of feedback that we received last year was that people would like to hear from a variety of companies, not just large ones.

In parallel to the workshop, a call will be launched for the first ever IEEE Software Special Issue on Release Engineering. So submit a talk or paper, polish it with feedback from the workshop, and submit it to the special issue for possible publication.

I'm happy to help if you have any questions regarding the submission process or the workshop in general. Please feel free to drop me a line (I'm kmoir and I work at mozilla dot com).

More information regarding from last year's workshop:

Release Engineering as a Force Multiplier - John O'Duinn
Releng 2013 recap - Kim Moir

As an aside, there is another release engineering event as part of Usenix on June 20 in Philadelphia with an open CFP too. As Dinah McNutt, the organizer of this event remarked to me "It's going to be a great year to be a release engineer".

>> Friday, June 20, 2014

>> Monday, June 09, 2014

>> Monday, June 02, 2014

>> Friday, May 16, 2014

>> Monday, May 12, 2014

>> Tuesday, May 06, 2014

>> Monday, May 05, 2014

>> Wednesday, April 02, 2014

>> Monday, March 31, 2014

>> Wednesday, March 26, 2014

>> Monday, March 24, 2014

>> Monday, March 10, 2014

>> Monday, February 17, 2014

>> Friday, January 24, 2014

Blog Archive