Mozilla Release Engineering++

Note: I wrote this a month ago but didn't have time to finish this post until now. Bugzilla > blogging.

In early August, I attended my inaugural work week with Mozilla release engineering.  I met many of our team for the first time in person at the San Francisco office.

It was a fantastic week.  Running along the beautiful waterfront in the morning and talking about open source builds all day is my idea of a good time :-) On the flight home, I started thinking about why working as a release engineer at Mozilla makes me very happy.

1)  The people on the release engineering team are friendly, welcoming and genuinely helpful.  Not only that, but they have had interesting experiences outside of work.   From volunteering at Burning Man, to dowsing burning trees as a firefighter, we heard many interesting stories.

2)  We have a lot of build and test infrastructure, both virtual and not.   Due to the sheer amount of hardware, we have scaling issues which are interesting and complex to explore.  For developers, the available capacity means that they don't have to wait long for a build to start to test their changes.   As one of my coworkers stated, the scale of our continuous integration systems is quite unique in the open source world.

3)  We share the the load of interruption driven work as individuals to make the overall team more productive.  Every two months, you're on buildduty.  That week, you're responsible for answering developer questions and ensuring our infrastructure is operational at optimal capacity.  This allows other members of the team to concentrate on the projects they are working on without being constantly pinged in IRC.  We have beta builds every week and releases every six weeks.  These are also covered by team members on a rotating basis, called releaseduty.

4)  The other teams at Mozilla are also very helpful.  I'm really stunned by their generousity to set aside their work at a moment's notice and help you debug a problem so you can move forward.  One of the great things about working at Mozilla is that people are all working toward a common goal, to make the web a better place. And this means that they want to help everyone move forward, not just the team they work on.

5) Due to the inherent scaling issues and the fact that very few organizations run the volume of builds and tests we do, we write a lot of our own tools.  This is a great opportunity to diversity your skill set. Python, Puppet, Buildbot, AWS, MySQL, system administration and tooling for Windows, Linux, Mac and Android, wrangling Hg, Git and more.

6)  Management is committed to hiring great release engineers, no matter where they live.  This allows you to do the work you love, from the place you love to live.

7) Release engineers aren't considered second class citizens within the engineering group.   They are just another group of engineers who contribute, and are respected members of the community.

Release Engineering branded laptop bags? Yes

8)  We are data driven and constantly seek to improve our metrics.  What's the wait time to run a build on a certain platform?  What 's the utilization of our infrastructure?  What's the trend for the number of checkins per month? What platform could benefit from additional test slaves?  How can we optimize the build to make it faster?  How can we improve our automation story?   Continuous improvement is fantastic.

9) Inherent in the fact that work on open source projects, our culture is very open.  This means that we have very interesting speakers visit and discuss their build story and we can learn from them to address challenges in our own environment.

10) Working as a Mozilla release engineer is no walk in the park.  The learning curve is steep and it's very challenging work.

Walks in the park are nice, but I like the endorphins from running hard too :-)

Read more...