Releng 2013 Call for Papers

There are many conferences that discuss various aspects of of release engineering such as PuppetConf, OSCON, LISA, EclipseCon, ApacheCon, Jenkins User conferences, devops days and Velocity.   But there isn't a conference dedicated specifically to release engineering... until now.

A solid build, test, packaging and deployment story is crucial to the success of a software project.  As John O'Duinn  says, "release engineers have a multiplier effect".  In other words, a release engineer can implement automation improvements that make every other developer more productive.  Traditionally, release engineering hasn't been a common area of academic research.  However,  this is starting to change.  Another challenge is there isn't a lot of communication between academic researchers and release engineers.  The aim of the Releng 2013 workshop on May 20, 2013 in San Francisco is to bring together those two communities together: people practicing release engineering and the academic researchers studying it.  It will be co-located with ICSE 2013, which is the largest academic software engineering conference. 

Image ©thomashawk,  http://www.flickr.com/photos/ekai/457004988/ licensed under Creative Commons by-nc-sa 2.0

Are you a release engineer who'd like to discuss the challenges you face and share your experiences with others? Or are you an academic looking to expand the audience for your research and discover new problems to analyze? If so, we encourage you to to submit a paper or a talk and attend the workshop.  Or if you just want to hear some great war stories, in both open source and commercial environments, this would be a fantastic place to learn.  More details are on the web site.  You can also follow us on twitter or Facebook.  We look forward to seeing you in San Francisco!

Notes:
Greg Wilson's article on the two solitudes (industry and academia) is an interesting read and underscores the importance for more interaction between these two communities

Read more...

Mozilla Release Engineering++

Note: I wrote this a month ago but didn't have time to finish this post until now. Bugzilla > blogging.

In early August, I attended my inaugural work week with Mozilla release engineering.  I met many of our team for the first time in person at the San Francisco office.

It was a fantastic week.  Running along the beautiful waterfront in the morning and talking about open source builds all day is my idea of a good time :-) On the flight home, I started thinking about why working as a release engineer at Mozilla makes me very happy.

1)  The people on the release engineering team are friendly, welcoming and genuinely helpful.  Not only that, but they have had interesting experiences outside of work.   From volunteering at Burning Man, to dowsing burning trees as a firefighter, we heard many interesting stories.

2)  We have a lot of build and test infrastructure, both virtual and not.   Due to the sheer amount of hardware, we have scaling issues which are interesting and complex to explore.  For developers, the available capacity means that they don't have to wait long for a build to start to test their changes.   As one of my coworkers stated, the scale of our continuous integration systems is quite unique in the open source world.

3)  We share the the load of interruption driven work as individuals to make the overall team more productive.  Every two months, you're on buildduty.  That week, you're responsible for answering developer questions and ensuring our infrastructure is operational at optimal capacity.  This allows other members of the team to concentrate on the projects they are working on without being constantly pinged in IRC.  We have beta builds every week and releases every six weeks.  These are also covered by team members on a rotating basis, called releaseduty.

4)  The other teams at Mozilla are also very helpful.  I'm really stunned by their generousity to set aside their work at a moment's notice and help you debug a problem so you can move forward.  One of the great things about working at Mozilla is that people are all working toward a common goal, to make the web a better place. And this means that they want to help everyone move forward, not just the team they work on.

5) Due to the inherent scaling issues and the fact that very few organizations run the volume of builds and tests we do, we write a lot of our own tools.  This is a great opportunity to diversity your skill set. Python, Puppet, Buildbot, AWS, MySQL, system administration and tooling for Windows, Linux, Mac and Android, wrangling Hg, Git and more.

6)  Management is committed to hiring great release engineers, no matter where they live.  This allows you to do the work you love, from the place you love to live.

7) Release engineers aren't considered second class citizens within the engineering group.   They are just another group of engineers who contribute, and are respected members of the community.

Release Engineering branded laptop bags? Yes

8)  We are data driven and constantly seek to improve our metrics.  What's the wait time to run a build on a certain platform?  What 's the utilization of our infrastructure?  What's the trend for the number of checkins per month? What platform could benefit from additional test slaves?  How can we optimize the build to make it faster?  How can we improve our automation story?   Continuous improvement is fantastic.

9) Inherent in the fact that work on open source projects, our culture is very open.  This means that we have very interesting speakers visit and discuss their build story and we can learn from them to address challenges in our own environment.

10) Working as a Mozilla release engineer is no walk in the park.  The learning curve is steep and it's very challenging work.

Walks in the park are nice, but I like the endorphins from running hard too :-)

Read more...

New platform tests: OS X Mountain Lion

Late Monday night, we moved a new testing platform into production.  We have 80+ shiny new Mountain Lion slaves running tests for desktop builds, with several reserved for staging tests.  There are a few issues with failing tests that are being tracked in bug 786084.

Image ©ekai,  http://www.flickr.com/photos/ekai/457004988/ licensed under Creative Commons by-nc-sa 2.0

This was one of the larger projects I've worked on since I started at Mozilla a few months ago.  Many of the existing test and builds slaves that we run are configured with an old installation of Puppet (0.24.8) that we call "pre-historic puppet".  We have a newer installation of Puppet running 2.7.19 nicknamed PuppetAgain that serves as the Puppet Master for these new Mountain Lion slaves.   The PuppetAgain installation used to just support CentOS slaves, so we added modules to support the Mac slaves and accommodate all the quirky Apple specific configuration needs.   
 
Given my experience as an Eclipse committer, I installed the Geppetto project as my Puppet manifest editor and it's a great tool.  Thanks to the folks at Cloudsmith for developing it, it has been very useful. 

In addition to becoming familiar with Puppet, I also had the opportunity to learn about the releng buildbot configs, adding new buildbot masters, updating graphing databases to store Talos results, running staging tests and all the steps to add a new platform. Since the process wasn't documented, I wrote up some notes to help others if they need to do the same.

Thanks to everyone on the release engineering and release engineering operations teams for all their help, especially answering my many questions and reviewing patches.  Its great to see something that you worked on in production, and finally generating green builds.

Read more...

Learning to speak like a native Mozillian

Aside from gaining experience with the technical requirements of a new job, there's also the challenge of learning the terms within the community as well as cultural norms.  There's time spent finding the right code repositories, documentation and test servers.  Who's the right person to cc on a bug or ping in IRC?  When learning a foreign language, they say that if you're still translating in your head before you speak,  you're still learning.  Here are some terms and conventions I learned during my first weeks at Mozilla. 

Release some code to a production branch or stream on the canonical repo
Mozilla:  to land
Eclipse:  to release or commit

People
Mozilla: Lots of people named John, Chris and Mike, many Canadians
Eclipse:  Lots of people named John, Chris and Mike, Canadians represent too

Image ©meddygarnet, http://www.flickr.com/photos/blakespot/5240888169/sizes/z/in/photostream/ licensed under Creative Commons by-nc-sa 2.0 

Where bugs rain down from
Mozilla: https://bugs.mozilla.org
Eclipse: https://bugs.eclipse.org

The source of all truth, which may contradict itself
Mozilla: http://wiki.mozilla.org
Eclipse:  http://wiki.eclipse.org

The version control system(s) to alternately love and shake your fist at
Mozilla: Hg with some Git, Subversion, CVS and Bazaar
Eclipse:  Git with a side order of CVS and Subversion.  (Side orders will be deprecated soon)

Image ©clsung, http://www.flickr.com/photos/clsung/310886130/sizes/z/in/photostream/ licensed under Creative Commons by-nc-sa 2.0

Colloquial expression for the associated open source foundation
Mozilla: MoFo
Eclipse: the foundation

Time for liquid refreshment
Mozilla: MFBT
Eclipse: Beer o'clock

Image ©Cambridge Brewing Company , http://www.flickr.com/photos/cambridgebrewingcompany/5619040409/sizes/z/in/photostream/ licensed under Creative Commons by-nc-sa 2.0

Communication channels in order of importance
Mozilla: IRC,  mailing lists, Bugzilla
Eclipse: Bugzilla, mailing lists, Twitter, IRC

Image ©blakespot, http://www.flickr.com/photos/blakespot/5240888169/sizes/z/in/photostream/ licensed under Creative Commons by-nc-sa 2.0

Code review
Mozilla: review flag in Bugzilla for all patches
Eclipse:  review flag in Bugzilla at the end of the release cycle, others use Gerrit all the time

Incredibly smart and friendly people with a passion for delivering fantastic open source software
 +1 Mozilla and Eclipse

What have I missed?

Disclaimer: This is just based on my own personal experience.  YMMV.

Read more...

Challenges in Release Engineering

I've been at Mozilla since the end of April.  The learning curve is steep but I'm having fun climbing.  My coworkers are very friendly and helpful while answering my barrage of questions with respect to how things work.   One of the things I've noticed is that there are many common challenges in release engineering, no matter what you're building. Here's my list so far:

1. Signing builds is like a falafel sandwich.  (Always includes some pita).

Image ©chotda, http://www.flickr.com/photos/santos/3531853459/sizes/z/in/photostream/ licensed under Creative Commons by-nc-sa 2.0

2.  Scaling your build to manage infrastructure utilization, tooling to manage that infrastructure and optimizing build parallelization is extremely challenging.  Developers will consume all available build infrastructure and then ask for more.  Scaling build infrastructure to accommodate future growth is an ongoing process. 

3. Proliferating numbers of platforms on which to build, test and run performance metrics add complexity.

 Image ©misterbisson, http://www.flickr.com/photos/maisonbisson/109211670/sizes/o/in/photostream/ licensed under Creative Commons by-nc-sa 2.0 

4.  Update game adventures.  When you have an open platform that included the installation of third-party components, it's inevitable that you will encounter unexpected update cases in the wild that weren't reflected in test cases.

5. Frequent releases generate faster user feedback on new features.  However, additional releases are expensive for the release engineering, quality assurance and release management teams.  Each additional release eats time, both people and machine.  Making the community aware that builds are not free is an ongoing communication exercise.

Image ©aarongeller, http://www.flickr.com/photos/aarongeller/360135019/  licensed under Creative Commons by-nc-sa 2.0

6.  You can have great documentation and process but the accumulated technical and tribal knowledge required to resolve a complex and broken build quickly is not earned by anything other than experience.

  Image ©Ian Muttoo, http://www.flickr.com/photos/imuttoo/2631466945/sizes/z/in/photostream/ licensed under Creative Commons by-nc-sa 2.0 

7.  If you can compile your code, this doesn't mean there won't be issues with packaging, signing and testing it.  Making a build available in a format for millions of users to consume > code that compiles.  Education is needed to make people aware of this distinction. 

What release engineering challenges do you face?

Read more...

Moving to Mozilla

Eclipse family: March 23 will be my last day at IBM. I'm happy to announce that I'll be joining the Mozilla Release Engineering team in April. 

Image ©flod, http://www.flickr.com/photos/flod/2221300134/sizes/l/in/photostream/ licensed under Creative Commons by-nc-sa 2.0

We all wear many hats during our lives.  I've always been fiercely proud of the privilege of calling myself an Eclipse committer.  We've accomplished amazing things together.  I look forward to seeing the continued growth and success of the Eclipse community. 

At Mozilla, I'll have the opportunity to work with a completely new software stack, and learn many new skills while working with very talented people. Best of all, my job will involve contributing to an open source community.  This is what I love to do.  The Mozilla goals of making the web better, and teaching a new generation to make content, instead of just consuming it, really resonate with me. Running builds on 1200+ machines raises some interesting scalability issues which I'm eager to learn more about.  It will be challenging work, and I'm honoured to have the opportunity to work at Mozilla.  I'll be working from my home in Ottawa. 

Where there are new challenges ahead for me, at the same time it has been a difficult decision to let my contribution to Eclipse wane.  I'd like to continue contribute to Eclipse in some way but on a part-time or occasional basis.

My IBM colleagues, my fellow committers and the Eclipse foundation staff:  you are all extraordinary.  It has been a privilege to work with you and learn so much.  To those in the larger Eclipse community, you inspire me.  To the people on the Eclipse SDK and Equinox teams, I have never had the opportunity to work with such a wonderful group of people.  Not only are you creative and technically brilliant, but you are all so dedicated to doing the right thing and getting things done.   You're all genuinely nice people, without ego, who are eager to share your knowledge with others.   It's not often that software project ships on time every year for ten years and continues to delight its user community.  It is a testament to the team behind it that this continues to be the case.  

In the process of working on Eclipse, I've met some fantastic people, many who have become great friends.  I'll miss the seeing your faces every day and discussing the finer points of Git migration strategies, but I'll still be on IRC, Twitter, Linkedln, G+ etc.  If you live in the Ottawa area, I'm always up for lunch :-) 

It has been a great run. Thank you all

Note #1:  I'll continue to write on this blog. The name still applies :-)
Note #2: I was going to call this post "Firefox-y Lady".  Then I read the lyrics to the Hendrix song "Foxy Lady" and decided maybe not.

Read more...

2011 by the numbers

2011 was an exciting year in the Eclipse community.  From my corner of the Eclipse universe, he's what it looked like:


One book chapter, many thanks

I contributed a chapter on Eclipse to the Architecture of Open Source Applications in 2010 and the book was published in May 2011.  Thanks to Amy Brown and Greg Wilson, for their long hours editing and providing feedback to the authors of this book.  It's a great read!  When Greg first approached me about writing this chapter, my immediate thought was "How hard could it be? I live and breathe Eclipse all day".  It was much more difficult that I imagined but in the process I learned a tremendous amount and am a better committer for the experience.  Many thanks to DJ Houghton and John Arthorne for reviewing my drafts and providing valuable feedback. A special thanks to Jeff McAffer who I interviewed about the decision to switch to OSGi in 3.0 and Steve Northover for his suggestions to make the SWT section into something more pixel perfect.  Merci Olivier Thomann for answering my many compiler questions,  and Boris Bokowski and Paul Webster for their thorough discussions with me regarding the modelled workbench and dependency injection in 4.x.  Also, thanks to Mike Wilson to allow me the flexibility in my job to spend some time at work working on this chapter.  I'm excited to see that Amy and Greg are now editing a second volume of this book.

Six milestones, many release candidates, two service releases, and one coordinated release, four streams, thousands of builds, millions of tests
No rest for the committers.

143 bug fixes
I closed about 143 bugs in the releng bucket in the past year.  That doesn't seem like much really.  I'd have liked to solve more.  The largest issues implemented from a releng perspective were shared licenses, code coverage, and the largest work item, the Git migration.

42 Git repos 
The Equinox and Eclipse projects migrated all their repos to Git.  We now have about 42 Git repos.  This involved a tremendous amount of work on the part of the Eclipse team as a whole.  There were many whiteboard drawings and detailed discussions about the migration process with John, Paul and a Mr. Gheorghe.  There was no Ringo.  Thank you Paul for all the huge amount of testing, script writing, and migration of all the ui and e4 repos. Thanks John for your work many sage suggestions on our Git migration, as well as your suggestion to implement the git flow method to simplify our development and build processes.  Thanks Andrew Niefer for migrating many of the Equinox and PDE repos, Bogdan Gheorghe for your work with SWT, and Oliver Thomann for testing JDT Core repos.  Thanks Tom Watson for your Git advice, having already climbed the Git learning curve while working on the OSGi Alliance repositories.  To Dani Megert and Markus Keller, your always fine attention to detail and pointing out areas that could be improved is appreciated. Paul is giving a talk about our migration at EclipseCon 2012 called Let's Git this Party Started.  I'm sure it will be insightful and entertaining.

One EclipseCon, two talks, one castle, many great people
I was privileged to attend EclipseCon Europe in Ludwidsberg this past November and present two talks.  I thoroughly enjoyed preparing these talks, and even more presenting them.  On the Wednesday morning, I talked about our Git Migration, and that evening I gave a talk with John Kellerman about history of Eclipse over the past 10 years.  After the second talk, a few people came up to me and said that the talk was so good that it should have been a keynote.  That was very fantastic to hear because we really put a huge amount of effort into that presentation.  I also had a lot of fun talking to people at our booth where we had posted many pictures of the Eclipse family from over the years. The Saturday after the conference Simon Kaegi, Eric Moffatt and I visited Heidelberg castle.  Canada scores very low on the castle index so this was a treat.   

You can't buy Eclipse magazines or giant pretzels at train stations in Canada either. I was impressed.

19 blog posts
I didn't have much time to write blogs posts this year.  The most popular one I wrote this year was about smashing open source stereotypes.

I'm never sure how popular a blog post will when I write them. It's always a surprise.  The comparison of Mozilla and Eclipse build infrastructure I wrote last year still holds the record for most popular (it ended up on reddit).


One marathon, many kilometers of training
How is running related to release engineering?  Running keeps me sane when release engineering gets crazy :-)  Preparing for the Ottawa marathon in May means that you have to start training at the end of January.  Running through snow, ice, wind and rain teaches you there isn't really anything you can't do when you are willing put in a lot of hard work to reach your goal.  And when you reach that goal, there's a lot of joy, because you know that you have conquered all the obstacles in your path and emerged victorious.

My sneakers after a 19K training run through deep slush

Open source is really a huge team effort and I had a lot of fun in the Eclipse community in 2011.

Who knows what 2012 will bring?

Read more...