Challenges in Release Engineering

I've been at Mozilla since the end of April.  The learning curve is steep but I'm having fun climbing.  My coworkers are very friendly and helpful while answering my barrage of questions with respect to how things work.   One of the things I've noticed is that there are many common challenges in release engineering, no matter what you're building. Here's my list so far:

1. Signing builds is like a falafel sandwich.  (Always includes some pita).

Image ©chotda, http://www.flickr.com/photos/santos/3531853459/sizes/z/in/photostream/ licensed under Creative Commons by-nc-sa 2.0

2.  Scaling your build to manage infrastructure utilization, tooling to manage that infrastructure and optimizing build parallelization is extremely challenging.  Developers will consume all available build infrastructure and then ask for more.  Scaling build infrastructure to accommodate future growth is an ongoing process. 

3. Proliferating numbers of platforms on which to build, test and run performance metrics add complexity.

 Image ©misterbisson, http://www.flickr.com/photos/maisonbisson/109211670/sizes/o/in/photostream/ licensed under Creative Commons by-nc-sa 2.0 

4.  Update game adventures.  When you have an open platform that included the installation of third-party components, it's inevitable that you will encounter unexpected update cases in the wild that weren't reflected in test cases.

5. Frequent releases generate faster user feedback on new features.  However, additional releases are expensive for the release engineering, quality assurance and release management teams.  Each additional release eats time, both people and machine.  Making the community aware that builds are not free is an ongoing communication exercise.

Image ©aarongeller, http://www.flickr.com/photos/aarongeller/360135019/  licensed under Creative Commons by-nc-sa 2.0

6.  You can have great documentation and process but the accumulated technical and tribal knowledge required to resolve a complex and broken build quickly is not earned by anything other than experience.

  Image ©Ian Muttoo, http://www.flickr.com/photos/imuttoo/2631466945/sizes/z/in/photostream/ licensed under Creative Commons by-nc-sa 2.0 

7.  If you can compile your code, this doesn't mean there won't be issues with packaging, signing and testing it.  Making a build available in a format for millions of users to consume > code that compiles.  Education is needed to make people aware of this distinction. 

What release engineering challenges do you face?

Read more...