Releng 2013 keynotes: John O'Duinn (Mozilla) and Roman Scheiter (LinkedIn)

>> Friday, May 10, 2013

There are two exciting keynotes planned for Releng 2013

John O'Duinn, Director of Release Engineering at Mozilla will kick off the workshop with his keynote Release Engineering as a Force Multiplier.  The build and release process used to be a pain point at Mozilla, but now makes the company and community more productive as a whole.  John will describe how the team added support for project branches to allow concurrent development, rethought continuous integration and increased capacity by moving to a hybrid-cloud build infrastructure. These changes improved several aspects of the business, including switching to a rapid release model and reducing turnaround time on a release from weeks to hoursAs a result,  Mozilla improved its abilities against much bigger and better funded competitors in the marketplace while also allowing them to enter new markets and help ensure its long-term success.


Roman Scheiter, Engineering Services Director at LinkedIn, will present the afternoon keynote entitled Against All Odds – Completely Overhauling Linkedin's Release Process.  This session will cover the evolution of LinkedIn’s release process from its earliest days to the point where the rapidly growing engineering team necessitated a radical shift. This shift, an executive sponsored effort to address technical debt and introduce new thinking to boost engineering efficiency allows six hundred developers to release thousands of changes per week without compromising quality.  As part of this undertaking, LinkedIn learned many best practices, developed tools and custom infrastructure, and lived through the internal cultural changes needed to make this independent release process work.   Roman will detail the evolution and results of this shift so you can learn directly from LinkedIn's pain.

 
In addition to these fantastic keynotes, we also have talks from release engineers and researchers from Netflix, Google,  Microsoft, Gnome, Red Hat, IBM, several universities and more!  We'll also have a panel at the end of the day to discuss the future of release engineering.

Check out the full program on the Releng 2013 site.  To register for the conference, which is managed as part of the larger ICSE conference, you can follow this link and choose the one-day-workshop.  See you in San Francisco on May 20!

Read more...

Thoughts on being a second generation geek

>> Tuesday, March 26, 2013

When I was in Toronto for our last work week,  I mentioned to one of my colleagues that I was a second generation geek.  He replied something to the effect of "There are families that have been fishing lobster for eight generations but not many second generation geeks."

I'm a second generation geek thanks to the influence of my Dad.  My sister also works in the software industry, as a technical writer.   My parents both had a tremendous influence on us.  If it wasn't for their support and encouragement, I doubt that we would be where we are today.

My Dad's first technology job was working at Burroughs  in Vancouver, BC as a field service technician and manager.  He would travel to customer sites to fix their computers, usually at banks,  insurance companies or government.  Back then computers were very large, not very common, and they had a longer lifecycle than we see today.  He had a briefcase full of tools to fix them,  given that most problems were mechanical or electrical in nature, as opposed to software.

About a decade later,  our family moved across the country and he started working as a technical services manager at a small software and services firm in Halifax, NS. This was the dawn of the PC era, and he brought home a succession of computers for him to learn about, as well as the rest of our family.  Dad would bring home stacks of computer magazines and manuals to read at nights on weekends, which I started to read as well.  We had many spirited discussions about technology, and what the future would hold.

In addition to having a wealth of technical knowledge, my Dad is also an accomplished woodworker  and builds beautiful furniture.  He bought my sister and I hammers so we could work along with him in the workshop.  There was also an abundance of Lego,  his old Meccano kit,  and stacks of science fiction books.  Christmas meant the inevitable gifts of conference swag in stockings, or a Linux book under the tree. 

Over the years, I have returned the favour and given my parents a lot of the swag I've received at work.  I'm sure they have the finest collection of Eclipse wear in Nova Scotia.  But this year begins a new tradition.


My Dad loves Firefox too

I've noticed that many of my (female) friends who work in software had a parent who worked in the industry and thus were a source of influence for their choice in career.  I learned from both my parents was to work very hard.  From my Dad, I learned that
1) Change in this industry is constant and it makes things interesting.
2) Continuous learning is essential.  Take courses, read voraciously,  break things and fix them.
3) STEM careers are for women.

Thanks Dad!

This past weekend, I asked him if there were ever any women at tech conferences when he attended them in the 1980s or 90s.  He replied that there was usually one or two, but that was all.

He then proceeded to tell me a story about a technical sales manager who he worked with for many years named Eli.  She had worked in an administrative capacity at Digital Equipment Corporation (DEC), but wanted to become a salesperson.  She was told that there was absolutely no way that she could be become a salesperson and represent DEC to customers because she was a woman.

In those days, computers were sold from OEMs (original equipment manufacturers like DEC) or VARs (valued added resellers).  The company where my Dad worked at was a VAR and had agreements with various OEMs, including DEC.  They would package hardware and software from several companies, along with installation and support services into sales proposals to meet customer requirements.  The funny thing was that VARs and OEMs would often compete for the same customers since they both sold the same branded hardware.  The VARs weren't limited to selling to a single vendor, but the OEMs had more flexibility in pricing in margins since they were the manufacturer.   Sales margins on hardware were quite high compared to what they are today, and thus a job as a salesperson could be quite financially lucrative if you were good.

Eli left DEC to work as a sales manager at the same office where my Dad worked.  She proceeded to outsell all the salesmen at her her previous employer who had refused to let her be a salesperson because of her gender, and won many sales awards.  So awesome!

I was always was a bit intimidated by Eli.  She was tough as nails, with a wicked sense of humour.  I can imagine that it wasn't always an easy path she had to walk, given that being a professional woman and working in technology was not common at the time, especially is socially conservative Nova Scotia.  But she did it, and proved that she could rise above her detractors.

A second thanks to women who have blazed trails in the past :-)  Who inspired you to choose your career in STEM?

Read more...

Who's speaking at Releng 2013?

>> Friday, March 15, 2013

Release engineering helps bring many of the products we use every day to market.  From entertainment, such as Netflix

http://www.flickr.com/photos/myetvmedia/8478881935/sizes/o/in/set-72157632783339196/
to search,

to operating systems and tooling,



to browsers and mobile operating systems that unleash the power of the web,


to software to configure the thousands of servers that allow us to build these products, and serve them to our customers.


Release engineers and researchers from Netflix, Puppet Labs, Mozilla. IBM, Google, Microsoft, Red Hat, Gnome and several universities will be giving talks at the Releng 2013 workshop on May 20, 2013 in San Francisco. I'm really excited by the different viewpoints that these speakers will bring to the conference and look forward to some interesting conversations.  I always enjoy learning about complex systems work and I'm sure that a behind-the-scenes look at how these companies build, test and deploy software will be fascinating.  A detailed list of talks is available on the conference web site.

In addition to the talks, we have two fantastic keynote speakers

John O'Duinn from Mozilla:  Release Engineering as a "force-multiplier
Alan Grosskurth a release engineering consultant (formally VMWare):
Release Engineering in the Cloud Era

To register for the conference, which is managed as part of the larger ICSE conference, you can follow this link and choose the one-day-workshop. We look forward to seeing you there!

Read more...

Disabling Bluetooth Assistant and app relaunch in 10.7 via Puppet

>> Monday, February 25, 2013

Intermittent test failures are very annoying. The create noise that cause people to question the test results - is it real data or just noise?

Image ©misspixels, http://www.flickr.com/photos/misspixels/8480711076/in/photostream/ under Creative Commons by-nc-sa 2.0
Recently, we were hit by intermittent mochitest failures on our Lion Talos boxes.  The issues causing this were two fold:
1) Despite Bluetooth being disabled, the Bluetooth setup assistant would randomly start and take focus away from running tests.
2) Our talos boxes reboot and repuppetize after a test job has completed.  However, on 10.7, the default is to relaunch apps that were running before reboot.  Thus, if an app crashed, it would be restarted upon reboot and again interfere with tests.

If this was just a machine on your desktop, you could disable the preferences in the the UI.  However, since we manage our infrastructure via Puppet, I needed a command line way to implement this preference change and update our 80+ Lion test machines automatically.

To disable setup assistant for Bluetooth mouse and keyboard
 defaults write /Library/Preferences/com.apple.Bluetooth BluetoothAutoSeekPointingDevice -bool false
 defaults write /Library/Preferences/com.apple.Bluetooth BluetoothAutoSeekKeyboard -bool false 
 
To disable apps from restarting upon reboot
 defaults write /Library/Preferences/com.apple.loginwindow LoginwindowLaunchesRelaunchApps -bool false

Note: the disabling the apps from restarting doesn't uncheck the preference in the UI, but it does work.

Having spent many years as a Linux sysadmin,  I find the way that Apple manages preferences within their operating systems non-intuitive and generally poorly documented.  In any case, it took me quite a bit of digging to find the correct defaults incantation, so I thought I'd blog about it in case others had the same problem :-)

References
Bug 843545 Intermittent OSX 10.7 mochitest failures that seem related to Bluetooth Setup Assistant crashing
Bug 743594 Stop 10.7 from restoring apps after a restart

Read more...

Reminder: Releng 2013 submission deadline is February 7

>> Tuesday, February 05, 2013

Just a friendly reminder that Feb 7 is the deadline to submit talk abstracts and papers to the Releng 2013 workshop.  It will be held on May 20 in San Francisco, in conjunction with the ICSE conference.

Are you involved in release engineering, release management or continuous deployment? Have you recently migrated a code repository to Git and implemented new branching strategies? Did your team implement amazing new automation to package and sign the binaries you deliver to customers?  Are you an academic looking for new problems to research and connect with practitioners in the field? Are you a maintainer for a continuous integration system or write a new build system? Did you manage to ship on time despite the fact that one file changed on a Friday afternoon and broke everything? If so, you have a story to share at this conference :-)



We have two fantastic keynote speakers lined up - John O'Duinn, director of release engineering at Mozilla, and Alan Grosskurth, who recently started working as a release engineering consultant after several years at VMWare.

If you have any questions regarding the submission process or the conference in general, please feel free to drop me a line (I'm kmoir and I work at mozilla dot com).  See you in San Francisco!

Read more...

Android Panda tests in production

>> Tuesday, January 29, 2013

Two weeks ago the Mozilla release engineering team had a work week in Toronto.  Toronto in January?  Balmy, according to one of our of colleagues from California.
 

Hal wears sandals in the snow.  Photo: John Hopkins
Over the past few months,  Ateam, IT and release engineering have been wrestling panda boards into submission and deploying them into production as part of our test infrastructure for Firefox for Android.  (There are pandas running B2G tests as well. )   Firefox is also a colloquial name for the red panda who apparently likes snow too.

Image ©maiac,  http://www.flickr.com/photos/maiac/5442715157/licensed under Creative Commons by-nc-sa 2.0
As Dustin described in his post on Mozpool, the panda boards are development boards that sit in specialized chassis that were developed for Mozilla.  During the work week, I finished putting the  last of the 400 panda boards in production running tests on the cedar, mozilla-central, try and mozilla-inbound branches.   From a releng perspective, this involved setting up 32 foopies and 5 new buildbot masters and updating our configs.   For those readers uninitiated with our release engineering configuration, we use buildbot as our continuous integration system to schedule builds and tests.  Foopies are are dedicated servers that handle the connections of the mobile devices to the buildbot masters.

Some of the issues that we encountered while we were rolling the pandas into production were quite difficult to overcome.  The Ateam, especially Joel spent many days debugging them, including this this particular nefarious one where the panda boards would spontaneously reboot while running tests

Bug 811444 - android panda boards magically reboot in the middle of the test

Joel's solution the solution was to slow down the running of the tests so the panda boards wouldn't reboot and allowed us to move forward.

Many thanks to Joel, Callek, Armen, Amy, Dustin and Jake for all their help putting these into production.

Next step: Manage the Android  Panda boards using Mozpool

Further reading
Dustin Mitchell's post on MozPool, including pictures of the awesome custom ruby red Mozilla chassis
Bug 802667 - configure new buildbot masters for use with android on pandas
Bug 803248 - buildbot config changes to support panda_android*
Bug 805658 - add all panda boards to slavealloc, disabled  
Bug 811723 - change android reftests to run with --ignore-window-size for panda only 
Bug 825984 - Turn on Android 4.0 test jobs on try and mozilla-inbound
Bug 829181 - put remaining pandas into production for Android 4.0 tests

Read more...

Releng 2013 Call for Papers

>> Wednesday, December 19, 2012

There are many conferences that discuss various aspects of of release engineering such as PuppetConf, OSCON, LISA, EclipseCon, ApacheCon, Jenkins User conferences, devops days and Velocity.   But there isn't a conference dedicated specifically to release engineering... until now.

A solid build, test, packaging and deployment story is crucial to the success of a software project.  As John O'Duinn  says, "release engineers have a multiplier effect".  In other words, a release engineer can implement automation improvements that make every other developer more productive.  Traditionally, release engineering hasn't been a common area of academic research.  However,  this is starting to change.  Another challenge is there isn't a lot of communication between academic researchers and release engineers.  The aim of the Releng 2013 workshop on May 20, 2013 in San Francisco is to bring together those two communities together: people practicing release engineering and the academic researchers studying it.  It will be co-located with ICSE 2013, which is the largest academic software engineering conference. 

Image ©thomashawk,  http://www.flickr.com/photos/ekai/457004988/ licensed under Creative Commons by-nc-sa 2.0

Are you a release engineer who'd like to discuss the challenges you face and share your experiences with others? Or are you an academic looking to expand the audience for your research and discover new problems to analyze? If so, we encourage you to to submit a paper or a talk and attend the workshop.  Or if you just want to hear some great war stories, in both open source and commercial environments, this would be a fantastic place to learn.  More details are on the web site.  You can also follow us on twitter or Facebook.  We look forward to seeing you in San Francisco!

Notes:
Greg Wilson's article on the two solitudes (industry and academia) is an interesting read and underscores the importance for more interaction between these two communities

Read more...

  © Blogger template Simple n' Sweet by Ourblogtemplates.com 2009

Back to TOP