Mozpool and Mozharness for Android Panda tests

Over the last few months, I migrated the test infrastructure that we run for Android 4.0 on Panda devices to use mozharness and mozpool. Before I go into a description of what this entailed, some background for those not familiar with our release engineering infrastructure.  We use Buildbot for our continuous integration engine. We have 50+ Buildbot masters are designated specific purposes such as scheduling, building, try servers and testing.  Test masters are further subdivided into ones specifically allocated to running tests on Mac, Linux, Windows, Tegras (Android 2.2 devices) and Pandas (Android 4.0 devices).  We have many devices that are allocated to each master to handle the volume of tests and builds we run each day. 

Why Mozpool?

We have over 800 panda devices in production that are used to run unit tests and talos (performance) tests for Firefox for Android 4.0.    Panda devices are rack mounted in a chassis.  One of the issues with these devices is that they are development boards and inherently quite unstable when running a large volume of tests 24/7.  Not many organizations run the volume of automated tests on mobile devices that we do.  Dealing with their issues in a non-automated fashion does not scale.

Pandas fall down

Mozpool is software that is used to mitigate that inherent stability and ensure that the devices that are available to run tests are in a verified state and have the correct Android image.  If Mozpool determines that the device has problems that makes it unsuitable to run tests, it changes its state so it won't be in the pool of devices running tests.  For instance, if it doesn't boot correctly or the requisite image cannot be applied. This reduces the number of jobs that fail due to infrastructure issues.  Mozpool also provides logging of the actions on devices and and a web page for looking at the state of your devices.  There is an API implemented vi REST and HTTP for simple actions on your devices, such as requesting a device, and returning it to the pool.  Another advantage of Mozpool is that it's easy to reimage the devices with a new image, either via the mozpool ui or simply specifying a new image in your mozpool client code.

Why Mozharness?
There are four main projects that are used to manage our infrastructure buildbotcustom, buildbot-configs, tools and mozharness. From the mozharness FAQ in Aki's words "Mozharness is a configuration-driven python script harness with full logging that allows production infrastructure and individual developers to use the same scripts."  Implementing mozharness scripts allows developers who don't have access to our infrastructure to run the same scripts we do on production hardware.   

Traditionally, the scripts that define the buildbot actions for our tests have been defined in the buildbotcustom project.  The code is very convoluted and it is difficult to new people to parse what bits of code apply to each platform.   Also,  every time we want to deploy changes for them we have to do a reconfig.  A reconfig is an operation where we upgrade our buildbot masters to the latest version of the code in the buildbotcustom, buildbot-configs and tools repositories.  A reconfig is usually initiated by the releng person on buildduty only a few times a week.  When you use mozharness, the buildbot config scripts point to the production branch of the mozharness repo. New changes can be deployed to production with a simpler merge to the production branch of mozharness.  No reconfig.

Closeup of zip line harness aka visual representation of some buildbotcustom code

 Image by Image ©timcaynes, http://www.flickr.com/photos/timcaynes/1588430260/under Creative Commons by-nc-sa 2.0

With mozharness scripts, you have a discrete script and config file that applies to each platform.  For instance, for panda android unittests, the config file is here, and the corresponding script is here. 

There are actions in mozharness that serve as boilerplate code that can be reused to common actions when running tests, such as installing Python packages into a virtual environment, closing repositories, downloading and extracting files and so on.  For instance, here are the default actions when running the android unit tests on Pandas:

These actions correspond to methods is classes that that your mozharness scripts will inherit. For instance, the PandaTest class in the mozharness script in inherits VirtualEnvMixin which has a method create_virtualenv to install the appropriate python modules for the test run.

 

The request-device action uses the mozpool client code within mozharness to interact with the mozpool and verify that the device is in a usable state to run tests. You can also override these methods with ones in your own mozharness script if you have changes that are unique to your platform.

If you are have questions regarding mozharness, please join the mozharness channel on irc.mozilla.org :-)

Further reading
Dustin Mitchell's description of Mozpool
Mozpool API documentation
Aki Sasaki's Mozharness FAQ 
Aki Sasaki's first blog post on why mozharness and more

Read more...

Mozilla Summit thoughts

The first weekend of October, I attended the Mozilla Summit in Toronto.  This was the first time I attended a summit since they only occur every few years.  The summit is a gathering of the Mozilla community, both paid contributors and volunteers.  Unlike other open source conferences I've attended,  in that the focus wasn't solely on the amazing new products and initiatives we are working on.  The major purpose of the summit was on the how to improve the open web and community as a whole, attract new contributors, and plan for the future. It was wonderful so talk to so many people in person that I only normally communicate with via IRC, as well as meet new people.

I really enjoyed the conference and would like to thank the organizers for all the hard work that went into it.  Conference organizing is an all consuming task and I'd like to thank those that make the summit happen, and run so smoothly.   Many people have already written about the amazing technical achievements that were unveiled, here's a good summary.  Coop blogged about the summit from the perspective of release engineering.  Many things struck me about the conference from a community standpoint.

There were many more diverse faces then I've seen at other open source conferences.  Lots of young people, even some teenagers who were excited to be contributing to the Mozilla community.  People from South America, Asia and Africa.  There were also a larger percentage of women than I've ever seen at an open source event.  I always joke that one of the good things about being a woman in open source is that there never is a lineup for the bathroom at break times.  For this conference, I had to wait in line once,  behind a colleague wearing an Ada Initiative shirt.  I had meals where I wasn't the only woman at the table, this was very refreshing.  I'm the only woman out of 18 people on the Mozilla release engineering team.  My colleagues are all great but at the same time it was amazing to meet and speak with so many talented women who are passionate about open source.

There were keynotes every morning and the remainder of the slots were working sessions where people could come together around a common topic and work together to formulate ideas to make things better for the future.  For instance, I attended workshops on topics such as organizing your project for contribution.  One of the interesting points that David Eaves and Mike Hoye brought up in this session is that the number one predictor that a contributor will continue to make contributions is if there initial bug is triaged into to the correct bucket and they receive feedback.  The faster they receive feedback, even if the feedback is that the bug won't be looked at for two weeks, the more likely it is that they will make another contribution. I also attended a session on how to scale our community to larger numbers which included discussions regarding the research of volunteer organizations scale, educational theory and how to maintain volunteer engagement.   Very different from my usual work topics of how to scale infrastructure and build capacity but very interesting the same.

Liz Henry set up a table where you could make jewellery with your favourite bug number or your irc nick.  What a fantastic idea, thank you Liz.

Saturday night had a lot of fun social events including a hockey (shinny) game at Maple Leaf Gardens where Potch served as announcer.  Thank you Lawrence Mandel and Lukas Blakk for organizing, it was amazing.  A couple of the people in the game were pretty new to skating and had never played hockey before but they did fine! In the end it was a pretty close game.

On the way back to the hotel I had the chance to explore the art of Nuit Blanche with some colleagues.  It was beautiful.

Ai Weiwei's Toronto installation of Forever Bicycles in Nathan Phillips Square.

I've never been to a conference where the focus is so much on "what can we do next" instead of "what we are doing now".  I wonder if other open source communities do the same. What did you enjoy most about the Mozilla Summit?

Read more...

Meeting little Releng

Seven years ago,  Mr. Releng and I filled out the paperwork for international adoption.  As anyone who has pursued adoption is aware, this path is often very long and fraught with unforseen obstacles, no matter the country where you apply.   So for a very long time, it seemed that everything that could go wrong, did go wrong and made for a very long wait.  But this summer, we travelled to Bangkok, Thailand and something went very right.

Our son aka little Releng

I may be biased, but he's pretty awesome.  I didn't realize how amazing it is to have a child in your life, it is such a gift.  In Canada, you can take up to nine months parental leave for adoption. Mr. Releng is taking the first half, while I'll take the second half. 

As an aside, we have never visited any countries in Asia before and so our visit to Thailand was quite an adventure.  We spent most of the time in Bangkok, so I can't comment on the rest of the country.  But it's quite an extraordinary city with beautiful sights, very friendly people and delicious food.  Even though it's literally half way around the world from here,  I would recommend as a fascinating place to visit.  Bangkok feels like it is in the future, with it's gleaming tall buildings,

yet at the same time it has an amazing past to explore as well.

Statue at Grand Palace

Read more...

Resources for learning Python

When I started working at Mozilla, I didn't know Python very well.  All of the code that drives our continuous integration is written in it so I've had a spent a lot of time becoming more proficient.  I've taken a number of free self-study classes/tutorials that others might find useful getting up to speed with Python.

Google Python class  - Two day class that describes how to manipulate strings, dictionaries, lists. File I/O, regular expressions and sorting algorithms.  Class lectures that were recorded at a Google training event are available on youtube, and the exercises are pretty straightforward with a solution key to check your work.  All the exercises are done using the Python installation on your local machine.

Code academy has a Python class.  I did the exercises for the topics that weren't covered in the Google class, including list comprehensions, lambda expressions and OO in Python.  The exercises are completed and validated on the website.

The best course I've taken so far has been the nine week Interactive Programming in Python class offered by Coursera. Coursera is a company that offers university courses for free.  You receive a grade and certification of completion,  but not an actual credit at the institution. The Python course I took is in conjunction with Rice University.  For this course, you have to register for the course when it's run and complete the work each week.   There are an hour or so of video lectures, a quiz or two and an assignment to complete for around 8-10 hours of work each week.  After you submit your assignment, you're asked to review the work of five other students.  This is how to scale marking assignments for the thousands of students enrolled in the class :-).  The code for your assignments is written in the browser,  on the CodeSkulptor website.

Syllabus for coursera course

Each week the assignment is to implement a different game which was a lot of fun.   We also learned how to implement the calculations for velocity, reflection and collisions to make the objects in our games appear as a realistic game.   The final project is to implement a version of the Asteroids game complete with missiles and exploding rocks.  I've also never done much with a the UI aspects of programming in the past, so it was interesting to learn how to incorporate these elements.

Screenshot of final asteroids game

This iteration of the course is almost over and the next one starts in October.  Essentially the course provides lots of time to practice writing code, which is the only path to getting better.  I found the course to be really well organized, and the video lectures were succinct in getting to the point of the lecture.  No profs going off on tangents about random subjects :-)

Mr. Releng also took this course at the same time I did as he is also using Python at work.  I said to him one day that I certainly wouldn't wanted to have attended all my university courses online as I always liked to ask questions in class.  With the pre-recorded lectures, the only way to ask questions is on the associated forums.  He agreed, but said while this was probably not the future of university, it was certainly great for continuing education.

Other Python resources I looked at  include http://www.learnpython.org/.  The pyvideo website includes talks from many of the PyCon and related conferences here which are interesting to watch.

What resources did you use to learn Python?

Read more...

Lessons learned organizing a technical event

I was on the organizing committee for Releng 2013.  This was the first time I've been involved in organizing a technical event.  I thought my experiences might be useful to others in the same situation thus this blog entry:-)

Reaching out to potential speakers
Three of the people on our organizing committee were researchers.  I was the only release engineer.  My goal was to get as many release engineers as possible to submit talks and attend the workshop.  I asked for suggestions from my coworkers and LinkedIn contacts to try to reach people that might be interested in submitting a talk.  I also wrote to multiple release engineering oriented mailing lists and Google+ groups.  I had a query for #relengcon and @relengcon mentions on Twitter.  If people mentioned that the relengcon sounded interesting, I replied to them and suggested that they submit a talk.  I also posted information about the workshop on several release engineering LinkedIn and meetup.com groups.  It's often a good idea to contact the organizer of a group and ask them to send text crafted by the organizers to their members.  Receiving a message from someone they already know lends some validity to the message, versus just another person adding to their inbox noise.

Carl and Gareth from Netflix talk about self-service build and delivery

Academic vs. Industry
Releng 2013 was a workshop under the larger ICSE academic conference.   The submission process was much more formal than the industry conferences I've attended in the past.   There were two types of submissions: a formal paper or an abstract for a talk.  There were strict submission guidelines with respect to the format of the talk abstract.  In contrast, the submission process for most industry conferences seems to be "submit a paragraph summarizing your talk into a form on web page".  Very easy.  The different approach stems from the fact that proceedings from academic conferences are usually published.  Academics often need the promise of publication to secure funding to attend a conference.  

Funding
This workshop had effectively zero budget.  ICSE workshops don't get any funding from the fees paid to the larger ISCE conference.   Attendees had to pay registration fees to attend the workshop whether they had a talk accepted or not.  Many industry conferences offer a reduced registration fee for speakers.  Since this was a first time event, I really concentrated on encouraging people to submit talks and register for the event.  Now that the event was a success, I would feel comfortable approaching a company and asking them to sponsor it so we could reduce speaker fees.  We had some funding from a research institution for one of our keynote speakers.  I reached out to Mozilla developer relations who provided some t-shirts,  mugs and stickers.  In the future, I'd also reach out to other companies for swag donations :-)

Recording of sessions
The cost of hiring a company to record the talks was prohibitive given our lack of budget.  Again, now that the first workshop was a success, this could be another item that could be sponsored. 

Post-workshop dinner organization
The OpenTable website is a great resource to see the availability of seating at  of various restaurants in several US cities.  It also links to their respective ratings.  This was easy way to make reservations at restaurants for dinner, especially since none of the organizers live in the Bay area.

Image ©photographus, http://www.flickr.com/photos/misspixels/8480711076/in/photostream/ under Creative Commons by-nc-sa 2.0

 
Panel
I had never moderated a panel before.  I found this article was quite helpful on how to successfully moderate a panel.  I spent quite a bit of time before the event thinking of panel questions and we also sought input from our attendees in a survey.   As for participants on the panel, I reached out to release engineers who didn't present a talk but looked like they had a wealth of release engineering experience on their LinkedIn profiles.  Sometimes diversity is just asking different people!

Speaker gifts
We wanted to give speaker gifts to our fine keynote speakers.  I was unsure what a typical speaker gift comprised so I asked on Twitter.   The answers ranged from fancy pens, laptop bags, local bottles of wine and local cookbook.  Another person suggested that the gift be something carried on the plane if the speaker wasn't local.  In fact, they usually personalized the gift by presenter and shipped it to their home to avoid dealing with airline hassles regarding gifts.  Release engineering is about building things.  Bram suggested Lego as a speaker gift.  I liked this idea a lot and added it as one of the items in the speaker gift bag.

In the end the workshop went pretty well.  The most important thing was to get people who are passionate about release engineering in the same room.  And that's where the interesting conversations begin and you make connections with people.  That's what really matters in the end.  

If you've been involved in organizing a technical event what advice do you have?

Read more...

Releng 2013 Recap

On May 20, we held the Releng 2013 workshop in San Francisco.  By all accounts it was a success!  I was unsure what to expect given that this was a first time for this workshop.   It seemed to be a topic that generated a lot of interest.  We had around 80 attendees which to my understanding was the highest attendance of any workshop at ICSE this year.   It was great to have so many people interested in release engineering and research in the room.  It was also fantastic to meet my co-organizers Bram, Foutse and Christian in person after working to organize this event since last fall.

The night before the workshop, I went out to dinner with the other organizers and some of their professor and post-doc colleagues.  I was struck by how enthusiastic they were about release engineering topics and interacting with industry.  One of them said  "We want to make sure that we're working on relevant problems." 

There were two keynotes at the workshop.  The opening keynote was Release Engineering as Force Multiplier by John O'Duinn of Mozilla.  The afternoon one was by Roman Scheiter of LinkedIn and entitled Against All Odds – Completely Overhauling Linkedin's Release Process.  Both were fantastic accounts of how the build, test and release pipelines at these companies were improved to make the organizations as a whole more effective.  It was an interesting contrast in that LinkedIn
moved from branch based development model to trunk based continuous integration while Mozilla moved from trunk based continuous integration to branch based development model.

One of the researchers commented that if you have negative results, you don't publish a paper.  So they were somewhat surprised to see the openness from those from industry on the things we did wrong.  My understanding is that's not that common in academia given the push to be the first to publish a new result.  A different culture.

Some highlights for me from the other talks:

• The software delivery model at Netflix by Curt Patrick and Gareth Bowles (development islands, no release engineers, their team provides tooling for self-serve builds)
• Hal Wine from Mozilla on using Hg and Git on the same codebase (many asked why - that sounds crazy!)
• Jim Buffenbarger from Boise State University gave a talk on amake which includes automatic dependency processing
• Akos Frohner and Boris Debic from Google gave a talk about the continuous release process at Google and which included some incredible numbers (100 million unit tests run a day!).  Boris also had some great things to say about the value of release engineering such as  "Release engineering should taught in business school, not in computer science classes.  It has real business value, and developers can learn it later. " "Startups should hire release enginners early, otherwise they will have to drain the swamp later".  "Companies that don't do release engineering well don't compete well in the marketplace."  Definitely words of wisdom.
• Dustin Mitchell from Mozilla gave a talk on the Buildbot continuous integration framework.  Later that afternoon, Moses Mendoza and Matthaus Owens from Puppet Labs presented how they build their packages for their consumers  and mentioned that they hoped to collaborate with Dustin and learn more about Buildbot.  Which was great - so glad that the workshop encouraged this collaboration.
• Ryan Hardt, a post-doc from University of Wisconsin (Milwaukee) gave a talk about Formiga, an Eclipse plug-in for refactoring Ant code which looks very promising. 
• Peter Rigby  from Concordia University described  DCVS systems facilitate different workflows between developers as opposed to more centralized systems.  Very interesting!

At the end of the day, I had the honour of moderating the discussion panel. The panel started out with professors Mike Godrey from Waterloo and Gunther Ruhe from the university of Calgary, as well as practitioners Jason Newblanc from Salesforce and Curt Patrick from Netflix.  Discussions included what topics needed more research (dependency analysis), to  a discussion of the skills do you look for when hiring a good release engineer (communication skills above all).  Given that one of the main goals of the workshop was to bring together researchers and release engineers it was very gratifying to hear from the researchers on the panel how happy they were that they had the opportunity to learn from the presentations earlier in the day.  After a while the panel shifted to include new members with new topics, and then we wrapped up to head to post-workshop dinners where the discussions continued.

Several attendees mentioned to me that they were happy to attend the workshop because it validated the release engineering work that they do is important.   I'm very fortunate to work at Mozilla on a large release engineering team where there are many people who love to talk about build optimization and automation.  But many release engineers work in organizations where they are the only ones who are interested in this subject.   To bring all these people together was a great experience!

We would like to thank everyone who attended the workshop.  Thank you to our session chairs who timed the talks and ensure that we stayed on schedule.   Thank you to Juliana Saraiva, our student volunteer, who helped with setup and throughout the day with audio and visual issues. 
A special thanks to all of our speakers.  Having been a speaker in the past, I understand all the work that goes into preparing talks on top of your regular day jobs and truly appreciate the effort that made this workshop successful.  At the end of the day, there was a resounding show of hands from attendees that they would like to attend another workshop.  In fact, several companies volunteered to  their space to host events which is very generous.   Please follow @relengcon on twitter for more information on upcoming events.  Many of the papers and presentations are available on the web site.  If your slides are missing from the web site, please email them to Bram.

Also thank you to my colleagues at Mozilla for arranging our releng work week to be same week so many of us could attend, despite it being a Canadian holiday weekend.  You're all amazing :-)

Read more...

Releng 2013 keynotes: John O'Duinn (Mozilla) and Roman Scheiter (LinkedIn)

There are two exciting keynotes planned for Releng 2013

John O'Duinn, Director of Release Engineering at Mozilla will kick off the workshop with his keynote Release Engineering as a Force Multiplier.  The build and release process used to be a pain point at Mozilla, but now makes the company and community more productive as a whole.  John will describe how the team added support for project branches to allow concurrent development, rethought continuous integration and increased capacity by moving to a hybrid-cloud build infrastructure. These changes improved several aspects of the business, including switching to a rapid release model and reducing turnaround time on a release from weeks to hours.  As a result,  Mozilla improved its abilities against much bigger and better funded competitors in the marketplace while also allowing them to enter new markets and help ensure its long-term success.


Roman Scheiter, Engineering Services Director at LinkedIn, will present the afternoon keynote entitled Against All Odds – Completely Overhauling Linkedin's Release Process.  This session will cover the evolution of LinkedIn’s release process from its earliest days to the point where the rapidly growing engineering team necessitated a radical shift. This shift, an executive sponsored effort to address technical debt and introduce new thinking to boost engineering efficiency allows six hundred developers to release thousands of changes per week without compromising quality.  As part of this undertaking, LinkedIn learned many best practices, developed tools and custom infrastructure, and lived through the internal cultural changes needed to make this independent release process work.   Roman will detail the evolution and results of this shift so you can learn directly from LinkedIn's pain.

 

In addition to these fantastic keynotes, we also have talks from release engineers and researchers from Netflix, Google,  Microsoft, Gnome, Red Hat, IBM, several universities and more!  We'll also have a panel at the end of the day to discuss the future of release engineering.

Check out the full program on the Releng 2013 site.  To register for the conference, which is managed as part of the larger ICSE conference, you can follow this link and choose the one-day-workshop.  See you in San Francisco on May 20!

Read more...

Thoughts on being a second generation geek

When I was in Toronto for our last work week,  I mentioned to one of my colleagues that I was a second generation geek.  He replied something to the effect of "There are families that have been fishing lobster for eight generations but not many second generation geeks."

I'm a second generation geek thanks to the influence of my Dad.  My sister also works in the software industry, as a technical writer.   My parents both had a tremendous influence on us.  If it wasn't for their support and encouragement, I doubt that we would be where we are today.

My Dad's first technology job was working at Burroughs  in Vancouver, BC as a field service technician and manager.  He would travel to customer sites to fix their computers, usually at banks,  insurance companies or government.  Back then computers were very large, not very common, and they had a longer lifecycle than we see today.  He had a briefcase full of tools to fix them,  given that most problems were mechanical or electrical in nature, as opposed to software.

About a decade later,  our family moved across the country and he started working as a technical services manager at a small software and services firm in Halifax, NS. This was the dawn of the PC era, and he brought home a succession of computers for him to learn about, as well as the rest of our family.  Dad would bring home stacks of computer magazines and manuals to read at nights on weekends, which I started to read as well.  We had many spirited discussions about technology, and what the future would hold.

In addition to having a wealth of technical knowledge, my Dad is also an accomplished woodworker  and builds beautiful furniture.  He bought my sister and I hammers so we could work along with him in the workshop.  There was also an abundance of Lego,  his old Meccano kit,  and stacks of science fiction books.  Christmas meant the inevitable gifts of conference swag in stockings, or a Linux book under the tree. 

Over the years, I have returned the favour and given my parents a lot of the swag I've received at work.  I'm sure they have the finest collection of Eclipse wear in Nova Scotia.  But this year begins a new tradition.

My Dad loves Firefox too

I've noticed that many of my (female) friends who work in software had a parent who worked in the industry and thus were a source of influence for their choice in career.  I learned from both my parents was to work very hard.  From my Dad, I learned that
1) Change in this industry is constant and it makes things interesting.
2) Continuous learning is essential.  Take courses, read voraciously,  break things and fix them.
3) STEM careers are for women.

Thanks Dad!

This past weekend, I asked him if there were ever any women at tech conferences when he attended them in the 1980s or 90s.  He replied that there was usually one or two, but that was all.

He then proceeded to tell me a story about a technical sales manager who he worked with for many years named Eli.  She had worked in an administrative capacity at Digital Equipment Corporation (DEC), but wanted to become a salesperson.  She was told that there was absolutely no way that she could be become a salesperson and represent DEC to customers because she was a woman.

In those days, computers were sold from OEMs (original equipment manufacturers like DEC) or VARs (valued added resellers).  The company where my Dad worked at was a VAR and had agreements with various OEMs, including DEC.  They would package hardware and software from several companies, along with installation and support services into sales proposals to meet customer requirements.  The funny thing was that VARs and OEMs would often compete for the same customers since they both sold the same branded hardware.  The VARs weren't limited to selling to a single vendor, but the OEMs had more flexibility in pricing in margins since they were the manufacturer.   Sales margins on hardware were quite high compared to what they are today, and thus a job as a salesperson could be quite financially lucrative if you were good.

Eli left DEC to work as a sales manager at the same office where my Dad worked.  She proceeded to outsell all the salesmen at her her previous employer who had refused to let her be a salesperson because of her gender, and won many sales awards.  So awesome!

I was always was a bit intimidated by Eli.  She was tough as nails, with a wicked sense of humour.  I can imagine that it wasn't always an easy path she had to walk, given that being a professional woman and working in technology was not common at the time, especially is socially conservative Nova Scotia.  But she did it, and proved that she could rise above her detractors.

A second thanks to women who have blazed trails in the past :-)  Who inspired you to choose your career in STEM?

Read more...

Who's speaking at Releng 2013?

Release engineering helps bring many of the products we use every day to market.  From entertainment, such as Netflix

http://www.flickr.com/photos/myetvmedia/8478881935/sizes/o/in/set-72157632783339196/

to search,

to operating systems and tooling,

to browsers and mobile operating systems that unleash the power of the web,

to software to configure the thousands of servers that allow us to build these products, and serve them to our customers.

Release engineers and researchers from Netflix, Puppet Labs, Mozilla. IBM, Google, Microsoft, Red Hat, Gnome and several universities will be giving talks at the Releng 2013 workshop on May 20, 2013 in San Francisco. I'm really excited by the different viewpoints that these speakers will bring to the conference and look forward to some interesting conversations.  I always enjoy learning about complex systems work and I'm sure that a behind-the-scenes look at how these companies build, test and deploy software will be fascinating.  A detailed list of talks is available on the conference web site.

In addition to the talks, we have two fantastic keynote speakers

John O'Duinn from Mozilla:  Release Engineering as a "force-multiplier
Alan Grosskurth a release engineering consultant (formally VMWare):
Release Engineering in the Cloud Era

To register for the conference, which is managed as part of the larger ICSE conference, you can follow this link and choose the one-day-workshop. We look forward to seeing you there!

Read more...

Disabling Bluetooth Assistant and app relaunch in 10.7 via Puppet

Intermittent test failures are very annoying. The create noise that cause people to question the test results - is it real data or just noise?

Image ©misspixels, http://www.flickr.com/photos/misspixels/8480711076/in/photostream/ under Creative Commons by-nc-sa 2.0

Recently, we were hit by intermittent mochitest failures on our Lion Talos boxes.  The issues causing this were two fold:
1) Despite Bluetooth being disabled, the Bluetooth setup assistant would randomly start and take focus away from running tests.
2) Our talos boxes reboot and repuppetize after a test job has completed.  However, on 10.7, the default is to relaunch apps that were running before reboot.  Thus, if an app crashed, it would be restarted upon reboot and again interfere with tests.

If this was just a machine on your desktop, you could disable the preferences in the the UI.  However, since we manage our infrastructure via Puppet, I needed a command line way to implement this preference change and update our 80+ Lion test machines automatically.

To disable setup assistant for Bluetooth mouse and keyboard

 defaults write /Library/Preferences/com.apple.Bluetooth BluetoothAutoSeekPointingDevice -bool false

 defaults write /Library/Preferences/com.apple.Bluetooth BluetoothAutoSeekKeyboard -bool false 

To disable apps from restarting upon reboot

 defaults write /Library/Preferences/com.apple.loginwindow LoginwindowLaunchesRelaunchApps -bool false

Note: the disabling the apps from restarting doesn't uncheck the preference in the UI, but it does work.

Having spent many years as a Linux sysadmin,  I find the way that Apple manages preferences within their operating systems non-intuitive and generally poorly documented.  In any case, it took me quite a bit of digging to find the correct defaults incantation, so I thought I'd blog about it in case others had the same problem :-)

References
Bug 843545 Intermittent OSX 10.7 mochitest failures that seem related to Bluetooth Setup Assistant crashing
Bug 743594 Stop 10.7 from restoring apps after a restart

Read more...

Reminder: Releng 2013 submission deadline is February 7

Just a friendly reminder that Feb 7 is the deadline to submit talk abstracts and papers to the Releng 2013 workshop.  It will be held on May 20 in San Francisco, in conjunction with the ICSE conference.

Are you involved in release engineering, release management or continuous deployment? Have you recently migrated a code repository to Git and implemented new branching strategies? Did your team implement amazing new automation to package and sign the binaries you deliver to customers?  Are you an academic looking for new problems to research and connect with practitioners in the field? Are you a maintainer for a continuous integration system or write a new build system? Did you manage to ship on time despite the fact that one file changed on a Friday afternoon and broke everything? If so, you have a story to share at this conference :-)

We have two fantastic keynote speakers lined up - John O'Duinn, director of release engineering at Mozilla, and Alan Grosskurth, who recently started working as a release engineering consultant after several years at VMWare.

If you have any questions regarding the submission process or the conference in general, please feel free to drop me a line (I'm kmoir and I work at mozilla dot com).  See you in San Francisco!

Read more...

Android Panda tests in production

Two weeks ago the Mozilla release engineering team had a work week in Toronto.  Toronto in January?  Balmy, according to one of our of colleagues from California.
 

Hal wears sandals in the snow.  Photo: John Hopkins

Over the past few months,  Ateam, IT and release engineering have been wrestling panda boards into submission and deploying them into production as part of our test infrastructure for Firefox for Android.  (There are pandas running B2G tests as well. )   Firefox is also a colloquial name for the red panda who apparently likes snow too.

Image ©maiac,  http://www.flickr.com/photos/maiac/5442715157/licensed under Creative Commons by-nc-sa 2.0

As Dustin described in his post on Mozpool, the panda boards are development boards that sit in specialized chassis that were developed for Mozilla.  During the work week, I finished putting the  last of the 400 panda boards in production running tests on the cedar, mozilla-central, try and mozilla-inbound branches.   From a releng perspective, this involved setting up 32 foopies and 5 new buildbot masters and updating our configs.   For those readers uninitiated with our release engineering configuration, we use buildbot as our continuous integration system to schedule builds and tests.  Foopies are are dedicated servers that handle the connections of the mobile devices to the buildbot masters.

Some of the issues that we encountered while we were rolling the pandas into production were quite difficult to overcome.  The Ateam, especially Joel spent many days debugging them, including this this particular nefarious one where the panda boards would spontaneously reboot while running tests

Bug 811444 - android panda boards magically reboot in the middle of the test

Joel's solution the solution was to slow down the running of the tests so the panda boards wouldn't reboot and allowed us to move forward.

Many thanks to Joel, Callek, Armen, Amy, Dustin and Jake for all their help putting these into production.

Next step: Manage the Android  Panda boards using Mozpool

Further reading
Dustin Mitchell's post on MozPool, including pictures of the awesome custom ruby red Mozilla chassis
Bug 802667 - configure new buildbot masters for use with android on pandas
Bug 803248 - buildbot config changes to support panda_android*
Bug 805658 - add all panda boards to slavealloc, disabled  
Bug 811723 - change android reftests to run with --ignore-window-size for panda only 
Bug 825984 - Turn on Android 4.0 test jobs on try and mozilla-inbound
Bug 829181 - put remaining pandas into production for Android 4.0 tests

Read more...