Android Panda tests in production

Two weeks ago the Mozilla release engineering team had a work week in Toronto.  Toronto in January?  Balmy, according to one of our of colleagues from California.
 

Hal wears sandals in the snow.  Photo: John Hopkins

Over the past few months,  Ateam, IT and release engineering have been wrestling panda boards into submission and deploying them into production as part of our test infrastructure for Firefox for Android.  (There are pandas running B2G tests as well. )   Firefox is also a colloquial name for the red panda who apparently likes snow too.

Image ©maiac,  http://www.flickr.com/photos/maiac/5442715157/licensed under Creative Commons by-nc-sa 2.0

As Dustin described in his post on Mozpool, the panda boards are development boards that sit in specialized chassis that were developed for Mozilla.  During the work week, I finished putting the  last of the 400 panda boards in production running tests on the cedar, mozilla-central, try and mozilla-inbound branches.   From a releng perspective, this involved setting up 32 foopies and 5 new buildbot masters and updating our configs.   For those readers uninitiated with our release engineering configuration, we use buildbot as our continuous integration system to schedule builds and tests.  Foopies are are dedicated servers that handle the connections of the mobile devices to the buildbot masters.

Some of the issues that we encountered while we were rolling the pandas into production were quite difficult to overcome.  The Ateam, especially Joel spent many days debugging them, including this this particular nefarious one where the panda boards would spontaneously reboot while running tests

Bug 811444 - android panda boards magically reboot in the middle of the test

Joel's solution the solution was to slow down the running of the tests so the panda boards wouldn't reboot and allowed us to move forward.

Many thanks to Joel, Callek, Armen, Amy, Dustin and Jake for all their help putting these into production.

Next step: Manage the Android  Panda boards using Mozpool

Further reading
Dustin Mitchell's post on MozPool, including pictures of the awesome custom ruby red Mozilla chassis
Bug 802667 - configure new buildbot masters for use with android on pandas
Bug 803248 - buildbot config changes to support panda_android*
Bug 805658 - add all panda boards to slavealloc, disabled  
Bug 811723 - change android reftests to run with --ignore-window-size for panda only 
Bug 825984 - Turn on Android 4.0 test jobs on try and mozilla-inbound
Bug 829181 - put remaining pandas into production for Android 4.0 tests

Read more...

Releng 2013 Call for Papers

There are many conferences that discuss various aspects of of release engineering such as PuppetConf, OSCON, LISA, EclipseCon, ApacheCon, Jenkins User conferences, devops days and Velocity.   But there isn't a conference dedicated specifically to release engineering... until now.

A solid build, test, packaging and deployment story is crucial to the success of a software project.  As John O'Duinn  says, "release engineers have a multiplier effect".  In other words, a release engineer can implement automation improvements that make every other developer more productive.  Traditionally, release engineering hasn't been a common area of academic research.  However,  this is starting to change.  Another challenge is there isn't a lot of communication between academic researchers and release engineers.  The aim of the Releng 2013 workshop on May 20, 2013 in San Francisco is to bring together those two communities together: people practicing release engineering and the academic researchers studying it.  It will be co-located with ICSE 2013, which is the largest academic software engineering conference. 

Image ©thomashawk,  http://www.flickr.com/photos/ekai/457004988/ licensed under Creative Commons by-nc-sa 2.0

Are you a release engineer who'd like to discuss the challenges you face and share your experiences with others? Or are you an academic looking to expand the audience for your research and discover new problems to analyze? If so, we encourage you to to submit a paper or a talk and attend the workshop.  Or if you just want to hear some great war stories, in both open source and commercial environments, this would be a fantastic place to learn.  More details are on the web site.  You can also follow us on twitter or Facebook.  We look forward to seeing you in San Francisco!

Notes:
Greg Wilson's article on the two solitudes (industry and academia) is an interesting read and underscores the importance for more interaction between these two communities

Read more...

Mozilla Release Engineering++

Note: I wrote this a month ago but didn't have time to finish this post until now. Bugzilla > blogging.

In early August, I attended my inaugural work week with Mozilla release engineering.  I met many of our team for the first time in person at the San Francisco office.

It was a fantastic week.  Running along the beautiful waterfront in the morning and talking about open source builds all day is my idea of a good time :-) On the flight home, I started thinking about why working as a release engineer at Mozilla makes me very happy.

1)  The people on the release engineering team are friendly, welcoming and genuinely helpful.  Not only that, but they have had interesting experiences outside of work.   From volunteering at Burning Man, to dowsing burning trees as a firefighter, we heard many interesting stories.

2)  We have a lot of build and test infrastructure, both virtual and not.   Due to the sheer amount of hardware, we have scaling issues which are interesting and complex to explore.  For developers, the available capacity means that they don't have to wait long for a build to start to test their changes.   As one of my coworkers stated, the scale of our continuous integration systems is quite unique in the open source world.

3)  We share the the load of interruption driven work as individuals to make the overall team more productive.  Every two months, you're on buildduty.  That week, you're responsible for answering developer questions and ensuring our infrastructure is operational at optimal capacity.  This allows other members of the team to concentrate on the projects they are working on without being constantly pinged in IRC.  We have beta builds every week and releases every six weeks.  These are also covered by team members on a rotating basis, called releaseduty.

4)  The other teams at Mozilla are also very helpful.  I'm really stunned by their generousity to set aside their work at a moment's notice and help you debug a problem so you can move forward.  One of the great things about working at Mozilla is that people are all working toward a common goal, to make the web a better place. And this means that they want to help everyone move forward, not just the team they work on.

5) Due to the inherent scaling issues and the fact that very few organizations run the volume of builds and tests we do, we write a lot of our own tools.  This is a great opportunity to diversity your skill set. Python, Puppet, Buildbot, AWS, MySQL, system administration and tooling for Windows, Linux, Mac and Android, wrangling Hg, Git and more.

6)  Management is committed to hiring great release engineers, no matter where they live.  This allows you to do the work you love, from the place you love to live.

7) Release engineers aren't considered second class citizens within the engineering group.   They are just another group of engineers who contribute, and are respected members of the community.

Release Engineering branded laptop bags? Yes

8)  We are data driven and constantly seek to improve our metrics.  What's the wait time to run a build on a certain platform?  What 's the utilization of our infrastructure?  What's the trend for the number of checkins per month? What platform could benefit from additional test slaves?  How can we optimize the build to make it faster?  How can we improve our automation story?   Continuous improvement is fantastic.

9) Inherent in the fact that work on open source projects, our culture is very open.  This means that we have very interesting speakers visit and discuss their build story and we can learn from them to address challenges in our own environment.

10) Working as a Mozilla release engineer is no walk in the park.  The learning curve is steep and it's very challenging work.

Walks in the park are nice, but I like the endorphins from running hard too :-)

Read more...

New platform tests: OS X Mountain Lion

Late Monday night, we moved a new testing platform into production.  We have 80+ shiny new Mountain Lion slaves running tests for desktop builds, with several reserved for staging tests.  There are a few issues with failing tests that are being tracked in bug 786084.

Image ©ekai,  http://www.flickr.com/photos/ekai/457004988/ licensed under Creative Commons by-nc-sa 2.0

This was one of the larger projects I've worked on since I started at Mozilla a few months ago.  Many of the existing test and builds slaves that we run are configured with an old installation of Puppet (0.24.8) that we call "pre-historic puppet".  We have a newer installation of Puppet running 2.7.19 nicknamed PuppetAgain that serves as the Puppet Master for these new Mountain Lion slaves.   The PuppetAgain installation used to just support CentOS slaves, so we added modules to support the Mac slaves and accommodate all the quirky Apple specific configuration needs.   
 
Given my experience as an Eclipse committer, I installed the Geppetto project as my Puppet manifest editor and it's a great tool.  Thanks to the folks at Cloudsmith for developing it, it has been very useful. 

In addition to becoming familiar with Puppet, I also had the opportunity to learn about the releng buildbot configs, adding new buildbot masters, updating graphing databases to store Talos results, running staging tests and all the steps to add a new platform. Since the process wasn't documented, I wrote up some notes to help others if they need to do the same.

Thanks to everyone on the release engineering and release engineering operations teams for all their help, especially answering my many questions and reviewing patches.  Its great to see something that you worked on in production, and finally generating green builds.

Read more...

Learning to speak like a native Mozillian

Aside from gaining experience with the technical requirements of a new job, there's also the challenge of learning the terms within the community as well as cultural norms.  There's time spent finding the right code repositories, documentation and test servers.  Who's the right person to cc on a bug or ping in IRC?  When learning a foreign language, they say that if you're still translating in your head before you speak,  you're still learning.  Here are some terms and conventions I learned during my first weeks at Mozilla. 

Release some code to a production branch or stream on the canonical repo
Mozilla:  to land
Eclipse:  to release or commit

People
Mozilla: Lots of people named John, Chris and Mike, many Canadians
Eclipse:  Lots of people named John, Chris and Mike, Canadians represent too

Image ©meddygarnet, http://www.flickr.com/photos/blakespot/5240888169/sizes/z/in/photostream/ licensed under Creative Commons by-nc-sa 2.0 

Where bugs rain down from
Mozilla: https://bugs.mozilla.org
Eclipse: https://bugs.eclipse.org

The source of all truth, which may contradict itself
Mozilla: http://wiki.mozilla.org
Eclipse:  http://wiki.eclipse.org

The version control system(s) to alternately love and shake your fist at
Mozilla: Hg with some Git, Subversion, CVS and Bazaar
Eclipse:  Git with a side order of CVS and Subversion.  (Side orders will be deprecated soon)

Image ©clsung, http://www.flickr.com/photos/clsung/310886130/sizes/z/in/photostream/ licensed under Creative Commons by-nc-sa 2.0

Colloquial expression for the associated open source foundation
Mozilla: MoFo
Eclipse: the foundation

Time for liquid refreshment
Mozilla: MFBT
Eclipse: Beer o'clock

Image ©Cambridge Brewing Company , http://www.flickr.com/photos/cambridgebrewingcompany/5619040409/sizes/z/in/photostream/ licensed under Creative Commons by-nc-sa 2.0

Communication channels in order of importance
Mozilla: IRC,  mailing lists, Bugzilla
Eclipse: Bugzilla, mailing lists, Twitter, IRC

Image ©blakespot, http://www.flickr.com/photos/blakespot/5240888169/sizes/z/in/photostream/ licensed under Creative Commons by-nc-sa 2.0

Code review
Mozilla: review flag in Bugzilla for all patches
Eclipse:  review flag in Bugzilla at the end of the release cycle, others use Gerrit all the time

Incredibly smart and friendly people with a passion for delivering fantastic open source software
 +1 Mozilla and Eclipse

What have I missed?

Disclaimer: This is just based on my own personal experience.  YMMV.

Read more...

Challenges in Release Engineering

I've been at Mozilla since the end of April.  The learning curve is steep but I'm having fun climbing.  My coworkers are very friendly and helpful while answering my barrage of questions with respect to how things work.   One of the things I've noticed is that there are many common challenges in release engineering, no matter what you're building. Here's my list so far:

1. Signing builds is like a falafel sandwich.  (Always includes some pita).

Image ©chotda, http://www.flickr.com/photos/santos/3531853459/sizes/z/in/photostream/ licensed under Creative Commons by-nc-sa 2.0

2.  Scaling your build to manage infrastructure utilization, tooling to manage that infrastructure and optimizing build parallelization is extremely challenging.  Developers will consume all available build infrastructure and then ask for more.  Scaling build infrastructure to accommodate future growth is an ongoing process. 

3. Proliferating numbers of platforms on which to build, test and run performance metrics add complexity.

 Image ©misterbisson, http://www.flickr.com/photos/maisonbisson/109211670/sizes/o/in/photostream/ licensed under Creative Commons by-nc-sa 2.0 

4.  Update game adventures.  When you have an open platform that included the installation of third-party components, it's inevitable that you will encounter unexpected update cases in the wild that weren't reflected in test cases.

5. Frequent releases generate faster user feedback on new features.  However, additional releases are expensive for the release engineering, quality assurance and release management teams.  Each additional release eats time, both people and machine.  Making the community aware that builds are not free is an ongoing communication exercise.

Image ©aarongeller, http://www.flickr.com/photos/aarongeller/360135019/  licensed under Creative Commons by-nc-sa 2.0

6.  You can have great documentation and process but the accumulated technical and tribal knowledge required to resolve a complex and broken build quickly is not earned by anything other than experience.

  Image ©Ian Muttoo, http://www.flickr.com/photos/imuttoo/2631466945/sizes/z/in/photostream/ licensed under Creative Commons by-nc-sa 2.0 

7.  If you can compile your code, this doesn't mean there won't be issues with packaging, signing and testing it.  Making a build available in a format for millions of users to consume > code that compiles.  Education is needed to make people aware of this distinction. 

What release engineering challenges do you face?

Read more...

Moving to Mozilla

Eclipse family: March 23 will be my last day at IBM. I'm happy to announce that I'll be joining the Mozilla Release Engineering team in April. 

Image ©flod, http://www.flickr.com/photos/flod/2221300134/sizes/l/in/photostream/ licensed under Creative Commons by-nc-sa 2.0

We all wear many hats during our lives.  I've always been fiercely proud of the privilege of calling myself an Eclipse committer.  We've accomplished amazing things together.  I look forward to seeing the continued growth and success of the Eclipse community. 

At Mozilla, I'll have the opportunity to work with a completely new software stack, and learn many new skills while working with very talented people. Best of all, my job will involve contributing to an open source community.  This is what I love to do.  The Mozilla goals of making the web better, and teaching a new generation to make content, instead of just consuming it, really resonate with me. Running builds on 1200+ machines raises some interesting scalability issues which I'm eager to learn more about.  It will be challenging work, and I'm honoured to have the opportunity to work at Mozilla.  I'll be working from my home in Ottawa. 

Where there are new challenges ahead for me, at the same time it has been a difficult decision to let my contribution to Eclipse wane.  I'd like to continue contribute to Eclipse in some way but on a part-time or occasional basis.

My IBM colleagues, my fellow committers and the Eclipse foundation staff:  you are all extraordinary.  It has been a privilege to work with you and learn so much.  To those in the larger Eclipse community, you inspire me.  To the people on the Eclipse SDK and Equinox teams, I have never had the opportunity to work with such a wonderful group of people.  Not only are you creative and technically brilliant, but you are all so dedicated to doing the right thing and getting things done.   You're all genuinely nice people, without ego, who are eager to share your knowledge with others.   It's not often that software project ships on time every year for ten years and continues to delight its user community.  It is a testament to the team behind it that this continues to be the case.  

In the process of working on Eclipse, I've met some fantastic people, many who have become great friends.  I'll miss the seeing your faces every day and discussing the finer points of Git migration strategies, but I'll still be on IRC, Twitter, Linkedln, G+ etc.  If you live in the Ottawa area, I'm always up for lunch :-) 

It has been a great run. Thank you all

Note #1:  I'll continue to write on this blog. The name still applies :-)
Note #2: I was going to call this post "Firefox-y Lady".  Then I read the lyrics to the Hendrix song "Foxy Lady" and decided maybe not.

Read more...

2011 by the numbers

2011 was an exciting year in the Eclipse community.  From my corner of the Eclipse universe, he's what it looked like:


One book chapter, many thanks

I contributed a chapter on Eclipse to the Architecture of Open Source Applications in 2010 and the book was published in May 2011.  Thanks to Amy Brown and Greg Wilson, for their long hours editing and providing feedback to the authors of this book.  It's a great read!  When Greg first approached me about writing this chapter, my immediate thought was "How hard could it be? I live and breathe Eclipse all day".  It was much more difficult that I imagined but in the process I learned a tremendous amount and am a better committer for the experience.  Many thanks to DJ Houghton and John Arthorne for reviewing my drafts and providing valuable feedback. A special thanks to Jeff McAffer who I interviewed about the decision to switch to OSGi in 3.0 and Steve Northover for his suggestions to make the SWT section into something more pixel perfect.  Merci Olivier Thomann for answering my many compiler questions,  and Boris Bokowski and Paul Webster for their thorough discussions with me regarding the modelled workbench and dependency injection in 4.x.  Also, thanks to Mike Wilson to allow me the flexibility in my job to spend some time at work working on this chapter.  I'm excited to see that Amy and Greg are now editing a second volume of this book.

Six milestones, many release candidates, two service releases, and one coordinated release, four streams, thousands of builds, millions of tests
No rest for the committers.

143 bug fixes
I closed about 143 bugs in the releng bucket in the past year.  That doesn't seem like much really.  I'd have liked to solve more.  The largest issues implemented from a releng perspective were shared licenses, code coverage, and the largest work item, the Git migration.

42 Git repos 
The Equinox and Eclipse projects migrated all their repos to Git.  We now have about 42 Git repos.  This involved a tremendous amount of work on the part of the Eclipse team as a whole.  There were many whiteboard drawings and detailed discussions about the migration process with John, Paul and a Mr. Gheorghe.  There was no Ringo.  Thank you Paul for all the huge amount of testing, script writing, and migration of all the ui and e4 repos. Thanks John for your work many sage suggestions on our Git migration, as well as your suggestion to implement the git flow method to simplify our development and build processes.  Thanks Andrew Niefer for migrating many of the Equinox and PDE repos, Bogdan Gheorghe for your work with SWT, and Oliver Thomann for testing JDT Core repos.  Thanks Tom Watson for your Git advice, having already climbed the Git learning curve while working on the OSGi Alliance repositories.  To Dani Megert and Markus Keller, your always fine attention to detail and pointing out areas that could be improved is appreciated. Paul is giving a talk about our migration at EclipseCon 2012 called Let's Git this Party Started.  I'm sure it will be insightful and entertaining.

One EclipseCon, two talks, one castle, many great people
I was privileged to attend EclipseCon Europe in Ludwidsberg this past November and present two talks.  I thoroughly enjoyed preparing these talks, and even more presenting them.  On the Wednesday morning, I talked about our Git Migration, and that evening I gave a talk with John Kellerman about history of Eclipse over the past 10 years.  After the second talk, a few people came up to me and said that the talk was so good that it should have been a keynote.  That was very fantastic to hear because we really put a huge amount of effort into that presentation.  I also had a lot of fun talking to people at our booth where we had posted many pictures of the Eclipse family from over the years. The Saturday after the conference Simon Kaegi, Eric Moffatt and I visited Heidelberg castle.  Canada scores very low on the castle index so this was a treat.   

You can't buy Eclipse magazines or giant pretzels at train stations in Canada either. I was impressed.

19 blog posts
I didn't have much time to write blogs posts this year.  The most popular one I wrote this year was about smashing open source stereotypes.

I'm never sure how popular a blog post will when I write them. It's always a surprise.  The comparison of Mozilla and Eclipse build infrastructure I wrote last year still holds the record for most popular (it ended up on reddit).


One marathon, many kilometers of training
How is running related to release engineering?  Running keeps me sane when release engineering gets crazy :-)  Preparing for the Ottawa marathon in May means that you have to start training at the end of January.  Running through snow, ice, wind and rain teaches you there isn't really anything you can't do when you are willing put in a lot of hard work to reach your goal.  And when you reach that goal, there's a lot of joy, because you know that you have conquered all the obstacles in your path and emerged victorious.

My sneakers after a 19K training run through deep slush

Open source is really a huge team effort and I had a lot of fun in the Eclipse community in 2011.

Who knows what 2012 will bring?

Read more...

We're the face of open source

I had some interesting discussions on Twitter this afternoon.  

Wayne replied:

Miles said:

One of the great things about the Eclipse community: that we cooperate on open source projects yet compete on commercial products.  This slide from the Eclipse 10 years talk  that John Kellerman and I  recently gave shows the diversity of the CDT project by company.

Here's another slide where we talked about the fact that there weren't originally enough non-IBM committers on the Eclipse project.  I called this "Too much blue in the Eclipse rainbow".

Image ©darrentunnicliff, http://www.flickr.com/photos/darrentunnicliff/4510834607/  licensed under Creative Commons by-nc-sa 2.0

I'd like see a more diverse community at Eclipse and in open source in general.  To spread the word that it's a rewarding career and we have a wonderful community.  Also, I'd like to find more people to fix bugs :-)

Occupy Open Source: We are the 1%

I don't know if the percentage of women at Eclipse is really 1% but it's pretty low.*

Ian later tweeted

My response was that it would be interesting to focus on the person, where they had come from and what they work and work the technology into the discussion.  Show a picture of the person, what their educational background is, how they got involved in open source, and what they work on.  I think computer science and open source have an image problem.  People think that we software isn't a social endeavour.  And yet it is.  Hello GitHub.  That the work we do doesn't change the world and make people's lives better.  No again.

One of the ways to combat stereotypes tell stories from the perspective of the person. How they came to work in open source. The interesting projects they work on.  Talks they presented at conferences.  What they do in their spare time outside work. Curtis writes code for PDE but he also likes to kayak.  Susan works on Orion but also runs an organic farm.  Andrew writes Linux tools but he also has interesting travel adventures.  Eric works on the next generation Eclipse UI and wins pool tournaments.  Tom works on that too, and he likes to ski and hike near his home in Innsbruck.  Ian lives in Victoria, works on p2 and plays hockey.  Introduce the person, then move on to talk about the technology they work on :-)

Yesterday, Syzmon asked me if me if could use our Eclipse 10 Years talk at a demo camp in Poland. I thought that was fantastic.  Our talk delivered in another country, in a different language.  Go Creative Commons.

Putting these too ideas together, I thought it would be interesting to have a common slide deck we as a community could use at schools or universities called "We're the face of open source".  I think it's important to showcase the different paths people take to get to their careers.  And kids need to to see something of themselves reflected in people who work in the industry.  It doesn't matter if you're a man or woman, your ethnicity,  where you live, if you're gay or straight, have five kids or three dogs. The important thing is that you have a story that you want to share to inspire a new generation to consider contributing to open source. 

Thoughts?

Notes
*1)This is not intended to be a statement for or against the Occupy movement.  I'm just trying to be funny. YMMV.
2) Standing out in the Crowd talk from OSCON 2009 has interesting numbers about open source diversity and the benefits it brings
3) I'm willing to help put the slide deck together in my spare time outside work.  We could use a Google Docs to allow multiple people to edit it. Maybe the slide deck could provide a list of Eclipse mentors that are willing to help out students fix their first bug, browse the source tree etc.  These are details.  Let me know if you are interested in contributing :-) 
4) This would make an interesting EclipseCon talk. Ten Eclipse committers/contributors you should know and why

Read more...

EclipseCon Europe presentations now available

My presentations from EclipseCon Europe are now available on slideshare.

The first one is the story of the Eclipse and Equinox team's migration from CVS to Git.

Migrating to Git: Rethinking the Commit

View more presentations from Kim Moir

There were a lot of great presentations on  Git migrations such as those by Steffen Pingel and Christian Campo.  One thing that I've learned is that the time to migrate is proportional to the size of your code base and history.  Someone asked me if we considered just starting in Git without our history. Well, no, but that would have solved a lot of problems. It was also interesting to talk to the EGit team, and meet some of the people working at GitHub. (Thanks for the octocat stickers Kevin!). In honour of Movember, this slide seems appropriate.

Image ©dealingwith, http://www.flickr.com/photos/dealingwith/4295488113/  licensed under Creative Commons by-nc-sa 2.0

The second talk I co-presented with John Kellerman. It's a look back at the last ten years of Eclipse history. I had several people come up to me after the talk and say they really enjoyed it. Thanks! It was fun to look back at Eclipse history and dig up funny pictures and bugs. I enjoyed presenting with John because he talked about the business side and I talked about the evolution of the Eclipse community from a down in the trenches commmiter perspective. A good balance. He has been involved in the Eclipse community since well before my time, lots of great stories!

Has it really been 10 years?

View more presentations from Kim Moir

You can also listen to the talk on FOSSLC website.  I love the fact that all the EclipseCon presentations were recorded. I plan to go back to and watch the sessions I missed!

A big thank you to the organizers of EclipseCon Europe for a fantastic conference. I've never attended one before, and was very impressed. Beautiful location, interesting talks, great running paths nearby and of course, the best people. It's great to be able to spend time with people who you work with but never get to see in person, like Ian Bull and Andrew Overholt. It was also great to meet new people at lunch and in the hallways.  I talked to some first time EclipseCon attendees who were very impressed with the caliber of talks at the conference so kudos to everyone who presented.

At the IBM booth, we pinned up many pictures of the Eclipse family over the past ten years. It was great to talk to people who visited the booth, especially those new the the Eclipse community.  I posted a link to them on my Google+ account. Looking back at them it's amazing to see so many smiling people.


It's hard to believe that it's been ten years. I had a number of people come up to me at the conference and say that they couldn't believe that I had been involved in the community for ten years. Well, I can't believe it either. It reminded me of the moment when I stood up on stage to receive my university diploma and I thought, how could these four years have gone so quickly?  I don't know but it's been a lot of fun.  By the way, here is a list of other committers who have been involved with Eclipse for ten years or more.

More importantly, there are now over a thousand Eclipse committers today.  I'm honoured to work with you all :-)

Presentations on slideshare:
Migrating to Git: Rethinking the Commit
Slideshare: Has it really been 10 years?
FOSSLC recording: Has it really been 10 years?
If you look at the speakers notes, you can see the text of the talk.  Most of the slides are just pictures in "Presentation Zen" style.

Read more...

In open source, all you have is social capital

Over a month ago, I watched this keynote by David Eaves that he gave at DjangoCon. Yes, I'm way behind on the list of things I'd like to blog about. I blame Bugzilla :-) Also, quite a few people at EclipseCon Europe came up to me and mentioned that they really enjoyed reading my blog. Thanks - I enjoy writing!



David Eaves is a negotiation consultant. He helps people on opposite sides of an issue come to an agreement, and has clients in open data, government, industry, and open source communities. He has done work at Mozilla to help manage contributor engagement and implement measures to keep contributors working in the community.

He starts off by telling the story, that as part of of his work with Mozilla, he thinks that he should submit a bug. He submits his first Thunderbird bug and announces on Twitter or Facebook that he's excited to submit his first bug. To which he gets the reply "I bet it's a duplicate".  According to this presentation, 50% of all bugs are duplicates.  It may be a duplicate, but this response isn't the best approach to encouraging future participation from a new contributor.

He then says that most open source communities don't have financial capital.  They don't have money flowing in to influence people.  "All you have is social capital".  Social capital consists of the people that contribute to your community and make it successful.  In theory, in a corporation, human resources tracks how to retain and manage its people.  In open source, we spend very little time managing our social capital or even better, tracking  it.

He then continues that one of the mantras of open source is that it's a meritocracy and your coding skills are the key to success within the community.  But if you look at people who have spent a lot of time in an open source community and are considered leaders,  many of them spend a lot more time working with people and  managing their community as opposed to coding.  To which he says "We pull people in based on their coding skills, and we promote people based on their negotiation skills".  Also, he remarks that nobody tells us that, and that we all stress that technical skills trumpet negotiation skills despite the evidence to the contrary.

He then gives examples of funny or misunderstood Mozilla bugs.  He states how it's harder to communicate with people when only via the written word.  I find that myself.  I've met quite a few people at conferences who I've found to be rude in Bugzilla to only find that in person they seem like a totally different person.  Reasonable.  Friendly.  Helpful.  Anyways, since Bugzilla is a written medium he suggests that the best way to interact on it is to ask questions.  He states that often people just have a solution in mind as a bug fix and aggressively push their solution where in fact they should ask the user what they want to do in the first place.  Also, you should paraphrase and repeat what the user said to gain understanding, acknowledge the problem and advocate for a solution. All great advice!

The next section of the talk discusses the architecture of a community.  Fork used to be a four letter word in open source  But with the advent of GitHub, it's not, but rather a way to empower the user.  It also absolves you from seeking anyone's permission before contributing.  He says we need to design our communities to "Architect for cooperation and away from collaboration"  It's great if a new contributor can start fixing a problem without the transaction costs associated with interacting with a committer.  We can spend our time on other tasks.  He also states that we need to empower the lowest people on the stack, such as those triaging bugs.  He also remarks that immediately marking a bug as invalid without acknowledgement of the effort required to report it doesn't build community loyalty.

GitHub Octocat

Image ©sunfox, http://www.flickr.com/photos/sunfox/4365495446/  licensed under Creative Commons by-nc-sa 2.0

The final section of the talk described applying metrics to the show what is going on in a project.  To measure contributor patches from non-paid staff is a way to measure social capital.  To determine why people have stopped contributing.  To measure wait time for before a patch is submitted.  He also states that it would be interesting to have a repository API and once you attach a patch to a bug, it would update the bug with the anticipated  wait time, to set expectations for the reporter.  He states that Wikipedia segments users based on metrics and they applies actions, such as suggesting mentors for new contributors.

I think having metrics for new Eclipse contributors would be very interesting.  I rarely have people contributing patches to my bucket other than from other Eclipse or Equinox project committers, but I'm sure the metrics would be very interesting for other components.   It would also be valuable to determine the reason that people stop contributing, and implement measures to encourage people to continue to their involvement.

He finishes by stating that there needs to be a social infrastructure for the community, not just code.  Also, in some cases, you many need to remove people from the community to reduce negative social capital.  A great talk, I highly recommend  it - extremely informative and funny. 


Notes:
 
David Eaves also blogs at http://www.eaves.ca on open data, open source and other interesting topics.
His keynote is here http://blip.tv/djangocon/keynote-david-eaves-5571777

Read more...

Excited about EclipseCon Europe

Ten years ago, we were burning the midnight oil getting Eclipse 1.0 ready to release.  I offer you proof

Me stuck behind server rack. Yes, I'm wearing COWS shirt.  Please don't judge me.

Today, we're working hard to polish our presentations as we celebrate ten years of  Eclipse at EclipseCon Europe.  I'm excited to have the opportunity to attend EclipseCon Europe, this will be my first time :-) 

I've been preparing two presentations over the past few weeks.  The first one is called Migrating to Git: Rethinking the Commit.  Here, I'll talk about the process we used to convert our large and historic CVS reposository to Git, the problems we encountered, how our development processes changed, and what advice we can offer other teams that are contemplating this migration.

Image ©venegas, http://www.flickr.com/photos/venegas/5549123/ licensed under Attribution-NonCommercial-ShareAlike 2.0 Generic (CC BY-NC-SA 2.0)

Image ©spool32, http://www.flickr.com/photos/spool32/5045502202/ licensed under Attribution-NonCommercial-ShareAlike 2.0 Generic (CC BY-NC-SA 2.0) 



The second talk is a light-hearted look back at the past ten years.  I received an email from EclipseCon Europe this morning that stated.  "After the Stammtisch, a couple of long-time Eclipse enthusiasts present Has it Really Been 10 Years?."  Long-time enthusiasts?  Okay, that made me feel old.  Anyways, in this talk John Kellerman and I look back at the the last ten years and discuss what what we expected when Eclipse was released as open source, the response we received, what mistakes were made, what surprised us.  Fair warning: I have uncovered some embarrassing pictures from our past Eclipse family.  I'll be using this occasion to showcase them.  If you have any interesting pictures to share, email me and I'd be happy to include them :-)   My understanding is that a Stammtisch involves beer so I think people will be in the right mood when they walk into the talk.

Last but not least, I invite you for to go for a run or walk each morning at 7am.  We'll meet in the lobby of the Nestor hotel.  Sign up for EclipseCon exercise here! 

See you soon!

Read more...

A history of lizard wrangling and other software stories

I've been thinking a lot lately about open source history.  I'm in the midst of preparing material for a presentation that John Kellerman and I will be giving at EclipseCon Europe about the history of Eclipse. Last week an interesting video crossed my twitter feed.  It was a talk by Mitchell Baker on the history of Mozilla.  Mitchell is the Chair of the Mozilla Foundation and Mozilla Corporation and has been involved in this community for many years.  Her title is Chief Lizard Wrangler.  That's the best job title I have ever heard.  Well, being called an astronaut would be fun too but you get the idea.  According to Wikipedia, she's a also skilled trapeze artist.  I like learning about people with interesting hobbies.




In the video, she talks about the history of  Mozilla, which started in 1998. She describes the tension between Netscape the corporation and Mozilla the community.  For instance, the belief in the Mozilla community that commit rights should be earned and voted on by your peers, not just assigned based on your employer.  It's a really interesting to learn about the conflict in those early years at Mozilla and how they worked so hard with very few resources to be where they are today.

One of my favourite lines from the talk is that "Technology alone does not change people's lives.  Our opportunity in the browser is to have a product that touches people." Also, she states that the problem with a lot of open source projects is that they are irrelevant to the marketplace and don't treat their users very well.  For instance, calling a user stupid isn't going to make their life better.  "You need to make a product that is elegant and beautiful and powerful under the covers but that people love."  Mozilla has a bit of a different outlook that the Eclipse community because their flagship product Firebox is used by everyday consumers.  Eclipse components are consumed by developers in the form of open source packages that the coordinated release provides, but at the same time many people consume components in commercial products that build upon our open source offerings.

Some other memorable lines "UI is always the most contentious issue".  Well, at Eclipse we never argue about UI.  Wait....hmm..maybe...this bug has a few contentious comments.....

This talk really resonated with me.  Eclipse and Mozilla are two well known open source communities with different histories but ultimately both are very successful.  I highly recommend taking the time to listen to it.

Happy Ada Lovelace Day from Ottawa!

Read more...

Hacker History

I recently read the book hackers: heroes of the computer revolution by Stephen Levy.  It looks back at the cast of characters that were involved in the early years of computing, starting in the late 1950s until the 1980s. The word hacker in this context doesn't mean someone that was breaking into machines for malicious intent, but rather someone who is breaking new ground in software and hardware.  The book is divided into three sections.  The first third of the book concentrates on people writing software at university labs such as MIT, the second third describes people making their own hardware in Silicon Valley computer clubs in the 1970s, and the final third looks at the beginnings of the PC game industry in the 1980s.

Interesting things I learned.

From xckd

• LISP was invented in 1958.  Old school.
• People used to single-handedly write a game and and then sell it to a game company such as Broderbund or Sierra On-line for a royalty on sales.  Like 30% of sales. A best selling game could make a young developer very very rich in a short period of time.
• Silicon Valley used to be called Silicon Gulch. Silicon Valley sounds classier.
• Bill Gates sold an early version of BASIC to a computer club which then copied it and distributed it to their members for free.  He then published a letter in a club newsletter that stated "As the majority of hobbyists may be aware, most of you steal your software."...."Who can afford to do professional work for nothing?"   A young Bill Gates writing about piracy in 1975. History repeats itself.
• Early hackers spent days, weeks even months on end in the lab without showering.  They also didn't date much.  Coincidence? No.
• The process to bring the Apple II to market and the interactions between Steve Jobs and Steve Wozniak were pretty interesting.

The book also talks about  the "hacker ethic", which underpins many things that we see in current open source communities today.  From the book

"Information should be free.
Mistrust authority--promote decentralization.
Hackers should be judged by their hacking, not bogus criteria such as degrees, age, race or position.
You can create art and beauty on a computer.
Computers can change your life for the better".


This book looks at the technical advancements that these hackers implemented, but is also interesting to see the culture as these nascent communities evolved. Also, it also describes the first computer conferences and magazines. It's also interesting to note the incredible passion and effort that these hackers dedicated to their  professional endeavours often had negative repercussions in their personal lives.

Anyways, if you are interested in reading about early computing history, this is a great book.   The first computer that my Dad (yes it runs in the family) brought home in the early 1980s was an Osborne , so I felt rather nostalgic reading about the this time period.

Source wikipedia

Read more...

Busting Build Myths

Who doesn't love Mythbusters? It must be great to wake up every morning and go to work in a fantastically equipped lab, build things and maybe set off a few explosions.  A dream job.

Unfortunately, my lab is filled with aging build machines, not cool toys.  When a build explodes I have to pick up the pieces and make it work again.  I've never been invited to the White House to talk science while sporting a snappy beret either.

Source Wikipedia: Washington Post blog, credit given to The White House/Chuck Kennedy, The White House/Chuck Kennedy

There are a few myths surrounding builds I'd like to bust.  I'll use the only tool I have available to dispel them, my keyboard:


1) My six bundle build is easy.  Thus your build is easy too.  There is a big difference between building a few bundles and building products, p2 repositories, API tests, tens of thousands of JUnit and performance tests for many platforms. Is running an Apache server in your basement the same as running Slashdot?  No. Builds are the same. Scalability issues and complexity.

2) Building with X will solve all your problems. Changing to a different build technology to compile and package your bundles can solve a lot of problems if it has been tested fully and works for all your requirements.  But there's a cost to migration.  To working out all the bugs and corner cases.  Is it worth the transition?  Only time will tell.  The more complex the build, the higher the cost of migration.  Yes, it would be wonderful if we could all run our builds the same way at Eclipse to help our consumers.  But this requires a considerable time investment to ensure that everything works.  As well, it means forgoing work in other areas to facilitate this transition. It's a tradeoff.

3) Once you get your build working, you never have to change it.  As my previous post described, build scripts evolve in concert with the code base. Constantly changing.  More code, more tests, more build gymnastics.  Building software is like owning a house.  You can try to to forgo all maintenance to save money, but your risk long term damage from broken pipes, a leaky roof or mice as the building deteriorates.

Homeowner forgot to visit Home Depot

Image ©graywolfx47, http://www.flickr.com/photos/graywolfx47/4308623603/sizes/z/in/photostream/licensed under Attribution-ShareAlike 2.0 Generic (CC BY-SA 2.0)  

4) Nobody in needs to understand how the build works, they just need to push a button.  That's great. Until the day before a release when your build fails with a cryptic message about unresolved dependencies.  And you have no idea how to fix it.  And neither does anyone else on the team.

5)  Release engineers are unskilled monkeys.  On the weekend, I attended the Software Developer's Summit here in Ottawa.  Interesting conference and it was great to meet some new  people. In the afternoon,  there was a meeting to discuss the long term build support with Eclipse foundation staff and other interested parties.  The term "monkey" was used by several participants as a colloquial expression for a release engineer.  I don't think it meant to be a malicious expression but at the same time, I found it interesting.  Many developers have limited knowledge of how builds work, to them it's a big black box.  Monkeys.  Big black box. Hmmmm. This sounds familiar.

Let's pretend these apes are monkeys.

Source: Wikipedia. Interpretations of 2001: A Space Odyssey

Since we release engineers understand how the black box works, maybe we're not the monkeys :-)

What other build myths are out there?

Read more...

Research, Release Engineering and ROI

I recently read some academic papers that quantify some release engineering practices in open source communities.  Very interesting.

The first one, "An Empirical Study of Build Maintenance Effort" (PDF), looks at how the time spent maintaining the build impacts the developers in a project.  It examines build coupling which refers to the how often changes in source code require changes in build code, as well as build ownership, which is the proportion of developers on the team who are responsible for maintaining the build.  The projects studied were ArgoUML, Hibernate, Eclipse (SDK), Jazz, GCC, Git, Linux, Mozilla, PLlot and PostgresSQL.

Here are some snippets I found interesting in this paper.  Note that when they refer to "Eclipse-core" they mean the Eclipse project's build. In this paragraph, they are talking about build coupling and how to reduce it

"In Eclipse-core, the coupling is reduced to 16% and in Jazz the observed coupling is a mere 4%.  Eclipse-core and Jazz both leverage the automated Eclipse Plugin Development Environment (PDE) build technology. ..... Since the developer must only maintain the high-level build.properties file (via the IDE), the daily build maintenance is reduced".

Thank you PDE family.  Dynamically generating Ant scripts at build time to compile and package our bundles makes us look good when compared with other open source projects.

"Other researches have found that the Linux build engineers have spent a considerable amount of time to make their build system as simple as possible for developers, as the expense of a very complex and hard to maintain core build system machinery. "

People always assume that writing software is hard, but for some reason building software should be easy. Build systems are complex beasts and obfuscating the complexity for the user is just as difficult as it might be when writing a full scale application. Writing good software is difficult, and so is constructing an elegant and effective build system.  It's just a different skill set.

 "Up to 79% of source code developers and 89% of test code developers are significantly impacted by build maintenance, yet investment in build experts can reduce the proportion of impacted developers by 22% of source code developers and 24% of test code developers."

So it looks like if you hire a release engineer and the productivity of your developers will increase.  You would buy a new machine to make the build faster, why not hire a fantastic release engineer and make the build better? The numbers indicate an great return on investment. 

Building complexity

Image ©algonquin_college, http://www.flickr.com/photos/algonquin_college/4971004199/in/photostream/ licensed under Attribution-ShareAlike 2.0 Generic (CC BY-SA 2.0)  


The second paper of is interest is entitled "The Evolution of Java Build Systems" (PDF). 

Again it looks at open source build systems that use either Ant (ArgoUML, Tomcat, JBoss, Eclipse-core) or Maven (Hibernate and Geronimo). The authors find that as the number of source lines of  code being built (SLOC)  is strongly correlated with the number of build lines of code (BLOC).

"Similar to Lehman's first law of software evolution, build system specifications tend to grow over time unless explicit effort is put into refactoring them."

and

"The Halstead complexity of a build system is highly correlated with the build system's size (BLOC), indicating the BLOC is a good approximation of build system complexity".

Lots of build code means increasing complexity.  If you are only building a few bundles, your build is easier to understand.  Makes sense. 

From their analysis, they concluded found that both Ant and Maven based builds evolved in similar fashion.

I like this sentence

"Despite the crucial role of build systems and their non-trivial maintenance effort, software engineering research rarely focuses in them".

I'd like to thank the authors for conducting this research and look forward to reading more in the future.  Build systems are complex systems and I welcome the efforts of these researchers to quantify way they can be improved. And it's extra special when they look at the projects that we work on every day!

References:
It will never work in theory
Queens University Software Analysis and Intelligence Lab

Read more...