We're the face of open source

I had some interesting discussions on Twitter this afternoon.  

Wayne replied:

Miles said:

One of the great things about the Eclipse community: that we cooperate on open source projects yet compete on commercial products.  This slide from the Eclipse 10 years talk  that John Kellerman and I  recently gave shows the diversity of the CDT project by company.

Here's another slide where we talked about the fact that there weren't originally enough non-IBM committers on the Eclipse project.  I called this "Too much blue in the Eclipse rainbow".

Image ©darrentunnicliff, http://www.flickr.com/photos/darrentunnicliff/4510834607/  licensed under Creative Commons by-nc-sa 2.0

I'd like see a more diverse community at Eclipse and in open source in general.  To spread the word that it's a rewarding career and we have a wonderful community.  Also, I'd like to find more people to fix bugs :-)

Occupy Open Source: We are the 1%

I don't know if the percentage of women at Eclipse is really 1% but it's pretty low.*

Ian later tweeted

My response was that it would be interesting to focus on the person, where they had come from and what they work and work the technology into the discussion.  Show a picture of the person, what their educational background is, how they got involved in open source, and what they work on.  I think computer science and open source have an image problem.  People think that we software isn't a social endeavour.  And yet it is.  Hello GitHub.  That the work we do doesn't change the world and make people's lives better.  No again.

One of the ways to combat stereotypes tell stories from the perspective of the person. How they came to work in open source. The interesting projects they work on.  Talks they presented at conferences.  What they do in their spare time outside work. Curtis writes code for PDE but he also likes to kayak.  Susan works on Orion but also runs an organic farm.  Andrew writes Linux tools but he also has interesting travel adventures.  Eric works on the next generation Eclipse UI and wins pool tournaments.  Tom works on that too, and he likes to ski and hike near his home in Innsbruck.  Ian lives in Victoria, works on p2 and plays hockey.  Introduce the person, then move on to talk about the technology they work on :-)

Yesterday, Syzmon asked me if me if could use our Eclipse 10 Years talk at a demo camp in Poland. I thought that was fantastic.  Our talk delivered in another country, in a different language.  Go Creative Commons.

Putting these too ideas together, I thought it would be interesting to have a common slide deck we as a community could use at schools or universities called "We're the face of open source".  I think it's important to showcase the different paths people take to get to their careers.  And kids need to to see something of themselves reflected in people who work in the industry.  It doesn't matter if you're a man or woman, your ethnicity,  where you live, if you're gay or straight, have five kids or three dogs. The important thing is that you have a story that you want to share to inspire a new generation to consider contributing to open source. 

Thoughts?

Notes
*1)This is not intended to be a statement for or against the Occupy movement.  I'm just trying to be funny. YMMV.
2) Standing out in the Crowd talk from OSCON 2009 has interesting numbers about open source diversity and the benefits it brings
3) I'm willing to help put the slide deck together in my spare time outside work.  We could use a Google Docs to allow multiple people to edit it. Maybe the slide deck could provide a list of Eclipse mentors that are willing to help out students fix their first bug, browse the source tree etc.  These are details.  Let me know if you are interested in contributing :-) 
4) This would make an interesting EclipseCon talk. Ten Eclipse committers/contributors you should know and why

Read more...

EclipseCon Europe presentations now available

My presentations from EclipseCon Europe are now available on slideshare.

The first one is the story of the Eclipse and Equinox team's migration from CVS to Git.

Migrating to Git: Rethinking the Commit

View more presentations from Kim Moir

There were a lot of great presentations on  Git migrations such as those by Steffen Pingel and Christian Campo.  One thing that I've learned is that the time to migrate is proportional to the size of your code base and history.  Someone asked me if we considered just starting in Git without our history. Well, no, but that would have solved a lot of problems. It was also interesting to talk to the EGit team, and meet some of the people working at GitHub. (Thanks for the octocat stickers Kevin!). In honour of Movember, this slide seems appropriate.

Image ©dealingwith, http://www.flickr.com/photos/dealingwith/4295488113/  licensed under Creative Commons by-nc-sa 2.0

The second talk I co-presented with John Kellerman. It's a look back at the last ten years of Eclipse history. I had several people come up to me after the talk and say they really enjoyed it. Thanks! It was fun to look back at Eclipse history and dig up funny pictures and bugs. I enjoyed presenting with John because he talked about the business side and I talked about the evolution of the Eclipse community from a down in the trenches commmiter perspective. A good balance. He has been involved in the Eclipse community since well before my time, lots of great stories!

Has it really been 10 years?

View more presentations from Kim Moir

You can also listen to the talk on FOSSLC website.  I love the fact that all the EclipseCon presentations were recorded. I plan to go back to and watch the sessions I missed!

A big thank you to the organizers of EclipseCon Europe for a fantastic conference. I've never attended one before, and was very impressed. Beautiful location, interesting talks, great running paths nearby and of course, the best people. It's great to be able to spend time with people who you work with but never get to see in person, like Ian Bull and Andrew Overholt. It was also great to meet new people at lunch and in the hallways.  I talked to some first time EclipseCon attendees who were very impressed with the caliber of talks at the conference so kudos to everyone who presented.

At the IBM booth, we pinned up many pictures of the Eclipse family over the past ten years. It was great to talk to people who visited the booth, especially those new the the Eclipse community.  I posted a link to them on my Google+ account. Looking back at them it's amazing to see so many smiling people.


It's hard to believe that it's been ten years. I had a number of people come up to me at the conference and say that they couldn't believe that I had been involved in the community for ten years. Well, I can't believe it either. It reminded me of the moment when I stood up on stage to receive my university diploma and I thought, how could these four years have gone so quickly?  I don't know but it's been a lot of fun.  By the way, here is a list of other committers who have been involved with Eclipse for ten years or more.

More importantly, there are now over a thousand Eclipse committers today.  I'm honoured to work with you all :-)

Presentations on slideshare:
Migrating to Git: Rethinking the Commit
Slideshare: Has it really been 10 years?
FOSSLC recording: Has it really been 10 years?
If you look at the speakers notes, you can see the text of the talk.  Most of the slides are just pictures in "Presentation Zen" style.

Read more...

In open source, all you have is social capital

Over a month ago, I watched this keynote by David Eaves that he gave at DjangoCon. Yes, I'm way behind on the list of things I'd like to blog about. I blame Bugzilla :-) Also, quite a few people at EclipseCon Europe came up to me and mentioned that they really enjoyed reading my blog. Thanks - I enjoy writing!



David Eaves is a negotiation consultant. He helps people on opposite sides of an issue come to an agreement, and has clients in open data, government, industry, and open source communities. He has done work at Mozilla to help manage contributor engagement and implement measures to keep contributors working in the community.

He starts off by telling the story, that as part of of his work with Mozilla, he thinks that he should submit a bug. He submits his first Thunderbird bug and announces on Twitter or Facebook that he's excited to submit his first bug. To which he gets the reply "I bet it's a duplicate".  According to this presentation, 50% of all bugs are duplicates.  It may be a duplicate, but this response isn't the best approach to encouraging future participation from a new contributor.

He then says that most open source communities don't have financial capital.  They don't have money flowing in to influence people.  "All you have is social capital".  Social capital consists of the people that contribute to your community and make it successful.  In theory, in a corporation, human resources tracks how to retain and manage its people.  In open source, we spend very little time managing our social capital or even better, tracking  it.

He then continues that one of the mantras of open source is that it's a meritocracy and your coding skills are the key to success within the community.  But if you look at people who have spent a lot of time in an open source community and are considered leaders,  many of them spend a lot more time working with people and  managing their community as opposed to coding.  To which he says "We pull people in based on their coding skills, and we promote people based on their negotiation skills".  Also, he remarks that nobody tells us that, and that we all stress that technical skills trumpet negotiation skills despite the evidence to the contrary.

He then gives examples of funny or misunderstood Mozilla bugs.  He states how it's harder to communicate with people when only via the written word.  I find that myself.  I've met quite a few people at conferences who I've found to be rude in Bugzilla to only find that in person they seem like a totally different person.  Reasonable.  Friendly.  Helpful.  Anyways, since Bugzilla is a written medium he suggests that the best way to interact on it is to ask questions.  He states that often people just have a solution in mind as a bug fix and aggressively push their solution where in fact they should ask the user what they want to do in the first place.  Also, you should paraphrase and repeat what the user said to gain understanding, acknowledge the problem and advocate for a solution. All great advice!

The next section of the talk discusses the architecture of a community.  Fork used to be a four letter word in open source  But with the advent of GitHub, it's not, but rather a way to empower the user.  It also absolves you from seeking anyone's permission before contributing.  He says we need to design our communities to "Architect for cooperation and away from collaboration"  It's great if a new contributor can start fixing a problem without the transaction costs associated with interacting with a committer.  We can spend our time on other tasks.  He also states that we need to empower the lowest people on the stack, such as those triaging bugs.  He also remarks that immediately marking a bug as invalid without acknowledgement of the effort required to report it doesn't build community loyalty.

GitHub Octocat

Image ©sunfox, http://www.flickr.com/photos/sunfox/4365495446/  licensed under Creative Commons by-nc-sa 2.0

The final section of the talk described applying metrics to the show what is going on in a project.  To measure contributor patches from non-paid staff is a way to measure social capital.  To determine why people have stopped contributing.  To measure wait time for before a patch is submitted.  He also states that it would be interesting to have a repository API and once you attach a patch to a bug, it would update the bug with the anticipated  wait time, to set expectations for the reporter.  He states that Wikipedia segments users based on metrics and they applies actions, such as suggesting mentors for new contributors.

I think having metrics for new Eclipse contributors would be very interesting.  I rarely have people contributing patches to my bucket other than from other Eclipse or Equinox project committers, but I'm sure the metrics would be very interesting for other components.   It would also be valuable to determine the reason that people stop contributing, and implement measures to encourage people to continue to their involvement.

He finishes by stating that there needs to be a social infrastructure for the community, not just code.  Also, in some cases, you many need to remove people from the community to reduce negative social capital.  A great talk, I highly recommend  it - extremely informative and funny. 


Notes:
 
David Eaves also blogs at http://www.eaves.ca on open data, open source and other interesting topics.
His keynote is here http://blip.tv/djangocon/keynote-david-eaves-5571777

Read more...

Excited about EclipseCon Europe

Ten years ago, we were burning the midnight oil getting Eclipse 1.0 ready to release.  I offer you proof

Me stuck behind server rack. Yes, I'm wearing COWS shirt.  Please don't judge me.

Today, we're working hard to polish our presentations as we celebrate ten years of  Eclipse at EclipseCon Europe.  I'm excited to have the opportunity to attend EclipseCon Europe, this will be my first time :-) 

I've been preparing two presentations over the past few weeks.  The first one is called Migrating to Git: Rethinking the Commit.  Here, I'll talk about the process we used to convert our large and historic CVS reposository to Git, the problems we encountered, how our development processes changed, and what advice we can offer other teams that are contemplating this migration.

Image ©venegas, http://www.flickr.com/photos/venegas/5549123/ licensed under Attribution-NonCommercial-ShareAlike 2.0 Generic (CC BY-NC-SA 2.0)

Image ©spool32, http://www.flickr.com/photos/spool32/5045502202/ licensed under Attribution-NonCommercial-ShareAlike 2.0 Generic (CC BY-NC-SA 2.0) 



The second talk is a light-hearted look back at the past ten years.  I received an email from EclipseCon Europe this morning that stated.  "After the Stammtisch, a couple of long-time Eclipse enthusiasts present Has it Really Been 10 Years?."  Long-time enthusiasts?  Okay, that made me feel old.  Anyways, in this talk John Kellerman and I look back at the the last ten years and discuss what what we expected when Eclipse was released as open source, the response we received, what mistakes were made, what surprised us.  Fair warning: I have uncovered some embarrassing pictures from our past Eclipse family.  I'll be using this occasion to showcase them.  If you have any interesting pictures to share, email me and I'd be happy to include them :-)   My understanding is that a Stammtisch involves beer so I think people will be in the right mood when they walk into the talk.

Last but not least, I invite you for to go for a run or walk each morning at 7am.  We'll meet in the lobby of the Nestor hotel.  Sign up for EclipseCon exercise here! 

See you soon!

Read more...

A history of lizard wrangling and other software stories

I've been thinking a lot lately about open source history.  I'm in the midst of preparing material for a presentation that John Kellerman and I will be giving at EclipseCon Europe about the history of Eclipse. Last week an interesting video crossed my twitter feed.  It was a talk by Mitchell Baker on the history of Mozilla.  Mitchell is the Chair of the Mozilla Foundation and Mozilla Corporation and has been involved in this community for many years.  Her title is Chief Lizard Wrangler.  That's the best job title I have ever heard.  Well, being called an astronaut would be fun too but you get the idea.  According to Wikipedia, she's a also skilled trapeze artist.  I like learning about people with interesting hobbies.




In the video, she talks about the history of  Mozilla, which started in 1998. She describes the tension between Netscape the corporation and Mozilla the community.  For instance, the belief in the Mozilla community that commit rights should be earned and voted on by your peers, not just assigned based on your employer.  It's a really interesting to learn about the conflict in those early years at Mozilla and how they worked so hard with very few resources to be where they are today.

One of my favourite lines from the talk is that "Technology alone does not change people's lives.  Our opportunity in the browser is to have a product that touches people." Also, she states that the problem with a lot of open source projects is that they are irrelevant to the marketplace and don't treat their users very well.  For instance, calling a user stupid isn't going to make their life better.  "You need to make a product that is elegant and beautiful and powerful under the covers but that people love."  Mozilla has a bit of a different outlook that the Eclipse community because their flagship product Firebox is used by everyday consumers.  Eclipse components are consumed by developers in the form of open source packages that the coordinated release provides, but at the same time many people consume components in commercial products that build upon our open source offerings.

Some other memorable lines "UI is always the most contentious issue".  Well, at Eclipse we never argue about UI.  Wait....hmm..maybe...this bug has a few contentious comments.....

This talk really resonated with me.  Eclipse and Mozilla are two well known open source communities with different histories but ultimately both are very successful.  I highly recommend taking the time to listen to it.

Happy Ada Lovelace Day from Ottawa!

Read more...

Hacker History

I recently read the book hackers: heroes of the computer revolution by Stephen Levy.  It looks back at the cast of characters that were involved in the early years of computing, starting in the late 1950s until the 1980s. The word hacker in this context doesn't mean someone that was breaking into machines for malicious intent, but rather someone who is breaking new ground in software and hardware.  The book is divided into three sections.  The first third of the book concentrates on people writing software at university labs such as MIT, the second third describes people making their own hardware in Silicon Valley computer clubs in the 1970s, and the final third looks at the beginnings of the PC game industry in the 1980s.

Interesting things I learned.

From xckd

• LISP was invented in 1958.  Old school.
• People used to single-handedly write a game and and then sell it to a game company such as Broderbund or Sierra On-line for a royalty on sales.  Like 30% of sales. A best selling game could make a young developer very very rich in a short period of time.
• Silicon Valley used to be called Silicon Gulch. Silicon Valley sounds classier.
• Bill Gates sold an early version of BASIC to a computer club which then copied it and distributed it to their members for free.  He then published a letter in a club newsletter that stated "As the majority of hobbyists may be aware, most of you steal your software."...."Who can afford to do professional work for nothing?"   A young Bill Gates writing about piracy in 1975. History repeats itself.
• Early hackers spent days, weeks even months on end in the lab without showering.  They also didn't date much.  Coincidence? No.
• The process to bring the Apple II to market and the interactions between Steve Jobs and Steve Wozniak were pretty interesting.

The book also talks about  the "hacker ethic", which underpins many things that we see in current open source communities today.  From the book

"Information should be free.
Mistrust authority--promote decentralization.
Hackers should be judged by their hacking, not bogus criteria such as degrees, age, race or position.
You can create art and beauty on a computer.
Computers can change your life for the better".


This book looks at the technical advancements that these hackers implemented, but is also interesting to see the culture as these nascent communities evolved. Also, it also describes the first computer conferences and magazines. It's also interesting to note the incredible passion and effort that these hackers dedicated to their  professional endeavours often had negative repercussions in their personal lives.

Anyways, if you are interested in reading about early computing history, this is a great book.   The first computer that my Dad (yes it runs in the family) brought home in the early 1980s was an Osborne , so I felt rather nostalgic reading about the this time period.

Source wikipedia

Read more...

Busting Build Myths

Who doesn't love Mythbusters? It must be great to wake up every morning and go to work in a fantastically equipped lab, build things and maybe set off a few explosions.  A dream job.

Unfortunately, my lab is filled with aging build machines, not cool toys.  When a build explodes I have to pick up the pieces and make it work again.  I've never been invited to the White House to talk science while sporting a snappy beret either.

Source Wikipedia: Washington Post blog, credit given to The White House/Chuck Kennedy, The White House/Chuck Kennedy

There are a few myths surrounding builds I'd like to bust.  I'll use the only tool I have available to dispel them, my keyboard:


1) My six bundle build is easy.  Thus your build is easy too.  There is a big difference between building a few bundles and building products, p2 repositories, API tests, tens of thousands of JUnit and performance tests for many platforms. Is running an Apache server in your basement the same as running Slashdot?  No. Builds are the same. Scalability issues and complexity.

2) Building with X will solve all your problems. Changing to a different build technology to compile and package your bundles can solve a lot of problems if it has been tested fully and works for all your requirements.  But there's a cost to migration.  To working out all the bugs and corner cases.  Is it worth the transition?  Only time will tell.  The more complex the build, the higher the cost of migration.  Yes, it would be wonderful if we could all run our builds the same way at Eclipse to help our consumers.  But this requires a considerable time investment to ensure that everything works.  As well, it means forgoing work in other areas to facilitate this transition. It's a tradeoff.

3) Once you get your build working, you never have to change it.  As my previous post described, build scripts evolve in concert with the code base. Constantly changing.  More code, more tests, more build gymnastics.  Building software is like owning a house.  You can try to to forgo all maintenance to save money, but your risk long term damage from broken pipes, a leaky roof or mice as the building deteriorates.

Homeowner forgot to visit Home Depot

Image ©graywolfx47, http://www.flickr.com/photos/graywolfx47/4308623603/sizes/z/in/photostream/licensed under Attribution-ShareAlike 2.0 Generic (CC BY-SA 2.0)  

4) Nobody in needs to understand how the build works, they just need to push a button.  That's great. Until the day before a release when your build fails with a cryptic message about unresolved dependencies.  And you have no idea how to fix it.  And neither does anyone else on the team.

5)  Release engineers are unskilled monkeys.  On the weekend, I attended the Software Developer's Summit here in Ottawa.  Interesting conference and it was great to meet some new  people. In the afternoon,  there was a meeting to discuss the long term build support with Eclipse foundation staff and other interested parties.  The term "monkey" was used by several participants as a colloquial expression for a release engineer.  I don't think it meant to be a malicious expression but at the same time, I found it interesting.  Many developers have limited knowledge of how builds work, to them it's a big black box.  Monkeys.  Big black box. Hmmmm. This sounds familiar.

Let's pretend these apes are monkeys.

Source: Wikipedia. Interpretations of 2001: A Space Odyssey

Since we release engineers understand how the black box works, maybe we're not the monkeys :-)

What other build myths are out there?

Read more...

Research, Release Engineering and ROI

I recently read some academic papers that quantify some release engineering practices in open source communities.  Very interesting.

The first one, "An Empirical Study of Build Maintenance Effort" (PDF), looks at how the time spent maintaining the build impacts the developers in a project.  It examines build coupling which refers to the how often changes in source code require changes in build code, as well as build ownership, which is the proportion of developers on the team who are responsible for maintaining the build.  The projects studied were ArgoUML, Hibernate, Eclipse (SDK), Jazz, GCC, Git, Linux, Mozilla, PLlot and PostgresSQL.

Here are some snippets I found interesting in this paper.  Note that when they refer to "Eclipse-core" they mean the Eclipse project's build. In this paragraph, they are talking about build coupling and how to reduce it

"In Eclipse-core, the coupling is reduced to 16% and in Jazz the observed coupling is a mere 4%.  Eclipse-core and Jazz both leverage the automated Eclipse Plugin Development Environment (PDE) build technology. ..... Since the developer must only maintain the high-level build.properties file (via the IDE), the daily build maintenance is reduced".

Thank you PDE family.  Dynamically generating Ant scripts at build time to compile and package our bundles makes us look good when compared with other open source projects.

"Other researches have found that the Linux build engineers have spent a considerable amount of time to make their build system as simple as possible for developers, as the expense of a very complex and hard to maintain core build system machinery. "

People always assume that writing software is hard, but for some reason building software should be easy. Build systems are complex beasts and obfuscating the complexity for the user is just as difficult as it might be when writing a full scale application. Writing good software is difficult, and so is constructing an elegant and effective build system.  It's just a different skill set.

 "Up to 79% of source code developers and 89% of test code developers are significantly impacted by build maintenance, yet investment in build experts can reduce the proportion of impacted developers by 22% of source code developers and 24% of test code developers."

So it looks like if you hire a release engineer and the productivity of your developers will increase.  You would buy a new machine to make the build faster, why not hire a fantastic release engineer and make the build better? The numbers indicate an great return on investment. 

Building complexity

Image ©algonquin_college, http://www.flickr.com/photos/algonquin_college/4971004199/in/photostream/ licensed under Attribution-ShareAlike 2.0 Generic (CC BY-SA 2.0)  


The second paper of is interest is entitled "The Evolution of Java Build Systems" (PDF). 

Again it looks at open source build systems that use either Ant (ArgoUML, Tomcat, JBoss, Eclipse-core) or Maven (Hibernate and Geronimo). The authors find that as the number of source lines of  code being built (SLOC)  is strongly correlated with the number of build lines of code (BLOC).

"Similar to Lehman's first law of software evolution, build system specifications tend to grow over time unless explicit effort is put into refactoring them."

and

"The Halstead complexity of a build system is highly correlated with the build system's size (BLOC), indicating the BLOC is a good approximation of build system complexity".

Lots of build code means increasing complexity.  If you are only building a few bundles, your build is easier to understand.  Makes sense. 

From their analysis, they concluded found that both Ant and Maven based builds evolved in similar fashion.

I like this sentence

"Despite the crucial role of build systems and their non-trivial maintenance effort, software engineering research rarely focuses in them".

I'd like to thank the authors for conducting this research and look forward to reading more in the future.  Build systems are complex systems and I welcome the efforts of these researchers to quantify way they can be improved. And it's extra special when they look at the projects that we work on every day!

References:
It will never work in theory
Queens University Software Analysis and Intelligence Lab

Read more...

Open source: She loves you, She loves you not

Working in an open source community is great, but there are also some disadvantages. From my admittedly narrow world view as a committer working in the Eclipse community as an employee of a large corporation, here are some of my observations.

Image ©emzee, http://www.flickr.com/photos/emzee/162362897/sizes/z/in/photostream/ licensed under Creative Commons by-nc-sa 2.0  



Things I love about working in open source

1. The people you meet are amazing and enthusiastic.   It's is a privilege to work with such a group of talented and thoughtful people.
2. Instead of non-disclosure agreements, we're free to talk about everything we work on.  Since everything is open, there are tremendous learning opportunities.  This inherent openness  means university researchers write papers about our work. Interesting!
3. Open source software is used by our consumers in unique ways.  When Jeff Norris gave his keynote at EclipseCon 2010 on how Eclipse was used to monitor robots on Mars, it was extraordinary to think that we as a community had contributed to something that was literally out of this world.  And when he said thanks to the committers during his talk, it was a very proud moment for us all.  There are many other examples of ways in which Eclipse is used to extend our knowledge of the world(s) around us.

Image ©owlbookdreams, http://www.flickr.com/photos/owlbookdreams/3721505610/ licensed under Creative Commons by-nc-sa 2.0  


Some things I don't love about working in open source

1. Many people complain but don't contribute. They consume the code we craft, complain about how its written, but are loathe to roll their sleeves up and write a useful patch.
2. We are constantly starved for resources, whether it be people, or new hardware.  There are too many bugs and not enough days :-)
3. My gender is an outlier in our community.

Ten years ago this week, I started working in at a small company called Object Technology International (OTI) as a system administrator.  I was asked by Jeff McAffer to install a server called dev.eclipse.org that would act as a CVS and bugzilla server for a project that they were going to open source called Eclipse.  I said something like "Open source, that's great.  Just like Linux".  To which Jeff replied along the lines of  "Well, I don't think it will be as popular as Linux, but we'll see what happens".  And the rest as they say, is history.

What do you love about working in open source?

Read more...

Kill Build: Vol. 1

Here are some of the ways I've seen a build killed during my years as an Eclipse committter.


1.  Lightning. Ottawa has tremendous thunderstorms in the summer.  What also happens during the early summer? Our coordinated release.  One summer, a transformer next to our office was hit by lightning.  Boom.  No power for about 12 hours. Bye bye build.

Image © ViaMoi, http://www.flickr.com/photos/viamoi/3338093351/ licensed under Creative Commons by-nc-sa 2.0  

2.  Ice.  The air conditioner in our lab filled with ice and ceased to function. The machines in our build lab then overheated and shut down.

3.  Flood: Our lab's air conditioner leaked the resulting water and flooded the lab. The build was relegated to a watery grave.

Image © thirsk, http://www.flickr.com/photos/thirsk/840616475/ licensed under Creative Commons by-nc-sa 2.0   

4. Random loss of power.  A committer plugged in a space heater in her office which tripped a circuit breaker and killed the power in our build lab. Or the total and intermittent loss of power from the Ottawa Hydro.  In the end, the results is the same: a dead build.

Image © ViaMoi, http://www.flickr.com/photos/viamoi/3339707547  licensed under Creative Commons by-nc-sa 2.0   


5.  Hardware self-destruction.  The eclipse foundation's power supply for a disk array sparked and brought down the switching gear. Dying switches, UPS's and drives have also wrought havoc.

6.  Permission issues: Permissions problems on the filesystem caused committing to the repository to fail. Or the signing process to time out.  Or both.

7. Network timeouts aka no network love.

Image © zoso_tc, http://www.flickr.com/photos/zoso_tc/3024964999/  licensed under Creative Commons by-nc-sa 2.0    

8. Human Error: The usual suspects compile errors, uncoordinated changes across projects, problems in builder itself, corrupted jars etc. have sent many builds to an early death.

9. Upgrades invoke the unexpected.  Changing a ssl certificate unleashed shenanigans.  An simple upgrade of a test machine caused failures.

Image © waferboard, http://www.flickr.com/photos/waferboard/5321533361/  licensed under Creative Commons by-nc-sa 2.0    


10.  Not respecting resource constraints.  Consuming all the disk space, CPU, and bandwidth available is a fine way to finish off a build.

How your builds failed?  Any exciting ways I've missed?

Read more...

Looking toward the Indigo finish line

This week, the Eclipse and Equinox projects released 3.7RC3.  For the past month, we have been in end game mode.  This means that make release candidates available about once a week, each with progressively stringent requirements to release a fix. All code changes require approval as follows

RC1 - Another committer must +1 your bug before you can submit a change.
RC2 - Two committers must +1
RC3 - Two committers and a component lead must approve.
RC4 - Two component leads must +1 your bug. A component lead can veto your change. Steve Northover coined the term intergalactic approval for a bug fix at this stage of the game.

Image © lrargerich, http://www.flickr.com/photos/lrargerich/4999906554/ licensed under Creative Commons by-nc-sa 2.0 

During this entire period, documentation changes can go in without approval.   During RC3 and later, all code changes must be announced to our project's release engineering mailing list to communicate to the other committers that you intend make a change.

There are some interesting parallels between the Eclipse end game and long distance running.  This weekend, 40,000 runners and 100,000 spectators will participate in Otttawa's Race weekend.   The week before their race, athletes participating in the longer distances will taper their runs.  In other words, their training mileage drops significantly so that they're well rested for race day.

Image © Gamma-Ray Productions, http://www.flickr.com/photos/29143375@N05/3574211684/ under Creative Commons by-nc-sa 2.0 


At Eclipse, the process is similar.  We taper our bug fixes as we approach release day. Eclipse is known for it's stable and predictable releases. At lunch yesterday,  my committer colleagues remarked that some of their dependencies (not Eclipse projects) just release at regular intervals and and don't have a ramp down process.  Other projects release occasionally and then don't have a release for several years.  No predictable release schedule.  At Eclipse, we are predictable and stable.  Some may call that boring, but it allows other projects and products who depend us to consume us at regular intervals.  Eclipse 3.7 is especially stable this year given that many committers have been busy releasing innovative new features to Orion and Eclipse 4.1.

We also have weekly test days where the committers and community test the builds and ensure that there aren't any regressions that were recently introduced.  For example, here is the PDE team's test plan. 

In addition to reducing the amount of changes we release to the build,  to ensure the stability of the code base, this tapering of bug fixes and the associated approval process also ensures that other committers have a chance to evaluate our patches.  This sanity check ensures that the fixes are both significant enough to warrant releasing to the build at this time and technically correct.  I can't speak for other committers, but at this stage of the Eclipse release, we are all a bit tired.  So it's good to have several sets of eyes looking at your patch.

Committer after Indigo release

Image © kaibara,http://www.flickr.com/photos/kaibara/4068996309/ licensed under Creative Commons by-nc-sa 2.0 

At this stage of the game, only serious bugs are fixed.  It's not worth causing ripples within the code to fix a minor bug only to find out after the release is available that this fix had intended consequences.

Image © Marc Dezemery, http://www.flickr.com/photos/mdezemery/295687769/ licensed under Creative Commons by-nc-sa 2.0

What process does your project use to ensure that your code base is ready for release?

Further Reading
Eclipse 3.7 Endgame plan

Read more...

SDK code coverage with JaCoCo

Today, I had to pleasure of closing  bug 241254 which has been open for over two years.  It was a request to run a code coverage tool against the SDK during the build.

We used JaCoCo which is an EPL licensed code coverage library brought to you by the same team that created EclEmma.  Why did we chose JaCoCo?

1. Help from JaCoCo committers. At EclipseCon 2010, Marc Hoffman (JaCoCo committer) indicated to Olivier Thomann that he was interested in helping us to implement JaCoCo in our build.  
2. Performance.  Other code coverage tools made the JUnit tests take significantly longer.  Our tests take long enough to finish as it is!  Olivier tested JaCoCo and found that it only took the JDT core tests about 2% longer to complete with JaCoCo enabled. Our 68,000 JUnit tests take about seven hours to run in parallel on three platforms.  The JDT Core tests comprise 37,000 of these tests, thus we estimated the impact of running  JaCoCo during the build would only increase the test run by about eight minutes.
3. Classes are instrumented on the fly and and remain untouched after the code coverage data is generated.  Other code coverage tools require you to instrument the classes for code coverage. Given the huge code base of the SDK, this didn't seem like a feasible alternative.
4.  JaCoCo reports look awesome.

How did we implement JaCoCo in the build?

1. Olivier tested an earlier version of JaCoCo and worked with the JaCoCo committers to address some issues.  Once these issues were resolved, we opened IPZilla bugs to consumed JaCoCo as part of the Indigo release.
2. The Eclipse SDK JUnit tests are run with the assistance of a project called org.eclipse.test.    An Ant script called libary.xml is used to invoke the test suites for each test bundle.  We added the JaCoCo agent and Ant jars to a new library directory of this project.These jars are then added to the extraVMargs property when the tests are run. The libary.xml also specifies the output directory of the coverage data (*.exec files). 
3. When the tests are run, corresponding exec files are created for each test bundle.
4. We modified the test.xml that is used to run all the tests suites to generate a code coverage report upon completion of the JUnit tests. All the *.exec files are merged into a single file.  A coverage report for each bundle in the SDK is run.  (excluding Orbit and doc bundles as well as fragments). The Ant for the report generation looks like this:

One problem that we encountered was that that JaCoCo report task was running out of heap space when the source files were passed to it as as zipfileset instead of a fileset.  This problem occurred especially when attempting to generate source reports for large bundles such as jdt.core. We overcame this problem by unzipping the source bundles into a temporary location and passing the fileset.

In addition to code coverage percentages, the report generates html of the source files that highlights the code coverage within the code itself.  For example:

All this information consumes about 1GB per build.  (Sorry Denis and Matt).  Thus I have only enabled it for our weekly integration builds, and the data will not be sent to the mirrors.

The code coverage reports  for 3.7M6 are now available. Check out your favourite Eclipse SDK bundle!



References
JaCoCo project
JaCoCo mission and design goals
JaCoco Ant tasks
Bug 241254 - run code coverage tool during build
Marc Hoffmans's JaCoCo talk at Eclipse Summit Europe 2010

Read more...

Thoughts on Making Software

My husband (aka Mr. Releng) and I were driving home from his office party in mid-December and had the following conversation.

Me:  A publicist from O'Reilly emailed me today and asked if I would like a free copy of Making Software.

MR:  Huh.  Why did they offer you a free book?

Me:  I don't know. I mentioned to one of my Eclipse colleagues on Twitter  that I thought it would be an interesting book to read.  Also, one of the editors is Greg Wilson, who's also the editor of the upcoming The Architecture of Open Source Applications.  (I contributed a chapter on Eclipse to AOSA).

MR:  Next time, mention on Twitter that you'd like a free car. What's the book about?

MR: Free car from Twitter?  Don't get your hopes up.  The book investigates the evidence that supports common software engineering practices. In medicine, a clinical trial is conducted to see if a new drug has a statistically significant effect compared to placebo or an existing treatment. The same evidence based principles can be applied to software engineering to determine if TDD or Agile methods are actually effective.  I've always thought that there was a lot of shouting but scant evidence to support different software development practices.

MR:  Yeah, undergraduate computer science is engineering.  Graduate level computer science is math.

From xkcd

As you may have guessed, I'm married to a mathematician.

In addition to being a fan of open source, I also enjoy reading about different scientific disciplines.  I love reading scienceblogs and have read many great books over the past year about evidence based medicine. So when I heard that there was a book available that examined what software engineering practices actually work, I was intrigued.

In any case, I finished reading it recently.   It was very interesting and I learned a lot!  The book is split into two sections.  The first section dealt with different research methods.  For instance,

• How to conduct a systematic literature review
• Empirical methods that can be applied to software engineering studies and why software engineering is inherently difficult to measure quantatively.
• How to decide on the papers to include in a meta-study. 

From xkcd

 
The second half of the book examines the evidence for different questions in software engineering.  For instance:

• How do you measure programmer productivity?  Are some developers really an order of magnitude more productive than their team mates?
• Does test driven development result in better software?
• What's the cost of discovering a bug early in the development cycle versus after the product has shipped?
• What working environment (cubicles/offices/open concept etc) are the most productive for software developers?
• Is there a measurable difference in software quality between open and closed source software?
• What are the reasons for the low proportion of women working in the software industry?
• Does the use of design patterns improve software quality?
• How do mine data from open source repositories for your own studies (Chapter 27 uses the Eclipse project as an example :-)

All in all, I found it a very interesting book that examined the actual empirical evidence to support or refute some of the sacred cows in software engineering. I think this this is a refreshing step forward for our profession.  If there aren't numbers to support that the way we work is effective, shouldn't we alter our path to use better methods?

Further reading
Software Carpentry: Empirical results

As an aside, some great books on evidence based medicine are
Bad Science by Ben Goldacre
Trick or Treatment by Simon Singh and Edzard Ernst
Snake Oil Science by R. Barker Bausell

Read more...

Eclipsing the build

A plan item for 3.7 is to transition the Eclipse and Equinox build process to run on eclipse.org hardware.  Why may you ask, isn't this the case today? Two reasons:

1.  Until recently, the Eclipse foundation didn't have hardware to support our requirements of running tests on multiple platforms (Windows, Mac, Linux).
2.  There wasn't a plan item to transition our build to eclipse.org until 3.7.   You may think that we open source developers are happy hippies who have the freedom to work on the bugs that interest them :-). The reality is that I prioritize my bugs as follows:

• Fix bugs that cause the build to fail.
• Fix bugs that block other committers from moving forward with their work or are necessary for legal reasons.  
• Plan items
• Everything else

I was going to write a post describing the progress migrating to eclipse.org hardware but I thought it would be a good idea to give you some background regarding how the Eclipse and Equinox project's build works today. 

The Eclipse build creates SDKs for 15 platforms.

A graphical representation of the 15 platforms we build  (os,ws,arch)

We use fragments to define platform specific code that is associated with a host bundle.  In some of these fragments, there's C code that needs to be compiled on the platform specific hardware to create binary libraries.  For instance, the SWT and equinox launcher fragments are examples of  these artifacts. We have about seven machines to compile the libraries for these 15 platforms, as some are cross compiled.  The SWT and Equinox team have hudson instances, which connect to all these internal IBM servers, compile the C code, and commit the resulting libraries in binary form to CVS at eclipse.org. This part of the build isn't transitioning to eclipse.org.

A simplified description of the existing Eclipse build that produces the artifacts that are available for download on eclipse.org is shown below.   Essentially, once a build is launched from cron, the build scripts check out all the code from eclipse.org to the IBM build machine which compiles the bundles. These bundles are copied to eclipse.org to be signed, then copied back to the IBM build machine to be assembled into a p2 repo, from which we create SDKs to run the tests. The zips, repos and test results are rsynced to eclipse.org.

Once we get the issues worked out with the new Hudson slaves to run tests, all the JUnit tests will run on eclipse.org hardware.  Eventually,  we hope to run the performance tests there once we have sufficient hardware.

Advantages of moving the build to hudson.eclipse.org

1. The build process will be more open and other Eclipse and Equinox committers will be able to start a build.  Today, I manage the crontab of the build machine.
2. Fewer issues with network timeouts due to the fact that the eclipse.org filesystem is local to Hudson.
3. Hudson has a rich feature set and many plugins to extend it. 

Disadvantages

1. Like kindergarten, we'll have to share the filesystem and machines with others.  Today the build runs on dedicated machines and thus we don't have resource constraint issues from other teams.  (The build process itself isn't faster at eclipse.org because of these resource constraints, but the tests are because at the moment we are the only team using the Windows and Mac test machines.)
2. Can't hack and slash the build on the the command line to rerun a single test suite or restart the build manually.

In a future post, I'll describe the process to transition our builds to run on eclipse.org and the obstacles that we have overcome.

How can you help?  Consider donating to Friends of Eclipse. 

References

Build at Eclipse.org

Graphics thanks to open source clip art

Servers
Database
iMac
Server cabinet
Tower
Database

Read more...

Implementing shared licenses with 3.7M5

This is the best Eclipse license I've ever seen.

Image © chriscardinal, http://www.flickr.com/photos/disatasu/3243844762/ licensed under Creative Commons by-nc-sa 2.0

Conversely, dealing with license files in Eclipse features is a bit painful.  Today, each time the Eclipse license changes, you need to copy the new license files to your features.  The license files included in the Eclipse SDK are highlighted below:

These same files are included in the 46 features that the Eclipse and RT Equinox project builds.  This is very tedious to update when the license files change. Lots of copy and paste. The license text also appears in the feature.properties.


In Eclipse 3.7M4,  the PDE and p2 teams added support for shared licenses.

The license files will be copied from the license feature to the feature that refers to it.  In 3.7M5, I modified our features to use this shared licenses. For instance, the beginning of the feature.xml for the Eclipse SDK now looks like this:

To implement this, I followed these steps. 

1. Create a license feature. This feature doesn't include any bundles or features. It only stores the license files which its build.properties reference.   The feature.properties has only two properties - the licenseURL and license. 

2. Update the existing features to reference the license feature.  Remove the license files from these features.  Remove the references to the license files from your feature's build.properties.  The values will be inherited from the build.properties of the license feature.  Remove the name and values of licenseURL and license from the feature.properties. These will be included from the license feature. Ensure that you don't have duplicate properties in the feature.properties of the license feature and feature your are updating. This will cause a build failure.


3.  Update your map files to include to the new feature. You don't need to nest the feature in another feature to fetch it, this happens automatically. For example, this is where our license feature is located

feature@org.eclipse.license=v20110121,:pserver:anonymous@dev.eclipse.org:/cvsroot/eclipse,,org.eclipse.sdk-feature/features/org.eclipse.license

4. Run a test build. Fix typos until you have a clean build. Verify that all your features still include the appropriate license files. 

There isn't an IU for the license feature included in the p2 repository.  The license text is included in the metadata.

Note that you'll need to build with 3.7M5 bundles.  There was a bug in the 3.7M4 bundles that prevented shared licenses from working in generated source features.

The advantage of this approach is that when the license files are updated, we'll only need to update the license feature.  We'll also have to increment the versions of the license feature in the referring feature, but that's a simple search and replace. 

Some other implementation details: I could have also stored the copyright information in the feature.properties of the license feature. However, the copyrights of our features are not identical, so I didn't choose this approach. This may be useful for other teams.

Thanks to the  PDE and p2 team for implementing this!


Further references

Bug 306818 - Improve license support in p2.
License mechanism description in wiki
Bug 332662 - Adopt new feature that allows to share the license file

Read more...