Research, Release Engineering and ROI

I recently read some academic papers that quantify some release engineering practices in open source communities.  Very interesting.

The first one, "An Empirical Study of Build Maintenance Effort" (PDF), looks at how the time spent maintaining the build impacts the developers in a project.  It examines build coupling which refers to the how often changes in source code require changes in build code, as well as build ownership, which is the proportion of developers on the team who are responsible for maintaining the build.  The projects studied were ArgoUML, Hibernate, Eclipse (SDK), Jazz, GCC, Git, Linux, Mozilla, PLlot and PostgresSQL.

Here are some snippets I found interesting in this paper.  Note that when they refer to "Eclipse-core" they mean the Eclipse project's build. In this paragraph, they are talking about build coupling and how to reduce it

"In Eclipse-core, the coupling is reduced to 16% and in Jazz the observed coupling is a mere 4%.  Eclipse-core and Jazz both leverage the automated Eclipse Plugin Development Environment (PDE) build technology. ..... Since the developer must only maintain the high-level build.properties file (via the IDE), the daily build maintenance is reduced".

Thank you PDE family.  Dynamically generating Ant scripts at build time to compile and package our bundles makes us look good when compared with other open source projects.

"Other researches have found that the Linux build engineers have spent a considerable amount of time to make their build system as simple as possible for developers, as the expense of a very complex and hard to maintain core build system machinery. "

People always assume that writing software is hard, but for some reason building software should be easy. Build systems are complex beasts and obfuscating the complexity for the user is just as difficult as it might be when writing a full scale application. Writing good software is difficult, and so is constructing an elegant and effective build system.  It's just a different skill set.

 "Up to 79% of source code developers and 89% of test code developers are significantly impacted by build maintenance, yet investment in build experts can reduce the proportion of impacted developers by 22% of source code developers and 24% of test code developers."

So it looks like if you hire a release engineer and the productivity of your developers will increase.  You would buy a new machine to make the build faster, why not hire a fantastic release engineer and make the build better? The numbers indicate an great return on investment. 

Building complexity

Image ©algonquin_college, http://www.flickr.com/photos/algonquin_college/4971004199/in/photostream/ licensed under Attribution-ShareAlike 2.0 Generic (CC BY-SA 2.0)  


The second paper of is interest is entitled "The Evolution of Java Build Systems" (PDF). 

Again it looks at open source build systems that use either Ant (ArgoUML, Tomcat, JBoss, Eclipse-core) or Maven (Hibernate and Geronimo). The authors find that as the number of source lines of  code being built (SLOC)  is strongly correlated with the number of build lines of code (BLOC).

"Similar to Lehman's first law of software evolution, build system specifications tend to grow over time unless explicit effort is put into refactoring them."

and

"The Halstead complexity of a build system is highly correlated with the build system's size (BLOC), indicating the BLOC is a good approximation of build system complexity".

Lots of build code means increasing complexity.  If you are only building a few bundles, your build is easier to understand.  Makes sense. 

From their analysis, they concluded found that both Ant and Maven based builds evolved in similar fashion.

I like this sentence

"Despite the crucial role of build systems and their non-trivial maintenance effort, software engineering research rarely focuses in them".

I'd like to thank the authors for conducting this research and look forward to reading more in the future.  Build systems are complex systems and I welcome the efforts of these researchers to quantify way they can be improved. And it's extra special when they look at the projects that we work on every day!

References:
It will never work in theory
Queens University Software Analysis and Intelligence Lab

Read more...

Open source: She loves you, She loves you not

Working in an open source community is great, but there are also some disadvantages. From my admittedly narrow world view as a committer working in the Eclipse community as an employee of a large corporation, here are some of my observations.

Image ©emzee, http://www.flickr.com/photos/emzee/162362897/sizes/z/in/photostream/ licensed under Creative Commons by-nc-sa 2.0  



Things I love about working in open source

1. The people you meet are amazing and enthusiastic.   It's is a privilege to work with such a group of talented and thoughtful people.
2. Instead of non-disclosure agreements, we're free to talk about everything we work on.  Since everything is open, there are tremendous learning opportunities.  This inherent openness  means university researchers write papers about our work. Interesting!
3. Open source software is used by our consumers in unique ways.  When Jeff Norris gave his keynote at EclipseCon 2010 on how Eclipse was used to monitor robots on Mars, it was extraordinary to think that we as a community had contributed to something that was literally out of this world.  And when he said thanks to the committers during his talk, it was a very proud moment for us all.  There are many other examples of ways in which Eclipse is used to extend our knowledge of the world(s) around us.

Image ©owlbookdreams, http://www.flickr.com/photos/owlbookdreams/3721505610/ licensed under Creative Commons by-nc-sa 2.0  


Some things I don't love about working in open source

1. Many people complain but don't contribute. They consume the code we craft, complain about how its written, but are loathe to roll their sleeves up and write a useful patch.
2. We are constantly starved for resources, whether it be people, or new hardware.  There are too many bugs and not enough days :-)
3. My gender is an outlier in our community.

Ten years ago this week, I started working in at a small company called Object Technology International (OTI) as a system administrator.  I was asked by Jeff McAffer to install a server called dev.eclipse.org that would act as a CVS and bugzilla server for a project that they were going to open source called Eclipse.  I said something like "Open source, that's great.  Just like Linux".  To which Jeff replied along the lines of  "Well, I don't think it will be as popular as Linux, but we'll see what happens".  And the rest as they say, is history.

What do you love about working in open source?

Read more...