Implementing shared licenses with 3.7M5

This is the best Eclipse license I've ever seen.

Image © chriscardinal, http://www.flickr.com/photos/disatasu/3243844762/ licensed under Creative Commons by-nc-sa 2.0

Conversely, dealing with license files in Eclipse features is a bit painful.  Today, each time the Eclipse license changes, you need to copy the new license files to your features.  The license files included in the Eclipse SDK are highlighted below:

These same files are included in the 46 features that the Eclipse and RT Equinox project builds.  This is very tedious to update when the license files change. Lots of copy and paste. The license text also appears in the feature.properties.


In Eclipse 3.7M4,  the PDE and p2 teams added support for shared licenses.

The license files will be copied from the license feature to the feature that refers to it.  In 3.7M5, I modified our features to use this shared licenses. For instance, the beginning of the feature.xml for the Eclipse SDK now looks like this:

To implement this, I followed these steps. 

1. Create a license feature. This feature doesn't include any bundles or features. It only stores the license files which its build.properties reference.   The feature.properties has only two properties - the licenseURL and license. 

2. Update the existing features to reference the license feature.  Remove the license files from these features.  Remove the references to the license files from your feature's build.properties.  The values will be inherited from the build.properties of the license feature.  Remove the name and values of licenseURL and license from the feature.properties. These will be included from the license feature. Ensure that you don't have duplicate properties in the feature.properties of the license feature and feature your are updating. This will cause a build failure.


3.  Update your map files to include to the new feature. You don't need to nest the feature in another feature to fetch it, this happens automatically. For example, this is where our license feature is located

feature@org.eclipse.license=v20110121,:pserver:anonymous@dev.eclipse.org:/cvsroot/eclipse,,org.eclipse.sdk-feature/features/org.eclipse.license

4. Run a test build. Fix typos until you have a clean build. Verify that all your features still include the appropriate license files. 

There isn't an IU for the license feature included in the p2 repository.  The license text is included in the metadata.

Note that you'll need to build with 3.7M5 bundles.  There was a bug in the 3.7M4 bundles that prevented shared licenses from working in generated source features.

The advantage of this approach is that when the license files are updated, we'll only need to update the license feature.  We'll also have to increment the versions of the license feature in the referring feature, but that's a simple search and replace. 

Some other implementation details: I could have also stored the copyright information in the feature.properties of the license feature. However, the copyrights of our features are not identical, so I didn't choose this approach. This may be useful for other teams.

Thanks to the  PDE and p2 team for implementing this!


Further references

Bug 306818 - Improve license support in p2.
License mechanism description in wiki
Bug 332662 - Adopt new feature that allows to share the license file

Read more...

Revert to an earlier p2 profile from the command line

Today, we're running builds toward 3.7M5 for the milestone that will be available Friday, January 28.  This morning I updated to our latest integration build from the repository. Unfortunately, one of the bundles had a compile error which was serious enough to cause Eclipse to crash on startup.

If you want to revert your installation to a previous version, this is easy to do from the UI. Chris describes this here. 

However, if you can't access the Eclipse UI, reverting becomes a bit more complicated. The online help illustrates how to use the p2 director application to revert your install to a previous profile state.

The list of profiles available in your Eclipse install are located in your profile registry.  The highlighted profile id below reflects my working installation from last week.

To revert to this profile, I downloaded a new build, extracted it to a directory and ran the following command

C:\N20110122-2000\eclipse\eclipsec.exe -application org.eclipse.equinox.p2.director -repository http://download.eclipse.org/eclipse/updates/3.7-I-builds -revert 1295719004390 -destination c:\I20100309-0809\eclipse -profile SDKProfile -bundlepool C:\I20100309-0809 -p2.os win32 -p2.ws win32 -p2.arch x86

Operation completed in 68759 ms. 

My install was reverted to the previous profile and I could once again start Eclipse.

Read more...

Hudson best practices and Eclipse

One of the plan items for Eclipse 3.7  I've assigned is to migrate the Eclipse and Equinox build to run on the Hudson installation at eclipse.org.  So I've spent a lot of quality time with Hudson lately. (Soon to be called Jenkins).

Today, I attended a webinar about optimizing Hudson in production hosted by Kohsuke Kawaguchi of CloudBees. Since my handwriting is similar to the last slide in this Oatmeal cartoon, I thought it would be better to jot my notes down in a blog post. As I was listening to his suggestions, I thought about how his seven best practices apply to our use of Hudson at Eclipse.


1. Backups 

• Backup your Hudson install to protect from disaster and accidental configuration changes.

 

Image © davemorris, http://www.flickr.com/photos/davemorris/2395980976/sizes/z/in/photostream/   licensed under Creative Commons by-nc-sa 2.0 

• $HUDSON_HOME on the master stores everything that needs to be backed up. The slaves doesn't need to be backed up.
• Live backups are fine as configuration changes are atomic
• In addition to the backup of the configuration, a backup of the filesystem is necessary to provide consistent snapshot.
• Backups are great but can you restore?  On a regular basis, try restoring your hudson configuration and starting up a test hudson instance with the restored hudson.war file.

2. Disk

• Prepare for disk utilization growth
• Space is more important than speed when ordering new drives
• Plan for expandable disk volumes
• If it's too late, symlink is your friend

3.  Use native packages

• For instance, *.deb, *.rpm.  Easy to update and install, and they include init scripts.
• Configuration is stored in /etc/default/hudson and /etc/sysconfig/hudson
• Existing $HUDSON_HOME can be migrated
• Native Windows package is a work in progress

@Eclipse:  Backups and disk configuration are managed by the webmasters, along with upgrading and installing hudson.


4. Distributed builds

• You will grow beyond a single system
• Increasing load on a single machine is not the only issue
• It's cheaper to grow horizontally (additional machines) versus vertically (more memory and CPUs on a single machine)
• It's also good to have isolation between build processes in case you want to diversify the test and build platforms
• Let master launch slaves via SSH or DCOM. This makes it easier to keep the cluster up and running.
• SSH public key authentication makes it easy to connect to slaves on disparate platforms.

@Eclipse: Yes, we have distributed builds and several platforms available for test purposes.

5.  Labels

Treat build machines like livestock, not like pets

Image © davegroth, http://www.flickr.com/photos/davegroth/9107888/sizes/m/in/photostream/   licensed under Creative Commons by-nc-sa 2.0 

• Build machines should be interchangeable
• Don't tie builds to a specific build machine,  unless there are platform limitations. This approach will yield better resource utilization and less downtime.  Queued builds can use the next available machine instead of waiting for a specific machine.
• When using labels for your Hudson slaves, name them to reflect their capability.  For instance, if the slave is Mac or Windows hardware, the label should reflect this.
• If  there is a real reason to limit build to run on a single machine (such as wanting to run platform specific tests), tou can use boolean expressions with labels.  For instance, this Hudson job shouldn't run the slaves running Mac or Windows

@Eclipse: We use label our build machines as livestock.  No pets for us. However, I'm quite envious that the Mozilla foundation has adopted red pandas (firefoxes).

  6.  Invest in a good URL

• By default Hudson runs on port 8080.  This is difficult for users to remember.
• Use the host's alias instead of the machine name.  Thus if the machine names changes, the users won't have to go to a new URL.
• Share port 80 with other applications using Apache reverse proxy with which will also allow you to run Hudson as a non-root user.   

@Eclipse: https://hudson.eclipse.org is short and simple.

7.  Try to keep the build records under control.

• Discard old build records if you can
• Removing old builds reduces start-up time and memory usage. 
• Settings are on a per-project basis. For instance, here are settings for one of my Eclipse builds.

@Eclipse: Yes, we do clean up old builds. Also, the webmasters have created a cron job to send a friendly email every week listing of the Hudson disk utilization to remind the release engineers to clean house.

The slides are available here and obviously provide a lot more detail than my short summary.  If you're interested the best practices for configuring Hudson for a production environment that has room to grow, this presentation is provides very useful suggestions.

Read more...

Smashing open source stereotypes

The Wall Street Journal published an article yesterday where it described the open-source software movement, whose activists tend to be fringe academics and ponytailed computer geeks. This is one of the many unfortunate stereotypes surrounding open source that I've heard over the years.   Let's break some of those down.


1.  Open source is a fringe movement.

Open source builds mainstream software. I work all day long contributing to open source software, as do many of my colleagues.  Why do corporations pay us to contribute to software that doesn't have a price tag?  It's in their self-interest to have employees improving the open source software that underlies their commercial offerings.  Here are some of the fringe groups involved in Eclipse.

Source: Eclipse Foundation Members' Meeting  Q4 Update December 16, 2010

These unknown companies are members of the Linux foundation.

2.  Open source is not professional.

At Eclipse, we ship our software on time every year.  We've done that for seven years.  That's professional.

 Source: Ian Skerrett: The Helios Train has arrived

3.  Open source contributors are loners sitting in their parents' basement working on arcane bugs.

Nope.  Open source is very social.  We interact on IRC and twitter.  We attend democamps, learn about new projects and build new businesses. We speak at conferences. The software we work on makes a difference in the world. It helps monitor robots on Mars, manages trains in Switzerland and models chemical reactions.

Source: Madhu Samuel: Eclipse Democamp Bangalore 2010

4.  Open source developers aren't in good physical condition and don't have interests outside of work

Again, a stereotype that has been proven wrong.    We ran 570K at EclipseCon 2010. 

Open source contributors run marathons, bike, curl, swim and play soccer or  hockey.  In fact, you can order an Eclipse bike shirt.

We also volunteer at schools and raise money for charity. 

5.  Open source contributors are always male.

Source: Miriam Ruiz: Women in Free Software

Wrong again. 

We may be a small minority, but here's hoping that these fine female role models will inspire a younger generation of women to consider the amazing career opportunities that open source has to offer.  Two generations ago, female doctors were rare.    Today, women are over 50% of new Canadian medical school graduates. So, may borrow a phrase from a recent campaign, it will get better.

What other open source stereotypes would you like to smash?

Read more...

Thoughts on Being Geek

Sitting on the plane travelling to Nova Scotia for the holidays, I read Being Geek: The Software Developer's Career Handbook by Michael Lopp. I've wanted to read this for a while so I was happy to finally have some time to do so.

It was a very informative and funny book, especially for someone like me who has spent a long time working in the software and IT services industry. Some key points it covered:

• You are responsible for your career. Don't depend on your manager or HR. They are looking out for their own interests.
• The importance of mentoring and having a career plan.
• How to negotiate your salary and benefits for a new job, and how this calculation should be altered if you are looking to move to a start-up versus an established company.
• Evaluating organizational culture outside the boundaries of hierarchy.  Where is the informal power  within an organization?
•  I really liked the chapter entitled "Your People" on the art of networking and the importance of connecting with people in your community face-to-face and getting to know them better.  As I read this I thought, my people are the Eclipse family. 
• Strategies for dealing with different types of managers and co-workers.
• How to decide if you want to be a manager.
• Dealing with a crisis situation such a major bug just before the ship date.
• The downsides of working in our industry.

As an Eclipse committer, this quotation from the chapter on selecting the tools to use as a developer made me laugh.

"Yeah, I know all about the glory of integrated debugging, and I see all you Eclipse guys having a ball, but what I have found in many years of developing is that embracing fancy tools means tinkering with tools to get them to behave how you want."

-Michael Lopp, Being Geek

There's something to be said for simplicity.

My only criticism about this book is that the example software developers in the book were overwhelmingly male.  There were very few female software developers if any used to as characters in his stories. Legend has it that they do exist.  All the examples of women in the book seemed to be in product marketing or similar roles.

Anyways, if you are interested in a thought provoking book about strategies for managing your software career this is definitely a good one.

Read more...