Tuesday, November 03, 2009

Happy Birthday Eclipse...now some pictures from your youth

Saturday November 7th, 2009 is Eclipse's 8th birthday. Eight years ago the first Eclipse downloads were publicly available along with the source code.

Here's a picture of the first www.eclipse.org. The website was slashdotted shortly after the "Eclipse is open source" announcement was made. The hard drive had to be replaced, thus the missing cover. This machine was replaced by a real server a few weeks later. A couple of years later, this was replaced by faster and more fault tolerant hardware managed by our most excellent webmasters.



I went through my desk the other day, and found some emails of the original requirements for the eclipse infrastructure, project structure and commit rights. A fast server has 512MB RAM and 20GB of disk space...really? Okay, I feel old.



Bugzilla was there from the start.



Some of the first mailing lists. A few disappeared but even more new ones were created. Today, we have eclipse on twitter, forums, conferences, marketplace, blogs and the list goes on. Pretty impressive!



In 2001, the linux kernel still fit on a floppy disk. It was useful to have a boot disk in the event that the boot partition became corrupted and the machine wouldn't boot. Much easier than booting from a rescue cd and mounting all the partitions by hand. Especially at 4am. Here's a picture for those who've never worked with a diskette (*ahem* co-ops :).




Initial project structure. Today we have so much much more.


The first commit rights - hello PDE family! Today we have an amazing diversity of committers from around the world and many companies. Eclipse also has friends, but would always like more....who doesn't?




In reality, the past doesn't matter that much, and we need to focus on the future. Sometimes I'm cynical about Eclipse, because as a long time committer, I notice that many people like to play (consume), but fewer want to pay (contribute). And to some degree we make it difficult to contribute without a significant investment of time and a steep learning curve. But most days, I'm absolutely amazed by the work that we as a community can do together, when smart, passionate people strive toward a common goal. Happy Birthday Eclipse!


Now I see IU, now IU don't

Eclipse 3.6M3 went out the door over the weekend, along with a lot of Halloween candy.



Sometimes, you can have too much Halloween candy. And sometimes, you can have too many IUs in your p2 repo. Don't believe me - just look at this repo with bogus bundles - scary!





Both scenarios can cause your friendly neighbourhood release engineer pain. This is unusual because we're a very pain tolerant people. To alleviate the suffering, the p2 team added an Ant task in 3.6M3 that allows you to remove bundles from your repo. As much as I love spending quality time at the command line modifying metadata, Ant tasks that automate tedious jobs are even better.

The p2.remove.iu task will remove both the metadata and the bundle from the repository for a specified IU. For example, if you had bogus packed com.ibm.icu.* bundles in your repo, this task would remove them.

<p2.remove.iu>
<repository location="file://${reposource}" />
<iu id="com.ibm.icu" artifacts="(format=packed)" />
<iu id="com.ibm.icu.base" artifacts="(format=packed)" />
<iu id="com.ibm.icu.source" artifacts="(format=packed)" />
<iu id="com.ibm.icu.base.source" artifacts="(format=packed)" />
</p2.remove.iu>

This task is useful if you'd like to remove some built time bundles from your repo. Or just correct a mistake after a release. It happens. In any case, it's all good. Almost as tasty as chocolate.




Related bugs

Support excluding bundles when running p2.process.artifacts task
p2.remove.iu task should have an option to specify to remove packed file only
ICU jars at Eclipse 3.5 update site have size of 0

Wednesday, October 21, 2009

Hudson is sweet, now the build can tweet

I recently set up a new Hudson build so the Equinox team could test their changes in a branch instead of releasing everything to HEAD. Hudson, like Eclipse, has a rich variety of plugins that can expand the functionality of your build. Hudson has a twitter plugin and thus Eclipse (test) builds tweet here

http://twitter.com/eclipsebuilds

Right now, there are only a few messages about the test builds I started, and stopped when I saw that build was proceeding as normal. In any case, it's pretty cool.

Hudson also has plugins to use additional slave machines on the network to run the build on a faster machine if the current build machine is too busy. It also has plugins to provision Amazon EC2 images and run the build on a slave in the cloud. We currently run about 54,000 JUnit test per platform multipled by five test machines during each build. We run tests on Windows, Mac and Linux machines. But the tests take a long time to complete simply because of the number of tests we run and the lack of additional hardware to run more tests in parallel. It would be fantastic to be able to run the tests on parallel on many machines and finish in fraction of the time they take today. That would allow us to reduce the number of breaking build issues. But of course, this will take significant testing to implement. Sounds like fun!

References
Run JUnit tests in parallel
Hudson build for the p2 branch

Tuesday, October 20, 2009

Test bundles switch from runnable to repo

Earlier this week, I changed the build so that the test bundles are provided in a zipped p2 repository format. A p2 repository looks something like this

...
artifacts.jar
content.jar
features/org.eclipse.sdk.tests_3.6.0.N20091019-1735-9J9fG6sFIKSg8a7j2ZwWY0UAV4BV.jar
plugins/org.eclipse.ua.tests_3.3.300.N20091019-1735.jar
plugins/org.eclipse.jdt.debug.tests_3.1.100.N20091019-1735.jar
binary/
...

The build will install the appropriate test bundles in the eclipse install being tested using the p2 director. Previously, our test bundles were assembled in the runnable format like this

...
eclipse/plugins/org.eclipse.jdt.debug.tests_3.1.100.N20091019-1735/...
eclipse/plugins/org.eclipse.ua.tests_3.3.300.N20091019-1735/..
eclipse/features/org.eclipse.sdk.tests_3.6.0.N20091019-1735-9J9fG6sFIKSg8a7j2ZwWY0UAV4BV
...

and unzipped into the dropins folder. However, the dropins folder is really for legacy purposes. Disadvantages of using the dropins folder include

  • It's to help those applications that expect that dropping a bunch of bundles into an install will work (!).
  • All the bundles in the dropins folder are treated as optional bundles.
  • You can't update the bundles in the dropins folder by using the UI, you are responsible for provisioning them.

The content of the org.eclipse.test framework bundle hasn't changed. However, if you're rerunning the Eclipse and Equinox projects' tests JUnit tests against your product, you'll need to install them into the eclipse your are testing using the p2 director. Unzipping the our JUnit bundles into the dropins folder won't work. Alternatively, you could use the reporunnable task to transform the repository into the old runnable format.

Why make this change? These days, I would expect that most users are installing bundles via a repository such as Galileo or one offered by a product team. Unzipping files is so Eclipse 1.0. So this will allow our tests to replicate the environment that reflects the reality of our users. As well, as I mentioned in my earlier post, we really need to run our tests in parallel on more machines to speed up the build process. Having the test bundles available in a repository, eventually in a shared location, is a step toward that goal.


References

Test bundles should be packaged as a repo


Supported dropins formats

repo2runnable task

Note: The Eclipse and Equinox projects provide their JUnit test bundles in the eclipse-Automated-Tests-${buildId}.zip in the eclipse-testing/eclipse-junit-tests-${buildId}.zip file that is available with every build.

Friday, September 25, 2009

Galileo SR1 has left the station




The Eclipse and Equinox teams are pleased to announce that our 3.5.1 release now available as part of the Galielo SR1 release. Congratulations to all contributors and committers whose hard work made this release possible!

Galileo

3.5.1 Readme

Eclipse

Equinox

You can also update to 3.5.1 using Help->Check for Updates

Enjoy!

Thursday, July 23, 2009

Eclipse Helios: More repos, fewer zips

In the 3.5 cycle, we introduced zipped p2 repositories in the build. These zips could be used to provision your target platform or update your install. We left the per-platform zips on the download page for Galileo. With 15 platforms multiplied by several permutations of features, and runtime versus source downloads, these zips consumed significant space on eclipse.org and the mirrors. Not to mention the fact that platforms other than Windows, Linux and Mac have very low download numbers.

For Helios, I've replaced all the platform specific RCP, JDT, PDE, CVS, examples and releng tools zips with p2 repositories specific to that feature. For instance, instead of 15 platform specific runtime RCP zips, there's a p2 source and runtime feature for each one. The SDK and Platform feature zips for each os.ws.arch combination continue to be built.

More information
How to Consume zipped repositories.
bug 280622

Monday, June 22, 2009

Galileo Release: Eclipse and Equinox 3.5 now available

After a year of development, Eclipse and Equinox 3.5 are now available for download. Please put your feet up and enjoy our new release.


Galileo, Eclipse, and Equinox pages.


Every release is different and this year is no exception.

There's planning and committing to a goal.



Learning new things. See the New and Noteworthy and the readme.

Discussions with committers and community. How many rebuilds will there be?




Drama and angst.



Making new friends. We'd especially like to thank these people who really contributed to the success of this release.



Other Eclipse friends decide to see what's behind a new door.




Some days are a blur.



The final steps at the end of the release seem to take forever.



But the end is sweet.




Congratulations to the Galileo teams on the release. I hope everyone takes time to relax. Because it won't be long before have to lace up your shoes and get ready to run towards Helios!

Friday, June 19, 2009

Adventures in Publishing

For Eclipse 3.5M7, we changed the build to take advantage of the new p2 publisher to to simplify the creation of the platform, RCP and SDK zips. Why switch to the publisher?

  • Metadata is generated on the source when pde build is invoked, instead of after the jars are built.

  • The publisher provides better support for the p2.inf

  • The metadata generator tasks are being deprecated

  • The publisher makes adding a new platform to the build much easier because metadata is generated for the root files. (We added five new platforms in the 3.5 cycle).


  • Also, the Equinox team also wanted to test the publisher in an active build environment and iron out any issues this might reveal. Similar to how Equinox is the reference implementation for OSGi, platform releng is the "reference build" for the Equinox team ;-).

    Previous to using the publisher, we built the sdk feature and the associated root file feature to construct the zips available on our build page. Root files are just files that reside at the root of the eclipse install such as the eclipse executables, config.ini, eclipse.ini and license files.

    To switch the our build to using the publisher I implemented the following steps

    1) Set p2.gathering=true in the build.properties of the builder for our master feature. The master feature contains all the bundles and features we compile in the build. I also set skipMirroring=true, since we'd like to continue to control the mirroring aspects later on in the build.

    2) Remove the call the p2.generate task after your bundles are built. While using the publisher, the metadata is generated when pde build is invoked in your headless build. So you don't need the separate generate call anymore.

    3) Switch the SDK a feature based to product based build. This isn't necessary for most projects that are switching to the publisher. However, the Eclipse SDK is a collection of artifacts for many platforms and product builds make life easier. We also had to to write a p2.inf file to include some platform specific bits into the product build since the PDE UI's product editor has support for four platforms. We build 16! There's a bug open to support more platforms in product files, hopefully this will be addressed in 3.6.

    4) Run the appropriate p2 publisher task. In our case, since we have product builds, I run the p2.publish.product task to publish metadata for the products to the repository.

    <p2.publish.product flavor="tooling" repository="file:${reposource}" productFile="${eclipse.build.configs}/sdk/builder/sdk.product" compress="true">
    <config os="win32" ws="win32" arch="x86" />
    <config os="win32" ws="win32" arch="x86_64" />
    <config os="win32" ws="wpf" arch="x86" />
    <config os="linux" ws="gtk" arch="x86" />
    <config os="linux" ws="gtk" arch="x86_64" />
    <config os="linux" ws="gtk" arch="ppc" />
    <config os="linux" ws="gtk" arch="s390" />
    <config os="linux" ws="gtk" arch="s390x" />
    <config os="linux" ws="motif" arch="x86" />
    <config os="solaris" ws="gtk" arch="sparc" />
    <config os="solaris" ws="gtk" arch="x86" />
    <config os="macosx" ws="cocoa" arch="x86" />
    <config os="macosx" ws="cocoa" arch="x86_64" />
    <config os="macosx" ws="carbon" arch="x86" />
    <config os="hpux" ws="motif" arch="ia64_32" />
    <config os="aix" ws="motif" arch="ppc" />
    <advice kind="featureVersions" file="${buildDirectory}/finalFeaturesVersions.properties" />
    <advice kind="pluginVersions" file="${buildDirectory}/finalPluginsVersions.properties" />
    </p2.publish.product>


    5) Update any custom build scripts with the publisher targets. The pde build included in 3.5 will generate this automatically for you. However, if you have bundles where build.properties indicates custom=true, you'll have to update your static build.xml files to include the new publishing parts. We use custom build scripts for our doc and swt bundles. See this patch on bug 266488 for an example. Basically, you're adding the following to the init target of your build.xml.

    <condition property="p2.publish.parts" value="true">
    <istrue value="${p2.gathering}">
    </istrue>
    </condition>

    and then this additional target.

    <target name="publish.bin.parts" depends="init" if="p2.publish.parts">
    <antcall target="gather.bin.parts">
    </antcall></target></pre><param name="destination.temp.folder" value="${build.result.folder}">

    <eclipse.gatherbundle metadatarepository="${p2.build.repo}" artifactrepository="${p2.build.repo}" buildresultfolder="${build.result.folder}" targetfolder="${build.result.folder}/${full.name}">
    </eclipse.gatherbundle>
    </target>

    After these steps are completed, the build proceeds as normal to provision the zips from the repository using the p2 director.

    Further References

    If you're contemplating switching to the publisher, these documents are useful references. Our build is quite complicated with it's 16 platforms and associated launchers, the transition for most teams will be much simpler than our scenario.

    The meta bug for the changes to reorganize the Eclipse and Equinox build to use the publisher.

    Wiki article on the p2 publisher

    Wiki article on customizing metadata

    Ian's blog entries about the publisher 1 and 2.