Implementing composite repos in your build

As DJ mentioned, composite repository support is included in 3.5. Our I-builds composite repo looks something like this...

compositeArtifacts.jar
compositeContent.jar
I20090210-0950/
I20090211-0900/

with a directory for each child repo such as
ls I20090211-0900/
artifacts.jar
binary/
content.jar
features/
plugins/
sdkinstaller.properties

Composite repositories allow you to identify which bundles are associated with each build in the repo. This allows you to cleanup your repositories more easily and avoid the wrath of the webmasters. How is this implemented in the build? As a neigbourhood committer would say, let me tell you a story......



One of the most important steps in our build is to create a master feature of all bundles and features that can sign it at the foundation in a single step. Once this feature is available, we extract it to a source location and publish metadata to a repo.

<p2.generator source="${reposource}" compress="true" append="true" flavor="tooling" metadataRepository="file:${repo}" artifactRepository="file:${repo}" metadataRepositoryName="${p2.repo.name}" artifactRepositoryName="${p2.repo.name}" publishArtifacts="true" p2OS="linux" mode="incremental" />

The next step is to run some packaging steps, then use the mirror application task to mirror the metadata and artifacts from the repo to the child repository. Why not just copy the bundles from the repo to the child repository instead of using the artifact.mirror task? One of the things that we want in our build is consistency. If a bundle in build A has the same version and qualifier as the bundle in build B, it should have the same content. Inconsistent bundles are scary.


How can a bundle have the same name but be different?

• New compiler changes the byte code
• New certificate changes the manifest
• Conditioning process changes the timestamp in the manifest
  
• A bundle has a compile error because of one the bundles it depends on has a bug. The dependancy is fixed, but the tag of the original bundle remains unchanged.

So we mirror the metadata and the artifacts instead of a simple copy.

<property name="buildRepo" value="${updateSite}/${buildId}">
<mkdir dir="${buildRepo}">
<p2.metadata.mirror source="file:${repo}" writemode="append" destination="file:${buildRepo}">

The artifact.mirror task has the baseline argument which refers to the existing composite repo. This means that the jars in the baseline repo will be used if the same jar exists both the baseline and the source repo. The new jars won't be used. This replicates the experience that a user will have when updating to the latest build. If the same jar already exists in their install, they won't download a new bundle.

<p2.artifact.mirror source="file:${repo}" baseline="file:${updateSite}" destination="file:${buildRepo}">
<copy todir="${buildRepo}">
<fileset file="${eclipse.build.configs}/../../extras/sdkinstaller.properties">
</fileset>

The next step is to create the composite repository
<p2.composite.artifact.repository.create location="file://${updateSite}" name="${p2.repo.name}" compressed="true" failonexists="false">
<p2.composite.metadata.repository.create location="file://${updateSite}" name="${p2.repo.name}" compressed="true" failonexists="false">

and add the child repository (current build) to the composite repository. The compositeArtifacts.jar and compositeContent.jar will have entries for the child repositories.

<children size='2'>
<child location='http://download.eclipse.org/eclipse/updates/3.5-I-builds/I20090210-0950'/>
<child location='http://download.eclipse.org/eclipse/updates/3.5-I-builds/I20090211-0900'/>
</children>


When we run the p2 director to provision the build zips, we provision from the composite repository.

Consistent metadata and artifacts ensures happiness all around.




This solution isn't perfect yet. I'll also be using the p2 team's comparator to provide warnings for certain scenarios . See bug 263272

Read more...

Eclipse 3.5 M5 now available

The 3.5M5 Eclipse and Equinox builds, as well as the new and noteworthy have been replicated to eclipse.org. The milestones p2 repo has the M5 bundles available.

M5 wasn't without drama. Late Friday afternoon, the UI team asked for a rebuild toward 3.5M5 to include a new splash screen. How could this go wrong? A rebuild with one file change. What could be simpler?

I woke up on Saturday morning confident that the build would be ready to promote. Instead of quality time with my laptop I'd be able to spend time snowshoeing with my favourite mathematician. Unfortunately, this was not to be. The build was missing the eclipse executables (eclipse.exe, eclipsec. exe etc). Hmm, I don't think this is a cosmetic bug. One file change and the executables went away? This seems like overkill.

When running the p2 director to provision the bundles for the build from the repo, there were signing errors in the .logs. I ran a test build with signing. The build had the same error. I ran a test build without signing. The executables were back. I ran jarsigner -verify on the bundles on a signed build. The junit 3.8.2 bundle had a signing error in the manifest. Interestingly enough, Denis had changed the signing certificate on eclipse.org on Friday.

At this point I suspected the problem was that the Orbit bundles were signed by the old certificate and the new bundles that we had just compiled in our build were signed in the old certificate. We resign the Orbit bundles in our build. (I've opened this bug to avoid this in the future).

Martin Oberhuber and David Williams ran a Orbit build on Sunday night which was signed by the new certificate. I ran a test build with this new build. The executables were back! Monday morning I promoted an Orbit build to Stable, and started a new build toward M5. Andrew Niefer determined that the root cause of the problem was that the junit bundle had a nested jar with a bogus signature. The end was near of this milestone saga! I could finally start releasing my patches that I had prepared for M6.

I was pinged on IRC by a certain PDE committer from Austin who will remain nameless. "Kim? Is there another build toward m5?" This PDE bug was approved by the PMC for inclusion in M5 yesterday afternoon. Another rebuild.



Here are the bugs with the details.

Promote 3.5M5
Certificate will expire within six months.
JUnit jars should exclude nested jars from signing and packing
[jarprocessor] Should we be signing nested jars?

I'll celebrate the release of M5 with a cookie. Enjoy.

Read more...