Implementing composite repos in your build

As DJ mentioned, composite repository support is included in 3.5. Our I-builds composite repo looks something like this...

compositeArtifacts.jar
compositeContent.jar
I20090210-0950/
I20090211-0900/

with a directory for each child repo such as
ls I20090211-0900/
artifacts.jar
binary/
content.jar
features/
plugins/
sdkinstaller.properties

Composite repositories allow you to identify which bundles are associated with each build in the repo. This allows you to cleanup your repositories more easily and avoid the wrath of the webmasters. How is this implemented in the build? As a neigbourhood committer would say, let me tell you a story......



One of the most important steps in our build is to create a master feature of all bundles and features that can sign it at the foundation in a single step. Once this feature is available, we extract it to a source location and publish metadata to a repo.

<p2.generator source="${reposource}" compress="true" append="true" flavor="tooling" metadataRepository="file:${repo}" artifactRepository="file:${repo}" metadataRepositoryName="${p2.repo.name}" artifactRepositoryName="${p2.repo.name}" publishArtifacts="true" p2OS="linux" mode="incremental" />

The next step is to run some packaging steps, then use the mirror application task to mirror the metadata and artifacts from the repo to the child repository. Why not just copy the bundles from the repo to the child repository instead of using the artifact.mirror task? One of the things that we want in our build is consistency. If a bundle in build A has the same version and qualifier as the bundle in build B, it should have the same content. Inconsistent bundles are scary.


How can a bundle have the same name but be different?

• New compiler changes the byte code
• New certificate changes the manifest
• Conditioning process changes the timestamp in the manifest
  
• A bundle has a compile error because of one the bundles it depends on has a bug. The dependancy is fixed, but the tag of the original bundle remains unchanged.

So we mirror the metadata and the artifacts instead of a simple copy.

<property name="buildRepo" value="${updateSite}/${buildId}">
<mkdir dir="${buildRepo}">
<p2.metadata.mirror source="file:${repo}" writemode="append" destination="file:${buildRepo}">

The artifact.mirror task has the baseline argument which refers to the existing composite repo. This means that the jars in the baseline repo will be used if the same jar exists both the baseline and the source repo. The new jars won't be used. This replicates the experience that a user will have when updating to the latest build. If the same jar already exists in their install, they won't download a new bundle.

<p2.artifact.mirror source="file:${repo}" baseline="file:${updateSite}" destination="file:${buildRepo}">
<copy todir="${buildRepo}">
<fileset file="${eclipse.build.configs}/../../extras/sdkinstaller.properties">
</fileset>

The next step is to create the composite repository
<p2.composite.artifact.repository.create location="file://${updateSite}" name="${p2.repo.name}" compressed="true" failonexists="false">
<p2.composite.metadata.repository.create location="file://${updateSite}" name="${p2.repo.name}" compressed="true" failonexists="false">

and add the child repository (current build) to the composite repository. The compositeArtifacts.jar and compositeContent.jar will have entries for the child repositories.

<children size='2'>
<child location='http://download.eclipse.org/eclipse/updates/3.5-I-builds/I20090210-0950'/>
<child location='http://download.eclipse.org/eclipse/updates/3.5-I-builds/I20090211-0900'/>
</children>


When we run the p2 director to provision the build zips, we provision from the composite repository.

Consistent metadata and artifacts ensures happiness all around.




This solution isn't perfect yet. I'll also be using the p2 team's comparator to provide warnings for certain scenarios . See bug 263272