Ben Konrath
<ben a-t bagu d-o-t org>
Phil Muldoon
<pmuldoon a-t redhat d-o-t com>
Most open source projects provide a source release of their software that has the ability to be compiled into a runnable binary. Currently there is no standard way for Eclipse plugins to generate this type of source drop. In this document I will discuss the problems that the Red Hat Eclipse team is experiencing building and packaging non-SDK plugins. I will also propose an enhancement to the pde.build plugin that will allow features to generate buildable source drops, therefore avoiding most of the problems we are experiencing.
The Eclipse SDK provides a uniform build process for plugins that are not part of the Eclipse SDK. This process integrates fetching the plugin's source from cvs, generating the plugin's build files, running the build from these generated files and finally packing up the compiled plugins into a distributable zip file. It seems that this build process was designed for developers who have control over the cvs servers that host their plugins and wish to easily make binary plugins.
The build process seems to work well for its designed use case, but it doesn't work well for Linux distributors partly because we are attempting to build plugins in a secure environment. Fetching from pserver cvs at build time opens the door to a possible "person in the middle" style security attack. Making cvs part of the build can introduce code that is not auditable to the build environment because not all projects tag their cvs repositories correctly. We realize that the SDK plugins are tagged consistently, but this is no the case for all plugins on eclipse.org. A recent example is of this happened with the CDT's 3.0M6 release - binary RC builds were available but there was no way to reproduce them because cvs was not tagged yet. We had to use a version fetched from HEAD as a substitute until the repository was tagged.
Another problem with this is that the build process largely dependent on a network connection being present and the cvs server being available. It would be much better if someone wanting to build the plugin could do so with an SDK and buildable plugin shipped to them on CD. Remember, not everyplace in the world has accessible broadband internet.
To work around these problems, the plugin's source is pre-fetched by calling the fetch target of the build.xml in a feature's releng plugin. These pre-fetched files are then tarred up into a custom source tarball. At build time, cvs access is disabled and the plugin is compiled from the pre-fetched tarball.
At first glance, this pre-fetch procedure seems to have solved most of our problems, but unfortunately this is not the case. For one, the security problem has only been moved from build time to pre-fetch time. This is because the source is still fetched directly from pserver cvs as opposed to using an upstream source package that can be verified with md5sums or gpg signatures. While Eclipse.org does not currently provide a verification process, having buildable source drops is the first step to getting such a process.
Another problem with this pre-fetch method is that it does not address the fact that cvs must be tagged in order to reproduce a build. As mentioned previously, this can result in having binary builds of plugins that cannot be audited or reproduced. In my opinion, this should be a concern for other members of the Eclipse community as well.
One final problem with this method is that it is distributed across several build steps, and requires heavy engineer intervention at several stages. This work difficult to automate across different versions of the same plugin because the releng plugins tend to change if a plugin or feature is under active development. If source drop generation were part of the releng process, we would avoid this time consuming pre-fetch process. Also, other Linux distributions could use the same source drops instead of pre-fetching the source tarballs on their own.
To generate the buildable source drops, I propose creating the zip the between the "fetch" and "generate" stage of pde.build/scripts/build.xml. This can be implemented by using a conditional antcall that is only run if the "buildSourceDrop" property is set in the script that calls pde.build/scripts/build.xml. Only features that want to generate source drops would set this property.
At this point the zip would contain only the Java source and would be missing a main build.xml file that is required to drive the compilation. I propose that a custom ant task be created that will generate this missing build.xml. This file would call the eclipse.buildScript ant task to generate the feature and plugin build.xml files. It would then call the feature's build.xml file that was just generated and finally it would call the assemble script to make the zip containing the binary plugin.
I should point out that the Eclipse SDK "srcIncluded" drops are generated with logic in the "postGenerate" target of its customTargets.xml. The "srcIncluded" drop must include the build.xml files that were just created in the "generate" target because it needs to be able build without an installed SDK. Plugins not included with the SDK are expected to have a pre-installed SDK which can generate the build.xml files. This proposal is consistent with this expectation in that the buildable source drops would not contain pre-generated build files for the plugins themselves.
One problem with the proposed solution is that native code in fragments will not be built automatically. This is because the Eclipse build process does not build native code as part of a regular plugin build. The native code will have to be explicitly built by the packager or user wanting to build the feature on their system. I have filed a bug (71637) about this shortcoming and have posted a patch that is awaiting comments. Hopefully this will be solved in the near future.
Another problem with this solution is that it only works for plugins that have an associated releng plugin. There are many open source plugins that don't have the need for the functionality that a releng plugin provides. A different solution should investigated for this case. My initial thought is that a "generate source drop" menu item should be added to the PDE tools menu that is visible when a user right clicks a the plugin.xml. This would use some of the mechanisms proposed in this document to create the buildable source drop.
I have started work on this proposal, but I it is not finished. Before I
proceed, I would like to get comments from some Eclipse pde.build and releng
developers to make sure that this proposal is acceptable. The patch can be
found here:
http://www.bagu.org/eclipse/org.eclipse.pde.build-buildable-source-drops.patch
Comments are appreciated.