development pmc release

Release Management

Prerequisites

  • To prepare or perform a release you MUST BE at least an Apache Sling Committer. Uploading a release to dist.apache.org additionally requires PMC membership, and it must happen before the artifacts are promoted to Maven Central; a non-PMC committer therefore asks a PMC member to finalize the release (the Committer CLI's [RESULT] email does this automatically).
  • Update to the most recent Sling parent POM before releasing.
  • Each release must be signed, see Appendix A below about creating and registering your key.
  • Make sure you have all Apache servers defined in your settings.xml.

Note: Listing the Apache servers in settings.xml also requires their passwords. Store them encrypted rather than in plaintext — see Password Encryption; the master password lives in $\{user.home\}/.m2/settings-security.xml.

In the past we staged release candidates on our local machines using a semi-manual process. Now that we inherit from the Apache parent POM, a repository manager will automatically handle staging for you. This means you now only need to make your GPG signing key and passphrase available to the release build in your $\{user.home\}/.m2/settings.xml.

Recent Sling and Apache parent POMs use maven-gpg-plugin 3.x, which reads the passphrase from a <server> whose id is given by gpg.passphraseServerId (default gpg.passphrase) and decrypts it via settings-security.xml. The older approach of a plain <gpg.passphrase> property in the apache-release profile is no longer used — with maven-gpg-plugin 3.x that value is passed to gpg literally (not decrypted) and signing fails.

<settings>
    ...
    <servers>
        <!-- maven-gpg-plugin 3.x reads the passphrase from this server id
             (gpg.passphraseServerId defaults to "gpg.passphrase") and
             decrypts it via settings-security.xml -->
        <server>
            <id>gpg.passphrase</id>
            <passphrase><!-- YOUR (encrypted) KEY PASSPHRASE --></passphrase>
        </server>
    </servers>
    <profiles>
        <profile>
            <id>apache-release</id>
            <properties>
                <!-- the id of your code signing key -->
                <gpg.keyname> <!-- YOUR KEY ID --> </gpg.keyname>
                <!-- avoid an interactive pinentry prompt -->
                <gpg.pinentryMode>loopback</gpg.pinentryMode>
            </properties>
        </profile>
    </profiles>
    ...
</settings>

The encrypted passphrase (and the encrypted Apache server passwords) are decrypted using the master password stored in $\{user.home\}/.m2/settings-security.xml; see Password Encryption for details.

Everything else is configured in the Sling parent POM; your module should inherit from its most recent version:

<parent>
    <groupId>org.apache.sling</groupId>
    <artifactId>sling</artifactId>
    <version><!-- the most recent release --></version>
</parent>

Releasing with the Sling Committer CLI (recommended)

The Sling Committer CLI is a Docker image that drives a release end-to-end: it closes the staging repository, verifies signatures, checksums and CI status, sends the [VOTE] and [RESULT] emails, promotes to Maven Central, updates dist.apache.org, JIRA and the Apache Reporter. It performs exactly the steps documented in the manual process below, so it is the recommended way to run a release.

One-time setup

  1. Install Docker and pull the image (or build it from the CLI repository):

    docker pull apache/sling-cli

  2. Create a docker-env file with your ASF credentials (used for Nexus, JIRA, Whimsy and the mailing lists):

    ASF_USERNAME=your-apache-id
ASF_PASSWORD=your-apache-password

Together with the GPG / settings.xml configuration from the Prerequisites above, that is all that is required. Every command runs in DRY_RUN mode by default (it only prints what it would do); append -x AUTO to actually perform the action. The examples below use this shorthand:

cli="docker run --rm --env-file=./docker-env apache/sling-cli"

1. Stage the release (Maven)

Staging is still performed with Maven, in the module you are releasing; the CLI drives everything afterwards:

mvn release:prepare
mvn release:perform

This creates the tag, builds and signs the artifacts, and uploads them to a Nexus staging repository. Note the staging repository id from the output (a line containing orgapachesling-NNNN). If you miss it, list the staging repositories — a freshly staged one is shown as [open], with the committer who staged it and its description:

$cli release list

2. Close, verify and call the vote

# close the staging repository (its description is derived from the staged POM)
$cli release close-staging -r <REPO_ID> -x AUTO

# verify PGP signatures, SHA-1/MD5 checksums and CI status
$cli release verify -r <REPO_ID>

# send the [VOTE] email (run once without -x AUTO to review it first)
$cli release prepare-email -r <REPO_ID> -x AUTO

3. Tally the votes

After at least 72 hours and three binding +1 votes, send the result email:

$cli release tally-votes -r <REPO_ID> -x AUTO

PMC membership is detected automatically from your ASF id: a PMC member's email states they will copy the release to the dist directory themselves; a non-PMC release manager's email asks a PMC member to finalize the release, since the dist upload comes first and is PMC-only.

If the vote does not pass, drop the staging repository and delete the tag, then start over with a new version (see Canceling the Release):

$cli release drop -r <REPO_ID> -x AUTO
git push --delete origin <TAG_NAME>

4. Finalize

$cli release finalize -r <REPO_ID> -x AUTO

finalize runs, in order: update dist.apache.org (only when you are a PMC member — the previous version to remove is detected automatically), promote to Maven Central, create the next JIRA version and move unresolved issues, mark the JIRA version as released, and update the Apache Reporter. The dist upload runs first because it is the only step that needs the staging repository, which promoting to Maven Central drops. A JIRA pre-flight check runs before any of these irreversible steps.

Afterwards update the website (releases / downloads / news) as described in Promoting the Release; the website diff can be generated with:

$cli release update-local-site -r <REPO_ID>

The sections below document the same steps performed manually. They are useful for understanding what the CLI does and as a fallback when it cannot be used.

Staging the Release Candidates

First prepare your POMs for release:

  1. Make sure there are no snapshots in the POMs to be released. In case you rely on a release version which is not yet promoted, you have to temporarily switch that dependency version to the release version. This might break the Jenkins CI build though, as the staged version is not yet visible to Jenkins, so revert this change after you have staged the release candidate.
  2. Check that your POMs will not lose content when they are rewritten during the release process 
    $ mvn release:prepare -DdryRun=true

    Compare the original pom.xml with the one called pom.xml.tag to see if the license or any other info has been removed. This has been known to happen if the starting <project> tag is not on a single line. The only things that should be different between these files are the <version> and <scm> elements. If there are any other changes, you must fix the original pom.xml file and commit before proceeding with the release.

  3. Publish a snapshot

    $ mvn deploy
...
[INFO] [deploy:deploy]
[INFO] Retrieving previous build number from apache.snapshots.https
...
    • If you experience an error during deployment like a HTTP 401 check your settings for the required server entries as outlined in the Prerequisites
    • Depending on the OS & the gpg version you have, you might hit https://issues.apache.org/jira/browse/MGPG-59, in which case you need, before maven command, to run gpg --use-agent --armor --detach-sign --output $(mktemp) pom.xml
    • Make sure the generated artifacts respect the Apache release rules: NOTICE and LICENSE files should be present in the META-INF directory within the jar. For -sources artifacts, be sure that your POM does not use the maven-source-plugin:2.0.3 which is broken. The recommended version at this time is 2.0.4
    • You should verify the deployment under the snapshot repository on Apache

  4. Prepare the release

    $ mvn release:clean
$ mvn release:prepare
    • Preparing the release will create the new tag in GIT, automatically checking in on your behalf

  5. Stage the release for a vote

    $ mvn release:perform
    • The release will automatically be inserted into a temporary staging repository for you, see the Nexus staging documentation for full details
    • You can continue to use mvn release:prepare and mvn release:perform on other sub-projects as necessary on the same machine and they will be combined in the same staging repository - this is useful when making a release of multiple Sling modules.

  6. Close the staging repository:

    • Login to https://repository.apache.org using your Apache credentials. Click on Staging Repositories on the left. Then click on org.apache.sling in the list of repositories. In the panel below you should see an open repository that is linked to your username and IP. Right click on this repository and select Close. This will close the repository from future deployments and make it available for others to view. If you are staging multiple releases together, skip this step until you have staged everything

  7. Verify the staged artifacts

    • If you click on your repository, a tree view will appear below. You can then browse the contents to ensure the artifacts are as you expect them. Pay particular attention to the existence of *.asc (signature) files. If you don't like the content of the repository, right click your repository and choose Drop. You can then redo (see Redoing release perform) or rollback your release (see Canceling the Release) and repeat the process
    • Note the staging repository URL, especially the number at the end of the URL. You will need this in your vote email

Redoing release perform

If perform fails for whatever reason (e.g. staged artifacts are incomplete or signed with a wrong key) drop the staging repository and create a required release.properties file containing scm.url and scm.tag manually.

Example release.properties file:

scm.url=scm:git:https://gitbox.apache.org/repos/asf/sling-org-apache-sling-settings.git
scm.tag=org.apache.sling.settings-1.3.10

Execute perform step again:

$ mvn release:perform

Redeploying staged artifacts

It may happen that deployment to Nexus fails or is partially successful. To fix such a deployment, you can re-deploy the artifacts from within the SCM checkout:

$ cd target/checkout
$ mvn deploy -Papache-release

Starting the Vote

Propose a vote on the dev list with the closed issues, the issues left, and the staging repository - for example:

To: "Sling Developers List" <dev@sling.apache.org>
Subject: [VOTE] Release Apache Sling ABC version X.Y.Z

Hi,

We solved N issues in this release:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310710&version=[YOUR JIRA RELEASE VERSION ID]&styleName=Text

Staging repository:
https://repository.apache.org/content/repositories/orgapachesling-[YOUR REPOSITORY ID]/

You can use this UNIX script to download the release and verify the signatures:
https://raw.githubusercontent.com/apache/sling-tooling-release/master/check_staged_release.sh

Usage:
sh check_staged_release.sh [YOUR REPOSITORY ID] /tmp/sling-staging

Please vote to approve this release:

  [ ] +1 Approve the release
  [ ]  0 Don't care
  [ ] -1 Don't release, because ...

This majority vote is open for at least 72 hours.

Wait for the Results

From Votes on Package Releases:

Votes on whether a package is ready to be released follow a format similar to majority approval -- except that the decision is officially determined solely by whether at least three +1 votes were registered. Releases may not be vetoed. Generally the community will table the vote to release if anyone identifies serious problems, but in most cases the ultimate decision, once three or more positive votes have been garnered, lies with the individual serving as release manager. The specifics of the process may vary from project to project, but the 'minimum of three +1 votes' rule is universal.

The list of binding voters is available on the Project Team page.

If the vote is successful, post the result to the dev list - for example:

To: "Sling Developers List" <dev@sling.apache.org>
Subject: [RESULT] [VOTE] Release Apache Sling ABC version X.Y.Z

Hi,

The vote has passed with the following result :

+1 (binding): <<list of names>>
+1 (non binding): <<list of names>>

I will copy this release to the Sling dist directory and
promote the artifacts to the central Maven repository.

If you are not a PMC member you cannot upload to dist.apache.org yourself, and it must be uploaded before the artifacts are promoted to Maven Central — so replace the closing line with a request for a PMC member to finalize the release, for example: "The release still needs to be finalized: the artifacts must first be copied to the Sling dist directory and only then promoted to the central Maven repository. As that first step requires PMC membership, which I do not have, can a PMC member please finalize this release?" (the Committer CLI generates the correct variant automatically.)

Be sure to include all votes in the list and indicate which votes were binding. Consider -1 votes very carefully. While there is technically no veto on release votes, there may be reasons for people to vote -1. So sometimes it may be better to cancel a release when someone, especially a member of the PMC, votes -1.

If the vote is unsuccessful, you need to fix the issues and restart the process - see Canceling the Release. Note that any changes to the artifacts under vote require a restart of the process, no matter how trivial. When restarting a vote version numbers must not be reused, since binaries might have already been copied around.

If the vote is successful, you need to promote and distribute the release - see Promoting the Release.

Canceling the Release

If the vote fails, or you decide to redo the release:

  1. Remove the release tag from Git (git push --delete origin ${tagName})
  2. Login to https://repository.apache.org using your Apache credentials. Click on Staging Repositories on the left. Then click on org.apache.sling in the list of repositories. In the panel below you should see a closed repository that is linked to your username and IP (if it's not yet closed you need to right click and select Close). Right click on this repository and select Drop.
  3. Remove the old version from Jira
    1. Create a new version in Jira with a version number following the one of the cancelled release
    2. Move all issues with the fix version set to the cancelled release to the next version
    3. Delete the old version from Jira
  4. Reply to the original release vote email to announce the cancellation
    1. Add [CANCELLED] to the subject line
    2. Briefly explain why the release needs to be cancelled
  5. Commit any fixes you need to make and start a vote for a new release.

Promoting the Release

If the vote passes:

  1. Push the release to https://dist.apache.org/repos/dist/release/sling/. This is only possible for PMC members (for a reasoning look at https://www.apache.org/dev/release.html#upload-ci). If you are not a PMC member, please ask one to do the upload for you.
    1. Commit the released artifacts to https://dist.apache.org/repos/dist/release/sling/ which is replicated to https://www.apache.org/dist/sling/ quickly via svnpubsub. See the section on quick artifact updates for a way to avoid having to checkout the whole folder first. The easiest to do this is to get the released artifact using the check script (check_staged_release.sh) and then simply copy the artifacts from the downloaded folder to your local checkout folder. Make sure to not add the checksum files for the signature file *.asc.*).
      • Make sure to not change the end-of-line encoding of the .pom when uploaded via svn import! Eg when a windows style eol encoded file is uploaded with the setting '*.pom = svn:eol-style=native' this would later fail the signature checks!
    2. Delete the old release artifacts from that same dist.apache.org svn folder (the dist directory is archived)
  2. Push the release to Maven Central
    1. Login to https://repository.apache.org with your Apache SVN credentials. Click on Staging Repositories. Find your closed staging repository and select it by checking the select box. Click Release from the menu above and confirm.
    2. Once the release is promoted click on Repositories on the left, select the Releases repository and validate that your artifacts are all there.
  3. Following the release promotion you will receive an email from the 'Apache Reporter Service'. Follow the link and add the release data, as it used by the PMC chair to prepare board reports. To simplify this task you can use the script from https://github.com/apache/sling-tooling-release/blob/master/update_reporter.sh. Alternatively you can add the release data directly via https://reporter.apache.org/addrelease.html?sling.
  4. Update the releases section on the website at releases.
  5. For new modules, update the download page on the website at downloads to point to the new release. For this you need to modify the according Groovy Template. For existing modules the renovate app will generate a pull request. The pull request must be manually merged.
  6. If you think that this release is worth a news entry, update the website at news

For the last two tasks, it's better to give the CDN some time to process the uploaded artifacts (15 minutes should be fine). This ensures that once the website (news and download page) is updated, people can actually download the artifacts.

Quick update of artifacts in dist

It is possible to update the artifacts without needing to checkout or update the full dist folder, which can be quite slow.

Assuming that we are releasing org.apache.sling.engine 2.6.22 and the old version artifact names start with org.apache.sling.engine-2.6.20, we can run the following commands

$ cd <folder where 2.6.22 is found>
$ curl -fsSL https://raw.githubusercontent.com/apache/sling-tooling-release/refs/heads/master/update_dist.sh \
        | bash -s -- org.apache.sling.engine 2.6.20 2.6.22

This makes sure that the new artifacts are imported and the old ones are deleted.

Update JIRA

Go to Manage Versions section on the SLING JIRA and mark the X.Y.Z version as released setting the release date to the date the vote has been closed.

Also create a new version X.Y.Z+2, if that hasn't already been done.

And keep the versions sorted, so when adding a new version moved it down to just above the previous versions.

Close all issues associated with the released version.

Update the Sling Starter Module

If the new release should be included in the Sling Starter, please create PR after the artifacts are available on Maven Central. That usually happens within 60 minutes of the staging repository being closed.

If the released module was already included in the Sling Starter, a pull request will be generated by the renovate app, you can approve it as soon as the automated checks pass.

Create an Announcement

We usually do such announcements only for "important" releases, as opposed to small individual module releases which are just announced on our news page.

To: "Sling Developers List" <dev@sling.apache.org>, "Apache Announcements" <announce@apache.org>
Subject: [ANN] Apache Sling ABC version X.Y.Z Released

The Apache Sling team is pleased to announce the release of Apache Sling ABC version X.Y.Z

Apache Sling is a web framework that uses a Java Content Repository, such as Apache
Jackrabbit, to store and manage content.  Sling applications use either scripts or
Java servlets, selected based on simple name conventions, to process HTTP requests
in a RESTful way.

<<insert short description of the sub-project>>

https://sling.apache.org/site/apache-sling-ABC.html

This release is available from https://sling.apache.org/site/downloads.cgi

Building from verified sources is recommended, but convenience binaries are
also available via Maven:

<dependency>
    <groupId>org.apache.sling</groupId>
    <artifactId>org.apache.sling.ABC</artifactId>
    <version>X.Y.Z</version>
</dependency>

Release Notes:

<<insert release notes in text format from JIRA>>

Enjoy!

-The Sling team

Important: Add the release to the Software section of the next board report below Reports.

Related Links

  1. https://www.apache.org/dev/release-signing.html
  2. https://wiki.apache.org/incubator/SigningReleases

Releasing the Sling IDE Tooling

The IDE tools for Eclipse are released using a different process. The artifacts are uploaded to the staging dist area and not published as Maven artifacts.

The instructions assume that you plan to release version 1.3.0 and that the previous release was 1.2.2.

Assuming that you plan to release version 1.3.0, you need to check out the project, ensure no local changes are present, and run

make release RELEASE_VERSION=1.3.0 NEXT_VERSION=1.3.1-SNAPSHOT

At the end of the execution the artifact will be uploaded to https://dist.apache.org/repos/dist/dev/sling/ide-tooling/1.3.0 .

At this point you can call the vote.

The format of the release vote should be

To: "Sling Developers List" <dev@sling.apache.org>
Subject: [VOTE] Release Apache Sling IDE Tooling version X.Y.Z

Hi,

We solved N issues in this release:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310710&version=[YOUR JIRA RELEASE VERSION ID]

Staging repository:
https://dist.apache.org/repos/dist/dev/sling/ide-tooling/X.Y.Z/

You can use this UNIX script to download the release and verify the signatures:
https://raw.githubusercontent.com/apache/sling-tooling-release/master/check_staged_release.sh

Usage:
sh check_staged_release.sh eclipse-1.3.0 /tmp/sling-staging
    
Please vote to approve this release:

  [ ] +1 Approve the release
  [ ]  0 Don't care
  [ ] -1 Don't release, because ...

This majority vote is open for at least 72 hours

Once the release has passed, the following must be done:

  1. announce the result of the vote, see Wait for the results
  2. update versions in jira, see Update JIRA
  3. upload p2update.zip (with signatures and checksums) to https://dist.apache.org/repos/dist/release/sling
  4. upload unzipped update site to https://dist.apache.org/repos/dist/release/sling/eclipse/1.3.0
  5. upload the source bundle to https://dist.apache.org/repos/dist/release/sling/eclipse/1.3.0
  6. update https://dist.apache.org/repos/dist/release/sling/eclipse/composite{Content,Artifacts}.xml to point version 1.3.0 instead of 1.2.2
    1. The timestamps in the composite xml files should be refreshed to "now", for instance by using the value of echo "`date +%s`000"
  7. remove the old update site from https://dist.apache.org/repos/dist/release/sling/eclipse/1.2.2/ versions but leave pointers to archive.apache.org, using compositeArtifacts.xml/compositeContent.xml , with a single child entry pointing to https://archive.apache.org/dist/sling/eclipse/1.2.2/
  8. remove the staged artifacts from https://dist.apache.org/repos/dist/dev/sling/ide-tooling-1.3.0
  9. update the news page and the download pages
  10. update the Eclipse Marketplace listing
    1. we do not maintain multiple 'solution versions' so just update the version and make sure that supported Eclipse releases and minimum Java version are correct

Appendix A: Creating and registering your PGP key

Each Sling release must be signed, and the corresponding keys must be available at https://downloads.apache.org/sling/KEYS .

This page only provides minimal information, the canonical reference for this is the ASF Infrastructure Release Signing page.

Assuming you are using a *nix system with a working OpenSSH, GnuPG, and bash you can create and add your own key with the following commands:

  1. Create a public/private pair key:

    $ gpg --gen-key

    When gpg asks for e-mail linked the key you MUST USE the <committer>@apache.org one. When gpg asks for comment linked the key you SHOULD USE "CODE SIGNING KEY"

  2. Add your public key to https://downloads.apache.org/sling/KEYS by adding it via SVN to https://dist.apache.org/repos/dist/release/sling/KEYS. This is only possible for PMC members (for a reasoning look at https://www.apache.org/dev/release.html#upload-ci). If you are not a PMC member, please ask one to do the upload for you. The actual update can be achieved e.g. via

    $ svn checkout https://dist.apache.org/repos/dist/release/sling/ sling --depth empty
$ cd sling
$ svn up KEYS

Add the public key to KEYS file with your favourite editor and afterwards

    $ svn commit -m "my key added" KEYS
  1. It's also good to upload your key to a public key server, see the ASF Infrastructure Release Signing page for more info.

Appendix B: Deploy Maven plugin documentation (if applicable)

When releasing a Maven plugin, the Maven-generated documentation published under https://sling.apache.org/components/ needs to be updated.

To publish the plugin documentation execute the following steps after the release:

  1. Checkout the release tag of the released plugin (or reset your workspace)

  2. Build the Maven site of the plugin locally.

    $ mvn clean site:site

  3. Checkout the Sling website

    $ git clone https://github.com/apache/sling-site.git

  4. Switch to branch asf-site

  5. Replace the content of the existing folder components/<plugin-name> with the generated Maven site from target/site

  6. Create a new folder src/main/jbake/assets/components/<plugin-name>-archives/<plugin-name>-<version> and copy the generated maven site there as well

  7. Commit the changes

  8. Check the results at https://sling.apache.org/components/

- ( Release Management )