LAVA Software Community Project Release Process
Introduction
This document is intended to describe the LAVA release process:
- What we release
- How we do it
Personnel
- Current LAVA Software Release Manager: ??
- Team member who also has access to the Release Signing Key: Steve
- Debian uploads: Steve
- Docker uploads: Remi/Stevan/Steve
Keys
Steve and Milosz have access to the release signing key secret key, the public key is on keyservers: A791358F2E49B100
Choose your Debian builder
If building packages for Debian upload, git-buildpackage can use a variety of builders. Most common in LAVA are either debuild or sbuild. For sbuild ensure that you override the default behaviour so that your builds include the source:
gbp buildpackage --git-builder="sbuild -s"
- or modify ~/.gbp.conf to set:
builder = sbuild -s
- For quicker builds, you may also want to set
--no-run-lintian
Build types
-
Daily master builds - Docker based images prepared from the
master branch using GitLab CI
- these are automatically included into the daily repository and copied into the snapshot directories.
-
Release tag builds - Docker based images prepared from git tags
using GitLab CI
- these are automatically included into the release repository by GitLab CI
- the release repository is then manually signed.
-
Daily Docker images
- amd64 images are built using GitLab CI
- lava-dispatcher
- lava-server
- arm64 images are built using GitLab CI on a SynQuacer or Mustang
- lava-dispatcher
- lava-server
- amd64 images are built using GitLab CI
-
Released Docker images
- Just run the script in share/release/docker-push.sh passing the tag name.
-
Debian uploads
- Built using git buildpackage and signed by the Debian Developer
- All changes for the Debian upload are to be included in the upstream debian directory.
- A tarball is downloaded from the git tag and imported into the upstream branch for gbp to use.
- Debian packaging is then signed by the DD
- Git tags for packaging changes are made and pushed using gbp.
- Built using git buildpackage and signed by the Debian Developer
Release process
- Outstanding merge requests are merged in time.
- Use the Milestone to identify issues and merge requests.
- Ensure there is time for functional testing of the release - future releases will hopefully(!) be via lavafed, but for now we also have some manual testing done.
- Agreement in the team that the release is good enough to go.
- Push
origin/master
to the master branch of your personal fork to trigger the master branch CI. Make sure all jobs pass. e.g.git push ivoire
- Create debian/changelog details with git log from last tag to master,
using the helper script:
./share/release/do-debian-release -v <release version>
- Include any Debian specific changes as everything happens in the
one repository.
- Make sure to specify the non-native version string for the
Debian version, e.g.
2019.01-1
- Make sure to use a two digit month for the release,
e.g.
2019.01
- Make a local tag on master for the eventual tag name so that the upstream code version matches the packaging version.
- Make sure to specify the non-native version string for the
Debian version, e.g.
- You can now do a local test build using the developer build
script. You can do this before creating the tag.
$ ./share/debian-dev-build.sh -s unstable
- The dev build happens in dedicated branches, so this should be clean.
- You may want to remove traces of the tarball and the build
artifacts in
../build-area/
- Release Manager makes the signed git tag on lava.git master branch
git tag -u C87D63FD935535CFB0CAF5C2A791358F2E49B100 -m "<MESSAGE>" <TAG_NAME>
- Check the tag with
git show <TAG_NAME>
andgit tag --verify <TAG_NAME>
- Push the git tag - this changes the version of all subsequent builds and starts the GitLab CI builds, as above.
- Wait for the CI to finish the builds from the tag.
- These will be used for the release archive.
- Edit the release notes of the tag to include the details of the commits, https://git.lavasoftware.org/lava/lava/tags
- Close the milestone in GitLab for this tag.
- The release repository is not signed by scripts or reprepro. The
reprepro-release.sh script will have set things up in
/home/gitlab-runner/repository/
:- The previously published tree will be in the
old-release
directory. Don't touch this yet - this is what people will still be using. - The new files will be in the
current-release
directory. Grab the Release files from here. - For each supported suite (stretch-backports and buster
currently) the RM downloads the Release files using
scp
, manually signs with theC87D63FD935535CFB0CAF5C2A791358F2E49B100
key and upload Release and Release.gpg back to the archive.- Download the Release files from the repository
(
current-release/dists/*/Release
) gpg -u C87D63FD935535CFB0CAF5C2A791358F2E49B100 -a --detach-sign Release
mv Release.asc Release.gpg
- Upload
Release.gpg
andRelease back
to the repository using scp. Be very careful with permissions here - they need to be owned by thegitlab-runner
user.
- Download the Release files from the repository
(
- The previously published tree will be in the
- Update the sym-link in
/home/gitlab-runner/repository
to change to publishing the new repo, and delete the old onecd /home/gitlab-runner/repository
ln -snf current-release release
rm -rf old-release
-
If uploading to Debian (beware of freeze guidelines etc.!)
- Download the .dsc from the release repository, using dget (will get the other source files automatically)
- Prepare a source-only upload to Debian, signed with the DD key.
- Docker image tagging & pushing to hub.docker.com
- Gitlab is building the 4 images automatically
- To push the docker image just
docker login
share/release/docker-push.sh <tag_name>
docker logout
- In order to push docker hub, you need to be a member of the lavasoftware organization.
- Send announcement email to the
lava-announce
mailing list.- Include the already-prepared release note
- append the
git log --one-line output
as included indebian/changelog
Issues to consider
- Whether to change lava-dispatcher to Architecture: all instead of a
restricted list of architectures.
- prove that we can get the right qemu-system dependencies for lava-dispatcher.
- decide which architectures get which qemu system architectures.
- Whether to continue building arm64 docker images on arm64
- https://lobradov.github.io/Building-docker-multiarch-images/
- Without an amd64 build farm, this might not give us any benefit.
Linaro support
- Copy all the files from
./lava_scheduler_app/tests/device-types/*
to the./shared/device-types/
directory of a fresh branch of an updated git clone of https://git.linaro.org/lava/lava-lab.git/ , commit all changes and all new files with a message relating to this new release and submit for review using$ git review
Background
Elements built by CI
(out of date now - please check!)
- When a git tag is pushed, the following steps are performed by the
CI:
- CI builds the master branch from the tag to create packages with
a version
lava_2018.10-1_amd64.changes
(note lack of git revision) for amd64 as artifacts in gitlab. Release Manager uses the./gitlab-api.py
script to locate the job number within the pipeline which produced the artifacts. - The same CI pipeline also builds the master branch from the tag
using the azrael arm64 gitlab runner and the -B option to create
lava-dispatcher_2018.10-1_arm64.changes
(without source) and associated files as artifacts. The wrapper script also downloads these artefacts. - CI has already run the unit tests in docker (both architectures)
- CI uses the packages to build two docker images of the release:
lava-dispatcher-amd64:2018.10
lava-server-amd64:2018.10
- Currently, the arm64 docker images need to be built manually.
lava-dispatcher-arm64:2018.10
lava-server-arm64:2018.10
- publish on
hub.lavasoftware.org
- CI builds the master branch from the tag to create packages with
a version
- Release manager then downloads the artifacts and imports into the
external reprepro using the manual signing key.
- Download and importing into reprepro can be scripted - signing
remains manual.
- The new option to repo-update.sh will allow downloading and importing into the release suite, without signing the repo. To do the import manually, check the content of the repo-update.sh script.
- The manual signing of the repository Release file (using C87D63FD935535CFB0CAF5C2A791358F2E49B100) is independent of reprepro.
- Download and importing into reprepro can be scripted - signing
remains manual.
- Release manager tags the docker images in hub.lavasoftware.org for
the lavasoftware organisation on hub.docker.com and pushes.
- Release Manager to be logged into hub.docker.com, then use docker images and docker tag commands.