Skip to content

GitLab

Last edited by Steve McIntyre
Page history

releasing

LAVA Software Community Project Release Process

Introduction

This document is intended to describe the LAVA release process:

  • What we release
  • How we do it

Personnel

  • Current LAVA Software Release Manager: ??
  • Team member who also has access to the Release Signing Key: Steve
  • Debian uploads: Steve
  • Docker uploads: Remi/Stevan/Steve

Keys

Steve and Milosz have access to the release signing key secret key, the public key is on keyservers: A791358F2E49B100

Choose your Debian builder

If building packages for Debian upload, git-buildpackage can use a variety of builders. Most common in LAVA are either debuild or sbuild. For sbuild ensure that you override the default behaviour so that your builds include the source:

  • gbp buildpackage --git-builder="sbuild -s"
  • or modify ~/.gbp.conf to set:
    • builder = sbuild -s
  • For quicker builds, you may also want to set --no-run-lintian

Build types

  1. Daily master builds - Docker based images prepared from the master branch using GitLab CI
    1. these are automatically included into the daily repository and copied into the snapshot directories.
  2. Release tag builds - Docker based images prepared from git tags using GitLab CI
    1. these are automatically included into the release repository by GitLab CI
    2. the release repository is then manually signed.
  3. Daily Docker images
    1. amd64 images are built using GitLab CI
      1. lava-dispatcher
      2. lava-server
    2. arm64 images are built using GitLab CI on a SynQuacer or Mustang
      1. lava-dispatcher
      2. lava-server
  4. Released Docker images
    1. Just run the script in share/release/docker-push.sh passing the tag name.
  5. Debian uploads
    1. Built using git buildpackage and signed by the Debian Developer
      1. All changes for the Debian upload are to be included in the upstream debian directory.
      2. A tarball is downloaded from the git tag and imported into the upstream branch for gbp to use.
    2. Debian packaging is then signed by the DD
      1. Git tags for packaging changes are made and pushed using gbp.

Release process

  1. Outstanding merge requests are merged in time.
    1. Use the Milestone to identify issues and merge requests.
  2. Ensure there is time for functional testing of the release - future releases will hopefully(!) be via lavafed, but for now we also have some manual testing done.
  3. Agreement in the team that the release is good enough to go.
  4. Push origin/master to the master branch of your personal fork to trigger the master branch CI. Make sure all jobs pass. e.g.
    1. git push ivoire
  5. Create debian/changelog details with git log from last tag to master, using the helper script:
    1. ./share/release/do-debian-release -v <release version>
    2. Include any Debian specific changes as everything happens in the one repository.
      1. Make sure to specify the non-native version string for the Debian version, e.g. 2019.01-1
      2. Make sure to use a two digit month for the release, e.g. 2019.01
      3. Make a local tag on master for the eventual tag name so that the upstream code version matches the packaging version.
    3. You can now do a local test build using the developer build script. You can do this before creating the tag.
      1. $ ./share/debian-dev-build.sh -s unstable
      2. The dev build happens in dedicated branches, so this should be clean.
      3. You may want to remove traces of the tarball and the build artifacts in ../build-area/
  6. Release Manager makes the signed git tag on lava.git master branch
    1. git tag -u C87D63FD935535CFB0CAF5C2A791358F2E49B100 -m "<MESSAGE>" <TAG_NAME>
    2. Check the tag with git show <TAG_NAME> and git tag --verify <TAG_NAME>
    3. Push the git tag - this changes the version of all subsequent builds and starts the GitLab CI builds, as above.
    4. Wait for the CI to finish the builds from the tag.
      1. These will be used for the release archive.
    5. Edit the release notes of the tag to include the details of the commits, https://git.lavasoftware.org/lava/lava/tags
    6. Close the milestone in GitLab for this tag.
  7. The release repository is not signed by scripts or reprepro. The reprepro-release.sh script will have set things up in /home/gitlab-runner/repository/ :
    1. The previously published tree will be in the old-release directory. Don't touch this yet - this is what people will still be using.
    2. The new files will be in the current-release directory. Grab the Release files from here.
    3. For each supported suite (stretch-backports and buster currently) the RM downloads the Release files using scp , manually signs with the C87D63FD935535CFB0CAF5C2A791358F2E49B100 key and upload Release and Release.gpg back to the archive.
      1. Download the Release files from the repository (current-release/dists/*/Release)
      2. gpg -u C87D63FD935535CFB0CAF5C2A791358F2E49B100 -a --detach-sign Release
      3. mv Release.asc Release.gpg
      4. Upload Release.gpg and Release back to the repository using scp. Be very careful with permissions here - they need to be owned by the gitlab-runner user.
  8. Update the sym-link in /home/gitlab-runner/repository to change to publishing the new repo, and delete the old one
    1. cd /home/gitlab-runner/repository
    2. ln -snf current-release release
    3. rm -rf old-release
  9. If uploading to Debian (beware of freeze guidelines etc.!)
    1. Download the .dsc from the release repository, using dget (will get the other source files automatically)
    2. Prepare a source-only upload to Debian, signed with the DD key.
  10. Docker image tagging & pushing to hub.docker.com
    1. Gitlab is building the 4 images automatically
    2. To push the docker image just
      1. docker login
      2. share/release/docker-push.sh <tag_name>
      3. docker logout
    3. In order to push docker hub, you need to be a member of the lavasoftware organization.
  11. Send announcement email to the lava-announce mailing list.
    1. Include the already-prepared release note
    2. append the git log --one-line output as included in debian/changelog

Issues to consider

  1. Whether to change lava-dispatcher to Architecture: all instead of a restricted list of architectures.
    1. prove that we can get the right qemu-system dependencies for lava-dispatcher.
    2. decide which architectures get which qemu system architectures.
  2. Whether to continue building arm64 docker images on arm64
    1. https://lobradov.github.io/Building-docker-multiarch-images/
    2. Without an amd64 build farm, this might not give us any benefit.

Linaro support

  1. Copy all the files from ./lava_scheduler_app/tests/device-types/* to the ./shared/device-types/ directory of a fresh branch of an updated git clone of https://git.linaro.org/lava/lava-lab.git/ , commit all changes and all new files with a message relating to this new release and submit for review using $ git review

Background

Elements built by CI

(out of date now - please check!)

  1. When a git tag is pushed, the following steps are performed by the CI:
    1. CI builds the master branch from the tag to create packages with a version lava_2018.10-1_amd64.changes (note lack of git revision) for amd64 as artifacts in gitlab. Release Manager uses the ./gitlab-api.py script to locate the job number within the pipeline which produced the artifacts.
    2. The same CI pipeline also builds the master branch from the tag using the azrael arm64 gitlab runner and the -B option to create lava-dispatcher_2018.10-1_arm64.changes (without source) and associated files as artifacts. The wrapper script also downloads these artefacts.
    3. CI has already run the unit tests in docker (both architectures)
    4. CI uses the packages to build two docker images of the release:
      1. lava-dispatcher-amd64:2018.10
      2. lava-server-amd64:2018.10
    5. Currently, the arm64 docker images need to be built manually.
      1. lava-dispatcher-arm64:2018.10
      2. lava-server-arm64:2018.10
    6. publish on hub.lavasoftware.org
  2. Release manager then downloads the artifacts and imports into the external reprepro using the manual signing key.
    1. Download and importing into reprepro can be scripted - signing remains manual.
      1. The new option to repo-update.sh will allow downloading and importing into the release suite, without signing the repo. To do the import manually, check the content of the repo-update.sh script.
      2. The manual signing of the repository Release file (using C87D63FD935535CFB0CAF5C2A791358F2E49B100) is independent of reprepro.
  3. Release manager tags the docker images in hub.lavasoftware.org for the lavasoftware organisation on hub.docker.com and pushes.
    1. Release Manager to be logged into hub.docker.com, then use docker images and docker tag commands.

Resources