How LAVA uses GitLab CI
One of the major advantages of GitLab for the LAVA Software Community Project is the ability to run a wide range of CI operations in a variety of situations. LAVA Software has always tried to run the unit tests on every code change prior to merge. GitLab has allowed this to be extended to static code analysis, packaging checks and building of custom and official docker images. These tasks are specified as jobs in GitLab. Jobs are grouped into Stages and Stages are grouped into Pipelines.
Pipelines, Stages and Jobs in GitLab
Pipelines are typically initiated when a change is pushed to a branch.
Note that the CI will start to run as soon as a branch is pushed, even if a merge request has not yet been created for that branch.
Jobs are grouped into Stages, for example test
, build
, publish
.
- Each stage must complete before jobs in the next stage will start.
- Stages are ordered by the
stages
list in the.gitlab-ci.yml
file. - All jobs in each stage will attempt to start simultaneously.
There needs to be a balance of the number of stages, the number of jobs in each stage and the available resources.
Pipelines can also be initiated after a change is merged (i.e. when the
master branch is updated) and when tags are pushed. GitLab allows
control over which jobs are run for which pipelines. Some jobs can be
set to only run on merge requests (skipping master
and tags
) or
only on master
and/or only on tags
.
Finally, pipelines can be initiated on a schedule. GitLab calls these
environments
. These are designed to deploy tested software to a
public location but can also be used to run other scheduled operations.
Only environments are able to run jobs outside of a docker container
by being configured for the deploy
stage.
Control and scripting
GitLab CI is controlled by the .gitlab-ci.yml
file in the top
directory of the source code repository. This file defines the stages
and jobs within each CI pipeline. There are likely to be a lot of
changes within this file as your CI develops. It is YAML syntax, so
use comments liberally to describe each part of the file.
Each stage defines a series of jobs. The name of each job needs to
match an executable script in the same path as the job name below
the .gitlab-ci/
directory.
dispatcher-debian-9
in thetests
stage runs a script at.gitlab-ci/test/dispatcher-debian-9.sh
Jobs can also use templates to allow re-use of control blocks. Think of the templates as a YAML dictionary - whatever occurs later will replace values set earlier. Specifically, a value set in the template can be modified in a job using that template just by repeating that part of the control block and setting a new value.
Templates start with a dot in the name and provide a reference which will be used by jobs based on that template:
.job: &job
Other templates can be based on templates themselves:
.dispatcher: &dispatcher
<<: *job
This can lead to a job control definition of just a single line:
dispatcher-debian-9: *dispatcher
The .gitlab-ci/test/dispatcher-debian-9.sh
then determines how this
job will differ from other jobs using the same template.
Dependencies and Artifacts
GitLab will run most jobs in a temporary docker container which gets discarded at the end of the job. To store files for use outside the container, the job needs to define artifacts.
Other jobs can then use these artifacts by expressing a dependency on the job which builds those artifacts.
For example, a job which builds a set of binary packages from the source code may define a set of artifacts using a wildcard:
amd64/pkg-debian-9:
<<: *job
artifacts:
paths:
- _build/*.deb
A job which deploys those binary packages to a public apt repository would define a dependency on the job providing the artifacts:
dependencies:
- amd64/pkg-debian-9
Some artifacts have special meaning to GitLab and are used to create
reports which are available in every merge request. Check the GitLab
documentation for more details. For example, the LAVA Software
Community Project uses bandit
to do static code analysis for security
issues:
sast:
<<: *dind
stage: analyze
allow_failure: true
artifacts:
reports:
sast: gl-sast-report.json
Note - this is an example of a job which executes the docker
program inside the docker image used by GitLab. The sast
job uses the
dind
template (DinD stands for Docker in Docker). This is useful
when building docker images or using prebuilt docker images for
specific tasks.
Runners
Jobs are assigned to runners which are instances of gitlab-runner
,
configured to work with particular projects in GitLab using secure
tokens. Runners can be restricted to particular projects and can use
tags.