GitLab

Make worker token available through manage command

See merge request !1477

When adding a new worker, everything can be done through the command line
management command. However, you need to go to the Django admin to obtain
the token that you'll need to set up the worker.

This change prints the token upon worker creation and also prints it when
showing details about a worker.

autopkgtest: install postgresql and apache2 for tests

See merge request !1478

96a30884 and 2b276ddc have made them optional, and that's what we want
for production deployments. However the test suite requires them to be
installed locally, so they now need to be listed explicitly as test
dependencies.

Bug: Fix duplicate mount point for container share.

See merge request !1457

docker: install qemu from buster backports

See merge request !1476

Remove hard dependency on postgresql and apache2 on the same system

See merge request !1473

Also install mips, misc and ppc.
Signed-off-by: Rémi Duraffort <remi.duraffort@linaro.org>

dispatcher: download: catch stderr when decompression fails

See merge request !1474

LAVA may be setup in a way in which the gunicorn runs on one host with
a gateway/cache/TLS termination server on a separate host, for instance.

Downgrade postgresql to a Recommends, making LAVA setup more flexible.
On some installations you may want to have the database on a container
or a on a separate host entirely.

To make that possible we also fix the postinst python script to check
for database availability earlier, while checking for options, so it
can set options.db to False and avoid trying to do any database-related
setup.

Allow master/worker version mismatch

See merge request !1466

testdef: do not crash if a test step is None

See merge request !1467

doc: authentication: add notes to GitLab auth integration

See merge request !1470

lava_server: show device type when devices come back from retired

See merge request !1472

docker: allow sharing network stack of long running containers

See merge request lava/lava!1463

Query devices' status one more time to mark device type as visible in
case its devices come back from retired status.
Signed-off-by: Chase Qi <chase.qi@amd.com>

doc: install latest version docker and docker-compose

See merge request !1471

The current version '18.09.1' docker installed via 'apt install
docker.io' doesn't work with 'lava-docker-worker'. The below error
returned.

```
docker: Error response from daemon: OCI runtime create failed:
container_linux.go:344: starting container process caused "exec:
\"/dev/init\": stat /dev/init: no such file or directory": unknown.
```

While the '20.10.5' version from Docker's official repo works well with
'lava-docker-worker'

Old version docker-compose also is problematic in some cases.
Signed-off-by: Chase Qi <chase.qi@amd.com>

This can happen when adding comments into the yaml file.
See https://lkft.validation.linaro.org/scheduler/job/2444894

```yaml
- test:
    definitions:
    - from: inline
      repository:
        run:
          steps:
          - # apt update
      [...]
```
Signed-off-by: Rémi Duraffort <remi.duraffort@linaro.org>

In multi-sites environment, it can be complex to synchronize master's
upgrade with workers' upgrades.
So add a new setting ALLOW_VERSION_MISMATCH to disable the version check.

tar test definition: fix compression support

See merge request !1464

boot/pyocd: Allow to connect to device before flashing

See merge request lava/lava!1462

lpcxpresso55s69: Add support for "pyocd" boot method.

See merge request !1461

The DownloaderAction was given the wrong parameters. It's now possible to use a
compressed tar archive.
Signed-off-by: Rémi Duraffort <remi.duraffort@linaro.org>

This makes it possible to start a docker test shell container that
shares the network stack of a fvp container.

Every usage of docker that is backed by lava_dispatcher.utils.docker now
supports sharing networking stacks between container. You need to start
a long-running container, say, a QEMU device that is using docker, and
specify a name for it using the `container_name` key in the job
definition. Later in the job definition, when defining another
container, say a docker test shell one, you reference the name used for
the first container in the `network_from` key. For example:

- In the QEMU boot action:

  docker:
    image: foo/bar
    container_name: qemu

- In a docker test shell after that:

  docker:
    image: something/else
    network_from: qemu

Note that this is only supported if the original container is a
long-running container that stays running in the background after the
action where it started finishes, e.g. a QEMU device container.

We have to drop support for container hostname because that conflicts
with the --network option. Also it's pointl...

Prompted by the addition of new "connect_before_flash" option which should
be set in device type template. To make it less confusing, describe other
device type level options too: "command" and "options". This is still
seems like first time such options are described for a boot method, but
hopefully this will set a positive example to extend to other methods.
Signed-off-by: Paul Sokolovsky <paul.sokolovsky@linaro.org>

Uses "connect_before_flash" method param recently added to "pyocd" boot
method, to avoid losing serial output immediately after flashing.
Signed-off-by: Paul Sokolovsky <paul.sokolovsky@linaro.org>

This will allow to capture serial output appearing immediately after
flashing. Specific reason for this change is lpcxpresso55s69 board,
which doesn't support reset-using-uart-break feature. In other words,
how we dealt re: this problem with other boards is that we flashed it
(and it started executing, which is not good), then connected to it,
which caused reset and executing from start, and that's how we were
able to capture the full output (but potentially with extra output
in the beginning, which is again not good).

That way doesn't work with lpcxpresso55s69 - as there's no reset,
we lose some initial output, and capture only tail of it. This problem
however doesn't happen with "jlink" boot method, and the difference
is exactly that it first issues ConnectDevice sub-action, and then
Flash() action. Looking at the other boot actions, that's also how
most of them do either. "cmsis-dap" and "pyocd" for about the only
clear exceptions.

Given that, it would be tempting to just swap the order unconditionally.
However, detailed gitlog review shows that such a change was already
made on 2018-07-31 in commit 4efd8586, just to be reverted the
same day in 610b8e58. Sadly, the revert doesn't include reasons
for it, by I was able to find the related dicussion at
https://review.linaro.org/c/lava/lava/+/27685. Quoting:

> There's a very specific reason this code was written how it was.
> On some pyocd devices the serial connection doesn't exist until
> the board is flashed, or the flashing process resets the serial
> connection so if already connected it will be closed.
> This change will mean they simply no longer work.

Specific boards weren't named, but it's easily to believe that could
happen, given all the variability in board behavior we see.
So instead (and as suggested in the review discussion by the link
above), a "connect_before_flash" boot method param is introduced,
and the default is kept the same as before (i.e.
connect_before_flash = False). For now, it will be set to True for
lpcxpresso55s69, and then other boards can be tested with that setting
as time permits.
Signed-off-by: Paul Sokolovsky <paul.sokolovsky@linaro.org>

doc: authentication: add redirect URI to Gitlab provider

See merge request !1460

Add namespace to a feedback channel.

See merge request !1459

Add a new field in the message comming from the feedback channel
called "namespace".

Signed-off-by: Fathi Boudra <fathi.boudra@linaro.org>

Fix authorization issues in REST API.

See merge request !1458