Need ability to throttle number of concurrent jobs per worker
There are many situations where an individual LAVA worker might run out of resources (CPU, memory, temp space) if many jobs are run concurrently.
LAVA should provide a "max concurrent jobs" per worker control feature so lab admins can control the max loading of workers.
Some example use cases:
- workers with limited memory quickly hit OOM conditions with lots of g[un]zip/tar/cpio running in parallel
- worker startup when all the healthchecks run in parallel basically at the same time (causes a major spike of network traffic if the healthcheck jobs are all downloading things over TFTP, (I've seen TFTP servers fall over in this case since they are also running on the worker machine.)
- workers with a bunch of qemu "devices" which will be highly CPU bound and will need to be throttled to not kill the worker host