RFC: Add benchmarks to the CI pipeline
This is follow-up to #521.
Context: LAVA documentation provides advice for growing the lab. Recent patches that reduce SQL query count (/scheduler, /results) include information on efficiency improvements. Adding performance tracking to the CI pipeline could help estimating the impact of the future patches on system's scalability.
Steps to add benchmarks:
- Mock environment closely resembling predicted workload (in progress as #521; still needs tuning)
- Store mocked environment for reuse (see details below)
- Load mocked environment for benchmark run (see details below)
- Execute benchmarks in the mocked environment (or its temporary copy; see dummy-db-api-benchmark)
Storing mocked environment might be useful due to the time required for its generation (~1.5h) compared to the time needed to load pre-generated one (under 5 minutes). After initial tuning it would probably rarely change (unless there are changes to the database schema). It would be convinent to have this environment generated by the CI pipeline but high cost of this operation calls for strict rules when the environment should be regenerated.
Loading mocked environment could be done in a similar manner to the official PostgreSQL image init files processing or built in ci-images. The latter option would be easier to implement but would break tools/data separation on containers in use.
Please let me know what is the LAVA team's view at this topic.