RFC: Change default counting mechanism for pagination
This is split off from #517.
Context: Default Django pagination handling requires applying access control filters and counting the entire dataset just to determine the exact number of pages to serve. Changing default counting mechanism for pagination (or even pagination mechanism) could improve perfomance but each approach comes with its drawbacks.
"A Dumb Paginator" (always returning constant instead of counting dataset) drops counting overhead entirely, but also drops precision and breaks the UI (while small datasets might not be affected by the pagination). Estimating returned constant is another point for consideration (instead of using e.g. sys.maxsize).
Time-limited paginator covers the case of dropping precision even for small datasets. For the worst case scenario (counting exceeds time limit) page "count" will be returned after period defined by the lab admin. Choice of the constant still applies.
Keyset pagination introduction could also result in significant performance boost by no longer requiring counting pages and filtering directly by the ordered field, e.g. TestJob ID or date fields.
COUNT approximation or COUNT caching complexity seems to be too much effort compared to possible gains (especially taking into account access control filters that would have to applied as well).
Please let me know if the LAVA team would be open to allow adding time-limited paginator and keyset pagination to the main codebase as a part of REST API v0.3 (on limited subset of endpoints for start).