Implement retries for HTTP notification callbacks
Currently the notification callback will only be tried once and if it did not successes (for example, connection is not perfect) it would be completely abandoned and forgotten about.
Relevant code: https://git.lavasoftware.org/lava/lava/-/blob/master/lava_scheduler_app/models.py#L2396
There are several several ways to implement retries for this.
The simple one would be to just use for _ in range loop with a break. However, there would be no backoff timer and it would block other callbacks.
The urllib3 provides a Retry class to implement request retry. It would need a little bit of boiler plate to work with requests library: https://stackoverflow.com/questions/49121365/implementing-retry-for-requests-in-python/49121508#49121508
Each callback can also be spawned in a separate cellerys shared task so that timeout on one would not block the other callbacks for the same test job.