Implement retries for HTTP notification callbacks
Currently the notification callback will only be tried once and if it did not successes (for example, connection is not perfect) it would be completely abandoned and forgotten about.
There are several several ways to implement retries for this.
The simple one would be to just use
for _ in range loop with a break. However, there would be no backoff timer and it would block other callbacks.
urllib3 provides a
Retry class to implement request retry. It would need a little bit of boiler plate to work with requests library: https://stackoverflow.com/questions/49121365/implementing-retry-for-requests-in-python/49121508#49121508
Each callback can also be spawned in a separate cellerys shared task so that timeout on one would not block the other callbacks for the same test job.