Advise on best practice for upgrading python dependencies on a LAVA instance
Python daemons will continue using the original bytecode for the lifetime of the process. If a python dependency is upgraded during the runtime of a process, the effect of that upgrade will not be seen until the daemon(s) restart.
Document best practice that if upgrading a python package on an instance already running LAVA, especially if that package is directly listed as a dependency of LAVA, then all LAVA daemons should be restarted. (Developers will be able to stipulate which daemon is actually affected but all LAVA daemons are safe to restart without affecting running tasks. There will be a brief moment where the UI will pause but that is all.)
service lava-server-gunicorn restart service lava-logs restart service lava-publisher restart service lava-master restart service lava-slave restart
Note: this does apply to workers as well as masters but the lava-slave daemon has only minimal dependencies. Most of the work is done by lava-run which gets a new process at the start of each test job. It is NOT possible to restart lava-run - any affected test jobs will need to be resubmitted but this is considered unlikely.