[UI] email notifications: non-verbose version misses critical information, verbose is too verbose and doesn't provide useful summary
I'm just starting to exploring LAVA email notifications as recently enabled on https://lite.validation.linaro.org/ . I first submitted this job: https://lite.validation.linaro.org/scheduler/job/956505 with
verbosity: verbose setting for notifications. The contents of the received email is attached: LAVA_notification_for_Test_Job_956505_lite-aeolus-micropython__1125-9d53095a.eml . It's definitely detailed, as was requested, and I may appreciate ability to check the status of each individual testcase right in the email, without need to open browser. But it lacks any useful summary of the test results, given that there're hundreds of testcases in there.
Next job I submitted is https://lite.validation.linaro.org/scheduler/job/956509, with
verbosity: verbose removed (i.e. with default verbosity setttings). Turned out, that job randomly failed due to infrastructural reasons. However, there's absolutely no indication of this in the email notification I received. This makes it rather useless, the whole idea of notifications (in my mind) is to get notified of any failure cases, so they can be acted upon promptly.
The email content can be just pasted inline:
Hello, The job with ID # 956509 is now in state Finished and health Complete. Job was submitted by ciadmin. Job details and log file: http://lite.validation.linaro.org/scheduler/job/956509 Device details: Hostname: frdm-k64f-03 Type: frdm-k64f Owner: Worker: lite-master.litelab Job details: Priority: Medium Description: lite-aeolus-micropython #1126-9d53095a Submitted: 2020-04-01 16:46:06 (+0000 UTC) Started: 2020-04-01 16:46:24 (+0000 UTC) Finished: 2020-04-01 16:46:56 (+0000 UTC) Duration: 0:00:32
Both types of emails miss exactly the same kind of information: number of passed/failed/skipped cases. For example, for "verbose" setting it could be:
Test Suite micropython_testsuite: http://lite.validation.linaro.org/results/956505/micropython_testsuite Totals: Passed: 345, Failed: 1, Skipped: 100 Test Case misc_sys_settrace_loop_py: Test skipped ...
For default setting, it could either per-testsuite totals, or cumulative totals for the entire job.