test: interactive: Not reliable test failure detection
While playing with "interactive" tests, I found that with multi-command scripts, it's possible that some command will fail, but the entire action won't be marked as failed.
The apparent reason for this is that the action implementation starts with a
fail status line 116, and then sets it to
pass line 172. Moreover, the status is again unconditionally set to
pass at the end line 121.
But that's backwards. Instead, the logic should be that the action is success, unless it fails, and a single intermediate failure somewhere means that the entire action is a failure, regardless of the number of intermediate successes it may have.
A few caveats/notes:
- This is a preliminary report I submit as "FYI" to avoid situation like with #349 (closed), to hopefully get a quick comment if I miss something with "interactive" tests.
- Based on my last week's playing, I don't have a testcase right away, would need to reproduce it again.
- In the code pointers above, I imagine that
resultsoperated there may be results of different levels (e.g. single command vs single script vs entire action). What I'm sure is that the logic should be as described above - failure of any of intermediate step should propagate as a failure of entire action, and be recorded in the test results as such.
- I have a preliminary patch, but due to all the above, so far submitting this ticket for initial vetting, and hope to do more homework (at least the repro case) before submitting a patch.