test: interactive: Without input command echo detection, result matching may be not reliable
This is 2nd issue I see with the current "interactive" tests implementation, after #350 (closed).
This is however more speculative, and based on my current understanding of the matter, which is in turn based on reading the docs (https://validation.linaro.org/static/docs/v2/actions-test.html?highlight=interactive#interactive-test-action), looking thru code, and of course looking at the test logs.
So, suppose we have an interactive test which feeds
echo foo style command into a device, and expects to get
foo back as a success. Now suppose that the above command leads to
bar output. I have suspicion that it still will match (pass), because the actual output captured by
pexpect will be:
echo foo bar
That's because the default behavior of the most interactive command-line UIs is to echo the input back to the user (or they wouldn't see the input typed). And of course, it's possible to match "foo" in the output above.
I see following steps of way forward with this:
- See if the issue of echo is handled on the level of
pexpect(my quick googling didn't lead to affirmation).
- If not, implement some kind of handling on the level of LAVA.
- As p.2 would represent a behavioral change of "interactive" tests, it apparently should be made optional.
- Whatever the result of p.1-3 is, the behavior should be explicitly documented, to allow users rely on it without guesswork and doubts.