[Openid-specs-ab] Issue #14: Clearly include overall test result in each log entry (openid/certification)

[Michael Jones]

New issue 14: Clearly include overall test result in each log entry
https://bitbucket.org/openid/certification/issue/14/clearly-include-overall-test-result-in

Michael Jones:

Each log should clearly include whether the overall test passed, failed, etc.  I am thinking that the last line of each log file should be one of these lines:

RESULT:  PASSED

RESULT:  PASSED WITH WARNINGS

RESULT:  FAILED

RESULT:  UNKNOWN - Explanation

The "UNKNOWN" cases arises when the tester has to provide additional information that can't be determined by the test tool, such as whether the user was prompted, to determine whether the test passed or failed.

In the "UNKNOWN" case, the log should probably include instructions to the tester about how to determine whether the test was successful or not.  For instance, the result line:

RESULT:  UNKNOWN - The test passed if you were not prompted to log in

might be appropriate for OP-G-03 (Request with prompt=none when logged in) and OP-B-05 (Requesting ID Token with max_age=1000 seconds Restriction).

Likewise, for OP-G-01 (Request with prompt=login) and OP-B-04 (Requesting ID Token with max_age=1 seconds Restriction), the right result line might be:

RESULT:  UNKNOWN - The test passed if you were prompted to log in