<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""><div><blockquote type="cite" class=""><div class="">On 13 Dec 2016, at 16:06, Hans Zandbelt via Openid-specs-ab <<a href="mailto:openid-specs-ab@lists.openid.net" class="">openid-specs-ab@lists.openid.net</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">sorry, I missed most of this thread because it ended up in my spam folder <div class=""><br class=""></div><div class="">I am doing a thing similar to Filip: the test harness has knowledge about the expected result in the client log files, which may be an error. My test harness script is here: <a href="https://github.com/pingidentity/mod_auth_openidc/blob/master/test/oidc-rp-certification.sh" class="">https://github.com/pingidentity/mod_auth_openidc/blob/master/test/oidc-rp-certification.sh</a> and sample output is here: <a href="https://github.com/pingidentity/mod_auth_openidc/blob/master/test/oidc-rp-certification.log" class="">https://github.com/pingidentity/mod_auth_openidc/blob/master/test/oidc-rp-certification.log</a>.</div></div></div></blockquote><div><br class=""></div>I do similar things too what Hans and Filip are doing.</div><div>Since I already had a ‘test harness’ from writing the OP test suite, I’ve reused that.</div><div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class=""><div class="">I don't think it provides more transparency to standardize the test harness output log format but perhaps it makes it easier to compare across different RP implementations.</div></div></div></div></blockquote><div><br class=""></div>Yes to both !</div><div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class=""><div class="">Hans.</div></div></div><div class="gmail_extra"><br class=""><div class="gmail_quote">On Sat, Dec 10, 2016 at 1:10 PM, Filip via Openid-specs-ab <span dir="ltr" class=""><<a href="mailto:openid-specs-ab@lists.openid.net" target="_blank" class="">openid-specs-ab@lists.openid.net</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="">I've prepared example <a href="https://gist.github.com/panva/b047e176f612d817c68ca57412ffcd2a" target="_blank" class="">output</a> of the current test suite and included the tests you refer to as examples. Two outputs - one where all pass, other when two tests fail.<div class=""><br class=""></div><div class="">In essence i'm just working around a <a href="https://mochajs.org/" target="_blank" class="">test framework</a> and assert what's described in 'Expected result' actually happens, hence the current barebone output. It is possible to enrich the results with more verbose output, push these into files similar to what the RP tool exposes - per test .log, every test would output the steps and assertions that are being taken. I'm assuming others can do the same or similar.</div><div class=""><br class=""></div><div class="">I come to think a detailed verbose output of the RP is even more of an evidence of a compliant RP behavior than a screenshot of just the result. Now to come up with what's necessary in the log file to validate the behavior, a standardized format for the messages.</div><div class=""><div class=""><div class="gmail_extra"><br clear="all" class=""><div class=""><div class="m_1211671221367141445gmail_signature">Best,<br class=""><b class="">Filip Skokan</b></div></div>
<br class=""><div class="gmail_quote">On Sat, Dec 10, 2016 at 2:46 AM, Mike Jones <span dir="ltr" class=""><<a href="mailto:Michael.Jones@microsoft.com" target="_blank" class="">Michael.Jones@microsoft.com</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="EN-US" class="">
<div class="m_1211671221367141445gmail-m_8968583395387046957WordSection1"><p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif;color:rgb(31,73,125)" class="">Hans Zandbelt and I have also exchanged thoughts on this and he’d also like the option to submit RP-collected logs rather than screen shots as auditable evidence
of compliant RP behavior. I’ll work on proposed language for the instructions allowing this possibility. I’ll be looking forward to your feedback on it.<u class=""></u><u class=""></u></span></p><p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif;color:rgb(31,73,125)" class=""><u class=""></u> <u class=""></u></span></p><p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif;color:rgb(31,73,125)" class="">It seems like your test harness must have knowledge of which tests succeed by detecting negative outcomes (such as rp-id_token-bad-sig-rs256 and rp-id_token-issuer-mismatch)
and which succeed by detecting positive outcomes (such as rp-nonce-unless-code-flow and rp-token_endpoint-client_secre<wbr class="">t_basic). Could you share your categorization with the working group? Hans, you must have this information too. Can you do the same? I
plan to use this list in the updated instructions to describe how people can verify the expected outcomes of the tests.<u class=""></u><u class=""></u></span></p><p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif;color:rgb(31,73,125)" class=""><u class=""></u> <u class=""></u></span></p><p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif;color:rgb(31,73,125)" class=""> <wbr class=""> <wbr class=""> Thanks all,<u class=""></u><u class=""></u></span></p><p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif;color:rgb(31,73,125)" class=""> <wbr class=""> <wbr class=""> -- Mike<u class=""></u><u class=""></u></span></p><p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif;color:rgb(31,73,125)" class=""><u class=""></u> <u class=""></u></span></p><p class="MsoNormal"><b class=""><span style="font-size:10pt;font-family:tahoma,sans-serif" class="">From:</span></b><span style="font-size:10pt;font-family:tahoma,sans-serif" class=""> Filip [mailto:<a href="mailto:panva.ip@gmail.com" target="_blank" class="">panva.ip@gmail.com</a>]
<br class="">
<b class="">Sent:</b> Thursday, December 08, 2016 10:41 AM<br class="">
<b class="">To:</b> Mike Jones<br class="">
<b class="">Cc:</b> Roland Hedberg; <a href="mailto:openid-specs-ab@lists.openid.net" target="_blank" class="">openid-specs-ab@lists.openid.n<wbr class="">et</a></span></p><div class=""><div class="m_1211671221367141445gmail-h5"><br class="">
<b class="">Subject:</b> Re: [Openid-specs-ab] RP Certification has launched to Pilot Phase<u class=""></u><u class=""></u></div></div><div class=""><br class="webkit-block-placeholder"></div><div class=""><div class="m_1211671221367141445gmail-h5"><p class="MsoNormal"><u class=""></u> <u class=""></u></p><p class="MsoNormal">In my suite<u class=""></u><u class=""></u></p><p class="MsoNormal"><u class=""></u> <u class=""></u></p>
<div class=""><p class="MsoNormal">When the test focuses on returned data (green path) then the data presence simply being asserted by the suite. Any errors encountered during the test run resolve in the test failing to finish, outputting the failed assertion.<u class=""></u><u class=""></u></p>
</div>
<div class=""><p class="MsoNormal"><u class=""></u> <u class=""></u></p>
</div>
<div class=""><p class="MsoNormal">When the test focuses on an error being thrown by the library, the part of the code that is supposed to throw is wrapped in a try / catch, with an ensuring throw right after the statement that is expected to throw in the first place, ensuring
there's always an error thrown. In the catch block i assert the error being thrown to be the expected one together with it's message. Should the expected exception not happen, the ensuring one will and the assertion for expected message fails.<u class=""></u><u class=""></u></p>
</div>
<div class=""><p class="MsoNormal"><u class=""></u> <u class=""></u></p>
</div>
<div class=""><p class="MsoNormal">Trying to understand the screenshots that you have in mind, are you expecting a screenshot from a user-agent? Or a console log outputting the expected data/error, or something completely different?<u class=""></u><u class=""></u></p>
</div><p class="MsoNormal"><br clear="all" class="">
<u class=""></u><u class=""></u></p>
<div class="">
<div class=""><p class="MsoNormal">Best,<br class="">
<b class="">Filip Skokan</b><u class=""></u><u class=""></u></p>
</div>
</div><p class="MsoNormal"><u class=""></u> <u class=""></u></p>
<div class=""><p class="MsoNormal">On Thu, Dec 8, 2016 at 7:27 PM, Mike Jones <<a href="mailto:Michael.Jones@microsoft.com" target="_blank" class="">Michael.Jones@microsoft.com</a>> wrote:<u class=""></u><u class=""></u></p><p class="MsoNormal">I'd like to know more about how your test harness code verifies the invariants and logs that they were met. The main thing that the screen shots are trying to achieve are transparency - that anyone can verify that your implementation got
it right. If there's another way of achieving that transparency, I'm sure that the working group would entertain it. Hopefully this would be easier than having to have third parties read your test harness code.<br class="">
<br class="">
If we can simplify things for developers while maintaining transparency, I'm all for it.<br class="">
<br class="">
Your thoughts?<br class="">
<span class="m_1211671221367141445gmail-m_8968583395387046957hoenzb"><span style="color:rgb(136,136,136)" class=""> -- Mike</span></span><u class=""></u><u class=""></u></p>
<div class="">
<div class=""><p class="MsoNormal"><br class="">
-----Original Message-----<br class="">
From: Openid-specs-ab [mailto:<a href="mailto:openid-specs-ab-bounces@lists.openid.net" target="_blank" class="">openid-specs-ab-bounce<wbr class="">s@lists.openid.net</a>] On Behalf Of Roland Hedberg via Openid-specs-ab<br class="">
Sent: Thursday, December 8, 2016 8:06 AM<br class="">
To: Filip <<a href="mailto:panva.ip@gmail.com" target="_blank" class="">panva.ip@gmail.com</a>><br class="">
Cc: <a href="mailto:openid-specs-ab@lists.openid.net" target="_blank" class="">openid-specs-ab@lists.openid.n<wbr class="">et</a><br class="">
Subject: Re: [Openid-specs-ab] RP Certification has launched to Pilot Phase<br class="">
<br class="">
<br class="">
> 8 dec. 2016 kl. 13:48 skrev Filip via Openid-specs-ab <<a href="mailto:openid-specs-ab@lists.openid.net" target="_blank" class="">openid-specs-ab@lists.openid.<wbr class="">net</a>>:<br class="">
><br class="">
> Hello Mike, everyone,<br class="">
><br class="">
> in case of a library, rather than a deployment being tested, the interface provided by Roland is excellent for writing a suite like so that executes one test after the other in a ”spec” like manner, without any browser involvement, seeing how it is expected
to submit image proofs of thrown errors is the described testing not eligible for certification submission?<br class="">
<br class="">
I have a similar suite as Filip for running tests on my library against the test tool and I think Hans might also.<br class="">
So, that is definitely a reasonable, if not even the preferred, use case.<br class="">
<br class="">
> Of course it is entirely possible to rewrite the test suite to use a browser and capture the results there instead, but i think providing the codebase used for executing the tests and it's output where the executed assertions for each test are clearly marked
could serve as proof as well.<br class="">
><br class="">
> What do you think?<br class="">
><br class="">
> Best,<br class="">
> Filip Skokan<br class="">
><br class="">
> On Thu, Dec 8, 2016 at 12:17 PM, Mike Jones via Openid-specs-ab <<a href="mailto:openid-specs-ab@lists.openid.net" target="_blank" class="">openid-specs-ab@lists.openid.<wbr class="">net</a>> wrote:<br class="">
> There are now complete RP certification submission instructions at <a href="http://openid.net/certification/rp_submission/" target="_blank" class="">
http://openid.net/certificatio<wbr class="">n/rp_submission/</a> and updated example submissions showing RP certifications referenced from it at
<a href="http://openid.net/wordpress-content/uploads/2016/12/Certification-Submission-Examples.pdf" target="_blank" class="">
http://openid.net/wordpress-co<wbr class="">ntent/uploads/2016/12/Certific<wbr class="">ation-Submission-Examples.pdf</a>.<wbr class=""> This means that we’re ready to accept real RP certification submissions!<br class="">
><br class="">
><br class="">
><br class="">
> Hans, Edmund, Filip, Rich (and of course Roland) – you’ve been actively testing. I encourage you to now take the final step to submit actual RP certification applications (thereby testing the instructions). Please contact me (and possibly also Roland) if
you have any questions about the instructions or suggestions on how to make them better. All other members are likewise encouraged to likewise participate in the pilot phase, during which RP certifications are free.<br class="">
><br class="">
><br class="">
><br class="">
> A huge thanks to Roland and the early testers for getting us to this point – especially Hans and Edmund!<br class="">
><br class="">
><br class="">
><br class="">
> We’ll talk about this progress and related items on the Connect working group call in 3.75 hours…<br class="">
><br class="">
><br class="">
><br class="">
> -- Mike<br class="">
><br class="">
><br class="">
> ______________________________<wbr class="">_________________<br class="">
> Openid-specs-ab mailing list<br class="">
> <a href="mailto:Openid-specs-ab@lists.openid.net" target="_blank" class="">Openid-specs-ab@lists.openid.n<wbr class="">et</a><br class="">
> <a href="http://lists.openid.net/mailman/listinfo/openid-specs-ab" target="_blank" class="">
http://lists.openid.net/mailma<wbr class="">n/listinfo/openid-specs-ab</a><br class="">
><br class="">
><br class="">
> ______________________________<wbr class="">_________________<br class="">
> Openid-specs-ab mailing list<br class="">
> <a href="mailto:Openid-specs-ab@lists.openid.net" target="_blank" class="">Openid-specs-ab@lists.openid.n<wbr class="">et</a><br class="">
> <a href="http://lists.openid.net/mailman/listinfo/openid-specs-ab" target="_blank" class="">
http://lists.openid.net/mailma<wbr class="">n/listinfo/openid-specs-ab</a><br class="">
<br class="">
-- Roland<br class="">
"Education is the path from cocky ignorance to miserable uncertainty.” - Mark Twain<br class="">
<br class="">
<br class="">
<br class="">
______________________________<wbr class="">_________________<br class="">
Openid-specs-ab mailing list<br class="">
<a href="mailto:Openid-specs-ab@lists.openid.net" target="_blank" class="">Openid-specs-ab@lists.openid.n<wbr class="">et</a><br class="">
<a href="http://lists.openid.net/mailman/listinfo/openid-specs-ab" target="_blank" class="">http://lists.openid.net/mailma<wbr class="">n/listinfo/openid-specs-ab</a><u class=""></u><u class=""></u></p>
</div>
</div>
</div>
<div class="">
<div class="">
<div class=""><p class="MsoNormal"><u class=""></u> <u class=""></u></p>
</div>
</div>
</div>
</div></div></div>
</div>
</blockquote></div><br class=""></div></div></div></div>
<br class="">______________________________<wbr class="">_________________<br class="">
Openid-specs-ab mailing list<br class="">
<a href="mailto:Openid-specs-ab@lists.openid.net" class="">Openid-specs-ab@lists.openid.<wbr class="">net</a><br class="">
<a href="http://lists.openid.net/mailman/listinfo/openid-specs-ab" rel="noreferrer" target="_blank" class="">http://lists.openid.net/<wbr class="">mailman/listinfo/openid-specs-<wbr class="">ab</a><br class="">
<br class=""></blockquote></div><br class=""><br clear="all" class=""><div class=""><br class=""></div>-- <br class=""><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr" class=""><div class=""><div style="padding:0px;margin:0" class=""> <table style="border-collapse:collapse;padding:0;margin:0" class=""> <tbody class=""><tr class=""> <td style="width:113px" class=""> <a href="https://www.pingidentity.com/" target="_blank" class=""></a><a href="https://www.pingidentity.com/" target="_blank" class=""><img alt="Ping Identity" src="https://www.pingidentity.com/content/dam/pic/images/misc/signature/ping-logo.png" class=""></a> </td> <td class=""> <table class=""> <tbody class=""><tr class=""> <td style="vertical-align:top" class=""> <span style="color:#e61d3c;display:inline-block;margin-bottom:3px;font-family:arial,helvetica,sans-serif;font-weight:bold;font-size:14px" class="">Hans Zandbelt</span> <br class=""> <span style="display: inline-block; margin-bottom: 2px; font-family: arial, helvetica, sans-serif; font-weight: normal; font-size: 14px;" class="">Principal Solutions Architect</span> <br class=""> <span style="font-family:arial,helvetica,sans-serif;font-size:14px;display:inline-block;margin-bottom:3px" class=""><a href="mailto:hzandbelt@pingidentity.com" target="_blank" class="">hzandbelt@pingidentity.com</a></span> <br class=""></td></tr></tbody></table></td></tr><tr class=""><td colspan="2" class=""><br class=""></td></tr></tbody></table><br class=""></div></div></div></div>
</div>
_______________________________________________<br class="">Openid-specs-ab mailing list<br class=""><a href="mailto:Openid-specs-ab@lists.openid.net" class="">Openid-specs-ab@lists.openid.net</a><br class="">http://lists.openid.net/mailman/listinfo/openid-specs-ab<br class=""></div></blockquote></div><br class=""></body></html>