[Openid-specs-ab] Conformance profile document updated

Mike Jones Michael.Jones at microsoft.com
Fri Apr 3 22:00:38 UTC 2015

Thanks for your thorough review, Garyl.  Replies inline…

From: Garyl Erickson [mailto:garyl.erickson at forgerock.com]
Sent: Thursday, April 02, 2015 8:13 PM
To: Mike Jones
Cc: openid-specs-ab at lists.openid.net; Don Thibeau; Mike Leszcz; Ian Glazer; Roshni Chandrashekhar; Eve Maler
Subject: Re: Conformance profile document updated

Hello Mike,

I do have some questions about the table in §3.1.

Regarding the Basic, Implicit & Hybrid conformance profile columns:

  1.  In the ID Token section, I'm seeing a number of tests with Test ID IdToken.verify() rather than an actual test name. Also in that section, I've not found a test in the online tool named OP-IDToken-Signature. Is this perhaps OP-IDToken-HS256, which is an Extra test online?
The tests that are function names, such as IdToken.verify(), are invoked by other tests.  They are in the spreadsheet for internal tracking purposes, so that we’re certain that all features are tested, even if not by a test specifically for that feature.  I will clarify that in the next version of the doc.
If you’re testing with dynamic registration support configured in, then OP-IDToken-Signature is replaced by OP-IDToken-RS256.  I personally think that it will simplify things for testers if for the next round of certification testing, we restructure the tool so that there are no either/or tests – just a fixed set of tests for each profile.  (Things are currently this way because some of the tests are testing multiple things, some of which don’t apply for some profiles.)  But we’re not going to change this during the lockdown period for phase 1.

  1.  In the UserInfo Endpoint section, for Test ID OP-UserInfo-Body, what does "Warning if broken" mean? Also, what is Test ID OpenIDSchema.verify()?
“Warning if broken” means that we aren’t going to require supporting the functionality for the Basic conformance profile, but we are leaving the test in place for interop testing purposes.  You can be certified if you get this warning but implementations should still aim to support the functionality.

  1.  In the scope Request Parameter section, again I've not found a test in the online tool named OP-IDToken-Signature.
Per my reply to 1, you’d be seeing this if you were using static registration.

  1.  In the Misc Request Parameters section, for Test ID OP-Req-id_token_hint, what does SHOULD mean? Should it be 'no err'?
The SHOULD means that support for this functionality is a SHOULD in the spec.  This is one of the places where the working group decided to require functionality that is designated as RECOMMENDED or SHOULD in the spec.  I agree that this should be clarified in the table.

  1.  In the OAuth Behaviors section, what is Test ID VerifyState()? Also, what do "Warning if under 30s", "OAuth MUST" and "OAuth SHOULD" mean? I'm expecting to see a simple 'y' or 'no err' (meaning a warning is ok) or a blank cell, or perhaps an * leading to a footnote explaining anything not as straightforward.
Yes, in the next version of the document, we should simplify it for consumption by testers.  The audience of the more nuanced descriptions was the working group to help them decide what we should do in each case.  But you’re right that testers don’t need or want this fine-grained information – they want yes/no values.  (See the Certification Submission Examples<http://openid.net/wordpress-content/uploads/2015/04/Certification-Submission-Examples.pdf> document for lists of actual tests presented by the test tool for different testing profiles.)

  1.  In the Client Authentication section, having separate lines for OP-ClientAuth-Basic-Dynamic and OP-ClientAuth-Basic-Static means both should pass, but the instructions elsewhere imply one or the other should be run, depending on whether dynamic registration is used by the tests. Similarly for OP-ClientAuth-SecretPost-Dynamic and OP-ClientAuth-SecretPost-Static.
This is one of the cases where related but different tests are presented by the tool, depending upon whether you’re testing with static or dynamic registration.  Again, per my answer to 1, I think that for subsequent certification phases, we should restructure these tests so none alternate.  But they do at present.  Those presented by the tool are the authoritative set.  The documents are there as supplemental information to help people understand why the test tool does what it does.
Regarding the Dynamic conformance profile column:

  1.  In the Discovery section, there are 11 lines with Test IDs that look like function names, not test names, with only 6 lines that list actual tests.
Per my answer to 1, these *are* function names. ☺  The functions are called by other tests you’ll be running, so there’s nothing extra that testers need to do for these.

  1.  In the Dynamic Client Registration section, there's a line with no Test ID. Also, should SHOULD be "no err"?
That means that there’s no test for the feature at present.  Therefore, there’s nothing you need to do for this.

  1.  In the request_uri Request Parameter section, I've not found an online test named OP-request_uri-Unsigned. (This might be a copy and paste typo of the next line for OP-request_uri-Unsigned-Dynamic (which does exist)).
This is another case where different versions of a test are presented based on the static/dynamic configuration.  No typo.
I’ll close by saying that the Certification Submission Examples<http://openid.net/wordpress-content/uploads/2015/04/Certification-Submission-Examples.pdf> document referenced from http://openid.net/certification/submission/ is the one that’s targeted at testers – not the Conformance Profiles document that you’re reviewing above.  That shouldn’t have any of the ambiguities or confusing content that you’re citing above.  If it does, please bring it to our attention.

Thanks again for the thorough review!
                                                            -- Mike

On Sun, Mar 29, 2015 at 10:51 PM, Mike Jones <Michael.Jones at microsoft.com<mailto:Michael.Jones at microsoft.com>> wrote:
I updated the conformance profile document to reflect the minor changes we’ve made to the profiles over the past month.  The new version is posted at http://openid.net/wordpress-content/uploads/2015/03/OpenID-Connect-Conformance-Profiles.pdf and linked to from the main certification page at http://openid.net/certification/.  Unless people find bugs in this version, this will be the one we go with for phase 1 of the certification program.

I’ll still working on writing the closed-form instructions to testers, and should have those for people to review within a day or so.

                                                                           -- Mike
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openid.net/pipermail/openid-specs-ab/attachments/20150403/a5eeba12/attachment-0001.html>

More information about the Openid-specs-ab mailing list