[OpenID] Backwards Compatibility

[Brett McDowell]

Thanks John!

To all:
Here is what struck me as actions we could take based on John's update:

Action: Form a work group to create a unit test API so that tests can  
be more automated, especially develop agreement on methods for testing  
the lower levels of the OpenID protocol.

Action: Join OSIS, review the wiki to assess how far informal interop  
of OpenID has progress so far. http://osis.idcommons.net/wiki/I5_User-Centric_Identity_Interop_through_RSA_2009

Action: Contribute your test cases to the new expanded test suite  
hosted at http://test-id.net/.

Action: Review the level of debugging, spoofing, etc. capabilities in  
Andrew Arnott's OpenID library since it may be a useful basis for any  
future automated test harness.

Action: It is up to the OIDF or some other organization to take on  
formal conformance testing if that is to happen.

Given all that, and the responses others have made so far.  I'll try  
to answer my own question below and folks should yell if they think  
I've misinterpreted the data.

> Which of the following testing methodologies sounds like the most  
> useful at this point in time?
> (a) Reference Implementation to test against
> (b) Conformance test suite with logging, verbose error handling and  
> reporting, etc.
> (c) Just a test procedures document that clearly lays out  
> interoperability testing per conformance "mode"
> (d) Online coordination support for voluntary testing using (c) from  
> above
> (e) In-person interop testing events based on (c) above
>


Answer: Hybrid of (a)+(b)+(c), a Test Procedures document with clear  
test steps for each feature (calling out the steps that can be  
automatically tested vs. those that require a real-time partner on the  
other end), several hosted implementations with publicly available end- 
points (not a single reference implementation), and at least one  
OpenID library optimized for automated testing (close to, but not  
officially a "reference implementation").  I suppose we need a little  
of (d) as well since OSIS work so far seems to have been enabled  
through online coordination support and it seems that's something that  
should continue.  No interest raised for in-person interoperability  
testing.

> And related to this, is there any need/demand for 3rd-party  
> proctored interoperability testing & certification of OpenID  
> implementations, or is all we need/want right now more support for  
> voluntary/informal testing?

All we know so far is that if there is a need for this, don't ask OSIS  
to do it.  But no one has really come out and said they want this (I'd  
assume the requirement for this would come from RP's or OP's who are  
purchasing their infrastructure from a vendor).

Does this look like an agreeable beginning?


Brett McDowell | +1.413.652.1248 | http://info.brettmcdowell.com

On Mar 17, 2009, at 10:53 PM, John Bradley wrote:

> Hi Brett,
>
> I think I have now exceeded my daily post limit to the openID  
> general list.
>
> As the one of the OSIS organizers, and the person who has worked on  
> the openID tests for the last two years I am happy to answer your  
> only slightly leading question.
>
> OSIS has posted openID interop tests for the last two years.  All of  
> the past results and participants are listed on the wiki.
> http://osis.idcommons.net/wiki/I5_User-Centric_Identity_Interop_through_RSA_2009
>
> The OIDF is contributing some resources to OSIS this year that has  
> allowed me to create over 40 new functional tests for OPs and RPs.
>
> The new tests are hosted at http://test-id.net/.
>
> Andrew Arnott who most of you are familiar with has been a huge help  
> with re-factoring his library to provide detailed debugging  
> information and the ability to perform spoofing and other tests that  
> would not normally be included in a regular library.
>
> Full debugging logs are available and the source for all the tests  
> is linked to GitHub from each page.
>
> We hope to refractor some of the tests I wrote last year to use the  
> new test endpoints and make them easier for people to run.
>
> We hope it will be a useful tool for people.   It is a work in  
> progress.
>
> OSIS is NOT about compliance testing for openID or Information Cards  
> (IMI).
>
> It is a IdentityCommons project to encourage user-centric identity  
> interoperability.
>
> All of the results are public and everyone is invited to participate  
> through the interop Wiki.
>
> I have to confess though I have not finished integrating the new  
> tests with the OSIS wiki test matrix.
>
> I hope to have that finished by the end of the week.   However  
> people can start testing with the existing tests.
>
> For the future, we hope to keep adding tests and reference examples  
> for people to use.
>
> I would like to see us come together to create a unit test API so  
> that tests can be more automated.
>
> However tests like checking that the OP is prompting you with a  
> reasonable dialog in response to a AX attribute request can never be  
> entirely automated.
>
> Hopefully we can agree on an automated way to test the lower levels  
> of the protocol.
>
> There probably needs to be a OIDF work group to peruse what the test  
> API will look like.
>
> I do understand that there will be some people looking for some sort  
> of conformance testing.
> I don't think the goal of OSIS is to do that directly.
>
> All of the tests and all of the code is available for people to  
> review and hopefully use to improve there implementations.
>
> It is up to the OIDF or some other organization to take on formal  
> conformance testing if that is to happen.
>
> So that is the story of OSIS testing to this point.
>
> If people have tests they desperately want to see included, let me  
> know an I will add them to the todo list.
>
> Or better yet join OSIS and add therm yourself:)
>
> Regards
> John Bradley
>
>> Date: Tue, 17 Mar 2009 19:26:50 -0400
>> From: Brett McDowell <brett at projectliberty.org>
>> Subject: Re: [OpenID] Backwards Compatibility
>> To: Martin Atkins <mart at degeneration.co.uk>
>> Cc: "general at openid.net" <general at openid.net>
>> Message-ID: <93AA4E7A-7A19-4EB8-A337-0C0D1571A774 at projectliberty.org>
>> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
>>
>> Would someone more familiar with OSIS interops step in and explain
>> what enhancements you all might have planned for the future of online
>> testing, if any?  I know there is some online testing now between the
>> major conference demonstrations, but I'm under the impression they  
>> are
>> far from "automatic" in nature.
>>
>> If we can collect a bit more input on requirements and goals, we  
>> might
>> have the beginning of a plan we could collectively deliver to the
>> community.
>>
>>
>> Brett McDowell | +1.413.652.1248 | http://info.brettmcdowell.com
>
> _______________________________________________
> general mailing list
> general at openid.net
> http://openid.net/mailman/listinfo/general



Brett McDowell | +1.413.652.1248 | http://info.brettmcdowell.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openid.net/pipermail/openid-general/attachments/20090318/91c08ebb/attachment-0002.htm>