<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Thanks John!<div><br></div><div>To all:</div><div>Here is what struck me as actions we could take based on John's update:</div><div><br></div><div>Action: Form a work group to create a unit test API so that tests can be more automated, especially develop agreement on methods for testing the lower levels of the OpenID protocol.</div><div><br></div><div>Action: Join OSIS, review the wiki to assess how far informal interop of OpenID has progress so far. <a href="http://osis.idcommons.net/wiki/I5_User-Centric_Identity_Interop_through_RSA_2009">http://osis.idcommons.net/wiki/I5_User-Centric_Identity_Interop_through_RSA_2009</a></div><div><br></div><div>Action: Contribute your test cases to the new expanded test suite hosted at <a href="http://test-id.net/">http://test-id.net/</a>.</div><div><br></div><div>Action: Review the level of debugging, spoofing, etc. capabilities in Andrew Arnott's OpenID library since it may be a useful basis for any future automated test harness.</div><div><br></div><div>Action: It is up to the OIDF or some other organization to take on formal conformance testing if that is to happen.</div><div><br></div><div>Given all that, and the responses others have made so far. I'll try to answer my own question below and folks should yell if they think I've misinterpreted the data.</div><div><br></div><div><blockquote type="cite" class=""><span class="Apple-style-span" style="color: rgb(0, 0, 0); ">Which of the following testing methodologies sounds like the most useful at this point in time?<br>(a) Reference Implementation to test against<br>(b) Conformance test suite with logging, verbose error handling and reporting, etc.<br>(c) Just a test procedures document that clearly lays out interoperability testing per conformance "mode"<br>(d) Online coordination support for voluntary testing using (c) from above<br>(e) In-person interop testing events based on (c) above<br><br></span></blockquote></div><div><br></div><div>Answer: Hybrid of (a)+(b)+(c), a Test Procedures document with clear test steps for each feature (calling out the steps that can be automatically tested vs. those that require a real-time partner on the other end), several hosted implementations with publicly available end-points (not a single reference implementation), and at least one OpenID library optimized for automated testing (close to, but not officially a "reference implementation"). I suppose we need a little of (d) as well since OSIS work so far seems to have been enabled through online coordination support and it seems that's something that should continue. No interest raised for in-person interoperability testing.</div><div><br></div><div><blockquote type="cite" class=""><span class="Apple-style-span" style="color: rgb(0, 0, 0); ">And related to this, is there any need/demand for 3rd-party proctored interoperability testing & certification of OpenID implementations, or is all we need/want right now more support for voluntary/informal testing?</span></blockquote><br></div><div>All we know so far is that if there is a need for this, don't ask OSIS to do it. But no one has really come out and said they want this (I'd assume the requirement for this would come from RP's or OP's who are purchasing their infrastructure from a vendor). </div><div><br></div><div>Does this look like an agreeable beginning?</div><div><br></div><div><br><div apple-content-edited="true"> <span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>Brett McDowell | +1.413.652.1248 | <a href="http://info.brettmcdowell.com">http://info.brettmcdowell.com</a></div></div></span> </div><br><div><div>On Mar 17, 2009, at 10:53 PM, John Bradley wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Hi Brett,<div><br></div><div>I think I have now exceeded my daily post limit to the openID general list.</div><div><br></div><div>As the one of the OSIS organizers, and the person who has worked on the openID tests for the last two years I am happy to answer your only slightly leading question.</div><div><br></div><div>OSIS has posted openID interop tests for the last two years. All of the past results and participants are listed on the wiki.</div><div><a href="http://osis.idcommons.net/wiki/I5_User-Centric_Identity_Interop_through_RSA_2009">http://osis.idcommons.net/wiki/I5_User-Centric_Identity_Interop_through_RSA_2009</a></div><div><br></div><div>The OIDF is contributing some resources to OSIS this year that has allowed me to create over 40 new functional tests for OPs and RPs. </div><div><br></div><div>The new tests are hosted at <a href="http://test-id.net/">http://test-id.net/</a>.</div><div><br></div><div>Andrew Arnott who most of you are familiar with has been a huge help with re-factoring his library to provide detailed debugging information and the ability to perform spoofing and other tests that would not normally be included in a regular library.</div><div><br></div><div>Full debugging logs are available and the source for all the tests is linked to GitHub from each page.</div><div><br></div><div>We hope to refractor some of the tests I wrote last year to use the new test endpoints and make them easier for people to run.</div><div><br></div><div>We hope it will be a useful tool for people. It is a work in progress.</div><div><br></div><div>OSIS is NOT about compliance testing for openID or Information Cards (IMI).</div><div><br></div><div>It is a IdentityCommons project to encourage user-centric identity interoperability. </div><div><br></div><div>All of the results are public and everyone is invited to participate through the interop Wiki.</div><div><br></div><div>I have to confess though I have not finished integrating the new tests with the OSIS wiki test matrix.</div><div><br></div><div>I hope to have that finished by the end of the week. However people can start testing with the existing tests.</div><div><br></div><div>For the future, we hope to keep adding tests and reference examples for people to use.</div><div><br></div><div>I would like to see us come together to create a unit test API so that tests can be more automated.</div><div><br></div><div>However tests like checking that the OP is prompting you with a reasonable dialog in response to a AX attribute request can never be entirely automated.</div><div><div><br></div><div>Hopefully we can agree on an automated way to test the lower levels of the protocol.</div><div><br></div><div>There probably needs to be a OIDF work group to peruse what the test API will look like.</div><div><br></div><div>I do understand that there will be some people looking for some sort of conformance testing.</div><div>I don't think the goal of OSIS is to do that directly. </div><div><br></div><div>All of the tests and all of the code is available for people to review and hopefully use to improve there implementations.</div><div><br></div><div>It is up to the OIDF or some other organization to take on formal conformance testing if that is to happen.</div><div><br></div><div>So that is the story of OSIS testing to this point.</div><div><br></div><div>If people have tests they desperately want to see included, let me know an I will add them to the todo list.</div><div><br></div><div>Or better yet join OSIS and add therm yourself:)</div><div><br></div><div>Regards</div><div>John Bradley</div><div><br></div><div><blockquote type="cite"><span class="Apple-style-span" style="color: rgb(0, 0, 0); font-family: -webkit-monospace; font-size: 10px; ">Date: Tue, 17 Mar 2009 19:26:50 -0400<br>From: Brett McDowell <<a href="mailto:brett@projectliberty.org">brett@projectliberty.org</a>><br>Subject: Re: [OpenID] Backwards Compatibility<br>To: Martin Atkins <<a href="mailto:mart@degeneration.co.uk">mart@degeneration.co.uk</a>><br>Cc: "<a href="mailto:general@openid.net">general@openid.net</a>" <<a href="mailto:general@openid.net">general@openid.net</a>><br>Message-ID: <<a href="mailto:93AA4E7A-7A19-4EB8-A337-0C0D1571A774@projectliberty.org">93AA4E7A-7A19-4EB8-A337-0C0D1571A774@projectliberty.org</a>><br>Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes<br><br>Would someone more familiar with OSIS interops step in and explain <br>what enhancements you all might have planned for the future of online <br>testing, if any? I know there is some online testing now between the <br>major conference demonstrations, but I'm under the impression they are <br>far from "automatic" in nature.<br><br>If we can collect a bit more input on requirements and goals, we might <br>have the beginning of a plan we could collectively deliver to the <br>community.<br><br><br>Brett McDowell | +1.413.652.1248 |<span class="Apple-converted-space"> </span><a href="http://info.brettmcdowell.com">http://info.brettmcdowell.com</a><br></span></blockquote></div><br></div></div>_______________________________________________<br>general mailing list<br><a href="mailto:general@openid.net">general@openid.net</a><br><a href="http://openid.net/mailman/listinfo/general">http://openid.net/mailman/listinfo/general</a><br></blockquote></div><br></div><br><br><div apple-content-edited="true"> <span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>Brett McDowell | +1.413.652.1248 | <a href="http://info.brettmcdowell.com">http://info.brettmcdowell.com</a></div></div></span> </div><br></body></html>