<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Sorry John, I've been traveling too much lately and missed that you had directed a question my way on this thread (see below).<div><div><div><br></div><div>On Mar 18, 2009, at 11:24 PM, John Bradley wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0; ">The latest Liberty Interoperability Testing report and Procedures, are interesting as a reference.<br>Testing<br><a href="http://www.projectliberty.org/liberty/content/download/4435/30275/file/SAML_3Q08_%20Interoperability_PUBLIC_Final_Report_102308-FINAL.pdf">http://www.projectliberty.org/liberty/content/download/4435/30275/file/SAML_3Q08_%20Interoperability_PUBLIC_Final_Report_102308-FINAL.pdf</a><br>Test Procedures<br><a href="http://www.projectliberty.org/liberty/content/download/2273/15001/file/ID-WSF-2-0-TestProcedures-v1-01.pdf">http://www.projectliberty.org/liberty/content/download/2273/15001/file/ID-WSF-2-0-TestProcedures-v1-01.pdf</a><br><a href="http://www.projectliberty.org/liberty/content/download/4160/27946/file/Liberty_Interoperability_SAML_Test_Plan_v3.1.pdf">http://www.projectliberty.org/liberty/content/download/4160/27946/file/Liberty_Interoperability_SAML_Test_Plan_v3.1.pdf</a><br><br>However even there most of the testing appears to be manual.<br><br>Perhaps Brett knows if there are IPR free/Open Source tools that could be used to construct these sorts of Conformance and Full Matrix Tests. I am certain that there are proprietary tools. I am looking for open-source friendly tools.<br></span></blockquote></div><br></div><div>We hired some test tool development done to created "negative testing" tools specific to the SAML 2.0 events. We didn't open source those... but probably only because no one asked. I don't see why we couldn't or wouldn't. I'll take that action item.</div><div><br></div><div>But I don't imagine that is either relevant to the OpenID test or what you were asking about. We once got a quote for doing a comprehensive automated test suite but it was a long time ago and incredibly expenses. We have found that full-matrix interoperability testing of actual implementations "from the wild" provides more value to deployers at the end of the day than a master automated conformance suite would (because deployers want stuff to work with each other more than they want it to be perfectly implemented, necessarily).</div></body></html>