[OpenID board] Building an OpenID technical interop framework

[David Recordon]

(bcc'd general@, please reply on the public board list)

One of the consistent pieces of feedback we've received from
developers is that it's difficult to correctly create a new OpenID
Relying Party or Provider due to the lack of Foundation run
interoperability tests that help developers understand if their
implementation is correct.  While JanRain used to run a set of tests
like this on OpenIDEnabled.com, they were taken offline almost a year
ago.  The Foundation has already funded some development of
http://test-id.net/, but it focuses largely on security driven tests.

Facebook and Google are each interested in contributing $10,000 to the
OpenID Foundation to develop an easy to use technical interoperability
site for OpenID if the Foundation also contributes at least $10,000 to
the effort, the following product specification is followed, the
companies are able to collaboratively choose the contractor which will
perform the development work, and the resulting software is released
under an open source license (Apache).  We believe that the existence
of this framework will be one of the highest leverage projects in both
driving broad adoption of interoperable OpenID implementations and in
increasing the overall quality of the open source OpenID libraries.

A framework to add tests:
Just as traditional unit tests are written, the software should
support the ability to add additional tests for RPs and OPs at any
time.  Each test should be a part of a given OpenID specification with
the ability to group multiple tests together based on functionality.
Some tests can be fully automated (i.e. discovery) and others will
require human interaction (i.e. sign in).

Built like developers think:
Developers implementing OpenID think in broad strokes such as "can I
sign in?" which the framework should be built around.  There should be
two major groups of tests, one which exercises a Relying Party
implementation and one which exercises a Provider.  Upon starting the
test, the software should direct the developer through the steps which
are needed to test their implementation in a logical order such as
discovery, association, authentication, and verification.  An
individual developer should not need to know to choose the "RP
protects against association poisoning" test, but it should be done
automatically.

Supports multiple specifications:
The framework should be extensible such that a developer can choose to
test their support for individual extensions to the OpenID
Authentication protocol.  It should include tests for AX, PAPE, and
the User Experience extensions.  Ideally this framework could grow to
support other protocols such as OAuth as well.

Supports multiple environments:
The framework should support multiple environments, with the ability
to override DNS settings using the equivalent of a hosts file to
switch between environments. A standard test framework would be an
invaluable resource for RPs and OPs to test their QA environment prior
to a production release.

Results should be logged:
The software should support recording the test results of a given RP
or OP and sharing them publicly.  This could ultimately evolve into
automated smoke testing of many different OPs and RPs.

It looks nice:
Yes, we might be software engineers but let's create something which
is usable.  Matching the OpenID.net site design is a fine place to
start.

Thoughts?

Thanks,
David and Eric (Sachs)