Re-defining the Key-Value format

[Johnny Bufu]

Hi Claus,

On 28-May-07, at 3:58 PM, Claus Färber wrote:
> Johnny Bufu schrieb:
>>> Encoded for AX using Key-Value Form Encoding  (OID 2, 4.1.1.)
>>>    openid.ax.foo.uri:http://example.com/foo/100%2525pure
>>
>> AX has nothing to do directly with key-value encoding. I see no
>> reference to percent-encoding from OpenID2's section 4.1.1.
>
>> But yes, using the AX 3.3.1 Default Encoding of a String Value [1],
>> if user_name=100%pure the field in an key-value representation  
>> would be:
>>
>> 	openid.ax.foo.value=100%25pure
>
> This looks wrong. In Key-Value Form, it would be:
>
>          ax.foo.value:100%25pure
>
> (A colon, no "openid." prefix.)

You're right, of course; I wasn't paying attention to prefixes /  
separator, just to the encoding part.


> In HTTP Encoding, it would be:
>
>          openid.foo.value=100%2525pure
>
> (First encoding from AX, second encoding from HTTP Encoding.)

Ok, so we're on the same page here: two encodings done by OpenID and  
AX, and one more by the application layer above.


>>> I don't think it's a good idea to introduce a solution to the "\n"
>>> problem in AX only. It should be part of the base spec (OpenId 2
>>> Authentication).
>>
>> What do you see as pros / cons for each proposed solution?
>
> AX is not the only OpenID extension that might need to encode "\n"
> characters.
>
> If other specifications need to encode "\n" characters, it is  
> easier to
> write such specifications if the base specification (OpenID 2.0
> Authentication) provides the encoding. It is also less likely that
> writers of such specifications invent their own ad-hoc encoding (or  
> miss
> the problem at all).
>
> The same is true for binary data: If the OpenID 2.0 specification
> RECOMMENDs base64, it's less likely that authors of extension specs
> invent their own encoding (which might be incompatible with software
> that expects UTF-8 and/or produces larger messages in HTTP Encoding.)

I agree with these points; the only drawback would then be having  
another thing added to the core spec and the difference from OpenID1.


>>> What about changing section 4.1.1. from:
>>>
>>>          A message in Key-Value form is a sequence of lines.  Each
>>>          line begins with a key, followed by a colon, and the value
>>>          associated with the key.  The line is terminated by a
>>>          single newline (UCS codepoint 10, "\n"). A key or value
>>>          MUST NOT contain a newline and a key also MUST NOT contain
>>>          a colon.
>>>
>>> to (wording adapted from RFC 2822):
>>>
>>> 	A message in Key-Value form consists of fields composed of
>>>          a key, followed by a colon (":"), followed by a value, and
>>>          terminated by a single LF (UCS codepoint 10, "\n").
>>>
>>>          The key MUST be composed of printable US-ASCII characters
>>> except
>>>          ":" (i.e. characters that have values between 33 and 57, or
>>>          between 59 and 126, inclusive). The key MUST NOT start with
>>>          a '*' (codepoint 32).
>>>
>>>          The value MUST be composed of a sequence of characters
>>> encoded
>>>          as UTF-8. If an extension to this specification allows  
>>> values
>>>          that contain LF (UCS codepoint 10, "\n") characters,  
>>> these LF
>>>          (UCS codepoint 10, "\n") characters MUST be encoded as a
>>>          sequence of LF, '*', ':' (UCS codepoints 10, 42, 32,
>>> "\n*:").
>>>
>>>     [Unlike the suggested %-encoding, this encoding is compatible  
>>> with
>>>     the current spec as long as LF characters are not actually  
>>> allowed
>>>     within the value.
>>
>> What makes the proposed percent-encoding incompatible with the
>> current OpenID spec?
>
> You can't use it as an encoding for _all_ Key-Value-Form messages,
> including those already specified in the base specification, as it
> encodes the '%' character differently:
>    <openid.return_to=http://example.com/f%E4rber>
> vs.
>    <openid.ax.foo.return_to=http://example.com/f%25E4rber>.
>
> If you want to change the encoding in the base specification (which I
> want to do), it better be identical for all characters except LF.

So you were considering it for encoding / escaping the newlines in  
the OpenID spec / key-value form encoding. If we went with that, then  
no encoding (newline escaping) would be needed in AX, so there would  
be no conflict.


>>>     It's similar to the RFC 2822 folding mechanism but folding is  
>>> only
>>>     allowed (and mandated) where a LF is to be encoded. Further, the
>>>     continuation line is compatible with the key-value format,
>>> using '*'
>>>     as a pseudo key value.]
>>>
>>>          If an extension to this specification needs to allows  
>>> binary
>>>          data in values, i.e. if it allows arbitrary bytes not to be
>>>          interpreted as UTF-8 characters, it MAY use Base64
>>> [<reference>]
>>>          encoding for the specification of the format of that value.
>>
>> I would be (mildly) ok with dealing with newline escaping in the core
>> if others agree, but:
>> - it does add some extra stuff, which some may not like / approve
>
> Otherwise, the extra stuff is still there but duplicated in every
> extension that needs it. No good.
>
>> - it would add another item on the 'compatibility list', and another
>> thing that OpenID 1/2 implementations would need to deal with twice
>
> It would be an addition to the compatibility list, yes. But it  
> would not
> require different handling for OpenID 1.x and 2.0:
> Unless there's a LF character (which can't happen in OpenID 1.x), the
> Key-Value Form message will be absolutely identical.

The OpenID2 implementations (which SHOULD keep backwards  
compatibility) will still need a separate code-path at that point.

>> - not sure what would be the net advantage of having it there (aside
>> from having consistency across all extensions).
>
> Well, isn't that enough?

Well :) it depends who you're asking. I would be in favor of it, if  
others agreed fairly quickly so that we don't delay the spec even more.


>>>     [Note: Base64, is quite efficient when it comes to encoding the
>>>     message in HTTP Encoding (OID 2, 4.1.2.). Unencoded bytes would
>>> have
>>>     to use the %-encoding, rougly doubling the size. Unencoded
>>> bytes also
>>>     create problems if implementations think they should be UTF-8,
>>> e.g.
>>>     if perl strings are used.]
>>>
>>>> - base64 must be used for encoding binary data, and defined
>>>>    an additional field for this:
>>>>    	openid.ax.encoding.<alias>=base64
>>> I think it's much simpler if the specification of the field value
>>> format just says UTF-8 or Base64
>>
>> The receiving party would need to distinguish between the two
>> somehow, no? So a flag of some kind would need to be passed through.
>
> No, the _spec_ should IMO say what encoding the value is in.
> There should be no flag (just as there's no flag that says the  
> value is
> an URI.)
>
> The specification should to say what the string means and what
> encoding/format it is in, for example:
>
> - an integer (encoded with digits U+0030..0+0039)
> - human-readable text
> - a URI
> - a PNG image (encoded as Base64)
> - ...
>
> There's no need for a flag. If the spec says that ax.foo.image is a  
> "PNG
> image encoded with Base64", it is always encoded that way.

I don't think we should go that far with the AX spec; the attribute  
type is specified by the openid.ax.foo.type=<uri>, and how each  
attribute is described is specified in the metadata spec.

AX should stay simple and lightweight - the "transport" layer for  
attributes. The encodings specified in the latest draft in SVN only  
deal with OpenID core spec compliance (default string encoding) and  
provide the convenience base64 encoding for transferring binary data,  
for which full metadata is not published.


Johnny