Subject: Re: Status of Bugzilla Bug 10, Round-tripping various information in properties

From: Geoffrey M Clemm <[email-masked]>
Date: Thu, 22 Dec 2005 13:25:30 -0500

There were several postings (most recently, one a couple of weeks ago)
that explained why escaping the XML as #PCDATA has a variety of problems.
For example:
<http://lists.w3.org/Archives/Public/w3c-dist-auth/2005OctDec/1056.html>
So until a non-problematic solution is proposed, I stick by my
preference for a recommendation that servers preserve namespaces
(namespace preserving parsers are not that hard to come by, so I
don't see this as an unreasonable recommendation).

Cheers,
Geoff

Wilfredo Sánchez Vega <[email-masked]> wrote on 12/22/2005 01:12:23 
PM:

>    For what it's worth, I talked to Greg Stein about this at 
> ApacheCon, and his position is that namespaces will not be preserved 
> by mod_dav, insofar as they are not semantically meaningful to a 
> namespace-aware XML parser.  The presence of other specifications 
> which add semantic meaning to then doesn't change that.
> 
>    This is basically the position I was taking.
> 
>    I (and I believe Greg) don't think this belongs in the spec, even 
> as a SHOULD.  An explicit recommendation that clients sent properties 
> as #PCDATA still makes a lot more sense to me.
> 
>    I have yet to hear a reason why serializing the property as 
> #PCDATA isn't an acceptable solution for whose who wish to avoid 
> forcing the server parse (and therefore interpret) the property value.
> 
>    -wsv
> 
> 
> On Dec 22, 2005, at 8:24 AM, Geoffrey M Clemm wrote:
> 
> > (1) I support adding an explicit recommendation that servers
> > preserve namespace prefixes in property values.  I'm neutral
> > on whether we use the term "SHOULD" in that recommendation.
>

---

From: Julian Reschke <[email-masked]>
Date: Thu, 22 Dec 2005 19:44:17 +0100

Hm,

Wilfredo's mail didn't arrive here (yet).

Geoffrey M Clemm wrote:
> Wilfredo Sánchez Vega <[email-masked]> wrote on 12/22/2005 01:12:23 PM:
> 

Yes, they are. This has been explained numerous time. But anyway:

- there's no W3C spec that would license an XML processor to throw away 
prefixes

- there are multiple W3C specs that use prefixes in attribute values (or 
even text content), including XSLT and XML Schema

So please everybody try to understand the issue before claiming prefixes 
aren't relevant. In *most* cases they aren't. In *some* cases they are.

>  > forcing the server parse (and therefore interpret) the property value.

A property value serialized as #PCDATA (thus as escaped XML) is 
something else than a property value serialized as XML. If you control 
the format, such as when you define the property in a spec, you sure 
have the freedom to say it's text, instead of XML. But this requires 
that senders and recipients agree on that. But in general, a client 
doesn't have that choice.

Furthermore, putting escaped XML into property values *will* have 
negative effects on generic clients (which will display angle brackets 
to the user) and some protocol extensions (such as DASL).

Best regards, Julian

---

From: Wilfredo Sánchez Vega <[email-masked]>
Date: Thu, 22 Dec 2005 18:50:47 -0800

> Wilfredo's mail didn't arrive here (yet).

   Sorry, I sent it from my apple.com account, and I haven't figured  
out how to subscribe to a W3C list without delivery (so it can post  
without moderation).

> - there's no W3C spec that would license an XML processor to throw  
> away prefixes

   I showed you an example of python code that parses XML and spits  
it right back out and the namespace prefixes were edits.  You said  
that wasn't a bug in PyXML.

   If PyXML is not a compliant XML library because it rewrites the  
prefixes, then I agree completely.  PyXML is broken and should be  
fixed, and we should move on.  But if that's not true, I don't think  
a server write should have switch to another library or do some  
haquery in order to preserve them.

   Geoff asserts that namespace preserving parsers are easy to come  
by.  That may be true for Geoff, and maybe he has a lot of  
flexibility in parser choices because he has no other such  
constraints.  I don't think that's true for everyone.

   Furthermore, in the case where a namespace is defined outside of  
the property container, some rewriting is required in order to  
maintain the association with the target URI.  So now we have  
language in 4.4 saying that changing the property container's prefix,  
unlike it's contents, are not significant, and suggest in 4.4.1  
suggesting that we should rewrite that tag to put all of the  
namespaces in there (seems like that should be explicitly stated in  
4.4 as a SHOULD as well, for consistency, unless we think tossing the  
namespace declarations out is OK, which is the opposite of what I'd  
expect).

   Aside from the mild weirdness of "it's significant, except for  
over here"... now it's not just good enough to use a parser that can  
preserves prefixes, but also I need it to let me specify that I can't  
edit them, "except for this in one element, where I need you to do  
the rewriting that you may otherwise have elsewhere."  What DOM  
methods exist for that sort of thing?

> - there are multiple W3C specs that use prefixes in attribute  
> values (or even text content), including XSLT and XML Schema

   And I'm all for servers that care about enabling the use of these  
specs bending over to accommodate them, but I don't think that other  
servers need to care.

   Additionally:

     http://www.w3.org/TR/REC-xml-names/

   Section 3: "Note that the prefix functions only as a placeholder  
for a namespace name." (Note "only" is emphasized in the text...)

   It's not unreasonable to think that having parsed that data, when  
rendering it back out, the placeholder might be relabeled without  
negative consequences.  Regardless of that assumption, that sentence  
makes it pretty clear than using the prefix for any other purposes is  
inconsistent with REC-xml-names.

> something else than a property value serialized as XML. If you  
> control the format, such as when you define the property in a spec,  
> you sure have the freedom to say it's text, instead of XML. But  
> this requires that senders and recipients agree on that. But in  
> general, a client doesn't have that choice.

   If you define the property, senders and receivers always have to  
agree to honor your definition.  That's not unique here.  To me  
that's an argument for saying that such definitions SHOULD use #PCDATA.

   Geoff, I'm not following your argument that some clients might not  
de-serialize the data.  If you have XML there, you have structured  
data.  Either the client understands the property and does handles it  
properly or it doesn't.

> negative effects on generic clients (which will display angle  
> brackets to the user) and some protocol extensions (such as DASL).

   What's a generic client going to show the user for structured XML  
data that it doesn't understand?  Generic client tends to refer to a  
tool for l33t haxx0rz.

   Perhaps I shouldn't get all bent out of shape over a SHOULD.

	-wsv

---

From: Julian Reschke <[email-masked]>
Date: Fri, 23 Dec 2005 10:31:48 +0100

Wilfredo Sánchez Vega wrote:
> ...
>   I showed you an example of python code that parses XML and spits it 
> right back out and the namespace prefixes were edits.  You said that 
> wasn't a bug in PyXML.

*Parsers* are not allowed to throw away prefixes. *Serializers* are 
separate specs, and have been neglected by the W3C for quite some time. 
I think DOM Level 3 talks about serialization.

> prefixes, then I agree completely.  PyXML is broken and should be fixed, 
> and we should move on.  But if that's not true, I don't think a server 
> write should have switch to another library or do some haquery in order 
> to preserve them.

That's hard to tell unless PyXML claims to conform to some particular 
specification that actually defines serialization.

>   Geoff asserts that namespace preserving parsers are easy to come by.  
> That may be true for Geoff, and maybe he has a lot of flexibility in 
> parser choices because he has no other such constraints.  I don't think 
> that's true for everyone.

Didn't we find out last time that it's not the parser (input), but the 
serializer (output)?

>   Furthermore, in the case where a namespace is defined outside of the 
> property container, some rewriting is required in order to maintain the 
> association with the target URI.  So now we have language in 4.4 saying 
> that changing the property container's prefix, unlike it's contents, are 
> not significant, and suggest in 4.4.1 suggesting that we should rewrite 
> that tag to put all of the namespaces in there (seems like that should 
> be explicitly stated in 4.4 as a SHOULD as well, for consistency, unless 
> we think tossing the namespace declarations out is OK, which is the 
> opposite of what I'd expect).

The spec doesn't say where the namespace declarations need to sit, as 
long as they're there. And no, I wouldn't want to mandate this, because 
that reallys seems to be irrelevant.

>   Aside from the mild weirdness of "it's significant, except for over 
> here"... now it's not just good enough to use a parser that can 
> edit them, "except for this in one element, where I need you to do the 
> rewriting that you may otherwise have elsewhere."  What DOM methods 
> exist for that sort of thing?

What does this have to do with editing?

>> - there are multiple W3C specs that use prefixes in attribute values 
>> (or even text content), including XSLT and XML Schema
> 

So you'd be happy would be "SHOULD" be a "MAY"?

>   Section 3: "Note that the prefix functions only as a placeholder for a 
> namespace name." (Note "only" is emphasized in the text...)

Too bad that other W3C specs have started to use the prefix anyway, and 
thus XML Infoset also considers it significant.

> inconsistent with REC-xml-names.

And that means that the world of W3C XML specs internally is 
inconsistent. Yes. I know. Everybody knows. There's nothing I can do 
about it.

The whole discussion started because the draft made the incorrect 
statement that prefixes do not need to be preserved because they are 
insignificant. Well, they aren't for certain applications, so this 
needed to be fixed.

>   If you define the property, senders and receivers always have to agree 
> to honor your definition.  That's not unique here.  To me that's an 
> argument for saying that such definitions SHOULD use #PCDATA.
> 

There are generic clients out there that display *every* property. How 
are they supposed to know that some text indeed consists of escaped XML?

>   What's a generic client going to show the user for structured XML data 
> that it doesn't understand?  Generic client tends to refer to a tool for 
> l33t haxx0rz.

In general, I'd expect them to only display the text content.

>   Perhaps I shouldn't get all bent out of shape over a SHOULD.

:-)

Best regards, Julian

---

From: Geoffrey M Clemm <[email-masked]>
Date: Fri, 23 Dec 2005 09:31:06 -0500

> that's an argument for saying that such definitions SHOULD use #PCDATA.

I agree that if the property definer defines the property value as being
escaped XML text, then all clients that read that property will know that
they have to explicitly apply their XML parser to the property value
(after the PROPFIND body has been parsed once by their WebDAV client 
library),
and that all clients that write that property will know that they have to
explicitly serialize their XML and wrap it with a #PCDATA wrapper).

My point was just that this decision has to be made by the property 
definer, not by individual clients, so that all property readers
and writers know that they have to explicitly do the parsing and
unparsing, i.e. you can't have one client deciding it is going to
escape the XML to preserve namespaces, and other clients not doing so.

But also note that all you've really done is pushed the problem to the 
client,
because now each client has to be sure to use a prefix-preserving parser
when they parse the string, and use a prefix-preserving serializer
when they serialize the XML for insertion in the #PCDATA. 

Cheers,
Geoff

---

From: Lisa Dusseault <[email-masked]>
Date: Fri, 23 Dec 2005 06:42:01 -0800

Wilfredo points out something I hadn't considered before, which I think 
addresses your concern, Geoff -- the idea that the only agents that 
need to preserve prefixes are those that want to.
  - Currently we have no known servers or clients that have a need to 
preserve prefixes.
  - If we had a server that decided to implement XPath-based search or 
something similar, that server could unilaterally begin to preserve 
prefixes as long as the standard doesn't prevent it.
  - If we had a client that decided to do something similar on its local 
cache/copy of data, the client would have to get a prefix-preserving 
parser and do a bunch of other work too -- but I don't see how the 
server behavior would entirely prevent this from working.

So although I'm OK with saying SHOULD preserve, I see good arguments 
for saying MAY preserve and I'm fine with that resolution too.

Lisa

---

From: Geoffrey M Clemm <[email-masked]>
Date: Fri, 23 Dec 2005 10:33:46 -0500

> need to preserve prefixes are those that want to.

My point below was that this is precisely what an agent cannot do
(if by "agent", you mean an individual client).  One agent cannot
decide to preserve prefixes by escaping the XML for a given
property value, unless *all* agents have agreed to do so (i.e.,
unless that property by definition always holds escaped XML text).

So at a minimum, you can never use this approach for an existing
property, because none of the existing clients will be escaping the
property value on write or unescaping the property value on read.

>   - Currently we have no known servers or clients that have a need to 

I'm not sure who "we" refers to here, but minimally it doesn't include
us (:-), because we will store XML schema and XSLT data in property 
values,
which do require that namespace tags be preserved.  I doubt we're the
only ones that will do so.

> prefixes as long as the standard doesn't prevent it.

Alternatively, a server could unilaterally decide to not
preserve prefixes, since the standard doesn't require it (i.e., it is
not a MUST).

> server behavior would entirely prevent this from working.

What prevents this from working (well) is that it could do so only if all
other clients that read or wrote this property did so as well. 
This approach is likely to be a source of subtle interoperability 
problems.

> for saying MAY preserve and I'm fine with that resolution too.

I very rarely care about the distinction between SHOULD vs. MAY,
and this is not one of those rare times (:-).

Cheers,
Geoff

---

From: Lisa Dusseault <[email-masked]>
Date: Fri, 23 Dec 2005 07:48:14 -0800

> unless that property by definition always holds escaped XML text).

That's true, but it's not quite what I was saying.

> only ones that will do so.

I didn't know about this or missed it.  :)  Good info.

> This approach is likely to be a source of subtle interoperability 

Let's say that
   1.  client X wants to use prefixes and store XML schema and XSLT data 
in property values.
   2.  client Y expects that value to be in ordinary XML, not 
encapsulated.
   3.  server S does not preserve prefixes.

I'm trying to understand here, how does the behavior of Y and S prevent 
X from doing what it wants to do?  Can't client X still store that 
information or at least reconstruct it, if it wants to use prefixes in 
local computations?  It might be messy, but it ought to be possible to 
preserve the original mappings somewhere the server won't mess with it 
(e.g. add an attribute 'originalmapping:x="http://example.com/ns"').  
Knowing how existing servers tend not to preserve prefixes anyway, I'd 
imagine this is what client X would have to do regardless, in order to 
work.

But there could well be something I'm missing here so I'm happy to be 
educated.

Thanks,
Lisa

---

From: Geoffrey M Clemm <[email-masked]>
Date: Fri, 23 Dec 2005 11:15:28 -0500

> imagine this is what client X would have to do regardless, in order to 

There might be something that client X could do, but the proposal we
were discussing was simply escaping the XML as a string via #PCDATA.
That proposal would cause client Y to break.

We could discuss an alternative proposal such as the one you describe
above, but a user of client Y would still get back broken XSLT or
broken XML-schema data, because client Y doesn't know how to use the 
originalmapping information to fix it up.  Also, this approach is
likely to be a lot of work for the client, so it isn't a "simple"
approach(which was what the #PCDATA approach was trying to achieve).

So I have not yet heard a "good" alternative to solving this problem
other than recommending that servers preserve namespace mappings.

Cheers,
Geoff

---

From: Wilfredo Sánchez Vega <[email-masked]>
Date: Fri, 23 Dec 2005 10:42:26 -0800

> So you'd be happy would be "SHOULD" be a "MAY"?

   Yes, absolutely.

   (The rest of this is banter...)

> Too bad that other W3C specs have started to use the prefix anyway,  
> and thus XML Infoset also considers it significant.

   Yeah, now that I've been forced to learn something about it, I  
hate XML.  Can you tell?  I should have challenged Tim Bray to a fist  
fight last week when I had the chance, but he's bigger than I am.

   The whole point of XML is that I shouldn't have to think about it,  
which is why I'm advocating against requirements that make me think  
really hard about it.  The serializer in PyXML decides where to park  
namespace definitions and what prefixes to use or not use.  I don't  
want to tweak the serializer, write my own, or find another library.   
If those are necessary, there's a spec problem, and in this case the  
spec problem would be in WebDAV.

   Sounds like there needs to be a serialization spec for XML as you  
suggest, then PyXML would want to conform to that, and ideally, that  
spec would address this namespace mess.  Like the ETag on PUT issue,  
I think it needs to be addressed upstream, instead of in DAV.

>>   Geoff, I'm not following your argument that some clients might  
>> not de-serialize the data.  If you have XML there, you have  
>> structured data.  Either the client understands the property and  
>> does handles it properly or it doesn't.
>
> There are generic clients out there that display *every* property.  
> How are they supposed to know that some text indeed consists of  
> escaped XML?

   Or XML?  Or binhex goo?  I dunno, which is why generic clients are  
of limited usefulness.  They are developer tools, not user tools.

>>   What's a generic client going to show the user for structured  
>> XML data that it doesn't understand?  Generic client tends to  
>> refer to a tool for l33t haxx0rz.
>

   The XML, you mean?  If not, what's the text content of "<foo><bar/ 
 ></foo>"?

   Only an engineer would think that a generic client is an  
acceptable UI for browsing structured data, even if it is XML "text".

   Lisa and some others in CalDAV think it's useful to have  
meaningful resource names in CalDAV because then you can browse it  
with Firefox or mount it in Finder, as if my Mom wants to crack open  
an iCalendar resource and read it for fun.  That's OK for developers,  
but it's _really_ bad UI.  Never mind localization issues and the like.

   If you are engineering for usability, generic clients aren't of  
any value.  My point being that for the audience which can actually  
use generic clients, showing angle brackets is fine.  Making  
compromises in the same of such "usability" is a mistake, I think.

   You are trading off knowing for sure, no doubt, your data won't be  
manipulated by the server for not having angle brackets in generic  
clients.  Not a great trade-off in my book.

	-wsv

---

From: Wilfredo Sánchez Vega <[email-masked]>
Date: Fri, 23 Dec 2005 11:50:55 -0800

> property value on write or unescaping the property value on read.

   Correct, but the existing spec (which is what applies to existing  
properties) doesn't require prefixes to be preserved.  You can't fix  
that with a spec change after the fact.

> only ones that will do so.

   Right.  So your options are:

	1- define properties that escape the XML.
	2- use a server that preserves the prefixes.

   I assume that you do #2 today, and that the lack of a requirement  
in the spec didn't prevent that.  Yes, that means you won't like my  
DAV server because I'm not catering to your (what, to me, are  
eccentric) needs.  I do not wish to be required to do so, and I'm  
willing to lose you as a possible client as a result of my lack of  
that feature.

   I agree that if you want this to work reliably across more DAV  
servers, something has to change.  I don't think that the server spec  
is what should change; I think the property specs should change,  
which has the advantage of working in the current WebDAV framework.   
I recognize that this isn't trivial for existing services, and in  
those cases, you can keep doing #2 and use a server that fits the bill.

> preserve prefixes, since the standard doesn't require it (i.e., it is

   Correct.  And twisted.web2, at present, doesn't guarantee this,  
nor does mod_dav.

> This approach is likely to be a source of subtle interoperability  

   You just told me that prefix-preserving parsers are easy to find,  
as if it was a non-issue.

   If you are storing data that required prefixes to be preserved in  
a property, of course clients that edit that data has to play nice.   
I'm not sure how even a server that guarantees the behavior you want  
can save you there.  It's going to have to preserve what the client  
sent, and if the client edits your prefixes, you're still SOL.

   On the other hand, if the data is escaped, and the client doesn't  
know that the property is escaped XML, it seems _less_ likely that if  
it did decide to write that data back out to the server that it will  
munge it before doing so.

> and this is not one of those rare times (:-).

   Ditto.  :-)

	-wsv

---

From: Elias Sinderson <[email-masked]>
Date: Fri, 23 Dec 2005 12:50:14 -0800

Geoffrey M Clemm wrote:

> [...] I have not yet heard a "good" alternative to solving this 
> problem other than recommending that servers preserve namespace mappings.

The above captures my take on the issue as well, and there seems to be 
rough consensus around this recommendation. AFAICT, the only thing 
remaining to be decided is whether the spec says SHOULD or MAY -- My 
vote is for SHOULD, because it will be more beneficial to clients if 
more servers tend to implement this feature in the long run. It is my 
opinion that a MAY level requirement in this situation will lead to more 
special casing for client writers (i.e. 'am I talking to a server of 
type A or a server of type B?') and, hence, interoperability issues.

Geoff supports the requirement but is indifferent wrt SHOULD vs. MAY.
Lisa is OK with SHOULD, but would be fine with MAY.
Jim is fine with SHOULD.
Wilfredo also seems to be okay with SHOULD, as it still leaves some 
wiggle room for server implementers.

Given the above I'll move to declare rough consensus on this issue and 
for the inclusion of the latest proposed text in the next draft.

Cheers,
Elias

---

From: Julian Reschke <[email-masked]>
Date: Sat, 24 Dec 2005 18:08:04 +0100

Wilfredo Sánchez Vega wrote:
> ---
>   The XML, you mean?  If not, what's the text content of 
> "<foo><bar/></foo>"?

The empty string.

>   Only an engineer would think that a generic client is an acceptable UI 
> for browsing structured data, even if it is XML "text".
> 
>   Lisa and some others in CalDAV think it's useful to have meaningful 
> resource names in CalDAV because then you can browse it with Firefox or 
> mount it in Finder, as if my Mom wants to crack open an iCalendar 
> resource and read it for fun.  That's OK for developers, but it's 
> _really_ bad UI.  Never mind localization issues and the like.

It may be bad UI, but that doesn't change the fact that meaningful 
(stable) URLs are a good thing...

Best regards, Julian