Subject: Re: [dav-dev] Problem with OfficeXP and Accented characters

From: Greg Stein <[email-masked]>
Date: Tue, 19 Feb 2002 18:52:40 -0800

On Wed, Feb 13, 2002 at 09:31:37PM +0100, Julian Reschke wrote:
>...
> > From: [email-masked] [mailto:[email-masked]]On Behalf Of
> > Peter Gillis
> > Sent: Wednesday, February 13, 2002 8:46 PM
>...
> > I am having a problem with Microsoft Web Folders after the client installs
> > Office XP on their machine and file names with accented characters are
> > involved.  Our server has been working in the past with earlier
> > implementations of Web Folders without a problem, however, when the client
> > machine is upgraded to Office XP, a document that was accessible
> > previously
> > is now not found.  I have tracked the problem down to the fact that Web
> > Folders is encoding the URI value a second time before sending
> > the request,
> > which then causes it not to be found on our server.  For example:
> >
> > In the folder listing we send back the following property:
> >
> > <href>/dav/webdav/%E8%E2temp%C9.xls</href>
> 
> Isn't this wrong in the first place? My understanding is that you should
> send URL-encoded UTF-8.

Nope.

The filename is in an "original character set". That is then encoded into
octets for the URL. That transformation is not specified anywhere. Ideally,
it is "original -> UTF-8", but nobody says it must be. In fact, I would say
it should match whatever encoding was used for the Request-URI (but that is
not specified/defined in the request, so you're out of luck again).

Once you have octets, then you perform the URL-escaping (using '%xx').

After that, you need to transform the URL into the character set of the
response body. That is usually UTF-8, but it is possible to have the XML in
a different character set (provided it is specified on the Content-Type
response header).

Finally, you must XML-escape the UTF-8 characters of the URL you're
inserting (e.g. translate '&' to '&amp;') so that you can embed the UTF-8
content into the XML response.

A long time ago, I captured this as a start of a technical FAQ. See:
    http://www.webdav.org/other/techfaq.html

Specifically, the second section.

Cheers,
-g

-- 
Greg Stein, http://www.lyra.org/

---

From: Stefan Eissing <[email-masked]>
Date: Wed, 20 Feb 2002 09:51:46 +0100

> not specified/defined in the request, so you're out of luck again).

I dare to disagree. The URI must be UTF-8 encoded. Otherwise you lose
interoperability. Assume you take 8859-1 in your response document and
apply a stylesheet which outputs UTF-8. XSTL does not know that it
should change the text child nodes of <href> elements.

You already mentioned request-URIs where the encoding is not known.

//Stefan

---

From: Erik Seaberg <[email-masked]>
Date: 21 Feb 2002 13:50:00 -0800

Julian Reschke <[email-masked]> writes:
> - In moddav (correct me if I'm wrong), people may be creating
> resources by directly accessing the filesystem.

<URL: http://www.webdav.org/mod_dav/install.html > says
# NOTE: the file repository is considered "private" to mod_dav and the
# web server.  Modifying files via FTP or through filesystem commands
# should not be allowed.

> - Broken clients may use request/destination URIs which are not
> encoded in UTF-8

I thought (per RFCs 2396 and 2616) the "http" scheme just specifies a
mapping for each part of the URL from US-ASCII to a sequence of
octets, and the mapping from those octets back to characters (if any!)
is at the whim of the server.  %-escaped UTF-8 is *recommended* for
new schemes (and for HTML that embeds URLs containing invalid
non-ASCII characters), but a server that uses Latin-15 should be able
to send Latin-15 in PROPFIND responses without having a client
complain about or mangle perfectly workable URLs just because it can't
display them nicely.

---

From: "Julian Reschke" <[email-masked]>
Date: Wed, 20 Feb 2002 11:38:28 +0100

Greg,

what you describe is a scenario where

- the server preserves whatever octet sequence was used in a URL
- the clients are consistent in what they send/expect

However, thinks are more complicated.

- In moddav (correct me if I'm wrong), people may be creating resources by
directly accessing the filesystem. If, for instance, I create a filename
containing "A umlaut", and the filesystem's filename encoding is ISO-8859-1,
I'll have the octet %c4 in the name. Returning %c4 in the PROPFIND response
in the general case will not work (because the client doesn't know about URL
character encodings, so it must use what the RFCs say, and that is UTF-8).

- Broken clients may use request/destination URIs which are not encoded in
UTF-8 (this may apply to older versions of Microsoft clients). So you
basically have the choice of failing the request if the octet stream doesn't
UTF-8-decode, or you can try to workaround the problem by making assumptions
about what the encoding in the request may have been.

As this issue is coming up every few weeks and as almost every server /
client I've seen in the last few months has some bug in this regard, it
should certainly clarified in the RFC2518 revision.

Julian

> -----Original Message-----
> From: [email-masked]
> [mailto:[email-masked]]On Behalf Of Greg Stein
> Sent: Wednesday, February 20, 2002 3:53 AM
> To: Julian Reschke
> Cc: Peter Gillis; [email-masked]; [email-masked];
> [email-masked]
> Subject: Re: [dav-dev] Problem with OfficeXP and Accented characters
>
>
> > > From: [email-masked]
> [mailto:[email-masked]]On Behalf Of
> > > Peter Gillis

---

From: Stefan Eissing <[email-masked]>
Date: Fri, 22 Feb 2002 10:12:29 +0100

Am Donnerstag den, 21. Februar 2002, um 22:50, schrieb Erik Seaberg:

> [...]

What you say is correct from a pure protocol perspective. Nowhere
in RFC 2518 or 2616 is anything said about decoded URIs. And
RFC 2396 says that there is no such thing as a decoded URI.

All true. The problem appears when you put a user interface on top
of webdav. Be it a Windows WebFolder or an OS X mounted volume.
People just expect nowadays that they can use Umlauts and the Euro
sign in filenames. They even expect that they can do that in Windows
and Linux and MacOS and will all see the same file on a particular
server.

As I see it, this requires a standard character encoding for conversion
of user input to URIs on the client side, namely UTF-8.

For the broken clients, Julian is talking about, it would be good
if servers make the effort to detect non-UTF-8 encoding and
apply characters conversion (most likely from 8859-1).

I've seen Tomcat looking for charset parameters in URIs to detect
char encoding. I'm not sure if I really understand what they are
trying to do. It looks like clients could use
   GET /test/folder/%d7%c4;charset=ISO-8859-7 HTTP/1.1
in their requests. Does anyone know more about that?

//Stefan