Subject: Re: UTF-8 Encoding Question

From: "John Glavin" <[email-masked]>
Date: Wed, 28 Feb 2001 22:57:05 -0800

Thanks, this makes sense now.  I guess what I can do is see if the URI
contains a valid UTF-8 sequence and if it does then assume it's UTF-8
encoded.  I got the following text from RFC-2279

UTF-8 strings can be fairly reliably recognized as such by a
      simple algorithm, i.e. the probability that a string of characters
      in any other encoding appears as valid UTF-8 is low, diminishing
      with increasing string length.

Webfolders must be doing something similar to this since it understands both
UTF-8 and Latin-1.

John Glavin
RiverFront Software
[email-masked]
http://www.webdrive.com

----- Original Message -----
From: "Greg Stein" <[email-masked]>
To: "John Glavin" <[email-masked]>
Cc: <[email-masked]>
Sent: Wednesday, February 28, 2001 4:54 PM
Subject: Re: UTF-8 Encoding Question

> We just had a discussion related to this on mod_dav's mailing list.
>
> On Wed, Feb 28, 2001 at 03:52:37PM -0800, John Glavin wrote:
> >...
> > But I run into a problem with the mydocsonline.com DAV server which says
> > it is using UTF-8 Encoding but returns the href as href: Magn%FCs.txt
> > This is not UTF-8 encoded, because characters > 0x80 in UTF-8 will be
> > encoded in a multibyte sequence.  This is normal ISO-8859 (Latin)
Encoding.
>
> There are two references to UTF-8 in the response: the Content-Type header
> and the XML document header:
>
>     Content-Type: text/xml; charset="utf-8"
>     <?xml version="1.0" encoding="utf-8"?>
>
> Both of these refer to the *response body*. In that sense, all characters
in
> the body are properly UTF-8 encoded.
>
> The URL itself is in its "escaped" form. See sections 2.4.2 of RFC 2396
for
> more info. Section 2.1 covers the general problem of UTF-8 encodings for
> URLs.
>
>
> To be more concrete. Section 2.1 defines two types of characters: "URI
> characters", and "original characters". The "utf-8" above refers to the
URI
> characters since that is what is sitting in the body of the response.
>
> The % escaping will give you a set of octets. The question then becomes,
> "what encoding will transform those octets into the 'original'
characters?"
> At the moment, you do not have enough information to do that. There is no
> attribute or header or other item that you can inspect for that.
>
> > In this case I am not sure what to do.  I use the Windows API call
> > MultiByteToWideChar function but I need to tell it to use either UTF-8
or
> > ANSI code pages.  For the mydocsonline server I need to use ANSI however
> > they are telling me to use UTF-8 and using UTF-8 wont work.
> >
> > When I use Webfolders it works properly on the mydocsonline server and
> > somehow knows to not use UTF-8 decoding.  Does anyone have any idea why
it
> > works or how I could really detect which code page to use ?
>
> I think your statement about it "working" for some servers, and not
working
> for mydocsonline (which is based on an early mod_dav; the current mod_dav
> has the same issue, tho) is based on a presumption that the character set
> for the URI characters == the charset of the original characters. That
> assumption is being made by servers and clients today.
>
> In mod_dav's case, we take the URI's (unescaped) octets and simply save
the
> resource under that name. We then return it using the same octet sequence
> (properly escaped). The net effect is that we keep the same encoding of
the
> "original characters" for the client. Of course, the problem arises when
one
> client saves using a UTF-8 encoding and another reads as Latin-1.
>
> But mod_dav does not have enough information from the client to decode the
> URL into (say) Unicode, and save that. If it could, then we could always
> return a UTF-8 encoding for the original characters (although we would
still
> have no way to tell that encoding to the client; clients would just
continue
> to assume the response encoding matches that encoding and it would
*happen*
> to match).
>
>
> To be really clear, let's go with a little diagram here:
>
>     Unicode resource name ("original characters")
>        |
>      [ UTF-8 encoding ]
>        |
>     URI octet sequence
>        |
>      [ URI %-escaping ]
>        |
>     Returned URI ("URI characters")
>
> But another totally legal scenario is:
>
>     Latin-1 resource name ("original characters")
>        |
>      [ identity (no) encoding ]
>        |
>     URI octet sequence
>        |
>      [ URI %-escaping ]
>        |
>     Returned URI ("URI characters")
>
> There is nothing in the response body to indicate which of the above two
> forms is occurring. Similarly, there is nothing in the request body to
> indicate which was used for the Request-URI. Because of the latter,
servers
> are just as broken if they make an assumption about how to decode from the
> URI octets into original characters.
>
> mod_dav does not attempt to decode/encode between octets and original
> characters. It just keeps them as octets. But that does imply that the
> encoding used by the client when it stored the resource better be the same
> encoding used when accessing the resource and the same decoding used for a
> PROPFIND result.
>
> RFC 2396, section 2.1 explicitly punts this issue to a future date. It
seems
> that I recall an internet draft, or even possibly a new RFC, but I'm not
> immediately aware of it.
>
> Cheers,
> -g
>
> --
> Greg Stein, http://www.lyra.org/

---

From: Greg Stein <[email-masked]>
Date: Thu, 1 Mar 2001 00:33:55 -0800

That would be a good heuristic for handling this.

We could augment the DAV:href element with an optional attribute, like this:

  <D:href D:original-charset="iso-8859-1">http://some.host/Magn%FCs.txt</D:href>
and
  <D:href D:original-charset="utf-8">http://some.host/C%C3%A9sar.txt</D:href>

That will help within the responses (if a server supplies it, then you don't
have to guess; otherwise, fall back to your heuristic). We'd still need a
way to determine the original charset of the Request-URI, though, to fully
solve the problem.

JimW: can we list this as an "Issue" for RFC 2518?

Cheers,
-g

---

From: "Dylan Barrell" <[email-masked]>
Date: Thu, 1 Mar 2001 09:10:22 -0800

Why, if webDAV is an extension of HTTP and HTTP specifies that the default
character set is ISO-8859-1 unless specified in the header, should it
therefore not be possible for the server to determine the character set from
the request header?

--Dylan

---

From: John Stracke <[email-masked]>
Date: Thu, 01 Mar 2001 09:15:36 -0500

Greg Stein wrote:

> RFC 2396, section 2.1 explicitly punts this issue to a future date. It seems

draft-masinter-url-i18n-07.txt

--
/==============================================================\
|John Stracke    | http://www.ecal.com |My opinions are my own.|
|Chief Scientist |=============================================|
|eCal Corp.      |You buttered your bread, now lie in it.      |
[email-masked]|                                             |
\==============================================================/

---

From: "Dan Brotsky" <[email-masked]>
Date: Thu, 1 Mar 2001 08:57:47 -0500

At 3:52 PM -0800 2/28/01, John Glavin wrote:
>But I run into a problem with the mydocsonline.com DAV server which 
>says it is using UTF-8 Encoding but returns the href as
>href: Magn%FCs.txt      This is not UTF-8 encoded, because 
>characters > 0x80 in UTF-8 will be encoded in a multibyte sequence. 
>This is normal ISO-8859 (Latin) Encoding.

As Greg pointed out on dav-dev, it's very important *not* to confuse 
the "UTF-8 encoding" explicitly employed for the XML response bodies 
with any possible "UTF-8 encoding" of non-ascii characters in the 
URLs in these bodies (before they are %-escaped).  XML bodies are 
explicitly encoded, but this encoding does NOT apply to any URL 
segments that are %-encoded *inside* the XML bodies.

The fact that URLs, in and of themselves, do not indicate the 
encoding used for their path segments is a known issue in the HTTP 
community, and is specifically referred to in several specs.  At this 
point, it is pretty much up to clients and servers to agree on 
conventions about what encoding will be used in such path segments. 
There seem to be two common appraches:

1. No encoding is done at all.  The platform-specific character set 
components of a filename are viewed as an octet stream and are then 
%-encoded directly into the URL.  Since many platform-specific 
character sets are very close to ISO 8859-1 (ISO Latin 1), this often 
looks like the client or server is using an ISO 8859-1 encoding of 
these segments, but it's often just the platform encoding.

2. Encoding is done by translating the platform-specific character 
set components of a filename into Unicode, and then UTF-8 (or 
sometimes ISO 8859-1) encoding that Unicode into an octet stream 
which is then %-encoded for use in an URL.

The advantage of approach 2 [in the case where UTF-8 is used], as I 
outlined in an earlier message to dav-dev, is that clients and 
servers with different platform character sets but who are both 
capable of encoding certain characters can unambiguously communicate 
about filenames that contain those characters.  This is increasingly 
important in today's global net community as wide-character-capable 
clients with divergent platform character sets proliferate.

I am encouraged to see that many of the "big" browser clients (IE and 
Netscape) are beginning to use UTF-8 encodings of platform character 
sets in the URLs they generate and to understand UTF-8 encodings in 
URLs they receive.  It would be nice to see the prevalent DAV servers 
(such as mod_dav) take approach #2 with UTF-8 as well.  (Approach #2 
using ISO 8859-1 has far fewer advantages, since ISO 8859-1 encodes a 
very restricted subset of Unicode.)

     dan

---

From: John Stracke <[email-masked]>
Date: Thu, 01 Mar 2001 14:19:05 -0500

> therefore not be possible for the server to determine the character set from

Because the charset applies to the message body, not the request-URI.

--
/=================================================================\
|John Stracke    | http://www.ecal.com |My opinions are my own.   |
|Chief Scientist |================================================|
|eCal Corp.      |If God had not given us duct tape, it would have|
[email-masked]|been necessary to invent it.                    |
\=================================================================/

---

From: "Dan Brotsky" <[email-masked]>
Date: Thu, 1 Mar 2001 10:35:51 -0500

At 12:33 AM -0800 3/1/01, Greg Stein wrote:
>   <D:href 
>D:original-charset="iso-8859-1">http://some.host/Magn%FCs.txt</D:href>
>and
>have to guess; otherwise, fall back to your heuristic).

This certainly would be a helpful thing for servers to say to clients 
(although maybe in some other way), and I second your request to JimW 
about adding it as an issue.

>  We'd still need a

I think there's a slightly different way to approach this, especially 
since (in general) there's no way to determine the original charset 
of the client request.

1. In situations where a returned URI in the response was present in 
the client request, I think the server really has to return it 
exactly as it was submitted.  Otherwise a lot of clients are going to 
get really confused, because for sure they're just looking for what 
they sent and not expecting to get into all sorts of "oh it's really 
the same in canonical form" discussions.

2. In situations where a returned URI in the response is not in the 
original request, the server is free to use whatever encoding it's 
going to use for path segments, and it can tell the client what is 
(either specifically as in Greg's proposal above or more generally 
via an OPTIONS setting or some such).

As both clients and servers get smarter about this issue, then 
presumably the distinction between these cases starts going away (and 
it becomes less important for servers to say what encoding they're 
using for path segments).  But on the way there this approach allows 
for backward compatibility with existing clients that are unaware of 
these issues without servers having to guess.

And it has the added advantage of not breaking any client/server 
combinations that currently work.

     dan

---

From: "Dan Brotsky" <[email-masked]>
Date: Thu, 1 Mar 2001 10:47:50 -0500

>therefore not be possible for the server to determine the character set from

The character set of the *body* and the character set of the *request 
url* are entirely different things.  The HTTP spec does not specify a 
default character set for path-segments or other segments in URLs, 
and the various URL/URI specs (as currently officially updated) don't 
do this either.

     dan

---

From: John Stracke <[email-masked]>
Date: Thu, 01 Mar 2001 15:11:22 -0500

> about adding it as an issue.

It might be better to wait and see whether IRIs take off
(<draft-masinter-url-i18n-07.txt>; broadly, the approach is to define that IRIs
are like URLs, but in Unicode; to use an IRI in a context that demands a URL, you
encode it in UTF-8, then apply %-encoding as normal).  No sense creating two
separate mechanisms to solve the same problem.

--
/==============================================================\
|John Stracke    | http://www.ecal.com |My opinions are my own.|
|Chief Scientist |=============================================|
|eCal Corp.      |Never underestimate the power of human       |
[email-masked]|stupidity. --I forget who                    |
\==============================================================/

---

From: "Dan Brotsky" <[email-masked]>
Date: Thu, 1 Mar 2001 14:09:20 -0500

At 3:11 PM -0500 3/1/01, John Stracke wrote:
>Dan Brotsky wrote:
>
>>  >   <D:href 
>>D:original-charset="utf-8">http://some.host/C%C3%A9sar.txt</D:href>
>>

I'm a big fan of IRI's (as you can tell from my earlier emails) but I 
think the issue here is the one of what to do when clients submit 
URIs with other encodings that the server returns to them in a body. 
Even when servers go to IRI discipline for URLs they generate, they 
shouldn't necessarily break clients who expect to get back what they 
asked for :^).

Here's another way of phrasing this issue that makes it not be about encoding:

The DAV spec says that it is *resources* that have properties, not 
*urls*, and that many different urls can refer to the same resource. 
When a client requests info about a resource by using a particular 
URL, but the server sends back information about a resource named by 
another URL, what guarantees does the client have about the returned 
URL:

1. Is it guaranteed to refer, in the context of this response, to the 
resource that the client asked about in the request?
2. Is it guaranteed *always* to refer to that same resource?  (In 
some sense, is there the same info in this response that one could 
glean from a permanent redirect?)

In this formulation, note that differences in %-escaping conventions 
can lead to different URLs that encode the same octet-stream.  Are 
there special conventions that apply in this case?

     dan