Subject: Collections and Request-URIs

From: Jim Luther <[email-masked]>
Date: Mon, 17 Jun 2002 16:45:44 -0700

RFC 2518, section 5.2 says:

"There is a standing convention that when a collection is referred to by 
its name without a trailing slash, the trailing slash is automatically 
appended.  Due to this, a resource may accept a URI without a 
trailing "/" to point to a collection. In this case it SHOULD return a 
content-location header in the response pointing to the URI ending with 
the "/". For example, if a client invokes a method on 
http://foo.bar/blah (no trailing slash), the resource 
http://foo.bar/blah/ (trailing slash) may respond as if the operation 
were invoked on it, and should return a content-location header with 
http://foo.bar/blah/ in it.  In general clients SHOULD use the "/" form 
of collection names."

It looks to me like the "standing convention" doesn't match the BNF. RFC 
2616, section 5.1.2 gives this rule for the Request-URI in the 
Request-Line:

	Request-URI 	= "*" | absoluteURI | abs_path | authority

and RFC 2396, section 3 gives these rules for abs_path and path_segments:

	abs_path      = "/"  path_segments
	path_segments = segment *( "/" segment )

So by my reading, abs_path should not have a trailing slash. Did I miss 
something? If not, I think the WebDAV spec should say that servers MUST 
handle Request-URIs to collections without the trail slash.

Anyway, trailing slashes issue causes an interoperability problem for 
our WebDAV file system and Apache 2.0 servers.  Since Mac OS X's WebDAV 
file system doesn't know if the end path component passed to it is a 
collection or non-collection resource, it doesn't put a trailing slash 
on the Request-URI. This works on all servers except for Apache 2 which 
sends us a "301 Moved Permanently" response. We don't redirect because 
the HTTP spec (RFC 2616, section 10.3.2) says:

"If the 301 status code is received in response to a request other than 
GET or HEAD, the user agent MUST NOT automatically redirect the request 
unless it can be confirmed by the user, since this might change the 
conditions under which the request was issued."

Hmmm... WebDAV file system can't confirm it with the user because it's a 
file system which doesn't do user interaction. I guess that I could 
compare the URI in the 301 response to see if it's the same as what I 
sent except for the trailing slash and retry the request. However, that 
would double the number of PROPFIND for collection transactions we send 
to Apache 2 servers and I'm trying to cut the number of transactions 
from our client, not increase them.

Apparently, Microsoft WebFolders had the same problem with Apache 2. 
Here's what we found in httpd.conf:

	#
	# The following directive disables redirects on non-GET requests for
	# a directory that does not include the trailing slash.  This fixes a
	# problem with Microsoft WebFolders which does not appropriately handle
	# redirects for folders with DAV methods.
	#
	BrowserMatch "Microsoft Data Access Internet Publishing Provider" 
redirect-carefully
	BrowserMatch "^WebDrive" redirect-carefully

So if we add a similar line:

	BrowserMatch "^WebDAVFS" redirect-carefully

to Apache's httpd.conf file to do a redirect-carefully for our 
User-Agent, our client works great with Apache 2 DAV servers.

- Jim Luther

---

From: "Julian Reschke" <[email-masked]>
Date: Tue, 18 Jun 2002 08:16:58 +0200

> From: [email-masked]
> [mailto:[email-masked]]On Behalf Of Jim Luther
> Sent: Tuesday, June 18, 2002 1:46 AM
> To: [email-masked]
> Subject: Collections and Request-URIs
>
>
>

However:

segment       = *pchar *( ";" param )
param         = *pchar

pchar         = unreserved | escaped |
                ":" | "@" | "&" | "=" | "+" | "$" | ","

([1]) and

"The second convention is a BNF-like grammar, used to define the formal URI
syntax. The grammar is that of [RFC822], except that "|" is used to
designate alternatives. Briefly, rules are separated from definitions by an
equal "=", indentation is used to continue a rule definition over more than
one line, literals are quoted with "", parentheses "(" and ")" are used to
group elements, optional elements are enclosed in "[" and "]" brackets, and
elements may be preceded with <n>* to designate n or more repetitions of the
following element; n defaults to 0."

([2]). So it seems that "segment" may be empty.

Julian

[1] <http://greenbytes.de/tech/webdav/rfc2396.html#rfc.section.3.3>
[2] <http://greenbytes.de/tech/webdav/rfc2396.html#rfc.section.1.6>

---

From: "Julian Reschke" <[email-masked]>
Date: Tue, 18 Jun 2002 18:52:55 +0200

Hi,

just did a few tests:

- Adobe Golive follows the 301 redirect without asking the user
- XML Spy always submits the folder URL without trailing / (no matter what
you enter), and reports the 301 as error

Also interesting:

- the default config for Apache 2.0.36 disables this behaviour for both the
Microsoft webfolder client and the WebDrive client:

> -----Original Message-----

---

From: Greg Stein <[email-masked]>
Date: Tue, 18 Jun 2002 15:37:04 -0700

On Mon, Jun 17, 2002 at 04:45:44PM -0700, Jim Luther wrote:
>...

As mentioned previously, the trailing-slash form matches the BNF.

> conditions under which the request was issued."

Roy Fielding has previous stated that RFC 2616's language is incorrect. The
spec is supposed to allow things like PROPFIND. I forget the exact
terminology that Roy used (i.e. was it "*all* idempotent methods"? or
something else), but PROPFIND was among them.

This topic was extensively discussed on the Apache development list when we
added that redirection code (actually, we fixed a bug that prevented the
"proper" behavior of redirecting for all methods). That was when Roy chimed
in about RFC 2616 being slightly incorrect, that a redirection is legal,
that Apache 2.0 "should" go ahead and redirect, and that the clients who are
*not* following the redirect are simply broken.

And given that the non-following clients are "broken", we went ahead and
made the default behavior to do the redirection, and called out the broken
clients in the standard distribution of httpd.conf.

>...

My response would be "fix your client. do the redirect."  :-)

If that is going to be impossible to do, then I can add that line to
httpd.conf. But I'm reluctant to just start adding stuff. Your client really
is "broken" (for some definition thereof :-) if it isn't following the 301.

Cheers,
-g

-- 
Greg Stein, http://www.lyra.org/

---

From: "Roy T. Fielding" <[email-masked]>
Date: Tue, 18 Jun 2002 16:22:29 -0700

> something else), but PROPFIND was among them.

Please see the official errata:

   http://world.std.com/~lawrence/http_errata.html#saferedirect

> clients in the standard distribution of httpd.conf.

That is correct.  And, BTW, RFC 2518 is in fact incorrect in the paragraph
quoted, something which I stated numerous times when it was a draft.
There are no known examples of HTTP servers for which the URI without a
slash is the SAME RESOURCE as the URI with a slash.  Automatically
providing two different URI for the same resource, particularly when
they cross hierarchical syntax, instantly doubles the application
name space (bad for IR and QA) and creates holes in access control.
That is why ALL general HTTP servers provide PERMANENT REDIRECTS
from a WRONG URL to a better URL for the sake of stupid authors of
broken references.  They have never been the same resource on any
server for which I have ever had readable code.

It is completely idiotic for the protocol standard to state that a
very expensive network round-trip correction needed because of indirect
stupidity on the part of the server in providing the wrong URL to the
client, or because clients have been encouraged to remove the trailing
slash because some moron in UI thought it looked better without the
slash, can be "worked around" by ignoring the difference between
the two resources.  Any WebDAV server that does not return a redirect
in that situation is a bad HTTP server.

A WebDAV client cannot assume anything about the nature of the URI,
since there is absolutely no requirement in HTTP that two URI that
differ only by the trailing slash are in fact the same resource.
An HTTP server cannot ignore the difference because the content of
a representation is interpreted in relation to the URI that was used to
access that resource.  Likewise, it often cannot ignore the difference
because an HTTP server consists of many interoperating components
that are configured according to the URI name space being accessed,
not necessarily all within the same machine.  Therefore, the ONLY
interoperable behavior when faced with a slashless collection request
is to perform an external redirect, even if that means existing
WebDAV clients will puke and die.

A WebDAV client must therefore handle the redirect.

Cheers,

Roy T. Fielding, Chief Scientist, Day Software
                  ([email-masked]) <http://www.day.com/>

                  Chairman, The Apache Software Foundation
                  ([email-masked])  <http://www.apache.org/>

---

From: "Julian Reschke" <[email-masked]>
Date: Wed, 19 Jun 2002 07:19:04 +0200

Hi,

first of all I really have a problem with the language here. WebDAV clients
that implement RFC2518 and RFC2616 (minus erratum published in March 2001)
*to the letter* suddenly are called "broken".

It's good that the erratum  for RFC2616 allows clients to follow the
redirect automatically.

However, I think the following issues remain:

> server for which I have ever had readable code.

In this case this needs to be put on the RFC2518 issues list.

2) In the particular case of moddav2, we're stuck with a situation where an
additional roundtrip is required to PROPFIND information about a resource
(if it's not known in advance to be a collection). I'm still interested in
an approch where the roundtip can be avoided.

3) WebDAV's namespace model is not completely compatible to HTTPs: every
WebDAV compliant URL namespace conforms to RFC2616, but not the other way
round. That's a fact the WebDAV WG has to deal with. In particular:

Given a collection /a and HTTP resources "/a/b" and "/a/b/" (where "/a/b"
and "/a/b/" are NOT the same resource) -- what should a PROPFIND on /a with
depth 1 report? I can think of (a) it should fail (because "/a" isn't a
WebDAV compliant collection), (b) it silently suppresses the response
element for "/a/b" or (c) it reports both. Looking at RFC2518, section 5.2
[1], (c) seems to be possible but I'm sure that few WebDAV clients are
prepared to handle this.

Is anybody aware of a WebDAV server that *does* dinstinguish between "/a/b"
and "/a/b/" and behaves as described in (c)? (Can I configure Apache2/moddav
to handle this situation?).

Feedback appreciated,

Julian

[1] <http://greenbytes.de/tech/webdav/rfc2518.html#rfc.section.5.2>

---

From: "Julian Reschke" <[email-masked]>
Date: Wed, 19 Jun 2002 09:31:24 +0200

Hi,

there was recently an xml-dev thread about security problems allowing
arbitrary XML in protocols (see for instance [1]).

As WebDAV doesn't need resolution of external entities / DTD validation, I'd
suggest to specfiy that servers and clients MUST NOT resolve external
entities, that is, MUST reject any WebDAV protocol message that contains
external entities.

Feedback appreciated.

[1] <http://lists.xml.org/archives/xml-dev/200206/msg00247.html>

---

From: "Jim Whitehead" <[email-masked]>
Date: Wed, 19 Jun 2002 14:30:13 -0700

> arbitrary XML in protocols (see for instance [1]).

This topic is also discussed in RFC 2518, in Section 17.7 (Implications of
XML External Entities).

> As WebDAV doesn't need resolution of external entities / DTD
> validation, I'd suggest to specfiy that servers and clients MUST NOT
> resolve external entities, that is, MUST reject any WebDAV protocol
> message that contains external entities.

In RFC 2518, we didn't go so far as to outlaw external entities, since (a)
it didn't seem that likely they would ever get shipped across the wire, and
(b) they might be useful for extensibility. But, after several years of
implementation, I don't know of any uses of XML external entities, so I'd be
fine with prohibiting them.

- Jim

---

From: "Julian Reschke" <[email-masked]>
Date: Wed, 19 Jun 2002 23:43:25 +0200

> From: Jim Whitehead [mailto:[email-masked]]
> Sent: Wednesday, June 19, 2002 11:30 PM
> To: Julian Reschke; [email-masked]
> Subject: RE: WebDAV XML handling vs. external entities
>
>

Indeed. Missed that part :-)

> fine with prohibiting them.

It think we should clarify. Right now, existing servers seem to either
ignore the external entitiy (mod_dav) or report an error (IIS). I think the
former is wrong because it means that part of the request wasn't parsed, so
the request shouldn't be executed. For the sake of clarity, I think it would
be a good thing to recommend that servers should fail the request.

Julian