Subject: A case for GETSRC (or other mechanism to that effect)

From: CJ Holmes <[email-masked]>
Date: Wed, 27 Feb 2002 21:58:50 -0800

I've been reading the archives, and following the long-running and 
sometimes contentious conversation over retrieval of source 
documents.  Maybe bringing this up will get me kicked off the list, 
but we'll see.

First, I appreciate what the DAV working group is trying to do by 
allowing multiple sources for a given document, and making each 
source a separate resource.  I won't bother denying that it is useful 
to some people and interesting in its own right.  I totally see the 
logic in it.  But it is an abstraction that is useful only to a small 
minority of users, and is very difficult to implement.  If you doubt 
that, I direct you to the total lack of implementations.  But I'll 
make a case on other grounds, as well.

With any DAV request except GET the process is rather straightforward:

	A virtual host is selected

	Security determines what rights the user has on the URL.  In 
the Apache universe that means PUT, POST, MKCOL, GET, etc..  In our 
universe (WebSTAR) that means read, write, list, mkdir, etc..

	We figure out what file the URL maps to.

	In deciding which plug-in will handle the request, a DAV 
plug-in simply claims anything with a DAV-specific method.  If it is 
MKCOL, DELETE, etc. then the DAV plug-in claims it.

	DAV handles the request, within the limits set by the security plug-in.

All of this works great, until we get to the GET method.  Plug-ins 
have no way of knowing the difference between a normal GET that 
should be handled normally, and a GET that is intended to retrieve 
the source of a document and requires special permissions.  In effect 
we have TWO semantics for GET.  And so we start creating fake URL 
spaces.  But we quickly run into security and configuration hassles.

The bottom line is that there's no way to provide good DAV access 
without some special configuration on the server.  And it isn't the 
result of some kind of design error by the implementor, but a basic 
issue with how the protocol is designed.  Unless you are only dealing 
with static content you MUST set up a whole new URL space to do DAV 
at all.

In practice setting this up is such a bother that what most 
administrators do is create a whole new virtual host with all 
plug-ins disabled in order to get it done.  Is it just me, or is it 
entirely ridiculous that server administrators have to double the 
number of virtual hosts they support just to give decent DAV access 
to their users?

And all of this is so complicated for something that *should* be 
really simple!  From most servers' point of view DAV is a way to let 
web authors access their .php, .cgi, .ssi, .shtml, and other dynamic 
files and edit them and save them back onto the server.  All of this 
business about an arbitrary number of sources of arbitrary types as 
properties of a single resource don't do our users one bit of good, 
but the implementation is quite burdensome.

What we really need from the DAV protocol is a way to know that a 
given request is intended to retrieve the source of a document, 
without having to also posses knowledge about the DAV configuration, 
or which URL spaces have been quarantined for use exclusively for 
WebDAV, and without having to create special areas in the URL space 
that don't actually map to anything else in the universe just so we 
can turn off all the plug-ins for a few requests.  Thus the 
suggestion for a GETSRC property.

I've read the arguments that ask about FINDPROP and other methods 
that act on a resource, and having to double the number of methods to 
support FINDPROP and FINDPROPSRC, etc..  But in most server 
implementations when you get the properties of a .php or .ssi file 
right now, what are you getting?  In practice you're getting the 
author and lastmod date and other information about the SOURCE, not 
the output.  I don't know of anybody who is executing a .php file and 
then returning the properties of the output -- it (a) isn't practical 
because of all the side-effects it would cause like CPU load and 
errors and code running that you didn't really intend to run and (b) 
isn't useful because nobody cares one whit about the DAV properties 
of PHP output.

I suggest that we just stop considering GET to be a DAV method at 
all.  GET has nothing to do with DAV any more than POST does. 
Instead of GET, we should be using GETSRC (or whatever you want to 
call it), always.  If you want the output of a source which has been 
run through PHP or SSI or whatever, then fire up your web browser and 
use a GET or POST method.  Otherwise, use your DAV client and GETSRC 
(or whatever).

When you work with the properties of a resource, you are working with 
the properties of the resource you receive from a GETSRC command. 
When you GETSRC, you receive the same data that you PUT.  When you 
GET or POST, you receive what the server generates from the source 
and your input.  In some cases that happens to be the same as the 
source (static files).

If OPTIONS does not indicate that GETSRC is avialable, then a DAV 
client may use GET instead.

Does not preclude a document from having multiple sources at 
different URIs, for the 5% of implementations that need such 
functionality.

Advantages of this approach:

	Works well with existing security modules for Apache and 
other servers.  Just add GETSRC to the list of methods a user is 
allowed to use.

	Provides one semantic for GET (the existing semantic from 
HTTP/1.1 spec) and only one semantic.

	Is far more likely to be implemented, and implemented WIDELY, 
than the src property ever will be.

This is a major, major problem with the protocol.  This is in not an 
indictment of anybody's abilities.  Even as a prospective implementor 
reading the spec carefully I didn't appreciate the magnitude of the 
problem until I set out to really implement it, and then it became 
nearly intractable.  Even the reference implementation doesn't 
implement it.

The DAV group can take this issue in hand and come up with a solution 
people are willing to implement, or the ad-hoc solutions will become 
de-facto standards.  (ie: Translate).  Personally, I would rather see 
this solved by the DAV working group.

cjh

--

---

From: "Eric Sedlar" <[email-masked]>
Date: Wed, 27 Feb 2002 22:18:51 -0800

Actually, I think Lisa & I convinced Geoff (the big holdout against the
Translate header) to give in at the last IETF meeting.  Are there any other
holdouts with good reason to have a problem with the translate header?

--Eric

> -----Original Message-----
> From: [email-masked]
> [mailto:[email-masked]]On Behalf Of CJ Holmes
> Sent: Wednesday, February 27, 2002 9:59 PM
> To: [email-masked]
> Subject: A case for GETSRC (or other mechanism to that effect)
>
>

---

From: "Julian Reschke" <[email-masked]>
Date: Thu, 28 Feb 2002 10:06:24 +0100

Eric,

that comes as a suprise.

As far as I remember the discussion we concluced that:

a) the Translate header (or GETSRC) solves only a very specific part of the
problem (when there's a one-to-one mapping between source and output),

b) the source property needs additional work to become usable, but it's a
more generic solution.

So as far as I am concerned, the plan was to fix the source property (maybe
be taking it out of RFC2518bis and to fix it in a sperate draft).

If there were out-of-band discussions, I'd certainly like to hear about it.

Julian

> [mailto:[email-masked]]On Behalf Of Eric Sedlar
> Sent: Thursday, February 28, 2002 7:19 AM
> Subject: RE: A case for GETSRC (or other mechanism to that effect)
>
>

---

From: "Julian Reschke" <[email-masked]>
Date: Thu, 28 Feb 2002 10:16:47 +0100

> Sent: Thursday, February 28, 2002 6:59 AM
> ...

I think the lack of implementations is caused by

- people choose to support the Translate header because they really haven't
a choice (as this is what the Microsoft clients use),

- the description of the source property is buggy and hard to understand
(this could be fixed).

> spaces.  But we quickly run into security and configuration hassles.

I object to the term "fake URL space". Defining separate namespaces for
sources and outputs seems like a logical solution to me.

> with static content you MUST set up a whole new URL space to do DAV

Yes. So? I don't understand why this is a problem per se.

> number of virtual hosts they support just to give decent DAV access

Well, they don't. How this is done is completely implementation-specific,
I'd say.

> ...

I think we need to answer this questions.

Given an ASP file and it's output -- are these two different resources, or
are they different representations of the same resource? In the first case
they'll *need* different URIs, in the other case normal HTTP variant
handling can be used.

Even when you choose the latter -- how do you treat then output resources
with multiple source resources?

---

From: "Eric Sedlar" <[email-masked]>
Date: Thu, 28 Feb 2002 01:24:58 -0800

I think the point is that for every method other than GET, the method
operates on a particular source object, not a set of them.  dav:source is
actually trying to track one level beyond that.  Translate: F, or GETSRC, is
just trying to get back the same contents that were PUT to that location.

The only time Translate/GETSRC is relevant is for executables (cgi's,
servlets, etc.) at a particular URL which generate some dynamic content.
So, there are really three different sets of content we are looking at:

  1)  the generated output returned by GET

  2)  the executable that generates the content for GET, and which is the
object of all other methods (PROPFIND, PROPPATCH, LOCK, UNLOCK, MOVE, COPY,
...)

  3)  the multiple source files that were used to generate the executable

The GET method returns resource #1, dav:source (if it were actually
implemented by anyone) would list #3.  Translate: F, or GETSRC, returns
resource #2, which is the one logically consistent with the other WebDAV
methods.

Actually figuring out #3 is something the equivalent of makefile dependency
tracking, and is not generally available even in a development environment,
which is something that WebDAV is certainly not.

Real-world experience has shown that:

* nobody actually tracks multiple source files for an executable
* people sometimes want to actually get the executable contents

Nobody has actually come up with a real-world use-case for dav:source, but
whether or not dav:source is deprecated or not, it's actually identifying a
different set of content than Translate: F / GETSRC.

And, once you realize that you need Translate: F or GETSRC, you might as
well use Translate: F, as you will have no problem getting interoperable
implementations, and it is functionally equivalent to GETSRC.

--Eric

> From: Julian Reschke [mailto:[email-masked]]
> Sent: Thursday, February 28, 2002 1:06 AM
> To: Eric Sedlar; [email-masked]

---

From: "Julian Reschke" <[email-masked]>
Date: Thu, 28 Feb 2002 11:59:28 +0100

> Sent: Thursday, February 28, 2002 10:25 AM
> To: Julian Reschke; [email-masked]

Actually, with dav:source and multiple URIs PUTting to the non-source result
would result in an error (or at least wouldn't have any effect at all).

> servlets, etc.) at a particular URL which generate some dynamic content.

I don't agree. DAV:source may be relevant for "compiled" resources as well.
For instance, static HTML might be generated by a one-time compilation
process, but you don't want anybody to *modify* the HTML. Rather, people
would need to be pointed to the modifiable source (for instance, XML and
XSLT input files).

> object of all other methods (PROPFIND, PROPPATCH, LOCK, UNLOCK,

I don't think I agree. PROPFIND etc. should access the same resource as GET
(if they are applied to the same URI).

> resource #2, which is the one logically consistent with the other WebDAV

Yes, but dav:source can do this for you as well. Why have two different
models if one is enough?

> which is something that WebDAV is certainly not.

That would only be a problem if WebDAV required that the list of sources is
exhaustive. It doesn't. It's just a hint to a client where to look for files
that actually *affect* the resource when edited.

> different set of content than Translate: F / GETSRC.

I disagree. Could you please explain why the model used in DAV:source can't
be applied to this use case?

> implementations, and it is functionally equivalent to GETSRC.

Using the same URI for "the" source (assuming there is only one) and the
output means that you can't point people explicitly to one of them. As user
agents generally aren't aware of the Translate header (or GETSRC), they
would fetch the output resource, so basically you have a source resource
which you can't refer to.

Actually, having a one-to-one mapping of executables to their output (for
instance foo.asp) is a very bad practice. Everytime you change your
underlying implementation, URIs will change. However, cool URIs don't
change.

Julian

---

From: "Eric Sedlar" <[email-masked]>
Date: Thu, 28 Feb 2002 09:44:31 -0800

> [mailto:[email-masked]]On Behalf Of Julian Reschke
> Sent: Thursday, February 28, 2002 2:59 AM

> would need to be pointed to the modifiable source (for instance, XML and

My point is not that dav:source is relevant (it is clearly useful to look at
the source files as in your example), but whether or not it is useful (three
years of industry experience say clearly it is not) is irrelevant to whether
or not Translate/GETSRC is useful.  You have pointed out a case (like
regular files) where GET and GETSRC would return the same result.  Whether
or not you want to allow people to directly write to compiled output (or an
executable) is a separate issue (which could be handled via an ACL, or the
server could generate an error).

> Yes, but dav:source can do this for you as well. Why have two different

dav:source deals with a completely separate use case.  I clearly need to
separate out the executable at the current URL from the files that generate
it.  In the case of a compiled executable, would you recommend putting both
the executable URL and the .o's & .c's that generate it in the same
undifferentiated dav:source bucket?  It's clearly necessary to separate the
two sets of things, for the case when you can directly PUT to the executable
resource.

> be applied to this use case?

You disagree?  Can you point to any implementations that actually keep track
of the source files?  I'm not saying you can't make up an example, just that
in practice, servers don't keep track of the references to the source.

> which you can't refer to.

So you are using the existing practice argument?  Existing user agents
aren't aware of dav:source either, so in any case we are talking about new
functionality.

> underlying implementation, URIs will change. However, cool URIs don't

What URIs are you talking about?  Can you give an example of this?  Clearly
there is widespread industry support for the one-to-one mapping you describe
(cgi, .asp).

--Eric

---

From: "Julian Reschke" <[email-masked]>
Date: Thu, 28 Feb 2002 19:55:37 +0100

> years of industry experience say clearly it is not) is irrelevant

I disagree. The way it's spec'd is broken. Not the generic concept.

> to whether

The example I gave was not about *forbidding* the modification, it was about
allowing the user agent to *discover* the files on which this
compiled/generated file is based (so it can offer to edit *those* instead).

> separate out the executable at the current URL from the files

Sure.

> two sets of things, for the case when you can directly PUT to the

Yes. So?

For instance, the source property for an executable might contain hrefs to C
sources and a Makefile.

What I'm saying is that the executable and it's output are different
resources (not different representations of the same resource) and thus
should have different URIs.

So, if we would have the URIs

a) <prefix>/sources/hello-world.c
b) <prefix>/sources/Makefile
c) <prefix>/bin/a.out
d) <prefix>/greeting

then c) would have the sources a) and b), while d) has the source c).

> ..

To take the popular use case (IIS and ASP files): an ASP file generates HTML
output, so the source property of the HTML output resource should/can point
to the ASP resource. Just because Microsoft decided a few years ago that
keeping both the output and the script at the same URI (for Frontpage)
doesn't make this the only possible approach.

Generelly, if a server doesn't know the source (or only some of them), or if
it doesn't want to expose them, that's fine. It's just a pointer for a user
agent where to find the editable source(s).

> aren't aware of dav:source either, so in any case we are talking about new

I was saying that we really don't have interoperable implementations, unless
you can properly define what the Translate header means on each single
HTTP/WebDAV method *and* get consensus on that and show that all Microsoft
products are consistent with this.

> there is widespread industry support for the one-to-one mapping

That doesn't mean it's good practice. See [1] (by Tim BL himself).

[1] <http://www.w3.org/Provider/Style/URI.html>

---

From: CJ Holmes <[email-masked]>
Date: Thu, 28 Feb 2002 11:32:53 -0800

>would need to be pointed to the modifiable source (for instance, XML and

Agreed.  I am not arguing against DAV:source at all.  I think it is a 
good idea, and I recognize its architectural superiority.

What I object to is that the current model makes simple things very 
hard to implement and a hassle to manage for no real gain to most 
people.

>Yes, but dav:source can do this for you as well. Why have two different

Nobody is suggesting two models.  I'm suggesting a change in the 
current model.  Instead of GETSRC, let's call it DAVGET just to 
relieve us of some preconceptions.

	GET is undefined by DAV
	PUT and DAVGET are symmetric
	Properties belong the result of DAVGET
	If URI "x" is compiled from sources at other URIs, then:
		DAVGET "x" will retrieve the resource
		PROPFIND "x" will retrieve the properties of DAVGET "x"
		GET "x" would probably return the same resource as 
DAVGET "x" (impl dependent)
		PUT "x" will result in an error (impl. dependent), 
since the sources lie elsewhere.
		PUT to the source URI(s) of "x" will affect the 
properties and value of DAVGET "x"

You can still have DAV:source and it will be just as useful as it is 
now.  But the common problem of how people use DAV to edit their 
source files is resolved.

>be applied to this use case?

It can, but it just isn't worth the trouble.  It is a LOT of extra 
work for absolutely no gain.  You gain nothing from DAV:source that 
you don't get for 1/20th the work from Translate.

If I were to spend a programmer month implementing DAV:source, 
administrators would still have to set aside special URL spaces and 
turn off all plug-ins for that space and add a lot of new security 
realms, just like they do now.  We could implement it, but in terms 
of managing the server we're left with the same mess that already 
afflicts our users.  There's just no point in implementing DAV:source 
unless you have some kind of tightly-integrated document 
management/compilation system that tracks multiple sources for the 
GET results of dynamic URI(s).

And so we're left in a situations where the simplest task is quite difficult.

>underlying implementation, URIs will change. However, cool URIs don't

Nobody is trying to tie down executables to output on a one-to-one 
basis.  Nobody is trying to get rid of DAV:source.  We're just trying 
to adjust the current model to make the simple things easy, while 
maintaining support for the the difficult (and architecturally more 
interesting) things.

cjh

--

---

From: "Julian Reschke" <[email-masked]>
Date: Thu, 28 Feb 2002 21:37:46 +0100

> Sent: Thursday, February 28, 2002 8:33 PM
> To: [email-masked]

I think you need to prove that if you want to introduce a second model for
something that could be solved using DAV:source.

> properties and value of DAVGET "x"

Having a new GET method that returns completely different resources than
GET, and defining that PROPFIND operates on DAVGET rather than GET is not
only a new model, it would change WebDAV at it's core. Why do this if this
isn't needed????

> source files is resolved.

It can *easily* be solved using DAV:source.

> you don't get for 1/20th the work from Translate.

That's a broad claim. Can you prove it?

> If I were to spend a programmer month implementing DAV:source,

Why would that take so long?

> turn off all plug-ins for that space and add a lot of new security

No, they don't.

Nobody says that they *have* to set aside a special URL space.

For instance, for any "output" resource "x", you could have a source
resource "x" + ".~~~~~source~~~~~" which represents the editable source code
of this resource. No separate URL space.

> GET results of dynamic URI(s).

The only difference is that you might have more than one source. It escapes
me why the special case of having just one source justifies breaking basic
Web axioms by having different resources with the same URI.

> And so we're left in a situations where the simplest task is

No, it's not.

Maybe you could step back and describe why you think it's so much more work
generating a second URI for the editable source resource.