Subject: Re: Why not an encapsulation for DAV over standard HTTP 1.0 or 1.1 without required server extension ?

From: "Jim Whitehead" <[email-masked]>
Date: Fri, 25 May 2001 10:45:34 -0700

Accidentally caught by the spam filter. I've added
[email-masked] to the accept2 list.

- Jim

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]
Sent: Friday, May 25, 2001 10:21 AM
To: [email-masked]
Cc: [email-masked]
Subject: [Moderator Action] Re: Why not an encapsulation for DAV over
standard HTTP 1.0 or 1.1 without required server extension ?

Larry Masinter wrote:
>
> The decision to use separate methods instead of just POST (which
> I think was the original question) was arrived at after a lengthy
> discussion; the first record I can find is from October 1996:
>
http://lists.w3.org/Archives/Public/w3c-dist-auth/1996OctDec/subject.html#25
>
> Note that SOAP takes a different view; it only uses POST. I don't
> think the considerations have changed substantially, except perhaps
> that we're talking about using other headers and/or message body
> elements rather than content-type to do method dispatching.

WebDAV is ugly precisely because the HTTP and XML are horribly entangled.
It's defined as a protocol, but the stack is not layered.
It really implies a data model, yet that data model is not rigourously
defined.
It has taken 5+ years to define, standardize, and implement.

In those 5 years the world has changed.

SOAP evolved from XML-RPC, which was largely invented to perform site
and content management at Userland, so the original domain and requirements
are similar to WebDAV. In much less than those 5 years, SOAP has evolved
from XML-RPC, been adopted by major corporations (Microsoft, IBM, etc.),
is reasonaly standardized and on its way through the W3C,
become the basis of all web services architectures,
acquired a complete layered stack (including WSDL and UDDI),
and is likely to be the basis of all future web-delivered functionality !

In the history of computing, it has often been the case that a
general purpose solution, bound to a specific application domain,
has delivered a more versatile, powerful, and ultimately longer-lived
solution, than a bespoke specialized language or protocol.
There are some high-value exceptions, like Verilog and VHDL,
but flexibility usually wins in a Darwinian world requiring constant
incremental adaptation.

SOAP-based web service architectures rely on a design-driven process
of data modelling/schemas, selecting an interface to publish,
followed by auto-generation of components for service discovery,
messaging, data (un)marshalling, method dispatching and invocation.

If WebDAV had a data model, it would be an almost trivial task to
create a web interface definition, auto-generate the communication
components and publish it as an web service !

Who can doubt that if WebDAV was starting today,
it would be defined as a SOAP-based service ?

Mikhail

__________________________________________________________________
Get your own FREE, personal Netscape Webmail account today at
http://webmail.netscape.com/

---

From: "Jim Whitehead" <[email-masked]>
Date: Fri, 25 May 2001 14:23:31 -0700

Can't resist...

> It's defined as a protocol, but the stack is not layered.

You could make the same claim about HTTP, that it horribly entangles RFC-822
headers in with the request line of the message, without any firm layering.

The simple fact is WebDAV defines a set of methods. These methods take
parameters. Some parameters are encoded in headers, others in XML. People
who have written DAV implementations don't have a problem with this.

> It really implies a data model, yet that data model is not rigourously

This is pretty typical of IETF standards, which are concerned with
on-the-wire encoding, and are less interested in the underlying data model.
The result is an implied, bot not normative data model. Thus, for example,
even though DAV implies a data model of a resource payload plus properties,
it is possible to implement this is a number of ways (Unix files + property
database, NT filesystem files, database tables, as objects only comprised of
attributes where a single distinguished attribute holds the payload).

> and content management at Userland, so the original domain and
> requirements are similar to WebDAV.

You're confusing a marshalling and basic transport mechanism (SOAP/XML-RPC)
with a semantic-rich protocol. XML-RPC and SOAP have almost no semantics
whatsoever.  The marshalling decisions are not the ones we spent most of our
time on -- it was deciding things like "how does a locked hierarchy interact
with move" and "what are the semantics of move/copy on live properties".
These are tricky, subtle issues which require a lot of time to sort out in a
consensus-driven standards group that is open to input from all comers, and
uses a transparent process.

For example, I'll note that people in W3C working groups feel no compulsion
to answer random questions from newbies questioning 4 year old design
decisions. Think about it.

> In much less than those 5 years, SOAP has evolved

Actually, I would trace the beginning of SOAP to the failed HTTP-NG effort
at the W3C, which was another attempt to define a "service-oriented"
infrastructure for the Web. It is no cooincidence that Henrik Nielsen was
heavy involved in HTTP-NG, then went to Microsoft, and became very heavily
involved in SOAP definition.

Furthermore, SOAP is not nearly as far along as WebDAV in terms of adoption.
Right now, people are using WebDAV in real-world settings to get their work
done. There has not yet been widespread adoption of SOAP. I have no doubt
that, given the marketing push by Microsoft and IBM, this will happen. But,
it's not there yet, and it will take some time yet before SOAP is at the
same level of standardization and adoption as WebDAV.

> solution, than a bespoke specialized language or protocol.

So this is why Internet-based services took off using infrastructures such
as RPC, ILU, and IIOP, which were also language independent marshalling
formats, right? I think it is incorrect to draw any technical lessons from
the adoption of SOAP. What makes SOAP different from RPC/IIOP is that
Microsoft and IBM both agree on this one. That is, it is the non-technical
issue of sponsorship of the standard that far outweighs any technical
properties of the solution. Yes, the use of HTTP POST and XML for
marshalling undoubtedly helped Microsoft and IBM agree to use SOAP,
especially given the mindshare of XML in the 1999-2000 timeframe (I'll note
that the same SOAP proposal would have flopped in, say 1995-96, when XML was
still tainted by the stigma of SGML).

> messaging, data (un)marshalling, method dispatching and invocation.

Except for service discovery, this is *exactly* what you did using CORBA and
RPC.  You created a data model, described interfaces, compiled those
interfaces into marshalling and unmarshalling stubs, with method dispatching
handled automatically.  So, why didn't these frameworks take off for
Internet use? Claiming that they don't go through firewalls is a red
herring -- if a company wants to host a service, or use a service, and have
a strong business need to do so, they'll make sure packets can get through.

> components and publish it as an web service !

Again, the marshalling issues were not the heart of the complexity of the
WebDAV protocol, and are not the heart of the complexity of WebDAV
implementations.

> it would be defined as a SOAP-based service ?

Me. I'm going to be very glad that, when administrators tell their firewalls
to not admit POST (so someone in the company doesn't set up a rogue Web
service and introduce a security hole), WebDAV will not be affected.

- Jim

---

From: Dan Brickley <[email-masked]>
Date: Fri, 25 May 2001 22:59:30 +0100 (BST)

> decisions. Think about it.

[Thinks about it... Tries to get annoyed... fails...]

Thanks for the thoughtful sideswipe Jim! I might add that people in W3C
working groups always decide things in secret unminuted meetings, are
the happless pawns of industry tyrants, and plot to undermine the plucky
freedom loving IETF groups at every opportunity. There, the secret's
out.

"people in W3C working groups" vary, as do the working habits of those
groups.  Only some of us are evil. Some of us are also people in IETF
WGs...

All I know is that when I try to answer questions from newbies asking
about 4-year old design decisions (eg. RDF), it's not through
"compulsion" but through common sense and (mailbox overload
permitting) common politeness. And when I fail to manage the same, more
likely its not because I'm on a W3C group, but because I'm human...

While I could pass the time bickering amiably about W3C and IETF, it
might nore interesting to try to get something more productive done. How
about we revisit the old "how do WebDAV and RDF work together" faq? I've
been hearing some nice things about Delta-V lately, maybe we could take
that as practical application where RDF and WebDAV could be shown
working together? As a minimum, I'd like to try to get a student project
or two looking at building some prototypes. What do you reckon?

Dan

--
industry pawn
http://purl.org/net/danbri/

---

From: "Jim Whitehead" <[email-masked]>
Date: Fri, 25 May 2001 15:08:35 -0700

*wiping egg from face*

The only purpose of my comment was a minor swipe at a well-known difference
between IETF and W3C process (openness of membership, and voting vs. rough
consensus). Given that it is well known, and that discussion of W3C vs IETF
process is off-topic for this list, I shouldn't have even said anything in
my post.

I will freely admit that there is large overlap between W3C and IETF working
groups, and that they are composed of very sharp, dedicated technical people
working on gnarly, tough problems to produce high quality technical
specifications.

I will also freely admit that there are times when the W3C process achieves
better results than the IETF process, HTML being an excellent example.

> might nore interesting to try to get something more productive done.

Agreed. I don't know what got into me today.

- Jim

---

From: "Jim Whitehead" <[email-masked]>
Date: Fri, 25 May 2001 15:22:43 -0700

> How
> about we revisit the old "how do WebDAV and RDF work together" faq?

Sure.

One area of overlap is use of WebDAV Properties to store Dublin Core
metadata.  I went to the Dublin Core site this morning to try and determine
the current status of the RDF representation of Dublin Core, and wasn't able
to find a document describing a standard representation of DC in RDF.  Does
such a document exist?  I found one working document
<http://www.ukoln.ac.uk/metadata/resources/dc/datamodel/WD-dc-rdf/> -- can I
do anything to help get that moved along (review a document, write an email,
etc.)?

Beyond that, it seems to me that use of RDF in DAV properties requires
providing some guidance on packaging issues. Some issues include:

* If I have a chunk of RDF, should I store it in one property, or in
multiple properties? What are the tradeoffs?
* How would XML-RPC via the DASL SEARCH method interact with RDF in DAV
properties?
* When should a chunk of RDF be its own resource, and when should it be
stored as properties?

> As a minimum, I'd like to try to get a student project

Sure, this could be interesting.  Many of the relationships between
resources in DeltaV could be represented, in an implementation, using RDF.
It would be interesting to see if an apps. framework like Redland could be
used as an implementation engine for a DeltaV server.

- Jim

---

From: "Jim Whitehead" <[email-masked]>
Date: Fri, 25 May 2001 15:43:07 -0700

> > or two looking at building some prototypes. What do you reckon?

Another idea is to tweak the RDF photo metadata project code, described at:

http://www.w3.org/TR/photo-rdf/

To use DAV properties.  At present, it returns the RDF metadata using HTTP
content negotiation -- ask for text/rdf or text/* and you get RDF, otherwise
you get the image.  Seems like it wouldn't be that difficult to add code to
have Jigsaw to output this metadata information in a DAV property...

- Jim

---

From: Dan Brickley <[email-masked]>
Date: Sat, 26 May 2001 01:29:28 +0100 (BST)

> do anything to help get that moved along (review a document, write an email,

Nudging my conscience in this fashion is as good a mechanism as any :)

The DC/RDF work did indeed languish for a while. Recently we've been
getting this back on track, with a new draft that makes better use of
RDF's in-built facilties. The Website (particularly the dc-architecture
pages, my responsibility) doesn't do the job it should in reflecting our
current thinking. 

Anyhow, the current draft is looking good, imho. I've recently started a
push for implementation testing, and would be glad to involve WebDAV
folks in the review process. It's hard to know what would count as
success/exit criteria, so feedback from this WG would be valued. See my
msg from Weds for link to the latest draft. I intend to rectify the
dc-architecture Web page situation asap.

http://www.jiscmail.ac.uk/cgi-bin/wa.exe?A2=ind0105&L=dc-architecture&F=&S=&P=1852
(hmm... .exe in a url, classy...)

> multiple properties? What are the tradeoffs?

Yes, that's the trouble with big squiggly graph structures; they're
great on the data-merging front, but when it comes to partitioning the
graph into bite-sized chunks, the chunking strategy often seems somewhat
arbitrary. I doubt we'll come up with one crisp answer that works for
everyone. In the simple unqualified DC front, there's often a mapping
onto single properties. For such purposes, my colleague Dave Beckett has
worked on a simple DC-in-XML format that parses as XML, yet has a more
predictable syntactic representation than arbitrary XML. This might
prove of interest in the WebDAV world...

http://dublincore.org/documents/dcmes-xml/

> * How would XML-RPC via the DASL SEARCH method interact with RDF in DAV

I've not followed any discussion about XML-RPC-meets-DASL, though the
idea (or with SOAP) sounds plausible. In RDF-land, we're starting to
think about ways in which RDF query might have commonalities with XML
Query, despite the difference in data models, database structure, query
optimisations etc. I need to learn more about DASL before attempting to
have a sensible answer on this.

> * When should a chunk of RDF be its own resource, and when should it be

Related to the one-property vs many properties issue. One might ask the
same question for XML docs on a webserver. Again, I don't know how we'd
tell a good answer from a bad one. Pre RDF-meets-WebDAV, RDF docs are
just new resources. If we could show some way of projecting them into
the WebDAV attribute/value model, then the stored as properties approach
(perhaps lossily, eg. just extracting DC) might make sense too.

> used as an implementation engine for a DeltaV server.

Cool! I'd like to see that attempted...

Is there any test data we could play with? One problem we've had
previously is little flurries of interest in this topic, with little
follow through (we're all busy...). Getting the problem described
adequately might be a step towards enticing someone to have a serious
crack at solving it.

I have a personal interest in document versioning, and the
representation of workflow and transformation metadata as events,
descriptions, with agent, date, time etc associated with those
events. But not enough time to play with implementations :( It'd be
really nice to have some Delta-V data for RDFists to take a look at though...

Dan

---

From: Dan Brickley <[email-masked]>
Date: Sat, 26 May 2001 01:39:09 +0100 (BST)

> have Jigsaw to output this metadata information in a DAV property...

Good idea! There's been a little discussion of this on the sinister and
secretive W3C www-annotation list(*), 
http://lists.w3.org/Archives/Public/www-annotation/2001JanJun/

Since Yves is involved in RDFPic, and Libby works with me at ILRT, this
might prove fun. I bought a digital camera recently so I'm all in
favour of better ways of interacting with such content. And I happen to
be sat on top of a wealth of interesting if unsettling image data +
metadata (http://www.brisbio.ac.uk/) so there's certainly testbed
content available, again perhaps student project material? 

Dan

(*) signup: send a secret handshake..
	To: [email-masked]
	Subject: subscribe

---

From: [email-masked]
Date: Tue, 29 May 2001 12:49:40 -0400

"Jim Whitehead" <[email-masked]> wrote:
>
> > WebDAV is ugly precisely because the HTTP and XML are 
> > horribly entangled. It's defined as a protocol, 
> > but the stack is not layered.
..
> who have written DAV implementations don't have a problem with this.

There seems to be a level-mismatch involved in changing the transport layer in order to implement high-level services, especially when it's a
stateless transport layer with extremely stateful high-level functionality.

This leads to awkward practical problems, such as: having to change the
source code of the web server to let through new methods and status codes;
having to rewrite http client libraries in order to send the extra
headers and methods (e.g. java.net cannot be used for WebDAV).

And these are not one-off changes, since every time a new WebDAV extension
comes out, the methods, extra headers, and status codes all change.
Of course, the second rewrite is implemented as a configurable set of
properties, perhaps as it should have been from the start :)

Perhaps the biggest benefit of layering is it enables reuse,
i.e. other people are developing the other pieces: from standards,
through to implementations, products and open source projects. 
Less work should be a powerful motivation to achieve orthogonality.

> > It really implies a data model, yet that data model is not rigorously
> > defined.

> on-the-wire encoding, and are less interested in the underlying 
> data model. The result is an implied, but not normative data model. 

> The marshalling decisions are not the ones we spent most of our
> time on -- it was deciding things like "how does a locked hierarchy 
> interact with move" and "what are the semantics of move/copy on 
> live properties". These are tricky, subtle issues which require a lot of
> time to sort out.

I agree the model and semantics are harder than the transport,
but all the more pity that at the end of a long process of discussing the
real details of the model, you do not actually have a normative model in
black and white, to hand to someone who wants to model in UML, RDF or ER.
The work has been done, but the clarity is obscured by recasting the
answers as a protocol. It also makes the discussions, and the
final standard, very obtuse, because you get circumlocutions such as:

"method postcondition: the object (which we do not really have), 
has a state (of which we dare not speak its name) of a certain 
(but unspoken) type, and the final value of this pseudo-state on this pseudo-object, should probably be X (but we can't insist)."

Of course, XML has almost been broken trying to make the transition
from serialization format to model-centric system (infoset, schema, query).
I would argue this is more to do with the explosion of XML politics,
rather than any deep technial problems, but it's hard to be sure.
Perhaps the best summary of the clash of cultures between the 
pointy-bracket syntax-driven doc-heads and the curly-brace class-conscious
object-geeks, is Ron Bourret's witty "myths":

  http://www.xml.com/pub/a/2000/03/08/namespaces/index.html
  
> > solution, than a bespoke specialized language or protocol.
..
> I think it is incorrect to draw any technical lessons from
> Microsoft and IBM both agree on this one. That is, it is the 
> non-technical issue of sponsorship of the standard that far 
> outweighs any technical properties of the solution. 

I agree that politics are more important than technical properties.
But there is another important pratical aspect, and that is the 
general 'developability' of the technology: is it flexible ? is it
easy to start developing solutions using standard tools ? does it
evolve smoothly to encompass new requirements ? does it compose
orthogonally with other layers and components ? etc.

> > messaging, data (un)marshalling, method dispatching and invocation.
..
> > components and publish it as an web service !

There's another pathway which might be more appropriate, 
and that is to specify WebDAV methods as WSDL interfaces,
then use various class generators to spit out the comms components.
This has the merit of being a closer match to the protocol, does not
require a complete data model, is language independent, and should be
able to bind to most existing data model implementations (given a 
flexible enough WSDL mapper-compiler).

Here are a couple of references for (quick hack) DAV-ish web services:

  http://www.xmethods.net/xfs/
  http://www.soapware.org/xmlStorageSystem

And thanks for answering a newbie's frustrations !

Mikhail
__________________________________________________________________
Get your own FREE, personal Netscape Webmail account today at http://webmail.netscape.com/