Subject: RE: Move and Delete (was: bind draft issues)

From: Jason Crawford <[email-masked]>
Date: Tue, 4 Mar 2003 19:35:32 -0500

> > Right.  To claim BIND spec support, you have enhance the implementation
to
> > support *this*  2518 compliant approach.... or to avoid a situations
where
> > bind spec
> > violations would happen.   It's not unreasonable to expect one to have
to
> > add code
> > in order to support a new feature like multiple-bindings.
>
> Actually, I'd have to remove code.
>
> The issue isn't that the server can't *technically* do this -- it's that
the
> this semantics isn't compatible with the existing DELETE semantics of our
> system. Just removing the collection binding without checking that all
> internal members may be deleted as well simply is *not* an option.

Okay, so I'll stop talking about symbolic links :-)

If there are multiple bindings to a collection and a WebDAV request asks
you do DELETE one of them, will your server end up deleting all the
descenent bindings also... or just the one requested?

Is it not possible to freeze the system while your system checks this?
Perhaps in a way similar to the way the Win32 has to before it allows
a MOVE.  In its case, is seems to find it acceptable to delay while
it's checking this.  It can cause a delay of a few seconds, but apparently
the folks that implemented that found it acceptable.

> So basically we have the choice between
>
> - supporting multiple bindings, but not the BIND method described in the
> draft, or
> - simply ignore this requirement.

Or I suppose, "not allow multiple bindings to collections?"

> I'd rather have a spec that I can implement without having to break a
> specific requirement (in particular if i'm not convinced it adds value to
my
> use case).

The other day I ran in to this in a non-WebDAV situation.  I tried to move
a huge
directory.  I found it frustrating to find that it was moving the files one
at a time and
ran in to a problem and left me with two partial trees.  (This was across
file systems.)
I would have been similarly, although less problematically, frustrated if
it had been
a delete request.

------------------------------------------
Phone: 914-784-7569,   [email-masked]
I do not check [email-masked]

---

From: "Julian Reschke" <[email-masked]>
Date: Wed, 5 Mar 2003 01:48:04 +0100

ailto:[email-masked]]
> Sent: Wednesday, March 05, 2003 1:36 AM
> To: Julian Reschke
> Cc: Clemm, Geoff; 'WebDAV'
> Subject: RE: Move and Delete (was: bind draft issues)
>
>
>
>
>

If it's one of many bindings: just the binding. If it's the last binding, it
will do a best effort DELETE as allowed in RFC2518.

> the folks that implemented that found it acceptable.

A MOVE is a simple namespace operation. All it needs to do is check locks.

A DELETE that cleans up in the foreground will need to check delete
privileges on all descendants. This set can be very huge. I think it's an
extremely bad idea to do this in a single transaction (yes, we tried).

> Or I suppose, "not allow multiple bindings to collections?"

Even then I can't complain conformance to the spec. It doesn't allow me to
have a best effort DELETE on collections at all.

> my use case).

I notice you keep mention MOVE. We don't disagree here. MOVE must be atomic.
If you can't MOVE, let if fail. If it fails the client still can do a COPY
and the DELETE the source.

DELETE is really a separate topic!

Julian

--
<green/>bytes GmbH -- http://www.greenbytes.de -- tel:+492512807760

---

From: Brian Korver <[email-masked]>
Date: Tue, 4 Mar 2003 17:22:54 -0800

On Tuesday, March 4, 2003, at 04:48  PM, Julian Reschke wrote:
[snip]
> extremely bad idea to do this in a single transaction (yes, we tried).
[snip]

Julian,

I agree, but I think it's even worse than this.  Semantically,
MOVE is merely a simple namespace operation, but in practice
it may be more than that.  For instance, a move across unix
filesystems must be implemented as a recursive copy (followed
afterward by delete).

-brian
[email-masked]

---

From: "Julian Reschke" <[email-masked]>
Date: Wed, 5 Mar 2003 09:21:57 +0100

> From: [email-masked]
> [mailto:[email-masked]]On Behalf Of Brian Korver
> Sent: Wednesday, March 05, 2003 2:23 AM
> To: 'WebDAV'
> Subject: Re: Move and Delete (was: bind draft issues)
>
>
>

Brian,

in the Unix API, you don't move at all -- you link() then unlink(). "mv" is
just a user command that does it's best when the files reside in different
partitions. I think WebDAV MOVE should just fail if the resource cannot
really be moved (preserving all dead & live properties), and fail otherwise.
Just like in the Unix API, the caller *then* can decide to do a COPY/DELETE
instead.

Julian

---

From: Brian Korver <[email-masked]>
Date: Wed, 5 Mar 2003 12:06:00 -0800

Julian,

I don't think we can push this behavior out to the client.

For the sake of illustration, let's take an hypothetical
repository built on a unix filesystem:

   Filesystem  1K-blocks     Used    Avail Capacity  Mounted on
   /dev/da0s1f    992239      372   912488     0%    /users/briank
   /dev/da0s1g   3048942  2509636   295391    89%    /users/briank/MP3s

Now, let's say I've got a file sitting at /users/briank/tmp/groovy.mp3
that I'd like to copy to my MP3s collection.  What is my WebDAV
client going to do?  We have two cases:

   1)  The server is "smart" and performs a copy/delete

       Ok, in this case my client issues a MOVE, the server performs
       a copy/delete (NOT rename(2) [link/unlink]).  But notice, this is
       a server-side copy/delete, not client-side.

   2)  The server isn't "smart" and thus the client must
       perform a COPY/DELETE

       What happens here?  The client issues a MOVE, but this
       must fail because rename(2) doesn't work across file
       systems.  What error does the server return to tell
       the client that a COPY/DELETE must be performed instead?
       What is the point of this extra round trip?

-brian
[email-masked]

---

From: "Julian Reschke" <[email-masked]>
Date: Wed, 5 Mar 2003 21:34:21 +0100

> Sent: Wednesday, March 05, 2003 9:06 PM
> To: 'WebDAV'

I guess you meant to say "move" here.

>        the client that a COPY/DELETE must be performed instead?

I guess we want a specific precondition failure that can be signaled. For
instance, DAV:preserve-versioning-properties.

>        What is the point of this extra round trip?

The same thing as the failure you'll get upon trying to rename. MOVE is
*not* the same thing (or should not be the same thing) as COPY/DELETE. If a
MOVE can't preserve the resource's live properties, it should fail.

Now I *do* agree that in many cases clients will actually *want* the "weak"
MOVE. So maybe we should consider supporting both (either by a new method,
or by adding parameters to MOVE).

Julian

---

From: Brian Korver <[email-masked]>
Date: Wed, 5 Mar 2003 13:48:27 -0800

> I guess you meant to say "move" here.

Oops.  Yes.

> or by adding parameters to MOVE).

That's an interesting idea.

-brian
[email-masked]

---

From: Brian Korver <[email-masked]>
Date: Wed, 5 Mar 2003 14:35:07 -0800

[snip]

Julian,

Were you thinking that this header (say "Atomic-Operation:") would be
used for only MOVE, or for all of the relevant operations (COPY,
DELETE, etc.)?

-brian
[email-masked]

---

From: "Julian Reschke" <[email-masked]>
Date: Wed, 5 Mar 2003 23:51:06 +0100

> Sent: Wednesday, March 05, 2003 11:35 PM
> To: 'WebDAV'

Actually, I'd really prefer not to define additional headers.

Thinking of it, we *also* can't agree on the right DELETE semantics (see
separate discussion). So one way to address this would be to leave DELETE
and MOVE as they are, and to add

- UNBIND (that really really really removes bindings, thus has the DELETE
semantics currently specified by the BIND draft) and
- RENAME (which would be a true MOVE that would fail when the server can't
implement it as internal namespace operation).

This would make discovery of the new functionality much easier.

Julian

---

From: Brian Korver <[email-masked]>
Date: Thu, 6 Mar 2003 14:55:49 -0800

Julian,

If it is the goal to have WebDAV implement the semantics of the
Unix API, then I would prefer that it more closely model those
semantics -- for instance by making DELETE fail on non-empty
collections.

Personally, I don't think that WebDAV should implement the
semantics of the Unix API, at the very least because of
the overhead -- imagine having to issue a DELETE for every
resource in order to delete a collection containing a million
files.  I feel that WebDAV should implement semantics that
are closer to the Unix CLI, which of course was designed to
be easier on the fingers (read: "lower overhead") than
the API.

-brian
[email-masked]

---

From: "Julian Reschke" <[email-masked]>
Date: Fri, 7 Mar 2003 09:57:50 +0100

> Sent: Thursday, March 06, 2003 11:56 PM
> To: 'WebDAV'
> ...

No, that's not the goal (at least I don't think so).

> resource in order to delete a collection containing a million

Correct. The overhead my be ok in a local fs, probably is problematic with a
network filesystem, and won't be acceptable for WebDAV.

> be easier on the fingers (read: "lower overhead") than

Yes and no. I really think that there may be separate use case that require
separate solutions.

A client should be able to request a MOVE operation that will fail if the
server can't support full MOVE semantics (for instance by changing the
DAV:resource-id property). A client would then be able to reconsider,
possibly issuing a COPY/DELETE sequence instead.

Hopefully we can agree that this type of request should be supported. If we
do, we're left with the alternatives of making this the standard behaviour
for the RFC2518 MOVE, or to invent some new marshalling. I'd prefer the
former, but I probably could live with the latter.

A similar problem obviously exists with the DELETE collection atomicity
issue.

---

From: Brian Korver <[email-masked]>
Date: Fri, 7 Mar 2003 12:22:14 -0800

On Friday, March 7, 2003, at 12:57  AM, Julian Reschke wrote:
[snip]

Julian,

We are in agreement here.

Even if a server is able (or desirable) to support atomic operations
in some cases, there needs to be a way for these servers to communicate
to clients when an atomic operation cannot be performed so that
the client can retry using the non-atomic operation (if so desired).
MOVEs across unix file systems are one example, but (as Jason pointed
out to my off-line), so are cross-repository MOVEs in the case of
cross-repository bindings.  A client has no way to know beforehand
whether a cross-repository MOVE will succeed, but may want to attempt
one before performing a COPY/DELETE instead.

-brian
[email-masked]

---

From: "Lisa Dusseault" <[email-masked]>
Date: Fri, 7 Mar 2003 13:37:05 -0800

> extremely bad idea to do this in a single transaction (yes, we tried).

Actually, collection MOVE suffers from the same problems as collection
DELETE.  Here are some of them.
 - You can't MOVE a file you don't have permission to write. Therefore,
the server must check write privileges on all descendants.
 - A WebDAV application server may be used to unify a number of back-end
storage servers under the same namespace.  So the server's URL
http://www.example.com/software/ may contain files stored on
//davfiles/software, whereas the server's URL
http://www.example.com/userdocs/ may contain files stored on
//davfiles/userdocs.  A server moving a collection from one of these
top-level-directories to the other would have to do a network
cross-repository move under the covers. This may not be feasible to do
atomically.

Lisa

---

From: "Julian Reschke" <[email-masked]>
Date: Sat, 8 Mar 2003 08:40:51 +0100

> [mailto:[email-masked]]On Behalf Of Lisa Dusseault
> Sent: Friday, March 07, 2003 10:37 PM
> To: 'Julian Reschke'; 'Jason Crawford'
> Cc: 'Clemm, Geoff'; 'WebDAV'

I think I disagree. I *can* MOVE a resource without having write permissions
to the resource itself. What I need is the write permission on source and
targer collection (+ some more when overwriting something else).

> //davfiles/userdocs.  A server moving a collection from one of these

Sure.

> cross-repository move under the covers. This may not be feasible to do

Yes, that's why many people would prefer it to fail (leaving the client the
choice to explicitly COPY/DELETE) rather than doing something the client
din't really want.

---

From: "Lisa Dusseault" <[email-masked]>
Date: Sat, 8 Mar 2003 11:03:21 -0800

> targer collection (+ some more when overwriting something else).

That's really interesting.  This is a tangent, but an important one, I
think.  I had assumed that if I have 'write' permission on a collection
"home/", but I do *not* have write permission on an internal member
"foo.txt", that an attempt to MOVE home/foo.txt (either to another
collection or a rename) would fail.  In the absence of the ACL
specification, my assumption is as good as yours!  

Now, I did check the ACL specification to see how servers supporting
that specification would have to behave. I don't think the ACL
specification is totally clear on that issue.  Is this a bug in the ACL
draft?  Even if we can't specify one or the other, the ACL draft should
specify that a user may or may not need write permission specifically on
a resource to move it, as well as write permission to the source and
target parent collections.

But back to the atomic vs. non-atomic move, the permissions example was
just an example.  You still have to recursively check privileges on all
descendent _collections_ if one accepts your permissions model.  And you
still have to recursively check locks, etc.  The basic argument for
non-atomic move was that in some situations, the server will have to
implement it as a super-user COPY-->DELETE.

> choice to explicitly COPY/DELETE) rather than doing something 

Right, but did you read the part about how the server might be able to
do a much better job of the under-the-cover copy-and-delete than the
client possibly could?  A client can only do operations based on its
authenticated context, whereas the server process may have more power to
change things.  My first example, and it's only one of many, was that a
client may not have permission to set "creationdate" but a server
obviously can.  So if the server accomplishes the MOVE by doing a
copy-and-delete behind the scenes, the server can clean up
"creationdate" so that it's the date of the original resource. However,
the client may not be allowed to set the value of "creationdate" to the
most correct value after it does a manual COPY operation.

Lisa

---

From: "Julian Reschke" <[email-masked]>
Date: Sat, 8 Mar 2003 20:16:06 +0100

> Sent: Saturday, March 08, 2003 8:03 PM
> To: 'Julian Reschke'; 'WebDAV'
> Cc: [email-masked]
> Subject: Atomic MOVE and move permissions
>
>
>

Yes. Eric, could you please publish the latest state of the ACL spec (which
includes the required-permission-per-method table)?

> target parent collections.

I think a rename operation should not require write access to the resource
itself (it only affects the state of the parent collection). I also think
this was the consensus of the attendees of the January meeting.

> descendent _collections_ if one accepts your permissions model.  And you

Why?

> still have to recursively check locks, etc.  The basic argument for

Yes.

> implement it as a super-user COPY-->DELETE.

Yes. And the basic argument for RENAME is that in some cases a client
doesn't *wan't* that to happen, because the resulting resource may behave
differently than the original.

> most correct value after it does a manual COPY operation.

Yes. That's why the current proposal says that MOVE can use the RFC2518
semantics, while REBIND will be really atomic.

Julian