Subject: RE: MOVEs across file systems

From: "Clemm, Geoff" <[email-masked]>
Date: Mon, 10 Mar 2003 14:40:19 -0500

I think the comments in the recent threads have pointed to
the appropriate path forward.  In particular, we can replace
the SHOULD statements with stronger MUST statements that have
a narrower target (SHOULD statements are pretty useless in a
spec anyway, so getting rid of them is a good thing).

In particular, instead of saying:

  "DELETE SHOULD be UNBIND if UNBIND is supported"

we should say something like:

  "When DELETE is applied to a collection, it MUST NOT modify the
   membership of another collection, except when the collection
   being deleted is itself a member of that other collection.

   For example, suppose /a/b/.../x identifies a collection C, and there
   is a second binding to C in a collection that is not a member of
   /a/b, then "DELETE /a/b" MUST NOT delete the internal member
   named "y" from C.

And instead of saying:

   "MOVE SHOULD be REBIND if REBIND is supported"

we should say something like:

   "When MOVE is applied to a resource, the other bindings
    to that resource MUST be unaffected, and if the
    resource being moved is a collection, the bindings to any
    members of that collection MUST be unaffected.
    Also, if MOVE is used with Overwrite:T to delete an
    existing resource, the constraints specified for DELETE apply."

Clearly, these constraints can be addressed by implementing DELETE
as an UNBIND and MOVE as REBIND, but in those situations that are
"safe" (i.e. there are not multiple bindings to the resources being
moved or deleted), the "best effort" semantics allowed by 2518
can be applied.

Is this an acceptable compromise?

Cheers,
Geoff

-----Original Message-----
From: Jason Crawford [mailto:[email-masked]]

> In this case, the resource at the target URL of the MOVE request
> would *not* be the "same" resource anymore (it would have a
> different DAV:resource-id).

We might have to accept that for the single binding cases and
encourage them to do better, but if they don't support cross-file
system bindings and they do a move to another file system and the
source resource also has additional bindings to it from the original
file system, then the request MUST be rejected.  If it is not, the
server can not claim to support the bind spec.

---

From: Jason Crawford <[email-masked]>
Date: Mon, 10 Mar 2003 15:15:34 -0500

> If you want me to to explain, you'll have to point me to the part you
don't
> understand :-)
>
> Assume a Unix fs, /a and /b being on different filesystems.
>
> ln /a/a /a/b
>
> will work.
>
> ln /a/a /b/a
>
> won't.
>
> However
>
> mv /a/a /b/a
>
> will work. However the result will not be the "same" resource (it will
have
> a different inode, and will reside in a different filesystem).

Thanks.  That; explains it for me.  Although not ideal since you said it
would
change resource ID's and we know it might not "fully complete", that is
probably
acceptable if there are no resources with two bindings to them being moved.
And even that's fine if the diamonds are totally internal to the moved
subtree.
As mentioned though, it is not acceptable if it breaks bindings that are
not even
part of the subtree being MOVE'd.   This is not something a client can
recover
from.

> > > would *not* be the "same" resource anymore (it would have a different
> > > DAV:resource-id).
> > We might have to accept that for the single binding cases and encourage
> > them to do better, but if they don't
> > support cross-file system bindings and they do a move to another file
> > system and the source resource also has additional bindings to it from
the
> > original file system, then the request MUST be rejected.   If it is
not, the
> Well, I think that's up to discussion. MOVE suffers from it's RFC2518
> definition (COPY then DELETE). REBIND doesn't. I don't see a problem with
a
> system supporting both (REBIND will fail for some operations, while MOVE
> won't).

:-)  Have we scared everyone away from this discussion yet?  Okay folks,
jump
right in.  :-)

---

From: "Dyer, Kevin" <[email-masked]>
Date: Tue, 11 Mar 2003 08:48:31 -0500

Just putting in my 0.02 Galactic credits to make sure we keep in mind that
some implementations will not be limited by a file system paradigm as the
underlying engines will provide an abstraction to the actual source data.
These systems use meta-data as the pointer to the actual source data which
can exist on completely different servers and file systems. So moving data
across a resources is as easy as creating a DB relationship. The actual
source data is not moved and the system retains one version of the truth.

Kevin 

  >From: Clemm, Geoff [mailto:[email-masked]]
  >Sent: Monday, March 10, 2003 3:18 PM
  >To: 'WebDAV'
  >Subject: RE: MOVEs across file systems
  >
  >
  >
  >That would be fine with me as well.
  >
  >Just to be clear, this means the binding spec would state:
  >
  >A server that supports BIND MUST implement MOVE/DELETE with
  >rebind/unbind semantics.  We will also define a parameter to
  >MOVE/DELETE that allows a user to explicitly request the
  >"best effort" style processing (that is OK because the user is
  >explicitly stating that trashing multiple binding semantics is
  >what they want).
  >
  >Cheers,
  >Geoff
  >

---

From: "Clemm, Geoff" <[email-masked]>
Date: Tue, 11 Mar 2003 09:25:55 -0500

Yes, one of the challenges of the binding protocol is to make sure
that we provide a simple and sensible model for the kind of system
that Kevin describes (since they are the systems that are most likely
to require and use a multiple-binding model), while still providing
sensible interoperation with file-system based implementations.

In particular, I believe it is likely that a repository such as the
one Kevin describes is likely to refuse to do anything other than
REBIND for MOVE and UNBIND for DELETE, both for performance and
semantic reasons.

Cheers,
Geoff

---

From: Jason Crawford <[email-masked]>
Date: Tue, 11 Mar 2003 14:54:43 -0500

Warning... this note is more interesting to server designers than
spec designers....  :-)

On Tuesday, 03/11/2003 at 08:27 PST, "Lisa Dusseault"
<[email-masked]> wrote:
> > Anyway... the thought I just wanted to point out is that if you have a
> > resource on one file system with multiple bindings to it, and you MOVE
> > that resource via one of those bindings "to another file system"...
> > you might actually want to NOT "copy/delete" it since you
> > still also have
> > several bindings to it from the original file system.  You might
> > *need* to keep a cross file system binding and not necessarily
> > move implementations around during MOVE/BIND and other
> > namespace operations.  And if you're forced to support cross
> > file system bindings, then perhaps you never really need to take the
> > COPY/DELETE approach?
>
> A couple counter-reasons:
>  - Quota within the system (top level directories or lower directories
> limiting space) this may depend on how the quota is counted - the
> system, through its acknowledged limitations, may need to actually move
> the resources (not leave them and rebind them) in order to calculate
> quota.
>
>  - Disk space outside the system - different parts of the namespace are
> frequently literally stored on different disks. Users/Administrators may
> move content from one part of the namespace to another *in order* to
> transfer to a less-used disk.

The system *might* not want to move the implementation to the other file
system because it has reason to keep it on the current file system also.
But as you say, perhaps it might want to move it for reasons of it's own.
Either way, the server has to support cross file system bindings.  And the
choice of which file system it will chose to keep it on becomes arbitrary
from the WebDAV perspective.   I would expect that it would actually end up
keeping it where it was and just get the operation done quickly.

You are probably talking about a situation where it's the last binding, so
it's probably best to move it to the same file system used by its final
parent.  Again, it probably could get away not moving it, but as you say,
users might make assumptions about certain parts of the URI space being
stored in certain file stores.   One complication with this approach is
that in order to try to maintain that relationship, the operation will try
to move the reources to file store that might not have enough quota from
one that already has enough.  That might make the operation abort despite
the fact that the user has enough quota overall.  Another is that if there
is a global quota, the safe copy approach might cause the global quota to
be exceeded (because for some period there will be two copies of one or
more resources) and possibly abort the operation despite there being plenty
of quota overall.  Either way, the half finished MOVE can be a real pain
for clients.   How bad this is depends on how the COPY/DELETE operations
are ordered and when the abort occurs.  (I know I find it a real mess when
I've encountered this situation.  It's tedious to fix if it's even fixable
and the user knows how to fix it.)

Interestingly enough... even if the root of a subtree ends up changing file
systems because it has only one parent binding, many of it's children might
not migrate if there are links from the original file system to them.

So in the presence of multiple bindings, this association between URI and
the file store used for the implementation is weak and sometimes
problematic.  For this reason it doesn't seem well advised to be
compromising WebDAV operations to maintain this mapping  that it can't
really guarantee anyway.  It does seem like there needs to be some other
way for the user to move resources between various storage systems in order
to manage the consumption of his quotas.  --  I repeat though... this is
only in the presence of multiple bindings.  In the absense of multiple
bindings, if we ignore the issue of aborts due to secondary issues, and
ignoring the unfortunate possibility that resource-id's might change as
implementations migrate, the server should be able to maintain the URL to
file store mappings and that can be attractive from a UI perspective.

> Jason, you've clearly got the right instincts for designing a WebDAV
> server -- thinking carefully about how to do the best possible job.
> It's just a question of can the *protocol* enforce a good job on server
> implementors?  Sometimes it can, but sometimes it can't.

Agreed.  In this thread I'm just pointing out possibilities that I, for
one, hadn't thought about or noticed mentioned before.  Thank you for
responding and cleaning up weaknesses in the message that started this
thread.  :-)

> We wouldn't
> want to force atomic DELETEs and have an implementation choose to reject
> a great number of large DELETE requests just because it can't guarantee
> them all atomic. In the extreme, that would be an even worse tradeoff.

Yes, for DELETE, there is a possibility that a client doesn't care if it's
atomic or not.  He just "wants to free resources eating up [his] quota...
and if [he] can't get all of the subtree freed, that's okay for now."   I
will point out weakly though, that if that's the user's thinking, even in
an atomic server, if the error message provided is informative enough,
there is a significant possibility that user will be able to free up the
portion of the subtree that isn't resisting the DELETE.  Obviously a server
can do that more efficiently though than the client can if the server
understands what the user/client wants.

Speaking of the server knowing what the user wants, that does bring up
another point... is the best effort DELETE operation allowed to
stop the first time it encounters a resource that prevents the unbinding of
the request-URI binding (for example to keep it reachable)?   Or must it
continue to find as many resources that it's free to delete as possible?
Or in the case of request URI is "protected" by a lock in the subtree, the
server will almost certainly be aware of the "protection" issue right away,
so is the server required to not remove any bindings at all? --  Anyway, if
we really think clients will *want* best effort DELETE available, we'll
have to specify what the prefered/required behavior is for that.  (Maybe
later if we don't see the demand yet.)

J.

------------------------------------------
Phone: 914-784-7569,   [email-masked]
I do not check [email-masked]

---

From: "Clemm, Geoff" <[email-masked]>
Date: Thu, 13 Mar 2003 14:01:45 -0500

I've made the proposed modifications to the MOVE and DELETE
constraints in the binding protocol in a new revision (01.3) of the
binding protocol, as well as added a constraint to COPY,
and posted it to the webdav binding web site.  Here
are the modified sections.  This will be on the agenda for the
WebDAV meeting next week, so if possible, please review this
before then.

Cheers,
Geoff

------------------------------------------------

2.3	COPY and Bindings

As defined in Section 8.8 of [RFC2518], COPY causes the resource
identified by the Request-URI to be duplicated, and makes the new
resource accessible using the URI spechified in the Destination
header.  Upon successful completion of a COPY, a new binding is
created between the last path segment of the Destination header, and
the destination resource. The new binding is added to its parent
collection, identified by the Destination header minus its final
segment.

The following figure shows an example: Suppose that a COPY is issued
to URI-3 for resource R (which is also mapped to URI-1 and URI-2),
with the Destination header set to URI-X.  After successful completion
of the COPY operation, resource R is duplicated to create resource R',
and a new binding has been created which creates at least the URI
mapping between URI-X and the new resource (although other URI
mappings may also have been created).

URI-1   URI-2    URI-3                           URI-X
   |       |        |                              |
   |       |        |   <---- URI Mappings ---->   |
   |       |        |                              |
+---------------------+                 +------------------------+
|     Resource R      |                 |     Resource R'        |
+---------------------+                 +------------------------+

It might be thought that a COPY request with "Depth: 0" on a
collection would duplicate its bindings, since bindings are part of
the collection's state.  This is not the case, however.  The
definition of Depth in [RFC2518] makes it clear that a "Depth: 0"
request does not apply to a collection's members.  Consequently, a
COPY with "Depth: 0" does not duplicate the bindings contained by the
collection.

If a COPY causes one or more existing resources to be updated, the
bindings to those resources MUST be unaffected by the COPY.  Using the
preceding example, suppose that a COPY is issued to URI-X for resource
R', with the Destination header set to URI-2.  The content and dead
properties of resource R would be updated to be a copy of those of
resource R', but the mappings from URI-1, URI-2, and URI-3 to resource
R remain unaffected.

2.4	DELETE and Bindings

When there are multiple bindings to a resource, a DELETE applied to
that resource MUST NOT remove any bindings to that resource other than
the one identified by the request URI.  For example, suppose the
collection identified by the URI "/a" has a binding named "x" to a
resource R, and another collection identified by "/b" has a binding
named "y" to the same resource R.  Then a DELETE applied to "/a/x"
removes the binding named "x" from "/a" but MUST NOT remove the
binding named "y" from "/b" (i.e. after the DELETE, "/y/b" continues
to identify the resource R).  In particular, although Section 8.6.1 of
[RFC2518] states that during DELETE processing, a server "MUST remove
any URI for the resource identified by the Request-URI from
collections which contain it as a member", a server that supports the
binding protocol MUST NOT follow this requirement.

When DELETE is applied to a collection, it MUST NOT modify the
membership of any other collection that is not itself a member of the
collection being deleted.  For example, if both "/a/.../x" and
"/b/.../y" identify the same collection, C, then applying DELETE to
"/a" MUST NOT delete an internal member from C or from any other
collection that is a member of C, because that would modify the
membership of "/b".

If a collection supports the UNBIND method (see Section 5), a DELETE
of an internal member of a collection MAY be implemented as an UNBIND
request.  In this case, applying DELETE to a Request-URI has the
effect of removing the binding identified by the final segment of the
Request-URI from the collection identified by the Request-URI minus
its final segment. Although [RFC2518] allows a DELETE to be a
non-atomic operation, when the DELETE operation is implemented as an
UNBIND, the operation is atomic.  In particular, a DELETE on a
hierarchy of resources is simply the removal of a binding to the
collection identified by the Request-URI.

2.5	MOVE and Bindings

When MOVE is applied to a resource, the other bindings to that
resource MUST be unaffected, and if the resource being moved is a
collection, the bindings to any members of that collection MUST be
unaffected. Also, if MOVE is used with Overwrite:T to delete an
existing resource, the constraints specified for DELETE apply.

If the destination collection of a MOVE request supports the REBIND
method (see Section 6), a MOVE of a resource into that collection MAY
be implemented as a REBIND request.  Although [RFC2518] allows a MOVE
to be a non-atomic operation, when the MOVE operation is implemented
as a REBIND, the operation is atomic.  In particular, applying a MOVE
to a Request-URI and a Destination URI has the effect of removing a
binding to a resource (at the Request-URI), and creating a new binding
to that resource (at the Destination URI).

As an example, suppose that a MOVE is issued to URI-3 for resource R
below (which is also mapped to URI-1 and URI-2), with the Destination
header set to URI-X.  After successful completion of the MOVE
operation, a new binding has been created which creates the URI
mapping between URI-X and resource R.  The binding corresponding to
the final segment of URI-3 has been removed, which also causes the URI
mapping between URI-3 and R to be removed.  If resource R were a
collection, old URI-3 based mappings to members of R would have been
removed, and new URI-X based mappings to members of R would have been
created.

>> Before Request:

 URI-1   URI-2    URI-3
   |       |        |                                
   |       |        |      <---- URI Mappings
   |       |        |
+---------------------+                   
|     Resource R      |
+---------------------+                   

>> After Request:

 URI-1   URI-2    URI-X
   |       |        |                                
   |       |        |      <---- URI Mappings
   |       |        |
+---------------------+                   
|     Resource R      |
+---------------------+                   

Although [RFC2518] allows a MOVE on a collection to be a non-atomic
operation, a MOVE implemented as an UNBIND MUST be atomic.  Even when
the Request-URI identifies a collection, the MOVE operation involves
only removing one binding to that collection and adding another.
There are no operations on bindings to any of its children, so the
case of MOVE on a collection is the same as the case of MOVE on a
non-collection resource.  Both are atomic.
------------------------------------------------

-----Original Message-----
From: Lisa Dusseault [mailto:[email-masked]]
Sent: Tuesday, March 11, 2003 1:11 PM
To: 'Clemm, Geoff'; 'WebDAV'
Subject: RE: MOVEs across file systems

This is pretty good!  I look forward to seeing it in context of the
draft, when we can submit drafts again of course.

> From: [email-masked] 
> [mailto:[email-masked]] On Behalf Of Clemm, Geoff
> Sent: Tuesday, March 11, 2003 9:10 AM
> To: 'WebDAV'
> Yeah, I didn't think that Julian's suggestion would be popular (:-).
> 
> So I think realistically, we should focus on constraining the
> behavior of MOVE/DELETE in the presence of multiple bindings
> to the same resource, so that they don't violate the basic 
> requirements of multiple bindings.
> 
> The current form of that proposal is:
> 
> ----------------------------
> 
> Instead of saying:
> 
> ------------------
> From: Lisa Dusseault [mailto:[email-masked]]
> Sent: Tuesday, March 11, 2003 11:40 AM
> This is equivalent to the previous behavior as long as 
> clients continue
> to issue the same MOVE and DELETE requests they have in the 
> past (which
> they will for quite a while).  Therefore, I don't see how this is a
> change, or how this could possibly be acceptable if the previous
> behavior was unacceptable.
> 
> Servers must be able to support the binding specification, and to
> support ordinary WebDAV clients, and to do what the server 
> implementors
> consider to be the most appropriate and best job they can of 
> fulfilling
> the request, and to report the results.  This proposed statement does
> not meet that requirement because it forces all servers to do atomic
> MOVE/DELETE in handling requests from ordinary WebDAV clients.
> 
> Lisa
> 
> > From: [email-masked] 
> > [mailto:[email-masked]] On Behalf Of Clemm, Geoff
> > Sent: Monday, March 10, 2003 12:18 PM
> > To: 'WebDAV'
> > From: Julian Reschke [mailto:[email-masked]]
> > 
> > > From: Clemm, Geoff
> > > So I'm happy to limit the constraints on MOVE and DELETE 
> to exactly
> > > what is needed to preserve the semantics of multiple bindings, but
> > > leaving them unconstrained makes the binding protocol pointless in
> > > practice.
> > 
> > On the other hand, a system that allows a "weak" MOVE if and 
> > only if there
> > aren't any multiple bindings seems very weird to me. So maybe 
> > we should
> > consider make MOVE "strong" by default, and only allow the 
> > old COPY/DELETE
> > semantics upon specific request?
> > 
> > 
> 
>