Subject: Re: UPDATE method and forking

From: Manfred Baedke <[email-masked]>
Date: Thu, 16 Mar 2006 20:16:55 +0100

Hi Werner,

comments inline:

>> Werner Donné <[email-masked]> wrote on 03/16/2006 08:01:57 AM:
>> 
>>> Geoffrey M Clemm wrote:
>>> >
>>> > (1) Why does this scenario have to be in one transaction?  In
>> particular,
>>> > why isn't it sufficient just to wrap the sequence of operations in
>>> > a LOCK/UNLOCK?  
>>>
>>> Because anything in between can go wrong, leaving the system in an
>>> inconsistent state. It can't be the responsibility of the client to
>>> keep the state consistent. That is a co-operative and hence unreliable
>>> design.
>> 
>> The only thing that can go wrong is that the client neglects to perform
>> an UNLOCK, which is why there are timeout and administrative override
>> mechanisms
>> in the locking protocol.  But I agree that many clients would rather not
>> deal with locks, which is why we introduced the workspace feature.  So lets
>> focus on whether or not the workspace feature provides what you need.
> 
> Between each step in the sequence the dialogue can be interrupted. Not only
> would the lock not be released, but the checked-in version of the VCR could
> also be on another version than it was originally. This is an unexpected
> global state change for all clients, which is not acceptable in a concurrent
> system.
> 

So the checked-in version of the VCR changed. Every client (at least 
those who do not hold a lock) must be aware of this possibility at any 
time. Why should this be unexpected?

> Regarding the lock release the timeout mechanism is not good enough if
> locking is used like that, because VCR is blocked too long. This impacts
> concurrency severely. When a transaction fails, for example, everything
> is rolled back and release immediately.
> 

Yes, long lasting exclusive locks are evil. Again, why not use 
workspaces or working resources?

> Again, I'm not discussing what I need. I'm not a user.
> 
>> 
>>> > (2) I agree that the locking mechanism is not designed to implement
>>> > transactional isolation.  That is what workspaces and working resources
>>> > are designed for.  If (for unexplained reasons :-) you don't want to
>>> > use the versioning features that were designed for transactional
>>> > isolation, it should come as no surprise that you have difficulty
>>> > achieving transactional isolation.  Some clients don't want/need
>>> > transactional isolation (they just want history to be kept, for
>> example),
>>> > which is why workspaces and working resources are optional features
>>> > (we define these two different ways of achieving transactional isolation
>>> > because some repositories can only support one, while other repositories
>>> > can only support the other, and we could find no way of unifying these
>>> > two different ways under a single feature).
>>>
>>> This is obviously not at all about what I want or don't want to use.
>>> I'm reasoning about the WebDAV model. I haven't said I was against
>> workspaces.
>> 
>> Well, to be honest, it is not at all clear to me what you do or do not
>> want to do (:-).  The purpose of the WebDAV model is provide a uniform
>> protocol for accessing the various authoring features provided by
>> a wide variety of repositories.  So the interesting questions are
>> "how do I as a client use the protocol to achieve this interesting
>> use case for a user" and "how do I as a repository use the protocol
>> to expose this particular feature of my repository to a client".
>> Without one of those questions in hand, one cannot have a productive
>> discussion about the WebDAV model.
> 
> I'm assessing the model, because I want to implement it. It is important
> that a model is consistent and stable, otherwise its implementation will
> be expensive. In my opinion exposing forking is not consistent. It is not
> because there is another feature that does the trick, that a particular
> feature shouldn't be scrutinised.
> 

Please explain in detail where you see inconsistencies. I can only see a 
non-linear version history so far, which is perfectly consistent.

> Inconsistent parts in a model have a life of their own and can have an
> impact on other areas of the model during evolution, just for the sake
> of compatibility.
> 
>> 
>>> > As for the why support forking, it is an (undesireable, but inevitable)
>>> > side-effect of parallel development.  So we don't define it as an
>>> > independent feature, but we "deal with it" for those features that
>>> > support parallel development (i.e., workspaces and working resources).
>>> > In particular, it is what tells you that parallel development has
>>> > occurred on a particular resource, and that therefore a merge is
>>> > required (the need for a merge when forking occurs is why it is
>>> > "undesireable").
>>> >
>>> > So creating forks in a non-parallel development context makes no
>> sense ...
>>> > you are just making trouble for yourself.  Which is why forking only
>>> > occurs as a side effect of parallel development, and not as a
>> feature that
>>> > you explicitly invoke.
>>>
>>> Which is what I reckoned and it is an encapsulation flaw. Clearly, forking
>>> on its own is useless, but it is exposed through the WebDAV interface with
>>> elements such as "checkin-fork", "checkout-fork", "fork-ok". There is no
>>> reason for this, nor for the UPDATE method to be allowed to change the
>>> checked-in version, which is the source of the problem.
>> 
>> The way WebDAV (and HTTP) achieves interoperability is by having a client
>> use a single uniform protocol against a wide variety of servers.  So the
>> client always uses the UPDATE method to "restore" a version from history,
>> whether it is working with a server that supports working resources,
>> supports
>> workspaces, supports both, or supports neither.  Changing the checked-in
>> version is important when the server supports either workspaces or
>> working resources, so you have clients uniformly use the UPDATE method.
>> This means that a client is aware of checkin-fork, checkout-fork, and
>> fork-ok behavior, so that in behaves properly when it is applied to a
>> server that supports either workspaces or working resources.
> 
> It strikes me that workspaces don't expose forking, while working resources
> do. Workspaces prove it is perfectly possible to offer branching and merging
> without exposing an implementation artifact such as forking. Note that they
> also make it possible to work transactionally within the repository without
> exposing transaction demarcation.
> 
> Workspaces are well designed, while working resources are not. Their difference
> should only about configurations, not in the way they offer branching. That
> should be exactly the same. The two things are orthogonal.
> 

The main difference between workspaces and working resources is, IMHO, 
whether the client controls the namespace or not. In both cases, you get 
multiple VCRs sharing a common version history. I really do not see a 
relevant difference in fork handling. What am I missing?

Regards,
Manfred

>> 
>>> Creating a new
>>> VCR in a workspace from a given version obviously requires forking, but
>>> that is merely an internal implementation matter.
>> 
>> No, it's part of the explicit user model, because when you invoke the MERGE
>> method, the result of the MERGE method is affected by whether or not
>> there has
>> been any branching, so the client needs to understand when a branch will
>> occur,
>> and when it won't.  
> 
> Merging and branching go together and you can do it with VCRs only. So there
> is no reason to offer another user model for branching than that of workspaces.
> Working resources should be made compatible with it.
> 
>> 
>> Cheers,
>> Geoff
>>  
>> 
>> 
> 
> Regards,
> 
> Werner.
> -- 
> Werner Donné  --  Re
> Engelbeekstraat 8
> B-3300 Tienen
> tel: (+32) 486 425803	e-mail: [email-masked]
> 
> Received on Thursday, 16 March 2006 17:34:50 GMT
> 
>     * This message: [ Message body ]
>     * Next message: Manfred Baedke: "Re: rfc2518bis-14: locking terminology"
>     * Previous message: John Barone: "RE: rfc2518bis-14: locking terminology"
>     * In reply to: Geoffrey M Clemm: "Re: UPDATE method and forking"
> 
>     * Mail actions: [ respond to this message ] [ mail a new topic ]
>     * Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ]
>     * Help: [ How to use the archives ] [ Search in the archives ] 
> 
> This archive was generated by hypermail 2.2.0+W3C-0.50 : Thursday, 16 March 2006 17:35:02 GMT

---

From: Werner Donné <[email-masked]>
Date: Fri, 17 Mar 2006 09:41:06 +0100

Hi Manfred,

> time. Why should this be unexpected?

Several clients can be executing such scenarios at the same time with the
same VCR. In order to avoid sudden changes of the checked-in version they
all have to use locking. Locking, however, is not mandatory, so it is up
to the clients to make sure there are no races. This is a co-operative
process. It is like saying that multiple clients can update tables in a
database, but they have to manage table or record locking themselves. It
was like that decades ago.

The forking and update mechanisms require isolation and locking is not
the good instrument for that. Workspaces achieve concurrency in a natural
way without imposing co-operative scenarios with locking. This is both
safer and simpler.

> workspaces or working resources?

They are not evil if the repository supports branching. It is common to
reserve a branch, because branches are a tool to let people work together
on the same topic. Long-lasting locks should not be used on the integration
branch in which all work will be merged.

> non-linear version history so far, which is perfectly consistent.

You have two alternatives for working in parallel, where one suffices. One
method, working resources, exposes physical aspects such as forking and
puts the resposibility for integrity at the client. The other is abstract.
It doesn't expose implementation details, because each "branch" has its
own VCR and the checked-in version of each VCR can be left to be the most
recent version. The integrity responsibility is within the server.

The only functional difference there is supposed to be between workspaces
and working resources is that the latter work in their own configuration,
a term that is nowhere defined by the way prior to section 9 of RFC 3253.

That makes it still possible to use workspaces in a client that would have
to resort to working resources. Workspaces are now positioned as a pure
server functionality. They are, however, an abstract mechanism that can
be perfectly implemented differently in a real server and in a stand-alone
client. For the latter the implementation may be lighter because there is
perhaps no cuncurrency, making the preservation of integrity easier. It
is as if such a client has its own built-in server.

> relevant difference in fork handling. What am I missing?

See above.

Regards,

Werner.

---

From: Manfred Baedke <[email-masked]>
Date: Fri, 17 Mar 2006 15:57:12 +0100

Hi Werner,

> recent version. The integrity responsibility is within the server.
First of all, checking out a working resource _does_ create a new VCR, 
much like a VERSION-CONTROL request to an unmapped URL which is a member 
URL of a workspace, the main difference being that in the latter case 
the client can choose the URL. Both mechanisms can be used to create 
multiple VCRs sharing a common version history and both can (but not 
must) be used in the way you describe (letting the checked-in version of 
each VCR be the "end of a branch").
I am still trying to understand what you mean when you speak of 
'integrity'. Am I right that you propose, speaking loosely, that the 
'original' VCR should belong to something like a 'main branch', its 
checked-in version always being the end of that branch, and that a new 
branch should be created by creating a new VCR (based on an 'old 
version') within a workspace?
I you do so, feel free not to support the UPDATE feature in your 
implementation.

Regards,
Manfred

---

From: Geoffrey M Clemm <[email-masked]>
Date: Fri, 17 Mar 2006 09:05:27 -0500

A few initial comments, and then additional comments in-line:

The reason for supporting locking is not because locking is a good way
to do parallel development on a versioning server (it isn't), but rather
because there are a lot of WebDAV locking clients out there, and we
want to ensure that they work as well as possible against a versioning
server.  So when we argue that "locking works", we aren't saying that
locking is a good way to solve this problem, but rather that if you have
a locking client, it will interoperate well with a versioning server
that supports the RFC-3253 versioning protocol.

The reason for supporting working resources is not because it provides
a good model for clients to work with (it doesn't), but rather that
it provides a different distribution of work between the client and
server.  In particular, with a workspace server, all of the work is
being done on the server, which means clients are much simpler to write.
With a working-resource server, a large part of the work has to be done
on a client (effectively, the client implements a client-side 
workspace, using local persistent state on the client, and the
low-level working-resource mechanisms provided by the server),
but because the server (the shared resource) does
significantly less work, it is actually much simpler to write a scalable
system using the working-resource mechanism than it is to write one
using the workspace mechanism.  This was the one architectural choice
which we found we couldn't hide beneath a common protocol (and I can
vouch for the fact that we tried very hard for a very long time to
do so). 

In that context, some additional comments below:

Werner wrote on 03/17/2006 03:41:06 AM:
> process. 

A single locking client on its own can ensure there are
no races by taking out the lock.  An attempt to modify this resource by
any method (PUT, UPDATE, MERGE) by another client will fail while the 
client
has the lock.  The only time co-operation is required is amoung a group of
non-locking clients ... but in this case they use etags, which will also
work against a versioning server.  To emphasize the point made above, this 
is
not an argument that locking the VCR is a good way to handle this problem 
...
it is an argument that existing locking clients will interoperate in a 
reasonable
way with a server that supports RFC-3253.

> It is like saying that multiple clients can update tables in a

Yes, but if you have thousands of deployed clients (written years ago
when there was no workspace protocol) that do this table and
record locking management, it is a very good thing if they will just work
unmodified against the new versioning servers.

> way without imposing co-operative scenarios with locking. This is both

Totally agree.  Newer clients that take advantage of the workspace 
protocol
will be much easier to write than it was to write the older clients that 
only
could use the locking protocol.

> branch in which all work will be merged.

A branch is modeled in the WebDAV protocol as an "activity".  Note that
we use the term "reserve" for what you do to a branch, to avoid confusion 
with the
"lock" protocol.  The lock protocol was not used for reserving branches,
because we didn't want the client to have to maintain lock tokens for the
branch they are working on.

> recent version. The integrity responsibility is within the server.

Yes, I agree that the workspace protocol allows for much simpler clients
to be written.

> a term that is nowhere defined by the way prior to section 9 of RFC 
3253.

Actually, it is section 10.2 ... "Advanced Versioning Terms".  Before 
then,
it was just used in an informal descriptive way, and was not part of the 
protocol.

Since my repository supports workspaces, I did try for quite a while to
convince the standards group that we only needed to support workspaces, 
but
the reality is that there are repositories that do not (and cannot) 
support
workspaces, but do support working resources, and unless we defined
working resources support in the protocol, clients would not be able to
interoperate in a useful way with those repositories.

> is as if such a client has its own built-in server.

But from a protocol perspective, you cannot hide this difference.  This is
in fact one of the major reasons why we started JSR-147, which is a 
client-level
API rather than a client-server protocol.  In JSR-147, there is only the
workspace model, because in a client API, you can hide the implementation 
choice
of "fat client" (working-resource model) or "skinny client" (workspace 
model)
under the client library layer that supports the JSR-147 API.

Cheers,
Geoff

---

From: Werner Donné <[email-masked]>
Date: Fri, 17 Mar 2006 16:25:00 +0100

Hi Manfred,

> each VCR be the "end of a branch").

I have reread section 9 of RFC 3253 and I can't find prove of a VCR being
created for a working resource, but I'm misinterpreting something perhaps.

> version') within a workspace?

Indeed, that is correct. With integrity I mean making sure that the
system is always in a consistent and predictable state. This is
normally achieved with transactions. Workspaces, being more abstract,
provide the opportunity for an "advanced" implementation to offer
such a guarantee. At the same time a less advanced implementation that
is deployed in a stand-alone environment, i.e. without concurrency
and the risk for races, can also comply to the workspaces specification.

Since the UPDATE method can change the checked-in version of a VCR
globally, the result may be unpredictable. (It will probably be
consistent, however, because the UPDATE method itself is normally
atomic.) This is because client may "forget" to use locking. If two
such clients are working concurrently on the same VCR, the checked-in
version may change in the middle of their scenario. They may end up
checking out from some other version without realising it. You can blame
those clients rightfully for it, but when there is a central system
it must make sure itself that this can't happen. The central system
can't rely on all clients to behave correctly.

Contrarily to what the spec suggests, I think it is simpler for both
client AND server if only workspaces are supported for working in
parallel.

> I you do so, feel free not to support the UPDATE feature in your

That is indeed possible, but I still would like to know where this
UPDATE and forking comes from and what purpose it serves. Turning it
the other way around, which problems would arise if it was left out?

Regards,

Werner.

---

From: Werner Donné <[email-masked]>
Date: Fri, 17 Mar 2006 16:48:23 +0100

Hi Geoff,

I understand now that there is in fact legacy to support, which is
fair enough. So Manfred doesn't have to answer the question in my
previous post anymore. The conclusion is important, because it
comes down to the choice whether an new server implementation wants
to support those older clients or not.

The JSR-147 initiative seems interesting. How does it relate to
JSR-170? Sections 4.10 and 4.11 of the latter may overlap.

Regards,

Werner.

Geoffrey M Clemm wrote:
> work against a versioning server.  To emphasize the point made above,
> this is
> not an argument that locking the VCR is a good way to handle this
> problem ...
> Totally agree.  Newer clients that take advantage of the workspace protocol
> we use the term "reserve" for what you do to a branch, to avoid
> confusion with the
> Actually, it is section 10.2 ... "Advanced Versioning Terms".  Before then,
> convince the standards group that we only needed to support workspaces, but
> the reality is that there are repositories that do not (and cannot) support
> workspace model, because in a client API, you can hide the
> implementation choice

---

From: Geoffrey M Clemm <[email-masked]>
Date: Fri, 17 Mar 2006 11:06:58 -0500

> created for a working resource, but I'm misinterpreting something 
perhaps.

Manfred's description isn't quite right ... you don't check out a working
resource ... you create a working resource by checking out a version.
You then update the working resource (with PUT) and check it in, which
deletes the working resource and creates a new version.  So Werner is 
correct
that there are two distinct ways of working.
The workspace way:
- you apply CHECKOUT to a VCR, which makes the VCR writeable
- you update the VCR with PUT
- you apply CHECKIN to the VCR, which creates a new version and makes the 
VCR
  unwriteable.
The working resource way:
- you apply CHECKOUT to a version, which creates a new working resource
- you update the working resource with PUT
- you apply CHECKIN to the working resource, which creates a new version 
and
  deletes the working resource.

> and the risk for races, can also comply to the workspaces specification.

Or with the working resource model, the state of the transaction is
basically the set of working resources currently held by the client.
Those working resources are effectively private to that client, and
therefore no other client can interfere with the transaction.  Much
lower level than workspaces, more work/complexity for the client, but much 
lower
overhead on the server.

> can't rely on all clients to behave correctly.

Yes, the workspace model is the simple/clean way to achieve this.
The locking model is the "works with existing clients" approach.

> client AND server if only workspaces are supported for working in

The working resource model is worse for the client than the workspace
model, but is significantly better than locking and a scalable
working resource server is significantly easier to implement than
a scalable workspace server.

> the other way around, which problems would arise if it was left out?

If your server does not support workspaces, then the only real benefit
of UPDATE is that it allows you to avoid creating a new version when
a client wants to restore an old version.  In particular,
if you don't support UPDATE, then the only way that your client can
tell the server that it wants to change the state back to that of some
previous version is to GET the content of the desired version,
CHECKOUT the VCR, PUT the content it got, and then CHECKIN, which creates
a new version (which is a copy of an existing version).  With UPDATE,
you can just send an UPDATE request to the server, with the URL of the
desired version (which both avoids creating a new version, and avoids
streaming the content back to the client via GET and then back to the
server via PUT).

Cheers,
Geoff

---

From: Geoffrey M Clemm <[email-masked]>
Date: Fri, 17 Mar 2006 11:16:37 -0500

Werner wrote on 03/17/2006 10:48:23 AM:

> previous post anymore.

Yes, the design of WebDAV versioning was especially challenging since
we were concerned both with legacy WebDAV clients, and legacy versioning
repositories, in addition to wanting to define a "good model" for
the future.

> The conclusion is important, because it

That is exactly right.

> JSR-170? Sections 4.10 and 4.11 of the latter may overlap.

Being semantically compatible with JSR-147 was an explicit goal of 
JSR-170.
We are currently considering adding the more advanced semantics of JSR-147
(activities, baselines) into JSR-283, the next version of JSR-170.

Cheers,
Geoff

---

From: Manfred Baedke <[email-masked]>
Date: Fri, 17 Mar 2006 17:34:14 +0100

Hi,

> resource ... you create a working resource by checking out a version.

This is of course true.

> deletes the working resource and creates a new version.  So Werner is 

My point was that the working resource itself is a VCR, whose deletion 
on a subsequent CHECKIN can be avoided using the DAV:keep-checked-out 
flag, in which case the working resource behaves much like any other VCR.

Regards,
Manfred

---

From: Geoffrey M Clemm <[email-masked]>
Date: Fri, 17 Mar 2006 14:32:07 -0500

Manfred Baedke <[email-masked]> wrote on 03/17/2006 11:34:14 

> flag, in which case the working resource behaves much like any other 
VCR.

Well, to be precise, a working resource is a "checked-out resource".
A checked-out VCR is also a checked-out resource,
but although both a working resource and a checked-out VCR share 
properties
and methods (i.e., all the properties and methods of a checked-out 
resource),
a working resource is actually not a VCR (in particular, a VCR has 
properties
that a working resource does not have).

Cheers,
Geoff

---

From: Manfred Baedke <[email-masked]>
Date: Fri, 17 Mar 2006 21:44:37 +0100

Geoff,

again you are right.
The reason for my confusion is that in our implementation (i.e. SAP 
Netweaver) a working resource _is_ a version controlled resource. I will 
have to meditate about this being a bug or just an implementation detail.

Regards,
Manfred

Geoffrey M Clemm wrote:
> 
> Manfred Baedke <[email-masked]> wrote on 03/17/2006 
> 11:34:14 AM:
> 
> but although both a working resource and a checked-out VCR share properties

---

From: Geoffrey M Clemm <[email-masked]>
Date: Fri, 17 Mar 2006 16:43:05 -0500

I would call this an implementation detail, and definitely not call
it a bug.  All it means is that working resources have more properties
in your system than the standard requires.

Also, since that is exactly what we would have done if we had
supported working resources on our workspace-based server (basically,
just allocating the working resources in some system-maintained
workspace), I consider it a very sensible implementation choice (:-).

Cheers,
Geoff

Manfred Baedke <[email-masked]> wrote on 03/17/2006 03:44:37 

> >  > flag, in which case the working resource behaves much like any 
other 
properties