Subject: The getcontentlength property

From: Werner Donné <[email-masked]>
Date: Fri, 22 Jun 2007 14:55:28 +0200

Hi,

RFC 2518 says that this property must be returned if the
Content-Length header is returned in a GET response for
the resource at hand. I assume that it is allowed to return
this property if the Content-Length header is never returned,
for example, because the chunked transfer encoding is used.

Can a client rely on a previously retrieved getcontentlength
to read only part of the chunked response stream of the
following GET response?

The particular case I have is where I detect the client is on
a windows machine and is retrieving a text/plain resource. In
that case carriage returns may be added on the fly. This
increases the physical length of the resource.

Regards,

Werner.
-- 
Werner Donné  --  Re
Engelbeekstraat 8
B-3300 Tienen
tel: (+32) 486 425803	e-mail: [email-masked]

---

From: Julian Reschke <[email-masked]>
Date: Fri, 22 Jun 2007 15:03:48 +0200

> for example, because the chunked transfer encoding is used.

Yes.

> to read only part of the chunked response stream of the

If it has reason to believe that the entity didn't change (ETag...), 
then yes.

> increases the physical length of the resource.

By doing this you're serving two different variants, depending on some 
part of the request ("User-Agent" header?)? So you'll have to specify 
"Vary" header in the response, and assign different entity tags. In that 
case a properly written client shouldn't have any problems because it 
can detect that the entity served by the second request is really different.

Best regards, Julian

---

From: Werner Donné <[email-masked]>
Date: Fri, 22 Jun 2007 16:49:47 +0200

> can detect that the entity served by the second request is really 

Yes, the "User-Agent" header was used.

The problem is when the resource is going to be fetched for the first
time by the client. It does a PROPFIND for retrieving the contents of
the parent collection and then a GET. The client I'm trying it with
doesn't ask for the "getetag" property in the PROPFIND, so whatever
ETag returns, it can't draw any conclusions from it. It just uses the
"getcontenttype" property to decide how many bytes to read, even if
the response stream is chunked.

Regards,

Werner.

---

From: Julian Reschke <[email-masked]>
Date: Fri, 22 Jun 2007 16:49:47 +0200

> the response stream is chunked.

Well, that's a bug (in the client).

Best regards, Julian

---

From: "Mr. Demeanour" <[email-masked]>
Date: Fri, 22 Jun 2007 16:05:26 +0100

Werner Donné wrote:
>>> Content-Length header is returned in a GET response for the
>>> resource at hand. I assume that it is allowed to return this
>>> property if the Content-Length header is never returned, for
>>> example, because the chunked transfer encoding is used.
>> 
>> Yes.
>> 
>>> Can a client rely on a previously retrieved getcontentlength to
>>> read only part of the chunked response stream of the following
>>> GET response?
>> 
>> If it has reason to believe that the entity didn't change
>> (ETag...), then yes.
>> 
>>> The particular case I have is where I detect the client is on a
>>> windows machine and is retrieving a text/plain resource. In that
>>> case carriage returns may be added on the fly. This increases the
>>> physical length of the resource.
>> 
>> By doing this you're serving two different variants, depending on
>> some part of the request ("User-Agent" header?)? So you'll have to
>> specify "Vary" header in the response, and assign different entity
>> tags. In that case a properly written client shouldn't have any
>> problems because it can detect that the entity served by the second
>> request is really different.
> 
>  time by the client. It does a PROPFIND for retrieving the contents
> of the parent collection and then a GET. The client I'm trying it
> with doesn't ask for the "getetag" property in the PROPFIND, so
> whatever ETag returns, it can't draw any conclusions from it. It just
> uses the "getcontenttype" property to decide how many bytes to read,
> even if the response stream is chunked.

I take it you meant "uses the getcontentlength property to decide".

If this client is going to ignore the HTTP 1.1 protocol, and instead
decides to read from the wire the number of bytes given in the response
to an earlier request, then I wouldn't expect it to work at all with a
chunked stream; because the length of the stream depends on the
transfer-encoding; and in particular, chunked encoding changes to the
number of bytes that comprise the stream.

I must say, I'm rather puzzled by this question; the whole point of the
chunked encoding is that it is suited to transferring content for which
it is inconvenient to compute a length in advance. If you can know the
length in advance, can't you just get your server to transfer unencoded
data, and use a Content-Length: header?

I presume I've missed the point.

-- 
Jack.

---

From: Werner Donné <[email-masked]>
Date: Sat, 23 Jun 2007 11:10:48 +0200

> I take it you meant "uses the getcontentlength property to decide".

Indeed. Sorry for the confusion.

> I presume I've missed the point.

This client does seem to read the chunked stream correctly, but after
reading it, it only retains as much bytes as indicated by the
"getcontentlength" property, which was fetched earlier. So if I have a
text/plain resource, for example, with ten lines of five characters each
and if I add a CR to each of the lines, the last ten characters will not
be returned. The "getcontentlength" returned 50, but because of the CRs
the response is 60 long, after decoding the chunked transfer encoding
that is. The result I see is that the returned file is cut off.

Werner.

---

From: "Mr. Demeanour" <[email-masked]>
Date: Sat, 23 Jun 2007 10:29:40 +0100

Werner Donné wrote:
> "getcontentlength" property, which was fetched earlier.

That's clearly a defect in the client. The server can't be expected to
compute a getcontentlength that may be dependent on whatever
Transfer-Encoding it decides on later (since the Transfer-Encoding may
be dependent on some Accept: header it hasn't seen yet).

But this is partly missing the point, I think. Your server is sending a
getcontentlength value for an entity which is dependent on the
User-Agent; but it seems to be generating the same getcontentlength
value, regardless of which entity is being requested (i.e. regardless of
the User-Agent setting). If you are going to vary the entity's value
according to the User-Agent, then surely you should vary the value of
getcontentlength according to the same criteria.

-- 
Jack.

---

From: Julian Reschke <[email-masked]>
Date: Sat, 23 Jun 2007 11:35:34 +0200

> be dependent on some Accept: header it hasn't seen yet).

Um, the value of content-length header as defined in RFC2616 does not 
depend on the transfer encoding.

> getcontentlength according to the same criteria.

I'd like to agree with this, but unfortunately RFC2518 has this language 
in (<http://greenbytes.de/tech/webdav/rfc2518.html#rfc.section.13.4>):

"Purpose:
     Contains the Content-Length header returned by a GET without accept 
headers."

Now U-A is not an accept header, but it *seems* like the original intent 
was to forbid variant selection here. Problem is, nobody seems to recall 
why this was added, and what exactly it means. Unfortunately, we have 
the same confusing language in 
<http://greenbytes.de/tech/webdav/draft-ietf-webdav-rfc2518bis-18.html#rfc.section.15.4>, 
soon to be published as RFC4918.

Best regards, Julian

---

From: "Mr. Demeanour" <[email-masked]>
Date: Sat, 23 Jun 2007 12:39:08 +0100

Julian Reschke wrote:
>>> This client does seem to read the chunked stream correctly, but
>>> after reading it, it only retains as much bytes as indicated by
>>> the "getcontentlength" property, which was fetched earlier.
>> 
>> That's clearly a defect in the client. The server can't be expected
>> to compute a getcontentlength that may be dependent on whatever 
>> Transfer-Encoding it decides on later (since the Transfer-Encoding
>> may be dependent on some Accept: header it hasn't seen yet).
> 

Agreed. But it's meaning does; in the presence of Transfer-Encoding:
chunked, Content-Length: should be ignored. I would suppose that means
that getcontentlength should also be ignored in those circumstances.
> 
>> But this is partly missing the point, I think. Your server is
>> sending a getcontentlength value for an entity which is dependent
>> on the User-Agent; but it seems to be generating the same
>> getcontentlength value, regardless of which entity is being
>> requested (i.e. regardless of the User-Agent setting). If you are
>> going to vary the entity's value according to the User-Agent, then
>> surely you should vary the value of getcontentlength according to
>> the same criteria.
> 
> I'd like to agree with this, but unfortunately RFC2518 has this
> language in
> (<http://greenbytes.de/tech/webdav/rfc2518.html#rfc.section.13.4>):
> 
> "Purpose: Contains the Content-Length header returned by a GET
> without accept headers."

Hah - OK. So Werner's client shouldn't be using getcontentlength to
control the transfer process /at all/.

Suppose that instead of a simple UA-dependent transformation of CRs into
CRLFs, the various entities that might be returned were translations
into various languages of some text? Without accept headers, the
getcontentlength value could represent any one of the renditions, chosen
arbitrarily. But if the browser submits an Accept-Language: header, and
the rendition chosen for the computation doesn't happen to match that
header, then there would be no correspondence whatsoever between the two
lengths.
> 
> Now U-A is not an accept header, but it *seems* like the original
> intent was to forbid variant selection here. Problem is, nobody seems
> to recall why this was added, and what exactly it means.
> Unfortunately, we have the same confusing language in 

So I'm inclined to interpret it as meaning that getcontentlength gives
the size of some entity of the resource, but which entity is undefined.
In particular, its value is unsuitable for controlling a transfer.

-- 
Jack.

---

From: Julian Reschke <[email-masked]>
Date: Sat, 23 Jun 2007 15:28:27 +0200

> that getcontentlength should also be ignored in those circumstances.

Nope. Either DAV:getcontentlength is present (then it indicates the size 
after Transfer-Decoding) or it's not present. But if the server returns 
it, a client can use it.

That being said, the information from the actual GET request is always 
authoritative, be it a Content-Length response header, or the actual 
content length after reading the full body using Transfer-Encoding: chunked.

> ...

Yep. The getcontentlength property is for things like directory 
listings. That's it.

> header, then there would be no correspondence whatsoever between the two

Right.

> In particular, its value is unsuitable for controlling a transfer.

+1.