]>

aaa@bzfx.net

Building Blocks for HTTP APIs Internet-Draft HTTP This document specifies a media type for PATCH payloads that overwrites a specific byte range, facilitating random access writes and segmented uploads of resources.

Introduction Filesystem interfaces typically provide some way to write at a specific position in a file. While HTTP supports reading byte range offsets using the Range header (), this technique cannot generally be used in PUT, because the server may ignore the Content-Range header while executing the write, causing data corruption. However, by using a method and media type that the server must understand, writes to byte ranges with Content-Range semantics becomes possible even when server support is undetermined. This media type is intended for use in a wide variety of applications where overwriting specific parts of the file is desired. This includes idempotently writing data to a stream, appending data to a file, overwriting specific byte ranges, or writing to multiple regions in a single operation (for example, appending audio to a recording in progress while updating metadata at the beginning of the file).

Notational Conventions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here. This document uses ABNF as defined in and imports grammar rules from and . For brevity, example HTTP requests or responses may add newlines or whitespace, or omit some headers necessary for message transfer. The term "byte" is used in the sense to mean "octet." Ranges are zero-indexed and inclusive. For example, "bytes 0-0" means the first byte of the document, and "bytes 1-2" is a range with two bytes, starting one byte into the document. Ranges of zero bytes are described by an address offset rather than a range. For example, "at byte 5" would separate the byte ranges 0-4 and 5-9.

Modifying a content range with PATCH Sending a Content-Range field in a PUT request (A "partial PUT", ) requires server support to be processed correctly. Without such support, the server's normal behavior would be to ignore the header, replacing the entire resource with just the part that changed, potentially causing corruption. To mitigate this, Content-Range may be used in conjunction with the PATCH method and a media type whose semantics are to write at the specified byte offset. This document re-uses the "multipart/byteranges" media type, and defines the "message/byterange" and "application/byteranges" media types, to opportunistically carry this field. A byte range patch lists one or more parts. Each part specifies two essential components:

1. Part fields: a list of HTTP fields that specify metadata, including the range being written to, the length of the body, and information about the target document that cannot be listed in the PATCH headers, e.g. Content-Type (where it would describe the patch itself, rather than the document being updated).
2. A part body: the actual data to write to the specified location.

Each part MUST indicate a single contiguous range to be written to. Servers MUST reject byte range patches that don't contain a known range with a 422 or 400 error. (This would mean the client may be using a yet-undefined mechanism to specify the target range.) The simplest form to represent a byte range patch is the "message/byterange" media type, which is similar to an HTTP message: This patch represents an instruction to write the four bytes "cdef" at an offset of 2 bytes. A document listing the digits 0-9 on a line, would look like this after applying the patch: Although this example is a text document with a line terminator, part bodies are only interperted as binary data, and can potentially overwrite individual bytes of multi-byte characters.

The Content-Range field The Content-Range field (as seen inside a patch document) is used to specify where in the target document the part body will be written. The client MAY indicate the anticipated final size of the document by providing the complete-length form, for example the "12" in bytes 0-11/12. The complete-length does not affect the write by itself, however the server MAY use it for other purposes, especially for preallocating disk space, or deciding when an upload in multiple parts has finished. If the client does not know or care about the final length of the document, it MAY use * in place of complete-length. For example, bytes 0-11/*. Most random access writes will take this form. The unsatisfied-range form (e.g. bytes */1000) sets the size of the document, but without writing any data. It may be used to truncate a document (to invert an append operation), to allocate space for a document without specifying any of its contents, or to signal that an upload has ended when the previous part or request did not specify a complete-length. The part body MUST be empty. If the "last-pos" is is unknown because the upload is indeterminate length (the Content-Length of the request is not known from the start, or the upload might continue indefinitely), then the Content-Offset field must be used instead.

The Content-Length field A "Content-Length" part field, if provided, describes the length of the part body. (To describe the size of the entire target resource, see the Content-Range field.) If provided, it MUST exactly match the length of the range specified in the Content-Range field, and servers MUST error when the Content-Length mismatches the length of the range.

The Content-Offset field The Content-Range field specifies an offset to write the content at, when the end of the write is not known. It is used instead of Content-Range when the Content-Length is not known. Its syntax is specified as a Structured Field : The value indicates how much of the target document is to be skipped over, typically the number of bytes. It MUST be an sf-integer. It accepts two parameters: The "unit" parameter indicates the unit associated with the value. A missing "unit" parameter is equivelant to providing unit=bytes. The "complete-length" parameter is equivelant to the complete-length value in Content-Range. When provided, it MUST be an sf-integer specifying the intended final length of the document. A missing "complete-length" is equivelant to providing * in Content-Range.

The Content-Type field A "Content-Type" part field MUST have the same effect as if provided in a PUT request uploading the entire resource (patch applied). Its use is typically limited to resource creation.

Other fields Other part fields in the patch document SHOULD have the same meaning as if it is a message-level header in a PUT request uploading the entire resource (patch applied). Use of such fields SHOULD be limited to cases where the meaning in the HTTP request headers would be different, where they would describe the entire patch, rather than the part body. For example, the "Content-Type" or "Content-Location" fields.

Applying a patch Servers SHOULD NOT accept requests that write beyond, and not adjacent to, the end of the resource. This would create a sparse file, where some bytes are undefined. For example, writing at byte 601 of a resource where bytes 0-599 are defined; this would leave byte 600 undefined. Servers that accept sparse writes MUST NOT disclose uninitialized content, and SHOULD fill in undefined regions with zeros. The expected length of the write can be computed from the part fields. If the actual length of the part body mismatches the expected length, this MUST be treated the same as a network interruption at the shorter length, but anticipating the longer length. Recovering from this interruption may involve rolling back the entire request, or saving as many bytes as possible. The client can then recover as it would recover from a network interruption.

Range units Currently, the only defined range unit is "bytes", however this may be other, yet-to-be-defined values. In the case of "bytes", the bytes that are read are exactly the same as the bytes that are changed. However, other units may define write semantics different from a read, if symmetric behavior would not make sense. For example, if a Content-Range field adds an item in a JSON array, this write may add a leading or trailing comma, not technically part of the item itself, in order to keep the resulting document well-formed. Even though the length in alternate units isn't changed, the byte length might. This might only be acceptable to servers storing these values in a database or memory structure, rather than on a byte-based filesystem.

Byterange Media Types

The multipart/byteranges media type The following illustrates a request with a "multipart/byteranges" body to write two ranges in a document: The syntax for multipart messages is defined in . While the body cannot contain the boundary, servers MAY use the Content-Length field to skip to the boundary (potentially ignoring a boundary in the body, which would be an error by the client). Content-Range MUST NOT be used in place of Content-Length for this purpose. The multipart/byteranges type may be used for operations where multiple regions must be updated at the same time; clients expect all the changes to be recorded as a single operation, or that if there's an interruption, all of the parts will be rolled back together. However, the exact behavior is at the discretion of the server.

The message/byterange media type When making a request with a single byte range, there is no need for a multipart boundary marker. This document defines a new media type "message/byterange" with the same semantics as a single byte range in a multipart/byteranges message, but with a simplified syntax that only needs to be parsed to the first empty line. The "message/byterange" form may be used in a PATCH request as so: This represents a request to modify a 600-byte document, starting at a 100-byte offset, overwriting 200 bytes of it.

Syntax The syntax re-uses concepts from the "multipart/byteranges" media type, except it omits the multipart separator, and so only allows a single range to be specified. It is also similar to the "message/http" media type, except the first line (the status line or request line) is omitted; a message/byterange document can be fed into a message/http parser by first prepending a line like "PATCH / HTTP/1.1". It follows the syntax of HTTP message headers and body. It MUST include the Content-Range header field. If the message length is known by the sender, it SHOULD contain the Content-Length header field. Unknown or nonapplicable header fields MUST be ignored. The field-line and message-body productions are specified in . This document has the same semantics as a single part in a "multipart/byteranges" document () or any response with a 206 (Partial Content) status code (). A "message/byterange" document may be trivially transformed into a "multipart/byteranges" document by prepending a dash-boundary and CRLF, and appending a close-delimiter (a CRLF, dash-boundary, terminating "--", and optional CRLF).

The application/byteranges media type The "application/byteranges" has the same semantics as "multipart/byteranges" but follows a binary format similar to "message/bhttp" , which may be more suitable for some clients and servers, as all variable length strings are tagged with their length.

Syntax Parsing starts by looking for a "Known-Length Message" or an "Indeterminate-Length Message". One or the other is distinguished by the different Framing Indicator. The remainder of the message is parsed by reading fields, then the content, by a method depending on if the message is known-length or indeterminate-length. If there are additional parts, they begin immediately after the end of a Content. The "Known-Length Field Section", "Known-Length Content", "Indeterminate-Length Field Section", "Indeterminate-Length Content", "Indeterminate-Length Content Chunk", "Field Line" definitions are identical to their definition in message/bhttp. They are used in one or more Known-Length Message and/or Indeterminate-Length Message productions, concatenated together.

Preserving Incomplete Uploads with "Prefer: transaction" The stateless design of HTTP generally implies that a request is atomic (otherwise parties would need to keep track of the state of a request while it's in progress). Clients need not need be concerned with the side-effects of error halfway through an upload. However, some clients may desire partial state changes, particularly when remaking the upload is more expensive than recovering from an interruption. In these cases, clients will prefer the incomplete upload to be preserved as much as possible, so they may resume from where the incomplete request was terminated. The client's preference for atomic or upload-preserving behavior may be signaled by a Prefer header: The transaction=atomic preference indicates that the request SHOULD commit only when a successful response is returned, and not any time before the end of the upload. The transaction=persist preference indicates that uploaded data SHOULD be continuously committed, so that if the upload is interrupted, it is possible to resume the upload from where it left off. This preference is generally applicable to any HTTP request (and not merely for PATCH or byte range patches). Servers SHOULD indicate when this preference was honored, using a "Preference-Applied" response header. For example: Servers might consider signaling this in a 103 (Early Hints) response , since once the final response is written, this may no longer be useful information.

Segmented document creation with PATCH As an alternative to using PUT to create a new resource, the contents of a resource may be uploaded in segments, written across several PATCH requests. A user-agent may also use PATCH to recover from an interrupted PUT request, if it was expected to create a new resource. The server will store the data sent to it by the user agent, but will not finalize the upload until the complete length of the document is known and received.

1. The client makes a PUT or PATCH request to a URL, a portion of which is randomized, to be unpredictable to other clients. This first request creates the resource, and should include Prefer: transaction=persist (to continuously commit the upload) and If-None-Match: * (to verify the target does not exist). If a PUT request, the server reads the Content-Length header and stores the intended complete length of the document. If a PATCH request, the "Content-Range" field in the "message/byterange" patch is read for the complete length. The complete length may also be undefined, and defined in a later request.
2. If any request is interrupted, the client may make a HEAD request to determine how much, if any, of the previous response was stored, and resumes uploading from that point. The server will return 200 (OK), but this may only indicate the write has been saved; the server is not obligated to begin acting on the upload until it is complete.
3. If the client sees from the HEAD response that additional data remains to be uploaded, it may make a PATCH request to resume uploading. Even if no data was uploaded or the resource was not created, the client should attempt creating the resource with PATCH to mitigate the possibility of another interrupted connection with a server that does not save incomplete transfers. However if in response to PATCH, the server reports 405 (Method Not Allowed), 415 (Unsupported Media Type), or 501 (Not Implemented), then the client must resort to retrying the PUT request.
4. The server detects the completion of the final request when the current received data matches the indicated complete length. For example, a Content-Range: 500-599/600 field is a write at the end of the resource. The server processes the upload and returns a response for it.

If the client does not know the complete length of its upload (for example, an upload of live audio), it may use the indeterminate length form "*" to append data to the document. The end of the upload is then signaled with an "unsatisfied-range" form (e.g. Content-Range: */1000), which instructs the server that the upload is complete if it has received all 1000 bytes. For building POST endpoints that support large uploads, clients can first upload the data to a scratch file as described above, and then submit a link to the file in a POST request. For updating an existing large file, the client can upload to a scratch file, then execute a MOVE () over the intended target.

Complete Length Example A single PUT request that creates a new resource may be split apart into multiple PATCH requests. Here is an example that uploads a 600-byte document across three 200-byte segments. The first PATCH request creates the resource: This request allocates a 600 byte document, and uploading the first 200 bytes of it. The server responds with 200, indicating that the complete upload was stored. Additional requests upload the remainder of the document: This second request also returns 200 (OK). A third request uploads the final portion of the document: The server responds with 200 (OK). Since this completely writes out the 600-byte document, the server may also perform final processing, for example, checking that the document is well formed. The server MAY return an error code if there is a syntax or other error, or in an earlier response as soon as it it able to detect an error, however the exact behavior is left undefined.

Registrations

message/byterange

Type name:

message

Subtype name:

byterange

Required parameters:

N/A

Optional parameters:

N/A

Encoding considerations:

binary

Security considerations:

See

Interoperability considerations:

See of this document

Published specification:

This document

Applications that use this media type:

HTTP applications that process filesystem-like writes to locations within a resource.

Fragment identifier considerations:

N/A

Additional information:

Deprecated alias names for this type:

N/A

Magic number(s):

N/A

File extension(s):

N/A

Macintosh file type code(s):

N/A

Person and email address to contact for further information:

See Authors' Addresses section.

Intended usage:

COMMON

Restrictions on usage:

None.

Author:

See Authors' Addresses section.

Change controller:

IESG

application/byteranges

Type name:

application

Subtype name:

byteranges

Required parameters:

N/A

Optional parameters:

N/A

Encoding considerations:

binary

Security considerations:

See

Interoperability considerations:

See of this document

Published specification:

This document

Applications that use this media type:

HTTP applications that process filesystem-like writes to locations within a resource.

Fragment identifier considerations:

N/A

Additional information:

Deprecated alias names for this type:

N/A

Magic number(s):

N/A

File extension(s):

N/A

Macintosh file type code(s):

N/A

Person and email address to contact for further information:

See Authors' Addresses section.

Intended usage:

COMMON

Restrictions on usage:

None.

Author:

See Authors' Addresses section.

Change controller:

IESG

Content-Offset

Field name:

Content-Offset

Status: permanent

Specification document:

This document.

"transaction" preference

Preference:

transaction

Value:

either "atomic" or "persist"

Description:

Specify if the client would prefer incomplete uploads to be saved, or committed only on success.

Reference:

Security Considerations

Uninitialized ranges A byterange patch may permit writes to offsets beyond the end of the resource. This may have non-obvious behavior. Servers supporting sparse files MUST NOT return uninitialized memory or storage contents. Uninitialized regions may be initialized prior to executing the write, or this may be left to the filesystem if it can guarantee that unallocated space will be read as a constant value.

Initializing large documents A byte range patch typically only requires server resources proportional to the patch size. One exception is pre-initializing space when growing a file. This occurs when a complete-length is specified in the Content-Range field, which hints at the final upload size, or when writing to an offset far after the end of the file, and the server initializes the space in between. This initialization could be a very expensive operation compared to the actual size of the patch. In general, servers SHOULD treat an initialization towards resource limits, and issue a 400 (Client Error) as appropriate. Note that 413 (Payload Too Large) would be misleading in this situation, as it would indicate the patch itself is too large, and the client should break up the patches into smaller chunks, rather than the intended signal that the final upload size isd too large.

References Normative References In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. The Hypertext Transfer Protocol (HTTP) is a stateless application-level protocol for distributed, collaborative, hypertext information systems. This document describes the overall architecture of HTTP, establishes common terminology, and defines aspects of the protocol that are shared by all versions. In this definition are core protocol elements, extensibility mechanisms, and the "http" and "https" Uniform Resource Identifier (URI) schemes. This document updates RFC 3864 and obsoletes RFCs 2818, 7231, 7232, 7233, 7235, 7538, 7615, 7694, and portions of 7230. This document describes a set of data types and associated algorithms that are intended to make it easier and safer to define and handle HTTP header and trailer fields, known as "Structured Fields", "Structured Headers", or "Structured Trailers". It is intended for use by specifications of new HTTP fields that wish to use a common syntax that is more restrictive than traditional HTTP field values. The Hypertext Transfer Protocol (HTTP) is a stateless application-level protocol for distributed, collaborative, hypertext information systems. This document specifies the HTTP/1.1 message syntax, message parsing, connection management, and related security concerns. This document obsoletes portions of RFC 7230. RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings. Internet technical specifications often need to define a formal syntax. Over the years, a modified version of Backus-Naur Form (BNF), called Augmented BNF (ABNF), has been popular among many Internet specifications. The current specification documents ABNF. It balances compactness and simplicity with reasonable representational power. The differences between standard BNF and ABNF involve naming rules, repetition, alternatives, order-independence, and value ranges. This specification also supplies additional rule definitions and encoding for a core lexical analyzer of the type common to several Internet specifications. [STANDARDS-TRACK] Informative References This second document defines the general structure of the MIME media typing system and defines an initial set of media types. [STANDARDS-TRACK] Web Distributed Authoring and Versioning (WebDAV) consists of a set of methods, headers, and content-types ancillary to HTTP/1.1 for the management of resource properties, creation and management of resource collections, URL namespace manipulation, and resource locking (collision avoidance). RFC 2518 was published in February 1999, and this specification obsoletes RFC 2518 with minor revisions mostly due to interoperability experience. [STANDARDS-TRACK] Several applications extending the Hypertext Transfer Protocol (HTTP) require a feature to do partial resource modification. The existing HTTP PUT method only allows a complete replacement of a document. This proposal adds a new HTTP method, PATCH, to modify an existing HTTP resource. [STANDARDS-TRACK] This memo introduces an informational HTTP status code that can be used to convey hints that help a client make preparations for processing the final response. This document defines a binary format for representing HTTP messages.

Discussion This section to be removed before final publication.

Sparse Documents This pattern can enable multiple, parallel uploads to a document at the same time. For example, uploading a large log file from multiple devices. However, this document does not define any ways for clients to track the unwritten regions in sparse documents, and the existing conditional request headers are designed to cause conflicts. Parallel uploads may require a byte-level locking scheme or conflict-free operators. This may be addressed in a later document.

Recovering from interrupted PUT Servers do not necessarily save the results of an incomplete upload; since most clients prefer atomic writes, many servers will discard an incomplete upload. A mechanism to indicate a preference for atomic vs. non-atomic writes may be defined at a later time. Byte range PATCH cannot by itself provide recovery from an interrupted PUT to an existing document, as it is not generally possible to distinguish what was received by the server from what already existed. One technique would be to use a 1xx interim response to indicate a location where the partial upload is being stored. If PUT request is interrupted, the client can make PATCH requests to this temporary, non-atomic location to complete the upload. When the last part is uploaded, the original interrupted PUT request will finish. Another technique would be to send a 1xx interim response to indicate how much of the upload has been committed. When a client receives this, it can skip that many bytes from the next attempt in a PATCH request.

Splicing and Binary Diff Operations more complicated than standard filesystem operations are out of scope for this media type. A feature of byte range patch is an upper limit on the complexity of applying the patch. In contrast, prepending, splicing, replace, or other complicated file operations could potentially require the entire file on disk be rewritten. Consider registering a media type for VCDIFF in this document, under the topic of "Media type registrations for byte-level patching".

Ending Indeterminate Length Uploads When uploading a patch with indeterminate-length form, Since the server doesn't know the complete-length, it might not be able to assume that the end of the request is also the end of the whole document, even if the request ended cleanly (the client may have reasons to split apart the upload). This needs to be signaled by a subsequent request with an unsatisfied-range specifying the final length of the document.

Reading Content-Range and Content-Offset in the headers Some parties may prefer that the message body need not be parsed at all, but instead leverage existing HTTP mechanisms to carry the part field data. Similar to how a 206 status code indicates that the response body is partial, a reserved value of Content-Type (in a PATCH request) could signal the server to read the Content-Range from the request headers, and the whole request body is the part body.