Welcome! Log In Create A New Profile

Advanced

http/2 PUT's without content-length fail to http 1.1 backend

Posted by Robert Samuel Newson 
Robert Samuel Newson
http/2 PUT's without content-length fail to http 1.1 backend
February 27, 2018 09:30PM
Hi,

I use haproxy (1.8.4) with http/2 support in front of a server that speaks http 1.1. This is working great with one exception. Several http/2 client libraries are sending PUT requests without sending the Content-Length header (as it' not strictly needed due to the framing). The http 1.1 request issued to the backend is therefore malformed as the header is required there.

I think the right thing here is for haproxy to convert the PUT to a chunked transfer request to the backend.

Thoughts?

B.
Hi Robert,

On Tue, Feb 27, 2018 at 08:26:01PM +0000, Robert Samuel Newson wrote:
> Hi,
>
> I use haproxy (1.8.4) with http/2 support in front of a server that speaks
> http 1.1. This is working great with one exception. Several http/2 client
> libraries are sending PUT requests without sending the Content-Length header
> (as it' not strictly needed due to the framing).

At first I thought this was not valid but in fact it obviously is since
it is the equivalent of chunked encoding for HTTP/1. Interestingly, just
like chunked uploads are not much documented in the HTTP/1 specs, this
frame-only transfer is never mentionned in the HTTP/2 spec and I totally
overlooked it.

> The http 1.1 request issued
> to the backend is therefore malformed as the header is required there.
>
> I think the right thing here is for haproxy to convert the PUT to a chunked
> transfer request to the backend.

Yes I think so as well. It will be a bit tricky though. From what I'm
imagining, we'll have to add "Transfer-encoding: chunked" if the HEADERS
frame doesn't carry the END_STREAM flag and there is no Content-length
header, then all DATA frames will have to be prefixed with a chunk size
and the frame carrying the ES flag will have to cause the final empty
chunk to be sent. This one will be a bit delicate I think, as we don't
want to lose it if the output buffer is full and we want not to miss the
event.

Also we must not do this for the CONNECT method!

I'm adding this to the todo list of fixes for H2. I don't promise to
have it by 1.8.5 though, but I'll try.

Thanks!
Willy
Robert Samuel Newson
Re: http/2 PUT's without content-length fail to http 1.1 backend
March 01, 2018 10:50AM
Hi,

Yup, agreed, the frame-only transfers are only really implied in the spec (8.1.2.6's, "A request or response that includes a payload body _can_ include a content-length header field", my emphasis). The http 2 spec does specifically prohibit the transfer-encoding: chunked header, again implying that the framing mechanism subsumes it.

There are a few libraries (like https://github.com/go-kivik/couchdb) which implement PUT without sending Content-Length, but other tools, like good old curl, which do send it.

The basic plan to transform to a chunked transfer encoding sounds right to me, and I'll keep an eye out for commits, I'm happy to test it out.

Thanks!

B.

> On 1 Mar 2018, at 06:16, Willy Tarreau <[email protected]> wrote:
>
> Hi Robert,
>
> On Tue, Feb 27, 2018 at 08:26:01PM +0000, Robert Samuel Newson wrote:
>> Hi,
>>
>> I use haproxy (1.8.4) with http/2 support in front of a server that speaks
>> http 1.1. This is working great with one exception. Several http/2 client
>> libraries are sending PUT requests without sending the Content-Length header
>> (as it' not strictly needed due to the framing).
>
> At first I thought this was not valid but in fact it obviously is since
> it is the equivalent of chunked encoding for HTTP/1. Interestingly, just
> like chunked uploads are not much documented in the HTTP/1 specs, this
> frame-only transfer is never mentionned in the HTTP/2 spec and I totally
> overlooked it.
>
>> The http 1.1 request issued
>> to the backend is therefore malformed as the header is required there.
>>
>> I think the right thing here is for haproxy to convert the PUT to a chunked
>> transfer request to the backend.
>
> Yes I think so as well. It will be a bit tricky though. From what I'm
> imagining, we'll have to add "Transfer-encoding: chunked" if the HEADERS
> frame doesn't carry the END_STREAM flag and there is no Content-length
> header, then all DATA frames will have to be prefixed with a chunk size
> and the frame carrying the ES flag will have to cause the final empty
> chunk to be sent. This one will be a bit delicate I think, as we don't
> want to lose it if the output buffer is full and we want not to miss the
> event.
>
> Also we must not do this for the CONNECT method!
>
> I'm adding this to the todo list of fixes for H2. I don't promise to
> have it by 1.8.5 though, but I'll try.
>
> Thanks!
> Willy
On Thu, Mar 01, 2018 at 09:38:10AM +0000, Robert Samuel Newson wrote:
> Yup, agreed, the frame-only transfers are only really implied in the spec
> (8.1.2.6's, "A request or response that includes a payload body _can_ include
> a content-length header field", my emphasis). The http 2 spec does
> specifically prohibit the transfer-encoding: chunked header, again implying
> that the framing mechanism subsumes it.

In fact it's even worse for me : I identified I had to do it, as indicated
by the comments on top of function h2_frt_transfer_data() which claim that
when no C-L nor tunnel is found, chunks are emitted. But apparently relying
too much on the checks I had on the paper-written RFC allowed me to completely
forget about this part! I've checked if I had a temporary implementation in
one of my previous dev branches but no, so I think I never wrote that code.
The good point is that the code was written with this in mind, so I hope I
won't face too big a surprise (and none of the usual "ah now I remember why").

> There are a few libraries (like https://github.com/go-kivik/couchdb) which
> implement PUT without sending Content-Length, but other tools, like good old
> curl, which do send it.

Good to know, thanks for the links. Indeed, curl always used C-L.

> The basic plan to transform to a chunked transfer encoding sounds right to
> me, and I'll keep an eye out for commits, I'm happy to test it out.

Thanks!
Willy
Robert Samuel Newson
Re: http/2 PUT's without content-length fail to http 1.1 backend
March 10, 2018 07:20PM
Hi,

Whenever you look into this, I noticed that a streamed upload also fails, but differently;

cat <large file> | acurl https://rnewson.cloudant.com/db1/doc1/att1 -XPUT -T-

curl chooses http/2 and sends the Transfer-Encoding: chunked header, and then streams the file.

The result is that the body is not received, the resulting upload is zero bytes.

I mention this for completeness but it would seem curl is at fault here, it should not send the Transfer-Encoding header, but I'm not sure haproxy is entirely innocent either?

B.

* acurl is simple a curl alias that adds my user:pass credentials.

> On 1 Mar 2018, at 11:06, Willy Tarreau <[email protected]> wrote:
>
> On Thu, Mar 01, 2018 at 09:38:10AM +0000, Robert Samuel Newson wrote:
>> Yup, agreed, the frame-only transfers are only really implied in the spec
>> (8.1.2.6's, "A request or response that includes a payload body _can_ include
>> a content-length header field", my emphasis). The http 2 spec does
>> specifically prohibit the transfer-encoding: chunked header, again implying
>> that the framing mechanism subsumes it.
>
> In fact it's even worse for me : I identified I had to do it, as indicated
> by the comments on top of function h2_frt_transfer_data() which claim that
> when no C-L nor tunnel is found, chunks are emitted. But apparently relying
> too much on the checks I had on the paper-written RFC allowed me to completely
> forget about this part! I've checked if I had a temporary implementation in
> one of my previous dev branches but no, so I think I never wrote that code.
> The good point is that the code was written with this in mind, so I hope I
> won't face too big a surprise (and none of the usual "ah now I remember why").
>
>> There are a few libraries (like https://github.com/go-kivik/couchdb) which
>> implement PUT without sending Content-Length, but other tools, like good old
>> curl, which do send it.
>
> Good to know, thanks for the links. Indeed, curl always used C-L.
>
>> The basic plan to transform to a chunked transfer encoding sounds right to
>> me, and I'll keep an eye out for commits, I'm happy to test it out.
>
> Thanks!
> Willy
Hi Robert,

On Sat, Mar 10, 2018 at 06:16:21PM +0000, Robert Samuel Newson wrote:
> Hi,
>
> Whenever you look into this, I noticed that a streamed upload also fails, but differently;
>
> cat <large file> | acurl https://rnewson.cloudant.com/db1/doc1/att1 -XPUT -T-
>
> curl chooses http/2 and sends the Transfer-Encoding: chunked header, and then streams the file.
>
> The result is that the body is not received, the resulting upload is zero bytes.
>
> I mention this for completeness but it would seem curl is at fault here, it
> should not send the Transfer-Encoding header, but I'm not sure haproxy is
> entirely innocent either?

The header should indeed not be sent, but regardless, haproxy is culprit here
for not being able to process DATA frames without a content-length. I have
already started to look into this and am trying to figure how to best address
the various issues currently pending in the same area without having to
reinvent everything each time ;-)

Thanks for the heads up, it will definitely help me.

Willy
Sorry, only registered users may post in this forum.

Click here to login