http-server-close and HTTP uploads/content-length header

Re: http-server-close and HTTP uploads/content-length header

Willy Tarreau — Wed, 28 Mar 2012 22:00:01 +0200

On Wed, Mar 28, 2012 at 09:06:33AM -0500, Rusty Geldmacher wrote:
> Hi Willy,
>
> Thanks so much for your responses! We're using version 1.4.19. I'll work
> on getting a network capture for you -- what tcpdump params would help the
> most? I usually use "-vv -A -w ". Probably the most helpful thing
> would be to dump port 81, which is the unencrypted traffic coming it
> HAProxy from stunnel.

Yes it should be enough. Flags you should use are "-s0 -w " so that
we get complete packets to see where data is missing or mangled. I also
need the data between haproxy and the server to compare the damage that
haproxy might have done there. And if you can offer me the haproxy logs
at the same time it would be wonderful :-)

Thank you!
Willy

Re: http-server-close and HTTP uploads/content-length header

Rusty Geldmacher — Wed, 28 Mar 2012 16:10:02 +0200

Hi Willy,

Thanks so much for your responses! We're using version 1.4.19. I'll work
on getting a network capture for you -- what tcpdump params would help the
most? I usually use "-vv -A -w ". Probably the most helpful thing
would be to dump port 81, which is the unencrypted traffic coming it
HAProxy from stunnel.

Thanks again,
rusty

On 3/27/12 2:21 PM, "Willy Tarreau" wrote:

>Hi Rusty,
>
>On Mon, Mar 26, 2012 at 02:07:40PM -0500, Rusty Geldmacher wrote:
>> Hi all,
>>
>> Was hoping someone could help shed some light on an issue we're
>>observing in
>> our setup. When we have "option http-server-close" set, then file
>>uploads via
>> multipart form data will not work. By "not work" I mean that the HTTP
>> content-length header will be reported as 0, which in turn causes the
>>final
>> destination to get seriously confused by the request and then give up. I
>> realize this sounds strange, as content-length is reported by the
>>client but
>> this is the behavior I'm witnessing. When I have http-server-close on,
>>the
>> content-length is 0, and when I remove the option the content-length is
>>as
>> expected ? with no changes on the client end.
>
>That's strange indeed. What version is this, so that we can check the
>changelog if any known bug might have had that side effect ?
>
>> Additionally, if we set "option httpclose" then everything will work.
>
>I'm not surprized, "option httpclose" only adjusts the Connection header
>and completely ignores framing. It's what we call the tunnel mode : both
>sides are free to exchange what they want past the headers.
>
>> However, I'm under the impression that using httpclose isn't ideal since
>> clients then wouldn't be able to send multiple HTTP requests across the
>>same
>> connection.
>
>You're right.
>
>> Since this is a mobile app, having that functionality would
>> enhance performance.
>
>For sure !
>
>> Setting http-server-close seems to be ideal because we
>> can correctly load balance multiple requests in the same session.
>
>Yes and it was designed exactly for this.
>
>> So, I'm wondering:
>>
>> * Does haproxy do anything to validate/calculate the content-length
>>header?
>
>Yes, it parses the Content-length header or alternatively considers the
>Transfer-Encoding if it's present and announced as "chunked".
>
>> * How would the presence of http-server-close affect the request's
>>content-length header?
>
>It only checks the header and should not mangle it. Well, there is only
>one
>case it changes it, it's when multiple similar content-length headers are
>sent, then it reduces them to only one.
>
>> At a high level, our request flow looks like this:
>>
>> stunnel -> haproxy -> apache -> passenger
>>
>> Where stunnel and haproxy are on the same machine, and apache/passenger
>>are
>> on three or four additional machines.
>>
>> Here's the relevant snippet from haproxy.cfg:
>>
>> defaults
>> mode http
>> log global
>> option dontlognull
>> option redispatch
>> retries 3
>> timeout http-request 10s
>> timeout queue 1m
>> timeout connect 10s
>> timeout client 1m
>> timeout server 1m
>> timeout http-keep-alive 500
>> timeout check 10s
>> maxconn 3000
>>
>> # Stunnel listens on 443 and forwards connections off to haproxy on 81
>> frontend https *:81
>> option httplog
>> # Enabling http-server-close will cause file uploads to fail!
>> # option http-server-close
>> option forwardfor except 127.0.0.0/8
>>
>> reqadd X-Forwarded-Proto:\ https
>>
>> # set some ACLS?
>> # choose a backend?
>> # Nothing else really special
>
>Nothing fancy here !
>
>> And here is a session log for each case, starting with the normal
>>operation
>> (no http-server-close set, or setting httpclose):
>>
>> 00000020:https.accept(0006)=000a from [10.0.1.30:43419]
>> 00000020:https.clireq[000a:ffff]: POST /URL-GOES-HERE HTTP/1.1
>> 00000020:https.clihdr[000a:ffff]: Host: dev.sermo.com
>> 00000020:https.clihdr[000a:ffff]: Content-Type: multipart/form-data;
>>charset=utf-8; boundary=0xKhTmLbOuNdArY
>> 00000020:https.clihdr[000a:ffff]: User-Agent: iPhone Simulator; iPhone
>>OS 4.3.2; en_US
>> 00000020:https.clihdr[000a:ffff]: Accept-Encoding: gzip
>> 00000020:https.clihdr[000a:ffff]: Content-Length: 62528
>> 00000020:https.clihdr[000a:ffff]: Connection: keep-alive
>>
>> And here is with nothing else changed except for the addition of the
>> http-server-close option:
>>
>> 00000007:https.accept(0006)=000a from [10.0.1.30:43688]
>> 00000007:https.clireq[000a:ffff]: POST /URL-GOES-HERE HTTP/1.1
>> 00000007:https.clihdr[000a:ffff]: Host: dev.sermo.com
>> 00000007:https.clihdr[000a:ffff]: Content-Type: multipart/form-data;
>>charset=utf-8; boundary=0xKhTmLbOuNdArY
>> 00000007:https.clihdr[000a:ffff]: User-Agent: iPhone Simulator; iPhone
>>OS 4.3.2; en_US
>> 00000007:https.clihdr[000a:ffff]: Accept-Encoding: gzip
>> 00000007:https.clihdr[000a:ffff]: Content-Length: 0
>> 00000007:https.clihdr[000a:ffff]: Connection: keep-alive
>
>Wow this is amazing !
>I've never seen that and don't understand how that may happen. I'm relly
>interested in knowing your version to try to reproduce.
>
>> We can get the system to work if instead of setting http-server-close,
>>we set
>> httpclose but like I explained above this doesn't seem to be ideal.
>
>I agree this is not a good solution, we need to fix this.
>
>If you could get a network capture for each side too, that would help
>a lot. Don't post it to the list if there are sensible information there.
>
>Regards,
>Willy
>

Re: http-server-close and HTTP uploads/content-length header

Willy Tarreau — Tue, 27 Mar 2012 20:30:01 +0200

Hi Rusty,

On Mon, Mar 26, 2012 at 02:07:40PM -0500, Rusty Geldmacher wrote:
> Hi all,
>
> Was hoping someone could help shed some light on an issue we're observing in
> our setup. When we have "option http-server-close" set, then file uploads via
> multipart form data will not work. By "not work" I mean that the HTTP
> content-length header will be reported as 0, which in turn causes the final
> destination to get seriously confused by the request and then give up. I
> realize this sounds strange, as content-length is reported by the client but
> this is the behavior I'm witnessing. When I have http-server-close on, the
> content-length is 0, and when I remove the option the content-length is as
> expected ? with no changes on the client end.

That's strange indeed. What version is this, so that we can check the
changelog if any known bug might have had that side effect ?

> Additionally, if we set "option httpclose" then everything will work.

I'm not surprized, "option httpclose" only adjusts the Connection header
and completely ignores framing. It's what we call the tunnel mode : both
sides are free to exchange what they want past the headers.

> However, I'm under the impression that using httpclose isn't ideal since
> clients then wouldn't be able to send multiple HTTP requests across the same
> connection.

You're right.

> Since this is a mobile app, having that functionality would
> enhance performance.

For sure !

> Setting http-server-close seems to be ideal because we
> can correctly load balance multiple requests in the same session.

Yes and it was designed exactly for this.

> So, I'm wondering:
>
> * Does haproxy do anything to validate/calculate the content-length header?

Yes, it parses the Content-length header or alternatively considers the
Transfer-Encoding if it's present and announced as "chunked".

> * How would the presence of http-server-close affect the request's content-length header?

It only checks the header and should not mangle it. Well, there is only one
case it changes it, it's when multiple similar content-length headers are
sent, then it reduces them to only one.

> At a high level, our request flow looks like this:
>
> stunnel -> haproxy -> apache -> passenger
>
> Where stunnel and haproxy are on the same machine, and apache/passenger are
> on three or four additional machines.
>
> Here's the relevant snippet from haproxy.cfg:
>
> defaults
> mode http
> log global
> option dontlognull
> option redispatch
> retries 3
> timeout http-request 10s
> timeout queue 1m
> timeout connect 10s
> timeout client 1m
> timeout server 1m
> timeout http-keep-alive 500
> timeout check 10s
> maxconn 3000
>
> # Stunnel listens on 443 and forwards connections off to haproxy on 81
> frontend https *:81
> option httplog
> # Enabling http-server-close will cause file uploads to fail!
> # option http-server-close
> option forwardfor except 127.0.0.0/8
>
> reqadd X-Forwarded-Proto:\ https
>
> # set some ACLS?
> # choose a backend?
> # Nothing else really special

Nothing fancy here !

> And here is a session log for each case, starting with the normal operation
> (no http-server-close set, or setting httpclose):
>
> 00000020:https.accept(0006)=000a from [10.0.1.30:43419]
> 00000020:https.clireq[000a:ffff]: POST /URL-GOES-HERE HTTP/1.1
> 00000020:https.clihdr[000a:ffff]: Host: dev.sermo.com
> 00000020:https.clihdr[000a:ffff]: Content-Type: multipart/form-data; charset=utf-8; boundary=0xKhTmLbOuNdArY
> 00000020:https.clihdr[000a:ffff]: User-Agent: iPhone Simulator; iPhone OS 4.3.2; en_US
> 00000020:https.clihdr[000a:ffff]: Accept-Encoding: gzip
> 00000020:https.clihdr[000a:ffff]: Content-Length: 62528
> 00000020:https.clihdr[000a:ffff]: Connection: keep-alive
>
> And here is with nothing else changed except for the addition of the
> http-server-close option:
>
> 00000007:https.accept(0006)=000a from [10.0.1.30:43688]
> 00000007:https.clireq[000a:ffff]: POST /URL-GOES-HERE HTTP/1.1
> 00000007:https.clihdr[000a:ffff]: Host: dev.sermo.com
> 00000007:https.clihdr[000a:ffff]: Content-Type: multipart/form-data; charset=utf-8; boundary=0xKhTmLbOuNdArY
> 00000007:https.clihdr[000a:ffff]: User-Agent: iPhone Simulator; iPhone OS 4.3.2; en_US
> 00000007:https.clihdr[000a:ffff]: Accept-Encoding: gzip
> 00000007:https.clihdr[000a:ffff]: Content-Length: 0
> 00000007:https.clihdr[000a:ffff]: Connection: keep-alive

Wow this is amazing !
I've never seen that and don't understand how that may happen. I'm relly
interested in knowing your version to try to reproduce.

> We can get the system to work if instead of setting http-server-close, we set
> httpclose but like I explained above this doesn't seem to be ideal.

I agree this is not a good solution, we need to fix this.

If you could get a network capture for each side too, that would help
a lot. Don't post it to the list if there are sensible information there.

Regards,
Willy

http-server-close and HTTP uploads/content-length header

Rusty Geldmacher — Mon, 26 Mar 2012 21:10:01 +0200

Hi all,

Was hoping someone could help shed some light on an issue we're observing in our setup. When we have "option http-server-close" set, then file uploads via multipart form data will not work. By "not work" I mean that the HTTP content-length header will be reported as 0, which in turn causes the final destination to get seriously confused by the request and then give up. I realize this sounds strange, as content-length is reported by the client but this is the behavior I'm witnessing. When I have http-server-close on, the content-length is 0, and when I remove the option the content-length is as expected — with no changes on the client end.

Additionally, if we set "option httpclose" then everything will work. However, I'm under the impression that using httpclose isn't ideal since clients then wouldn't be able to send multiple HTTP requests across the same connection. Since this is a mobile app, having that functionality would enhance performance. Setting http-server-close seems to be ideal because we can correctly load balance multiple requests in the same session.

So, I'm wondering:

* Does haproxy do anything to validate/calculate the content-length header?
* How would the presence of http-server-close affect the request's content-length header?

At a high level, our request flow looks like this:

stunnel -> haproxy -> apache -> passenger

Where stunnel and haproxy are on the same machine, and apache/passenger are on three or four additional machines.

Here's the relevant snippet from haproxy.cfg:

defaults
mode http
log global
option dontlognull
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 500
timeout check 10s
maxconn 3000

# Stunnel listens on 443 and forwards connections off to haproxy on 81
frontend https *:81
option httplog
# Enabling http-server-close will cause file uploads to fail!
# option http-server-close
option forwardfor except 127.0.0.0/8

reqadd X-Forwarded-Proto:\ https

# set some ACLS…
# choose a backend…
# Nothing else really special

And here is a session log for each case, starting with the normal operation (no http-server-close set, or setting httpclose):

00000020:https.accept(0006)=000a from [10.0.1.30:43419]
00000020:https.clireq[000a:ffff]: POST /URL-GOES-HERE HTTP/1.1
00000020:https.clihdr[000a:ffff]: Host: dev.sermo.com
00000020:https.clihdr[000a:ffff]: Content-Type: multipart/form-data; charset=utf-8; boundary=0xKhTmLbOuNdArY
00000020:https.clihdr[000a:ffff]: User-Agent: iPhone Simulator; iPhone OS 4..3.2; en_US
00000020:https.clihdr[000a:ffff]: Accept-Encoding: gzip
00000020:https.clihdr[000a:ffff]: Content-Length: 62528
00000020:https.clihdr[000a:ffff]: Connection: keep-alive

And here is with nothing else changed except for the addition of the http-server-close option:

00000007:https.accept(0006)=000a from [10.0.1.30:43688]
00000007:https.clireq[000a:ffff]: POST /URL-GOES-HERE HTTP/1.1
00000007:https.clihdr[000a:ffff]: Host: dev.sermo.com
00000007:https.clihdr[000a:ffff]: Content-Type: multipart/form-data; charset=utf-8; boundary=0xKhTmLbOuNdArY
00000007:https.clihdr[000a:ffff]: User-Agent: iPhone Simulator; iPhone OS 4..3.2; en_US
00000007:https.clihdr[000a:ffff]: Accept-Encoding: gzip
00000007:https.clihdr[000a:ffff]: Content-Length: 0
00000007:https.clihdr[000a:ffff]: Connection: keep-alive

We can get the system to work if instead of setting http-server-close, we set httpclose but like I explained above this doesn't seem to be ideal.

Any help or pointers would be appreciated!

Thanks,
rg