Welcome! Log In Create A New Profile

Advanced

Issue in HAproxy requests with http-send-name-header

Posted by Roque Porchetto 
Roque Porchetto
Issue in HAproxy requests with http-send-name-header
February 06, 2018 05:50PM
Hello,

I'm working on a scalability test project that currently involves to
simulate several user logins on a system whose load balancing relies on
haproxy (latest stable version 1.8.3).

In a scenario where 20 or more users try to login to the system, around 20%
of them fail. After intense debugging and analysis of tcpdumps, as far as I
see, the issue seems to come from haproxy: some of the http requests are
malformed, due the headers are inserted in a bad way. Specifically, the
header "http-send-name-header". If that keyword is removed from the haproxy
configuration, the error is no longer reproduced. But http-send-name-header
is needed by the system to manage user-backends relations through cookies.

Here an example of a request malformed by haproxy:
[image: Imágenes integradas 2]
That request must have been:

....

X-Balancer-Current-Cookie: SERVERID

X-Balancer-Current-Server: user-0

cancel_url=http.......&__ac_password=myPassword
....


But we can see in the malformed request that the X-Balancer-Current-Server
key-value was inserted inside the body changing the value of __ac_password.

The same happens using other http-send-name-header values, like:
'http-send-name-header Host'.

Here is my haproxy configuration (here only one backend server for testing
purpose, final setup would have many backends servers) :

global
maxconn 4096
stats socket /srv/slapgrid/slappart15/var/run/haproxy.sock level admin

defaults
log global
mode http
option httplog
option dontlognull
retries 1
option redispatch
maxconn 2000
cookie SERVERID rewrite
http-send-name-header X-Balancer-Current-Server
balance roundrobin
stats uri /haproxy
stats realm Global\ statistics
timeout server 305s
timeout queue 60s
timeout connect 5s
timeout client 305s
option forceclose

listen user
bind 10.0.15.62:2152
http-request set-header X-Balancer-Current-Cookie SERVERID
server user-0 10.0.5.183:2200 cookie user-0 check inter 3s rise 1 fall 2
maxqueue 5 maxconn 1
option httpchk GET /



Potential related issue:
I found this discussion about a similar issue: http://www.serverphorums.com/
read.php?10,592865. I don't think is the same case, but maybe they are
related.

Please let me know if you need any extra information.

Thank you very much in advance.
Regards,
Roque
Attachments:
open | download - image.png (103.7 KB)
Willy Tarreau
Re: Issue in HAproxy requests with http-send-name-header
February 08, 2018 09:50AM
Hello Roque,

On Tue, Feb 06, 2018 at 05:37:32PM +0100, Roque Porchetto wrote:
> Hello,
>
> I'm working on a scalability test project that currently involves to
> simulate several user logins on a system whose load balancing relies on
> haproxy (latest stable version 1.8.3).
>
> In a scenario where 20 or more users try to login to the system, around 20%
> of them fail. After intense debugging and analysis of tcpdumps, as far as I
> see, the issue seems to come from haproxy: some of the http requests are
> malformed, due the headers are inserted in a bad way. Specifically, the
> header "http-send-name-header". If that keyword is removed from the haproxy
> configuration, the error is no longer reproduced. But http-send-name-header
> is needed by the system to manage user-backends relations through cookies.

Argh, that's not fun, because it's the worst option ever brought to haproxy,
regularly breaking due to insignificant changes :-(

> Here an example of a request malformed by haproxy:
> [image: Imágenes integradas 2]

(note: in the future, please avoid posting images for header captures,
it's quite a pain to read, search and comment on).

> That request must have been:
>
> ...
>
> X-Balancer-Current-Cookie: SERVERID
>
> X-Balancer-Current-Server: user-0
>
> cancel_url=http.......&__ac_password=myPassword
> ...
>
>
> But we can see in the malformed request that the X-Balancer-Current-Server
> key-value was inserted inside the body changing the value of __ac_password.

Now I'm seeing it. So it means that we're having something wrong with the
position computation once data start to come in the buffer. Did you notice
this problem with 1.7 or not ? The random speed at which data arrive in the
buffer explain the fact that you observe neither 0% nor 100% hit.

Could you please check if adding "option http-buffer-request" makes it
worse or better ? I guess you'll get either 100% failure or 100% success,
this will help troubleshooting (and will possibly help you workaround the
issue for some time).

Thanks!
Willy
Willy Tarreau
Re: Issue in HAproxy requests with http-send-name-header
February 08, 2018 10:30AM
On Thu, Feb 08, 2018 at 09:44:38AM +0100, Willy Tarreau wrote:
> Now I'm seeing it. So it means that we're having something wrong with the
> position computation once data start to come in the buffer. Did you notice
> this problem with 1.7 or not ? The random speed at which data arrive in the
> buffer explain the fact that you observe neither 0% nor 100% hit.
>
> Could you please check if adding "option http-buffer-request" makes it
> worse or better ? I guess you'll get either 100% failure or 100% success,
> this will help troubleshooting (and will possibly help you workaround the
> issue for some time).

By the way, I tried hard but never managed to get it to fail at all, so the
tests above will be useful. Also please check in your logs if the faulty
requests have experienced any retry or redispatch. It's in these cases that
the http-send-name-header gets trickier. And since you're using maxconn 1,
I'm wondering if one reason might not be because the server behind is a bit
limited, possibly causing a significant enough number of retries exhibiting
the bug.

Thanks,
Willy
Roque Porchetto
Re: Issue in HAproxy requests with http-send-name-header
February 08, 2018 02:40PM
Hello Willy,

thank you very much for your time on this. I'm adding here some extra
information:

Did you notice this problem with 1.7 or not ?


No, I tried with 1.8.1 (required by the system I'm using) and then I moved
to last version to try. Downgrading the version would take me some hours
due I would have to rebuild the whole system.

Could you please check if adding "option http-buffer-request" makes it
> worse or better ?


Yes! With that option I get 100% success.

Also please check in your logs if the faulty requests have experienced any
> retry or redispatch.


The faulty requests are not being retried or similar, because they don't
generate a failure, they just arrive to the login form with a wrong
(corrupt) password, so the login fails but the requests are ok.

I tried also another thing, related to the content-type of the requests,
and maybe could help you troubleshooting.
If I sent the requests from the client with the header "
*X-Content-Type-Options*" = "*nosniff*", the error is no longer reproduced.

I hope this helps, please let me know of you need any extra details or to
try more tests.

Regards,
Roque





2018-02-08 10:22 GMT+01:00 Willy Tarreau <[email protected]>:

> On Thu, Feb 08, 2018 at 09:44:38AM +0100, Willy Tarreau wrote:
> > Now I'm seeing it. So it means that we're having something wrong with the
> > position computation once data start to come in the buffer. Did you
> notice
> > this problem with 1.7 or not ? The random speed at which data arrive in
> the
> > buffer explain the fact that you observe neither 0% nor 100% hit.
> >
> > Could you please check if adding "option http-buffer-request" makes it
> > worse or better ? I guess you'll get either 100% failure or 100% success,
> > this will help troubleshooting (and will possibly help you workaround the
> > issue for some time).
>
> By the way, I tried hard but never managed to get it to fail at all, so the
> tests above will be useful. Also please check in your logs if the faulty
> requests have experienced any retry or redispatch. It's in these cases that
> the http-send-name-header gets trickier. And since you're using maxconn 1,
> I'm wondering if one reason might not be because the server behind is a bit
> limited, possibly causing a significant enough number of retries exhibiting
> the bug.
>
> Thanks,
> Willy
>



--
Roque Porchetto
Sorry, only registered users may post in this forum.

Click here to login