Welcome! Log In Create A New Profile

Advanced

Request bodies don't get proxied sometimes in version 1.8.x

Posted by Nick Vatamaniuc 
Hello,

Sometimes PUT/POST request bodies don't get proxied. Headers are sent
through, server then waits for the body to be sent but the body data never
arrives and the socket times out after 5 minutes.

This seems to be a regression in 1.8, reverting to 1.7.9 fixes the issue.

haproxy version: 1.8.7. 3 backend servers. Using Lua on all the request
paths.
nbproc > 1 and not using threads.

Normally the setup handles 500k to 1.5m successful PUT/POST requests per
day.
About 5-10 or so end up failing as described above.

Some config settings:

option http-server-close
option log-health-checks
option log-separate-errors
option redispatch
retries 4
timeout client 150000
timeout connect 5000
timeout queue 5000
timeout server 3600000

More details about the failing requests:

Body size: about 30k-50k
Tw = 0
Tc = 0 in 90% and Tc=1 in 10% of the cases
Tq is about 5-10 on average (with some outliers)
actconn/feconn: 140 on average
beconn: 30 on average
Termination state: ---- for all requests
None are http/2

Things we tried which failed to fix the issue:

Upgraded to haproxy 1.8.8 + git up to
80e179128cfd78d95cdebf7195fd21299e7931b6

Disabled HTTP/2. None of the failing connections are HTTP/2, but was trying
to
test the theory that successful HTTP/2 connections somehow altered the
internal state of the server and affected those failing requests.

The only thing that fixed the issue was downgrading to haproxy 1.7.9

Thank you,
-Nick



/usr/sbin/haproxy -vv

HA-Proxy version 1.8.7 2018/04/07
Copyright 2000-2018 Willy Tarreau <[email protected]>
Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -m64 -march=x86-64 -O2 -g -fno-strict-aliasing
-Wdeclaration-after-statement -fwrapv -fno-strict-overflow -Wno-unused-label
OPTIONS = USE_SLZ=1 USE_REGPARM=1 USE_THREAD=1 USE_OPENSSL=1 USE_LUA=1
USE_SYSTEMD=1 USE_STATIC_PCRE2=1 USE_PCRE2_JIT=1 USE_TFO=1
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with OpenSSL version : OpenSSL 1.1.0h 27 Mar 2018
Running on OpenSSL version : OpenSSL 1.1.0h 27 Mar 2018
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2
Built with Lua version : Lua 5.3.4
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT
IP_FREEBIND
Encrypted password support via crypt(3): yes
Built with multi-threading support.
Built with PCRE2 version : 10.30 2017-08-14
PCRE2 library supports JIT : yes
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"),
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with network namespace support.
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available filters :
[SPOE] spoe
[COMP] compression
[TRACE] trace
Robert Samuel Newson
Re: Request bodies don't get proxied sometimes in version 1.8.x
May 10, 2018 05:10PM
We've found that just switching back to 1.7.9 makes this issue disappear. It's a really hard one to pin down though. It's quite rare and we've had no luck building a reliable reproducing script.

Is there anything more we can provide to give insight into why haproxy might sometimes hang during request processing?

B.

> On 8 May 2018, at 16:54, Nick Vatamaniuc <[email protected]> wrote:
>
> Hello,
>
> Sometimes PUT/POST request bodies don't get proxied. Headers are sent
> through, server then waits for the body to be sent but the body data never
> arrives and the socket times out after 5 minutes.
>
> This seems to be a regression in 1.8, reverting to 1.7.9 fixes the issue.
>
> haproxy version: 1.8.7. 3 backend servers. Using Lua on all the request paths.
> nbproc > 1 and not using threads.
>
> Normally the setup handles 500k to 1.5m successful PUT/POST requests per day.
> About 5-10 or so end up failing as described above.
>
> Some config settings:
>
> option http-server-close
> option log-health-checks
> option log-separate-errors
> option redispatch
> retries 4
> timeout client 150000
> timeout connect 5000
> timeout queue 5000
> timeout server 3600000
>
> More details about the failing requests:
>
> Body size: about 30k-50k
> Tw = 0
> Tc = 0 in 90% and Tc=1 in 10% of the cases
> Tq is about 5-10 on average (with some outliers)
> actconn/feconn: 140 on average
> beconn: 30 on average
> Termination state: ---- for all requests
> None are http/2
>
> Things we tried which failed to fix the issue:
>
> Upgraded to haproxy 1.8.8 + git up to 80e179128cfd78d95cdebf7195fd21299e7931b6
>
> Disabled HTTP/2. None of the failing connections are HTTP/2, but was trying to
> test the theory that successful HTTP/2 connections somehow altered the
> internal state of the server and affected those failing requests.
>
> The only thing that fixed the issue was downgrading to haproxy 1.7.9
>
> Thank you,
> -Nick
>
>
>
> /usr/sbin/haproxy -vv
>
> HA-Proxy version 1.8.7 2018/04/07
> Copyright 2000-2018 Willy Tarreau <[email protected]>
> Build options :
> TARGET = linux2628
> CPU = generic
> CC = gcc
> CFLAGS = -m64 -march=x86-64 -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -fno-strict-overflow -Wno-unused-label
> OPTIONS = USE_SLZ=1 USE_REGPARM=1 USE_THREAD=1 USE_OPENSSL=1 USE_LUA=1 USE_SYSTEMD=1 USE_STATIC_PCRE2=1 USE_PCRE2_JIT=1 USE_TFO=1
> Default settings :
> maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
> Built with OpenSSL version : OpenSSL 1.1.0h 27 Mar 2018
> Running on OpenSSL version : OpenSSL 1.1.0h 27 Mar 2018
> OpenSSL library supports TLS extensions : yes
> OpenSSL library supports SNI : yes
> OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2
> Built with Lua version : Lua 5.3.4
> Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
> Encrypted password support via crypt(3): yes
> Built with multi-threading support.
> Built with PCRE2 version : 10.30 2017-08-14
> PCRE2 library supports JIT : yes
> Built with libslz for stateless compression.
> Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
> Built with network namespace support.
> Available polling systems :
> epoll : pref=300, test result OK
> poll : pref=200, test result OK
> select : pref=150, test result OK
> Total: 3 (3 usable), will use epoll.
> Available filters :
> [SPOE] spoe
> [COMP] compression
> [TRACE] trace
Sorry, only registered users may post in this forum.

Click here to login