Mildis
Diagnose a PD-- status
November 01, 2017 12:50PM
Hi,

I got a request ending in a PD status.
However, ‘show errors’ does not tell anything about that.

backend server returned 200, haproxy returned 200 to the client.
The entire request took 202ms and returned 15k of data : 3/0/0/199/202 200 15262

Is there a way to diagnose further the PD status ?
Maybe make haproxy log the reason why it ended in PD ?

Thanks,
mildis
Mildis
Re: Diagnose a PD-- status
November 02, 2017 05:20PM
I ran in debug mode and found the issue :

0000156e:ft-public.clireq[000c:000d]: PUT /api/products/5/versions/5/documentations HTTP/1.1
0000156e:ft-public.clihdr[000c:000d]: X-CSRF-TOKEN: de035ec0-58a3-4668-9e43-e4b36911d2ff
0000156e:ft-public.clihdr[000c:000d]: Content-Type: application/json
0000156e:ft-public.clihdr[000c:000d]: accept: application/json
0000156e:ft-public.clihdr[000c:000d]: Content-Length: 12605
0000156e:ft-public.clihdr[000c:000d]: Host: edc-ci.geomath.fr
0000156e:ft-public.clihdr[000c:000d]: Connection: Keep-Alive
0000156e:ft-public.clihdr[000c:000d]: User-Agent: Apache-HttpClient/4.5.3 (Java/1.8.0_131)
0000156e:ft-public.clihdr[000c:000d]: Cookie: CSRF-TOKEN=de035ec0-58a3-4668-9e43-e4b36911d2ff; JSESSIONID=8Xn2-NKJJuaMo-eI6c5PvSTwxNf5BLugv7e7rmes; remember-me=Y1luT2VGVXZnMHZvRWN6ZkluY3F6Zz09Olh4UmZmM3BpQVJxaVlZUmhWbkI1MlE9PQ
0000156e:ft-public.clihdr[000c:000d]: Accept-Encoding: gzip,deflate
0000156e:bck-traefik.srvrep[000c:000d]: HTTP/1.1 200 OK
0000156e:bck-traefik.srvhdr[000c:000d]: Cache-Control: no-cache, no-store, max-age=0, must-revalidate
0000156e:bck-traefik.srvhdr[000c:000d]: Content-Type: application/json;charset=UTF-8
0000156e:bck-traefik.srvhdr[000c:000d]: Date: Thu, 02 Nov 2017 13:47:18 GMT
0000156e:bck-traefik.srvhdr[000c:000d]: Expires: 0
0000156e:bck-traefik.srvhdr[000c:000d]: Pragma: no-cache
0000156e:bck-traefik.srvhdr[000c:000d]: Strict-Transport-Security: max-age=31536000 ; includeSubDomains
0000156e:bck-traefik.srvhdr[000c:000d]: X-Application-Context: application:prod,mysql:8081
0000156e:bck-traefik.srvhdr[000c:000d]: X-Content-Type-Options: nosniff
0000156e:bck-traefik.srvhdr[000c:000d]: X-Xss-Protection: 1; mode=block
0000156e:bck-traefik.srvhdr[000c:000d]: Transfer-Encoding: chunked
[WARNING] 305/144718 (21260) : HTTP compression failed: unexpected behavior of previous filters


Compression was enabled in the default section with :
compression algo gzip
compression type text/css text/html text/javascript application/javascript text/plain text/xml application/json

By commenting these two lines, it went OK.

However, I still can’t figure out what is causing this behavior : there are many other calls to the same URL.

Any clues ?

Regards,
mildis

> Le 1 nov. 2017 à 12:37, Mildis <[email protected]> a écrit :
>
> Hi,
>
> I got a request ending in a PD status.
> However, ‘show errors’ does not tell anything about that.
>
> backend server returned 200, haproxy returned 200 to the client.
> The entire request took 202ms and returned 15k of data : 3/0/0/199/202 200 15262
>
> Is there a way to diagnose further the PD status ?
> Maybe make haproxy log the reason why it ended in PD ?
>
> Thanks,
> mildis
Christopher Faulet
Re: Diagnose a PD-- status
November 06, 2017 10:20AM
Hi,


Le 02/11/2017 à 17:16, Mildis a écrit :
> [WARNING] 305/144718 (21260) : HTTP compression failed: unexpected behavior of previous filters

This warning is very suspicious. It should not happen. Could you share
your configuration and "haproxy -vv" output please ?


--
Christopher Faulet
Mildis
Re: Diagnose a PD-- status
November 06, 2017 05:00PM
Hi Christopher,

Configuration is attached.
The domain2backend map sends data mostly to bck-traefik.

$ haproxy -vv
HA-Proxy version 1.7.91~bpo7+1 2017/08/24
Copyright 2000-2017 Willy Tarreau <[email protected]>

Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -g -O2 -fPIE -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2
OPTIONS = USE_GETADDRINFO=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1

Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.7
Running on zlib version : 1.2.7
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with OpenSSL version : OpenSSL 1.0.1e 11 Feb 2013
Running on OpenSSL version : OpenSSL 1.0.1t 3 May 2016 (VERSIONS DIFFER!)
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.30 2012-02-04
Running on PCRE version : 8.30 2012-02-04
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with Lua version : Lua 5.3.1
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND

Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.

Available filters :
[COMP] compression
[TRACE] trace
[SPOE] spoe




Mildis


> Le 6 nov. 2017 à 10:10, Christopher Faulet <[email protected]> a écrit :
>
> Hi,
>
>
> Le 02/11/2017 à 17:16, Mildis a écrit :
>> [WARNING] 305/144718 (21260) : HTTP compression failed: unexpected behavior of previous filters
>
> This warning is very suspicious. It should not happen. Could you share your configuration and "haproxy -vv" output please ?
>
>
> --
> Christopher Faulet
Attachments:
open | download - haproxy.cfg (5.8 KB)
Christopher Faulet
Re: Diagnose a PD-- status
November 07, 2017 10:20AM
Le 06/11/2017 à 16:47, Mildis a écrit :
> Hi Christopher,
>
> Configuration is attached.
> The domain2backend map sends data mostly to bck-traefik.
>

Hi Mildis,

At first glance, I don't see anything strange here. The compression
filter is not supposed to fail this way. So there is definitely
something strange I don't understand for now.

Could you reproduce the bug on a single request on an HAProxy instance
without load, maybe using a dedicated frontend ? If yes, it could be
useful to enable the trace filter adding following lines in your
configuration, in the used frontend section:

filter trace name BEFORE
filter compression
filter trace name AFTER

Then, you can retry, starting HAProxy in foreground. The output is
verbose, but it give more information about the message filtering.

Thanks,
--
Christopher Faulet
Mildis
Re: Diagnose a PD-- status
November 08, 2017 05:40PM
Christopher,

We couldn’t had the error reproduced today : both dataset and client tool were updated.
However, I keep the configuration options at hand if I ever need to debug this issue further again.

Thanks for your time on this.
Regards,
Mildis


> Le 7 nov. 2017 à 10:14, Christopher Faulet <[email protected]> a écrit :
>
> Le 06/11/2017 à 16:47, Mildis a écrit :
>> Hi Christopher,
>> Configuration is attached.
>> The domain2backend map sends data mostly to bck-traefik.
>
> Hi Mildis,
>
> At first glance, I don't see anything strange here. The compression filter is not supposed to fail this way. So there is definitely something strange I don't understand for now.
>
> Could you reproduce the bug on a single request on an HAProxy instance without load, maybe using a dedicated frontend ? If yes, it could be useful to enable the trace filter adding following lines in your configuration, in the used frontend section:
>
> filter trace name BEFORE
> filter compression
> filter trace name AFTER
>
> Then, you can retry, starting HAProxy in foreground. The output is verbose, but it give more information about the message filtering.
>
> Thanks,
> --
> Christopher Faulet
Sorry, only registered users may post in this forum.

Click here to login