Welcome! Log In Create A New Profile

Advanced

Connections stuck in CLOSE_WAIT state with h2

Posted by Janusz Dziemidowicz 
Willy Tarreau
Re: Connections stuck in CLOSE_WAIT state with h2
July 19, 2018 11:20AM
Hi Milan, Janusz,

I suspect I managed to reliably trigger the issue you were facing and
found a good explanation for it. It is caused by unprocessed bytes at
the end of the H1 stream. I manage to reproduce it if I chain two layers
of haproxy with the last one sending stats. It does not happen with a
single layer, so the scheduling matters a lot (I think it's important
that the final CRLF is present in the same response packet as the final
chunk).

Could you please try the attached patch to see if you're on the same
issue ?

Thanks,
Willy
From 00610960a196f01b6e6b549e29eb1cf2426d253a Mon Sep 17 00:00:00 2001
From: Willy Tarreau <[email protected]>
Date: Thu, 19 Jul 2018 10:58:28 +0200
Subject: BUG/MEDIUM: h2: never leave pending data in the output buffer on
close

We currently don't process trailers on H2, but this has an impact : on
chunked HTTP/1 responses, we decide to emit the ES bit once we see the
0CRLF. From this point the stream switches to the CLOSED state, which
aborts processing of the remaining bytes. Thus the extra CRLF which ends
trailers is not processed and remains in the buffer. This prevents the
stream from being notified about end of transmission, which in turn keeps
the mux busy and prevents the connection from quitting.

The case of the trailers is not the root cause of this issue, though it
is what triggers it. The root cause is that upon error and/or close, once
we know we're not going to process any more data, we must absolutely flush
any remaining bytes from the output buffer, otherwise there is no way the
stream can quit. This is what this patch does.

It looks very likely related to the issues reported and debugged by
Janusz Dziemidowicz and Milan Petruzelka.

One way to reproduce it is to chain two proxies with the last one emitting
chunked data (typically using the stats page) :

global
stats socket /tmp/sock1 mode 666 level admin
stats timeout 1h
tune.ssl.default-dh-param 1024
tune.bufsize 16384

defaults
mode http
timeout connect 4s
timeout client 10s
timeout server 20s

listen px1
bind :4443 ssl crt rsa+dh2048.pem npn h2 alpn h2
server s1 127.0.0.1:4445

listen px2
bind :4444 ssl crt rsa+dh2048.pem npn h2 alpn h2
bind :4445
stats uri /

Then use curl to fetch the stats through px1 :

curl --http2 -k "https://127.0.0.1:4443/";

When curl is sent to the first one, "show sess" issued to the CLI will
show a remaining session during the client timeout. When curl is aimed at
port 4444 (px2), there is no such remaining session.

This fix needs to be backported to 1.8.
---
src/mux_h2.c | 15 +++++++++++++++
1 file changed, 15 insertions(+)

diff --git a/src/mux_h2.c b/src/mux_h2.c
index 0d0101e..47df89e 100644
--- a/src/mux_h2.c
+++ b/src/mux_h2.c
@@ -3302,6 +3302,14 @@ static int h2s_frt_make_resp_data(struct h2s *h2s, struct buffer *buf)
/* we may need to add END_STREAM */
/* FIXME: we should also detect shutdown(w) below, but how ? Maybe we
* could rely on the MSG_MORE flag as a hint for this ?
+ *
+ * FIXME: what we do here is not correct because we send end_stream
+ * before knowing if we'll have to send a HEADERS frame for the
+ * trailers. More importantly we're not consuming the trailing CRLF
+ * after the end of trailers, so it will be left to the caller to
+ * eat it. The right way to do it would be to measure trailers here
+ * and to send ES only if there are no trailers.
+ *
*/
if (((h1m->flags & H1_MF_CLEN) && !(h1m->curr_len - size)) ||
!h1m->curr_len || h1m->state >= HTTP_MSG_DONE)
@@ -3404,6 +3412,13 @@ static int h2_snd_buf(struct conn_stream *cs, struct buffer *buf, int flags)
}
}

+ if (h2s->st >= H2_SS_ERROR) {
+ /* trim any possibly pending data after we close (extra CR-LF,
+ * unprocessed trailers, abnormal extra data, ...)
+ */
+ bo_del(buf, buf->o);
+ }
+
/* RST are sent similarly to frame acks */
if (h2s->st == H2_SS_ERROR || h2s->flags & H2_SF_RST_RCVD) {
cs->flags |= CS_FL_ERROR;
--
1.7.12.1
Janusz Dziemidowicz
Re: Connections stuck in CLOSE_WAIT state with h2
July 20, 2018 01:40PM
czw., 19 lip 2018 o 11:14 Willy Tarreau <[email protected]> napisał(a):
>
> Hi Milan, Janusz,
>
> I suspect I managed to reliably trigger the issue you were facing and
> found a good explanation for it. It is caused by unprocessed bytes at
> the end of the H1 stream. I manage to reproduce it if I chain two layers
> of haproxy with the last one sending stats. It does not happen with a
> single layer, so the scheduling matters a lot (I think it's important
> that the final CRLF is present in the same response packet as the final
> chunk).
>
> Could you please try the attached patch to see if you're on the same
> issue ?

I've been running 1.8.12 with this patch for an hour. It seems that it
helped somewhat, but not entirely. After an hour I still see about 10
CLOSE_WAIT sockets. The number seems to grow a lot slower, but still
grows (and some of them have been sitting in CLOSE_WAIT for over 30
minutes).
Since I'm also affected by SPDY_PROTOCOL_ERROR I mentioned earlier I
must disable h2 now.

--
Janusz Dziemidowicz
Willy Tarreau
Re: Connections stuck in CLOSE_WAIT state with h2
July 20, 2018 01:50PM
On Fri, Jul 20, 2018 at 01:30:00PM +0200, Janusz Dziemidowicz wrote:
> I've been running 1.8.12 with this patch for an hour. It seems that it
> helped somewhat, but not entirely. After an hour I still see about 10
> CLOSE_WAIT sockets. The number seems to grow a lot slower, but still
> grows (and some of them have been sitting in CLOSE_WAIT for over 30
> minutes).

Thank you Janusz for testing. I've also merged the other patch that could
lead to some CLOSE_WAIT that we tested a few months ago. It was different
but maybe you were suffering from the two causes.

> Since I'm also affected by SPDY_PROTOCOL_ERROR I mentioned earlier I
> must disable h2 now.

OK. If you happen to figure that it's caused by haproxy, do not hesitate
to provide any information so that we can debug this one further.

Thanks!
Willy
Milan Petruželka
Re: Connections stuck in CLOSE_WAIT state with h2
July 20, 2018 02:40PM
On Fri, 20 Jul 2018 at 13:37, Willy Tarreau <[email protected]> wrote:

> Thank you Janusz for testing. I've also merged the other patch that could
> lead to some CLOSE_WAIT that we tested a few months ago. It was different
> but maybe you were suffering from the two causes.
>

I've applied both patches to vanilla haproxy 1.8.12. I'll leave it running
and report back.

BUG/MEDIUM: h2: never leave pending data in the output buffer on close
BUG/MEDIUM: h2: make sure the last stream closes the connection after a
timeout

Milan
Milan Petruželka
Re: Connections stuck in CLOSE_WAIT state with h2
July 23, 2018 08:50AM
On Fri, 20 Jul 2018 at 14:36, Milan Petruželka <[email protected]> wrote:

>
> I've applied both patches to vanilla haproxy 1.8.12. I'll leave it running
> and report back.
>

>

Hi,

After weekend CLOSE_WAIT connections are still there. What
does cflg=0x80203300 in "show fd" mean? FDs with cflg=0x80203300 are either
CLOSE_WAIT or "sock - protocol: TCP" - see FDs 14, 15, 16, 18, 19 and 25 in
following dumps. And - sockets in lsof state "sock - protocol: TCP" can't
be found in netstat.

SHOW FD 3300
14 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x23d0340
iocb=0x4d4c90(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300
fe=fe-http mux=H2 mux_ctx=0x2494cc0
15 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x245c6f0
iocb=0x4d4c90(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300
fe=fe-http mux=H2 mux_ctx=0x23c1db0
16 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x25598e0
iocb=0x4d4c90(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300
fe=fe-http mux=H2 mux_ctx=0x23d0900
18 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x23940a0
iocb=0x4d4c90(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300
fe=fe-http mux=H2 mux_ctx=0x242a030
19 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x24a8b90
iocb=0x4d4c90(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300
fe=fe-http mux=H2 mux_ctx=0x24820b0
25 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x2457a10
iocb=0x4d4c90(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300
fe=fe-http mux=H2 mux_ctx=0x2394660

LSOF
haproxy 31313 haproxy 0u CHR 136,1 0t0 4
/dev/pts/1
haproxy 31313 haproxy 1w FIFO 0,10 0t0 2004495
pipe
haproxy 31313 haproxy 2w FIFO 0,10 0t0 2004495
pipe
haproxy 31313 haproxy 3u a_inode 0,11 0 7017
[eventpoll]
haproxy 31313 haproxy 4u unix 0xffff88042aa3b400 0t0 2002869
/www/server/haproxy/cmd.sock.31313.tmp type=STREAM
haproxy 31313 haproxy 5u IPv4 2002872 0t0 TCP
some.ip:http (LISTEN)
haproxy 31313 haproxy 6u IPv4 2002873 0t0 TCP
some.ip:https (LISTEN)
haproxy 31313 haproxy 7u IPv4 2002874 0t0 TCP
*:http-alt (LISTEN)
haproxy 31313 haproxy 8u IPv4 2002875 0t0 TCP
*:8443 (LISTEN)
haproxy 31313 haproxy 9r FIFO 0,10 0t0 2002876
pipe
haproxy 31313 haproxy 10w FIFO 0,10 0t0 2002876
pipe
haproxy 31313 haproxy 11u IPv4 6560416 0t0 TCP
some.ip:https->some.ip:49375 (ESTABLISHED)
haproxy 31313 haproxy 12u IPv4 2002883 0t0 UDP
*:52068
haproxy 31313 haproxy 13u IPv4 6656750 0t0 TCP
some.ip:https->some.ip:50544 (ESTABLISHED)
haproxy 31313 haproxy 14u IPv4 4951212 0t0 TCP
some.ip:https->some.ip:55554 (CLOSE_WAIT)
haproxy 31313 haproxy 15u sock 0,8 0t0 4111815
protocol: TCP
haproxy 31313 haproxy 16u sock 0,8 0t0 6236118
protocol: TCP
haproxy 31313 haproxy 17u IPv4 6657419 0t0 TCP
some.ip:https->some.ip:64934 (ESTABLISHED)
haproxy 31313 haproxy 18u sock 0,8 0t0 2653890
protocol: TCP
haproxy 31313 haproxy 19u IPv4 5699053 0t0 TCP
some.ip:https->some.ip:59601 (CLOSE_WAIT)
haproxy 31313 haproxy 20u IPv4 6656756 0t0 TCP
some.ip:https->some.ip:29233 (ESTABLISHED)
haproxy 31313 haproxy 21u IPv4 6656760 0t0 TCP
some.ip:https->some.ip:59058 (ESTABLISHED)
haproxy 31313 haproxy 22u IPv4 6654620 0t0 TCP
some.ip:https->some.ip:49306 (ESTABLISHED)
haproxy 31313 haproxy 23u IPv4 6656769 0t0 TCP
some.ip:https->some.ip:17513 (ESTABLISHED)
haproxy 31313 haproxy 25u IPv4 5873818 0t0 TCP
some.ip:https->some.ip:58413 (CLOSE_WAIT)
haproxy 31313 haproxy 26u unix 0xffff8802f9240000 0t0 6656772
type=STREAM
haproxy 31313 haproxy 27u IPv4 6656639 0t0 TCP
some.ip:https->some.ip:2926 (ESTABLISHED)

SHOW FD
4 : st=0x05(R:PrA W:pra) ev=0x01(heopI) [nlc] cache=0 owner=0x232ac80
iocb=0x4c0be0(listener_accept) tmask=0xffffffffffffffff
umask=0xfffffffffffffffe l.st=RDY fe=GLOBAL
5 : st=0x05(R:PrA W:pra) ev=0x01(heopI) [nlc] cache=0 owner=0x232ce80
iocb=0x4c0be0(listener_accept) tmask=0xffffffffffffffff
umask=0xfffffffffffffffe l.st=RDY fe=fe-http
6 : st=0x05(R:PrA W:pra) ev=0x01(heopI) [nlc] cache=0 owner=0x232d390
iocb=0x4c0be0(listener_accept) tmask=0xffffffffffffffff
umask=0xfffffffffffffffe l.st=RDY fe=fe-http
7 : st=0x05(R:PrA W:pra) ev=0x01(heopI) [nlc] cache=0 owner=0x234cb00
iocb=0x4c0be0(listener_accept) tmask=0xffffffffffffffff
umask=0xfffffffffffffffe l.st=RDY fe=fe-service
8 : st=0x05(R:PrA W:pra) ev=0x00(heopi) [nlc] cache=0 owner=0x234d010
iocb=0x4c0be0(listener_accept) tmask=0xffffffffffffffff
umask=0xfffffffffffffffe l.st=RDY fe=fe-service
9 : st=0x05(R:PrA W:pra) ev=0x00(heopi) [nlc] cache=0 owner=0x4e9260
iocb=0x4e9260(thread_sync_io_handler) tmask=0xffffffffffffffff
umask=0xfffffffffffffffe
11 : st=0x25(R:PrA W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x25be850
iocb=0x4d4c90(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203306
fe=fe-http mux=H2 mux_ctx=0x2480820
13 : st=0x25(R:PrA W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x24283d0
iocb=0x4d4c90(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203306
fe=fe-http mux=H2 mux_ctx=0x25c47e0
14 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x23d0340
iocb=0x4d4c90(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300
fe=fe-http mux=H2 mux_ctx=0x2494cc0
15 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x245c6f0
iocb=0x4d4c90(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300
fe=fe-http mux=H2 mux_ctx=0x23c1db0
16 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x25598e0
iocb=0x4d4c90(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300
fe=fe-http mux=H2 mux_ctx=0x23d0900
17 : st=0x25(R:PrA W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x25c35f0
iocb=0x4d4c90(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203306
fe=fe-http mux=H2 mux_ctx=0x2585d80
18 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x23940a0
iocb=0x4d4c90(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300
fe=fe-http mux=H2 mux_ctx=0x242a030
19 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x24a8b90
iocb=0x4d4c90(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300
fe=fe-http mux=H2 mux_ctx=0x24820b0
20 : st=0x25(R:PrA W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x25c3460
iocb=0x4d4c90(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203306
fe=fe-http mux=H2 mux_ctx=0x23b76f0
21 : st=0x25(R:PrA W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x23bc6e0
iocb=0x4d4c90(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203306
fe=fe-http mux=H2 mux_ctx=0x23cd7b0
22 : st=0x25(R:PrA W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x2570b30
iocb=0x4d4c90(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203306
fe=fe-http mux=H2 mux_ctx=0x23be980
23 : st=0x25(R:PrA W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x23b78b0
iocb=0x4d4c90(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203306
fe=fe-http mux=PASS mux_ctx=0x245d0e0
24 : st=0x22(R:pRa W:pRa) ev=0x00(heopi) [Nlc] cache=0 owner=0x25c3910
iocb=0x4d4c90(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x00241300
fe=GLOBAL mux=PASS mux_ctx=0x245bf00
25 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x2457a10
iocb=0x4d4c90(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300
fe=fe-http mux=H2 mux_ctx=0x2394660
26 : st=0x25(R:PrA W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x257a510
iocb=0x4d4c90(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x00202306
sv=nginx01/be-nginx01 mux=PASS mux_ctx=0x23b4110
27 : st=0x25(R:PrA W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x23be2d0
iocb=0x4d4c90(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203306
fe=fe-http mux=H2 mux_ctx=0x23c5d50

Milan
Milan Petruželka
Re: Connections stuck in CLOSE_WAIT state with h2
July 23, 2018 01:40PM
Hi,

I've compiled latest haproxy 1.8.12 from Git repo (HAProxy version
1.8.12-5e100b-15, released 2018/07/20) with latest h2 patches and extended
h2 debug info. And after some time I caught one CLOSE_WAIT connection. Here
is extended show fd debug:
25 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x24f0a70
iocb=0x4d34c0(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300
fe=fe-http mux=H2 mux_ctx=0x258a880 st0=7 flg=0x00001000 nbst=8 nbcs=0
fctl_cnt=0 send_cnt=8 tree_cnt=8 orph_cnt=8 dbuf=0/0 mbuf=0/16384

LSOF CLOSE_WAIT
haproxy 26364 haproxy 25u IPv4 7140390 0t0 TCP
ip:https->ip:50041 (CLOSE_WAIT)

Milan
Willy Tarreau
Re: Connections stuck in CLOSE_WAIT state with h2
July 24, 2018 08:10AM
Hi Milan,

On Mon, Jul 23, 2018 at 08:41:03AM +0200, Milan Petruzelka wrote:
> After weekend CLOSE_WAIT connections are still there.

Ah bad :-(

> What
> does cflg=0x80203300 in "show fd" mean?

These are the connection flags. You can figure them with contrib/debug/flags :

$ ./flags 0x80203300 | grep ^conn
conn->flags = CO_FL_XPRT_TRACKED | CO_FL_CONNECTED | CO_FL_ADDR_TO_SET | CO_FL_ADDR_FROM_SET | CO_FL_XPRT_READY | CO_FL_CTRL_READY

So basically it says that everything is configured on the connection and
that there is no request for polling (CO_FL_{CURR,SOCK,XPRT}_{WR,RD}_ENA).

> FDs with cflg=0x80203300 are either
> CLOSE_WAIT or "sock - protocol: TCP" - see FDs 14, 15, 16, 18, 19 and 25 in
> following dumps. And - sockets in lsof state "sock - protocol: TCP" can't
> be found in netstat.

That totally makes sense. If the connection is not monitored at all (but
why, this is the question), and has no timeout, definitely it will wander
forever.

Do you *think* that you got less CLOSE_WAITs or that the latest fixes
didn't change anything ? I suspect that for some reason you might be
hit by several bugs, which is what has complicated the diagnostic, but
that's just pure guess.

>
> SHOW FD 3300
> 14 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x23d0340
> iocb=0x4d4c90(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300
> fe=fe-http mux=H2 mux_ctx=0x2494cc0
(...)

If you run this with the latest 1.8 (not just the two patches above), some
extra debug information is provided in show fd :

$ echo show fd | socat - /tmp/sock1 | grep -i mux=H2
19 : st=0x25(R:PrA W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x7ffff7fd8e80 iocb=0x58cb44(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80201306 fe=httpgw mux=H2 mux_ctx=0x7ffff7f8ff10 st0=2 flg=0x00000000 nbst=1 nbcs=1 fctl_cnt=0 send_cnt=0 tree_cnt=1 orph_cnt=0 dbuf=0/0 mbuf=0/0

In particular, st0, the mux flags, the number of streams and the buffer
states will give important information about what state the connection
is in (and whether it still has streams attached or not).

Oh I'm just seeing you already did that in the next e-mail. Thank you :-)

So we have this :

25 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x24f0a70
iocb=0x4d34c0(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300
fe=fe-http mux=H2 mux_ctx=0x258a880 st0=7 flg=0x00001000 nbst=8 nbcs=0
fctl_cnt=0 send_cnt=8 tree_cnt=8 orph_cnt=8 dbuf=0/0 mbuf=0/16384

- st0=7 => H2_CS_ERROR2 : an error was sent, either it succeeded or
could not be sent and had to be aborted nonetheless ;

- flg=1000 => H2_CF_GOAWAY_SENT : the GOAWAY frame was sent to the mux
buffer.

- nbst=8 => 8 streams still attached

- nbcs=0 => 0 conn_streams found (application layer detached or not
attached yet)

- send_cnt=8 => 8 streams still in the send_list, waiting for the mux
to pick their contentx.

- tree_cnt=8 => 8 streams known in the tree (hence they are still valid
from the H2 protocol perspective)

- orph_cnt=8 => 8 streams are orphaned : these streams have quit at the
application layer (very likely a timeout).

- mbuf=0/16384 : the mux buffer is empty but allocated. It's not very
common.

At this point what it indicates is that :
- 8 streams were active on this connection and a response was sent (at
least partially) and probably waited for the mux buffer to be empty
due to data from other previous streams. I'm realising it would be
nice to also report the highest stream index to get an idea of the
number of past streams on the connection.

- an error happened (protocol error, network issue, etc, no more info
at the moment) and caused haproxy to emit a GOAWAY frame. While doing
so, the pending streams in the send_list were not destroyed.

- then for an unknown reason the situation doesn't move anymore. I'm
realising that one case I figured in the past with an error possibly
blocking the connection at least partially covers one point here, it
causes the mux buffer to remain allocated, so this patch would have
caused it to be released, but it's still incomplete.

Now I have some elements to dig through, I'll try to mentally reproduce
the complex sequence of a blocked response with a GOAWAY being sent at
the same time to see what happens.

Thank you very much for all these information!
Willy
Milan Petruželka
Re: Connections stuck in CLOSE_WAIT state with h2
July 24, 2018 12:30PM
Hi Willy,

Do you *think* that you got less CLOSE_WAITs or that the latest fixes
> didn't change anything ? I suspect that for some reason you might be
> hit by several bugs, which is what has complicated the diagnostic, but
> that's just pure guess.
>
>
I'm not sure. I left patched haproxy running over last weekend. I have a
slight feeling that there was less hanging connections than over last week,
but it could be because of lower weekend traffic. Now i'm running latest
GIT version (1.8.12-5e100b-15) and i'll compare the speed of blocked
connections increase against vannila 1.8.12 from last week.

Oh I'm just seeing you already did that in the next e-mail. Thank you :-)
>

I'll send more extended "show fd" dumps as soon as i catch some more.
Please let me know If you add more h2 state info into 1.8 git version and
i'll run it in production to get more info.

So we have this :
>
> 25 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0
> owner=0x24f0a70
> iocb=0x4d34c0(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300
>
> fe=fe-http mux=H2 mux_ctx=0x258a880 st0=7 flg=0x00001000 nbst=8 nbcs=0
>
> fctl_cnt=0 send_cnt=8 tree_cnt=8 orph_cnt=8 dbuf=0/0 mbuf=0/16384
>
>
> - st0=7 => H2_CS_ERROR2 : an error was sent, either it succeeded or
> could not be sent and had to be aborted nonetheless ;
>
> - flg=1000 => H2_CF_GOAWAY_SENT : the GOAWAY frame was sent to the mux
> buffer.
>
> - nbst=8 => 8 streams still attached
>
> - nbcs=0 => 0 conn_streams found (application layer detached or not
> attached yet)
>
> - send_cnt=8 => 8 streams still in the send_list, waiting for the mux
> to pick their contentx.
>
> - tree_cnt=8 => 8 streams known in the tree (hence they are still valid
> from the H2 protocol perspective)
>
> - orph_cnt=8 => 8 streams are orphaned : these streams have quit at the
> application layer (very likely a timeout).
>
> - mbuf=0/16384 : the mux buffer is empty but allocated. It's not very
> common.
>
> At this point what it indicates is that :
> - 8 streams were active on this connection and a response was sent (at
> least partially) and probably waited for the mux buffer to be empty
> due to data from other previous streams. I'm realising it would be
> nice to also report the highest stream index to get an idea of the
> number of past streams on the connection.
>
> - an error happened (protocol error, network issue, etc, no more info
> at the moment) and caused haproxy to emit a GOAWAY frame. While doing
> so, the pending streams in the send_list were not destroyed.
>
> - then for an unknown reason the situation doesn't move anymore. I'm
> realising that one case I figured in the past with an error possibly
> blocking the connection at least partially covers one point here, it
> causes the mux buffer to remain allocated, so this patch would have
> caused it to be released, but it's still incomplete.
>
> Now I have some elements to dig through, I'll try to mentally reproduce
> the complex sequence of a blocked response with a GOAWAY being sent at
> the same time to see what happens.
>
>
Thanks a lot for a detailed description.
Milan
Willy Tarreau
Re: Connections stuck in CLOSE_WAIT state with h2
July 24, 2018 02:50PM
Hi Milan,

On Tue, Jul 24, 2018 at 12:23:37PM +0200, Milan Petruzelka wrote:
> Hi Willy,
>
> Do you *think* that you got less CLOSE_WAITs or that the latest fixes
> > didn't change anything ? I suspect that for some reason you might be
> > hit by several bugs, which is what has complicated the diagnostic, but
> > that's just pure guess.
> >
> >
> I'm not sure. I left patched haproxy running over last weekend. I have a
> slight feeling that there was less hanging connections than over last week,
> but it could be because of lower weekend traffic.

OK, it's not that obvious at least.

> Now i'm running latest
> GIT version (1.8.12-5e100b-15) and i'll compare the speed of blocked
> connections increase against vannila 1.8.12 from last week.
>
> I'll send more extended "show fd" dumps as soon as i catch some more.
> Please let me know If you add more h2 state info into 1.8 git version and
> i'll run it in production to get more info.

So I'm having one update to emit the missing info on "show fd" (patch merged
and pushed already, that I'm attaching here if it's easier for you), and
two other ones which fix a situation which can definitely cause this to
happen, but which I still fail to reproduce using the various tools I
have (curl, h2c, nghttp, ...). The case is the following and derives from
your previous capture :

- multiple streams fight for the mux and are queued in the send_list
- at this point the mux has to emit a GOAWAY for a reason that's still
to be figured (probably it received a bad message but we could guess
anything)
- the streams are woken up, notified about the error
- h2_detach() is called for each of them
- they are detached from the h2 stream (struct h2s technically
speaking, which is the internal representation of the h2 stream
state)
- since the streams are marked as blocked for some room, they are orphaned
and nothing more is done on them.
- at this point, any activity on the connection goes through h2_wake()
which sees the conneciton in ERROR2 state, tries again to release the
streams, cannot, and stops polling.

=> from this point, no more events can be received on the connection, and
the streams remain orphaned forever. This case was partially addressed
by the patch I sent a while ago but I wasn't completely sure about the
sequence that could lead to this. I'm attaching this patch (0001-WIP-*).

However this patch alone (0001-WIP-*) cannot prevent the streams from being
orphaned (6th step above) so while it's needed, it's not enough. The third
one is required for this (h2-error.diff).

And I *think* (and hope) that with these 2 patches on top of latest 1.8
we're OK now. What I would appreciate quite a lot if you're willing to
let me abuse your time is to either git pull or apply
0001-MINOR-h2-add-the-error-code-and-the-max-last-stream-.patch on top
of your up-to-date branch, then apply
0001-WIP-h2-try-to-address-possible-causes-for-the-close_.patch then
apply h2-error.diff and test again. The last two will apply with an
offset but that's not a problem.

If you still see some close_waits (I really hope not), it means they're
caused by something totally different and then "show fd" will help better
than previously thanks to the extra info in the first patch.

I think we're about to nail it down, to be honest ;-)

Thanks!
Willy
From 12a4b5c69d3eb0cc01bf92df215c7cc8d70ee8bc Mon Sep 17 00:00:00 2001
From: Willy Tarreau <[email protected]>
Date: Tue, 24 Jul 2018 14:12:42 +0200
Subject: MINOR: h2: add the error code and the max/last stream IDs to "show
fd"

This is intented to help debugging H2 in field.

(cherry picked from commit 616ac81dec5759990ee600047d8ad900f6eba6e8)
[wt: adapted context a little bit]
Signed-off-by: Willy Tarreau <[email protected]>
---
src/mux_h2.c | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/src/mux_h2.c b/src/mux_h2.c
index 7f14de4..95effb9 100644
--- a/src/mux_h2.c
+++ b/src/mux_h2.c
@@ -3515,8 +3515,11 @@ static void h2_show_fd(struct chunk *msg, struct connection *conn)
node = eb32_next(node);
}

- chunk_appendf(msg, " st0=%d flg=0x%08x nbst=%u nbcs=%u fctl_cnt=%d send_cnt=%d tree_cnt=%d orph_cnt=%d dbuf=%u/%u mbuf=%u/%u",
- h2c->st0, h2c->flags, h2c->nb_streams, h2c->nb_cs, fctl_cnt, send_cnt, tree_cnt, orph_cnt, h2c->dbuf->i, h2c->dbuf->size, h2c->mbuf->o, h2c->mbuf->size);
+ chunk_appendf(msg, " st0=%d err=%d maxid=%d lastid=%d flg=0x%08x nbst=%u nbcs=%u"
+ " fctl_cnt=%d send_cnt=%d tree_cnt=%d orph_cnt=%d dbuf=%u/%u mbuf=%u/%u",
+ h2c->st0, h2c->errcode, h2c->max_id, h2c->last_sid, h2c->flags,
+ h2c->nb_streams, h2c->nb_cs, fctl_cnt, send_cnt, tree_cnt, orph_cnt,
+ h2c->dbuf->i, h2c->dbuf->size, h2c->mbuf->o, h2c->mbuf->size);
}

/*******************************************************/
--
1.7.12.1

From f055296f7598ba84b08e78f8309ffd7fa0c9522b Mon Sep 17 00:00:00 2001
From: Willy Tarreau <[email protected]>
Date: Wed, 13 Jun 2018 09:19:29 +0200
Subject: WIP: h2: try to address possible causes for the close_wait issues

It is uncertain whether certain errors could prevent pending outgoing
data from being emitted, and from releasing attached streams. Indeed,
for h2_release() to be called, the mux buf must be empty or an error
must have been met. A clean shutdown will not constitute an error and
it's likely that refraining from sending may prevent the buffer from
flushing. Thus maybe we can end up with data forever in the buffer.

The timeout task should normally take care of this though. It's worth
noting that if there's no more stream and the output buffer is empty
on wake(), the task's timeout is eternity.

This fix should be backported to 1.8.
---
src/mux_h2.c | 9 +--------
1 file changed, 1 insertion(+), 8 deletions(-)

diff --git a/src/mux_h2.c b/src/mux_h2.c
index bc33ce2..2f2384d 100644
--- a/src/mux_h2.c
+++ b/src/mux_h2.c
@@ -2271,14 +2271,6 @@ static int h2_wake(struct connection *conn)
h2_release(conn);
return -1;
}
- else {
- /* some streams still there, we need to signal them all and
- * wait for their departure.
- */
- __conn_xprt_stop_recv(conn);
- __conn_xprt_stop_send(conn);
- return 0;
- }
}

if (!h2c->dbuf->i)
@@ -2294,6 +2286,7 @@ static int h2_wake(struct connection *conn)

/* adjust output polling */
if (!(conn->flags & CO_FL_SOCK_WR_SH) &&
+ h2c->st0 != H2_CS_ERROR2 && !(h2c->flags & H2_CF_GOAWAY_FAILED) &&
(h2c->st0 == H2_CS_ERROR ||
h2c->mbuf->o ||
(h2c->mws > 0 && !LIST_ISEMPTY(&h2c->fctl_list)) ||
--
1.7.12.1

diff --git a/src/mux_h2.c b/src/mux_h2.c
index 7f14de4..a4db89d 100644
--- a/src/mux_h2.c
+++ b/src/mux_h2.c
@@ -2512,6 +2512,7 @@ static void h2_detach(struct conn_stream *cs)
* an ES or RST frame), so orphan it in this case.
*/
if (!(cs->conn->flags & CO_FL_ERROR) &&
+ (h2c->st0 < H2_CS_ERROR) &&
(h2s->flags & (H2_SF_BLK_MBUSY | H2_SF_BLK_MROOM | H2_SF_BLK_MFCTL)))
return;
Milan Petruželka
Re: Connections stuck in CLOSE_WAIT state with h2
July 25, 2018 10:20AM
Hi Willy,

On Tue, 24 Jul 2018 at 14:37, Willy Tarreau <[email protected]> wrote:

> So I'm having one update to emit the missing info on "show fd" (patch
> merged
> and pushed already, that I'm attaching here if it's easier for you)


I've compiled version 1.8.12-12a4b5-16 with from Git and let it run
overnight. Now I have 4 blocked connections. Maxid is always equal to
lastid:

14 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x17c51f0
iocb=0x4d34d0(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300
fe=fe-http mux=H2 mux_ctx=0x17a5240 st0=7 err=5 maxid=133 lastid=133
flg=0x00001000 nbst=6 nbcs=0 fctl_cnt=0 send_cnt=6 tree_cnt=6 orph_cnt=6
dbuf=0/0 mbuf=0/16384
19 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x178fab0
iocb=0x4d34d0(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300
fe=fe-http mux=H2 mux_ctx=0x1798be0 st0=7 err=5 maxid=131 lastid=131
flg=0x00001000 nbst=12 nbcs=0 fctl_cnt=0 send_cnt=12 tree_cnt=12
orph_cnt=12 dbuf=0/0 mbuf=0/16384
20 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x17bac40
iocb=0x4d34d0(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300
fe=fe-http mux=H2 mux_ctx=0x19e6220 st0=7 err=5 maxid=27 lastid=27
flg=0x00001000 nbst=1 nbcs=0 fctl_cnt=0 send_cnt=1 tree_cnt=1 orph_cnt=1
dbuf=0/0 mbuf=0/16384
22 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x18d2410
iocb=0x4d34d0(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300
fe=fe-http mux=H2 mux_ctx=0x1a0dab0 st0=7 err=5 maxid=107 lastid=107
flg=0x00001000 nbst=5 nbcs=0 fctl_cnt=0 send_cnt=5 tree_cnt=5 orph_cnt=5
dbuf=0/0 mbuf=0/16384



> And I *think* (and hope) that with these 2 patches on top of latest 1.8
>
we're OK now. What I would appreciate quite a lot if you're willing to
> let me abuse your time is to either git pull or apply
> 0001-MINOR-h2-add-the-error-code-and-the-max-last-stream-.patch on top
> of your up-to-date branch, then apply
> 0001-WIP-h2-try-to-address-possible-causes-for-the-close_.patch then
> apply h2-error.diff and test again.
>

Now I'll add both patches (WIP-h2 and h2-error.diff) and give it a try in
production.

I think we're about to nail it down, to be honest ;-)
>

That would be great :-)
Milan
Willy Tarreau
Re: Connections stuck in CLOSE_WAIT state with h2
July 25, 2018 10:30AM
Hi Milan,

On Wed, Jul 25, 2018 at 10:15:50AM +0200, Milan Petruzelka wrote:
> Now I'll add both patches (WIP-h2 and h2-error.diff) and give it a try in
> production.

Thank you. At first I thought you still had the errors with them applied
and was despeared, now I understand there's still hope :-)

Cheers,
Willy
Olivier Doucet
Re: Connections stuck in CLOSE_WAIT state with h2
July 25, 2018 06:00PM
Hello,

2018-07-25 10:20 GMT+02:00 Willy Tarreau <[email protected]>:

> Hi Milan,
>
> On Wed, Jul 25, 2018 at 10:15:50AM +0200, Milan Petruzelka wrote:
> > Now I'll add both patches (WIP-h2 and h2-error.diff) and give it a try in
> > production.
>
> Thank you. At first I thought you still had the errors with them applied
> and was despeared, now I understand there's still hope :-)
>
> Cheers,
> Willy
>


It seems I have the same issue as Milan :
We activated HTTP/2 on production a few weeks ago, and on some customers
(not all !) we can observe a very strange behaviour : it seems some
frontend sessions are not closed, leading to 'slim' reached if HAProxy runs
for several days without being reloaded.

What can be observed:
- on a specific frontend, scur keeps growing over and over.
- reloading haproxy (with -sf parameter) clears sessions connected
- it happens only on specific frontends, but I failed to cross info (we
have issue on 2 frontends, both have reasonable trafic, but some frontends
with much more do not have the issue).
- disabling HTTP/2 solve the problem for these specific frontends, so this
is definitely HTTP/2 related

I ran several "show fd" and "show sess" on haproxy process, and filter it
with the frontend name. Both shows different number of lines, and the
difference is growing over time.

I am running HAProxy 1.8.12 with OpenSSL 1.1.0h compiled statically.

Here is one fd with the issue:
58 : st=0x25(R:PrA W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x46ceeb0
iocb=0x530c20(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80201300
fe=xxxx:443 mux=H2 mux_ctx=0x45e27e0

With "flag" debug binary I debugged cflg and all "lost" sessions are in
this state :
conn->flags = CO_FL_XPRT_TRACKED | CO_FL_CONNECTED | CO_FL_ADDR_FROM_SET |
CO_FL_XPRT_READY | CO_FL_CTRL_READY

This issue is very close to Milan bug, that's why I posted as a reply. If
I'm wrong, I'll split it in another thread.

Willy, are your patches "production-safe" (meaning it is reasonable enough
to run it a few hours in production) ? Can it be applied on 1.8.12 release,
or do I need to download latest trunk ?

I can reproduce the issue quickly (~ 2 hours to be sure) on my side to help
!

Olivier
Willy Tarreau
Re: Connections stuck in CLOSE_WAIT state with h2
July 25, 2018 06:50PM
Hi Olivier,

On Wed, Jul 25, 2018 at 05:51:46PM +0200, Olivier Doucet wrote:
> It seems I have the same issue as Milan :
> We activated HTTP/2 on production a few weeks ago, and on some customers
> (not all !) we can observe a very strange behaviour : it seems some
> frontend sessions are not closed, leading to 'slim' reached if HAProxy runs
> for several days without being reloaded.
(...)
> With "flag" debug binary I debugged cflg and all "lost" sessions are in
> this state :
> conn->flags = CO_FL_XPRT_TRACKED | CO_FL_CONNECTED | CO_FL_ADDR_FROM_SET |
> CO_FL_XPRT_READY | CO_FL_CTRL_READY
>
> This issue is very close to Milan bug, that's why I posted as a reply. If
> I'm wrong, I'll split it in another thread.

It's definitely the same as Milan's from my point of view. Many thanks
for reporting it as well.

> Willy, are your patches "production-safe" (meaning it is reasonable enough
> to run it a few hours in production) ? Can it be applied on 1.8.12 release,
> or do I need to download latest trunk ?

Yes they are safe enough so that I expect them to be backported for
1.8.13. I'm even reasonably confident that they should fix the problem.
You need to apply them on the latest 1.8 git maintenance version (in
which case you don't need the first one which is already merged).

> I can reproduce the issue quickly (~ 2 hours to be sure) on my side to help
> !

That could be great!

Thanks!
Willy
Milan Petruželka
Re: Connections stuck in CLOSE_WAIT state with h2
July 26, 2018 10:40AM
On Wed, 25 Jul 2018 at 18:39, Willy Tarreau <[email protected]> wrote:

> > This issue is very close to Milan bug, that's why I posted as a reply. If
> > I'm wrong, I'll split it in another thread.
>
> It's definitely the same as Milan's from my point of view. Many thanks
> for reporting it as well.
>
>
Hi Willy,

my haproxy with last two patches has no blocked connection since it started
24 hours ago. Before last patches it always had at least one. I think you
finally nailed it. To be on the safe side, I would wait till Monday, before
we start to celebrate.

Milan
Willy Tarreau
Re: Connections stuck in CLOSE_WAIT state with h2
July 26, 2018 10:40AM
Hi Milan!

On Thu, Jul 26, 2018 at 10:31:24AM +0200, Milan Petruzelka wrote:
> my haproxy with last two patches has no blocked connection since it started
> 24 hours ago. Before last patches it always had at least one. I think you
> finally nailed it. To be on the safe side, I would wait till Monday, before
> we start to celebrate.

Ah that's an awesome news! I'm fine with waiting before Monday to open
the Champagne though :-)

Thanks!
Willy
Olivier Doucet
Re: Connections stuck in CLOSE_WAIT state with h2
July 26, 2018 11:00AM
Hello,


2018-07-26 10:33 GMT+02:00 Willy Tarreau <[email protected]>:

> Hi Milan!
>
> On Thu, Jul 26, 2018 at 10:31:24AM +0200, Milan Petruzelka wrote:
> > my haproxy with last two patches has no blocked connection since it
> started
> > 24 hours ago. Before last patches it always had at least one. I think you
> > finally nailed it. To be on the safe side, I would wait till Monday,
> before
> > we start to celebrate.
>
> Ah that's an awesome news! I'm fine with waiting before Monday to open
> the Champagne though :-)
>
> Thanks!
> Willy
>

I grab latest haproxy 1.8 from git repository
http://git.haproxy.org/git/haproxy-1.8.git/
I applied these patches:
h2-error.diff
0001-WIP-h2-try-to-address-possible-causes-for-the-close_.patch
(as you said,
0001-MINOR-h2-add-the-error-code-and-the-max-last-stream-.patch was already
applied).

new binary:
HA-Proxy version 1.8.12-f8a0ef4-19 2018/07/24

Previous build:
https://tof.cx/images/2018/07/26/f31243bfede22e20a7a991ae6c39506d.png
(we can clearly see when reload happens :p)

New build:
https://tof.cx/images/2018/07/26/e402d7fe15604d50418891071628019b.png


Seems you found what was wrong :) Great work ! Thank you to Milan too for
first raising the issue.

I will keep this new binary for a few days, just to check that every cases
are handled correctly.

Olivier
Willy Tarreau
Re: Connections stuck in CLOSE_WAIT state with h2
July 26, 2018 11:20AM
Hi Olivier,

On Thu, Jul 26, 2018 at 10:53:33AM +0200, Olivier Doucet wrote:
> Previous build:
> https://tof.cx/images/2018/07/26/f31243bfede22e20a7a991ae6c39506d.png
> (we can clearly see when reload happens :p)
>
> New build:
> https://tof.cx/images/2018/07/26/e402d7fe15604d50418891071628019b.png

Impressive!

> Seems you found what was wrong :) Great work ! Thank you to Milan too for
> first raising the issue.

Yeah, and to you as well for confirming, that's always pleasant!

> I will keep this new binary for a few days, just to check that every cases
> are handled correctly.

Perfect, thanks!

Willy
Olivier Doucet
Re: Connections stuck in CLOSE_WAIT state with h2
July 27, 2018 09:10AM
Hello,


2018-07-26 11:09 GMT+02:00 Willy Tarreau <[email protected]>:

> Hi Olivier,
>
> On Thu, Jul 26, 2018 at 10:53:33AM +0200, Olivier Doucet wrote:
> > Previous build:
> > https://tof.cx/images/2018/07/26/f31243bfede22e20a7a991ae6c39506d.png
> > (we can clearly see when reload happens :p)
> >
> > New build:
> > https://tof.cx/images/2018/07/26/e402d7fe15604d50418891071628019b.png
>
> Impressive!
>
> > Seems you found what was wrong :) Great work ! Thank you to Milan too for
> > first raising the issue.
>
> Yeah, and to you as well for confirming, that's always pleasant!
>
> > I will keep this new binary for a few days, just to check that every
> cases
> > are handled correctly.
>

24 hours later, still no issue to be reported. All sessions are expiring
just fine. I think you can merge :)

Olivier
Willy Tarreau
Re: Connections stuck in CLOSE_WAIT state with h2
July 27, 2018 10:20AM
Hi Olivier,

On Fri, Jul 27, 2018 at 09:04:04AM +0200, Olivier Doucet wrote:
> 24 hours later, still no issue to be reported. All sessions are expiring
> just fine. I think you can merge :)

Yes I think you're right, I'll do this, it will at least help all the users
who don't want to patch their versions. We'll probably emit another 1.8 soon.

Thanks!
Willy
Milan Petruželka
Re: Connections stuck in CLOSE_WAIT state with h2
July 27, 2018 10:40AM
On Fri, 27 Jul 2018 at 10:08, Willy Tarreau <[email protected]> wrote:

> Hi Olivier,
>
> On Fri, Jul 27, 2018 at 09:04:04AM +0200, Olivier Doucet wrote:
> > 24 hours later, still no issue to be reported. All sessions are expiring
> > just fine. I think you can merge :)
>
> Yes I think you're right, I'll do this, it will at least help all the users
> who don't want to patch their versions. We'll probably emit another 1.8
> soon.


Hi,

after 2 days I also have no blocked connections. There's no need to wait
until Monday as I suggested yesterday.

Milan
Willy Tarreau
Re: Connections stuck in CLOSE_WAIT state with h2
July 27, 2018 10:40AM
On Fri, Jul 27, 2018 at 10:28:36AM +0200, Milan Petruzelka wrote:
> after 2 days I also have no blocked connections. There's no need to wait
> until Monday as I suggested yesterday.

Perfect, many thanks Milan.

Willy
Janusz Dziemidowicz
Re: Connections stuck in CLOSE_WAIT state with h2
August 02, 2018 11:40AM
pt., 27 lip 2018 o 10:35 Willy Tarreau <[email protected]> napisał(a):
>
> On Fri, Jul 27, 2018 at 10:28:36AM +0200, Milan Petruzelka wrote:
> > after 2 days I also have no blocked connections. There's no need to wait
> > until Monday as I suggested yesterday.
>
> Perfect, many thanks Milan.

Sorry for being late, but 1.8.13 fixes the CLOSE_WAIT problem for me too :)
Now I have to dig into protocol errors I get when enabling h2, but
this will probably happen next week. I will create a new thread for
this.

--
Janusz Dziemidowicz
Willy Tarreau
Re: Connections stuck in CLOSE_WAIT state with h2
August 02, 2018 11:40AM
On Thu, Aug 02, 2018 at 11:29:17AM +0200, Janusz Dziemidowicz wrote:
> pt., 27 lip 2018 o 10:35 Willy Tarreau <[email protected]> napisal(a):
> >
> > On Fri, Jul 27, 2018 at 10:28:36AM +0200, Milan Petruzelka wrote:
> > > after 2 days I also have no blocked connections. There's no need to wait
> > > until Monday as I suggested yesterday.
> >
> > Perfect, many thanks Milan.
>
> Sorry for being late, but 1.8.13 fixes the CLOSE_WAIT problem for me too :)
> Now I have to dig into protocol errors I get when enabling h2, but
> this will probably happen next week. I will create a new thread for
> this.

Perfect, thank you!

willy
Sorry, only registered users may post in this forum.

Click here to login