Welcome! Log In Create A New Profile

Advanced

Problem using http2 termination in 1.8

Posted by Glen Huang 
Glen Huang
Problem using http2 termination in 1.8
December 04, 2017 05:10AM
Hi,

I’m using 1.8 to do http2 termination, but when my backend tried to read an uploaded image from haproxy and save it to the file system, it could only read some data, and then haproxy stopped responding, and after awhile my backend failed:

read tcp 172.18.0.4:80->172.18.0.5:44760: read: connection reset by peer

And on the file system, I was left with a partial image. This doesn’t always happen, but like 20% of the time. It saves the full image correctly in the other 80%.

My haproxy live in docker, and is from the official haproxy:1.8-alpine image, and my backend is a simple go http server. I wonder if I did something wrong or it’s a bug in 1.8?

I also tried manually upload the image with curl, could reproduce it.

Any help is much appreciated.

Here is my haproxy config:

global
hard-stop-after 30s
tune.ssl.default-dh-param 2048

# Default ciphers to use on SSL-enabled listening sockets.
# For more information, see ciphers(1SSL). This list is from:
# https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
ssl-default-bind-options no-sslv3
ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS

defaults
timeout connect 5s
timeout client 1m
timeout server 1m

frontend http
mode http
bind :80
redirect scheme https code 301

frontend https
mode http
bind :443 ssl crt /usr/local/etc/haproxy/certs/all.pem alpn h2,http/1.1
http-request set-header Forwarded for=%ci
default_backend server

backend server
mode http
server server server:80
Glen Huang
Re: Problem using http2 termination in 1.8
December 04, 2017 05:20AM
Forgot to mention, if I upload directly to the backend, without going through haproxy, the image is saved correctly every time. So I’m pretty sure the problem lies within haproxy.

> On 4 Dec 2017, at 11:59 AM, Glen Huang <[email protected]> wrote:
>
> Hi,
>
> I’m using 1.8 to do http2 termination, but when my backend tried to read an uploaded image from haproxy and save it to the file system, it could only read some data, and then haproxy stopped responding, and after awhile my backend failed:
>
> read tcp 172.18.0.4:80->172.18.0.5:44760: read: connection reset by peer
>
> And on the file system, I was left with a partial image. This doesn’t always happen, but like 20% of the time. It saves the full image correctly in the other 80%.
>
> My haproxy live in docker, and is from the official haproxy:1.8-alpine image, and my backend is a simple go http server. I wonder if I did something wrong or it’s a bug in 1.8?
>
> I also tried manually upload the image with curl, could reproduce it.
>
> Any help is much appreciated.
>
> Here is my haproxy config:
>
> global
> hard-stop-after 30s
> tune.ssl.default-dh-param 2048
>
> # Default ciphers to use on SSL-enabled listening sockets.
> # For more information, see ciphers(1SSL). This list is from:
> # https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
> ssl-default-bind-options no-sslv3
> ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
>
> defaults
> timeout connect 5s
> timeout client 1m
> timeout server 1m
>
> frontend http
> mode http
> bind :80
> redirect scheme https code 301
>
> frontend https
> mode http
> bind :443 ssl crt /usr/local/etc/haproxy/certs/all.pem alpn h2,http/1.1
> http-request set-header Forwarded for=%ci
> default_backend server
>
> backend server
> mode http
> server server server:80
>
>
>
>
Glen Huang
Re: Problem using http2 termination in 1.8
December 04, 2017 08:30AM
Something more to report:

- I initially was on 1.8.0, but upgrading to 1.8.1 doesn’t fix the problem, alpine or not.
- If instead of doing http2 termination, I ask haproxy to relay http directly, it works correctly.
- If instead of doing http2 termination, I ask haproxy to relay https1.1, it works correctly.

Seems the problem lies within the process of converting http2 to http1.1

> On 4 Dec 2017, at 11:59 AM, Glen Huang <[email protected]> wrote:
>
> Hi,
>
> I’m using 1.8 to do http2 termination, but when my backend tried to read an uploaded image from haproxy and save it to the file system, it could only read some data, and then haproxy stopped responding, and after awhile my backend failed:
>
> read tcp 172.18.0.4:80->172.18.0.5:44760: read: connection reset by peer
>
> And on the file system, I was left with a partial image. This doesn’t always happen, but like 20% of the time. It saves the full image correctly in the other 80%.
>
> My haproxy live in docker, and is from the official haproxy:1.8-alpine image, and my backend is a simple go http server. I wonder if I did something wrong or it’s a bug in 1.8?
>
> I also tried manually upload the image with curl, could reproduce it.
>
> Any help is much appreciated.
>
> Here is my haproxy config:
>
> global
> hard-stop-after 30s
> tune.ssl.default-dh-param 2048
>
> # Default ciphers to use on SSL-enabled listening sockets.
> # For more information, see ciphers(1SSL). This list is from:
> # https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
> ssl-default-bind-options no-sslv3
> ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
>
> defaults
> timeout connect 5s
> timeout client 1m
> timeout server 1m
>
> frontend http
> mode http
> bind :80
> redirect scheme https code 301
>
> frontend https
> mode http
> bind :443 ssl crt /usr/local/etc/haproxy/certs/all.pem alpn h2,http/1.1
> http-request set-header Forwarded for=%ci
> default_backend server
>
> backend server
> mode http
> server server server:80
>
>
>
>
Willy Tarreau
Re: Problem using http2 termination in 1.8
December 04, 2017 06:40PM
Hi Glen,

On Mon, Dec 04, 2017 at 03:18:12PM +0800, Glen Huang wrote:
> Something more to report:
>
> - I initially was on 1.8.0, but upgrading to 1.8.1 doesn't fix the problem, alpine or not.
> - If instead of doing http2 termination, I ask haproxy to relay http directly, it works correctly.
> - If instead of doing http2 termination, I ask haproxy to relay https1.1, it works correctly.
>
> Seems the problem lies within the process of converting http2 to http1.1

It looks very much like this indeed. I'll try again with curl (I did
most of the upload tests with it). Just to narrow down the issue, what
approximate size is your image ? I'm interested in figuring if it's
small (smaller than a buffer), medium (a few hundreds of kB) or large
(tens of MB).

Thanks!
Willy
Willy Tarreau
Re: Problem using http2 termination in 1.8
December 04, 2017 07:10PM
On Mon, Dec 04, 2017 at 06:35:09PM +0100, Willy Tarreau wrote:
> Hi Glen,
>
> On Mon, Dec 04, 2017 at 03:18:12PM +0800, Glen Huang wrote:
> > Something more to report:
> >
> > - I initially was on 1.8.0, but upgrading to 1.8.1 doesn't fix the problem, alpine or not.
> > - If instead of doing http2 termination, I ask haproxy to relay http directly, it works correctly.
> > - If instead of doing http2 termination, I ask haproxy to relay https1.1, it works correctly.
> >
> > Seems the problem lies within the process of converting http2 to http1.1
>
> It looks very much like this indeed. I'll try again with curl (I did
> most of the upload tests with it). Just to narrow down the issue, what
> approximate size is your image ? I'm interested in figuring if it's
> small (smaller than a buffer), medium (a few hundreds of kB) or large
> (tens of MB).

I seem to be able to generate this case by uploading a few megabytes to
a cpu-bound script (reading data using dd bs=10). In this case, the file
is indeed truncated. I'm investigating now.

Willy
Willy Tarreau
Re: Problem using http2 termination in 1.8
December 04, 2017 07:30PM
On Mon, Dec 04, 2017 at 07:03:24PM +0100, Willy Tarreau wrote:
> I seem to be able to generate this case by uploading a few megabytes to
> a cpu-bound script (reading data using dd bs=10). In this case, the file
> is indeed truncated. I'm investigating now.

I manage to reproduce it more easily when haproxy is running under
strace. I've seen that some data frames were received by the SSL stack
but not sent to the other side. I suspect an issue with how/when the
connection is woken up to enable reading again.

Willy
Sorry, only registered users may post in this forum.

Click here to login