Welcome! Log In Create A New Profile

Advanced

HAProxy 1.8.x not serving errorfiles with H2

Posted by J. Casalino 
J. Casalino
HAProxy 1.8.x not serving errorfiles with H2
June 05, 2018 06:40PM
We are in the process of testing HAProxy 1.8.x with ALPN and H2 on some of our servers. We have default 502 and 503 errorfiles defined (ex. errorfile 503 /etc/haproxy/errors/503.http), but we've noticed that these errorfiles are not served to the user's browser when the error occurs (for instance, if the backend is down, a user should get the 503 errorfile).

Chrome returns "ERR_SPDY_PROTOCOL_ERROR", Curl [1] returns "curl: (92) HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2)", and Firefox shows "The connection to <servername> was interrupted while the page was loading."

With debug logging turned on, I can see that HAProxy is recognizing a 503 if the back-end server is down [2], but it doesn't seem to pass that error through to the client browser. If the backend is up and a 502 is generated, users do not receive the errorfile either. If we turn off H2 and drop back to HTTP/1.1, the errorfiles are displayed properly (though via HTTP/0.9)

This has been observed in both 1.8.4 and 1.8.9. Our platform is Amazon Linux, using openssl-1.0.2k-12.109.amzn1.x86_64.

Thanks in advance for any thoughts you might have -

[1]
Curl verbose (curl -vvvvI) output:
* Trying <ipaddr>...
* TCP_NODELAY set
* Connected to <servername> (<ipaddr>) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: [removed]
* start date: Mar 20 00:00:00 2017 GMT
* expire date: Mar 24 12:00:00 2020 GMT
* subjectAltName: host "<hostname>" matched cert's "<certname>"
* issuer: C=US; O=DigiCert Inc; CN=DigiCert SHA2 Secure Server CA
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fbded005400)
> HEAD /libs/cq/core/content/welcome.html HTTP/2
> Host: <hostname>
> User-Agent: curl/7.54.0
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
* HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2)
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, Client hello (1):
curl: (92) HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2)

[2] haproxy[19803]: <ipaddr>:63832 [05/Jun/2018:15:36:24.202] incoming_https~ local_author_app_http/<NOSRV> 0/-1/-1/-1/0 503 1441 - - SCDN 3/1/0/0/0 0/0 "GET /libs/cq/core/content/welcome.html HTTP/1.1"
J. Casalino
RE: HAProxy 1.8.x not serving errorfiles with H2
June 11, 2018 04:50PM
Trying again - has anyone else seen this?


From: J. Casalino [mailto:[email protected]]
Sent: Tuesday, June 5, 2018 12:27 PM
To: haproxy@formilux.org
Subject: HAProxy 1.8.x not serving errorfiles with H2

We are in the process of testing HAProxy 1.8.x with ALPN and H2 on some of our servers. We have default 502 and 503 errorfiles defined (ex. errorfile 503 /etc/haproxy/errors/503.http), but we've noticed that these errorfiles are not served to the user's browser when the error occurs (for instance, if the backend is down, a user should get the 503 errorfile).

Chrome returns "ERR_SPDY_PROTOCOL_ERROR", Curl [1] returns "curl: (92) HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2)", and Firefox shows "The connection to <servername> was interrupted while the page was loading."

With debug logging turned on, I can see that HAProxy is recognizing a 503 if the back-end server is down [2], but it doesn't seem to pass that error through to the client browser. If the backend is up and a 502 is generated, users do not receive the errorfile either. If we turn off H2 and drop back to HTTP/1.1, the errorfiles are displayed properly (though via HTTP/0.9)

This has been observed in both 1.8.4 and 1.8.9. Our platform is Amazon Linux, using openssl-1.0.2k-12.109.amzn1.x86_64.

Thanks in advance for any thoughts you might have -

[1]
Curl verbose (curl -vvvvI) output:
* Trying <ipaddr>...
* TCP_NODELAY set
* Connected to <servername> (<ipaddr>) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: [removed]
* start date: Mar 20 00:00:00 2017 GMT
* expire date: Mar 24 12:00:00 2020 GMT
* subjectAltName: host "<hostname>" matched cert's "<certname>"
* issuer: C=US; O=DigiCert Inc; CN=DigiCert SHA2 Secure Server CA
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fbded005400)
> HEAD /libs/cq/core/content/welcome.html HTTP/2
> Host: <hostname>
> User-Agent: curl/7.54.0
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
* HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2)
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, Client hello (1):
curl: (92) HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2)

[2] haproxy[19803]: <ipaddr>:63832 [05/Jun/2018:15:36:24.202] incoming_https~ local_author_app_http/<NOSRV> 0/-1/-1/-1/0 503 1441 - - SCDN 3/1/0/0/0 0/0 "GET /libs/cq/core/content/welcome.html HTTP/1.1"
Cyril Bonté
Re: HAProxy 1.8.x not serving errorfiles with H2
June 11, 2018 10:40PM
Hi,

Le 11/06/2018 à 16:39, J. Casalino a écrit :
> Trying again – has anyone else seen this?
>
> *From:* J. Casalino [mailto:[email protected]]
> *Sent:* Tuesday, June 5, 2018 12:27 PM
> *To:* haproxy@formilux.org
> *Subject:* HAProxy 1.8.x not serving errorfiles with H2
>
> We are in the process of testing HAProxy 1.8.x with ALPN and H2 on some
> of our servers. We have default 502 and 503 errorfiles defined (ex.
> errorfile 503 /etc/haproxy/errors/503.http), but we’ve noticed that
> these errorfiles are not served to the user’s browser when the error
> occurs (for instance, if the backend is down, a user should get the 503
> errorfile).
>
> Chrome returns "ERR_SPDY_PROTOCOL_ERROR”, Curl [1] returns “curl: (92)
> HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2)”, and
> Firefox shows “The connection to <servername> was interrupted while the
> page was loading.”
>
> With debug logging turned on, I can see that HAProxy is recognizing a
> 503 if the back-end server is down [2], but it doesn’t seem to pass that
> error through to the client browser. If the backend is up and a 502 is
> generated, users do not receive the errorfile either. If we turn off H2
> and drop back to HTTP/1.1, the errorfiles are displayed properly (though
> via HTTP/0.9)

If it looks like HTTP/0.9, I tend to think that your errorfile is not
properly set. Such files must contain the status line, the headers and
the response body.

And indeed, from a quick test, if I remove the status line, I get the
same error with a HTTP/2 request. Once the file is correctly defined,
everything is OK.

Can you provide the content of your errorfile(s) ?


> This has been observed in both 1.8.4 and 1.8.9. Our platform is Amazon
> Linux, using openssl-1.0.2k-12.109.amzn1.x86_64.
>
> Thanks in advance for any thoughts you might have –
>
> [1]
>
> Curl verbose (curl -vvvvI) output:
>
> *   Trying <ipaddr>...
>
> * TCP_NODELAY set
>
> * Connected to <servername> (<ipaddr>) port 443 (#0)
>
> * ALPN, offering h2
>
> * ALPN, offering http/1.1
>
> * Cipher selection:
> ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
>
> * successfully set certificate verify locations:
>
> *   CAfile: /etc/ssl/cert.pem
>
>   CApath: none
>
> * TLSv1.2 (OUT), TLS handshake, Client hello (1):
>
> * TLSv1.2 (IN), TLS handshake, Server hello (2):
>
> * TLSv1.2 (IN), TLS handshake, Certificate (11):
>
> * TLSv1.2 (IN), TLS handshake, Server key exchange (12):
>
> * TLSv1.2 (IN), TLS handshake, Server finished (14):
>
> * TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
>
> * TLSv1.2 (OUT), TLS change cipher, Client hello (1):
>
> * TLSv1.2 (OUT), TLS handshake, Finished (20):
>
> * TLSv1.2 (IN), TLS change cipher, Client hello (1):
>
> * TLSv1.2 (IN), TLS handshake, Finished (20):
>
> * SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
>
> * ALPN, server accepted to use h2
>
> * Server certificate:
>
> *  subject: [removed]
>
> *  start date: Mar 20 00:00:00 2017 GMT
>
> *  expire date: Mar 24 12:00:00 2020 GMT
>
> *  subjectAltName: host "<hostname>" matched cert's "<certname>"
>
> *  issuer: C=US; O=DigiCert Inc; CN=DigiCert SHA2 Secure Server CA
>
> *  SSL certificate verify ok.
>
> * Using HTTP2, server supports multi-use
>
> * Connection state changed (HTTP/2 confirmed)
>
> * Copying HTTP/2 data in stream buffer to connection buffer after
> upgrade: len=0
>
> * Using Stream ID: 1 (easy handle 0x7fbded005400)
>
> > HEAD /libs/cq/core/content/welcome.html HTTP/2
>
> > Host: <hostname>
>
> > User-Agent: curl/7.54.0
>
> > Accept: */*
>
> >
>
> * Connection state changed (MAX_CONCURRENT_STREAMS updated)!
>
> * HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2)
>
> * Closing connection 0
>
> * TLSv1.2 (OUT), TLS alert, Client hello (1):
>
> curl: (92) HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2)
>
> [2] haproxy[19803]: <ipaddr>:63832 [05/Jun/2018:15:36:24.202]
> incoming_https~ local_author_app_http/<NOSRV> 0/-1/-1/-1/0 503 1441 - -
> SCDN 3/1/0/0/0 0/0 "GET /libs/cq/core/content/welcome.html HTTP/1.1"
>


--
Cyril Bonté
Willy Tarreau
Re: HAProxy 1.8.x not serving errorfiles with H2
June 12, 2018 02:20PM
Hi Cyril,

On Mon, Jun 11, 2018 at 10:36:43PM +0200, Cyril Bonté wrote:
> If it looks like HTTP/0.9, I tend to think that your errorfile is not
> properly set. Such files must contain the status line, the headers and the
> response body.
>
> And indeed, from a quick test, if I remove the status line, I get the same
> error with a HTTP/2 request. Once the file is correctly defined, everything
> is OK.
>
> Can you provide the content of your errorfile(s) ?

That's indeed very possible. The H2 to H1 gateway requires that the
server side respects HTTP/1.1 messaging and semantics, otherwise it
will not be able to parse the response. It's obviously the same for
error messages since these ones are converted back to H2 by the same
gateway.

Willy
J. Casalino
RE: HAProxy 1.8.x not serving errorfiles with H2
June 12, 2018 06:00PM
Hi Cyril,

Thank you so much for your response. [1] is the simplest example of our 503 errorfile. Based on what you're saying it sounds like we need to add some additional information into the file. Is this documented somewhere?

[1]
<html>
<head>
<title>HTTP 503</title>
</head>
<body style="font-family:Arial,Helvetica,sans-serif;">
<div style="margin: 0 auto; width: 960px;">
<h2 >Author is unavailable</h2>
<p>
Either the instance is temporarily unavailable, down or being restarted. If this issue persists, please <a href="REMOVED">file a ticket</a>
<br />
<pre>
W W W
W W W W
'. W
.-""-._ \ \.--|
/ "-..__) .-'
| _ /
\'-.__, .__.,'
`'----'._\--'
VVVVVVVVVVVVVVVVVVVVV
</pre>
</div>
</body>
</html>


-----Original Message-----
From: Cyril Bonté [mailto:[email protected]]
Sent: Monday, June 11, 2018 4:37 PM
To: J. Casalino <[email protected]>
Cc: haproxy@formilux.org
Subject: Re: HAProxy 1.8.x not serving errorfiles with H2

Hi,

Le 11/06/2018 à 16:39, J. Casalino a écrit :
> Trying again - has anyone else seen this?
>
> *From:* J. Casalino [mailto:[email protected]]
> *Sent:* Tuesday, June 5, 2018 12:27 PM
> *To:* haproxy@formilux.org
> *Subject:* HAProxy 1.8.x not serving errorfiles with H2
>
> We are in the process of testing HAProxy 1.8.x with ALPN and H2 on
> some of our servers. We have default 502 and 503 errorfiles defined (ex.
> errorfile 503 /etc/haproxy/errors/503.http), but we've noticed that
> these errorfiles are not served to the user's browser when the error
> occurs (for instance, if the backend is down, a user should get the
> 503 errorfile).
>
> Chrome returns "ERR_SPDY_PROTOCOL_ERROR", Curl [1] returns "curl: (92)
> HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2)", and
> Firefox shows "The connection to <servername> was interrupted while
> the page was loading."
>
> With debug logging turned on, I can see that HAProxy is recognizing a
> 503 if the back-end server is down [2], but it doesn't seem to pass
> that error through to the client browser. If the backend is up and a
> 502 is generated, users do not receive the errorfile either. If we
> turn off H2 and drop back to HTTP/1.1, the errorfiles are displayed
> properly (though via HTTP/0.9)

If it looks like HTTP/0.9, I tend to think that your errorfile is not properly set. Such files must contain the status line, the headers and the response body.

And indeed, from a quick test, if I remove the status line, I get the same error with a HTTP/2 request. Once the file is correctly defined, everything is OK.

Can you provide the content of your errorfile(s) ?


> This has been observed in both 1.8.4 and 1.8.9. Our platform is Amazon
> Linux, using openssl-1.0.2k-12.109.amzn1.x86_64.
>
> Thanks in advance for any thoughts you might have -
>
> [1]
>
> Curl verbose (curl -vvvvI) output:
>
> *   Trying <ipaddr>...
>
> * TCP_NODELAY set
>
> * Connected to <servername> (<ipaddr>) port 443 (#0)
>
> * ALPN, offering h2
>
> * ALPN, offering http/1.1
>
> * Cipher selection:
> ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
>
> * successfully set certificate verify locations:
>
> *   CAfile: /etc/ssl/cert.pem
>
>   CApath: none
>
> * TLSv1.2 (OUT), TLS handshake, Client hello (1):
>
> * TLSv1.2 (IN), TLS handshake, Server hello (2):
>
> * TLSv1.2 (IN), TLS handshake, Certificate (11):
>
> * TLSv1.2 (IN), TLS handshake, Server key exchange (12):
>
> * TLSv1.2 (IN), TLS handshake, Server finished (14):
>
> * TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
>
> * TLSv1.2 (OUT), TLS change cipher, Client hello (1):
>
> * TLSv1.2 (OUT), TLS handshake, Finished (20):
>
> * TLSv1.2 (IN), TLS change cipher, Client hello (1):
>
> * TLSv1.2 (IN), TLS handshake, Finished (20):
>
> * SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
>
> * ALPN, server accepted to use h2
>
> * Server certificate:
>
> *  subject: [removed]
>
> *  start date: Mar 20 00:00:00 2017 GMT
>
> *  expire date: Mar 24 12:00:00 2020 GMT
>
> *  subjectAltName: host "<hostname>" matched cert's "<certname>"
>
> *  issuer: C=US; O=DigiCert Inc; CN=DigiCert SHA2 Secure Server CA
>
> *  SSL certificate verify ok.
>
> * Using HTTP2, server supports multi-use
>
> * Connection state changed (HTTP/2 confirmed)
>
> * Copying HTTP/2 data in stream buffer to connection buffer after
> upgrade: len=0
>
> * Using Stream ID: 1 (easy handle 0x7fbded005400)
>
> > HEAD /libs/cq/core/content/welcome.html HTTP/2
>
> > Host: <hostname>
>
> > User-Agent: curl/7.54.0
>
> > Accept: */*
>
> >
>
> * Connection state changed (MAX_CONCURRENT_STREAMS updated)!
>
> * HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2)
>
> * Closing connection 0
>
> * TLSv1.2 (OUT), TLS alert, Client hello (1):
>
> curl: (92) HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err
> 2)
>
> [2] haproxy[19803]: <ipaddr>:63832 [05/Jun/2018:15:36:24.202]
> incoming_https~ local_author_app_http/<NOSRV> 0/-1/-1/-1/0 503 1441 -
> - SCDN 3/1/0/0/0 0/0 "GET /libs/cq/core/content/welcome.html HTTP/1.1"
>


--
Cyril Bonté
Tim Düsterhus
Re: HAProxy 1.8.x not serving errorfiles with H2
June 12, 2018 08:30PM
Hi

Am 12.06.2018 um 17:43 schrieb J. Casalino:
> Thank you so much for your response. [1] is the simplest example of our 503 errorfile. Based on what you're saying it sounds like we need to add some additional information into the file. Is this documented somewhere?
>

it is [1]:

> The files are returned verbatim on the TCP socket. This allows any trick such
> as redirections to another URL or site, as well as tricks to clean cookies,
> force enable or disable caching, etc... The package provides default error
> files returning the same contents as default errors.

You need to prefix HTTP response headers. Something like this should
work [2]:

> HTTP/1.0 503 Service Unavailable
> Cache-Control: no-cache
> Connection: close
> Content-Type: text/html

Best regards
Tim Düsterhus

[1]
https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#4.2-errorfile
[2]
http://git.haproxy.org/?p=haproxy.git;a=blob;f=examples/errorfiles/503.http;h=48fde5881e680857462c32cc305f9179af19091f;hb=HEAD
J. Casalino
RE: HAProxy 1.8.x not serving errorfiles with H2
June 12, 2018 09:40PM
Thank you! That fixed the issue.

Might I suggest adding the below snippet into the "example" section of the errorfile documentation at [1]? The current example only shows the configuration itself but not what the file format should be - since it's this specific, the description given is just simply not enough for a casual proxy admin like myself to successfully configure it.

-----Original Message-----
From: Tim Düsterhus [mailto:[email protected]]
Sent: Tuesday, June 12, 2018 2:22 PM
To: J. Casalino <[email protected]>; Cyril Bonté <[email protected]>
Cc: haproxy@formilux.org
Subject: Re: HAProxy 1.8.x not serving errorfiles with H2

Hi

Am 12.06.2018 um 17:43 schrieb J. Casalino:
> Thank you so much for your response. [1] is the simplest example of our 503 errorfile. Based on what you're saying it sounds like we need to add some additional information into the file. Is this documented somewhere?
>

it is [1]:

> The files are returned verbatim on the TCP socket. This allows any
> trick such as redirections to another URL or site, as well as tricks
> to clean cookies, force enable or disable caching, etc... The package
> provides default error files returning the same contents as default errors.

You need to prefix HTTP response headers. Something like this should work [2]:

> HTTP/1.0 503 Service Unavailable
> Cache-Control: no-cache
> Connection: close
> Content-Type: text/html

Best regards
Tim Düsterhus

[1]
https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#4.2-errorfile
[2]
http://git.haproxy.org/?p=haproxy.git;a=blob;f=examples/errorfiles/503.http;h=48fde5881e680857462c32cc305f9179af19091f;hb=HEAD
Sorry, only registered users may post in this forum.

Click here to login