Help with Chrome preconnects

hi,

I am using HAProxy 1.4.x infront of Apache 2.2.x. For SSLs, I just do a tcp
redirect from port 443. Like
==
listen ssl-relay 0.0.0.0:443
mode tcp
balance roundrobin
server inst1 machinename:443 check inter 2000 fall 3
==

Everything was running fine till Chrome introduced preconnects. I have
logged a bug at http://code.google.com/p/chromium/issues/detail?id=87121
Its a fairly long thread but the gist is the following :
Chrome does some speculative SSL connects to the backend and does not close
the handshake. The problem for us now is that the request goes to an Apache
process and that process gets blocked for the entire duration of the
timeout! If in httpd.conf we have 60seconds as timeout, there are one or
two Apache processes that will get blocked in "Reading request" state for
60seconds thinking that the chrome user will use the connection! As you can
easily see, this is really a drain on the process pool and very soon it
maxes out on child processes.

Is there anyway HAProxy can help here? As in, is there anyway HAProxy does
not open an apache connection till there is any "activity" on the
connection? Please let me know. I guess most systems would have this
problem but for some reason, I can not find enough links on google.
Or if you can think of other ways of handling this, please let me know that
too.

Thanks,
Vikram

Hey,

Depends at which phase of the health check Chrome maintains the
connection opened, you can give a try to HAProxy's content inspection:
listen https
mode tcp
balance roundrobin
acl clienthello req_ssl_hello_type 1

# use tcp content accepts to detects ssl client and server hello.
tcp-request inspect-delay 5s
tcp-request content accept if clienthello

server s1 192.168.1.1:443


It requires HAProxy 1.5.

The client (chrome) will have 5s to send its SSL clienthello, as long
as it does not do it, HAProxy won't even open a connection to the
server.
So if the chrome "bug" is related to this phase, it may help, if
chrome keeps the connection opened after this phase, it won't help.

You could try setting up "client timeout" and "server timeout".

Cheers


On Wed, Jul 11, 2012 at 3:45 AM, Vikram Nayak <[email protected]> wrote:
> hi,
>
> I am using HAProxy 1.4.x infront of Apache 2.2.x. For SSLs, I just do a tcp
> redirect from port 443. Like
> ==
> listen ssl-relay 0.0.0.0:443
> mode tcp
> balance roundrobin
> server inst1 machinename:443 check inter 2000 fall 3
> ==
>
> Everything was running fine till Chrome introduced preconnects. I have
> logged a bug at http://code.google.com/p/chromium/issues/detail?id=87121
> Its a fairly long thread but the gist is the following :
> Chrome does some speculative SSL connects to the backend and does not close
> the handshake. The problem for us now is that the request goes to an Apache
> process and that process gets blocked for the entire duration of the
> timeout! If in httpd.conf we have 60seconds as timeout, there are one or two
> Apache processes that will get blocked in "Reading request" state for
> 60seconds thinking that the chrome user will use the connection! As you can
> easily see, this is really a drain on the process pool and very soon it
> maxes out on child processes.
>
> Is there anyway HAProxy can help here? As in, is there anyway HAProxy does
> not open an apache connection till there is any "activity" on the
> connection? Please let me know. I guess most systems would have this problem
> but for some reason, I can not find enough links on google.
> Or if you can think of other ways of handling this, please let me know that
> too.
>
> Thanks,
> Vikram

I would suggest terminating SSL on the haproxy box (with stud in front of it), thus switching haproxy from tcp to http mode. That way, longrunning keepalive-enabled HTTPS sessions terminate there and apache only sees real non-SSL request without blocking any threads.
If you would like to avoid terminating SSL on the proxy, you could enable a haproxy+stud on the apache box (apache listens on 80/8080, and stud with haproxy listens on 443). This way, you can still use a dedicated tcp reverse proxy withouth ssl encryption, doing all the ssl work in the backend and you avoid blocking apache threads because you have an event based proxy in front of it (even if it is on the same box).





________________________________
> Date: Wed, 11 Jul 2012 07:15:19 +0530
> Subject: Help with Chrome preconnects
> From: vikram.nayak.p@gmail.com
> To: haproxy@formilux.org
>
> hi,
>
> I am using HAProxy 1.4.x infront of Apache 2.2.x. For SSLs, I just do a
> tcp redirect from port 443. Like
> ==
> listen ssl-relay 0.0.0.0:443http://0.0.0.0:443
> mode tcp
> balance roundrobin
> server inst1 machinename:443 check inter 2000 fall 3
> ==
>
> Everything was running fine till Chrome introduced preconnects. I have
> logged a bug
> at http://code.google.com/p/chromium/issues/detail?id=87121
> Its a fairly long thread but the gist is the following :
> Chrome does some speculative SSL connects to the backend and does not
> close the handshake. The problem for us now is that the request goes to
> an Apache process and that process gets blocked for the entire duration
> of the timeout! If in httpd.conf we have 60seconds as timeout, there
> are one or two Apache processes that will get blocked in "Reading
> request" state for 60seconds thinking that the chrome user will use the
> connection! As you can easily see, this is really a drain on the
> process pool and very soon it maxes out on child processes.
>
> Is there anyway HAProxy can help here? As in, is there anyway HAProxy
> does not open an apache connection till there is any "activity" on the
> connection? Please let me know. I guess most systems would have this
> problem but for some reason, I can not find enough links on google.
> Or if you can think of other ways of handling this, please let me know
> that too.
>
> Thanks,
> Vikram

On Wed, Jul 11, 2012 at 07:03:52AM +0200, Baptiste wrote:
> Hey,
>
> Depends at which phase of the health check Chrome maintains the
> connection opened, you can give a try to HAProxy's content inspection:
> listen https
> mode tcp
> balance roundrobin
> acl clienthello req_ssl_hello_type 1
>
> # use tcp content accepts to detects ssl client and server hello.
> tcp-request inspect-delay 5s
> tcp-request content accept if clienthello

You need to add this line here :

tcp-request content reject

because the tcp-request rules default to accept when none match. Otherwise
this should indeed work.

BTW, someone recently reported to me that chrome sometimes failed on SNI.
Now I understand : the guy was mixing SSL and SSH on the same port : if
the client does not send anything then it's SSH. Chrome was getting the
SSH page from time to time and decided that the server did not support
TLS so it then stopped using SNI. Now I understand why it was not sending
any handshake, this is because of this new broken behaviour which saves
one speculative roundtrip but which also consumes memory on servers for
nothing...

Regards,
Willy

Thanks for all the replies Willy, Baptiste and Lukas.

Unfortunately we could not get the "tcp content" trick to work. Our guess
is Chrome preconnect actually does a SSL handshake and so HAProxy does not
have a choice and has to engage a Apache worker. We used the following
==
listen https 0.0.0.0:44 http://0.0.0.0:44453
mode tcp
balance roundrobin
acl clienthello req_ssl_hello_type 1
# use tcp content accepts to detects ssl client and server hello.
tcp-request inspect-delay 20s
tcp-request content accept if clienthello
tcp-request content reject
server inst1 127.0.0.1:443 check inter 2000 fall 3
==

Lukas's suggestion was really helpful. I do not have any experience with
stud/stunnel. So I will get to these in a week or so when I get some time.
For now, we are using nginx for SSL termination and going the HAProxy route
for all non-SSL. nginx will route from 443 to non-ssl Apache on 80.
Hopefully will replace it with HAProxy+stud some time later.


Thanks,
Vikram

On Thu, Jul 12, 2012 at 11:39 AM, Willy Tarreau <[email protected]> wrote:

> On Wed, Jul 11, 2012 at 07:03:52AM +0200, Baptiste wrote:
> > Hey,
> >
> > Depends at which phase of the health check Chrome maintains the
> > connection opened, you can give a try to HAProxy's content inspection:
> > listen https
> > mode tcp
> > balance roundrobin
> > acl clienthello req_ssl_hello_type 1
> >
> > # use tcp content accepts to detects ssl client and server hello.
> > tcp-request inspect-delay 5s
> > tcp-request content accept if clienthello
>
> You need to add this line here :
>
> tcp-request content reject
>
> because the tcp-request rules default to accept when none match. Otherwise
> this should indeed work.
>
> BTW, someone recently reported to me that chrome sometimes failed on SNI.
> Now I understand : the guy was mixing SSL and SSH on the same port : if
> the client does not send anything then it's SSH. Chrome was getting the
> SSH page from time to time and decided that the server did not support
> TLS so it then stopped using SNI. Now I understand why it was not sending
> any handshake, this is because of this new broken behaviour which saves
> one speculative roundtrip but which also consumes memory on servers for
> nothing...
>
> Regards,
> Willy
>
>

Sorry for the confusing config that I gave. I actually meant
==
listen https 0.0.0.0:44 http://0.0.0.0:4445/3

mode tcp
balance roundrobin
acl clienthello req_ssl_hello_type 1
# use tcp content accepts to detects ssl client and server hello.
tcp-request inspect-delay 20s

tcp-request content accept if clienthello
tcp-request content reject
server inst1 127.0.0.1:44 http://127.0.0.1:443/4 check inter
2000 fall 3
==
And on 444 I have Apache running in ssl mode.

Thanks,
Vikram

On Fri, Jul 13, 2012 at 2:16 AM, Vikram Nayak <[email protected]>wrote:

> Thanks for all the replies Willy, Baptiste and Lukas.
>
> Unfortunately we could not get the "tcp content" trick to work. Our guess
> is Chrome preconnect actually does a SSL handshake and so HAProxy does not
> have a choice and has to engage a Apache worker. We used the following
> ==
> listen https 0.0.0.0:44 http://0.0.0.0:44453
>
> mode tcp
> balance roundrobin
> acl clienthello req_ssl_hello_type 1
> # use tcp content accepts to detects ssl client and server hello.
> tcp-request inspect-delay 20s
>
> tcp-request content accept if clienthello
> tcp-request content reject
> server inst1 127.0.0.1:443 check inter 2000 fall 3
> ==
>
> Lukas's suggestion was really helpful. I do not have any experience with
> stud/stunnel. So I will get to these in a week or so when I get some time.
> For now, we are using nginx for SSL termination and going the HAProxy route
> for all non-SSL. nginx will route from 443 to non-ssl Apache on 80.
> Hopefully will replace it with HAProxy+stud some time later.
>
>
> Thanks,
> Vikram
>
>
> On Thu, Jul 12, 2012 at 11:39 AM, Willy Tarreau <[email protected]> wrote:
>
>> On Wed, Jul 11, 2012 at 07:03:52AM +0200, Baptiste wrote:
>> > Hey,
>> >
>> > Depends at which phase of the health check Chrome maintains the
>> > connection opened, you can give a try to HAProxy's content inspection:
>> > listen https
>> > mode tcp
>> > balance roundrobin
>> > acl clienthello req_ssl_hello_type 1
>> >
>> > # use tcp content accepts to detects ssl client and server
>> hello.
>> > tcp-request inspect-delay 5s
>> > tcp-request content accept if clienthello
>>
>> You need to add this line here :
>>
>> tcp-request content reject
>>
>> because the tcp-request rules default to accept when none match. Otherwise
>> this should indeed work.
>>
>> BTW, someone recently reported to me that chrome sometimes failed on SNI.
>> Now I understand : the guy was mixing SSL and SSH on the same port : if
>> the client does not send anything then it's SSH. Chrome was getting the
>> SSH page from time to time and decided that the server did not support
>> TLS so it then stopped using SNI. Now I understand why it was not sending
>> any handshake, this is because of this new broken behaviour which saves
>> one speculative roundtrip but which also consumes memory on servers for
>> nothing...
>>
>> Regards,
>> Willy
>>
>>
>