Welcome! Log In Create A New Profile

Advanced

maxsslconn vs maxsslrate

Posted by Mihir Shirali 
Mihir Shirali
maxsslconn vs maxsslrate
June 07, 2018 07:10AM
Hi Team,

We use haproxy to front tls for a large number of endpoints, haproxy
prcesses the TLS session and then forwards the request to the backend
application.
What we have noticed is that if there are a large number of connections
from different clients - the CPU usage goes up significantly. This
primarily because haproxy is handling a lot ofSSL connections. I came
across 2 options above and tested them out.

With maxsslrate - CPU is better controlled and if I combine this with 503
response in the front end I see great results. Is there a possibility of
connection timeout on the client here if there are a very large number of
requests?

With maxsslconn, CPU is still pegged high - and clients receive a tcp
reset. This is also good, because there is no chance of tcp time out on the
client. Clients can retry after a bit and they are aware that the
connection is closed instead of waiting on timeout. However, CPU still
seems pegged high. What is the reason for high CPU on the server here - Is
it because SSL stack is still hit with this setting?

--
Regards,
Mihir
Aleksandar Lazic
Re: maxsslconn vs maxsslrate
June 07, 2018 10:50AM
Hi Mihir.

On 07/06/2018 10:27, Mihir Shirali wrote:
>Hi Team,
>
>We use haproxy to front tls for a large number of endpoints, haproxy
>prcesses the TLS session and then forwards the request to the backend
>application.
>
>What we have noticed is that if there are a large number of connections
>from different clients - the CPU usage goes up significantly. This
>primarily because haproxy is handling a lot ofSSL connections. I came
>across 2 options above and tested them out.

What do you mean with *large number*?

https://medium.freecodecamp.org/how-we-fine-tuned-haproxy-to-achieve-2-000-000-concurrent-ssl-connections-d017e61a4d27

>With maxsslrate - CPU is better controlled and if I combine this with
>503 response in the front end I see great results. Is there a
>possibility of connection timeout on the client here if there are a
>very large number of requests?
>
>With maxsslconn, CPU is still pegged high - and clients receive a tcp
>reset. This is also good, because there is no chance of tcp time out on
>the client. Clients can retry after a bit and they are aware that the
>connection is closed instead of waiting on timeout. However, CPU still
>seems pegged high. What is the reason for high CPU on the server here -
>Is it because SSL stack is still hit with this setting?

SSL/TLS handling isn't that easy.

Please can you share some more information's, because in the latest
versions of haproxy are a lot optimisation's introduced also for TLS.

haproxy -vv

Anonymized haproxy conf.

>--
>Regards,
>Mihir

Best regards
Aleks
Mihir Shirali
Re: maxsslconn vs maxsslrate
June 07, 2018 11:10AM
We have a large number of ip phones connecting to this port. They could be
as large as 80k. They request for a file from a custom application. haproxy
front ends the tls connection and then forwards the request to the
application's http port.

HA-Proxy version 1.8.8 2018/04/19
Copyright 2000-2018 Willy Tarreau <[email protected]>

Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement
-fwrapv -fno-strict-overflow -Wno-unused-label
OPTIONS = USE_OPENSSL=1

Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Running on OpenSSL version : OpenSSL 1.0.2l.6.2.83
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT
IP_FREEBIND
Encrypted password support via crypt(3): yes
Built with multi-threading support.
Built without PCRE or PCRE2 support (using libc's regex instead)
Built without compression support (neither USE_ZLIB nor USE_SLZ are set).
Compression algorithms supported : identity("identity")
Built with network namespace support.

Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.

Available filters :
[SPOE] spoe
[COMP] compression
[TRACE] trace


On Thu, Jun 7, 2018 at 2:13 PM, Aleksandar Lazic <[email protected]> wrote:

> Hi Mihir.
>
> On 07/06/2018 10:27, Mihir Shirali wrote:
>
>> Hi Team,
>>
>> We use haproxy to front tls for a large number of endpoints, haproxy
>> prcesses the TLS session and then forwards the request to the backend
>> application.
>>
>> What we have noticed is that if there are a large number of connections
>> from different clients - the CPU usage goes up significantly. This
>> primarily because haproxy is handling a lot ofSSL connections. I came
>> across 2 options above and tested them out.
>>
>
> What do you mean with *large number*?
>
> https://medium.freecodecamp.org/how-we-fine-tuned-haproxy-to
> -achieve-2-000-000-concurrent-ssl-connections-d017e61a4d27
>
> With maxsslrate - CPU is better controlled and if I combine this with
>> 503 response in the front end I see great results. Is there a
>> possibility of connection timeout on the client here if there are a
>> very large number of requests?
>>
>> With maxsslconn, CPU is still pegged high - and clients receive a tcp
>> reset. This is also good, because there is no chance of tcp time out on
>> the client. Clients can retry after a bit and they are aware that the
>> connection is closed instead of waiting on timeout. However, CPU still
>> seems pegged high. What is the reason for high CPU on the server here -
>> Is it because SSL stack is still hit with this setting?
>>
>
> SSL/TLS handling isn't that easy.
>
> Please can you share some more information's, because in the latest
> versions of haproxy are a lot optimisation's introduced also for TLS.
>
> haproxy -vv
>
> Anonymized haproxy conf.
>
> --
>> Regards,
>> Mihir
>>
>
> Best regards
> Aleks
>



--
Regards,
Mihir
Aleksandar Lazic
Re: maxsslconn vs maxsslrate
June 07, 2018 11:20AM
On 07/06/2018 14:30, Mihir Shirali wrote:
>We have a large number of ip phones connecting to this port. They could
>be as large as 80k. They request for a file from a custom
>application. haproxy front ends the tls connection and then forwards
>the request to the application's http port.

Have you take a look into the link below for some tunings for the system
and haproxy.

>HA-Proxy version 1.8.8 2018/04/19
>Copyright 2000-2018 Willy Tarreau <[email protected]>

[snipp]

Any change to update to 1.8.9?

Thanks can you also send the "Anonymized haproxy conf".
The main questions are do you use thread and or nbprocs?
This will be answered by the conf

Best regards
aleks

>On Thu, Jun 7, 2018 at 2:13 PM, Aleksandar Lazic <[email protected]> wrote:
>
>> Hi Mihir.
>>
>> On 07/06/2018 10:27, Mihir Shirali wrote:
>>
>>> Hi Team,
>>>
>>> We use haproxy to front tls for a large number of endpoints, haproxy
>>> prcesses the TLS session and then forwards the request to the backend
>>> application.
>>>
>>> What we have noticed is that if there are a large number of connections
>>> from different clients - the CPU usage goes up significantly. This
>>> primarily because haproxy is handling a lot ofSSL connections. I came
>>> across 2 options above and tested them out.
>>>
>>
>> What do you mean with *large number*?
>>
>> https://medium.freecodecamp.org/how-we-fine-tuned-haproxy-to
>> -achieve-2-000-000-concurrent-ssl-connections-d017e61a4d27
>>
>> With maxsslrate - CPU is better controlled and if I combine this with
>>> 503 response in the front end I see great results. Is there a
>>> possibility of connection timeout on the client here if there are a
>>> very large number of requests?
>>>
>>> With maxsslconn, CPU is still pegged high - and clients receive a tcp
>>> reset. This is also good, because there is no chance of tcp time out on
>>> the client. Clients can retry after a bit and they are aware that the
>>> connection is closed instead of waiting on timeout. However, CPU still
>>> seems pegged high. What is the reason for high CPU on the server here -
>>> Is it because SSL stack is still hit with this setting?
>>>
>>
>> SSL/TLS handling isn't that easy.
>>
>> Please can you share some more information's, because in the latest
>> versions of haproxy are a lot optimisation's introduced also for TLS.
>>
>> haproxy -vv
>>
>> Anonymized haproxy conf.
>>
>> --
>>> Regards,
>>> Mihir
>>>
>>
>> Best regards
>> Aleks
>
>--
>Regards,
>Mihir
Mihir Shirali
Re: maxsslconn vs maxsslrate
June 07, 2018 11:30AM
Hi Alexander,

I have looked at the link. What I am looking for is an answer to the
difference between maxsslconn and maxsslrate. The former does not result in
CPU savings while the latter does. Again the former does result in large
number of tcp connection resets while the latter does not. What I'd like to
know and understand is why that is the case.
I am using nbproc set to 2.

On Thu, Jun 7, 2018 at 2:43 PM, Aleksandar Lazic <[email protected]> wrote:

> On 07/06/2018 14:30, Mihir Shirali wrote:
>
>> We have a large number of ip phones connecting to this port. They could
>> be as large as 80k. They request for a file from a custom
>> application. haproxy front ends the tls connection and then forwards
>> the request to the application's http port.
>>
>
> Have you take a look into the link below for some tunings for the system
> and haproxy.
>
> HA-Proxy version 1.8.8 2018/04/19
>> Copyright 2000-2018 Willy Tarreau <[email protected]>
>>
>
> [snipp]
>
> Any change to update to 1.8.9?
>
> Thanks can you also send the "Anonymized haproxy conf".
> The main questions are do you use thread and or nbprocs?
> This will be answered by the conf
>
> Best regards
> aleks
>
>
> On Thu, Jun 7, 2018 at 2:13 PM, Aleksandar Lazic <[email protected]>
>> wrote:
>>
>> Hi Mihir.
>>>
>>> On 07/06/2018 10:27, Mihir Shirali wrote:
>>>
>>> Hi Team,
>>>>
>>>> We use haproxy to front tls for a large number of endpoints, haproxy
>>>> prcesses the TLS session and then forwards the request to the backend
>>>> application.
>>>>
>>>> What we have noticed is that if there are a large number of connections
>>>> from different clients - the CPU usage goes up significantly. This
>>>> primarily because haproxy is handling a lot ofSSL connections. I came
>>>> across 2 options above and tested them out.
>>>>
>>>>
>>> What do you mean with *large number*?
>>>
>>> https://medium.freecodecamp.org/how-we-fine-tuned-haproxy-to
>>> -achieve-2-000-000-concurrent-ssl-connections-d017e61a4d27
>>>
>>> With maxsslrate - CPU is better controlled and if I combine this with
>>>
>>>> 503 response in the front end I see great results. Is there a
>>>> possibility of connection timeout on the client here if there are a
>>>> very large number of requests?
>>>>
>>>> With maxsslconn, CPU is still pegged high - and clients receive a tcp
>>>> reset. This is also good, because there is no chance of tcp time out on
>>>> the client. Clients can retry after a bit and they are aware that the
>>>> connection is closed instead of waiting on timeout. However, CPU still
>>>> seems pegged high. What is the reason for high CPU on the server here -
>>>> Is it because SSL stack is still hit with this setting?
>>>>
>>>>
>>> SSL/TLS handling isn't that easy.
>>>
>>> Please can you share some more information's, because in the latest
>>> versions of haproxy are a lot optimisation's introduced also for TLS.
>>>
>>> haproxy -vv
>>>
>>> Anonymized haproxy conf.
>>>
>>> --
>>>
>>>> Regards,
>>>> Mihir
>>>>
>>>>
>>> Best regards
>>> Aleks
>>>
>>
>> --
>> Regards,
>> Mihir
>>
>


--
Regards,
Mihir
Aleksandar Lazic
Re: maxsslconn vs maxsslrate
June 09, 2018 09:10AM
Hi Mihir.

On 07/06/2018 14:47, Mihir Shirali wrote:
>Hi Alexander,
>
>I have looked at the link. What I am looking for is an answer to the
>difference between maxsslconn and maxsslrate. The former does not result in
>CPU savings while the latter does. Again the former does result in large
>number of tcp connection resets while the latter does not. What I'd like to
>know and understand is why that is the case.
>I am using nbproc set to 2.

As far as I understand the doc right makes that behaviour sense.

https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#3.2-maxconn
https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#3.2-maxsslconn
https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#3.2-maxsslrate

The sslconn limits **only** the connection rate, **not** the ssl
handshakes.

The sslrate limits the ssl handshakes.

To answer you original question means to me that you want to save some
CPU usage and therefor you should use maxsslrate.

Hth
Aleks


>On Thu, Jun 7, 2018 at 2:43 PM, Aleksandar Lazic <[email protected]> wrote:
>
>> On 07/06/2018 14:30, Mihir Shirali wrote:
>>
>>> We have a large number of ip phones connecting to this port. They could
>>> be as large as 80k. They request for a file from a custom
>>> application. haproxy front ends the tls connection and then forwards
>>> the request to the application's http port.
>>>
>>
>> Have you take a look into the link below for some tunings for the system
>> and haproxy.
>>
>> HA-Proxy version 1.8.8 2018/04/19
>>> Copyright 2000-2018 Willy Tarreau <[email protected]>
>>>
>>
>> [snipp]
>>
>> Any change to update to 1.8.9?
>>
>> Thanks can you also send the "Anonymized haproxy conf".
>> The main questions are do you use thread and or nbprocs?
>> This will be answered by the conf
>>
>> Best regards
>> aleks
>>
>>
>> On Thu, Jun 7, 2018 at 2:13 PM, Aleksandar Lazic <[email protected]>
>>> wrote:
>>>
>>> Hi Mihir.
>>>>
>>>> On 07/06/2018 10:27, Mihir Shirali wrote:
>>>>
>>>> Hi Team,
>>>>>
>>>>> We use haproxy to front tls for a large number of endpoints, haproxy
>>>>> prcesses the TLS session and then forwards the request to the backend
>>>>> application.
>>>>>
>>>>> What we have noticed is that if there are a large number of connections
>>>>> from different clients - the CPU usage goes up significantly. This
>>>>> primarily because haproxy is handling a lot ofSSL connections. I came
>>>>> across 2 options above and tested them out.
>>>>>
>>>>>
>>>> What do you mean with *large number*?
>>>>
>>>> https://medium.freecodecamp.org/how-we-fine-tuned-haproxy-to
>>>> -achieve-2-000-000-concurrent-ssl-connections-d017e61a4d27
>>>>
>>>> With maxsslrate - CPU is better controlled and if I combine this with
>>>>
>>>>> 503 response in the front end I see great results. Is there a
>>>>> possibility of connection timeout on the client here if there are a
>>>>> very large number of requests?
>>>>>
>>>>> With maxsslconn, CPU is still pegged high - and clients receive a tcp
>>>>> reset. This is also good, because there is no chance of tcp time out on
>>>>> the client. Clients can retry after a bit and they are aware that the
>>>>> connection is closed instead of waiting on timeout. However, CPU still
>>>>> seems pegged high. What is the reason for high CPU on the server here -
>>>>> Is it because SSL stack is still hit with this setting?
>>>>>
>>>>>
>>>> SSL/TLS handling isn't that easy.
>>>>
>>>> Please can you share some more information's, because in the latest
>>>> versions of haproxy are a lot optimisation's introduced also for TLS.
>>>>
>>>> haproxy -vv
>>>>
>>>> Anonymized haproxy conf.
>>>>
>>>> --
>>>>
>>>>> Regards,
>>>>> Mihir
>>>>>
>>>>>
>>>> Best regards
>>>> Aleks
>>>>
>>>
>>> --
>>> Regards,
>>> Mihir
>>>
>>
>
>
>--
>Regards,
>Mihir
Willy Tarreau
Re: maxsslconn vs maxsslrate
June 22, 2018 10:30AM
On Thu, Jun 07, 2018 at 02:47:01PM +0530, Mihir Shirali wrote:
> Hi Alexander,
>
> I have looked at the link. What I am looking for is an answer to the
> difference between maxsslconn and maxsslrate. The former does not result in
> CPU savings while the latter does.

That's expected :
- the former limits the number of concurrent connections : it will refuse
to allocate more SSL sessions than configured. The problem is that this
can only be done after the TCP connection is accepted, so extraneous
connections will be rejected instead of being delayed. This is used to
limit the amount of memory used by SSL.

- the latter limits the rate at which incoming connections are accepted on
all SSL listeners. Once the limit is reached, new connections will not be
accepted anymore until the average accept rate over the last sliding
second falls lower than this value. This effectively results in a
smoothing of the SSL traffic. For example if you accept only 200 conns/s
you'll ultimately see one connection accepted every 5 ms.

In both cases there are limitations though :
- with maxsslconn, some users will not be able to connect at all if the
limit is reached ;

- with maxsslrate, some users will experience a slowdown so that the max
rate you have configured is respected.

Overall you should use this with high enough values to protect your system
but not really for QoS.

Regards,
willy
Sorry, only registered users may post in this forum.

Click here to login