Welcome! Log In Create A New Profile

Advanced

HAProxy's health checks and maxconn limits

Posted by Jiri Mencak 
Jiri Mencak
HAProxy's health checks and maxconn limits
January 04, 2017 11:50AM
Hi,

we are using HAProxy with its default 2000 maxconn limit and a listen block:

listen stats :1936
mode http
monitor-uri /healthz

which we use to check HAProxy's "health" by external HTTP probes. The
behaviour I'm seeing is that once the default 2000 maxconn limit is
reached, HAProxy stops listening for new connections and these are queued
in the kernel. This includes the ":1936/healthz" probes which timeout and
HAProxy's state is being interpreted as "unhealthy", rather than "busy".
While we can health-check HAProxy in other ways (e.g. stats UNIX domain
socket in stream mode), HTTP health-checks are preferred since they better
reflect HAProxy's ability to process requests.

As far as I can see, staying with the HTTP probe model, we can increase the
global maxconn limit or/and increase the health-check's timeout period.

Do you see any other options? What are the dangers of setting the global
maxconn limits really high apart from increased HAProxy's memory usage?
I've seen reports https://www.linangran.com/?p=547 of people going as
high as maxconn=2000000 and achieving 300k concurrent TCP connections on a
reportedly outdated PC.

Many thanks.

Jiri
Lukas Tribus
Re: HAProxy's health checks and maxconn limits
January 05, 2017 01:10AM
Hi Jiri,


Am 04.01.2017 um 11:38 schrieb Jiri Mencak:
> Hi,
>
> we are using HAProxy with its default 2000 maxconn limit and a listen
> block:
>
> listen stats :1936
> mode http
> monitor-uri /healthz
>
> which we use to check HAProxy's "health" by external HTTP probes. The
> behaviour I'm seeing is that once the default 2000 maxconn limit is
> reached,
> HAProxy stops listening
> [...]
> As far as I can see, staying with the HTTP probe model, we can
> increase the
> global maxconn limit or/and increase the health-check's timeout period.
>
> Do you see any other options?

Increasing global maxconn is the correct thing to do.

Here's why:
They are multiple maxconn "levels" that work independently from each other.

We have maxconn at global (process) level, at listen/frontend level, and
server level.


What you wanna do is make sure that the queuing happens in the lower
tiers first:
1. maxconn at server level is exhausted, haproxy will queue or
redispatch to another
backend server
2. maxconn at bind/frontend/listen level is exhausted, haproxy will stop
accepting
new connections and the kernel will queue for this particular level
(e.g. the specific frontend)
3. maxconn at the process level is exhausted, the kernel will queue for
all frontends
on this kernel of the specific process

The reason is impact. If one frontend/listen section is overloaded and
we are hitting its "own"
maxconn, other sections remain unaffected (your http probes in the
listen section for example).

But if the global maxconn is exhausted, everything is gonna be affected
(which is exactly what
you are seeing).



> What are the dangers of setting the global maxconn limits really high
> apart
> from increased HAProxy's memory usage?

Its really only memory (haproxy and kernel). But that also means that
when you
go beyond what your box can handle in ram and swap, you gonna get haproxy
OOM-killed.

If you have 1GB of memory to spare on this box, you can set global maxconn
to 20000 without any problems (estimate from the docs). And because each
listen/frontend section is still bound to the per section maxconn
default (2000),
your monitor section won't be affected by exhaustion in the other sections.

Consider tuning per listen/frontend maxconn as well.

maxconn is the single most import performance setting in haproxy, and it
should
be adjusted for each use case, considering available memory.


Also see:
https://cbonte.github.io/haproxy-dconv/1.6/configuration.html#4.2-maxconn



Regards,

Lukas
Sorry, only registered users may post in this forum.

Click here to login