HAProxy for persistent TCP connections

Hello,
I am evaluating HAProxy for an application which utilizes persistent
TCP connections. We often go days, weeks, or even months without
causing a TCP disconnect/reconnect. Any issues I should be aware of
before I get too far down the path? This will just be straight TCP
load-balancing, no HTTP involved.

thanks.

Hello,

On Sun, Mar 07, 2010 at 08:46:08PM -0500, Geoffrey Mina wrote:
> Hello,
> I am evaluating HAProxy for an application which utilizes persistent
> TCP connections. We often go days, weeks, or even months without
> causing a TCP disconnect/reconnect. Any issues I should be aware of
> before I get too far down the path? This will just be straight TCP
> load-balancing, no HTTP involved.

Having some connections remain established for months is not an issue
at all. However you should still be careful about your timeouts, because
if a connection remains inactive for a long time, there is no was to
distinguish that from a frozen connection. So maybe you can set large
timeouts of 1 hour, 1 day or something like this in order to get rid
of dead connections without having to restart the process. The largest
timeout you can set is 24 days, due to the internal precision of one
millisecond.

You should also enable tcp keepalives (option tcpka) so that the
system regularly tests connections. By default, some systems send
them every 2 hours, and actively kill the connection after several
failures.

You should also be careful about your config changes, because if you
restart, you'll break your connections (unless you restart with -sf,
but it will be hard to troubleshoot when running multiple concurrent
processes).

Regards,
Willy

Great. None of this should be an issue. My application sends it's
own keepalive/heartbeat packets every 30-60 seconds. So, it sounds
like the timeout will only kick in if there is no activity on the
socket, correct? If that's the case, then I'll probably have fairly
short timeout settings, to ensure we don't have a bunch of garbage
connections up.

thanks.

On Mon, Mar 8, 2010 at 1:16 AM, Willy Tarreau <[email protected]> wrote:
> Hello,
>
> On Sun, Mar 07, 2010 at 08:46:08PM -0500, Geoffrey Mina wrote:
>> Hello,
>> I am evaluating HAProxy for an application which utilizes persistent
>> TCP connections.  We often go days, weeks, or even months without
>> causing a TCP disconnect/reconnect.  Any issues I should be aware of
>> before I get too far down the path?  This will just be straight TCP
>> load-balancing, no HTTP involved.
>
> Having some connections remain established for months is not an issue
> at all. However you should still be careful about your timeouts, because
> if a connection remains inactive for a long time, there is no was to
> distinguish that from a frozen connection. So maybe you can set large
> timeouts of 1 hour, 1 day or something like this in order to get rid
> of dead connections without having to restart the process. The largest
> timeout you can set is 24 days, due to the internal precision of one
> millisecond.
>
> You should also enable tcp keepalives (option tcpka) so that the
> system regularly tests connections. By default, some systems send
> them every 2 hours, and actively kill the connection after several
> failures.
>
> You should also be careful about your config changes, because if you
> restart, you'll break your connections (unless you restart with -sf,
> but it will be hard to troubleshoot when running multiple concurrent
> processes).
>
> Regards,
> Willy
>
>

On Tue, Mar 09, 2010 at 05:58:07PM -0500, Geoffrey Mina wrote:
> Great. None of this should be an issue. My application sends it's
> own keepalive/heartbeat packets every 30-60 seconds. So, it sounds
> like the timeout will only kick in if there is no activity on the
> socket, correct? If that's the case, then I'll probably have fairly
> short timeout settings, to ensure we don't have a bunch of garbage
> connections up.

yes indeed that's better that way. And it's nice to see that some
people still think about implementing application level keep-alives !

Willy

Willy,
I know you said that HAProxy would work just fine with persistent TCP
connections, unfortunately I am not seeing that behavior. We are
establishing a socket connection and sending application level "heart beat"
messages every 60 seconds. I am seeing that HAProxy is shutting down my
connection after a period of time. Attached is a pcap file of the
shutdown... from the HAProxy server. I have also included my configuration
below.

Any ideas on what's up here?


global
maxconn 4096 # Total Max Connections. This is dependent on
ulimit
daemon
nbproc 4 # Number of processing cores. Dual Dual-core Opteron
is 4 cores for example.
log 127.0.0.1 local0 debug

defaults
mode http
clitimeout 150000
srvtimeout 30000
contimeout 4000
log global
#option tcplog
#option httplog
#option httpclose # Disable Keepalive


listen services X.X.X.131:1312
mode tcp
balance roundrobin # Load Balancing algorithm
option tcpka
retries 3
## Define your servers to balance
server rs-webserver1 X.X.X.217:1312
server rs-webserver2 X.X.X.216:1312
server rs-webserver3 X.X.X.136:1312
server rs-webserver4 X.X.X.220:1312
server rs-webserver5 X.X.X.126:1312


listen stats :8080
mode http
stats uri /



Thanks!
Geoff

On Tue, Mar 9, 2010 at 6:03 PM, Willy Tarreau <[email protected]> wrote:

> On Tue, Mar 09, 2010 at 05:58:07PM -0500, Geoffrey Mina wrote:
> > Great. None of this should be an issue. My application sends it's
> > own keepalive/heartbeat packets every 30-60 seconds. So, it sounds
> > like the timeout will only kick in if there is no activity on the
> > socket, correct? If that's the case, then I'll probably have fairly
> > short timeout settings, to ensure we don't have a bunch of garbage
> > connections up.
>
> yes indeed that's better that way. And it's nice to see that some
> people still think about implementing application level keep-alives !
>
> Willy
>
>

Hi,

Le Mercredi 31 Mars 2010 23:33:05, Geoffrey Mina a écrit :
> Willy,
> I know you said that HAProxy would work just fine with persistent TCP
> connections, unfortunately I am not seeing that behavior. We are
> establishing a socket connection and sending application level "heart beat"
> messages every 60 seconds.

Isn't it every 30 seconds, with a "BEAT" message coming from the backend ?

> I am seeing that HAProxy is shutting down my
> connection after a period of time. Attached is a pcap file of the
> shutdown... from the HAProxy server. I have also included my configuration
> below.

If it's 30 seconds, then "srvtimeout 30000" is not a good value, as HAProxy can close the connection nearly at the same time the packet is received.
Can you try with a value a little bigger than the heart beat interval ?
And if it's 60 seconds, this timeout is definitly too low.

--
Cyril Bonté

Great. I guess I wasn't clear on what those actually did. I will try
increasing those numbers and see if things get better.

gracias!
Geoff

On Wed, Mar 31, 2010 at 6:03 PM, Cyril Bonté <[email protected]> wrote:

> Hi,
>
> Le Mercredi 31 Mars 2010 23:33:05, Geoffrey Mina a écrit :
> > Willy,
> > I know you said that HAProxy would work just fine with persistent TCP
> > connections, unfortunately I am not seeing that behavior. We are
> > establishing a socket connection and sending application level "heart
> beat"
> > messages every 60 seconds.
>
> Isn't it every 30 seconds, with a "BEAT" message coming from the backend ?
>
> > I am seeing that HAProxy is shutting down my
> > connection after a period of time. Attached is a pcap file of the
> > shutdown... from the HAProxy server. I have also included my
> configuration
> > below.
>
> If it's 30 seconds, then "srvtimeout 30000" is not a good value, as
> HAProxy can close the connection nearly at the same time the packet is
> received.
> Can you try with a value a little bigger than the heart beat interval ?
> And if it's 60 seconds, this timeout is definitly too low.
>
> --
> Cyril Bonté
>