Welcome! Log In Create A New Profile

Advanced

Force connection close after a haproxy reload

Posted by Robson Roberto Souza Peixoto 
Robson Roberto Souza Peixoto
Force connection close after a haproxy reload
March 14, 2017 02:20AM
Hi!

I'm using HaProxy as a reverse proxy to my applications running on Marathon.

The Marathon-lb(https://github.com/mesosphere/marathon-lb) is the
responsabel to create the configuration file and reload the HaProxy.

But there are a lot of thrift and another tcp services and because of that
the backend is configured as layer 4. All clients of these services are
using persistent connection and because of that we have a lot of process
waiting for all connections close to finish the process after a reload.

But sometimes it take days or weeks to hapen a new deploy of the software
and release the connection.

I'd like to know if is possible to force the connection close if the
process if waiting the connection to die.

Thanks!
--
Robson Roberto Souza Peixoto
Robinho
Master in Computer Science, University of Campinas
IRC: robsonpeixoto
Twitter: http://twitter.com/robinhopeixoto
github: https://github.com/robsonpeixoto
Willy Tarreau
Re: Force connection close after a haproxy reload
March 14, 2017 10:50AM
Hi Robinho,

On Tue, Mar 14, 2017 at 01:12:45AM +0000, Robson Roberto Souza Peixoto wrote:
> Hi!
>
> I'm using HaProxy as a reverse proxy to my applications running on Marathon.
>
> The Marathon-lb(https://github.com/mesosphere/marathon-lb) is the
> responsabel to create the configuration file and reload the HaProxy.
>
> But there are a lot of thrift and another tcp services and because of that
> the backend is configured as layer 4. All clients of these services are
> using persistent connection and because of that we have a lot of process
> waiting for all connections close to finish the process after a reload.
>
> But sometimes it take days or weeks to hapen a new deploy of the software
> and release the connection.
>
> I'd like to know if is possible to force the connection close if the
> process if waiting the connection to die.

In fact in HTTP it's already the case. In TCP however there's no way to
know when to close because by definition we're just forwarding bytes
between two sides without any interpretation of the transported protocol.
If you want to close such connections upon reload, you have two possibilities :
- shorten the timeouts (but how short is enough solely depends on your
workload and what works today may not work tomorrow) ;

- reload with -st instead of -sf : it will instruct the old process to
terminate all connections and quit instead of letting them close first.

Given your description I think the latter is exactly what you're looking for.

Cheers,
Willy
Robson Roberto Souza Peixoto
Re: Force connection close after a haproxy reload
March 14, 2017 12:20PM
On Tue, Mar 14, 2017 at 6:39 AM Willy Tarreau <[email protected]> wrote:

> Hi Robinho,
>
> On Tue, Mar 14, 2017 at 01:12:45AM +0000, Robson Roberto Souza Peixoto
> wrote:
> > Hi!
> >
> > I'm using HaProxy as a reverse proxy to my applications running on
> Marathon.
> >
> > The Marathon-lb(https://github.com/mesosphere/marathon-lb) is the
> > responsabel to create the configuration file and reload the HaProxy.
> >
> > But there are a lot of thrift and another tcp services and because of
> that
> > the backend is configured as layer 4. All clients of these services are
> > using persistent connection and because of that we have a lot of process
> > waiting for all connections close to finish the process after a reload.
> >
> > But sometimes it take days or weeks to hapen a new deploy of the software
> > and release the connection.
> >
> > I'd like to know if is possible to force the connection close if the
> > process if waiting the connection to die.
>
> In fact in HTTP it's already the case. In TCP however there's no way to
> know when to close because by definition we're just forwarding bytes
> between two sides without any interpretation of the transported protocol.
> If you want to close such connections upon reload, you have two
> possibilities :
> - shorten the timeouts (but how short is enough solely depends on your
> workload and what works today may not work tomorrow) ;
>
> - reload with -st instead of -sf : it will instruct the old process to
> terminate all connections and quit instead of letting them close first.
>
>
But will `-st` mode wait for current http requests finish? Or will
interrupt all connections without waiting for the responses?


> Given your description I think the latter is exactly what you're looking
> for.
>
> Cheers,
> Willy
>
--
Robson Roberto Souza Peixoto
Robinho
Master in Computer Science, University of Campinas
IRC: robsonpeixoto
Twitter: http://twitter.com/robinhopeixoto
github: https://github.com/robsonpeixoto
Willy Tarreau
Re: Force connection close after a haproxy reload
March 14, 2017 01:30PM
On Tue, Mar 14, 2017 at 11:16:26AM +0000, Robson Roberto Souza Peixoto wrote:
> But will `-st` mode wait for current http requests finish? Or will
> interrupt all connections without waiting for the responses?

It will interrupt them all but you said you were running TCP and not HTTP
so with TCP there's no notion of end of request nor response.

Willy
Robson Roberto Souza Peixoto
Re: Force connection close after a haproxy reload
March 14, 2017 01:50PM
On Tue, 14 Mar 2017 at 09:20 Willy Tarreau <[email protected]> wrote:

> On Tue, Mar 14, 2017 at 11:16:26AM +0000, Robson Roberto Souza Peixoto
> wrote:
> > But will `-st` mode wait for current http requests finish? Or will
> > interrupt all connections without waiting for the responses?
>
> It will interrupt them all but you said you were running TCP and not HTTP
> so with TCP there's no notion of end of request nor response.
>

Sorry, my bad! I didn't put all information. On the same haproxy we have
all kind of applications:
- redis
- websocket
- http
- thrift
....

If it interrupt the connection without wait for http responses from the
current connection, maybe I'll create a instance for layer4 and another one
for layer7 proxy.

One more question. What do you think about add a parameter that wait for
layer7 requests ends and close all layer4 connections?

Willy thanks to help me again.


> Willy
>
--
Robson Roberto Souza Peixoto
Robinho
Master in Computer Science, University of Campinas
IRC: robsonpeixoto
Twitter: http://twitter.com/robinhopeixoto
github: https://github.com/robsonpeixoto
Cyril Bonté
Re: Force connection close after a haproxy reload
March 15, 2017 11:50AM
Hi all,

> De: "Willy Tarreau" <[email protected]>
> À: "Robson Roberto Souza Peixoto" <[email protected]>
> Cc: haproxy@formilux.org
> Envoyé: Mardi 14 Mars 2017 13:20:46
> Objet: Re: Force connection close after a haproxy reload
>
> On Tue, Mar 14, 2017 at 11:16:26AM +0000, Robson Roberto Souza
> Peixoto wrote:
> > But will `-st` mode wait for current http requests finish? Or will
> > interrupt all connections without waiting for the responses?
>
> It will interrupt them all but you said you were running TCP and not
> HTTP
> so with TCP there's no notion of end of request nor response.

As a reminder (to me), I sent a patch in december (just before the 1.7.0 release), which immediately closes the HTTP keep-alived connections. Currently, during the soft stop, HTTP connections are only closed when a request is processed, it doesn't do anything on connections already in an idle state.
I didn't spend more time on it but having a quick look at it, it may be ready to merge soon.

About TCP connections, while I wrote the patch, I was thinking about a global "grace timeout", which will enforce haproxy exit if the soft stop takes too long (for example when tcp connections don't expire). Something like :

global
grace 30s

Would it be something acceptable Willy ?

Cheers,
Cyril Bonté
Pavlos Parissis
Re: Force connection close after a haproxy reload
March 15, 2017 12:10PM
On 15/03/2017 11:48 πμ, Cyril Bonté wrote:
> Hi all,
>
>> De: "Willy Tarreau" <[email protected]> À: "Robson Roberto Souza Peixoto"
>> <[email protected]> Cc: haproxy@formilux.org Envoyé: Mardi 14 Mars
>> 2017 13:20:46 Objet: Re: Force connection close after a haproxy reload
>>
>> On Tue, Mar 14, 2017 at 11:16:26AM +0000, Robson Roberto Souza Peixoto
>> wrote:
>>> But will `-st` mode wait for current http requests finish? Or will
>>> interrupt all connections without waiting for the responses?
>>
>> It will interrupt them all but you said you were running TCP and not HTTP
>> so with TCP there's no notion of end of request nor response.
>
> As a reminder (to me), I sent a patch in december (just before the 1.7.0
> release), which immediately closes the HTTP keep-alived connections.
> Currently, during the soft stop, HTTP connections are only closed when a
> request is processed, it doesn't do anything on connections already in an
> idle state. I didn't spend more time on it but having a quick look at it, it
> may be ready to merge soon.
>

What I have observed in production is that in HTTP mode the timeout settings
(client, http-requests) contribute to how long older process stay around. Longer
timeouts makes older processes to stay around longer time.
Having said this, it all depends to the traffic composition. For instance, when
you have radio streaming then older processes stay for days regardless to
timeout settings.


> About TCP connections, while I wrote the patch, I was thinking about a global
> "grace timeout", which will enforce haproxy exit if the soft stop takes too
> long (for example when tcp connections don't expire). Something like :
>
> global grace 30s

This is a very interesting approach, as it solves the issue of old processes
staying around for days for TCP mode. +1 from me.


Cheers,
Pavlos
Willy Tarreau
Re: Force connection close after a haproxy reload
March 15, 2017 12:10PM
Hi Cyril!

On Wed, Mar 15, 2017 at 11:48:01AM +0100, Cyril Bonté wrote:
> As a reminder (to me), I sent a patch in december (just before the 1.7.0
> release), which immediately closes the HTTP keep-alived connections.
> Currently, during the soft stop, HTTP connections are only closed when a
> request is processed, it doesn't do anything on connections already in an
> idle state.

Ah yes I vaguely remember about this discussion now.

> I didn't spend more time on it but having a quick look at it, it may be ready
> to merge soon.

Cool!

> About TCP connections, while I wrote the patch, I was thinking about a global
> "grace timeout", which will enforce haproxy exit if the soft stop takes too
> long (for example when tcp connections don't expire). Something like :
>
> global
> grace 30s

It's pretty similar to what we have with the "grace" setting in frontends
which maintains the listener alive for some time and automatically shuts
it down. And yes I think it does make sense because in the end people
complain about "too old" processes.

Please note, if you go down that route, you don't need to kill idle sessions
upon signal receipt anymore, all of them should be dealt with by this grace
period. That will make http quit even more cleanly as well.

In the past I thought about shortening the client/server timeouts upon
reload, or switching them to client-fin and server-fin only. But that
wouldn't work because some connections can remain very active for a long
time and would never die.

> Would it be something acceptable Willy ?

Yes I think that this grace setting would solve all use cases and avoid
having to kill each and every pending connection (they would simply go
away with the process). And since it's a new setting we could possibly
even imagine backporting it to 1.7 if there's strong demand and it's
riskless.

Cheers,
Willy
Dave Cottlehuber
Re: Force connection close after a haproxy reload
March 15, 2017 04:10PM
On Wed, 15 Mar 2017, at 12:02, Willy Tarreau wrote:
> Hi Cyril!
>
> On Wed, Mar 15, 2017 at 11:48:01AM +0100, Cyril Bonté wrote:
> > As a reminder (to me), I sent a patch in december (just before the 1.7.0
> > release), which immediately closes the HTTP keep-alived connections.
> > Currently, during the soft stop, HTTP connections are only closed when a
> > request is processed, it doesn't do anything on connections already in an
> > idle state.
>
> Ah yes I vaguely remember about this discussion now.
>
> > I didn't spend more time on it but having a quick look at it, it may be ready
> > to merge soon.
>
> Cool!
>
> > About TCP connections, while I wrote the patch, I was thinking about a global
> > "grace timeout", which will enforce haproxy exit if the soft stop takes too
> > long (for example when tcp connections don't expire). Something like :
> >
> > global
> > grace 30s

Yes please. I have a reasonable number of websocket connections that
run for hours or days. I'd much prefer having an operational guarantee
that
a restart/reload will take no longer than 5 minutes by which time all of
the
transactional HTTP-only (non-upgraded) connections will have been long
time.

A+
Dave
Sorry, only registered users may post in this forum.

Click here to login