Welcome! Log In Create A New Profile

Advanced

Peer tables don't synch on clear

Posted by Franks Andy (IT Technical Architecture Manager) 
Franks Andy (IT Technical Architecture Manager)
Peer tables don't synch on clear
February 08, 2018 11:30AM
Hi all,
Haproxy 1.6.13
I've checked the documentation again but can't see an option for this.
We sometimes clear backup path server use for individual connections and whilst the peers synchronisation works for new connections, it doesn't clear on the secondary peer node we're using.
Is this by design or an option I'm not seeing?
Thanks
Andy
Frederic Lecaille
Re: Peer tables don't synch on clear
February 12, 2018 02:00PM
On 02/08/2018 11:22 AM, Franks Andy (IT Technical Architecture Manager)
wrote:
> Hi all,

Hello Franks,

>   Haproxy 1.6.13
>
>   I’ve checked the documentation again but can’t see an option for this.
>
> We sometimes clear backup path server use for individual connections and
> whilst the peers synchronisation works for new connections, it doesn’t
> clear on the secondary peer node we’re using.
>
> Is this by design or an option I’m not seeing?

Please give us more information about your configuration. If possible,
also provide us with the information of stick-table entries concerned
with this issue (see "show table" CLI command).

Do not forget to obfuscate the critical data.

Regards,

Fred.
Franks Andy (IT Technical Architecture Manager)
RE: Peer tables don't synch on clear
February 12, 2018 04:40PM
Hi Fred,
Thanks for the reply.
I have two peers synchronising (we use keepalived over the two to control which is live).

HAProxy config:

peers lb_replication
peer server1 10.128.176.141:1024
peer server2 10.128.176.142:1024

backend sourceaddr
stick-table type ip size 10240k expire 30m peers lb_replication

frontend ft_web_ssl
bind 0.0.0.0:443 name https ssl crt /etc/haproxy/certs/main.pem
mode http
option httplog

acl is_from_outside src 192.168.110.0/24
acl is_empty_path path /
acl is_webmail hdr(host) -i webmail
acl is_webmail_fqdn hdr(host) -i webmail.domain

redirect location /owa/ code 302 if is_webmail is_empty_path ! is_from_outside
redirect location /owa/ code 302 if is_webmail_fqdn is_empty_path ! is_from_outside
default_backend bk_web_ssl

backend bk_web_ssl
mode http
option httplog
cookie SERVERID insert nocache indirect
stick on src table sourceaddr
server server1 10.128.176.150:443 check ssl
server server2 10.51.0.150:443 check ssl backup

It's fine for new connections - it records the correct server1/server2 information. It's hard to demonstrate, but I can see when I use haproxyctl to clear an entry :

Haproxyctl clear table sourceaddr key <key>

... it doesn't clear the secondary node entry. When that entry for the client re-presents the expiry time on the secondary updates but the entry never clears.

I can't really include pictures on these emails, but the tables are kind of standard:

e.g.

0x7fa8b247a4f4: key=217.40.203.34 use=0 exp=1574957 server_id=1

Thanks
Andy

-----Original Message-----
From: Frederic Lecaille [mailto:[email protected]]
Sent: 12 February 2018 12:56
To: Franks Andy (IT Technical Architecture Manager); '[email protected]'
Subject: Re: Peer tables don't synch on clear

On 02/08/2018 11:22 AM, Franks Andy (IT Technical Architecture Manager)
wrote:
> Hi all,

Hello Franks,

>   Haproxy 1.6.13
>
>   I've checked the documentation again but can't see an option for this.
>
> We sometimes clear backup path server use for individual connections and
> whilst the peers synchronisation works for new connections, it doesn't
> clear on the secondary peer node we're using.
>
> Is this by design or an option I'm not seeing?

Please give us more information about your configuration. If possible,
also provide us with the information of stick-table entries concerned
with this issue (see "show table" CLI command).

Do not forget to obfuscate the critical data.

Regards,

Fred.
Frederic Lecaille
Re: Peer tables don't synch on clear
February 13, 2018 08:40AM
On 02/12/2018 04:28 PM, Franks Andy (IT Technical Architecture Manager)
wrote:
> Hi Fred,

Hi Franks,

Please bottom post when you reply.

> Thanks for the reply.
> I have two peers synchronising (we use keepalived over the two to control which is live).
>
> HAProxy config:
>
> peers lb_replication
> peer server1 10.128.176.141:1024
> peer server2 10.128.176.142:1024
>
> backend sourceaddr
> stick-table type ip size 10240k expire 30m peers lb_replication
>
> frontend ft_web_ssl
> bind 0.0.0.0:443 name https ssl crt /etc/haproxy/certs/main.pem
> mode http
> option httplog
>
> acl is_from_outside src 192.168.110.0/24
> acl is_empty_path path /
> acl is_webmail hdr(host) -i webmail
> acl is_webmail_fqdn hdr(host) -i webmail.domain
>
> redirect location /owa/ code 302 if is_webmail is_empty_path ! is_from_outside
> redirect location /owa/ code 302 if is_webmail_fqdn is_empty_path ! is_from_outside
> default_backend bk_web_ssl
>
> backend bk_web_ssl
> mode http
> option httplog
> cookie SERVERID insert nocache indirect
> stick on src table sourceaddr
> server server1 10.128.176.150:443 check ssl
> server server2 10.51.0.150:443 check ssl backup
>
> It's fine for new connections - it records the correct server1/server2 information. It's hard to demonstrate, but I can see when I use haproxyctl to clear an entry :
>
> Haproxyctl clear table sourceaddr key <key>

Haproxy stick-table are synchronized between peers but only to create or
update entries. The deletions are not synchronized.

The stick-table synchronizations are performed thanks to peers protocol
(see doc/peers* files). There is nothing in this protocol which
synchronize the deletions.

So you cannot reproduce your issue with haproxyctl.

The stick-table entries are cleared when they expire (exp == 0) and when
there is no more usage of these entries (use == 0). As the expiry values
are synchronized,the stick-table are supposed to be purged at almost the
same time.

> .. it doesn't clear the secondary node entry. When that entry for the client re-presents the expiry time on the secondary updates but the entry never clears.
>
> I can't really include pictures on these emails, but the tables are kind of standard:
>
> e.g.
>
> 0x7fa8b247a4f4: key=217.40.203.34 use=0 exp=1574957 server_id=1
>
> Thanks
> Andy
>
> -----Original Message-----
> From: Frederic Lecaille [mailto:[email protected]]
> Sent: 12 February 2018 12:56
> To: Franks Andy (IT Technical Architecture Manager); '[email protected]'
> Subject: Re: Peer tables don't synch on clear
>
> On 02/08/2018 11:22 AM, Franks Andy (IT Technical Architecture Manager)
> wrote:
>> Hi all,
>
> Hello Franks,
>
>>   Haproxy 1.6.13
>>
>>   I've checked the documentation again but can't see an option for this.
>>
>> We sometimes clear backup path server use for individual connections and
>> whilst the peers synchronisation works for new connections, it doesn't
>> clear on the secondary peer node we're using.
>>
>> Is this by design or an option I'm not seeing?
>
> Please give us more information about your configuration. If possible,
> also provide us with the information of stick-table entries concerned
> with this issue (see "show table" CLI command).
>
> Do not forget to obfuscate the critical data.
>
> Regards,
>
> Fred.
>
>
>
Franks Andy (IT Technical Architecture Manager)
RE: Peer tables don't synch on clear
February 13, 2018 12:10PM
Thanks for the update,
Looks like I need to clear from both nodes simultaneously then, or use the option to shut down connections on return of the non-backup server(s).
Thanks again
Andy

-----Original Message-----
From: Frederic Lecaille [mailto:[email protected]]
Sent: 13 February 2018 07:35
To: Franks Andy (IT Technical Architecture Manager); '[email protected]'
Subject: Re: Peer tables don't synch on clear

On 02/12/2018 04:28 PM, Franks Andy (IT Technical Architecture Manager)
wrote:
> Hi Fred,

Hi Franks,

Please bottom post when you reply.

> Thanks for the reply.
> I have two peers synchronising (we use keepalived over the two to control which is live).
>
> HAProxy config:
>
> peers lb_replication
> peer server1 10.128.176.141:1024
> peer server2 10.128.176.142:1024
>
> backend sourceaddr
> stick-table type ip size 10240k expire 30m peers lb_replication
>
> frontend ft_web_ssl
> bind 0.0.0.0:443 name https ssl crt /etc/haproxy/certs/main.pem
> mode http
> option httplog
>
> acl is_from_outside src 192.168.110.0/24
> acl is_empty_path path /
> acl is_webmail hdr(host) -i webmail
> acl is_webmail_fqdn hdr(host) -i webmail.domain
>
> redirect location /owa/ code 302 if is_webmail is_empty_path ! is_from_outside
> redirect location /owa/ code 302 if is_webmail_fqdn is_empty_path ! is_from_outside
> default_backend bk_web_ssl
>
> backend bk_web_ssl
> mode http
> option httplog
> cookie SERVERID insert nocache indirect
> stick on src table sourceaddr
> server server1 10.128.176.150:443 check ssl
> server server2 10.51.0.150:443 check ssl backup
>
> It's fine for new connections - it records the correct server1/server2 information. It's hard to demonstrate, but I can see when I use haproxyctl to clear an entry :
>
> Haproxyctl clear table sourceaddr key <key>

Haproxy stick-table are synchronized between peers but only to create or
update entries. The deletions are not synchronized.

The stick-table synchronizations are performed thanks to peers protocol
(see doc/peers* files). There is nothing in this protocol which
synchronize the deletions.

So you cannot reproduce your issue with haproxyctl.

The stick-table entries are cleared when they expire (exp == 0) and when
there is no more usage of these entries (use == 0). As the expiry values
are synchronized,the stick-table are supposed to be purged at almost the
same time.

> .. it doesn't clear the secondary node entry. When that entry for the client re-presents the expiry time on the secondary updates but the entry never clears.
>
> I can't really include pictures on these emails, but the tables are kind of standard:
>
> e.g.
>
> 0x7fa8b247a4f4: key=217.40.203.34 use=0 exp=1574957 server_id=1
>
> Thanks
> Andy
>
> -----Original Message-----
> From: Frederic Lecaille [mailto:[email protected]]
> Sent: 12 February 2018 12:56
> To: Franks Andy (IT Technical Architecture Manager); '[email protected]'
> Subject: Re: Peer tables don't synch on clear
>
> On 02/08/2018 11:22 AM, Franks Andy (IT Technical Architecture Manager)
> wrote:
>> Hi all,
>
> Hello Franks,
>
>>   Haproxy 1.6.13
>>
>>   I've checked the documentation again but can't see an option for this.
>>
>> We sometimes clear backup path server use for individual connections and
>> whilst the peers synchronisation works for new connections, it doesn't
>> clear on the secondary peer node we're using.
>>
>> Is this by design or an option I'm not seeing?
>
> Please give us more information about your configuration. If possible,
> also provide us with the information of stick-table entries concerned
> with this issue (see "show table" CLI command).
>
> Do not forget to obfuscate the critical data.
>
> Regards,
>
> Fred.
>
>
>
Willy Tarreau
Re: Peer tables don't synch on clear
February 19, 2018 08:40AM
On Tue, Feb 13, 2018 at 11:00:08AM +0000, Franks Andy (IT Technical Architecture Manager) wrote:
> Thanks for the update,
> Looks like I need to clear from both nodes simultaneously then, or use the
> option to shut down connections on return of the non-backup server(s).

If you only need to clear to kill stickiness, instead of clearing you can
simply modify the "serverid" data. Just put an invalid value into it, the
change should be propagated, and it will kill stickiness.

Hoping this helps,
Willy
Sorry, only registered users may post in this forum.

Click here to login