Welcome! Log In Create A New Profile

Advanced

haproxy with keepalived

Posted by Esteban Torres Rodríguez 
Esteban Torres Rodríguez
haproxy with keepalived
March 19, 2012 08:10PM
Maybe I have to ask this in the keepalived list.

I have 2 servers configured in debian with keepalived and haproxy.

Today I found that both servers are running.

Keepalived is assumed that this active-passive, but ......

When I stop a server, the other works perfectly!

I followed this manual:

http://www.howtoforge.com/setting-up-a-high-availability-load-balancer-with-haproxy-keepalived-on-debian-lenny-p2

Has anyone been the same?

In /var/log/messages:

Keepalived_healthcheckers: Registering Kernel netlink command channel
Keepalived_healthcheckers: Opening file '/etc/keepalived/keepalived.conf'.
Keepalived_healthcheckers: Configuration is using: 3739 Bytes
Keepalived_vrrp: IPVS: Can not initialize IPVS: Protocol not available
Keepalived_healthcheckers: Using kernel netlink reflector LinkWatch ...
Keepalived_vrrp: Opening file '/etc/keepalived/keepalived.conf'.
Keepalived_vrrp: Configuration is using: 34986 Bytes
Keepalived_vrrp: Using kernel netlink reflector LinkWatch ...
Keepalived_vrrp: VRRP_Instance (VI_1) Entering MASTER STATE
Sander Klein
Re: haproxy with keepalived
March 20, 2012 10:20AM
Hey Esteban,

Your config looks good to me.

Sometimes it can happen that during failover not all servers receive
the gratuitous arp and they keep sending traffic to the backup router.

I normally force another failover to force another gratuitous arp get
it working again. It shouldn't happen to often tho....

Greets,

Sander
Willy Tarreau
Re: haproxy with keepalived
March 21, 2012 07:50AM
On Mon, Mar 19, 2012 at 08:08:37PM +0100, Esteban Torres Rodríguez wrote:
> Maybe I have to ask this in the keepalived list.
>
> I have 2 servers configured in debian with keepalived and haproxy.
>
> Today I found that both servers are running.

You mean that both servers own the VIP ?

Willy
Esteban Torres Rodríguez
Re: haproxy with keepalived
March 21, 2012 09:20AM
El día 21 de marzo de 2012 07:39, Willy Tarreau <[email protected]> escribió:
> On Mon, Mar 19, 2012 at 08:08:37PM +0100, Esteban Torres Rodríguez wrote:
>> Maybe I have to ask this in the keepalived list.
>>
>> I have 2 servers configured in debian with keepalived and haproxy.
>>
>> Today I found that both servers are running.
>
> You mean that both servers own the VIP ?

Of the 6 ip, 4 are answered by the master and 2 backup. it is dynamic.
Ip are exchanged randomly.
The server has 4 network cards. Is it of significance that has 3 virtual ip?


vrrp_script chk_haproxy {
script "killall -0 haproxy"
interval 2
weight 2
}

vrrp_instance VI_1 {
interface eth0
state MASTER
virtual_router_id 51
priority 101
virtual_ipaddress {
10.239.212.28
10.239.212.30
10.239.212.58
10.239.212.59
10.239.212.60
10.239.212.77
}
track_script {
chk_haproxy
}
}



>
> Willy
>
Willy Tarreau
Re: haproxy with keepalived
March 24, 2012 10:00PM
On Wed, Mar 21, 2012 at 09:15:16AM +0100, Esteban Torres Rodríguez wrote:
> El día 21 de marzo de 2012 07:39, Willy Tarreau <[email protected]> escribió:
> > On Mon, Mar 19, 2012 at 08:08:37PM +0100, Esteban Torres Rodríguez wrote:
> >> Maybe I have to ask this in the keepalived list.
> >>
> >> I have 2 servers configured in debian with keepalived and haproxy.
> >>
> >> Today I found that both servers are running.
> >
> > You mean that both servers own the VIP ?
>
> Of the 6 ip, 4 are answered by the master and 2 backup. it is dynamic.
> Ip are exchanged randomly.

Is this what you're observing or what you want ? Also, my concerns were about
finding some IPs at the two places at once, which fortunately is not the case !

> The server has 4 network cards. Is it of significance that has 3 virtual ip?

No, it should be unrelated. Are the NICs on the same network ? If so I wonder
what happens when keepalived receives its own packets via another NIC, maybe
it forces a re-election but I may be wrong, Alex would know that much better
than me !

> vrrp_script chk_haproxy {
> script "killall -0 haproxy"
> interval 2
> weight 2
> }
>
> vrrp_instance VI_1 {
> interface eth0
> state MASTER
> virtual_router_id 51
> priority 101
> virtual_ipaddress {
> 10.239.212.28
> 10.239.212.30
> 10.239.212.58
> 10.239.212.59
> 10.239.212.60
> 10.239.212.77
> }
> track_script {
> chk_haproxy
> }
> }

I really see nothing wrong here, nor anything which could explain how only
some of the addresses would be added to an interface !

Regards,
Willy
Esteban Torres Rodríguez
Re: haproxy with keepalived
March 25, 2012 05:10PM
El día 24 de marzo de 2012 21:51, Willy Tarreau <[email protected]> escribió:
> On Wed, Mar 21, 2012 at 09:15:16AM +0100, Esteban Torres Rodríguez wrote:
>> El día 21 de marzo de 2012 07:39, Willy Tarreau <[email protected]> escribió:
>> > On Mon, Mar 19, 2012 at 08:08:37PM +0100, Esteban Torres Rodríguez wrote:
>> >> Maybe I have to ask this in the keepalived list.
>> >>
>> >> I have 2 servers configured in debian with keepalived and haproxy.
>> >>
>> >> Today I found that both servers are running.
>> >
>> > You mean that both servers own the VIP ?
>>
>> Of the 6 ip, 4 are answered by the master and 2 backup. it is dynamic.
>> Ip are exchanged randomly.
>
> Is this what you're observing or what you want ? Also, my concerns were about
> finding some IPs at the two places at once, which fortunately is not the case !
>
>> The server has 4 network cards. Is it of significance that has 3 virtual ip?
>
> No, it should be unrelated. Are the NICs on the same network ? If so I wonder
> what happens when keepalived receives its own packets via another NIC, maybe
> it forces a re-election but I may be wrong, Alex would know that much better
> than me !

both servers are virtual with vmware. Each server has 4 physical cards
and 3 virtual (1 physical server for management, 3 physical and 3
virtual VIP).

perhaps it is vmware management making changes ownership the nic.


>
>> vrrp_script chk_haproxy {
>>       script "killall -0 haproxy"
>>       interval 2
>>       weight 2
>> }
>>
>> vrrp_instance VI_1 {
>>       interface eth0
>>       state MASTER
>>       virtual_router_id 51
>>       priority 101
>>       virtual_ipaddress {
>>               10.239.212.28
>>               10.239.212.30
>>               10.239.212.58
>>               10.239.212.59
>>               10.239.212.60
>>               10.239.212.77
>>       }
>>       track_script {
>>               chk_haproxy
>>       }
>> }
>
> I really see nothing wrong here, nor anything which could explain how only
> some of the addresses would be added to an interface !
>
> Regards,
> Willy
>
Willy Tarreau
Re: haproxy with keepalived
March 25, 2012 09:30PM
On Sun, Mar 25, 2012 at 05:00:53PM +0200, Esteban Torres Rodríguez wrote:
> El día 24 de marzo de 2012 21:51, Willy Tarreau <[email protected]> escribió:
> > On Wed, Mar 21, 2012 at 09:15:16AM +0100, Esteban Torres Rodríguez wrote:
> >> El día 21 de marzo de 2012 07:39, Willy Tarreau <[email protected]> escribió:
> >> > On Mon, Mar 19, 2012 at 08:08:37PM +0100, Esteban Torres Rodríguez wrote:
> >> >> Maybe I have to ask this in the keepalived list.
> >> >>
> >> >> I have 2 servers configured in debian with keepalived and haproxy.
> >> >>
> >> >> Today I found that both servers are running.
> >> >
> >> > You mean that both servers own the VIP ?
> >>
> >> Of the 6 ip, 4 are answered by the master and 2 backup. it is dynamic.
> >> Ip are exchanged randomly.
> >
> > Is this what you're observing or what you want ? Also, my concerns were about
> > finding some IPs at the two places at once, which fortunately is not the case !
> >
> >> The server has 4 network cards. Is it of significance that has 3 virtual ip?
> >
> > No, it should be unrelated. Are the NICs on the same network ? If so I wonder
> > what happens when keepalived receives its own packets via another NIC, maybe
> > it forces a re-election but I may be wrong, Alex would know that much better
> > than me !
>
> both servers are virtual with vmware. Each server has 4 physical cards
> and 3 virtual (1 physical server for management, 3 physical and 3
> virtual VIP).
>
> perhaps it is vmware management making changes ownership the nic.

I really have no idea. That's the beauty of virtualization, adding (clouds
of) smoke between where the problems occur and where they are observed.
What's funny is that the cost of debugging these issues is *much* higher
than the cost of the server that was saved by the operation :-)

The only thing you can do at the moment is to try to stabilize the lower
layers and ensure they're not doing any more magics in your back.

Regards,
Willy
Jens Dueholm Christensen (JEDC)
RE: haproxy with keepalived
March 27, 2012 09:50AM
Hi

I've got a setup with haproxy and keepalived in front handling ~10 IP's.

When I was testing my setup (watching with tcpdump etc) I saw a strange behaviour, that I eventually found a solution to a long while ago.

A bit of googling today lead me to answer 2 (answered Mar 16 '11 at 6:12) on this page: http://serverfault.com/questions/247472/arp-replies-contain-wrong-mac-address

For my setup to work correctly - ie. have the right NIC send out the Gratuitous ARP packet - I'm doing this in my iptables-based firewall script:

for interface in /proc/sys/net/ipv4/conf/*/arp_filter; do
/bin/echo "1" > ${interface}
done

The documentation for arp_filter says that 0 (the default value) "usually makes sense", but in multi-home and failover senarios it does not make sense at all!
Before reading the documentation I had never thought it possible that an arp request comming in through NIC 1 could be answered by NIC 2, but alas..

Before changing that I had a 1-in-6 (I've got 6 NIC's in this machine - 2 bundled for failover to 3 different networks) chance of the GARP-packet beeing sent out correctly and failover working as intended.

Oh, and I should mention that my firewall-script already did contain the mentioned rp_filter "fix".

for interface in /proc/sys/net/ipv4/conf/*/rp_filter; do
/bin/echo "0" > ${interface}
done

Regards,
Jens Dueholm Christensen 
Rambøll Survey IT

-----Original Message-----
From: Willy Tarreau [mailto:[email protected]]
Sent: Saturday, March 24, 2012 9:51 PM
To: Esteban Torres Rodríguez
Cc: haproxy@formilux.org
Subject: Re: haproxy with keepalived

On Wed, Mar 21, 2012 at 09:15:16AM +0100, Esteban Torres Rodríguez wrote:
> El día 21 de marzo de 2012 07:39, Willy Tarreau <[email protected]> escribió:
> > On Mon, Mar 19, 2012 at 08:08:37PM +0100, Esteban Torres Rodríguez wrote:
> >> Maybe I have to ask this in the keepalived list.
> >>
> >> I have 2 servers configured in debian with keepalived and haproxy.
> >>
> >> Today I found that both servers are running.
> >
> > You mean that both servers own the VIP ?
>
> Of the 6 ip, 4 are answered by the master and 2 backup. it is dynamic.
> Ip are exchanged randomly.

Is this what you're observing or what you want ? Also, my concerns were about
finding some IPs at the two places at once, which fortunately is not the case !

> The server has 4 network cards. Is it of significance that has 3 virtual ip?

No, it should be unrelated. Are the NICs on the same network ? If so I wonder
what happens when keepalived receives its own packets via another NIC, maybe
it forces a re-election but I may be wrong, Alex would know that much better
than me !

> vrrp_script chk_haproxy {
> script "killall -0 haproxy"
> interval 2
> weight 2
> }
>
> vrrp_instance VI_1 {
> interface eth0
> state MASTER
> virtual_router_id 51
> priority 101
> virtual_ipaddress {
> 10.239.212.28
> 10.239.212.30
> 10.239.212.58
> 10.239.212.59
> 10.239.212.60
> 10.239.212.77
> }
> track_script {
> chk_haproxy
> }
> }

I really see nothing wrong here, nor anything which could explain how only
some of the addresses would be added to an interface !

Regards,
Willy
Baptiste
Re: haproxy with keepalived
March 28, 2012 06:20AM
hey,

thanks for sharing your fix.
This is an interesting one :)

Baptiste
Jens Dueholm Christensen (JEDC)
RE: haproxy with keepalived
March 28, 2012 11:50PM
Hey Baptiste

You're very welcome - hopefully others can use it.

At least there was a (somewhat) reasonable explanation and easy fix to the problem once I figured out what was happening.

I re-read my previous post, and realised that one thing was a bit unclear:

The homepage I linked to mentioned that the rp_filter setting also *might* have something to do with correcting the problem.
I cannot verify this, as I already had turned off rp_filter in my setup.

Your milage may also vary depending on distribution - Debian, Suse, RHEL, CentOS etc etc - I have only tested with RHEL.

At least I'm sure that this strange behaviour is limited to linux-flavour OSes. FreeBSD (that I also use) does not exibit the same behaviour.

Regards,
Jens Dueholm Christensen
________________________________________
From: Baptiste [[email protected]]
Sent: 28 March 2012 06:14
To: Jens Dueholm Christensen (JEDC)
Cc: haproxy@formilux.org
Subject: Re: haproxy with keepalived

hey,

thanks for sharing your fix.
This is an interesting one :)

Baptiste
Willy Tarreau
Re: haproxy with keepalived
March 31, 2012 06:40PM
On Wed, Mar 28, 2012 at 09:47:47PM +0000, Jens Dueholm Christensen (JEDC) wrote:
> Hey Baptiste
>
> You're very welcome - hopefully others can use it.
>
> At least there was a (somewhat) reasonable explanation and easy fix to the problem once I figured out what was happening.
>
> I re-read my previous post, and realised that one thing was a bit unclear:
>
> The homepage I linked to mentioned that the rp_filter setting also *might* have something to do with correcting the problem.
> I cannot verify this, as I already had turned off rp_filter in my setup.
>
> Your milage may also vary depending on distribution - Debian, Suse, RHEL, CentOS etc etc - I have only tested with RHEL.
>
> At least I'm sure that this strange behaviour is limited to linux-flavour OSes. FreeBSD (that I also use) does not exibit the same behaviour.

Clearly on linux it's common to have ARP working in a "strange way" for some
people, because its IPv4 stack works exactly like the IPv6 one, with addresses
having a host scope, so any network card is able to respond to an ARP request.

I've been using Julian Anastsov's patchset for more than 10 years on 2.2 then
2.4 to add the arp_announce, arp_filter, arp_ignore etc... sysctls. Now they're
in 2.6 by default but I too think that the default values are confusing, so one
of the very first things I do when I install a system is to switch them. The
second one is to set ip_nonlocal_bind :-)

Regards,
Willy
Jens Dueholm Christensen (JEDC)
RE: haproxy with keepalived
April 02, 2012 10:00AM
(once again I apologize for top-posting)

Would you mind listing what you change (and if possible a reason)?

I've grown up with *BSD-style environments (started out on NetBSD 1.2 back in the 90's on non-x86 hardware), and I keep beeing "amazed" by Linux.
While some vendors and distros are doing good jobs with documentation and features, statements like you own about using a patchset for more than 10 years always gives me a queasy stomach.

What if - oh the horror - I should have done something differently than what I think is "the right way"?
My recent experiences with arp_filter etc tells me that I've still got something to learn..

Regards,
Jens Dueholm Christensen 

-----Original Message-----
From: Willy Tarreau [mailto:[email protected]]
Sent: Saturday, March 31, 2012 6:36 PM
To: Jens Dueholm Christensen (JEDC)
Cc: haproxy@formilux.org
Subject: Re: haproxy with keepalived

Clearly on linux it's common to have ARP working in a "strange way" for some
people, because its IPv4 stack works exactly like the IPv6 one, with addresses
having a host scope, so any network card is able to respond to an ARP request.

I've been using Julian Anastsov's patchset for more than 10 years on 2.2 then
2.4 to add the arp_announce, arp_filter, arp_ignore etc... sysctls. Now they're
in 2.6 by default but I too think that the default values are confusing, so one
of the very first things I do when I install a system is to switch them. The
second one is to set ip_nonlocal_bind :-)

Regards,
Willy
Sorry, only registered users may post in this forum.

Click here to login