HAProxy in High Availability

Hi,

We use HAProxy in our all high availability setups. We set up HAProyx
instances as active-active or active-backup instances.

We use DNS roundrobin to distribute requests to active haproxy
instances and use keepalived for failover.

--
Türker Sezer
TS Design Informatics LTD.

http://www.tsdesign.info/

Hi Türker,

With an Active/Active setup and Round Robin DNS, what happen when one of
the HAProxy is failing/stopped ?

I guess, that the client has the IP in cache and will fail to connect,
won't it?

Do you have some sample configuration to share for DNS round robin &
Keepalived ?

Thanks !


Regards,
Thomas.


On Thu, Jun 28, 2012 at 12:39 PM, Türker Sezer <[email protected]>wrote:

> Hi,
>
> We use HAProxy in our all high availability setups. We set up HAProyx
> instances as active-active or active-backup instances.
>
> We use DNS roundrobin to distribute requests to active haproxy
> instances and use keepalived for failover.
>
> --
> Türker Sezer
> TS Design Informatics LTD.
>
> http://www.tsdesign.info/
>

You're better off running haproxy via pacemaker, so if haproxy dies then
you can not have your VIP run on that host.

We've been doing this for a while and it works nicely. Simple to
configure too.

On 6/28/12 6:39 AM, Türker Sezer wrote:
> Hi,
>
> We use HAProxy in our all high availability setups. We set up HAProyx
> instances as active-active or active-backup instances.
>
> We use DNS roundrobin to distribute requests to active haproxy
> instances and use keepalived for failover.
>

On Thu, Jun 28, 2012 at 11:43 AM, Thomas Manson
<[email protected]> wrote:
> Hi Türker,
>
> With an Active/Active setup and Round Robin DNS, what happen when one of the
> HAProxy is failing/stopped ?
>
> I guess, that the client has the IP in cache and will fail to connect, won't
> it?
>
>  Do you have some sample configuration to share for DNS round robin &
> Keepalived ?

We use DNS roundrobin and keepalived together. If one of the HAProxy
fails, the other HAProxy instance takes its IP.

DNS roundrobin is so simple. You can set two or more A record for a
domain. Then It works as roundrobin.

--
Türker Sezer
TS Design Informatics LTD.

http://www.tsdesign.info/

usually a client will cache the IP served by the DNS server, in order to
not query each time the DNS system.

So how can the client switch to another server once it has resolved one.

Regards,
Thomas.

On Thu, Jun 28, 2012 at 12:53 PM, Türker Sezer <[email protected]>wrote:

> On Thu, Jun 28, 2012 at 11:43 AM, Thomas Manson
> <[email protected]> wrote:
> > Hi Türker,
> >
> > With an Active/Active setup and Round Robin DNS, what happen when one of
> the
> > HAProxy is failing/stopped ?
> >
> > I guess, that the client has the IP in cache and will fail to connect,
> won't
> > it?
> >
> > Do you have some sample configuration to share for DNS round robin &
> > Keepalived ?
>
> We use DNS roundrobin and keepalived together. If one of the HAProxy
> fails, the other HAProxy instance takes its IP.
>
> DNS roundrobin is so simple. You can set two or more A record for a
> domain. Then It works as roundrobin.
>
> --
> Türker Sezer
> TS Design Informatics LTD.
>
> http://www.tsdesign.info/
>

usually a client will cache the IP served by the DNS server, in order to
not query each time the DNS system.

So how can the client switch to another server once it has resolved one.

Regards,
Thomas.

On Thu, Jun 28, 2012 at 12:53 PM, Türker Sezer <[email protected]>wrote:

> On Thu, Jun 28, 2012 at 11:43 AM, Thomas Manson
> <[email protected]> wrote:
> > Hi Türker,
> >
> > With an Active/Active setup and Round Robin DNS, what happen when one of
> the
> > HAProxy is failing/stopped ?
> >
> > I guess, that the client has the IP in cache and will fail to connect,
> won't
> > it?
> >
> > Do you have some sample configuration to share for DNS round robin &
> > Keepalived ?
>
> We use DNS roundrobin and keepalived together. If one of the HAProxy
> fails, the other HAProxy instance takes its IP.
>
> DNS roundrobin is so simple. You can set two or more A record for a
> domain. Then It works as roundrobin.
>
> --
> Türker Sezer
> TS Design Informatics LTD.
>
> http://www.tsdesign.info/
>

I don't know Pacemaker, do you have a sample config to share ?

On Thu, Jun 28, 2012 at 12:50 PM, David Coulson <[email protected]>wrote:

> You're better off running haproxy via pacemaker, so if haproxy dies then
> you can not have your VIP run on that host.
>
> We've been doing this for a while and it works nicely. Simple to configure
> too.
>
>
> On 6/28/12 6:39 AM, Türker Sezer wrote:
>
>> Hi,
>>
>> We use HAProxy in our all high availability setups. We set up HAProyx
>> instances as active-active or active-backup instances.
>>
>> We use DNS roundrobin to distribute requests to active haproxy
>> instances and use keepalived for failover.
>>
>>
>
>

primitive re-haproxy-lsb lsb:haproxy \
meta failure-timeout="60" \
op monitor interval="30" timeout="5s" \
op start interval="0" timeout="5s" \
op stop interval="0" timeout="5s"
primitive re-adproxy-ip ocf:heartbeat:IPaddr \
meta failure-timeout="60" \
params ip="172.31.0.5" cidr_netmask="32" nic="lo" \
meta failure-timeout="60" \
op monitor interval="30s"
group gr-haproxy re-haproxy-lsb re-adproxy-ip



On 6/28/12 7:01 AM, Thomas Manson wrote:
> I don't know Pacemaker, do you have a sample config to share ?
>
>
>
>
>
>

Many thanks, I'm pretty sur it will be usefull for many ;)

Regards,
Thomas.

On Thu, Jun 28, 2012 at 1:03 PM, David Coulson <[email protected]>wrote:

> primitive re-haproxy-lsb lsb:haproxy \
> meta failure-timeout="60" \
> op monitor interval="30" timeout="5s" \
> op start interval="0" timeout="5s" \
> op stop interval="0" timeout="5s"
> primitive re-adproxy-ip ocf:heartbeat:IPaddr \
> meta failure-timeout="60" \
> params ip="172.31.0.5" cidr_netmask="32" nic="lo" \
> meta failure-timeout="60" \
> op monitor interval="30s"
> group gr-haproxy re-haproxy-lsb re-adproxy-ip
>
>
>
>
> On 6/28/12 7:01 AM, Thomas Manson wrote:
>
> I don't know Pacemaker, do you have a sample config to share ?
>
>
>
>>
>>
>
>
>
>

They failover IPs between hosts running haproxy using keepalived - The 2
(or more) IPs references by the DNS record will always be 'alive'.

On 6/28/12 7:00 AM, Thomas Manson wrote:
> usually a client will cache the IP served by the DNS server, in order
> to not query each time the DNS system.
>
> So how can the client switch to another server once it has resolved one.
>
> Regards,
> Thomas.

On Thu, Jun 28, 2012 at 11:59 AM, Manson Thomas <[email protected]> wrote:
> usually a client will cache the IP served by the DNS server, in order to not
> query each time the DNS system.
>
> So how can the client switch to another server once it has resolved one.

Clients dont switch ip address. They connect same ip address. But we
move ip address to backup or another active instance using keepalived
so they connect another server using same ip address.

--
Türker Sezer
TS Design Informatics LTD.

http://www.tsdesign.info/

Ok,

but then, I don't get where is used DNS Round Robin, if only one IP is
used. (it may be obvious, sorry ;);)

Regards,
Thomas.

On Thu, Jun 28, 2012 at 1:08 PM, Türker Sezer <[email protected]>wrote:

> On Thu, Jun 28, 2012 at 11:59 AM, Manson Thomas <[email protected]>
> wrote:
> > usually a client will cache the IP served by the DNS server, in order to
> not
> > query each time the DNS system.
> >
> > So how can the client switch to another server once it has resolved one..
>
> Clients dont switch ip address. They connect same ip address. But we
> move ip address to backup or another active instance using keepalived
> so they connect another server using same ip address.
>
> --
> Türker Sezer
> TS Design Informatics LTD.
>
> http://www.tsdesign.info/
>

Multiple IP addresses are used, and managed by keepalived.

On 6/28/12 7:11 AM, Thomas Manson wrote:
> Ok,
>
> but then, I don't get where is used DNS Round Robin, if only one IP
> is used. (it may be obvious, sorry ;);)
>
> Regards,
> Thomas.
>
> On Thu, Jun 28, 2012 at 1:08 PM, Türker Sezer
> <[email protected] <mailto:[email protected]>> wrote:
>
> On Thu, Jun 28, 2012 at 11:59 AM, Manson Thomas
> <[email protected] <mailto:[email protected]>> wrote:
> > usually a client will cache the IP served by the DNS server, in
> order to not
> > query each time the DNS system.
> >
> > So how can the client switch to another server once it has
> resolved one.
>
> Clients dont switch ip address. They connect same ip address. But we
> move ip address to backup or another active instance using keepalived
> so they connect another server using same ip address.
>
> --
> Türker Sezer
> TS Design Informatics LTD.
>
> http://www.tsdesign.info/
>
>

With this approach you really want 1 fewer public IP than you have public facing servers. With 2 servers this means 1 IP. DNS is used to distribute the load around, and keepalived is used to move traffic when a server fails. But you always want at least 1 servers worth of spare capacity in your HA environment, otherwise after you fail over the server getting 2X traffic will also die.

Actually that's an oversimplified example, since to distribute the load correctly you'd need n-1 public addresses on each of n servers (with diminishing returns). Thankfully few LB scenarios require more than 1X1 machines.

-Richard

On Jun 28, 2012, at 6:17 AM, David Coulson <[email protected]> wrote:

> Multiple IP addresses are used, and managed by keepalived.
>
> On 6/28/12 7:11 AM, Thomas Manson wrote:
>> Ok,
>>
>> but then, I don't get where is used DNS Round Robin, if only one IP is used. (it may be obvious, sorry ;);)
>>
>> Regards,
>> Thomas.
>>
>> On Thu, Jun 28, 2012 at 1:08 PM, Türker Sezer <[email protected]> wrote:
>> On Thu, Jun 28, 2012 at 11:59 AM, Manson Thomas <[email protected]> wrote:
>> > usually a client will cache the IP served by the DNS server, in order to not
>> > query each time the DNS system.
>> >
>> > So how can the client switch to another server once it has resolved one..
>>
>> Clients dont switch ip address. They connect same ip address. But we
>> move ip address to backup or another active instance using keepalived
>> so they connect another server using same ip address.
>>
>> --
>> Türker Sezer
>> TS Design Informatics LTD.
>>
>> http://www.tsdesign.info/
>>
>
>

Thomas,

Check out this tutorial:
http://www.howtoforge.com/setting-up-a-high-availability-load-balancer-with-haproxy-keepalived-on-debian-lenny

I have a much more complex setup running and I have tested the failover one
too many times, works like a charm

Good Luck

On Thu, Jun 28, 2012 at 7:29 AM, Richard Stanford <[email protected]>wrote:

> With this approach you really want 1 fewer public IP than you have public
> facing servers. With 2 servers this means 1 IP. DNS is used to distribute
> the load around, and keepalived is used to move traffic when a server
> fails. But you always want at least 1 servers worth of spare capacity in
> your HA environment, otherwise after you fail over the server getting 2X
> traffic will also die.
>
> Actually that's an oversimplified example, since to distribute the load
> correctly you'd need n-1 public addresses on each of n servers (with
> diminishing returns). Thankfully few LB scenarios require more than 1X1
> machines.
>
> -Richard
>
> On Jun 28, 2012, at 6:17 AM, David Coulson <[email protected]> wrote:
>
> Multiple IP addresses are used, and managed by keepalived.
>
> On 6/28/12 7:11 AM, Thomas Manson wrote:
>
> Ok,
>
> but then, I don't get where is used DNS Round Robin, if only one IP is
> used. (it may be obvious, sorry ;);)
>
> Regards,
> Thomas.
>
> On Thu, Jun 28, 2012 at 1:08 PM, Türker Sezer <[email protected]>wrote:
>
>> On Thu, Jun 28, 2012 at 11:59 AM, Manson Thomas <[email protected]>
>> wrote:
>> > usually a client will cache the IP served by the DNS server, in order
>> to not
>> > query each time the DNS system.
>> >
>> > So how can the client switch to another server once it has resolved one.
>>
>> Clients dont switch ip address. They connect same ip address. But we
>> move ip address to backup or another active instance using keepalived
>> so they connect another server using same ip address.
>>
>> --
>> Türker Sezer
>> TS Design Informatics LTD.
>>
>> http://www.tsdesign.info/
>>
>
>
>
>

On Thu, Jun 28, 2012 at 06:50:26AM -0400, David Coulson wrote:
> You're better off running haproxy via pacemaker, so if haproxy dies then
> you can not have your VIP run on that host.

That's already what keepalived does, and it goes a bit further
in that you can monitor the service for real, not just the
process presence, and even decide several failover scenarios
using floating VRRP priorities. For instance, I usually assign
a weight of 4 to my haproxy process and 2 to sshd. That way, if
haproxy dies, the other one takes the VIP. However, if both
haproxy work and one sshd dies, the one taking the VIP is the
one still reachable for administration. If one of each dies,
the one with the running haproxy wins.

Some more complex setups involve switching a VIP depending on the
number of application servers that haproxy sees, using
"monitor-fail if ...". This is handy for multi-layer architectures
with an inter-DC LAN for instance.

Regards,
Willy

On 6/28/12 7:15 PM, Willy Tarreau wrote:
> That's already what keepalived does, and it goes a bit further
> in that you can monitor the service for real, not just the
> process presence, and even decide several failover scenarios
> using floating VRRP priorities. For instance, I usually assign
> a weight of 4 to my haproxy process and 2 to sshd. That way, if
> haproxy dies, the other one takes the VIP. However, if both
> haproxy work and one sshd dies, the one taking the VIP is the
> one still reachable for administration. If one of each dies,
> the one with the running haproxy wins.
You can do similar functionality with Pacemaker - Not sure if it is more
complex, the same, or simpler to support than keepalived. It's just
'different'. I've used keepalived twice in the last 10 years, and
pacemaker pretty much every day - Probably a little biased.

Pacemaker is tech preview in RHEL6, and SuSE 11 uses it as it's standard
resource manager for clustering. Would be nice if there was
documentation or at least config snippets from users who have
implemented it in the field.
>
> Some more complex setups involve switching a VIP depending on the
> number of application servers that haproxy sees, using
> "monitor-fail if ...". This is handy for multi-layer architectures
> with an inter-DC LAN for instance.
Do you have a configuration example of this? I don't think there is a
custom HAProxy OCF for Pacemaker yet - I've just been using the init.d
script, but sounds like it could use a 'real' Pacemaker script to
support it properly, especially if you can feed availability information
of backend systems as attributes to influence where resources are placed.

Is parsing out the http stats page (in csv, xml, or whatever) the
simplest way to get the current 'state' of HAProxy and the systems it is
routing to, or is there a better/cleaner/other way to do it?

On 29/06/2012 11:53, David Coulson wrote:
> Is parsing out the http stats page (in csv, xml, or whatever) the
> simplest way to get the current 'state' of HAProxy and the systems it is
> routing to, or is there a better/cleaner/other way to do it?

Hi,

You can use the haprox stats socket to get the current 'state'.

echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio

The output is plain text that you can easily parse.

For a more complete description of how the unix socket is used in
haproxy, go to the doc and look for "9.2. Unix Socket commands".
http://haproxy.1wt.eu/download/1.5/doc/configuration.txt

Regards,
Laurent