Welcome! Log In Create A New Profile

Advanced

haproxy and changing ELB IPs

Posted by K3 
K3
haproxy and changing ELB IPs
August 03, 2018 06:00PM
Hi,
We are running into a problem and would like to hear any advice.

Our Setup:
We use haproxy 1.7.7 with two backends.
One of the backends is AWS ELB
The haproxy is running on a linux machine in our data center (on premises)

Problem:
The ELB is available on 3 AZ so the endpoint can resolve to 3 IPs at a
given time.
After startup, when one of the ELB's IP changes, our haproxy shows the ELB
as down with L4TMOUT and never recovers the backend.

From 1.7.7 doc, under section 5.3.1 we see the below
"A few other events can trigger a name resolution at run time:

- when a server's health check ends up in a connection timeout: this may be
because the server has a new IP address. So we need to trigger a name
resolution to know this new IP."

Its not clear if the resolvers section is required for the above
statement to be true.

Unfortunately, we cannot define name server's IPs in our configuration
since these IPs can change.

On the system (/etc/resolv.conf), these are automatically updated by dhclient.

Question:

1. Is there any way by which haproxy can use the latest DNS entries
from the system config and use it during the runtime?

2. Is there any way to configure haproxy to expire name resolution
after X secs without defining the nameserver IPs?


Please advice.

Thanks,
Karthik
Igor Cicimov
Re: haproxy and changing ELB IPs
August 04, 2018 02:30PM
Hi,

On Sat, Aug 4, 2018 at 1:50 AM, K3 <[email protected]> wrote:

> Hi,
> We are running into a problem and would like to hear any advice.
>
> Our Setup:
> We use haproxy 1.7.7 with two backends.
> One of the backends is AWS ELB
> The haproxy is running on a linux machine in our data center (on premises)
>
> Problem:
> The ELB is available on 3 AZ so the endpoint can resolve to 3 IPs at a
> given time.
> After startup, when one of the ELB's IP changes, our haproxy shows the ELB
> as down with L4TMOUT and never recovers the backend.
>
> From 1.7.7 doc, under section 5.3.1 we see the below
> "A few other events can trigger a name resolution at run time:
>
> - when a server's health check ends up in a connection timeout: this may be
> because the server has a new IP address. So we need to trigger a name
> resolution to know this new IP."
>
> Its not clear if the resolvers section is required for the above statement to be true.
>
> Unfortunately, we cannot define name server's IPs in our configuration since these IPs can change.
>
> On the system (/etc/resolv.conf), these are automatically updated by dhclient.
>
> Question:
>
> 1. Is there any way by which haproxy can use the latest DNS entries from the system config and use it during the runtime?
>
> I just saw this announced in 1.9-dev1


- the "resolvers" section can now be fed directly from resolv.conf
using the "parse-resolv-conf" directive. The DNS code also supports
new options to enable/disable address deduplication within a farm.


so you might give it a try.


> 2. Is there any way to configure haproxy to expire name resolution after X secs without defining the nameserver IPs?
>
> There is "hold valid" option for the "resolvers" section to tell haproxy
to refresh the resolved names lets say every 10 seconds. Makes sense in
case of

reading from resolv.conf to re-read its content as well but not sure, maybe
it does it once on startup and then it's cached, we need someone from
haproxy team to comment on this.


> Please advice.
>
> Thanks,
> Karthik
>
Lukas Tribus
Re: haproxy and changing ELB IPs
August 04, 2018 03:30PM
On Sat, 4 Aug 2018 at 14:21, Igor Cicimov
<[email protected]> wrote:
>
> Hi,
>
> On Sat, Aug 4, 2018 at 1:50 AM, K3 <[email protected]> wrote:
>>
>> Hi,
>> We are running into a problem and would like to hear any advice.
>>
>> Our Setup:
>> We use haproxy 1.7.7 with two backends.
>> One of the backends is AWS ELB
>> The haproxy is running on a linux machine in our data center (on premises)
>>
>> Problem:
>> The ELB is available on 3 AZ so the endpoint can resolve to 3 IPs at a given time.
>> After startup, when one of the ELB's IP changes, our haproxy shows the ELB as down with L4TMOUT and never recovers the backend.
>>
>> From 1.7.7 doc, under section 5.3.1 we see the below
>> "A few other events can trigger a name resolution at run time:
>>
>> - when a server's health check ends up in a connection timeout: this may be
>> because the server has a new IP address. So we need to trigger a name
>> resolution to know this new IP."
>>
>> Its not clear if the resolvers section is required for the above statement to be true.
>>
>> Unfortunately, we cannot define name server's IPs in our configuration since these IPs can change.
>>
>> On the system (/etc/resolv.conf), these are automatically updated by dhclient.
>>
>> Question:
>>
>> 1. Is there any way by which haproxy can use the latest DNS entries from the system config and use it during the runtime?
>
> I just saw this announced in 1.9-dev1
>
>
> - the "resolvers" section can now be fed directly from resolv.conf
> using the "parse-resolv-conf" directive. The DNS code also supports
> new options to enable/disable address deduplication within a farm.

This is on startup only though. Haproxy cannot continuously re-read
resolv.conf, not even with this feature.

If the DNS servers on your system keep changing unpredictably, there
is no solution other than restarting haproxy, with the
parse-resolv-conf configuration and the haproxy internal resolver.



Lukas
Karthik P
Re: haproxy and changing ELB IPs
August 06, 2018 07:40PM
Thats a great news! Thanks.

For the #2 (haproxy to refresh DNS entries when changed), is it possible to
optionally log the NS IPs during every health check?

We recently had an outage for short time related to NameServer's h/w
failure (both primary and secondary went down). We were told that it is
possible for these IPs to change in the future. It never happened so far
though.

Thanks,
Karthik

On Sat, Aug 4, 2018 at 6:19 AM, Lukas Tribus <[email protected]> wrote:

> On Sat, 4 Aug 2018 at 14:21, Igor Cicimov
> <[email protected]> wrote:
> >
> > Hi,
> >
> > On Sat, Aug 4, 2018 at 1:50 AM, K3 <[email protected]> wrote:
> >>
> >> Hi,
> >> We are running into a problem and would like to hear any advice.
> >>
> >> Our Setup:
> >> We use haproxy 1.7.7 with two backends.
> >> One of the backends is AWS ELB
> >> The haproxy is running on a linux machine in our data center (on
> premises)
> >>
> >> Problem:
> >> The ELB is available on 3 AZ so the endpoint can resolve to 3 IPs at a
> given time.
> >> After startup, when one of the ELB's IP changes, our haproxy shows the
> ELB as down with L4TMOUT and never recovers the backend.
> >>
> >> From 1.7.7 doc, under section 5.3.1 we see the below
> >> "A few other events can trigger a name resolution at run time:
> >>
> >> - when a server's health check ends up in a connection timeout: this
> may be
> >> because the server has a new IP address. So we need to trigger a
> name
> >> resolution to know this new IP."
> >>
> >> Its not clear if the resolvers section is required for the above
> statement to be true.
> >>
> >> Unfortunately, we cannot define name server's IPs in our configuration
> since these IPs can change.
> >>
> >> On the system (/etc/resolv.conf), these are automatically updated by
> dhclient.
> >>
> >> Question:
> >>
> >> 1. Is there any way by which haproxy can use the latest DNS entries
> from the system config and use it during the runtime?
> >
> > I just saw this announced in 1.9-dev1
> >
> >
> > - the "resolvers" section can now be fed directly from resolv.conf
> > using the "parse-resolv-conf" directive. The DNS code also supports
> > new options to enable/disable address deduplication within a farm.
>
> This is on startup only though. Haproxy cannot continuously re-read
> resolv.conf, not even with this feature.
>
> If the DNS servers on your system keep changing unpredictably, there
> is no solution other than restarting haproxy, with the
> parse-resolv-conf configuration and the haproxy internal resolver.
>
>
>
> Lukas
>
Igor Cicimov
Re: haproxy and changing ELB IPs
August 07, 2018 02:10AM
Hi Lukas,

On Sat, Aug 4, 2018 at 11:19 PM, Lukas Tribus <[email protected]> wrote:

> On Sat, 4 Aug 2018 at 14:21, Igor Cicimov
> <[email protected]> wrote:
> >
> > Hi,
> >
> > On Sat, Aug 4, 2018 at 1:50 AM, K3 <[email protected]> wrote:
> >>
> >> Hi,
> >> We are running into a problem and would like to hear any advice.
> >>
> >> Our Setup:
> >> We use haproxy 1.7.7 with two backends.
> >> One of the backends is AWS ELB
> >> The haproxy is running on a linux machine in our data center (on
> premises)
> >>
> >> Problem:
> >> The ELB is available on 3 AZ so the endpoint can resolve to 3 IPs at a
> given time.
> >> After startup, when one of the ELB's IP changes, our haproxy shows the
> ELB as down with L4TMOUT and never recovers the backend.
> >>
> >> From 1.7.7 doc, under section 5.3.1 we see the below
> >> "A few other events can trigger a name resolution at run time:
> >>
> >> - when a server's health check ends up in a connection timeout: this
> may be
> >> because the server has a new IP address. So we need to trigger a
> name
> >> resolution to know this new IP."
> >>
> >> Its not clear if the resolvers section is required for the above
> statement to be true.
> >>
> >> Unfortunately, we cannot define name server's IPs in our configuration
> since these IPs can change.
> >>
> >> On the system (/etc/resolv.conf), these are automatically updated by
> dhclient.
> >>
> >> Question:
> >>
> >> 1. Is there any way by which haproxy can use the latest DNS entries
> from the system config and use it during the runtime?
> >
> > I just saw this announced in 1.9-dev1
> >
> >
> > - the "resolvers" section can now be fed directly from resolv.conf
> > using the "parse-resolv-conf" directive. The DNS code also supports
> > new options to enable/disable address deduplication within a farm.
>
> This is on startup only though. Haproxy cannot continuously re-read
> resolv.conf, not even with this feature.
>
> If the DNS servers on your system keep changing unpredictably, there
> is no solution other than restarting haproxy, with the
> parse-resolv-conf configuration and the haproxy internal resolver.
>
> Would a reload suffice instead of restart? It should not be difficult to
create a monitor for *resolv.conf* file using inotify lets say and
automatically reload/restart haproxy in case it's content has changed.
Lukas Tribus
Re: haproxy and changing ELB IPs
August 07, 2018 11:50AM
Hello,


> We recently had an outage for short time related to NameServer's h/w failure (both primary and secondary went down).
> We were told that it is possible for these IPs to change in the future. It never happened so far though.

So you don't have changing nameservers at all, but it is possible that
the IPs will change once.

I suggest you don't over-engineer this. Automating a possible one time
occurrence is a waste of time, imho.


> is it possible to optionally log the NS IPs during every health check?

No.


> Would a reload suffice instead of restart? It should not be difficult to create a monitor
> for resolv.conf file using inotify lets say and automatically reload/restart haproxy in case
> it's content has changed.

Sure, a reload would suffice.


Regards,
Lukas
Patrick Hemmer
Re: haproxy and changing ELB IPs
August 07, 2018 05:00PM
On 2018/8/7 05:45, Lukas Tribus wrote:
> Hello,
>
>
>> We recently had an outage for short time related to NameServer's h/w failure (both primary and secondary went down).
>> We were told that it is possible for these IPs to change in the future. It never happened so far though.
> So you don't have changing nameservers at all, but it is possible that
> the IPs will change once.
>
> I suggest you don't over-engineer this. Automating a possible one time
> occurrence is a waste of time, imho.
>
>
>> is it possible to optionally log the NS IPs during every health check?
> No.
>
>
>> Would a reload suffice instead of restart? It should not be difficult to create a monitor
>> for resolv.conf file using inotify lets say and automatically reload/restart haproxy in case
>> it's content has changed.
> Sure, a reload would suffice.
>
>
> Regards,
> Lukas
>
As an alternative option, if the system utilizes NetworkManager, then
solving this becomes very easy. NetworkManager can be configured to
provide a local dnsmasq instance as a DNS proxy. If this is enabled,
then your resolver becomes a static "127.0.0.1". And since
NetworkManager also integrates with the DHCP client, if the nameserver
IPs change, then it'll reload dnsmasq, and you don't need to do anything
with haproxy.
Enabling this is as simple as adding "dns = dnsmasq" to NetworkManager.conf.

-Patrick
Sorry, only registered users may post in this forum.

Click here to login