Welcome! Log In Create A New Profile

Advanced

Observations about reloads and DNS SRV records

Posted by Tait Clarridge 
Tait Clarridge
Observations about reloads and DNS SRV records
June 06, 2018 05:20PM
I've been testing DNS service discovery and the use of SRV records and have
a few thoughts on a couple things that I noticed.

1. Reloading with SRV records ignores server-state-file
- While this is not a huge deal, it does mean that the backend in
question becomes unavailable when the proxy is reloaded until the SRV and
subsequent A records are resolved
- I understand that the point of these records is to dynamically build
the backend, and populating servers from outdated and potentially invalid
information is not ideal, but it might be nice to seed the backend before
the first SRV query returns
- Below is the related config since it is very likely that I have
misconfigured something :)

globals
...
server-state-base /var/lib/haproxy
server-state-file state

defaults
...
load-server-state-from-file global
default-server init-addr last,libc,none

....

resolvers sd
nameserver sd 127.0.0.1:9999

backend dynamic
server-template srv 5 _http._tcp.serviceA.tait.testing resolvers sd
resolve-prefer ipv4 check

2. Additional record responses from the nameserver are not parsed
- This just means that any servers that are populated from the SRV
records require a second round of querying for each of the hosts after the
fqdn is stored. It might be more efficient if these records are also parsed
but I can see that it might be pretty challenging in the current DNS
resolver
- Only reason I thought of this was to try and reduce up the time it
takes to populate the backend servers with addresses in an effort to lessen
the effects of #1

I'm happy with the workaround I'll be pursing for now where my SD service
(that originally was going to be a resolver and populate via SRV records)
is going to write all the backend definitions to disk so this is not a
pressing issue, just thought I'd share the limitations I discovered. My
knowledge of C (and the internal workings of HAproxy) is not great
otherwise this would probably be a patch submission for #1 :)

Tait
Aleksandar Lazic
Re: Observations about reloads and DNS SRV records
June 06, 2018 05:40PM
Hi Tait.

On 06/06/2018 11:16, Tait Clarridge wrote:
>I've been testing DNS service discovery and the use of SRV records and have
>a few thoughts on a couple things that I noticed.

In this area was a lot of changes in the last version of haproxy, do you have
tested the setup with the latest haproxy 1.8 release?

Please can you be so kind and send us the output of

haproxy -vv

>1. Reloading with SRV records ignores server-state-file
> - While this is not a huge deal, it does mean that the backend in
>question becomes unavailable when the proxy is reloaded until the SRV and
>subsequent A records are resolved
> - I understand that the point of these records is to dynamically build
>the backend, and populating servers from outdated and potentially invalid
>information is not ideal, but it might be nice to seed the backend before
>the first SRV query returns
> - Below is the related config since it is very likely that I have
>misconfigured something :)
>
>globals
> ...
> server-state-base /var/lib/haproxy
> server-state-file state
>
>defaults
> ...
> load-server-state-from-file global
> default-server init-addr last,libc,none
>
>...
>
>resolvers sd
> nameserver sd 127.0.0.1:9999
>
>backend dynamic
> server-template srv 5 _http._tcp.serviceA.tait.testing resolvers sd
>resolve-prefer ipv4 check
>
>2. Additional record responses from the nameserver are not parsed
> - This just means that any servers that are populated from the SRV
>records require a second round of querying for each of the hosts after the
>fqdn is stored. It might be more efficient if these records are also parsed
>but I can see that it might be pretty challenging in the current DNS
>resolver
> - Only reason I thought of this was to try and reduce up the time it
>takes to populate the backend servers with addresses in an effort to lessen
>the effects of #1
>
>I'm happy with the workaround I'll be pursing for now where my SD service
>(that originally was going to be a resolver and populate via SRV records)
>is going to write all the backend definitions to disk so this is not a
>pressing issue, just thought I'd share the limitations I discovered. My
>knowledge of C (and the internal workings of HAproxy) is not great
>otherwise this would probably be a patch submission for #1 :)
>
>Tait
Tait Clarridge
Re: Observations about reloads and DNS SRV records
June 06, 2018 05:50PM
On Wed, Jun 6, 2018 at 11:35 AM Aleksandar Lazic <[email protected]> wrote:

> Hi Tait.
>
> On 06/06/2018 11:16, Tait Clarridge wrote:
> >I've been testing DNS service discovery and the use of SRV records and
> have
> >a few thoughts on a couple things that I noticed.
>
> In this area was a lot of changes in the last version of haproxy, do you
> have
> tested the setup with the latest haproxy 1.8 release?
>
> Please can you be so kind and send us the output of
>
> haproxy -vv
>
>
Hey Aleksandar,

I'm on 1.8.8 so I will try to give it a shot with 1.8.9 later this
afternoon. I've added the output below.

HA-Proxy version 1.8.8 2018/04/19
Copyright 2000-2018 Willy Tarreau <[email protected]>

Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement
-fwrapv -fno-strict-overflow -Wno-format-truncation -Wno-null-dereference
-Wno-unused-label
OPTIONS = USE_LINUX_TPROXY=1 USE_CRYPT_H=1 USE_GETADDRINFO=1 USE_ZLIB=1
USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_SYSTEMD=1 USE_PCRE=1

Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with OpenSSL version : OpenSSL 1.1.0h-fips 27 Mar 2018
Running on OpenSSL version : OpenSSL 1.1.0g-fips 2 Nov 2017
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2
Built with Lua version : Lua 5.3.4
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT
IP_FREEBIND
Encrypted password support via crypt(3): yes
Built with multi-threading support.
Built with PCRE version : 8.42 2018-03-20
Running on PCRE version : 8.41 2017-07-05
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"),
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with network namespace support.

Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.

Available filters :
[SPOE] spoe
[COMP] compression
[TRACE] trace

Thanks,
Tait
Hi Tait

A few comments inline:

1. Reloading with SRV records ignores server-state-file
> - While this is not a huge deal, it does mean that the backend in
> question becomes unavailable when the proxy is reloaded until the SRV and
> subsequent A records are resolved
>
- I understand that the point of these records is to dynamically build
> the backend, and populating servers from outdated and potentially invalid
> information is not ideal, but it might be nice to seed the backend before
> the first SRV query returns
>

This should not happen and it's a known issue that we're working on.



> - Below is the related config since it is very likely that I have
> misconfigured something :)
>
> globals
> ...
> server-state-base /var/lib/haproxy
> server-state-file state
>
> defaults
> ...
> load-server-state-from-file global
> default-server init-addr last,libc,none
>
> ...
>
> resolvers sd
> nameserver sd 127.0.0.1:9999
>
> backend dynamic
> server-template srv 5 _http._tcp.serviceA.tait.testing resolvers sd
> resolve-prefer ipv4 check
>
> 2. Additional record responses from the nameserver are not parsed
>

This is true, by design.


> - This just means that any servers that are populated from the SRV
> records require a second round of querying for each of the hosts after the
> fqdn is stored. It might be more efficient if these records are also parsed
> but I can see that it might be pretty challenging in the current DNS
> resolver
> - Only reason I thought of this was to try and reduce up the time it
> takes to populate the backend servers with addresses in an effort to lessen
> the effects of #1
>

Actually, I tested many DNS server and some of them simply did not send the
additional records when they could not fit in the response (too small
payload for the number of SRV records).
Technically, we could try to use additional records if available and then
failover to current way of working if none found.



> I'm happy with the workaround I'll be pursing for now where my SD service
> (that originally was going to be a resolver and populate via SRV records)
> is going to write all the backend definitions to disk so this is not a
> pressing issue, just thought I'd share the limitations I discovered. My
> knowledge of C (and the internal workings of HAproxy) is not great
> otherwise this would probably be a patch submission for #1 :)
>
> Tait
>
>
I'll check that for you. (In the mean time, please keep on answering to
Aleksandar emails, the more info I'll have, the best).

Baptiste
Tait Clarridge
Re: Observations about reloads and DNS SRV records
June 07, 2018 09:20PM
Hi Baptiste, thanks for the response.

On Wed, Jun 6, 2018 at 6:32 PM Baptiste <[email protected]> wrote:

>
> This should not happen and it's a known issue that we're working on.
>
>
Excellent, figured you guys were probably already aware of it. Let me know
if I can assist in testing.


>
> Actually, I tested many DNS server and some of them simply did not send
> the additional records when they could not fit in the response (too small
> payload for the number of SRV records).
> Technically, we could try to use additional records if available and then
> failover to current way of working if none found.
>
>
True, not a lot do and I don't have a lot of hosts. I assumed it already
parsed them based on reading
https://www.haproxy.com/documentation/aloha/9-5/traffic-management/lb-layer7/dns-srv-records/
and
took that into account when writing my DNS service.

I'm a little swamped with other work at the moment, but when I get a chance
I would be able to provide a DNS server (written in Go) that returns
additional records to test with if that helps.

>
>
>> I'm happy with the workaround I'll be pursing for now where my SD service
>> (that originally was going to be a resolver and populate via SRV records)
>> is going to write all the backend definitions to disk so this is not a
>> pressing issue, just thought I'd share the limitations I discovered. My
>> knowledge of C (and the internal workings of HAproxy) is not great
>> otherwise this would probably be a patch submission for #1 :)
>>
>> Tait
>>
>>
> I'll check that for you. (In the mean time, please keep on answering to
> Aleksandar emails, the more info I'll have, the best).
>
> Baptiste
>

Thanks again, I don't have a lot of time to do any testing right now but
hope to soon.
Hi Tait,

1. Reloading with SRV records ignores server-state-file
>

Can you tell me more about this one. How do you see that?
I mean, I have a conf similar to yours and I can see HAProxy parsing the
server state file (and returning a bunch of warning I'm working on about
backend name and ID mismatch).

==> while writing this mail, I am able to reproduce the issue I think:
- start HAProxy with SRV records
- dump servers state
- update haproxy conf to prevent dns resolution at runtime
- reload haproxy
==> my servers are now "unconfigured"... (no IP, no FQDN, nothing from the
servers state file)


- While this is not a huge deal, it does mean that the backend in
> question becomes unavailable when the proxy is reloaded until the SRV and
> subsequent A records are resolved
>

Well, this is the role of the server state file: populating server info at
configuration parsing time so you don't have to wait until first DNS
resolution to get servers' information.
==> this might be caused by the bug found above.


2. Additional record responses from the nameserver are not parsed
> - This just means that any servers that are populated from the SRV
> records require a second round of querying for each of the hosts after the
> fqdn is stored. It might be more efficient if these records are also parsed
> but I can see that it might be pretty challenging in the current DNS
> resolver
> - Only reason I thought of this was to try and reduce up the time it
> takes to populate the backend servers with addresses in an effort to lessen
> the effects of #1
>
>
I'll work on this one as soon as I fixed the bug above/

Baptiste
>
> I'm a little swamped with other work at the moment, but when I get a
> chance I would be able to provide a DNS server (written in Go) that returns
> additional records to test with if that helps.
>

Thanks for giving me this idea!
I wrote a quick and inflexible one here: https://github.com/bedis/dnsserver
So feel free to contribute to it or write your own :)

I'm going to use it to troubleshoot the issue you reported. That said,
nothing is better than other real DNS servers (bind / unbound / powerdns
and others) for real production.

Baptiste
Tait Clarridge
Re: Observations about reloads and DNS SRV records
June 10, 2018 04:20PM
Hey Baptiste,

On Sun, Jun 10, 2018 at 3:19 AM Baptiste <[email protected]> wrote:

>
> ==> while writing this mail, I am able to reproduce the issue I think:
> - start HAProxy with SRV records
> - dump servers state
> - update haproxy conf to prevent dns resolution at runtime
> - reload haproxy
> ==> my servers are now "unconfigured"... (no IP, no FQDN, nothing from the
> servers state file)
>
>

That is exactly the issue I was seeing, my apologies for not describing the
actual issue I was seeing when I opened this thread.


>>
> 2. Additional record responses from the nameserver are not parsed
>> - This just means that any servers that are populated from the SRV
>> records require a second round of querying for each of the hosts after the
>> fqdn is stored. It might be more efficient if these records are also parsed
>> but I can see that it might be pretty challenging in the current DNS
>> resolver
>> - Only reason I thought of this was to try and reduce up the time it
>> takes to populate the backend servers with addresses in an effort to lessen
>> the effects of #1
>>
>>
> I'll work on this one as soon as I fixed the bug above/
>

Great! The first one will go a long way in making reloads pretty seamless
for DNS, at least for my use case :).

Thanks again,
Tait
Tait Clarridge
Re: Observations about reloads and DNS SRV records
June 10, 2018 04:30PM
On Sun, Jun 10, 2018 at 3:22 AM Baptiste <[email protected]> wrote:

>
> Thanks for giving me this idea!
> I wrote a quick and inflexible one here:
> https://github.com/bedis/dnsserver
> So feel free to contribute to it or write your own :)
>
>
I'm going to use it to troubleshoot the issue you reported. That said,
> nothing is better than other real DNS servers (bind / unbound / powerdns
> and others) for real production.
>

Looks good! My application is using the same DNS library (miekg/dns) which
seems to be the most well-rounded and mature for go. If I think of anything
to add I'll open a PR but it is currently exactly how I am returning my
results.


>
> Baptiste
>
Sorry, only registered users may post in this forum.

Click here to login