Welcome! Log In Create A New Profile

Advanced

1.8.3 dns resolver ipv4/ipv6 undesirable behaviour

Posted by Marc Fournier 
Marc Fournier
1.8.3 dns resolver ipv4/ipv6 undesirable behaviour
January 08, 2018 05:50PM
Hello,

Using the following (simplified) configuration, all the servers go (and
stay) into maintenance mode about 30s after start up or config
reload. Default "resolvers/hold timeout" I guess. These log lines get emitted:

2018-01-08T15:38:10.209195+00:00: Proxy dockercloud_hello-world started.
2018-01-08T15:38:10.209200+00:00: Proxy tutum_hello-world started.
[...]
2018-01-08T15:38:41.565222+00:00: Server tutum_hello-world/tutum1 is going DOWN for maintenance (DNS timeout status). 3 active and 1 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
2018-01-08T15:38:41.565233+00:00: Server tutum_hello-world/tutum2 is going DOWN for maintenance (DNS timeout status). 2 active and 1 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
2018-01-08T15:38:41.565238+00:00: Server tutum_hello-world/tutum3 is going DOWN for maintenance (DNS timeout status). 1 active and 1 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
2018-01-08T15:38:41.565244+00:00: Server tutum_hello-world/tutum4 is going DOWN for maintenance (DNS timeout status). 0 active and 1 backup servers left. Running on backup. 0 sessions active, 0 requeued, 0 remaining in queue.
2018-01-08T15:38:41.565250+00:00: Server dockercloud_hello-world/dockercloud1 is going DOWN for maintenance (DNS timeout status). 0 active and 1 backup servers left. Running on backup. 0 sessions active, 0 requeued, 0 remaining in queue.


resolvers rancher
nameserver dnsmasq 169.254.169.250:53

backend dockercloud_hello-world
default-server inter 2000 rise 2 fall 3 port 80
server sorry 127.0.0.1:8082 backup
server-template dockercloud 1 hello-world.dockercloud.rancher.internal:80 resolvers rancher check

backend tutum_hello-world
default-server inter 2000 rise 2 fall 3 port 80
server sorry 127.0.0.1:8082 backup
server-template tutum 4 hello-world.tutum.rancher.internal:80 resolvers rancher check


I tcpdumped the DNS traffic and noticed the AAAA answers were empty.
(full output of wireshark's "text export" here:
https://gist.github.com/mfournier/0642a32df759ee0b1fbbd505a862f191)

Simply adding "resolve-prefer ipv4" makes the symptom go away, so no big
deal. But I wanted to point this out, as it might bite others, and I'm
pretty sure 1.7.x didn't have this issue.

Also, it doesn't seem right to me that a whole backend can get knocked
out by an incomplete DNS config (ie: the setup works well the first 30
seconds, as long as only the ipv4 A records get considered).

Thanks !

Marc
Marc Fournier
Re: 1.8.3 dns resolver ipv4/ipv6 undesirable behaviour
January 09, 2018 01:30PM
Marc Fournier <[email protected]> writes:

> Simply adding "resolve-prefer ipv4" makes the symptom go away, so no big
> deal. But I wanted to point this out, as it might bite others, and I'm
> pretty sure 1.7.x didn't have this issue.

It turns out that "resolve-prefer ipv4" wasn't what fixed my problem
after all. It was reordering the options on the server-template line
which made the difference.

So basically changing:
server-template tutum 4 hello-world.tutum.rancher.internal:80 resolvers rancher check
to:
server-template tutum 4 hello-world.tutum.rancher.internal:80 check resolvers rancher

With this change, tcpdump shows DNS queries/responses every
10s. Previously queries seemed to be sent only once, at start up/reload.

If that's the expected behaviour, this statement in the documentation is
misleading:
The "server" and "default-server" keywords support a certain number of settings
which are all passed as arguments on the server line. The order in which those
arguments appear does not count, and they are all optional.

nb: this problem seems to also happen with standard "server"
definitions, not only with the new "server-template" one.

Marc
Sorry, only registered users may post in this forum.

Click here to login