Welcome! Log In Create A New Profile

Advanced

DNS resolver and mixed case responses

Posted by Dale Smith 
Dale Smith
DNS resolver and mixed case responses
April 03, 2018 04:30PM
Hello All,

I have noticed a potential issue in HAProxy 1.7 where a backend server has
reported in logs an 'unspecified DNS error' and been marked down for
maintenance.

After some investigation, I have found that our internal DNS server is
responding with non-lowercase domain names in the Answer section. When
HAProxy 'check' performs the lookup in dns.c the validation on the Answer
domain with memcmp fails.

I'm trying to understand what system is at fault here; the DNS server for
not responding with the same case as the query, or HAProxy which should be
performing a case insensitive match.
I have made some test changes to src/dns.c to implement case insensitive
matching in line 260 (in dns_resolve_recv) and unifying case before
returning in dns_read_name. After these changes I can no longer reproduce
the problem. I'm happy to share my patch file, but expect that your own
libs and implementation will be cleaner.
Relevant RFC: https://tools.ietf.org/html/rfc4343#section-3

The current viable workaround appears to be simply matching the case that
the DNS server is configured to reply with. For us this means specifying
something like 'Server.db.example.com' instead of 'server.db.example.com'
in haproxy.cfg.


Reproduction steps:
* Use 'check' and 'resolvers' in server section to enable
post-initialisation dns lookups.
* Have the DNS resolver reply in upper/lower/mixed case that does not match
config file. (I can share a small python script that can be used to test
this)
The initial resolution during config read is fine, as this is done via
libc. The subsequent DNS resolutions during checks fail and eventually set
the server as offline.

---
Sample configuration to reproduce:
defaults
log global
mode tcp
timeout connect 10s
timeout client 1m
timeout server 1m
timeout check 10s

resolvers dns-resolver
nameserver ns1 127.0.0.1:1153
resolve_retries 3
timeout retry 1s
hold valid 2s # small, so we can reproduce quickly
hold other 10s

frontend fe15000
bind :15000
default_backend be15000

backend be15000
server dest15000 example.com:5000 check resolvers dns-resolver
init-addr libc,none
---

Sample DNS response that triggers the issue:
(Note the reply CaSe does not match query. I can also provide a simple
Python server that performs uppercase in its reply, for replication of
this.)
---
$ dig @127.0.0.1 -p 1153 example.com
<snip>
;; QUESTION SECTION:
;EXAMPLE.COM. IN A

;; ANSWER SECTION:
EXAMPLE.COM. 60 IN A 127.0.0.1
<snip>


Thanks for any assistance,
Dale Smith
Baptiste
Re: DNS resolver and mixed case responses
April 03, 2018 06:40PM
On Tue, Apr 3, 2018 at 4:17 PM, Dale Smith <[email protected]> wrote:

> Hello All,
>
> I have noticed a potential issue in HAProxy 1.7 where a backend server has
> reported in logs an 'unspecified DNS error' and been marked down for
> maintenance.
>
> After some investigation, I have found that our internal DNS server is
> responding with non-lowercase domain names in the Answer section. When
> HAProxy 'check' performs the lookup in dns.c the validation on the Answer
> domain with memcmp fails.
>
> I'm trying to understand what system is at fault here; the DNS server for
> not responding with the same case as the query, or HAProxy which should be
> performing a case insensitive match.
> I have made some test changes to src/dns.c to implement case insensitive
> matching in line 260 (in dns_resolve_recv) and unifying case before
> returning in dns_read_name. After these changes I can no longer reproduce
> the problem. I'm happy to share my patch file, but expect that your own
> libs and implementation will be cleaner.
> Relevant RFC: https://tools.ietf.org/html/rfc4343#section-3
>
> The current viable workaround appears to be simply matching the case that
> the DNS server is configured to reply with. For us this means specifying
> something like 'Server.db.example.com' instead of 'server.db.example.com'
> in haproxy.cfg.
>
>
> Reproduction steps:
> * Use 'check' and 'resolvers' in server section to enable
> post-initialisation dns lookups.
> * Have the DNS resolver reply in upper/lower/mixed case that does not
> match config file. (I can share a small python script that can be used to
> test this)
> The initial resolution during config read is fine, as this is done via
> libc. The subsequent DNS resolutions during checks fail and eventually set
> the server as offline.
>
> ---
> Sample configuration to reproduce:
> defaults
> log global
> mode tcp
> timeout connect 10s
> timeout client 1m
> timeout server 1m
> timeout check 10s
>
> resolvers dns-resolver
> nameserver ns1 127.0.0.1:1153
> resolve_retries 3
> timeout retry 1s
> hold valid 2s # small, so we can reproduce quickly
> hold other 10s
>
> frontend fe15000
> bind :15000
> default_backend be15000
>
> backend be15000
> server dest15000 example.com:5000 check resolvers dns-resolver
> init-addr libc,none
> ---
>
> Sample DNS response that triggers the issue:
> (Note the reply CaSe does not match query. I can also provide a simple
> Python server that performs uppercase in its reply, for replication of
> this.)
> ---
> $ dig @127.0.0.1 -p 1153 example.com
> <snip>
> ;; QUESTION SECTION:
> ;EXAMPLE.COM. IN A
>
> ;; ANSWER SECTION:
> EXAMPLE.COM. 60 IN A 127.0.0.1
> <snip>
>
>
> Thanks for any assistance,
> Dale Smith
>
>

Hi Dale,

Thanks for the report!
Please share your patch here and I'll have a look, so we could merge it.

Baptiste
Dale Smith
Re: DNS resolver and mixed case responses
April 04, 2018 11:40AM
On Tue, 3 Apr 2018 at 17:35 Baptiste <[email protected]> wrote:

>
> Hi Dale,
>
> Thanks for the report!
> Please share your patch here and I'll have a look, so we could merge it.
>
> Baptiste
>

Hi Baptiste,

I've attached a patch file for haproxy-1.7 that performs the case
insensitive match instead of a direct memcmp. This resolves the issue in my
testing, I'd be interested in your thoughts on this; and happy to rework or
re-test any suggestions you have for the patch.

I previously mentioned an additional dns_toupper within dns_read_name, but
this does not appear to be necessary to resolve this particular issue so I
have removed it.

Regards,
Dale Smith
Attachments:
open | download - dns_strnicmp.patch (1.7 KB)
Tim Düsterhus
Re: DNS resolver and mixed case responses
April 04, 2018 04:40PM
Dale,

Am 03.04.2018 um 16:17 schrieb Dale Smith:
> I'm trying to understand what system is at fault here; the DNS server for
> not responding with the same case as the query, or HAProxy which should be
> performing a case insensitive match.

This is left unspecified in the standards, but on the other hand there
is this Internet Draft:
https://tools.ietf.org/html/draft-vixie-dnsext-dns0x20-00 which wants to
mandate case preserval to make DNS spoofing harder by introducing more
entropy in the DNS request.

I recommend to fix your internal DNS server, because case preserving
behaviour seems to be somewhat expected according to a quick Google search.

Best regards
Tim Düsterhus
Dennis Jacobfeuerborn
Re: DNS resolver and mixed case responses
April 06, 2018 11:20AM
On 04.04.2018 16:30, Tim Düsterhus wrote:
> Dale,
>
> Am 03.04.2018 um 16:17 schrieb Dale Smith:
>> I'm trying to understand what system is at fault here; the DNS server for
>> not responding with the same case as the query, or HAProxy which
>> should be
>> performing a case insensitive match.
>
> This is left unspecified in the standards, but on the other hand there
> is this Internet Draft:
> https://tools.ietf.org/html/draft-vixie-dnsext-dns0x20-00 which wants to
> mandate case preserval to make DNS spoofing harder by introducing more
> entropy in the DNS request.
>
> I recommend to fix your internal DNS server, because case preserving
> behaviour seems to be somewhat expected according to a quick Google search.

There is this:

Domain Name System (DNS) Case Insensitivity Clarification:
https://tools.ietf.org/html/rfc4343#section-3.1

In section 3 it says this:

3. Name Lookup, Label Types, and CLASS

According to the original DNS design decision, comparisons on name
lookup for DNS queries should be case insensitive [STD13]. That is
to say, a lookup string octet with a value in the inclusive range
from 0x41 to 0x5A, the uppercase ASCII letters, MUST match the
identical value and also match the corresponding value in the
inclusive range from 0x61 to 0x7A, the lowercase ASCII letters. A
lookup string octet with a lowercase ASCII letter value MUST
similarly match the identical value and also match the corresponding
value in the uppercase ASCII letter range.

(Historical note: The terms "uppercase" and "lowercase" were invented
after movable type. The terms originally referred to the two font
trays for storing, in partitioned areas, the different physical type
elements. Before movable type, the nearest equivalent terms were
"majuscule" and "minuscule".)

This reads to me like HAProxy should match characters in the ranges 0x41
to 0x5A and 0x61 to 0x7A insensitively as long as the label type is ASCII.

Section 4.1 "DNS Output Case Preservation" mentions this: "No "case
conversion" or "case folding" is done during such output operations,
thus "preserving" case."

Regrads,
Dennis
Baptiste
Re: DNS resolver and mixed case responses
April 09, 2018 09:40AM
On Fri, Apr 6, 2018 at 11:10 AM, Dennis Jacobfeuerborn <
[email protected]> wrote:

> On 04.04.2018 16:30, Tim Düsterhus wrote:
> > Dale,
> >
> > Am 03.04.2018 um 16:17 schrieb Dale Smith:
> >> I'm trying to understand what system is at fault here; the DNS server
> for
> >> not responding with the same case as the query, or HAProxy which
> >> should be
> >> performing a case insensitive match.
> >
> > This is left unspecified in the standards, but on the other hand there
> > is this Internet Draft:
> > https://tools.ietf.org/html/draft-vixie-dnsext-dns0x20-00 which wants to
> > mandate case preserval to make DNS spoofing harder by introducing more
> > entropy in the DNS request.
> >
> > I recommend to fix your internal DNS server, because case preserving
> > behaviour seems to be somewhat expected according to a quick Google
> search.
>
> There is this:
>
> Domain Name System (DNS) Case Insensitivity Clarification:
> https://tools.ietf.org/html/rfc4343#section-3.1
>
> In section 3 it says this:
>
> 3. Name Lookup, Label Types, and CLASS
>
> According to the original DNS design decision, comparisons on name
> lookup for DNS queries should be case insensitive [STD13]. That is
> to say, a lookup string octet with a value in the inclusive range
> from 0x41 to 0x5A, the uppercase ASCII letters, MUST match the
> identical value and also match the corresponding value in the
> inclusive range from 0x61 to 0x7A, the lowercase ASCII letters. A
> lookup string octet with a lowercase ASCII letter value MUST
> similarly match the identical value and also match the corresponding
> value in the uppercase ASCII letter range.
>
> (Historical note: The terms "uppercase" and "lowercase" were invented
> after movable type. The terms originally referred to the two font
> trays for storing, in partitioned areas, the different physical type
> elements. Before movable type, the nearest equivalent terms were
> "majuscule" and "minuscule".)
>
> This reads to me like HAProxy should match characters in the ranges 0x41
> to 0x5A and 0x61 to 0x7A insensitively as long as the label type is ASCII..
>
> Section 4.1 "DNS Output Case Preservation" mentions this: "No "case
> conversion" or "case folding" is done during such output operations,
> thus "preserving" case."
>
> Regrads,
> Dennis
>
>
Hi All,

Let me ask some advices to our friends of PowerDNS :)

Baptiste
Baptiste
Re: DNS resolver and mixed case responses
April 09, 2018 10:20AM
So, it seems that responses that does not match the case should be dropped:
https://twitter.com/PowerDNS_Bert/status/983254222694240257

Baptiste
Ben Draut
Re: DNS resolver and mixed case responses
April 10, 2018 11:20PM
It's interesting that the default behavior of HAProxy resolvers can
conflict with the default behavior of bind. (If you're unlucky with
whatever bind has cached)

By default, bind uses case-insensitive compression, which can cause it to
use a different case in the ANSWER than in the QUESTION. (See
'no-case-compress':
https://ftp.isc.org/isc/bind9/cur/9.9/doc/arm/Bv9ARM.ch06.html) We were
impacted by this recently.

Also interesting:
https://indico.dns-oarc.net/event/20/session/2/contribution/12/material/slides/0.pdf


On Mon, Apr 9, 2018 at 2:12 AM, Baptiste <[email protected]> wrote:

> So, it seems that responses that does not match the case should be dropped:
> https://twitter.com/PowerDNS_Bert/status/983254222694240257
>
> Baptiste
>
Jim Freeman
Re: DNS resolver and mixed case responses
April 12, 2018 03:20PM
It will be important to know which behavior AWS's Route53/DNS servers use ?

Using stock Debian/Stretch BIND9 (1:9.10.3.dfsg.P4-12.3+deb9u4), we see
haproxy downing backend servers with
"Server<foo> is going DOWN for maintenance (unspecified DNS error)."
https://github.com/haproxy/haproxy/search?q=unspecified+dns+error

We're expecting/testing to see if bind9's "no-case-compress { any; }"
directive
addresses this, but many folks do not control their DNS services (and as
requisite
AWS/Route53 capabilities mature, neither will we).


On Tue, Apr 10, 2018 at 3:11 PM, Ben Draut <[email protected]> wrote:

> It's interesting that the default behavior of HAProxy resolvers can
> conflict with the default behavior of bind. (If you're unlucky with
> whatever bind has cached)
>
> By default, bind uses case-insensitive compression, which can cause it to
> use a different case in the ANSWER than in the QUESTION. (See
> 'no-case-compress': https://ftp.isc.org/isc/bind9/cur/9.9/
> doc/arm/Bv9ARM.ch06.html) We were impacted by this recently.
>
> Also interesting: https://indico.dns-oarc.net/event/20/session/
> 2/contribution/12/material/slides/0.pdf
>
>
> On Mon, Apr 9, 2018 at 2:12 AM, Baptiste <[email protected]> wrote:
>
>> So, it seems that responses that does not match the case should be
>> dropped:
>> https://twitter.com/PowerDNS_Bert/status/983254222694240257
>>
>> Baptiste
>>
>
>
Ben Draut
Re: DNS resolver and mixed case responses
April 12, 2018 04:20PM
At the moment, AWS's provided DNS servers and Route53 appear to always
match the question case in the answer. (As do Google's DNS servers)

Bind seems to be the odd man out in not doing that by default.

On Thu, Apr 12, 2018 at 7:08 AM, Jim Freeman <[email protected]> wrote:

> It will be important to know which behavior AWS's Route53/DNS servers use ?
>
> Using stock Debian/Stretch BIND9 (1:9.10.3.dfsg.P4-12.3+deb9u4), we see
> haproxy downing backend servers with
> "Server<foo> is going DOWN for maintenance (unspecified DNS error)."
> https://github.com/haproxy/haproxy/search?q=unspecified+dns+error
>
> We're expecting/testing to see if bind9's "no-case-compress { any; }"
> directive
> addresses this, but many folks do not control their DNS services (and as
> requisite
> AWS/Route53 capabilities mature, neither will we).
>
>
> On Tue, Apr 10, 2018 at 3:11 PM, Ben Draut <[email protected]> wrote:
>
>> It's interesting that the default behavior of HAProxy resolvers can
>> conflict with the default behavior of bind. (If you're unlucky with
>> whatever bind has cached)
>>
>> By default, bind uses case-insensitive compression, which can cause it to
>> use a different case in the ANSWER than in the QUESTION. (See
>> 'no-case-compress': https://ftp.isc.org/isc/bind9/cur/9.9/do
>> c/arm/Bv9ARM.ch06.html) We were impacted by this recently.
>>
>> Also interesting: https://indico.dns-oarc.net/event/20/session/2/
>> contribution/12/material/slides/0.pdf
>>
>>
>> On Mon, Apr 9, 2018 at 2:12 AM, Baptiste <[email protected]> wrote:
>>
>>> So, it seems that responses that does not match the case should be
>>> dropped:
>>> https://twitter.com/PowerDNS_Bert/status/983254222694240257
>>>
>>> Baptiste
>>>
>>
>>
>
Baptiste
Re: DNS resolver and mixed case responses
April 17, 2018 04:10PM
Hi all,

Thanks a lot for your various investigations!
As a conclusion, HAProxy's behavior is "as expected".

Baptiste
Sorry, only registered users may post in this forum.

Click here to login