Michael Schwartzkopff
Problem: Connect() failed for backend: no free ports.
November 06, 2017 10:20PM
Hi,

I have a problem setting up a haproxy 1.6.13 that starts several
processes. In the config I have nbroc 3. In the logs I find lots of
entries like:


haproxy[: Connect() failed for backend XXX: no free ports


Searching the mailing list this seems to be a known problem when the
kernel still thinks some ports are open but haproxy wants to reuse it. I
alreay set "option nolinger" but the error messages remain, especially
when I start haproxy.


Any other solution?


Mit freundlichen Grüßen,

--

[*] sys4 AG

https://sys4.de, +49 (89) 30 90 46 64
Schleißheimer Straße 26/MG,80333 München

Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
Vorstand: Patrick Ben Koetter, Marc Schiffbauer, Wolfgang Stief
Aufsichtsratsvorsitzender: Florian Kirstein
On Mon, Nov 6, 2017 at 10:14 PM, Michael Schwartzkopff <[email protected]> wrote:

> Hi,
>
> I have a problem setting up a haproxy 1.6.13 that starts several
> processes. In the config I have nbroc 3. In the logs I find lots of
> entries like:
>
>
> haproxy[: Connect() failed for backend XXX: no free ports
>
>
> Searching the mailing list this seems to be a known problem when the
> kernel still thinks some ports are open but haproxy wants to reuse it. I
> alreay set "option nolinger" but the error messages remain, especially
> when I start haproxy.
>
>
> Any other solution?
>
>
Hi Michael,

Maybe you could tell us more about your workload and share with us your
configuration.
This will help the diagnostic.
Also, can you confirm you tuned some sysctls? (I mainly think about the
port range one)

Baptiste
Michael Schwartzkopff
Re: Problem: Connect() failed for backend: no free ports.
November 06, 2017 11:00PM
Am 06.11.2017 um 22:39 schrieb Baptiste:
> On Mon, Nov 6, 2017 at 10:14 PM, Michael Schwartzkopff <[email protected]> wrote:
>
>> Hi,
>>
>> I have a problem setting up a haproxy 1.6.13 that starts several
>> processes. In the config I have nbroc 3. In the logs I find lots of
>> entries like:
>>
>>
>> haproxy[: Connect() failed for backend XXX: no free ports
>>
>>
>> Searching the mailing list this seems to be a known problem when the
>> kernel still thinks some ports are open but haproxy wants to reuse it. I
>> alreay set "option nolinger" but the error messages remain, especially
>> when I start haproxy.
>>
>>
>> Any other solution?
>>
>>
> Hi Michael,
>
> Maybe you could tell us more about your workload and share with us your
> configuration.
> This will help the diagnostic.
> Also, can you confirm you tuned some sysctls? (I mainly think about the
> port range one)
>
> Baptiste
>
global
  maxconn 2000000
  nbproc 3
  cpu-map 1 0
  cpu-map 2 1
  cpu-map 3 2

defaults
  mode          tcp
  option        tcplog
  option        dontlognull
  option        dontlog-normal
  option        redispatch
  option        nolinger
  balance       leastconn
  retries       5

frontend IMAP-fe
  bind <myIP>:143 name IMAP tcp-ut 30s
  default_backend IMAP-be
  maxconn 400000

backend IMAP-be
  option tcp-check
  tcp-check connect port 143
  tcp-check expect string * OK
  default-server on-marked-down shutdown-sessions
  fullconn 400000
  server proxy01 192.168.0.101 source 192.168.0.201:10000-60000 check
  server proxy02 192.168.0.102 source 192.168.0.202:10000-60000 check
  server proxy03 192.168.0.103 source 192.168.0.203:10000-60000 check
  server proxy04 192.168.0.104 source 192.168.0.204:10000-60000 check
  (...)


And yes, sysctl is adjusted. Interesting enough, the errors above only
appear in my test when I start haproxy. After some flapping haproxy does
not emit any further log entries.


Mit freundlichen Grüßen,

--

[*] sys4 AG

https://sys4.de, +49 (89) 30 90 46 64
Schleißheimer Straße 26/MG,80333 München

Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
Vorstand: Patrick Ben Koetter, Marc Schiffbauer, Wolfgang Stief
Aufsichtsratsvorsitzender: Florian Kirstein
Lukas Tribus
Re: Problem: Connect() failed for backend: no free ports.
November 06, 2017 11:50PM
Hallo Michael,



2017-11-06 22:47 GMT+01:00 Michael Schwartzkopff <[email protected]>:
> Am 06.11.2017 um 22:39 schrieb Baptiste:
>> On Mon, Nov 6, 2017 at 10:14 PM, Michael Schwartzkopff <[email protected]> wrote:
>>
>>> Hi,
>>>
>>> I have a problem setting up a haproxy 1.6.13 that starts several
>>> processes. In the config I have nbroc 3. In the logs I find lots of
>>> entries like:
>>>
>>> haproxy[: Connect() failed for backend XXX: no free ports
> global
> maxconn 2000000
> nbproc 3
> cpu-map 1 0
> cpu-map 2 1
> cpu-map 3 2
> [...]
> backend IMAP-be
> option tcp-check
> tcp-check connect port 143
> tcp-check expect string * OK
> default-server on-marked-down shutdown-sessions
> fullconn 400000
> server proxy01 192.168.0.101 source 192.168.0.201:10000-60000 check
> server proxy02 192.168.0.102 source 192.168.0.202:10000-60000 check
> server proxy03 192.168.0.103 source 192.168.0.203:10000-60000 check
> server proxy04 192.168.0.104 source 192.168.0.204:10000-60000 check


You are using multiprocess mode together with static source port
ranges. That's a bad idea, because the processes will compete for the
same exact source ports and the syscalls will continue to fail as
different processes are trying to use the same ports.

There are a few possibilities here, but we will have to know:

- why are you using different source IP's for each backend server?
- why are you using static port ranges?

What I would suggest is to make sure that the kernel does the source
port selection, but the kernel needs to be able to use the full
5-tuple at this point, otherwise I imagine you'd run into source port
exhaustion soon.

If you don't require specific source IP's per server, than just remove
the "source ip:port-range" keyword altogether, the kernel will take
care of everything. Just make sure that your sysctls permit a similar
source port range.

If you need specific source IPs (for reasons unrelated to source port
exhaustion), then drop the port range and specify only the IP. However
for the kernel to be able to use the full 5-table, you will need
IP_BIND_ADDRESS_NO_PORT [1], which requires haproxy 1.7, linux 4.2 and
libc 2.23.



> And yes, sysctl is adjusted. Interesting enough, the errors above only
> appear in my test when I start haproxy. After some flapping haproxy does
> not emit any further log entries.

Still, this is a recipe for disaster. Haproxy is fighting among its
own processes on the back of the kernel. I'd advise against using this
configuration in production.


cheers,
lukas


[1] https://github.com/torvalds/linux/commit/90c337da
Michael Schwartzkopff
Re: Problem: Connect() failed for backend: no free ports.
November 07, 2017 03:40PM
Am 06.11.2017 um 23:43 schrieb Lukas Tribus:
> Hallo Michael,
>
>
>
> 2017-11-06 22:47 GMT+01:00 Michael Schwartzkopff <[email protected]>:
>> Am 06.11.2017 um 22:39 schrieb Baptiste:
>>> On Mon, Nov 6, 2017 at 10:14 PM, Michael Schwartzkopff <[email protected]> wrote:
>>>
>>>> Hi,
>>>>
>>>> I have a problem setting up a haproxy 1.6.13 that starts several
>>>> processes. In the config I have nbroc 3. In the logs I find lots of
>>>> entries like:
>>>>
>>>> haproxy[: Connect() failed for backend XXX: no free ports
>> global
>> maxconn 2000000
>> nbproc 3
>> cpu-map 1 0
>> cpu-map 2 1
>> cpu-map 3 2
>> [...]
>> backend IMAP-be
>> option tcp-check
>> tcp-check connect port 143
>> tcp-check expect string * OK
>> default-server on-marked-down shutdown-sessions
>> fullconn 400000
>> server proxy01 192.168.0.101 source 192.168.0.201:10000-60000 check
>> server proxy02 192.168.0.102 source 192.168.0.202:10000-60000 check
>> server proxy03 192.168.0.103 source 192.168.0.203:10000-60000 check
>> server proxy04 192.168.0.104 source 192.168.0.204:10000-60000 check
>
> You are using multiprocess mode together with static source port
> ranges. That's a bad idea, because the processes will compete for the
> same exact source ports and the syscalls will continue to fail as
> different processes are trying to use the same ports.
>
> There are a few possibilities here, but we will have to know:
>
> - why are you using different source IP's for each backend server?
> - why are you using static port ranges?
>
> What I would suggest is to make sure that the kernel does the source
> port selection, but the kernel needs to be able to use the full
> 5-tuple at this point, otherwise I imagine you'd run into source port
> exhaustion soon.
>
> If you don't require specific source IP's per server, than just remove
> the "source ip:port-range" keyword altogether, the kernel will take
> care of everything. Just make sure that your sysctls permit a similar
> source port range.

thanks. That helps.


> If you need specific source IPs (for reasons unrelated to source port
> exhaustion), then drop the port range and specify only the IP. However
> for the kernel to be able to use the full 5-table, you will need
> IP_BIND_ADDRESS_NO_PORT [1], which requires haproxy 1.7, linux 4.2 and
> libc 2.23.

We will see if we can install a 4.2 or later kernel.


Mit freundlichen Grüßen,

--

[*] sys4 AG

https://sys4.de, +49 (89) 30 90 46 64
Schleißheimer Straße 26/MG,80333 München

Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
Vorstand: Patrick Ben Koetter, Marc Schiffbauer, Wolfgang Stief
Aufsichtsratsvorsitzender: Florian Kirstein
Lukas Tribus
Re: Problem: Connect() failed for backend: no free ports.
November 07, 2017 03:50PM
Hello,


>> If you don't require specific source IP's per server, than just remove
>> the "source ip:port-range" keyword altogether, the kernel will take
>> care of everything. Just make sure that your sysctls permit a similar
>> source port range.
>
> thanks. That helps.
>
>
>> If you need specific source IPs (for reasons unrelated to source port
>> exhaustion), then drop the port range and specify only the IP. However
>> for the kernel to be able to use the full 5-table, you will need
>> IP_BIND_ADDRESS_NO_PORT [1], which requires haproxy 1.7, linux 4.2 and
>> libc 2.23.
>
> We will see if we can install a 4.2 or later kernel.

This is only necessary if you need the to use specific source IPs. If
you can remove the source keyword completely, then you don't need to
do this at all.


Grüße,
Lukas
Sorry, only registered users may post in this forum.

Click here to login