Welcome! Log In Create A New Profile

Advanced

cannot bind socket - Need help with config file

Posted by Imam Toufique 
Imam Toufique
cannot bind socket - Need help with config file
January 08, 2018 09:30AM
Hi,

I need some help figuring out why my config below is failing to start the
haproxy daemon. I am totally new to this.

Below is my confg:


global
# local2.* /var/log/haproxy.log
#
log 127.0.0.1 local2
#local2.* /var/log/haproxy.log
chroot /var/log/haproxy
#stats timeout 30s
user haproxy
group haproxy
daemon

defaults
log global
mode tcp
option tcplog
option dontlognull
timeout connect 5000
timeout client 50000
timeout server 50000


frontend sftp-server
bind *:22
default_backend sftp_server
timeout client 1h


listen stats 10.0.15.23:22
bind :22
mode tcp
maxconn 2000
option redis-check
retries 3
option redispatch
balance roundrobin

use_backend sftp_server
backend sftp_server
balance roundrobin
server web 10.0.15.21:22 check weight 2
server nagios 10.0.15.15:22 check weight 2

When I run a config check, i get this:

[[email protected] haproxy]# haproxy -f ./haproxy.cfg -c
Configuration file is valid

when I try to start haproxy, I get the following error:

[[email protected] haproxy]# haproxy -f ./haproxy.cfg -d
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result FAILED
Total: 3 (2 usable), will use epoll.
Using epoll() as the polling mechanism.
[ALERT] 007/081940 (1416) : Starting frontend sftp-server: cannot bind
socket [0.0.0.0:22]
[ALERT] 007/081940 (1416) : Starting proxy stats: cannot bind socket [
10.0.15.23:22]
[ALERT] 007/081940 (1416) : Starting proxy stats: cannot bind socket [
0.0.0.0:22]

In the config above, I am trying to setup 2 SFTP servers load-balanced with
haproxy. I would like to use port 22 , for sftp.

Please help, I need to get this going.

thanks.
Jonathan Matthews
Re: cannot bind socket - Need help with config file
January 08, 2018 11:30AM
On Mon, 8 Jan 2018 at 08:29, Imam Toufique <[email protected]> wrote:

> [ALERT] 007/081940 (1416) : Starting frontend sftp-server: cannot bind
> socket [0.0.0.0:22]
> [ALERT] 007/081940 (1416) : Starting proxy stats: cannot bind socket [
> 10.0.15.23:22]
> [ALERT] 007/081940 (1416) : Starting proxy stats: cannot bind socket [
> 0.0.0.0:22]
>

I would strongly suspect that the server already has something bound to
port 22. It's probably your SSH daemon.

You'll need to fix that, by dedicating either a different port or interface
to the SFTP listener.

J

> --
Jonathan Matthews
London, UK
http://www.jpluscplusm.com/contact.html
Lukas Tribus
Re: cannot bind socket - Need help with config file
January 08, 2018 11:50AM
Hello Imam,


On Mon, Jan 8, 2018 at 11:24 AM, Jonathan Matthews
<[email protected]> wrote:
> On Mon, 8 Jan 2018 at 08:29, Imam Toufique <[email protected]> wrote:
>>
>> [ALERT] 007/081940 (1416) : Starting frontend sftp-server: cannot bind
>> socket [0.0.0.0:22]
>> [ALERT] 007/081940 (1416) : Starting proxy stats: cannot bind socket
>> [10.0.15.23:22]
>> [ALERT] 007/081940 (1416) : Starting proxy stats: cannot bind socket
>> [0.0.0.0:22]
>
>
> I would strongly suspect that the server already has something bound to port
> 22. It's probably your SSH daemon.
>
> You'll need to fix that, by dedicating either a different port or interface
> to the SFTP listener.

Correct.

Also:
- you can't bind the stats socket to the same port as your actual frontend
- you are binding twice for the stats socket already (you must not
have "bind :ABC" AND listen stats 1.2.3.4:ABC as that will cause 2
different sockets to be created - don't specify IP and port in the
"listen" line to avoid that kind of confusing)


Lukas
Lukas Tribus
Re: cannot bind socket - Need help with config file
January 09, 2018 11:20AM
Hello Imam,


On Tue, Jan 9, 2018 at 2:30 AM, Imam Toufique <[email protected]> wrote:
>
> Hi Jonathan, and Lucas,
>
> Thanks for your replies. With your help, I was able to get it work
> partially.

Please always CC the mailing list though.



> frontend main *:2200
> #bind *:22
> default_backend sftp
> timeout client 1h

While this works, it's causing a lot of confusion. Please do follow my
advice and DON'T specify the port in the frontend/listen line. Use the
bind directive instead.
So in this case:

> frontend main
> bind :2200
> default_backend sftp
> timeout client 1h

It's much more readable like this.



> listen stats
> #bind *:22

You disbled your stats section with this configuration. Either decide
for a port, or remove it if you don't need it.



> But haproxy starts and I was able to get ssh to one of the servers. Now I
> have a different problem where I get a ssh ket fingerprint error warning and
> my connection drops.
>
> I get the error below:
>
> [[email protected] ~]$ ssh file -p 2200
> @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
> @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
> @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
> IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
> Someone could be eavesdropping on you right now (man-in-the-middle attack)!
> It is also possible that a host key has just been changed.
> The fingerprint for the RSA key sent by the remote host is
> SHA256:MHkXThp4cSltDn0/mRsq7Se+qcDz6cz1dD+kCiyE9e0.
> Please contact your system administrator.
> Add correct host key in /home/vagrant/.ssh/known_hosts to get rid of this
> message.
> Offending ECDSA key in /home/vagrant/.ssh/known_hosts:4
> RSA host key for [file]:2200 has changed and you have requested strict
> checking.
> Host key verification failed
>
> It looks like host keys are changing, and the host key becomes unknown to
> both servers that are behind HAProxy. what do you recommend doing in a case
> like this?

That's what happens when you load-balance between 2 different SSH
servers with a different private key. What is it that you want to
achieve in the first place?



cheers,
lukas
Imam Toufique
Re: cannot bind socket - Need help with config file
January 09, 2018 07:00PM
Hi Lukus,

thanks again for your continued help and support! Here is my config file
with updates now:

frontend main
bind :2200
default_backend sftp
timeout client 5d


listen stats
bind *:2200
mode tcp
maxconn 2000
option redis-check
retries 3
option redispatch
balance roundrobin


Please correct me if you see something that is not right.

You asked about my SSH/SFTP use-case. Basically, here is my use-case. I
have several SFTP servers that I would like to load-balance. I was
thinking about using HAProxy to load-balance SFTP connections between my
SFTP servers. As I was testing my setup yesterday, I was sending sftp file
transfers to the HAproxy node, I noticed that HAProxy node CPU usage was
pretty high. I am beginning to wonder if it is the right setup for my
environment.
Is HAProxy is the right solution for SFTP server load-balancing?

thanks

On Tue, Jan 9, 2018 at 2:12 AM, Lukas Tribus <[email protected]> wrote:

> Hello Imam,
>
>
> On Tue, Jan 9, 2018 at 2:30 AM, Imam Toufique <[email protected]> wrote:
> >
> > Hi Jonathan, and Lucas,
> >
> > Thanks for your replies. With your help, I was able to get it work
> > partially.
>
> Please always CC the mailing list though.
>
>
>
> > frontend main *:2200
> > #bind *:22
> > default_backend sftp
> > timeout client 1h
>
> While this works, it's causing a lot of confusion. Please do follow my
> advice and DON'T specify the port in the frontend/listen line. Use the
> bind directive instead.
> So in this case:
>
> > frontend main
> > bind :2200
> > default_backend sftp
> > timeout client 1h
>
> It's much more readable like this.
>
>
>
> > listen stats
> > #bind *:22
>
> You disbled your stats section with this configuration. Either decide
> for a port, or remove it if you don't need it.
>
>
>
> > But haproxy starts and I was able to get ssh to one of the servers. Now
> I
> > have a different problem where I get a ssh ket fingerprint error warning
> and
> > my connection drops.
> >
> > I get the error below:
> >
> > [[email protected] ~]$ ssh file -p 2200
> > @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
> > @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
> > @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
> > IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
> > Someone could be eavesdropping on you right now (man-in-the-middle
> attack)!
> > It is also possible that a host key has just been changed.
> > The fingerprint for the RSA key sent by the remote host is
> > SHA256:MHkXThp4cSltDn0/mRsq7Se+qcDz6cz1dD+kCiyE9e0.
> > Please contact your system administrator.
> > Add correct host key in /home/vagrant/.ssh/known_hosts to get rid of this
> > message.
> > Offending ECDSA key in /home/vagrant/.ssh/known_hosts:4
> > RSA host key for [file]:2200 has changed and you have requested strict
> > checking.
> > Host key verification failed
> >
> > It looks like host keys are changing, and the host key becomes unknown to
> > both servers that are behind HAProxy. what do you recommend doing in a
> case
> > like this?
>
> That's what happens when you load-balance between 2 different SSH
> servers with a different private key. What is it that you want to
> achieve in the first place?
>
>
>
> cheers,
> lukas
>



--
Regards,
*Imam Toufique*
*213-700-5485*
Lukas Tribus
Re: cannot bind socket - Need help with config file
January 10, 2018 05:30PM
Hi Imam,


On Tue, Jan 9, 2018 at 6:54 PM, Imam Toufique <[email protected]> wrote:
> Hi Lukus,
>
> thanks again for your continued help and support! Here is my config file
> with updates now:
>
> frontend main
> bind :2200
> default_backend sftp
> timeout client 5d
>
>
> listen stats
> bind *:2200
> mode tcp
> maxconn 2000
> option redis-check
> retries 3
> option redispatch
> balance roundrobin
>
>
> Please correct me if you see something that is not right.

That's wrong. You are again configuring 2 services on a single port.
In this case, the kernel will load-balance between the two causing
chaos.

What is the "listen stats" section supposed to do anyway in your
configuration? Why do you need a main frontend and this listen
section?



> You asked about my SSH/SFTP use-case. Basically, here is my use-case. I
> have several SFTP servers that I would like to load-balance. I was thinking
> about using HAProxy to load-balance SFTP connections between my SFTP
> servers. As I was testing my setup yesterday, I was sending sftp file
> transfers to the HAproxy node, I noticed that HAProxy node CPU usage was
> pretty high. I am beginning to wonder if it is the right setup for my
> environment.
> Is HAProxy is the right solution for SFTP server load-balancing?

Load-balancing SSH/SFTP generally should be very easy to do, as SSH
only uses a single port and doesn't have any layering violations (as
opposed to FTP).
The only thing to be aware of is the public key issue with different
servers, as you are load-balancing between them. Use the same private
key on all the backend server to avoid this problem.

As for the high CPU usage, I'd recommend fixing the configuration
first, before troubleshooting the CPU load. You may see strange
effects due to unintended load-balancing.


The rule is is simple: you are specifying the same listening port more
than once in the configuration, then something is and will go wrong.
You must have one single reference to port 2200 only.



Lukas
Lukas Tribus
Re: cannot bind socket - Need help with config file
January 11, 2018 01:10AM
Hello Imam,


On Wed, Jan 10, 2018 at 11:49 PM, Imam Toufique <[email protected]> wrote:
> Lukas,
>
> Sorry to keep on dragging this, I am confused here. I will admit that I
> have not had the time to read the documentation on this. From what I was
> able to read, I slapped togather this config to get me started.
>
> I am not sure exactly what the 'listen' part do. From what I can gather, I
> found this in the user documentation:

Again please "Reply-All" so the mailing list remains CC'ed.


The frontend and listen functionality overlap, they can do the same
thing, with a slightly different syntax. You either use a frontend OR
a listen section. You don't use both for the same exact purpose.

The frontend is fine, just delete everything related to the listen
section and that's it.




Regards,
Lukas
Imam Toufique
Re: cannot bind socket - Need help with config file
January 11, 2018 01:10AM
Thanks, Lukas! Sorry, I think I have been just replying to you by
accidentally hitting the 'reply' button.

So, I have everything in the listen section commented out:

frontend main
bind :2200
default_backend sftp
timeout client 5d


#listen stats
# bind *:2200
# mode tcp
# maxconn 2000
# option redis-check
# retries 3
# option redispatch
# balance roundrobin

#use_backend sftp_server
backend sftp
balance roundrobin
server web 10.0.15.21:2200 check weight 2
server nagios 10.0.15.15:2200 check weight 2

Is that what I need, right?

thanks.

On Wed, Jan 10, 2018 at 4:00 PM, Lukas Tribus <[email protected]> wrote:

> Hello Imam,
>
>
> On Wed, Jan 10, 2018 at 11:49 PM, Imam Toufique <[email protected]>
> wrote:
> > Lukas,
> >
> > Sorry to keep on dragging this, I am confused here. I will admit that I
> > have not had the time to read the documentation on this. From what I was
> > able to read, I slapped togather this config to get me started.
> >
> > I am not sure exactly what the 'listen' part do. From what I can
> gather, I
> > found this in the user documentation:
>
> Again please "Reply-All" so the mailing list remains CC'ed.
>
>
> The frontend and listen functionality overlap, they can do the same
> thing, with a slightly different syntax. You either use a frontend OR
> a listen section. You don't use both for the same exact purpose.
>
> The frontend is fine, just delete everything related to the listen
> section and that's it.
>
>
>
>
> Regards,
> Lukas
>



--
Regards,
*Imam Toufique*
*213-700-5485*
Jonathan Matthews
Re: cannot bind socket - Need help with config file
January 11, 2018 04:40PM
On 11 January 2018 at 00:03, Imam Toufique <[email protected]> wrote:
> So, I have everything in the listen section commented out:
>
> frontend main
> bind :2200
> default_backend sftp
> timeout client 5d
>
>
> #listen stats
> # bind *:2200
> # mode tcp
> # maxconn 2000
> # option redis-check
> # retries 3
> # option redispatch
> # balance roundrobin
>
> #use_backend sftp_server
> backend sftp
> balance roundrobin
> server web 10.0.15.21:2200 check weight 2
> server nagios 10.0.15.15:2200 check weight 2
>
> Is that what I need, right?

I suspect you won't need to have your *backend*'s ports changed to
2200. Your SSH server on those machines is *probably* also your SFTP
server. I don't recall if you can serve a different/sync'd host key
per port in sshd, but this might be a reason to run a different daemon
on a higher port as you're doing.

As an aside, it's not clear why you're trying to do this. You've
already hit the host-key-changing problem, and unless you have a
*very* specific use case, your users will hit the "50% of the time I
connect, my files have gone away" problem soon. So you've probably got
to solve the shared-storage problem on your backends ... which turns
them in to stateless SFTP-to-FS servers.

In my opinion adding haproxy as a TCP proxy in your architecture adds
very little, if anything. If I were you, I'd strongly consider just
sync'ing the same host key to each server, putting their IPs in a
low-TTL DNS record, and leaving haproxy out of the setup.

J
Lukas Tribus
Re: cannot bind socket - Need help with config file
January 11, 2018 07:10PM
Hello,


On 11 January 2018 at 16:36, Jonathan Matthews <[email protected]> wrote:
> On 11 January 2018 at 00:03, Imam Toufique <[email protected]> wrote:
>> So, I have everything in the listen section commented out:
>>
>> frontend main
>> bind :2200
>> default_backend sftp
>> timeout client 5d
>>
>>
>> #listen stats
>> # bind *:2200
>> # mode tcp
>> # maxconn 2000
>> # option redis-check
>> # retries 3
>> # option redispatch
>> # balance roundrobin
>>
>> #use_backend sftp_server
>> backend sftp
>> balance roundrobin
>> server web 10.0.15.21:2200 check weight 2
>> server nagios 10.0.15.15:2200 check weight 2
>>
>> Is that what I need, right?
>
> I suspect you won't need to have your *backend*'s ports changed to
> 2200. Your SSH server on those machines is *probably* also your SFTP
> server

That's exactly right, your backend destination port should probably
22, there is no need to bump that one to 2200.



> As an aside, it's not clear why you're trying to do this. You've
> already hit the host-key-changing problem, and unless you have a
> *very* specific use case, your users will hit the "50% of the time I
> connect, my files have gone away" problem soon. So you've probably got
> to solve the shared-storage problem on your backends ... which turns
> them in to stateless SFTP-to-FS servers.
>
> In my opinion adding haproxy as a TCP proxy in your architecture adds
> very little, if anything. If I were you, I'd strongly consider just
> sync'ing the same host key to each server, putting their IPs in a
> low-TTL DNS record, and leaving haproxy out of the setup.

With DNS round-robin instead of haproxy you have the same exact
requirements regarding SSH keys and filesystem synchronization, with
all the disadvantages (no health checks, no direct control of the
actual load-balancing, no stats, no logs, etc).

I'm really not sure why you'd recommend DNS RR instead of haproxy
here. Load-balancing a single-port TCP protocol between 2 backends is
a bread and butter use-case for haproxy.



Regards,
Lukas
Sorry, only registered users may post in this forum.

Click here to login