Welcome! Log In Create A New Profile

Advanced

Trouble with ECC/RSA shared IP/port SSL setup and using unix sockets (localhost method works)

Posted by Vitaly Pecharsky 
Hello

We have been trying to test a shared IP/port ECC/RSA SSL
implementation that is available in HAProxy, and largely followed this
basic setup guide
http://blog.haproxy.com/2015/07/15/serving-ecc-and-rsa-certificates-on-same-ip-with-haproxy/
and adapted it to our setup.

It works fine using a localhost as a listener (see commented lines in
the config below), however with a [email protected] socket setup I can not seem to
get the setup to work.

Log messages while accessing the site using sockets are:

Jan 5 15:35:16 localhost haproxy[12713]: Z.Z.Z.Z:61393
[05/Jan/2017:15:35:16.502] example-ecc-rsa-relay
example-ecc-relay/example-ecc-rsa-in 2/-1/1 0 SC 0/0/0/0/5 0/0
Jan 5 15:35:16 localhost haproxy[12713]: Z.Z.Z.Z:61394
[05/Jan/2017:15:35:16.512] example-ecc-rsa-relay
example-ecc-relay/example-ecc-rsa-in 1/-1/0 0 SC 0/0/0/0/5 0/0
Jan 5 15:35:16 localhost haproxy[12713]: Z.Z.Z.Z:61395
[05/Jan/2017:15:35:16.732] example-ecc-rsa-relay
example-ecc-relay/example-ecc-rsa-in 1/-1/0 0 SC 0/0/0/0/5 0/0
Jan 5 15:35:16 localhost haproxy[12713]: Z.Z.Z.Z:61396
[05/Jan/2017:15:35:16.760] example-ecc-rsa-relay
example-ecc-relay/example-ecc-rsa-in 1/-1/0 0 SC 0/0/0/0/5 0/0
Jan 5 15:35:21 localhost haproxy[12713]: Z.Z.Z.Z:5909
[05/Jan/2017:15:35:21.779] example-ecc-rsa-relay
example-ecc-relay/example-ecc-rsa-in 1/-1/0 0 SC 0/0/0/0/5 0/0
Jan 5 15:35:21 localhost haproxy[12713]: Z.Z.Z.Z:61400
[05/Jan/2017:15:35:21.786] example-ecc-rsa-relay
example-ecc-relay/example-ecc-rsa-in 2/-1/1 0 SC 0/0/0/0/5 0/0

After doing some research before posting to this list, suggestions
here pointed to either a chroot or permissions issue with the socket.
I have tested removing (commenting out) chroot as well as setting
permissions on the socket to 777 as well as ensuring that the socket
is set to haproxy uid/gid via bind-unix global statement, and neither
seem to make any difference.

Does anyone have any additional suggestions to try or sees an obvious
thing I may have missed?

Thanks in advance!

HAProxy compile string:

haproxy -vv
HA-Proxy version 1.7.1 2016/12/13
Copyright 2000-2016 Willy Tarreau <[email protected]>

Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement
OPTIONS = USE_ZLIB=1 USE_OPENSSL=1 USE_PCRE=1

Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.3
Running on zlib version : 1.2.3
Compression algorithms supported : identity("identity"),
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with OpenSSL version : OpenSSL 1.1.0c 10 Nov 2016
Running on OpenSSL version : OpenSSL 1.1.0c 10 Nov 2016
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 7.8 2008-09-05
Running on PCRE version : 7.8 2008-09-05
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built without Lua support
Built with transparent proxy support using: IP_TRANSPARENT
IPV6_TRANSPARENT IP_FREEBIND

Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.

Available filters :
[SPOE] spoe
[TRACE] trace
[COMP] compression


Relevant HAProxy config:

global
log 127.0.0.1 local2 notice
pidfile /var/run/haproxy.pid
maxconn 400000
user haproxy
group haproxy
#chroot /var/lib/haproxy
unix-bind prefix /var/run/ mode 600 user haproxy group haproxy
#unix-bind prefix /var/run/ mode 777 user haproxy group haproxy
daemon
nbproc 23

tune.maxrewrite 1024
tune.bufsize 16384

cpu-map 1 1
cpu-map 2 2
cpu-map 3 3
cpu-map 4 4
cpu-map 5 5
cpu-map 6 6
cpu-map 7 7
cpu-map 8 8
cpu-map 9 9
cpu-map 10 10
cpu-map 11 11
cpu-map 12 12
cpu-map 13 13
cpu-map 14 14
cpu-map 15 15
cpu-map 16 16
cpu-map 17 17
cpu-map 18 18
cpu-map 19 19
cpu-map 20 20
cpu-map 21 21
cpu-map 22 22
cpu-map 23 23

stats socket /var/run/haproxy01.stat process 1
stats socket /var/run/haproxy02.stat process 2
stats socket /var/run/haproxy03.stat process 3
stats socket /var/run/haproxy04.stat process 4
stats socket /var/run/haproxy05.stat process 5
stats socket /var/run/haproxy06.stat process 6
stats socket /var/run/haproxy07.stat process 7
stats socket /var/run/haproxy08.stat process 8
stats socket /var/run/haproxy09.stat process 9
stats socket /var/run/haproxy10.stat process 10
stats socket /var/run/haproxy11.stat process 11
stats socket /var/run/haproxy12.stat process 12
stats socket /var/run/haproxy13.stat process 13
stats socket /var/run/haproxy14.stat process 14
stats socket /var/run/haproxy15.stat process 15
stats socket /var/run/haproxy16.stat process 16
stats socket /var/run/haproxy17.stat process 17
stats socket /var/run/haproxy18.stat process 18
stats socket /var/run/haproxy19.stat process 19
stats socket /var/run/haproxy20.stat process 20
stats socket /var/run/haproxy21.stat process 21
stats socket /var/run/haproxy22.stat process 22
stats socket /var/run/haproxy23.stat process 23

ca-base /etc/ssl/certs
crt-base /etc/ssl/certs

tune.ssl.cachesize 256000
tune.ssl.lifetime 900

ssl-default-bind-ciphers
ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS
ssl-default-bind-options no-sslv3 no-tls-tickets

tune.ssl.default-dh-param 2048


defaults
log global
option redispatch
retries 5
timeout http-request 30s
timeout queue 60s
timeout connect 30s
timeout client 60s
timeout server 60s
timeout http-keep-alive 30s
timeout check 30s
timeout tarpit 15s
maxconn 400000
errorfile 408 /dev/null


frontend example-ecc-rsa-relay
bind-process 12-13
bind X.X.X.X:443
mode tcp
option tcplog
option dontlog-normal
tcp-request inspect-delay 5s
log 127.0.0.1 local2 debug

acl ip_blacklist src -f /etc/haproxy/ip_blacklist.lst
tcp-request connection reject if ip_blacklist

tcp-request content accept if { req.ssl_ec_ext 0 }
use_backend example-ecc-relay if { req.ssl_ec_ext 1 }
default_backend example-rsa-relay

backend example-ecc-relay
bind-process 12-13
mode tcp
option tcplog
log 127.0.0.1 local2 debug
server example-ecc-rsa-in [email protected]_example_ecc_in.sock send-proxy
#server example_ecc_in 127.1.1.67:80 send-proxy

backend example-rsa-relay
bind-process 12-13
mode tcp
option tcplog
log 127.0.0.1 local2 debug
server example-ecc-rsa-in [email protected]_example_rsa_in.sock send-proxy
#server example_rsa_in 127.1.1.67:81 send-proxy

frontend example-ecc-rsa-in
bind [email protected]_example_ecc_in.sock ssl crt
../ecc_letsencrypt/example.com.ecc.san.dh2048.bundle.pem accept-proxy
bind [email protected]_example_rsa_in.sock ssl crt
../rsa_letsencrypt/example.com.rsa.san.dh2048.bundle.pem accept-proxy
#bind 127.1.1.67:80 accept-proxy name example_ecc_in ssl crt
../ecc_letsencrypt/example.com.ecc.san.dh2048.bundle.pem
#bind 127.1.1.67:81 accept-proxy name example_rsa_in ssl crt
../rsa_letsencrypt/example.com.rsa.san.dh2048.bundle.pem
bind-process 12-13
mode tcp
option tcplog
log 127.0.0.1 local2 debug
option dontlog-normal

default_backend example-ssl-out

backend example-ssl-out
bind-process 11
mode http
option httplog
log 127.0.0.1 local2 debug

http-request set-header X-Forced-Ssl on
http-request set-header X-Forwarded-Proto https
http-request set-header X-Forwarded-Ssl on

rspadd Strict-Transport-Security:\ max-age=15552000

server example-ssl-proxy-in-070 127.1.1.70:80 send-proxy

frontend example-http-in
bind-process 11
bind 127.1.1.70:80 accept-proxy name example_ssl_proxy_in_070
bind 1X.X.X.Y:80 name example_http_in
bind 1X.X.X.X:80 name ecc_http_in
mode http
option httplog
option dontlog-normal
log 127.0.0.1 local2 debug
option forwardfor if-none
option http-server-close
tcp-request inspect-delay 5s
default_backend example-http-out

backend example-http-out
bind-process 11
mode http
option httplog
log 127.0.0.1 local2 debug
option http-server-close
option httpchk OPTIONS * HTTP/1.1\r\nHost:\ example.com
balance roundrobin
http-request allow
server www001 10.2.51.1:80 check inter 5s rise 1 fall 1

Socket permissions (with chmod 777):

srwxrwxrwx 1 haproxy haproxy 0 Jan 5 15:46 haproxy_example_ecc_in.sock
srwxrwxrwx 1 haproxy haproxy 0 Jan 5 15:46 haproxy_example_rsa_in.sock
Hello,

On 1/6/2017 1:55 AM, Vitaly Pecharsky wrote:
> haproxy -vv
> HA-Proxy version 1.7.1 2016/12/13
> Copyright 2000-2016 Willy Tarreau <[email protected]>

As you are running 1.7 and OpenSSL 1.1.0, you don't need to do this any
more. HAProxy can now natively support ECC/RSA/DSA based on client
support. Check
https://cbonte.github.io/haproxy-dconv/configuration-1.7.html#5.1-crt

> unix-bind prefix /var/run/ mode 600 user haproxy group haproxy

unix-bind only affects bind lines and because of this:

> server example-ecc-rsa-in [email protected]_example_rsa_in.sock send-proxy

You need to change this to [email protected]/var/run/haproxy_example_rsa_in.sock.

Regards,
Nenad
Nenad

That makes total sense - and solved the issue with sockets like a charm.

Thanks for the tip on combining the certs, that makes configuration
even simpler - that's the approach I am going with for production
setup. No more multi-chained backends, yay!



On Thu, Jan 5, 2017 at 7:06 PM, Nenad Merdanovic <[email protected]> wrote:
> Hello,
>
> On 1/6/2017 1:55 AM, Vitaly Pecharsky wrote:
>> haproxy -vv
>> HA-Proxy version 1.7.1 2016/12/13
>> Copyright 2000-2016 Willy Tarreau <[email protected]>
>
> As you are running 1.7 and OpenSSL 1.1.0, you don't need to do this any
> more. HAProxy can now natively support ECC/RSA/DSA based on client
> support. Check
> https://cbonte.github.io/haproxy-dconv/configuration-1.7.html#5.1-crt
>
>> unix-bind prefix /var/run/ mode 600 user haproxy group haproxy
>
> unix-bind only affects bind lines and because of this:
>
>> server example-ecc-rsa-in [email protected]_example_rsa_in.sock send-proxy
>
> You need to change this to [email protected]/var/run/haproxy_example_rsa_in.sock.
>
> Regards,
> Nenad



--
Sincerely
Vitaly Pecharsky
mailto:[email protected]
Sorry, only registered users may post in this forum.

Click here to login