Welcome! Log In Create A New Profile

Advanced

Failed checks issue

Posted by Haim Ari 
Haim Ari
Failed checks issue
December 05, 2017 10:40AM
Hello,


After upgrading to 1.8.1 i'm now facing this issue... (screenshot attached)


[cid:1f0e0dc5-269b-4c6b-98ae-9885001f1781]


After restarting a tomcat backend the check remains at DOWN state and does not recover even though the check responds with HTTP 200 response from this haproxy:


[email protected]:~# curl -i http://web27:8480/keepalive


HTTP/1.1 200
Content-Length: 5
Date: Tue, 05 Dec 2017 09:24:34 GMT
Connection: close

if i reload/restart the haproxy, it becomes UP again

Here is the config:

global
tune.http.logurilen 65535
log /dev/log len 65535 local0

# to have these messages end up in /var/log/haproxy.log you will
# need to:
#
# 1) configure syslog to accept network log events. This is done
# by adding the '-r' option to the SYSLOGD_OPTIONS in
# /etc/sysconfig/syslog
#
# 2) configure local2 events to go to the /var/log/haproxy.log
# file. A line like the following can be added to
# /etc/sysconfig/syslog
#
# local2.* /var/log/haproxy.log

tune.ssl.default-dh-param 2048
#log 127.0.0.1 local2
#log 192.168.249.244:10514 local2
#log elk:10514 local2
#log-send-hostname

#log 127.0.0.1 local2
#log-send-hostname haproxy
#log 192.168.249.131 local2

chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 10000
user haproxy
group haproxy
daemon

stats socket :1996 level admin process 1 expose-fd listeners
stats socket :1997 level admin process 2 expose-fd listeners
stats socket :1998 level admin process 3 expose-fd listeners
stats socket :1999 level admin process 4 expose-fd listeners
nbproc 8

### CPU MAPS ###

# HTTP #
cpu-map 1 1-2
# HTTPS #
cpu-map 2 3-4
# SDP #
cpu-map 3 5-6
# TCP #
cpu-map 4 7-8

spread-checks 5


# turn on stats unix socket
#stats socket /var/lib/haproxy/stats level admin process 1


#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
#
defaults
mode http
log global
log-format "body: %[capture.req.hdr(0)]"
option httplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
option abortonclose
retries 3
timeout http-request 10s
timeout queue 2s
timeout connect 5s
timeout client 2m
timeout server 2m
timeout http-keep-alive 10s

frontend AdPlatform-In
option http-buffer-request
bind-process 1
#http-request set-log-level silent unless { path_beg -i /trackinstall } or { path_beg -i /trackpostinstall } or { hdr(host) -i nj.shooter.xxxxxxx.ccc }
bind 217.65.36.201:80
bind 217.65.36.202:80
bind 217.65.36.203:80
bind 217.65.36.204:80
maxconn 10000
timeout client-fin 10s
default_backend adplatform
capture request header User-Agent len 128
capture request header Referer len 64
capture request header Host len 64
capture request header X-Forwarded-For len 64
capture request header True-Client-IP len 500
capture request header X-EdgeConnect-Session-Id len 500
capture request header X-StartApp-Info len 500

backend adplatform
fullconn 2000
timeout queue 1s
option httpchk GET /keepalive
balance roundrobin
option abortonclose
#option http-server-close
option forwardfor
timeout server 5s
http-request set-header X-Forwarded-Port %[dst_port]
http-request add-header X-Forwarded-Proto https if { ssl_fc }

# server web13 192.168.249.53:8480 weight 80 check inter 10s minconn 10 maxconn 2 maxqueue 1 slowstart 120s
#server web22 192.168.249.148:8480 weight 50 check inter 10s minconn 10 maxconn 100 maxqueue 1 slowstart 120s
server web23 192.168.249.149:8480 weight 30 check inter 10s minconn 10 maxconn 100 maxqueue 1 slowstart 120s
server web24 192.168.249.150:8480 weight 30 check inter 10s minconn 10 maxconn 100 maxqueue 1 slowstart 120s
#server web25 192.168.249.151:8480 weight 150 check inter 10s minconn 10 maxconn 100 maxqueue 1 slowstart 120s
# server web26 192.168.249.152:8480 weight 150 check inter 10s minconn 10 maxconn 100 maxqueue 1 slowstart 120s
server web27 192.168.249.153:8480 weight 150 check inter 10s minconn 10 maxconn 100 maxqueue 1 slowstart 120s
server web31 192.168.249.181:8480 weight 180 check inter 10s minconn 10 maxconn 150 maxqueue 1 slowstart 120s
server web32 192.168.249.182:8480 weight 180 check inter 10s minconn 10 maxconn 150 maxqueue 1 slowstart 120s
server web33 192.168.249.183:8480 weight 180 check inter 10s minconn 10 maxconn 150 maxqueue 1 slowstart 120s
server web34 192.168.249.184:8480 weight 180 check inter 10s minconn 10 maxconn 150 maxqueue 1 slowstart 120s
server web41 192.168.249.61:8480 weight 215 check inter 10s minconn 10 maxconn 200 maxqueue 1 slowstart 120s
server web42 192.168.249.62:8480 weight 215 check inter 10s minconn 10 maxconn 200 maxqueue 1 slowstart 120s
server web43 192.168.249.63:8480 weight 215 check inter 10s minconn 10 maxconn 200 maxqueue 1 slowstart 120s
server web44 192.168.249.64:8480 weight 215 check inter 10s minconn 10 maxconn 200 maxqueue 1 slowstart 120s
server web45 192.168.249.65:8480 weight 215 check inter 10s minconn 10 maxconn 200 maxqueue 1 slowstart 120s
server web46 192.168.249.66:8480 weight 215 check inter 10s minconn 10 maxconn 200 maxqueue 1 slowstart 120s
server web47 192.168.249.67:8480 weight 215 check inter 10s minconn 10 maxconn 200 maxqueue 1 slowstart 120s
server web48 192.168.249.68:8480 weight 215 check inter 10s minconn 10 maxconn 200 maxqueue 1 slowstart 120s
server web51 192.168.249.101:8480 weight 220 check inter 10s minconn 10 maxconn 200 maxqueue 1 slowstart 120s
server web52 192.168.249.102:8480 weight 220 check inter 10s minconn 10 maxconn 200 maxqueue 1 slowstart 120s
server web53 192.168.249.103:8480 weight 220 check inter 10s minconn 10 maxconn 200 maxqueue 1 slowstart 120s
server web54 192.168.249.104:8480 weight 220 check inter 10s minconn 10 maxconn 200 maxqueue 1 slowstart 120s
server web55 192.168.249.105:8480 weight 220 check inter 10s minconn 10 maxconn 200 maxqueue 1 slowstart 120s
server web56 192.168.249.106:8480 weight 220 check inter 10s minconn 10 maxconn 200 maxqueue 1 slowstart 120s
server web57 192.168.249.107:8480 weight 220 check inter 10s minconn 10 maxconn 200 maxqueue 1 slowstart 120s
server web58 192.168.249.108:8480 weight 220 check inter 10s minconn 10 maxconn 200 maxqueue 1 slowstart 120s
server stg-web2 192.168.249.178:8480 weight 150 check inter 10s minconn 10 maxconn 100 maxqueue 1 slowstart 120s
# server stg-web1 192.168.249.171:8480 weight 80 check inter 10s minconn 10 maxconn 500 maxqueue 1 slowstart 120s

i see this issue on 4 haproxy nodes
Can you please advise ?







Haim Ari / SysOps Manager

M: 972.584563032 / T: 972.722288367



[https://www.startapp.com/wp-content/uploads/2017/11/signature-250-1.jpg] https://goo.gl/R83eLy
Attachments:
open | download - Screenshot from 2017-12-05 11-17-05.png (90.2 KB)
Aleksandar Lazic
Re: Failed checks issue
December 06, 2017 12:50AM
------ Originalnachricht ------
Von: "Haim Ari" <[email protected]>
An: "[email protected]" <[email protected]>
Gesendet: 05.12.2017 10:31:36
Betreff: Failed checks issue

>Hello,
>
>
>
>After upgrading to 1.8.1 i'm now facing this issue... (screenshot
>attached)
[snipp]

>
>After restarting a tomcat backend the check remains at DOWN state and
>does not recover even though the check responds with HTTP 200 response
>from this haproxy:
>
>
>
>[email protected]:~# curl -i http://web27:8480/keepalive
>
>
>
>
>HTTP/1.1 200
>Content-Length: 5
>Date: Tue, 05 Dec 2017 09:24:34 GMT
>Connection: close
>
>if i reload/restart the haproxy, it becomes UP again
>
>Here is the config:
>
>global
>tune.http.logurilen 65535
>log /dev/log len 65535 local0
>
> # to have these messages end up in /var/log/haproxy.log you will
> # need to:
> #
> # 1) configure syslog to accept network log events. This is done
> # by adding the '-r' option to the SYSLOGD_OPTIONS in
> # /etc/sysconfig/syslog
> #
> # 2) configure local2 events to go to the /var/log/haproxy.log
> # file. A line like the following can be added to
> # /etc/sysconfig/syslog
> #
> # local2.* /var/log/haproxy.log
>
> tune.ssl.default-dh-param 2048
> #log 127.0.0.1 local2
> #log 192.168.249.244:10514 local2
> #log elk:10514 local2
> #log-send-hostname
>
> #log 127.0.0.1 local2
> #log-send-hostname haproxy
> #log 192.168.249.131 local2
>
> chroot /var/lib/haproxy
> pidfile /var/run/haproxy.pid
> maxconn 10000
> user haproxy
> group haproxy
> daemon
>
> stats socket :1996 level admin process 1 expose-fd listeners
> stats socket :1997 level admin process 2 expose-fd listeners
> stats socket :1998 level admin process 3 expose-fd listeners
> stats socket :1999 level admin process 4 expose-fd listeners
> nbproc 8
>
> ### CPU MAPS ###
>
> # HTTP #
> cpu-map 1 1-2
> # HTTPS #
> cpu-map 2 3-4
> # SDP #
> cpu-map 3 5-6
> # TCP #
> cpu-map 4 7-8
>
> spread-checks 5
>
>
> # turn on stats unix socket
> #stats socket /var/lib/haproxy/stats level admin process 1
>
>
>#---------------------------------------------------------------------
># common defaults that all the 'listen' and 'backend' sections will
># use if not designated in their block
>#---------------------------------------------------------------------
>#
>defaults
> mode http
> log global
> log-format "body: %[capture.req.hdr(0)]"
> option httplog
> option dontlognull
> option http-server-close
> option forwardfor except 127.0.0.0/8
> option redispatch
> option abortonclose
> retries 3
> timeout http-request 10s
> timeout queue 2s
> timeout connect 5s
> timeout client 2m
> timeout server 2m
> timeout http-keep-alive 10s
>
>frontend AdPlatform-In
> option http-buffer-request
> bind-process 1
> #http-request set-log-level silent unless { path_beg -i
>/trackinstall } or { path_beg -i /trackpostinstall } or { hdr(host) -i
>nj.shooter.xxxxxxx.ccc }
> bind 217.65.36.201:80
> bind 217.65.36.202:80
> bind 217.65.36.203:80
> bind 217.65.36.204:80
> maxconn 10000
> timeout client-fin 10s
> default_backend adplatform
> capture request header User-Agent len 128
> capture request header Referer len 64
> capture request header Host len 64
> capture request header X-Forwarded-For len 64
> capture request header True-Client-IP len 500
> capture request header X-EdgeConnect-Session-Id len 500
> capture request header X-StartApp-Info len 500
>
>backend adplatform
> fullconn 2000
> timeout queue 1s
> option httpchk GET /keepalive
> balance roundrobin
> option abortonclose
> #option http-server-close
> option forwardfor
> timeout server 5s
> http-request set-header X-Forwarded-Port %[dst_port]
> http-request add-header X-Forwarded-Proto https if { ssl_fc }
>
> # server web13 192.168.249.53:8480 weight 80 check inter 10s
>minconn 10 maxconn 2 maxqueue 1 slowstart 120s
> #server web22 192.168.249.148:8480 weight 50 check inter 10s
>minconn 10 maxconn 100 maxqueue 1 slowstart 120s
> server web23 192.168.249.149:8480 weight 30 check inter 10s
>minconn 10 maxconn 100 maxqueue 1 slowstart 120s
> server web24 192.168.249.150:8480 weight 30 check inter 10s
>minconn 10 maxconn 100 maxqueue 1 slowstart 120s
> #server web25 192.168.249.151:8480 weight 150 check inter 10s
>minconn 10 maxconn 100 maxqueue 1 slowstart 120s
> # server web26 192.168.249.152:8480 weight 150 check inter 10s
>minconn 10 maxconn 100 maxqueue 1 slowstart 120s
> server web27 192.168.249.153:8480 weight 150 check inter 10s
>minconn 10 maxconn 100 maxqueue 1 slowstart 120s
> server web31 192.168.249.181:8480 weight 180 check inter 10s
>minconn 10 maxconn 150 maxqueue 1 slowstart 120s
> server web32 192.168.249.182:8480 weight 180 check inter 10s
>minconn 10 maxconn 150 maxqueue 1 slowstart 120s
> server web33 192.168.249.183:8480 weight 180 check inter 10s
>minconn 10 maxconn 150 maxqueue 1 slowstart 120s
> server web34 192.168.249.184:8480 weight 180 check inter 10s
>minconn 10 maxconn 150 maxqueue 1 slowstart 120s
> server web41 192.168.249.61:8480 weight 215 check inter 10s
>minconn 10 maxconn 200 maxqueue 1 slowstart 120s
> server web42 192.168.249.62:8480 weight 215 check inter 10s
>minconn 10 maxconn 200 maxqueue 1 slowstart 120s
> server web43 192.168.249.63:8480 weight 215 check inter 10s
>minconn 10 maxconn 200 maxqueue 1 slowstart 120s
> server web44 192.168.249.64:8480 weight 215 check inter 10s
>minconn 10 maxconn 200 maxqueue 1 slowstart 120s
> server web45 192.168.249.65:8480 weight 215 check inter 10s
>minconn 10 maxconn 200 maxqueue 1 slowstart 120s
> server web46 192.168.249.66:8480 weight 215 check inter 10s
>minconn 10 maxconn 200 maxqueue 1 slowstart 120s
> server web47 192.168.249.67:8480 weight 215 check inter 10s
>minconn 10 maxconn 200 maxqueue 1 slowstart 120s
> server web48 192.168.249.68:8480 weight 215 check inter 10s
>minconn 10 maxconn 200 maxqueue 1 slowstart 120s
> server web51 192.168.249.101:8480 weight 220 check inter 10s
>minconn 10 maxconn 200 maxqueue 1 slowstart 120s
> server web52 192.168.249.102:8480 weight 220 check inter 10s
>minconn 10 maxconn 200 maxqueue 1 slowstart 120s
> server web53 192.168.249.103:8480 weight 220 check inter 10s
>minconn 10 maxconn 200 maxqueue 1 slowstart 120s
> server web54 192.168.249.104:8480 weight 220 check inter 10s
>minconn 10 maxconn 200 maxqueue 1 slowstart 120s
> server web55 192.168.249.105:8480 weight 220 check inter 10s
>minconn 10 maxconn 200 maxqueue 1 slowstart 120s
> server web56 192.168.249.106:8480 weight 220 check inter 10s
>minconn 10 maxconn 200 maxqueue 1 slowstart 120s
> server web57 192.168.249.107:8480 weight 220 check inter 10s
>minconn 10 maxconn 200 maxqueue 1 slowstart 120s
> server web58 192.168.249.108:8480 weight 220 check inter 10s
>minconn 10 maxconn 200 maxqueue 1 slowstart 120s
> server stg-web2 192.168.249.178:8480 weight 150 check inter 10s
>minconn 10 maxconn 100 maxqueue 1 slowstart 120s
> # server stg-web1 192.168.249.171:8480 weight 80 check inter
>10s minconn 10 maxconn 500 maxqueue 1 slowstart 120s
>
>i see this issue on 4 haproxy nodes
>Can you please advise ?
>
>Haim Ari / SysOps Manager
>
>M: 972.584563032 / T: 972.722288367
>
>https://goo.gl/R83eLy
Please can you show us some logs when haproxy tell you that the backends
are down.

Regards
Aleks
Sorry, only registered users may post in this forum.

Click here to login