Welcome! Log In Create A New Profile

Advanced

Removed health check in combination with load-server-state-from-file (Bug)

Posted by Tim Düsterhus 
Hi

I run haproxy with 'load-server-state-from-file'. Before reloading
haproxy I dump the state using:

echo show servers state |nc -U admin.sock > /etc/haproxy/state/global

I noticed a buggy behaviour with this:

1. Check that the backend is 'DOWN'.
2. Dump the state using the command above (the 'DOWN' state is written
into the file).
3. Remove the health check of the backend.
4. Reload haproxy.
5. The backend will now be 'DOWN' forever, as the initial state taken
from the file is 'DOWN' and no health checks are running.

I attached an example configuration and an example state file. To
reproduce the issue:

1. Start haproxy.
2. Open the Stats page.
3. Place the state file.
4. Remove the 'check' from the configuration.
5. Reload haproxy.
6. Start the backend.
7. Reload the Stats page and notice that the backend still is 'DOWN'.

Tim
1
# be_id be_name srv_id srv_name srv_addr srv_op_state srv_admin_state srv_uweight srv_iweight srv_time_since_last_change srv_check_status srv_check_result srv_check_health srv_check_state srv_agent_state bk_f_forced_id srv_f_forced_id
3 bk_http 1 nginx 172.17.0.3 0 0 1 1 374 7 0 0 7 0 0 0

global
stats socket /admin.sock mode 666 level admin

server-state-file global
server-state-base /etc/haproxy/state/

defaults
log global
timeout connect 5s
timeout client 50s
timeout server 50s

load-server-state-from-file global

frontend fe_http
mode http
bind :::80 v4v6

default_backend bk_http

backend bk_http
mode http

option httpchk GET /

server nginx 172.17.0.3:80 check

listen stats
bind :1936

mode http
stats enable
stats uri /
stats hide-version
Hi

as I did not receive any reply at all to my email from Aug 13 I thought
I resend it (Quoted below). Can anyone at least verify that my bug
report is valid? :-)

Tim

Am 13.08.2017 um 13:19 schrieb Tim Düsterhus:
> Hi
>
> I run haproxy with 'load-server-state-from-file'. Before reloading
> haproxy I dump the state using:
>
> echo show servers state |nc -U admin.sock > /etc/haproxy/state/global
>
> I noticed a buggy behaviour with this:
>
> 1. Check that the backend is 'DOWN'.
> 2. Dump the state using the command above (the 'DOWN' state is written
> into the file).
> 3. Remove the health check of the backend.
> 4. Reload haproxy.
> 5. The backend will now be 'DOWN' forever, as the initial state taken
> from the file is 'DOWN' and no health checks are running.
>
> I attached an example configuration and an example state file. To
> reproduce the issue:
>
> 1. Start haproxy.
> 2. Open the Stats page.
> 3. Place the state file.
> 4. Remove the 'check' from the configuration.
> 5. Reload haproxy.
> 6. Start the backend.
> 7. Reload the Stats page and notice that the backend still is 'DOWN'.
>
> Tim
>
Hello,

I am experiencing the same problem.
Is this the expected behaviour ? Or is it a bug ?

Regards,
Julien

On Sat, Aug 26, 2017 at 2:55 AM, Tim Düsterhus <[email protected]> wrote:

> Hi
>
> as I did not receive any reply at all to my email from Aug 13 I thought
> I resend it (Quoted below). Can anyone at least verify that my bug
> report is valid? :-)
>
> Tim
>
> Am 13.08.2017 um 13:19 schrieb Tim Düsterhus:
> > Hi
> >
> > I run haproxy with 'load-server-state-from-file'. Before reloading
> > haproxy I dump the state using:
> >
> > echo show servers state |nc -U admin.sock > /etc/haproxy/state/global
> >
> > I noticed a buggy behaviour with this:
> >
> > 1. Check that the backend is 'DOWN'.
> > 2. Dump the state using the command above (the 'DOWN' state is written
> > into the file).
> > 3. Remove the health check of the backend.
> > 4. Reload haproxy.
> > 5. The backend will now be 'DOWN' forever, as the initial state taken
> > from the file is 'DOWN' and no health checks are running.
> >
> > I attached an example configuration and an example state file. To
> > reproduce the issue:
> >
> > 1. Start haproxy.
> > 2. Open the Stats page.
> > 3. Place the state file.
> > 4. Remove the 'check' from the configuration.
> > 5. Reload haproxy.
> > 6. Start the backend.
> > 7. Reload the Stats page and notice that the backend still is 'DOWN'.
> >
> > Tim
> >
>
>
Hi Julien and Tim,

Le 28/08/2017 à 10:32, Julien Laffaye a écrit :
> Hello,
>
> I am experiencing the same problem.
> Is this the expected behaviour ? Or is it a bug ?

Yes, it's expected.
One use case is to start all servers in a DOWN state then
programmatically set one or several of them UP once everything is
initialized in the architecture, using the CLI command "set server
<backend>/<server> health up".

>
> Regards,
> Julien
>
> On Sat, Aug 26, 2017 at 2:55 AM, Tim Düsterhus <[email protected]
> <mailto:[email protected]>> wrote:
>
> Hi
>
> as I did not receive any reply at all to my email from Aug 13 I thought
> I resend it (Quoted below). Can anyone at least verify that my bug
> report is valid? :-)
>
> Tim
>
> Am 13.08.2017 um 13:19 schrieb Tim Düsterhus:
> > Hi
> >
> > I run haproxy with 'load-server-state-from-file'. Before reloading
> > haproxy I dump the state using:
> >
> > echo show servers state |nc -U admin.sock > /etc/haproxy/state/global
> >
> > I noticed a buggy behaviour with this:
> >
> > 1. Check that the backend is 'DOWN'.
> > 2. Dump the state using the command above (the 'DOWN' state is
> written
> > into the file).
> > 3. Remove the health check of the backend.
> > 4. Reload haproxy.
> > 5. The backend will now be 'DOWN' forever, as the initial state taken
> > from the file is 'DOWN' and no health checks are running.
> >
> > I attached an example configuration and an example state file. To
> > reproduce the issue:
> >
> > 1. Start haproxy.
> > 2. Open the Stats page.
> > 3. Place the state file.
> > 4. Remove the 'check' from the configuration.
> > 5. Reload haproxy.
> > 6. Start the backend.
> > 7. Reload the Stats page and notice that the backend still is 'DOWN'.
> >
> > Tim
> >
>
>


--
Cyril Bonté
Hello Cyril,


This also can be achieved by using 'disabled' keyword on server, and update CLI to enable it. Are you sure that using server-state file to keep server DOWN from previous health check is the expected behaviour ?

May be i'm wrong, but when i have a server in DOWN state because of health check, if a disable health check on it, i expected the server to be UP.



________________________________
De : Cyril Bonté <[email protected]>
Envoyé : mardi 29 août 2017 00:12
À : Julien Laffaye; Tim Düsterhus
Cc : haproxy@formilux.org
Objet : Re: Removed health check in combination with load-server-state-from-file (Bug)

Hi Julien and Tim,

Le 28/08/2017 à 10:32, Julien Laffaye a écrit :
> Hello,
>
> I am experiencing the same problem.
> Is this the expected behaviour ? Or is it a bug ?

Yes, it's expected.
One use case is to start all servers in a DOWN state then
programmatically set one or several of them UP once everything is
initialized in the architecture, using the CLI command "set server
<backend>/<server> health up".

>
> Regards,
> Julien
>
> On Sat, Aug 26, 2017 at 2:55 AM, Tim Düsterhus <[email protected]
> <mailto:[email protected]>> wrote:
>
> Hi
>
> as I did not receive any reply at all to my email from Aug 13 I thought
> I resend it (Quoted below). Can anyone at least verify that my bug
> report is valid? :-)
>
> Tim
>
> Am 13.08.2017 um 13:19 schrieb Tim Düsterhus:
> > Hi
> >
> > I run haproxy with 'load-server-state-from-file'. Before reloading
> > haproxy I dump the state using:
> >
> > echo show servers state |nc -U admin.sock > /etc/haproxy/state/global
> >
> > I noticed a buggy behaviour with this:
> >
> > 1. Check that the backend is 'DOWN'.
> > 2. Dump the state using the command above (the 'DOWN' state is
> written
> > into the file).
> > 3. Remove the health check of the backend.
> > 4. Reload haproxy.
> > 5. The backend will now be 'DOWN' forever, as the initial state taken
> > from the file is 'DOWN' and no health checks are running.
> >
> > I attached an example configuration and an example state file. To
> > reproduce the issue:
> >
> > 1. Start haproxy.
> > 2. Open the Stats page.
> > 3. Place the state file.
> > 4. Remove the 'check' from the configuration.
> > 5. Reload haproxy.
> > 6. Start the backend.
> > 7. Reload the Stats page and notice that the backend still is 'DOWN'.
> >
> > Tim
> >
>
>


--
Cyril Bonté
> De: "Arnaud Jost" <[email protected]>
> À: haproxy@formilux.org
> Envoyé: Mercredi 30 Août 2017 10:01:37
> Objet: RE: Removed health check in combination with load-server-state-from-file (Bug)
>
>
>
>
> Hello Cyril,
>
>
> This also can be achieved by using 'disabled' keyword on server, and
> update CLI to enable it. Are you sure that using server-state file
> to keep server DOWN from previous health check is the expected
> behaviour ?

I am sure ;-)

> May be i'm wrong, but when i have a server in DOWN state because of
> health check, if a disable health check on it, i expected the server
> to be UP.

Well health checks or not, the state file reflects the state you want.
You can absolutely imagine cases where health checks were never activated, and someone used the CLI to set a server DOWN. If you save the servers state and reload haproxy, You'd certainly want to have that server still DOWN (which is a different state than DOWN for maintenance, the one set by the "disabled" keyword).

Please avoid top posting ;-)

>
>
>
>
>
>
>
>
> De : Cyril Bonté <[email protected]>
> Envoyé : mardi 29 août 2017 00:12
> À : Julien Laffaye; Tim Düsterhus
> Cc : haproxy@formilux.org
> Objet : Re: Removed health check in combination with
> load-server-state-from-file (Bug)
>
> Hi Julien and Tim,
>
> Le 28/08/2017 à 10:32, Julien Laffaye a écrit :
> > Hello,
> >
> > I am experiencing the same problem.
> > Is this the expected behaviour ? Or is it a bug ?
>
> Yes, it's expected.
> One use case is to start all servers in a DOWN state then
> programmatically set one or several of them UP once everything is
> initialized in the architecture, using the CLI command "set server
> <backend>/<server> health up".
>
> >
> > Regards,
> > Julien
> >
> > On Sat, Aug 26, 2017 at 2:55 AM, Tim Düsterhus <[email protected]
> > < mailto:[email protected] >> wrote:
> >
> > Hi
> >
> > as I did not receive any reply at all to my email from Aug 13 I
> > thought
> > I resend it (Quoted below). Can anyone at least verify that my bug
> > report is valid? :-)
> >
> > Tim
> >
> > Am 13.08.2017 um 13:19 schrieb Tim Düsterhus:
> > > Hi
> > >
> > > I run haproxy with 'load-server-state-from-file'. Before
> > > reloading
> > > haproxy I dump the state using:
> > >
> > > echo show servers state |nc -U admin.sock >
> > > /etc/haproxy/state/global
> > >
> > > I noticed a buggy behaviour with this:
> > >
> > > 1. Check that the backend is 'DOWN'.
> > > 2. Dump the state using the command above (the 'DOWN' state is
> > written
> > > into the file).
> > > 3. Remove the health check of the backend.
> > > 4. Reload haproxy.
> > > 5. The backend will now be 'DOWN' forever, as the initial state
> > > taken
> > > from the file is 'DOWN' and no health checks are running.
> > >
> > > I attached an example configuration and an example state file. To
> > > reproduce the issue:
> > >
> > > 1. Start haproxy.
> > > 2. Open the Stats page.
> > > 3. Place the state file.
> > > 4. Remove the 'check' from the configuration.
> > > 5. Reload haproxy.
> > > 6. Start the backend.
> > > 7. Reload the Stats page and notice that the backend still is
> > > 'DOWN'.
> > >
> > > Tim
> > >
> >
> >
>
>
> --
> Cyril Bonté
>
>
On Wed, Aug 30, 2017 at 10:38 AM, Cyril Bonté <[email protected]> wrote:

> Well health checks or not, the state file reflects the state you want.
> You can absolutely imagine cases where health checks were never activated,
> and someone used the CLI to set a server DOWN. If you save the servers
> state and reload haproxy, You'd certainly want to have that server still
> DOWN (which is a different state than DOWN for maintenance, the one set by
> the "disabled" keyword).


Hello Cyril,

Well, you can also think of this use case which I find legit:
- you don't have probe
- you want to have probe so you add one and reload haproxy
- you realize your probe does not do want you intended, your backends are
turning down
- you panic, you decide to rollback to the previous working configuration
and reload haproxy
- it still does not work
- you panic :)

How can we make this rollback-to-previous-configuration works in this case ?

Regards,
Julien
> De: "Julien Laffaye" <[email protected]>
> Hello Cyril,
>
>
> Well, you can also think of this use case which I find legit:
> - you don't have probe
> - you want to have probe so you add one and reload haproxy
> - you realize your probe does not do want you intended, your backends
> are turning down
> - you panic, you decide to rollback to the previous working
> configuration and reload haproxy
> - it still does not work
> - you panic :)
>
>
> How can we make this rollback-to-previous-configuration works in this
> case ?

Isn't the state file part of the configuration? :-)
On 17-08-30 11:12:15, Julien Laffaye wrote:
> On Wed, Aug 30, 2017 at 10:38 AM, Cyril Bonté <[email protected]> wrote:
>
> > Well health checks or not, the state file reflects the state you want.
> > You can absolutely imagine cases where health checks were never activated,
> > and someone used the CLI to set a server DOWN. If you save the servers
> > state and reload haproxy, You'd certainly want to have that server still
> > DOWN (which is a different state than DOWN for maintenance, the one set by
> > the "disabled" keyword).
>
> Well, you can also think of this use case which I find legit:
> - you don't have probe
> - you want to have probe so you add one and reload haproxy
> - you realize your probe does not do want you intended, your backends are
> turning down
> - you panic, you decide to rollback to the previous working configuration
> and reload haproxy
> - it still does not work
> - you panic :)
>
> How can we make this rollback-to-previous-configuration works in this
> case ?

Remove / rename the file load-server-state-from-file is referencing.

Cheers,
Georg
On Wed, Aug 30, 2017 at 11:18 AM, Cyril Bonté <[email protected]> wrote:

> Isn't the state file part of the configuration? :-)
>

yes it is. it is also part of the previous working configuration.
Sorry, only registered users may post in this forum.

Click here to login