Welcome! Log In Create A New Profile

Advanced

BUG: changed server IPs do not get restored from saved state

Posted by Peter Fröhlich 
Peter Fröhlich
BUG: changed server IPs do not get restored from saved state
September 12, 2018 03:00PM
Hi gurus

we are using HAProxy 1.8.13 on Centos 7 with dynamically assigned
backends (backend server IPs and ports assigned over stats socket by
some python glue accessing namerd). This works fine.

We needed a method to make the dynamically assigned stuff stick over
restarts and try to use the "load-server-state-from-file" directive.

Sadly we are running into a bug:
HAProxy does not restore the configured IP address present in the
state file but falls back to the one in the config file at startup.
Other information like uptime counter, state, port get restored.
This only happens if the default server address in the config file is
an IP. This does not happen with DNS names. If we switch out the
default server addresses '127.0.0.1' in the config with 'localhost' it
works as expected. We found this behavior with all tested versions
(1.8.7, 1.8.13, 1.9-dev).

The test config and example output are available in a Stack Exchange
question I opened:
https://serverfault.com/questions/930422/haproxy-use-saved-state-to-restore-backend-ips-possible

You'll find two tiny patches attached, one for 1.8.13 and one for
1.9-dev. They work for our system and use case.

Thanks for your great work
Peter

peter@froehlich.cc
---

1.9-dev
--- src/server.c 2018-09-12 12:02:48.000000000 +0000
+++ server.c 2018-09-12 12:02:41.000000000 +0000
@@ -4143,7 +4143,7 @@
goto srv_init_addr_next;

for (srv = curproxy->srv; srv; srv = srv->next)
- if (srv->hostname)
+ if (srv->hostname || srv->lastaddr)
return_code |= srv_iterate_initaddr(srv);

srv_init_addr_next:


1.8.13
--- server.c 2018-09-12 12:25:47.000000000 +0000
+++ src/server.c 2018-09-12 12:26:05.000000000 +0000
@@ -3957,7 +3957,7 @@
goto srv_init_addr_next;

for (srv = curproxy->srv; srv; srv = srv->next)
- if (srv->hostname)
+ if (srv->hostname || srv->lastaddr)
return_code |= srv_iterate_initaddr(srv);

srv_init_addr_next:
Willy Tarreau
Re: BUG: changed server IPs do not get restored from saved state
September 13, 2018 08:00AM
Hi Peter,

On Wed, Sep 12, 2018 at 02:46:52PM +0200, Peter Fröhlich wrote:
> HAProxy does not restore the configured IP address present in the
> state file but falls back to the one in the config file at startup.
> Other information like uptime counter, state, port get restored.
> This only happens if the default server address in the config file is
> an IP. This does not happen with DNS names. If we switch out the
> default server addresses '127.0.0.1' in the config with 'localhost' it
> works as expected. We found this behavior with all tested versions
> (1.8.7, 1.8.13, 1.9-dev).

This is interesting. I'm pretty sure this derivates from the initial
choice of only restoring IP addresses learned over DNS.

> 1.9-dev
> --- src/server.c 2018-09-12 12:02:48.000000000 +0000
> +++ server.c 2018-09-12 12:02:41.000000000 +0000
> @@ -4143,7 +4143,7 @@
> goto srv_init_addr_next;
>
> for (srv = curproxy->srv; srv; srv = srv->next)
> - if (srv->hostname)
> + if (srv->hostname || srv->lastaddr)
> return_code |= srv_iterate_initaddr(srv);
>
> srv_init_addr_next:

I just have a doubt about this one. I suspect that if you change the
address in your config file, it will be ignored, which will be a much
worse issue.

Basically the problem we have with the state file is that some fields
do not report the original value, so it's difficult to know if the
state change is more recent than the config change. And typically the
choice above was made to work around this (possibly not the most
effective way). It's important to keep in mind that a config change
must always have percedence over a state change, otherwise you'll end
up having to remove your state file to make sure config changes are
taken into account, which would be by far the worst solution.

Ideally we should have the config address in the state file so that
upon reload we can detect if the config changed, and only pick the
new config value if it changed, otherwise use the state. I *think*
(but have not tested) that a workaround to this miss and to achieve
what you need could be to use a dummy name and set "init-addr last,<ip>"
for the initial address. This way the server name will be set, not
used for resolution, and the state file will always be applied. But
similarly, any IP address change made to the config file will be
ignored as long as the entry exists in the state file. But you'll
have the option to force that if really needed by removing "last"
on the first reload which needs to enforce a config change.

Probably that some doc updates on the subject would be needed :-/

Regards,
Willy
Peter Fröhlich
Re: BUG: changed server IPs do not get restored from saved state
September 13, 2018 09:40AM
On Thu, 13 Sep 2018, 07:52 Willy Tarreau, <[email protected]> wrote:
> On Wed, Sep 12, 2018 at 02:46:52PM +0200, Peter Fröhlich wrote:
> > HAProxy does not restore the configured IP address present in the
> > state file but falls back to the one in the config file at startup.
> > Other information like uptime counter, state, port get restored.
> > This only happens if the default server address in the config file is
> > an IP. This does not happen with DNS names. If we switch out the
> > default server addresses '127.0.0.1' in the config with 'localhost' it
> > works as expected. We found this behavior with all tested versions
> > (1.8.7, 1.8.13, 1.9-dev).
>
> This is interesting. I'm pretty sure this derivates from the initial
> choice of only restoring IP addresses learned over DNS.

This makes sense and I expected that much, reading the source.
But the documentation didn't reflect that so I took it as a bug.

> Ideally we should have the config address in the state file so that
> upon reload we can detect if the config changed, and only pick the
> new config value if it changed, otherwise use the state. I *think*
> (but have not tested) that a workaround to this miss and to achieve
> what you need could be to use a dummy name and set "init-addr last,<ip>"
> for the initial address. This way the server name will be set, not
> used for resolution, and the state file will always be applied. But
> similarly, any IP address change made to the config file will be
> ignored as long as the entry exists in the state file. But you'll
> have the option to force that if really needed by removing "last"
> on the first reload which needs to enforce a config change.

Thanks, I'll try that.
That said, I'm actually happy with just setting 'localhost' everywhere.
We are using this config in our service discovery layer and set all the
backend servers to an invalid default value - the real settings always
get set over the socket or over the state file in case of a restart.

> Probably that some doc updates on the subject would be needed :-/

I agree, as now its not clear that IPs get ignored in the state file if the
server does not have a DNS name set in the config file.

Would our use case of setting the backend servers only through the socket,
and the necessary workaround of setting the server entries to 'localhost'
be worth a 'Notes:' entry in the 'load-server-state-from-file' section?

Kind regards,
Peter
Willy Tarreau
Re: BUG: changed server IPs do not get restored from saved state
September 13, 2018 10:10AM
On Thu, Sep 13, 2018 at 09:29:42AM +0200, Peter Fröhlich wrote:
> Would our use case of setting the backend servers only through the socket,
> and the necessary workaround of setting the server entries to 'localhost'
> be worth a 'Notes:' entry in the 'load-server-state-from-file' section?

Yes, very likely. Feel free to propose a patch for the doc. Generally doc
written by users who were bitten by a problem is much clearer than the one
written by the initial developer!

Thanks,
Willy
Peter Fröhlich
Re: BUG: changed server IPs do not get restored from saved state
September 13, 2018 03:30PM
Am Do., 13. Sep. 2018 um 10:06 Uhr schrieb Willy Tarreau <[email protected]>:
> On Thu, Sep 13, 2018 at 09:29:42AM +0200, Peter Fröhlich wrote:
> > Would our use case of setting the backend servers only through the socket,
> > and the necessary workaround of setting the server entries to 'localhost'
> > be worth a 'Notes:' entry in the 'load-server-state-from-file' section?
>
> Yes, very likely. Feel free to propose a patch for the doc. Generally doc
> written by users who were bitten by a problem is much clearer than the one
> written by the initial developer!
>

Ok, I'll see what I can do. :-)

-Peter
Am Do., 13. Sep. 2018 um 15:23 Uhr schrieb Peter Fröhlich
<[email protected]>:
>
> Am Do., 13. Sep. 2018 um 10:06 Uhr schrieb Willy Tarreau <[email protected]>:
> > On Thu, Sep 13, 2018 at 09:29:42AM +0200, Peter Fröhlich wrote:
> > > Would our use case of setting the backend servers only through the
socket,
> > > and the necessary workaround of setting the server entries to
'localhost'
> > > be worth a 'Notes:' entry in the 'load-server-state-from-file'
section?
> >
> > Yes, very likely. Feel free to propose a patch for the doc. Generally
doc
> > written by users who were bitten by a problem is much clearer than the
one
> > written by the initial developer!
> >

For your consideration, the doc patch with context.
Please give me feedback if I should incorporate some other aspects.

Peter

--- configuration.txt 2018-09-14 17:35:35.000000000 +0200
+++ configuration.txt.edited 2018-09-14 17:35:29.000000000 +0200
@@ -4894,69 +4894,79 @@

load-server-state-from-file { global | local | none }
Allow seamless reload of HAProxy
May be used in sections: defaults | frontend | listen | backend
yes | no | yes | yes

This directive points HAProxy to a file where server state from previous
running process has been saved. That way, when starting up, before
handling
traffic, the new process can apply old states to servers exactly has if
no
reload occurred. The purpose of the "load-server-state-from-file"
directive is
to tell haproxy which file to use. For now, only 2 arguments to either
prevent
loading state or load states from a file containing all backends and
servers.
The state file can be generated by running the command "show servers
state"
over the stats socket and redirect output.

The format of the file is versioned and is very specific. To understand
it,
please read the documentation of the "show servers state" command
(chapter
9.3 of Management Guide).

Arguments:
global load the content of the file pointed by the global directive
named "server-state-file".

local load the content of the file pointed by the directive
"server-state-file-name" if set. If not set, then the
backend
name is used as a file name.

none don't load any stat for this backend

Notes:
- - server's IP address is preserved across reloads by default, but the
- order can be changed thanks to the server's "init-addr" setting. This
- means that an IP address change performed on the CLI at run time will
+ - a servers IP address will only be applied from the state file if the
+ servers config file line refers to a DNS address. If the server has a
+ static IP configured in the config file the IP from the state file
gets
+ ignored silently (the other settings get restored).
+
+ - the resolved IP address of a server is preserved across reloads by
default,
+ but the order can be changed thanks to the servers "init-addr"
setting.
+ This means that an IP address change performed on the CLI at run
time will
be preserved, and that any change to the local resolver (e.g.
/etc/hosts)
will possibly not have any effect if the state file is in use.

- - server's weight is applied from previous running process unless it
has
- has changed between previous and new configuration files.
+ - a continuous reconfiguration of server IPs + ports over the state
socket
+ could require to always restore the IP from the state file at reload..
+ To enable this use a safe default DNS name (like "localhost") in the
+ servers config line, that gets resolved over the local resolver.

+ - servers weight is applied from previous running process unless it has
+ has changed between previous and new configuration files.
+
Example: Minimal configuration

global
stats socket /tmp/socket
server-state-file /tmp/server_state

defaults
load-server-state-from-file global

backend bk
server s1 127.0.0.1:22 check weight 11
server s2 127.0.0.1:22 check weight 12


Then one can run :

socat /tmp/socket - <<< "show servers state" > /tmp/server_state

Content of the file /tmp/server_state would be like this:

1
# <field names skipped for the doc example>
1 bk 1 s1 127.0.0.1 2 0 11 11 4 6 3 4 6 0 0
1 bk 2 s2 127.0.0.1 2 0 12 12 4 6 3 4 6 0 0

Example: Minimal configuration

global
stats socket /tmp/socket
server-state-base /etc/haproxy/states
Hi Peter,

On Fri, Sep 14, 2018 at 05:43:14PM +0200, Peter Fröhlich wrote:
> For your consideration, the doc patch with context.
> Please give me feedback if I should incorporate some other aspects.

Thank you, however as you can see below, your mailer mangled the patch
by wrapping some lines and making it unusable. Could you please send it
as an attachment instead ? Oh by the way, if you're doing your patches
by hand, it's generally better to perform the diff from one directory
above the development tree, or by prepending "./" in front of the file
names so that the patch always applies fine using "patch -p1". Example
in your case :

$ diff -u ./doc/configuration.txt{.edited,} > file.diff

thanks!
Willy

> --- configuration.txt 2018-09-14 17:35:35.000000000 +0200
> +++ configuration.txt.edited 2018-09-14 17:35:29.000000000 +0200
> @@ -4894,69 +4894,79 @@
>
> load-server-state-from-file { global | local | none }
> Allow seamless reload of HAProxy
> May be used in sections: defaults | frontend | listen | backend
> yes | no | yes | yes
>
> This directive points HAProxy to a file where server state from previous
> running process has been saved. That way, when starting up, before
> handling
> traffic, the new process can apply old states to servers exactly has if
> no
> reload occurred. The purpose of the "load-server-state-from-file"
> directive is
> to tell haproxy which file to use. For now, only 2 arguments to either
> prevent
> loading state or load states from a file containing all backends and
> servers.
> The state file can be generated by running the command "show servers
> state"
> over the stats socket and redirect output.
(...)
Hi Willy

Am Fr., 14. Sep. 2018 um 19:15 Uhr schrieb Willy Tarreau <[email protected]>:
> On Fri, Sep 14, 2018 at 05:43:14PM +0200, Peter Fröhlich wrote:
> > For your consideration, the doc patch with context.
> > Please give me feedback if I should incorporate some other aspects.
>
> Thank you, however as you can see below, your mailer mangled the patch
> by wrapping some lines and making it unusable. Could you please send it
> as an attachment instead ?

Sorry, I'm not used to do this without pull requests. ^^;;;
Here is the diff.

Thanks
Peter
Attachments:
open | download - doc_load-server-state-from-file.diff (1.7 KB)
Sorry, only registered users may post in this forum.

Click here to login