Welcome! Log In Create A New Profile

Advanced

What is a nice way to bypass the maintenance mode for certain IP's?

Posted by Pieter Vogelaar 
Hi,

At the moment if we set backends in maintenance mode, the servers can’t be reached by anyone.
Is it possible to still allow traffic from certain IP’s (of the office network) so that testing can be done, before the backend is available to the general public again?


Best regards,
Pieter Vogelaar
Hi,

On Mon, Feb 19, 2018 at 12:18:36PM +0000, Pieter Vogelaar wrote:
> Hi,
>
> At the moment if we set backends in maintenance mode, the servers can't be
> reached by anyone.
> Is it possible to still allow traffic from certain IP's (of the office
> network) so that testing can be done, before the backend is available to the
> general public again?

Please take a look at "force-persist", it's designed exactly for what
you want to do.

Regards,
Willy
Hi Willy,

Thanks I will look into that!

On the statistics report page it's possible to set all servers of a backend in maintence mode. Is it also possible to set the servers of all backends in maintenance mode?

Best regards,
Pieter Vogelaar


Op 19-02-18 14:04 heeft Willy Tarreau <[email protected]> geschreven:

Hi,

On Mon, Feb 19, 2018 at 12:18:36PM +0000, Pieter Vogelaar wrote:
> Hi,
>
> At the moment if we set backends in maintenance mode, the servers can't be
> reached by anyone.
> Is it possible to still allow traffic from certain IP's (of the office
> network) so that testing can be done, before the backend is available to the
> general public again?

Please take a look at "force-persist", it's designed exactly for what
you want to do.

Regards,
Willy
On Tue, Feb 20, 2018 at 08:11:25AM +0000, Pieter Vogelaar wrote:
> On the statistics report page it's possible to set all servers of a backend
> in maintence mode. Is it also possible to set the servers of all backends in
> maintenance mode?

I don't think so. At least the page doesn't offer the possibility, but if you
process it using a script, you should be able to send all IDs at once in a
single POST (provided it fits in a single buffer).

Willy
On 20/02/2018 09:11 πμ, Pieter Vogelaar wrote:
> Hi Willy,
>
> Thanks I will look into that!
>
> On the statistics report page it's possible to set all servers of a backend in maintence mode. Is it also possible to set the servers of all backends in maintenance mode?
>
> Best regards,
> Pieter Vogelaar
>

You may want to look at https://github.com/unixsurfer/haproxytool, as it does exactly what you want
% haproxytool server -d bck_all_srv1
Are you sure we want to disable 3 servers y/n?: y
bck_all_srv1 disabled in backend1_proc34 backend
bck_all_srv1 disabled in backend_proc1 backend
bck_all_srv1 disabled in backend2_proc34 backend

Cheers,
Pavlos
Am 2018-02-19 14:04, schrieb Willy Tarreau:
> Hi,
>
> On Mon, Feb 19, 2018 at 12:18:36PM +0000, Pieter Vogelaar wrote:
>> Hi,
>>
>> At the moment if we set backends in maintenance mode, the servers
>> can't be
>> reached by anyone.
>> Is it possible to still allow traffic from certain IP's (of the office
>> network) so that testing can be done, before the backend is available
>> to the
>> general public again?
>
> Please take a look at "force-persist", it's designed exactly for what
> you want to do.
>
> Regards,
> Willy



Hi,


can you point out what is wrong with this config?


https://stackoverflow.com/questions/29248144/working-configuration-for-haproxy-with-the-force-persist-setting


This pretty much how I would end up doing it and I'm curious to know if
there are any errors in my thinking.
(haproxy 1.7.9)




Regards
Rainer
On Tue, Feb 20, 2018 at 12:33:59PM +0100, rainer@ultra-secure.de wrote:
> can you point out what is wrong with this config?
>
> https://stackoverflow.com/questions/29248144/working-configuration-for-haproxy-with-the-force-persist-setting

Thanks for the link, I've responded there so that the response can be
found for future readers.

Willy
Am 2018-02-20 13:44, schrieb Willy Tarreau:
> On Tue, Feb 20, 2018 at 12:33:59PM +0100, rainer@ultra-secure.de wrote:
>> can you point out what is wrong with this config?
>>
>> https://stackoverflow.com/questions/29248144/working-configuration-for-haproxy-with-the-force-persist-setting
>
> Thanks for the link, I've responded there so that the response can be
> found for future readers.
>
> Willy



Thank you!



Best Regards
Rainer
Hi Willy,

I have the following (stripped down) configuration:

-----------

defaults
log global
maxconn 8000
option redispatch
option allbackups
retries 3
stats enable
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout check 10s

frontend default-http
bind :80
mode http
force-persist if TRUE
option httplog
use_backend %[base,regsub(^www\.,,i),map_beg(/etc/haproxy/http-bases-to-backends.map,default)]

frontend default-https
bind :443 ssl crt /etc/ssl/haproxy
mode http
force-persist if TRUE
option httplog
option http-server-close
option forwardfor
reqadd X-Forwarded-Proto:\ https
reqadd X-Forwarded-Port:\ 443
use_backend %[base,regsub(^www\.,,i),map_beg(/etc/haproxy/http-bases-to-backends.map,default)]

backend pieter-tomcat-tst
mode http
balance roundrobin
cookie JSESSIONID prefix nocache
redirect scheme https code 302 if !{ ssl_fc }
server pieter-tomcat-01t:8080 10.15.17.142:8080 check cookie s01
server pieter-tomcat-02t:8080 10.15.33.183:8080 check cookie s02

------------

In defaults I have "option redispatch" and in the frontends "force-persist if TRUE". But when I put both tomcat servers in maintenance mode I get a 503 served.
Why am I not getting access even though the servers are in maintenance mode?


Best regards,
Pieter Vogelaar

Op 19-02-18 14:04 heeft Willy Tarreau <[email protected]> geschreven:

Hi,

On Mon, Feb 19, 2018 at 12:18:36PM +0000, Pieter Vogelaar wrote:
> Hi,
>
> At the moment if we set backends in maintenance mode, the servers can't be
> reached by anyone.
> Is it possible to still allow traffic from certain IP's (of the office
> network) so that testing can be done, before the backend is available to the
> general public again?

Please take a look at "force-persist", it's designed exactly for what
you want to do.

Regards,
Willy
Hi Pieter,

On Thu, Mar 01, 2018 at 01:16:36PM +0000, Pieter Vogelaar wrote:
> Hi Willy,
>
> I have the following (stripped down) configuration:
>
> -----------
>
> defaults
> log global
> maxconn 8000
> option redispatch
> option allbackups
> retries 3
> stats enable
> timeout http-request 10s
> timeout queue 1m
> timeout connect 10s
> timeout client 1m
> timeout server 1m
> timeout check 10s
>
> frontend default-http
> bind :80
> mode http
> force-persist if TRUE
> option httplog
> use_backend %[base,regsub(^www\.,,i),map_beg(/etc/haproxy/http-bases-to-backends.map,default)]
>
> frontend default-https
> bind :443 ssl crt /etc/ssl/haproxy
> mode http
> force-persist if TRUE
> option httplog
> option http-server-close
> option forwardfor
> reqadd X-Forwarded-Proto:\ https
> reqadd X-Forwarded-Port:\ 443
> use_backend %[base,regsub(^www\.,,i),map_beg(/etc/haproxy/http-bases-to-backends.map,default)]
>
> backend pieter-tomcat-tst
> mode http
> balance roundrobin
> cookie JSESSIONID prefix nocache
> redirect scheme https code 302 if !{ ssl_fc }
> server pieter-tomcat-01t:8080 10.15.17.142:8080 check cookie s01
> server pieter-tomcat-02t:8080 10.15.33.183:8080 check cookie s02
>
> ------------
>
> In defaults I have "option redispatch" and in the frontends "force-persist if
> TRUE". But when I put both tomcat servers in maintenance mode I get a 503
> served.
> Why am I not getting access even though the servers are in maintenance mode?

For me it works here. The difference is that I'm using cookie insertion
since my server do not provide a cookie. Are you certain that you reached
the page where your servers delivered this cookie so that the persistence
can work ? I suspect that you don't have the JSESSIONID cookie yet in your
case.

Willy
Hi Willy,

We use Memcached Session Manager that stores the Tomcat sessions to a Couchbase cluster. It suffixes the session ID with "-n1" like:

JSESSIONID=s01~1C7985929CDF981D9ACC79EBD8A3293D-n1

Could this JSESSIONID format somehow have impact on HAProxy?


Best regards,
Pieter Vogelaar


Op 01-03-18 15:25 heeft Willy Tarreau <[email protected]> geschreven:

Hi Pieter,

On Thu, Mar 01, 2018 at 01:16:36PM +0000, Pieter Vogelaar wrote:
> Hi Willy,
>
> I have the following (stripped down) configuration:
>
> -----------
>
> defaults
> log global
> maxconn 8000
> option redispatch
> option allbackups
> retries 3
> stats enable
> timeout http-request 10s
> timeout queue 1m
> timeout connect 10s
> timeout client 1m
> timeout server 1m
> timeout check 10s
>
> frontend default-http
> bind :80
> mode http
> force-persist if TRUE
> option httplog
> use_backend %[base,regsub(^www\.,,i),map_beg(/etc/haproxy/http-bases-to-backends.map,default)]
>
> frontend default-https
> bind :443 ssl crt /etc/ssl/haproxy
> mode http
> force-persist if TRUE
> option httplog
> option http-server-close
> option forwardfor
> reqadd X-Forwarded-Proto:\ https
> reqadd X-Forwarded-Port:\ 443
> use_backend %[base,regsub(^www\.,,i),map_beg(/etc/haproxy/http-bases-to-backends.map,default)]
>
> backend pieter-tomcat-tst
> mode http
> balance roundrobin
> cookie JSESSIONID prefix nocache
> redirect scheme https code 302 if !{ ssl_fc }
> server pieter-tomcat-01t:8080 10.15.17.142:8080 check cookie s01
> server pieter-tomcat-02t:8080 10.15.33.183:8080 check cookie s02
>
> ------------
>
> In defaults I have "option redispatch" and in the frontends "force-persist if
> TRUE". But when I put both tomcat servers in maintenance mode I get a 503
> served.
> Why am I not getting access even though the servers are in maintenance mode?

For me it works here. The difference is that I'm using cookie insertion
since my server do not provide a cookie. Are you certain that you reached
the page where your servers delivered this cookie so that the persistence
can work ? I suspect that you don't have the JSESSIONID cookie yet in your
case.

Willy
On Thu, Mar 01, 2018 at 02:29:57PM +0000, Pieter Vogelaar wrote:
> Hi Willy,
>
> We use Memcached Session Manager that stores the Tomcat sessions to a Couchbase cluster. It suffixes the session ID with "-n1" like:
>
> JSESSIONID=s01~1C7985929CDF981D9ACC79EBD8A3293D-n1
>
> Could this JSESSIONID format somehow have impact on HAProxy?

No, that's fine, but my point is : are you certain it is present in your
browser's request when you get the 503 ? If the 3 char of the termination
flags in the logs indicates '-' or 'N', it means it's absent. Or if you
can run haproxy in debug mode (with -d), you will see all requests on the
screen and it will be easy to tell whether the cookie is there or not.

Willy
Hi Willy,

Yes I'm absolutely certain that the cookie is present in the browser request when I get the 503.
I changed the JSESSIONID line to "cookie SERVERID insert indirect nocache", but that didn't make a difference.

Log line when both servers in backend are in maintenance mode:

172.30.214.130:56887 [01/Mar/2018:15:37:24.928] default-https~ pieter-tomcat-tst/<NOSRV> 0/-1/-1/-1/0 503 3576 - - SCDN 1/0/0/0/0 0/0 {tst-pieter.example.org||nl-NL,nl;q=0.9,en-US;q=0.8,en;q=0.7||Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.186 Safari/537.36} {|||} "GET /hello-world/ HTTP/1.1"

Log line when both servers in backend are active:

172.30.214.130:56451 [01/Mar/2018:15:37:09.734] default-https~ pieter-tomcat-tst/pieter-tomcat-01t:8080 0/0/1/2/3 200 5243 - - --VN 2/1/0/0/0 0/0 {tst-pieter.example.org||nl-NL,nl;q=0.9,en-US;q=0.8,en;q=0.7||Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.186 Safari/537.36} {text/html;charset=ISO-8859-1|||} "GET /hello-world/ HTTP/1.1"


Best regards,
Pieter Vogelaar


Op 01-03-18 15:32 heeft Willy Tarreau <[email protected]> geschreven:

On Thu, Mar 01, 2018 at 02:29:57PM +0000, Pieter Vogelaar wrote:
> Hi Willy,
>
> We use Memcached Session Manager that stores the Tomcat sessions to a Couchbase cluster. It suffixes the session ID with "-n1" like:
>
> JSESSIONID=s01~1C7985929CDF981D9ACC79EBD8A3293D-n1
>
> Could this JSESSIONID format somehow have impact on HAProxy?

No, that's fine, but my point is : are you certain it is present in your
browser's request when you get the 503 ? If the 3 char of the termination
flags in the logs indicates '-' or 'N', it means it's absent. Or if you
can run haproxy in debug mode (with -d), you will see all requests on the
screen and it will be easy to tell whether the cookie is there or not.

Willy
Hi Pieter and Willy,

Le 01/03/2018 à 16:09, Pieter Vogelaar a écrit :
> Hi Willy,
>
> Yes I'm absolutely certain that the cookie is present in the browser request when I get the 503.
> I changed the JSESSIONID line to "cookie SERVERID insert indirect nocache", but that didn't make a difference.
>
> Log line when both servers in backend are in maintenance mode:
>
> 172.30.214.130:56887 [01/Mar/2018:15:37:24.928] default-https~ pieter-tomcat-tst/<NOSRV> 0/-1/-1/-1/0 503 3576 - - SCDN 1/0/0/0/0 0/0 {tst-pieter.example.org||nl-NL,nl;q=0.9,en-US;q=0.8,en;q=0.7||Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.186 Safari/537.36} {|||} "GET /hello-world/ HTTP/1.1"
>
> Log line when both servers in backend are active:
>
> 172.30.214.130:56451 [01/Mar/2018:15:37:09.734] default-https~ pieter-tomcat-tst/pieter-tomcat-01t:8080 0/0/1/2/3 200 5243 - - --VN 2/1/0/0/0 0/0 {tst-pieter.example.org||nl-NL,nl;q=0.9,en-US;q=0.8,en;q=0.7||Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.186 Safari/537.36} {text/html;charset=ISO-8859-1|||} "GET /hello-world/ HTTP/1.1"

Well, I think your issue will be resolved by moving "force-persist" on
the backend side instead of the frontend one.

The issue seems to exist from the first day of "force-persist", where
the code and the documentation don't agree about the sections where it
can be used.

From the documentation and the config parser, it can be used in a
frontend, but the code only loop on the s->be to evaluate the
persistence options.

See commit
http://git.haproxy.org/?p=haproxy.git;a=commit;h=4de9149f876cc0c63495b71a2c7a3aefc722c9c0
for details.

The same issue was also introduced with "ignore-persist".

>
>
> Best regards,
> Pieter Vogelaar
>
>
> Op 01-03-18 15:32 heeft Willy Tarreau <[email protected]> geschreven:
>
> On Thu, Mar 01, 2018 at 02:29:57PM +0000, Pieter Vogelaar wrote:
> > Hi Willy,
> >
> > We use Memcached Session Manager that stores the Tomcat sessions to a Couchbase cluster. It suffixes the session ID with "-n1" like:
> >
> > JSESSIONID=s01~1C7985929CDF981D9ACC79EBD8A3293D-n1
> >
> > Could this JSESSIONID format somehow have impact on HAProxy?
>
> No, that's fine, but my point is : are you certain it is present in your
> browser's request when you get the 503 ? If the 3 char of the termination
> flags in the logs indicates '-' or 'N', it means it's absent. Or if you
> can run haproxy in debug mode (with -d), you will see all requests on the
> screen and it will be easy to tell whether the cookie is there or not.
>
> Willy
>
>


--
Cyril Bonté
Hi Cyril,

On Thu, Mar 01, 2018 at 08:50:55PM +0100, Cyril Bonté wrote:
> Well, I think your issue will be resolved by moving "force-persist" on the
> backend side instead of the frontend one.
>
> The issue seems to exist from the first day of "force-persist", where the
> code and the documentation don't agree about the sections where it can be
> used.
>
> From the documentation and the config parser, it can be used in a frontend,
> but the code only loop on the s->be to evaluate the
> persistence options.

I think you are totally right. At first this caught my eye, but I looked
at the doc, and found that it was apparently supported in the frontend,
which surprised me. Then a quick look at the code told me that we were
setting SF_FORCE_PRST regardless of the frontend or backend. But after
your comment I looked closer and noticed that the place where it's done
is when processing the content switching rules, so by definition it can
only be done on the backend.

>
> See commit http://git.haproxy.org/?p=haproxy.git;a=commit;h=4de9149f876cc0c63495b71a2c7a3aefc722c9c0
> for details.
>
> The same issue was also introduced with "ignore-persist".

You're right, both are set in the same loop. Usually I prefer to adapt
the code to make it match the doc, but here I don't see a reasonable
way to do it, so I think we'll instead update the code to emit a warning
and update the doc. Any objection ?

Thanks,
Willy
When I move force-persist to the backend, it indeed works.

From some other post I understand it's only possible to bypass the maintenance mode where stickiness is used?


Best regards,
Pieter Vogelaar

Op 02-03-18 06:32 heeft Willy Tarreau <[email protected]> geschreven:

Hi Cyril,

On Thu, Mar 01, 2018 at 08:50:55PM +0100, Cyril Bonté wrote:
> Well, I think your issue will be resolved by moving "force-persist" on the
> backend side instead of the frontend one.
>
> The issue seems to exist from the first day of "force-persist", where the
> code and the documentation don't agree about the sections where it can be
> used.
>
> From the documentation and the config parser, it can be used in a frontend,
> but the code only loop on the s->be to evaluate the
> persistence options.

I think you are totally right. At first this caught my eye, but I looked
at the doc, and found that it was apparently supported in the frontend,
which surprised me. Then a quick look at the code told me that we were
setting SF_FORCE_PRST regardless of the frontend or backend. But after
your comment I looked closer and noticed that the place where it's done
is when processing the content switching rules, so by definition it can
only be done on the backend.

>
> See commit http://git.haproxy.org/?p=haproxy.git;a=commit;h=4de9149f876cc0c63495b71a2c7a3aefc722c9c0
> for details.
>
> The same issue was also introduced with "ignore-persist".

You're right, both are set in the same loop. Usually I prefer to adapt
the code to make it match the doc, but here I don't see a reasonable
way to do it, so I think we'll instead update the code to emit a warning
and update the doc. Any objection ?

Thanks,
Willy
On Fri, Mar 02, 2018 at 01:51:42PM +0000, Pieter Vogelaar wrote:
> When I move force-persist to the backend, it indeed works.

Great, thanks for the feedback.

> From some other post I understand it's only possible to bypass the
> maintenance mode where stickiness is used?

Yes, or you can use the "use-server" directive which respects force-persist
as well. What I've seen a number of people use to test deployments was a
static page at a hidden URL where you had a list of all configured servers
and a radio button (or a link), allowing you to decide what server to connect
to. Then clicking on the server would set the cookie so that you can directly
try to connect there and see if it works or not. If it doesn't, you just have
to go back to the server selection page and try another one. I remember having
implemented one such page with extra stuff like setting/clearing the cookie,
setting/clearing a "force-persist" cookie that was used to decide whether to
go there forcing the access or as a regular user. Thus using stickiness it's
very convenient. But if it doesn't suit your needs well, use-server will let
you do exactly what you want.

Willy
Does use-server also accept some keyword to address the first server in the backend instead of a specific valid server name of the backend?

That would save quite a bit logic complexity in Puppet.

Best regards,
Pieter Vogelaar


Op 02-03-18 15:41 heeft Willy Tarreau <[email protected]> geschreven:

On Fri, Mar 02, 2018 at 01:51:42PM +0000, Pieter Vogelaar wrote:
> When I move force-persist to the backend, it indeed works.

Great, thanks for the feedback.

> From some other post I understand it's only possible to bypass the
> maintenance mode where stickiness is used?

Yes, or you can use the "use-server" directive which respects force-persist
as well. What I've seen a number of people use to test deployments was a
static page at a hidden URL where you had a list of all configured servers
and a radio button (or a link), allowing you to decide what server to connect
to. Then clicking on the server would set the cookie so that you can directly
try to connect there and see if it works or not. If it doesn't, you just have
to go back to the server selection page and try another one. I remember having
implemented one such page with extra stuff like setting/clearing the cookie,
setting/clearing a "force-persist" cookie that was used to decide whether to
go there forcing the access or as a regular user. Thus using stickiness it's
very convenient. But if it doesn't suit your needs well, use-server will let
you do exactly what you want.

Willy
On Tue, Mar 06, 2018 at 09:48:11AM +0000, Pieter Vogelaar wrote:
> Does use-server also accept some keyword to address the first server in the
> backend instead of a specific valid server name of the backend?

Hmmm no there's no such feature. I'm not sure I'm seeing well the real
use case to be honnest, considering that you should have a farm with
many servers. Or maybe you'd only want to target a specific server
position, like server #1, server #2 etc to be sure to iterate over all
of them using enough rules ?

> That would save quite a bit logic complexity in Puppet.

It could be useful if you could shortly describe what the challenge is
in this case and how you would see it addressed. Maybe it's something
not too difficult to implement if it can be useful.

Cheers,
Willy
Hi Willy,

Le 02/03/2018 à 06:32, Willy Tarreau a écrit :
> You're right, both are set in the same loop. Usually I prefer to adapt
> the code to make it match the doc, but here I don't see a reasonable
> way to do it, so I think we'll instead update the code to emit a warning
> and update the doc. Any objection ?

With some delay, no objection at all. This is probably the safest option
as it used to work like this since the beginning. I'm working on the patch.

--
Cyril Bonté
Sorry, only registered users may post in this forum.

Click here to login