Welcome! Log In Create A New Profile

Advanced

HAProxy 1.7.5: conn_cur value problem with peer stick-table

Posted by Maxime Guillet 
Hello all,

I have problem to setup rate limiting with "conn_cur" information and
stick-table shared using peers. The counter always increase without
decrementing until key has expired.
Following my configuration and my tests:
I have 2 haproxy servers, a peers definition containing 2 servers and a
stick-table definition storing conn_cur and conn_rate(10s):

...
peers haproxysrv
peer haproxy1 10.0.0.2:9001
peer haproxy2 10.0.0.3:9001

frontend http
bind 0.0.0.0:80
stick-table type ip size 500k expire 10s store conn_cur,conn_rate(10s)
peers haproxysrv
tcp-request connection track-sc0 src
...


1/ Without the peer configuration and a test with ab binary, I have a
correct conn_cur value on each servers:

$ ab -n 2000 -c 20 http://10.0.0.2/
$ ab -n 2000 -c 20 http://10.0.0.3/

haproxy1# echo "show table http" | socat stdio
/var/run/haproxy/haproxy.stats
# table: http, type: ip, size:512000, used:1
0x7f59d568bb10: key=10.10.10.10 use=0 exp=6359 conn_rate(10000)=211
conn_cur=20
0x7f59d568bb10: key=10.10.10.10 use=20 exp=8116 conn_rate(10000)=190
conn_cur=20
0x7f59d568bb10: key=10.10.10.10 use=20 exp=5682 conn_rate(10000)=212
conn_cur=20

=> ab concurrency option set to 20 => conn_curr=20, the same when trying on
haproxy2

2/ If I launch the same test on both haproxy servers and peers
configuration activated, I can see the conn_cur counter always increasing

$ ab -n 2000 -c 20 http://10.0.0.2/
$ ab -n 2000 -c 20 http://10.0.0.3/

haproxy1# echo "show table http" | socat stdio
/var/run/haproxy/haproxy.stats
# table: http, type: ip, size:512000, used:1
0x7f59d568b9a0: key=10.10.10.10 use=20 exp=7285 conn_rate(10000)=225
conn_cur=47
0x7f59d568b9a0: key=10.10.10.10 use=20 exp=5337 conn_rate(10000)=213
conn_cur=52
0x7f59d568b9a0: key=10.10.10.10 use=20 exp=7952 conn_rate(10000)=178
conn_cur=133
0x7f59d568b9a0: key=10.10.10.10 use=20 exp=9589 conn_rate(10000)=218
conn_cur=259
0x7f59d568b9a0: key=10.10.10.10 use=20 exp=6144 conn_rate(10000)=143
conn_cur=321
0x7f59d568b9a0: key=10.10.10.10 use=20 exp=8115 conn_rate(10000)=190
conn_cur=553
0x7f59d568b9a0: key=10.10.10.10 use=20 exp=7815 conn_rate(10000)=180
conn_cur=676
0x7f59d568b9a0: key=10.10.10.10 use=20 exp=7285 conn_rate(10000)=162
conn_cur=738

=> the conn_cur information becomes more and more higher, while it should
be at maximum 40 (2 x 20 concurrent connection with ab)

I don't believe this behavior is intended, it seems more to be a bug.

Please help me if I did a mistake in my configuration, with my
underdstanding of the conn_cur use or please report this problem as a bug.

My haproxy version is 1.7.5 in Debian (1.7.5-1~bpo8+1)

Thanks for your help.

Maxime
Hi Maxime,

On Fri, May 19, 2017 at 02:28:40PM +0200, Maxime Guillet wrote:
> 2/ If I launch the same test on both haproxy servers and peers
> configuration activated, I can see the conn_cur counter always increasing
>
> $ ab -n 2000 -c 20 http://10.0.0.2/
> $ ab -n 2000 -c 20 http://10.0.0.3/
>
> haproxy1# echo "show table http" | socat stdio
> /var/run/haproxy/haproxy.stats
> # table: http, type: ip, size:512000, used:1
> 0x7f59d568b9a0: key=10.10.10.10 use=20 exp=7285 conn_rate(10000)=225
> conn_cur=47
> 0x7f59d568b9a0: key=10.10.10.10 use=20 exp=5337 conn_rate(10000)=213
> conn_cur=52
> 0x7f59d568b9a0: key=10.10.10.10 use=20 exp=7952 conn_rate(10000)=178
> conn_cur=133
> 0x7f59d568b9a0: key=10.10.10.10 use=20 exp=9589 conn_rate(10000)=218
> conn_cur=259
> 0x7f59d568b9a0: key=10.10.10.10 use=20 exp=6144 conn_rate(10000)=143
> conn_cur=321
> 0x7f59d568b9a0: key=10.10.10.10 use=20 exp=8115 conn_rate(10000)=190
> conn_cur=553
> 0x7f59d568b9a0: key=10.10.10.10 use=20 exp=7815 conn_rate(10000)=180
> conn_cur=676
> 0x7f59d568b9a0: key=10.10.10.10 use=20 exp=7285 conn_rate(10000)=162
> conn_cur=738
>
> => the conn_cur information becomes more and more higher, while it should
> be at maximum 40 (2 x 20 concurrent connection with ab)
>
> I don't believe this behavior is intended, it seems more to be a bug.

Well, yes and no. The principle of stick table synchronization is primarily
for stickiness and secondarily for active-backup table replication, so the
last which pushes a value should overwrite the previous one. In that sense,
it's expected that the values you find are around 20. What you're seeing
still makes me think it's a bug (since it sounds like we're adding up some
values here), but I don't know if it's a side effect of the update or not.
Specifically it's possible that this replication would prevent a session
from decrementing its reference on close for example. It could also be a
stupid copy-paste of the conn_cnt replication, which would then be wrong.
At first glance what I'm seeing in the code looks like a simple replication
so I can't explain how what you're seeing happens for now :-/

Willy
Le 27/05/2017 à 08:49, Willy Tarreau a écrit :
> Hi Maxime,
>
> On Fri, May 19, 2017 at 02:28:40PM +0200, Maxime Guillet wrote:
>> 2/ If I launch the same test on both haproxy servers and peers
>> configuration activated, I can see the conn_cur counter always increasing
>>
>> $ ab -n 2000 -c 20 http://10.0.0.2/
>> $ ab -n 2000 -c 20 http://10.0.0.3/
>>
>> haproxy1# echo "show table http" | socat stdio
>> /var/run/haproxy/haproxy.stats
>> # table: http, type: ip, size:512000, used:1
>> 0x7f59d568b9a0: key=10.10.10.10 use=20 exp=7285 conn_rate(10000)=225
>> conn_cur=47
>> 0x7f59d568b9a0: key=10.10.10.10 use=20 exp=5337 conn_rate(10000)=213
>> conn_cur=52
>> 0x7f59d568b9a0: key=10.10.10.10 use=20 exp=7952 conn_rate(10000)=178
>> conn_cur=133
>> 0x7f59d568b9a0: key=10.10.10.10 use=20 exp=9589 conn_rate(10000)=218
>> conn_cur=259
>> 0x7f59d568b9a0: key=10.10.10.10 use=20 exp=6144 conn_rate(10000)=143
>> conn_cur=321
>> 0x7f59d568b9a0: key=10.10.10.10 use=20 exp=8115 conn_rate(10000)=190
>> conn_cur=553
>> 0x7f59d568b9a0: key=10.10.10.10 use=20 exp=7815 conn_rate(10000)=180
>> conn_cur=676
>> 0x7f59d568b9a0: key=10.10.10.10 use=20 exp=7285 conn_rate(10000)=162
>> conn_cur=738
>>
>> => the conn_cur information becomes more and more higher, while it should
>> be at maximum 40 (2 x 20 concurrent connection with ab)
>>
>> I don't believe this behavior is intended, it seems more to be a bug.
> Well, yes and no. The principle of stick table synchronization is primarily
> for stickiness and secondarily for active-backup table replication, so the
> last which pushes a value should overwrite the previous one. In that sense,
> it's expected that the values you find are around 20. What you're seeing
> still makes me think it's a bug (since it sounds like we're adding up some
> values here), but I don't know if it's a side effect of the update or not.
> Specifically it's possible that this replication would prevent a session
> from decrementing its reference on close for example. It could also be a
> stupid copy-paste of the conn_cnt replication, which would then be wrong.
> At first glance what I'm seeing in the code looks like a simple replication
> so I can't explain how what you're seeing happens for now :-/
>
> Willy
>
Hi willy,

i found this topic because we have exactly the same problem , conn_cur
increasing endlessly when peers are activated. ( HA-Proxy version
1.7.9-1~bpo8+1  )
We use it for rate limiting purpose but from what i read it's a bad idea
after all, i thought that stick table were synchronised like this :

 Server A  [ conn_cur = 10 ], Server B [ conn_cur = 20 ], with peers =>
Server A [ conn_cur = 30 ], Server B [ conn_cur = 30 ]

But from what you write ( in fact it's in the documentation too ) it
seems that A overwrite B, or B overwrite A so it's impossible to use it
for rate limiting purpose.( i certainly don't want a [ conn_cur = 10 ]
overwriting a [ conn_cur = 20 ] )

Am i right ?

Thanks
Le 27/10/2017 à 18:06, Arnall a écrit :
> Le 27/05/2017 à 08:49, Willy Tarreau a écrit :
>> Hi Maxime,
>>
>> On Fri, May 19, 2017 at 02:28:40PM +0200, Maxime Guillet wrote:
>>> 2/ If I launch the same test on both haproxy servers and peers
>>> configuration activated, I can see the conn_cur counter always
>>> increasing
>>>
>>>    $ ab -n 2000 -c 20 http://10.0.0.2/
>>>    $ ab -n 2000 -c 20 http://10.0.0.3/
>>>
>>>    haproxy1# echo "show table http" | socat stdio
>>> /var/run/haproxy/haproxy.stats
>>>    # table: http, type: ip, size:512000, used:1
>>>    0x7f59d568b9a0: key=10.10.10.10 use=20 exp=7285 conn_rate(10000)=225
>>> conn_cur=47
>>>    0x7f59d568b9a0: key=10.10.10.10 use=20 exp=5337 conn_rate(10000)=213
>>> conn_cur=52
>>>    0x7f59d568b9a0: key=10.10.10.10 use=20 exp=7952 conn_rate(10000)=178
>>> conn_cur=133
>>>    0x7f59d568b9a0: key=10.10.10.10 use=20 exp=9589 conn_rate(10000)=218
>>> conn_cur=259
>>>    0x7f59d568b9a0: key=10.10.10.10 use=20 exp=6144 conn_rate(10000)=143
>>> conn_cur=321
>>>    0x7f59d568b9a0: key=10.10.10.10 use=20 exp=8115 conn_rate(10000)=190
>>> conn_cur=553
>>>    0x7f59d568b9a0: key=10.10.10.10 use=20 exp=7815 conn_rate(10000)=180
>>> conn_cur=676
>>>    0x7f59d568b9a0: key=10.10.10.10 use=20 exp=7285 conn_rate(10000)=162
>>> conn_cur=738
>>>
>>> => the conn_cur information becomes more and more higher, while it
>>> should
>>> be at maximum 40 (2 x 20 concurrent connection with ab)
>>>
>>> I don't believe this behavior is intended, it seems more to be a bug.
>> Well, yes and no. The principle of stick table synchronization is
>> primarily
>> for stickiness and secondarily for active-backup table replication,
>> so the
>> last which pushes a value should overwrite the previous one. In that
>> sense,
>> it's expected that the values you find are around 20. What you're seeing
>> still makes me think it's a bug (since it sounds like we're adding up
>> some
>> values here), but I don't know if it's a side effect of the update or
>> not.
>> Specifically it's possible that this replication would prevent a session
>> from decrementing its reference on close for example. It could also be a
>> stupid copy-paste of the conn_cnt replication, which would then be
>> wrong.
>> At first glance what I'm seeing in the code looks like a simple
>> replication
>> so I can't explain how what you're seeing happens for now :-/
>>
>> Willy
>>
> Hi willy,
>
> i found this topic because we have exactly the same problem , conn_cur
> increasing endlessly when peers are activated. ( HA-Proxy version
> 1.7.9-1~bpo8+1  )
> We use it for rate limiting purpose but from what i read it's a bad
> idea after all, i thought that stick table were synchronised like this :
>
>  Server A  [ conn_cur = 10 ], Server B [ conn_cur = 20 ], with peers
> => Server A [ conn_cur = 30 ], Server B [ conn_cur = 30 ]
>
> But from what you write ( in fact it's in the documentation too ) it
> seems that A overwrite B, or B overwrite A so it's impossible to use
> it for rate limiting purpose.( i certainly don't want a [ conn_cur =
> 10 ] overwriting a [ conn_cur = 20 ] )
>
> Am i right ?
>
> Thanks
>
Hi Willy,

as you ask i bump this message about endlessly increasing conn_cur
counter when used with peers.

At the time of the test it was with HA-Proxy version 1.7.9-1~bpo8+1, i
didn't test with newer version.

Regards.

Arnaud.


---
L'absence de virus dans ce courrier électronique a été vérifiée par le logiciel antivirus Avast.
https://www.avast.com/antivirus
Sorry, only registered users may post in this forum.

Click here to login