Welcome! Log In Create A New Profile

Advanced

Request rate limiting on the backend section

Posted by Krishna Kumar (Engineering) 
Krishna Kumar (Engineering)
Request rate limiting on the backend section
November 07, 2017 10:50AM
Hi all,

I am trying to implement request rate limiting to protect our servers from
too
many requests. While we were able to get this working correctly in the
frontend section, it is required to implement the same in the backend
section
due to the configuration we use in our data center for different services.

This is the current configuration that I tried to get 1000 RPS for the
entire
backend section:

backend HTTP-be
http-request track-sc2 fe_id
stick-table type integer size 1m expire 60s store
http_req_rate(100ms),gpc0,gpc0_rate(100ms)
acl mark_seen sc2_inc_gpc0 ge 0
acl outside_limit sc2_gpc0_rate() gt 100
http-request deny deny_status 429 if mark_seen outside_limit
server my-server <ip>:80

But running "wrk -t 1 -c 1 <args>" gives:
RPS: 19.60 (Total requests: 18669 Good: 100 Errors: 18569)
with following haproxy metrics:
0x2765d1c: key=3 use=1 exp=60000 gpc0=7270 gpc0_rate(100)=364
http_req_rate(100)=364

and "wrk -t 20 -c 1000 <args>" gives:
RPS: 6.62 (Total requests: 1100022 Good: 100 Errors: 1099922)
with following haproxy metrics:
0x2765d1c: key=3 use=94 exp=59999 gpc0=228218 gpc0_rate(100)=7229
http_req_rate(100)=7229

As seen above, only a total of 100 requests succeeded, not 100 * 10 * 20 =
20K
(for the 20 second duration). gpc0_rate does not *seem* to get reset to
zero at
configured interval (100ms), may be due to a mistake in the configuration
settings
above.

Another setting that we tried was:

backend HTTP-be
acl too_fast be_sess_rate gt 1000
tcp-request inspect-delay 1s
tcp-request content accept if ! too_fast
tcp-request content accept if WAIT_END
server my-server <ip>:80

While this behaved much better than earlier, I still get a lot of
discrepancy based
on the traffic volume:
wrk -t 40 -c 200 -d 15s <url>
RPS: 841.14 (Total requests: 12634 Good: 12634 Errors: 0 Time: 15.02)

wrk -t 40 -c 400 -d 15s <url>
RPS: 1063.78 (Total requests: 15978 Good: 15978 Errors: 0 Time: 15.02)

wrk -t 40 -c 800 -d 15s <url>
RPS: 1123.03 (Total requests: 16868 Good: 16868 Errors: 0 Time: 15.02)

wrk -t 40 -c 1600 -d 15s <url>
RPS: 1382.98 (Total requests: 20883 Good: 20883 Errors: 0 Time: 15.10)

The last one is off by 38%

Could someone help on what I am doing wrong, or how to achieve this? I
prefer using the first approach with stick table if possible, as it
provides finer
time granularity.

Thanks for any help.

Regards,
- Krishna
Lukas Tribus
Re: Request rate limiting on the backend section
November 07, 2017 02:30PM
Hello,



2017-11-07 10:46 GMT+01:00 Krishna Kumar (Engineering)
<[email protected]>:
> Hi all,
>
> I am trying to implement request rate limiting to protect our servers from
> too
> many requests. While we were able to get this working correctly in the
> frontend section, it is required to implement the same in the backend
> section
> due to the configuration we use in our data center for different services.

I'd suggest to use maxconn. This limits the amount of connections opened
to a single server, and is therefor equivalent to in-flight requests. That's is
a more appropriate limit than RPS because it doesn't matter if the responses
take a long time to compute or not.

You simply specify maxconn on each server:

backend bk_default
server s1 10.0.0.1:80 maxconn 5000
server s2 10.0.0.2:80 maxconn 3000
server s3 10.0.0.3:80 maxconn 2500


http://cbonte.github.io/haproxy-dconv/1.7/configuration.html#5.2-maxconn



Regards,
Lukas
Krishna Kumar (Engineering)
Re: Request rate limiting on the backend section
November 07, 2017 06:00PM
Hi Lukas,

On Tue, Nov 7, 2017 at 6:46 PM, Lukas Tribus <[email protected]> wrote:

> I'd suggest to use maxconn. This limits the amount of connections opened
> to a single server, and is therefor equivalent to in-flight requests.
That's is
> a more appropriate limit than RPS because it doesn't matter if the
responses
> take a long time to compute or not.

Thanks for your suggestion. Unfortunately per server maxconn may be
unsuitable
for our particular case due to 2 reasons:

1. We want to modify the limit dynamically at high frequency, e.g. every
second.
maxconn setting on server does not seem to be modifiable at present
without
a reload. We are using haproxy-1.6.3, is this feature present in a
newer release?

2. We have haproxy running on *many* servers, even for a single service
(many
configuration files also use nbproc). It is easier to give a RPS=1000
for the entire
service instead of breaking up per process and per server, which may
not be
possible at the rate we plan.

Is there any way that this can be done at the backend using stick-tables? I
was
wondering if there was an elementary mistake in the configuration options
being
used.

Regards,
- Krishna
Lukas Tribus
Re: Request rate limiting on the backend section
November 07, 2017 07:40PM
Hello,



2017-11-07 17:55 GMT+01:00 Krishna Kumar (Engineering)
<[email protected]>:
> Hi Lukas,
>
> On Tue, Nov 7, 2017 at 6:46 PM, Lukas Tribus <[email protected]> wrote:
>
>> I'd suggest to use maxconn. This limits the amount of connections opened
>> to a single server, and is therefor equivalent to in-flight requests.
>> That's is
>> a more appropriate limit than RPS because it doesn't matter if the
>> responses
>> take a long time to compute or not.
>
> Thanks for your suggestion. Unfortunately per server maxconn may be
> unsuitable
> for our particular case due to 2 reasons:
>
> 1. We want to modify the limit dynamically at high frequency, e.g. every second.
> maxconn setting on server does not seem to be modifiable at present without
> a reload. We are using haproxy-1.6.3, is this feature present in a newer release?

Yes, in 1.7 you can change server maxconn values in real time using
the admin socket:
https://cbonte.github.io/haproxy-dconv/1.7/management.html#9.3-set%20maxconn%20server



> 2. We have haproxy running on *many* servers, even for a single service (many
> configuration files also use nbproc). It is easier to give a RPS=1000 for the entire
> service instead of breaking up per process and per server, which may not be
> possible at the rate we plan.

You are reluctant to elaborate on the bigger picture, so I guess
generic advice is not what you are looking for. I just hope you are
not trying to build some kind of distributed rate-limiting
functionality with this.



> Is there any way that this can be done at the backend using stick-tables?
> We are using haproxy-1.6.3

I don't have enough experience with stick-tables to comment on this
generally, but I would suggest you upgrade to a current 1.7 release
first of all and retry your tests. There are currently 223 bugs fixed
in releases AFTER 1.6.3:
http://www.haproxy.org/bugs/bugs-1.6.3.html

Maybe someone more stick-table savvy can comment on your specific question.



Regards,
Lukas
Krishna Kumar (Engineering)
Re: Request rate limiting on the backend section
November 08, 2017 05:40AM
On Tue, Nov 7, 2017 at 11:57 PM, Lukas Tribus <[email protected]> wrote:

Hi Lukas,

> Yes, in 1.7 you can change server maxconn values in real time using
> the admin socket:
> https://cbonte.github.io/haproxy-dconv/1.7/management.
html#9.3-set%20maxconn%20server

Thanks, will take a look at if we can use this. The only issue is that we
want to
be able to change rps very often, and some backend sections contain upto
500 servers (and much more during sales), and doing that on the fly at high
frequency may not scale.

> You are reluctant to elaborate on the bigger picture, so I guess
> generic advice is not what you are looking for. I just hope you are
> not trying to build some kind of distributed rate-limiting
> functionality with this.

Sorry, not reluctance, I thought giving too much detail would put off people
from taking a look :) So you are right, we are trying to build a
distributed rate
limiting feature, and the control plane is mostly ready (thanks to HAProxy
developers for making such a performant/configurable system). The service
monitors current http_request_rate and current RPS setting via uds every
second, and updates these values to a central repository (zookeeper), and
on demand, tries to increase capacity by requesting capacity from other
servers so as to keep capacity constant at the configured value (e.g. 1000
RPS). Is this something you would not recommend?

> I don't have enough experience with stick-tables to comment on this
> generally, but I would suggest you upgrade to a current 1.7 release
> first of all and retry your tests. There are currently 223 bugs fixed
> in releases AFTER 1.6.3:
> http://www.haproxy.org/bugs/bugs-1.6.3.html

Thanks, we are considering moving to this version.

> Maybe someone more stick-table savvy can comment on your specific
question.

If anyone else has done something similar, would really like to hear from
you on
how to control RPS in the backend.

Regards,
- Krishna
Sorry, only registered users may post in this forum.

Click here to login