Welcome! Log In Create A New Profile

Advanced

Nginx Rate limiting for HTTPS requests

Posted by rickGsp 
rickGsp
Nginx Rate limiting for HTTPS requests
May 15, 2018 06:30PM
Hi,

I have been experimenting with Nginx rate limiting and I need some inputs on
it’s working and what can be expected from this feature. I see some
difference in what I expected from this feature going by the documentation
and what I observed in my experiments.

Here is the detail on my testing:
I have a test server running Nginx and a backend server. Nginx is configured
as HTTPS server listening on 443. I have configured Nginx as reverse proxy
to my backend. We have a proprietary tool which feeds configured number of
HTTPS requests (one request/connection) to test server and generates reports
at the end of test. Report will have details how many requests return status
as 200 and 503.

Observation 1:
As per my observations, more requests are getting processed with return
status as 200 than expected if input request rate to Nginx is much higher
than the rate limit configured.
For example, with the following configuration in Nginx for rate limiting,
Here are my tests:
limit_req_zone $host zone=perhost:1m rate=100r/s;
limit_req zone=perhost burst=100 nodelay;

Test1: With input as 250 req/sec and rate limit configured at 100r/s, rate
limiting works as expected since on average ~100 requests return with 200
status every second

Test2: With input as 500 req/sec and rate limit configured at 100r/s, rate
limiting does not work as expected since on average ~150 requests return
with 200 status every second

Test3: With input as 600 req/sec and rate limit configured at 100r/s, rate
limiting does not work as expected since on average ~200 requests return
with 200 status every second

Test4: With input as 800 req/sec and rate limit configured at 100r/s, rate
limiting does not work as expected since on average ~350 requests return
with 200 status every second

Observation 2:
On the other side, If Nginx is configured as HTTP server listening on 80,
rate limiting feature seems to be working fine for the same tests.

I am not very sure what is happening here for HTTPS based testing. One
observation I have made is that in HTTP case, requests gets processed very
quickly whereas for HTTPS case, complete transaction takes relatively
longer. Also, for low input rate of HTTPS requests transaction completion is
not taking very long where as when input rate goes up, this delay further
increase and then rate limiting start behaving unexpectedly. Can this be the
cause of this difference in anyway? Please share your inputs on this.

Thanks in advance

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,279802,279802#msg-279802

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Maxim Dounin
Re: Nginx Rate limiting for HTTPS requests
May 15, 2018 08:00PM
Hello!

On Tue, May 15, 2018 at 12:20:31PM -0400, rickGsp wrote:

> I have been experimenting with Nginx rate limiting and I need some inputs on
> it’s working and what can be expected from this feature. I see some
> difference in what I expected from this feature going by the documentation
> and what I observed in my experiments.
>
> Here is the detail on my testing:
> I have a test server running Nginx and a backend server. Nginx is configured
> as HTTPS server listening on 443. I have configured Nginx as reverse proxy
> to my backend. We have a proprietary tool which feeds configured number of
> HTTPS requests (one request/connection) to test server and generates reports
> at the end of test. Report will have details how many requests return status
> as 200 and 503.
>
> Observation 1:
> As per my observations, more requests are getting processed with return
> status as 200 than expected if input request rate to Nginx is much higher
> than the rate limit configured.
> For example, with the following configuration in Nginx for rate limiting,
> Here are my tests:
> limit_req_zone $host zone=perhost:1m rate=100r/s;
> limit_req zone=perhost burst=100 nodelay;
>
> Test1: With input as 250 req/sec and rate limit configured at 100r/s, rate
> limiting works as expected since on average ~100 requests return with 200
> status every second
>
> Test2: With input as 500 req/sec and rate limit configured at 100r/s, rate
> limiting does not work as expected since on average ~150 requests return
> with 200 status every second

The question is: how did you get the ~150 r/s number?

As per your description, the tool you are using reports number of
requests returned corresponding status, but not rate. Make sure
that calculation is not based on initial numbers, but counts real
responses received and uses wall clock to calculate rate.

That is, if you tool is expected to generate 500 r/s load for 10
seconds (5000 requests in total) and you've got 1500 requests with
status 200, success rate is _not_ 150 r/s. To calculate success rate
properly we need to know how long requests processing took. E.g.,
if it took 15 seconds from the load start to the last request
finished, the real rate is 100 r/s.

[...]

> I am not very sure what is happening here for HTTPS based testing. One
> observation I have made is that in HTTP case, requests gets processed very
> quickly whereas for HTTPS case, complete transaction takes relatively
> longer. Also, for low input rate of HTTPS requests transaction completion is
> not taking very long where as when input rate goes up, this delay further
> increase and then rate limiting start behaving unexpectedly. Can this be the
> cause of this difference in anyway? Please share your inputs on this.

Sure, see above. As long as request processing takes significant
time, it becomes more important to measure time properly. Failing
to do so will result in wrong numbers.

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
rickGsp
Re: Nginx Rate limiting for HTTPS requests
May 16, 2018 11:10AM
Thanks for responding Maxim. I understood what you are pointing at. Yes I
have taken care of time measurement. Actually my test runs for 60 seconds
and in total I expect 6000 requests returning 200 status with rate limit
configured at 100r/s. However I see 9000 requests returning 200 status which
means 150 req/sec.

Shall I expect that even for HTTPS, rate limiting should work as perfectly
as plain HTTP case. If yes, I am just wondering if there is something I am
missing while configuring Nginx rate limiting for HTTPS.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,279802,279814#msg-279814

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Maxim Dounin
Re: Nginx Rate limiting for HTTPS requests
May 16, 2018 03:30PM
Hello!

On Wed, May 16, 2018 at 05:00:20AM -0400, rickGsp wrote:

> Thanks for responding Maxim. I understood what you are pointing at. Yes I
> have taken care of time measurement. Actually my test runs for 60 seconds
> and in total I expect 6000 requests returning 200 status with rate limit
> configured at 100r/s. However I see 9000 requests returning 200 status which
> means 150 req/sec.

As I tried to explain in my previous message, "test runs for 60
seconds" can have two different meanings: 1) the load is generated
for 60 seconds and 2) from first request started to the last
request finished it takes 60 seconds.

Make sure you are using the correct meaning. Also, it might
be a good idea to look into nginx access logs to verify both time
and numbers reported by your tool.

> Shall I expect that even for HTTPS, rate limiting should work as perfectly
> as plain HTTP case. If yes, I am just wondering if there is something I am
> missing while configuring Nginx rate limiting for HTTPS.

Yes, request rate limiting is expected to work identically for
both HTTP and HTTPS. The difference of HTTPS is that it, in
contrast to HTTP, requires a lot of resources for SSL handshakes,
and it is perfectly normal if your server cannot handle 500
handshakes per second at all. As such, total test time might be
significantly different from load generation time.

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
rickGsp
Re: Nginx Rate limiting for HTTPS requests
May 20, 2018 10:30AM
>>As I tried to explain in my previous message, "test runs for 60
>>seconds" can have two different meanings: 1) the load is generated
>>for 60 seconds and 2) from first request started to the last
>>request finished it takes 60 seconds.

>>Make sure you are using the correct meaning. Also, it might
>>be a good idea to look into nginx access logs to verify both time
>>and numbers reported by your tool.

Yes Maxim, I had understood your point. My test actually ran for 60 to 65
seconds which means it took 5 additional seconds to process the requests.
Even access logs says the same. Also, on more powerful machine, I get
expected result for the same test i.e 500 req/sec load but start seeing
difference at relatively higher load.It seems to me that a results also
depends on the resources available on the machine running Nginx.
Surprisingly, CPU was not hitting the peak on both the machines.I am using
CentOS systems for this testings.

Actually in another test with plain HTTP requests, I observed the same issue
of more requests than expected getting processed. However, for HTTP case,
this behaviour appeared at 700 req/sec input load instead of 500 req/sec as
in HTTPS. In this test requests got processed within 60 secs.

With all the test results, I am being forced to think that Nginx rate
limiting may not be able to stop DDoS attack with very high input load but
is decent enough to handle sudden spikes and load which is slightly higher
than configured rate limit, and computing power available also plays some
role here. Do you think I am right?

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,279802,279874#msg-279874

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Sorry, only registered users may post in this forum.

Click here to login