Welcome! Log In Create A New Profile

Advanced

Nginx Rate limiting for HTTPS requests

Posted by rickGsp 
rickGsp
Nginx Rate limiting for HTTPS requests
May 15, 2018 06:30PM
Hi,

I have been experimenting with Nginx rate limiting and I need some inputs on
it’s working and what can be expected from this feature. I see some
difference in what I expected from this feature going by the documentation
and what I observed in my experiments.

Here is the detail on my testing:
I have a test server running Nginx and a backend server. Nginx is configured
as HTTPS server listening on 443. I have configured Nginx as reverse proxy
to my backend. We have a proprietary tool which feeds configured number of
HTTPS requests (one request/connection) to test server and generates reports
at the end of test. Report will have details how many requests return status
as 200 and 503.

Observation 1:
As per my observations, more requests are getting processed with return
status as 200 than expected if input request rate to Nginx is much higher
than the rate limit configured.
For example, with the following configuration in Nginx for rate limiting,
Here are my tests:
limit_req_zone $host zone=perhost:1m rate=100r/s;
limit_req zone=perhost burst=100 nodelay;

Test1: With input as 250 req/sec and rate limit configured at 100r/s, rate
limiting works as expected since on average ~100 requests return with 200
status every second

Test2: With input as 500 req/sec and rate limit configured at 100r/s, rate
limiting does not work as expected since on average ~150 requests return
with 200 status every second

Test3: With input as 600 req/sec and rate limit configured at 100r/s, rate
limiting does not work as expected since on average ~200 requests return
with 200 status every second

Test4: With input as 800 req/sec and rate limit configured at 100r/s, rate
limiting does not work as expected since on average ~350 requests return
with 200 status every second

Observation 2:
On the other side, If Nginx is configured as HTTP server listening on 80,
rate limiting feature seems to be working fine for the same tests.

I am not very sure what is happening here for HTTPS based testing. One
observation I have made is that in HTTP case, requests gets processed very
quickly whereas for HTTPS case, complete transaction takes relatively
longer. Also, for low input rate of HTTPS requests transaction completion is
not taking very long where as when input rate goes up, this delay further
increase and then rate limiting start behaving unexpectedly. Can this be the
cause of this difference in anyway? Please share your inputs on this.

Thanks in advance

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,279802,279802#msg-279802

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Maxim Dounin
Re: Nginx Rate limiting for HTTPS requests
May 15, 2018 08:00PM
Hello!

On Tue, May 15, 2018 at 12:20:31PM -0400, rickGsp wrote:

> I have been experimenting with Nginx rate limiting and I need some inputs on
> it’s working and what can be expected from this feature. I see some
> difference in what I expected from this feature going by the documentation
> and what I observed in my experiments.
>
> Here is the detail on my testing:
> I have a test server running Nginx and a backend server. Nginx is configured
> as HTTPS server listening on 443. I have configured Nginx as reverse proxy
> to my backend. We have a proprietary tool which feeds configured number of
> HTTPS requests (one request/connection) to test server and generates reports
> at the end of test. Report will have details how many requests return status
> as 200 and 503.
>
> Observation 1:
> As per my observations, more requests are getting processed with return
> status as 200 than expected if input request rate to Nginx is much higher
> than the rate limit configured.
> For example, with the following configuration in Nginx for rate limiting,
> Here are my tests:
> limit_req_zone $host zone=perhost:1m rate=100r/s;
> limit_req zone=perhost burst=100 nodelay;
>
> Test1: With input as 250 req/sec and rate limit configured at 100r/s, rate
> limiting works as expected since on average ~100 requests return with 200
> status every second
>
> Test2: With input as 500 req/sec and rate limit configured at 100r/s, rate
> limiting does not work as expected since on average ~150 requests return
> with 200 status every second

The question is: how did you get the ~150 r/s number?

As per your description, the tool you are using reports number of
requests returned corresponding status, but not rate. Make sure
that calculation is not based on initial numbers, but counts real
responses received and uses wall clock to calculate rate.

That is, if you tool is expected to generate 500 r/s load for 10
seconds (5000 requests in total) and you've got 1500 requests with
status 200, success rate is _not_ 150 r/s. To calculate success rate
properly we need to know how long requests processing took. E.g.,
if it took 15 seconds from the load start to the last request
finished, the real rate is 100 r/s.

[...]

> I am not very sure what is happening here for HTTPS based testing. One
> observation I have made is that in HTTP case, requests gets processed very
> quickly whereas for HTTPS case, complete transaction takes relatively
> longer. Also, for low input rate of HTTPS requests transaction completion is
> not taking very long where as when input rate goes up, this delay further
> increase and then rate limiting start behaving unexpectedly. Can this be the
> cause of this difference in anyway? Please share your inputs on this.

Sure, see above. As long as request processing takes significant
time, it becomes more important to measure time properly. Failing
to do so will result in wrong numbers.

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
rickGsp
Re: Nginx Rate limiting for HTTPS requests
May 16, 2018 11:10AM
Thanks for responding Maxim. I understood what you are pointing at. Yes I
have taken care of time measurement. Actually my test runs for 60 seconds
and in total I expect 6000 requests returning 200 status with rate limit
configured at 100r/s. However I see 9000 requests returning 200 status which
means 150 req/sec.

Shall I expect that even for HTTPS, rate limiting should work as perfectly
as plain HTTP case. If yes, I am just wondering if there is something I am
missing while configuring Nginx rate limiting for HTTPS.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,279802,279814#msg-279814

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Maxim Dounin
Re: Nginx Rate limiting for HTTPS requests
May 16, 2018 03:30PM
Hello!

On Wed, May 16, 2018 at 05:00:20AM -0400, rickGsp wrote:

> Thanks for responding Maxim. I understood what you are pointing at. Yes I
> have taken care of time measurement. Actually my test runs for 60 seconds
> and in total I expect 6000 requests returning 200 status with rate limit
> configured at 100r/s. However I see 9000 requests returning 200 status which
> means 150 req/sec.

As I tried to explain in my previous message, "test runs for 60
seconds" can have two different meanings: 1) the load is generated
for 60 seconds and 2) from first request started to the last
request finished it takes 60 seconds.

Make sure you are using the correct meaning. Also, it might
be a good idea to look into nginx access logs to verify both time
and numbers reported by your tool.

> Shall I expect that even for HTTPS, rate limiting should work as perfectly
> as plain HTTP case. If yes, I am just wondering if there is something I am
> missing while configuring Nginx rate limiting for HTTPS.

Yes, request rate limiting is expected to work identically for
both HTTP and HTTPS. The difference of HTTPS is that it, in
contrast to HTTP, requires a lot of resources for SSL handshakes,
and it is perfectly normal if your server cannot handle 500
handshakes per second at all. As such, total test time might be
significantly different from load generation time.

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
rickGsp
Re: Nginx Rate limiting for HTTPS requests
May 20, 2018 10:30AM
>>As I tried to explain in my previous message, "test runs for 60
>>seconds" can have two different meanings: 1) the load is generated
>>for 60 seconds and 2) from first request started to the last
>>request finished it takes 60 seconds.

>>Make sure you are using the correct meaning. Also, it might
>>be a good idea to look into nginx access logs to verify both time
>>and numbers reported by your tool.

Yes Maxim, I had understood your point. My test actually ran for 60 to 65
seconds which means it took 5 additional seconds to process the requests.
Even access logs says the same. Also, on more powerful machine, I get
expected result for the same test i.e 500 req/sec load but start seeing
difference at relatively higher load.It seems to me that a results also
depends on the resources available on the machine running Nginx.
Surprisingly, CPU was not hitting the peak on both the machines.I am using
CentOS systems for this testings.

Actually in another test with plain HTTP requests, I observed the same issue
of more requests than expected getting processed. However, for HTTP case,
this behaviour appeared at 700 req/sec input load instead of 500 req/sec as
in HTTPS. In this test requests got processed within 60 secs.

With all the test results, I am being forced to think that Nginx rate
limiting may not be able to stop DDoS attack with very high input load but
is decent enough to handle sudden spikes and load which is slightly higher
than configured rate limit, and computing power available also plays some
role here. Do you think I am right?

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,279802,279874#msg-279874

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Peter Booth
Re: Nginx Rate limiting for HTTPS requests
May 20, 2018 08:50PM
Rate limiting is a useful but crude tool that should only be one if four or five different things you do to protect your backend:

1 browser caching
2 cDN
3 rate limiting
4 nginx caching reverse proxy

What are your requests? Are they static content or proxied to a back end?
Do users login?
Is it valid for dynamic content built for one user to be returned to another?

Sent from my iPhone

On May 20, 2018, at 4:24 AM, rickGsp <[email protected]> wrote:

>>> As I tried to explain in my previous message, "test runs for 60
>>> seconds" can have two different meanings: 1) the load is generated
>>> for 60 seconds and 2) from first request started to the last
>>> request finished it takes 60 seconds.
>
>>> Make sure you are using the correct meaning. Also, it might
>>> be a good idea to look into nginx access logs to verify both time
>>> and numbers reported by your tool.
>
> Yes Maxim, I had understood your point. My test actually ran for 60 to 65
> seconds which means it took 5 additional seconds to process the requests.
> Even access logs says the same. Also, on more powerful machine, I get
> expected result for the same test i.e 500 req/sec load but start seeing
> difference at relatively higher load.It seems to me that a results also
> depends on the resources available on the machine running Nginx.
> Surprisingly, CPU was not hitting the peak on both the machines.I am using
> CentOS systems for this testings.
>
> Actually in another test with plain HTTP requests, I observed the same issue
> of more requests than expected getting processed. However, for HTTP case,
> this behaviour appeared at 700 req/sec input load instead of 500 req/sec as
> in HTTPS. In this test requests got processed within 60 secs.
>
> With all the test results, I am being forced to think that Nginx rate
> limiting may not be able to stop DDoS attack with very high input load but
> is decent enough to handle sudden spikes and load which is slightly higher
> than configured rate limit, and computing power available also plays some
> role here. Do you think I am right?
>
> Posted at Nginx Forum: https://forum.nginx.org/read.php?2,279802,279874#msg-279874
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Peter Booth
Re: Nginx Rate limiting for HTTPS requests
May 20, 2018 09:00PM
5. Do you use keepslive?

Sent from my iPhone

> On May 20, 2018, at 2:45 PM, Peter Booth <[email protected]> wrote:
>
> Rate limiting is a useful but crude tool that should only be one if four or five different things you do to protect your backend:
>
> 1 browser caching
> 2 cDN
> 3 rate limiting
> 4 nginx caching reverse proxy
>
> What are your requests? Are they static content or proxied to a back end?
> Do users login?
> Is it valid for dynamic content built for one user to be returned to another?
>
> Sent from my iPhone
>
> On May 20, 2018, at 4:24 AM, rickGsp <[email protected]> wrote:
>
>>>> As I tried to explain in my previous message, "test runs for 60
>>>> seconds" can have two different meanings: 1) the load is generated
>>>> for 60 seconds and 2) from first request started to the last
>>>> request finished it takes 60 seconds.
>>
>>>> Make sure you are using the correct meaning. Also, it might
>>>> be a good idea to look into nginx access logs to verify both time
>>>> and numbers reported by your tool.
>>
>> Yes Maxim, I had understood your point. My test actually ran for 60 to 65
>> seconds which means it took 5 additional seconds to process the requests.
>> Even access logs says the same. Also, on more powerful machine, I get
>> expected result for the same test i.e 500 req/sec load but start seeing
>> difference at relatively higher load.It seems to me that a results also
>> depends on the resources available on the machine running Nginx.
>> Surprisingly, CPU was not hitting the peak on both the machines.I am using
>> CentOS systems for this testings.
>>
>> Actually in another test with plain HTTP requests, I observed the same issue
>> of more requests than expected getting processed. However, for HTTP case,
>> this behaviour appeared at 700 req/sec input load instead of 500 req/sec as
>> in HTTPS. In this test requests got processed within 60 secs.
>>
>> With all the test results, I am being forced to think that Nginx rate
>> limiting may not be able to stop DDoS attack with very high input load but
>> is decent enough to handle sudden spikes and load which is slightly higher
>> than configured rate limit, and computing power available also plays some
>> role here. Do you think I am right?
>>
>> Posted at Nginx Forum: https://forum.nginx.org/read.php?2,279802,279874#msg-279874
>>
>> _______________________________________________
>> nginx mailing list
>> nginx@nginx.org
>> http://mailman.nginx.org/mailman/listinfo/nginx
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
rickGsp
Re: Nginx Rate limiting for HTTPS requests
May 21, 2018 07:10AM
> Rate limiting is a useful but crude tool that should only be one if four
or five different things you do to protect your backend:
>
> 1 browser caching
> 2 cDN
> 3 rate limiting
> 4 nginx caching reverse proxy
>
> What are your requests? Are they static content or proxied to a back end?
> Do users login?
> Is it valid for dynamic content built for one user to be returned to
another?

I am mainly using it to do reverse proxy to the backend.

>Do you use keepalive?

Here is the cleaned up version of the configuration in use:

# configuration file /etc/nginx/nginx.conf:
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;

events {
worker_connections 4096 ;
}

http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';

access_log /var/log/nginx/access.log main;
sendfile on;
client_header_buffer_size 64k;
#tcp_nopush on;
keepalive_timeout 65s;
#gzip on;
include /etc/nginx/conf.d/*.conf;

limit_req_zone $host zone=perhost:10m rate=100r/s;
limit_req zone=perhost burst=100 nodelay;

upstream service_lb {
server 127.0.0.1:8020;
server 127.0.0.1:8021;
}
}

worker_rlimit_nofile 10000;

# configuration file /etc/nginx/conf.d/nginx_ssl.conf:
server {
listen 192.168.0.50:443 ssl backlog=1024;
listen 127.0.0.1:443 ssl;

ssl_certificate /etc/nginx/conf.d/nginx.crt;
ssl_certificate_key /etc/nginx/conf.d/nginx.key;
ssl_protocols TLSv1.1 TLSv1.2;
ssl_ciphers
EECDH+AESGCM:EECDH+AES256:EECDH+AES128:EECDH+AES:kRSA+AESGCM:kRSA+AES:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-GCM-SHA256

:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:!aNULL:!ADH:!eNULL:!EXP:!LOW:!DES:!3DES:!RC4:!MD5:!SEED;
ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:1024000;
ssl_session_timeout 300;
ssl_verify_client off;

#charset koi8-r;
access_log /var/log/nginx/access.log main;

location /service/ {
proxy_pass http://service_lb;
break;
}
}

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,279802,279879#msg-279879

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Maxim Dounin
Re: Nginx Rate limiting for HTTPS requests
May 21, 2018 02:20PM
Hello!

On Sun, May 20, 2018 at 04:24:21AM -0400, rickGsp wrote:

> >>As I tried to explain in my previous message, "test runs for 60
> >>seconds" can have two different meanings: 1) the load is generated
> >>for 60 seconds and 2) from first request started to the last
> >>request finished it takes 60 seconds.
>
> >>Make sure you are using the correct meaning. Also, it might
> >>be a good idea to look into nginx access logs to verify both time
> >>and numbers reported by your tool.
>
> Yes Maxim, I had understood your point. My test actually ran for 60 to 65
> seconds which means it took 5 additional seconds to process the requests.
> Even access logs says the same. Also, on more powerful machine, I get
> expected result for the same test i.e 500 req/sec load but start seeing
> difference at relatively higher load.It seems to me that a results also
> depends on the resources available on the machine running Nginx.
> Surprisingly, CPU was not hitting the peak on both the machines.I am using
> CentOS systems for this testings.
>
> Actually in another test with plain HTTP requests, I observed the same issue
> of more requests than expected getting processed. However, for HTTP case,
> this behaviour appeared at 700 req/sec input load instead of 500 req/sec as
> in HTTPS. In this test requests got processed within 60 secs.
>
> With all the test results, I am being forced to think that Nginx rate
> limiting may not be able to stop DDoS attack with very high input load but
> is decent enough to handle sudden spikes and load which is slightly higher
> than configured rate limit, and computing power available also plays some
> role here. Do you think I am right?

I'm pretty sure the problem is with your tests, not with nginx
request rate limiting. Unfortunately, it is not possible to
reproduce your tests and check what's going wrong as you are using
proprietary software for tests.

As suggested previously, it might be a good idea to verify numbers
using nginx access logs. Seeing numbers of requests per seconds
should be as trivial as

grep ' 200 ' /path/to/log | awk '{print $4}' | uniq -c

assuming default log format and only test requests in the log.

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
rickGsp
Re: Nginx Rate limiting for HTTPS requests
May 21, 2018 04:00PM
>>I'm pretty sure the problem is with your tests, not with nginx
>>request rate limiting. Unfortunately, it is not possible to
>>reproduce your tests and check what's going wrong as you are using
>>proprietary software for tests.

>>As suggested previously, it might be a good idea to verify numbers
>>using nginx access logs. Seeing numbers of requests per seconds
>>should be as trivial as

>>grep ' 200 ' /path/to/log | awk '{print $4}' | uniq -c

>>assuming default log format and only test requests in the log.

Hi Maxim,

Here is a piece of output for the following command as per our success
return value as 202.
grep ' 202 ' /path/to/log | awk '{print $4}' | uniq -c

232 [17/May/2018:03:46:03
171 [17/May/2018:03:46:04
101 [17/May/2018:03:46:05
124 [17/May/2018:03:46:06
169 [17/May/2018:03:46:07
105 [17/May/2018:03:46:08
5 [17/May/2018:03:46:09
1 [17/May/2018:03:46:08
218 [17/May/2018:03:46:09
104 [17/May/2018:03:46:10
269 [17/May/2018:03:46:11
130 [17/May/2018:03:46:12
97 [17/May/2018:03:46:13
96 [17/May/2018:03:46:14
124 [17/May/2018:03:46:15
248 [17/May/2018:03:46:16
237 [17/May/2018:03:46:17
126 [17/May/2018:03:46:18

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,279802,279887#msg-279887

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Maxim Dounin
Re: Nginx Rate limiting for HTTPS requests
May 21, 2018 06:50PM
Hello!

On Mon, May 21, 2018 at 09:55:20AM -0400, rickGsp wrote:

> >>I'm pretty sure the problem is with your tests, not with nginx
> >>request rate limiting. Unfortunately, it is not possible to
> >>reproduce your tests and check what's going wrong as you are using
> >>proprietary software for tests.
>
> >>As suggested previously, it might be a good idea to verify numbers
> >>using nginx access logs. Seeing numbers of requests per seconds
> >>should be as trivial as
>
> >>grep ' 200 ' /path/to/log | awk '{print $4}' | uniq -c
>
> >>assuming default log format and only test requests in the log.
>
> Hi Maxim,
>
> Here is a piece of output for the following command as per our success
> return value as 202.
> grep ' 202 ' /path/to/log | awk '{print $4}' | uniq -c
>
> 232 [17/May/2018:03:46:03
> 171 [17/May/2018:03:46:04
> 101 [17/May/2018:03:46:05
> 124 [17/May/2018:03:46:06
> 169 [17/May/2018:03:46:07
> 105 [17/May/2018:03:46:08
> 5 [17/May/2018:03:46:09
> 1 [17/May/2018:03:46:08
> 218 [17/May/2018:03:46:09
> 104 [17/May/2018:03:46:10
> 269 [17/May/2018:03:46:11
> 130 [17/May/2018:03:46:12
> 97 [17/May/2018:03:46:13
> 96 [17/May/2018:03:46:14
> 124 [17/May/2018:03:46:15
> 248 [17/May/2018:03:46:16
> 237 [17/May/2018:03:46:17
> 126 [17/May/2018:03:46:18

This certainly does not look right. Either there are some
unrelated requests in the log, or requests are not limited as it
can be expected from your configuration. Some additional things
to check:

- Make sure the $host variable you use for the limiting is not
empty and not changed between requests created by your testing
tool. Try logging the variable to see if it changes or not.
Alternatively, replace it with a static string to see if it helps.

- Make sure there are no unrelated requests in the log. In
particular, you may want to use different logs in the server{}
block you are limiting and in the http{} block.

- Try another tool to see if you are able to reproduce the same
effect. Something simple like "ab" or "http_load" might be a
good choice.

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
rickGsp
Re: Nginx Rate limiting for HTTPS requests
May 22, 2018 06:40PM
>>- Make sure the $host variable you use for the limiting is not
>>empty and not changed between requests created by your testing
>>tool. Try logging the variable to see if it changes or not.
>>Alternatively, replace it with a static string to see if it helps.

Checked. $host variable is set for all the requests.

>>- Make sure there are no unrelated requests in the log. In
>>particular, you may want to use different logs in the server{}
>>block you are limiting and in the http{} block.

Checked. There are no unrelated requests in the log.

>>- Try another tool to see if you are able to reproduce the same
>>effect. Something simple like "ab" or "http_load" might be a
>>good choice.

Checked with "ab" with as following (concurrency 700 requests);
ab -n 20000 -c 700 https://9.0.0.10:443/test.html

Here is the piece of output. As per the report test ran for approx. 50
seconds and 20000-14622 = 5278 requests returned with success. This is as
expected as per rate limiting at 100r/s for 50 seconds test. Notice that
Mean requests processed per second is 396.

Concurrency Level: 700
Time taken for tests: 50.437 seconds
Complete requests: 20000
Failed requests: 14722
Requests per second: 396.53 [#/sec] (mean)

Access log report for this test as per the following command seems to be
fine: grep ' 200 ' /path/to/log | awk '{print $4}' | uniq -c
111 [22/May/2018:15:35:04
101 [22/May/2018:15:35:05
95 [22/May/2018:15:35:06
98 [22/May/2018:15:35:07
97 [22/May/2018:15:35:08
106 [22/May/2018:15:35:09
95 [22/May/2018:15:35:10
99 [22/May/2018:15:35:11
104 [22/May/2018:15:35:12
106 [22/May/2018:15:35:13


In another test, I ran two instances of "ab" in parallel with same
configuration and following is the output.This is again approx. 50 seconds
test. By combining both the reports (20000+20000) - (9344+10239) = 20417
requests returned with success. This is four times of expected 5000
requests/sec rate. I would like to understand this behaviour. I guess this
is happening in my tests as well. In my case I just keep pushing requests
without waiting for response.

First instance:
Concurrency Level: 700
Time taken for tests: 46.944 seconds
Complete requests: 20000
Failed requests: 9344
Requests per second: 426.04 [#/sec] (mean)

Second Instance:
Concurrency Level: 700
Time taken for tests: 53.344 seconds
Complete requests: 20000
Failed requests: 10239
Requests per second: 374.92 [#/sec] (mean)


Access log report for this test as per the following command does not seem
to be fine: grep ' 200 ' /path/to/log | awk '{print $4}' | uniq -c
180 [22/May/2018:15:52:59
276 [22/May/2018:15:53:00
33 [22/May/2018:15:53:01
20 [22/May/2018:15:53:00
70 [22/May/2018:15:53:01
1 [22/May/2018:15:53:00
181 [22/May/2018:15:53:01
16 [22/May/2018:15:53:02
2 [22/May/2018:15:53:01
99 [22/May/2018:15:53:02
1 [22/May/2018:15:53:01
177 [22/May/2018:15:53:02
329 [22/May/2018:15:53:03
8 [22/May/2018:15:53:02

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,279802,279908#msg-279908

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Maxim Dounin
Re: Nginx Rate limiting for HTTPS requests
May 22, 2018 08:10PM
Hello!

On Tue, May 22, 2018 at 12:37:18PM -0400, rickGsp wrote:

> >>- Make sure the $host variable you use for the limiting is not
> >>empty and not changed between requests created by your testing
> >>tool. Try logging the variable to see if it changes or not.
> >>Alternatively, replace it with a static string to see if it helps.
>
> Checked. $host variable is set for all the requests.
>
> >>- Make sure there are no unrelated requests in the log. In
> >>particular, you may want to use different logs in the server{}
> >>block you are limiting and in the http{} block.
>
> Checked. There are no unrelated requests in the log.
>
> >>- Try another tool to see if you are able to reproduce the same
> >>effect. Something simple like "ab" or "http_load" might be a
> >>good choice.
>
> Checked with "ab" with as following (concurrency 700 requests);
> ab -n 20000 -c 700 https://9.0.0.10:443/test.html
>
> Here is the piece of output. As per the report test ran for approx. 50
> seconds and 20000-14622 = 5278 requests returned with success. This is as
> expected as per rate limiting at 100r/s for 50 seconds test. Notice that
> Mean requests processed per second is 396.
>
> Concurrency Level: 700
> Time taken for tests: 50.437 seconds
> Complete requests: 20000
> Failed requests: 14722
> Requests per second: 396.53 [#/sec] (mean)
>
> Access log report for this test as per the following command seems to be
> fine: grep ' 200 ' /path/to/log | awk '{print $4}' | uniq -c
> 111 [22/May/2018:15:35:04
> 101 [22/May/2018:15:35:05
> 95 [22/May/2018:15:35:06
> 98 [22/May/2018:15:35:07
> 97 [22/May/2018:15:35:08
> 106 [22/May/2018:15:35:09
> 95 [22/May/2018:15:35:10
> 99 [22/May/2018:15:35:11
> 104 [22/May/2018:15:35:12
> 106 [22/May/2018:15:35:13
>
>
> In another test, I ran two instances of "ab" in parallel with same
> configuration and following is the output.This is again approx. 50 seconds
> test. By combining both the reports (20000+20000) - (9344+10239) = 20417
> requests returned with success. This is four times of expected 5000
> requests/sec rate. I would like to understand this behaviour. I guess this
> is happening in my tests as well. In my case I just keep pushing requests
> without waiting for response.
>
> First instance:
> Concurrency Level: 700
> Time taken for tests: 46.944 seconds
> Complete requests: 20000
> Failed requests: 9344
> Requests per second: 426.04 [#/sec] (mean)
>
> Second Instance:
> Concurrency Level: 700
> Time taken for tests: 53.344 seconds
> Complete requests: 20000
> Failed requests: 10239
> Requests per second: 374.92 [#/sec] (mean)
>
>
> Access log report for this test as per the following command does not seem
> to be fine: grep ' 200 ' /path/to/log | awk '{print $4}' | uniq -c
> 180 [22/May/2018:15:52:59
> 276 [22/May/2018:15:53:00
> 33 [22/May/2018:15:53:01
> 20 [22/May/2018:15:53:00
> 70 [22/May/2018:15:53:01
> 1 [22/May/2018:15:53:00
> 181 [22/May/2018:15:53:01
> 16 [22/May/2018:15:53:02
> 2 [22/May/2018:15:53:01
> 99 [22/May/2018:15:53:02
> 1 [22/May/2018:15:53:01
> 177 [22/May/2018:15:53:02
> 329 [22/May/2018:15:53:03
> 8 [22/May/2018:15:53:02

Please show "uname -a", "nginx -V", and "ps -alxww | grep nginx"
output.

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
rickGsp
Re: Nginx Rate limiting for HTTPS requests
May 23, 2018 07:00AM
>>Please show "uname -a", "nginx -V", and "ps -alxww | grep nginx"
>>output.

#uname -a
Linux localhost.localdomain 3.10.0-693.11.6.el7.x86_64 #1 SMP Thu Jan 4
01:06:37 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

#nginx -V
nginx version: nginx/1.14.0
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-16) (GCC)
built with OpenSSL 1.0.2k-fips 26 Jan 2017
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx
--modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf
--error-log-path=/var/log/nginx/error.log
--http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid
--lock-path=/var/run/nginx.lock
--http-client-body-temp-path=/var/cache/nginx/client_temp
--http-proxy-temp-path=/var/cache/nginx/proxy_temp
--http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp
--http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp
--http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx
--with-compat --with-file-aio --with-threads --with-http_addition_module
--with-http_auth_request_module --with-http_dav_module
--with-http_flv_module --with-http_gunzip_module
--with-http_gzip_static_module --with-http_mp4_module
--with-http_random_index_module --with-http_realip_module
--with-http_secure_link_module --with-http_slice_module
--with-http_ssl_module --with-http_stub_status_module --with-http_sub_module
--with-http_v2_module --with-mail --with-mail_ssl_module --with-stream
--with-stream_realip_module --with-stream_ssl_module
--with-stream_ssl_preread_module --with-cc-opt='-O2 -g -pipe -Wall
-Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong
--param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fPIC'
--with-ld-opt='-Wl,-z,relro -Wl,-z,now -pie'


#ps -alxww | grep nginx
5 0 9613 1 20 0 48516 1352 sigsus Ss ? 0:00 nginx:
master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
5 996 9614 9613 20 0 53204 7892 ep_pol S ? 1:35 nginx:
worker process
5 996 9615 9613 20 0 53800 8476 ep_pol S ? 1:37 nginx:
worker process
5 996 9616 9613 20 0 54888 9648 ep_pol S ? 1:56 nginx:
worker process
5 996 9617 9613 20 0 53008 7696 ep_pol S ? 2:22 nginx:
worker process
5 996 9618 9613 20 0 53452 8140 ep_pol S ? 2:12 nginx:
worker process
5 996 9619 9613 20 0 55036 9712 ep_pol S ? 2:14 nginx:
worker process
5 996 9620 9613 20 0 58700 13484 ep_pol S ? 2:18 nginx:
worker process
5 996 9621 9613 20 0 55532 10316 ep_pol S ? 2:20 nginx:
worker process
5 996 9622 9613 20 0 53504 8300 ep_pol S ? 2:18 nginx:
worker process
5 996 9623 9613 20 0 53204 7892 ep_pol S ? 2:12 nginx:
worker process
5 996 9624 9613 20 0 52196 6992 ep_pol S ? 2:32 nginx:
worker process
5 996 9625 9613 20 0 57164 11944 ep_pol S ? 2:24 nginx:
worker process
0 0 26753 26580 20 0 112648 964 pipe_w S+ pts/0 0:00 grep
--color=auto nginx

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,279802,279913#msg-279913

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Maxim Dounin
Re: Nginx Rate limiting for HTTPS requests
May 23, 2018 09:40PM
Hello!

On Wed, May 23, 2018 at 12:58:53AM -0400, rickGsp wrote:

> >>Please show "uname -a", "nginx -V", and "ps -alxww | grep nginx"
> >>output.
>
> #uname -a
> Linux localhost.localdomain 3.10.0-693.11.6.el7.x86_64 #1 SMP Thu Jan 4
> 01:06:37 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
>
> #nginx -V
> nginx version: nginx/1.14.0
> built by gcc 4.8.5 20150623 (Red Hat 4.8.5-16) (GCC)
> built with OpenSSL 1.0.2k-fips 26 Jan 2017

[...]

Thank you, I was able to reproduce this.

This indeed seems to be a problem with request rate limiting which
manifests itself when high request rates are used, there are
multiple worker processes, and request processing results in long
delays during event loop iterations (so that's why HTTPS requests
were special).

The problem is that time, as updated by each worker process at the
event loop iteration start, can be different in different workers
due to delays. Moreover, a worker may be in the future relative
to another worker. And this interfered nicely with nginx request
rate limiting logic which trying to be tolerant to time changes.

Quick and dirty patch below. It is expected to work fine on
systems with monotonic clocks available, but may need more work to
handle time changes on systems without monotonic clocks.

diff --git a/src/http/modules/ngx_http_limit_req_module.c b/src/http/modules/ngx_http_limit_req_module.c
--- a/src/http/modules/ngx_http_limit_req_module.c
+++ b/src/http/modules/ngx_http_limit_req_module.c
@@ -398,8 +398,9 @@ ngx_http_limit_req_lookup(ngx_http_limit
ngx_queue_insert_head(&ctx->sh->queue, &lr->queue);

ms = (ngx_msec_int_t) (now - lr->last);
+ ms = ngx_max(ms, 0);

- excess = lr->excess - ctx->rate * ngx_abs(ms) / 1000 + 1000;
+ excess = lr->excess - ctx->rate * ms / 1000 + 1000;

if (excess < 0) {
excess = 0;
@@ -413,7 +414,7 @@ ngx_http_limit_req_lookup(ngx_http_limit

if (account) {
lr->excess = excess;
- lr->last = now;
+ lr->last += ms;
return NGX_OK;
}

@@ -508,14 +509,15 @@ ngx_http_limit_req_account(ngx_http_limi

now = ngx_current_msec;
ms = (ngx_msec_int_t) (now - lr->last);
+ ms = ngx_max(ms, 0);

- excess = lr->excess - ctx->rate * ngx_abs(ms) / 1000 + 1000;
+ excess = lr->excess - ctx->rate * ms / 1000 + 1000;

if (excess < 0) {
excess = 0;
}

- lr->last = now;
+ lr->last += ms;
lr->excess = excess;
lr->count--;



--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Sorry, only registered users may post in this forum.

Click here to login