Welcome! Log In Create A New Profile

Advanced

Nginx Rate limiting for HTTPS requests

Posted by rickGsp 
rickGsp
Nginx Rate limiting for HTTPS requests
May 15, 2018 06:30PM
Hi,

I have been experimenting with Nginx rate limiting and I need some inputs on
it’s working and what can be expected from this feature. I see some
difference in what I expected from this feature going by the documentation
and what I observed in my experiments.

Here is the detail on my testing:
I have a test server running Nginx and a backend server. Nginx is configured
as HTTPS server listening on 443. I have configured Nginx as reverse proxy
to my backend. We have a proprietary tool which feeds configured number of
HTTPS requests (one request/connection) to test server and generates reports
at the end of test. Report will have details how many requests return status
as 200 and 503.

Observation 1:
As per my observations, more requests are getting processed with return
status as 200 than expected if input request rate to Nginx is much higher
than the rate limit configured.
For example, with the following configuration in Nginx for rate limiting,
Here are my tests:
limit_req_zone $host zone=perhost:1m rate=100r/s;
limit_req zone=perhost burst=100 nodelay;

Test1: With input as 250 req/sec and rate limit configured at 100r/s, rate
limiting works as expected since on average ~100 requests return with 200
status every second

Test2: With input as 500 req/sec and rate limit configured at 100r/s, rate
limiting does not work as expected since on average ~150 requests return
with 200 status every second

Test3: With input as 600 req/sec and rate limit configured at 100r/s, rate
limiting does not work as expected since on average ~200 requests return
with 200 status every second

Test4: With input as 800 req/sec and rate limit configured at 100r/s, rate
limiting does not work as expected since on average ~350 requests return
with 200 status every second

Observation 2:
On the other side, If Nginx is configured as HTTP server listening on 80,
rate limiting feature seems to be working fine for the same tests.

I am not very sure what is happening here for HTTPS based testing. One
observation I have made is that in HTTP case, requests gets processed very
quickly whereas for HTTPS case, complete transaction takes relatively
longer. Also, for low input rate of HTTPS requests transaction completion is
not taking very long where as when input rate goes up, this delay further
increase and then rate limiting start behaving unexpectedly. Can this be the
cause of this difference in anyway? Please share your inputs on this.

Thanks in advance

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,279802,279802#msg-279802

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Maxim Dounin
Re: Nginx Rate limiting for HTTPS requests
May 15, 2018 08:00PM
Hello!

On Tue, May 15, 2018 at 12:20:31PM -0400, rickGsp wrote:

> I have been experimenting with Nginx rate limiting and I need some inputs on
> it’s working and what can be expected from this feature. I see some
> difference in what I expected from this feature going by the documentation
> and what I observed in my experiments.
>
> Here is the detail on my testing:
> I have a test server running Nginx and a backend server. Nginx is configured
> as HTTPS server listening on 443. I have configured Nginx as reverse proxy
> to my backend. We have a proprietary tool which feeds configured number of
> HTTPS requests (one request/connection) to test server and generates reports
> at the end of test. Report will have details how many requests return status
> as 200 and 503.
>
> Observation 1:
> As per my observations, more requests are getting processed with return
> status as 200 than expected if input request rate to Nginx is much higher
> than the rate limit configured.
> For example, with the following configuration in Nginx for rate limiting,
> Here are my tests:
> limit_req_zone $host zone=perhost:1m rate=100r/s;
> limit_req zone=perhost burst=100 nodelay;
>
> Test1: With input as 250 req/sec and rate limit configured at 100r/s, rate
> limiting works as expected since on average ~100 requests return with 200
> status every second
>
> Test2: With input as 500 req/sec and rate limit configured at 100r/s, rate
> limiting does not work as expected since on average ~150 requests return
> with 200 status every second

The question is: how did you get the ~150 r/s number?

As per your description, the tool you are using reports number of
requests returned corresponding status, but not rate. Make sure
that calculation is not based on initial numbers, but counts real
responses received and uses wall clock to calculate rate.

That is, if you tool is expected to generate 500 r/s load for 10
seconds (5000 requests in total) and you've got 1500 requests with
status 200, success rate is _not_ 150 r/s. To calculate success rate
properly we need to know how long requests processing took. E.g.,
if it took 15 seconds from the load start to the last request
finished, the real rate is 100 r/s.

[...]

> I am not very sure what is happening here for HTTPS based testing. One
> observation I have made is that in HTTP case, requests gets processed very
> quickly whereas for HTTPS case, complete transaction takes relatively
> longer. Also, for low input rate of HTTPS requests transaction completion is
> not taking very long where as when input rate goes up, this delay further
> increase and then rate limiting start behaving unexpectedly. Can this be the
> cause of this difference in anyway? Please share your inputs on this.

Sure, see above. As long as request processing takes significant
time, it becomes more important to measure time properly. Failing
to do so will result in wrong numbers.

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
rickGsp
Re: Nginx Rate limiting for HTTPS requests
May 16, 2018 11:10AM
Thanks for responding Maxim. I understood what you are pointing at. Yes I
have taken care of time measurement. Actually my test runs for 60 seconds
and in total I expect 6000 requests returning 200 status with rate limit
configured at 100r/s. However I see 9000 requests returning 200 status which
means 150 req/sec.

Shall I expect that even for HTTPS, rate limiting should work as perfectly
as plain HTTP case. If yes, I am just wondering if there is something I am
missing while configuring Nginx rate limiting for HTTPS.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,279802,279814#msg-279814

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Maxim Dounin
Re: Nginx Rate limiting for HTTPS requests
May 16, 2018 03:30PM
Hello!

On Wed, May 16, 2018 at 05:00:20AM -0400, rickGsp wrote:

> Thanks for responding Maxim. I understood what you are pointing at. Yes I
> have taken care of time measurement. Actually my test runs for 60 seconds
> and in total I expect 6000 requests returning 200 status with rate limit
> configured at 100r/s. However I see 9000 requests returning 200 status which
> means 150 req/sec.

As I tried to explain in my previous message, "test runs for 60
seconds" can have two different meanings: 1) the load is generated
for 60 seconds and 2) from first request started to the last
request finished it takes 60 seconds.

Make sure you are using the correct meaning. Also, it might
be a good idea to look into nginx access logs to verify both time
and numbers reported by your tool.

> Shall I expect that even for HTTPS, rate limiting should work as perfectly
> as plain HTTP case. If yes, I am just wondering if there is something I am
> missing while configuring Nginx rate limiting for HTTPS.

Yes, request rate limiting is expected to work identically for
both HTTP and HTTPS. The difference of HTTPS is that it, in
contrast to HTTP, requires a lot of resources for SSL handshakes,
and it is perfectly normal if your server cannot handle 500
handshakes per second at all. As such, total test time might be
significantly different from load generation time.

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
rickGsp
Re: Nginx Rate limiting for HTTPS requests
May 20, 2018 10:30AM
>>As I tried to explain in my previous message, "test runs for 60
>>seconds" can have two different meanings: 1) the load is generated
>>for 60 seconds and 2) from first request started to the last
>>request finished it takes 60 seconds.

>>Make sure you are using the correct meaning. Also, it might
>>be a good idea to look into nginx access logs to verify both time
>>and numbers reported by your tool.

Yes Maxim, I had understood your point. My test actually ran for 60 to 65
seconds which means it took 5 additional seconds to process the requests.
Even access logs says the same. Also, on more powerful machine, I get
expected result for the same test i.e 500 req/sec load but start seeing
difference at relatively higher load.It seems to me that a results also
depends on the resources available on the machine running Nginx.
Surprisingly, CPU was not hitting the peak on both the machines.I am using
CentOS systems for this testings.

Actually in another test with plain HTTP requests, I observed the same issue
of more requests than expected getting processed. However, for HTTP case,
this behaviour appeared at 700 req/sec input load instead of 500 req/sec as
in HTTPS. In this test requests got processed within 60 secs.

With all the test results, I am being forced to think that Nginx rate
limiting may not be able to stop DDoS attack with very high input load but
is decent enough to handle sudden spikes and load which is slightly higher
than configured rate limit, and computing power available also plays some
role here. Do you think I am right?

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,279802,279874#msg-279874

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Peter Booth
Re: Nginx Rate limiting for HTTPS requests
May 20, 2018 08:50PM
Rate limiting is a useful but crude tool that should only be one if four or five different things you do to protect your backend:

1 browser caching
2 cDN
3 rate limiting
4 nginx caching reverse proxy

What are your requests? Are they static content or proxied to a back end?
Do users login?
Is it valid for dynamic content built for one user to be returned to another?

Sent from my iPhone

On May 20, 2018, at 4:24 AM, rickGsp <[email protected]> wrote:

>>> As I tried to explain in my previous message, "test runs for 60
>>> seconds" can have two different meanings: 1) the load is generated
>>> for 60 seconds and 2) from first request started to the last
>>> request finished it takes 60 seconds.
>
>>> Make sure you are using the correct meaning. Also, it might
>>> be a good idea to look into nginx access logs to verify both time
>>> and numbers reported by your tool.
>
> Yes Maxim, I had understood your point. My test actually ran for 60 to 65
> seconds which means it took 5 additional seconds to process the requests.
> Even access logs says the same. Also, on more powerful machine, I get
> expected result for the same test i.e 500 req/sec load but start seeing
> difference at relatively higher load.It seems to me that a results also
> depends on the resources available on the machine running Nginx.
> Surprisingly, CPU was not hitting the peak on both the machines.I am using
> CentOS systems for this testings.
>
> Actually in another test with plain HTTP requests, I observed the same issue
> of more requests than expected getting processed. However, for HTTP case,
> this behaviour appeared at 700 req/sec input load instead of 500 req/sec as
> in HTTPS. In this test requests got processed within 60 secs.
>
> With all the test results, I am being forced to think that Nginx rate
> limiting may not be able to stop DDoS attack with very high input load but
> is decent enough to handle sudden spikes and load which is slightly higher
> than configured rate limit, and computing power available also plays some
> role here. Do you think I am right?
>
> Posted at Nginx Forum: https://forum.nginx.org/read.php?2,279802,279874#msg-279874
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Peter Booth
Re: Nginx Rate limiting for HTTPS requests
May 20, 2018 09:00PM
5. Do you use keepslive?

Sent from my iPhone

> On May 20, 2018, at 2:45 PM, Peter Booth <[email protected]> wrote:
>
> Rate limiting is a useful but crude tool that should only be one if four or five different things you do to protect your backend:
>
> 1 browser caching
> 2 cDN
> 3 rate limiting
> 4 nginx caching reverse proxy
>
> What are your requests? Are they static content or proxied to a back end?
> Do users login?
> Is it valid for dynamic content built for one user to be returned to another?
>
> Sent from my iPhone
>
> On May 20, 2018, at 4:24 AM, rickGsp <[email protected]> wrote:
>
>>>> As I tried to explain in my previous message, "test runs for 60
>>>> seconds" can have two different meanings: 1) the load is generated
>>>> for 60 seconds and 2) from first request started to the last
>>>> request finished it takes 60 seconds.
>>
>>>> Make sure you are using the correct meaning. Also, it might
>>>> be a good idea to look into nginx access logs to verify both time
>>>> and numbers reported by your tool.
>>
>> Yes Maxim, I had understood your point. My test actually ran for 60 to 65
>> seconds which means it took 5 additional seconds to process the requests.
>> Even access logs says the same. Also, on more powerful machine, I get
>> expected result for the same test i.e 500 req/sec load but start seeing
>> difference at relatively higher load.It seems to me that a results also
>> depends on the resources available on the machine running Nginx.
>> Surprisingly, CPU was not hitting the peak on both the machines.I am using
>> CentOS systems for this testings.
>>
>> Actually in another test with plain HTTP requests, I observed the same issue
>> of more requests than expected getting processed. However, for HTTP case,
>> this behaviour appeared at 700 req/sec input load instead of 500 req/sec as
>> in HTTPS. In this test requests got processed within 60 secs.
>>
>> With all the test results, I am being forced to think that Nginx rate
>> limiting may not be able to stop DDoS attack with very high input load but
>> is decent enough to handle sudden spikes and load which is slightly higher
>> than configured rate limit, and computing power available also plays some
>> role here. Do you think I am right?
>>
>> Posted at Nginx Forum: https://forum.nginx.org/read.php?2,279802,279874#msg-279874
>>
>> _______________________________________________
>> nginx mailing list
>> nginx@nginx.org
>> http://mailman.nginx.org/mailman/listinfo/nginx
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
rickGsp
Re: Nginx Rate limiting for HTTPS requests
May 21, 2018 07:10AM
> Rate limiting is a useful but crude tool that should only be one if four
or five different things you do to protect your backend:
>
> 1 browser caching
> 2 cDN
> 3 rate limiting
> 4 nginx caching reverse proxy
>
> What are your requests? Are they static content or proxied to a back end?
> Do users login?
> Is it valid for dynamic content built for one user to be returned to
another?

I am mainly using it to do reverse proxy to the backend.

>Do you use keepalive?

Here is the cleaned up version of the configuration in use:

# configuration file /etc/nginx/nginx.conf:
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;

events {
worker_connections 4096 ;
}

http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';

access_log /var/log/nginx/access.log main;
sendfile on;
client_header_buffer_size 64k;
#tcp_nopush on;
keepalive_timeout 65s;
#gzip on;
include /etc/nginx/conf.d/*.conf;

limit_req_zone $host zone=perhost:10m rate=100r/s;
limit_req zone=perhost burst=100 nodelay;

upstream service_lb {
server 127.0.0.1:8020;
server 127.0.0.1:8021;
}
}

worker_rlimit_nofile 10000;

# configuration file /etc/nginx/conf.d/nginx_ssl.conf:
server {
listen 192.168.0.50:443 ssl backlog=1024;
listen 127.0.0.1:443 ssl;

ssl_certificate /etc/nginx/conf.d/nginx.crt;
ssl_certificate_key /etc/nginx/conf.d/nginx.key;
ssl_protocols TLSv1.1 TLSv1.2;
ssl_ciphers
EECDH+AESGCM:EECDH+AES256:EECDH+AES128:EECDH+AES:kRSA+AESGCM:kRSA+AES:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-GCM-SHA256

:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:!aNULL:!ADH:!eNULL:!EXP:!LOW:!DES:!3DES:!RC4:!MD5:!SEED;
ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:1024000;
ssl_session_timeout 300;
ssl_verify_client off;

#charset koi8-r;
access_log /var/log/nginx/access.log main;

location /service/ {
proxy_pass http://service_lb;
break;
}
}

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,279802,279879#msg-279879

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Maxim Dounin
Re: Nginx Rate limiting for HTTPS requests
May 21, 2018 02:20PM
Hello!

On Sun, May 20, 2018 at 04:24:21AM -0400, rickGsp wrote:

> >>As I tried to explain in my previous message, "test runs for 60
> >>seconds" can have two different meanings: 1) the load is generated
> >>for 60 seconds and 2) from first request started to the last
> >>request finished it takes 60 seconds.
>
> >>Make sure you are using the correct meaning. Also, it might
> >>be a good idea to look into nginx access logs to verify both time
> >>and numbers reported by your tool.
>
> Yes Maxim, I had understood your point. My test actually ran for 60 to 65
> seconds which means it took 5 additional seconds to process the requests.
> Even access logs says the same. Also, on more powerful machine, I get
> expected result for the same test i.e 500 req/sec load but start seeing
> difference at relatively higher load.It seems to me that a results also
> depends on the resources available on the machine running Nginx.
> Surprisingly, CPU was not hitting the peak on both the machines.I am using
> CentOS systems for this testings.
>
> Actually in another test with plain HTTP requests, I observed the same issue
> of more requests than expected getting processed. However, for HTTP case,
> this behaviour appeared at 700 req/sec input load instead of 500 req/sec as
> in HTTPS. In this test requests got processed within 60 secs.
>
> With all the test results, I am being forced to think that Nginx rate
> limiting may not be able to stop DDoS attack with very high input load but
> is decent enough to handle sudden spikes and load which is slightly higher
> than configured rate limit, and computing power available also plays some
> role here. Do you think I am right?

I'm pretty sure the problem is with your tests, not with nginx
request rate limiting. Unfortunately, it is not possible to
reproduce your tests and check what's going wrong as you are using
proprietary software for tests.

As suggested previously, it might be a good idea to verify numbers
using nginx access logs. Seeing numbers of requests per seconds
should be as trivial as

grep ' 200 ' /path/to/log | awk '{print $4}' | uniq -c

assuming default log format and only test requests in the log.

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Sorry, only registered users may post in this forum.

Click here to login