Welcome! Log In Create A New Profile

Advanced

Nginx runs out of memory with large value for 'keepalive_requests'

Posted by prajos 
Hi all,
I'm using nginx as a Revers proxy to a service (A). nginx receives a large
number of persistent connections from a single client service(B).
Service B sends a lot of requests (2K rps) over these persistent
connections.

The amount of memory nginx uses seems to increase as a function of
'keepalive_requests 2147483647' . The memory used keeps raising until the
machine runs out of memory (4GB, aws instance). While a smaller
''keepalive_requests 8192' doesn't create the exact problem.

Some additional observations:
When I reload nginx the memory usage comes down and then slowly starts
building up.
when I test nginx with a gatling test tool as a client, this behaviour is
not observed.
When I use the actual service(B), this behaviour seems to reappear.

I curious to know what exactly is happening and how can I fix this issue of
high memory usage ?

my nginx server side configuration looks like:

server {
listen 443 ssl default_server;
....
....

location / {
# keepalive_timeout 14400s;
# keepalive_requests 2147483647; ----> over 10 hrs, memory usages go to 4
GB

keepalive_timeout 600s;
keepalive_requests 8192;

proxy_pass http://ingress;
proxy_redirect off;
proxy_http_version 1.1;
proxy_set_header Connection "";
}
...
}


Thanks for all the help,

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,280505,280505#msg-280505

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Hello!

On Thu, Jul 12, 2018 at 05:48:55AM -0400, prajos wrote:

> Hi all,
> I'm using nginx as a Revers proxy to a service (A). nginx receives a large
> number of persistent connections from a single client service(B).
> Service B sends a lot of requests (2K rps) over these persistent
> connections.
>
> The amount of memory nginx uses seems to increase as a function of
> 'keepalive_requests 2147483647' . The memory used keeps raising until the
> machine runs out of memory (4GB, aws instance). While a smaller
> ''keepalive_requests 8192' doesn't create the exact problem.
>
> Some additional observations:
> When I reload nginx the memory usage comes down and then slowly starts
> building up.
> when I test nginx with a gatling test tool as a client, this behaviour is
> not observed.
> When I use the actual service(B), this behaviour seems to reappear.
>
> I curious to know what exactly is happening and how can I fix this issue of
> high memory usage ?

There may be allocations from a connection memory pool, and these
allocations are freed only on connection close. Trying to use
"keepalive_requests 2147483647" is expected to result in memory
usage growth as long as connections are never closed.

Tuning various settings might help to eliminate connection-related
allocations. In particular, if you've already tuned some settings
from their default values, switching back to defaults might be a
good starting point. Though in general it is a bad idea to never
close keepalive connections, the number of requests is limited for
a reason.

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Sorry, only registered users may post in this forum.

Click here to login