Welcome! Log In Create A New Profile

Advanced

nginx cache mounted on tmpf getting fulled

Posted by omkar_jadhav_20 
omkar_jadhav_20
nginx cache mounted on tmpf getting fulled
January 06, 2017 06:50AM
Hi,

I am using nginx as webserver with nginx version: nginx/1.10.2. For faster
access we have mounted cache of nginx of different application on RAM.But
even after giving enough buffer of size , now and then cache is getting
filled , below are few details of files for your reference :
maximum size given in nginx conf file is 500G , while mouting we have given
600G of space i.e. 100G of buffer.But still it is getting filled 100%.

fstab entries :
tmpfs /cache/123 tmpfs defaults,size=600G
0 0
tmpfs /cache/456 tmpfs defaults,size=60G
0 0
tmpfs /cache/789 tmpfs defaults,size=110G
0 0

cache getting filled , df output:

tmpfs tmpfs 60G 17G 44G 28%
/cache/456
tmpfs tmpfs 110G 323M 110G 1%
/cache/789
tmpfs tmpfs 600G 600G 0 100%
/cache/123

nginx conf details :

proxy_cache_path /cache/123 keys_zone=a123:200m levels=1:2 max_size=500g
inactive=3d;

server{
listen 80;
server_name dvr.catchup.com;
location ~.*.m3u8 {
access_log /var/log/nginx/access_123.log access;
proxy_cache off;
root /xyz/123;
if (!-e $request_filename) {
#origin url will be used if content is not available on DS
proxy_pass http://10.10.10.1X;
}
}
location / {
access_log /var/log/nginx/access_123.log access;
proxy_cache_valid 3d;
proxy_cache a123;
root /xyz/123;
if (!-e $request_filename) {
#origin url will be used if content is not available on server
proxy_pass http://10.10.10.1X;
}
proxy_cache_key $proxy_host$uri;
}
}

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,271842,271842#msg-271842

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
omkar_jadhav_20
Re: nginx cache mounted on tmpf getting filled 100%
January 06, 2017 06:50AM
omkar_jadhav_20 Wrote:
-------------------------------------------------------
> Hi,
>
> I am using nginx as webserver with nginx version: nginx/1.10.2. For
> faster access we have mounted cache of nginx of different application
> on RAM.But even after giving enough buffer of size , now and then
> cache is getting filled , below are few details of files for your
> reference :
> maximum size given in nginx conf file is 500G , while mouting we have
> given 600G of space i.e. 100G of buffer.But still it is getting filled
> 100%.
>
> fstab entries :
> tmpfs /cache/123 tmpfs
> defaults,size=600G 0 0
> tmpfs /cache/456 tmpfs
> defaults,size=60G 0 0
> tmpfs /cache/789 tmpfs
> defaults,size=110G 0 0
>
> cache getting filled , df output:
>
> tmpfs tmpfs 60G 17G 44G 28%
> /cache/456
> tmpfs tmpfs 110G 323M 110G 1%
> /cache/789
> tmpfs tmpfs 600G 600G 0 100%
> /cache/123
>
> nginx conf details :
>
> proxy_cache_path /cache/123 keys_zone=a123:200m levels=1:2
> max_size=500g inactive=3d;
>
> server{
> listen 80;
> server_name dvr.catchup.com;
> location ~.*.m3u8 {
> access_log /var/log/nginx/access_123.log access;
> proxy_cache off;
> root /xyz/123;
> if (!-e $request_filename) {
> #origin url will be used if content is not available on DS
> proxy_pass http://10.10.10.1X;
> }
> }
> location / {
> access_log /var/log/nginx/access_123.log access;
> proxy_cache_valid 3d;
> proxy_cache a123;
> root /xyz/123;
> if (!-e $request_filename) {
> #origin url will be used if content is not available on server
> proxy_pass http://10.10.10.1X;
> }
> proxy_cache_key $proxy_host$uri;
> }
> }

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,271842,271843#msg-271843

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
omkar_jadhav_20
Re: nginx cache mounted on tmpf getting filled 100%
January 09, 2017 10:10AM
can someone please respond. Let me know if any additional information is
required.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,271842,271870#msg-271870

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Matthew Slowe
Re: nginx cache mounted on tmpf getting fulled
January 09, 2017 10:20AM
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 06/01/2017 05:40, omkar_jadhav_20 wrote:
> Hi,
>
> I am using nginx as webserver with nginx version: nginx/1.10.2. For
> faster access we have mounted cache of nginx of different
> application on RAM.But even after giving enough buffer of size ,
> now and then cache is getting filled , below are few details of
> files for your reference : maximum size given in nginx conf file is
> 500G , while mouting we have given 600G of space i.e. 100G of
> buffer.But still it is getting filled 100%.

Do you actually have enough RAM / swap to facilitate these
requirements? It looks like you'd need about 800G of RAM/swap space to
make this work?

If you do then I don't know enough about how nginx works to advise,
sorry :-)

- --
Matthew Slowe | Server Infrastructure Officer
IT Infrastructure, Information Services, University of Kent
Room S21, Cornwallis South
Canterbury, Kent, CT2 7NZ, UK
Tel: +44 (0)1227 824265

www.kent.ac.uk/is | @UnikentUnseenIT | @UKCLibraryIt
PGP: https://keybase.io/fooflington
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0

iQIcBAEBCAAGBQJYc1NyAAoJEGc7JS6kpZi5IqcP/jjt31iEpOIc9iKa3kd/pE8i
uXa1QIkm2Pw6sU23SYu2tlMo2ExlhG8gCIYHs8nEh0uZvYX/uEaRbCSLXlkdzPBH
RZw3ngv48PhxFZqPWJXWZyDEElE8ybPtf5/ST5VA1Y/QMMwT8sFy3yTMbCD55MV9
RA3SIDFByMAzeMgHaQBErJzFW5UU+NenlGKZjQRZ6upxy1D0YGk1fV5HsI2h4MRx
L9i1afpbbmPE1p6v5q8RYEXxcQsTieRwsV0BSCBlfzcy/VoAC55p9sGy1wFlk4bD
b5XYUi7kX3/absk2vxHLGPJLOSMwD77BAWt3SzYb8RlPwEFvQc4JNED4uNykR51I
mUdAE1F+rxzHMZStoggkeSLKqoRleyJ5ZKRrP6gq5jVEl0m3cWo7pljM5By/eLNR
lyPZkbfIHWQvyRN8pnLVzvYMcrXXjONMOdJCngSppb6Ae1cSC0esGvLXxRIYzySF
sgNfB04H3R66hq3uLGz7rYrPKaAgEKg3XaASso4r3uYYWUvfYmAO5bE8pkZ+M+QW
Lx5vbyNZgPFVNSihZBnryh2WUs1c4LjHYEOoUYSTKirxV6daQiOoPrYfBXXUMwk4
FaBkimO3gu7xMmoe2lOc3KGZuEz2JV1Sqo/orxQ4Pmk5lkgVfSrPDn3h/oL8/ZOW
Zdlw+tZNDm0ULOGlS2tE
=YUuq
-----END PGP SIGNATURE-----
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
omkar_jadhav_20
Re: nginx cache mounted on tmpf getting fulled
January 09, 2017 10:20AM
Yes I do have RAM of size : 1.5T and swap space of around 200G. It has been
observed that swap is not getting used in this case. But it seems either OS
is not clearing the RAM that fast or nginx is not able to control RAM as
mentioned in the nginx configuration file.
Could you please suggest what can be the possible solution for this issue.
Every time max_size is getting crossed by RAM even after mentioning it in
nginx config.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,271842,271873#msg-271873

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
omkar_jadhav_20
Re: nginx cache mounted on tmpf getting fulled
January 16, 2017 08:10AM
can someone please respond and suggest best way to manage cache in highly
utilized cache dependent environment. As number of nginx requests increases
and cache hit ratio becomes significant , the tmpfs crosses the max_size
limit mentioned in the nginx.conf file and use full cache mounted . This in
returns increases the server load by great extent.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,271842,271995#msg-271995

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Igor A. Ippolitov
Re: nginx cache mounted on tmpf getting fulled
January 16, 2017 10:50AM
Hello,

Your cache have 200m space for keys. This is around 1.6M items, isn't it?
How much files do you have in your cache? May we have a look at
`df -i ` and `du -s /cache/123` output, please?

On 06.01.2017 08:40, omkar_jadhav_20 wrote:
> Hi,
>
> I am using nginx as webserver with nginx version: nginx/1.10.2. For faster
> access we have mounted cache of nginx of different application on RAM.But
> even after giving enough buffer of size , now and then cache is getting
> filled , below are few details of files for your reference :
> maximum size given in nginx conf file is 500G , while mouting we have given
> 600G of space i.e. 100G of buffer.But still it is getting filled 100%.
>
> fstab entries :
> tmpfs /cache/123 tmpfs defaults,size=600G
> 0 0
> tmpfs /cache/456 tmpfs defaults,size=60G
> 0 0
> tmpfs /cache/789 tmpfs defaults,size=110G
> 0 0
>
> cache getting filled , df output:
>
> tmpfs tmpfs 60G 17G 44G 28%
> /cache/456
> tmpfs tmpfs 110G 323M 110G 1%
> /cache/789
> tmpfs tmpfs 600G 600G 0 100%
> /cache/123
>
> nginx conf details :
>
> proxy_cache_path /cache/123 keys_zone=a123:200m levels=1:2 max_size=500g
> inactive=3d;
>
> server{
> listen 80;
> server_name dvr.catchup.com;
> location ~.*.m3u8 {
> access_log /var/log/nginx/access_123.log access;
> proxy_cache off;
> root /xyz/123;
> if (!-e $request_filename) {
> #origin url will be used if content is not available on DS
> proxy_pass http://10.10.10.1X;
> }
> }
> location / {
> access_log /var/log/nginx/access_123.log access;
> proxy_cache_valid 3d;
> proxy_cache a123;
> root /xyz/123;
> if (!-e $request_filename) {
> #origin url will be used if content is not available on server
> proxy_pass http://10.10.10.1X;
> }
> proxy_cache_key $proxy_host$uri;
> }
> }
>
> Posted at Nginx Forum: https://forum.nginx.org/read.php?2,271842,271842#msg-271842
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx


_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Peter Booth
Re: nginx cache mounted on tmpf getting fulled
January 17, 2017 02:40AM
I'm curious, why are you using tmpfs for your cache store? With fast local storage bring so cheap, why don't you devote a few TB to your cache?

When I look at the techempower benchmarks I see that openresty (an nginx build that comes with lots of lua value add) can serve 440,000 JSON responses per sec with 3ms latency. That's on five year old E7-4850 Westmere hardware at 2.0GHz, with 10G NICs. The min latency to get a packet from nginx through the kernel stack and onto the wire is about 4uS for a NIC of that vintage, dropping to 2uS with openonload (sokarflare's kernel bypass).

As ippolitiv suggests, your cache already has room for 1.6M items- that's a huge amount. What kind of hit rate are you seeing for your cache?

One way to manage cache size is to only cache popular items- if you set proxy_cache_min_uses =4 then only objects that are requested four times will be cached, which will increase your hit rates and reduce the space needed for the cache.

Peter

Sent from my iPhone

> On Jan 16, 2017, at 4:41 AM, Igor A. Ippolitov <[email protected]> wrote:
>
> Hello,
>
> Your cache have 200m space for keys. This is around 1.6M items, isn't it?
> How much files do you have in your cache? May we have a look at
> `df -i ` and `du -s /cache/123` output, please?
>
>> On 06.01.2017 08:40, omkar_jadhav_20 wrote:
>> Hi,
>>
>> I am using nginx as webserver with nginx version: nginx/1.10.2. For faster
>> access we have mounted cache of nginx of different application on RAM.But
>> even after giving enough buffer of size , now and then cache is getting
>> filled , below are few details of files for your reference :
>> maximum size given in nginx conf file is 500G , while mouting we have given
>> 600G of space i.e. 100G of buffer.But still it is getting filled 100%.
>>
>> fstab entries :
>> tmpfs /cache/123 tmpfs defaults,size=600G
>> 0 0
>> tmpfs /cache/456 tmpfs defaults,size=60G
>> 0 0
>> tmpfs /cache/789 tmpfs defaults,size=110G
>> 0 0
>>
>> cache getting filled , df output:
>>
>> tmpfs tmpfs 60G 17G 44G 28%
>> /cache/456
>> tmpfs tmpfs 110G 323M 110G 1%
>> /cache/789
>> tmpfs tmpfs 600G 600G 0 100%
>> /cache/123
>>
>> nginx conf details :
>>
>> proxy_cache_path /cache/123 keys_zone=a123:200m levels=1:2 max_size=500g
>> inactive=3d;
>>
>> server{
>> listen 80;
>> server_name dvr.catchup.com;
>> location ~.*.m3u8 {
>> access_log /var/log/nginx/access_123.log access;
>> proxy_cache off;
>> root /xyz/123;
>> if (!-e $request_filename) {
>> #origin url will be used if content is not available on DS
>> proxy_pass http://10.10.10.1X;
>> }
>> }
>> location / {
>> access_log /var/log/nginx/access_123.log access;
>> proxy_cache_valid 3d;
>> proxy_cache a123;
>> root /xyz/123;
>> if (!-e $request_filename) {
>> #origin url will be used if content is not available on server
>> proxy_pass http://10.10.10.1X;
>> }
>> proxy_cache_key $proxy_host$uri;
>> }
>> }
>>
>> Posted at Nginx Forum: https://forum.nginx.org/read.php?2,271842,271842#msg-271842
>>
>> _______________________________________________
>> nginx mailing list
>> nginx@nginx.org
>> http://mailman.nginx.org/mailman/listinfo/nginx
>
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
steve
Re: nginx cache mounted on tmpf getting fulled
January 17, 2017 04:40AM
Hi,


On 01/17/2017 02:37 PM, Peter Booth wrote:
> I'm curious, why are you using tmpfs for your cache store? With fast local storage bring so cheap, why don't you devote a few TB to your cache?
>
> When I look at the techempower benchmarks I see that openresty (an nginx build that comes with lots of lua value add) can serve 440,000 JSON responses per sec with 3ms latency. That's on five year old E7-4850 Westmere hardware at 2.0GHz, with 10G NICs. The min latency to get a packet from nginx through the kernel stack and onto the wire is about 4uS for a NIC of that vintage, dropping to 2uS with openonload (sokarflare's kernel bypass).
>
> As ippolitiv suggests, your cache already has room for 1.6M items- that's a huge amount. What kind of hit rate are you seeing for your cache?
>
> One way to manage cache size is to only cache popular items- if you set proxy_cache_min_uses =4 then only objects that are requested four times will be cached, which will increase your hit rates and reduce the space needed for the cache.
>
> Peter
>
> Sent from my iPhone
>
>> On Jan 16, 2017, at 4:41 AM, Igor A. Ippolitov <[email protected]> wrote:
>>
>> Hello,
>>
>> Your cache have 200m space for keys. This is around 1.6M items, isn't it?
>> How much files do you have in your cache? May we have a look at
>> `df -i ` and `du -s /cache/123` output, please?
>>
>>> On 06.01.2017 08:40, omkar_jadhav_20 wrote:
>>> Hi,
>>>
>>> I am using nginx as webserver with nginx version: nginx/1.10.2. For faster
>>> access we have mounted cache of nginx of different application on RAM.But
>>> even after giving enough buffer of size , now and then cache is getting
>>> filled , below are few details of files for your reference :
>>> maximum size given in nginx conf file is 500G , while mouting we have given
>>> 600G of space i.e. 100G of buffer.But still it is getting filled 100%.
>>>
>>> fstab entries :
>>> tmpfs /cache/123 tmpfs defaults,size=600G
>>> 0 0
>>> tmpfs /cache/456 tmpfs defaults,size=60G
>>> 0 0
>>> tmpfs /cache/789 tmpfs defaults,size=110G
>>> 0 0
>>>
>>> cache getting filled , df output:
>>>
>>> tmpfs tmpfs 60G 17G 44G 28%
>>> /cache/456
>>> tmpfs tmpfs 110G 323M 110G 1%
>>> /cache/789
>>> tmpfs tmpfs 600G 600G 0 100%
>>> /cache/123
>>>
>>> nginx conf details :
>>>
>>> proxy_cache_path /cache/123 keys_zone=a123:200m levels=1:2 max_size=500g
>>> inactive=3d;
>>>
>>> server{
>>> listen 80;
>>> server_name dvr.catchup.com;
>>> location ~.*.m3u8 {
>>> access_log /var/log/nginx/access_123.log access;
>>> proxy_cache off;
>>> root /xyz/123;
>>> if (!-e $request_filename) {
>>> #origin url will be used if content is not available on DS
>>> proxy_pass http://10.10.10.1X;
>>> }
>>> }
>>> location / {
>>> access_log /var/log/nginx/access_123.log access;
>>> proxy_cache_valid 3d;
>>> proxy_cache a123;
>>> root /xyz/123;
>>> if (!-e $request_filename) {
>>> #origin url will be used if content is not available on server
>>> proxy_pass http://10.10.10.1X;
>>> }
>>> proxy_cache_key $proxy_host$uri;
>>> }
>>> }
>>>
>>> Posted at Nginx Forum: https://forum.nginx.org/read.php?2,271842,271842#msg-271842
>>>
>>> _______________________________________________
>>> nginx mailing list
>>> nginx@nginx.org
>>> http://mailman.nginx.org/mailman/listinfo/nginx
>>
>> _______________________________________________
>> nginx mailing list
>> nginx@nginx.org
>> http://mailman.nginx.org/mailman/listinfo/nginx
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx

So that's a total of 1TB of memory allocated to caches. Do you have that
much spare on your server? Linux will allocate *up to* the specified
amount *as long as it's spare*. It would be worth looking at your server
to ensure that 1TB memory is spare before blaming nginx.

You can further improve performance and safety by mounting them

nodev,noexec,nosuid,noatime,async,size=xxxM,mode=0755,uid=xx,gid=xx

To answer this poster... memory is even faster!

Steve

--
Steve Holdoway BSc(Hons) MIITP
http://www.greengecko.co.nz
Linkedin: http://www.linkedin.com/in/steveholdoway
Skype: sholdowa

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Sorry, only registered users may post in this forum.

Click here to login