Welcome! Log In Create A New Profile

Advanced

nginx cache mounted on tmpf getting fulled

Posted by omkar_jadhav_20 
omkar_jadhav_20
nginx cache mounted on tmpf getting fulled
January 06, 2017 06:50AM
Hi,

I am using nginx as webserver with nginx version: nginx/1.10.2. For faster
access we have mounted cache of nginx of different application on RAM.But
even after giving enough buffer of size , now and then cache is getting
filled , below are few details of files for your reference :
maximum size given in nginx conf file is 500G , while mouting we have given
600G of space i.e. 100G of buffer.But still it is getting filled 100%.

fstab entries :
tmpfs /cache/123 tmpfs defaults,size=600G
0 0
tmpfs /cache/456 tmpfs defaults,size=60G
0 0
tmpfs /cache/789 tmpfs defaults,size=110G
0 0

cache getting filled , df output:

tmpfs tmpfs 60G 17G 44G 28%
/cache/456
tmpfs tmpfs 110G 323M 110G 1%
/cache/789
tmpfs tmpfs 600G 600G 0 100%
/cache/123

nginx conf details :

proxy_cache_path /cache/123 keys_zone=a123:200m levels=1:2 max_size=500g
inactive=3d;

server{
listen 80;
server_name dvr.catchup.com;
location ~.*.m3u8 {
access_log /var/log/nginx/access_123.log access;
proxy_cache off;
root /xyz/123;
if (!-e $request_filename) {
#origin url will be used if content is not available on DS
proxy_pass http://10.10.10.1X;
}
}
location / {
access_log /var/log/nginx/access_123.log access;
proxy_cache_valid 3d;
proxy_cache a123;
root /xyz/123;
if (!-e $request_filename) {
#origin url will be used if content is not available on server
proxy_pass http://10.10.10.1X;
}
proxy_cache_key $proxy_host$uri;
}
}

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,271842,271842#msg-271842

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
omkar_jadhav_20
Re: nginx cache mounted on tmpf getting filled 100%
January 06, 2017 06:50AM
omkar_jadhav_20 Wrote:
-------------------------------------------------------
> Hi,
>
> I am using nginx as webserver with nginx version: nginx/1.10.2. For
> faster access we have mounted cache of nginx of different application
> on RAM.But even after giving enough buffer of size , now and then
> cache is getting filled , below are few details of files for your
> reference :
> maximum size given in nginx conf file is 500G , while mouting we have
> given 600G of space i.e. 100G of buffer.But still it is getting filled
> 100%.
>
> fstab entries :
> tmpfs /cache/123 tmpfs
> defaults,size=600G 0 0
> tmpfs /cache/456 tmpfs
> defaults,size=60G 0 0
> tmpfs /cache/789 tmpfs
> defaults,size=110G 0 0
>
> cache getting filled , df output:
>
> tmpfs tmpfs 60G 17G 44G 28%
> /cache/456
> tmpfs tmpfs 110G 323M 110G 1%
> /cache/789
> tmpfs tmpfs 600G 600G 0 100%
> /cache/123
>
> nginx conf details :
>
> proxy_cache_path /cache/123 keys_zone=a123:200m levels=1:2
> max_size=500g inactive=3d;
>
> server{
> listen 80;
> server_name dvr.catchup.com;
> location ~.*.m3u8 {
> access_log /var/log/nginx/access_123.log access;
> proxy_cache off;
> root /xyz/123;
> if (!-e $request_filename) {
> #origin url will be used if content is not available on DS
> proxy_pass http://10.10.10.1X;
> }
> }
> location / {
> access_log /var/log/nginx/access_123.log access;
> proxy_cache_valid 3d;
> proxy_cache a123;
> root /xyz/123;
> if (!-e $request_filename) {
> #origin url will be used if content is not available on server
> proxy_pass http://10.10.10.1X;
> }
> proxy_cache_key $proxy_host$uri;
> }
> }

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,271842,271843#msg-271843

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
omkar_jadhav_20
Re: nginx cache mounted on tmpf getting filled 100%
January 09, 2017 10:10AM
can someone please respond. Let me know if any additional information is
required.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,271842,271870#msg-271870

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Matthew Slowe
Re: nginx cache mounted on tmpf getting fulled
January 09, 2017 10:20AM
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 06/01/2017 05:40, omkar_jadhav_20 wrote:
> Hi,
>
> I am using nginx as webserver with nginx version: nginx/1.10.2. For
> faster access we have mounted cache of nginx of different
> application on RAM.But even after giving enough buffer of size ,
> now and then cache is getting filled , below are few details of
> files for your reference : maximum size given in nginx conf file is
> 500G , while mouting we have given 600G of space i.e. 100G of
> buffer.But still it is getting filled 100%.

Do you actually have enough RAM / swap to facilitate these
requirements? It looks like you'd need about 800G of RAM/swap space to
make this work?

If you do then I don't know enough about how nginx works to advise,
sorry :-)

- --
Matthew Slowe | Server Infrastructure Officer
IT Infrastructure, Information Services, University of Kent
Room S21, Cornwallis South
Canterbury, Kent, CT2 7NZ, UK
Tel: +44 (0)1227 824265

www.kent.ac.uk/is | @UnikentUnseenIT | @UKCLibraryIt
PGP: https://keybase.io/fooflington
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0

iQIcBAEBCAAGBQJYc1NyAAoJEGc7JS6kpZi5IqcP/jjt31iEpOIc9iKa3kd/pE8i
uXa1QIkm2Pw6sU23SYu2tlMo2ExlhG8gCIYHs8nEh0uZvYX/uEaRbCSLXlkdzPBH
RZw3ngv48PhxFZqPWJXWZyDEElE8ybPtf5/ST5VA1Y/QMMwT8sFy3yTMbCD55MV9
RA3SIDFByMAzeMgHaQBErJzFW5UU+NenlGKZjQRZ6upxy1D0YGk1fV5HsI2h4MRx
L9i1afpbbmPE1p6v5q8RYEXxcQsTieRwsV0BSCBlfzcy/VoAC55p9sGy1wFlk4bD
b5XYUi7kX3/absk2vxHLGPJLOSMwD77BAWt3SzYb8RlPwEFvQc4JNED4uNykR51I
mUdAE1F+rxzHMZStoggkeSLKqoRleyJ5ZKRrP6gq5jVEl0m3cWo7pljM5By/eLNR
lyPZkbfIHWQvyRN8pnLVzvYMcrXXjONMOdJCngSppb6Ae1cSC0esGvLXxRIYzySF
sgNfB04H3R66hq3uLGz7rYrPKaAgEKg3XaASso4r3uYYWUvfYmAO5bE8pkZ+M+QW
Lx5vbyNZgPFVNSihZBnryh2WUs1c4LjHYEOoUYSTKirxV6daQiOoPrYfBXXUMwk4
FaBkimO3gu7xMmoe2lOc3KGZuEz2JV1Sqo/orxQ4Pmk5lkgVfSrPDn3h/oL8/ZOW
Zdlw+tZNDm0ULOGlS2tE
=YUuq
-----END PGP SIGNATURE-----
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
omkar_jadhav_20
Re: nginx cache mounted on tmpf getting fulled
January 09, 2017 10:20AM
Yes I do have RAM of size : 1.5T and swap space of around 200G. It has been
observed that swap is not getting used in this case. But it seems either OS
is not clearing the RAM that fast or nginx is not able to control RAM as
mentioned in the nginx configuration file.
Could you please suggest what can be the possible solution for this issue.
Every time max_size is getting crossed by RAM even after mentioning it in
nginx config.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,271842,271873#msg-271873

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
omkar_jadhav_20
Re: nginx cache mounted on tmpf getting fulled
January 16, 2017 08:10AM
can someone please respond and suggest best way to manage cache in highly
utilized cache dependent environment. As number of nginx requests increases
and cache hit ratio becomes significant , the tmpfs crosses the max_size
limit mentioned in the nginx.conf file and use full cache mounted . This in
returns increases the server load by great extent.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,271842,271995#msg-271995

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Igor A. Ippolitov
Re: nginx cache mounted on tmpf getting fulled
January 16, 2017 10:50AM
Hello,

Your cache have 200m space for keys. This is around 1.6M items, isn't it?
How much files do you have in your cache? May we have a look at
`df -i ` and `du -s /cache/123` output, please?

On 06.01.2017 08:40, omkar_jadhav_20 wrote:
> Hi,
>
> I am using nginx as webserver with nginx version: nginx/1.10.2. For faster
> access we have mounted cache of nginx of different application on RAM.But
> even after giving enough buffer of size , now and then cache is getting
> filled , below are few details of files for your reference :
> maximum size given in nginx conf file is 500G , while mouting we have given
> 600G of space i.e. 100G of buffer.But still it is getting filled 100%.
>
> fstab entries :
> tmpfs /cache/123 tmpfs defaults,size=600G
> 0 0
> tmpfs /cache/456 tmpfs defaults,size=60G
> 0 0
> tmpfs /cache/789 tmpfs defaults,size=110G
> 0 0
>
> cache getting filled , df output:
>
> tmpfs tmpfs 60G 17G 44G 28%
> /cache/456
> tmpfs tmpfs 110G 323M 110G 1%
> /cache/789
> tmpfs tmpfs 600G 600G 0 100%
> /cache/123
>
> nginx conf details :
>
> proxy_cache_path /cache/123 keys_zone=a123:200m levels=1:2 max_size=500g
> inactive=3d;
>
> server{
> listen 80;
> server_name dvr.catchup.com;
> location ~.*.m3u8 {
> access_log /var/log/nginx/access_123.log access;
> proxy_cache off;
> root /xyz/123;
> if (!-e $request_filename) {
> #origin url will be used if content is not available on DS
> proxy_pass http://10.10.10.1X;
> }
> }
> location / {
> access_log /var/log/nginx/access_123.log access;
> proxy_cache_valid 3d;
> proxy_cache a123;
> root /xyz/123;
> if (!-e $request_filename) {
> #origin url will be used if content is not available on server
> proxy_pass http://10.10.10.1X;
> }
> proxy_cache_key $proxy_host$uri;
> }
> }
>
> Posted at Nginx Forum: https://forum.nginx.org/read.php?2,271842,271842#msg-271842
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx


_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Peter Booth
Re: nginx cache mounted on tmpf getting fulled
January 17, 2017 02:40AM
I'm curious, why are you using tmpfs for your cache store? With fast local storage bring so cheap, why don't you devote a few TB to your cache?

When I look at the techempower benchmarks I see that openresty (an nginx build that comes with lots of lua value add) can serve 440,000 JSON responses per sec with 3ms latency. That's on five year old E7-4850 Westmere hardware at 2.0GHz, with 10G NICs. The min latency to get a packet from nginx through the kernel stack and onto the wire is about 4uS for a NIC of that vintage, dropping to 2uS with openonload (sokarflare's kernel bypass).

As ippolitiv suggests, your cache already has room for 1.6M items- that's a huge amount. What kind of hit rate are you seeing for your cache?

One way to manage cache size is to only cache popular items- if you set proxy_cache_min_uses =4 then only objects that are requested four times will be cached, which will increase your hit rates and reduce the space needed for the cache.

Peter

Sent from my iPhone

> On Jan 16, 2017, at 4:41 AM, Igor A. Ippolitov <[email protected]> wrote:
>
> Hello,
>
> Your cache have 200m space for keys. This is around 1.6M items, isn't it?
> How much files do you have in your cache? May we have a look at
> `df -i ` and `du -s /cache/123` output, please?
>
>> On 06.01.2017 08:40, omkar_jadhav_20 wrote:
>> Hi,
>>
>> I am using nginx as webserver with nginx version: nginx/1.10.2. For faster
>> access we have mounted cache of nginx of different application on RAM.But
>> even after giving enough buffer of size , now and then cache is getting
>> filled , below are few details of files for your reference :
>> maximum size given in nginx conf file is 500G , while mouting we have given
>> 600G of space i.e. 100G of buffer.But still it is getting filled 100%.
>>
>> fstab entries :
>> tmpfs /cache/123 tmpfs defaults,size=600G
>> 0 0
>> tmpfs /cache/456 tmpfs defaults,size=60G
>> 0 0
>> tmpfs /cache/789 tmpfs defaults,size=110G
>> 0 0
>>
>> cache getting filled , df output:
>>
>> tmpfs tmpfs 60G 17G 44G 28%
>> /cache/456
>> tmpfs tmpfs 110G 323M 110G 1%
>> /cache/789
>> tmpfs tmpfs 600G 600G 0 100%
>> /cache/123
>>
>> nginx conf details :
>>
>> proxy_cache_path /cache/123 keys_zone=a123:200m levels=1:2 max_size=500g
>> inactive=3d;
>>
>> server{
>> listen 80;
>> server_name dvr.catchup.com;
>> location ~.*.m3u8 {
>> access_log /var/log/nginx/access_123.log access;
>> proxy_cache off;
>> root /xyz/123;
>> if (!-e $request_filename) {
>> #origin url will be used if content is not available on DS
>> proxy_pass http://10.10.10.1X;
>> }
>> }
>> location / {
>> access_log /var/log/nginx/access_123.log access;
>> proxy_cache_valid 3d;
>> proxy_cache a123;
>> root /xyz/123;
>> if (!-e $request_filename) {
>> #origin url will be used if content is not available on server
>> proxy_pass http://10.10.10.1X;
>> }
>> proxy_cache_key $proxy_host$uri;
>> }
>> }
>>
>> Posted at Nginx Forum: https://forum.nginx.org/read.php?2,271842,271842#msg-271842
>>
>> _______________________________________________
>> nginx mailing list
>> nginx@nginx.org
>> http://mailman.nginx.org/mailman/listinfo/nginx
>
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
steve
Re: nginx cache mounted on tmpf getting fulled
January 17, 2017 04:40AM
Hi,


On 01/17/2017 02:37 PM, Peter Booth wrote:
> I'm curious, why are you using tmpfs for your cache store? With fast local storage bring so cheap, why don't you devote a few TB to your cache?
>
> When I look at the techempower benchmarks I see that openresty (an nginx build that comes with lots of lua value add) can serve 440,000 JSON responses per sec with 3ms latency. That's on five year old E7-4850 Westmere hardware at 2.0GHz, with 10G NICs. The min latency to get a packet from nginx through the kernel stack and onto the wire is about 4uS for a NIC of that vintage, dropping to 2uS with openonload (sokarflare's kernel bypass).
>
> As ippolitiv suggests, your cache already has room for 1.6M items- that's a huge amount. What kind of hit rate are you seeing for your cache?
>
> One way to manage cache size is to only cache popular items- if you set proxy_cache_min_uses =4 then only objects that are requested four times will be cached, which will increase your hit rates and reduce the space needed for the cache.
>
> Peter
>
> Sent from my iPhone
>
>> On Jan 16, 2017, at 4:41 AM, Igor A. Ippolitov <[email protected]> wrote:
>>
>> Hello,
>>
>> Your cache have 200m space for keys. This is around 1.6M items, isn't it?
>> How much files do you have in your cache? May we have a look at
>> `df -i ` and `du -s /cache/123` output, please?
>>
>>> On 06.01.2017 08:40, omkar_jadhav_20 wrote:
>>> Hi,
>>>
>>> I am using nginx as webserver with nginx version: nginx/1.10.2. For faster
>>> access we have mounted cache of nginx of different application on RAM.But
>>> even after giving enough buffer of size , now and then cache is getting
>>> filled , below are few details of files for your reference :
>>> maximum size given in nginx conf file is 500G , while mouting we have given
>>> 600G of space i.e. 100G of buffer.But still it is getting filled 100%.
>>>
>>> fstab entries :
>>> tmpfs /cache/123 tmpfs defaults,size=600G
>>> 0 0
>>> tmpfs /cache/456 tmpfs defaults,size=60G
>>> 0 0
>>> tmpfs /cache/789 tmpfs defaults,size=110G
>>> 0 0
>>>
>>> cache getting filled , df output:
>>>
>>> tmpfs tmpfs 60G 17G 44G 28%
>>> /cache/456
>>> tmpfs tmpfs 110G 323M 110G 1%
>>> /cache/789
>>> tmpfs tmpfs 600G 600G 0 100%
>>> /cache/123
>>>
>>> nginx conf details :
>>>
>>> proxy_cache_path /cache/123 keys_zone=a123:200m levels=1:2 max_size=500g
>>> inactive=3d;
>>>
>>> server{
>>> listen 80;
>>> server_name dvr.catchup.com;
>>> location ~.*.m3u8 {
>>> access_log /var/log/nginx/access_123.log access;
>>> proxy_cache off;
>>> root /xyz/123;
>>> if (!-e $request_filename) {
>>> #origin url will be used if content is not available on DS
>>> proxy_pass http://10.10.10.1X;
>>> }
>>> }
>>> location / {
>>> access_log /var/log/nginx/access_123.log access;
>>> proxy_cache_valid 3d;
>>> proxy_cache a123;
>>> root /xyz/123;
>>> if (!-e $request_filename) {
>>> #origin url will be used if content is not available on server
>>> proxy_pass http://10.10.10.1X;
>>> }
>>> proxy_cache_key $proxy_host$uri;
>>> }
>>> }
>>>
>>> Posted at Nginx Forum: https://forum.nginx.org/read.php?2,271842,271842#msg-271842
>>>
>>> _______________________________________________
>>> nginx mailing list
>>> nginx@nginx.org
>>> http://mailman.nginx.org/mailman/listinfo/nginx
>>
>> _______________________________________________
>> nginx mailing list
>> nginx@nginx.org
>> http://mailman.nginx.org/mailman/listinfo/nginx
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx

So that's a total of 1TB of memory allocated to caches. Do you have that
much spare on your server? Linux will allocate *up to* the specified
amount *as long as it's spare*. It would be worth looking at your server
to ensure that 1TB memory is spare before blaming nginx.

You can further improve performance and safety by mounting them

nodev,noexec,nosuid,noatime,async,size=xxxM,mode=0755,uid=xx,gid=xx

To answer this poster... memory is even faster!

Steve

--
Steve Holdoway BSc(Hons) MIITP
http://www.greengecko.co.nz
Linkedin: http://www.linkedin.com/in/steveholdoway
Skype: sholdowa

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
omkar_jadhav_20
Re: nginx cache mounted on tmpf getting fulled
January 25, 2017 08:10AM
Hi,

We have used tmpfs to mount frequently used media application cache serving.
We have also mounted rest applications on disk where we are facing similar
issues of max_size being breached . Please find below details for your
reference :

# du -sh /cache/*
245G /cache/12007
161G /cache/12152

# grep a12007 /etc/nginx/nginx.conf
proxy_cache_path /cache/12007 keys_zone=a12007:200m levels=1:2
max_size=200g inactive=10d;
# grep a12152 /etc/nginx/nginx.conf
proxy_cache_path /cache/12152 keys_zone=a12152:200m levels=1:2
max_size=100g inactive=10d;

# df -Th
Filesystem Type Size Used Avail Use% Mounted on
/dev/mapper/vg_os-lv_root ext4 99G 1.5G 92G 2% /
devtmpfs devtmpfs 756G 0 756G 0% /dev
tmpfs tmpfs 756G 0 756G 0% /dev/shm
tmpfs tmpfs 756G 50M 756G 1% /run
tmpfs tmpfs 756G 0 756G 0%
/sys/fs/cgroup
/dev/mapper/vg_os-lv_usr ext4 99G 1.4G 92G 2% /usr
/dev/sda1 ext4 477M 106M 343M 24% /boot
/dev/mapper/vg_cache-lv_cache ext4 1.5T 1.4T 0 100% /cache

Please let us what went wrong here and how this can be corrected.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,271842,272136#msg-272136

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
omkar_jadhav_20
Re: nginx cache mounted on tmpf getting fulled
January 25, 2017 08:20AM
omkar_jadhav_20 Wrote:
-------------------------------------------------------
> Hi,
>
> We have used tmpfs to mount frequently used media application cache
> serving. We have also mounted rest applications on disk where we are
> facing similar issues of max_size being breached . Please find below
> details for your reference :
>
> # du -sh /cache/*
> 245G /cache/12007
> 161G /cache/12152
>
> # grep a12007 /etc/nginx/nginx.conf
> proxy_cache_path /cache/12007 keys_zone=a12007:200m levels=1:2
> max_size=200g inactive=10d;
> # grep a12152 /etc/nginx/nginx.conf
> proxy_cache_path /cache/12152 keys_zone=a12152:200m levels=1:2
> max_size=100g inactive=10d;
>
> # df -Th
> Filesystem Type Size Used Avail Use% Mounted
> on
> /dev/mapper/vg_os-lv_root ext4 99G 1.5G 92G 2% /
> devtmpfs devtmpfs 756G 0 756G 0% /dev
> tmpfs tmpfs 756G 0 756G 0%
> /dev/shm
> tmpfs tmpfs 756G 50M 756G 1% /run
> tmpfs tmpfs 756G 0 756G 0%
> /sys/fs/cgroup
> /dev/mapper/vg_os-lv_usr ext4 99G 1.4G 92G 2% /usr
> /dev/sda1 ext4 477M 106M 343M 24% /boot
> /dev/mapper/vg_cache-lv_cache ext4 1.5T 1.4T 0 100% /cache
>
> Please let us what went wrong here and how this can be corrected.
In addition we are getting 90% of cache hit ratio. as suggested by Steve ,
will just making below mounting option changes in fstab will resolve this
issue or I am missing/misscalculating anything :
nodev,noexec,nosuid,noatime,async,size=xxxM,mode=0755,uid=xx,gid=xx

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,271842,272137#msg-272137

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Peter Booth
Re: nginx cache mounted on tmpf getting fulled
February 20, 2017 06:40AM
Just to be pedantic. It’s counterintuitive but, in general, tmpfs is not faster than local storage, for the use case of caching static content for web servers.
Sounds weird? Here’s why:

tmpfs is a file system view of all of the system’s virtual memory - that is both physical memory and swap space.

If you use local storage for your cache store then every time a file is requested for the first time it gets read from local storage and written to the page cache. Subsequent requests are served from page cache.. The OS manages memory allocation so as to maximize the size of the page cache (subject to config settings snappiness, min_free_kbytes, and the watermark setting for kswapd. The fact that you are using a cache suggests that you have expensive back-end queries that your cache is front of. So the cost of reading a cached file from disk << cost of recreating dynamic content. With real-world web systems the probability distribution of requests for different resources is never uniform, There are clusters of popular resources.

Any popular requests that get served from your disk cache will in fact be being served from page cache (memory), so there is no reason to interfere with the OS. In general its likely that you have less physical memory than disk space., so that using tmpfs for your nginx cache could mean that you’re serving cached files from swap space.

Allowing the OS to do its job (with the page cache) means that you will already get the tmpfs like latencies for the popular resources - which is what you want to maximize performance across your entire site. This is another example of why with issues of web performance, its usually better to test theories than to rely on logical reasoning.

Peter Booth


> On 16 Jan 2017, at 10:34 PM, steve <[email protected]> wrote:
>
> Hi,
>
>
> On 01/17/2017 02:37 PM, Peter Booth wrote:
>> I'm curious, why are you using tmpfs for your cache store? With fast local storage bring so cheap, why don't you devote a few TB to your cache?
>>
>> When I look at the techempower benchmarks I see that openresty (an nginx build that comes with lots of lua value add) can serve 440,000 JSON responses per sec with 3ms latency. That's on five year old E7-4850 Westmere hardware at 2.0GHz, with 10G NICs. The min latency to get a packet from nginx through the kernel stack and onto the wire is about 4uS for a NIC of that vintage, dropping to 2uS with openonload (sokarflare's kernel bypass).
>>
>> As ippolitiv suggests, your cache already has room for 1.6M items- that's a huge amount. What kind of hit rate are you seeing for your cache?
>>
>> One way to manage cache size is to only cache popular items- if you set proxy_cache_min_uses =4 then only objects that are requested four times will be cached, which will increase your hit rates and reduce the space needed for the cache.
>>
>> Peter
>>
>> Sent from my iPhone
>>
>>> On Jan 16, 2017, at 4:41 AM, Igor A. Ippolitov <[email protected]> wrote:
>>>
>>> Hello,
>>>
>>> Your cache have 200m space for keys. This is around 1.6M items, isn't it?
>>> How much files do you have in your cache? May we have a look at
>>> `df -i ` and `du -s /cache/123` output, please?
>>>
>>>> On 06.01.2017 08:40, omkar_jadhav_20 wrote:
>>>> Hi,
>>>>
>>>> I am using nginx as webserver with nginx version: nginx/1.10.2. For faster
>>>> access we have mounted cache of nginx of different application on RAM.But
>>>> even after giving enough buffer of size , now and then cache is getting
>>>> filled , below are few details of files for your reference :
>>>> maximum size given in nginx conf file is 500G , while mouting we have given
>>>> 600G of space i.e. 100G of buffer.But still it is getting filled 100%.
>>>>
>>>> fstab entries :
>>>> tmpfs /cache/123 tmpfs defaults,size=600G
>>>> 0 0
>>>> tmpfs /cache/456 tmpfs defaults,size=60G
>>>> 0 0
>>>> tmpfs /cache/789 tmpfs defaults,size=110G
>>>> 0 0
>>>>
>>>> cache getting filled , df output:
>>>>
>>>> tmpfs tmpfs 60G 17G 44G 28%
>>>> /cache/456
>>>> tmpfs tmpfs 110G 323M 110G 1%
>>>> /cache/789
>>>> tmpfs tmpfs 600G 600G 0 100%
>>>> /cache/123
>>>>
>>>> nginx conf details :
>>>>
>>>> proxy_cache_path /cache/123 keys_zone=a123:200m levels=1:2 max_size=500g
>>>> inactive=3d;
>>>>
>>>> server{
>>>> listen 80;
>>>> server_name dvr.catchup.com;
>>>> location ~.*.m3u8 {
>>>> access_log /var/log/nginx/access_123.log access;
>>>> proxy_cache off;
>>>> root /xyz/123;
>>>> if (!-e $request_filename) {
>>>> #origin url will be used if content is not available on DS
>>>> proxy_pass http://10.10.10.1X;
>>>> }
>>>> }
>>>> location / {
>>>> access_log /var/log/nginx/access_123.log access;
>>>> proxy_cache_valid 3d;
>>>> proxy_cache a123;
>>>> root /xyz/123;
>>>> if (!-e $request_filename) {
>>>> #origin url will be used if content is not available on server
>>>> proxy_pass http://10.10.10.1X;
>>>> }
>>>> proxy_cache_key $proxy_host$uri;
>>>> }
>>>> }
>>>>
>>>> Posted at Nginx Forum: https://forum.nginx.org/read.php?2,271842,271842#msg-271842
>>>>
>>>> _______________________________________________
>>>> nginx mailing list
>>>> nginx@nginx.org
>>>> http://mailman.nginx.org/mailman/listinfo/nginx
>>>
>>> _______________________________________________
>>> nginx mailing list
>>> nginx@nginx.org
>>> http://mailman.nginx.org/mailman/listinfo/nginx
>> _______________________________________________
>> nginx mailing list
>> nginx@nginx.org
>> http://mailman.nginx.org/mailman/listinfo/nginx
>
> So that's a total of 1TB of memory allocated to caches. Do you have that much spare on your server? Linux will allocate *up to* the specified amount *as long as it's spare*. It would be worth looking at your server to ensure that 1TB memory is spare before blaming nginx.
>
> You can further improve performance and safety by mounting them
>
> nodev,noexec,nosuid,noatime,async,size=xxxM,mode=0755,uid=xx,gid=xx
>
> To answer this poster... memory is even faster!
>
> Steve
>
> --
> Steve Holdoway BSc(Hons) MIITP
> http://www.greengecko.co.nz
> Linkedin: http://www.linkedin.com/in/steveholdoway
> Skype: sholdowa
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Sorry, only registered users may post in this forum.

Click here to login