All workers in 'D' state using sendfile

Hello,

I have tried to summarize this as much as possible but it's still a lot of
text. I apologize but wanted to make sure that I provide enough
information to explain the issue properly.

I'm hoping that somebody that uses nginx as a high traffic/concurrency
download server will be able to shed some light on this issue. I've tried
as many things as I can think of and everything keeps pointing to it being
an issue with nginx, not the server - but I am of course more than willing
to try any suggestions provided.

*Background:*
Approx. 1,500 - 5,000 concurrent connections (peak / off-peak),
Files vary in size from 5MB to 2GB,
All downloads; only very small dynamic content scripts run on these servers
and none take more than 1-3 seconds,
File are hosted on direct-attached AoE storage with a dedicated 10GE link,
Server is running nginx-1.0.11, php-fpm 5.3 and CentOS 5.8x64
(2.6.18-308.4.1.el5.centos.plus).
Specs are: Dual Xeon E5649 (6 Core), 32GB RAM, 300GB 10k SAS HDD, AoE DAS
over 10GE
Download speeds are restricted by the PHP handoff using X-Accel-Redirect,
but obviously not when I'm testing ;)

*Issue:*
After running for a short, but random period of time (5min ~ 90min) all
nginx workers will eventually end up in a 'D' state according to ps/top.
This causes all downloads to run extremely slowly (~25kb/s) but it doesn't
seem to be caused by I/O because an scp of the same file will complete at
the expected speed of ~750MB+/s.

I usually run with worker_processes set to 13, but I've had to raise this
to 50 to prevent the issue. This works short term, but I'm guessing
eventually I will need to restart nginx to fix it.

*Config:*
I'm using sendfile with epoll, and using the following events / http
settings (I've removed the location block with the fastcgi handler, etc):

events {
worker_connections 16384;
use epoll;
}

http {
....

sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 0;

....

location /internalAccess/ {
internal;
alias /data/;
}
}



Kind Regards,

Drew
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx

Hello!

On Sat, May 12, 2012 at 08:28:14PM +1000, Drew Wareham wrote:

> Hello,
>
> I have tried to summarize this as much as possible but it's still a lot of
> text. I apologize but wanted to make sure that I provide enough
> information to explain the issue properly.
>
> I'm hoping that somebody that uses nginx as a high traffic/concurrency
> download server will be able to shed some light on this issue. I've tried
> as many things as I can think of and everything keeps pointing to it being
> an issue with nginx, not the server - but I am of course more than willing
> to try any suggestions provided.
>
> *Background:*
> Approx. 1,500 - 5,000 concurrent connections (peak / off-peak),
> Files vary in size from 5MB to 2GB,
> All downloads; only very small dynamic content scripts run on these servers
> and none take more than 1-3 seconds,
> File are hosted on direct-attached AoE storage with a dedicated 10GE link,
> Server is running nginx-1.0.11, php-fpm 5.3 and CentOS 5.8x64
> (2.6.18-308.4.1.el5.centos.plus).
> Specs are: Dual Xeon E5649 (6 Core), 32GB RAM, 300GB 10k SAS HDD, AoE DAS
> over 10GE
> Download speeds are restricted by the PHP handoff using X-Accel-Redirect,
> but obviously not when I'm testing ;)
>
> *Issue:*
> After running for a short, but random period of time (5min ~ 90min) all
> nginx workers will eventually end up in a 'D' state according to ps/top.
> This causes all downloads to run extremely slowly (~25kb/s) but it doesn't
> seem to be caused by I/O because an scp of the same file will complete at
> the expected speed of ~750MB+/s.
>
> I usually run with worker_processes set to 13, but I've had to raise this
> to 50 to prevent the issue. This works short term, but I'm guessing
> eventually I will need to restart nginx to fix it.
>
> *Config:*
> I'm using sendfile with epoll, and using the following events / http
> settings (I've removed the location block with the fastcgi handler, etc):

With rotational disks you have to optimize iops to minimize seeks.
This includes:

1. Switch off sendfile, it works bad on such workloads under linux
due to no ability to control readahead (and hence blocks read from
disk).

2. Use large output buffers, something like

output_buffers 1 512k

would be a good starting point.

3. Try using aio to ensure better disk concurrency (and note under
linux it needs directio as well), i.e. something like this

aio on;
directio 512;

(this will require newer kernel though, but using 2.6.18 nowadays
looks like bad idea, at least if you need speed)

4. Try tuning io scheduler, there have been reports that deadline
might be better for such workloads.

More details can be found here:

http://nginx.org/r/output_buffers
http://nginx.org/r/aio
http://nginx.org/r/directio

Maxim Dounin

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx

Hi Maxim,

Thanks for your reply and sorry for the delay in responding!

I've applied your suggested changes to three servers in the cluster -
hopefully that will give me an accurate idea of their effectiveness. I'll
report back when I have more useful info.


Thanks again,

Drew


On Sat, May 12, 2012 at 9:18 PM, Maxim Dounin <[email protected]> wrote:

> Hello!
>
> On Sat, May 12, 2012 at 08:28:14PM +1000, Drew Wareham wrote:
>
> > Hello,
> >
> > I have tried to summarize this as much as possible but it's still a lot
> of
> > text. I apologize but wanted to make sure that I provide enough
> > information to explain the issue properly.
> >
> > I'm hoping that somebody that uses nginx as a high traffic/concurrency
> > download server will be able to shed some light on this issue. I've
> tried
> > as many things as I can think of and everything keeps pointing to it
> being
> > an issue with nginx, not the server - but I am of course more than
> willing
> > to try any suggestions provided.
> >
> > *Background:*
> > Approx. 1,500 - 5,000 concurrent connections (peak / off-peak),
> > Files vary in size from 5MB to 2GB,
> > All downloads; only very small dynamic content scripts run on these
> servers
> > and none take more than 1-3 seconds,
> > File are hosted on direct-attached AoE storage with a dedicated 10GE
> link,
> > Server is running nginx-1.0.11, php-fpm 5.3 and CentOS 5.8x64
> > (2.6.18-308.4.1.el5.centos.plus).
> > Specs are: Dual Xeon E5649 (6 Core), 32GB RAM, 300GB 10k SAS HDD, AoE DAS
> > over 10GE
> > Download speeds are restricted by the PHP handoff using X-Accel-Redirect,
> > but obviously not when I'm testing ;)
> >
> > *Issue:*
> > After running for a short, but random period of time (5min ~ 90min) all
> > nginx workers will eventually end up in a 'D' state according to ps/top.
> > This causes all downloads to run extremely slowly (~25kb/s) but it
> doesn't
> > seem to be caused by I/O because an scp of the same file will complete at
> > the expected speed of ~750MB+/s.
> >
> > I usually run with worker_processes set to 13, but I've had to raise this
> > to 50 to prevent the issue. This works short term, but I'm guessing
> > eventually I will need to restart nginx to fix it.
> >
> > *Config:*
> > I'm using sendfile with epoll, and using the following events / http
> > settings (I've removed the location block with the fastcgi handler, etc):
>
> With rotational disks you have to optimize iops to minimize seeks.
> This includes:
>
> 1. Switch off sendfile, it works bad on such workloads under linux
> due to no ability to control readahead (and hence blocks read from
> disk).
>
> 2. Use large output buffers, something like
>
> output_buffers 1 512k
>
> would be a good starting point.
>
> 3. Try using aio to ensure better disk concurrency (and note under
> linux it needs directio as well), i.e. something like this
>
> aio on;
> directio 512;
>
> (this will require newer kernel though, but using 2.6.18 nowadays
> looks like bad idea, at least if you need speed)
>
> 4. Try tuning io scheduler, there have been reports that deadline
> might be better for such workloads.
>
> More details can be found here:
>
> http://nginx.org/r/output_buffers
> http://nginx.org/r/aio
> http://nginx.org/r/directio
>
> Maxim Dounin
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx
>
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx

Hi,

Unfortunately the changes did not seem to be the silver bullet I was hoping
for - however - although the processes are still sleeping, and the servers
are still getting very high loads, it does seem to have helped - downloads
are no longer unable to start, which is great!

I'm not sure if it's helpful, but an strace on a sleeping process looks
like this:

io_submit(47044328767488, 1, {{0x13dab860, 0, 0, 0, 66}}) = 1
epoll_wait(9, {{EPOLLOUT, {u32=1554189657, u64=47044330982745}}, {EPOLLOUT,
{u32=1554179536, u64=47044330972624}}}, 512, 36) = 2
writev(250,
[{"\321\4\vS\0313\237F\222\337\246\322\33(\30\245=g\352\271\2\361\244p\240\377Q\314\2371^\256"...,
161636}], 1) = 20440
writev(69,
[{"\235\337}\33\201\214)\321\343\233\22\346$z\374\2126T\\j\210\250L\250\331{\220\333\323\343\341J"...,
386840}], 1) = 130320
epoll_wait(9, {{EPOLLOUT, {u32=1554194624, u64=47044330987712}}}, 512, 9) =
1
writev(222,
[{"<\247\260`P\237\2455\236\244\352!\237s\223h\25\206N3[{\351f\31\275\344b\5\204\f\v"...,
396976}], 1) = 95568
epoll_wait(9, {{EPOLLOUT, {u32=1554205480, u64=47044330998568}}}, 512, 26)
= 1
writev(286,
[{"E\260q\214\346X[\376\305\5\275\352\344`\256q\327\344m\r\236\t\321\354\200\325\333\351E\340\374\232"...,
240116}], 1) = 64240
epoll_wait(9, {{EPOLLOUT, {u32=1554195361, u64=47044330988449}}}, 512, 25)
= 1
writev(133,
[{"\243Y\373y\10\0252\34\32\22\2\36\227\325e\345\333\372=\365./\340\34V\251U\0373\234\35\250"...,
13732}], 1) = 13732
io_submit(47044328767488, 1, {{0x12599628, 0, 0, 0, 209}}) = 1
epoll_wait(9, {{EPOLLOUT, {u32=1554199961, u64=47044330993049}}}, 512, 5) =
1
writev(49,
[{"+\347^\17\322\354\201\20=\35\246b\200\0\214K'z>\344k\331\272Svh\234`\334)|\205"...,
176592}], 1) = 84120
epoll_wait(9, {}, 512, 1) = 0
epoll_wait(9, {}, 512, 4) = 0
epoll_wait(9, {{EPOLLOUT, {u32=1554179905, u64=47044330972993}}}, 512, 14)
= 1
epoll_wait(9, {{EPOLLOUT, {u32=1554193521, u64=47044330986609}}}, 512, 10)
= 1
writev(137,
[{"\212\375\216\330'\315^\20|\350N\362\25j\272\304=v\227\210?\3539S\343\6D\265C-\360J"...,
336856}], 1) = 96360
epoll_wait(9, {{EPOLLOUT, {u32=1554181193, u64=47044330974281}}}, 512, 9) =
1
writev(79,
[{"\321\277\340\360E\323A\352\352\377\357w\357m_\377\377R\0\200\177\365l\200
\314D\24z\21U\0"..., 228056}], 1) = 128480
epoll_wait(9, {}, 512, 3) = 0
epoll_wait(9, {}, 512, 8) = 0
epoll_wait(9, {}, 512, 2) = 0
epoll_wait(9, {{EPOLLOUT, {u32=1554204009, u64=47044330997097}}}, 512, 26)
= 1
writev(67, [{"\204-&
V\325?\375\33\202B\236\216\r\240\360\17\0103\25\274\3\300>\352\267\211BJ\265\23\327"...,
166588}], 1) = 26280
epoll_wait(9, {}, 512, 12) = 0
epoll_wait(9, {{EPOLLIN, {u32=6779072, u64=6779072}}}, 512, 14) = 1
read(10, "\1\0\0\0\0\0\0\0", 8) = 8
io_getevents(47044328767488, 1, 64, {{0x13dab860, 0x13dab820, 524288, 0}},
{0, 0}) = 1
writev(80,
[{"X\361N8\2\214\203n\263t\240\\\335\241k\212N\366\24\222\32\201u\267\272\32\v\326=\373\34\v"...,
524288}], 1) = 56608
epoll_wait(9, {{EPOLLOUT, {u32=1554187265, u64=47044330980353}}}, 512, 13)
= 1
epoll_wait(9, {{EPOLLOUT, {u32=1554183217, u64=47044330976305}}}, 512, 3) =
1
epoll_wait(9, {}, 512, 1) = 0
epoll_wait(9, {{EPOLLOUT, {u32=1554181744, u64=47044330974832}}}, 512, 13)
= 1
writev(121,
[{"\371s\222d\231\313\17\t\227\31\33a\315\304\365NZ7\323\200\347\337\260\355\253\203\30\215N\313\260d"...,
331027}], 1) = 49640
epoll_wait(9, {}, 512, 2) = 0
epoll_wait(9, {}, 512, 8) = 0
epoll_wait(9, {{EPOLLOUT, {u32=1554192968, u64=47044330986056}}}, 512, 12)
= 1
epoll_wait(9, {{EPOLLIN, {u32=6779072, u64=6779072}}}, 512, 12) = 1
read(10, "\1\0\0\0\0\0\0\0", 8) = 8
io_getevents(47044328767488, 1, 64, {{0x12599628, 0x125995e8, 524288, 0}},
{0, 0}) = 1
writev(133,
[{"&yj\373dw\335\364\232k\310\6\204\363\365=c{V\257\6:\225\354\233\253b\27*\221\4\264"...,
524288}], 1) = 114748
epoll_wait(9, {{EPOLLOUT, {u32=1554203825, u64=47044330996913}}}, 512, 6) =
1
epoll_wait(9, {{EPOLLOUT, {u32=1554179536, u64=47044330972624}}}, 512, 5) =
1
writev(69,
[{"X\4\250\274\3415\212A\20D\30\2122.\23\351n%\226\245\250\242b$\271\t\22/fX\303\263"...,
256520}], 1) = 130320
epoll_wait(9, {{EPOLLOUT, {u32=1554192784, u64=47044330985872}}}, 512, 27)
= 1


An strace -c taken only when a process is in the 'D' state shows:

[root@HOST16 ~]# time strace -p 22334 -c
Process 22334 attached - interrupt to quit
Process 22334 detached
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
70.65 0.008273 109 76 io_submit
29.35 0.003437 10 360 writev
0.00 0.000000 0 26 read
0.00 0.000000 0 3 open
0.00 0.000000 0 8 close
0.00 0.000000 0 3 fstat
0.00 0.000000 0 1 ioctl
0.00 0.000000 0 1 socket
0.00 0.000000 0 1 1 connect
0.00 0.000000 0 11 3 recvfrom
0.00 0.000000 0 1 getsockname
0.00 0.000000 0 3 getsockopt
0.00 0.000000 0 44 fcntl
0.00 0.000000 0 26 io_getevents
0.00 0.000000 0 180 epoll_wait
0.00 0.000000 0 4 epoll_ctl
------ ----------- ----------- --------- --------- ----------------
100.00 0.011710 748 4 total

real 0m8.570s
user 0m0.016s
sys 0m0.028s

When it's out of sleeping state, about 25% of the time is spent in
epoll_wait. I've also noticed that vmstat shows far less frequent
swapping, although now instead of 5-10MB regularly, it'll swap ~100MB every
30+ seconds.


Cheers,

Drew

On Fri, May 25, 2012 at 2:57 PM, Drew Wareham <[email protected]> wrote:

> Hi Maxim,
>
> Thanks for your reply and sorry for the delay in responding!
>
> I've applied your suggested changes to three servers in the cluster -
> hopefully that will give me an accurate idea of their effectiveness. I'll
> report back when I have more useful info.
>
>
> Thanks again,
>
> Drew
>
>
>
> On Sat, May 12, 2012 at 9:18 PM, Maxim Dounin <[email protected]> wrote:
>
>> Hello!
>>
>> On Sat, May 12, 2012 at 08:28:14PM +1000, Drew Wareham wrote:
>>
>> > Hello,
>> >
>> > I have tried to summarize this as much as possible but it's still a lot
>> of
>> > text. I apologize but wanted to make sure that I provide enough
>> > information to explain the issue properly.
>> >
>> > I'm hoping that somebody that uses nginx as a high traffic/concurrency
>> > download server will be able to shed some light on this issue. I've
>> tried
>> > as many things as I can think of and everything keeps pointing to it
>> being
>> > an issue with nginx, not the server - but I am of course more than
>> willing
>> > to try any suggestions provided.
>> >
>> > *Background:*
>> > Approx. 1,500 - 5,000 concurrent connections (peak / off-peak),
>> > Files vary in size from 5MB to 2GB,
>> > All downloads; only very small dynamic content scripts run on these
>> servers
>> > and none take more than 1-3 seconds,
>> > File are hosted on direct-attached AoE storage with a dedicated 10GE
>> link,
>> > Server is running nginx-1.0.11, php-fpm 5.3 and CentOS 5.8x64
>> > (2.6.18-308.4.1.el5.centos.plus).
>> > Specs are: Dual Xeon E5649 (6 Core), 32GB RAM, 300GB 10k SAS HDD, AoE
>> DAS
>> > over 10GE
>> > Download speeds are restricted by the PHP handoff using
>> X-Accel-Redirect,
>> > but obviously not when I'm testing ;)
>> >
>> > *Issue:*
>> > After running for a short, but random period of time (5min ~ 90min) all
>> > nginx workers will eventually end up in a 'D' state according to ps/top.
>> > This causes all downloads to run extremely slowly (~25kb/s) but it
>> doesn't
>> > seem to be caused by I/O because an scp of the same file will complete
>> at
>> > the expected speed of ~750MB+/s.
>> >
>> > I usually run with worker_processes set to 13, but I've had to raise
>> this
>> > to 50 to prevent the issue. This works short term, but I'm guessing
>> > eventually I will need to restart nginx to fix it.
>> >
>> > *Config:*
>> > I'm using sendfile with epoll, and using the following events / http
>> > settings (I've removed the location block with the fastcgi handler,
>> etc):
>>
>> With rotational disks you have to optimize iops to minimize seeks.
>> This includes:
>>
>> 1. Switch off sendfile, it works bad on such workloads under linux
>> due to no ability to control readahead (and hence blocks read from
>> disk).
>>
>> 2. Use large output buffers, something like
>>
>> output_buffers 1 512k
>>
>> would be a good starting point.
>>
>> 3. Try using aio to ensure better disk concurrency (and note under
>> linux it needs directio as well), i.e. something like this
>>
>> aio on;
>> directio 512;
>>
>> (this will require newer kernel though, but using 2.6.18 nowadays
>> looks like bad idea, at least if you need speed)
>>
>> 4. Try tuning io scheduler, there have been reports that deadline
>> might be better for such workloads.
>>
>> More details can be found here:
>>
>> http://nginx.org/r/output_buffers
>> http://nginx.org/r/aio
>> http://nginx.org/r/directio
>>
>> Maxim Dounin
>>
>> _______________________________________________
>> nginx mailing list
>> nginx@nginx.org
>> http://mailman.nginx.org/mailman/listinfo/nginx
>>
>
>
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx

Hi Maxim,

Is there any planning to develop the thread version?

2012/5/12 Maxim Dounin <[email protected]>:
> Hello!
>
> On Sat, May 12, 2012 at 08:28:14PM +1000, Drew Wareham wrote:
>
>> Hello,
>>
>> I have tried to summarize this as much as possible but it's still a lot of
>> text.  I apologize but wanted to make sure that I provide enough
>> information to explain the issue properly.
>>
>> I'm hoping that somebody that uses nginx as a high traffic/concurrency
>> download server will be able to shed some light on this issue.  I've tried
>> as many things as I can think of and everything keeps pointing to it being
>> an issue with nginx, not the server - but I am of course more than willing
>> to try any suggestions provided.
>>
>> *Background:*
>> Approx. 1,500 - 5,000 concurrent connections (peak / off-peak),
>> Files vary in size from 5MB to 2GB,
>> All downloads; only very small dynamic content scripts run on these servers
>> and none take more than 1-3 seconds,
>> File are hosted on direct-attached AoE storage with a dedicated 10GE link,
>> Server is running nginx-1.0.11, php-fpm 5.3 and CentOS 5.8x64
>> (2.6.18-308.4.1.el5.centos.plus).
>> Specs are: Dual Xeon E5649 (6 Core), 32GB RAM, 300GB 10k SAS HDD, AoE DAS
>> over 10GE
>> Download speeds are restricted by the PHP handoff using X-Accel-Redirect,
>> but obviously not when I'm testing ;)
>>
>> *Issue:*
>> After running for a short, but random period of time (5min ~ 90min) all
>> nginx workers will eventually end up in a 'D' state according to ps/top.
>> This causes all downloads to run extremely slowly (~25kb/s) but it doesn't
>> seem to be caused by I/O because an scp of the same file will complete at
>> the expected speed of ~750MB+/s.
>>
>> I usually run with worker_processes set to 13, but I've had to raise this
>> to 50 to prevent the issue.  This works short term, but I'm guessing
>> eventually I will need to restart nginx to fix it.
>>
>> *Config:*
>> I'm using sendfile with epoll, and using the following events / http
>> settings (I've removed the location block with the fastcgi handler, etc):
>
> With rotational disks you have to optimize iops to minimize seeks.
> This includes:
>
> 1. Switch off sendfile, it works bad on such workloads under linux
> due to no ability to control readahead (and hence blocks read from
> disk).
>
> 2. Use large output buffers, something like
>
>    output_buffers 1 512k
>
> would be a good starting point.
>
> 3. Try using aio to ensure better disk concurrency (and note under
> linux it needs directio as well), i.e. something like this
>
>    aio on;
>    directio 512;
>
> (this will require newer kernel though, but using 2.6.18 nowadays
> looks like bad idea, at least if you need speed)
>
> 4. Try tuning io scheduler, there have been reports that deadline
> might be better for such workloads.
>
> More details can be found here:
>
> http://nginx.org/r/output_buffers
> http://nginx.org/r/aio
> http://nginx.org/r/directio
>
> Maxim Dounin
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx

If you comment/remove "use epoll;" does that help?


On Mon, May 28, 2012 at 10:06 PM, 姚伟斌 <[email protected]> wrote:
> Hi Maxim,
>
> Is there any planning to develop the thread version?
>
> 2012/5/12 Maxim Dounin <[email protected]>:
>> Hello!
>>
>> On Sat, May 12, 2012 at 08:28:14PM +1000, Drew Wareham wrote:
>>
>>> Hello,
>>>
>>> I have tried to summarize this as much as possible but it's still a lot of
>>> text.  I apologize but wanted to make sure that I provide enough
>>> information to explain the issue properly.
>>>
>>> I'm hoping that somebody that uses nginx as a high traffic/concurrency
>>> download server will be able to shed some light on this issue.  I've tried
>>> as many things as I can think of and everything keeps pointing to it being
>>> an issue with nginx, not the server - but I am of course more than willing
>>> to try any suggestions provided.
>>>
>>> *Background:*
>>> Approx. 1,500 - 5,000 concurrent connections (peak / off-peak),
>>> Files vary in size from 5MB to 2GB,
>>> All downloads; only very small dynamic content scripts run on these servers
>>> and none take more than 1-3 seconds,
>>> File are hosted on direct-attached AoE storage with a dedicated 10GE link,
>>> Server is running nginx-1.0.11, php-fpm 5.3 and CentOS 5.8x64
>>> (2.6.18-308.4.1.el5.centos.plus).
>>> Specs are: Dual Xeon E5649 (6 Core), 32GB RAM, 300GB 10k SAS HDD, AoE DAS
>>> over 10GE
>>> Download speeds are restricted by the PHP handoff using X-Accel-Redirect,
>>> but obviously not when I'm testing ;)
>>>
>>> *Issue:*
>>> After running for a short, but random period of time (5min ~ 90min) all
>>> nginx workers will eventually end up in a 'D' state according to ps/top.
>>> This causes all downloads to run extremely slowly (~25kb/s) but it doesn't
>>> seem to be caused by I/O because an scp of the same file will complete at
>>> the expected speed of ~750MB+/s.
>>>
>>> I usually run with worker_processes set to 13, but I've had to raise this
>>> to 50 to prevent the issue.  This works short term, but I'm guessing
>>> eventually I will need to restart nginx to fix it.
>>>
>>> *Config:*
>>> I'm using sendfile with epoll, and using the following events / http
>>> settings (I've removed the location block with the fastcgi handler, etc):
>>
>> With rotational disks you have to optimize iops to minimize seeks.
>> This includes:
>>
>> 1. Switch off sendfile, it works bad on such workloads under linux
>> due to no ability to control readahead (and hence blocks read from
>> disk).
>>
>> 2. Use large output buffers, something like
>>
>>    output_buffers 1 512k
>>
>> would be a good starting point.
>>
>> 3. Try using aio to ensure better disk concurrency (and note under
>> linux it needs directio as well), i.e. something like this
>>
>>    aio on;
>>    directio 512;
>>
>> (this will require newer kernel though, but using 2.6.18 nowadays
>> looks like bad idea, at least if you need speed)
>>
>> 4. Try tuning io scheduler, there have been reports that deadline
>> might be better for such workloads.
>>
>> More details can be found here:
>>
>> http://nginx.org/r/output_buffers
>> http://nginx.org/r/aio
>> http://nginx.org/r/directio
>>
>> Maxim Dounin
>>
>> _______________________________________________
>> nginx mailing list
>> nginx@nginx.org
>> http://mailman.nginx.org/mailman/listinfo/nginx
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx

Hi Ryan,

If I comment out use epoll;, it still seems to use epoll according to
strace. These are the compile flags I use:

../configure --prefix=/usr/local/nginx-1.2.0 --with-pcre
--add-module=/usr/local/src/nginx_upload_module-2.2.0
--with-http_stub_status_module --with-file-aio --without-http_proxy_module
--without-http_memcached_module --without-http_gzip_module
--without-http_ssi_module --without-http_userid_module
--without-http_autoindex_module --without-http_geo_module
--without-http_map_module --without-http_empty_gif_module
--without-http_browser_module --without-http_upstream_ip_hash_module
--without-http_charset_module

This is what I'm running with now:

events {
worker_connections 51200;
# use epoll;
# multi_accept on;
}

http {
include mime.types;
default_type application/octet-stream;

server_tokens off;
sendfile off;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 10;

aio on;
directio 4k;
output_buffers 1 512k;
max_ranges 5;
...


Cheers,

Drew


On Tue, May 29, 2012 at 12:31 AM, Ryan Brown <[email protected]> wrote:

> If you comment/remove "use epoll;" does that help?
>
>
> On Mon, May 28, 2012 at 10:06 PM, 姚伟斌 <[email protected]> wrote:
> > Hi Maxim,
> >
> > Is there any planning to develop the thread version?
> >
> > 2012/5/12 Maxim Dounin <[email protected]>:
> >> Hello!
> >>
> >> On Sat, May 12, 2012 at 08:28:14PM +1000, Drew Wareham wrote:
> >>
> >>> Hello,
> >>>
> >>> I have tried to summarize this as much as possible but it's still a
> lot of
> >>> text. I apologize but wanted to make sure that I provide enough
> >>> information to explain the issue properly.
> >>>
> >>> I'm hoping that somebody that uses nginx as a high traffic/concurrency
> >>> download server will be able to shed some light on this issue. I've
> tried
> >>> as many things as I can think of and everything keeps pointing to it
> being
> >>> an issue with nginx, not the server - but I am of course more than
> willing
> >>> to try any suggestions provided.
> >>>
> >>> *Background:*
> >>> Approx. 1,500 - 5,000 concurrent connections (peak / off-peak),
> >>> Files vary in size from 5MB to 2GB,
> >>> All downloads; only very small dynamic content scripts run on these
> servers
> >>> and none take more than 1-3 seconds,
> >>> File are hosted on direct-attached AoE storage with a dedicated 10GE
> link,
> >>> Server is running nginx-1.0.11, php-fpm 5.3 and CentOS 5.8x64
> >>> (2.6.18-308.4.1.el5.centos.plus).
> >>> Specs are: Dual Xeon E5649 (6 Core), 32GB RAM, 300GB 10k SAS HDD, AoE
> DAS
> >>> over 10GE
> >>> Download speeds are restricted by the PHP handoff using
> X-Accel-Redirect,
> >>> but obviously not when I'm testing ;)
> >>>
> >>> *Issue:*
> >>> After running for a short, but random period of time (5min ~ 90min) all
> >>> nginx workers will eventually end up in a 'D' state according to
> ps/top.
> >>> This causes all downloads to run extremely slowly (~25kb/s) but it
> doesn't
> >>> seem to be caused by I/O because an scp of the same file will complete
> at
> >>> the expected speed of ~750MB+/s.
> >>>
> >>> I usually run with worker_processes set to 13, but I've had to raise
> this
> >>> to 50 to prevent the issue. This works short term, but I'm guessing
> >>> eventually I will need to restart nginx to fix it.
> >>>
> >>> *Config:*
> >>> I'm using sendfile with epoll, and using the following events / http
> >>> settings (I've removed the location block with the fastcgi handler,
> etc):
> >>
> >> With rotational disks you have to optimize iops to minimize seeks.
> >> This includes:
> >>
> >> 1. Switch off sendfile, it works bad on such workloads under linux
> >> due to no ability to control readahead (and hence blocks read from
> >> disk).
> >>
> >> 2. Use large output buffers, something like
> >>
> >> output_buffers 1 512k
> >>
> >> would be a good starting point.
> >>
> >> 3. Try using aio to ensure better disk concurrency (and note under
> >> linux it needs directio as well), i.e. something like this
> >>
> >> aio on;
> >> directio 512;
> >>
> >> (this will require newer kernel though, but using 2.6.18 nowadays
> >> looks like bad idea, at least if you need speed)
> >>
> >> 4. Try tuning io scheduler, there have been reports that deadline
> >> might be better for such workloads.
> >>
> >> More details can be found here:
> >>
> >> http://nginx.org/r/output_buffers
> >> http://nginx.org/r/aio
> >> http://nginx.org/r/directio
> >>
> >> Maxim Dounin
> >>
> >> _______________________________________________
> >> nginx mailing list
> >> nginx@nginx.org
> >> http://mailman.nginx.org/mailman/listinfo/nginx
> >
> > _______________________________________________
> > nginx mailing list
> > nginx@nginx.org
> > http://mailman.nginx.org/mailman/listinfo/nginx
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx
>
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx

Hi,

I am facing the same exact issue as explained by Drew,

is there any working solution to tune nginx for higher throughput?

or how to deal with sleeping D state nginx processes ?

i can post my server specs and nginx conf is needed but I would like to ask
Drew if he found the working properly solution or not

Regards
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx

Hello!

On Sun, Jun 09, 2013 at 06:17:10AM +0430, Host DL wrote:

> I am facing the same exact issue as explained by Drew,
>
> is there any working solution to tune nginx for higher throughput?
>
> or how to deal with sleeping D state nginx processes ?

See this reply for basic tuning suggestions:

http://mailman.nginx.org/pipermail/nginx/2012-May/033761.html

--
Maxim Dounin
http://nginx.org/en/donation.html

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx

Hello Maxim,

Thanks for your response, and sorry that I am new to mailing list and my
1st message may was not very clear to you

I've already read all posts in this conversation and all tuning options has
been tested

I'm using 8x 2TB SATA ENT in RAID10 level + 64G RAM on my box
CentOS 5.9 x64_84 / 2.6.18-348.6.1.el5

nginx.conf:

worker_priority -10;
worker_processes 64;
worker_rlimit_nofile 20000;

events {
worker_connections 2048;
use epoll;
worker_aio_requests 128;
}

http {
sendfile off;
tcp_nopush on;
tcp_nodelay on;
aio on;
directio 2m;
#directio_alignment 4k;
output_buffers 1 1m;

keepalive_timeout 15;

......
}

During the peak time connections will reach up to 14-15K in total and more
than 1Gbit/s outgoing throughput
Please note that the server was stable with about 12K connections in the
peak time and about 1-1.1Gbit/s throughput but after adding another VH with
about 2-3K connections it seems that server is unable to handle the request
properly at the peak time
Its expected the throughput to exceed the previous ~1.1Gbit/s rate but it
doesn't, Even it doesn't reach to 1Gbit/s while the connections are now
getting more and bigger

During every peak time the LA will each to the number of nginx workers ( 64
for my current config ) and will stay at the same rate to the end of peak
time, all processes are in D state and the interesting thing, memory is not
being used fully and it may push about 30-40G with about 20-30% I/O wait

Connections are not being processed fast and it will stay in connecting and
request sent/wait for a few seconds and then data transfer will start at a
very slow rate

I've already tried both sendfile and AIO, and AIO seems to handle the
connections better
Played with output_buffer and increased both number and size of buffers but
no any special effect, even the throughput got lower

The interesting thing is that at the same peak time the read transfer rate
is more than 500mbit/s using scp/rsync from the RAID to /dev/null

Sorry for my long post and if my english sux
Any suggestion would be greatly appreciarted

Respect,
Moozi




============================================


On Sun, Jun 9, 2013 at 4:43 PM, Maxim Dounin <[email protected]> wrote:

> Hello!
>
> On Sun, Jun 09, 2013 at 06:17:10AM +0430, Host DL wrote:
>
> > I am facing the same exact issue as explained by Drew,
> >
> > is there any working solution to tune nginx for higher throughput?
> >
> > or how to deal with sleeping D state nginx processes ?
>
> See this reply for basic tuning suggestions:
>
> http://mailman.nginx.org/pipermail/nginx/2012-May/033761.html
>
> --
> Maxim Dounin
> http://nginx.org/en/donation.html
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx
>
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx

Hello!

On Sun, Jun 09, 2013 at 07:24:49PM +0430, Host DL wrote:

> Hello Maxim,
>
> Thanks for your response, and sorry that I am new to mailing list and my
> 1st message may was not very clear to you
>
> I've already read all posts in this conversation and all tuning options has
> been tested
>
> I'm using 8x 2TB SATA ENT in RAID10 level + 64G RAM on my box
> CentOS 5.9 x64_84 / 2.6.18-348.6.1.el5
>
> nginx.conf:
>
> worker_priority -10;
> worker_processes 64;
> worker_rlimit_nofile 20000;
>
> events {
> worker_connections 2048;
> use epoll;
> worker_aio_requests 128;
> }
>
> http {
> sendfile off;
> tcp_nopush on;
> tcp_nodelay on;
> aio on;
> directio 2m;
> #directio_alignment 4k;
> output_buffers 1 1m;
>
> keepalive_timeout 15;
>
> ......
> }
>
> During the peak time connections will reach up to 14-15K in total and more
> than 1Gbit/s outgoing throughput
> Please note that the server was stable with about 12K connections in the
> peak time and about 1-1.1Gbit/s throughput but after adding another VH with
> about 2-3K connections it seems that server is unable to handle the request
> properly at the peak time
> Its expected the throughput to exceed the previous ~1.1Gbit/s rate but it
> doesn't, Even it doesn't reach to 1Gbit/s while the connections are now
> getting more and bigger
>
> During every peak time the LA will each to the number of nginx workers ( 64
> for my current config ) and will stay at the same rate to the end of peak
> time, all processes are in D state and the interesting thing, memory is not
> being used fully and it may push about 30-40G with about 20-30% I/O wait

Main problem with AIO on Linux is that it requires directio to
actually work asynchronously. I would assume you've just reached
a critical number of synchronous requests to disks as due to
"directio 2m" in your config (and that's why you see all workers
in D state). Try tuning directio to a lower value to see if it
helps.

Note well: memory is not used for filesystem cache with directio,
so there is no surprise it's not being used fully.

--
Maxim Dounin
http://nginx.org/en/donation.html

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx

Dear Maxim,

Thanks for your response,
I don't think it will help much since all of my files are larger enough
than 2MB

Regarding AIO problem in linux, do you think using AIO + sendfile together
on FreeBSD will be better in performance in my case?

Respect



============================================


On Mon, Jun 10, 2013 at 3:25 PM, Maxim Dounin <[email protected]> wrote:

> Hello!
>
> On Sun, Jun 09, 2013 at 07:24:49PM +0430, Host DL wrote:
>
> > Hello Maxim,
> >
> > Thanks for your response, and sorry that I am new to mailing list and my
> > 1st message may was not very clear to you
> >
> > I've already read all posts in this conversation and all tuning options
> has
> > been tested
> >
> > I'm using 8x 2TB SATA ENT in RAID10 level + 64G RAM on my box
> > CentOS 5.9 x64_84 / 2.6.18-348.6.1.el5
> >
> > nginx.conf:
> >
> > worker_priority -10;
> > worker_processes 64;
> > worker_rlimit_nofile 20000;
> >
> > events {
> > worker_connections 2048;
> > use epoll;
> > worker_aio_requests 128;
> > }
> >
> > http {
> > sendfile off;
> > tcp_nopush on;
> > tcp_nodelay on;
> > aio on;
> > directio 2m;
> > #directio_alignment 4k;
> > output_buffers 1 1m;
> >
> > keepalive_timeout 15;
> >
> > ......
> > }
> >
> > During the peak time connections will reach up to 14-15K in total and
> more
> > than 1Gbit/s outgoing throughput
> > Please note that the server was stable with about 12K connections in the
> > peak time and about 1-1.1Gbit/s throughput but after adding another VH
> with
> > about 2-3K connections it seems that server is unable to handle the
> request
> > properly at the peak time
> > Its expected the throughput to exceed the previous ~1.1Gbit/s rate but it
> > doesn't, Even it doesn't reach to 1Gbit/s while the connections are now
> > getting more and bigger
> >
> > During every peak time the LA will each to the number of nginx workers (
> 64
> > for my current config ) and will stay at the same rate to the end of peak
> > time, all processes are in D state and the interesting thing, memory is
> not
> > being used fully and it may push about 30-40G with about 20-30% I/O wait
>
> Main problem with AIO on Linux is that it requires directio to
> actually work asynchronously. I would assume you've just reached
> a critical number of synchronous requests to disks as due to
> "directio 2m" in your config (and that's why you see all workers
> in D state). Try tuning directio to a lower value to see if it
> helps.
>
> Note well: memory is not used for filesystem cache with directio,
> so there is no surprise it's not being used fully.
>
> --
> Maxim Dounin
> http://nginx.org/en/donation.html
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx
>
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx

On Monday 10 June 2013 15:08:45 Host DL wrote:
> Dear Maxim,
>
> Thanks for your response,
> I don't think it will help much since all of my files are larger enough
> than 2MB
[...]

How large? And how big is the entire dataset? Perhaps, quite the opposite you
should increase the directio value for better utilization of pagecache.

wbr, Valentin V. Bartenev

--
http://nginx.org/en/donation.html

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx

Hello!

On Mon, Jun 10, 2013 at 03:38:45PM +0430, Host DL wrote:

> Dear Maxim,
>
> Thanks for your response,
> I don't think it will help much since all of my files are larger enough
> than 2MB

The "D" state of nginx workers has only one explanation: blocking
operations on disks. When serving static files with nginx on
Linux, this basically means one of the following:

1. opening / stat()'ing files
2. blocking aio reads due to no directio on unaligned reads
3. blocking aio reads due to no directio on small files

Within nginx, you may reduce possibility of (3) using directio
directive with a smaller value. Both (1) and (2) are more or less
unavoidable, but aren't likely to happen, at least with proper OS
tuning.

If in doubt, try tracing where worker processes are blocked. As a
very first step, ps(1) output should be examined for a wait
channel column (wchan).

> Regarding AIO problem in linux, do you think using AIO + sendfile together
> on FreeBSD will be better in performance in my case?

Yes.

--
Maxim Dounin
http://nginx.org/en/donation.html

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx

Thanks you for clarification on this issue

The last modification that worked properly was reducing workers
I tried both 64, 24 and 12 workers and less workers was better in
performances during the peak time

I can't recognize why but it works like a charm

PS: Dual E5-2620 has been used which leads to 24 HT core and 12 real core
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx