Welcome! Log In Create A New Profile

Advanced

big difference between request time and upstreams time

Posted by Руслан Закиров 
Руслан Закиров
big difference between request time and upstreams time
May 08, 2018 01:50PM
Hello,

Some selected log records:
14:27:46 1.609 [0.013] [0.002] [192.168.1.44:5002]
14:27:50 1.017 [0.017] [0.001] [192.168.1.24:9000]
14:27:51 1.522 [0.021] [0.000] [192.168.1.92:9000]
14:27:50 1.019 [0.019] [0.000] [192.168.1.41:9000]
14:27:52 1.019 [0.018] [0.000] [192.168.1.49:9000]
14:27:52 1.019 [0.018] [0.001] [192.168.1.59:9000]
14:27:55 1.515 [0.014] [0.000] [192.168.1.92:9000]
14:27:57 0.510 [0.010] [0.001] [192.168.1.21:9000]
14:28:03 1.521 [0.021] [0.001] [192.168.1.48:9000]
14:28:04 0.660 [0.007] [0.002] [192.168.1.24:5002]
14:28:05 2.216 [0.018] [0.002] [192.168.1.44:5002]
14:28:11 0.510 [0.010] [0.000] [192.168.1.49:9000]
14:28:26 0.937 [0.008] [0.002] [192.168.1.92:5002]
14:28:28 1.019 [0.019] [0.000] [192.168.1.49:9000]
14:28:28 0.508 [0.007] [0.000] [192.168.1.42:9000]
14:28:31 1.021 [0.019] [0.000] [192.168.1.44:9000]
14:28:32 0.509 [0.008] [0.000] [192.168.1.48:9000]
14:28:36 1.015 [0.015] [0.000] [192.168.1.43:9000]
14:28:39 0.358 [0.007] [0.001] [192.168.1.92:5002]

columns: wallclock time, request time, upstream_request_time,
upstream_connect_time, upstream.

Please, help me diagnose this problem further as I stuck. This is subset
where request_time 50x bigger than upstream_request_time (just to make
subset less noisy). I see request times up to 60 seconds. Can not tie it to
some periodicity. It happens so often that don't see anything helpful in
strace... I stuck... Any ideas?

This is nginx/1.10.2 on FreeBSD 10.3-RELEASE-p7.

--
Руслан Закиров
Руководитель отдела разработки веб-сервисов
+7(916) 597-92-69, ruz @ http://www.sports.ru/
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Igor A. Ippolitov
Re: big difference between request time and upstreams time
May 08, 2018 02:00PM
Можно я отвечу ссылкой?
http://mailman.nginx.org/pipermail/nginx/2008-October/008025.html
Проблема, скорее всего, в клиентах.

On 08.05.2018 14:43, Руслан Закиров wrote:
> Hello,
>
> Some selected log records:
> 14:27:46 1.609 [0.013] [0.002] [192.168.1.44:5002
> http://192.168.1.44:5002]
> 14:27:50 1.017 [0.017] [0.001] [192.168.1.24:9000
> http://192.168.1.24:9000]
> 14:27:51 1.522 [0.021] [0.000] [192.168.1.92:9000
> http://192.168.1.92:9000]
> 14:27:50 1.019 [0.019] [0.000] [192.168.1.41:9000
> http://192.168.1.41:9000]
> 14:27:52 1.019 [0.018] [0.000] [192.168.1.49:9000
> http://192.168.1.49:9000]
> 14:27:52 1.019 [0.018] [0.001] [192.168.1.59:9000
> http://192.168.1.59:9000]
> 14:27:55 1.515 [0.014] [0.000] [192.168.1.92:9000
> http://192.168.1.92:9000]
> 14:27:57 0.510 [0.010] [0.001] [192.168.1.21:9000
> http://192.168.1.21:9000]
> 14:28:03 1.521 [0.021] [0.001] [192.168.1.48:9000
> http://192.168.1.48:9000]
> 14:28:04 0.660 [0.007] [0.002] [192.168.1.24:5002
> http://192.168.1.24:5002]
> 14:28:05 2.216 [0.018] [0.002] [192.168.1.44:5002
> http://192.168.1.44:5002]
> 14:28:11 0.510 [0.010] [0.000] [192.168.1.49:9000
> http://192.168.1.49:9000]
> 14:28:26 0.937 [0.008] [0.002] [192.168.1.92:5002
> http://192.168.1.92:5002]
> 14:28:28 1.019 [0.019] [0.000] [192.168.1.49:9000
> http://192.168.1.49:9000]
> 14:28:28 0.508 [0.007] [0.000] [192.168.1.42:9000
> http://192.168.1.42:9000]
> 14:28:31 1.021 [0.019] [0.000] [192.168.1.44:9000
> http://192.168.1.44:9000]
> 14:28:32 0.509 [0.008] [0.000] [192.168.1.48:9000
> http://192.168.1.48:9000]
> 14:28:36 1.015 [0.015] [0.000] [192.168.1.43:9000
> http://192.168.1.43:9000]
> 14:28:39 0.358 [0.007] [0.001] [192.168.1.92:5002
> http://192.168.1.92:5002]
>
> columns: wallclock time, request time, upstream_request_time,
> upstream_connect_time, upstream.
>
> Please, help me diagnose this problem further as I stuck. This is
> subset where request_time 50x bigger than upstream_request_time (just
> to make subset less noisy). I see request times up to 60 seconds. Can
> not tie it to some periodicity. It happens so often that don't see
> anything helpful in strace... I stuck... Any ideas?
>
> This is nginx/1.10.2 on FreeBSD 10.3-RELEASE-p7.
>
> --
> Руслан Закиров
> Руководитель отдела разработки веб-сервисов
> +7(916) 597-92-69, ruz @ http://www.sports.ru/
>
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx


_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Igor A. Ippolitov
Re: big difference between request time and upstreams time
May 08, 2018 02:20PM
Sorry, didn't realize this is an English mailing list.

To sum it up: the problem is most likely about clients and not the server.
Discrepancy between request time and upstream time usually means that a
client is slow or uses a bad connection.
Basically, this is OK unless you have the only server out of many with
this problem.
This in turn may mean that the problem is with that server's network
connection.

Regards.

On 08.05.2018 14:50, Igor A. Ippolitov wrote:
> Можно я отвечу ссылкой?
> http://mailman.nginx.org/pipermail/nginx/2008-October/008025.html
> Проблема, скорее всего, в клиентах.
>
> On 08.05.2018 14:43, Руслан Закиров wrote:
>> Hello,
>>
>> Some selected log records:
>> 14:27:46 1.609 [0.013] [0.002] [192.168.1.44:5002
>> http://192.168.1.44:5002]
>> 14:27:50 1.017 [0.017] [0.001] [192.168.1.24:9000
>> http://192.168.1.24:9000]
>> 14:27:51 1.522 [0.021] [0.000] [192.168.1.92:9000
>> http://192.168.1.92:9000]
>> 14:27:50 1.019 [0.019] [0.000] [192.168.1.41:9000
>> http://192.168.1.41:9000]
>> 14:27:52 1.019 [0.018] [0.000] [192.168.1.49:9000
>> http://192.168.1.49:9000]
>> 14:27:52 1.019 [0.018] [0.001] [192.168.1.59:9000
>> http://192.168.1.59:9000]
>> 14:27:55 1.515 [0.014] [0.000] [192.168.1.92:9000
>> http://192.168.1.92:9000]
>> 14:27:57 0.510 [0.010] [0.001] [192.168.1.21:9000
>> http://192.168.1.21:9000]
>> 14:28:03 1.521 [0.021] [0.001] [192.168.1.48:9000
>> http://192.168.1.48:9000]
>> 14:28:04 0.660 [0.007] [0.002] [192.168.1.24:5002
>> http://192.168.1.24:5002]
>> 14:28:05 2.216 [0.018] [0.002] [192.168.1.44:5002
>> http://192.168.1.44:5002]
>> 14:28:11 0.510 [0.010] [0.000] [192.168.1.49:9000
>> http://192.168.1.49:9000]
>> 14:28:26 0.937 [0.008] [0.002] [192.168.1.92:5002
>> http://192.168.1.92:5002]
>> 14:28:28 1.019 [0.019] [0.000] [192.168.1.49:9000
>> http://192.168.1.49:9000]
>> 14:28:28 0.508 [0.007] [0.000] [192.168.1.42:9000
>> http://192.168.1.42:9000]
>> 14:28:31 1.021 [0.019] [0.000] [192.168.1.44:9000
>> http://192.168.1.44:9000]
>> 14:28:32 0.509 [0.008] [0.000] [192.168.1.48:9000
>> http://192.168.1.48:9000]
>> 14:28:36 1.015 [0.015] [0.000] [192.168.1.43:9000
>> http://192.168.1.43:9000]
>> 14:28:39 0.358 [0.007] [0.001] [192.168.1.92:5002
>> http://192.168.1.92:5002]
>>
>> columns: wallclock time, request time, upstream_request_time,
>> upstream_connect_time, upstream.
>>
>> Please, help me diagnose this problem further as I stuck. This is
>> subset where request_time 50x bigger than upstream_request_time (just
>> to make subset less noisy). I see request times up to 60 seconds. Can
>> not tie it to some periodicity. It happens so often that don't see
>> anything helpful in strace... I stuck... Any ideas?
>>
>> This is nginx/1.10.2 on FreeBSD 10.3-RELEASE-p7.
>>
>> --
>> Руслан Закиров
>> Руководитель отдела разработки веб-сервисов
>> +7(916) 597-92-69, ruz @ http://www.sports.ru/
>>
>>
>> _______________________________________________
>> nginx mailing list
>> nginx@nginx.org
>> http://mailman.nginx.org/mailman/listinfo/nginx
>
>
>
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx


_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Руслан Закиров
Re: big difference between request time and upstreams time
May 08, 2018 06:00PM
On Tue, May 8, 2018 at 3:11 PM, Igor A. Ippolitov <[email protected]>
wrote:

> Sorry, didn't realize this is an English mailing list.
>
> To sum it up: the problem is most likely about clients and not the server..
> Discrepancy between request time and upstream time usually means that a
> client is slow or uses a bad connection.
> Basically, this is OK unless you have the only server out of many with
> this problem.
> This in turn may mean that the problem is with that server's network
> connection.
>


The issue affects all of our primary nginx servers.

However, they receive requests from 4 "routing" nginx servers and all
backends via haproxy. The problem affects only
requests from the routing nginxs, not backends. I would expect routing
servers pull data from upstream ASAP. So slow
clients in my mind should only affect those routing servers standing in
front.

Am I wrong?


> Regards.
>
>
> On 08.05.2018 14:50, Igor A. Ippolitov wrote:
>
> Можно я отвечу ссылкой? http://mailman.nginx.org/
> pipermail/nginx/2008-October/008025.html
> Проблема, скорее всего, в клиентах.
>
> On 08.05.2018 14:43, Руслан Закиров wrote:
>
> Hello,
>
> Some selected log records:
> 14:27:46 1.609 [0.013] [0.002] [192.168.1.44:5002]
> 14:27:50 1.017 [0.017] [0.001] [192.168.1.24:9000]
> 14:27:51 1.522 [0.021] [0.000] [192.168.1.92:9000]
> 14:27:50 1.019 [0.019] [0.000] [192.168.1.41:9000]
> 14:27:52 1.019 [0.018] [0.000] [192.168.1.49:9000]
> 14:27:52 1.019 [0.018] [0.001] [192.168.1.59:9000]
> 14:27:55 1.515 [0.014] [0.000] [192.168.1.92:9000]
> 14:27:57 0.510 [0.010] [0.001] [192.168.1.21:9000]
> 14:28:03 1.521 [0.021] [0.001] [192.168.1.48:9000]
> 14:28:04 0.660 [0.007] [0.002] [192.168.1.24:5002]
> 14:28:05 2.216 [0.018] [0.002] [192.168.1.44:5002]
> 14:28:11 0.510 [0.010] [0.000] [192.168.1.49:9000]
> 14:28:26 0.937 [0.008] [0.002] [192.168.1.92:5002]
> 14:28:28 1.019 [0.019] [0.000] [192.168.1.49:9000]
> 14:28:28 0.508 [0.007] [0.000] [192.168.1.42:9000]
> 14:28:31 1.021 [0.019] [0.000] [192.168.1.44:9000]
> 14:28:32 0.509 [0.008] [0.000] [192.168.1.48:9000]
> 14:28:36 1.015 [0.015] [0.000] [192.168.1.43:9000]
> 14:28:39 0.358 [0.007] [0.001] [192.168.1.92:5002]
>
> columns: wallclock time, request time, upstream_request_time,
> upstream_connect_time, upstream.
>
> Please, help me diagnose this problem further as I stuck. This is subset
> where request_time 50x bigger than upstream_request_time (just to make
> subset less noisy). I see request times up to 60 seconds. Can not tie it to
> some periodicity. It happens so often that don't see anything helpful in
> strace... I stuck... Any ideas?
>
> This is nginx/1.10.2 on FreeBSD 10.3-RELEASE-p7.
>
> --
> Руслан Закиров
> Руководитель отдела разработки веб-сервисов
> +7(916) 597-92-69, ruz @ http://www.sports.ru/
>
>
> _______________________________________________
> nginx mailing listnginx@nginx.orghttp://mailman.nginx.org/mailman/listinfo/nginx
>
>
>
>
> _______________________________________________
> nginx mailing listnginx@nginx.orghttp://mailman.nginx.org/mailman/listinfo/nginx
>
>
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx
>



--
Руслан Закиров
Руководитель отдела разработки веб-сервисов
+7(916) 597-92-69, ruz @ http://www.sports.ru/
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Igor A. Ippolitov
Re: big difference between request time and upstreams time
May 08, 2018 06:30PM
Ruslan,

This depends on your routing nginx configuration.
If doesn't have enough buffers to contain a response completely and
temporary files are turned off, then you will run into a situation, when
the delay is propagated from client facing nginx to a middle layer nginx.

The fact that only client facing requests are affected proves this idea.

On 08.05.2018 18:51, Руслан Закиров wrote:
>
>
> On Tue, May 8, 2018 at 3:11 PM, Igor A. Ippolitov
> <[email protected] <mailto:[email protected]>> wrote:
>
> Sorry, didn't realize this is an English mailing list.
>
> To sum it up: the problem is most likely about clients and not the
> server.
> Discrepancy between request time and upstream time usually means
> that a client is slow or uses a bad connection.
> Basically, this is OK unless you have the only server out of many
> with this problem.
> This in turn may mean that the problem is with that server's
> network connection.
>
>
>
> The issue affects all of our primary nginx servers.
>
> However, they receive requests from 4 "routing" nginx servers and all
> backends via haproxy. The problem affects only
> requests from the routing nginxs, not backends. I would expect routing
> servers pull data from upstream ASAP. So slow
> clients in my mind should only affect those routing servers standing
> in front.
>
> Am I wrong?
>
>
> Regards.
>
>
> On 08.05.2018 14:50, Igor A. Ippolitov wrote:
>> Можно я отвечу ссылкой?
>> http://mailman.nginx.org/pipermail/nginx/2008-October/008025.html
>> http://mailman.nginx.org/pipermail/nginx/2008-October/008025.html
>> Проблема, скорее всего, в клиентах.
>>
>> On 08.05.2018 14:43, Руслан Закиров wrote:
>>> Hello,
>>>
>>> Some selected log records:
>>> 14:27:46 1.609 [0.013] [0.002] [192.168.1.44:5002
>>> http://192.168.1.44:5002]
>>> 14:27:50 1.017 [0.017] [0.001] [192.168.1.24:9000
>>> http://192.168.1.24:9000]
>>> 14:27:51 1.522 [0.021] [0.000] [192.168.1.92:9000
>>> http://192.168.1.92:9000]
>>> 14:27:50 1.019 [0.019] [0.000] [192.168.1.41:9000
>>> http://192.168.1.41:9000]
>>> 14:27:52 1.019 [0.018] [0.000] [192.168.1.49:9000
>>> http://192.168.1.49:9000]
>>> 14:27:52 1.019 [0.018] [0.001] [192.168.1.59:9000
>>> http://192.168.1.59:9000]
>>> 14:27:55 1.515 [0.014] [0.000] [192.168.1.92:9000
>>> http://192.168.1.92:9000]
>>> 14:27:57 0.510 [0.010] [0.001] [192.168.1.21:9000
>>> http://192.168.1.21:9000]
>>> 14:28:03 1.521 [0.021] [0.001] [192.168.1.48:9000
>>> http://192.168.1.48:9000]
>>> 14:28:04 0.660 [0.007] [0.002] [192.168.1.24:5002
>>> http://192.168.1.24:5002]
>>> 14:28:05 2.216 [0.018] [0.002] [192.168.1.44:5002
>>> http://192.168.1.44:5002]
>>> 14:28:11 0.510 [0.010] [0.000] [192.168.1.49:9000
>>> http://192.168.1.49:9000]
>>> 14:28:26 0.937 [0.008] [0.002] [192.168.1.92:5002
>>> http://192.168.1.92:5002]
>>> 14:28:28 1.019 [0.019] [0.000] [192.168.1.49:9000
>>> http://192.168.1.49:9000]
>>> 14:28:28 0.508 [0.007] [0.000] [192.168.1.42:9000
>>> http://192.168.1.42:9000]
>>> 14:28:31 1.021 [0.019] [0.000] [192.168.1.44:9000
>>> http://192.168.1.44:9000]
>>> 14:28:32 0.509 [0.008] [0.000] [192.168.1.48:9000
>>> http://192.168.1.48:9000]
>>> 14:28:36 1.015 [0.015] [0.000] [192.168.1.43:9000
>>> http://192.168.1.43:9000]
>>> 14:28:39 0.358 [0.007] [0.001] [192.168.1.92:5002
>>> http://192.168.1.92:5002]
>>>
>>> columns: wallclock time, request time, upstream_request_time,
>>> upstream_connect_time, upstream.
>>>
>>> Please, help me diagnose this problem further as I stuck. This
>>> is subset where request_time 50x bigger than
>>> upstream_request_time (just to make subset less noisy). I see
>>> request times up to 60 seconds. Can not tie it to some
>>> periodicity. It happens so often that don't see anything helpful
>>> in strace... I stuck... Any ideas?
>>>
>>> This is nginx/1.10.2 on FreeBSD 10.3-RELEASE-p7.
>>>
>>> --
>>> Руслан Закиров
>>> Руководитель отдела разработки веб-сервисов
>>> +7(916) 597-92-69, ruz @ http://www.sports.ru/
>>>
>>>
>>> _______________________________________________
>>> nginx mailing list
>>> nginx@nginx.org <mailto:[email protected]>
>>> http://mailman.nginx.org/mailman/listinfo/nginx
>>> http://mailman.nginx.org/mailman/listinfo/nginx
>>
>>
>>
>>
>> _______________________________________________
>> nginx mailing list
>> nginx@nginx.org <mailto:[email protected]>
>> http://mailman.nginx.org/mailman/listinfo/nginx
>> http://mailman.nginx.org/mailman/listinfo/nginx
>
>
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org <mailto:[email protected]>
> http://mailman.nginx.org/mailman/listinfo/nginx
> http://mailman.nginx.org/mailman/listinfo/nginx
>
>
>
>
> --
> Руслан Закиров
> Руководитель отдела разработки веб-сервисов
> +7(916) 597-92-69, ruz @ http://www.sports.ru/
>
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx


_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Руслан Закиров
Re: big difference between request time and upstreams time
May 08, 2018 08:10PM
On Tue, May 8, 2018 at 7:22 PM, Igor A. Ippolitov <[email protected]>
wrote:

> Ruslan,
>
> This depends on your routing nginx configuration.
> If doesn't have enough buffers to contain a response completely and
> temporary files are turned off, then you will run into a situation, when
> the delay is propagated from client facing nginx to a middle layer nginx.
>
> The fact that only client facing requests are affected proves this idea.
>

Sure it sounds very much like my case. Any pointers on good article on this
subject? Probably my goal is to free "primary" nginx servers as soon as
possible and leave last mile delivery job to "routing" nginx in front. If
there is no articles you know about on this matter then just point me at
nginx options I should start from.


> On 08.05.2018 18:51, Руслан Закиров wrote:
>
>
>
> On Tue, May 8, 2018 at 3:11 PM, Igor A. Ippolitov <[email protected]>
> wrote:
>
>> Sorry, didn't realize this is an English mailing list.
>>
>> To sum it up: the problem is most likely about clients and not the server.
>> Discrepancy between request time and upstream time usually means that a
>> client is slow or uses a bad connection.
>> Basically, this is OK unless you have the only server out of many with
>> this problem.
>> This in turn may mean that the problem is with that server's network
>> connection.
>>
>
>
> The issue affects all of our primary nginx servers.
>
> However, they receive requests from 4 "routing" nginx servers and all
> backends via haproxy. The problem affects only
> requests from the routing nginxs, not backends. I would expect routing
> servers pull data from upstream ASAP. So slow
> clients in my mind should only affect those routing servers standing in
> front.
>
> Am I wrong?
>
>
>> Regards.
>>
>>
>> On 08.05.2018 14:50, Igor A. Ippolitov wrote:
>>
>> Можно я отвечу ссылкой? http://mailman.nginx.org/piper
>> mail/nginx/2008-October/008025.html
>> Проблема, скорее всего, в клиентах.
>>
>> On 08.05.2018 14:43, Руслан Закиров wrote:
>>
>> Hello,
>>
>> Some selected log records:
>> 14:27:46 1.609 [0.013] [0.002] [192.168.1.44:5002]
>> 14:27:50 1.017 [0.017] [0.001] [192.168.1.24:9000]
>> 14:27:51 1.522 [0.021] [0.000] [192.168.1.92:9000]
>> 14:27:50 1.019 [0.019] [0.000] [192.168.1.41:9000]
>> 14:27:52 1.019 [0.018] [0.000] [192.168.1.49:9000]
>> 14:27:52 1.019 [0.018] [0.001] [192.168.1.59:9000]
>> 14:27:55 1.515 [0.014] [0.000] [192.168.1.92:9000]
>> 14:27:57 0.510 [0.010] [0.001] [192.168.1.21:9000]
>> 14:28:03 1.521 [0.021] [0.001] [192.168.1.48:9000]
>> 14:28:04 0.660 [0.007] [0.002] [192.168.1.24:5002]
>> 14:28:05 2.216 [0.018] [0.002] [192.168.1.44:5002]
>> 14:28:11 0.510 [0.010] [0.000] [192.168.1.49:9000]
>> 14:28:26 0.937 [0.008] [0.002] [192.168.1.92:5002]
>> 14:28:28 1.019 [0.019] [0.000] [192.168.1.49:9000]
>> 14:28:28 0.508 [0.007] [0.000] [192.168.1.42:9000]
>> 14:28:31 1.021 [0.019] [0.000] [192.168.1.44:9000]
>> 14:28:32 0.509 [0.008] [0.000] [192.168.1.48:9000]
>> 14:28:36 1.015 [0.015] [0.000] [192.168.1.43:9000]
>> 14:28:39 0.358 [0.007] [0.001] [192.168.1.92:5002]
>>
>> columns: wallclock time, request time, upstream_request_time,
>> upstream_connect_time, upstream.
>>
>> Please, help me diagnose this problem further as I stuck. This is subset
>> where request_time 50x bigger than upstream_request_time (just to make
>> subset less noisy). I see request times up to 60 seconds. Can not tie it to
>> some periodicity. It happens so often that don't see anything helpful in
>> strace... I stuck... Any ideas?
>>
>> This is nginx/1.10.2 on FreeBSD 10.3-RELEASE-p7.
>>
>> --
>> Руслан Закиров
>> Руководитель отдела разработки веб-сервисов
>> +7(916) 597-92-69, ruz @ http://www.sports.ru/
>>
>>
>> _______________________________________________
>> nginx mailing listnginx@nginx.orghttp://mailman.nginx.org/mailman/listinfo/nginx
>>
>>
>>
>>
>> _______________________________________________
>> nginx mailing listnginx@nginx.orghttp://mailman.nginx.org/mailman/listinfo/nginx
>>
>>
>>
>> _______________________________________________
>> nginx mailing list
>> nginx@nginx.org
>> http://mailman.nginx.org/mailman/listinfo/nginx
>>
>
>
>
> --
> Руслан Закиров
> Руководитель отдела разработки веб-сервисов
> +7(916) 597-92-69, ruz @ http://www.sports.ru/
>
>
> _______________________________________________
> nginx mailing listnginx@nginx.orghttp://mailman.nginx.org/mailman/listinfo/nginx
>
>
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx
>



--
Руслан Закиров
Руководитель отдела разработки веб-сервисов
+7(916) 597-92-69, ruz @ http://www.sports.ru/
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Igor A. Ippolitov
Re: big difference between request time and upstreams time
May 08, 2018 08:20PM
Ruslan,

Not sure if I know a good article on the topic.
Just ensure proxy_buffering is 'on', proxy_buffer size covers maximum
possible reply headers  size and proxy_buffers matches 90% margin of
your replies (or whatever you think is appropriate).
Most of time these recommendations ensures optimal performance for nginx
as a proxy.

But more interesting question is if you really need to tune anything.
If your front edge servers are well loaded, do you really need to load
them even more?

May be someone else will help with a proper text to read.

On 08.05.2018 21:04, Руслан Закиров wrote:
>
>
> On Tue, May 8, 2018 at 7:22 PM, Igor A. Ippolitov
> <[email protected] <mailto:[email protected]>> wrote:
>
> Ruslan,
>
> This depends on your routing nginx configuration.
> If doesn't have enough buffers to contain a response completely
> and temporary files are turned off, then you will run into a
> situation, when the delay is propagated from client facing nginx
> to a middle layer nginx.
>
> The fact that only client facing requests are affected proves this
> idea.
>
>
> Sure it sounds very much like my case. Any pointers on good article on
> this subject? Probably my goal is to free "primary" nginx servers as
> soon as possible and leave last mile delivery job to "routing" nginx
> in front. If there is no articles you know about on this matter then
> just point me at nginx options I should start from.
>
>
> On 08.05.2018 18:51, Руслан Закиров wrote:
>>
>>
>> On Tue, May 8, 2018 at 3:11 PM, Igor A. Ippolitov
>> <[email protected] <mailto:[email protected]>> wrote:
>>
>> Sorry, didn't realize this is an English mailing list.
>>
>> To sum it up: the problem is most likely about clients and
>> not the server.
>> Discrepancy between request time and upstream time usually
>> means that a client is slow or uses a bad connection.
>> Basically, this is OK unless you have the only server out of
>> many with this problem.
>> This in turn may mean that the problem is with that server's
>> network connection.
>>
>>
>>
>> The issue affects all of our primary nginx servers.
>>
>> However, they receive requests from 4 "routing" nginx servers and
>> all backends via haproxy. The problem affects only
>> requests from the routing nginxs, not backends. I would expect
>> routing servers pull data from upstream ASAP. So slow
>> clients in my mind should only affect those routing servers
>> standing in front.
>>
>> Am I wrong?
>>
>>
>> Regards.
>>
>>
>> On 08.05.2018 14:50, Igor A. Ippolitov wrote:
>>> Можно я отвечу ссылкой?
>>> http://mailman.nginx.org/pipermail/nginx/2008-October/008025.html
>>> http://mailman.nginx.org/pipermail/nginx/2008-October/008025.html
>>> Проблема, скорее всего, в клиентах.
>>>
>>> On 08.05.2018 14:43, Руслан Закиров wrote:
>>>> Hello,
>>>>
>>>> Some selected log records:
>>>> 14:27:46 1.609 [0.013] [0.002] [192.168.1.44:5002
>>>> http://192.168.1.44:5002]
>>>> 14:27:50 1.017 [0.017] [0.001] [192.168.1.24:9000
>>>> http://192.168.1.24:9000]
>>>> 14:27:51 1.522 [0.021] [0.000] [192.168.1.92:9000
>>>> http://192.168.1.92:9000]
>>>> 14:27:50 1.019 [0.019] [0.000] [192.168.1.41:9000
>>>> http://192.168.1.41:9000]
>>>> 14:27:52 1.019 [0.018] [0.000] [192.168.1.49:9000
>>>> http://192.168.1.49:9000]
>>>> 14:27:52 1.019 [0.018] [0.001] [192.168.1.59:9000
>>>> http://192.168.1.59:9000]
>>>> 14:27:55 1.515 [0.014] [0.000] [192.168.1.92:9000
>>>> http://192.168.1.92:9000]
>>>> 14:27:57 0.510 [0.010] [0.001] [192.168.1.21:9000
>>>> http://192.168.1.21:9000]
>>>> 14:28:03 1.521 [0.021] [0.001] [192.168.1.48:9000
>>>> http://192.168.1.48:9000]
>>>> 14:28:04 0.660 [0.007] [0.002] [192.168.1.24:5002
>>>> http://192.168.1.24:5002]
>>>> 14:28:05 2.216 [0.018] [0.002] [192.168.1.44:5002
>>>> http://192.168.1.44:5002]
>>>> 14:28:11 0.510 [0.010] [0.000] [192.168.1.49:9000
>>>> http://192.168.1.49:9000]
>>>> 14:28:26 0.937 [0.008] [0.002] [192.168.1.92:5002
>>>> http://192.168.1.92:5002]
>>>> 14:28:28 1.019 [0.019] [0.000] [192.168.1.49:9000
>>>> http://192.168.1.49:9000]
>>>> 14:28:28 0.508 [0.007] [0.000] [192.168.1.42:9000
>>>> http://192.168.1.42:9000]
>>>> 14:28:31 1.021 [0.019] [0.000] [192.168.1.44:9000
>>>> http://192.168.1.44:9000]
>>>> 14:28:32 0.509 [0.008] [0.000] [192.168.1.48:9000
>>>> http://192.168.1.48:9000]
>>>> 14:28:36 1.015 [0.015] [0.000] [192.168.1.43:9000
>>>> http://192.168.1.43:9000]
>>>> 14:28:39 0.358 [0.007] [0.001] [192.168.1.92:5002
>>>> http://192.168.1.92:5002]
>>>>
>>>> columns: wallclock time, request time,
>>>> upstream_request_time, upstream_connect_time, upstream.
>>>>
>>>> Please, help me diagnose this problem further as I stuck.
>>>> This is subset where request_time 50x bigger than
>>>> upstream_request_time (just to make subset less noisy). I
>>>> see request times up to 60 seconds. Can not tie it to some
>>>> periodicity. It happens so often that don't see anything
>>>> helpful in strace... I stuck... Any ideas?
>>>>
>>>> This is nginx/1.10.2 on FreeBSD 10.3-RELEASE-p7.
>>>>
>>>> --
>>>> Руслан Закиров
>>>> Руководитель отдела разработки веб-сервисов
>>>> +7(916) 597-92-69, ruz @ http://www.sports.ru/
>>>>
>>>>
>>>> _______________________________________________
>>>> nginx mailing list
>>>> nginx@nginx.org <mailto:[email protected]>
>>>> http://mailman.nginx.org/mailman/listinfo/nginx
>>>> http://mailman.nginx.org/mailman/listinfo/nginx
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> nginx mailing list
>>> nginx@nginx.org <mailto:[email protected]>
>>> http://mailman.nginx.org/mailman/listinfo/nginx
>>> http://mailman.nginx.org/mailman/listinfo/nginx
>>
>>
>>
>> _______________________________________________
>> nginx mailing list
>> nginx@nginx.org <mailto:[email protected]>
>> http://mailman.nginx.org/mailman/listinfo/nginx
>> http://mailman.nginx.org/mailman/listinfo/nginx
>>
>>
>>
>>
>> --
>> Руслан Закиров
>> Руководитель отдела разработки веб-сервисов
>> +7(916) 597-92-69, ruz @ http://www.sports.ru/
>>
>>
>> _______________________________________________
>> nginx mailing list
>> nginx@nginx.org <mailto:[email protected]>
>> http://mailman.nginx.org/mailman/listinfo/nginx
>> http://mailman.nginx.org/mailman/listinfo/nginx
>
>
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org <mailto:[email protected]>
> http://mailman.nginx.org/mailman/listinfo/nginx
> http://mailman.nginx.org/mailman/listinfo/nginx
>
>
>
>
> --
> Руслан Закиров
> Руководитель отдела разработки веб-сервисов
> +7(916) 597-92-69, ruz @ http://www.sports.ru/
>
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx


_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Sorry, only registered users may post in this forum.

Click here to login