Welcome! Log In Create A New Profile

Advanced

proxy_cache_background_update leads to 200 ms delay

Posted by stephan13360 
stephan13360
proxy_cache_background_update leads to 200 ms delay
July 06, 2018 02:40PM
I recently set proxy_cache_valid 200 to 1 second, down from 15 minutes to
refresh the content more often (With proxy_cache_background_update on;
already activated long before that).
Our Web monitoring, checking our site every minute, showed an increase in
response time following this change.

After some investigating I pinned it down to a ~200 ms delay coming from
using proxy_cache_background_update.

With an EXPIRED cache and proxy_cache_background_update disabled the site
takes around 500 ms to load.
With a cache HIT its takes around ~5 ms.
Then I stopped php-fpm, disabled proxy_cache_background_update and got a
STALE response (proxy_cache_use_stale error) taking ~6 ms.
The I started php-fpm again and enabled proxy_cache_background_update, wehen
the cache expired I get a STALE response taking around ~200 ms

I got the times I measured using curl's time_total on localhost, so network
delay is out of the picture. I ran the tests dozens of times and the
milliseconds are very consistent.

I have a few sites / server showing this behavior and some that don't.
The server in this example is an AWS EC2 t2.small with SSD storage. Only
NGINX and the cache is running on this Instance, PHP ist running on a
different server.

Currently I don't now why this is happening an would appreciate any hints.

Example Output from cURL:

================EXPIRED================
HTTP/1.1 200 OK
Server: nginx
Date: Fri, 06 Jul 2018 12:00:46 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 112834
Connection: keep-alive
Vary: Accept-Encoding
Vary: Accept-Encoding
Content-Language: de
X-Cache: EXPIRED

Status Code: 200
Lookup time: 0.004181 s
Connect time (TCP): 0.004236 s
Connect time (SSL): 0.000000 s
Pretransfer time: 0.004259 s
Starttransfer time: 0.503897 s
Size download: 112834 bytes
Speed download: 223248.000 bytes/s

Total time: 0.505418 s

================HIT================
HTTP/1.1 200 OK
Server: nginx
Date: Fri, 06 Jul 2018 12:01:23 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 112834
Connection: keep-alive
Vary: Accept-Encoding
Vary: Accept-Encoding
Content-Language: de
X-Cache: HIT

Status Code: 200
Lookup time: 0.004200 s
Connect time (TCP): 0.004257 s
Connect time (SSL): 0.000000 s
Pretransfer time: 0.004281 s
Starttransfer time: 0.005348 s
Size download: 112834 bytes
Speed download: 20642883.000 bytes/s

Total time: 0.005466 s

================STALE (proxy_cache_background_update on)================
HTTP/1.1 200 OK
Server: nginx
Date: Fri, 06 Jul 2018 12:02:28 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 112834
Connection: keep-alive
Vary: Accept-Encoding
Vary: Accept-Encoding
Content-Language: de
X-Cache: STALE

Status Code: 200
Lookup time: 0.004203 s
Connect time (TCP): 0.004267 s
Connect time (SSL): 0.000000 s
Pretransfer time: 0.004294 s
Starttransfer time: 0.004870 s
Size download: 112834 bytes
Speed download: 537563.000 bytes/s

Total time: 0.209899 s

================STALE (proxy_cache_background_update off)================
HTTP/1.1 200 OK
Server: nginx
Date: Fri, 06 Jul 2018 12:03:03 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 112834
Connection: keep-alive
Vary: Accept-Encoding
Vary: Accept-Encoding
Content-Language: de
X-Cache: STALE

Status Code: 200
Lookup time: 0.004204 s
Connect time (TCP): 0.004262 s
Connect time (SSL): 0.000000 s
Pretransfer time: 0.004289 s
Starttransfer time: 0.005906 s
Size download: 112834 bytes
Speed download: 18752534.000 bytes/s

Total time: 0.006017 s

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,280434,280434#msg-280434

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
stephan13360
Re: proxy_cache_background_update leads to 200 ms delay
July 06, 2018 04:00PM
To clarify a few things:

The delay has nothing to do with the proxy_cache_valid 200 time, this change
only made me realize that there seems to be a problem. Before this change
our web monitoring would always get a cache HIT, and after it mostly got a
STALE response because it checks every minute.

We use NGINX 1.15.0 mainline.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,280434,280437#msg-280437

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Valentin V. Bartenev
Re: proxy_cache_background_update leads to 200 ms delay
July 07, 2018 01:20PM
On Friday, 6 July 2018 15:32:07 MSK stephan13360 wrote:
> I recently set proxy_cache_valid 200 to 1 second, down from 15 minutes to
> refresh the content more often (With proxy_cache_background_update on;
> already activated long before that).
> Our Web monitoring, checking our site every minute, showed an increase in
> response time following this change.
>
> After some investigating I pinned it down to a ~200 ms delay coming from
> using proxy_cache_background_update.
>
[..]

I assume you have tcp_nopush directive enabled, then please try switching it off.

wbr, Valentin V. Bartenev



_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
stephan13360
Re: proxy_cache_background_update leads to 200 ms delay
July 07, 2018 02:20PM
Wow, thats it! The delay is gone.

For now I am satisfied that the delay is gone and will read up some more on
tcp_nopush.

For the future: Is there any information on why the combination of
tcp_nopush and proxy_cache_background_update create the delay and not the
STALE response you get when the backend ist down and
proxy_cache_background_update is off?

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,280434,280444#msg-280444

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
It's not a combination of tcp_nopush and proxy_cache_background_update that creates this delay.

tcp_nopush (TCP_CORK in Linux) introduces a delay of packets being sent for up to 200ms or until the packet size gets to the defined MTU.

proxy_cache_background_update (if I remember correctly), will do the common checks at the origin to check if a file changed, since this request performed is (often) less than the MTU, you'll end up having to wait for the 200ms delay.

So disabling tcp_nopush also disables the 200ms delay.

´╗┐On 07/07/2018, 14.15, "nginx on behalf of stephan13360" <[email protected] on behalf of nginx-forum@forum.nginx.org> wrote:

Wow, thats it! The delay is gone.

For now I am satisfied that the delay is gone and will read up some more on
tcp_nopush.

For the future: Is there any information on why the combination of
tcp_nopush and proxy_cache_background_update create the delay and not the
STALE response you get when the backend ist down and
proxy_cache_background_update is off?

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,280434,280444#msg-280444

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx


_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Valentin V. Bartenev
Re: proxy_cache_background_update leads to 200 ms delay
July 09, 2018 02:20PM
On Saturday 07 July 2018 08:15:43 stephan13360 wrote:
> Wow, thats it! The delay is gone.
>
> For now I am satisfied that the delay is gone and will read up some more on
> tcp_nopush.
>
> For the future: Is there any information on why the combination of
> tcp_nopush and proxy_cache_background_update create the delay and not the
> STALE response you get when the backend ist down and
> proxy_cache_background_update is off?
>

When a client connection is closing or switching to keepalive state,
the response body left in the socket is explicitly pushed. In current
implementation the client connection is kept "busy" during background
update and the last chunk may rest in the socket until kernel will send
it.

wbr, Valentin V. Bartenev

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Sorry, only registered users may post in this forum.

Click here to login