Welcome! Log In Create A New Profile

Advanced

status/usage of FRiCKLE/ngx_cache_purge. still reliable? alternatives?

Posted by PGNet Dev 
For some new WordPress sites, I'll be deploying fastcgi_cache as reverse proxy / page cache, instead of usual Varnish.

Although there are a number of WP-module-based PURGE options, I prefer that it's handled by the web server.

A commonly referenced approach is to use the 'FRiCKLE/ngx_cache_purge',

https://github.com/FRiCKLE/ngx_cache_purge/

with associated nginx conf additions,

https://easyengine.io/wordpress-nginx/tutorials/single-site/fastcgi-cache-with-purging/
https://www.ryadel.com/en/nginx-purge-proxy-cache-delete-invalidate-linux-centos-7/

ngx_cache_purge module development appears to have gone stale; no commits since ~ 2014.

What are your experiences with current use of that module, with latest 1.15x nginx releases?

Is there a cleaner, nginx-native approach? Or other nginx purge module that's better maintained?

Comments &/or pointers to any docs, etc would be helpful.
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Hi,

On Wed, Jun 6, 2018 at 3:05 PM, PGNet Dev <[email protected]> wrote:

> For some new WordPress sites, I'll be deploying fastcgi_cache as reverse
> proxy / page cache, instead of usual Varnish.
>
> Although there are a number of WP-module-based PURGE options, I prefer
> that it's handled by the web server.
>
> A commonly referenced approach is to use the 'FRiCKLE/ngx_cache_purge',
>
> https://github.com/FRiCKLE/ngx_cache_purge/
>
> with associated nginx conf additions,
>
> https://easyengine.io/wordpress-nginx/tutorials/
> single-site/fastcgi-cache-with-purging/
> https://www.ryadel.com/en/nginx-purge-proxy-cache-
> delete-invalidate-linux-centos-7/
>
> ngx_cache_purge module development appears to have gone stale; no commits
> since ~ 2014.
>
> What are your experiences with current use of that module, with latest
> 1.15x nginx releases?
>
> Is there a cleaner, nginx-native approach? Or other nginx purge module
> that's better maintained?
>
> Comments &/or pointers to any docs, etc would be helpful.
>

My $0.02 coming from experience building out scalable WP clusters is, stick
to Varnish here.

FRiCKLE's module is great, but it would be scary to put into production-
have fun with that test/release cycle :p

The overhead of putting Nginx in front of Varnish is fairly small in the
grand scheme of things. What's your motivation to strictly use Nginx?

There is official support for cache purging with the commercial version of
Nginx: https://www.nginx.com/products/nginx/caching/.

I've seen moderate hardware running Nginx (for TLS offload + WAF) ->
Varnish (cache + purge) -> Apache/mod_php do 50k r/s on a single node. One
would hope this suffices; it's a stable and proven stack. Again,
ngx_cache_purge is great, but any unsupported module in a prod environment
is scary when you're not writing the code. ;)
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Hi

> My $0.02 coming from experience building out scalable WP clusters is,
> stick to Varnish here.

Miscommunication on my part -- my aforementioned Varnish-in-front
referred to site dev in general.

To date, it's been in front of Symfony sites. Works like a champ there.

Since you're apparently working with WP under real-world loads, do you
perchance have a production-ready, V6-compatible VCL & nginx config you
can share? or point to?

> FRiCKLE's module is great, but it would be scary to put into production-
> have fun with that test/release cycle :p

Yep. Hence my question(s)!

> The overhead of putting Nginx in front of Varnish is fairly small in the
> grand scheme of things. What's your motivation to strictly use Nginx?

This time 'round, it's not entirely 'my' motivation; came with the job's
"prefer to haves".

Based, in apparently large part, on the usual use of TheGoogle; these 2
in particular:


https://deliciousbrains.com/page-caching-varnish-vs-nginx-fastcgi-cache-2018/
https://www.scalescale.com/tips/nginx/nginx-vs-varnish/

> There is official support for cache purging with the commercial version
> of Nginx: https://www.nginx.com/products/nginx/caching/.

Ah, so not (yet) in the FOSS product. I see it's proxy_cache, not
fastcgi_cache, based ...


> I've seen moderate hardware running Nginx (for TLS offload + WAF) ->
> Varnish (cache + purge) -> Apache/mod_php do 50k r/s on a single node.
> One would hope this suffices; it's a stable and proven stack. Again,
> ngx_cache_purge is great, but any unsupported module in a prod
> environment is scary when you're not writing the code. ;)

Again, yep.

Thx!
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Hi,

On Wed, Jun 6, 2018 at 3:42 PM, PGNet Dev <[email protected]> wrote:

> Hi
>
> My $0.02 coming from experience building out scalable WP clusters is,
>> stick to Varnish here.
>>
>
> Miscommunication on my part -- my aforementioned Varnish-in-front referred
> to site dev in general.
>
> To date, it's been in front of Symfony sites. Works like a champ there.
>
> Since you're apparently working with WP under real-world loads, do you
> perchance have a production-ready, V6-compatible VCL & nginx config you can
> share? or point to?
>


Nothing off the top of my head/isn't NDA-protected ;) But basic configs
will generally serve you well. Varnish and Nginx are mature, stable
projects; basic proxy_pass design with Nginx + basic Varnish config and a
PURGE method handler should suffice for most operations. Beyond that, tune
Nginx for buffer sizes and do a bit of kernel tweaking if necessary for
windowing, if you need.



> FRiCKLE's module is great, but it would be scary to put into production-
>> have fun with that test/release cycle :p
>
> Yep. Hence my question(s)!
>


Right- my point is, it's not officially supported, and Nginx has no stable
API/ABI. With every release you want to leverage you need to walk through
your entire test/canary/B-G/whatever cycle. That's a question only you can
answer, but asking about "what about X release" is fruitless because of a
complete lack of ABI support. In six month's it's an obsolete question,
whose only two answers are "be the developer and watching the changelog" or
"compile the module, test it, and pray to the diety of your choice that it
doesn't explode".


>
> The overhead of putting Nginx in front of Varnish is fairly small in the
>> grand scheme of things. What's your motivation to strictly use Nginx?
>>
>
> This time 'round, it's not entirely 'my' motivation; came with the job's
> "prefer to haves".
>
> Based, in apparently large part, on the usual use of TheGoogle; these 2 in
> particular:
>
>
> https://deliciousbrains.com/page-caching-varnish-vs-nginx-fa
> stcgi-cache-2018/
> https://www.scalescale.com/tips/nginx/nginx-vs-varnish/



Stepping back, these articles compare Nginx vs. Varnish straight-up. There
is considerable difference to take into account in examining a stack
leverage both.

And of course, always always always take into strong account the context
and limitations in which these articles were written. They do not care
about your particular business limitations, context, financial/resource
restrictions, or anything else that makes your situation useful. A large
grain of salt is always important to hold here.

In particular, the first article doesn't leverage keepalive (I maintain
"ab" is a horrid tool in this day and age), uses a cloud service with the
client living in who-knows-what-geographic/network-topology, and quite
frankly was written by an author who does not focus on systems/operations.
Tread wisely.

The second article is two and a half years old, offers no data whatsoever,
and touches on a number of irrelevant topics (SSL, h2). I'd steer clear of
any opinion offered here.

If I were you I would strongly question this "prefer to have" if the only
question is manageable cache purging. :)



> There is official support for cache purging with the commercial version of
>> Nginx: https://www.nginx.com/products/nginx/caching/.
>>
>
> Ah, so not (yet) in the FOSS product. I see it's proxy_cache, not
> fastcgi_cache, based ...
>


I imagine that's a question for the sales folks, outside of this list :D
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
On 6/6/18 4:09 PM, Robert Paprocki wrote:
> Nginx has no stable API/ABI. With every release you want to leverage you need to walk
> through your entire test/canary/B-G/whatever cycle. That's a question
> only you can answer, but asking about "what about X release" is
> fruitless because of a complete lack of ABI support. In six month's it's
> an obsolete question, whose only two answers are "be the developer and
> watching the changelog" or "compile the module, test it, and pray to the
> diety of your choice that it doesn't explode".

That's an excellent point. Esp since I tend to keep production current
with Nginx releases.

TBH, tho, I've said such a prayer-or-three re: Varnish!

> Stepping back, these articles compare Nginx vs. Varnish straight-up.
> There is considerable difference to take into account in examining a
> stack leverage both.
> ...

Much agreed. Apparently my reference to 'TheGoogle' refs wasn't snarky
or dismissive enough! ;-)

> If I were you I would strongly question this "prefer to have" if the
> only question is manageable cache purging. :)

Been done. Not convincingly enough, apparently.
You can lead a horse ...
It's a Nordstrom's(-of-long-ago) moment: "Customer's Right. Because they
say so."

Thx agn!


_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Hi,

> On Jun 6, 2018, at 16:18, PGNet Dev <[email protected]> wrote:
>
>> On 6/6/18 4:09 PM, Robert Paprocki wrote:
>> Nginx has no stable API/ABI. With every release you want to leverage you need to walk through your entire test/canary/B-G/whatever cycle. That's a question only you can answer, but asking about "what about X release" is fruitless because of a complete lack of ABI support. In six month's it's an obsolete question, whose only two answers are "be the developer and watching the changelog" or "compile the module, test it, and pray to the diety of your choice that it doesn't explode".
>
> That's an excellent point. Esp since I tend to keep production current with Nginx releases.
>
> TBH, tho, I've said such a prayer-or-three re: Varnish!

Certainly ;) I'm unfamiliar with Varnish's lifecycle. Just pointing out what should be noted (frankly, with the last few years of releases, unless there's a specific feature or bug you need to overcome, upgrading nginx to "latest" doesn't offer much value. I would love to be proved wrong here though ;) ).


>
>> Stepping back, these articles compare Nginx vs. Varnish straight-up. There is considerable difference to take into account in examining a stack leverage both.
> > ...
>
> Much agreed. Apparently my reference to 'TheGoogle' refs wasn't snarky or dismissive enough! ;-)
>
>> If I were you I would strongly question this "prefer to have" if the only question is manageable cache purging. :)
>
> Been done. Not convincingly enough, apparently.
> You can lead a horse ...
> It's a Nordstrom's(-of-long-ago) moment: "Customer's Right. Because they say so."
>
> Thx agn!

I got you :) good luck with it! You have our sympathies ;)
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
On Thu, Jun 7, 2018 at 6:05 AM, PGNet Dev <[email protected]> wrote:
> For some new WordPress sites, I'll be deploying fastcgi_cache as reverse proxy / page cache, instead of usual Varnish.
>
> Although there are a number of WP-module-based PURGE options, I prefer that it's handled by the web server.
>
> A commonly referenced approach is to use the 'FRiCKLE/ngx_cache_purge',
>
> https://github.com/FRiCKLE/ngx_cache_purge/
>
> with associated nginx conf additions,
>
> https://easyengine.io/wordpress-nginx/tutorials/single-site/fastcgi-cache-with-purging/
> https://www.ryadel.com/en/nginx-purge-proxy-cache-delete-invalidate-linux-centos-7/
>
> ngx_cache_purge module development appears to have gone stale; no commits since ~ 2014.
>
> What are your experiences with current use of that module, with latest 1.15x nginx releases?
>
> Is there a cleaner, nginx-native approach? Or other nginx purge module that's better maintained?
>
> Comments &/or pointers to any docs, etc would be helpful.
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx


You can try this:
https://github.com/nginx-modules/ngx_cache_purge
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
On 6/6/18 11:31 PM, Jon Franklin wrote:
> You can try this:
> https://github.com/nginx-modules/ngx_cache_purge

Thx! I'd aptly managed to not find/notice that fork.

Does address the 'stale' development status. Still, leaves some of the
concerns about nginx ABI, etc. mentioned earlier.
I'll set up a test instance and take it all for a spin.

OTOH, I've setup a Varnish instance in front of WP. As predicted, it's
straightforward.

And, the test WP site 'feels' a *lot* more responsive than using the
FastCGI cache alternative.

I've no quantitative benchmarks ... yet ... and I've not yet run all the
'Canary' tests I need to by any stretch. But it certainly looks promising.


_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
> For some new WordPress sites, I'll be deploying fastcgi_cache as reverse
> proxy / page cache, instead of usual Varnish.
>
> A commonly referenced approach is to use the 'FRiCKLE/ngx_cache_purge',
>
> https://github.com/FRiCKLE/ngx_cache_purge/
>
> ngx_cache_purge module development appears to have gone stale; no commits
> since ~ 2014.


Works just fine just for the current nginx versions you need to apply this
patch
https://github.com/FRiCKLE/ngx_cache_purge/commit/c7345057ad5429617fc0823e92e3fa8043840cef.diff
..
(or maybe the forked repo has allready this implemented).


There are some situations where nginx is "better" suited than Varnish.

In my case at one project we decided/had to switch to nginx caching from
varnish because varnish (even you are using disk based (mmap/file) backend
storage) has a memory overhead per cacheable object (like ~1Kb)

While 1Kb doesn't sound much when you start to have milions of objects it
adds up and in this case even we had several terabytes of fast SSDs the
actual bottleneck ended was there was not enough ram - the instances had
only limited 32 Gb so in general there couldnt be more than 33 milion cached
objects. Nginx on the other on the same hardware deals with 800+ milion (and
increasing) objects without a problem.


p.s. there is also obviously the ssl thing with varnish vs nginx .. but
thats another topic.

rr

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
On 6/7/18 9:27 AM, Reinis Rozitis wrote:
> this patch
> https://github.com/FRiCKLE/ngx_cache_purge/commit/c7345057ad5429617fc0823e92e3fa8043840cef.diff

Noted, thx.

> In my case at one project we decided/had to switch to nginx caching from
> varnish because varnish (even you are using disk based (mmap/file)
> backend storage) has a memory overhead per cacheable object (like ~1Kb)
>
> While 1Kb doesn't sound much when you start to have milions of objects
> it adds up and in this case even we had several terabytes of fast SSDs
> the actual bottleneck ended was there was not enough ramĀ  - the
> instances had only limited 32 Gb so in general there couldnt be more
> than 33 milion cached objects. Nginx on the other on the same hardware
> deals with 800+ milion (and increasing) objects without a problem.

Point taken. Not an issue for my typical use case; may come up in
future, so good to remember.

> p.s. there is also obviously the ssl thing with varnish vs nginx .. but
> thats another topic.

No real "vs" or "thing" IME. nginx(ssl terminator) -> varnish -> nginx
works quite nicely.

There's also Varnish's terminator, Hitch, as an alternative,

https://www.varnish-software.com/plus/ssl-tls-support/
https://github.com/varnish/hitch

which I've been told works well; I haven't bothered since I've already
got nginx in place on the backend -- adding a listener on the frontend
is trivial.
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
On Wednesday 06 June 2018 15:42:25 PGNet Dev wrote:
[..]
> > There is official support for cache purging with the commercial version
> > of Nginx: https://www.nginx.com/products/nginx/caching/.
>
> Ah, so not (yet) in the FOSS product. I see it's proxy_cache, not
> fastcgi_cache, based ...
>

Like almost all official modules, it's independent from the protocol used.

http://nginx.org/r/proxy_cache_purge
http://nginx.org/r/fastcgi_cache_purge
http://nginx.org/r/uwsgi_cache_purge
http://nginx.org/r/scgi_cache_purge

wbr, Valentin V. Bartenev

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
> No real "vs" or "thing" IME. nginx(ssl terminator) -> varnish -> nginx
> works quite nicely.
>
> There's also Varnish's terminator, Hitch, as an alternative,

Sure in general there is no problem offloading varnish (done it with nginx /
stud / haproxy / hitch / h2o .. etc and still running several setups).

But again depends on your needs and willingness to deal with larger software
stack (that's why I said it's another topic) as you end up with 2+ moving
parts (which have their own configuration / own resources / network buffers
/ sockets / timeouts etc) but obviously there are things which one does
better than other (and vice versa).

I just added it because you initially asked to comment on "nginx-native"
approach (if we can consider a third-party (in non-commercial version)
module as native) ;)


p.s. for some time varnish has http2 support .. maybe at some point in
future either openssl gets cleaned-up/rewritten enough for them to link with
it or they find some good-enough alternative :)

rr

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
I ran both Varnish (for caching), Nginx (ssl offloading) for quite some
time in production, but then switched to Nginx only. The main reasons being:

- The sheer amount of added context switches (proxying was done local on a
cPanel box, seeing 20-30k reqs/sec during peak hours)
- Issues with managing hacks/changes for spoofing the HTTPS env in Apache,
while maintaining the option of regular updates (CloudLinux ended up adding
this patch for me in it's builds
https://alex-at.net/blog/apache-mod_remoteip-mod_rpaf =>
https://www.cloudlinux.com/cloudlinux-os-blog/entry/beta-easyapache-4-updated-1-31
to make things easier, but it was already too late as I had already jumped
to Nginx)
- Having to manage two software versions, configs, auto config builders
used by internal tools, etc
- More added headaches with central logging
- No projected TLS support in Varnish
- Bare minimum H2 support in Varnish vs a more mature implementation in
Nginx

Since Nginx can pretty much do everything Varnish does, and more, I decided
to avoid the headaches and just jump over to Nginx (even though I've been
an avid Varnish fan since 2.1.5). As for a VCL replacement and purging in
Nginx, I suggest reading up on Lua and checking out openresty if you want
streamlined updates and don't want to manually compile/manage modules. To
avoid overloading the filesystem with added I/O from purge
requests/scans/etc, I wrote a simple Perl script that handles all the PURGE
requests in order to have regex support and control over the remoals (it
basically validates ownership to purge on the related domain, queues
removals, then has another thread for the cleanup).

Hope this helps some :)


On Thu, Jun 7, 2018 at 9:12 PM, Reinis Rozitis <[email protected]> wrote:

> No real "vs" or "thing" IME. nginx(ssl terminator) -> varnish -> nginx
>> works quite nicely.
>>
>> There's also Varnish's terminator, Hitch, as an alternative,
>>
>
> Sure in general there is no problem offloading varnish (done it with nginx
> / stud / haproxy / hitch / h2o .. etc and still running several setups).
>
> But again depends on your needs and willingness to deal with larger
> software stack (that's why I said it's another topic) as you end up with 2+
> moving parts (which have their own configuration / own resources / network
> buffers / sockets / timeouts etc) but obviously there are things which one
> does better than other (and vice versa).
>
> I just added it because you initially asked to comment on "nginx-native"
> approach (if we can consider a third-party (in non-commercial version)
> module as native) ;)
>
>
> p.s. for some time varnish has http2 support .. maybe at some point in
> future either openssl gets cleaned-up/rewritten enough for them to link
> with it or they find some good-enough alternative :)
>
> rr
>
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx
>
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Hi

On 6/12/18 12:03 AM, Andrei wrote:
> - The sheer amount of added context switches (proxying was done local on
> a cPanel box, seeing 20-30k reqs/sec during peak hours)

Not clear what you mean here

> - Having to manage two software versions, configs, auto config builders
> used by internal tools, etc

Not a huge headache here. I can see this gets possibly annoying a scale
with # of sites.

> - More added headaches with central logging

Having Varnish's detailed logging is a bit plus, IME, for tracking down
cache issues, specifically, and header issues in general.

No issues with 'central' logging.

> - No projected TLS support in Varnish

Having a terminator out front hasn't been a problem, save for the
additional config considerations.

> - Bare minimum H2 support in Varnish vs a more mature implementation in
> Nginx

This one I'm somewhat aware of -- haven't yet convinced myself of
if/where there's a really problematic bottleneck.

> Since Nginx can pretty much do everything Varnish does, and more,

Except for the richness of the VCL ...

> I decided to avoid the headaches and just jump over to Nginx (even though
> I've been an avid Varnish fan since 2.1.5). As for a VCL replacement and
> purging in Nginx, I suggest reading up on Lua and checking out openresty
> if you want streamlined updates and don't want to manually
> compile/manage modules. To avoid overloading the filesystem with added
> I/O from purge requests/scans/etc, I wrote a simple Perl script that
> handles all the PURGE requests in order to have regex support and
> control over the remoals (it basically validates ownership to purge on
> the related domain, queues removals, then has another thread for the
> cleanup).

My main problem so far is that WordPress appears to be generally
Varnish-UNfriendly.

Not core, but plugins. With Varnish, I'm having all SORTS of
issues/artifacts cropping up. So far, (my) VCL pass exceptions haven't
been sufficient.

Without Varnish, there are far fewer 'surprises'.

Then again, I'm not a huge WP fan to begin with; it's a pain to debug
anything beyond standard server config issues. Caching in particular.

OTOH, my sites with Nginx+Varnish+Varnish with Symfony work without a hitch.

My leaning is, for WP, Nginx only. For SF, Nginx+Varnish. And, TBH,
avoiding WP if/when I can.

> Hope this helps some :)

It does, thx!
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Sorry, only registered users may post in this forum.

Click here to login