http_log_module filter by status

I was interested in having nginx log 404s to their own file.
Essentially, i _hate_ 404s, so I like to parse access logs find and
report all 404s. However, as-is, parsing large access logs can be quite
inefficient since 404s represent such a small % of the entire file. I
was thinking nginx could filter it out at write-time:

access_log not_found.log combined buffer=16K 404;


Apologies for the lameness of the code, but this is what I came up
with:
https://gist.github.com/2906701

I certainly don't recommend anyone uses it, I'm mostly just looking for
feedback. Is this better off in its own module? (there's so much code in
the http_log_module that I want to leverage though). There's much more
filtering that could go on that perhaps a new directive is a better
approach:

access_log_filter $status /(40\d)/
access_log_filter $method GET

(which is certainly beyond my capabilities).

Thoughts?

Posted at Nginx Forum: http://forum.nginx.org/read.php?2,227401,227401#msg-227401

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx

Hello!

On Sun, Jun 10, 2012 at 01:40:12PM -0400, karlseguin wrote:

> I was interested in having nginx log 404s to their own file.
> Essentially, i _hate_ 404s, so I like to parse access logs find and
> report all 404s. However, as-is, parsing large access logs can be quite
> inefficient since 404s represent such a small % of the entire file. I
> was thinking nginx could filter it out at write-time:
>
> access_log not_found.log combined buffer=16K 404;
>
>
> Apologies for the lameness of the code, but this is what I came up
> with:
> https://gist.github.com/2906701
>
> I certainly don't recommend anyone uses it, I'm mostly just looking for
> feedback. Is this better off in its own module? (there's so much code in
> the http_log_module that I want to leverage though). There's much more
> filtering that could go on that perhaps a new directive is a better
> approach:
>
> access_log_filter $status /(40\d)/
> access_log_filter $method GET
>
> (which is certainly beyond my capabilities).
>
> Thoughts?

error_page 404 /404.html;

location = /404.html {
access_log /path/to/log;
}

Maxim Dounin

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx

The documentation <http://wiki.nginx.org/HttpCoreModule#error_page>; also
says that if you don't want to redirect to another page, you can use a
named redirection :
location / {
error_page 404 @404;
}

location @404 {
access_log /path/to/log;
}

The wiki syntax seems to be wrong though, since it is using brackets and
not braces.
---
*B. R.*


On Tue, Jun 12, 2012 at 4:08 PM, Maxim Dounin <[email protected]> wrote:

> Hello!
>
> On Sun, Jun 10, 2012 at 01:40:12PM -0400, karlseguin wrote:
>
> > I was interested in having nginx log 404s to their own file.
> > Essentially, i _hate_ 404s, so I like to parse access logs find and
> > report all 404s. However, as-is, parsing large access logs can be quite
> > inefficient since 404s represent such a small % of the entire file. I
> > was thinking nginx could filter it out at write-time:
> >
> > access_log not_found.log combined buffer=16K 404;
> >
> >
> > Apologies for the lameness of the code, but this is what I came up
> > with:
> > https://gist.github.com/2906701
> >
> > I certainly don't recommend anyone uses it, I'm mostly just looking for
> > feedback. Is this better off in its own module? (there's so much code in
> > the http_log_module that I want to leverage though). There's much more
> > filtering that could go on that perhaps a new directive is a better
> > approach:
> >
> > access_log_filter $status /(40\d)/
> > access_log_filter $method GET
> >
> > (which is certainly beyond my capabilities).
> >
> > Thoughts?
>
> error_page 404 /404.html;
>
> location = /404.html {
> access_log /path/to/log;
> }
>
> Maxim Dounin
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx
>
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx

heh, thanks :) much better!

Posted at Nginx Forum: http://forum.nginx.org/read.php?2,227401,227474#msg-227474

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx

Hello!

On Tue, Jun 12, 2012 at 06:08:28PM -0400, B.R. wrote:

> The documentation <http://wiki.nginx.org/HttpCoreModule#error_page>; also
> says that if you don't want to redirect to another page, you can use a
> named redirection :
> location / {
> error_page 404 @404;
> }
>
> location @404 {
> access_log /path/to/log;
> }

This will cause another 404 as written.

Note well that named location are really needed only if you don't
want *internal* redirect to happen, i.e. want to preserve original
uri (e.g. for an additional processing as in various fallback
schemes) and/or really can't touch uri namespace for some reason.

Maxim Dounin

>
> The wiki syntax seems to be wrong though, since it is using brackets and
> not braces.
> ---
> *B. R.*
>
>
> On Tue, Jun 12, 2012 at 4:08 PM, Maxim Dounin <[email protected]> wrote:
>
> > Hello!
> >
> > On Sun, Jun 10, 2012 at 01:40:12PM -0400, karlseguin wrote:
> >
> > > I was interested in having nginx log 404s to their own file.
> > > Essentially, i _hate_ 404s, so I like to parse access logs find and
> > > report all 404s. However, as-is, parsing large access logs can be quite
> > > inefficient since 404s represent such a small % of the entire file. I
> > > was thinking nginx could filter it out at write-time:
> > >
> > > access_log not_found.log combined buffer=16K 404;
> > >
> > >
> > > Apologies for the lameness of the code, but this is what I came up
> > > with:
> > > https://gist.github.com/2906701
> > >
> > > I certainly don't recommend anyone uses it, I'm mostly just looking for
> > > feedback. Is this better off in its own module? (there's so much code in
> > > the http_log_module that I want to leverage though). There's much more
> > > filtering that could go on that perhaps a new directive is a better
> > > approach:
> > >
> > > access_log_filter $status /(40\d)/
> > > access_log_filter $method GET
> > >
> > > (which is certainly beyond my capabilities).
> > >
> > > Thoughts?
> >
> > error_page 404 /404.html;
> >
> > location = /404.html {
> > access_log /path/to/log;
> > }
> >
> > Maxim Dounin
> >
> > _______________________________________________
> > nginx mailing list
> > nginx@nginx.org
> > http://mailman.nginx.org/mailman/listinfo/nginx
> >

> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx