Welcome! Log In Create A New Profile

Advanced

Nginx error log parser

Posted by mohit Agrawal 
mohit Agrawal
Nginx error log parser
January 10, 2018 09:50AM
Hi ,

I am looking to parse nginx error log so as to find out which particular IP
is throttled during specific amount of time on connection throttling /
request throttling. The format looks like :

2018/01/10 06:26:31 [error] 13485#13485: *64285471 limiting
connections by zone "rl_conn", client: xx.xx.xx.xx, server:
www.xyz.com, request: "GET /api/xyz HTTP/1.1", host: "www.xyz.com"

And the sample that I am looking for is :

{client: "xx.xx.xx.xx", server: "www.xyz.com", host: "www.xyz.com",
"request": "GET /api/xyz HTTP/1.1", reason: "limiting connections by
zone "rl_conn""}

so that I can pass it through ELK stack and find out the root ip which is
causing issue.


--
Mohit Agrawal
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Aziz Rozyev
Re: Nginx error log parser
January 10, 2018 10:40AM
is the 'log_format json’ is what you’re asking for?

http://nginx.org/en/docs/http/ngx_http_log_module.html#log_format

br,
Aziz.





> On 10 Jan 2018, at 11:45, mohit Agrawal <[email protected]> wrote:
>
> Hi ,
>
> I am looking to parse nginx error log so as to find out which particular IP is throttled during specific amount of time on connection throttling / request throttling. The format looks like :
>
> 2018/01/10 06:26:31 [error] 13485#13485: *64285471 limiting connections by zone "rl_conn", client: xx.xx.xx.xx, server: www.xyz.com, request: "GET /api/xyz HTTP/1.1", host: "www.xyz.com"
> And the sample that I am looking for is :
>
> {client: "xx.xx.xx.xx", server: "www.xyz.com", host: "www.xyz.com", "request": "GET /api/xyz HTTP/1.1", reason: "limiting connections by zone "rl_conn""}
> so that I can pass it through ELK stack and find out the root ip which is causing issue.
>
>
> --
> Mohit Agrawal
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Aziz Rozyev
Re: Nginx error log parser
January 10, 2018 11:00AM
btw, after re-reading the your questing, it looks like you need something like logstash grok filter.

br,
Aziz.





> On 10 Jan 2018, at 11:45, mohit Agrawal <[email protected]> wrote:
>
> Hi ,
>
> I am looking to parse nginx error log so as to find out which particular IP is throttled during specific amount of time on connection throttling / request throttling. The format looks like :
>
> 2018/01/10 06:26:31 [error] 13485#13485: *64285471 limiting connections by zone "rl_conn", client: xx.xx.xx.xx, server: www.xyz.com, request: "GET /api/xyz HTTP/1.1", host: "www.xyz.com"
> And the sample that I am looking for is :
>
> {client: "xx.xx.xx.xx", server: "www.xyz.com", host: "www.xyz.com", "request": "GET /api/xyz HTTP/1.1", reason: "limiting connections by zone "rl_conn""}
> so that I can pass it through ELK stack and find out the root ip which is causing issue.
>
>
> --
> Mohit Agrawal
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
mohit Agrawal
Re: Nginx error log parser
January 10, 2018 12:40PM
Hi Aziz,

log_format directive only provides formatting for access log, I am looking
to format error.log which doesn't take log_format directive.
Above example that I gave is just for nginx error logs.

Thanks

On 10 January 2018 at 15:26, Aziz Rozyev <[email protected]> wrote:

> btw, after re-reading the your questing, it looks like you need something
> like logstash grok filter.
>
> br,
> Aziz.
>
>
>
>
>
> > On 10 Jan 2018, at 11:45, mohit Agrawal <[email protected]> wrote:
> >
> > Hi ,
> >
> > I am looking to parse nginx error log so as to find out which particular
> IP is throttled during specific amount of time on connection throttling /
> request throttling. The format looks like :
> >
> > 2018/01/10 06:26:31 [error] 13485#13485: *64285471 limiting connections
> by zone "rl_conn", client: xx.xx.xx.xx, server: www.xyz.com, request:
> "GET /api/xyz HTTP/1.1", host: "www.xyz.com"
> > And the sample that I am looking for is :
> >
> > {client: "xx.xx.xx.xx", server: "www.xyz.com", host: "www.xyz.com",
> "request": "GET /api/xyz HTTP/1.1", reason: "limiting connections by zone
> "rl_conn""}
> > so that I can pass it through ELK stack and find out the root ip which
> is causing issue.
> >
> >
> > --
> > Mohit Agrawal
> > _______________________________________________
> > nginx mailing list
> > nginx@nginx.org
> > http://mailman.nginx.org/mailman/listinfo/nginx
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx
>



--
Mohit Agrawal
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Aziz Rozyev
Re: Nginx error log parser
January 10, 2018 12:50PM
Hi Mohit,

check the second reply. I’m not sure that there is a conventional pretty printing
tools for nginx error log.


br,
Aziz.





> On 10 Jan 2018, at 14:37, mohit Agrawal <[email protected]> wrote:
>
> Hi Aziz,
>
> log_format directive only provides formatting for access log, I am looking to format error.log which doesn't take log_format directive.
> Above example that I gave is just for nginx error logs.
>
> Thanks
>
> On 10 January 2018 at 15:26, Aziz Rozyev <[email protected]> wrote:
> btw, after re-reading the your questing, it looks like you need something like logstash grok filter.
>
> br,
> Aziz.
>
>
>
>
>
> > On 10 Jan 2018, at 11:45, mohit Agrawal <[email protected]> wrote:
> >
> > Hi ,
> >
> > I am looking to parse nginx error log so as to find out which particular IP is throttled during specific amount of time on connection throttling / request throttling. The format looks like :
> >
> > 2018/01/10 06:26:31 [error] 13485#13485: *64285471 limiting connections by zone "rl_conn", client: xx.xx.xx.xx, server: www.xyz.com, request: "GET /api/xyz HTTP/1.1", host: "www.xyz.com"
> > And the sample that I am looking for is :
> >
> > {client: "xx.xx.xx.xx", server: "www.xyz.com", host: "www.xyz.com", "request": "GET /api/xyz HTTP/1.1", reason: "limiting connections by zone "rl_conn""}
> > so that I can pass it through ELK stack and find out the root ip which is causing issue.
> >
> >
> > --
> > Mohit Agrawal
> > _______________________________________________
> > nginx mailing list
> > nginx@nginx.org
> > http://mailman.nginx.org/mailman/listinfo/nginx
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx
>
>
>
> --
> Mohit Agrawal

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
mohit Agrawal
Re: Nginx error log parser
January 10, 2018 12:50PM
Yeah I have tried grok / regex pattern as well. But not extensive success
that I achieved. grok didn't work for me, I tried regex then it was able to
segregate time , pid, tid, log_level and message. I also need message break
up for above pattern

On 10 January 2018 at 17:12, Aziz Rozyev <[email protected]> wrote:

> Hi Mohit,
>
> check the second reply. I’m not sure that there is a conventional pretty
> printing
> tools for nginx error log.
>
>
> br,
> Aziz.
>
>
>
>
>
> > On 10 Jan 2018, at 14:37, mohit Agrawal <[email protected]> wrote:
> >
> > Hi Aziz,
> >
> > log_format directive only provides formatting for access log, I am
> looking to format error.log which doesn't take log_format directive.
> > Above example that I gave is just for nginx error logs.
> >
> > Thanks
> >
> > On 10 January 2018 at 15:26, Aziz Rozyev <[email protected]> wrote:
> > btw, after re-reading the your questing, it looks like you need
> something like logstash grok filter.
> >
> > br,
> > Aziz.
> >
> >
> >
> >
> >
> > > On 10 Jan 2018, at 11:45, mohit Agrawal <[email protected]>
> wrote:
> > >
> > > Hi ,
> > >
> > > I am looking to parse nginx error log so as to find out which
> particular IP is throttled during specific amount of time on connection
> throttling / request throttling. The format looks like :
> > >
> > > 2018/01/10 06:26:31 [error] 13485#13485: *64285471 limiting
> connections by zone "rl_conn", client: xx.xx.xx.xx, server: www.xyz.com,
> request: "GET /api/xyz HTTP/1.1", host: "www.xyz.com"
> > > And the sample that I am looking for is :
> > >
> > > {client: "xx.xx.xx.xx", server: "www.xyz.com", host: "www.xyz.com",
> "request": "GET /api/xyz HTTP/1.1", reason: "limiting connections by zone
> "rl_conn""}
> > > so that I can pass it through ELK stack and find out the root ip which
> is causing issue.
> > >
> > >
> > > --
> > > Mohit Agrawal
> > > _______________________________________________
> > > nginx mailing list
> > > nginx@nginx.org
> > > http://mailman.nginx.org/mailman/listinfo/nginx
> >
> > _______________________________________________
> > nginx mailing list
> > nginx@nginx.org
> > http://mailman.nginx.org/mailman/listinfo/nginx
> >
> >
> >
> > --
> > Mohit Agrawal
>
>


--
Mohit Agrawal
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Aziz Rozyev
Re: Nginx error log parser
January 10, 2018 01:20PM
If you need parse exactly the same format, as you’ve shown in you question, it’s fairly easy to create something e.g. perl/awk/sed script.

for instance:

################# tst.awk #################
BEGIN {FS = "," }
{
split($1, m, "\ ")
printf "%s", "{ "
printf "%s",$2
printf "%s",$3
printf "%s",$5
printf "%s",$4
printf "reason: %s %s %s %s \"%s\"\n", m[6], m[7], m[8], m[9], m[10]
print " }”

}
#############################################


result:

echo 2018/01/10 06:26:31 [error] 13485#13485: *64285471 limiting connections by zone "rl_conn", client: xx.xx.xx.xx, server: www.xyz.com, request: "GET /api/xyz HTTP/1.1", host: "www.xyz.com" | awk -f /tmp/test.awk
{ client: xx.xx.xx.xx server: www.xyz.com host: www.xyz.com request: GET /api/xyz HTTP/1.1reason: limiting connections by zone "rl_conn"
}


br,
Aziz.





> On 10 Jan 2018, at 14:45, mohit Agrawal <[email protected]> wrote:
>
> Yeah I have tried grok / regex pattern as well. But not extensive success that I achieved. grok didn't work for me, I tried regex then it was able to segregate time , pid, tid, log_level and message. I also need message break up for above pattern
>
> On 10 January 2018 at 17:12, Aziz Rozyev <[email protected]> wrote:
> Hi Mohit,
>
> check the second reply. I’m not sure that there is a conventional pretty printing
> tools for nginx error log.
>
>
> br,
> Aziz.
>
>
>
>
>
> > On 10 Jan 2018, at 14:37, mohit Agrawal <[email protected]> wrote:
> >
> > Hi Aziz,
> >
> > log_format directive only provides formatting for access log, I am looking to format error.log which doesn't take log_format directive.
> > Above example that I gave is just for nginx error logs.
> >
> > Thanks
> >
> > On 10 January 2018 at 15:26, Aziz Rozyev <[email protected]> wrote:
> > btw, after re-reading the your questing, it looks like you need something like logstash grok filter.
> >
> > br,
> > Aziz.
> >
> >
> >
> >
> >
> > > On 10 Jan 2018, at 11:45, mohit Agrawal <[email protected]> wrote:
> > >
> > > Hi ,
> > >
> > > I am looking to parse nginx error log so as to find out which particular IP is throttled during specific amount of time on connection throttling / request throttling. The format looks like :
> > >
> > > 2018/01/10 06:26:31 [error] 13485#13485: *64285471 limiting connections by zone "rl_conn", client: xx.xx.xx.xx, server: www.xyz.com, request: "GET /api/xyz HTTP/1.1", host: "www.xyz.com"
> > > And the sample that I am looking for is :
> > >
> > > {client: "xx.xx.xx.xx", server: "www.xyz.com", host: "www.xyz.com", "request": "GET /api/xyz HTTP/1.1", reason: "limiting connections by zone "rl_conn""}
> > > so that I can pass it through ELK stack and find out the root ip which is causing issue.
> > >
> > >
> > > --
> > > Mohit Agrawal
> > > _______________________________________________
> > > nginx mailing list
> > > nginx@nginx.org
> > > http://mailman.nginx.org/mailman/listinfo/nginx
> >
> > _______________________________________________
> > nginx mailing list
> > nginx@nginx.org
> > http://mailman.nginx.org/mailman/listinfo/nginx
> >
> >
> >
> > --
> > Mohit Agrawal
>
>
>
>
> --
> Mohit Agrawal

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
mohit Agrawal
Re: Nginx error log parser
January 10, 2018 01:30PM
Thanks Aziz for this, I get your point, but can we do awking in fluentd
cons file ? Basically we are looking for realtime awking a nginx error log
file, how heavy this would be according to you.

On 10 January 2018 at 17:44, Aziz Rozyev <[email protected]> wrote:

> If you need parse exactly the same format, as you’ve shown in you
> question, it’s fairly easy to create something e.g. perl/awk/sed script.
>
> for instance:
>
> ################# tst.awk #################
> BEGIN {FS = "," }
> {
> split($1, m, "\ ")
> printf "%s", "{ "
> printf "%s",$2
> printf "%s",$3
> printf "%s",$5
> printf "%s",$4
> printf "reason: %s %s %s %s \"%s\"\n", m[6], m[7], m[8], m[9], m[10]
> print " }”
>
> }
> #############################################
>
>
> result:
>
> echo 2018/01/10 06:26:31 [error] 13485#13485: *64285471 limiting
> connections by zone "rl_conn", client: xx.xx.xx.xx, server: www.xyz.com,
> request: "GET /api/xyz HTTP/1.1", host: "www.xyz.com" | awk -f
> /tmp/test.awk
> { client: xx.xx.xx.xx server: www.xyz.com host: www.xyz.com request: GET
> /api/xyz HTTP/1.1reason: limiting connections by zone "rl_conn"
> }
>
>
> br,
> Aziz.
>
>
>
>
>
> > On 10 Jan 2018, at 14:45, mohit Agrawal <[email protected]> wrote:
> >
> > Yeah I have tried grok / regex pattern as well. But not extensive
> success that I achieved. grok didn't work for me, I tried regex then it was
> able to segregate time , pid, tid, log_level and message. I also need
> message break up for above pattern
> >
> > On 10 January 2018 at 17:12, Aziz Rozyev <[email protected]> wrote:
> > Hi Mohit,
> >
> > check the second reply. I’m not sure that there is a conventional pretty
> printing
> > tools for nginx error log.
> >
> >
> > br,
> > Aziz.
> >
> >
> >
> >
> >
> > > On 10 Jan 2018, at 14:37, mohit Agrawal <[email protected]>
> wrote:
> > >
> > > Hi Aziz,
> > >
> > > log_format directive only provides formatting for access log, I am
> looking to format error.log which doesn't take log_format directive.
> > > Above example that I gave is just for nginx error logs.
> > >
> > > Thanks
> > >
> > > On 10 January 2018 at 15:26, Aziz Rozyev <[email protected]> wrote:
> > > btw, after re-reading the your questing, it looks like you need
> something like logstash grok filter.
> > >
> > > br,
> > > Aziz.
> > >
> > >
> > >
> > >
> > >
> > > > On 10 Jan 2018, at 11:45, mohit Agrawal <[email protected]>
> wrote:
> > > >
> > > > Hi ,
> > > >
> > > > I am looking to parse nginx error log so as to find out which
> particular IP is throttled during specific amount of time on connection
> throttling / request throttling. The format looks like :
> > > >
> > > > 2018/01/10 06:26:31 [error] 13485#13485: *64285471 limiting
> connections by zone "rl_conn", client: xx.xx.xx.xx, server: www.xyz.com,
> request: "GET /api/xyz HTTP/1.1", host: "www.xyz.com"
> > > > And the sample that I am looking for is :
> > > >
> > > > {client: "xx.xx.xx.xx", server: "www.xyz.com", host: "www.xyz.com",
> "request": "GET /api/xyz HTTP/1.1", reason: "limiting connections by zone
> "rl_conn""}
> > > > so that I can pass it through ELK stack and find out the root ip
> which is causing issue.
> > > >
> > > >
> > > > --
> > > > Mohit Agrawal
> > > > _______________________________________________
> > > > nginx mailing list
> > > > nginx@nginx.org
> > > > http://mailman.nginx.org/mailman/listinfo/nginx
> > >
> > > _______________________________________________
> > > nginx mailing list
> > > nginx@nginx.org
> > > http://mailman.nginx.org/mailman/listinfo/nginx
> > >
> > >
> > >
> > > --
> > > Mohit Agrawal
> >
> >
> >
> >
> > --
> > Mohit Agrawal
>
>


--
Mohit Agrawal
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
itpp2012
Re: Nginx error log parser
January 10, 2018 01:30PM
Aziz Rozyev Wrote:
-------------------------------------------------------
> Hi Mohit,
>
> check the second reply. I’m not sure that there is a conventional
> pretty printing
> tools for nginx error log.

Look at awstats.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,278065,278080#msg-278080

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
mohit Agrawal
Re: Nginx error log parser
January 10, 2018 03:00PM
Hi All,


I have something like this. I tested the `tail -f /var/log/nginx/error.log
| awk -f /var/log/nginx/test.awk` part and it just works fine. But when i
try to run it through fluentd, it doesn't do anything. Any idea why ?


<source>

@type exec

format json

tag sample

command tail -f /var/log/nginx/error.log | awk -f
/var/log/nginx/test.awk

</source>


<match sample >

@type stdout

</match>


Also /var/log/nginx/test.awk, is as follow :


################# tst.awk #################

BEGIN {FS = "," }

{

split($1, m, "\ ")

gsub(/ /, "", $2)

split($2, a, ":")

gsub(/ /, "", $3)

split($3, b, ":")

gsub(/ /, "", $4)

split($4, c, ":")

gsub(/ /, "", $5)

split($5, d, ":")

printf "%s", "{"

printf "\"%s\" : \"%s\",",a[1], a[2]

printf "\"%s\" : \"%s\",",b[1], b[2]

#printf "%s",$3 ","

#printf "%s",$5 ","

#printf "%s",$4 ","

printf "\"%s\" : %s,",c[1], c[2]

printf "\"%s\" : %s,",d[1], d[2]

split(m[10], e, "\"")

printf " \"reason\": \"%s %s %s %s %s\"}\n", m[6], m[7], m[8], m[9], e[2
]




}

#############################################




On 10 January 2018 at 17:53, mohit Agrawal <[email protected]> wrote:

> Thanks Aziz for this, I get your point, but can we do awking in fluentd
> cons file ? Basically we are looking for realtime awking a nginx error log
> file, how heavy this would be according to you.
>
> On 10 January 2018 at 17:44, Aziz Rozyev <[email protected]> wrote:
>
>> If you need parse exactly the same format, as you’ve shown in you
>> question, it’s fairly easy to create something e.g. perl/awk/sed script.
>>
>> for instance:
>>
>> ################# tst.awk #################
>> BEGIN {FS = "," }
>> {
>> split($1, m, "\ ")
>> printf "%s", "{ "
>> printf "%s",$2
>> printf "%s",$3
>> printf "%s",$5
>> printf "%s",$4
>> printf "reason: %s %s %s %s \"%s\"\n", m[6], m[7], m[8], m[9], m[10]
>> print " }”
>>
>> }
>> #############################################
>>
>>
>> result:
>>
>> echo 2018/01/10 06:26:31 [error] 13485#13485: *64285471 limiting
>> connections by zone "rl_conn", client: xx.xx.xx.xx, server: www.xyz.com,
>> request: "GET /api/xyz HTTP/1.1", host: "www.xyz.com" | awk -f
>> /tmp/test.awk
>> { client: xx.xx.xx.xx server: www.xyz.com host: www.xyz.com request:
>> GET /api/xyz HTTP/1.1reason: limiting connections by zone "rl_conn"
>> }
>>
>>
>> br,
>> Aziz.
>>
>>
>>
>>
>>
>> > On 10 Jan 2018, at 14:45, mohit Agrawal <[email protected]> wrote:
>> >
>> > Yeah I have tried grok / regex pattern as well. But not extensive
>> success that I achieved. grok didn't work for me, I tried regex then it was
>> able to segregate time , pid, tid, log_level and message. I also need
>> message break up for above pattern
>> >
>> > On 10 January 2018 at 17:12, Aziz Rozyev <[email protected]> wrote:
>> > Hi Mohit,
>> >
>> > check the second reply. I’m not sure that there is a conventional
>> pretty printing
>> > tools for nginx error log.
>> >
>> >
>> > br,
>> > Aziz.
>> >
>> >
>> >
>> >
>> >
>> > > On 10 Jan 2018, at 14:37, mohit Agrawal <[email protected]>
>> wrote:
>> > >
>> > > Hi Aziz,
>> > >
>> > > log_format directive only provides formatting for access log, I am
>> looking to format error.log which doesn't take log_format directive.
>> > > Above example that I gave is just for nginx error logs.
>> > >
>> > > Thanks
>> > >
>> > > On 10 January 2018 at 15:26, Aziz Rozyev <[email protected]> wrote:
>> > > btw, after re-reading the your questing, it looks like you need
>> something like logstash grok filter.
>> > >
>> > > br,
>> > > Aziz.
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > > On 10 Jan 2018, at 11:45, mohit Agrawal <[email protected]>
>> wrote:
>> > > >
>> > > > Hi ,
>> > > >
>> > > > I am looking to parse nginx error log so as to find out which
>> particular IP is throttled during specific amount of time on connection
>> throttling / request throttling. The format looks like :
>> > > >
>> > > > 2018/01/10 06:26:31 [error] 13485#13485: *64285471 limiting
>> connections by zone "rl_conn", client: xx.xx.xx.xx, server: www.xyz.com,
>> request: "GET /api/xyz HTTP/1.1", host: "www.xyz.com"
>> > > > And the sample that I am looking for is :
>> > > >
>> > > > {client: "xx.xx.xx.xx", server: "www.xyz.com", host: "www.xyz.com",
>> "request": "GET /api/xyz HTTP/1.1", reason: "limiting connections by zone
>> "rl_conn""}
>> > > > so that I can pass it through ELK stack and find out the root ip
>> which is causing issue.
>> > > >
>> > > >
>> > > > --
>> > > > Mohit Agrawal
>> > > > _______________________________________________
>> > > > nginx mailing list
>> > > > nginx@nginx.org
>> > > > http://mailman.nginx.org/mailman/listinfo/nginx
>> > >
>> > > _______________________________________________
>> > > nginx mailing list
>> > > nginx@nginx.org
>> > > http://mailman.nginx.org/mailman/listinfo/nginx
>> > >
>> > >
>> > >
>> > > --
>> > > Mohit Agrawal
>> >
>> >
>> >
>> >
>> > --
>> > Mohit Agrawal
>>
>>
>
>
> --
> Mohit Agrawal
>



--
Mohit Agrawal
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Aziz Rozyev
Re: Nginx error log parser
January 11, 2018 10:00AM
Hi,

seems, that fluentd has an nginx_parser plugin already, another solution that probably should work is to use the grep filters,
something as follows:

<fitler foo.bar>
@type grep
<regexp>
key client
patter ^client.*\ $
</regexp>
<regexp>
key server
pattern ^server.*\ $
</regexp>
<regexp>
key host
pattern ^host.*$
</regexp>
<regexp>
key zone
pattern ^zone.*\ $
</regexp>
…..
</filter>


then use record_trasformer type, to make further modifications. But, I didn’t tried above,
probably it’s something that better to be asked from fluentd community..


br,
Aziz.





> On 10 Jan 2018, at 15:23, mohit Agrawal <[email protected]> wrote:
>
> Thanks Aziz for this, I get your point, but can we do awking in fluentd cons file ? Basically we are looking for realtime awking a nginx error log file, how heavy this would be according to you.
>
> On 10 January 2018 at 17:44, Aziz Rozyev <[email protected]> wrote:
> If you need parse exactly the same format, as you’ve shown in you question, it’s fairly easy to create something e.g. perl/awk/sed script.
>
> for instance:
>
> ################# tst.awk #################
> BEGIN {FS = "," }
> {
> split($1, m, "\ ")
> printf "%s", "{ "
> printf "%s",$2
> printf "%s",$3
> printf "%s",$5
> printf "%s",$4
> printf "reason: %s %s %s %s \"%s\"\n", m[6], m[7], m[8], m[9], m[10]
> print " }”
>
> }
> #############################################
>
>
> result:
>
> echo 2018/01/10 06:26:31 [error] 13485#13485: *64285471 limiting connections by zone "rl_conn", client: xx.xx.xx.xx, server: www.xyz.com, request: "GET /api/xyz HTTP/1.1", host: "www.xyz.com" | awk -f /tmp/test.awk
> { client: xx.xx.xx.xx server: www.xyz.com host: www.xyz.com request: GET /api/xyz HTTP/1.1reason: limiting connections by zone "rl_conn"
> }
>
>
> br,
> Aziz.
>
>
>
>
>
> > On 10 Jan 2018, at 14:45, mohit Agrawal <[email protected]> wrote:
> >
> > Yeah I have tried grok / regex pattern as well. But not extensive success that I achieved. grok didn't work for me, I tried regex then it was able to segregate time , pid, tid, log_level and message. I also need message break up for above pattern
> >
> > On 10 January 2018 at 17:12, Aziz Rozyev <[email protected]> wrote:
> > Hi Mohit,
> >
> > check the second reply. I’m not sure that there is a conventional pretty printing
> > tools for nginx error log.
> >
> >
> > br,
> > Aziz.
> >
> >
> >
> >
> >
> > > On 10 Jan 2018, at 14:37, mohit Agrawal <[email protected]> wrote:
> > >
> > > Hi Aziz,
> > >
> > > log_format directive only provides formatting for access log, I am looking to format error.log which doesn't take log_format directive.
> > > Above example that I gave is just for nginx error logs.
> > >
> > > Thanks
> > >
> > > On 10 January 2018 at 15:26, Aziz Rozyev <[email protected]> wrote:
> > > btw, after re-reading the your questing, it looks like you need something like logstash grok filter.
> > >
> > > br,
> > > Aziz.
> > >
> > >
> > >
> > >
> > >
> > > > On 10 Jan 2018, at 11:45, mohit Agrawal <[email protected]> wrote:
> > > >
> > > > Hi ,
> > > >
> > > > I am looking to parse nginx error log so as to find out which particular IP is throttled during specific amount of time on connection throttling / request throttling. The format looks like :
> > > >
> > > > 2018/01/10 06:26:31 [error] 13485#13485: *64285471 limiting connections by zone "rl_conn", client: xx.xx.xx.xx, server: www.xyz.com, request: "GET /api/xyz HTTP/1.1", host: "www.xyz.com"
> > > > And the sample that I am looking for is :
> > > >
> > > > {client: "xx.xx.xx.xx", server: "www.xyz.com", host: "www.xyz.com", "request": "GET /api/xyz HTTP/1.1", reason: "limiting connections by zone "rl_conn""}
> > > > so that I can pass it through ELK stack and find out the root ip which is causing issue.
> > > >
> > > >
> > > > --
> > > > Mohit Agrawal
> > > > _______________________________________________
> > > > nginx mailing list
> > > > nginx@nginx.org
> > > > http://mailman.nginx.org/mailman/listinfo/nginx
> > >
> > > _______________________________________________
> > > nginx mailing list
> > > nginx@nginx.org
> > > http://mailman.nginx.org/mailman/listinfo/nginx
> > >
> > >
> > >
> > > --
> > > Mohit Agrawal
> >
> >
> >
> >
> > --
> > Mohit Agrawal
>
>
>
>
> --
> Mohit Agrawal

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
mohit Agrawal
Re: Nginx error log parser
January 11, 2018 12:50PM
I finally end up writing my own error log fluentd custom parser in ruby.
It's working now.

Thanks for help anyways, much appreciated


On 11 January 2018 at 14:26, Aziz Rozyev <[email protected]> wrote:

> Hi,
>
> seems, that fluentd has an nginx_parser plugin already, another solution
> that probably should work is to use the grep filters,
> something as follows:
>
> <fitler foo.bar>
> @type grep
> <regexp>
> key client
> patter ^client.*\ $
> </regexp>
> <regexp>
> key server
> pattern ^server.*\ $
> </regexp>
> <regexp>
> key host
> pattern ^host.*$
> </regexp>
> <regexp>
> key zone
> pattern ^zone.*\ $
> </regexp>
> …..
> </filter>
>
>
> then use record_trasformer type, to make further modifications. But, I
> didn’t tried above,
> probably it’s something that better to be asked from fluentd community..
>
>
> br,
> Aziz.
>
>
>
>
>
> > On 10 Jan 2018, at 15:23, mohit Agrawal <[email protected]> wrote:
> >
> > Thanks Aziz for this, I get your point, but can we do awking in fluentd
> cons file ? Basically we are looking for realtime awking a nginx error log
> file, how heavy this would be according to you.
> >
> > On 10 January 2018 at 17:44, Aziz Rozyev <[email protected]> wrote:
> > If you need parse exactly the same format, as you’ve shown in you
> question, it’s fairly easy to create something e.g. perl/awk/sed script.
> >
> > for instance:
> >
> > ################# tst.awk #################
> > BEGIN {FS = "," }
> > {
> > split($1, m, "\ ")
> > printf "%s", "{ "
> > printf "%s",$2
> > printf "%s",$3
> > printf "%s",$5
> > printf "%s",$4
> > printf "reason: %s %s %s %s \"%s\"\n", m[6], m[7], m[8], m[9], m[10]
> > print " }”
> >
> > }
> > #############################################
> >
> >
> > result:
> >
> > echo 2018/01/10 06:26:31 [error] 13485#13485: *64285471 limiting
> connections by zone "rl_conn", client: xx.xx.xx.xx, server: www.xyz.com,
> request: "GET /api/xyz HTTP/1.1", host: "www.xyz.com" | awk -f
> /tmp/test.awk
> > { client: xx.xx.xx.xx server: www.xyz.com host: www.xyz.com request:
> GET /api/xyz HTTP/1.1reason: limiting connections by zone "rl_conn"
> > }
> >
> >
> > br,
> > Aziz.
> >
> >
> >
> >
> >
> > > On 10 Jan 2018, at 14:45, mohit Agrawal <[email protected]>
> wrote:
> > >
> > > Yeah I have tried grok / regex pattern as well. But not extensive
> success that I achieved. grok didn't work for me, I tried regex then it was
> able to segregate time , pid, tid, log_level and message. I also need
> message break up for above pattern
> > >
> > > On 10 January 2018 at 17:12, Aziz Rozyev <[email protected]> wrote:
> > > Hi Mohit,
> > >
> > > check the second reply. I’m not sure that there is a conventional
> pretty printing
> > > tools for nginx error log.
> > >
> > >
> > > br,
> > > Aziz.
> > >
> > >
> > >
> > >
> > >
> > > > On 10 Jan 2018, at 14:37, mohit Agrawal <[email protected]>
> wrote:
> > > >
> > > > Hi Aziz,
> > > >
> > > > log_format directive only provides formatting for access log, I am
> looking to format error.log which doesn't take log_format directive.
> > > > Above example that I gave is just for nginx error logs.
> > > >
> > > > Thanks
> > > >
> > > > On 10 January 2018 at 15:26, Aziz Rozyev <[email protected]> wrote:
> > > > btw, after re-reading the your questing, it looks like you need
> something like logstash grok filter.
> > > >
> > > > br,
> > > > Aziz.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > > On 10 Jan 2018, at 11:45, mohit Agrawal <[email protected]>
> wrote:
> > > > >
> > > > > Hi ,
> > > > >
> > > > > I am looking to parse nginx error log so as to find out which
> particular IP is throttled during specific amount of time on connection
> throttling / request throttling. The format looks like :
> > > > >
> > > > > 2018/01/10 06:26:31 [error] 13485#13485: *64285471 limiting
> connections by zone "rl_conn", client: xx.xx.xx.xx, server: www.xyz.com,
> request: "GET /api/xyz HTTP/1.1", host: "www.xyz.com"
> > > > > And the sample that I am looking for is :
> > > > >
> > > > > {client: "xx.xx.xx.xx", server: "www.xyz.com", host: "www.xyz.com",
> "request": "GET /api/xyz HTTP/1.1", reason: "limiting connections by zone
> "rl_conn""}
> > > > > so that I can pass it through ELK stack and find out the root ip
> which is causing issue.
> > > > >
> > > > >
> > > > > --
> > > > > Mohit Agrawal
> > > > > _______________________________________________
> > > > > nginx mailing list
> > > > > nginx@nginx.org
> > > > > http://mailman.nginx.org/mailman/listinfo/nginx
> > > >
> > > > _______________________________________________
> > > > nginx mailing list
> > > > nginx@nginx.org
> > > > http://mailman.nginx.org/mailman/listinfo/nginx
> > > >
> > > >
> > > >
> > > > --
> > > > Mohit Agrawal
> > >
> > >
> > >
> > >
> > > --
> > > Mohit Agrawal
> >
> >
> >
> >
> > --
> > Mohit Agrawal
>
>


--
Mohit Agrawal
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Sorry, only registered users may post in this forum.

Click here to login