Welcome! Log In Create A New Profile

Advanced

nginx access logs to mysql

Posted by SARA QUISPE MEJIA 
SARA QUISPE MEJIA
nginx access logs to mysql
April 20, 2017 09:10AM
I want to parse the log file respect to a client that means to make a
report of how the client is using my application through the information
provided by a log file.

So I need to filter some url where I find an especific name of a function.
For that I thought in insert my log file to database like mysql. Could I do
that with my log file ? (I use nginx )

I tried to do that with syslog-ng but it doesn't work

Do you have any ideas?

Thanks

Sara
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
oscaretu .
Re: nginx access logs to mysql
April 20, 2017 09:20AM
Sara, why don't you process the log file just with grep / pcregrep to get
just the lines that containt that function name?


On Thu, Apr 20, 2017 at 9:05 AM, SARA QUISPE MEJIA <[email protected]>
wrote:

> I want to parse the log file respect to a client that means to make a
> report of how the client is using my application through the information
> provided by a log file.
>
> So I need to filter some url where I find an especific name of a
> function. For that I thought in insert my log file to database like mysql.
> Could I do that with my log file ? (I use nginx )
>
> I tried to do that with syslog-ng but it doesn't work
>
> Do you have any ideas?
>
> Thanks
>
> Sara
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx
>



--
Oscar Fernandez Sierra
oscaretu@gmail.com
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
SARA QUISPE MEJIA
Re: nginx access logs to mysql
April 20, 2017 09:40AM
yes, I tried to do that and then I did the analysis with goaccess,
but now I need to do an interface(page.php) for the user can choose
the function that want to filter and then generate a report.
So, I thought put all my information on a database like mysql

2017-04-20 9:11 GMT+02:00 oscaretu . <[email protected]>:

> Sara, why don't you process the log file just with grep / pcregrep to get
> just the lines that containt that function name?
>
>
> On Thu, Apr 20, 2017 at 9:05 AM, SARA QUISPE MEJIA <[email protected]>
> wrote:
>
>> I want to parse the log file respect to a client that means to make a
>> report of how the client is using my application through the information
>> provided by a log file.
>>
>> So I need to filter some url where I find an especific name of a
>> function. For that I thought in insert my log file to database like mysql.
>> Could I do that with my log file ? (I use nginx )
>>
>> I tried to do that with syslog-ng but it doesn't work
>>
>> Do you have any ideas?
>>
>> Thanks
>>
>> Sara
>>
>> _______________________________________________
>> nginx mailing list
>> nginx@nginx.org
>> http://mailman.nginx.org/mailman/listinfo/nginx
>>
>
>
>
> --
> Oscar Fernandez Sierra
> oscaretu@gmail.com
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx
>
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
oscaretu .
Re: nginx access logs to mysql
April 20, 2017 10:00AM
If you extract the lines containing all the function names, and filter them
with a script that counts the number of times that the function appears,
you only have to store (for example in a JSON) the number of times that
each function was used. And you can avoid storing all the logs in a
database.


On Thu, Apr 20, 2017 at 9:29 AM, SARA QUISPE MEJIA <[email protected]>
wrote:

> yes, I tried to do that and then I did the analysis with goaccess,
> but now I need to do an interface(page.php) for the user can choose
> the function that want to filter and then generate a report.
> So, I thought put all my information on a database like mysql
>
> 2017-04-20 9:11 GMT+02:00 oscaretu . <[email protected]>:
>
>> Sara, why don't you process the log file just with grep / pcregrep to get
>> just the lines that containt that function name?
>>
>>
>> On Thu, Apr 20, 2017 at 9:05 AM, SARA QUISPE MEJIA <[email protected]>
>> wrote:
>>
>>> I want to parse the log file respect to a client that means to make a
>>> report of how the client is using my application through the information
>>> provided by a log file.
>>>
>>> So I need to filter some url where I find an especific name of a
>>> function. For that I thought in insert my log file to database like mysql.
>>> Could I do that with my log file ? (I use nginx )
>>>
>>> I tried to do that with syslog-ng but it doesn't work
>>>
>>> Do you have any ideas?
>>>
>>> Thanks
>>>
>>> Sara
>>>
>>> _______________________________________________
>>> nginx mailing list
>>> nginx@nginx.org
>>> http://mailman.nginx.org/mailman/listinfo/nginx
>>>
>>
>>
>>
>> --
>> Oscar Fernandez Sierra
>> oscaretu@gmail.com
>>
>> _______________________________________________
>> nginx mailing list
>> nginx@nginx.org
>> http://mailman.nginx.org/mailman/listinfo/nginx
>>
>
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx
>



--
Oscar Fernandez Sierra
oscaretu@gmail.com
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
SARA QUISPE MEJIA
Re: nginx access logs to mysql
April 20, 2017 10:20AM
yes, I could do it but I want just store for example the customer, the
function name, the date after with those information I could filter my
report through a page.php
But with json I don't know how can I do that?


2017-04-20 9:57 GMT+02:00 oscaretu . <[email protected]>:

> If you extract the lines containing all the function names, and filter
> them with a script that counts the number of times that the function
> appears, you only have to store (for example in a JSON) the number of times
> that each function was used. And you can avoid storing all the logs in a
> database.
>
>
> On Thu, Apr 20, 2017 at 9:29 AM, SARA QUISPE MEJIA <[email protected]>
> wrote:
>
>> yes, I tried to do that and then I did the analysis with goaccess,
>> but now I need to do an interface(page.php) for the user can choose
>> the function that want to filter and then generate a report.
>> So, I thought put all my information on a database like mysql
>>
>> 2017-04-20 9:11 GMT+02:00 oscaretu . <[email protected]>:
>>
>>> Sara, why don't you process the log file just with grep / pcregrep to
>>> get just the lines that containt that function name?
>>>
>>>
>>> On Thu, Apr 20, 2017 at 9:05 AM, SARA QUISPE MEJIA <[email protected]>
>>> wrote:
>>>
>>>> I want to parse the log file respect to a client that means to make a
>>>> report of how the client is using my application through the information
>>>> provided by a log file.
>>>>
>>>> So I need to filter some url where I find an especific name of a
>>>> function. For that I thought in insert my log file to database like mysql.
>>>> Could I do that with my log file ? (I use nginx )
>>>>
>>>> I tried to do that with syslog-ng but it doesn't work
>>>>
>>>> Do you have any ideas?
>>>>
>>>> Thanks
>>>>
>>>> Sara
>>>>
>>>> _______________________________________________
>>>> nginx mailing list
>>>> nginx@nginx.org
>>>> http://mailman.nginx.org/mailman/listinfo/nginx
>>>>
>>>
>>>
>>>
>>> --
>>> Oscar Fernandez Sierra
>>> oscaretu@gmail.com
>>>
>>> _______________________________________________
>>> nginx mailing list
>>> nginx@nginx.org
>>> http://mailman.nginx.org/mailman/listinfo/nginx
>>>
>>
>>
>> _______________________________________________
>> nginx mailing list
>> nginx@nginx.org
>> http://mailman.nginx.org/mailman/listinfo/nginx
>>
>
>
>
> --
> Oscar Fernandez Sierra
> oscaretu@gmail.com
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx
>
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Reinis Rozitis
Re: nginx access logs to mysql
April 20, 2017 12:00PM
> I tried to do that with syslog-ng but it doesn't work

What exactly didn't work (before writing a longer configuration example)?

It's kind of simple with syslog-ng - you set the nginx log file as source /
provide mysql as destination and that's it.


rr

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
oscaretu .
Re: nginx access logs to mysql
April 20, 2017 12:40PM
I was trying to say you that you don't need to store lots of unnecesary
logs when you can store just the valuable information contained in the
logs, in a processed way.

Why store thousands of log lines that show some called function
functionname1 instead of store something like this:


{
"date" : "20170420",
"functionmname1": 33,
"functionmname2": 18,
"functionmname3": 51,
"functionmname4": 68,
...
}

In JSON, in a database, or where you want...


On Thu, Apr 20, 2017 at 10:10 AM, SARA QUISPE MEJIA <[email protected]>
wrote:

> yes, I could do it but I want just store for example the customer, the
> function name, the date after with those information I could filter my
> report through a page.php
> But with json I don't know how can I do that?
>
>
> 2017-04-20 9:57 GMT+02:00 oscaretu . <[email protected]>:
>
>> If you extract the lines containing all the function names, and filter
>> them with a script that counts the number of times that the function
>> appears, you only have to store (for example in a JSON) the number of times
>> that each function was used. And you can avoid storing all the logs in a
>> database.
>>
>>
>> On Thu, Apr 20, 2017 at 9:29 AM, SARA QUISPE MEJIA <[email protected]>
>> wrote:
>>
>>> yes, I tried to do that and then I did the analysis with goaccess,
>>> but now I need to do an interface(page.php) for the user can choose
>>> the function that want to filter and then generate a report.
>>> So, I thought put all my information on a database like mysql
>>>
>>> 2017-04-20 9:11 GMT+02:00 oscaretu . <[email protected]>:
>>>
>>>> Sara, why don't you process the log file just with grep / pcregrep to
>>>> get just the lines that containt that function name?
>>>>
>>>>
>>>> On Thu, Apr 20, 2017 at 9:05 AM, SARA QUISPE MEJIA <[email protected]>
>>>> wrote:
>>>>
>>>>> I want to parse the log file respect to a client that means to make a
>>>>> report of how the client is using my application through the information
>>>>> provided by a log file.
>>>>>
>>>>> So I need to filter some url where I find an especific name of a
>>>>> function. For that I thought in insert my log file to database like mysql.
>>>>> Could I do that with my log file ? (I use nginx )
>>>>>
>>>>> I tried to do that with syslog-ng but it doesn't work
>>>>>
>>>>> Do you have any ideas?
>>>>>
>>>>> Thanks
>>>>>
>>>>> Sara
>>>>>
>>>>> _______________________________________________
>>>>> nginx mailing list
>>>>> nginx@nginx.org
>>>>> http://mailman.nginx.org/mailman/listinfo/nginx
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Oscar Fernandez Sierra
>>>> oscaretu@gmail.com
>>>>
>>>> _______________________________________________
>>>> nginx mailing list
>>>> nginx@nginx.org
>>>> http://mailman.nginx.org/mailman/listinfo/nginx
>>>>
>>>
>>>
>>> _______________________________________________
>>> nginx mailing list
>>> nginx@nginx.org
>>> http://mailman.nginx.org/mailman/listinfo/nginx
>>>
>>
>>
>>
>> --
>> Oscar Fernandez Sierra
>> oscaretu@gmail.com
>>
>> _______________________________________________
>> nginx mailing list
>> nginx@nginx.org
>> http://mailman.nginx.org/mailman/listinfo/nginx
>>
>
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx
>



--
Oscar Fernandez Sierra
oscaretu@gmail.com
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Sorry, only registered users may post in this forum.

Click here to login