Michael Lopez
HA_PROXY multiline tcp payload
November 09, 2017 07:10PM
I am utilizing HA_Proxy for TCP load balancing and my configuration is
(Keepalive Virtual IP -> Ha_Proxy -> syslog-ng -> source_ip.log).

When receiving TCP payloads which are multi line the first line is
forwarded appropriately to the source IP log file and rest of the payload
goes to a secondary file with the Virtual IP address which means my logs
for those devices are split into 2 log files and the secondary file does
not contain the source.

Now the key is that when I remove HA_PROXY from the equation and only
utilize (Keepalive Virtual IP -> syslog-ng -> source_ip.log) I only have 1
file per source and even the multi-line is added to the appropriate source
file.

Wondering if anyone has experienced this issue and how they resolved it.
Any help would be greatly appreciated

Sincerely,
Michael
Willy Tarreau
Re: HA_PROXY multiline tcp payload
November 10, 2017 06:00AM
On Thu, Nov 09, 2017 at 01:04:03PM -0500, Michael Lopez wrote:
> I am utilizing HA_Proxy for TCP load balancing and my configuration is
> (Keepalive Virtual IP -> Ha_Proxy -> syslog-ng -> source_ip.log).
>
> When receiving TCP payloads which are multi line the first line is
> forwarded appropriately to the source IP log file and rest of the payload
> goes to a secondary file with the Virtual IP address which means my logs
> for those devices are split into 2 log files and the secondary file does
> not contain the source.
>
> Now the key is that when I remove HA_PROXY from the equation and only
> utilize (Keepalive Virtual IP -> syslog-ng -> source_ip.log) I only have 1
> file per source and even the multi-line is added to the appropriate source
> file.
>
> Wondering if anyone has experienced this issue and how they resolved it.
> Any help would be greatly appreciated

I suspect that your log client is establishing a new connection for each and
every new line, and that syslog-ng correctly routes the second half of the
message to the correct file when it doesn't find the address in it. But when
adding haproxy in the middle, the source address changes, it's haproxy's. So
the first half of the log message contains the IP address set by the client
and goes to the correct file, the second half doesn't contain it and syslog-ng
has to fall back to the source IP address of the connection which is now
haproxy's, and the log goes to the file containing haproxy's source address
logs.

If that's the case, either there's a way to prevent the client from closing
between two messages, or you'll have to adapt haproxy to work in transparent
mode so that it spoofs the client's address when connecting to syslog-ng.
Another more elegant solution would be to try to implement the PROXY protocol
in syslog-ng, it would be a perfect fit for this. Apparently this has already
been discussed, and Balasz even considered implementing it, so it might just
be a matter of priority :

https://lists.balabit.hu/pipermail/syslog-ng/2015-October/022412.html

Willy
Sorry, only registered users may post in this forum.

Click here to login