Welcome! Log In Create A New Profile

Advanced

Using haproxy together with NFS

Posted by Lucas Rolff 
Lucas Rolff
Using haproxy together with NFS
August 01, 2018 10:10PM
Hi guys,

I’ve been playing around today with two NFS servers (each on their own storage array), synced by Unison to provide a bit higher uptime.

To allow NFS clients to use a single IP, I’ve configured an haproxy install (1 now, two when in prod), where I want to talk over tcp mode to the NFS servers.
My idea is that all traffic is directed based on the source IP balancing, so the traffic will be somewhat 50/50 on each NFS server.

My question is if anyone have actually ever got a setup like this to work, I’m using NFS 4.0, and whenever I try to mount the NFS mount on the client, it does communicate with haproxy, and I do see traffic on the NFS server itself, meaning the communication seems to work.

The issue I’m facing, is that the mounting will actually never complete due to some weird behavior when I go through haproxy:

Aug 01 21:44:44 nfs-server-f8209dc4-a1a6-4baf-86fa-eba0b0254bc9 kernel: RPC: fragment too large: 1347571544
Aug 01 21:44:44 nfs-server-f8209dc4-a1a6-4baf-86fa-eba0b0254bc9 kernel: RPC: fragment too large: 1347571544
Aug 01 21:44:44 nfs-server-f8209dc4-a1a6-4baf-86fa-eba0b0254bc9 kernel: RPC: fragment too large: 1347571544
Aug 01 21:44:45 nfs-server-f8209dc4-a1a6-4baf-86fa-eba0b0254bc9 kernel: RPC: fragment too large: 1347571544

It will continue to give this “fragment too large”.
If I bypass haproxy it works completely fine, so I know the NFS Server is configured correctly for the client to connect.

My haproxy configuration looks like this:

global
log 127.0.0.1 local1 debug
nbproc 4
user haproxy
group haproxy
daemon
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid

defaults
mode tcp
log global
option tcplog
timeout client 1m
timeout server 1m
timeout connect 10s
balance source

frontend nfs-in1
bind *:2049
use_backend nfs_backend1
frontend nfs-in2
bind *:111
use_backend nfs_backend2
frontend nfs-in3
bind *:46716
use_backend nfs_backend3
frontend nfs-in4
bind *:36856
use_backend nfs_backend4

backend nfs_backend1
server nfs1 217.xx.xx.xx:2049 send-proxy
backend nfs_backend2
server nfs1 217.xx.xx.xx:111 send-proxy
backend nfs_backend3
server nfs1 217.xx.xx.xx:46716 send-proxy
backend nfs_backend4
server nfs1 217.xx.xx.xx:36856 send-proxy

I use the “send-proxy” to let the NFS Server see the actual source IP, instead of the haproxy machine IP.

If anyone has any idea what can be the cause of the “fragment too large” when going via haproxy, or an actual working config for haproxy to do NFS 4.0 or 4.1 traffic – then please let me know!

Best Regards,
Lucas Rolff
Michael Ezzell
Re: Using haproxy together with NFS
August 02, 2018 02:50AM
On Wed, Aug 1, 2018, 16:00 Lucas Rolff <[email protected]> wrote:

>
> I use the “send-proxy” to let the NFS Server see the actual source IP,
> instead of the haproxy machine IP.
>
You'll probably need remove that. Unless the destination service
explicitly supports the Proxy Protocol (in which case, it must not, by
definition, process connections where the protocol's preamble is *absent*
from the stream), then this would just look like corrupt data. This option
doesn't actually change the source address.

HAProxy in TCP mode should work fine with NFS -- at least, it does with
NFS4.1 as implemented in Amazon Elastic File System -- which is the only
version I've tested against.

https://serverfault.com/a/799213/153161
Lucas Rolff
Re: Using haproxy together with NFS
August 02, 2018 06:10AM
Hi michael,

Without the send-proxy, the client IP in the export would have to be the haproxy server in that case right?

The issue there is then, that I end up with all clients having access to haproxy can suddenly mount all shares in nfs, which I would like to prevent

There’s still different shares that different servers need access to

I’ll try not the sample config from the link above! Thanks!

Get Outlook for iOShttps://aka.ms/o0ukef
________________________________
From: Michael Ezzell <[email protected]>
Sent: Thursday, August 2, 2018 2:38:06 AM
To: Lucas Rolff
Cc: HAproxy Mailing Lists
Subject: Re: Using haproxy together with NFS



On Wed, Aug 1, 2018, 16:00 Lucas Rolff <[email protected]<mailto:[email protected]>> wrote:

I use the “send-proxy” to let the NFS Server see the actual source IP, instead of the haproxy machine IP.
You'll probably need remove that. Unless the destination service explicitly supports the Proxy Protocol (in which case, it must not, by definition, process connections where the protocol's preamble is *absent* from the stream), then this would just look like corrupt data. This option doesn't actually change the source address.

HAProxy in TCP mode should work fine with NFS -- at least, it does with NFS4.1 as implemented in Amazon Elastic File System -- which is the only version I've tested against.

https://serverfault.com/a/799213/153161
Willy Tarreau
Re: Using haproxy together with NFS
August 02, 2018 06:20PM
On Thu, Aug 02, 2018 at 04:05:24AM +0000, Lucas Rolff wrote:
> Hi michael,
>
> Without the send-proxy, the client IP in the export would have to be the
> haproxy server in that case right?

That's it. But Michael is absolutely right, your NFS server doesn't support
the proxy protocol, and the lines it emits below indicate it :

Aug 01 21:44:44 nfs-server-f8209dc4-a1a6-4baf-86fa-eba0b0254bc9 kernel: RPC: fragment too large: 1347571544
Aug 01 21:44:44 nfs-server-f8209dc4-a1a6-4baf-86fa-eba0b0254bc9 kernel: RPC: fragment too large: 1347571544
Aug 01 21:44:44 nfs-server-f8209dc4-a1a6-4baf-86fa-eba0b0254bc9 kernel: RPC: fragment too large: 1347571544
Aug 01 21:44:45 nfs-server-f8209dc4-a1a6-4baf-86fa-eba0b0254bc9 kernel: RPC: fragment too large: 1347571544

This fragment size (1347571544) is "PROX" encoded in big endian, which are
the first 4 chars of the proxy protocol header :-)

> The issue there is then, that I end up with all clients having access to
> haproxy can suddenly mount all shares in nfs, which I would like to prevent

Maybe you can modify your NFS server to support the proxy protocol, that
could possibly make sense for your use case ? Otherwise on Linux you may
be able to configure haproxy to work in transparent mode using "source
0.0.0.0 usesrc clientip" but beware that it requires some specific iptables
rules to divert the traffic and send it back to haproxy. It will also require
that all your NFS servers route the clients via haproxy for the response
traffic. This is not always very convenient.

Regards,
Willy
Lucas Rolff
Re: Using haproxy together with NFS
August 02, 2018 06:50PM
I indeed removed the send-proxy - then I had to put the IP of haproxy in the NFS exports file instead to be able to mount the share (which makes sense seen from a NFS perspective).

Making the NFS server support proxy protocol, isn't something I think will happen - I rely on the upstream packages (CentOS 7 packages in this case).

And using transparency mode - I think relying on stuff going via haproxy for routing won't be a possibility in this case - so I guess I have to drop my wish about haproxy + NFS in this case, I'd like something that is fairly standard without too much modifications on the current NFS infrastructure (since it would introduce more complexity).

Thanks for your replies both of you!

Best Regards,

On 02/08/2018, 18.09, "Willy Tarreau" <[email protected]> wrote:

On Thu, Aug 02, 2018 at 04:05:24AM +0000, Lucas Rolff wrote:
> Hi michael,
>
> Without the send-proxy, the client IP in the export would have to be the
> haproxy server in that case right?

That's it. But Michael is absolutely right, your NFS server doesn't support
the proxy protocol, and the lines it emits below indicate it :

Aug 01 21:44:44 nfs-server-f8209dc4-a1a6-4baf-86fa-eba0b0254bc9 kernel: RPC: fragment too large: 1347571544
Aug 01 21:44:44 nfs-server-f8209dc4-a1a6-4baf-86fa-eba0b0254bc9 kernel: RPC: fragment too large: 1347571544
Aug 01 21:44:44 nfs-server-f8209dc4-a1a6-4baf-86fa-eba0b0254bc9 kernel: RPC: fragment too large: 1347571544
Aug 01 21:44:45 nfs-server-f8209dc4-a1a6-4baf-86fa-eba0b0254bc9 kernel: RPC: fragment too large: 1347571544

This fragment size (1347571544) is "PROX" encoded in big endian, which are
the first 4 chars of the proxy protocol header :-)

> The issue there is then, that I end up with all clients having access to
> haproxy can suddenly mount all shares in nfs, which I would like to prevent

Maybe you can modify your NFS server to support the proxy protocol, that
could possibly make sense for your use case ? Otherwise on Linux you may
be able to configure haproxy to work in transparent mode using "source
0.0.0.0 usesrc clientip" but beware that it requires some specific iptables
rules to divert the traffic and send it back to haproxy. It will also require
that all your NFS servers route the clients via haproxy for the response
traffic. This is not always very convenient.

Regards,
Willy
Sander Klein
Re: Using haproxy together with NFS
August 03, 2018 11:40AM
Hi,

You might want to have a look at IPVS for instance in combination with Keepalived. You can then even use udp mounts if you want.

Just my 2 cents.

Regards,

Sander


> On 2 Aug 2018, at 18:40, Lucas Rolff <[email protected]> wrote:
>
> I indeed removed the send-proxy - then I had to put the IP of haproxy in the NFS exports file instead to be able to mount the share (which makes sense seen from a NFS perspective).
>
> Making the NFS server support proxy protocol, isn't something I think will happen - I rely on the upstream packages (CentOS 7 packages in this case).
>
> And using transparency mode - I think relying on stuff going via haproxy for routing won't be a possibility in this case - so I guess I have to drop my wish about haproxy + NFS in this case, I'd like something that is fairly standard without too much modifications on the current NFS infrastructure (since it would introduce more complexity).
>
> Thanks for your replies both of you!
>
> Best Regards,
>
> On 02/08/2018, 18.09, "Willy Tarreau" <[email protected]> wrote:
>
>> On Thu, Aug 02, 2018 at 04:05:24AM +0000, Lucas Rolff wrote:
>> Hi michael,
>>
>> Without the send-proxy, the client IP in the export would have to be the
>> haproxy server in that case right?
>
> That's it. But Michael is absolutely right, your NFS server doesn't support
> the proxy protocol, and the lines it emits below indicate it :
>
> Aug 01 21:44:44 nfs-server-f8209dc4-a1a6-4baf-86fa-eba0b0254bc9 kernel: RPC: fragment too large: 1347571544
> Aug 01 21:44:44 nfs-server-f8209dc4-a1a6-4baf-86fa-eba0b0254bc9 kernel: RPC: fragment too large: 1347571544
> Aug 01 21:44:44 nfs-server-f8209dc4-a1a6-4baf-86fa-eba0b0254bc9 kernel: RPC: fragment too large: 1347571544
> Aug 01 21:44:45 nfs-server-f8209dc4-a1a6-4baf-86fa-eba0b0254bc9 kernel: RPC: fragment too large: 1347571544
>
> This fragment size (1347571544) is "PROX" encoded in big endian, which are
> the first 4 chars of the proxy protocol header :-)
>
>> The issue there is then, that I end up with all clients having access to
>> haproxy can suddenly mount all shares in nfs, which I would like to prevent
>
> Maybe you can modify your NFS server to support the proxy protocol, that
> could possibly make sense for your use case ? Otherwise on Linux you may
> be able to configure haproxy to work in transparent mode using "source
> 0.0.0.0 usesrc clientip" but beware that it requires some specific iptables
> rules to divert the traffic and send it back to haproxy. It will also require
> that all your NFS servers route the clients via haproxy for the response
> traffic. This is not always very convenient.
>
> Regards,
> Willy
>
>
Sorry, only registered users may post in this forum.

Click here to login