Welcome! Log In Create A New Profile

Advanced

HAProxy SMTP Rate limiting

Posted by SAFENTRIX Administrator 
SAFENTRIX Administrator
HAProxy SMTP Rate limiting
November 12, 2017 03:30PM
Hi

We are using HAProxy v 1.6.4 for SMTP load balancing (in transparent mode) and it works fine.

Now we want to block connections from IP addresses that abuse the Email server.

One way to identify this is by looking at connections where server returns a response starting with letter "5" (SMTP permanent error codes start with this).

We are trying to implement this in HAProxy, but it is not working. I have put the relevant parts of haproxy.conf in here.

** START
frontend nsmtpproxy
bind :25 transparent
mode tcp
tcp-request inspect-delay 5s
tcp-request connection reject if { src_conn_cnt(nappsmtp) gt 2 }
default_backend nappsmtp

backend nappsmtp
source 0.0.0.0 usesrc clientip
mode tcp
acl isfail res.payload(0,3) -m beg 5
stick-table type ip size 30k expire 30m store conn_cnt
stick store-response src_updt_conn_cnt if isfail
server smtp1 192.168.1.6:25 check inter 60000
server smtp2 192.168.1.7:25 check inter 60000
option smtpchk
** END

Pl let us know where we are going wrong.
Willy Tarreau
Re: HAProxy SMTP Rate limiting
November 19, 2017 07:50AM
Hello,

On Sun, Nov 12, 2017 at 09:19:33AM -0500, SAFENTRIX Administrator wrote:
> We are using HAProxy v 1.6.4 for SMTP load balancing (in transparent mode)
> and it works fine.
>
> Now we want to block connections from IP addresses that abuse the Email
> server.
>
> One way to identify this is by looking at connections where server returns a
> response starting with letter "5" (SMTP permanent error codes start with
> this).
>
> We are trying to implement this in HAProxy, but it is not working. I have
> put the relevant parts of haproxy.conf in here.
>
> ** START
> frontend nsmtpproxy
> bind :25 transparent
> mode tcp
> tcp-request inspect-delay 5s
> tcp-request connection reject if { src_conn_cnt(nappsmtp) gt 2 }
> default_backend nappsmtp
>
> backend nappsmtp
> source 0.0.0.0 usesrc clientip
> mode tcp
> acl isfail res.payload(0,3) -m beg 5
> stick-table type ip size 30k expire 30m store conn_cnt
> stick store-response src_updt_conn_cnt if isfail
> server smtp1 192.168.1.6:25 check inter 60000
> server smtp2 192.168.1.7:25 check inter 60000
> option smtpchk
> ** END
>
> Pl let us know where we are going wrong.

I'm seeing something wrong indeed. "stick store-response" is only used
to perform some stickiness, that's not what you want. If I were you,
I'd instead use the gpt0 tag to indicate that the client was refused
by the server. The frontend would then do this :

tcp-request connection track-sc0 src table nappsmtp
tcp-request connection reject if { sc0_get_gpt0 gt 0 }

and the backend :

tcp-response content sc-set-gpt0(0) 1 if isfail

This will mark the IP in the table on the first occurrence of a 5xx
response. If you want to leave a small margin to start blocking after
the 3rd failure as you did, better then the gpc0 counter then.

Hoping this helps,
Willy
SAFENTRIX Administrator
Re: HAProxy SMTP Rate limiting
November 20, 2017 08:40AM
Hi:

Thanks for the quick response. We changed the configuration as follows and it still does not work. The problem seems to be getting the response payload and comparing it against letter "5". We have also tried upgrading to 1.7.9 and face the same issue. The stick table does not get updated at all.

Is the format of the ACL fine? Also is there any way we can make haproxy print out the contents of res.payload(0,3) to ensure that we are comparing against the proper string?

Also we are using "mode tcp".. Will res.payload work with this mode?

** START
frontend nsmtpproxy
bind :25 transparent
mode tcp
tcp-request connection track-sc0 src table nappsmtp
tcp-request connection reject if { sc0_get_gpc0 gt 2 }
option tcplog
default_backend nappsmtp

backend nappsmtp

source 0.0.0.0 usesrc clientip
mode tcp
acl isfail res.payload(0,3),hex -m sub 53
stick-table type ip size 30k expire 30m store gpc0
tcp-response content sc-inc-gpc0(0) if isfail
server smtp1 192.168.1.6:25 check inter 60000
server smtp2 192.168.1.7:25 check inter 60000
option smtpchk

** END


-----Original Message-----
Hello,

On Sun, Nov 12, 2017 at 09:19:33AM -0500, SAFENTRIX Administrator wrote:
> We are using HAProxy v 1.6.4 for SMTP load balancing (in transparent mode)
> and it works fine.
>
> Now we want to block connections from IP addresses that abuse the Email
> server.
>
> One way to identify this is by looking at connections where server returns a
> response starting with letter "5" (SMTP permanent error codes start with
> this).
>
> We are trying to implement this in HAProxy, but it is not working. I have
> put the relevant parts of haproxy.conf in here.
>
> ** START
> frontend nsmtpproxy
> bind :25 transparent
> mode tcp
> tcp-request inspect-delay 5s
> tcp-request connection reject if { src_conn_cnt(nappsmtp) gt 2 }
> default_backend nappsmtp
>
> backend nappsmtp
> source 0.0.0.0 usesrc clientip
> mode tcp
> acl isfail res.payload(0,3) -m beg 5
> stick-table type ip size 30k expire 30m store conn_cnt
> stick store-response src_updt_conn_cnt if isfail
> server smtp1 192.168.1.6:25 check inter 60000
> server smtp2 192.168.1.7:25 check inter 60000
> option smtpchk
> ** END
>
> Pl let us know where we are going wrong.

I'm seeing something wrong indeed. "stick store-response" is only used
to perform some stickiness, that's not what you want. If I were you,
I'd instead use the gpt0 tag to indicate that the client was refused
by the server. The frontend would then do this :

tcp-request connection track-sc0 src table nappsmtp
tcp-request connection reject if { sc0_get_gpt0 gt 0 }

and the backend :

tcp-response content sc-set-gpt0(0) 1 if isfail

This will mark the IP in the table on the first occurrence of a 5xx
response. If you want to leave a small margin to start blocking after
the 3rd failure as you did, better then the gpc0 counter then.

Hoping this helps,
Willy

--
STOP Virus, STOP SPAM, SAVE Bandwidth!
http://www.safentrix.com/adlink?cid=0
--
Moemen MHEDHBI
Re: HAProxy SMTP Rate limiting
November 20, 2017 09:50PM
Hi,

Probably the problem you are having is that HAProxy is only checking the
first server reply which is 220.

In fact, in TCP mode load balancing decisions (ACLs and such,..) are
taken for the whole connection (In HTTP mode, decisions are taken per
request.).

In this case the  res.payload(0,3) is only checked for the first server
reply and decision will be taken for the whole connecion (so no further
ACL executions for other responses).

I have checked with an smtp server, and it is the first to talk whenever
a new connection is established (220 blabla ) so res.payload(0,3),hex -m
sub 53 will never match.

You can print the content of the payload using the following config:

log-format %ci:%cp\ [%t]\ %ft\ %b/%s\ %Tq/%Tw/%Tc/%Tr/%Tt\ %ST\ %B\ %CC\
%CS\ %tsc\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq\ %hr\ %hs\ %[var(sess.smtpresp]\
%{+Q}r
tcp-response  content set-var(sess.smtpresp) res.payload(0,3)


++


On 20/11/2017 08:34, SAFENTRIX Administrator wrote:
>
> Hi:
>
> Thanks for the quick response. We changed the configuration as follows
> and it still does not work. The problem seems to be getting the
> response payload and comparing it against letter "5". We have also
> tried upgrading to 1.7.9 and face the same issue. The stick table does
> not get updated at all.
>
>  
>
> Is the format of the ACL fine? Also is there any way we can make
> haproxy print out the contents of res.payload(0,3) to ensure that we
> are comparing against the proper string?
>
>  
>
> Also we are using "mode tcp". Will res.payload work with this mode?
>
>  
>
> ** START
> frontend      nsmtpproxy
>         bind :25 transparent
>         mode tcp
>         tcp-request connection track-sc0 src table nappsmtp
>         tcp-request connection reject if { sc0_get_gpc0 gt 2 }
>         option tcplog
>         default_backend nappsmtp
>
>  
>
> backend nappsmtp
>
>
>     source 0.0.0.0 usesrc clientip
>     mode tcp
>     acl isfail res.payload(0,3),hex -m sub 53
>     stick-table type ip size 30k expire 30m store gpc0
>     tcp-response content sc-inc-gpc0(0) if isfail
>     server smtp1 192.168.1.6:25 check inter 60000
>
>     server smtp2 192.168.1.7:25 check inter 60000  
>
>     option smtpchk
>
>
> ** END
>
>
>
> -----Original Message-----
> Hello,
>
> On Sun, Nov 12, 2017 at 09:19:33AM -0500, SAFENTRIX Administrator wrote:
> > We are using HAProxy v 1.6.4 for SMTP load balancing (in transparent
> mode)
> > and it works fine.
> >
> > Now we want to block connections from IP addresses that abuse the Email
> > server.
> >
> > One way to identify this is by looking at connections where server
> returns a
> > response starting with letter "5" (SMTP permanent error codes start with
> > this).
> >
> > We are trying to implement this in HAProxy, but it is not working. I
> have
> > put the relevant parts of haproxy.conf in here.
> >
> > ** START
> > frontend nsmtpproxy
> > bind :25 transparent
> > mode tcp
> > tcp-request inspect-delay 5s
> > tcp-request connection reject if { src_conn_cnt(nappsmtp) gt 2 }
> > default_backend nappsmtp
> >
> > backend nappsmtp
> > source 0.0.0.0 usesrc clientip
> > mode tcp
> > acl isfail res.payload(0,3) -m beg 5
> > stick-table type ip size 30k expire 30m store conn_cnt
> > stick store-response src_updt_conn_cnt if isfail
> > server smtp1 192.168.1.6:25 check inter 60000
> > server smtp2 192.168.1.7:25 check inter 60000
> > option smtpchk
> > ** END
> >
> > Pl let us know where we are going wrong.
>
> I'm seeing something wrong indeed. "stick store-response" is only used
> to perform some stickiness, that's not what you want. If I were you,
> I'd instead use the gpt0 tag to indicate that the client was refused
> by the server. The frontend would then do this :
>
> tcp-request connection track-sc0 src table nappsmtp
> tcp-request connection reject if { sc0_get_gpt0 gt 0 }
>
> and the backend :
>
> tcp-response content sc-set-gpt0(0) 1 if isfail
>
> This will mark the IP in the table on the first occurrence of a 5xx
> response. If you want to leave a small margin to start blocking after
> the 3rd failure as you did, better then the gpc0 counter then.
>
> Hoping this helps,
> Willy
>
> --
> STOP Virus, STOP SPAM, SAVE Bandwidth!
> http://www.safentrix.com/adlink?cid=0
> --
>

--
Moemen MHEDHBI
SAFENTRIX Administrator
Re: HAProxy SMTP Rate limiting
November 21, 2017 05:10AM
HI:


Thanks for the reply. Your theory is absolutely correct. Though we could not print out the logs, we did the following to confirm:

1. Changed acl to

acl isfail res.payload(0,3) -m beg 220

2. We could see the gpc0 getting hit and incremented.

So this means that our approach will not work. We would much prefer to do small tweaks in HAProxy than put in a separate SMTP Application proxy for this (as we are much more confident about HAProxy capability).

Before giving up on HAProxy for this, we would like to try just one more time.

1. Is there any way to intercept start of every server response traffic stream, and compare it against "5"?
2. Could LUA scripting give a solution?

-----Original Message-----



Hi,
Probably the problem you are having is that HAProxy is only checking the first server reply which is 220.
In fact, in TCP mode load balancing decisions (ACLs and such,...) are taken for the whole connection (In HTTP mode, decisions are taken per request.).
In this case the res.payload(0,3) is only checked for the first server reply and decision will be taken for the whole connecion (so no further ACL executions for other responses).
I have checked with an smtp server, and it is the first to talk whenever a new connection is established (220 blabla ) so res.payload(0,3),hex -m sub 53 will never match.
You can print the content of the payload using the following config:

log-format %ci:%cp\ [%t]\ %ft\ %b/%s\ %Tq/%Tw/%Tc/%Tr/%Tt\ %ST\ %B\ %CC\ %CS\ %tsc\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq\ %hr\ %hs\ %[var(sess.smtpresp]\ %{+Q}r
tcp-response content set-var(sess.smtpresp) res.payload(0,3)

++

On 20/11/2017 08:34, SAFENTRIX Administrator wrote:
Hi:

Thanks for the quick response. We changed the configuration as follows and it still does not work. The problem seems to be getting the response payload and comparing it against letter "5". We have also tried upgrading to 1.7.9 and face the same issue. The stick table does not get updated at all.

Is the format of the ACL fine? Also is there any way we can make haproxy print out the contents of res.payload(0,3) to ensure that we are comparing against the proper string?

Also we are using "mode tcp". Will res.payload work with this mode?

** START
frontend nsmtpproxy
bind :25 transparent
mode tcp
tcp-request connection track-sc0 src table nappsmtp
tcp-request connection reject if { sc0_get_gpc0 gt 2 }
option tcplog
default_backend nappsmtp

backend nappsmtp

source 0.0.0.0 usesrc clientip
mode tcp
acl isfail res.payload(0,3),hex -m sub 53
stick-table type ip size 30k expire 30m store gpc0
tcp-response content sc-inc-gpc0(0) if isfail
server smtp1 192.168.1.6:25 check inter 60000
server smtp2 192.168.1.7:25 check inter 60000
option smtpchk

** END


-----Original Message-----
Hello,

On Sun, Nov 12, 2017 at 09:19:33AM -0500, SAFENTRIX Administrator wrote:
> We are using HAProxy v 1.6.4 for SMTP load balancing (in transparent mode)
> and it works fine.
>
> Now we want to block connections from IP addresses that abuse the Email
> server.
>
> One way to identify this is by looking at connections where server returns a
> response starting with letter "5" (SMTP permanent error codes start with
> this).
>
> We are trying to implement this in HAProxy, but it is not working. I have
> put the relevant parts of haproxy.conf in here.
>
> ** START
> frontend nsmtpproxy
> bind :25 transparent
> mode tcp
> tcp-request inspect-delay 5s
> tcp-request connection reject if { src_conn_cnt(nappsmtp) gt 2 }
> default_backend nappsmtp
>
> backend nappsmtp
> source 0.0.0.0 usesrc clientip
> mode tcp
> acl isfail res.payload(0,3) -m beg 5
> stick-table type ip size 30k expire 30m store conn_cnt
> stick store-response src_updt_conn_cnt if isfail
> server smtp1 192.168.1.6:25 check inter 60000
> server smtp2 192.168.1.7:25 check inter 60000
> option smtpchk
> ** END
>
> Pl let us know where we are going wrong.

I'm seeing something wrong indeed. "stick store-response" is only used
to perform some stickiness, that's not what you want. If I were you,
I'd instead use the gpt0 tag to indicate that the client was refused
by the server. The frontend would then do this :

tcp-request connection track-sc0 src table nappsmtp
tcp-request connection reject if { sc0_get_gpt0 gt 0 }

and the backend :

tcp-response content sc-set-gpt0(0) 1 if isfail

This will mark the IP in the table on the first occurrence of a 5xx
response. If you want to leave a small margin to start blocking after
the 3rd failure as you did, better then the gpc0 counter then.

Hoping this helps,
Willy

--
STOP Virus, STOP SPAM, SAVE Bandwidth!
[ http://www.safentrix.com/adlink?cid=0 ]( http://www.safentrix.com/adlink?cid=0 )
--

-- Moemen MHEDHBI STOP Virus, STOP SPAM, SAVE Bandwidth!
[ www.safentrix.com ]( http://www.safentrix.com/adlink?cid=0 )
Sorry, only registered users may post in this forum.

Click here to login