Welcome! Log In Create A New Profile

Advanced

Haproxy support for handling concurrent requests from different clients

Posted by Mihir Shirali 
Hi Team,

We have haproxy installed on a server which is being used primarily for
front ending TLS. After session establishment it sets certain headers in
the http request and forwards it to the application in the backend. The
back end application is a tftp server and hence it can receive requests
from a large number of clients.
What we observe on our server is that when we have large number of clients
haproxy gets quite busy and the CPU clocks pretty high. Since both haproxy
and our backend application run on the same server - this combined CPU can
get close to the limit.
What we’d like to know is if there is a way to throttle the number of
requests per second. All the searches so far - seem to indicate that we
could rate limit based on src ip or http header. However, since our client
ips will be different in the real world we wont be able to use that (less
recurrence)
Could you please help? Is this possible?
Am 10.05.2018 um 18:27 schrieb Mihir Shirali:
> Hi Team,
>
> We have haproxy installed on a server which is being used primarily for front
> ending TLS. After session establishment it sets certain headers in the http
> request and forwards it to the application in the backend. The back end
> application is a tftp server and hence it can receive requests from a large
> number of clients.

Why do you add http header for a tftp service?
Do you really mean https://de.wikipedia.org/wiki/Trivial_File_Transfer_Protocol

> What we observe on our server is that when we have large number of clients
> haproxy gets quite busy and the CPU clocks pretty high. Since both haproxy and
> our backend application run on the same server - this combined CPU can get close
> to the limit.
> What we’d like to know is if there is a way to throttle the number of requests
> per second. All the searches so far - seem to indicate that we could rate limit
> based on src ip or http header. However, since our client ips will be different
> in the real world we wont be able to use that (less recurrence)
> Could you please help? Is this possible?

What's the output of haproxy -vv ?
There was some issues about high CPU Usage so maybe you will need to update.

Could this be a option?
https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#4.2-rate-limit%20sessions
https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#7.3.3-src_updt_conn_cnt

What's 'less recurrence' , hours, days?

Regards
Aleks
Hi Aleksandar,

Why do you add http header for a tftp service?
Do you really mean https://de.wikipedia.org/wiki/
Trivial_File_Transfer_Protocol
[Mihir]>>This TFTP is a custom application written by us. The http headers
also have custom attributes which are used by the backend application.

haproxy version is
HA-Proxy version 1.5.11 2015/01/31

https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#4.2-rate-
limit%20sessions
[Mihir]>>I believe this only queues the packets right? Is there a way we
could tell the client to back off and retry after a bit (like a 503). This
decision based on the high number of requests.


On Fri, May 11, 2018 at 1:58 AM, Aleksandar Lazic <[email protected]t>
wrote:

> Am 10.05.2018 um 18:27 schrieb Mihir Shirali:
> > Hi Team,
> >
> > We have haproxy installed on a server which is being used primarily for
> front
> > ending TLS. After session establishment it sets certain headers in the
> http
> > request and forwards it to the application in the backend. The back end
> > application is a tftp server and hence it can receive requests from a
> large
> > number of clients.
>
> Why do you add http header for a tftp service?
> Do you really mean https://de.wikipedia.org/wiki/
> Trivial_File_Transfer_Protocol
>
> > What we observe on our server is that when we have large number of
> clients
> > haproxy gets quite busy and the CPU clocks pretty high. Since both
> haproxy and
> > our backend application run on the same server - this combined CPU can
> get close
> > to the limit.
> > What we’d like to know is if there is a way to throttle the number of
> requests
> > per second. All the searches so far - seem to indicate that we could
> rate limit
> > based on src ip or http header. However, since our client ips will be
> different
> > in the real world we wont be able to use that (less recurrence)
> > Could you please help? Is this possible?
>
> What's the output of haproxy -vv ?
> There was some issues about high CPU Usage so maybe you will need to
> update.
>
> Could this be a option?
> https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#4.2-rate-
> limit%20sessions
> https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#7.3.3-src_
> updt_conn_cnt
>
> What's 'less recurrence' , hours, days?
>
> Regards
> Aleks
>



--
Regards,
Mihir
Hi Mihir.

Am 11.05.2018 um 05:57 schrieb Mihir Shirali:
> Hi Aleksandar,
>
> Why do you add http header for a tftp service?
> Do you really mean https://de.wikipedia.org/wiki/Trivial_File_Transfer_Protocol
> https://de.wikipedia.org/wiki/Trivial_File_Transfer_Protocol
> [Mihir]>>This TFTP is a custom application written by us. The http headers also
> have custom attributes which are used by the backend application.
>
> haproxy version is 
> HA-Proxy version 1.5.11 2015/01/31

Could you try to update at least to the latest 1.5 or better to 1.8?
https://www.haproxy.org/bugs/bugs-1.5.11.html

> https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#4.2-rate-limit%20sessions
> <https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#4.2-rate-limit%20sessions>;
>
> [Mihir]>>I believe this only queues the packets right? Is there a way we could
> tell the client to back off and retry after a bit (like a 503). This decision
> based on the high number of requests.

Yes it's possible but I haven't done it before.
I would try this, but I hope that someone with more experience in this topic
step forward and show us a working solution.

https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#4.2-http-request
https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#7.3.3-src_conn_rate

http-request connection track-sc0 src
http-request deny deny_status 503 if { src_conn_rate gt 10 }

This lines are shameless copied from the examples in
https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#4.2-tcp-request%20connection

Regards
Aleks

> On Fri, May 11, 2018 at 1:58 AM, Aleksandar Lazic <[email protected]
> <mailto:[email protected]>> wrote:
>
> Am 10.05.2018 um 18:27 schrieb Mihir Shirali:
> > Hi Team,
> >
> > We have haproxy installed on a server which is being used primarily for front
> > ending TLS. After session establishment it sets certain headers in the http
> > request and forwards it to the application in the backend. The back end
> > application is a tftp server and hence it can receive requests from a large
> > number of clients.
>
> Why do you add http header for a tftp service?
> Do you really mean
> https://de.wikipedia.org/wiki/Trivial_File_Transfer_Protocol
> https://de.wikipedia.org/wiki/Trivial_File_Transfer_Protocol
>
> > What we observe on our server is that when we have large number of clients
> > haproxy gets quite busy and the CPU clocks pretty high. Since both haproxy and
> > our backend application run on the same server - this combined CPU can get close
> > to the limit.
> > What we’d like to know is if there is a way to throttle the number of requests
> > per second. All the searches so far - seem to indicate that we could rate limit
> > based on src ip or http header. However, since our client ips will be different
> > in the real world we wont be able to use that (less recurrence)
> > Could you please help? Is this possible?
>
> What's the output of haproxy -vv ?
> There was some issues about high CPU Usage so maybe you will need to update.
>
> Could this be a option?
> https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#4.2-rate-limit%20sessions
> <https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#4.2-rate-limit%20sessions>;
> https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#7.3.3-src_updt_conn_cnt
> <https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#7.3.3-src_updt_conn_cnt>;
>
> What's 'less recurrence' , hours, days?
>
> Regards
> Aleks
>
>
>
>
> --
> Regards,
> Mihir
Thanks Aleksandar for the help!
I did look up some examples for setting 503 - but all of them (as you've
indicated) seem based on src ip or src header. I'm guessing this is more
suitable for a DOS/DDOS attack? In our deployment, the likelihood of
getting one request from multiple clients is more than multiple requests
from a single client.
As an update the rate-limit directive has helped. However, the only problem
is that the client does not know that the server is busy and *could* time
out. It would be great if it were possible to somehow send a 503 out , so
the clients could retry after a random time.

With respect to the update - we are evaluating this and have run into some
issues since we need to host 2 different certificates on the port (served
based on the cipher). We should be able to fix this on our own though.

On Fri, May 11, 2018 at 11:41 AM, Aleksandar Lazic <[email protected]>
wrote:

> Hi Mihir.
>
> Am 11.05.2018 um 05:57 schrieb Mihir Shirali:
> > Hi Aleksandar,
> >
> > Why do you add http header for a tftp service?
> > Do you really mean https://de.wikipedia.org/wiki/Trivial_File_Transfer_
> Protocol
> > https://de.wikipedia.org/wiki/Trivial_File_Transfer_Protocol
> > [Mihir]>>This TFTP is a custom application written by us. The http
> headers also
> > have custom attributes which are used by the backend application.
> >
> > haproxy version is
> > HA-Proxy version 1.5.11 2015/01/31
>
> Could you try to update at least to the latest 1.5 or better to 1.8?
> https://www.haproxy.org/bugs/bugs-1.5.11.html
>
> > https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#4.2-rate-
> limit%20sessions
> > <https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#4.2-rate-
> limit%20sessions>
> >
> > [Mihir]>>I believe this only queues the packets right? Is there a way we
> could
> > tell the client to back off and retry after a bit (like a 503). This
> decision
> > based on the high number of requests.
>
> Yes it's possible but I haven't done it before.
> I would try this, but I hope that someone with more experience in this
> topic
> step forward and show us a working solution.
>
> https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#4.2-http-
> request
> https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#7.3.3-src_
> conn_rate
>
> http-request connection track-sc0 src
> http-request deny deny_status 503 if { src_conn_rate gt 10 }
>
> This lines are shameless copied from the examples in
> https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#4.2-tcp-
> request%20connection
>
> Regards
> Aleks
>
> > On Fri, May 11, 2018 at 1:58 AM, Aleksandar Lazic <[email protected]
> > <mailto:[email protected]>> wrote:
> >
> > Am 10.05.2018 um 18:27 schrieb Mihir Shirali:
> > > Hi Team,
> > >
> > > We have haproxy installed on a server which is being used
> primarily for front
> > > ending TLS. After session establishment it sets certain headers in
> the http
> > > request and forwards it to the application in the backend. The
> back end
> > > application is a tftp server and hence it can receive requests
> from a large
> > > number of clients.
> >
> > Why do you add http header for a tftp service?
> > Do you really mean
> > https://de.wikipedia.org/wiki/Trivial_File_Transfer_Protocol
> > https://de.wikipedia.org/wiki/Trivial_File_Transfer_Protocol
> >
> > > What we observe on our server is that when we have large number of
> clients
> > > haproxy gets quite busy and the CPU clocks pretty high. Since both
> haproxy and
> > > our backend application run on the same server - this combined CPU
> can get close
> > > to the limit.
> > > What we’d like to know is if there is a way to throttle the number
> of requests
> > > per second. All the searches so far - seem to indicate that we
> could rate limit
> > > based on src ip or http header. However, since our client ips will
> be different
> > > in the real world we wont be able to use that (less recurrence)
> > > Could you please help? Is this possible?
> >
> > What's the output of haproxy -vv ?
> > There was some issues about high CPU Usage so maybe you will need to
> update.
> >
> > Could this be a option?
> > https://cbonte.github.io/haproxy-dconv/1.8/
> configuration.html#4.2-rate-limit%20sessions
> > <https://cbonte.github.io/haproxy-dconv/1.8/
> configuration.html#4.2-rate-limit%20sessions>
> > https://cbonte.github.io/haproxy-dconv/1.8/
> configuration.html#7.3.3-src_updt_conn_cnt
> > <https://cbonte.github.io/haproxy-dconv/1.8/
> configuration.html#7.3.3-src_updt_conn_cnt>
> >
> > What's 'less recurrence' , hours, days?
> >
> > Regards
> > Aleks
> >
> >
> >
> >
> > --
> > Regards,
> > Mihir
>
>


--
Regards,
Mihir
On Fri, 11 May 2018 8:01 pm Mihir Shirali <[email protected]> wrote:

> Thanks Aleksandar for the help!
> I did look up some examples for setting 503 - but all of them (as you've
> indicated) seem based on src ip or src header. I'm guessing this is more
> suitable for a DOS/DDOS attack? In our deployment, the likelihood of
> getting one request from multiple clients is more than multiple requests
> from a single client.
> As an update the rate-limit directive has helped. However, the only
> problem is that the client does not know that the server is busy and
> *could* time out. It would be great if it were possible to somehow send a
> 503 out , so the clients could retry after a random time.
>

Or even better 429.
Hi,

On Fri, May 11, Mihir Shirali wrote:
> I did look up some examples for setting 503 - but all of them (as you've
> indicated) seem based on src ip or src header. I'm guessing this is more
> suitable for a DOS/DDOS attack? In our deployment, the likelihood of
> getting one request from multiple clients is more than multiple requests
> from a single client.

Can you explain how/when(on what condition) you'd like to limit the number
of requests and haproxy return 503 status to clients (429 seems more
appropriate status code for this) ?

If you just want haproxy to return 503 for all new requests when
there're X number of sessions/connections/session rate then
take a look at fe_conn, fe_req_rate, fe_sess_rate, be_conn and
be_sess_rate
(https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#7.3.2-fe_conn)
so for example something like
http-request deny deny_status 503 if { fe_req_rate gt 50 }

> As an update the rate-limit directive has helped. However, the only problem
> is that the client does not know that the server is busy and *could* time
> out. It would be great if it were possible to somehow send a 503 out , so
> the clients could retry after a random time.

-Jarno

--
Jarno Huuskonen
Thanks Jamo!
This is just what we were looking for!



On Tue, May 15, 2018 at 10:17 PM, Jarno Huuskonen <[email protected]>
wrote:

> Hi,
>
> On Fri, May 11, Mihir Shirali wrote:
> > I did look up some examples for setting 503 - but all of them (as you've
> > indicated) seem based on src ip or src header. I'm guessing this is more
> > suitable for a DOS/DDOS attack? In our deployment, the likelihood of
> > getting one request from multiple clients is more than multiple requests
> > from a single client.
>
> Can you explain how/when(on what condition) you'd like to limit the number
> of requests and haproxy return 503 status to clients (429 seems more
> appropriate status code for this) ?
>
> If you just want haproxy to return 503 for all new requests when
> there're X number of sessions/connections/session rate then
> take a look at fe_conn, fe_req_rate, fe_sess_rate, be_conn and
> be_sess_rate
> (https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#7.3.2-fe_
> conn)
> so for example something like
> http-request deny deny_status 503 if { fe_req_rate gt 50 }
>
> > As an update the rate-limit directive has helped. However, the only
> problem
> > is that the client does not know that the server is busy and *could* time
> > out. It would be great if it were possible to somehow send a 503 out , so
> > the clients could retry after a random time.
>
> -Jarno
>
> --
> Jarno Huuskonen
>



--
Regards,
Mihir
Sorry, only registered users may post in this forum.

Click here to login