Stats for backend queue

Hey,

I have a dynamic backend with maxconn 80 with multiple servers.
Many times I can see on the haproxy stats page that servers on this backend
are reaching their maximum 80, but I don't see the number of requests
currently in queue. The maximum number I ever see is 80. Why is that? Can I
somehow see the number of requests in the queue?

Also, with a munin plugin that checks the HTTP page with ";csv", I see that
sometimes the graphs shows 400+ req/sec for this backend, which is not
possible since the maximum is 80...

Last, what is the difference between "Sessions" and "Session rate" ?
How can I tell when I need another dynamic backend server?

Thanks!
Bar.

On Sat, May 12, 2012 at 07:01:19PM +0300, Bar Ziony wrote:
> Hey,
>
> I have a dynamic backend with maxconn 80 with multiple servers.
> Many times I can see on the haproxy stats page that servers on this backend
> are reaching their maximum 80, but I don't see the number of requests
> currently in queue. The maximum number I ever see is 80. Why is that? Can I
> somehow see the number of requests in the queue?

The queue is split between servers and backend. In the servers' queue, you
only see the requests which absolutely need to be served by the given server
(due to persistence cookie or stick-tables). Otherwise the request lies in
the backend's queue so that it will be served by the first available server.
It's very normal not to have too many requests in the server's queue and have
more in the backend's queue.

> Also, with a munin plugin that checks the HTTP page with ";csv", I see that
> sometimes the graphs shows 400+ req/sec for this backend, which is not
> possible since the maximum is 80...
> Last, what is the difference between "Sessions" and "Session rate" ?

You seem to be really confusing concurrency and rate I'm afraid. Imagine a
highway, it's the same. Session rate is the number of cars you see pass an
observation point each second. Session concurrency is the number of parallel
lanes that are occupied at a given instant. If the traffic slows down, you
need more lanes to drain the same number of cars without slowing the rate
down. If your cars drive faster, you need less lanes for a same cars rate.

Regards,
Willy

> How can I tell when I need another dynamic backend server?

It's simple : observe the total queue size in a backend (backend + sum of
servers). Divide the number by the maxconn and it will tell you the number
of servers that would allow the requests to be processed without queuing.
Note that it's fine to have a bit of queueing, it saves you from buying
more hardware at the expense of a slightly delayed processing. You just
need to ensure the queue is not too deep. The average time spent in the
queue is the average queue size divided by the maxconn and multiplied by
the average response time.

So in order to get an idea :

srv1 has maxconn 80 and queue around 10
srv2 has maxconn 80 and queue around 10
backend has a queue around 100

The total queue is 120, which is the equivalent of 1.5 server. Let's say
you add a single server, you'll then have around 80 requests spread over
the last server, and 40 requests still in the queues. If your servers
exhibit an average response time of 50 ms, the average time spent in the
queue will be 40/80*50 ms = 25ms, so the total response time will increase
from 50ms to 75ms due to the queue. For many sites this will not be
noticeable and probably acceptable. Now if your site is already slow (eg:
2 seconds response time), adding 50% more will give you 3 seconds and your
users will clearly notice the difference.

That's why you first need to maintain the response times as low as possible
by limiting the maxconn, and only then estimate the number of servers needed
to keep the response time low.

Hoping this helps,
Willy

On Sat, May 12, 2012 at 6:01 PM, Bar Ziony <[email protected]> wrote:
> Hey,
>
> I have a dynamic backend with maxconn 80 with multiple servers.
> Many times I can see on the haproxy stats page that servers on this backend
> are reaching their maximum 80, but I don't see the number of requests
> currently in queue. The maximum number I ever see is 80. Why is that? Can I
> somehow see the number of requests in the queue?
>

Do you have any kind of persistence or affinity?
(cookie or predictive algorithm)
When persistence or affinity is in use, the requests are queued on the
server, otherwise, they are queue on the backend.
You can see the column Queue Max and check if you have a number per
server line or if each server line is 0 and a number is printed on the
backend line.

> Also, with a munin plugin that checks the HTTP page with ";csv", I see that
> sometimes the graphs shows 400+ req/sec for this backend, which is not
> possible since the maximum is 80...

You seem to misunderstand connection and request.
80 connections per server means 80 request in the mean time, there is
no relation to time.
If your server manage to process 400 req/s, it means each request
lasts, in average, 50ms.


>
> Last, what is the difference between "Sessions" and "Session rate" ?
> How can I tell when I need another dynamic backend server?
>



> Thanks!
> Bar.

Willy, thanks for your answer.


On Sat, May 12, 2012 at 7:21 PM, Willy Tarreau <[email protected]> wrote:

> On Sat, May 12, 2012 at 07:01:19PM +0300, Bar Ziony wrote:
> > Hey,
> >
> > I have a dynamic backend with maxconn 80 with multiple servers.
> > Many times I can see on the haproxy stats page that servers on this
> backend
> > are reaching their maximum 80, but I don't see the number of requests
> > currently in queue. The maximum number I ever see is 80. Why is that?
> Can I
> > somehow see the number of requests in the queue?
>
> The queue is split between servers and backend. In the servers' queue, you
> only see the requests which absolutely need to be served by the given
> server
> (due to persistence cookie or stick-tables). Otherwise the request lies in
> the backend's queue so that it will be served by the first available
> server.
> It's very normal not to have too many requests in the server's queue and
> have
> more in the backend's queue.
>

I now see the "Queue" part in the backend line and indeed I can see the
numbers rising on load! Thanks :)
I have no persistency and my backend servers are totally agnostic to what
user they're serving (user sessions are stored on a centeralized memcached).


> > Also, with a munin plugin that checks the HTTP page with ";csv", I see
> that
> > sometimes the graphs shows 400+ req/sec for this backend, which is not
> > possible since the maximum is 80...
> > Last, what is the difference between "Sessions" and "Session rate" ?
>
> You seem to be really confusing concurrency and rate I'm afraid. Imagine a
> highway, it's the same. Session rate is the number of cars you see pass an
> observation point each second. Session concurrency is the number of
> parallel
> lanes that are occupied at a given instant. If the traffic slows down, you
> need more lanes to drain the same number of cars without slowing the rate
> down. If your cars drive faster, you need less lanes for a same cars rate.
>

So session rate is the number of requests per second ? Why is it called
session then if it's really requests?

And "Sessions" is just plain sessions number, without caring for how much
of them were happening in 1 sec?


> Regards,
> Willy
>
> > How can I tell when I need another dynamic backend server?
>
> It's simple : observe the total queue size in a backend (backend + sum of
> servers). Divide the number by the maxconn and it will tell you the number
> of servers that would allow the requests to be processed without queuing.
> Note that it's fine to have a bit of queueing, it saves you from buying
> more hardware at the expense of a slightly delayed processing. You just
> need to ensure the queue is not too deep. The average time spent in the
> queue is the average queue size divided by the maxconn and multiplied by
> the average response time.
>
> So in order to get an idea :
>
> srv1 has maxconn 80 and queue around 10
> srv2 has maxconn 80 and queue around 10
> backend has a queue around 100
>
> The total queue is 120, which is the equivalent of 1.5 server. Let's say
> you add a single server, you'll then have around 80 requests spread over
> the last server, and 40 requests still in the queues. If your servers
> exhibit an average response time of 50 ms, the average time spent in the
> queue will be 40/80*50 ms = 25ms, so the total response time will increase
> from 50ms to 75ms due to the queue. For many sites this will not be
> noticeable and probably acceptable. Now if your site is already slow (eg:
> 2 seconds response time), adding 50% more will give you 3 seconds and your
> users will clearly notice the difference.
>

How can I know the average response time of my servers? haproxy provides
that data somewhere?

I have a max of 800 requests in the backend queue (none in the servers
queue since there is no persistence). Is that a lot ? :|

I also see 3,400 sessions in the frontend, and only ~100 in the dynamic
backend and 15 in the static backend (in the "cur" column). How is that
possible? So many requests are not valid, or sessions are kept and are not
for 1 request only ? :\ I don't understand that..


Thanks,
Bar.



> That's why you first need to maintain the response times as low as possible
> by limiting the maxconn, and only then estimate the number of servers
> needed
> to keep the response time low.
>
> Hoping this helps,
> Willy
>
>

On Sat, May 12, 2012 at 08:43:43PM +0300, Bar Ziony wrote:
> So session rate is the number of requests per second ? Why is it called
> session then if it's really requests?

You have the two. Initially in haproxy, you had no keepalive, so
1 req = 1 session. Now you have the numbers in the session column,
and if you pass your mouse over the number, you'll see the requests
too.

> And "Sessions" is just plain sessions number, without caring for how much
> of them were happening in 1 sec?

"sessions cur" is the number of concurrent sessions observed at the moment
it is reported. It is not related to any timing, it's what is observed at
one instant. To keep the analogy with the highway, it's how many lanes are
occupied at the precise instant you're taking the snapshot.

> How can I know the average response time of my servers? haproxy provides
> that data somewhere?

Yes you have each response time value in your logs :-)

> I have a max of 800 requests in the backend queue (none in the servers
> queue since there is no persistence). Is that a lot ? :|

It depends. If you're doing 800 reqs/s, you know that on average it will
take one second to drain these 800 requests, so it can be a lot. But if
you're facing an exceptional event, maybe you accept to delay requests
by up to 1s instead of seeing your servers die or stop responding.

> I also see 3,400 sessions in the frontend, and only ~100 in the dynamic
> backend and 15 in the static backend (in the "cur" column). How is that
> possible? So many requests are not valid, or sessions are kept and are not
> for 1 request only ? :\ I don't understand that..

It's because a connection is only forwarded to the backend once the client
has sent a request. And for some clients, sending a request takes some time
(eg: large requests, or simply because of poor network connectivity), so
you're always having more connections on the frontend than on the backend.
Also, haproxy closes the connection to the server as soon as it has the
last byte of the response, but it still forwards those data to the client
(so it acts as a TCP buffer). During this response buffering, the clients
are still connected to the frontend but the backend is already released.

This behaviour enables some multiplexing of the server connections, because
they never remain idle, even if the clients are slow to read the responses.

> I'm sorry but I didn't quite get what does Concurrency means.

It is the number of parallel sessions you have at one instant. When you do
"netstat -an | grep -c ESTAB", you get a number of concurrent connections.

> Connection/sec * response time ? Why is that = concurrency?

If you're not used to this, you need to draw it on paper to understand.

Imagine a road passing on a bridge. Your bridge is designed to support
100 cars. This is its concurrency limit. The response time is the time
a car spends on the bridge. The cars enter the bridge at a rate of 4
per second. If the cars last more than 25 seconds on the bridge, you'll
have more than 100 cars on your bridge and it might break. If you make
your cars run faster on the bridge, they will last there less time and
there will be less cars on it. If you are on a holiday season, you'll
get a higher "car rate" at the input of the bridge, and if they don't
run faster, you'll break the limit again.

> Here they are:
> net.ipv4.tcp_mem = 24372 32496 48744
> net.ipv4.tcp_wmem = 4096 16384 1039872
> net.ipv4.tcp_rmem = 4096 87380 1039872
>
> Are those valid?

So you have between 24372*4096 and 48744*4096 = 100..200 MB of RAM
assigned to the TCP stack, which is fine considering your VM size.
Your read and write buffers are correct too (the kernel automatically
adjusts them depending on the available memory).

However you have to be aware that a socket buffer needs at least 4kB
in each direction (hence why the min limit is 4kB), so 200 MB limits
you to 200/2/4 = 25k sockets, 20k of which can be on the frontend side
and 5k of which may be on the backend side.

> > You can reduce haproxy's memory usage by reducing buffer sizes this
> > way here :
> > tune.bufsize 8030
> > tune.maxrewrite 1030
>
> But would this hurt somehow?

It would only hurt if your average object size is larger than 7kB. And it
would not hurt that much, it would only eat a bit more CPU because haproxy
would have to perform twice the number of recv/send to forward data between
sockets. If you're using mostly large objects (more than twice the buffer
size), you can also enable "option splice-response" which will permit
forwarding between the two sockets without user-land copy. It becomes
insensible to the buffer size and generally saves some CPU cycles.

> I can increase the RAM if that will solve the
> problem! I just wonder how it is possible that haproxy is using so much
> RAM, when I didn't see so much RAM usage from my old single web server
> (nginx).

As I said, it all depends on how many connections you were having when
you observed the issue. With the 16kB default buffers, 20k conns * (2
buffers + ~1kB for internal structs) will take around 650 MB of RAM.
This plus the 200 MB for the kernel buffers almost reaches your VM
memory. I forgot to say that you also have conntrack to account for.
What is unsure is if you really reached those 20k conns.

> I don't want to configure stuff for a low-RAM machine if I actually need
> more RAM. We have no problem paying for a bigger VPS (but unfortunately we
> must stay on this VPS infrastructure).

OK, then let's tune more finely and increase the RAM size at the same
time : use the 8kB buffers in haproxy as I indicated above, and increase
your RAM to 1.5-2GB to be safe.

> Only nginx is running on this machine as well to terminate SSL, but it
> seems like haproxy is the one that consumes all memory. Nothing else is
> running on the machines besides syslog for haproxy and the machine itself,
> regular processes and munin plugins every 5 minutes (which are not causing
> any RAM issues)...

OK, so it should be easy to get the correct numbers once for all.

Regards,
Willy

Hi Willy :)

On Sat, May 12, 2012 at 10:06 PM, Willy Tarreau <[email protected]> wrote:

> On Sat, May 12, 2012 at 08:43:43PM +0300, Bar Ziony wrote:
> > So session rate is the number of requests per second ? Why is it called
> > session then if it's really requests?
>
> You have the two. Initially in haproxy, you had no keepalive, so
> 1 req = 1 session. Now you have the numbers in the session column,
> and if you pass your mouse over the number, you'll see the requests
> too.
>

Great, just saw that. and that's only for the frontend because only it
enables keepalive and then req/sec > session/sec ?


>
> > And "Sessions" is just plain sessions number, without caring for how much
> > of them were happening in 1 sec?
>
> "sessions cur" is the number of concurrent sessions observed at the moment
> it is reported. It is not related to any timing, it's what is observed at
> one instant. To keep the analogy with the highway, it's how many lanes are
> occupied at the precise instant you're taking the snapshot.
>
> > How can I know the average response time of my servers? haproxy provides
> > that data somewhere?
>
> Yes you have each response time value in your logs :-)
>

But can I get the average response time? Also, this is the response time of
the backend, or the full response time ?


>
> > I have a max of 800 requests in the backend queue (none in the servers
> > queue since there is no persistence). Is that a lot ? :|
>
> It depends. If you're doing 800 reqs/s, you know that on average it will
> take one second to drain these 800 requests, so it can be a lot. But if
> you're facing an exceptional event, maybe you accept to delay requests
> by up to 1s instead of seeing your servers die or stop responding.
>
> > I also see 3,400 sessions in the frontend, and only ~100 in the dynamic
> > backend and 15 in the static backend (in the "cur" column). How is that
> > possible? So many requests are not valid, or sessions are kept and are
> not
> > for 1 request only ? :\ I don't understand that..
>
> It's because a connection is only forwarded to the backend once the client
> has sent a request. And for some clients, sending a request takes some time
> (eg: large requests, or simply because of poor network connectivity), so
> you're always having more connections on the frontend than on the backend.
> Also, haproxy closes the connection to the server as soon as it has the
> last byte of the response, but it still forwards those data to the client
> (so it acts as a TCP buffer). During this response buffering, the clients
> are still connected to the frontend but the backend is already released.
>
> This behaviour enables some multiplexing of the server connections, because
> they never remain idle, even if the clients are slow to read the responses.
>
> OK, got why there are more frontend sessions than backend sessions. But is
it usual to see so much more?


> > I'm sorry but I didn't quite get what does Concurrency means.
>
> It is the number of parallel sessions you have at one instant. When you do
> "netstat -an | grep -c ESTAB", you get a number of concurrent connections.
>
> > Connection/sec * response time ? Why is that = concurrency?
>
> If you're not used to this, you need to draw it on paper to understand.
>
> Imagine a road passing on a bridge. Your bridge is designed to support
> 100 cars. This is its concurrency limit. The response time is the time
> a car spends on the bridge. The cars enter the bridge at a rate of 4
> per second. If the cars last more than 25 seconds on the bridge, you'll
> have more than 100 cars on your bridge and it might break. If you make
> your cars run faster on the bridge, they will last there less time and
> there will be less cars on it. If you are on a holiday season, you'll
> get a higher "car rate" at the input of the bridge, and if they don't
> run faster, you'll break the limit again.
>
> Cool, thanks :)


> > Here they are:
> > net.ipv4.tcp_mem = 24372 32496 48744
> > net.ipv4.tcp_wmem = 4096 16384 1039872
> > net.ipv4.tcp_rmem = 4096 87380 1039872
> >
> > Are those valid?
>
> So you have between 24372*4096 and 48744*4096 = 100..200 MB of RAM
> assigned to the TCP stack, which is fine considering your VM size.
> Your read and write buffers are correct too (the kernel automatically
> adjusts them depending on the available memory).
>
> However you have to be aware that a socket buffer needs at least 4kB
> in each direction (hence why the min limit is 4kB), so 200 MB limits
> you to 200/2/4 = 25k sockets, 20k of which can be on the frontend side
> and 5k of which may be on the backend side.
>

Didn't get much of that, I'll read about tcp_[rm]?mem some more :)


>
> > > You can reduce haproxy's memory usage by reducing buffer sizes this
> > > way here :
> > > tune.bufsize 8030
> > > tune.maxrewrite 1030
> >
> > But would this hurt somehow?
>
> It would only hurt if your average object size is larger than 7kB. And it
> would not hurt that much, it would only eat a bit more CPU because haproxy
> would have to perform twice the number of recv/send to forward data between
> sockets. If you're using mostly large objects (more than twice the buffer
> size), you can also enable "option splice-response" which will permit
> forwarding between the two sockets without user-land copy. It becomes
> insensible to the buffer size and generally saves some CPU cycles.
>

How can I know if I mostly use small or large objects?

>
> > I can increase the RAM if that will solve the
> > problem! I just wonder how it is possible that haproxy is using so much
> > RAM, when I didn't see so much RAM usage from my old single web server
> > (nginx).
>
> As I said, it all depends on how many connections you were having when
> you observed the issue. With the 16kB default buffers, 20k conns * (2
> buffers + ~1kB for internal structs) will take around 650 MB of RAM.
> This plus the 200 MB for the kernel buffers almost reaches your VM
> memory. I forgot to say that you also have conntrack to account for.
> What is unsure is if you really reached those 20k conns.
>

why 2 buffers? frontend + backend?

>
> > I don't want to configure stuff for a low-RAM machine if I actually need
> > more RAM. We have no problem paying for a bigger VPS (but unfortunately
> we
> > must stay on this VPS infrastructure).
>
> OK, then let's tune more finely and increase the RAM size at the same
> time : use the 8kB buffers in haproxy as I indicated above, and increase
> your RAM to 1.5-2GB to be safe.
>

I'm just now upgrading my LB's (one at a time, failing them over as I
upgrade) to 2GB RAM, but I don't want to make the buffers change before I
understand the thing with the object sizes - it's really not dangerous ?
It's a regular web app, some file uploads, lots of images downloads (small
and big)... I don't know the average size of a "dynamic" response.


>
> > Only nginx is running on this machine as well to terminate SSL, but it
> > seems like haproxy is the one that consumes all memory. Nothing else is
> > running on the machines besides syslog for haproxy and the machine
> itself,
> > regular processes and munin plugins every 5 minutes (which are not
> causing
> > any RAM issues)...
>
> OK, so it should be easy to get the correct numbers once for all.
>
> Regards,
> Willy
>
>
Thanks!
Bar.

Hi,

Le 12/05/2012 21:42, Bar Ziony a écrit :
> OK, got why there are more frontend sessions than backend sessions. But
> is it usual to see so much more?

In the configuration you provided, you didn't set any "timeout
http-keep-alive". It means that on your frontend, your keep-alive
timeout is equal to your client timeout : 90 seconds is maybe too long
for your server. Try to add a timeout with a small value.

--
Cyril Bonté

Oh, thanks.
Small value = 10 sec for example? :| What is an optimal keepalive timeout?

Thanks,
Bar.

On Sat, May 12, 2012 at 10:51 PM, Cyril Bonté <[email protected]> wrote:

> Hi,
>
> Le 12/05/2012 21:42, Bar Ziony a écrit :
>
> OK, got why there are more frontend sessions than backend sessions. But
>> is it usual to see so much more?
>>
>
> In the configuration you provided, you didn't set any "timeout
> http-keep-alive". It means that on your frontend, your keep-alive timeout
> is equal to your client timeout : 90 seconds is maybe too long for your
> server. Try to add a timeout with a small value.
>
> --
> Cyril Bonté
>
>
>

On Sat, May 12, 2012 at 11:04:49PM +0300, Bar Ziony wrote:
> Oh, thanks.
> Small value = 10 sec for example? :| What is an optimal keepalive timeout?

I like to use just a few seconds so that all objects from the same page are
fetched at once and the connection automatically closes after this. But your
mileage may vary.

Willy

Is there a benefit to allow a larger keepalive timeout so more than
resources from 1 page will be downloaded, or is it just best to create a
new connection for succeeding pages?

Willy, did you see my previous email in this correspondence? :)

Thanks,
Bar.

On Sat, May 12, 2012 at 11:18 PM, Willy Tarreau <[email protected]> wrote:

> On Sat, May 12, 2012 at 11:04:49PM +0300, Bar Ziony wrote:
> > Oh, thanks.
> > Small value = 10 sec for example? :| What is an optimal keepalive
> timeout?
>
> I like to use just a few seconds so that all objects from the same page are
> fetched at once and the connection automatically closes after this. But
> your
> mileage may vary.
>
> Willy
>
>

On Sat, May 12, 2012 at 10:42:24PM +0300, Bar Ziony wrote:
> > > How can I know the average response time of my servers? haproxy provides
> > > that data somewhere?
> >
> > Yes you have each response time value in your logs :-)
>
> But can I get the average response time?

By doing an average of what appears in your logs ! Oh if you want,
halog can do it for you per URL or per server (halog -ua / halog -srv).

> Also, this is the response time of
> the backend, or the full response time ?

These are the server response times, as well as the total times so you
know what part is server-side computing and what part is data transfer
(which might partially include the client's download time).

> OK, got why there are more frontend sessions than backend sessions. But is
> it usual to see so much more?

Sometimes yes. I know a website which has excellent response times (<< 1ms)
and very good connectivity between haproxy and the servers. For a long time
we believed there was a bug with the reporting of the session counts, until
we figured that since the time was 100* larger between the client and haproxy
than between haproxy and the server, it was normal to have that large a ratio
between the frontend and the backend!

But if your server's response time is larger than the client-to-haproxy RTT,
or if you have objects much larger than the buffers, the ratio between frontend
and backend will be closer to 1. There is no rule, it all depends on your
site. What you can say for sure is that the higher the ratio, the more
savings you're doing on backend servers since you don't need to add more
to stand the same load.

> > It would only hurt if your average object size is larger than 7kB. And it
> > would not hurt that much, it would only eat a bit more CPU because haproxy
> > would have to perform twice the number of recv/send to forward data between
> > sockets. If you're using mostly large objects (more than twice the buffer
> > size), you can also enable "option splice-response" which will permit
> > forwarding between the two sockets without user-land copy. It becomes
> > insensible to the buffer size and generally saves some CPU cycles.
> >
>
> How can I know if I mostly use small or large objects?

You have this in your logs too, but more generally, I think you're the one
who knows your site the best. You certainly have an idea of the object size
distribution on your site, and of the dynamic object sizes, otherwise it's
quite hard for you to guess how large your infrastructure has to be before
being hit by the load.

> > > I can increase the RAM if that will solve the
> > > problem! I just wonder how it is possible that haproxy is using so much
> > > RAM, when I didn't see so much RAM usage from my old single web server
> > > (nginx).
> >
> > As I said, it all depends on how many connections you were having when
> > you observed the issue. With the 16kB default buffers, 20k conns * (2
> > buffers + ~1kB for internal structs) will take around 650 MB of RAM.
> > This plus the 200 MB for the kernel buffers almost reaches your VM
> > memory. I forgot to say that you also have conntrack to account for.
> > What is unsure is if you really reached those 20k conns.
>
> why 2 buffers? frontend + backend?

No, just because of system call optimizations. You need at least one
recv() to parse a request (or response), and at least another one to
detect a close(). So each connection will have at least 2 recv() calls
per direction, and since each recv() can fill a full buffer, there is
generally nothing to save by enabling splice if you don't have enough
data to fill between 1 and 2 buffers.

> > > I don't want to configure stuff for a low-RAM machine if I actually need
> > > more RAM. We have no problem paying for a bigger VPS (but unfortunately
> > we
> > > must stay on this VPS infrastructure).
> >
> > OK, then let's tune more finely and increase the RAM size at the same
> > time : use the 8kB buffers in haproxy as I indicated above, and increase
> > your RAM to 1.5-2GB to be safe.
> >
>
> I'm just now upgrading my LB's (one at a time, failing them over as I
> upgrade) to 2GB RAM, but I don't want to make the buffers change before I
> understand the thing with the object sizes - it's really not dangerous ?

Really not. For instance, I've just run a benchmark here. On a 2.6 GHz
core2duo, with 30kB objects, I'm seeing :

- 13600 req/s and 3.42 Gbps with 16kB buffers
- 13440 req/s and 3.38 Gbps with 8kB buffers
- 15440 req/s and 3.90 Gbps with 8kB buffers + splice on 64kB pipes
- 16950 req/s and 4.26 Gbps with 8kB buffers + splice on 128kB pipes

So you see, the difference between 16 and 8kB is very minor compared to
what you can have with splicing (but splicing can vary a lot depending
on network cards and drivers).

> It's a regular web app, some file uploads, lots of images downloads (small
> and big)... I don't know the average size of a "dynamic" response.

Well, you have the network bandwidth of the dynamic servers at least (well I
hope you're monitoring at least this). You just divide this bandwidth with
these servers request rate and you'll have the average object size for these
servers.

It's very important for a web site to know its average response size. It
directly impacts page load time for users, that's the most important
indicator after the number of objects per page. You must absolutely
figure out these elements otherwise I can predict that your site will
never take off because you'll never be able to optimize it for your
users.

Regards,
Willy

On Sat, May 12, 2012 at 11:31:03PM +0300, Bar Ziony wrote:
> Is there a benefit to allow a larger keepalive timeout so more than
> resources from 1 page will be downloaded, or is it just best to create a
> new connection for succeeding pages?

It depends on your available memory. Ideally you'd keep connections open
long enough for two consecutive clicks to be performed in the same
connection, but if this means having 10* the amount of memory, it
probably isn't worth it.

> Willy, did you see my previous email in this correspondence? :)

Yes, just replied.

Willy

Thank you Willy!

I increased the RAM to 2GB and now I don't see the problem. I will change
the buffers size and report back as well.
Why did you recommend bufsize of 8130 and not 8192? Also, Why do I need to
change tune.maxrewrite ?

Thanks,
Bar.

On Sat, May 12, 2012 at 11:42 PM, Willy Tarreau <[email protected]> wrote:

> On Sat, May 12, 2012 at 11:31:03PM +0300, Bar Ziony wrote:
> > Is there a benefit to allow a larger keepalive timeout so more than
> > resources from 1 page will be downloaded, or is it just best to create a
> > new connection for succeeding pages?
>
> It depends on your available memory. Ideally you'd keep connections open
> long enough for two consecutive clicks to be performed in the same
> connection, but if this means having 10* the amount of memory, it
> probably isn't worth it.
>
> > Willy, did you see my previous email in this correspondence? :)
>
> Yes, just replied.
>
> Willy
>
>

On Sun, May 13, 2012 at 12:50:39AM +0300, Bar Ziony wrote:
> Thank you Willy!
>
> I increased the RAM to 2GB and now I don't see the problem. I will change
> the buffers size and report back as well.
> Why did you recommend bufsize of 8130 and not 8192?

As I explained in an earlier mail in this thread, 8030 is 5.5*1460 which is
the default TCP MSS. This ensures that TCP segments remain as much aligned
as possible. Also, not using the kill 8kB leaves some room for the malloc
heads in the same pages, which reduces memory fragmentation.

> Also, Why do I need to change tune.maxrewrite ?

because by default it's half of a buffer, so it would protect 4kB of the
8kB buffer, limiting your request and response headers to 4kB and preventing
to read more than that on the first segments. 1kB is more than enough for
the usual rewrites, so that leaves 7kB usable for the first read.

Willy