Welcome! Log In Create A New Profile

Advanced

stats socket, show errors always blank

Posted by Jeremy 
Jeremy
stats socket, show errors always blank
September 02, 2009 12:30AM
I've set up the stats socket and get results ok running "show info" or "show
stats" etc, but for some reason "show errors" is always blank. Any tips to
getting "show errors" to return the sexy output like is shown at the very
bottom of http://haproxy.1wt.eu/download/1.3/doc/configuration.txt ?

Basically we're still having an intermittent problem with a client that is
getting 502 Bad Gateway errors, most likely due to some bad characters in
their HTTP headers (specifically cookies we think), but I'm unable to tell
the client exactly what is wrong since "show errors" is not showing
anything. They want to track down and fix whatever's wrong with their
website, so they have not wanted us to enable the options to accept invalid
http requests/responses just yet.

This particular load balancer just has 1 site running on it with a couple
backends and one front end. I've also tried doing show errors <IID> using
the various IID's in use but it always just comes back with nothing.

The site in question has a ton of css includes, images, javascript files
etc... The only thing I can think of is maybe once a valid request passes
through HAProxy "show errors" is cleared, so we somehow need them to stop
making any more requests once they run into 502 bad gateway until we can
capture the "show errors" output?

I have spammed "show errors" as fast as I possibly can while they're doing
some testing but I've yet to actually see it return anything.

Thanks for any tips!
Willy Tarreau
Re: stats socket, show errors always blank
September 02, 2009 11:00PM
On Tue, Sep 01, 2009 at 05:26:54PM -0500, Jeremy wrote:
> I've set up the stats socket and get results ok running "show info" or "show
> stats" etc, but for some reason "show errors" is always blank. Any tips to
> getting "show errors" to return the sexy output like is shown at the very
> bottom of http://haproxy.1wt.eu/download/1.3/doc/configuration.txt ?

It simply means you have had no request nor response error. If you want
to try to get one, send it an invalid request using telnet, such as :

GET / / HTTP/1.0

You will find it in your "show errors" report.

> Basically we're still having an intermittent problem with a client that is
> getting 502 Bad Gateway errors, most likely due to some bad characters in
> their HTTP headers (specifically cookies we think), but I'm unable to tell
> the client exactly what is wrong since "show errors" is not showing
> anything.

Do you see it in your logs too ? It is possible that another proxy between
the client and haproxy returns the 502. Also, it is possible that haproxy
sends a 502 because the application randomly kills sessions without responding.

> They want to track down and fix whatever's wrong with their
> website, so they have not wanted us to enable the options to accept invalid
> http requests/responses just yet.

You can congratulate them, it's extremely rare these days to see
people who want to comply with standards !

> This particular load balancer just has 1 site running on it with a couple
> backends and one front end. I've also tried doing show errors <IID> using
> the various IID's in use but it always just comes back with nothing.
>
> The site in question has a ton of css includes, images, javascript files
> etc... The only thing I can think of is maybe once a valid request passes
> through HAProxy "show errors" is cleared,

Not at all. It was designed exactly for the case you're trying to
troubleshoot, which means a hosted customer having some bad characters
once in a while without the ability to find which ones. So harpoxy will
capture a bad request/response and keep it until you restart it or until
a new invalid request/response replaces it, even one year after if needed.

> so we somehow need them to stop
> making any more requests once they run into 502 bad gateway until we can
> capture the "show errors" output?

No, not at all.

> I have spammed "show errors" as fast as I possibly can while they're doing
> some testing but I've yet to actually see it return anything.

Not needed either :-) However, I suggest to send it one last time before
restarting haproxy, just in case you trap one.

But quite honnestly, if you wait a long time without any, the application
is fine IMHO. However, it's very likely that it breaks sessions. Check
your logs for "SH" flags or even for 502 status codes. I'm sure your
customer's server is randomly sending TCP RST packets in response to
requests. One user reported such an issue with round-robin link load-
balancing (which reorders packets) and made a windows server nervously
send RST packets, thus aborting connections.

Regards,
Willy
Sorry, only registered users may post in this forum.

Click here to login