Crash with ss-20120310 and ss-20120311

Hi,

today I've experienced 3 crashes on 2 servers with haproxy. I've never
had any before so I thought I would just put a note up here.

20120310 crashed with:
Server 1
haproxy[3065] general protection ip:452ddf sp:7fff02906808 error:0 in
haproxy[400000+6e000]

Server 2
haproxy[30329]: segfault at a22312e314e ip 0000000000452db4 sp
00007fff0553dbb8 error 4 in haproxy[400000+6e000]

20120311 crashed with:
Server 1
haproxy[30497]: segfault at a223156 ip 0000000000452def sp
00007fff343786c8 error 4 in haproxy[400000+6e000]

I know they are snapshots, but since none of the snapshot ever crashed
on me before I thought it was worth noticing. Maybe it's already a known
issue or it might be a new bug....

Greets,

Sander

Hey,

I guess Willy would be keen to get the core dump and the haproxy
binary with its configuration.
You should try to reach him directly.

cheers


On Mon, Mar 12, 2012 at 4:03 PM, Sander Klein <[email protected]> wrote:
> Hi,
>
> today I've experienced 3 crashes on 2 servers with haproxy. I've never had
> any before so I thought I would just put a note up here.
>
> 20120310 crashed with:
> Server 1
> haproxy[3065] general protection ip:452ddf sp:7fff02906808 error:0 in
> haproxy[400000+6e000]
>
> Server 2
> haproxy[30329]: segfault at a22312e314e ip 0000000000452db4 sp
> 00007fff0553dbb8 error 4 in haproxy[400000+6e000]
>
> 20120311 crashed with:
> Server 1
> haproxy[30497]: segfault at a223156 ip 0000000000452def sp 00007fff343786c8
> error 4 in haproxy[400000+6e000]
>
> I know they are snapshots, but since none of the snapshot ever crashed on me
> before I thought it was worth noticing. Maybe it's already a known issue or
> it might be a new bug....
>
> Greets,
>
> Sander
>

Hi,

On Tue, Mar 13, 2012 at 07:05:36PM +0100, Baptiste wrote:
> Hey,
>
> I guess Willy would be keen to get the core dump and the haproxy
> binary with its configuration.
> You should try to reach him directly.

Yes Sander, please can you send me a core if you're still willing
to run it ? For this, you need to set "ulimit -c unlimited" before
starting haproxy and to disable "user", "group" and "chroot" settings
in the global section. I really understand it can be too much for you
depending on your usage.

If you could send me your config in private (without sensible info
such as stats password), it would immensely help.

BTW, if you cannot get a core but you have gdb on the machine, attaching
gdb to the running process is another solution :

# gdb
> attach <haproxy_pid>
> cont
Continuing

Then wait for gdb to return to prompt (indicating a crash) then please
do :
> bt

then you can send the output. Then you can exit with "quit" and restart
the process.

Thanks very much for reporting this, we'll do our best to get it fixed!
Willy

Hey Willy,

On 15.03.2012 07:53, Willy Tarreau wrote:
> Hi,
>
> On Tue, Mar 13, 2012 at 07:05:36PM +0100, Baptiste wrote:
>> Hey,
>>
>> I guess Willy would be keen to get the core dump and the haproxy
>> binary with its configuration.
>> You should try to reach him directly.
>
> Yes Sander, please can you send me a core if you're still willing
> to run it ? For this, you need to set "ulimit -c unlimited" before
> starting haproxy and to disable "user", "group" and "chroot" settings
> in the global section. I really understand it can be too much for you
> depending on your usage.

Of course we are using it in production ;-) But I do have a fail-over
situation so I think I dare running it again. It can take some time
before crashing.

> If you could send me your config in private (without sensible info
> such as stats password), it would immensely help.

Will do.

Do you care which snapshot I run?

Greets,

Sander

Hi Sander,

On Thu, Mar 15, 2012 at 08:44:43AM +0100, Sander Klein wrote:
> Of course we are using it in production ;-) But I do have a fail-over
> situation so I think I dare running it again. It can take some time
> before crashing.

OK, so be careful anyway!

> >If you could send me your config in private (without sensible info
> >such as stats password), it would immensely help.
>
> Will do.
>
> Do you care which snapshot I run?

Ideally the first one which exhibited the issue. BTW, do you know which
most recent one you used without the issue ? Eg: do you know if 20120306
has the same issue ?

Thanks a lot for your help,
Willy

On 15.03.2012 10:10, Willy Tarreau wrote:

>> Do you care which snapshot I run?
>
> Ideally the first one which exhibited the issue. BTW, do you know
> which
> most recent one you used without the issue ? Eg: do you know if
> 20120306
> has the same issue ?

I'm currently running 20120207 which doesn't give me any problems. I
upgraded straight to 20120310.

Sander