Welcome! Log In Create A New Profile

Advanced

regtest lua/b00002.vtc fails with 1.9-dev2 / master

Posted by PiBa-NL 
PiBa-NL
regtest lua/b00002.vtc fails with 1.9-dev2 / master
September 13, 2018 05:20PM
Hi List, Olivier,

Just tried another run of regtests on FreeBSD, and found that
lua/b00002.vtc fails (coredump, gdb bt below) with todays snapshot:
HA-Proxy version 1.9-dev2-253006d 2018/09/12

While on yesterday's snapshot it works properly, below some tests on
different commits, and while the 'log' and 'connection' seem to have
been fixed, the 'lua' check fails also with todays latest commit.

MINOR: h1: parse the Connection header field        98f5cf7
/lua/b00002.vtc FAILED
.....
MINOR: conn_streams: Remove wait_list from conn_streams. 7138455
/lua/b00002.vtc FAILED (0.029) exit=2 ---- h1    0.0 Bad exit status:
0x008b exit 0x0 signal 11 core 128
MINOR: checks: Give checks their own wait_list. 26e1a8f /lua/b00002.vtc
FAILED
MEDIUM: stream_interfaces: Starts receiving from the... f653528
/log/b00000.vtc FAILED
MEDIUM: mux_h2: Revamp the send path when blocking. 8ae735d
/log/b00000.vtc FAILED  /connection/b00000.vtc TIMED OUT (kill -9)
MINOR: connections: Add a "handle" field to wait_list. cb1f49f
log/b00000.vtc FAILED
MEDIUM: stream_interface: Make recv() subscribe when...
MEDIUM: h2: Don't use a wake() method anymore. 7505f94 log/b00000.vtc FAILED
MEDIUM: h2: always subscribe to receive if allowed. a1411e
log/b00000.vtc FAILED
MINOR: h2: Let user of h2_recv() and h2_send() know... d4dd22
log/b00000.vtc FAILED
MEDIUM: connections: Get rid of the recv() method. af4021 log/b00000.vtc
FAILED, ---- s1    2.0 HTTP rx failed (fd:5 read: Connection reset by peer)
MEDIUM: connections/mux: Add a recv and a send+recv... 4cf7fb OK

Did something go astray?

p.s. should reg-tests maybe get run before the -dev releases get tagged?
(or is that already done, and this could be a FreeBSD specific issue and
as such not have been spotted?)

Regards,

PiBa-NL (Pieter)

#0  si_cs_recv (cs=0x802c71180) at src/stream_interface.c:1163
1163            if (conn->xprt->rcv_pipe && conn->mux->rcv_pipe &&
(gdb) bt full
#0  si_cs_recv (cs=0x802c71180) at src/stream_interface.c:1163
        conn = (struct connection *) 0x802c63700
        si = (struct stream_interface *) 0x802c74848
        ic = (struct channel *) 0x802c74570
        ret = 0
        max = 0
        cur_read = 0
        read_poll = 4
#1  0x000000000054e11f in si_cs_io_cb (t=0x0, ctx=0x802c74848, state=0)
at src/stream_interface.c:741
        si = (struct stream_interface *) 0x802c74848
        cs = (struct conn_stream *) 0x802c71180
        ret = 0
#2  0x00000000004b717d in process_stream (t=0x802c79280,
context=0x802c74500, state=257) at src/stream.c:1660
        srv = (struct server *) 0x5a5c8b
        s = (struct stream *) 0x802c74500
        sess = (struct session *) 0x802c6a1c0
        rqf_last = 0
        rpf_last = 4500720
        rq_prod_last = 8
        rq_cons_last = 46544272
        rp_cons_last = 0
        rp_prod_last = 0
        req_ana_back = 0
        req = (struct channel *) 0x802c74510
        res = (struct channel *) 0x802c74570
        si_f = (struct stream_interface *) 0x802c747f8
        si_b = (struct stream_interface *) 0x802c74848
#3  0x00000000005a5629 in process_runnable_tasks () at src/task.c:381
        t = (struct task *) 0x802c79280
        state = 257
        ctx = (void *) 0x802c74500
        process = (struct task *(*)(struct task *, void *, unsigned
short)) 0x4b70b0 <process_stream>
        t = (struct task *) 0x802c79280
---Type <return> to continue, or q <return> to quit---
        max_processed = 199
#4  0x0000000000518d02 in run_poll_loop () at src/haproxy.c:2511
        next = 0
        exp = -750267441
#5  0x0000000000515930 in run_thread_poll_loop (data=0x8024873d4) at
src/haproxy.c:2576
        start_lock = {lock = 0, info = {owner = 0, waiters = 0,
last_location = {function = 0x0, file = 0x0, line = 0}}}
        ptif = (struct per_thread_init_fct *) 0x8c7838
        ptdf = (struct per_thread_deinit_fct *) 0x800f1d7cc
#6  0x0000000800f18bc5 in pthread_create () from /lib/libthr.so.3
No symbol table info available.
#7  0x0000000000000000 in ?? ()
No symbol table info available.
Current language:  auto; currently minimal
(gdb) info thread
  3 process 101654  0x0000000801e11e3a in _kevent () from /lib/libc.so.7
  2 process 101287  0x0000000801e11e3a in _kevent () from /lib/libc.so.7
* 1 process 101624  si_cs_recv (cs=0x802c71180) at
src/stream_interface.c:1163
Willy Tarreau
Re: regtest lua/b00002.vtc fails with 1.9-dev2 / master
September 14, 2018 09:50AM
Hi Pieter,

On Thu, Sep 13, 2018 at 05:13:43PM +0200, PiBa-NL wrote:
> Just tried another run of regtests on FreeBSD, and found that lua/b00002.vtc
> fails (coredump, gdb bt below) with todays snapshot: HA-Proxy version
> 1.9-dev2-253006d 2018/09/12

Yesterday we've looked at this with Olivier and it raised some questions
between us regarding the nature of certain events that need to be reported
upwards. Here the issue is related to the sequencing of the connect_server()
call and the subscription to try to receive. If we do it the current way,
we read from a non-existent mux and if we do it the other way around we'll
lose connect events if there's no data to send. We have some plans for this
that we're discussing today in front of the white board. Getting rid of ~15
years of accumulated hacks in the connection layers is amazingly difficult,
and much more than what it looks like before trying to do it!

I hope that today we'll have clear ideas about a very short-term workaround
and how we want to address this situation from an architecture perspective.

> p.s. should reg-tests maybe get run before the -dev releases get tagged? (or
> is that already done, and this could be a FreeBSD specific issue and as such
> not have been spotted?)

It's my fault. Yes I'm supposed to run these tests. But after finally
managed to collect all the long-awaited code a few minutes before I really
had to leave, I was impatient to push it out so that we can all resync. And
not having vtc anymore after I rebooted my PC (last time it was built in
/dev/shm and I still didn't take the time to rebuild it) didn't give me
enough motivation to add this extra task. The process will improve over
time, we need it to become more natural and usual. You don't kill 18
years of bad habits in a few weeks! Just to be clear, spotting bugs at
this moment will not prevent me from emitting a dev release, but at least
I'll report them in the changelog so that nobody wastes their time ;-)

Cheers,
Willy
Sorry, only registered users may post in this forum.

Click here to login