Welcome! Log In Create A New Profile

Advanced

Segfault in haproxy v1.8 with Lua

Posted by Hessam Mirsadeghi 
Hessam Mirsadeghi
Segfault in haproxy v1.8 with Lua
April 11, 2018 01:40AM
Hi,

I have a simple Lua http-response action script that leads to segmentation
fault in haproxy. The Lua script is a simple call to txn.res:forward(0).
A sample haproxy config and the Lua script files are attached. The backend
is simply an nginx instance which responds with 204 No Content.

The commit that introduces this problem is:
commit 8a5949f2d74c3a3a6c6da25449992c312b183ef3
BUG/MEDIUM: http: Switch the HTTP response in tunnel mode as earlier as
possible

Any ideas?

Best,
Seyed
Attachments:
open | download - foo.lua (111 bytes)
open | download - haproxy.cfg (2.5 KB)
Christopher Faulet
Re: Segfault in haproxy v1.8 with Lua
April 11, 2018 11:50AM
Le 11/04/2018 à 01:31, Hessam Mirsadeghi a écrit :
> Hi,
>
> I have a simple Lua http-response action script that leads to
> segmentation fault in haproxy. The Lua script is a simple call
> to txn.res:forward(0).
> A sample haproxy config and the Lua script files are attached. The
> backend is simply an nginx instance which responds with 204 No Content.
>
> The commit that introduces this problem is:
> commit 8a5949f2d74c3a3a6c6da25449992c312b183ef3
>     BUG/MEDIUM: http: Switch the HTTP response in tunnel mode as
> earlier as possible
>
> Any ideas?
>

Hi,

I'm unable to reproduce the segfault using your example. Could you
provide the output of "haproxy -vv" and the full backtrace of your
segfault ?

Regards,

--
Christopher Faulet
Hessam Mirsadeghi
Re: Segfault in haproxy v1.8 with Lua
April 11, 2018 04:20PM
Hi Christopher,

You're right; that segfault happens with the build at the faulty commit and
not later versions such as v1.8.5.
However, version v1.8.5 does segfault with the attached modified Lua
script. As far as I can tell, the problem arises after any call to
"txn.res:set()".

In the attached Lua script, if you remove the call to either of
"txn.res:set(txn.res:get())" or "txn.res:forward(txn.res:get_in_len())",
the segfault will disappear.
Also, when I only have a call to "txn.res:set(txn.res:get())" in the
script, haproxy becomes unresponsive to all but the first request on each
persistent connection. That is, something like "curl -sig localhost:80
localhost:80" will only get the response for the first request; the second
one times out on the existing connection and succeeds only on a a second
connection established by curl.

--------------- Here is the output of "haproxy -vv" for the new segfault
---------------

HA-Proxy version 1.8.5 2018/03/23
Copyright 2000-2018 Willy Tarreau <[email protected]>

Build options :
TARGET = linux26
CPU = generic
CC = gcc
CFLAGS = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement
-fwrapv -fno-strict-overflow -Wno-format-truncation -Wno-null-dereference
-Wno-unused-label
OPTIONS = USE_THREAD=1 USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1

Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with OpenSSL version : OpenSSL 1.1.0h-fips 27 Mar 2018
Running on OpenSSL version : OpenSSL 1.1.0h-fips 27 Mar 2018
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2
Built with Lua version : Lua 5.3.4
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT
IP_FREEBIND
Encrypted password support via crypt(3): yes
Built with multi-threading support.
Built with PCRE version : 8.41 2017-07-05
Running on PCRE version : 8.41 2017-07-05
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built without compression support (neither USE_ZLIB nor USE_SLZ are set).
Compression algorithms supported : identity("identity")
Built with network namespace support.

Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.

Available filters :
[SPOE] spoe
[COMP] compression
[TRACE] trace


Best,
Seyed


On Wed, Apr 11, 2018 at 5:46 AM, Christopher Faulet <[email protected]>
wrote:

> Le 11/04/2018 à 01:31, Hessam Mirsadeghi a écrit :
>
>> Hi,
>>
>> I have a simple Lua http-response action script that leads to
>> segmentation fault in haproxy. The Lua script is a simple call
>> to txn.res:forward(0).
>> A sample haproxy config and the Lua script files are attached. The
>> backend is simply an nginx instance which responds with 204 No Content.
>>
>> The commit that introduces this problem is:
>> commit 8a5949f2d74c3a3a6c6da25449992c312b183ef3
>> BUG/MEDIUM: http: Switch the HTTP response in tunnel mode as earlier
>> as possible
>>
>> Any ideas?
>>
>>
> Hi,
>
> I'm unable to reproduce the segfault using your example. Could you provide
> the output of "haproxy -vv" and the full backtrace of your segfault ?
>
> Regards,
>
> --
> Christopher Faulet
>
Christopher Faulet
Re: Segfault in haproxy v1.8 with Lua
April 12, 2018 02:50PM
Le 11/04/2018 à 16:11, Hessam Mirsadeghi a écrit :
> Hi Christopher,
>
> You're right; that segfault happens with the build at the faulty commit
> and not later versions such as v1.8.5.
> However, version v1.8.5 does segfault with the attached modified Lua
> script. As far as I can tell, the problem arises after any call to
> "txn.res:set()".
>
> In the attached Lua script, if you remove the call to either of
> "txn.res:set(txn.res:get())" or "txn.res:forward(txn.res:get_in_len())",
> the segfault will disappear.
> Also, when I only have a call to "txn.res:set(txn.res:get())" in the
> script, haproxy becomes unresponsive to all but the first request on
> each persistent connection. That is, something like "curl -sig
> localhost:80 localhost:80" will only get the response for the first
> request; the second one times out on the existing connection and
> succeeds only on a a second connection established by curl.

So, with this script, I have a segfault. But not only with HAProxy 1.8
and higher. It also crashes on HAProxy 1.7. Actually, It will crash on
all versions. This is a Lua problem not a HTTP one.

IMHO, "Channel.set" or "Channel.get" must never be used on HTTP
messages. These functions totally hijack the HTTP parser. There is an
attempt to restore the HTTP parser state. But this is only partially
done (the headers' analysis is done, but not all of the ensuing
analysis). So the result is undefined. It will seem to work for simpler
cases but will fail most of time. Same remarks are true for
"Channel.getline" and "Channel.append".

"Channel.forward" (and "Channel.send") must also never be used on HTTP
messages. When called, the messages analysis is in progress. The HTTP
parser expects to have all the headers at this stage. Mixing message
analysis and data forward is totally unexpected.

I will let Thierry (the Lua maintainer) decide what to do. But, I guess
he will restrict usage of all these functions to TCP streams only.

So now, maybe you can describe what you try to do. There is probably
another way to achieve it. I hope so, because it will never work this way.

--
Christopher Faulet
Hessam
Re: Segfault in haproxy v1.8 with Lua
April 12, 2018 03:00PM
Thanks for your response Christopher.

What I want is to overwrite each http response with another custom response. I'm not talking about manipulating headers only; I want to overwrite the whole response.

Thanks,
Seyed

> On Apr 12, 2018, at 8:44 AM, Christopher Faulet <[email protected]> wrote:
>
>> Le 11/04/2018 à 16:11, Hessam Mirsadeghi a écrit :
>> Hi Christopher,
>> You're right; that segfault happens with the build at the faulty commit and not later versions such as v1.8.5.
>> However, version v1.8.5 does segfault with the attached modified Lua script. As far as I can tell, the problem arises after any call to "txn.res:set()".
>> In the attached Lua script, if you remove the call to either of "txn.res:set(txn.res:get())" or "txn.res:forward(txn.res:get_in_len())", the segfault will disappear.
>> Also, when I only have a call to "txn.res:set(txn.res:get())" in the script, haproxy becomes unresponsive to all but the first request on each persistent connection. That is, something like "curl -sig localhost:80 localhost:80" will only get the response for the first request; the second one times out on the existing connection and succeeds only on a a second connection established by curl.
>
> So, with this script, I have a segfault. But not only with HAProxy 1.8 and higher. It also crashes on HAProxy 1.7. Actually, It will crash on all versions. This is a Lua problem not a HTTP one.
>
> IMHO, "Channel.set" or "Channel.get" must never be used on HTTP messages. These functions totally hijack the HTTP parser. There is an attempt to restore the HTTP parser state. But this is only partially done (the headers' analysis is done, but not all of the ensuing analysis). So the result is undefined. It will seem to work for simpler cases but will fail most of time. Same remarks are true for "Channel.getline" and "Channel.append".
>
> "Channel.forward" (and "Channel.send") must also never be used on HTTP messages. When called, the messages analysis is in progress. The HTTP parser expects to have all the headers at this stage. Mixing message analysis and data forward is totally unexpected.
>
> I will let Thierry (the Lua maintainer) decide what to do. But, I guess he will restrict usage of all these functions to TCP streams only.
>
> So now, maybe you can describe what you try to do. There is probably another way to achieve it. I hope so, because it will never work this way.
>
> --
> Christopher Faulet
Christopher Faulet
Re: Segfault in haproxy v1.8 with Lua
April 12, 2018 03:30PM
Le 12/04/2018 à 14:51, Hessam a écrit :
> Thanks for your response Christopher.
>
> What I want is to overwrite each http response with another custom response. I'm not talking about manipulating headers only; I want to overwrite the whole response.
>

AKAIK, using Lua it is no possible. I guess you want to alter the
response. If so, the only way to achieve this is to write a filter to do
what you want. But the current API is far to be easy to use, especially
for the body rewriting. We planned to make things simpler but it takes
time and needs internal (and important) refactoring first.

On the other hand, if you just need to generate a custom response on
some criteria of the request, you can use a Lua applet.

--
Christopher Faulet
Hessam Mirsadeghi
Re: Segfault in haproxy v1.8 with Lua
April 12, 2018 05:50PM
But using an applet on a request will prevent the request from being sent
to the backend servers; I still want backend servers to receive the request..

On Thu, Apr 12, 2018 at 9:16 AM, Christopher Faulet <[email protected]>
wrote:

> Le 12/04/2018 à 14:51, Hessam a écrit :
>
>> Thanks for your response Christopher.
>>
>> What I want is to overwrite each http response with another custom
>> response. I'm not talking about manipulating headers only; I want to
>> overwrite the whole response.
>>
>>
> AKAIK, using Lua it is no possible. I guess you want to alter the
> response. If so, the only way to achieve this is to write a filter to do
> what you want. But the current API is far to be easy to use, especially for
> the body rewriting. We planned to make things simpler but it takes time and
> needs internal (and important) refactoring first.
>
> On the other hand, if you just need to generate a custom response on some
> criteria of the request, you can use a Lua applet.
>
> --
> Christopher Faulet
>
Christopher Faulet
Re: Segfault in haproxy v1.8 with Lua
April 13, 2018 10:30AM
Le 12/04/2018 à 17:41, Hessam Mirsadeghi a écrit :
> But using an applet on a request will prevent the request from being
> sent to the backend servers; I still want backend servers to receive the
> request.
So, there is nothing much to do apart writing a filter. As I said, this
is not trivial at all. And unfortunately, to rewrite the entire
response, the current filter API will not be your friend. But,
definitely, this cannot be achieved using Lua. HAProxy was not designed
to do HTTP messages rewriting.

--
Christopher Faulet
Hessam Mirsadeghi
Re: Segfault in haproxy v1.8 with Lua
April 13, 2018 08:40PM
I see your point. Thanks for your help.

On Fri, Apr 13, 2018 at 4:20 AM, Christopher Faulet <[email protected]>
wrote:

> Le 12/04/2018 à 17:41, Hessam Mirsadeghi a écrit :
>
>> But using an applet on a request will prevent the request from being sent
>> to the backend servers; I still want backend servers to receive the request.
>>
> So, there is nothing much to do apart writing a filter. As I said, this is
> not trivial at all. And unfortunately, to rewrite the entire response, the
> current filter API will not be your friend. But, definitely, this cannot be
> achieved using Lua. HAProxy was not designed to do HTTP messages rewriting.
>
> --
> Christopher Faulet
>
Sorry, only registered users may post in this forum.

Click here to login