Welcome! Log In Create A New Profile

Advanced

haproxy+QAT memory usage very high under busy traffic

Posted by Julian Zhu 
Julian Zhu
haproxy+QAT memory usage very high under busy traffic
January 09, 2018 10:00AM
We are testing haproxy+QAT card(Intel QuickAssist-Technology) and find that the memory usage of haproxy+QAT is much higher than that of haproxy alone.
Under the same traffic (about 120K connections), haproxy alone only takes 6G memory while haproxy+QAT takes 36G.
The only difference in config is as below for haproxy+QAT:
.......
ssl-engine qat
ssl-mode-async
.......
and the memory is not released even if we terminate all haproxy processes until we reboot the server.

OS:
Centos-7

SW version:
haproxy-1.8.3
openssl-1.1.0e
QAT_Engine: 0.5.20
QAT1.7.Upstream.L.1.0.3_42

Does anyone encounter the same issue? Is there any way to debug or fix it?

Best Regards,
Julian
Willy Tarreau
Re: haproxy+QAT memory usage very high under busy traffic
January 09, 2018 03:40PM
Hi Julian,

On Tue, Jan 09, 2018 at 08:50:48AM +0000, Julian Zhu wrote:
> We are testing haproxy+QAT card(Intel QuickAssist-Technology) and find that the memory usage of haproxy+QAT is much higher than that of haproxy alone.
> Under the same traffic (about 120K connections), haproxy alone only takes 6G memory while haproxy+QAT takes 36G.
> The only difference in config is as below for haproxy+QAT:
> ......
> ssl-engine qat
> ssl-mode-async
> ......
> and the memory is not released even if we terminate all haproxy processes until we reboot the server.

Then this indicates that the leak (if any but it sounds like this) is in
the engine. You'd need to unload and reload the modules to see if it's a
real leak or just some extreme memory usage.

> OS:
> Centos-7
>
> SW version:
> haproxy-1.8.3
> openssl-1.1.0e
> QAT_Engine: 0.5.20
> QAT1.7.Upstream.L.1.0.3_42
>
> Does anyone encounter the same issue? Is there any way to debug or fix it?

Unfortunately at this point it will be entirely dependant on QAT, and I
don't really know what can be done there. Maybe there's a debug mode. You
should possibly check with other versions of the engine (higher or lower).
If you're still experiencing the leak with an up to date version, you
should contact your intel support so that they can take a look at this.

It's possible that haproxy triggers the leak (ie does something not cool
triggering an unusual code path leading to a leak in the engine), so they
may not be aware of it yet.

Hoping this helps,
Willy
Emeric Brun
Re: haproxy+QAT memory usage very high under busy traffic
January 09, 2018 04:10PM
Hi Julian,

On 01/09/2018 03:28 PM, Willy Tarreau wrote:
> Hi Julian,
>
> On Tue, Jan 09, 2018 at 08:50:48AM +0000, Julian Zhu wrote:
>> We are testing haproxy+QAT card(Intel QuickAssist-Technology) and find that the memory usage of haproxy+QAT is much higher than that of haproxy alone.
>> Under the same traffic (about 120K connections), haproxy alone only takes 6G memory while haproxy+QAT takes 36G.
>> The only difference in config is as below for haproxy+QAT:
>> ......
>> ssl-engine qat
>> ssl-mode-async
>> ......
>> and the memory is not released even if we terminate all haproxy processes until we reboot the server.
>
> Then this indicates that the leak (if any but it sounds like this) is in
> the engine. You'd need to unload and reload the modules to see if it's a
> real leak or just some extreme memory usage.
>
>> OS:
>> Centos-7
>>
>> SW version:
>> haproxy-1.8.3
>> openssl-1.1.0e
>> QAT_Engine: 0.5.20
>> QAT1.7.Upstream.L.1.0.3_42
>>
>> Does anyone encounter the same issue? Is there any way to debug or fix it?
>
> Unfortunately at this point it will be entirely dependant on QAT, and I
> don't really know what can be done there. Maybe there's a debug mode. You
> should possibly check with other versions of the engine (higher or lower).
> If you're still experiencing the leak with an up to date version, you
> should contact your intel support so that they can take a look at this.
>
> It's possible that haproxy triggers the leak (ie does something not cool
> triggering an unusual code path leading to a leak in the engine), so they
> may not be aware of it yet.
>
> Hoping this helps,
> Willy
>

There is no specific memory allocation in the code of haproxy to handle the async engines.

In addition, we are also using an other dfferent engine than QAT which is using a the "async engine mode" and we didn't notice any memory consumption.

So the issue seems related to the openssl ".so" used for the engine. As Willy told you, you should try a different QAT version.

R,
Emeric
Sorry, only registered users may post in this forum.

Click here to login