Welcome! Log In Create A New Profile

Advanced

Fwd: Haproxy-1.5.12 High memory usage problem

Posted by lizj3624 lizj3624 
lizj3624 lizj3624
Fwd: Haproxy-1.5.12 High memory usage problem
April 21, 2017 09:00AM
hi willy,


Our production environment is use Haproxy-1.5.12 as load balancer,
our haproxy loads a lot of configuration information, resulting in the
start up of memory to reach 300m, after running for a period of time, a
single process memory occupancy rate reached 1G,
We must reload haproxy, memory usage can be reduced by 300m,
Why run a period of time, while dealing with a large number of HTTP
request, memory soared from 300m to 1G?

Please help me analyze, the relevant information is as follows:

1、Our production environment:
OS: Linux-2.6.32-431.el6.x86_64
Haproxy:
version: 1.5.12, On the basis of that, we made some patches.
32 processes.
USE_OPENSSL.


2、pmap info:
Address Kbytes RSS Dirty Mode Mapping
0000000000400000 1120 804 0 r-x-- haproxy
0000000000718000 40 40 40 rw--- haproxy
0000000000722000 52 52 52 rw--- [ anon ]
00000000020f2000 333900 333892 333892 rw--- [ anon ]
0000000016705000 526312 526184 526184 rw--- [ anon ]
0000003a67e00000 128 24 0 r-x-- ld-2.12.so
0000003a6801f000 4 4 4 r---- ld-2.12.so
0000003a68020000 4 4 4 rw--- ld-2.12.so
0000003a68021000 4 4 4 rw--- [ anon ]
0000003a68200000 8 4 0 r-x-- libpcreposix.so.0.0.0
0000003a68202000 2044 0 0 ----- libpcreposix.so.0.0.0
0000003a68401000 4 4 4 rw--- libpcreposix.so.0.0.0
0000003a68600000 1576 492 0 r-x-- libc-2.12.so
0000003a6878a000 2048 0 0 ----- libc-2.12.so
0000003a6898a000 16 16 8 r---- libc-2.12.so
0000003a6898e000 4 4 4 rw--- libc-2.12.so
0000003a6898f000 20 20 20 rw--- [ anon ]
0000003a68a00000 92 20 0 r-x-- libpthread-2.12.so
0000003a68a17000 2048 0 0 ----- libpthread-2.12.so
0000003a68c17000 4 4 4 r---- libpthread-2.12.so
0000003a68c18000 4 4 4 rw--- libpthread-2.12.so
0000003a68c19000 16 4 4 rw--- [ anon ]
0000003a6aa00000 176 44 0 r-x-- libpcre.so.0.0.1
0000003a6aa2c000 2044 0 0 ----- libpcre.so.0.0.1
0000003a6ac2b000 4 4 4 rw--- libpcre.so.0.0.1
0000003a6c200000 28 4 0 r-x-- libcrypt-2.12.so
0000003a6c207000 2048 0 0 ----- libcrypt-2.12.so
0000003a6c407000 4 4 4 r---- libcrypt-2.12.so
0000003a6c408000 4 4 4 rw--- libcrypt-2.12.so
0000003a6c409000 184 0 0 rw--- [ anon ]
0000003a6d200000 460 0 0 r-x-- libfreebl3.so
0000003a6d273000 2044 0 0 ----- libfreebl3.so
0000003a6d472000 8 8 4 r---- libfreebl3.so
0000003a6d474000 4 4 4 rw--- libfreebl3.so
0000003a6d475000 16 12 12 rw--- [ anon ]
00007fd4b4eda000 8592 4100 4100 rw--- [ anon ]
00007fd4b573e000 172808 172808 172808 rw-s- zero (deleted)
00007fd4c0000000 136 8 8 rw--- [ anon ]
00007fd4c0022000 65400 0 0 ----- [ anon ]
00007fd4c4136000 17176 60 60 rw--- [ anon ]
00007fd4c51fc000 4 0 0 ----- [ anon ]
00007fd4c51fd000 10240 8 8 rwx-- [ anon ]
00007fd4c5bfd000 56 24 0 r-x-- libeng_dx.so.2.0.0
00007fd4c5c0b000 2044 0 0 ----- libeng_dx.so.2.0.0
00007fd4c5e0a000 8 8 8 rw--- libeng_dx.so.2.0.0
00007fd4c5e0c000 48 0 0 r-x-- libnss_files-2.12.so
00007fd4c5e18000 2048 0 0 ----- libnss_files-2.12.so
00007fd4c6018000 4 4 4 r---- libnss_files-2.12.so
00007fd4c6019000 4 4 4 rw--- libnss_files-2.12.so
00007fd4c6023000 12 12 12 rw--- [ anon ]
00007fd4c6026000 8 0 0 r-x-- libdl-2.12.so
00007fd4c6028000 2048 0 0 ----- libdl-2.12.so
00007fd4c6228000 4 4 4 r---- libdl-2.12.so
00007fd4c6229000 4 4 4 rw--- libdl-2.12.so
00007fd4c622a000 8 8 8 rw--- [ anon ]
00007fd4c622c000 2096 1044 0 r-x-- libcrypto.so.1.0.0
00007fd4c6438000 2044 0 0 ----- libcrypto.so.1.0.0
00007fd4c6637000 160 160 160 rw--- libcrypto.so.1.0.0
00007fd4c665f000 12 8 8 rw--- [ anon ]
00007fd4c6662000 404 268 0 r-x-- libssl.so.1.0.0
00007fd4c66c7000 2048 0 0 ----- libssl.so.1.0.0
00007fd4c68c7000 40 40 40 rw--- libssl.so.1.0.0
00007fd4c68d1000 4 4 4 rw--- [ anon ]
00007fd4c68db000 4 4 4 rw--- [ anon ]
00007fffed065000 84 32 32 rwx-- [ stack ]
00007fffed1ff000 4 4 0 r-x-- [ anon ]
ffffffffff600000 4 0 0 r-x-- [ anon ]
---------------- ------ ------ ------
total kB 1164028 1040280 1037536



3、show pools:
Dumping pools usage. Use SIGQUIT to flush them.
- Pool ptrcap (16 bytes) : 0 allocated (0 bytes), 0 used, 6528 users
[SHARED]
- Pool pipe (32 bytes) : 7 allocated (224 bytes), 7 used, 3 users [SHARED]
- Pool caphdr (48 bytes) : 0 allocated (0 bytes), 0 used, 2 users [SHARED]
- Pool channel (80 bytes) : 620 allocated (49600 bytes), 308 used, 1
users [SHARED]
- Pool task (112 bytes) : 4797 allocated (537264 bytes), 4659 used, 11
users [SHARED]
- Pool uniqueid (128 bytes) : 0 allocated (0 bytes), 0 used, 1 users
[SHARED]
- Pool capture (256 bytes) : 60 allocated (15360 bytes), 2 used, 1 users
[SHARED]
- Pool caphdr (304 bytes) : 0 allocated (0 bytes), 0 used, 10 users
[SHARED]
- Pool connection (352 bytes) : 338 allocated (118976 bytes), 162 used, 1
users [SHARED]
- Pool hdr_idx (416 bytes) : 309 allocated (128544 bytes), 152 used, 1
users [SHARED]
- Pool session (896 bytes) : 328 allocated (293888 bytes), 158 used, 1
users [SHARED]
- Pool requri (1024 bytes) : 30 allocated (30720 bytes), 1 used, 1 users
[SHARED]
- Pool buffer (9248 bytes) : 620 allocated (5733760 bytes), 308 used, 1
users [SHARED]
Total: 13 pools, 6908336 bytes allocated, 3658416 used.


Thanks!
Attachments:
open | download - Haproxy memory.png (72.3 KB)
Lukas Tribus
Re: Fwd: Haproxy-1.5.12 High memory usage problem
April 21, 2017 07:40PM
Hello lizj3624,


we are gonna need the configuration, at least redacted. Especially
important are timeouts and maxconn values. Also provide the output of
haproxy -vv.




Am 21.04.2017 um 08:56 schrieb lizj3624 lizj3624:
>
> Why run a period of time, while dealing with a large number of HTTP
> request, memory soared from 300m to 1G?

Because as connection slots are getting used, more memory is allocated.
What are your maxconn values?




>
> Please help me analyze, the relevant information is as follows:
>
> 1、Our production environment:
> OS: Linux-2.6.32-431.el6.x86_64
> Haproxy:
> version: 1.5.12, On the basis of that, we made some patches.

You probably wanna try a a current vanilla release then (although I can't
remember any memory leaks).

(not a release with 107 known bugs, 10 of the MAJOR and 44 of them MEDIUM:
[email protected]:~/haproxy-1.5$ git log --oneline v1.5.12.. | grep BUG | wc -l
107
[email protected]:~/haproxy-1.5$ git log --oneline v1.5.12.. | grep BUG | grep
MAJ | wc -l
10
[email protected]:~/haproxy-1.5$ git log --oneline v1.5.12.. | grep BUG | grep
MED | wc -l
44
[email protected]:~/haproxy-1.5$
)



Regards,
Lukas
Sorry, only registered users may post in this forum.

Click here to login