Hi :)
We have a small farm of 8 memcached boxes (Dell R210s, Debian Lenny
x64) with 8GB RAM. I've just noticed that a) there is swap configured
on these machines (probably my bad) and b) many gigabytes of swap is
being used.
memcache09:~# ps faux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME
COMMAND
[kernel processes omitted]
root 1 0.0 0.0 10312 296 ? Ss Feb18 1:19 init
[2]
root 1013 0.0 0.0 16920 236 ? S<s Feb18 0:00 udevd
--daemon
root 1943 0.0 0.0 126172 1532 ? Sl Feb18 0:23 /usr/
sbin/rsyslogd -c3
root 1954 0.0 0.0 3796 228 ? Ss Feb18 0:00 /usr/
sbin/acpid
root 1970 0.0 0.0 48864 516 ? Ss Feb18 0:00 /usr/
sbin/sshd
root 10929 0.0 0.0 66068 3112 ? Ss 13:10 0:00 \_
sshd:
[email protected]/0
root 10931 0.0 0.0 18756 1836 pts/0 Ss 13:10 0:00
\_ -bash
root 10962 0.0 0.0 15980 1052 pts/0 R+ 13:27
0:00 \_ ps faxu
root 2011 0.0 0.0 19804 372 ? Ss Feb18 0:09 /usr/
sbin/cron
root 2028 0.0 0.0 3796 208 tty1 Ss+ Feb18 0:00 /sbin/
getty 38400 tty1
root 2029 0.0 0.0 3796 208 tty2 Ss+ Feb18 0:00 /sbin/
getty 38400 tty2
root 2030 0.0 0.0 3796 208 tty3 Ss+ Feb18 0:00 /sbin/
getty 38400 tty3
root 2032 0.0 0.0 3796 208 tty4 Ss+ Feb18 0:00 /sbin/
getty 38400 tty4
root 2034 0.0 0.0 3796 208 tty5 Ss+ Feb18 0:00 /sbin/
getty 38400 tty5
root 2036 0.0 0.0 3796 208 tty6 Ss+ Feb18 0:00 /sbin/
getty 38400 tty6
nobody 31164 1.0 37.4 3521308 3072284 ? Sl Apr28 2413:10 /usr/
bin/memcached -m 2928 -p 11212 -u nobody -c 8192
nobody 31179 0.0 1.7 429080 144436 ? Ssl Apr28 50:44 /usr/
bin/memcached -m 192 -p 11213 -u nobody -c 8192
nobody 31381 0.2 54.6 5427948 4478588 ? Sl Apr28 594:38 /usr/
bin/memcached -m 4680 -p 11211 -u nobody -c 8192
memcache09:~# free
total used free shared buffers
cached
Mem: 8199492 8099508 99984 0 16052
16900
-/+ buffers/cache: 8066556 132936
Swap: 9936160 3620640 6315520
The 'swappiness' proc knob was on its default of 60, but I just don't
get it... the VIRT columns of all running process don't add up to
anywhere near (physical RAM + swap in use) (they add to under 10GB)
It can't even be disk cache because there's only 660MB in use on the
rootfs...
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda1 230528596 669032 218149388 1% /
I fully expect if I restart the 3 memcache processes (not something I
want to do since it will hurt live performance) the swap will zero,
but that doesn't solve the root cause.
Any ideas would be warmly welcomed! :)
Cheers,
Gavin