Gigs of swap being used, but why?

Hi :)

We have a small farm of 8 memcached boxes (Dell R210s, Debian Lenny
x64) with 8GB RAM. I've just noticed that a) there is swap configured
on these machines (probably my bad) and b) many gigabytes of swap is
being used.

memcache09:~# ps faux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME
COMMAND
[kernel processes omitted]
root 1 0.0 0.0 10312 296 ? Ss Feb18 1:19 init
[2]
root 1013 0.0 0.0 16920 236 ? S<s Feb18 0:00 udevd
--daemon
root 1943 0.0 0.0 126172 1532 ? Sl Feb18 0:23 /usr/
sbin/rsyslogd -c3
root 1954 0.0 0.0 3796 228 ? Ss Feb18 0:00 /usr/
sbin/acpid
root 1970 0.0 0.0 48864 516 ? Ss Feb18 0:00 /usr/
sbin/sshd
root 10929 0.0 0.0 66068 3112 ? Ss 13:10 0:00 \_
sshd: [email protected]/0
root 10931 0.0 0.0 18756 1836 pts/0 Ss 13:10 0:00
\_ -bash
root 10962 0.0 0.0 15980 1052 pts/0 R+ 13:27
0:00 \_ ps faxu
root 2011 0.0 0.0 19804 372 ? Ss Feb18 0:09 /usr/
sbin/cron
root 2028 0.0 0.0 3796 208 tty1 Ss+ Feb18 0:00 /sbin/
getty 38400 tty1
root 2029 0.0 0.0 3796 208 tty2 Ss+ Feb18 0:00 /sbin/
getty 38400 tty2
root 2030 0.0 0.0 3796 208 tty3 Ss+ Feb18 0:00 /sbin/
getty 38400 tty3
root 2032 0.0 0.0 3796 208 tty4 Ss+ Feb18 0:00 /sbin/
getty 38400 tty4
root 2034 0.0 0.0 3796 208 tty5 Ss+ Feb18 0:00 /sbin/
getty 38400 tty5
root 2036 0.0 0.0 3796 208 tty6 Ss+ Feb18 0:00 /sbin/
getty 38400 tty6
nobody 31164 1.0 37.4 3521308 3072284 ? Sl Apr28 2413:10 /usr/
bin/memcached -m 2928 -p 11212 -u nobody -c 8192
nobody 31179 0.0 1.7 429080 144436 ? Ssl Apr28 50:44 /usr/
bin/memcached -m 192 -p 11213 -u nobody -c 8192
nobody 31381 0.2 54.6 5427948 4478588 ? Sl Apr28 594:38 /usr/
bin/memcached -m 4680 -p 11211 -u nobody -c 8192

memcache09:~# free
total used free shared buffers
cached
Mem: 8199492 8099508 99984 0 16052
16900
-/+ buffers/cache: 8066556 132936
Swap: 9936160 3620640 6315520

The 'swappiness' proc knob was on its default of 60, but I just don't
get it... the VIRT columns of all running process don't add up to
anywhere near (physical RAM + swap in use) (they add to under 10GB)

It can't even be disk cache because there's only 660MB in use on the
rootfs...

Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda1 230528596 669032 218149388 1% /

I fully expect if I restart the 3 memcache processes (not something I
want to do since it will hurt live performance) the swap will zero,
but that doesn't solve the root cause.

Any ideas would be warmly welcomed! :)

Cheers,
Gavin

> The 'swappiness' proc knob was on its default of 60, but I just don't
> get it... the VIRT columns of all running process don't add up to
> anywhere near (physical RAM + swap in use) (they add to under 10GB)
>
> It can't even be disk cache because there's only 660MB in use on the
> rootfs...
>
> Filesystem 1K-blocks Used Available Use% Mounted on
> /dev/sda1 230528596 669032 218149388 1% /
>

I don't want to dive deep into a discussion about how Linux swap works
and such. I will leave that for a Linux mailing list and people who have
the full deep understanding. I will only say that, in my experience, if
you give Linux swap it is going to use it for something sometime.

The big issue I see here is that you have allocated 7.5GB to memcached
on a system with only 8GB or RAM. That leaves only 500MB for the system
AND memcached overhead. For example, on one of my servers, I have
memcached set to use 512MB of RAM. Its RSS in ps is 590712. It is using
80MB of memory for other uses. This includes the application itself,
connection overhead, etc. I like to keep 1GB of free memory on all my
systems for the system.

> I fully expect if I restart the 3 memcache processes (not something I
> want to do since it will hurt live performance) the swap will zero,
> but that doesn't solve the root cause.

Well, that is a whole other discussion. Restarting one memcached daemon
should never be a problem assuming the rest of the system is up and
running. If it is, you may want to reconsider your cache architecture.

--

Brian.
--------
http://brian.moonspot.net/

Hi Brian,

Thanks for that; I'll try pulling back the RAM usage on each instance
a little to clear some more headroom, and disable swap on one of the
mc nodes so I can monitor for OOM process kills.

The RSS numbers still don't add up to the amount of swap being used,
so I'm still curious about that - but in the meantime I want to make
sure memcache pumps data out rather than chugging with a cheap SATA
disk :)

gdh

On 10/8/10 3:55 PM, Gavin Hamill wrote:
> Hi Brian,
>
> Thanks for that; I'll try pulling back the RAM usage on each instance
> a little to clear some more headroom, and disable swap on one of the
> mc nodes so I can monitor for OOM process kills.
>
> The RSS numbers still don't add up to the amount of swap being used,
> so I'm still curious about that - but in the meantime I want to make
> sure memcache pumps data out rather than chugging with a cheap SATA
> disk :)
>
> gdh

RSS is the size of the memory in physical RAM. It does not include swap.
VSZ is virtual space which includes RSS and other factors, but may not
include all of swap. There is no column in ps that reflects all swap
usage afaik.

--

Brian.
--------
http://brian.moonspot.net/

> On 10/8/10 3:55 PM, Gavin Hamill wrote:
> > Hi Brian,
> >
> > Thanks for that; I'll try pulling back the RAM usage on each instance
> > a little to clear some more headroom, and disable swap on one of the
> > mc nodes so I can monitor for OOM process kills.
> >
> > The RSS numbers still don't add up to the amount of swap being used,
> > so I'm still curious about that - but in the meantime I want to make
> > sure memcache pumps data out rather than chugging with a cheap SATA
> > disk :)
> >
> > gdh
>
> RSS is the size of the memory in physical RAM. It does not include swap. VSZ
> is virtual space which includes RSS and other factors, but may not include all
> of swap. There is no column in ps that reflects all swap usage afaik.

You need to dig through /proc/slabinfo to figure what all the memory is...

I prefer setting swappiness to 0 and vfs_cache_pressure to 200 or
similar... ie; really really deprioritize filecache over application
memory.

I assume you'll find a bunch of socket memory used in the slabinfo. People
tend to forget that TCP connections use buffers.

-Dormando

Ta for the suggestions - much appreciated. I'll work with swappiness
and vfs_cache_pressure and see if I can make any sense from the slab
cache :)

slabtop seems to suggest only 16MB is in use for TCP buffers, so the
answer is likely a little more involved.

OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE
NAME
8148 7736 94% 1.62K 2037 4 16296K TCP

Cheers,
Gavin.