Welcome! Log In Create A New Profile

Advanced

Slab allocation starvation

Posted by Owen Tran 
Owen Tran
Slab allocation starvation
September 09, 2010 06:40AM
I'm running 4 servers with 96 Gb of memory and it has been a great
work horse for two years. We recently upgrade to version 1.4.5. Slab
reallocation won't work with 1.4.5 (at least I can't compile the flag
in on CentOS 5). What I'm experiencing is that once all the memory has
been allocated to all pages in different slab sizes, memcached will
not allocate more pages for the particular slab size, causing
premature evictions. This is particular evident since our application
has lots of large objects that require lots of pages.

From the wiki by dormando...

The previous chunk-size-multiple optimization is the default behavior.
You can switch it off, though, as a tradeoff in order to gain an
interesting, underdocumented feature: the ability to reassign slab
memory from one slabclass to another slabclass at runtime. To do so,
recompile with the ALLOW_SLABS_REASSIGN flag. After that, you can send
memcached messages like 'slabs reassign (src_slabclass)
(dest_slabclass)'. This feature can be useful, as long running
memcached instances will eventually allocate its max memory into
slabs. Unfortunately, slabs allocated early in memcached's process
life might over time be effectively in the wrong slabclass. Imagine,
for example, that you store session data in memcached, and memcached
has been up and running for months. Finally, you deploy a new version
of your application code, which stores more interesting information in
your sessions -- so your session sizes have grown. Suddenly, memcached
starts thrashing with huge amounts of evictions. What might have
happened is that since the session size grew, the slab allocator needs
to use a different slabclass. Most of the slabs are now sitting idle
and unused in the old slabclass. The usual solution is to just restart
memcached, unless you've turned on ALLOW_SLABS_REASSIGN.

So besides trying to fill out page sizes in memcached with quick
expiring seed data to allocate enough pages per slab, are there any
other solutions for getting around not being able to reallocate slabs
on the fly? I'm actually leaning towards moving towards ehCache server
RESTful implementation at this point, since slab reassignment doesn't
seem to be a big issue for most people making our use of memcached
probably an edge case best suited for another technology.

Thanks.
dormando
Re: Slab allocation starvation
October 01, 2010 03:00AM
We do intend to fix the slab reassignment issue, sorry :/

The old ALLOW_SLABS_REASSIGN code has some mythical bugs in it, which is
why it's presently unused.

Hopefully not that much longer...

On Wed, 8 Sep 2010, Owen Tran wrote:

> I'm running 4 servers with 96 Gb of memory and it has been a great
> work horse for two years. We recently upgrade to version 1.4.5. Slab
> reallocation won't work with 1.4.5 (at least I can't compile the flag
> in on CentOS 5). What I'm experiencing is that once all the memory has
> been allocated to all pages in different slab sizes, memcached will
> not allocate more pages for the particular slab size, causing
> premature evictions. This is particular evident since our application
> has lots of large objects that require lots of pages.
>
> From the wiki by dormando...
>
> The previous chunk-size-multiple optimization is the default behavior.
> You can switch it off, though, as a tradeoff in order to gain an
> interesting, underdocumented feature: the ability to reassign slab
> memory from one slabclass to another slabclass at runtime. To do so,
> recompile with the ALLOW_SLABS_REASSIGN flag. After that, you can send
> memcached messages like 'slabs reassign (src_slabclass)
> (dest_slabclass)'. This feature can be useful, as long running
> memcached instances will eventually allocate its max memory into
> slabs. Unfortunately, slabs allocated early in memcached's process
> life might over time be effectively in the wrong slabclass. Imagine,
> for example, that you store session data in memcached, and memcached
> has been up and running for months. Finally, you deploy a new version
> of your application code, which stores more interesting information in
> your sessions -- so your session sizes have grown. Suddenly, memcached
> starts thrashing with huge amounts of evictions. What might have
> happened is that since the session size grew, the slab allocator needs
> to use a different slabclass. Most of the slabs are now sitting idle
> and unused in the old slabclass. The usual solution is to just restart
> memcached, unless you've turned on ALLOW_SLABS_REASSIGN.
>
> So besides trying to fill out page sizes in memcached with quick
> expiring seed data to allocate enough pages per slab, are there any
> other solutions for getting around not being able to reallocate slabs
> on the fly? I'm actually leaning towards moving towards ehCache server
> RESTful implementation at this point, since slab reassignment doesn't
> seem to be a big issue for most people making our use of memcached
> probably an edge case best suited for another technology.
>
> Thanks.
>
Sorry, only registered users may post in this forum.

Click here to login