Welcome! Log In Create A New Profile

Advanced

flush_all and evictions

Posted by mjy 
mjy
flush_all and evictions
July 08, 2010 05:20PM
Hi,

we are trying to implement better memcached monitoring and have come
across this problem with 1.4.5:

- after flush_all on a filled (long-running) memcached, we get
evictions immediately (within seconds, none of the buckets should fill
up that quickly)

This would mean that either flushed items count toward evictions when
replaced (even though they should not be valid items after flush_all
and evictions should count only valid items that were replaced) or
that somehow flush_all does not work as expected, or that we have a
pathetically bad distribution of item sizes and memcached will not
reallocate slabs/buckets after a flush_all(?).

If this behaviour is normal, does anyone have a better suggestion
regarding what to monitor other than evictions to see if more
memcached instances would be needed?

Memcached options were:

memcached -v -m 2000 -l 10.0.0.22 -p 11211 -u root

stats items reports evictions (shortly after flushing) in 2 buckets:

STAT items:5:number 8737
STAT items:5:age 24656
STAT items:5:evicted 16
STAT items:5:evicted_nonzero 0
STAT items:5:evicted_time 130774
STAT items:5:outofmemory 0
STAT items:5:tailrepairs 0
STAT items:5:reclaimed 6

STAT items:19:number 8832
STAT items:19:age 141322
STAT items:19:evicted 45
STAT items:19:evicted_nonzero 45
STAT items:19:evicted_time 14103
STAT items:19:outofmemory 0
STAT items:19:tailrepairs 0
STAT items:19:reclaimed 8

stats slabs:

STAT 5:chunk_size 240
STAT 5:chunks_per_page 4369
STAT 5:total_pages 2
STAT 5:total_chunks 8738
STAT 5:used_chunks 8738
STAT 5:free_chunks 0
STAT 5:free_chunks_end 0
STAT 5:mem_requested 1826616
STAT 5:get_hits 0
STAT 5:cmd_set 226
STAT 5:delete_hits 0
STAT 5:incr_hits 0
STAT 5:decr_hits 0
STAT 5:cas_hits 0
STAT 5:cas_badval 0

STAT 19:chunk_size 5680
STAT 19:chunks_per_page 184
STAT 19:total_pages 48
STAT 19:total_chunks 8832
STAT 19:used_chunks 8832
STAT 19:free_chunks 0
STAT 19:free_chunks_end 0
STAT 19:mem_requested 42671558
STAT 19:get_hits 17
STAT 19:cmd_set 217
STAT 19:delete_hits 0
STAT 19:incr_hits 0
STAT 19:decr_hits 0
STAT 19:cas_hits 0
STAT 19:cas_badval 0

Thanks,
Marinos
dormando
Re: flush_all and evictions
July 09, 2010 08:50AM
>
> we are trying to implement better memcached monitoring and have come
> across this problem with 1.4.5:
>
> - after flush_all on a filled (long-running) memcached, we get
> evictions immediately (within seconds, none of the buckets should fill
> up that quickly)
>
> This would mean that either flushed items count toward evictions when
> replaced (even though they should not be valid items after flush_all
> and evictions should count only valid items that were replaced) or
> that somehow flush_all does not work as expected, or that we have a
> pathetically bad distribution of item sizes and memcached will not
> reallocate slabs/buckets after a flush_all(?).
>
> If this behaviour is normal, does anyone have a better suggestion
> regarding what to monitor other than evictions to see if more
> memcached instances would be needed?

Huhhh I guess you're right. The evictions stat doesn't take into account
if the item has been flushed, so they'll still show up as evictions. I'm
not sure why you'd be doing flush_all to see if you need more memcached
instances? Are you doing it often? If flush's are rare then the various
evictions stats still mean something.

Watch for evictions_nonzero and how old evicted_time is to see if you're
probably running low on memory. Cross that with your hit rate and graph
it. So if your hit rate starts fluctuating you should be able to trace it
back to changes in traffic patterns or code pushes.

Sometimes high eviction rates can mean you're shoveling in too many items
that're never fetched again (but still set to expire after a long time).
So it's a matter of watching these stats + being dilligent about what your
app is doing.

Would you mind opening a bug report about the evictions + flush_all thing?
that's an easy fix.

Thanks,
-Dormando
Sorry, only registered users may post in this forum.

Click here to login