Welcome! Log In Create A New Profile

Advanced

Memcached underutilized

Posted by amix 
amix
Memcached underutilized
October 22, 2010 11:30PM
We have recently added a new memcached server, but it's underutilized
or at least not as much utilized as our other memcached servers. The
difference between this new server and the older server are following:
* memcached 1.4.5 is run (instead of memcached 1.2.2)
* Ubuntu 10 is run (instead of Ubuntu 8)

Has something drastically changed between 1.2.2 and 1.4.5? Or has
something changed between Ubuntu 8 and Ubuntu 10? Anybody else having
a similar issue?

Here is usage patterns from these two servers. Each of them have 3GB
available, but the server running 1.4.5 only uses around 300MB. We are
unsure what the cause is. These servers are run like this:

memcached -m 3072 -k -p 11221 -u plurk -l 192.168.0.35 -c 10240

Regards,
amix

[   (   '192.168.0.35:11221 (1)',
       {   'accepting_conns': '1',
           'auth_cmds': '0',
           'auth_errors': '0',
           'bytes': '71301667',
           'bytes_read': '68596588527',
           'bytes_written': '215641263049',
           'cas_badval': '0',
           'cas_hits': '0',
           'cas_misses': '0',
           'cmd_flush': '0',
           'cmd_get': '246655883',
           'cmd_set': '62909113',
           'conn_yields': '0',
           'connection_structures': '2526',
           'curr_connections': '2262',
           'curr_items': '132819',
           'decr_hits': '0',
           'decr_misses': '0',
           'delete_hits': '2297822',
           'delete_misses': '23286159',
           'evictions': '0',
           'get_hits': '163185291',
           'get_misses': '83470592',
           'incr_hits': '25466',
           'incr_misses': '0',
           'limit_maxbytes': '3221225472',
           'listen_disabled_num': '0',
           'pid': '7319',
           'pointer_size': '64',
           'reclaimed': '36224211',
           'rusage_system': '8341.840000',
           'rusage_user': '3173.390000',
           'threads': '4',
           'time': '1287781046',
           'total_connections': '139794',
           'total_items': '62899221',
           'uptime': '514507',
           'version': '1.4.5'}),
   (   '192.168.0.70:11221 (1)',
       {   'bytes': '2879886510',
           'bytes_read': '65548028841',
           'bytes_written': '179395226207',
           'cmd_get': '277646387',
           'cmd_set': '68425776',
           'connection_structures': '2553',
           'curr_connections': '2291',
           'curr_items': '3907837',
           'evictions': '0',
           'get_hits': '193391910',
           'get_misses': '84254477',
           'limit_maxbytes': '3221225472',
           'pid': '17580',
           'pointer_size': '64',
           'rusage_system': '6113.240000',
           'rusage_user': '3507.560000',
           'threads': '1',
           'time': '1287781045',
           'total_connections': '138271',
           'total_items': '68416527',
           'uptime': '514517',
           'version': '1.2.2'})]
amix
Re: Memcached underutilized
October 23, 2010 12:00AM
Have done some more debugging and here is what I have found.

The general main differences for these two servers seems to be:
'curr_items': '132819', (memcached 1.4.5)
'curr_items': '3907837', (memcached 1.2.2)

Total items seem to be very similar, i.e.
'total_items': '62899221', (memcached 1.4.5)
'total_items': '68416527', (memcached 1.2.2)

Testing memcached 1.2.2 and memcached 1.4.5 and filling it up with
sample data it seems like memcached 1.4.5 is much better at using
memory that has been allocated by expired items. Is this correct?

If so, it would mean that our cache is way underutilized in general
and that 1.4.5 is much better at managing memory (i.e. it won't
allocate memory if it does not need to).

Could someone that's familiar with memcached internals please confirm
this.

Regards,
amix

On 22 Okt., 23:12, amix <ami...@gmail.com> wrote:
> We have recently added a new memcached server, but it's underutilized
> or at least not as much utilized as our other memcached servers. The
> difference between this new server and the older server are following:
> * memcached 1.4.5 is run (instead of memcached 1.2.2)
> * Ubuntu 10 is run (instead of Ubuntu 8)
>
> Has something drastically changed between 1.2.2 and 1.4.5? Or has
> something changed between Ubuntu 8 and Ubuntu 10? Anybody else having
> a similar issue?
>
> Here is usage patterns from these two servers. Each of them have 3GB
> available, but the server running 1.4.5 only uses around 300MB. We are
> unsure what the cause is. These servers are run like this:
>
> memcached -m 3072 -k -p 11221 -u plurk -l 192.168.0.35 -c 10240
>
> Regards,
> amix
>
> [   (   '192.168.0.35:11221 (1)',
>         {   'accepting_conns': '1',
>             'auth_cmds': '0',
>             'auth_errors': '0',
>             'bytes': '71301667',
>             'bytes_read': '68596588527',
>             'bytes_written': '215641263049',
>             'cas_badval': '0',
>             'cas_hits': '0',
>             'cas_misses': '0',
>             'cmd_flush': '0',
>             'cmd_get': '246655883',
>             'cmd_set': '62909113',
>             'conn_yields': '0',
>             'connection_structures': '2526',
>             'curr_connections': '2262',
>             'curr_items': '132819',
>             'decr_hits': '0',
>             'decr_misses': '0',
>             'delete_hits': '2297822',
>             'delete_misses': '23286159',
>             'evictions': '0',
>             'get_hits': '163185291',
>             'get_misses': '83470592',
>             'incr_hits': '25466',
>             'incr_misses': '0',
>             'limit_maxbytes': '3221225472',
>             'listen_disabled_num': '0',
>             'pid': '7319',
>             'pointer_size': '64',
>             'reclaimed': '36224211',
>             'rusage_system': '8341.840000',
>             'rusage_user': '3173.390000',
>             'threads': '4',
>             'time': '1287781046',
>             'total_connections': '139794',
>             'total_items': '62899221',
>             'uptime': '514507',
>             'version': '1.4.5'}),
>     (   '192.168.0.70:11221 (1)',
>         {   'bytes': '2879886510',
>             'bytes_read': '65548028841',
>             'bytes_written': '179395226207',
>             'cmd_get': '277646387',
>             'cmd_set': '68425776',
>             'connection_structures': '2553',
>             'curr_connections': '2291',
>             'curr_items': '3907837',
>             'evictions': '0',
>             'get_hits': '193391910',
>             'get_misses': '84254477',
>             'limit_maxbytes': '3221225472',
>             'pid': '17580',
>             'pointer_size': '64',
>             'rusage_system': '6113.240000',
>             'rusage_user': '3507.560000',
>             'threads': '1',
>             'time': '1287781045',
>             'total_connections': '138271',
>             'total_items': '68416527',
>             'uptime': '514517',
>             'version': '1.2.2'})]
Maxim Dounin
Re: Memcached underutilized
October 23, 2010 12:10AM
Hello!

On Fri, Oct 22, 2010 at 02:51:05PM -0700, amix wrote:

> Have done some more debugging and here is what I have found.
>
> The general main differences for these two servers seems to be:
> 'curr_items': '132819', (memcached 1.4.5)
> 'curr_items': '3907837', (memcached 1.2.2)
>
> Total items seem to be very similar, i.e.
> 'total_items': '62899221', (memcached 1.4.5)
> 'total_items': '68416527', (memcached 1.2.2)
>
> Testing memcached 1.2.2 and memcached 1.4.5 and filling it up with
> sample data it seems like memcached 1.4.5 is much better at using
> memory that has been allocated by expired items. Is this correct?
>
> If so, it would mean that our cache is way underutilized in general
> and that 1.4.5 is much better at managing memory (i.e. it won't
> allocate memory if it does not need to).
>
> Could someone that's familiar with memcached internals please confirm
> this.

Yes.

Re-use of expired items before allocating new items was done by
Trond Norbye[1][2] (in 1.4.0+). Note that "reclaimed" stats shows
how many times memcached re-used expired items.

[1] http://code.google.com/p/memcached/issues/detail?id=14
[2] http://github.com/memcached/memcached/commit/d3807d06f0693b0435545d35462de2829a61a7d1

Maxim Dounin
amix
Re: Memcached underutilized
October 23, 2010 02:30PM
Hi Maxim

Thanks for the clarification!

Kind regards,
amix

On 23 Okt., 00:05, Maxim Dounin <mdou...@mdounin.ru> wrote:
> Hello!
>
>
>
>
>
>
>
>
>
> On Fri, Oct 22, 2010 at 02:51:05PM -0700, amix wrote:
> > Have done some more debugging and here is what I have found.
>
> > The general main differences for these two servers seems to be:
> >             'curr_items': '132819',  (memcached 1.4.5)
> >             'curr_items': '3907837', (memcached 1.2.2)
>
> > Total items seem to be very similar, i.e.
> >             'total_items': '62899221',  (memcached 1.4.5)
> >             'total_items': '68416527', (memcached 1.2.2)
>
> > Testing memcached 1.2.2 and memcached 1.4.5 and filling it up with
> > sample data it seems like memcached 1.4.5 is much better at using
> > memory that has been allocated by expired items. Is this correct?
>
> > If so, it would mean that our cache is way underutilized in general
> > and that 1.4.5 is much better at managing memory (i.e. it won't
> > allocate memory if it does not need to).
>
> > Could someone that's familiar with memcached internals please confirm
> > this.
>
> Yes.
>
> Re-use of expired items before allocating new items was done by
> Trond Norbye[1][2] (in 1.4.0+).  Note that "reclaimed" stats shows
> how many times memcached re-used expired items.
>
> [1]http://code.google.com/p/memcached/issues/detail?id=14
> [2]http://github.com/memcached/memcached/commit/d3807d06f0693b0435545d35....
>
> Maxim Dounin
Sorry, only registered users may post in this forum.

Click here to login