Welcome! Log In Create A New Profile

Advanced

Get multi error - too many keys returned

Posted by Piotr Husiatyński 
Piotr Husiatyński
Get multi error - too many keys returned
July 16, 2010 09:40AM
I'm having an issue with my memcached farm. I'm using 3 memcached
servers (v1.2.6) and python-memcached client (v1.43).

When using get_multi function, memcached returns sometimes keys that I
wasn't asking for. Memcached can return less data than expected (if
key-value does not exist), but documentation says nothing about
returning more data than it should. What can cause this?
dormando
Re: Get multi error - too many keys returned
July 21, 2010 10:40AM
> I'm having an issue with my memcached farm. I'm using 3 memcached
> servers (v1.2.6) and python-memcached client (v1.43).
>
> When using get_multi function, memcached returns sometimes keys that I
> wasn't asking for. Memcached can return less data than expected (if
> key-value does not exist), but documentation says nothing about
> returning more data than it should. What can cause this?

Client bugs, or buggy keys I assume? Memcached the server most certainly
only returns the keys you ask for, so if you're getting other ones you
might be using the client object wrong and getting out of sync with the
protocol. Or the client's just buggy...

Sorry, don't have any ideas about what it could be in particular. make
sure you're sending keys that don't have newlines/spaces/etc in them and
that you're not clobbering the object somehow.

-Dormando
Guille -bisho-
Re: Get multi error - too many keys returned
July 21, 2010 11:00AM
On Wed, Jul 21, 2010 at 10:37, dormando <[email protected]> wrote:
>> I'm having an issue with my memcached farm. I'm using 3 memcached
>> servers (v1.2.6) and python-memcached client (v1.43).
>>
>> When using get_multi function, memcached returns sometimes keys that I
>> wasn't asking for. Memcached can return less data than expected (if
>> key-value does not exist), but documentation says nothing about
>> returning more data than it should. What can cause this?

We have experienced similar behaviours with php memcache under heavy
load. Sometimes the clients seems to mix requests from different
replies.

There is not much you can do to avoid this. Switch to another client
if happens a lot, or just ignore the unneeded keys.

--
Guille -ℬḭṩḩø- <[email protected]>
:wq
David Morel
Re: Get multi error - too many keys returned
July 22, 2010 08:10AM
On 21 juil, 10:59, Guille -bisho- <[email protected]> wrote:
> On Wed, Jul 21, 2010 at 10:37, dormando <[email protected]> wrote:
> >> I'm having an issue with my memcached farm. I'm using 3 memcached
> >> servers (v1.2.6) and python-memcached client (v1.43).
>
> >> When using get_multi function, memcached returns sometimes keys that I
> >> wasn't asking for. Memcached can return less data than expected (if
> >> key-value does not exist), but documentation says nothing about
> >> returning more data than it should. What can cause this?
>
> We have experienced similar behaviours with php memcache under heavy
> load. Sometimes the clients seems to mix requests from different
> replies.

This happens when your client forks and forgets to reopen a
connection, but re-uses the one that was open before the fork.
As a results, several clients use the same socket which ends up
interleaving the traffic.
This is completely evil, not about memcache, really (it will happen
just the same with any network protocol), and is solved by
disconnecting and reconnecting.
Also, it is only visible under heavy load because this is when you
have a chance of seeing actually concurrent requests despite the very
short roundtrip of a typical memcached query.
Solutions: #1 make very sure you open the connection after the fork,
or #2 have a simple check in your memcached layer that checks the PID
hasn't changed. If it has, do a disconnect_all() or whatever it's
called in your language, and then run your queries. There is some
overhead when you adopt solution #2, but #1 is not always applicable
if you have spaghetti code and/or lax coding policies.

David
Guille -bisho-
Re: Get multi error - too many keys returned
July 22, 2010 02:30PM
On Thu, Jul 22, 2010 at 08:07, David Morel <[email protected]> wrote:
> On 21 juil, 10:59, Guille -bisho- <[email protected]> wrote:
>> On Wed, Jul 21, 2010 at 10:37, dormando <[email protected]> wrote:
>> >> I'm having an issue with my memcached farm. I'm using 3 memcached
>> >> servers (v1.2.6) and python-memcached client (v1.43).
>>
>> >> When using get_multi function, memcached returns sometimes keys that I
>> >> wasn't asking for. Memcached can return less data than expected (if
>> >> key-value does not exist), but documentation says nothing about
>> >> returning more data than it should. What can cause this?
>>
>> We have experienced similar behaviours with php memcache under heavy
>> load. Sometimes the clients seems to mix requests from different
>> replies.
>
> This happens when your client forks and forgets to reopen a
> connection, but re-uses the one that was open before the fork.
> As a results, several clients use the same socket which ends up
> interleaving the traffic.
> This is completely evil, not about memcache, really (it will happen
> just the same with any network protocol), and is solved by
> disconnecting and reconnecting.
> Also, it is only visible under heavy load because this is when you
> have a chance of seeing actually concurrent requests despite the very
> short roundtrip of a typical memcached query.
> Solutions: #1 make very sure you open the connection after the fork,
> or #2 have a simple check in your memcached layer that checks the PID
> hasn't changed. If it has, do a disconnect_all() or whatever it's
> called in your language, and then run your queries. There is some
> overhead when you adopt solution #2, but #1 is not always applicable
> if you have spaghetti code and/or lax coding policies.

Yeah, I know. PHP Memcache module is not really good. PECL Memcache
has his own implementation, and while more complete (supports UDP)
shows some problems like this, not too much fortunately. Memcached
(that uses libmemcached) seems more robust, but UDP support in
libmemcached is useless (has not implemented the wait for replies in
UDP).

It would be nice to give a bit more love to UDP implementation in
libmemcached (after all it's the official library and it's a pity his
implementation is not complete), so many drivers get advantage of a
well written client code.

--
Guille -ℬḭṩḩø- <[email protected]>
:wq
Sorry, only registered users may post in this forum.

Click here to login