Welcome! Log In Create A New Profile

Advanced

Issue 151 in memcached: Append/Prepend to existing key whose value that grows larger than 1MB and non-existing key returns NOT_STORED error

Posted by Anonymous User 
Status: New
Owner: ----
Labels: Type-Defect Priority-Medium

New issue 151 by tru64ufs: Append/Prepend to existing key whose value that
grows larger than 1MB and non-existing key returns NOT_STORED error
http://code.google.com/p/memcached/issues/detail?id=151

What steps will reproduce the problem?

1. append a value to an existing key that can grow larger than 1MB
2. the error returned is NOT_STORED
3. append a value to non-existing key also returns NOT_STORED


What is the expected output? What do you see instead?
We need a way to distinguish them. Otherwise we need get(key) before
append()


What version of the product are you using? On what operating system?
1.4.5 on Linux

Please provide any additional information below.

do_store_item() returns

new_it = do_item_alloc(key, it->nkey, flags,
old_it->exptime, it->nbytes + old_it->nbytes - 2 /* CRLF */);

if (new_it == NULL) {
/* SERVER_ERROR out of memory */
if (old_it != NULL)
do_item_remove(old_it);

return NOT_STORED;
}

I think "append/prepend" should return the same error
as "set/replace/update" larger than 1MB blob.

"SERVER_ERROR object too large for cache"
Comment #1 on issue 151 by ingenthr: Append/Prepend to existing key whose
value that grows larger than 1MB and non-existing key returns NOT_STORED
error
http://code.google.com/p/memcached/issues/detail?id=151

The problem with this is that other clients may be expecting different
behavior. There have been problems in the past with changing responses. I
don't know enough to anticipate how it would affect most of the clients.

I think this makes sense to do in 1.6 with an envvar to get the old
behavior if needed.

In terms of addressing your current needs with 1.4.5, perhaps the client
could fail back to doing the append on the client to see if it's a memory
issue or another kind of issue.
Comment #2 on issue 151 by tru64ufs: Append/Prepend to existing key whose
value that grows larger than 1MB and non-existing key returns NOT_STORED
error
http://code.google.com/p/memcached/issues/detail?id=151

I understand this would change client's expectation.

The problem with 1.4.5 is that the client trims to appropriate size on
get() not on append().

The client simply does not distinguish between non-existing key and
existing key on append. So the client's operation simply starts with
append() regardless of its existence. Doing get() on every append()
increase latency and although it may be rare, get() does not guarantee its
existence (no atomicity across multiple operation)

The client only trims to appropriate size on get().
I think we need to change this behavior on major upgrade due to backward
compatibility issue.

Thank you.
Comment #3 on issue 151 by dorma...@rydia.net: Append/Prepend to existing
key whose value that grows larger than 1MB and non-existing key returns
NOT_STORED error
http://code.google.com/p/memcached/issues/detail?id=151

Are you trying to ensure that the key doesn't go over 1MB, or that you
append against something that already exists?

Memcached removes the item if it fails to append to it, so your best bet is
to actually run 'add' if append fails for any reason. If the 'add' fails
you then lost the race for recreating the item and you can decide to either
run append again (depending on what your app is doing here...) or give up.

I never noticed the issue here since we already use the append/add routine.
Comment #4 on issue 151 by tru64ufs: Append/Prepend to existing key whose
value that grows larger than 1MB and non-existing key returns NOT_STORED
error
http://code.google.com/p/memcached/issues/detail?id=151

Dormando.

Here's what I am trying to do.

Workload: many append/prened(), a few read

1. append against non-existing key (allowed) - error ignored. no need to
check its existence
we only create key on DB read operation.
We are trying not to create too many keys so there are many append
operations to simplify logic

2. But once a key had been created, and unless there is another get()
operation, we only append.
since there are so many appends (append against non-existing key and
existing keys), we cannot do
get() on every append() NOT_STORED failure.
Again, NOT_STORED failure is ok for non-exsiting key.

3. Values only gets trims on get().

The problem with this is that once a key has been created, and workload
requires lots of append and few get,
when we get NOT_STORED, we do not know if this append() failure was due to
non-exsiting key or out-of-memory. Since there is no need to check for
existence when the key does not exist, we cannot add get() after each
append() failure. If you do, you are simply adding 1 RTT to each append
operation which doubles response time. Could be much worse if the value is
large (close to 1M)

Memcached does NOT remove the item when it fails to append/prepend with
out-of-mem error. It simply returns NOT_STORED.

Thank you, Dormando.
Comment #5 on issue 151 by dorma...@rydia.net: Append/Prepend to existing
key whose value that grows larger than 1MB and non-existing key returns
NOT_STORED error
http://code.google.com/p/memcached/issues/detail?id=151

Looks like you have the one edge case that's annoyed by this. Thanks for
explaining it.

If you need to use a workaround, I'd still suggest something like add. You
throw away an RTT still but don't fetch a lot of data back.

Then you're somewhat stuck with appending to keys you don't want to append
to... You could mitigate it a bit by having the "add" add a key with a
timeout of 1s, making it less likely that it'll fill to the brim with data.

Your case does seem a little weird though. If it's failing to append due to
hitting the item size limit, you just leave it at 1M? You don't trim it or
want to rewrite it? In most apps I run a failure due to overlarge value or
nonexistent key are both good reasons to re-add a compacted item.

This kinda makes me want to add a new command... something like "-X" which
enables protocol fixes, with the caveat that we can add protocol fixes in
every release. A method to bridge the gap between major releases, perhaps?
Comment #6 on issue 151 by tru64ufs: Append/Prepend to existing key whose
value that grows larger than 1MB and non-existing key returns NOT_STORED
error
http://code.google.com/p/memcached/issues/detail?id=151

Dormando.

Very good question. We are actually prepend(), not append().
That means we are using memcached as a FIFO Q. Only the oldest items need
to be discarded.
If we leave it at 1M, then we are stuck with old items in the queue.

I will have to play with "add" then.
Thanks a lot.
Sorry, only registered users may post in this forum.

Click here to login