Welcome! Log In Create A New Profile

Advanced

REST API

Posted by jsm 
jsm
REST API
July 28, 2010 04:40PM
Anyone writing or planning to write a REST API for memcached?
If no such plan, I would be interested in writing a REST API.
Any suggestions, comments welcome.
Gavin M. Roy
Re: REST API
July 28, 2010 04:50PM
Why add the HTTP protocol overhead? REST/HTTP would add ~75Mbps of
additional traffic at 100k gets per second by saying there's a rough 100
byte overhead per request over the ASCII protocol. I base the 100 bytes by
the HTTP GET request, minimal request headers and minimal response
headers. The binary protocol is very terse in comparison to the ASCII
protocol. In addition netcat or telnet works as good as curl for drop dead
simplicity. Don't get me wrong, it would be neat, but shouldn't be
considered in moderately well used memcached environments.

Regards,

Gavin

On Wed, Jul 28, 2010 at 8:43 AM, jsm <[email protected]> wrote:

> Anyone writing or planning to write a REST API for memcached?
> If no such plan, I would be interested in writing a REST API.
> Any suggestions, comments welcome.
>
Rajesh Nair
Re: REST API
July 28, 2010 05:20PM
Gavin,

If you go by the strict sense of word, HTTP protocol is not a pre-requisite
for REST service.
It requires a protocol which supports linking entities through URIs. It is
very much possible to implement a RESTful service by coming up with own URI
protocol for memcached messages

something like :
mc://<memcached-cluster>/messages/<key>

and the transport layer can be pretty much the same TCP to not add any
overhead.

JSM,

What is the value-add you are looking from the RESTful version of the
memcached API?

Regards,
Rajesh Nair



On Wed, Jul 28, 2010 at 8:13 PM, Gavin M. Roy <[email protected]> wrote:

> Why add the HTTP protocol overhead? REST/HTTP would add ~75Mbps of
> additional traffic at 100k gets per second by saying there's a rough 100
> byte overhead per request over the ASCII protocol. I base the 100 bytes by
> the HTTP GET request, minimal request headers and minimal response
> headers. The binary protocol is very terse in comparison to the ASCII
> protocol. In addition netcat or telnet works as good as curl for drop dead
> simplicity. Don't get me wrong, it would be neat, but shouldn't be
> considered in moderately well used memcached environments.
>
> Regards,
>
> Gavin
>
>
> On Wed, Jul 28, 2010 at 8:43 AM, jsm <[email protected]> wrote:
>
>> Anyone writing or planning to write a REST API for memcached?
>> If no such plan, I would be interested in writing a REST API.
>> Any suggestions, comments welcome.
>>
>
>
jsm
Re: REST API
July 28, 2010 05:50PM
On Jul 28, 8:02 pm, Rajesh Nair <[email protected]> wrote:
> Gavin,
>
> If you go by the strict sense of word, HTTP protocol is not a pre-requisite
> for REST service.
> It requires a protocol which supports linking entities through URIs.  It is
> very much possible to implement a RESTful service by coming up with own URI
> protocol for memcached messages
>
> something like :
> mc://<memcached-cluster>/messages/<key>
>
> and the transport layer can be pretty much the same TCP to not add any
> overhead.
>
> JSM,
>
> What is the value-add you are looking from the RESTful version of the
> memcached API?

Basically to be able to use without binding to any particular
language.

>
> Regards,
> Rajesh Nair
>
> On Wed, Jul 28, 2010 at 8:13 PM, Gavin M. Roy <[email protected]> wrote:
>
>
>
> > Why add the HTTP protocol overhead?  REST/HTTP would add ~75Mbps of
> > additional traffic at 100k gets per second by saying there's a rough 100
> > byte overhead per request over the ASCII protocol.  I base the 100 bytes by
> > the HTTP GET request, minimal request headers and minimal response
> > headers. The binary protocol is very terse in comparison to the ASCII
> > protocol.  In addition netcat or telnet works as good as curl for drop dead
> > simplicity.  Don't get me wrong, it would be neat, but shouldn't be
> > considered in moderately well used memcached environments.
>
> > Regards,
>
> > Gavin
>
> > On Wed, Jul 28, 2010 at 8:43 AM, jsm <[email protected]> wrote:
>
> >> Anyone writing or planning to write a REST API for memcached?
> >> If no such plan, I would be interested in writing a REST API.
> >> Any suggestions, comments welcome.
jsm
Re: REST API
July 28, 2010 05:50PM
Gavin,
You are right about the overhead and also saw that API's exist for
most of the languages as well.
I thought REST API would make memcached language agnostic.
I would like to hear from the community if the REST API should be
pursued or not?

On Jul 28, 7:43 pm, "Gavin M. Roy" <[email protected]> wrote:
> Why add the HTTP protocol overhead?  REST/HTTP would add ~75Mbps of
> additional traffic at 100k gets per second by saying there's a rough 100
> byte overhead per request over the ASCII protocol.  I base the 100 bytes by
> the HTTP GET request, minimal request headers and minimal response
> headers. The binary protocol is very terse in comparison to the ASCII
> protocol.  In addition netcat or telnet works as good as curl for drop dead
> simplicity.  Don't get me wrong, it would be neat, but shouldn't be
> considered in moderately well used memcached environments.
>
> Regards,
>
> Gavin
>
>
>
> On Wed, Jul 28, 2010 at 8:43 AM, jsm <[email protected]> wrote:
> > Anyone writing or planning to write a REST API for memcached?
> > If no such plan, I would be interested in writing a REST API.
> > Any suggestions, comments welcome.
Gavin M. Roy
Re: REST API
July 28, 2010 06:00PM
On Wed, Jul 28, 2010 at 11:37 AM, jsm <[email protected]> wrote:

> > What is the value-add you are looking from the RESTful version of the
> > memcached API?
>
> Basically to be able to use without binding to any particular
> language.
>

Not to push too hard on his, but the ascii protocol is simple and easy to
implement using sockets in any language:

http://code.sixapart.com/svn/memcached/trunk/server/doc/protocol.txt

I'd go as far as to say if you're focused on HTTP REST, it's a more simple
client implementation to just use the ascii protocol. If you do go down the
route of implementing HTTP REST, you'll want to look at stuff like
libevent's HTTP server as well as something like libmicrohttpd.

Regards,

Gavin
Michael Shadle
Re: REST API
July 28, 2010 06:00PM
While the idea is noble, there's already APIs for every language under the sun, and the ASCII protocol is plaintext, so you can just use sockets instead of sockets+http required for a RESTful interface.

On Jul 28, 2010, at 8:16 AM, jsm <[email protected]> wrote:

> Gavin,
> You are right about the overhead and also saw that API's exist for
> most of the languages as well.
> I thought REST API would make memcached language agnostic.
> I would like to hear from the community if the REST API should be
> pursued or not?
>
> On Jul 28, 7:43 pm, "Gavin M. Roy" <[email protected]> wrote:
>> Why add the HTTP protocol overhead? REST/HTTP would add ~75Mbps of
>> additional traffic at 100k gets per second by saying there's a rough 100
>> byte overhead per request over the ASCII protocol. I base the 100 bytes by
>> the HTTP GET request, minimal request headers and minimal response
>> headers. The binary protocol is very terse in comparison to the ASCII
>> protocol. In addition netcat or telnet works as good as curl for drop dead
>> simplicity. Don't get me wrong, it would be neat, but shouldn't be
>> considered in moderately well used memcached environments.
>>
>> Regards,
>>
>> Gavin
>>
>>
>>
>> On Wed, Jul 28, 2010 at 8:43 AM, jsm <[email protected]> wrote:
>>> Anyone writing or planning to write a REST API for memcached?
>>> If no such plan, I would be interested in writing a REST API.
>>> Any suggestions, comments welcome.
Les Mikesell
Re: REST API
July 28, 2010 06:10PM
On 7/28/2010 10:16 AM, jsm wrote:
> Gavin,
> You are right about the overhead and also saw that API's exist for
> most of the languages as well.
> I thought REST API would make memcached language agnostic.
> I would like to hear from the community if the REST API should be
> pursued or not?

I'm not quite sure how a rest api could deal with the distributed
servers without a special client anyway. But, it might be handy to have
a web service that mapped a rest api as directly as possible to memcache
operations where the http side would use traditional load balance/fail
over methods and handle the http 1.1 connection caching. I'm sure there
would be places where this could be used by components that have/want
data in a cache shared by more efficient clients.

--
Les Mikesell
lesmikesell@gmail.com
Adam Lee
Re: REST API
July 28, 2010 08:00PM
memcached's protocol is, as has been pointed out, already language agnostic
and much more efficient than trying to do HTTP. If you're saying RESTful in
the "not necessarily HTTP" sense, though, then I'd say that memcached's text
protocol is basically already as RESTful as you're going to get- think of
commands as verbs ('get,' 'set,' 'add,' 'delete,' etc...) and the key as a
URI and you're basically in an analogous situation that I think basically
meets the criteria as much as you can (hard to have a stateless cache)...
http://en.wikipedia.org/wiki/Representational_State_Transfer#Constraints

If you want a key-value datastore with an HTTP interface, though, might I
recommend Tokyo Tyrant? It speaks memcached and its own binary protocol as
well: http://1978th.net/tokyotyrant/spex.html#protocol

On Wed, Jul 28, 2010 at 12:03 PM, Les Mikesell <[email protected]>wrote:

> On 7/28/2010 10:16 AM, jsm wrote:
>
>> Gavin,
>> You are right about the overhead and also saw that API's exist for
>> most of the languages as well.
>> I thought REST API would make memcached language agnostic.
>> I would like to hear from the community if the REST API should be
>> pursued or not?
>>
>
> I'm not quite sure how a rest api could deal with the distributed servers
> without a special client anyway. But, it might be handy to have a web
> service that mapped a rest api as directly as possible to memcache
> operations where the http side would use traditional load balance/fail over
> methods and handle the http 1.1 connection caching. I'm sure there would be
> places where this could be used by components that have/want data in a cache
> shared by more efficient clients.
>
> --
> Les Mikesell
> lesmikesell@gmail.com
>



--
awl
Les Mikesell
Re: REST API
July 28, 2010 08:20PM
There's no argument that embedding a locally-configured memcache client
library for the appropriate language into your program would be more
efficient, but consider the case where you have many programs in many
languages sharing the cache data and some of them have inherent http
capability and aren't used enough to care about that last 10% efficiency
when it means rewriting a bunch of code with new libraries to get it.
However, I still think the http interface would have to be a separate
standalone piece, sitting over a stock client that knows about the local
distributed servers or you'd need a special client library anyway.

-Les


On 7/28/2010 12:53 PM, Adam Lee wrote:
> memcached's protocol is, as has been pointed out, already language
> agnostic and much more efficient than trying to do HTTP. If you're
> saying RESTful in the "not necessarily HTTP" sense, though, then I'd say
> that memcached's text protocol is basically already as RESTful as you're
> going to get- think of commands as verbs ('get,' 'set,' 'add,' 'delete,'
> etc...) and the key as a URI and you're basically in an analogous
> situation that I think basically meets the criteria as much as you can
> (hard to have a stateless cache)...
> http://en.wikipedia.org/wiki/Representational_State_Transfer#Constraints
>
> If you want a key-value datastore with an HTTP interface, though, might
> I recommend Tokyo Tyrant? It speaks memcached and its own binary
> protocol as well: http://1978th.net/tokyotyrant/spex.html#protocol
>
> On Wed, Jul 28, 2010 at 12:03 PM, Les Mikesell <[email protected]
> <mailto:[email protected]>> wrote:
>
> On 7/28/2010 10:16 AM, jsm wrote:
>
> Gavin,
> You are right about the overhead and also saw that API's exist for
> most of the languages as well.
> I thought REST API would make memcached language agnostic.
> I would like to hear from the community if the REST API should be
> pursued or not?
>
>
> I'm not quite sure how a rest api could deal with the distributed
> servers without a special client anyway. But, it might be handy to
> have a web service that mapped a rest api as directly as possible to
> memcache operations where the http side would use traditional load
> balance/fail over methods and handle the http 1.1 connection
> caching. I'm sure there would be places where this could be used by
> components that have/want data in a cache shared by more efficient
> clients.
>
> --
> Les Mikesell
> lesmikesell@gmail.com <mailto:[email protected]>
>
>
>
>
> --
> awl
Adam Lee
Re: REST API
July 28, 2010 08:20PM
That seems an odd case to me, to be honest. One of the key benefits of
memcached is its ultra-low latency, which is negated somewhat by using a
much heavier protocol. Also, writing a simple client library for the text
protocol is seriously achievable in an afternoon.

Anyway, there's nothing to say that you can't change your "hashing" strategy
to work with HTTP/URIs. Just hash the URI or use characters from it to
select a server, for example... As long as all clients agree, it doesn't
matter how you shard the data.

Again, I'd say you should take a look at TokyoTyrant for a fast, simple
key-value store that speaks both memcached and HTTP.

On Wed, Jul 28, 2010 at 2:11 PM, Les Mikesell <[email protected]> wrote:

> There's no argument that embedding a locally-configured memcache client
> library for the appropriate language into your program would be more
> efficient, but consider the case where you have many programs in many
> languages sharing the cache data and some of them have inherent http
> capability and aren't used enough to care about that last 10% efficiency
> when it means rewriting a bunch of code with new libraries to get it.
> However, I still think the http interface would have to be a separate
> standalone piece, sitting over a stock client that knows about the local
> distributed servers or you'd need a special client library anyway.
>
> -Les
>
>
>
> On 7/28/2010 12:53 PM, Adam Lee wrote:
>
>> memcached's protocol is, as has been pointed out, already language
>> agnostic and much more efficient than trying to do HTTP. If you're
>> saying RESTful in the "not necessarily HTTP" sense, though, then I'd say
>> that memcached's text protocol is basically already as RESTful as you're
>> going to get- think of commands as verbs ('get,' 'set,' 'add,' 'delete,'
>> etc...) and the key as a URI and you're basically in an analogous
>> situation that I think basically meets the criteria as much as you can
>> (hard to have a stateless cache)...
>> http://en.wikipedia.org/wiki/Representational_State_Transfer#Constraints
>>
>> If you want a key-value datastore with an HTTP interface, though, might
>> I recommend Tokyo Tyrant? It speaks memcached and its own binary
>> protocol as well: http://1978th.net/tokyotyrant/spex.html#protocol
>>
>> On Wed, Jul 28, 2010 at 12:03 PM, Les Mikesell <[email protected]
>> <mailto:[email protected]>> wrote:
>>
>> On 7/28/2010 10:16 AM, jsm wrote:
>>
>> Gavin,
>> You are right about the overhead and also saw that API's exist for
>> most of the languages as well.
>> I thought REST API would make memcached language agnostic.
>> I would like to hear from the community if the REST API should be
>> pursued or not?
>>
>>
>> I'm not quite sure how a rest api could deal with the distributed
>> servers without a special client anyway. But, it might be handy to
>> have a web service that mapped a rest api as directly as possible to
>> memcache operations where the http side would use traditional load
>> balance/fail over methods and handle the http 1.1 connection
>> caching. I'm sure there would be places where this could be used by
>> components that have/want data in a cache shared by more efficient
>> clients.
>>
>> --
>> Les Mikesell
>> lesmikesell@gmail.com <mailto:[email protected]>
>>
>>
>>
>>
>> --
>> awl
>>
>
>


--
awl
Marc Bollinger
Re: REST API
July 28, 2010 08:50PM
"and some of them have inherent http capability and aren't used enough to
care about that last 10% efficiency when it means rewriting a bunch of code
with new libraries to get it."

But you're..._adding_ support for memcached to that system. If this were a
system already using memcached, it'd be...speaking memcached. If it were
using a different, external caching system, you'd probably expect some
measure of integration code, or at least testing it if, like Tyrant, it did
speak the same protocol you were expecting. The only other case would be if
you weren't using some external cache already, in which event you're going
to be adding logic, regardless.

- Marc

On Wed, Jul 28, 2010 at 11:19 AM, Adam Lee <[email protected]> wrote:

> That seems an odd case to me, to be honest. One of the key benefits of
> memcached is its ultra-low latency, which is negated somewhat by using a
> much heavier protocol. Also, writing a simple client library for the text
> protocol is seriously achievable in an afternoon.
>
> Anyway, there's nothing to say that you can't change your "hashing"
> strategy to work with HTTP/URIs. Just hash the URI or use characters from
> it to select a server, for example... As long as all clients agree, it
> doesn't matter how you shard the data.
>
> Again, I'd say you should take a look at TokyoTyrant for a fast, simple
> key-value store that speaks both memcached and HTTP.
>
> On Wed, Jul 28, 2010 at 2:11 PM, Les Mikesell <[email protected]>wrote:
>
>> There's no argument that embedding a locally-configured memcache client
>> library for the appropriate language into your program would be more
>> efficient, but consider the case where you have many programs in many
>> languages sharing the cache data and some of them have inherent http
>> capability and aren't used enough to care about that last 10% efficiency
>> when it means rewriting a bunch of code with new libraries to get it.
>> However, I still think the http interface would have to be a separate
>> standalone piece, sitting over a stock client that knows about the local
>> distributed servers or you'd need a special client library anyway.
>>
>> -Les
>>
>>
>>
>> On 7/28/2010 12:53 PM, Adam Lee wrote:
>>
>>> memcached's protocol is, as has been pointed out, already language
>>> agnostic and much more efficient than trying to do HTTP. If you're
>>> saying RESTful in the "not necessarily HTTP" sense, though, then I'd say
>>> that memcached's text protocol is basically already as RESTful as you're
>>> going to get- think of commands as verbs ('get,' 'set,' 'add,' 'delete,'
>>> etc...) and the key as a URI and you're basically in an analogous
>>> situation that I think basically meets the criteria as much as you can
>>> (hard to have a stateless cache)...
>>> http://en.wikipedia.org/wiki/Representational_State_Transfer#Constraints
>>>
>>> If you want a key-value datastore with an HTTP interface, though, might
>>> I recommend Tokyo Tyrant? It speaks memcached and its own binary
>>> protocol as well: http://1978th.net/tokyotyrant/spex.html#protocol
>>>
>>> On Wed, Jul 28, 2010 at 12:03 PM, Les Mikesell <[email protected]
>>> <mailto:[email protected]>> wrote:
>>>
>>> On 7/28/2010 10:16 AM, jsm wrote:
>>>
>>> Gavin,
>>> You are right about the overhead and also saw that API's exist for
>>> most of the languages as well.
>>> I thought REST API would make memcached language agnostic.
>>> I would like to hear from the community if the REST API should be
>>> pursued or not?
>>>
>>>
>>> I'm not quite sure how a rest api could deal with the distributed
>>> servers without a special client anyway. But, it might be handy to
>>> have a web service that mapped a rest api as directly as possible to
>>> memcache operations where the http side would use traditional load
>>> balance/fail over methods and handle the http 1.1 connection
>>> caching. I'm sure there would be places where this could be used by
>>> components that have/want data in a cache shared by more efficient
>>> clients.
>>>
>>> --
>>> Les Mikesell
>>> lesmikesell@gmail.com <mailto:[email protected]>
>>>
>>>
>>>
>>>
>>> --
>>> awl
>>>
>>
>>
>
>
> --
> awl
>
Les Mikesell
Re: REST API
July 28, 2010 09:10PM
On 7/28/2010 1:43 PM, Marc Bollinger wrote:
> "and some of them have inherent http capability and aren't used enough
> to care about that last 10% efficiency when it means rewriting a bunch
> of code with new libraries to get it."
>
> But you're..._adding_ support for memcached to that system. If this were
> a system already using memcached, it'd be...speaking memcached. If it
> were using a different, external caching system, you'd probably expect
> some measure of integration code, or at least testing it if, like
> Tyrant, it did speak the same protocol you were expecting. The only
> other case would be if you weren't using some external cache already, in
> which event you're going to be adding logic, regardless.

I think we don't have the same concept of "the system". Mine is that
unrelated programs may be using the same data and only some of them are
designed around memcache and need its speed. Maybe it would be better
to embed memcache into a traditional http proxy for the things that
don't to avoid having the cache miss logic everywhere.

--
Les Mikesell
lesmikesell@gmail.com
Aaron Stone
Re: REST API
July 28, 2010 11:00PM
On Wed, Jul 28, 2010 at 8:37 AM, jsm <[email protected]> wrote:
>
>
> On Jul 28, 8:02 pm, Rajesh Nair <[email protected]> wrote:
>> Gavin,
>>
>> If you go by the strict sense of word, HTTP protocol is not a pre-requisite
>> for REST service.
>> It requires a protocol which supports linking entities through URIs.  It is
>> very much possible to implement a RESTful service by coming up with own URI
>> protocol for memcached messages
>>
>> something like :
>> mc://<memcached-cluster>/messages/<key>
>>
>> and the transport layer can be pretty much the same TCP to not add any
>> overhead.
>>
>> JSM,
>>
>> What is the value-add you are looking from the RESTful version of the
>> memcached API?
>
> Basically to be able to use without binding to any particular
> language.

I read this as requesting memcached native support for structured
values (e.g. hashes, lists, etc.) -- is that what you meant?

Aaron

>>
>> Regards,
>> Rajesh Nair
>>
>> On Wed, Jul 28, 2010 at 8:13 PM, Gavin M. Roy <[email protected]> wrote:
>>
>>
>>
>> > Why add the HTTP protocol overhead?  REST/HTTP would add ~75Mbps of
>> > additional traffic at 100k gets per second by saying there's a rough 100
>> > byte overhead per request over the ASCII protocol.  I base the 100 bytes by
>> > the HTTP GET request, minimal request headers and minimal response
>> > headers. The binary protocol is very terse in comparison to the ASCII
>> > protocol.  In addition netcat or telnet works as good as curl for drop dead
>> > simplicity.  Don't get me wrong, it would be neat, but shouldn't be
>> > considered in moderately well used memcached environments.
>>
>> > Regards,
>>
>> > Gavin
>>
>> > On Wed, Jul 28, 2010 at 8:43 AM, jsm <[email protected]> wrote:
>>
>> >> Anyone writing or planning to write a REST API for memcached?
>> >> If no such plan, I would be interested in writing a REST API.
>> >> Any suggestions, comments welcome.
jsm
Re: REST API
July 29, 2010 06:20AM
What I meant was to add a REST protocol to memcached layer, just like
you have a binary protocol and ascii.
Its up to the user to decide which protocol to use when accessing
memcached objects.
Regards,
J.S.Mammen

On Jul 29, 1:49 am, Aaron Stone <[email protected]> wrote:
> On Wed, Jul 28, 2010 at 8:37 AM, jsm <[email protected]> wrote:
>
> > On Jul 28, 8:02 pm, Rajesh Nair <[email protected]> wrote:
> >> Gavin,
>
> >> If you go by the strict sense of word, HTTP protocol is not a pre-requisite
> >> for REST service.
> >> It requires a protocol which supports linking entities through URIs.  It is
> >> very much possible to implement a RESTful service by coming up with own URI
> >> protocol for memcached messages
>
> >> something like :
> >> mc://<memcached-cluster>/messages/<key>
>
> >> and the transport layer can be pretty much the same TCP to not add any
> >> overhead.
>
> >> JSM,
>
> >> What is the value-add you are looking from the RESTful version of the
> >> memcached API?
>
> > Basically to be able to use without binding to any particular
> > language.
>
> I read this as requesting memcached native support for structured
> values (e.g. hashes, lists, etc.) -- is that what you meant?
>
> Aaron
>
>
>
>
>
> >> Regards,
> >> Rajesh Nair
>
> >> On Wed, Jul 28, 2010 at 8:13 PM, Gavin M. Roy <[email protected]> wrote:
>
> >> > Why add the HTTP protocol overhead?  REST/HTTP would add ~75Mbps of
> >> > additional traffic at 100k gets per second by saying there's a rough 100
> >> > byte overhead per request over the ASCII protocol.  I base the 100 bytes by
> >> > the HTTP GET request, minimal request headers and minimal response
> >> > headers. The binary protocol is very terse in comparison to the ASCII
> >> > protocol.  In addition netcat or telnet works as good as curl for drop dead
> >> > simplicity.  Don't get me wrong, it would be neat, but shouldn't be
> >> > considered in moderately well used memcached environments.
>
> >> > Regards,
>
> >> > Gavin
>
> >> > On Wed, Jul 28, 2010 at 8:43 AM, jsm <[email protected]> wrote:
>
> >> >> Anyone writing or planning to write a REST API for memcached?
> >> >> If no such plan, I would be interested in writing a REST API.
> >> >> Any suggestions, comments welcome.
j.s. mammen
Re: REST API
July 29, 2010 10:50AM
Here is an example use case found in Memcached WIKI. I think the below
usecase could benefit from REST
====
Example use-case¶

My company Widgets&Sprockets has a distributed SOA architecture that
uses a brokerage object for distributing web service traffic across
several hundred services. Large portions of the respones to this
traffic can be cached in the brokerage object (they are just xml
documents stored as UTF8 encoded strings). These requests are
optomised by first checking the cache before even hitting the
brokerage object, sadly however the company has a legacy architecture
leading to some of the calls being sent to the brokerage object coming
from Java and some from C#. Currently it is not a trivial task to
retrieve the XML documents from the cache from both Java & C# without
some heavy patching. The two clients chosen for this use-case are the
excellent C# memcached client library Enyim.Memcached and the fabulous
Java memcached client library Spy.Net. Please note this code is not
curently possible on the C# side as several patches are pending being
applied to the client (it is entirely possible that they won't be
applied, so this may remain a proof-of-concept rather than a true
example implementation)

=====

On Jul 29, 8:42 am, jsm <[email protected]> wrote:
> What I meant was to add a REST protocol to memcached layer, just like
> you have a binary protocol and ascii.
> Its up to the user to decide which protocol to use when accessing
> memcached objects.
> Regards,
> J.S.Mammen
>
> On Jul 29, 1:49 am, Aaron Stone <[email protected]> wrote:
>
>
>
> > On Wed, Jul 28, 2010 at 8:37 AM, jsm <[email protected]> wrote:
>
> > > On Jul 28, 8:02 pm, Rajesh Nair <[email protected]> wrote:
> > >> Gavin,
>
> > >> If you go by the strict sense of word, HTTP protocol is not a pre-requisite
> > >> for REST service.
> > >> It requires a protocol which supports linking entities through URIs.  It is
> > >> very much possible to implement a RESTful service by coming up with own URI
> > >> protocol for memcached messages
>
> > >> something like :
> > >> mc://<memcached-cluster>/messages/<key>
>
> > >> and the transport layer can be pretty much the same TCP to not add any
> > >> overhead.
>
> > >> JSM,
>
> > >> What is the value-add you are looking from the RESTful version of the
> > >> memcached API?
>
> > > Basically to be able to use without binding to any particular
> > > language.
>
> > I read this as requesting memcached native support for structured
> > values (e.g. hashes, lists, etc.) -- is that what you meant?
>
> > Aaron
>
> > >> Regards,
> > >> Rajesh Nair
>
> > >> On Wed, Jul 28, 2010 at 8:13 PM, Gavin M. Roy <[email protected]> wrote:
>
> > >> > Why add the HTTP protocol overhead?  REST/HTTP would add ~75Mbps of
> > >> > additional traffic at 100k gets per second by saying there's a rough 100
> > >> > byte overhead per request over the ASCII protocol.  I base the 100 bytes by
> > >> > the HTTP GET request, minimal request headers and minimal response
> > >> > headers. The binary protocol is very terse in comparison to the ASCII
> > >> > protocol.  In addition netcat or telnet works as good as curl for drop dead
> > >> > simplicity.  Don't get me wrong, it would be neat, but shouldn't be
> > >> > considered in moderately well used memcached environments.
>
> > >> > Regards,
>
> > >> > Gavin
>
> > >> > On Wed, Jul 28, 2010 at 8:43 AM, jsm <[email protected]> wrote:
>
> > >> >> Anyone writing or planning to write a REST API for memcached?
> > >> >> If no such plan, I would be interested in writing a REST API.
> > >> >> Any suggestions, comments welcome.
Henrik Schröder
Re: REST API
July 29, 2010 11:00AM
The whole point of adding the binary protocol was to create something even
more efficient and structured than the ascii protocol, so what would adding
REST to it do? It'd just be an even more verbose protocol than the ascii
one, and well yeah, you can point a web browser instead of a telnet client
to a server, but come on...


/Henrik

On Thu, Jul 29, 2010 at 05:42, jsm <[email protected]> wrote:

> What I meant was to add a REST protocol to memcached layer, just like
> you have a binary protocol and ascii.
> Its up to the user to decide which protocol to use when accessing
> memcached objects.
> Regards,
> J.S.Mammen
>
> On Jul 29, 1:49 am, Aaron Stone <[email protected]> wrote:
> > On Wed, Jul 28, 2010 at 8:37 AM, jsm <[email protected]> wrote:
> >
> > > On Jul 28, 8:02 pm, Rajesh Nair <[email protected]> wrote:
> > >> Gavin,
> >
> > >> If you go by the strict sense of word, HTTP protocol is not a
> pre-requisite
> > >> for REST service.
> > >> It requires a protocol which supports linking entities through URIs.
> It is
> > >> very much possible to implement a RESTful service by coming up with
> own URI
> > >> protocol for memcached messages
> >
> > >> something like :
> > >> mc://<memcached-cluster>/messages/<key>
> >
> > >> and the transport layer can be pretty much the same TCP to not add any
> > >> overhead.
> >
> > >> JSM,
> >
> > >> What is the value-add you are looking from the RESTful version of the
> > >> memcached API?
> >
> > > Basically to be able to use without binding to any particular
> > > language.
> >
> > I read this as requesting memcached native support for structured
> > values (e.g. hashes, lists, etc.) -- is that what you meant?
> >
> > Aaron
> >
> >
> >
> >
> >
> > >> Regards,
> > >> Rajesh Nair
> >
> > >> On Wed, Jul 28, 2010 at 8:13 PM, Gavin M. Roy <[email protected]>
> wrote:
> >
> > >> > Why add the HTTP protocol overhead? REST/HTTP would add ~75Mbps of
> > >> > additional traffic at 100k gets per second by saying there's a rough
> 100
> > >> > byte overhead per request over the ASCII protocol. I base the 100
> bytes by
> > >> > the HTTP GET request, minimal request headers and minimal response
> > >> > headers. The binary protocol is very terse in comparison to the
> ASCII
> > >> > protocol. In addition netcat or telnet works as good as curl for
> drop dead
> > >> > simplicity. Don't get me wrong, it would be neat, but shouldn't be
> > >> > considered in moderately well used memcached environments.
> >
> > >> > Regards,
> >
> > >> > Gavin
> >
> > >> > On Wed, Jul 28, 2010 at 8:43 AM, jsm <[email protected]> wrote:
> >
> > >> >> Anyone writing or planning to write a REST API for memcached?
> > >> >> If no such plan, I would be interested in writing a REST API.
> > >> >> Any suggestions, comments welcome.
>
Henrik Schröder
Re: REST API
July 29, 2010 11:10AM
Exactly how would the below use-case be solved with REST?

The incompatibilities between clients are because there is no universal way
of mapping keys->server, most clients do it a little bit differently, and
there is no universal list of how the flags attributes should be used, and
there is no cross-platform serialization strategy.

None of those problems would be solved by REST. They're all outside the
communication between a single client and a single server, and instead have
everything to do with how a client handles a cluster of servers, and what a
client does with the bytestreams that are actually stored in the servers.


/Henrik

On Thu, Jul 29, 2010 at 10:48, j.s. mammen <[email protected]> wrote:

> Here is an example use case found in Memcached WIKI. I think the below
> usecase could benefit from REST
> ====
> Example use-case¶
>
> My company Widgets&Sprockets has a distributed SOA architecture that
> uses a brokerage object for distributing web service traffic across
> several hundred services. Large portions of the respones to this
> traffic can be cached in the brokerage object (they are just xml
> documents stored as UTF8 encoded strings). These requests are
> optomised by first checking the cache before even hitting the
> brokerage object, sadly however the company has a legacy architecture
> leading to some of the calls being sent to the brokerage object coming
> from Java and some from C#. Currently it is not a trivial task to
> retrieve the XML documents from the cache from both Java & C# without
> some heavy patching. The two clients chosen for this use-case are the
> excellent C# memcached client library Enyim.Memcached and the fabulous
> Java memcached client library Spy.Net. Please note this code is not
> curently possible on the C# side as several patches are pending being
> applied to the client (it is entirely possible that they won't be
> applied, so this may remain a proof-of-concept rather than a true
> example implementation)
>
> =====
>
> On Jul 29, 8:42 am, jsm <[email protected]> wrote:
> > What I meant was to add a REST protocol to memcached layer, just like
> > you have a binary protocol and ascii.
> > Its up to the user to decide which protocol to use when accessing
> > memcached objects.
> > Regards,
> > J.S.Mammen
> >
> > On Jul 29, 1:49 am, Aaron Stone <[email protected]> wrote:
> >
> >
> >
> > > On Wed, Jul 28, 2010 at 8:37 AM, jsm <[email protected]> wrote:
> >
> > > > On Jul 28, 8:02 pm, Rajesh Nair <[email protected]> wrote:
> > > >> Gavin,
> >
> > > >> If you go by the strict sense of word, HTTP protocol is not a
> pre-requisite
> > > >> for REST service.
> > > >> It requires a protocol which supports linking entities through URIs.
> It is
> > > >> very much possible to implement a RESTful service by coming up with
> own URI
> > > >> protocol for memcached messages
> >
> > > >> something like :
> > > >> mc://<memcached-cluster>/messages/<key>
> >
> > > >> and the transport layer can be pretty much the same TCP to not add
> any
> > > >> overhead.
> >
> > > >> JSM,
> >
> > > >> What is the value-add you are looking from the RESTful version of
> the
> > > >> memcached API?
> >
> > > > Basically to be able to use without binding to any particular
> > > > language.
> >
> > > I read this as requesting memcached native support for structured
> > > values (e.g. hashes, lists, etc.) -- is that what you meant?
> >
> > > Aaron
> >
> > > >> Regards,
> > > >> Rajesh Nair
> >
> > > >> On Wed, Jul 28, 2010 at 8:13 PM, Gavin M. Roy <[email protected]>
> wrote:
> >
> > > >> > Why add the HTTP protocol overhead? REST/HTTP would add ~75Mbps
> of
> > > >> > additional traffic at 100k gets per second by saying there's a
> rough 100
> > > >> > byte overhead per request over the ASCII protocol. I base the 100
> bytes by
> > > >> > the HTTP GET request, minimal request headers and minimal response
> > > >> > headers. The binary protocol is very terse in comparison to the
> ASCII
> > > >> > protocol. In addition netcat or telnet works as good as curl for
> drop dead
> > > >> > simplicity. Don't get me wrong, it would be neat, but shouldn't
> be
> > > >> > considered in moderately well used memcached environments.
> >
> > > >> > Regards,
> >
> > > >> > Gavin
> >
> > > >> > On Wed, Jul 28, 2010 at 8:43 AM, jsm <[email protected]> wrote:
> >
> > > >> >> Anyone writing or planning to write a REST API for memcached?
> > > >> >> If no such plan, I would be interested in writing a REST API.
> > > >> >> Any suggestions, comments welcome.
>
Aaron Stone
Re: REST API
July 29, 2010 01:00PM
What's a ReST protocol? ReST is a model.

On Wed, Jul 28, 2010 at 8:42 PM, jsm <[email protected]> wrote:
> What I meant was to add a REST protocol to memcached layer, just like
> you have a binary protocol and ascii.
> Its up to the user to decide which protocol to use when accessing
> memcached objects.
> Regards,
> J.S.Mammen
>
> On Jul 29, 1:49 am, Aaron Stone <[email protected]> wrote:
>> On Wed, Jul 28, 2010 at 8:37 AM, jsm <[email protected]> wrote:
>>
>> > On Jul 28, 8:02 pm, Rajesh Nair <[email protected]> wrote:
>> >> Gavin,
>>
>> >> If you go by the strict sense of word, HTTP protocol is not a pre-requisite
>> >> for REST service.
>> >> It requires a protocol which supports linking entities through URIs.  It is
>> >> very much possible to implement a RESTful service by coming up with own URI
>> >> protocol for memcached messages
>>
>> >> something like :
>> >> mc://<memcached-cluster>/messages/<key>
>>
>> >> and the transport layer can be pretty much the same TCP to not add any
>> >> overhead.
>>
>> >> JSM,
>>
>> >> What is the value-add you are looking from the RESTful version of the
>> >> memcached API?
>>
>> > Basically to be able to use without binding to any particular
>> > language.
>>
>> I read this as requesting memcached native support for structured
>> values (e.g. hashes, lists, etc.) -- is that what you meant?
>>
>> Aaron
>>
>>
>>
>>
>>
>> >> Regards,
>> >> Rajesh Nair
>>
>> >> On Wed, Jul 28, 2010 at 8:13 PM, Gavin M. Roy <[email protected]> wrote:
>>
>> >> > Why add the HTTP protocol overhead?  REST/HTTP would add ~75Mbps of
>> >> > additional traffic at 100k gets per second by saying there's a rough 100
>> >> > byte overhead per request over the ASCII protocol.  I base the 100 bytes by
>> >> > the HTTP GET request, minimal request headers and minimal response
>> >> > headers. The binary protocol is very terse in comparison to the ASCII
>> >> > protocol.  In addition netcat or telnet works as good as curl for drop dead
>> >> > simplicity.  Don't get me wrong, it would be neat, but shouldn't be
>> >> > considered in moderately well used memcached environments.
>>
>> >> > Regards,
>>
>> >> > Gavin
>>
>> >> > On Wed, Jul 28, 2010 at 8:43 AM, jsm <[email protected]> wrote:
>>
>> >> >> Anyone writing or planning to write a REST API for memcached?
>> >> >> If no such plan, I would be interested in writing a REST API.
>> >> >> Any suggestions, comments welcome.
Henrik Schröder
Re: REST API
July 29, 2010 01:40PM
I would assume he's talking about making memcached expose some sort of
simple web service api over http.

Although, you could argue that both the ascii protocol and binary protocol
are restful, the sure seem to me to fit the definition pretty closely.


/Henrik

On Thu, Jul 29, 2010 at 12:56, Aaron Stone <[email protected]> wrote:

> What's a ReST protocol? ReST is a model.
>
> On Wed, Jul 28, 2010 at 8:42 PM, jsm <[email protected]> wrote:
> > What I meant was to add a REST protocol to memcached layer, just like
> > you have a binary protocol and ascii.
> > Its up to the user to decide which protocol to use when accessing
> > memcached objects.
> > Regards,
> > J.S.Mammen
> >
> > On Jul 29, 1:49 am, Aaron Stone <[email protected]> wrote:
> >> On Wed, Jul 28, 2010 at 8:37 AM, jsm <[email protected]> wrote:
> >>
> >> > On Jul 28, 8:02 pm, Rajesh Nair <[email protected]> wrote:
> >> >> Gavin,
> >>
> >> >> If you go by the strict sense of word, HTTP protocol is not a
> pre-requisite
> >> >> for REST service.
> >> >> It requires a protocol which supports linking entities through URIs.
> It is
> >> >> very much possible to implement a RESTful service by coming up with
> own URI
> >> >> protocol for memcached messages
> >>
> >> >> something like :
> >> >> mc://<memcached-cluster>/messages/<key>
> >>
> >> >> and the transport layer can be pretty much the same TCP to not add
> any
> >> >> overhead.
> >>
> >> >> JSM,
> >>
> >> >> What is the value-add you are looking from the RESTful version of the
> >> >> memcached API?
> >>
> >> > Basically to be able to use without binding to any particular
> >> > language.
> >>
> >> I read this as requesting memcached native support for structured
> >> values (e.g. hashes, lists, etc.) -- is that what you meant?
> >>
> >> Aaron
> >>
> >>
> >>
> >>
> >>
> >> >> Regards,
> >> >> Rajesh Nair
> >>
> >> >> On Wed, Jul 28, 2010 at 8:13 PM, Gavin M. Roy <[email protected]>
> wrote:
> >>
> >> >> > Why add the HTTP protocol overhead? REST/HTTP would add ~75Mbps of
> >> >> > additional traffic at 100k gets per second by saying there's a
> rough 100
> >> >> > byte overhead per request over the ASCII protocol. I base the 100
> bytes by
> >> >> > the HTTP GET request, minimal request headers and minimal response
> >> >> > headers. The binary protocol is very terse in comparison to the
> ASCII
> >> >> > protocol. In addition netcat or telnet works as good as curl for
> drop dead
> >> >> > simplicity. Don't get me wrong, it would be neat, but shouldn't be
> >> >> > considered in moderately well used memcached environments.
> >>
> >> >> > Regards,
> >>
> >> >> > Gavin
> >>
> >> >> > On Wed, Jul 28, 2010 at 8:43 AM, jsm <[email protected]> wrote:
> >>
> >> >> >> Anyone writing or planning to write a REST API for memcached?
> >> >> >> If no such plan, I would be interested in writing a REST API.
> >> >> >> Any suggestions, comments welcome.
>
j.s. mammen
Re: REST API
July 29, 2010 04:00PM
Folks, lets not get bogged down by REST defined by Roy Fielding in
2000.

My question was simple.
Here it is again, rephrased.

Do we need to implement a memcached layer whereby we can access the
cached objects by using HTTP protocol. Here is an example of getting a
cached object from a server
GET [server]/mc/object/id1

Hope the question is clearer now?

On Jul 29, 4:30 pm, Henrik Schröder <[email protected]> wrote:
> I would assume he's talking about making memcached expose some sort of
> simple web service api over http.
>
> Although, you could argue that both the ascii protocol and binary protocol
> are restful, the sure seem to me to fit the definition pretty closely.
>
> /Henrik
>
>
>
> On Thu, Jul 29, 2010 at 12:56, Aaron Stone <[email protected]> wrote:
> > What's a ReST protocol? ReST is a model.
>
> > On Wed, Jul 28, 2010 at 8:42 PM, jsm <[email protected]> wrote:
> > > What I meant was to add a REST protocol to memcached layer, just like
> > > you have a binary protocol and ascii.
> > > Its up to the user to decide which protocol to use when accessing
> > > memcached objects.
> > > Regards,
> > > J.S.Mammen
>
> > > On Jul 29, 1:49 am, Aaron Stone <[email protected]> wrote:
> > >> On Wed, Jul 28, 2010 at 8:37 AM, jsm <[email protected]> wrote:
>
> > >> > On Jul 28, 8:02 pm, Rajesh Nair <[email protected]> wrote:
> > >> >> Gavin,
>
> > >> >> If you go by the strict sense of word, HTTP protocol is not a
> > pre-requisite
> > >> >> for REST service.
> > >> >> It requires a protocol which supports linking entities through URIs.
> >  It is
> > >> >> very much possible to implement a RESTful service by coming up with
> > own URI
> > >> >> protocol for memcached messages
>
> > >> >> something like :
> > >> >> mc://<memcached-cluster>/messages/<key>
>
> > >> >> and the transport layer can be pretty much the same TCP to not add
> > any
> > >> >> overhead.
>
> > >> >> JSM,
>
> > >> >> What is the value-add you are looking from the RESTful version of the
> > >> >> memcached API?
>
> > >> > Basically to be able to use without binding to any particular
> > >> > language.
>
> > >> I read this as requesting memcached native support for structured
> > >> values (e.g. hashes, lists, etc.) -- is that what you meant?
>
> > >> Aaron
>
> > >> >> Regards,
> > >> >> Rajesh Nair
>
> > >> >> On Wed, Jul 28, 2010 at 8:13 PM, Gavin M. Roy <[email protected]>
> > wrote:
>
> > >> >> > Why add the HTTP protocol overhead?  REST/HTTP would add ~75Mbps of
> > >> >> > additional traffic at 100k gets per second by saying there's a
> > rough 100
> > >> >> > byte overhead per request over the ASCII protocol.  I base the 100
> > bytes by
> > >> >> > the HTTP GET request, minimal request headers and minimal response
> > >> >> > headers. The binary protocol is very terse in comparison to the
> > ASCII
> > >> >> > protocol.  In addition netcat or telnet works as good as curl for
> > drop dead
> > >> >> > simplicity.  Don't get me wrong, it would be neat, but shouldn't be
> > >> >> > considered in moderately well used memcached environments.
>
> > >> >> > Regards,
>
> > >> >> > Gavin
>
> > >> >> > On Wed, Jul 28, 2010 at 8:43 AM, jsm <[email protected]> wrote:
>
> > >> >> >> Anyone writing or planning to write a REST API for memcached?
> > >> >> >> If no such plan, I would be interested in writing a REST API.
> > >> >> >> Any suggestions, comments welcome.
Les Mikesell
Re: REST API
July 29, 2010 04:50PM
On 7/29/2010 8:54 AM, j.s. mammen wrote:
> Folks, lets not get bogged down by REST defined by Roy Fielding in
> 2000.
>
> My question was simple.
> Here it is again, rephrased.
>
> Do we need to implement a memcached layer whereby we can access the
> cached objects by using HTTP protocol. Here is an example of getting a
> cached object from a server
> GET [server]/mc/object/id1
>
> Hope the question is clearer now?

I can see an advantage if it let you get/put data using facilities most
languages have built-in instead of needing an add-on client library.
But, I don't see how you would handle the [server] part without
re-creating all of the logic in the client library anyway.

--
Les Mikesell
lesmikesell@gmail.com
Bill Moseley
Re: REST API
July 29, 2010 04:50PM
On Thu, Jul 29, 2010 at 6:54 AM, j.s. mammen <[email protected]> wrote:

> Folks, lets not get bogged down by REST defined by Roy Fielding in
> 2000.
>
> My question was simple.
> Here it is again, rephrased.
>
> Do we need to implement a memcached layer whereby we can access the
> cached objects by using HTTP protocol. Here is an example of getting a
> cached object from a server
> GET [server]/mc/object/id1
>

I'm finding this an interesting discussion because at work we have a number
of custom back-end services that all require custom clients wherever they
are used. In some ways they are similar to memcached where each server
needs a client and all the servers that use the client need to be configured
the same. The server-side of these services tend to be pretty simple just
serving requests, and the client code often makes some decision about which
back-end server to use from a list of configured servers (mostly for HA,
though).

The problem I see with our set up is that all the clients must be managed on
every machine that needs to talk to a back end service. And because of the
binary protocol any changes to the request (e.g. adding a new parameter)
requires that the clients get recompiled and pushed out to every machine.
Leaves room for breakage. Our existing clients are really bad at returning
any useful error messages, which http responses can do easily.

I've been pushing to move to a http protocol and get rid of the custom
clients. We have load balancers that are really good at http so we could
use those in front of a few machines so the client code is just used in
those few places -- thus, easier to manage. In some cases we could have we
clients use these services directly if it we used http instead of using our
web serves as proxies.

Now, the same discussion has happened at work about the binary protocol
being much more efficient. Indeed, that's true, but in some cases that
overhead is not that significant compared to the overall request. It may
add up, sure, but how critical that is depends on our needs - you don't here
much talk of moving http to a binary protocol, after all (although I'm sure
Google and Yahoo think about keeping their request and response headers as
small as possible).


Anyway, specific to this thread, would there be any reason memcached would
need to talk http? Isn't that something that a pretty simple wrapper could
add to your favorite memcached client? Certainly doesn't seem like
something memcached would need to support natively. It's up to the end-user
to determine if the http overhead is significant or not, no?



--
Bill Moseley
moseley@hank.org
John Reilly
Re: REST API
July 29, 2010 07:20PM
You could easily develop an http-to-memcached proxy to allow this. All the
partitioning logic could exist in the memcache client embedded in your
proxy. This might make some sense because then you would not have to
implement the partitioning logic into your clients. I would say the answer
to the question is no (memcached does not need to support http).


On Thu, Jul 29, 2010 at 6:54 AM, j.s. mammen <[email protected]> wrote:

> Folks, lets not get bogged down by REST defined by Roy Fielding in
> 2000.
>
> My question was simple.
> Here it is again, rephrased.
>
> Do we need to implement a memcached layer whereby we can access the
> cached objects by using HTTP protocol. Here is an example of getting a
> cached object from a server
> GET [server]/mc/object/id1
>
> Hope the question is clearer now?
>
> On Jul 29, 4:30 pm, Henrik Schröder <[email protected]> wrote:
> > I would assume he's talking about making memcached expose some sort of
> > simple web service api over http.
> >
> > Although, you could argue that both the ascii protocol and binary
> protocol
> > are restful, the sure seem to me to fit the definition pretty closely.
> >
> > /Henrik
> >
> >
> >
> > On Thu, Jul 29, 2010 at 12:56, Aaron Stone <[email protected]> wrote:
> > > What's a ReST protocol? ReST is a model.
> >
> > > On Wed, Jul 28, 2010 at 8:42 PM, jsm <[email protected]> wrote:
> > > > What I meant was to add a REST protocol to memcached layer, just like
> > > > you have a binary protocol and ascii.
> > > > Its up to the user to decide which protocol to use when accessing
> > > > memcached objects.
> > > > Regards,
> > > > J.S.Mammen
> >
> > > > On Jul 29, 1:49 am, Aaron Stone <[email protected]> wrote:
> > > >> On Wed, Jul 28, 2010 at 8:37 AM, jsm <[email protected]> wrote:
> >
> > > >> > On Jul 28, 8:02 pm, Rajesh Nair <[email protected]> wrote:
> > > >> >> Gavin,
> >
> > > >> >> If you go by the strict sense of word, HTTP protocol is not a
> > > pre-requisite
> > > >> >> for REST service.
> > > >> >> It requires a protocol which supports linking entities through
> URIs.
> > > It is
> > > >> >> very much possible to implement a RESTful service by coming up
> with
> > > own URI
> > > >> >> protocol for memcached messages
> >
> > > >> >> something like :
> > > >> >> mc://<memcached-cluster>/messages/<key>
> >
> > > >> >> and the transport layer can be pretty much the same TCP to not
> add
> > > any
> > > >> >> overhead.
> >
> > > >> >> JSM,
> >
> > > >> >> What is the value-add you are looking from the RESTful version of
> the
> > > >> >> memcached API?
> >
> > > >> > Basically to be able to use without binding to any particular
> > > >> > language.
> >
> > > >> I read this as requesting memcached native support for structured
> > > >> values (e.g. hashes, lists, etc.) -- is that what you meant?
> >
> > > >> Aaron
> >
> > > >> >> Regards,
> > > >> >> Rajesh Nair
> >
> > > >> >> On Wed, Jul 28, 2010 at 8:13 PM, Gavin M. Roy <
> [email protected]>
> > > wrote:
> >
> > > >> >> > Why add the HTTP protocol overhead? REST/HTTP would add
> ~75Mbps of
> > > >> >> > additional traffic at 100k gets per second by saying there's a
> > > rough 100
> > > >> >> > byte overhead per request over the ASCII protocol. I base the
> 100
> > > bytes by
> > > >> >> > the HTTP GET request, minimal request headers and minimal
> response
> > > >> >> > headers. The binary protocol is very terse in comparison to the
> > > ASCII
> > > >> >> > protocol. In addition netcat or telnet works as good as curl
> for
> > > drop dead
> > > >> >> > simplicity. Don't get me wrong, it would be neat, but
> shouldn't be
> > > >> >> > considered in moderately well used memcached environments.
> >
> > > >> >> > Regards,
> >
> > > >> >> > Gavin
> >
> > > >> >> > On Wed, Jul 28, 2010 at 8:43 AM, jsm <[email protected]>
> wrote:
> >
> > > >> >> >> Anyone writing or planning to write a REST API for memcached?
> > > >> >> >> If no such plan, I would be interested in writing a REST API..
> > > >> >> >> Any suggestions, comments welcome.
>
Marc Bollinger
Re: REST API
July 29, 2010 07:50PM
Along those lines, I just did some digging and it looks like there's a
third-party nginx plugin that supports using REST to address the cache at
the proxy level: http://wiki.nginx.org/NginxHttpMemcModule, and I agree with
the others that's where you'd want to place something like this. Note: I
just found the above link now, and am in no way advocating its use in
particular, just that there are already efforts to do this in more
appropriate layers.

- Marc

On Thu, Jul 29, 2010 at 10:17 AM, John Reilly <[email protected]> wrote:

> You could easily develop an http-to-memcached proxy to allow this. All the
> partitioning logic could exist in the memcache client embedded in your
> proxy. This might make some sense because then you would not have to
> implement the partitioning logic into your clients. I would say the answer
> to the question is no (memcached does not need to support http).
>
>
> On Thu, Jul 29, 2010 at 6:54 AM, j.s. mammen <[email protected]> wrote:
>
>> Folks, lets not get bogged down by REST defined by Roy Fielding in
>> 2000.
>>
>> My question was simple.
>> Here it is again, rephrased.
>>
>> Do we need to implement a memcached layer whereby we can access the
>> cached objects by using HTTP protocol. Here is an example of getting a
>> cached object from a server
>> GET [server]/mc/object/id1
>>
>> Hope the question is clearer now?
>>
>> On Jul 29, 4:30 pm, Henrik Schröder <[email protected]> wrote:
>> > I would assume he's talking about making memcached expose some sort of
>> > simple web service api over http.
>> >
>> > Although, you could argue that both the ascii protocol and binary
>> protocol
>> > are restful, the sure seem to me to fit the definition pretty closely.
>> >
>> > /Henrik
>> >
>> >
>> >
>> > On Thu, Jul 29, 2010 at 12:56, Aaron Stone <[email protected]> wrote:
>> > > What's a ReST protocol? ReST is a model.
>> >
>> > > On Wed, Jul 28, 2010 at 8:42 PM, jsm <[email protected]> wrote:
>> > > > What I meant was to add a REST protocol to memcached layer, just
>> like
>> > > > you have a binary protocol and ascii.
>> > > > Its up to the user to decide which protocol to use when accessing
>> > > > memcached objects.
>> > > > Regards,
>> > > > J.S.Mammen
>> >
>> > > > On Jul 29, 1:49 am, Aaron Stone <[email protected]> wrote:
>> > > >> On Wed, Jul 28, 2010 at 8:37 AM, jsm <[email protected]> wrote:
>> >
>> > > >> > On Jul 28, 8:02 pm, Rajesh Nair <[email protected]>
>> wrote:
>> > > >> >> Gavin,
>> >
>> > > >> >> If you go by the strict sense of word, HTTP protocol is not a
>> > > pre-requisite
>> > > >> >> for REST service.
>> > > >> >> It requires a protocol which supports linking entities through
>> URIs.
>> > > It is
>> > > >> >> very much possible to implement a RESTful service by coming up
>> with
>> > > own URI
>> > > >> >> protocol for memcached messages
>> >
>> > > >> >> something like :
>> > > >> >> mc://<memcached-cluster>/messages/<key>
>> >
>> > > >> >> and the transport layer can be pretty much the same TCP to not
>> add
>> > > any
>> > > >> >> overhead.
>> >
>> > > >> >> JSM,
>> >
>> > > >> >> What is the value-add you are looking from the RESTful version
>> of the
>> > > >> >> memcached API?
>> >
>> > > >> > Basically to be able to use without binding to any particular
>> > > >> > language.
>> >
>> > > >> I read this as requesting memcached native support for structured
>> > > >> values (e.g. hashes, lists, etc.) -- is that what you meant?
>> >
>> > > >> Aaron
>> >
>> > > >> >> Regards,
>> > > >> >> Rajesh Nair
>> >
>> > > >> >> On Wed, Jul 28, 2010 at 8:13 PM, Gavin M. Roy <
>> [email protected]>
>> > > wrote:
>> >
>> > > >> >> > Why add the HTTP protocol overhead? REST/HTTP would add
>> ~75Mbps of
>> > > >> >> > additional traffic at 100k gets per second by saying there's a
>> > > rough 100
>> > > >> >> > byte overhead per request over the ASCII protocol. I base the
>> 100
>> > > bytes by
>> > > >> >> > the HTTP GET request, minimal request headers and minimal
>> response
>> > > >> >> > headers. The binary protocol is very terse in comparison to
>> the
>> > > ASCII
>> > > >> >> > protocol. In addition netcat or telnet works as good as curl
>> for
>> > > drop dead
>> > > >> >> > simplicity. Don't get me wrong, it would be neat, but
>> shouldn't be
>> > > >> >> > considered in moderately well used memcached environments.
>> >
>> > > >> >> > Regards,
>> >
>> > > >> >> > Gavin
>> >
>> > > >> >> > On Wed, Jul 28, 2010 at 8:43 AM, jsm <[email protected]>
>> wrote:
>> >
>> > > >> >> >> Anyone writing or planning to write a REST API for memcached?
>> > > >> >> >> If no such plan, I would be interested in writing a REST API.
>> > > >> >> >> Any suggestions, comments welcome.
>>
>
Aaron Stone
Re: REST API
July 29, 2010 09:20PM
We have just such a thing at Linden Lab, too. I was appalled when I
heard about it, but then looked under the hood at a bunch of very
convenient features. And it's pretty nice to be able to check your
caches with curl.

The mapping from memcached onto HTTP is really non-obvious. HTTP can
be a beast of a protocol -- caching, content-types, the various verbs,
and so on. These are really decisions that you'd want to make within
your own organization.

Aaron


On Thu, Jul 29, 2010 at 10:17 AM, John Reilly <[email protected]> wrote:
> You could easily develop an http-to-memcached proxy to allow this.  All the
> partitioning logic could exist in the memcache client embedded in your
> proxy.  This might make some sense because then you would not have to
> implement the partitioning logic into your clients.  I would say the answer
> to the question is no (memcached does not need to support http).
>
>
> On Thu, Jul 29, 2010 at 6:54 AM, j.s. mammen <[email protected]> wrote:
>>
>> Folks, lets not get bogged down by REST defined by  Roy Fielding in
>> 2000.
>>
>> My question was simple.
>> Here it is again, rephrased.
>>
>> Do we need to implement a memcached layer whereby we can access the
>> cached objects by using HTTP protocol. Here is an example of getting a
>> cached object from a server
>> GET [server]/mc/object/id1
>>
>> Hope the question is clearer now?
>>
>> On Jul 29, 4:30 pm, Henrik Schröder <[email protected]> wrote:
>> > I would assume he's talking about making memcached expose some sort of
>> > simple web service api over http.
>> >
>> > Although, you could argue that both the ascii protocol and binary
>> > protocol
>> > are restful, the sure seem to me to fit the definition pretty closely.
>> >
>> > /Henrik
>> >
>> >
>> >
>> > On Thu, Jul 29, 2010 at 12:56, Aaron Stone <[email protected]> wrote:
>> > > What's a ReST protocol? ReST is a model.
>> >
>> > > On Wed, Jul 28, 2010 at 8:42 PM, jsm <[email protected]> wrote:
>> > > > What I meant was to add a REST protocol to memcached layer, just
>> > > > like
>> > > > you have a binary protocol and ascii.
>> > > > Its up to the user to decide which protocol to use when accessing
>> > > > memcached objects.
>> > > > Regards,
>> > > > J.S.Mammen
>> >
>> > > > On Jul 29, 1:49 am, Aaron Stone <[email protected]> wrote:
>> > > >> On Wed, Jul 28, 2010 at 8:37 AM, jsm <[email protected]> wrote:
>> >
>> > > >> > On Jul 28, 8:02 pm, Rajesh Nair <[email protected]> wrote:
>> > > >> >> Gavin,
>> >
>> > > >> >> If you go by the strict sense of word, HTTP protocol is not a
>> > > pre-requisite
>> > > >> >> for REST service.
>> > > >> >> It requires a protocol which supports linking entities through
>> > > >> >> URIs.
>> > >  It is
>> > > >> >> very much possible to implement a RESTful service by coming up
>> > > >> >> with
>> > > own URI
>> > > >> >> protocol for memcached messages
>> >
>> > > >> >> something like :
>> > > >> >> mc://<memcached-cluster>/messages/<key>
>> >
>> > > >> >> and the transport layer can be pretty much the same TCP to not
>> > > >> >> add
>> > > any
>> > > >> >> overhead.
>> >
>> > > >> >> JSM,
>> >
>> > > >> >> What is the value-add you are looking from the RESTful version
>> > > >> >> of the
>> > > >> >> memcached API?
>> >
>> > > >> > Basically to be able to use without binding to any particular
>> > > >> > language.
>> >
>> > > >> I read this as requesting memcached native support for structured
>> > > >> values (e.g. hashes, lists, etc.) -- is that what you meant?
>> >
>> > > >> Aaron
>> >
>> > > >> >> Regards,
>> > > >> >> Rajesh Nair
>> >
>> > > >> >> On Wed, Jul 28, 2010 at 8:13 PM, Gavin M. Roy
>> > > >> >> <[email protected]>
>> > > wrote:
>> >
>> > > >> >> > Why add the HTTP protocol overhead?  REST/HTTP would add
>> > > >> >> > ~75Mbps of
>> > > >> >> > additional traffic at 100k gets per second by saying there's a
>> > > rough 100
>> > > >> >> > byte overhead per request over the ASCII protocol.  I base the
>> > > >> >> > 100
>> > > bytes by
>> > > >> >> > the HTTP GET request, minimal request headers and minimal
>> > > >> >> > response
>> > > >> >> > headers. The binary protocol is very terse in comparison to
>> > > >> >> > the
>> > > ASCII
>> > > >> >> > protocol.  In addition netcat or telnet works as good as curl
>> > > >> >> > for
>> > > drop dead
>> > > >> >> > simplicity.  Don't get me wrong, it would be neat, but
>> > > >> >> > shouldn't be
>> > > >> >> > considered in moderately well used memcached environments.
>> >
>> > > >> >> > Regards,
>> >
>> > > >> >> > Gavin
>> >
>> > > >> >> > On Wed, Jul 28, 2010 at 8:43 AM, jsm <[email protected]>
>> > > >> >> > wrote:
>> >
>> > > >> >> >> Anyone writing or planning to write a REST API for memcached?
>> > > >> >> >> If no such plan, I would be interested in writing a REST API.
>> > > >> >> >> Any suggestions, comments welcome.
>
>
dormando
Re: REST API
August 01, 2010 12:40AM
On Thu, 29 Jul 2010, j.s. mammen wrote:

> Folks, lets not get bogged down by REST defined by Roy Fielding in
> 2000.
>
> My question was simple.
> Here it is again, rephrased.
>
> Do we need to implement a memcached layer whereby we can access the
> cached objects by using HTTP protocol. Here is an example of getting a
> cached object from a server
> GET [server]/mc/object/id1
>
> Hope the question is clearer now?

I'm not sure how this thread has gone on for so long? Adding a new API
won't fix this problem aside from happening to force you to write a new
set of clients that use the same hashing model and don't mess with the
blob on their way back.

We should be better about building a compatibility standard into the
clients so they have the ability to use matching hashing algorithms to
blind fetch binary blobs. Bonus points for standardizing compression.

If you're talking about using a raw REST approach on top of thin air
you'll be missing all of the client features that make memcached what it
is; a distributed cache.

If you take your existing java/C# clients and use a single server instead
of multiple, you should be able to fetch the same XML blobs through each
without having them mangle it?
Sorry, only registered users may post in this forum.

Click here to login