Welcome! Log In Create A New Profile

Advanced

memcached-1.4.5 without multithread support (or with `-t 0')

Posted by tudorica 
The current memcached-1.4.5 version I downloaded appears to always be
built with multithreaded support (unless something subtle is happening
during configure that I haven't noticed). Would it be OK if I
submitted a patch that allows a single-threaded memcached build? Here
is the rationale: instead of peppering the code with expensive user-
space locking and events (e.g. pthread_mutex_lock, and the producer-
consumers), why not just have the alternative to deploy N instances of
plain singlethreaded memcached distinct/isolated processes, where N is
the number of available CPUs (e.g. each instance on a different port)?
Each such memcached process will utilize 1/Nth of the memory that a
`memcached -t N' would have otherwise utilized, and there would be no
user-space locking (unlike when memcached is launched with `-t 1'),
i.e. all locking is performed by the in-kernel network stack when
traffic is demuxed onto the N sockets. Sure, this would mean that the
clients will have to deal with more memcached instances (albeit
virtual), but my impression is that this is already the norm (see the
consistent hashing libraries like libketama), and proper hashing (in
the client) to choose the target memcached server (ip:port) is already
commonplace. The only down-side I may envision is clients utilizing
non-uniform hash functions to choose the target memcached server, but
that's their problem.

Regards,
T
We took it out for a reason, + if you run with -t 1 you won't really see
contention. 'Cuz it's running single threaded and using futexes under
linux. Those don't have much of a performance hit until you do contend.

I know some paper just came out which showed people using multiple
memcached instances and scaling some kernel locks, along with the whole
redis "ONE INSTANCE PER CORE IS AWESOME GUYS" thing.

But I'd really love it if you would prove that this is better, and prove
that there is no loss when running multiple instances. This is all
conjecture.

I'm slowly chipping away at the 1.6 branch and some lock scaling patches,
which feels a lot more productive than anecdotally naysaying progress.

memcached -t 4 will run 140,000 sets and 300,000+ gets per second on a box
of mine. An unrefined patch on an older version from trond gets that to
400,000 sets and 630,000 gets. I expect to get that to be a bit higher.

I assume you have some 10GE memcached instances pushing 5gbps+ of traffic
in order for this patch to be worth your time?

Or are all of your keys 1 byte and you're fetching 1 million of them per
second?

On Mon, 4 Oct 2010, tudorica wrote:

> The current memcached-1.4.5 version I downloaded appears to always be
> built with multithreaded support (unless something subtle is happening
> during configure that I haven't noticed). Would it be OK if I
> submitted a patch that allows a single-threaded memcached build? Here
> is the rationale: instead of peppering the code with expensive user-
> space locking and events (e.g. pthread_mutex_lock, and the producer-
> consumers), why not just have the alternative to deploy N instances of
> plain singlethreaded memcached distinct/isolated processes, where N is
> the number of available CPUs (e.g. each instance on a different port)?
> Each such memcached process will utilize 1/Nth of the memory that a
> `memcached -t N' would have otherwise utilized, and there would be no
> user-space locking (unlike when memcached is launched with `-t 1'),
> i.e. all locking is performed by the in-kernel network stack when
> traffic is demuxed onto the N sockets. Sure, this would mean that the
> clients will have to deal with more memcached instances (albeit
> virtual), but my impression is that this is already the norm (see the
> consistent hashing libraries like libketama), and proper hashing (in
> the client) to choose the target memcached server (ip:port) is already
> commonplace. The only down-side I may envision is clients utilizing
> non-uniform hash functions to choose the target memcached server, but
> that's their problem.
>
> Regards,
> T
>
I agree that if you have futexes on your platform and you don't contend
(i.e. don't even have to call into the kernel) the overhead is small(er),
however, there is also the overhead between the producer-consumer contexts,
namely the event base and the `-t 1' thread (again, unless I misread the
code, in which case I apologize).

I am not sure what paper you mean, but my thesis did deal with the
scalability of sockets (raw sockets in particular) and how a single-producer
multiple-consumer approach doesn't scale nearly as good as a ``one instance
per core." Sure enough the details are more gory and I won't get into them.
I'll try to find some time to test and compare and get back to you with some
numbers. By the way, I do ot see how there would be ``loss when running
multiple instances'' since your traffic is disjoint, and if you do have
``loss'' then your OS kernel is broken.

As a matter of fact, I do have a 10Gbps machine, actually two of them, each
with a dual socket Xeon X5570 and 2x Myri-10G NICs that I was planning on
using for tests. Would you be so kind to tell me if there's any standard
performance test suite for memcached that is typically used? Or should I
just write my own trivial client---in particular, as you mentioned, I am
interested in the scalability of memcached (-t 4 versus proper
singlethreaded/multi-process) with respect to the key and/or value size.

Regards,
T

On Mon, Oct 4, 2010 at 4:21 PM, dormando <[email protected]> wrote:

> We took it out for a reason, + if you run with -t 1 you won't really see
> contention. 'Cuz it's running single threaded and using futexes under
> linux. Those don't have much of a performance hit until you do contend.
>
> I know some paper just came out which showed people using multiple
> memcached instances and scaling some kernel locks, along with the whole
> redis "ONE INSTANCE PER CORE IS AWESOME GUYS" thing.
>
> But I'd really love it if you would prove that this is better, and prove
> that there is no loss when running multiple instances. This is all
> conjecture.
>
> I'm slowly chipping away at the 1.6 branch and some lock scaling patches,
> which feels a lot more productive than anecdotally naysaying progress.
>
> memcached -t 4 will run 140,000 sets and 300,000+ gets per second on a box
> of mine. An unrefined patch on an older version from trond gets that to
> 400,000 sets and 630,000 gets. I expect to get that to be a bit higher.
>
> I assume you have some 10GE memcached instances pushing 5gbps+ of traffic
> in order for this patch to be worth your time?
>
> Or are all of your keys 1 byte and you're fetching 1 million of them per
> second?
>
> On Mon, 4 Oct 2010, tudorica wrote:
>
> > The current memcached-1.4.5 version I downloaded appears to always be
> > built with multithreaded support (unless something subtle is happening
> > during configure that I haven't noticed). Would it be OK if I
> > submitted a patch that allows a single-threaded memcached build? Here
> > is the rationale: instead of peppering the code with expensive user-
> > space locking and events (e.g. pthread_mutex_lock, and the producer-
> > consumers), why not just have the alternative to deploy N instances of
> > plain singlethreaded memcached distinct/isolated processes, where N is
> > the number of available CPUs (e.g. each instance on a different port)?
> > Each such memcached process will utilize 1/Nth of the memory that a
> > `memcached -t N' would have otherwise utilized, and there would be no
> > user-space locking (unlike when memcached is launched with `-t 1'),
> > i.e. all locking is performed by the in-kernel network stack when
> > traffic is demuxed onto the N sockets. Sure, this would mean that the
> > clients will have to deal with more memcached instances (albeit
> > virtual), but my impression is that this is already the norm (see the
> > consistent hashing libraries like libketama), and proper hashing (in
> > the client) to choose the target memcached server (ip:port) is already
> > commonplace. The only down-side I may envision is clients utilizing
> > non-uniform hash functions to choose the target memcached server, but
> > that's their problem.
> >
> > Regards,
> > T
> >
>
Hey,

No actually... in -t 1 mode the only producer/consumer is between the
accept thread and the worker thread. Once a connection is open the socket
events are local to the thread. Persistent connections would remove almost
all of the overhead aside from the futexes existing.

There's also work we're presently doing to scale that lock, but it's not
really necessary as nobody hits that limit.

There is loss... There's a lot of loss to running multiple instances:

- Management overhead of running multiple instances instead of one (yes I
know blah blah blah go hire a junior guy and have him set it up right.
it's harder than it looks).

- Less efficient multigets. "Natural" multigets are more efficient if you
use fewer instances. Less natural multigets don't care nearly as much, but
can suffer if you accidentally cluster too much data on a single (too
small) instance. I *have* seen this happen.

- Socket overhead. This is a big deal if you're running 8x or more
instances on a single box. Now in order to fetch individual keys spread
about, both clients and servers need to manage *much, much* more socket
"crap". This includes the DMA allocation overhead I think you noted in
your paper. If you watch /proc/slabinfo between maintaining 100 sockets or
800 sockets you'll see that there's a large memory loss, and the kernel
has to do a lot more work to manage all of that extra shuffling. Memcached
itself will lose memory to connection states and buffers.

You can tune it down with various /proc knobs but adding 8x+ connection
overhead is a very real loss.

On the note of a performance test suite... I've been trying to fix one up
to be that sort of thing. I have Matt's "memcachetest" able to saturate
memcached instead of itself, and I've gotten the redis test to do some
damage: http://dormando.livejournal.com/525147.html

....but it needs more work over the next few weeks.

Also, finally, I'm asking about a real actual usage. Academically it's
interesting that memcached should be able to run 1million+ sets per second
off of a 48 core box, but reality always *always* pins performance to
other areas.

Namely:

1) memory overhead. the more crap you can stuff in memcached the faster
your app goes per dollar.
2) other very important scientific-style advancements in cache usage are
more fixated on benefits shown from the binary protocol. Utilizing
multiset and asyncronous get/set stacking can shave real milliseconds off
of serving real responses.

Except people aren't using binprot, and they're finding bugs when they do
(it seems to generate more packets? But I haven't gotten that far yet in
fixing the benchmarks).

I wish we could focus over there for a while.

Btw, your paper was very good. There's no question that a lot of the
kernel-level fixes in there will have real world benefits.

I just question the hyperfocus here on an aspect of a practical
application that is never the actual problem.

On Mon, 4 Oct 2010, Tudor Marian wrote:

> I agree that if you have futexes on your platform and you don't contend (i.e. don't even have to call into the kernel) the overhead is small(er), however, there is also the overhead between the
> producer-consumer contexts, namely the event base and the `-t 1' thread (again, unless I misread the code, in which case I apologize).
>
> I am not sure what paper you mean, but my thesis did deal with the scalability of sockets (raw sockets in particular) and how a single-producer multiple-consumer approach doesn't scale nearly as good as a
> ``one instance per core." Sure enough the details are more gory and I won't get into them.  I'll try to find some time to test and compare and get back to you with some numbers.  By the way, I do ot see
> how there would be ``loss when running multiple instances'' since your traffic is disjoint, and if you do have ``loss'' then your OS kernel is broken.
>
> As a matter of fact, I do have a 10Gbps machine, actually two of them, each with a dual socket Xeon X5570 and 2x Myri-10G NICs that I was planning on using for tests. Would you be so kind to tell me if
> there's any standard performance test suite for memcached that is typically used? Or should I just write my own trivial client---in particular, as you mentioned, I am interested in the scalability of
> memcached (-t 4 versus proper singlethreaded/multi-process) with respect to the key and/or value size.
>
> Regards,
> T
>
> On Mon, Oct 4, 2010 at 4:21 PM, dormando <[email protected]> wrote:
> We took it out for a reason, + if you run with -t 1 you won't really see
> contention. 'Cuz it's running single threaded and using futexes under
> linux. Those don't have much of a performance hit until you do contend.
>
> I know some paper just came out which showed people using multiple
> memcached instances and scaling some kernel locks, along with the whole
> redis "ONE INSTANCE PER CORE IS AWESOME GUYS" thing.
>
> But I'd really love it if you would prove that this is better, and prove
> that there is no loss when running multiple instances. This is all
> conjecture.
>
> I'm slowly chipping away at the 1.6 branch and some lock scaling patches,
> which feels a lot more productive than anecdotally naysaying progress.
>
> memcached -t 4 will run 140,000 sets and 300,000+ gets per second on a box
> of mine. An unrefined patch on an older version from trond gets that to
> 400,000 sets and 630,000 gets. I expect to get that to be a bit higher.
>
> I assume you have some 10GE memcached instances pushing 5gbps+ of traffic
> in order for this patch to be worth your time?
>
> Or are all of your keys 1 byte and you're fetching 1 million of them per
> second?
>
> On Mon, 4 Oct 2010, tudorica wrote:
>
> > The current memcached-1.4.5 version I downloaded appears to always be
> > built with multithreaded support (unless something subtle is happening
> > during configure that I haven't noticed).  Would it be OK if I
> > submitted a patch that allows a single-threaded memcached build? Here
> > is the rationale: instead of peppering the code with expensive user-
> > space locking and events (e.g. pthread_mutex_lock, and the producer-
> > consumers), why not just have the alternative to deploy N instances of
> > plain singlethreaded memcached distinct/isolated processes, where N is
> > the number of available CPUs (e.g. each instance on a different port)?
> > Each such memcached process will utilize 1/Nth of the memory that a
> > `memcached -t N' would have otherwise utilized, and there would be no
> > user-space locking (unlike when memcached is launched with `-t 1'),
> > i.e. all locking is performed by the in-kernel network stack when
> > traffic is demuxed onto the N sockets.  Sure, this would mean that the
> > clients will have to deal with more memcached instances (albeit
> > virtual), but my impression is that this is already the norm (see the
> > consistent hashing libraries like libketama), and proper hashing (in
> > the client) to choose the target memcached server (ip:port) is already
> > commonplace.  The only down-side I may envision is clients utilizing
> > non-uniform hash functions to choose the target memcached server, but
> > that's their problem.
> >
> > Regards,
> > T
> >
>
>
>
>
Sorry, only registered users may post in this forum.

Click here to login