Welcome! Log In Create A New Profile

Advanced

Memcached make many SI (Software Interrupts)

Posted by ARccRA 
ARccRA
Memcached make many SI (Software Interrupts)
September 27, 2010 05:40PM
Hi

We have an 2 x quad core server with 32 gb ram. If many clients
connect to this server (only memcached runs on it) the first core run
to nearly 100 % use by si (software interrups) and so some client
can't reach the server.
Memcached runs currently with 4 threads and with version (1.4.2). All
other cores have 70 % idle so I ask me is there a possibility to
improve the performance?
dormando
Re: Memcached make many SI (Software Interrupts)
September 27, 2010 08:40PM
> We have an 2 x quad core server with 32 gb ram. If many clients
> connect to this server (only memcached runs on it) the first core run
> to nearly 100 % use by si (software interrups) and so some client
> can't reach the server.
> Memcached runs currently with 4 threads and with version (1.4.2). All
> other cores have 70 % idle so I ask me is there a possibility to
> improve the performance?
>

This is an issue with how your network interrupts are being routed, not
with how memcached is being threaded.

Wish I had some good links offhand for this, because it's a little obscure
to deal with. In short; you'll want to balance your network interrupts
across cores. Google for blog posts about smp_affinity for network cards
and irqbalance (which poorly tries to automatically do this).

Depending on how many NIC's you have and if it's multiqueued or not you'll
have to tune it differently. Linux 2.6.35 has some features for extending
the speed of single-queued NIC's (find the pages discussing it on
kernelnewbies.org).
Jay Paroline
Re: Memcached make many SI (Software Interrupts)
September 28, 2010 09:00PM
We've run into this exact same issue and narrowed it down to the NIC,
but don't really know where to go from there. I'm going to look up
Dormando's suggestions but if anyone else has experience with this and
can point us in the right direction, it would be greatly appreciated.

Thanks,

Jay

On Sep 27, 2:34 pm, dormando <[email protected]> wrote:
> > We have an 2 x quad core server with 32 gb ram. If many clients
> > connect to this server (only memcached runs on it) the first core run
> > to nearly 100 % use by si (software interrups) and so some client
> > can't reach the server.
> > Memcached runs currently with 4 threads and with version (1.4.2). All
> > other cores have 70 % idle so I ask me is there a possibility to
> > improve the performance?
>
> This is an issue with how your network interrupts are being routed, not
> with how memcached is being threaded.
>
> Wish I had some good links offhand for this, because it's a little obscure
> to deal with. In short; you'll want to balance your network interrupts
> across cores. Google for blog posts about smp_affinity for network cards
> and irqbalance (which poorly tries to automatically do this).
>
> Depending on how many NIC's you have and if it's multiqueued or not you'll
> have to tune it differently. Linux 2.6.35 has some features for extending
> the speed of single-queued NIC's (find the pages discussing it on
> kernelnewbies.org).
Rohit Karlupia
Re: Memcached make many SI (Software Interrupts)
September 28, 2010 11:00PM
This is a bit complicated and I still don't understand it fully, but here
are few pointers. BTW all that follows has nothing specific to memcached.

1) cat /proc/interrupts
This will show the list of which core(s) are handling the interrupts for
given devices.
Check eth0/eth1 whichever you are using. The first column is the irq id.

2) cat /proc/irq/<irq-number>/smp_affinity
If things are good, you should be able to do "echo "xxxx"
> /proc/irq/<irq-number>/smp_affinity to modify the smb affinity. You can
do a bit of googling to find about this.

If this doesn't works, it is either your NIC, or its driver or the kernel.
For our hardware centOS (2.6.18) was not spreading the interrupts but with
Ubuntu (2.6.32) this worked.
I have seen this working with kernel 2.6.24 with Intel NICs.

If nothing else works try setting up multiple memcached instances on the
same machine using different NICs. For example if you have 4 NICs, create 4
memcached instances each with 7/8GB memory. Usually OS will assign different
NICs to different cores (check with cat /proc/interrupts). So let memcached1
listen on eth0, memcached2 on eth1...etc.

Hope this helps.

thanks!
rohitk



On Wed, Sep 29, 2010 at 12:28 AM, Jay Paroline <[email protected]> wrote:

> We've run into this exact same issue and narrowed it down to the NIC,
> but don't really know where to go from there. I'm going to look up
> Dormando's suggestions but if anyone else has experience with this and
> can point us in the right direction, it would be greatly appreciated.
>
> Thanks,
>
> Jay
>
> On Sep 27, 2:34 pm, dormando <[email protected]> wrote:
> > > We have an 2 x quad core server with 32 gb ram. If many clients
> > > connect to this server (only memcached runs on it) the first core run
> > > to nearly 100 % use by si (software interrups) and so some client
> > > can't reach the server.
> > > Memcached runs currently with 4 threads and with version (1.4.2). All
> > > other cores have 70 % idle so I ask me is there a possibility to
> > > improve the performance?
> >
> > This is an issue with how your network interrupts are being routed, not
> > with how memcached is being threaded.
> >
> > Wish I had some good links offhand for this, because it's a little
> obscure
> > to deal with. In short; you'll want to balance your network interrupts
> > across cores. Google for blog posts about smp_affinity for network cards
> > and irqbalance (which poorly tries to automatically do this).
> >
> > Depending on how many NIC's you have and if it's multiqueued or not
> you'll
> > have to tune it differently. Linux 2.6.35 has some features for extending
> > the speed of single-queued NIC's (find the pages discussing it on
> > kernelnewbies.org).
>
dormando
Re: Memcached make many SI (Software Interrupts)
September 28, 2010 11:10PM
It's a big topic, here's the stuff to google:

- cat /proc/interrupts - look for "eth0", "eth1", etc. If you have one
interrupt assigned to eth0, you have a single-queue NIC.

If you have many interrupts that look like "eth0-0, eth0-1", etc, you have
a multi-queue NIC. These can have their interrupts spread out more.

Use either irqbalance (probably a bad idea) or echoing values into
/proc/irq/nn/smp_affinity (google for help with this), to spread out the
interrupts. You can then experiment with using `taskset` to bind memcached
to the same CPUs as the interrupts, or to different CPU's and see if the
throughput changes.

- look up "linux sysctl network tuning"

This tends to give you crap like this:
net.ipv4.ip_local_port_range = 9500 65536
net.core.rmem_max = 1048576
net.core.wmem_max = 1048576
net.ipv4.tcp_rmem = 4096 87380 4194304
net.ipv4.tcp_wmem = 4096 43690 4194304
net.ipv4.tcp_fin_timeout = 30
net.ipv4.tcp_syncookies = 0
net.ipv4.tcp_max_orphans = 65536
net.ipv4.tcp_max_syn_backlog = 16384
#net.ipv4.tcp_synack_retries = 2
net.core.netdev_budget=1000
net.ipv4.tcp_max_tw_buckets = 1512000

Which will vary by if you're using persistent connections or not (ie; how
high the turnover is). Don't blindly copy/paste this stuff.

- Read up on all the "ethtool" options available for your NIC. ensure the
defaults work.

NICs are all configured for a balance between packet latency and
throughput. The more interrupts they coalesce within the driver, the
higher potential latency of packet return. This can be really hard to go
through and the settings will vary by every NIC you fiddle with. I've
managed to get differences of 40k-60k pps by tuning these values. Often
the latency doesn't get much worse.

- Use a recent kernel.

On a particular piece of recent hardware I doubled packet throughput by
moving from 2.6.27 to 2.6.32. I was not able to push 2.6.18 hard without
having it drop networking.

- Use a more recent kernel

http://kernelnewbies.org/Linux_2_6_35#head-94daf753b96280181e79a71ca4bb7f7a423e302a

I haven't played with this much yet, but it looks like a BFD, especially
if you're stuck with single-queue NIC's.

- Get a better NIC.

10ge NICs have awesomesauce features for shoveling more packets around.
There're different levels of how awesome straight-gbit NICs are as well. I
like the high-end intels more than most of the broadcoms, for instance.

- Don't think running multiple instances of memcached will make much of a
difference.

Maybe run more threads though, or try pinning them to a set of CPU's or a
particular CPU.

On Tue, 28 Sep 2010, Jay Paroline wrote:

> We've run into this exact same issue and narrowed it down to the NIC,
> but don't really know where to go from there. I'm going to look up
> Dormando's suggestions but if anyone else has experience with this and
> can point us in the right direction, it would be greatly appreciated.
>
> Thanks,
>
> Jay
>
> On Sep 27, 2:34 pm, dormando <[email protected]> wrote:
> > > We have an 2 x quad core server with 32 gb ram. If many clients
> > > connect to this server (only memcached runs on it) the first core run
> > > to nearly 100 % use by si (software interrups) and so some client
> > > can't reach the server.
> > > Memcached runs currently with 4 threads and with version (1.4.2). All
> > > other cores have 70 % idle so I ask me is there a possibility to
> > > improve the performance?
> >
> > This is an issue with how your network interrupts are being routed, not
> > with how memcached is being threaded.
> >
> > Wish I had some good links offhand for this, because it's a little obscure
> > to deal with. In short; you'll want to balance your network interrupts
> > across cores. Google for blog posts about smp_affinity for network cards
> > and irqbalance (which poorly tries to automatically do this).
> >
> > Depending on how many NIC's you have and if it's multiqueued or not you'll
> > have to tune it differently. Linux 2.6.35 has some features for extending
> > the speed of single-queued NIC's (find the pages discussing it on
> > kernelnewbies.org).
>
Sorry, only registered users may post in this forum.

Click here to login