cannot assign requested address (99)

We have a very busy high traffic site and recently started using memached to
store frequently accessed queries. We put memcached on each php application
server so that the connection is local to the system.

Today, during one of our peek times, we started getting this error and had
to disable the caching of queries:

[20-Oct-2010 12:05:57] PHP Warning: Memcache::connect() [<a
href='memcache.connect'>memcache.connect</a>]: Can't connect to
127.0.0.1:11211, Cannot assign requested address (99)

In searching online for this error, I came across several articles, but
nothing that I though was relevant to our case. I tried increasing the
number of connections from 1024 (default) to 5000 but the error continued.

Does anyone have any suggestions / thoughts of where to look?

stats information is below:

STAT pid 13425
STAT uptime 1940
STAT time 1287591991
STAT version 1.4.5
STAT pointer_size 64
STAT rusage_user 6.443020
STAT rusage_system 25.109182
STAT curr_connections 15
STAT total_connections 340578
STAT connection_structures 118
STAT cmd_get 357224
STAT cmd_set 136334
STAT cmd_flush 0
STAT get_hits 326956
STAT get_misses 30268
STAT delete_misses 0
STAT delete_hits 0
STAT incr_misses 0
STAT incr_hits 0
STAT decr_misses 0
STAT decr_hits 0
STAT cas_misses 0
STAT cas_hits 0
STAT cas_badval 0
STAT auth_cmds 0
STAT auth_errors 0
STAT bytes_read 21641880
STAT bytes_written 41425437
STAT limit_maxbytes 1073741824
STAT accepting_conns 1
STAT listen_disabled_num 0
STAT threads 4
STAT conn_yields 0
STAT bytes 2925203
STAT curr_items 19070
STAT total_items 136334
STAT evictions 0
STAT reclaimed 12
END

> We have a very busy high traffic site and recently started using memached to store frequently accessed queries.  We put memcached on each
> php application server so that the connection is local to the system.
> Today, during one of our peek times, we started getting this error and had to disable the caching of queries:
>
> [20-Oct-2010 12:05:57] PHP Warning: Memcache::connect() [<a href='memcache.connect'>memcache.connect</a>]: Can't connect to
> 127.0.0.1:11211, Cannot assign requested address (99) 

You're probably running out of local ports. Google for that phrase and
you'll probably find help :)

Just as a side note; you're meant to set up memcached as a big cluster and
use all the cache space, as noted at http://memcached.org/about - sticking
them all to localhost only defeats half the purpose. yes I know that
"localhost is faster" but it's more than adequate over the network.

-Dormando

STAT curr_connections 15
STAT total_connections 340578

Looks like your client code is creating and closing connections, instead of
reusing them.
You could increase the port range and/or decrease the tcp close timeout, but
much easier fix would be to use the client libraries connection reusing
ability.

thanks!
Rohit Karlupia


On Fri, Oct 22, 2010 at 11:50 AM, dormando <[email protected]> wrote:

>
> > We have a very busy high traffic site and recently started using memached
> to store frequently accessed queries. We put memcached on each
> > php application server so that the connection is local to the system.
> > Today, during one of our peek times, we started getting this error and
> had to disable the caching of queries:
> >
> > [20-Oct-2010 12:05:57] PHP Warning: Memcache::connect() [<a
> href='memcache.connect'>memcache.connect</a>]: Can't connect to
> > 127.0.0.1:11211, Cannot assign requested address (99)
>
> You're probably running out of local ports. Google for that phrase and
> you'll probably find help :)
>
> Just as a side note; you're meant to set up memcached as a big cluster and
> use all the cache space, as noted at http://memcached.org/about - sticking
> them all to localhost only defeats half the purpose. yes I know that
> "localhost is faster" but it's more than adequate over the network.
>
> -Dormando