(tcp 11211) failed with: Connection timed out (110)

Guys,

We are seeing tons of these messages reported;

Memcache::get(): Server 10.3.230.15 (tcp 11211) failed with:
Connection timed out (110)

They are reported on all six of our webservers. Memcache is working
but lots of connections are timing out as reported with the above
messages.

We are running memcached 1.2.5 with php-pecl-memcache-2.2.3 (we also
tried with memcache 1.4.5 but had the same errors). We have PHP 5.2.6
running on x86 Hardware with Red Hat 5.5.

We have six web servers all running Apache 2.2 on prefork mode.
Prefork MaxClients is set to 192.

Our memcached config looks like this;

PORT="11211"
USER="memcached"
MAXCONN="2048"
CACHESIZE="2048"
OPTIONS=""

We are running memcache between webservers on a LAN, no Firewall or
iptables being used.

Memcache is used to cache output of scripts most of the time. It could
be HTML or XML that is delivered to client.

Our tcp values look like this;

tcp_fin_timeout
60
tcp_max_orphans
65536
tcp_orphan_retries
0
tcp_keepalive_probes
9
tcp_keepalive_time
7200

I am totally out of ideas, I can't see any dropped packets on the
network, I just don't know what else to check.

Any ideas as I am pulling my hair out!

Thanks

Upgrade back to 1.4.5 and look for the 'listen_disabled_num' value in the
stats output. If the number is increasing, you're hitting maxconns.

If not, you're probably seeing packet loss, or have a firewall in the way
that's maxing out.

On Tue, 7 Sep 2010, the_fonz wrote:

> Guys,
>
> We are seeing tons of these messages reported;
>
> Memcache::get(): Server 10.3.230.15 (tcp 11211) failed with:
> Connection timed out (110)
>
> They are reported on all six of our webservers. Memcache is working
> but lots of connections are timing out as reported with the above
> messages.
>
> We are running memcached 1.2.5 with php-pecl-memcache-2.2.3 (we also
> tried with memcache 1.4.5 but had the same errors). We have PHP 5.2.6
> running on x86 Hardware with Red Hat 5.5.
>
> We have six web servers all running Apache 2.2 on prefork mode.
> Prefork MaxClients is set to 192.
>
> Our memcached config looks like this;
>
> PORT="11211"
> USER="memcached"
> MAXCONN="2048"
> CACHESIZE="2048"
> OPTIONS=""
>
> We are running memcache between webservers on a LAN, no Firewall or
> iptables being used.
>
> Memcache is used to cache output of scripts most of the time. It could
> be HTML or XML that is delivered to client.
>
> Our tcp values look like this;
>
> tcp_fin_timeout
> 60
> tcp_max_orphans
> 65536
> tcp_orphan_retries
> 0
> tcp_keepalive_probes
> 9
> tcp_keepalive_time
> 7200
>
> I am totally out of ideas, I can't see any dropped packets on the
> network, I just don't know what else to check.
>
> Any ideas as I am pulling my hair out!
>
> Thanks
>
>

the_fonz wrote:
> Guys,
>
> We are seeing tons of these messages reported;
>
> Memcache::get(): Server 10.3.230.15 (tcp 11211) failed with:
> Connection timed out (110)
>
> They are reported on all six of our webservers. Memcache is working
> but lots of connections are timing out as reported with the above
> messages.
>
> We are running memcached 1.2.5 with php-pecl-memcache-2.2.3 (we also
> tried with memcache 1.4.5 but had the same errors). We have PHP 5.2.6
> running on x86 Hardware with Red Hat 5.5.
>
> We have six web servers all running Apache 2.2 on prefork mode.
> Prefork MaxClients is set to 192.
>
Fonz,

Ayye Cunningham!! (ok, I couldn't help it!)

Can you telnet port 11211 on each of your servers? If you write a simple
command-line script, can it connect?

Also, maybe try PECL/memcached?

--Patrick

> Our memcached config looks like this;
>
> PORT="11211"
> USER="memcached"
> MAXCONN="2048"
> CACHESIZE="2048"
> OPTIONS=""
>
> We are running memcache between webservers on a LAN, no Firewall or
> iptables being used.
>
> Memcache is used to cache output of scripts most of the time. It could
> be HTML or XML that is delivered to client.
>
> Our tcp values look like this;
>
> tcp_fin_timeout
> 60
> tcp_max_orphans
> 65536
> tcp_orphan_retries
> 0
> tcp_keepalive_probes
> 9
> tcp_keepalive_time
> 7200
>
> I am totally out of ideas, I can't see any dropped packets on the
> network, I just don't know what else to check.
>
> Any ideas as I am pulling my hair out!
>
> Thanks
>
>

On 9/7/10 10:50 AM, the_fonz wrote:
> We are running memcached 1.2.5 with php-pecl-memcache-2.2.3 (we also
> tried with memcache 1.4.5 but had the same errors). We have PHP 5.2.6
> running on x86 Hardware with Red Hat 5.5.
>
> We have six web servers all running Apache 2.2 on prefork mode.
> Prefork MaxClients is set to 192.

Are you using persistent connections?

--

Brian.
--------
http://brian.moonspot.net/

Sorry, my memcache knowledge is about two weeks old (am learning though!)

Do you mean are we pooling connections? No we are not.

PHP should just drop the connection once it's finished with it. there is no "timeout" or "close connection" set within PHP.


> Date: Tue, 7 Sep 2010 11:19:18 -0500
> From: brian@moonspot.net
> To: memcached@googlegroups.com
> CC: tim_stockford@hotmail.com
> Subject: Re: (tcp 11211) failed with: Connection timed out (110)
>
> On 9/7/10 10:50 AM, the_fonz wrote:
> > We are running memcached 1.2.5 with php-pecl-memcache-2.2.3 (we also
> > tried with memcache 1.4.5 but had the same errors). We have PHP 5.2.6
> > running on x86 Hardware with Red Hat 5.5.
> >
> > We have six web servers all running Apache 2.2 on prefork mode.
> > Prefork MaxClients is set to 192.
>
> Are you using persistent connections?
>
> --
>
> Brian.
> --------
> http://brian.moonspot.net/

We reverted back to 1.4.5. Still get continuous timed out connections
with no increase in listen_disabled_num

I read http://brian.moonspot.net/php-memcached-issues

I see that PECL/memcached is buggy for persistent connections but am
wondering if I should enable persistent connections with
php-pecl-memcache-2.2.3, would this speed things up?

Is it worth trying PECL/memcached without persistent connections?

Thanks




On Sep 7, 5:40 pm, Tim Stockford <tim_stockf...@hotmail.com> wrote:
> Sorry, my memcache knowledge is about two weeks old (am learning though!)
>
> Do you mean are we pooling connections? No we are not.
>
> PHP should just drop the connection once it's finished with it. there is no "timeout" or "close connection" set within PHP.
>
> > Date: Tue, 7 Sep 2010 11:19:18 -0500
> > From: br...@moonspot.net
> > To: memcached@googlegroups.com
> > CC: tim_stockf...@hotmail.com
> > Subject: Re: (tcp 11211) failed with: Connection timed out (110)
>
> > On 9/7/10 10:50 AM, the_fonz wrote:
> > > We are running memcached 1.2.5 with php-pecl-memcache-2.2.3 (we also
> > > tried with memcache 1.4.5 but had the same errors). We have PHP 5.2.6
> > > running on x86 Hardware with Red Hat 5.5.
>
> > > We have six web servers all running Apache 2.2 on prefork mode.
> > > Prefork MaxClients is set to 192.
>
> > Are you using persistent connections?
>
> > --
>
> > Brian.
> > --------
> >http://brian.moonspot.net/

On 9/9/10 10:20 AM, the_fonz wrote:
> We reverted back to 1.4.5. Still get continuous timed out connections
> with no increase in listen_disabled_num
>
> I read http://brian.moonspot.net/php-memcached-issues
>
> I see that PECL/memcached is buggy for persistent connections but am
> wondering if I should enable persistent connections with
> php-pecl-memcache-2.2.3, would this speed things up?
>
> Is it worth trying PECL/memcached without persistent connections?

I run PECL/memcache with persistent connections always.

If you dig through the comments on my blog post there is a work around
for the persistent connection issue in PECL/memcached. But, there are
other issues with that library that will hopefully be fixed soon.

--

Brian.
--------
http://brian.moonspot.net/

the_fonz wrote:
> We reverted back to 1.4.5. Still get continuous timed out connections
> with no increase in listen_disabled_num
>
> I read http://brian.moonspot.net/php-memcached-issues
>
> I see that PECL/memcached is buggy for persistent connections but am
> wondering if I should enable persistent connections with
> php-pecl-memcache-2.2.3, would this speed things up?
>
> Is it worth trying PECL/memcached without persistent connections?
>
> Thanks
>
>
>
>
> On Sep 7, 5:40 pm, Tim Stockford <tim_stockf...@hotmail.com> wrote:
>
>> Sorry, my memcache knowledge is about two weeks old (am learning though!)
>>
>> Do you mean are we pooling connections? No we are not.
>>
>> PHP should just drop the connection once it's finished with it. there is no "timeout" or "close connection" set within PHP.
>>
>>
>>> Date: Tue, 7 Sep 2010 11:19:18 -0500
>>> From: br...@moonspot.net
>>> To: memcached@googlegroups.com
>>> CC: tim_stockf...@hotmail.com
>>> Subject: Re: (tcp 11211) failed with: Connection timed out (110)
>>>
>>> On 9/7/10 10:50 AM, the_fonz wrote:
>>>
>>>> We are running memcached 1.2.5 with php-pecl-memcache-2.2.3 (we also
>>>> tried with memcache 1.4.5 but had the same errors). We have PHP 5.2.6
>>>> running on x86 Hardware with Red Hat 5.5.
>>>>
>>>> We have six web servers all running Apache 2.2 on prefork mode.
>>>> Prefork MaxClients is set to 192.
>>>>
>>> Are you using persistent connections?
>>>
Fonz,

I would certainly give PECL/memcached a test if you can. I'm in the
process (taking longer than I intended) to upgrade libmemcached that
PECL/memcached is built against, so that might end up fixing the buggy
issues.

--Patrick
>>> --
>>>
>>> Brian.
>>> --------
>>> http://brian.moonspot.net/
>>>