Welcome! Log In Create A New Profile

Advanced

Making memchaed more secure

Posted by samwyse 
samwyse
Making memchaed more secure
August 07, 2010 03:20PM
I've just now suggested this on Slashdot: At startup, issue a big
multi-line warning if the IP addresses that are getting bound aren't
on the loopback address or a private internet. The private internets
are defined in RFC 1918 as:

10.0.0.0 - 10.255.255.255 (10/8 prefix)
172.16.0.0 - 172.31.255.255 (172.16/12 prefix)
192.168.0.0 - 192.168.255.255 (192.168/16 prefix)
Brian Moon
Re: Making memchaed more secure
August 07, 2010 04:00PM
On 8/7/10 7:09 AM, samwyse wrote:
> I've just now suggested this on Slashdot: At startup, issue a big
> multi-line warning if the IP addresses that are getting bound aren't
> on the loopback address or a private internet. The private internets
> are defined in RFC 1918 as:
>
> 10.0.0.0 - 10.255.255.255 (10/8 prefix)
> 172.16.0.0 - 172.31.255.255 (172.16/12 prefix)
> 192.168.0.0 - 192.168.255.255 (192.168/16 prefix)

That would be a knee jerk reaction to a poorly worded headline on
Slashdot because some idiot sysadmins at some high profile sites should
be fire for setting up memcached in the one way it was never ever ever
intended to be setup.

--

Brian.
--------
http://brian.moonspot.net/
Henrik Schröder
Re: Making memchaed more secure
August 07, 2010 04:30PM
What do you mean "at startup"? I click "start service" in my service control
panel, and then... where would that warning be displayed?

Seriously though, there are many ways to solve this problem, binding to
private IPs is one way to do it, but not necessarily the best way, and
definitely not the only way. If you run memcached you should realize that
there's no security whatsoever on it, and leaving it open to the internet at
large is a pretty stupid idea. How to best secure it depends greatly on the
local circumstances, and that is not something memcached itself should start
second-guessing the local admins about.


/Henrik

On Sat, Aug 7, 2010 at 14:09, samwyse <[email protected]> wrote:

> I've just now suggested this on Slashdot: At startup, issue a big
> multi-line warning if the IP addresses that are getting bound aren't
> on the loopback address or a private internet. The private internets
> are defined in RFC 1918 as:
>
> 10.0.0.0 - 10.255.255.255 (10/8 prefix)
> 172.16.0.0 - 172.31.255.255 (172.16/12 prefix)
> 192.168.0.0 - 192.168.255.255 (192.168/16 prefix)
>
Loganaden Velvindron
Re: Making memchaed more secure
August 07, 2010 04:40PM
Here's a git-diff.

It disables remote debug & cachedump support.

Feedback is welcomed.

It works fine on our server since this morning.
Sorry for not having posted this earlier.

//Logan
C-x-C-c
Esokia Web Agency
http://www.esokia-webagency.com/

diff --git a/memcached.c b/memcached.c
index 750c8b3..eb0343f 100644
--- a/memcached.c
+++ b/memcached.c
@@ -2336,8 +2336,7 @@ inline static void process_stats_detail(conn *c, const
char *command) {
assert(c != NULL);

if (strcmp(command, "on") == 0) {
- settings.detail_enabled = 1;
- out_string(c, "OK");
+ out_string(c, "Remote debug support disabled");
}
else if (strcmp(command, "off") == 0) {
settings.detail_enabled = 0;
@@ -2469,27 +2468,7 @@ static void process_stat(conn *c, token_t *tokens,
const size_t ntokens) {
} else if (strcmp(subcommand, "settings") == 0) {
process_stat_settings(&append_stats, c);
} else if (strcmp(subcommand, "cachedump") == 0) {
- char *buf;
- unsigned int bytes, id, limit = 0;
-
- if (ntokens < 5) {
- out_string(c, "CLIENT_ERROR bad command line");
- return;
- }
-
- if (!safe_strtoul(tokens[2].value, &id) ||
- !safe_strtoul(tokens[3].value, &limit)) {
- out_string(c, "CLIENT_ERROR bad command line format");
- return;
- }
-
- if (id >= POWER_LARGEST) {
- out_string(c, "CLIENT_ERROR Illegal slab id");
- return;
- }
-
- buf = item_cachedump(id, limit, &bytes);
- write_and_free(c, buf, bytes);
+ out_string(c, "Cachedump disabled");
return ;
} else {
/* getting here means that the subcommand is either engine specific
or


On Sat, Aug 7, 2010 at 6:24 PM, Henrik Schröder <[email protected]> wrote:

> What do you mean "at startup"? I click "start service" in my service
> control panel, and then... where would that warning be displayed?
>
> Seriously though, there are many ways to solve this problem, binding to
> private IPs is one way to do it, but not necessarily the best way, and
> definitely not the only way. If you run memcached you should realize that
> there's no security whatsoever on it, and leaving it open to the internet at
> large is a pretty stupid idea. How to best secure it depends greatly on the
> local circumstances, and that is not something memcached itself should start
> second-guessing the local admins about.
>
>
> /Henrik
>
>
> On Sat, Aug 7, 2010 at 14:09, samwyse <[email protected]> wrote:
>
>> I've just now suggested this on Slashdot: At startup, issue a big
>> multi-line warning if the IP addresses that are getting bound aren't
>> on the loopback address or a private internet. The private internets
>> are defined in RFC 1918 as:
>>
>> 10.0.0.0 - 10.255.255.255 (10/8 prefix)
>> 172.16.0.0 - 172.31.255.255 (172.16/12 prefix)
>> 192.168.0.0 - 192.168.255.255 (192.168/16 prefix)
>>
>
>


--
`` Real men run current !''
Loganaden Velvindron
Re: Making memchaed more secure
August 07, 2010 05:00PM
There seems to be a problem when I pasted it in gmail.

Here's a link to the git diff:

http://devio.us/~loganaden/memcached.git.diff

//Logan
C-x-C-c
Esokia Webagency



On Sat, Aug 7, 2010 at 6:30 PM, Loganaden Velvindron <[email protected]>wrote:

> Here's a git-diff.
>
> It disables remote debug & cachedump support.
>
> Feedback is welcomed.
>
> It works fine on our server since this morning.
> Sorry for not having posted this earlier.
>
> //Logan
> C-x-C-c
> Esokia Web Agency
> http://www.esokia-webagency.com/
>
> diff --git a/memcached.c b/memcached.c
> index 750c8b3..eb0343f 100644
> --- a/memcached.c
> +++ b/memcached.c
> @@ -2336,8 +2336,7 @@ inline static void process_stats_detail(conn *c,
> const char *command) {
> assert(c != NULL);
>
> if (strcmp(command, "on") == 0) {
> - settings.detail_enabled = 1;
> - out_string(c, "OK");
> + out_string(c, "Remote debug support disabled");
> }
> else if (strcmp(command, "off") == 0) {
> settings.detail_enabled = 0;
> @@ -2469,27 +2468,7 @@ static void process_stat(conn *c, token_t *tokens,
> const size_t ntokens) {
> } else if (strcmp(subcommand, "settings") == 0) {
> process_stat_settings(&append_stats, c);
> } else if (strcmp(subcommand, "cachedump") == 0) {
> - char *buf;
> - unsigned int bytes, id, limit = 0;
> -
> - if (ntokens < 5) {
> - out_string(c, "CLIENT_ERROR bad command line");
> - return;
> - }
> -
> - if (!safe_strtoul(tokens[2].value, &id) ||
> - !safe_strtoul(tokens[3].value, &limit)) {
> - out_string(c, "CLIENT_ERROR bad command line format");
> - return;
> - }
> -
> - if (id >= POWER_LARGEST) {
> - out_string(c, "CLIENT_ERROR Illegal slab id");
> - return;
> - }
> -
> - buf = item_cachedump(id, limit, &bytes);
> - write_and_free(c, buf, bytes);
> + out_string(c, "Cachedump disabled");
> return ;
> } else {
> /* getting here means that the subcommand is either engine
> specific or
>
>
>
> On Sat, Aug 7, 2010 at 6:24 PM, Henrik Schröder <[email protected]> wrote:
>
>> What do you mean "at startup"? I click "start service" in my service
>> control panel, and then... where would that warning be displayed?
>>
>> Seriously though, there are many ways to solve this problem, binding to
>> private IPs is one way to do it, but not necessarily the best way, and
>> definitely not the only way. If you run memcached you should realize that
>> there's no security whatsoever on it, and leaving it open to the internet at
>> large is a pretty stupid idea. How to best secure it depends greatly on the
>> local circumstances, and that is not something memcached itself should start
>> second-guessing the local admins about.
>>
>>
>> /Henrik
>>
>>
>> On Sat, Aug 7, 2010 at 14:09, samwyse <[email protected]> wrote:
>>
>>> I've just now suggested this on Slashdot: At startup, issue a big
>>> multi-line warning if the IP addresses that are getting bound aren't
>>> on the loopback address or a private internet. The private internets
>>> are defined in RFC 1918 as:
>>>
>>> 10.0.0.0 - 10.255.255.255 (10/8 prefix)
>>> 172.16.0.0 - 172.31.255.255 (172.16/12 prefix)
>>> 192.168.0.0 - 192.168.255.255 (192.168/16 prefix)
>>>
>>
>>
>
>
> --
> `` Real men run current !''
>
>
>
>
>
>


--
`` Real men run current !''
Dustin
Re: Making memchaed more secure
August 07, 2010 08:50PM
On Aug 7, 5:09 am, samwyse <[email protected]> wrote:
> I've just now suggested this on Slashdot:  At startup, issue a big
> multi-line warning if the IP addresses that are getting bound aren't
> on the loopback address or a private internet.  The private internets
> are defined in RFC 1918 as:
>
>           10.0.0.0 - 10.255.255.255 (10/8 prefix)
>           172.16.0.0 - 172.31.255.255 (172.16/12 prefix)
>           192.168.0.0 - 192.168.255.255 (192.168/16 prefix)

Shouldn't it first verify itself against my firewall configuration
and ensure I don't have SASL enabled? Should you also verify the IPv6
address class isn't link-local or perhaps using an intrasite tunneling
policy for cross datacenter invalidations?

I wouldn't want to get an alert when I'm doing it right just because
some people are doing it wrong.
Dustin
Re: Making memchaed more secure
August 07, 2010 08:50PM
On Aug 7, 7:52 am, Loganaden Velvindron <[email protected]> wrote:
> There seems to be a problem when I pasted it in gmail.
>
> Here's a link to the git diff:
>
> http://devio.us/~loganaden/memcached.git.diff

This makes some sense to me. That functionality is kind of a
plague. On one hand we've got people trying to use it for things it
doesn't do and on the other hand, we've got people who configure
memcached incorrectly and put themselves at risk.
samwyse
Re: Making memchaed more secure
August 07, 2010 09:10PM
So you don't think it's a good idea to warn idiot sysadmins if they
set up memcached in the one way it was never ever ever intended to be
setup? I disagree. If people would RTFM, we wouldn't need the
acronym. Checking the address that is being bound would only incur a
cost at startup, and could help the users of sites that hire idiot
sysadmins (who have plenty of ways to get themselves fired without
risking other people's personal information).

On Sat, Aug 7, 2010 at 8:54 AM, Brian Moon <[email protected]> wrote:
> On 8/7/10 7:09 AM, samwyse wrote:
>>
>> I've just now suggested this on Slashdot:  At startup, issue a big
>> multi-line warning if the IP addresses that are getting bound aren't
>> on the loopback address or a private internet.  The private internets
>> are defined in RFC 1918 as:
>>
>>           10.0.0.0 - 10.255.255.255 (10/8 prefix)
>>           172.16.0.0 - 172.31.255.255 (172.16/12 prefix)
>>           192.168.0.0 - 192.168.255.255 (192.168/16 prefix)
>
> That would be a knee jerk reaction to a poorly worded headline on Slashdot
> because some idiot sysadmins at some high profile sites should be fire for
> setting up memcached in the one way it was never ever ever intended to be
> setup.
>
> --
>
> Brian.
> --------
> http://brian.moonspot.net/
>
Michael Shadle
Re: Making memchaed more secure
August 07, 2010 09:20PM
memcached has never claimed to be "secure" in the sense you're thinking. It's a domain specific application that was designed the lightest way possible with certain expectations of its usage.

Do you also want mysql to spit out a warning if its listening on a public ip?

Do you want your webserver to spit out a notice when its not listening on a public ip?

Some people need the "for external use only" type warnings but honestly if you don't grok the basics of server administration then you shouldn't be in a position to do it.

Or, fork memached (you're totally allowed to) and make a version that offers kid gloves. Notices on startup reminding you about IP address binding (even though the config file makes it very clear how to bind) a username/password in the protocol (slowing it down an negating the original purpose) and why not also remind the admin to take a shower or eat a healthy breakfast while you're at it.


On Aug 7, 2010, at 7:45 AM, samwyse <[email protected]> wrote:

> So you don't think it's a good idea to warn idiot sysadmins if they
> set up memcached in the one way it was never ever ever intended to be
> setup? I disagree. If people would RTFM, we wouldn't need the
> acronym. Checking the address that is being bound would only incur a
> cost at startup, and could help the users of sites that hire idiot
> sysadmins (who have plenty of ways to get themselves fired without
> risking other people's personal information).
>
> On Sat, Aug 7, 2010 at 8:54 AM, Brian Moon <[email protected]> wrote:
>> On 8/7/10 7:09 AM, samwyse wrote:
>>>
>>> I've just now suggested this on Slashdot: At startup, issue a big
>>> multi-line warning if the IP addresses that are getting bound aren't
>>> on the loopback address or a private internet. The private internets
>>> are defined in RFC 1918 as:
>>>
>>> 10.0.0.0 - 10.255.255.255 (10/8 prefix)
>>> 172.16.0.0 - 172.31.255.255 (172.16/12 prefix)
>>> 192.168.0.0 - 192.168.255.255 (192.168/16 prefix)
>>
>> That would be a knee jerk reaction to a poorly worded headline on Slashdot
>> because some idiot sysadmins at some high profile sites should be fire for
>> setting up memcached in the one way it was never ever ever intended to be
>> setup.
>>
>> --
>>
>> Brian.
>> --------
>> http://brian.moonspot.net/
>>
samwyse
Re: Making memchaed more secure
August 08, 2010 01:20AM
On Aug 7, 2:17 pm, Michael Shadle <[email protected]> wrote:
> memcached has never claimed to be "secure" in the sense you're thinking.

In what sense am I thinking?

> It's a domain specific application that was designed the lightest way possible with certain expectations of its usage.

My suggestion follows that design.

> Do you also want mysql to spit out a warning if its listening on a public ip?

Does the documentation for mysql warn admins to not do that?

> Do you want your webserver to spit out a notice when its not listening on a public ip?

Ditto.

> Some people need the "for external use only" type warnings but honestly if you don't grok the basics of server administration then you shouldn't be in a position to do it.

I believe that I do (the jury's still out on that question, I only
have 25 years experience), but apparently some very well known sites
don't.

> Or, fork memached (you're totally allowed to) and make a version that offers kid gloves. Notices on startup reminding you about IP address binding (even though the config file makes it very clear how to bind) a username/password in the protocol (slowing it down an negating the original purpose) and why not also remind the admin to take a shower or eat a healthy breakfast while you're at it.

My suggestion neither slows memcached down nor negates its original
purpose. Ditto Loganaden's patch, which to all appearances you should
also be objecting to. He's breaking the functionality of documented
commands, for gawd's sake; I won't be able to use remote debug or dump
the cache. How will I ever be able to find and fix problems?
dormando
Re: Making memchaed more secure
August 08, 2010 02:40AM
On Sat, 7 Aug 2010, Dustin wrote:

>
> On Aug 7, 7:52 am, Loganaden Velvindron <[email protected]> wrote:
> > There seems to be a problem when I pasted it in gmail.
> >
> > Here's a link to the git diff:
> >
> > http://devio.us/~loganaden/memcached.git.diff
>
> This makes some sense to me. That functionality is kind of a
> plague. On one hand we've got people trying to use it for things it
> doesn't do and on the other hand, we've got people who configure
> memcached incorrectly and put themselves at risk.
>

I propose:

1.4.6 would come with a -D option or something which would disable
cachedump, etc. possibly also stats sizes.

1.6.0 will have them disabled by default, with a different option for
enabling them? Yeah people abuse the shit out of them and I was
entertaining the idea of randomizing the names every release or removing
it before, but I don't want to listen to the whining on both sides of the
fence.

Definitely don't think printing warnings will do much. Honestly we do warn
you in the docs, it's clear that you never provide a username/password,
and dozens of articles on "running memcached in the cloud" tell you to
firewall the damn thing.

What's funny is despite all this people still screw it up. Even if we make
it harder to run debug commands that's only limiting the sort of damage
you can do by a teeny bit.

Then in six months some security geek will write something to run 'stats
slabs/stats items' and bomb your weakest slabs with junk data until your
site goes away. Or like, connect to it until it hits maxconns, or guess at
common keys. Yawn.
Dustin
Re: Making memchaed more secure
August 08, 2010 05:40AM
On Aug 7, 12:17 pm, Michael Shadle <[email protected]> wrote:
>
> ) a username/password in the protocol (slowing it down an negating the original purpose)

Note that SASL does allow username/password and many other types of
authentication. If your app uses long-lived connections (as they all
should), authentication doesn't make it more slower in any meaningful
way.

On Aug 7, 4:19 pm, samwyse <[email protected]> wrote:

> apparently some very well known sites don't.

Warnings just don't help all that much. We have a warning that
tells you that we won't start up as root if you try to start memcached
as root. People figured out that they could work around this by
passing ``-u root'' *sigh*

No sysadmin with any idea how things work would ever voluntarily run
software as root without really, really good justification and
isolation, yet there was a report that memcached had a (theoretical)
remote root vulnerability because of how far people were willing to go
out of their way to make things insecure.

> My suggestion neither slows memcached down nor negates its original
> purpose.  Ditto Loganaden's patch, which to all appearances you should
> also be objecting to.  He's breaking the functionality of documented
> commands, for gawd's sake;

stats cachedump is explicitly undocumented. There is no reference
to it in the protocol specifications and the only place you'll find it
in the wiki is in the programming FAQ where it says that it's a
debugging interface and you shouldn't use it.

See this thread for more: http://groups.google.com/group/memcached/browse_thread/thread/a936dbd74a2d9a5f

> I won't be able to use remote debug or dump the cache.

You can't do it now.

> How will I ever be able to find and fix problems?

What kind of problem do you think such functionality would ever help
you fix? I'm asking a completely serious question because when it's
come up in the past all of the answers have come down to, ``I don't
know – I just need it.''
Michael Shadle
Re: Making memchaed more secure
August 08, 2010 05:50AM
On Sat, Aug 7, 2010 at 8:31 PM, Dustin <[email protected]> wrote:

>  Note that SASL does allow username/password and many other types of
> authentication.  If your app uses long-lived connections (as they all
> should), authentication doesn't make it more slower in any meaningful
> way.

If using persistent connections yes, only initial negotiation would be
impactful you're right.

>> I won't be able to use remote debug or dump the cache.
>
>  You can't do it now.

Brian said you can dump the cache... My notes from OSCON: "you can
dump your data using "memdump" - pulls back a meg worth of keys, also
has a C API for it" so it sounds like you -can- not that you have a
legitimate reason to need to... :p

(yes it's sort of not really the point.)

Also the whole "you should display warnings" thing is -annoying-

I have an open with mod_passenger folks because starting nginx with
mod_passenger compiled in, but not actually activated results in a
warning about "hey, you have passenger compiled but not enabled"

I can't even think of a proper metaphor for that. Obviously if you
your app isn't working, -maybe- you didn't configure it properly in
the webserver... at some point you have to stop treating
administrators and such as idiots. If they -are- idiots hopefully they
get demoted or fired. Not hope that the software is so overly verbose
about everything it prods you along with everything (and where do you
stop? Someone else will want more than a notice about binding IP
addresses. They'll want something telling them "hey you could use more
RAM for the cache" or who knows what else)

There's a reason you interview someone and look for their skills for a
position. Otherwise we could go hire a hobo on the street for as
little as possible just to have a warm body pushing buttons.
Dustin
Re: Making memchaed more secure
August 08, 2010 06:10AM
On Aug 7, 8:49 pm, Michael Shadle <[email protected]> wrote:
> >> I won't be able to use remote debug or dump the cache.
>
> >  You can't do it now.
>
> Brian said you can dump the cache... My notes from OSCON: "you can
> dump your data using "memdump" - pulls back a meg worth of keys, also
> has a C API for it" so it sounds like you -can- not that you have a
> legitimate reason to need to... :p

I wrote a java api for it a while back, but pulled it out of my
client because I didn't think it was responsible to ship my client
with support for it -- people kept trying to use it and thinking it
was supported.

> I can't even think of a proper metaphor for that. Obviously if you
> your app isn't working, -maybe- you didn't configure it properly in
> the webserver... at some point you have to stop treating
> administrators and such as idiots. If they -are- idiots hopefully they
> get demoted or fired. Not hope that the software is so overly verbose
> about everything it prods you along with everything (and where do you
> stop? Someone else will want more than a notice about binding IP
> addresses. They'll want something telling them "hey you could use more
> RAM for the cache" or who knows what else)

I agree. If someone can't perform a task correctly, train the
person or give the task to someone who can.
Sorry, only registered users may post in this forum.

Click here to login