Welcome! Log In Create A New Profile

Advanced

Issue 158 in memcached: Single packet DoS on UDP channel

Posted by Anonymous User 
Anonymous User
Issue 158 in memcached: Single packet DoS on UDP channel
October 06, 2010 12:10AM
Status: New
Owner: ----
Labels: Type-Defect Priority-Medium

New issue 158 by marcolslaviero: Single packet DoS on UDP channel
http://code.google.com/p/memcached/issues/detail?id=158

What steps will reproduce the problem?
1. Download attached script
2. Start a memcached server on a UDP port
3. Run "python largeudp.py a <server_ip> <port> 1200"

What is the expected output? What do you see instead?
The server is expected to continue processing UDP traffic after the script
has copmleted.

Instead, it no longer responds to any traffic sent via UDP, legitimate or
otherwise.

Note: while UDP packets are no longer processed, the TCP channel remains
operational.

What version of the product are you using? On what operating system?
Tested on memcached-1.4.5, libevent-2.0.7-rc. Linux-2.6.24-28

Please provide any additional information below.

The attached script takes four arguments:
arg-1: either "a" or "b", to use the "a"scii or "b"inary protocol
arg-2: target ip
arg-3: target port
arg-4: length of packet to generate. 1200 works nicely for hanging 1.4.5
while not exceeding MTUs.


Attachments:
largeudp.py 3.6 KB
Anonymous User
Re: Issue 158 in memcached: Single packet DoS on UDP channel
October 06, 2010 12:00PM
Comment #1 on issue 158 by marcolslaviero: Single packet DoS on UDP channel
http://code.google.com/p/memcached/issues/detail?id=158

I omitted to mention that once the UDP channel stops responding, the
memcached process consumes 100% cpu; in this way the bug indirectly affects
users of the TCP channel too as well as other processes on the same machine.
Anonymous User
Re: Issue 158 in memcached: Single packet DoS on UDP channel
October 06, 2010 07:30PM
Updates:
Owner: eric.d.lambert

Comment #2 on issue 158 by eric.d.lambert: Single packet DoS on UDP channel
http://code.google.com/p/memcached/issues/detail?id=158

(No comment was entered for this change.)
Anonymous User
Re: Issue 158 in memcached: Single packet DoS on UDP channel
October 07, 2010 11:00PM
Comment #3 on issue 158 by eric.d.lambert: Single packet DoS on UDP channel
http://code.google.com/p/memcached/issues/detail?id=158

Just FYI, was not able to reproduce this on mac using 1.4.5 with libevent
2.07-rc ... will take a look on linux when I get a chance.

smacky:Downloads elambert$ python ./largeudp.py a localhost 11211 1200
[-] Using ascii protocol. Good choice!
[-] Creating socket
[-] Is memcached responding? Yes
[-] Great, server responds
[-] Sending large packet with size 1208
[-] Timeout expired waiting for response to large packet
[-] Is memcached responding? Yes
[E] DoS appears to have failed
smacky:Downloads elambert$ echo "stats" | nc localhost 11211 |
egrep "version|libevent"
STAT version 1.4.5_1_gb4936c4
STAT libevent 2.0.7-rc
smacky:Downloads elambert$ uname -a
Darwin smacky.local 10.4.0 Darwin Kernel Version 10.4.0: Fri Apr 23
18:28:53 PDT 2010; root:xnu-1504.7.4~1/RELEASE_I386 i386 i386
smacky:Downloads elambert$
Anonymous User
Re: Issue 158 in memcached: Single packet DoS on UDP channel
October 08, 2010 01:20AM
Comment #4 on issue 158 by marcolslaviero: Single packet DoS on UDP channel
http://code.google.com/p/memcached/issues/detail?id=158

Thanks for the feedback, apologies for supplying a PoC that didn't demo the
issue sufficiently, that was pretty fail for a PoC. However, I believe the
bug to still be present, as I'll show below. I'm less sure as to the cause:
large packet sizes trigger the bug sooner, but using a packet size of 10
also triggers the bug on my side, eventually.

I can trigger the bug on Snow Leopard (Darwin insurrection.local 10.4.0
Darwin Kernel Version 10.4.0: Fri Apr 23 18:28:53 PDT 2010;
root:xnu-1504.7.4~1/RELEASE_I386 i386) by running the PoC four times (works
for both ascii and binary protocols) using a packet size of 1200. This too
is the case on Linux, repeating the PoC four times triggers the bug.

Below is an OSX backtrace after the bug had been triggered, and memcached
was consuming full CPU:
(gdb) bt
#0 0x00007fff85ab508a in kevent ()
#1 0x00000001000300e3 in kq_dispatch ()
#2 0x000000010002380e in event_base_loop ()
#3 0x0000000100002be7 in main (argc=3, argv=0x7fff5fbff9b0) at
memcached.c:4681
(Note: the libevent for the OSX version was 1.4.2. Latest libevent was on
Linux)

Not much to go on, I'm afraid.

It may also help to mention that if "-v -v" is enabled then once the bug is
triggered, no further debug messages are printed to the console for UDP
comms. TCP traffic still generates the regular debug notices.
Anonymous User
Re: Issue 158 in memcached: Single packet DoS on UDP channel
October 08, 2010 06:50AM
Comment #5 on issue 158 by eric.d.lambert: Single packet DoS on UDP channel
http://code.google.com/p/memcached/issues/detail?id=158

I think i have somewhat of an idea as to what is happening. The UDP
connections behave a little bit different than TCP. At start up, the server
creates a UDP "connection handler" for each thread in the server (as
controlled by the -t option). By default there are 4 threads so there will
be 4 UDP "connection handlers".

What appears to be happening is that the message you are sending is not a
valid/wellformed message which causes the server to close the "connection
handler" in such a way that it can not be used again. So in a default
scenario with the server running 4 threads, the first time you run your
test one connect handler is closed leaving three left, the second run
closes a second handler leaving two left and so on. If you start the server
with only one thread (-t 1) you should hit the issue on the first run.

I dont have a root cause yet, but the issue does appear to be related to
how the server clears up the connection handler when it encounters a
problem. Will let you know when I've figured this out
Anonymous User
Re: Issue 158 in memcached: Single packet DoS on UDP channel
October 08, 2010 07:00AM
Updates:
Status: Accepted

Comment #6 on issue 158 by eric.d.lambert: Single packet DoS on UDP channel
http://code.google.com/p/memcached/issues/detail?id=158

(No comment was entered for this change.)
Anonymous User
Re: Issue 158 in memcached: Single packet DoS on UDP channel
October 08, 2010 11:30PM
Comment #7 on issue 158 by marcolslaviero: Single packet DoS on UDP channel
http://code.google.com/p/memcached/issues/detail?id=158

Well spotted, I was indeed running the initial tests with -t 1. At least
that clears up some minor confusion this side.
Sorry, only registered users may post in this forum.

Click here to login