Welcome! Log In Create A New Profile

Advanced

stream module on 100% cpu load

Posted by A. Schulze 
A. Schulze
stream module on 100% cpu load
January 03, 2017 02:30PM
Hello,

last days I setup a server to encapsulate DNS over TLS.

- DNS-Server @localhost, Port 53 TCP
- NGINX Stream module on public IP, Port 853 TCP, SSL enabled.

That work so far.
Now I thought to scan this setup using ssllabs.com

I shutdown my HTTPS webserver an let nginx stream module listen on port 443.
To make it easier I switched also the stream proxy target to ::1, Port 80
Now I could again access my website but not via nginx ssl but nginx
stream module.
Work also so far...

Now I pointed SSLlasbs to the server and ... surprise!

The scan terminate with "Assessment failed: Unexpected failure"
last loglines nginx wrote was:

2017/01/03 13:26:49 [info] 19253#0: *25 client
[2600:c02:1020:4202::ac10:8267]:50918 connected to [2001:db8::53]:443
2017/01/03 13:26:49 [info] 19253#0: *25 proxy [2001:db8::53]:42534
connected to [::1]:80
2017/01/03 13:26:50 [notice] 19253#0: *25 SSL renegotiation disabled
while proxying connection, client: 2600:c02:1020:4202::ac10:8267,
server: [2001:db8::53]:443, upstream: "[::1]:80", bytes from/to
client:138/0, bytes from/to upstream:0/138

The nginx process stop responding and eat up 100% cpu time.

After reading again http://nginx.org/en/docs/stream/ngx_stream_ssl_module.html
I added "worker_processes auto;" to nginx.conf.

That changed the picture a little bit.
The ssllabs scan do no longer terminate but finish with a usual result.
Still one nginx process consume 100% cpu time.

I guess there is something broken with my setup or nginx. What further
information are needed to nail down the problem?

Andreas


nginx-1.11.8 with this (simplified) /etc/nginx/nginx.conf:

error_log /path/to/nginx-error.log info;
daemon off;

events {
worker_connections 1024;
}

http {
server {
listen [::1]:80;
location / {
root /path/to/htdocs/;
}
}
}

worker_processes auto;
stream {
upstream dns {
server [::1]:80;
}
server {
listen [2001:db8::53]:443 ssl;
proxy_pass dns;
ssl_certificate /path/to/cert+intermediate.pem;
ssl_certificate_key /path/to/key.pem;
}
}

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Vladimir Homutov
Re: stream module on 100% cpu load
January 10, 2017 04:30PM
03.01.2017 16:20, A. Schulze пишет:
>
> Hello,
>
> last days I setup a server to encapsulate DNS over TLS.
>
> - DNS-Server @localhost, Port 53 TCP
> - NGINX Stream module on public IP, Port 853 TCP, SSL enabled.
>
> That work so far.
> Now I thought to scan this setup using ssllabs.com
>
> I shutdown my HTTPS webserver an let nginx stream module listen on port
> 443.
> To make it easier I switched also the stream proxy target to ::1, Port 80
> Now I could again access my website but not via nginx ssl but nginx
> stream module.
> Work also so far...
>
> Now I pointed SSLlasbs to the server and ... surprise!
>
> The scan terminate with "Assessment failed: Unexpected failure"
> last loglines nginx wrote was:
>
> 2017/01/03 13:26:49 [info] 19253#0: *25 client
> [2600:c02:1020:4202::ac10:8267]:50918 connected to [2001:db8::53]:443
> 2017/01/03 13:26:49 [info] 19253#0: *25 proxy [2001:db8::53]:42534
> connected to [::1]:80
> 2017/01/03 13:26:50 [notice] 19253#0: *25 SSL renegotiation disabled
> while proxying connection, client: 2600:c02:1020:4202::ac10:8267,
> server: [2001:db8::53]:443, upstream: "[::1]:80", bytes from/to
> client:138/0, bytes from/to upstream:0/138
>
> The nginx process stop responding and eat up 100% cpu time.
>
> After reading again
> http://nginx.org/en/docs/stream/ngx_stream_ssl_module.html
> I added "worker_processes auto;" to nginx.conf.
>
> That changed the picture a little bit.
> The ssllabs scan do no longer terminate but finish with a usual result.
> Still one nginx process consume 100% cpu time.
>
> I guess there is something broken with my setup or nginx. What further
> information are needed to nail down the problem?
>
> Andreas

Thank you for reporting.

You may try the following patch:

diff --git a/src/stream/ngx_stream_proxy_module.c
b/src/stream/ngx_stream_proxy_module.c
--- a/src/stream/ngx_stream_proxy_module.c
+++ b/src/stream/ngx_stream_proxy_module.c
@@ -1564,6 +1564,7 @@ ngx_stream_proxy_process(ngx_stream_sess
return;
}

+ src->read->ready = 0;
src->read->eof = 1;
n = 0;
}


_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
A. Schulze
Re: stream module on 100% cpu load
January 16, 2017 08:30AM
Vladimir Homutov:

> You may try the following patch:
>
> diff --git a/src/stream/ngx_stream_proxy_module.c
> b/src/stream/ngx_stream_proxy_module.c
> --- a/src/stream/ngx_stream_proxy_module.c
> +++ b/src/stream/ngx_stream_proxy_module.c
> @@ -1564,6 +1564,7 @@ ngx_stream_proxy_process(ngx_stream_sess
> return;
> }
>
> + src->read->ready = 0;
> src->read->eof = 1;
> n = 0;
> }


Hello Vladimir,

I can confirm the patch fix the issue.
Thanks!

Andreas


_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Sorry, only registered users may post in this forum.

Click here to login