Welcome! Log In Create A New Profile

Advanced

Large CRL file crashing nginx on reload

Posted by Shaun Tarves 
Shaun Tarves
Large CRL file crashing nginx on reload
July 26, 2018 10:20PM
Hi,

We are trying to use nginx to support the DoD PKI infrastructure, which
includes many DoD and contractor CRLs. The combined CRL file is over 350MB
in size, which seems to crash nginx during a reload (at least on Red Hat
6). Our cert/key/crl set up is valid and working, and when only including a
subset of the CRL files we have, reloads work fine.

When we concatenate all the CRLs we need to support, the config reload
request causes worker threads to become defunct and messages in the error
log indicate the following:

2018/07/26 16:05:25 [alert] 30624#30624: fork() failed while spawning
"worker process" (12: Cannot allocate memory)

2018/07/26 16:05:25 [alert] 30624#30624: sendmsg() failed (9: Bad file
descriptor)

2018/07/26 16:08:42 [alert] 30624#30624: worker process 1611 exited on
signal 9

Is there any way we can get nginx to support such a large volume of CRLs?
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Igor A. Ippolitov
Re: Large CRL file crashing nginx on reload
July 26, 2018 11:50PM
Shaun,

Can you post a snippet on how you include crl into your configuration
and 'ps aux | grep nginx' output, please?

The wild guess is that you include the crl several times. And on reload
you get twice as many workers as there are usually.
You can try moving ssl_crl statement into http{} context.

On 26.07.2018 23:16, Shaun Tarves wrote:
> Hi,
>
> We are trying to use nginx to support the DoD PKI infrastructure,
> which includes many DoD and contractor CRLs. The combined CRL file is
> over 350MB in size, which seems to crash nginx during a reload (at
> least on Red Hat 6). Our cert/key/crl set up is valid and working, and
> when only including a subset of the CRL files we have, reloads work fine.
>
> When we concatenate all the CRLs we need to support, the config reload
> request causes worker threads to become defunct and messages in the
> error log indicate the following:
>
> 2018/07/26 16:05:25 [alert] 30624#30624: fork() failed while spawning
> "worker process" (12: Cannot allocate memory)
>
> 2018/07/26 16:05:25 [alert] 30624#30624: sendmsg() failed (9: Bad file
> descriptor)
>
> 2018/07/26 16:08:42 [alert] 30624#30624: worker process 1611 exited on
> signal 9
>
>
> Is there any way we can get nginx to support such a large volume of CRLs?
>
>
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx


_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Maxim Dounin
Re: Large CRL file crashing nginx on reload
July 27, 2018 02:20AM
Hello!

On Thu, Jul 26, 2018 at 04:16:11PM -0400, Shaun Tarves wrote:

> We are trying to use nginx to support the DoD PKI infrastructure, which
> includes many DoD and contractor CRLs. The combined CRL file is over 350MB
> in size, which seems to crash nginx during a reload (at least on Red Hat
> 6). Our cert/key/crl set up is valid and working, and when only including a
> subset of the CRL files we have, reloads work fine.
>
> When we concatenate all the CRLs we need to support, the config reload
> request causes worker threads to become defunct and messages in the error
> log indicate the following:
>
> 2018/07/26 16:05:25 [alert] 30624#30624: fork() failed while spawning
> "worker process" (12: Cannot allocate memory)

The error suggest you've run out of memory.

> 2018/07/26 16:05:25 [alert] 30624#30624: sendmsg() failed (9: Bad file
> descriptor)
>
> 2018/07/26 16:08:42 [alert] 30624#30624: worker process 1611 exited on
> signal 9

And this one suggests nginx worker was killed with signal 9,
likely by the OOM Killer. That is, again, you've run out of
memory.

> Is there any way we can get nginx to support such a large volume
> of CRLs?

It looks like you problem is that you don't have enough memory for
your configuration. Most trivial solution would be to add more
memory. Another possible solution would be to carefully inspect
the configuration, and, if possible, reduce amount of memory
required. In particular, when using such a big CRLs it is
important to only specify them in configuration context they are
needed, as each SSL context with a CRL configured will load its
own copy of the CRL.

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Shaun Tarves
Re: Large CRL file crashing nginx on reload
July 27, 2018 05:00PM
Here are the relevant parts of our configuration:

worker_processes 1;
pid /var/run/nginx.pid;
events {
worker_connections 512;
}
http {
server {
listen xx.xx.xx.xx:443 default_server ssl;
ssl on;
ssl_certificate /opt/xxx.pem;
ssl_certificate_key /opt/xxx.key
ssl_ciphers 'AES128+EECDH:AES128+EDH:!aNULL';
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_session_cache shared:SSL:10m;
ssl_prefer_server_ciphers on;
ssl_verify_client optional;
ssl_client_certificate /opt/ca.crt.pem
ssl_crl /opt/ca.crl/.pem;
}
}

During a "reload" command, here is how our ps looks:

[[email protected] nginx]# service nginx reload

Reloading nginx: [ OK ]

[[email protected] nginx]# ps -ef | grep nginx

root 9605 1 9 15:06 ? 00:00:17 nginx: master process
/usr/sbin/nginx -c /etc/nginx/nginx.conf

cons3rt 9606 9605 0 15:06 ? 00:00:00 nginx: worker process


root 11009 27847 0 15:09 pts/2 00:00:00 grep nginx

[[email protected] nginx]# ps -ef | grep nginx

root 9605 1 10 15:06 ? 00:00:24 nginx: master process
/usr/sbin/nginx -c /etc/nginx/nginx.conf

cons3rt 9606 9605 0 15:06 ? 00:00:00 nginx: worker process is
shutting down

root 11091 27847 0 15:10 pts/2 00:00:00 grep nginx

[[email protected] nginx]# ps -ef | grep nginx

root 9605 1 10 15:06 ? 00:00:24 nginx: master process
/usr/sbin/nginx -c /etc/nginx/nginx.conf

cons3rt 9606 9605 0 15:06 ? 00:00:00 nginx: worker process is
shutting down

root 11362 27847 0 15:10 pts/2 00:00:00 grep nginx

[[email protected] nginx]# ps -ef | grep nginx

root 9605 1 9 15:06 ? 00:00:24 nginx: master process
/usr/sbin/nginx -c /etc/nginx/nginx.conf

cons3rt 9606 9605 1 15:06 ? 00:00:02 nginx: worker process is
shutting down

root 11395 27847 0 15:10 pts/2 00:00:00 grep nginx

[[email protected] nginx]# vi /var/log/nginx/error.log

[[email protected] nginx]# ps -ef | grep nginx

root 9605 1 7 15:06 ? 00:00:24 nginx: master process
/usr/sbin/nginx -c /etc/nginx/nginx.conf

cons3rt 9606 9605 5 15:06 ? 00:00:19 nginx: worker process is
shutting down

root 11771 27847 0 15:12 pts/2 00:00:00 grep nginx

[[email protected] nginx]# service nginx stop

Stopping nginx: [FAILED]



On Thu, Jul 26, 2018 at 4:16 PM Shaun Tarves <[email protected]>
wrote:

> Hi,
>
> We are trying to use nginx to support the DoD PKI infrastructure, which
> includes many DoD and contractor CRLs. The combined CRL file is over 350MB
> in size, which seems to crash nginx during a reload (at least on Red Hat
> 6). Our cert/key/crl set up is valid and working, and when only including a
> subset of the CRL files we have, reloads work fine.
>
> When we concatenate all the CRLs we need to support, the config reload
> request causes worker threads to become defunct and messages in the error
> log indicate the following:
>
> 2018/07/26 16:05:25 [alert] 30624#30624: fork() failed while spawning
> "worker process" (12: Cannot allocate memory)
>
> 2018/07/26 16:05:25 [alert] 30624#30624: sendmsg() failed (9: Bad file
> descriptor)
>
> 2018/07/26 16:08:42 [alert] 30624#30624: worker process 1611 exited on
> signal 9
>
> Is there any way we can get nginx to support such a large volume of CRLs?
>
>
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Maxim Dounin
Re: Large CRL file crashing nginx on reload
July 27, 2018 05:30PM
Hello!

On Fri, Jul 27, 2018 at 10:56:38AM -0400, Shaun Tarves wrote:

> Here are the relevant parts of our configuration:
>
> worker_processes 1;
> pid /var/run/nginx.pid;
> events {
> worker_connections 512;
> }
> http {
> server {
> listen xx.xx.xx.xx:443 default_server ssl;
> ssl on;
> ssl_certificate /opt/xxx.pem;
> ssl_certificate_key /opt/xxx.key
> ssl_ciphers 'AES128+EECDH:AES128+EDH:!aNULL';
> ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
> ssl_session_cache shared:SSL:10m;
> ssl_prefer_server_ciphers on;
> ssl_verify_client optional;
> ssl_client_certificate /opt/ca.crt.pem
> ssl_crl /opt/ca.crl/.pem;
> }
> }

Configuration looks fine - there is only one server{} block where
the "ssl_crl" directive is used, so there should be only one copy
of CRL loaded per configuration.

Accordingly, it looks like you've simply run out of memory. Check
the amount of memory as available on your server (and/or memory
limits, if any) and the amount of memory as used by nginx with the
CRL loaded. Note that for the configuration reload to work you
will need extra memory to load an additional copy of the
configuration and to start new worker processes. See
http://nginx.org/en/docs/control.html#reconfiguration for details
on who configuration reload works.

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Shaun Tarves
Re: Large CRL file crashing nginx on reload
July 27, 2018 08:20PM
That is exactly the issue. Seeing what the "reload" did to the memory
(starting a new worker process) was the culprit. I was thinking the
configuration reload should just refresh what's in memory, but it clearly
doubles the requirement of memory and must wait until the previous child
can stop gracefully.

Thank you for the help!

On Fri, Jul 27, 2018 at 10:56 AM Shaun Tarves <[email protected]>
wrote:

> Here are the relevant parts of our configuration:
>
> worker_processes 1;
> pid /var/run/nginx.pid;
> events {
> worker_connections 512;
> }
> http {
> server {
> listen xx.xx.xx.xx:443 default_server ssl;
> ssl on;
> ssl_certificate /opt/xxx.pem;
> ssl_certificate_key /opt/xxx.key
> ssl_ciphers 'AES128+EECDH:AES128+EDH:!aNULL';
> ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
> ssl_session_cache shared:SSL:10m;
> ssl_prefer_server_ciphers on;
> ssl_verify_client optional;
> ssl_client_certificate /opt/ca.crt.pem
> ssl_crl /opt/ca.crl/.pem;
> }
> }
>
> During a "reload" command, here is how our ps looks:
>
> [[email protected] nginx]# service nginx reload
>
> Reloading nginx: [ OK ]
>
> [[email protected] nginx]# ps -ef | grep nginx
>
> root 9605 1 9 15:06 ? 00:00:17 nginx: master process
> /usr/sbin/nginx -c /etc/nginx/nginx.conf
>
> cons3rt 9606 9605 0 15:06 ? 00:00:00 nginx: worker process
>
>
> root 11009 27847 0 15:09 pts/2 00:00:00 grep nginx
>
> [[email protected] nginx]# ps -ef | grep nginx
>
> root 9605 1 10 15:06 ? 00:00:24 nginx: master process
> /usr/sbin/nginx -c /etc/nginx/nginx.conf
>
> cons3rt 9606 9605 0 15:06 ? 00:00:00 nginx: worker process is
> shutting down
>
> root 11091 27847 0 15:10 pts/2 00:00:00 grep nginx
>
> [[email protected] nginx]# ps -ef | grep nginx
>
> root 9605 1 10 15:06 ? 00:00:24 nginx: master process
> /usr/sbin/nginx -c /etc/nginx/nginx.conf
>
> cons3rt 9606 9605 0 15:06 ? 00:00:00 nginx: worker process is
> shutting down
>
> root 11362 27847 0 15:10 pts/2 00:00:00 grep nginx
>
> [[email protected] nginx]# ps -ef | grep nginx
>
> root 9605 1 9 15:06 ? 00:00:24 nginx: master process
> /usr/sbin/nginx -c /etc/nginx/nginx.conf
>
> cons3rt 9606 9605 1 15:06 ? 00:00:02 nginx: worker process is
> shutting down
>
> root 11395 27847 0 15:10 pts/2 00:00:00 grep nginx
>
> [[email protected] nginx]# vi /var/log/nginx/error.log
>
> [[email protected] nginx]# ps -ef | grep nginx
>
> root 9605 1 7 15:06 ? 00:00:24 nginx: master process
> /usr/sbin/nginx -c /etc/nginx/nginx.conf
>
> cons3rt 9606 9605 5 15:06 ? 00:00:19 nginx: worker process is
> shutting down
>
> root 11771 27847 0 15:12 pts/2 00:00:00 grep nginx
>
> [[email protected] nginx]# service nginx stop
>
> Stopping nginx: [FAILED]
>
>
>
> On Thu, Jul 26, 2018 at 4:16 PM Shaun Tarves <
> [email protected]> wrote:
>
>> Hi,
>>
>> We are trying to use nginx to support the DoD PKI infrastructure, which
>> includes many DoD and contractor CRLs. The combined CRL file is over 350MB
>> in size, which seems to crash nginx during a reload (at least on Red Hat
>> 6). Our cert/key/crl set up is valid and working, and when only including a
>> subset of the CRL files we have, reloads work fine.
>>
>> When we concatenate all the CRLs we need to support, the config reload
>> request causes worker threads to become defunct and messages in the error
>> log indicate the following:
>>
>> 2018/07/26 16:05:25 [alert] 30624#30624: fork() failed while spawning
>> "worker process" (12: Cannot allocate memory)
>>
>> 2018/07/26 16:05:25 [alert] 30624#30624: sendmsg() failed (9: Bad file
>> descriptor)
>>
>> 2018/07/26 16:08:42 [alert] 30624#30624: worker process 1611 exited on
>> signal 9
>>
>> Is there any way we can get nginx to support such a large volume of CRLs?
>>
>>
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Sorry, only registered users may post in this forum.

Click here to login