Welcome! Log In Create A New Profile

Advanced

2 of 16 cores are constantly maxing out - how to balance the load?

Posted by Raffael Vogler 
Hello!

I have nginx with php-fpm running on a 16 core Ubuntu 16.04 instance. The
server is handling more than 10 million requests per hour.

https://imgur.com/a/iRZ7V

As you can see on the htop screenshot cores 6 and 7 are maxed out and
that's the case constantly - even after restarting nginx those two cores
stay at that level.

I wonder why is that so and how to balance the load more evenly?

Also I'm curious to know whether this might indicate a performance relevant
issue or if it is most likely harmless and just looks odd.

> cat /etc/nginx/nginx.conf | grep -v '^\s*#'

user www-data;
worker_processes auto;
pid /run/nginx.pid;
events {
worker_connections 768;
}
http {
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
ssl_prefer_server_ciphers on;
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
gzip on;
gzip_disable "msie6";
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}

Thanks

Raffael
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
If it’s the same two cores, it might be another process that uses the same two cores and thus happens to max out.
One very likely possibility would be interrupts from e.g. networking. You can check /proc/interrupts to see where interrupts from the network happens.

From: nginx <[email protected]> on behalf of Raffael Vogler <[email protected]>
Reply-To: "[email protected]" <[email protected]>
Date: Thursday, 11 January 2018 at 11.14
To: "[email protected]" <[email protected]>
Subject: 2 of 16 cores are constantly maxing out - how to balance the load?

Hello!

I have nginx with php-fpm running on a 16 core Ubuntu 16.04 instance. The server is handling more than 10 million requests per hour.

https://imgur.com/a/iRZ7V

As you can see on the htop screenshot cores 6 and 7 are maxed out and that's the case constantly - even after restarting nginx those two cores stay at that level.

I wonder why is that so and how to balance the load more evenly?

Also I'm curious to know whether this might indicate a performance relevant issue or if it is most likely harmless and just looks odd.

> cat /etc/nginx/nginx.conf | grep -v '^\s*#'



user www-data;

worker_processes auto;

pid /run/nginx.pid;

events {

worker_connections 768;

}

http {

sendfile on;

tcp_nopush on;

tcp_nodelay on;

keepalive_timeout 65;

types_hash_max_size 2048;

include /etc/nginx/mime.types;

default_type application/octet-stream;

ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE

ssl_prefer_server_ciphers on;

access_log /var/log/nginx/access.log;

error_log /var/log/nginx/error.log;

gzip on;

gzip_disable "msie6";

include /etc/nginx/conf.d/*.conf;

include /etc/nginx/sites-enabled/*;

}

Thanks

Raffael
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Hey Lucas, your assumption seems to be correct. According to
/proc/interrupts the following stats are significantly higher for those two
cores (CPU5, CPU6 - 0-based indexing):

- CPU5: xen-percpu-ipi callfuncsingle5
- CPU6: xen-percpu-ipi callfuncsingle6
- CPU5: xen-pirq-msi-x eth0-TxRx-0
- CPU6: xen-pirq-msi-x eth0-TxRx-1
- CPU5,6: TLB shootdowns
- CPU5,6: Hypervisor callback interrupts

Is this something that can and should be optimized or is it simply a matter
of fact due to the high load on the (fixed) available network card capacity?


On Thu, Jan 11, 2018 at 11:16 AM, Lucas Rolff <[email protected]> wrote:

> If it’s the same two cores, it might be another process that uses the same
> two cores and thus happens to max out.
>
> One very likely possibility would be interrupts from e.g. networking. You
> can check /proc/interrupts to see where interrupts from the network happens.
>
>
>
> *From: *nginx <[email protected]> on behalf of Raffael Vogler <
> [email protected]>
> *Reply-To: *"[email protected]" <[email protected]>
> *Date: *Thursday, 11 January 2018 at 11.14
> *To: *"[email protected]" <[email protected]>
> *Subject: *2 of 16 cores are constantly maxing out - how to balance the
> load?
>
>
>
> Hello!
>
> I have nginx with php-fpm running on a 16 core Ubuntu 16.04 instance. The
> server is handling more than 10 million requests per hour.
>
> https://imgur.com/a/iRZ7V
>
> As you can see on the htop screenshot cores 6 and 7 are maxed out and
> that's the case constantly - even after restarting nginx those two cores
> stay at that level.
>
> I wonder why is that so and how to balance the load more evenly?
>
> Also I'm curious to know whether this might indicate a performance
> relevant issue or if it is most likely harmless and just looks odd.
>
> > cat /etc/nginx/nginx.conf | grep -v '^\s*#'
>
>
>
> user www-data;
>
> worker_processes auto;
>
> pid /run/nginx.pid;
>
> events {
>
> worker_connections 768;
>
> }
>
> http {
>
> sendfile on;
>
> tcp_nopush on;
>
> tcp_nodelay on;
>
> keepalive_timeout 65;
>
> types_hash_max_size 2048;
>
> include /etc/nginx/mime.types;
>
> default_type application/octet-stream;
>
> ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
>
> ssl_prefer_server_ciphers on;
>
> access_log /var/log/nginx/access.log;
>
> error_log /var/log/nginx/error.log;
>
> gzip on;
>
> gzip_disable "msie6";
>
> include /etc/nginx/conf.d/*.conf;
>
> include /etc/nginx/sites-enabled/*;
>
> }
>
> Thanks
>
> Raffael
>
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx
>



--
Raffael Vogler, Chief Technology Officer

Yieldlove GmbH
Neuer Pferdemarkt 1, 20359 Hamburg

www.yieldlove.com

raffael.vogler@yieldlove.com
XING https://www.xing.com/profile/Raffael_Vogler - LinkedIn
https://www.linkedin.com/in/raffaelvogler
Skype: joyofdata

Registernummer: HRB 127559; Registergericht: Amtsgericht Hamburg; USt-ID:
DE815426709; Geschäftsführung:Benjamin Gries, Timo Hagenow, Ivan Tomic
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Or would it make sense (if possible at all) to assign two or three more
cores to networking interrupts?
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
In high traffic environments it generally make sense to “dedicate” a core to each RX and TX queue you have on the NIC – this way you lower the chances of a single core being overloaded from handling network and thus degrading performance.

And then at same time within nginx, map the individual processes to other cores.

So, let’s say say you have 8 cores and 1 RX and 1 TX queue:
Core 0: RX queue
Core 1: TX queue
Core 2 to 7: nginx processes

You’d then set nginx to 6 workers (if you’re not running other stuff on the box).

Now, in your case with php-fpm in the mix as well, controlling that can be hard ( not sure if you can pin php-fpm processes to cores ) – but for nginx and RX/TX queues, it’s for sure possible.

From: nginx <[email protected]> on behalf of Raffael Vogler <[email protected]>
Reply-To: "[email protected]" <[email protected]>
Date: Thursday, 11 January 2018 at 11.55
To: "[email protected]" <[email protected]>
Subject: Re: 2 of 16 cores are constantly maxing out - how to balance the load?

Or would it make sense (if possible at all) to assign two or three more cores to networking interrupts?
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
On 2018-01-11 11:59, Lucas Rolff wrote:
>
> Now, in your case with php-fpm in the mix as well, controlling that
> can be hard ( not sure if you can pin php-fpm processes to cores ) –
> but for nginx and RX/TX queues, it’s for sure possible.


Should be doable with cgroups / cpusets? CPUAffinity directive in the
service unit file ( see systemd.exec(5) )?



--
Vlad K.
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Perhaps you should use pidstat to validate which processes are running on the two busy cores?

> On Jan 11, 2018, at 6:25 AM, Vlad K. <[email protected]> wrote:
>
> On 2018-01-11 11:59, Lucas Rolff wrote:
>> Now, in your case with php-fpm in the mix as well, controlling that
>> can be hard ( not sure if you can pin php-fpm processes to cores ) –
>> but for nginx and RX/TX queues, it’s for sure possible.
>
>
> Should be doable with cgroups / cpusets? CPUAffinity directive in the service unit file ( see systemd.exec(5) )?
>
>
>
> --
> Vlad K.
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
> Perhaps you should use pidstat to validate which processes are running on the two busy cores?

Did that an can confirm that CPU 5 and 6 are not exclusively used by networking - but also by nginx and php-fpm.

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
> So, let’s say say you have 8 cores and 1 RX and 1 TX queue:
> Core 0: RX queue
> Core 1: TX queue
> Core 2 to 7: nginx processes

what tool or configuration file would I have to use to dedicate cores to processes?

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Sorry, only registered users may post in this forum.

Click here to login