Welcome! Log In Create A New Profile

Advanced

Performance test caps at 600 Mbit/s

Posted by MrFastDie 
MrFastDie
Performance test caps at 600 Mbit/s
January 06, 2017 04:30PM
Hello,

the last days I played a little with the NGINX settings and the tcp stack to
test the best performance. I used direct connection between my testing
machine and my server using a cat5e cable.
My nginx.conf can be found at pastebin: http://pastebin.com/rRAEwvNc
My sysctl.conf also provides lot of changes: http://pastebin.com/KmPjEnHN
My /etc/security/limits.conf provides soft nofile at 1048576 and hard nofile
at the same value.

I'm not using any kind of firewall, all logs are disabled, neither the cpu
nor the ram is at it's limit, the i/o load seems fine to me too, but the
internet speed caps round about 600 Mbit/s. This test was made with 2000
connections per time and a 1MB file.

The server contains an Intel(R) Pentium(R) Dual CPU E2220 @ 2.40GHz and
has 3.5G of Ram.
My local computer contains an Intel(R) Core(TM) i7-4710HQ CPU @ 2.50GHz and
has 16G of Ram.

The limits and tcp changes were also made on my local computer.

Is there someone who can help me to deal with this problem?

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,271846,271846#msg-271846

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
CJ Ess
Re: Performance test caps at 600 Mbit/s
January 06, 2017 05:30PM
Which OS? What NIC? You also have to consider the traffic source, is it
known capable of of saturating the NIC on your server?

On Fri, Jan 6, 2017 at 10:24 AM, MrFastDie <nginx-forum@forum.nginx.org>
wrote:

> Hello,
>
> the last days I played a little with the NGINX settings and the tcp stack
> to
> test the best performance. I used direct connection between my testing
> machine and my server using a cat5e cable.
> My nginx.conf can be found at pastebin: http://pastebin.com/rRAEwvNc
> My sysctl.conf also provides lot of changes: http://pastebin.com/KmPjEnHN
> My /etc/security/limits.conf provides soft nofile at 1048576 and hard
> nofile
> at the same value.
>
> I'm not using any kind of firewall, all logs are disabled, neither the cpu
> nor the ram is at it's limit, the i/o load seems fine to me too, but the
> internet speed caps round about 600 Mbit/s. This test was made with 2000
> connections per time and a 1MB file.
>
> The server contains an Intel(R) Pentium(R) Dual CPU E2220 @ 2.40GHz and
> has 3.5G of Ram.
> My local computer contains an Intel(R) Core(TM) i7-4710HQ CPU @ 2.50GHz and
> has 16G of Ram.
>
> The limits and tcp changes were also made on my local computer.
>
> Is there someone who can help me to deal with this problem?
>
> Posted at Nginx Forum: https://forum.nginx.org/read.
> php?2,271846,271846#msg-271846
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx
>
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
MrFastDie
Re: Performance test caps at 600 Mbit/s
January 09, 2017 09:40AM
The OS of the server is debian 8, at my testing machine i'm using arch
linux. Both NIC supports the speed of 1000Mb/s the server got round about
600 Mb/s up and 13Mb/s down.

CJ Ess Wrote:
-------------------------------------------------------
> Which OS? What NIC? You also have to consider the traffic source, is
> it
> known capable of of saturating the NIC on your server?
>
> On Fri, Jan 6, 2017 at 10:24 AM, MrFastDie
> <nginx-forum@forum.nginx.org>
> wrote:
>
> > Hello,
> >
> > the last days I played a little with the NGINX settings and the tcp
> stack
> > to
> > test the best performance. I used direct connection between my
> testing
> > machine and my server using a cat5e cable.
> > My nginx.conf can be found at pastebin: http://pastebin.com/rRAEwvNc
> > My sysctl.conf also provides lot of changes:
> http://pastebin.com/KmPjEnHN
> > My /etc/security/limits.conf provides soft nofile at 1048576 and
> hard
> > nofile
> > at the same value.
> >
> > I'm not using any kind of firewall, all logs are disabled, neither
> the cpu
> > nor the ram is at it's limit, the i/o load seems fine to me too, but
> the
> > internet speed caps round about 600 Mbit/s. This test was made with
> 2000
> > connections per time and a 1MB file.
> >
> > The server contains an Intel(R) Pentium(R) Dual CPU E2220 @
> 2.40GHz and
> > has 3.5G of Ram.
> > My local computer contains an Intel(R) Core(TM) i7-4710HQ CPU @
> 2.50GHz and
> > has 16G of Ram.
> >
> > The limits and tcp changes were also made on my local computer.
> >
> > Is there someone who can help me to deal with this problem?
> >
> > Posted at Nginx Forum: https://forum.nginx.org/read.
> > php?2,271846,271846#msg-271846
> >
> > _______________________________________________
> > nginx mailing list
> > nginx@nginx.org
> > http://mailman.nginx.org/mailman/listinfo/nginx
> >
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,271846,271869#msg-271869

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Valentin V. Bartenev
Re: Performance test caps at 600 Mbit/s
January 09, 2017 03:50PM
On Friday 06 January 2017 10:24:06 MrFastDie wrote:
> Hello,
>
> the last days I played a little with the NGINX settings and the tcp stack to
> test the best performance. I used direct connection between my testing
> machine and my server using a cat5e cable.
> My nginx.conf can be found at pastebin: http://pastebin.com/rRAEwvNc
> My sysctl.conf also provides lot of changes: http://pastebin.com/KmPjEnHN
> My /etc/security/limits.conf provides soft nofile at 1048576 and hard nofile
> at the same value.
>
[..]

You should configure worker_processes to use all cpu cores and disable
accept_mutex.

You can find more information in this talk:
https://www.youtube.com/watch?v=eLW_NSuwYU0

wbr, Valentin V. Bartenev

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Anonymous User
Re: Performance test caps at 600 Mbit/s
January 09, 2017 04:50PM
‎FYI, benchmark mentioned in the video.
https://github.com/wg/wrk

Wouldn't a number of test machine ls on the Internet make more sense than flogging nginx locally on your network?

With VPS time being sold by the hour, seems to me you should get one VPS tester running acceptably, then clone a dozen and do your test. With SSD based VPS, you can literally clone one a minute.
  Original Message  
From: Valentin V. Bartenev
Sent: Monday, January 9, 2017 6:44 AM
To: nginx@nginx.org
Reply To: nginx@nginx.org
Subject: Re: Performance test caps at 600 Mbit/s

On Friday 06 January 2017 10:24:06 MrFastDie wrote:
> Hello,
>
> the last days I played a little with the NGINX settings and the tcp stack to
> test the best performance. I used direct connection between my testing
> machine and my server using a cat5e cable.
> My nginx.conf can be found at pastebin: http://pastebin.com/rRAEwvNc
> My sysctl.conf also provides lot of changes: http://pastebin.com/KmPjEnHN
> My /etc/security/limits.conf provides soft nofile at 1048576 and hard nofile
> at the same value.
>
[..]

You should configure worker_processes to use all cpu cores and disable
accept_mutex.

You can find more information in this talk:
https://www.youtube.com/watch?v=eLW_NSuwYU0

wbr, Valentin V. Bartenev

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
CJ Ess
Re: Performance test caps at 600 Mbit/s
January 09, 2017 05:40PM
Out of the box Linux isn't optimized for high speed networking - if you
google for linux and 1g or 10g you'll find a ton of pages about
configuration changes.

I think you'll want to do something like:

ifconfig eth0 txqueuelen 10000
ethtool -G eth0 rx 4096 tx 4096
ip route | while read x; do ip route change $x initcwnd 10 initrwnd 10; done

And these sysctl settings:
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 10
net.ipv4.tcp_keepalive_time = 60
net.ipv4.tcp_keepalive_probes = 14
net.ipv4.tcp_keepalive_intvl = 15
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.tcp_syncookies = 1
net.ipv4.ip_nonlocal_bind = 1
net.ipv4.tcp_sack = 1
net.ipv4.ip_local_port_range = 1024 65535
net.core.rmem_max = 33554432
net.core.wmem_max = 33554432
net.core.rmem_default = 524288
net.core.rmem_default = 524288
net.ipv4.tcp_rmem = 4096 524288 33554432
net.ipv4.tcp_wmem = 4096 524288 33554432
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.tcp_max_tw_buckets = 2000000
net.ipv4.tcp_max_syn_backlog = 65536
net.ipv4.tcp_max_orphans = 262144
net.core.netdev_max_backlog = 300000
net.core.somaxconn = 65536
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_syn_retries = 2
net.ipv4.tcp_slow_start_after_idle = 0

At least these are the settings I've used for benchmarking, may do less or
more on your setup. The NIC vendor can be really important - Intel NICs are
really good and work really well with Linux. Broadcom NICs tend to flake
out around line rate and are also notorious for dropping packets in
hardware and not telling the kernel, so I try not to use those.


On Mon, Jan 9, 2017 at 3:31 AM, MrFastDie <nginx-forum@forum.nginx.org>
wrote:

> The OS of the server is debian 8, at my testing machine i'm using arch
> linux. Both NIC supports the speed of 1000Mb/s the server got round about
> 600 Mb/s up and 13Mb/s down.
>
> CJ Ess Wrote:
> -------------------------------------------------------
> > Which OS? What NIC? You also have to consider the traffic source, is
> > it
> > known capable of of saturating the NIC on your server?
> >
> > On Fri, Jan 6, 2017 at 10:24 AM, MrFastDie
> > <nginx-forum@forum.nginx.org>
> > wrote:
> >
> > > Hello,
> > >
> > > the last days I played a little with the NGINX settings and the tcp
> > stack
> > > to
> > > test the best performance. I used direct connection between my
> > testing
> > > machine and my server using a cat5e cable.
> > > My nginx.conf can be found at pastebin: http://pastebin.com/rRAEwvNc
> > > My sysctl.conf also provides lot of changes:
> > http://pastebin.com/KmPjEnHN
> > > My /etc/security/limits.conf provides soft nofile at 1048576 and
> > hard
> > > nofile
> > > at the same value.
> > >
> > > I'm not using any kind of firewall, all logs are disabled, neither
> > the cpu
> > > nor the ram is at it's limit, the i/o load seems fine to me too, but
> > the
> > > internet speed caps round about 600 Mbit/s. This test was made with
> > 2000
> > > connections per time and a 1MB file.
> > >
> > > The server contains an Intel(R) Pentium(R) Dual CPU E2220 @
> > 2.40GHz and
> > > has 3.5G of Ram.
> > > My local computer contains an Intel(R) Core(TM) i7-4710HQ CPU @
> > 2.50GHz and
> > > has 16G of Ram.
> > >
> > > The limits and tcp changes were also made on my local computer.
> > >
> > > Is there someone who can help me to deal with this problem?
> > >
> > > Posted at Nginx Forum: https://forum.nginx.org/read.
> > > php?2,271846,271846#msg-271846
> > >
> > > _______________________________________________
> > > nginx mailing list
> > > nginx@nginx.org
> > > http://mailman.nginx.org/mailman/listinfo/nginx
> > >
> > _______________________________________________
> > nginx mailing list
> > nginx@nginx.org
> > http://mailman.nginx.org/mailman/listinfo/nginx
>
> Posted at Nginx Forum: https://forum.nginx.org/read.
> php?2,271846,271869#msg-271869
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx
>
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
pbooth
Re: Performance test caps at 600 Mbit/s
January 09, 2017 10:10PM
> Both NIC supports the speed of 1000Mb/s

How do you know? Your kernel or NIC config might be limiting you.

iperf, snfnettest, or etherate will show you the maximum possible
bandwidth at the TCP or IP layer.
If it's under 700 then you know to focus on the NIC and OS. If it's above
900 then the
problem is in your nginx or your test workload.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,271846,271895#msg-271895

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
oscaretu .
Re: Performance test caps at 600 Mbit/s
January 09, 2017 10:40PM
Hello, I don't find anything in Google about

snfnettest

Can you confirm that the name is OK?

Kind regards,
Oscar


On Mon, Jan 9, 2017 at 10:02 PM, pbooth <nginx-forum@forum.nginx.org> wrote:

> > Both NIC supports the speed of 1000Mb/s
>
> How do you know? Your kernel or NIC config might be limiting you.
>
> iperf, snfnettest, or etherate will show you the maximum possible
> bandwidth at the TCP or IP layer.
> If it's under 700 then you know to focus on the NIC and OS. If it's above
> 900 then the
> problem is in your nginx or your test workload.
>
> Posted at Nginx Forum: https://forum.nginx.org/read.
> php?2,271846,271895#msg-271895
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx
>



--
Oscar Fernandez Sierra
oscaretu@gmail.com
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Sorry, only registered users may post in this forum.

Click here to login