Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer
Locked
12
12
 
|
Well, to add some datapoints, I just did a quick test om my private homeserver, testing with iperf3
-s / iperf3 -c <server-ip> I have not much knowledge of iperf3 network test and it's validity, but there is definitely some differences to be seen. Summary: FBSD <-> FBSD 3,5Gbit/sec FBSD -> Centos: 3GBit/Sec Centos -> FBSD: 12 Gbit/sec Centos <-> Centos: 12 Gbit/sec OpenSuse <-> OpenSuse 13 GBit/sec So it ssems there is some fishy when Freebsd is used as client xen running on opensuse, 11 vm's, freebsd , openSuse and Centos. No vm has been optimized in any way, everything is out of the box. =========================== Test 1, Freebsd 12 -> freebsd 12 =========================== Server Freebsd 12: ccepted connection from 192.168.1.100, port 39518 [ 5] local 192.168.1.99 port 5201 connected to 192.168.1.100 port 48518 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 374 MBytes 3.14 Gbits/sec [ 5] 1.00-2.00 sec 444 MBytes 3.72 Gbits/sec [ 5] 2.00-3.00 sec 411 MBytes 3.45 Gbits/sec [ 5] 3.00-4.00 sec 438 MBytes 3.67 Gbits/sec [ 5] 4.00-5.00 sec 413 MBytes 3.46 Gbits/sec [ 5] 5.00-6.00 sec 417 MBytes 3.50 Gbits/sec [ 5] 6.00-7.00 sec 412 MBytes 3.45 Gbits/sec [ 5] 7.00-8.00 sec 432 MBytes 3.63 Gbits/sec [ 5] 8.00-9.00 sec 418 MBytes 3.51 Gbits/sec [ 5] 9.00-10.00 sec 375 MBytes 3.15 Gbits/sec [ 5] 10.00-10.00 sec 332 KBytes 2.17 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.00 sec 4.04 GBytes 3.47 Gbits/sec receiver Client, freebsd 12: Connecting to host 10.192.168.99, port 5201 [ 5] local 192.168.1.100 port 48518 connected to 192.168.1.99 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 375 MBytes 3.14 Gbits/sec 239 178 KBytes [ 5] 1.00-2.00 sec 445 MBytes 3.73 Gbits/sec 834 74.1 KBytes [ 5] 2.00-3.00 sec 411 MBytes 3.45 Gbits/sec 794 38.5 KBytes [ 5] 3.00-4.00 sec 438 MBytes 3.68 Gbits/sec 808 198 KBytes [ 5] 4.00-5.00 sec 413 MBytes 3.46 Gbits/sec 777 67.0 KBytes [ 5] 5.00-6.00 sec 417 MBytes 3.50 Gbits/sec 795 120 KBytes [ 5] 6.00-7.00 sec 412 MBytes 3.45 Gbits/sec 763 72.7 KBytes [ 5] 7.00-8.00 sec 432 MBytes 3.62 Gbits/sec 810 329 KBytes [ 5] 8.00-9.00 sec 418 MBytes 3.51 Gbits/sec 777 219 KBytes [ 5] 9.00-10.00 sec 375 MBytes 3.15 Gbits/sec 688 304 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 4.04 GBytes 3.47 Gbits/sec 7285 sender [ 5] 0.00-10.00 sec 4.04 GBytes 3.47 Gbits/sec receiver ========================== test 2, centos 7.6 -> Freebsd 12 ========================== Server Freebsd 12: Accepted connection from 192.168.1.162, port 49180 [ 5] local 192.168.1.99 port 5201 connected to 192.168.1.162 port 49182 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 1.17 GBytes 10.0 Gbits/sec [ 5] 1.00-2.00 sec 1.47 GBytes 12.6 Gbits/sec [ 5] 2.00-3.00 sec 1.30 GBytes 11.2 Gbits/sec [ 5] 3.00-4.00 sec 1.32 GBytes 11.3 Gbits/sec [ 5] 4.00-5.00 sec 1.44 GBytes 12.4 Gbits/sec [ 5] 5.00-6.00 sec 1.29 GBytes 11.1 Gbits/sec [ 5] 6.00-7.00 sec 1.32 GBytes 11.3 Gbits/sec [ 5] 7.00-8.00 sec 1.36 GBytes 11.7 Gbits/sec [ 5] 8.00-9.00 sec 1.32 GBytes 11.3 Gbits/sec [ 5] 9.00-10.00 sec 1.41 GBytes 12.1 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.00 sec 13.4 GBytes 11.5 Gbits/sec receiver Client, centos 7.6: Connecting to host 192.168.1.99, port 5201 [ 4] local 192.168.1.162 port 49182 connected to 192.168.1.99 port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 1.17 GBytes 10.1 Gbits/sec 0 822 KBytes [ 4] 1.00-2.00 sec 1.47 GBytes 12.6 Gbits/sec 0 1.53 MBytes [ 4] 2.00-3.00 sec 1.30 GBytes 11.2 Gbits/sec 0 1.57 MBytes [ 4] 3.00-4.00 sec 1.32 GBytes 11.3 Gbits/sec 0 1.64 MBytes [ 4] 4.00-5.00 sec 1.44 GBytes 12.4 Gbits/sec 0 1.70 MBytes [ 4] 5.00-6.00 sec 1.29 GBytes 11.1 Gbits/sec 0 1.70 MBytes [ 4] 6.00-7.00 sec 1.32 GBytes 11.3 Gbits/sec 0 1.70 MBytes [ 4] 7.00-8.00 sec 1.36 GBytes 11.7 Gbits/sec 0 1.70 MBytes [ 4] 8.00-9.00 sec 1.32 GBytes 11.3 Gbits/sec 0 1.70 MBytes [ 4] 9.00-10.00 sec 1.41 GBytes 12.1 Gbits/sec 0 1.70 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 13.4 GBytes 11.5 Gbits/sec 0 sender [ 4] 0.00-10.00 sec 13.4 GBytes 11.5 Gbits/sec receiver ========================== test 3, Freebsd 12 -> Centos 7.6 ========================== Server, Centos 7.6: Accepted connection from 192.168.1.100, port 20309 [ 5] local 192.168.1.155 port 5201 connected to 192.168.1.100 port 17917 [ ID] Interval Transfer Bandwidth [ 5] 0.00-1.00 sec 335 MBytes 2.81 Gbits/sec [ 5] 1.00-2.00 sec 379 MBytes 3.18 Gbits/sec [ 5] 2.00-3.00 sec 368 MBytes 3.08 Gbits/sec [ 5] 3.00-4.00 sec 371 MBytes 3.11 Gbits/sec [ 5] 4.00-5.00 sec 374 MBytes 3.14 Gbits/sec [ 5] 5.00-6.00 sec 362 MBytes 3.04 Gbits/sec [ 5] 6.00-7.00 sec 381 MBytes 3.19 Gbits/sec [ 5] 7.00-8.00 sec 384 MBytes 3.22 Gbits/sec [ 5] 8.00-9.00 sec 374 MBytes 3.14 Gbits/sec [ 5] 9.00-10.00 sec 366 MBytes 3.07 Gbits/sec [ 5] 10.00-10.13 sec 56.4 MBytes 3.52 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 5] 0.00-10.13 sec 0.00 Bytes 0.00 bits/sec sender [ 5] 0.00-10.13 sec 3.66 GBytes 3.10 Gbits/sec receiver Client, Freebsd 12: 5] local 192.168.1.100 port 17917 connected to 192.168.1.155 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 387 MBytes 3.25 Gbits/sec 83 309 KBytes [ 5] 1.00-2.00 sec 380 MBytes 3.19 Gbits/sec 88 11.4 KBytes [ 5] 2.00-3.00 sec 365 MBytes 3.06 Gbits/sec 83 54.2 KBytes [ 5] 3.00-4.00 sec 372 MBytes 3.12 Gbits/sec 86 322 KBytes [ 5] 4.00-5.00 sec 370 MBytes 3.11 Gbits/sec 83 2.85 KBytes [ 5] 5.00-6.00 sec 366 MBytes 3.07 Gbits/sec 84 298 KBytes [ 5] 6.00-7.00 sec 382 MBytes 3.20 Gbits/sec 87 356 KBytes [ 5] 7.00-8.00 sec 380 MBytes 3.19 Gbits/sec 87 369 KBytes [ 5] 8.00-9.00 sec 374 MBytes 3.14 Gbits/sec 87 29.9 KBytes [ 5] 9.00-10.00 sec 375 MBytes 3.15 Gbits/sec 86 419 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 3.66 GBytes 3.15 Gbits/sec 854 sender [ 5] 0.00-10.00 sec 3.66 GBytes 3.15 Gbits/sec receiver ========================== test 4, Centos 7.6 -> Centos 7.6 ========================== Server: Accepted connection from 192.168.1.162, port 47606 [ 5] local 192.168.1.155 port 5201 connected to 192.168.1.162 port 47608 [ ID] Interval Transfer Bandwidth [ 5] 0.00-1.00 sec 1.60 GBytes 13.7 Gbits/sec [ 5] 1.00-2.00 sec 1.59 GBytes 13.7 Gbits/sec [ 5] 2.00-3.00 sec 1.60 GBytes 13.8 Gbits/sec [ 5] 3.00-4.00 sec 1.71 GBytes 14.7 Gbits/sec [ 5] 4.00-5.00 sec 1.49 GBytes 12.8 Gbits/sec [ 5] 5.00-6.00 sec 1.38 GBytes 11.9 Gbits/sec [ 5] 6.00-7.00 sec 1.52 GBytes 13.1 Gbits/sec [ 5] 7.00-8.00 sec 1.51 GBytes 13.0 Gbits/sec [ 5] 8.00-9.00 sec 1.51 GBytes 13.0 Gbits/sec [ 5] 9.00-10.00 sec 1.51 GBytes 12.9 Gbits/sec [ 5] 10.00-10.03 sec 48.1 MBytes 11.9 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 5] 0.00-10.03 sec 0.00 Bytes 0.00 bits/sec sender [ 5] 0.00-10.03 sec 15.5 GBytes 13.2 Gbits/sec receiver Client, Centos 7.6 Connecting to host 192.168.1.155, port 5201 [ 4] local 192.168.1.162 port 47608 connected to 192.168.1.155 port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 1.66 GBytes 14.3 Gbits/sec 0 1.41 MBytes [ 4] 1.00-2.00 sec 1.58 GBytes 13.5 Gbits/sec 0 1.46 MBytes [ 4] 2.00-3.00 sec 1.61 GBytes 13.9 Gbits/sec 0 1.91 MBytes [ 4] 3.00-4.00 sec 1.71 GBytes 14.7 Gbits/sec 0 2.13 MBytes [ 4] 4.00-5.00 sec 1.47 GBytes 12.6 Gbits/sec 0 2.49 MBytes [ 4] 5.00-6.00 sec 1.39 GBytes 11.9 Gbits/sec 0 2.56 MBytes [ 4] 6.00-7.00 sec 1.53 GBytes 13.1 Gbits/sec 0 2.71 MBytes [ 4] 7.00-8.00 sec 1.50 GBytes 12.9 Gbits/sec 0 2.73 MBytes [ 4] 8.00-9.00 sec 1.50 GBytes 12.9 Gbits/sec 0 2.81 MBytes [ 4] 9.00-10.00 sec 1.51 GBytes 12.9 Gbits/sec 0 2.84 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 15.5 GBytes 13.3 Gbits/sec 0 sender [ 4] 0.00-10.00 sec 15.5 GBytes 13.3 Gbits/sec receiver =============================== test4, Opensuse 15.1 -> OpenSuse 15.1 ================================ Server, Opensuse 15.1: [ 5] local 192.168.1.158 port 5201 connected to 192.168.1.163 port 39334 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 1.51 GBytes 13.0 Gbits/sec [ 5] 1.00-2.00 sec 1.61 GBytes 13.8 Gbits/sec [ 5] 2.00-3.00 sec 1.58 GBytes 13.6 Gbits/sec [ 5] 3.00-4.00 sec 1.58 GBytes 13.6 Gbits/sec [ 5] 4.00-5.00 sec 1.56 GBytes 13.4 Gbits/sec [ 5] 5.00-6.00 sec 1.60 GBytes 13.7 Gbits/sec [ 5] 6.00-7.00 sec 1.58 GBytes 13.5 Gbits/sec [ 5] 7.00-8.00 sec 1.60 GBytes 13.8 Gbits/sec [ 5] 8.00-9.00 sec 1.60 GBytes 13.7 Gbits/sec [ 5] 9.00-10.00 sec 1.61 GBytes 13.8 Gbits/sec [ 5] 10.00-10.04 sec 68.0 MBytes 14.2 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.04 sec 15.9 GBytes 13.6 Gbits/sec receiver Client, OpenSuse 15.1: [ 5] local 192.168.1.163 port 39334 connected to 192.168.1.158 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 1.58 GBytes 13.6 Gbits/sec 0 723 KBytes [ 5] 1.00-2.00 sec 1.61 GBytes 13.8 Gbits/sec 0 805 KBytes [ 5] 2.00-3.00 sec 1.57 GBytes 13.5 Gbits/sec 0 926 KBytes [ 5] 3.00-4.00 sec 1.58 GBytes 13.6 Gbits/sec 0 926 KBytes [ 5] 4.00-5.00 sec 1.56 GBytes 13.4 Gbits/sec 0 970 KBytes [ 5] 5.00-6.00 sec 1.60 GBytes 13.7 Gbits/sec 0 1.05 MBytes [ 5] 6.00-7.00 sec 1.57 GBytes 13.5 Gbits/sec 0 1.05 MBytes [ 5] 7.00-8.00 sec 1.61 GBytes 13.8 Gbits/sec 0 1.05 MBytes [ 5] 8.00-9.00 sec 1.59 GBytes 13.7 Gbits/sec 0 1.05 MBytes [ 5] 9.00-10.00 sec 1.62 GBytes 13.9 Gbits/sec 0 1.05 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 15.9 GBytes 13.7 Gbits/sec 0 sender [ 5] 0.00-10.04 sec 15.9 GBytes 13.6 Gbits/sec receiver > I've noticed very slow networking speed between VM's with FreeBSD on the > same host (XCP-ng 7.6.0) for more recent FreeBSD-versions. > > > I've made some tests that show me that something happened from 10.4-RELEASE > to 11.0-RELEASE that had a huge impact on network performance, and > 12.0-RELEASE is even slower. > > > My test setup: > > Host: XCP-ng 7.6.0, managed with XenOrchestra. Open source. > > Network: Internal Private Network on the host (not connected to a PIF). > Each VM has only one VIF connected to this network. > > VM's: > > 2 x 12.0-RELEASE > > 2 x 11.0-RELEASE > > 2 x 10.4-RELEASE > > > All clean identical installs from XenOrchestra, only installed iperf on > each VM for testing. (xe-guest-utilities makes no difference in my tests, > I've tried with and without). > > > iperf -s on first server listed below, and iperf -c <ip> -r on the second > to test speed back and forth: > > > 12.0 <-> 12.0: 50Mbit as client and server > > 12.0 <-> 11.0: 800Mbit/s (11.0 as client), and 140Mbit/s (11.0 as server) > > 12.0 <-> 10.4: 2.76Gbit (10.4 as client), and 1.25Gbit (10.4 as server). > > 11.0 <-> 11.0: 219Mbit as client, 99Mbit as server > > 10.4 <-> 10.4: 11.2Gbit as client, 10.9Gbit as server > > > As I side note, not sure if related, but I've noticed that I can't run > iperf with -r flag on 10.4-RELEASE. I get this error message: > > > iperf -c 172.31.16.122 -r > > ------------------------------------------------------------ > > Server listening on TCP port 5001 > > TCP window size: 64.0 KByte (default) > > ------------------------------------------------------------ > > write failed: Broken pipe > > ------------------------------------------------------------ > > Client connecting to 172.31.16.122, TCP port 5001 > > TCP window size: 32.5 KByte (default) > > ------------------------------------------------------------ > > [ 5] local 172.31.16.121 port 19231 connected with 172.31.16.122 port 5001 > > [ ID] Interval Transfer Bandwidth > > [ 5] 0.0- 0.0 sec 0.00 Bytes 0.00 bits/sec > > > > I can run iperf -s fine, and iperf -c <ip> from the other 10.4 VM though: > > > > iperf -c 172.31.16.122 > > ------------------------------------------------------------ > > Client connecting to 172.31.16.122, TCP port 5001 > > TCP window size: 32.5 KByte (default) > > ------------------------------------------------------------ > > [ 3] local 172.31.16.121 port 22055 connected with 172.31.16.122 port 5001 > > [ ID] Interval Transfer Bandwidth > > [ 3] 0.0-10.0 sec 12.9 GBytes 11.1 Gbits/sec > > > > What have I tried to solve this? > > I've tried to disable checksum offloading for the 12.0-RELEASE VIF's via > XCP-ng. Disabled basically everything without any difference in iperf > results: other-config (MRW): ethtool-sg: off; ethtool-tso: off; > ethtool-ufo: off; ethtool-gso: off; ethtool-rx: off; ethtool-tx: off > > > Also tried disabling offloading in FreeBSD with ifconfig xn0 -txcsum > -rxcsum -tso -lro and no difference here either. > > > Any ideas of how to proceed now to find a solution for this? > > > Regards, > > Christian > -- mvh, Bjarne _______________________________________________ [hidden email] mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "[hidden email]" |
Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer
|
|
Thanks for your input Bjarne.
Previously I tested with iperf2, but I have made som tests with iperf3 now also and noticed there are a lot of "Retr" (TCP retries) in some cases. Went back here to your post and saw that your results also showed a lot of retries in some cases. My new tests showed similar results to yours (although not nearly as hi throughput as you have). When Linux is client the retries are 0, with FreeBSD as client the number is > 0. I'm not sure if our numbers are considered high though, and something that actually is a problem? Although, on a internal network with no external factors like interference, I feel this should always be 0 no matter what? What hypervisor were you running in your tests? Version? Regards, Christian Den mån 24 juni 2019 kl 17:56 skrev Bjarne <[hidden email]>: > Well, to add some datapoints, I just did a quick test om my private > homeserver, testing with iperf3 > -s / iperf3 -c <server-ip> > > I have not much knowledge of iperf3 network test and it's validity, but > there is definitely some > differences to be seen. > > Summary: > > FBSD <-> FBSD 3,5Gbit/sec > FBSD -> Centos: 3GBit/Sec > Centos -> FBSD: 12 Gbit/sec > > Centos <-> Centos: 12 Gbit/sec > OpenSuse <-> OpenSuse 13 GBit/sec > > So it ssems there is some fishy when Freebsd is used as client > > xen running on opensuse, 11 vm's, freebsd , openSuse and Centos. No vm > has been optimized in any > way, everything is out of the box. > > > =========================== > Test 1, Freebsd 12 -> freebsd 12 > =========================== > > Server Freebsd 12: > ccepted connection from 192.168.1.100, port 39518 > [ 5] local 192.168.1.99 port 5201 connected to 192.168.1.100 port 48518 > [ ID] Interval Transfer Bitrate > [ 5] 0.00-1.00 sec 374 MBytes 3.14 Gbits/sec > [ 5] 1.00-2.00 sec 444 MBytes 3.72 Gbits/sec > [ 5] 2.00-3.00 sec 411 MBytes 3.45 Gbits/sec > [ 5] 3.00-4.00 sec 438 MBytes 3.67 Gbits/sec > [ 5] 4.00-5.00 sec 413 MBytes 3.46 Gbits/sec > [ 5] 5.00-6.00 sec 417 MBytes 3.50 Gbits/sec > [ 5] 6.00-7.00 sec 412 MBytes 3.45 Gbits/sec > [ 5] 7.00-8.00 sec 432 MBytes 3.63 Gbits/sec > [ 5] 8.00-9.00 sec 418 MBytes 3.51 Gbits/sec > [ 5] 9.00-10.00 sec 375 MBytes 3.15 Gbits/sec > [ 5] 10.00-10.00 sec 332 KBytes 2.17 Gbits/sec > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bitrate > [ 5] 0.00-10.00 sec 4.04 GBytes 3.47 Gbits/sec > receiver > > Client, freebsd 12: > Connecting to host 10.192.168.99, port 5201 > [ 5] local 192.168.1.100 port 48518 connected to 192.168.1.99 port 5201 > [ ID] Interval Transfer Bitrate Retr Cwnd > [ 5] 0.00-1.00 sec 375 MBytes 3.14 Gbits/sec 239 178 KBytes > [ 5] 1.00-2.00 sec 445 MBytes 3.73 Gbits/sec 834 74.1 KBytes > [ 5] 2.00-3.00 sec 411 MBytes 3.45 Gbits/sec 794 38.5 KBytes > [ 5] 3.00-4.00 sec 438 MBytes 3.68 Gbits/sec 808 198 KBytes > [ 5] 4.00-5.00 sec 413 MBytes 3.46 Gbits/sec 777 67.0 KBytes > [ 5] 5.00-6.00 sec 417 MBytes 3.50 Gbits/sec 795 120 KBytes > [ 5] 6.00-7.00 sec 412 MBytes 3.45 Gbits/sec 763 72.7 KBytes > [ 5] 7.00-8.00 sec 432 MBytes 3.62 Gbits/sec 810 329 KBytes > [ 5] 8.00-9.00 sec 418 MBytes 3.51 Gbits/sec 777 219 KBytes > [ 5] 9.00-10.00 sec 375 MBytes 3.15 Gbits/sec 688 304 KBytes > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bitrate Retr > [ 5] 0.00-10.00 sec 4.04 GBytes 3.47 Gbits/sec 7285 > sender > [ 5] 0.00-10.00 sec 4.04 GBytes 3.47 Gbits/sec > receiver > > > ========================== > test 2, centos 7.6 -> Freebsd 12 > ========================== > > Server Freebsd 12: > Accepted connection from 192.168.1.162, port 49180 > [ 5] local 192.168.1.99 port 5201 connected to 192.168.1.162 port 49182 > [ ID] Interval Transfer Bitrate > [ 5] 0.00-1.00 sec 1.17 GBytes 10.0 Gbits/sec > [ 5] 1.00-2.00 sec 1.47 GBytes 12.6 Gbits/sec > [ 5] 2.00-3.00 sec 1.30 GBytes 11.2 Gbits/sec > [ 5] 3.00-4.00 sec 1.32 GBytes 11.3 Gbits/sec > [ 5] 4.00-5.00 sec 1.44 GBytes 12.4 Gbits/sec > [ 5] 5.00-6.00 sec 1.29 GBytes 11.1 Gbits/sec > [ 5] 6.00-7.00 sec 1.32 GBytes 11.3 Gbits/sec > [ 5] 7.00-8.00 sec 1.36 GBytes 11.7 Gbits/sec > [ 5] 8.00-9.00 sec 1.32 GBytes 11.3 Gbits/sec > [ 5] 9.00-10.00 sec 1.41 GBytes 12.1 Gbits/sec > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bitrate > [ 5] 0.00-10.00 sec 13.4 GBytes 11.5 Gbits/sec > receiver > > Client, centos 7.6: > Connecting to host 192.168.1.99, port 5201 > [ 4] local 192.168.1.162 port 49182 connected to 192.168.1.99 port 5201 > [ ID] Interval Transfer Bandwidth Retr Cwnd > [ 4] 0.00-1.00 sec 1.17 GBytes 10.1 Gbits/sec 0 822 KBytes > [ 4] 1.00-2.00 sec 1.47 GBytes 12.6 Gbits/sec 0 1.53 MBytes > [ 4] 2.00-3.00 sec 1.30 GBytes 11.2 Gbits/sec 0 1.57 MBytes > [ 4] 3.00-4.00 sec 1.32 GBytes 11.3 Gbits/sec 0 1.64 MBytes > [ 4] 4.00-5.00 sec 1.44 GBytes 12.4 Gbits/sec 0 1.70 MBytes > [ 4] 5.00-6.00 sec 1.29 GBytes 11.1 Gbits/sec 0 1.70 MBytes > [ 4] 6.00-7.00 sec 1.32 GBytes 11.3 Gbits/sec 0 1.70 MBytes > [ 4] 7.00-8.00 sec 1.36 GBytes 11.7 Gbits/sec 0 1.70 MBytes > [ 4] 8.00-9.00 sec 1.32 GBytes 11.3 Gbits/sec 0 1.70 MBytes > [ 4] 9.00-10.00 sec 1.41 GBytes 12.1 Gbits/sec 0 1.70 MBytes > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bandwidth Retr > [ 4] 0.00-10.00 sec 13.4 GBytes 11.5 Gbits/sec 0 sender > [ 4] 0.00-10.00 sec 13.4 GBytes 11.5 Gbits/sec > receiver > > > ========================== > test 3, Freebsd 12 -> Centos 7.6 > ========================== > > Server, Centos 7.6: > Accepted connection from 192.168.1.100, port 20309 > [ 5] local 192.168.1.155 port 5201 connected to 192.168.1.100 port 17917 > [ ID] Interval Transfer Bandwidth > [ 5] 0.00-1.00 sec 335 MBytes 2.81 Gbits/sec > [ 5] 1.00-2.00 sec 379 MBytes 3.18 Gbits/sec > [ 5] 2.00-3.00 sec 368 MBytes 3.08 Gbits/sec > [ 5] 3.00-4.00 sec 371 MBytes 3.11 Gbits/sec > [ 5] 4.00-5.00 sec 374 MBytes 3.14 Gbits/sec > [ 5] 5.00-6.00 sec 362 MBytes 3.04 Gbits/sec > [ 5] 6.00-7.00 sec 381 MBytes 3.19 Gbits/sec > [ 5] 7.00-8.00 sec 384 MBytes 3.22 Gbits/sec > [ 5] 8.00-9.00 sec 374 MBytes 3.14 Gbits/sec > [ 5] 9.00-10.00 sec 366 MBytes 3.07 Gbits/sec > [ 5] 10.00-10.13 sec 56.4 MBytes 3.52 Gbits/sec > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bandwidth > [ 5] 0.00-10.13 sec 0.00 Bytes 0.00 bits/sec sender > [ 5] 0.00-10.13 sec 3.66 GBytes 3.10 Gbits/sec > receiver > > Client, Freebsd 12: > 5] local 192.168.1.100 port 17917 connected to 192.168.1.155 port 5201 > [ ID] Interval Transfer Bitrate Retr Cwnd > [ 5] 0.00-1.00 sec 387 MBytes 3.25 Gbits/sec 83 309 KBytes > [ 5] 1.00-2.00 sec 380 MBytes 3.19 Gbits/sec 88 11.4 KBytes > [ 5] 2.00-3.00 sec 365 MBytes 3.06 Gbits/sec 83 54.2 KBytes > [ 5] 3.00-4.00 sec 372 MBytes 3.12 Gbits/sec 86 322 KBytes > [ 5] 4.00-5.00 sec 370 MBytes 3.11 Gbits/sec 83 2.85 KBytes > [ 5] 5.00-6.00 sec 366 MBytes 3.07 Gbits/sec 84 298 KBytes > [ 5] 6.00-7.00 sec 382 MBytes 3.20 Gbits/sec 87 356 KBytes > [ 5] 7.00-8.00 sec 380 MBytes 3.19 Gbits/sec 87 369 KBytes > [ 5] 8.00-9.00 sec 374 MBytes 3.14 Gbits/sec 87 29.9 KBytes > [ 5] 9.00-10.00 sec 375 MBytes 3.15 Gbits/sec 86 419 KBytes > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bitrate Retr > [ 5] 0.00-10.00 sec 3.66 GBytes 3.15 Gbits/sec 854 sender > [ 5] 0.00-10.00 sec 3.66 GBytes 3.15 Gbits/sec > receiver > > > ========================== > test 4, Centos 7.6 -> Centos 7.6 > ========================== > > Server: > Accepted connection from 192.168.1.162, port 47606 > [ 5] local 192.168.1.155 port 5201 connected to 192.168.1.162 port 47608 > [ ID] Interval Transfer Bandwidth > [ 5] 0.00-1.00 sec 1.60 GBytes 13.7 Gbits/sec > [ 5] 1.00-2.00 sec 1.59 GBytes 13.7 Gbits/sec > [ 5] 2.00-3.00 sec 1.60 GBytes 13.8 Gbits/sec > [ 5] 3.00-4.00 sec 1.71 GBytes 14.7 Gbits/sec > [ 5] 4.00-5.00 sec 1.49 GBytes 12.8 Gbits/sec > [ 5] 5.00-6.00 sec 1.38 GBytes 11.9 Gbits/sec > [ 5] 6.00-7.00 sec 1.52 GBytes 13.1 Gbits/sec > [ 5] 7.00-8.00 sec 1.51 GBytes 13.0 Gbits/sec > [ 5] 8.00-9.00 sec 1.51 GBytes 13.0 Gbits/sec > [ 5] 9.00-10.00 sec 1.51 GBytes 12.9 Gbits/sec > [ 5] 10.00-10.03 sec 48.1 MBytes 11.9 Gbits/sec > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bandwidth > [ 5] 0.00-10.03 sec 0.00 Bytes 0.00 bits/sec sender > [ 5] 0.00-10.03 sec 15.5 GBytes 13.2 Gbits/sec > receiver > > Client, Centos 7.6 > Connecting to host 192.168.1.155, port 5201 > [ 4] local 192.168.1.162 port 47608 connected to 192.168.1.155 port 5201 > [ ID] Interval Transfer Bandwidth Retr Cwnd > [ 4] 0.00-1.00 sec 1.66 GBytes 14.3 Gbits/sec 0 1.41 MBytes > [ 4] 1.00-2.00 sec 1.58 GBytes 13.5 Gbits/sec 0 1.46 MBytes > [ 4] 2.00-3.00 sec 1.61 GBytes 13.9 Gbits/sec 0 1.91 MBytes > [ 4] 3.00-4.00 sec 1.71 GBytes 14.7 Gbits/sec 0 2.13 MBytes > [ 4] 4.00-5.00 sec 1.47 GBytes 12.6 Gbits/sec 0 2.49 MBytes > [ 4] 5.00-6.00 sec 1.39 GBytes 11.9 Gbits/sec 0 2.56 MBytes > [ 4] 6.00-7.00 sec 1.53 GBytes 13.1 Gbits/sec 0 2.71 MBytes > [ 4] 7.00-8.00 sec 1.50 GBytes 12.9 Gbits/sec 0 2.73 MBytes > [ 4] 8.00-9.00 sec 1.50 GBytes 12.9 Gbits/sec 0 2.81 MBytes > [ 4] 9.00-10.00 sec 1.51 GBytes 12.9 Gbits/sec 0 2.84 MBytes > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bandwidth Retr > [ 4] 0.00-10.00 sec 15.5 GBytes 13.3 Gbits/sec 0 sender > [ 4] 0.00-10.00 sec 15.5 GBytes 13.3 Gbits/sec > receiver > > > =============================== > test4, Opensuse 15.1 -> OpenSuse 15.1 > ================================ > > Server, Opensuse 15.1: > [ 5] local 192.168.1.158 port 5201 connected to 192.168.1.163 port 39334 > [ ID] Interval Transfer Bitrate > [ 5] 0.00-1.00 sec 1.51 GBytes 13.0 Gbits/sec > [ 5] 1.00-2.00 sec 1.61 GBytes 13.8 Gbits/sec > [ 5] 2.00-3.00 sec 1.58 GBytes 13.6 Gbits/sec > [ 5] 3.00-4.00 sec 1.58 GBytes 13.6 Gbits/sec > [ 5] 4.00-5.00 sec 1.56 GBytes 13.4 Gbits/sec > [ 5] 5.00-6.00 sec 1.60 GBytes 13.7 Gbits/sec > [ 5] 6.00-7.00 sec 1.58 GBytes 13.5 Gbits/sec > [ 5] 7.00-8.00 sec 1.60 GBytes 13.8 Gbits/sec > [ 5] 8.00-9.00 sec 1.60 GBytes 13.7 Gbits/sec > [ 5] 9.00-10.00 sec 1.61 GBytes 13.8 Gbits/sec > [ 5] 10.00-10.04 sec 68.0 MBytes 14.2 Gbits/sec > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bitrate > [ 5] 0.00-10.04 sec 15.9 GBytes 13.6 Gbits/sec > receiver > > Client, OpenSuse 15.1: > [ 5] local 192.168.1.163 port 39334 connected to 192.168.1.158 port 5201 > [ ID] Interval Transfer Bitrate Retr Cwnd > [ 5] 0.00-1.00 sec 1.58 GBytes 13.6 Gbits/sec 0 723 KBytes > [ 5] 1.00-2.00 sec 1.61 GBytes 13.8 Gbits/sec 0 805 KBytes > [ 5] 2.00-3.00 sec 1.57 GBytes 13.5 Gbits/sec 0 926 KBytes > [ 5] 3.00-4.00 sec 1.58 GBytes 13.6 Gbits/sec 0 926 KBytes > [ 5] 4.00-5.00 sec 1.56 GBytes 13.4 Gbits/sec 0 970 KBytes > [ 5] 5.00-6.00 sec 1.60 GBytes 13.7 Gbits/sec 0 1.05 MBytes > [ 5] 6.00-7.00 sec 1.57 GBytes 13.5 Gbits/sec 0 1.05 MBytes > [ 5] 7.00-8.00 sec 1.61 GBytes 13.8 Gbits/sec 0 1.05 MBytes > [ 5] 8.00-9.00 sec 1.59 GBytes 13.7 Gbits/sec 0 1.05 MBytes > [ 5] 9.00-10.00 sec 1.62 GBytes 13.9 Gbits/sec 0 1.05 MBytes > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bitrate Retr > [ 5] 0.00-10.00 sec 15.9 GBytes 13.7 Gbits/sec 0 sender > [ 5] 0.00-10.04 sec 15.9 GBytes 13.6 Gbits/sec > receiver > > > > > > > [hidden email] mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "[hidden email]" |
Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer
|
|
On Mon, Jun 24, 2019 at 10:24:09PM +0200, Christian M wrote:
> Thanks for your input Bjarne. > > Previously I tested with iperf2, but I have made som tests with iperf3 now > also and noticed there are a lot of "Retr" (TCP retries) in some cases. > Went back here to your post and saw that your results also showed a lot of > retries in some cases. My new tests showed similar results to yours > (although not nearly as hi throughput as you have). When Linux is client > the retries are 0, with FreeBSD as client the number is > 0. Can you figure out what caused those retries? Wrong MTU, bad checksums? > I'm not sure > if our numbers are considered high though, and something that actually is a > problem? Although, on a internal network with no external factors like > interference, I feel this should always be 0 no matter what? If properly configured yes, there should be no retries when doing intra-VM connections. Thanks, Roger. _______________________________________________ [hidden email] mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "[hidden email]" |
Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer
|
|
I've made two tests while running tcpdump on the xcp-ng host. I'm not at
all qualified to interpret the .pcap files from tcpdump, but I've put them on Google Drive and linked them below the two tests. Perhaps someone more qualified could have a look for anything useful in there. Please note the extremely uneven throughput for test 2 below. It's almost like the throughput increased when running tcpdump simultaneously. Host: XCP-ng 7.6.0 Network: Private Network on host, not connected to any PIF. VM1: 12.0-RELEASE (1 VIF, 172.31.16.125) VM2: 12.0-RELEASE (1 VIF, 172.31.15.126) On the host I listen with tcpdump on the VIF for VM1 in both tests. VM1 as client: On XCP-ng: tcpdump -i vif42.0 -s 0 -w xcp-ng-vm1-client.pcap xcp-ng-vm1-client.pcap (80M): https://drive.google.com/open?id=1eR3fetvKRz3vFSXCxDKuJYFrQ3wLqjrU On VM1: iperf3 -c 172.31.16.126 On VM2: iperf3 -s VM1 iperf3 output: Connecting to host 172.31.16.126, port 5201 [ 5] local 172.31.16.125 port 18182 connected to 172.31.16.126 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 8.00 MBytes 67.1 Mbits/sec 156 15.6 KBytes [ 5] 1.00-2.00 sec 14.1 MBytes 118 Mbits/sec 318 1.43 KBytes [ 5] 2.00-3.00 sec 7.51 MBytes 63.1 Mbits/sec 218 1.43 KBytes [ 5] 3.00-4.00 sec 8.29 MBytes 69.3 Mbits/sec 193 1.43 KBytes [ 5] 4.00-5.01 sec 10.7 MBytes 89.4 Mbits/sec 252 1.43 KBytes [ 5] 5.01-6.00 sec 13.6 MBytes 115 Mbits/sec 313 31.3 KBytes [ 5] 6.00-7.00 sec 8.41 MBytes 70.2 Mbits/sec 309 1.43 KBytes [ 5] 7.00-8.01 sec 12.6 MBytes 106 Mbits/sec 223 1.43 KBytes [ 5] 8.01-9.00 sec 12.5 MBytes 106 Mbits/sec 227 1.43 KBytes [ 5] 9.00-10.00 sec 13.5 MBytes 113 Mbits/sec 263 1.43 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 109 MBytes 91.6 Mbits/sec 2472 sender [ 5] 0.00-10.01 sec 109 MBytes 91.3 Mbits/sec receiver VM1 as server: On XCP-ng: tcpdump -i vif42.0 -s 0 -w xcp-ng-vm1-server.pcap xcp-ng-vm1-server.pcap: (337M): https://drive.google.com/open?id=1N3-GccXaBc6hlzFrgshi2vhcCt8vekxF On VM1: iperf3 -s On VM2: iperf3 -c 172.31.16.125 VM1 iperf3 output: ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- Accepted connection from 172.31.16.126, port 31523 [ 5] local 172.31.16.125 port 5201 connected to 172.31.16.126 port 34605 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 10.9 MBytes 91.6 Mbits/sec [ 5] 1.00-2.00 sec 14.1 MBytes 118 Mbits/sec [ 5] 2.00-3.01 sec 4.92 MBytes 41.0 Mbits/sec [ 5] 3.01-4.00 sec 10.4 MBytes 87.9 Mbits/sec [ 5] 4.00-5.00 sec 8.52 MBytes 71.2 Mbits/sec [ 5] 5.00-6.00 sec 22.0 MBytes 185 Mbits/sec [ 5] 6.00-7.00 sec 9.25 MBytes 77.8 Mbits/sec [ 5] 7.00-8.00 sec 70.8 MBytes 594 Mbits/sec [ 5] 8.00-9.00 sec 89.5 MBytes 752 Mbits/sec [ 5] 9.00-10.00 sec 105 MBytes 884 Mbits/sec [ 5] 10.00-10.00 sec 180 KBytes 1.19 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.00 sec 346 MBytes 290 Mbits/sec receiver Den tis 25 juni 2019 kl 10:21 skrev Roger Pau Monné <[hidden email]>: > On Mon, Jun 24, 2019 at 10:24:09PM +0200, Christian M wrote: > > Thanks for your input Bjarne. > > > > Previously I tested with iperf2, but I have made som tests with iperf3 > now > > also and noticed there are a lot of "Retr" (TCP retries) in some cases. > > Went back here to your post and saw that your results also showed a lot > of > > retries in some cases. My new tests showed similar results to yours > > (although not nearly as hi throughput as you have). When Linux is client > > the retries are 0, with FreeBSD as client the number is > 0. > > Can you figure out what caused those retries? Wrong MTU, bad > checksums? > > > I'm not sure > > if our numbers are considered high though, and something that actually > is a > > problem? Although, on a internal network with no external factors like > > interference, I feel this should always be 0 no matter what? > > If properly configured yes, there should be no retries when doing > intra-VM connections. > > Thanks, Roger. > [hidden email] mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "[hidden email]" |
Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer
|
|
25.06.2019 18:55, Christian M wrote:
> Please note the > extremely uneven throughput for test 2 below. It's almost like the > throughput increased when running tcpdump simultaneously. Generally this indicates broken "hardware" offload for some features like txcsum/rxcsum/vlan_hwcsum/vlan_hwtagging etc. Some of them may be automatically disabled while interface is put in promiscuous mode by running tcpdump and enabled back after tcpdump terminated. You should look at "options" line in the output of ifconfig for the interface and manually disable offloads (look for ifconfig(8) manual for details). They retry tests without tcpdump running. _______________________________________________ [hidden email] mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "[hidden email]" |
Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer
|
|
I've disabled them all on both VM's with FreeBSD 12.0-RELEASE (ifconfig xn0
-txcsum -rxcsum -lro -tso), and also tried disabling everything for the VIF via XCP-ng earlier. It made no difference unfortunately. Den tis 25 juni 2019 kl 14:08 skrev Eugene Grosbein <[hidden email]>: > 25.06.2019 18:55, Christian M wrote: > > > Please note the > > extremely uneven throughput for test 2 below. It's almost like the > > throughput increased when running tcpdump simultaneously. > > Generally this indicates broken "hardware" offload for some features > like txcsum/rxcsum/vlan_hwcsum/vlan_hwtagging etc. > > Some of them may be automatically disabled while interface is put in > promiscuous mode > by running tcpdump and enabled back after tcpdump terminated. > > You should look at "options" line in the output of ifconfig for the > interface > and manually disable offloads (look for ifconfig(8) manual for details). > They retry tests without tcpdump running. > > [hidden email] mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "[hidden email]" |
Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer
|
|
In reply to this post by Christian Marcos
On Tue, Jun 25, 2019 at 01:55:40PM +0200, Christian M wrote:
> I've made two tests while running tcpdump on the xcp-ng host. I'm not at > all qualified to interpret the .pcap files from tcpdump, but I've put them > on Google Drive and linked them below the two tests. Perhaps someone more > qualified could have a look for anything useful in there. Please note the > extremely uneven throughput for test 2 below. It's almost like the > throughput increased when running tcpdump simultaneously. > > Host: XCP-ng 7.6.0 > Network: Private Network on host, not connected to any PIF. > VM1: 12.0-RELEASE (1 VIF, 172.31.16.125) > VM2: 12.0-RELEASE (1 VIF, 172.31.15.126) > > On the host I listen with tcpdump on the VIF for VM1 in both tests. > > VM1 as client: > > On XCP-ng: tcpdump -i vif42.0 -s 0 -w xcp-ng-vm1-client.pcap Can you check the capabilities of vif42.0? (ie: whether csum offloading is actually disabled on the host?) > xcp-ng-vm1-client.pcap (80M): > https://drive.google.com/open?id=1eR3fetvKRz3vFSXCxDKuJYFrQ3wLqjrU > On VM1: iperf3 -c 172.31.16.126 > On VM2: iperf3 -s I've taken a look at the dump and the checksum is wrong (or maybe missing) for all? packets. Packets with source 172.31.16.125 all have the TCP checksum set to 0x7f80 and all packets with source 172.31.16.125 have the TCP checksum set to 0x7960. FTR, can you also paste the ifconfig output of each interface before running iperf? Thanks, Roger. _______________________________________________ [hidden email] mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "[hidden email]" |
Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer
|
|
There's a mistake in my reply below.
On Tue, Jun 25, 2019 at 04:56:43PM +0200, Roger Pau Monné wrote: > On Tue, Jun 25, 2019 at 01:55:40PM +0200, Christian M wrote: > > I've made two tests while running tcpdump on the xcp-ng host. I'm not at > > all qualified to interpret the .pcap files from tcpdump, but I've put them > > on Google Drive and linked them below the two tests. Perhaps someone more > > qualified could have a look for anything useful in there. Please note the > > extremely uneven throughput for test 2 below. It's almost like the > > throughput increased when running tcpdump simultaneously. > > > > Host: XCP-ng 7.6.0 > > Network: Private Network on host, not connected to any PIF. > > VM1: 12.0-RELEASE (1 VIF, 172.31.16.125) > > VM2: 12.0-RELEASE (1 VIF, 172.31.15.126) > > > > On the host I listen with tcpdump on the VIF for VM1 in both tests. > > > > VM1 as client: > > > > On XCP-ng: tcpdump -i vif42.0 -s 0 -w xcp-ng-vm1-client.pcap > > Can you check the capabilities of vif42.0? (ie: whether csum > offloading is actually disabled on the host?) > > > xcp-ng-vm1-client.pcap (80M): > > https://drive.google.com/open?id=1eR3fetvKRz3vFSXCxDKuJYFrQ3wLqjrU > > On VM1: iperf3 -c 172.31.16.126 > > On VM2: iperf3 -s > > I've taken a look at the dump and the checksum is wrong (or maybe > missing) for all? packets. > > Packets with source 172.31.16.125 all have the TCP checksum set to > 0x7f80 and all packets with source 172.31.16.125 have the TCP checksum _______________________________________________ [hidden email] mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "[hidden email]" |
Re: Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer
|
|
In reply to this post by Christian Marcos
oh yes, sorry I am using:
0:~ # xl info host : x0 release : 4.12.14-lp150.12.64-default version : #1 SMP Mon Jun 17 16:53:50 UTC 2019 (3edfd41) machine : x86_64 nr_cpus : 4 max_cpu_id : 3 nr_nodes : 1 cores_per_socket : 4 threads_per_core : 1 cpu_mhz : 3192 hw_caps : bfebfbff:77faf3bf:2c100800:00000021:00000001:000027ab:00000000:00000100 virt_caps : hvm hvm_directio total_memory : 32461 free_memory : 5319 sharing_freed_memory : 0 sharing_used_memory : 0 outstanding_claims : 0 free_cpus : 0 xen_major : 4 xen_minor : 10 xen_extra : .3_04-lp150.2.1 xen_version : 4.10.3_04-lp150.2.1 xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 xen_scheduler : credit xen_pagesize : 4096 platform_params : virt_start=0xffff800000000000 xen_changeset : xen_commandline : dom0_mem=3072M,max:3072M cc_compiler : gcc (SUSE Linux) 7.4.0 cc_compile_by : abuild cc_compile_domain : suse.de cc_compile_date : Thu May 9 16:10:08 UTC 2019 build_id : bf77d035ff05eb55d9616e418f73cbcbd96f747b xend_config_format : 4 Since you peaked my interest, I tryed playing around a little. Initially to day, the retransmits where between 4-15 retransmits. I then changed the network interface from hypervisor default to E1000 (still xn0 in freebsd) and the retransmits jumped up to 700 to 800! After changing the network interface back to hypervisor default, the number of retranmits remains high in the 3-digit area. Fooling around with mtu size did not change anything. Something is not quite right. On 6/24/19 10:24 PM, Christian M wrote: > Thanks for your input Bjarne. > > Previously I tested with iperf2, but I have made som tests with iperf3 now > also and noticed there are a lot of "Retr" (TCP retries) in some cases. > Went back here to your post and saw that your results also showed a lot of > retries in some cases. My new tests showed similar results to yours > (although not nearly as hi throughput as you have). When Linux is client > the retries are 0, with FreeBSD as client the number is > 0. I'm not sure > if our numbers are considered high though, and something that actually is a > problem? Although, on a internal network with no external factors like > interference, I feel this should always be 0 no matter what? > > What hypervisor were you running in your tests? Version? > > Regards, > Christian > > > Den mån 24 juni 2019 kl 17:56 skrev Bjarne <[hidden email]>: > >> Well, to add some datapoints, I just did a quick test om my private >> homeserver, testing with iperf3 >> -s / iperf3 -c <server-ip> >> >> I have not much knowledge of iperf3 network test and it's validity, but >> there is definitely some >> differences to be seen. >> >> Summary: >> >> FBSD <-> FBSD 3,5Gbit/sec >> FBSD -> Centos: 3GBit/Sec >> Centos -> FBSD: 12 Gbit/sec >> :snip >> >> >> -- mvh, Bjarne _______________________________________________ [hidden email] mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "[hidden email]" |
Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer
|
|
In reply to this post by Christian Marcos
25.06.2019 19:40, Christian M wrote:
> I've disabled them all on both VM's with FreeBSD 12.0-RELEASE (ifconfig xn0 > -txcsum -rxcsum -lro -tso), and also tried disabling everything for the VIF > via XCP-ng earlier. It made no difference unfortunately. Then you should try to bring the interface to promiscuous mode manually with "ifconfig xn0 promisc" any retry. If this does not make a difference, then you may be facing some timekeeping problem as tcpdump makes additional CPU load on the system that can force scheduler behaviour to change. As next step, you should check your kern.timecounter.* and kern.eventtimer.* sysctls. If default value of kern.eventtimer.periodic is 0 (it depends on detected "hardware"), you should change it kern.eventtimer.periodic=1 and retry the test. _______________________________________________ [hidden email] mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "[hidden email]" |
Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer
|
|
checksum offloading doe not seem to have any affect when done on the VM
VIF's. Both 12.0-RELEASE VIF's now have: other-config (MRW): ethtool-rx: off; ethtool-tx: off; ethtool-sg: off; ethtool-tso: off; ethtool-ufo: off; ethtool-gso: off ethtool -k vif54.0 Features for vif54.0: rx-checksumming: on [fixed] tx-checksumming: off tx-checksum-ipv4: off tx-checksum-ip-generic: off [fixed] tx-checksum-ipv6: off tx-checksum-fcoe-crc: off [fixed] tx-checksum-sctp: off [fixed] scatter-gather: off tx-scatter-gather: off tx-scatter-gather-fraglist: off tcp-segmentation-offload: off tx-tcp-segmentation: off tx-tcp-ecn-segmentation: off [fixed] tx-tcp6-segmentation: off udp-fragmentation-offload: off [fixed] generic-segmentation-offload: off generic-receive-offload: on large-receive-offload: off [fixed] rx-vlan-offload: off [fixed] tx-vlan-offload: off [fixed] ntuple-filters: off [fixed] receive-hashing: off [fixed] highdma: off [fixed] rx-vlan-filter: off [fixed] vlan-challenged: off [fixed] tx-lockless: off [fixed] netns-local: off [fixed] tx-gso-robust: off [fixed] tx-fcoe-segmentation: off [fixed] tx-gre-segmentation: off [fixed] tx-ipip-segmentation: off [fixed] tx-sit-segmentation: off [fixed] tx-udp_tnl-segmentation: off [fixed] fcoe-mtu: off [fixed] tx-nocache-copy: off loopback: off [fixed] rx-fcs: off [fixed] rx-all: off [fixed] tx-vlan-stag-hw-insert: off [fixed] rx-vlan-stag-hw-parse: off [fixed] rx-vlan-stag-filter: off [fixed] l2-fwd-offload: off [fixed] busy-poll: off [fixed] Both vif's look the same as above. Running tcpdump on the host while running iperf3 between the 12.0 VM's results in a lot of incorrect cksum like this. tcpdump -i vif54.0 -v -nn| grep -i incorrect 172.31.16.125.63013 > 172.31.16.126.5201: Flags [.], cksum 0x7f08 (incorrect -> 0x030f), seq 223153882:223155330, ack 1, win 1026, options [nop,nop,TS val 4104002274 ecr 3926764642], length 1448 ... Also noticed that iperf3 does not work to run every time, more than 50% of the tries with iperf3 -c <ip> I get: Connecting to host 172.31.16.126, port 5201 iperf3: error - unable to initialize stream: Socket is not connected Or nothing happens and I need to abort, then restart iperf3 -s on the other VM, and retry. The results from iperf3 are really inconsistent still, and I cant get a good feel of any pattern. Currently as I type this, a result looks like this: Host: tcpdump running VIF's: no promisc xn0 on both VM's: no promisc [ 5] local 172.31.16.125 port 47738 connected to 172.31.16.126 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 37.8 MBytes 316 Mbits/sec 176 1.43 KBytes [ 5] 1.00-2.00 sec 133 MBytes 1.11 Gbits/sec 1022 49.8 KBytes [ 5] 2.00-3.00 sec 179 MBytes 1.50 Gbits/sec 784 191 KBytes [ 5] 3.00-4.00 sec 132 MBytes 1.11 Gbits/sec 201 345 KBytes [ 5] 4.00-5.00 sec 95.3 MBytes 799 Mbits/sec 438 180 KBytes [ 5] 5.00-6.00 sec 126 MBytes 1.06 Gbits/sec 1256 353 KBytes [ 5] 6.00-7.00 sec 144 MBytes 1.20 Gbits/sec 1242 275 KBytes [ 5] 7.00-8.00 sec 94.5 MBytes 792 Mbits/sec 593 167 KBytes [ 5] 8.00-9.00 sec 125 MBytes 1.05 Gbits/sec 326 169 KBytes [ 5] 9.00-10.00 sec 57.0 MBytes 478 Mbits/sec 1013 1.43 KBytes [ 5] 10.00-11.00 sec 31.3 MBytes 262 Mbits/sec 463 1.43 KBytes [ 5] 11.00-12.00 sec 42.3 MBytes 355 Mbits/sec 544 78.3 KBytes [ 5] 12.00-13.00 sec 41.3 MBytes 346 Mbits/sec 624 1.43 KBytes [ 5] 13.00-14.00 sec 52.7 MBytes 442 Mbits/sec 739 1.43 KBytes [ 5] 14.00-15.00 sec 20.4 MBytes 171 Mbits/sec 384 1.43 KBytes [ 5] 15.00-16.00 sec 74.5 MBytes 625 Mbits/sec 489 299 KBytes [ 5] 16.00-17.00 sec 40.1 MBytes 337 Mbits/sec 835 1.43 KBytes [ 5] 17.00-18.00 sec 43.2 MBytes 363 Mbits/sec 372 1.43 KBytes [ 5] 18.00-19.00 sec 58.9 MBytes 494 Mbits/sec 723 1.43 KBytes [ 5] 19.00-20.00 sec 21.2 MBytes 178 Mbits/sec 300 1.43 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-20.00 sec 1.51 GBytes 649 Mbits/sec 12524 sender [ 5] 0.00-20.00 sec 1.51 GBytes 649 Mbits/sec receiver But please be sure, this can change dramatically without making any changes what so ever. Offloading is not turned off in FreeBSD (but on the VIF's as stated before), and tcpdump is running on the host. xn0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=503<RXCSUM,TXCSUM,TSO4,LRO> ether 12:ec:6a:5c:af:df inet 172.31.16.125 netmask 0xffffff00 broadcast 172.31.16.255 media: Ethernet manual status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> xn0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=503<RXCSUM,TXCSUM,TSO4,LRO> ether aa:4f:f7:4c:a0:e6 inet 172.31.16.126 netmask 0xffffff00 broadcast 172.31.16.255 media: Ethernet manual status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> Absolutely nothing changed this time (restarted iperf3 -s though since I ran ifconfig do get the info above), tcpdump still running on host (never even aborted): Host: tcpdump still running VIF's: no promisc xn0 on both VM's: no promisc [ 5] local 172.31.16.125 port 30543 connected to 172.31.16.126 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 10.3 MBytes 86.8 Mbits/sec 165 38.4 KBytes [ 5] 1.00-2.00 sec 39.7 MBytes 333 Mbits/sec 175 167 KBytes [ 5] 2.00-3.00 sec 40.0 MBytes 335 Mbits/sec 437 54.1 KBytes [ 5] 3.00-4.00 sec 11.2 MBytes 93.6 Mbits/sec 320 1.43 KBytes [ 5] 4.00-5.00 sec 11.2 MBytes 94.2 Mbits/sec 305 1.43 KBytes [ 5] 5.00-6.00 sec 7.63 MBytes 64.1 Mbits/sec 179 2.84 KBytes [ 5] 6.00-7.00 sec 6.77 MBytes 56.7 Mbits/sec 191 1.43 KBytes [ 5] 7.00-8.00 sec 4.26 MBytes 35.9 Mbits/sec 145 21.3 KBytes [ 5] 8.00-9.00 sec 5.38 MBytes 45.1 Mbits/sec 176 1.43 KBytes [ 5] 9.00-10.00 sec 8.24 MBytes 69.1 Mbits/sec 225 1.43 KBytes [ 5] 10.00-11.00 sec 16.1 MBytes 135 Mbits/sec 422 1.43 KBytes [ 5] 11.00-12.00 sec 3.76 MBytes 31.5 Mbits/sec 113 1.43 KBytes [ 5] 12.00-13.00 sec 6.62 MBytes 55.5 Mbits/sec 161 1.43 KBytes [ 5] 13.00-14.00 sec 5.28 MBytes 44.3 Mbits/sec 156 18.4 KBytes [ 5] 14.00-15.00 sec 5.21 MBytes 43.7 Mbits/sec 146 1.43 KBytes [ 5] 15.00-16.00 sec 6.37 MBytes 53.4 Mbits/sec 199 1.43 KBytes [ 5] 16.00-17.00 sec 10.7 MBytes 89.5 Mbits/sec 193 1.43 KBytes [ 5] 17.00-18.00 sec 9.49 MBytes 79.6 Mbits/sec 390 1.43 KBytes [ 5] 18.00-19.00 sec 5.36 MBytes 44.9 Mbits/sec 125 1.43 KBytes [ 5] 19.00-20.00 sec 3.65 MBytes 30.6 Mbits/sec 58 1.43 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-20.00 sec 217 MBytes 91.1 Mbits/sec 4281 sender [ 5] 0.00-20.00 sec 217 MBytes 90.8 Mbits/sec receiver Host: tcpdump still running VIF's: promisc xn0 on both VM's: promisc Connecting to host 172.31.16.126, port 5201 [ 5] local 172.31.16.125 port 61196 connected to 172.31.16.126 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 25.9 MBytes 217 Mbits/sec 271 90.5 KBytes [ 5] 1.00-2.00 sec 56.4 MBytes 473 Mbits/sec 94 255 KBytes [ 5] 2.00-3.00 sec 83.4 MBytes 699 Mbits/sec 872 201 KBytes [ 5] 3.00-4.00 sec 76.4 MBytes 641 Mbits/sec 684 176 KBytes [ 5] 4.00-5.00 sec 80.4 MBytes 674 Mbits/sec 742 1.43 KBytes [ 5] 5.00-6.00 sec 81.4 MBytes 684 Mbits/sec 248 333 KBytes [ 5] 6.00-7.00 sec 86.1 MBytes 722 Mbits/sec 826 160 KBytes [ 5] 7.00-8.00 sec 77.5 MBytes 650 Mbits/sec 209 253 KBytes [ 5] 8.00-9.00 sec 9.64 MBytes 80.9 Mbits/sec 433 12.8 KBytes [ 5] 9.00-10.00 sec 8.31 MBytes 69.6 Mbits/sec 210 1.43 KBytes [ 5] 10.00-11.00 sec 8.64 MBytes 72.5 Mbits/sec 199 1.43 KBytes [ 5] 11.00-12.00 sec 7.02 MBytes 58.9 Mbits/sec 240 1.43 KBytes [ 5] 12.00-13.00 sec 20.1 MBytes 169 Mbits/sec 579 1.43 KBytes [ 5] 13.00-14.00 sec 10.8 MBytes 90.4 Mbits/sec 277 22.7 KBytes [ 5] 14.00-15.00 sec 2.75 MBytes 23.1 Mbits/sec 107 1.43 KBytes [ 5] 15.00-16.00 sec 8.79 MBytes 73.8 Mbits/sec 203 12.8 KBytes [ 5] 16.00-17.00 sec 9.23 MBytes 77.3 Mbits/sec 158 1.43 KBytes [ 5] 17.00-18.00 sec 13.5 MBytes 114 Mbits/sec 381 1.43 KBytes [ 5] 18.00-19.00 sec 41.6 MBytes 349 Mbits/sec 331 99.5 KBytes [ 5] 19.00-20.00 sec 40.6 MBytes 340 Mbits/sec 70 1.43 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-20.00 sec 749 MBytes 314 Mbits/sec 7134 sender [ 5] 0.00-20.00 sec 748 MBytes 314 Mbits/sec receiver Host: tcpdump NOT running VIF's: no promisc xn0: no promisc Run1: Connecting to host 172.31.16.126, port 5201 [ 5] local 172.31.16.125 port 20600 connected to 172.31.16.126 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 28.4 MBytes 238 Mbits/sec 451 1.43 KBytes [ 5] 1.00-2.00 sec 21.5 MBytes 181 Mbits/sec 546 2.84 KBytes [ 5] 2.00-3.00 sec 26.7 MBytes 224 Mbits/sec 595 1.43 KBytes [ 5] 3.00-4.00 sec 15.9 MBytes 133 Mbits/sec 347 1.43 KBytes [ 5] 4.00-5.00 sec 4.45 MBytes 37.4 Mbits/sec 187 2.84 KBytes [ 5] 5.00-6.00 sec 15.4 MBytes 129 Mbits/sec 239 66.5 KBytes [ 5] 6.00-7.00 sec 10.6 MBytes 89.1 Mbits/sec 366 1.43 KBytes [ 5] 7.00-8.00 sec 30.8 MBytes 258 Mbits/sec 510 1.43 KBytes [ 5] 8.00-9.00 sec 16.6 MBytes 140 Mbits/sec 332 2.84 KBytes [ 5] 9.00-10.00 sec 12.4 MBytes 104 Mbits/sec 376 1.43 KBytes [ 5] 10.00-11.00 sec 11.3 MBytes 94.9 Mbits/sec 223 1.43 KBytes [ 5] 11.00-12.00 sec 9.00 MBytes 75.5 Mbits/sec 206 1.43 KBytes [ 5] 12.00-13.00 sec 8.95 MBytes 75.1 Mbits/sec 201 1.43 KBytes [ 5] 13.00-14.00 sec 12.3 MBytes 103 Mbits/sec 179 1.43 KBytes [ 5] 14.00-15.00 sec 20.5 MBytes 172 Mbits/sec 445 1.43 KBytes [ 5] 15.00-16.00 sec 3.92 MBytes 32.9 Mbits/sec 163 1.43 KBytes [ 5] 16.00-17.00 sec 6.82 MBytes 57.2 Mbits/sec 122 2.84 KBytes [ 5] 17.00-18.00 sec 14.3 MBytes 120 Mbits/sec 238 1.43 KBytes [ 5] 18.00-19.00 sec 14.2 MBytes 119 Mbits/sec 226 1.43 KBytes [ 5] 19.00-20.00 sec 12.9 MBytes 108 Mbits/sec 297 1.43 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-20.00 sec 297 MBytes 125 Mbits/sec 6249 sender [ 5] 0.00-20.00 sec 297 MBytes 124 Mbits/sec receiver Run2 right after a iperf3 -s restart on second VM: Connecting to host 172.31.16.126, port 5201 [ 5] local 172.31.16.125 port 40622 connected to 172.31.16.126 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 6.49 MBytes 54.3 Mbits/sec 163 1.43 KBytes [ 5] 1.00-2.00 sec 9.93 MBytes 83.3 Mbits/sec 263 1.43 KBytes [ 5] 2.00-3.00 sec 7.71 MBytes 64.7 Mbits/sec 173 1.43 KBytes [ 5] 3.00-4.00 sec 5.58 MBytes 47.0 Mbits/sec 83 1.43 KBytes [ 5] 4.00-5.00 sec 14.8 MBytes 124 Mbits/sec 347 1.43 KBytes [ 5] 5.00-6.00 sec 16.8 MBytes 141 Mbits/sec 349 1.43 KBytes [ 5] 6.00-7.00 sec 19.2 MBytes 161 Mbits/sec 205 1.43 KBytes [ 5] 7.00-8.00 sec 6.39 MBytes 53.6 Mbits/sec 79 1.43 KBytes [ 5] 8.00-9.00 sec 7.19 MBytes 60.3 Mbits/sec 133 1.43 KBytes [ 5] 9.00-10.00 sec 18.2 MBytes 153 Mbits/sec 265 1.43 KBytes [ 5] 10.00-11.00 sec 11.0 MBytes 92.6 Mbits/sec 220 1.43 KBytes [ 5] 11.00-12.00 sec 7.93 MBytes 66.6 Mbits/sec 172 1.43 KBytes [ 5] 12.00-13.00 sec 10.1 MBytes 84.8 Mbits/sec 198 1.43 KBytes [ 5] 13.00-14.00 sec 7.92 MBytes 66.5 Mbits/sec 196 1.43 KBytes [ 5] 14.00-15.00 sec 7.63 MBytes 64.0 Mbits/sec 144 1.43 KBytes [ 5] 15.00-16.00 sec 8.70 MBytes 73.0 Mbits/sec 267 1.43 KBytes [ 5] 16.00-17.00 sec 16.0 MBytes 134 Mbits/sec 296 1.43 KBytes [ 5] 17.00-18.00 sec 10.9 MBytes 91.5 Mbits/sec 261 1.43 KBytes [ 5] 18.00-19.00 sec 23.0 MBytes 193 Mbits/sec 444 1.43 KBytes [ 5] 19.00-20.00 sec 8.86 MBytes 74.3 Mbits/sec 224 1.43 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-20.00 sec 224 MBytes 94.1 Mbits/sec 4482 sender [ 5] 0.00-20.00 sec 224 MBytes 93.9 Mbits/sec receiver As suggested before, I have set kern.eventtimer.periodic: 1, the above tests are done with that setting. I don't see any difference with it set or not (kern.eventtimer.periodic: 0 was default btw). There was also a suggestion to have a look at kern.timecouter.*, but I'm not sure what to look for here: kern.timecounter.tsc_shift: 1 kern.timecounter.smp_tsc_adjust: 0 kern.timecounter.smp_tsc: 0 kern.timecounter.invariant_tsc: 0 kern.timecounter.fast_gettime: 1 kern.timecounter.tick: 1 kern.timecounter.choice: XENTIMER(950) ACPI-fast(900) i8254(0) HPET(950) TSC-low(-100) dummy(-1000000) kern.timecounter.hardware: XENTIMER kern.timecounter.alloweddeviation: 5 kern.timecounter.stepwarnings: 0 kern.timecounter.tc.XENTIMER.quality: 950 kern.timecounter.tc.XENTIMER.frequency: 1000000000 kern.timecounter.tc.XENTIMER.counter: 3517394506 kern.timecounter.tc.XENTIMER.mask: 4294967295 kern.timecounter.tc.ACPI-fast.quality: 900 kern.timecounter.tc.ACPI-fast.frequency: 3579545 kern.timecounter.tc.ACPI-fast.counter: 1749326798 kern.timecounter.tc.ACPI-fast.mask: 4294967295 kern.timecounter.tc.i8254.quality: 0 kern.timecounter.tc.i8254.frequency: 1193182 kern.timecounter.tc.i8254.counter: 58432 kern.timecounter.tc.i8254.mask: 65535 kern.timecounter.tc.HPET.quality: 950 kern.timecounter.tc.HPET.frequency: 62500000 kern.timecounter.tc.HPET.counter: 3606880145 kern.timecounter.tc.HPET.mask: 4294967295 kern.timecounter.tc.TSC-low.quality: -100 kern.timecounter.tc.TSC-low.frequency: 1200075192 kern.timecounter.tc.TSC-low.counter: 3248698599 kern.timecounter.tc.TSC-low.mask: 4294967295 Regards, Christian Den ons 26 juni 2019 kl 01:14 skrev Eugene Grosbein <[hidden email]>: > 25.06.2019 19:40, Christian M wrote: > > > I've disabled them all on both VM's with FreeBSD 12.0-RELEASE (ifconfig > xn0 > > -txcsum -rxcsum -lro -tso), and also tried disabling everything for the > VIF > > via XCP-ng earlier. It made no difference unfortunately. > > Then you should try to bring the interface to promiscuous mode manually > with "ifconfig xn0 promisc" any retry. If this does not make a difference, > then you may be facing some timekeeping problem as tcpdump makes additional > CPU load on the system that can force scheduler behaviour to change. > > As next step, you should check your kern.timecounter.* and > kern.eventtimer.* sysctls. > If default value of kern.eventtimer.periodic is 0 (it depends on detected > "hardware"), > you should change it kern.eventtimer.periodic=1 and retry the test. > > [hidden email] mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "[hidden email]" |
Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer
|
|
26.06.2019 15:11, Christian M wrote:
> Running tcpdump on the host while running iperf3 between the 12.0 VM's > results in a lot of incorrect cksum like this. > > tcpdump -i vif54.0 -v -nn| grep -i incorrect > 172.31.16.125.63013 > 172.31.16.126.5201: Flags [.], cksum 0x7f08 > (incorrect -> 0x030f), seq 223153882:223155330, ack 1, win 1026, options > [nop,nop,TS val 4104002274 ecr 3926764642], length 1448 Incorrect checksum shown by tcpdump is normal and expected for outgoing packets if txcsum offload is not disabled because networking stack skips checksum calculation offloading it to "hardware". It still indicates a bug at sending side if seen for outgoing packets when txcsum is disabled. Incorrect checksum shown by tcpdump for incoming packets always indicates a problem at sending side or intermediate network (if any). You should check output of "netstat -sp ip" and "netstat -sp tcp" for non-zero and growing checksum error counters at receiving side. _______________________________________________ [hidden email] mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "[hidden email]" |
Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer
|
|
I ran ifconfig xn0 -txcsum on both test VM's, and all incorrect checksum
disappeared. netstat -sp ip for VM1 (172.31.16.125, running iperf3 -c always) and VM2 (172.31.16.126, running iperf3 -s always): ip: 3664084 total packets received 0 bad header checksums 0 with size smaller than minimum 0 with data size < data length 0 with ip length > max ip packet size 0 with header length < data size 0 with data length < header length 0 with bad options 0 with incorrect version number 0 fragments received 0 fragments dropped (dup or out of space) 0 fragments dropped after timeout 0 packets reassembled ok 3664080 packets for this host 0 packets for unknown/unsupported protocol 0 packets forwarded (0 packets fast forwarded) 4 packets not forwardable 0 packets received for unknown multicast group 0 redirects sent 7003245 packets sent from this host 0 packets sent with fabricated ip header 0 output packets dropped due to no bufs, etc. 0 output packets discarded due to no route 0 output datagrams fragmented 0 fragments created 0 datagrams that can't be fragmented 0 tunneling packets that can't find gif 0 datagrams with bad address in header ip: 8217827 total packets received 0 bad header checksums 0 with size smaller than minimum 0 with data size < data length 0 with ip length > max ip packet size 0 with header length < data size 0 with data length < header length 0 with bad options 0 with incorrect version number 0 fragments received 0 fragments dropped (dup or out of space) 0 fragments dropped after timeout 0 packets reassembled ok 8217827 packets for this host 0 packets for unknown/unsupported protocol 0 packets forwarded (0 packets fast forwarded) 0 packets not forwardable 0 packets received for unknown multicast group 0 redirects sent 4300027 packets sent from this host 0 packets sent with fabricated ip header 0 output packets dropped due to no bufs, etc. 0 output packets discarded due to no route 0 output datagrams fragmented 0 fragments created 0 datagrams that can't be fragmented 0 tunneling packets that can't find gif 0 datagrams with bad address in header netstat -sp tcp for VM1 (172.31.16.125, running iperf3 -c always) and VM2 (172.31.16.126, running iperf3 -s always): tcp: 7002523 packets sent 6866670 data packets (9939548357 bytes) 109393 data packets (101233068 bytes) retransmitted 0 data packets unnecessarily retransmitted 0 resends initiated by MTU discovery 27039 ack-only packets (90 delayed) 0 URG only packets 0 window probe packets 0 window update packets 125 control packets 3664099 packets received 3387212 acks (for 9939333715 bytes) 270936 duplicate acks 0 acks for unsent data 915 packets (36547 bytes) received in-sequence 4887 completely duplicate packets (208 bytes) 4853 old duplicate packets 0 packets with some dup. data (0 bytes duped) 0 out-of-order packets (0 bytes) 0 packets (0 bytes) of data after window 0 window probes 5694 window update packets 6 packets received after close 0 discarded for bad checksums 0 discarded for bad header offset fields 0 discarded because packet too short 0 discarded due to memory problems 88 connection requests 2 connection accepts 0 bad connection attempts 0 listen queue overflows 0 ignored RSTs in the windows 89 connections established (including accepts) 85 times used RTT from hostcache 85 times used RTT variance from hostcache 85 times used slow-start threshold from hostcache 90 connections closed (including 57 drops) 51 connections updated cached RTT on close 51 connections updated cached RTT variance on close 23 connections updated cached ssthresh on close 1 embryonic connection dropped 3387212 segments updated rtt (of 3313837 attempts) 0 retransmit timeouts 0 connections dropped by rexmit timeout 0 persist timeouts 0 connections dropped by persist timeout 0 Connections (fin_wait_2) dropped because of timeout 0 keepalive timeouts 0 keepalive probes sent 0 connections dropped by keepalive 759139 correct ACK header predictions 70 correct data packet header predictions 2 syncache entries added 0 retransmitted 0 dupsyn 0 dropped 2 completed 0 bucket overflow 0 cache overflow 0 reset 0 stale 0 aborted 0 badack 0 unreach 0 zone failures 2 cookies sent 0 cookies received 1 hostcache entry added 0 bucket overflow 5602 SACK recovery episodes 109329 segment rexmits in SACK recovery episodes 101663360 byte rexmits in SACK recovery episodes 249723 SACK options (SACK blocks) received 0 SACK options (SACK blocks) sent 0 SACK scoreboard overflow 0 packets with ECN CE bit set 0 packets with ECN ECT(0) bit set 0 packets with ECN ECT(1) bit set 0 successful ECN handshakes 0 times ECN reduced the congestion window 0 packets with matching signature received 0 packets with bad signature received 0 times failed to make signature due to no SA 0 times unexpected signature received 0 times no signature provided by segment 0 Path MTU discovery black hole detection activations 0 Path MTU discovery black hole detection min MSS activations 0 Path MTU discovery black hole detection failures TCP connection count by state: 0 connections in CLOSED state 3 connections in LISTEN state 0 connections in SYN_SENT state 0 connections in SYN_RCVD state 2 connections in ESTABLISHED state 0 connections in CLOSE_WAIT state 0 connections in FIN_WAIT_1 state 0 connections in CLOSING state 0 connections in LAST_ACK state 0 connections in FIN_WAIT_2 state 0 connections in TIME_WAIT state tcp: 4299563 packets sent 1510 data packets (132487 bytes) 0 data packets (0 bytes) retransmitted 0 data packets unnecessarily retransmitted 0 resends initiated by MTU discovery 4297568 ack-only packets (2610 delayed) 0 URG only packets 0 window probe packets 392 window update packets 93 control packets 8217839 packets received 1623 acks (for 132605 bytes) 150324 duplicate acks 0 acks for unsent data 7770336 packets (11249613126 bytes) received in-sequence 126189 completely duplicate packets (117973252 bytes) 4853 old duplicate packets 194 packets with some dup. data (59900 bytes duped) 290196 out-of-order packets (415026177 bytes) 0 packets (0 bytes) of data after window 0 window probes 0 window update packets 25 packets received after close 0 discarded for bad checksums 0 discarded for bad header offset fields 0 discarded because packet too short 0 discarded due to memory problems 6 connection requests 102 connection accepts 0 bad connection attempts 0 listen queue overflows 1 ignored RSTs in the window 108 connections established (including accepts) 95 times used RTT from hostcache 95 times used RTT variance from hostcache 0 times used slow-start threshold from hostcache 163 connections closed (including 47 drops) 29 connections updated cached RTT on close 29 connections updated cached RTT variance on close 0 connections updated cached ssthresh on close 0 embryonic connections dropped 1623 segments updated rtt (of 1314 attempts) 0 retransmit timeouts 0 connections dropped by rexmit timeout 0 persist timeouts 0 connections dropped by persist timeout 0 Connections (fin_wait_2) dropped because of timeout 9 keepalive timeouts 8 keepalive probes sent 1 connection dropped by keepalive 593 correct ACK header predictions 7769705 correct data packet header predictions 102 syncache entries added 0 retransmitted 0 dupsyn 0 dropped 102 completed 0 bucket overflow 0 cache overflow 0 reset 0 stale 0 aborted 0 badack 0 unreach 0 zone failures 102 cookies sent 0 cookies received 3 hostcache entries added 0 bucket overflow 0 SACK recovery episodes 0 segment rexmits in SACK recovery episodes 0 byte rexmits in SACK recovery episodes 0 SACK options (SACK blocks) received 293795 SACK options (SACK blocks) sent 0 SACK scoreboard overflow 0 packets with ECN CE bit set 0 packets with ECN ECT(0) bit set 0 packets with ECN ECT(1) bit set 0 successful ECN handshakes 0 times ECN reduced the congestion window 0 packets with matching signature received 0 packets with bad signature received 0 times failed to make signature due to no SA 0 times unexpected signature received 0 times no signature provided by segment 0 Path MTU discovery black hole detection activations 0 Path MTU discovery black hole detection min MSS activations 0 Path MTU discovery black hole detection failures TCP connection count by state: 0 connections in CLOSED state 4 connections in LISTEN state 0 connections in SYN_SENT state 0 connections in SYN_RCVD state 2 connections in ESTABLISHED state 0 connections in CLOSE_WAIT state 0 connections in FIN_WAIT_1 state 0 connections in CLOSING state 0 connections in LAST_ACK state 0 connections in FIN_WAIT_2 state 0 connections in TIME_WAIT state The data collected is after a number of runs with iperf3. I'm really not qualified to interpret any of this, so my hopes are that someone else sees something unusual here. Thanks, Christian Den ons 26 juni 2019 kl 10:37 skrev Eugene Grosbein <[hidden email]>: > 26.06.2019 15:11, Christian M wrote: > > > Running tcpdump on the host while running iperf3 between the 12.0 VM's > > results in a lot of incorrect cksum like this. > > > > tcpdump -i vif54.0 -v -nn| grep -i incorrect > > 172.31.16.125.63013 > 172.31.16.126.5201: Flags [.], cksum 0x7f08 > > (incorrect -> 0x030f), seq 223153882:223155330, ack 1, win 1026, options > > [nop,nop,TS val 4104002274 ecr 3926764642], length 1448 > > Incorrect checksum shown by tcpdump is normal and expected for outgoing > packets > if txcsum offload is not disabled because networking stack skips checksum > calculation > offloading it to "hardware". It still indicates a bug at sending side if > seen for outgoing packets > when txcsum is disabled. > > Incorrect checksum shown by tcpdump for incoming packets always indicates > a problem > at sending side or intermediate network (if any). > > You should check output of "netstat -sp ip" and "netstat -sp tcp" for > non-zero > and growing checksum error counters at receiving side. > > [hidden email] mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "[hidden email]" |
Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer
|
|
Sorry to say, but no. Nothing changed :(
The iperf issue concerns me a bit also: root@:/usr/home/asdf # iperf3 -c 172.31.16.126 iperf3: error - unable to receive control message: Connection reset by peer root@:/usr/home/asdf # iperf3 -c 172.31.16.126 Connecting to host 172.31.16.126, port 5201 iperf3: error - unable to initialize stream: Socket is not connected root@:/usr/home/asdf # iperf3 -c 172.31.16.126 ^C- - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr iperf3: interrupt - the client has terminated root@:/usr/home/asdf # iperf3 -c 172.31.16.126 Connecting to host 172.31.16.126, port 5201 [ 5] local 172.31.16.125 port 23884 connected to 172.31.16.126 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 6.00 MBytes 50.2 Mbits/sec 217 1.43 KBytes [ 5] 1.00-2.00 sec 26.7 MBytes 224 Mbits/sec 566 1.43 KBytes [ 5] 2.00-3.00 sec 15.6 MBytes 131 Mbits/sec 445 1.43 KBytes [ 5] 3.00-4.00 sec 13.5 MBytes 113 Mbits/sec 453 1.43 KBytes [ 5] 4.00-5.00 sec 13.7 MBytes 115 Mbits/sec 512 1.43 KBytes [ 5] 5.00-6.00 sec 6.51 MBytes 54.6 Mbits/sec 201 1.43 KBytes [ 5] 6.00-7.00 sec 13.5 MBytes 114 Mbits/sec 366 1.43 KBytes [ 5] 7.00-8.00 sec 16.9 MBytes 142 Mbits/sec 325 1.43 KBytes [ 5] 8.00-9.00 sec 24.2 MBytes 203 Mbits/sec 275 1.43 KBytes [ 5] 9.00-10.00 sec 38.8 MBytes 325 Mbits/sec 277 1.43 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 175 MBytes 147 Mbits/sec 3637 sender [ 5] 0.00-10.00 sec 175 MBytes 147 Mbits/sec receiver iperf Done. root@:/usr/home/asdf # iperf3 -s ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- iperf3: error - unable to receive parameters from client: Connection reset by peer ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- Accepted connection from 172.31.16.125, port 21052 ^Ciperf3: interrupt - the server has terminated root@:/usr/home/asdf # iperf3 -s ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- Accepted connection from 172.31.16.125, port 29365 [ 5] local 172.31.16.126 port 5201 connected to 172.31.16.125 port 23884 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 5.85 MBytes 49.0 Mbits/sec [ 5] 1.00-2.00 sec 26.6 MBytes 223 Mbits/sec [ 5] 2.00-3.00 sec 15.6 MBytes 131 Mbits/sec [ 5] 3.00-4.00 sec 13.5 MBytes 113 Mbits/sec [ 5] 4.00-5.00 sec 13.7 MBytes 115 Mbits/sec [ 5] 5.00-6.00 sec 6.51 MBytes 54.6 Mbits/sec [ 5] 6.00-7.00 sec 13.5 MBytes 114 Mbits/sec [ 5] 7.00-8.00 sec 16.9 MBytes 142 Mbits/sec [ 5] 8.00-9.00 sec 24.1 MBytes 202 Mbits/sec [ 5] 9.00-10.00 sec 38.8 MBytes 325 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.00 sec 175 MBytes 147 Mbits/sec receiver ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- Den ons 26 juni 2019 kl 12:36 skrev Eugene Grosbein <[hidden email]>: > 26.06.2019 17:26, Christian M wrote: > > > I ran ifconfig xn0 -txcsum on both test VM's, and all incorrect checksum > disappeared. > > [skip] > > Data looks good. Did it run better this time? > > [hidden email] mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "[hidden email]" |
Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer
|
|
26.06.2019 17:44, Christian M wrote:
> Sorry to say, but no. Nothing changed :( > > The iperf issue concerns me a bit also: Try starting ntpd inside VM guests, wait an hour then post contents of /var/db/ntpd.drift This is to verify quality of time source provided to VMs by hypervisor. _______________________________________________ [hidden email] mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "[hidden email]" |
Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer
|
|
Started ntpd in the two main 12-0-RELEASE vm's I've been testing with.
drift files show -0.262 and -1.144 after about two hours. Den ons 26 juni 2019 kl 12:50 skrev Eugene Grosbein <[hidden email]>: > 26.06.2019 17:44, Christian M wrote: > > > Sorry to say, but no. Nothing changed :( > > > > The iperf issue concerns me a bit also: > > Try starting ntpd inside VM guests, wait an hour then post contents of > /var/db/ntpd.drift > This is to verify quality of time source provided to VMs by hypervisor. > > [hidden email] mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "[hidden email]" |
Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer
|
|
I've installed 12.0-STABLE on two new VM's now. 172.31.16.127 and .128. VIF
cheksum offloading is turned off, and -txcsum for xn0 for both VM's. I feel the throughput is more consistent now, not all over the place as before, even between runs. But the Retr column (tcp retries) in iperf3 has jumped up considerably from hundreds/s to thousands/s. Just a reminder, I have tested this with 11.0-RELEASE also, where the issue appeared first for me. 10.4-RELEASE is as fast as I could expect it to be, and 0 retries. 12.0-STABLE: Connecting to host 172.31.16.128, port 5201 [ 5] local 172.31.16.127 port 16833 connected to 172.31.16.128 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 96.3 MBytes 808 Mbits/sec 2401 2.85 KBytes [ 5] 1.00-2.00 sec 118 MBytes 991 Mbits/sec 3120 17.0 KBytes [ 5] 2.00-3.00 sec 121 MBytes 1.02 Gbits/sec 3203 69.8 KBytes [ 5] 3.00-4.00 sec 102 MBytes 853 Mbits/sec 3126 15.6 KBytes [ 5] 4.00-5.00 sec 110 MBytes 921 Mbits/sec 2890 15.6 KBytes [ 5] 5.00-6.00 sec 108 MBytes 908 Mbits/sec 3308 17.0 KBytes [ 5] 6.00-7.00 sec 104 MBytes 869 Mbits/sec 3046 48.2 KBytes [ 5] 7.00-8.00 sec 98.9 MBytes 830 Mbits/sec 2845 2.85 KBytes [ 5] 8.00-9.00 sec 104 MBytes 874 Mbits/sec 2711 86.8 KBytes [ 5] 9.00-10.00 sec 108 MBytes 904 Mbits/sec 2696 14.2 KBytes [ 5] 10.00-11.00 sec 103 MBytes 864 Mbits/sec 2660 31.3 KBytes [ 5] 11.00-12.00 sec 98.8 MBytes 828 Mbits/sec 2476 19.9 KBytes [ 5] 12.00-13.00 sec 99.9 MBytes 838 Mbits/sec 2857 11.3 KBytes [ 5] 13.00-14.00 sec 107 MBytes 894 Mbits/sec 2685 24.1 KBytes [ 5] 14.00-15.00 sec 114 MBytes 953 Mbits/sec 2321 25.5 KBytes [ 5] 15.00-16.00 sec 93.1 MBytes 781 Mbits/sec 2427 48.3 KBytes [ 5] 16.00-17.00 sec 107 MBytes 895 Mbits/sec 2219 29.8 KBytes [ 5] 17.00-18.00 sec 92.5 MBytes 776 Mbits/sec 2441 12.8 KBytes [ 5] 18.00-19.00 sec 116 MBytes 976 Mbits/sec 2840 38.2 KBytes [ 5] 19.00-20.00 sec 102 MBytes 853 Mbits/sec 2573 43.9 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-20.00 sec 2.05 GBytes 882 Mbits/sec 54845 sender [ 5] 0.00-20.00 sec 2.05 GBytes 881 Mbits/sec receiver 10.3-RELEASE (offloading enabled everywhere for these VM's VIF, and for xn0): Connecting to host 172.31.16.122, port 5201 [ 5] local 172.31.16.121 port 20712 connected to 172.31.16.122 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 832 MBytes 6.98 Gbits/sec 0 580 KBytes [ 5] 1.00-2.00 sec 1.24 GBytes 10.6 Gbits/sec 0 1.04 MBytes [ 5] 2.00-3.00 sec 1.34 GBytes 11.5 Gbits/sec 0 1.57 MBytes [ 5] 3.00-4.00 sec 1.41 GBytes 12.1 Gbits/sec 0 1.76 MBytes [ 5] 4.00-5.00 sec 1.37 GBytes 11.7 Gbits/sec 0 1.77 MBytes [ 5] 5.00-6.00 sec 1.34 GBytes 11.5 Gbits/sec 0 1.77 MBytes [ 5] 6.00-7.00 sec 1.37 GBytes 11.8 Gbits/sec 0 1.77 MBytes [ 5] 7.00-8.00 sec 1.39 GBytes 12.0 Gbits/sec 0 1.77 MBytes [ 5] 8.00-9.00 sec 1.35 GBytes 11.6 Gbits/sec 0 1.77 MBytes [ 5] 9.00-10.00 sec 1.42 GBytes 12.2 Gbits/sec 0 1.77 MBytes [ 5] 10.00-11.00 sec 1.34 GBytes 11.5 Gbits/sec 0 1.77 MBytes [ 5] 11.00-12.00 sec 1.32 GBytes 11.4 Gbits/sec 0 1.77 MBytes [ 5] 12.00-13.00 sec 1.34 GBytes 11.6 Gbits/sec 0 1.77 MBytes [ 5] 13.00-14.00 sec 1.49 GBytes 12.8 Gbits/sec 0 1.77 MBytes [ 5] 14.00-15.00 sec 1.36 GBytes 11.7 Gbits/sec 0 1.77 MBytes [ 5] 15.00-16.00 sec 1.37 GBytes 11.7 Gbits/sec 0 1.77 MBytes [ 5] 16.00-17.00 sec 1.48 GBytes 12.7 Gbits/sec 0 1.77 MBytes [ 5] 17.00-18.00 sec 1.36 GBytes 11.7 Gbits/sec 0 1.77 MBytes [ 5] 18.00-19.00 sec 1.47 GBytes 12.6 Gbits/sec 0 1.77 MBytes [ 5] 19.00-20.00 sec 1.36 GBytes 11.7 Gbits/sec 0 1.77 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-20.00 sec 26.9 GBytes 11.6 Gbits/sec 0 sender [ 5] 0.00-20.01 sec 26.9 GBytes 11.6 Gbits/sec receiver _______________________________________________ [hidden email] mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "[hidden email]" |
Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer
|
|
Den tors 27 juni 2019 kl 12:19 skrev Roger Pau Monné <[hidden email]>:
> On Thu, Jun 27, 2019 at 12:14:33PM +0200, Christian M wrote: > > I've installed 12.0-STABLE on two new VM's now. 172.31.16.127 and .128. > VIF > > cheksum offloading is turned off, and -txcsum for xn0 for both VM's. > > > > I feel the throughput is more consistent now, not all over the place as > > before, even between runs. But the Retr column (tcp retries) in iperf3 > has > > jumped up considerably from hundreds/s to thousands/s. > > > > Just a reminder, I have tested this with 11.0-RELEASE also, where the > issue > > appeared first for me. 10.4-RELEASE is as fast as I could expect it to > be, > > and 0 retries. > > > > 12.0-STABLE: > > > > Connecting to host 172.31.16.128, port 5201 > > [ 5] local 172.31.16.127 port 16833 connected to 172.31.16.128 port 5201 > > [ ID] Interval Transfer Bitrate Retr Cwnd > > [ 5] 0.00-1.00 sec 96.3 MBytes 808 Mbits/sec 2401 2.85 KBytes > > > > [ 5] 1.00-2.00 sec 118 MBytes 991 Mbits/sec 3120 17.0 KBytes > > > > [ 5] 2.00-3.00 sec 121 MBytes 1.02 Gbits/sec 3203 69.8 KBytes > > > > [ 5] 3.00-4.00 sec 102 MBytes 853 Mbits/sec 3126 15.6 KBytes > > > > [ 5] 4.00-5.00 sec 110 MBytes 921 Mbits/sec 2890 15.6 KBytes > > > > [ 5] 5.00-6.00 sec 108 MBytes 908 Mbits/sec 3308 17.0 KBytes > > > > [ 5] 6.00-7.00 sec 104 MBytes 869 Mbits/sec 3046 48.2 KBytes > > > > [ 5] 7.00-8.00 sec 98.9 MBytes 830 Mbits/sec 2845 2.85 KBytes > > > > [ 5] 8.00-9.00 sec 104 MBytes 874 Mbits/sec 2711 86.8 KBytes > > > > [ 5] 9.00-10.00 sec 108 MBytes 904 Mbits/sec 2696 14.2 KBytes > > > > [ 5] 10.00-11.00 sec 103 MBytes 864 Mbits/sec 2660 31.3 KBytes > > > > [ 5] 11.00-12.00 sec 98.8 MBytes 828 Mbits/sec 2476 19.9 KBytes > > > > [ 5] 12.00-13.00 sec 99.9 MBytes 838 Mbits/sec 2857 11.3 KBytes > > > > [ 5] 13.00-14.00 sec 107 MBytes 894 Mbits/sec 2685 24.1 KBytes > > > > [ 5] 14.00-15.00 sec 114 MBytes 953 Mbits/sec 2321 25.5 KBytes > > > > [ 5] 15.00-16.00 sec 93.1 MBytes 781 Mbits/sec 2427 48.3 KBytes > > > > [ 5] 16.00-17.00 sec 107 MBytes 895 Mbits/sec 2219 29.8 KBytes > > > > [ 5] 17.00-18.00 sec 92.5 MBytes 776 Mbits/sec 2441 12.8 KBytes > > > > [ 5] 18.00-19.00 sec 116 MBytes 976 Mbits/sec 2840 38.2 KBytes > > > > [ 5] 19.00-20.00 sec 102 MBytes 853 Mbits/sec 2573 43.9 KBytes > > > > - - - - - - - - - - - - - - - - - - - - - - - - - > > [ ID] Interval Transfer Bitrate Retr > > [ 5] 0.00-20.00 sec 2.05 GBytes 882 Mbits/sec 54845 > > Can you paste the output of ifconfig for both the interfaces used in > the test? > > Are you sure all hardware offloading capabilities are turned off on > both interfaces? > > Can you check what's causing those retries? > > Either using tcpdump, whireshark or some other tool to analyze the > network traffic and detect the errors that cause such retries? > > Thanks, Roger. > 172.31.16.127 (12.0-STABLE): lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 inet 127.0.0.1 netmask 0xff000000 groups: lo nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> xn0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=501<RXCSUM,TSO4,LRO> ether 6e:83:99:ed:ce:f7 inet 172.31.16.127 netmask 0xffffff00 broadcast 172.31.16.255 media: Ethernet manual status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> ethtool -k vif68.0 Features for vif68.0: rx-checksumming: on [fixed] tx-checksumming: off tx-checksum-ipv4: off tx-checksum-ip-generic: off [fixed] tx-checksum-ipv6: off tx-checksum-fcoe-crc: off [fixed] tx-checksum-sctp: off [fixed] scatter-gather: off tx-scatter-gather: off tx-scatter-gather-fraglist: off tcp-segmentation-offload: off tx-tcp-segmentation: off tx-tcp-ecn-segmentation: off [fixed] tx-tcp6-segmentation: off udp-fragmentation-offload: off [fixed] generic-segmentation-offload: off generic-receive-offload: on large-receive-offload: off [fixed] rx-vlan-offload: off [fixed] tx-vlan-offload: off [fixed] ntuple-filters: off [fixed] receive-hashing: off [fixed] highdma: off [fixed] rx-vlan-filter: off [fixed] vlan-challenged: off [fixed] tx-lockless: off [fixed] netns-local: off [fixed] tx-gso-robust: off [fixed] tx-fcoe-segmentation: off [fixed] tx-gre-segmentation: off [fixed] tx-ipip-segmentation: off [fixed] tx-sit-segmentation: off [fixed] tx-udp_tnl-segmentation: off [fixed] fcoe-mtu: off [fixed] tx-nocache-copy: off loopback: off [fixed] rx-fcs: off [fixed] rx-all: off [fixed] tx-vlan-stag-hw-insert: off [fixed] rx-vlan-stag-hw-parse: off [fixed] rx-vlan-stag-filter: off [fixed] l2-fwd-offload: off [fixed] busy-poll: off [fixed] 172.31.16.128 (12.0-STABLE): lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 inet 127.0.0.1 netmask 0xff000000 groups: lo nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> xn0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=501<RXCSUM,TSO4,LRO> ether 32:a7:9f:cc:94:8a inet 172.31.16.128 netmask 0xffffff00 broadcast 172.31.16.255 media: Ethernet manual status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> ethtool -k vif67.0 Features for vif67.0: rx-checksumming: on [fixed] tx-checksumming: off tx-checksum-ipv4: off tx-checksum-ip-generic: off [fixed] tx-checksum-ipv6: off tx-checksum-fcoe-crc: off [fixed] tx-checksum-sctp: off [fixed] scatter-gather: off tx-scatter-gather: off tx-scatter-gather-fraglist: off tcp-segmentation-offload: off tx-tcp-segmentation: off tx-tcp-ecn-segmentation: off [fixed] tx-tcp6-segmentation: off udp-fragmentation-offload: off [fixed] generic-segmentation-offload: off generic-receive-offload: on large-receive-offload: off [fixed] rx-vlan-offload: off [fixed] tx-vlan-offload: off [fixed] ntuple-filters: off [fixed] receive-hashing: off [fixed] highdma: off [fixed] rx-vlan-filter: off [fixed] vlan-challenged: off [fixed] tx-lockless: off [fixed] netns-local: off [fixed] tx-gso-robust: off [fixed] tx-fcoe-segmentation: off [fixed] tx-gre-segmentation: off [fixed] tx-ipip-segmentation: off [fixed] tx-sit-segmentation: off [fixed] tx-udp_tnl-segmentation: off [fixed] fcoe-mtu: off [fixed] tx-nocache-copy: off loopback: off [fixed] rx-fcs: off [fixed] rx-all: off [fixed] tx-vlan-stag-hw-insert: off [fixed] rx-vlan-stag-hw-parse: off [fixed] rx-vlan-stag-filter: off [fixed] l2-fwd-offload: off [fixed] busy-poll: off [fixed] Should I run tcpdump on xen host, or on FreeBSD while running iperf3? As I side note, I installed XCP-ng 7.6.0 on a older server I had laying around just to see how 12.0-RELEASE behaved there, and It was the same thing, super slow between VM's. _______________________________________________ [hidden email] mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "[hidden email]" |
Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer
|
|
On Thu, Jun 27, 2019 at 12:31:35PM +0200, Christian M wrote:
> Den tors 27 juni 2019 kl 12:19 skrev Roger Pau Monné <[hidden email]>: > > > On Thu, Jun 27, 2019 at 12:14:33PM +0200, Christian M wrote: > > > I've installed 12.0-STABLE on two new VM's now. 172.31.16.127 and .128. > > VIF > > > cheksum offloading is turned off, and -txcsum for xn0 for both VM's. > > > > > > I feel the throughput is more consistent now, not all over the place as > > > before, even between runs. But the Retr column (tcp retries) in iperf3 > > has > > > jumped up considerably from hundreds/s to thousands/s. > > > > > > Just a reminder, I have tested this with 11.0-RELEASE also, where the > > issue > > > appeared first for me. 10.4-RELEASE is as fast as I could expect it to > > be, > > > and 0 retries. > > > > > > 12.0-STABLE: > > > > > > Connecting to host 172.31.16.128, port 5201 > > > [ 5] local 172.31.16.127 port 16833 connected to 172.31.16.128 port 5201 > > > [ ID] Interval Transfer Bitrate Retr Cwnd > > > [ 5] 0.00-1.00 sec 96.3 MBytes 808 Mbits/sec 2401 2.85 KBytes > > > > > > [ 5] 1.00-2.00 sec 118 MBytes 991 Mbits/sec 3120 17.0 KBytes > > > > > > [ 5] 2.00-3.00 sec 121 MBytes 1.02 Gbits/sec 3203 69.8 KBytes > > > > > > [ 5] 3.00-4.00 sec 102 MBytes 853 Mbits/sec 3126 15.6 KBytes > > > > > > [ 5] 4.00-5.00 sec 110 MBytes 921 Mbits/sec 2890 15.6 KBytes > > > > > > [ 5] 5.00-6.00 sec 108 MBytes 908 Mbits/sec 3308 17.0 KBytes > > > > > > [ 5] 6.00-7.00 sec 104 MBytes 869 Mbits/sec 3046 48.2 KBytes > > > > > > [ 5] 7.00-8.00 sec 98.9 MBytes 830 Mbits/sec 2845 2.85 KBytes > > > > > > [ 5] 8.00-9.00 sec 104 MBytes 874 Mbits/sec 2711 86.8 KBytes > > > > > > [ 5] 9.00-10.00 sec 108 MBytes 904 Mbits/sec 2696 14.2 KBytes > > > > > > [ 5] 10.00-11.00 sec 103 MBytes 864 Mbits/sec 2660 31.3 KBytes > > > > > > [ 5] 11.00-12.00 sec 98.8 MBytes 828 Mbits/sec 2476 19.9 KBytes > > > > > > [ 5] 12.00-13.00 sec 99.9 MBytes 838 Mbits/sec 2857 11.3 KBytes > > > > > > [ 5] 13.00-14.00 sec 107 MBytes 894 Mbits/sec 2685 24.1 KBytes > > > > > > [ 5] 14.00-15.00 sec 114 MBytes 953 Mbits/sec 2321 25.5 KBytes > > > > > > [ 5] 15.00-16.00 sec 93.1 MBytes 781 Mbits/sec 2427 48.3 KBytes > > > > > > [ 5] 16.00-17.00 sec 107 MBytes 895 Mbits/sec 2219 29.8 KBytes > > > > > > [ 5] 17.00-18.00 sec 92.5 MBytes 776 Mbits/sec 2441 12.8 KBytes > > > > > > [ 5] 18.00-19.00 sec 116 MBytes 976 Mbits/sec 2840 38.2 KBytes > > > > > > [ 5] 19.00-20.00 sec 102 MBytes 853 Mbits/sec 2573 43.9 KBytes > > > > > > - - - - - - - - - - - - - - - - - - - - - - - - - > > > [ ID] Interval Transfer Bitrate Retr > > > [ 5] 0.00-20.00 sec 2.05 GBytes 882 Mbits/sec 54845 > > > > Can you paste the output of ifconfig for both the interfaces used in > > the test? > > > > Are you sure all hardware offloading capabilities are turned off on > > both interfaces? > > > > Can you check what's causing those retries? > > > > Either using tcpdump, whireshark or some other tool to analyze the > > network traffic and detect the errors that cause such retries? > > > > Thanks, Roger. > > > > 172.31.16.127 (12.0-STABLE): > > lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 > options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> > inet6 ::1 prefixlen 128 > inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 > inet 127.0.0.1 netmask 0xff000000 > groups: lo > nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> > xn0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 > options=501<RXCSUM,TSO4,LRO> I would try to disable rxcsum, tso4 and lro also. > ether 6e:83:99:ed:ce:f7 > inet 172.31.16.127 netmask 0xffffff00 broadcast 172.31.16.255 > media: Ethernet manual > status: active > nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> > > ethtool -k vif68.0 > Features for vif68.0: > rx-checksumming: on [fixed] > tx-checksumming: off > tx-checksum-ipv4: off > tx-checksum-ip-generic: off [fixed] > tx-checksum-ipv6: off > tx-checksum-fcoe-crc: off [fixed] > tx-checksum-sctp: off [fixed] > scatter-gather: off > tx-scatter-gather: off > tx-scatter-gather-fraglist: off > tcp-segmentation-offload: off > tx-tcp-segmentation: off > tx-tcp-ecn-segmentation: off [fixed] > tx-tcp6-segmentation: off > udp-fragmentation-offload: off [fixed] > generic-segmentation-offload: off > generic-receive-offload: on > large-receive-offload: off [fixed] > rx-vlan-offload: off [fixed] > tx-vlan-offload: off [fixed] > ntuple-filters: off [fixed] > receive-hashing: off [fixed] > highdma: off [fixed] > rx-vlan-filter: off [fixed] > vlan-challenged: off [fixed] > tx-lockless: off [fixed] > netns-local: off [fixed] > tx-gso-robust: off [fixed] > tx-fcoe-segmentation: off [fixed] > tx-gre-segmentation: off [fixed] > tx-ipip-segmentation: off [fixed] > tx-sit-segmentation: off [fixed] > tx-udp_tnl-segmentation: off [fixed] > fcoe-mtu: off [fixed] > tx-nocache-copy: off > loopback: off [fixed] > rx-fcs: off [fixed] > rx-all: off [fixed] > tx-vlan-stag-hw-insert: off [fixed] > rx-vlan-stag-hw-parse: off [fixed] > rx-vlan-stag-filter: off [fixed] > l2-fwd-offload: off [fixed] > busy-poll: off [fixed] > > 172.31.16.128 (12.0-STABLE): > > lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 > options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> > inet6 ::1 prefixlen 128 > inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 > inet 127.0.0.1 netmask 0xff000000 > groups: lo > nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> > xn0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 > options=501<RXCSUM,TSO4,LRO> > ether 32:a7:9f:cc:94:8a > inet 172.31.16.128 netmask 0xffffff00 broadcast 172.31.16.255 > media: Ethernet manual > status: active > nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> > > ethtool -k vif67.0 > Features for vif67.0: > rx-checksumming: on [fixed] > tx-checksumming: off > tx-checksum-ipv4: off > tx-checksum-ip-generic: off [fixed] > tx-checksum-ipv6: off > tx-checksum-fcoe-crc: off [fixed] > tx-checksum-sctp: off [fixed] > scatter-gather: off > tx-scatter-gather: off > tx-scatter-gather-fraglist: off > tcp-segmentation-offload: off > tx-tcp-segmentation: off > tx-tcp-ecn-segmentation: off [fixed] > tx-tcp6-segmentation: off > udp-fragmentation-offload: off [fixed] > generic-segmentation-offload: off > generic-receive-offload: on > large-receive-offload: off [fixed] > rx-vlan-offload: off [fixed] > tx-vlan-offload: off [fixed] > ntuple-filters: off [fixed] > receive-hashing: off [fixed] > highdma: off [fixed] > rx-vlan-filter: off [fixed] > vlan-challenged: off [fixed] > tx-lockless: off [fixed] > netns-local: off [fixed] > tx-gso-robust: off [fixed] > tx-fcoe-segmentation: off [fixed] > tx-gre-segmentation: off [fixed] > tx-ipip-segmentation: off [fixed] > tx-sit-segmentation: off [fixed] > tx-udp_tnl-segmentation: off [fixed] > fcoe-mtu: off [fixed] > tx-nocache-copy: off > loopback: off [fixed] > rx-fcs: off [fixed] > rx-all: off [fixed] > tx-vlan-stag-hw-insert: off [fixed] > rx-vlan-stag-hw-parse: off [fixed] > rx-vlan-stag-filter: off [fixed] > l2-fwd-offload: off [fixed] > busy-poll: off [fixed] > > Should I run tcpdump on xen host, or on FreeBSD while running iperf3? On the Xen host would be better, since that tcpdump will read the packets as FreeBSD puts them on the wire. > As I side note, I installed XCP-ng 7.6.0 on a older server I had laying > around just to see how 12.0-RELEASE behaved there, and It was the same > thing, super slow between VM's. Yes, I don't think this is related to hardware at all. I will try to find some time to look into this, but it's not going to be soon since I'm currently preparing a talk for a conference I'm attending next week, so I won't likely be able to get to it until the end of July, sorry. Also, I'm no networking expert, so I cannot assure my success in fixing this. As a workaround you can switch to the emulated network card by setting 'hw.xen.disable_pv_nics=1' in /boot/loader.conf. That will give you worse performance than a fully working PV network card, but at least should be consistent. There are others that have switched to virtio-net, but I have no idea how to do that with XCP. Roger. _______________________________________________ [hidden email] mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "[hidden email]" |
Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer
|
|
Thanks Roger.
I disabled pv nic entirely on my two 12.0-RELEASE test VM's. I got 1000baseT full duplex auto selected, so I expected throughput close to that, but to my surprise this are the results: $ ifconfig em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=812099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER> ether 56:65:6f:f3:02:fb inet 172.31.16.125 netmask 0xffffff00 broadcast 172.31.16.255 media: Ethernet autoselect (1000baseT <full-duplex>) status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 inet 127.0.0.1 netmask 0xff000000 groups: lo nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> $ iperf3 -c 172.31.16.126 Connecting to host 172.31.16.126, port 5201 [ 5] local 172.31.16.125 port 11247 connected to 172.31.16.126 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 6.50 MBytes 54.5 Mbits/sec 0 368 KBytes [ 5] 1.00-2.00 sec 7.22 MBytes 60.6 Mbits/sec 0 368 KBytes [ 5] 2.00-3.00 sec 7.26 MBytes 60.9 Mbits/sec 0 368 KBytes [ 5] 3.00-4.00 sec 7.09 MBytes 59.4 Mbits/sec 0 368 KBytes [ 5] 4.00-5.00 sec 7.32 MBytes 61.4 Mbits/sec 0 368 KBytes [ 5] 5.00-6.00 sec 7.22 MBytes 60.6 Mbits/sec 0 385 KBytes [ 5] 6.00-7.00 sec 7.24 MBytes 60.7 Mbits/sec 0 385 KBytes [ 5] 7.00-8.00 sec 7.28 MBytes 61.1 Mbits/sec 0 385 KBytes [ 5] 8.00-9.00 sec 7.55 MBytes 63.3 Mbits/sec 0 385 KBytes [ 5] 9.00-10.00 sec 7.12 MBytes 59.7 Mbits/sec 0 385 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 71.8 MBytes 60.2 Mbits/sec 0 sender [ 5] 0.00-10.06 sec 71.8 MBytes 59.9 Mbits/sec receiver iperf Done. $ iperf3 -c 172.31.16.126 -R Connecting to host 172.31.16.126, port 5201 Reverse mode, remote host 172.31.16.126 is sending [ 5] local 172.31.16.125 port 22443 connected to 172.31.16.126 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 5.06 MBytes 42.5 Mbits/sec [ 5] 1.00-2.00 sec 5.60 MBytes 47.0 Mbits/sec [ 5] 2.00-3.00 sec 6.22 MBytes 52.2 Mbits/sec [ 5] 3.00-4.00 sec 6.92 MBytes 58.0 Mbits/sec [ 5] 4.00-5.00 sec 7.39 MBytes 62.0 Mbits/sec [ 5] 5.00-6.00 sec 7.17 MBytes 60.1 Mbits/sec [ 5] 6.00-7.00 sec 7.18 MBytes 60.3 Mbits/sec [ 5] 7.00-8.00 sec 7.32 MBytes 61.4 Mbits/sec [ 5] 8.00-9.00 sec 6.94 MBytes 58.2 Mbits/sec [ 5] 9.00-10.00 sec 7.04 MBytes 59.0 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 67.3 MBytes 56.5 Mbits/sec 0 sender [ 5] 0.00-10.00 sec 66.8 MBytes 56.1 Mbits/sec receiver iperf Done. Good luck with your talk. Regards, Christian Den tors 4 juli 2019 kl 09:24 skrev Roger Pau Monné <[hidden email]>: > On Thu, Jun 27, 2019 at 12:31:35PM +0200, Christian M wrote: > > Den tors 27 juni 2019 kl 12:19 skrev Roger Pau Monné < > [hidden email]>: > > > > > On Thu, Jun 27, 2019 at 12:14:33PM +0200, Christian M wrote: > > > > I've installed 12.0-STABLE on two new VM's now. 172.31.16.127 and > .128. > > > VIF > > > > cheksum offloading is turned off, and -txcsum for xn0 for both VM's. > > > > > > > > I feel the throughput is more consistent now, not all over the place > as > > > > before, even between runs. But the Retr column (tcp retries) in > iperf3 > > > has > > > > jumped up considerably from hundreds/s to thousands/s. > > > > > > > > Just a reminder, I have tested this with 11.0-RELEASE also, where the > > > issue > > > > appeared first for me. 10.4-RELEASE is as fast as I could expect it > to > > > be, > > > > and 0 retries. > > > > > > > > 12.0-STABLE: > > > > > > > > Connecting to host 172.31.16.128, port 5201 > > > > [ 5] local 172.31.16.127 port 16833 connected to 172.31.16.128 port > 5201 > > > > [ ID] Interval Transfer Bitrate Retr Cwnd > > > > [ 5] 0.00-1.00 sec 96.3 MBytes 808 Mbits/sec 2401 2.85 > KBytes > > > > > > > > [ 5] 1.00-2.00 sec 118 MBytes 991 Mbits/sec 3120 17.0 > KBytes > > > > > > > > [ 5] 2.00-3.00 sec 121 MBytes 1.02 Gbits/sec 3203 69.8 > KBytes > > > > > > > > [ 5] 3.00-4.00 sec 102 MBytes 853 Mbits/sec 3126 15.6 > KBytes > > > > > > > > [ 5] 4.00-5.00 sec 110 MBytes 921 Mbits/sec 2890 15.6 > KBytes > > > > > > > > [ 5] 5.00-6.00 sec 108 MBytes 908 Mbits/sec 3308 17.0 > KBytes > > > > > > > > [ 5] 6.00-7.00 sec 104 MBytes 869 Mbits/sec 3046 48.2 > KBytes > > > > > > > > [ 5] 7.00-8.00 sec 98.9 MBytes 830 Mbits/sec 2845 2.85 > KBytes > > > > > > > > [ 5] 8.00-9.00 sec 104 MBytes 874 Mbits/sec 2711 86.8 > KBytes > > > > > > > > [ 5] 9.00-10.00 sec 108 MBytes 904 Mbits/sec 2696 14.2 > KBytes > > > > > > > > [ 5] 10.00-11.00 sec 103 MBytes 864 Mbits/sec 2660 31.3 > KBytes > > > > > > > > [ 5] 11.00-12.00 sec 98.8 MBytes 828 Mbits/sec 2476 19.9 > KBytes > > > > > > > > [ 5] 12.00-13.00 sec 99.9 MBytes 838 Mbits/sec 2857 11.3 > KBytes > > > > > > > > [ 5] 13.00-14.00 sec 107 MBytes 894 Mbits/sec 2685 24.1 > KBytes > > > > > > > > [ 5] 14.00-15.00 sec 114 MBytes 953 Mbits/sec 2321 25.5 > KBytes > > > > > > > > [ 5] 15.00-16.00 sec 93.1 MBytes 781 Mbits/sec 2427 48.3 > KBytes > > > > > > > > [ 5] 16.00-17.00 sec 107 MBytes 895 Mbits/sec 2219 29.8 > KBytes > > > > > > > > [ 5] 17.00-18.00 sec 92.5 MBytes 776 Mbits/sec 2441 12.8 > KBytes > > > > > > > > [ 5] 18.00-19.00 sec 116 MBytes 976 Mbits/sec 2840 38.2 > KBytes > > > > > > > > [ 5] 19.00-20.00 sec 102 MBytes 853 Mbits/sec 2573 43.9 > KBytes > > > > > > > > - - - - - - - - - - - - - - - - - - - - - - - - - > > > > [ ID] Interval Transfer Bitrate Retr > > > > [ 5] 0.00-20.00 sec 2.05 GBytes 882 Mbits/sec 54845 > > > > > > Can you paste the output of ifconfig for both the interfaces used in > > > the test? > > > > > > Are you sure all hardware offloading capabilities are turned off on > > > both interfaces? > > > > > > Can you check what's causing those retries? > > > > > > Either using tcpdump, whireshark or some other tool to analyze the > > > network traffic and detect the errors that cause such retries? > > > > > > Thanks, Roger. > > > > > > > 172.31.16.127 (12.0-STABLE): > > > > lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 > > options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> > > inet6 ::1 prefixlen 128 > > inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 > > inet 127.0.0.1 netmask 0xff000000 > > groups: lo > > nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> > > xn0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 > > options=501<RXCSUM,TSO4,LRO> > > I would try to disable rxcsum, tso4 and lro also. > > > ether 6e:83:99:ed:ce:f7 > > inet 172.31.16.127 netmask 0xffffff00 broadcast 172.31.16.255 > > media: Ethernet manual > > status: active > > nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> > > > > ethtool -k vif68.0 > > Features for vif68.0: > > rx-checksumming: on [fixed] > > tx-checksumming: off > > tx-checksum-ipv4: off > > tx-checksum-ip-generic: off [fixed] > > tx-checksum-ipv6: off > > tx-checksum-fcoe-crc: off [fixed] > > tx-checksum-sctp: off [fixed] > > scatter-gather: off > > tx-scatter-gather: off > > tx-scatter-gather-fraglist: off > > tcp-segmentation-offload: off > > tx-tcp-segmentation: off > > tx-tcp-ecn-segmentation: off [fixed] > > tx-tcp6-segmentation: off > > udp-fragmentation-offload: off [fixed] > > generic-segmentation-offload: off > > generic-receive-offload: on > > large-receive-offload: off [fixed] > > rx-vlan-offload: off [fixed] > > tx-vlan-offload: off [fixed] > > ntuple-filters: off [fixed] > > receive-hashing: off [fixed] > > highdma: off [fixed] > > rx-vlan-filter: off [fixed] > > vlan-challenged: off [fixed] > > tx-lockless: off [fixed] > > netns-local: off [fixed] > > tx-gso-robust: off [fixed] > > tx-fcoe-segmentation: off [fixed] > > tx-gre-segmentation: off [fixed] > > tx-ipip-segmentation: off [fixed] > > tx-sit-segmentation: off [fixed] > > tx-udp_tnl-segmentation: off [fixed] > > fcoe-mtu: off [fixed] > > tx-nocache-copy: off > > loopback: off [fixed] > > rx-fcs: off [fixed] > > rx-all: off [fixed] > > tx-vlan-stag-hw-insert: off [fixed] > > rx-vlan-stag-hw-parse: off [fixed] > > rx-vlan-stag-filter: off [fixed] > > l2-fwd-offload: off [fixed] > > busy-poll: off [fixed] > > > > 172.31.16.128 (12.0-STABLE): > > > > lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 > > options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> > > inet6 ::1 prefixlen 128 > > inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 > > inet 127.0.0.1 netmask 0xff000000 > > groups: lo > > nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> > > xn0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 > > options=501<RXCSUM,TSO4,LRO> > > ether 32:a7:9f:cc:94:8a > > inet 172.31.16.128 netmask 0xffffff00 broadcast 172.31.16.255 > > media: Ethernet manual > > status: active > > nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> > > > > ethtool -k vif67.0 > > Features for vif67.0: > > rx-checksumming: on [fixed] > > tx-checksumming: off > > tx-checksum-ipv4: off > > tx-checksum-ip-generic: off [fixed] > > tx-checksum-ipv6: off > > tx-checksum-fcoe-crc: off [fixed] > > tx-checksum-sctp: off [fixed] > > scatter-gather: off > > tx-scatter-gather: off > > tx-scatter-gather-fraglist: off > > tcp-segmentation-offload: off > > tx-tcp-segmentation: off > > tx-tcp-ecn-segmentation: off [fixed] > > tx-tcp6-segmentation: off > > udp-fragmentation-offload: off [fixed] > > generic-segmentation-offload: off > > generic-receive-offload: on > > large-receive-offload: off [fixed] > > rx-vlan-offload: off [fixed] > > tx-vlan-offload: off [fixed] > > ntuple-filters: off [fixed] > > receive-hashing: off [fixed] > > highdma: off [fixed] > > rx-vlan-filter: off [fixed] > > vlan-challenged: off [fixed] > > tx-lockless: off [fixed] > > netns-local: off [fixed] > > tx-gso-robust: off [fixed] > > tx-fcoe-segmentation: off [fixed] > > tx-gre-segmentation: off [fixed] > > tx-ipip-segmentation: off [fixed] > > tx-sit-segmentation: off [fixed] > > tx-udp_tnl-segmentation: off [fixed] > > fcoe-mtu: off [fixed] > > tx-nocache-copy: off > > loopback: off [fixed] > > rx-fcs: off [fixed] > > rx-all: off [fixed] > > tx-vlan-stag-hw-insert: off [fixed] > > rx-vlan-stag-hw-parse: off [fixed] > > rx-vlan-stag-filter: off [fixed] > > l2-fwd-offload: off [fixed] > > busy-poll: off [fixed] > > > > Should I run tcpdump on xen host, or on FreeBSD while running iperf3? > > On the Xen host would be better, since that tcpdump will read the > packets as FreeBSD puts them on the wire. > > > As I side note, I installed XCP-ng 7.6.0 on a older server I had laying > > around just to see how 12.0-RELEASE behaved there, and It was the same > > thing, super slow between VM's. > > Yes, I don't think this is related to hardware at all. > > I will try to find some time to look into this, but it's not going to > be soon since I'm currently preparing a talk for a conference I'm > attending next week, so I won't likely be able to get to it until the > end of July, sorry. Also, I'm no networking expert, so I cannot assure > my success in fixing this. > > As a workaround you can switch to the emulated network card by > setting 'hw.xen.disable_pv_nics=1' in /boot/loader.conf. That will > give you worse performance than a fully working PV network card, but > at least should be consistent. There are others that have switched to > virtio-net, but I have no idea how to do that with XCP. > > Roger. > [hidden email] mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "[hidden email]" |
«
Return to freebsd-xen
|
1 view|%1 views
| Free forum by Nabble | Edit this page |