Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer

classic Classic list List threaded Threaded
22 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer

Bjarne-4
Well, to add some datapoints,  I just did a quick test om my private homeserver, testing with iperf3
-s  / iperf3 -c <server-ip>

I have not much knowledge of iperf3 network test and it's validity, but there is definitely some
differences to be seen.

Summary:

FBSD <-> FBSD   3,5Gbit/sec
FBSD -> Centos:  3GBit/Sec
Centos -> FBSD: 12 Gbit/sec

Centos <-> Centos: 12 Gbit/sec
OpenSuse <-> OpenSuse 13 GBit/sec

So it ssems there is some fishy when Freebsd is used as client

xen running on opensuse,  11 vm's,  freebsd , openSuse and Centos. No vm has been optimized in any
way,  everything is out of the box.


===========================
Test 1,  Freebsd 12 -> freebsd 12
===========================

Server Freebsd 12:
ccepted connection from 192.168.1.100, port 39518
[  5] local 192.168.1.99 port 5201 connected to 192.168.1.100 port 48518
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   374 MBytes  3.14 Gbits/sec
[  5]   1.00-2.00   sec   444 MBytes  3.72 Gbits/sec
[  5]   2.00-3.00   sec   411 MBytes  3.45 Gbits/sec
[  5]   3.00-4.00   sec   438 MBytes  3.67 Gbits/sec
[  5]   4.00-5.00   sec   413 MBytes  3.46 Gbits/sec
[  5]   5.00-6.00   sec   417 MBytes  3.50 Gbits/sec
[  5]   6.00-7.00   sec   412 MBytes  3.45 Gbits/sec
[  5]   7.00-8.00   sec   432 MBytes  3.63 Gbits/sec
[  5]   8.00-9.00   sec   418 MBytes  3.51 Gbits/sec
[  5]   9.00-10.00  sec   375 MBytes  3.15 Gbits/sec
[  5]  10.00-10.00  sec   332 KBytes  2.17 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  4.04 GBytes  3.47 Gbits/sec                  receiver

Client, freebsd 12:
Connecting to host 10.192.168.99, port 5201
[  5] local 192.168.1.100 port 48518 connected to 192.168.1.99 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   375 MBytes  3.14 Gbits/sec  239    178 KBytes
[  5]   1.00-2.00   sec   445 MBytes  3.73 Gbits/sec  834   74.1 KBytes
[  5]   2.00-3.00   sec   411 MBytes  3.45 Gbits/sec  794   38.5 KBytes
[  5]   3.00-4.00   sec   438 MBytes  3.68 Gbits/sec  808    198 KBytes
[  5]   4.00-5.00   sec   413 MBytes  3.46 Gbits/sec  777   67.0 KBytes
[  5]   5.00-6.00   sec   417 MBytes  3.50 Gbits/sec  795    120 KBytes
[  5]   6.00-7.00   sec   412 MBytes  3.45 Gbits/sec  763   72.7 KBytes
[  5]   7.00-8.00   sec   432 MBytes  3.62 Gbits/sec  810    329 KBytes
[  5]   8.00-9.00   sec   418 MBytes  3.51 Gbits/sec  777    219 KBytes
[  5]   9.00-10.00  sec   375 MBytes  3.15 Gbits/sec  688    304 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  4.04 GBytes  3.47 Gbits/sec 7285             sender
[  5]   0.00-10.00  sec  4.04 GBytes  3.47 Gbits/sec                  receiver


==========================
test 2, centos 7.6 -> Freebsd 12
==========================

Server Freebsd 12:
Accepted connection from 192.168.1.162, port 49180
[  5] local 192.168.1.99 port 5201 connected to 192.168.1.162 port 49182
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  1.17 GBytes  10.0 Gbits/sec
[  5]   1.00-2.00   sec  1.47 GBytes  12.6 Gbits/sec
[  5]   2.00-3.00   sec  1.30 GBytes  11.2 Gbits/sec
[  5]   3.00-4.00   sec  1.32 GBytes  11.3 Gbits/sec
[  5]   4.00-5.00   sec  1.44 GBytes  12.4 Gbits/sec
[  5]   5.00-6.00   sec  1.29 GBytes  11.1 Gbits/sec
[  5]   6.00-7.00   sec  1.32 GBytes  11.3 Gbits/sec
[  5]   7.00-8.00   sec  1.36 GBytes  11.7 Gbits/sec
[  5]   8.00-9.00   sec  1.32 GBytes  11.3 Gbits/sec
[  5]   9.00-10.00  sec  1.41 GBytes  12.1 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  13.4 GBytes  11.5 Gbits/sec                  receiver

Client, centos 7.6:
Connecting to host 192.168.1.99, port 5201
[  4] local 192.168.1.162 port 49182 connected to 192.168.1.99 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  1.17 GBytes  10.1 Gbits/sec    0    822 KBytes
[  4]   1.00-2.00   sec  1.47 GBytes  12.6 Gbits/sec    0   1.53 MBytes
[  4]   2.00-3.00   sec  1.30 GBytes  11.2 Gbits/sec    0   1.57 MBytes
[  4]   3.00-4.00   sec  1.32 GBytes  11.3 Gbits/sec    0   1.64 MBytes
[  4]   4.00-5.00   sec  1.44 GBytes  12.4 Gbits/sec    0   1.70 MBytes
[  4]   5.00-6.00   sec  1.29 GBytes  11.1 Gbits/sec    0   1.70 MBytes
[  4]   6.00-7.00   sec  1.32 GBytes  11.3 Gbits/sec    0   1.70 MBytes
[  4]   7.00-8.00   sec  1.36 GBytes  11.7 Gbits/sec    0   1.70 MBytes
[  4]   8.00-9.00   sec  1.32 GBytes  11.3 Gbits/sec    0   1.70 MBytes
[  4]   9.00-10.00  sec  1.41 GBytes  12.1 Gbits/sec    0   1.70 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  13.4 GBytes  11.5 Gbits/sec 0 sender
[  4]   0.00-10.00  sec  13.4 GBytes  11.5 Gbits/sec                  receiver


==========================
test 3, Freebsd 12 -> Centos 7.6
==========================

Server, Centos 7.6:
Accepted connection from 192.168.1.100, port 20309
[  5] local 192.168.1.155 port 5201 connected to 192.168.1.100 port 17917
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec   335 MBytes  2.81 Gbits/sec
[  5]   1.00-2.00   sec   379 MBytes  3.18 Gbits/sec
[  5]   2.00-3.00   sec   368 MBytes  3.08 Gbits/sec
[  5]   3.00-4.00   sec   371 MBytes  3.11 Gbits/sec
[  5]   4.00-5.00   sec   374 MBytes  3.14 Gbits/sec
[  5]   5.00-6.00   sec   362 MBytes  3.04 Gbits/sec
[  5]   6.00-7.00   sec   381 MBytes  3.19 Gbits/sec
[  5]   7.00-8.00   sec   384 MBytes  3.22 Gbits/sec
[  5]   8.00-9.00   sec   374 MBytes  3.14 Gbits/sec
[  5]   9.00-10.00  sec   366 MBytes  3.07 Gbits/sec
[  5]  10.00-10.13  sec  56.4 MBytes  3.52 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-10.13  sec  0.00 Bytes  0.00 bits/sec sender
[  5]   0.00-10.13  sec  3.66 GBytes  3.10 Gbits/sec                  receiver

Client, Freebsd 12:
   5] local 192.168.1.100 port 17917 connected to 192.168.1.155 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   387 MBytes  3.25 Gbits/sec   83    309 KBytes
[  5]   1.00-2.00   sec   380 MBytes  3.19 Gbits/sec   88   11.4 KBytes
[  5]   2.00-3.00   sec   365 MBytes  3.06 Gbits/sec   83   54.2 KBytes
[  5]   3.00-4.00   sec   372 MBytes  3.12 Gbits/sec   86    322 KBytes
[  5]   4.00-5.00   sec   370 MBytes  3.11 Gbits/sec   83   2.85 KBytes
[  5]   5.00-6.00   sec   366 MBytes  3.07 Gbits/sec   84    298 KBytes
[  5]   6.00-7.00   sec   382 MBytes  3.20 Gbits/sec   87    356 KBytes
[  5]   7.00-8.00   sec   380 MBytes  3.19 Gbits/sec   87    369 KBytes
[  5]   8.00-9.00   sec   374 MBytes  3.14 Gbits/sec   87   29.9 KBytes
[  5]   9.00-10.00  sec   375 MBytes  3.15 Gbits/sec   86    419 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  3.66 GBytes  3.15 Gbits/sec 854 sender
[  5]   0.00-10.00  sec  3.66 GBytes  3.15 Gbits/sec                  receiver


==========================
test 4, Centos 7.6 -> Centos 7.6
==========================

Server:
Accepted connection from 192.168.1.162, port 47606
[  5] local 192.168.1.155 port 5201 connected to 192.168.1.162 port 47608
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec  1.60 GBytes  13.7 Gbits/sec
[  5]   1.00-2.00   sec  1.59 GBytes  13.7 Gbits/sec
[  5]   2.00-3.00   sec  1.60 GBytes  13.8 Gbits/sec
[  5]   3.00-4.00   sec  1.71 GBytes  14.7 Gbits/sec
[  5]   4.00-5.00   sec  1.49 GBytes  12.8 Gbits/sec
[  5]   5.00-6.00   sec  1.38 GBytes  11.9 Gbits/sec
[  5]   6.00-7.00   sec  1.52 GBytes  13.1 Gbits/sec
[  5]   7.00-8.00   sec  1.51 GBytes  13.0 Gbits/sec
[  5]   8.00-9.00   sec  1.51 GBytes  13.0 Gbits/sec
[  5]   9.00-10.00  sec  1.51 GBytes  12.9 Gbits/sec
[  5]  10.00-10.03  sec  48.1 MBytes  11.9 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-10.03  sec  0.00 Bytes  0.00 bits/sec sender
[  5]   0.00-10.03  sec  15.5 GBytes  13.2 Gbits/sec                  receiver

Client, Centos 7.6
Connecting to host 192.168.1.155, port 5201
[  4] local 192.168.1.162 port 47608 connected to 192.168.1.155 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  1.66 GBytes  14.3 Gbits/sec    0   1.41 MBytes
[  4]   1.00-2.00   sec  1.58 GBytes  13.5 Gbits/sec    0   1.46 MBytes
[  4]   2.00-3.00   sec  1.61 GBytes  13.9 Gbits/sec    0   1.91 MBytes
[  4]   3.00-4.00   sec  1.71 GBytes  14.7 Gbits/sec    0   2.13 MBytes
[  4]   4.00-5.00   sec  1.47 GBytes  12.6 Gbits/sec    0   2.49 MBytes
[  4]   5.00-6.00   sec  1.39 GBytes  11.9 Gbits/sec    0   2.56 MBytes
[  4]   6.00-7.00   sec  1.53 GBytes  13.1 Gbits/sec    0   2.71 MBytes
[  4]   7.00-8.00   sec  1.50 GBytes  12.9 Gbits/sec    0   2.73 MBytes
[  4]   8.00-9.00   sec  1.50 GBytes  12.9 Gbits/sec    0   2.81 MBytes
[  4]   9.00-10.00  sec  1.51 GBytes  12.9 Gbits/sec    0   2.84 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  15.5 GBytes  13.3 Gbits/sec 0 sender
[  4]   0.00-10.00  sec  15.5 GBytes  13.3 Gbits/sec                  receiver


===============================
test4, Opensuse 15.1 -> OpenSuse 15.1
================================

Server, Opensuse 15.1:
[  5] local 192.168.1.158 port 5201 connected to 192.168.1.163 port 39334
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  1.51 GBytes  13.0 Gbits/sec
[  5]   1.00-2.00   sec  1.61 GBytes  13.8 Gbits/sec
[  5]   2.00-3.00   sec  1.58 GBytes  13.6 Gbits/sec
[  5]   3.00-4.00   sec  1.58 GBytes  13.6 Gbits/sec
[  5]   4.00-5.00   sec  1.56 GBytes  13.4 Gbits/sec
[  5]   5.00-6.00   sec  1.60 GBytes  13.7 Gbits/sec
[  5]   6.00-7.00   sec  1.58 GBytes  13.5 Gbits/sec
[  5]   7.00-8.00   sec  1.60 GBytes  13.8 Gbits/sec
[  5]   8.00-9.00   sec  1.60 GBytes  13.7 Gbits/sec
[  5]   9.00-10.00  sec  1.61 GBytes  13.8 Gbits/sec
[  5]  10.00-10.04  sec  68.0 MBytes  14.2 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.04  sec  15.9 GBytes  13.6 Gbits/sec                  receiver

Client, OpenSuse 15.1:
[  5] local 192.168.1.163 port 39334 connected to 192.168.1.158 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.58 GBytes  13.6 Gbits/sec    0    723 KBytes
[  5]   1.00-2.00   sec  1.61 GBytes  13.8 Gbits/sec    0    805 KBytes
[  5]   2.00-3.00   sec  1.57 GBytes  13.5 Gbits/sec    0    926 KBytes
[  5]   3.00-4.00   sec  1.58 GBytes  13.6 Gbits/sec    0    926 KBytes
[  5]   4.00-5.00   sec  1.56 GBytes  13.4 Gbits/sec    0    970 KBytes
[  5]   5.00-6.00   sec  1.60 GBytes  13.7 Gbits/sec    0   1.05 MBytes
[  5]   6.00-7.00   sec  1.57 GBytes  13.5 Gbits/sec    0   1.05 MBytes
[  5]   7.00-8.00   sec  1.61 GBytes  13.8 Gbits/sec    0   1.05 MBytes
[  5]   8.00-9.00   sec  1.59 GBytes  13.7 Gbits/sec    0   1.05 MBytes
[  5]   9.00-10.00  sec  1.62 GBytes  13.9 Gbits/sec    0   1.05 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  15.9 GBytes  13.7 Gbits/sec 0 sender
[  5]   0.00-10.04  sec  15.9 GBytes  13.6 Gbits/sec                  receiver






> I've noticed very slow networking speed between VM's with FreeBSD on the
> same host (XCP-ng 7.6.0) for more recent FreeBSD-versions.
>
>
> I've made some tests that show me that something happened from 10.4-RELEASE
> to 11.0-RELEASE that had a huge impact on network performance, and
> 12.0-RELEASE is even slower.
>
>
> My test setup:
>
> Host: XCP-ng 7.6.0, managed with XenOrchestra. Open source.
>
> Network: Internal Private Network on the host (not connected to a PIF).
> Each VM has only one VIF connected to this network.
>
> VM's:
>
> 2 x 12.0-RELEASE
>
> 2 x 11.0-RELEASE
>
> 2 x 10.4-RELEASE
>
>
> All clean identical installs from XenOrchestra, only installed iperf on
> each VM for testing. (xe-guest-utilities makes no difference in my tests,
> I've tried with and without).
>
>
> iperf -s on first server listed below, and iperf -c <ip> -r on the second
> to test speed back and forth:
>
>
> 12.0 <-> 12.0: 50Mbit as client and server
>
> 12.0 <-> 11.0: 800Mbit/s (11.0 as client), and 140Mbit/s (11.0 as server)
>
> 12.0 <-> 10.4: 2.76Gbit (10.4 as client), and 1.25Gbit (10.4 as server).
>
> 11.0 <-> 11.0: 219Mbit as client, 99Mbit as server
>
> 10.4 <-> 10.4: 11.2Gbit as client, 10.9Gbit as server
>
>
> As I side note, not sure if related, but I've noticed that I can't run
> iperf with -r flag on 10.4-RELEASE. I get this error message:
>
>
> iperf -c 172.31.16.122 -r
>
> ------------------------------------------------------------
>
> Server listening on TCP port 5001
>
> TCP window size: 64.0 KByte (default)
>
> ------------------------------------------------------------
>
> write failed: Broken pipe
>
> ------------------------------------------------------------
>
> Client connecting to 172.31.16.122, TCP port 5001
>
> TCP window size: 32.5 KByte (default)
>
> ------------------------------------------------------------
>
> [ 5] local 172.31.16.121 port 19231 connected with 172.31.16.122 port 5001
>
> [ ID] Interval Transfer Bandwidth
>
> [ 5] 0.0- 0.0 sec 0.00 Bytes 0.00 bits/sec
>
>
>
> I can run iperf -s fine, and iperf -c <ip> from the other 10.4 VM though:
>
>
>
> iperf -c 172.31.16.122
>
> ------------------------------------------------------------
>
> Client connecting to 172.31.16.122, TCP port 5001
>
> TCP window size: 32.5 KByte (default)
>
> ------------------------------------------------------------
>
> [ 3] local 172.31.16.121 port 22055 connected with 172.31.16.122 port 5001
>
> [ ID] Interval Transfer Bandwidth
>
> [ 3] 0.0-10.0 sec 12.9 GBytes 11.1 Gbits/sec
>
>
>
> What have I tried to solve this?
>
> I've tried to disable checksum offloading for the 12.0-RELEASE VIF's via
> XCP-ng. Disabled basically everything without any difference in iperf
> results: other-config (MRW): ethtool-sg: off; ethtool-tso: off;
> ethtool-ufo: off; ethtool-gso: off; ethtool-rx: off; ethtool-tx: off
>
>
> Also tried disabling offloading in FreeBSD with ifconfig xn0 -txcsum
> -rxcsum -tso -lro and no difference here either.
>
>
> Any ideas of how to proceed now to find a solution for this?
>
>
> Regards,
>
> Christian
>

--
mvh,
Bjarne

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer

Christian Marcos
Thanks for your input Bjarne.

Previously I tested with iperf2, but I have made som tests with iperf3 now
also and noticed there are a lot of "Retr" (TCP retries) in some cases.
Went back here to your post and saw that your results also showed a lot of
retries in some cases. My new tests showed similar results to yours
(although not nearly as hi throughput as you have). When Linux is client
the retries are 0, with FreeBSD as client the number is > 0. I'm not sure
if our numbers are considered high though, and something that actually is a
problem? Although, on a internal network with no external factors like
interference, I feel this should always be 0 no matter what?

What hypervisor were you running in your tests? Version?

Regards,
Christian


Den mån 24 juni 2019 kl 17:56 skrev Bjarne <[hidden email]>:

> Well, to add some datapoints,  I just did a quick test om my private
> homeserver, testing with iperf3
> -s  / iperf3 -c <server-ip>
>
> I have not much knowledge of iperf3 network test and it's validity, but
> there is definitely some
> differences to be seen.
>
> Summary:
>
> FBSD <-> FBSD   3,5Gbit/sec
> FBSD -> Centos:  3GBit/Sec
> Centos -> FBSD: 12 Gbit/sec
>
> Centos <-> Centos: 12 Gbit/sec
> OpenSuse <-> OpenSuse 13 GBit/sec
>
> So it ssems there is some fishy when Freebsd is used as client
>
> xen running on opensuse,  11 vm's,  freebsd , openSuse and Centos. No vm
> has been optimized in any
> way,  everything is out of the box.
>
>
> ===========================
> Test 1,  Freebsd 12 -> freebsd 12
> ===========================
>
> Server Freebsd 12:
> ccepted connection from 192.168.1.100, port 39518
> [  5] local 192.168.1.99 port 5201 connected to 192.168.1.100 port 48518
> [ ID] Interval           Transfer     Bitrate
> [  5]   0.00-1.00   sec   374 MBytes  3.14 Gbits/sec
> [  5]   1.00-2.00   sec   444 MBytes  3.72 Gbits/sec
> [  5]   2.00-3.00   sec   411 MBytes  3.45 Gbits/sec
> [  5]   3.00-4.00   sec   438 MBytes  3.67 Gbits/sec
> [  5]   4.00-5.00   sec   413 MBytes  3.46 Gbits/sec
> [  5]   5.00-6.00   sec   417 MBytes  3.50 Gbits/sec
> [  5]   6.00-7.00   sec   412 MBytes  3.45 Gbits/sec
> [  5]   7.00-8.00   sec   432 MBytes  3.63 Gbits/sec
> [  5]   8.00-9.00   sec   418 MBytes  3.51 Gbits/sec
> [  5]   9.00-10.00  sec   375 MBytes  3.15 Gbits/sec
> [  5]  10.00-10.00  sec   332 KBytes  2.17 Gbits/sec
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID] Interval           Transfer     Bitrate
> [  5]   0.00-10.00  sec  4.04 GBytes  3.47 Gbits/sec
> receiver
>
> Client, freebsd 12:
> Connecting to host 10.192.168.99, port 5201
> [  5] local 192.168.1.100 port 48518 connected to 192.168.1.99 port 5201
> [ ID] Interval           Transfer     Bitrate         Retr  Cwnd
> [  5]   0.00-1.00   sec   375 MBytes  3.14 Gbits/sec  239    178 KBytes
> [  5]   1.00-2.00   sec   445 MBytes  3.73 Gbits/sec  834   74.1 KBytes
> [  5]   2.00-3.00   sec   411 MBytes  3.45 Gbits/sec  794   38.5 KBytes
> [  5]   3.00-4.00   sec   438 MBytes  3.68 Gbits/sec  808    198 KBytes
> [  5]   4.00-5.00   sec   413 MBytes  3.46 Gbits/sec  777   67.0 KBytes
> [  5]   5.00-6.00   sec   417 MBytes  3.50 Gbits/sec  795    120 KBytes
> [  5]   6.00-7.00   sec   412 MBytes  3.45 Gbits/sec  763   72.7 KBytes
> [  5]   7.00-8.00   sec   432 MBytes  3.62 Gbits/sec  810    329 KBytes
> [  5]   8.00-9.00   sec   418 MBytes  3.51 Gbits/sec  777    219 KBytes
> [  5]   9.00-10.00  sec   375 MBytes  3.15 Gbits/sec  688    304 KBytes
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID] Interval           Transfer     Bitrate         Retr
> [  5]   0.00-10.00  sec  4.04 GBytes  3.47 Gbits/sec 7285
> sender
> [  5]   0.00-10.00  sec  4.04 GBytes  3.47 Gbits/sec
> receiver
>
>
> ==========================
> test 2, centos 7.6 -> Freebsd 12
> ==========================
>
> Server Freebsd 12:
> Accepted connection from 192.168.1.162, port 49180
> [  5] local 192.168.1.99 port 5201 connected to 192.168.1.162 port 49182
> [ ID] Interval           Transfer     Bitrate
> [  5]   0.00-1.00   sec  1.17 GBytes  10.0 Gbits/sec
> [  5]   1.00-2.00   sec  1.47 GBytes  12.6 Gbits/sec
> [  5]   2.00-3.00   sec  1.30 GBytes  11.2 Gbits/sec
> [  5]   3.00-4.00   sec  1.32 GBytes  11.3 Gbits/sec
> [  5]   4.00-5.00   sec  1.44 GBytes  12.4 Gbits/sec
> [  5]   5.00-6.00   sec  1.29 GBytes  11.1 Gbits/sec
> [  5]   6.00-7.00   sec  1.32 GBytes  11.3 Gbits/sec
> [  5]   7.00-8.00   sec  1.36 GBytes  11.7 Gbits/sec
> [  5]   8.00-9.00   sec  1.32 GBytes  11.3 Gbits/sec
> [  5]   9.00-10.00  sec  1.41 GBytes  12.1 Gbits/sec
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID] Interval           Transfer     Bitrate
> [  5]   0.00-10.00  sec  13.4 GBytes  11.5 Gbits/sec
> receiver
>
> Client, centos 7.6:
> Connecting to host 192.168.1.99, port 5201
> [  4] local 192.168.1.162 port 49182 connected to 192.168.1.99 port 5201
> [ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
> [  4]   0.00-1.00   sec  1.17 GBytes  10.1 Gbits/sec    0    822 KBytes
> [  4]   1.00-2.00   sec  1.47 GBytes  12.6 Gbits/sec    0   1.53 MBytes
> [  4]   2.00-3.00   sec  1.30 GBytes  11.2 Gbits/sec    0   1.57 MBytes
> [  4]   3.00-4.00   sec  1.32 GBytes  11.3 Gbits/sec    0   1.64 MBytes
> [  4]   4.00-5.00   sec  1.44 GBytes  12.4 Gbits/sec    0   1.70 MBytes
> [  4]   5.00-6.00   sec  1.29 GBytes  11.1 Gbits/sec    0   1.70 MBytes
> [  4]   6.00-7.00   sec  1.32 GBytes  11.3 Gbits/sec    0   1.70 MBytes
> [  4]   7.00-8.00   sec  1.36 GBytes  11.7 Gbits/sec    0   1.70 MBytes
> [  4]   8.00-9.00   sec  1.32 GBytes  11.3 Gbits/sec    0   1.70 MBytes
> [  4]   9.00-10.00  sec  1.41 GBytes  12.1 Gbits/sec    0   1.70 MBytes
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID] Interval           Transfer     Bandwidth       Retr
> [  4]   0.00-10.00  sec  13.4 GBytes  11.5 Gbits/sec 0 sender
> [  4]   0.00-10.00  sec  13.4 GBytes  11.5 Gbits/sec
> receiver
>
>
> ==========================
> test 3, Freebsd 12 -> Centos 7.6
> ==========================
>
> Server, Centos 7.6:
> Accepted connection from 192.168.1.100, port 20309
> [  5] local 192.168.1.155 port 5201 connected to 192.168.1.100 port 17917
> [ ID] Interval           Transfer     Bandwidth
> [  5]   0.00-1.00   sec   335 MBytes  2.81 Gbits/sec
> [  5]   1.00-2.00   sec   379 MBytes  3.18 Gbits/sec
> [  5]   2.00-3.00   sec   368 MBytes  3.08 Gbits/sec
> [  5]   3.00-4.00   sec   371 MBytes  3.11 Gbits/sec
> [  5]   4.00-5.00   sec   374 MBytes  3.14 Gbits/sec
> [  5]   5.00-6.00   sec   362 MBytes  3.04 Gbits/sec
> [  5]   6.00-7.00   sec   381 MBytes  3.19 Gbits/sec
> [  5]   7.00-8.00   sec   384 MBytes  3.22 Gbits/sec
> [  5]   8.00-9.00   sec   374 MBytes  3.14 Gbits/sec
> [  5]   9.00-10.00  sec   366 MBytes  3.07 Gbits/sec
> [  5]  10.00-10.13  sec  56.4 MBytes  3.52 Gbits/sec
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID] Interval           Transfer     Bandwidth
> [  5]   0.00-10.13  sec  0.00 Bytes  0.00 bits/sec sender
> [  5]   0.00-10.13  sec  3.66 GBytes  3.10 Gbits/sec
> receiver
>
> Client, Freebsd 12:
>    5] local 192.168.1.100 port 17917 connected to 192.168.1.155 port 5201
> [ ID] Interval           Transfer     Bitrate         Retr  Cwnd
> [  5]   0.00-1.00   sec   387 MBytes  3.25 Gbits/sec   83    309 KBytes
> [  5]   1.00-2.00   sec   380 MBytes  3.19 Gbits/sec   88   11.4 KBytes
> [  5]   2.00-3.00   sec   365 MBytes  3.06 Gbits/sec   83   54.2 KBytes
> [  5]   3.00-4.00   sec   372 MBytes  3.12 Gbits/sec   86    322 KBytes
> [  5]   4.00-5.00   sec   370 MBytes  3.11 Gbits/sec   83   2.85 KBytes
> [  5]   5.00-6.00   sec   366 MBytes  3.07 Gbits/sec   84    298 KBytes
> [  5]   6.00-7.00   sec   382 MBytes  3.20 Gbits/sec   87    356 KBytes
> [  5]   7.00-8.00   sec   380 MBytes  3.19 Gbits/sec   87    369 KBytes
> [  5]   8.00-9.00   sec   374 MBytes  3.14 Gbits/sec   87   29.9 KBytes
> [  5]   9.00-10.00  sec   375 MBytes  3.15 Gbits/sec   86    419 KBytes
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID] Interval           Transfer     Bitrate         Retr
> [  5]   0.00-10.00  sec  3.66 GBytes  3.15 Gbits/sec 854 sender
> [  5]   0.00-10.00  sec  3.66 GBytes  3.15 Gbits/sec
> receiver
>
>
> ==========================
> test 4, Centos 7.6 -> Centos 7.6
> ==========================
>
> Server:
> Accepted connection from 192.168.1.162, port 47606
> [  5] local 192.168.1.155 port 5201 connected to 192.168.1.162 port 47608
> [ ID] Interval           Transfer     Bandwidth
> [  5]   0.00-1.00   sec  1.60 GBytes  13.7 Gbits/sec
> [  5]   1.00-2.00   sec  1.59 GBytes  13.7 Gbits/sec
> [  5]   2.00-3.00   sec  1.60 GBytes  13.8 Gbits/sec
> [  5]   3.00-4.00   sec  1.71 GBytes  14.7 Gbits/sec
> [  5]   4.00-5.00   sec  1.49 GBytes  12.8 Gbits/sec
> [  5]   5.00-6.00   sec  1.38 GBytes  11.9 Gbits/sec
> [  5]   6.00-7.00   sec  1.52 GBytes  13.1 Gbits/sec
> [  5]   7.00-8.00   sec  1.51 GBytes  13.0 Gbits/sec
> [  5]   8.00-9.00   sec  1.51 GBytes  13.0 Gbits/sec
> [  5]   9.00-10.00  sec  1.51 GBytes  12.9 Gbits/sec
> [  5]  10.00-10.03  sec  48.1 MBytes  11.9 Gbits/sec
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID] Interval           Transfer     Bandwidth
> [  5]   0.00-10.03  sec  0.00 Bytes  0.00 bits/sec sender
> [  5]   0.00-10.03  sec  15.5 GBytes  13.2 Gbits/sec
> receiver
>
> Client, Centos 7.6
> Connecting to host 192.168.1.155, port 5201
> [  4] local 192.168.1.162 port 47608 connected to 192.168.1.155 port 5201
> [ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
> [  4]   0.00-1.00   sec  1.66 GBytes  14.3 Gbits/sec    0   1.41 MBytes
> [  4]   1.00-2.00   sec  1.58 GBytes  13.5 Gbits/sec    0   1.46 MBytes
> [  4]   2.00-3.00   sec  1.61 GBytes  13.9 Gbits/sec    0   1.91 MBytes
> [  4]   3.00-4.00   sec  1.71 GBytes  14.7 Gbits/sec    0   2.13 MBytes
> [  4]   4.00-5.00   sec  1.47 GBytes  12.6 Gbits/sec    0   2.49 MBytes
> [  4]   5.00-6.00   sec  1.39 GBytes  11.9 Gbits/sec    0   2.56 MBytes
> [  4]   6.00-7.00   sec  1.53 GBytes  13.1 Gbits/sec    0   2.71 MBytes
> [  4]   7.00-8.00   sec  1.50 GBytes  12.9 Gbits/sec    0   2.73 MBytes
> [  4]   8.00-9.00   sec  1.50 GBytes  12.9 Gbits/sec    0   2.81 MBytes
> [  4]   9.00-10.00  sec  1.51 GBytes  12.9 Gbits/sec    0   2.84 MBytes
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID] Interval           Transfer     Bandwidth       Retr
> [  4]   0.00-10.00  sec  15.5 GBytes  13.3 Gbits/sec 0 sender
> [  4]   0.00-10.00  sec  15.5 GBytes  13.3 Gbits/sec
> receiver
>
>
> ===============================
> test4, Opensuse 15.1 -> OpenSuse 15.1
> ================================
>
> Server, Opensuse 15.1:
> [  5] local 192.168.1.158 port 5201 connected to 192.168.1.163 port 39334
> [ ID] Interval           Transfer     Bitrate
> [  5]   0.00-1.00   sec  1.51 GBytes  13.0 Gbits/sec
> [  5]   1.00-2.00   sec  1.61 GBytes  13.8 Gbits/sec
> [  5]   2.00-3.00   sec  1.58 GBytes  13.6 Gbits/sec
> [  5]   3.00-4.00   sec  1.58 GBytes  13.6 Gbits/sec
> [  5]   4.00-5.00   sec  1.56 GBytes  13.4 Gbits/sec
> [  5]   5.00-6.00   sec  1.60 GBytes  13.7 Gbits/sec
> [  5]   6.00-7.00   sec  1.58 GBytes  13.5 Gbits/sec
> [  5]   7.00-8.00   sec  1.60 GBytes  13.8 Gbits/sec
> [  5]   8.00-9.00   sec  1.60 GBytes  13.7 Gbits/sec
> [  5]   9.00-10.00  sec  1.61 GBytes  13.8 Gbits/sec
> [  5]  10.00-10.04  sec  68.0 MBytes  14.2 Gbits/sec
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID] Interval           Transfer     Bitrate
> [  5]   0.00-10.04  sec  15.9 GBytes  13.6 Gbits/sec
> receiver
>
> Client, OpenSuse 15.1:
> [  5] local 192.168.1.163 port 39334 connected to 192.168.1.158 port 5201
> [ ID] Interval           Transfer     Bitrate         Retr  Cwnd
> [  5]   0.00-1.00   sec  1.58 GBytes  13.6 Gbits/sec    0    723 KBytes
> [  5]   1.00-2.00   sec  1.61 GBytes  13.8 Gbits/sec    0    805 KBytes
> [  5]   2.00-3.00   sec  1.57 GBytes  13.5 Gbits/sec    0    926 KBytes
> [  5]   3.00-4.00   sec  1.58 GBytes  13.6 Gbits/sec    0    926 KBytes
> [  5]   4.00-5.00   sec  1.56 GBytes  13.4 Gbits/sec    0    970 KBytes
> [  5]   5.00-6.00   sec  1.60 GBytes  13.7 Gbits/sec    0   1.05 MBytes
> [  5]   6.00-7.00   sec  1.57 GBytes  13.5 Gbits/sec    0   1.05 MBytes
> [  5]   7.00-8.00   sec  1.61 GBytes  13.8 Gbits/sec    0   1.05 MBytes
> [  5]   8.00-9.00   sec  1.59 GBytes  13.7 Gbits/sec    0   1.05 MBytes
> [  5]   9.00-10.00  sec  1.62 GBytes  13.9 Gbits/sec    0   1.05 MBytes
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID] Interval           Transfer     Bitrate         Retr
> [  5]   0.00-10.00  sec  15.9 GBytes  13.7 Gbits/sec 0 sender
> [  5]   0.00-10.04  sec  15.9 GBytes  13.6 Gbits/sec
> receiver
>
>
>
>
>
>
>
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer

Roger Pau Monné
On Mon, Jun 24, 2019 at 10:24:09PM +0200, Christian M wrote:
> Thanks for your input Bjarne.
>
> Previously I tested with iperf2, but I have made som tests with iperf3 now
> also and noticed there are a lot of "Retr" (TCP retries) in some cases.
> Went back here to your post and saw that your results also showed a lot of
> retries in some cases. My new tests showed similar results to yours
> (although not nearly as hi throughput as you have). When Linux is client
> the retries are 0, with FreeBSD as client the number is > 0.

Can you figure out what caused those retries? Wrong MTU, bad
checksums?

> I'm not sure
> if our numbers are considered high though, and something that actually is a
> problem? Although, on a internal network with no external factors like
> interference, I feel this should always be 0 no matter what?

If properly configured yes, there should be no retries when doing
intra-VM connections.

Thanks, Roger.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer

Christian Marcos
I've made two tests while running tcpdump on the xcp-ng host. I'm not at
all qualified to interpret the .pcap files from tcpdump, but I've put them
on Google Drive and linked them below the two tests. Perhaps someone more
qualified could have a look for anything useful in there. Please note the
extremely uneven throughput for test 2 below. It's almost like the
throughput increased when running tcpdump simultaneously.

Host: XCP-ng 7.6.0
Network: Private Network on host, not connected to any PIF.
VM1: 12.0-RELEASE (1 VIF, 172.31.16.125)
VM2: 12.0-RELEASE (1 VIF, 172.31.15.126)

On the host I listen with tcpdump on the VIF for VM1 in both tests.

VM1 as client:

On XCP-ng: tcpdump -i vif42.0 -s 0 -w xcp-ng-vm1-client.pcap
xcp-ng-vm1-client.pcap (80M):
https://drive.google.com/open?id=1eR3fetvKRz3vFSXCxDKuJYFrQ3wLqjrU
On VM1: iperf3 -c 172.31.16.126
On VM2: iperf3 -s

VM1 iperf3 output:
Connecting to host 172.31.16.126, port 5201
[  5] local 172.31.16.125 port 18182 connected to 172.31.16.126 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  8.00 MBytes  67.1 Mbits/sec  156   15.6 KBytes
[  5]   1.00-2.00   sec  14.1 MBytes   118 Mbits/sec  318   1.43 KBytes
[  5]   2.00-3.00   sec  7.51 MBytes  63.1 Mbits/sec  218   1.43 KBytes
[  5]   3.00-4.00   sec  8.29 MBytes  69.3 Mbits/sec  193   1.43 KBytes
[  5]   4.00-5.01   sec  10.7 MBytes  89.4 Mbits/sec  252   1.43 KBytes
[  5]   5.01-6.00   sec  13.6 MBytes   115 Mbits/sec  313   31.3 KBytes
[  5]   6.00-7.00   sec  8.41 MBytes  70.2 Mbits/sec  309   1.43 KBytes
[  5]   7.00-8.01   sec  12.6 MBytes   106 Mbits/sec  223   1.43 KBytes
[  5]   8.01-9.00   sec  12.5 MBytes   106 Mbits/sec  227   1.43 KBytes
[  5]   9.00-10.00  sec  13.5 MBytes   113 Mbits/sec  263   1.43 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   109 MBytes  91.6 Mbits/sec  2472
sender
[  5]   0.00-10.01  sec   109 MBytes  91.3 Mbits/sec
 receiver


VM1 as server:

On XCP-ng: tcpdump -i vif42.0 -s 0 -w xcp-ng-vm1-server.pcap
xcp-ng-vm1-server.pcap: (337M):
https://drive.google.com/open?id=1N3-GccXaBc6hlzFrgshi2vhcCt8vekxF
On VM1: iperf3 -s
On VM2: iperf3 -c 172.31.16.125

VM1 iperf3 output:
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 172.31.16.126, port 31523
[  5] local 172.31.16.125 port 5201 connected to 172.31.16.126 port 34605
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  10.9 MBytes  91.6 Mbits/sec
[  5]   1.00-2.00   sec  14.1 MBytes   118 Mbits/sec
[  5]   2.00-3.01   sec  4.92 MBytes  41.0 Mbits/sec
[  5]   3.01-4.00   sec  10.4 MBytes  87.9 Mbits/sec
[  5]   4.00-5.00   sec  8.52 MBytes  71.2 Mbits/sec
[  5]   5.00-6.00   sec  22.0 MBytes   185 Mbits/sec
[  5]   6.00-7.00   sec  9.25 MBytes  77.8 Mbits/sec
[  5]   7.00-8.00   sec  70.8 MBytes   594 Mbits/sec
[  5]   8.00-9.00   sec  89.5 MBytes   752 Mbits/sec
[  5]   9.00-10.00  sec   105 MBytes   884 Mbits/sec
[  5]  10.00-10.00  sec   180 KBytes  1.19 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec   346 MBytes   290 Mbits/sec
 receiver



Den tis 25 juni 2019 kl 10:21 skrev Roger Pau Monné <[hidden email]>:

> On Mon, Jun 24, 2019 at 10:24:09PM +0200, Christian M wrote:
> > Thanks for your input Bjarne.
> >
> > Previously I tested with iperf2, but I have made som tests with iperf3
> now
> > also and noticed there are a lot of "Retr" (TCP retries) in some cases.
> > Went back here to your post and saw that your results also showed a lot
> of
> > retries in some cases. My new tests showed similar results to yours
> > (although not nearly as hi throughput as you have). When Linux is client
> > the retries are 0, with FreeBSD as client the number is > 0.
>
> Can you figure out what caused those retries? Wrong MTU, bad
> checksums?
>
> > I'm not sure
> > if our numbers are considered high though, and something that actually
> is a
> > problem? Although, on a internal network with no external factors like
> > interference, I feel this should always be 0 no matter what?
>
> If properly configured yes, there should be no retries when doing
> intra-VM connections.
>
> Thanks, Roger.
>
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer

Eugene Grosbein-10
25.06.2019 18:55, Christian M wrote:

> Please note the
> extremely uneven throughput for test 2 below. It's almost like the
> throughput increased when running tcpdump simultaneously.

Generally this indicates broken "hardware" offload for some features
like txcsum/rxcsum/vlan_hwcsum/vlan_hwtagging etc.

Some of them may be automatically disabled while interface is put in promiscuous mode
by running tcpdump and enabled back after tcpdump terminated.

You should look at "options" line in the output of ifconfig for the interface
and manually disable offloads (look for ifconfig(8) manual for details).
They retry tests without tcpdump running.

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer

Christian Marcos
I've disabled them all on both VM's with FreeBSD 12.0-RELEASE (ifconfig xn0
-txcsum -rxcsum -lro -tso), and also tried disabling everything for the VIF
via XCP-ng earlier. It made no difference unfortunately.

Den tis 25 juni 2019 kl 14:08 skrev Eugene Grosbein <[hidden email]>:

> 25.06.2019 18:55, Christian M wrote:
>
> > Please note the
> > extremely uneven throughput for test 2 below. It's almost like the
> > throughput increased when running tcpdump simultaneously.
>
> Generally this indicates broken "hardware" offload for some features
> like txcsum/rxcsum/vlan_hwcsum/vlan_hwtagging etc.
>
> Some of them may be automatically disabled while interface is put in
> promiscuous mode
> by running tcpdump and enabled back after tcpdump terminated.
>
> You should look at "options" line in the output of ifconfig for the
> interface
> and manually disable offloads (look for ifconfig(8) manual for details).
> They retry tests without tcpdump running.
>
>
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer

Roger Pau Monné
In reply to this post by Christian Marcos
On Tue, Jun 25, 2019 at 01:55:40PM +0200, Christian M wrote:

> I've made two tests while running tcpdump on the xcp-ng host. I'm not at
> all qualified to interpret the .pcap files from tcpdump, but I've put them
> on Google Drive and linked them below the two tests. Perhaps someone more
> qualified could have a look for anything useful in there. Please note the
> extremely uneven throughput for test 2 below. It's almost like the
> throughput increased when running tcpdump simultaneously.
>
> Host: XCP-ng 7.6.0
> Network: Private Network on host, not connected to any PIF.
> VM1: 12.0-RELEASE (1 VIF, 172.31.16.125)
> VM2: 12.0-RELEASE (1 VIF, 172.31.15.126)
>
> On the host I listen with tcpdump on the VIF for VM1 in both tests.
>
> VM1 as client:
>
> On XCP-ng: tcpdump -i vif42.0 -s 0 -w xcp-ng-vm1-client.pcap

Can you check the capabilities of vif42.0? (ie: whether csum
offloading is actually disabled on the host?)

> xcp-ng-vm1-client.pcap (80M):
> https://drive.google.com/open?id=1eR3fetvKRz3vFSXCxDKuJYFrQ3wLqjrU
> On VM1: iperf3 -c 172.31.16.126
> On VM2: iperf3 -s

I've taken a look at the dump and the checksum is wrong (or maybe
missing) for all? packets.

Packets with source 172.31.16.125 all have the TCP checksum set to
0x7f80 and all packets with source 172.31.16.125 have the TCP checksum
set to 0x7960.

FTR, can you also paste the ifconfig output of each interface before
running iperf?

Thanks, Roger.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer

Roger Pau Monné
There's a mistake in my reply below.

On Tue, Jun 25, 2019 at 04:56:43PM +0200, Roger Pau Monné wrote:

> On Tue, Jun 25, 2019 at 01:55:40PM +0200, Christian M wrote:
> > I've made two tests while running tcpdump on the xcp-ng host. I'm not at
> > all qualified to interpret the .pcap files from tcpdump, but I've put them
> > on Google Drive and linked them below the two tests. Perhaps someone more
> > qualified could have a look for anything useful in there. Please note the
> > extremely uneven throughput for test 2 below. It's almost like the
> > throughput increased when running tcpdump simultaneously.
> >
> > Host: XCP-ng 7.6.0
> > Network: Private Network on host, not connected to any PIF.
> > VM1: 12.0-RELEASE (1 VIF, 172.31.16.125)
> > VM2: 12.0-RELEASE (1 VIF, 172.31.15.126)
> >
> > On the host I listen with tcpdump on the VIF for VM1 in both tests.
> >
> > VM1 as client:
> >
> > On XCP-ng: tcpdump -i vif42.0 -s 0 -w xcp-ng-vm1-client.pcap
>
> Can you check the capabilities of vif42.0? (ie: whether csum
> offloading is actually disabled on the host?)
>
> > xcp-ng-vm1-client.pcap (80M):
> > https://drive.google.com/open?id=1eR3fetvKRz3vFSXCxDKuJYFrQ3wLqjrU
> > On VM1: iperf3 -c 172.31.16.126
> > On VM2: iperf3 -s
>
> I've taken a look at the dump and the checksum is wrong (or maybe
> missing) for all? packets.
>
> Packets with source 172.31.16.125 all have the TCP checksum set to
> 0x7f80 and all packets with source 172.31.16.125 have the TCP checksum
                                     ^ 172.31.16.126
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer

freebsd-xen mailing list
In reply to this post by Christian Marcos
oh yes, sorry I am using:

0:~ # xl info
host                   : x0
release                : 4.12.14-lp150.12.64-default
version                : #1 SMP Mon Jun 17 16:53:50 UTC 2019 (3edfd41)
machine                : x86_64
nr_cpus                : 4
max_cpu_id             : 3
nr_nodes               : 1
cores_per_socket       : 4
threads_per_core       : 1
cpu_mhz                : 3192
hw_caps                : bfebfbff:77faf3bf:2c100800:00000021:00000001:000027ab:00000000:00000100
virt_caps              : hvm hvm_directio
total_memory           : 32461
free_memory            : 5319
sharing_freed_memory   : 0
sharing_used_memory    : 0
outstanding_claims     : 0
free_cpus              : 0
xen_major              : 4
xen_minor              : 10
xen_extra              : .3_04-lp150.2.1
xen_version            : 4.10.3_04-lp150.2.1
xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler          : credit
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          :
xen_commandline        : dom0_mem=3072M,max:3072M
cc_compiler            : gcc (SUSE Linux) 7.4.0
cc_compile_by          : abuild
cc_compile_domain      : suse.de
cc_compile_date        : Thu May  9 16:10:08 UTC 2019
build_id               : bf77d035ff05eb55d9616e418f73cbcbd96f747b
xend_config_format     : 4


Since you peaked my interest, I tryed playing around a little.
Initially to day, the retransmits where between 4-15   retransmits. I then changed the network
interface from hypervisor default to  E1000 (still xn0 in freebsd) and the retransmits jumped up to
700 to 800!
After changing the network interface back to hypervisor default, the number of retranmits remains
high in the 3-digit area.  Fooling around with mtu size did not change anything.
Something is not quite right.



On 6/24/19 10:24 PM, Christian M wrote:

> Thanks for your input Bjarne.
>
> Previously I tested with iperf2, but I have made som tests with iperf3 now
> also and noticed there are a lot of "Retr" (TCP retries) in some cases.
> Went back here to your post and saw that your results also showed a lot of
> retries in some cases. My new tests showed similar results to yours
> (although not nearly as hi throughput as you have). When Linux is client
> the retries are 0, with FreeBSD as client the number is > 0. I'm not sure
> if our numbers are considered high though, and something that actually is a
> problem? Although, on a internal network with no external factors like
> interference, I feel this should always be 0 no matter what?
>
> What hypervisor were you running in your tests? Version?
>
> Regards,
> Christian
>
>
> Den mån 24 juni 2019 kl 17:56 skrev Bjarne <[hidden email]>:
>
>> Well, to add some datapoints,  I just did a quick test om my private
>> homeserver, testing with iperf3
>> -s  / iperf3 -c <server-ip>
>>
>> I have not much knowledge of iperf3 network test and it's validity, but
>> there is definitely some
>> differences to be seen.
>>
>> Summary:
>>
>> FBSD <-> FBSD   3,5Gbit/sec
>> FBSD -> Centos:  3GBit/Sec
>> Centos -> FBSD: 12 Gbit/sec
>> :snip
>>
>>
>>

--
mvh,
Bjarne
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer

Eugene Grosbein-10
In reply to this post by Christian Marcos
25.06.2019 19:40, Christian M wrote:

> I've disabled them all on both VM's with FreeBSD 12.0-RELEASE (ifconfig xn0
> -txcsum -rxcsum -lro -tso), and also tried disabling everything for the VIF
> via XCP-ng earlier. It made no difference unfortunately.

Then you should try to bring the interface to promiscuous mode manually
with "ifconfig xn0 promisc" any retry. If this does not make a difference,
then you may be facing some timekeeping problem as tcpdump makes additional
CPU load on the system that can force scheduler behaviour to change.

As next step, you should check your kern.timecounter.* and kern.eventtimer.* sysctls.
If default value of kern.eventtimer.periodic is 0 (it depends on detected "hardware"),
you should change it kern.eventtimer.periodic=1 and retry the test.

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer

Christian Marcos
checksum offloading doe not seem to have any affect when done on the VM
VIF's. Both 12.0-RELEASE VIF's now have:

other-config (MRW): ethtool-rx: off; ethtool-tx: off; ethtool-sg: off;
ethtool-tso: off; ethtool-ufo: off; ethtool-gso: off

ethtool -k vif54.0
Features for vif54.0:
rx-checksumming: on [fixed]
tx-checksumming: off
tx-checksum-ipv4: off
tx-checksum-ip-generic: off [fixed]
tx-checksum-ipv6: off
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: off [fixed]
scatter-gather: off
tx-scatter-gather: off
tx-scatter-gather-fraglist: off
tcp-segmentation-offload: off
tx-tcp-segmentation: off
tx-tcp-ecn-segmentation: off [fixed]
tx-tcp6-segmentation: off
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: off
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: off [fixed]
tx-vlan-offload: off [fixed]
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: off [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-ipip-segmentation: off [fixed]
tx-sit-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
busy-poll: off [fixed]

Both vif's look the same as above.

Running tcpdump on the host while running iperf3 between the 12.0 VM's
results in a lot of incorrect cksum like this.

tcpdump -i vif54.0 -v -nn| grep -i incorrect
172.31.16.125.63013 > 172.31.16.126.5201: Flags [.], cksum 0x7f08
(incorrect -> 0x030f), seq 223153882:223155330, ack 1, win 1026, options
[nop,nop,TS val 4104002274 ecr 3926764642], length 1448
...

Also noticed that iperf3 does not work to run every time, more than 50% of
the tries with iperf3 -c <ip> I get:
Connecting to host 172.31.16.126, port 5201
iperf3: error - unable to initialize stream: Socket is not connected

Or nothing happens and I need to abort, then restart iperf3 -s on the other
VM, and retry.

The results from iperf3 are really inconsistent still, and I cant get a
good feel of any pattern. Currently as I type this, a result looks like
this:

Host: tcpdump running
VIF's: no promisc
xn0 on both VM's: no promisc

[  5] local 172.31.16.125 port 47738 connected to 172.31.16.126 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  37.8 MBytes   316 Mbits/sec  176   1.43 KBytes
[  5]   1.00-2.00   sec   133 MBytes  1.11 Gbits/sec  1022   49.8 KBytes
[  5]   2.00-3.00   sec   179 MBytes  1.50 Gbits/sec  784    191 KBytes
[  5]   3.00-4.00   sec   132 MBytes  1.11 Gbits/sec  201    345 KBytes
[  5]   4.00-5.00   sec  95.3 MBytes   799 Mbits/sec  438    180 KBytes
[  5]   5.00-6.00   sec   126 MBytes  1.06 Gbits/sec  1256    353 KBytes
[  5]   6.00-7.00   sec   144 MBytes  1.20 Gbits/sec  1242    275 KBytes
[  5]   7.00-8.00   sec  94.5 MBytes   792 Mbits/sec  593    167 KBytes
[  5]   8.00-9.00   sec   125 MBytes  1.05 Gbits/sec  326    169 KBytes
[  5]   9.00-10.00  sec  57.0 MBytes   478 Mbits/sec  1013   1.43 KBytes
[  5]  10.00-11.00  sec  31.3 MBytes   262 Mbits/sec  463   1.43 KBytes
[  5]  11.00-12.00  sec  42.3 MBytes   355 Mbits/sec  544   78.3 KBytes
[  5]  12.00-13.00  sec  41.3 MBytes   346 Mbits/sec  624   1.43 KBytes
[  5]  13.00-14.00  sec  52.7 MBytes   442 Mbits/sec  739   1.43 KBytes
[  5]  14.00-15.00  sec  20.4 MBytes   171 Mbits/sec  384   1.43 KBytes
[  5]  15.00-16.00  sec  74.5 MBytes   625 Mbits/sec  489    299 KBytes
[  5]  16.00-17.00  sec  40.1 MBytes   337 Mbits/sec  835   1.43 KBytes
[  5]  17.00-18.00  sec  43.2 MBytes   363 Mbits/sec  372   1.43 KBytes
[  5]  18.00-19.00  sec  58.9 MBytes   494 Mbits/sec  723   1.43 KBytes
[  5]  19.00-20.00  sec  21.2 MBytes   178 Mbits/sec  300   1.43 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-20.00  sec  1.51 GBytes   649 Mbits/sec  12524
sender
[  5]   0.00-20.00  sec  1.51 GBytes   649 Mbits/sec
 receiver

But please be sure, this can change dramatically without making any changes
what so ever. Offloading is not turned off in FreeBSD (but on the VIF's as
stated before), and tcpdump is running on the host.

xn0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=503<RXCSUM,TXCSUM,TSO4,LRO>
ether 12:ec:6a:5c:af:df
inet 172.31.16.125 netmask 0xffffff00 broadcast 172.31.16.255
media: Ethernet manual
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

xn0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=503<RXCSUM,TXCSUM,TSO4,LRO>
ether aa:4f:f7:4c:a0:e6
inet 172.31.16.126 netmask 0xffffff00 broadcast 172.31.16.255
media: Ethernet manual
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

Absolutely nothing changed this time (restarted iperf3 -s though since I
ran ifconfig do get the info above), tcpdump still running on host (never
even aborted):

Host: tcpdump still running
VIF's: no promisc
xn0 on both VM's: no promisc

[  5] local 172.31.16.125 port 30543 connected to 172.31.16.126 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  10.3 MBytes  86.8 Mbits/sec  165   38.4 KBytes
[  5]   1.00-2.00   sec  39.7 MBytes   333 Mbits/sec  175    167 KBytes
[  5]   2.00-3.00   sec  40.0 MBytes   335 Mbits/sec  437   54.1 KBytes
[  5]   3.00-4.00   sec  11.2 MBytes  93.6 Mbits/sec  320   1.43 KBytes
[  5]   4.00-5.00   sec  11.2 MBytes  94.2 Mbits/sec  305   1.43 KBytes
[  5]   5.00-6.00   sec  7.63 MBytes  64.1 Mbits/sec  179   2.84 KBytes
[  5]   6.00-7.00   sec  6.77 MBytes  56.7 Mbits/sec  191   1.43 KBytes
[  5]   7.00-8.00   sec  4.26 MBytes  35.9 Mbits/sec  145   21.3 KBytes
[  5]   8.00-9.00   sec  5.38 MBytes  45.1 Mbits/sec  176   1.43 KBytes
[  5]   9.00-10.00  sec  8.24 MBytes  69.1 Mbits/sec  225   1.43 KBytes
[  5]  10.00-11.00  sec  16.1 MBytes   135 Mbits/sec  422   1.43 KBytes
[  5]  11.00-12.00  sec  3.76 MBytes  31.5 Mbits/sec  113   1.43 KBytes
[  5]  12.00-13.00  sec  6.62 MBytes  55.5 Mbits/sec  161   1.43 KBytes
[  5]  13.00-14.00  sec  5.28 MBytes  44.3 Mbits/sec  156   18.4 KBytes
[  5]  14.00-15.00  sec  5.21 MBytes  43.7 Mbits/sec  146   1.43 KBytes
[  5]  15.00-16.00  sec  6.37 MBytes  53.4 Mbits/sec  199   1.43 KBytes
[  5]  16.00-17.00  sec  10.7 MBytes  89.5 Mbits/sec  193   1.43 KBytes
[  5]  17.00-18.00  sec  9.49 MBytes  79.6 Mbits/sec  390   1.43 KBytes
[  5]  18.00-19.00  sec  5.36 MBytes  44.9 Mbits/sec  125   1.43 KBytes
[  5]  19.00-20.00  sec  3.65 MBytes  30.6 Mbits/sec   58   1.43 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-20.00  sec   217 MBytes  91.1 Mbits/sec  4281
sender
[  5]   0.00-20.00  sec   217 MBytes  90.8 Mbits/sec
 receiver

Host: tcpdump still running
VIF's: promisc
xn0 on both VM's: promisc

Connecting to host 172.31.16.126, port 5201
[  5] local 172.31.16.125 port 61196 connected to 172.31.16.126 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  25.9 MBytes   217 Mbits/sec  271   90.5 KBytes
[  5]   1.00-2.00   sec  56.4 MBytes   473 Mbits/sec   94    255 KBytes
[  5]   2.00-3.00   sec  83.4 MBytes   699 Mbits/sec  872    201 KBytes
[  5]   3.00-4.00   sec  76.4 MBytes   641 Mbits/sec  684    176 KBytes
[  5]   4.00-5.00   sec  80.4 MBytes   674 Mbits/sec  742   1.43 KBytes
[  5]   5.00-6.00   sec  81.4 MBytes   684 Mbits/sec  248    333 KBytes
[  5]   6.00-7.00   sec  86.1 MBytes   722 Mbits/sec  826    160 KBytes
[  5]   7.00-8.00   sec  77.5 MBytes   650 Mbits/sec  209    253 KBytes
[  5]   8.00-9.00   sec  9.64 MBytes  80.9 Mbits/sec  433   12.8 KBytes
[  5]   9.00-10.00  sec  8.31 MBytes  69.6 Mbits/sec  210   1.43 KBytes
[  5]  10.00-11.00  sec  8.64 MBytes  72.5 Mbits/sec  199   1.43 KBytes
[  5]  11.00-12.00  sec  7.02 MBytes  58.9 Mbits/sec  240   1.43 KBytes
[  5]  12.00-13.00  sec  20.1 MBytes   169 Mbits/sec  579   1.43 KBytes
[  5]  13.00-14.00  sec  10.8 MBytes  90.4 Mbits/sec  277   22.7 KBytes
[  5]  14.00-15.00  sec  2.75 MBytes  23.1 Mbits/sec  107   1.43 KBytes
[  5]  15.00-16.00  sec  8.79 MBytes  73.8 Mbits/sec  203   12.8 KBytes
[  5]  16.00-17.00  sec  9.23 MBytes  77.3 Mbits/sec  158   1.43 KBytes
[  5]  17.00-18.00  sec  13.5 MBytes   114 Mbits/sec  381   1.43 KBytes
[  5]  18.00-19.00  sec  41.6 MBytes   349 Mbits/sec  331   99.5 KBytes
[  5]  19.00-20.00  sec  40.6 MBytes   340 Mbits/sec   70   1.43 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-20.00  sec   749 MBytes   314 Mbits/sec  7134
sender
[  5]   0.00-20.00  sec   748 MBytes   314 Mbits/sec
 receiver

Host: tcpdump NOT running
VIF's: no promisc
xn0: no promisc

Run1:
Connecting to host 172.31.16.126, port 5201
[  5] local 172.31.16.125 port 20600 connected to 172.31.16.126 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  28.4 MBytes   238 Mbits/sec  451   1.43 KBytes
[  5]   1.00-2.00   sec  21.5 MBytes   181 Mbits/sec  546   2.84 KBytes
[  5]   2.00-3.00   sec  26.7 MBytes   224 Mbits/sec  595   1.43 KBytes
[  5]   3.00-4.00   sec  15.9 MBytes   133 Mbits/sec  347   1.43 KBytes
[  5]   4.00-5.00   sec  4.45 MBytes  37.4 Mbits/sec  187   2.84 KBytes
[  5]   5.00-6.00   sec  15.4 MBytes   129 Mbits/sec  239   66.5 KBytes
[  5]   6.00-7.00   sec  10.6 MBytes  89.1 Mbits/sec  366   1.43 KBytes
[  5]   7.00-8.00   sec  30.8 MBytes   258 Mbits/sec  510   1.43 KBytes
[  5]   8.00-9.00   sec  16.6 MBytes   140 Mbits/sec  332   2.84 KBytes
[  5]   9.00-10.00  sec  12.4 MBytes   104 Mbits/sec  376   1.43 KBytes
[  5]  10.00-11.00  sec  11.3 MBytes  94.9 Mbits/sec  223   1.43 KBytes
[  5]  11.00-12.00  sec  9.00 MBytes  75.5 Mbits/sec  206   1.43 KBytes
[  5]  12.00-13.00  sec  8.95 MBytes  75.1 Mbits/sec  201   1.43 KBytes
[  5]  13.00-14.00  sec  12.3 MBytes   103 Mbits/sec  179   1.43 KBytes
[  5]  14.00-15.00  sec  20.5 MBytes   172 Mbits/sec  445   1.43 KBytes
[  5]  15.00-16.00  sec  3.92 MBytes  32.9 Mbits/sec  163   1.43 KBytes
[  5]  16.00-17.00  sec  6.82 MBytes  57.2 Mbits/sec  122   2.84 KBytes
[  5]  17.00-18.00  sec  14.3 MBytes   120 Mbits/sec  238   1.43 KBytes
[  5]  18.00-19.00  sec  14.2 MBytes   119 Mbits/sec  226   1.43 KBytes
[  5]  19.00-20.00  sec  12.9 MBytes   108 Mbits/sec  297   1.43 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-20.00  sec   297 MBytes   125 Mbits/sec  6249
sender
[  5]   0.00-20.00  sec   297 MBytes   124 Mbits/sec
 receiver

Run2 right after a iperf3 -s restart on second VM:
Connecting to host 172.31.16.126, port 5201
[  5] local 172.31.16.125 port 40622 connected to 172.31.16.126 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  6.49 MBytes  54.3 Mbits/sec  163   1.43 KBytes
[  5]   1.00-2.00   sec  9.93 MBytes  83.3 Mbits/sec  263   1.43 KBytes
[  5]   2.00-3.00   sec  7.71 MBytes  64.7 Mbits/sec  173   1.43 KBytes
[  5]   3.00-4.00   sec  5.58 MBytes  47.0 Mbits/sec   83   1.43 KBytes
[  5]   4.00-5.00   sec  14.8 MBytes   124 Mbits/sec  347   1.43 KBytes
[  5]   5.00-6.00   sec  16.8 MBytes   141 Mbits/sec  349   1.43 KBytes
[  5]   6.00-7.00   sec  19.2 MBytes   161 Mbits/sec  205   1.43 KBytes
[  5]   7.00-8.00   sec  6.39 MBytes  53.6 Mbits/sec   79   1.43 KBytes
[  5]   8.00-9.00   sec  7.19 MBytes  60.3 Mbits/sec  133   1.43 KBytes
[  5]   9.00-10.00  sec  18.2 MBytes   153 Mbits/sec  265   1.43 KBytes
[  5]  10.00-11.00  sec  11.0 MBytes  92.6 Mbits/sec  220   1.43 KBytes
[  5]  11.00-12.00  sec  7.93 MBytes  66.6 Mbits/sec  172   1.43 KBytes
[  5]  12.00-13.00  sec  10.1 MBytes  84.8 Mbits/sec  198   1.43 KBytes
[  5]  13.00-14.00  sec  7.92 MBytes  66.5 Mbits/sec  196   1.43 KBytes
[  5]  14.00-15.00  sec  7.63 MBytes  64.0 Mbits/sec  144   1.43 KBytes
[  5]  15.00-16.00  sec  8.70 MBytes  73.0 Mbits/sec  267   1.43 KBytes
[  5]  16.00-17.00  sec  16.0 MBytes   134 Mbits/sec  296   1.43 KBytes
[  5]  17.00-18.00  sec  10.9 MBytes  91.5 Mbits/sec  261   1.43 KBytes
[  5]  18.00-19.00  sec  23.0 MBytes   193 Mbits/sec  444   1.43 KBytes
[  5]  19.00-20.00  sec  8.86 MBytes  74.3 Mbits/sec  224   1.43 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-20.00  sec   224 MBytes  94.1 Mbits/sec  4482
sender
[  5]   0.00-20.00  sec   224 MBytes  93.9 Mbits/sec
 receiver

As suggested before, I have set kern.eventtimer.periodic: 1, the above
tests are done with that setting. I don't see any difference with it set or
not (kern.eventtimer.periodic: 0 was default btw).

There was also a suggestion to have a look at kern.timecouter.*, but I'm
not sure what to look for here:

kern.timecounter.tsc_shift: 1
kern.timecounter.smp_tsc_adjust: 0
kern.timecounter.smp_tsc: 0
kern.timecounter.invariant_tsc: 0
kern.timecounter.fast_gettime: 1
kern.timecounter.tick: 1
kern.timecounter.choice: XENTIMER(950) ACPI-fast(900) i8254(0) HPET(950)
TSC-low(-100) dummy(-1000000)
kern.timecounter.hardware: XENTIMER
kern.timecounter.alloweddeviation: 5
kern.timecounter.stepwarnings: 0
kern.timecounter.tc.XENTIMER.quality: 950
kern.timecounter.tc.XENTIMER.frequency: 1000000000
kern.timecounter.tc.XENTIMER.counter: 3517394506
kern.timecounter.tc.XENTIMER.mask: 4294967295
kern.timecounter.tc.ACPI-fast.quality: 900
kern.timecounter.tc.ACPI-fast.frequency: 3579545
kern.timecounter.tc.ACPI-fast.counter: 1749326798
kern.timecounter.tc.ACPI-fast.mask: 4294967295
kern.timecounter.tc.i8254.quality: 0
kern.timecounter.tc.i8254.frequency: 1193182
kern.timecounter.tc.i8254.counter: 58432
kern.timecounter.tc.i8254.mask: 65535
kern.timecounter.tc.HPET.quality: 950
kern.timecounter.tc.HPET.frequency: 62500000
kern.timecounter.tc.HPET.counter: 3606880145
kern.timecounter.tc.HPET.mask: 4294967295
kern.timecounter.tc.TSC-low.quality: -100
kern.timecounter.tc.TSC-low.frequency: 1200075192
kern.timecounter.tc.TSC-low.counter: 3248698599
kern.timecounter.tc.TSC-low.mask: 4294967295

Regards,
Christian

Den ons 26 juni 2019 kl 01:14 skrev Eugene Grosbein <[hidden email]>:

> 25.06.2019 19:40, Christian M wrote:
>
> > I've disabled them all on both VM's with FreeBSD 12.0-RELEASE (ifconfig
> xn0
> > -txcsum -rxcsum -lro -tso), and also tried disabling everything for the
> VIF
> > via XCP-ng earlier. It made no difference unfortunately.
>
> Then you should try to bring the interface to promiscuous mode manually
> with "ifconfig xn0 promisc" any retry. If this does not make a difference,
> then you may be facing some timekeeping problem as tcpdump makes additional
> CPU load on the system that can force scheduler behaviour to change.
>
> As next step, you should check your kern.timecounter.* and
> kern.eventtimer.* sysctls.
> If default value of kern.eventtimer.periodic is 0 (it depends on detected
> "hardware"),
> you should change it kern.eventtimer.periodic=1 and retry the test.
>
>
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer

Eugene Grosbein-10
26.06.2019 15:11, Christian M wrote:

> Running tcpdump on the host while running iperf3 between the 12.0 VM's
> results in a lot of incorrect cksum like this.
>
> tcpdump -i vif54.0 -v -nn| grep -i incorrect
> 172.31.16.125.63013 > 172.31.16.126.5201: Flags [.], cksum 0x7f08
> (incorrect -> 0x030f), seq 223153882:223155330, ack 1, win 1026, options
> [nop,nop,TS val 4104002274 ecr 3926764642], length 1448

Incorrect checksum shown by tcpdump is normal and expected for outgoing packets
if txcsum offload is not disabled because networking stack skips checksum calculation
offloading it to "hardware". It still indicates a bug at sending side if seen for outgoing packets
when txcsum is disabled.

Incorrect checksum shown by tcpdump for incoming packets always indicates a problem
at sending side or intermediate network (if any).

You should check output of "netstat -sp ip" and "netstat -sp tcp" for non-zero
and growing checksum error counters at receiving side.

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer

Christian Marcos
I ran ifconfig xn0 -txcsum on both test VM's, and all incorrect checksum
disappeared.

netstat -sp ip for  VM1 (172.31.16.125, running iperf3 -c always) and VM2
(172.31.16.126, running iperf3 -s always):

ip:
3664084 total packets received
0 bad header checksums
0 with size smaller than minimum
0 with data size < data length
0 with ip length > max ip packet size
0 with header length < data size
0 with data length < header length
0 with bad options
0 with incorrect version number
0 fragments received
0 fragments dropped (dup or out of space)
0 fragments dropped after timeout
0 packets reassembled ok
3664080 packets for this host
0 packets for unknown/unsupported protocol
0 packets forwarded (0 packets fast forwarded)
4 packets not forwardable
0 packets received for unknown multicast group
0 redirects sent
7003245 packets sent from this host
0 packets sent with fabricated ip header
0 output packets dropped due to no bufs, etc.
0 output packets discarded due to no route
0 output datagrams fragmented
0 fragments created
0 datagrams that can't be fragmented
0 tunneling packets that can't find gif
0 datagrams with bad address in header

ip:
8217827 total packets received
0 bad header checksums
0 with size smaller than minimum
0 with data size < data length
0 with ip length > max ip packet size
0 with header length < data size
0 with data length < header length
0 with bad options
0 with incorrect version number
0 fragments received
0 fragments dropped (dup or out of space)
0 fragments dropped after timeout
0 packets reassembled ok
8217827 packets for this host
0 packets for unknown/unsupported protocol
0 packets forwarded (0 packets fast forwarded)
0 packets not forwardable
0 packets received for unknown multicast group
0 redirects sent
4300027 packets sent from this host
0 packets sent with fabricated ip header
0 output packets dropped due to no bufs, etc.
0 output packets discarded due to no route
0 output datagrams fragmented
0 fragments created
0 datagrams that can't be fragmented
0 tunneling packets that can't find gif
0 datagrams with bad address in header

netstat -sp tcp for VM1 (172.31.16.125, running iperf3 -c always) and VM2
(172.31.16.126, running iperf3 -s always):

tcp:
7002523 packets sent
6866670 data packets (9939548357 bytes)
109393 data packets (101233068 bytes) retransmitted
0 data packets unnecessarily retransmitted
0 resends initiated by MTU discovery
27039 ack-only packets (90 delayed)
0 URG only packets
0 window probe packets
0 window update packets
125 control packets
3664099 packets received
3387212 acks (for 9939333715 bytes)
270936 duplicate acks
0 acks for unsent data
915 packets (36547 bytes) received in-sequence
4887 completely duplicate packets (208 bytes)
4853 old duplicate packets
0 packets with some dup. data (0 bytes duped)
0 out-of-order packets (0 bytes)
0 packets (0 bytes) of data after window
0 window probes
5694 window update packets
6 packets received after close
0 discarded for bad checksums
0 discarded for bad header offset fields
0 discarded because packet too short
0 discarded due to memory problems
88 connection requests
2 connection accepts
0 bad connection attempts
0 listen queue overflows
0 ignored RSTs in the windows
89 connections established (including accepts)
85 times used RTT from hostcache
85 times used RTT variance from hostcache
85 times used slow-start threshold from hostcache
90 connections closed (including 57 drops)
51 connections updated cached RTT on close
51 connections updated cached RTT variance on close
23 connections updated cached ssthresh on close
1 embryonic connection dropped
3387212 segments updated rtt (of 3313837 attempts)
0 retransmit timeouts
0 connections dropped by rexmit timeout
0 persist timeouts
0 connections dropped by persist timeout
0 Connections (fin_wait_2) dropped because of timeout
0 keepalive timeouts
0 keepalive probes sent
0 connections dropped by keepalive
759139 correct ACK header predictions
70 correct data packet header predictions
2 syncache entries added
0 retransmitted
0 dupsyn
0 dropped
2 completed
0 bucket overflow
0 cache overflow
0 reset
0 stale
0 aborted
0 badack
0 unreach
0 zone failures
2 cookies sent
0 cookies received
1 hostcache entry added
0 bucket overflow
5602 SACK recovery episodes
109329 segment rexmits in SACK recovery episodes
101663360 byte rexmits in SACK recovery episodes
249723 SACK options (SACK blocks) received
0 SACK options (SACK blocks) sent
0 SACK scoreboard overflow
0 packets with ECN CE bit set
0 packets with ECN ECT(0) bit set
0 packets with ECN ECT(1) bit set
0 successful ECN handshakes
0 times ECN reduced the congestion window
0 packets with matching signature received
0 packets with bad signature received
0 times failed to make signature due to no SA
0 times unexpected signature received
0 times no signature provided by segment
0 Path MTU discovery black hole detection activations
0 Path MTU discovery black hole detection min MSS activations
0 Path MTU discovery black hole detection failures
TCP connection count by state:
0 connections in CLOSED state
3 connections in LISTEN state
0 connections in SYN_SENT state
0 connections in SYN_RCVD state
2 connections in ESTABLISHED state
0 connections in CLOSE_WAIT state
0 connections in FIN_WAIT_1 state
0 connections in CLOSING state
0 connections in LAST_ACK state
0 connections in FIN_WAIT_2 state
0 connections in TIME_WAIT state

tcp:
4299563 packets sent
1510 data packets (132487 bytes)
0 data packets (0 bytes) retransmitted
0 data packets unnecessarily retransmitted
0 resends initiated by MTU discovery
4297568 ack-only packets (2610 delayed)
0 URG only packets
0 window probe packets
392 window update packets
93 control packets
8217839 packets received
1623 acks (for 132605 bytes)
150324 duplicate acks
0 acks for unsent data
7770336 packets (11249613126 bytes) received in-sequence
126189 completely duplicate packets (117973252 bytes)
4853 old duplicate packets
194 packets with some dup. data (59900 bytes duped)
290196 out-of-order packets (415026177 bytes)
0 packets (0 bytes) of data after window
0 window probes
0 window update packets
25 packets received after close
0 discarded for bad checksums
0 discarded for bad header offset fields
0 discarded because packet too short
0 discarded due to memory problems
6 connection requests
102 connection accepts
0 bad connection attempts
0 listen queue overflows
1 ignored RSTs in the window
108 connections established (including accepts)
95 times used RTT from hostcache
95 times used RTT variance from hostcache
0 times used slow-start threshold from hostcache
163 connections closed (including 47 drops)
29 connections updated cached RTT on close
29 connections updated cached RTT variance on close
0 connections updated cached ssthresh on close
0 embryonic connections dropped
1623 segments updated rtt (of 1314 attempts)
0 retransmit timeouts
0 connections dropped by rexmit timeout
0 persist timeouts
0 connections dropped by persist timeout
0 Connections (fin_wait_2) dropped because of timeout
9 keepalive timeouts
8 keepalive probes sent
1 connection dropped by keepalive
593 correct ACK header predictions
7769705 correct data packet header predictions
102 syncache entries added
0 retransmitted
0 dupsyn
0 dropped
102 completed
0 bucket overflow
0 cache overflow
0 reset
0 stale
0 aborted
0 badack
0 unreach
0 zone failures
102 cookies sent
0 cookies received
3 hostcache entries added
0 bucket overflow
0 SACK recovery episodes
0 segment rexmits in SACK recovery episodes
0 byte rexmits in SACK recovery episodes
0 SACK options (SACK blocks) received
293795 SACK options (SACK blocks) sent
0 SACK scoreboard overflow
0 packets with ECN CE bit set
0 packets with ECN ECT(0) bit set
0 packets with ECN ECT(1) bit set
0 successful ECN handshakes
0 times ECN reduced the congestion window
0 packets with matching signature received
0 packets with bad signature received
0 times failed to make signature due to no SA
0 times unexpected signature received
0 times no signature provided by segment
0 Path MTU discovery black hole detection activations
0 Path MTU discovery black hole detection min MSS activations
0 Path MTU discovery black hole detection failures
TCP connection count by state:
0 connections in CLOSED state
4 connections in LISTEN state
0 connections in SYN_SENT state
0 connections in SYN_RCVD state
2 connections in ESTABLISHED state
0 connections in CLOSE_WAIT state
0 connections in FIN_WAIT_1 state
0 connections in CLOSING state
0 connections in LAST_ACK state
0 connections in FIN_WAIT_2 state
0 connections in TIME_WAIT state

The data collected is after a number of runs with iperf3. I'm really not
qualified to interpret any of this, so my hopes are that someone else sees
something unusual here.

Thanks,
Christian



Den ons 26 juni 2019 kl 10:37 skrev Eugene Grosbein <[hidden email]>:

> 26.06.2019 15:11, Christian M wrote:
>
> > Running tcpdump on the host while running iperf3 between the 12.0 VM's
> > results in a lot of incorrect cksum like this.
> >
> > tcpdump -i vif54.0 -v -nn| grep -i incorrect
> > 172.31.16.125.63013 > 172.31.16.126.5201: Flags [.], cksum 0x7f08
> > (incorrect -> 0x030f), seq 223153882:223155330, ack 1, win 1026, options
> > [nop,nop,TS val 4104002274 ecr 3926764642], length 1448
>
> Incorrect checksum shown by tcpdump is normal and expected for outgoing
> packets
> if txcsum offload is not disabled because networking stack skips checksum
> calculation
> offloading it to "hardware". It still indicates a bug at sending side if
> seen for outgoing packets
> when txcsum is disabled.
>
> Incorrect checksum shown by tcpdump for incoming packets always indicates
> a problem
> at sending side or intermediate network (if any).
>
> You should check output of "netstat -sp ip" and "netstat -sp tcp" for
> non-zero
> and growing checksum error counters at receiving side.
>
>
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer

Christian Marcos
Sorry to say, but no. Nothing changed :(

The iperf issue concerns me a bit also:

root@:/usr/home/asdf # iperf3 -c 172.31.16.126
iperf3: error - unable to receive control message: Connection reset by peer
root@:/usr/home/asdf # iperf3 -c 172.31.16.126
Connecting to host 172.31.16.126, port 5201
iperf3: error - unable to initialize stream: Socket is not connected
root@:/usr/home/asdf # iperf3 -c 172.31.16.126
^C- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
iperf3: interrupt - the client has terminated
root@:/usr/home/asdf # iperf3 -c 172.31.16.126
Connecting to host 172.31.16.126, port 5201
[  5] local 172.31.16.125 port 23884 connected to 172.31.16.126 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  6.00 MBytes  50.2 Mbits/sec  217   1.43 KBytes
[  5]   1.00-2.00   sec  26.7 MBytes   224 Mbits/sec  566   1.43 KBytes
[  5]   2.00-3.00   sec  15.6 MBytes   131 Mbits/sec  445   1.43 KBytes
[  5]   3.00-4.00   sec  13.5 MBytes   113 Mbits/sec  453   1.43 KBytes
[  5]   4.00-5.00   sec  13.7 MBytes   115 Mbits/sec  512   1.43 KBytes
[  5]   5.00-6.00   sec  6.51 MBytes  54.6 Mbits/sec  201   1.43 KBytes
[  5]   6.00-7.00   sec  13.5 MBytes   114 Mbits/sec  366   1.43 KBytes
[  5]   7.00-8.00   sec  16.9 MBytes   142 Mbits/sec  325   1.43 KBytes
[  5]   8.00-9.00   sec  24.2 MBytes   203 Mbits/sec  275   1.43 KBytes
[  5]   9.00-10.00  sec  38.8 MBytes   325 Mbits/sec  277   1.43 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   175 MBytes   147 Mbits/sec  3637
sender
[  5]   0.00-10.00  sec   175 MBytes   147 Mbits/sec
 receiver

iperf Done.



root@:/usr/home/asdf # iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
iperf3: error - unable to receive parameters from client: Connection reset
by peer
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 172.31.16.125, port 21052
^Ciperf3: interrupt - the server has terminated
root@:/usr/home/asdf # iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 172.31.16.125, port 29365
[  5] local 172.31.16.126 port 5201 connected to 172.31.16.125 port 23884
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  5.85 MBytes  49.0 Mbits/sec
[  5]   1.00-2.00   sec  26.6 MBytes   223 Mbits/sec
[  5]   2.00-3.00   sec  15.6 MBytes   131 Mbits/sec
[  5]   3.00-4.00   sec  13.5 MBytes   113 Mbits/sec
[  5]   4.00-5.00   sec  13.7 MBytes   115 Mbits/sec
[  5]   5.00-6.00   sec  6.51 MBytes  54.6 Mbits/sec
[  5]   6.00-7.00   sec  13.5 MBytes   114 Mbits/sec
[  5]   7.00-8.00   sec  16.9 MBytes   142 Mbits/sec
[  5]   8.00-9.00   sec  24.1 MBytes   202 Mbits/sec
[  5]   9.00-10.00  sec  38.8 MBytes   325 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec   175 MBytes   147 Mbits/sec
 receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------

Den ons 26 juni 2019 kl 12:36 skrev Eugene Grosbein <[hidden email]>:

> 26.06.2019 17:26, Christian M wrote:
>
> > I ran ifconfig xn0 -txcsum on both test VM's, and all incorrect checksum
> disappeared.
>
> [skip]
>
> Data looks good. Did it run better this time?
>
>
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer

Eugene Grosbein-10
26.06.2019 17:44, Christian M wrote:

> Sorry to say, but no. Nothing changed :(
>
> The iperf issue concerns me a bit also:

Try starting ntpd inside VM guests, wait an hour then post contents of /var/db/ntpd.drift
This is to verify quality of time source provided to VMs by hypervisor.

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer

Christian Marcos
Started ntpd in the two main 12-0-RELEASE vm's I've been testing with.
drift files show -0.262 and -1.144 after about two hours.

Den ons 26 juni 2019 kl 12:50 skrev Eugene Grosbein <[hidden email]>:

> 26.06.2019 17:44, Christian M wrote:
>
> > Sorry to say, but no. Nothing changed :(
> >
> > The iperf issue concerns me a bit also:
>
> Try starting ntpd inside VM guests, wait an hour then post contents of
> /var/db/ntpd.drift
> This is to verify quality of time source provided to VMs by hypervisor.
>
>
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer

Christian Marcos
I've installed 12.0-STABLE on two new VM's now. 172.31.16.127 and .128. VIF
cheksum offloading is turned off, and -txcsum for xn0 for both VM's.

I feel the throughput is more consistent now, not all over the place as
before, even between runs. But the Retr column (tcp retries) in iperf3 has
jumped up considerably from hundreds/s to thousands/s.

Just a reminder, I have tested this with 11.0-RELEASE also, where the issue
appeared first for me. 10.4-RELEASE is as fast as I could expect it to be,
and 0 retries.

12.0-STABLE:

Connecting to host 172.31.16.128, port 5201
[  5] local 172.31.16.127 port 16833 connected to 172.31.16.128 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  96.3 MBytes   808 Mbits/sec  2401   2.85 KBytes

[  5]   1.00-2.00   sec   118 MBytes   991 Mbits/sec  3120   17.0 KBytes

[  5]   2.00-3.00   sec   121 MBytes  1.02 Gbits/sec  3203   69.8 KBytes

[  5]   3.00-4.00   sec   102 MBytes   853 Mbits/sec  3126   15.6 KBytes

[  5]   4.00-5.00   sec   110 MBytes   921 Mbits/sec  2890   15.6 KBytes

[  5]   5.00-6.00   sec   108 MBytes   908 Mbits/sec  3308   17.0 KBytes

[  5]   6.00-7.00   sec   104 MBytes   869 Mbits/sec  3046   48.2 KBytes

[  5]   7.00-8.00   sec  98.9 MBytes   830 Mbits/sec  2845   2.85 KBytes

[  5]   8.00-9.00   sec   104 MBytes   874 Mbits/sec  2711   86.8 KBytes

[  5]   9.00-10.00  sec   108 MBytes   904 Mbits/sec  2696   14.2 KBytes

[  5]  10.00-11.00  sec   103 MBytes   864 Mbits/sec  2660   31.3 KBytes

[  5]  11.00-12.00  sec  98.8 MBytes   828 Mbits/sec  2476   19.9 KBytes

[  5]  12.00-13.00  sec  99.9 MBytes   838 Mbits/sec  2857   11.3 KBytes

[  5]  13.00-14.00  sec   107 MBytes   894 Mbits/sec  2685   24.1 KBytes

[  5]  14.00-15.00  sec   114 MBytes   953 Mbits/sec  2321   25.5 KBytes

[  5]  15.00-16.00  sec  93.1 MBytes   781 Mbits/sec  2427   48.3 KBytes

[  5]  16.00-17.00  sec   107 MBytes   895 Mbits/sec  2219   29.8 KBytes

[  5]  17.00-18.00  sec  92.5 MBytes   776 Mbits/sec  2441   12.8 KBytes

[  5]  18.00-19.00  sec   116 MBytes   976 Mbits/sec  2840   38.2 KBytes

[  5]  19.00-20.00  sec   102 MBytes   853 Mbits/sec  2573   43.9 KBytes

- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-20.00  sec  2.05 GBytes   882 Mbits/sec  54845
sender
[  5]   0.00-20.00  sec  2.05 GBytes   881 Mbits/sec
 receiver

10.3-RELEASE (offloading enabled everywhere for these VM's VIF, and for
xn0):

Connecting to host 172.31.16.122, port 5201
[  5] local 172.31.16.121 port 20712 connected to 172.31.16.122 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   832 MBytes  6.98 Gbits/sec    0    580 KBytes

[  5]   1.00-2.00   sec  1.24 GBytes  10.6 Gbits/sec    0   1.04 MBytes

[  5]   2.00-3.00   sec  1.34 GBytes  11.5 Gbits/sec    0   1.57 MBytes

[  5]   3.00-4.00   sec  1.41 GBytes  12.1 Gbits/sec    0   1.76 MBytes

[  5]   4.00-5.00   sec  1.37 GBytes  11.7 Gbits/sec    0   1.77 MBytes

[  5]   5.00-6.00   sec  1.34 GBytes  11.5 Gbits/sec    0   1.77 MBytes

[  5]   6.00-7.00   sec  1.37 GBytes  11.8 Gbits/sec    0   1.77 MBytes

[  5]   7.00-8.00   sec  1.39 GBytes  12.0 Gbits/sec    0   1.77 MBytes

[  5]   8.00-9.00   sec  1.35 GBytes  11.6 Gbits/sec    0   1.77 MBytes

[  5]   9.00-10.00  sec  1.42 GBytes  12.2 Gbits/sec    0   1.77 MBytes

[  5]  10.00-11.00  sec  1.34 GBytes  11.5 Gbits/sec    0   1.77 MBytes

[  5]  11.00-12.00  sec  1.32 GBytes  11.4 Gbits/sec    0   1.77 MBytes

[  5]  12.00-13.00  sec  1.34 GBytes  11.6 Gbits/sec    0   1.77 MBytes

[  5]  13.00-14.00  sec  1.49 GBytes  12.8 Gbits/sec    0   1.77 MBytes

[  5]  14.00-15.00  sec  1.36 GBytes  11.7 Gbits/sec    0   1.77 MBytes

[  5]  15.00-16.00  sec  1.37 GBytes  11.7 Gbits/sec    0   1.77 MBytes

[  5]  16.00-17.00  sec  1.48 GBytes  12.7 Gbits/sec    0   1.77 MBytes

[  5]  17.00-18.00  sec  1.36 GBytes  11.7 Gbits/sec    0   1.77 MBytes

[  5]  18.00-19.00  sec  1.47 GBytes  12.6 Gbits/sec    0   1.77 MBytes

[  5]  19.00-20.00  sec  1.36 GBytes  11.7 Gbits/sec    0   1.77 MBytes

- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-20.00  sec  26.9 GBytes  11.6 Gbits/sec    0             sender
[  5]   0.00-20.01  sec  26.9 GBytes  11.6 Gbits/sec
 receiver
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer

Christian Marcos
Den tors 27 juni 2019 kl 12:19 skrev Roger Pau Monné <[hidden email]>:

> On Thu, Jun 27, 2019 at 12:14:33PM +0200, Christian M wrote:
> > I've installed 12.0-STABLE on two new VM's now. 172.31.16.127 and .128.
> VIF
> > cheksum offloading is turned off, and -txcsum for xn0 for both VM's.
> >
> > I feel the throughput is more consistent now, not all over the place as
> > before, even between runs. But the Retr column (tcp retries) in iperf3
> has
> > jumped up considerably from hundreds/s to thousands/s.
> >
> > Just a reminder, I have tested this with 11.0-RELEASE also, where the
> issue
> > appeared first for me. 10.4-RELEASE is as fast as I could expect it to
> be,
> > and 0 retries.
> >
> > 12.0-STABLE:
> >
> > Connecting to host 172.31.16.128, port 5201
> > [  5] local 172.31.16.127 port 16833 connected to 172.31.16.128 port 5201
> > [ ID] Interval           Transfer     Bitrate         Retr  Cwnd
> > [  5]   0.00-1.00   sec  96.3 MBytes   808 Mbits/sec  2401   2.85 KBytes
> >
> > [  5]   1.00-2.00   sec   118 MBytes   991 Mbits/sec  3120   17.0 KBytes
> >
> > [  5]   2.00-3.00   sec   121 MBytes  1.02 Gbits/sec  3203   69.8 KBytes
> >
> > [  5]   3.00-4.00   sec   102 MBytes   853 Mbits/sec  3126   15.6 KBytes
> >
> > [  5]   4.00-5.00   sec   110 MBytes   921 Mbits/sec  2890   15.6 KBytes
> >
> > [  5]   5.00-6.00   sec   108 MBytes   908 Mbits/sec  3308   17.0 KBytes
> >
> > [  5]   6.00-7.00   sec   104 MBytes   869 Mbits/sec  3046   48.2 KBytes
> >
> > [  5]   7.00-8.00   sec  98.9 MBytes   830 Mbits/sec  2845   2.85 KBytes
> >
> > [  5]   8.00-9.00   sec   104 MBytes   874 Mbits/sec  2711   86.8 KBytes
> >
> > [  5]   9.00-10.00  sec   108 MBytes   904 Mbits/sec  2696   14.2 KBytes
> >
> > [  5]  10.00-11.00  sec   103 MBytes   864 Mbits/sec  2660   31.3 KBytes
> >
> > [  5]  11.00-12.00  sec  98.8 MBytes   828 Mbits/sec  2476   19.9 KBytes
> >
> > [  5]  12.00-13.00  sec  99.9 MBytes   838 Mbits/sec  2857   11.3 KBytes
> >
> > [  5]  13.00-14.00  sec   107 MBytes   894 Mbits/sec  2685   24.1 KBytes
> >
> > [  5]  14.00-15.00  sec   114 MBytes   953 Mbits/sec  2321   25.5 KBytes
> >
> > [  5]  15.00-16.00  sec  93.1 MBytes   781 Mbits/sec  2427   48.3 KBytes
> >
> > [  5]  16.00-17.00  sec   107 MBytes   895 Mbits/sec  2219   29.8 KBytes
> >
> > [  5]  17.00-18.00  sec  92.5 MBytes   776 Mbits/sec  2441   12.8 KBytes
> >
> > [  5]  18.00-19.00  sec   116 MBytes   976 Mbits/sec  2840   38.2 KBytes
> >
> > [  5]  19.00-20.00  sec   102 MBytes   853 Mbits/sec  2573   43.9 KBytes
> >
> > - - - - - - - - - - - - - - - - - - - - - - - - -
> > [ ID] Interval           Transfer     Bitrate         Retr
> > [  5]   0.00-20.00  sec  2.05 GBytes   882 Mbits/sec  54845
>
> Can you paste the output of ifconfig for both the interfaces used in
> the test?
>
> Are you sure all hardware offloading capabilities are turned off on
> both interfaces?
>
> Can you check what's causing those retries?
>
> Either using tcpdump, whireshark or some other tool to analyze the
> network traffic and detect the errors that cause such retries?
>
> Thanks, Roger.
>

172.31.16.127 (12.0-STABLE):

lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
inet 127.0.0.1 netmask 0xff000000
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
xn0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=501<RXCSUM,TSO4,LRO>
ether 6e:83:99:ed:ce:f7
inet 172.31.16.127 netmask 0xffffff00 broadcast 172.31.16.255
media: Ethernet manual
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

ethtool -k vif68.0
Features for vif68.0:
rx-checksumming: on [fixed]
tx-checksumming: off
tx-checksum-ipv4: off
tx-checksum-ip-generic: off [fixed]
tx-checksum-ipv6: off
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: off [fixed]
scatter-gather: off
tx-scatter-gather: off
tx-scatter-gather-fraglist: off
tcp-segmentation-offload: off
tx-tcp-segmentation: off
tx-tcp-ecn-segmentation: off [fixed]
tx-tcp6-segmentation: off
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: off
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: off [fixed]
tx-vlan-offload: off [fixed]
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: off [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-ipip-segmentation: off [fixed]
tx-sit-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
busy-poll: off [fixed]

172.31.16.128 (12.0-STABLE):

lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
inet 127.0.0.1 netmask 0xff000000
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
xn0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=501<RXCSUM,TSO4,LRO>
ether 32:a7:9f:cc:94:8a
inet 172.31.16.128 netmask 0xffffff00 broadcast 172.31.16.255
media: Ethernet manual
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

ethtool -k vif67.0
Features for vif67.0:
rx-checksumming: on [fixed]
tx-checksumming: off
tx-checksum-ipv4: off
tx-checksum-ip-generic: off [fixed]
tx-checksum-ipv6: off
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: off [fixed]
scatter-gather: off
tx-scatter-gather: off
tx-scatter-gather-fraglist: off
tcp-segmentation-offload: off
tx-tcp-segmentation: off
tx-tcp-ecn-segmentation: off [fixed]
tx-tcp6-segmentation: off
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: off
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: off [fixed]
tx-vlan-offload: off [fixed]
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: off [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-ipip-segmentation: off [fixed]
tx-sit-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
busy-poll: off [fixed]

Should I run tcpdump on xen host, or on FreeBSD while running iperf3?

As I side note, I installed XCP-ng 7.6.0 on a older server I had laying
around just to see how 12.0-RELEASE behaved there, and It was the same
thing, super slow between VM's.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer

Roger Pau Monné
On Thu, Jun 27, 2019 at 12:31:35PM +0200, Christian M wrote:

> Den tors 27 juni 2019 kl 12:19 skrev Roger Pau Monné <[hidden email]>:
>
> > On Thu, Jun 27, 2019 at 12:14:33PM +0200, Christian M wrote:
> > > I've installed 12.0-STABLE on two new VM's now. 172.31.16.127 and .128.
> > VIF
> > > cheksum offloading is turned off, and -txcsum for xn0 for both VM's.
> > >
> > > I feel the throughput is more consistent now, not all over the place as
> > > before, even between runs. But the Retr column (tcp retries) in iperf3
> > has
> > > jumped up considerably from hundreds/s to thousands/s.
> > >
> > > Just a reminder, I have tested this with 11.0-RELEASE also, where the
> > issue
> > > appeared first for me. 10.4-RELEASE is as fast as I could expect it to
> > be,
> > > and 0 retries.
> > >
> > > 12.0-STABLE:
> > >
> > > Connecting to host 172.31.16.128, port 5201
> > > [  5] local 172.31.16.127 port 16833 connected to 172.31.16.128 port 5201
> > > [ ID] Interval           Transfer     Bitrate         Retr  Cwnd
> > > [  5]   0.00-1.00   sec  96.3 MBytes   808 Mbits/sec  2401   2.85 KBytes
> > >
> > > [  5]   1.00-2.00   sec   118 MBytes   991 Mbits/sec  3120   17.0 KBytes
> > >
> > > [  5]   2.00-3.00   sec   121 MBytes  1.02 Gbits/sec  3203   69.8 KBytes
> > >
> > > [  5]   3.00-4.00   sec   102 MBytes   853 Mbits/sec  3126   15.6 KBytes
> > >
> > > [  5]   4.00-5.00   sec   110 MBytes   921 Mbits/sec  2890   15.6 KBytes
> > >
> > > [  5]   5.00-6.00   sec   108 MBytes   908 Mbits/sec  3308   17.0 KBytes
> > >
> > > [  5]   6.00-7.00   sec   104 MBytes   869 Mbits/sec  3046   48.2 KBytes
> > >
> > > [  5]   7.00-8.00   sec  98.9 MBytes   830 Mbits/sec  2845   2.85 KBytes
> > >
> > > [  5]   8.00-9.00   sec   104 MBytes   874 Mbits/sec  2711   86.8 KBytes
> > >
> > > [  5]   9.00-10.00  sec   108 MBytes   904 Mbits/sec  2696   14.2 KBytes
> > >
> > > [  5]  10.00-11.00  sec   103 MBytes   864 Mbits/sec  2660   31.3 KBytes
> > >
> > > [  5]  11.00-12.00  sec  98.8 MBytes   828 Mbits/sec  2476   19.9 KBytes
> > >
> > > [  5]  12.00-13.00  sec  99.9 MBytes   838 Mbits/sec  2857   11.3 KBytes
> > >
> > > [  5]  13.00-14.00  sec   107 MBytes   894 Mbits/sec  2685   24.1 KBytes
> > >
> > > [  5]  14.00-15.00  sec   114 MBytes   953 Mbits/sec  2321   25.5 KBytes
> > >
> > > [  5]  15.00-16.00  sec  93.1 MBytes   781 Mbits/sec  2427   48.3 KBytes
> > >
> > > [  5]  16.00-17.00  sec   107 MBytes   895 Mbits/sec  2219   29.8 KBytes
> > >
> > > [  5]  17.00-18.00  sec  92.5 MBytes   776 Mbits/sec  2441   12.8 KBytes
> > >
> > > [  5]  18.00-19.00  sec   116 MBytes   976 Mbits/sec  2840   38.2 KBytes
> > >
> > > [  5]  19.00-20.00  sec   102 MBytes   853 Mbits/sec  2573   43.9 KBytes
> > >
> > > - - - - - - - - - - - - - - - - - - - - - - - - -
> > > [ ID] Interval           Transfer     Bitrate         Retr
> > > [  5]   0.00-20.00  sec  2.05 GBytes   882 Mbits/sec  54845
> >
> > Can you paste the output of ifconfig for both the interfaces used in
> > the test?
> >
> > Are you sure all hardware offloading capabilities are turned off on
> > both interfaces?
> >
> > Can you check what's causing those retries?
> >
> > Either using tcpdump, whireshark or some other tool to analyze the
> > network traffic and detect the errors that cause such retries?
> >
> > Thanks, Roger.
> >
>
> 172.31.16.127 (12.0-STABLE):
>
> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
> options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
> inet6 ::1 prefixlen 128
> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
> inet 127.0.0.1 netmask 0xff000000
> groups: lo
> nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
> xn0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
> options=501<RXCSUM,TSO4,LRO>

I would try to disable rxcsum, tso4 and lro also.

> ether 6e:83:99:ed:ce:f7
> inet 172.31.16.127 netmask 0xffffff00 broadcast 172.31.16.255
> media: Ethernet manual
> status: active
> nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
>
> ethtool -k vif68.0
> Features for vif68.0:
> rx-checksumming: on [fixed]
> tx-checksumming: off
> tx-checksum-ipv4: off
> tx-checksum-ip-generic: off [fixed]
> tx-checksum-ipv6: off
> tx-checksum-fcoe-crc: off [fixed]
> tx-checksum-sctp: off [fixed]
> scatter-gather: off
> tx-scatter-gather: off
> tx-scatter-gather-fraglist: off
> tcp-segmentation-offload: off
> tx-tcp-segmentation: off
> tx-tcp-ecn-segmentation: off [fixed]
> tx-tcp6-segmentation: off
> udp-fragmentation-offload: off [fixed]
> generic-segmentation-offload: off
> generic-receive-offload: on
> large-receive-offload: off [fixed]
> rx-vlan-offload: off [fixed]
> tx-vlan-offload: off [fixed]
> ntuple-filters: off [fixed]
> receive-hashing: off [fixed]
> highdma: off [fixed]
> rx-vlan-filter: off [fixed]
> vlan-challenged: off [fixed]
> tx-lockless: off [fixed]
> netns-local: off [fixed]
> tx-gso-robust: off [fixed]
> tx-fcoe-segmentation: off [fixed]
> tx-gre-segmentation: off [fixed]
> tx-ipip-segmentation: off [fixed]
> tx-sit-segmentation: off [fixed]
> tx-udp_tnl-segmentation: off [fixed]
> fcoe-mtu: off [fixed]
> tx-nocache-copy: off
> loopback: off [fixed]
> rx-fcs: off [fixed]
> rx-all: off [fixed]
> tx-vlan-stag-hw-insert: off [fixed]
> rx-vlan-stag-hw-parse: off [fixed]
> rx-vlan-stag-filter: off [fixed]
> l2-fwd-offload: off [fixed]
> busy-poll: off [fixed]
>
> 172.31.16.128 (12.0-STABLE):
>
> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
> options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
> inet6 ::1 prefixlen 128
> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
> inet 127.0.0.1 netmask 0xff000000
> groups: lo
> nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
> xn0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
> options=501<RXCSUM,TSO4,LRO>
> ether 32:a7:9f:cc:94:8a
> inet 172.31.16.128 netmask 0xffffff00 broadcast 172.31.16.255
> media: Ethernet manual
> status: active
> nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
>
> ethtool -k vif67.0
> Features for vif67.0:
> rx-checksumming: on [fixed]
> tx-checksumming: off
> tx-checksum-ipv4: off
> tx-checksum-ip-generic: off [fixed]
> tx-checksum-ipv6: off
> tx-checksum-fcoe-crc: off [fixed]
> tx-checksum-sctp: off [fixed]
> scatter-gather: off
> tx-scatter-gather: off
> tx-scatter-gather-fraglist: off
> tcp-segmentation-offload: off
> tx-tcp-segmentation: off
> tx-tcp-ecn-segmentation: off [fixed]
> tx-tcp6-segmentation: off
> udp-fragmentation-offload: off [fixed]
> generic-segmentation-offload: off
> generic-receive-offload: on
> large-receive-offload: off [fixed]
> rx-vlan-offload: off [fixed]
> tx-vlan-offload: off [fixed]
> ntuple-filters: off [fixed]
> receive-hashing: off [fixed]
> highdma: off [fixed]
> rx-vlan-filter: off [fixed]
> vlan-challenged: off [fixed]
> tx-lockless: off [fixed]
> netns-local: off [fixed]
> tx-gso-robust: off [fixed]
> tx-fcoe-segmentation: off [fixed]
> tx-gre-segmentation: off [fixed]
> tx-ipip-segmentation: off [fixed]
> tx-sit-segmentation: off [fixed]
> tx-udp_tnl-segmentation: off [fixed]
> fcoe-mtu: off [fixed]
> tx-nocache-copy: off
> loopback: off [fixed]
> rx-fcs: off [fixed]
> rx-all: off [fixed]
> tx-vlan-stag-hw-insert: off [fixed]
> rx-vlan-stag-hw-parse: off [fixed]
> rx-vlan-stag-filter: off [fixed]
> l2-fwd-offload: off [fixed]
> busy-poll: off [fixed]
>
> Should I run tcpdump on xen host, or on FreeBSD while running iperf3?

On the Xen host would be better, since that tcpdump will read the
packets as FreeBSD puts them on the wire.

> As I side note, I installed XCP-ng 7.6.0 on a older server I had laying
> around just to see how 12.0-RELEASE behaved there, and It was the same
> thing, super slow between VM's.

Yes, I don't think this is related to hardware at all.

I will try to find some time to look into this, but it's not going to
be soon since I'm currently preparing a talk for a conference I'm
attending next week, so I won't likely be able to get to it until the
end of July, sorry. Also, I'm no networking expert, so I cannot assure
my success in fixing this.

As a workaround you can switch to the emulated network card by
setting 'hw.xen.disable_pv_nics=1' in /boot/loader.conf. That will
give you worse performance than a fully working PV network card, but
at least should be consistent. There are others that have switched to
virtio-net, but I have no idea how to do that with XCP.

Roger.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer

Christian Marcos
Thanks Roger.

I disabled pv nic entirely on my two 12.0-RELEASE test VM's. I got
1000baseT full duplex auto selected, so I expected throughput close to
that, but to my surprise this are the results:

$ ifconfig
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=812099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER>
ether 56:65:6f:f3:02:fb
inet 172.31.16.125 netmask 0xffffff00 broadcast 172.31.16.255
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
inet 127.0.0.1 netmask 0xff000000
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
$ iperf3 -c 172.31.16.126
Connecting to host 172.31.16.126, port 5201
[  5] local 172.31.16.125 port 11247 connected to 172.31.16.126 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  6.50 MBytes  54.5 Mbits/sec    0    368 KBytes
[  5]   1.00-2.00   sec  7.22 MBytes  60.6 Mbits/sec    0    368 KBytes
[  5]   2.00-3.00   sec  7.26 MBytes  60.9 Mbits/sec    0    368 KBytes
[  5]   3.00-4.00   sec  7.09 MBytes  59.4 Mbits/sec    0    368 KBytes
[  5]   4.00-5.00   sec  7.32 MBytes  61.4 Mbits/sec    0    368 KBytes
[  5]   5.00-6.00   sec  7.22 MBytes  60.6 Mbits/sec    0    385 KBytes
[  5]   6.00-7.00   sec  7.24 MBytes  60.7 Mbits/sec    0    385 KBytes
[  5]   7.00-8.00   sec  7.28 MBytes  61.1 Mbits/sec    0    385 KBytes
[  5]   8.00-9.00   sec  7.55 MBytes  63.3 Mbits/sec    0    385 KBytes
[  5]   9.00-10.00  sec  7.12 MBytes  59.7 Mbits/sec    0    385 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  71.8 MBytes  60.2 Mbits/sec    0             sender
[  5]   0.00-10.06  sec  71.8 MBytes  59.9 Mbits/sec
 receiver

iperf Done.
$ iperf3 -c 172.31.16.126 -R
Connecting to host 172.31.16.126, port 5201
Reverse mode, remote host 172.31.16.126 is sending
[  5] local 172.31.16.125 port 22443 connected to 172.31.16.126 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  5.06 MBytes  42.5 Mbits/sec
[  5]   1.00-2.00   sec  5.60 MBytes  47.0 Mbits/sec
[  5]   2.00-3.00   sec  6.22 MBytes  52.2 Mbits/sec
[  5]   3.00-4.00   sec  6.92 MBytes  58.0 Mbits/sec
[  5]   4.00-5.00   sec  7.39 MBytes  62.0 Mbits/sec
[  5]   5.00-6.00   sec  7.17 MBytes  60.1 Mbits/sec
[  5]   6.00-7.00   sec  7.18 MBytes  60.3 Mbits/sec
[  5]   7.00-8.00   sec  7.32 MBytes  61.4 Mbits/sec
[  5]   8.00-9.00   sec  6.94 MBytes  58.2 Mbits/sec
[  5]   9.00-10.00  sec  7.04 MBytes  59.0 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  67.3 MBytes  56.5 Mbits/sec    0             sender
[  5]   0.00-10.00  sec  66.8 MBytes  56.1 Mbits/sec
 receiver

iperf Done.

Good luck with your talk.

Regards,
Christian


Den tors 4 juli 2019 kl 09:24 skrev Roger Pau Monné <[hidden email]>:

> On Thu, Jun 27, 2019 at 12:31:35PM +0200, Christian M wrote:
> > Den tors 27 juni 2019 kl 12:19 skrev Roger Pau Monné <
> [hidden email]>:
> >
> > > On Thu, Jun 27, 2019 at 12:14:33PM +0200, Christian M wrote:
> > > > I've installed 12.0-STABLE on two new VM's now. 172.31.16.127 and
> .128.
> > > VIF
> > > > cheksum offloading is turned off, and -txcsum for xn0 for both VM's.
> > > >
> > > > I feel the throughput is more consistent now, not all over the place
> as
> > > > before, even between runs. But the Retr column (tcp retries) in
> iperf3
> > > has
> > > > jumped up considerably from hundreds/s to thousands/s.
> > > >
> > > > Just a reminder, I have tested this with 11.0-RELEASE also, where the
> > > issue
> > > > appeared first for me. 10.4-RELEASE is as fast as I could expect it
> to
> > > be,
> > > > and 0 retries.
> > > >
> > > > 12.0-STABLE:
> > > >
> > > > Connecting to host 172.31.16.128, port 5201
> > > > [  5] local 172.31.16.127 port 16833 connected to 172.31.16.128 port
> 5201
> > > > [ ID] Interval           Transfer     Bitrate         Retr  Cwnd
> > > > [  5]   0.00-1.00   sec  96.3 MBytes   808 Mbits/sec  2401   2.85
> KBytes
> > > >
> > > > [  5]   1.00-2.00   sec   118 MBytes   991 Mbits/sec  3120   17.0
> KBytes
> > > >
> > > > [  5]   2.00-3.00   sec   121 MBytes  1.02 Gbits/sec  3203   69.8
> KBytes
> > > >
> > > > [  5]   3.00-4.00   sec   102 MBytes   853 Mbits/sec  3126   15.6
> KBytes
> > > >
> > > > [  5]   4.00-5.00   sec   110 MBytes   921 Mbits/sec  2890   15.6
> KBytes
> > > >
> > > > [  5]   5.00-6.00   sec   108 MBytes   908 Mbits/sec  3308   17.0
> KBytes
> > > >
> > > > [  5]   6.00-7.00   sec   104 MBytes   869 Mbits/sec  3046   48.2
> KBytes
> > > >
> > > > [  5]   7.00-8.00   sec  98.9 MBytes   830 Mbits/sec  2845   2.85
> KBytes
> > > >
> > > > [  5]   8.00-9.00   sec   104 MBytes   874 Mbits/sec  2711   86.8
> KBytes
> > > >
> > > > [  5]   9.00-10.00  sec   108 MBytes   904 Mbits/sec  2696   14.2
> KBytes
> > > >
> > > > [  5]  10.00-11.00  sec   103 MBytes   864 Mbits/sec  2660   31.3
> KBytes
> > > >
> > > > [  5]  11.00-12.00  sec  98.8 MBytes   828 Mbits/sec  2476   19.9
> KBytes
> > > >
> > > > [  5]  12.00-13.00  sec  99.9 MBytes   838 Mbits/sec  2857   11.3
> KBytes
> > > >
> > > > [  5]  13.00-14.00  sec   107 MBytes   894 Mbits/sec  2685   24.1
> KBytes
> > > >
> > > > [  5]  14.00-15.00  sec   114 MBytes   953 Mbits/sec  2321   25.5
> KBytes
> > > >
> > > > [  5]  15.00-16.00  sec  93.1 MBytes   781 Mbits/sec  2427   48.3
> KBytes
> > > >
> > > > [  5]  16.00-17.00  sec   107 MBytes   895 Mbits/sec  2219   29.8
> KBytes
> > > >
> > > > [  5]  17.00-18.00  sec  92.5 MBytes   776 Mbits/sec  2441   12.8
> KBytes
> > > >
> > > > [  5]  18.00-19.00  sec   116 MBytes   976 Mbits/sec  2840   38.2
> KBytes
> > > >
> > > > [  5]  19.00-20.00  sec   102 MBytes   853 Mbits/sec  2573   43.9
> KBytes
> > > >
> > > > - - - - - - - - - - - - - - - - - - - - - - - - -
> > > > [ ID] Interval           Transfer     Bitrate         Retr
> > > > [  5]   0.00-20.00  sec  2.05 GBytes   882 Mbits/sec  54845
> > >
> > > Can you paste the output of ifconfig for both the interfaces used in
> > > the test?
> > >
> > > Are you sure all hardware offloading capabilities are turned off on
> > > both interfaces?
> > >
> > > Can you check what's causing those retries?
> > >
> > > Either using tcpdump, whireshark or some other tool to analyze the
> > > network traffic and detect the errors that cause such retries?
> > >
> > > Thanks, Roger.
> > >
> >
> > 172.31.16.127 (12.0-STABLE):
> >
> > lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
> > options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
> > inet6 ::1 prefixlen 128
> > inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
> > inet 127.0.0.1 netmask 0xff000000
> > groups: lo
> > nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
> > xn0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
> > options=501<RXCSUM,TSO4,LRO>
>
> I would try to disable rxcsum, tso4 and lro also.
>
> > ether 6e:83:99:ed:ce:f7
> > inet 172.31.16.127 netmask 0xffffff00 broadcast 172.31.16.255
> > media: Ethernet manual
> > status: active
> > nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> >
> > ethtool -k vif68.0
> > Features for vif68.0:
> > rx-checksumming: on [fixed]
> > tx-checksumming: off
> > tx-checksum-ipv4: off
> > tx-checksum-ip-generic: off [fixed]
> > tx-checksum-ipv6: off
> > tx-checksum-fcoe-crc: off [fixed]
> > tx-checksum-sctp: off [fixed]
> > scatter-gather: off
> > tx-scatter-gather: off
> > tx-scatter-gather-fraglist: off
> > tcp-segmentation-offload: off
> > tx-tcp-segmentation: off
> > tx-tcp-ecn-segmentation: off [fixed]
> > tx-tcp6-segmentation: off
> > udp-fragmentation-offload: off [fixed]
> > generic-segmentation-offload: off
> > generic-receive-offload: on
> > large-receive-offload: off [fixed]
> > rx-vlan-offload: off [fixed]
> > tx-vlan-offload: off [fixed]
> > ntuple-filters: off [fixed]
> > receive-hashing: off [fixed]
> > highdma: off [fixed]
> > rx-vlan-filter: off [fixed]
> > vlan-challenged: off [fixed]
> > tx-lockless: off [fixed]
> > netns-local: off [fixed]
> > tx-gso-robust: off [fixed]
> > tx-fcoe-segmentation: off [fixed]
> > tx-gre-segmentation: off [fixed]
> > tx-ipip-segmentation: off [fixed]
> > tx-sit-segmentation: off [fixed]
> > tx-udp_tnl-segmentation: off [fixed]
> > fcoe-mtu: off [fixed]
> > tx-nocache-copy: off
> > loopback: off [fixed]
> > rx-fcs: off [fixed]
> > rx-all: off [fixed]
> > tx-vlan-stag-hw-insert: off [fixed]
> > rx-vlan-stag-hw-parse: off [fixed]
> > rx-vlan-stag-filter: off [fixed]
> > l2-fwd-offload: off [fixed]
> > busy-poll: off [fixed]
> >
> > 172.31.16.128 (12.0-STABLE):
> >
> > lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
> > options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
> > inet6 ::1 prefixlen 128
> > inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
> > inet 127.0.0.1 netmask 0xff000000
> > groups: lo
> > nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
> > xn0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
> > options=501<RXCSUM,TSO4,LRO>
> > ether 32:a7:9f:cc:94:8a
> > inet 172.31.16.128 netmask 0xffffff00 broadcast 172.31.16.255
> > media: Ethernet manual
> > status: active
> > nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> >
> > ethtool -k vif67.0
> > Features for vif67.0:
> > rx-checksumming: on [fixed]
> > tx-checksumming: off
> > tx-checksum-ipv4: off
> > tx-checksum-ip-generic: off [fixed]
> > tx-checksum-ipv6: off
> > tx-checksum-fcoe-crc: off [fixed]
> > tx-checksum-sctp: off [fixed]
> > scatter-gather: off
> > tx-scatter-gather: off
> > tx-scatter-gather-fraglist: off
> > tcp-segmentation-offload: off
> > tx-tcp-segmentation: off
> > tx-tcp-ecn-segmentation: off [fixed]
> > tx-tcp6-segmentation: off
> > udp-fragmentation-offload: off [fixed]
> > generic-segmentation-offload: off
> > generic-receive-offload: on
> > large-receive-offload: off [fixed]
> > rx-vlan-offload: off [fixed]
> > tx-vlan-offload: off [fixed]
> > ntuple-filters: off [fixed]
> > receive-hashing: off [fixed]
> > highdma: off [fixed]
> > rx-vlan-filter: off [fixed]
> > vlan-challenged: off [fixed]
> > tx-lockless: off [fixed]
> > netns-local: off [fixed]
> > tx-gso-robust: off [fixed]
> > tx-fcoe-segmentation: off [fixed]
> > tx-gre-segmentation: off [fixed]
> > tx-ipip-segmentation: off [fixed]
> > tx-sit-segmentation: off [fixed]
> > tx-udp_tnl-segmentation: off [fixed]
> > fcoe-mtu: off [fixed]
> > tx-nocache-copy: off
> > loopback: off [fixed]
> > rx-fcs: off [fixed]
> > rx-all: off [fixed]
> > tx-vlan-stag-hw-insert: off [fixed]
> > rx-vlan-stag-hw-parse: off [fixed]
> > rx-vlan-stag-filter: off [fixed]
> > l2-fwd-offload: off [fixed]
> > busy-poll: off [fixed]
> >
> > Should I run tcpdump on xen host, or on FreeBSD while running iperf3?
>
> On the Xen host would be better, since that tcpdump will read the
> packets as FreeBSD puts them on the wire.
>
> > As I side note, I installed XCP-ng 7.6.0 on a older server I had laying
> > around just to see how 12.0-RELEASE behaved there, and It was the same
> > thing, super slow between VM's.
>
> Yes, I don't think this is related to hardware at all.
>
> I will try to find some time to look into this, but it's not going to
> be soon since I'm currently preparing a talk for a conference I'm
> attending next week, so I won't likely be able to get to it until the
> end of July, sorry. Also, I'm no networking expert, so I cannot assure
> my success in fixing this.
>
> As a workaround you can switch to the emulated network card by
> setting 'hw.xen.disable_pv_nics=1' in /boot/loader.conf. That will
> give you worse performance than a fully working PV network card, but
> at least should be consistent. There are others that have switched to
> virtio-net, but I have no idea how to do that with XCP.
>
> Roger.
>
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "[hidden email]"
12