Determining cause of transfer limit

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Determining cause of transfer limit

Sean C. Farley-2
I have recently upped my Internet service and have now noticed a limit
being reached, but I am not certain which limit and best option to
resolve it.

I am using a circa 2007 system as a multi-purpose router running FreeBSD
12-STABLE (r367740).  The issue is that it maxes out around 400Mb/s when
running a speed test through it between my workstation and various test
sites (i.e., DSL Reports and Speedtest).  There are two NIC's (both are
Intel 82541PI) in use with one to the ISP and one to my workstation.

At first, I saw one of them apparently hitting an interrupt rate of just
over 8000, so I bumped their rate limits higher with little to no
improvement.

What makes me believe I can theoretically get faster speeds is that I
can use the onboard NIC (Marvell 88E8056) to replace one of the NIC's
and nearly double the speed.  The difference is that it is on the PCI-E
bus and has MSI support.

irq16: em0:irq0+
irq17: em1:irq0
irq20: hpet0
irq258: mskc0

I have many network settings, but changing them did nothing.  Here are
the settings I am trying now that seem to squeak a little extra
performance.  The commented-out lines are ones I tried without seeing
any change.  I have also tested without these settings.

/boot/loader.conf
hw.em.rx_process_limit="-1"
# dev.em.0.iflib.override_nrxds="2048"
# dev.em.1.iflib.override_nrxds="2048"
# dev.em.2.iflib.override_nrxds="2048"
# dev.em.0.iflib.override_ntxds="2048"
# net.link.ifqmaxlen="2048"
hw.em.max_interrupt_rate="32000"
# net.isr.maxthreads="-1"
# net.isr.bindthreads="1"

/etc/sysctl.conf
kern.random.harvest.mask=351
dev.em.0.fc=0
dev.em.1.fc=0
dev.em.0.itr=122        # Allow past 8000 interrupts/second.
dev.em.1.itr=122
net.inet.ip.redirect=0
net.inet6.ip6.redirect=0

Increasing these from 66 to 250 did not help:
hw.em.rx_abs_int_delay: 66
hw.em.tx_abs_int_delay: 66
hw.em.tx_int_delay: 66

I am utilizing pf, but I doubt it is the issue since using the same
rules with the msk driver would have held the speed down to 400Mb/s.

Am I hitting the limit of the PCI bus (memory or interrupt) or something
else?  I can buy a new PCI-E NIC for the internal network, but I rather
fully utilize the Intel NIC's I have, if possible.

Sean
--
[hidden email]
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Determining cause of transfer limit

Michael Sierchio
Sorry for the top post.  Have you tried device polling? From
/usr/src/sys/amd64/conf/NOTES:

#####################################################################

# NETWORKING OPTIONS


#

# DEVICE_POLLING adds support for mixed interrupt-polling handling

# of network device drivers, which has significant benefits in terms

# of robustness to overloads and responsivity, as well as permitting

# accurate scheduling of the CPU time between kernel network processing

# and other activities.  The drawback is a moderate (up to 1/HZ seconds)

# potential increase in response times.

# It is strongly recommended to use HZ=1000 or 2000 with DEVICE_POLLING

# to achieve smoother behaviour.

# Additionally, you can enable/disable polling at runtime with help of

# the ifconfig(8) utility, and select the CPU fraction reserved to

# userland with the sysctl variable kern.polling.user_frac

# (default 50, range 0..100).

#

# Not all device drivers support this mode of operation at the time of

# this writing.  See polling(4) for more details.


options         DEVICE_POLLING

On Sat, Nov 21, 2020 at 10:24 AM Sean C. Farley <[hidden email]> wrote:

> I have recently upped my Internet service and have now noticed a limit
> being reached, but I am not certain which limit and best option to
> resolve it.
>
> I am using a circa 2007 system as a multi-purpose router running FreeBSD
> 12-STABLE (r367740).  The issue is that it maxes out around 400Mb/s when
> running a speed test through it between my workstation and various test
> sites (i.e., DSL Reports and Speedtest).  There are two NIC's (both are
> Intel 82541PI) in use with one to the ISP and one to my workstation.
>
> At first, I saw one of them apparently hitting an interrupt rate of just
> over 8000, so I bumped their rate limits higher with little to no
> improvement.
>
> What makes me believe I can theoretically get faster speeds is that I
> can use the onboard NIC (Marvell 88E8056) to replace one of the NIC's
> and nearly double the speed.  The difference is that it is on the PCI-E
> bus and has MSI support.
>
> irq16: em0:irq0+
> irq17: em1:irq0
> irq20: hpet0
> irq258: mskc0
>
> I have many network settings, but changing them did nothing.  Here are
> the settings I am trying now that seem to squeak a little extra
> performance.  The commented-out lines are ones I tried without seeing
> any change.  I have also tested without these settings.
>
> /boot/loader.conf
> hw.em.rx_process_limit="-1"
> # dev.em.0.iflib.override_nrxds="2048"
> # dev.em.1.iflib.override_nrxds="2048"
> # dev.em.2.iflib.override_nrxds="2048"
> # dev.em.0.iflib.override_ntxds="2048"
> # net.link.ifqmaxlen="2048"
> hw.em.max_interrupt_rate="32000"
> # net.isr.maxthreads="-1"
> # net.isr.bindthreads="1"
>
> /etc/sysctl.conf
> kern.random.harvest.mask=351
> dev.em.0.fc=0
> dev.em.1.fc=0
> dev.em.0.itr=122        # Allow past 8000 interrupts/second.
> dev.em.1.itr=122
> net.inet.ip.redirect=0
> net.inet6.ip6.redirect=0
>
> Increasing these from 66 to 250 did not help:
> hw.em.rx_abs_int_delay: 66
> hw.em.tx_abs_int_delay: 66
> hw.em.tx_int_delay: 66
>
> I am utilizing pf, but I doubt it is the issue since using the same
> rules with the msk driver would have held the speed down to 400Mb/s.
>
> Am I hitting the limit of the PCI bus (memory or interrupt) or something
> else?  I can buy a new PCI-E NIC for the internal network, but I rather
> fully utilize the Intel NIC's I have, if possible.
>
> Sean
> --
> [hidden email]
> _______________________________________________
> [hidden email] mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "[hidden email]"
>


--

"Well," Brahmā said, "even after ten thousand explanations, a fool is no
wiser, but an intelligent person requires only two thousand five hundred."

- The Mahābhārata
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Determining cause of transfer limit

Saad, Mark
scf
  can I use polling on sfxge ? Also if I want to use polling on say ix or sfxge
does this break things like pf, bird or nsd ?

---
Mark Saad
Lucera Financial Infrastructures, LLC
[hidden email]


________________________________________
From: [hidden email] <[hidden email]> on behalf of Michael Sierchio <[hidden email]>
Sent: Saturday, November 21, 2020 4:55 PM
To: Sean C. Farley
Cc: [hidden email]
Subject: Re: Determining cause of transfer limit

Sorry for the top post.  Have you tried device polling? From
/usr/src/sys/amd64/conf/NOTES:

#####################################################################

# NETWORKING OPTIONS


#

# DEVICE_POLLING adds support for mixed interrupt-polling handling

# of network device drivers, which has significant benefits in terms

# of robustness to overloads and responsivity, as well as permitting

# accurate scheduling of the CPU time between kernel network processing

# and other activities.  The drawback is a moderate (up to 1/HZ seconds)

# potential increase in response times.

# It is strongly recommended to use HZ=1000 or 2000 with DEVICE_POLLING

# to achieve smoother behaviour.

# Additionally, you can enable/disable polling at runtime with help of

# the ifconfig(8) utility, and select the CPU fraction reserved to

# userland with the sysctl variable kern.polling.user_frac

# (default 50, range 0..100).

#

# Not all device drivers support this mode of operation at the time of

# this writing.  See polling(4) for more details.


options         DEVICE_POLLING

On Sat, Nov 21, 2020 at 10:24 AM Sean C. Farley <[hidden email]> wrote:

> I have recently upped my Internet service and have now noticed a limit
> being reached, but I am not certain which limit and best option to
> resolve it.
>
> I am using a circa 2007 system as a multi-purpose router running FreeBSD
> 12-STABLE (r367740).  The issue is that it maxes out around 400Mb/s when
> running a speed test through it between my workstation and various test
> sites (i.e., DSL Reports and Speedtest).  There are two NIC's (both are
> Intel 82541PI) in use with one to the ISP and one to my workstation.
>
> At first, I saw one of them apparently hitting an interrupt rate of just
> over 8000, so I bumped their rate limits higher with little to no
> improvement.
>
> What makes me believe I can theoretically get faster speeds is that I
> can use the onboard NIC (Marvell 88E8056) to replace one of the NIC's
> and nearly double the speed.  The difference is that it is on the PCI-E
> bus and has MSI support.
>
> irq16: em0:irq0+
> irq17: em1:irq0
> irq20: hpet0
> irq258: mskc0
>
> I have many network settings, but changing them did nothing.  Here are
> the settings I am trying now that seem to squeak a little extra
> performance.  The commented-out lines are ones I tried without seeing
> any change.  I have also tested without these settings.
>
> /boot/loader.conf
> hw.em.rx_process_limit="-1"
> # dev.em.0.iflib.override_nrxds="2048"
> # dev.em.1.iflib.override_nrxds="2048"
> # dev.em.2.iflib.override_nrxds="2048"
> # dev.em.0.iflib.override_ntxds="2048"
> # net.link.ifqmaxlen="2048"
> hw.em.max_interrupt_rate="32000"
> # net.isr.maxthreads="-1"
> # net.isr.bindthreads="1"
>
> /etc/sysctl.conf
> kern.random.harvest.mask=351
> dev.em.0.fc=0
> dev.em.1.fc=0
> dev.em.0.itr=122        # Allow past 8000 interrupts/second.
> dev.em.1.itr=122
> net.inet.ip.redirect=0
> net.inet6.ip6.redirect=0
>
> Increasing these from 66 to 250 did not help:
> hw.em.rx_abs_int_delay: 66
> hw.em.tx_abs_int_delay: 66
> hw.em.tx_int_delay: 66
>
> I am utilizing pf, but I doubt it is the issue since using the same
> rules with the msk driver would have held the speed down to 400Mb/s.
>
> Am I hitting the limit of the PCI bus (memory or interrupt) or something
> else?  I can buy a new PCI-E NIC for the internal network, but I rather
> fully utilize the Intel NIC's I have, if possible.
>
> Sean
> --
> [hidden email]
> _______________________________________________
> [hidden email] mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "[hidden email]"
>


--

"Well," Brahmā said, "even after ten thousand explanations, a fool is no
wiser, but an intelligent person requires only two thousand five hundred."

- The Mahābhārata
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[hidden email]"

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Determining cause of transfer limit

Sean C. Farley-2
In reply to this post by Michael Sierchio
On Sat, 21 Nov 2020, Michael Sierchio wrote:

> Sorry for the top post.  Have you tried device polling? From
> /usr/src/sys/amd64/conf/NOTES:
...
> options         DEVICE_POLLING

*snip*

I had not tried it and had actually forgotten about that capability.
Anyway, I just tried it (kernel built with polling and enabled via
ifconfig on the devices) and saw no observable change.  I am not certain
if the code paths can really make use of it with the switch to iflib.

However, I was able to squeak a bit more from the NIC's by the
equivalent of polling(4) by bumping the delay for interrupts with the
key one being hw.em.rx_int_delay.  It improved transfers going through
the router, in both directions, by about 20Mb/s.  The number of
interrupts dropped a lot, so I doubt I am hitting an interrupt limit.

/boot/loader.conf:
hw.em.rx_int_delay="264"
hw.em.rx_abs_int_delay="264"
hw.em.tx_int_delay="264"
hw.em.tx_abs_int_delay="264"

One other thing to note, is that when I am testing going through the
router, there are many drops and stalls/restarts when downloading but
not uploading.

Before test:
dev.em.1.iflib.txq0.r_abdications: 0
dev.em.1.iflib.txq0.r_restarts: 0
dev.em.1.iflib.txq0.r_stalls: 0
dev.em.1.iflib.txq0.r_starts: 9239
dev.em.1.iflib.txq0.r_drops: 0
dev.em.1.iflib.txq0.r_enqueues: 9240
dev.em.1.iflib.txq0.txq_cleaned: 9640
dev.em.1.iflib.txq0.txq_processed: 9680

After test:
dev.em.1.iflib.txq0.r_abdications: 14
dev.em.1.iflib.txq0.r_restarts: 7419
dev.em.1.iflib.txq0.r_stalls: 7419
dev.em.1.iflib.txq0.r_starts: 411677
dev.em.1.iflib.txq0.r_drops: 198
dev.em.1.iflib.txq0.r_enqueues: 878486
dev.em.1.iflib.txq0.txq_cleaned: 875005
dev.em.1.iflib.txq0.txq_processed: 875045

After 2nd test with Marvell as outgoing NIC starting from zero:
dev.em.1.iflib.txq0.r_abdications: 1
dev.em.1.iflib.txq0.r_restarts: 7647
dev.em.1.iflib.txq0.r_stalls: 7647
dev.em.1.iflib.txq0.r_starts: 1092018
dev.em.1.iflib.txq0.r_drops: 811
dev.em.1.iflib.txq0.r_enqueues: 1570425
dev.em.1.iflib.txq0.txq_cleaned: 1567629
dev.em.1.iflib.txq0.txq_processed: 1567671

Unless someone knows anything else, I think I am going to have to buy a
PCI-E NIC for the router to replace one of the Intel NIC's.  I will
either use it along with the Marvell or the other Intel NIC.  When
testing between my workstation and any one Intel NIC on the router using
iperf3, I can achieve just over 900Mb/s.

Sean
--
[hidden email]



> On Sat, Nov 21, 2020 at 10:24 AM Sean C. Farley <[hidden email]> wrote:
>
>> I have recently upped my Internet service and have now noticed a limit
>> being reached, but I am not certain which limit and best option to
>> resolve it.
>>
>> I am using a circa 2007 system as a multi-purpose router running FreeBSD
>> 12-STABLE (r367740).  The issue is that it maxes out around 400Mb/s when
>> running a speed test through it between my workstation and various test
>> sites (i.e., DSL Reports and Speedtest).  There are two NIC's (both are
>> Intel 82541PI) in use with one to the ISP and one to my workstation.
>>
>> At first, I saw one of them apparently hitting an interrupt rate of just
>> over 8000, so I bumped their rate limits higher with little to no
>> improvement.
>>
>> What makes me believe I can theoretically get faster speeds is that I
>> can use the onboard NIC (Marvell 88E8056) to replace one of the NIC's
>> and nearly double the speed.  The difference is that it is on the PCI-E
>> bus and has MSI support.
>>
>> irq16: em0:irq0+
>> irq17: em1:irq0
>> irq20: hpet0
>> irq258: mskc0
>>
>> I have many network settings, but changing them did nothing.  Here are
>> the settings I am trying now that seem to squeak a little extra
>> performance.  The commented-out lines are ones I tried without seeing
>> any change.  I have also tested without these settings.
>>
>> /boot/loader.conf
>> hw.em.rx_process_limit="-1"
>> # dev.em.0.iflib.override_nrxds="2048"
>> # dev.em.1.iflib.override_nrxds="2048"
>> # dev.em.2.iflib.override_nrxds="2048"
>> # dev.em.0.iflib.override_ntxds="2048"
>> # net.link.ifqmaxlen="2048"
>> hw.em.max_interrupt_rate="32000"
>> # net.isr.maxthreads="-1"
>> # net.isr.bindthreads="1"
>>
>> /etc/sysctl.conf
>> kern.random.harvest.mask=351
>> dev.em.0.fc=0
>> dev.em.1.fc=0
>> dev.em.0.itr=122        # Allow past 8000 interrupts/second.
>> dev.em.1.itr=122
>> net.inet.ip.redirect=0
>> net.inet6.ip6.redirect=0
>>
>> Increasing these from 66 to 250 did not help:
>> hw.em.rx_abs_int_delay: 66
>> hw.em.tx_abs_int_delay: 66
>> hw.em.tx_int_delay: 66
>>
>> I am utilizing pf, but I doubt it is the issue since using the same
>> rules with the msk driver would have held the speed down to 400Mb/s.
>>
>> Am I hitting the limit of the PCI bus (memory or interrupt) or something
>> else?  I can buy a new PCI-E NIC for the internal network, but I rather
>> fully utilize the Intel NIC's I have, if possible.
>>
>> Sean
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Determining cause of transfer limit

Sean C. Farley-2
In reply to this post by Saad, Mark
On Sat, 21 Nov 2020, Saad, Mark wrote:

> scf
>  can I use polling on sfxge ? Also if I want to use polling on say ix
> or sfxge does this break things like pf, bird or nsd ?
>
> ---
> Mark Saad
> Lucera Financial Infrastructures, LLC
> [hidden email]

I wish I could help, but I have little knowledge about the device
polling capability.  Did you intend to ask Michael?

However, I can say that according to polling(4) man page, neither ix(4)
nor sfxge(4) are listed as supported by it.  I saw no change regarding
pf with polling in the kernel.  Finally, polling is recommended
(somewhere?) for slower hardware.  If you are using a 10Gb adapter, you
most likely have a faster system that does not need it.

You should ask in a separate E-mail thread if you have issues with that
driver or hardware.

Sean
--
[hidden email]

> ________________________________________
> From: [hidden email] <[hidden email]> on behalf of Michael Sierchio <[hidden email]>
> Sent: Saturday, November 21, 2020 4:55 PM
> To: Sean C. Farley
> Cc: [hidden email]
> Subject: Re: Determining cause of transfer limit
>
> Sorry for the top post.  Have you tried device polling? From
> /usr/src/sys/amd64/conf/NOTES:
> options         DEVICE_POLLING

*snip*

> On Sat, Nov 21, 2020 at 10:24 AM Sean C. Farley <[hidden email]> wrote:
>
>> I have recently upped my Internet service and have now noticed a limit
>> being reached, but I am not certain which limit and best option to
>> resolve it.

*snip*
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[hidden email]"