Netmap bridge not working with 10G Ethernet ports

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Netmap bridge not working with 10G Ethernet ports

Rajesh Kumar
Hi,

I am testing a 10G Network driver with Netmap "bridge" utility, where it
doesn't seem to work. Here is my setup details.

*System under Test:*  Running FreeBSD CURRENT.  Has two inbuilt 10G NIC
ports.
*System 1:* Running Ubuntu, whose network port is connected to Port1 of
System Under Test
*System 2:* Running FreeBSD CURRENT, whose network port is connected to
Port 0 of System Under Test.

Bridged the Port0 and Port1 of System Under Test using the Netmap "bridge"
utility. Able to see interfaces coming up active and Link UP.
# bridge -c -v -i netmap:ax0 -i netmap:ax1

Then tried pinging from System 1 to System 2. It fails.

*Observations:*
1. ARP request from System 1 goes to bridge port 1 (netmap_rxsync) and then
forwarded to port 0 (netmap_txsync)
2. ARP request is received in System 2 (via bridge port 0) and ARP reply is
being sent from System 2.
3. ARP reply from System 2 seems to be not reaching bridge port 0 to get
forwarded to bridge 1 and hence to System 1.
4. Above 3 steps happen 3 times for ARP resolution cycle and then fails.
Hence the ping fails.

On Debugging, when the ARP reply is being sent from System 2, I don't see
any interrupt triggered on the bridge port 0 in system under test.

Netstat in system under test, doesn't show any receive or drop counters
incremented. But as I understand netstat capture the stats above the netmap
stack. Hence not reflecting the counts.

*Note:*
a) I tried with another vendor 10G NIC card. It behaves the same way. So
this issue doesn't seem to be generic and not hardware specific.
b) Trying with another vendor 1G NIC card, things are working.  So not
sure, what makes a difference here.  The ports in System 1 and System 2 are
USB attached Ethernet capable of maximum speed of 1G.  So does connecting
1G to 10G bridge ports is having any impact?
c) We have verified the same 10G driver with pkt-gen utility and things are
working. Facing issue only when using "bridge" utility.

So, wondering how the ARP reply packet is getting lost here. Any ideas to
debug?

Thanks,
Rajesh.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Netmap bridge not working with 10G Ethernet ports

Vincenzo Maffione-2
Hi,

Il giorno mer 18 nov 2020 alle ore 08:13 Rajesh Kumar <[hidden email]>
ha scritto:

> Hi,
>
> I am testing a 10G Network driver with Netmap "bridge" utility, where it
> doesn't seem to work. Here is my setup details.
>
> *System under Test:*  Running FreeBSD CURRENT.  Has two inbuilt 10G NIC
> ports.
> *System 1:* Running Ubuntu, whose network port is connected to Port1 of
> System Under Test
> *System 2:* Running FreeBSD CURRENT, whose network port is connected to
> Port 0 of System Under Test.
>
> Bridged the Port0 and Port1 of System Under Test using the Netmap "bridge"
> utility. Able to see interfaces coming up active and Link UP.
> # bridge -c -v -i netmap:ax0 -i netmap:ax1
>
> This looks like if_axe(4) driver, and therefore there's no native netmap
support, which means you are falling back on
the emulated netmap adapter. Are these USB dongles? If so, how can they be
10G?


> Then tried pinging from System 1 to System 2. It fails.
>
> *Observations:*
> 1. ARP request from System 1 goes to bridge port 1 (netmap_rxsync) and then
> forwarded to port 0 (netmap_txsync)
> 2. ARP request is received in System 2 (via bridge port 0) and ARP reply is
> being sent from System 2.
> 3. ARP reply from System 2 seems to be not reaching bridge port 0 to get
> forwarded to bridge 1 and hence to System 1.
> 4. Above 3 steps happen 3 times for ARP resolution cycle and then fails.
> Hence the ping fails.
>
> On Debugging, when the ARP reply is being sent from System 2, I don't see
> any interrupt triggered on the bridge port 0 in system under test.
>
> In this kind of configuration it is mandatory to disable all the NIC
offloads, because netmap does not program the NIC
to honor them, e.g.:

# ifconfig ax0 -txcsum -rxcsum -tso4 -tso6 -lro -txcsum6 -rxcsum6
# ifconfig ax1 -txcsum -rxcsum -tso4 -tso6 -lro -txcsum6 -rxcsum6


> Netstat in system under test, doesn't show any receive or drop counters
> incremented. But as I understand netstat capture the stats above the netmap
> stack. Hence not reflecting the counts.
>

Correct.


>
> *Note:*
> a) I tried with another vendor 10G NIC card. It behaves the same way. So
> this issue doesn't seem to be generic and not hardware specific.
>

Which driver are those NICs using? That makes the difference. I guess it's
still a driver with no native netmap support, hence
you are using the same emulated adapter.


> b) Trying with another vendor 1G NIC card, things are working.  So not
> sure, what makes a difference here.  The ports in System 1 and System 2 are
> USB attached Ethernet capable of maximum speed of 1G.  So does connecting
> 1G to 10G bridge ports is having any impact?
>

I don't think so. On each p2p link the NICs will negotiate 1G speed.
In any case, what driver was this one?


> c) We have verified the same 10G driver with pkt-gen utility and things are
> working. Facing issue only when using "bridge" utility.
>

That may be because pkt-gen does not care about checksums, whereas the
TCP/IP stack does.
Hence the need to disable offloads (see above).

Cheers,
  Vincenzo


> So, wondering how the ARP reply packet is getting lost here. Any ideas to
> debug?
>
> Thanks,
> Rajesh.
> _______________________________________________
> [hidden email] mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "[hidden email]"
>
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Netmap bridge not working with 10G Ethernet ports

Rajesh Kumar
Hi Vincenzo,

Thanks for your reply.

On Thu, Nov 19, 2020 at 3:16 AM Vincenzo Maffione <[hidden email]>
wrote:

>
> This looks like if_axe(4) driver, and therefore there's no native netmap
> support, which means you are falling back on
> the emulated netmap adapter. Are these USB dongles? If so, how can they be
> 10G?
>

The Driver I am working with is "if_axp" (sys/dev/axgbe).  This is AMD
10Gigabit Ethernet Driver. This is recently committed upstream. Yes, it
doesn't have a Native netmap support, but uses the netmap stack which is
existing already.  These are inbuilt SFP ports with our test board and not
USB dongles.

Does Native netmap mean the hardware capability which needs to be
programmed appropriately from driver side?  Any generic documentation
regarding the same?


> In this kind of configuration it is mandatory to disable all the NIC
> offloads, because netmap does not program the NIC
> to honor them, e.g.:
>
> # ifconfig ax0 -txcsum -rxcsum -tso4 -tso6 -lro -txcsum6 -rxcsum6
> # ifconfig ax1 -txcsum -rxcsum -tso4 -tso6 -lro -txcsum6 -rxcsum6
>

Earlier, I haven't tried disabling the Offload capabilities.  But I tried
now, but it still behaves the same way.  ARP replies doesn't seem to reach
the bridge (or dropped) to be forwarded.  I will collect the details for
AMD driver. Tried the same test with another 10G card (Intel "ix" driver)
also exhibits similar behavior.  Details below.


> a) I tried with another vendor 10G NIC card. It behaves the same way. So
>> this issue doesn't seem to be generic and not hardware specific.
>>
>
> Which driver are those NICs using? That makes the difference. I guess it's
> still a driver with no native netmap support, hence
> you are using the same emulated adapter
>

I am using the "ix" driver (Intel 10G NIC adapter).  I guess this driver
also doesn't support Native Netmap.  Please correct me if I am wrong.  I
tried disabling the offload capabilities with this device/driver and tested
and still observed the netmap bridging fails.

root@fbsd_cur# sysctl dev.ix.0 | grep tx_packets
dev.ix.0.queue7.tx_packets: 0
dev.ix.0.queue6.tx_packets: 0
dev.ix.0.queue5.tx_packets: 0
dev.ix.0.queue4.tx_packets: 0
dev.ix.0.queue3.tx_packets: 0
dev.ix.0.queue2.tx_packets: 0
dev.ix.0.queue1.tx_packets: 0
*dev.ix.0.queue0.tx_packets: 3*
root@fbsd_cur# sysctl dev.ix.0 | grep rx_packets
dev.ix.0.queue7.rx_packets: 0
dev.ix.0.queue6.rx_packets: 0
dev.ix.0.queue5.rx_packets: 0
dev.ix.0.queue4.rx_packets: 0
dev.ix.0.queue3.rx_packets: 0
dev.ix.0.queue2.rx_packets: 0
dev.ix.0.queue1.rx_packets: 0
dev.ix.0.queue0.rx_packets: 0
root@fbsd_cur # sysctl dev.ix.1 | grep tx_packets
dev.ix.1.queue7.tx_packets: 0
dev.ix.1.queue6.tx_packets: 0
dev.ix.1.queue5.tx_packets: 0
dev.ix.1.queue4.tx_packets: 0
dev.ix.1.queue3.tx_packets: 0
dev.ix.1.queue2.tx_packets: 0
dev.ix.1.queue1.tx_packets: 0
dev.ix.1.queue0.tx_packets: 0
root@fbsd_cur # sysctl dev.ix.1 | grep rx_packets
dev.ix.1.queue7.rx_packets: 0
dev.ix.1.queue6.rx_packets: 0
dev.ix.1.queue5.rx_packets: 0
dev.ix.1.queue4.rx_packets: 0
dev.ix.1.queue3.rx_packets: 0
dev.ix.1.queue2.rx_packets: 0
dev.ix.1.queue1.rx_packets: 0

*dev.ix.1.queue0.rx_packets: 3*

You can see "ix1" received 3 packets (ARP requests) from system 1 and
transmitted 3 packets to system 2 via "ix0". But ARP reply from system 2 is
not captured or forwared properly.

You can see the checksum features disabled (except VLAN_HWCSIM) on both
interfaces.  And you can see both interfaces active and Link up.

root@fbsd_cur # ifconfig -a
ix0: flags=8862<BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500

options=48538b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO>
        ether a0:36:9f:a5:49:90
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

ix1: flags=8862<BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500

options=48538b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO>
        ether a0:36:9f:a5:49:92
        media: Ethernet autoselect (1000baseT <full-duplex,rxpause,txpause>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

>
> b) Trying with another vendor 1G NIC card, things are working.  So not
>> sure, what makes a difference here.  The ports in System 1 and System 2
>> are
>> USB attached Ethernet capable of maximum speed of 1G.  So does connecting
>> 1G to 10G bridge ports is having any impact?
>>
>
> I don't think so. On each p2p link the NICs will negotiate 1G speed.
> In any case, what driver was this one?
>

This is "igb" driver. Intel 1G NIC Card.

Thanks,
Rajesh.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Netmap bridge not working with 10G Ethernet ports

Vincenzo Maffione-2
Il giorno gio 19 nov 2020 alle ore 12:28 Rajesh Kumar <[hidden email]>
ha scritto:

> Hi Vincenzo,
>
> Thanks for your reply.
>
> On Thu, Nov 19, 2020 at 3:16 AM Vincenzo Maffione <[hidden email]>
> wrote:
>
>>
>> This looks like if_axe(4) driver, and therefore there's no native netmap
>> support, which means you are falling back on
>> the emulated netmap adapter. Are these USB dongles? If so, how can they
>> be 10G?
>>
>
> The Driver I am working with is "if_axp" (sys/dev/axgbe).  This is AMD
> 10Gigabit Ethernet Driver. This is recently committed upstream. Yes, it
> doesn't have a Native netmap support, but uses the netmap stack which is
> existing already.  These are inbuilt SFP ports with our test board and not
> USB dongles.
>

Ok, now it makes sense. Thanks for clarifying. I see that if_axp(4) uses
iflib(4). This means that actually if_axp(4) has native netmap support,
because iflib(4) has native netmap support.


> Does Native netmap mean the hardware capability which needs to be
> programmed appropriately from driver side?  Any generic documentation
> regarding the same?
>

It means that the driver has some modifications to allow netmap to directly
program the NIC rings. These modifications are mostly the per-driver txsync
and rxsyng routines.
In case of iflib(4) drivers, these modifications are provided directly
within the iflib(4) code, and therefore any driver using iflib will have
native netmap support.


>
>> In this kind of configuration it is mandatory to disable all the NIC
>> offloads, because netmap does not program the NIC
>> to honor them, e.g.:
>>
>> # ifconfig ax0 -txcsum -rxcsum -tso4 -tso6 -lro -txcsum6 -rxcsum6
>> # ifconfig ax1 -txcsum -rxcsum -tso4 -tso6 -lro -txcsum6 -rxcsum6
>>
>
> Earlier, I haven't tried disabling the Offload capabilities.  But I tried
> now, but it still behaves the same way.  ARP replies doesn't seem to reach
> the bridge (or dropped) to be forwarded.  I will collect the details for
> AMD driver. Tried the same test with another 10G card (Intel "ix" driver)
> also exhibits similar behavior.  Details below.
>

Ok, this makes sense, because also ix(4) uses iflib, and therefore you are
basically hitting the same issue of if_axp(4)
At this point I must think that there is still some issue with the
interaction between iflib(4) and netmap(4).


>
>
>> a) I tried with another vendor 10G NIC card. It behaves the same way. So
>>> this issue doesn't seem to be generic and not hardware specific.
>>>
>>
>> Which driver are those NICs using? That makes the difference. I guess
>> it's still a driver with no native netmap support, hence
>> you are using the same emulated adapter
>>
>
> I am using the "ix" driver (Intel 10G NIC adapter).  I guess this driver
> also doesn't support Native Netmap.  Please correct me if I am wrong.  I
> tried disabling the offload capabilities with this device/driver and tested
> and still observed the netmap bridging fails.
>

As I stated above, ix(4) has netmap support, like any iflib(4) driver.


> root@fbsd_cur# sysctl dev.ix.0 | grep tx_packets
> dev.ix.0.queue7.tx_packets: 0
> dev.ix.0.queue6.tx_packets: 0
> dev.ix.0.queue5.tx_packets: 0
> dev.ix.0.queue4.tx_packets: 0
> dev.ix.0.queue3.tx_packets: 0
> dev.ix.0.queue2.tx_packets: 0
> dev.ix.0.queue1.tx_packets: 0
> *dev.ix.0.queue0.tx_packets: 3*
> root@fbsd_cur# sysctl dev.ix.0 | grep rx_packets
> dev.ix.0.queue7.rx_packets: 0
> dev.ix.0.queue6.rx_packets: 0
> dev.ix.0.queue5.rx_packets: 0
> dev.ix.0.queue4.rx_packets: 0
> dev.ix.0.queue3.rx_packets: 0
> dev.ix.0.queue2.rx_packets: 0
> dev.ix.0.queue1.rx_packets: 0
> dev.ix.0.queue0.rx_packets: 0
> root@fbsd_cur # sysctl dev.ix.1 | grep tx_packets
> dev.ix.1.queue7.tx_packets: 0
> dev.ix.1.queue6.tx_packets: 0
> dev.ix.1.queue5.tx_packets: 0
> dev.ix.1.queue4.tx_packets: 0
> dev.ix.1.queue3.tx_packets: 0
> dev.ix.1.queue2.tx_packets: 0
> dev.ix.1.queue1.tx_packets: 0
> dev.ix.1.queue0.tx_packets: 0
> root@fbsd_cur # sysctl dev.ix.1 | grep rx_packets
> dev.ix.1.queue7.rx_packets: 0
> dev.ix.1.queue6.rx_packets: 0
> dev.ix.1.queue5.rx_packets: 0
> dev.ix.1.queue4.rx_packets: 0
> dev.ix.1.queue3.rx_packets: 0
> dev.ix.1.queue2.rx_packets: 0
> dev.ix.1.queue1.rx_packets: 0
>
> *dev.ix.1.queue0.rx_packets: 3*
>
> You can see "ix1" received 3 packets (ARP requests) from system 1 and
> transmitted 3 packets to system 2 via "ix0". But ARP reply from system 2 is
> not captured or forwared properly.
>

I see. This info may be useful. Have you tried to look at interrupts (e.g.
`vmstat -i`), to see if "ix0" gets any RX interrupts (for the missing ARP
replies)?


>
> You can see the checksum features disabled (except VLAN_HWCSIM) on both
> interfaces.  And you can see both interfaces active and Link up.
>
> root@fbsd_cur # ifconfig -a
> ix0: flags=8862<BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
>
> options=48538b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO>
>         ether a0:36:9f:a5:49:90
>         media: Ethernet autoselect (100baseTX <full-duplex>)
>         status: active
>         nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
>
> ix1: flags=8862<BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
>
> options=48538b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO>
>         ether a0:36:9f:a5:49:92
>         media: Ethernet autoselect (1000baseT
> <full-duplex,rxpause,txpause>)
>         status: active
>         nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
>
>>
>> b) Trying with another vendor 1G NIC card, things are working.  So not
>>> sure, what makes a difference here.  The ports in System 1 and System 2
>>> are
>>> USB attached Ethernet capable of maximum speed of 1G.  So does connecting
>>> 1G to 10G bridge ports is having any impact?
>>>
>>
>> I don't think so. On each p2p link the NICs will negotiate 1G speed.
>> In any case, what driver was this one?
>>
>
> This is "igb" driver. Intel 1G NIC Card.
>

Also the igb(4) driver is using iflib(4). So the involved netmap code is
the same as ix(4) and if_axp(4).
This is something that I'm not able to understand right now.
It does not look like something related to offloads.

Next week I will try to see if I can reproduce your issue with em(4), and
report back. That's still an Intel driver using iflib(4).

Thanks,
  Vincenzo


>
> Thanks,
> Rajesh.
>
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Netmap bridge not working with 10G Ethernet ports

Rajesh Kumar
Hi Vincenzo,

On Fri, Nov 20, 2020 at 3:20 AM Vincenzo Maffione <[hidden email]>
wrote:

>
> Ok, now it makes sense. Thanks for clarifying. I see that if_axp(4) uses
> iflib(4). This means that actually if_axp(4) has native netmap support,
> because iflib(4) has native netmap support.
>
>
It means that the driver has some modifications to allow netmap to directly
> program the NIC rings. These modifications are mostly the per-driver txsync
> and rxsyng routines.
> In case of iflib(4) drivers, these modifications are provided directly
> within the iflib(4) code, and therefore any driver using iflib will have
> native netmap support.
>

Thanks for clarifying on the Native Netmap support.

Ok, this makes sense, because also ix(4) uses iflib, and therefore you are
> basically hitting the same issue of if_axp(4)
> At this point I must think that there is still some issue with the
> interaction between iflib(4) and netmap(4).
>

Ok. Let me know if any more debug info needed in this part.

I see. This info may be useful. Have you tried to look at interrupts (e.g.
> `vmstat -i`), to see if "ix0" gets any RX interrupts (for the missing ARP
> replies)?
>

It's interesting here. When I try with Intel NIC card. I see atleast 1
interrupt raised.  But not sure whether that is for ARP reply. Because,
when I try to dump the packet from "bridge"(modified) utility, I don't see
any ARP reply packet getting dumped.


*irq59: ix0:rxq0                        1          0 (only 1 interrupt on
the opposite side)*irq67: ix0:aq                          2          0

*irq68: ix1:rxq0                        3          0  (you can see 3
interrupts for 3 ARP requests from System 1)*irq76: ix1:aq
         2          0

The same experiment, when I try with AMD inbuilt ports, I don't see that 1
interrupt also raised.

irq81: ax0:dev_irq                    16          0
irq83: ax0                          2541          4
irq93: ax1:dev_irq                    27          0
irq95: ax1                          2371          3
*irq97: ax1:rxq0                        3          0 (you can see 3
interrupts for 3 ARP requests from System 1, but no interrupt is seen from
"ax0:rxq0" for ARP reply from System 2)*

I will do some more testing to see whether this behavior is consistent or
intermittent.

Also the igb(4) driver is using iflib(4). So the involved netmap code is
> the same as ix(4) and if_axp(4).
> This is something that I'm not able to understand right now.
> It does not look like something related to offloads.
>
> Next week I will try to see if I can reproduce your issue with em(4), and
> report back. That's still an Intel driver using iflib(4).
>

The "igb(4)" driver, with which things are working now is related to em(4)
driver (may be for newer hardware version).  Initially we faced similar
issue with igb(4) driver as well. After reverting the following commits,
things started to work.  Thanks to Stephan Dewt (copied) for pointing
this.  But it still fails with ix(4) driver and if_axp(4) driver.

https://github.com/freebsd/freebsd/commit/e12efc2c9e434075d0740e2e2e9e2fca2ad5f7cf

Thanks for providing your inputs on this issue Vincenzo.  Let me know for
any more details that you need.

Thanks,
Rajesh.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Netmap bridge not working with 10G Ethernet ports

Vincenzo Maffione-2
Il giorno ven 20 nov 2020 alle ore 14:31 Rajesh Kumar <[hidden email]>
ha scritto:

> Hi Vincenzo,
>
> On Fri, Nov 20, 2020 at 3:20 AM Vincenzo Maffione <[hidden email]>
> wrote:
>
>>
>> Ok, now it makes sense. Thanks for clarifying. I see that if_axp(4) uses
>> iflib(4). This means that actually if_axp(4) has native netmap support,
>> because iflib(4) has native netmap support.
>>
>>
> It means that the driver has some modifications to allow netmap to
>> directly program the NIC rings. These modifications are mostly the
>> per-driver txsync and rxsyng routines.
>> In case of iflib(4) drivers, these modifications are provided directly
>> within the iflib(4) code, and therefore any driver using iflib will have
>> native netmap support.
>>
>
> Thanks for clarifying on the Native Netmap support.
>
> Ok, this makes sense, because also ix(4) uses iflib, and therefore you are
>> basically hitting the same issue of if_axp(4)
>> At this point I must think that there is still some issue with the
>> interaction between iflib(4) and netmap(4).
>>
>
> Ok. Let me know if any more debug info needed in this part.
>
> I see. This info may be useful. Have you tried to look at interrupts (e.g.
>> `vmstat -i`), to see if "ix0" gets any RX interrupts (for the missing ARP
>> replies)?
>>
>
> It's interesting here. When I try with Intel NIC card. I see atleast 1
> interrupt raised.  But not sure whether that is for ARP reply. Because,
> when I try to dump the packet from "bridge"(modified) utility, I don't see
> any ARP reply packet getting dumped.
>
>
> *irq59: ix0:rxq0                        1          0 (only 1 interrupt on
> the opposite side)*irq67: ix0:aq                          2          0
>
> *irq68: ix1:rxq0                        3          0  (you can see 3
> interrupts for 3 ARP requests from System 1)*irq76: ix1:aq
>            2          0
>
> The same experiment, when I try with AMD inbuilt ports, I don't see that 1
> interrupt also raised.
>
> irq81: ax0:dev_irq                    16          0
> irq83: ax0                          2541          4
> irq93: ax1:dev_irq                    27          0
> irq95: ax1                          2371          3
> *irq97: ax1:rxq0                        3          0 (you can see 3
> interrupts for 3 ARP requests from System 1, but no interrupt is seen from
> "ax0:rxq0" for ARP reply from System 2)*
>
> I will do some more testing to see whether this behavior is consistent or
> intermittent.
>
> Also the igb(4) driver is using iflib(4). So the involved netmap code is
>> the same as ix(4) and if_axp(4).
>> This is something that I'm not able to understand right now.
>> It does not look like something related to offloads.
>>
>> Next week I will try to see if I can reproduce your issue with em(4), and
>> report back. That's still an Intel driver using iflib(4).
>>
>
> The "igb(4)" driver, with which things are working now is related to em(4)
> driver (may be for newer hardware version).  Initially we faced similar
> issue with igb(4) driver as well. After reverting the following commits,
> things started to work.  Thanks to Stephan Dewt (copied) for pointing
> this.  But it still fails with ix(4) driver and if_axp(4) driver.
>
>
> https://github.com/freebsd/freebsd/commit/e12efc2c9e434075d0740e2e2e9e2fca2ad5f7cf
>
> Thanks for providing your inputs on this issue Vincenzo.  Let me know for
> any more details that you need.
>
>
I was able to reproduce your issue on FreeBSD-CURRENT running within a QEMU
VM, with two em(4) devices and the netmap bridge running between them.
I see the ARP request packet received on em0 (with associated IRQ), and
forwarded on em1. However, the ARP reply coming on em1 does not trigger an
IRQ on em1, and indeed the NIC RX head/tail pointers are not incremented as
they should (`sysctl -a | grep em.1 | grep queue_rx`) ... that is weird,
and lets me think that the issue is more likely driver-related than
netmap/iflib-related.
In any case, would you mind filing the issue on the bugzilla, so that we
can properly track this issue?

Thanks,
  Vincenzo


> Thanks,
> Rajesh.
>
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Netmap bridge not working with 10G Ethernet ports

Vincenzo Maffione-2
Hi Rajesh,
  I think the issue here is simply that you have not enabled promiscuous
mode on your interfaces.
# ifconfig ix0 promisc
# ifconfig ix1 promisc

This is an additional requirement when using netmap bridge, because that is
not done automatically (differently from what happens with if_bridge(4)).
If promisc is not enabled, the NIC will drop any unicast packet that is not
directed to the NIC's address (e.g. the ARP reply in your case). Broadcast
packets will of course pass (e.g. the ARP request). This explains the
absence of IRQs and the head/tail pointers not being updated.
So no bugs AFAIK.

I figured it out the hard way, but it was actually also documented on the
github (https://github.com/luigirizzo/netmap#receiver-does-not-receive).
I will add it to the netmap bridge man page.

Cheers,
  Vincenzo


Il giorno sab 21 nov 2020 alle ore 17:06 Vincenzo Maffione <
[hidden email]> ha scritto:

>
>
> Il giorno ven 20 nov 2020 alle ore 14:31 Rajesh Kumar <[hidden email]>
> ha scritto:
>
>> Hi Vincenzo,
>>
>> On Fri, Nov 20, 2020 at 3:20 AM Vincenzo Maffione <[hidden email]>
>> wrote:
>>
>>>
>>> Ok, now it makes sense. Thanks for clarifying. I see that if_axp(4) uses
>>> iflib(4). This means that actually if_axp(4) has native netmap support,
>>> because iflib(4) has native netmap support.
>>>
>>>
>> It means that the driver has some modifications to allow netmap to
>>> directly program the NIC rings. These modifications are mostly the
>>> per-driver txsync and rxsyng routines.
>>> In case of iflib(4) drivers, these modifications are provided directly
>>> within the iflib(4) code, and therefore any driver using iflib will have
>>> native netmap support.
>>>
>>
>> Thanks for clarifying on the Native Netmap support.
>>
>> Ok, this makes sense, because also ix(4) uses iflib, and therefore you
>>> are basically hitting the same issue of if_axp(4)
>>> At this point I must think that there is still some issue with the
>>> interaction between iflib(4) and netmap(4).
>>>
>>
>> Ok. Let me know if any more debug info needed in this part.
>>
>> I see. This info may be useful. Have you tried to look at interrupts
>>> (e.g. `vmstat -i`), to see if "ix0" gets any RX interrupts (for the missing
>>> ARP replies)?
>>>
>>
>> It's interesting here. When I try with Intel NIC card. I see atleast 1
>> interrupt raised.  But not sure whether that is for ARP reply. Because,
>> when I try to dump the packet from "bridge"(modified) utility, I don't see
>> any ARP reply packet getting dumped.
>>
>>
>> *irq59: ix0:rxq0                        1          0 (only 1 interrupt on
>> the opposite side)*irq67: ix0:aq                          2          0
>>
>> *irq68: ix1:rxq0                        3          0  (you can see 3
>> interrupts for 3 ARP requests from System 1)*irq76: ix1:aq
>>            2          0
>>
>> The same experiment, when I try with AMD inbuilt ports, I don't see that
>> 1 interrupt also raised.
>>
>> irq81: ax0:dev_irq                    16          0
>> irq83: ax0                          2541          4
>> irq93: ax1:dev_irq                    27          0
>> irq95: ax1                          2371          3
>> *irq97: ax1:rxq0                        3          0 (you can see 3
>> interrupts for 3 ARP requests from System 1, but no interrupt is seen from
>> "ax0:rxq0" for ARP reply from System 2)*
>>
>> I will do some more testing to see whether this behavior is consistent or
>> intermittent.
>>
>> Also the igb(4) driver is using iflib(4). So the involved netmap code is
>>> the same as ix(4) and if_axp(4).
>>> This is something that I'm not able to understand right now.
>>> It does not look like something related to offloads.
>>>
>>> Next week I will try to see if I can reproduce your issue with em(4),
>>> and report back. That's still an Intel driver using iflib(4).
>>>
>>
>> The "igb(4)" driver, with which things are working now is related to
>> em(4) driver (may be for newer hardware version).  Initially we faced
>> similar issue with igb(4) driver as well. After reverting the following
>> commits, things started to work.  Thanks to Stephan Dewt (copied) for
>> pointing this.  But it still fails with ix(4) driver and if_axp(4) driver.
>>
>>
>> https://github.com/freebsd/freebsd/commit/e12efc2c9e434075d0740e2e2e9e2fca2ad5f7cf
>>
>> Thanks for providing your inputs on this issue Vincenzo.  Let me know for
>> any more details that you need.
>>
>>
> I was able to reproduce your issue on FreeBSD-CURRENT running within a
> QEMU VM, with two em(4) devices and the netmap bridge running between them.
> I see the ARP request packet received on em0 (with associated IRQ), and
> forwarded on em1. However, the ARP reply coming on em1 does not trigger an
> IRQ on em1, and indeed the NIC RX head/tail pointers are not incremented as
> they should (`sysctl -a | grep em.1 | grep queue_rx`) ... that is weird,
> and lets me think that the issue is more likely driver-related than
> netmap/iflib-related.
> In any case, would you mind filing the issue on the bugzilla, so that we
> can properly track this issue?
>
> Thanks,
>   Vincenzo
>
>
>> Thanks,
>> Rajesh.
>>
>
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Netmap bridge not working with 10G Ethernet ports

Rajesh Kumar
Hi Vincenzo,

Thanks for pointing this.

On Sat, Nov 21, 2020 at 10:40 PM Vincenzo Maffione <[hidden email]>
wrote:

> # ifconfig ix0 promisc
> # ifconfig ix1 promisc
>
> This is an additional requirement when using netmap bridge, because that
> is not done automatically (differently from what happens with if_bridge(4)).
> If promisc is not enabled, the NIC will drop any unicast packet that is
> not directed to the NIC's address (e.g. the ARP reply in your case).
> Broadcast packets will of course pass (e.g. the ARP request). This explains
> the absence of IRQs and the head/tail pointers not being updated.
> So no bugs AFAIK.
>

Setting the interfaces in promiscous mode makes things to work properly.

I tried the same with AMD Ports and it's still not working.  I believe this
is something specific to if_axp driver. I will see what is going wrong with
packet forwarding with AMD ports. Thanks for pointing this out.

I figured it out the hard way, but it was actually also documented on the
> github (https://github.com/luigirizzo/netmap#receiver-does-not-receive).
> I will add it to the netmap bridge man page.
>

That would be helpful. Thanks.


> Il giorno sab 21 nov 2020 alle ore 17:06 Vincenzo Maffione <
> [hidden email]> ha scritto:
>
>>
>>
>> Il giorno ven 20 nov 2020 alle ore 14:31 Rajesh Kumar <[hidden email]>
>> ha scritto:
>>
>>> Hi Vincenzo,
>>>
>>> On Fri, Nov 20, 2020 at 3:20 AM Vincenzo Maffione <[hidden email]>
>>> wrote:
>>>
>>>>
>>>> Ok, now it makes sense. Thanks for clarifying. I see that if_axp(4)
>>>> uses iflib(4). This means that actually if_axp(4) has native netmap
>>>> support, because iflib(4) has native netmap support.
>>>>
>>>>
>>> It means that the driver has some modifications to allow netmap to
>>>> directly program the NIC rings. These modifications are mostly the
>>>> per-driver txsync and rxsyng routines.
>>>> In case of iflib(4) drivers, these modifications are provided directly
>>>> within the iflib(4) code, and therefore any driver using iflib will have
>>>> native netmap support.
>>>>
>>>
>>> Thanks for clarifying on the Native Netmap support.
>>>
>>> Ok, this makes sense, because also ix(4) uses iflib, and therefore you
>>>> are basically hitting the same issue of if_axp(4)
>>>> At this point I must think that there is still some issue with the
>>>> interaction between iflib(4) and netmap(4).
>>>>
>>>
>>> Ok. Let me know if any more debug info needed in this part.
>>>
>>> I see. This info may be useful. Have you tried to look at interrupts
>>>> (e.g. `vmstat -i`), to see if "ix0" gets any RX interrupts (for the missing
>>>> ARP replies)?
>>>>
>>>
>>> It's interesting here. When I try with Intel NIC card. I see atleast 1
>>> interrupt raised.  But not sure whether that is for ARP reply. Because,
>>> when I try to dump the packet from "bridge"(modified) utility, I don't see
>>> any ARP reply packet getting dumped.
>>>
>>>
>>> *irq59: ix0:rxq0                        1          0 (only 1 interrupt
>>> on the opposite side)*irq67: ix0:aq                          2
>>>  0
>>>
>>> *irq68: ix1:rxq0                        3          0  (you can see 3
>>> interrupts for 3 ARP requests from System 1)*irq76: ix1:aq
>>>              2          0
>>>
>>> The same experiment, when I try with AMD inbuilt ports, I don't see that
>>> 1 interrupt also raised.
>>>
>>> irq81: ax0:dev_irq                    16          0
>>> irq83: ax0                          2541          4
>>> irq93: ax1:dev_irq                    27          0
>>> irq95: ax1                          2371          3
>>> *irq97: ax1:rxq0                        3          0 (you can see 3
>>> interrupts for 3 ARP requests from System 1, but no interrupt is seen from
>>> "ax0:rxq0" for ARP reply from System 2)*
>>>
>>> I will do some more testing to see whether this behavior is consistent
>>> or intermittent.
>>>
>>> Also the igb(4) driver is using iflib(4). So the involved netmap code is
>>>> the same as ix(4) and if_axp(4).
>>>> This is something that I'm not able to understand right now.
>>>> It does not look like something related to offloads.
>>>>
>>>> Next week I will try to see if I can reproduce your issue with em(4),
>>>> and report back. That's still an Intel driver using iflib(4).
>>>>
>>>
>>> The "igb(4)" driver, with which things are working now is related to
>>> em(4) driver (may be for newer hardware version).  Initially we faced
>>> similar issue with igb(4) driver as well. After reverting the following
>>> commits, things started to work.  Thanks to Stephan Dewt (copied) for
>>> pointing this.  But it still fails with ix(4) driver and if_axp(4) driver.
>>>
>>>
>>> https://github.com/freebsd/freebsd/commit/e12efc2c9e434075d0740e2e2e9e2fca2ad5f7cf
>>>
>>> Thanks for providing your inputs on this issue Vincenzo.  Let me know
>>> for any more details that you need.
>>>
>>>
>> I was able to reproduce your issue on FreeBSD-CURRENT running within a
>> QEMU VM, with two em(4) devices and the netmap bridge running between them.
>> I see the ARP request packet received on em0 (with associated IRQ), and
>> forwarded on em1. However, the ARP reply coming on em1 does not trigger an
>> IRQ on em1, and indeed the NIC RX head/tail pointers are not incremented as
>> they should (`sysctl -a | grep em.1 | grep queue_rx`) ... that is weird,
>> and lets me think that the issue is more likely driver-related than
>> netmap/iflib-related.
>> In any case, would you mind filing the issue on the bugzilla, so that we
>> can properly track this issue?
>>
>> Thanks,
>>   Vincenzo
>>
>>
>>> Thanks,
>>> Rajesh.
>>>
>>
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Netmap bridge not working with 10G Ethernet ports

Vincenzo Maffione-2
Il giorno mar 24 nov 2020 alle ore 03:57 Rajesh Kumar <[hidden email]>
ha scritto:

> Hi Vincenzo,
>
> Thanks for pointing this.
>
> On Sat, Nov 21, 2020 at 10:40 PM Vincenzo Maffione <[hidden email]>
> wrote:
>
>> # ifconfig ix0 promisc
>> # ifconfig ix1 promisc
>>
>> This is an additional requirement when using netmap bridge, because that
>> is not done automatically (differently from what happens with if_bridge(4)).
>> If promisc is not enabled, the NIC will drop any unicast packet that is
>> not directed to the NIC's address (e.g. the ARP reply in your case).
>> Broadcast packets will of course pass (e.g. the ARP request). This explains
>> the absence of IRQs and the head/tail pointers not being updated.
>> So no bugs AFAIK.
>>
>
> Setting the interfaces in promiscous mode makes things to work properly.
>
> I tried the same with AMD Ports and it's still not working.  I believe
> this is something specific to if_axp driver. I will see what is going wrong
> with packet forwarding with AMD ports. Thanks for pointing this out.
>

Yeah, it's weird because axgbe also uses iflib(4). If the driver exposes
NIC head/tail pointers (sysctl) it may be useful to check what happens
there.
It may be that the NIC is dropping these packets for some reason.


>
> I figured it out the hard way, but it was actually also documented on the
>> github (https://github.com/luigirizzo/netmap#receiver-does-not-receive).
>> I will add it to the netmap bridge man page.
>>
>
> That would be helpful. Thanks.
>

Done in r367920.

Cheers,
  Vincenzo


>
>
>> Il giorno sab 21 nov 2020 alle ore 17:06 Vincenzo Maffione <
>> [hidden email]> ha scritto:
>>
>>>
>>>
>>> Il giorno ven 20 nov 2020 alle ore 14:31 Rajesh Kumar <[hidden email]>
>>> ha scritto:
>>>
>>>> Hi Vincenzo,
>>>>
>>>> On Fri, Nov 20, 2020 at 3:20 AM Vincenzo Maffione <
>>>> [hidden email]> wrote:
>>>>
>>>>>
>>>>> Ok, now it makes sense. Thanks for clarifying. I see that if_axp(4)
>>>>> uses iflib(4). This means that actually if_axp(4) has native netmap
>>>>> support, because iflib(4) has native netmap support.
>>>>>
>>>>>
>>>> It means that the driver has some modifications to allow netmap to
>>>>> directly program the NIC rings. These modifications are mostly the
>>>>> per-driver txsync and rxsyng routines.
>>>>> In case of iflib(4) drivers, these modifications are provided directly
>>>>> within the iflib(4) code, and therefore any driver using iflib will have
>>>>> native netmap support.
>>>>>
>>>>
>>>> Thanks for clarifying on the Native Netmap support.
>>>>
>>>> Ok, this makes sense, because also ix(4) uses iflib, and therefore you
>>>>> are basically hitting the same issue of if_axp(4)
>>>>> At this point I must think that there is still some issue with the
>>>>> interaction between iflib(4) and netmap(4).
>>>>>
>>>>
>>>> Ok. Let me know if any more debug info needed in this part.
>>>>
>>>> I see. This info may be useful. Have you tried to look at interrupts
>>>>> (e.g. `vmstat -i`), to see if "ix0" gets any RX interrupts (for the missing
>>>>> ARP replies)?
>>>>>
>>>>
>>>> It's interesting here. When I try with Intel NIC card. I see atleast 1
>>>> interrupt raised.  But not sure whether that is for ARP reply. Because,
>>>> when I try to dump the packet from "bridge"(modified) utility, I don't see
>>>> any ARP reply packet getting dumped.
>>>>
>>>>
>>>> *irq59: ix0:rxq0                        1          0 (only 1 interrupt
>>>> on the opposite side)*irq67: ix0:aq                          2
>>>>  0
>>>>
>>>> *irq68: ix1:rxq0                        3          0  (you can see 3
>>>> interrupts for 3 ARP requests from System 1)*irq76: ix1:aq
>>>>              2          0
>>>>
>>>> The same experiment, when I try with AMD inbuilt ports, I don't see
>>>> that 1 interrupt also raised.
>>>>
>>>> irq81: ax0:dev_irq                    16          0
>>>> irq83: ax0                          2541          4
>>>> irq93: ax1:dev_irq                    27          0
>>>> irq95: ax1                          2371          3
>>>> *irq97: ax1:rxq0                        3          0 (you can see 3
>>>> interrupts for 3 ARP requests from System 1, but no interrupt is seen from
>>>> "ax0:rxq0" for ARP reply from System 2)*
>>>>
>>>> I will do some more testing to see whether this behavior is consistent
>>>> or intermittent.
>>>>
>>>> Also the igb(4) driver is using iflib(4). So the involved netmap code
>>>>> is the same as ix(4) and if_axp(4).
>>>>> This is something that I'm not able to understand right now.
>>>>> It does not look like something related to offloads.
>>>>>
>>>>> Next week I will try to see if I can reproduce your issue with em(4),
>>>>> and report back. That's still an Intel driver using iflib(4).
>>>>>
>>>>
>>>> The "igb(4)" driver, with which things are working now is related to
>>>> em(4) driver (may be for newer hardware version).  Initially we faced
>>>> similar issue with igb(4) driver as well. After reverting the following
>>>> commits, things started to work.  Thanks to Stephan Dewt (copied) for
>>>> pointing this.  But it still fails with ix(4) driver and if_axp(4) driver.
>>>>
>>>>
>>>> https://github.com/freebsd/freebsd/commit/e12efc2c9e434075d0740e2e2e9e2fca2ad5f7cf
>>>>
>>>> Thanks for providing your inputs on this issue Vincenzo.  Let me know
>>>> for any more details that you need.
>>>>
>>>>
>>> I was able to reproduce your issue on FreeBSD-CURRENT running within a
>>> QEMU VM, with two em(4) devices and the netmap bridge running between them.
>>> I see the ARP request packet received on em0 (with associated IRQ), and
>>> forwarded on em1. However, the ARP reply coming on em1 does not trigger an
>>> IRQ on em1, and indeed the NIC RX head/tail pointers are not incremented as
>>> they should (`sysctl -a | grep em.1 | grep queue_rx`) ... that is weird,
>>> and lets me think that the issue is more likely driver-related than
>>> netmap/iflib-related.
>>> In any case, would you mind filing the issue on the bugzilla, so that we
>>> can properly track this issue?
>>>
>>> Thanks,
>>>   Vincenzo
>>>
>>>
>>>> Thanks,
>>>> Rajesh.
>>>>
>>>
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Netmap bridge not working with 10G Ethernet ports

Rajesh Kumar
Hi Vincenzo,


On Tue, Nov 24, 2020 at 8:54 PM Vincenzo Maffione <[hidden email]>
wrote:

>
> Yeah, it's weird because axgbe also uses iflib(4). If the driver exposes
> NIC head/tail pointers (sysctl) it may be useful to check what happens
> there.
> It may be that the NIC is dropping these packets for some reason.
>

Looks like, "ifdi_promisc_set" routine of the driver is not getting
triggered properly.  Forcibly setting the promisc mode from driver is
solving the packet drop issue. Now I see the ARP reply packet.

axgbe has split header support, which causes trouble starting from ping
packets due to incompatibility with Iflib/Netmap and associated utilities.
I made some changes to the driver, Iflib and Netmap utilities and having
the netmap bridge working with axgbe now.

I am working on a clean fix for promisc mode setting and incompatibility
issue. Will submit the changes for review once done.

Thanks for your inputs and support.

Done in r367920.
>

Thank you.

Thanks,
Rajesh.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"