FreeBSD does not reply to IPv6 Neighbor Solicitations

[Victor Sudakov-6]

Dear Colleagues,

Why could it be that a FreeBSD 12.2 host does not reply to ICMPv6
Neighbor Solicitations from the router?

Interface configuration on host:

$ ifconfig re1
re1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
        ether c4:12:f5:33:c9:7c
        inet 192.168.170.5/24 broadcast 192.168.170.255
        inet6 fe80::c612:f5ff:fe33:c97c%re1/64 scopeid 0x2
        inet6 2001:470:ecba:3::5/64
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
$

Interface configuration on router:

[admin@MikroTik] > /ipv6 address print where interface=bridge
Flags: X - disabled, I - invalid, D - dynamic, G - global, L - link-local
 #    ADDRESS                                     FROM-POOL INTERFACE
 0 DL fe80::4a8f:5aff:feab:b0c1/64                          bridge
 1  G 2001:470:ecba:3::1/64                                 bridge
[admin@MikroTik] >


Packet dump: http://admin.sibptus.ru/~vas/nd1.pcapng
where I ping a host in the IPv6 Internet from 2001:470:ecba:3::5, the
router wants to learn the L2 address for 2001:470:ecba:3::5 to reply
to, and receives no answer.

Where could be the problem?

--
Victor Sudakov,  VAS4-RIPE, VAS47-RIPN
2:5005/49@fidonet http://vas.tomsk.ru/

[Victor Sudakov-6]

Victor Sudakov wrote:
> Dear Colleagues,
>
> Why could it be that a FreeBSD 12.2 host does not reply to ICMPv6
> Neighbor Solicitations from the router?

Any ideas please?


--
Victor Sudakov,  VAS4-RIPE, VAS47-RIPN
2:5005/49@fidonet http://vas.tomsk.ru/

[Victor Sudakov-6]

Michael Sierchio wrote:

> On Sun, Jan 3, 2021 at 6:35 PM Victor Sudakov <[hidden email]> wrote:
>
> > > Why could it be that a FreeBSD 12.2 host does not reply to ICMPv6
> > > Neighbor Solicitations from the router?
> >
> > Any ideas please?
> >
> >
> Are you permitting the required udp and icmp?  These could be tighter, but
>
> ################################################################################
>
> # dhcp / bootp
>
> $FW add 00128 allow udp from any 67,68,546,547 to any 67,68,546,547

There is no firewall on the FreeBSD host in question. There is no need,
the host is on the LAN of a Mikrotik router.

>
> The method I have found to be reliable is to use dhcp6c, which requires the
> pkg 'dhcp6'

Why? On the host in question, I have a statically configured global IPv6
address, and auto_linklocal enabled on all interfaces:

$ ifconfig re1
re1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
        ether c4:12:f5:33:c9:7c
        inet 192.168.170.5/24 broadcast 192.168.170.255
        inet6 fe80::c612:f5ff:fe33:c97c%re1/64 scopeid 0x2
        inet6 2001:470:ecba:3::5/64
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
$

--
Victor Sudakov,  VAS4-RIPE, VAS47-RIPN
2:5005/49@fidonet http://vas.tomsk.ru/

[Lutz Donnerhacke]

> Victor Sudakov wrote:
> > Dear Colleagues,
> >
> > Why could it be that a FreeBSD 12.2 host does not reply to ICMPv6
> > Neighbor Solicitations from the router?
>
> Any ideas please?

Thank you for pointing this out.
I do have an similar effect, after upgrading, and you point me to a good
direction.
I'll investigate and report back.

Lutz Donnerhacke

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[hidden email]"

[Bjoern A. Zeeb]

On 4 Jan 2021, at 14:17, Lutz Donnerhacke wrote:

>> Victor Sudakov wrote:
>>> Dear Colleagues,
>>>
>>> Why could it be that a FreeBSD 12.2 host does not reply to ICMPv6
>>> Neighbor Solicitations from the router?
>>
>> Any ideas please?
>
> Thank you for pointing this out.
> I do have an similar effect, after upgrading, and you point me to a
> good
> direction.
> I'll investigate and report back.

I’d start by checking netstat -s -p icmp6 and netstat -s -p ip6  for
any suspicious counter updates.

Another thing to do might be to turn on nd6 log/debugging by sysctl  
(sysctl net.inet6.icmp6.nd6_debug=0xff should do it) and keep an eye on
the kernel messages.


/bz

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[hidden email]"

[Lutz Donnerhacke]

> I’d start by checking netstat -s -p icmp6 and netstat -s -p ip6  for
> any suspicious counter updates.

Great idea. It points me tot he most stupid error I could make.

Instead of
  ifconfig_lagg140_aliases="inet6 2a01:75c0:1000:140::/64 anycast"
I wrote
  ifconfig_vlan140_aliases="inet6 2a01:75c0:1000:140::/64 anycast"
so the IPv6 address was not set after reboot.

This fails to get noticed, due the long lifetime of the announced prefix.
(the error has been visible since a few days only, I had no time to investigate)

So I can confess, plain 12.2-STABLE is no broken.

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[hidden email]"

[Victor Sudakov-6]

Paul Mather wrote:
> >>>> Why could it be that a FreeBSD 12.2 host does not reply to ICMPv6
> >>>> Neighbor Solicitations from the router?

[dd]

> >
> > $ ifconfig re1
> > re1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
> >        options=8209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
> >        ether c4:12:f5:33:c9:7c
> >        inet 192.168.170.5/24 broadcast 192.168.170.255
> >        inet6 fe80::c612:f5ff:fe33:c97c%re1/64 scopeid 0x2
> >        inet6 2001:470:ecba:3::5/64
> >        media: Ethernet autoselect (1000baseT <full-duplex>)
> >        status: active
> >        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
>
>
> I notice your nd6 options do not include ACCEPT_RTADV.  Perhaps this
> is a reason why your interface is ignoring routing messages?  

Well, Neighbor Solicitations (ICMPv6 type 135) and Neighbor
Advertisements (ICMPv6 type 136) are not exactly routing messages, they
are the equivalent of the ARP protocol in IPv6, and AFAIK should work
between any two IPv6 nodes to map L3 addresses to L2 addresses, even if
there are no routers on the segment. Correct me if I'm wrong.

You may be right but then it is certainly a bug. Unfortunately I cannot
reproduce the problem with any reliability, this thing works more often
than not.

> My interface ifconfig shows "nd6
> options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>"

>
> I also use a statically-defined[*] IPv6 address, but include "accept_rtadv" in the interface definition in /etc/rc.conf.  Furthermore, I also set rtsold_enable="YES" to send router solicitation messages on the interface.

This would add one or two autoconfigured global IPv6 addresses to your
interface. There is no harm in that, I agree, but it's important to
understand if this is a bug and can be reproduced and reported.

>
> [*] As well as a static IPv6 address I also enable SLAAC to get autoconfigured and privacy addresses on the interface.
>

I see your point, this makes sense, but I would like to try and isolate
the problem.

--
Victor Sudakov,  VAS4-RIPE, VAS47-RIPN
2:5005/49@fidonet http://vas.tomsk.ru/

[Victor Sudakov-6]

Lutz Donnerhacke wrote:

> > Victor Sudakov wrote:
> > > Dear Colleagues,
> > >
> > > Why could it be that a FreeBSD 12.2 host does not reply to ICMPv6
> > > Neighbor Solicitations from the router?
> >
> > Any ideas please?
>
> Thank you for pointing this out.
> I do have an similar effect, after upgrading, and you point me to a good
> direction.
> I'll investigate and report back.

Problem is, I cannot reproduce it reliably. Sometimes everything "just works."
Maybe the absence of traffic causes this, I really don't know.

--
Victor Sudakov,  VAS4-RIPE, VAS47-RIPN
2:5005/49@fidonet http://vas.tomsk.ru/

[Lutz Donnerhacke]

Victor Sudakov wrote:
> Paul Mather wrote:
> > >>>> Why could it be that a FreeBSD 12.2 host does not reply to ICMPv6
> > >>>> Neighbor Solicitations from the router?
>
> Well, Neighbor Solicitations (ICMPv6 type 135) and Neighbor
> Advertisements (ICMPv6 type 136) are not exactly routing messages, they
> are the equivalent of the ARP protocol in IPv6, and AFAIK should work
> between any two IPv6 nodes to map L3 addresses to L2 addresses, even if
> there are no routers on the segment. Correct me if I'm wrong.

Correct.

> You may be right but then it is certainly a bug. Unfortunately I cannot
> reproduce the problem with any reliability, this thing works more often
> than not.

May you be able to capture the icmp6 traffic of this interface with respect
to ND? I'm really interested in seeing, that the box does not respond to a
given NS query.

There are various reasons, why this may happen, i.e. sender IP in the NS is
out of prefix of the target IP. This may happen, if multiple prefixes are
added to the interface. Some devices (like Cisco ASA) are very picky on
matching source/target IPs. So it might be possible, that the problem is not
the the FreeBSD box, but the querying device (Mircotik?)

> > My interface ifconfig shows "nd6
> > options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>"
[...]
> > [*] As well as a static IPv6 address I also enable SLAAC to get
> > autoconfigured and privacy addresses on the interface.
>
> I see your point, this makes sense, but I would like to try and isolate
> the problem.

There is no problem with neighbour discovery without the ACCEPT_RTADV
option. It simply works.
# uname -a
FreeBSD ... 12.2-STABLE FreeBSD 12.2-STABLE r368820 ENCOLINE-NAT  amd64

# ifconfig vlan1111
vlan1111: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
1500
       options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
       ether 48:df:37:3c:d3:50
       inet6 fe80::4adf:37ff:fe3c:d350%vlan1111 prefixlen 64 scopeid 0x1e
       inet6 2a01:75c0:1000:1111:5:102:160:146 prefixlen 64
       inet 5.102.160.146 netmask 0xfffffff0 broadcast 5.102.160.159
       groups: vlan
       vlan: 1111 vlanpcp: 0 parent interface: ixl0
       media: Ethernet autoselect (10Gbase-SR <full-duplex,rxpause,txpause>)
       status: active
       nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

# tcpdump -ni vlan1111 icmp6 | fgrep neighbor
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vlan1111, link-type EN10MB (Ethernet), capture size 262144
bytes
09:06:17.823698 IP6 fe80::50:1111 > 2a01:75c0:1000:1111:5:102:160:146:
ICMP6, neighbor solicitation, who has 2a01:75c0:1000:1111:5:102:160:146,
length 32
09:06:17.823708 IP6 fe80::4adf:37ff:fe3c:d350 > fe80::50:1111: ICMP6,
neighbor advertisement, tgt is 2a01:75c0:1000:1111:5:102:160:146, length 24
09:06:22.782809 IP6 fe80::4adf:37ff:fe3c:d350 > fe80::50:1111: ICMP6,
neighbor solicitation, who has fe80::50:1111, length 32
09:06:22.787620 IP6 fe80::50:1111 > fe80::4adf:37ff:fe3c:d350: ICMP6,
neighbor advertisement, tgt is fe80::50:1111, length 24
^C271 packets captured
5149447 packets received by filter
0 packets dropped by kernel

So it works in both directions.
Please note, that the first NS query is coming from a link-local address and
requesting a global IP. This will not always be answered by any device out
there (especially if the roles are reversed)

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[hidden email]"

[Victor Sudakov-6]

Lutz Donnerhacke wrote:

> Victor Sudakov wrote:
> > Paul Mather wrote:
> > > >>>> Why could it be that a FreeBSD 12.2 host does not reply to ICMPv6
> > > >>>> Neighbor Solicitations from the router?
> >
> > Well, Neighbor Solicitations (ICMPv6 type 135) and Neighbor
> > Advertisements (ICMPv6 type 136) are not exactly routing messages, they
> > are the equivalent of the ARP protocol in IPv6, and AFAIK should work
> > between any two IPv6 nodes to map L3 addresses to L2 addresses, even if
> > there are no routers on the segment. Correct me if I'm wrong.
>
> Correct.
>
> > You may be right but then it is certainly a bug. Unfortunately I cannot
> > reproduce the problem with any reliability, this thing works more often
> > than not.
>
> May you be able to capture the icmp6 traffic of this interface with respect
> to ND? I'm really interested in seeing, that the box does not respond to a
> given NS query.

Here you are http://admin.sibptus.ru/~vas/nd1.pcapng

>
> There are various reasons, why this may happen, i.e. sender IP in the NS is
> out of prefix of the target IP. This may happen, if multiple prefixes are
> added to the interface. Some devices (like Cisco ASA) are very picky on
> matching source/target IPs. So it might be possible, that the problem is not
> the the FreeBSD box, but the querying device (Mircotik?)

Maybe. The Mikrotik sends neighbor solicitations from a link-local
address, as you can see in the packet dump above. Is this correct
behavior?

>
> There is no problem with neighbour discovery without the ACCEPT_RTADV
> option. It simply works.

I thought as much.

> So it works in both directions.
> Please note, that the first NS query is coming from a link-local address and
> requesting a global IP. This will not always be answered by any device out
> there (especially if the roles are reversed)

Hmm, this is an interesting observation, please see the packet dump
above, what do you say?

And what do standards say, what should be the source address of a
neighbor solicitation when the target address is a global address?

--
Victor Sudakov,  VAS4-RIPE, VAS47-RIPN
2:5005/49@fidonet http://vas.tomsk.ru/

[Lutz Donnerhacke]

> > May you be able to capture the icmp6 traffic of this interface with
> > respect to ND? I'm really interested in seeing, that the box does
> > not respond to a given NS query.
>
> Here you are http://admin.sibptus.ru/~vas/nd1.pcapng

The device, where the capture was taken does not respond tot he NS packet.
This might be caused by:
 a) the device has a different configured IP address, than requested
 b) the network card does not listen to the multicast group, which is
    used by the request (you see it only due to the promisc mode of the
    capture). But this is unlikely (due to the promisc mode)
 c) your system is broken

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[hidden email]"

[Victor Sudakov-6]

Lutz Donnerhacke wrote:
> > > May you be able to capture the icmp6 traffic of this interface with
> > > respect to ND? I'm really interested in seeing, that the box does
> > > not respond to a given NS query.
> >
> > Here you are http://admin.sibptus.ru/~vas/nd1.pcapng
>
> The device, where the capture was taken does not respond to the NS packet.

So your interpretation agrees with mine, that's great.

> This might be caused by:
>  a) the device has a different configured IP address, than requested

I have already published the output of `ifconfig re1`, here it is again,
do you think anything important is missing? This is the very interface
where the capture was taken:

$ ifconfig re1
re1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
        ether c4:12:f5:33:c9:7c
        inet 192.168.170.5/24 broadcast 192.168.170.255
        inet6 fe80::c612:f5ff:fe33:c97c%re1/64 scopeid 0x2
        inet6 2001:470:ecba:3::5/64
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
$


>  b) the network card does not listen to the multicast group, which is
>     used by the request

Why could that be? Hardware problem or software/FreeBSD glitch?

> (you see it only due to the promisc mode of the
>     capture). But this is unlikely (due to the promisc mode)
>  c) your system is broken

Very likely. That's why I'm here looking for advice and enlightenment.

--
Victor Sudakov,  VAS4-RIPE, VAS47-RIPN
2:5005/49@fidonet http://vas.tomsk.ru/

[John-Mark Gurney-2]

Lutz Donnerhacke wrote this message on Tue, Jan 05, 2021 at 12:58 +0100:

> > > May you be able to capture the icmp6 traffic of this interface with
> > > respect to ND? I'm really interested in seeing, that the box does
> > > not respond to a given NS query.
> >
> > Here you are http://admin.sibptus.ru/~vas/nd1.pcapng
>
> The device, where the capture was taken does not respond tot he NS packet.
> This might be caused by:
>  a) the device has a different configured IP address, than requested
>  b) the network card does not listen to the multicast group, which is
>     used by the request (you see it only due to the promisc mode of the
>     capture). But this is unlikely (due to the promisc mode)
>  c) your system is broken

I have some test scripts where something similar to this happens.

I tcpdump shows the request coming into the FreeBSD box (in this case,
13-current main-c255640-gc38e59ce1b0), addressed to the IPv6 of the
box, and FreeBSD failing to respond w/ an answer for it's own IP...

This is inconsistent and hard to reproduce, but it does happen with
somewhat regularity.

This is from the host w/ ipv6 address fc00:b5d:41c:7e37::c43c:
02:11:53.065550 IP6 :: > ff02::1:ff00:7e37: ICMP6, neighbor solicitation, who has fc00:b5d:41c:7e37::7e37, length 32
02:11:53.069274 IP6 :: > ff02::1:ff00:c43c: ICMP6, neighbor solicitation, who has fc00:b5d:41c:7e37::c43c, length 32
02:11:54.639001 IP6 fc00:b5d:41c:7e37::7e37 > ff02::1:ff00:c43c: ICMP6, neighbor solicitation, who has fc00:b5d:41c:7e37::c43c, length 32
02:11:55.659956 IP6 fc00:b5d:41c:7e37::7e37 > ff02::1:ff00:c43c: ICMP6, neighbor solicitation, who has fc00:b5d:41c:7e37::c43c, length 32
02:11:56.667880 IP6 fc00:b5d:41c:7e37::7e37 > ff02::1:ff00:c43c: ICMP6, neighbor solicitation, who has fc00:b5d:41c:7e37::c43c, length 32

and this is from the host w/ ipv6 address fc00:b5d:41c:7e37::7e37:
02:11:53.065345 IP6 :: > ff02::1:ff00:7e37: ICMP6, neighbor solicitation, who has fc00:b5d:41c:7e37::7e37, length 32
02:11:54.638742 IP6 fc00:b5d:41c:7e37::7e37 > ff02::1:ff00:c43c: ICMP6, neighbor solicitation, who has fc00:b5d:41c:7e37::c43c, length 32
02:11:55.658801 IP6 fc00:b5d:41c:7e37::7e37 > ff02::1:ff00:c43c: ICMP6, neighbor solicitation, who has fc00:b5d:41c:7e37::c43c, length 32
02:11:56.667187 IP6 fc00:b5d:41c:7e37::7e37 > ff02::1:ff00:c43c: ICMP6, neighbor solicitation, who has fc00:b5d:41c:7e37::c43c, length 32

So, these captures are from the same host, but w/ the two interfaces
in different vnet jails, but wired back to back, so the time stamps
came from same clock, so can give you close to absolute ordering between
the two captures...

This is pretty much, bring up interface, configure a/ ipv6 addresses,
and then ping the address, and fail after a couple tries.

I'm not sure why the 7e37 host didn't receive c43c's hosts broadcast
announcing their address, but other times, it will properly respond,
for example:
05:08:32.158342 IP6 fc00:b5d:41c:7e37::7e37 > ff02::1:ff00:c43c: ICMP6, neighbor solicitation, who has fc00:b5d:41c:7e37::c43c, length 32
05:08:32.158377 IP6 fc00:b5d:41c:7e37::c43c > fc00:b5d:41c:7e37::7e37: ICMP6, neighbor advertisement, tgt is fc00:b5d:41c:7e37::c43c, length 32
05:08:32.215624 IP6 fc00:b5d:41c:7e37::7e37 > fc00:b5d:41c:7e37::c43c: ICMP6, echo request, seq 0, length 16
05:08:32.215646 IP6 fc00:b5d:41c:7e37::c43c > fc00:b5d:41c:7e37::7e37: ICMP6, echo reply, seq 0, length 16

--
  John-Mark Gurney Voice: [phone]

     "All that I will do, has been done, All that I have, has not."
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[hidden email]"

[Andrey V. Elsukov]

On 12.01.2021 05:25, John-Mark Gurney wrote:

>> The device, where the capture was taken does not respond tot he NS packet.
>> This might be caused by:
>>  a) the device has a different configured IP address, than requested
>>  b) the network card does not listen to the multicast group, which is
>>     used by the request (you see it only due to the promisc mode of the
>>     capture). But this is unlikely (due to the promisc mode)
>>  c) your system is broken
>
> I have some test scripts where something similar to this happens.
>
> I tcpdump shows the request coming into the FreeBSD box (in this case,
> 13-current main-c255640-gc38e59ce1b0), addressed to the IPv6 of the
> box, and FreeBSD failing to respond w/ an answer for it's own IP...
>
> This is inconsistent and hard to reproduce, but it does happen with
> somewhat regularity.

Hi,

when this will happen again, it would be nice to make sure that NS
packets hit the IP stack. E.g. with attached dtrace script.

Also net.inet6.icmp6.nd6_debug variable should be set to see error
messages from ND code.

If it doesn't show expected info, this means that packets don't hit IP
stack. Probably some multicast related problem. In this case it could be
useful to obtain output of ifmcstat(8).

--
WBR, Andrey V. Elsukov

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[hidden email]"

[John-Mark Gurney-2]

Andrey V. Elsukov wrote this message on Tue, Jan 12, 2021 at 16:33 +0300:

> On 12.01.2021 05:25, John-Mark Gurney wrote:
> >> The device, where the capture was taken does not respond tot he NS packet.
> >> This might be caused by:
> >>  a) the device has a different configured IP address, than requested
> >>  b) the network card does not listen to the multicast group, which is
> >>     used by the request (you see it only due to the promisc mode of the
> >>     capture). But this is unlikely (due to the promisc mode)
> >>  c) your system is broken
> >
> > I have some test scripts where something similar to this happens.
> >
> > I tcpdump shows the request coming into the FreeBSD box (in this case,
> > 13-current main-c255640-gc38e59ce1b0), addressed to the IPv6 of the
> > box, and FreeBSD failing to respond w/ an answer for it's own IP...
> >
> > This is inconsistent and hard to reproduce, but it does happen with
> > somewhat regularity.
>
> when this will happen again, it would be nice to make sure that NS
> packets hit the IP stack. E.g. with attached dtrace script.

Ok, I ran the dtrace script when I reproduced the problem, and it did
not produce any output.

Here are the steps that I use to setup the interfaces for this test,
which seems to trigger it...  This is on a single machine, usings the
on board bge0 and a USB ure...  Each interface is put in their own
vnet jail...  I've attached the script, just need to init the jails,
then run the test a couple of times:

sh testinterfaces.sh init ue0 bge0
sh testinterfaces.sh csum ue0 bge0
sh testinterfaces.sh csum ue0 bge0

When it fails, you'll see something like:
[...]
ping FAILED on -rxcsum6!!!!                              
27018 27019                                              
test failed, data in /tmp/testiface.moLZMgFH                                                                      
csumtest failed.                                        

the numbers are the pid's of the tcpdumps that were run... There are also
pcaps of the two sides in the directory...

These are effectively what the script does:
1) start w/ both interfaces in down state (they have ipv6 addresses set
   from previous run)
2) configure csum flags on ure (in this case, -rxcsum6)
3) verify that flag is cleared
4) disable all csum flags on bge0 (-txcsum -rxcsum -txcsum6 -rxcsum6)
5) verify that -txcsum and -rxcsum on bge0 (just realized this is a bug
   in my script that I don't verify the ipv6 versions of the flag)
6) sleep for .5 seconds
7) bring up ure
8) bring up bge0
9) configure inet6 addresses on ure and bge (duplicating the addresses
   already configured)
10) wait for both interfaces to have link, AND the inet6 addresses to
    not be in tentative state
11) sleep .5
12) run ping, and see it fail due to the ND problem

I ran the dtrace script in the host system, which iirc, should be fine.

If you enable the removal of the inet6 addresses in cleaniface, this
error does not seem to trigger...

> Also net.inet6.icmp6.nd6_debug variable should be set to see error
> messages from ND code.

I set it in the jail, and did not see any messages... I reset the
interface, and was able to see the messages when it "worked", so
it looks like the packets aren't hitting the ip stack properly...

> If it doesn't show expected info, this means that packets don't hit IP
> stack. Probably some multicast related problem. In this case it could be
> useful to obtain output of ifmcstat(8).

So, the output of ifmcstat when it isn't working:
root@test:~ # jexec checkjail ifmcstat
lo0:
        inet6 fe80::1%lo0 scopeid 0x1
sysctl net.inet6.mld.ifinfo: No such file or directory
                group ff01::1%lo0 scopeid 0x1 mode exclude
                group ff02::1%lo0 scopeid 0x1 mode exclude
                group ff02::1:ff00:1%lo0 scopeid 0x1 mode exclude
bge0:
        inet6 fe80::12e7:c6ff:fexx:xxxx%bge0 scopeid 0x2
        mldv2 flags=2<USEALLOW> rv 2 qi 125 qri 10 uri 3
                group ff01::1%bge0 scopeid 0x2 mode exclude
                        mcast-macaddr 33:33:00:00:00:01
                group ff02::1%bge0 scopeid 0x2 mode exclude
                        mcast-macaddr 33:33:00:00:00:01
                group ff02::1:ffxx:xxxx%bge0 scopeid 0x2 mode exclude
                        mcast-macaddr 33:33:ff:xx:xx:xx

I'm not sure why there's an error on net.inet6.mld.ifinfo, as both my
kernel and userland should be in sync, as of Jan 9th...

so, I made things works, and ran ifmcstat again, and this time it has
an additional group in the output:
[...]
bge0:
        inet6 fe80::12e7:c6ff:fexx:xxxx%bge0 scopeid 0x2
        mldv2 flags=2<USEALLOW> rv 2 qi 125 qri 10 uri 3
                group ff02::1:ff00:c43c%bge0 scopeid 0x2 mode exclude
                        mcast-macaddr 33:33:ff:00:c4:3c
                group ff01::1%bge0 scopeid 0x2 mode exclude
                        mcast-macaddr 33:33:00:00:00:01
                group ff02::1%bge0 scopeid 0x2 mode exclude
                        mcast-macaddr 33:33:00:00:00:01
                group ff02::1:ffxx:xxxx%bge0 scopeid 0x2 mode exclude
                        mcast-macaddr 33:33:ff:xx:xx:xx

and the tcpdump output:
21:10:53.938655 IP6 fc00:b5d:41c:7e37::7e37 > ff02::1:ff00:c43c: ICMP6, neighbor solicitation, who has fc00:b5d:41c:7e37::c43c, length 32
21:10:55.001428 IP6 fc00:b5d:41c:7e37::7e37 > ff02::1:ff00:c43c: ICMP6, neighbor solicitation, who has fc00:b5d:41c:7e37::c43c, length 32

In this case, the interface that is having issues is bge0, so it's a
"real" NIC and should be well supported...

--
  John-Mark Gurney Voice: [phone]

     "All that I will do, has been done, All that I have, has not."
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[hidden email]"

[Andrey V. Elsukov]

On 13.01.2021 00:37, John-Mark Gurney wrote:
>> when this will happen again, it would be nice to make sure that NS
>> packets hit the IP stack. E.g. with attached dtrace script.
>
> Ok, I ran the dtrace script when I reproduced the problem, and it did
> not produce any output.
>
> These are effectively what the script does:
> 9) configure inet6 addresses on ure and bge (duplicating the addresses
>    already configured)

Does it mean that you just reconfigure address without removing it? It
looks like the problem, that was discussed here
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=233535

> bge0:
>         inet6 fe80::12e7:c6ff:fexx:xxxx%bge0 scopeid 0x2
>         mldv2 flags=2<USEALLOW> rv 2 qi 125 qri 10 uri 3
>                 group ff01::1%bge0 scopeid 0x2 mode exclude
>                         mcast-macaddr 33:33:00:00:00:01
>                 group ff02::1%bge0 scopeid 0x2 mode exclude
>                         mcast-macaddr 33:33:00:00:00:01
>                 group ff02::1:ffxx:xxxx%bge0 scopeid 0x2 mode exclude
>                         mcast-macaddr 33:33:ff:xx:xx:xx
>
> so, I made things works, and ran ifmcstat again, and this time it has
> an additional group in the output:
> [...]
> bge0:
>         inet6 fe80::12e7:c6ff:fexx:xxxx%bge0 scopeid 0x2
>         mldv2 flags=2<USEALLOW> rv 2 qi 125 qri 10 uri 3
>                 group ff02::1:ff00:c43c%bge0 scopeid 0x2 mode exclude
>                         mcast-macaddr 33:33:ff:00:c4:3c
>                 group ff01::1%bge0 scopeid 0x2 mode exclude
>                         mcast-macaddr 33:33:00:00:00:01
>                 group ff02::1%bge0 scopeid 0x2 mode exclude
>                         mcast-macaddr 33:33:00:00:00:01
>                 group ff02::1:ffxx:xxxx%bge0 scopeid 0x2 mode exclude
>                         mcast-macaddr 33:33:ff:xx:xx:xx
>
> and the tcpdump output:
> 21:10:53.938655 IP6 fc00:b5d:41c:7e37::7e37 > ff02::1:ff00:c43c: ICMP6, neighbor solicitation, who has fc00:b5d:41c:7e37::c43c, length 32
> 21:10:55.001428 IP6 fc00:b5d:41c:7e37::7e37 > ff02::1:ff00:c43c: ICMP6, neighbor solicitation, who has fc00:b5d:41c:7e37::c43c, length 32

Since ff02::1:ff00:c43c%bge0 is not configured in first case, IP stack
just ignores NS messages and they don't hit ND6 code.

In the PR 233535 the problem was reproducible with MLDv1, so if you
disable MLDv2 will it work (to reduce possible scope of problematic code)?

net.inet6.mld.v2enable=0

--
WBR, Andrey V. Elsukov
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[hidden email]"

[John-Mark Gurney-2]

Andrey V. Elsukov wrote this message on Wed, Jan 13, 2021 at 11:42 +0300:

> On 13.01.2021 00:37, John-Mark Gurney wrote:
> >> when this will happen again, it would be nice to make sure that NS
> >> packets hit the IP stack. E.g. with attached dtrace script.
> >
> > Ok, I ran the dtrace script when I reproduced the problem, and it did
> > not produce any output.
> >
> > These are effectively what the script does:
> > 9) configure inet6 addresses on ure and bge (duplicating the addresses
> >    already configured)
>
> Does it mean that you just reconfigure address without removing it? It
> looks like the problem, that was discussed here
> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=233535

Yeah, I guess it is the same...

> > bge0:
> >         inet6 fe80::12e7:c6ff:fexx:xxxx%bge0 scopeid 0x2
> >         mldv2 flags=2<USEALLOW> rv 2 qi 125 qri 10 uri 3
> >                 group ff01::1%bge0 scopeid 0x2 mode exclude
> >                         mcast-macaddr 33:33:00:00:00:01
> >                 group ff02::1%bge0 scopeid 0x2 mode exclude
> >                         mcast-macaddr 33:33:00:00:00:01
> >                 group ff02::1:ffxx:xxxx%bge0 scopeid 0x2 mode exclude
> >                         mcast-macaddr 33:33:ff:xx:xx:xx
> >
> > so, I made things works, and ran ifmcstat again, and this time it has
> > an additional group in the output:
> > [...]
> > bge0:
> >         inet6 fe80::12e7:c6ff:fexx:xxxx%bge0 scopeid 0x2
> >         mldv2 flags=2<USEALLOW> rv 2 qi 125 qri 10 uri 3
> >                 group ff02::1:ff00:c43c%bge0 scopeid 0x2 mode exclude
> >                         mcast-macaddr 33:33:ff:00:c4:3c
> >                 group ff01::1%bge0 scopeid 0x2 mode exclude
> >                         mcast-macaddr 33:33:00:00:00:01
> >                 group ff02::1%bge0 scopeid 0x2 mode exclude
> >                         mcast-macaddr 33:33:00:00:00:01
> >                 group ff02::1:ffxx:xxxx%bge0 scopeid 0x2 mode exclude
> >                         mcast-macaddr 33:33:ff:xx:xx:xx
> >
> > and the tcpdump output:
> > 21:10:53.938655 IP6 fc00:b5d:41c:7e37::7e37 > ff02::1:ff00:c43c: ICMP6, neighbor solicitation, who has fc00:b5d:41c:7e37::c43c, length 32
> > 21:10:55.001428 IP6 fc00:b5d:41c:7e37::7e37 > ff02::1:ff00:c43c: ICMP6, neighbor solicitation, who has fc00:b5d:41c:7e37::c43c, length 32
>
> Since ff02::1:ff00:c43c%bge0 is not configured in first case, IP stack
> just ignores NS messages and they don't hit ND6 code.
>
> In the PR 233535 the problem was reproducible with MLDv1, so if you
> disable MLDv2 will it work (to reduce possible scope of problematic code)?
>
> net.inet6.mld.v2enable=0

I just tested this, and it does not fix the problem for me.

Do I need to reboot or something?  If I set it to 0, the bug
still appears, and also the ifmcstat has the line:
mldv2 flags=2<USEALLOW> rv 2 qi 125 qri 10 uri 3

is there something else that needs to be done to disable mldv2?

--
  John-Mark Gurney Voice: [phone]

     "All that I will do, has been done, All that I have, has not."
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[hidden email]"

[John-Mark Gurney-2]

John-Mark Gurney wrote this message on Wed, Jan 13, 2021 at 17:59 -0800:

> Andrey V. Elsukov wrote this message on Wed, Jan 13, 2021 at 11:42 +0300:
> > On 13.01.2021 00:37, John-Mark Gurney wrote:
> > >> when this will happen again, it would be nice to make sure that NS
> > >> packets hit the IP stack. E.g. with attached dtrace script.
> > >
> > > Ok, I ran the dtrace script when I reproduced the problem, and it did
> > > not produce any output.
> > >
> > > These are effectively what the script does:
> > > 9) configure inet6 addresses on ure and bge (duplicating the addresses
> > >    already configured)
> >
> > Does it mean that you just reconfigure address without removing it? It
> > looks like the problem, that was discussed here
> > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=233535
>
> Yeah, I guess it is the same...
>
> > > bge0:
> > >         inet6 fe80::12e7:c6ff:fexx:xxxx%bge0 scopeid 0x2
> > >         mldv2 flags=2<USEALLOW> rv 2 qi 125 qri 10 uri 3
> > >                 group ff01::1%bge0 scopeid 0x2 mode exclude
> > >                         mcast-macaddr 33:33:00:00:00:01
> > >                 group ff02::1%bge0 scopeid 0x2 mode exclude
> > >                         mcast-macaddr 33:33:00:00:00:01
> > >                 group ff02::1:ffxx:xxxx%bge0 scopeid 0x2 mode exclude
> > >                         mcast-macaddr 33:33:ff:xx:xx:xx
> > >
> > > so, I made things works, and ran ifmcstat again, and this time it has
> > > an additional group in the output:
> > > [...]
> > > bge0:
> > >         inet6 fe80::12e7:c6ff:fexx:xxxx%bge0 scopeid 0x2
> > >         mldv2 flags=2<USEALLOW> rv 2 qi 125 qri 10 uri 3
> > >                 group ff02::1:ff00:c43c%bge0 scopeid 0x2 mode exclude
> > >                         mcast-macaddr 33:33:ff:00:c4:3c
> > >                 group ff01::1%bge0 scopeid 0x2 mode exclude
> > >                         mcast-macaddr 33:33:00:00:00:01
> > >                 group ff02::1%bge0 scopeid 0x2 mode exclude
> > >                         mcast-macaddr 33:33:00:00:00:01
> > >                 group ff02::1:ffxx:xxxx%bge0 scopeid 0x2 mode exclude
> > >                         mcast-macaddr 33:33:ff:xx:xx:xx
> > >
> > > and the tcpdump output:
> > > 21:10:53.938655 IP6 fc00:b5d:41c:7e37::7e37 > ff02::1:ff00:c43c: ICMP6, neighbor solicitation, who has fc00:b5d:41c:7e37::c43c, length 32
> > > 21:10:55.001428 IP6 fc00:b5d:41c:7e37::7e37 > ff02::1:ff00:c43c: ICMP6, neighbor solicitation, who has fc00:b5d:41c:7e37::c43c, length 32
> >
> > Since ff02::1:ff00:c43c%bge0 is not configured in first case, IP stack
> > just ignores NS messages and they don't hit ND6 code.
> >
> > In the PR 233535 the problem was reproducible with MLDv1, so if you
> > disable MLDv2 will it work (to reduce possible scope of problematic code)?
> >
> > net.inet6.mld.v2enable=0
>
> I just tested this, and it does not fix the problem for me.
>
> Do I need to reboot or something?  If I set it to 0, the bug
> still appears, and also the ifmcstat has the line:
> mldv2 flags=2<USEALLOW> rv 2 qi 125 qri 10 uri 3
>
> is there something else that needs to be done to disable mldv2?

I was able to reproduce using epair on a single system.  It did
take a few runs of the test before it failed, but it was able to
fail.

So, to reproduce, on a -current system (mine is
main-c255640-gc38e59ce1b0), do:

ifconfig epair0 create
sh ipv6.failure.sh init epair0a epair0b
sh ipv6.failure.sh run epair0a epair0b
sh ipv6.failure.sh run epair0a epair0b
sh ipv6.failure.sh run epair0a epair0b
sh ipv6.failure.sh run epair0a epair0b

In my case, it failed on the fourth try, but it does appear to be
inconsistent, as a fifth run worked, but then the sixth run failed
again, and additional runs worked...

If you want to see the live tcpdump, you can run the following
commands in a coupld different windows:
jexec testjail tcpdump -n -p -i epair0a

and:
jexec checkjail tcpdump -n -p -i epair0b

uname -a:
FreeBSD test 13.0-CURRENT FreeBSD 13.0-CURRENT #7 main-c255640-gc38e59ce1b0: Sat Jan  9 01:55:25 UTC 2021     root@test:/usr/obj/usr/src.git/amd64.amd64/sys/GENERIC  amd64

and the cpu:
CPU: AMD PRO A10-8770E R7, 10 COMPUTE CORES 4C+6G    (2794.80-MHz K8-class CPU)

--
  John-Mark Gurney Voice: [phone]

     "All that I will do, has been done, All that I have, has not."
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[hidden email]"