SNMP/No Bufferspace

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

SNMP/No Bufferspace

Larry Rosenman
I have a mrtg job that runs every 5 minutes on a FreeBSD-CURRENT box and
gets this randomly:

SNMP Error:
send_query: No buffer space available
SNMPv1_Session (remote host: "main-switch.lerctr.org"
[192.168.200.224].161)
                   community: "lerctr"
                  request ID: 1517898706
                 PDU bufsize: 8000 bytes
                     timeout: 2s
                     retries: 5
                     backoff: 1)
  at /usr/local/lib/perl5/site_perl/SNMP_util.pm line 492.
SNMPGET Problem for ifInOctets.39 ifOutOctets.39 sysUptime sysName on
[hidden email]::::::v4only
  at /usr/local/bin/mrtg line 2330.
2019-06-03 10:50:00: WARNING: Skipping host main-switch.lerctr.org as it
does not resolve to an IPv4 address
2019-06-03 10:50:00: WARNING: no data for
ifInOctets&ifOutOctets:[hidden email]. Skipping further
queries for Host main-switch.lerctr.org in this round.
2019-06-03 10:50:00: WARNING: Skipping host sewingroomswitch.lerctr.org
as it does not resolve to an IPv4 address
2019-06-03 10:50:00: WARNING: no data for
ifInOctets&ifOutOctets:[hidden email]. Skipping
further queries for Host sewingroomswitch.lerctr.org in this round.
2019-06-03 10:50:01: ERROR: Target[main-switch.lerctr.org_39][_IN_] '
$target->[24]{$mode} ' did not eval into defined data
2019-06-03 10:50:01: ERROR: Target[main-switch.lerctr.org_39][_OUT_] '
$target->[24]{$mode} ' did not eval into defined data
2019-06-03 10:50:01: ERROR: Target[main-switch.lerctr.org_40][_IN_] '
$target->[25]{$mode} ' did not eval into defined data
2019-06-03 10:50:01: ERROR: Target[main-switch.lerctr.org_40][_OUT_] '
$target->[25]{$mode} ' did not eval into defined data
2019-06-03 10:50:01: ERROR: Target[main-switch.lerctr.org_41][_IN_] '
$target->[26]{$mode} ' did not eval into defined data
2019-06-03 10:50:01: ERROR: Target[main-switch.lerctr.org_41][_OUT_] '
$target->[26]{$mode} ' did not eval into defined data
2019-06-03 10:50:01: ERROR: Target[main-switch.lerctr.org_42][_IN_] '
$target->[27]{$mode} ' did not eval into defined data
2019-06-03 10:50:01: ERROR: Target[main-switch.lerctr.org_42][_OUT_] '
$target->[27]{$mode} ' did not eval into defined data
2019-06-03 10:50:01: ERROR: Target[main-switch.lerctr.org_43][_IN_] '
$target->[28]{$mode} ' did not eval into defined data
2019-06-03 10:50:01: ERROR: Target[main-switch.lerctr.org_43][_OUT_] '
$target->[28]{$mode} ' did not eval into defined data
2019-06-03 10:50:01: ERROR: Target[main-switch.lerctr.org_44][_IN_] '
$target->[29]{$mode} ' did not eval into defined data
2019-06-03 10:50:01: ERROR: Target[main-switch.lerctr.org_44][_OUT_] '
$target->[29]{$mode} ' did not eval into defined data
2019-06-03 10:50:01: ERROR: Target[main-switch.lerctr.org_45][_IN_] '
$target->[30]{$mode} ' did not eval into defined data
2019-06-03 10:50:01: ERROR: Target[main-switch.lerctr.org_45][_OUT_] '
$target->[30]{$mode} ' did not eval into defined data
2019-06-03 10:50:01: ERROR: Target[main-switch.lerctr.org_46][_IN_] '
$target->[31]{$mode} ' did not eval into defined data
2019-06-03 10:50:01: ERROR: Target[main-switch.lerctr.org_46][_OUT_] '
$target->[31]{$mode} ' did not eval into defined data
2019-06-03 10:50:01: ERROR: Target[main-switch.lerctr.org_47][_IN_] '
$target->[32]{$mode} ' did not eval into defined data
2019-06-03 10:50:01: ERROR: Target[main-switch.lerctr.org_47][_OUT_] '
$target->[32]{$mode} ' did not eval into defined data
2019-06-03 10:50:01: ERROR: Target[main-switch.lerctr.org_48][_IN_] '
$target->[33]{$mode} ' did not eval into defined data
2019-06-03 10:50:01: ERROR: Target[main-switch.lerctr.org_48][_OUT_] '
$target->[33]{$mode} ' did not eval into defined data
2019-06-03 10:50:01: ERROR: Target[sewingroomswitch.lerctr.org_1][_IN_]
' $target->[34]{$mode} ' did not eval into defined data
2019-06-03 10:50:01: ERROR: Target[sewingroomswitch.lerctr.org_1][_OUT_]
' $target->[34]{$mode} ' did not eval into defined data


I have:
kern.ipc.maxsockbuf=2147483648

in my sysctl.conf as suggested in a bunch of places, and the box has
plenty (128G) of memory.

How can I debug this?


--
Larry Rosenman                     http://www.lerctr.org/~ler
Phone: +1 214-642-9640                 E-Mail: [hidden email]
US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: SNMP/No Bufferspace

Eugene Grosbein-10
03.06.2019 22:56, Larry Rosenman wrote:

> I have a mrtg job that runs every 5 minutes on a FreeBSD-CURRENT box and gets this randomly:
>
> SNMP Error:
> send_query: No buffer space available

[skip]

> How can I debug this?

Your outgoing network interface stalls for some reason.
Where is this traffic directed to? Some kind of VPN tunnel? Physical NIC that looses link?
Some Wifi or mobile network?


_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: SNMP/No Bufferspace

Larry Rosenman
On 06/03/2019 12:23 pm, Eugene Grosbein wrote:

> 03.06.2019 22:56, Larry Rosenman wrote:
>
>> I have a mrtg job that runs every 5 minutes on a FreeBSD-CURRENT box
>> and gets this randomly:
>>
>> SNMP Error:
>> send_query: No buffer space available
>
> [skip]
>
>> How can I debug this?
>
> Your outgoing network interface stalls for some reason.
> Where is this traffic directed to? Some kind of VPN tunnel? Physical
> NIC that looses link?
> Some Wifi or mobile network?
>
physical NIC on the same physical network (all Ubiquiti gear).


⌂66% [[hidden email]:~] $ ifconfig
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bce0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric
0 mtu 1500
        options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
        ether a4:ba:db:29:66:95
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
bce1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
        ether a4:ba:db:29:66:97
        media: Ethernet autoselect
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
bce2: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
        ether a4:ba:db:29:66:99
        media: Ethernet autoselect
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
bce3: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
        ether a4:ba:db:29:66:9b
        media: Ethernet autoselect
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
1500
        ether 02:d7:b8:51:f2:00
        inet6 fe80::d7:b8ff:fe51:f200%bridge0 prefixlen 64 scopeid 0x6
        inet6 2600:1700:210:b180:d7:b8ff:fe51:f200 prefixlen 64 autoconf
        inet 192.168.200.4 netmask 0xfffffc00 broadcast 192.168.203.255
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: bce0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 2 priority 128 path cost 55
        groups: bridge
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
⌂68% [[hidden email]:~] $

bridge0 via bce0 to the "main-switch" . and all the
AP's,switches, etc.



--
Larry Rosenman                     http://www.lerctr.org/~ler
Phone: +1 214-642-9640                 E-Mail: [hidden email]
US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: SNMP/No Bufferspace

Eugene Grosbein-10
04.06.2019 0:49, Larry Rosenman wrote:

> On 06/03/2019 12:23 pm, Eugene Grosbein wrote:
>> 03.06.2019 22:56, Larry Rosenman wrote:
>>
>>> I have a mrtg job that runs every 5 minutes on a FreeBSD-CURRENT box and gets this randomly:
>>>
>>> SNMP Error:
>>> send_query: No buffer space available
>>
>> [skip]
>>
>>> How can I debug this?
>>
>> Your outgoing network interface stalls for some reason.
>> Where is this traffic directed to? Some kind of VPN tunnel? Physical
>> NIC that looses link?
>> Some Wifi or mobile network?
>>
> physical NIC on the same physical network (all Ubiquiti gear).
>
>
> ⌂66% [[hidden email]:~] $ ifconfig
> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
>     options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
>     inet6 ::1 prefixlen 128
>     inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
>     inet 127.0.0.1 netmask 0xff000000
>     groups: lo
>     nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
> bce0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
>     options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
>     ether a4:ba:db:29:66:95
>     media: Ethernet autoselect (1000baseT <full-duplex>)
>     status: active
>     nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> bce1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
>     options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
>     ether a4:ba:db:29:66:97
>     media: Ethernet autoselect
>     nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> bce2: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
>     options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
>     ether a4:ba:db:29:66:99
>     media: Ethernet autoselect
>     nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> bce3: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
>     options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
>     ether a4:ba:db:29:66:9b
>     media: Ethernet autoselect
>     nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
>     ether 02:d7:b8:51:f2:00
>     inet6 fe80::d7:b8ff:fe51:f200%bridge0 prefixlen 64 scopeid 0x6
>     inet6 2600:1700:210:b180:d7:b8ff:fe51:f200 prefixlen 64 autoconf
>     inet 192.168.200.4 netmask 0xfffffc00 broadcast 192.168.203.255
>     id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
>     maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
>     root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
>     member: bce0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
>             ifmaxaddr 0 port 2 priority 128 path cost 55
>     groups: bridge
>     nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
> ⌂68% [[hidden email]:~] $
>
> bridge0 via bce0 to the "main-switch" . and all the
> AP's,switches, etc.

You should search system logs for possible link problems
and check counters shown by "netstat -idnh" (errs/drop/coll).

Also check counters by "netstat -m" for mbufs/mbuf clusters.

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: SNMP/No Bufferspace

Larry Rosenman
On 06/03/2019 1:09 pm, Eugene Grosbein wrote:

> 04.06.2019 0:49, Larry Rosenman wrote:
>
>> On 06/03/2019 12:23 pm, Eugene Grosbein wrote:
>>> 03.06.2019 22:56, Larry Rosenman wrote:
>>>
>>>> I have a mrtg job that runs every 5 minutes on a FreeBSD-CURRENT box
>>>> and gets this randomly:
>>>>
>>>> SNMP Error:
>>>> send_query: No buffer space available
>>>
>>> [skip]
>>>
>>>> How can I debug this?
>>>
>>> Your outgoing network interface stalls for some reason.
>>> Where is this traffic directed to? Some kind of VPN tunnel? Physical
>>> NIC that looses link?
>>> Some Wifi or mobile network?
>>>
>> physical NIC on the same physical network (all Ubiquiti gear).
>>
>>
>> ⌂66% [[hidden email]:~] $ ifconfig
>> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
>>     options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
>>     inet6 ::1 prefixlen 128
>>     inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
>>     inet 127.0.0.1 netmask 0xff000000
>>     groups: lo
>>     nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
>> bce0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST>
>> metric 0 mtu 1500
>>    
>> options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
>>     ether a4:ba:db:29:66:95
>>     media: Ethernet autoselect (1000baseT <full-duplex>)
>>     status: active
>>     nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
>> bce1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
>>    
>> options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
>>     ether a4:ba:db:29:66:97
>>     media: Ethernet autoselect
>>     nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
>> bce2: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
>>    
>> options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
>>     ether a4:ba:db:29:66:99
>>     media: Ethernet autoselect
>>     nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
>> bce3: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
>>    
>> options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
>>     ether a4:ba:db:29:66:9b
>>     media: Ethernet autoselect
>>     nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
>> bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0
>> mtu 1500
>>     ether 02:d7:b8:51:f2:00
>>     inet6 fe80::d7:b8ff:fe51:f200%bridge0 prefixlen 64 scopeid 0x6
>>     inet6 2600:1700:210:b180:d7:b8ff:fe51:f200 prefixlen 64 autoconf
>>     inet 192.168.200.4 netmask 0xfffffc00 broadcast 192.168.203.255
>>     id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
>>     maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
>>     root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
>>     member: bce0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
>>             ifmaxaddr 0 port 2 priority 128 path cost 55
>>     groups: bridge
>>     nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
>> ⌂68% [[hidden email]:~] $
>>
>> bridge0 via bce0 to the "main-switch" . and all the
>> AP's,switches, etc.
>
> You should search system logs for possible link problems
> and check counters shown by "netstat -idnh" (errs/drop/coll).
>
> Also check counters by "netstat -m" for mbufs/mbuf clusters.


⌂79% [[hidden email]:~] $ netstat -idnh
Name    Mtu Network       Address              Ipkts Ierrs Idrop    
Opkts Oerrs  Coll  Drop
lo0     16K <Link#1>      lo0                    27M     0     0      
27M     0     0     0
lo0       - ::1/128       ::1                   582k     -     -    
582k     -     -     -
lo0       - fe80::%lo0/64 fe80::1%lo0              0     -     -        
0     -     -     -
lo0       - 127.0.0.0/8   127.0.0.1             190k     -     -      
26M     -     -     -
bce0   1.5K <Link#2>      a4:ba:db:29:66:95     715M  186k     0    
533M     0     0   15M
bce1*  1.5K <Link#3>      a4:ba:db:29:66:97        0     0     0        
0     0     0     0
bce2*  1.5K <Link#4>      a4:ba:db:29:66:99        0     0     0        
0     0     0     0
bce3*  1.5K <Link#5>      a4:ba:db:29:66:9b        0     0     0        
0     0     0     0
bridg  1.5K <Link#6>      02:d7:b8:51:f2:00     715M     0     0    
533M   15M     0     0
bridg     - fe80::%bridge fe80::d7:b8ff:fe5     7.2k     -     -    
7.2k     -     -     -
bridg     - 2600:1700:210 2600:1700:210:b18     578M     -     -    
476M     -     -     -
bridg     - 192.168.200.0 192.168.200.4         161M     -     -      
72M     -     -     -
⌂86% [[hidden email]:~] $ netstat -m
3212/18193/21405 mbufs in use (current/cache/total)
2078/11604/13682/8175421 mbuf clusters in use (current/cache/total/max)
2045/8075 mbuf+clusters out of packet secondary zone in use
(current/cache)
6/9355/9361/4087710 4k (page size) jumbo clusters in use
(current/cache/total/max)
0/0/0/1211173 9k jumbo clusters in use (current/cache/total/max)
0/0/0/681285 16k jumbo clusters in use (current/cache/total/max)
4983K/65176K/70159K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters delayed (4k/9k/16k)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0 sendfile syscalls
0 sendfile syscalls completed without I/O request
0 requests for I/O initiated by sendfile
0 pages read by sendfile as part of a request
0 pages were valid at time of a sendfile request
0 pages were valid and substituted to bogus page
0 pages were requested for read ahead by applications
0 pages were read ahead by sendfile
0 times sendfile encountered an already busy page
0 requests for sfbufs denied
0 requests for sfbufs delayed
⌂83% [[hidden email]:~] $

Hrm.  I may be able to move directly away from the bridge.....

⌂83% [[hidden email]:~] $ grep bce0 /var/log/messages
⌂80% [[hidden email]:~] 1 $ grep bridge /var/log/messages
Jun  3 02:48:30 borg kernel: arp: 192.168.200.54 moved from
04:c9:d9:63:22:87 to 04:c9:d9:63:22:89 on bridge0
Jun  3 02:49:49 borg kernel: arp: 192.168.200.54 moved from
04:c9:d9:63:22:89 to 04:c9:d9:63:22:87 on bridge0
⌂83% [[hidden email]:~] $ uptime
  1:15PM  up 1 day, 18:11, 1 user, load averages: 20.08, 18.82, 18.13
⌂83% [[hidden email]:~] $
--
Larry Rosenman                     http://www.lerctr.org/~ler
Phone: +1 214-642-9640                 E-Mail: [hidden email]
US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: SNMP/No Bufferspace

Eugene Grosbein-10
04.06.2019 1:15, Larry Rosenman wrote:

> Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll  Drop
> bce0   1.5K <Link#2>      a4:ba:db:29:66:95     715M  186k     0     533M     0     0   15M

Something really bad happening at driver level or physical level:
way too many input errors (perhaps, bad CRC) and output drops (due to busy media?).

I prefer to max out interface FIFO length while using bce(4) by means of /boot/loader.conf
when link is heavily utilized:

# max = 8*255 = 2040
hw.bce.tx_pages=8

Also, check driver-level stats with sysctl dev.bce.0
If possible, check link partner - managed switch port counters or what does it have.
Maybe, replace twisted pair cable.

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: SNMP/No Bufferspace

Larry Rosenman
On 06/03/2019 1:27 pm, Eugene Grosbein wrote:

> 04.06.2019 1:15, Larry Rosenman wrote:
>
>> Name    Mtu Network       Address              Ipkts Ierrs Idrop    
>> Opkts Oerrs  Coll  Drop
>> bce0   1.5K <Link#2>      a4:ba:db:29:66:95     715M  186k     0    
>> 533M     0     0   15M
>
> Something really bad happening at driver level or physical level:
> way too many input errors (perhaps, bad CRC) and output drops (due to
> busy media?).
>
> I prefer to max out interface FIFO length while using bce(4) by means
> of /boot/loader.conf
> when link is heavily utilized:
>
> # max = 8*255 = 2040
> hw.bce.tx_pages=8
>
> Also, check driver-level stats with sysctl dev.bce.0
> If possible, check link partner - managed switch port counters or what
> does it have.
> Maybe, replace twisted pair cable.


Ok, I've removed the bridge (don't need it).  Added the tunable
and will check it after a few hours.  I *REALLY* appreciate
the help diagnosing this.  (If need be I can replace the cable, but
let's
see what happens now).

--
Larry Rosenman                     http://www.lerctr.org/~ler
Phone: +1 214-642-9640                 E-Mail: [hidden email]
US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: SNMP/No Bufferspace

Eugene Grosbein-10
04.06.2019 1:42, Larry Rosenman wrote:

> Ok, I've removed the bridge (don't need it).  Added the tunable
> and will check it after a few hours.  I *REALLY* appreciate
> the help diagnosing this.  (If need be I can replace the cable, but let's
> see what happens now).

You already run mrtg, so make sure you have graphs for bce0 usage.
Maybe you have short-lived traffic storms.


_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: SNMP/No Bufferspace

Larry Rosenman
On 06/03/2019 2:23 pm, Eugene Grosbein wrote:

> 04.06.2019 1:42, Larry Rosenman wrote:
>
>> Ok, I've removed the bridge (don't need it).  Added the tunable
>> and will check it after a few hours.  I *REALLY* appreciate
>> the help diagnosing this.  (If need be I can replace the cable, but
>> let's
>> see what happens now).
>
> You already run mrtg, so make sure you have graphs for bce0 usage.
> Maybe you have short-lived traffic storms.

I upped the RX pages as well, and that seems(!) to help with having all
3 of my network backups
running (bacula) which apparently was what was tripping it up.

⌂75% [[hidden email]:~] $ sysctl hw.bce
hw.bce.rx_ticks: 18
hw.bce.rx_ticks_int: 18
hw.bce.rx_quick_cons_trip: 6
hw.bce.rx_quick_cons_trip_int: 6
hw.bce.tx_ticks: 80
hw.bce.tx_ticks_int: 80
hw.bce.tx_quick_cons_trip: 20
hw.bce.tx_quick_cons_trip_int: 20
hw.bce.strict_rx_mtu: 0
hw.bce.hdr_split: 1
hw.bce.tx_pages: 8
hw.bce.rx_pages: 8
hw.bce.msi_enable: 1
hw.bce.tso_enable: 1
hw.bce.verbose: 1
⌂77% [[hidden email]:~] $ sysctl dev.bce.0
dev.bce.0.com_no_buffers: 0
dev.bce.0.stat_CatchupInRuleCheckerP4Hit: 0
dev.bce.0.stat_CatchupInMBUFDiscards: 0
dev.bce.0.stat_CatchupInFTQDiscards: 0
dev.bce.0.stat_CatchupInRuleCheckerDiscards: 0
dev.bce.0.stat_IfInRuleCheckerP4Hit: 10301
dev.bce.0.stat_IfInMBUFDiscards: 0
dev.bce.0.stat_IfInFTQDiscards: 0
dev.bce.0.stat_IfInRuleCheckerDiscards: 0
dev.bce.0.stat_IfInFramesL2FilterDiscards: 5741
dev.bce.0.stat_XoffStateEntered: 0
dev.bce.0.stat_MacControlFramesReceived: 0
dev.bce.0.stat_FlowControlDone: 0
dev.bce.0.stat_OutXoffSent: 0
dev.bce.0.stat_OutXonSent: 0
dev.bce.0.stat_XoffPauseFramesReceived: 0
dev.bce.0.stat_XonPauseFramesReceived: 0
dev.bce.0.stat_EtherStatsPktsTx1523Octetsto9022Octets: 0
dev.bce.0.stat_EtherStatsPktsTx1024Octetsto1522Octets: 7461404
dev.bce.0.stat_EtherStatsPktsTx512Octetsto1023Octets: 488
dev.bce.0.stat_EtherStatsPktsTx256Octetsto511Octets: 1331
dev.bce.0.stat_EtherStatsPktsTx128Octetsto255Octets: 139674
dev.bce.0.stat_EtherStatsPktsTx65Octetsto127Octets: 1492188
dev.bce.0.stat_EtherStatsPktsTx64Octets: 325
dev.bce.0.stat_EtherStatsPktsRx1523Octetsto9022Octets: 0
dev.bce.0.stat_EtherStatsPktsRx1024Octetsto1522Octets: 2890915
dev.bce.0.stat_EtherStatsPktsRx512Octetsto1023Octets: 1494
dev.bce.0.stat_EtherStatsPktsRx256Octetsto511Octets: 167073
dev.bce.0.stat_EtherStatsPktsRx128Octetsto255Octets: 4091
dev.bce.0.stat_EtherStatsPktsRx65Octetsto127Octets: 7602388
dev.bce.0.stat_EtherStatsPktsRx64Octets: 3913
dev.bce.0.stat_EtherStatsOversizePkts: 0
dev.bce.0.stat_EtherStatsUndersizePkts: 0
dev.bce.0.stat_EtherStatsJabbers: 0
dev.bce.0.stat_EtherStatsFragments: 0
dev.bce.0.stat_EtherStatsCollisions: 0
dev.bce.0.stat_Dot3StatsLateCollisions: 0
dev.bce.0.stat_Dot3StatsExcessiveCollisions: 0
dev.bce.0.stat_Dot3StatsDeferredTransmissions: 0
dev.bce.0.stat_Dot3StatsMultipleCollisionFrames: 0
dev.bce.0.stat_Dot3StatsSingleCollisionFrames: 0
dev.bce.0.stat_Dot3StatsAlignmentErrors: 0
dev.bce.0.stat_Dot3StatsFCSErrors: 0
dev.bce.0.stat_Dot3StatsCarrierSenseErrors: 0
dev.bce.0.stat_emac_tx_stat_dot3statsinternalmactransmiterrors: 0
dev.bce.0.stat_IfHCOutBroadcastPkts: 694
dev.bce.0.stat_IfHCOutMulticastPkts: 290
dev.bce.0.stat_IfHCOutUcastPkts: 9094426
dev.bce.0.stat_IfHCInBroadcastPkts: 3746
dev.bce.0.stat_IfHCInMulticastPkts: 6555
dev.bce.0.stat_IfHCInUcastPkts: 10659572
dev.bce.0.stat_IfHCOutBadOctets: 0
dev.bce.0.stat_IfHCOutOctets: 11458190364
dev.bce.0.stat_IfHCInBadOctets: 2384121
dev.bce.0.stat_IfHcInOctets: 5141824959
dev.bce.0.unexpected_attention_count: 0
dev.bce.0.dma_map_addr_tx_failed_count: 0
dev.bce.0.dma_map_addr_rx_failed_count: 0
dev.bce.0.mbuf_frag_count: 0
dev.bce.0.mbuf_alloc_failed_count: 0
dev.bce.0.l2fhdr_error_count: 0
dev.bce.0.%parent: pci1
dev.bce.0.%pnpinfo: vendor=0x14e4 device=0x1639 subvendor=0x1028
subdevice=0x0235 class=0x020000
dev.bce.0.%location: slot=0 function=0 dbsf=pci0:1:0:0
dev.bce.0.%driver: bce
dev.bce.0.%desc: QLogic NetXtreme II BCM5709 1000Base-T (C0)
⌂78% [[hidden email]:~] $

We'll see over the next few hours.

Thanks again.

--
Larry Rosenman                     http://www.lerctr.org/~ler
Phone: +1 214-642-9640                 E-Mail: [hidden email]
US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: SNMP/No Bufferspace

Larry Rosenman
On 06/03/2019 2:36 pm, Larry Rosenman wrote:

> On 06/03/2019 2:23 pm, Eugene Grosbein wrote:
>> 04.06.2019 1:42, Larry Rosenman wrote:
>>
>>> Ok, I've removed the bridge (don't need it).  Added the tunable
>>> and will check it after a few hours.  I *REALLY* appreciate
>>> the help diagnosing this.  (If need be I can replace the cable, but
>>> let's
>>> see what happens now).
>>
>> You already run mrtg, so make sure you have graphs for bce0 usage.
>> Maybe you have short-lived traffic storms.
>
> I upped the RX pages as well, and that seems(!) to help with having
> all 3 of my network backups
> running (bacula) which apparently was what was tripping it up.
>

Definitely happier:
[[hidden email]:~] 1 $ sysctl -d dev.bce.0.com_no_buffers
dev.bce.0.com_no_buffers: Valid packets received but no RX buffers
available
[[hidden email]:~] $

[[hidden email]:~] $ sysctl  dev.bce.0.com_no_buffers
dev.bce.0.com_no_buffers: 0
⌂63% [[hidden email]:~] $ netstat -iadh
Name    Mtu Network       Address              Ipkts Ierrs Idrop    
Opkts Oerrs  Coll  Drop
lo0     16K <Link#1>      lo0                   2.1M     0     0    
2.1M     0     0     0
lo0       - localhost     localhost             150k     -     -    
150k     -     -     -
                           ff02::202%lo0
                           ff01::1%lo0
                           ff02::2:a7c1:4da6
                           ff02::2:ffa7:c14d
                           ff02::1%lo0
                           ff02::1:ff00:1%lo
lo0       - fe80::%lo0/64 fe80::1%lo0              0     -     -        
0     -     -     -
                           ff02::202%lo0
                           ff01::1%lo0
                           ff02::2:a7c1:4da6
                           ff02::2:ffa7:c14d
                           ff02::1%lo0
                           ff02::1:ff00:1%lo
lo0       - your-net      localhost             6.3k     -     -    
2.0M     -     -     -
                           224.0.0.1
bce0   1.5K <Link#2>      a4:ba:db:29:66:95     118M     0     0      
92M     0     0     0
bce0      - 192.168.200.0 borg                   53M     -     -      
26M     -     -     -
bce0      - fe80::%bce0/6 fe80::a6ba:dbff:f      418     -     -      
428     -     -     -
bce0      - 2600:1700:210 2600:1700:210:b18      68M     -     -      
66M     -     -     -
bce1*  1.5K <Link#3>      a4:ba:db:29:66:97        0     0     0        
0     0     0     0
bce2*  1.5K <Link#4>      a4:ba:db:29:66:99        0     0     0        
0     0     0     0
bce3*  1.5K <Link#5>      a4:ba:db:29:66:9b        0     0     0        
0     0     0     0
⌂62% [[hidden email]:~] $


⌂71% [[hidden email]:~] $ sysctl hw.bce
hw.bce.rx_ticks: 18
hw.bce.rx_ticks_int: 18
hw.bce.rx_quick_cons_trip: 6
hw.bce.rx_quick_cons_trip_int: 6
hw.bce.tx_ticks: 80
hw.bce.tx_ticks_int: 80
hw.bce.tx_quick_cons_trip: 20
hw.bce.tx_quick_cons_trip_int: 20
hw.bce.strict_rx_mtu: 0
hw.bce.hdr_split: 1
hw.bce.tx_pages: 8
hw.bce.rx_pages: 8
hw.bce.msi_enable: 1
hw.bce.tso_enable: 1
hw.bce.verbose: 1
⌂74% [[hidden email]:~]


--
Larry Rosenman                     http://www.lerctr.org/~ler
Phone: +1 214-642-9640                 E-Mail: [hidden email]
US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[hidden email]"