'dropped due to full socket buffers' by SNMP

classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|

'dropped due to full socket buffers' by SNMP

Victor Gamov-2
Hi All

Can somebody help me to get UDP 'dropped due to full socket buffers' by
SNMP?  Is it possible?  Now I'm getting it with
`netstat -n -p udp -f inet -s`
but SNMP will be more useful for remote monitoring.

Thanks!

--
CU,
Victor Gamov
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 'dropped due to full socket buffers' by SNMP

Victor Gamov-2
And one more question about 'dropped due to full socket buffers':  how
to avoid it?  Which params must be tuned?

Thanks.


On 30/11/2020 18:33, Victor Gamov wrote:
> Hi All
>
> Can somebody help me to get UDP 'dropped due to full socket buffers' by
> SNMP?  Is it possible?  Now I'm getting it with
> `netstat -n -p udp -f inet -s`
> but SNMP will be more useful for remote monitoring.
>
> Thanks!

--
CU,
Victor Gamov
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 'dropped due to full socket buffers' by SNMP

Eugene Grosbein-10
29.12.2020 23:36, Victor Gamov wrote:

Please do not top-post.

> Thanks.
>
>
> On 30/11/2020 18:33, Victor Gamov wrote:
>> Hi All
>>
>> Can somebody help me to get UDP 'dropped due to full socket buffers' by SNMP?  Is it possible?  Now I'm getting it with
>> `netstat -n -p udp -f inet -s`
>> but SNMP will be more useful for remote monitoring.
>>
> And one more question about 'dropped due to full socket buffers':  how to avoid it?  Which params must be tuned?

Both problems generally mean that outgoing link for that UDP packets is congested,
so socket buffers start to fill until they are full.

What is "media" for that UDP data - some syntetic tunnel or plain ethernet link or something else?

You may find useful this old thread dealing with similar problem:
http://freebsd.1045724.x6.nabble.com/SNMP-No-Bufferspace-td6333081.html

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 'dropped due to full socket buffers' by SNMP

Victor Gamov-2
Hi Eugene

Thank for your reply

On 30/12/2020 02:13, Eugene Grosbein wrote:

> 29.12.2020 23:36, Victor Gamov wrote:
>
> Please do not top-post.
>
>> Thanks.
>>
>>
>> On 30/11/2020 18:33, Victor Gamov wrote:
>>> Hi All
>>>
>>> Can somebody help me to get UDP 'dropped due to full socket buffers' by SNMP?  Is it possible?  Now I'm getting it with
>>> `netstat -n -p udp -f inet -s`
>>> but SNMP will be more useful for remote monitoring.
>>>
>> And one more question about 'dropped due to full socket buffers':  how to avoid it?  Which params must be tuned?
>
> Both problems generally mean that outgoing link for that UDP packets is congested,
> so socket buffers start to fill until they are full.
>
> What is "media" for that UDP data - some syntetic tunnel or plain ethernet link or something else?

It's 10G Intel card (via ix driver) attached to D-Link switch

This Host-A got many multicast UDP streams via many VLANs and resend
streams to one VLAN-750 (mainly).  Every multicast serviced by its own
process.

Interface utilization about 1Gbit/s for receive and 800Mbit/s for transmit.

> You may find useful this old thread dealing with similar problem:
> http://freebsd.1045724.x6.nabble.com/SNMP-No-Bufferspace-td6333081.html

Thanks for this link.

'vmstat -i | grep ix0' output followed:
=====
irq264: ix0:rxq0             90527644527      21675
irq265: ix0:rxq1             95793018351      22936
irq266: ix0:rxq2             96075009541      23004
irq267: ix0:rxq3             93444280619      22374
irq268: ix0:aq                         3          0
=====


'netstat -idnh -W' show no errs/drops


'netstat -m'
=====
12525/7470/19995 mbufs in use (current/cache/total)
12484/4766/17250/1005602 mbuf clusters in use (current/cache/total/max)
158/1866 mbuf+clusters out of packet secondary zone in use (current/cache)
0/116/116/502800 4k (page size) jumbo clusters in use
(current/cache/total/max)
0/0/0/148978 9k jumbo clusters in use (current/cache/total/max)
0/0/0/83800 16k jumbo clusters in use (current/cache/total/max)
28099K/11863K/39962K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters delayed (4k/9k/16k)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0 sendfile syscalls
0 sendfile syscalls completed without I/O request
0 requests for I/O initiated by sendfile
0 pages read by sendfile as part of a request
0 pages were valid at time of a sendfile request
0 pages were valid and substituted to bogus page
0 pages were requested for read ahead by applications
0 pages were read ahead by sendfile
0 times sendfile encountered an already busy page
0 requests for sfbufs denied
0 requests for sfbufs delayed
======



Currently I'm thinking about ethernet flow control: Host-B connected to
VLAN-750 on the third switch has 1G link (via igb driver) and both
Host-A and Host-B has fc=3.  So when Host-B get microburst it can send
PAUSE and Host-A start to fill queue.  But FC disabled for all switches
and I'm not sure about PAUSE can be propagated from Host-A to Host-B

--
CU,
Victor Gamov
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 'dropped due to full socket buffers' by SNMP

Eugene Grosbein-10
30.12.2020 16:44, Victor Gamov wrote:

> Currently I'm thinking about ethernet flow control: Host-B connected to VLAN-750 on the third switch has 1G link (via igb driver)
> and both Host-A and Host-B has fc=3.  So when Host-B get microburst it can send PAUSE and Host-A start to fill queue.
> But FC disabled for all switches and I'm not sure about PAUSE can be propagated from Host-A to Host-B

AFAIK, pause frames are not forwarded. For short term you should make flow control settings consistent
between link partners, maybe increase kern.ipc.maxsockbuf and then net.inet.udp.recvspace.
But for long term you should consider adding more links and create LACP aggregate
so not input nor output link be close to congestion.


_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 'dropped due to full socket buffers' by SNMP

Victor Gamov-2


On 30/12/2020 12:57, Eugene Grosbein wrote:
> 30.12.2020 16:44, Victor Gamov wrote:
>
>> Currently I'm thinking about ethernet flow control: Host-B connected to VLAN-750 on the third switch has 1G link (via igb driver)
>> and both Host-A and Host-B has fc=3.  So when Host-B get microburst it can send PAUSE and Host-A start to fill queue.
>> But FC disabled for all switches and I'm not sure about PAUSE can be propagated from Host-A to Host-B
>
> AFAIK, pause frames are not forwarded. For short term you should make flow control settings consistent
> between link partners,

As I understand hw.ix.flow_control=3 to allow flow-control for
negotiation.  Real PAUSE setting will be set during negotiation.  So
where I can find active flow-control setting for host interface?

> maybe increase kern.ipc.maxsockbuf and then net.inet.udp.recvspace.

Eugene, at first message you suppose Host-A (sender) "outgoing link for
that UDP packets is congested" because this host shows non-zero "dropped
due to full socket buffers".   So is net.inet.udp.recvspace increasing
on Host-B (mainly receiver) will be affected for this congestion?  Or I
need to try to increase both kern.ipc.maxsockbuf and
net.inet.udp.recvspace on both hosts?

Also how I can check current sockbuf usage?

> But for long term you should consider adding more links and create LACP aggregate
> so not input nor output link be close to congestion.

I'll migrate to 10G on Host-B at next month.


Thanks for your advise!

--
CU,
Victor Gamov
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 'dropped due to full socket buffers' by SNMP

Eugene Grosbein-10
30.12.2020 23:08, Victor Gamov wrote:

> As I understand hw.ix.flow_control=3 to allow flow-control for negotiation.
> Real PAUSE setting will be set during negotiation.

At the moment of congestion.

>  So where I can find active flow-control setting for host interface?

Can't check for ix just now, but for em(4) there is sysctl dev.em.0.fc.
It should be similar for ix.

>> maybe increase kern.ipc.maxsockbuf and then net.inet.udp.recvspace.
> Eugene, at first message you suppose Host-A (sender) "outgoing link for that UDP packets is congested"
> because this host shows non-zero "dropped due to full socket buffers".
> So is net.inet.udp.recvspace increasing on Host-B (mainly receiver) will be affected for this congestion?

Can't tell in details without going deep into your setup :-)
You can try it yourself and verify quickly.

> Or I need to try to increase both kern.ipc.maxsockbuf and net.inet.udp.recvspace on both hosts?

Tune one that drops UDP.

> Also how I can check current sockbuf usage?

netstat -xn

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 'dropped due to full socket buffers' by SNMP

Victor Gamov-2
Hi Eugene!

Thanks for your responces.

And Happy New Year for everyone!

On 01.01.2021 03:19, Eugene Grosbein wrote:
> 30.12.2020 23:08, Victor Gamov wrote:
>
>> As I understand hw.ix.flow_control=3 to allow flow-control for negotiation.
>> Real PAUSE setting will be set during negotiation.
>
> At the moment of congestion.

As I understand PAUSE feature negotiated during auto-negotiation
process. If flow-control disabled on one side (switch for example) then
other side (host) will not to use this feature too.  Is it right?

>>   So where I can find active flow-control setting for host interface?
>
> Can't check for ix just now, but for em(4) there is sysctl dev.em.0.fc.
> It should be similar for ix.

I have hw.ix.flow_control=3 (what does is it means ?) and dev.ix.0.fc=3
(and what does is it means?)


>>> maybe increase kern.ipc.maxsockbuf and then net.inet.udp.recvspace.
>> Eugene, at first message you suppose Host-A (sender) "outgoing link for that UDP packets is congested"
>> because this host shows non-zero "dropped due to full socket buffers".
>> So is net.inet.udp.recvspace increasing on Host-B (mainly receiver) will be affected for this congestion?
>
> Can't tell in details without going deep into your setup :-)
> You can try it yourself and verify quickly.
>
>> Or I need to try to increase both kern.ipc.maxsockbuf and net.inet.udp.recvspace on both hosts?
>
> Tune one that drops UDP.
>
>> Also how I can check current sockbuf usage?
>
> netstat -xn

Unfortunately it never shoes counters about UDP multicast traffic.

I'll increase kern.ipc.maxsockbuf and net.inet.udp.recvspace at next
week and write about results.


Back to my original question: is it possible to monitor `netstat -n -p
udp -f inet -s` counters by SNMP?


--
CU,
Victor Gamov

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 'dropped due to full socket buffers' by SNMP

Eugene Grosbein-10
05.01.2021 16:39, Victor Gamov wrote:

>>> As I understand hw.ix.flow_control=3 to allow flow-control for negotiation.
>>> Real PAUSE setting will be set during negotiation.
>>
>> At the moment of congestion.
>
> As I understand PAUSE feature negotiated during auto-negotiation process. If flow-control disabled on one side (switch for example) then other side (host) will not to use this feature too.  Is it right?

Modulo firmware bugs.

>>>   So where I can find active flow-control setting for host interface?
>> Can't check for ix just now, but for em(4) there is sysctl dev.em.0.fc.
>> It should be similar for ix.
> I have hw.ix.flow_control=3 (what does is it means ?) and dev.ix.0.fc=3 (and what does is it means?)

/* Flow Control Settings */
enum ixgbe_fc_mode {
        ixgbe_fc_none = 0,
        ixgbe_fc_rx_pause,
        ixgbe_fc_tx_pause,
        ixgbe_fc_full,
        ixgbe_fc_default
};

So, ixgbe_fc_full = 3 (full rx/tx flow control enabled, see sys/dev/ixgbe/ixgbe_type.h).
 
> Back to my original question: is it possible to monitor `netstat -n -p udp -f inet -s` counters by SNMP?

There is no standard MIB for these counters but you may always export it with net-mgmt/bsnmp-ucd port
and custom OIDs if you use stock bsnmpd (bsnmp-ucd man page has an example).
There is similar possibility for net-snmpd.


_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 'dropped due to full socket buffers' by SNMP

Victor Gamov-2
In reply to this post by Victor Gamov-2
Hi All

On 05.01.2021 12:39, Victor Gamov wrote:

> Hi Eugene!
>
> Thanks for your responces.
>
> And Happy New Year for everyone!
>
> On 01.01.2021 03:19, Eugene Grosbein wrote:
>> 30.12.2020 23:08, Victor Gamov wrote:
>>
>>> As I understand hw.ix.flow_control=3 to allow flow-control for
>>> negotiation.
>>> Real PAUSE setting will be set during negotiation.
>>
>> At the moment of congestion.
>
> As I understand PAUSE feature negotiated during auto-negotiation
> process. If flow-control disabled on one side (switch for example) then
> other side (host) will not to use this feature too.  Is it right?
>
>>>   So where I can find active flow-control setting for host interface?
>>
>> Can't check for ix just now, but for em(4) there is sysctl dev.em.0.fc.
>> It should be similar for ix.
>
> I have hw.ix.flow_control=3 (what does is it means ?) and dev.ix.0.fc=3
> (and what does is it means?)
>
>
>>>> maybe increase kern.ipc.maxsockbuf and then net.inet.udp.recvspace.
>>> Eugene, at first message you suppose Host-A (sender) "outgoing link
>>> for that UDP packets is congested"
>>> because this host shows non-zero "dropped due to full socket buffers".
>>> So is net.inet.udp.recvspace increasing on Host-B (mainly receiver)
>>> will be affected for this congestion?
>>
>> Can't tell in details without going deep into your setup :-)
>> You can try it yourself and verify quickly.
>>
>>> Or I need to try to increase both kern.ipc.maxsockbuf and
>>> net.inet.udp.recvspace on both hosts?
>>
>> Tune one that drops UDP.
>>
>>> Also how I can check current sockbuf usage?
>>
>> netstat -xn
>
> Unfortunately it never shoes counters about UDP multicast traffic.
>
> I'll increase kern.ipc.maxsockbuf and net.inet.udp.recvspace at next
> week and write about results.

I increase kern.ipc.maxsockbuf from 2097152 -> 2597152 -> 3145728 but
netstat -sn -p udp | grep 'dropped due to full socket buffers' still
show dropped packets.

Then I increase net.inet.udp.recvspace 84160 -> 105200 but 'dropped due
to full socket buffers' packets still here.

Do I need to try to increase something else?

--
CU,
Victor Gamov
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 'dropped due to full socket buffers' by SNMP

Eugene Grosbein-10
22.01.2021 16:27, Victor Gamov wrote:

> I increase kern.ipc.maxsockbuf from 2097152 -> 2597152 -> 3145728 but
> netstat -sn -p udp | grep 'dropped due to full socket buffers' still show dropped packets.
>
> Then I increase net.inet.udp.recvspace 84160 -> 105200 but 'dropped due to full socket buffers' packets still here.
>
> Do I need to try to increase something else?

Increase of various buffers may help against very short traffic bursts or limited time interface congestion.
It won't help in case you have permanent (or nearly permanent) link overload.

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 'dropped due to full socket buffers' by SNMP

Victor Gamov-2


On 22.01.2021 12:52, Eugene Grosbein wrote:

> 22.01.2021 16:27, Victor Gamov wrote:
>
>> I increase kern.ipc.maxsockbuf from 2097152 -> 2597152 -> 3145728 but
>> netstat -sn -p udp | grep 'dropped due to full socket buffers' still show dropped packets.
>>
>> Then I increase net.inet.udp.recvspace 84160 -> 105200 but 'dropped due to full socket buffers' packets still here.
>>
>> Do I need to try to increase something else?
>
> Increase of various buffers may help against very short traffic bursts or limited time interface congestion.
> It won't help in case you have permanent (or nearly permanent) link overload.

No link overload: this host attached to network via 10G ix0, many VLANs
on this ix0 and some sender on every VLAN sends multicast traffic to
this host.  Total input traffic about 1Gbit/s

--
CU,
Victor Gamov
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 'dropped due to full socket buffers' by SNMP

Eugene Grosbein-10
22.01.2021 17:02, Victor Gamov wrote:

> No link overload: this host attached to network via 10G ix0, many VLANs on this ix0 and some sender on every VLAN sends multicast traffic to this host.  Total input traffic about 1Gbit/s

What FreeBSD version do you use currently? Do you use IPv6 for UDP or IPv4 only?


_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 'dropped due to full socket buffers' by SNMP

Victor Gamov-2


On 22.01.2021 13:21, Eugene Grosbein wrote:
> 22.01.2021 17:02, Victor Gamov wrote:
>
>> No link overload: this host attached to network via 10G ix0, many
>> VLANs on this ix0 and some sender on every VLAN sends multicast
>> traffic to this host.  Total input traffic about 1Gbit/s
>
> What FreeBSD version do you use currently? Do you use IPv6 for UDP or
> IPv4 only?


FreeBSD 12.2-STABLE r366543 GENERIC  amd64

UDP-4 only

--
CU,
Victor Gamov
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 'dropped due to full socket buffers' by SNMP

Eugene Grosbein-10
22.01.2021 19:28, Victor Gamov wrote:

> On 22.01.2021 13:21, Eugene Grosbein wrote:
>> 22.01.2021 17:02, Victor Gamov wrote:
>>
>>> No link overload: this host attached to network via 10G ix0, many
>>> VLANs on this ix0 and some sender on every VLAN sends multicast
>>> traffic to this host.  Total input traffic about 1Gbit/s
>>
>> What FreeBSD version do you use currently? Do you use IPv6 for UDP or
>> IPv4 only?
>
>
> FreeBSD 12.2-STABLE r366543 GENERIC  amd64
>
> UDP-4 only

In case of IPv4 UDP the counter "dropped due to full socket buffers"
is increased for incoming packets only. Therefore, the problem is in a code
processing incoming stream(s): either it locks for long time on something,
or it just has no enough raw CPU horsepower to deal with incoming stream.

Look at "top -SHPI" CPU counters, if your CPU cores are loaded evenly;
if some of cores became overloaded sometimes. You should also draw per-cpu load graphs.
(f.e. sysctl kern.cp_times)



_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 'dropped due to full socket buffers' by SNMP

Victor Gamov-2


On 22.01.2021 16:09, Eugene Grosbein wrote:

> 22.01.2021 19:28, Victor Gamov wrote:
>
>> On 22.01.2021 13:21, Eugene Grosbein wrote:
>>> 22.01.2021 17:02, Victor Gamov wrote:
>>>
>>>> No link overload: this host attached to network via 10G ix0, many
>>>> VLANs on this ix0 and some sender on every VLAN sends multicast
>>>> traffic to this host.  Total input traffic about 1Gbit/s
>>>
>>> What FreeBSD version do you use currently? Do you use IPv6 for UDP or
>>> IPv4 only?
>>
>>
>> FreeBSD 12.2-STABLE r366543 GENERIC  amd64
>>
>> UDP-4 only
>
> In case of IPv4 UDP the counter "dropped due to full socket buffers"
> is increased for incoming packets only. Therefore, the problem is in a code
> processing incoming stream(s): either it locks for long time on something,
> or it just has no enough raw CPU horsepower to deal with incoming stream.
>
> Look at "top -SHPI" CPU counters, if your CPU cores are loaded evenly;

it's CPU E3-1270 v6 @ 3.80GHz with WCPU about 60% idle + 9%
kernel{if_io_tqg_X) + 5% intr{swi1: netisr X).  And many processes about
1% WCPU for multicast receive/resend (one for every multicast)

Also "netstat -na -p udp" shows me zero or very small Recv-Q

> if some of cores became overloaded sometimes. You should also draw per-cpu load graphs.
> (f.e. sysctl kern.cp_times)

I have SNMP stats and it show me about 40% load


I have no visible problems with growing "dropped due to full socket
buffers" but I think it's not well when this counter increasing.

--
CU,
Victor Gamov
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 'dropped due to full socket buffers' by SNMP

Eugene Grosbein-10
22.01.2021 20:26, Victor Gamov wrote:

>>>> What FreeBSD version do you use currently? Do you use IPv6 for UDP or
>>>> IPv4 only?
>>>
>>>
>>> FreeBSD 12.2-STABLE r366543 GENERIC  amd64
>>>
>>> UDP-4 only
>>
>> In case of IPv4 UDP the counter "dropped due to full socket buffers"
>> is increased for incoming packets only. Therefore, the problem is in a code
>> processing incoming stream(s): either it locks for long time on something,
>> or it just has no enough raw CPU horsepower to deal with incoming stream.
>>
>> Look at "top -SHPI" CPU counters, if your CPU cores are loaded evenly;
>
> it's CPU E3-1270 v6 @ 3.80GHz with WCPU about 60% idle + 9% kernel{if_io_tqg_X) + 5% intr{swi1: netisr X).  And many processes about 1% WCPU for multicast receive/resend (one for every multicast)
>
> Also "netstat -na -p udp" shows me zero or very small Recv-Q
>
>> if some of cores became overloaded sometimes. You should also draw per-cpu load graphs.
>> (f.e. sysctl kern.cp_times)
>
> I have SNMP stats and it show me about 40% load

There is no standard MIBs to show per-core load. You get only an average via standard SNMP MIBs
and average can be misleading and hide 100% load of some cores.

If you really have no overloaded cores, then this should be locking problem that prevents processing code
to do its job timely.



_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[hidden email]"