[diskless] pkg takes 100% of a CPU

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

[diskless] pkg takes 100% of a CPU

BERTRAND Joël
        Hello,

        I have installed a diskless FreeBSD (11.1-RELEASE). This workstation
runs a customized kernel to use Realtek re driver as I have seen some
bugs in FreeBSD one's.

        Since I have upgraded 11.0 to 11.1, each night, pkg stalls and takes
100% of a CPU :

PID USERNAME    THR PRI NICE   SIZE    RES STATE   C TIME  WCPU COMMAND
65609 root        1 103    0 51320K 12956K CPU1  1 425:16  98.43% pkg

        I can removed this command from crontab, but I'm pretty sure there is a
bug somewhere between FreeBSD, pkg and nfs root. For information, my
server runs NetBSD 8 (thus root nfs is mounted with v3/TCP).

        Best regards,

        JB
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: [diskless] pkg takes 100% of a CPU

Steven Hartland
When we’ve seen it using 100% it’s been doing comprehension stuff which
usually finishes you just have to wait. Not sure if that’s what you’re
seeing?

On Sat, 7 Apr 2018 at 09:12, BERTRAND Joël <[hidden email]>
wrote:

>         Hello,
>
>         I have installed a diskless FreeBSD (11.1-RELEASE). This
> workstation
> runs a customized kernel to use Realtek re driver as I have seen some
> bugs in FreeBSD one's.
>
>         Since I have upgraded 11.0 to 11.1, each night, pkg stalls and
> takes
> 100% of a CPU :
>
> PID USERNAME    THR PRI NICE   SIZE    RES STATE   C TIME  WCPU COMMAND
> 65609 root        1 103    0 51320K 12956K CPU1  1 425:16  98.43% pkg
>
>         I can removed this command from crontab, but I'm pretty sure there
> is a
> bug somewhere between FreeBSD, pkg and nfs root. For information, my
> server runs NetBSD 8 (thus root nfs is mounted with v3/TCP).
>
>         Best regards,
>
>         JB
> _______________________________________________
> [hidden email] mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
> To unsubscribe, send any mail to "[hidden email]"
>
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: [diskless] pkg takes 100% of a CPU

BERTRAND Joël
Steven Hartland a écrit :
> When we’ve seen it using 100% it’s been doing comprehension stuff which
> usually finishes you just have to wait. Not sure if that’s what you’re
> seeing?

        Yesterday, I have killed pkg after more than 100 hours of CPU time...

        Best regards,

        JB
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: [diskless] pkg takes 100% of a CPU

Steven Hartland
 What operation where you performing? If you kill with -SEGV you should get
a core dump which can be investigated. Alternatively connect gdb to the
process and get stack traces

On Sat, 7 Apr 2018 at 10:54, BERTRAND Joël <[hidden email]>
wrote:

> Steven Hartland a écrit :
> > When we’ve seen it using 100% it’s been doing comprehension stuff which
> > usually finishes you just have to wait. Not sure if that’s what you’re
> > seeing?
>
>         Yesterday, I have killed pkg after more than 100 hours of CPU
> time...
>
>         Best regards,
>
>         JB
> _______________________________________________
> [hidden email] mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
> To unsubscribe, send any mail to "[hidden email]"
>
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: [diskless] pkg takes 100% of a CPU

Ian Lepore-3
In reply to this post by BERTRAND Joël
On Sat, 2018-04-07 at 11:50 +0200, BERTRAND Jol wrote:

> Steven Hartland a crit:
> >
> > When we’ve seen it using 100% it’s been doing comprehension stuff which
> > usually finishes you just have to wait. Not sure if that’s what you’re
> > seeing?
> Yesterday, I have killed pkg after more than 100 hours of CPU time...
>
> Best regards,
>
> JB

For me, pkg(8) quit working on systems that have /var/db mounted from
nfs long ago, maybe as much as a year ago at this point. I mentioned
it on irc, and was told "It's probably something to do with locking",
but I already have boot.nfsroot.options="nolockd" in loader.conf
(because that's pretty much the only option because the rc(8) system
was broken years ago when it comes to nfsroot).

-- Ian
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: [diskless] pkg takes 100% of a CPU

Rodney W. Grimes-4
> On Sat, 2018-04-07 at 11:50 +0200, BERTRAND Jol wrote:
> > Steven Hartland a crit:
> > >
> > > When we?ve seen it using 100% it?s been doing comprehension stuff which
> > > usually finishes you just have to wait. Not sure if that?s what you?re
> > > seeing?
> > Yesterday, I have killed pkg after more than 100 hours of CPU time...
> >
> > Best regards,
> >
> > JB
>
> For me, pkg(8) quit working on systems that have /var/db mounted from
> nfs long ago, maybe as much as a year ago at this point. I mentioned
> it on irc, and was told "It's probably something to do with locking",
> but I already have boot.nfsroot.options="nolockd" in loader.conf
> (because that's pretty much the only option because the rc(8) system
> was broken years ago when it comes to nfsroot).

My understanding of the current broken state of afairs is that pkg
does not work over nfs unless you have proper and full lockd support,
AND set NFS_WITH_PROPER_LOCKING in pkg's environment.  This really
really really just needs to work without a lot of fuss out of the
box.

The /etc/rc.initdiskless is presenty (11.x and later) in a usable
state, though it has some issues if your /usr is seperate from /,
and you have to override the default size of /var due to bloat,
though I think the sizes of all the tmpfs's got bumped in ^head/.

I had a very nice working iPXE based boot menu system that you
could boot all of {10|11|12-snaps}{i386|amd54}, but it is for
some strange reason having a problem with the NFS load of the
kernel going at an abismal pace, like 10 to 15 minutes to load
the kernel.  Stuff before and after that is normal speed.  I believe
this to be my fault, I changed something some place, and it broke,
and I did not notice for a good long time.  Now I do not know
what it was that I changed that broken it :-(.

The only real special thing in the above setup was a pxeboot that
I got from a very specific point in time that has a nfsroot fix
in it, without that you can not easily override the nfsroot ip:path
from iPXE menu.   /boot/pxeboot  before and after this specific
version is/was broke (since my test system is borked I can not
test ^/heads latest pxeboot).

Regards,
--
Rod Grimes                                                 [hidden email]
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: [diskless] pkg takes 100% of a CPU

Rodney W. Grimes-4
In reply to this post by Ian Lepore-3
[ Charset ISO-2022-JP unsupported, converting... ]

> On Sat, 2018-04-07 at 11:50 +0200, BERTRAND Jol wrote:
> > Steven Hartland a crit:
> > >
> > > When we?ve seen it using 100% it?s been doing comprehension stuff which
> > > usually finishes you just have to wait. Not sure if that?s what you?re
> > > seeing?
> > Yesterday, I have killed pkg after more than 100 hours of CPU time...
> >
> > Best regards,
> >
> > JB
>
> For me, pkg(8) quit working on systems that have /var/db mounted from
> nfs long ago, maybe as much as a year ago at this point. I mentioned
> it on irc, and was told "It's probably something to do with locking",
> but I already have boot.nfsroot.options="nolockd" in loader.conf
> (because that's pretty much the only option because the rc(8) system
> was broken years ago when it comes to nfsroot).

Further information in:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=213611


--
Rod Grimes                                                 [hidden email]
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: [diskless] pkg takes 100% of a CPU

Ian Lepore-3
In reply to this post by Rodney W. Grimes-4
On Sat, 2018-04-07 at 07:35 -0700, Rodney W. Grimes wrote:

> >
> > On Sat, 2018-04-07 at 11:50 +0200, BERTRAND Jol wrote:
> > >
> > > Steven Hartland a crit:
> > > >
> > > >
> > > > When we?ve seen it using 100% it?s been doing comprehension stuff which
> > > > usually finishes you just have to wait. Not sure if that?s what you?re
> > > > seeing?
> > > Yesterday, I have killed pkg after more than 100 hours of CPU time...
> > >
> > > Best regards,
> > >
> > > JB
> > For me, pkg(8) quit working on systems that have /var/db mounted from
> > nfs long ago, maybe as much as a year ago at this point. I mentioned
> > it on irc, and was told "It's probably something to do with locking",
> > but I already have boot.nfsroot.options="nolockd" in loader.conf
> > (because that's pretty much the only option because the rc(8) system
> > was broken years ago when it comes to nfsroot).
> My understanding of the current broken state of afairs is that pkg
> does not work over nfs unless you have proper and full lockd support,
> AND set NFS_WITH_PROPER_LOCKING in pkg's environment.  This really
> really really just needs to work without a lot of fuss out of the
> box.
>

nfs locking support is compiled into the kernel on both the server and
client side, and rpc.lockd is running on both, but there is no need for
actual locking on the mount because the rootfs isn't accessed by
multiple clients at once -- each of my arm boards has its own private
rootfs exported from the server. I'm pretty sure I tried remounting
root

I agree that pkg(8) needs to just work.

> The /etc/rc.initdiskless is presenty (11.x and later) in a usable
> state, though it has some issues if your /usr is seperate from /,
> and you have to override the default size of /var due to bloat,
> though I think the sizes of all the tmpfs's got bumped in ^head/.
>

I don't want an mfs-based /var. I want /var, which is just a dir in the
rootfs, to work the way it's supposed to even if rootfs is nfs-mounted.
Changes to the rc(8) subsystem done long ago, related to managing
pidfiles, cause the bulk of the problems. The problem would be fixable
if we had an mfs-based /run filesystem mounted very early to hold such
things, then later during rc processing a symlink could be made so that
/var/run would point to /run. I'm afraid all of that is easy to say,
but probably has a zillion little nits and "that won't work in my crazy
setup" objections from people, so I've never pursued it seriously.

-- Ian

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: [diskless] pkg takes 100% of a CPU

Rodney W. Grimes-4
> On Sat, 2018-04-07 at 07:35 -0700, Rodney W. Grimes wrote:
> > >
> > > On Sat, 2018-04-07 at 11:50 +0200, BERTRAND Jol wrote:
> > > >
> > > > Steven Hartland a crit:
> > > > >
> > > > >
> > > > > When we?ve seen it using 100% it?s been doing comprehension stuff which
> > > > > usually finishes you just have to wait. Not sure if that?s what you?re
> > > > > seeing?
> > > > Yesterday, I have killed pkg after more than 100 hours of CPU time...
> > > >
> > > > Best regards,
> > > >
> > > > JB
> > > For me, pkg(8) quit working on systems that have /var/db mounted from
> > > nfs long ago, maybe as much as a year ago at this point. I mentioned
> > > it on irc, and was told "It's probably something to do with locking",
> > > but I already have boot.nfsroot.options="nolockd" in loader.conf
> > > (because that's pretty much the only option because the rc(8) system
> > > was broken years ago when it comes to nfsroot).
> > My understanding of the current broken state of afairs is that pkg
> > does not work over nfs unless you have proper and full lockd support,
> > AND set NFS_WITH_PROPER_LOCKING in pkg's environment.??This really
> > really really just needs to work without a lot of fuss out of the
> > box.
> >
>
> nfs locking support is compiled into the kernel on both the server and
> client side, and rpc.lockd is running on both, but there is no need for
> actual locking on the mount because the rootfs isn't accessed by
> multiple clients at once -- each of my arm boards has its own private
> rootfs exported from the server. I'm pretty sure I tried remounting
> root

pkg wants to do locking, multiple copies of pkg may be running on one
client, you might get pkg to work for you if you do the

echo "NFS_WITH_PROPER_LOCKING = true;" >> /usr/local/etc/pkg.conf

I mentioned in my follow up.

>
> I agree that pkg(8) needs to just work.

:-)

> > The /etc/rc.initdiskless is presenty (11.x and later) in a usable
> > state, though it has some issues if your /usr is seperate from /,
> > and you have to override the default size of /var due to bloat,
> > though I think the sizes of all the tmpfs's got bumped in ^head/.
> >
>
> I don't want an mfs-based /var. I want /var, which is just a dir in the
> rootfs, to work the way it's supposed to even if rootfs is nfs-mounted.
> Changes to the rc(8) subsystem done long ago, related to managing
> pidfiles, cause the bulk of the problems. The problem would be fixable
> if we had an mfs-based /run filesystem mounted very early to hold such
> things, then later during rc processing a symlink could be made so that
> /var/run would point to /run. I'm afraid all of that is easy to say,
> but probably has a zillion little nits and "that won't work in my crazy
> setup" objections from people, so I've never pursued it seriously.

I agree we should support a non mfs based /var.  I'll take a crack
at seeing how hard it is to turn that off.

What about making /var/run mfs based when rc.initdiskless runs?
I think you can actually get that done under the current scheme
once I get /var turned off from doing the mfs/var.cpio.gz extract.

Need to see if it processes conf/var/run/*, I do not think so.

--
Rod Grimes                                                 [hidden email]
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: [diskless] pkg takes 100% of a CPU

BERTRAND Joël
Rodney W. Grimes a écrit :

>> On Sat, 2018-04-07 at 07:35 -0700, Rodney W. Grimes wrote:
>>>>
>>>> On Sat, 2018-04-07 at 11:50 +0200, BERTRAND Jol wrote:
>>>>>
>>>>> Steven Hartland a crit:
>>>>>>
>>>>>>
>>>>>> When we?ve seen it using 100% it?s been doing comprehension stuff which
>>>>>> usually finishes you just have to wait. Not sure if that?s what you?re
>>>>>> seeing?
>>>>> Yesterday, I have killed pkg after more than 100 hours of CPU time...
>>>>>
>>>>> Best regards,
>>>>>
>>>>> JB
>>>> For me, pkg(8) quit working on systems that have /var/db mounted from
>>>> nfs long ago, maybe as much as a year ago at this point. I mentioned
>>>> it on irc, and was told "It's probably something to do with locking",
>>>> but I already have boot.nfsroot.options="nolockd" in loader.conf
>>>> (because that's pretty much the only option because the rc(8) system
>>>> was broken years ago when it comes to nfsroot).
>>> My understanding of the current broken state of afairs is that pkg
>>> does not work over nfs unless you have proper and full lockd support,
>>> AND set NFS_WITH_PROPER_LOCKING in pkg's environment.??This really
>>> really really just needs to work without a lot of fuss out of the
>>> box.
>>>
>>
>> nfs locking support is compiled into the kernel on both the server and
>> client side, and rpc.lockd is running on both, but there is no need for
>> actual locking on the mount because the rootfs isn't accessed by
>> multiple clients at once -- each of my arm boards has its own private
>> rootfs exported from the server. I'm pretty sure I tried remounting
>> root
>
> pkg wants to do locking, multiple copies of pkg may be running on one
> client, you might get pkg to work for you if you do the
>
> echo "NFS_WITH_PROPER_LOCKING = true;" >> /usr/local/etc/pkg.conf

        Of course, I have set this line in /usr/local/etc/pkg.conf.

        I cannot attach pkg to gdb :
root@pythagore:~ # gdb -p 75520
GNU gdb 6.1.1 [FreeBSD]
.........
This GDB was configured as "amd64-marcel-freebsd".
Attaching to process 75520
/usr/src/contrib/gdb/gdb/solib-svr4.c:1444: internal-error:
legacy_fetch_link_map_offsets called without legacy link_map support
enabled.
A problem internal to GDB has been detected,
further debugging may prove unreliable.

pkg seems to be called by /usr/local/sbin/pkgcheck-qsa

        I have killed pkg with -SEGV but I'm unable to find core file.

        Best regards,

        JB
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: [diskless] pkg takes 100% of a CPU

freebsd-hackers mailing list
In reply to this post by Ian Lepore-3
On Sat, 07 Apr 2018 08:19:51 -0600
Ian Lepore <[hidden email]> wrote:

> On Sat, 2018-04-07 at 11:50 +0200, BERTRAND Jol wrote:
> > Steven Hartland a crit:
> > >
> > > When we?ve seen it using 100% it?s been doing comprehension stuff
> > > which usually finishes you just have to wait. Not sure if that?s
> > > what you?re seeing?
> > Yesterday, I have killed pkg after more than 100 hours of
> > CPU time...
> >
> > Best regards,
> >
> > JB
>
> For me, pkg(8) quit working on systems that have /var/db mounted from
> nfs long ago, maybe as much as a year ago at this point. I mentioned
> it on irc, and was told "It's probably something to do with locking",
> but I already have boot.nfsroot.options="nolockd" in loader.conf
> (because that's pretty much the only option because the rc(8) system
> was broken years ago when it comes to nfsroot).

Is the db is on netbsd side? If yes, sqlite db over nfs are pita
because nfs lies about file locking. It's true that it works 99% of
time, the problematic part is the 1%. Documentation talks about
incorrect NFS implementation, but a correct implementation can fail too
because network latency, separate memory data between clients and
server or different filesystem semantics; or all of them.

https://www.sqlite.org/faq.html#q5

https://stackoverflow.com/questions/9907429/locking-sqlite-file-on-nfs-filesystem-possible#9962003



>
> -- Ian

---   ---
Eduardo Morras <[hidden email]>
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: [diskless] pkg takes 100% of a CPU

BERTRAND Joël
Eduardo Morras via freebsd-hackers a écrit :

> On Sat, 07 Apr 2018 08:19:51 -0600
> Ian Lepore <[hidden email]> wrote:
>
>> On Sat, 2018-04-07 at 11:50 +0200, BERTRAND Jol wrote:
>>> Steven Hartland a crit:
>>>>
>>>> When we?ve seen it using 100% it?s been doing comprehension stuff
>>>> which usually finishes you just have to wait. Not sure if that?s
>>>> what you?re seeing?
>>> Yesterday, I have killed pkg after more than 100 hours of
>>> CPU time...
>>>
>>> Best regards,
>>>
>>> JB
>>
>> For me, pkg(8) quit working on systems that have /var/db mounted from
>> nfs long ago, maybe as much as a year ago at this point. I mentioned
>> it on irc, and was told "It's probably something to do with locking",
>> but I already have boot.nfsroot.options="nolockd" in loader.conf
>> (because that's pretty much the only option because the rc(8) system
>> was broken years ago when it comes to nfsroot).
>
> Is the db is on netbsd side? If yes, sqlite db over nfs are pita
> because nfs lies about file locking. It's true that it works 99% of
> time, the problematic part is the 1%. Documentation talks about
> incorrect NFS implementation, but a correct implementation can fail too
> because network latency, separate memory data between clients and
> server or different filesystem semantics; or all of them.

        All filesystems are mounted from NetBSD server :

root@pythagore:/var # mount
192.168.10.128:/srv/pythagore on / (nfs, asynchronous)
devfs on /dev (devfs, local, multilabel)
procfs on /proc (procfs, local)
fdescfs on /dev/fd (fdescfs)
192.168.10.128:/home on /home (nfs, asynchronous)
root@pythagore:/var # cat /etc/fstab
# Device        Mountpoint      FStype  Options Dump    Pass#
192.168.10.128:/srv/pythagore / nfs nfsv3,tcp,soft,intr,rw,async,nolockd
   0       0
proc                      /proc procfs rw                      0 0
fdesc                     /dev/fd      fdescfs         rw      0 0
192.168.10.128:/home      /home nfs    nfsv3,tcp,soft,intr,rw,async 0 0

        On server, /var/log/messages are full of :
Apr  8 13:40:49 legendre rpc.lockd: duplicate lock from hilbert.45141
Apr  8 13:40:49 legendre rpc.lockd: no matching entry for hilbert
Apr  8 13:40:52 legendre rpc.lockd: duplicate lock from pythagore.68734
Apr  8 13:40:52 legendre rpc.lockd: duplicate lock from pythagore.68734
Apr  8 13:40:52 legendre rpc.lockd: no matching entry for pythagore
Apr  8 13:40:52 legendre rpc.lockd: no matching entry for pythagore
Apr  8 13:40:55 legendre rpc.lockd: duplicate lock from pythagore.68734

even if all filesystems are mounted with nolockd option.

        JB
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: [diskless] pkg takes 100% of a CPU

Rick Macklem
BERTRAND Joël wrote:
[stuff snipped]
>        All filesystems are mounted from NetBSD server :
>
>root@pythagore:/var # mount
>192.168.10.128:/srv/pythagore on / (nfs, asynchronous)
>devfs on /dev (devfs, local, multilabel)
>procfs on /proc (procfs, local)
>fdescfs on /dev/fd (fdescfs)
>192.168.10.128:/home on /home (nfs, asynchronous)
Just fyi, "nfsstat -m" on the client will show you what NFS mount options
are actually in use.

>root@pythagore:/var # cat /etc/fstab
># Device        Mountpoint      FStype  Options Dump    Pass#
>192.168.10.128:/srv/pythagore / nfs nfsv3,tcp,soft,intr,rw,async,nolockd
I'd suggest you try without the "soft,intr" options. When those are set,
a system call like write(2) can return with EINTR and very few (if any)
applications handle that correctly.

[more stuff snipped]

>        On server, /var/log/messages are full of :
>Apr  8 13:40:49 legendre rpc.lockd: duplicate lock from hilbert.45141
>Apr  8 13:40:49 legendre rpc.lockd: no matching entry for hilbert
>Apr  8 13:40:52 legendre rpc.lockd: duplicate lock from pythagore.68734
>Apr  8 13:40:52 legendre rpc.lockd: duplicate lock from pythagore.68734
>Apr  8 13:40:52 legendre rpc.lockd: no matching entry for pythagore
>Apr  8 13:40:52 legendre rpc.lockd: no matching entry for pythagore
>Apr  8 13:40:55 legendre rpc.lockd: duplicate lock from pythagore.68734
>
>even if all filesystems are mounted with nolockd option.
If you use "nolockd" on all mounts, you do not need to run rpc.lockd or
rpc.statd, which is what I prefer. (The NLM and NSM protocols were undocumented
and fundamentally broken. Eventually they were published in an X/Open XNFS
manual, but it was a vague 2 page summary. The only real doc would be the
OpenSolaris implementation. NFSv4 does a much better job of locking, so if you need the locks to be visible to other clients, that is what I'd suggest.

rick
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"