12.0-RELEASE zfs/vnode deadlock issue

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

12.0-RELEASE zfs/vnode deadlock issue

Courtland
Recently a number of my production 12.0 systems have experienced what I can
only gather is a ZFS deadlock related to vnodes. It seems similar to the
relatively recent FreeBSD-EN-18:18.zfs (ZFS vnode reclaim deadlock)
problem. Previously the same systems were running 11.1-RELEASE without
problem.

Threads are always stuck with the stack around
vn_lock->zfs_root->lookup->namei. When the system is in this state, a
simple `ls /` or `ls /tmp` always hangs, but other datasets seem
unaffected. I have a fairly straightforward ZFS root setup on a single pool
with one SSD. The workload is a ruby/rails/nginx/postgresql backed web
application combined with some data warehousing and other periodic tasks.

Sometimes I can remote SSH in, other times that fails because the user
shell fails to load, and I can run commands via `ssh ... command`.
Sometimes the system is not accessible remotely at all, or it eventually
becomes inaccessible if left long enough in this state. I always have to
physically reboot the device because the shutdown procedure fails. The
network stack (e.g. ping) seems to work completely fine whilst this is
going on, until you try to interact with or spawn a process that hits the
deadlock.

Like previous similar ZFS deadlock issues, increasing kern.vnodes seems to
make the system last longer by up to a few weeks, but is still a bandaid.
However, I have yet to witness vnodes usage actually getting close to the
maximum.

I haven't had any luck reproducing this reliably, but eventually it happens
after a few days or a few weeks... I managed to connect to a system in this
state and grab a procstat and get (hopefully) something useful out of kgdb.
I will note that although I was able to install debug symbols, I couldn't
manage to get the source files onto it for kgdb purposes before I lost SSH
access.

I am hoping someone can help me figure out if this is a legitimate bug, or
something already fixed in 12-STABLE. I wish I could reproduce it reliably
to try against STABLE, but there doesn't appear to be any related ZFS fixes
not in RELEASE. Thanks.

Below is an abbreviated procstat and what I was able to get out of kgdb for
an affected thread. Note that the thread backtrace is from a simple `ls`
command. The procstat dump below is truncated because my last attempt to
send this was rejected by this list for being too long - so a number of
sh/cron processes and some zfs threads in a hung state were removed.

ld# kgdb
GNU gdb (GDB) 8.2.1 [GDB v8.2.1 for FreeBSD]
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html
>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-portbld-freebsd12.0".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /boot/kernel/kernel...Reading symbols from
/usr/lib/debug//boot/kernel/kernel.debug...done.
done.
sched_switch (td=0xfffff8002452a000, newtd=0xfffff80003625580,
flags=<optimized out>)
    at /usr/src/sys/kern/sched_ule.c:2112
2112 /usr/src/sys/kern/sched_ule.c: No such file or directory.
(kgdb) tid 102023
(kgdb) bt
#0  sched_switch (td=0xfffff801a83dc580, newtd=0xfffff80003550580,
flags=<optimized out>)
    at /usr/src/sys/kern/sched_ule.c:2112
#1  0xffffffff80d0e0a1 in mi_switch (flags=<optimized out>, newtd=0x0) at
/usr/src/sys/kern/kern_synch.c:439
#2  0xffffffff80d5c80c in sleepq_wait (wchan=<unavailable>,
pri=<unavailable>)
    at /usr/src/sys/kern/subr_sleepqueue.c:692
#3  0xffffffff80cd9105 in sleeplk (lk=0xfffff800247307e8, flags=<optimized
out>, ilk=<optimized out>,
    wmesg=<optimized out>, pri=<optimized out>, timo=51, queue=1) at
/usr/src/sys/kern/kern_lock.c:300
#4  0xffffffff80cd7f85 in lockmgr_slock_hard (lk=<optimized out>,
flags=<optimized out>, ilk=<optimized out>,
    file=<optimized out>, line=0, lwa=<optimized out>) at
/usr/src/sys/kern/kern_lock.c:646
#5  0xffffffff813acc5e in VOP_LOCK1_APV (vop=<optimized out>,
a=0xfffffe00f89dd450) at vnode_if.c:2087
#6  0xffffffff80de2820 in VOP_LOCK1 (vp=0xfffff80024730780, flags=2105344,
    file=0xffffffff814d4f74
"/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vfsops.c",
line=2074)
    at ./vnode_if.h:859
#7  _vn_lock (vp=0xfffff80024730780, flags=2105344,
    file=0xffffffff814d4f74
"/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vfsops.c",
line=2074)
    at /usr/src/sys/kern/vfs_vnops.c:1533
#8  0xffffffff8049f68d in zfs_root (vfsp=<optimized out>, flags=2105344,
vpp=0xfffffe00f89dd558)
    at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vfsops.c:2074
#9  0xffffffff80dc5d43 in lookup (ndp=0xfffffe00f89dd780) at
/usr/src/sys/kern/vfs_lookup.c:961
#10 0xffffffff80dc4f9b in namei (ndp=0xfffffe00f89dd780) at
/usr/src/sys/kern/vfs_lookup.c:444
#11 0xffffffff80ddc637 in kern_statat (td=0xfffff801a83dc580,
flag=<optimized out>, fd=<optimized out>,
    path=0x800729c78 <error: Cannot access memory at address 0x800729c78>,
pathseg=UIO_USERSPACE,
    sbp=0xfffffe00f89dd898, hook=0x0) at
/usr/src/sys/kern/vfs_syscalls.c:2283
#12 0xffffffff80ddcdcf in sys_fstatat (td=<unavailable>,
uap=0xfffff801a83dc940)
    at /usr/src/sys/kern/vfs_syscalls.c:2260
#13 0xffffffff81222449 in syscallenter (td=<optimized out>)
    at /usr/src/sys/amd64/amd64/../../kern/subr_syscall.c:135
#14 amd64_syscall (td=0xfffff801a83dc580, traced=0) at
/usr/src/sys/amd64/amd64/trap.c:1076
#15 <signal handler called>
#16 0x000000080047941a in ?? ()
Backtrace stopped: Cannot access memory at address 0x7fffffffe178
(kgdb) fr 7
#7  _vn_lock (vp=0xfffff80024730780, flags=2105344,
    file=0xffffffff814d4f74
"/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vfsops.c",
line=2074)
    at /usr/src/sys/kern/vfs_vnops.c:1533
1533 /usr/src/sys/kern/vfs_vnops.c: No such file or directory.
(kgdb) print *vp
$1 = {v_tag = 0xffffffff8144af45 "zfs", v_op = 0xffffffff81c64fd0
<zfs_vnodeops>, v_data = 0xfffff800247e8110,
  v_mount = 0xfffff800247e1000, v_nmntvnodes = {tqe_next =
0xfffff800247305a0, tqe_prev = 0xfffff800247e1060}, {
    v_mountedhere = 0x0, v_unpcb = 0x0, v_rdev = 0x0, v_fifoinfo = 0x0},
v_hashlist = {le_next = 0x0,
    le_prev = 0x0}, v_cache_src = {lh_first = 0xfffff8016f1ff5b0},
v_cache_dst = {tqh_first = 0x0,
    tqh_last = 0xfffff800247307d0}, v_cache_dd = 0x0, v_lock = {lock_object
= {
      lo_name = 0xffffffff8144af45 "zfs", lo_flags = 117112832, lo_data =
0, lo_witness = 0x0},
    lk_lock = 18446735294395164034, lk_exslpfail = 0, lk_timo = 51, lk_pri
= 96}, v_interlock = {lock_object = {
      lo_name = 0xffffffff814e4508 "vnode interlock", lo_flags = 16973824,
lo_data = 0, lo_witness = 0x0},
    mtx_lock = 0}, v_vnlock = 0xfffff800247307e8, v_actfreelist = {tqe_next
= 0x0,
    tqe_prev = 0xfffff80024730660}, v_bufobj = {bo_lock = {lock_object = {
        lo_name = 0xffffffff814a9e5f "bufobj interlock", lo_flags =
86179840, lo_data = 0, lo_witness = 0x0},
      rw_lock = 1}, bo_ops = 0xffffffff81d38600 <buf_ops_bio>, bo_object =
0xfffff80006f5d100, bo_synclist = {
      le_next = 0x0, le_prev = 0x0}, bo_private = 0xfffff80024730780,
bo_clean = {bv_hd = {tqh_first = 0x0,
        tqh_last = 0xfffff80024730898}, bv_root = {pt_root = 0}, bv_cnt =
0}, bo_dirty = {bv_hd = {
        tqh_first = 0x0, tqh_last = 0xfffff800247308b8}, bv_root = {pt_root
= 0}, bv_cnt = 0}, bo_numoutput = 0,
    bo_flag = 0, bo_domain = 1, bo_bsize = 131072}, v_pollinfo = 0x0,
v_label = 0x0, v_lockf = 0x0, v_rl = {
    rl_waiters = {tqh_first = 0x0, tqh_last = 0xfffff80024730908},
rl_currdep = 0x0}, v_cstart = 0, v_lasta = 0,
  v_lastw = 0, v_clen = 0, v_holdcnt = 241, v_usecount = 240, v_iflag =
512, v_vflag = 1, v_mflag = 0,
  v_writecount = 0, v_hash = 4, v_type = VDIR}
(kgdb)


ld# procstat -kka
  PID    TID COMM                TDNAME              KSTACK

    0 100000 kernel              swapper             mi_switch+0xe1
sleepq_timedwait+0x2f _sleep+0x217 swapper+0x6b btext+0x2c
    0 100012 kernel              thread taskq        mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100014 kernel              kqueue_ctx taskq    mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100015 kernel              config_0            mi_switch+0xe1
sleepq_wait+0x2c msleep_spin_sbt+0x177 gtaskqueue_thread_loop+0xc3
fork_exit+0x83 fork_trampoline+0xe
    0 100017 kernel              aiod_kick taskq     mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100018 kernel              if_io_tqg_0         mi_switch+0xe1
sleepq_wait+0x2c msleep_spin_sbt+0x177 gtaskqueue_thread_loop+0xc3
fork_exit+0x83 fork_trampoline+0xe
    0 100019 kernel              if_io_tqg_1         mi_switch+0xe1
sleepq_wait+0x2c msleep_spin_sbt+0x177 gtaskqueue_thread_loop+0xc3
fork_exit+0x83 fork_trampoline+0xe
    0 100020 kernel              if_io_tqg_2         mi_switch+0xe1
sleepq_wait+0x2c msleep_spin_sbt+0x177 gtaskqueue_thread_loop+0xc3
fork_exit+0x83 fork_trampoline+0xe
    0 100021 kernel              if_io_tqg_3         mi_switch+0xe1
sleepq_wait+0x2c msleep_spin_sbt+0x177 gtaskqueue_thread_loop+0xc3
fork_exit+0x83 fork_trampoline+0xe
    0 100022 kernel              if_io_tqg_4         mi_switch+0xe1
sleepq_wait+0x2c msleep_spin_sbt+0x177 gtaskqueue_thread_loop+0xc3
fork_exit+0x83 fork_trampoline+0xe
    0 100023 kernel              if_io_tqg_5         mi_switch+0xe1
sleepq_wait+0x2c msleep_spin_sbt+0x177 gtaskqueue_thread_loop+0xc3
fork_exit+0x83 fork_trampoline+0xe
    0 100024 kernel              if_io_tqg_6         mi_switch+0xe1
sleepq_wait+0x2c msleep_spin_sbt+0x177 gtaskqueue_thread_loop+0xc3
fork_exit+0x83 fork_trampoline+0xe
    0 100025 kernel              if_io_tqg_7         mi_switch+0xe1
sleepq_wait+0x2c msleep_spin_sbt+0x177 gtaskqueue_thread_loop+0xc3
fork_exit+0x83 fork_trampoline+0xe
    0 100026 kernel              softirq_0           mi_switch+0xe1
sleepq_wait+0x2c msleep_spin_sbt+0x177 gtaskqueue_thread_loop+0xc3
fork_exit+0x83 fork_trampoline+0xe
    0 100027 kernel              softirq_1           mi_switch+0xe1
sleepq_wait+0x2c msleep_spin_sbt+0x177 gtaskqueue_thread_loop+0xc3
fork_exit+0x83 fork_trampoline+0xe
    0 100028 kernel              softirq_2           mi_switch+0xe1
sleepq_wait+0x2c msleep_spin_sbt+0x177 gtaskqueue_thread_loop+0xc3
fork_exit+0x83 fork_trampoline+0xe
    0 100029 kernel              softirq_3           mi_switch+0xe1
sleepq_wait+0x2c msleep_spin_sbt+0x177 gtaskqueue_thread_loop+0xc3
fork_exit+0x83 fork_trampoline+0xe
    0 100030 kernel              softirq_4           mi_switch+0xe1
sleepq_wait+0x2c msleep_spin_sbt+0x177 gtaskqueue_thread_loop+0xc3
fork_exit+0x83 fork_trampoline+0xe
    0 100031 kernel              softirq_5           mi_switch+0xe1
sleepq_wait+0x2c msleep_spin_sbt+0x177 gtaskqueue_thread_loop+0xc3
fork_exit+0x83 fork_trampoline+0xe
    0 100032 kernel              softirq_6           mi_switch+0xe1
sleepq_wait+0x2c msleep_spin_sbt+0x177 gtaskqueue_thread_loop+0xc3
fork_exit+0x83 fork_trampoline+0xe
    0 100033 kernel              softirq_7           mi_switch+0xe1
sleepq_wait+0x2c msleep_spin_sbt+0x177 gtaskqueue_thread_loop+0xc3
fork_exit+0x83 fork_trampoline+0xe
    0 100034 kernel              if_config_tqg_0     mi_switch+0xe1
sleepq_wait+0x2c msleep_spin_sbt+0x177 gtaskqueue_thread_loop+0xc3
fork_exit+0x83 fork_trampoline+0xe
    0 100056 kernel              firmware taskq      mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100057 kernel              crypto_0            mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100058 kernel              crypto_1            mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100059 kernel              crypto_2            mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100060 kernel              crypto_3            mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100061 kernel              crypto_4            mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100062 kernel              crypto_5            mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100063 kernel              crypto_6            mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100064 kernel              crypto_7            mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100096 kernel              system_taskq_0      mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100097 kernel              system_taskq_1      mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100098 kernel              system_taskq_2      mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100099 kernel              system_taskq_3      mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100100 kernel              system_taskq_4      mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100101 kernel              system_taskq_5      mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100102 kernel              system_taskq_6      mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100103 kernel              system_taskq_7      mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100104 kernel              mca taskq           mi_switch+0xe1
sleepq_wait+0x2c msleep_spin_sbt+0x177 taskqueue_thread_loop+0xc3
fork_exit+0x83 fork_trampoline+0xe
    0 100109 kernel              arc_prune_0         mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100110 kernel              arc_prune_1         mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100111 kernel              arc_prune_2         mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100112 kernel              arc_prune_3         mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100113 kernel              arc_prune_4         mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100114 kernel              arc_prune_5         mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100115 kernel              arc_prune_6         mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100116 kernel              arc_prune_7         mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100119 kernel              dbu_evict           mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100121 kernel              z_vdev_file_0       mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100122 kernel              z_vdev_file_1       mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100123 kernel              z_vdev_file_2       mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100124 kernel              z_vdev_file_3       mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100125 kernel              z_vdev_file_4       mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100126 kernel              z_vdev_file_5       mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100127 kernel              z_vdev_file_6       mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100128 kernel              z_vdev_file_7       mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100129 kernel              z_vdev_file_8       mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100130 kernel              z_vdev_file_9       mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100131 kernel              z_vdev_file_10      mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100132 kernel              z_vdev_file_11      mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100133 kernel              z_vdev_file_12      mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100134 kernel              z_vdev_file_13      mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100135 kernel              z_vdev_file_14      mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100136 kernel              z_vdev_file_15      mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100138 kernel              zfsvfs              mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100143 kernel              acpi_task_0         mi_switch+0xe1
sleepq_wait+0x2c msleep_spin_sbt+0x177 taskqueue_thread_loop+0xc3
fork_exit+0x83 fork_trampoline+0xe
    0 100144 kernel              acpi_task_1         mi_switch+0xe1
sleepq_wait+0x2c msleep_spin_sbt+0x177 taskqueue_thread_loop+0xc3
fork_exit+0x83 fork_trampoline+0xe
    0 100145 kernel              acpi_task_2         mi_switch+0xe1
sleepq_wait+0x2c msleep_spin_sbt+0x177 taskqueue_thread_loop+0xc3
fork_exit+0x83 fork_trampoline+0xe
    0 100146 kernel              CAM taskq           mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100167 kernel              zio_null_issue      mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100168 kernel              zio_null_intr       mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100169 kernel              zio_read_issue_0    mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100170 kernel              zio_read_issue_1    mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100171 kernel              zio_read_issue_2    mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100172 kernel              zio_read_issue_3    mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100173 kernel              zio_read_issue_4    mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100174 kernel              zio_read_issue_5    mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100175 kernel              zio_read_issue_6    mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100176 kernel              zio_read_issue_7    mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100177 kernel              zio_read_intr_0_0   mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100178 kernel              zio_read_intr_0_1   mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100179 kernel              zio_read_intr_0_2   mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100180 kernel              zio_read_intr_0_3   mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100181 kernel              zio_read_intr_0_4   mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100182 kernel              zio_read_intr_0_5   mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100183 kernel              zio_read_intr_0_6   mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100184 kernel              zio_read_intr_0_7   mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100185 kernel              zio_read_intr_0_8   mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100186 kernel              zio_read_intr_0_9   mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100187 kernel              zio_read_intr_0_10  mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100188 kernel              zio_read_intr_0_11  mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100189 kernel              zio_read_intr_1_0   mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100393 kernel              zio_free_intr       mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100394 kernel              zio_claim_issue     mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100395 kernel              zio_claim_intr      mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100396 kernel              zio_ioctl_issue     mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100397 kernel              zio_ioctl_intr      mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100403 kernel              dp_sync_taskq_0     mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100404 kernel              dp_sync_taskq_1     mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100405 kernel              dp_sync_taskq_2     mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100406 kernel              dp_sync_taskq_3     mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100407 kernel              dp_sync_taskq_4     mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100408 kernel              dp_sync_taskq_5     mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100409 kernel              dp_zil_clean_taskq_ mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100410 kernel              dp_zil_clean_taskq_ mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100411 kernel              dp_zil_clean_taskq_ mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100412 kernel              dp_zil_clean_taskq_ mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100413 kernel              dp_zil_clean_taskq_ mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100414 kernel              dp_zil_clean_taskq_ mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100415 kernel              dp_zil_clean_taskq_ mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100416 kernel              dp_zil_clean_taskq_ mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100417 kernel              zfs_vn_rele_taskq   mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100418 kernel              metaslab_group_task mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100419 kernel              metaslab_group_task mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100420 kernel              metaslab_group_task mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100421 kernel              metaslab_group_task mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    1 100002 init                -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
kern_wait6+0x66d sys_wait4+0x78 amd64_syscall+0x369
fast_syscall_common+0x101
    2 100065 crypto              -                   mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 crypto_proc+0x2e5 fork_exit+0x83
fork_trampoline+0xe
    3 100066 crypto returns 0    -                   mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 crypto_ret_proc+0x1cd fork_exit+0x83
fork_trampoline+0xe
    4 100067 crypto returns 1    -                   mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 crypto_ret_proc+0x1cd fork_exit+0x83
fork_trampoline+0xe
    5 100068 crypto returns 2    -                   mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 crypto_ret_proc+0x1cd fork_exit+0x83
fork_trampoline+0xe
    6 100069 crypto returns 3    -                   mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 crypto_ret_proc+0x1cd fork_exit+0x83
fork_trampoline+0xe
    7 100070 crypto returns 4    -                   mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 crypto_ret_proc+0x1cd fork_exit+0x83
fork_trampoline+0xe
    8 100071 crypto returns 5    -                   mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 crypto_ret_proc+0x1cd fork_exit+0x83
fork_trampoline+0xe
    9 100072 crypto returns 6    -                   mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 crypto_ret_proc+0x1cd fork_exit+0x83
fork_trampoline+0xe
   10 100001 audit               -                   mi_switch+0xe1
sleepq_wait+0x2c _cv_wait+0x152 audit_worker+0x73 fork_exit+0x83
fork_trampoline+0xe
   11 100003 idle                idle: cpu0          acpi_cpu_idle+0x2e7
cpu_idle_acpi+0x3f cpu_idle+0xa7 sched_idletd+0x515 fork_exit+0x83
fork_trampoline+0xe
   11 100004 idle                idle: cpu1          mi_switch+0xe1
critical_exit_preempt+0x58 sched_idletd+0x515 fork_exit+0x83
fork_trampoline+0xe
   11 100005 idle                idle: cpu2          acpi_cpu_idle+0x2e7
cpu_idle_acpi+0x3f cpu_idle+0xa7 sched_idletd+0x515 fork_exit+0x83
fork_trampoline+0xe
   11 100006 idle                idle: cpu3          mi_switch+0xe1
critical_exit_preempt+0x58 sched_idletd+0x515 fork_exit+0x83
fork_trampoline+0xe
   11 100007 idle                idle: cpu4          acpi_cpu_idle+0x2e7
cpu_idle_acpi+0x3f cpu_idle+0xa7 sched_idletd+0x515 fork_exit+0x83
fork_trampoline+0xe
   11 100008 idle                idle: cpu5          mi_switch+0xe1
critical_exit_preempt+0x58 sched_idletd+0x515 fork_exit+0x83
fork_trampoline+0xe
   11 100009 idle                idle: cpu6          acpi_cpu_idle+0x2e7
cpu_idle_acpi+0x3f cpu_idle+0xa7 sched_idletd+0x515 fork_exit+0x83
fork_trampoline+0xe
   11 100010 idle                idle: cpu7          acpi_cpu_idle+0x2e7
cpu_idle_acpi+0x3f cpu_idle+0xa7 sched_idletd+0x515 fork_exit+0x83
fork_trampoline+0xe
   12 100011 intr                swi6: Giant taskq   mi_switch+0xe1
ithread_loop+0x393 fork_exit+0x83 fork_trampoline+0xe
   12 100013 intr                swi5: fast taskq

   12 100016 intr                swi6: task queue    mi_switch+0xe1
ithread_loop+0x393 fork_exit+0x83 fork_trampoline+0xe
   12 100035 intr                swi4: clock (0)     mi_switch+0xe1
ithread_loop+0x393 fork_exit+0x83 fork_trampoline+0xe
   12 100036 intr                swi4: clock (1)

   12 100037 intr                swi4: clock (2)

   12 100038 intr                swi4: clock (3)

   12 100039 intr                swi4: clock (4)

   12 100040 intr                swi4: clock (5)

   12 100041 intr                swi4: clock (6)

   12 100042 intr                swi4: clock (7)

   12 100043 intr                swi1: netisr 0      mi_switch+0xe1
ithread_loop+0x393 fork_exit+0x83 fork_trampoline+0xe
   12 100044 intr                swi3: vm

   12 100077 intr                irq274: xhci0       mi_switch+0xe1
ithread_loop+0x393 fork_exit+0x83 fork_trampoline+0xe
   12 100083 intr                irq18: ehci0 ehci1  mi_switch+0xe1
ithread_loop+0x393 fork_exit+0x83 fork_trampoline+0xe
   12 100094 intr                irq305: ahci0       mi_switch+0xe1
ithread_loop+0x393 fork_exit+0x83 fork_trampoline+0xe
   12 100095 intr                swi0: uart uart     mi_switch+0xe1
ithread_loop+0x393 fork_exit+0x83 fork_trampoline+0xe
   12 100141 intr                swi1: pf send       mi_switch+0xe1
ithread_loop+0x393 fork_exit+0x83 fork_trampoline+0xe
   13 100045 ng_queue            ng_queue0           mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 ngthread+0x9c fork_exit+0x83
fork_trampoline+0xe
   13 100046 ng_queue            ng_queue1           mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 ngthread+0x9c fork_exit+0x83
fork_trampoline+0xe
   13 100047 ng_queue            ng_queue2           mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 ngthread+0x9c fork_exit+0x83
fork_trampoline+0xe
   13 100048 ng_queue            ng_queue3           mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 ngthread+0x9c fork_exit+0x83
fork_trampoline+0xe
   13 100049 ng_queue            ng_queue4           mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 ngthread+0x9c fork_exit+0x83
fork_trampoline+0xe
   13 100050 ng_queue            ng_queue5           mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 ngthread+0x9c fork_exit+0x83
fork_trampoline+0xe
   13 100051 ng_queue            ng_queue6           mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 ngthread+0x9c fork_exit+0x83
fork_trampoline+0xe
   13 100052 ng_queue            ng_queue7           mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 ngthread+0x9c fork_exit+0x83
fork_trampoline+0xe
   14 100053 geom                g_event             mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 g_run_events+0x4f fork_exit+0x83
fork_trampoline+0xe
   14 100054 geom                g_up                mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 g_io_schedule_up+0xcc g_up_procbody+0x6d
fork_exit+0x83 fork_trampoline+0xe
   14 100055 geom                g_down              mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 g_io_schedule_down+0xcc g_down_procbody+0x6d
fork_exit+0x83 fork_trampoline+0xe
   15 100073 crypto returns 7    -                   mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 crypto_ret_proc+0x1cd fork_exit+0x83
fork_trampoline+0xe
   16 100074 sequencer 00        -                   mi_switch+0xe1
sleepq_wait+0x2c _cv_wait+0x152 seq_eventthread+0xdc fork_exit+0x83
fork_trampoline+0xe
   17 100075 cam                 doneq0              mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 xpt_done_td+0x9b fork_exit+0x83
fork_trampoline+0xe
   17 100076 cam                 doneq1              mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 xpt_done_td+0x9b fork_exit+0x83
fork_trampoline+0xe
   17 100147 cam                 scanner             mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 xpt_scanner_thread+0x99 fork_exit+0x83
fork_trampoline+0xe
   18 100078 usb                 usbus0              mi_switch+0xe1
sleepq_wait+0x2c _cv_wait+0x152 usb_process+0x137 fork_exit+0x83
fork_trampoline+0xe
   18 100079 usb                 usbus0              mi_switch+0xe1
sleepq_wait+0x2c _cv_wait+0x152 usb_process+0x137 fork_exit+0x83
fork_trampoline+0xe
   18 100080 usb                 usbus0              mi_switch+0xe1
sleepq_wait+0x2c _cv_wait+0x152 usb_process+0x137 fork_exit+0x83
fork_trampoline+0xe
   18 100081 usb                 usbus0              mi_switch+0xe1
sleepq_wait+0x2c _cv_wait+0x152 usb_process+0x137 fork_exit+0x83
fork_trampoline+0xe
   18 100082 usb                 usbus0              mi_switch+0xe1
sleepq_wait+0x2c _cv_wait+0x152 usb_process+0x137 fork_exit+0x83
fork_trampoline+0xe
   18 100084 usb                 usbus1              mi_switch+0xe1
sleepq_wait+0x2c _cv_wait+0x152 usb_process+0x137 fork_exit+0x83
fork_trampoline+0xe
   18 100085 usb                 usbus1              mi_switch+0xe1
sleepq_wait+0x2c _cv_wait+0x152 usb_process+0x137 fork_exit+0x83
fork_trampoline+0xe
   18 100086 usb                 usbus1              mi_switch+0xe1
sleepq_wait+0x2c _cv_wait+0x152 usb_process+0x137 fork_exit+0x83
fork_trampoline+0xe
   18 100087 usb                 usbus1              mi_switch+0xe1
sleepq_wait+0x2c _cv_wait+0x152 usb_process+0x137 fork_exit+0x83
fork_trampoline+0xe
   18 100088 usb                 usbus1              mi_switch+0xe1
sleepq_wait+0x2c _cv_wait+0x152 usb_process+0x137 fork_exit+0x83
fork_trampoline+0xe
   18 100089 usb                 usbus2              mi_switch+0xe1
sleepq_wait+0x2c _cv_wait+0x152 usb_process+0x137 fork_exit+0x83
fork_trampoline+0xe
   18 100090 usb                 usbus2              mi_switch+0xe1
sleepq_wait+0x2c _cv_wait+0x152 usb_process+0x137 fork_exit+0x83
fork_trampoline+0xe
   18 100091 usb                 usbus2              mi_switch+0xe1
sleepq_wait+0x2c _cv_wait+0x152 usb_process+0x137 fork_exit+0x83
fork_trampoline+0xe
   18 100092 usb                 usbus2              mi_switch+0xe1
sleepq_wait+0x2c _cv_wait+0x152 usb_process+0x137 fork_exit+0x83
fork_trampoline+0xe
   18 100093 usb                 usbus2              mi_switch+0xe1
sleepq_wait+0x2c _cv_wait+0x152 usb_process+0x137 fork_exit+0x83
fork_trampoline+0xe
   19 100105 soaiod1             -                   mi_switch+0xe1
sleepq_timedwait+0x2f _sleep+0x217 soaio_kproc_loop+0x1bf fork_exit+0x83
fork_trampoline+0xe
   20 100106 soaiod2             -                   mi_switch+0xe1
sleepq_timedwait+0x2f _sleep+0x217 soaio_kproc_loop+0x1bf fork_exit+0x83
fork_trampoline+0xe
   21 100107 soaiod3             -                   mi_switch+0xe1
sleepq_timedwait+0x2f _sleep+0x217 soaio_kproc_loop+0x1bf fork_exit+0x83
fork_trampoline+0xe
   22 100108 soaiod4             -                   mi_switch+0xe1
sleepq_timedwait+0x2f _sleep+0x217 soaio_kproc_loop+0x1bf fork_exit+0x83
fork_trampoline+0xe
   23 100117 zfskern             arc_reclaim_thread  mi_switch+0xe1
sleepq_timedwait+0x2f _cv_timedwait_sbt+0x17a arc_reclaim_thread+0x146
fork_exit+0x83 fork_trampoline+0xe
   23 100118 zfskern             arc_dnlc_evicts_thr mi_switch+0xe1
sleepq_wait+0x2c _cv_wait+0x152 arc_dnlc_evicts_thread+0x16f fork_exit+0x83
fork_trampoline+0xe
   23 100120 zfskern             dbuf_evict_thread   mi_switch+0xe1
sleepq_timedwait+0x2f _cv_timedwait_sbt+0x17a dbuf_evict_thread+0x1c8
fork_exit+0x83 fork_trampoline+0xe
   23 100137 zfskern             l2arc_feed_thread   mi_switch+0xe1
sleepq_timedwait+0x2f _cv_timedwait_sbt+0x17a l2arc_feed_thread+0x219
fork_exit+0x83 fork_trampoline+0xe
   23 100398 zfskern             trim zroot          mi_switch+0xe1
sleepq_timedwait+0x2f _cv_timedwait_sbt+0x17a trim_thread+0x11f
fork_exit+0x83 fork_trampoline+0xe
   23 100434 zfskern             txg_thread_enter    mi_switch+0xe1
sleepq_wait+0x2c _cv_wait+0x152 txg_quiesce+0x21b txg_quiesce_thread+0x11b
fork_exit+0x83 fork_trampoline+0xe
   23 100435 zfskern             txg_thread_enter    mi_switch+0xe1
sleepq_wait+0x2c _cv_wait+0x152 txg_sync_thread+0x13b fork_exit+0x83
fork_trampoline+0xe
   23 100436 zfskern             solthread 0xfffffff mi_switch+0xe1
sleepq_wait+0x2c _cv_wait+0x152 zthr_procedure+0xcc fork_exit+0x83
fork_trampoline+0xe
   23 100437 zfskern             solthread 0xfffffff mi_switch+0xe1
sleepq_wait+0x2c _cv_wait+0x152 zthr_procedure+0xcc fork_exit+0x83
fork_trampoline+0xe
   24 100139 sctp_iterator       -                   mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 sctp_iterator_thread+0x59 fork_exit+0x83
fork_trampoline+0xe
   25 100140 pf purge            -                   mi_switch+0xe1
sleepq_timedwait+0x2f _sleep+0x217 pf_purge_thread+0xb9 fork_exit+0x83
fork_trampoline+0xe
   26 100142 rand_harvestq       -                   mi_switch+0xe1
sleepq_timedwait+0x2f _sleep+0x217 random_kthread+0x286 fork_exit+0x83
fork_trampoline+0xe
   27 100148 enc_daemon0         -                   mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 enc_daemon+0x349 fork_exit+0x83
fork_trampoline+0xe
   28 100149 pagedaemon          dom0                mi_switch+0xe1
sleepq_timedwait+0x2f _sleep+0x217 vm_pageout_worker+0x28e vm_pageout+0x15a
fork_exit+0x83 fork_trampoline+0xe
   28 100151 pagedaemon          laundry: dom0       mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 vm_pageout_laundry_worker+0xfdd
fork_exit+0x83 fork_trampoline+0xe
   28 100153 pagedaemon          uma                 mi_switch+0xe1
sleepq_timedwait+0x2f _sleep+0x217 uma_reclaim_worker+0xca fork_exit+0x83
fork_trampoline+0xe
   29 100150 vmdaemon            -                   mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 vm_daemon+0x94 fork_exit+0x83
fork_trampoline+0xe
   30 100152 bufdaemon           -                   mi_switch+0xe1
sleepq_timedwait+0x2f _sleep+0x217 buf_daemon+0xe8 fork_exit+0x83
fork_trampoline+0xe
   30 100156 bufdaemon           bufspacedaemon-0    mi_switch+0xe1
sleepq_timedwait+0x2f _sleep+0x217 bufspace_daemon+0xa4 fork_exit+0x83
fork_trampoline+0xe
   30 100157 bufdaemon           bufspacedaemon-1    mi_switch+0xe1
sleepq_timedwait+0x2f _sleep+0x217 bufspace_daemon+0xa4 fork_exit+0x83
fork_trampoline+0xe
   30 100158 bufdaemon           bufspacedaemon-2    mi_switch+0xe1
sleepq_timedwait+0x2f _sleep+0x217 bufspace_daemon+0xa4 fork_exit+0x83
fork_trampoline+0xe
   30 100159 bufdaemon           bufspacedaemon-3    mi_switch+0xe1
sleepq_timedwait+0x2f _sleep+0x217 bufspace_daemon+0xa4 fork_exit+0x83
fork_trampoline+0xe
   30 100160 bufdaemon           bufspacedaemon-4    mi_switch+0xe1
sleepq_timedwait+0x2f _sleep+0x217 bufspace_daemon+0xa4 fork_exit+0x83
fork_trampoline+0xe
   30 100161 bufdaemon           bufspacedaemon-5    mi_switch+0xe1
sleepq_timedwait+0x2f _sleep+0x217 bufspace_daemon+0xa4 fork_exit+0x83
fork_trampoline+0xe
   30 100162 bufdaemon           bufspacedaemon-6    mi_switch+0xe1
sleepq_timedwait+0x2f _sleep+0x217 bufspace_daemon+0xa4 fork_exit+0x83
fork_trampoline+0xe
   31 100154 vnlru               -                   mi_switch+0xe1
sleepq_timedwait+0x2f _sleep+0x217 vnlru_proc+0x95 fork_exit+0x83
fork_trampoline+0xe
   32 100155 syncer              -                   mi_switch+0xe1
sleepq_timedwait+0x2f _cv_timedwait_sbt+0x17a sched_sync+0xc4
fork_exit+0x83 fork_trampoline+0xe
 2925 100473 devd                -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14
_cv_timedwait_sig_sbt+0x17a seltdwait+0x7b kern_select+0x87f
sys_select+0x56 amd64_syscall+0x369 fast_syscall_common+0x101
 3014 100487 syslogd             -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_select+0x87f sys_select+0x56 amd64_syscall+0x369
fast_syscall_common+0x101
 3121 100443 automountd          -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
autofs_ioctl+0xbd devfs_ioctl+0xad VOP_IOCTL_APV+0x7e vn_ioctl+0x1a4
devfs_ioctl_f+0x1f kern_ioctl+0x26d sys_ioctl+0x15e amd64_syscall+0x369
fast_syscall_common+0x101
 3127 100461 autounmountd        -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
kqueue_kevent+0x29e kern_kevent+0xb5 kern_kevent_generic+0x71
sys_kevent+0x61 amd64_syscall+0x369 fast_syscall_common+0x101
 3246 100488 sshd                -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_select+0x87f sys_select+0x56 amd64_syscall+0x369
fast_syscall_common+0x101
 7053 100664 smbd                -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14
_cv_timedwait_sig_sbt+0x17a seltdwait+0x7b kern_poll+0x43d sys_poll+0x50
amd64_syscall+0x369 fast_syscall_common+0x101
 7062 100616 smbd                -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14
_cv_timedwait_sig_sbt+0x17a seltdwait+0x7b kern_poll+0x43d sys_poll+0x50
amd64_syscall+0x369 fast_syscall_common+0x101
 7063 100670 smbd                -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14
_cv_timedwait_sig_sbt+0x17a seltdwait+0x7b kern_poll+0x43d sys_poll+0x50
amd64_syscall+0x369 fast_syscall_common+0x101
 7064 100455 smbd                -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14
_cv_timedwait_sig_sbt+0x17a seltdwait+0x7b kern_poll+0x43d sys_poll+0x50
amd64_syscall+0x369 fast_syscall_common+0x101
 7065 100519 snmptrapd           -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_select+0x87f sys_select+0x56 amd64_syscall+0x369
fast_syscall_common+0x101
10568 100483 nginx               -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
kern_sigsuspend+0x174 sys_sigsuspend+0x31 amd64_syscall+0x369
fast_syscall_common+0x101
10569 100803 nginx               -                   mi_switch+0xe1
sleepq_wait+0x2c sleeplk+0x1c5 lockmgr_slock_hard+0x2c5 VOP_LOCK1_APV+0x7e
_vn_lock+0x40 zfs_root+0x6d lookup+0x933 namei+0x44b unp_connectat+0x174
soconnect+0xad kern_connectat+0x106 sys_connect+0x77 amd64_syscall+0x369
fast_syscall_common+0x101
10570 100448 nginx               -                   mi_switch+0xe1
sleepq_wait+0x2c sleeplk+0x1c5 lockmgr_slock_hard+0x2c5 VOP_LOCK1_APV+0x7e
_vn_lock+0x40 zfs_root+0x6d lookup+0x933 namei+0x44b unp_connectat+0x174
soconnect+0xad kern_connectat+0x106 sys_connect+0x77 amd64_syscall+0x369
fast_syscall_common+0x101
10571 100527 nginx               -                   mi_switch+0xe1
sleepq_wait+0x2c sleeplk+0x1c5 lockmgr_slock_hard+0x2c5 VOP_LOCK1_APV+0x7e
_vn_lock+0x40 zfs_root+0x6d lookup+0x933 namei+0x44b unp_connectat+0x174
soconnect+0xad kern_connectat+0x106 sys_connect+0x77 amd64_syscall+0x369
fast_syscall_common+0x101
10572 100552 nginx               -                   mi_switch+0xe1
sleepq_wait+0x2c sleeplk+0x1c5 lockmgr_slock_hard+0x2c5 VOP_LOCK1_APV+0x7e
_vn_lock+0x40 zfs_root+0x6d lookup+0x933 namei+0x44b unp_connectat+0x174
soconnect+0xad kern_connectat+0x106 sys_connect+0x77 amd64_syscall+0x369
fast_syscall_common+0x101
10573 100617 nginx               -                   mi_switch+0xe1
sleepq_wait+0x2c sleeplk+0x1c5 lockmgr_slock_hard+0x2c5 VOP_LOCK1_APV+0x7e
_vn_lock+0x40 zfs_root+0x6d lookup+0x933 namei+0x44b unp_connectat+0x174
soconnect+0xad kern_connectat+0x106 sys_connect+0x77 amd64_syscall+0x369
fast_syscall_common+0x101
10574 101643 nginx               -                   mi_switch+0xe1
sleepq_wait+0x2c sleeplk+0x1c5 lockmgr_slock_hard+0x2c5 VOP_LOCK1_APV+0x7e
_vn_lock+0x40 zfs_root+0x6d lookup+0x933 namei+0x44b unp_connectat+0x174
soconnect+0xad kern_connectat+0x106 sys_connect+0x77 amd64_syscall+0x369
fast_syscall_common+0x101
10575 100610 nginx               -                   mi_switch+0xe1
sleepq_wait+0x2c sleeplk+0x1c5 lockmgr_slock_hard+0x2c5 VOP_LOCK1_APV+0x7e
_vn_lock+0x40 zfs_root+0x6d lookup+0x933 namei+0x44b unp_connectat+0x174
soconnect+0xad kern_connectat+0x106 sys_connect+0x77 amd64_syscall+0x369
fast_syscall_common+0x101
10576 100653 nginx               -                   mi_switch+0xe1
sleepq_wait+0x2c sleeplk+0x1c5 lockmgr_slock_hard+0x2c5 VOP_LOCK1_APV+0x7e
_vn_lock+0x40 zfs_root+0x6d lookup+0x933 namei+0x44b unp_connectat+0x174
soconnect+0xad kern_connectat+0x106 sys_connect+0x77 amd64_syscall+0x369
fast_syscall_common+0x101
20584 100460 openvpn             -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14
_cv_timedwait_sig_sbt+0x17a seltdwait+0x7b kern_poll+0x43d sys_poll+0x50
amd64_syscall+0x369 fast_syscall_common+0x101
21660 100615 postgres            -                   mi_switch+0xe1
sleepq_wait+0x2c sleeplk+0x1c5 lockmgr_slock_hard+0x2c5 VOP_LOCK1_APV+0x7e
_vn_lock+0x40 zfs_root+0x6d lookup+0x933 namei+0x44b kern_utimesat+0x110
amd64_syscall+0x369 fast_syscall_common+0x101
21662 100612 postgres            -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14
_cv_timedwait_sig_sbt+0x17a seltdwait+0x7b kern_poll+0x43d sys_poll+0x50
amd64_syscall+0x369 fast_syscall_common+0x101
21663 100626 postgres            -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14
_cv_timedwait_sig_sbt+0x17a seltdwait+0x7b kern_poll+0x43d sys_poll+0x50
amd64_syscall+0x369 fast_syscall_common+0x101
21664 100450 postgres            -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14
_cv_timedwait_sig_sbt+0x17a seltdwait+0x7b kern_poll+0x43d sys_poll+0x50
amd64_syscall+0x369 fast_syscall_common+0x101
21665 100638 postgres            -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14
_cv_timedwait_sig_sbt+0x17a seltdwait+0x7b kern_poll+0x43d sys_poll+0x50
amd64_syscall+0x369 fast_syscall_common+0x101
21666 100635 postgres            -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369
fast_syscall_common+0x101
21667 100700 postgres            -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14
_cv_timedwait_sig_sbt+0x17a seltdwait+0x7b kern_poll+0x43d sys_poll+0x50
amd64_syscall+0x369 fast_syscall_common+0x101
22183 100518 getty               -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
tty_wait+0x1c ttydisc_read+0x1f2 ttydev_read+0x64 devfs_read_f+0xdc
dofileread+0x95 sys_read+0xc3 amd64_syscall+0x369 fast_syscall_common+0x101
22184 100509 getty               -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
tty_wait+0x1c ttydisc_read+0x1f2 ttydev_read+0x64 devfs_read_f+0xdc
dofileread+0x95 sys_read+0xc3 amd64_syscall+0x369 fast_syscall_common+0x101
22185 101647 getty               -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
tty_wait+0x1c ttydisc_read+0x1f2 ttydev_read+0x64 devfs_read_f+0xdc
dofileread+0x95 sys_read+0xc3 amd64_syscall+0x369 fast_syscall_common+0x101
22435 100440 getty               -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
tty_wait+0x1c ttydisc_read+0x1f2 ttydev_read+0x64 devfs_read_f+0xdc
dofileread+0x95 sys_read+0xc3 amd64_syscall+0x369 fast_syscall_common+0x101
24825 101365 postgres            -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369
fast_syscall_common+0x101
24851 100599 perl                -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205
kern_clock_nanosleep+0x1a7 sys_nanosleep+0x5f amd64_syscall+0x369
fast_syscall_common+0x101
24855 101640 ntpd                -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_select+0x87f sys_select+0x56 amd64_syscall+0x369
fast_syscall_common+0x101
24865 100459 radiusd             -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
kqueue_kevent+0x29e kern_kevent+0xb5 kern_kevent_generic+0x71
sys_kevent+0x61 amd64_syscall+0x369 fast_syscall_common+0x101
24865 101450 radiusd             -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
umtxq_sleep+0x143 do_sem2_wait+0x68a __umtx_op_sem2_wait+0x4b
amd64_syscall+0x369 fast_syscall_common+0x101
24869 100545 ruby25              -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14
_cv_timedwait_sig_sbt+0x17a seltdwait+0x7b kern_select+0x87f
sys_select+0x56 amd64_syscall+0x369 fast_syscall_common+0x101
24869 101382 ruby25              ruby-timer-thr      mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369
fast_syscall_common+0x101
24870 100813 ruby25              -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
24870 100960 ruby25              ruby-timer-thr      mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369
fast_syscall_common+0x101
24870 100961 ruby25              cluster.rb:253      mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_select+0x87f sys_select+0x56 amd64_syscall+0x369
fast_syscall_common+0x101
24870 100962 ruby25              cluster.rb:287      mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
24870 100968 ruby25              puma 001            mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
24870 100973 ruby25              reactor.rb:249      mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14
_cv_timedwait_sig_sbt+0x17a seltdwait+0x7b kern_select+0x87f
sys_select+0x56 amd64_syscall+0x369 fast_syscall_common+0x101
24870 100979 ruby25              thread_pool.rb*     mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
24870 100981 ruby25              thread_pool.rb*     mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
24870 100984 ruby25              server.rb:358       mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_select+0x87f sys_select+0x56 amd64_syscall+0x369
fast_syscall_common+0x101
24871 100541 ruby25              -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
24871 100665 ruby25              cluster.rb:253      mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_select+0x87f sys_select+0x56 amd64_syscall+0x369
fast_syscall_common+0x101
24871 100986 ruby25              ruby-timer-thr      mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369
fast_syscall_common+0x101
24871 100987 ruby25              cluster.rb:287      mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
24871 100989 ruby25              puma 001            mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
24871 101001 ruby25              reactor.rb:249      mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14
_cv_timedwait_sig_sbt+0x17a seltdwait+0x7b kern_select+0x87f
sys_select+0x56 amd64_syscall+0x369 fast_syscall_common+0x101
24871 101037 ruby25              thread_pool.rb*     mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
24871 101047 ruby25              thread_pool.rb*     mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
24871 101051 ruby25              server.rb:358       mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_select+0x87f sys_select+0x56 amd64_syscall+0x369
fast_syscall_common+0x101
24873 100637 ruby25              -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
24873 100828 ruby25              cluster.rb:253      mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_select+0x87f sys_select+0x56 amd64_syscall+0x369
fast_syscall_common+0x101
24873 101055 ruby25              ruby-timer-thr      mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369
fast_syscall_common+0x101
24873 101064 ruby25              cluster.rb:287      mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
24873 101077 ruby25              puma 001            mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
24873 101098 ruby25              reactor.rb:249      mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14
_cv_timedwait_sig_sbt+0x17a seltdwait+0x7b kern_select+0x87f
sys_select+0x56 amd64_syscall+0x369 fast_syscall_common+0x101
24873 101102 ruby25              thread_pool.rb*     mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
24873 101118 ruby25              thread_pool.rb*     mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
24873 101122 ruby25              server.rb:358       mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_select+0x87f sys_select+0x56 amd64_syscall+0x369
fast_syscall_common+0x101
24874 100516 ruby25              -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
24874 101036 ruby25              cluster.rb:253      mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_select+0x87f sys_select+0x56 amd64_syscall+0x369
fast_syscall_common+0x101
24874 101129 ruby25              ruby-timer-thr      mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369
fast_syscall_common+0x101
24874 101143 ruby25              cluster.rb:287      mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
24874 101144 ruby25              puma 001            mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
24874 101147 ruby25              reactor.rb:249      mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14
_cv_timedwait_sig_sbt+0x17a seltdwait+0x7b kern_select+0x87f
sys_select+0x56 amd64_syscall+0x369 fast_syscall_common+0x101
24874 101194 ruby25              thread_pool.rb*     mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
24874 101259 ruby25              thread_pool.rb*     mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
24874 101267 ruby25              server.rb:358       mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_select+0x87f sys_select+0x56 amd64_syscall+0x369
fast_syscall_common+0x101
24875 100539 ruby25              -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
24875 101071 ruby25              cluster.rb:253      mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_select+0x87f sys_select+0x56 amd64_syscall+0x369
fast_syscall_common+0x101
24875 101284 ruby25              ruby-timer-thr      mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369
fast_syscall_common+0x101
24875 101289 ruby25              cluster.rb:287      mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
24875 101298 ruby25              puma 001            mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
24875 101334 ruby25              reactor.rb:249      mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14
_cv_timedwait_sig_sbt+0x17a seltdwait+0x7b kern_select+0x87f
sys_select+0x56 amd64_syscall+0x369 fast_syscall_common+0x101
24875 101347 ruby25              thread_pool.rb*     mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
24875 101350 ruby25              thread_pool.rb*     mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
24875 101369 ruby25              server.rb:358       mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_select+0x87f sys_select+0x56 amd64_syscall+0x369
fast_syscall_common+0x101
24876 100642 ruby25              -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
24876 101384 ruby25              ruby-timer-thr      mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369
fast_syscall_common+0x101
24876 101411 ruby25              cluster.rb:253      mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_select+0x87f sys_select+0x56 amd64_syscall+0x369
fast_syscall_common+0x101
24876 101423 ruby25              cluster.rb:287      mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
24876 101429 ruby25              puma 001            mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
24876 101431 ruby25              reactor.rb:249      mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14
_cv_timedwait_sig_sbt+0x17a seltdwait+0x7b kern_select+0x87f
sys_select+0x56 amd64_syscall+0x369 fast_syscall_common+0x101
24876 101441 ruby25              thread_pool.rb*     mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
24876 101444 ruby25              thread_pool.rb*     mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
24876 101447 ruby25              server.rb:358       mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_select+0x87f sys_select+0x56 amd64_syscall+0x369
fast_syscall_common+0x101
24885 100816 snort               -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205
bpfread+0x318 devfs_read_f+0xdc dofileread+0x95 sys_read+0xc3
amd64_syscall+0x369 fast_syscall_common+0x101
24885 101457 snort               -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205
kern_clock_nanosleep+0x1a7 sys_nanosleep+0x5f amd64_syscall+0x369
fast_syscall_common+0x101
25043 100810 ruby25              -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
25043 101658 ruby25              ruby-timer-thr      mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369
fast_syscall_common+0x101
25044 100682 postgres            -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369
fast_syscall_common+0x101
25422 100603 postgres            -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369
fast_syscall_common+0x101
25433 100691 postgres            -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369
fast_syscall_common+0x101
25442 100504 postgres            -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369
fast_syscall_common+0x101
26015 100805 postgres            -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369
fast_syscall_common+0x101
27660 100636 postgres            -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369
fast_syscall_common+0x101
29381 100553 postgres            -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369
fast_syscall_common+0x101
34928 100480 dhcpd               -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14
_cv_timedwait_sig_sbt+0x17a seltdwait+0x7b kern_select+0x87f
sys_select+0x56 amd64_syscall+0x369 fast_syscall_common+0x101
36704 101146 perl                -                   mi_switch+0xe1
sleepq_wait+0x2c sleeplk+0x1c5 lockmgr_xlock_hard+0x19c VOP_LOCK1_APV+0x7e
_vn_lock+0x40 zfs_znode_alloc+0x434 zfs_mknode+0xa9d
zfs_freebsd_create+0x512 VOP_CREATE_APV+0x78 vn_open_cred+0x2c9
kern_openat+0x20c amd64_syscall+0x369 fast_syscall_common+0x101
36751 101709 cron                -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a pipe_read+0x446
dofileread+0x95 sys_read+0xc3 amd64_syscall+0x369 fast_syscall_common+0x101
39368 101861 sendmail            -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
kern_wait6+0x66d sys_wait4+0x78 amd64_syscall+0x369
fast_syscall_common+0x101
39370 101856 mail.local          -                   mi_switch+0xe1
sleepq_wait+0x2c sleeplk+0x1c5 lockmgr_slock_hard+0x2c5 VOP_LOCK1_APV+0x7e
_vn_lock+0x40 zfs_root+0x6d lookup+0x933 namei+0x44b kern_statat+0x77
sys_fstatat+0x2f amd64_syscall+0x369 fast_syscall_common+0x101
40693 101766 perl                -                   mi_switch+0xe1
sleepq_wait+0x2c sleeplk+0x1c5 lockmgr_slock_hard+0x2c5 VOP_LOCK1_APV+0x7e
_vn_lock+0x40 zfs_root+0x6d lookup+0x933 namei+0x44b kern_statat+0x77
sys_fstatat+0x2f amd64_syscall+0x369 fast_syscall_common+0x101
40696 101911 sshd                -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369
fast_syscall_common+0x101
40698 101972 sshd                -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14
_cv_timedwait_sig_sbt+0x17a seltdwait+0x7b kern_select+0x87f
sys_select+0x56 amd64_syscall+0x369 fast_syscall_common+0x101
40699 101310 bash                -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
kern_wait6+0x66d sys_wait4+0x78 amd64_syscall+0x369
fast_syscall_common+0x101
40711 101974 cron                -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a pipe_read+0x446
dofileread+0x95 sys_read+0xc3 amd64_syscall+0x369 fast_syscall_common+0x101
40712 101395 sh                  -                   mi_switch+0xe1
sleepq_wait+0x2c sleeplk+0x1c5 lockmgr_slock_hard+0x2c5 VOP_LOCK1_APV+0x7e
_vn_lock+0x40 zfs_root+0x6d lookup+0x933 namei+0x44b kern_statat+0x77
sys_fstatat+0x2f amd64_syscall+0x369 fast_syscall_common+0x101
40715 101969 su                  -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
kern_wait6+0x66d sys_wait4+0x78 amd64_syscall+0x369
fast_syscall_common+0x101
40716 101976 csh                 -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
kern_sigsuspend+0x174 sys_sigsuspend+0x31 amd64_syscall+0x369
fast_syscall_common+0x101
40722 101977 procstat            -
 sysctl_kern_proc_kstack+0x282 sysctl_root_handler_locked+0x8b
sysctl_root+0x257 userland_sysctl+0x17a sys___sysctl+0x5f
amd64_syscall+0x369 fast_syscall_common+0x101
81907 100546 racoon              -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14
_cv_timedwait_sig_sbt+0x17a seltdwait+0x7b kern_select+0x87f
sys_select+0x56 amd64_syscall+0x369 fast_syscall_common+0x101
83204 100656 perl                -                   mi_switch+0xe1
sleepq_wait+0x2c sleeplk+0x1c5 lockmgr_slock_hard+0x2c5 VOP_LOCK1_APV+0x7e
_vn_lock+0x40 zfs_root+0x6d lookup+0x933 namei+0x44b kern_statat+0x77
sys_fstatat+0x2f amd64_syscall+0x369 fast_syscall_common+0x101
83208 100501 postgres            -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369
fast_syscall_common+0x101
83209 100525 postgres            -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369
fast_syscall_common+0x101
83210 100628 postgres            -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369
fast_syscall_common+0x101
83424 100485 ladvd               -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
kqueue_kevent+0x29e kern_kevent+0xb5 kern_kevent_generic+0x71
sys_kevent+0x61 amd64_syscall+0x369 fast_syscall_common+0x101
83425 100554 ladvd               -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205
kqueue_kevent+0x29e kern_kevent+0xb5 kern_kevent_generic+0x71
sys_kevent+0x61 amd64_syscall+0x369 fast_syscall_common+0x101
83428 101357 perl                -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205
kern_clock_nanosleep+0x1a7 sys_nanosleep+0x5f amd64_syscall+0x369
fast_syscall_common+0x101
83444 100542 snmpd               -                   <running>

83545 100486 sendmail            -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14
_cv_timedwait_sig_sbt+0x17a seltdwait+0x7b kern_select+0x87f
sys_select+0x56 amd64_syscall+0x369 fast_syscall_common+0x101
83548 100814 sendmail            -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
kern_sigsuspend+0x174 sys_sigsuspend+0x31 amd64_syscall+0x369
fast_syscall_common+0x101
83554 100427 named               isc-worker0000      mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
83554 100431 named               isc-worker0001      mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
83554 100432 named               isc-worker0002      mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
83554 100556 named               -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
kern_sigtimedwait+0x3d4 sys_sigwait+0x5c amd64_syscall+0x369
fast_syscall_common+0x101
83554 100774 named               isc-worker0003      mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
83554 100819 named               isc-worker0004      mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
83554 100820 named               isc-worker0005      mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
83554 100849 named               isc-worker0006      mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
83554 100990 named               isc-worker0007      mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
83554 101481 named               isc-timer           mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
83554 101534 named               isc-socket          mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
kqueue_kevent+0x29e kern_kevent+0xb5 kern_kevent_generic+0x71
sys_kevent+0x61 amd64_syscall+0x369 fast_syscall_common+0x101
83556 100608 named               -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
kern_sigtimedwait+0x3d4 sys_sigwait+0x5c amd64_syscall+0x369
fast_syscall_common+0x101
83556 101540 named               isc-worker0000      mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
83556 101541 named               isc-worker0001      mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
83556 101543 named               isc-worker0002      mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
83556 101545 named               isc-worker0003      mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
83556 101546 named               isc-worker0004      mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
83556 101549 named               isc-worker0005      mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
83556 101560 named               isc-worker0006      mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
83556 101583 named               isc-worker0007      mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
83556 101596 named               isc-timer           mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
83556 101600 named               isc-socket          mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
kqueue_kevent+0x29e kern_kevent+0xb5 kern_kevent_generic+0x71
sys_kevent+0x61 amd64_syscall+0x369 fast_syscall_common+0x101
83559 100514 ruby25              -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
83559 101604 ruby25              ruby-timer-thr      mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369
fast_syscall_common+0x101
83562 100463 ruby25              -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
83562 101486 ruby25              ruby-timer-thr      mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369
fast_syscall_common+0x101
83565 100821 ruby25              -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
83565 101606 ruby25              ruby-timer-thr      mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369
fast_syscall_common+0x101
84111 100668 postgres            -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369
fast_syscall_common+0x101
84112 100613 postgres            -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369
fast_syscall_common+0x101
84113 101274 postgres            -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369
fast_syscall_common+0x101
84416 100550 cron                -                   mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205
kern_clock_nanosleep+0x1a7 sys_nanosleep+0x5f amd64_syscall+0x369
fast_syscall_common+0x101
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 12.0-RELEASE zfs/vnode deadlock issue

Andriy Gapon
On 01/03/2019 17:00, Nick Rogers wrote:
> 36704 101146 perl                -                   mi_switch+0xe1
> sleepq_wait+0x2c sleeplk+0x1c5 lockmgr_xlock_hard+0x19c VOP_LOCK1_APV+0x7e
> _vn_lock+0x40 zfs_znode_alloc+0x434 zfs_mknode+0xa9d
> zfs_freebsd_create+0x512 VOP_CREATE_APV+0x78 vn_open_cred+0x2c9
> kern_openat+0x20c amd64_syscall+0x369 fast_syscall_common+0x101

I suspect that this thread is a root cause of the problem.
In this place, the vnode should be freshly created and not visible to anything
but the current thread.  So, vn_lock() should always immediately succeed.  I
cannot understand how the vnode lock could be held by another thread.

--
Andriy Gapon
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 12.0-RELEASE zfs/vnode deadlock issue

freebsd-stable mailing list
In reply to this post by Courtland

> On Mar 1, 2019, at 7:00 AM, Nick Rogers <[hidden email]> wrote:
>
> I am hoping someone can help me figure out if this is a legitimate bug, or
> something already fixed in 12-STABLE. I wish I could reproduce it reliably
> to try against STABLE, but there doesn't appear to be any related ZFS fixes
> not in RELEASE. Thanks.
>

I have also experienced this problem, but I haven’t been able to troubleshoot it at all.

Peter
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 12.0-RELEASE zfs/vnode deadlock issue

Courtland
On Sat, Mar 2, 2019 at 5:27 PM Peter Avalos via freebsd-stable <
[hidden email]> wrote:

>
> > On Mar 1, 2019, at 7:00 AM, Nick Rogers <[hidden email]> wrote:
> >
> > I am hoping someone can help me figure out if this is a legitimate bug,
> or
> > something already fixed in 12-STABLE. I wish I could reproduce it
> reliably
> > to try against STABLE, but there doesn't appear to be any related ZFS
> fixes
> > not in RELEASE. Thanks.
> >
>
> I have also experienced this problem, but I haven’t been able to
> troubleshoot it at all.
>

I've opened a bug report, so if you have any more information about how it
is affecting you that may be helpful to share here.

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=236220


>
> Peter
> _______________________________________________
> [hidden email] mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "[hidden email]"
>
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 12.0-RELEASE zfs/vnode deadlock issue

Courtland
In reply to this post by Andriy Gapon
Thanks for the insight, it does appear that in all instances of this
problem there is always one thread stuck on zfs_znode_alloc. Unfortunately
its always a different application (e.g., perl, sh, postgres). I will post
more information in the bug.

On Sat, Mar 2, 2019 at 12:48 PM Andriy Gapon <[hidden email]> wrote:

> On 01/03/2019 17:00, Nick Rogers wrote:
> > 36704 101146 perl                -                   mi_switch+0xe1
> > sleepq_wait+0x2c sleeplk+0x1c5 lockmgr_xlock_hard+0x19c
> VOP_LOCK1_APV+0x7e
> > _vn_lock+0x40 zfs_znode_alloc+0x434 zfs_mknode+0xa9d
> > zfs_freebsd_create+0x512 VOP_CREATE_APV+0x78 vn_open_cred+0x2c9
> > kern_openat+0x20c amd64_syscall+0x369 fast_syscall_common+0x101
>
> I suspect that this thread is a root cause of the problem.
> In this place, the vnode should be freshly created and not visible to
> anything
> but the current thread.  So, vn_lock() should always immediately succeed.
> I
> cannot understand how the vnode lock could be held by another thread.
>
> --
> Andriy Gapon
>
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 12.0-RELEASE zfs/vnode deadlock issue

Courtland
In reply to this post by Andriy Gapon
On Sat, Mar 2, 2019 at 12:48 PM Andriy Gapon <[hidden email]> wrote:

> On 01/03/2019 17:00, Nick Rogers wrote:
> > 36704 101146 perl                -                   mi_switch+0xe1
> > sleepq_wait+0x2c sleeplk+0x1c5 lockmgr_xlock_hard+0x19c
> VOP_LOCK1_APV+0x7e
> > _vn_lock+0x40 zfs_znode_alloc+0x434 zfs_mknode+0xa9d
> > zfs_freebsd_create+0x512 VOP_CREATE_APV+0x78 vn_open_cred+0x2c9
> > kern_openat+0x20c amd64_syscall+0x369 fast_syscall_common+0x101
>
> I suspect that this thread is a root cause of the problem.
> In this place, the vnode should be freshly created and not visible to
> anything
> but the current thread.  So, vn_lock() should always immediately succeed.
> I
> cannot understand how the vnode lock could be held by another thread.
>

It happened again. I tried to get a backtrace from the offending thread and
one of the others waiting for it. At the moment I have access to this
particular system in its bad state and can leave it like this for as long
as possible, so let me know if there's something else useful I can get out
of the debugger.

courtland# procstat -kka | grep zfs
    0 100140 kernel              zfsvfs              mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
    0 100424 kernel              zfs_vn_rele_taskq   mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
   23 100119 zfskern             arc_reclaim_thread  mi_switch+0xe1
sleepq_timedwait+0x2f _cv_timedwait_sbt+0x17a arc_reclaim_thread+0x146
fork_exit+0x83 fork_trampoline+0xe
   23 100120 zfskern             arc_dnlc_evicts_thr mi_switch+0xe1
sleepq_wait+0x2c _cv_wait+0x152 arc_dnlc_evicts_thread+0x16f fork_exit+0x83
fork_trampoline+0xe
   23 100122 zfskern             dbuf_evict_thread   mi_switch+0xe1
sleepq_timedwait+0x2f _cv_timedwait_sbt+0x17a dbuf_evict_thread+0x1c8
fork_exit+0x83 fork_trampoline+0xe
   23 100139 zfskern             l2arc_feed_thread   mi_switch+0xe1
sleepq_timedwait+0x2f _cv_timedwait_sbt+0x17a l2arc_feed_thread+0x219
fork_exit+0x83 fork_trampoline+0xe
   23 100405 zfskern             trim zroot          mi_switch+0xe1
sleepq_timedwait+0x2f _cv_timedwait_sbt+0x17a trim_thread+0x11f
fork_exit+0x83 fork_trampoline+0xe
   23 100441 zfskern             txg_thread_enter    mi_switch+0xe1
sleepq_wait+0x2c _cv_wait+0x152 txg_quiesce+0x21b txg_quiesce_thread+0x11b
fork_exit+0x83 fork_trampoline+0xe
   23 100442 zfskern             txg_thread_enter    mi_switch+0xe1
sleepq_wait+0x2c _cv_wait+0x152 txg_sync_thread+0x13b fork_exit+0x83
fork_trampoline+0xe
   23 100443 zfskern             solthread 0xfffffff mi_switch+0xe1
sleepq_wait+0x2c _cv_wait+0x152 zthr_procedure+0xcc fork_exit+0x83
fork_trampoline+0xe
   23 100444 zfskern             solthread 0xfffffff mi_switch+0xe1
sleepq_wait+0x2c _cv_wait+0x152 zthr_procedure+0xcc fork_exit+0x83
fork_trampoline+0xe
 7476 100751 postgres            -                   mi_switch+0xe1
sleepq_wait+0x2c _cv_wait+0x152 dmu_tx_wait+0x2eb dmu_tx_assign+0x48
zfs_freebsd_create+0x4c8 VOP_CREATE_APV+0x78 vn_open_cred+0x2c9
kern_openat+0x20c amd64_syscall+0x369 fast_syscall_common+0x101
 7480 100527 postgres            -                   mi_switch+0xe1
sleepq_wait+0x2c sleeplk+0x1c5 lockmgr_xlock_hard+0x19c VOP_LOCK1_APV+0x7e
_vn_lock+0x40 zfs_znode_alloc+0x434 zfs_mknode+0xa9d
zfs_freebsd_create+0x512 VOP_CREATE_APV+0x78 vn_open_cred+0x2c9
kern_openat+0x20c amd64_syscall+0x369 fast_syscall_common+0x101
46101 100471 postgres            -                   mi_switch+0xe1
sleepq_wait+0x2c _cv_wait+0x152 dmu_tx_wait+0x2eb dmu_tx_assign+0x48
zfs_freebsd_create+0x4c8 VOP_CREATE_APV+0x78 vn_open_cred+0x2c9
kern_openat+0x20c amd64_syscall+0x369 fast_syscall_common+0x101
52625 100488 perl                -                   mi_switch+0xe1
sleepq_wait+0x2c sleeplk+0x1c5 lockmgr_slock_hard+0x2c5 VOP_LOCK1_APV+0x7e
_vn_lock+0x40 zfs_root+0x6d lookup+0x933 namei+0x44b kern_statat+0x77
sys_fstatat+0x2f amd64_syscall+0x369 fast_syscall_common+0x101
52675 100643 csh                 -                   mi_switch+0xe1
sleepq_wait+0x2c sleeplk+0x1c5 lockmgr_slock_hard+0x2c5 VOP_LOCK1_APV+0x7e
_vn_lock+0x40 zfs_root+0x6d lookup+0x933 namei+0x44b kern_statat+0x77
sys_fstatat+0x2f amd64_syscall+0x369 fast_syscall_common+0x101
52826 100562 ls                  -                   mi_switch+0xe1
sleepq_wait+0x2c sleeplk+0x1c5 lockmgr_slock_hard+0x2c5 VOP_LOCK1_APV+0x7e
_vn_lock+0x40 zfs_root+0x6d lookup+0x933 namei+0x44b kern_statat+0x77
sys_fstatat+0x2f amd64_syscall+0x369 fast_syscall_common+0x101
52889 100641 bash                -                   mi_switch+0xe1
sleepq_wait+0x2c sleeplk+0x1c5 lockmgr_slock_hard+0x2c5 VOP_LOCK1_APV+0x7e
_vn_lock+0x40 zfs_root+0x6d lookup+0x933 namei+0x44b kern_statat+0x77
sys_fstatat+0x2f amd64_syscall+0x369 fast_syscall_common+0x101
courtland# kgdb
GNU gdb (GDB) 8.2.1 [GDB v8.2.1 for FreeBSD]
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html
>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-portbld-freebsd12.0".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /boot/kernel/kernel...Reading symbols from
/usr/lib/debug//boot/kernel/kernel.debug...done.
done.
<signal handler called>
(kgdb) tid 100471
(kgdb) bt
#0  sched_switch (td=0xfffff8001196c580, newtd=0xfffff8000354f580,
flags=<optimized out>) at /usr/src/sys/kern/sched_ule.c:2112
#1  0xffffffff80d0e0a1 in mi_switch (flags=<optimized out>, newtd=0x0) at
/usr/src/sys/kern/kern_synch.c:439
#2  0xffffffff80d5c80c in sleepq_wait (wchan=<unavailable>,
pri=<unavailable>) at /usr/src/sys/kern/subr_sleepqueue.c:692
#3  0xffffffff80c9ca52 in _cv_wait (cvp=0xfffff80011354d28,
lock=0xfffff80011354c50) at /usr/src/sys/kern/kern_condvar.c:146
#4  0xffffffff80403dfb in dmu_tx_wait (tx=0xfffff8037afd1c00) at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_tx.c:1090
#5  0xffffffff80403748 in dmu_tx_assign (tx=0xfffff8037afd1c00, txg_how=1)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_tx.c:1032
#6  0xffffffff804a4ce8 in zfs_create (excl=0, mode=0, dvp=<optimized out>,
name=<optimized out>, vap=<optimized out>, vpp=<optimized out>,
cr=<optimized out>, td=<optimized out>) at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:1875
#7  zfs_freebsd_create (ap=0xfffffe008e25a790) at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:4996
#8  0xffffffff813aaa68 in VOP_CREATE_APV (vop=<optimized out>,
a=0xfffffe008e25a790) at vnode_if.c:263
#9  0xffffffff80de20f9 in VOP_CREATE (dvp=<unavailable>,
vpp=0xfffffe008e25a888, cnp=<optimized out>, vap=0x18000000001) at
./vnode_if.h:109
#10 vn_open_cred (ndp=0xfffffe008e25a828, flagp=0xfffffe008e25a954,
cmode=<optimized out>, vn_open_flags=<optimized out>,
cred=0xfffff8002b529b00, fp=0xfffff80359185aa0) at
/usr/src/sys/kern/vfs_vnops.c:243
#11 0xffffffff80dda74c in kern_openat (td=0xfffff8001196c580, fd=-100,
path=0x88d3a0f70 <error: Cannot access memory at address 0x88d3a0f70>,
pathseg=UIO_USERSPACE, flags=2563, mode=<optimized out>) at
/usr/src/sys/kern/vfs_syscalls.c:1082
#12 0xffffffff81222449 in syscallenter (td=<optimized out>) at
/usr/src/sys/amd64/amd64/../../kern/subr_syscall.c:135
#13 amd64_syscall (td=0xfffff8001196c580, traced=0) at
/usr/src/sys/amd64/amd64/trap.c:1076
#14 <signal handler called>
#15 0x000000080131e82a in ?? ()
Backtrace stopped: Cannot access memory at address 0x7fffffffb4c8
(kgdb) tid 100527
(kgdb) bt
#0  sched_switch (td=0xfffff8026af9d580, newtd=0xfffff80003551000,
flags=<optimized out>) at /usr/src/sys/kern/sched_ule.c:2112
#1  0xffffffff80d0e0a1 in mi_switch (flags=<optimized out>, newtd=0x0) at
/usr/src/sys/kern/kern_synch.c:439
#2  0xffffffff80d5c80c in sleepq_wait (wchan=<unavailable>,
pri=<unavailable>) at /usr/src/sys/kern/subr_sleepqueue.c:692
#3  0xffffffff80cd9105 in sleeplk (lk=0xfffff802f256c248, flags=<optimized
out>, ilk=<optimized out>, wmesg=<optimized out>, pri=<optimized out>,
timo=51, queue=0) at /usr/src/sys/kern/kern_lock.c:300
#4  0xffffffff80cd819c in lockmgr_xlock_hard (lk=0xfffff802f256c248,
flags=525312, ilk=0xfffff802f256c278, file=<optimized out>, line=<optimized
out>, lwa=<optimized out>) at /usr/src/sys/kern/kern_lock.c:822
#5  0xffffffff813acc5e in VOP_LOCK1_APV (vop=<optimized out>,
a=0xfffffe008e36b280) at vnode_if.c:2087
#6  0xffffffff80de2820 in VOP_LOCK1 (vp=0xfffff802f256c1e0, flags=525312,
file=0xffffffff813c47d0
"/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_znode.c",
line=730) at ./vnode_if.h:859
#7  _vn_lock (vp=0xfffff802f256c1e0, flags=525312, file=0xffffffff813c47d0
"/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_znode.c",
line=730) at /usr/src/sys/kern/vfs_vnops.c:1533
#8  0xffffffff804ae1f4 in zfs_znode_alloc (zfsvfs=<optimized out>,
db=<optimized out>, blksz=0, obj_type=<optimized out>, hdl=<optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_znode.c:730
#9  0xffffffff804adc0d in zfs_mknode (dzp=0xfffff802f461f440,
vap=0xfffffe008e36b6e0, tx=<optimized out>, cr=0xfffff8002b529b00,
flag=<optimized out>, zpp=0xfffffe008e36b638, acl_ids=0xfffffe008e36b5f8)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_znode.c:988
#10 0xffffffff804a4d32 in zfs_create (excl=0, mode=0, dvp=<optimized out>,
name=<optimized out>, vap=<optimized out>, vpp=<optimized out>,
cr=<optimized out>, td=<optimized out>) at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:1883
#11 zfs_freebsd_create (ap=0xfffffe008e36b790) at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:4996
#12 0xffffffff813aaa68 in VOP_CREATE_APV (vop=<optimized out>,
a=0xfffffe008e36b790) at vnode_if.c:263
#13 0xffffffff80de20f9 in VOP_CREATE (dvp=<unavailable>,
vpp=0xfffffe008e36b888, cnp=<optimized out>, vap=0x18000000001) at
./vnode_if.h:109
#14 vn_open_cred (ndp=0xfffffe008e36b828, flagp=0xfffffe008e36b954,
cmode=<optimized out>, vn_open_flags=<optimized out>,
cred=0xfffff8002b529b00, fp=0xfffff8026d933d70) at
/usr/src/sys/kern/vfs_vnops.c:243
#15 0xffffffff80dda74c in kern_openat (td=0xfffff8026af9d580, fd=-100,
path=0x7fffffffc630 <error: Cannot access memory at address
0x7fffffffc630>, pathseg=UIO_USERSPACE, flags=1538, mode=<optimized out>)
at /usr/src/sys/kern/vfs_syscalls.c:1082
#16 0xffffffff81222449 in syscallenter (td=<optimized out>) at
/usr/src/sys/amd64/amd64/../../kern/subr_syscall.c:135
#17 amd64_syscall (td=0xfffff8026af9d580, traced=0) at
/usr/src/sys/amd64/amd64/trap.c:1076
#18 <signal handler called>
#19 0x000000080138b4fa in ?? ()
Backtrace stopped: Cannot access memory at address 0x7fffffffc138
(kgdb) fr 7
#7  _vn_lock (vp=0xfffff802f256c1e0, flags=525312, file=0xffffffff813c47d0
"/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_znode.c",
line=730) at /usr/src/sys/kern/vfs_vnops.c:1533
warning: Source file is more recent than executable.
1533 error = VOP_LOCK1(vp, flags, file, line);
(kgdb) print *vp
$1 = {v_tag = 0xffffffff8144af45 "zfs", v_op = 0xffffffff81c64fd0
<zfs_vnodeops>, v_data = 0xfffff802608c3440, v_mount = 0x0, v_nmntvnodes =
{tqe_next = 0xfffff8037a047000, tqe_prev = 0xfffff802f23feb60},
{v_mountedhere = 0x0, v_unpcb = 0x0, v_rdev = 0x0, v_fifoinfo = 0x0},
v_hashlist = {le_next = 0x0, le_prev = 0x0}, v_cache_src = {
    lh_first = 0x0}, v_cache_dst = {tqh_first = 0x0, tqh_last =
0xfffff802f256c230}, v_cache_dd = 0x0, v_lock = {lock_object = {lo_name =
0xffffffff8144af45 "zfs", lo_flags = 117112840, lo_data = 0, lo_witness =
0x0}, lk_lock = 18446744073709551605, lk_exslpfail = 0, lk_timo = 51,
lk_pri = 96}, v_interlock = {lock_object = {
      lo_name = 0xffffffff814e4508 "vnode interlock", lo_flags = 16973824,
lo_data = 0, lo_witness = 0x0}, mtx_lock = 0}, v_vnlock =
0xfffff802f256c248, v_actfreelist = {tqe_next = 0xfffff8037a047000,
tqe_prev = 0xfffff80011333a38}, v_bufobj = {bo_lock = {lock_object =
{lo_name = 0xffffffff814a9e5f "bufobj interlock", lo_flags = 86179840,
        lo_data = 0, lo_witness = 0x0}, rw_lock = 1}, bo_ops =
0xffffffff81d38600 <buf_ops_bio>, bo_object = 0x0, bo_synclist = {le_next =
0x0, le_prev = 0x0}, bo_private = 0xfffff802f256c1e0, bo_clean = {bv_hd =
{tqh_first = 0x0, tqh_last = 0xfffff802f256c2f8}, bv_root = {pt_root = 0},
bv_cnt = 0}, bo_dirty = {bv_hd = {tqh_first = 0x0,
        tqh_last = 0xfffff802f256c318}, bv_root = {pt_root = 0}, bv_cnt =
0}, bo_numoutput = 0, bo_flag = 0, bo_domain = 1, bo_bsize = 16384},
v_pollinfo = 0x0, v_label = 0x0, v_lockf = 0x0, v_rl = {rl_waiters =
{tqh_first = 0x0, tqh_last = 0xfffff802f256c368}, rl_currdep = 0x0},
v_cstart = 0, v_lasta = 0, v_lastw = 0, v_clen = 0,
  v_holdcnt = 1, v_usecount = 1, v_iflag = 0, v_vflag = 0, v_mflag = 0,
v_writecount = 0, v_hash = 49436353, v_type = VREG}
(kgdb)




> --
> Andriy Gapon
>
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 12.0-RELEASE zfs/vnode deadlock issue

Andriy Gapon
On 04/03/2019 22:35, Nick Rogers wrote:
> v_lock = {lock_object = {lo_name =
> 0xffffffff8144af45 "zfs", lo_flags = 117112840, lo_data = 0, lo_witness =
> 0x0}, lk_lock = 18446744073709551605, lk_exslpfail = 0, lk_timo = 51,
> lk_pri = 96}

Hmm, lk_lock looks bogus.
18446744073709551605 == 0xfffffffffffffff5 and it's LK_SHARE |
LK_EXCLUSIVE_WAITERS with 0xfffffffffffffff shared owners.
Perhaps, this is a result of LK_SHARERS_LOCK(-1).

Is your kernel compiled with INVARIANTS and INVARIANT_SUPPORT?
I suspect that the vnode was accessed (unlocked?) through a stale pointer after
it was recycled.

--
Andriy Gapon
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 12.0-RELEASE zfs/vnode deadlock issue

Courtland
On Mon, Mar 4, 2019 at 5:29 PM Andriy Gapon <[hidden email]> wrote:

> On 04/03/2019 22:35, Nick Rogers wrote:
> > v_lock = {lock_object = {lo_name =
> > 0xffffffff8144af45 "zfs", lo_flags = 117112840, lo_data = 0, lo_witness =
> > 0x0}, lk_lock = 18446744073709551605, lk_exslpfail = 0, lk_timo = 51,
> > lk_pri = 96}
>
> Hmm, lk_lock looks bogus.
> 18446744073709551605 == 0xfffffffffffffff5 and it's LK_SHARE |
> LK_EXCLUSIVE_WAITERS with 0xfffffffffffffff shared owners.
> Perhaps, this is a result of LK_SHARERS_LOCK(-1).
>
> Is your kernel compiled with INVARIANTS and INVARIANT_SUPPORT?
> I suspect that the vnode was accessed (unlocked?) through a stale pointer
> after
> it was recycled.
>

I don't believe so - it's basically amd64 GENERIC w/ a reduced set of
modules and static zfs option.


> --
> Andriy Gapon
>
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"