Disk/ZFS activity crash on 11.2-STABLE

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Disk/ZFS activity crash on 11.2-STABLE

list
I have a server running a Supermicro X11DPH-T system board (dmesg below)
which is unstable under heavy disk read load.  The following command is
sufficient to trigger a crash:

# zfs send -R pool@snapshot > /dev/null

That command takes an unpredictable amount of time to trigger the crash,
but I have never seen it complete successfully.

Further, crash dumps of this are difficult to obtain (see below),
despite the fact that dumps are enabled and tested via:

# sysctl debug.kdb.panic=1

As disclosure, this machine sometimes exhibits a second failure mode
with a "kernel trap 12 with interrupts disabled."  This typically occurs
on boot, just about the point where the kernel begins enumerating the
CPUs.  But that is not (yet) my primary concern.

I typically perform the 'zfs send' stress test over an ssh session,
although the server is nearby on a test bench.  I have not yet been at
the console when the panic or crash happens, so I am not certain that
the kernel is "panicking gracefully," if you'll pardon the oxymoron, I
have yet to find any crash dumps in /var/crash after a disk-induced
panic.

Is a serial console the next step in troubleshooting this?  I am fairly
strong in bash scripting, but inexperienced in kernel crash debugging.
I already have remote syslogging set on this server, so it's feasible
to write certain terse tidbits to syslog ("still alive at xx:yy PDT")
and/or dump some key statistics every minute to the server's local ZFS
pool, and then see what the last stats dump file shows upon reboot.

What are some key stats I should capture at periodic intervals while
waiting for the 'zfs send' to induce the crash -- iostat?  zfs-stats is
also installed, but I'm ignorant about what stats I should dump, and how
to infer any post-mortem meaning from them.

This problem first appeared when doing a backup of this machine.  The
'zfs send' was invoked by a remote host over an ssh session with jumbo
frames enabled, achieving a throughput of around 115 MB/sec going out
over GigE.  To eliminate the network driver as a possible point of
failure, I discovered that sending the 'zfs send' output to /dev/null
was sufficient to trigger the crash.  I may try rate-limiting the
output pipe with:

# zfs send -R pool@snapshot | pv -Wbrat -L 1M > /dev/null

to see whether the -L value has any effect on the ability of this
command to complete successfully.

After I started drafting this write-up, I finally got a crash dump.  I
ran the stress script:

#!/usr/bin/env bash

FILE="/var/crash/zfs-stress.start"
SLEEP_TIME=10

# if this script causes a crash, then we should expect to see a
# crash dump newer than this file:

touch $FILE

logger "begin $0"

zfs destroy -rv tank@stress-test 2> /dev/null

zfs snap -r tank@stress-test

logger "begin 'zfs send -R tank@stress-test"

zfs send -R tank@stress-test > /dev/null &

logger "zfs send running in background"

while [ -f $FILE ] && sleep $SLEEP_TIME; do

  logger "still alive @ $(date +%H:%M:%S)"

done

-- snip

while also running 'systat -vm' in one virtual terminal and writing
'zpool iostat -v' to disk every couple seconds.  I did finally observe a
panic and this time got a dump.

One quirk I forgot to mention is that after this ZFS-induced crash, on the
next boot, the /etc/rc somehow does not run dhclient on the primary
interface, despite its presence in /etc/rc.conf:

ifconfig_ixl0="DHCP"

I have no idea whether that is at all related, but it does mean that
even if the system reboots after a crash, it is inaccessible except by
the physical console.  I have to log in as root and say 'dhclient ixl0'.
On the NEXT boot, the DHCP client is invoked by /etc/rc as one would
expect.  I have:

rc_debug="YES"

defined in /etc/rc.conf but I am not seeing any messages that explain why
dhclient is not being executed.

I would be very grateful to anyone who has time to examine these post-mortems
and offer any suggestions on where to go next.

Thank you for your time reading this lengthy trouble report.

Jim



As promised, dmesg:


Copyright (c) 1992-2018 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.2-STABLE #0 r335674: Tue Jun 26 13:20:24 PDT 2018
    [hidden email]:/usr/obj/usr/src/sys/GENERIC amd64
FreeBSD clang version 6.0.0 (tags/RELEASE_600/final 326565) (based on LLVM 6.0.0)
VT(efifb): resolution 1680x1050
CPU: Intel(R) Xeon(R) Gold 5118 CPU @ 2.30GHz (2300.08-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x50654  Family=0x6  Model=0x55  Stepping=4
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x7ffefbff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x121<LAHF,ABM,Prefetch>
  Structured Extended Features=0xd39ffffb<FSGSBASE,TSCADJ,BMI1,HLE,AVX2,FDPEXC,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,NFPUSG,MPX,PQE,AVX512F,AVX512DQ,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,PROCTRACE,AVX512CD,AVX512BW,AVX512VL>
  Structured Extended Features2=0x8<PKU>
  Structured Extended Features3=0xc000000<IBPB,STIBP>
  XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
  TSC: P-state invariant, performance statistics
real memory  = 103079215104 (98304 MB)
avail memory = 98749128704 (94174 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <SUPERM SMCI--MB>
FreeBSD/SMP: Multiprocessor System Detected: 48 CPUs
FreeBSD/SMP: 2 package(s) x 12 core(s) x 2 hardware threads
ioapic6: Changing APIC ID to 16
ioapic7: Changing APIC ID to 17
ioapic8: Changing APIC ID to 18
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 24-31 on motherboard
ioapic2 <Version 2.0> irqs 32-39 on motherboard
ioapic3 <Version 2.0> irqs 40-47 on motherboard
ioapic4 <Version 2.0> irqs 48-55 on motherboard
ioapic5 <Version 2.0> irqs 72-79 on motherboard
ioapic6 <Version 2.0> irqs 80-87 on motherboard
ioapic7 <Version 2.0> irqs 88-95 on motherboard
ioapic8 <Version 2.0> irqs 96-103 on motherboard
SMP: AP CPU #1 Launched!
SMP: AP CPU #38 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #16 Launched!
SMP: AP CPU #12 Launched!
SMP: AP CPU #6 Launched!
SMP: AP CPU #17 Launched!
SMP: AP CPU #40 Launched!
SMP: AP CPU #29 Launched!
SMP: AP CPU #5 Launched!
SMP: AP CPU #27 Launched!
SMP: AP CPU #19 Launched!
SMP: AP CPU #13 Launched!
SMP: AP CPU #22 Launched!
SMP: AP CPU #4 Launched!
SMP: AP CPU #28 Launched!
SMP: AP CPU #30 Launched!
SMP: AP CPU #11 Launched!
SMP: AP CPU #42 Launched!
SMP: AP CPU #32 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #34 Launched!
SMP: AP CPU #14 Launched!
SMP: AP CPU #37 Launched!
SMP: AP CPU #20 Launched!
SMP: AP CPU #41 Launched!
SMP: AP CPU #21 Launched!
SMP: AP CPU #18 Launched!
SMP: AP CPU #35 Launched!
SMP: AP CPU #23 Launched!
SMP: AP CPU #10 Launched!
SMP: AP CPU #36 Launched!
SMP: AP CPU #31 Launched!
SMP: AP CPU #43 Launched!
SMP: AP CPU #44 Launched!
SMP: AP CPU #45 Launched!
SMP: AP CPU #39 Launched!
SMP: AP CPU #8 Launched!
SMP: AP CPU #24 Launched!
SMP: AP CPU #25 Launched!
SMP: AP CPU #46 Launched!
SMP: AP CPU #9 Launched!
SMP: AP CPU #26 Launched!
SMP: AP CPU #33 Launched!
SMP: AP CPU #15 Launched!
SMP: AP CPU #7 Launched!
SMP: AP CPU #47 Launched!
Timecounter "TSC-low" frequency 1150040467 Hz quality 1000
random: entropy device external interface
kbd1 at kbdmux0
netmap: loaded module
module_register_init: MOD_LOAD (vesa, 0xffffffff80ff4590, 0) error 19
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
nexus0
cryptosoft0: <software crypto> on motherboard
acpi0: <SUPERM SUPERM> on motherboard
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> numa-domain 0 on acpi0
cpu1: <ACPI CPU> numa-domain 0 on acpi0
cpu2: <ACPI CPU> numa-domain 0 on acpi0
cpu3: <ACPI CPU> numa-domain 0 on acpi0
cpu4: <ACPI CPU> numa-domain 0 on acpi0
cpu5: <ACPI CPU> numa-domain 0 on acpi0
cpu6: <ACPI CPU> numa-domain 0 on acpi0
cpu7: <ACPI CPU> numa-domain 0 on acpi0
cpu8: <ACPI CPU> numa-domain 0 on acpi0
cpu9: <ACPI CPU> numa-domain 0 on acpi0
cpu10: <ACPI CPU> numa-domain 0 on acpi0
cpu11: <ACPI CPU> numa-domain 0 on acpi0
cpu12: <ACPI CPU> numa-domain 0 on acpi0
cpu13: <ACPI CPU> numa-domain 0 on acpi0
cpu14: <ACPI CPU> numa-domain 0 on acpi0
cpu15: <ACPI CPU> numa-domain 0 on acpi0
cpu16: <ACPI CPU> numa-domain 0 on acpi0
cpu17: <ACPI CPU> numa-domain 0 on acpi0
cpu18: <ACPI CPU> numa-domain 0 on acpi0
cpu19: <ACPI CPU> numa-domain 0 on acpi0
cpu20: <ACPI CPU> numa-domain 0 on acpi0
cpu21: <ACPI CPU> numa-domain 0 on acpi0
cpu22: <ACPI CPU> numa-domain 0 on acpi0
cpu23: <ACPI CPU> numa-domain 0 on acpi0
cpu24: <ACPI CPU> numa-domain 1 on acpi0
cpu25: <ACPI CPU> numa-domain 1 on acpi0
cpu26: <ACPI CPU> numa-domain 1 on acpi0
cpu27: <ACPI CPU> numa-domain 1 on acpi0
cpu28: <ACPI CPU> numa-domain 1 on acpi0
cpu29: <ACPI CPU> numa-domain 1 on acpi0
cpu30: <ACPI CPU> numa-domain 1 on acpi0
cpu31: <ACPI CPU> numa-domain 1 on acpi0
cpu32: <ACPI CPU> numa-domain 1 on acpi0
cpu33: <ACPI CPU> numa-domain 1 on acpi0
cpu34: <ACPI CPU> numa-domain 1 on acpi0
cpu35: <ACPI CPU> numa-domain 1 on acpi0
cpu36: <ACPI CPU> numa-domain 1 on acpi0
cpu37: <ACPI CPU> numa-domain 1 on acpi0
cpu38: <ACPI CPU> numa-domain 1 on acpi0
cpu39: <ACPI CPU> numa-domain 1 on acpi0
cpu40: <ACPI CPU> numa-domain 1 on acpi0
cpu41: <ACPI CPU> numa-domain 1 on acpi0
cpu42: <ACPI CPU> numa-domain 1 on acpi0
cpu43: <ACPI CPU> numa-domain 1 on acpi0
cpu44: <ACPI CPU> numa-domain 1 on acpi0
cpu45: <ACPI CPU> numa-domain 1 on acpi0
cpu46: <ACPI CPU> numa-domain 1 on acpi0
cpu47: <ACPI CPU> numa-domain 1 on acpi0
atrtc0: <AT realtime clock> port 0x70-0x71,0x74-0x77 irq 8 on acpi0
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 24000000 Hz quality 950
Event timer "HPET" frequency 24000000 Hz quality 350
Event timer "HPET1" frequency 24000000 Hz quality 340
Event timer "HPET2" frequency 24000000 Hz quality 340
Event timer "HPET3" frequency 24000000 Hz quality 340
Event timer "HPET4" frequency 24000000 Hz quality 340
Event timer "HPET5" frequency 24000000 Hz quality 340
Event timer "HPET6" frequency 24000000 Hz quality 340
Event timer "HPET7" frequency 24000000 Hz quality 340
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x508-0x50b on acpi0
acpi_syscontainer0: <System Container> on acpi0
acpi_syscontainer1: <System Container> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff numa-domain 0 on acpi0
pcib0: _OSC returned error 0x10
pci0: <ACPI PCI bus> numa-domain 0 on pcib0
pci0: <dasp, performance counters> at device 8.1 (no driver attached)
pci0: <unknown> at device 17.0 (no driver attached)
pci0: <unknown> at device 17.1 (no driver attached)
ahci0: <Intel Lewisburg AHCI SATA controller> port 0x3070-0x3077,0x3060-0x3063,0x3020-0x303f mem 0x9d206000-0x9d207fff,0x9d209000-0x9d2090ff,0x9d180000-0x9d1fffff irq 16 at device 17.5 numa-domain 0 on pci0
ahci0: AHCI v1.31 with 6 6Gbps ports, Port Multiplier not supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich2: <AHCI channel> at channel 2 on ahci0
ahcich3: <AHCI channel> at channel 3 on ahci0
ahcich4: <AHCI channel> at channel 4 on ahci0
ahcich5: <AHCI channel> at channel 5 on ahci0
ahciem0: <AHCI enclosure management bridge> on ahci0
xhci0: <Intel Lewisburg USB 3.0 controller> mem 0x383ffff00000-0x383ffff0ffff irq 16 at device 20.0 numa-domain 0 on pci0
xhci0: 32 bytes context size, 64-bit DMA
usbus0: waiting for BIOS to give up control
xhci_interrupt: host controller halted
usbus0 numa-domain 0 on xhci0
usbus0: 5.0Gbps Super Speed USB v3.0
pci0: <simple comms> at device 22.0 (no driver attached)
pci0: <simple comms> at device 22.1 (no driver attached)
pci0: <simple comms> at device 22.4 (no driver attached)
ahci1: <Intel Lewisburg AHCI SATA controller> port 0x3050-0x3057,0x3040-0x3043,0x3000-0x301f mem 0x9d204000-0x9d205fff,0x9d208000-0x9d2080ff,0x9d100000-0x9d17ffff irq 16 at device 23.0 numa-domain 0 on pci0
ahci1: AHCI v1.31 with 8 6Gbps ports, Port Multiplier not supported
ahcich6: <AHCI channel> at channel 0 on ahci1
ahcich7: <AHCI channel> at channel 1 on ahci1
ahcich8: <AHCI channel> at channel 2 on ahci1
ahcich9: <AHCI channel> at channel 3 on ahci1
ahcich10: <AHCI channel> at channel 4 on ahci1
ahcich11: <AHCI channel> at channel 5 on ahci1
ahcich12: <AHCI channel> at channel 6 on ahci1
ahcich13: <AHCI channel> at channel 7 on ahci1
ahciem1: <AHCI enclosure management bridge> on ahci1
pcib1: <ACPI PCI-PCI bridge> irq 16 at device 28.0 numa-domain 0 on pci0
pci1: <ACPI PCI bus> numa-domain 0 on pcib1
pcib2: <ACPI PCI-PCI bridge> irq 17 at device 28.5 numa-domain 0 on pci0
pci2: <ACPI PCI bus> numa-domain 0 on pcib2
pcib3: <ACPI PCI-PCI bridge> irq 17 at device 0.0 numa-domain 0 on pci2
pci3: <ACPI PCI bus> numa-domain 0 on pcib3
vgapci0: <VGA-compatible display> port 0x2000-0x207f mem 0x9c000000-0x9cffffff,0x9d000000-0x9d01ffff irq 17 at device 0.0 numa-domain 0 on pci3
vgapci0: Boot video device
isab0: <PCI-ISA bridge> at device 31.0 numa-domain 0 on pci0
isa0: <ISA bus> numa-domain 0 on isab0
pci0: <memory> at device 31.2 (no driver attached)
pci0: <serial bus> at device 31.5 (no driver attached)
pcib4: <ACPI Host-PCI bridge> numa-domain 0 on acpi0
pcib4: _OSC returned error 0x10
pci4: <ACPI PCI bus> numa-domain 0 on pcib4
pcib5: <ACPI PCI-PCI bridge> irq 39 at device 0.0 numa-domain 0 on pci4
pci5: <ACPI PCI bus> numa-domain 0 on pcib5
pcib6: <ACPI PCI-PCI bridge> mem 0xaae00000-0xaae1ffff irq 32 at device 0.0 numa-domain 0 on pci5
pci6: <ACPI PCI bus> numa-domain 0 on pcib6
pcib7: <PCI-PCI bridge> irq 38 at device 3.0 numa-domain 0 on pci6
pci7: <PCI bus> numa-domain 0 on pcib7
ixl0: <Intel(R) Ethernet Connection 700 Series PF Driver, Version - 1.9.9-k> mem 0xa9000000-0xa9ffffff,0xaa008000-0xaa00ffff irq 38 at device 0.0 numa-domain 0 on pci7
ixl0: using 1024 tx descriptors and 1024 rx descriptors
ixl0: fw 3.1.52349 api 1.5 nvm 3.25 etid 800009e7 oem 1.262.0
ixl0: PF-ID[0]: VFs 32, MSIX 129, VF MSIX 5, QPs 768, MDIO shared
ixl0: Using MSIX interrupts with 9 vectors
ixl0: Allocating 8 queues for PF LAN VSI; 8 queues active
ixl0: Ethernet address: ac:1f:6b:61:54:06
ixl0: SR-IOV ready
queues is 0xfffffe0001d13000
ixl0: netmap queues/slots: TX 8/1024, RX 8/1024
ixl1: <Intel(R) Ethernet Connection 700 Series PF Driver, Version - 1.9.9-k> mem 0xa8000000-0xa8ffffff,0xaa000000-0xaa007fff irq 38 at device 0.1 numa-domain 0 on pci7
ixl1: using 1024 tx descriptors and 1024 rx descriptors
ixl1: fw 3.1.52349 api 1.5 nvm 3.25 etid 800009e7 oem 1.262.0
ixl1: PF-ID[1]: VFs 32, MSIX 129, VF MSIX 5, QPs 768, MDIO shared
ixl1: Using MSIX interrupts with 9 vectors
ixl1: Allocating 8 queues for PF LAN VSI; 8 queues active
ixl1: Ethernet address: ac:1f:6b:61:54:07
ixl1: SR-IOV ready
queues is 0xfffffe0001e69000
ixl1: netmap queues/slots: TX 8/1024, RX 8/1024
pcib8: <ACPI Host-PCI bridge> numa-domain 0 on acpi0
pcib8: _OSC returned error 0x10
pci8: <ACPI PCI bus> numa-domain 0 on pcib8
pcib9: <ACPI PCI-PCI bridge> irq 47 at device 2.0 numa-domain 0 on pci8
pci9: <ACPI PCI bus> numa-domain 0 on pcib9
mpr0: <Avago Technologies (LSI) SAS3008> port 0x7000-0x70ff mem 0xb8640000-0xb864ffff,0xb8600000-0xb863ffff irq 42 at device 0.0 numa-domain 0 on pci9
mpr0: Firmware: 15.00.03.00, Driver: 18.03.00.00-fbsd
mpr0: IOCCapabilities: 7a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc,FastPath,RDPQArray>
pcib10: <ACPI Host-PCI bridge> numa-domain 0 on acpi0
pcib10: _OSC returned error 0x10
pci10: <ACPI PCI bus> numa-domain 0 on pcib10
pci10: <dasp, performance counters> at device 14.0 (no driver attached)
pci10: <dasp, performance counters> at device 15.0 (no driver attached)
pci10: <dasp, performance counters> at device 18.0 (no driver attached)
pci10: <dasp, performance counters> at device 18.1 (no driver attached)
pcib11: <ACPI Host-PCI bridge> numa-domain 1 on acpi0
ixl0: Link is up, 1 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: None, Autoneg: True, Flow Control: None
ixl0: link state changed to UP
ixl1: Link is up, 1 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: None, Autoneg: True, Flow Control: Full
ixl1: link state changed to UP
pcib11: _OSC returned error 0x10
pci11: <ACPI PCI bus> numa-domain 1 on pcib11
pci11: <dasp, performance counters> at device 8.1 (no driver attached)
pcib12: <ACPI Host-PCI bridge> numa-domain 1 on acpi0
pcib12: _OSC returned error 0x10
pci12: <ACPI PCI bus> numa-domain 1 on pcib12
pcib13: <ACPI Host-PCI bridge> numa-domain 1 on acpi0
pcib13: _OSC returned error 0x10
pci13: <ACPI PCI bus> numa-domain 1 on pcib13
pcib14: <ACPI Host-PCI bridge> numa-domain 1 on acpi0
pcib14: _OSC returned error 0x10
pci14: <ACPI PCI bus> numa-domain 1 on pcib14
pci14: <dasp, performance counters> at device 14.0 (no driver attached)
pci14: <dasp, performance counters> at device 15.0 (no driver attached)
pci14: <dasp, performance counters> at device 18.0 (no driver attached)
pci14: <dasp, performance counters> at device 18.1 (no driver attached)
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
orm0: <ISA Option ROM> at iomem 0xc0000-0xc7fff on isa0
ppc0: cannot reserve I/O port range
est0: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu0
est1: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu1
est2: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu2
est3: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu3
est4: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu4
est5: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu5
est6: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu6
est7: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu7
est8: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu8
est9: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu9
est10: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu10
est11: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu11
est12: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu12
est13: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu13
est14: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu14
est15: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu15
est16: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu16
est17: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu17
est18: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu18
est19: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu19
est20: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu20
est21: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu21
est22: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu22
est23: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu23
est24: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu24
est25: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu25
est26: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu26
est27: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu27
est28: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu28
est29: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu29
est30: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu30
est31: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu31
est32: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu32
est33: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu33
est34: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu34
est35: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu35
est36: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu36
est37: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu37
est38: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu38
est39: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu39
est40: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu40
est41: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu41
est42: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu42
est43: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu43
est44: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu44
est45: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu45
est46: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu46
est47: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu47
random: unblocking device.
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
Timecounters tick every 1.000 msec
ugen0.1: <0x8086 XHCI root HUB> at usbus0
mpr0: uhub0: <0x8086 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
SAS Address from SAS device page0 = 5000c500955fb571
mpr0: Found device <401<SspTarg>,End Device> <12.0Gbps> handle<0x000a> enclosureHandle<0x0002> slot 0
mpr0: At enclosure level 0 and connector name (    )
mpr0: SAS Address from SAS device page0 = 5000c5009550e4f1
mpr0: Found device <401<SspTarg>,End Device> <12.0Gbps> handle<0x000b> enclosureHandle<0x0002> slot 1
mpr0: At enclosure level 0 and connector name (    )
mpr0: SAS Address from SAS device page0 = 5000c50095518c95
mpr0: Found device <401<SspTarg>,End Device> <12.0Gbps> handle<0x000c> enclosureHandle<0x0002> slot 2
mpr0: At enclosure level 0 and connector name (    )
mpr0: SAS Address from SAS device page0 = 5000c500955fb3c5
mpr0: Found device <401<SspTarg>,End Device> <12.0Gbps> handle<0x000d> enclosureHandle<0x0002> slot 3
mpr0: At enclosure level 0 and connector name (    )
mpr0: SAS Address from SAS device page0 = 5000c500955fb021
mpr0: Found device <401<SspTarg>,End Device> <12.0Gbps> handle<0x000e> enclosureHandle<0x0002> slot 4
mpr0: At enclosure level 0 and connector name (    )
mpr0: SAS Address from SAS device page0 = 5000c5009550e891
mpr0: Found device <401<SspTarg>,End Device> <12.0Gbps> handle<0x000f> enclosureHandle<0x0002> slot 5
mpr0: At enclosure level 0 and connector name (    )
mpr0: SAS Address from SAS device page0 = 500304801eb5dbfd
mpr0: Found device <4411<SmpInit,SspTarg,SepDev>,End Device> <12.0Gbps> handle<0x0010> enclosureHandle<0x0002> slot 12
mpr0: At enclosure level 0 and connector name (    )
ses0 at ahciem0 bus 0 scbus6 target 0 lun 0
ses0: <AHCI SGPIO Enclosure 1.00 0001> SEMB S-E-S 2.00 device
ses0: SEMB SES Device
ses1 at ahciem1 bus 0 scbus15 target 0 lun 0
ses1: <AHCI SGPIO Enclosure 1.00 0001> SEMB S-E-S 2.00 device
ses1: SEMB SES Device
ses2 at mpr0 bus 0 scbus16 target 20 lun 0
ses2: <LSI SAS3x28 0601> Fixed Enclosure Services SPC-3 SCSI device
ses2: 1200.000MB/s transfers
ses2: Command Queueing enabled
ses2: SCSI-3 ENC Device
ada0 at ahcich4 bus 0 scbus4 target 0 lun 0
ada0: <INTEL SSDSC2KB240G7 SCV10121> ACS-3 ATA SATA 3.x device
ada0: Serial Number BTYS81150CWH240AGN
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 512bytes)
ada0: Command Queueing enabled
ada0: 228936MB (468862128 512 byte sectors)
ada1 at ahcich5 bus 0 scbus5 target 0 lun 0
ada1: <INTEL SSDSC2KB240G7 SCV10121> ACS-3 ATA SATA 3.x device
ada1: Serial Number BTYS814205JS240AGN
ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 512bytes)
ada1: Command Queueing enabled
ada1: 228936MB (468862128 512 byte sectors)
uhub0: 26 ports with 26 removable, self powered
ses2: pass4,da0: Element descriptor: 'Slot00'
da1 at mpr0 bus 0 scbus16 target 9 lun 0
da2 at mpr0 bus 0 scbus16 target 10 lun 0
da0 at mpr0 bus 0 scbus16 target 8 lun 0
da5 at mpr0 bus 0 scbus16 target 13 lun 0
da4 at mpr0 bus 0 scbus16 target 12 lun 0
da2: <SEAGATE ST4000NM0095 K004> Fixed Direct Access SPC-4 SCSI device
da2: Serial Number ZC15T3GB0000C832EM0C
da2: 1200.000MB/s transfers
da2: Command Queueing enabled
da2: 3815447MB (976754646 4096 byte sectors)
da0: <SEAGATE ST4000NM0095 K004> Fixed Direct Access SPC-4 SCSI device
da0: Serial Number ZC1606NJ0000R833G9JK
da0: 1200.000MB/s transfers
da0: Command Queueing enabled
da0: 3815447MB (976754646 4096 byte sectors)
da4: <SEAGATE ST4000NM0095 K004> Fixed Direct Access SPC-4 SCSI device
da4: Serial Number ZC1606RR0000R833G8YR
da4: 1200.000MB/s transfers
da4: Command Queueing enabled
da4: 3815447MB (976754646 4096 byte sectors)
ses2: pass4,da0: SAS Device Slot Element: 1 Phys at Slot 0
ses2:  phy 0: SAS device type 1 id 0
da5: da1: <SEAGATE ST4000NM0095 K004> Fixed Direct Access SPC-4 SCSI device
da1: Serial Number ZC15V7DV0000C8258M38
ugen0.2: <Dell Dell USB Entry Keyboard> at usbus0
da1: 1200.000MB/s transfers
da1: Command Queueing enabled
da1: 3815447MB (976754646 4096 byte sectors)
<SEAGATE ST4000NM0095 K004> Fixed Direct Access SPC-4 SCSI device
ukbd0da5: Serial Number ZC15KB320000R832RAR7
da5: 1200.000MB/s transfers
da5: Command Queueing enabled
da5: 3815447MB (976754646 4096 byte sectors)
 numa-domain 0 on uhub0
da3 at mpr0 bus 0 scbus16 target 11 lun 0
ukbd0: <Dell Dell USB Entry Keyboard, class 0/0, rev 1.10/1.15, addr 1> on usbus0
kbd2 at ukbd0
ses2:  phy 0: protocols: Initiator( None ) Target( SSP )
ses2:  phy 0: parent 500304801eb5dbff addr 5000c500955fb571
da3: <SEAGATE ST4000NM0095 K004> Fixed Direct Access SPC-4 SCSI device
da3: Serial Number ZC1606Q80000R833WKE8
da3: 1200.000MB/s transfers
da3: Command Queueing enabled
da3: 3815447MB (976754646 4096 byte sectors)
ses2: pass5,da1: Element descriptor: 'Slot01'
ses2: pass5,da1: SAS Device Slot Element: 1 Phys at Slot 1
ses2:  phy 0: SAS device type 1 id 0
ses2:  phy 0: protocols: Initiator( None ) Target( SSP )
ses2:  phy 0: parent 500304801eb5dbff addr 5000c5009550e4f1
ses2: pass6,da2: Element descriptor: 'Slot02'
ses2: pass6,da2: SAS Device Slot Element: 1 Phys at Slot 2
ses2:  phy 0: SAS device type 1 id 0
ses2:  phy 0: protocols: Initiator( None ) Target( SSP )
ses2:  phy 0: parent 500304801eb5dbff addr 5000c50095518c95
ses2: pass7,da3: Element descriptor: 'Slot03'
ses2: pass7,da3: SAS Device Slot Element: 1 Phys at Slot 3
ses2:  phy 0: SAS device type 1 id 0
ses2:  phy 0: protocols: Initiator( None ) Target( SSP )
ses2:  phy 0: parent 500304801eb5dbff addr 5000c500955fb3c5
ses2: pass8,da4: Element descriptor: 'Slot04'
ses2: pass8,da4: SAS Device Slot Element: 1 Phys at Slot 4
ugen0.3: <vendor 0x0557 product 0x7000> at usbus0
ses2:  phy 0: SAS device type 1 id 0
ses2:  phy 0: protocols: Initiator( None ) Target( SSP )
ses2:  phy 0: parent 500304801eb5dbff addr 5000c500955fb021
ses2: pass9,da5: Element descriptor: 'Slot05'
ses2: pass9,da5: SAS Device Slot Element: 1 Phys at Slot 5
ses2:  phy 0: SAS device type 1 id 0
ses2:  phy 0: protocols: Initiator( None ) Target( SSP )
ses2:  phy 0: parent 500304801eb5dbff addr 5000c5009550e891
uhub1 numa-domain 0 on uhub0
uhub1: <vendor 0x0557 product 0x7000, class 9/0, rev 2.00/0.00, addr 2> on usbus0
Trying to mount root from zfs:tank/default/root []...
Root mount waiting for: usbus0
uhub1: 4 ports with 3 removable, self powered
ugen0.4: <vendor 0x0557 product 0x2419> at usbus0
ukbd1 numa-domain 0 on uhub1
ukbd1: <vendor 0x0557 product 0x2419, class 0/0, rev 1.10/1.00, addr 3> on usbus0
kbd3 at ukbd1
lo0: link state changed to UP
ums0 numa-domain 0 on uhub1
ums0: <vendor 0x0557 product 0x2419, class 0/0, rev 1.10/1.00, addr 3> on usbus0
ums0: 3 buttons and [Z] coordinates ID=0



And pciconf -lvb:


hostb0@pci0:0:0:0: class=0x060000 card=0x098115d9 chip=0x20208086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E DMI3 Registers'
    class      = bridge
    subclass   = HOST-PCI
none0@pci0:0:4:0: class=0x088000 card=0x00008086 chip=0x20218086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CBDMA Registers'
    class      = base peripheral
    bar   [10] = type Memory, range 64, base 0x383ffff2c000, size 16384, enabled
none1@pci0:0:4:1: class=0x088000 card=0x00008086 chip=0x20218086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CBDMA Registers'
    class      = base peripheral
    bar   [10] = type Memory, range 64, base 0x383ffff28000, size 16384, enabled
none2@pci0:0:4:2: class=0x088000 card=0x00008086 chip=0x20218086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CBDMA Registers'
    class      = base peripheral
    bar   [10] = type Memory, range 64, base 0x383ffff24000, size 16384, enabled
none3@pci0:0:4:3: class=0x088000 card=0x00008086 chip=0x20218086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CBDMA Registers'
    class      = base peripheral
    bar   [10] = type Memory, range 64, base 0x383ffff20000, size 16384, enabled
none4@pci0:0:4:4: class=0x088000 card=0x00008086 chip=0x20218086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CBDMA Registers'
    class      = base peripheral
    bar   [10] = type Memory, range 64, base 0x383ffff1c000, size 16384, enabled
none5@pci0:0:4:5: class=0x088000 card=0x00008086 chip=0x20218086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CBDMA Registers'
    class      = base peripheral
    bar   [10] = type Memory, range 64, base 0x383ffff18000, size 16384, enabled
none6@pci0:0:4:6: class=0x088000 card=0x00008086 chip=0x20218086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CBDMA Registers'
    class      = base peripheral
    bar   [10] = type Memory, range 64, base 0x383ffff14000, size 16384, enabled
none7@pci0:0:4:7: class=0x088000 card=0x00008086 chip=0x20218086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CBDMA Registers'
    class      = base peripheral
    bar   [10] = type Memory, range 64, base 0x383ffff10000, size 16384, enabled
none8@pci0:0:5:0: class=0x088000 card=0x00008086 chip=0x20248086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E MM/Vt-d Configuration Registers'
    class      = base peripheral
none9@pci0:0:5:2: class=0x088000 card=0x00000000 chip=0x20258086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
ioapic0@pci0:0:5:4: class=0x080020 card=0x20268086 chip=0x20268086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
    subclass   = interrupt controller
    bar   [10] = type Memory, range 32, base 0x9d20a000, size 4096, enabled
none10@pci0:0:8:0: class=0x088000 card=0x00008086 chip=0x20148086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E Ubox Registers'
    class      = base peripheral
none11@pci0:0:8:1: class=0x110100 card=0x00008086 chip=0x20158086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E Ubox Registers'
    class      = dasp
    subclass   = performance counters
none12@pci0:0:8:2: class=0x088000 card=0x00008086 chip=0x20168086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E Ubox Registers'
    class      = base peripheral
none13@pci0:0:17:0: class=0xff0000 card=0x098115d9 chip=0xa1ec8086 rev=0x09 hdr=0x00
    vendor     = 'Intel Corporation'
none14@pci0:0:17:1: class=0xff0000 card=0x098115d9 chip=0xa1ed8086 rev=0x09 hdr=0x00
    vendor     = 'Intel Corporation'
ahci0@pci0:0:17:5: class=0x010601 card=0x098115d9 chip=0xa1d28086 rev=0x09 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Lewisburg SSATA Controller [AHCI mode]'
    class      = mass storage
    subclass   = SATA
    bar   [10] = type Memory, range 32, base 0x9d206000, size 8192, enabled
    bar   [14] = type Memory, range 32, base 0x9d209000, size 256, enabled
    bar   [18] = type I/O Port, range 32, base 0x3070, size 8, enabled
    bar   [1c] = type I/O Port, range 32, base 0x3060, size 4, enabled
    bar   [20] = type I/O Port, range 32, base 0x3020, size 32, enabled
    bar   [24] = type Memory, range 32, base 0x9d180000, size 524288, enabled
xhci0@pci0:0:20:0: class=0x0c0330 card=0x098115d9 chip=0xa1af8086 rev=0x09 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Lewisburg USB 3.0 xHCI Controller'
    class      = serial bus
    subclass   = USB
    bar   [10] = type Memory, range 64, base 0x383ffff00000, size 65536, enabled
none15@pci0:0:20:2: class=0x118000 card=0x098115d9 chip=0xa1b18086 rev=0x09 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Lewisburg Thermal Subsystem'
    class      = dasp
    bar   [10] = type Memory, range 64, base 0x383ffff33000, size 4096, enabled
none16@pci0:0:22:0: class=0x078000 card=0x098115d9 chip=0xa1ba8086 rev=0x09 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Lewisburg CSME: HECI'
    class      = simple comms
    bar   [10] = type Memory, range 64, base 0x383ffff32000, size 4096, enabled
none17@pci0:0:22:1: class=0x078000 card=0x098115d9 chip=0xa1bb8086 rev=0x09 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Lewisburg CSME: HECI'
    class      = simple comms
    bar   [10] = type Memory, range 64, base 0x383ffff31000, size 4096, enabled
none18@pci0:0:22:4: class=0x078000 card=0x098115d9 chip=0xa1be8086 rev=0x09 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Lewisburg CSME: HECI'
    class      = simple comms
    bar   [10] = type Memory, range 64, base 0x383ffff30000, size 4096, enabled
ahci1@pci0:0:23:0: class=0x010601 card=0x098115d9 chip=0xa1828086 rev=0x09 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Lewisburg SATA Controller [AHCI mode]'
    class      = mass storage
    subclass   = SATA
    bar   [10] = type Memory, range 32, base 0x9d204000, size 8192, enabled
    bar   [14] = type Memory, range 32, base 0x9d208000, size 256, enabled
    bar   [18] = type I/O Port, range 32, base 0x3050, size 8, enabled
    bar   [1c] = type I/O Port, range 32, base 0x3040, size 4, enabled
    bar   [20] = type I/O Port, range 32, base 0x3000, size 32, enabled
    bar   [24] = type Memory, range 32, base 0x9d100000, size 524288, enabled
pcib1@pci0:0:28:0: class=0x060400 card=0x098115d9 chip=0xa1908086 rev=0xf9 hdr=0x01
    vendor     = 'Intel Corporation'
    device     = 'Lewisburg PCI Express Root Port'
    class      = bridge
    subclass   = PCI-PCI
pcib2@pci0:0:28:5: class=0x060400 card=0x098115d9 chip=0xa1958086 rev=0xf9 hdr=0x01
    vendor     = 'Intel Corporation'
    device     = 'Lewisburg PCI Express Root Port'
    class      = bridge
    subclass   = PCI-PCI
isab0@pci0:0:31:0: class=0x060100 card=0x098115d9 chip=0xa1c38086 rev=0x09 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Lewisburg LPC Controller'
    class      = bridge
    subclass   = PCI-ISA
none19@pci0:0:31:2: class=0x058000 card=0x098115d9 chip=0xa1a18086 rev=0x09 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Lewisburg PMC'
    class      = memory
    bar   [10] = type Memory, range 32, base 0x9d200000, size 16384, enabled
none20@pci0:0:31:4: class=0x0c0500 card=0x098115d9 chip=0xa1a38086 rev=0x09 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Lewisburg SMBus'
    class      = serial bus
    subclass   = SMBus
    bar   [10] = type Memory, range 64, base 0, size 256, disabled
    bar   [20] = type I/O Port, range 32, base 0x780, size 32, enabled
none21@pci0:0:31:5: class=0x0c8000 card=0x098115d9 chip=0xa1a48086 rev=0x09 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Lewisburg SPI Controller'
    class      = serial bus
    bar   [10] = type Memory, range 32, base 0xfe010000, size 4096, enabled
pcib3@pci0:2:0:0: class=0x060400 card=0x11501a03 chip=0x11501a03 rev=0x04 hdr=0x01
    vendor     = 'ASPEED Technology, Inc.'
    device     = 'AST1150 PCI-to-PCI Bridge'
    class      = bridge
    subclass   = PCI-PCI
vgapci0@pci0:3:0:0: class=0x030000 card=0x098115d9 chip=0x20001a03 rev=0x41 hdr=0x00
    vendor     = 'ASPEED Technology, Inc.'
    device     = 'ASPEED Graphics Family'
    class      = display
    subclass   = VGA
    bar   [10] = type Memory, range 32, base 0x9c000000, size 16777216, enabled
    bar   [14] = type Memory, range 32, base 0x9d000000, size 131072, enabled
    bar   [18] = type I/O Port, range 32, base 0x2000, size 128, enabled
pcib5@pci0:23:0:0: class=0x060400 card=0x098115d9 chip=0x20308086 rev=0x04 hdr=0x01
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E PCI Express Root Port A'
    class      = bridge
    subclass   = PCI-PCI
none22@pci0:23:5:0: class=0x088000 card=0x00008086 chip=0x20348086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
none23@pci0:23:5:2: class=0x088000 card=0x00000000 chip=0x20358086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E RAS Configuration Registers'
    class      = base peripheral
ioapic1@pci0:23:5:4: class=0x080020 card=0x20368086 chip=0x20368086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
    subclass   = interrupt controller
    bar   [10] = type Memory, range 32, base 0xaaf00000, size 4096, enabled
none24@pci0:23:8:0: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none25@pci0:23:8:1: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none26@pci0:23:8:2: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none27@pci0:23:8:3: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none28@pci0:23:8:4: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none29@pci0:23:8:5: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none30@pci0:23:8:6: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none31@pci0:23:8:7: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none32@pci0:23:9:0: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none33@pci0:23:9:1: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none34@pci0:23:9:2: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none35@pci0:23:9:3: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none36@pci0:23:9:4: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none37@pci0:23:9:5: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none38@pci0:23:9:6: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none39@pci0:23:9:7: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none40@pci0:23:10:0: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none41@pci0:23:10:1: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none42@pci0:23:14:0: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none43@pci0:23:14:1: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none44@pci0:23:14:2: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none45@pci0:23:14:3: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none46@pci0:23:14:4: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none47@pci0:23:14:5: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none48@pci0:23:14:6: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none49@pci0:23:14:7: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none50@pci0:23:15:0: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none51@pci0:23:15:1: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none52@pci0:23:15:2: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none53@pci0:23:15:3: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none54@pci0:23:15:4: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none55@pci0:23:15:5: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none56@pci0:23:15:6: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none57@pci0:23:15:7: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none58@pci0:23:16:0: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none59@pci0:23:16:1: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none60@pci0:23:29:0: class=0x088000 card=0x00008086 chip=0x20548086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none61@pci0:23:29:1: class=0x088000 card=0x00008086 chip=0x20558086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none62@pci0:23:29:2: class=0x088000 card=0x00008086 chip=0x20568086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none63@pci0:23:29:3: class=0x088000 card=0x00008086 chip=0x20578086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none64@pci0:23:30:0: class=0x088000 card=0x00008086 chip=0x20808086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E PCU Registers'
    class      = base peripheral
none65@pci0:23:30:1: class=0x088000 card=0x00008086 chip=0x20818086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E PCU Registers'
    class      = base peripheral
none66@pci0:23:30:2: class=0x088000 card=0x00008086 chip=0x20828086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E PCU Registers'
    class      = base peripheral
none67@pci0:23:30:3: class=0x088000 card=0x00008086 chip=0x20838086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E PCU Registers'
    class      = base peripheral
none68@pci0:23:30:4: class=0x088000 card=0x00008086 chip=0x20848086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E PCU Registers'
    class      = base peripheral
none69@pci0:23:30:5: class=0x088000 card=0x00008086 chip=0x20858086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E PCU Registers'
    class      = base peripheral
none70@pci0:23:30:6: class=0x088000 card=0x00008086 chip=0x20868086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E PCU Registers'
    class      = base peripheral
pcib6@pci0:24:0:0: class=0x060400 card=0xdeadbeef chip=0x37c08086 rev=0x09 hdr=0x01
    vendor     = 'Intel Corporation'
    class      = bridge
    subclass   = PCI-PCI
    bar   [10] = type Memory, range 64, base 0xaae00000, size 131072, enabled
pcib7@pci0:25:3:0: class=0x060400 card=0xdeadbeef chip=0x37c58086 rev=0x09 hdr=0x01
    vendor     = 'Intel Corporation'
    class      = bridge
    subclass   = PCI-PCI
ixl0@pci0:26:0:0: class=0x020000 card=0x37d215d9 chip=0x37d28086 rev=0x09 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Connection X722 for 10GBASE-T'
    class      = network
    subclass   = ethernet
    bar   [10] = type Prefetchable Memory, range 64, base 0xa9000000, size 16777216, enabled
    bar   [1c] = type Prefetchable Memory, range 64, base 0xaa008000, size 32768, enabled
ixl1@pci0:26:0:1: class=0x020000 card=0x37d215d9 chip=0x37d28086 rev=0x09 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Connection X722 for 10GBASE-T'
    class      = network
    subclass   = ethernet
    bar   [10] = type Prefetchable Memory, range 64, base 0xa8000000, size 16777216, enabled
    bar   [1c] = type Prefetchable Memory, range 64, base 0xaa000000, size 32768, enabled
pcib9@pci0:58:2:0: class=0x060400 card=0x098115d9 chip=0x20328086 rev=0x04 hdr=0x01
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E PCI Express Root Port C'
    class      = bridge
    subclass   = PCI-PCI
none71@pci0:58:5:0: class=0x088000 card=0x00008086 chip=0x20348086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
none72@pci0:58:5:2: class=0x088000 card=0x00000000 chip=0x20358086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E RAS Configuration Registers'
    class      = base peripheral
ioapic2@pci0:58:5:4: class=0x080020 card=0x20368086 chip=0x20368086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
    subclass   = interrupt controller
    bar   [10] = type Memory, range 32, base 0xb8700000, size 4096, enabled
none73@pci0:58:8:0: class=0x088000 card=0x00008086 chip=0x20668086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
none74@pci0:58:9:0: class=0x088000 card=0x00008086 chip=0x20668086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
none75@pci0:58:10:0: class=0x088000 card=0x00008086 chip=0x20408086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
none76@pci0:58:10:1: class=0x088000 card=0x00008086 chip=0x20418086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
none77@pci0:58:10:2: class=0x088000 card=0x00008086 chip=0x20428086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
none78@pci0:58:10:3: class=0x088000 card=0x00008086 chip=0x20438086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
none79@pci0:58:10:4: class=0x088000 card=0x00008086 chip=0x20448086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
none80@pci0:58:10:5: class=0x088000 card=0x00008086 chip=0x20458086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
none81@pci0:58:10:6: class=0x088000 card=0x00008086 chip=0x20468086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
none82@pci0:58:10:7: class=0x088000 card=0x00008086 chip=0x20478086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
none83@pci0:58:11:0: class=0x088000 card=0x00008086 chip=0x20488086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
none84@pci0:58:11:1: class=0x088000 card=0x00008086 chip=0x20498086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
none85@pci0:58:11:2: class=0x088000 card=0x00008086 chip=0x204a8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
none86@pci0:58:11:3: class=0x088000 card=0x00008086 chip=0x204b8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
none87@pci0:58:12:0: class=0x088000 card=0x00008086 chip=0x20408086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
none88@pci0:58:12:1: class=0x088000 card=0x00008086 chip=0x20418086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
none89@pci0:58:12:2: class=0x088000 card=0x00008086 chip=0x20428086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
none90@pci0:58:12:3: class=0x088000 card=0x00008086 chip=0x20438086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
none91@pci0:58:12:4: class=0x088000 card=0x00008086 chip=0x20448086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
none92@pci0:58:12:5: class=0x088000 card=0x00008086 chip=0x20458086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
none93@pci0:58:12:6: class=0x088000 card=0x00008086 chip=0x20468086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
none94@pci0:58:12:7: class=0x088000 card=0x00008086 chip=0x20478086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
none95@pci0:58:13:0: class=0x088000 card=0x00008086 chip=0x20488086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
none96@pci0:58:13:1: class=0x088000 card=0x00008086 chip=0x20498086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
none97@pci0:58:13:2: class=0x088000 card=0x00008086 chip=0x204a8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
none98@pci0:58:13:3: class=0x088000 card=0x00008086 chip=0x204b8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
mpr0@pci0:59:0:0: class=0x010700 card=0x080815d9 chip=0x00971000 rev=0x02 hdr=0x00
    vendor     = 'LSI Logic / Symbios Logic'
    device     = 'SAS3008 PCI-Express Fusion-MPT SAS-3'
    class      = mass storage
    subclass   = SAS
    bar   [10] = type I/O Port, range 32, base 0x7000, size 256, enabled
    bar   [14] = type Memory, range 64, base 0xb8640000, size 65536, enabled
    bar   [1c] = type Memory, range 64, base 0xb8600000, size 262144, enabled
none99@pci0:93:5:0: class=0x088000 card=0x00008086 chip=0x20348086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
none100@pci0:93:5:2: class=0x088000 card=0x00000000 chip=0x20358086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E RAS Configuration Registers'
    class      = base peripheral
ioapic3@pci0:93:5:4: class=0x080020 card=0x20368086 chip=0x20368086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
    subclass   = interrupt controller
    bar   [10] = type Memory, range 32, base 0xc5f00000, size 4096, enabled
none101@pci0:93:14:0: class=0x110100 card=0x00008086 chip=0x20588086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = dasp
    subclass   = performance counters
none102@pci0:93:14:1: class=0x088000 card=0x00008086 chip=0x20598086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
none103@pci0:93:15:0: class=0x110100 card=0x00008086 chip=0x20588086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = dasp
    subclass   = performance counters
none104@pci0:93:15:1: class=0x088000 card=0x00008086 chip=0x20598086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
none105@pci0:93:18:0: class=0x110100 card=0x00008086 chip=0x204c8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E M3KTI Registers'
    class      = dasp
    subclass   = performance counters
none106@pci0:93:18:1: class=0x110100 card=0x00008086 chip=0x204d8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E M3KTI Registers'
    class      = dasp
    subclass   = performance counters
none107@pci0:93:18:2: class=0x088000 card=0x00008086 chip=0x204e8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E M3KTI Registers'
    class      = base peripheral
none108@pci0:93:21:0: class=0x088000 card=0x00008086 chip=0x20188086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E M2PCI Registers'
    class      = base peripheral
none109@pci0:93:22:0: class=0x088000 card=0x00008086 chip=0x20188086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E M2PCI Registers'
    class      = base peripheral
none110@pci0:93:22:4: class=0x088000 card=0x00008086 chip=0x20188086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E M2PCI Registers'
    class      = base peripheral
none111@pci0:128:4:0: class=0x088000 card=0x00008086 chip=0x20218086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CBDMA Registers'
    class      = base peripheral
    bar   [10] = type Memory, range 64, base 0x393ffff1c000, size 16384, enabled
none112@pci0:128:4:1: class=0x088000 card=0x00008086 chip=0x20218086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CBDMA Registers'
    class      = base peripheral
    bar   [10] = type Memory, range 64, base 0x393ffff18000, size 16384, enabled
none113@pci0:128:4:2: class=0x088000 card=0x00008086 chip=0x20218086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CBDMA Registers'
    class      = base peripheral
    bar   [10] = type Memory, range 64, base 0x393ffff14000, size 16384, enabled
none114@pci0:128:4:3: class=0x088000 card=0x00008086 chip=0x20218086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CBDMA Registers'
    class      = base peripheral
    bar   [10] = type Memory, range 64, base 0x393ffff10000, size 16384, enabled
none115@pci0:128:4:4: class=0x088000 card=0x00008086 chip=0x20218086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CBDMA Registers'
    class      = base peripheral
    bar   [10] = type Memory, range 64, base 0x393ffff0c000, size 16384, enabled
none116@pci0:128:4:5: class=0x088000 card=0x00008086 chip=0x20218086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CBDMA Registers'
    class      = base peripheral
    bar   [10] = type Memory, range 64, base 0x393ffff08000, size 16384, enabled
none117@pci0:128:4:6: class=0x088000 card=0x00008086 chip=0x20218086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CBDMA Registers'
    class      = base peripheral
    bar   [10] = type Memory, range 64, base 0x393ffff04000, size 16384, enabled
none118@pci0:128:4:7: class=0x088000 card=0x00008086 chip=0x20218086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CBDMA Registers'
    class      = base peripheral
    bar   [10] = type Memory, range 64, base 0x393ffff00000, size 16384, enabled
none119@pci0:128:5:0: class=0x088000 card=0x00008086 chip=0x20248086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E MM/Vt-d Configuration Registers'
    class      = base peripheral
none120@pci0:128:5:2: class=0x088000 card=0x00000000 chip=0x20258086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
ioapic4@pci0:128:5:4: class=0x080020 card=0x20268086 chip=0x20268086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
    subclass   = interrupt controller
    bar   [10] = type Memory, range 32, base 0xd3700000, size 4096, enabled
none121@pci0:128:8:0: class=0x088000 card=0x00008086 chip=0x20148086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E Ubox Registers'
    class      = base peripheral
none122@pci0:128:8:1: class=0x110100 card=0x00008086 chip=0x20158086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E Ubox Registers'
    class      = dasp
    subclass   = performance counters
none123@pci0:128:8:2: class=0x088000 card=0x00008086 chip=0x20168086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E Ubox Registers'
    class      = base peripheral
none124@pci0:133:5:0: class=0x088000 card=0x00008086 chip=0x20348086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
none125@pci0:133:5:2: class=0x088000 card=0x00000000 chip=0x20358086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E RAS Configuration Registers'
    class      = base peripheral
ioapic5@pci0:133:5:4: class=0x080020 card=0x20368086 chip=0x20368086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
    subclass   = interrupt controller
    bar   [10] = type Memory, range 32, base 0xe0f00000, size 4096, enabled
none126@pci0:133:8:0: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none127@pci0:133:8:1: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none128@pci0:133:8:2: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none129@pci0:133:8:3: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none130@pci0:133:8:4: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none131@pci0:133:8:5: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none132@pci0:133:8:6: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none133@pci0:133:8:7: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none134@pci0:133:9:0: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none135@pci0:133:9:1: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none136@pci0:133:9:2: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none137@pci0:133:9:3: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none138@pci0:133:9:4: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none139@pci0:133:9:5: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none140@pci0:133:9:6: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none141@pci0:133:9:7: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none142@pci0:133:10:0: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none143@pci0:133:10:1: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none144@pci0:133:14:0: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none145@pci0:133:14:1: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none146@pci0:133:14:2: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none147@pci0:133:14:3: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none148@pci0:133:14:4: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none149@pci0:133:14:5: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none150@pci0:133:14:6: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none151@pci0:133:14:7: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none152@pci0:133:15:0: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none153@pci0:133:15:1: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none154@pci0:133:15:2: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none155@pci0:133:15:3: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none156@pci0:133:15:4: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none157@pci0:133:15:5: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none158@pci0:133:15:6: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none159@pci0:133:15:7: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none160@pci0:133:16:0: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none161@pci0:133:16:1: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none162@pci0:133:29:0: class=0x088000 card=0x00008086 chip=0x20548086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none163@pci0:133:29:1: class=0x088000 card=0x00008086 chip=0x20558086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none164@pci0:133:29:2: class=0x088000 card=0x00008086 chip=0x20568086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none165@pci0:133:29:3: class=0x088000 card=0x00008086 chip=0x20578086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none166@pci0:133:30:0: class=0x088000 card=0x00008086 chip=0x20808086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E PCU Registers'
    class      = base peripheral
none167@pci0:133:30:1: class=0x088000 card=0x00008086 chip=0x20818086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E PCU Registers'
    class      = base peripheral
none168@pci0:133:30:2: class=0x088000 card=0x00008086 chip=0x20828086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E PCU Registers'
    class      = base peripheral
none169@pci0:133:30:3: class=0x088000 card=0x00008086 chip=0x20838086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E PCU Registers'
    class      = base peripheral
none170@pci0:133:30:4: class=0x088000 card=0x00008086 chip=0x20848086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E PCU Registers'
    class      = base peripheral
none171@pci0:133:30:5: class=0x088000 card=0x00008086 chip=0x20858086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E PCU Registers'
    class      = base peripheral
none172@pci0:133:30:6: class=0x088000 card=0x00008086 chip=0x20868086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E PCU Registers'
    class      = base peripheral
none173@pci0:174:5:0: class=0x088000 card=0x00008086 chip=0x20348086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
none174@pci0:174:5:2: class=0x088000 card=0x00000000 chip=0x20358086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E RAS Configuration Registers'
    class      = base peripheral
ioapic6@pci0:174:5:4: class=0x080020 card=0x20368086 chip=0x20368086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
    subclass   = interrupt controller
    bar   [10] = type Memory, range 32, base 0xee700000, size 4096, enabled
none175@pci0:174:8:0: class=0x088000 card=0x00008086 chip=0x20668086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
none176@pci0:174:9:0: class=0x088000 card=0x00008086 chip=0x20668086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
none177@pci0:174:10:0: class=0x088000 card=0x00008086 chip=0x20408086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
none178@pci0:174:10:1: class=0x088000 card=0x00008086 chip=0x20418086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
none179@pci0:174:10:2: class=0x088000 card=0x00008086 chip=0x20428086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
none180@pci0:174:10:3: class=0x088000 card=0x00008086 chip=0x20438086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
none181@pci0:174:10:4: class=0x088000 card=0x00008086 chip=0x20448086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
none182@pci0:174:10:5: class=0x088000 card=0x00008086 chip=0x20458086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
none183@pci0:174:10:6: class=0x088000 card=0x00008086 chip=0x20468086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
none184@pci0:174:10:7: class=0x088000 card=0x00008086 chip=0x20478086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
none185@pci0:174:11:0: class=0x088000 card=0x00008086 chip=0x20488086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
none186@pci0:174:11:1: class=0x088000 card=0x00008086 chip=0x20498086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
none187@pci0:174:11:2: class=0x088000 card=0x00008086 chip=0x204a8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
none188@pci0:174:11:3: class=0x088000 card=0x00008086 chip=0x204b8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
none189@pci0:174:12:0: class=0x088000 card=0x00008086 chip=0x20408086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
none190@pci0:174:12:1: class=0x088000 card=0x00008086 chip=0x20418086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
none191@pci0:174:12:2: class=0x088000 card=0x00008086 chip=0x20428086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
none192@pci0:174:12:3: class=0x088000 card=0x00008086 chip=0x20438086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
none193@pci0:174:12:4: class=0x088000 card=0x00008086 chip=0x20448086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
none194@pci0:174:12:5: class=0x088000 card=0x00008086 chip=0x20458086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
none195@pci0:174:12:6: class=0x088000 card=0x00008086 chip=0x20468086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
none196@pci0:174:12:7: class=0x088000 card=0x00008086 chip=0x20478086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
none197@pci0:174:13:0: class=0x088000 card=0x00008086 chip=0x20488086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
none198@pci0:174:13:1: class=0x088000 card=0x00008086 chip=0x20498086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
none199@pci0:174:13:2: class=0x088000 card=0x00008086 chip=0x204a8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
none200@pci0:174:13:3: class=0x088000 card=0x00008086 chip=0x204b8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
none201@pci0:215:5:0: class=0x088000 card=0x00008086 chip=0x20348086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
none202@pci0:215:5:2: class=0x088000 card=0x00000000 chip=0x20358086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E RAS Configuration Registers'
    class      = base peripheral
ioapic7@pci0:215:5:4: class=0x080020 card=0x20368086 chip=0x20368086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
    subclass   = interrupt controller
    bar   [10] = type Memory, range 32, base 0xfbf00000, size 4096, enabled
none203@pci0:215:14:0: class=0x110100 card=0x00008086 chip=0x20588086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = dasp
    subclass   = performance counters
none204@pci0:215:14:1: class=0x088000 card=0x00008086 chip=0x20598086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
none205@pci0:215:15:0: class=0x110100 card=0x00008086 chip=0x20588086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = dasp
    subclass   = performance counters
none206@pci0:215:15:1: class=0x088000 card=0x00008086 chip=0x20598086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
none207@pci0:215:18:0: class=0x110100 card=0x00008086 chip=0x204c8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E M3KTI Registers'
    class      = dasp
    subclass   = performance counters
none208@pci0:215:18:1: class=0x110100 card=0x00008086 chip=0x204d8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E M3KTI Registers'
    class      = dasp
    subclass   = performance counters
none209@pci0:215:18:2: class=0x088000 card=0x00008086 chip=0x204e8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E M3KTI Registers'
    class      = base peripheral
none210@pci0:215:21:0: class=0x088000 card=0x00008086 chip=0x20188086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E M2PCI Registers'
    class      = base peripheral
none211@pci0:215:22:0: class=0x088000 card=0x00008086 chip=0x20188086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E M2PCI Registers'
    class      = base peripheral
none212@pci0:215:22:4: class=0x088000 card=0x00008086 chip=0x20188086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E M2PCI Registers'
    class      = base peripheral


# grep swap /etc/fstab ; echo
#/dev/mirror/swap                       none            swap    sw
/dev/gpt/swap0                  none            swap    sw
#/dev/gpt/swap1                 none            swap    sw

# swapinfo -h
Device          1024-blocks     Used    Avail Capacity
/dev/gpt/swap0     16777216       0B      16G     0%

# ls -ltr /var/crash/
total 3844652
-rw-r--r--  1 root  wheel           5 Jun 21 21:34 minfree
-rw-------  1 root  wheel         462 Jul  5 11:46 info.0
-rw-------  1 root  wheel  5043531776 Jul  5 11:47 vmcore.0
-rw-------  1 root  wheel      128912 Jul  5 11:47 core.txt.0
-rw-------  1 root  wheel         464 Jul  6 12:24 info.1
-rw-------  1 root  wheel  3628683264 Jul  6 12:24 vmcore.1
-rw-------  1 root  wheel      127947 Jul  6 12:24 core.txt.1
-rw-r--r--  1 root  wheel         639 Jul  9 12:51 ls.out.0
-rw-r--r--  1 root  wheel        3960 Jul 11 06:36 zpool-iostat.out
-rw-------  1 root  wheel         464 Jul 11 12:47 info.2
-rw-------  1 root  wheel  3777585152 Jul 11 12:48 vmcore.2
-rw-------  1 root  wheel      127775 Jul 11 12:48 core.txt.2
-rw-r--r--  1 root  wheel           0 Jul 11 13:12 zfs-stress.start
-rw-r--r--  1 root  wheel           2 Jul 11 13:37 bounds
-rw-------  1 root  wheel         458 Jul 11 13:37 info.3
-rw-------  1 root  wheel  7352229888 Jul 11 13:38 vmcore.3
lrwxr-xr-x  1 root  wheel           6 Jul 11 13:38 info.last@ -> info.3
lrwxr-xr-x  1 root  wheel           8 Jul 11 13:38 vmcore.last@ -> vmcore.3
-rw-------  1 root  wheel      143973 Jul 11 13:38 core.txt.3

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Disk/ZFS activity crash on 11.2-STABLE

mdtancsa
On 7/11/2018 5:29 PM, Jim Long wrote:
> Further, crash dumps of this are difficult to obtain (see below),
> despite the fact that dumps are enabled and tested via:
> while also running 'systat -vm' in one virtual terminal and writing
> 'zpool iostat -v' to disk every couple seconds.  I did finally observe a
> panic and this time got a dump.

prior to the crash, if you run gstat does it show the disks at 100% load ?

You have a lot of RAM, is ARC by chance using all of it ? Try putting a
hard limit on it. So if you have 98G of RAM, place a limit at something
like 70G for now.
Also, what is the file system like ? Is it a few big files or MANY
little files.  Post the output of zfs-stats -a

>
> One quirk I forgot to mention is that after this ZFS-induced crash, on the
> next boot, the /etc/rc somehow does not run dhclient on the primary
> interface, despite its presence in /etc/rc.conf:



>
> ifconfig_ixl0="DHCP"

Thats odd.  does
dmesg -a
show anything about ixl0 and dhclient ?

Perhaps in /usr/local/etc/rc.d/
put a small script like "dhclient-safety.sh"


in in it have something like
#!/bin/sh

LOGGER=/usr/bin/logger
case "$1" in
start)
        /bin/ps -auxww | /usr/bin/grep dhclient |  $LOGGER
        /sbin/ifconfig -v ixl0 | $LOGGER
        killall dhclient
        /sbin/ifconfig  ixl0 down
        sleep 1
        /sbin/ifconfig  ixl0 up
        /sbin/dhclient ixl0 &
        ;;
esac



... At least this way you are not locked out post crash.  And speaking
of crash, post the contents of the
core.txt.3

file in /var/crash

        ---Mike

>
>
>
> As promised, dmesg:
>
>
> Copyright (c) 1992-2018 The FreeBSD Project.
> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
> The Regents of the University of California. All rights reserved.
> FreeBSD is a registered trademark of The FreeBSD Foundation.
> FreeBSD 11.2-STABLE #0 r335674: Tue Jun 26 13:20:24 PDT 2018
>     [hidden email]:/usr/obj/usr/src/sys/GENERIC amd64
> FreeBSD clang version 6.0.0 (tags/RELEASE_600/final 326565) (based on LLVM 6.0.0)
> VT(efifb): resolution 1680x1050
> CPU: Intel(R) Xeon(R) Gold 5118 CPU @ 2.30GHz (2300.08-MHz K8-class CPU)
>   Origin="GenuineIntel"  Id=0x50654  Family=0x6  Model=0x55  Stepping=4
>   Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
>   Features2=0x7ffefbff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
>   AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
>   AMD Features2=0x121<LAHF,ABM,Prefetch>
>   Structured Extended Features=0xd39ffffb<FSGSBASE,TSCADJ,BMI1,HLE,AVX2,FDPEXC,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,NFPUSG,MPX,PQE,AVX512F,AVX512DQ,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,PROCTRACE,AVX512CD,AVX512BW,AVX512VL>
>   Structured Extended Features2=0x8<PKU>
>   Structured Extended Features3=0xc000000<IBPB,STIBP>
>   XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
>   VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
>   TSC: P-state invariant, performance statistics
> real memory  = 103079215104 (98304 MB)
> avail memory = 98749128704 (94174 MB)
> Event timer "LAPIC" quality 600
> ACPI APIC Table: <SUPERM SMCI--MB>
> FreeBSD/SMP: Multiprocessor System Detected: 48 CPUs
> FreeBSD/SMP: 2 package(s) x 12 core(s) x 2 hardware threads
> ioapic6: Changing APIC ID to 16
> ioapic7: Changing APIC ID to 17
> ioapic8: Changing APIC ID to 18
> ioapic0 <Version 2.0> irqs 0-23 on motherboard
> ioapic1 <Version 2.0> irqs 24-31 on motherboard
> ioapic2 <Version 2.0> irqs 32-39 on motherboard
> ioapic3 <Version 2.0> irqs 40-47 on motherboard
> ioapic4 <Version 2.0> irqs 48-55 on motherboard
> ioapic5 <Version 2.0> irqs 72-79 on motherboard
> ioapic6 <Version 2.0> irqs 80-87 on motherboard
> ioapic7 <Version 2.0> irqs 88-95 on motherboard
> ioapic8 <Version 2.0> irqs 96-103 on motherboard
> SMP: AP CPU #1 Launched!
> SMP: AP CPU #38 Launched!
> SMP: AP CPU #2 Launched!
> SMP: AP CPU #16 Launched!
> SMP: AP CPU #12 Launched!
> SMP: AP CPU #6 Launched!
> SMP: AP CPU #17 Launched!
> SMP: AP CPU #40 Launched!
> SMP: AP CPU #29 Launched!
> SMP: AP CPU #5 Launched!
> SMP: AP CPU #27 Launched!
> SMP: AP CPU #19 Launched!
> SMP: AP CPU #13 Launched!
> SMP: AP CPU #22 Launched!
> SMP: AP CPU #4 Launched!
> SMP: AP CPU #28 Launched!
> SMP: AP CPU #30 Launched!
> SMP: AP CPU #11 Launched!
> SMP: AP CPU #42 Launched!
> SMP: AP CPU #32 Launched!
> SMP: AP CPU #3 Launched!
> SMP: AP CPU #34 Launched!
> SMP: AP CPU #14 Launched!
> SMP: AP CPU #37 Launched!
> SMP: AP CPU #20 Launched!
> SMP: AP CPU #41 Launched!
> SMP: AP CPU #21 Launched!
> SMP: AP CPU #18 Launched!
> SMP: AP CPU #35 Launched!
> SMP: AP CPU #23 Launched!
> SMP: AP CPU #10 Launched!
> SMP: AP CPU #36 Launched!
> SMP: AP CPU #31 Launched!
> SMP: AP CPU #43 Launched!
> SMP: AP CPU #44 Launched!
> SMP: AP CPU #45 Launched!
> SMP: AP CPU #39 Launched!
> SMP: AP CPU #8 Launched!
> SMP: AP CPU #24 Launched!
> SMP: AP CPU #25 Launched!
> SMP: AP CPU #46 Launched!
> SMP: AP CPU #9 Launched!
> SMP: AP CPU #26 Launched!
> SMP: AP CPU #33 Launched!
> SMP: AP CPU #15 Launched!
> SMP: AP CPU #7 Launched!
> SMP: AP CPU #47 Launched!
> Timecounter "TSC-low" frequency 1150040467 Hz quality 1000
> random: entropy device external interface
> kbd1 at kbdmux0
> netmap: loaded module
> module_register_init: MOD_LOAD (vesa, 0xffffffff80ff4590, 0) error 19
> random: registering fast source Intel Secure Key RNG
> random: fast provider: "Intel Secure Key RNG"
> nexus0
> cryptosoft0: <software crypto> on motherboard
> acpi0: <SUPERM SUPERM> on motherboard
> acpi0: Power Button (fixed)
> cpu0: <ACPI CPU> numa-domain 0 on acpi0
> cpu1: <ACPI CPU> numa-domain 0 on acpi0
> cpu2: <ACPI CPU> numa-domain 0 on acpi0
> cpu3: <ACPI CPU> numa-domain 0 on acpi0
> cpu4: <ACPI CPU> numa-domain 0 on acpi0
> cpu5: <ACPI CPU> numa-domain 0 on acpi0
> cpu6: <ACPI CPU> numa-domain 0 on acpi0
> cpu7: <ACPI CPU> numa-domain 0 on acpi0
> cpu8: <ACPI CPU> numa-domain 0 on acpi0
> cpu9: <ACPI CPU> numa-domain 0 on acpi0
> cpu10: <ACPI CPU> numa-domain 0 on acpi0
> cpu11: <ACPI CPU> numa-domain 0 on acpi0
> cpu12: <ACPI CPU> numa-domain 0 on acpi0
> cpu13: <ACPI CPU> numa-domain 0 on acpi0
> cpu14: <ACPI CPU> numa-domain 0 on acpi0
> cpu15: <ACPI CPU> numa-domain 0 on acpi0
> cpu16: <ACPI CPU> numa-domain 0 on acpi0
> cpu17: <ACPI CPU> numa-domain 0 on acpi0
> cpu18: <ACPI CPU> numa-domain 0 on acpi0
> cpu19: <ACPI CPU> numa-domain 0 on acpi0
> cpu20: <ACPI CPU> numa-domain 0 on acpi0
> cpu21: <ACPI CPU> numa-domain 0 on acpi0
> cpu22: <ACPI CPU> numa-domain 0 on acpi0
> cpu23: <ACPI CPU> numa-domain 0 on acpi0
> cpu24: <ACPI CPU> numa-domain 1 on acpi0
> cpu25: <ACPI CPU> numa-domain 1 on acpi0
> cpu26: <ACPI CPU> numa-domain 1 on acpi0
> cpu27: <ACPI CPU> numa-domain 1 on acpi0
> cpu28: <ACPI CPU> numa-domain 1 on acpi0
> cpu29: <ACPI CPU> numa-domain 1 on acpi0
> cpu30: <ACPI CPU> numa-domain 1 on acpi0
> cpu31: <ACPI CPU> numa-domain 1 on acpi0
> cpu32: <ACPI CPU> numa-domain 1 on acpi0
> cpu33: <ACPI CPU> numa-domain 1 on acpi0
> cpu34: <ACPI CPU> numa-domain 1 on acpi0
> cpu35: <ACPI CPU> numa-domain 1 on acpi0
> cpu36: <ACPI CPU> numa-domain 1 on acpi0
> cpu37: <ACPI CPU> numa-domain 1 on acpi0
> cpu38: <ACPI CPU> numa-domain 1 on acpi0
> cpu39: <ACPI CPU> numa-domain 1 on acpi0
> cpu40: <ACPI CPU> numa-domain 1 on acpi0
> cpu41: <ACPI CPU> numa-domain 1 on acpi0
> cpu42: <ACPI CPU> numa-domain 1 on acpi0
> cpu43: <ACPI CPU> numa-domain 1 on acpi0
> cpu44: <ACPI CPU> numa-domain 1 on acpi0
> cpu45: <ACPI CPU> numa-domain 1 on acpi0
> cpu46: <ACPI CPU> numa-domain 1 on acpi0
> cpu47: <ACPI CPU> numa-domain 1 on acpi0
> atrtc0: <AT realtime clock> port 0x70-0x71,0x74-0x77 irq 8 on acpi0
> atrtc0: registered as a time-of-day clock, resolution 1.000000s
> Event timer "RTC" frequency 32768 Hz quality 0
> attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
> Timecounter "i8254" frequency 1193182 Hz quality 0
> Event timer "i8254" frequency 1193182 Hz quality 100
> hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
> Timecounter "HPET" frequency 24000000 Hz quality 950
> Event timer "HPET" frequency 24000000 Hz quality 350
> Event timer "HPET1" frequency 24000000 Hz quality 340
> Event timer "HPET2" frequency 24000000 Hz quality 340
> Event timer "HPET3" frequency 24000000 Hz quality 340
> Event timer "HPET4" frequency 24000000 Hz quality 340
> Event timer "HPET5" frequency 24000000 Hz quality 340
> Event timer "HPET6" frequency 24000000 Hz quality 340
> Event timer "HPET7" frequency 24000000 Hz quality 340
> Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
> acpi_timer0: <24-bit timer at 3.579545MHz> port 0x508-0x50b on acpi0
> acpi_syscontainer0: <System Container> on acpi0
> acpi_syscontainer1: <System Container> on acpi0
> pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff numa-domain 0 on acpi0
> pcib0: _OSC returned error 0x10
> pci0: <ACPI PCI bus> numa-domain 0 on pcib0
> pci0: <dasp, performance counters> at device 8.1 (no driver attached)
> pci0: <unknown> at device 17.0 (no driver attached)
> pci0: <unknown> at device 17.1 (no driver attached)
> ahci0: <Intel Lewisburg AHCI SATA controller> port 0x3070-0x3077,0x3060-0x3063,0x3020-0x303f mem 0x9d206000-0x9d207fff,0x9d209000-0x9d2090ff,0x9d180000-0x9d1fffff irq 16 at device 17.5 numa-domain 0 on pci0
> ahci0: AHCI v1.31 with 6 6Gbps ports, Port Multiplier not supported
> ahcich0: <AHCI channel> at channel 0 on ahci0
> ahcich1: <AHCI channel> at channel 1 on ahci0
> ahcich2: <AHCI channel> at channel 2 on ahci0
> ahcich3: <AHCI channel> at channel 3 on ahci0
> ahcich4: <AHCI channel> at channel 4 on ahci0
> ahcich5: <AHCI channel> at channel 5 on ahci0
> ahciem0: <AHCI enclosure management bridge> on ahci0
> xhci0: <Intel Lewisburg USB 3.0 controller> mem 0x383ffff00000-0x383ffff0ffff irq 16 at device 20.0 numa-domain 0 on pci0
> xhci0: 32 bytes context size, 64-bit DMA
> usbus0: waiting for BIOS to give up control
> xhci_interrupt: host controller halted
> usbus0 numa-domain 0 on xhci0
> usbus0: 5.0Gbps Super Speed USB v3.0
> pci0: <simple comms> at device 22.0 (no driver attached)
> pci0: <simple comms> at device 22.1 (no driver attached)
> pci0: <simple comms> at device 22.4 (no driver attached)
> ahci1: <Intel Lewisburg AHCI SATA controller> port 0x3050-0x3057,0x3040-0x3043,0x3000-0x301f mem 0x9d204000-0x9d205fff,0x9d208000-0x9d2080ff,0x9d100000-0x9d17ffff irq 16 at device 23.0 numa-domain 0 on pci0
> ahci1: AHCI v1.31 with 8 6Gbps ports, Port Multiplier not supported
> ahcich6: <AHCI channel> at channel 0 on ahci1
> ahcich7: <AHCI channel> at channel 1 on ahci1
> ahcich8: <AHCI channel> at channel 2 on ahci1
> ahcich9: <AHCI channel> at channel 3 on ahci1
> ahcich10: <AHCI channel> at channel 4 on ahci1
> ahcich11: <AHCI channel> at channel 5 on ahci1
> ahcich12: <AHCI channel> at channel 6 on ahci1
> ahcich13: <AHCI channel> at channel 7 on ahci1
> ahciem1: <AHCI enclosure management bridge> on ahci1
> pcib1: <ACPI PCI-PCI bridge> irq 16 at device 28.0 numa-domain 0 on pci0
> pci1: <ACPI PCI bus> numa-domain 0 on pcib1
> pcib2: <ACPI PCI-PCI bridge> irq 17 at device 28.5 numa-domain 0 on pci0
> pci2: <ACPI PCI bus> numa-domain 0 on pcib2
> pcib3: <ACPI PCI-PCI bridge> irq 17 at device 0.0 numa-domain 0 on pci2
> pci3: <ACPI PCI bus> numa-domain 0 on pcib3
> vgapci0: <VGA-compatible display> port 0x2000-0x207f mem 0x9c000000-0x9cffffff,0x9d000000-0x9d01ffff irq 17 at device 0.0 numa-domain 0 on pci3
> vgapci0: Boot video device
> isab0: <PCI-ISA bridge> at device 31.0 numa-domain 0 on pci0
> isa0: <ISA bus> numa-domain 0 on isab0
> pci0: <memory> at device 31.2 (no driver attached)
> pci0: <serial bus> at device 31.5 (no driver attached)
> pcib4: <ACPI Host-PCI bridge> numa-domain 0 on acpi0
> pcib4: _OSC returned error 0x10
> pci4: <ACPI PCI bus> numa-domain 0 on pcib4
> pcib5: <ACPI PCI-PCI bridge> irq 39 at device 0.0 numa-domain 0 on pci4
> pci5: <ACPI PCI bus> numa-domain 0 on pcib5
> pcib6: <ACPI PCI-PCI bridge> mem 0xaae00000-0xaae1ffff irq 32 at device 0.0 numa-domain 0 on pci5
> pci6: <ACPI PCI bus> numa-domain 0 on pcib6
> pcib7: <PCI-PCI bridge> irq 38 at device 3.0 numa-domain 0 on pci6
> pci7: <PCI bus> numa-domain 0 on pcib7
> ixl0: <Intel(R) Ethernet Connection 700 Series PF Driver, Version - 1.9.9-k> mem 0xa9000000-0xa9ffffff,0xaa008000-0xaa00ffff irq 38 at device 0.0 numa-domain 0 on pci7
> ixl0: using 1024 tx descriptors and 1024 rx descriptors
> ixl0: fw 3.1.52349 api 1.5 nvm 3.25 etid 800009e7 oem 1.262.0
> ixl0: PF-ID[0]: VFs 32, MSIX 129, VF MSIX 5, QPs 768, MDIO shared
> ixl0: Using MSIX interrupts with 9 vectors
> ixl0: Allocating 8 queues for PF LAN VSI; 8 queues active
> ixl0: Ethernet address: ac:1f:6b:61:54:06
> ixl0: SR-IOV ready
> queues is 0xfffffe0001d13000
> ixl0: netmap queues/slots: TX 8/1024, RX 8/1024
> ixl1: <Intel(R) Ethernet Connection 700 Series PF Driver, Version - 1.9.9-k> mem 0xa8000000-0xa8ffffff,0xaa000000-0xaa007fff irq 38 at device 0.1 numa-domain 0 on pci7
> ixl1: using 1024 tx descriptors and 1024 rx descriptors
> ixl1: fw 3.1.52349 api 1.5 nvm 3.25 etid 800009e7 oem 1.262.0
> ixl1: PF-ID[1]: VFs 32, MSIX 129, VF MSIX 5, QPs 768, MDIO shared
> ixl1: Using MSIX interrupts with 9 vectors
> ixl1: Allocating 8 queues for PF LAN VSI; 8 queues active
> ixl1: Ethernet address: ac:1f:6b:61:54:07
> ixl1: SR-IOV ready
> queues is 0xfffffe0001e69000
> ixl1: netmap queues/slots: TX 8/1024, RX 8/1024
> pcib8: <ACPI Host-PCI bridge> numa-domain 0 on acpi0
> pcib8: _OSC returned error 0x10
> pci8: <ACPI PCI bus> numa-domain 0 on pcib8
> pcib9: <ACPI PCI-PCI bridge> irq 47 at device 2.0 numa-domain 0 on pci8
> pci9: <ACPI PCI bus> numa-domain 0 on pcib9
> mpr0: <Avago Technologies (LSI) SAS3008> port 0x7000-0x70ff mem 0xb8640000-0xb864ffff,0xb8600000-0xb863ffff irq 42 at device 0.0 numa-domain 0 on pci9
> mpr0: Firmware: 15.00.03.00, Driver: 18.03.00.00-fbsd
> mpr0: IOCCapabilities: 7a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc,FastPath,RDPQArray>
> pcib10: <ACPI Host-PCI bridge> numa-domain 0 on acpi0
> pcib10: _OSC returned error 0x10
> pci10: <ACPI PCI bus> numa-domain 0 on pcib10
> pci10: <dasp, performance counters> at device 14.0 (no driver attached)
> pci10: <dasp, performance counters> at device 15.0 (no driver attached)
> pci10: <dasp, performance counters> at device 18.0 (no driver attached)
> pci10: <dasp, performance counters> at device 18.1 (no driver attached)
> pcib11: <ACPI Host-PCI bridge> numa-domain 1 on acpi0
> ixl0: Link is up, 1 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: None, Autoneg: True, Flow Control: None
> ixl0: link state changed to UP
> ixl1: Link is up, 1 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: None, Autoneg: True, Flow Control: Full
> ixl1: link state changed to UP
> pcib11: _OSC returned error 0x10
> pci11: <ACPI PCI bus> numa-domain 1 on pcib11
> pci11: <dasp, performance counters> at device 8.1 (no driver attached)
> pcib12: <ACPI Host-PCI bridge> numa-domain 1 on acpi0
> pcib12: _OSC returned error 0x10
> pci12: <ACPI PCI bus> numa-domain 1 on pcib12
> pcib13: <ACPI Host-PCI bridge> numa-domain 1 on acpi0
> pcib13: _OSC returned error 0x10
> pci13: <ACPI PCI bus> numa-domain 1 on pcib13
> pcib14: <ACPI Host-PCI bridge> numa-domain 1 on acpi0
> pcib14: _OSC returned error 0x10
> pci14: <ACPI PCI bus> numa-domain 1 on pcib14
> pci14: <dasp, performance counters> at device 14.0 (no driver attached)
> pci14: <dasp, performance counters> at device 15.0 (no driver attached)
> pci14: <dasp, performance counters> at device 18.0 (no driver attached)
> pci14: <dasp, performance counters> at device 18.1 (no driver attached)
> uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
> uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
> orm0: <ISA Option ROM> at iomem 0xc0000-0xc7fff on isa0
> ppc0: cannot reserve I/O port range
> est0: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu0
> est1: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu1
> est2: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu2
> est3: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu3
> est4: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu4
> est5: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu5
> est6: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu6
> est7: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu7
> est8: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu8
> est9: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu9
> est10: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu10
> est11: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu11
> est12: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu12
> est13: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu13
> est14: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu14
> est15: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu15
> est16: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu16
> est17: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu17
> est18: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu18
> est19: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu19
> est20: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu20
> est21: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu21
> est22: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu22
> est23: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu23
> est24: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu24
> est25: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu25
> est26: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu26
> est27: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu27
> est28: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu28
> est29: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu29
> est30: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu30
> est31: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu31
> est32: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu32
> est33: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu33
> est34: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu34
> est35: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu35
> est36: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu36
> est37: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu37
> est38: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu38
> est39: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu39
> est40: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu40
> est41: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu41
> est42: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu42
> est43: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu43
> est44: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu44
> est45: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu45
> est46: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu46
> est47: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu47
> random: unblocking device.
> ZFS filesystem version: 5
> ZFS storage pool version: features support (5000)
> Timecounters tick every 1.000 msec
> ugen0.1: <0x8086 XHCI root HUB> at usbus0
> mpr0: uhub0: <0x8086 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
> SAS Address from SAS device page0 = 5000c500955fb571
> mpr0: Found device <401<SspTarg>,End Device> <12.0Gbps> handle<0x000a> enclosureHandle<0x0002> slot 0
> mpr0: At enclosure level 0 and connector name (    )
> mpr0: SAS Address from SAS device page0 = 5000c5009550e4f1
> mpr0: Found device <401<SspTarg>,End Device> <12.0Gbps> handle<0x000b> enclosureHandle<0x0002> slot 1
> mpr0: At enclosure level 0 and connector name (    )
> mpr0: SAS Address from SAS device page0 = 5000c50095518c95
> mpr0: Found device <401<SspTarg>,End Device> <12.0Gbps> handle<0x000c> enclosureHandle<0x0002> slot 2
> mpr0: At enclosure level 0 and connector name (    )
> mpr0: SAS Address from SAS device page0 = 5000c500955fb3c5
> mpr0: Found device <401<SspTarg>,End Device> <12.0Gbps> handle<0x000d> enclosureHandle<0x0002> slot 3
> mpr0: At enclosure level 0 and connector name (    )
> mpr0: SAS Address from SAS device page0 = 5000c500955fb021
> mpr0: Found device <401<SspTarg>,End Device> <12.0Gbps> handle<0x000e> enclosureHandle<0x0002> slot 4
> mpr0: At enclosure level 0 and connector name (    )
> mpr0: SAS Address from SAS device page0 = 5000c5009550e891
> mpr0: Found device <401<SspTarg>,End Device> <12.0Gbps> handle<0x000f> enclosureHandle<0x0002> slot 5
> mpr0: At enclosure level 0 and connector name (    )
> mpr0: SAS Address from SAS device page0 = 500304801eb5dbfd
> mpr0: Found device <4411<SmpInit,SspTarg,SepDev>,End Device> <12.0Gbps> handle<0x0010> enclosureHandle<0x0002> slot 12
> mpr0: At enclosure level 0 and connector name (    )
> ses0 at ahciem0 bus 0 scbus6 target 0 lun 0
> ses0: <AHCI SGPIO Enclosure 1.00 0001> SEMB S-E-S 2.00 device
> ses0: SEMB SES Device
> ses1 at ahciem1 bus 0 scbus15 target 0 lun 0
> ses1: <AHCI SGPIO Enclosure 1.00 0001> SEMB S-E-S 2.00 device
> ses1: SEMB SES Device
> ses2 at mpr0 bus 0 scbus16 target 20 lun 0
> ses2: <LSI SAS3x28 0601> Fixed Enclosure Services SPC-3 SCSI device
> ses2: 1200.000MB/s transfers
> ses2: Command Queueing enabled
> ses2: SCSI-3 ENC Device
> ada0 at ahcich4 bus 0 scbus4 target 0 lun 0
> ada0: <INTEL SSDSC2KB240G7 SCV10121> ACS-3 ATA SATA 3.x device
> ada0: Serial Number BTYS81150CWH240AGN
> ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 512bytes)
> ada0: Command Queueing enabled
> ada0: 228936MB (468862128 512 byte sectors)
> ada1 at ahcich5 bus 0 scbus5 target 0 lun 0
> ada1: <INTEL SSDSC2KB240G7 SCV10121> ACS-3 ATA SATA 3.x device
> ada1: Serial Number BTYS814205JS240AGN
> ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 512bytes)
> ada1: Command Queueing enabled
> ada1: 228936MB (468862128 512 byte sectors)
> uhub0: 26 ports with 26 removable, self powered
> ses2: pass4,da0: Element descriptor: 'Slot00'
> da1 at mpr0 bus 0 scbus16 target 9 lun 0
> da2 at mpr0 bus 0 scbus16 target 10 lun 0
> da0 at mpr0 bus 0 scbus16 target 8 lun 0
> da5 at mpr0 bus 0 scbus16 target 13 lun 0
> da4 at mpr0 bus 0 scbus16 target 12 lun 0
> da2: <SEAGATE ST4000NM0095 K004> Fixed Direct Access SPC-4 SCSI device
> da2: Serial Number ZC15T3GB0000C832EM0C
> da2: 1200.000MB/s transfers
> da2: Command Queueing enabled
> da2: 3815447MB (976754646 4096 byte sectors)
> da0: <SEAGATE ST4000NM0095 K004> Fixed Direct Access SPC-4 SCSI device
> da0: Serial Number ZC1606NJ0000R833G9JK
> da0: 1200.000MB/s transfers
> da0: Command Queueing enabled
> da0: 3815447MB (976754646 4096 byte sectors)
> da4: <SEAGATE ST4000NM0095 K004> Fixed Direct Access SPC-4 SCSI device
> da4: Serial Number ZC1606RR0000R833G8YR
> da4: 1200.000MB/s transfers
> da4: Command Queueing enabled
> da4: 3815447MB (976754646 4096 byte sectors)
> ses2: pass4,da0: SAS Device Slot Element: 1 Phys at Slot 0
> ses2:  phy 0: SAS device type 1 id 0
> da5: da1: <SEAGATE ST4000NM0095 K004> Fixed Direct Access SPC-4 SCSI device
> da1: Serial Number ZC15V7DV0000C8258M38
> ugen0.2: <Dell Dell USB Entry Keyboard> at usbus0
> da1: 1200.000MB/s transfers
> da1: Command Queueing enabled
> da1: 3815447MB (976754646 4096 byte sectors)
> <SEAGATE ST4000NM0095 K004> Fixed Direct Access SPC-4 SCSI device
> ukbd0da5: Serial Number ZC15KB320000R832RAR7
> da5: 1200.000MB/s transfers
> da5: Command Queueing enabled
> da5: 3815447MB (976754646 4096 byte sectors)
>  numa-domain 0 on uhub0
> da3 at mpr0 bus 0 scbus16 target 11 lun 0
> ukbd0: <Dell Dell USB Entry Keyboard, class 0/0, rev 1.10/1.15, addr 1> on usbus0
> kbd2 at ukbd0
> ses2:  phy 0: protocols: Initiator( None ) Target( SSP )
> ses2:  phy 0: parent 500304801eb5dbff addr 5000c500955fb571
> da3: <SEAGATE ST4000NM0095 K004> Fixed Direct Access SPC-4 SCSI device
> da3: Serial Number ZC1606Q80000R833WKE8
> da3: 1200.000MB/s transfers
> da3: Command Queueing enabled
> da3: 3815447MB (976754646 4096 byte sectors)
> ses2: pass5,da1: Element descriptor: 'Slot01'
> ses2: pass5,da1: SAS Device Slot Element: 1 Phys at Slot 1
> ses2:  phy 0: SAS device type 1 id 0
> ses2:  phy 0: protocols: Initiator( None ) Target( SSP )
> ses2:  phy 0: parent 500304801eb5dbff addr 5000c5009550e4f1
> ses2: pass6,da2: Element descriptor: 'Slot02'
> ses2: pass6,da2: SAS Device Slot Element: 1 Phys at Slot 2
> ses2:  phy 0: SAS device type 1 id 0
> ses2:  phy 0: protocols: Initiator( None ) Target( SSP )
> ses2:  phy 0: parent 500304801eb5dbff addr 5000c50095518c95
> ses2: pass7,da3: Element descriptor: 'Slot03'
> ses2: pass7,da3: SAS Device Slot Element: 1 Phys at Slot 3
> ses2:  phy 0: SAS device type 1 id 0
> ses2:  phy 0: protocols: Initiator( None ) Target( SSP )
> ses2:  phy 0: parent 500304801eb5dbff addr 5000c500955fb3c5
> ses2: pass8,da4: Element descriptor: 'Slot04'
> ses2: pass8,da4: SAS Device Slot Element: 1 Phys at Slot 4
> ugen0.3: <vendor 0x0557 product 0x7000> at usbus0
> ses2:  phy 0: SAS device type 1 id 0
> ses2:  phy 0: protocols: Initiator( None ) Target( SSP )
> ses2:  phy 0: parent 500304801eb5dbff addr 5000c500955fb021
> ses2: pass9,da5: Element descriptor: 'Slot05'
> ses2: pass9,da5: SAS Device Slot Element: 1 Phys at Slot 5
> ses2:  phy 0: SAS device type 1 id 0
> ses2:  phy 0: protocols: Initiator( None ) Target( SSP )
> ses2:  phy 0: parent 500304801eb5dbff addr 5000c5009550e891
> uhub1 numa-domain 0 on uhub0
> uhub1: <vendor 0x0557 product 0x7000, class 9/0, rev 2.00/0.00, addr 2> on usbus0
> Trying to mount root from zfs:tank/default/root []...
> Root mount waiting for: usbus0
> uhub1: 4 ports with 3 removable, self powered
> ugen0.4: <vendor 0x0557 product 0x2419> at usbus0
> ukbd1 numa-domain 0 on uhub1
> ukbd1: <vendor 0x0557 product 0x2419, class 0/0, rev 1.10/1.00, addr 3> on usbus0
> kbd3 at ukbd1
> lo0: link state changed to UP
> ums0 numa-domain 0 on uhub1
> ums0: <vendor 0x0557 product 0x2419, class 0/0, rev 1.10/1.00, addr 3> on usbus0
> ums0: 3 buttons and [Z] coordinates ID=0
>
>
>
> And pciconf -lvb:
>
>
> hostb0@pci0:0:0:0: class=0x060000 card=0x098115d9 chip=0x20208086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E DMI3 Registers'
>     class      = bridge
>     subclass   = HOST-PCI
> none0@pci0:0:4:0: class=0x088000 card=0x00008086 chip=0x20218086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CBDMA Registers'
>     class      = base peripheral
>     bar   [10] = type Memory, range 64, base 0x383ffff2c000, size 16384, enabled
> none1@pci0:0:4:1: class=0x088000 card=0x00008086 chip=0x20218086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CBDMA Registers'
>     class      = base peripheral
>     bar   [10] = type Memory, range 64, base 0x383ffff28000, size 16384, enabled
> none2@pci0:0:4:2: class=0x088000 card=0x00008086 chip=0x20218086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CBDMA Registers'
>     class      = base peripheral
>     bar   [10] = type Memory, range 64, base 0x383ffff24000, size 16384, enabled
> none3@pci0:0:4:3: class=0x088000 card=0x00008086 chip=0x20218086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CBDMA Registers'
>     class      = base peripheral
>     bar   [10] = type Memory, range 64, base 0x383ffff20000, size 16384, enabled
> none4@pci0:0:4:4: class=0x088000 card=0x00008086 chip=0x20218086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CBDMA Registers'
>     class      = base peripheral
>     bar   [10] = type Memory, range 64, base 0x383ffff1c000, size 16384, enabled
> none5@pci0:0:4:5: class=0x088000 card=0x00008086 chip=0x20218086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CBDMA Registers'
>     class      = base peripheral
>     bar   [10] = type Memory, range 64, base 0x383ffff18000, size 16384, enabled
> none6@pci0:0:4:6: class=0x088000 card=0x00008086 chip=0x20218086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CBDMA Registers'
>     class      = base peripheral
>     bar   [10] = type Memory, range 64, base 0x383ffff14000, size 16384, enabled
> none7@pci0:0:4:7: class=0x088000 card=0x00008086 chip=0x20218086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CBDMA Registers'
>     class      = base peripheral
>     bar   [10] = type Memory, range 64, base 0x383ffff10000, size 16384, enabled
> none8@pci0:0:5:0: class=0x088000 card=0x00008086 chip=0x20248086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E MM/Vt-d Configuration Registers'
>     class      = base peripheral
> none9@pci0:0:5:2: class=0x088000 card=0x00000000 chip=0x20258086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
> ioapic0@pci0:0:5:4: class=0x080020 card=0x20268086 chip=0x20268086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
>     subclass   = interrupt controller
>     bar   [10] = type Memory, range 32, base 0x9d20a000, size 4096, enabled
> none10@pci0:0:8:0: class=0x088000 card=0x00008086 chip=0x20148086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E Ubox Registers'
>     class      = base peripheral
> none11@pci0:0:8:1: class=0x110100 card=0x00008086 chip=0x20158086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E Ubox Registers'
>     class      = dasp
>     subclass   = performance counters
> none12@pci0:0:8:2: class=0x088000 card=0x00008086 chip=0x20168086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E Ubox Registers'
>     class      = base peripheral
> none13@pci0:0:17:0: class=0xff0000 card=0x098115d9 chip=0xa1ec8086 rev=0x09 hdr=0x00
>     vendor     = 'Intel Corporation'
> none14@pci0:0:17:1: class=0xff0000 card=0x098115d9 chip=0xa1ed8086 rev=0x09 hdr=0x00
>     vendor     = 'Intel Corporation'
> ahci0@pci0:0:17:5: class=0x010601 card=0x098115d9 chip=0xa1d28086 rev=0x09 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Lewisburg SSATA Controller [AHCI mode]'
>     class      = mass storage
>     subclass   = SATA
>     bar   [10] = type Memory, range 32, base 0x9d206000, size 8192, enabled
>     bar   [14] = type Memory, range 32, base 0x9d209000, size 256, enabled
>     bar   [18] = type I/O Port, range 32, base 0x3070, size 8, enabled
>     bar   [1c] = type I/O Port, range 32, base 0x3060, size 4, enabled
>     bar   [20] = type I/O Port, range 32, base 0x3020, size 32, enabled
>     bar   [24] = type Memory, range 32, base 0x9d180000, size 524288, enabled
> xhci0@pci0:0:20:0: class=0x0c0330 card=0x098115d9 chip=0xa1af8086 rev=0x09 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Lewisburg USB 3.0 xHCI Controller'
>     class      = serial bus
>     subclass   = USB
>     bar   [10] = type Memory, range 64, base 0x383ffff00000, size 65536, enabled
> none15@pci0:0:20:2: class=0x118000 card=0x098115d9 chip=0xa1b18086 rev=0x09 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Lewisburg Thermal Subsystem'
>     class      = dasp
>     bar   [10] = type Memory, range 64, base 0x383ffff33000, size 4096, enabled
> none16@pci0:0:22:0: class=0x078000 card=0x098115d9 chip=0xa1ba8086 rev=0x09 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Lewisburg CSME: HECI'
>     class      = simple comms
>     bar   [10] = type Memory, range 64, base 0x383ffff32000, size 4096, enabled
> none17@pci0:0:22:1: class=0x078000 card=0x098115d9 chip=0xa1bb8086 rev=0x09 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Lewisburg CSME: HECI'
>     class      = simple comms
>     bar   [10] = type Memory, range 64, base 0x383ffff31000, size 4096, enabled
> none18@pci0:0:22:4: class=0x078000 card=0x098115d9 chip=0xa1be8086 rev=0x09 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Lewisburg CSME: HECI'
>     class      = simple comms
>     bar   [10] = type Memory, range 64, base 0x383ffff30000, size 4096, enabled
> ahci1@pci0:0:23:0: class=0x010601 card=0x098115d9 chip=0xa1828086 rev=0x09 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Lewisburg SATA Controller [AHCI mode]'
>     class      = mass storage
>     subclass   = SATA
>     bar   [10] = type Memory, range 32, base 0x9d204000, size 8192, enabled
>     bar   [14] = type Memory, range 32, base 0x9d208000, size 256, enabled
>     bar   [18] = type I/O Port, range 32, base 0x3050, size 8, enabled
>     bar   [1c] = type I/O Port, range 32, base 0x3040, size 4, enabled
>     bar   [20] = type I/O Port, range 32, base 0x3000, size 32, enabled
>     bar   [24] = type Memory, range 32, base 0x9d100000, size 524288, enabled
> pcib1@pci0:0:28:0: class=0x060400 card=0x098115d9 chip=0xa1908086 rev=0xf9 hdr=0x01
>     vendor     = 'Intel Corporation'
>     device     = 'Lewisburg PCI Express Root Port'
>     class      = bridge
>     subclass   = PCI-PCI
> pcib2@pci0:0:28:5: class=0x060400 card=0x098115d9 chip=0xa1958086 rev=0xf9 hdr=0x01
>     vendor     = 'Intel Corporation'
>     device     = 'Lewisburg PCI Express Root Port'
>     class      = bridge
>     subclass   = PCI-PCI
> isab0@pci0:0:31:0: class=0x060100 card=0x098115d9 chip=0xa1c38086 rev=0x09 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Lewisburg LPC Controller'
>     class      = bridge
>     subclass   = PCI-ISA
> none19@pci0:0:31:2: class=0x058000 card=0x098115d9 chip=0xa1a18086 rev=0x09 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Lewisburg PMC'
>     class      = memory
>     bar   [10] = type Memory, range 32, base 0x9d200000, size 16384, enabled
> none20@pci0:0:31:4: class=0x0c0500 card=0x098115d9 chip=0xa1a38086 rev=0x09 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Lewisburg SMBus'
>     class      = serial bus
>     subclass   = SMBus
>     bar   [10] = type Memory, range 64, base 0, size 256, disabled
>     bar   [20] = type I/O Port, range 32, base 0x780, size 32, enabled
> none21@pci0:0:31:5: class=0x0c8000 card=0x098115d9 chip=0xa1a48086 rev=0x09 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Lewisburg SPI Controller'
>     class      = serial bus
>     bar   [10] = type Memory, range 32, base 0xfe010000, size 4096, enabled
> pcib3@pci0:2:0:0: class=0x060400 card=0x11501a03 chip=0x11501a03 rev=0x04 hdr=0x01
>     vendor     = 'ASPEED Technology, Inc.'
>     device     = 'AST1150 PCI-to-PCI Bridge'
>     class      = bridge
>     subclass   = PCI-PCI
> vgapci0@pci0:3:0:0: class=0x030000 card=0x098115d9 chip=0x20001a03 rev=0x41 hdr=0x00
>     vendor     = 'ASPEED Technology, Inc.'
>     device     = 'ASPEED Graphics Family'
>     class      = display
>     subclass   = VGA
>     bar   [10] = type Memory, range 32, base 0x9c000000, size 16777216, enabled
>     bar   [14] = type Memory, range 32, base 0x9d000000, size 131072, enabled
>     bar   [18] = type I/O Port, range 32, base 0x2000, size 128, enabled
> pcib5@pci0:23:0:0: class=0x060400 card=0x098115d9 chip=0x20308086 rev=0x04 hdr=0x01
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E PCI Express Root Port A'
>     class      = bridge
>     subclass   = PCI-PCI
> none22@pci0:23:5:0: class=0x088000 card=0x00008086 chip=0x20348086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
> none23@pci0:23:5:2: class=0x088000 card=0x00000000 chip=0x20358086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E RAS Configuration Registers'
>     class      = base peripheral
> ioapic1@pci0:23:5:4: class=0x080020 card=0x20368086 chip=0x20368086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
>     subclass   = interrupt controller
>     bar   [10] = type Memory, range 32, base 0xaaf00000, size 4096, enabled
> none24@pci0:23:8:0: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none25@pci0:23:8:1: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none26@pci0:23:8:2: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none27@pci0:23:8:3: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none28@pci0:23:8:4: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none29@pci0:23:8:5: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none30@pci0:23:8:6: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none31@pci0:23:8:7: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none32@pci0:23:9:0: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none33@pci0:23:9:1: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none34@pci0:23:9:2: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none35@pci0:23:9:3: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none36@pci0:23:9:4: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none37@pci0:23:9:5: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none38@pci0:23:9:6: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none39@pci0:23:9:7: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none40@pci0:23:10:0: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none41@pci0:23:10:1: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none42@pci0:23:14:0: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none43@pci0:23:14:1: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none44@pci0:23:14:2: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none45@pci0:23:14:3: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none46@pci0:23:14:4: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none47@pci0:23:14:5: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none48@pci0:23:14:6: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none49@pci0:23:14:7: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none50@pci0:23:15:0: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none51@pci0:23:15:1: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none52@pci0:23:15:2: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none53@pci0:23:15:3: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none54@pci0:23:15:4: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none55@pci0:23:15:5: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none56@pci0:23:15:6: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none57@pci0:23:15:7: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none58@pci0:23:16:0: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none59@pci0:23:16:1: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none60@pci0:23:29:0: class=0x088000 card=0x00008086 chip=0x20548086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none61@pci0:23:29:1: class=0x088000 card=0x00008086 chip=0x20558086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none62@pci0:23:29:2: class=0x088000 card=0x00008086 chip=0x20568086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none63@pci0:23:29:3: class=0x088000 card=0x00008086 chip=0x20578086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none64@pci0:23:30:0: class=0x088000 card=0x00008086 chip=0x20808086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E PCU Registers'
>     class      = base peripheral
> none65@pci0:23:30:1: class=0x088000 card=0x00008086 chip=0x20818086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E PCU Registers'
>     class      = base peripheral
> none66@pci0:23:30:2: class=0x088000 card=0x00008086 chip=0x20828086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E PCU Registers'
>     class      = base peripheral
> none67@pci0:23:30:3: class=0x088000 card=0x00008086 chip=0x20838086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E PCU Registers'
>     class      = base peripheral
> none68@pci0:23:30:4: class=0x088000 card=0x00008086 chip=0x20848086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E PCU Registers'
>     class      = base peripheral
> none69@pci0:23:30:5: class=0x088000 card=0x00008086 chip=0x20858086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E PCU Registers'
>     class      = base peripheral
> none70@pci0:23:30:6: class=0x088000 card=0x00008086 chip=0x20868086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E PCU Registers'
>     class      = base peripheral
> pcib6@pci0:24:0:0: class=0x060400 card=0xdeadbeef chip=0x37c08086 rev=0x09 hdr=0x01
>     vendor     = 'Intel Corporation'
>     class      = bridge
>     subclass   = PCI-PCI
>     bar   [10] = type Memory, range 64, base 0xaae00000, size 131072, enabled
> pcib7@pci0:25:3:0: class=0x060400 card=0xdeadbeef chip=0x37c58086 rev=0x09 hdr=0x01
>     vendor     = 'Intel Corporation'
>     class      = bridge
>     subclass   = PCI-PCI
> ixl0@pci0:26:0:0: class=0x020000 card=0x37d215d9 chip=0x37d28086 rev=0x09 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Ethernet Connection X722 for 10GBASE-T'
>     class      = network
>     subclass   = ethernet
>     bar   [10] = type Prefetchable Memory, range 64, base 0xa9000000, size 16777216, enabled
>     bar   [1c] = type Prefetchable Memory, range 64, base 0xaa008000, size 32768, enabled
> ixl1@pci0:26:0:1: class=0x020000 card=0x37d215d9 chip=0x37d28086 rev=0x09 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Ethernet Connection X722 for 10GBASE-T'
>     class      = network
>     subclass   = ethernet
>     bar   [10] = type Prefetchable Memory, range 64, base 0xa8000000, size 16777216, enabled
>     bar   [1c] = type Prefetchable Memory, range 64, base 0xaa000000, size 32768, enabled
> pcib9@pci0:58:2:0: class=0x060400 card=0x098115d9 chip=0x20328086 rev=0x04 hdr=0x01
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E PCI Express Root Port C'
>     class      = bridge
>     subclass   = PCI-PCI
> none71@pci0:58:5:0: class=0x088000 card=0x00008086 chip=0x20348086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
> none72@pci0:58:5:2: class=0x088000 card=0x00000000 chip=0x20358086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E RAS Configuration Registers'
>     class      = base peripheral
> ioapic2@pci0:58:5:4: class=0x080020 card=0x20368086 chip=0x20368086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
>     subclass   = interrupt controller
>     bar   [10] = type Memory, range 32, base 0xb8700000, size 4096, enabled
> none73@pci0:58:8:0: class=0x088000 card=0x00008086 chip=0x20668086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
> none74@pci0:58:9:0: class=0x088000 card=0x00008086 chip=0x20668086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
> none75@pci0:58:10:0: class=0x088000 card=0x00008086 chip=0x20408086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
> none76@pci0:58:10:1: class=0x088000 card=0x00008086 chip=0x20418086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
> none77@pci0:58:10:2: class=0x088000 card=0x00008086 chip=0x20428086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
> none78@pci0:58:10:3: class=0x088000 card=0x00008086 chip=0x20438086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
> none79@pci0:58:10:4: class=0x088000 card=0x00008086 chip=0x20448086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
> none80@pci0:58:10:5: class=0x088000 card=0x00008086 chip=0x20458086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
> none81@pci0:58:10:6: class=0x088000 card=0x00008086 chip=0x20468086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
> none82@pci0:58:10:7: class=0x088000 card=0x00008086 chip=0x20478086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
> none83@pci0:58:11:0: class=0x088000 card=0x00008086 chip=0x20488086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
> none84@pci0:58:11:1: class=0x088000 card=0x00008086 chip=0x20498086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
> none85@pci0:58:11:2: class=0x088000 card=0x00008086 chip=0x204a8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
> none86@pci0:58:11:3: class=0x088000 card=0x00008086 chip=0x204b8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
> none87@pci0:58:12:0: class=0x088000 card=0x00008086 chip=0x20408086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
> none88@pci0:58:12:1: class=0x088000 card=0x00008086 chip=0x20418086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
> none89@pci0:58:12:2: class=0x088000 card=0x00008086 chip=0x20428086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
> none90@pci0:58:12:3: class=0x088000 card=0x00008086 chip=0x20438086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
> none91@pci0:58:12:4: class=0x088000 card=0x00008086 chip=0x20448086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
> none92@pci0:58:12:5: class=0x088000 card=0x00008086 chip=0x20458086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
> none93@pci0:58:12:6: class=0x088000 card=0x00008086 chip=0x20468086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
> none94@pci0:58:12:7: class=0x088000 card=0x00008086 chip=0x20478086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
> none95@pci0:58:13:0: class=0x088000 card=0x00008086 chip=0x20488086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
> none96@pci0:58:13:1: class=0x088000 card=0x00008086 chip=0x20498086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
> none97@pci0:58:13:2: class=0x088000 card=0x00008086 chip=0x204a8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
> none98@pci0:58:13:3: class=0x088000 card=0x00008086 chip=0x204b8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
> mpr0@pci0:59:0:0: class=0x010700 card=0x080815d9 chip=0x00971000 rev=0x02 hdr=0x00
>     vendor     = 'LSI Logic / Symbios Logic'
>     device     = 'SAS3008 PCI-Express Fusion-MPT SAS-3'
>     class      = mass storage
>     subclass   = SAS
>     bar   [10] = type I/O Port, range 32, base 0x7000, size 256, enabled
>     bar   [14] = type Memory, range 64, base 0xb8640000, size 65536, enabled
>     bar   [1c] = type Memory, range 64, base 0xb8600000, size 262144, enabled
> none99@pci0:93:5:0: class=0x088000 card=0x00008086 chip=0x20348086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
> none100@pci0:93:5:2: class=0x088000 card=0x00000000 chip=0x20358086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E RAS Configuration Registers'
>     class      = base peripheral
> ioapic3@pci0:93:5:4: class=0x080020 card=0x20368086 chip=0x20368086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
>     subclass   = interrupt controller
>     bar   [10] = type Memory, range 32, base 0xc5f00000, size 4096, enabled
> none101@pci0:93:14:0: class=0x110100 card=0x00008086 chip=0x20588086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = dasp
>     subclass   = performance counters
> none102@pci0:93:14:1: class=0x088000 card=0x00008086 chip=0x20598086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
> none103@pci0:93:15:0: class=0x110100 card=0x00008086 chip=0x20588086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = dasp
>     subclass   = performance counters
> none104@pci0:93:15:1: class=0x088000 card=0x00008086 chip=0x20598086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
> none105@pci0:93:18:0: class=0x110100 card=0x00008086 chip=0x204c8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E M3KTI Registers'
>     class      = dasp
>     subclass   = performance counters
> none106@pci0:93:18:1: class=0x110100 card=0x00008086 chip=0x204d8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E M3KTI Registers'
>     class      = dasp
>     subclass   = performance counters
> none107@pci0:93:18:2: class=0x088000 card=0x00008086 chip=0x204e8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E M3KTI Registers'
>     class      = base peripheral
> none108@pci0:93:21:0: class=0x088000 card=0x00008086 chip=0x20188086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E M2PCI Registers'
>     class      = base peripheral
> none109@pci0:93:22:0: class=0x088000 card=0x00008086 chip=0x20188086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E M2PCI Registers'
>     class      = base peripheral
> none110@pci0:93:22:4: class=0x088000 card=0x00008086 chip=0x20188086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E M2PCI Registers'
>     class      = base peripheral
> none111@pci0:128:4:0: class=0x088000 card=0x00008086 chip=0x20218086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CBDMA Registers'
>     class      = base peripheral
>     bar   [10] = type Memory, range 64, base 0x393ffff1c000, size 16384, enabled
> none112@pci0:128:4:1: class=0x088000 card=0x00008086 chip=0x20218086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CBDMA Registers'
>     class      = base peripheral
>     bar   [10] = type Memory, range 64, base 0x393ffff18000, size 16384, enabled
> none113@pci0:128:4:2: class=0x088000 card=0x00008086 chip=0x20218086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CBDMA Registers'
>     class      = base peripheral
>     bar   [10] = type Memory, range 64, base 0x393ffff14000, size 16384, enabled
> none114@pci0:128:4:3: class=0x088000 card=0x00008086 chip=0x20218086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CBDMA Registers'
>     class      = base peripheral
>     bar   [10] = type Memory, range 64, base 0x393ffff10000, size 16384, enabled
> none115@pci0:128:4:4: class=0x088000 card=0x00008086 chip=0x20218086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CBDMA Registers'
>     class      = base peripheral
>     bar   [10] = type Memory, range 64, base 0x393ffff0c000, size 16384, enabled
> none116@pci0:128:4:5: class=0x088000 card=0x00008086 chip=0x20218086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CBDMA Registers'
>     class      = base peripheral
>     bar   [10] = type Memory, range 64, base 0x393ffff08000, size 16384, enabled
> none117@pci0:128:4:6: class=0x088000 card=0x00008086 chip=0x20218086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CBDMA Registers'
>     class      = base peripheral
>     bar   [10] = type Memory, range 64, base 0x393ffff04000, size 16384, enabled
> none118@pci0:128:4:7: class=0x088000 card=0x00008086 chip=0x20218086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CBDMA Registers'
>     class      = base peripheral
>     bar   [10] = type Memory, range 64, base 0x393ffff00000, size 16384, enabled
> none119@pci0:128:5:0: class=0x088000 card=0x00008086 chip=0x20248086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E MM/Vt-d Configuration Registers'
>     class      = base peripheral
> none120@pci0:128:5:2: class=0x088000 card=0x00000000 chip=0x20258086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
> ioapic4@pci0:128:5:4: class=0x080020 card=0x20268086 chip=0x20268086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
>     subclass   = interrupt controller
>     bar   [10] = type Memory, range 32, base 0xd3700000, size 4096, enabled
> none121@pci0:128:8:0: class=0x088000 card=0x00008086 chip=0x20148086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E Ubox Registers'
>     class      = base peripheral
> none122@pci0:128:8:1: class=0x110100 card=0x00008086 chip=0x20158086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E Ubox Registers'
>     class      = dasp
>     subclass   = performance counters
> none123@pci0:128:8:2: class=0x088000 card=0x00008086 chip=0x20168086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E Ubox Registers'
>     class      = base peripheral
> none124@pci0:133:5:0: class=0x088000 card=0x00008086 chip=0x20348086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
> none125@pci0:133:5:2: class=0x088000 card=0x00000000 chip=0x20358086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E RAS Configuration Registers'
>     class      = base peripheral
> ioapic5@pci0:133:5:4: class=0x080020 card=0x20368086 chip=0x20368086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
>     subclass   = interrupt controller
>     bar   [10] = type Memory, range 32, base 0xe0f00000, size 4096, enabled
> none126@pci0:133:8:0: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none127@pci0:133:8:1: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none128@pci0:133:8:2: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none129@pci0:133:8:3: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none130@pci0:133:8:4: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none131@pci0:133:8:5: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none132@pci0:133:8:6: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none133@pci0:133:8:7: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none134@pci0:133:9:0: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none135@pci0:133:9:1: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none136@pci0:133:9:2: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none137@pci0:133:9:3: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none138@pci0:133:9:4: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none139@pci0:133:9:5: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none140@pci0:133:9:6: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none141@pci0:133:9:7: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none142@pci0:133:10:0: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none143@pci0:133:10:1: class=0x088000 card=0x00008086 chip=0x208d8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none144@pci0:133:14:0: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none145@pci0:133:14:1: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none146@pci0:133:14:2: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none147@pci0:133:14:3: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none148@pci0:133:14:4: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none149@pci0:133:14:5: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none150@pci0:133:14:6: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none151@pci0:133:14:7: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none152@pci0:133:15:0: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none153@pci0:133:15:1: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none154@pci0:133:15:2: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none155@pci0:133:15:3: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none156@pci0:133:15:4: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none157@pci0:133:15:5: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none158@pci0:133:15:6: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none159@pci0:133:15:7: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none160@pci0:133:16:0: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none161@pci0:133:16:1: class=0x088000 card=0x00008086 chip=0x208e8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none162@pci0:133:29:0: class=0x088000 card=0x00008086 chip=0x20548086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none163@pci0:133:29:1: class=0x088000 card=0x00008086 chip=0x20558086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none164@pci0:133:29:2: class=0x088000 card=0x00008086 chip=0x20568086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none165@pci0:133:29:3: class=0x088000 card=0x00008086 chip=0x20578086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E CHA Registers'
>     class      = base peripheral
> none166@pci0:133:30:0: class=0x088000 card=0x00008086 chip=0x20808086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E PCU Registers'
>     class      = base peripheral
> none167@pci0:133:30:1: class=0x088000 card=0x00008086 chip=0x20818086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E PCU Registers'
>     class      = base peripheral
> none168@pci0:133:30:2: class=0x088000 card=0x00008086 chip=0x20828086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E PCU Registers'
>     class      = base peripheral
> none169@pci0:133:30:3: class=0x088000 card=0x00008086 chip=0x20838086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E PCU Registers'
>     class      = base peripheral
> none170@pci0:133:30:4: class=0x088000 card=0x00008086 chip=0x20848086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E PCU Registers'
>     class      = base peripheral
> none171@pci0:133:30:5: class=0x088000 card=0x00008086 chip=0x20858086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E PCU Registers'
>     class      = base peripheral
> none172@pci0:133:30:6: class=0x088000 card=0x00008086 chip=0x20868086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E PCU Registers'
>     class      = base peripheral
> none173@pci0:174:5:0: class=0x088000 card=0x00008086 chip=0x20348086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
> none174@pci0:174:5:2: class=0x088000 card=0x00000000 chip=0x20358086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E RAS Configuration Registers'
>     class      = base peripheral
> ioapic6@pci0:174:5:4: class=0x080020 card=0x20368086 chip=0x20368086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
>     subclass   = interrupt controller
>     bar   [10] = type Memory, range 32, base 0xee700000, size 4096, enabled
> none175@pci0:174:8:0: class=0x088000 card=0x00008086 chip=0x20668086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
> none176@pci0:174:9:0: class=0x088000 card=0x00008086 chip=0x20668086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
> none177@pci0:174:10:0: class=0x088000 card=0x00008086 chip=0x20408086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
> none178@pci0:174:10:1: class=0x088000 card=0x00008086 chip=0x20418086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
> none179@pci0:174:10:2: class=0x088000 card=0x00008086 chip=0x20428086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
> none180@pci0:174:10:3: class=0x088000 card=0x00008086 chip=0x20438086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
> none181@pci0:174:10:4: class=0x088000 card=0x00008086 chip=0x20448086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
> none182@pci0:174:10:5: class=0x088000 card=0x00008086 chip=0x20458086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
> none183@pci0:174:10:6: class=0x088000 card=0x00008086 chip=0x20468086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
> none184@pci0:174:10:7: class=0x088000 card=0x00008086 chip=0x20478086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
> none185@pci0:174:11:0: class=0x088000 card=0x00008086 chip=0x20488086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
> none186@pci0:174:11:1: class=0x088000 card=0x00008086 chip=0x20498086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
> none187@pci0:174:11:2: class=0x088000 card=0x00008086 chip=0x204a8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
> none188@pci0:174:11:3: class=0x088000 card=0x00008086 chip=0x204b8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
> none189@pci0:174:12:0: class=0x088000 card=0x00008086 chip=0x20408086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
> none190@pci0:174:12:1: class=0x088000 card=0x00008086 chip=0x20418086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
> none191@pci0:174:12:2: class=0x088000 card=0x00008086 chip=0x20428086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
> none192@pci0:174:12:3: class=0x088000 card=0x00008086 chip=0x20438086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
> none193@pci0:174:12:4: class=0x088000 card=0x00008086 chip=0x20448086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
> none194@pci0:174:12:5: class=0x088000 card=0x00008086 chip=0x20458086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
> none195@pci0:174:12:6: class=0x088000 card=0x00008086 chip=0x20468086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
> none196@pci0:174:12:7: class=0x088000 card=0x00008086 chip=0x20478086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
> none197@pci0:174:13:0: class=0x088000 card=0x00008086 chip=0x20488086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
> none198@pci0:174:13:1: class=0x088000 card=0x00008086 chip=0x20498086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
> none199@pci0:174:13:2: class=0x088000 card=0x00008086 chip=0x204a8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
> none200@pci0:174:13:3: class=0x088000 card=0x00008086 chip=0x204b8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
> none201@pci0:215:5:0: class=0x088000 card=0x00008086 chip=0x20348086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
> none202@pci0:215:5:2: class=0x088000 card=0x00000000 chip=0x20358086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E RAS Configuration Registers'
>     class      = base peripheral
> ioapic7@pci0:215:5:4: class=0x080020 card=0x20368086 chip=0x20368086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
>     subclass   = interrupt controller
>     bar   [10] = type Memory, range 32, base 0xfbf00000, size 4096, enabled
> none203@pci0:215:14:0: class=0x110100 card=0x00008086 chip=0x20588086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = dasp
>     subclass   = performance counters
> none204@pci0:215:14:1: class=0x088000 card=0x00008086 chip=0x20598086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
> none205@pci0:215:15:0: class=0x110100 card=0x00008086 chip=0x20588086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = dasp
>     subclass   = performance counters
> none206@pci0:215:15:1: class=0x088000 card=0x00008086 chip=0x20598086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = base peripheral
> none207@pci0:215:18:0: class=0x110100 card=0x00008086 chip=0x204c8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E M3KTI Registers'
>     class      = dasp
>     subclass   = performance counters
> none208@pci0:215:18:1: class=0x110100 card=0x00008086 chip=0x204d8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E M3KTI Registers'
>     class      = dasp
>     subclass   = performance counters
> none209@pci0:215:18:2: class=0x088000 card=0x00008086 chip=0x204e8086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E M3KTI Registers'
>     class      = base peripheral
> none210@pci0:215:21:0: class=0x088000 card=0x00008086 chip=0x20188086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E M2PCI Registers'
>     class      = base peripheral
> none211@pci0:215:22:0: class=0x088000 card=0x00008086 chip=0x20188086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E M2PCI Registers'
>     class      = base peripheral
> none212@pci0:215:22:4: class=0x088000 card=0x00008086 chip=0x20188086 rev=0x04 hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = 'Sky Lake-E M2PCI Registers'
>     class      = base peripheral
>
>
> # grep swap /etc/fstab ; echo
> #/dev/mirror/swap                       none            swap    sw
> /dev/gpt/swap0                  none            swap    sw
> #/dev/gpt/swap1                 none            swap    sw
>
> # swapinfo -h
> Device          1024-blocks     Used    Avail Capacity
> /dev/gpt/swap0     16777216       0B      16G     0%
>
> # ls -ltr /var/crash/
> total 3844652
> -rw-r--r--  1 root  wheel           5 Jun 21 21:34 minfree
> -rw-------  1 root  wheel         462 Jul  5 11:46 info.0
> -rw-------  1 root  wheel  5043531776 Jul  5 11:47 vmcore.0
> -rw-------  1 root  wheel      128912 Jul  5 11:47 core.txt.0
> -rw-------  1 root  wheel         464 Jul  6 12:24 info.1
> -rw-------  1 root  wheel  3628683264 Jul  6 12:24 vmcore.1
> -rw-------  1 root  wheel      127947 Jul  6 12:24 core.txt.1
> -rw-r--r--  1 root  wheel         639 Jul  9 12:51 ls.out.0
> -rw-r--r--  1 root  wheel        3960 Jul 11 06:36 zpool-iostat.out
> -rw-------  1 root  wheel         464 Jul 11 12:47 info.2
> -rw-------  1 root  wheel  3777585152 Jul 11 12:48 vmcore.2
> -rw-------  1 root  wheel      127775 Jul 11 12:48 core.txt.2
> -rw-r--r--  1 root  wheel           0 Jul 11 13:12 zfs-stress.start
> -rw-r--r--  1 root  wheel           2 Jul 11 13:37 bounds
> -rw-------  1 root  wheel         458 Jul 11 13:37 info.3
> -rw-------  1 root  wheel  7352229888 Jul 11 13:38 vmcore.3
> lrwxr-xr-x  1 root  wheel           6 Jul 11 13:38 info.last@ -> info.3
> lrwxr-xr-x  1 root  wheel           8 Jul 11 13:38 vmcore.last@ -> vmcore.3
> -rw-------  1 root  wheel      143973 Jul 11 13:38 core.txt.3
>
> _______________________________________________
> [hidden email] mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-questions
> To unsubscribe, send any mail to "[hidden email]"
>
>


--
-------------------
Mike Tancsa, tel +1 519 651 3400 x203
Sentex Communications, [hidden email]
Providing Internet services since 1994 www.sentex.net
Cambridge, Ontario Canada
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Disk/ZFS activity crash on 11.2-STABLE

list
On Thu, Jul 12, 2018 at 11:53:31AM -0400, Mike Tancsa wrote:
>
> prior to the crash, if you run gstat does it show the disks at 100% load ?
>
> You have a lot of RAM, is ARC by chance using all of it ? Try putting a
> hard limit on it. So if you have 98G of RAM, place a limit at something
> like 70G for now.
> Also, what is the file system like ? Is it a few big files or MANY
> little files.  Post the output of zfs-stats -a
>

..snip..

> And speaking
> of crash, post the contents of the
> core.txt.3
>
> file in /var/crash
>
> ---Mike


Mike:

Thank you for copying me on your reply, I forgot to request that.  I
read the digest version of the list, so a direct reply is much more
timely.

The core.txt.3 file is attached.  Your suggestion of dhclient-safety.sh
makes sense, I should have thought of using a legacy rc.d script in
/usr/local/etc/rc.d/

zfs-stats.out is also attached.

Yes, ARC usage is high.  top shows:

last pid:  2215;  load averages:  1.50,  1.69,  1.69   up 0+00:44:59  11:02:00
27 processes:  2 running, 25 sleeping
CPU:  0.0% user,  0.0% nice,  9.0% system,  0.1% interrupt, 90.9% idle
Mem: 1416K Active, 11M Inact, 2788K Laundry, 91G Wired, 1236M Free
ARC: 84G Total, 68M MFU, 83G MRU, 15M Anon, 877M Header, 4078K Other
     83G Compressed, 148G Uncompressed, 1.78:1 Ratio
Swap: 16G Total, 15M Used, 16G Free

I'm re-running the test now.  I had shut off hyperthreading to go down to
24 cores, but it's back on now.  This server will be a computational resource,
not just storage, so I'd like to find a way to stablize the configuration with
hyperthreading enabled, if possible.

Thanks again,

Jim



zfs-stress.sh:

#!/usr/bin/env bash

FILE="/var/crash/zfs-stress.start"
SLEEP_TIME=10

# if this script causes a crash, then we should expect to see a
# crash dump newer than this file:

touch $FILE

logger "$0 : begin"

echo 'zfs destroy -r mendeleev@stress-test' | sh -v 2>&1 | logger

echo 'zfs snap    -r mendeleev@stress-test' | sh -v 2>&1 | logger

logger "$0 : backgrounding 'zfs send -R mendeleev@stress-test > /dev/null'"

zfs send -R mendeleev@stress-test > /dev/null &

while [ -f $FILE ] && sleep $SLEEP_TIME; do

        printf "gstat -ab:\n$(gstat -ab)\n\n" | logger
        top | head -8 | logger

done

[ -f $FILE ] || echo "$0 : $FILE has disappeared ... terminating."

-- snip




_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[hidden email]"

core.txt.3 (143K) Download Attachment
zfs-stats.out (11K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Disk/ZFS activity crash on 11.2-STABLE

mdtancsa
On 7/12/2018 2:35 PM, Jim Long wrote:

>
> Yes, ARC usage is high.  top shows:
>
> last pid:  2215;  load averages:  1.50,  1.69,  1.69   up 0+00:44:59  11:02:00
> 27 processes:  2 running, 25 sleeping
> CPU:  0.0% user,  0.0% nice,  9.0% system,  0.1% interrupt, 90.9% idle
> Mem: 1416K Active, 11M Inact, 2788K Laundry, 91G Wired, 1236M Free
> ARC: 84G Total, 68M MFU, 83G MRU, 15M Anon, 877M Header, 4078K Other
>      83G Compressed, 148G Uncompressed, 1.78:1 Ratio
> Swap: 16G Total, 15M Used, 16G Free
>
> I'm re-running the test now.  I had shut off hyperthreading to go down to
> 24 cores, but it's back on now.  This server will be a computational resource,
> not just storage, so I'd like to find a way to stablize the configuration with
> hyperthreading enabled, if possible.
I would leave HT on.  Based on the zfs stats

CACHE HITS BY DATA TYPE:
          Demand Data: 51.38% 457.85k
          Prefetch Data: 0.00% 2
          Demand Metadata: 44.71% 398.41k
          Prefetch Metadata: 3.91% 34.86k

        CACHE MISSES BY DATA TYPE:
          Demand Data: 3.22% 17.71k
          Prefetch Data: 81.77% 449.51k
          Demand Metadata: 8.57% 47.14k
          Prefetch Metadata: 6.43% 35.35k


If I read this right, you are not hurting too much for meta data misses.
I find things like
zfs list -t snapshots
REALLY slow down if there is not enough caching for meta data where
there are a lot of files and directories.

eg. on a box that has several million small files and many many
directories, I increased the amount of RAM to vfs.zfs.arc_meta_limit and
it helped a LOT. Even then I still see a lot of misses (it was worse before)

        CACHE HITS BY DATA TYPE:
          Demand Data:                  10.82%  5.58b
          Prefetch Data:                0.66%   339.34m
          Demand Metadata:              50.83%  26.22b
          Prefetch Metadata:            37.69%  19.44b

        CACHE MISSES BY DATA TYPE:
          Demand Data:                  6.79%   680.20m
          Prefetch Data:                8.43%   844.48m
          Demand Metadata:              79.52%  7.97b
          Prefetch Metadata:            5.26%   527.45m

But I dont think thats your case.  I would try and set a ceiling. On
RELENG_11 you dont need to reboot

Try
sysctl -w vfs.zfs.arc_max=77946198016

which shaves off 20G from what ARC can gobble up. Not sure if thats your
issue, but it is an issue for some users.

If you are still hurting for caching, an SSD drive or NVME and make it a
caching device for your pool.

and what does
zpool status
show ?

        ---Mike


--
-------------------
Mike Tancsa, tel +1 519 651 3400 x203
Sentex Communications, [hidden email]
Providing Internet services since 1994 www.sentex.net
Cambridge, Ontario Canada
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Disk/ZFS activity crash on 11.2-STABLE

list
On Thu, Jul 12, 2018 at 02:49:53PM -0400, Mike Tancsa wrote:

--snip--

> I would try and set a ceiling. On
> RELENG_11 you dont need to reboot
>
> Try
> sysctl -w vfs.zfs.arc_max=77946198016
>
> which shaves off 20G from what ARC can gobble up. Not sure if thats your
> issue, but it is an issue for some users.
>
> If you are still hurting for caching, an SSD drive or NVME and make it a
> caching device for your pool.
>
> and what does
> zpool status
> show ?

I set the limit to the value you suggested, and the next test ran less
than three minutes before the machine rebooted, with no crash dump produced.

I further reduced the limit to 50G and it's been running for about 50 minutes
so far.  Fingers crossed.  I do have L2ARC I can add if need be.

I'll keep you posted on how this run goes.

Thank you,

Jim



# zpool list; echo; zpool status
NAME        SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
mendeleev  21.6T  5.02T  16.6T        -         -     0%    23%  1.00x  ONLINE  -

  pool: mendeleev
 state: ONLINE
  scan: scrub repaired 0 in 3h39m with 0 errors on Wed Jun  6 18:24:08 2018
config:

        NAME          STATE     READ WRITE CKSUM
        mendeleev     ONLINE       0     0     0
          raidz2-0    ONLINE       0     0     0
            gpt/zfs0  ONLINE       0     0     0
            gpt/zfs1  ONLINE       0     0     0
            gpt/zfs2  ONLINE       0     0     0
            gpt/zfs3  ONLINE       0     0     0
            gpt/zfs4  ONLINE       0     0     0
            gpt/zfs5  ONLINE       0     0     0

errors: No known data errors


Here's zfs-stats after about 50 min of activity.

# zfs-stats -a

------------------------------------------------------------------------
ZFS Subsystem Report                            Thu Jul 12 14:38:04 2018
------------------------------------------------------------------------

System Information:

        Kernel Version:                         1102501 (osreldate)
        Hardware Platform:                      amd64
        Processor Architecture:                 amd64

        ZFS Storage pool Version:               5000
        ZFS Filesystem Version:                 5

FreeBSD 11.2-STABLE #0 r335674: Tue Jun 26 13:20:24 PDT 2018 root
 2:38PM  up 55 mins, 2 users, load averages: 3.44, 3.88, 3.83

------------------------------------------------------------------------

System Memory:

        0.02%   14.62   MiB Active,     0.02%   21.34   MiB Inact
        62.00%  57.18   GiB Wired,      0.00%   0 Cache
        37.96%  35.01   GiB Free,       -0.00%  -204800 Bytes Gap

        Real Installed:                         96.00   GiB
        Real Available:                 98.57%  94.63   GiB
        Real Managed:                   97.45%  92.22   GiB

        Logical Total:                          96.00   GiB
        Logical Used:                   63.51%  60.97   GiB
        Logical Free:                   36.49%  35.03   GiB

Kernel Memory:                                  968.90  MiB
        Data:                           96.26%  932.67  MiB
        Text:                           3.74%   36.24   MiB

Kernel Memory Map:                              92.22   GiB
        Size:                           1.70%   1.57    GiB
        Free:                           98.30%  90.65   GiB

------------------------------------------------------------------------

ARC Summary: (HEALTHY)
        Memory Throttle Count:                  0

ARC Misc:
        Deleted:                                17.53m
        Recycle Misses:                         0
        Mutex Misses:                           46.41k
        Evict Skips:                            26.28k

ARC Size:                               100.02% 50.01   GiB
        Target Size: (Adaptive)         100.00% 50.00   GiB
        Min Size (Hard Limit):          22.80%  11.40   GiB
        Max Size (High Water):          4:1     50.00   GiB

ARC Size Breakdown:
        Recently Used Cache Size:       99.19%  49.60   GiB
        Frequently Used Cache Size:     0.81%   416.12  MiB

ARC Hash Breakdown:
        Elements Max:                           3.96m
        Elements Current:               95.60%  3.79m
        Collisions:                             3.22m
        Chain Max:                              5
        Chains:                                 367.37k

------------------------------------------------------------------------

ARC Efficiency:                                 63.22m
        Cache Hit Ratio:                43.75%  27.66m
        Cache Miss Ratio:               56.25%  35.56m
        Actual Hit Ratio:               39.39%  24.90m

        Data Demand Efficiency:         69.68%  26.96m
        Data Prefetch Efficiency:       0.00%   18.77m

        CACHE HITS BY CACHE LIST:
          Anonymously Used:             9.95%   2.75m
          Most Recently Used:           87.72%  24.26m
          Most Frequently Used:         2.32%   640.73k
          Most Recently Used Ghost:     0.00%   0
          Most Frequently Used Ghost:   0.01%   2.43k

        CACHE HITS BY DATA TYPE:
          Demand Data:                  67.92%  18.79m
          Prefetch Data:                0.00%   8
          Demand Metadata:              22.12%  6.12m
          Prefetch Metadata:            9.96%   2.76m

        CACHE MISSES BY DATA TYPE:
          Demand Data:                  22.99%  8.18m
          Prefetch Data:                52.77%  18.77m
          Demand Metadata:              17.07%  6.07m
          Prefetch Metadata:            7.17%   2.55m

------------------------------------------------------------------------

L2ARC is disabled

------------------------------------------------------------------------

File-Level Prefetch: (HEALTHY)

DMU Efficiency:                                 137.36k
        Hit Ratio:                      0.34%   470
        Miss Ratio:                     99.66%  136.89k

        Colinear:                               0
          Hit Ratio:                    100.00% 0
          Miss Ratio:                   100.00% 0

        Stride:                                 0
          Hit Ratio:                    100.00% 0
          Miss Ratio:                   100.00% 0

DMU Misc:
        Reclaim:                                0
          Successes:                    100.00% 0
          Failures:                     100.00% 0

        Streams:                                0
          +Resets:                      100.00% 0
          -Resets:                      100.00% 0
          Bogus:                                0

------------------------------------------------------------------------

VDEV cache is disabled

------------------------------------------------------------------------

ZFS Tunables (sysctl):
        kern.maxusers                           6392
        vm.kmem_size                            99019939840
        vm.kmem_size_scale                      1
        vm.kmem_size_min                        0
        vm.kmem_size_max                        1319413950874
        vfs.zfs.trim.max_interval               1
        vfs.zfs.trim.timeout                    30
        vfs.zfs.trim.txg_delay                  32
        vfs.zfs.trim.enabled                    1
        vfs.zfs.vol.immediate_write_sz          32768
        vfs.zfs.vol.unmap_sync_enabled          0
        vfs.zfs.vol.unmap_enabled               1
        vfs.zfs.vol.recursive                   0
        vfs.zfs.vol.mode                        1
        vfs.zfs.version.zpl                     5
        vfs.zfs.version.spa                     5000
        vfs.zfs.version.acl                     1
        vfs.zfs.version.ioctl                   7
        vfs.zfs.debug                           0
        vfs.zfs.super_owner                     0
        vfs.zfs.immediate_write_sz              32768
        vfs.zfs.sync_pass_rewrite               2
        vfs.zfs.sync_pass_dont_compress         5
        vfs.zfs.sync_pass_deferred_free         2
        vfs.zfs.zio.dva_throttle_enabled        1
        vfs.zfs.zio.exclude_metadata            0
        vfs.zfs.zio.use_uma                     1
        vfs.zfs.zil_slog_bulk                   786432
        vfs.zfs.cache_flush_disable             0
        vfs.zfs.zil_replay_disable              0
        vfs.zfs.standard_sm_blksz               131072
        vfs.zfs.dtl_sm_blksz                    4096
        vfs.zfs.min_auto_ashift                 12
        vfs.zfs.max_auto_ashift                 13
        vfs.zfs.vdev.trim_max_pending           10000
        vfs.zfs.vdev.bio_delete_disable         0
        vfs.zfs.vdev.bio_flush_disable          0
        vfs.zfs.vdev.queue_depth_pct            1000
        vfs.zfs.vdev.write_gap_limit            4096
        vfs.zfs.vdev.read_gap_limit             32768
        vfs.zfs.vdev.aggregation_limit          131072
        vfs.zfs.vdev.trim_max_active            64
        vfs.zfs.vdev.trim_min_active            1
        vfs.zfs.vdev.scrub_max_active           2
        vfs.zfs.vdev.scrub_min_active           1
        vfs.zfs.vdev.async_write_max_active     10
        vfs.zfs.vdev.async_write_min_active     1
        vfs.zfs.vdev.async_read_max_active      3
        vfs.zfs.vdev.async_read_min_active      1
        vfs.zfs.vdev.sync_write_max_active      10
        vfs.zfs.vdev.sync_write_min_active      10
        vfs.zfs.vdev.sync_read_max_active       10
        vfs.zfs.vdev.sync_read_min_active       10
        vfs.zfs.vdev.max_active                 1000
        vfs.zfs.vdev.async_write_active_max_dirty_percent60
        vfs.zfs.vdev.async_write_active_min_dirty_percent30
        vfs.zfs.vdev.mirror.non_rotating_seek_inc1
        vfs.zfs.vdev.mirror.non_rotating_inc    0
        vfs.zfs.vdev.mirror.rotating_seek_offset1048576
        vfs.zfs.vdev.mirror.rotating_seek_inc   5
        vfs.zfs.vdev.mirror.rotating_inc        0
        vfs.zfs.vdev.trim_on_init               1
        vfs.zfs.vdev.cache.bshift               16
        vfs.zfs.vdev.cache.size                 0
        vfs.zfs.vdev.cache.max                  16384
        vfs.zfs.vdev.default_ms_shift           29
        vfs.zfs.vdev.min_ms_count               16
        vfs.zfs.vdev.max_ms_count               200
        vfs.zfs.txg.timeout                     5
        vfs.zfs.spa_min_slop                    134217728
        vfs.zfs.spa_slop_shift                  5
        vfs.zfs.spa_asize_inflation             24
        vfs.zfs.deadman_enabled                 0
        vfs.zfs.deadman_checktime_ms            5000
        vfs.zfs.deadman_synctime_ms             1000000
        vfs.zfs.debug_flags                     0
        vfs.zfs.debugflags                      0
        vfs.zfs.recover                         0
        vfs.zfs.spa_load_verify_data            1
        vfs.zfs.spa_load_verify_metadata        1
        vfs.zfs.spa_load_verify_maxinflight     10000
        vfs.zfs.max_missing_tvds_scan           0
        vfs.zfs.max_missing_tvds_cachefile      2
        vfs.zfs.max_missing_tvds                0
        vfs.zfs.spa_load_print_vdev_tree        0
        vfs.zfs.ccw_retry_interval              300
        vfs.zfs.check_hostid                    1
        vfs.zfs.mg_fragmentation_threshold      85
        vfs.zfs.mg_noalloc_threshold            0
        vfs.zfs.condense_pct                    200
        vfs.zfs.metaslab_sm_blksz               4096
        vfs.zfs.metaslab.bias_enabled           1
        vfs.zfs.metaslab.lba_weighting_enabled  1
        vfs.zfs.metaslab.fragmentation_factor_enabled1
        vfs.zfs.metaslab.preload_enabled        1
        vfs.zfs.metaslab.preload_limit          3
        vfs.zfs.metaslab.unload_delay           8
        vfs.zfs.metaslab.load_pct               50
        vfs.zfs.metaslab.min_alloc_size         33554432
        vfs.zfs.metaslab.df_free_pct            4
        vfs.zfs.metaslab.df_alloc_threshold     131072
        vfs.zfs.metaslab.debug_unload           0
        vfs.zfs.metaslab.debug_load             0
        vfs.zfs.metaslab.fragmentation_threshold70
        vfs.zfs.metaslab.force_ganging          16777217
        vfs.zfs.free_bpobj_enabled              1
        vfs.zfs.free_max_blocks                 -1
        vfs.zfs.no_scrub_prefetch               0
        vfs.zfs.no_scrub_io                     0
        vfs.zfs.resilver_min_time_ms            3000
        vfs.zfs.free_min_time_ms                1000
        vfs.zfs.scan_min_time_ms                1000
        vfs.zfs.scan_idle                       50
        vfs.zfs.scrub_delay                     4
        vfs.zfs.resilver_delay                  2
        vfs.zfs.top_maxinflight                 32
        vfs.zfs.zfetch.array_rd_sz              1048576
        vfs.zfs.zfetch.max_idistance            67108864
        vfs.zfs.zfetch.max_distance             8388608
        vfs.zfs.zfetch.min_sec_reap             2
        vfs.zfs.zfetch.max_streams              8
        vfs.zfs.prefetch_disable                0
        vfs.zfs.delay_scale                     500000
        vfs.zfs.delay_min_dirty_percent         60
        vfs.zfs.dirty_data_sync                 67108864
        vfs.zfs.dirty_data_max_percent          10
        vfs.zfs.dirty_data_max_max              4294967296
        vfs.zfs.dirty_data_max                  4294967296
        vfs.zfs.max_recordsize                  1048576
        vfs.zfs.default_ibs                     17
        vfs.zfs.default_bs                      9
        vfs.zfs.send_holes_without_birth_time   1
        vfs.zfs.mdcomp_disable                  0
        vfs.zfs.per_txg_dirty_frees_percent     30
        vfs.zfs.nopwrite_enabled                1
        vfs.zfs.dedup.prefetch                  1
        vfs.zfs.dbuf_cache_lowater_pct          10
        vfs.zfs.dbuf_cache_hiwater_pct          10
        vfs.zfs.dbuf_cache_shift                5
        vfs.zfs.dbuf_cache_max_bytes            3060818688
        vfs.zfs.l2c_only_size                   0
        vfs.zfs.mfu_ghost_data_esize            44354878976
        vfs.zfs.mfu_ghost_metadata_esize        8191800320
        vfs.zfs.mfu_ghost_size                  52546679296
        vfs.zfs.mfu_data_esize                  0
        vfs.zfs.mfu_metadata_esize              84992
        vfs.zfs.mfu_size                        64779776
        vfs.zfs.mru_ghost_data_esize            48840192
        vfs.zfs.mru_ghost_metadata_esize        926682624
        vfs.zfs.mru_ghost_size                  975522816
        vfs.zfs.mru_data_esize                  47824171520
        vfs.zfs.mru_metadata_esize              5086352896
        vfs.zfs.mru_size                        52926374400
        vfs.zfs.anon_data_esize                 0
        vfs.zfs.anon_metadata_esize             0
        vfs.zfs.anon_size                       69500928
        vfs.zfs.l2arc_norw                      1
        vfs.zfs.l2arc_feed_again                1
        vfs.zfs.l2arc_noprefetch                1
        vfs.zfs.l2arc_feed_min_ms               200
        vfs.zfs.l2arc_feed_secs                 1
        vfs.zfs.l2arc_headroom                  2
        vfs.zfs.l2arc_write_boost               8388608
        vfs.zfs.l2arc_write_max                 8388608
        vfs.zfs.arc_meta_limit                  13421772800
        vfs.zfs.arc_free_target                 167651
        vfs.zfs.compressed_arc_enabled          1
        vfs.zfs.arc_grow_retry                  60
        vfs.zfs.arc_shrink_shift                7
        vfs.zfs.arc_average_blocksize           8192
        vfs.zfs.arc_no_grow_shift               5
        vfs.zfs.arc_min                         12243274752
        vfs.zfs.arc_max                         53687091200
        vfs.zfs.abd_chunk_size                  4096

------------------------------------------------------------------------


A few iterations of 'zpool iostat -v':

                 capacity     operations    bandwidth
pool          alloc   free   read  write   read  write
------------  -----  -----  -----  -----  -----  -----
mendeleev     5.02T  16.6T  3.58K      0   389M      0
  raidz2      5.02T  16.6T  3.58K      0   389M      0
    gpt/zfs0      -      -    783      0  45.9M      0
    gpt/zfs1      -      -  1.29K      0   100M      0
    gpt/zfs2      -      -    711      0  54.9M      0
    gpt/zfs3      -      -    729      0  54.3M      0
    gpt/zfs4      -      -  1.23K      0  99.7M      0
    gpt/zfs5      -      -    752      0  46.3M      0
------------  -----  -----  -----  -----  -----  -----

                 capacity     operations    bandwidth
pool          alloc   free   read  write   read  write
------------  -----  -----  -----  -----  -----  -----
mendeleev     5.02T  16.6T  8.71K      0   286M      0
  raidz2      5.02T  16.6T  8.71K      0   286M      0
    gpt/zfs0      -      -    715      0  35.6M      0
    gpt/zfs1      -      -  2.15K      0  82.7M      0
    gpt/zfs2      -      -    791      0  34.4M      0
    gpt/zfs3      -      -    706      0  33.6M      0
    gpt/zfs4      -      -  2.29K      0  82.3M      0
    gpt/zfs5      -      -    693      0  37.0M      0
------------  -----  -----  -----  -----  -----  -----

                 capacity     operations    bandwidth
pool          alloc   free   read  write   read  write
------------  -----  -----  -----  -----  -----  -----
mendeleev     5.02T  16.6T  6.28K      0   775M      0
  raidz2      5.02T  16.6T  6.28K      0   775M      0
    gpt/zfs0      -      -  1.37K      0   112M      0
    gpt/zfs1      -      -  1.57K      0   194M      0
    gpt/zfs2      -      -    802      0  87.0M      0
    gpt/zfs3      -      -    851      0  86.9M      0
    gpt/zfs4      -      -  2.03K      0   199M      0
    gpt/zfs5      -      -  1.09K      0   112M      0
------------  -----  -----  -----  -----  -----  -----

                 capacity     operations    bandwidth
pool          alloc   free   read  write   read  write
------------  -----  -----  -----  -----  -----  -----
mendeleev     5.02T  16.6T  9.15K      0   272M      0
  raidz2      5.02T  16.6T  9.15K      0   272M      0
    gpt/zfs0      -      -    507      0  23.1M      0
    gpt/zfs1      -      -  2.77K      0  80.7M      0
    gpt/zfs2      -      -    727      0  41.8M      0
    gpt/zfs3      -      -    707      0  41.2M      0
    gpt/zfs4      -      -  2.58K      0  76.1M      0
    gpt/zfs5      -      -    509      0  23.6M      0
------------  -----  -----  -----  -----  -----  -----


_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Disk/ZFS activity crash on 11.2-STABLE [SOLVED]

list
On Thu, Jul 12, 2018 at 02:42:48PM -0700, Jim Long wrote:

> On Thu, Jul 12, 2018 at 02:49:53PM -0400, Mike Tancsa wrote:
>
> --snip--
>
> > I would try and set a ceiling. On
> > RELENG_11 you dont need to reboot
> >
> > Try
> > sysctl -w vfs.zfs.arc_max=77946198016
> >
> > which shaves off 20G from what ARC can gobble up. Not sure if thats your
> > issue, but it is an issue for some users.
> >
> > If you are still hurting for caching, an SSD drive or NVME and make it a
> > caching device for your pool.
> >
> > and what does
> > zpool status
> > show ?
>
> I set the limit to the value you suggested, and the next test ran less
> than three minutes before the machine rebooted, with no crash dump produced.
>
> I further reduced the limit to 50G and it's been running for about 50 minutes
> so far.  Fingers crossed.  I do have L2ARC I can add if need be.
>
> I'll keep you posted on how this run goes.
>
> Thank you,
>
> Jim

It appears that limiting the ARC size did it.  The 'zfs send -R' was
able to complete with ARC limited to 50G, and a second run with a 60G
ARC limit also completed.

That is a very handy tunable to know about.  Being able to reduce cache
size on a running system when needed, to free up RAM, or whatever.  I
was curious to find the answer to your query about the average size of
files on the system, so I ran a 'zdb -b' on the pool.  That process
began to page out large amounts of RAM into swap, which was making the
system rather sluggish, especially once I decided to kill the zdb
process.  By dropping the ARC size limit, I was able to temporarily free
some RAM so that the process could succumb to the SIGKILL signal.

Thank you very much for your advice in guiding me to this resolution!

Jim

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Disk/ZFS activity crash on 11.2-STABLE [SOLVED]

mdtancsa
On 7/13/2018 3:10 PM, Jim Long wrote:
>
> It appears that limiting the ARC size did it.  The 'zfs send -R' was
> able to complete with ARC limited to 50G, and a second run with a 60G
> ARC limit also completed.

Great! I am glad that helped. Its helpful to me too as I have some
systems I am upgrading to RELENG_11 from RELENG_10 soon.  I was aware of
this particular issue, but was worried perhaps your crashes were related
to something else I might run into :)

If you ever have a system with a LOT of small files and directories, a
handy value to tune / keep an eye on is the mix allocated to metadata vs
file data. vfs.zfs.arc_meta_limit.  You can tell when doing things like
"ls" in a directory takes a LONG time to list files. In my case, I had
many directories with 50,000+ files.

Also things like 'zfs list -t snapshot' start to take a long time.

        ---Mike


--
-------------------
Mike Tancsa, tel +1 519 651 3400 x203
Sentex Communications, [hidden email]
Providing Internet services since 1994 www.sentex.net
Cambridge, Ontario Canada
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Disk/ZFS activity crash on 11.2-STABLE [SOLVED]

list
On Fri, Jul 13, 2018 at 03:22:39PM -0400, Mike Tancsa wrote:
>
> If you ever have a system with a LOT of small files and directories, a
> handy value to tune / keep an eye on is the mix allocated to metadata vs
> file data. vfs.zfs.arc_meta_limit.  You can tell when doing things like
> "ls" in a directory takes a LONG time to list files. In my case, I had
> many directories with 50,000+ files.
>
> Also things like 'zfs list -t snapshot' start to take a long time.

I think I already have that symptom on another new server, a backup
retention server.  It's slow (CPU) and fat (disk).

# time zfs list -Hrt snap | wc -l
   27365

real    2m47.811s
user    0m1.757s
sys     0m20.828s


Almost three minutes to list all snapshots found.  So when that
symptomatic slowness appears, is the tweak to *raise* arc_meta_limit ?
I don't immediately see how to tell what the arc_meta usage is, and thus
see how close it is to the limit.

From that storage server ("electron"):

ARC Efficiency:                                 20.18m
        Cache Hit Ratio:                91.20%  18.40m
        Cache Miss Ratio:               8.80%   1.78m
        Actual Hit Ratio:               91.18%  18.40m

        Data Demand Efficiency:         87.10%  11.95k

        CACHE HITS BY CACHE LIST:
          Anonymously Used:             0.02%   3.15k
          Most Recently Used:           0.30%   55.95k
          Most Frequently Used:         99.68%  18.34m
          Most Recently Used Ghost:     0.00%   0
          Most Frequently Used Ghost:   0.00%   0

        CACHE HITS BY DATA TYPE:
          Demand Data:                  0.06%   10.41k
          Prefetch Data:                0.00%   0
          Demand Metadata:              99.93%  18.39m
          Prefetch Metadata:            0.02%   3.15k

        CACHE MISSES BY DATA TYPE:
          Demand Data:                  0.09%   1.54k
          Prefetch Data:                0.00%   0
          Demand Metadata:              99.73%  1.77m
          Prefetch Metadata:            0.19%   3.30k

# sysctl -a | grep arc | grep ^vfs.zfs
vfs.zfs.l2arc_norw: 1
vfs.zfs.l2arc_feed_again: 1
vfs.zfs.l2arc_noprefetch: 1
vfs.zfs.l2arc_feed_min_ms: 200
vfs.zfs.l2arc_feed_secs: 1
vfs.zfs.l2arc_headroom: 2
vfs.zfs.l2arc_write_boost: 8388608
vfs.zfs.l2arc_write_max: 8388608
vfs.zfs.arc_meta_limit: 16432737280
vfs.zfs.arc_free_target: 113124
vfs.zfs.compressed_arc_enabled: 1
vfs.zfs.arc_grow_retry: 60
vfs.zfs.arc_shrink_shift: 7
vfs.zfs.arc_average_blocksize: 8192
vfs.zfs.arc_no_grow_shift: 5
vfs.zfs.arc_min: 8216368640
vfs.zfs.arc_max: 65730949120

# top | head -8
last pid:   943;  load averages:  0.14,  0.15,  0.10  up 0+00:30:43    14:42:26
22 processes:  1 running, 21 sleeping

Mem: 16M Active, 13M Inact, 1063M Wired, 61G Free
ARC: 374M Total, 214M MFU, 63M MRU, 32K Anon, 5614K Header, 91M Other
     79M Compressed, 223M Uncompressed, 2.81:1 Ratio
Swap: 8192M Total, 8192M Free


Thanks again, Mike.


Jim

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Disk/ZFS activity crash on 11.2-STABLE [SOLVED]

Shane Ambler-5
In reply to this post by mdtancsa
On 14/07/2018 04:52, Mike Tancsa wrote:

> On 7/13/2018 3:10 PM, Jim Long wrote:
>>
>> It appears that limiting the ARC size did it.  The 'zfs send -R' was
>> able to complete with ARC limited to 50G, and a second run with a 60G
>> ARC limit also completed.
>
> Great! I am glad that helped. Its helpful to me too as I have some
> systems I am upgrading to RELENG_11 from RELENG_10 soon.  I was aware of
> this particular issue, but was worried perhaps your crashes were related
> to something else I might run into :)

To add my 2c -

I recently realised that the arc_max defaults allow all ram to be
allocated as wired, which is bad.

arc_max should not be set higher than physical ram - max_wired - buffer

It is time I put this into a bug report -
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=229764

--
FreeBSD - the place to B...Storing Data

Shane Ambler

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Disk/ZFS activity crash on 11.2-STABLE [SOLVED]

Shane Ambler-5
On 16/07/2018 01:21, Kevin P. Neal wrote:

> On Sat, Jul 14, 2018 at 03:11:21PM +0930, Shane Ambler wrote:
>> To add my 2c -
>>
>> I recently realised that the arc_max defaults allow all ram to be
>> allocated as wired, which is bad.
>>
>> arc_max should not be set higher than physical ram - max_wired - buffer
>>
>> It is time I put this into a bug report -
>> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=229764
>
> Does this affect 11.2-RELEASE? Is it new?
>
> I've got a machine with 8GB memory that I need to not fall over after an
> upgrade from 11.1-RELEASE.
>

It is no different between any recent system releases. You shouldn't see
any difference between 11.1 and 11.2 in this matter.

If you are willing to buildworld, I would suggest looking at
https://reviews.freebsd.org/D7538
For me, I see much better memory use and less swapping, improvements
from this should prevent wired amounts being an issue.

--
FreeBSD - the place to B...Software Developing

Shane Ambler

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[hidden email]"