[Bug 213903] Kernel crashes from turnstile_broadcast (/usr/src/sys/kern/subr_turnstile.c:837)

classic Classic list List threaded Threaded
35 messages Options
12
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Bug 213903] Kernel crashes from turnstile_broadcast (/usr/src/sys/kern/subr_turnstile.c:837)

bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=213903

            Bug ID: 213903
           Summary: Kernel crashes from turnstile_broadcast
                    (/usr/src/sys/kern/subr_turnstile.c:837)
           Product: Base System
           Version: CURRENT
          Hardware: Any
                OS: Any
            Status: New
          Severity: Affects Only Me
          Priority: ---
         Component: kern
          Assignee: [hidden email]
          Reporter: [hidden email]

I am currently experiencing semi-regular kernel crashes on my FreeBSD
12-current machine. I am new to kernel debugging, and hoping someone can have a
look at the debugging output below to point me in the direction of what the
problem might be.

My machine is a FreeNAS-mini from iXsystems which I have formatted and
installed stock FreeBSD onto. My kernel is the default generic-nodebug with the
VIMAGE options added.

$ uname -a
FreeBSD freenas.woods.am 12.0-CURRENT FreeBSD 12.0-CURRENT #0 r305311M: Sat Sep
 3 12:29:01 AWST 2016    
[hidden email]:/usr/obj/usr/src/sys/GENERIC-NODEBUG-VIMAGE  amd64

$ cat /usr/src/sys/amd64/conf/GENERIC-NODEBUG-VIMAGE
# SPARTICUS -- WITNESS and INVARIANTS free kernel configuration file
#               for FreeBSD/amd64

include GENERIC-NODEBUG

ident   GENERIC-NODEBUG-VIMAGE

#nooptions      SCTP    # Stream Control Transmission Protocol
options         VIMAGE  # VNET/Vimage support


Output from kernel crash dump debug with kgdb below:

/usr/obj/usr/src/sys/GENERIC-NODEBUG-VIMAGE)# kgdb kernel.debug
/var/crash/vmcore.last
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:
kernel trap 12 with interrupts disabled


Fatal trap 12: page fault while in kernel mode
cpuid = 2; apic id = 04
fault virtual address   = 0x30
fault code              = supervisor read data, page not present
instruction pointer     = 0x20:0xffffffff80b4d91c
stack pointer           = 0x28:0xfffffe046813a440
frame pointer           = 0x28:0xfffffe046813a470
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = resume, IOPL = 0
current process         = 33487 (sh)
Uptime: 15m16s
Dumping 1664 out of 16338 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%

Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/zfs.ko...Reading
symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/zfs.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/zfs.ko
Reading symbols from
/boot/kernel.GENERIC-NODEBUG-VIMAGE/opensolaris.ko...Reading symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/opensolaris.ko.debug...
done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/opensolaris.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/geom_eli.ko...Reading
symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/geom_eli.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/geom_eli.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/accf_http.ko...Reading
symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/accf_http.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/accf_http.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/coretemp.ko...Reading
symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/coretemp.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/coretemp.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/aesni.ko...Reading
symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/aesni.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/aesni.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/if_bridge.ko...Reading
symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/if_bridge.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/if_bridge.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/bridgestp.ko...Reading
symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/bridgestp.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/bridgestp.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/ums.ko...Reading
symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/ums.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/ums.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/netgraph.ko...Reading
symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/netgraph.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/netgraph.ko
Reading symbols from
/boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_netflow.ko...Reading symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_netflow.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_netflow.ko
Reading symbols from
/boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_ksocket.ko...Reading symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_ksocket.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_ksocket.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_ether.ko...Reading
symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_ether.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_ether.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_socket.ko...Reading
symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_socket.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_socket.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/linux.ko...Reading
symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/linux.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/linux.ko
Reading symbols from
/boot/kernel.GENERIC-NODEBUG-VIMAGE/linux_common.ko...Reading symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/linux_common.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/linux_common.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/linux64.ko...Reading
symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/linux64.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/linux64.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/fdescfs.ko...Reading
symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/fdescfs.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/fdescfs.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/if_epair.ko...Reading
symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/if_epair.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/if_epair.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/nullfs.ko...Reading
symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/nullfs.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/nullfs.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/tmpfs.ko...Reading
symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/tmpfs.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/tmpfs.ko
#0  doadump (textdump=1) at pcpu.h:221
221             __asm("movq %%gs:%1,%0" : "=r" (td)
(kgdb) list *0xffffffff80b4d91c
0xffffffff80b4d91c is in turnstile_broadcast
(/usr/src/sys/kern/subr_turnstile.c:837).
832
833             /*
834              * Transfer the blocked list to the pending list.
835              */
836             mtx_lock_spin(&td_contested_lock);
837             TAILQ_CONCAT(&ts->ts_pending, &ts->ts_blocked[queue],
td_lockq);
838             mtx_unlock_spin(&td_contested_lock);
839
840             /*
841              * Give a turnstile to each thread.  The last thread gets
Current language:  auto; currently minimal
(kgdb) backtrace
#0  doadump (textdump=1) at pcpu.h:221
#1  0xffffffff80aea40e in kern_reboot (howto=260) at
/usr/src/sys/kern/kern_shutdown.c:366
#2  0xffffffff80aea9db in vpanic (fmt=<value optimized out>, ap=<value
optimized out>) at /usr/src/sys/kern/kern_shutdown.c:759
#3  0xffffffff80aea813 in panic (fmt=0x0) at
/usr/src/sys/kern/kern_shutdown.c:690
#4  0xffffffff8039e197 in db_panic (addr=<value optimized out>,
have_addr=false, count=0, modif=0x0) at /usr/src/sys/ddb/db_command.c:486
#5  0xffffffff8039d689 in db_command (cmd_table=<value optimized out>) at
/usr/src/sys/ddb/db_command.c:453
#6  0xffffffff8039d3e4 in db_command_loop () at
/usr/src/sys/ddb/db_command.c:506
#7  0xffffffff803a053b in db_trap (type=<value optimized out>, code=<value
optimized out>) at /usr/src/sys/ddb/db_main.c:251
#8  0xffffffff80b36b33 in kdb_trap (type=<value optimized out>, code=<value
optimized out>, tf=<value optimized out>) at /usr/src/sys/kern/subr_kdb.c:654
#9  0xffffffff80fdd441 in trap_fatal (frame=0xfffffe046813a390, eva=48) at
/usr/src/sys/amd64/amd64/trap.c:836
#10 0xffffffff80fdd673 in trap_pfault (frame=0xfffffe046813a390, usermode=0) at
/usr/src/sys/amd64/amd64/trap.c:691
#11 0xffffffff80fdcbfc in trap (frame=0xfffffe046813a390) at
/usr/src/sys/amd64/amd64/trap.c:442
#12 0xffffffff80fbf491 in calltrap () at
/usr/src/sys/amd64/amd64/exception.S:236
#13 0xffffffff80b4d91c in turnstile_broadcast (ts=0x0, queue=1) at
/usr/src/sys/kern/subr_turnstile.c:837
#14 0xffffffff80ae5e1f in __rw_wunlock_hard (c=0xfffff803f886d960, tid=<value
optimized out>, file=<value optimized out>, line=<value optimized out>)
    at /usr/src/sys/kern/kern_rwlock.c:1027
#15 0xffffffff80e525dc in vm_map_delete (map=<value optimized out>,
start=<value optimized out>, end=<value optimized out>) at
/usr/src/sys/vm/vm_map.c:2960
#16 0xffffffff80e54477 in vm_map_remove (map=0xfffff8035540f000,
start=140737488355328, end=1) at /usr/src/sys/vm/vm_map.c:3077
#17 0xffffffff80a9863f in exec_new_vmspace (imgp=0xfffffe046813a860,
sv=0xffffffff81a596e8) at /usr/src/sys/kern/kern_exec.c:1096
#18 0xffffffff80a6ced8 in exec_elf64_imgact (imgp=<value optimized out>) at
/usr/src/sys/kern/imgact_elf.c:896
#19 0xffffffff80a9670d in kern_execve (td=<value optimized out>, args=<value
optimized out>, mac_p=0x0) at /usr/src/sys/kern/kern_exec.c:603
#20 0xffffffff80a95b9c in sys_execve (td=0xfffff8032893aa00,
uap=0xfffffe046813ab80) at /usr/src/sys/kern/kern_exec.c:219
#21 0xffffffff80fddde8 in amd64_syscall (td=<value optimized out>, traced=0) at
subr_syscall.c:135
#22 0xffffffff80fbf77b in Xfast_syscall () at
/usr/src/sys/amd64/amd64/exception.S:396
#23 0x0000000800b468ea in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb) up 11
#11 0xffffffff80fdcbfc in trap (frame=0xfffffe046813a390) at
/usr/src/sys/amd64/amd64/trap.c:442
442                             (void) trap_pfault(frame, FALSE);
(kgdb) list
437
438                     KASSERT(cold || td->td_ucred != NULL,
439                         ("kernel trap doesn't have ucred"));
440                     switch (type) {
441                     case T_PAGEFLT:                 /* page fault */
442                             (void) trap_pfault(frame, FALSE);
443                             goto out;
444
445                     case T_DNA:
446                             if (PCB_USER_FPU(td->td_pcb))
(kgdb) print td
$1 = (struct thread *) 0xfffff8032893aa00
(kgdb) print td->td_ucred
$2 = (struct ucred *) 0xfffff8004005ec00
(kgdb) print type
$3 = 12
(kgdb) up
#12 0xffffffff80fbf491 in calltrap () at
/usr/src/sys/amd64/amd64/exception.S:236
236             call    trap_check
Current language:  auto; currently asm
(kgdb) up
#13 0xffffffff80b4d91c in turnstile_broadcast (ts=0x0, queue=1) at
/usr/src/sys/kern/subr_turnstile.c:837
837             TAILQ_CONCAT(&ts->ts_pending, &ts->ts_blocked[queue],
td_lockq);
Current language:  auto; currently minimal
(kgdb) up
#14 0xffffffff80ae5e1f in __rw_wunlock_hard (c=0xfffff803f886d960, tid=<value
optimized out>, file=<value optimized out>, line=<value optimized out>)
    at /usr/src/sys/kern/kern_rwlock.c:1027
1027            turnstile_broadcast(ts, queue);
(kgdb) up
#15 0xffffffff80e525dc in vm_map_delete (map=<value optimized out>,
start=<value optimized out>, end=<value optimized out>) at
/usr/src/sys/vm/vm_map.c:2960
2960                    VM_OBJECT_WUNLOCK(object);
(kgdb) up
#16 0xffffffff80e54477 in vm_map_remove (map=0xfffff8035540f000,
start=140737488355328, end=1) at /usr/src/sys/vm/vm_map.c:3077
3077            result = vm_map_delete(map, start, end);
(kgdb) up
#17 0xffffffff80a9863f in exec_new_vmspace (imgp=0xfffffe046813a860,
sv=0xffffffff81a596e8) at /usr/src/sys/kern/kern_exec.c:1096
1096                    vm_map_remove(map, vm_map_min(map), vm_map_max(map));
(kgdb) up
#18 0xffffffff80a6ced8 in exec_elf64_imgact (imgp=<value optimized out>) at
/usr/src/sys/kern/imgact_elf.c:896
896             error = exec_new_vmspace(imgp, sv);
(kgdb) up
#19 0xffffffff80a9670d in kern_execve (td=<value optimized out>, args=<value
optimized out>, mac_p=0x0) at /usr/src/sys/kern/kern_exec.c:603
603                     error = (*execsw[i]->ex_imgact)(imgp);
(kgdb) up
#20 0xffffffff80a95b9c in sys_execve (td=0xfffff8032893aa00,
uap=0xfffffe046813ab80) at /usr/src/sys/kern/kern_exec.c:219
219                     error = kern_execve(td, &args, NULL);
(kgdb) up
#21 0xffffffff80fddde8 in amd64_syscall (td=<value optimized out>, traced=0) at
subr_syscall.c:135
135                     error = (sa->callp->sy_call)(td, sa->args);
(kgdb) up
#22 0xffffffff80fbf77b in Xfast_syscall () at
/usr/src/sys/amd64/amd64/exception.S:396
396             call    amd64_syscall
Current language:  auto; currently asm
(kgdb) up
#23 0x0000000800b468ea in ?? ()
(kgdb) up
Initial frame selected; you cannot go up.
(kgdb) quit

After being requested for more info from Mateusz Guzik <[hidden email]>, I
also performed the following kgdb command:
(kgdb) f 14
#14 0xffffffff80ae5e1f in __rw_wunlock_hard (c=0xfffff803f886d960, tid=<value
optimized out>, file=<value optimized out>, line=<value optimized out>)
    at /usr/src/sys/kern/kern_rwlock.c:1027
1027            turnstile_broadcast(ts, queue);
Current language:  auto; currently minimal
(kgdb) x/xg c
0xfffff803f886d960:     0xfffff8032893aa00

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Bug 213903] Kernel crashes from turnstile_broadcast (/usr/src/sys/kern/subr_turnstile.c:837)

bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=213903

Ben Woods <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email],
                   |                            |[hidden email]
           Keywords|                            |crash
             Status|New                         |Open

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Bug 213903] Kernel crashes from turnstile_broadcast (/usr/src/sys/kern/subr_turnstile.c:837)

bugzilla-noreply
In reply to this post by bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=213903

Shawn Webb <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]

--- Comment #1 from Shawn Webb <[hidden email]> ---
We at OPNsense have a user that might be affected by this as well. Here's the
bug report on OPNsense's side:
https://forum.opnsense.org/index.php?topic=4315.0

Any updates on this issue?

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Bug 213903] Kernel crashes from turnstile_broadcast (/usr/src/sys/kern/subr_turnstile.c:837)

bugzilla-noreply
In reply to this post by bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=213903

--- Comment #2 from Ben Woods <[hidden email]> ---
I still experience the issue about 2-3 times a month. Not sure how to
investigate further...

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Bug 213903] Kernel crashes from turnstile_broadcast (/usr/src/sys/kern/subr_turnstile.c:837)

bugzilla-noreply
In reply to this post by bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=213903

--- Comment #3 from Franco Fichtner <[hidden email]> ---
We have over a dozen user reports on this collected in two weeks, some with
daily crashes.  Trying to bring in a developer now...

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Bug 213903] Kernel crashes from turnstile_broadcast (/usr/src/sys/kern/subr_turnstile.c:837)

bugzilla-noreply
In reply to this post by bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=213903

Mateusz Guzik <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]

--- Comment #4 from Mateusz Guzik <[hidden email]> ---
Please reproduce with:
https://people.freebsd.org/~mjg/patches/rwlock-debug.diff

the patch is against 11.0

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Bug 213903] Kernel crashes from turnstile_broadcast (/usr/src/sys/kern/subr_turnstile.c:837)

bugzilla-noreply
In reply to this post by bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=213903

--- Comment #5 from Franco Fichtner <[hidden email]> ---
Thank you. A public call for testing will be out today based on your diff. :)


Cheers,
Franco

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Bug 213903] Kernel crashes from turnstile_broadcast (/usr/src/sys/kern/subr_turnstile.c:837)

bugzilla-noreply
In reply to this post by bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=213903

John Baldwin <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]

--- Comment #6 from John Baldwin <[hidden email]> ---
In kgdb, can you go up to the frame that faulted (13) and print out 'ts' and
'queue'?

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Bug 213903] Kernel crashes from turnstile_broadcast (/usr/src/sys/kern/subr_turnstile.c:837)

bugzilla-noreply
In reply to this post by bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=213903

--- Comment #7 from Franco Fichtner <[hidden email]> ---
So far we haven't heard back from users regarding the debug info. But we have a
suspect:
https://lists.freebsd.org/pipermail/freebsd-current/2015-November/058122.html

Some CPU info of previous user reports:

CPU: Intel(R) Atom(TM) CPU  C2758  @ 2.40GHz (2400.06-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x406d8  Family=0x6  Model=0x4d  Stepping=8
 
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
 
Features2=0x43d8e3bf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,MOVBE,POPCNT,TSCDLT,AESNI,RDRAND>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x101<LAHF,Prefetch>
  Structured Extended Features=0x2282<TSCADJ,SMEP,ERMS,NFPUSG>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
real memory  = 17179869184 (16384 MB)
avail memory = 16515608576 (15750 MB)


CPU: Intel(R) Celeron(R) CPU  N2930  @ 1.83GHz (1833.38-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x30678  Family=0x6  Model=0x37  Stepping=8
 
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
 
Features2=0x41d8e3bf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,MOVBE,POPCNT,TSCDLT,RDRAND>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x101<LAHF,Prefetch>
  Structured Extended Features=0x2282<TSCADJ,SMEP,ERMS,NFPUSG>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
real memory  = 8589934592 (8192 MB)
avail memory = 8072794112 (7698 MB)


CPU: Intel(R) Celeron(R) CPU  J1900  @ 1.99GHz (2000.05-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x30678  Family=0x6  Model=0x37  Stepping=8
 
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
 
Features2=0x41d8e3bf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,MOVBE,POPCNT,TSCDLT,RDRAND>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x101<LAHF,Prefetch>
  Structured Extended Features=0x2282<TSCADJ,SMEP,ERMS,NFPUSG>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
real memory  = 4294967296 (4096 MB)
avail memory = 3961688064 (3778 MB)

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Bug 213903] Kernel crashes from turnstile_broadcast (/usr/src/sys/kern/subr_turnstile.c:837)

bugzilla-noreply
In reply to this post by bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=213903

--- Comment #8 from Ben Woods <[hidden email]> ---
(In reply to John Baldwin from comment #6)
> In kgdb, can you go up to the frame that faulted (13) and print out 'ts' and 'queue'?

Apologies for the delay in replying, I had deleted this core, and moved
poudriere to another box to avoid these crashes. I had to reinstall poudriere
to recreate the high load on the server in order to reproduce this kernel
crash.

I have printed the values you requested below. Please let me know if you would
like any more info. I'll keep the core handy this time :)


# kgdb /usr/lib/debug/boot/kernel.GENERIC-NODEBUG-VIMAGE/kernel.debug
/var/crash/vmcore.last
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:
kernel trap 12 with interrupts disabled


Fatal trap 12: page fault while in kernel mode
cpuid = 2; apic id = 04
fault virtual address   = 0x30
fault code              = supervisor read data, page not present
instruction pointer     = 0x20:0xffffffff80b2db5c
stack pointer           = 0x28:0xfffffe04687ad8c0
frame pointer           = 0x28:0xfffffe04687ad8e0
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = resume, IOPL = 0
current process         = 98303 (sh)

Reading symbols from
/usr/lib/debug/boot/kernel.GENERIC-NODEBUG-VIMAGE/zfs.ko.debug...done.
Loaded symbols for
/usr/lib/debug/boot/kernel.GENERIC-NODEBUG-VIMAGE/zfs.ko.debug
Reading symbols from
/boot/kernel.GENERIC-NODEBUG-VIMAGE/opensolaris.ko...Reading symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/opensolaris.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/opensolaris.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/geom_eli.ko...Reading
symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/geom_eli.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/geom_eli.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/accf_http.ko...Reading
symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/accf_http.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/accf_http.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/coretemp.ko...Reading
symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/coretemp.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/coretemp.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/aesni.ko...Reading
symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/aesni.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/aesni.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/fdescfs.ko...Reading
symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/fdescfs.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/fdescfs.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/if_bridge.ko...Reading
symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/if_bridge.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/if_bridge.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/bridgestp.ko...Reading
symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/bridgestp.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/bridgestp.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/ums.ko...Reading
symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/ums.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/ums.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/linux.ko...Reading
symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/linux.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/linux.ko
Reading symbols from
/boot/kernel.GENERIC-NODEBUG-VIMAGE/linux_common.ko...Reading symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/linux_common.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/linux_common.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/linux64.ko...Reading
symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/linux64.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/linux64.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/cpuctl.ko...Reading
symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/cpuctl.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/cpuctl.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/nullfs.ko...Reading
symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/nullfs.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/nullfs.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/tmpfs.ko...Reading
symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/tmpfs.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/tmpfs.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/if_epair.ko...Reading
symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/if_epair.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/if_epair.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/linprocfs.ko...Reading
symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/linprocfs.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/linprocfs.ko
#0  doadump (textdump=0) at pcpu.h:222
222     pcpu.h: No such file or directory.
        in pcpu.h
(kgdb) bt
#0  doadump (textdump=0) at pcpu.h:222
#1  0xffffffff803a571c in db_fncall (dummy1=<value optimized out>,
dummy2=<value optimized out>, dummy3=<value optimized out>, dummy4=<value
optimized out>) at /usr/src/sys/ddb/db_command.c:581
#2  0xffffffff803a529f in db_command (cmd_table=<value optimized out>) at
/usr/src/sys/ddb/db_command.c:453
#3  0xffffffff803a5014 in db_command_loop () at
/usr/src/sys/ddb/db_command.c:506
#4  0xffffffff803a806f in db_trap (type=<value optimized out>, code=<value
optimized out>) at /usr/src/sys/ddb/db_main.c:248
#5  0xffffffff80b168b3 in kdb_trap (type=<value optimized out>, code=<value
optimized out>, tf=<value optimized out>) at /usr/src/sys/kern/subr_kdb.c:654
#6  0xffffffff80fbb322 in trap_fatal (frame=0xfffffe04687ad800, eva=48) at
/usr/src/sys/amd64/amd64/trap.c:796
#7  0xffffffff80fbb52c in trap_pfault (frame=0xfffffe04687ad800, usermode=0) at
/usr/src/sys/amd64/amd64/trap.c:658
#8  0xffffffff80fbabe0 in trap (frame=0xfffffe04687ad800) at
/usr/src/sys/amd64/amd64/trap.c:421
#9  0xffffffff80f9d9f1 in calltrap () at
/usr/src/sys/amd64/amd64/exception.S:236
#10 0xffffffff80b2db5c in turnstile_broadcast (ts=0x0, queue=1) at
/usr/src/sys/kern/subr_turnstile.c:840
#11 0xffffffff80ac693f in __rw_wunlock_hard (c=0xfffff803574082e8, tid=<value
optimized out>, file=<value optimized out>, line=<value optimized out>) at
/usr/src/sys/kern/kern_rwlock.c:1051
#12 0xffffffff80e317ec in vm_map_delete (map=<value optimized out>,
start=<value optimized out>, end=<value optimized out>) at
/usr/src/sys/vm/vm_map.c:2956
#13 0xffffffff80e2f40e in vmspace_exit (td=<value optimized out>) at
/usr/src/sys/vm/vm_map.c:3073
#14 0xffffffff80a7bd19 in exit1 (td=<value optimized out>, rval=<value
optimized out>, signo=<value optimized out>) at
/usr/src/sys/kern/kern_exit.c:399
#15 0xffffffff80a7b4cd in sys_sys_exit (td=<value optimized out>, uap=<value
optimized out>) at /usr/src/sys/kern/kern_exit.c:178
#16 0xffffffff80fbbcee in amd64_syscall (td=0xfffff801ec058a00, traced=0) at
subr_syscall.c:135
#17 0xffffffff80f9dcdb in Xfast_syscall () at
/usr/src/sys/amd64/amd64/exception.S:396
#18 0x0000000800b661fa in ?? ()
Previous frame inner to this frame (corrupt stack?)
Current language:  auto; currently minimal
(kgdb) up 10
#10 0xffffffff80b2db5c in turnstile_broadcast (ts=0x0, queue=1) at
/usr/src/sys/kern/subr_turnstile.c:840
840             TAILQ_CONCAT(&ts->ts_pending, &ts->ts_blocked[queue],
td_lockq);
(kgdb) print ts
$1 = (struct turnstile *) 0x0
(kgdb) print queue
$2 = 1
(kgdb) %

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Bug 213903] Kernel crashes from turnstile_broadcast (/usr/src/sys/kern/subr_turnstile.c:837)

bugzilla-noreply
In reply to this post by bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=213903

--- Comment #9 from Ben Woods <[hidden email]> ---
(In reply to Franco Fichtner from comment #7)
Indeed, my CPU is one of these. I am experiencing this on a FreeNAS mini bought
from IxSystems.

Some details from my dmesg:
CPU: Intel(R) Atom(TM) CPU  C2750  @ 2.40GHz (2400.14-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x406d8  Family=0x6  Model=0x4d  Stepping=8
 
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
 
Features2=0x43d8e3bf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,MOVBE,POPCNT,TSCDLT,AESNI,RDRAND>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x101<LAHF,Prefetch>
  Structured Extended Features=0x2282<TSCADJ,SMEP,ERMS,NFPUSG>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
real memory  = 17179869184 (16384 MB)
avail memory = 16534667264 (15768 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <INTEL  TIANO   >
WARNING: L1 data cache covers less APIC IDs than a core
0 < 1
FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
FreeBSD/SMP: 1 package(s) x 8 core(s)
random: unblocking device.
WARNING: VIMAGE (virtualized network stack) is a highly experimental feature.
ACPI BIOS Warning (bug): Invalid length for FADT/Pm1aControlBlock: 32, using
default 16 (20170119/tbfadt-742)
ioapic0 <Version 2.0> irqs 0-23 on motherboard
SMP: AP CPU #6 Launched!
SMP: AP CPU #7 Launched!
SMP: AP CPU #4 Launched!
SMP: AP CPU #5 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #1 Launched!
SMP: AP CPU #3 Launched!

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Bug 213903] Kernel crashes from turnstile_broadcast (/usr/src/sys/kern/subr_turnstile.c:837)

bugzilla-noreply
In reply to this post by bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=213903

--- Comment #10 from Mateusz Guzik <[hidden email]> ---
turnstile of NULL value and the queue of 1 are consistent with the previous
core showing the lock value being that of the curthread.

Can you please reproduce with this
https://people.freebsd.org/~mjg/patches/rwlock-debug-head.diff

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Bug 213903] Kernel crashes from turnstile_broadcast (/usr/src/sys/kern/subr_turnstile.c:837)

bugzilla-noreply
In reply to this post by bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=213903

Cassiano Peixoto <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email],
                   |                            |[hidden email]

--- Comment #11 from Cassiano Peixoto <[hidden email]> ---
Guys,

I'm having the same issue here on FreeBSD 10.3-STABLE. I'm using Atom C2758 as
well. It has began after 10.3 update. It's very serious issue because many
production servers are crashing. Can someone take a look please?

Thanks.

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Bug 213903] Kernel crashes from turnstile_broadcast (/usr/src/sys/kern/subr_turnstile.c:837)

bugzilla-noreply
In reply to this post by bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=213903

--- Comment #12 from Franco Fichtner <[hidden email]> ---
r301157 was backported to 10-STABLE, but 10.3 is unaffected.  There is no
10.3-STABLE.  Which one did you mean?

From our experience r301157 is the bad commit as the panics have disappeared in
our latest OPNsense version which reverted the rwlock bits of this particular
patch.

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Bug 213903] Kernel crashes from turnstile_broadcast (/usr/src/sys/kern/subr_turnstile.c:837)

bugzilla-noreply
In reply to this post by bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=213903

--- Comment #13 from Cassiano Peixoto <[hidden email]> ---
(In reply to Franco Fichtner from comment #12)
Hi Franco,

I don't know exactly which svn version i'm using, because when i run uname -a
it doesn't show me. But anyway i updated my FreeBSD 10.3 on February 6th. Is it
makes sense to you?  How can i revert this commit?

Thanks.

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Bug 213903] Kernel crashes from turnstile_broadcast (/usr/src/sys/kern/subr_turnstile.c:837)

bugzilla-noreply
In reply to this post by bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=213903

--- Comment #14 from Franco Fichtner <[hidden email]> ---
Hi Cassiano,

What's your output of uname -v?

Can you make sure to include a backtrace here from ddb? type "bt" at the prompt
when the panic happens. It may be related but not the same code path.


Cheers,
Franco

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Bug 213903] Kernel crashes from turnstile_broadcast (/usr/src/sys/kern/subr_turnstile.c:837)

bugzilla-noreply
In reply to this post by bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=213903

--- Comment #15 from Cassiano Peixoto <[hidden email]> ---
(In reply to Franco Fichtner from comment #14)
Hi Franco,

Here it is:
FreeBSD 10.3-STABLE #4: Mon Feb  6 09:29:52 BRST 2017    
[hidden email]:/usr/obj/usr/src/sys/GENERIC

My debug bellow:

# kgdb kernel.debug /var/crash/vmcore.last
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:
kernel trap 12 with interrupts disabled


Fatal trap 12: page fault while in kernel mode
cpuid = 7; apic id = 0e
fault virtual address   = 0x30
fault code              = supervisor read data, page not present
instruction pointer     = 0x20:0xffffffff80b2b4fa
stack pointer           = 0x28:0xfffffe0237a4f450
frame pointer           = 0x28:0xfffffe0237a4f480
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = resume, IOPL = 0
current process         = 79530 (sh)
trap number             = 12
panic: page fault
cpuid = 7
KDB: stack backtrace:
#0 0xffffffff80b16230 at kdb_backtrace+0x60
#1 0xffffffff80ad7036 at vpanic+0x126
#2 0xffffffff80ad6f03 at panic+0x43
#3 0xffffffff80f810cd at trap_fatal+0x35d
#4 0xffffffff80f813e8 at trap_pfault+0x308
#5 0xffffffff80f80a2a at trap+0x47a
#6 0xffffffff80f661dc at calltrap+0x8
#7 0xffffffff80ad4d80 at __rw_wunlock_hard+0x90
#8 0xffffffff80dffd9a at vm_map_delete+0x33a
#9 0xffffffff80e01b47 at vm_map_remove+0x47
#10 0xffffffff80a96759 at exec_new_vmspace+0x1e9
#11 0xffffffff80a73284 at exec_elf64_imgact+0xa44
#12 0xffffffff80a94ec4 at kern_execve+0x7d4
#13 0xffffffff80a9438c at sys_execve+0x4c
#14 0xffffffff80f81b00 at amd64_syscall+0x450
#15 0xffffffff80f664cb at Xfast_syscall+0xfb
Uptime: 19h0m34s
Dumping 1063 out of 8149 MB: (CTRL-C to abort)
..2%..11%..22%..31%..41%..52%..61%..71%..82%..91%

Reading symbols from /boot/kernel.off/coretemp.ko.symbols...done.
Loaded symbols for /boot/kernel.off/coretemp.ko.symbols
Reading symbols from /boot/modules/plcm.ko...done.
Loaded symbols for /boot/modules/plcm.ko
#0  doadump (textdump=<value optimized out>) at pcpu.h:219
219     pcpu.h: No such file or directory.
        in pcpu.h
(kgdb) list *0xffffffff80b2b4fa
0xffffffff80b2b4fa is in turnstile_broadcast
(/usr/src/sys/kern/subr_turnstile.c:838).
833    
834             /*
835              * Transfer the blocked list to the pending list.
836              */
837             mtx_lock_spin(&td_contested_lock);
838             TAILQ_CONCAT(&ts->ts_pending, &ts->ts_blocked[queue],
td_lockq);
839             mtx_unlock_spin(&td_contested_lock);
840    
841             /*
842              * Give a turnstile to each thread.  The last thread gets
Current language:  auto; currently minimal
(kgdb) bt
#0  doadump (textdump=<value optimized out>) at pcpu.h:219
#1  0xffffffff80ad6c53 in kern_reboot (howto=260) at
/usr/src/sys/kern/kern_shutdown.c:486
#2  0xffffffff80ad7075 in vpanic (fmt=<value optimized out>, ap=<value
optimized out>) at /usr/src/sys/kern/kern_shutdown.c:889
#3  0xffffffff80ad6f03 in panic (fmt=0x0) at
/usr/src/sys/kern/kern_shutdown.c:818
#4  0xffffffff80f810cd in trap_fatal (frame=<value optimized out>, eva=<value
optimized out>) at /usr/src/sys/amd64/amd64/trap.c:858
#5  0xffffffff80f813e8 in trap_pfault (frame=0xfffffe0237a4f3a0,
usermode=<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:681
#6  0xffffffff80f80a2a in trap (frame=0xfffffe0237a4f3a0) at
/usr/src/sys/amd64/amd64/trap.c:447
#7  0xffffffff80f661dc in calltrap () at
/usr/src/sys/amd64/amd64/exception.S:238
#8  0xffffffff80b2b4fa in turnstile_broadcast (ts=0x0, queue=1) at
/usr/src/sys/kern/subr_turnstile.c:838
#9  0xffffffff80ad4d80 in __rw_wunlock_hard (c=0xfffff8013a3b5318, tid=1,
file=0xfffff80009947001 "8?\201????", line=1) at
/usr/src/sys/kern/kern_rwlock.c:1027
#10 0xffffffff80dffd9a in vm_map_delete (map=0xfffff8000c22b8c0, start=<value
optimized out>, end=140737488355328) at /usr/src/sys/vm/vm_map.c:2911
#11 0xffffffff80e01b47 in vm_map_remove (map=0xfffff8000c22b8c0,
start=140737488355328, end=1) at /usr/src/sys/vm/vm_map.c:3028
#12 0xffffffff80a96759 in exec_new_vmspace (imgp=0xfffffe0237a4f868,
sv=0xffffffff819858e8) at /usr/src/sys/kern/kern_exec.c:1084
#13 0xffffffff80a73284 in exec_elf64_imgact (imgp=0xfffffe0237a4f868) at
/usr/src/sys/kern/imgact_elf.c:881
#14 0xffffffff80a94ec4 in kern_execve (td=0xfffff80009947000,
args=0xfffffe0237a4fa78, mac_p=<value optimized out>) at
/usr/src/sys/kern/kern_exec.c:606
#15 0xffffffff80a9438c in sys_execve (td=0xfffff80009947000, uap=<value
optimized out>) at /usr/src/sys/kern/kern_exec.c:222
#16 0xffffffff80f81b00 in amd64_syscall (td=0xfffff80009947000, traced=0) at
subr_syscall.c:141
#17 0xffffffff80f664cb in Xfast_syscall () at
/usr/src/sys/amd64/amd64/exception.S:398
#18 0x0000000800d7a97a in ?? ()
Previous frame inner to this frame (corrupt stack?)

Thanks for your help.

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Bug 213903] Kernel crashes from turnstile_broadcast (/usr/src/sys/kern/subr_turnstile.c:837)

bugzilla-noreply
In reply to this post by bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=213903

--- Comment #16 from Mateusz Guzik <[hidden email]> ---
Can you please reproduce with
https://people.freebsd.org/~mjg/patches/rwlock-debug-10.diff appliled on top.
E.g. like this:

cd /usr/src
fetch https://people.freebsd.org/~mjg/patches/rwlock-debug-10.diff
patch -p1 < rwlock-debug-10.diff

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Bug 213903] Kernel crashes from turnstile_broadcast (/usr/src/sys/kern/subr_turnstile.c:837)

bugzilla-noreply
In reply to this post by bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=213903

--- Comment #17 from Cassiano Peixoto <[hidden email]> ---
(In reply to Mateusz Guzik from comment #16)
Hi Mateusz,

Sorry but i can't try this patch, i had to rollback the old kernel to avoid
crashes. It's a production server and i can't let it down. :(

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Bug 213903] Kernel crashes from turnstile_broadcast (/usr/src/sys/kern/subr_turnstile.c:837)

bugzilla-noreply
In reply to this post by bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=213903

--- Comment #18 from Cassiano Peixoto <[hidden email]> ---
(In reply to Cassiano Peixoto from comment #17)
Hi Mateusz,

I got another customer to try. Can i apply this patch and only replace the
kernel or do i need to replace userland as well?

After that do you need only kgdb trace output?

Thanks.

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "[hidden email]"
12
Loading...