panic: deadlkres: possible deadlock detected for 0xfffff80141b04560, blocked for 1801695 ticks

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

panic: deadlkres: possible deadlock detected for 0xfffff80141b04560, blocked for 1801695 ticks

Eitan Adler-4
How to reproduce?

# kldload sem
# kldunload sem
< wait debug.deadlkres.slptime_threshold seconds >

https://reviews.freebsd.org/P168
Reading symbols from ./kernel/kernel...Reading symbols from
/usr/home/eax/crashes/sem_load_dklres/kernel/kernel.debug...done.
done.

Unread portion of the kernel message buffer:
[29474] panic: deadlkres: possible deadlock detected for
0xfffff80141b04560, blocked for 1801695 ticks
[29474]
[29474] cpuid = 31
[29474] time = 1523493929
[29474] KDB: stack backtrace:
[29474] db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame
0xfffffe00a6204960
[29474] vpanic() at vpanic+0x18d/frame 0xfffffe00a62049c0
[29474] panic() at panic+0x43/frame 0xfffffe00a6204a20
[29474] deadlkres() at deadlkres+0x3a6/frame 0xfffffe00a6204a70
[29474] fork_exit() at fork_exit+0x84/frame 0xfffffe00a6204ab0
[29474] fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a6204ab0
[29474] --- trap 0, rip = 0, rsp = 0, rbp = 0 ---
[29474] KDB: enter: panic

__curthread () at ./machine/pcpu.h:230
230             __asm("movq %%gs:%1,%0" : "=r" (td)
(kgdb) bt
#0  __curthread () at ./machine/pcpu.h:230
#1  doadump (textdump=0x1) at /usr/src/sys/kern/kern_shutdown.c:361
#2  0xffffffff80434f4c in db_fncall_generic (addr=<optimized out>,
rv=<optimized out>, nargs=<optimized out>, args=<optimized out>)
    at /usr/src/sys/ddb/db_command.c:609
#3  db_fncall (dummy1=<optimized out>, dummy2=<optimized out>,
dummy3=<optimized out>, dummy4=<optimized out>) at
/usr/src/sys/ddb/db_command.c:657
#4  0xffffffff80434a99 in db_command (last_cmdp=<optimized out>,
cmd_table=<optimized out>, dopager=<optimized out>) at
/usr/src/sys/ddb/db_command.c:481
#5  0xffffffff80434814 in db_command_loop () at
/usr/src/sys/ddb/db_command.c:534
#6  0xffffffff80437a3f in db_trap (type=<optimized out>,
code=<optimized out>) at /usr/src/sys/ddb/db_main.c:250
#7  0xffffffff80babf53 in kdb_trap (type=0x3, code=0xffff0ff0,
tf=<optimized out>) at /usr/src/sys/kern/subr_kdb.c:697
#8  0xffffffff81024aa8 in trap (frame=0xfffffe00a6204890) at
/usr/src/sys/amd64/amd64/trap.c:548
#9  <signal handler called>
#10 kdb_enter (why=0xffffffff8129f663 "panic", msg=<optimized out>) at
/usr/src/sys/kern/subr_kdb.c:479
#11 0xffffffff80b66b5a in vpanic (fmt=<optimized out>,
ap=0xfffffe00a6204a00) at /usr/src/sys/kern/kern_shutdown.c:826
#12 0xffffffff80b66be3 in panic (fmt=0xffffffff81deab08 <cnputs_mtx>
"5i&\201\377\377\377\377") at /usr/src/sys/kern/kern_shutdown.c:764
#13 0xffffffff80b00466 in deadlkres () at /usr/src/sys/kern/kern_clock.c:288
#14 0xffffffff80b26e34 in fork_exit (callout=0xffffffff80b000c0
<deadlkres>, arg=0x0, frame=0xfffffe00a6204ac0) at
/usr/src/sys/kern/kern_fork.c:1039
#15 <signal handler called>

(kgdb)
#13 0xffffffff80b00466 in deadlkres () at /usr/src/sys/kern/kern_clock.c:288
288             panic("%s: possible deadlock detected for %p, blocked
for %d ticks\n",
(kgdb) info locals
tryl = <optimized out>
slpticks = 0x1b7740
blkticks = 0xdbba0
p = 0xfffff80141b00538
td = <optimized out>
wchan = 0xffffffff81deb2d8 <kld_sx>
tticks = 0x1b7ddf
slptype = <optimized out>
i = <error reading variable i (Cannot access memory at address 0x2)>

(more in the paste)

--
Eitan Adler
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: panic: deadlkres: possible deadlock detected for 0xfffff80141b04560, blocked for 1801695 ticks

Andriy Gapon
On 12/04/2018 07:00, Eitan Adler wrote:

> How to reproduce?
>
> # kldload sem
> # kldunload sem
> < wait debug.deadlkres.slptime_threshold seconds >
>
> https://reviews.freebsd.org/P168
> Reading symbols from ./kernel/kernel...Reading symbols from
> /usr/home/eax/crashes/sem_load_dklres/kernel/kernel.debug...done.
> done.
>
> Unread portion of the kernel message buffer:
> [29474] panic: deadlkres: possible deadlock detected for
> 0xfffff80141b04560, blocked for 1801695 ticks

The debugging information you provided does not help.
You need to do
(gdb) p *(struct thread*)0xfffff80141b04560  <=== taken from the panic message
find td_tid value and then do
(gdb) tid <td_tid>
And then examine that thread.

I guess that the deadlkres could be enhanced to print the tid and the stack
trace of the deadlocked thread, so that it is faster to get started.


--
Andriy Gapon
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: panic: deadlkres: possible deadlock detected for 0xfffff80141b04560, blocked for 1801695 ticks

Eitan Adler-4
On 11 April 2018 at 23:11, Andriy Gapon <[hidden email]> wrote:

> On 12/04/2018 07:00, Eitan Adler wrote:
>> How to reproduce?
>>
>> # kldload sem
>> # kldunload sem
>> < wait debug.deadlkres.slptime_threshold seconds >
>>
>> https://reviews.freebsd.org/P168
>> Reading symbols from ./kernel/kernel...Reading symbols from
>> /usr/home/eax/crashes/sem_load_dklres/kernel/kernel.debug...done.
>> done.
>>
>> Unread portion of the kernel message buffer:
>> [29474] panic: deadlkres: possible deadlock detected for
>> 0xfffff80141b04560, blocked for 1801695 ticks
>
> The debugging information you provided does not help.
> You need to do
> (gdb) p *(struct thread*)0xfffff80141b04560  <=== taken from the panic message
> find td_tid value and then do
> (gdb) tid <td_tid>
> And then examine that thread.
>
> I guess that the deadlkres could be enhanced to print the tid and the stack
> trace of the deadlocked thread, so that it is faster to get started.

(kgdb) p (*(struct thread*)0xfffff80141b04560).td_tid
$2 = 0x18bd8
(kgdb) tid 18bd8
(kgdb) bt
#0  sched_switch (td=0xfffff80141b04560, newtd=0xfffff80003699000,
flags=<optimized out>) at /usr/src/sys/kern/sched_ule.c:2115
#1  0xffffffff80b7156c in mi_switch (flags=0x104, newtd=0x0) at
/usr/src/sys/kern/kern_synch.c:437
#2  0xffffffff80bba7ed in sleepq_switch (wchan=0xffffffff81deb2d8
<kld_sx>, pri=0x0) at /usr/src/sys/kern/subr_sleepqueue.c:613
#3  0xffffffff80bba693 in sleepq_wait (wchan=0xffffffff81deb2d8
<kld_sx>, pri=0x0) at /usr/src/sys/kern/subr_sleepqueue.c:692
#4  0xffffffff80b6f329 in _sx_xlock_hard (sx=0xffffffff81deb2d8
<kld_sx>, x=<optimized out>, opts=<optimized out>, file=<optimized
out>, line=<optimized out>) at /usr/src/sys/kern/kern_sx.c:777
#5  0xffffffff80b6ef31 in _sx_xlock (sx=0xffffffff81deb2d8 <kld_sx>,
opts=0x0, file=0xffffffff811a5d65 "/usr/src/sys/kern/kern_linker.c",
line=0x42e) at /usr/src/sys/kern/kern_sx.c:319
#6  0xffffffff80b3834e in kern_kldload (td=<optimized out>,
file=<optimized out>, fileid=0xfffffe00af1f8964) at
/usr/src/sys/kern/kern_linker.c:1070
#7  0xffffffff80b3848b in sys_kldload (td=0xfffff80141b04560,
uap=<optimized out>) at /usr/src/sys/kern/kern_linker.c:1097
#8  0xffffffff8102606b in syscallenter (td=0xfffff80141b04560) at
/usr/src/sys/amd64/amd64/../../kern/subr_syscall.c:134
#9  amd64_syscall (td=0xfffff80141b04560, traced=0x0) at
/usr/src/sys/amd64/amd64/trap.c:936
#10 <signal handler called>
#11 0x00000008002cfd8a in ?? ()
Backtrace stopped: Cannot access memory at address 0x7fffffffd478

(kgdb) frame
Stack level 3, frame at 0xfffffe00af1f8820:
 rip = 0xffffffff80bba693 in sleepq_wait
(/usr/src/sys/kern/subr_sleepqueue.c:692); saved rip =
0xffffffff80b6f329
 called by frame at 0xfffffe00af1f88d0, caller of frame at 0xfffffe00af1f87f0
 source language c.
 Arglist at 0xfffffe00af1f8810, args: wchan=0xffffffff81deb2d8 <kld_sx>, pri=0x0
 Locals at 0xfffffe00af1f8810, Previous frame's sp is 0xfffffe00af1f8820
 Saved registers:
  rbx at 0xfffffe00af1f87f8, rbp at 0xfffffe00af1f8810, r14 at
0xfffffe00af1f8800, r15 at 0xfffffe00af1f8808, rip at
0xfffffe00af1f8818
wchan = 0xffffffff81deb2d8 <kld_sx>
pri = 0x0
td = 0xfffff80141b04560

Updated paste with more info as well
--
Eitan Adler
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"