Regression between 10 & 11, Freeswitch hangs when exiting

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Regression between 10 & 11, Freeswitch hangs when exiting

Eric Masson-3
Hi,

Freeswitch, a multiprotocol softswitch (http://freewitch.org) runs on
FreeBSD and a port is available (net/freeswitch/)

Freeswitch runs fine on any 10.x installation, issueing the shutdown
command from cli makes the application exit cleanly.

On any 11.x installation, issueing the shutdown command from cli makes
the application hang.

FreeBSD version is :
FreeBSD newsrv.interne.associated-bears.org 11.1-STABLE FreeBSD 11.1-STABLE #0 r327876M: Sat Jan 13 16:19:26 CET 2018     [hidden email]:/usr/obj/usr/src/sys/SE7525GP2  amd64

Generic or custom kernel behave the same.

On Freeswitch side, 1.6.19 (net/freeswitch) and also git branches master
& v1.6 show the same issue.

I've reproduced the issue on the latest v1.6, the backtrace of stuck
process shows the following :
#0  0x00000008030ea91c in _umtx_op_err () from /lib/libthr.so.3
#1  0x00000008030e6f97 in join_common (pthread=0x807ed7700, thread_return=0x7fffffffb658, abstime=0x0) at /usr/src/lib/libthr/thread/thr_join.c:125
#2  0x0000000800f01421 in apr_thread_join (retval=0x7fffffffb68c, thd=0x8059e9658) at threadproc/unix/thread.c:234
#3  0x0000000800cd889e in do_shutdown (module=0x2, shutdown=<optimized out>, unload=SWITCH_TRUE, fail_if_busy=(unknown: 1523667750), err=<optimized out>) at src/switch_loadable_module.c:2031
#4  0x0000000800cdb84a in switch_loadable_module_shutdown () at src/switch_loadable_module.c:2083
#5  0x0000000800c9d92c in switch_core_destroy () at src/switch_core.c:2940
#6  0x0000000000403b2b in main (argc=<optimized out>, argv=0x7fffffffe988) at src/switch.c:1210

It seems the issue lies in thread handling.

I've attached :
- the output of a Freeswitch related script used to gather relevant
  information.
- source files showing in bt

Thank for help.

Éric Masson

--
Warning: file "/home/emss/misc/fortune/En_sig.dat" unreadable
Warning: file "/home/emss/misc/fortune/Fr_sig.dat" unreadable
Faut vraiment que je m'occupe de ce problème de signature :)

GNU gdb (GDB) 8.0.1 [GDB v8.0.1 for FreeBSD]
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-portbld-freebsd11.1".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /opt/freeswitch/bin/freeswitch...done.
[New LWP 100834]
[New LWP 101041]
[New LWP 101042]
[New LWP 101044]
[New LWP 101045]
[New LWP 101046]
[New LWP 101047]
[New LWP 101050]
[New LWP 101051]
[New LWP 101080]
Core was generated by `/opt/freeswitch/bin/freeswitch -c'.
#0  0x00000008030ea91c in _umtx_op_err () from /lib/libthr.so.3
[Current thread is 1 (LWP 100834)]
(gdb)

================================================================================
# GDB session generated by FS backtrace-from-core
# FreeSWITCH version: 1.9.0+git~20180202T143051Z~38153a37ed~64bit
# FreeSWITCH version (human): 1.9.0 git 38153a3 2018-02-02 14:30:51Z 64bit
================================================================================


================================================================================
# info threads
================================================================================
  Id   Target Id         Frame
* 1    LWP 100834        0x00000008030ea91c in _umtx_op_err () from /lib/libthr.so.3
  2    LWP 101041        0x0000000803de665a in _select () from /lib/libc.so.7
  3    LWP 101042        0x00000008030ea91c in _umtx_op_err () from /lib/libthr.so.3
  4    LWP 101044        0x00000008030ea91c in _umtx_op_err () from /lib/libthr.so.3
  5    LWP 101045        0x00000008030ea91c in _umtx_op_err () from /lib/libthr.so.3
  6    LWP 101046        0x0000000803de665a in _select () from /lib/libc.so.7
  7    LWP 101047        0x00000008030ea91c in _umtx_op_err () from /lib/libthr.so.3
  8    LWP 101050        0x0000000803d7797a in _accept () from /lib/libc.so.7
  9    LWP 101051        0x0000000803d7797a in _accept () from /lib/libc.so.7
  10   LWP 101080        0x0000000803d7797a in _accept () from /lib/libc.so.7
================================================================================
# bt
================================================================================
#0  0x00000008030ea91c in _umtx_op_err () from /lib/libthr.so.3
#1  0x00000008030e6f97 in join_common (pthread=0x807ed7700, thread_return=0x7fffffffb658, abstime=0x0) at /usr/src/lib/libthr/thread/thr_join.c:125
#2  0x0000000800f01421 in apr_thread_join (retval=0x7fffffffb68c, thd=0x8059e9658) at threadproc/unix/thread.c:234
#3  0x0000000800cd889e in do_shutdown (module=0x2, shutdown=<optimized out>, unload=SWITCH_TRUE, fail_if_busy=(unknown: 1523667750), err=<optimized out>) at src/switch_loadable_module.c:2031
#4  0x0000000800cdb84a in switch_loadable_module_shutdown () at src/switch_loadable_module.c:2083
#5  0x0000000800c9d92c in switch_core_destroy () at src/switch_core.c:2940
#6  0x0000000000403b2b in main (argc=<optimized out>, argv=0x7fffffffe988) at src/switch.c:1210
================================================================================
# bt full
================================================================================
#0  0x00000008030ea91c in _umtx_op_err () from /lib/libthr.so.3
No symbol table info available.
#1  0x00000008030e6f97 in join_common (pthread=0x807ed7700, thread_return=0x7fffffffb658, abstime=0x0) at /usr/src/lib/libthr/thread/thr_join.c:125
        ret = <error reading variable ret (Cannot access memory at address 0x2d)>
        tid = 101080
        ts = {tv_sec = 34411006815, tv_nsec = 34514890792}
        tsp = 0x18ad8
#2  0x0000000800f01421 in apr_thread_join (retval=0x7fffffffb68c, thd=0x8059e9658) at threadproc/unix/thread.c:234
        stat = <optimized out>
#3  0x0000000800cd889e in do_shutdown (module=0x2, shutdown=<optimized out>, unload=SWITCH_TRUE, fail_if_busy=(unknown: 1523667750), err=<optimized out>) at src/switch_loadable_module.c:2031
        st = SWITCH_STATUS_SUCCESS
        pool = <optimized out>
        flags = 12798607
#4  0x0000000800cdb84a in switch_loadable_module_shutdown () at src/switch_loadable_module.c:2083
        i = <error reading variable i (Cannot access memory at address 0x0)>
        hi = <optimized out>
        val = 0x805626460
        module = 0x807ed7700
#5  0x0000000800c9d92c in switch_core_destroy () at src/switch_core.c:2940
        event = 0x0
#6  0x0000000000403b2b in main (argc=<optimized out>, argv=0x7fffffffe988) at src/switch.c:1210
        pid_path = "/opt/freeswitch/var/run/freeswitch/freeswitch.pid", '\000' <repeats 974 times>
        pid_buffer = "45848", '\000' <repeats 26 times>
        old_pid_buffer = "45690", '\000' <repeats 26 times>
        opts_str = '\000' <repeats 1023 times>
        local_argv = {0x7fffffffec40 "/opt/freeswitch/bin/freeswitch", 0x7fffffffec5f "-c", 0x0 <repeats 1022 times>}
        arg_argv = {0x0 <repeats 128 times>}
        local_argc = 2
        reincarnate_reexec = <error reading variable reincarnate_reexec (Cannot access memory at address 0x0)>
        reincarnate = <error reading variable reincarnate (Cannot access memory at address 0x0)>
        do_wait = <error reading variable do_wait (Cannot access memory at address 0x0)>
        nf = <error reading variable nf (Cannot access memory at address 0x0)>
        pid = <optimized out>
        nc = <error reading variable nc (Cannot access memory at address 0x1)>
        ret = <error reading variable ret (Cannot access memory at address 0x0)>
        flags = 68225
        priority = <error reading variable priority (Cannot access memory at address 0x0)>
        do_kill = 0
        run_set = <error reading variable run_set (Cannot access memory at address 0x1)>
        log_set = 0
        alt_base = <error reading variable alt_base (Cannot access memory at address 0x1)>
        alt_dirs = 0
        x = <optimized out>
        opts = <optimized out>
        i = <optimized out>
        runas_group = 0x0
        pid_len = 5
        pool = <optimized out>
        fd = <optimized out>
        old_pid_len = 5
        err = 0x800f28b86 "Success"
        destroy_status = <optimized out>
================================================================================
# thread apply all bt
================================================================================

Thread 10 (LWP 101080):
#0  0x0000000803d7797a in _accept () from /lib/libc.so.7
#1  0x00000008030df2c6 in __thr_accept (s=67, addr=0x8088f5238, addrlen=0x8088f5218) at /usr/src/lib/libthr/thread/thr_syscalls.c:106
#2  0x0000000800eff5e0 in apr_socket_accept (new=0x7fffdec1aed8, sock=0x8088f31b0, connection_context=0x8088f5028) at network_io/unix/sockets.c:191
#3  0x000000080766bbb6 in mod_event_socket_runtime () at mod_event_socket.c:2987
#4  0x0000000800cd8da9 in switch_loadable_module_exec (thread=0x8059e9658, obj=0x8059e9238) at src/switch_loadable_module.c:113
#5  0x0000000800f013a6 in dummy_worker (opaque=0x8059e9658) at threadproc/unix/thread.c:151
#6  0x00000008030dcc05 in thread_start (curthread=0x807ed7700) at /usr/src/lib/libthr/thread/thr_create.c:289
#7  0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x7fffdec1b000

Thread 9 (LWP 101051):
#0  0x0000000803d7797a in _accept () from /lib/libc.so.7
#1  0x00000008030df2c6 in __thr_accept (s=10, addr=0x806618238, addrlen=0x806618218) at /usr/src/lib/libthr/thread/thr_syscalls.c:106
#2  0x0000000800eff5e0 in apr_socket_accept (new=0x7fffdfa13f20, sock=0x8059de610, connection_context=0x806618028) at network_io/unix/sockets.c:191
#3  0x0000000800d5ae20 in msrp_listener (thread=<optimized out>, obj=0x8011be5e8 <globals+120>) at src/switch_msrp.c:1379
#4  0x0000000800f013a6 in dummy_worker (opaque=0x8059de838) at threadproc/unix/thread.c:151
#5  0x00000008030dcc05 in thread_start (curthread=0x804e19200) at /usr/src/lib/libthr/thread/thr_create.c:289
#6  0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x7fffdfa14000

Thread 8 (LWP 101050):
#0  0x0000000803d7797a in _accept () from /lib/libc.so.7
#1  0x00000008030df2c6 in __thr_accept (s=8, addr=0x806018238, addrlen=0x806018218) at /usr/src/lib/libthr/thread/thr_syscalls.c:106
#2  0x0000000800eff5e0 in apr_socket_accept (new=0x7fffdfa50f20, sock=0x8059de1e8, connection_context=0x806018028) at network_io/unix/sockets.c:191
#3  0x0000000800d5ae20 in msrp_listener (thread=<optimized out>, obj=0x8011be5c8 <globals+88>) at src/switch_msrp.c:1379
#4  0x0000000800f013a6 in dummy_worker (opaque=0x8059de510) at threadproc/unix/thread.c:151
#5  0x00000008030dcc05 in thread_start (curthread=0x804e18d00) at /usr/src/lib/libthr/thread/thr_create.c:289
#6  0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x7fffdfa51000

Thread 7 (LWP 101047):
#0  0x00000008030ea91c in _umtx_op_err () from /lib/libthr.so.3
#1  0x00000008030debb0 in _thr_umtx_timedwait_uint (mtx=0x80062e098, id=<optimized out>, clockid=<optimized out>, abstime=<optimized out>, shared=<optimized out>) at /usr/src/lib/libthr/thread/thr_umtx.c:234
#2  0x00000008030e8698 in cond_wait_user (abstime=<optimized out>, cancel=1, cvp=<optimized out>, mp=<optimized out>) at /usr/src/lib/libthr/thread/thr_cond.c:304
#3  cond_wait_common (cond=<optimized out>, mutex=<optimized out>, abstime=0x7fffdfe8fec8, cancel=1) at /usr/src/lib/libthr/thread/thr_cond.c:364
#4  0x0000000800efb3ef in apr_thread_cond_timedwait (cond=<optimized out>, mutex=<optimized out>, timeout=500000) at locks/unix/thread_cond.c:89
#5  0x0000000800ef16fb in apr_queue_pop_timeout (queue=0x8051dd158, data=0x7fffdfe8ff48, timeout=500000) at misc/apr_queue.c:339
#6  0x0000000800cd490b in switch_scheduler_task_thread (thread=<optimized out>, obj=<optimized out>) at src/switch_scheduler.c:188
#7  0x0000000800f013a6 in dummy_worker (opaque=0x805de85c8) at threadproc/unix/thread.c:151
#8  0x00000008030dcc05 in thread_start (curthread=0x804e18300) at /usr/src/lib/libthr/thread/thr_create.c:289
#9  0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x7fffdfe90000

Thread 6 (LWP 101046):
#0  0x0000000803de665a in _select () from /lib/libc.so.7
#1  0x00000008030df9a2 in __thr_select (numfds=0, readfds=0x0, writefds=0x0, exceptfds=0x0, timeout=0x7fffdfeccf00) at /usr/src/lib/libthr/thread/thr_syscalls.c:493
#2  0x0000000800f02749 in apr_sleep (t=<optimized out>) at time/unix/time.c:246
#3  0x0000000800d4ce25 in do_sleep (t=0) at src/switch_time.c:164
#4  switch_sleep (t=0) at src/switch_time.c:632
#5  0x0000000800c86b8d in switch_core_sql_db_thread (thread=<optimized out>, obj=<optimized out>) at src/switch_core_sqldb.c:1395
#6  0x0000000800f013a6 in dummy_worker (opaque=0x805371c48) at threadproc/unix/thread.c:151
#7  0x00000008030dcc05 in thread_start (curthread=0x804e18800) at /usr/src/lib/libthr/thread/thr_create.c:289
#8  0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x7fffdfecd000

Thread 5 (LWP 101045):
#0  0x00000008030ea91c in _umtx_op_err () from /lib/libthr.so.3
#1  0x00000008030debb0 in _thr_umtx_timedwait_uint (mtx=0x80062e068, id=<optimized out>, clockid=<optimized out>, abstime=<optimized out>, shared=<optimized out>) at /usr/src/lib/libthr/thread/thr_umtx.c:234
#2  0x00000008030e8698 in cond_wait_user (abstime=<optimized out>, cancel=1, cvp=<optimized out>, mp=<optimized out>) at /usr/src/lib/libthr/thread/thr_cond.c:304
#3  cond_wait_common (cond=<optimized out>, mutex=<optimized out>, abstime=0x0, cancel=1) at /usr/src/lib/libthr/thread/thr_cond.c:364
#4  0x0000000800c81e46 in switch_user_sql_thread (thread=<optimized out>, obj=0x8051fd0f0) at src/switch_core_sqldb.c:2135
#5  0x0000000800f013a6 in dummy_worker (opaque=0x805ad0880) at threadproc/unix/thread.c:151
#6  0x00000008030dcc05 in thread_start (curthread=0x804e17e00) at /usr/src/lib/libthr/thread/thr_create.c:289
#7  0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x7fffdff0a000

Thread 4 (LWP 101044):
#0  0x00000008030ea91c in _umtx_op_err () from /lib/libthr.so.3
#1  0x00000008030debb0 in _thr_umtx_timedwait_uint (mtx=0x80062e050, id=<optimized out>, clockid=<optimized out>, abstime=<optimized out>, shared=<optimized out>) at /usr/src/lib/libthr/thread/thr_umtx.c:234
#2  0x00000008030e8698 in cond_wait_user (abstime=<optimized out>, cancel=1, cvp=<optimized out>, mp=<optimized out>) at /usr/src/lib/libthr/thread/thr_cond.c:304
#3  cond_wait_common (cond=<optimized out>, mutex=<optimized out>, abstime=0x0, cancel=1) at /usr/src/lib/libthr/thread/thr_cond.c:364
#4  0x0000000800ef1608 in apr_queue_pop (queue=0x805140908, data=0x7fffdff46f50) at misc/apr_queue.c:276
#5  0x0000000800d40a57 in log_thread (t=<optimized out>, obj=<optimized out>) at src/switch_log.c:294
#6  0x0000000800f013a6 in dummy_worker (opaque=0x805371b38) at threadproc/unix/thread.c:151
#7  0x00000008030dcc05 in thread_start (curthread=0x804e17900) at /usr/src/lib/libthr/thread/thr_create.c:289
#8  0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x7fffdff47000

Thread 3 (LWP 101042):
#0  0x00000008030ea91c in _umtx_op_err () from /lib/libthr.so.3
#1  0x00000008030debb0 in _thr_umtx_timedwait_uint (mtx=0x80062e020, id=<optimized out>, clockid=<optimized out>, abstime=<optimized out>, shared=<optimized out>) at /usr/src/lib/libthr/thread/thr_umtx.c:234
#2  0x00000008030e8698 in cond_wait_user (abstime=<optimized out>, cancel=1, cvp=<optimized out>, mp=<optimized out>) at /usr/src/lib/libthr/thread/thr_cond.c:304
#3  cond_wait_common (cond=<optimized out>, mutex=<optimized out>, abstime=0x0, cancel=1) at /usr/src/lib/libthr/thread/thr_cond.c:364
#4  0x0000000800ef1608 in apr_queue_pop (queue=0x804e31d08, data=0x7fffdffc0f38) at misc/apr_queue.c:276
#5  0x0000000800ce578b in switch_event_dispatch_thread (thread=<optimized out>, obj=0x804e31d08) at src/switch_event.c:324
#6  0x0000000800f013a6 in dummy_worker (opaque=0x805140780) at threadproc/unix/thread.c:151
#7  0x00000008030dcc05 in thread_start (curthread=0x804e16a00) at /usr/src/lib/libthr/thread/thr_create.c:289
#8  0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x7fffdffc1000

Thread 2 (LWP 101041):
#0  0x0000000803de665a in _select () from /lib/libc.so.7
#1  0x00000008030df9a2 in __thr_select (numfds=0, readfds=0x0, writefds=0x0, exceptfds=0x0, timeout=0x7fffdfffdf10) at /usr/src/lib/libthr/thread/thr_syscalls.c:493
#2  0x0000000800f02749 in apr_sleep (t=<optimized out>) at time/unix/time.c:246
#3  0x0000000800d4ce25 in do_sleep (t=0) at src/switch_time.c:164
#4  switch_sleep (t=0) at src/switch_time.c:632
#5  0x0000000800c78e51 in pool_thread (thread=<optimized out>, obj=<optimized out>) at src/switch_core_memory.c:569
#6  0x0000000800f013a6 in dummy_worker (opaque=0x804f13740) at threadproc/unix/thread.c:151
#7  0x00000008030dcc05 in thread_start (curthread=0x804e16500) at /usr/src/lib/libthr/thread/thr_create.c:289
#8  0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x7fffdfffe000

Thread 1 (LWP 100834):
#0  0x00000008030ea91c in _umtx_op_err () from /lib/libthr.so.3
#1  0x00000008030e6f97 in join_common (pthread=0x807ed7700, thread_return=0x7fffffffb658, abstime=0x0) at /usr/src/lib/libthr/thread/thr_join.c:125
#2  0x0000000800f01421 in apr_thread_join (retval=0x7fffffffb68c, thd=0x8059e9658) at threadproc/unix/thread.c:234
#3  0x0000000800cd889e in do_shutdown (module=0x2, shutdown=<optimized out>, unload=SWITCH_TRUE, fail_if_busy=(unknown: 1523667750), err=<optimized out>) at src/switch_loadable_module.c:2031
#4  0x0000000800cdb84a in switch_loadable_module_shutdown () at src/switch_loadable_module.c:2083
#5  0x0000000800c9d92c in switch_core_destroy () at src/switch_core.c:2940
#6  0x0000000000403b2b in main (argc=<optimized out>, argv=0x7fffffffe988) at src/switch.c:1210
================================================================================
# thread apply all bt full
================================================================================

Thread 10 (LWP 101080):
#0  0x0000000803d7797a in _accept () from /lib/libc.so.7
No symbol table info available.
#1  0x00000008030df2c6 in __thr_accept (s=67, addr=0x8088f5238, addrlen=0x8088f5218) at /usr/src/lib/libthr/thread/thr_syscalls.c:106
        curthread = 0x807ed7700
        ret = <optimized out>
#2  0x0000000800eff5e0 in apr_socket_accept (new=0x7fffdec1aed8, sock=0x8088f31b0, connection_context=0x8088f5028) at network_io/unix/sockets.c:191
No locals.
#3  0x000000080766bbb6 in mod_event_socket_runtime () at mod_event_socket.c:2987
        errs = <error reading variable errs (Cannot access memory at address 0x0)>
        x = <optimized out>
        pool = <optimized out>
        listener_pool = 0x8088f5218
        sa = 0x8088f30e8
        rv = <optimized out>
        inbound_socket = 0x8088f50e8
        listener = <optimized out>
#4  0x0000000800cd8da9 in switch_loadable_module_exec (thread=0x8059e9658, obj=0x8059e9238) at src/switch_loadable_module.c:113
        ts = 0x8059e9238
        status = <error reading variable status (Cannot access memory at address 0x0)>
        module = 0x8056263c8
#5  0x0000000800f013a6 in dummy_worker (opaque=0x8059e9658) at threadproc/unix/thread.c:151
No locals.
#6  0x00000008030dcc05 in thread_start (curthread=0x807ed7700) at /usr/src/lib/libthr/thread/thr_create.c:289
        set = {__bits = {0, 0, 0, 0}}
#7  0x0000000000000000 in ?? ()
No symbol table info available.
Backtrace stopped: Cannot access memory at address 0x7fffdec1b000

Thread 9 (LWP 101051):
#0  0x0000000803d7797a in _accept () from /lib/libc.so.7
No symbol table info available.
#1  0x00000008030df2c6 in __thr_accept (s=10, addr=0x806618238, addrlen=0x806618218) at /usr/src/lib/libthr/thread/thr_syscalls.c:106
        curthread = 0x804e19200
        ret = <optimized out>
#2  0x0000000800eff5e0 in apr_socket_accept (new=0x7fffdfa13f20, sock=0x8059de610, connection_context=0x806618028) at network_io/unix/sockets.c:191
No locals.
#3  0x0000000800d5ae20 in msrp_listener (thread=<optimized out>, obj=0x8011be5e8 <globals+120>) at src/switch_msrp.c:1379
        msock = 0x8011be5e8 <globals+120>
        sock = 0x8066180e8
        thd_attr = <optimized out>
        pool = 0x806618218
        rv = <optimized out>
#4  0x0000000800f013a6 in dummy_worker (opaque=0x8059de838) at threadproc/unix/thread.c:151
No locals.
#5  0x00000008030dcc05 in thread_start (curthread=0x804e19200) at /usr/src/lib/libthr/thread/thr_create.c:289
        set = {__bits = {0, 0, 0, 0}}
#6  0x0000000000000000 in ?? ()
No symbol table info available.
Backtrace stopped: Cannot access memory at address 0x7fffdfa14000

Thread 8 (LWP 101050):
#0  0x0000000803d7797a in _accept () from /lib/libc.so.7
No symbol table info available.
#1  0x00000008030df2c6 in __thr_accept (s=8, addr=0x806018238, addrlen=0x806018218) at /usr/src/lib/libthr/thread/thr_syscalls.c:106
        curthread = 0x804e18d00
        ret = <optimized out>
#2  0x0000000800eff5e0 in apr_socket_accept (new=0x7fffdfa50f20, sock=0x8059de1e8, connection_context=0x806018028) at network_io/unix/sockets.c:191
No locals.
#3  0x0000000800d5ae20 in msrp_listener (thread=<optimized out>, obj=0x8011be5c8 <globals+88>) at src/switch_msrp.c:1379
        msock = 0x8011be5c8 <globals+88>
        sock = 0x8060180e8
        thd_attr = <optimized out>
        pool = 0x806018218
        rv = <optimized out>
#4  0x0000000800f013a6 in dummy_worker (opaque=0x8059de510) at threadproc/unix/thread.c:151
No locals.
#5  0x00000008030dcc05 in thread_start (curthread=0x804e18d00) at /usr/src/lib/libthr/thread/thr_create.c:289
        set = {__bits = {0, 0, 0, 0}}
#6  0x0000000000000000 in ?? ()
No symbol table info available.
Backtrace stopped: Cannot access memory at address 0x7fffdfa51000

Thread 7 (LWP 101047):
#0  0x00000008030ea91c in _umtx_op_err () from /lib/libthr.so.3
No symbol table info available.
#1  0x00000008030debb0 in _thr_umtx_timedwait_uint (mtx=0x80062e098, id=<optimized out>, clockid=<optimized out>, abstime=<optimized out>, shared=<optimized out>) at /usr/src/lib/libthr/thread/thr_umtx.c:234
        tm_p = 0x5a7f38d8
        tm_size = <error reading variable tm_size (Cannot access memory at address 0x18)>
#2  0x00000008030e8698 in cond_wait_user (abstime=<optimized out>, cancel=1, cvp=<optimized out>, mp=<optimized out>) at /usr/src/lib/libthr/thread/thr_cond.c:304
        curthread = 0x804e18300
        deferred = <error reading variable deferred (Cannot access memory at address 0x0)>
        recurse = 0
        error = <optimized out>
        sq = <optimized out>
        error2 = <optimized out>
#3  cond_wait_common (cond=<optimized out>, mutex=<optimized out>, abstime=0x7fffdfe8fec8, cancel=1) at /usr/src/lib/libthr/thread/thr_cond.c:364
        cvp = 0x8053de1a0
        mp = 0x8051df720
        error = <optimized out>
#4  0x0000000800efb3ef in apr_thread_cond_timedwait (cond=<optimized out>, mutex=<optimized out>, timeout=500000) at locks/unix/thread_cond.c:89
        then = <optimized out>
        abstime = {tv_sec = 1518287064, tv_nsec = 389979000}
        rv = <optimized out>
#5  0x0000000800ef16fb in apr_queue_pop_timeout (queue=0x8051dd158, data=0x7fffdfe8ff48, timeout=500000) at misc/apr_queue.c:339
        rv = 0
#6  0x0000000800cd490b in switch_scheduler_task_thread (thread=<optimized out>, obj=<optimized out>) at src/switch_scheduler.c:188
        pop = 0x805e262a0
#7  0x0000000800f013a6 in dummy_worker (opaque=0x805de85c8) at threadproc/unix/thread.c:151
No locals.
#8  0x00000008030dcc05 in thread_start (curthread=0x804e18300) at /usr/src/lib/libthr/thread/thr_create.c:289
        set = {__bits = {0, 0, 0, 0}}
#9  0x0000000000000000 in ?? ()
No symbol table info available.
Backtrace stopped: Cannot access memory at address 0x7fffdfe90000

Thread 6 (LWP 101046):
#0  0x0000000803de665a in _select () from /lib/libc.so.7
No symbol table info available.
#1  0x00000008030df9a2 in __thr_select (numfds=0, readfds=0x0, writefds=0x0, exceptfds=0x0, timeout=0x7fffdfeccf00) at /usr/src/lib/libthr/thread/thr_syscalls.c:493
        curthread = 0x804e18800
        ret = <optimized out>
#2  0x0000000800f02749 in apr_sleep (t=<optimized out>) at time/unix/time.c:246
        tv = {tv_sec = 1, tv_usec = 0}
#3  0x0000000800d4ce25 in do_sleep (t=0) at src/switch_time.c:164
        ts = {tv_sec = 34375559564, tv_nsec = 34378191880}
#4  switch_sleep (t=0) at src/switch_time.c:632
No locals.
#5  0x0000000800c86b8d in switch_core_sql_db_thread (thread=<optimized out>, obj=<optimized out>) at src/switch_core_sqldb.c:1395
        sec = <error reading variable sec (Cannot access memory at address 0x0)>
        reg_sec = <error reading variable reg_sec (Cannot access memory at address 0x0)>
#6  0x0000000800f013a6 in dummy_worker (opaque=0x805371c48) at threadproc/unix/thread.c:151
No locals.
#7  0x00000008030dcc05 in thread_start (curthread=0x804e18800) at /usr/src/lib/libthr/thread/thr_create.c:289
        set = {__bits = {0, 0, 0, 0}}
#8  0x0000000000000000 in ?? ()
No symbol table info available.
Backtrace stopped: Cannot access memory at address 0x7fffdfecd000

Thread 5 (LWP 101045):
#0  0x00000008030ea91c in _umtx_op_err () from /lib/libthr.so.3
No symbol table info available.
#1  0x00000008030debb0 in _thr_umtx_timedwait_uint (mtx=0x80062e068, id=<optimized out>, clockid=<optimized out>, abstime=<optimized out>, shared=<optimized out>) at /usr/src/lib/libthr/thread/thr_umtx.c:234
        tm_p = 0x8030e17bc <_thr_ast+44>
        tm_size = <error reading variable tm_size (Cannot access memory at address 0x18)>
#2  0x00000008030e8698 in cond_wait_user (abstime=<optimized out>, cancel=1, cvp=<optimized out>, mp=<optimized out>) at /usr/src/lib/libthr/thread/thr_cond.c:304
        curthread = 0x804e17e00
        deferred = <error reading variable deferred (Cannot access memory at address 0x0)>
        recurse = 0
        error = <optimized out>
        sq = <optimized out>
        error2 = <optimized out>
#3  cond_wait_common (cond=<optimized out>, mutex=<optimized out>, abstime=0x0, cancel=1) at /usr/src/lib/libthr/thread/thr_cond.c:364
        cvp = 0x8053e7a00
        mp = 0x8051df4e0
        error = <optimized out>
#4  0x0000000800c81e46 in switch_user_sql_thread (thread=<optimized out>, obj=0x8051fd0f0) at src/switch_core_sqldb.c:2135
        iterations = 0
        lc = <optimized out>
        i = <optimized out>
        i = <optimized out>
#5  0x0000000800f013a6 in dummy_worker (opaque=0x805ad0880) at threadproc/unix/thread.c:151
No locals.
#6  0x00000008030dcc05 in thread_start (curthread=0x804e17e00) at /usr/src/lib/libthr/thread/thr_create.c:289
        set = {__bits = {0, 0, 0, 0}}
#7  0x0000000000000000 in ?? ()
No symbol table info available.
Backtrace stopped: Cannot access memory at address 0x7fffdff0a000

Thread 4 (LWP 101044):
#0  0x00000008030ea91c in _umtx_op_err () from /lib/libthr.so.3
No symbol table info available.
#1  0x00000008030debb0 in _thr_umtx_timedwait_uint (mtx=0x80062e050, id=<optimized out>, clockid=<optimized out>, abstime=<optimized out>, shared=<optimized out>) at /usr/src/lib/libthr/thread/thr_umtx.c:234
        tm_p = 0x8030e17bc <_thr_ast+44>
        tm_size = <error reading variable tm_size (Cannot access memory at address 0x18)>
#2  0x00000008030e8698 in cond_wait_user (abstime=<optimized out>, cancel=1, cvp=<optimized out>, mp=<optimized out>) at /usr/src/lib/libthr/thread/thr_cond.c:304
        curthread = 0x804e17900
        deferred = <error reading variable deferred (Cannot access memory at address 0x0)>
        recurse = 0
        error = <optimized out>
        sq = <optimized out>
        error2 = <optimized out>
#3  cond_wait_common (cond=<optimized out>, mutex=<optimized out>, abstime=0x0, cancel=1) at /usr/src/lib/libthr/thread/thr_cond.c:364
        cvp = 0x8051d1180
        mp = 0x8051d2ea0
        error = <optimized out>
#4  0x0000000800ef1608 in apr_queue_pop (queue=0x805140908, data=0x7fffdff46f50) at misc/apr_queue.c:276
        rv = 0
#5  0x0000000800d40a57 in log_thread (t=<optimized out>, obj=<optimized out>) at src/switch_log.c:294
        pop = 0x0
        node = <optimized out>
        binding = <optimized out>
#6  0x0000000800f013a6 in dummy_worker (opaque=0x805371b38) at threadproc/unix/thread.c:151
No locals.
#7  0x00000008030dcc05 in thread_start (curthread=0x804e17900) at /usr/src/lib/libthr/thread/thr_create.c:289
        set = {__bits = {0, 0, 0, 0}}
#8  0x0000000000000000 in ?? ()
No symbol table info available.
Backtrace stopped: Cannot access memory at address 0x7fffdff47000

Thread 3 (LWP 101042):
#0  0x00000008030ea91c in _umtx_op_err () from /lib/libthr.so.3
No symbol table info available.
#1  0x00000008030debb0 in _thr_umtx_timedwait_uint (mtx=0x80062e020, id=<optimized out>, clockid=<optimized out>, abstime=<optimized out>, shared=<optimized out>) at /usr/src/lib/libthr/thread/thr_umtx.c:234
        tm_p = 0x8030e17bc <_thr_ast+44>
        tm_size = <error reading variable tm_size (Cannot access memory at address 0x18)>
#2  0x00000008030e8698 in cond_wait_user (abstime=<optimized out>, cancel=1, cvp=<optimized out>, mp=<optimized out>) at /usr/src/lib/libthr/thread/thr_cond.c:304
        curthread = 0x804e16a00
        deferred = <error reading variable deferred (Cannot access memory at address 0x0)>
        recurse = 0
        error = <optimized out>
        sq = <optimized out>
        error2 = <optimized out>
#3  cond_wait_common (cond=<optimized out>, mutex=<optimized out>, abstime=0x0, cancel=1) at /usr/src/lib/libthr/thread/thr_cond.c:364
        cvp = 0x804ff75c0
        mp = 0x804e1da40
        error = <optimized out>
#4  0x0000000800ef1608 in apr_queue_pop (queue=0x804e31d08, data=0x7fffdffc0f38) at misc/apr_queue.c:276
        rv = 0
#5  0x0000000800ce578b in switch_event_dispatch_thread (thread=<optimized out>, obj=0x804e31d08) at src/switch_event.c:324
        pop = 0x0
        event = <optimized out>
#6  0x0000000800f013a6 in dummy_worker (opaque=0x805140780) at threadproc/unix/thread.c:151
No locals.
#7  0x00000008030dcc05 in thread_start (curthread=0x804e16a00) at /usr/src/lib/libthr/thread/thr_create.c:289
        set = {__bits = {0, 0, 0, 0}}
#8  0x0000000000000000 in ?? ()
No symbol table info available.
Backtrace stopped: Cannot access memory at address 0x7fffdffc1000

Thread 2 (LWP 101041):
#0  0x0000000803de665a in _select () from /lib/libc.so.7
No symbol table info available.
#1  0x00000008030df9a2 in __thr_select (numfds=0, readfds=0x0, writefds=0x0, exceptfds=0x0, timeout=0x7fffdfffdf10) at /usr/src/lib/libthr/thread/thr_syscalls.c:493
        curthread = 0x804e16500
        ret = <optimized out>
#2  0x0000000800f02749 in apr_sleep (t=<optimized out>) at time/unix/time.c:246
        tv = {tv_sec = 1, tv_usec = 0}
#3  0x0000000800d4ce25 in do_sleep (t=0) at src/switch_time.c:164
        ts = {tv_sec = 140736951476056, tv_nsec = 140736951476112}
#4  switch_sleep (t=0) at src/switch_time.c:632
No locals.
#5  0x0000000800c78e51 in pool_thread (thread=<optimized out>, obj=<optimized out>) at src/switch_core_memory.c:569
        len = 0
#6  0x0000000800f013a6 in dummy_worker (opaque=0x804f13740) at threadproc/unix/thread.c:151
No locals.
#7  0x00000008030dcc05 in thread_start (curthread=0x804e16500) at /usr/src/lib/libthr/thread/thr_create.c:289
        set = {__bits = {0, 0, 0, 0}}
#8  0x0000000000000000 in ?? ()
No symbol table info available.
Backtrace stopped: Cannot access memory at address 0x7fffdfffe000

Thread 1 (LWP 100834):
#0  0x00000008030ea91c in _umtx_op_err () from /lib/libthr.so.3
No symbol table info available.
#1  0x00000008030e6f97 in join_common (pthread=0x807ed7700, thread_return=0x7fffffffb658, abstime=0x0) at /usr/src/lib/libthr/thread/thr_join.c:125
        ret = <error reading variable ret (Cannot access memory at address 0x2d)>
        tid = 101080
        ts = {tv_sec = 34411006815, tv_nsec = 34514890792}
        tsp = 0x18ad8
#2  0x0000000800f01421 in apr_thread_join (retval=0x7fffffffb68c, thd=0x8059e9658) at threadproc/unix/thread.c:234
        stat = <optimized out>
#3  0x0000000800cd889e in do_shutdown (module=0x2, shutdown=<optimized out>, unload=SWITCH_TRUE, fail_if_busy=(unknown: 1523667750), err=<optimized out>) at src/switch_loadable_module.c:2031
        st = SWITCH_STATUS_SUCCESS
        pool = <optimized out>
        flags = 12798607
#4  0x0000000800cdb84a in switch_loadable_module_shutdown () at src/switch_loadable_module.c:2083
        i = <error reading variable i (Cannot access memory at address 0x0)>
        hi = <optimized out>
        val = 0x805626460
        module = 0x807ed7700
#5  0x0000000800c9d92c in switch_core_destroy () at src/switch_core.c:2940
        event = 0x0
#6  0x0000000000403b2b in main (argc=<optimized out>, argv=0x7fffffffe988) at src/switch.c:1210
        pid_path = "/opt/freeswitch/var/run/freeswitch/freeswitch.pid", '\000' <repeats 974 times>
        pid_buffer = "45848", '\000' <repeats 26 times>
        old_pid_buffer = "45690", '\000' <repeats 26 times>
        opts_str = '\000' <repeats 1023 times>
        local_argv = {0x7fffffffec40 "/opt/freeswitch/bin/freeswitch", 0x7fffffffec5f "-c", 0x0 <repeats 1022 times>}
        arg_argv = {0x0 <repeats 128 times>}
        local_argc = 2
        reincarnate_reexec = <error reading variable reincarnate_reexec (Cannot access memory at address 0x0)>
        reincarnate = <error reading variable reincarnate (Cannot access memory at address 0x0)>
        do_wait = <error reading variable do_wait (Cannot access memory at address 0x0)>
        nf = <error reading variable nf (Cannot access memory at address 0x0)>
        pid = <optimized out>
        nc = <error reading variable nc (Cannot access memory at address 0x1)>
        ret = <error reading variable ret (Cannot access memory at address 0x0)>
        flags = 68225
        priority = <error reading variable priority (Cannot access memory at address 0x0)>
        do_kill = 0
        run_set = <error reading variable run_set (Cannot access memory at address 0x1)>
        log_set = 0
        alt_base = <error reading variable alt_base (Cannot access memory at address 0x1)>
        alt_dirs = 0
        x = <optimized out>
        opts = <optimized out>
        i = <optimized out>
        runas_group = 0x0
        pid_len = 5
        pool = <optimized out>
        fd = <optimized out>
        old_pid_len = 5
        err = 0x800f28b86 "Success"
        destroy_status = <optimized out>

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"

thread.c (11K) Download Attachment
switch_loadable_module.c (133K) Download Attachment
switch_core.c (136K) Download Attachment
switch.c (47K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Regression between 10 & 11, Freeswitch hangs when exiting

Eugene Grosbein-10
11.02.2018 2:29, Eric Masson пишет:

> Hi,
>
> Freeswitch, a multiprotocol softswitch (http://freewitch.org) runs on
> FreeBSD and a port is available (net/freeswitch/)
>
> Freeswitch runs fine on any 10.x installation, issueing the shutdown
> command from cli makes the application exit cleanly.
>
> On any 11.x installation, issueing the shutdown command from cli makes
> the application hang.
>
> FreeBSD version is :
> FreeBSD newsrv.interne.associated-bears.org 11.1-STABLE FreeBSD 11.1-STABLE #0 r327876M: Sat Jan 13 16:19:26 CET 2018     [hidden email]:/usr/obj/usr/src/sys/SE7525GP2  amd64
>
> Generic or custom kernel behave the same.
>
> On Freeswitch side, 1.6.19 (net/freeswitch) and also git branches master
> & v1.6 show the same issue.
>
> I've reproduced the issue on the latest v1.6, the backtrace of stuck
> process shows the following :
> #0  0x00000008030ea91c in _umtx_op_err () from /lib/libthr.so.3
> #1  0x00000008030e6f97 in join_common (pthread=0x807ed7700, thread_return=0x7fffffffb658, abstime=0x0) at /usr/src/lib/libthr/thread/thr_join.c:125
> #2  0x0000000800f01421 in apr_thread_join (retval=0x7fffffffb68c, thd=0x8059e9658) at threadproc/unix/thread.c:234
> #3  0x0000000800cd889e in do_shutdown (module=0x2, shutdown=<optimized out>, unload=SWITCH_TRUE, fail_if_busy=(unknown: 1523667750), err=<optimized out>) at src/switch_loadable_module.c:2031
> #4  0x0000000800cdb84a in switch_loadable_module_shutdown () at src/switch_loadable_module.c:2083
> #5  0x0000000800c9d92c in switch_core_destroy () at src/switch_core.c:2940
> #6  0x0000000000403b2b in main (argc=<optimized out>, argv=0x7fffffffe988) at src/switch.c:1210
>
> It seems the issue lies in thread handling.
>
> I've attached :
> - the output of a Freeswitch related script used to gather relevant
>   information.
> - source files showing in bt
>
> Thank for help.

See also: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=225584
What CPU do you use?

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Regression between 10 & 11, Freeswitch hangs when exiting

Eric Masson-3
Eugene Grosbein <[hidden email]> writes:

Hi Eugene,

> See also: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=225584

I'll check.

> What CPU do you use?

Older ones...

Intel SE-7525GP2 fitted with 2 Xeons

        Type: Central Processor
        Family: Xeon
        Manufacturer: Intel
        ID: 4A 0F 00 00 FF FB EB BF
        Signature: Type 0, Family 15, Model 4, Stepping 10
        Version: Intel(R) Xeon(TM) CPU 3.60GHz
        Voltage: 1.3 V
        External Clock: 200 MHz
        Max Speed: 3600 MHz
        Current Speed: 3600 MHz

Regards

Éric Masson
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"