kern/174822: Page fault in tcp_discardcb under high traffic

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

kern/174822: Page fault in tcp_discardcb under high traffic

Adam Twardowski

>Number:         174822
>Category:       kern
>Synopsis:       Page fault in tcp_discardcb under high traffic
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:      
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sun Dec 30 08:40:00 UTC 2012
>Closed-Date:
>Last-Modified:
>Originator:     Adam Twardowski
>Release:        9.1-RELEASE
>Organization:
>Environment:
FreeBSD ID11278.choopa.com 9.1-RELEASE FreeBSD 9.1-RELEASE #0 r243825: Tue Dec  4 09:23:10 UTC 2012     [hidden email]:/usr/obj/usr/src/sys/GENERIC  amd64
>Description:
Random kernel page fault's under high network traffic traffic.  I'm running haproxy 1.4.22 to load balance traffic to a cluster of web servers.  I'm seeing the same error on multiple machines and I already ran memtest to verify that there wasn't any issues with the memory.

[root@ID11278 /var/crash]# kgdb /boot/kernel/kernel.symbols vmcore.0
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:


Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address   = 0x28
fault code              = supervisor read data, page not present
instruction pointer     = 0x20:0xffffffff80a7a951
stack pointer           = 0x28:0xffffff88c1836780
frame pointer           = 0x28:0xffffff88c1836800
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 22494 (haproxy)
trap number             = 12
panic: page fault
cpuid = 0
KDB: stack backtrace:
#0 0xffffffff809208a6 at kdb_backtrace+0x66
#1 0xffffffff808ea8be at panic+0x1ce
#2 0xffffffff80bd8240 at trap_fatal+0x290
#3 0xffffffff80bd857d at trap_pfault+0x1ed
#4 0xffffffff80bd8b9e at trap+0x3ce
#5 0xffffffff80bc315f at calltrap+0x8
#6 0xffffffff80a80ffa at tcp_usr_detach+0x11a
#7 0xffffffff80952f84 at sofree+0x134
#8 0xffffffff80953a81 at soclose+0x3c1
#9 0xffffffff808a9653 at _fdrop+0x23
#10 0xffffffff808aa952 at closef+0x52
#11 0xffffffff808ab192 at kern_close+0x172
#12 0xffffffff80bd7ae6 at amd64_syscall+0x546
#13 0xffffffff80bc3447 at Xfast_syscall+0xf7
Uptime: 3d2h59m9s
Dumping 1980 out of 32726 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%

Reading symbols from /boot/kernel/geom_mirror.ko...Reading symbols from /boot/kernel/geom_mirror.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/geom_mirror.ko
Reading symbols from /boot/modules/dscp_rewrite.ko...done.
Loaded symbols for /boot/modules/dscp_rewrite.ko
Reading symbols from /boot/kernel/ipfw.ko...Reading symbols from /boot/kernel/ipfw.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ipfw.ko
#0  doadump (textdump=Variable "textdump" is not available.
) at pcpu.h:224
224     pcpu.h: No such file or directory.
        in pcpu.h
(kgdb) bt
#0  doadump (textdump=Variable "textdump" is not available.
) at pcpu.h:224
#1  0xffffffff808ea3a1 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:448
#2  0xffffffff808ea897 in panic (fmt=0x1 <Address 0x1 out of bounds>) at /usr/src/sys/kern/kern_shutdown.c:636
#3  0xffffffff80bd8240 in trap_fatal (frame=0xc, eva=Variable "eva" is not available.
) at /usr/src/sys/amd64/amd64/trap.c:857
#4  0xffffffff80bd857d in trap_pfault (frame=0xffffff88c18366d0, usermode=0) at /usr/src/sys/amd64/amd64/trap.c:773
#5  0xffffffff80bd8b9e in trap (frame=0xffffff88c18366d0) at /usr/src/sys/amd64/amd64/trap.c:456
#6  0xffffffff80bc315f in calltrap () at /usr/src/sys/amd64/amd64/exception.S:228
#7  0xffffffff80a7a951 in tcp_discardcb (tp=0x0) at /usr/src/sys/netinet/tcp_subr.c:839
#8  0xffffffff80a80ffa in tcp_usr_detach (so=Variable "so" is not available.
) at /usr/src/sys/netinet/tcp_usrreq.c:205
#9  0xffffffff80952f84 in sofree (so=0xfffffe05e6dbe7f8) at /usr/src/sys/kern/uipc_socket.c:647
#10 0xffffffff80953a81 in soclose (so=0xfffffe05e6dbe7f8) at /usr/src/sys/kern/uipc_socket.c:741
#11 0xffffffff808a9653 in _fdrop (fp=0xfffffe01d5f18280, td=Variable "td" is not available.
) at file.h:316
#12 0xffffffff808aa952 in closef (fp=0xfffffe01d5f18280, td=0xfffffe000fcd28e0) at /usr/src/sys/kern/kern_descrip.c:2250
#13 0xffffffff808ab192 in kern_close (td=0xfffffe000fcd28e0, fd=Variable "fd" is not available.
) at /usr/src/sys/kern/kern_descrip.c:1236
#14 0xffffffff80bd7ae6 in amd64_syscall (td=0xfffffe000fcd28e0, traced=0) at subr_syscall.c:135
#15 0xffffffff80bc3447 in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:387
#16 0x0000000800ff1cfc in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb)

>How-To-Repeat:
Not easily repeatable, but it seems to happen to me every couple of days at least.
>Fix:


>Release-Note:
>Audit-Trail:
>Unformatted:
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: kern/174822: Page fault in tcp_discardcb under high traffic

vishal kumar
Hi Adam, did you find any solution for this crash. I too facing a similar crash in my system:

#0  doadump () at  /src/freebsd/sys/kern/kern_shutdown.c:267
#1  0xffffffff80497140 in boot (howto=260) at  /src/freebsd/sys/kern/kern_shutdown.c:495
#2  0xffffffff804978af in panic (fmt=0x0) at /src/freebsd/sys/kern/kern_shutdown.c:693
#3  0xffffffff8071378f in trap_fatal (frame=0xffffff00025a9468, eva=18446744071571844127) at  /src/freebsd/sys/amd64/amd64/trap.c:817
#4  0xffffffff80713c3f in trap_pfault (frame=0xffffff80000612a0, usermode=0) at  /src/freebsd/sys/amd64/amd64/trap.c:714
#5  0xffffffff8071402c in trap (frame=0xffffff80000612a0) at  /src/freebsd/sys/amd64/amd64/trap.c:464
#6  0xffffffff806fb164 in calltrap () at  /src/freebsd/sys/amd64/amd64/exception.S:224
#7  0xffffffff805adc06 in tcp_discardcb (tp=0x0) at  /src/freebsd/sys/netinet/tcp_subr.c:849
#8  0xffffffff805b7475 in tcp_usr_detach (so=Variable "so" is not available.) at  /src/freebsd/sys/netinet/tcp_usrreq.c:235
#9  0xffffffff804fc6d4 in sofree (so=0xffffff001110e000) at  /src/freebsd/sys/kern/uipc_socket.c:753
#10 0xffffffff8059fbfb in tcp_do_segment (m=0xffffff0066233500, th=0x1, so=0xffffff001110e000, tp=0xffffff0025e5b3a0, drop_hdrlen=52, tlen=0, iptos=0 '\0', ti_locked=3) at  /src/freebsd/sys/netinet/tcp_input.c:4350
#11 0xffffffff805a5677 in tcp_input (m=0xffffff0066233500, off0=Variable "off0" is not available.) at  /src/freebsd/sys/netinet/tcp_input.c:1723
#12 0xffffffff8058faac in ip_input (m=0xffffff0066233500) at  /src/freebsd/sys/netinet/ip_input.c:1720
#13 0xffffffff80565139 in swi_net (arg=Variable "arg" is not available.) at  /src/freebsd/sys/net/netisr.c:739
#14 0xffffffff8046f2f4 in intr_event_execute_handlers (p=Variable "p" is not available.) at  /src/freebsd/sys/kern/kern_intr.c:1233
#15 0xffffffff804709a5 in ithread_loop (arg=0xffffff00024c1980) at  /src/freebsd/sys/kern/kern_intr.c:1246
#16 0xffffffff8046ccc8 in fork_exit ( callout=0xffffffff80470910 <ithread_loop>, arg=0xffffff00024c1980, frame=0xffffff8000061c40) at  /src/freebsd/sys/kern/kern_fork.c:866
#17 0xffffffff806fb5be in fork_trampoline () at  /src/freebsd/sys/amd64/amd64/exception.S:565
#18 0x0000000000000000 in ?? ()