libcrypto.so.111 linked binaries SIGSEGV (in bhyve guest)

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

libcrypto.so.111 linked binaries SIGSEGV (in bhyve guest)

Harry Schmalzbauer
Hello,

I'm tryint to upgrade a bhyve guest from stable/11 to stable/12.

pkg(8) for example crashes with signal 11.

I looked for other binaries affected by
ldd /usr/sbin/* | & grep 'signal 11$'
wich gives
/usr/sbin/auditdistd: signal 11
/usr/sbin/bhyve: signal 11
/usr/sbin/bsnmpd: signal
/usr/sbin/gssd: signal 11
/usr/sbin/hostapd: signal 11
/usr/sbin/iprop-log: signal 11
/usr/sbin/keyserv: signal 11
/usr/sbin/kstash: signal 11
/usr/sbin/ktutil: signal 11
/usr/sbin/local-unbound: signal 11
/usr/sbin/local-unbound-anchor: signal 11
/usr/sbin/local-unbound-checkconf: signal 11
/usr/sbin/local-unbound-control: signal 11
/usr/sbin/ntp-keygen: signal 11
/usr/sbin/ntpd: signal 11
/usr/sbin/ntpdate: signal 11
/usr/sbin/ntpdc: signal 11
/usr/sbin/pkg: signal 11
/usr/sbin/ppp: signal 11
/usr/sbin/sntp: signal 11
/usr/sbin/sshd: signal 11
/usr/sbin/tcpdump: signal 11
/usr/sbin/uefisign: signal 11
/usr/sbin/wpa_supplicant: signal 11

They all seem to have in common beeing linked against
'/lib/libcrypto.so.111'

truss /usr/sbin/auditdistd
:
close(3)                                         = 0 (0x0)
openat(AT_FDCWD,"/lib/libcrypto.so.111",O_RDONLY|O_CLOEXEC|O_VERIFY,00)
= 3 (0x3)
fstat(3,{ mode=-r--r--r-- ,inode=150033332,size=3006464,blksize=4096 })
= 0 (0x0)
mmap(0x0,4096,PROT_READ,MAP_PRIVATE|MAP_PREFAULT_READ,3,0x0) =
34362249216 (0x800265000)
mmap(0x0,3104768,PROT_NONE,MAP_GUARD,-1,0x0)     = 34362347520 (0x80027d000)
mmap(0x80027d000,1138688,PROT_READ,MAP_PRIVATE|MAP_FIXED|MAP_NOCORE|MAP_PREFAULT_READ,3,0x0)
= 34362347520 (0x80027d000)
mmap(0x800393000,1757184,PROT_READ|PROT_EXEC,MAP_PRIVATE|MAP_FIXED|MAP_NOCORE|MAP_PREFAULT_READ,3,0x116000)
= 34363486208 (0x800393000)
mmap(0x800540000,196608,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_PREFAULT_READ,3,0x2c3000)
= 34365243392 (0x800540000) SIGNAL 11 (SIGSEGV) code=SEGV_ACCERR
trapno=12 addr=0x80056f790
process killed, signal = 11 (core dumped)

I have no idea how to analyze further or what the reason could be (like
mentioned, all binaries listed dump core after opening lib/libcrypto.so.111

gdb shows:
Core was generated by `/usr/sbin/auditdistd'.
Program terminated with signal 11, Segmentation fault.
Reading symbols from /lib/libutil.so.9...Reading symbols from
/usr/lib/debug//lib/libutil.so.9.debug...done.
done.
Loaded symbols for /lib/libutil.so.9
Reading symbols from /libexec/ld-elf.so.1...Reading symbols from
/usr/lib/debug//libexec/ld-elf.so.1.debug...done.
done.
Loaded symbols for /libexec/ld-elf.so.1
#0  memset (dest=0x80056f790, c=0, len=<value optimized out>)
     at
/usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/rtld.c:5624
5624                    ((char *)dest)[i] = c;
(gdb) bt
#0  memset (dest=0x80056f790, c=0, len=<value optimized out>)
     at
/usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/rtld.c:5624
#1  0x0000000800235b07 in map_object (fd=3, path=0x800246140
"/lib/libcrypto.so.111",
     sb=0x7fffffffd4a8)
     at
/usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/map_object.c:249
#2  0x0000000800230806 in load_object (name=0x201dba "libcrypto.so.111",
fd_u=-1,
     refobj=0x800248000, flags=<value optimized out>)
     at
/usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/rtld.c:2493
#3  0x0000000800229972 in _rtld (sp=<value optimized out>,
exit_proc=0x7fffffffea30,
     objp=0x7fffffffea38)
     at
/usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/rtld.c:2315
#4  0x0000000800228019 in .rtld_start ()
     at
/usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/amd64/rtld_start.S:39
#5  0x0000000000000000 in ?? ()
Current language:  auto; currently minimal

Any help highly appreciated.

This is with a live CD (amd64), compiled with stable/12 from today (so
clang 7.01).
The bhyve guest has 2GB hardwired and ran stable/11 beforehand, which
compiled the live CD.
bhyve host is 11.2.  But that shouldn't play a role, does it?

-harry

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Strange rtld-elf failure on stable/12 [Was: libcrypto.so.111 linked binaries SIGSEGV (in bhyve guest)]

Harry Schmalzbauer
Am 20.02.2019 um 17:51 schrieb Harry Schmalzbauer:
> Hello,
>

> gdb shows:
> Core was generated by `/usr/sbin/auditdistd'.
> Program terminated with signal 11, Segmentation fault.
> Reading symbols from /lib/libutil.so.9...Reading symbols from
> /usr/lib/debug//lib/libutil.so.9.debug...done.
> done.
> Loaded symbols for /lib/libutil.so.9
> Reading symbols from /libexec/ld-elf.so.1...Reading symbols from
> /usr/lib/debug//libexec/ld-elf.so.1.debug...done.
> done.
> Loaded symbols for /libexec/ld-elf.so.1
> #0  memset (dest=0x80056f790, c=0, len=<value optimized out>)
>     at
> /usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/rtld.c:5624
> 5624                    ((char *)dest)[i] = c;
> (gdb) bt
> #0  memset (dest=0x80056f790, c=0, len=<value optimized out>)
>     at
> /usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/rtld.c:5624
> #1  0x0000000800235b07 in map_object (fd=3, path=0x800246140
> "/lib/libcrypto.so.111",
>     sb=0x7fffffffd4a8)
>     at
> /usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/map_object.c:249
> #2  0x0000000800230806 in load_object (name=0x201dba
> "libcrypto.so.111", fd_u=-1,
>     refobj=0x800248000, flags=<value optimized out>)
>     at
> /usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/rtld.c:2493
> #3  0x0000000800229972 in _rtld (sp=<value optimized out>,
> exit_proc=0x7fffffffea30,
>     objp=0x7fffffffea38)
>     at
> /usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/rtld.c:2315
> #4  0x0000000800228019 in .rtld_start ()
>     at
> /usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/amd64/rtld_start.S:39
> #5  0x0000000000000000 in ?? ()
> Current language:  auto; currently minimal
>
> Any help highly appreciated.
>
> This is with a live CD (amd64), compiled with stable/12 from today (so
> clang 7.01).
> The bhyve guest has 2GB hardwired and ran stable/11 beforehand, which
> compiled the live CD.
> bhyve host is 11.2.  But that shouldn't play a role, does it?

I'm really interested what happens here.
I built stable/11 in that bhyve guest and updated that guest to
stable/11 from yesterday.
To my surpise llvm 7.01 was also merged to stable/11.  Thank you for
that great supprt!
No problems with any binary in the stable/11 bhyve guest.

Then I built stable/12 in that re-built stable/11 guest.
As result, again all binaries linked to /lib/libcrypto.so.111 crash
(signal 11) with the stable/12 iso in the same bhyve guest.

Here the example from ntpq:
Program terminated with signal 11, Segmentation fault.
Reading symbols from /lib/libedit.so.7...Reading symbols from
/usr/lib/debug//lib/libedit.so.7.debug...done.
done.
Loaded symbols for /lib/libedit.so.7
Reading symbols from /lib/libm.so.5...Reading symbols from
/usr/lib/debug//lib/libm.so.5.debug...done.
done.
Loaded symbols for /lib/libm.so.5
Reading symbols from /libexec/ld-elf.so.1...Reading symbols from
/usr/lib/debug//libexec/ld-elf.so.1.debug...done.
done.
#0  memset (dest=0x8005ef790, c=0, len=<value optimized out>) at
/usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/rtld.c:5624
5624                    ((char *)dest)[i] = c;
(gdb) bt
#0  memset (dest=0x8005ef790, c=0, len=<value optimized out>) at
/usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/rtld.c:5624
#1  0x000000080025db07 in map_object (fd=3, path=0x80026e1a0
"/lib/libcrypto.so.111", sb=0x7fffffffd4c8) at
/usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/map_object.c:249
#2  0x0000000800258806 in load_object (name=0x201b40 "libcrypto.so.111",
fd_u=-1, refobj=0x800270000, flags=<value optimized out>) at
/usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/rtld.c:2493
#3  0x0000000800251972 in _rtld (sp=<value optimized out>,
exit_proc=0x7fffffffea50, objp=0x7fffffffea58) at
/usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/rtld.c:2315
#4  0x0000000800250019 in .rtld_start () at
/usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/amd64/rtld_start.S:39
#5  0x0000000000000000 in ?? ()

So please correct me if I'm comletely wrong, but the problem here seems
to be reproducably rtld-elf related.
Unfortunately I don't know anything about object files and linkers and
the related fundamental stuff.
But maybe someone else has an idea what's going wrong here?

Thanks,

-Harry
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Strange rtld-elf failure on stable/12 [Was: libcrypto.so.111 linked binaries SIGSEGV (in bhyve guest)]

Konstantin Belousov
On Thu, Feb 21, 2019 at 09:24:43AM +0100, Harry Schmalzbauer wrote:

> Am 20.02.2019 um 17:51 schrieb Harry Schmalzbauer:
> > Hello,
> >
> …
> > gdb shows:
> > Core was generated by `/usr/sbin/auditdistd'.
> > Program terminated with signal 11, Segmentation fault.
> > Reading symbols from /lib/libutil.so.9...Reading symbols from
> > /usr/lib/debug//lib/libutil.so.9.debug...done.
> > done.
> > Loaded symbols for /lib/libutil.so.9
> > Reading symbols from /libexec/ld-elf.so.1...Reading symbols from
> > /usr/lib/debug//libexec/ld-elf.so.1.debug...done.
> > done.
> > Loaded symbols for /libexec/ld-elf.so.1
> > #0  memset (dest=0x80056f790, c=0, len=<value optimized out>)
> >     at
> > /usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/rtld.c:5624
> > 5624                    ((char *)dest)[i] = c;
> > (gdb) bt
> > #0  memset (dest=0x80056f790, c=0, len=<value optimized out>)
> >     at
> > /usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/rtld.c:5624
> > #1  0x0000000800235b07 in map_object (fd=3, path=0x800246140
> > "/lib/libcrypto.so.111",
> >     sb=0x7fffffffd4a8)
> >     at
> > /usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/map_object.c:249
> > #2  0x0000000800230806 in load_object (name=0x201dba
> > "libcrypto.so.111", fd_u=-1,
> >     refobj=0x800248000, flags=<value optimized out>)
> >     at
> > /usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/rtld.c:2493
> > #3  0x0000000800229972 in _rtld (sp=<value optimized out>,
> > exit_proc=0x7fffffffea30,
> >     objp=0x7fffffffea38)
> >     at
> > /usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/rtld.c:2315
> > #4  0x0000000800228019 in .rtld_start ()
> >     at
> > /usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/amd64/rtld_start.S:39
> > #5  0x0000000000000000 in ?? ()
> > Current language:  auto; currently minimal
> >
> > Any help highly appreciated.
> >
> > This is with a live CD (amd64), compiled with stable/12 from today (so
> > clang 7.01).
> > The bhyve guest has 2GB hardwired and ran stable/11 beforehand, which
> > compiled the live CD.
> > bhyve host is 11.2.  But that shouldn't play a role, does it?
>
> I'm really interested what happens here.
> I built stable/11 in that bhyve guest and updated that guest to
> stable/11 from yesterday.
> To my surpise llvm 7.01 was also merged to stable/11.  Thank you for
> that great supprt!
> No problems with any binary in the stable/11 bhyve guest.
>
> Then I built stable/12 in that re-built stable/11 guest.
> As result, again all binaries linked to /lib/libcrypto.so.111 crash
> (signal 11) with the stable/12 iso in the same bhyve guest.
>
> Here the example from ntpq:
> Program terminated with signal 11, Segmentation fault.
> Reading symbols from /lib/libedit.so.7...Reading symbols from
> /usr/lib/debug//lib/libedit.so.7.debug...done.
> done.
> Loaded symbols for /lib/libedit.so.7
> Reading symbols from /lib/libm.so.5...Reading symbols from
> /usr/lib/debug//lib/libm.so.5.debug...done.
> done.
> Loaded symbols for /lib/libm.so.5
> Reading symbols from /libexec/ld-elf.so.1...Reading symbols from
> /usr/lib/debug//libexec/ld-elf.so.1.debug...done.
> done.
> #0  memset (dest=0x8005ef790, c=0, len=<value optimized out>) at
> /usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/rtld.c:5624
> 5624                    ((char *)dest)[i] = c;
> (gdb) bt
> #0  memset (dest=0x8005ef790, c=0, len=<value optimized out>) at
> /usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/rtld.c:5624
> #1  0x000000080025db07 in map_object (fd=3, path=0x80026e1a0
> "/lib/libcrypto.so.111", sb=0x7fffffffd4c8) at
> /usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/map_object.c:249
> #2  0x0000000800258806 in load_object (name=0x201b40 "libcrypto.so.111",
> fd_u=-1, refobj=0x800270000, flags=<value optimized out>) at
> /usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/rtld.c:2493
> #3  0x0000000800251972 in _rtld (sp=<value optimized out>,
> exit_proc=0x7fffffffea50, objp=0x7fffffffea58) at
> /usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/rtld.c:2315
> #4  0x0000000800250019 in .rtld_start () at
> /usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/amd64/rtld_start.S:39
> #5  0x0000000000000000 in ?? ()
>
> So please correct me if I'm comletely wrong, but the problem here seems
> to be reproducably rtld-elf related.
> Unfortunately I don't know anything about object files and linkers and
> the related fundamental stuff.
If you do not know about linkers, why do you claim that the problem
is related to rtld ?

> But maybe someone else has an idea what's going wrong here?

The fault happens during zeroing of bss.  Most likely it is due to some
strangeness of the object being loaded.  For diagnostic, show
the output of "readelf -a libcrypto.so.111".
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: libcrypto.so.111 linked binaries SIGSEGV (in bhyve guest)

Harry Schmalzbauer
Am 21.02.2019 um 09:54 schrieb Konstantin Belousov:

> On Thu, Feb 21, 2019 at 09:24:43AM +0100, Harry Schmalzbauer wrote:
>> Am 20.02.2019 um 17:51 schrieb Harry Schmalzbauer:
>>> Hello,
>>>
>> …
>>> gdb shows:
>>> Core was generated by `/usr/sbin/auditdistd'.
>>> Program terminated with signal 11, Segmentation fault.
>>> Reading symbols from /lib/libutil.so.9...Reading symbols from
>>> /usr/lib/debug//lib/libutil.so.9.debug...done.
>>> done.
>>> Loaded symbols for /lib/libutil.so.9
>>> Reading symbols from /libexec/ld-elf.so.1...Reading symbols from
>>> /usr/lib/debug//libexec/ld-elf.so.1.debug...done.
>>> done.
>>> Loaded symbols for /libexec/ld-elf.so.1
>>> #0  memset (dest=0x80056f790, c=0, len=<value optimized out>)
>>>      at
>>> /usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/rtld.c:5624
>>> 5624                    ((char *)dest)[i] = c;
>>> (gdb) bt
>>> #0  memset (dest=0x80056f790, c=0, len=<value optimized out>)
>>>      at
>>> /usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/rtld.c:5624
>>> #1  0x0000000800235b07 in map_object (fd=3, path=0x800246140
>>> "/lib/libcrypto.so.111",
>>>      sb=0x7fffffffd4a8)
>>>      at
>>> /usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/map_object.c:249
>>> #2  0x0000000800230806 in load_object (name=0x201dba
>>> "libcrypto.so.111", fd_u=-1,
>>>      refobj=0x800248000, flags=<value optimized out>)
>>>      at
>>> /usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/rtld.c:2493
>>> #3  0x0000000800229972 in _rtld (sp=<value optimized out>,
>>> exit_proc=0x7fffffffea30,
>>>      objp=0x7fffffffea38)
>>>      at
>>> /usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/rtld.c:2315
>>> #4  0x0000000800228019 in .rtld_start ()
>>>      at
>>> /usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/amd64/rtld_start.S:39
>>> #5  0x0000000000000000 in ?? ()
>>> Current language:  auto; currently minimal
>>>
>>> Any help highly appreciated.
>>>
>>> This is with a live CD (amd64), compiled with stable/12 from today (so
>>> clang 7.01).
>>> The bhyve guest has 2GB hardwired and ran stable/11 beforehand, which
>>> compiled the live CD.
>>> bhyve host is 11.2.  But that shouldn't play a role, does it?
>>
>> I'm really interested what happens here.
>> I built stable/11 in that bhyve guest and updated that guest to
>> stable/11 from yesterday.
>> To my surpise llvm 7.01 was also merged to stable/11.  Thank you for
>> that great supprt!
>> No problems with any binary in the stable/11 bhyve guest.
>>
>> Then I built stable/12 in that re-built stable/11 guest.
>> As result, again all binaries linked to /lib/libcrypto.so.111 crash
>> (signal 11) with the stable/12 iso in the same bhyve guest.
>>
>> Here the example from ntpq:
>> Program terminated with signal 11, Segmentation fault.
>> Reading symbols from /lib/libedit.so.7...Reading symbols from
>> /usr/lib/debug//lib/libedit.so.7.debug...done.
>> done.
>> Loaded symbols for /lib/libedit.so.7
>> Reading symbols from /lib/libm.so.5...Reading symbols from
>> /usr/lib/debug//lib/libm.so.5.debug...done.
>> done.
>> Loaded symbols for /lib/libm.so.5
>> Reading symbols from /libexec/ld-elf.so.1...Reading symbols from
>> /usr/lib/debug//libexec/ld-elf.so.1.debug...done.
>> done.
>> #0  memset (dest=0x8005ef790, c=0, len=<value optimized out>) at
>> /usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/rtld.c:5624
>> 5624                    ((char *)dest)[i] = c;
>> (gdb) bt
>> #0  memset (dest=0x8005ef790, c=0, len=<value optimized out>) at
>> /usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/rtld.c:5624
>> #1  0x000000080025db07 in map_object (fd=3, path=0x80026e1a0
>> "/lib/libcrypto.so.111", sb=0x7fffffffd4c8) at
>> /usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/map_object.c:249
>> #2  0x0000000800258806 in load_object (name=0x201b40 "libcrypto.so.111",
>> fd_u=-1, refobj=0x800270000, flags=<value optimized out>) at
>> /usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/rtld.c:2493
>> #3  0x0000000800251972 in _rtld (sp=<value optimized out>,
>> exit_proc=0x7fffffffea50, objp=0x7fffffffea58) at
>> /usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/rtld.c:2315
>> #4  0x0000000800250019 in .rtld_start () at
>> /usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/amd64/rtld_start.S:39
>> #5  0x0000000000000000 in ?? ()
>>
>> So please correct me if I'm comletely wrong, but the problem here seems
>> to be reproducably rtld-elf related.
>> Unfortunately I don't know anything about object files and linkers and
>> the related fundamental stuff.
> If you do not know about linkers, why do you claim that the problem
> is related to rtld ?
>
>> But maybe someone else has an idea what's going wrong here?
>
> The fault happens during zeroing of bss.  Most likely it is due to some
> strangeness of the object being loaded.  For diagnostic, show
> the output of "readelf -a libcrypto.so.111".

Thanks for your help!
I just guess it's rtld related, since I obviously misinterpreted the
backtrace.  Reverting topic change…

ELF Header:
   Magic:   7f 45 4c 46 02 01 01 09 00 00 00 00 00 00 00 00
   Class:                             ELF64
   Data:                              2's complement, little endian
   Version:                           1 (current)
   OS/ABI:                            FreeBSD
   ABI Version:                       0
   Type:                              DYN (Shared object file)
   Machine:                           Advanced Micro Devices x86-64
   Version:                           0x1
   Entry point address:               0x116000
   Start of program headers:          64 (bytes into file)
   Start of section headers:          3090864 (bytes into file)
   Flags:                             0
   Size of this header:               64 (bytes)
   Size of program headers:           56 (bytes)
   Number of program headers:         8
   Size of section headers:           64 (bytes)
   Number of section headers:         29
   Section header string table index: 28

Elf file type is DYN (Shared object file)
Entry point 0x116000
There are 8 program headers, starting at offset 64

Program Headers:
   Type           Offset             VirtAddr           PhysAddr
                  FileSiz            MemSiz              Flg    Align
   PHDR           0x0000000000000040 0x0000000000000040 0x0000000000000040
                  0x00000000000001c0 0x00000000000001c0  R      0x8
   LOAD           0x0000000000000000 0x0000000000000000 0x0000000000000000
                  0x0000000000115a7c 0x0000000000115a7c  R      0x1000
   LOAD           0x0000000000116000 0x0000000000116000 0x0000000000116000
                  0x00000000001acb20 0x00000000001acb20  R E    0x1000
   LOAD           0x00000000002c3000 0x00000000002c3000 0x00000000002c3000
                  0x000000000002f790 0x00000000000325e0  RW     0x1000
   DYNAMIC        0x00000000002f1a80 0x00000000002f1a80 0x00000000002f1a80
                  0x0000000000000190 0x0000000000000190  RW     0x8
   GNU_RELRO      0x00000000002c9000 0x00000000002c9000 0x00000000002c9000
                  0x0000000000029790 0x0000000000029790  R      0x1
   GNU_EH_FRAME   0x00000000000d0050 0x00000000000d0050 0x00000000000d0050
                  0x000000000000bc74 0x000000000000bc74  R      0x4
   GNU_STACK      0x0000000000000000 0x0000000000000000 0x0000000000000000
                  0x0000000000000000 0x0000000000000000  RW     0

  Section to Segment mapping:
   Segment Sections...
    00
    01     (null) (null) (null) (null) (null) (null) (null) (null)
(null) (null) (null) (null) (null) (null) (null) (null) (null) (null)
(null) (null) (null) (null) (null) (null) (null) (null) (null) (null)
    02
    03
    04
    05
    06
    07     (null) (null) (null) (null) (null) (null) (null) (null)
(null) (null) (null) (null) (null) (null) (null) (null) (null) (null)
(null) (null) (null) (null) (null) (null) (null) (null) (null) (null)
There are 29 section headers, starting at offset 0x2f29b0:

Section Headers:
   [Nr] Name              Type             Address           Offset
        Size              EntSize          Flags  Link  Info  Align
   [ 0] (null)            NULL             0000000000000000  00000000
        0000000000000000  0000000000000000           0     0     0
:
:
:
   [28] (null)            NULL             0000000000000000  00000000
        0000000000000000  0000000000000000           0     0     0
Key to Flags:
   W (write), A (alloc), X (execute), M (merge), S (strings)
   I (info), L (link order), G (group), x (unknown)
   O (extra OS processing required) o (OS specific), p (processor specific)

There is no dynamic section in this file.

Thanks a lot,

-harry
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: libcrypto.so.111 linked binaries SIGSEGV (in bhyve guest)

Konstantin Belousov
On Thu, Feb 21, 2019 at 10:03:29AM +0100, Harry Schmalzbauer wrote:

> Am 21.02.2019 um 09:54 schrieb Konstantin Belousov:
> > On Thu, Feb 21, 2019 at 09:24:43AM +0100, Harry Schmalzbauer wrote:
> >> Am 20.02.2019 um 17:51 schrieb Harry Schmalzbauer:
> >>> Hello,
> >>>
> >> …
> >>> gdb shows:
> >>> Core was generated by `/usr/sbin/auditdistd'.
> >>> Program terminated with signal 11, Segmentation fault.
> >>> Reading symbols from /lib/libutil.so.9...Reading symbols from
> >>> /usr/lib/debug//lib/libutil.so.9.debug...done.
> >>> done.
> >>> Loaded symbols for /lib/libutil.so.9
> >>> Reading symbols from /libexec/ld-elf.so.1...Reading symbols from
> >>> /usr/lib/debug//libexec/ld-elf.so.1.debug...done.
> >>> done.
> >>> Loaded symbols for /libexec/ld-elf.so.1
> >>> #0  memset (dest=0x80056f790, c=0, len=<value optimized out>)
> >>>      at
> >>> /usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/rtld.c:5624
> >>> 5624                    ((char *)dest)[i] = c;
> >>> (gdb) bt
> >>> #0  memset (dest=0x80056f790, c=0, len=<value optimized out>)
> >>>      at
> >>> /usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/rtld.c:5624
> >>> #1  0x0000000800235b07 in map_object (fd=3, path=0x800246140
> >>> "/lib/libcrypto.so.111",
> >>>      sb=0x7fffffffd4a8)
> >>>      at
> >>> /usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/map_object.c:249
> >>> #2  0x0000000800230806 in load_object (name=0x201dba
> >>> "libcrypto.so.111", fd_u=-1,
> >>>      refobj=0x800248000, flags=<value optimized out>)
> >>>      at
> >>> /usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/rtld.c:2493
> >>> #3  0x0000000800229972 in _rtld (sp=<value optimized out>,
> >>> exit_proc=0x7fffffffea30,
> >>>      objp=0x7fffffffea38)
> >>>      at
> >>> /usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/rtld.c:2315
> >>> #4  0x0000000800228019 in .rtld_start ()
> >>>      at
> >>> /usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/amd64/rtld_start.S:39
> >>> #5  0x0000000000000000 in ?? ()
> >>> Current language:  auto; currently minimal
> >>>
> >>> Any help highly appreciated.
> >>>
> >>> This is with a live CD (amd64), compiled with stable/12 from today (so
> >>> clang 7.01).
> >>> The bhyve guest has 2GB hardwired and ran stable/11 beforehand, which
> >>> compiled the live CD.
> >>> bhyve host is 11.2.  But that shouldn't play a role, does it?
> >>
> >> I'm really interested what happens here.
> >> I built stable/11 in that bhyve guest and updated that guest to
> >> stable/11 from yesterday.
> >> To my surpise llvm 7.01 was also merged to stable/11.  Thank you for
> >> that great supprt!
> >> No problems with any binary in the stable/11 bhyve guest.
> >>
> >> Then I built stable/12 in that re-built stable/11 guest.
> >> As result, again all binaries linked to /lib/libcrypto.so.111 crash
> >> (signal 11) with the stable/12 iso in the same bhyve guest.
> >>
> >> Here the example from ntpq:
> >> Program terminated with signal 11, Segmentation fault.
> >> Reading symbols from /lib/libedit.so.7...Reading symbols from
> >> /usr/lib/debug//lib/libedit.so.7.debug...done.
> >> done.
> >> Loaded symbols for /lib/libedit.so.7
> >> Reading symbols from /lib/libm.so.5...Reading symbols from
> >> /usr/lib/debug//lib/libm.so.5.debug...done.
> >> done.
> >> Loaded symbols for /lib/libm.so.5
> >> Reading symbols from /libexec/ld-elf.so.1...Reading symbols from
> >> /usr/lib/debug//libexec/ld-elf.so.1.debug...done.
> >> done.
> >> #0  memset (dest=0x8005ef790, c=0, len=<value optimized out>) at
> >> /usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/rtld.c:5624
> >> 5624                    ((char *)dest)[i] = c;
> >> (gdb) bt
> >> #0  memset (dest=0x8005ef790, c=0, len=<value optimized out>) at
> >> /usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/rtld.c:5624
> >> #1  0x000000080025db07 in map_object (fd=3, path=0x80026e1a0
> >> "/lib/libcrypto.so.111", sb=0x7fffffffd4c8) at
> >> /usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/map_object.c:249
> >> #2  0x0000000800258806 in load_object (name=0x201b40 "libcrypto.so.111",
> >> fd_u=-1, refobj=0x800270000, flags=<value optimized out>) at
> >> /usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/rtld.c:2493
> >> #3  0x0000000800251972 in _rtld (sp=<value optimized out>,
> >> exit_proc=0x7fffffffea50, objp=0x7fffffffea58) at
> >> /usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/rtld.c:2315
> >> #4  0x0000000800250019 in .rtld_start () at
> >> /usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/amd64/rtld_start.S:39
> >> #5  0x0000000000000000 in ?? ()
> >>
> >> So please correct me if I'm comletely wrong, but the problem here seems
> >> to be reproducably rtld-elf related.
> >> Unfortunately I don't know anything about object files and linkers and
> >> the related fundamental stuff.
> > If you do not know about linkers, why do you claim that the problem
> > is related to rtld ?
> >
> >> But maybe someone else has an idea what's going wrong here?
> >
> > The fault happens during zeroing of bss.  Most likely it is due to some
> > strangeness of the object being loaded.  For diagnostic, show
> > the output of "readelf -a libcrypto.so.111".
>
> Thanks for your help!
> I just guess it's rtld related, since I obviously misinterpreted the
> backtrace.  Reverting topic change…
>
> ELF Header:
>    Magic:   7f 45 4c 46 02 01 01 09 00 00 00 00 00 00 00 00
>    Class:                             ELF64
>    Data:                              2's complement, little endian
>    Version:                           1 (current)
>    OS/ABI:                            FreeBSD
>    ABI Version:                       0
>    Type:                              DYN (Shared object file)
>    Machine:                           Advanced Micro Devices x86-64
>    Version:                           0x1
>    Entry point address:               0x116000
>    Start of program headers:          64 (bytes into file)
>    Start of section headers:          3090864 (bytes into file)
>    Flags:                             0
>    Size of this header:               64 (bytes)
>    Size of program headers:           56 (bytes)
>    Number of program headers:         8
>    Size of section headers:           64 (bytes)
>    Number of section headers:         29
>    Section header string table index: 28
>
> Elf file type is DYN (Shared object file)
> Entry point 0x116000
> There are 8 program headers, starting at offset 64
>
> Program Headers:
>    Type           Offset             VirtAddr           PhysAddr
>                   FileSiz            MemSiz              Flg    Align
>    PHDR           0x0000000000000040 0x0000000000000040 0x0000000000000040
>                   0x00000000000001c0 0x00000000000001c0  R      0x8
>    LOAD           0x0000000000000000 0x0000000000000000 0x0000000000000000
>                   0x0000000000115a7c 0x0000000000115a7c  R      0x1000
>    LOAD           0x0000000000116000 0x0000000000116000 0x0000000000116000
>                   0x00000000001acb20 0x00000000001acb20  R E    0x1000
>    LOAD           0x00000000002c3000 0x00000000002c3000 0x00000000002c3000
>                   0x000000000002f790 0x00000000000325e0  RW     0x1000
>    DYNAMIC        0x00000000002f1a80 0x00000000002f1a80 0x00000000002f1a80
>                   0x0000000000000190 0x0000000000000190  RW     0x8
>    GNU_RELRO      0x00000000002c9000 0x00000000002c9000 0x00000000002c9000
>                   0x0000000000029790 0x0000000000029790  R      0x1
>    GNU_EH_FRAME   0x00000000000d0050 0x00000000000d0050 0x00000000000d0050
>                   0x000000000000bc74 0x000000000000bc74  R      0x4
>    GNU_STACK      0x0000000000000000 0x0000000000000000 0x0000000000000000
>                   0x0000000000000000 0x0000000000000000  RW     0
>
>   Section to Segment mapping:
>    Segment Sections...
>     00
>     01     (null) (null) (null) (null) (null) (null) (null) (null)
> (null) (null) (null) (null) (null) (null) (null) (null) (null) (null)
> (null) (null) (null) (null) (null) (null) (null) (null) (null) (null)
>     02
>     03
>     04
>     05
>     06
>     07     (null) (null) (null) (null) (null) (null) (null) (null)
> (null) (null) (null) (null) (null) (null) (null) (null) (null) (null)
> (null) (null) (null) (null) (null) (null) (null) (null) (null) (null)
> There are 29 section headers, starting at offset 0x2f29b0:
>
> Section Headers:
>    [Nr] Name              Type             Address           Offset
>         Size              EntSize          Flags  Link  Info  Align
>    [ 0] (null)            NULL             0000000000000000  00000000
>         0000000000000000  0000000000000000           0     0     0
> :
> :
> :
>    [28] (null)            NULL             0000000000000000  00000000
>         0000000000000000  0000000000000000           0     0     0
> Key to Flags:
>    W (write), A (alloc), X (execute), M (merge), S (strings)
>    I (info), L (link order), G (group), x (unknown)
>    O (extra OS processing required) o (OS specific), p (processor specific)
>
> There is no dynamic section in this file.

The object is clearly corrupted.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: libcrypto.so.111 linked binaries SIGSEGV (in bhyve guest)

Patrick M. Hausen
In reply to this post by Harry Schmalzbauer
Hello,

I don’t know if this is related or not, but when I compile
the Nextcloud client port

        https://svnweb.freebsd.org/ports/head/deskutils/nextcloudclient/

on 11.2 by setting

        DEFAULT_VERSIONS+=ssl=openssl111

it dumps core, too.

Kind regards
Patrick
--
punkt.de GmbH Internet - Dienstleistungen - Beratung
Kaiserallee 13a Tel.: 0721 9109-0 Fax: -100
76133 Karlsruhe [hidden email] http://punkt.de
AG Mannheim 108285 Gf: Juergen Egeling

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: libcrypto.so.111 linked binaries SIGSEGV (in bhyve guest)

Harry Schmalzbauer
In reply to this post by Konstantin Belousov
Am 21.02.2019 um 10:36 schrieb Konstantin Belousov:

>>
>> ELF Header:
>>     Magic:   7f 45 4c 46 02 01 01 09 00 00 00 00 00 00 00 00
>>     Class:                             ELF64
>>     Data:                              2's complement, little endian
>>     Version:                           1 (current)
>>     OS/ABI:                            FreeBSD
>>     ABI Version:                       0
>>     Type:                              DYN (Shared object file)
>>     Machine:                           Advanced Micro Devices x86-64
>>     Version:                           0x1
>>     Entry point address:               0x116000
>>     Start of program headers:          64 (bytes into file)
>>     Start of section headers:          3090864 (bytes into file)
>>     Flags:                             0
>>     Size of this header:               64 (bytes)
>>     Size of program headers:           56 (bytes)
>>     Number of program headers:         8
>>     Size of section headers:           64 (bytes)
>>     Number of section headers:         29
>>     Section header string table index: 28
>>
>> Elf file type is DYN (Shared object file)
>> Entry point 0x116000
>> There are 8 program headers, starting at offset 64
>>
>> Program Headers:
>>     Type           Offset             VirtAddr           PhysAddr
>>                    FileSiz            MemSiz              Flg    Align
>>     PHDR           0x0000000000000040 0x0000000000000040 0x0000000000000040
>>                    0x00000000000001c0 0x00000000000001c0  R      0x8
>>     LOAD           0x0000000000000000 0x0000000000000000 0x0000000000000000
>>                    0x0000000000115a7c 0x0000000000115a7c  R      0x1000
>>     LOAD           0x0000000000116000 0x0000000000116000 0x0000000000116000
>>                    0x00000000001acb20 0x00000000001acb20  R E    0x1000
>>     LOAD           0x00000000002c3000 0x00000000002c3000 0x00000000002c3000
>>                    0x000000000002f790 0x00000000000325e0  RW     0x1000
>>     DYNAMIC        0x00000000002f1a80 0x00000000002f1a80 0x00000000002f1a80
>>                    0x0000000000000190 0x0000000000000190  RW     0x8
>>     GNU_RELRO      0x00000000002c9000 0x00000000002c9000 0x00000000002c9000
>>                    0x0000000000029790 0x0000000000029790  R      0x1
>>     GNU_EH_FRAME   0x00000000000d0050 0x00000000000d0050 0x00000000000d0050
>>                    0x000000000000bc74 0x000000000000bc74  R      0x4
>>     GNU_STACK      0x0000000000000000 0x0000000000000000 0x0000000000000000
>>                    0x0000000000000000 0x0000000000000000  RW     0
>>
>>    Section to Segment mapping:
>>     Segment Sections...
>>      00
>>      01     (null) (null) (null) (null) (null) (null) (null) (null)
>> (null) (null) (null) (null) (null) (null) (null) (null) (null) (null)
>> (null) (null) (null) (null) (null) (null) (null) (null) (null) (null)
>>      02
>>      03
>>      04
>>      05
>>      06
>>      07     (null) (null) (null) (null) (null) (null) (null) (null)
>> (null) (null) (null) (null) (null) (null) (null) (null) (null) (null)
>> (null) (null) (null) (null) (null) (null) (null) (null) (null) (null)
>> There are 29 section headers, starting at offset 0x2f29b0:
>>



> The object is clearly corrupted.

Thanks to your hint to readelf, I found out that it gets corrupted
during dump(8) (or resotore, not yet analyzed).
The obj tree contains the good version, the dump archive not.
The dump archive is used as source for the ISO, hence the described errors.
Now I have to dig in 10 years old deployment scripts to track down and
reproduce the corruption.  No explanation so far, but for sure no
rtld-elf problem :-)
And also not a problem in the FreeBSD make chain, building stable/12 on
stable/11 works as intended and doesn't produce the mutilated
libcrypto.so.111!

Thanks,

-harry
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: libcrypto.so.111 linked binaries SIGSEGV (in bhyve guest)

Eugene Grosbein-10
21.02.2019 22:27, Harry Schmalzbauer wrote:

>> The object is clearly corrupted.
>
> Thanks to your hint to readelf, I found out that it gets corrupted during dump(8) (or resotore, not yet analyzed).
> The obj tree contains the good version, the dump archive not.
> The dump archive is used as source for the ISO, hence the described errors.
> Now I have to dig in 10 years old deployment scripts to track down and reproduce the corruption.  No explanation so far, but for sure no rtld-elf problem :-)
> And also not a problem in the FreeBSD make chain, building stable/12 on stable/11 works as intended and doesn't produce the mutilated libcrypto.so.111!

You may find useful reading trail of this PR https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=228174

Long story short: dump(8) will read inconsistent data (or even garbage) from mounted file system
unless used with -L to make and dump a snapshot. And UFS snapshots are not compatible with SU+J UFS
created with installer by default in some versions of FreeBSD.

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: libcrypto.so.111 linked binaries SIGSEGV (in bhyve guest)

Harry Schmalzbauer
Am 22.02.2019 um 04:51 schrieb Eugene Grosbein:

> 21.02.2019 22:27, Harry Schmalzbauer wrote:
>
>>> The object is clearly corrupted.
>>
>> Thanks to your hint to readelf, I found out that it gets corrupted during dump(8) (or resotore, not yet analyzed).
>> The obj tree contains the good version, the dump archive not.
>> The dump archive is used as source for the ISO, hence the described errors.
>> Now I have to dig in 10 years old deployment scripts to track down and reproduce the corruption.  No explanation so far, but for sure no rtld-elf problem :-)
>> And also not a problem in the FreeBSD make chain, building stable/12 on stable/11 works as intended and doesn't produce the mutilated libcrypto.so.111!
>
> You may find useful reading trail of this PR https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=228174
>
> Long story short: dump(8) will read inconsistent data (or even garbage) from mounted file system
> unless used with -L to make and dump a snapshot. And UFS snapshots are not compatible with SU+J UFS
> created with installer by default in some versions of FreeBSD.

Thanks a lot for that additional relevant information.  I'm aware about
the -L & SU+J problem.  And I'm not conviced, the default installer
settings handle this situation correctly, at least not for the root
filesystem!

My issue was unrelated though.
I dump(8)ed a unmounted md(4), but restore(8) hasn't had enough space
(only view bytes, so size of the corrupted file wasn't obviously wrong)
and the deployment script hasn't checked the return status at all.
Fixed the script and now the restore(8)ed libcrypto.so.111 works.

Thanks,

-harry

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"