ptrace: SIGTRAP and EXIT race

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

ptrace: SIGTRAP and EXIT race

Robert
Hello.

Before creating a bug and providing some test code, would ask a community
here.
When tracing a process using ptrace and there are multiple threads in the
tracing process hitting the same breakpoint, sometimes main thread exits
(WIFEXITED(status) is TRUE) before last queued TRAP_BKPT signal(s) have
been delivered to the tracing process. So a final breakpoint hits counter
is less than it should be.

So in the example below:

#include <iostream>
#include <thread>

#include <pthread_np.h>

static const int num_threads = 2;

void foo() {
    for (int i = 0; i < 2; ++i) {
        printf("hi: %d (tid: %d)\n", i, pthread_getthreadid_np());
    }
}

int main() {
    std::thread t[num_threads];

    for (int i = 0; i < num_threads; ++i) {
        t[i] = std::thread(foo);
    }

    for (int i = 0; i < num_threads; ++i) {
        t[i].join();
    }

    return 0;
}

If we set breakpoint to printf, it should be triggered 4 times (tracing
process should receive TRAP_BKPT 4 times). However, in ~1 of 5 runs, it
receives TRAP_BKPT just 2 or 3 times.

Is this expected? Thanks.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: ptrace: SIGTRAP and EXIT race

Konstantin Belousov
On Thu, Feb 21, 2019 at 08:43:20PM -0800, Robert Ayrapetyan wrote:

> Hello.
>
> Before creating a bug and providing some test code, would ask a community
> here.
> When tracing a process using ptrace and there are multiple threads in the
> tracing process hitting the same breakpoint, sometimes main thread exits
> (WIFEXITED(status) is TRUE) before last queued TRAP_BKPT signal(s) have
> been delivered to the tracing process. So a final breakpoint hits counter
> is less than it should be.
>
> So in the example below:
>
> #include <iostream>
> #include <thread>
>
> #include <pthread_np.h>
>
> static const int num_threads = 2;
>
> void foo() {
>     for (int i = 0; i < 2; ++i) {
>         printf("hi: %d (tid: %d)\n", i, pthread_getthreadid_np());
>     }
> }
>
> int main() {
>     std::thread t[num_threads];
>
>     for (int i = 0; i < num_threads; ++i) {
>         t[i] = std::thread(foo);
>     }
>
>     for (int i = 0; i < num_threads; ++i) {
>         t[i].join();
>     }
>
>     return 0;
> }
>
> If we set breakpoint to printf, it should be triggered 4 times (tracing
> process should receive TRAP_BKPT 4 times). However, in ~1 of 5 runs, it
> receives TRAP_BKPT just 2 or 3 times.
>
> Is this expected? Thanks.

I indeed would expect that all four breakpoints triggered before the
main thread exits, assuming that the breakpoints were installed before
the threads are created.  Please provide the stand-alone (and preferrably
non-interactive) test to reproduce the issue.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: ptrace: SIGTRAP and EXIT race

Robert
Hi, thanks for a prompt reply. Here are the instructions of how to
reproduce (sorry for inconvenient way of specifying BP address when running
app):

uname -a
FreeBSD XXX 12.0-RELEASE-p3 FreeBSD 12.0-RELEASE-p3 GENERIC  amd64

cd /tmp
git clone https://github.com/rayrapetyan/ptrace_bug_poc.git
cd ptrace_bug_poc
mkdir build
cd build
cmake ..
make

Run ~20 times:

/tmp/ptrace_bug_poc/build/src/ptrace_test/ptrace_test
/tmp/ptrace_bug_poc/build/src/mt_example/mt_example 0x201385

-------
Note: make sure 0x201385 is a call to <printf@plt> in
"/tmp/ptrace_bug_poc/build/src/mt_example/mt_example":
gdb /tmp/ptrace_bug_poc/build/src/mt_example/mt_example
disassemble foo
-------

Wait fo appearance of:
"BOOM! Invalid BP hits counter (hits: 1, tid: XXXX)"
at the end of the output (most of the times it will be "SUCCESS")

Thanks.


On Fri, Feb 22, 2019 at 2:10 AM Konstantin Belousov <[hidden email]>
wrote:

> On Thu, Feb 21, 2019 at 08:43:20PM -0800, Robert Ayrapetyan wrote:
> > Hello.
> >
> > Before creating a bug and providing some test code, would ask a community
> > here.
> > When tracing a process using ptrace and there are multiple threads in the
> > tracing process hitting the same breakpoint, sometimes main thread exits
> > (WIFEXITED(status) is TRUE) before last queued TRAP_BKPT signal(s) have
> > been delivered to the tracing process. So a final breakpoint hits counter
> > is less than it should be.
> >
> > So in the example below:
> >
> > #include <iostream>
> > #include <thread>
> >
> > #include <pthread_np.h>
> >
> > static const int num_threads = 2;
> >
> > void foo() {
> >     for (int i = 0; i < 2; ++i) {
> >         printf("hi: %d (tid: %d)\n", i, pthread_getthreadid_np());
> >     }
> > }
> >
> > int main() {
> >     std::thread t[num_threads];
> >
> >     for (int i = 0; i < num_threads; ++i) {
> >         t[i] = std::thread(foo);
> >     }
> >
> >     for (int i = 0; i < num_threads; ++i) {
> >         t[i].join();
> >     }
> >
> >     return 0;
> > }
> >
> > If we set breakpoint to printf, it should be triggered 4 times (tracing
> > process should receive TRAP_BKPT 4 times). However, in ~1 of 5 runs, it
> > receives TRAP_BKPT just 2 or 3 times.
> >
> > Is this expected? Thanks.
>
> I indeed would expect that all four breakpoints triggered before the
> main thread exits, assuming that the breakpoints were installed before
> the threads are created.  Please provide the stand-alone (and preferrably
> non-interactive) test to reproduce the issue.
>
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: ptrace: SIGTRAP and EXIT race

Konstantin Belousov
On Fri, Feb 22, 2019 at 03:57:49PM -0800, Robert Ayrapetyan wrote:

> Hi, thanks for a prompt reply. Here are the instructions of how to
> reproduce (sorry for inconvenient way of specifying BP address when running
> app):
>
> uname -a
> FreeBSD XXX 12.0-RELEASE-p3 FreeBSD 12.0-RELEASE-p3 GENERIC  amd64
>
> cd /tmp
> git clone https://github.com/rayrapetyan/ptrace_bug_poc.git
> cd ptrace_bug_poc
> mkdir build
> cd build
> cmake ..
> make
>
> Run ~20 times:
>
> /tmp/ptrace_bug_poc/build/src/ptrace_test/ptrace_test
> /tmp/ptrace_bug_poc/build/src/mt_example/mt_example 0x201385
>
> -------
> Note: make sure 0x201385 is a call to <printf@plt> in
> "/tmp/ptrace_bug_poc/build/src/mt_example/mt_example":
> gdb /tmp/ptrace_bug_poc/build/src/mt_example/mt_example
> disassemble foo
> -------
>
> Wait fo appearance of:
> "BOOM! Invalid BP hits counter (hits: 1, tid: XXXX)"
> at the end of the output (most of the times it will be "SUCCESS")
>

~700 lines of C++ code definitely do not fall under the 'minimal repro'
spec.  I do not to read all of it.

From looking at Debugger::Launch(), it seems that you missed the
required debugger/child synchronization for PT_TRACE_ME. Typically child
does
        raise(SIGSTOP);
immediately after PT_TRACE_ME, and the tracer must consume this signal.
Otherwise the child continues the execution and might just execute the
place where you intend to set a breakpoint. I may missed the sync (or it
might be done by other means in your code), because as I said, I do not
want to read 700 lines of C++.

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: ptrace: SIGTRAP and EXIT race

Robert
raise(SIGSTOP) is not necessary - child stops by itself on the first
instruction after execve.

Let's see what's in these 700 lines of code:

Read\Write memory (this is for Add\Remove breakpoints);
TrapWait (main debugger cycle, handles threads born\destroy);
Init\Launch\Attach - this is all necessary to bootstrap;
GetNumLwps\GetLwpList - misc, you can't work with threads without these;
Step\Continue - self explanatory, you can't reproduce BP issue without
these functions;
GetRegs\SetRegs\GetEip\SetEip - all are needed for handling logic around
BPs.

That's it! There is one more function - EntryPoint, it just retrieves the
entry point from executable, you can ignore it.

If you can do the same using less amount of code (with all error checks in
place) - perfect, but I'm afraid it will not be less than 500 lines. So if
you don't like - just don't do that, no one can force you to lol.

Thanks!

On Sat, Feb 23, 2019 at 3:32 AM Konstantin Belousov <[hidden email]>
wrote:

> On Fri, Feb 22, 2019 at 03:57:49PM -0800, Robert Ayrapetyan wrote:
> > Hi, thanks for a prompt reply. Here are the instructions of how to
> > reproduce (sorry for inconvenient way of specifying BP address when
> running
> > app):
> >
> > uname -a
> > FreeBSD XXX 12.0-RELEASE-p3 FreeBSD 12.0-RELEASE-p3 GENERIC  amd64
> >
> > cd /tmp
> > git clone https://github.com/rayrapetyan/ptrace_bug_poc.git
> > cd ptrace_bug_poc
> > mkdir build
> > cd build
> > cmake ..
> > make
> >
> > Run ~20 times:
> >
> > /tmp/ptrace_bug_poc/build/src/ptrace_test/ptrace_test
> > /tmp/ptrace_bug_poc/build/src/mt_example/mt_example 0x201385
> >
> > -------
> > Note: make sure 0x201385 is a call to <printf@plt> in
> > "/tmp/ptrace_bug_poc/build/src/mt_example/mt_example":
> > gdb /tmp/ptrace_bug_poc/build/src/mt_example/mt_example
> > disassemble foo
> > -------
> >
> > Wait fo appearance of:
> > "BOOM! Invalid BP hits counter (hits: 1, tid: XXXX)"
> > at the end of the output (most of the times it will be "SUCCESS")
> >
>
> ~700 lines of C++ code definitely do not fall under the 'minimal repro'
> spec.  I do not to read all of it.
>
> From looking at Debugger::Launch(), it seems that you missed the
> required debugger/child synchronization for PT_TRACE_ME. Typically child
> does
>         raise(SIGSTOP);
> immediately after PT_TRACE_ME, and the tracer must consume this signal.
> Otherwise the child continues the execution and might just execute the
> place where you intend to set a breakpoint. I may missed the sync (or it
> might be done by other means in your code), because as I said, I do not
> want to read 700 lines of C++.
>
>
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"