Un-kill-able process hang at umtxpi state with 100% cpu

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Un-kill-able process hang at umtxpi state with 100% cpu

BBlister
Hi,

On the FreeBSD multiuser student server that I administer, a student managed to created a program that takes 100% of one of the CPU cores with a process that cannot be killed.

uname -a
FreeBSD zafora 9.2-STABLE FreeBSD 9.2-STABLE #1 r264312: Thu Apr 10 15:25:33 EEST 2014     root@zafora.icte.uowm.gr:/usr/obj/usr/src/sys/zafora  amd64

The process (uses 100% cpu):
[root@zafora ~]# ps axuwww | grep  35662
ictest00301 35662 100.0  0.0  12684  1912 19- R    Tue04PM   8343:29.56 ./c1

and

[root@zafora ~]# procstat 35662
  PID  PPID  PGID   SID  TSID THR LOGIN    WCHAN     EMUL          COMM
35662 35661 35661 20142 20142   4 ictest00301 umtxpi    FreeBSD ELF64 c1

which has the parent process of a make command (I killed the parent process, but nothing happened)
ictest00301 35661   0.0  0.0   6280    320 19- I    Tue04PM      0:00.01 make execute

The specific process is blocked (cannot be killed) with the following wait channels:
[root@zafora ~]# procstat -t 35662
  PID    TID COMM             TDNAME           CPU  PRI STATE   WCHAN
35662 100196 c1               -                  2  122 sleep   umtxpi
35662 100997 c1               -                  0  122 sleep   umtxpi
35662 101757 c1               -                  1  122 sleep   uwait
35662 102464 c1               -                  2  120 run     umtxpi


Even though I have implemented resource limits in /etc/login.conf as to the CPU usage, they cannot be enforced, and the server's logfile is filled with messages:

Mar 23 13:06:46 zafora kernel: pid 35662 (c1), uid 1100301, was killed: exceeded maximum CPU limit

but nothing happens. Of course
kill -9 35662

has not effect.



How I can kill this process? If one student managed to make an un-killable process that gets 100% of the CPU with restrictions cannot be enforced, then soon the server will be filled with such processes, and a
reboot will be mandatory.


Thank you.
Reply | Threaded
Open this post in threaded view
|

Re: Un-kill-able process hang at umtxpi state with 100% cpu

BBlister

Following an advice from a user of this list,
I did a

kill -STOP 35662

in order to minimize the CPU consumption

and as soon as I hit enter the process disappeared (died). I had already killed its parent, but
it was not possible to kill this process. By sending the STOP signal it was killed.

Thanks again.
Reply | Threaded
Open this post in threaded view
|

Re: Un-kill-able process hang at umtxpi state with 100% cpu

jd1008


On 03/23/2015 12:14 PM, BBlister wrote:

> Following an advice from a user of this list,
> I did a
>
> kill -STOP 35662
>
> in order to minimize the CPU consumption
>
> and as soon as I hit enter the process disappeared (died). I had already
> killed its parent, but
> it was not possible to kill this process. By sending the STOP signal it was
> killed.
>
> Thanks again.
>
>
>
> --
> View this message in context: http://freebsd.1045724.n5.nabble.com/Un-kill-able-process-hang-at-umtxpi-state-with-100-cpu-tp5999345p5999426.html
> Sent from the freebsd-questions mailing list archive at Nabble.com.
> _______________________________________________
> [hidden email] mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-questions
> To unsubscribe, send any mail to "[hidden email]"
>
Do you know the process name?
You might have a malware!!!!
Beware!!

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Un-kill-able process hang at umtxpi state with 100% cpu

BBlister
The process named c1 in the home directory of that student, had an accompanied c1.c file that actually is one that spawn many working pthreads and synchronizes them at the end. I executed by a normal user
and nothing strange happened.

So I assume that was a rare timing issue/race condition that triggered this error.
I will have to upgrade of course to 9.3 in order to rule out old problems.

So, the takeaway of this is that sometimes when kill -KILL does not work, kill -STOP may do the job :)
Reply | Threaded
Open this post in threaded view
|

Re: Un-kill-able process hang at umtxpi state with 100% cpu

Matthew Seaman-2
In reply to this post by BBlister
On 2015/03/23 18:14, BBlister wrote:

> Following an advice from a user of this list,
> I did a
>
> kill -STOP 35662
>
> in order to minimize the CPU consumption
>
> and as soon as I hit enter the process disappeared (died). I had already
> killed its parent, but
> it was not possible to kill this process. By sending the STOP signal it was
> killed.
Are you sure?  STOP usually does exactly what it says on the tin.  It's
like Ctrl-Z in the shell.  You can in theory use 'kill -CONT pid' to
wake the process up again.

If the process died from a SIGSTOP it must be because there were other
pending signals that could somehow be delivered once the process was
stopped, but that strikes me as pretty odd.

        Cheers,

        Matthew



signature.asc (900 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Un-kill-able process hang at umtxpi state with 100% cpu

jd1008
In reply to this post by BBlister


On 03/23/2015 12:32 PM, BBlister wrote:

> The process named c1 in the home directory of that student, had an
> accompanied c1.c file that actually is one that spawn many working pthreads
> and synchronizes them at the end. I executed by a normal user
> and nothing strange happened.
>
> So I assume that was a rare timing issue/race condition that triggered this
> error.
> I will have to upgrade of course to 9.3 in order to rule out old problems.
>
> So, the takeaway of this is that sometimes when kill -KILL does not work,
> kill -STOP may do the job :)

About signal 9 (SIGKILL)

SIGKILL
    The SIGKILL signal is sent to a process to cause it to terminate
    immediately (*kill*). In contrast to SIGTERM and SIGINT, this signal
    cannot be caught or ignored, and the receiving process cannot
    perform any clean-up upon receiving this signal.

So, the only way a process cannot be killed by a signal 9 is if it is
sleeping on an uniterruptible
priority (such as when it is holding a lock on a system resource, and
sleeps waiting for an even
from low lever driver or other process (thread) (which will send a wake
signal to the sleeping process). And as soon it is awakened, it is killed
and the resource is released.

This is why that I suspect that signal STOP did not kill your process
that was not killable by SIGKILL (9).
What happened is that possibly, the process had woken and the system
finally delivered the SIGKILL to to it. I believe you had a conicidence of
sending the STOP and then you saw that the process had disappeared.
As you know a lot of time can pass  (as far as the cpu is concerned)
between doing a ps command, seeing the process still running, and then
issuing a kill command and then doing ps again to see that
the process has disappeared.

Cheers

JD
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[hidden email]"