vfs_aio.c is still not safe

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

vfs_aio.c is still not safe

David Xu-2
Even with recently change to vfs_aio.c, the kernel AIO code is
still not safe to be used. The problem is a AIO daemon thread
may be blocked on sockets, pipe, and fifo if peer does not
transfer any data, the problem can be accumulated and all
daemon threads will be blocked if such user process increases.
I don't know who hacked socket code to support some level of
callback, it seems work, but in fact, it may only work for
first AIO request, if user queued multiple requests, same problem
will happen, I tried to workaround this problem by using
non-blocking I/O, but with current file ops, there is no such
support, I can not change O_NONBLOCK on fly because userland
and kernel have race, PR: kernel/41331 is a well explained
problem, userland will hit the race.

So possible solution could be:
1) disable AIO support for none disk file.
2) someone implement callbacks for pipe, fifo, and add
   non-blocking feature to fo_read/fo_write.

The former is simple, the later needs some effort, however
with superio kqueue, the AIO support for socket and pipe is
less important, I prefer 1) to make the AIO code usable.

David Xu




_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: vfs_aio.c is still not safe

Dag-Erling Smørgrav
David Xu <[hidden email]> writes:

> Even with recently change to vfs_aio.c, the kernel AIO code is
> still not safe to be used. The problem is a AIO daemon thread
> may be blocked on sockets, pipe, and fifo if peer does not
> transfer any data, the problem can be accumulated and all
> daemon threads will be blocked if such user process increases.
> [...]
> So possible solution could be:
> 1) disable AIO support for none disk file.
> 2) someone implement callbacks for pipe, fifo, and add
>    non-blocking feature to fo_read/fo_write.

3) Rewrite the aio code to use kthreads attached to each process, so
   problems with one process's aio does not propagate to other
   processes.

> The former is simple, the later needs some effort, however
> with superio kqueue, the AIO support for socket and pipe is
> less important, I prefer 1) to make the AIO code usable.

DES
--
Dag-Erling Smørgrav - [hidden email]
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: vfs_aio.c is still not safe

David Xu-2
Dag-Erling Smørgrav wrote:

>3) Rewrite the aio code to use kthreads attached to each process, so
>   problems with one process's aio does not propagate to other
>   processes.
>
>  
>
This is not a complete solution, it shifts the problem to another side,
allow user to kill process.

>DES
>  
>

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: vfs_aio.c is still not safe

Julian Elischer
David Xu wrote:

> Dag-Erling Smørgrav wrote:
>
>> 3) Rewrite the aio code to use kthreads attached to each process, so
>>   problems with one process's aio does not propagate to other
>>   processes.
>>
>>  
>>
> This is not a complete solution, it shifts the problem to another side,
> allow user to kill process.

A kernel thread cannot be "killed" by the user. it can only
agree to kill itself.

>
>> DES
>>  
>>
>
> _______________________________________________
> [hidden email] mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-arch
> To unsubscribe, send any mail to "[hidden email]"

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: vfs_aio.c is still not safe

David Xu-2
Julian Elischer wrote:

>
> David Xu wrote:
>
>> Dag-Erling Smørgrav wrote:
>>
>>> 3) Rewrite the aio code to use kthreads attached to each process, so
>>>   problems with one process's aio does not propagate to other
>>>   processes.
>>>
>>>  
>>>
>> This is not a complete solution, it shifts the problem to another side,
>> allow user to kill process.
>
>
> A kernel thread cannot be "killed" by the user. it can only
> agree to kill itself.
>

By attaching kthreads to each process, there still has an serious issue,
if I allow max queued AIO requests to be 1000 (sysctl can adjust this),
then I will allow 1000 aio threads to be created and be blocked for each
process, otherwise, it defeats the purpose of aio, this is why aio
thread should not be blocked on sockets, fifo, pipe, and then they can
be reused.
Using fixed number of aio threads for disk file is ok, since disk data
will be availble in foreseeable period.

If a user process got a signal which will bring it down, thread_single()
call in exit1() should cause the kthreads to exit, this can be done.
This also reminds me another problem, if all user threads exited by
calling e.g: thr_exit or kse_exit, now, these kthreads should exit too,
so the code should be adjusted,not only ptrace code.

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: vfs_aio.c is still not safe

David Xu-2
David Xu wrote:

> If a user process got a signal which will bring it down, thread_single()
> call in exit1() should cause the kthreads to exit, this can be done.
> This also reminds me another problem, if all user threads exited by
> calling e.g: thr_exit or kse_exit, now, these kthreads should exit too,
> so the code should be adjusted,not only ptrace code.
>
>

I have rechecked thr_exit(), there is a band-aid, the lastest thread can
not exit via thr_exit(), so thr_exit and kse_exit need not be worried.



_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: vfs_aio.c is still not safe

Julian Elischer
In reply to this post by David Xu-2
David Xu wrote:

> Julian Elischer wrote:
>
>>
>> David Xu wrote:
>>
>>> Dag-Erling Smørgrav wrote:
>>>
>>>> 3) Rewrite the aio code to use kthreads attached to each process, so
>>>>   problems with one process's aio does not propagate to other
>>>>   processes.
>>>>
>>>>  
>>>>
>>> This is not a complete solution, it shifts the problem to another side,
>>> allow user to kill process.
>>
>>
>>
>> A kernel thread cannot be "killed" by the user. it can only
>> agree to kill itself.
>>
>
> By attaching kthreads to each process, there still has an serious issue,
> if I allow max queued AIO requests to be 1000 (sysctl can adjust this),
> then I will allow 1000 aio threads to be created and be blocked for each
> process, otherwise, it defeats the purpose of aio, this is why aio
> thread should not be blocked on sockets, fifo, pipe, and then they can
> be reused.
> Using fixed number of aio threads for disk file is ok, since disk data
> will be availble in foreseeable period.


I was thinking that we would use the same scheme and code used for the
current AIO
just that the threads would be attached to the requesting process
instead of the
AIO process.
i.e. not just one thread per request.

>
> If a user process got a signal which will bring it down, thread_single()
> call in exit1() should cause the kthreads to exit, this can be done.
> This also reminds me another problem, if all user threads exited by
> calling e.g: thr_exit or kse_exit, now, these kthreads should exit too,
> so the code should be adjusted,not only ptrace code.

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "[hidden email]"