CPU Isolation

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

CPU Isolation

Mark Saad-5
All
 I am looking to setup a 12-STABLE box in a way that is similar to a
CentOS server I am running.
That server is setup using the boot option isocpus, to remove
cores/cpus from the kernel scheduler at boot.  What I want to try on
12-STABLE is to set aside some cores to run process and some kernel
threads that will only always run on a dedicated core with out the
chance for being preempted / interrupted by something else on that
core / cpu . What I do not see is a way to  evict processes off a
running cpu post boot or a way to isolate them preboot. Am  I missing
something?

--
mark saad | [hidden email]
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: CPU Isolation

Wojciech Puchar-8
man cpuset

On Tue, 22 Jan 2019, Mark Saad wrote:

> All
> I am looking to setup a 12-STABLE box in a way that is similar to a
> CentOS server I am running.
> That server is setup using the boot option isocpus, to remove
> cores/cpus from the kernel scheduler at boot.  What I want to try on
> 12-STABLE is to set aside some cores to run process and some kernel
> threads that will only always run on a dedicated core with out the
> chance for being preempted / interrupted by something else on that
> core / cpu . What I do not see is a way to  evict processes off a
> running cpu post boot or a way to isolate them preboot. Am  I missing
> something?
>
> --
> mark saad | [hidden email]
> _______________________________________________
> [hidden email] mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
> To unsubscribe, send any mail to "[hidden email]"
>
>
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: CPU Isolation

Babak Farrokhi-7
In reply to this post by Mark Saad-5
AFAIK there is no way to do it pre-boot. To do the isolation post-boot your best bet is cpuset(1) (with which I could not totally isolate cpu cores as dedicated application cores).


On 22 Jan 2019, at 19:29, Mark Saad wrote:

> All
>  I am looking to setup a 12-STABLE box in a way that is similar to a
> CentOS server I am running.
> That server is setup using the boot option isocpus, to remove
> cores/cpus from the kernel scheduler at boot.  What I want to try on
> 12-STABLE is to set aside some cores to run process and some kernel
> threads that will only always run on a dedicated core with out the
> chance for being preempted / interrupted by something else on that
> core / cpu . What I do not see is a way to  evict processes off a
> running cpu post boot or a way to isolate them preboot. Am  I missing
> something?
>
> --
> mark saad | [hidden email]
> _______________________________________________
> [hidden email] mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
> To unsubscribe, send any mail to "[hidden email]"

signature.asc (981 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: CPU Isolation

Poul-Henning Kamp
--------
In message <[hidden email]>, "Babak Farrokhi" writes:

>AFAIK there is no way to do it pre-boot. To do the isolation post-boot your
>best bet is cpuset(1) (with which I could not totally isolate cpu cores
>as dedicated application cores).

The trick is to move /sbin/init to /sbin/init.real and create a
/sbin/init shell script which cpuset's PID=1 then exec's /sbin/init.real

Be aware that the surroundings of that shell script are very sparse.

--
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[hidden email]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: CPU Isolation

Christian Stærk
On 1/28/19 9:34 AM, Poul-Henning Kamp wrote:

> --------
> In message <[hidden email]>, "Babak Farrokhi" writes:
>
>> AFAIK there is no way to do it pre-boot. To do the isolation post-boot your
>> best bet is cpuset(1) (with which I could not totally isolate cpu cores
>> as dedicated application cores).
> The trick is to move /sbin/init to /sbin/init.real and create a
> /sbin/init shell script which cpuset's PID=1 then exec's /sbin/init.real
>
> Be aware that the surroundings of that shell script are very sparse.
>
Or maybe set "init_path" in loader.conf to point to the init wrapper and
then keep /sbin/init unchanged.

Then subsequent system updates would not break the setup.


Best regards
Christian
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: CPU Isolation

Andrey V. Elsukov
In reply to this post by Mark Saad-5
On 22.01.2019 21:29, Mark Saad wrote:

> All
>  I am looking to setup a 12-STABLE box in a way that is similar to a
> CentOS server I am running.
> That server is setup using the boot option isocpus, to remove
> cores/cpus from the kernel scheduler at boot.  What I want to try on
> 12-STABLE is to set aside some cores to run process and some kernel
> threads that will only always run on a dedicated core with out the
> chance for being preempted / interrupted by something else on that
> core / cpu . What I do not see is a way to  evict processes off a
> running cpu post boot or a way to isolate them preboot. Am  I missing
> something?
>
After r331723 it is possible to make such separation after boot using
cpuset(8). If you need this a bit early, you need to modify sbin/init.

--
WBR, Andrey V. Elsukov


signature.asc (566 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: CPU Isolation

Mark Saad-5
All
  Thanks for the replies, I had experimented with the init script
method but ; I kept thinking that I was doing this wrong. I think that
adding a loader option to
isolate cpus from the scheduler would be useful; but I have no idea
how to get there.  My current plans are to setup init to use cpus 0-4
and to use 5-7 for my applications
and see how it goes. The other test I want to try is to run some
kernel threads on dedicated cores. Let me try to break stuff now.

On Mon, Jan 28, 2019 at 7:20 AM Andrey V. Elsukov <[hidden email]> wrote:

>
> On 22.01.2019 21:29, Mark Saad wrote:
> > All
> >  I am looking to setup a 12-STABLE box in a way that is similar to a
> > CentOS server I am running.
> > That server is setup using the boot option isocpus, to remove
> > cores/cpus from the kernel scheduler at boot.  What I want to try on
> > 12-STABLE is to set aside some cores to run process and some kernel
> > threads that will only always run on a dedicated core with out the
> > chance for being preempted / interrupted by something else on that
> > core / cpu . What I do not see is a way to  evict processes off a
> > running cpu post boot or a way to isolate them preboot. Am  I missing
> > something?
> >
>
> After r331723 it is possible to make such separation after boot using
> cpuset(8). If you need this a bit early, you need to modify sbin/init.
>
> --
> WBR, Andrey V. Elsukov
>


--
mark saad | [hidden email]
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: CPU Isolation

Andrey V. Elsukov
On 28.01.2019 22:25, Mark Saad wrote:
> All
>   Thanks for the replies, I had experimented with the init script
> method but ; I kept thinking that I was doing this wrong. I think that
> adding a loader option to
> isolate cpus from the scheduler would be useful; but I have no idea
> how to get there.  My current plans are to setup init to use cpus 0-4
> and to use 5-7 for my applications
> and see how it goes. The other test I want to try is to run some
> kernel threads on dedicated cores. Let me try to break stuff now.

As r331723 says, now kernel threads belong to cpuset #2, all other
processes belong to cpuset 1. So, you can try to use commands:
Limit all userlevel application to use only CPU cores 0-4:
 # cpuset -s 1 -l 0-4
Limit kernel level threads to use CPU only CPU cores 0-4:
 # cpuset -s 2 -l 0-4
and then dedicate CPUs 5-7 for your application with pid PID
 # cpuset -l 5-7 -p PID

--
WBR, Andrey V. Elsukov


signature.asc (566 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: CPU Isolation

Andrey V. Elsukov
On 29.01.2019 00:01, Andrey V. Elsukov wrote:
> As r331723 says, now kernel threads belong to cpuset #2, all other
> processes belong to cpuset 1. So, you can try to use commands:
> Limit all userlevel application to use only CPU cores 0-4:
>  # cpuset -s 1 -l 0-4
> Limit kernel level threads to use CPU only CPU cores 0-4:
>  # cpuset -s 2 -l 0-4
> and then dedicate CPUs 5-7 for your application with pid PID
>  # cpuset -l 5-7 -p PID

Sorry, you need to create new set for your application, then you will be
able to move it to CPU cores 5-7:
  # cpuset -C -l 5-7 -p PID

--
WBR, Andrey V. Elsukov


signature.asc (566 bytes) Download Attachment