"cpuset -n prefer:?" --what values for "?" are supposed to be allowed? (only 1 is, despite two numa domains)

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

"cpuset -n prefer:?" --what values for "?" are supposed to be allowed? (only 1 is, despite two numa domains)

Mark Millard-4
In a context with:

# cpuset -g
pid -1 mask: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27
pid -1 domain policy: first-touch mask: 0, 1

I get:

# cpuset -l0 -n prefer:0 COMMAND
cpuset: setdomain: Invalid argument

# cpuset -l0 -n prefer:2 COMMAND
cpuset: setdomain: Invalid argument

But one prefer:? value does allow the COMMAND
to run:

# cpuset -l0 -n prefer:1 COMMAND

This seem odd to me. Am I missing something?

For reference: I'm using a ThreadRipper 1950X
with a head -r351227 based context for this
activity. The above happens to have been run
in a Windows 10 Pro HyperV session, instead
of in a native-boot of the same media. (A
native-boot would have had 32 CPUs.)


===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: "cpuset -n prefer:?" --what values for "?" are supposed to be allowed? (only 1 is, despite two numa domains)

Mark Johnston-2
On Tue, Sep 10, 2019 at 10:58:05PM -0700, Mark Millard wrote:

> In a context with:
>
> # cpuset -g
> pid -1 mask: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27
> pid -1 domain policy: first-touch mask: 0, 1
>
> I get:
>
> # cpuset -l0 -n prefer:0 COMMAND
> cpuset: setdomain: Invalid argument
>
> # cpuset -l0 -n prefer:2 COMMAND
> cpuset: setdomain: Invalid argument
>
> But one prefer:? value does allow the COMMAND
> to run:
>
> # cpuset -l0 -n prefer:1 COMMAND
>
> This seem odd to me. Am I missing something?
>
> For reference: I'm using a ThreadRipper 1950X
> with a head -r351227 based context for this
> activity. The above happens to have been run
> in a Windows 10 Pro HyperV session, instead
> of in a native-boot of the same media. (A
> native-boot would have had 32 CPUs.)

Can you please show the output of "sysctl vm.phys_segs" from this
setup?
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: "cpuset -n prefer:?" --what values for "?" are supposed to be allowed? (only 1 is, despite two numa domains)

Mark Millard-4


On 2019-Sep-11, at 07:31, Mark Johnston <markj at freebsd.org> wrote:

> On Tue, Sep 10, 2019 at 10:58:05PM -0700, Mark Millard wrote:
>> In a context with:
>>
>> # cpuset -g
>> pid -1 mask: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27
>> pid -1 domain policy: first-touch mask: 0, 1
>>
>> I get:
>>
>> # cpuset -l0 -n prefer:0 COMMAND
>> cpuset: setdomain: Invalid argument
>>
>> # cpuset -l0 -n prefer:2 COMMAND
>> cpuset: setdomain: Invalid argument
>>
>> But one prefer:? value does allow the COMMAND
>> to run:
>>
>> # cpuset -l0 -n prefer:1 COMMAND
>>
>> This seem odd to me. Am I missing something?
>>
>> For reference: I'm using a ThreadRipper 1950X
>> with a head -r351227 based context for this
>> activity. The above happens to have been run
>> in a Windows 10 Pro HyperV session, instead
>> of in a native-boot of the same media. (A
>> native-boot would have had 32 CPUs.)
>
> Can you please show the output of "sysctl vm.phys_segs" from this
> setup?

Sure:

# sysctl vm.phys_segs
vm.phys_segs:
SEGMENT 0:

start:     0x1000
end:       0x9f000
domain:    0
free list: 0xffffffff8281daa0

SEGMENT 1:

start:     0x103000
end:       0x1000000
domain:    0
free list: 0xffffffff8281daa0

SEGMENT 2:

start:     0x1000000
end:       0x2ee1000
domain:    0
free list: 0xffffffff8281d830

SEGMENT 3:

start:     0x2eea000
end:       0x2f23000
domain:    0
free list: 0xffffffff8281d830

SEGMENT 4:

start:     0x3000000
end:       0xf7ff0000
domain:    0
free list: 0xffffffff8281d830

SEGMENT 5:

start:     0x100002000
end:       0x9c5562000
domain:    0
free list: 0xffffffff8281d5c0

SEGMENT 6:

start:     0xa07c00000
end:       0xa07d50000
domain:    0
free list: 0xffffffff8281d5c0

SEGMENT 7:

start:     0xa08001000
end:       0xf9ee00000
domain:    1
free list: 0xffffffff8281dd10

SEGMENT 8:

start:     0x1000000000
end:       0x1427fe6000
domain:    1
free list: 0xffffffff8281dd10


And confirming the oddity is still the case
(I'd rebooted since the report):

# cpuset -g
pid -1 mask: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27
pid -1 domain policy: first-touch mask: 0, 1

# cpuset -l0 -n prefer:0 COMMAND
cpuset: setdomain: Invalid argument

# cpuset -l0 -n prefer:2 COMMAND
cpuset: setdomain: Invalid argument

# cpuset -l0 -n prefer:1 COMMAND
cpuset: COMMAND: No such file or directory


===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: "cpuset -n prefer:?" --what values for "?" are supposed to be allowed? (only 1 is, despite two numa domains)

Mark Johnston-2
On Wed, Sep 11, 2019 at 07:57:26AM -0700, Mark Millard wrote:

>
>
> On 2019-Sep-11, at 07:31, Mark Johnston <markj at freebsd.org> wrote:
>
> > On Tue, Sep 10, 2019 at 10:58:05PM -0700, Mark Millard wrote:
> >> In a context with:
> >>
> >> # cpuset -g
> >> pid -1 mask: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27
> >> pid -1 domain policy: first-touch mask: 0, 1
> >>
> >> I get:
> >>
> >> # cpuset -l0 -n prefer:0 COMMAND
> >> cpuset: setdomain: Invalid argument
> >>
> >> # cpuset -l0 -n prefer:2 COMMAND
> >> cpuset: setdomain: Invalid argument
> >>
> >> But one prefer:? value does allow the COMMAND
> >> to run:
> >>
> >> # cpuset -l0 -n prefer:1 COMMAND
> >>
> >> This seem odd to me. Am I missing something?
> >>
> >> For reference: I'm using a ThreadRipper 1950X
> >> with a head -r351227 based context for this
> >> activity. The above happens to have been run
> >> in a Windows 10 Pro HyperV session, instead
> >> of in a native-boot of the same media. (A
> >> native-boot would have had 32 CPUs.)
> >
> > Can you please show the output of "sysctl vm.phys_segs" from this
> > setup?
>
> Sure:

I was wondering if you had only one domain populated, but it seems not
to be the case.  Could you try updating to r351672 or later and see if
the behaviour persists?
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: "cpuset -n prefer:?" --what values for "?" are supposed to be allowed? (only 1 is, despite two numa domains)

Mark Millard-4


On 2019-Sep-11, at 08:15, Mark Johnston <markj at freebsd.org> wrote:

> On Wed, Sep 11, 2019 at 07:57:26AM -0700, Mark Millard wrote:
>>
>>
>> On 2019-Sep-11, at 07:31, Mark Johnston <markj at freebsd.org> wrote:
>>
>>> On Tue, Sep 10, 2019 at 10:58:05PM -0700, Mark Millard wrote:
>>>> In a context with:
>>>>
>>>> # cpuset -g
>>>> pid -1 mask: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27
>>>> pid -1 domain policy: first-touch mask: 0, 1
>>>>
>>>> I get:
>>>>
>>>> # cpuset -l0 -n prefer:0 COMMAND
>>>> cpuset: setdomain: Invalid argument
>>>>
>>>> # cpuset -l0 -n prefer:2 COMMAND
>>>> cpuset: setdomain: Invalid argument
>>>>
>>>> But one prefer:? value does allow the COMMAND
>>>> to run:
>>>>
>>>> # cpuset -l0 -n prefer:1 COMMAND
>>>>
>>>> This seem odd to me. Am I missing something?
>>>>
>>>> For reference: I'm using a ThreadRipper 1950X
>>>> with a head -r351227 based context for this
>>>> activity. The above happens to have been run
>>>> in a Windows 10 Pro HyperV session, instead
>>>> of in a native-boot of the same media. (A
>>>> native-boot would have had 32 CPUs.)
>>>
>>> Can you please show the output of "sysctl vm.phys_segs" from this
>>> setup?
>>
>> Sure:
>
> I was wondering if you had only one domain populated, but it seems not
> to be the case.  Could you try updating to r351672 or later and see if
> the behaviour persists?

It may be a bit before I do that.

FYI: I had set MAXMEMDOM to match the number of
actual domains for the context:

/usr/src/sys/amd64/conf/GENERIC-DBG:options     MAXMEMDOM=2
/usr/src/sys/amd64/conf/GENERIC-NODBG:options   MAXMEMDOM=2

(These kernel configuration files include GENERIC.)

===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: "cpuset -n prefer:?" --what values for "?" are supposed to be allowed? (only 1 is, despite two numa domains)

Mark Millard-4


On 2019-Sep-11, at 10:11, Mark Millard <marklmi at yahoo.com> wrote:



> On 2019-Sep-11, at 08:15, Mark Johnston <markj at freebsd.org> wrote:
>
>> On Wed, Sep 11, 2019 at 07:57:26AM -0700, Mark Millard wrote:
>>>
>>>
>>> On 2019-Sep-11, at 07:31, Mark Johnston <markj at freebsd.org> wrote:
>>>
>>>> On Tue, Sep 10, 2019 at 10:58:05PM -0700, Mark Millard wrote:
>>>>> In a context with:
>>>>>
>>>>> # cpuset -g
>>>>> pid -1 mask: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27
>>>>> pid -1 domain policy: first-touch mask: 0, 1
>>>>>
>>>>> I get:
>>>>>
>>>>> # cpuset -l0 -n prefer:0 COMMAND
>>>>> cpuset: setdomain: Invalid argument
>>>>>
>>>>> # cpuset -l0 -n prefer:2 COMMAND
>>>>> cpuset: setdomain: Invalid argument
>>>>>
>>>>> But one prefer:? value does allow the COMMAND
>>>>> to run:
>>>>>
>>>>> # cpuset -l0 -n prefer:1 COMMAND
>>>>>
>>>>> This seem odd to me. Am I missing something?
>>>>>
>>>>> For reference: I'm using a ThreadRipper 1950X
>>>>> with a head -r351227 based context for this
>>>>> activity. The above happens to have been run
>>>>> in a Windows 10 Pro HyperV session, instead
>>>>> of in a native-boot of the same media. (A
>>>>> native-boot would have had 32 CPUs.)
>>>>
>>>> Can you please show the output of "sysctl vm.phys_segs" from this
>>>> setup?
>>>
>>> Sure:
>>
>> I was wondering if you had only one domain populated, but it seems not
>> to be the case.  Could you try updating to r351672 or later and see if
>> the behaviour persists?
>
> It may be a bit before I do that.
>
> FYI: I had set MAXMEMDOM to match the number of
> actual domains for the context:
>
> /usr/src/sys/amd64/conf/GENERIC-DBG:options     MAXMEMDOM=2
> /usr/src/sys/amd64/conf/GENERIC-NODBG:options   MAXMEMDOM=2
>
> (These kernel configuration files include GENERIC.)

Not that the below is the problem that I reported, but
cpuset_modify_domain has an oddity. In the below, note
the "root->" use followed by the "root &&" test: the
root-> use would have failed first. Should the && be
"dset &&" instead? Should "root &&" just be removed for
being redundant?

793                root = cpuset_getroot(set);
794                mtx_lock_spin(&cpuset_lock);
795                dset = root->cs_domain;
796                /*
797                 * Verify that we have access to this set of domains.
798                 */
799                if (root && !domainset_valid(dset, domain)) {
800                        error = EINVAL;
801                        goto out;
802                }

===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: "cpuset -n prefer:?" --what values for "?" are supposed to be allowed? (only 1 is, despite two numa domains)

Mark Johnston-2
On Wed, Sep 11, 2019 at 11:14:42AM -0700, Mark Millard wrote:

>
>
> On 2019-Sep-11, at 10:11, Mark Millard <marklmi at yahoo.com> wrote:
>
>
>
> > On 2019-Sep-11, at 08:15, Mark Johnston <markj at freebsd.org> wrote:
> >
> >> On Wed, Sep 11, 2019 at 07:57:26AM -0700, Mark Millard wrote:
> >>>
> >>>
> >>> On 2019-Sep-11, at 07:31, Mark Johnston <markj at freebsd.org> wrote:
> >>>
> >>>> On Tue, Sep 10, 2019 at 10:58:05PM -0700, Mark Millard wrote:
> >>>>> In a context with:
> >>>>>
> >>>>> # cpuset -g
> >>>>> pid -1 mask: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27
> >>>>> pid -1 domain policy: first-touch mask: 0, 1
> >>>>>
> >>>>> I get:
> >>>>>
> >>>>> # cpuset -l0 -n prefer:0 COMMAND
> >>>>> cpuset: setdomain: Invalid argument
> >>>>>
> >>>>> # cpuset -l0 -n prefer:2 COMMAND
> >>>>> cpuset: setdomain: Invalid argument
> >>>>>
> >>>>> But one prefer:? value does allow the COMMAND
> >>>>> to run:
> >>>>>
> >>>>> # cpuset -l0 -n prefer:1 COMMAND
> >>>>>
> >>>>> This seem odd to me. Am I missing something?
> >>>>>
> >>>>> For reference: I'm using a ThreadRipper 1950X
> >>>>> with a head -r351227 based context for this
> >>>>> activity. The above happens to have been run
> >>>>> in a Windows 10 Pro HyperV session, instead
> >>>>> of in a native-boot of the same media. (A
> >>>>> native-boot would have had 32 CPUs.)
> >>>>
> >>>> Can you please show the output of "sysctl vm.phys_segs" from this
> >>>> setup?
> >>>
> >>> Sure:
> >>
> >> I was wondering if you had only one domain populated, but it seems not
> >> to be the case.  Could you try updating to r351672 or later and see if
> >> the behaviour persists?
> >
> > It may be a bit before I do that.
> >
> > FYI: I had set MAXMEMDOM to match the number of
> > actual domains for the context:
> >
> > /usr/src/sys/amd64/conf/GENERIC-DBG:options     MAXMEMDOM=2
> > /usr/src/sys/amd64/conf/GENERIC-NODBG:options   MAXMEMDOM=2
> >
> > (These kernel configuration files include GENERIC.)

Ok, that helps.  I believe you are hitting a bug that will be fixed by
r351672 and a couple of preceding commits to the same area.

> Not that the below is the problem that I reported, but
> cpuset_modify_domain has an oddity. In the below, note
> the "root->" use followed by the "root &&" test: the
> root-> use would have failed first. Should the && be
> "dset &&" instead? Should "root &&" just be removed for
> being redundant?

Good catch.  I believe cpusets are not allowed to have a NULL domain
set, so dset should never be NULL either.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"