MSR accesses that slows down the hypervisor/host

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

MSR accesses that slows down the hypervisor/host

freebsd-hackers mailing list
Hello,

There are couple AMD processor related MSRs which are being accessed in FreeBSD.

#define MSR_AMDK8_IPM           0xc0010055
#define MSR_LS_CFG      0xc0011020

We are seeing a lot of CPU time being spent in the host (Hyper-V) in handling traps when accessing these MSRs. Especially the first MRS is frequently being accessed in cpu_idle() so the performance impact to host is significant.

We noted that Linux made some code changes in the 4.10 kernel to access the first MSR much less frequently. So we are wondering if there are similar changes in FreeBSD that might be in the plan. Microsoft Hyper-V team is also planning some work to speed up the accesses to these MSRs. However any suggestions or plan to improve in the FreeBSD guest are welcome.

Thanks,
Wei
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: MSR accesses that slows down the hypervisor/host

Konstantin Belousov
On Wed, Sep 16, 2020 at 06:52:42AM +0000, Wei Hu via freebsd-hackers wrote:

> Hello,
>
> There are couple AMD processor related MSRs which are being accessed in FreeBSD.
>
> #define MSR_AMDK8_IPM           0xc0010055
> #define MSR_LS_CFG      0xc0011020
>
> We are seeing a lot of CPU time being spent in the host (Hyper-V) in handling traps when accessing these MSRs. Especially the first MRS is frequently being accessed in cpu_idle() so the performance impact to host is significant.
>
> We noted that Linux made some code changes in the 4.10 kernel to access the first MSR much less frequently. So we are wondering if there are similar changes in FreeBSD that might be in the plan. Microsoft Hyper-V team is also planning some work to speed up the accesses to these MSRs. However any suggestions or plan to improve in the FreeBSD guest are welcome.
>

Where do you see accesses to MSR_LS_CFG ?  I can only find manipulations
of that MSR in init_amd(), and then it is all under check that we are not
virtualized.

For MSR_AMDK8_IPM access in cpu_idle(), it seems that the workaround was
applied too wide. It might be that we do not need to do it on recent CPUs,
but I need to spent more time looking at datasheets to confirm/deny.

But, do you (hypervisor) indeed allow guest to initiate C1 or deeper idle
state ?  If not, perhaps as the first measure, we can avoid manipulating
MSR_AMDK8_IPM under hypervisor at all.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: MSR accesses that slows down the hypervisor/host

Andriy Gapon
On 16/09/2020 16:57, Konstantin Belousov wrote:

> On Wed, Sep 16, 2020 at 06:52:42AM +0000, Wei Hu via freebsd-hackers wrote:
>> Hello,
>>
>> There are couple AMD processor related MSRs which are being accessed in FreeBSD.
>>
>> #define MSR_AMDK8_IPM           0xc0010055
>> #define MSR_LS_CFG      0xc0011020
>>
>> We are seeing a lot of CPU time being spent in the host (Hyper-V) in handling traps when accessing these MSRs. Especially the first MRS is frequently being accessed in cpu_idle() so the performance impact to host is significant.
>>
>> We noted that Linux made some code changes in the 4.10 kernel to access the first MSR much less frequently. So we are wondering if there are similar changes in FreeBSD that might be in the plan. Microsoft Hyper-V team is also planning some work to speed up the accesses to these MSRs. However any suggestions or plan to improve in the FreeBSD guest are welcome.
>>
>
> Where do you see accesses to MSR_LS_CFG ?  I can only find manipulations
> of that MSR in init_amd(), and then it is all under check that we are not
> virtualized.
>
> For MSR_AMDK8_IPM access in cpu_idle(), it seems that the workaround was
> applied too wide. It might be that we do not need to do it on recent CPUs,
> but I need to spent more time looking at datasheets to confirm/deny.
>
> But, do you (hypervisor) indeed allow guest to initiate C1 or deeper idle
> state ?  If not, perhaps as the first measure, we can avoid manipulating
> MSR_AMDK8_IPM under hypervisor at all.

I agree with these observations and suggestions.
Additionally, I am not sure why we check and clear AMDK8_CMPHALT bit on every
call of cpu_idle().

Maybe in the past we had to deal with firmware / SMM code that aggresivekly
restored that bit despite the wishes of our OS.  I would assume that "firmware"
of virtual machines does not do that.

Having said that, echoing what Kostik said, I really doubt that any hypervisor
faithfully emulates AMDK8_CMPHALT.  I do not see any reason to do that.


--
Andriy Gapon
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

RE: MSR accesses that slows down the hypervisor/host

freebsd-hackers mailing list
In reply to this post by Konstantin Belousov
> From: Konstantin Belousov <[hidden email]>
> Sent: Wednesday, September 16, 2020 9:57 PM
> To: Wei Hu <[hidden email]>
> Cc: [hidden email]
> Subject: Re: MSR accesses that slows down the hypervisor/host
>
> Where do you see accesses to MSR_LS_CFG ?  I can only find manipulations of
> that MSR in init_amd(), and then it is all under check that we are not
> virtualized.
>
Yes, it is only accessed in init_amd() at boot time. So it is less concerned. MSR_AMDK8_IPM
is accessed in cpu_idle() all the time, so it is the key place to optimize.

> For MSR_AMDK8_IPM access in cpu_idle(), it seems that the workaround was
> applied too wide. It might be that we do not need to do it on recent CPUs, but I
> need to spent more time looking at datasheets to confirm/deny.
>
> But, do you (hypervisor) indeed allow guest to initiate C1 or deeper idle state ?
> If not, perhaps as the first measure, we can avoid manipulating
> MSR_AMDK8_IPM under hypervisor at all.

You are right a guest cannot initiate C1 or deeper idle state when running on Hyper-V.
So skipping the read of MSR_AMDK8_IPM when running under this hypervisor would
Be a viable solution.

Thanks,
Wei
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: MSR accesses that slows down the hypervisor/host

Konstantin Belousov
On Thu, Sep 17, 2020 at 05:34:22AM +0000, Wei Hu wrote:

> > From: Konstantin Belousov <[hidden email]>
> > Sent: Wednesday, September 16, 2020 9:57 PM
> > To: Wei Hu <[hidden email]>
> > Cc: [hidden email]
> > Subject: Re: MSR accesses that slows down the hypervisor/host
> >
> > Where do you see accesses to MSR_LS_CFG ?  I can only find manipulations of
> > that MSR in init_amd(), and then it is all under check that we are not
> > virtualized.
> >
> Yes, it is only accessed in init_amd() at boot time. So it is less concerned. MSR_AMDK8_IPM
> is accessed in cpu_idle() all the time, so it is the key place to optimize.
>
> > For MSR_AMDK8_IPM access in cpu_idle(), it seems that the workaround was
> > applied too wide. It might be that we do not need to do it on recent CPUs, but I
> > need to spent more time looking at datasheets to confirm/deny.
> >
> > But, do you (hypervisor) indeed allow guest to initiate C1 or deeper idle state ?
> > If not, perhaps as the first measure, we can avoid manipulating
> > MSR_AMDK8_IPM under hypervisor at all.
>
> You are right a guest cannot initiate C1 or deeper idle state when running on Hyper-V.
> So skipping the read of MSR_AMDK8_IPM when running under this hypervisor would
> Be a viable solution.

https://reviews.freebsd.org/D26470
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

RE: MSR accesses that slows down the hypervisor/host

freebsd-hackers mailing list
> > > From: Konstantin Belousov <[hidden email]>
> > > Sent: Wednesday, September 16, 2020 9:57 PM
> > > To: Wei Hu <[hidden email]>
> > > Cc: [hidden email]
> > > Subject: Re: MSR accesses that slows down the hypervisor/host
> > >
> > > Where do you see accesses to MSR_LS_CFG ?  I can only find
> > > manipulations of that MSR in init_amd(), and then it is all under
> > > check that we are not virtualized.
> > >
> > Yes, it is only accessed in init_amd() at boot time. So it is less
> > concerned. MSR_AMDK8_IPM is accessed in cpu_idle() all the time, so it is
> the key place to optimize.
> >
> > > For MSR_AMDK8_IPM access in cpu_idle(), it seems that the workaround
> > > was applied too wide. It might be that we do not need to do it on
> > > recent CPUs, but I need to spent more time looking at datasheets to
> confirm/deny.
> > >
> > > But, do you (hypervisor) indeed allow guest to initiate C1 or deeper idle
> state ?
> > > If not, perhaps as the first measure, we can avoid manipulating
> > > MSR_AMDK8_IPM under hypervisor at all.
> >
> > You are right a guest cannot initiate C1 or deeper idle state when running on
> Hyper-V.
> > So skipping the read of MSR_AMDK8_IPM when running under this
> > hypervisor would Be a viable solution.
>
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Freviews
> .freebsd.org%2FD26470&amp;data=02%7C01%7Cweh%40microsoft.com%7C8
> 7f5f82583ee428f49c108d85b38e9c3%7C72f988bf86f141af91ab2d7cd011db4
> 7%7C1%7C0%7C637359647283571535&amp;sdata=JdKxHO1sM2InD7Eo793FF
> RIpj5AQmowcc%2BLGW19dlH4%3D&amp;reserved=0

Thanks for the quick response. I will review the change and do some tests on Hyper-V.

Wei
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"