TSC calibration in virtual machines

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

TSC calibration in virtual machines

Andriy Gapon

It seems that TSC calibration in virtual machines sometimes can do more harm
than good.  Should we default to trusting the information provided by a hypervisor?

Specifically, I am observing a problem on GCE instances where calibrated TSC
frequency is about 10% lower than advertised frequency.  And apparently the
advertised frequency is the right one.

I found this thread with similar reports and a variety of workarounds from
administratively disabling the calibration to switching to a different timecounter:
https://lists.freebsd.org/pipermail/freebsd-cloud/2017-January/000080.html

--
Andriy Gapon
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: TSC calibration in virtual machines

John Baldwin
On 6/27/18 12:14 AM, Andriy Gapon wrote:

>
> It seems that TSC calibration in virtual machines sometimes can do more harm
> than good.  Should we default to trusting the information provided by a hypervisor?
>
> Specifically, I am observing a problem on GCE instances where calibrated TSC
> frequency is about 10% lower than advertised frequency.  And apparently the
> advertised frequency is the right one.
>
> I found this thread with similar reports and a variety of workarounds from
> administratively disabling the calibration to switching to a different timecounter:
> https://lists.freebsd.org/pipermail/freebsd-cloud/2017-January/000080.html

I suspect you are probably right that we should just "trust" TSC frequencies
provided by a hypervisor.  We could perhaps choose to whitelist hypervisors
known to provide accurate values if we wanted to be cautious.

--
John Baldwin
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: TSC calibration in virtual machines

Jung-uk Kim
In reply to this post by Andriy Gapon
On 06/27/2018 03:14, Andriy Gapon wrote:

>
> It seems that TSC calibration in virtual machines sometimes can do more harm
> than good.  Should we default to trusting the information provided by a hypervisor?
>
> Specifically, I am observing a problem on GCE instances where calibrated TSC
> frequency is about 10% lower than advertised frequency.  And apparently the
> advertised frequency is the right one.
>
> I found this thread with similar reports and a variety of workarounds from
> administratively disabling the calibration to switching to a different timecounter:
> https://lists.freebsd.org/pipermail/freebsd-cloud/2017-January/000080.html
We already do that for VMware hosts since r221214.

https://svnweb.freebsd.org/changeset/base/221214

We should do the same for each hypervisor.

Jung-uk Kim


signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: TSC calibration in virtual machines

Alan Somers-2
On Wed, Jun 27, 2018 at 10:36 AM, Jung-uk Kim <[hidden email]> wrote:

> On 06/27/2018 03:14, Andriy Gapon wrote:
> >
> > It seems that TSC calibration in virtual machines sometimes can do more
> harm
> > than good.  Should we default to trusting the information provided by a
> hypervisor?
> >
> > Specifically, I am observing a problem on GCE instances where calibrated
> TSC
> > frequency is about 10% lower than advertised frequency.  And apparently
> the
> > advertised frequency is the right one.
> >
> > I found this thread with similar reports and a variety of workarounds
> from
> > administratively disabling the calibration to switching to a different
> timecounter:
> > https://lists.freebsd.org/pipermail/freebsd-cloud/2017-
> January/000080.html
>
> We already do that for VMware hosts since r221214.
>
> https://svnweb.freebsd.org/changeset/base/221214
>
> We should do the same for each hypervisor.
>
> Jung-uk Kim
>
>
We probably should.  But why does calibration fail in the first place?  If
it can fail in a VM, then it can probably fail on bare metal too.  It would
be worth investigating.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: TSC calibration in virtual machines

Stephen J. Kiernan
On Wed, Jun 27, 2018, 12:48 PM Alan Somers <[hidden email]> wrote:

> On Wed, Jun 27, 2018 at 10:36 AM, Jung-uk Kim <[hidden email]> wrote:
>
> > On 06/27/2018 03:14, Andriy Gapon wrote:
> > >
> > > It seems that TSC calibration in virtual machines sometimes can do more
> > harm
> > > than good.  Should we default to trusting the information provided by a
> > hypervisor?
> > >
> > > Specifically, I am observing a problem on GCE instances where
> calibrated
> > TSC
> > > frequency is about 10% lower than advertised frequency.  And apparently
> > the
> > > advertised frequency is the right one.
> > >
> > > I found this thread with similar reports and a variety of workarounds
> > from
> > > administratively disabling the calibration to switching to a different
> > timecounter:
> > > https://lists.freebsd.org/pipermail/freebsd-cloud/2017-
> > January/000080.html
> >
> > We already do that for VMware hosts since r221214.
> >
> > https://svnweb.freebsd.org/changeset/base/221214
> >
> > We should do the same for each hypervisor.
> >
> > Jung-uk Kim
> >
> >
> We probably should.  But why does calibration fail in the first place?  If
> it can fail in a VM, then it can probably fail on bare metal too.  It would
> be worth investigating.
>

The main problem is you can't be assured that the DELAY call will be
accurate in those cases. Also the way that some VMs implement the rdtsc
insrruction may not be as accurate as we would need. In some cases it can
compound the issue.

-Steve
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: TSC calibration in virtual machines

Ryan Stone-2
In reply to this post by Alan Somers-2
I would guess that the calibration can fail because when running under
the hypervisor, the FreeBSD guest code can be descheduled at the wrong
time.  As I recall, the current algorithm looks like:

1. Sample rdtsc
2. Use a fixed-frequency timer to busy-wait for exactly 1 second
3. Sample rdtsc again
4. tsc_freq = sample2 - sample1;

If we are descheduled between 2 and 3, the time we spend off-cpu will
not be accounted for at step 4.  On bare-metal this is not possible as
neither the scheduler nor interrupts are not running yet.

Although, come to think of it, I seem to recall something about SMI
interrupts mucking this up long in the past, for exactly the same
reason.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: TSC calibration in virtual machines

Jung-uk Kim
In reply to this post by Alan Somers-2
On 06/27/2018 12:47, Alan Somers wrote:

> On Wed, Jun 27, 2018 at 10:36 AM, Jung-uk Kim <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     On 06/27/2018 03:14, Andriy Gapon wrote:
>     >
>     > It seems that TSC calibration in virtual machines sometimes can do more harm
>     > than good.  Should we default to trusting the information provided by a hypervisor?
>     >
>     > Specifically, I am observing a problem on GCE instances where calibrated TSC
>     > frequency is about 10% lower than advertised frequency.  And apparently the
>     > advertised frequency is the right one.
>     >
>     > I found this thread with similar reports and a variety of workarounds from
>     > administratively disabling the calibration to switching to a different timecounter:
>     > https://lists.freebsd.org/pipermail/freebsd-cloud/2017-January/000080.html
>     <https://lists.freebsd.org/pipermail/freebsd-cloud/2017-January/000080.html>
>
>     We already do that for VMware hosts since r221214.
>
>     https://svnweb.freebsd.org/changeset/base/221214
>     <https://svnweb.freebsd.org/changeset/base/221214>
>
>     We should do the same for each hypervisor.
>
> We probably should.  But why does calibration fail in the first place?
Because multiple guests are sharing same physical CPUs and guest OS has
no control, timing cannot be 100% accurate.

> If it can fail in a VM, then it can probably fail on bare metal too.  It
> would be worth investigating.
It does not "fail" in bare metal because we have almost complete control.

Jung-uk Kim


signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: TSC calibration in virtual machines

Rodney W. Grimes-4
In reply to this post by Alan Somers-2
> On Wed, Jun 27, 2018 at 10:36 AM, Jung-uk Kim <[hidden email]> wrote:
>
> > On 06/27/2018 03:14, Andriy Gapon wrote:
> > >
> > > It seems that TSC calibration in virtual machines sometimes can do more
> > harm
> > > than good.  Should we default to trusting the information provided by a
> > hypervisor?
> > >
> > > Specifically, I am observing a problem on GCE instances where calibrated
> > TSC
> > > frequency is about 10% lower than advertised frequency.  And apparently
> > the
> > > advertised frequency is the right one.
> > >
> > > I found this thread with similar reports and a variety of workarounds
> > from
> > > administratively disabling the calibration to switching to a different
> > timecounter:
> > > https://lists.freebsd.org/pipermail/freebsd-cloud/2017-
> > January/000080.html
> >
> > We already do that for VMware hosts since r221214.
> >
> > https://svnweb.freebsd.org/changeset/base/221214
> >
> > We should do the same for each hypervisor.
> >
> > Jung-uk Kim
> >
> >
> We probably should.  But why does calibration fail in the first place?  If
> it can fail in a VM, then it can probably fail on bare metal too.  It would
> be worth investigating.

No, the failure in a VM is unique to a VM, it has to do with the fact
your have the hypervisor timeslicing a CPU that you believe to be 100%
dedicated to you.

There are several white papers, including one from VMWare about what
they have done to help with the time keeping problems.

What is suggested above would be a correct thing to do.
Bhyve creates these issues as well, and use of certain timers
in a bhyve guest can cause you nightmares with ntp.


--
Rod Grimes                                                 [hidden email]
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: TSC calibration in virtual machines

Alan Somers-2
In reply to this post by Jung-uk Kim
On Wed, Jun 27, 2018 at 11:05 AM, Jung-uk Kim <[hidden email]> wrote:

> On 06/27/2018 12:47, Alan Somers wrote:
> > On Wed, Jun 27, 2018 at 10:36 AM, Jung-uk Kim <[hidden email]
> > <mailto:[hidden email]>> wrote:
> >
> >     On 06/27/2018 03:14, Andriy Gapon wrote:
> >     >
> >     > It seems that TSC calibration in virtual machines sometimes can do
> more harm
> >     > than good.  Should we default to trusting the information provided
> by a hypervisor?
> >     >
> >     > Specifically, I am observing a problem on GCE instances where
> calibrated TSC
> >     > frequency is about 10% lower than advertised frequency.  And
> apparently the
> >     > advertised frequency is the right one.
> >     >
> >     > I found this thread with similar reports and a variety of
> workarounds from
> >     > administratively disabling the calibration to switching to a
> different timecounter:
> >     > https://lists.freebsd.org/pipermail/freebsd-cloud/2017-
> January/000080.html
> >     <https://lists.freebsd.org/pipermail/freebsd-cloud/2017-
> January/000080.html>
> >
> >     We already do that for VMware hosts since r221214.
> >
> >     https://svnweb.freebsd.org/changeset/base/221214
> >     <https://svnweb.freebsd.org/changeset/base/221214>
> >
> >     We should do the same for each hypervisor.
> >
> > We probably should.  But why does calibration fail in the first place?
> Because multiple guests are sharing same physical CPUs and guest OS has
> no control, timing cannot be 100% accurate.
>
> > If it can fail in a VM, then it can probably fail on bare metal too.  It
> > would be worth investigating.
> It does not "fail" in bare metal because we have almost complete control.
>
> Jung-uk Kim
>
>
Makes sense.  I didn't realize that it ran before the scheduler or
interrupts were started.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: TSC calibration in virtual machines

Jung-uk Kim
In reply to this post by Ryan Stone-2
On 06/27/2018 13:01, Ryan Stone wrote:

> I would guess that the calibration can fail because when running under
> the hypervisor, the FreeBSD guest code can be descheduled at the wrong
> time.  As I recall, the current algorithm looks like:
>
> 1. Sample rdtsc
> 2. Use a fixed-frequency timer to busy-wait for exactly 1 second
> 3. Sample rdtsc again
> 4. tsc_freq = sample2 - sample1;
>
> If we are descheduled between 2 and 3, the time we spend off-cpu will
> not be accounted for at step 4.  On bare-metal this is not possible as
> neither the scheduler nor interrupts are not running yet.
>
> Although, come to think of it, I seem to recall something about SMI
> interrupts mucking this up long in the past, for exactly the same
> reason.
I think it was legacy USB device emulation for certain Intel
chipset-based motherboards.

Jung-uk Kim


signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: TSC calibration in virtual machines

Jung-uk Kim
In reply to this post by Rodney W. Grimes-4
On 06/27/2018 13:05, Rodney W. Grimes wrote:
> There are several white papers, including one from VMWare about what
> they have done to help with the time keeping problems.

https://www.vmware.com/files/pdf/techpaper/Timekeeping-In-VirtualMachines.pdf

Jung-uk Kim


signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: TSC calibration in virtual machines

Ian Lepore-3
In reply to this post by Rodney W. Grimes-4
On Wed, 2018-06-27 at 10:05 -0700, Rodney W. Grimes wrote:

> >
> > On Wed, Jun 27, 2018 at 10:36 AM, Jung-uk Kim <[hidden email]>
> > wrote:
> >
> > >
> > > On 06/27/2018 03:14, Andriy Gapon wrote:
> > > >
> > > >
> > > > It seems that TSC calibration in virtual machines sometimes can
> > > > do more
> > > harm
> > > >
> > > > than good.  Should we default to trusting the information
> > > > provided by a
> > > hypervisor?
> > > >
> > > >
> > > > Specifically, I am observing a problem on GCE instances where
> > > > calibrated
> > > TSC
> > > >
> > > > frequency is about 10% lower than advertised frequency.  And
> > > > apparently
> > > the
> > > >
> > > > advertised frequency is the right one.
> > > >
> > > > I found this thread with similar reports and a variety of
> > > > workarounds
> > > from
> > > >
> > > > administratively disabling the calibration to switching to a
> > > > different
> > > timecounter:
> > > >
> > > > https://lists.freebsd.org/pipermail/freebsd-cloud/2017-
> > > January/000080.html
> > >
> > > We already do that for VMware hosts since r221214.
> > >
> > > https://svnweb.freebsd.org/changeset/base/221214
> > >
> > > We should do the same for each hypervisor.
> > >
> > > Jung-uk Kim
> > >
> > >
> > We probably should.  But why does calibration fail in the first
> > place?  If
> > it can fail in a VM, then it can probably fail on bare metal
> > too.  It would
> > be worth investigating.
> No, the failure in a VM is unique to a VM, it has to do with the fact
> your have the hypervisor timeslicing a CPU that you believe to be
> 100%
> dedicated to you.
>
> There are several white papers, including one from VMWare about what
> they have done to help with the time keeping problems.
>
> What is suggested above would be a correct thing to do.
> Bhyve creates these issues as well, and use of certain timers
> in a bhyve guest can cause you nightmares with ntp.

Iirc, bhyve's arithmetic when doing timer emulation leads to roundoff
errors that accumulate to effectively make the emulated timer run off-
frequency. The hpet timer was trivial to fix by just redefining it to
run at a power-of-2 frequency to eliminate rounding errors. The other
timers have to run at fixed frequencies, so better arithmetic will be
the way to fix them. I vaguely remember that being harder to do than to
say because of the way the code is currently structured, which is why I
just did the easy fix to the hpet so that people would have at least
one usable timer that didn't give ntpd fits in guest OSes.

-- Ian
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"