early x86 microcode loading

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

early x86 microcode loading

Mark Johnston-2
I've been working on support for early loading of microcode updates and
wanted to solicit feedback on the approach before starting to get any
code changes reviewed.

Currently we support microcode updates via cpuctl(4), where
cpucontrol(8) passes microcode blobs to the kernel via an ioctl
interface.  Updates are distributed by the sysutils/devcpu-data port.
The scheme has a few shortcomings:
- Microcode updates may introduce new CPU features, but since we load
  microcode from userland, updates are performed well after the kernel
  has done CPU feature detection.
- Updates need to be reapplied after an ACPI suspend/resume, and there's
  currently no mechanism to automatically reapply the update after a
  resume.
- Updates aren't applied until userspace starts running, so there exists
  a window in which the kernel is running without vulnerability
  mitigations provided by microcode updates.

The aim of this work is to instead use the boot loader to load microcode
updates into kernel memory, and modify the kernel to apply the updates
as the first step in BSP and AP initialization, as well as after an ACPI
resume.  To configure an update, one would then just need to add the
following lines to loader.conf:

cpu_ucode_load="YES"
cpu_ucode_name="/boot/firmware/microcode.bin"
cpu_ucode_type="cpu_ucode"

The kernel would then automatically load the update during processor
initialization in the subsequent boot-up.

A given Intel microcode update applies only to CPUs of a specific
<family, model, stepping> tuple, while AMD releases a single update per
processor family.  My plan is to extend cpucontrol(8) to determine the
correct microcode update for the running system, and have the devcpu-data
port install the corresponding file to /boot/firmware.  The port could
then add the following to loader.conf.local:

devcpu_data_load="YES"
devcpu_data_name="/boot/firmware/<update file>"
devcpu_data_type="cpu_ucode"

Currently, the port doesn't automatically enable microcode updates; one
needs to enable the microcode_update rc script after installing the
port.  I'm not yet sure how this should be integrated with early
loading.  For example, if "service microcode_update onestart" enables
early loading, how should early loading be disabled?  Would it be
reasonable for the port to automatically enable updates when it is
installed?

For at least Intel, my intention is to support loading of multiple
update files concatenated together, and have the kernel determine the
correct update to apply.  This is to make it easier to support a cluster
of mixed systems which boot from some shared environment; rather than
needing to select and configure the correct update for each machine, one
can supply all updates as a single file and have each machine select the
right one automatically.  This would come at the expense of some wasted
memory: the combined set of Intel microcode files is currently about
1.8MB in size.

A small patch to the loader is needed to guarantee the 16 byte-alignment
of the loaded microcode update required by Intel.  This will require a
bootcode update on GPT-based systems in order for early loading to work
reliably.

I'm interested in any feedback on the above, and especially any
suggestions on how this feature should be integrated with the
devcpu-data port.  Thanks in advance.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: early x86 microcode loading

John Baldwin
On 7/12/18 11:31 AM, Mark Johnston wrote:
> I'm interested in any feedback on the above, and especially any
> suggestions on how this feature should be integrated with the
> devcpu-data port.  Thanks in advance.

In general this sounds good to me.  I'm not quite sure how to manage
setting the lines in loader.conf.  Using 'service foo start/stop'
seems to be a bit of an abuse of what service is used for as opposed
to just including a standalone script in the devcpu-data port with a
suitable pkg-message.  sysrc seems to be a closer analog than service
in theory, but shoehorning this into sysrc would seem to be quite
ugly and doesn't seem like a good idea to me either.

--
John Baldwin
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: early x86 microcode loading

Shawn Webb-3
In reply to this post by Mark Johnston-2
Hey Mark,

Thank you very much for working on this and opening the discussion.
What you've drafted sounds reasonable to me, but perhaps with one
edge case.

On Thu, Jul 12, 2018 at 02:31:16PM -0400, Mark Johnston wrote:

> I've been working on support for early loading of microcode updates and
> wanted to solicit feedback on the approach before starting to get any
> code changes reviewed.
>
> Currently we support microcode updates via cpuctl(4), where
> cpucontrol(8) passes microcode blobs to the kernel via an ioctl
> interface.  Updates are distributed by the sysutils/devcpu-data port.
> The scheme has a few shortcomings:
> - Microcode updates may introduce new CPU features, but since we load
>   microcode from userland, updates are performed well after the kernel
>   has done CPU feature detection.
> - Updates need to be reapplied after an ACPI suspend/resume, and there's
>   currently no mechanism to automatically reapply the update after a
>   resume.
> - Updates aren't applied until userspace starts running, so there exists
>   a window in which the kernel is running without vulnerability
>   mitigations provided by microcode updates.
>
> The aim of this work is to instead use the boot loader to load microcode
> updates into kernel memory, and modify the kernel to apply the updates
> as the first step in BSP and AP initialization, as well as after an ACPI
> resume.  To configure an update, one would then just need to add the
> following lines to loader.conf:
>
> cpu_ucode_load="YES"
> cpu_ucode_name="/boot/firmware/microcode.bin"
> cpu_ucode_type="cpu_ucode"
>
> The kernel would then automatically load the update during processor
> initialization in the subsequent boot-up.
>
> A given Intel microcode update applies only to CPUs of a specific
> <family, model, stepping> tuple, while AMD releases a single update per
> processor family.  My plan is to extend cpucontrol(8) to determine the
> correct microcode update for the running system, and have the devcpu-data
> port install the corresponding file to /boot/firmware.  The port could
> then add the following to loader.conf.local:
>
> devcpu_data_load="YES"
> devcpu_data_name="/boot/firmware/<update file>"
> devcpu_data_type="cpu_ucode"
I'm curious about what would happen if I moved the drives to a new
system and booted off of them, perhaps forgetting to comment out the
above lines in loader.conf beforehand.

Additionally, how would I instruct the system in such a case to
re-probe which firmware file I need?

I recognize this could be construed as an edge case, but I've done
this multiple times (and, thanks to ZFS, really easily).

Thanks,

--
Shawn Webb
Cofounder and Security Engineer
HardenedBSD

Tor-ified Signal:    +1 443-546-8752
Tor+XMPP+OTR:        [hidden email]
GPG Key ID:          0x6A84658F52456EEE
GPG Key Fingerprint: 2ABA B6BD EF6A F486 BE89  3D9E 6A84 658F 5245 6EEE

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: early x86 microcode loading

Eric van Gyzen-2
In reply to this post by John Baldwin
On 07/12/2018 14:24, John Baldwin wrote:

> On 7/12/18 11:31 AM, Mark Johnston wrote:
>> I'm interested in any feedback on the above, and especially any
>> suggestions on how this feature should be integrated with the
>> devcpu-data port.  Thanks in advance.
>
> In general this sounds good to me.  I'm not quite sure how to manage
> setting the lines in loader.conf.  Using 'service foo start/stop'
> seems to be a bit of an abuse of what service is used for as opposed
> to just including a standalone script in the devcpu-data port with a
> suitable pkg-message.  sysrc seems to be a closer analog than service
> in theory, but shoehorning this into sysrc would seem to be quite
> ugly and doesn't seem like a good idea to me either.

I hesitate to suggest this, due to the necessary change in /two/ boot
loaders, but...  The foo.d approach is very convenient for packages.  An
activation script installed by the port/package could create a new
/boot/loader.conf.d/devcpu-data file containing these lines.  A
pkg-message would instruct the user to run it.

Eric
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: early x86 microcode loading

Poul-Henning Kamp
In reply to this post by Mark Johnston-2
--------
In message <20180712183116.GB15892@raichu>, Mark Johnston writes:

>My plan is to extend cpucontrol(8) to determine the
>correct microcode update for the running system, and have the devcpu-data
>port install the corresponding file to /boot/firmware.

This is problematic when a diskimage is migrated to a different CPU,
only on the second reboot on the new hardware are you certain to
have the correct microcode.

For images which are resurrected on demand on whatever hardware is
available this really problematic.

NB: before anybody misunderstands:  This is not a problem for
guest systems under virtualization - they don't need microcode updates.

--
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[hidden email]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: early x86 microcode loading

Simon J. Gerraty
In reply to this post by Eric van Gyzen-2
Eric van Gyzen <[hidden email]> wrote:
> I hesitate to suggest this, due to the necessary change in /two/ boot
> loaders, but...  The foo.d approach is very convenient for packages.  An
> activation script installed by the port/package could create a new
> /boot/loader.conf.d/devcpu-data file containing these lines.  A
> pkg-message would instruct the user to run it.

The ability for packages to add to the boot env is indeed nice,
(we do that in Junos) but a loader.conf.d/ would be problematic for
anyone whating verification (at least using my implementation) during
boot.

Allowing subdirs such that a loader.conf snippet, the module (or
whatever) and associated signatures could all co-exist without
interference with other packages works nicely.

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: early x86 microcode loading

Mark Johnston-2
In reply to this post by Poul-Henning Kamp
On Thu, Jul 12, 2018 at 08:52:29PM +0000, Poul-Henning Kamp wrote:

> --------
> In message <20180712183116.GB15892@raichu>, Mark Johnston writes:
>
> >My plan is to extend cpucontrol(8) to determine the
> >correct microcode update for the running system, and have the devcpu-data
> >port install the corresponding file to /boot/firmware.
>
> This is problematic when a diskimage is migrated to a different CPU,
> only on the second reboot on the new hardware are you certain to
> have the correct microcode.
>
> For images which are resurrected on demand on whatever hardware is
> available this really problematic.

I can think of three ways to address this case:

1a) Always load all of the updates as a single file, and select the
    correct update during boot.  As I pointed out, this wastes some
    memory (a couple of megabytes currently).  On at least amd64 it
    doesn't look very practical to release the pages backing the
    update file back to the VM.  That is, I don't think we can easily
    "shed" the preloaded file data once the correct update has been
    selected and saved.

1b) Have the devcpu-data port operate in one of two modes: either the
    port selects the update for the current machine, as I outlined in my
    original mail, or it concatenates all of the updates as in 1a) and
    the kernel selects the correct update.  This way we'd only
    waste memory if the disk image is to be shared among multiple
    machines.  I'm not sure what the mechanism should be for selecting
    the mode.

 2) Install all updates to a directory under /boot and add code to the
    loader to perform the selection, and pass only the required microcode
    file to the kernel.  This seems straightforward to me, though I'm not
    yet sure exactly where in the loader this logic should go.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: early x86 microcode loading

Konstantin Belousov
On Thu, Jul 12, 2018 at 06:46:31PM -0400, Mark Johnston wrote:

> On Thu, Jul 12, 2018 at 08:52:29PM +0000, Poul-Henning Kamp wrote:
> > --------
> > In message <20180712183116.GB15892@raichu>, Mark Johnston writes:
> >
> > >My plan is to extend cpucontrol(8) to determine the
> > >correct microcode update for the running system, and have the devcpu-data
> > >port install the corresponding file to /boot/firmware.
> >
> > This is problematic when a diskimage is migrated to a different CPU,
> > only on the second reboot on the new hardware are you certain to
> > have the correct microcode.
> >
> > For images which are resurrected on demand on whatever hardware is
> > available this really problematic.
>
> I can think of three ways to address this case:
>
> 1a) Always load all of the updates as a single file, and select the
>     correct update during boot.  As I pointed out, this wastes some
>     memory (a couple of megabytes currently).  On at least amd64 it
>     doesn't look very practical to release the pages backing the
>     update file back to the VM.  That is, I don't think we can easily
>     "shed" the preloaded file data once the correct update has been
>     selected and saved.
>
> 1b) Have the devcpu-data port operate in one of two modes: either the
>     port selects the update for the current machine, as I outlined in my
>     original mail, or it concatenates all of the updates as in 1a) and
>     the kernel selects the correct update.  This way we'd only
>     waste memory if the disk image is to be shared among multiple
>     machines.  I'm not sure what the mechanism should be for selecting
>     the mode.
>
>  2) Install all updates to a directory under /boot and add code to the
>     loader to perform the selection, and pass only the required microcode
>     file to the kernel.  This seems straightforward to me, though I'm not
>     yet sure exactly where in the loader this logic should go.

What is the problem with having the microcode blob unmatched ? The
result would be only lack of the update for the CPU. If user cares about
having the updated microcode, he would run the required command anew.
Or you might add an automatic run of such command on shutdown.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: early x86 microcode loading

Mark Johnston-2
On Fri, Jul 13, 2018 at 03:50:54PM +0300, Konstantin Belousov wrote:

> On Thu, Jul 12, 2018 at 06:46:31PM -0400, Mark Johnston wrote:
> > On Thu, Jul 12, 2018 at 08:52:29PM +0000, Poul-Henning Kamp wrote:
> > > --------
> > > In message <20180712183116.GB15892@raichu>, Mark Johnston writes:
> > >
> > > >My plan is to extend cpucontrol(8) to determine the
> > > >correct microcode update for the running system, and have the devcpu-data
> > > >port install the corresponding file to /boot/firmware.
> > >
> > > This is problematic when a diskimage is migrated to a different CPU,
> > > only on the second reboot on the new hardware are you certain to
> > > have the correct microcode.
> > >
> > > For images which are resurrected on demand on whatever hardware is
> > > available this really problematic.

(To be clear, this case can be handled with my proposal: one would
concatenate all of the updates together and load the result, and the
kernel would select the correct update and apply it during boot.  The
issue is with the default behaviour of the devcpu-data port.)

> > I can think of three ways to address this case:
> >
> > 1a) Always load all of the updates as a single file, and select the
> >     correct update during boot.  As I pointed out, this wastes some
> >     memory (a couple of megabytes currently).  On at least amd64 it
> >     doesn't look very practical to release the pages backing the
> >     update file back to the VM.  That is, I don't think we can easily
> >     "shed" the preloaded file data once the correct update has been
> >     selected and saved.
> >
> > 1b) Have the devcpu-data port operate in one of two modes: either the
> >     port selects the update for the current machine, as I outlined in my
> >     original mail, or it concatenates all of the updates as in 1a) and
> >     the kernel selects the correct update.  This way we'd only
> >     waste memory if the disk image is to be shared among multiple
> >     machines.  I'm not sure what the mechanism should be for selecting
> >     the mode.
> >
> >  2) Install all updates to a directory under /boot and add code to the
> >     loader to perform the selection, and pass only the required microcode
> >     file to the kernel.  This seems straightforward to me, though I'm not
> >     yet sure exactly where in the loader this logic should go.
>
> What is the problem with having the microcode blob unmatched ? The
> result would be only lack of the update for the CPU. If user cares about
> having the updated microcode, he would run the required command anew.
> Or you might add an automatic run of such command on shutdown.

Given that the trend seems to be for new CPU vulnerabilities to be
mitigated by microcode updates, I think we'd want a mechanism that makes
a reasonable effort to work reliably once it is configured by the
administrator.  From this perspective, special cases which require an
extra reboot or an extra command invocation at shutdown (what if the
system panics?) are undesirable.  Perhaps we should indeed declare these
special cases as unsupported by devcpu-data, but I would prefer not to
do so if possible.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: early x86 microcode loading

Mark Johnston-2
In reply to this post by Shawn Webb-3
On Thu, Jul 12, 2018 at 03:30:15PM -0400, Shawn Webb wrote:

> Hey Mark,
>
> Thank you very much for working on this and opening the discussion.
> What you've drafted sounds reasonable to me, but perhaps with one
> edge case.
>
> On Thu, Jul 12, 2018 at 02:31:16PM -0400, Mark Johnston wrote:
> > I've been working on support for early loading of microcode updates and
> > wanted to solicit feedback on the approach before starting to get any
> > code changes reviewed.
> >
> > Currently we support microcode updates via cpuctl(4), where
> > cpucontrol(8) passes microcode blobs to the kernel via an ioctl
> > interface.  Updates are distributed by the sysutils/devcpu-data port.
> > The scheme has a few shortcomings:
> > - Microcode updates may introduce new CPU features, but since we load
> >   microcode from userland, updates are performed well after the kernel
> >   has done CPU feature detection.
> > - Updates need to be reapplied after an ACPI suspend/resume, and there's
> >   currently no mechanism to automatically reapply the update after a
> >   resume.
> > - Updates aren't applied until userspace starts running, so there exists
> >   a window in which the kernel is running without vulnerability
> >   mitigations provided by microcode updates.
> >
> > The aim of this work is to instead use the boot loader to load microcode
> > updates into kernel memory, and modify the kernel to apply the updates
> > as the first step in BSP and AP initialization, as well as after an ACPI
> > resume.  To configure an update, one would then just need to add the
> > following lines to loader.conf:
> >
> > cpu_ucode_load="YES"
> > cpu_ucode_name="/boot/firmware/microcode.bin"
> > cpu_ucode_type="cpu_ucode"
> >
> > The kernel would then automatically load the update during processor
> > initialization in the subsequent boot-up.
> >
> > A given Intel microcode update applies only to CPUs of a specific
> > <family, model, stepping> tuple, while AMD releases a single update per
> > processor family.  My plan is to extend cpucontrol(8) to determine the
> > correct microcode update for the running system, and have the devcpu-data
> > port install the corresponding file to /boot/firmware.  The port could
> > then add the following to loader.conf.local:
> >
> > devcpu_data_load="YES"
> > devcpu_data_name="/boot/firmware/<update file>"
> > devcpu_data_type="cpu_ucode"
>
> I'm curious about what would happen if I moved the drives to a new
> system and booted off of them, perhaps forgetting to comment out the
> above lines in loader.conf beforehand.
>
> Additionally, how would I instruct the system in such a case to
> re-probe which firmware file I need?
>
> I recognize this could be construed as an edge case, but I've done
> this multiple times (and, thanks to ZFS, really easily).

This is being discussed in another subthread.  Right now, an update
simply wouldn't be applied during the first boot.  My notion is that an
rc script provided by the port would automatically reconfigure the
update, so it'd be applied upon a subsequent reboot.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: early x86 microcode loading

Poul-Henning Kamp
In reply to this post by Mark Johnston-2
--------
In message <20180712224631.GF15892@raichu>, Mark Johnston writes:

>I can think of three ways to address this case:
>
>1a) Always load all of the updates as a single file, and select the
>    correct update during boot.  As I pointed out, this wastes some
>    memory (a couple of megabytes currently).  On at least amd64 it
>    doesn't look very practical to release the pages backing the
>    update file back to the VM.  That is, I don't think we can easily
>    "shed" the preloaded file data once the correct update has been
>    selected and saved.

Then we should fix that problem, rather than build an elaborate
workaround for in each and every subsystem which runs into this.

Isn't this the same issue with the splash-screen for instance ?

--
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[hidden email]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: early x86 microcode loading

Mark Johnston-2
On Fri, Jul 13, 2018 at 11:05:30PM +0000, Poul-Henning Kamp wrote:

> --------
> In message <20180712224631.GF15892@raichu>, Mark Johnston writes:
>
> >I can think of three ways to address this case:
> >
> >1a) Always load all of the updates as a single file, and select the
> >    correct update during boot.  As I pointed out, this wastes some
> >    memory (a couple of megabytes currently).  On at least amd64 it
> >    doesn't look very practical to release the pages backing the
> >    update file back to the VM.  That is, I don't think we can easily
> >    "shed" the preloaded file data once the correct update has been
> >    selected and saved.
>
> Then we should fix that problem, rather than build an elaborate
> workaround for in each and every subsystem which runs into this.

I implemented this in r336505 and plan to use it for the microcode
update file.

> Isn't this the same issue with the splash-screen for instance ?

I didn't look into the splash screen code, but it can be made to
give memory back to the system by using preload_delete_name().
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "[hidden email]"