bhyve: Detecting that a guest kernel has booted

classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|

bhyve: Detecting that a guest kernel has booted

freebsd-virtualization mailing list
Hello!

Leaving aside userland monitoring tools such as Prometheus[0], is there
any way to detect on the host that a guest kernel running in bhyve has
booted? I'm assuming "booted" in this sense to mean "PID 1 has probably
started". I'm guessing that there probably isn't, but I thought I'd
better ask anyway. "Not booted" would mean something like "Is sitting
at the Grub prompt doing nothing".

[0] https://prometheus.io/

--
Mark Raynsford | http://www.io7m.com


attachment0 (235 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: bhyve: Detecting that a guest kernel has booted

Shawn Webb-3
On Mon, Mar 11, 2019 at 05:04:20PM +0000, Mark Raynsford via freebsd-virtualization wrote:
> Hello!
>
> Leaving aside userland monitoring tools such as Prometheus[0], is there
> any way to detect on the host that a guest kernel running in bhyve has
> booted? I'm assuming "booted" in this sense to mean "PID 1 has probably
> started". I'm guessing that there probably isn't, but I thought I'd
> better ask anyway. "Not booted" would mean something like "Is sitting
> at the Grub prompt doing nothing".

If your guest OS supports it, you could probably write two scripts that
uses virtio_console(4), one for the guest to tell the host "HELLO" and
one for the host to say "NICE TO SEE YOU!" once the guest's "HELLO" is
received.

Of course, replace the "HELLO" and "NICE TO SEE YOU!" with the right
logic you're looking for. ;)

The "HELLO" bit could be written as an init script.

Thanks,

--
Shawn Webb
Cofounder and Security Engineer
HardenedBSD

Tor-ified Signal:    +1 443-546-8752
Tor+XMPP+OTR:        [hidden email]
GPG Key ID:          0x6A84658F52456EEE
GPG Key Fingerprint: 2ABA B6BD EF6A F486 BE89  3D9E 6A84 658F 5245 6EEE

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: bhyve: Detecting that a guest kernel has booted

Jason Barbier-2
In reply to this post by freebsd-virtualization mailing list
None I know off the top of my head nor would that be really something easily done without some sort of agent. We could probably make some sort of inference based calls to the emulated registers but even then that wouldn't be that correct.

---
Jason Barbier | E: [hidden email]

On Mon, Mar 11, 2019, at 10:05 AM, Mark Raynsford via freebsd-virtualization wrote:

> Hello!
>
> Leaving aside userland monitoring tools such as Prometheus[0], is there
> any way to detect on the host that a guest kernel running in bhyve has
> booted? I'm assuming "booted" in this sense to mean "PID 1 has probably
> started". I'm guessing that there probably isn't, but I thought I'd
> better ask anyway. "Not booted" would mean something like "Is sitting
> at the Grub prompt doing nothing".
>
> [0] https://prometheus.io/
>
> --
> Mark Raynsford | http://www.io7m.com
>
>
> Attachments:
> * null
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: bhyve: Detecting that a guest kernel has booted

freebsd-virtualization mailing list
In reply to this post by Shawn Webb-3
On 2019-03-11T13:08:53 -0400
Shawn Webb <[hidden email]> wrote:
>
> If your guest OS supports it, you could probably write two scripts that
> uses virtio_console(4), one for the guest to tell the host "HELLO" and
> one for the host to say "NICE TO SEE YOU!" once the guest's "HELLO" is
> received.
>

They're a mix of FreeBSD, OpenBSD, and Debian guests. So I'm guessing
one out of three of those supports it...

I suppose my other option would be to add (another) NFS mount in each
guest, and have them touch a file early in the init script (and
possibly touch a different file early in the shutdown script).

--
Mark Raynsford | http://www.io7m.com


attachment0 (235 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: bhyve: Detecting that a guest kernel has booted

Shawn Webb-3
On Mon, Mar 11, 2019 at 05:21:58PM +0000, Mark Raynsford wrote:

> On 2019-03-11T13:08:53 -0400
> Shawn Webb <[hidden email]> wrote:
> >
> > If your guest OS supports it, you could probably write two scripts that
> > uses virtio_console(4), one for the guest to tell the host "HELLO" and
> > one for the host to say "NICE TO SEE YOU!" once the guest's "HELLO" is
> > received.
> >
>
> They're a mix of FreeBSD, OpenBSD, and Debian guests. So I'm guessing
> one out of three of those supports it...
>
> I suppose my other option would be to add (another) NFS mount in each
> guest, and have them touch a file early in the init script (and
> possibly touch a different file early in the shutdown script).
Both FreeBSD and Linux supports virtio_console(4). I have no idea
about OpenBSD, but I'm sure they'd be open to an implementation if
asked.

The NFS solution would work, but it would be somewhat fragile. What
happens when a VM crashes? What happens when the host crashes?

Thanks,

--
Shawn Webb
Cofounder and Security Engineer
HardenedBSD

Tor-ified Signal:    +1 443-546-8752
Tor+XMPP+OTR:        [hidden email]
GPG Key ID:          0x6A84658F52456EEE
GPG Key Fingerprint: 2ABA B6BD EF6A F486 BE89  3D9E 6A84 658F 5245 6EEE

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: bhyve: Detecting that a guest kernel has booted

freebsd-virtualization mailing list
On 2019-03-11T13:27:23 -0400
Shawn Webb <[hidden email]> wrote:
>
> Both FreeBSD and Linux supports virtio_console(4). I have no idea
> about OpenBSD, but I'm sure they'd be open to an implementation if
> asked.

Right.

>
> The NFS solution would work, but it would be somewhat fragile. What
> happens when a VM crashes? What happens when the host crashes?

At least in my case:

If the VM crashes, it'll be restarted by a process supervisor (runit,
here).

If the host crashes, I likely have bigger problems. In any case, I
think that's still fine because all the host would care about is if the
guest's file was touched more recently than the last time the host
tried to start a bhyve process for it.

I suppose I should elaborate a bit: I do have monitoring via Prometheus
in place, but I'd like to try to stagger VM startups a little as
starting up a lot of them in parallel on boot tends to overwhelm the
machine slightly. Once they're all up and running in a steady state,
things are fine. I would typically stagger the startup of
ordinary services (inside a jail, for example) by using dependencies in
runit - it has a facility to pause a service until a dependent service
has been started. Unfortunately, that can't work in this case because
once the bhyve process has been started, the host can't tell if/when
the guest has actually fully started up. As far as runit is concerned,
the service is up and so any dependent services should be started too.

--
Mark Raynsford | http://www.io7m.com


attachment0 (235 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: bhyve: Detecting that a guest kernel has booted

Rodney W. Grimes-6
In reply to this post by freebsd-virtualization mailing list
-- Start of PGP signed section.

> On 2019-03-11T13:08:53 -0400
> Shawn Webb <[hidden email]> wrote:
> >
> > If your guest OS supports it, you could probably write two scripts that
> > uses virtio_console(4), one for the guest to tell the host "HELLO" and
> > one for the host to say "NICE TO SEE YOU!" once the guest's "HELLO" is
> > received.
> >
>
> They're a mix of FreeBSD, OpenBSD, and Debian guests. So I'm guessing
> one out of three of those supports it...
>
> I suppose my other option would be to add (another) NFS mount in each
> guest, and have them touch a file early in the init script (and
> possibly touch a different file early in the shutdown script).

Well ICMP is in the kernel, and should be working as soon as the
interface is up, long before you could do anything with NFS,
so rather than the complexity above a simple ping would suffice.

There is also the phase of vmm(8) startup that when you are
running bhyveload vs bhyve and iirc grubload vs bhyve, that
can be detected.  vmbhyve does so and says you are in state
looader when you do a vm list.

--
Rod Grimes                                                 [hidden email]
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: bhyve: Detecting that a guest kernel has booted

Rodney W. Grimes-6
In reply to this post by freebsd-virtualization mailing list
> On 2019-03-11T13:27:23 -0400
> Shawn Webb <[hidden email]> wrote:
> >
> > Both FreeBSD and Linux supports virtio_console(4). I have no idea
> > about OpenBSD, but I'm sure they'd be open to an implementation if
> > asked.
>
> Right.
>
> >
> > The NFS solution would work, but it would be somewhat fragile. What
> > happens when a VM crashes? What happens when the host crashes?
>
> At least in my case:
>
> If the VM crashes, it'll be restarted by a process supervisor (runit,
> here).

I think ping would suffice to make that determination?

> If the host crashes, I likely have bigger problems. In any case, I
> think that's still fine because all the host would care about is if the
> guest's file was touched more recently than the last time the host
> tried to start a bhyve process for it.

You could inside the vm simply touch /tmp/foo from cron and from the host
see that the diskimage last modified time updated.  Assuming some cacheing
does not get in the way.

> I suppose I should elaborate a bit: I do have monitoring via Prometheus
> in place, but I'd like to try to stagger VM startups a little as
> starting up a lot of them in parallel on boot tends to overwhelm the
> machine slightly. Once they're all up and running in a steady state,
> things are fine. I would typically stagger the startup of
> ordinary services (inside a jail, for example) by using dependencies in
> runit - it has a facility to pause a service until a dependent service
> has been started. Unfortunately, that can't work in this case because
> once the bhyve process has been started, the host can't tell if/when
> the guest has actually fully started up. As far as runit is concerned,
> the service is up and so any dependent services should be started too.

The package vmbhyve has starggered startup in a specific ordered list
implemented.  It is all writtin in /bin/sh, so easy to adapt.

> Mark Raynsford | http://www.io7m.com
--
Rod Grimes                                                 [hidden email]
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: bhyve: Detecting that a guest kernel has booted

Shawn Webb-3
In reply to this post by freebsd-virtualization mailing list
On Mon, Mar 11, 2019 at 10:58:55AM -0700, Rodney W. Grimes wrote:

> -- Start of PGP signed section.
> > On 2019-03-11T13:08:53 -0400
> > Shawn Webb <[hidden email]> wrote:
> > >
> > > If your guest OS supports it, you could probably write two scripts that
> > > uses virtio_console(4), one for the guest to tell the host "HELLO" and
> > > one for the host to say "NICE TO SEE YOU!" once the guest's "HELLO" is
> > > received.
> > >
> >
> > They're a mix of FreeBSD, OpenBSD, and Debian guests. So I'm guessing
> > one out of three of those supports it...
> >
> > I suppose my other option would be to add (another) NFS mount in each
> > guest, and have them touch a file early in the init script (and
> > possibly touch a different file early in the shutdown script).
>
> Well ICMP is in the kernel, and should be working as soon as the
> interface is up, long before you could do anything with NFS,
> so rather than the complexity above a simple ping would suffice.
Just a note: Windows systems disable inbound ICMP by default, but
inbound ICMP support can be enabled post-installation.

> There is also the phase of vmm(8) startup that when you are
> running bhyveload vs bhyve and iirc grubload vs bhyve, that
> can be detected.  vmbhyve does so and says you are in state
> looader when you do a vm list.

I would suggest using bhyve with UEFI. I wish a death upon bhyveload
and grub2-bhyve.

Thanks,

--
Shawn Webb
Cofounder and Security Engineer
HardenedBSD

Tor-ified Signal:    +1 443-546-8752
Tor+XMPP+OTR:        [hidden email]
GPG Key ID:          0x6A84658F52456EEE
GPG Key Fingerprint: 2ABA B6BD EF6A F486 BE89  3D9E 6A84 658F 5245 6EEE

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: bhyve: Detecting that a guest kernel has booted

Rodney W. Grimes-6
> On Mon, Mar 11, 2019 at 10:58:55AM -0700, Rodney W. Grimes wrote:
> > -- Start of PGP signed section.
> > > On 2019-03-11T13:08:53 -0400
> > > Shawn Webb <[hidden email]> wrote:
> > > >
> > > > If your guest OS supports it, you could probably write two scripts that
> > > > uses virtio_console(4), one for the guest to tell the host "HELLO" and
> > > > one for the host to say "NICE TO SEE YOU!" once the guest's "HELLO" is
> > > > received.
> > > >
> > >
> > > They're a mix of FreeBSD, OpenBSD, and Debian guests. So I'm guessing
> > > one out of three of those supports it...
> > >
> > > I suppose my other option would be to add (another) NFS mount in each
> > > guest, and have them touch a file early in the init script (and
> > > possibly touch a different file early in the shutdown script).
> >
> > Well ICMP is in the kernel, and should be working as soon as the
> > interface is up, long before you could do anything with NFS,
> > so rather than the complexity above a simple ping would suffice.
>
> Just a note: Windows systems disable inbound ICMP by default, but
> inbound ICMP support can be enabled post-installation.
>
> > There is also the phase of vmm(8) startup that when you are
> > running bhyveload vs bhyve and iirc grubload vs bhyve, that
> > can be detected.  vmbhyve does so and says you are in state
> > looader when you do a vm list.
>
> I would suggest using bhyve with UEFI. I wish a death upon bhyveload
> and grub2-bhyve.

I have no love for them either, but until we get our UEFI updated
it is a rather sad state of affairs.  If we could a) get on a modern
version of ed2k, and b) bet the CSM fixed so that we could actually
boot bios mode stuff with it and c) maybe make a port of seabios
that could be used then we would be kicking some seriuos stuff!

> Thanks,
> Shawn Webb

--
Rod Grimes                                                 [hidden email]
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: bhyve: Detecting that a guest kernel has booted

Shawn Webb-3
In reply to this post by Shawn Webb-3
On Mon, Mar 11, 2019 at 11:09:07AM -0700, Rodney W. Grimes wrote:

> > On Mon, Mar 11, 2019 at 10:58:55AM -0700, Rodney W. Grimes wrote:
> > > -- Start of PGP signed section.
> > > > On 2019-03-11T13:08:53 -0400
> > > > Shawn Webb <[hidden email]> wrote:
> > > > >
> > > > > If your guest OS supports it, you could probably write two scripts that
> > > > > uses virtio_console(4), one for the guest to tell the host "HELLO" and
> > > > > one for the host to say "NICE TO SEE YOU!" once the guest's "HELLO" is
> > > > > received.
> > > > >
> > > >
> > > > They're a mix of FreeBSD, OpenBSD, and Debian guests. So I'm guessing
> > > > one out of three of those supports it...
> > > >
> > > > I suppose my other option would be to add (another) NFS mount in each
> > > > guest, and have them touch a file early in the init script (and
> > > > possibly touch a different file early in the shutdown script).
> > >
> > > Well ICMP is in the kernel, and should be working as soon as the
> > > interface is up, long before you could do anything with NFS,
> > > so rather than the complexity above a simple ping would suffice.
> >
> > Just a note: Windows systems disable inbound ICMP by default, but
> > inbound ICMP support can be enabled post-installation.
> >
> > > There is also the phase of vmm(8) startup that when you are
> > > running bhyveload vs bhyve and iirc grubload vs bhyve, that
> > > can be detected.  vmbhyve does so and says you are in state
> > > looader when you do a vm list.
> >
> > I would suggest using bhyve with UEFI. I wish a death upon bhyveload
> > and grub2-bhyve.
>
> I have no love for them either, but until we get our UEFI updated
> it is a rather sad state of affairs.  If we could a) get on a modern
> version of ed2k, and b) bet the CSM fixed so that we could actually
> boot bios mode stuff with it and c) maybe make a port of seabios
> that could be used then we would be kicking some seriuos stuff!
If I'm able to run an internship this year like I did last year, I
might have my interns update our UEFI firmware to the latest release
and/or make a BSDL seabios port.

My employer was recently acquired, so I'm likely not going to run an
internship this year.

Thanks,

--
Shawn Webb
Cofounder and Security Engineer
HardenedBSD

Tor-ified Signal:    +1 443-546-8752
Tor+XMPP+OTR:        [hidden email]
GPG Key ID:          0x6A84658F52456EEE
GPG Key Fingerprint: 2ABA B6BD EF6A F486 BE89  3D9E 6A84 658F 5245 6EEE

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: bhyve: Detecting that a guest kernel has booted

Patrick M. Hausen
In reply to this post by freebsd-virtualization mailing list
Hi all,

> Am 11.03.2019 um 18:04 schrieb Mark Raynsford via freebsd-virtualization <[hidden email]>:
> Leaving aside userland monitoring tools such as Prometheus[0], is there
> any way to detect on the host that a guest kernel running in bhyve has
> booted? I'm assuming "booted" in this sense to mean "PID 1 has probably
> started". I'm guessing that there probably isn't, but I thought I'd
> better ask anyway. "Not booted" would mean something like "Is sitting
> at the Grub prompt doing nothing“.

The abandoned FreeNAS Corral release did have guest additions
that served as a simple health monitor for FreeBSD and Linux
guests.

They used virtio_console(4) and the last state of the project can
be found here:
https://github.com/freenas/freenas-vm-tools

They compile cleanly on FreeBSD 12 and I even made a port that
I submitted to iX Systems, but we all know what happened to
Corral, unfortunately.

Possibly they can serve as a starting point for a new project.

And then there seems to be a newer version, again from FreeNAS,
written in Go:
https://github.com/freenas/bhyve-vm-goagent

HTH,
Patrick

--
punkt.de GmbH Internet - Dienstleistungen - Beratung
Kaiserallee 13a Tel.: 0721 9109-0 Fax: -100
76133 Karlsruhe [hidden email] http://punkt.de
AG Mannheim 108285 Gf: Juergen Egeling


signature.asc (541 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: bhyve: Detecting that a guest kernel has booted

Patrick M. Hausen
In reply to this post by Rodney W. Grimes-6
Hi all,

> Am 11.03.2019 um 19:09 schrieb Rodney W. Grimes <[hidden email]>:
> I have no love for them either, but until we get our UEFI updated
> it is a rather sad state of affairs.

I routinely boot FreeBSD, Ubuntu and Windows 10 with
UEFI, so I don’t see much that I would consider broken.

Missing persistence of boot variables (correct term?) is
the only thing that I know of - what else makes it a
sad state, currently?

Kind regards
Patrick
--
punkt.de GmbH Internet - Dienstleistungen - Beratung
Kaiserallee 13a Tel.: 0721 9109-0 Fax: -100
76133 Karlsruhe [hidden email] http://punkt.de
AG Mannheim 108285 Gf: Juergen Egeling

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: bhyve: Detecting that a guest kernel has booted

Rodney W. Grimes-6
> Hi all,
>
> > Am 11.03.2019 um 19:09 schrieb Rodney W. Grimes <[hidden email]>:
> > I have no love for them either, but until we get our UEFI updated
> > it is a rather sad state of affairs.
>
> I routinely boot FreeBSD, Ubuntu and Windows 10 with
> UEFI, so I don?t see much that I would consider broken.

All 64 bit oses, afaik we can not boot anything 32 bit
with our current uefi, which is:
This branch is 8930 commits behind tianocore:master.
And depends on an external git belonging to Peter Grehan,
who is now gone from the project.

>
> Missing persistence of boot variables (correct term?) is
> the only thing that I know of - what else makes it a
> sad state, currently?

Our GOP code frequently blows up vnc clients is another
issue, not sure if that is in the uefi stuff, or if we
just have bad vnc layered on top of it.  It throws recs
that are out of bounds.

The embeded PXE code is also rather dated.

> Patrick

--
Rod Grimes                                                 [hidden email]
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: bhyve: Detecting that a guest kernel has booted

Peter Grehan
> This branch is 8930 commits behind tianocore:master.
> And depends on an external git belonging to Peter Grehan,

  No, it isn't.

  https://github.com/freebsd/uefi-edk2

  Under FreeBSD project control since Oct 16, 2015.

later,

Peter.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: bhyve: Detecting that a guest kernel has booted

Rodney W. Grimes-6
> > This branch is 8930 commits behind tianocore:master.
> > And depends on an external git belonging to Peter Grehan,
>
>   No, it isn't.
>
>   https://github.com/freebsd/uefi-edk2
>
>   Under FreeBSD project control since Oct 16, 2015.

My appologies, the page does not clearly show who owns
the git project and I wrongly assumed it was you.

Though perhaps there is another edk2 some place
that some people seem to have copies of?

Is the source the same for the versions in
people.freebsd.org/~grehan/?


> later,
> Peter.
--
Rod Grimes                                                 [hidden email]
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: bhyve: Detecting that a guest kernel has booted

Nick Wolff-2
We actually have some people it looks like working on an updated UEFI and
have a CFT out in Updating "uefi-edk2-bhyve" thread in this mailing list.

I hope we can get bhyve-vm-goagent
<https://github.com/freenas/bhyve-vm-goagent> back in the tree and also
built for any other operating systems people need.

Thanks,

Nick Wolff


On Mon, Mar 11, 2019 at 8:21 PM Rodney W. Grimes <
[hidden email]> wrote:

> > > This branch is 8930 commits behind tianocore:master.
> > > And depends on an external git belonging to Peter Grehan,
> >
> >   No, it isn't.
> >
> >   https://github.com/freebsd/uefi-edk2
> >
> >   Under FreeBSD project control since Oct 16, 2015.
>
> My appologies, the page does not clearly show who owns
> the git project and I wrongly assumed it was you.
>
> Though perhaps there is another edk2 some place
> that some people seem to have copies of?
>
> Is the source the same for the versions in
> people.freebsd.org/~grehan/?
>
>
> > later,
> > Peter.
> --
> Rod Grimes
> [hidden email]
> _______________________________________________
> [hidden email] mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> To unsubscribe, send any mail to "
> [hidden email]"
>
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to "[hidden email]"