After 12.1->12.2 upgrade FreeBSD bhyve guest does not start anymore

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

After 12.1->12.2 upgrade FreeBSD bhyve guest does not start anymore

Andrea Venturoli
Hello.

Due to a 12.x specific bug, I run a 11.3 guest using vm-bhyve.
This is its config:

> loader="bhyveload"
> cpu=1
> memory=512M
> network0_type="virtio-net"
> network0_switch="public"
> disk0_type="virtio-blk"
> disk0_name="disk0"
> disk0_dev="sparse-zvol"
> uuid="..."
> network0_mac="..."

On monday I upgraded the host from 12.1 to 12.2 and, since then, it
won't boot automatically, but will stop at the loader prompt.

Dmesg:

> /boot/kernel/kernel text=0x1564b08 data=0x145330+0x4cdf30 syms=[0x8+0x16daf0+0x8+0x186a43]
> /boot/entropy size=0x1000
> Booting...
> Copyright (c) 1992-2019 The FreeBSD Project.
> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
>         The Regents of the University of California. All rights reserved.
> FreeBSD is a registered trademark of The FreeBSD Foundation.
> FreeBSD 11.3-RELEASE #0 r349754: Fri Jul  5 04:45:24 UTC 2019
>     [hidden email]:/usr/obj/usr/src/sys/GENERIC amd64
> FreeBSD clang version 8.0.0 (tags/RELEASE_800/final 356365) (based on LLVM 8.0.0)
> VT: init without driver.
> CPU: Intel(R) Core(TM) i5-7400 CPU @ 3.00GHz (3000.12-MHz K8-class CPU)
>   Origin="GenuineIntel"  Id=0x906e9  Family=0x6  Model=0x9e  Stepping=9
>   Features=0x9f83fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,SS,HTT,PBE>
>   Features2=0xfeda7a17<SSE3,PCLMULQDQ,DTES64,DS_CPL,SSSE3,SDBG,FMA,CX16,xTPR,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
>   AMD Features=0x24100800<SYSCALL,NX,Page1GB,LM>
>   AMD Features2=0x121<LAHF,ABM,Prefetch>
>   Structured Extended Features=0x40729<FSGSBASE,BMI1,AVX2,BMI2,ERMS,INVPCID,RDSEED>
>   Structured Extended Features3=0x400<MD_CLEAR>
>   XSAVE Features=0x1<XSAVEOPT>
>   TSC: P-state invariant
> Hypervisor: Origin = "bhyve bhyve "
> real memory  = 536870912 (512 MB)
> avail memory = 474779648 (452 MB)
> Event timer "LAPIC" quality 600
> ACPI APIC Table: <BHYVE  BVMADT  >
> ioapic0 <Version 1.1> irqs 0-31 on motherboard
> Timecounter "TSC-low" frequency 1500060031 Hz quality 1000
> random: entropy device external interface
> kbd1 at kbdmux0
> 000.000029 [4213] netmap_init               netmap: loaded module
> module_register_init: MOD_LOAD (vesa, 0xffffffff81005eb0, 0) error 19
> random: registering fast source Intel Secure Key RNG
> random: fast provider: "Intel Secure Key RNG"
> nexus0
> cryptosoft0: <software crypto> on motherboard
> acpi0: <BHYVE BVXSDT> on motherboard
> acpi0: Power Button (fixed)
> atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
> atrtc0: registered as a time-of-day clock, resolution 1.000000s
> Event timer "RTC" frequency 32768 Hz quality 0
> attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
> Timecounter "i8254" frequency 1193182 Hz quality 0
> Event timer "i8254" frequency 1193182 Hz quality 100
> hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
> Timecounter "HPET" frequency 16777216 Hz quality 950
> Event timer "HPET" frequency 16777216 Hz quality 550
> Event timer "HPET1" frequency 16777216 Hz quality 450
> Event timer "HPET2" frequency 16777216 Hz quality 450
> Event timer "HPET3" frequency 16777216 Hz quality 450
> Event timer "HPET4" frequency 16777216 Hz quality 450
> Event timer "HPET5" frequency 16777216 Hz quality 450
> Event timer "HPET6" frequency 16777216 Hz quality 450
> Event timer "HPET7" frequency 16777216 Hz quality 450
> Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
> acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
> pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
> pcib0: could not evaluate _ADR - AE_NOT_FOUND
> pci0: <ACPI PCI bus> on pcib0
> virtio_pci0: <VirtIO PCI Block adapter> port 0x2000-0x207f mem 0xc0000000-0xc0001fff irq 16 at device 4.0 on pci0
> vtblk0: <VirtIO Block Adapter> on virtio_pci0
> vtblk0: 10240MB (20971520 512 byte sectors)
> virtio_pci1: <VirtIO PCI Network adapter> port 0x2080-0x209f mem 0xc0002000-0xc0003fff irq 17 at device 5.0 on pci0
> vtnet0: <VirtIO Networking Adapter> on virtio_pci1
> vtnet0: Ethernet address: 58:9c:fc:09:e1:26
> vtnet0: netmap queues/slots: TX 1/1024, RX 1/512
> 000.000566 [ 503] vtnet_netmap_attach       vtnet attached txq=1, txd=1024 rxq=1, rxd=512
> isab0: <PCI-ISA bridge> at device 31.0 on pci0
> isa0: <ISA bus> on isab0
> atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
> atkbd0: <AT Keyboard> irq 1 on atkbdc0
> kbd0 at atkbd0
> atkbd0: [GIANT-LOCKED]
> driver bug: Unable to set devclass (class: atkbdc devname: (unknown))
> psm0: <PS/2 Mouse> irq 12 on atkbdc0
> psm0: [GIANT-LOCKED]
> psm0: model Generic PS/2 mouse, device ID 0
> uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
> uart0: console (9600,n,8,1)
> uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
> vga0: <Generic ISA VGA> at port 0x3b0-0x3bb iomem 0xb0000-0xb7fff on isa0
> ppc0: cannot reserve I/O port range
> Timecounters tick every 10.000 msec
> usb_needs_explore_all: no devclass
>
> Loader variables:
>
> Manual root filesystem specification:
>   <fstype>:<device> [options]
>       Mount <device> using filesystem <fstype>
>       and with the specified (optional) option list.
>
>     eg. ufs:/dev/da0s1a
>         zfs:tank
>         cd9660:/dev/cd0 ro
>           (which is equivalent to: mount -t cd9660 -o ro /dev/cd0 /)
>
>   ?               List valid disk boot devices
>   .               Yield 1 second (for background tasks)
>   <empty line>    Abort manual input
>
> mountroot>

Then, if I just type "ufs:/dev/vtbd0a" it will start correctly.


Can anyone explain why this happens and how to avoid it?


  bye & Thanks
        av.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: After 12.1->12.2 upgrade FreeBSD bhyve guest does not start anymore

Allan Jude-9
On 2021-01-13 07:45, Andrea Venturoli wrote:

> Hello.
>
> Due to a 12.x specific bug, I run a 11.3 guest using vm-bhyve.
> This is its config:
>> loader="bhyveload"
>> cpu=1
>> memory=512M
>> network0_type="virtio-net"
>> network0_switch="public"
>> disk0_type="virtio-blk"
>> disk0_name="disk0"
>> disk0_dev="sparse-zvol"
>> uuid="..."
>> network0_mac="..."
>
> On monday I upgraded the host from 12.1 to 12.2 and, since then, it
> won't boot automatically, but will stop at the loader prompt.
>
> Dmesg:
>> /boot/kernel/kernel text=0x1564b08 data=0x145330+0x4cdf30
>> syms=[0x8+0x16daf0+0x8+0x186a43]
>> /boot/entropy size=0x1000
>> Booting...
>> Copyright (c) 1992-2019 The FreeBSD Project.
>> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
>>         The Regents of the University of California. All rights reserved.
>> FreeBSD is a registered trademark of The FreeBSD Foundation.
>> FreeBSD 11.3-RELEASE #0 r349754: Fri Jul  5 04:45:24 UTC 2019
>>     [hidden email]:/usr/obj/usr/src/sys/GENERIC amd64
>> FreeBSD clang version 8.0.0 (tags/RELEASE_800/final 356365) (based on
>> LLVM 8.0.0)
>> VT: init without driver.
>> CPU: Intel(R) Core(TM) i5-7400 CPU @ 3.00GHz (3000.12-MHz K8-class CPU)
>>   Origin="GenuineIntel"  Id=0x906e9  Family=0x6  Model=0x9e  Stepping=9
>>  
>> Features=0x9f83fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,SS,HTT,PBE>
>>
>>  
>> Features2=0xfeda7a17<SSE3,PCLMULQDQ,DTES64,DS_CPL,SSSE3,SDBG,FMA,CX16,xTPR,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
>>
>>   AMD Features=0x24100800<SYSCALL,NX,Page1GB,LM>
>>   AMD Features2=0x121<LAHF,ABM,Prefetch>
>>   Structured Extended
>> Features=0x40729<FSGSBASE,BMI1,AVX2,BMI2,ERMS,INVPCID,RDSEED>
>>   Structured Extended Features3=0x400<MD_CLEAR>
>>   XSAVE Features=0x1<XSAVEOPT>
>>   TSC: P-state invariant
>> Hypervisor: Origin = "bhyve bhyve "
>> real memory  = 536870912 (512 MB)
>> avail memory = 474779648 (452 MB)
>> Event timer "LAPIC" quality 600
>> ACPI APIC Table: <BHYVE  BVMADT  >
>> ioapic0 <Version 1.1> irqs 0-31 on motherboard
>> Timecounter "TSC-low" frequency 1500060031 Hz quality 1000
>> random: entropy device external interface
>> kbd1 at kbdmux0
>> 000.000029 [4213] netmap_init               netmap: loaded module
>> module_register_init: MOD_LOAD (vesa, 0xffffffff81005eb0, 0) error 19
>> random: registering fast source Intel Secure Key RNG
>> random: fast provider: "Intel Secure Key RNG"
>> nexus0
>> cryptosoft0: <software crypto> on motherboard
>> acpi0: <BHYVE BVXSDT> on motherboard
>> acpi0: Power Button (fixed)
>> atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
>> atrtc0: registered as a time-of-day clock, resolution 1.000000s
>> Event timer "RTC" frequency 32768 Hz quality 0
>> attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
>> Timecounter "i8254" frequency 1193182 Hz quality 0
>> Event timer "i8254" frequency 1193182 Hz quality 100
>> hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
>> Timecounter "HPET" frequency 16777216 Hz quality 950
>> Event timer "HPET" frequency 16777216 Hz quality 550
>> Event timer "HPET1" frequency 16777216 Hz quality 450
>> Event timer "HPET2" frequency 16777216 Hz quality 450
>> Event timer "HPET3" frequency 16777216 Hz quality 450
>> Event timer "HPET4" frequency 16777216 Hz quality 450
>> Event timer "HPET5" frequency 16777216 Hz quality 450
>> Event timer "HPET6" frequency 16777216 Hz quality 450
>> Event timer "HPET7" frequency 16777216 Hz quality 450
>> Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
>> acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
>> pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
>> pcib0: could not evaluate _ADR - AE_NOT_FOUND
>> pci0: <ACPI PCI bus> on pcib0
>> virtio_pci0: <VirtIO PCI Block adapter> port 0x2000-0x207f mem
>> 0xc0000000-0xc0001fff irq 16 at device 4.0 on pci0
>> vtblk0: <VirtIO Block Adapter> on virtio_pci0
>> vtblk0: 10240MB (20971520 512 byte sectors)
>> virtio_pci1: <VirtIO PCI Network adapter> port 0x2080-0x209f mem
>> 0xc0002000-0xc0003fff irq 17 at device 5.0 on pci0
>> vtnet0: <VirtIO Networking Adapter> on virtio_pci1
>> vtnet0: Ethernet address: 58:9c:fc:09:e1:26
>> vtnet0: netmap queues/slots: TX 1/1024, RX 1/512
>> 000.000566 [ 503] vtnet_netmap_attach       vtnet attached txq=1,
>> txd=1024 rxq=1, rxd=512
>> isab0: <PCI-ISA bridge> at device 31.0 on pci0
>> isa0: <ISA bus> on isab0
>> atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
>> atkbd0: <AT Keyboard> irq 1 on atkbdc0
>> kbd0 at atkbd0
>> atkbd0: [GIANT-LOCKED]
>> driver bug: Unable to set devclass (class: atkbdc devname: (unknown))
>> psm0: <PS/2 Mouse> irq 12 on atkbdc0
>> psm0: [GIANT-LOCKED]
>> psm0: model Generic PS/2 mouse, device ID 0
>> uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
>> uart0: console (9600,n,8,1)
>> uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
>> vga0: <Generic ISA VGA> at port 0x3b0-0x3bb iomem 0xb0000-0xb7fff on isa0
>> ppc0: cannot reserve I/O port range
>> Timecounters tick every 10.000 msec
>> usb_needs_explore_all: no devclass
>>
>> Loader variables:
>>
>> Manual root filesystem specification:
>>   <fstype>:<device> [options]
>>       Mount <device> using filesystem <fstype>
>>       and with the specified (optional) option list.
>>
>>     eg. ufs:/dev/da0s1a
>>         zfs:tank
>>         cd9660:/dev/cd0 ro
>>           (which is equivalent to: mount -t cd9660 -o ro /dev/cd0 /)
>>
>>   ?               List valid disk boot devices
>>   .               Yield 1 second (for background tasks)
>>   <empty line>    Abort manual input
>>
>> mountroot>
>
> Then, if I just type "ufs:/dev/vtbd0a" it will start correctly.
>
>
> Can anyone explain why this happens and how to avoid it?
>
>
>  bye & Thanks
>     av.
> _______________________________________________
> [hidden email] mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> To unsubscribe, send any mail to
> "[hidden email]"

Do you happen to have a ZFS partition?

I think some changes to some of the bootstrap (pre-loader) code that is
shared with bhyve-load, means it always prefers ZFS over UFS (normally
for booting a real machine, you use a different bootstrap for UFS than
for ZFS, but bhyve-load is a special case.

--
Allan Jude
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: After 12.1->12.2 upgrade FreeBSD bhyve guest does not start anymore

Andrea Venturoli
On 1/13/21 7:30 PM, Allan Jude wrote:

> Do you happen to have a ZFS partition?

Not sure what you mean:
_ the host is all ZFS based;
_ the guest has only an UFS partition.



> I think some changes to some of the bootstrap (pre-loader) code that is
> shared with bhyve-load, means it always prefers ZFS over UFS (normally
> for booting a real machine, you use a different bootstrap for UFS than
> for ZFS, but bhyve-load is a special case.

I'm not following you here.
Could you provide some pointers that explain this more in depth?



  bye & Thanks
        av.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: After 12.1->12.2 upgrade FreeBSD bhyve guest does not start anymore

Jason Tubnor
On Thu, 14 Jan 2021 at 05:50, Andrea Venturoli <[hidden email]> wrote:

>
>
> > I think some changes to some of the bootstrap (pre-loader) code that is
> > shared with bhyve-load, means it always prefers ZFS over UFS (normally
> > for booting a real machine, you use a different bootstrap for UFS than
> > for ZFS, but bhyve-load is a special case.
>
> I'm not following you here.
> Could you provide some pointers that explain this more in depth?
>

Is your guest running within a zvol? What is the volmode parameter set to
on that zvol?

Cheers,

Jason
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: After 12.1->12.2 upgrade FreeBSD bhyve guest does not start anymore

Andrea Venturoli
On 1/14/21 5:47 AM, Jason Tubnor wrote:

> Is your guest running within a zvol?

Yes!
I've got disk0_dev="sparse-zvol" in my vm-bhyve config file.



> What is the volmode parameter set  to on that zvol?

# zfs get volmode zroot/vm/f11b/disk0
NAME                 PROPERTY  VALUE    SOURCE
zroot/vm/f11b/disk0  volmode   default  default

# sysctl vfs.zfs.vol.mode
vfs.zfs.vol.mode: 1

I.e. "geom"

Should I change that to "dev"?

While I know I can change only one volume at a time, how would this
impact Windows VMs?

  bye & Thanks
        av.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: After 12.1->12.2 upgrade FreeBSD bhyve guest does not start anymore

Jason Tubnor
On Thu, 14 Jan 2021 at 18:52, Andrea Venturoli <[hidden email]> wrote:

>
>
> # sysctl vfs.zfs.vol.mode
> vfs.zfs.vol.mode: 1
>
> I.e. "geom"
>
> Should I change that to "dev"?
>
> While I know I can change only one volume at a time, how would this
> impact Windows VMs?


We clearly define volmode=dev on the zvol for all guests to prevent the
host geom activities taking hold of the volume.  Setting it to 'dev' will
make the host ignore it and won't cause issues when bhyve goes to use it.

This should fix the problem you are having.  Let us know how you get on.

Cheers,

Jason.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: After 12.1->12.2 upgrade FreeBSD bhyve guest does not start anymore

Andrea Venturoli
On 1/14/21 10:04 PM, Jason Tubnor wrote:

> We clearly define volmode=dev on the zvol for all guests to prevent the
> host geom activities taking hold of the volume.  Setting it to 'dev'
> will make the host ignore it and won't cause issues when bhyve goes to
> use it.
>
> This should fix the problem you are having.  Let us know how you get on.

I tried this, but it did not help.
I still need to connect to the guest via (virtual) serial line and
manually specify the root device in order to boot it.

  bye & thanks
        av.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to "[hidden email]"