NVMe and Bhyve

classic Classic list List threaded Threaded
24 messages Options
12
Reply | Threaded
Open this post in threaded view
|

NVMe and Bhyve

Michael Reifenberger
Hi,
first I tried to install windows10 ltsc 2019 onto a nvme disk.
This failed, the windows installer did not find a disk to install.

Then I tried the following setup:
...
disk0_type="ahci-hd"
disk0_dev="zvol"
disk0_name="disk0"
disk1_type="nvme"
disk1_dev="zvol"
disk1_name="disk1"
disk1_type="ahci-cd"
disk1_name="disk2.img"
...

And Installed windows on disk0.
Disk0 is found as expected, disk1 is not.

The devicemanager reports an error:
https://imgur.com/a/zrHx23y

Shouldn't nvme be supported in behyve?

Greetings
---
mike

Gruß
---
Michael Reifenberger

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: NVMe and Bhyve

yuripv
Michael Reifenberger wrote:

> Hi,
> first I tried to install windows10 ltsc 2019 onto a nvme disk.
> This failed, the windows installer did not find a disk to install.
>
> Then I tried the following setup:
> ...
> disk0_type="ahci-hd"
> disk0_dev="zvol"
> disk0_name="disk0"
> disk1_type="nvme"
> disk1_dev="zvol"
> disk1_name="disk1"
> disk1_type="ahci-cd"
> disk1_name="disk2.img"

If this is exact copy/paste, shouldn't the 2 lines above use disk2
prefix instead?

> ...
>
> And Installed windows on disk0.
> Disk0 is found as expected, disk1 is not.
>
> The devicemanager reports an error:
> https://imgur.com/a/zrHx23y
>
> Shouldn't nvme be supported in behyve?
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

RE: NVMe and Bhyve

freebsd-virtualization mailing list
Michael Reifenberger wrote:

> Hi,
> first I tried to install windows10 ltsc 2019 onto a nvme disk.
> This failed, the windows installer did not find a disk to install.
>
> Then I tried the following setup:
> ...
> disk0_type="ahci-hd"
> disk0_dev="zvol"
> disk0_name="disk0"
> disk1_type="nvme"
> disk1_dev="zvol"
> disk1_name="disk1"
> disk1_type="ahci-cd"
> disk1_name="disk2.img"

>If this is exact copy/paste, shouldn't the 2 lines above use disk2 prefix instead?

There's definitely an issue with the configuration file, but vm-bhyve retrieves config settings by looking through the file (which is loaded into a variable) line by line and returning the first match. As such, it will likely be ignoring the ahci-cd & disk2.img settings. Strange why the last disk is a CD and pointing at a .img file though.

More useful would be the actual bhyve command from /path/to/guest/vm-bhyve.log, and maybe also confirmation that the zvol path that appears in that command line correctly points to an existing zvol.

Matt

> ...
>
> And Installed windows on disk0.
> Disk0 is found as expected, disk1 is not.
>
> The devicemanager reports an error:
> https://imgur.com/a/zrHx23y
>
> Shouldn't nvme be supported in behyve?
_______________________________________________
[hidden email] mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to "[hidden email]"
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: NVMe and Bhyve

mike-6
In reply to this post by yuripv
Hi,
yes its a typo but not relevant for the mentioned problem.
It just leads to not adding disk2.

Disk0 and the nvme get added to bhyve.
Only windows is unwilling to use nvme.

Greetings
...
Mike


On February 7, 2019 9:08:53 PM GMT+01:00, Yuri Pankov <[hidden email]> wrote:

>Michael Reifenberger wrote:
>> Hi,
>> first I tried to install windows10 ltsc 2019 onto a nvme disk.
>> This failed, the windows installer did not find a disk to install.
>>
>> Then I tried the following setup:
>> ...
>> disk0_type="ahci-hd"
>> disk0_dev="zvol"
>> disk0_name="disk0"
>> disk1_type="nvme"
>> disk1_dev="zvol"
>> disk1_name="disk1"
>> disk1_type="ahci-cd"
>> disk1_name="disk2.img"
>
>If this is exact copy/paste, shouldn't the 2 lines above use disk2
>prefix instead?
>
>> ...
>>
>> And Installed windows on disk0.
>> Disk0 is found as expected, disk1 is not.
>>
>> The devicemanager reports an error:
>> https://imgur.com/a/zrHx23y
>>
>> Shouldn't nvme be supported in behyve?
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: NVMe and Bhyve

Victor Sudakov-3
In reply to this post by yuripv
What's that disk0_type="nvme" thing?

I'm curious. In 11.2-RELEASE there is nothing about nvme in bhyve(8).
Should I be running CURRENT or what?

--
Victor Sudakov,  VAS4-RIPE, VAS47-RIPN
2:5005/49@fidonet http://vas.tomsk.ru/

signature.asc (465 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: NVMe and Bhyve

Jason Tubnor
On Tue, 12 Feb 2019 at 19:43, Victor Sudakov <[hidden email]> wrote:

> What's that disk0_type="nvme" thing?
>
> I'm curious. In 11.2-RELEASE there is nothing about nvme in bhyve(8).
> Should I be running CURRENT or what?
>
>
>
It is valid on hosts 12.0-RELEASE and newer.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: NVMe and Bhyve

Victor Sudakov-3
Jason Tubnor wrote:
>
> > What's that disk0_type="nvme" thing?
> >
> > I'm curious. In 11.2-RELEASE there is nothing about nvme in bhyve(8).
> > Should I be running CURRENT or what?
> >
> >
> >
> It is valid on hosts 12.0-RELEASE and newer.

Is it worth it? I mean upgrading from 11.2-RELEASE to 12.0-RELEASE for
the sake of nvme, is it useful for running FreeBSD, Linux (Mint,
Centos) and Windows10 guests?

--
Victor Sudakov,  VAS4-RIPE, VAS47-RIPN
2:5005/49@fidonet http://vas.tomsk.ru/

signature.asc (465 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: NVMe and Bhyve

Marcelo Araujo-7
If you have Nvme devices or you want store disks in ram yes, otherwise no.

BR,

On Tue, Feb 12, 2019, 9:05 PM Victor Sudakov <[hidden email] wrote:

> Jason Tubnor wrote:
> >
> > > What's that disk0_type="nvme" thing?
> > >
> > > I'm curious. In 11.2-RELEASE there is nothing about nvme in bhyve(8).
> > > Should I be running CURRENT or what?
> > >
> > >
> > >
> > It is valid on hosts 12.0-RELEASE and newer.
>
> Is it worth it? I mean upgrading from 11.2-RELEASE to 12.0-RELEASE for
> the sake of nvme, is it useful for running FreeBSD, Linux (Mint,
> Centos) and Windows10 guests?
>
> --
> Victor Sudakov,  VAS4-RIPE, VAS47-RIPN
> 2:5005/49@fidonet http://vas.tomsk.ru/
>
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: NVMe and Bhyve

Jason Tubnor
On Wed, 13 Feb 2019 at 00:57, Marcelo Araujo <[hidden email]>
wrote:

> If you have Nvme devices or you want store disks in ram yes, otherwise no.
>
> >
> > Is it worth it? I mean upgrading from 11.2-RELEASE to 12.0-RELEASE for
> > the sake of nvme, is it useful for running FreeBSD, Linux (Mint,
> > Centos) and Windows10 guests?
> >


Or if you are using good SLOG and ARC for your VM zvols.

I'm still sorting out some Windows issues with the bhyve nvme presentation
for a variety of guest (except freebsd, that works fine) at the moment.  I
would avoid 12.0-RELEASE as it may introduce other issues with your
environment.  I am staying on the 11.x branch until 12.1 is released so no
nvme except for in the test environment that is running 12-STABLE.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: NVMe and Bhyve

freebsd-virtualization mailing list
On Wed, Feb 13, 2019 at 07:50:32AM +1100, Jason Tubnor wrote:

> On Wed, 13 Feb 2019 at 00:57, Marcelo Araujo <[hidden email]>
> wrote:
>
> > If you have Nvme devices or you want store disks in ram yes, otherwise no.
> >
> > >
> > > Is it worth it? I mean upgrading from 11.2-RELEASE to 12.0-RELEASE for
> > > the sake of nvme, is it useful for running FreeBSD, Linux (Mint,
> > > Centos) and Windows10 guests?
> > >
>
>
> Or if you are using good SLOG and ARC for your VM zvols.
>
> I'm still sorting out some Windows issues with the bhyve nvme presentation
> for a variety of guest (except freebsd, that works fine) at the moment.  I
> would avoid 12.0-RELEASE as it may introduce other issues with your
> environment.  I am staying on the 11.x branch until 12.1 is released so no
> nvme except for in the test environment that is running 12-STABLE.

I will stay with 11.x until 12.2 is released.

> _______________________________________________
> [hidden email] mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> To unsubscribe, send any mail to "[hidden email]"

--
Member - Liberal International This is doctor@@nl2k.ab.ca Ici doctor@@nl2k.ab.ca
Yahweh, Queen & country!Never Satan President Republic!Beware AntiChrist rising!
https://www.empire.kred/ROOTNK?t=94a1f39b  Look at Psalms 14 and 53 on Atheism
Do not consider painful what is good for you.  -Euripides
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: NVMe and Bhyve

Victor Sudakov-3
In reply to this post by Jason Tubnor
Jason Tubnor wrote:

>
> > If you have Nvme devices or you want store disks in ram yes, otherwise no.
> >
> > >
> > > Is it worth it? I mean upgrading from 11.2-RELEASE to 12.0-RELEASE for
> > > the sake of nvme, is it useful for running FreeBSD, Linux (Mint,
> > > Centos) and Windows10 guests?
> > >
>
>
> Or if you are using good SLOG and ARC for your VM zvols.
>
> I'm still sorting out some Windows issues with the bhyve nvme presentation
> for a variety of guest (except freebsd, that works fine) at the moment.  I
> would avoid 12.0-RELEASE as it may introduce other issues with your
I was just worried that using ahci-hd instead of paravirtualized disks
causes suboptimal performance of guests. Do you think there is a hope to get
paravirtualized disks support for Windows guests in bhyve?

--
Victor Sudakov,  VAS4-RIPE, VAS47-RIPN
2:5005/49@fidonet http://vas.tomsk.ru/

signature.asc (465 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: NVMe and Bhyve

Stefan Bethke-2


> Am 15.02.2019 um 05:41 schrieb Victor Sudakov <[hidden email]>:
>
> Jason Tubnor wrote:
>>
>>> If you have Nvme devices or you want store disks in ram yes, otherwise no.
>>>
>>>>
>>>> Is it worth it? I mean upgrading from 11.2-RELEASE to 12.0-RELEASE for
>>>> the sake of nvme, is it useful for running FreeBSD, Linux (Mint,
>>>> Centos) and Windows10 guests?
>>>>
>>
>>
>> Or if you are using good SLOG and ARC for your VM zvols.
>>
>> I'm still sorting out some Windows issues with the bhyve nvme presentation
>> for a variety of guest (except freebsd, that works fine) at the moment.  I
>> would avoid 12.0-RELEASE as it may introduce other issues with your
>
> I was just worried that using ahci-hd instead of paravirtualized disks
> causes suboptimal performance of guests. Do you think there is a hope to get
> paravirtualized disks support for Windows guests in bhyve?

I was under the impression that the Windows VirtIO disk driver works with the Bhyve virtio-blk device. You need to provide these drivers to Windows somehow (for example, as a second CD image during installation): https://www.linux-kvm.org/page/WindowsGuestDrivers/Download_Drivers


Stefan

--
Stefan Bethke <[hidden email]>   Fon +49 151 14070811

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: NVMe and Bhyve

Victor Sudakov-3
Stefan Bethke wrote:

> >
> > I was just worried that using ahci-hd instead of paravirtualized disks
> > causes suboptimal performance of guests. Do you think there is a hope to get
> > paravirtualized disks support for Windows guests in bhyve?
>
> I was under the impression that the Windows VirtIO disk driver works
> with the Bhyve virtio-blk device. You need to provide these drivers to
> Windows somehow (for example, as a second CD image during
> installation):
> https://www.linux-kvm.org/page/WindowsGuestDrivers/Download_Drivers
I use NetKVM (VirtIO network) driver from there all the time, because
e1000 is bad.

However, I've read a number of bhyve resources including
https://wiki.freebsd.org/bhyve/Windows and I've never found a success
story with paravirtualized Windows disk drivers.

If anyone has been successful with viostor(?), please share your experience.

I would be especially grateful for instruction how to install Windows on
a viostor disk instead of ahci-hd.

--
Victor Sudakov,  VAS4-RIPE, VAS47-RIPN
2:5005/49@fidonet http://vas.tomsk.ru/

signature.asc (465 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: NVMe and Bhyve

Victor Sudakov-3
Victor Sudakov wrote:
>
> If anyone has been successful with viostor(?), please share your experience.

Or at least please tell me how I can attach another iso image (with
drivers) when running "vm install myhost windows.iso"

--
Victor Sudakov,  VAS4-RIPE, VAS47-RIPN
2:5005/49@fidonet http://vas.tomsk.ru/

signature.asc (465 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: NVMe and Bhyve

Jason Tubnor
On Mon, 18 Feb 2019 at 02:58, Victor Sudakov <[hidden email]> wrote:

>
> Or at least please tell me how I can attach another iso image (with
> drivers) when running "vm install myhost windows.iso"


Technically, you should be able to define something like:

disk1_type="ahci-cd"
disk1_name="virtio-win-0.1.164.iso"
disk1_dev="file"

in the vm.conf file if the virtio-win-0.1.164.iso is located in
/vm/servername/ directory.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: NVMe and Bhyve

Victor Sudakov-6
Jason Tubnor wrote:

> On Mon, 18 Feb 2019 at 02:58, Victor Sudakov <[hidden email]> wrote:
>
> >
> > Or at least please tell me how I can attach another iso image (with
> > drivers) when running "vm install myhost windows.iso"
>
>
> Technically, you should be able to define something like:
>
> disk1_type="ahci-cd"
> disk1_name="virtio-win-0.1.164.iso"
> disk1_dev="file"
>
> in the vm.conf file if the virtio-win-0.1.164.iso is located in
> /vm/servername/ directory.
Well, disk0 will be the guest HDD to install to, disk1 will be the ISO with
drivers, and which disk will be the ISO attached by the "vm install"
command?

--
Victor Sudakov,  VAS4-RIPE, VAS47-RIPN
2:5005/49@fidonet http://vas.tomsk.ru/

signature.asc (465 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: NVMe and Bhyve

Jason Tubnor
On Mon, 18 Feb 2019 at 12:09, Victor Sudakov <[hidden email]> wrote:

>
> Well, disk0 will be the guest HDD to install to, disk1 will be the ISO with
> drivers, and which disk will be the ISO attached by the "vm install"
> command?
>
>
 From a user perspective, this is not of any value to know.  vm-bhyve
manages that in the background and usually puts the regular disk0 as -s 4:0
regardless of normal boot or install and when the installer is invoked,
puts the installer iso in as -s 3:0

So based on previous examples, you'd see something like this in the log:
-s 3:0,ahci-cd,/vm/.iso/myinstaliso.iso -s
4:0,ahci-hd,/dev/zvol/mypool/vm/myserver/disk0 -s
5:0,ahci-cd,/vm/myserver/virtio-win-0.1.164.iso
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: NVMe and Bhyve

Rodney W. Grimes-4
In reply to this post by Jason Tubnor
> On Mon, 18 Feb 2019 at 02:58, Victor Sudakov <[hidden email]> wrote:
>
> >
> > Or at least please tell me how I can attach another iso image (with
> > drivers) when running "vm install myhost windows.iso"
>
>
> Technically, you should be able to define something like:
>
> disk1_type="ahci-cd"
> disk1_name="virtio-win-0.1.164.iso"
> disk1_dev="file"
or
disk1_dev="custom" if you wish to use a path in disk1_name,
I still do not know why this needs to be treated differently,
a path should just work :-(

> in the vm.conf file if the virtio-win-0.1.164.iso is located in
> /vm/servername/ directory.
--
Rod Grimes                                                 [hidden email]
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: NVMe and Bhyve

Victor Sudakov-3
In reply to this post by Jason Tubnor
Jason Tubnor wrote:

>
> >
> > Well, disk0 will be the guest HDD to install to, disk1 will be the ISO with
> > drivers, and which disk will be the ISO attached by the "vm install"
> > command?
> >
> >
>  From a user perspective, this is not of any value to know.  vm-bhyve
> manages that in the background and usually puts the regular disk0 as -s 4:0
> regardless of normal boot or install and when the installer is invoked,
> puts the installer iso in as -s 3:0
Oh, if vm-bhyve manages that the installer ISO is always attached
before other ancillary ISOs, that's fine with me.

>
> So based on previous examples, you'd see something like this in the log:
> -s 3:0,ahci-cd,/vm/.iso/myinstaliso.iso -s
> 4:0,ahci-hd,/dev/zvol/mypool/vm/myserver/disk0 -s
> 5:0,ahci-cd,/vm/myserver/virtio-win-0.1.164.iso

I see. I have tried this and it works as you describe, which is good
news.

However, there is bad news too. I've tried to install Windows 10 and
Windows Server 2012R2 onto the virtio-blk device, but...

The guest installer says that there are no disks to install to and no
drivers, and suggests a driver to be loaded. It can be told to find the
appropriate driver in my virtio-win-0.1.141.iso CD:

http://admin.sibptus.ru/~vas/bhyve1.png

but when it begins loading the viostor driver, the VM crashes with
"bhyve exited with status 134"

Therefore I ask again if someone has an actual success story of running
a Windows guest on a virtio-blk device.

--
Victor Sudakov,  VAS4-RIPE, VAS47-RIPN
2:5005/49@fidonet http://vas.tomsk.ru/

signature.asc (465 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: NVMe and Bhyve

Victor Sudakov-3
In reply to this post by Jason Tubnor
Jason Tubnor wrote:

>
> >
> > Well, disk0 will be the guest HDD to install to, disk1 will be the ISO with
> > drivers, and which disk will be the ISO attached by the "vm install"
> > command?
> >
> >
>  From a user perspective, this is not of any value to know.  vm-bhyve
> manages that in the background and usually puts the regular disk0 as -s 4:0
> regardless of normal boot or install and when the installer is invoked,
> puts the installer iso in as -s 3:0

Oh, if vm-bhyve manages that the installer ISO is always attached
before other ancillary ISOs, that's fine with me.

>
> So based on previous examples, you'd see something like this in the log:
> -s 3:0,ahci-cd,/vm/.iso/myinstaliso.iso -s
> 4:0,ahci-hd,/dev/zvol/mypool/vm/myserver/disk0 -s
> 5:0,ahci-cd,/vm/myserver/virtio-win-0.1.164.iso

I see. I have tried this and it works as you describe, which is good
news.

However, there is bad news too. I've tried to install Windows 10 and
Windows Server 2012R2 onto the virtio-blk device, but...

The guest installer says that there are no disks to install to and no
drivers, and suggests a driver to be loaded. It can be told to find the
appropriate driver in my virtio-win-0.1.141.iso CD:

http://admin.sibptus.ru/~vas/bhyve1.png

but when it begins loading the viostor driver, the VM crashes with
"bhyve exited with status 134"

Therefore I ask again if someone has an actual success story of running
a Windows guest on a virtio-blk device.

--
Victor Sudakov,  VAS4-RIPE, VAS47-RIPN
2:5005/49@fidonet http://vas.tomsk.ru/
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to "[hidden email]"
12