emulated PCI device BAR size

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

emulated PCI device BAR size

Chuck Tuffli-3
In debugging why Windows doesn't like bhyve's NVMe device emulation,
another developer mentioned that the NVMe specification defines the
minimum size of BAR[0] to be 16K bytes. And while most OS don't
enforce this requirement, evidently, Windows does.

On the surface, a "minimum BAR size" sounds odd, but the MMIO BAR in
NVMe includes the doorbell registers for each queue. And since the
maximum number of queues is implementation specific, there isn't a
fixed size for the BAR.

So my question is how to best fix this. The easiest would be to add a
check to the BAR size calculation in pci_nvme.c along the lines of:
    #define NVME_MMIO_SPACE_MIN    (1 << 14)
    ...
    pci_membar_sz = MAX( <calculation based on queue count>,
NVME_MMIO_SPACE_MIN);

While this makes Windows happy, are there any implications to this
approach other than chewing up more memory than the device actually
needs? Alternatively, is there a better mechanism to use in this case?

--chuck
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: emulated PCI device BAR size

Chuck Tuffli-3
On Mon, Mar 18, 2019 at 10:15 AM Chuck Tuffli <[hidden email]> wrote:

>
> In debugging why Windows doesn't like bhyve's NVMe device emulation,
> another developer mentioned that the NVMe specification defines the
> minimum size of BAR[0] to be 16K bytes. And while most OS don't
> enforce this requirement, evidently, Windows does.
>
> On the surface, a "minimum BAR size" sounds odd, but the MMIO BAR in
> NVMe includes the doorbell registers for each queue. And since the
> maximum number of queues is implementation specific, there isn't a
> fixed size for the BAR.
>
> So my question is how to best fix this. The easiest would be to add a
> check to the BAR size calculation in pci_nvme.c along the lines of:
>     #define NVME_MMIO_SPACE_MIN    (1 << 14)
>     ...
>     pci_membar_sz = MAX( <calculation based on queue count>,
> NVME_MMIO_SPACE_MIN);

I went down this route and have a patch up for review
(https://reviews.freebsd.org/D19676) if anyone is interested.

--chuck
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: emulated PCI device BAR size

Jason Tubnor-2
On Sun., 24 Mar. 2019, 6:35 am Chuck Tuffli, <[hidden email]> wrote:

> On Mon, Mar 18, 2019 at 10:15 AM Chuck Tuffli <[hidden email]> wrote:
>
> >
> > So my question is how to best fix this. The easiest would be to add a
> > check to the BAR size calculation in pci_nvme.c along the lines of:
> >     #define NVME_MMIO_SPACE_MIN    (1 << 14)
> >     ...
> >     pci_membar_sz = MAX( <calculation based on queue count>,
> > NVME_MMIO_SPACE_MIN);
>
> I went down this route and have a patch up for review
> (https://reviews.freebsd.org/D19676) if anyone is interested.
>
> --chuck
>

I've built and installed this patch on FreeBSD
12.0-STABLE-amd64-20190314-r345117.  Testing occurred against Windows 10
Pro guest with a presented NVMe bhyve device and all IO transactions worked
without issue.

Can this be committed to head and MFD to 12-STABLE.  Thanks!
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: emulated PCI device BAR size

Chuck Tuffli-3
On Sat, Mar 23, 2019 at 6:44 PM Jason Tubnor <[hidden email]> wrote:

>
>
>
> On Sun., 24 Mar. 2019, 6:35 am Chuck Tuffli, <[hidden email]> wrote:
>>
>> On Mon, Mar 18, 2019 at 10:15 AM Chuck Tuffli <[hidden email]> wrote:
>>
>> >
>> > So my question is how to best fix this. The easiest would be to add a
>> > check to the BAR size calculation in pci_nvme.c along the lines of:
>> >     #define NVME_MMIO_SPACE_MIN    (1 << 14)
>> >     ...
>> >     pci_membar_sz = MAX( <calculation based on queue count>,
>> > NVME_MMIO_SPACE_MIN);
>>
>> I went down this route and have a patch up for review
>> (https://reviews.freebsd.org/D19676) if anyone is interested.
>>
>> --chuck
>
>
> I've built and installed this patch on FreeBSD 12.0-STABLE-amd64-20190314-r345117.  Testing occurred against Windows 10 Pro guest with a presented NVMe bhyve device and all IO transactions worked without issue.
>
> Can this be committed to head and MFD to 12-STABLE.  Thanks!

Committed with r345956. MFC in 2 weeks if nothing pops up. Thanks
again for all your testing!

--chuck

https://svnweb.freebsd.org/changeset/base/345956
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: emulated PCI device BAR size

Rodney W. Grimes-6
> On Sat, Mar 23, 2019 at 6:44 PM Jason Tubnor <[hidden email]> wrote:
> >
> >
> >
> > On Sun., 24 Mar. 2019, 6:35 am Chuck Tuffli, <[hidden email]> wrote:
> >>
> >> On Mon, Mar 18, 2019 at 10:15 AM Chuck Tuffli <[hidden email]> wrote:
> >>
> >> >
> >> > So my question is how to best fix this. The easiest would be to add a
> >> > check to the BAR size calculation in pci_nvme.c along the lines of:
> >> >     #define NVME_MMIO_SPACE_MIN    (1 << 14)
> >> >     ...
> >> >     pci_membar_sz = MAX( <calculation based on queue count>,
> >> > NVME_MMIO_SPACE_MIN);
> >>
> >> I went down this route and have a patch up for review
> >> (https://reviews.freebsd.org/D19676) if anyone is interested.
> >>
> >> --chuck
> >
> >
> > I've built and installed this patch on FreeBSD 12.0-STABLE-amd64-20190314-r345117.  Testing occurred against Windows 10 Pro guest with a presented NVMe bhyve device and all IO transactions worked without issue.
> >
> > Can this be committed to head and MFD to 12-STABLE.  Thanks!
>
> Committed with r345956. MFC in 2 weeks if nothing pops up. Thanks
> again for all your testing!

A small prod Chuck you should of mentioned Jason in the
commit message for his testing work.  I know it is easy
to forget this, but it is the right thing to do.

Jason,
  I'll thank you here, in a more personal maner, for working with
Chuck on getting this fixed.

Thanks,
--
Rod Grimes                                                 [hidden email]
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to "[hidden email]"