bhyve PCIe passthrough on AMD Epyc

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

bhyve PCIe passthrough on AMD Epyc

Kevin Day-3
I'm trying to get PCIe device (video card) passthrough working to a Windows bhyve VM, should this work now?

I have the device blocked off in loader.conf, and added hw.vmm.amdvi.enable="1".  After doing so this is how it appears in pciconf:

ppt0@pci0:97:0:0: class=0x030000 card=0x27173842 chip=0x128b10de rev=0xa1 hdr=0x00
    bar   [10] = type Memory, range 32, base 0xe0000000, size 16777216, disabled
    bar   [14] = type Prefetchable Memory, range 64, base 0xfcf0000000, size 134217728, disabled
    bar   [1c] = type Prefetchable Memory, range 64, base 0xfcf8000000, size 33554432, disabled
    bar   [24] = type I/O Port, range 32, base 0x3000, size 128, disabled
    cap 01[60] = powerspec 3  supports D0 D3  current D0
    cap 05[68] = MSI supports 1 message, 64 bit
    cap 10[78] = PCI-Express 2 legacy endpoint max data 256(256) RO NS
                 link x8(x8) speed 5.0(5.0)
    ecap 0002[100] = VC 1 max VC0
    ecap 0004[128] = Power Budgeting 1
    ecap 000b[600] = Vendor 1 ID 1

When trying to attach it to a VM with "-s 9:0,passthru,97/0/0" I get this almost immediately after it starts up:

Assertion failed: (error == 0), function modify_bar_registration, file /usr/src/usr.sbin/bhyve/pci_emul.c, line 504.

Also on the console I'm seeing a lot of messages like:

ivhd0: Error: completion failed tail:0x1c0, head:0x0.
ivhd0: Dump all the commands:
  [CMD0, off:0xff0] opcode= 0x1 0x304f1629 0x30 0xa5a5
  [CMD1, off:0x0] opcode= 0x1 0x304f1629 0x30 0xa5a5
  [CMD2, off:0x10] opcode= 0x1 0x304f1629 0x30 0xa5a5
  [CMD3, off:0x20] opcode= 0x1 0x304f1629 0x30 0xa5a5
  [CMD4, off:0x30] opcode= 0x1 0x304f1629 0x30 0xa5a5
  [CMD5, off:0x40] opcode= 0x1 0x304f1629 0x30 0xa5a5
  [CMD6, off:0x50] opcode= 0x1 0x304f1629 0x30 0xa5a5
  [CMD7, off:0x60] opcode= 0x2 0x6101 0x0 0x0
  [CMD8, off:0x70] opcode= 0x1 0x304f1629 0x30 0xa5a5
  [CMD9, off:0x80] opcode= 0x1 0x304f1629 0x30 0xa5a5
  [CMD10, off:0x90] opcode= 0x1 0x304f1629 0x30 0xa5a5
  [CMD11, off:0xa0] opcode= 0x1 0x304f1629 0x30 0xa5a5
  [CMD12, off:0xb0] opcode= 0x1 0x304f1629 0x30 0xa5a5
  [CMD13, off:0xc0] opcode= 0x1 0x304f1629 0x30 0xa5a5
  [CMD14, off:0xd0] opcode= 0x1 0x304f1629 0x30 0xa5a5
  [CMD15, off:0xe0] opcode= 0x1 0x304f1629 0x30 0xa5a5
  [CMD16, off:0xf0] opcode= 0x1 0x304f1629 0x30 0xa5a5
  [CMD17, off:0x100] opcode= 0x1 0x304f1629 0x30 0xa5a5
  [CMD18, off:0x110] opcode= 0x2 0x6100 0x0 0x0
  [CMD19, off:0x120] opcode= 0x1 0x304f1629 0x30 0xa5a5
  [CMD20, off:0x130] opcode= 0x1 0x304f1629 0x30 0xa5a5
  [CMD21, off:0x140] opcode= 0x1 0x304f1629 0x30 0xa5a5
  [CMD22, off:0x150] opcode= 0x1 0x304f1629 0x30 0xa5a5
  [CMD23, off:0x160] opcode= 0x1 0x304f1629 0x30 0xa5a5
  [CMD24, off:0x170] opcode= 0x1 0x304f1629 0x30 0xa5a5
  [CMD25, off:0x180] opcode= 0x1 0x304f1629 0x30 0xa5a5
  [CMD26, off:0x190] opcode= 0x1 0x304f1629 0x30 0xa5a5
  [CMD27, off:0x1a0] opcode= 0x1 0x304f1629 0x30 0xa5a5
  [CMD28, off:0x1b0] opcode= 0x1 0x304f1629 0x30 0xa5a5


_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: bhyve PCIe passthrough on AMD Epyc

Rodney W. Grimes-4
> I'm trying to get PCIe device (video card) passthrough working to a Windows bhyve VM, should this work now?

No, this should not work now.

> I have the device blocked off in loader.conf, and added hw.vmm.amdvi.enable="1".  After doing so this is how it appears in pciconf:
>
> ppt0@pci0:97:0:0: class=0x030000 card=0x27173842 chip=0x128b10de rev=0xa1 hdr=0x00
>     bar   [10] = type Memory, range 32, base 0xe0000000, size 16777216, disabled
>     bar   [14] = type Prefetchable Memory, range 64, base 0xfcf0000000, size 134217728, disabled
>     bar   [1c] = type Prefetchable Memory, range 64, base 0xfcf8000000, size 33554432, disabled
>     bar   [24] = type I/O Port, range 32, base 0x3000, size 128, disabled
>     cap 01[60] = powerspec 3  supports D0 D3  current D0
>     cap 05[68] = MSI supports 1 message, 64 bit
>     cap 10[78] = PCI-Express 2 legacy endpoint max data 256(256) RO NS
>                  link x8(x8) speed 5.0(5.0)
>     ecap 0002[100] = VC 1 max VC0
>     ecap 0004[128] = Power Budgeting 1
>     ecap 000b[600] = Vendor 1 ID 1
>
> When trying to attach it to a VM with "-s 9:0,passthru,97/0/0" I get this almost immediately after it starts up:
>
> Assertion failed: (error == 0), function modify_bar_registration, file /usr/src/usr.sbin/bhyve/pci_emul.c, line 504.
>
> Also on the console I'm seeing a lot of messages like:
>
> ivhd0: Error: completion failed tail:0x1c0, head:0x0.
> ivhd0: Dump all the commands:
>   [CMD0, off:0xff0] opcode= 0x1 0x304f1629 0x30 0xa5a5
>   [CMD1, off:0x0] opcode= 0x1 0x304f1629 0x30 0xa5a5
>   [CMD2, off:0x10] opcode= 0x1 0x304f1629 0x30 0xa5a5
>   [CMD3, off:0x20] opcode= 0x1 0x304f1629 0x30 0xa5a5
>   [CMD4, off:0x30] opcode= 0x1 0x304f1629 0x30 0xa5a5
>   [CMD5, off:0x40] opcode= 0x1 0x304f1629 0x30 0xa5a5
>   [CMD6, off:0x50] opcode= 0x1 0x304f1629 0x30 0xa5a5
>   [CMD7, off:0x60] opcode= 0x2 0x6101 0x0 0x0
>   [CMD8, off:0x70] opcode= 0x1 0x304f1629 0x30 0xa5a5
>   [CMD9, off:0x80] opcode= 0x1 0x304f1629 0x30 0xa5a5
>   [CMD10, off:0x90] opcode= 0x1 0x304f1629 0x30 0xa5a5
>   [CMD11, off:0xa0] opcode= 0x1 0x304f1629 0x30 0xa5a5
>   [CMD12, off:0xb0] opcode= 0x1 0x304f1629 0x30 0xa5a5
>   [CMD13, off:0xc0] opcode= 0x1 0x304f1629 0x30 0xa5a5
>   [CMD14, off:0xd0] opcode= 0x1 0x304f1629 0x30 0xa5a5
>   [CMD15, off:0xe0] opcode= 0x1 0x304f1629 0x30 0xa5a5
>   [CMD16, off:0xf0] opcode= 0x1 0x304f1629 0x30 0xa5a5
>   [CMD17, off:0x100] opcode= 0x1 0x304f1629 0x30 0xa5a5
>   [CMD18, off:0x110] opcode= 0x2 0x6100 0x0 0x0
>   [CMD19, off:0x120] opcode= 0x1 0x304f1629 0x30 0xa5a5
>   [CMD20, off:0x130] opcode= 0x1 0x304f1629 0x30 0xa5a5
>   [CMD21, off:0x140] opcode= 0x1 0x304f1629 0x30 0xa5a5
>   [CMD22, off:0x150] opcode= 0x1 0x304f1629 0x30 0xa5a5
>   [CMD23, off:0x160] opcode= 0x1 0x304f1629 0x30 0xa5a5
>   [CMD24, off:0x170] opcode= 0x1 0x304f1629 0x30 0xa5a5
>   [CMD25, off:0x180] opcode= 0x1 0x304f1629 0x30 0xa5a5
>   [CMD26, off:0x190] opcode= 0x1 0x304f1629 0x30 0xa5a5
>   [CMD27, off:0x1a0] opcode= 0x1 0x304f1629 0x30 0xa5a5
>   [CMD28, off:0x1b0] opcode= 0x1 0x304f1629 0x30 0xa5a5
>
>
> _______________________________________________
> [hidden email] mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> To unsubscribe, send any mail to "[hidden email]"
>

--
Rod Grimes                                                 [hidden email]
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: bhyve PCIe passthrough on AMD Epyc

Kevin Day-3


> On Feb 27, 2019, at 12:32 AM, Rodney W. Grimes <[hidden email]> wrote:
>
>> I'm trying to get PCIe device (video card) passthrough working to a Windows bhyve VM, should this work now?
>
> No, this should not work now.

Ok, easy enough. Should *any* PCIe passthrough work on Epyc right now? Just trying the same thing with a simple PCIe USB card gives me:

CPU0: local APIC error 0x40
CPU0: local APIC error 0x40
ivhd0: Error: completion failed tail:0xda0, head:0x0.
ivhd0: Dump all the commands:
  [CMD0, off:0xff0] opcode= 0x0 0x0 0x0 0x0
  [CMD1, off:0x0] opcode= 0x3 0x0 0x0 0x7ffffffffffff003
  [CMD2, off:0x10] opcode= 0x0 0x0 0x0 0x0
  [CMD3, off:0x20] opcode= 0x2 0x0 0x0 0x0
  [CMD4, off:0x30] opcode= 0x2 0x2 0x0 0x0
[...snip... lots of similar stuff]

Then I get a bunch more APIC errors, and the system freezes for about 60 seconds before recovering.

-- Kevin

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: bhyve PCIe passthrough on AMD Epyc

Rodney W. Grimes-4
> > On Feb 27, 2019, at 12:32 AM, Rodney W. Grimes <[hidden email]> wrote:
> >
> >> I'm trying to get PCIe device (video card) passthrough working to a Windows bhyve VM, should this work now?
> >
> > No, this should not work now.
>
> Ok, easy enough. Should *any* PCIe passthrough work on Epyc right now?
I do not know what would be special about Epyc so can not say
for certain.

> Just trying the same thing with a simple PCIe USB card gives me:
>
> CPU0: local APIC error 0x40
> CPU0: local APIC error 0x40
> ivhd0: Error: completion failed tail:0xda0, head:0x0.
> ivhd0: Dump all the commands:
>   [CMD0, off:0xff0] opcode= 0x0 0x0 0x0 0x0
>   [CMD1, off:0x0] opcode= 0x3 0x0 0x0 0x7ffffffffffff003
>   [CMD2, off:0x10] opcode= 0x0 0x0 0x0 0x0
>   [CMD3, off:0x20] opcode= 0x2 0x0 0x0 0x0
>   [CMD4, off:0x30] opcode= 0x2 0x2 0x0 0x0
> [...snip... lots of similar stuff]
>
> Then I get a bunch more APIC errors, and the system freezes for about 60 seconds before recovering.

You do not get any host dmesg errors about unaligned BAR
ranges or sizes?

Can you get the pciconf -l -b info for the device you passed, something like:
# pciconf -l -b uhci0
uhci0@pci0:0:26:0:      class=0x0c0300 card=0x20f017aa chip=0x29378086 rev=0x03 hdr=0x00
    bar   [20] = type I/O Port, range 32, base 0x1840, size 32, enabled

> -- Kevin
--
Rod Grimes                                                 [hidden email]
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: bhyve PCIe passthrough on AMD Epyc

Kevin Day-3


> On Feb 27, 2019, at 11:34 PM, Rodney W. Grimes <[hidden email]> wrote:
>
>>> On Feb 27, 2019, at 12:32 AM, Rodney W. Grimes <[hidden email]> wrote:
>>>
>>>> I'm trying to get PCIe device (video card) passthrough working to a Windows bhyve VM, should this work now?
>>>
>>> No, this should not work now.
>>
>> Ok, easy enough. Should *any* PCIe passthrough work on Epyc right now?
> I do not know what would be special about Epyc so can not say
> for certain.

I'm just pointing out that it's an AMD Epyc because I know the AMD IOMMU/AMD-Vi code was added somewhat more recently than Intel's and is maybe less tested.

>> Just trying the same thing with a simple PCIe USB card gives me:
>>
>> CPU0: local APIC error 0x40
>> CPU0: local APIC error 0x40
>> ivhd0: Error: completion failed tail:0xda0, head:0x0.
>> ivhd0: Dump all the commands:
>>  [CMD0, off:0xff0] opcode= 0x0 0x0 0x0 0x0
>>  [CMD1, off:0x0] opcode= 0x3 0x0 0x0 0x7ffffffffffff003
>>  [CMD2, off:0x10] opcode= 0x0 0x0 0x0 0x0
>>  [CMD3, off:0x20] opcode= 0x2 0x0 0x0 0x0
>>  [CMD4, off:0x30] opcode= 0x2 0x2 0x0 0x0
>> [...snip... lots of similar stuff]
>>
>> Then I get a bunch more APIC errors, and the system freezes for about 60 seconds before recovering.
>
> You do not get any host dmesg errors about unaligned BAR
> ranges or sizes?
>

Correct. I did with the video card, but not with the xhci USB card. I get a ton of dmesg errors, and the host system repeatedly freezes until I stop the VM. (Console is non-responsive, no ICMP replies, etc)

I didn't actually look at this until now, but Windows actually does see the USB controller, but it complains that "port reset failed" on any device I attach to it.

> Can you get the pciconf -l -b info for the device you passed, something like:
> # pciconf -l -b uhci0
> uhci0@pci0:0:26:0:      class=0x0c0300 card=0x20f017aa chip=0x29378086 rev=0x03 hdr=0x00
>    bar   [20] = type I/O Port, range 32, base 0x1840, size 32, enabled

xhci3@pci0:97:0:0: class=0x0c0330 card=0xffffffff chip=0x00151912 rev=0x02 hdr=0x00
    bar   [10] = type Memory, range 64, base 0xe1d00000, size 8192, enabled
    cap 01[50] = powerspec 3  supports D0 D3  current D0
    cap 05[70] = MSI supports 8 messages, 64 bit enabled with 1 message
    cap 11[90] = MSI-X supports 8 messages
                 Table in map 0x10[0x1000], PBA in map 0x10[0x1080]
    cap 10[a0] = PCI-Express 2 endpoint max data 128(128) NS
                 link x1(x1) speed 5.0(5.0) ASPM L1(L0s/L1)
    ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
    ecap 0018[150] = LTR 1

Also just in case it's relevant, "acpidump" on this system hangs after allocating ~3GB of ram and just spins at 100% CPU. I am seeing that AMD-Vi is enabled in the boot dmesg though:

AMD-Vi: IVRS Info VAsize = 64 PAsize = 48 GVAsize = 2 flags:0






_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: bhyve PCIe passthrough on AMD Epyc

Anish Gupta
Hi Kevin,
>I'm just pointing out that it's an AMD Epyc because I know the AMD
IOMMU/AMD-Vi code was added somewhat more recently than Intel's and is
maybe less tested.
I tested on Ryzen-I, not sure if anyone tried it on newer Ryzen/Epyc
systems.

>CPU0: local APIC error 0x40
This is concerning, as per AMD docs, bit6 is for illegal vector which
indicate CPU is getting invalid interrupts.

Can you share hw.vmm sysctl output?

-Anish


On Thu, Feb 28, 2019 at 9:59 AM Kevin Day <[hidden email]> wrote:

>
>
> > On Feb 27, 2019, at 11:34 PM, Rodney W. Grimes <
> [hidden email]> wrote:
> >
> >>> On Feb 27, 2019, at 12:32 AM, Rodney W. Grimes <
> [hidden email]> wrote:
> >>>
> >>>> I'm trying to get PCIe device (video card) passthrough working to a
> Windows bhyve VM, should this work now?
> >>>
> >>> No, this should not work now.
> >>
> >> Ok, easy enough. Should *any* PCIe passthrough work on Epyc right now?
> > I do not know what would be special about Epyc so can not say
> > for certain.
>
> I'm just pointing out that it's an AMD Epyc because I know the AMD
> IOMMU/AMD-Vi code was added somewhat more recently than Intel's and is
> maybe less tested.
>
> >> Just trying the same thing with a simple PCIe USB card gives me:
> >>
> >> CPU0: local APIC error 0x40
> >> CPU0: local APIC error 0x40
> >> ivhd0: Error: completion failed tail:0xda0, head:0x0.
> >> ivhd0: Dump all the commands:
> >>  [CMD0, off:0xff0] opcode= 0x0 0x0 0x0 0x0
> >>  [CMD1, off:0x0] opcode= 0x3 0x0 0x0 0x7ffffffffffff003
> >>  [CMD2, off:0x10] opcode= 0x0 0x0 0x0 0x0
> >>  [CMD3, off:0x20] opcode= 0x2 0x0 0x0 0x0
> >>  [CMD4, off:0x30] opcode= 0x2 0x2 0x0 0x0
> >> [...snip... lots of similar stuff]
> >>
> >> Then I get a bunch more APIC errors, and the system freezes for about
> 60 seconds before recovering.
> >
> > You do not get any host dmesg errors about unaligned BAR
> > ranges or sizes?
> >
>
> Correct. I did with the video card, but not with the xhci USB card. I get
> a ton of dmesg errors, and the host system repeatedly freezes until I stop
> the VM. (Console is non-responsive, no ICMP replies, etc)
>
> I didn't actually look at this until now, but Windows actually does see
> the USB controller, but it complains that "port reset failed" on any device
> I attach to it.
>
> > Can you get the pciconf -l -b info for the device you passed, something
> like:
> > # pciconf -l -b uhci0
> > uhci0@pci0:0:26:0:      class=0x0c0300 card=0x20f017aa chip=0x29378086
> rev=0x03 hdr=0x00
> >    bar   [20] = type I/O Port, range 32, base 0x1840, size 32, enabled
>
> xhci3@pci0:97:0:0:      class=0x0c0330 card=0xffffffff chip=0x00151912
> rev=0x02 hdr=0x00
>     bar   [10] = type Memory, range 64, base 0xe1d00000, size 8192, enabled
>     cap 01[50] = powerspec 3  supports D0 D3  current D0
>     cap 05[70] = MSI supports 8 messages, 64 bit enabled with 1 message
>     cap 11[90] = MSI-X supports 8 messages
>                  Table in map 0x10[0x1000], PBA in map 0x10[0x1080]
>     cap 10[a0] = PCI-Express 2 endpoint max data 128(128) NS
>                  link x1(x1) speed 5.0(5.0) ASPM L1(L0s/L1)
>     ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
>     ecap 0018[150] = LTR 1
>
> Also just in case it's relevant, "acpidump" on this system hangs after
> allocating ~3GB of ram and just spins at 100% CPU. I am seeing that AMD-Vi
> is enabled in the boot dmesg though:
>
> AMD-Vi: IVRS Info VAsize = 64 PAsize = 48 GVAsize = 2 flags:0
>
>
>
>
>
>
> _______________________________________________
> [hidden email] mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> To unsubscribe, send any mail to "
> [hidden email]"
>
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: bhyve PCIe passthrough on AMD Epyc

Kevin Day-3


> On Feb 28, 2019, at 12:28 PM, Anish <[hidden email]> wrote:
>
> Hi Kevin,
> >I'm just pointing out that it's an AMD Epyc because I know the AMD IOMMU/AMD-Vi code was added somewhat more recently than Intel's and is maybe less tested.
> I tested on Ryzen-I, not sure if anyone tried it on newer Ryzen/Epyc systems.
>
> >CPU0: local APIC error 0x40
> This is concerning, as per AMD docs, bit6 is for illegal vector which indicate CPU is getting invalid interrupts.
>
> Can you share hw.vmm sysctl output?
>
> -Anish

Sure:

hw.vmm.amdvi.domain_id: 4
hw.vmm.amdvi.disable_io_fault: 0
hw.vmm.amdvi.ptp_level: 4
hw.vmm.amdvi.host_ptp: 1
hw.vmm.amdvi.enable: 1
hw.vmm.amdvi.count: 8
hw.vmm.npt.pmap_flags: 507
hw.vmm.svm.num_asids: 32768
hw.vmm.svm.disable_npf_assist: 0
hw.vmm.svm.features: 113919
hw.vmm.svm.vmcb_clean: 1023
hw.vmm.vmx.l1d_flush_sw: 0
hw.vmm.vmx.l1d_flush: 0
hw.vmm.vmx.vpid_alloc_failed: 0
hw.vmm.vmx.posted_interrupt_vector: -1
hw.vmm.vmx.cap.posted_interrupts: 0
hw.vmm.vmx.cap.virtual_interrupt_delivery: 0
hw.vmm.vmx.cap.invpcid: 0
hw.vmm.vmx.cap.monitor_trap: 0
hw.vmm.vmx.cap.unrestricted_guest: 0
hw.vmm.vmx.cap.pause_exit: 0
hw.vmm.vmx.cap.halt_exit: 0
hw.vmm.vmx.initialized: 0
hw.vmm.vmx.cr4_zeros_mask: 0
hw.vmm.vmx.cr4_ones_mask: 0
hw.vmm.vmx.cr0_zeros_mask: 0
hw.vmm.vmx.cr0_ones_mask: 0
hw.vmm.vmx.no_flush_rsb: 0
hw.vmm.ept.pmap_flags: 0
hw.vmm.vrtc.flag_broken_time: 1
hw.vmm.ppt.devices: 1
hw.vmm.iommu.enable: 1
hw.vmm.iommu.initialized: 1
hw.vmm.bhyve_xcpuids: 312
hw.vmm.topology.cpuid_leaf_b: 1
hw.vmm.create: beavis
hw.vmm.destroy: beavis
hw.vmm.trace_guest_exceptions: 0
hw.vmm.ipinum: 251
hw.vmm.halt_detection: 1



_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to "[hidden email]"