[Bug 224003] xen kernel panics

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

[Bug 224003] xen kernel panics

Chris-2
I'm having similar problems with xen on 11.1, AMD64, June 2017.
The OS runs fine, but installed xen from package, setup exactly
as per the handbook and get a kernel panic on two machines,
complaining about iommu not being enabled.

Machines are: a Sun X4170 to start, then  a Proliant DL380 G7
with E5630 cpu. Both have all the virtualisation options
enabled in the bios, but there are no options on either machine
  for iommu.

Have other machines running FreeBSD, an Atom pc running as a
nas box, a couple of other Intel based systems and a Sparc
machine running 10.2, with a shedload of packages built to try
and get  an X desktop running. Got as far as xvnc remote
desktop working, but still trying to get Firefox built.

Normally, everything works out of the box with FreeBSD , so has
this really been tested, or is xen just not mainstream ?. Trying
to find info on xen / FreeBSD brings up a load of sometimes
conflicting info. The xen site is no help either, with possibly
out of date / not in sync with handbook info.

FreeBSD is the escape plan from Solaris and zones, but need
zfs, jails and xen to make it all work. Any ideas ?...

Regards,

Chris
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: [Bug 224003] xen kernel panics

Roger Pau Monné
On Thu, Mar 29, 2018 at 11:32:19PM +0000, Chris wrote:
> I'm having similar problems with xen on 11.1, AMD64, June 2017.
> The OS runs fine, but installed xen from package, setup exactly
> as per the handbook and get a kernel panic on two machines,
> complaining about iommu not being enabled.

FreeBSD/Xen Dom0 requires a working IOMMU, that's documented in the
handbook [0] section 21.8.1.

Can you paste the full output that you get when booting under Xen?

> Machines are: a Sun X4170 to start, then  a Proliant DL380 G7
> with E5630 cpu. Both have all the virtualisation options
> enabled in the bios, but there are no options on either machine
>  for iommu.

On Intel hardware the IOMMU is called VT-d. I have no idea if the
hardware that you list has an IOMMU, it depends on both the CPU and
the motherboard.

Without VT-d (an IOMMU) FreeBSD/Xen Dom0 won't work.

Roger.

[0] https://www.freebsd.org/doc/handbook/virtualization-host-xen.html
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: [Bug 224003] xen kernel panics

Chris-2
On 03/30/18 12:07, Roger Pau Monné wrote:

> On Thu, Mar 29, 2018 at 11:32:19PM +0000, Chris wrote:
>> I'm having similar problems with xen on 11.1, AMD64, June 2017.
>> The OS runs fine, but installed xen from package, setup exactly
>> as per the handbook and get a kernel panic on two machines,
>> complaining about iommu not being enabled.
>
> FreeBSD/Xen Dom0 requires a working IOMMU, that's documented in the
> handbook [0] section 21.8.1.
>
> Can you paste the full output that you get when booting under Xen?
>
>> Machines are: a Sun X4170 to start, then  a Proliant DL380 G7
>> with E5630 cpu. Both have all the virtualisation options
>> enabled in the bios, but there are no options on either machine
>>   for iommu.
>
> On Intel hardware the IOMMU is called VT-d. I have no idea if the
> hardware that you list has an IOMMU, it depends on both the CPU and
> the motherboard.
>
> Without VT-d (an IOMMU) FreeBSD/Xen Dom0 won't work.
>
> Roger.
>
> [0] https://www.freebsd.org/doc/handbook/virtualization-host-xen.html
> .
>

Roger,

Thanks for the reply and clarification on the meaning of iommu. A bit
more info on the machine:

Cpu is actually an E5645, hex core, 2.4GHz
32 Gb ram, full ecc
bios is dated 2011, with:

VT-d, enabled
Intel virtualisation tech, enabled

It's difficult to log info on this, as there is just the "needs iommu"
message on boot, then halt. Live cd and file edits brought the base
system back, but nothing in /var/log/xen at all.

DL380 Proliant is industry standard vanilla and they are very common
and thus affordable second user. I know VMware runs on a G5 version
of this model, so doubt there is any problem with the hardware. One
thing found in the searches was a discussion about cpu "advisories ?"
which produces this problem, but no info on how to get round it. This
was from 2015, but would think such a bug would have been fixed by
now. This a first attempt at getting xen running under FreeBSD or Xen at
all, so a compete newbie in this area. Do Embedded rtos systems here,
no specialisation in xen or freebsd in particular, but have time and
can try various solutions if someone has any ideas.

Today: deleted packages, updated system amd ports and rebuilding
xen and tools from source. Might just be something out of sync, but
will report back once it's finished...

Regards & Thanks,

Chris



_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: [Bug 224003] xen kernel panics

Roger Pau Monné
On Fri, Mar 30, 2018 at 05:43:21PM +0000, Chris wrote:

> On 03/30/18 12:07, Roger Pau Monné wrote:
> > On Thu, Mar 29, 2018 at 11:32:19PM +0000, Chris wrote:
> > > I'm having similar problems with xen on 11.1, AMD64, June 2017.
> > > The OS runs fine, but installed xen from package, setup exactly
> > > as per the handbook and get a kernel panic on two machines,
> > > complaining about iommu not being enabled.
> >
> > FreeBSD/Xen Dom0 requires a working IOMMU, that's documented in the
> > handbook [0] section 21.8.1.
> >
> > Can you paste the full output that you get when booting under Xen?
> >
> > > Machines are: a Sun X4170 to start, then  a Proliant DL380 G7
> > > with E5630 cpu. Both have all the virtualisation options
> > > enabled in the bios, but there are no options on either machine
> > >   for iommu.
> >
> > On Intel hardware the IOMMU is called VT-d. I have no idea if the
> > hardware that you list has an IOMMU, it depends on both the CPU and
> > the motherboard.
> >
> > Without VT-d (an IOMMU) FreeBSD/Xen Dom0 won't work.
> >
> > Roger.
> >
> > [0] https://www.freebsd.org/doc/handbook/virtualization-host-xen.html
> > .
> >
>
> Roger,
>
> Thanks for the reply and clarification on the meaning of iommu. A bit
> more info on the machine:
>
> Cpu is actually an E5645, hex core, 2.4GHz
> 32 Gb ram, full ecc
> bios is dated 2011, with:
>
> VT-d, enabled
> Intel virtualisation tech, enabled
>
> It's difficult to log info on this, as there is just the "needs iommu"
> message on boot, then halt. Live cd and file edits brought the base
> system back, but nothing in /var/log/xen at all.

You can also boot into FreeBSD (without Xen) by dropping into the
loader prompt and typing:

> unload
> unset xen_kernel

I'm afraid you will need to connect to the server using a serial
console and get the output from there. Without the full serial output
it's hard to figure out what's going on.

Thanks, Roger.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: [Bug 224003] xen kernel panics

Rodney W. Grimes-4
In reply to this post by Chris-2
[ Charset ISO-8859-1 unsupported, converting... ]

> On 03/30/18 12:07, Roger Pau Monn? wrote:
> > On Thu, Mar 29, 2018 at 11:32:19PM +0000, Chris wrote:
> >> I'm having similar problems with xen on 11.1, AMD64, June 2017.
> >> The OS runs fine, but installed xen from package, setup exactly
> >> as per the handbook and get a kernel panic on two machines,
> >> complaining about iommu not being enabled.
> >
> > FreeBSD/Xen Dom0 requires a working IOMMU, that's documented in the
> > handbook [0] section 21.8.1.
> >
> > Can you paste the full output that you get when booting under Xen?
> >
> >> Machines are: a Sun X4170 to start, then  a Proliant DL380 G7
> >> with E5630 cpu. Both have all the virtualisation options
> >> enabled in the bios, but there are no options on either machine
> >>   for iommu.
> >
> > On Intel hardware the IOMMU is called VT-d. I have no idea if the
> > hardware that you list has an IOMMU, it depends on both the CPU and
> > the motherboard.
> >
> > Without VT-d (an IOMMU) FreeBSD/Xen Dom0 won't work.
> >
> > Roger.
> >
> > [0] https://www.freebsd.org/doc/handbook/virtualization-host-xen.html
> > .
> >
>
> Roger,
>
> Thanks for the reply and clarification on the meaning of iommu. A bit
> more info on the machine:
>
> Cpu is actually an E5645, hex core, 2.4GHz
> 32 Gb ram, full ecc
> bios is dated 2011, with:

I can affirm that a E56xx cpu should be very capable of supporting Xen.
CPU: Intel(R) Xeon(R) CPU           X5675  @ 3.07GHz (3059.07-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x206c2  Family=0x6  Model=0x2c  Stepping=2
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x29ee3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,POPCNT,AESNI>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x1<LAHF>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics

I do not have VT-d enabled above infact I can not find that there
is any setting for it on this hardware, though I am pretty sure
I have used hardware passthrough in vmware with that box,
which does requrire an iommu.

But a wiki for bhyve tells me to do this on intel to find if
there is a IOMMU avaliable:
# acpidump -t | grep -A12 DMAR
  DMAR: Length=432, Revision=1, Checksum=213,
        OEMID=DELL, OEM Table ID=PE_SC3, OEM Revision=0x1,
        Creator ID=DELL, Creator Revision=0x1
        Host Address Width=40
        Flags=

        Type=DRHD
        Length=16
        Flags={INCLUDE_ALL}
        Segment=0
        Address=0x00000000fed90000

        Type=RMRR

If you have a DMAR table, you have a visible to FreeBSD iommu

> VT-d, enabled
> Intel virtualisation tech, enabled
>
> It's difficult to log info on this, as there is just the "needs iommu"
> message on boot, then halt. Live cd and file edits brought the base
> system back, but nothing in /var/log/xen at all.
>
> DL380 Proliant is industry standard vanilla and they are very common
> and thus affordable second user. I know VMware runs on a G5 version
> of this model, so doubt there is any problem with the hardware.

VMware can run on almost any 64 bit processor, including those
that do not have nested page tables, iommus, or for that matter
even VT-x.  They have software emulation of this stuff since there
software predates many of these features.

> One
> thing found in the searches was a discussion about cpu "advisories ?"
> which produces this problem, but no info on how to get round it. This
> was from 2015, but would think such a bug would have been fixed by
> now. This a first attempt at getting xen running under FreeBSD or Xen at
> all, so a compete newbie in this area. Do Embedded rtos systems here,
> no specialisation in xen or freebsd in particular, but have time and
> can try various solutions if someone has any ideas.
>
> Today: deleted packages, updated system amd ports and rebuilding
> xen and tools from source. Might just be something out of sync, but
> will report back once it's finished...

Can you boot FreeBSD without Xen on it and get some dmesg output
like I did above, that would help to atleast affirm or deny that
is being seen with respect to VT-x and VT-d?

> Regards & Thanks,
> Chris

--
Rod Grimes                                                 [hidden email]
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: [Bug 224003] xen kernel panics

Chris-2
On 03/30/18 18:05, Rodney W. Grimes wrote:

> I can affirm that a E56xx cpu should be very capable of supporting Xen.
> CPU: Intel(R) Xeon(R) CPU           X5675  @ 3.07GHz (3059.07-MHz K8-class CPU)
>    Origin="GenuineIntel"  Id=0x206c2  Family=0x6  Model=0x2c  Stepping=2
>    Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
>    Features2=0x29ee3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,POPCNT,AESNI>
>    AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
>    AMD Features2=0x1<LAHF>
>    VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
>    TSC: P-state invariant, performance statistics
>

Rodney,

Thanks for that and that test does confirm the DMAP table.

Built xen from source yesterday, which pulls in a gcc build and
loads of warnings during that and the build of various other
packages, but when rebooting, am getting a stack underflow at
the boot screen. Probably a typo in one of the config files,
but they look ok, so include the config files below. Haven't
defined any networking yet, as gettng the bare kernel
running must be the first step. A  bit like bringing up a
an embedded system board from bare metal, though far more
complex.

Building from source and getting the same error suggests it's
nothing out of sync, packages etc, put the disk aside and
reinstalled the os and xen from scratch on another disk, to
get a fresh start.  It's a slightly later iso image,
31-July-2017. Still panics, with the following boot log via
the serial port com 1. Sorry, lots of it, some line lengths
edited:-

-----------------------------------------------------------
ProLiant System BIOS - P67 (01/30/2011)
Copyright 1982, 2011 Hewlett-Packard Development Company, L.P.

32 GB Installed 1 Processor(s) detected, 6 total cores enabled,
Hyperthreading is disabled
Proc 1: Intel(R) Xeon(R) CPU E5645 @ 2.40GHz
QPI Speed: 5.8 GT/s
HP Power Profile Mode: Balanced Power and Performance
Power Regulator Mode: Dynamic Power Savings
Advanced Memory Protection Mode: Advanced ECC Support
Redundant ROM Detected - This system contains a valid backup
system ROM.
Inlet Ambient Temperature: 13C/55F

<snipped>

Setup Utility
  Press "F10" key for System Maintenance Menu
  Press "F11" key for Default Boot Override Options
  Press "F12" key for Network Boot
  For access via BIOS Serial Console
  Press "ESC+9" for ROM-Based Setup Utility
  Press "ESC+0" for System Maintenance Menu
  Press "ESC+!" for Default Boot Override Options
  Press "ESC+@" for Network Boot
Attempting Boot From CD-ROM
Attempting Boot From Hard Drive (C:)-\|/- <etc>
Consoles: internal video/keyboard  BIOS drive A: is
disk0BIOS
drive C: is disk1BIOS drive D: is disk2BIOS drive
E: is disk3BIOS
637kB/3396796kB available memory

FreeBSD/x86 bootstrap loader, Revision 1.1(Fri Jul 21
02:03:08 UTC 2017
[hidden email])|/-\|/- <etc>
Loading /boot/defaults/loader.conf/-\|/- <etc>

<Beast Image>

stack underflow message ???

                       \|/-(XEN) Bad console= option '' ???
  Xen 4.7.2
(XEN) Xen version 4.7.2 (root@) (FreeBSD clang version 4.0.0
(tags/RELEASE_400/final 297347) (based on LLVM 4.0.0))
debug=n Thu Mar 15 18:36:39 UTC 2018
(XEN) Latest ChangeSet:
(XEN) Bootloader: FreeBSD Loader
(XEN) Command line: dom0_mem=2048M dom0_max_vcpus=6
dom0pvh=1 console=com1, vga com1=115200,8n1
guest_loglvl=all loglvl=all

(XEN) Video information:
(XEN)  VGA is text mode 80x25, font 8x16
(XEN)  VBE/DDC methods: V2; EDID transfer time: 1 seconds
(XEN) Disc information:
(XEN)  Found 3 MBR signatures
(XEN)  Found 3 EDD information structures
(XEN) Xen-e820 RAM map:
(XEN)  0000000000000000 - 000000000009f400 (usable)
(XEN)  000000000009f400 - 00000000000a0000 (reserved)
(XEN)  00000000000f0000 - 0000000000100000 (reserved)
(XEN)  0000000000100000 - 00000000cf62f000 (usable)
(XEN)  00000000cf62f000 - 00000000cf63c000 (ACPI data)
(XEN)  00000000cf63c000 - 00000000cf63d000 (usable)
(XEN)  00000000cf63d000 - 00000000d4000000 (reserved)
(XEN)  00000000fec00000 - 00000000fee10000 (reserved)
(XEN)  00000000ff800000 - 0000000100000000 (reserved)
(XEN)  0000000100000000 - 000000082ffff000 (usable)
(XEN) ACPI: RSDP 000F4F00, 0024 (r2 HP    )
(XEN) ACPI: XSDT CF630340, 00BC (r1 HP     ProLiant        2   Ò     162E)
(XEN) ACPI: FACP CF630440, 00F4 (r3 HP     ProLiant        2   Ò     162E)
(XEN) ACPI: DSDT CF630540, 20BD (r1 HP         DSDT        1 INTL 20030228)
(XEN) ACPI: FACS CF62F100, 0040
(XEN) ACPI: SPCR CF62F140, 0050 (r1 HP     SPCRRBSU        1   Ò     162E)
(XEN) ACPI: MCFG CF62F1C0, 003C (r1 HP     ProLiant        1             0)
(XEN) ACPI: HPET CF62F200, 0038 (r1 HP     ProLiant        2   Ò     162E)
(XEN) ACPI: FFFF CF62F240, 0064 (r2 HP     ProLiant        2   Ò     162E)
(XEN) ACPI: SPMI CF62F2C0, 0040 (r5 HP     ProLiant        1   Ò     162E)
(XEN) ACPI: ERST CF62F300, 01D0 (r1 HP     ProLiant        1   Ò     162E)
(XEN) ACPI: APIC CF62F500, 015E (r1 HP     ProLiant        2             0)
(XEN) ACPI: SRAT CF62F680, 0570 (r1 HP     Proliant        1   Ò     162E)
(XEN) ACPI: FFFF CF62FC00, 0176 (r1 HP     ProLiant        1   Ò     162E)
(XEN) ACPI: BERT CF62FD80, 0030 (r1 HP     ProLiant        1   Ò     162E)
(XEN) ACPI: HEST CF62FDC0, 00BC (r1 HP     ProLiant        1   Ò     162E)
(XEN) ACPI: DMAR CF62FE80, 0168 (r1 HP     ProLiant        1   Ò     162E)
(XEN) ACPI: SSDT CF632600, 0125 (r3     HP  CRSPCI0        2   HP        1)
(XEN) ACPI: SSDT CF632740, 0255 (r3     HP  riser1a        2 INTL 20061109)
(XEN) ACPI: SSDT CF6329C0, 025D (r3     HP  riser2a        2 INTL 20061109)
(XEN) ACPI: SSDT CF632C40, 03BB (r1     HP      pcc        1 INTL 20090625)
(XEN) ACPI: SSDT CF633000, 0377 (r1     HP     pmab        1 INTL 20090625)
(XEN) ACPI: SSDT CF633380, 105C (r1  INTEL PPM RCM         1 INTL 20061109)
(XEN) System RAM: 32757MB (33543992kB)
(XEN) SRAT: PXM 0 -> APIC 00 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 02 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 04 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 10 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 12 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 14 -> Node 0
(XEN) SRAT: Node 0 PXM 0 0-d0000000
(XEN) SRAT: Node 0 PXM 0 100000000-830000000
(XEN) NUMA: Using 20 for the hash shift.
(XEN) Domain heap initialised
(XEN) found SMP MP-table at 000f4f80
(XEN) DMI 2.7 present.
(XEN) Using APIC driver default
(XEN) ACPI: PM-Timer IO Port: 0x908
(XEN) ACPI: SLEEP INFO: pm1x_cnt[1:904,1:0], pm1x_evt[1:900,1:0]
(XEN) ACPI:             wakeup_vec[cf62f10c], vec_size[20]
(XEN) ACPI: Local APIC address 0xfee00000
(XEN) ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
(XEN) ACPI: LAPIC (acpi_id[0x10] lapic_id[0x20] disabled)
(XEN) ACPI: LAPIC (acpi_id[0x08] lapic_id[0x10] enabled)
(XEN) ACPI: LAPIC (acpi_id[0x18] lapic_id[0x30] disabled)
(XEN) ACPI: LAPIC (acpi_id[0x04] lapic_id[0x04] enabled)
(XEN) ACPI: LAPIC (acpi_id[0x14] lapic_id[0x24] disabled)
(XEN) ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x14] enabled)
(XEN) ACPI: LAPIC (acpi_id[0x1c] lapic_id[0x34] disabled)
(XEN) ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
(XEN) ACPI: LAPIC (acpi_id[0x12] lapic_id[0x22] disabled)
(XEN) ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x12] enabled)
(XEN) ACPI: LAPIC (acpi_id[0x1a] lapic_id[0x32] disabled)
(XEN) ACPI: LAPIC (acpi_id[0x06] lapic_id[0x06] disabled)
(XEN) ACPI: LAPIC (acpi_id[0x16] lapic_id[0x26] disabled)
(XEN) ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x16] disabled)
(XEN) ACPI: LAPIC (acpi_id[0x1e] lapic_id[0x36] disabled)
(XEN) ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] disabled)
(XEN) ACPI: LAPIC (acpi_id[0x11] lapic_id[0x21] disabled)
(XEN) ACPI: LAPIC (acpi_id[0x09] lapic_id[0x11] disabled)
(XEN) ACPI: LAPIC (acpi_id[0x19] lapic_id[0x31] disabled)
(XEN) ACPI: LAPIC (acpi_id[0x05] lapic_id[0x05] disabled)
(XEN) ACPI: LAPIC (acpi_id[0x15] lapic_id[0x25] disabled)
(XEN) ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x15] disabled)
(XEN) ACPI: LAPIC (acpi_id[0x1d] lapic_id[0x35] disabled)
(XEN) ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] disabled)
(XEN) ACPI: LAPIC (acpi_id[0x13] lapic_id[0x23] disabled)
(XEN) ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x13] disabled)
(XEN) ACPI: LAPIC (acpi_id[0x1b] lapic_id[0x33] disabled)
(XEN) ACPI: LAPIC (acpi_id[0x07] lapic_id[0x07] disabled)
(XEN) ACPI: LAPIC (acpi_id[0x17] lapic_id[0x27] disabled)
(XEN) ACPI: LAPIC (acpi_id[0x0f] lapic_id[0x17] disabled)
(XEN) ACPI: LAPIC (acpi_id[0x1f] lapic_id[0x37] disabled)
(XEN) ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
(XEN) ACPI: IOAPIC (id[0x08] address[0xfec00000] gsi_base[0])
(XEN) IOAPIC[0]: apic_id 8, version 32, address 0xfec00000, GSI 0-23
(XEN) ACPI: IOAPIC (id[0x00] address[0xfec80000] gsi_base[24])
(XEN) IOAPIC[1]: apic_id 0, version 32, address 0xfec80000, GSI 24-47
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 high edge)
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
(XEN) ACPI: IRQ0 used by override.
(XEN) ACPI: IRQ2 used by override.
(XEN) ACPI: IRQ9 used by override.
(XEN) Enabling APIC mode:  Flat.  Using 2 I/O APICs
(XEN) ACPI: HPET id: 0x8086a201 base: 0xfed00000
(XEN) Failed to get Error Log Address Range.
(XEN) HEST: Table parsing has been initialized
(XEN) Using ACPI (MADT) for SMP configuration information
(XEN) SMP: Allowing 32 CPUs (26 hotplug CPUs)
(XEN) IRQ limits: 48 GSI, 1120 MSI/MSI-X
(XEN) Intel machine check reporting enabled
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Detected 2399.372 MHz processor.
(XEN) Initing memory sharing.
(XEN) alt table ffff82d0802bf2f0 -> ffff82d0802bfc44
(XEN) PCI: MCFG configuration 0: base d0000000 segment 0000 buses 00 - 3f
(XEN) PCI: MCFG area at d0000000 reserved in E820
(XEN) PCI: Using MCFG for segment 0000 bus 00-3f
(XEN) [VT-D]Disabling IOMMU due to Intel 5500/5520/X58 Chipset errata
#47, #53
(XEN) I/O virtualisation disabled
(XEN) nr_sockets: 5
(XEN) Enabled directed EOI with ioapic_ack_old on!
(XEN) ENABLING IO-APIC IRQs
(XEN)  -> Using old ACK method
(XEN) ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1
(XEN) Platform timer is 14.318MHz HPET
(XEN) Allocated console ring of 64 KiB.
(XEN) mwait-idle: MWAIT substates: 0x1120
(XEN) mwait-idle: v0.4.1 model 0x2c
(XEN) mwait-idle: lapic_timer_reliable_states 0xffffffff
(XEN) VMX: Supported advanced features:
(XEN)  - APIC MMIO access virtualisation
(XEN)  - APIC TPR shadow
(XEN)  - Extended Page Tables (EPT)
(XEN)  - Virtual-Processor Identifiers (VPID)
(XEN)  - Virtual NMI
(XEN)  - MSR direct-access bitmap
(XEN)  - Unrestricted Guest
(XEN) HVM: ASIDs enabled.
(XEN) HVM: VMX enabled
(XEN) HVM: Hardware Assisted Paging (HAP) detected
(XEN) HVM: HAP page sizes: 4kB, 2MB, 1GB
(XEN) Brought up 6 CPUs
(XEN) ACPI sleep modes: S3
(XEN) VPMU: disabled
(XEN) mcheck_poll: Machine check polling timer started.
(XEN) Dom0 has maximum 624 PIRQs
(XEN) NX (Execute Disable) protection active
(XEN) *** LOADING DOMAIN 0 ***
(XEN)  Xen  kernel: 64-bit, lsb, compat32
(XEN)  Dom0 kernel: 64-bit, PAE, lsb, paddr 0x200000 -> 0x1e90da0
(XEN)  Dom0 symbol map 0x1e90da0 -> 0x2167b80
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN)  Dom0 alloc.:   0000000814000000->0000000818000000 (507901 pages
to be allocated)
(XEN)  Init. ramdisk: 000000082fdfd000->000000082fe00000
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN)  Loaded kernel: ffffffff80200000->ffffffff82167b80
(XEN)  Init. ramdisk: ffffffff82168000->ffffffff8216b000
(XEN)  Phys-Mach map: ffffffff8216b000->ffffffff8256b000
(XEN)  Start info:    ffffffff8256b000->ffffffff8256c4b4
(XEN)  Page tables:   ffffffff8256d000->ffffffff82584000
(XEN)  Boot stack:    ffffffff82584000->ffffffff82585000
(XEN)  TOTAL:         ffffffff80000000->ffffffff82800000
(XEN)  ENTRY ADDRESS: ffffffff80ec8000
(XEN) Dom0 has maximum 6 VCPUs
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) Presently, iommu must be enabled for PVH hardware domain
(XEN)
(XEN) ****************************************
(XEN)
(XEN) Reboot in five seconds...

Included kernel debugging with this install. No idea how to
drive that though.

Hope this is some use and thanks again for the help with this...

Regards,

Chris


_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: [Bug 224003] xen kernel panics

Chris-2
In reply to this post by Rodney W. Grimes-4

Here are all the config files:


### /etc/sysctl.conf:

#security.bsd.see_other_uids=0
kern.randompid=3854
#
# Xen Kernel
vm.max_wired=-1

### /etc/login.conf

<snipped>

  ~/bin:\
        :nologin=/var/run/nologin:\
        :cputime=unlimited:\
        :datasize=unlimited:\
        :stacksize=unlimited:\
        :memorylocked=unlimited:\
        :memoryuse=unlimited:\
        :filesize=unlimited:\
        :coredumpsize=unlimited:\
        :openfiles=unlimited:\
        :maxproc=unlimited:\
        :sbsize=unlimited:\
        :vmemoryuse=unlimited:\
        :swapuse=unlimited:\
        :pseudoterminals=unlimited:\
        :kqueues=unlimited:\
        :umtxp=unlimited:\
        :priority=0:\
        :ignoretime@:\
        :umask=022:
       
### /etc/ttys:

#
# Xen Console
# -----------
xc0     "/usr/libexec/getty Pc"         xterm   on secure
#

### /boot/menu.rc.local

#
# Xen kernel
try-include /boot/xen.4th

### /boot/loader.conf

#
# Xen Kernel
# ----------
hw.pci.mcfg=0
xen_kernel="/boot/xen"
xen_cmdline="dom0_mem=2048M dom0_max_vcpus=6 dom0pvh=1 console=com1, vga
com1=115200,8n1 guest_loglvl=all loglvl=all"

### /etc/rc/conf

#
# Local rc.conf, icq, 30-Jun-2017
#                     09-Mar-2018
# -------------------------------
hostname="freebsdxen"
#
# Network
ifconfig_bce0="inet 192.9.xxx.xxx netmask 255.255.255.0"
defaultrouter="192.9.xxx.xxx"
#
# Services
sshd_enable="YES"
ntpd_enable="YES"
ntpdate_enable="YES"
ntpd_sync_on_start="YES"
zfs_enable="YES"
powerd_enable="YES"
#
# Radeon Kernel Mode Driver
kld_list="radeonkms"
#
# Gui Services
moused_enable="YES"
dbus_enable="YES"
hald_enable="YES"
slim_enable="YES"
#
# Disabled Services
sendmail_enable="NONE"
#
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"
#
# Xen Configuration
# -----------------
xencommons_enable="yes"
# autobridge_interfaces="bridge0"
# autobridge_bridge0="igb0"


Regards,

Chris

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: [Bug 224003] xen kernel panics

Chris-2
In reply to this post by Rodney W. Grimes-4
On 03/31/18 18:11, Chris wrote:

> On 03/30/18 18:05, Rodney W. Grimes wrote:
>
>> I can affirm that a E56xx cpu should be very capable of supporting Xen.
>> CPU: Intel(R) Xeon(R) CPU X5675 @ 3.07GHz (3059.07-MHz K8-class CPU)
>> Origin="GenuineIntel" Id=0x206c2 Family=0x6 Model=0x2c Stepping=2
>> Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
>>
>> Features2=0x29ee3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,POPCNT,AESNI>
>>
>> AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
>> AMD Features2=0x1<LAHF>
>> VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
>> TSC: P-state invariant, performance statistics
>>

Looks like this might be the problem:-

> (XEN) [VT-D]Disabling IOMMU due to Intel 5500/5520/X58 Chipset errata
> #47, #53

Been in a rush today, but should have looked at that a bit more closely
before posting, as istr, saw that  mentioned as a bug eleswhere back
from 2015 or so....

Regards,

Chris
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: [Bug 224003] xen kernel panics

mail-4
Am 2018-03-31 20:24, schrieb Chris:

> On 03/31/18 18:11, Chris wrote:
>> On 03/30/18 18:05, Rodney W. Grimes wrote:
>>
>>> I can affirm that a E56xx cpu should be very capable of supporting
>>> Xen.
>>> CPU: Intel(R) Xeon(R) CPU X5675 @ 3.07GHz (3059.07-MHz K8-class CPU)
>>> Origin="GenuineIntel" Id=0x206c2 Family=0x6 Model=0x2c Stepping=2
>>> Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
>>>
>>> Features2=0x29ee3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,POPCNT,AESNI>
>>>
>>> AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
>>> AMD Features2=0x1<LAHF>
>>> VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
>>> TSC: P-state invariant, performance statistics
>>>
>
> Looks like this might be the problem:-
>
>> (XEN) [VT-D]Disabling IOMMU due to Intel 5500/5520/X58 Chipset errata
>> #47, #53
>
> Been in a rush today, but should have looked at that a bit more closely
> before posting, as istr, saw that  mentioned as a bug eleswhere back
> from 2015 or so....

i had the same problem with an older poweredge r410, just for fun (and
profit) i patched this and xen started working like expected. it has
been tested just for a short time,  but booting vms was possible. i am
sure that i figured out which limitation this had but i cant remember

see my notes at gist

https://gist.github.com/jserviceorg/e8479c6cc1b485cacf21682ebb22bd27

>
> Regards,
>
> Chris
> _______________________________________________
> [hidden email] mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-xen
> To unsubscribe, send any mail to "[hidden email]"
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: [Bug 224003] xen kernel panics

Rodney W. Grimes-4
In reply to this post by Chris-2
> On 03/31/18 18:11, Chris wrote:
> > On 03/30/18 18:05, Rodney W. Grimes wrote:
> >
> >> I can affirm that a E56xx cpu should be very capable of supporting Xen.
> >> CPU: Intel(R) Xeon(R) CPU X5675 @ 3.07GHz (3059.07-MHz K8-class CPU)
> >> Origin="GenuineIntel" Id=0x206c2 Family=0x6 Model=0x2c Stepping=2
> >> Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
> >>
> >> Features2=0x29ee3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,POPCNT,AESNI>
> >>
> >> AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
> >> AMD Features2=0x1<LAHF>
> >> VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
> >> TSC: P-state invariant, performance statistics
> >>
>
> Looks like this might be the problem:-
>
> > (XEN) [VT-D]Disabling IOMMU due to Intel 5500/5520/X58 Chipset errata
> > #47, #53
>
> Been in a rush today, but should have looked at that a bit more closely
> before posting, as istr, saw that  mentioned as a bug eleswhere back
> from 2015 or so....

https://support.citrix.com/article/CTX136517

That has some info on how to work around this issue.  You can turn
of the interrupt remapping and still have an iommu, though it is
not going be very effective for passthrough of devices.

Its linux centric, and I am not quiet sure how to apply those
knobs to loader.conf to make xen do the right thing, but it
should be possible.


--
Rod Grimes                                                 [hidden email]
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: [Bug 224003] xen kernel panics

Roger Pau Monné
On Sat, Mar 31, 2018 at 07:38:28PM -0700, Rodney W. Grimes wrote:

> > On 03/31/18 18:11, Chris wrote:
> > > On 03/30/18 18:05, Rodney W. Grimes wrote:
> > >
> > >> I can affirm that a E56xx cpu should be very capable of supporting Xen.
> > >> CPU: Intel(R) Xeon(R) CPU X5675 @ 3.07GHz (3059.07-MHz K8-class CPU)
> > >> Origin="GenuineIntel" Id=0x206c2 Family=0x6 Model=0x2c Stepping=2
> > >> Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
> > >>
> > >> Features2=0x29ee3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,POPCNT,AESNI>
> > >>
> > >> AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
> > >> AMD Features2=0x1<LAHF>
> > >> VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
> > >> TSC: P-state invariant, performance statistics
> > >>
> >
> > Looks like this might be the problem:-
> >
> > > (XEN) [VT-D]Disabling IOMMU due to Intel 5500/5520/X58 Chipset errata
> > > #47, #53
> >
> > Been in a rush today, but should have looked at that a bit more closely
> > before posting, as istr, saw that  mentioned as a bug eleswhere back
> > from 2015 or so....
>
> https://support.citrix.com/article/CTX136517
>
> That has some info on how to work around this issue.  You can turn
> of the interrupt remapping and still have an iommu, though it is
> not going be very effective for passthrough of devices.

Just add iommu=no-intremap to your xen_cmdline in loader.conf.

Roger.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "[hidden email]"