Linux domU only works with xen_platform_pci=0 ?

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Linux domU only works with xen_platform_pci=0 ?

Kai Otto-2
Hello,

I'm trying to set up a FreeBSD 11.1 system as Xen virtualization host.
Following a combination of handbook [1] and wiki [2], I was able to get
get a FreeBSD dom0 in PVH mode, and FreeBSD domU in HVM mode running,
including installation to disk and networking.

It seemed to me as if the switch 'xen_platform_pci' doesn't have an
effect on FreeBSD domU's, as I see e.g. a xenpci0 and xbd0 in dmesg no
matter if I set it to 1 or 0.

Do I understand it correctly, that this switch makes the difference
between HVM and PVHVM mode?
According to xl.cfg(5), it 'enables a guest Operating System [...] to
make use of paravirtualization features such as disk and network devices'.


Afterwards, I tried to create a Linux domU. Both Centos 7 and Alpine
Linux only detected the harddisk with xen_platform_pci=0.

For Alpine Linux, with xen_platform_pci=1, I get the following messages
on the console:

vbd vbd-5632: 19 xenbus_dev_probe on device/vbd/5632
vbd vbd-5632: failed to write error node for device/vbd/5632 (19
xenbus_dev_probe on device/vbd/5632)

After waiting for a couple minutes it boots, but doesn't detect the disk.

Looking in /var/log/xen/qemu-dm-alpine-hvm.log, I see the following
messages:

xen be core: can't open gnttab device
xen be core: can't open gnttab device
xen be: vkbd-0: initalize() failed
xen be: vkbd-0: initalize() failed
xen be: vkbd-0: initalize() failed



Googling around yielded [3], where someone apparently ran into the same
problem, and used xen_platform_pci=0 as workaround.



So my question is:
Is xen_platform_pci=1 required for PVHVM mode?
If yes, is PVHVM mode only available for FreeBSD domU's?
If no, how can I enable it for Linux guests?


Any response would be appreciated, thanks!

(Unrelated: I also couldn't get PVH guests to boot, neither FreeBSD nor
Linux. The console stayed blank and CPU went to 100%. But that's a whole
other story I guess.)

- Kai




My system:

OS: FreeBSD-11.1-RELEASE amd64
xen installed via pkg
Processor: E3-1240 v2, (which supports VT-x, VT-d and EPT)


My alpine.cfg:

builder="hvm"
name="alpine-hvm"
memory=1024
vcpus=1
vif = [ 'bridge=bridge0' ]
disk = [ '/dev/zvol/zbulk/vm/alpine-hvm_disk0,raw,hda,rw',
 '/zbulk/mediashare/isos/linux/apline-virt-3.7.0-x86_64.iso,raw,hdb:cdrom,r']

xen_platform_pci=1 # or 0, see above
vnc = 1
vnclisten = "0.0.0.0"
serial = "pty"
usbdevice = "tablet"

--

[1]
https://www.freebsd.org/doc/en/books/handbook/virtualization-host-xen.html
[2] https://wiki.freebsd.org/Xen
[3]
https://www.reddit.com/r/freebsd/comments/6uldys/freebsd_111_linux_domu_on_xen_wont_boot/
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Linux domU only works with xen_platform_pci=0 ?

Roger Pau Monné-2
On Sun, May 13, 2018 at 03:51:36PM +0200, Kai Otto wrote:

> Hello,
>
> I'm trying to set up a FreeBSD 11.1 system as Xen virtualization host.
> Following a combination of handbook [1] and wiki [2], I was able to get
> get a FreeBSD dom0 in PVH mode, and FreeBSD domU in HVM mode running,
> including installation to disk and networking.
>
> It seemed to me as if the switch 'xen_platform_pci' doesn't have an
> effect on FreeBSD domU's, as I see e.g. a xenpci0 and xbd0 in dmesg no
> matter if I set it to 1 or 0.
>
> Do I understand it correctly, that this switch makes the difference
> between HVM and PVHVM mode?
> According to xl.cfg(5), it 'enables a guest Operating System [...] to
> make use of paravirtualization features such as disk and network devices'.
>
>
> Afterwards, I tried to create a Linux domU. Both Centos 7 and Alpine
> Linux only detected the harddisk with xen_platform_pci=0.
>
> For Alpine Linux, with xen_platform_pci=1, I get the following messages
> on the console:
>
> vbd vbd-5632: 19 xenbus_dev_probe on device/vbd/5632

That's ENODEV IIRC, I think there's something wrong with FreeBSD disk
backend.

> vbd vbd-5632: failed to write error node for device/vbd/5632 (19
> xenbus_dev_probe on device/vbd/5632)
>
> After waiting for a couple minutes it boots, but doesn't detect the disk.
>
> Looking in /var/log/xen/qemu-dm-alpine-hvm.log, I see the following
> messages:
>
> xen be core: can't open gnttab device
> xen be core: can't open gnttab device
> xen be: vkbd-0: initalize() failed
> xen be: vkbd-0: initalize() failed
> xen be: vkbd-0: initalize() failed
>
>
>
> Googling around yielded [3], where someone apparently ran into the same
> problem, and used xen_platform_pci=0 as workaround.
>
>
>
> So my question is:
> Is xen_platform_pci=1 required for PVHVM mode?

No, it shouldn't be.

> If yes, is PVHVM mode only available for FreeBSD domU's?
> If no, how can I enable it for Linux guests?

If you boot FreeBSD Dom0 kernel with boot_verbose=YES (in
/boot/loader.conf), can you paste the messages you get when trying to
boot the PVHVM guest with xen_platform_pci=1?

Also, can you paste the output of `xenstore-ls -fp` executed on Dom0
after the PVHVM guest has failed to boot but is still running?

> Any response would be appreciated, thanks!
>
> (Unrelated: I also couldn't get PVH guests to boot, neither FreeBSD nor
> Linux. The console stayed blank and CPU went to 100%. But that's a whole
> other story I guess.)

FreeBSD Dom0 runs in PVH mode, so you are indeed able to boot at least
one PVH guest :).

Let's look into the PVHVM issue first.

> My system:
>
> OS: FreeBSD-11.1-RELEASE amd64
> xen installed via pkg
> Processor: E3-1240 v2, (which supports VT-x, VT-d and EPT)
>
>
> My alpine.cfg:
>
> builder="hvm"
> name="alpine-hvm"
> memory=1024
> vcpus=1
> vif = [ 'bridge=bridge0' ]
> disk = [ '/dev/zvol/zbulk/vm/alpine-hvm_disk0,raw,hda,rw',
>  '/zbulk/mediashare/isos/linux/apline-virt-3.7.0-x86_64.iso,raw,hdb:cdrom,r']
>
> xen_platform_pci=1 # or 0, see above
> vnc = 1
> vnclisten = "0.0.0.0"
> serial = "pty"
> usbdevice = "tablet"

Your domain config file seems OK to me.

Roger.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Linux domU only works with xen_platform_pci=0 ?

Marcin Cieslak-3
In reply to this post by Kai Otto-2
On Sun, 13 May 2018, Kai Otto wrote:

> Hello,
>
> Linux only detected the harddisk with xen_platform_pci=0.
>
> For Alpine Linux, with xen_platform_pci=1, I get the following messages
> on the console:
>
> vbd vbd-5632: 19 xenbus_dev_probe on device/vbd/5632
> vbd vbd-5632: failed to write error node for device/vbd/5632 (19
> xenbus_dev_probe on device/vbd/5632)
I have reported this here, too:

https://lists.freebsd.org/pipermail/freebsd-xen/2017-August/003079.html

I think it is needed for some newer Linux kernels only.

I am successfully using Windows, Solaris and some other wierd HVM
guests without this option successfully. xen_platform_pci=1
seems to confuse Linux 4.* line (2.6.32 boots fine).

Marcin

smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Linux domU only works with xen_platform_pci=0 ?

Kai Otto-2
In reply to this post by Roger Pau Monné-2
On 13/05/18 17:16, Roger Pau Monné wrote:

> On Sun, May 13, 2018 at 03:51:36PM +0200, Kai Otto wrote:
>> Hello,
>>
>> I'm trying to set up a FreeBSD 11.1 system as Xen virtualization host.
>> Following a combination of handbook [1] and wiki [2], I was able to get
>> get a FreeBSD dom0 in PVH mode, and FreeBSD domU in HVM mode running,
>> including installation to disk and networking.
>>
>> It seemed to me as if the switch 'xen_platform_pci' doesn't have an
>> effect on FreeBSD domU's, as I see e.g. a xenpci0 and xbd0 in dmesg no
>> matter if I set it to 1 or 0.
>>
>> Do I understand it correctly, that this switch makes the difference
>> between HVM and PVHVM mode?
>> According to xl.cfg(5), it 'enables a guest Operating System [...] to
>> make use of paravirtualization features such as disk and network devices'.
>>
>>
>> Afterwards, I tried to create a Linux domU. Both Centos 7 and Alpine
>> Linux only detected the harddisk with xen_platform_pci=0.
>>
>> For Alpine Linux, with xen_platform_pci=1, I get the following messages
>> on the console:
>>
>> vbd vbd-5632: 19 xenbus_dev_probe on device/vbd/5632
> That's ENODEV IIRC, I think there's something wrong with FreeBSD disk
> backend.
>
>> vbd vbd-5632: failed to write error node for device/vbd/5632 (19
>> xenbus_dev_probe on device/vbd/5632)
>>
>> After waiting for a couple minutes it boots, but doesn't detect the disk.
>>
>> Looking in /var/log/xen/qemu-dm-alpine-hvm.log, I see the following
>> messages:
>>
>> xen be core: can't open gnttab device
>> xen be core: can't open gnttab device
>> xen be: vkbd-0: initalize() failed
>> xen be: vkbd-0: initalize() failed
>> xen be: vkbd-0: initalize() failed
>>
>>
>>
>> Googling around yielded [3], where someone apparently ran into the same
>> problem, and used xen_platform_pci=0 as workaround.
>>
>>
>>
>> So my question is:
>> Is xen_platform_pci=1 required for PVHVM mode?
> No, it shouldn't be.
>
>> If yes, is PVHVM mode only available for FreeBSD domU's?
>> If no, how can I enable it for Linux guests?
> If you boot FreeBSD Dom0 kernel with boot_verbose=YES (in
> /boot/loader.conf), can you paste the messages you get when trying to
> boot the PVHVM guest with xen_platform_pci=1?
>
> Also, can you paste the output of `xenstore-ls -fp` executed on Dom0
> after the PVHVM guest has failed to boot but is still running?
>
Hello again,
thanks for your response Roger!

I attached both. Because I don't know how the mailinglist handles
attachments, here is a pastebin of xenstore-ls:
https://pastebin.com/s7wa1ee7
And a direct paste of dmesg messages:

random: harvesting attach, 8 bytes (4 bits) from xbbd0
tap0: bpf attached
tap0: Ethernet address: 00:bd:ee:99:ff:00
tap0: link state changed to UP
tap0: changing name to 'xnb2.0-emu'
xnb(xnb_probe:1127): Claiming device 0, xnb
xnb2.0: bpf attached
xnb(xnb_attach:1271): Attaching to backend/vif/2/0
random: harvesting attach, 8 bytes (4 bits) from xnb0
xnb(xnb_frontend_changed:1395): frontend_state=Initialising,
xnb_state=InitWait
xnb2.0: link state changed to DOWN
xnb2.0: link state changed to UP
xnb2.0: link state changed to DOWN
xnb2.0: promiscuous mode enabled
xnb2.0: link state changed to UP
xnb2.0-emu: promiscuous mode enabled
nd6_dad_timer: cancel DAD on xnb2.0 because of ND6_IFF_IFDISABLED.
xnb(xnb_frontend_changed:1395): frontend_state=Connected, xnb_state=InitWait
xnb(xnb_connect_comms:788): rings connected!


- Kai

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "[hidden email]"

dmesg (1K) Download Attachment
xenstore-ls (18K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Linux domU only works with xen_platform_pci=0 ?

Roger Pau Monné-2
On Sun, May 13, 2018 at 07:33:03PM +0200, Kai Otto wrote:

> On 13/05/18 17:16, Roger Pau Monné wrote:
> > On Sun, May 13, 2018 at 03:51:36PM +0200, Kai Otto wrote:
> >> Hello,
> >>
> >> I'm trying to set up a FreeBSD 11.1 system as Xen virtualization host.
> >> Following a combination of handbook [1] and wiki [2], I was able to get
> >> get a FreeBSD dom0 in PVH mode, and FreeBSD domU in HVM mode running,
> >> including installation to disk and networking.
> >>
> >> It seemed to me as if the switch 'xen_platform_pci' doesn't have an
> >> effect on FreeBSD domU's, as I see e.g. a xenpci0 and xbd0 in dmesg no
> >> matter if I set it to 1 or 0.
> >>
> >> Do I understand it correctly, that this switch makes the difference
> >> between HVM and PVHVM mode?
> >> According to xl.cfg(5), it 'enables a guest Operating System [...] to
> >> make use of paravirtualization features such as disk and network devices'.
> >>
> >>
> >> Afterwards, I tried to create a Linux domU. Both Centos 7 and Alpine
> >> Linux only detected the harddisk with xen_platform_pci=0.
> >>
> >> For Alpine Linux, with xen_platform_pci=1, I get the following messages
> >> on the console:
> >>
> >> vbd vbd-5632: 19 xenbus_dev_probe on device/vbd/5632
> > That's ENODEV IIRC, I think there's something wrong with FreeBSD disk
> > backend.
> >
> >> vbd vbd-5632: failed to write error node for device/vbd/5632 (19
> >> xenbus_dev_probe on device/vbd/5632)
> >>
> >> After waiting for a couple minutes it boots, but doesn't detect the disk.
> >>
> >> Looking in /var/log/xen/qemu-dm-alpine-hvm.log, I see the following
> >> messages:
> >>
> >> xen be core: can't open gnttab device
> >> xen be core: can't open gnttab device
> >> xen be: vkbd-0: initalize() failed
> >> xen be: vkbd-0: initalize() failed
> >> xen be: vkbd-0: initalize() failed
> >>
> >>
> >>
> >> Googling around yielded [3], where someone apparently ran into the same
> >> problem, and used xen_platform_pci=0 as workaround.
> >>
> >>
> >>
> >> So my question is:
> >> Is xen_platform_pci=1 required for PVHVM mode?
> > No, it shouldn't be.
> >
> >> If yes, is PVHVM mode only available for FreeBSD domU's?
> >> If no, how can I enable it for Linux guests?
> > If you boot FreeBSD Dom0 kernel with boot_verbose=YES (in
> > /boot/loader.conf), can you paste the messages you get when trying to
> > boot the PVHVM guest with xen_platform_pci=1?
> >
> > Also, can you paste the output of `xenstore-ls -fp` executed on Dom0
> > after the PVHVM guest has failed to boot but is still running?
> >
> Hello again,
> thanks for your response Roger!
>
> I attached both. Because I don't know how the mailinglist handles
> attachments, here is a pastebin of xenstore-ls:
> https://pastebin.com/s7wa1ee7
> And a direct paste of dmesg messages:
>
> random: harvesting attach, 8 bytes (4 bits) from xbbd0
> tap0: bpf attached
> tap0: Ethernet address: 00:bd:ee:99:ff:00
> tap0: link state changed to UP
> tap0: changing name to 'xnb2.0-emu'
> xnb(xnb_probe:1127): Claiming device 0, xnb
> xnb2.0: bpf attached
> xnb(xnb_attach:1271): Attaching to backend/vif/2/0
> random: harvesting attach, 8 bytes (4 bits) from xnb0
> xnb(xnb_frontend_changed:1395): frontend_state=Initialising,
> xnb_state=InitWait
> xnb2.0: link state changed to DOWN
> xnb2.0: link state changed to UP
> xnb2.0: link state changed to DOWN
> xnb2.0: promiscuous mode enabled
> xnb2.0: link state changed to UP
> xnb2.0-emu: promiscuous mode enabled
> nd6_dad_timer: cancel DAD on xnb2.0 because of ND6_IFF_IFDISABLED.
> xnb(xnb_frontend_changed:1395): frontend_state=Connected, xnb_state=InitWait
> xnb(xnb_connect_comms:788): rings connected!

Can you please give a try to the following patch? I think it should
apply cleanly to stable/11 (11.1-RELEASE). You will have to rebuild
and install the kernel.


Thanks, Roger.
---8<---
diff --git a/sys/dev/xen/blkback/blkback.c b/sys/dev/xen/blkback/blkback.c
index 9744c8da73cb..e9c7afe7d18c 100644
--- a/sys/dev/xen/blkback/blkback.c
+++ b/sys/dev/xen/blkback/blkback.c
@@ -806,6 +806,9 @@ struct xbb_softc {
 
  /** Watch to wait for hotplug script execution */
  struct xs_watch  hotplug_watch;
+
+ /** Got the needed data from hotplug scripts? */
+ bool  hotplug_done;
 };
 
 /*---------------------------- Request Processing ----------------------------*/
@@ -3310,10 +3313,9 @@ xbb_connect(struct xbb_softc *xbb)
 {
  int error;
 
- if (xenbus_get_state(xbb->dev) != XenbusStateInitialised)
- return;
-
- if (xbb_collect_frontend_info(xbb) != 0)
+ if (!xbb->hotplug_done ||
+    (xenbus_get_state(xbb->dev) != XenbusStateInitWait) ||
+    (xbb_collect_frontend_info(xbb) != 0))
  return;
 
  xbb->flags &= ~XBBF_SHUTDOWN;
@@ -3412,6 +3414,7 @@ xbb_shutdown(struct xbb_softc *xbb)
  free(xbb->hotplug_watch.node, M_XENBLOCKBACK);
  xbb->hotplug_watch.node = NULL;
  }
+ xbb->hotplug_done = false;
 
  if (xenbus_get_state(xbb->dev) < XenbusStateClosing)
  xenbus_set_state(xbb->dev, XenbusStateClosing);
@@ -3692,8 +3695,7 @@ xbb_attach_disk(struct xs_watch *watch, const char **vec, unsigned int len)
  return;
  }
 
- /* Tell the front end that we are ready to connect. */
- xenbus_set_state(dev, XenbusStateInitialised);
+ xbb->hotplug_done = true;
 }
 
 /**
@@ -3757,6 +3759,7 @@ xbb_attach(device_t dev)
  * We need to wait for hotplug script execution before
  * moving forward.
  */
+ KASSERT(!xbb->hotplug_done, ("Hotplug scripts already executed"));
  watch_path = xs_join(xenbus_get_node(xbb->dev), "physical-device-path");
  xbb->hotplug_watch.callback_data = (uintptr_t)dev;
  xbb->hotplug_watch.callback = xbb_attach_disk;

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Linux domU only works with xen_platform_pci=0 ?

Nathan Friess
On 2018-05-14 07:04 AM, Roger Pau Monné wrote:

> On Sun, May 13, 2018 at 07:33:03PM +0200, Kai Otto wrote:
>> On 13/05/18 17:16, Roger Pau Monné wrote:
>>> On Sun, May 13, 2018 at 03:51:36PM +0200, Kai Otto wrote:
>>>> Hello,
>>>>
>>>> I'm trying to set up a FreeBSD 11.1 system as Xen virtualization host.
>>>> Following a combination of handbook [1] and wiki [2], I was able to get
>>>> get a FreeBSD dom0 in PVH mode, and FreeBSD domU in HVM mode running,
>>>> including installation to disk and networking.
>>>>
>>>> It seemed to me as if the switch 'xen_platform_pci' doesn't have an
>>>> effect on FreeBSD domU's, as I see e.g. a xenpci0 and xbd0 in dmesg no
>>>> matter if I set it to 1 or 0.
>>>>
>>>> Do I understand it correctly, that this switch makes the difference
>>>> between HVM and PVHVM mode?
>>>> According to xl.cfg(5), it 'enables a guest Operating System [...] to
>>>> make use of paravirtualization features such as disk and network devices'.
>>>>
>>>>
>>>> Afterwards, I tried to create a Linux domU. Both Centos 7 and Alpine
>>>> Linux only detected the harddisk with xen_platform_pci=0.
>>>>
>>>> For Alpine Linux, with xen_platform_pci=1, I get the following messages
>>>> on the console:
>>>>
>>>> vbd vbd-5632: 19 xenbus_dev_probe on device/vbd/5632
>>> That's ENODEV IIRC, I think there's something wrong with FreeBSD disk
>>> backend.
>>>
>>>> vbd vbd-5632: failed to write error node for device/vbd/5632 (19
>>>> xenbus_dev_probe on device/vbd/5632)
>>>>
>>>> After waiting for a couple minutes it boots, but doesn't detect the disk.
I had similar issues with Linux domUs being unable to detect their disks
when FreeBSD 11.1-RELEASE is the backend:

https://lists.freebsd.org/pipermail/freebsd-xen/2016-December/002924.html


What seems to be happening is that on my system the frontend and backend
may go from state InitWait to Initialised in different orders and so
either end may end up getting stuck waiting for the other side to change
state when the other side already has done so.

I have been running with the attached patch since my last message above
and Linux domUs have been working perfectly since then.  I realize that
the call to pause() is not the correct solution but it demonstrates that
some fine tuning of how the states are handled will help.

Nathan

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "[hidden email]"

missed-notifications.patch (925 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Linux domU only works with xen_platform_pci=0 ?

Roger Pau Monné-2
On Mon, May 14, 2018 at 08:34:54PM -0600, Nathan Friess wrote:

> On 2018-05-14 07:04 AM, Roger Pau Monné wrote:
> > On Sun, May 13, 2018 at 07:33:03PM +0200, Kai Otto wrote:
> > > On 13/05/18 17:16, Roger Pau Monné wrote:
> > > > On Sun, May 13, 2018 at 03:51:36PM +0200, Kai Otto wrote:
> > > > > Hello,
> > > > >
> > > > > I'm trying to set up a FreeBSD 11.1 system as Xen virtualization host.
> > > > > Following a combination of handbook [1] and wiki [2], I was able to get
> > > > > get a FreeBSD dom0 in PVH mode, and FreeBSD domU in HVM mode running,
> > > > > including installation to disk and networking.
> > > > >
> > > > > It seemed to me as if the switch 'xen_platform_pci' doesn't have an
> > > > > effect on FreeBSD domU's, as I see e.g. a xenpci0 and xbd0 in dmesg no
> > > > > matter if I set it to 1 or 0.
> > > > >
> > > > > Do I understand it correctly, that this switch makes the difference
> > > > > between HVM and PVHVM mode?
> > > > > According to xl.cfg(5), it 'enables a guest Operating System [...] to
> > > > > make use of paravirtualization features such as disk and network devices'.
> > > > >
> > > > >
> > > > > Afterwards, I tried to create a Linux domU. Both Centos 7 and Alpine
> > > > > Linux only detected the harddisk with xen_platform_pci=0.
> > > > >
> > > > > For Alpine Linux, with xen_platform_pci=1, I get the following messages
> > > > > on the console:
> > > > >
> > > > > vbd vbd-5632: 19 xenbus_dev_probe on device/vbd/5632
> > > > That's ENODEV IIRC, I think there's something wrong with FreeBSD disk
> > > > backend.
> > > >
> > > > > vbd vbd-5632: failed to write error node for device/vbd/5632 (19
> > > > > xenbus_dev_probe on device/vbd/5632)
> > > > >
> > > > > After waiting for a couple minutes it boots, but doesn't detect the disk.
>
> I had similar issues with Linux domUs being unable to detect their disks
> when FreeBSD 11.1-RELEASE is the backend:
>
> https://lists.freebsd.org/pipermail/freebsd-xen/2016-December/002924.html
>
>
> What seems to be happening is that on my system the frontend and backend may
> go from state InitWait to Initialised in different orders and so either end
> may end up getting stuck waiting for the other side to change state when the
> other side already has done so.
>
> I have been running with the attached patch since my last message above and
> Linux domUs have been working perfectly since then.  I realize that the call
> to pause() is not the correct solution but it demonstrates that some fine
> tuning of how the states are handled will help.

Can you please give a try to the patch at:

https://lists.freebsd.org/pipermail/freebsd-xen/2018-May/003132.html

I think this is the proper way to solve the issue, and I would like to
commit it ASAP in order to MFC it to stable-11 before 11.2 is
released, but it could benefit from some more testing.

Thanks, Roger.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Linux domU only works with xen_platform_pci=0 ?

Nathan Friess
On 2018-05-15 02:08 AM, Roger Pau Monné wrote:

> On Mon, May 14, 2018 at 08:34:54PM -0600, Nathan Friess wrote:
>>
>> I had similar issues with Linux domUs being unable to detect their disks
>> when FreeBSD 11.1-RELEASE is the backend:
>>
>> https://lists.freebsd.org/pipermail/freebsd-xen/2016-December/002924.html
>>
>>
>> What seems to be happening is that on my system the frontend and backend may
>> go from state InitWait to Initialised in different orders and so either end
>> may end up getting stuck waiting for the other side to change state when the
>> other side already has done so.
>>
>> I have been running with the attached patch since my last message above and
>> Linux domUs have been working perfectly since then.  I realize that the call
>> to pause() is not the correct solution but it demonstrates that some fine
>> tuning of how the states are handled will help.
>
> Can you please give a try to the patch at:
>
> https://lists.freebsd.org/pipermail/freebsd-xen/2018-May/003132.html
>
> I think this is the proper way to solve the issue, and I would like to
> commit it ASAP in order to MFC it to stable-11 before 11.2 is
> released, but it could benefit from some more testing.
>
> Thanks, Roger.
>


I tried the patch on and I my Linux domU did was not able to complete
the attachment to the FreeBSD backend.  I applied the patch to the
11-RELEASE kernel. (I couldn't get 11-STABLE to compile.)

With xen_platform_pci=0 the frontend and backend vbds were both stuck in
state 1.  With xen_platform_pci=1 the frontend was in state 1 and
backend in state 3.

This second case is one that I think was happening before and why I
added the pause as a hack.  Things seems to work best if both ends see
each other go into state 2 and then proceed from there.  Adding the
pause back in allows the frontend to go into state 3 but for some reason
the backend stays in state 2.

So the above patch is an improvement but it is still possible to miss
state changes.  I didn't have time last night to do full debugging to
see why the backend didn't move from state 2 to 3 in the last case.

Nathan
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Linux domU only works with xen_platform_pci=0 ?

Roger Pau Monné-2
On Thu, May 17, 2018 at 06:10:15PM -0600, Nathan Friess wrote:

> On 2018-05-15 02:08 AM, Roger Pau Monné wrote:
> > On Mon, May 14, 2018 at 08:34:54PM -0600, Nathan Friess wrote:
> > >
> > > I had similar issues with Linux domUs being unable to detect their disks
> > > when FreeBSD 11.1-RELEASE is the backend:
> > >
> > > https://lists.freebsd.org/pipermail/freebsd-xen/2016-December/002924.html
> > >
> > >
> > > What seems to be happening is that on my system the frontend and backend may
> > > go from state InitWait to Initialised in different orders and so either end
> > > may end up getting stuck waiting for the other side to change state when the
> > > other side already has done so.
> > >
> > > I have been running with the attached patch since my last message above and
> > > Linux domUs have been working perfectly since then.  I realize that the call
> > > to pause() is not the correct solution but it demonstrates that some fine
> > > tuning of how the states are handled will help.
> >
> > Can you please give a try to the patch at:
> >
> > https://lists.freebsd.org/pipermail/freebsd-xen/2018-May/003132.html
> >
> > I think this is the proper way to solve the issue, and I would like to
> > commit it ASAP in order to MFC it to stable-11 before 11.2 is
> > released, but it could benefit from some more testing.
> >
> > Thanks, Roger.
> >
>
>
> I tried the patch on and I my Linux domU did was not able to complete the
> attachment to the FreeBSD backend.  I applied the patch to the 11-RELEASE
> kernel. (I couldn't get 11-STABLE to compile.)
>
> With xen_platform_pci=0 the frontend and backend vbds were both stuck in
> state 1.  With xen_platform_pci=1 the frontend was in state 1 and backend in
> state 3.

Thanks for the testing! I however think you had some issue applying
the patch or building/installing the kernel, with the attached patch
the backend should never go into state 3.

Can you confirm that you have the patch correctly applied to the
kernel you are running?

And if so, can you try to debug why the backend goes into state 3?

FWIW, the patch works fine for me and I've been able to boot Debian
and Alpine Linux guests without having to add xen_platform_pci=0.

Thanks, Roger.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Linux domU only works with xen_platform_pci=0 ?

Nathan Friess
On 2018-05-19 02:02 AM, Roger Pau Monné wrote:

> On Thu, May 17, 2018 at 06:10:15PM -0600, Nathan Friess wrote:
>> I tried the patch on and I my Linux domU did was not able to complete the
>> attachment to the FreeBSD backend.  I applied the patch to the 11-RELEASE
>> kernel. (I couldn't get 11-STABLE to compile.)
>>
>> With xen_platform_pci=0 the frontend and backend vbds were both stuck in
>> state 1.  With xen_platform_pci=1 the frontend was in state 1 and backend in
>> state 3.
>
> Thanks for the testing! I however think you had some issue applying
> the patch or building/installing the kernel, with the attached patch
> the backend should never go into state 3.
>
> Can you confirm that you have the patch correctly applied to the
> kernel you are running?
>
> And if so, can you try to debug why the backend goes into state 3?
>
> FWIW, the patch works fine for me and I've been able to boot Debian
> and Alpine Linux guests without having to add xen_platform_pci=0.
>
> Thanks, Roger.
>
Sorry, I don't which run I was looking at there.  Here is the order of
events.  There are two zvols exposed to the domU,
/dev/zvol/zroot/vmachines/smyth/rootfs and .../swap, so the messages are
interspersed between the two.

xbb(xbb_attach:3738): Attaching to backend/vbd/10/51713
xbb(xbb_attach:3802): Attach done, set state InitWait
xbb(xbb_attach_disk:3612): Enter xbb_attach_disk
xbb(xbb_attach_disk:3620):   Read physical-device-path failed
xbb(xbb_frontend_changed:3918): frontend_state=Initialising,
xbb_state=InitWait
xbb(xbb_attach:3738): Attaching to backend/vbd/10/51714
xbb(xbb_attach:3802): Attach done, set state InitWait
xbb(xbb_attach_disk:3612): Enter xbb_attach_disk
xbb(xbb_attach_disk:3620):   Read physical-device-path failed
xbb(xbb_frontend_changed:3918): frontend_state=Initialising,
xbb_state=InitWait
(Shell script) Start block script /local/domain/9/backend/vbd/10/51714 add
(Shell script) Start block script /local/domain/9/backend/vbd/10/51713 add
[1] xbb(xbb_frontend_changed:3918): frontend_state=Initialised,
xbb_state=InitWait
xbb(xbb_connect:3316): Enter xbb_connect: hotplug=0, get_state=2,
collect_frontend=0
[1] xbb(xbb_frontend_changed:3918): frontend_state=Initialised,
xbb_state=InitWait
xbb(xbb_connect:3316): Enter xbb_connect: hotplug=0, get_state=2,
collect_frontend=0
(Shell script) Write /dev/zvol/zroot/vmachines/smyth/rootfs to
/local/domain/9/backend/vbd/10/51713/physical-device-path
xbb(xbb_attach_disk:3612): Enter xbb_attach_disk
xbb(xbb_open_backend:2687): opening
dev=/dev/zvol/zroot/vmachines/smyth/rootfs
xbb(xbb_open_backend:2757): opened
dev=/dev/zvol/zroot/vmachines/smyth/rootfs sector_size=512
media_size=21474836480
xbb(xbb_attach_disk:3718): Set hotplug done, attach_disk done
(Shell script) Write /dev/zvol/zroot/vmachines/smyth/swap to
/local/domain/9/backend/vbd/10/51714/physical-device-path
xbb(xbb_attach_disk:3612): Enter xbb_attach_disk
xbb(xbb_open_backend:2687): opening dev=/dev/zvol/zroot/vmachines/smyth/swap
xbb(xbb_open_backend:2757): opened
dev=/dev/zvol/zroot/vmachines/smyth/swap sector_size=512
media_size=2147483648
xbb(xbb_attach_disk:3718): Set hotplug done, attach_disk done

... is stuck here...

At [1] the frontend has transitioned to Initialised but the block script
has not completed yet so hotplug_done is 0 and the notification about
the frontend status change is lost.  Calling xbb_connect at the end of
xbb_attach_disk like in the attached patch will catch this case. Now
with your recent change and the attached file I can use Linux domUs
successfully!


On a related note, did these patches ever make it into 11-stable?  I
don't see them at svn.freebsd.org/base/stable/11 but I may have missed
something.

https://lists.freebsd.org/pipermail/freebsd-xen/2016-December/002918.html

There were 4 patches, the third one being required for xl devd to
trigger block scripts.  I have been using them as well for over a year
with no issues:

https://lists.freebsd.org/pipermail/freebsd-xen/2016-December/002918.html

Thanks,

Nathan

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "[hidden email]"

missed-watch2.patch (365 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Linux domU only works with xen_platform_pci=0 ?

Roger Pau Monné-2
On Mon, May 21, 2018 at 01:51:13PM -0600, Nathan Friess wrote:

> On 2018-05-19 02:02 AM, Roger Pau Monné wrote:
> > On Thu, May 17, 2018 at 06:10:15PM -0600, Nathan Friess wrote:
> > > I tried the patch on and I my Linux domU did was not able to complete the
> > > attachment to the FreeBSD backend.  I applied the patch to the 11-RELEASE
> > > kernel. (I couldn't get 11-STABLE to compile.)
> > >
> > > With xen_platform_pci=0 the frontend and backend vbds were both stuck in
> > > state 1.  With xen_platform_pci=1 the frontend was in state 1 and backend in
> > > state 3.
> >
> > Thanks for the testing! I however think you had some issue applying
> > the patch or building/installing the kernel, with the attached patch
> > the backend should never go into state 3.
> >
> > Can you confirm that you have the patch correctly applied to the
> > kernel you are running?
> >
> > And if so, can you try to debug why the backend goes into state 3?
> >
> > FWIW, the patch works fine for me and I've been able to boot Debian
> > and Alpine Linux guests without having to add xen_platform_pci=0.
> >
> > Thanks, Roger.
> >
>
> Sorry, I don't which run I was looking at there.  Here is the order of
> events.  There are two zvols exposed to the domU,
> /dev/zvol/zroot/vmachines/smyth/rootfs and .../swap, so the messages are
> interspersed between the two.
>
> xbb(xbb_attach:3738): Attaching to backend/vbd/10/51713
> xbb(xbb_attach:3802): Attach done, set state InitWait
> xbb(xbb_attach_disk:3612): Enter xbb_attach_disk
> xbb(xbb_attach_disk:3620):   Read physical-device-path failed
> xbb(xbb_frontend_changed:3918): frontend_state=Initialising,
> xbb_state=InitWait
> xbb(xbb_attach:3738): Attaching to backend/vbd/10/51714
> xbb(xbb_attach:3802): Attach done, set state InitWait
> xbb(xbb_attach_disk:3612): Enter xbb_attach_disk
> xbb(xbb_attach_disk:3620):   Read physical-device-path failed
> xbb(xbb_frontend_changed:3918): frontend_state=Initialising,
> xbb_state=InitWait
> (Shell script) Start block script /local/domain/9/backend/vbd/10/51714 add
> (Shell script) Start block script /local/domain/9/backend/vbd/10/51713 add
> [1] xbb(xbb_frontend_changed:3918): frontend_state=Initialised,
> xbb_state=InitWait
> xbb(xbb_connect:3316): Enter xbb_connect: hotplug=0, get_state=2,
> collect_frontend=0
> [1] xbb(xbb_frontend_changed:3918): frontend_state=Initialised,
> xbb_state=InitWait
> xbb(xbb_connect:3316): Enter xbb_connect: hotplug=0, get_state=2,
> collect_frontend=0
> (Shell script) Write /dev/zvol/zroot/vmachines/smyth/rootfs to
> /local/domain/9/backend/vbd/10/51713/physical-device-path
> xbb(xbb_attach_disk:3612): Enter xbb_attach_disk
> xbb(xbb_open_backend:2687): opening
> dev=/dev/zvol/zroot/vmachines/smyth/rootfs
> xbb(xbb_open_backend:2757): opened
> dev=/dev/zvol/zroot/vmachines/smyth/rootfs sector_size=512
> media_size=21474836480
> xbb(xbb_attach_disk:3718): Set hotplug done, attach_disk done
> (Shell script) Write /dev/zvol/zroot/vmachines/smyth/swap to
> /local/domain/9/backend/vbd/10/51714/physical-device-path
> xbb(xbb_attach_disk:3612): Enter xbb_attach_disk
> xbb(xbb_open_backend:2687): opening dev=/dev/zvol/zroot/vmachines/smyth/swap
> xbb(xbb_open_backend:2757): opened dev=/dev/zvol/zroot/vmachines/smyth/swap
> sector_size=512 media_size=2147483648
> xbb(xbb_attach_disk:3718): Set hotplug done, attach_disk done
>
> ... is stuck here...
>
> At [1] the frontend has transitioned to Initialised but the block script has
> not completed yet so hotplug_done is 0 and the notification about the
> frontend status change is lost.  Calling xbb_connect at the end of
> xbb_attach_disk like in the attached patch will catch this case. Now with
> your recent change and the attached file I can use Linux domUs successfully!

Your Linux kernel is certainly very fast to boot, so that it wins the
race with blkback processing the result from the hotplug script
execution.

I've now committed the fix to HEAD, will MFC in a week if everything
goes fine:

https://svnweb.freebsd.org/base?view=revision&revision=334027

TYVM for the testing and the fix.

> On a related note, did these patches ever make it into 11-stable?  I don't
> see them at svn.freebsd.org/base/stable/11 but I may have missed something.
>
> https://lists.freebsd.org/pipermail/freebsd-xen/2016-December/002918.html

I don't think they are on HEAD either?

I don't have a lot of time ATM, so if you want to pick the patches,
rebase them onto HEAD, test them and give me a branch I'm more than
happy to review and commit them.

> There were 4 patches, the third one being required for xl devd to trigger
> block scripts.  I have been using them as well for over a year with no
> issues:
>
> https://lists.freebsd.org/pipermail/freebsd-xen/2016-December/002918.html

The Xen side of this is already upstream AFAIK, see:

http://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=7ff99b232e0f91a5189f429498868bfccf8d7154

Thanks, Roger.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Linux domU only works with xen_platform_pci=0 ?

Nathan Friess
On 2018-05-22 03:01 AM, Roger Pau Monné wrote:

> On Mon, May 21, 2018 at 01:51:13PM -0600, Nathan Friess wrote:
>> On 2018-05-19 02:02 AM, Roger Pau Monné wrote:
>> On a related note, did these patches ever make it into 11-stable?  I don't
>> see them at svn.freebsd.org/base/stable/11 but I may have missed something.
>>
>> https://lists.freebsd.org/pipermail/freebsd-xen/2016-December/002918.html
>
> I don't think they are on HEAD either?
>
> I don't have a lot of time ATM, so if you want to pick the patches,
> rebase them onto HEAD, test them and give me a branch I'm more than
> happy to review and commit them.
>
Attached are 4 patches from your git repository, applied to HEAD with
some testing on an up-to-date 12-CURRENT install.  I did see one kernel
message that I assume is not fatal (more on that later).  I'm not able
to test running as dom0 right now because I don't have any newish
hardware available.  My main use case is FreeBSD as domU serving disks
to other domUs.

The 0001-0003 patches applied cleanly to HEAD.  0004 required moving a
function around to fix a compile error but otherwise is unchanged.  I
also copied your commit messages from git into the patches for reference.


I was going to attach a patch to fix xen-tools with iasl in 12-CURRENT,
but as I type this I see that you just committed a fix yesterday.
Thanks! :)


The kernel message that I'm seeing appears when a domU shuts down or a
disk is detached from a running domU, while the backend for the disk is
my 12-CURRENT test system.  The backend spits this out:


lock order reversal: (sleepable after non-sleepable)
  1st 0xfffff8000802fe90 xbbd1 (xbbd1) @
/usr/src/sys/dev/xen/blkback/blkback.c:3423
  2nd 0xffffffff81fdf890 intrsrc (intrsrc) @
/usr/src/sys/x86/x86/intr_machdep.c:224
stack backtrace:
#0 0xffffffff80bdd993 at witness_debugger+0x73
#1 0xffffffff80bdd814 at witness_checkorder+0xe34
#2 0xffffffff80b7d798 at _sx_xlock+0x68
#3 0xffffffff811b3913 at intr_remove_handler+0x43
#4 0xffffffff811c63ef at xen_intr_unbind+0x10f
#5 0xffffffff80a12ecf at xbb_disconnect+0x2f
#6 0xffffffff80a12e54 at xbb_shutdown+0x1e4
#7 0xffffffff80a10be4 at xbb_frontend_changed+0x54
#8 0xffffffff80ed66a4 at xenbusb_back_otherend_changed+0x14
#9 0xffffffff80a2a382 at xenwatch_thread+0x182
#10 0xffffffff80b34164 at fork_exit+0x84
#11 0xffffffff8101ec9e at fork_trampoline+0xe
lock order reversal: (sleepable after non-sleepable)
  1st 0xfffff80008030690 xbbd0 (xbbd0) @
/usr/src/sys/dev/xen/blkback/blkback.c:3423
  2nd 0xffffffff81fdf890 intrsrc (intrsrc) @
/usr/src/sys/x86/x86/intr_machdep.c:224
stack backtrace:
#0 0xffffffff80bdd993 at witness_debugger+0x73
#1 0xffffffff80bdd814 at witness_checkorder+0xe34
#2 0xffffffff80b7d798 at _sx_xlock+0x68
#3 0xffffffff811b3913 at intr_remove_handler+0x43
#4 0xffffffff811c63ef at xen_intr_unbind+0x10f
#5 0xffffffff80a12ecf at xbb_disconnect+0x2f
#6 0xffffffff80a12e54 at xbb_shutdown+0x1e4
#7 0xffffffff80a10be4 at xbb_frontend_changed+0x54
#8 0xffffffff80ed66a4 at xenbusb_back_otherend_changed+0x14
#9 0xffffffff80a2a382 at xenwatch_thread+0x182
#10 0xffffffff80b34164 at fork_exit+0x84
#11 0xffffffff8101ec9e at fork_trampoline+0xe


Otherwise disk backends seem to work fine.

Nathan

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "[hidden email]"

0001-xenstore-remove-the-suspend-sx-lock.patch (6K) Download Attachment
0002-xenstore-don-t-wait-with-the-PCATCH-flag.patch (1K) Download Attachment
0003-dev-xenstore-add-support-for-watches.patch (9K) Download Attachment
0004-xenstore-dev-prevent-user-space-xenstore-device-from.patch (2K) Download Attachment