ELF - panic on installworld

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

ELF - panic on installworld

Wolfgang Zenker-2
Hi,

I'm trying to run installworld using 11-STABLE on an Ubiquity Edge
Router Lite (mips64, 2 cores, 512 MB Ram). Unfortunately I haven't
managed to finish the installworld yet, I always get a
panic: kernel stack overflow - trapframe at 0xffffffff80917eb0
in slightly different places during the installworld. Of the 4 panics I
have seen on the serial console, 3 had the trapframe at 0xffffffff80917eb0
and one at 0xffffffff80915eb0
/usr/src and /usr/obj are nfs-mounted, and I have configured almost 2 GB
of swap. The build was done in a Qemu environment.

Any hints how to proceed from here?

Wolfgang
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-mips
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

ERL (was: ELF - panic on installworld)

Wolfgang Zenker-2
Sorry for the messup in the subject.

* Wolfgang Zenker <[hidden email]> [180305 22:16]:

> Hi,
>
> I'm trying to run installworld using 11-STABLE on an Ubiquity Edge
> Router Lite (mips64, 2 cores, 512 MB Ram). Unfortunately I haven't
> managed to finish the installworld yet, I always get a
> panic: kernel stack overflow - trapframe at 0xffffffff80917eb0
> in slightly different places during the installworld. Of the 4 panics I
> have seen on the serial console, 3 had the trapframe at 0xffffffff80917eb0
> and one at 0xffffffff80915eb0
> /usr/src and /usr/obj are nfs-mounted, and I have configured almost 2 GB
> of swap. The build was done in a Qemu environment.
>
> Any hints how to proceed from here?
>
> Wolfgang
> _______________________________________________
> [hidden email] mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-mips
> To unsubscribe, send any mail to "[hidden email]"
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-mips
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: ELF - panic on installworld

Eugene Grosbein-10
In reply to this post by Wolfgang Zenker-2
06.03.2018 4:16, Wolfgang Zenker пишет:

> Hi,
>
> I'm trying to run installworld using 11-STABLE on an Ubiquity Edge
> Router Lite (mips64, 2 cores, 512 MB Ram). Unfortunately I haven't
> managed to finish the installworld yet, I always get a
> panic: kernel stack overflow - trapframe at 0xffffffff80917eb0
> in slightly different places during the installworld. Of the 4 panics I
> have seen on the serial console, 3 had the trapframe at 0xffffffff80917eb0
> and one at 0xffffffff80915eb0
> /usr/src and /usr/obj are nfs-mounted, and I have configured almost 2 GB
> of swap. The build was done in a Qemu environment.
>
> Any hints how to proceed from here?

Try increasing kernel stack size from default 2 pages to 4 by rebuilding
the kernel with options KSTACK_PAGES=4


_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-mips
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: ERL (was: ELF - panic on installworld)

Alexander Kabaev-3
In reply to this post by Wolfgang Zenker-2
On Mon, 5 Mar 2018 22:45:28 +0100
Wolfgang Zenker <[hidden email]> wrote:

> Sorry for the messup in the subject.
>
> * Wolfgang Zenker <[hidden email]> [180305 22:16]:
> > Hi,
> >
> > I'm trying to run installworld using 11-STABLE on an Ubiquity Edge
> > Router Lite (mips64, 2 cores, 512 MB Ram). Unfortunately I haven't
> > managed to finish the installworld yet, I always get a
> > panic: kernel stack overflow - trapframe at 0xffffffff80917eb0
> > in slightly different places during the installworld. Of the 4
> > panics I have seen on the serial console, 3 had the trapframe at
> > 0xffffffff80917eb0 and one at 0xffffffff80915eb0
> > /usr/src and /usr/obj are nfs-mounted, and I have configured almost
> > 2 GB of swap. The build was done in a Qemu environment.
> >
> > Any hints how to proceed from here?
> >
> > Wolfgang
NFS is is known kernel stack hog. Try increasing KSTACK_PAGES up from
whatever defaults in your kernel config file. IIRC, mips defaults to 2
pages, which is not enough for your workload.

--
Alexander Kabaev

attachment0 (981 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: ELF - panic on installworld

Eugene Grosbein-10
In reply to this post by Eugene Grosbein-10
06.03.2018 8:27, Eugene Grosbein wrote:

> 06.03.2018 4:16, Wolfgang Zenker wrote:

>> I'm trying to run installworld using 11-STABLE on an Ubiquity Edge
>> Router Lite (mips64, 2 cores, 512 MB Ram). Unfortunately I haven't
>> managed to finish the installworld yet, I always get a
>> panic: kernel stack overflow - trapframe at 0xffffffff80917eb0
>> in slightly different places during the installworld. Of the 4 panics I
>> have seen on the serial console, 3 had the trapframe at 0xffffffff80917eb0
>> and one at 0xffffffff80915eb0
>> /usr/src and /usr/obj are nfs-mounted, and I have configured almost 2 GB
>> of swap. The build was done in a Qemu environment.
>>
>> Any hints how to proceed from here?
>
> Try increasing kernel stack size from default 2 pages to 4 by rebuilding
> the kernel with options KSTACK_PAGES=4

Note also, that depending on your network configuration, KSTACK_PAGES=4
may or may not be enough. If it does not help, you need to double it once more.

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-mips
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: ERL (was: ELF - panic on installworld)

Alexander Kabaev-3
In reply to this post by Alexander Kabaev-3
On Mon, 5 Mar 2018 20:30:42 -0500
Alexander Kabaev <[hidden email]> wrote:

> On Mon, 5 Mar 2018 22:45:28 +0100
> Wolfgang Zenker <[hidden email]> wrote:
>
> > Sorry for the messup in the subject.
> >
> > * Wolfgang Zenker <[hidden email]> [180305 22:16]:  
> > > Hi,
> > >
> > > I'm trying to run installworld using 11-STABLE on an Ubiquity Edge
> > > Router Lite (mips64, 2 cores, 512 MB Ram). Unfortunately I haven't
> > > managed to finish the installworld yet, I always get a
> > > panic: kernel stack overflow - trapframe at 0xffffffff80917eb0
> > > in slightly different places during the installworld. Of the 4
> > > panics I have seen on the serial console, 3 had the trapframe at
> > > 0xffffffff80917eb0 and one at 0xffffffff80915eb0
> > > /usr/src and /usr/obj are nfs-mounted, and I have configured
> > > almost 2 GB of swap. The build was done in a Qemu environment.
> > >
> > > Any hints how to proceed from here?
> > >
> > > Wolfgang  
>
> NFS is is known kernel stack hog. Try increasing KSTACK_PAGES up from
> whatever defaults in your kernel config file. IIRC, mips defaults to 2
> pages, which is not enough for your workload.
>
> --
> Alexander Kabaev
I got reminded privately that variable stack sizes do not work on MIPS
without extra patches:

 http://people.freebsd.org/~sson/mips/kstack/kstack_large_page.diff

Not sure if it can still be applied to -current.
--
Alexander Kabaev

attachment0 (981 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: ERL (was: ELF - panic on installworld)

Wolfgang Zenker-2
* Alexander Kabaev <[hidden email]> [180306 03:15]:
> On Mon, 5 Mar 2018 20:30:42 -0500
> Alexander Kabaev <[hidden email]> wrote:

>> On Mon, 5 Mar 2018 22:45:28 +0100
>> Wolfgang Zenker <[hidden email]> wrote:

>>> Sorry for the messup in the subject.

>>> * Wolfgang Zenker <[hidden email]> [180305 22:16]:  

>>>> I'm trying to run installworld using 11-STABLE on an Ubiquity Edge
>>>> Router Lite (mips64, 2 cores, 512 MB Ram). Unfortunately I haven't
>>>> managed to finish the installworld yet, I always get a
>>>> panic: kernel stack overflow - trapframe at 0xffffffff80917eb0
>>>> in slightly different places during the installworld. Of the 4
>>>> panics I have seen on the serial console, 3 had the trapframe at
>>>> 0xffffffff80917eb0 and one at 0xffffffff80915eb0
>>>> /usr/src and /usr/obj are nfs-mounted, and I have configured
>>>> almost 2 GB of swap. The build was done in a Qemu environment.

>>>> Any hints how to proceed from here?

>> NFS is is known kernel stack hog. Try increasing KSTACK_PAGES up from
>> whatever defaults in your kernel config file. IIRC, mips defaults to 2
>> pages, which is not enough for your workload.

> I got reminded privately that variable stack sizes do not work on MIPS
> without extra patches:

True; due to KSTACK_PAGES being set to 2 unconditionally in
sys/mips/include/param.h, buildkernel fails when KSTACK_PAGES is added
to the conf file. Adding an "#ifndef KSTACK_PAGES" to param.h allows
buildkernel to work, but the resulting kernel panics at boot.

>  http://people.freebsd.org/~sson/mips/kstack/kstack_large_page.diff

That looks like quite a bit more than just a simple increase of the
kernel stack space. I think I will go with updating by building media
images (and less frequent upgrades) instead.

> Not sure if it can still be applied to -current.

I will at least try it once on 11-STABLE, just to find out. Will take a
while, currently down with flu and not really that good with
concentrating on that task.

Wolfgang
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-mips
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: ELF - panic on installworld

John Baldwin
In reply to this post by Eugene Grosbein-10
On Tuesday, March 06, 2018 08:37:26 AM Eugene Grosbein wrote:

> 06.03.2018 8:27, Eugene Grosbein wrote:
>
> > 06.03.2018 4:16, Wolfgang Zenker wrote:
>
> >> I'm trying to run installworld using 11-STABLE on an Ubiquity Edge
> >> Router Lite (mips64, 2 cores, 512 MB Ram). Unfortunately I haven't
> >> managed to finish the installworld yet, I always get a
> >> panic: kernel stack overflow - trapframe at 0xffffffff80917eb0
> >> in slightly different places during the installworld. Of the 4 panics I
> >> have seen on the serial console, 3 had the trapframe at 0xffffffff80917eb0
> >> and one at 0xffffffff80915eb0
> >> /usr/src and /usr/obj are nfs-mounted, and I have configured almost 2 GB
> >> of swap. The build was done in a Qemu environment.
> >>
> >> Any hints how to proceed from here?
> >
> > Try increasing kernel stack size from default 2 pages to 4 by rebuilding
> > the kernel with options KSTACK_PAGES=4
>
> Note also, that depending on your network configuration, KSTACK_PAGES=4
> may or may not be enough. If it does not help, you need to double it once more.

KSTACK_PAGES doesn't work on MIPS because the MIPS kstack has to be hardwired
into the TLB and the code that does that assumes a hardcoded stack size.

--
John Baldwin
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-mips
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: ELF - panic on installworld

Juli Mallett-4
On 6 March 2018 at 10:29, John Baldwin <[hidden email]> wrote:

> On Tuesday, March 06, 2018 08:37:26 AM Eugene Grosbein wrote:
> > 06.03.2018 8:27, Eugene Grosbein wrote:
> >
> > > 06.03.2018 4:16, Wolfgang Zenker wrote:
> >
> > >> I'm trying to run installworld using 11-STABLE on an Ubiquity Edge
> > >> Router Lite (mips64, 2 cores, 512 MB Ram). Unfortunately I haven't
> > >> managed to finish the installworld yet, I always get a
> > >> panic: kernel stack overflow - trapframe at 0xffffffff80917eb0
> > >> in slightly different places during the installworld. Of the 4 panics
> I
> > >> have seen on the serial console, 3 had the trapframe at
> 0xffffffff80917eb0
> > >> and one at 0xffffffff80915eb0
> > >> /usr/src and /usr/obj are nfs-mounted, and I have configured almost 2
> GB
> > >> of swap. The build was done in a Qemu environment.
> > >>
> > >> Any hints how to proceed from here?
> > >
> > > Try increasing kernel stack size from default 2 pages to 4 by
> rebuilding
> > > the kernel with options KSTACK_PAGES=4
> >
> > Note also, that depending on your network configuration, KSTACK_PAGES=4
> > may or may not be enough. If it does not help, you need to double it
> once more.
>
> KSTACK_PAGES doesn't work on MIPS because the MIPS kstack has to be
> hardwired
> into the TLB and the code that does that assumes a hardcoded stack size.
>

That said, we could easily use a more flexible wired TLB entry scheme,
including smartly using pagemask in the cases where the number of pages is
suitable.  If we wanted to allow wiring of mappings into the TLB flexibly
at runtime we could do that, or we could just at compile-time have
different code to handle different KSTACK_PAGES values.  People have strong
feelings about some of those options, but if there's a workload-oriented
pressure to move in a different direction, it should be very easy to do.

Juli.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-mips
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: ELF - panic on installworld

Eugene Grosbein-10
07.03.2018 3:14, Juli Mallett wrote:

> That said, we could easily use a more flexible wired TLB entry scheme,
> including smartly using pagemask in the cases where the number of pages is
> suitable.  If we wanted to allow wiring of mappings into the TLB flexibly
> at runtime we could do that, or we could just at compile-time have
> different code to handle different KSTACK_PAGES values.  People have strong
> feelings about some of those options, but if there's a workload-oriented
> pressure to move in a different direction, it should be very easy to do.

I cannot understand what "feelings" may exist about kernel stack shortage leading to panics.


_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-mips
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: ELF - panic on installworld

Wolfgang Zenker-2
In reply to this post by Juli Mallett-4
* Juli Mallett <[hidden email]> [180306 21:14]:
> On 6 March 2018 at 10:29, John Baldwin <[hidden email]> wrote:
>> On Tuesday, March 06, 2018 08:37:26 AM Eugene Grosbein wrote:
>>> 06.03.2018 8:27, Eugene Grosbein wrote:
>>>> 06.03.2018 4:16, Wolfgang Zenker wrote:

>>>>> I'm trying to run installworld using 11-STABLE on an Ubiquity Edge
>>>>> Router Lite (mips64, 2 cores, 512 MB Ram). Unfortunately I haven't
>>>>> managed to finish the installworld yet, I always get a
>>>>> panic: kernel stack overflow - trapframe at 0xffffffff80917eb0
>>>>> in slightly different places during the installworld. Of the 4 panics
>>>>> I have seen on the serial console, 3 had the trapframe at
>>>>> 0xffffffff80917eb0 and one at 0xffffffff80915eb0
>>>>> /usr/src and /usr/obj are nfs-mounted, and I have configured almost 2 GB
>>>>> of swap. The build was done in a Qemu environment.

>>>>> Any hints how to proceed from here?

>>>> Try increasing kernel stack size from default 2 pages to 4 by rebuilding
>>>> the kernel with options KSTACK_PAGES=4

>>> Note also, that depending on your network configuration, KSTACK_PAGES=4
>>> may or may not be enough. If it does not help, you need to double it
>>> once more.

>> KSTACK_PAGES doesn't work on MIPS because the MIPS kstack has to be
>> hardwired
>> into the TLB and the code that does that assumes a hardcoded stack size.

> That said, we could easily use a more flexible wired TLB entry scheme,
> including smartly using pagemask in the cases where the number of pages is
> suitable.  If we wanted to allow wiring of mappings into the TLB flexibly
> at runtime we could do that, or we could just at compile-time have
> different code to handle different KSTACK_PAGES values.  People have strong
> feelings about some of those options, but if there's a workload-oriented
> pressure to move in a different direction, it should be very easy to do.

I would welcome a more flexible solution, of course. However, I guess my
situation is a bit unusual: the ERL is an unusually powerful machine for
a mips based network device, and therefore I want to experiment with it
trying different options without dismantling the device for every update
to switch in a new usb stick. I think most users just set it up once
and only update in case of security problems. Anyhow, if a more flexible
solution could be done without to much effort, please do it. If I can
help by testing, please let me know.

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-mips
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: ELF - panic on installworld

Wolfgang Zenker-2
In reply to this post by Eugene Grosbein-10
* Eugene Grosbein <[hidden email]> [180306 22:48]:
> 07.03.2018 3:14, Juli Mallett wrote:

>> That said, we could easily use a more flexible wired TLB entry scheme,
>> including smartly using pagemask in the cases where the number of pages is
>> suitable.  If we wanted to allow wiring of mappings into the TLB flexibly
>> at runtime we could do that, or we could just at compile-time have
>> different code to handle different KSTACK_PAGES values.  People have strong
>> feelings about some of those options, but if there's a workload-oriented
>> pressure to move in a different direction, it should be very easy to do.

> I cannot understand what "feelings" may exist about kernel stack shortage leading to panics.

People being people, they have feelings about everything, no matter if
that is logical or not. So it is better to be prepared for it.

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-mips
To unsubscribe, send any mail to "[hidden email]"