nosh init system

classic Classic list List threaded Threaded
28 messages Options
12
Reply | Threaded
Open this post in threaded view
|

nosh init system

freebsd-hackers mailing list
Hi everyone.

I might be missing something since I have only been in the group for a few months, but is anyone looking at the "nosh" init system ( https://jdebp.eu/Softwares/nosh/ )?
From what I have read there is some talk of writing a new init system; is nosh known to be bad in some way or just obscure (it did take me a decent while to find)?

From what I can find it is aiming to fill an systemd-shaped hole in a better way and while maintaining compatibility with BSD. I am not exceptionally read in and may be missing some pitfalls.

I am curious what you think of it.

/Sidju
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: nosh init system

Wojciech Puchar-8
>
> I might be missing something since I have only been in the group for a few months, but is anyone looking at the "nosh" init system ( https://jdebp.eu/Softwares/nosh/ )?
>> From what I have read there is some talk of writing a new init system; is nosh known to be bad in some way or just obscure (it did take me a decent while to find)?

What is wrong in existing init system?

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: nosh init system

Cy Schubert-4
On February 8, 2019 1:13:04 PM PST, Wojciech Puchar <[hidden email]> wrote:

>>
>> I might be missing something since I have only been in the group for
>a few months, but is anyone looking at the "nosh" init system (
>https://jdebp.eu/Softwares/nosh/ )?
>>> From what I have read there is some talk of writing a new init
>system; is nosh known to be bad in some way or just obscure (it did
>take me a decent while to find)?
>
>What is wrong in existing init system?
>
>_______________________________________________
>[hidden email] mailing list
>https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
>To unsubscribe, send any mail to
>"[hidden email]"

nosh isn't really mainstream. Why not make an port?
--
Pardon the typos and autocorrect, small keyboard in use.
Cheers,
Cy Schubert <[hidden email]>
FreeBSD UNIX: <[hidden email]> Web: http://www.FreeBSD.org

        The need of the many outweighs the greed of the few.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: nosh init system

Conrad Meyer-2
In reply to this post by freebsd-hackers mailing list
Hi Sidju,

On Fri, Feb 8, 2019 at 12:52 PM Sidju via freebsd-hackers
<[hidden email]> wrote:
> I might be missing something since I have only been in the group for a few months, but is anyone looking at the "nosh" init system ( https://jdebp.eu/Softwares/nosh/ )?

Not that I know of.

> From what I have read there is some talk of writing a new init system; is nosh known to be bad in some way or just obscure (it did take me a decent while to find)?

It's good to be use precise terminology when discussing these things,
because confusion can lead to the wrong idea and pedantic arguments.
To be clear, FreeBSD isn't missing an *init*, which serves to launch
pid 2 and reap zombies.  We're missing a half-decent service
management system.  On Linux, systemd performs both roles.  On
FreeBSD, init(8) serves one role, and rc(8) is a tiny subset of a real
service management system like systemd.

(I think the piece we would consider replacing or supplementing would
be rc(8).  Part of that might be migrating some responsibilities from
pid 1 to pid 2, such as spawning gettys.  I don't hold strong opinions
about that.)

Nosh may be useful for inspiration (and portions may be used directly;
it's MIT and BSD licensed).  I haven't taken a thorough look at it,
and I don't know of anyone else who has either.  I don't believe it is
suitable to be dropped in (for a host of reasons, not all technical),
but I could be persuaded I am mistaken.

Best regards,
Conrad
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: nosh init system

Cy Schubert-4
On February 8, 2019 1:51:16 PM PST, Conrad Meyer <[hidden email]> wrote:

>Hi Sidju,
>
>On Fri, Feb 8, 2019 at 12:52 PM Sidju via freebsd-hackers
><[hidden email]> wrote:
>> I might be missing something since I have only been in the group for
>a few months, but is anyone looking at the "nosh" init system (
>https://jdebp.eu/Softwares/nosh/ )?
>
>Not that I know of.
>
>> From what I have read there is some talk of writing a new init
>system; is nosh known to be bad in some way or just obscure (it did
>take me a decent while to find)?
>
>It's good to be use precise terminology when discussing these things,
>because confusion can lead to the wrong idea and pedantic arguments.
>To be clear, FreeBSD isn't missing an *init*, which serves to launch
>pid 2 and reap zombies.  We're missing a half-decent service
>management system.  On Linux, systemd performs both roles.  On
>FreeBSD, init(8) serves one role, and rc(8) is a tiny subset of a real
>service management system like systemd.
>
>(I think the piece we would consider replacing or supplementing would
>be rc(8).  Part of that might be migrating some responsibilities from
>pid 1 to pid 2, such as spawning gettys.  I don't hold strong opinions
>about that.)
>
>Nosh may be useful for inspiration (and portions may be used directly;
>it's MIT and BSD licensed).  I haven't taken a thorough look at it,
>and I don't know of anyone else who has either.  I don't believe it is
>suitable to be dropped in (for a host of reasons, not all technical),
>but I could be persuaded I am mistaken.
>
>Best regards,
>Conrad
>_______________________________________________
>[hidden email] mailing list
>https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
>To unsubscribe, send any mail to
>"[hidden email]"

I've been partial to Solaris (illumos) smf,  started by init (through their sysv inittab). CDDL may not be palatable to some but if we could I'd consider it again.

--
Pardon the typos and autocorrect, small keyboard in use.
Cheers,
Cy Schubert <[hidden email]>
FreeBSD UNIX: <[hidden email]> Web: http://www.FreeBSD.org

        The need of the many outweighs the greed of the few.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: nosh init system

Wojciech Puchar-8
In reply to this post by Conrad Meyer-2
> pid 2 and reap zombies.  We're missing a half-decent service
> management system.  On Linux, systemd performs both roles.  On
> FreeBSD, init(8) serves one role, and rc(8) is a tiny subset of a real
> service management system like systemd.

systemd is overcomplex crap. And a reason many people migrated to FreeBSD
from linux.

>
> (I think the piece we would consider replacing or supplementing would
> be rc(8).  Part of that might be migrating some responsibilities from
> pid 1 to pid 2, such as spawning gettys.  I don't hold strong opinions
> about that.)

this make sense but with spawning gettys left to init.


what do you want to improve in rc? starting services in parallel doesn't
seem to be major problem to make i think.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: nosh init system

Enji Cooper
On Feb 9, 2019, at 09:32, Wojciech Puchar <[hidden email]> wrote:

>> pid 2 and reap zombies.  We're missing a half-decent service
>> management system.  On Linux, systemd performs both roles.  On
>> FreeBSD, init(8) serves one role, and rc(8) is a tiny subset of a real
>> service management system like systemd.
>
> systemd is overcomplex crap. And a reason many people migrated to FreeBSD from linux.
>
>>
>> (I think the piece we would consider replacing or supplementing would
>> be rc(8).  Part of that might be migrating some responsibilities from
>> pid 1 to pid 2, such as spawning gettys.  I don't hold strong opinions
>> about that.)
>
> this make sense but with spawning gettys left to init.
>
>
> what do you want to improve in rc? starting services in parallel doesn't seem to be major problem to make i think.

Starting and stopping services based on logical events and “run levels”, apart from what devd handles with hardware events is what comes to mind for me.

rc(8) is also incredibly fragile when it comes to starting or stopping services beyond first boot, or when dealing with “optional” services, like nis/yp. I tried to clean this up a few years ago, but it’s not close to my ideal design (it feels like  a bubblegum and duct tape solution).

Cheers,
-Enji

> _______________________________________________
> [hidden email] mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
> To unsubscribe, send any mail to "[hidden email]"
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: nosh init system

D. Ebdrup
On 2/9/19, Enji Cooper <[hidden email]> wrote:

> On Feb 9, 2019, at 09:32, Wojciech Puchar <[hidden email]> wrote:
>
>>> pid 2 and reap zombies.  We're missing a half-decent service
>>> management system.  On Linux, systemd performs both roles.  On
>>> FreeBSD, init(8) serves one role, and rc(8) is a tiny subset of a real
>>> service management system like systemd.
>>
>> systemd is overcomplex crap. And a reason many people migrated to FreeBSD
>> from linux.
>>
>>>
>>> (I think the piece we would consider replacing or supplementing would
>>> be rc(8).  Part of that might be migrating some responsibilities from
>>> pid 1 to pid 2, such as spawning gettys.  I don't hold strong opinions
>>> about that.)
>>
>> this make sense but with spawning gettys left to init.
>>
>>
>> what do you want to improve in rc? starting services in parallel doesn't
>> seem to be major problem to make i think.
>
> Starting and stopping services based on logical events and “run levels”,
> apart from what devd handles with hardware events is what comes to mind for
> me.
>
> rc(8) is also incredibly fragile when it comes to starting or stopping
> services beyond first boot, or when dealing with “optional” services, like
> nis/yp. I tried to clean this up a few years ago, but it’s not close to my
> ideal design (it feels like  a bubblegum and duct tape solution).
>
> Cheers,
> -Enji
>
>> _______________________________________________
>> [hidden email] mailing list
>> https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
>> To unsubscribe, send any mail to "[hidden email]"
> _______________________________________________
> [hidden email] mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
> To unsubscribe, send any mail to "[hidden email]"
>
It seems to me that while there isn't that much to be gained from the
base system being parallel, trasz@ effort to get rcorder -p (which is
ready for review at [1]) is still something that can massively improve
startup when you're running a lot of jails that don't necessarily have
any or very little interdependence.

I've also heard talk of improving devd to make it react to a lot more
hardware events, which I think is a laudible goal if anyone uses
FreeBSD on a desktop or laptop as I do.

As I see it, FreeBSD already has the basis for something similar to
SMF or systemd - and while I'd prefer whatever FreeBSD gets to be
closer to SMF, I still think there are valuable lessons to be learned
from both systemd and launchd with regard to what's missing.

I also find it a little interesting that not a lot of people seem to
know about the 6th paragraph of init(8) [2]. Since I found out about
it, I've referred often enough to it that I can probably paraphrase it
pretty accurately: "The init utility can also be used to keep
arbitrary daemons running, automatically restarting them if they die."

Also, not to toot my own horn here, but in editing the latest batch of
status reports, I noticed that nosh has a section that people might
wanna read.

[1] https://reviews.freebsd.org/D3715
[2] https://man.freebsd.org/init(8)
--
Daniel Ebdrup aka. D. Ebdrup.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: nosh init system

Adam Vande More
In reply to this post by Enji Cooper
On Sat, Feb 9, 2019 at 11:57 AM Enji Cooper <[hidden email]> wrote:

> On Feb 9, 2019, at 09:32, Wojciech Puchar <[hidden email]> wrote:
>
> >> pid 2 and reap zombies.  We're missing a half-decent service
> >> management system.  On Linux, systemd performs both roles.  On
> >> FreeBSD, init(8) serves one role, and rc(8) is a tiny subset of a real
> >> service management system like systemd.
> >
> > systemd is overcomplex crap. And a reason many people migrated to
> FreeBSD from linux.
> >
> >>
> >> (I think the piece we would consider replacing or supplementing would
> >> be rc(8).  Part of that might be migrating some responsibilities from
> >> pid 1 to pid 2, such as spawning gettys.  I don't hold strong opinions
> >> about that.)
> >
> > this make sense but with spawning gettys left to init.
> >
> >
> > what do you want to improve in rc? starting services in parallel doesn't
> seem to be major problem to make i think.
>
> Starting and stopping services based on logical events and “run levels”,
> apart from what devd handles with hardware events is what comes to mind for
> me.
>
> rc(8) is also incredibly fragile when it comes to starting or stopping
> services beyond first boot, or when dealing with “optional” services, like
> nis/yp. I tried to clean this up a few years ago, but it’s not close to my
> ideal design (it feels like  a bubblegum and duct tape solution).
>

"incredibly fragile" indicates there is some common, easily triggered issue
with it.  Can you elaborate please?  I stop and restart base services and
others on a regular basis and don't see an issue there although I also
haven't use NIS for some time.
--
Adam
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: nosh init system

Eugene Grosbein-10
In reply to this post by Enji Cooper
10.02.2019 0:55, Enji Cooper write:

> rc(8) is also incredibly fragile when it comes to starting or stopping services beyond first boot,
> or when dealing with “optional” services, like nis/yp.

Starting/stopping services beyound first boot has really only slight connection to first boot init system,
it can and should be dealt with distinct facility.

Yes, our init system can be improved but please do not overcomplicate it with tasks
it should not solve.

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: nosh init system

Cy Schubert-4
In reply to this post by freebsd-hackers mailing list
In message <[hidden email]>, Enji
Cooper writes
:

> On Feb 9, 2019, at 09:32, Wojciech Puchar <[hidden email]> wrote:
>
> >> pid 2 and reap zombies.  We're missing a half-decent service
> >> management system.  On Linux, systemd performs both roles.  On
> >> FreeBSD, init(8) serves one role, and rc(8) is a tiny subset of a real
> >> service management system like systemd.
> >
> > systemd is overcomplex crap. And a reason many people migrated to FreeBSD f
> rom linux.
> >
> >>
> >> (I think the piece we would consider replacing or supplementing would
> >> be rc(8).  Part of that might be migrating some responsibilities from
> >> pid 1 to pid 2, such as spawning gettys.  I don't hold strong opinions
> >> about that.)
> >
> > this make sense but with spawning gettys left to init.
> >
> >
> > what do you want to improve in rc? starting services in parallel doesn't se
> em to be major problem to make i think.
>
> Starting and stopping services based on logical events and “run levels”,
> apart from what devd handles with hardware events is what comes to mind for m
> e.
>
> rc(8) is also incredibly fragile when it comes to starting or stopping servic
> es beyond first boot, or when dealing with “optional” services, like nis/
> yp. I tried to clean this up a few years ago, but it’s not close to my idea
> l design (it feels like  a bubblegum and duct tape solution).

I've been using NIS on FreeBSD for a couple of decades and still using
it on my network. Except for one bad patch a few years ago it's been
100% solid. There are no startup or shutdown issues.

I don't see what's so "incredibly fragile" about rc(8). That's not to
say there aren't better solutions, like SMF.

Where rc(8) falls down is any port or a customer's (user of FreeBSD) rc
script could fail hosing the boot or worse hosing the system*. Where a
solution like SMF solves the problem is that should a service which
other services depend on fail, only that branch of the startup tree
would fail. In that scenario, if a service fails but sshd start, a
sysadmin would still be able to login remotely to resolve the problem.
So in this regard rc(8) is at a disadvantage.

We could address the above paragraph by starting sshd earlier during
boot thereby allowing the opportunity to fix remotely.

Regarding SMF, it could be implemented by rc(8) invoking smf in similar
fashion as Solaris does -- Solaris invokes it through inittab(5) -- or
it could through a special yet to be determined entry in ttys(5). The
Solaris approach is init(8)'s sole job, through the inittab(5) entry,
is to restart smf, while smf does the rest. I prefer not to discuss
implementation details right now, it's premature.

* Incredibly stupid people can hose SMF too. It is more complex. OTOH
that complexity might scare the uninitiated from attempting something
dumb.


--
Cheers,
Cy Schubert <[hidden email]>
FreeBSD UNIX:  <[hidden email]>   Web:  http://www.FreeBSD.org

        The need of the many outweighs the greed of the few.


_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: nosh init system

Conrad Meyer-2
Hi Cy,

On Sat, Feb 9, 2019 at 3:35 PM Cy Schubert <[hidden email]> wrote:
> I don't see what's so "incredibly fragile" about rc(8). That's not to
> say there aren't better solutions, like SMF.

Maybe "incredibly" as a choice of adjective is inappropriate.  I think
we (you, me, and ngie@) can all agree it is somewhat fragile, and
there are things SMF/systemd/nosh get right that rc(8) does not
(today).  Anyway, your next paragraph goes on to be a good start at
describing some of rc's fragility.  :-)

> Where rc(8) falls down is any port or a customer's (user of FreeBSD) rc
> script could fail hosing the boot or worse hosing the system*. Where a
> solution like SMF solves the problem is that should a service which
> other services depend on fail, only that branch of the startup tree
> would fail.

Right; that's a great example.

> In that scenario, if a service fails but sshd start, a
> sysadmin would still be able to login remotely to resolve the problem.
> So in this regard rc(8) is at a disadvantage.
>
> We could address the above paragraph by starting sshd earlier during
> boot thereby allowing the opportunity to fix remotely.

I don't think that is really sufficient without substantially
modifying init+rc to be closer to something like systemd or SMF,
anyway.  And then we'd rather just have something like SMF :-).

As soon as *any* rc service fails to start (signal, non-zero exit,
stop_boot), rc(8) exits non-zero, causing init(8) to go to single
user.  All service state is thrown away with rc(8) exit, but any rc.d
"services" that managed to start before boot failed are not
terminated.  Even if an admin manages to log in and fix the
configuration, re-starting rc(8) restarts the runcom process from
scratch, as if nothing had already been done, without first stopping
anything that was already running.  The only safe, reproducible way to
re-start rc(8) is to fully reboot the system.

E.g., the major pain point we run into repeatedly with restarted boot
is that cleanvar / cleartmp run again.  This breaks ld-elf.so.hints
cache (anything linking /usr/local libraries — hope your admin is
running base sshd and not openssh-portable!) as well as wiping out
/var/run pid files (breaking "already running?" rc pid checks).  As a
result, services get double-started.

Cleanvar could maybe be improved to avoid this problem — e.g., we
could coordinate with the kernel to set a per-boot, persisted flag
that cleanvar has completed, even if rc(8) exits — but the broad class
of problems would remain (rc.d autostart is stateful, but any partial
failure destroys all state).

Best regards,
Conrad
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: nosh init system

Cy Schubert-4
In reply to this post by freebsd-hackers mailing list
In message <[hidden email]
il.com>
, Conrad Meyer writes:

> Hi Cy,
>
> On Sat, Feb 9, 2019 at 3:35 PM Cy Schubert <[hidden email]> wrote:
> > I don't see what's so "incredibly fragile" about rc(8). That's not to
> > say there aren't better solutions, like SMF.
>
> Maybe "incredibly" as a choice of adjective is inappropriate.  I think
> we (you, me, and ngie@) can all agree it is somewhat fragile, and
> there are things SMF/systemd/nosh get right that rc(8) does not
> (today).  Anyway, your next paragraph goes on to be a good start at
> describing some of rc's fragility.  :-)
>
> > Where rc(8) falls down is any port or a customer's (user of FreeBSD) rc
> > script could fail hosing the boot or worse hosing the system*. Where a
> > solution like SMF solves the problem is that should a service which
> > other services depend on fail, only that branch of the startup tree
> > would fail.
>
> Right; that's a great example.
>
> > In that scenario, if a service fails but sshd start, a
> > sysadmin would still be able to login remotely to resolve the problem.
> > So in this regard rc(8) is at a disadvantage.
> >
> > We could address the above paragraph by starting sshd earlier during
> > boot thereby allowing the opportunity to fix remotely.
>
> I don't think that is really sufficient without substantially
> modifying init+rc to be closer to something like systemd or SMF,
> anyway.  And then we'd rather just have something like SMF :-).

I'd rather see SMF but a number felt a CDDL licensed init was
unacceptable -- except for the fact that SMF doesn't replace init.

>
> As soon as *any* rc service fails to start (signal, non-zero exit,
> stop_boot), rc(8) exits non-zero, causing init(8) to go to single
> user.  All service state is thrown away with rc(8) exit, but any rc.d
> "services" that managed to start before boot failed are not
> terminated.  Even if an admin manages to log in and fix the
> configuration, re-starting rc(8) restarts the runcom process from
> scratch, as if nothing had already been done, without first stopping
> anything that was already running.  The only safe, reproducible way to
> re-start rc(8) is to fully reboot the system.

It wasn't that way 10-15 years ago. It's evolved to become this. Even
if we stay with rc(8), quickly cobbling together a patch isn't going to
fix it long term. Whether we use another init, an add-on like SMF, or
make rc(8) more robust, it will not be fixed by a simple tweak here or
there.

>
> E.g., the major pain point we run into repeatedly with restarted boot
> is that cleanvar / cleartmp run again.  This breaks ld-elf.so.hints
> cache (anything linking /usr/local libraries — hope your admin is
> running base sshd and not openssh-portable!) as well as wiping out
> /var/run pid files (breaking "already running?" rc pid checks).  As a
> result, services get double-started.
>
> Cleanvar could maybe be improved to avoid this problem — e.g., we
> could coordinate with the kernel to set a per-boot, persisted flag
> that cleanvar has completed, even if rc(8) exits — but the broad class
> of problems would remain (rc.d autostart is stateful, but any partial
> failure destroys all state).

This needs more than improving cleanvar or some other script. It's like
whack-a-mole. (The rest of this not specifically talking to you
Conrad.) This is why every one to two months this topic comes up again
and again and again. It's a pain point. (And also the shiny new object
syndrome.) Various people suggest their favourite init(8) replacement
and the bikeshed starts up again.

To avoid bikeshedding this to death again, we enumerated two issues so
far. Let's continue to list issues. I also think that this should be a
BSDCan devsummit whiteboard topic where we list issues in one column
and next to it we list possible solutions, after listing all the issues
first. And finally if this is too large for one person to work on,
assign the various issues to willing developers.

One final thought. init(8) and rc(8) requirements for desktop/laptop,
server, embedded, and mobile are probably different enough that their
requirements may compete with each other. Some embedded applications
may desire a simple rc(8) whereas server or desktop a heavier weight
solution.


--
Cheers,
Cy Schubert <[hidden email]>
FreeBSD UNIX:  <[hidden email]>   Web:  http://www.FreeBSD.org

        The need of the many outweighs the greed of the few.


_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: nosh init system

Enji Cooper
In reply to this post by Adam Vande More

> On Feb 9, 2019, at 10:32, Adam <[hidden email]> wrote:
>
>> On Sat, Feb 9, 2019 at 11:57 AM Enji Cooper <[hidden email]> wrote:
>
>> On Feb 9, 2019, at 09:32, Wojciech Puchar <[hidden email]> wrote:
>>
>> >> pid 2 and reap zombies.  We're missing a half-decent service
>> >> management system.  On Linux, systemd performs both roles.  On
>> >> FreeBSD, init(8) serves one role, and rc(8) is a tiny subset of a real
>> >> service management system like systemd.
>> >
>> > systemd is overcomplex crap. And a reason many people migrated to FreeBSD from linux.
>> >
>> >>
>> >> (I think the piece we would consider replacing or supplementing would
>> >> be rc(8).  Part of that might be migrating some responsibilities from
>> >> pid 1 to pid 2, such as spawning gettys.  I don't hold strong opinions
>> >> about that.)
>> >
>> > this make sense but with spawning gettys left to init.
>> >
>> >
>> > what do you want to improve in rc? starting services in parallel doesn't seem to be major problem to make i think.
>>
>> Starting and stopping services based on logical events and “run levels”, apart from what devd handles with hardware events is what comes to mind for me.
>>
>> rc(8) is also incredibly fragile when it comes to starting or stopping services beyond first boot, or when dealing with “optional” services, like nis/yp. I tried to clean this up a few years ago, but it’s not close to my ideal design (it feels like  a bubblegum and duct tape solution).
>
> "incredibly fragile" indicates there is some common, easily triggered issue with it.  Can you elaborate please?  I stop and restart base services and others on a regular basis and don't see an issue there although I also haven't use NIS for some time.

Try restarting netif if you have a static route set; it will break routing (until you restart the routing pseudo service), causing your machine to become unreachable if you’re not on the same network segment.

Linux doesn’t have this problem; neither does OS X.

Thanks,
-Enji
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: nosh init system

Enji Cooper
In reply to this post by Cy Schubert-4

> On Feb 9, 2019, at 15:34, Cy Schubert <[hidden email]> wrote:

...

> I've been using NIS on FreeBSD for a couple of decades and still using
> it on my network. Except for one bad patch a few years ago it's been
> 100% solid. There are no startup or shutdown issues.

My example of yp was probably a bad example for others to glom on to.

Some real examples are netif vs routing, samba’s myriad of scripts, or restarting rpcbind (which doesn’t trigger a restart of mountd, nfsd, etc). In the latter case, one can argue that the services can be made more fault tolerant to their dependencies not being available (that’s one part of a solution), however, the example of netif vs routing is much, much harder to solve without having an external service manager/monitor.

> I don't see what's so "incredibly fragile" about rc(8). That's not to
> say there aren't better solutions, like SMF.
>
> Where rc(8) falls down is any port or a customer's (user of FreeBSD) rc
> script could fail hosing the boot or worse hosing the system*. Where a
> solution like SMF solves the problem is that should a service which
> other services depend on fail, only that branch of the startup tree
> would fail. In that scenario, if a service fails but sshd start, a
> sysadmin would still be able to login remotely to resolve the problem.
> So in this regard rc(8) is at a disadvantage.

This is another item that yes, rc doesn’t solve properly. Good call (I didn’t notice this shortcoming).

> We could address the above paragraph by starting sshd earlier during
> boot thereby allowing the opportunity to fix remotely.

Ehhhhhhh... this is probably not that easy.

> Regarding SMF, it could be implemented by rc(8) invoking smf in similar
> fashion as Solaris does -- Solaris invokes it through inittab(5) -- or
> it could through a special yet to be determined entry in ttys(5). The
> Solaris approach is init(8)'s sole job, through the inittab(5) entry,
> is to restart smf, while smf does the rest. I prefer not to discuss
> implementation details right now, it's premature.
>
> * Incredibly stupid people can hose SMF too. It is more complex. OTOH
> that complexity might scare the uninitiated from attempting something
> dumb.

Quite frankly, service managers/monitors should manage services in a best effort manner. Unless, there’s danger of something like filesystem corruption occurring, I don’t think that a ton of effort should be put into making complicated cases work.

Cheers,
-Enji
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: nosh init system

Rodney W. Grimes-4
In reply to this post by Cy Schubert-4
> In message <[hidden email]
> il.com>
> , Conrad Meyer writes:
> > Hi Cy,
> >
> > On Sat, Feb 9, 2019 at 3:35 PM Cy Schubert <[hidden email]> wrote:
> > > I don't see what's so "incredibly fragile" about rc(8). That's not to
> > > say there aren't better solutions, like SMF.
> >
> > Maybe "incredibly" as a choice of adjective is inappropriate.  I think
> > we (you, me, and ngie@) can all agree it is somewhat fragile, and
> > there are things SMF/systemd/nosh get right that rc(8) does not
> > (today).  Anyway, your next paragraph goes on to be a good start at
> > describing some of rc's fragility.  :-)
> >
> > > Where rc(8) falls down is any port or a customer's (user of FreeBSD) rc
> > > script could fail hosing the boot or worse hosing the system*. Where a
> > > solution like SMF solves the problem is that should a service which
> > > other services depend on fail, only that branch of the startup tree
> > > would fail.
> >
> > Right; that's a great example.
> >
> > > In that scenario, if a service fails but sshd start, a
> > > sysadmin would still be able to login remotely to resolve the problem.
> > > So in this regard rc(8) is at a disadvantage.
> > >
> > > We could address the above paragraph by starting sshd earlier during
> > > boot thereby allowing the opportunity to fix remotely.
> >
> > I don't think that is really sufficient without substantially
> > modifying init+rc to be closer to something like systemd or SMF,
> > anyway.  And then we'd rather just have something like SMF :-).
>
> I'd rather see SMF but a number felt a CDDL licensed init was
> unacceptable -- except for the fact that SMF doesn't replace init.
>
> >
> > As soon as *any* rc service fails to start (signal, non-zero exit,
> > stop_boot), rc(8) exits non-zero, causing init(8) to go to single
> > user.  All service state is thrown away with rc(8) exit, but any rc.d
> > "services" that managed to start before boot failed are not
> > terminated.  Even if an admin manages to log in and fix the
> > configuration, re-starting rc(8) restarts the runcom process from
> > scratch, as if nothing had already been done, without first stopping
> > anything that was already running.  The only safe, reproducible way to
> > re-start rc(8) is to fully reboot the system.

It -should- be safe to restart rc, as rc scripts should check to
see if the item they are being requested to start is already running,
rc scripts that fail to have this check are defective and should be
fixed.  You should be able to invate /etc/rc.d/foo start as many
times as you want in a row and only get 1 instance of foo, with the
other starts returning "foo already running"   Same with stop.

>
> It wasn't that way 10-15 years ago. It's evolved to become this. Even
> if we stay with rc(8), quickly cobbling together a patch isn't going to
> fix it long term. Whether we use another init, an add-on like SMF, or
> make rc(8) more robust, it will not be fixed by a simple tweak here or
> there.

Much gets broken in the name of new features sadly.

>
> >
> > E.g., the major pain point we run into repeatedly with restarted boot
> > is that cleanvar / cleartmp run again.  This breaks ld-elf.so.hints
> > cache (anything linking /usr/local libraries ??? hope your admin is
> > running base sshd and not openssh-portable!) as well as wiping out
> > /var/run pid files (breaking "already running?" rc pid checks).  As a
> > result, services get double-started.
> >
> > Cleanvar could maybe be improved to avoid this problem ??? e.g., we
> > could coordinate with the kernel to set a per-boot, persisted flag
> > that cleanvar has completed, even if rc(8) exits ??? but the broad class
> > of problems would remain (rc.d autostart is stateful, but any partial
> > failure destroys all state).
>
> This needs more than improving cleanvar or some other script. It's like
> whack-a-mole. (The rest of this not specifically talking to you
> Conrad.) This is why every one to two months this topic comes up again
> and again and again. It's a pain point. (And also the shiny new object
> syndrome.) Various people suggest their favourite init(8) replacement
> and the bikeshed starts up again.

Shiny new things also come with shiny new problems, I would actually
often rather repair a broken old something than get a new shiny
something as I know the defects of the raty old something.

> To avoid bikeshedding this to death again, we enumerated two issues so
> far. Let's continue to list issues. I also think that this should be a
> BSDCan devsummit whiteboard topic where we list issues in one column
> and next to it we list possible solutions, after listing all the issues
> first. And finally if this is too large for one person to work on,
> assign the various issues to willing developers.

We do not need to wait for BSDCan, there are more of us here on this
list than at any dev summit.

>
> One final thought. init(8) and rc(8) requirements for desktop/laptop,
> server, embedded, and mobile are probably different enough that their
> requirements may compete with each other. Some embedded applications
> may desire a simple rc(8) whereas server or desktop a heavier weight
> solution.

It is rather simple to just drop the whole rc.d and rewrite
/etc/rc for the embeded situtaion, going back to the 4.3 era.

Though we might want to go over to the rc mailling list?

> Cy Schubert <[hidden email]>
--
Rod Grimes                                                 [hidden email]
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: nosh init system

Cy Schubert-4
In reply to this post by freebsd-hackers mailing list
In message <[hidden email]>,
"Rodney W. Gri
mes" writes:

> > In message <[hidden email]
> > il.com>
> > , Conrad Meyer writes:
> > > Hi Cy,
> > >
> > > On Sat, Feb 9, 2019 at 3:35 PM Cy Schubert <[hidden email]> wr
> ote:
> > > > I don't see what's so "incredibly fragile" about rc(8). That's not to
> > > > say there aren't better solutions, like SMF.
> > >
> > > Maybe "incredibly" as a choice of adjective is inappropriate.  I think
> > > we (you, me, and ngie@) can all agree it is somewhat fragile, and
> > > there are things SMF/systemd/nosh get right that rc(8) does not
> > > (today).  Anyway, your next paragraph goes on to be a good start at
> > > describing some of rc's fragility.  :-)
> > >
> > > > Where rc(8) falls down is any port or a customer's (user of FreeBSD) rc
> > > > script could fail hosing the boot or worse hosing the system*. Where a
> > > > solution like SMF solves the problem is that should a service which
> > > > other services depend on fail, only that branch of the startup tree
> > > > would fail.
> > >
> > > Right; that's a great example.
> > >
> > > > In that scenario, if a service fails but sshd start, a
> > > > sysadmin would still be able to login remotely to resolve the problem.
> > > > So in this regard rc(8) is at a disadvantage.
> > > >
> > > > We could address the above paragraph by starting sshd earlier during
> > > > boot thereby allowing the opportunity to fix remotely.
> > >
> > > I don't think that is really sufficient without substantially
> > > modifying init+rc to be closer to something like systemd or SMF,
> > > anyway.  And then we'd rather just have something like SMF :-).
> >
> > I'd rather see SMF but a number felt a CDDL licensed init was
> > unacceptable -- except for the fact that SMF doesn't replace init.
> >
> > >
> > > As soon as *any* rc service fails to start (signal, non-zero exit,
> > > stop_boot), rc(8) exits non-zero, causing init(8) to go to single
> > > user.  All service state is thrown away with rc(8) exit, but any rc.d
> > > "services" that managed to start before boot failed are not
> > > terminated.  Even if an admin manages to log in and fix the
> > > configuration, re-starting rc(8) restarts the runcom process from
> > > scratch, as if nothing had already been done, without first stopping
> > > anything that was already running.  The only safe, reproducible way to
> > > re-start rc(8) is to fully reboot the system.
>
> It -should- be safe to restart rc, as rc scripts should check to
> see if the item they are being requested to start is already running,
> rc scripts that fail to have this check are defective and should be
> fixed.  You should be able to invate /etc/rc.d/foo start as many
> times as you want in a row and only get 1 instance of foo, with the
> other starts returning "foo already running"   Same with stop.
>
> >
> > It wasn't that way 10-15 years ago. It's evolved to become this. Even
> > if we stay with rc(8), quickly cobbling together a patch isn't going to
> > fix it long term. Whether we use another init, an add-on like SMF, or
> > make rc(8) more robust, it will not be fixed by a simple tweak here or
> > there.
>
> Much gets broken in the name of new features sadly.

That was my point. Tunnel vision.

>
> >
> > >
> > > E.g., the major pain point we run into repeatedly with restarted boot
> > > is that cleanvar / cleartmp run again.  This breaks ld-elf.so.hints
> > > cache (anything linking /usr/local libraries ??? hope your admin is
> > > running base sshd and not openssh-portable!) as well as wiping out
> > > /var/run pid files (breaking "already running?" rc pid checks).  As a
> > > result, services get double-started.
> > >
> > > Cleanvar could maybe be improved to avoid this problem ??? e.g., we
> > > could coordinate with the kernel to set a per-boot, persisted flag
> > > that cleanvar has completed, even if rc(8) exits ??? but the broad class
> > > of problems would remain (rc.d autostart is stateful, but any partial
> > > failure destroys all state).
> >
> > This needs more than improving cleanvar or some other script. It's like
> > whack-a-mole. (The rest of this not specifically talking to you
> > Conrad.) This is why every one to two months this topic comes up again
> > and again and again. It's a pain point. (And also the shiny new object
> > syndrome.) Various people suggest their favourite init(8) replacement
> > and the bikeshed starts up again.
>
> Shiny new things also come with shiny new problems, I would actually
> often rather repair a broken old something than get a new shiny
> something as I know the defects of the raty old something.

Agreed. Like building on a foundation of sand.

>
> > To avoid bikeshedding this to death again, we enumerated two issues so
> > far. Let's continue to list issues. I also think that this should be a
> > BSDCan devsummit whiteboard topic where we list issues in one column
> > and next to it we list possible solutions, after listing all the issues
> > first. And finally if this is too large for one person to work on,
> > assign the various issues to willing developers.
>
> We do not need to wait for BSDCan, there are more of us here on this
> list than at any dev summit.

True but whiteboards help. My point is let's itemize a list of issues
first. Write down (figuratively speaking) all ideas to address the
listed issues. Select the best ideas. Implement them.

>
> >
> > One final thought. init(8) and rc(8) requirements for desktop/laptop,
> > server, embedded, and mobile are probably different enough that their
> > requirements may compete with each other. Some embedded applications
> > may desire a simple rc(8) whereas server or desktop a heavier weight
> > solution.
>
> It is rather simple to just drop the whole rc.d and rewrite
> /etc/rc for the embeded situtaion, going back to the 4.3 era.
>
> Though we might want to go over to the rc mailling list?

Excellent idea!


--
Cheers,
Cy Schubert <[hidden email]>
FreeBSD UNIX:  <[hidden email]>   Web:  http://www.FreeBSD.org

        The need of the many outweighs the greed of the few.


_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: nosh init system

Enji Cooper
In reply to this post by Rodney W. Grimes-4
On Feb 9, 2019, at 20:20, Rodney W. Grimes <[hidden email]> wrote:

>> In message <[hidden email]
>> il.com>
>> , Conrad Meyer writes:
>>> Hi Cy,
>>>
>>>> On Sat, Feb 9, 2019 at 3:35 PM Cy Schubert <[hidden email]> wrote:
>>>> I don't see what's so "incredibly fragile" about rc(8). That's not to
>>>> say there aren't better solutions, like SMF.
>>>
>>> Maybe "incredibly" as a choice of adjective is inappropriate.  I think
>>> we (you, me, and ngie@) can all agree it is somewhat fragile, and
>>> there are things SMF/systemd/nosh get right that rc(8) does not
>>> (today).  Anyway, your next paragraph goes on to be a good start at
>>> describing some of rc's fragility.  :-)
>>>
>>>> Where rc(8) falls down is any port or a customer's (user of FreeBSD) rc
>>>> script could fail hosing the boot or worse hosing the system*. Where a
>>>> solution like SMF solves the problem is that should a service which
>>>> other services depend on fail, only that branch of the startup tree
>>>> would fail.
>>>
>>> Right; that's a great example.
>>>
>>>> In that scenario, if a service fails but sshd start, a
>>>> sysadmin would still be able to login remotely to resolve the problem.
>>>> So in this regard rc(8) is at a disadvantage.
>>>>
>>>> We could address the above paragraph by starting sshd earlier during
>>>> boot thereby allowing the opportunity to fix remotely.
>>>
>>> I don't think that is really sufficient without substantially
>>> modifying init+rc to be closer to something like systemd or SMF,
>>> anyway.  And then we'd rather just have something like SMF :-).
>>
>> I'd rather see SMF but a number felt a CDDL licensed init was
>> unacceptable -- except for the fact that SMF doesn't replace init.
>>
>>>
>>> As soon as *any* rc service fails to start (signal, non-zero exit,
>>> stop_boot), rc(8) exits non-zero, causing init(8) to go to single
>>> user.  All service state is thrown away with rc(8) exit, but any rc.d
>>> "services" that managed to start before boot failed are not
>>> terminated.  Even if an admin manages to log in and fix the
>>> configuration, re-starting rc(8) restarts the runcom process from
>>> scratch, as if nothing had already been done, without first stopping
>>> anything that was already running.  The only safe, reproducible way to
>>> re-start rc(8) is to fully reboot the system.
>
> It -should- be safe to restart rc, as rc scripts should check to
> see if the item they are being requested to start is already running,
> rc scripts that fail to have this check are defective and should be
> fixed.  You should be able to invate /etc/rc.d/foo start as many
> times as you want in a row and only get 1 instance of foo, with the
> other starts returning "foo already running"   Same with stop.

I’m not sure if Conrad is referring to the isilon way of restarting services. If so, the isilon parallel start process would effectively wipe the slate clean and restart everything if interrupted, which (because of the nature of cleanvar, etc), would wipe out any and all pidfiles, resulting in in weird set of services which fail to start on next run through.

In short, I think the fact that isilon didn’t mount tmpfs to /var/run was begging for pain, as it’s a directory one should only setup once at boot.

That being said, there are other pseudo services that aren’t necessarily idempotent. If they run twice, the second run could result in breakage to other dependent services run after them.

Thanks,
-Enji
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: nosh init system

Cy Schubert-4
In reply to this post by freebsd-hackers mailing list
In message <[hidden email]>, Enji
Cooper writes
:

> On Feb 9, 2019, at 20:20, Rodney W. Grimes <[hidden email]
> t> wrote:
>
> >> In message <[hidden email]
> >> il.com>
> >> , Conrad Meyer writes:
> >>> Hi Cy,
> >>>
> >>>> On Sat, Feb 9, 2019 at 3:35 PM Cy Schubert <[hidden email]> w
> rote:
> >>>> I don't see what's so "incredibly fragile" about rc(8). That's not to
> >>>> say there aren't better solutions, like SMF.
> >>>
> >>> Maybe "incredibly" as a choice of adjective is inappropriate.  I think
> >>> we (you, me, and ngie@) can all agree it is somewhat fragile, and
> >>> there are things SMF/systemd/nosh get right that rc(8) does not
> >>> (today).  Anyway, your next paragraph goes on to be a good start at
> >>> describing some of rc's fragility.  :-)
> >>>
> >>>> Where rc(8) falls down is any port or a customer's (user of FreeBSD) rc
> >>>> script could fail hosing the boot or worse hosing the system*. Where a
> >>>> solution like SMF solves the problem is that should a service which
> >>>> other services depend on fail, only that branch of the startup tree
> >>>> would fail.
> >>>
> >>> Right; that's a great example.
> >>>
> >>>> In that scenario, if a service fails but sshd start, a
> >>>> sysadmin would still be able to login remotely to resolve the problem.
> >>>> So in this regard rc(8) is at a disadvantage.
> >>>>
> >>>> We could address the above paragraph by starting sshd earlier during
> >>>> boot thereby allowing the opportunity to fix remotely.
> >>>
> >>> I don't think that is really sufficient without substantially
> >>> modifying init+rc to be closer to something like systemd or SMF,
> >>> anyway.  And then we'd rather just have something like SMF :-).
> >>
> >> I'd rather see SMF but a number felt a CDDL licensed init was
> >> unacceptable -- except for the fact that SMF doesn't replace init.
> >>
> >>>
> >>> As soon as *any* rc service fails to start (signal, non-zero exit,
> >>> stop_boot), rc(8) exits non-zero, causing init(8) to go to single
> >>> user.  All service state is thrown away with rc(8) exit, but any rc.d
> >>> "services" that managed to start before boot failed are not
> >>> terminated.  Even if an admin manages to log in and fix the
> >>> configuration, re-starting rc(8) restarts the runcom process from
> >>> scratch, as if nothing had already been done, without first stopping
> >>> anything that was already running.  The only safe, reproducible way to
> >>> re-start rc(8) is to fully reboot the system.
> >
> > It -should- be safe to restart rc, as rc scripts should check to
> > see if the item they are being requested to start is already running,
> > rc scripts that fail to have this check are defective and should be
> > fixed.  You should be able to invate /etc/rc.d/foo start as many
> > times as you want in a row and only get 1 instance of foo, with the
> > other starts returning "foo already running"   Same with stop.
>
> I’m not sure if Conrad is referring to the isilon way of restarting service
> s. If so, the isilon parallel start process would effectively wipe the slate
> clean and restart everything if interrupted, which (because of the nature of
> cleanvar, etc), would wipe out any and all pidfiles, resulting in in weird se
> t of services which fail to start on next run through.
>
> In short, I think the fact that isilon didn’t mount tmpfs to /var/run was b
> egging for pain, as it’s a directory one should only setup once at boot.

Regardless of whether they use tmpfs or not, services should be
constructed in a manner such that it should still work if the customer
chooses not to use tmpfs.

This also goes for those who mount /usr separately like I do (which has
saved my bacon as recently as a couple of weeks ago). A change made to
one of the RC scripts assumed /usr was on rootfs. (When I raised the
issue the reply was "you should /usr on / anyway.") My point is that we
assume our way of setting up a server is the only way and we bulldoze.
In reality FreeBSD and prior to that commercial UNIX were set up
variously. It's only since Linux became so popular that it has been
assumed that one size fits all.

These are two examples of why this approach doesn't work. POLA is
painful.

>
> That being said, there are other pseudo services that aren’t necessarily id
> empotent. If they run twice, the second run could result in breakage to other
>  dependent services run after them.

Cleanvar being the focus of much of our discussion should be able to
determine it has run before.

I'm purposely not discussing implementation details.


--
Cheers,
Cy Schubert <[hidden email]>
FreeBSD UNIX:  <[hidden email]>   Web:  http://www.FreeBSD.org

        The need of the many outweighs the greed of the few.


_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: nosh init system

Wojciech Puchar-8
In reply to this post by Eugene Grosbein-10
> Starting/stopping services beyound first boot has really only slight connection to first boot init system,
> it can and should be dealt with distinct facility.
>
> Yes, our init system can be improved but please do not overcomplicate it with tasks
> it should not solve.
>
>
first someone wanted FreeBSD rewritten in rust, then other want to change
simple and working init system with something overcomplex.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
12