pkgbase: Move of head/etc/ files

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

pkgbase: Move of head/etc/ files

Brad Davis
Hello,

I want to give some information on what is happening to files in head/etc/ and open a discussion.

I am trying to move all config files out of head/etc/ and place them with the source files for the particular use.  For example, some of these are obvious moves like moving say portsnap.conf to usr.sbin/portsnap/portsnap.conf.  The review for this move shows exactly what would happen here: https://reviews.freebsd.org/D16554.  For these cases, there are some easy benefits, like:
1) Keeping all files as part of a program together, which makes maintenance for maintainers a little easier.
2) Simplicity, we do not need all the .if ${MK_FOO} lines in etc/Makefile because that is handled a level above the programs Makefile and so one CONFS= line is all that is needed.
3) Easier to avoid bugs where we install things that should not be installed.  Like for apmd, we installed apmd.conf on all archs even though it only works on i386.  i.e. we don't have to keep knobs in sync in multiple locations.

Some of the more complicated things to move like etc/shells and the init scripts.  I picked the location of etc/shells based on it being used by getusershell(3) in libc, so it ended up in lib/libc/gen/shells.  The move of the init scripts was based on a few things, including the reasons above, but additionally:
1) Isolating a subcomponent, for example work is happening to bring in OpenRC, so we will have multiple init systems with different types of scripts, so moving them away from head/etc/ keeps them isolated.

This work was driven by needing to tag these files as config files in the METALOG so that pkg knows what is a config file so that pkg can 3-way merge them.

More info on pkgbase: https://wiki.freebsd.org/PkgBase


Regards,
Brad Davis
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: pkgbase: Move of head/etc/ files

Simon J. Gerraty
Brad Davis <[hidden email]> wrote:

> I am trying to move all config files out of head/etc/ and place them
> with the source files for the particular use.  For example, some of
> these are obvious moves like moving say portsnap.conf to
> usr.sbin/portsnap/portsnap.conf.  The review for this move shows

In the case of foo.conf for foo, this is a reasonable approach.

There are however many files in etc/ that do not fit that model.

IMO one of the nice things about the BSD src tree, is that the location
of a file in the src tree can generally be inferred from its installed
location on the system.
This property should not be lost without substantial countervailing
benefit.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: pkgbase: Move of head/etc/ files

Brad Davis
On Wed, Aug 1, 2018, at 6:11 PM, Simon J. Gerraty wrote:

> Brad Davis <[hidden email]> wrote:
>
> > I am trying to move all config files out of head/etc/ and place them
> > with the source files for the particular use.  For example, some of
> > these are obvious moves like moving say portsnap.conf to
> > usr.sbin/portsnap/portsnap.conf.  The review for this move shows
>
> In the case of foo.conf for foo, this is a reasonable approach.
>
> There are however many files in etc/ that do not fit that model.
>
> IMO one of the nice things about the BSD src tree, is that the location
> of a file in the src tree can generally be inferred from its installed
> location on the system.
> This property should not be lost without substantial countervailing
> benefit.

Hi Simon,

I generally agree with you, but I think where to draw that line is different based on who is looking at it at the moment.

Lots, not all, but lots, of our config files have the $FreeBSD$ line that shows the path of where the file came from in the source tree, so that helps some.  Using /etc/shells as an example, the first line is:

# $FreeBSD: head/lib/libc/gen/shells 336840 2018-07-28 20:21:23Z brd $


Regards,
Brad Davis
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: pkgbase: Move of head/etc/ files

Guido Falsi-7
In reply to this post by Brad Davis
On 8/2/18 1:54 AM, Brad Davis wrote:
> Hello,
>
> I want to give some information on what is happening to files in head/etc/ and open a discussion.
>

Today while updating a pkgbase machine I got this error(with trimmed
debug output):

The following 1 package(s) will be affected (of 0 checked):

Installed packages to be UPGRADED:
        FreeBSD-runtime: 12.0.s20180725172903 -> 12.0.s20180803092917
[mpnet-base]

Number of packages to be upgraded: 1

Proceed with this action? [y/N]: y
DBG(1)[71153]> want to upgrade advisory to exclusive lock
[1/1] Upgrading FreeBSD-runtime from 12.0.s20180725172903 to
12.0.s20180803092917...
[1/1] Extracting FreeBSD-runtime-12.0.s20180803092917:   1%
DBG(1)[71153]> Populating config_file /etc/blacklistd.conf
[1/1] Extracting FreeBSD-runtime-12.0.s20180803092917:   1%
DBG(1)[71153]> Populating config_file /etc/defaults/rc.conf
Segmentation fault (core dumped)

I'm not sure if it's related to r336845 and r336847, but since those are
the only one touching rc.conf recently it looks possible.

Anyone has an idea what I've stumbled upon?

I'm investigating this, but if I can't fix it shortly I'll revert to my
previous pkg set and try again at another time. If some tests are needed
I'll try to perform them as requested.

Thanks in advance!

BTW I have no objection to moving the configuration files if it's needed
for code maintainability.

--
Guido Falsi <[hidden email]>
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: pkgbase: Move of head/etc/ files

Chris H-2
In reply to this post by Brad Davis
On Wed, 01 Aug 2018 17:54:10 -0600 "Brad Davis" <[hidden email]> said

> Hello,
>
> I want to give some information on what is happening to files in head/etc/
> and open a discussion.
>
> I am trying to move all config files out of head/etc/ and place them with the
> source files for the particular use.  For example, some of these are obvious
> moves like moving say portsnap.conf to usr.sbin/portsnap/portsnap.conf.  The
> review for this move shows exactly what would happen here:
> https://reviews.freebsd.org/D16554.  For these cases, there are some easy
> benefits, like:
> 1) Keeping all files as part of a program together, which makes maintenance
> for maintainers a little easier.
> 2) Simplicity, we do not need all the .if ${MK_FOO} lines in etc/Makefile
> because that is handled a level above the programs Makefile and so one CONFS=
> line is all that is needed.
> 3) Easier to avoid bugs where we install things that should not be installed.
> Like for apmd, we installed apmd.conf on all archs even though it only works
> on i386.  i.e. we don't have to keep knobs in sync in multiple locations.
>
> Some of the more complicated things to move like etc/shells and the init
> scripts.  I picked the location of etc/shells based on it being used by
> getusershell(3) in libc, so it ended up in lib/libc/gen/shells.  The move of
> the init scripts was based on a few things, including the reasons above, but
> additionally:
> 1) Isolating a subcomponent, for example work is happening to bring in
> OpenRC, so we will have multiple init systems with different types of
> scripts, so moving them away from head/etc/ keeps them isolated.

Is the current rc(8) broken? Is OpenRC an intended replacement? How is this
going to work?

>
> This work was driven by needing to tag these files as config files in the
> METALOG so that pkg knows what is a config file so that pkg can 3-way merge
> them.
>
> More info on pkgbase: https://wiki.freebsd.org/PkgBase
>
>
> Regards,
> Brad Davis

--Chris


_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: pkgbase: Move of head/etc/ files

Brad Davis
In reply to this post by Guido Falsi-7
On Fri, Aug 3, 2018, at 4:43 AM, Guido Falsi wrote:

> On 8/2/18 1:54 AM, Brad Davis wrote:
> > Hello,
> >
> > I want to give some information on what is happening to files in head/etc/ and open a discussion.
> >
>
> Today while updating a pkgbase machine I got this error(with trimmed
> debug output):
>
> The following 1 package(s) will be affected (of 0 checked):
>
> Installed packages to be UPGRADED:
>         FreeBSD-runtime: 12.0.s20180725172903 -> 12.0.s20180803092917
> [mpnet-base]
>
> Number of packages to be upgraded: 1
>
> Proceed with this action? [y/N]: y
> DBG(1)[71153]> want to upgrade advisory to exclusive lock
> [1/1] Upgrading FreeBSD-runtime from 12.0.s20180725172903 to
> 12.0.s20180803092917...
> [1/1] Extracting FreeBSD-runtime-12.0.s20180803092917:   1%
> DBG(1)[71153]> Populating config_file /etc/blacklistd.conf
> [1/1] Extracting FreeBSD-runtime-12.0.s20180803092917:   1%
> DBG(1)[71153]> Populating config_file /etc/defaults/rc.conf
> Segmentation fault (core dumped)
>
> I'm not sure if it's related to r336845 and r336847, but since those are
> the only one touching rc.conf recently it looks possible.
>
> Anyone has an idea what I've stumbled upon?
>
> I'm investigating this, but if I can't fix it shortly I'll revert to my
> previous pkg set and try again at another time. If some tests are needed
> I'll try to perform them as requested.

This is a bug in pkg and has to do with these files transitioning from normal files to config files.

I will try and work on this in the next week.


Regards,
Brad Davis
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: pkgbase: Move of head/etc/ files

Brad Davis
In reply to this post by Chris H-2
On Sat, Aug 4, 2018, at 3:43 AM, Chris H wrote:

> On Wed, 01 Aug 2018 17:54:10 -0600 "Brad Davis" <[hidden email]> said
>
> > Hello,
> >
> > I want to give some information on what is happening to files in head/etc/
> > and open a discussion.
> >
> > I am trying to move all config files out of head/etc/ and place them with the
> > source files for the particular use.  For example, some of these are obvious
> > moves like moving say portsnap.conf to usr.sbin/portsnap/portsnap.conf.  The
> > review for this move shows exactly what would happen here:
> > https://reviews.freebsd.org/D16554.  For these cases, there are some easy
> > benefits, like:
> > 1) Keeping all files as part of a program together, which makes maintenance
> > for maintainers a little easier.
> > 2) Simplicity, we do not need all the .if ${MK_FOO} lines in etc/Makefile
> > because that is handled a level above the programs Makefile and so one CONFS=
> > line is all that is needed.
> > 3) Easier to avoid bugs where we install things that should not be installed.
> > Like for apmd, we installed apmd.conf on all archs even though it only works
> > on i386.  i.e. we don't have to keep knobs in sync in multiple locations.
> >
> > Some of the more complicated things to move like etc/shells and the init
> > scripts.  I picked the location of etc/shells based on it being used by
> > getusershell(3) in libc, so it ended up in lib/libc/gen/shells.  The move of
> > the init scripts was based on a few things, including the reasons above, but
> > additionally:
> > 1) Isolating a subcomponent, for example work is happening to bring in
> > OpenRC, so we will have multiple init systems with different types of
> > scripts, so moving them away from head/etc/ keeps them isolated.
>
> Is the current rc(8) broken? Is OpenRC an intended replacement? How is this
> going to work?

I am not sure what the status or progress is for this.  Probably worth contacting Kris Moore.


Regards,
Brad Davis
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: pkgbase: Move of head/etc/ files

Guido Falsi-7
In reply to this post by Brad Davis
On 8/4/18 6:50 PM, Brad Davis wrote:

> On Fri, Aug 3, 2018, at 4:43 AM, Guido Falsi wrote:
>> On 8/2/18 1:54 AM, Brad Davis wrote:
>>> Hello,
>>>
>>> I want to give some information on what is happening to files in head/etc/ and open a discussion.
>>>
>>
>> Today while updating a pkgbase machine I got this error(with trimmed
>> debug output):
>>
>> The following 1 package(s) will be affected (of 0 checked):
>>
>> Installed packages to be UPGRADED:
>>         FreeBSD-runtime: 12.0.s20180725172903 -> 12.0.s20180803092917
>> [mpnet-base]
>>
>> Number of packages to be upgraded: 1
>>
>> Proceed with this action? [y/N]: y
>> DBG(1)[71153]> want to upgrade advisory to exclusive lock
>> [1/1] Upgrading FreeBSD-runtime from 12.0.s20180725172903 to
>> 12.0.s20180803092917...
>> [1/1] Extracting FreeBSD-runtime-12.0.s20180803092917:   1%
>> DBG(1)[71153]> Populating config_file /etc/blacklistd.conf
>> [1/1] Extracting FreeBSD-runtime-12.0.s20180803092917:   1%
>> DBG(1)[71153]> Populating config_file /etc/defaults/rc.conf
>> Segmentation fault (core dumped)
>>
>> I'm not sure if it's related to r336845 and r336847, but since those are
>> the only one touching rc.conf recently it looks possible.
>>
>> Anyone has an idea what I've stumbled upon?
>>
>> I'm investigating this, but if I can't fix it shortly I'll revert to my
>> previous pkg set and try again at another time. If some tests are needed
>> I'll try to perform them as requested.
>
> This is a bug in pkg and has to do with these files transitioning from normal files to config files.
>
> I will try and work on this in the next week.

Thanks in advance!

For me there's no hurry. I have reverted my machine and will not be
updating for at least a full week anyway.

If you need testing I'm available though.

--
Guido Falsi <[hidden email]>
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: pkgbase: Move of head/etc/ files

Guido Falsi-7
On 8/4/18 7:21 PM, Guido Falsi wrote:

> On 8/4/18 6:50 PM, Brad Davis wrote:
>> On Fri, Aug 3, 2018, at 4:43 AM, Guido Falsi wrote:
>>> On 8/2/18 1:54 AM, Brad Davis wrote:
>>>> Hello,
>>>>
>>>> I want to give some information on what is happening to files in head/etc/ and open a discussion.
>>>>
>>>
>>> Today while updating a pkgbase machine I got this error(with trimmed
>>> debug output):
>>>
>>> The following 1 package(s) will be affected (of 0 checked):
>>>
>>> Installed packages to be UPGRADED:
>>>         FreeBSD-runtime: 12.0.s20180725172903 -> 12.0.s20180803092917
>>> [mpnet-base]
>>>
>>> Number of packages to be upgraded: 1
>>>
>>> Proceed with this action? [y/N]: y
>>> DBG(1)[71153]> want to upgrade advisory to exclusive lock
>>> [1/1] Upgrading FreeBSD-runtime from 12.0.s20180725172903 to
>>> 12.0.s20180803092917...
>>> [1/1] Extracting FreeBSD-runtime-12.0.s20180803092917:   1%
>>> DBG(1)[71153]> Populating config_file /etc/blacklistd.conf
>>> [1/1] Extracting FreeBSD-runtime-12.0.s20180803092917:   1%
>>> DBG(1)[71153]> Populating config_file /etc/defaults/rc.conf
>>> Segmentation fault (core dumped)
>>>
>>> I'm not sure if it's related to r336845 and r336847, but since those are
>>> the only one touching rc.conf recently it looks possible.
>>>
>>> Anyone has an idea what I've stumbled upon?
>>>
>>> I'm investigating this, but if I can't fix it shortly I'll revert to my
>>> previous pkg set and try again at another time. If some tests are needed
>>> I'll try to perform them as requested.
>>
>> This is a bug in pkg and has to do with these files transitioning from normal files to config files.
>>
>> I will try and work on this in the next week.
>
> Thanks in advance!
>
> For me there's no hurry. I have reverted my machine and will not be
> updating for at least a full week anyway.
>
> If you need testing I'm available though.
>

Hi I just updated a machine using pkg with this patch added:

https://github.com/freebsd/pkg/commit/30644237c1c655e43911110198fe23a9c835aa24

and it now works fine, so thanks a lot!

--
Guido Falsi <[hidden email]>
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: pkgbase: Move of head/etc/ files

Brad Davis

On Sat, Aug 18, 2018, at 11:34 AM, Guido Falsi wrote:

> On 8/4/18 7:21 PM, Guido Falsi wrote:
> > On 8/4/18 6:50 PM, Brad Davis wrote:
> >> On Fri, Aug 3, 2018, at 4:43 AM, Guido Falsi wrote:
> >>> On 8/2/18 1:54 AM, Brad Davis wrote:
> >>>> Hello,
> >>>>
> >>>> I want to give some information on what is happening to files in head/etc/ and open a discussion.
> >>>>
> >>>
> >>> Today while updating a pkgbase machine I got this error(with trimmed
> >>> debug output):
> >>>
> >>> The following 1 package(s) will be affected (of 0 checked):
> >>>
> >>> Installed packages to be UPGRADED:
> >>>         FreeBSD-runtime: 12.0.s20180725172903 -> 12.0.s20180803092917
> >>> [mpnet-base]
> >>>
> >>> Number of packages to be upgraded: 1
> >>>
> >>> Proceed with this action? [y/N]: y
> >>> DBG(1)[71153]> want to upgrade advisory to exclusive lock
> >>> [1/1] Upgrading FreeBSD-runtime from 12.0.s20180725172903 to
> >>> 12.0.s20180803092917...
> >>> [1/1] Extracting FreeBSD-runtime-12.0.s20180803092917:   1%
> >>> DBG(1)[71153]> Populating config_file /etc/blacklistd.conf
> >>> [1/1] Extracting FreeBSD-runtime-12.0.s20180803092917:   1%
> >>> DBG(1)[71153]> Populating config_file /etc/defaults/rc.conf
> >>> Segmentation fault (core dumped)
> >>>
> >>> I'm not sure if it's related to r336845 and r336847, but since those are
> >>> the only one touching rc.conf recently it looks possible.
> >>>
> >>> Anyone has an idea what I've stumbled upon?
> >>>
> >>> I'm investigating this, but if I can't fix it shortly I'll revert to my
> >>> previous pkg set and try again at another time. If some tests are needed
> >>> I'll try to perform them as requested.
> >>
> >> This is a bug in pkg and has to do with these files transitioning from normal files to config files.
> >>
> >> I will try and work on this in the next week.
> >
> > Thanks in advance!
> >
> > For me there's no hurry. I have reverted my machine and will not be
> > updating for at least a full week anyway.
> >
> > If you need testing I'm available though.
> >
>
> Hi I just updated a machine using pkg with this patch added:
>
> https://github.com/freebsd/pkg/commit/30644237c1c655e43911110198fe23a9c835aa24
>
> and it now works fine, so thanks a lot!

Great, and I just committed this to ports as pkg 1.10.5_2.


Regards,
Brad Davis
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "[hidden email]"