Centralized building

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Centralized building

Eirik Øverby
Hi all!

I've spent about a week trying to accomplish a rather simple task: To  
build kernel and world once for each architecture we have, and  
distribute this precompiled src and obj tree via NFS to all the  
systems that need updating. I have combined this with a locally  
maintained CVS tree, in order to assure coherent releases being  
installed on all our systems.

However, I am seeing some peculiar issues that I simply don't manage  
to get around.
Scenario:

I've got one server running 6.0-STABLE-i386. On this host I've  
created a jail for building. We have both i386 and amd64 platforms in-
house, so I've created a script that build for both:
  make TARGET_ARCH=i386  MAKEOBJDIRPREFIX=/usr/obj.i386  buildworld
  make TARGET_ARCH=amd64 MAKEOBJDIRPREFIX=/usr/obj.amd64 buildworld
And the same for buildkernel.

Starting out trying to upgrade the amd64 hosts, I export the two obj  
directories via NFS, and mount them as /usr/obj on the amd64 hosts  
that need upgrading. This was, at least, my initial approach. I then  
found out that the /usr/src tree in the build jail is somehow tainted  
by the build (and by the options I specified), so I need to export  
that as well (which, I am afraid, means I have to maintain two  
different build jails). Therefore I also export /usr/src and mount it  
on the target hosts.

I then realized that I need to use the same objdir on the target  
hosts as in the build jail, so I try mounting to /usr/obj.<arch> on  
the target hosts. This allows me to get somewhat further.  
Installworld now progresses for a while, until it bombs out with the  
following error:
   ===> sys/boot/i386/boot2 (install)
   dd if=/dev/zero of=boot2.ldr bs=276 count=1
   dd: not found
   *** Error code 127

When looking for dd, I find it in the host PATH, and also in the obj  
dir:
   [root@build] /usr/obj.amd64# find . -name dd -type f
   ./amd64/usr/src/bin/dd/dd

At this point, I get rid of the MAKEOBJDIRPREFIX option and rebuild  
everything with just TARGET_ARCH, only exporting /usr/obj from the  
build jail. I notice that when using TARGET_ARCH with something else  
than the architecture the build is running on (i.e. amd64 on an i386  
host), the resulting build is NOT to be found in /usr/obj, but in /
usr/obj/amd64. Thus I need to specify MAKEOBJDIRPREFIX=/usr/obj/amd64  
on the target host for installworld to get anything done at all.

I'm still getting the dd: not found error, and I do believe I've  
tried every combination and variation I can think of. Clocks are in  
sync between all the systems, so that is not the problem. Is the  
build system partially broken in 6.0? Have I missed something? Do I  
actually need an amd64 host to be able to build for amd64 systems, or  
are there other ways to accomplish what I'm trying to do? Should I  
prehaps try doing centralized binary upgrades instead?

Any help would be appreciated.
With best regards,
/Eirik

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Centralized building

Joseph Koshy
> Starting out trying to upgrade the amd64 hosts, I export the
> two obj directories via NFS, and mount them as /usr/obj on the
> amd64 hosts that need upgrading.

I done upgrades the other way, by having the build machine
mount the clients to-be-root partition and installing to it
using NFS.

--
FreeBSD Volunteer,     http://people.freebsd.org/~jkoshy
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Centralized building

Eirik Øverby

On Nov 19, 2005, at 13:28 , Joseph Koshy wrote:

>> Starting out trying to upgrade the amd64 hosts, I export the
>> two obj directories via NFS, and mount them as /usr/obj on the
>> amd64 hosts that need upgrading.
>
> I done upgrades the other way, by having the build machine
> mount the clients to-be-root partition and installing to it
> using NFS.

Would I have to export every filesystem mount point on the hosts  
then? Or does an -alldirs option do the trick (in exports)?

In any case this would not be compatible with our security policy,  
unfortunately.

/Eirik


_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Centralized building

Joseph Koshy
> Would I have to export every filesystem mount point on the
> hosts then? Or does an -alldirs option do the trick (in
> exports)?

I use a self-contained '/' partition on the machines where I
need to cross-install.

Note that one still needs to run mergemaster or etcmerge after
the cross-install step.

You may want to look at: http://trac.t7a.org/isconf/ and
http://www.infrastructures.org/.

--
FreeBSD Volunteer,     http://people.freebsd.org/~jkoshy
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Centralized building

Brandon Fosdick
In reply to this post by Eirik Øverby
Eirik Øverby wrote:
> I've spent about a week trying to accomplish a rather simple task: To
> build kernel and world once for each architecture we have, and
> distribute this precompiled src and obj tree via NFS to all the  systems
> that need updating. I have combined this with a locally  maintained CVS
> tree, in order to assure coherent releases being  installed on all our
> systems.

AFAICT cross-compiling amd64 on a i386 machine isn't supported yet. I ran into a similar problem when I upgraded an i386 machine to amd64. I thought I could just set CPUTYPE=athlon-64 and buildworld would do the right thing. Apparently not.

I haven't bothered to try building i386 on an amd64 box though. Maybe you'll have better luck with that.
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Centralized building

Joseph Koshy
> AFAICT cross-compiling amd64 on a i386 machine isn't supported
> yet. I ran into a similar problem when I upgraded an i386
> machine to amd64. I thought I could just set CPUTYPE=athlon-64
> and buildworld would do the right thing. Apparently not.

Bootstrapping a single machine is supported:

# make buildworld TARGET_ARCH=new-arch

plus a few other steps.  (See build(7)).

There have been a couple of postings on the mailing lists
on this topic in the recent past.  I've taken a stab at
describing how to cross-bootstrap too:

http://edoofus.blogspot.com/2005/10/cross-building-freebsd.html

The OP wanted to do a 'buildworld TARGET_ARCH=foo' on one
machine and then an 'installworld' on a different set of
machines.

--
FreeBSD Volunteer,     http://people.freebsd.org/~jkoshy
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Centralized building

Eirik Øverby

On Nov 19, 2005, at 19:43 , Joseph Koshy wrote:

>> AFAICT cross-compiling amd64 on a i386 machine isn't supported
>> yet. I ran into a similar problem when I upgraded an i386
>> machine to amd64. I thought I could just set CPUTYPE=athlon-64
>> and buildworld would do the right thing. Apparently not.
>
> Bootstrapping a single machine is supported:
>
> # make buildworld TARGET_ARCH=new-arch
>
> plus a few other steps.  (See build(7)).
>
> There have been a couple of postings on the mailing lists
> on this topic in the recent past.  I've taken a stab at
> describing how to cross-bootstrap too:
>
> http://edoofus.blogspot.com/2005/10/cross-building-freebsd.html
>
> The OP wanted to do a 'buildworld TARGET_ARCH=foo' on one
> machine and then an 'installworld' on a different set of
> machines.

Yes, and he still wonders if this is supposed to be doable or not.
I think the culprit is (partly) the fact that every architecture is  
built into its own subdirectory in /usr/obj, EXCEPT the architecture  
the build is running on. The same goes for the install part, and if  
the build and install architectures differ, it cannot ever work.  
Setting MAKEOBJDIRPREFIX on the target host makes the install start,  
but it fails after a couple of minutes with the "dd: not found" error.
(I do notice that there is a /usr/obj/usr directory created also when  
cross-building; I'm assuming this contains the build bootstrap tools).



>
> --
> FreeBSD Volunteer,     http://people.freebsd.org/~jkoshy
>
>

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Centralized building

Eirik Øverby

On Nov 20, 2005, at 09:50 , Eirik Øverby wrote:

>
> On Nov 19, 2005, at 19:43 , Joseph Koshy wrote:
>
>>> AFAICT cross-compiling amd64 on a i386 machine isn't supported
>>> yet. I ran into a similar problem when I upgraded an i386
>>> machine to amd64. I thought I could just set CPUTYPE=athlon-64
>>> and buildworld would do the right thing. Apparently not.
>>
>> Bootstrapping a single machine is supported:
>>
>> # make buildworld TARGET_ARCH=new-arch
>>
>> plus a few other steps.  (See build(7)).
>>
>> There have been a couple of postings on the mailing lists
>> on this topic in the recent past.  I've taken a stab at
>> describing how to cross-bootstrap too:
>>
>> http://edoofus.blogspot.com/2005/10/cross-building-freebsd.html
>>
>> The OP wanted to do a 'buildworld TARGET_ARCH=foo' on one
>> machine and then an 'installworld' on a different set of
>> machines.
>
> Yes, and he still wonders if this is supposed to be doable or not.
> I think the culprit is (partly) the fact that every architecture is  
> built into its own subdirectory in /usr/obj, EXCEPT the  
> architecture the build is running on. The same goes for the install  
> part, and if the build and install architectures differ, it cannot  
> ever work. Setting MAKEOBJDIRPREFIX on the target host makes the  
> install start, but it fails after a couple of minutes with the "dd:  
> not found" error.
> (I do notice that there is a /usr/obj/usr directory created also  
> when cross-building; I'm assuming this contains the build bootstrap  
> tools).

Follow-up. If I enter src/sys and do a "make install", the dd step  
works perfectly - however it stops later when trying to install  
cdboot. I am assuming this is due to missing options or wrong target  
for make, but - from all I can tell - shows a weakness in the build/
install system. Or maybe not...

Anyone??

/Eirik

>
>
>
>>
>> --
>> FreeBSD Volunteer,     http://people.freebsd.org/~jkoshy
>>
>>
>
> _______________________________________________
> [hidden email] mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-
> [hidden email]"
>
>

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Centralized building

Ruslan Ermilov
On Sun, Nov 20, 2005 at 10:14:37AM +0100, Eirik ?verby wrote:

>
> On Nov 20, 2005, at 09:50 , Eirik ?verby wrote:
>
> >
> >On Nov 19, 2005, at 19:43 , Joseph Koshy wrote:
> >
> >>>AFAICT cross-compiling amd64 on a i386 machine isn't supported
> >>>yet. I ran into a similar problem when I upgraded an i386
> >>>machine to amd64. I thought I could just set CPUTYPE=athlon-64
> >>>and buildworld would do the right thing. Apparently not.
> >>
> >>Bootstrapping a single machine is supported:
> >>
> >># make buildworld TARGET_ARCH=new-arch
> >>
> >>plus a few other steps.  (See build(7)).
> >>
> >>There have been a couple of postings on the mailing lists
> >>on this topic in the recent past.  I've taken a stab at
> >>describing how to cross-bootstrap too:
> >>
> >>http://edoofus.blogspot.com/2005/10/cross-building-freebsd.html
> >>
> >>The OP wanted to do a 'buildworld TARGET_ARCH=foo' on one
> >>machine and then an 'installworld' on a different set of
> >>machines.
> >
> >Yes, and he still wonders if this is supposed to be doable or not.
> >I think the culprit is (partly) the fact that every architecture is  
> >built into its own subdirectory in /usr/obj, EXCEPT the  
> >architecture the build is running on. The same goes for the install  
> >part, and if the build and install architectures differ, it cannot  
> >ever work. Setting MAKEOBJDIRPREFIX on the target host makes the  
> >install start, but it fails after a couple of minutes with the "dd:  
> >not found" error.
> >(I do notice that there is a /usr/obj/usr directory created also  
> >when cross-building; I'm assuming this contains the build bootstrap  
> >tools).
>
> Follow-up. If I enter src/sys and do a "make install", the dd step  
> works perfectly - however it stops later when trying to install  
> cdboot. I am assuming this is due to missing options or wrong target  
> for make, but - from all I can tell - shows a weakness in the build/
> install system. Or maybe not...
>
> Anyone??
>
We don't support build host != install host, in a strict sense.
But as Joseph pointed out, we do support NFS installs to different
architectures.  The build host != install host is supported only
if a number of conditions are met, most noticeable are that they
should be running the same OS version and the same (or compatible)
CPU, and of course the same set of options (/etc/make.conf, etc.)


Cheers,
--
Ruslan Ermilov
[hidden email]
FreeBSD committer

attachment0 (194 bytes) Download Attachment