[Bug 237290] PORTS_MODULES - single thread ports build

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

[Bug 237290] PORTS_MODULES - single thread ports build

bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=237290

            Bug ID: 237290
           Summary: PORTS_MODULES - single thread ports build
           Product: Base System
           Version: 12.0-STABLE
          Hardware: Any
                OS: Any
            Status: New
          Severity: Affects Some People
          Priority: ---
         Component: kern
          Assignee: [hidden email]
          Reporter: [hidden email]

Then some ports set via PORTS_MODULES then they builded with single thread,
with takes a lot of time.

Reproduce:
Add to /etc/make.conf
PORTS_MODULES+= graphics/drm-fbsd12.0-kmod

build kernel:
/usr/bin/time -h make -j`getconf NPROCESSORS_ONLN` -s -DALWAYS_CHECK_MAKE
buildkernel

top show only 1 cpu usage during drm-fbsd12.0-kmod port building.

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

[Bug 237290] build(7): PORTS_MODULES does not build in parallel

bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=237290

Kubilay Kocak <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|PORTS_MODULES - single      |build(7): PORTS_MODULES
                   |thread ports build          |does not build in parallel
             Status|New                         |Open
           Keywords|                            |needs-qa, performance
                 CC|                            |[hidden email]

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

[Bug 237290] build(7): PORTS_MODULES does not build in parallel

bugzilla-noreply
In reply to this post by bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=237290

--- Comment #1 from [hidden email] ---
Created attachment 203705
  --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=203705&action=edit
patch

Works for me.

Known issue:  if in /etc/make.conf set WRKDIRPREFIX?=/tmp/ports or
WRKDIRPREFIX=/tmp/ports then it overwrites value that set in kern.post.mk
(/usr/obj...).

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

[Bug 237290] [PATCH] build(7): PORTS_MODULES does not build in parallel

bugzilla-noreply
In reply to this post by bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=237290

[hidden email] changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|build(7): PORTS_MODULES     |[PATCH] build(7):
                   |does not build in parallel  |PORTS_MODULES does not
                   |                            |build in parallel

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

[Bug 237290] [PATCH] build(7): PORTS_MODULES does not build in parallel

bugzilla-noreply
In reply to this post by bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=237290

[hidden email] changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
 Attachment #203705|                            |maintainer-approval?
              Flags|                            |

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

[Bug 237290] [PATCH] build(7): PORTS_MODULES does not build in parallel

bugzilla-noreply
In reply to this post by bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=237290

Mark Linimon <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Keywords|                            |patch

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

[Bug 237290] [PATCH] build(7): PORTS_MODULES does not build in parallel

bugzilla-noreply
In reply to this post by bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=237290

--- Comment #2 from Bryan Drewery <[hidden email]> ---
Don't remove -B please. We do not want to pass -j down to the ports build.
Ports does not support -j. It uses an internal mechanism of MAKE_JOBS. Some
ports support it and others do not. Passing down -j doesn't do what you would
expect.

Separate issue, but it's also not supported to build multiple ports at once
simply via 'make'. Something like Poudriere is needed for that.

# MAKE_JOBS_NUMBER
#               - Override the number of make jobs to be used.  User settable.
# MAKE_JOBS_NUMBER_LIMIT
#               - Set a limit for maximum number of make jobs allowed to be
#                 used.

MAKE_JOBS_NUMBER defaults to the number of CPU threads on the system as long as
you do not set DISABLE_MAKE_JOBS and the port isn't MAKE_JOBS_UNSAFE.

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

[Bug 237290] [PATCH] build(7): PORTS_MODULES does not build in parallel

bugzilla-noreply
In reply to this post by bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=237290

--- Comment #3 from [hidden email] ---
(In reply to Bryan Drewery from comment #2)

Keeping -B is terrible idea: compilation takes too many time.

-j can be filtered out here:
MAKEFLAGS="${MAKEFLAGS:M*:tW:S/^-m /-m_/g:S/ -m /
-m_/g:tw:N-m_*:NMK_AUTO_OBJ=*}"
also I can extract "-j N" N arg and pass it to MAKE_JOBS_NUMBER.
I dont know why -m filered out here.

Now MAKEFLAGS contains -j 8 -s -J15,16 and some other strange staff.
Probably better not pass MAKEFLAGS or keep only known good flags, like -s.

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

[Bug 237290] [PATCH] build(7): PORTS_MODULES does not build in parallel

bugzilla-noreply
In reply to this post by bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=237290

--- Comment #4 from [hidden email] ---
(In reply to rozhuk.im from comment #3)
MAKEFLAGS:  -j 8 -s -D ALWAYS_CHECK_MAKE -J 15,16 -m /usr/src/share/mk -j 8 -s
-D ALWAYS_CHECK_MAKE -J 15,16 -m /usr/src/share/mk -D NO_MODULES_OBJ
.MAKE.LEVEL.ENV=MAKELEVEL KERNEL=kernel TARGET=amd64 TARGET_ARCH=amd64

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

[Bug 237290] [PATCH] build(7): PORTS_MODULES does not build in parallel

bugzilla-noreply
In reply to this post by bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=237290

[hidden email] changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
 Attachment #203705|maintainer-approval?        |
              Flags|                            |
 Attachment #203705|0                           |1
        is obsolete|                            |
 Attachment #203755|                            |maintainer-approval?
              Flags|                            |

--- Comment #5 from [hidden email] ---
Created attachment 203755
  --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=203755&action=edit
more clean MAKEFLAGS

src MAKEFLAGS:  -j 8 -s -D ALWAYS_CHECK_MAKE -J 15,16 -m /usr/src/share/mk -j 8
-s -D ALWAYS_CHECK_MAKE -J 15,16 -m /usr/src/share/mk -D NO_MODULES_OBJ
.MAKE.LEVEL.ENV=MAKELEVEL KERNEL=kernel TARGET=amd64 TARGET_ARCH=amd64

Passed to ports:
MAKE_JOBS_NUMBER: 8
MAKEFLAGS: -s -D ALWAYS_CHECK_MAKE -s -D ALWAYS_CHECK_MAKE -D NO_MODULES_OBJ
.MAKE.LEVEL.ENV=MAKELEVEL KERNEL=kernel TARGET=amd64 TARGET_ARCH=amd64

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

[Bug 237290] [PATCH] build(7): PORTS_MODULES does not build in parallel

bugzilla-noreply
In reply to this post by bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=237290

Mark Linimon <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
          Component|kern                        |conf

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "[hidden email]"