[Bug 241148] Mk/Scripts/do-fetch.sh: script made by fetch-list fetches the same file several times

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

[Bug 241148] Mk/Scripts/do-fetch.sh: script made by fetch-list fetches the same file several times

bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=241148

            Bug ID: 241148
           Summary: Mk/Scripts/do-fetch.sh: script made by fetch-list
                    fetches the same file several times
           Product: Ports & Packages
           Version: Latest
          Hardware: Any
                OS: Any
            Status: New
          Severity: Affects Only Me
          Priority: ---
         Component: Individual Port(s)
          Assignee: [hidden email]
          Reporter: [hidden email]
                CC: [hidden email]
             Flags: maintainer-feedback?([hidden email])
          Assignee: [hidden email]

After r507705[1] fetch-list for ports with DISTFILES having entries with
subdirectories, produces shell script that causes downloading of the same file
several time.

For example:

$ cd /usr/ports/lang/rust
$ make fetch-list
mkdir -p /usr/ports/distfiles/rust && cd /usr/ports/distfiles/rust && { env
/usr/bin/fetch -Fpr  -S 96163304
https://static.rust-lang.org/dist/rustc-1.38.0-src.tar.xz  || env
/usr/bin/fetch -Fpr  -S 96163304
http://distcache.FreeBSD.org/ports-distfiles/rust/rustc-1.38.0-src.tar.xz  ||
echo "rustc-1.38.0-src.tar.xz" not fetched; }
mkdir -p /usr/ports/distfiles/rust && cd /usr/ports/distfiles/rust && { mkdir
-p "2019-08-15" &&
env /usr/bin/fetch -Fpr  -S 70318740 -o
2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz
https://static.rust-lang.org/dist/2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz
 || mkdir -p "2019-08-15" &&
env /usr/bin/fetch -Fpr  -S 70318740 -o
2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz
http://distcache.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz
 || mkdir -p "2019-08-15" &&
env /usr/bin/fetch -Fpr  -S 70318740 -o
2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz
http://distcache.us-east.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz
 || mkdir -p "2019-08-15" &&
env /usr/bin/fetch -Fpr  -S 70318740 -o
2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz
http://distcache.eu.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz
 || mkdir -p "2019-08-15" &&
env /usr/bin/fetch -Fpr  -S 70318740 -o
2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz
http://distcache.us-west.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz
 || mkdir -p "2019-08-15" &&
env /usr/bin/fetch -Fpr  -S 70318740 -o
2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz
http://distcache.FreeBSD.org/ports-distfiles/rust/2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz
 || echo "2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz" not fetched; }
mkdir -p /usr/ports/distfiles/rust && cd /usr/ports/distfiles/rust && { mkdir
-p "2019-08-15" &&
env /usr/bin/fetch -Fpr  -S 87956779 -o
2019-08-15/rust-std-1.37.0-x86_64-unknown-freebsd.tar.gz
https://static.rust-lang.org/dist/2019-08-15/rust-std-1.37.0-x86_64-unknown-freebsd.tar.gz
 || mkdir -p "2019-08-15" &&
env /usr/bin/fetch -Fpr  -S 87956779 -o
2019-08-15/rust-std-1.37.0-x86_64-unknown-freebsd.tar.gz
http://distcache.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rust-std-1.37.0-x86_64-unknown-freebsd.tar.gz
 || mkdir -p "2019-08-15" &&
env /usr/bin/fetch -Fpr  -S 87956779 -o
2019-08-15/rust-std-1.37.0-x86_64-unknown-freebsd.tar.gz
http://distcache.us-east.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rust-std-1.37.0-x86_64-unknown-freebsd.tar.gz
 || mkdir -p "2019-08-15" &&
env /usr/bin/fetch -Fpr  -S 87956779 -o
2019-08-15/rust-std-1.37.0-x86_64-unknown-freebsd.tar.gz
http://distcache.eu.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rust-std-1.37.0-x86_64-unknown-freebsd.tar.gz
 || mkdir -p "2019-08-15" &&
env /usr/bin/fetch -Fpr  -S 87956779 -o
2019-08-15/rust-std-1.37.0-x86_64-unknown-freebsd.tar.gz
http://distcache.us-west.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rust-std-1.37.0-x86_64-unknown-freebsd.tar.gz
 || mkdir -p "2019-08-15" &&
env /usr/bin/fetch -Fpr  -S 87956779 -o
2019-08-15/rust-std-1.37.0-x86_64-unknown-freebsd.tar.gz
http://distcache.FreeBSD.org/ports-distfiles/rust/2019-08-15/rust-std-1.37.0-x86_64-unknown-freebsd.tar.gz
 || echo "2019-08-15/rust-std-1.37.0-x86_64-unknown-freebsd.tar.gz" not
fetched; }
mkdir -p /usr/ports/distfiles/rust && cd /usr/ports/distfiles/rust && { mkdir
-p "2019-08-15" &&
env /usr/bin/fetch -Fpr  -S 6712004 -o
2019-08-15/cargo-0.38.0-x86_64-unknown-freebsd.tar.gz
https://static.rust-lang.org/dist/2019-08-15/cargo-0.38.0-x86_64-unknown-freebsd.tar.gz
 || mkdir -p "2019-08-15" &&
env /usr/bin/fetch -Fpr  -S 6712004 -o
2019-08-15/cargo-0.38.0-x86_64-unknown-freebsd.tar.gz
http://distcache.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/cargo-0.38.0-x86_64-unknown-freebsd.tar.gz
 || mkdir -p "2019-08-15" &&
env /usr/bin/fetch -Fpr  -S 6712004 -o
2019-08-15/cargo-0.38.0-x86_64-unknown-freebsd.tar.gz
http://distcache.us-east.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/cargo-0.38.0-x86_64-unknown-freebsd.tar.gz
 || mkdir -p "2019-08-15" &&
env /usr/bin/fetch -Fpr  -S 6712004 -o
2019-08-15/cargo-0.38.0-x86_64-unknown-freebsd.tar.gz
http://distcache.eu.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/cargo-0.38.0-x86_64-unknown-freebsd.tar.gz
 || mkdir -p "2019-08-15" &&
env /usr/bin/fetch -Fpr  -S 6712004 -o
2019-08-15/cargo-0.38.0-x86_64-unknown-freebsd.tar.gz
http://distcache.us-west.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/cargo-0.38.0-x86_64-unknown-freebsd.tar.gz
 || mkdir -p "2019-08-15" &&
env /usr/bin/fetch -Fpr  -S 6712004 -o
2019-08-15/cargo-0.38.0-x86_64-unknown-freebsd.tar.gz
http://distcache.FreeBSD.org/ports-distfiles/rust/2019-08-15/cargo-0.38.0-x86_64-unknown-freebsd.tar.gz
 || echo "2019-08-15/cargo-0.38.0-x86_64-unknown-freebsd.tar.gz" not fetched; }

Execution of that script ends up with this result:

$ su root -c "/usr/home/user/fetch_rust.sh"
Password:
fetch: https://static.rust-lang.org/dist/rustc-1.38.0-src.tar.xz: size unknown
fetch: https://static.rust-lang.org/dist/rustc-1.38.0-src.tar.xz: size of
remote file is not known
rustc-1.38.0-src.tar.xz                                 91 MB 4241 kBps    22s
2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd          67 MB 3724 kBps    08s
fetch:
http://distcache.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz:
Not Found
fetch:
http://distcache.us-east.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz:
Not Found
fetch:
http://distcache.eu.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz:
Not Found
fetch:
http://distcache.us-west.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz:
Not Found
2019-08-15/rust-std-1.37.0-x86_64-unknown-free          83 MB 4212 kBps    21s
fetch:
http://distcache.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rust-std-1.37.0-x86_64-unknown-freebsd.tar.gz:
Not Found
fetch:
http://distcache.us-east.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rust-std-1.37.0-x86_64-unknown-freebsd.tar.gz:
Not Found
fetch:
http://distcache.eu.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rust-std-1.37.0-x86_64-unknown-freebsd.tar.gz:
Not Found
fetch:
http://distcache.us-west.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rust-std-1.37.0-x86_64-unknown-freebsd.tar.gz:
Not Found
2019-08-15/cargo-0.38.0-x86_64-unknown-freebsd        6554 kB 3455 kBps    02s
fetch:
http://distcache.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/cargo-0.38.0-x86_64-unknown-freebsd.tar.gz:
Not Found
fetch:
http://distcache.us-east.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/cargo-0.38.0-x86_64-unknown-freebsd.tar.gz:
Not Found
fetch:
http://distcache.eu.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/cargo-0.38.0-x86_64-unknown-freebsd.tar.gz:
Not Found
fetch:
http://distcache.us-west.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/cargo-0.38.0-x86_64-unknown-freebsd.tar.gz:
Not Found

As you can see, the script tries to download, for example,
rustc-1.37.0-x86_64-unknown-freebsd.tar.gz five times.  Even after the script
has downloaded it successfully on the first try.

That happens because of the following lines in fetch-list's script:

mkdir -p /usr/ports/distfiles/rust && cd /usr/ports/distfiles/rust && { mkdir
-p "2019-08-15" &&
env /usr/bin/fetch -Fpr  -S 70318740 -o
2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz
https://static.rust-lang.org/dist/2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz
 || mkdir -p "2019-08-15" &&
env /usr/bin/fetch -Fpr  -S 70318740 -o
2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz
http://distcache.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz
 || mkdir -p "2019-08-15" &&
env /usr/bin/fetch -Fpr  -S 70318740 -o
2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz
http://distcache.us-east.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz
 || mkdir -p "2019-08-15" &&
env /usr/bin/fetch -Fpr  -S 70318740 -o
2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz
http://distcache.eu.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz
 || mkdir -p "2019-08-15" &&
env /usr/bin/fetch -Fpr  -S 70318740 -o
2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz
http://distcache.us-west.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz
 || mkdir -p "2019-08-15" &&
env /usr/bin/fetch -Fpr  -S 70318740 -o
2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz
http://distcache.FreeBSD.org/ports-distfiles/rust/2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz
 || echo "2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz" not fetched; }

It makes "2019-08-15", downloads (successfully)
rustc-1.37.0-x86_64-unknown-freebsd.tar.gz from
https://static.rust-lang.org/dist/2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz
(having zero exit status), skips `mkdir -p "2019-08-15"` (because of "no
errors" exit status), and fetches the file again after `&&` (once again --
because of zero exit status).  This `fetch` is subsequently failing, but
`mkdir` after that `fetch` succeeds (because it had `-p` specified).  The
script `fetch`es again.  And so on.

I believe `mkdir` and `fetch` must be grouped together between all those `||`
operators.

See also: bug 239293[2]

P.S. If someone will fix this, please also take bug 239761[3] into account,
where my patch allows to combine several fetching lines into single one, as
usual.

[1] https://svnweb.freebsd.org/ports?view=revision&revision=507705
[2] https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=239293
[3] https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=239761

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports-bugs
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

[Bug 241148] Mk/Scripts/do-fetch.sh: script made by fetch-list fetches the same file several times

bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=241148

Tobias Kortkamp <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
              Flags|maintainer-feedback?(rust@F |
                   |reeBSD.org)                 |
          Component|Individual Port(s)          |Ports Framework
           Assignee|[hidden email]            |[hidden email]
                 CC|                            |[hidden email]

--- Comment #1 from Tobias Kortkamp <[hidden email]> ---
This has little to do with lang/rust (except that it was used as an example)
and rust@ does not need to be involved here.

--
You are receiving this mail because:
You are on the CC list for the bug.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports-bugs
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

[Bug 241148] Mk/Scripts/do-fetch.sh: script made by fetch-list fetches the same file several times

bugzilla-noreply
In reply to this post by bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=241148

Ruslan Garipov <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
              Flags|                            |maintainer-feedback?

--- Comment #2 from Ruslan Garipov <[hidden email]> ---
(In reply to Tobias Kortkamp from comment #1)

> This has little to do with lang/rust
Yes.  But assignment to the rust team was made automatically, I didn't assigned
them:

> Bugzilla Automation 2019-10-09 09:52:19 UTC
> Assignee: [hidden email][hidden email]

So, rust@, I'm sorry for bother.

--
You are receiving this mail because:
You are on the CC list for the bug.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports-bugs
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

[Bug 241148] Mk/Scripts/do-fetch.sh: script made by fetch-list fetches the same file several times

bugzilla-noreply
In reply to this post by bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=241148

Adam Weinberger <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]

--- Comment #3 from Adam Weinberger <[hidden email]> ---
Tobias wasn't chiding you, and yes bugzilla does the initial assignment
automatically. He was adding a note documenting why he reassigned it to
portmgr.

--
You are receiving this mail because:
You are on the CC list for the bug.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports-bugs
To unsubscribe, send any mail to "[hidden email]"