Is pkg site forbidden by brower?

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Is pkg site forbidden by brower?

Yoshihiro Ota
Hi,

Is "403 Forbidden" an intended response for a brower access to http://pkg.freebsd.org/FreeBSD:12:i386/ nowdays?

I used to see available packages with a brower and decided which one to use.
How can I find distributions like "latest", "release_X", etc?

Hiro

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Is pkg site forbidden by brower?

Kevin Oberman-4
On Sat, Sep 5, 2020 at 8:04 PM Yoshihiro Ota <[hidden email]> wrote:

> Hi,
>
> Is "403 Forbidden" an intended response for a brower access to
> http://pkg.freebsd.org/FreeBSD:12:i386/ nowdays?
>
> I used to see available packages with a brower and decided which one to
> use.
> How can I find distributions like "latest", "release_X", etc?
>
> Hiro


Does https://pkg-status.freebsd.org/builds?jailname=121amd64 have what you
want? I can't believe that there is no way to see a log of failed builds,
but I can only see the new failures and no information on previous builds.
--
Kevin Oberman, Part time kid herder and retired Network Engineer
E-mail: [hidden email]
PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Is pkg site forbidden by brower?

grarpamp
On 9/6/20, Kevin Oberman <[hidden email]> wrote:
> On Sat, Sep 5, 2020 at 8:04 PM Yoshihiro Ota <[hidden email]> wrote:
>> Is "403 Forbidden" an intended response for a brower access to
>> http://pkg.freebsd.org/FreeBSD:12:i386/ nowdays?
>>
>> I used to see available packages with a brower and decided which one to
>> use.

Some more people have noted this change
as breaking tool scripts, etc.

And useful meta files are unfortunately now invisible:
packagesite.txz, meta.txz, pkg.txz, pkg.txz.sig

If someone want to block the '/.../All/' dir full of pkgs,
maybe, but do not block any other part of the hier.

>> How can I find distributions like "latest", "release_X", etc?

Yes, there does not appear to be any docs enumerating all
the available live names for use in PACKAGESITE url.
Reopening the above dirs would be self documenting.

The name for the term in <term> position of /${ABI}/<term>/All/...
might be "REPOSITORY_ROOT" or "repo-path" or simply "repository",
but it does not seem defined for users in pkg or pkg.conf manpages.
"distribution" is unlikely the correct term, "branch" might be
a useful connotation regarding ports source tree.

> Does https://pkg-status.freebsd.org/builds?jailname=121amd64 have what you
> want?

Those names don't correspond 1:1 to anything on pkg.freebsd.org.

> I can't believe that there is no way to see a log of failed builds,
> but I can only see the new failures and no information on previous builds.

Pkg buildlogs are a separate issue.
They should be available for browsing, same as kernel, base...
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Is pkg site forbidden by brower?

Niclas Zeising-5
On 2020-09-06 09:00, grarpamp wrote:

> On 9/6/20, Kevin Oberman <[hidden email]> wrote:
>> On Sat, Sep 5, 2020 at 8:04 PM Yoshihiro Ota <[hidden email]> wrote:
>>> Is "403 Forbidden" an intended response for a brower access to
>>> http://pkg.freebsd.org/FreeBSD:12:i386/ nowdays?
>>>
>>> I used to see available packages with a brower and decided which one to
>>> use.
>
> Some more people have noted this change
> as breaking tool scripts, etc.
>
> And useful meta files are unfortunately now invisible:
> packagesite.txz, meta.txz, pkg.txz, pkg.txz.sig
>
> If someone want to block the '/.../All/' dir full of pkgs,
> maybe, but do not block any other part of the hier.

The reason that folder listing was disabled on the package download
sites is that it used too much resources.  For every hit on those URLs,
the web server had to dynamically generate the folder listing, and send
it.  This caused DDoS-like scenarios, where these were hit repeatedly,
which caused problems for legitimate traffic.  Since the relevant
information is available in the txz files above, and also on freshports,
and since pkg have no need for directory listing, it was disabled.

I would suggest using freshports.org, which has information on which
version of a package is available for the various FreeBSD versions and
architectures, both in the latest and the quarterly branch.

>
>>> How can I find distributions like "latest", "release_X", etc?
>
> Yes, there does not appear to be any docs enumerating all
> the available live names for use in PACKAGESITE url.
> Reopening the above dirs would be self documenting.

I am not sure what you are looking for here.  Can you explain the use
case, what are you trying to accomplish?

>
> The name for the term in <term> position of /${ABI}/<term>/All/...
> might be "REPOSITORY_ROOT" or "repo-path" or simply "repository",
> but it does not seem defined for users in pkg or pkg.conf manpages.
> "distribution" is unlikely the correct term, "branch" might be
> a useful connotation regarding ports source tree.

Once again, I'm not sure what you are looking for.  Have you looked at
the manual for pkg.conf, which is fairly extensive and have several
examples.

>
>> Does https://pkg-status.freebsd.org/builds?jailname=121amd64 have what you
>> want?
>
> Those names don't correspond 1:1 to anything on pkg.freebsd.org.

Actually, they do. You can see both the ports tree built (default for
top of the ports tree, and quarterly for the quarterly branch), as well
as architecture and FreeBSD version.  You can even see exactly which svn
revision is used.

>
>> I can't believe that there is no way to see a log of failed builds,
>> but I can only see the new failures and no information on previous builds.
>
> Pkg buildlogs are a separate issue.
> They should be available for browsing, same as kernel, base...

Build logs are available, but not all of them are available over IPv4,
since IPv4 addresses are scarce.  If you open a specific builder, you
can see a list of new failures, and links to the build logs.  There are
also links to previous builds, so that you can compare, and find earlier
failures.

Regards
--
Niclas
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Is pkg site forbidden by brower?

Michael Gmelin-2


> On 6. Sep 2020, at 12:00, Niclas Zeising <[hidden email]> wrote:
>
> On 2020-09-06 09:00, grarpamp wrote:
>>> On 9/6/20, Kevin Oberman <[hidden email]> wrote:
>>> On Sat, Sep 5, 2020 at 8:04 PM Yoshihiro Ota <[hidden email]> wrote:
>>>> Is "403 Forbidden" an intended response for a brower access to
>>>> http://pkg.freebsd.org/FreeBSD:12:i386/ nowdays?
>>>>
>>>> I used to see available packages with a brower and decided which one to
>>>> use.
>> Some more people have noted this change
>> as breaking tool scripts, etc.
>> And useful meta files are unfortunately now invisible:
>> packagesite.txz, meta.txz, pkg.txz, pkg.txz.sig
>> If someone want to block the '/.../All/' dir full of pkgs,
>> maybe, but do not block any other part of the hier.
>
> The reason that folder listing was disabled on the package download sites is that it used too much resources.  For every hit on those URLs, the web server had to dynamically generate the folder listing, and send it.  This caused DDoS-like scenarios, where these were hit repeatedly, which caused problems for legitimate traffic.  Since the relevant information is available in the txz files above, and also on freshports, and since pkg have no need for directory listing, it was disabled.
>

Is this part of why pkg repos are performing so much better recently? I‘m quite happy about that :)

If there’s a use case for having access to this information, we could simply provide it through a static index.html that’s recreated every time the directory changes.

Cheers,
Michael


_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Is pkg site forbidden by brower?

Yoshihiro Ota
On Sun, 6 Sep 2020 14:47:00 +0200
Michael Gmelin <[hidden email]> wrote:

>
>
> > On 6. Sep 2020, at 12:00, Niclas Zeising <[hidden email]> wrote:
> >
> > 〓On 2020-09-06 09:00, grarpamp wrote:
> >>> On 9/6/20, Kevin Oberman <[hidden email]> wrote:
> >>> On Sat, Sep 5, 2020 at 8:04 PM Yoshihiro Ota <[hidden email]> wrote:
> >>>> Is "403 Forbidden" an intended response for a brower access to
> >>>> http://pkg.freebsd.org/FreeBSD:12:i386/ nowdays?
> >>>>
> >>>> I used to see available packages with a brower and decided which one to
> >>>> use.
> >> Some more people have noted this change
> >> as breaking tool scripts, etc.
> >> And useful meta files are unfortunately now invisible:
> >> packagesite.txz, meta.txz, pkg.txz, pkg.txz.sig
> >> If someone want to block the '/.../All/' dir full of pkgs,
> >> maybe, but do not block any other part of the hier.
> >
> > The reason that folder listing was disabled on the package download sites is that it used too
> > much resources.  For every hit on those URLs, the web server had to dynamically generate the
> > folder listing, and send it.  This caused DDoS-like scenarios, where these were hit repeatedly,
> > which caused problems for legitimate traffic.  Since the relevant information is available in
> > the txz files above, and also on freshports, and since pkg have no need for directory listing,
> > it was disabled.
> >
>
> Is this part of why pkg repos are performing so much better recently? I‘m quite happy about
> that :)
>
> If there’s a use case for having access to this information, we could simply provide it through a
> static index.html that’s recreated every time the directory changes.
>
> Cheers,
> Michael

Michael,

Do you own pkg sites?

I like the idea of creating static index.html file as there are modules with version numbers and also modules with multiple version numbers.
Brower helps to find which versions are available, for example.

Regards,
Hiro
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Is pkg site forbidden by brower?

Michael Gmelin-2


> On 6. Sep 2020, at 21:09, Yoshihiro Ota <[hidden email]> wrote:
>
> On Sun, 6 Sep 2020 14:47:00 +0200
> Michael Gmelin <[hidden email]> wrote:
>
>>
>>
>>>> On 6. Sep 2020, at 12:00, Niclas Zeising <[hidden email]> wrote:
>>>
>>> 〓On 2020-09-06 09:00, grarpamp wrote:
>>>>> On 9/6/20, Kevin Oberman <[hidden email]> wrote:
>>>>> On Sat, Sep 5, 2020 at 8:04 PM Yoshihiro Ota <[hidden email]> wrote:
>>>>>> Is "403 Forbidden" an intended response for a brower access to
>>>>>> http://pkg.freebsd.org/FreeBSD:12:i386/ nowdays?
>>>>>>
>>>>>> I used to see available packages with a brower and decided which one to
>>>>>> use.
>>>> Some more people have noted this change
>>>> as breaking tool scripts, etc.
>>>> And useful meta files are unfortunately now invisible:
>>>> packagesite.txz, meta.txz, pkg.txz, pkg.txz.sig
>>>> If someone want to block the '/.../All/' dir full of pkgs,
>>>> maybe, but do not block any other part of the hier.
>>>
>>> The reason that folder listing was disabled on the package download sites is that it used too
>>> much resources.  For every hit on those URLs, the web server had to dynamically generate the
>>> folder listing, and send it.  This caused DDoS-like scenarios, where these were hit repeatedly,
>>> which caused problems for legitimate traffic.  Since the relevant information is available in
>>> the txz files above, and also on freshports, and since pkg have no need for directory listing,
>>> it was disabled.
>>>
>>
>> Is this part of why pkg repos are performing so much better recently? I‘m quite happy about
>> that :)
>>
>> If there’s a use case for having access to this information, we could simply provide it through a
>> static index.html that’s recreated every time the directory changes.
>>
>> Cheers,
>> Michael
>
> Michael,
>
> Do you own pkg sites?

Unfortunately not, but you could ask clusteradm@ about that.

Cheers,
Michael

>
> I like the idea of creating static index.html file as there are modules with version numbers and also modules with multiple version numbers.
> Brower helps to find which versions are available, for example.
>
> Regards,
> Hiro
> _______________________________________________
> [hidden email] mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "[hidden email]"

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Is pkg site forbidden by brower?

Mark Linimon-2
In reply to this post by Yoshihiro Ota
On Sun, Sep 06, 2020 at 03:05:49PM -0400, Yoshihiro Ota wrote:
> Do you own pkg sites?

Only portmgr@ does that.  So this matter can only be resolved by a
discussion between portmgr@ and clusteradm@.

mcl
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"