pkgng mirrors

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

pkgng mirrors

Sergey V. Dyatko
Hi,

Today I setup test jail on my laptop with head/amd64 and trying to use
pkgng, as I can see we have just one(?) host for pkgbeta.freebsd.org:

pkgbeta.freebsd.org.    3600    IN      CNAME   ftp.dk.freebsd.org.
ftp.dk.freebsd.org.     3600    IN      CNAME   ftp.beastie.tdk.net.
ftp.beastie.tdk.net.    80      IN      A       193.162.146.4

that host doesn't respond 'quickly' (for me):
[tiger@laptop]:~%curl -w '%{time_total}: %{time_connect}
%{time_starttransfer} %{size_download} \n' -s -o /dev/null
'http://pkgbeta.freebsd.org'
18,988: 0,003 18,988 3126
I have some 'free' resources for mirroring and ready to become official
mirror.

That host:
* already have full ftp mirror (synced @daily from
  ftp5.ru.freebsd.org)
* available via rsync/ftp/http for everyone
* have 2x1Gbit interfaces, powered 24x7
* available free disk space 1.4T

In private mail bapt@ wrote "Package beta, is temporary and will be
removed quite soon (I hope)".
So my questions is:
1) when it will be removed?
2) I'm confused...  Is the project does not need more resources ? (5.2
on [1]).

[1]
http://www.freebsd.org/doc/en_US.ISO8859-1/articles/hubs/mirror-official.html 


--
wbr, tiger
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hubs
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: pkgng mirrors

Mark Blackman-4

On 5 Sep 2012, at 09:20, "Sergey V. Dyatko" <[hidden email]> wrote:

> Hi,
>
> Today I setup test jail on my laptop with head/amd64 and trying to use
> pkgng, as I can see we have just one(?) host for pkgbeta.freebsd.org:
>
> pkgbeta.freebsd.org.    3600    IN      CNAME   ftp.dk.freebsd.org.
> ftp.dk.freebsd.org.     3600    IN      CNAME   ftp.beastie.tdk.net.
> ftp.beastie.tdk.net.    80      IN      A       193.162.146.4
>
> that host doesn't respond 'quickly' (for me):
> [tiger@laptop]:~%curl -w '%{time_total}: %{time_connect}
> %{time_starttransfer} %{size_download} \n' -s -o /dev/null
> 'http://pkgbeta.freebsd.org'
> 18,988: 0,003 18,988 3126
> I have some 'free' resources for mirroring and ready to become official
> mirror.
>
> That host:
> * already have full ftp mirror (synced @daily from
>  ftp5.ru.freebsd.org)
> * available via rsync/ftp/http for everyone
> * have 2x1Gbit interfaces, powered 24x7
> * available free disk space 1.4T
>
> In private mail bapt@ wrote "Package beta, is temporary and will be
> removed quite soon (I hope)".
> So my questions is:
> 1) when it will be removed?
> 2) I'm confused...  Is the project does not need more resources ? (5.2
> on [1]).

I believe the plan is to move to a small set of globally distributed
pkgng mirrors which are directly administered by those @freebsd.org, while
leaving the existing ad-hoc mirrors in place.

See..

http://wiki.freebsd.org/201208DevSummit?action=AttachFile&do=get&target=clusteradm_DevSummit201208_Presentation.key

As for schedules, that's less clear, other than quite soon (2-3 months?)


- Mark_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hubs
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: pkgng mirrors

Bjoern A. Zeeb
On Wed, 5 Sep 2012, Mark Blackman wrote:

Hi,

> On 5 Sep 2012, at 09:20, "Sergey V. Dyatko" <[hidden email]> wrote:
>
>> Hi,
>>
>> Today I setup test jail on my laptop with head/amd64 and trying to use
>> pkgng, as I can see we have just one(?) host for pkgbeta.freebsd.org:
>>
....
>> that host doesn't respond 'quickly' (for me):
...
>> I have some 'free' resources for mirroring and ready to become official
>> mirror.
...

>> In private mail bapt@ wrote "Package beta, is temporary and will be
>> removed quite soon (I hope)".
>> So my questions is:
>> 1) when it will be removed?
>> 2) I'm confused...  Is the project does not need more resources ? (5.2
>> on [1]).
>
> I believe the plan is to move to a small set of globally distributed
> pkgng mirrors which are directly administered by those @freebsd.org, while
> leaving the existing ad-hoc mirrors in place.
...
> As for schedules, that's less clear, other than quite soon (2-3 months?)

So let me unofficially leak a few more things here:

1) the tool is out as version 1.0; there will certainly be updates coming.
    You talk to bapt about that.

2) as bapt said in the announcement email pkgbeta.f.o is for testing
    and best effort.

3) no officially supported sets of pkg packages are published yet
    in my view.

4) we (FreeBSD) will be running the own official mirrors much the same
    way as indicated on the SVN mirror thread.  This will be what we
    will be supporting.

5) you will be able to run your own mirror, for your local community/enterprise
    if you'll feel you'll need to do that.  We will ask you not to
    do that under *.freebsd.org but it could be another local domain.
    However we (and you (plural)) will have to see and find out how
    things will go.  Please understand that with each mirror great
    advocacy statistics are reduced. Also udnerstand that we are not
    just doing it for statistics reasons.
    We will provide rsync access for people who wish mirror.

6) We currently setup mirrors at us-west, us-east, are in the progress
    for one in the EU and will be looking for more, esp, Asia after
    that.  We may be looking for 2nd sites in these areas if needed but
    we want see first.

7) Currently scheduled requirements are full root, isolated fully
    dedicated to FreeBSD machines/cluster resources for these mirror
    sites, not operated by any third party, but hands on for hardware,
    ...

8) for pkg mirrors, we are currently starting with about 7TB (yeah,
    you read correctly seven Terabyte as a start) of disk space
    requirements just for pkg packages and to hold them for up to a year.
    We'll also see how that'll work out, esp. if we might get more
    arch/platform sets in the future but it's a huge *up* from the
    current total of ftp space we have on mirrors.

9) The timeline for this to become public depends on a couple of
    factors, in part setup, in part package building, in part shaking
    out procedures and a few of us did quite some talking during
    the Cambridge Devsummit about this.  A few weeks to end this year
    depending on branch might be expected; I would assume HEAD comes
    first, stbale/9 after that at some time.
    We hope to publish more detailed information as things proceed.

/bz

--
Bjoern A. Zeeb                                 You have to have visions!
          Stop bit received. Insert coin for new address family.
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hubs
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: pkgng mirrors

Garrett Wollman-4
<<On Wed, 5 Sep 2012 09:54:33 +0000 (UTC), "Bjoern A. Zeeb" <[hidden email]> said:

> 4) we (FreeBSD) will be running the own official mirrors much the same
>     way as indicated on the SVN mirror thread.  This will be what we
>     will be supporting.

It would have been nice if this had been discussed with the people who
have volunteered significant resources over the past nearly two
decades to support FreeBSD by operating mirrors.  Say, the ones on
this mailing-list, perhaps?  Rather than just a peremptory "thanks but
we don't need you any more"?

It may very well be the right thing to do, but it certainly doesn't
come over well, particularly when (as in my case) people have just
recently put substantial new resources into their mirrors.

-GAWollman

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hubs
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: pkgng mirrors

Bjoern A. Zeeb
On Wed, 5 Sep 2012, Garrett Wollman wrote:

> <<On Wed, 5 Sep 2012 09:54:33 +0000 (UTC), "Bjoern A. Zeeb" <[hidden email]> said:
>
>> 4) we (FreeBSD) will be running the own official mirrors much the same
>>     way as indicated on the SVN mirror thread.  This will be what we
>>     will be supporting.
>
> It would have been nice if this had been discussed with the people who
> have volunteered significant resources over the past nearly two
> decades to support FreeBSD by operating mirrors.  Say, the ones on
> this mailing-list, perhaps?  Rather than just a peremptory "thanks but
> we don't need you any more"?
>
> It may very well be the right thing to do, but it certainly doesn't
> come over well, particularly when (as in my case) people have just
> recently put substantial new resources into their mirrors.

Sorry I am direct rather than beating about the bush, esp. as you
agree it's time to get the news out.   You'll find that some of this is
a result of last weeks discussions and the proper writeup is on TODO
lists.  The ideas aren't new -- I can easily go back to 2008 and find
postings about CDNs for packages.  The fact that it took us quite long
doesn't make it better.  But to speak more of a timeline, you'll want to
know that about 2 months ago, I couldn't have told you anything of this
more than general ponderings that had existed for years one way or
another and possibly mostly totally different.  The fact that you do
not see an official pkg mirror with packages yet is equally telling I
guess.

HOWEVER:

beware that the old stuff will continue to run for quite a while still,
probably longer than some of us would love to see or think currently,
but we need it, especially if run well (though the majority of mirrors
sadly aren't run well);  so your investment in a good FreeBSD mirror site
is worth it and welcome!   And we are talking into 2015 here probably (and
things tend to slip) by the current plans to my understanding and probably
a lot of people haven't realized that.  So you are getting a multi-year
notice.

That's a lot of time for us to get experience with the current plans.
We might very well as well find that we do have a larger digit number
of users by an order of magnitude or two and suddenly say "oh *beep*,
let's forget about that plan quickly and then again have to figure out
how to equip mirrors with 10TB of usable disk space and 128+G of memory
or half a rack of machines";)  The numbers might not be too far off in
total of what we are currently looking at -- though I am guessing
rather than doing maths.

In addition, as I mentioned, feel free to continue to run a mirror for
your local community it's not that you have to stop that out of a sudden
even after the magic year.  The fact that we'll provide rsync from the
mirrors for you should tell that.  In fact we'll provide you with way
better resources to do this than we currently do should you in a few years
still need or want to mirror.

And yes, we are aware at this point of having "3rd party mirrors" and we
are discussing how to continue to "endorse" (strong word) them to the best
we can for new stuff.  The Australian SVN mirror case has clearly shown us
that we need to figure it out.  The answer is - don't know yet and certainly
don't know before our own stuff is up and running but will let you as time
comes.


I think that's again my personal summary of what I think is the satte
of affairs but again, consider it informal.

/bz

--
Bjoern A. Zeeb                                 You have to have visions!
          Stop bit received. Insert coin for new address family.
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hubs
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: pkgng mirrors

Tim Bishop
On Wed, Sep 05, 2012 at 06:35:48PM +0000, Bjoern A. Zeeb wrote:
> ... we need it, especially if run well (though the majority of mirrors
> sadly aren't run well); ...

I run mirrorservice.org so I've interacted with quite a few open source
projects that have their stuff mirrored by third parties. The FreeBSD
system seems fairly primitive compared to others; whilst you have
various things to check the status of the mirrors there's nothing (as
far as I know) automatically happening with it. Other projects have
systems to automatically remove mirrors from listings if they're not
updating properly or are missing content.

If FreeBSD was to do something similar you'd be able to retain a well
established mirror network whilst effectively weeding out broken sites.

That said, 10TB worth of packages? That's a lot and I can imagine most
mirrors couldn't handle that quantity of data.

Tim.

--
Tim Bishop
http://www.bishnet.net/tim/
PGP Key: 0x5AE7D984
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hubs
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: pkgng mirrors

Bjoern A. Zeeb
On Wed, 5 Sep 2012, Tim Bishop wrote:

> On Wed, Sep 05, 2012 at 06:35:48PM +0000, Bjoern A. Zeeb wrote:
>> ... we need it, especially if run well (though the majority of mirrors
>> sadly aren't run well); ...
>
> I run mirrorservice.org so I've interacted with quite a few open source

And I noticed that service about two weeks ago and have a reminder to
go back out of curiosity to see;-)


> projects that have their stuff mirrored by third parties. The FreeBSD
> system seems fairly primitive compared to others; whilst you have
> various things to check the status of the mirrors there's nothing (as
> far as I know) automatically happening with it. Other projects have
> systems to automatically remove mirrors from listings if they're not
> updating properly or are missing content.
>
> If FreeBSD was to do something similar you'd be able to retain a well
> established mirror network whilst effectively weeding out broken sites.

Part of the historic problem here is that we have little control over
*.cc.FreeBSD.org currently though we gain more as time passes.  So
it's not that easy.  We can remove them from website and release lists
and I am not sure to which extend this happening but well worth
investigating.  I seem to remember that Release Engineering in the past
had at least made sure that the mirrors mentioned in the release
announcement had the bits at that time.

The other problems would however equally remain - no statistics,
maintainance, unable to "atomically" deploy things, and a couple of
other stuff ...  The fact that the world has moved by a decade but we
did not may make these changes look more radical than they are.


> That said, 10TB worth of packages? That's a lot and I can imagine most
> mirrors couldn't handle that quantity of data.

7TB to start within the first year, yes, we are upping the game a
little to give a lot better support to users we hope.  It won't be there
from day 1 but that's the direction thigns are heading.  A lot of the
mirrors can't cope with the 1G or 2G of ftp, and most of the old stuff
is not on that anymore but on ftp-archive, so yeah...

--
Bjoern A. Zeeb                                 You have to have visions!
          Stop bit received. Insert coin for new address family.
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hubs
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: pkgng mirrors

Mark Blackman-4
In reply to this post by Tim Bishop

On 5 Sep 2012, at 20:09, Tim Bishop <[hidden email]> wrote:

> On Wed, Sep 05, 2012 at 06:35:48PM +0000, Bjoern A. Zeeb wrote:
>> ... we need it, especially if run well (though the majority of mirrors
>> sadly aren't run well); ...
>
> I run mirrorservice.org so I've interacted with quite a few open source
> projects that have their stuff mirrored by third parties. The FreeBSD
> system seems fairly primitive compared to others; whilst you have
> various things to check the status of the mirrors there's nothing (as
> far as I know) automatically happening with it. Other projects have
> systems to automatically remove mirrors from listings if they're not
> updating properly or are missing content.
>
> If FreeBSD was to do something similar you'd be able to retain a well
> established mirror network whilst effectively weeding out broken sites.
>
> That said, 10TB worth of packages? That's a lot and I can imagine most
> mirrors couldn't handle that quantity of data.

10TB seems like a fairly modest requirement in 2012 with 3TB drives
going for 100GBP.

- Mark

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hubs
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: pkgng mirrors

Garrett Wollman-4
<<On Wed, 5 Sep 2012 22:04:18 +0100, Mark Blackman <[hidden email]> said:

> 10TB seems like a fairly modest requirement in 2012 with 3TB drives
> going for 100GBP.

Depends on who's paying for it, I think.  The recently upgraded
ftp5/cvsup3 (xyz.csail.mit.edu) has 12 x 1G enterprise SATA drives in
it, with a total (non-expandable) size of 8.12T for the zpool where
the public data lives.  We got all this hardware free from the
manufacturer, and wanted to put it to good use (since it's a bit too
small to be a file server for our own users).

-GAWollman

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hubs
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: pkgng mirrors

Jason Harris
In reply to this post by Bjoern A. Zeeb
On Wed, Sep 05, 2012 at 07:23:25PM +0000, Bjoern A. Zeeb wrote:
> On Wed, 5 Sep 2012, Tim Bishop wrote:
 
> > projects that have their stuff mirrored by third parties. The FreeBSD
> > system seems fairly primitive compared to others; whilst you have
> > various things to check the status of the mirrors there's nothing (as
> > far as I know) automatically happening with it. Other projects have
> > systems to automatically remove mirrors from listings if they're not
> > updating properly or are missing content.
> >
> > If FreeBSD was to do something similar you'd be able to retain a well
> > established mirror network whilst effectively weeding out broken sites.

Sounds like MirrorBrain:

  http://www.mirrorbrain.org/

I can't believe it hasn't been discussed more in public - Google
for "mirrorbrain" site:lists.freebsd.org and find one hit in 2008,
2010, and 2011...  :(

> Part of the historic problem here is that we have little control over
> *.cc.FreeBSD.org currently though we gain more as time passes.  So
> it's not that easy.  We can remove them from website and release lists
> and I am not sure to which extend this happening but well worth
> investigating.  I seem to remember that Release Engineering in the past
> had at least made sure that the mirrors mentioned in the release
> announcement had the bits at that time.

AIUI, MirrorBrain scans the mirrors and does most of this automagically.
I really like that it can generate .torrents on the fly with web seeding;
that seems perfect for .iso images around release time.

> The other problems would however equally remain - no statistics,

MB would run on central, project-run webservers and direct users
to mirrors with the desired files/collections, so you do know who
is requesting what and where they're being sent.

As for the server logs, why not remote syslog?  Also, I have changed
my Apache logs to CSV format and log the server's IP, making it easy
to feed logs from multiple servers to a single DB.

> maintainance, unable to "atomically" deploy things, and a couple of
> other stuff ...  The fact that the world has moved by a decade but we
> did not may make these changes look more radical than they are.

Clearly CDNs do this stuff quite well, but what other Open Source
projects have had to resort to using CDNs?

> > That said, 10TB worth of packages? That's a lot and I can imagine most
> > mirrors couldn't handle that quantity of data.
>
> 7TB to start within the first year, yes, we are upping the game a
> little to give a lot better support to users we hope.  It won't be there
> from day 1 but that's the direction thigns are heading.  A lot of the
> mirrors can't cope with the 1G or 2G of ftp, and most of the old stuff
> is not on that anymore but on ftp-archive, so yeah...

MB could redirect the archived stuff straight to ftp-archive, of course.

--
Jason Harris           |  PGP:  This _is_ PGP-signed, isn't it?
[hidden email] _|_ Got photons? (TM), (C) 2004

attachment0 (323 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: pkgng mirrors

Garrett Wollman-4
<<On Wed, 5 Sep 2012 20:50:49 -0400, Jason Harris <[hidden email]> said:

> Clearly CDNs do this stuff quite well, but what other Open Source
> projects have had to resort to using CDNs?

A lot of the major ones get it for "free" from either the provider
that hosts their development resources (e.g. sourceforge) or their
principal corporate sponsor.

-GAWollman
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hubs
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: pkgng mirrors

Garrett Wollman-4
In reply to this post by Bjoern A. Zeeb
<<On Wed, 5 Sep 2012 18:35:48 +0000 (UTC), "Bjoern A. Zeeb" <[hidden email]> said:

> I think that's again my personal summary of what I think is the satte
> of affairs but again, consider it informal.

But you don't really answer the question: WHO is discussing this, and
WHERE?  In what role are you speaking?  And shouldn't the community be
involved, rather than (as has happened disturbingly frequently in
recent months) making a decision behind closed doors and then
announcing it as a fait accompli in a post about something else?  I'm
all for making things better, but I don't even know what the people
who are working on this (whoever they may be) are trying to
accomplish.

Weren't we just having a discussion on another list a month ago about
how important decisions about how FreeBSD works shouldn't be made in
private?

-GAWollman
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hubs
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: pkgng mirrors

Alexandr Kovalenko-4
In reply to this post by Bjoern A. Zeeb
On Wed, Sep 5, 2012 at 12:54 PM, Bjoern A. Zeeb
<[hidden email]> wrote:

> So let me unofficially leak a few more things here:
>
> 4) we (FreeBSD) will be running the own official mirrors much the same
>    way as indicated on the SVN mirror thread.  This will be what we
>    will be supporting.
>
>
> 5) you will be able to run your own mirror, for your local
> community/enterprise
>    if you'll feel you'll need to do that.  We will ask you not to
>    do that under *.freebsd.org but it could be another local domain.
>    However we (and you (plural)) will have to see and find out how
>    things will go.  Please understand that with each mirror great
>    advocacy statistics are reduced. Also udnerstand that we are not
>    just doing it for statistics reasons.
>    We will provide rsync access for people who wish mirror.

So just "thank you and f.o." without any discussion at least with
mirror maintainers. We here (and I personally) provide
ftp/cvsup.ua.freebsd.org mirror for >10 years, put lots of efforts and
money (with the help of community), and now you just discussed
something behind the closed doors and say "f.o., we don't need you
anymore, but we let you run mirror". What a releif that you don't
prohibit us from running local (unofficial) mirrors. Thanks! >:E

Some of mirrors have done upgrades recently (hardware) and now you
also say - go away, we'll do everything on our own.

For me it looks very unpolite and and unfair.

First freebsd-udpdate, then portsnap without publicly available
server-side parts, now everything else? Oracle-way? :)

--
Alexandr Kovalenko
ftp/cvsup.ua.freebsd.org maintainer.
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hubs
To unsubscribe, send any mail to "[hidden email]"