Please check the current beta git conversions

classic Classic list List threaded Threaded
104 messages Options
1234 ... 6
Reply | Threaded
Open this post in threaded view
|

Please check the current beta git conversions

Ed Maste-2
We've been updating the svn-git converter and pushing out a new
converted repo every two weeks, and are now approaching the time where
we'd like to commit to the tree generated by the exporter, and
guarantee that hashes will remain consistent from this point. At this
point the Git Working Group believes the conversion represents a
high-fidelity translation of the Subversion history, but would
appreciate additional review in case we've missed anything.

I'd ask folks with an interest in the FreeBSD repository to clone the
beta conversion and review the converted history in the specific areas
they are interested in - e.g., specific contrib/ software, device
drivers, etc. This will also present our final opportunity to change
the author map file, if necessary.

The beta src tree can be cloned via:
% git clone https://cgit-beta.freebsd.org/src.git freebsd-cgit-beta

Please follow up this week if you notice any discrepancies or author
entries that require updates.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Please check the current beta git conversions

Goran Mekić
On Tue, Sep 01, 2020 at 01:14:10PM -0400, Ed Maste wrote:

> We've been updating the svn-git converter and pushing out a new
> converted repo every two weeks, and are now approaching the time where
> we'd like to commit to the tree generated by the exporter, and
> guarantee that hashes will remain consistent from this point. At this
> point the Git Working Group believes the conversion represents a
> high-fidelity translation of the Subversion history, but would
> appreciate additional review in case we've missed anything.
>
> I'd ask folks with an interest in the FreeBSD repository to clone the
> beta conversion and review the converted history in the specific areas
> they are interested in - e.g., specific contrib/ software, device
> drivers, etc. This will also present our final opportunity to change
> the author map file, if necessary.
>
> The beta src tree can be cloned via:
> % git clone https://cgit-beta.freebsd.org/src.git freebsd-cgit-beta
>
> Please follow up this week if you notice any discrepancies or author
> entries that require updates.
Hello,

While I can clone src nicely, it is very slow in Serbia/Europe. The best
speed I got is 1.45MB/s but it's mostly around 700kB/s (still cloning). It is
probably not what you asked for, I just wanted you to know what's the speed
over here. Are there any mirrors planned? If not, is github mirror
recommended (mostly, what's the difference if I clone from cgit vs
github)?

Also, you probably want to hide gitolite-admin repo from cgit (not a biggie).

Regards,
meka

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Please check the current beta git conversions

Poul-Henning Kamp
--------
Goran Mekić writes:

> While I can clone src nicely, it is very slow in Serbia/Europe. The best
> speed I got is 1.45MB/s but it's mostly around 700kB/s (still cloning).

I was about to ask about that:

Any guidance on amount of diskspace and how long it takes to clone the repo ?


--
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[hidden email]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Please check the current beta git conversions

Shawn Webb-3
In reply to this post by Ed Maste-2
On Tue, Sep 01, 2020 at 01:14:10PM -0400, Ed Maste wrote:

> We've been updating the svn-git converter and pushing out a new
> converted repo every two weeks, and are now approaching the time where
> we'd like to commit to the tree generated by the exporter, and
> guarantee that hashes will remain consistent from this point. At this
> point the Git Working Group believes the conversion represents a
> high-fidelity translation of the Subversion history, but would
> appreciate additional review in case we've missed anything.
>
> I'd ask folks with an interest in the FreeBSD repository to clone the
> beta conversion and review the converted history in the specific areas
> they are interested in - e.g., specific contrib/ software, device
> drivers, etc. This will also present our final opportunity to change
> the author map file, if necessary.
>
> The beta src tree can be cloned via:
> % git clone https://cgit-beta.freebsd.org/src.git freebsd-cgit-beta
>
> Please follow up this week if you notice any discrepancies or author
> entries that require updates.
I'm curious if there's any plans for read-only access over ssh.
Trusting FreeBSD's ssh key material is likely easier than trusting
HTTPS in certain regions.

Thanks,

--
Shawn Webb
Cofounder / Security Engineer
HardenedBSD

GPG Key ID:          0xFF2E67A277F8E1FA
GPG Key Fingerprint: D206 BB45 15E0 9C49 0CF9  3633 C85B 0AF8 AB23 0FB2
https://git-01.md.hardenedbsd.org/HardenedBSD/pubkeys/src/branch/master/Shawn_Webb/03A4CBEBB82EA5A67D9F3853FF2E67A277F8E1FA.pub.asc

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Please check the current beta git conversions

Ed Maste-2
In reply to this post by Goran Mekić
On Tue, 1 Sep 2020 at 13:23, Goran Mekić <[hidden email]> wrote:
>
> Hello,
>
> While I can clone src nicely, it is very slow in Serbia/Europe.

Thanks for the report - I wouldn't be surprised if the host is a bit
bogged down from folks trying this just after my message went out.
Once your clone finishes please do let us know how long it took
overall.

We will have mirrors set up, which will be some combination of our own
hosts, GitHub, GitLab, or other third party hosts.

The current GitHub mirror is generated from an old version of svn2git,
which misses a lot of metadata - in particular, merges (for vendor
branch updates and such) are just missing. There are also some
mistakes that originated in our Subversion mirror network that are now
stuck in the current GitHub Mirror (commits authored by "svnmir"
instead of the correct committer, for example). We expect we will
continue to have a mirror on GitHub since many folks consider it "the"
repository of open source projects. There will be a documented process
for folks to migrate from the old to the new version of the mirror
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Please check the current beta git conversions

Ed Maste-2
In reply to this post by Poul-Henning Kamp
On Tue, 1 Sep 2020 at 13:37, Poul-Henning Kamp <[hidden email]> wrote:

>
> --------
> Goran Mekić writes:
>
> > While I can clone src nicely, it is very slow in Serbia/Europe. The best
> > speed I got is 1.45MB/s but it's mostly around 700kB/s (still cloning).
>
> I was about to ask about that:
>
> Any guidance on amount of diskspace and how long it takes to clone the repo ?

I see just over 3GB in my clone, including about 2.5GB in the .git directory.

If you have only one checkout git will require a bit more disk space
than svn. However, if you have two or more working trees (say, vanilla
FreeBSD and multiple work-in-progress trees, or head and stable
branches, etc.) using "git worktree" will share the .git directory and
in total will occupy less space than the equivalent in svn.

I'd expect clones to take minutes, although cgit-beta is running on a
lower spec jail host and might have trouble if many people are cloning
at the same time.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Please check the current beta git conversions

Goran Mekić
In reply to this post by Ed Maste-2
On Tue, Sep 01, 2020 at 01:43:19PM -0400, Ed Maste wrote:
> Thanks for the report - I wouldn't be surprised if the host is a bit
> bogged down from folks trying this just after my message went out.
> Once your clone finishes please do let us know how long it took
> overall.

time git clone https://cgit-beta.freebsd.org/src.git freebsd-cgit-beta
Cloning into 'freebsd-cgit-beta'...
remote: Enumerating objects: 390952, done.
remote: Counting objects: 100% (390952/390952), done.
remote: Compressing objects: 100% (30728/30728), done.
remote: Total 3763254 (delta 382506), reused 360224 (delta 360224), pack-reused 3372302
Receiving objects: 100% (3763254/3763254), 1.10 GiB | 876.00 KiB/s, done.
Resolving deltas: 100% (2984823/2984823), done.
Updating files: 100% (81780/81780), done.

________________________________________________________
Executed in   30.36 mins   fish           external
   usr time  666.58 secs  672.00 micros  666.58 secs
   sys time   30.38 secs  144.00 micros   30.38 secs

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Please check the current beta git conversions

Thomas Mueller-4
In reply to this post by Ed Maste-2
from Ed Maste:

> > Any guidance on amount of diskspace and how long it takes to clone the repo ?

> I see just over 3GB in my clone, including about 2.5GB in the .git directory.

> If you have only one checkout git will require a bit more disk space
> than svn. However, if you have two or more working trees (say, vanilla
> FreeBSD and multiple work-in-progress trees, or head and stable
> branches, etc.) using "git worktree" will share the .git directory and
> in total will occupy less space than the equivalent in svn.

> I'd expect clones to take minutes, although cgit-beta is running on a
> lower spec jail host and might have trouble if many people are cloning
> at the same time.

2.5 GB in .git directory sounds crazy and incomprehensible to me.

Only src tree I would be interested in now is HEAD (13-current), though I would also want the ports and doc trees.  Would that mollify the diskspace bloat?

I had trouble with the ethernet and wireless network drivers in FreeBSD 12 and 13-current, but there subsequently was a post on a big improvement to the network drivers for HEAD but not 12-stable.

So I am abandoning FreeBSD 12.x .

Hopefully I could update 13-current from within 13-current where I have no internet access but could use git from NetBSD, where I also have svn.

Tom

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Please check the current beta git conversions

Kevin Oberman-4
In reply to this post by Ed Maste-2
On Tue, Sep 1, 2020 at 7:38 PM Thomas Mueller <[hidden email]> wrote:

> from Ed Maste:
>
> > > Any guidance on amount of diskspace and how long it takes to clone the
> repo ?
>
> > I see just over 3GB in my clone, including about 2.5GB in the .git
> directory.
>
> > If you have only one checkout git will require a bit more disk space
> > than svn. However, if you have two or more working trees (say, vanilla
> > FreeBSD and multiple work-in-progress trees, or head and stable
> > branches, etc.) using "git worktree" will share the .git directory and
> > in total will occupy less space than the equivalent in svn.
>
> > I'd expect clones to take minutes, although cgit-beta is running on a
> > lower spec jail host and might have trouble if many people are cloning
> > at the same time.
>
> 2.5 GB in .git directory sounds crazy and incomprehensible to me.
>
> Only src tree I would be interested in now is HEAD (13-current), though I
> would also want the ports and doc trees.  Would that mollify the diskspace
> bloat?
>
> I had trouble with the ethernet and wireless network drivers in FreeBSD 12
> and 13-current, but there subsequently was a post on a big improvement to
> the network drivers for HEAD but not 12-stable.
>
> So I am abandoning FreeBSD 12.x .
>
> Hopefully I could update 13-current from within 13-current where I have no
> internet access but could use git from NetBSD, where I also have svn.
>
> Tom
>
Not really much different from subversion. .svn in /usr/sys is also 2.5G,
at least for 12.1.
--
Kevin Oberman, Part time kid herder and retired Network Engineer
E-mail: [hidden email]
PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Plans for git (was: Please check the current beta git conversions)

Greg 'groggy' Lehey-3
In reply to this post by Ed Maste-2
On Tuesday,  1 September 2020 at 13:14:10 -0400, Ed Maste wrote:
> We've been updating the svn-git converter and pushing out a new
> converted repo every two weeks, and are now approaching the time where
> we'd like to commit to the tree generated by the exporter,
> ...

Somehow I've missed this development.  Reading between the lines, it
seems that we're planning to move from svn to git, but I can't recall
seeing any announcement on the subject.  Can you give some background?
It would also be nice to find a HOWTO both for the migration and for
life with git.

Greg
--
Sent from my desktop computer.
See complete headers for address and phone numbers.
This message is digitally signed.  If your Microsoft mail program
reports problems, please read http://lemis.com/broken-MUA

signature.asc (169 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Plans for git (was: Please check the current beta git conversions)

grarpamp
The underlying initializing 'git init' commit hash must be
signed by security officer key having sufficient human PGP-WoT.

Git also supports sha-256 soon now, adoption should
be researched from various online article series and
work product before committing plans...
https://lwn.net/Articles/823352/
https://git-scm.com/docs/hash-function-transition
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Plans for git (was: Please check the current beta git conversions)

Kurt Jaeger-6
In reply to this post by Greg 'groggy' Lehey-3
Hi!

> On Tuesday,  1 September 2020 at 13:14:10 -0400, Ed Maste wrote:
> > We've been updating the svn-git converter and pushing out a new
> > converted repo every two weeks, and are now approaching the time where
> > we'd like to commit to the tree generated by the exporter,
> > ...
>
> Somehow I've missed this development.

The list archive

https://lists.freebsd.org/pipermail/freebsd-git/

has additional information.

http://wiki.freebsd.org/GitWorkflow
http://wiki.freebsd.org/UlrichSpoerlein/GitTalkDevSummit

has more. The repo: https://cgit-beta.freebsd.org/

Freshports using git:
devgit.freshports.org

A short intro on git for svn users:
https://hackmd.io/ML5TSl8mQ5-27B5eqDf7YA?view

--
[hidden email]            +49 171 3101372                    Now what ?
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Plans for git (was: Please check the current beta git conversions)

Steve Kargl
On Wed, Sep 02, 2020 at 08:01:17AM +0200, Kurt Jaeger wrote:

> Hi!
>
> > On Tuesday,  1 September 2020 at 13:14:10 -0400, Ed Maste wrote:
> > > We've been updating the svn-git converter and pushing out a new
> > > converted repo every two weeks, and are now approaching the time where
> > > we'd like to commit to the tree generated by the exporter,
> > > ...
> >
> > Somehow I've missed this development.
>
> The list archive
>
> https://lists.freebsd.org/pipermail/freebsd-git/
>
> has additional information.
>
> http://wiki.freebsd.org/GitWorkflow
> http://wiki.freebsd.org/UlrichSpoerlein/GitTalkDevSummit
>
> has more. The repo: https://cgit-beta.freebsd.org/
>
> Freshports using git:
> devgit.freshports.org
>
> A short intro on git for svn users:
> https://hackmd.io/ML5TSl8mQ5-27B5eqDf7YA?view
>

ROTFL.  From the "short intro", 2nd sentence.

New committers are assumed to already be familiar with the basic
operation of Git.  If not, start by reading the Git Book.

Define "short".

--
Steve
20170425 https://www.youtube.com/watch?v=VWUpyCsUKR4
20161221 https://www.youtube.com/watch?v=IbCHE-hONow
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Please check the current beta git conversions

Thomas Mueller-4
In reply to this post by Kevin Oberman-4
from Kevin Oberman:

> > So I am abandoning FreeBSD 12.x .

> > Hopefully I could update 13-current from within 13-current where I have no
> > internet access but could use git from NetBSD, where I also have svn.
       
> > Tom  

> Not really much different from subversion. .svn in /usr/sys is also 2.5G,
> at least for 12.1.
>-      
> Kevin Oberman, Part time kid herder and retired Network Engineer

My cross-post to [hidden email] was rejected because I was/am not subscribed to that list.

So I deactivate that part for this post.

I just checked du /usr/src/.svn ; this is on 11.1-STABLE old system, but this tree is for HEAD : 5817268 : is this 1K-blocks?  Now I think it's 512 bytes, in contrast to df that shows diskspace in 1-KB blocks: confusing.  I no longer track FreeBSD-11 src trees.

Now I wonder how this compares to git and hg: NetBSD plans to switch from cvs to Mercurial.

Tom

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Please check the current beta git conversions

Gary Jennejohn-6
In reply to this post by Kevin Oberman-4
On Wed, 02 Sep 2020 08:30:28 +0000
"Thomas Mueller" <[hidden email]> wrote:

> from Kevin Oberman:
>
> > > So I am abandoning FreeBSD 12.x .  
>
> > > Hopefully I could update 13-current from within 13-current where I have no
> > > internet access but could use git from NetBSD, where I also have svn.  
>        
> > > Tom    
>
> > Not really much different from subversion. .svn in /usr/sys is also 2.5G,
> > at least for 12.1.
> >-      
> > Kevin Oberman, Part time kid herder and retired Network Engineer  
>
> My cross-post to [hidden email] was rejected because I was/am not subscribed to that list.
>
> So I deactivate that part for this post.
>
> I just checked du /usr/src/.svn ; this is on 11.1-STABLE old system, but this tree is for HEAD : 5817268 : is this 1K-blocks?  Now I think it's 512 bytes, in contrast to df that shows diskspace in 1-KB blocks: confusing.  I no longer track FreeBSD-11 src trees.
>
> Now I wonder how this compares to git and hg: NetBSD plans to switch from cvs to Mercurial.
>

This is HEAD updated a short time ago (r365233):

du -sh /usr/src/.svn/
2.0G    /usr/src/.svn/

--
Gary Jennejohn
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Plans for git (was: Please check the current beta git conversions)

Mathieu Arnold-2
In reply to this post by grarpamp
On Wed, Sep 02, 2020 at 01:20:22AM -0400, grarpamp wrote:
> The underlying initializing 'git init' commit hash must be
> signed by security officer key having sufficient human PGP-WoT.
>
> Git also supports sha-256 soon now, adoption should
> be researched from various online article series and
> work product before committing plans...
> https://lwn.net/Articles/823352/
> https://git-scm.com/docs/hash-function-transition

"soon now" seems a bit vague, from what I have read on the subject,
whilst the local repository operations are working with SHA256 hashes,
it is still lacking remote transport, clones, and such.

--
Mathieu Arnold

signature.asc (981 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Please check the current beta git conversions

Mathieu Arnold-2
In reply to this post by Ed Maste-2
On Wed, Sep 02, 2020 at 02:37:45AM +0000, Thomas Mueller wrote:

> from Ed Maste:
>
> > > Any guidance on amount of diskspace and how long it takes to clone the repo ?
>
> > I see just over 3GB in my clone, including about 2.5GB in the .git directory.
>
> > If you have only one checkout git will require a bit more disk space
> > than svn. However, if you have two or more working trees (say, vanilla
> > FreeBSD and multiple work-in-progress trees, or head and stable
> > branches, etc.) using "git worktree" will share the .git directory and
> > in total will occupy less space than the equivalent in svn.
>
> > I'd expect clones to take minutes, although cgit-beta is running on a
> > lower spec jail host and might have trouble if many people are cloning
> > at the same time.
>
> 2.5 GB in .git directory sounds crazy and incomprehensible to me.
2.5GB is for the full history, if you only care about the main branch,
you can clone with:

  git clone --branch=main --single-branch https...

It will cut .git down to about 1.2GB.


If you do not care about the history and only want the tip of the
branch, you can also use:

  git clone --branch=main --depth 1 https...

This will cut it down even more to about 270MB.

--
Mathieu Arnold

signature.asc (981 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Please check the current beta git conversions

Olivier Certner-3
Hi,

> > 2.5 GB in .git directory sounds crazy and incomprehensible to me.
>
> 2.5GB is for the full history, if you only care about the main branch,
> you can clone with:
>
> (snip)

I do not reproduce this, but instead see `du` reporting 1.2GiB just after a
simple clone (no options), which AFAIK contains the whole history.

I have a complete mirror of SVN that I update with scripts around `svnrdump
dump` and `svnadmin load`, and its size is 6.1GiB (I pack it regularly).

For the record, here are the sizes of the `.svn` directories of some checkouts
I currently have:
- stable/12 at rev 355888: 1.9GiB
- stable/12 at rev 361580: 2.4GiB
- head at rev 361584: 2.7GiB
I must point out that I don't remember the exact creation history of these. It
is probable that, to create new ones, I did full copies (`cp -a`) of an old
one followed by `svn switch`. Don't know if this can have an influence on
these numbers (didn't test with pristine checkouts).

Important tip: When using `du`, don't forget to pass the '-A' option,
especially if using ZFS + compression.

--
Olivier Certner


_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Please check the current beta git conversions

Olivier Certner-3
In reply to this post by Goran Mekić
> Receiving objects: 100% (3763254/3763254), 1.10 GiB | 876.00 KiB/s, done.
> Executed in   30.36 mins

        Experienced a comparable full clone time yesterday afternoon from
France (did not precisely `time` it). The bandwidth oscillated between 600 and
1.2MiB/s steadily during the transfer. The final reported bandwidth was
991KiB/s (same transfer size).

--
Olivier Certner


_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Plans for git (was: Please check the current beta git conversions)

Ed Maste-2
In reply to this post by Mathieu Arnold-2
On Wed, 2 Sep 2020 at 07:51, Mathieu Arnold <[hidden email]> wrote:

>
> > Git also supports sha-256 soon now, adoption should
> > be researched from various online article series and
> > work product before committing plans...
> > https://lwn.net/Articles/823352/
> > https://git-scm.com/docs/hash-function-transition
>
> "soon now" seems a bit vague, from what I have read on the subject,
> whilst the local repository operations are working with SHA256 hashes,
> it is still lacking remote transport, clones, and such.

Yes, Git will migrate to SHA256 but will not be completely finished
sufficiently soon to matter for our needs. We'll eventually deal with
the migration the same way as everyone else.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
1234 ... 6