How to handle the pack files now we have switched to git?

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

How to handle the pack files now we have switched to git?

Pete French-3
So, for me the switch to git went very smoothly. I havent moved to
etcupdate yet, but will probably do that soon. Hopwever I did hit one
issue. What I do is to build on a single machine, and then send that to
a number of places using rsync.

But what seems to happen wuth git is that it has a big pack file of
objects, and the name of the opack file is the SHA1 of whatever is
inside it. So if something chnages then the filename chnages - and thus
rsync tries to move the entire lot all over again, even if the change is
tiny.

Does anyone have a solution to this ? Currently I am telling rsync to
ignore .git when it sends the source, but am not entirely happy with that.

But apart from that, git works fine ;)

-pete.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: How to handle the pack files now we have switched to git?

Ronald Klop
 
Van: Pete French <[hidden email]>
Datum: dinsdag, 5 januari 2021 12:08
Aan: FreeBSD Stable Mailing List <[hidden email]>
Onderwerp: How to handle the pack files now we have switched to git?

>
> So, for me the switch to git went very smoothly. I havent moved to etcupdate yet, but will probably do that soon. Hopwever I did hit one issue. What I do is to build on a single machine, and then send that to a number of places using rsync.
>
> But what seems to happen wuth git is that it has a big pack file of objects, and the name of the opack file is the SHA1 of whatever is inside it. So if something chnages then the filename chnages - and thus rsync tries to move the entire lot all over again, even if the change is tiny.
>
> Does anyone have a solution to this ? Currently I am telling rsync to ignore .git when it sends the source, but am not entirely happy with that.
>
> But apart from that, git works fine ;)
>
> -pete.
> _______________________________________________
> [hidden email] mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "[hidden email]"
>
>
>


Why are you not happy with ignoring .git? It sounds like a pretty reasonable thing to do for your setup.
Otherwise (if you keep the .git dir) there is nothing you can do about this (AFAIK) and you will rsync quite some data now and then.

Regards,
Ronald.
 
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: How to handle the pack files now we have switched to git?

Chris H-2
In reply to this post by Pete French-3
On 2021-01-05 03:08, Pete French wrote:

> So, for me the switch to git went very smoothly. I havent moved to etcupdate
> yet,
> but will probably do that soon. Hopwever I did hit one issue. What I do is
> to
> build on a single machine, and then send that to a number of places using
> rsync.
>
> But what seems to happen wuth git is that it has a big pack file of objects,
> and
> the name of the opack file is the SHA1 of whatever is inside it. So if
> something
> chnages then the filename chnages - and thus rsync tries to move the entire
> lot
> all over again, even if the change is tiny.
>
> Does anyone have a solution to this ? Currently I am telling rsync to ignore
> .git
> when it sends the source, but am not entirely happy with that.
>
> But apart from that, git works fine ;)
Brilliant minds think alike. I use nearly the same routine. ;-)
Have been doing so on svn(1) as well. FWIW the only difference between
passing
the archive with vs without the .git. Is that getting the details of what rev
the source is at will not be available without the .git. Same on svn.
For me this is only a minor inconvenience. To help overcome this in the past.
I issued a 'svn info /usr/src >./SRC-REV'. Then simply passed that file along
to the receiving host(s).
OTOH it won't be (easily) possible to "up" the repo(s) from the receiving
hosts
w/o the .git.

HTH

--Chris
>
> -pete.
> _______________________________________________
> [hidden email] mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "[hidden email]"
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: How to handle the pack files now we have switched to git?

Pete French-3


On 05/01/2021 15:58, Chris wrote:
> Brilliant minds think alike. I use nearly the same routine. ;-)

Always good to know that I am not doing something completely idiotic ;)

> OTOH it won't be (easily) possible to "up" the repo(s) from the
> receiving hosts w/o the .git.

Yes, this is my issue really. Because though I liek to think I have one
place which builds the system and rsyncs it out, in practice here are
three of them, and each of them can push to the otjers. Which one I use
rather depoends where I am, and what I am doing - which is a problem
which has become more acute in the current world!

What I have decided to do is keep the.git on all fo them, but not rsycn
it, and to 'git pull' at the same time in each place - a 'it status'
will show me if any files are actually different after the rsync.

Should work fine I think.

thanks,

-pete.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: How to handle the pack files now we have switched to git?

Ulrich Spörlein-4
In reply to this post by Pete French-3
On Tue, 2021-01-05 at 11:08:48 +0000, Pete French wrote:

>So, for me the switch to git went very smoothly. I havent moved to
>etcupdate yet, but will probably do that soon. Hopwever I did hit one
>issue. What I do is to build on a single machine, and then send that to
>a number of places using rsync.
>
>But what seems to happen wuth git is that it has a big pack file of
>objects, and the name of the opack file is the SHA1 of whatever is
>inside it. So if something chnages then the filename chnages - and thus
>rsync tries to move the entire lot all over again, even if the change is
>tiny.

That's not entirely correct. The packfile will only be rewritten if
a) enough other stuff has accumulated
b) you force a repack.

You are fighting against git's GC mechanism a bit here, but you can
still make it work. On your source of truth host, do the following:

1. git gc --aggressive
2. look at .git/objects/pack, there should be a single big pack
3. touch .git/objects/pack/pack-<whateverhash>.keep  (or was it .pack.keep?)
4. rsync --del to all other hosts

Now future git gc runs will not delete that big pack, you'll only get
churn in the new, much smaller, packs.

hth
Uli
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: How to handle the pack files now we have switched to git?

Pete French-3


On 06/01/2021 15:47, Ulrich Spörlein wrote:

> That's not entirely correct. The packfile will only be rewritten if
> a) enough other stuff has accumulated
> b) you force a repack.
>
> You are fighting against git's GC mechanism a bit here, but you can
> still make it work. On your source of truth host, do the following:
>
> 1. git gc --aggressive
> 2. look at .git/objects/pack, there should be a single big pack
> 3. touch .git/objects/pack/pack-<whateverhash>.keep  (or was it
> .pack.keep?)
> 4. rsync --del to all other hosts
>
> Now future git gc runs will not delete that big pack, you'll only get
> churn in the new, much smaller, packs.

I missed seeing this until now, sorry, but thankyou! This is excellent,
and looks like a much better solution - or at least one which lets me
know 100% that I have percisely the same files on each machine. I will
gve this a try next time I need to do an update.

thanks,

-pete.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"