The "git clone /path/to/git/objects_symlink" couldn't work after the following
change:

https://github.com/git/git/commit/6f054f9fb3a501c35b55c65e547a244f14c38d56

But repo command manages the git repo as symlinks, so check whether the objects
is an symlink to fix the problem:

* Nothing is changed if git/objects is not a symlink
* Use "git clone file://" when git/objects is a symlink

Signed-off-by: Robert Yang <liezhi.yang@...>
---
bitbake/lib/bb/fetch2/git.py | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py
index d0d68538e2..2a3c06fe4e 100644
--- a/bitbake/lib/bb/fetch2/git.py
+++ b/bitbake/lib/bb/fetch2/git.py
@@ -367,9 +367,13 @@ class Git(FetchMethod):

# If the repo still doesn't exist, fallback to cloning it
if not os.path.exists(ud.clonedir):
- # We do this since git will use a "-l" option automatically for local urls where possible
+ # We do this since git will use a "-l" option automatically for local urls where possible,
+ # but it doesn't work when git/objects is a symlink, only works when it is a directory.
if repourl.startswith("file://"):
- repourl = repourl[7:]
+ repourl_path = repourl[7:]
+ objects = os.path.join(repourl_path, 'objects')
+ if os.path.isdir(objects) and not os.path.islink(objects):
+ repourl = repourl_path
clone_cmd = "LANG=C %s clone --bare --mirror %s %s --progress" % (ud.basecmd, shlex.quote(repourl), ud.clonedir)
if ud.proto.lower() != 'file':
bb.fetch2.check_network_access(d, clone_cmd, ud.url)
--
2.33.1

On Fri, Mar 24, 2023 at 1:09 AM Robert Yang <liezhi.yang@...> wrote:

The "git clone /path/to/git/objects_symlink" couldn't work after the following
change:

https://github.com/git/git/commit/6f054f9fb3a501c35b55c65e547a244f14c38d56

But repo command manages the git repo as symlinks, so check whether the objects
is an symlink to fix the problem:

* Nothing is changed if git/objects is not a symlink
* Use "git clone file://" when git/objects is a symlink

Signed-off-by: Robert Yang <liezhi.yang@...>

Tested-by: Theodore A. Roth <troth@...>

Hi Sam,

On 3/26/23 10:18 PM, Sam Liddicott wrote:

Thanks for responding to this.
Does this method allow symlinking for local clones?

Did you mean "git clone -s/--shared"? No, it doesn't support that as the code shows. Use "git clone --shared" would save time and disk spaces, but
it would cause problems, for example, it doesn't work with BB_GENERATE_MIRROR_TARBALL since the .git/objects/info/alternates will be
invalid on other hosts.

I'm not sure whether we need fix other parts to make "--shared" work, let's see
whether RP has any suggestions.

I have a large git repository checked out using the repo --mirror option, which is then cloned for each target workspace.

Maybe consider using a shared DL_DIR among the builds, e.g.:

DL_DIR = "/path/to/shared/downloads"

// Robert

I'd like to avoid doing large local clones for each MACHINE workspace (there's a good dozen) but instead just symlink git objects as has been working previously and according to the patch I submitted.
Sam
On Fri, 24 Mar 2023, 19:51 Theodore A. Roth, <troth@... <mailto:troth@...>> wrote:
On Fri, Mar 24, 2023 at 1:09 AM Robert Yang <liezhi.yang@...
<mailto:liezhi.yang@...>> wrote:
>
> The "git clone /path/to/git/objects_symlink" couldn't work after the
following
> change:
>
>
https://github.com/git/git/commit/6f054f9fb3a501c35b55c65e547a244f14c38d56
<https://urldefense.com/v3/__https://github.com/git/git/commit/6f054f9fb3a501c35b55c65e547a244f14c38d56__;!!AjveYdw8EvQ!bAhzaY_csImWvjvDB4Fa1yvAmmEJ32SC4duzLyDarpR7o36SD7vwI2lra1HmH7jt_x22MPCDWbmT2knQRA$>
>
> But repo command manages the git repo as symlinks, so check whether the
objects
> is an symlink to fix the problem:
>
> * Nothing is changed if git/objects is not a symlink
> * Use "git clone file://" when git/objects is a symlink
>
> Signed-off-by: Robert Yang <liezhi.yang@...
<mailto:liezhi.yang@...>>
Tested-by: Theodore A. Roth <troth@... <mailto:troth@...>>

On Sun, 2023-03-26 at 15:18 +0100, Sam Liddicott wrote:

Thanks for responding to this.

Does this method allow symlinking for local clones?

I have a large git repository checked out using the repo --mirror
option, which is then cloned for each target workspace.

I'd like to avoid doing large local clones for each MACHINE workspace
(there's a good dozen) but instead just symlink git objects as has
been working previously and according to the patch I submitted.

I don't think we can make everything work for everyone.

As Robert mentioned, using the alternates mechanism will break the
mirror code. I think some users will also have issues if DL_DIR changes
break the checkout from under them.

We are going to have to do something to address the changes made in
upstream git and Robert's patch looks like a reasonable step to me to
fix that.

I'm open to other proposals but we can't regress other workflows. I'm
also reluctant to complicate the code more than we need to, there are
already far too many different codepaths we have to worry about :/.

Cheers,

Richard