Date   

OpenEmbedded Developer meeting reminder

Philip Balister
 

We are finalizing the agenda for the developer meeting this Friday (May 20). The agenda has some topics carried from prior meetings and some new ideas we will cover at the bottom. We would like to get additional topics that matter to users and developers. Please review:

https://www.openembedded.org/wiki/OEDVM_2022_05

and add your ideas for topics we should discuss. Members of the OpenEmbedded board and Technical steering committee will be around on irc in #oe on libera.chat.

Philip


Thoughts on the eSDK

Richard Purdie
 

I've been asked about my thoughts on the eSDK. I'm going to try and
write down some of my ideas and thinking about it. I don't have a fully
thought out plan to pull off the shelf but can at least share the
ideas.

In summary, I think many of the concepts in the eSDK are good but the
implementation isn't great, it is effectively a beta where we needed to
go back and learn from the implementation to improve things.

To understand why, keep in mind it was put together as an experiment.
The idea was to have an environment which rather than being an SDK or a
full build system, could be configured to be different steps in a
spectrum between those two positions. The goals included:

* using prebuild binary objects rather than building from source
* a prebuilt toolchain
* being able to generate customised images quickly
* could be updated to new versions of the underlying environment
* making commands like devtool available to the "SDK" user
* hiding the build system complexity from the "SDK" user where possible
* allowing usage as per the current "standard SDK".

The design decisions this resulted in were:

* being able to lock the sstate hashes to a known version
* using sstate as the binary object mechanism (so not package 
management based)
* needing to capture a locked down version of the metadata (i.e. the 
layers in use)

The eSDK as implemented did do many of the things it was designed to
do. As with most engineering, there were challenges that were
encountered, things that didn't work as expected and also unanticipated
issues. It really needed someone to go back over it armed with that
knowledge for a v2 and that is still where we stand today as
development stopped.

The things which bother me about our current situation:

* there is no way to unlock the sstate hash of a specific component
like a bblock/bbunlock command.
https://bugzilla.yoctoproject.org/show_bug.cgi?id=13425
* we need to work out the layer setup and capture it into the eSDK. If 
we had a standard mechanism for layer setup and config this would be 
much easier. The current approach is very sensitive to different 
layouts and lacks any tests for anything that isn't poky
* where sstate checksum mismatches occur, they are fiendishly 
difficult to debug (a wider problem with bitbake but they affect 
eSDK particularly badly)
* no mechanism to turn an existing build into an eSDK
* no mechanism to turn an eSDK install into a full build
* can't build eSDK within eSDK

You can see the list of open bugs against eSDK:

https://bugzilla.yoctoproject.org/buglist.cgi?quicksearch=esdk&list_id=649733

In particular there are the following issues:

* it only works targeting a system of the same as it is being built 
on (an aarch64 eSDK needs to be built on an aarch64 system)
* multilib may or may not work (and has no tests)
https://bugzilla.yoctoproject.org/show_bug.cgi?id=10693
* devtool commands may need to be disabled depending on context
https://bugzilla.yoctoproject.org/show_bug.cgi?id=8789
* sdk-update has issues, e.g. 
https://bugzilla.yoctoproject.org/show_bug.cgi?id=10798
https://bugzilla.yoctoproject.org/show_bug.cgi?id=14462
https://bugzilla.yoctoproject.org/show_bug.cgi?id=14462
* several things don't build within the eSDK, e.g. ovmf, multilib, 
lttng, gstreamer-plugins-good


I've put off fixing some elements as I've hoped that if we fixed the
layer setup problem, it would then help elements of the eSDK. I think
we probably do need to go ahead and try and fix other elements without
that piece for now though (layer setup is a different problem in it's
own right).

I'm sharing this in the hopes it gets people thinking a little more and
can at least start the discussion.

Cheers,

Richard


Re: golang module handling

Andrei Gherzan
 

On Fri, 6 May 2022, at 18:10, lukas.funke@... wrote:

Hello all,

I just posted a series of patches for the purpose of generating recipies for a go offline build:


The series generates recipies of the following form (grpc-web as an example):

SRC_URI = "git://${GO_IMPORT};nobranch=1;name=${PN};protocol=https \

           file://go.mod.patch;patchdir=src/${GO_IMPORT} \

           ${@go_src_uri('github.com/cenkalti/backoff',path='github.com/cenkalti/backoff/v4')} \

           ${@go_src_uri('github.com/desertbit/timer')} \

...

#TODO: Subdirectories are heuristically derived from the import path and might be incorrect.

# github.com/cenkalti/backoff/v4@....3 => 6b0e4ad0cd65431b217393bff47d1dff727b264b

SRCREV_github.com.cenkalti.backoff.v4 = "6b0e4ad0cd65431b217393bff47d1dff727b264b"

GO_MODULE_PATH[github.com.cenkalti.backoff.v4] = "github.com/cenkalti/backoff/v4"

GO_MODULE_VERSION[github.com.cenkalti.backoff.v4] = "v4.1.3"

# github.com/desertbit/timer@....0-20180107155436-c41aec40b27f => c41aec40b27f0eeb2b94300fffcd624c69b02990

SRCREV_github.com.desertbit.timer = "c41aec40b27f0eeb2b94300fffcd624c69b02990"

GO_MODULE_PATH[github.com.desertbit.timer] = "github.com/desertbit/timer"

GO_MODULE_VERSION[github.com.desertbit.timer] = "v0.0.0-20180107155436-c41aec40b27f"


How it works:

the recipetool resolves the go-module-path for each dependency in the go.mod file to a corresponding src-url (mostly git repository urls, but other vcs would work as well). The recipetools also resolves the (pseudo)-semver to the corresponding git commit. git-url and commit hash are added to the generated go-recipe and processed later during build by the go-vendor.bbclass. During 'unpack' the bbclass copies the fetched dependency sources to the 'vendor' subdirectory and generates a 'modules.txt' manifest.
Most of my test projects worked right away. However, I'm sure that I missed some corner cases, as I'm not a go expert. Thus, I would be grateful for your opinion on my approach.


The implementation approach looks fine to me and it's what I had in mind too. What do you guys feel about taking the same approach in cargo-bitbake - writing this tool in go and maintaining it as such? I was just starting doing that but I stopped seeing this. There is an argument that go people would be able to help more in a go project while maintaining support for go packages in yocto/OE. And that seems to have worked fine in cargo/rust.

Andrei


Re: golang module handling

Bruce Ashfield
 

On Fri, May 6, 2022 at 3:39 PM Mark Asselstine
<mark.asselstine@...> wrote:



On 2022-05-06 14:41, Bruce Ashfield wrote:
On Fri, May 6, 2022 at 1:10 PM <lukas.funke@...> wrote:

Hello all,

I just posted a series of patches for the purpose of generating recipies for a go offline build:

[PATCH 1/5] recipetool-create: add ensure_native_cmd function (openembedded.org)
[PATCH 2/5] create_npm: reuse ensure_native_cmd from create.py (openembedded.org)
[PATCH 3/5] poky-meta: add go vendor class for offline builds (openembedded.org)
[PATCH 4/5] recipetool: add go recipe generator (openembedded.org)
[PATCH 5/5] oe-selftest: add go recipe create selftest (openembedded.org)
Hi Lukas,

As you might have figured out from last week's thread on this subject I
am interested in this work, as well as Bruce's scripts. We will be sure
to try out your work over the coming days and provide feedback.
At this point, I probably won't bother releasing my scripts. They suit
my needs, but the amount of effort (and time) to get them in a more
usable state (i.e. for more upstreams) isn't worth the effort.

What's here doesn't impact what I'm doing and vice versa, so there's
no need to consolidate.

Bruce


MarkA



The series generates recipies of the following form (grpc-web as an example):

SRC_URI = "git://${GO_IMPORT};nobranch=1;name=${PN};protocol=https \

file://go.mod.patch;patchdir=src/${GO_IMPORT} \

${@go_src_uri('github.com/cenkalti/backoff',path='github.com/cenkalti/backoff/v4')} \

${@go_src_uri('github.com/desertbit/timer')} \

...

#TODO: Subdirectories are heuristically derived from the import path and might be incorrect.

# github.com/cenkalti/backoff/v4@....3 => 6b0e4ad0cd65431b217393bff47d1dff727b264b

SRCREV_github.com.cenkalti.backoff.v4 = "6b0e4ad0cd65431b217393bff47d1dff727b264b"

GO_MODULE_PATH[github.com.cenkalti.backoff.v4] = "github.com/cenkalti/backoff/v4"

GO_MODULE_VERSION[github.com.cenkalti.backoff.v4] = "v4.1.3"

# github.com/desertbit/timer@....0-20180107155436-c41aec40b27f => c41aec40b27f0eeb2b94300fffcd624c69b02990

SRCREV_github.com.desertbit.timer = "c41aec40b27f0eeb2b94300fffcd624c69b02990"

GO_MODULE_PATH[github.com.desertbit.timer] = "github.com/desertbit/timer"

GO_MODULE_VERSION[github.com.desertbit.timer] = "v0.0.0-20180107155436-c41aec40b27f"


How it works:

the recipetool resolves the go-module-path for each dependency in the go.mod file to a corresponding src-url (mostly git repository urls, but other vcs would work as well). The recipetools also resolves the (pseudo)-semver to the corresponding git commit. git-url and commit hash are added to the generated go-recipe and processed later during build by the go-vendor.bbclass. During 'unpack' the bbclass copies the fetched dependency sources to the 'vendor' subdirectory and generates a 'modules.txt' manifest.
Most of my test projects worked right away. However, I'm sure that I missed some corner cases, as I'm not a go expert. Thus, I would be grateful for your opinion on my approach.
I saw the patches on the list, the two approaches are very similar to
what I've been using and generating for meta-virtualization. I even
see some similar hoops you had to jump through that I did as well
(indirect dependencies, etc).

The recipe format is different, but the approaches of resolving the
upstreams are very similar.

Some folks will like the generated information from the python, others
won't (I fall into the latter), but that is more of a preference than
a technical issue.

I unfortunately won't have any time to review or otherwise look at
this much until June, but luckily, no matter which direction this
heads, what I've been using continues to work .. and that's what I
like about the approaches.

Bruce

Best regards
Lukas

On Sat, Apr 30, 2022 at 12:06 AM, Bruce Ashfield wrote:

On Fri, Apr 29, 2022 at 4:21 PM Khem Raj <raj.khem@...> wrote:




On 4/29/22 12:15 PM, Bruce Ashfield wrote:

On Fri, Apr 29, 2022 at 2:59 PM Mark Asselstine
<mark.asselstine@...> wrote:


Folks,

For those of us who have had to work on GO recipes I think there has
been an open item that would be nice to get resolved in this new release
cycle.

Specifically, as with many of the modern languages GO includes its own
fetcher for 'dependencies', in GO these are referred to as 'modules'. As
it stands there is no unified or easy way to handle these and it would
be good to have this addressed in bitbake.

For example I can use two GO projects which we wanted to write recipes
for recently.

#1 buildah - https://github.com/containers/buildah - as you can see in
this project they actively maintain a 'vendor' directory and regularly
make commits to the vendor directory to keep it up-to-date. If you do
something like the following you can verify that no additional downloads
are needed during a build

$> export GOPATH=/tmp/go
$> git clone https://github.com/containers/buildah
$> cd buildah
$> go build -mod vendor ./cmd/buildah
$> ls /tmp/go/pkg/mod/

The /tmp/go/pkg/mod directory remains empty and no download msgs are
seen during the build to confirm that no downloads take place.

#2 Minio Client (MC) - https://github.com/minio/mc - as you can see in
this project there is no maintained 'vendor' directory. Similarly if you
clone and build this project you will experience additional downloads

$> export GOPATH=/tmp/go
$> git clone https://github.com/minio/mc
$> cd mc
$> go build -trimpath -tags kqueue -o ./mc
$> ls /tmp/go/pkg/mod/

This time you will find /tmp/go/pkg/mod is filled with mods and during
the build download msgs will have been displayed.

For folks that follow the mailing lists you might have seen discussions
about writing recipes for each of the "dependencies" but as you will
have noted in these threads GO projects typically have their own
versions and even specific commits of dependencies which they are
building against. In most cases you want to use what is dictated in the
project's go.mod and such to not invalidate the projects build and
runtime testing.

Solutions for writing recipes for projects like Minio Client:

Potential (actually currently in use) solution #1 is what has been done
for a recipe like k3s
(https://git.yoctoproject.org/meta-virtualization/tree/recipes-containers/k3s/k3s_git.bb).
Where each dependency has a SRC_URI and related information, which is
used to populate the vendor directory.

Potential solution #2 would be to run 'go mod vendor' to populate the
'vendor' directory, after which a patch can be generated which would be
added to the SRC_URI and as long as we capture the license information
in the recipe for all "dependencies" we can build without worries about
additional download attempts.

This was looked at (and discarded) as a solution for meta-virtualization.

It can work, but has big maintenance drawbacks around patching, CVEs.
It also doesn't allow for download caching, etc, as it has no fetcher
integration. If you want to do something like this, you need to do a
fetcher implementation, setup module proxies, etc, rather than
generating a giant patch to restore 'vendor'

I think a recipe tool to write recipes for go projects is going to serve
the community well, like what cargo-bitbake is in rust ecosystem. And
then bitbake fetcher could aide a bit in fetching part.

That's what I'm describing with the tool/script, it generates the
recipe elements. It could be dynamic, or as it currently stands, is
manually run to create a recipe.



Plus it makes us reliant on the golang module infrastructure, which
has shown to not always be reliable.

I would agree this was in flux before go modules were accepted as
standard way of doing dependencies, there were many solutions for
dependency management but I think we should now very much settle that it
is go modules now, since its part of core tooling in go community now a
days. So if we build tooling to use go modules underneath we wont be
painting ourselves to a corner.

There's no real painting into a corner in what I'm describing, it
isn't going around the modules and modules specification, it just
isn't using go itself to fetch. I've watched enough changes in golang,
that maybe this is the last .. maybe it isn't :) The go module format
hasn't been in flux for quite some time. It is the infrastructure that
goes through by default, has a few choke points and can have issues.
Of course the go mod infrastructure is flexible and can setup
mirrors/proxy, caching, etc, and that is what a fetcher implementation
could absolutely do.

It is the ongoing license management, maintenance/security, developer
flow, etc, that I found as a challenge.

Someone can absolutely implement a fetcher for it, set up the
variables to point to downloads for the proxy/mirrors (and avoid the
network), lock down go.mod/go.sum as the dependency declaration and
build without vendoring (and network). We talked about that model in
some of the earlier threads as well. That someone who writes it, just
isn't me at the moment :) I have something working that is independent
of those entanglements and can work even if something else is
developed.

Cheers,

Bruce






Potential solution #3 would be to add a 'go get' to the fetcher. This
would allow transposing go.mod information into the SRC_URI, drastically
reducing the tedious work done for solution #1.

This is difficult to implement, and leaves us with challenges for
locking down versions, licensing, etc. Definitely doable, but also
makes us again reliant on the go module infrastructure, unless we
setup mirrors, etc.


Potential solution #4 extend the tooling to facilitate the SRC_URI
generation for #1.

Not necessary. This is how #1 is actually done :)


Each of these have pros and cons, for instance #2 is the least amount of
work but also does nothing for saving download space when multiple GO
recipes do share dependencies. #3 and #2 are less secure compared to #1.
None of these address licensing very well...

I know this subject came up a few months ago but I don't recall there
being a resolution and since we are starting a new dev cycle I thought
now would be a good time to get thoughts from folks who have written GO
project recipes and take steps to making updates to moving this forward,
even if just by a little bit.

For go, we did come to a conclusion amongst the interested parties.

The "solution" that I'm using, is what you see in meta-virt. Use the
well established fetchers to go to the source repositories. That
allows us all the re-use and well tested functionality of git, etc,
and no need to re-invent the wheel.

I've written a tool that generates everything required for those
SRC_URIs, so for the most part, it isn't a manual process.

My presentation on the topic was accepted to the yocto summit, and
before that, I'll make the script available .. I just haven't had time
with all the release chaos and kernel issues at the moment.

Bruce


MarkA










--
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II









--
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II


Re: golang module handling

Mark Asselstine
 

On 2022-05-06 14:41, Bruce Ashfield wrote:
On Fri, May 6, 2022 at 1:10 PM <lukas.funke@...> wrote:

Hello all,

I just posted a series of patches for the purpose of generating recipies for a go offline build:

[PATCH 1/5] recipetool-create: add ensure_native_cmd function (openembedded.org)
[PATCH 2/5] create_npm: reuse ensure_native_cmd from create.py (openembedded.org)
[PATCH 3/5] poky-meta: add go vendor class for offline builds (openembedded.org)
[PATCH 4/5] recipetool: add go recipe generator (openembedded.org)
[PATCH 5/5] oe-selftest: add go recipe create selftest (openembedded.org)
Hi Lukas,

As you might have figured out from last week's thread on this subject I am interested in this work, as well as Bruce's scripts. We will be sure to try out your work over the coming days and provide feedback.

MarkA



The series generates recipies of the following form (grpc-web as an example):

SRC_URI = "git://${GO_IMPORT};nobranch=1;name=${PN};protocol=https \

file://go.mod.patch;patchdir=src/${GO_IMPORT} \

${@go_src_uri('github.com/cenkalti/backoff',path='github.com/cenkalti/backoff/v4')} \

${@go_src_uri('github.com/desertbit/timer')} \

...

#TODO: Subdirectories are heuristically derived from the import path and might be incorrect.

# github.com/cenkalti/backoff/v4@....3 => 6b0e4ad0cd65431b217393bff47d1dff727b264b

SRCREV_github.com.cenkalti.backoff.v4 = "6b0e4ad0cd65431b217393bff47d1dff727b264b"

GO_MODULE_PATH[github.com.cenkalti.backoff.v4] = "github.com/cenkalti/backoff/v4"

GO_MODULE_VERSION[github.com.cenkalti.backoff.v4] = "v4.1.3"

# github.com/desertbit/timer@....0-20180107155436-c41aec40b27f => c41aec40b27f0eeb2b94300fffcd624c69b02990

SRCREV_github.com.desertbit.timer = "c41aec40b27f0eeb2b94300fffcd624c69b02990"

GO_MODULE_PATH[github.com.desertbit.timer] = "github.com/desertbit/timer"

GO_MODULE_VERSION[github.com.desertbit.timer] = "v0.0.0-20180107155436-c41aec40b27f"


How it works:

the recipetool resolves the go-module-path for each dependency in the go.mod file to a corresponding src-url (mostly git repository urls, but other vcs would work as well). The recipetools also resolves the (pseudo)-semver to the corresponding git commit. git-url and commit hash are added to the generated go-recipe and processed later during build by the go-vendor.bbclass. During 'unpack' the bbclass copies the fetched dependency sources to the 'vendor' subdirectory and generates a 'modules.txt' manifest.
Most of my test projects worked right away. However, I'm sure that I missed some corner cases, as I'm not a go expert. Thus, I would be grateful for your opinion on my approach.
I saw the patches on the list, the two approaches are very similar to
what I've been using and generating for meta-virtualization. I even
see some similar hoops you had to jump through that I did as well
(indirect dependencies, etc).
The recipe format is different, but the approaches of resolving the
upstreams are very similar.
Some folks will like the generated information from the python, others
won't (I fall into the latter), but that is more of a preference than
a technical issue.
I unfortunately won't have any time to review or otherwise look at
this much until June, but luckily, no matter which direction this
heads, what I've been using continues to work .. and that's what I
like about the approaches.
Bruce

Best regards
Lukas

On Sat, Apr 30, 2022 at 12:06 AM, Bruce Ashfield wrote:

On Fri, Apr 29, 2022 at 4:21 PM Khem Raj <raj.khem@...> wrote:




On 4/29/22 12:15 PM, Bruce Ashfield wrote:

On Fri, Apr 29, 2022 at 2:59 PM Mark Asselstine
<mark.asselstine@...> wrote:


Folks,

For those of us who have had to work on GO recipes I think there has
been an open item that would be nice to get resolved in this new release
cycle.

Specifically, as with many of the modern languages GO includes its own
fetcher for 'dependencies', in GO these are referred to as 'modules'. As
it stands there is no unified or easy way to handle these and it would
be good to have this addressed in bitbake.

For example I can use two GO projects which we wanted to write recipes
for recently.

#1 buildah - https://github.com/containers/buildah - as you can see in
this project they actively maintain a 'vendor' directory and regularly
make commits to the vendor directory to keep it up-to-date. If you do
something like the following you can verify that no additional downloads
are needed during a build

$> export GOPATH=/tmp/go
$> git clone https://github.com/containers/buildah
$> cd buildah
$> go build -mod vendor ./cmd/buildah
$> ls /tmp/go/pkg/mod/

The /tmp/go/pkg/mod directory remains empty and no download msgs are
seen during the build to confirm that no downloads take place.

#2 Minio Client (MC) - https://github.com/minio/mc - as you can see in
this project there is no maintained 'vendor' directory. Similarly if you
clone and build this project you will experience additional downloads

$> export GOPATH=/tmp/go
$> git clone https://github.com/minio/mc
$> cd mc
$> go build -trimpath -tags kqueue -o ./mc
$> ls /tmp/go/pkg/mod/

This time you will find /tmp/go/pkg/mod is filled with mods and during
the build download msgs will have been displayed.

For folks that follow the mailing lists you might have seen discussions
about writing recipes for each of the "dependencies" but as you will
have noted in these threads GO projects typically have their own
versions and even specific commits of dependencies which they are
building against. In most cases you want to use what is dictated in the
project's go.mod and such to not invalidate the projects build and
runtime testing.

Solutions for writing recipes for projects like Minio Client:

Potential (actually currently in use) solution #1 is what has been done
for a recipe like k3s
(https://git.yoctoproject.org/meta-virtualization/tree/recipes-containers/k3s/k3s_git.bb).
Where each dependency has a SRC_URI and related information, which is
used to populate the vendor directory.

Potential solution #2 would be to run 'go mod vendor' to populate the
'vendor' directory, after which a patch can be generated which would be
added to the SRC_URI and as long as we capture the license information
in the recipe for all "dependencies" we can build without worries about
additional download attempts.

This was looked at (and discarded) as a solution for meta-virtualization.

It can work, but has big maintenance drawbacks around patching, CVEs.
It also doesn't allow for download caching, etc, as it has no fetcher
integration. If you want to do something like this, you need to do a
fetcher implementation, setup module proxies, etc, rather than
generating a giant patch to restore 'vendor'

I think a recipe tool to write recipes for go projects is going to serve
the community well, like what cargo-bitbake is in rust ecosystem. And
then bitbake fetcher could aide a bit in fetching part.

That's what I'm describing with the tool/script, it generates the
recipe elements. It could be dynamic, or as it currently stands, is
manually run to create a recipe.



Plus it makes us reliant on the golang module infrastructure, which
has shown to not always be reliable.

I would agree this was in flux before go modules were accepted as
standard way of doing dependencies, there were many solutions for
dependency management but I think we should now very much settle that it
is go modules now, since its part of core tooling in go community now a
days. So if we build tooling to use go modules underneath we wont be
painting ourselves to a corner.

There's no real painting into a corner in what I'm describing, it
isn't going around the modules and modules specification, it just
isn't using go itself to fetch. I've watched enough changes in golang,
that maybe this is the last .. maybe it isn't :) The go module format
hasn't been in flux for quite some time. It is the infrastructure that
goes through by default, has a few choke points and can have issues.
Of course the go mod infrastructure is flexible and can setup
mirrors/proxy, caching, etc, and that is what a fetcher implementation
could absolutely do.

It is the ongoing license management, maintenance/security, developer
flow, etc, that I found as a challenge.

Someone can absolutely implement a fetcher for it, set up the
variables to point to downloads for the proxy/mirrors (and avoid the
network), lock down go.mod/go.sum as the dependency declaration and
build without vendoring (and network). We talked about that model in
some of the earlier threads as well. That someone who writes it, just
isn't me at the moment :) I have something working that is independent
of those entanglements and can work even if something else is
developed.

Cheers,

Bruce






Potential solution #3 would be to add a 'go get' to the fetcher. This
would allow transposing go.mod information into the SRC_URI, drastically
reducing the tedious work done for solution #1.

This is difficult to implement, and leaves us with challenges for
locking down versions, licensing, etc. Definitely doable, but also
makes us again reliant on the go module infrastructure, unless we
setup mirrors, etc.


Potential solution #4 extend the tooling to facilitate the SRC_URI
generation for #1.

Not necessary. This is how #1 is actually done :)


Each of these have pros and cons, for instance #2 is the least amount of
work but also does nothing for saving download space when multiple GO
recipes do share dependencies. #3 and #2 are less secure compared to #1.
None of these address licensing very well...

I know this subject came up a few months ago but I don't recall there
being a resolution and since we are starting a new dev cycle I thought
now would be a good time to get thoughts from folks who have written GO
project recipes and take steps to making updates to moving this forward,
even if just by a little bit.

For go, we did come to a conclusion amongst the interested parties.

The "solution" that I'm using, is what you see in meta-virt. Use the
well established fetchers to go to the source repositories. That
allows us all the re-use and well tested functionality of git, etc,
and no need to re-invent the wheel.

I've written a tool that generates everything required for those
SRC_URIs, so for the most part, it isn't a manual process.

My presentation on the topic was accepted to the yocto summit, and
before that, I'll make the script available .. I just haven't had time
with all the release chaos and kernel issues at the moment.

Bruce


MarkA










--
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II




Re: golang module handling

Bruce Ashfield
 

On Fri, May 6, 2022 at 1:10 PM <lukas.funke@...> wrote:

Hello all,

I just posted a series of patches for the purpose of generating recipies for a go offline build:

[PATCH 1/5] recipetool-create: add ensure_native_cmd function (openembedded.org)
[PATCH 2/5] create_npm: reuse ensure_native_cmd from create.py (openembedded.org)
[PATCH 3/5] poky-meta: add go vendor class for offline builds (openembedded.org)
[PATCH 4/5] recipetool: add go recipe generator (openembedded.org)
[PATCH 5/5] oe-selftest: add go recipe create selftest (openembedded.org)

The series generates recipies of the following form (grpc-web as an example):

SRC_URI = "git://${GO_IMPORT};nobranch=1;name=${PN};protocol=https \

file://go.mod.patch;patchdir=src/${GO_IMPORT} \

${@go_src_uri('github.com/cenkalti/backoff',path='github.com/cenkalti/backoff/v4')} \

${@go_src_uri('github.com/desertbit/timer')} \

...

#TODO: Subdirectories are heuristically derived from the import path and might be incorrect.

# github.com/cenkalti/backoff/v4@....3 => 6b0e4ad0cd65431b217393bff47d1dff727b264b

SRCREV_github.com.cenkalti.backoff.v4 = "6b0e4ad0cd65431b217393bff47d1dff727b264b"

GO_MODULE_PATH[github.com.cenkalti.backoff.v4] = "github.com/cenkalti/backoff/v4"

GO_MODULE_VERSION[github.com.cenkalti.backoff.v4] = "v4.1.3"

# github.com/desertbit/timer@....0-20180107155436-c41aec40b27f => c41aec40b27f0eeb2b94300fffcd624c69b02990

SRCREV_github.com.desertbit.timer = "c41aec40b27f0eeb2b94300fffcd624c69b02990"

GO_MODULE_PATH[github.com.desertbit.timer] = "github.com/desertbit/timer"

GO_MODULE_VERSION[github.com.desertbit.timer] = "v0.0.0-20180107155436-c41aec40b27f"


How it works:

the recipetool resolves the go-module-path for each dependency in the go.mod file to a corresponding src-url (mostly git repository urls, but other vcs would work as well). The recipetools also resolves the (pseudo)-semver to the corresponding git commit. git-url and commit hash are added to the generated go-recipe and processed later during build by the go-vendor.bbclass. During 'unpack' the bbclass copies the fetched dependency sources to the 'vendor' subdirectory and generates a 'modules.txt' manifest.
Most of my test projects worked right away. However, I'm sure that I missed some corner cases, as I'm not a go expert. Thus, I would be grateful for your opinion on my approach.
I saw the patches on the list, the two approaches are very similar to
what I've been using and generating for meta-virtualization. I even
see some similar hoops you had to jump through that I did as well
(indirect dependencies, etc).

The recipe format is different, but the approaches of resolving the
upstreams are very similar.

Some folks will like the generated information from the python, others
won't (I fall into the latter), but that is more of a preference than
a technical issue.

I unfortunately won't have any time to review or otherwise look at
this much until June, but luckily, no matter which direction this
heads, what I've been using continues to work .. and that's what I
like about the approaches.

Bruce

Best regards
Lukas

On Sat, Apr 30, 2022 at 12:06 AM, Bruce Ashfield wrote:

On Fri, Apr 29, 2022 at 4:21 PM Khem Raj <raj.khem@...> wrote:




On 4/29/22 12:15 PM, Bruce Ashfield wrote:

On Fri, Apr 29, 2022 at 2:59 PM Mark Asselstine
<mark.asselstine@...> wrote:


Folks,

For those of us who have had to work on GO recipes I think there has
been an open item that would be nice to get resolved in this new release
cycle.

Specifically, as with many of the modern languages GO includes its own
fetcher for 'dependencies', in GO these are referred to as 'modules'. As
it stands there is no unified or easy way to handle these and it would
be good to have this addressed in bitbake.

For example I can use two GO projects which we wanted to write recipes
for recently.

#1 buildah - https://github.com/containers/buildah - as you can see in
this project they actively maintain a 'vendor' directory and regularly
make commits to the vendor directory to keep it up-to-date. If you do
something like the following you can verify that no additional downloads
are needed during a build

$> export GOPATH=/tmp/go
$> git clone https://github.com/containers/buildah
$> cd buildah
$> go build -mod vendor ./cmd/buildah
$> ls /tmp/go/pkg/mod/

The /tmp/go/pkg/mod directory remains empty and no download msgs are
seen during the build to confirm that no downloads take place.

#2 Minio Client (MC) - https://github.com/minio/mc - as you can see in
this project there is no maintained 'vendor' directory. Similarly if you
clone and build this project you will experience additional downloads

$> export GOPATH=/tmp/go
$> git clone https://github.com/minio/mc
$> cd mc
$> go build -trimpath -tags kqueue -o ./mc
$> ls /tmp/go/pkg/mod/

This time you will find /tmp/go/pkg/mod is filled with mods and during
the build download msgs will have been displayed.

For folks that follow the mailing lists you might have seen discussions
about writing recipes for each of the "dependencies" but as you will
have noted in these threads GO projects typically have their own
versions and even specific commits of dependencies which they are
building against. In most cases you want to use what is dictated in the
project's go.mod and such to not invalidate the projects build and
runtime testing.

Solutions for writing recipes for projects like Minio Client:

Potential (actually currently in use) solution #1 is what has been done
for a recipe like k3s
(https://git.yoctoproject.org/meta-virtualization/tree/recipes-containers/k3s/k3s_git.bb).
Where each dependency has a SRC_URI and related information, which is
used to populate the vendor directory.

Potential solution #2 would be to run 'go mod vendor' to populate the
'vendor' directory, after which a patch can be generated which would be
added to the SRC_URI and as long as we capture the license information
in the recipe for all "dependencies" we can build without worries about
additional download attempts.

This was looked at (and discarded) as a solution for meta-virtualization.

It can work, but has big maintenance drawbacks around patching, CVEs.
It also doesn't allow for download caching, etc, as it has no fetcher
integration. If you want to do something like this, you need to do a
fetcher implementation, setup module proxies, etc, rather than
generating a giant patch to restore 'vendor'

I think a recipe tool to write recipes for go projects is going to serve
the community well, like what cargo-bitbake is in rust ecosystem. And
then bitbake fetcher could aide a bit in fetching part.

That's what I'm describing with the tool/script, it generates the
recipe elements. It could be dynamic, or as it currently stands, is
manually run to create a recipe.



Plus it makes us reliant on the golang module infrastructure, which
has shown to not always be reliable.

I would agree this was in flux before go modules were accepted as
standard way of doing dependencies, there were many solutions for
dependency management but I think we should now very much settle that it
is go modules now, since its part of core tooling in go community now a
days. So if we build tooling to use go modules underneath we wont be
painting ourselves to a corner.

There's no real painting into a corner in what I'm describing, it
isn't going around the modules and modules specification, it just
isn't using go itself to fetch. I've watched enough changes in golang,
that maybe this is the last .. maybe it isn't :) The go module format
hasn't been in flux for quite some time. It is the infrastructure that
goes through by default, has a few choke points and can have issues.
Of course the go mod infrastructure is flexible and can setup
mirrors/proxy, caching, etc, and that is what a fetcher implementation
could absolutely do.

It is the ongoing license management, maintenance/security, developer
flow, etc, that I found as a challenge.

Someone can absolutely implement a fetcher for it, set up the
variables to point to downloads for the proxy/mirrors (and avoid the
network), lock down go.mod/go.sum as the dependency declaration and
build without vendoring (and network). We talked about that model in
some of the earlier threads as well. That someone who writes it, just
isn't me at the moment :) I have something working that is independent
of those entanglements and can work even if something else is
developed.

Cheers,

Bruce






Potential solution #3 would be to add a 'go get' to the fetcher. This
would allow transposing go.mod information into the SRC_URI, drastically
reducing the tedious work done for solution #1.

This is difficult to implement, and leaves us with challenges for
locking down versions, licensing, etc. Definitely doable, but also
makes us again reliant on the go module infrastructure, unless we
setup mirrors, etc.


Potential solution #4 extend the tooling to facilitate the SRC_URI
generation for #1.

Not necessary. This is how #1 is actually done :)


Each of these have pros and cons, for instance #2 is the least amount of
work but also does nothing for saving download space when multiple GO
recipes do share dependencies. #3 and #2 are less secure compared to #1.
None of these address licensing very well...

I know this subject came up a few months ago but I don't recall there
being a resolution and since we are starting a new dev cycle I thought
now would be a good time to get thoughts from folks who have written GO
project recipes and take steps to making updates to moving this forward,
even if just by a little bit.

For go, we did come to a conclusion amongst the interested parties.

The "solution" that I'm using, is what you see in meta-virt. Use the
well established fetchers to go to the source repositories. That
allows us all the re-use and well tested functionality of git, etc,
and no need to re-invent the wheel.

I've written a tool that generates everything required for those
SRC_URIs, so for the most part, it isn't a manual process.

My presentation on the topic was accepted to the yocto summit, and
before that, I'll make the script available .. I just haven't had time
with all the release chaos and kernel issues at the moment.

Bruce


MarkA










--
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II




--
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II


Re: golang module handling

lukas.funke@...
 

Hello all,

I just posted a series of patches for the purpose of generating recipies for a go offline build:

[PATCH 1/5] recipetool-create: add ensure_native_cmd function (openembedded.org)
[PATCH 2/5] create_npm: reuse ensure_native_cmd from create.py (openembedded.org)
[PATCH 3/5] poky-meta: add go vendor class for offline builds (openembedded.org)
[PATCH 4/5] recipetool: add go recipe generator (openembedded.org)
[PATCH 5/5] oe-selftest: add go recipe create selftest (openembedded.org)

The series generates recipies of the following form (grpc-web as an example):

SRC_URI = "git://${GO_IMPORT};nobranch=1;name=${PN};protocol=https \

           file://go.mod.patch;patchdir=src/${GO_IMPORT} \

           ${@go_src_uri('github.com/cenkalti/backoff',path='github.com/cenkalti/backoff/v4')} \

           ${@go_src_uri('github.com/desertbit/timer')} \

...

#TODO: Subdirectories are heuristically derived from the import path and might be incorrect.

# github.com/cenkalti/backoff/v4@....3 => 6b0e4ad0cd65431b217393bff47d1dff727b264b

SRCREV_github.com.cenkalti.backoff.v4 = "6b0e4ad0cd65431b217393bff47d1dff727b264b"

GO_MODULE_PATH[github.com.cenkalti.backoff.v4] = "github.com/cenkalti/backoff/v4"

GO_MODULE_VERSION[github.com.cenkalti.backoff.v4] = "v4.1.3"

# github.com/desertbit/timer@....0-20180107155436-c41aec40b27f => c41aec40b27f0eeb2b94300fffcd624c69b02990

SRCREV_github.com.desertbit.timer = "c41aec40b27f0eeb2b94300fffcd624c69b02990"

GO_MODULE_PATH[github.com.desertbit.timer] = "github.com/desertbit/timer"

GO_MODULE_VERSION[github.com.desertbit.timer] = "v0.0.0-20180107155436-c41aec40b27f"


How it works:

the recipetool resolves the go-module-path for each dependency in the go.mod file to a corresponding src-url (mostly git repository urls, but other vcs would work as well). The recipetools also resolves the (pseudo)-semver to the corresponding git commit. git-url and commit hash are added to the generated go-recipe and processed later during build by the go-vendor.bbclass. During 'unpack' the bbclass copies the fetched dependency sources to the 'vendor' subdirectory and generates a 'modules.txt' manifest.
Most of my test projects worked right away. However, I'm sure that I missed some corner cases, as I'm not a go expert. Thus, I would be grateful for your opinion on my approach.

Best regards
Lukas

On Sat, Apr 30, 2022 at 12:06 AM, Bruce Ashfield wrote:

On Fri, Apr 29, 2022 at 4:21 PM Khem Raj <raj.khem@...> wrote:



On 4/29/22 12:15 PM, Bruce Ashfield wrote:
On Fri, Apr 29, 2022 at 2:59 PM Mark Asselstine
<mark.asselstine@...> wrote:

Folks,

For those of us who have had to work on GO recipes I think there has
been an open item that would be nice to get resolved in this new release
cycle.

Specifically, as with many of the modern languages GO includes its own
fetcher for 'dependencies', in GO these are referred to as 'modules'. As
it stands there is no unified or easy way to handle these and it would
be good to have this addressed in bitbake.

For example I can use two GO projects which we wanted to write recipes
for recently.

#1 buildah - https://github.com/containers/buildah - as you can see in
this project they actively maintain a 'vendor' directory and regularly
make commits to the vendor directory to keep it up-to-date. If you do
something like the following you can verify that no additional downloads
are needed during a build

$> export GOPATH=/tmp/go
$> git clone https://github.com/containers/buildah
$> cd buildah
$> go build -mod vendor ./cmd/buildah
$> ls /tmp/go/pkg/mod/

The /tmp/go/pkg/mod directory remains empty and no download msgs are
seen during the build to confirm that no downloads take place.

#2 Minio Client (MC) - https://github.com/minio/mc - as you can see in
this project there is no maintained 'vendor' directory. Similarly if you
clone and build this project you will experience additional downloads

$> export GOPATH=/tmp/go
$> git clone https://github.com/minio/mc
$> cd mc
$> go build -trimpath -tags kqueue -o ./mc
$> ls /tmp/go/pkg/mod/

This time you will find /tmp/go/pkg/mod is filled with mods and during
the build download msgs will have been displayed.

For folks that follow the mailing lists you might have seen discussions
about writing recipes for each of the "dependencies" but as you will
have noted in these threads GO projects typically have their own
versions and even specific commits of dependencies which they are
building against. In most cases you want to use what is dictated in the
project's go.mod and such to not invalidate the projects build and
runtime testing.

Solutions for writing recipes for projects like Minio Client:

Potential (actually currently in use) solution #1 is what has been done
for a recipe like k3s
(https://git.yoctoproject.org/meta-virtualization/tree/recipes-containers/k3s/k3s_git.bb).
Where each dependency has a SRC_URI and related information, which is
used to populate the vendor directory.

Potential solution #2 would be to run 'go mod vendor' to populate the
'vendor' directory, after which a patch can be generated which would be
added to the SRC_URI and as long as we capture the license information
in the recipe for all "dependencies" we can build without worries about
additional download attempts.
This was looked at (and discarded) as a solution for meta-virtualization.

It can work, but has big maintenance drawbacks around patching, CVEs.
It also doesn't allow for download caching, etc, as it has no fetcher
integration. If you want to do something like this, you need to do a
fetcher implementation, setup module proxies, etc, rather than
generating a giant patch to restore 'vendor'
I think a recipe tool to write recipes for go projects is going to serve
the community well, like what cargo-bitbake is in rust ecosystem. And
then bitbake fetcher could aide a bit in fetching part.
That's what I'm describing with the tool/script, it generates the
recipe elements. It could be dynamic, or as it currently stands, is
manually run to create a recipe.



Plus it makes us reliant on the golang module infrastructure, which
has shown to not always be reliable.
I would agree this was in flux before go modules were accepted as
standard way of doing dependencies, there were many solutions for
dependency management but I think we should now very much settle that it
is go modules now, since its part of core tooling in go community now a
days. So if we build tooling to use go modules underneath we wont be
painting ourselves to a corner.
There's no real painting into a corner in what I'm describing, it
isn't going around the modules and modules specification, it just
isn't using go itself to fetch. I've watched enough changes in golang,
that maybe this is the last .. maybe it isn't :) The go module format
hasn't been in flux for quite some time. It is the infrastructure that
goes through by default, has a few choke points and can have issues.
Of course the go mod infrastructure is flexible and can setup
mirrors/proxy, caching, etc, and that is what a fetcher implementation
could absolutely do.

It is the ongoing license management, maintenance/security, developer
flow, etc, that I found as a challenge.

Someone can absolutely implement a fetcher for it, set up the
variables to point to downloads for the proxy/mirrors (and avoid the
network), lock down go.mod/go.sum as the dependency declaration and
build without vendoring (and network). We talked about that model in
some of the earlier threads as well. That someone who writes it, just
isn't me at the moment :) I have something working that is independent
of those entanglements and can work even if something else is
developed.

Cheers,

Bruce






Potential solution #3 would be to add a 'go get' to the fetcher. This
would allow transposing go.mod information into the SRC_URI, drastically
reducing the tedious work done for solution #1.
This is difficult to implement, and leaves us with challenges for
locking down versions, licensing, etc. Definitely doable, but also
makes us again reliant on the go module infrastructure, unless we
setup mirrors, etc.


Potential solution #4 extend the tooling to facilitate the SRC_URI
generation for #1.
Not necessary. This is how #1 is actually done :)


Each of these have pros and cons, for instance #2 is the least amount of
work but also does nothing for saving download space when multiple GO
recipes do share dependencies. #3 and #2 are less secure compared to #1.
None of these address licensing very well...

I know this subject came up a few months ago but I don't recall there
being a resolution and since we are starting a new dev cycle I thought
now would be a good time to get thoughts from folks who have written GO
project recipes and take steps to making updates to moving this forward,
even if just by a little bit.
For go, we did come to a conclusion amongst the interested parties.

The "solution" that I'm using, is what you see in meta-virt. Use the
well established fetchers to go to the source repositories. That
allows us all the re-use and well tested functionality of git, etc,
and no need to re-invent the wheel.

I've written a tool that generates everything required for those
SRC_URIs, so for the most part, it isn't a manual process.

My presentation on the topic was accepted to the yocto summit, and
before that, I'll make the script available .. I just haven't had time
with all the release chaos and kernel issues at the moment.

Bruce


MarkA










--
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II


OEDvM 2022.05

Josef Holzmayr
 

There will be an OpenEmbedded Developer Virtual Meeting (OEDVM) held
on Friday, 20 May 2022. This will be the day after the Yocto
Project® Summit 2022.05 concludes. The scheduled time will match the
YP Summit's start at 12 UTC/7AM Eastern, and run until 20 UTC/3PM (or
earlier if we run out of topics, but then will most likely turn into a
happy hour).

The event has been added to the OE Calendar
https://www.openembedded.org/wiki/Calendar

We have added some topics that should be interesting to the larger audience at:
https://www.openembedded.org/wiki/OEDVM_2022_05

We are looking for "moderators" for the currently listed topics. This
entails helping to get the discussion started by presenting 1-5 slides
on the topic and problems currently being experienced.

If you do not see something listed you would like to discuss, please
add it and put yourself as the "moderator" (and a rough time estimate
for discussion).

As always, if you are not an OpenEmbedded Member (but are interested
in joining), please attend OEDVM and you can be voted in.

If there are any questions, please feel free to email me and/or the
entire OE board (board@...).

Josef, on behalf of the OpenEmbedded Board


Re: golang module handling

Bruce Ashfield
 

On Fri, Apr 29, 2022 at 4:21 PM Khem Raj <raj.khem@...> wrote:



On 4/29/22 12:15 PM, Bruce Ashfield wrote:
On Fri, Apr 29, 2022 at 2:59 PM Mark Asselstine
<mark.asselstine@...> wrote:

Folks,

For those of us who have had to work on GO recipes I think there has
been an open item that would be nice to get resolved in this new release
cycle.

Specifically, as with many of the modern languages GO includes its own
fetcher for 'dependencies', in GO these are referred to as 'modules'. As
it stands there is no unified or easy way to handle these and it would
be good to have this addressed in bitbake.

For example I can use two GO projects which we wanted to write recipes
for recently.

#1 buildah - https://github.com/containers/buildah - as you can see in
this project they actively maintain a 'vendor' directory and regularly
make commits to the vendor directory to keep it up-to-date. If you do
something like the following you can verify that no additional downloads
are needed during a build

$> export GOPATH=/tmp/go
$> git clone https://github.com/containers/buildah
$> cd buildah
$> go build -mod vendor ./cmd/buildah
$> ls /tmp/go/pkg/mod/

The /tmp/go/pkg/mod directory remains empty and no download msgs are
seen during the build to confirm that no downloads take place.

#2 Minio Client (MC) - https://github.com/minio/mc - as you can see in
this project there is no maintained 'vendor' directory. Similarly if you
clone and build this project you will experience additional downloads

$> export GOPATH=/tmp/go
$> git clone https://github.com/minio/mc
$> cd mc
$> go build -trimpath -tags kqueue -o ./mc
$> ls /tmp/go/pkg/mod/

This time you will find /tmp/go/pkg/mod is filled with mods and during
the build download msgs will have been displayed.

For folks that follow the mailing lists you might have seen discussions
about writing recipes for each of the "dependencies" but as you will
have noted in these threads GO projects typically have their own
versions and even specific commits of dependencies which they are
building against. In most cases you want to use what is dictated in the
project's go.mod and such to not invalidate the projects build and
runtime testing.

Solutions for writing recipes for projects like Minio Client:

Potential (actually currently in use) solution #1 is what has been done
for a recipe like k3s
(https://git.yoctoproject.org/meta-virtualization/tree/recipes-containers/k3s/k3s_git.bb).
Where each dependency has a SRC_URI and related information, which is
used to populate the vendor directory.

Potential solution #2 would be to run 'go mod vendor' to populate the
'vendor' directory, after which a patch can be generated which would be
added to the SRC_URI and as long as we capture the license information
in the recipe for all "dependencies" we can build without worries about
additional download attempts.
This was looked at (and discarded) as a solution for meta-virtualization.

It can work, but has big maintenance drawbacks around patching, CVEs.
It also doesn't allow for download caching, etc, as it has no fetcher
integration. If you want to do something like this, you need to do a
fetcher implementation, setup module proxies, etc, rather than
generating a giant patch to restore 'vendor'
I think a recipe tool to write recipes for go projects is going to serve
the community well, like what cargo-bitbake is in rust ecosystem. And
then bitbake fetcher could aide a bit in fetching part.
That's what I'm describing with the tool/script, it generates the
recipe elements. It could be dynamic, or as it currently stands, is
manually run to create a recipe.



Plus it makes us reliant on the golang module infrastructure, which
has shown to not always be reliable.
I would agree this was in flux before go modules were accepted as
standard way of doing dependencies, there were many solutions for
dependency management but I think we should now very much settle that it
is go modules now, since its part of core tooling in go community now a
days. So if we build tooling to use go modules underneath we wont be
painting ourselves to a corner.
There's no real painting into a corner in what I'm describing, it
isn't going around the modules and modules specification, it just
isn't using go itself to fetch. I've watched enough changes in golang,
that maybe this is the last .. maybe it isn't :) The go module format
hasn't been in flux for quite some time. It is the infrastructure that
goes through by default, has a few choke points and can have issues.
Of course the go mod infrastructure is flexible and can setup
mirrors/proxy, caching, etc, and that is what a fetcher implementation
could absolutely do.

It is the ongoing license management, maintenance/security, developer
flow, etc, that I found as a challenge.

Someone can absolutely implement a fetcher for it, set up the
variables to point to downloads for the proxy/mirrors (and avoid the
network), lock down go.mod/go.sum as the dependency declaration and
build without vendoring (and network). We talked about that model in
some of the earlier threads as well. That someone who writes it, just
isn't me at the moment :) I have something working that is independent
of those entanglements and can work even if something else is
developed.

Cheers,

Bruce






Potential solution #3 would be to add a 'go get' to the fetcher. This
would allow transposing go.mod information into the SRC_URI, drastically
reducing the tedious work done for solution #1.
This is difficult to implement, and leaves us with challenges for
locking down versions, licensing, etc. Definitely doable, but also
makes us again reliant on the go module infrastructure, unless we
setup mirrors, etc.


Potential solution #4 extend the tooling to facilitate the SRC_URI
generation for #1.
Not necessary. This is how #1 is actually done :)


Each of these have pros and cons, for instance #2 is the least amount of
work but also does nothing for saving download space when multiple GO
recipes do share dependencies. #3 and #2 are less secure compared to #1.
None of these address licensing very well...

I know this subject came up a few months ago but I don't recall there
being a resolution and since we are starting a new dev cycle I thought
now would be a good time to get thoughts from folks who have written GO
project recipes and take steps to making updates to moving this forward,
even if just by a little bit.
For go, we did come to a conclusion amongst the interested parties.

The "solution" that I'm using, is what you see in meta-virt. Use the
well established fetchers to go to the source repositories. That
allows us all the re-use and well tested functionality of git, etc,
and no need to re-invent the wheel.

I've written a tool that generates everything required for those
SRC_URIs, so for the most part, it isn't a manual process.

My presentation on the topic was accepted to the yocto summit, and
before that, I'll make the script available .. I just haven't had time
with all the release chaos and kernel issues at the moment.

Bruce


MarkA










--
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II


Re: golang module handling

Khem Raj
 

On 4/29/22 12:15 PM, Bruce Ashfield wrote:
On Fri, Apr 29, 2022 at 2:59 PM Mark Asselstine
<mark.asselstine@...> wrote:

Folks,

For those of us who have had to work on GO recipes I think there has
been an open item that would be nice to get resolved in this new release
cycle.

Specifically, as with many of the modern languages GO includes its own
fetcher for 'dependencies', in GO these are referred to as 'modules'. As
it stands there is no unified or easy way to handle these and it would
be good to have this addressed in bitbake.

For example I can use two GO projects which we wanted to write recipes
for recently.

#1 buildah - https://github.com/containers/buildah - as you can see in
this project they actively maintain a 'vendor' directory and regularly
make commits to the vendor directory to keep it up-to-date. If you do
something like the following you can verify that no additional downloads
are needed during a build

$> export GOPATH=/tmp/go
$> git clone https://github.com/containers/buildah
$> cd buildah
$> go build -mod vendor ./cmd/buildah
$> ls /tmp/go/pkg/mod/

The /tmp/go/pkg/mod directory remains empty and no download msgs are
seen during the build to confirm that no downloads take place.

#2 Minio Client (MC) - https://github.com/minio/mc - as you can see in
this project there is no maintained 'vendor' directory. Similarly if you
clone and build this project you will experience additional downloads

$> export GOPATH=/tmp/go
$> git clone https://github.com/minio/mc
$> cd mc
$> go build -trimpath -tags kqueue -o ./mc
$> ls /tmp/go/pkg/mod/

This time you will find /tmp/go/pkg/mod is filled with mods and during
the build download msgs will have been displayed.

For folks that follow the mailing lists you might have seen discussions
about writing recipes for each of the "dependencies" but as you will
have noted in these threads GO projects typically have their own
versions and even specific commits of dependencies which they are
building against. In most cases you want to use what is dictated in the
project's go.mod and such to not invalidate the projects build and
runtime testing.

Solutions for writing recipes for projects like Minio Client:

Potential (actually currently in use) solution #1 is what has been done
for a recipe like k3s
(https://git.yoctoproject.org/meta-virtualization/tree/recipes-containers/k3s/k3s_git.bb).
Where each dependency has a SRC_URI and related information, which is
used to populate the vendor directory.

Potential solution #2 would be to run 'go mod vendor' to populate the
'vendor' directory, after which a patch can be generated which would be
added to the SRC_URI and as long as we capture the license information
in the recipe for all "dependencies" we can build without worries about
additional download attempts.
This was looked at (and discarded) as a solution for meta-virtualization.
It can work, but has big maintenance drawbacks around patching, CVEs.
It also doesn't allow for download caching, etc, as it has no fetcher
integration. If you want to do something like this, you need to do a
fetcher implementation, setup module proxies, etc, rather than
generating a giant patch to restore 'vendor'
I think a recipe tool to write recipes for go projects is going to serve the community well, like what cargo-bitbake is in rust ecosystem. And then bitbake fetcher could aide a bit in fetching part.

Plus it makes us reliant on the golang module infrastructure, which
has shown to not always be reliable.
I would agree this was in flux before go modules were accepted as standard way of doing dependencies, there were many solutions for dependency management but I think we should now very much settle that it is go modules now, since its part of core tooling in go community now a days. So if we build tooling to use go modules underneath we wont be painting ourselves to a corner.



Potential solution #3 would be to add a 'go get' to the fetcher. This
would allow transposing go.mod information into the SRC_URI, drastically
reducing the tedious work done for solution #1.
This is difficult to implement, and leaves us with challenges for
locking down versions, licensing, etc. Definitely doable, but also
makes us again reliant on the go module infrastructure, unless we
setup mirrors, etc.


Potential solution #4 extend the tooling to facilitate the SRC_URI
generation for #1.
Not necessary. This is how #1 is actually done :)


Each of these have pros and cons, for instance #2 is the least amount of
work but also does nothing for saving download space when multiple GO
recipes do share dependencies. #3 and #2 are less secure compared to #1.
None of these address licensing very well...

I know this subject came up a few months ago but I don't recall there
being a resolution and since we are starting a new dev cycle I thought
now would be a good time to get thoughts from folks who have written GO
project recipes and take steps to making updates to moving this forward,
even if just by a little bit.
For go, we did come to a conclusion amongst the interested parties.
The "solution" that I'm using, is what you see in meta-virt. Use the
well established fetchers to go to the source repositories. That
allows us all the re-use and well tested functionality of git, etc,
and no need to re-invent the wheel.
I've written a tool that generates everything required for those
SRC_URIs, so for the most part, it isn't a manual process.
My presentation on the topic was accepted to the yocto summit, and
before that, I'll make the script available .. I just haven't had time
with all the release chaos and kernel issues at the moment.
Bruce


MarkA





Re: golang module handling

Mark Asselstine
 

Thanks for replying Bruce. I wasn't expecting to see anything until next week but you were someone I did expect to hear from.

On 2022-04-29 15:15, Bruce Ashfield wrote:
On Fri, Apr 29, 2022 at 2:59 PM Mark Asselstine
<mark.asselstine@...> wrote:

Folks,

For those of us who have had to work on GO recipes I think there has
been an open item that would be nice to get resolved in this new release
cycle.

Specifically, as with many of the modern languages GO includes its own
fetcher for 'dependencies', in GO these are referred to as 'modules'. As
it stands there is no unified or easy way to handle these and it would
be good to have this addressed in bitbake.

For example I can use two GO projects which we wanted to write recipes
for recently.

#1 buildah - https://github.com/containers/buildah - as you can see in
this project they actively maintain a 'vendor' directory and regularly
make commits to the vendor directory to keep it up-to-date. If you do
something like the following you can verify that no additional downloads
are needed during a build

$> export GOPATH=/tmp/go
$> git clone https://github.com/containers/buildah
$> cd buildah
$> go build -mod vendor ./cmd/buildah
$> ls /tmp/go/pkg/mod/

The /tmp/go/pkg/mod directory remains empty and no download msgs are
seen during the build to confirm that no downloads take place.

#2 Minio Client (MC) - https://github.com/minio/mc - as you can see in
this project there is no maintained 'vendor' directory. Similarly if you
clone and build this project you will experience additional downloads

$> export GOPATH=/tmp/go
$> git clone https://github.com/minio/mc
$> cd mc
$> go build -trimpath -tags kqueue -o ./mc
$> ls /tmp/go/pkg/mod/

This time you will find /tmp/go/pkg/mod is filled with mods and during
the build download msgs will have been displayed.

For folks that follow the mailing lists you might have seen discussions
about writing recipes for each of the "dependencies" but as you will
have noted in these threads GO projects typically have their own
versions and even specific commits of dependencies which they are
building against. In most cases you want to use what is dictated in the
project's go.mod and such to not invalidate the projects build and
runtime testing.

Solutions for writing recipes for projects like Minio Client:

Potential (actually currently in use) solution #1 is what has been done
for a recipe like k3s
(https://git.yoctoproject.org/meta-virtualization/tree/recipes-containers/k3s/k3s_git.bb).
Where each dependency has a SRC_URI and related information, which is
used to populate the vendor directory.

Potential solution #2 would be to run 'go mod vendor' to populate the
'vendor' directory, after which a patch can be generated which would be
added to the SRC_URI and as long as we capture the license information
in the recipe for all "dependencies" we can build without worries about
additional download attempts.
This was looked at (and discarded) as a solution for meta-virtualization.
It can work, but has big maintenance drawbacks around patching, CVEs.
It also doesn't allow for download caching, etc, as it has no fetcher
integration. If you want to do something like this, you need to do a
fetcher implementation, setup module proxies, etc, rather than
generating a giant patch to restore 'vendor'
Plus it makes us reliant on the golang module infrastructure, which
has shown to not always be reliable.

Agreed. In a pinch it can come in handy but long term I also believe it has many drawbacks. I just wanted to include it as it has come up and wanted to have it covered here.



Potential solution #3 would be to add a 'go get' to the fetcher. This
would allow transposing go.mod information into the SRC_URI, drastically
reducing the tedious work done for solution #1.
This is difficult to implement, and leaves us with challenges for
locking down versions, licensing, etc. Definitely doable, but also
makes us again reliant on the go module infrastructure, unless we
setup mirrors, etc.


Potential solution #4 extend the tooling to facilitate the SRC_URI
generation for #1.
Not necessary. This is how #1 is actually done :)
Well don't leave me hanging. Did I miss something that has been made available that I am blissfully/ignorantly unaware of, or is this something you have access to but hasn't been shared? Ah, more below.




Each of these have pros and cons, for instance #2 is the least amount of
work but also does nothing for saving download space when multiple GO
recipes do share dependencies. #3 and #2 are less secure compared to #1.
None of these address licensing very well...

I know this subject came up a few months ago but I don't recall there
being a resolution and since we are starting a new dev cycle I thought
now would be a good time to get thoughts from folks who have written GO
project recipes and take steps to making updates to moving this forward,
even if just by a little bit.
For go, we did come to a conclusion amongst the interested parties.
The "solution" that I'm using, is what you see in meta-virt. Use the
well established fetchers to go to the source repositories. That
allows us all the re-use and well tested functionality of git, etc,
and no need to re-invent the wheel.
I've written a tool that generates everything required for those
SRC_URIs, so for the most part, it isn't a manual process.
My presentation on the topic was accepted to the yocto summit, and
before that, I'll make the script available .. I just haven't had time
with all the release chaos and kernel issues at the moment.
Phew, so I am not ignorant, at least not in relation to the tooling. That would be great if you can share your work and I think it goes without saying that I appreciate your work on this.

I opted for this email instead of just pinging you on IRC as I thought it would be fair for others that might have followed the discussion from a few months back. Again I wasn't expecting a Friday afternoon response, but thanks for taking the time to get back so quickly. I look forward to seeing your scripts and talk.

Have a great weekend,
MarkA



Bruce


MarkA




Re: golang module handling

Bruce Ashfield
 

On Fri, Apr 29, 2022 at 2:59 PM Mark Asselstine
<mark.asselstine@...> wrote:

Folks,

For those of us who have had to work on GO recipes I think there has
been an open item that would be nice to get resolved in this new release
cycle.

Specifically, as with many of the modern languages GO includes its own
fetcher for 'dependencies', in GO these are referred to as 'modules'. As
it stands there is no unified or easy way to handle these and it would
be good to have this addressed in bitbake.

For example I can use two GO projects which we wanted to write recipes
for recently.

#1 buildah - https://github.com/containers/buildah - as you can see in
this project they actively maintain a 'vendor' directory and regularly
make commits to the vendor directory to keep it up-to-date. If you do
something like the following you can verify that no additional downloads
are needed during a build

$> export GOPATH=/tmp/go
$> git clone https://github.com/containers/buildah
$> cd buildah
$> go build -mod vendor ./cmd/buildah
$> ls /tmp/go/pkg/mod/

The /tmp/go/pkg/mod directory remains empty and no download msgs are
seen during the build to confirm that no downloads take place.

#2 Minio Client (MC) - https://github.com/minio/mc - as you can see in
this project there is no maintained 'vendor' directory. Similarly if you
clone and build this project you will experience additional downloads

$> export GOPATH=/tmp/go
$> git clone https://github.com/minio/mc
$> cd mc
$> go build -trimpath -tags kqueue -o ./mc
$> ls /tmp/go/pkg/mod/

This time you will find /tmp/go/pkg/mod is filled with mods and during
the build download msgs will have been displayed.

For folks that follow the mailing lists you might have seen discussions
about writing recipes for each of the "dependencies" but as you will
have noted in these threads GO projects typically have their own
versions and even specific commits of dependencies which they are
building against. In most cases you want to use what is dictated in the
project's go.mod and such to not invalidate the projects build and
runtime testing.

Solutions for writing recipes for projects like Minio Client:

Potential (actually currently in use) solution #1 is what has been done
for a recipe like k3s
(https://git.yoctoproject.org/meta-virtualization/tree/recipes-containers/k3s/k3s_git.bb).
Where each dependency has a SRC_URI and related information, which is
used to populate the vendor directory.

Potential solution #2 would be to run 'go mod vendor' to populate the
'vendor' directory, after which a patch can be generated which would be
added to the SRC_URI and as long as we capture the license information
in the recipe for all "dependencies" we can build without worries about
additional download attempts.
This was looked at (and discarded) as a solution for meta-virtualization.

It can work, but has big maintenance drawbacks around patching, CVEs.
It also doesn't allow for download caching, etc, as it has no fetcher
integration. If you want to do something like this, you need to do a
fetcher implementation, setup module proxies, etc, rather than
generating a giant patch to restore 'vendor'

Plus it makes us reliant on the golang module infrastructure, which
has shown to not always be reliable.


Potential solution #3 would be to add a 'go get' to the fetcher. This
would allow transposing go.mod information into the SRC_URI, drastically
reducing the tedious work done for solution #1.
This is difficult to implement, and leaves us with challenges for
locking down versions, licensing, etc. Definitely doable, but also
makes us again reliant on the go module infrastructure, unless we
setup mirrors, etc.


Potential solution #4 extend the tooling to facilitate the SRC_URI
generation for #1.
Not necessary. This is how #1 is actually done :)


Each of these have pros and cons, for instance #2 is the least amount of
work but also does nothing for saving download space when multiple GO
recipes do share dependencies. #3 and #2 are less secure compared to #1.
None of these address licensing very well...

I know this subject came up a few months ago but I don't recall there
being a resolution and since we are starting a new dev cycle I thought
now would be a good time to get thoughts from folks who have written GO
project recipes and take steps to making updates to moving this forward,
even if just by a little bit.
For go, we did come to a conclusion amongst the interested parties.

The "solution" that I'm using, is what you see in meta-virt. Use the
well established fetchers to go to the source repositories. That
allows us all the re-use and well tested functionality of git, etc,
and no need to re-invent the wheel.

I've written a tool that generates everything required for those
SRC_URIs, so for the most part, it isn't a manual process.

My presentation on the topic was accepted to the yocto summit, and
before that, I'll make the script available .. I just haven't had time
with all the release chaos and kernel issues at the moment.

Bruce


MarkA





--
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II


golang module handling

Mark Asselstine
 

Folks,

For those of us who have had to work on GO recipes I think there has been an open item that would be nice to get resolved in this new release cycle.

Specifically, as with many of the modern languages GO includes its own fetcher for 'dependencies', in GO these are referred to as 'modules'. As it stands there is no unified or easy way to handle these and it would be good to have this addressed in bitbake.

For example I can use two GO projects which we wanted to write recipes for recently.

#1 buildah - https://github.com/containers/buildah - as you can see in this project they actively maintain a 'vendor' directory and regularly make commits to the vendor directory to keep it up-to-date. If you do something like the following you can verify that no additional downloads are needed during a build

$> export GOPATH=/tmp/go
$> git clone https://github.com/containers/buildah
$> cd buildah
$> go build -mod vendor ./cmd/buildah
$> ls /tmp/go/pkg/mod/

The /tmp/go/pkg/mod directory remains empty and no download msgs are seen during the build to confirm that no downloads take place.

#2 Minio Client (MC) - https://github.com/minio/mc - as you can see in this project there is no maintained 'vendor' directory. Similarly if you clone and build this project you will experience additional downloads

$> export GOPATH=/tmp/go
$> git clone https://github.com/minio/mc
$> cd mc
$> go build -trimpath -tags kqueue -o ./mc
$> ls /tmp/go/pkg/mod/

This time you will find /tmp/go/pkg/mod is filled with mods and during the build download msgs will have been displayed.

For folks that follow the mailing lists you might have seen discussions about writing recipes for each of the "dependencies" but as you will have noted in these threads GO projects typically have their own versions and even specific commits of dependencies which they are building against. In most cases you want to use what is dictated in the project's go.mod and such to not invalidate the projects build and runtime testing.

Solutions for writing recipes for projects like Minio Client:

Potential (actually currently in use) solution #1 is what has been done for a recipe like k3s (https://git.yoctoproject.org/meta-virtualization/tree/recipes-containers/k3s/k3s_git.bb). Where each dependency has a SRC_URI and related information, which is used to populate the vendor directory.

Potential solution #2 would be to run 'go mod vendor' to populate the 'vendor' directory, after which a patch can be generated which would be added to the SRC_URI and as long as we capture the license information in the recipe for all "dependencies" we can build without worries about additional download attempts.

Potential solution #3 would be to add a 'go get' to the fetcher. This would allow transposing go.mod information into the SRC_URI, drastically reducing the tedious work done for solution #1.

Potential solution #4 extend the tooling to facilitate the SRC_URI generation for #1.

Each of these have pros and cons, for instance #2 is the least amount of work but also does nothing for saving download space when multiple GO recipes do share dependencies. #3 and #2 are less secure compared to #1. None of these address licensing very well...

I know this subject came up a few months ago but I don't recall there being a resolution and since we are starting a new dev cycle I thought now would be a good time to get thoughts from folks who have written GO project recipes and take steps to making updates to moving this forward, even if just by a little bit.

MarkA


Re: Dependent task hashes in depsig.*

Richard Purdie
 

On Wed, 2022-04-27 at 08:26 -0500, Joshua Watt wrote:
On Wed, Apr 27, 2022 at 6:04 AM Richard Purdie
<richard.purdie@...> wrote:

On Wed, 2022-04-27 at 12:39 +0200, Jacob Kroon wrote:
On 4/27/22 12:12, Richard Purdie wrote:
On Wed, 2022-04-27 at 11:06 +0200, Jacob Kroon wrote:
Hi Richard and Joshua,

When using hash equivalency, since commits

https://git.openembedded.org/openembedded-core/commit/?id=d6c7b9f4f0e
https://git.openembedded.org/openembedded-core/commit/?id=1cf62882bba

scrambling a header in one of the gcc patches causes all target packages
to rebuild.
That is probably unfortunately inevitable. If the output has changed (i.e. the
headers are different), it shouldn't be matching a previous build as we can't
know what has changed.
I don't think I was being clear enough. With "scrambling a header in a
gcc patch" I mean that I change something in the section before the
"---" line in a .patch file, like modifying the "Upstream-Status" line.
To me, hash equivalence should be able to optimize that scenario, since
the output from building gcc-cross is not changed.
That definitely wasn't clear! I now understand better what you mean and yes,
we're supposed to be optimising that scenario.


This is because the depsig.do_populate_sysroot in "libgcc"
changes:

[jkroon@fedora work]$ diff -u i686-oe-linux/libgcc/11.3.0-r0/temp/depsig.do_populate_sysroot.1*
--- i686-oe-linux/libgcc/11.3.0-r0/temp/depsig.do_populate_sysroot.1589812 2022-04-27 10:14:22.403251775 +0200
+++ i686-oe-linux/libgcc/11.3.0-r0/temp/depsig.do_populate_sysroot.1674014 2022-04-27 10:26:45.329365448 +0200
@@ -1,7 +1,7 @@
OEOuthashBasic
12
glibc: 8feab297dd38b103daa4f26eeabb5690a74b8b5700d16e4eca7b56e6fd667a5e
-libgcc: dfd38409a4cc5320b781edc14de2af8321180c3f194a58b798870ad7ff6a9226
+libgcc: 195f6a155dac8e450e72a7432ab91959a8e095e057d5b79e3adba41721dc7ea5
linux-libc-headers: 12a5aaf8aec9554ac3c778cdc6c65df4db52fc573e84b7110572d459a15c9d6a
SSTATE_PKGSPEC=sstate:libgcc:i686-oe-linux:11.3.0:r0:i686:8:
task=populate_sysroot
Is it the case that it is the dependent task hashes that are added
above, and that the checksum of patches are included in the those task
hashes ?
The dependent resolved hashes are used, as resolved by hashequiv which is a key
difference.
So it is the outhashes that are listed above?
No, they are hash equiv resolved hashes which would have a one to one mapping
with an outhash.

Then I don't understand
the diff above. libgcc depends on itself ? But apparently no files in
the sysroot changed, since the above is the only diff I get.
To be honest, I don't remember/understand offhand either. I'd need to go and
spend time trying to page in all the information.

We have too few people with the knowledge in these areas and I'm rapidly burning
out. I don't like this reply but I just don't have the time to dive into it and
debug it right now and I can't really give much more of a helpful comment/reply
without doing so.

I agree there is some issue here which does need investigation. At least do file
a bug so it doesn't get forgotten but we don't have many people taking on bugs
either. This one would get triaged to me or Joshua.

I'd also add that gcc is pretty horrific in that it bundles up a lot of it's
build tree into the sysroot. It is possible those bundled files are varying
somehow reproducibility wise causing some instability. I've worried about this
kind of issue for a while but I don't scale and there are a load of other issues
going on too :(.
It looks like this could possibly be a bug in the code in
meta/classes/staging.bbclass that injects the dependencies into
do_populate_sysroot & do_package.

It doesn't explicitly filter out ${PN} from BB_TASKDEPDATA, although
it perhaps should? I'm not sure if the bug is that
libgcc:do_populate_sysroot is in BB_TASKDEPDATA to begin with, or that
the code isn't filtering it out. If we do need to filter it out,
that's a pretty easy change to make.
The current task has to be in BB_TASKDEPDATA so I think you're right, we should
filter out "ourselves". I'll send a patch.

Cheers,

Richard


Re: Dependent task hashes in depsig.*

Joshua Watt
 

On Wed, Apr 27, 2022 at 6:04 AM Richard Purdie
<richard.purdie@...> wrote:

On Wed, 2022-04-27 at 12:39 +0200, Jacob Kroon wrote:
On 4/27/22 12:12, Richard Purdie wrote:
On Wed, 2022-04-27 at 11:06 +0200, Jacob Kroon wrote:
Hi Richard and Joshua,

When using hash equivalency, since commits

https://git.openembedded.org/openembedded-core/commit/?id=d6c7b9f4f0e
https://git.openembedded.org/openembedded-core/commit/?id=1cf62882bba

scrambling a header in one of the gcc patches causes all target packages
to rebuild.
That is probably unfortunately inevitable. If the output has changed (i.e. the
headers are different), it shouldn't be matching a previous build as we can't
know what has changed.
I don't think I was being clear enough. With "scrambling a header in a
gcc patch" I mean that I change something in the section before the
"---" line in a .patch file, like modifying the "Upstream-Status" line.
To me, hash equivalence should be able to optimize that scenario, since
the output from building gcc-cross is not changed.
That definitely wasn't clear! I now understand better what you mean and yes,
we're supposed to be optimising that scenario.


This is because the depsig.do_populate_sysroot in "libgcc"
changes:

[jkroon@fedora work]$ diff -u i686-oe-linux/libgcc/11.3.0-r0/temp/depsig.do_populate_sysroot.1*
--- i686-oe-linux/libgcc/11.3.0-r0/temp/depsig.do_populate_sysroot.1589812 2022-04-27 10:14:22.403251775 +0200
+++ i686-oe-linux/libgcc/11.3.0-r0/temp/depsig.do_populate_sysroot.1674014 2022-04-27 10:26:45.329365448 +0200
@@ -1,7 +1,7 @@
OEOuthashBasic
12
glibc: 8feab297dd38b103daa4f26eeabb5690a74b8b5700d16e4eca7b56e6fd667a5e
-libgcc: dfd38409a4cc5320b781edc14de2af8321180c3f194a58b798870ad7ff6a9226
+libgcc: 195f6a155dac8e450e72a7432ab91959a8e095e057d5b79e3adba41721dc7ea5
linux-libc-headers: 12a5aaf8aec9554ac3c778cdc6c65df4db52fc573e84b7110572d459a15c9d6a
SSTATE_PKGSPEC=sstate:libgcc:i686-oe-linux:11.3.0:r0:i686:8:
task=populate_sysroot
Is it the case that it is the dependent task hashes that are added
above, and that the checksum of patches are included in the those task
hashes ?
The dependent resolved hashes are used, as resolved by hashequiv which is a key
difference.
So it is the outhashes that are listed above?
No, they are hash equiv resolved hashes which would have a one to one mapping
with an outhash.

Then I don't understand
the diff above. libgcc depends on itself ? But apparently no files in
the sysroot changed, since the above is the only diff I get.
To be honest, I don't remember/understand offhand either. I'd need to go and
spend time trying to page in all the information.

We have too few people with the knowledge in these areas and I'm rapidly burning
out. I don't like this reply but I just don't have the time to dive into it and
debug it right now and I can't really give much more of a helpful comment/reply
without doing so.

I agree there is some issue here which does need investigation. At least do file
a bug so it doesn't get forgotten but we don't have many people taking on bugs
either. This one would get triaged to me or Joshua.

I'd also add that gcc is pretty horrific in that it bundles up a lot of it's
build tree into the sysroot. It is possible those bundled files are varying
somehow reproducibility wise causing some instability. I've worried about this
kind of issue for a while but I don't scale and there are a load of other issues
going on too :(.
It looks like this could possibly be a bug in the code in
meta/classes/staging.bbclass that injects the dependencies into
do_populate_sysroot & do_package.

It doesn't explicitly filter out ${PN} from BB_TASKDEPDATA, although
it perhaps should? I'm not sure if the bug is that
libgcc:do_populate_sysroot is in BB_TASKDEPDATA to begin with, or that
the code isn't filtering it out. If we do need to filter it out,
that's a pretty easy change to make.


Cheers,

Richard


Re: Dependent task hashes in depsig.*

Richard Purdie
 

On Wed, 2022-04-27 at 12:39 +0200, Jacob Kroon wrote:
On 4/27/22 12:12, Richard Purdie wrote:
On Wed, 2022-04-27 at 11:06 +0200, Jacob Kroon wrote:
Hi Richard and Joshua,

When using hash equivalency, since commits

https://git.openembedded.org/openembedded-core/commit/?id=d6c7b9f4f0e
https://git.openembedded.org/openembedded-core/commit/?id=1cf62882bba

scrambling a header in one of the gcc patches causes all target packages
to rebuild.
That is probably unfortunately inevitable. If the output has changed (i.e. the
headers are different), it shouldn't be matching a previous build as we can't
know what has changed.
I don't think I was being clear enough. With "scrambling a header in a
gcc patch" I mean that I change something in the section before the
"---" line in a .patch file, like modifying the "Upstream-Status" line.
To me, hash equivalence should be able to optimize that scenario, since
the output from building gcc-cross is not changed.
That definitely wasn't clear! I now understand better what you mean and yes,
we're supposed to be optimising that scenario.


This is because the depsig.do_populate_sysroot in "libgcc"
changes:

[jkroon@fedora work]$ diff -u i686-oe-linux/libgcc/11.3.0-r0/temp/depsig.do_populate_sysroot.1*
--- i686-oe-linux/libgcc/11.3.0-r0/temp/depsig.do_populate_sysroot.1589812 2022-04-27 10:14:22.403251775 +0200
+++ i686-oe-linux/libgcc/11.3.0-r0/temp/depsig.do_populate_sysroot.1674014 2022-04-27 10:26:45.329365448 +0200
@@ -1,7 +1,7 @@
OEOuthashBasic
12
glibc: 8feab297dd38b103daa4f26eeabb5690a74b8b5700d16e4eca7b56e6fd667a5e
-libgcc: dfd38409a4cc5320b781edc14de2af8321180c3f194a58b798870ad7ff6a9226
+libgcc: 195f6a155dac8e450e72a7432ab91959a8e095e057d5b79e3adba41721dc7ea5
linux-libc-headers: 12a5aaf8aec9554ac3c778cdc6c65df4db52fc573e84b7110572d459a15c9d6a
SSTATE_PKGSPEC=sstate:libgcc:i686-oe-linux:11.3.0:r0:i686:8:
task=populate_sysroot
Is it the case that it is the dependent task hashes that are added
above, and that the checksum of patches are included in the those task
hashes ?
The dependent resolved hashes are used, as resolved by hashequiv which is a key
difference.
So it is the outhashes that are listed above?
No, they are hash equiv resolved hashes which would have a one to one mapping
with an outhash.

Then I don't understand
the diff above. libgcc depends on itself ? But apparently no files in
the sysroot changed, since the above is the only diff I get.
To be honest, I don't remember/understand offhand either. I'd need to go and
spend time trying to page in all the information.

We have too few people with the knowledge in these areas and I'm rapidly burning
out. I don't like this reply but I just don't have the time to dive into it and
debug it right now and I can't really give much more of a helpful comment/reply
without doing so.

I agree there is some issue here which does need investigation. At least do file
a bug so it doesn't get forgotten but we don't have many people taking on bugs
either. This one would get triaged to me or Joshua.

I'd also add that gcc is pretty horrific in that it bundles up a lot of it's
build tree into the sysroot. It is possible those bundled files are varying
somehow reproducibility wise causing some instability. I've worried about this
kind of issue for a while but I don't scale and there are a load of other issues
going on too :(.

Cheers,

Richard


Re: Dependent task hashes in depsig.*

Jacob Kroon
 

On 4/27/22 12:12, Richard Purdie wrote:
On Wed, 2022-04-27 at 11:06 +0200, Jacob Kroon wrote:
Hi Richard and Joshua,

When using hash equivalency, since commits

https://git.openembedded.org/openembedded-core/commit/?id=d6c7b9f4f0e
https://git.openembedded.org/openembedded-core/commit/?id=1cf62882bba

scrambling a header in one of the gcc patches causes all target packages
to rebuild.
That is probably unfortunately inevitable. If the output has changed (i.e. the
headers are different), it shouldn't be matching a previous build as we can't
know what has changed.
I don't think I was being clear enough. With "scrambling a header in a
gcc patch" I mean that I change something in the section before the
"---" line in a .patch file, like modifying the "Upstream-Status" line.
To me, hash equivalence should be able to optimize that scenario, since
the output from building gcc-cross is not changed.

This is because the depsig.do_populate_sysroot in "libgcc"
changes:

[jkroon@fedora work]$ diff -u i686-oe-linux/libgcc/11.3.0-r0/temp/depsig.do_populate_sysroot.1*
--- i686-oe-linux/libgcc/11.3.0-r0/temp/depsig.do_populate_sysroot.1589812 2022-04-27 10:14:22.403251775 +0200
+++ i686-oe-linux/libgcc/11.3.0-r0/temp/depsig.do_populate_sysroot.1674014 2022-04-27 10:26:45.329365448 +0200
@@ -1,7 +1,7 @@
OEOuthashBasic
12
glibc: 8feab297dd38b103daa4f26eeabb5690a74b8b5700d16e4eca7b56e6fd667a5e
-libgcc: dfd38409a4cc5320b781edc14de2af8321180c3f194a58b798870ad7ff6a9226
+libgcc: 195f6a155dac8e450e72a7432ab91959a8e095e057d5b79e3adba41721dc7ea5
linux-libc-headers: 12a5aaf8aec9554ac3c778cdc6c65df4db52fc573e84b7110572d459a15c9d6a
SSTATE_PKGSPEC=sstate:libgcc:i686-oe-linux:11.3.0:r0:i686:8:
task=populate_sysroot
Is it the case that it is the dependent task hashes that are added
above, and that the checksum of patches are included in the those task
hashes ?
The dependent resolved hashes are used, as resolved by hashequiv which is a key
difference.
So it is the outhashes that are listed above ? Then I don't understand
the diff above. libgcc depends on itself ? But apparently no files in
the sysroot changed, since the above is the only diff I get.

In order to solve the original problem that those patches were fixing,
would it not be possible to instead include the *outhashes* of the
dependent recipes ?
Since the resolved hashes should map to a single outhash, I don't think it would
change anything?
Jacob


Re: Dependent task hashes in depsig.*

Richard Purdie
 

On Wed, 2022-04-27 at 11:06 +0200, Jacob Kroon wrote:
Hi Richard and Joshua,

When using hash equivalency, since commits

https://git.openembedded.org/openembedded-core/commit/?id=d6c7b9f4f0e
https://git.openembedded.org/openembedded-core/commit/?id=1cf62882bba

scrambling a header in one of the gcc patches causes all target packages
to rebuild.
That is probably unfortunately inevitable. If the output has changed (i.e. the
headers are different), it shouldn't be matching a previous build as we can't
know what has changed.

This is because the depsig.do_populate_sysroot in "libgcc"
changes:

[jkroon@fedora work]$ diff -u i686-oe-linux/libgcc/11.3.0-r0/temp/depsig.do_populate_sysroot.1*
--- i686-oe-linux/libgcc/11.3.0-r0/temp/depsig.do_populate_sysroot.1589812 2022-04-27 10:14:22.403251775 +0200
+++ i686-oe-linux/libgcc/11.3.0-r0/temp/depsig.do_populate_sysroot.1674014 2022-04-27 10:26:45.329365448 +0200
@@ -1,7 +1,7 @@
OEOuthashBasic
12
glibc: 8feab297dd38b103daa4f26eeabb5690a74b8b5700d16e4eca7b56e6fd667a5e
-libgcc: dfd38409a4cc5320b781edc14de2af8321180c3f194a58b798870ad7ff6a9226
+libgcc: 195f6a155dac8e450e72a7432ab91959a8e095e057d5b79e3adba41721dc7ea5
linux-libc-headers: 12a5aaf8aec9554ac3c778cdc6c65df4db52fc573e84b7110572d459a15c9d6a
SSTATE_PKGSPEC=sstate:libgcc:i686-oe-linux:11.3.0:r0:i686:8:
task=populate_sysroot
Is it the case that it is the dependent task hashes that are added
above, and that the checksum of patches are included in the those task
hashes ?
The dependent resolved hashes are used, as resolved by hashequiv which is a key
difference.

In order to solve the original problem that those patches were fixing,
would it not be possible to instead include the *outhashes* of the
dependent recipes ?
Since the resolved hashes should map to a single outhash, I don't think it would
change anything?

Cheers,

Richard


Dependent task hashes in depsig.*

Jacob Kroon
 

Hi Richard and Joshua,

When using hash equivalency, since commits

https://git.openembedded.org/openembedded-core/commit/?id=d6c7b9f4f0e
https://git.openembedded.org/openembedded-core/commit/?id=1cf62882bba

scrambling a header in one of the gcc patches causes all target packages
to rebuild. This is because the depsig.do_populate_sysroot in "libgcc"
changes:

[jkroon@fedora work]$ diff -u i686-oe-linux/libgcc/11.3.0-r0/temp/depsig.do_populate_sysroot.1*
--- i686-oe-linux/libgcc/11.3.0-r0/temp/depsig.do_populate_sysroot.1589812 2022-04-27 10:14:22.403251775 +0200
+++ i686-oe-linux/libgcc/11.3.0-r0/temp/depsig.do_populate_sysroot.1674014 2022-04-27 10:26:45.329365448 +0200
@@ -1,7 +1,7 @@
OEOuthashBasic
12
glibc: 8feab297dd38b103daa4f26eeabb5690a74b8b5700d16e4eca7b56e6fd667a5e
-libgcc: dfd38409a4cc5320b781edc14de2af8321180c3f194a58b798870ad7ff6a9226
+libgcc: 195f6a155dac8e450e72a7432ab91959a8e095e057d5b79e3adba41721dc7ea5
linux-libc-headers: 12a5aaf8aec9554ac3c778cdc6c65df4db52fc573e84b7110572d459a15c9d6a
SSTATE_PKGSPEC=sstate:libgcc:i686-oe-linux:11.3.0:r0:i686:8:
task=populate_sysroot
Is it the case that it is the dependent task hashes that are added
above, and that the checksum of patches are included in the those task
hashes ?

In order to solve the original problem that those patches were fixing,
would it not be possible to instead include the *outhashes* of the
dependent recipes ?

Jacob


Re: [Automated-testing] RFC Linter in meta-openembedded

Richard Purdie
 

On Sat, 2022-04-16 at 17:15 +0200, Marius Kriegerowski wrote:
Dear Richard, 

Thanks for the comprehensive answer! That helped me to get a better overview of
what is there and how to proceed.

I'll leave cosmetics for a later phase, stick to unittest and fix the
integration so that we have a baseline from where to decide how to proceed.
FYI I did notice some docs here:

https://git.yoctoproject.org/patchtest/tree/usage.adoc

Cheers,

Richard

141 - 160 of 1685