On Tue, Jan 18, 2022 at 8:40 AM Richard Purdie
<richard.purdie@...> wrote:

On Tue, 2022-01-18 at 14:00 +0100, Stefan Herbrechtsmeier wrote:

In summary we use a language specific lock file based approach which
support offline build, license checks and CVE scans and leaves the
dependency management and fixing outside of OE to limit the recipe count
and required resources.

I think so. It isn't the perfect solution but it is what will likely be the most
successful/practical.

I think there might be a slight ambiguity in what people think of as
dependency management.

We've been allowing projects in go to carry their dependencies using
vendor/ directories since almost the beginning, but we have in the
past patched and updated those dependencies ourselves (by overlaying
the directory via clone, or by patching) to fix both bugs and CVEs.

Having the actual hashes specified by the main project repository and
than translated/transformed into something that our existing fetchers
can handle (both the fetch and placement into the appropriate
directory for language specific proxies/mirrors/search paths) isn't
really leaving the CVE and dependency management outside of OE. It is
leaving it outside of the OE package dependencies, but they are still
controlled, understood and something we can patch/manage within
recipes if required.

Bruce

Should this be unified between Node.js / npm, Go, Rust / Cargo and
Python / Pipfile?

I don't think it makes sense to dictate that and make a hard rule. Where there
are many dependencies and we can't easily control the dependency mechanism in
the language, yes. Not everything has as granular dependencies as npm though.

The objectives we're trying to meet remain the same. How we do that could vary
case by case.

This means you have to manage the dependencies outside of OE. This leads to the
following questions:
1) Should it be possible to use a lock file from the source?
2) Should it be possible to patch the lock file?
3) Should it be possible to patch a dependency?
4) Must a patch be applied to all packages with the same version or
should it be applied to an individual package inside the dependency tree?
4Hi Richard,) Should the recipe detect license changes of dependencies?
5) Should the recipetool or build process generate the licenses and
license checksums?
6) Should the recipetool or build process extract the CVE product name
and version?

It would be nice if you could help me to get a clear vision about the
integration of language specific package managers so that I can adapt my
work.

I'm acutely aware that I'm not the one doing this work and I don't really want
to impose impractical constraints. My response has therefore been open ended
deliberately as I don't want to pin us into a corner. I've highlighted the
specific issues I'm aware of, e.g. that thousands of dependencies at the bitbake
level likely won't work and I'm not aware of any way to make that happen right
now. I don't really know the correct answer to some of the above questions.

Okay, I will use a minimal solution for now.

Ideally, yes, tools would generate the correct license and CVE data. Ideally,
the build would verify that data is still correct, much as we do with the
license checksums for other recipes.

I will check if the CVE scan can support multiple packages.

The good news there is we own that code so we have some flexibility to extend it
if/as needed.

Cheers,

Richard

--
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II