Re: [oe] INCOMPATIBLE_LICENSES and WHITELIST_<license> usage


Peter Kjellerstedt
 

Warning: wall of text ahead. Sorry about that, but I believe it is
important that we get this right if we are redesigning it.

TL;DR: see the end for the new suggested licensing variables.

-----Original Message-----
From: openembedded-devel@... <openembedded-
devel@...> On Behalf Of Richard Purdie
Sent: den 18 februari 2022 15:14
To: Saul Wold <Saul.Wold@...>; openembedded-
architecture@...; OE-core <openembedded-
core@...>; OpenEmbedded Devel List <openembedded-
devel@...>
Subject: Re: [oe] INCOMPATIBLE_LICENSES and WHITELIST_<license> usage

On Thu, 2022-02-17 at 15:01 -0800, Saul Wold wrote:
I am working on a proposal to re-write how INCOMPATIBLE_LICENSES is used
and processed to possibly include a COMPATIBLE_LICENSES variable as
well, see PeterK's email [0]

I am trying to determine the usage of WHITELIST_<license> which would be
used to override a license that might be listed in INCOMPATIBLE_LICENSES
variable.

Randy and I have done a quick and dirty survey of a 100 or so layers
(thanks Randy) and could not find any real usage other than what's
currently in OE-Core for WHITELIST_GPL-3.0.

If you are using WHITELIST_<license>, please let me reply with your
usage.


[0] https://lists.openembedded.org/g/openembedded-devel/message/95166
We need to be mindful that we need to resolve this to unblock the other
language changes and feature creep here is potentially problematic. I do
think it is worth trying to improve things rather than blindly allowing
the horrible syntax in this variable to continue though.

The test case we have for this currently is:

WHITELIST_GPL-3.0:pn-core-image-minimal = "bash"

so I'd wondered about an alternative of:

INCOMPATIBLE_LICENSE_EXCEPTIONS:pn-core-image-minimal = "bash:GPL-3.0"
You do not really need the license here (more than possibly as
information to anyone reading the code). Specifying that a package is
allowed in an image regardless of its licenses can just as well be
done by allowing the variable to take a list of packages:

INCOMPATIBLE_LICENSE_EXCEPTIONS:pn-core-image-minimal = "bash readline"

An alternative (and in my opinion better) variable name would be:

IMAGE_ALLOW_PACKAGES:pn-core-image-minimal = "bash readline"

focusing on that this is a list of _packages_ allowed for a given
_image_. This does not explicitly mention licenses, but I do not
believe that is needed. After all there could be multiple reasons a
package is not allowed in an image and this variable would allow to
ignore them all because that is typically what you want to do:
specify that you really want that package in that image. I guess
this is in some sense the opposite of PACKAGE_EXCLUDE. And I guess
like :append vs :remove, if someone for some reason specifies a
package in both IMAGE_ALLOW_PACKAGES and PACKAGE_EXCLUDE, then the
later should win (to err on the side of caution).

For the case where you want to allow a recipe to be built despite
it using licenses that are otherwise not allowed you can simply use

INCOMPATIBLE_LICENSE:pn-foo = ""

and don't really need a separate variable for it.

which matches the current functionality, removes the issue that the name
of the variable is unknown without iterating every possible license name
and makes it clear where it is applying to.

I don't really like INCOMPATIBLE_LICENSE_ALLOWED_RECIPES since:

a) it is long
b) it refers to recipes when it works against packages

(INCOMPATIBLE_LICENSE_PACKAGE_EXCEPTIONS is more correct but still long)

I do like it mentioning INCOMPATIBLE_LICENSE in full since it works in
conjunction with that variable and that is definitely not clear from the
current WHITELIST_XXX until you look at the code.

I'm still of the opinion the AVAILABLE_LICENSES variable is something we
should be aiming to remove ultimately too, it has horrible issues with
layers changing hashes for all recipes.
Since I was the one to introduce it, I will answer to that. It was
introduced so that it is possible to specify compatible licenses
instead of incompatible licenses by basically calculating the
incompatible licenses by taking the set of available licenses minus
the set of compatible licenses. If a mechanism to do that is
introduced, e.g., by adding support for a COMPATIBLE_LICENSES in
addition to INCOMPATIBLE_LICENSE, then I am of the opinion that we
can remove AVAILABLE_LICENSES again.

We also need this mechanism in the code for handling allowed vs
disallowed licenses because the current code really cannot handled a
list of many hundreds of incompatible licenses, which is what we got
after all SPDX licenses were added to OE-Core. The code is extremely
inefficient and evaluates the list of incompatible licenses over and
over again. In our case that meant the recipe parsing time tripled.

Cheers,

Richard
That said, we really need two sets of variables. One for blocking
the building of recipes because of its license(s), and one for
preventing packages with disallowed licenses to be included in a
given image. These are very different use cases and they take
totally different lists of items (recipes vs packages). The current
mess where the same variables are used for both cases needs to be
resolved.

So why do we need both cases? The first case where recipes are
prevented from being built makes it possible to, e.g., prevent a
newer versioned recipe that uses GPL-3.0-or-later to be built and
instead build an older version that uses GPL-2.0-or-later (i.e.,
what meta-gplv2 can be used for today).

The second case allows to build code that uses disallowed licenses,
as long as the packages are not added to the image. Why is this
useful? Because many recipes are built for some packages that are
never used in the given image, and it is then much easier to allow
them to be built as long as their output is not used. E.g., very
many recipes depend on bash which is GPL-3.0-or-later, making it
near impossible to avoid having to build it. However, it is
perfectly possible to build production images that do not need bash
to be installed.

I believe for this second case we should have two variables,
IMAGE_COMPATIBLE_LICENSES and IMAGE_INCOMPATIBLE_LICENSES. And to
make the naming clear for the first case, I would suggest calling
those variables RECIPE_COMPATIBLE_LICENSES and
RECIPE_INCOMPATIBLE_LICENSES.

Also note that the use of RECIPE_COMPATIBLE_LICENSES and
RECIPE_INCOMPATIBLE_LICENSES is mutually exclusive, as is the use
of IMAGE_COMPATIBLE_LICENSES and IMAGE_INCOMPATIBLE_LICENSES. I.e.,
you will have to choose whether to specify what licenses to allow
or what licenses to disallow. You cannot do both. This is because
specifying a list of compatible licenses means that all other
licenses are by definition incompatible, and vice versa. However,
this also means that it makes perfect sense to be able to, e.g.,
specify a few RECIPE_INCOMPATIBLE_LICENSES together with a list of
IMAGE_COMPATIBLE_LICENSES.

Oh, and another thing I would like to take the opportunity to raise
is whether we should continue to support patterns in these list, or
if we should change it so that the lists of licenses need to
explicitly specify all licenses. The latter would greatly simplify
the code, especially when combined with only allowing SPDX names.
If we decide to remove support for patterns, and based on the
assumption that the pattern is typically used to specify "*GPL-3.0*",
we could make available a variable or two that contain the typical
sets of GPL licenses. This would also allow us to remove the code
that handles how an or-later licenses specified as '<license>+'
should be treated in combination with patterns.

So, to reiterate, these are the variables that I suggest we adopt
to be able to handle the various use cases regarding licenses:

RECIPE_COMPATIBLE_LICENSES - list of compatible licenses for
recipes
RECIPE_INCOMPATIBLE_LICENSES - list of incompatible licenses for
recipes
IMAGE_COMPATIBLE_LICENSES - list of compatible licenses for
packages in images
IMAGE_INCOMPATIBLE_LICENSES - list of incompatible licenses for
packages in images
IMAGE_ALLOW_PACKAGES - list of packages allowed in the
image regardless of licenses and
other restrictions

This also means that the old WHITELIST_*, INCOMPATIBLE_LICENSE
and AVAILABLE_LICENSES variables are removed. I also suggest we
remove the support for patterns in the new variables.

//Peter

Join {openembedded-architecture@lists.openembedded.org to automatically receive all group messages.