On Tue, Jul 27, 2021 at 08:18:13AM +0100, Richard Purdie wrote:

On Mon, 2021-07-26 at 23:51 -0400, Denys Dmytriyenko wrote:

All,

The layer has been updated and moved to git.yoctoproject.org under a central
mixin repo called "meta-lts-mixins", branch "dunfell/kernel":
http://git.yoctoproject.org/cgit/cgit.cgi/meta-lts-mixins/

The naming is in accordance with the LTS mixin layer guidelines:
https://wiki.yoctoproject.org/wiki/Stable_Release_and_LTS#LTS_.E2.80.9CMixin.E2.80.9D_repositories

Other mixin layers are also welcome!

Should the layer be called kernel-5.10 since there potentially may be other
versions?

Either that, or it can hold multiple versions...

On Mon, 2021-07-26 at 23:51 -0400, Denys Dmytriyenko wrote:

All,

The layer has been updated and moved to git.yoctoproject.org under a central
mixin repo called "meta-lts-mixins", branch "dunfell/kernel":
http://git.yoctoproject.org/cgit/cgit.cgi/meta-lts-mixins/

The naming is in accordance with the LTS mixin layer guidelines:
https://wiki.yoctoproject.org/wiki/Stable_Release_and_LTS#LTS_.E2.80.9CMixin.E2.80.9D_repositories

Other mixin layers are also welcome!

Should the layer be called kernel-5.10 since there potentially may be other
versions?

All,

The layer has been updated and moved to git.yoctoproject.org under a central
mixin repo called "meta-lts-mixins", branch "dunfell/kernel":
http://git.yoctoproject.org/cgit/cgit.cgi/meta-lts-mixins/

The naming is in accordance with the LTS mixin layer guidelines:
https://wiki.yoctoproject.org/wiki/Stable_Release_and_LTS#LTS_.E2.80.9CMixin.E2.80.9D_repositories

Other mixin layers are also welcome!

On Mon, 19 Jul 2021 13:55:42 -0400
"Denys Dmytriyenko" <denis@...> wrote:

All,

Recently several Yocto Project members have expressed interest in being able
to upgrade the Linux kernel in Dunfell LTS release from 5.4 to 5.10. Both of
those Linux kernel versions are designated "longterm"[1], but at the time
Dunfell was released in April 2020, 5.10 was not out yet.

Recommended way to add new features or upgrade components in Yocto LTS
releases is with the so called "mixin" layers[2]. After discussing some of
the details of such layer among the Yocto Project TSC and other interested
developers, as well as testing the concept for few weeks, I'd like to make
this open to a wider consumption in hopes it would be useful to others.

This Linux kernel "mixin" layer is currently hosted on GitHub under my
name[3], but can be moved to a more public space, like Yocto Project git
server, if it gets traction and is found useful to others. Please feel free
to try it and provide feedback. Thanks!

[1] https://kernel.org/
[2] https://wiki.yoctoproject.org/wiki/Stable_Release_and_LTS#LTS_.E2.80.9CMixin.E2.80.9D_repositories
[3] https://github.com/denix0/meta-lts-kernel-mixin

Hi Denys,

This looks like a good initiative. A couple of initial comments:

layer.conf contains `PREFERRED_VERSION_linux-yocto ?= "5.10%"`. It'd
be better to leave that out and in the readme say that users need to
add this in their distro or local conf. That would bring the layer in
line with best practices and what's needed for Yocto Project Compatible
status as adding simply a layer to a build should not result in changes
to the images you build.

The layer needs a COPYING or LICENSE file.

It'd be good to explicitly say how contributions to the layer can be
made - should this be by GitHub pull request or via a mailing list?

Did someone already backport the change to Dunfell to fix the module.lds for out
of tree kernel builds?

Master commit 596b0474d3d?

For gatesgarth, we ended up porting a copy of this into our distribution branch,
as it was rejected for a backport.

Also there was a necessary change for kernel-devsrc as well, I believe the
oe-core commit was: cb940d8af359fa370254bd4c2b36ba26708bb54b

See:

https://github.com/Xilinx/meta-petalinux/tree/rel-v2021.1/oe-core/meta

classes/kernel.bbclass and recipes-kernel/linux

this was the best way we could find to override these files for gatesgarth.

--Mark


On 7/19/21 12:55 PM, Denys Dmytriyenko wrote:

All,

Recently several Yocto Project members have expressed interest in being able
to upgrade the Linux kernel in Dunfell LTS release from 5.4 to 5.10. Both of
those Linux kernel versions are designated "longterm"[1], but at the time
Dunfell was released in April 2020, 5.10 was not out yet.

Recommended way to add new features or upgrade components in Yocto LTS
releases is with the so called "mixin" layers[2]. After discussing some of
the details of such layer among the Yocto Project TSC and other interested
developers, as well as testing the concept for few weeks, I'd like to make
this open to a wider consumption in hopes it would be useful to others.

This Linux kernel "mixin" layer is currently hosted on GitHub under my
name[3], but can be moved to a more public space, like Yocto Project git
server, if it gets traction and is found useful to others. Please feel free
to try it and provide feedback. Thanks!

[1] https://kernel.org/
[2] https://wiki.yoctoproject.org/wiki/Stable_Release_and_LTS#LTS_.E2.80.9CMixin.E2.80.9D_repositories
[3] https://github.com/denix0/meta-lts-kernel-mixin

All,

Recently several Yocto Project members have expressed interest in being able
to upgrade the Linux kernel in Dunfell LTS release from 5.4 to 5.10. Both of
those Linux kernel versions are designated "longterm"[1], but at the time
Dunfell was released in April 2020, 5.10 was not out yet.

Recommended way to add new features or upgrade components in Yocto LTS
releases is with the so called "mixin" layers[2]. After discussing some of
the details of such layer among the Yocto Project TSC and other interested
developers, as well as testing the concept for few weeks, I'd like to make
this open to a wider consumption in hopes it would be useful to others.

This Linux kernel "mixin" layer is currently hosted on GitHub under my
name[3], but can be moved to a more public space, like Yocto Project git
server, if it gets traction and is found useful to others. Please feel free
to try it and provide feedback. Thanks!

[1] https://kernel.org/
[2] https://wiki.yoctoproject.org/wiki/Stable_Release_and_LTS#LTS_.E2.80.9CMixin.E2.80.9D_repositories
[3] https://github.com/denix0/meta-lts-kernel-mixin

Just to update, I have an experiment:

http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=rpurdie/overrides-convert

This contains a conversion script which can convert poky to use ":" 
instead of "_" as the override character (i.e. all of oe-core, meta-yocto, 
docs and bitbake inc. tests). I had to add in only two special hand cases
to the conversion code which is better than I expected (so far).

There is also a quick hack to make bitbake use the new override format 
(but no optimisation to the code) and a commit showing the result of the 
conversion.

Right now, this will successfully "bitbake -p", i.e. it parses. It will
even run a build but crashes out in gcc-cross do_compile.

On Fri, 2021-07-16 at 11:44 -0500, Mark Hatle wrote:

On 7/16/21 4:22 AM, Richard Purdie wrote:

On Thu, 2021-07-15 at 09:26 -0500, Mark Hatle wrote:

On 7/15/21 8:56 AM, Richard Purdie wrote:

Breaking things down a bit, one thing I keep running into with our current 
codebase and metadata is that overrides are not clear. In my previous email,
the example was:

do_configure_class-native

A human can usually spot "class-native" is and "configure" or 
"configure_class-native" is not. Bitbake's parser struggles. It has huge 
internal lists including variables like x86_64 where it has to track 
whether "64" in an override.

One way of fixing this is to be explicit about overrides and use a different 
separator. See an example patch below I made to the quilt recipe using ":" 
instead to see how it looks. Personally, I think this looks like an improvement.

There are two challenges:

a) The work of migration. Do we migrate piecemeal or with a flag day? I'm not
   sure piecemeal is even possible unfortunately.

b) Some of the packaging code (at the very least) needs rewriting as it accesses
   both RDEPENDS_${PN} and puts ${PN} in OVERRIDES and accesses RDEPENDS. I'm not
   sure what else may be assuming this works.

Ya, this is all over the packaging code. Should be trivial to move to treating
it as a variable (and not override). I can help with this if you want.

I've been pondering this a bit more and my preference is to move this to use 
overrides more explicitly.

The part of the code I worked on, the override was used as an optimization to avoid:

d.getVar("RDEPENDS_%s" % package) or d.getVar("RDEPENDS")

(and similar for all of the other variables that could be package specific.)

Overrides (at least to me) are system (or at least recipe) wide settings that
allow you to 'override' key things. The package items, based on that definitely
really aren't overrides -- they're package specific variables. (This is also
why I suggested RDEPENDS[package] might actually be a better syntax.)

Overrides are one of the key things which make OE function as an architecture and 
how a lot of the "magic" in our metadata happens. There is a hierarchy of overrides,
some are system wide some are local to specific functional areas but using them for
packaging is a natural fit.

As such, I'm very strongly in favour of explicitly exposing that rather than hiding
it.

FWIW the image class code also does something similar for IMAGE_CMD and image types.

I'm all for making override usage more clear to people. But in this ONE case, I
don't think these are really overrides -- and worse they could cause confusion
because they are ONLY overrides when packaging. So if you use the override
chunk in something that doesn't affect a package specific setting it won't do
anything. It also won't do anything on the functions (I think), as those are
run before the override is loaded.

Specific code I'm talking about BTW is:

http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/classes/package_rpm.bbclass

starting at line 287. (srcname is PN-src BTW)

then the loop starting at 328. Iterates over PACKAGES and sets the override one
at a time to the package name to get (conffiles, dirnames, summary/description,
pkgv, pkgr, pkge, license, section, description, rdepends, rrecommends,
rsuggests, rprovides, rreplaces, rconflicts, and I suspect a few others...)

Note, I could make the above changes pretty quickly and just submit this as an
implementation change in a few hours. I'd be happy to do that and look for
other 'in-place' usages of d.setVar("OVERRIDE"... to see if there are other
things like this.

As above, I'm very much of the opposite view, I think we want to promote this,
not remove it. Yes, you can emulate overrides with other forms of variable access
but that really isn't how the system was designed to be used.

Cheers,

Richard

On 7/16/21 4:22 AM, Richard Purdie wrote:

On Thu, 2021-07-15 at 09:26 -0500, Mark Hatle wrote:

On 7/15/21 8:56 AM, Richard Purdie wrote:

Breaking things down a bit, one thing I keep running into with our current 
codebase and metadata is that overrides are not clear. In my previous email,
the example was:

do_configure_class-native

A human can usually spot "class-native" is and "configure" or 
"configure_class-native" is not. Bitbake's parser struggles. It has huge 
internal lists including variables like x86_64 where it has to track 
whether "64" in an override.

One way of fixing this is to be explicit about overrides and use a different 
separator. See an example patch below I made to the quilt recipe using ":" 
instead to see how it looks. Personally, I think this looks like an improvement.

There are two challenges:

a) The work of migration. Do we migrate piecemeal or with a flag day? I'm not
   sure piecemeal is even possible unfortunately.

b) Some of the packaging code (at the very least) needs rewriting as it accesses
   both RDEPENDS_${PN} and puts ${PN} in OVERRIDES and accesses RDEPENDS. I'm not
   sure what else may be assuming this works.

Ya, this is all over the packaging code. Should be trivial to move to treating
it as a variable (and not override). I can help with this if you want.

I've been pondering this a bit more and my preference is to move this to use 
overrides more explicitly.

The part of the code I worked on, the override was used as an optimization to avoid:

d.getVar("RDEPENDS_%s" % package) or d.getVar("RDEPENDS")

(and similar for all of the other variables that could be package specific.)

Overrides (at least to me) are system (or at least recipe) wide settings that
allow you to 'override' key things. The package items, based on that definitely
really aren't overrides -- they're package specific variables. (This is also
why I suggested RDEPENDS[package] might actually be a better syntax.)

I'm all for making override usage more clear to people. But in this ONE case, I
don't think these are really overrides -- and worse they could cause confusion
because they are ONLY overrides when packaging. So if you use the override
chunk in something that doesn't affect a package specific setting it won't do
anything. It also won't do anything on the functions (I think), as those are
run before the override is loaded.

Specific code I'm talking about BTW is:

http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/classes/package_rpm.bbclass

starting at line 287. (srcname is PN-src BTW)

then the loop starting at 328. Iterates over PACKAGES and sets the override one
at a time to the package name to get (conffiles, dirnames, summary/description,
pkgv, pkgr, pkge, license, section, description, rdepends, rrecommends,
rsuggests, rprovides, rreplaces, rconflicts, and I suspect a few others...)

Note, I could make the above changes pretty quickly and just submit this as an
implementation change in a few hours. I'd be happy to do that and look for
other 'in-place' usages of d.setVar("OVERRIDE"... to see if there are other
things like this.

On Fri, 2021-07-16 at 14:35 +0000, Peter Kjellerstedt wrote:

As to RDEPENDS_${PN} & co, wouldn't it make sense to use RDEPENDS_pn-${PN},
since "pn-<package name>" already exists as an override? 

PN is really badly named since it is effectively the recipe name in many 
contexts. We're specifying one single thing in overrides with it rather than
a suite of packages a recipe generates. I can't see how using pn- like that
would work for the use cases we have for the package name override.

On that note,
I would like to see overrides being prefixed more (maybe even as a
requirement). Some suggestions:

pn- - Package name
df- - Distro feature
mf- - Machine feature
pf- - Package feature (can we rename/alias PACKAGECONFIG to
          PACKAGE_FEATURES while at it, for consistency?)
arch- - Architecture
dist- - Distribution

That way it would be more obvious what an override does. Especially the
use of the distro name as an override can have some unforeseen consequences
if the name is somewhat common).

We have tried to do this with newer overrides where it makes sense. It is 
a tradeoff between longer names and usability. I don't think adding something
for the package name would help. Having "distro-" and "machine-" could
help in some cases but it comes at the price of more complicated conversions
and it becomes a question of whether things become more readable. Worth thinking
about but the above list doesn't really make sense since features are in overrides.

(PACKAGECONFIG should really be RECIPE_FEATURES if we're going to talk names!)

Cheers,

Richard

On Thu, 2021-07-15 at 09:26 -0500, Mark Hatle wrote:

On 7/15/21 8:56 AM, Richard Purdie wrote:

Breaking things down a bit, one thing I keep running into with our current 
codebase and metadata is that overrides are not clear. In my previous email,
the example was:

do_configure_class-native

A human can usually spot "class-native" is and "configure" or 
"configure_class-native" is not. Bitbake's parser struggles. It has huge 
internal lists including variables like x86_64 where it has to track 
whether "64" in an override.

One way of fixing this is to be explicit about overrides and use a different 
separator. See an example patch below I made to the quilt recipe using ":" 
instead to see how it looks. Personally, I think this looks like an improvement.

There are two challenges:

a) The work of migration. Do we migrate piecemeal or with a flag day? I'm not
   sure piecemeal is even possible unfortunately.

b) Some of the packaging code (at the very least) needs rewriting as it accesses
   both RDEPENDS_${PN} and puts ${PN} in OVERRIDES and accesses RDEPENDS. I'm not
   sure what else may be assuming this works.

Ya, this is all over the packaging code. Should be trivial to move to treating
it as a variable (and not override). I can help with this if you want.

I've been pondering this a bit more and my preference is to move this to use 
overrides more explicitly.

What I did realise is that d.getVar("RDEPENDS:${PN}") can actually split out
the override itself and apply it so I think the existing code can be made to
work just with the character change if we teach getVar how to handle it which
should be possible.

(It can of course also be moved to overrides, but that won't be as obvious to a
recipe creator, since they're not used to the package name being an 'override'.)

Whilst the instinct is to "hide" that, perhaps it could also make overrides more
obvious since people could see them in a different setting more clearly? I think
that is where I'm leaning right now.

This change does buy us cleaner looking metadata and ultimately, a faster
and cleaner internal bitbake that at least would use less memory. It could set
the stage to allow the defval idea I mentioned in a previous mail to happen.

It is a huge change and would need a lot of work to make it happen. Is it worth 
doing? Not sure. I'm putting it out there for discussion.

Is this something we do in parallel with master, and then flag day it? I could
easily see this work take a few weeks (if a bunch of people help).

I think we might be able to block convert much of the metadata with some kind of 
script. If that is possible, we'd have a flag day and a bunch of code tested in 
parallel before a final semi-automated conversion.

It is also going to be tempting that "if we're breaking things, lets break lots".
I want to be mindful that tempting as that may be, if users can't convert clearly,
it will be worlds of pain. The benefit of this change is that at least in the
general case, the transition should be clear. Taking steps rather than changing the
world may therefore be a better idea...

One of those things (variable syntax) I'm not sure 'steps' make sense. If we're
changing a fundamental syntax, it really becomes a new major version of bitbake
and thus a new version of OE.

Having lots of things change at once may make things much harder for layer maintainers
than having a controlled change to handle though?

 EXTRA_OECONF = "--with-perl='${USRBINPATH}/env perl' --with-patch=patch"
-EXTRA_OECONF_append_class-native = " --disable-nls"
+EXTRA_OECONF:append:class-native = " --disable-nls"

So operations (append) and overrides would both be delimited by ':'? It keeps
the ordering and makes it easier to read for sure, but I'm wondering if we need
to deliminate the operation differently. (No I have no suggestions, just a
thought. I don't object to the proposal at all.)

I wondered but this kind of felt neatest to me and it does work nicely with
the code backend handling it...

-RDEPENDS_${PN}-ptest = "make file sed gawk diffutils findutils ed perl \
+RDEPENDS:${PN}-ptest = "make file sed gawk diffutils findutils ed perl \
                         perl-module-filehandle perl-module-getopt-std \
                         perl-module-posix perl-module-file-temp \
                         perl-module-text-parsewords perl-module-overloading \

The above would work without code changes to the packaging class, but maybe we
really should make the change for readability. (I've often wondered if it
really should use 'flag' syntax instead, but lack of overrides make flag syntax
difficult to use.)

i.e.:

-FILES_${PN} = "${sysconfdir} ${datadir}/quilt \
+FILES[${PN}] = "${sysconfdir} ${datadir}/quilt \

or maybe something like? (This is a larger syntax change though)

-RDEPENDS_${PN} = "bash patch diffstat bzip2 util-linux less"
-RDEPENDS_${PN}_class-native = "diffstat-native patch-native bzip2-native"
+RDEPENDS[${PN}] = "bash patch diffstat bzip2 util-linux less"
+RDEPENDS[${PN}]:class-native = "diffstat-native patch-native bzip2-native"

'flags' in bitbake are really a poor cousin to the main variables and behave 
quite differently. Whether flags should have override support is a different 
question. It would hugely complicate the datastore code and I'm not sure the
above is more readable compared to the simpler ":". I've always seen flags as
supplemental data for a variable rather than anything like the above.

Cheers,

Richard

This is a long answer, and responding to multiple subjects that have been
brought up in the different threads, so please bear with me.

I would love to see improvements to the bitbake syntax when it comes to
overrides. Unfortunately, I do not see how I would be able to ever update
to said version if it is done as a flag day release. The way we work is
to have our layers compatible with the version of bitbake we are using
(currently Hardknott). Then I (who basically do all work to build our
layers with master of Poky) have a special adaptation layer where I add
bbappends and some occasional backported bbclass to make our layers work
with Poky master. This work runs in parallel with the normal recipe updates
that the developers do so that when a new major release of Poky is ready,
so are we. However, if we would have to use a whole different version of
all our recipes, there is no way this process would work, and I can't stop
the normal development for a flag day upgrade of Poky. :(

The way I see it can be done would involve supporting both syntaxes in
one LTS-release. In the next non-LTS release, the support for the old
syntax can be dropped. That way there is always an intermediate release
that one can update to, update one's own recipes and then proceed to the
latest release. This of course means that the Yocto Project cannot reap
the benefits of the new and improved syntax immediately, but would have
to hold off for one release. On the other hand, this would mean the
transitioning path for all users of the Yocto Project would be trivial
(for some definition of trivial), compared to a major PITA.

Now, I don't know if supporting both syntaxes at once is feasible given
some of the comments given in the thread. However, most of them seem
related to being able to drop support for the old syntax and thereby
gaining the expected improvements to the internal data, which would not
be a goal for the release that actually has to support both syntaxes.
Its goal is compatibility and a way forward.

As to the actual syntax changes, one thing I would like to se is the
_append/_prepend/_remove operators actually turn into real operators.
I definitely do not want to see the same syntax for the operators as
for overrides (that is one of the most confusing things with the
current solution).

Here is one suggestion:

PACKAGECONFIG_append = "foo" => PACKAGECONFIG{} .= "foo"
PACKAGECONFIG_append = " foo" => PACKAGECONFIG{} += "foo"
PACKAGECONFIG_prepend = "foo " => PACKAGECONFIG{} =. "foo"
PACKAGECONFIG_prepend = " foo " => PACKAGECONFIG{} =+ "foo"
PACKAGECONFIG_remove = "foo " => PACKAGECONFIG{} -= "foo"

PACKAGECONFIG_append_bar = "foo" => PACKAGECONFIG{bar} .= "foo"
PACKAGECONFIG_append_bar = " foo" => PACKAGECONFIG{bar} += "foo"
PACKAGECONFIG_prepend_bar = "foo " => PACKAGECONFIG{bar} =. "foo"
PACKAGECONFIG_prepend_bar = " foo " => PACKAGECONFIG{bar} =+ "foo"
PACKAGECONFIG_remove_bar = "foo " => PACKAGECONFIG{bar} -= "foo"

PACKAGECONFIG_append_bar_foo = "foo" => PACKAGECONFIG{bar&foo} .= "foo"
PACKAGECONFIG_append_bar_foo = " foo" => PACKAGECONFIG{bar&foo} += "foo"
PACKAGECONFIG_prepend_bar_foo = "foo " => PACKAGECONFIG{bar&foo} =. "foo"
PACKAGECONFIG_prepend_bar_foo = " foo " => PACKAGECONFIG{bar&foo} =+ "foo"
PACKAGECONFIG_remove_bar_foo = "foo " => PACKAGECONFIG{bar&foo} -= "foo"

This would not have a corresponding solution for, e.g.,
PACKAGECONFIG_bar_foo_append, but I actually think that is good.

This would also allow for clearing out the pending appends for a
given override by doing:

PACKAGECONFIG{} = ""

or:

PACKAGECONFIG{bar} = ""

And a totally different solution (and probably more controversial) would
be to add a more programmatical way to handle overrides, e.g.:

PACKAGECONFIG += "foo" if overrides("foo bar")

or:

if override("foo") && override("bar"): PACKAGECONFIG += "foo"

It doesn't help with whether the update to the variable is immediate or
deferred though.

As to RDEPENDS_${PN} & co, wouldn't it make sense to use RDEPENDS_pn-${PN},
since "pn-<package name>" already exists as an override? 

On that note,
I would like to see overrides being prefixed more (maybe even as a
requirement). Some suggestions:

pn- - Package name
df- - Distro feature
mf- - Machine feature
pf- - Package feature (can we rename/alias PACKAGECONFIG to
          PACKAGE_FEATURES while at it, for consistency?)
arch- - Architecture
dist- - Distribution

That way it would be more obvious what an override does. Especially the
use of the distro name as an override can have some unforeseen consequences
if the name is somewhat common).