Date   

Linux 5.10 LTS "mixin" layer for Dunfell

Denys Dmytriyenko
 

All,

Recently several Yocto Project members have expressed interest in being able
to upgrade the Linux kernel in Dunfell LTS release from 5.4 to 5.10. Both of
those Linux kernel versions are designated "longterm"[1], but at the time
Dunfell was released in April 2020, 5.10 was not out yet.

Recommended way to add new features or upgrade components in Yocto LTS
releases is with the so called "mixin" layers[2]. After discussing some of
the details of such layer among the Yocto Project TSC and other interested
developers, as well as testing the concept for few weeks, I'd like to make
this open to a wider consumption in hopes it would be useful to others.

This Linux kernel "mixin" layer is currently hosted on GitHub under my
name[3], but can be moved to a more public space, like Yocto Project git
server, if it gets traction and is found useful to others. Please feel free
to try it and provide feedback. Thanks!

[1] https://kernel.org/
[2] https://wiki.yoctoproject.org/wiki/Stable_Release_and_LTS#LTS_.E2.80.9CMixin.E2.80.9D_repositories
[3] https://github.com/denix0/meta-lts-kernel-mixin

--
Regards,
Denys Dmytriyenko <denis@...>
PGP: 0x420902729A92C964 - https://denix.org/0x420902729A92C964
Fingerprint: 25FC E4A5 8A72 2F69 1186 6D76 4209 0272 9A92 C964


Override syntax change proposal

Richard Purdie
 

Hi All,

I now have an actual proposed plan for changing override syntax which I hope
will mitigate the issues raised in the previous discussion as far a is possible
whilst still letting the project move forward.

Firstly, I'd like to merge a patch I sent to the bitbake list:

https://lists.openembedded.org/g/bitbake-devel/topic/patch_data_smart_parse/84285770

This patch allows bitbake to accept ":" in variable/function names. It is designed
to be simple and could be easily backported to previous bitbake versions. At the
very least I'd like to do that for master, allowing at least OE-Core master branch 
to start using an explicit override syntax.

I'd like to go further and suggest we backport to older bitbake versions (back to 
dunfell?) so that anyone using metadata over multiple versions could continue to
do so.

I'm hoping the above isn't too controversial and would at least let us move forward
to trying to improve.

The next steps beyond that are a little trickier as they start to require the change.
The next logical move is to require ":" as the separator for current master branches.
I believe it is reasonable to do this if we can show how layers can be converted and 
have a way for older layers to continue to work with master.

There is a script I wrote and linked to (~100 lines of python) which with some
priming about the overrides in use in a layer (which a layer maintainer should know), 
is able to automatically convert a layer. For oe-core+meta-yocto+docs+bitbake, that
is 9000+ changes but the script seems reliable if not very nice code. I did end up
adding a handful of manual fixes but that was for python code manipulating variables
which most layers wouldn't have, in general the core is way more complex than most
layers.

This means that people with older layers trying to work with master now have two
options:

a) they convert to use ":" and update to a version of bitbake with the above tweak
to allow it to accept that (no behaviour change)
b) they dynamically translate their layer with a script when testing with master

Given those options two options both being shown to be viable, I think this
is enough to allow us to proceed and update the syntax on shorter timescales such
as during this release, ready for the next LTS.

In working out compatibility pieces, I don't have any optimisations using the
new syntax yet, or a plan worked out for what we can do beyond this other that ideas
and my previous experiments. Some changes "feel" right and this one definitely does 
as a good move in it's own right regardless of whether we ultimately can improve 
syntax or not. I'm therefore keen to proceed.

Cheers.

Richard


Re: Should we change variable override formatting?

Richard Purdie
 

On Fri, 2021-07-16 at 17:46 +0100, Richard Purdie via lists.openembedded.org wrote:
Just to update, I have an experiment:

http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=rpurdie/overrides-convert

This contains a conversion script which can convert poky to use ":" 
instead of "_" as the override character (i.e. all of oe-core, meta-yocto, 
docs and bitbake inc. tests). I had to add in only two special hand cases
to the conversion code which is better than I expected (so far).

There is also a quick hack to make bitbake use the new override format 
(but no optimisation to the code) and a commit showing the result of the 
conversion.

Right now, this will successfully "bitbake -p", i.e. it parses. It will
even run a build but crashes out in gcc-cross do_compile.
Branch updated, it now lets a core-image-sato build successfully.

Cheers,

Richard


Re: Should we change variable override formatting?

Mark Hatle
 

On 7/16/21 11:56 AM, Richard Purdie wrote:
On Fri, 2021-07-16 at 11:44 -0500, Mark Hatle wrote:

On 7/16/21 4:22 AM, Richard Purdie wrote:
On Thu, 2021-07-15 at 09:26 -0500, Mark Hatle wrote:

On 7/15/21 8:56 AM, Richard Purdie wrote:
Breaking things down a bit, one thing I keep running into with our current 
codebase and metadata is that overrides are not clear. In my previous email,
the example was:

do_configure_class-native

A human can usually spot "class-native" is and "configure" or 
"configure_class-native" is not. Bitbake's parser struggles. It has huge 
internal lists including variables like x86_64 where it has to track 
whether "64" in an override.

One way of fixing this is to be explicit about overrides and use a different 
separator. See an example patch below I made to the quilt recipe using ":" 
instead to see how it looks. Personally, I think this looks like an improvement.

There are two challenges:

a) The work of migration. Do we migrate piecemeal or with a flag day? I'm not
   sure piecemeal is even possible unfortunately.

b) Some of the packaging code (at the very least) needs rewriting as it accesses
   both RDEPENDS_${PN} and puts ${PN} in OVERRIDES and accesses RDEPENDS. I'm not
   sure what else may be assuming this works.
Ya, this is all over the packaging code. Should be trivial to move to treating
it as a variable (and not override). I can help with this if you want.
I've been pondering this a bit more and my preference is to move this to use 
overrides more explicitly.
The part of the code I worked on, the override was used as an optimization to avoid:

d.getVar("RDEPENDS_%s" % package) or d.getVar("RDEPENDS")

(and similar for all of the other variables that could be package specific.)

Overrides (at least to me) are system (or at least recipe) wide settings that
allow you to 'override' key things. The package items, based on that definitely
really aren't overrides -- they're package specific variables. (This is also
why I suggested RDEPENDS[package] might actually be a better syntax.)
Overrides are one of the key things which make OE function as an architecture and 
how a lot of the "magic" in our metadata happens. There is a hierarchy of overrides,
some are system wide some are local to specific functional areas but using them for
packaging is a natural fit.

As such, I'm very strongly in favour of explicitly exposing that rather than hiding
it.

FWIW the image class code also does something similar for IMAGE_CMD and image types.

I'm all for making override usage more clear to people. But in this ONE case, I
don't think these are really overrides -- and worse they could cause confusion
because they are ONLY overrides when packaging. So if you use the override
chunk in something that doesn't affect a package specific setting it won't do
anything. It also won't do anything on the functions (I think), as those are
run before the override is loaded.

Specific code I'm talking about BTW is:

http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/classes/package_rpm.bbclass

starting at line 287. (srcname is PN-src BTW)

then the loop starting at 328. Iterates over PACKAGES and sets the override one
at a time to the package name to get (conffiles, dirnames, summary/description,
pkgv, pkgr, pkge, license, section, description, rdepends, rrecommends,
rsuggests, rprovides, rreplaces, rconflicts, and I suspect a few others...)

Note, I could make the above changes pretty quickly and just submit this as an
implementation change in a few hours. I'd be happy to do that and look for
other 'in-place' usages of d.setVar("OVERRIDE"... to see if there are other
things like this.
As above, I'm very much of the opposite view, I think we want to promote this,
not remove it. Yes, you can emulate overrides with other forms of variable access
but that really isn't how the system was designed to be used.
The view I've also had and used to explain to others. There are two scopes of
variables in OE. global and recipe specific. Adding an override anywhere else,
really adds additional (hidden) scopes to the system.

I guess this metal model of scopes is influenced by 'bitbake -e'. I can run it
against the global system, or a specific recipe. But there is no way for me to
run it on a 'package' scope.

Additional mental issue I have with PACKAGE being a scope, is the limited nature
of the scope. The majority of variables in the system are not modified or used
when in this scope, only a VERY limited amount.. (Now this limited amount can
reference variables in the recipe scope.. But would you ever do:

PACKAGES = "${PN} ${PN}-dev ${PN}-src"

FILES_${PN} = "/foo/${MYFILE}"

MYFILE = "bar"
MYFILE_${PN} = "foo"

Current implementation, /foo/foo would be selected, only during package
processing -- but if you use bitbake -e recipe you can /foo/bar...


So back to what I offered before, from a quicker survey of poky the following
are the only non-global/recipe specific scopes that I was able to find.

'PACKAGE' specific overrides:
package_*.bbclass
update-rc.d.bbclass
insane.bbclass
buildhistory.bbclass

The variables allowed to work in the current form of VAR_pkg is very specific
and tailored.


uboot/syslinux/grub have a concept of 'labels', they each implement it slightly
differently but it's the same idea:
uboot-extlinux-config.bbclass
syslinux.bbclass
grub-efi-cfg.bbclass

Again, the list appears to be very specific and minimal from what I could tell.
Unlike the package side, I don't have a lot of experience here. (Variables are
specific to each of those three classes though, but still a limited set.)


'debugfs' usage for images. This appears to be how the debugfs stuff is
implemented:
image.bbclass

I'm not completely sure the range of variables used here. But again it looks
somewhat limited in scope... but this scope could quickly expand if you end up
nesting things variables, but I'm unsure that it would work based on the manual
processing of vardeps that I see.

Cheers,

Richard


Re: Should we change variable override formatting?

Richard Purdie
 

On Fri, 2021-07-16 at 11:44 -0500, Mark Hatle wrote:

On 7/16/21 4:22 AM, Richard Purdie wrote:
On Thu, 2021-07-15 at 09:26 -0500, Mark Hatle wrote:

On 7/15/21 8:56 AM, Richard Purdie wrote:
Breaking things down a bit, one thing I keep running into with our current 
codebase and metadata is that overrides are not clear. In my previous email,
the example was:

do_configure_class-native

A human can usually spot "class-native" is and "configure" or 
"configure_class-native" is not. Bitbake's parser struggles. It has huge 
internal lists including variables like x86_64 where it has to track 
whether "64" in an override.

One way of fixing this is to be explicit about overrides and use a different 
separator. See an example patch below I made to the quilt recipe using ":" 
instead to see how it looks. Personally, I think this looks like an improvement.

There are two challenges:

a) The work of migration. Do we migrate piecemeal or with a flag day? I'm not
   sure piecemeal is even possible unfortunately.

b) Some of the packaging code (at the very least) needs rewriting as it accesses
   both RDEPENDS_${PN} and puts ${PN} in OVERRIDES and accesses RDEPENDS. I'm not
   sure what else may be assuming this works.
Ya, this is all over the packaging code. Should be trivial to move to treating
it as a variable (and not override). I can help with this if you want.
I've been pondering this a bit more and my preference is to move this to use 
overrides more explicitly.
The part of the code I worked on, the override was used as an optimization to avoid:

d.getVar("RDEPENDS_%s" % package) or d.getVar("RDEPENDS")

(and similar for all of the other variables that could be package specific.)

Overrides (at least to me) are system (or at least recipe) wide settings that
allow you to 'override' key things. The package items, based on that definitely
really aren't overrides -- they're package specific variables. (This is also
why I suggested RDEPENDS[package] might actually be a better syntax.)
Overrides are one of the key things which make OE function as an architecture and 
how a lot of the "magic" in our metadata happens. There is a hierarchy of overrides,
some are system wide some are local to specific functional areas but using them for
packaging is a natural fit.

As such, I'm very strongly in favour of explicitly exposing that rather than hiding
it.

FWIW the image class code also does something similar for IMAGE_CMD and image types.

I'm all for making override usage more clear to people. But in this ONE case, I
don't think these are really overrides -- and worse they could cause confusion
because they are ONLY overrides when packaging. So if you use the override
chunk in something that doesn't affect a package specific setting it won't do
anything. It also won't do anything on the functions (I think), as those are
run before the override is loaded.

Specific code I'm talking about BTW is:

http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/classes/package_rpm.bbclass

starting at line 287. (srcname is PN-src BTW)

then the loop starting at 328. Iterates over PACKAGES and sets the override one
at a time to the package name to get (conffiles, dirnames, summary/description,
pkgv, pkgr, pkge, license, section, description, rdepends, rrecommends,
rsuggests, rprovides, rreplaces, rconflicts, and I suspect a few others...)

Note, I could make the above changes pretty quickly and just submit this as an
implementation change in a few hours. I'd be happy to do that and look for
other 'in-place' usages of d.setVar("OVERRIDE"... to see if there are other
things like this.
As above, I'm very much of the opposite view, I think we want to promote this,
not remove it. Yes, you can emulate overrides with other forms of variable access
but that really isn't how the system was designed to be used.

Cheers,

Richard


Re: Should we change variable override formatting?

Mark Hatle
 

On 7/16/21 11:28 AM, Richard Purdie wrote:
On Fri, 2021-07-16 at 14:35 +0000, Peter Kjellerstedt wrote:
(trimmed to add a minor comment)

As to RDEPENDS_${PN} & co, wouldn't it make sense to use RDEPENDS_pn-${PN},
since "pn-<package name>" already exists as an override? 
PN is really badly named since it is effectively the recipe name in many 
contexts. We're specifying one single thing in overrides with it rather than
a suite of packages a recipe generates. I can't see how using pn- like that
would work for the use cases we have for the package name override.
In this case ${PN} is just a name. It's because PACKAGES contains ${PN} by default.

Really it's RDEPENDS_<package>. Which is distinctly different then
RDEPENDS_pn-${PN} which is 'RDEPENDS' when evaluated.

I know minor distinction, but this is why I don't advocate treating RDEPENDS
(and other package specific variables) as overrides, but instead think of this
as RDEPENDS_<package>_<operation/override>...

So with proposed syntax becomes RDEPENDS_<package>:operation/override

On that note,
I would like to see overrides being prefixed more (maybe even as a
requirement). Some suggestions:

pn- - Package name
df- - Distro feature
mf- - Machine feature
pf- - Package feature (can we rename/alias PACKAGECONFIG to
          PACKAGE_FEATURES while at it, for consistency?)
arch- - Architecture
dist- - Distribution

That way it would be more obvious what an override does. Especially the
use of the distro name as an override can have some unforeseen consequences
if the name is somewhat common).
We have tried to do this with newer overrides where it makes sense. It is 
a tradeoff between longer names and usability. I don't think adding something
for the package name would help. Having "distro-" and "machine-" could
help in some cases but it comes at the price of more complicated conversions
and it becomes a question of whether things become more readable. Worth thinking
about but the above list doesn't really make sense since features are in overrides.
I've definitely run into problems where distro name or machine name collided
with a different override in the past and took a long time for me to realize the
clash was happening and thus the wrong variables were being evaluated.

--Mark

(PACKAGECONFIG should really be RECIPE_FEATURES if we're going to talk names!)

Cheers,

Richard








Re: Should we change variable override formatting?

Richard Purdie
 

Just to update, I have an experiment:

http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=rpurdie/overrides-convert

This contains a conversion script which can convert poky to use ":" 
instead of "_" as the override character (i.e. all of oe-core, meta-yocto, 
docs and bitbake inc. tests). I had to add in only two special hand cases
to the conversion code which is better than I expected (so far).

There is also a quick hack to make bitbake use the new override format 
(but no optimisation to the code) and a commit showing the result of the 
conversion.

Right now, this will successfully "bitbake -p", i.e. it parses. It will
even run a build but crashes out in gcc-cross do_compile.

You can see from the diff it generates, it isn't perfect and there are 
issues. At this point I should probably step back and finish the "diff"
tool I started a while ago so that we can compare to different builds,
see if (well, where!) the resulting output as parsed is different.

The point of this is to:

a) see if automated conversion is viable
b) see if the proposed syntax looks/feels better
c) allow experiments on things like performance to be made if I can
figure out where the differences are in the conversion and get builds working
d) if it can work, we could experiment further and see if there are follow
up improvements that can be based on this such as my previous experiments
with defval

So this is known to be rather rough and ready, it is very much experimental 
and brainstorming but probably worth sharing for anyone wanting to experiment.

Cheers,

Richard


Re: Should we change variable override formatting?

Mark Hatle
 

On 7/16/21 4:22 AM, Richard Purdie wrote:
On Thu, 2021-07-15 at 09:26 -0500, Mark Hatle wrote:

On 7/15/21 8:56 AM, Richard Purdie wrote:
Breaking things down a bit, one thing I keep running into with our current 
codebase and metadata is that overrides are not clear. In my previous email,
the example was:

do_configure_class-native

A human can usually spot "class-native" is and "configure" or 
"configure_class-native" is not. Bitbake's parser struggles. It has huge 
internal lists including variables like x86_64 where it has to track 
whether "64" in an override.

One way of fixing this is to be explicit about overrides and use a different 
separator. See an example patch below I made to the quilt recipe using ":" 
instead to see how it looks. Personally, I think this looks like an improvement.

There are two challenges:

a) The work of migration. Do we migrate piecemeal or with a flag day? I'm not
   sure piecemeal is even possible unfortunately.

b) Some of the packaging code (at the very least) needs rewriting as it accesses
   both RDEPENDS_${PN} and puts ${PN} in OVERRIDES and accesses RDEPENDS. I'm not
   sure what else may be assuming this works.
Ya, this is all over the packaging code. Should be trivial to move to treating
it as a variable (and not override). I can help with this if you want.
I've been pondering this a bit more and my preference is to move this to use 
overrides more explicitly.
The part of the code I worked on, the override was used as an optimization to avoid:

d.getVar("RDEPENDS_%s" % package) or d.getVar("RDEPENDS")

(and similar for all of the other variables that could be package specific.)

Overrides (at least to me) are system (or at least recipe) wide settings that
allow you to 'override' key things. The package items, based on that definitely
really aren't overrides -- they're package specific variables. (This is also
why I suggested RDEPENDS[package] might actually be a better syntax.)

What I did realise is that d.getVar("RDEPENDS:${PN}") can actually split out
the override itself and apply it so I think the existing code can be made to
work just with the character change if we teach getVar how to handle it which
should be possible.

(It can of course also be moved to overrides, but that won't be as obvious to a
recipe creator, since they're not used to the package name being an 'override'.)
Whilst the instinct is to "hide" that, perhaps it could also make overrides more
obvious since people could see them in a different setting more clearly? I think
that is where I'm leaning right now.
I'm all for making override usage more clear to people. But in this ONE case, I
don't think these are really overrides -- and worse they could cause confusion
because they are ONLY overrides when packaging. So if you use the override
chunk in something that doesn't affect a package specific setting it won't do
anything. It also won't do anything on the functions (I think), as those are
run before the override is loaded.

Specific code I'm talking about BTW is:

http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/classes/package_rpm.bbclass

starting at line 287. (srcname is PN-src BTW)

then the loop starting at 328. Iterates over PACKAGES and sets the override one
at a time to the package name to get (conffiles, dirnames, summary/description,
pkgv, pkgr, pkge, license, section, description, rdepends, rrecommends,
rsuggests, rprovides, rreplaces, rconflicts, and I suspect a few others...)

Note, I could make the above changes pretty quickly and just submit this as an
implementation change in a few hours. I'd be happy to do that and look for
other 'in-place' usages of d.setVar("OVERRIDE"... to see if there are other
things like this.

This change does buy us cleaner looking metadata and ultimately, a faster
and cleaner internal bitbake that at least would use less memory. It could set
the stage to allow the defval idea I mentioned in a previous mail to happen.

It is a huge change and would need a lot of work to make it happen. Is it worth 
doing? Not sure. I'm putting it out there for discussion.
Is this something we do in parallel with master, and then flag day it? I could
easily see this work take a few weeks (if a bunch of people help).
I think we might be able to block convert much of the metadata with some kind of 
script. If that is possible, we'd have a flag day and a bunch of code tested in 
parallel before a final semi-automated conversion.
Agreed. I'd expect a 90+% success rate on a conversion.

It is also going to be tempting that "if we're breaking things, lets break lots".
I want to be mindful that tempting as that may be, if users can't convert clearly,
it will be worlds of pain. The benefit of this change is that at least in the
general case, the transition should be clear. Taking steps rather than changing the
world may therefore be a better idea...
One of those things (variable syntax) I'm not sure 'steps' make sense. If we're
changing a fundamental syntax, it really becomes a new major version of bitbake
and thus a new version of OE.
Having lots of things change at once may make things much harder for layer maintainers
than having a controlled change to handle though?
(been reading the other replies)

I don't have a great answer to this except for personal experience. I greatly
prefer [when things pass a compatibility point] to maintain multiple layers. If
we make the conversion script? available to layer maintainers, then it becomes
easier for them to transition. Yes, there is absolutely still work if they have
to maintain multiple branches.


 EXTRA_OECONF = "--with-perl='${USRBINPATH}/env perl' --with-patch=patch"
-EXTRA_OECONF_append_class-native = " --disable-nls"
+EXTRA_OECONF:append:class-native = " --disable-nls"
So operations (append) and overrides would both be delimited by ':'? It keeps
the ordering and makes it easier to read for sure, but I'm wondering if we need
to deliminate the operation differently. (No I have no suggestions, just a
thought. I don't object to the proposal at all.)
I wondered but this kind of felt neatest to me and it does work nicely with
the code backend handling it...



-RDEPENDS_${PN}-ptest = "make file sed gawk diffutils findutils ed perl \
+RDEPENDS:${PN}-ptest = "make file sed gawk diffutils findutils ed perl \
                         perl-module-filehandle perl-module-getopt-std \
                         perl-module-posix perl-module-file-temp \
                         perl-module-text-parsewords perl-module-overloading \
The above would work without code changes to the packaging class, but maybe we
really should make the change for readability. (I've often wondered if it
really should use 'flag' syntax instead, but lack of overrides make flag syntax
difficult to use.)

i.e.:

-FILES_${PN} = "${sysconfdir} ${datadir}/quilt \
+FILES[${PN}] = "${sysconfdir} ${datadir}/quilt \

or maybe something like? (This is a larger syntax change though)

-RDEPENDS_${PN} = "bash patch diffstat bzip2 util-linux less"
-RDEPENDS_${PN}_class-native = "diffstat-native patch-native bzip2-native"
+RDEPENDS[${PN}] = "bash patch diffstat bzip2 util-linux less"
+RDEPENDS[${PN}]:class-native = "diffstat-native patch-native bzip2-native"
'flags' in bitbake are really a poor cousin to the main variables and behave 
quite differently. Whether flags should have override support is a different 
question. It would hugely complicate the datastore code and I'm not sure the
above is more readable compared to the simpler ":". I've always seen flags as
supplemental data for a variable rather than anything like the above.
This change would certainly allow flags to more easily have overrides as you
avoid the multiple "_" is it a literal _ or an override deliminator. But
supporting future override of flags would be possible -- I the more I think
about this, the more I agree that is a bad idea for a first pass at this.

So I amend what I said above to say, avoid flags and my suggestion would be
(because I don't like package name as an override):

-RDEPENDS_${PN} = "bash patch diffstat bzip2 util-linux less"
-RDEPENDS_${PN}_class-native = "diffstat-native patch-native bzip2-native"
+RDEPENDS_${PN} = "bash patch diffstat bzip2 util-linux less"
+RDEPENDS_${PN}:class-native = "diffstat-native patch-native bzip2-native"

With the limited number of variables that are used by "package" implementation
today, it should be easy to teach the conversion script to NOT convert
RDEPENDS_first_second_third_fourth to RDEPENDS:first, unless first is an
operation (append, prepend, etc.. which it really shouldn't be!)

--Mark

Cheers,

Richard


Re: Should we change variable override formatting?

Richard Purdie
 

On Fri, 2021-07-16 at 14:35 +0000, Peter Kjellerstedt wrote:

This is a long answer, and responding to multiple subjects that have been
brought up in the different threads, so please bear with me.

I would love to see improvements to the bitbake syntax when it comes to
overrides. Unfortunately, I do not see how I would be able to ever update
to said version if it is done as a flag day release. The way we work is
to have our layers compatible with the version of bitbake we are using
(currently Hardknott). Then I (who basically do all work to build our
layers with master of Poky) have a special adaptation layer where I add
bbappends and some occasional backported bbclass to make our layers work
with Poky master. This work runs in parallel with the normal recipe updates
that the developers do so that when a new major release of Poky is ready,
so are we. However, if we would have to use a whole different version of
all our recipes, there is no way this process would work, and I can't stop
the normal development for a flag day upgrade of Poky. :(

The way I see it can be done would involve supporting both syntaxes in
one LTS-release. In the next non-LTS release, the support for the old
syntax can be dropped. That way there is always an intermediate release
that one can update to, update one's own recipes and then proceed to the
latest release. This of course means that the Yocto Project cannot reap
the benefits of the new and improved syntax immediately, but would have
to hold off for one release. On the other hand, this would mean the
transitioning path for all users of the Yocto Project would be trivial
(for some definition of trivial), compared to a major PITA.

Now, I don't know if supporting both syntaxes at once is feasible given
some of the comments given in the thread. However, most of them seem
related to being able to drop support for the old syntax and thereby
gaining the expected improvements to the internal data, which would not
be a goal for the release that actually has to support both syntaxes.
Its goal is compatibility and a way forward.
I'm not sure it is going to be feasible to take two years to make this kind
of transition. I've been looking for alternatives and one possibility may be
semi-automated translation.

By that, I mean that if you have a script with some knowledge of the layer
it is translating, it appears from some quick tests locally that you can
automate translation of a layer. My test was with poky (so bitbake, oe-core, 
meta-yocto and even the docs).

The kind of knowledge a script needs is the names of the overrides in use
and also function names or named parameter which contain keywords like
"prepend", "append" and "remove". Once you prime it with that infomation,
which a layer maintainer should have a good idea of, the script seems to
be able to handle the bulk of translation.

Code speaks volumes so:

http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=rpurdie/overrides-convert

which has a script to translate poky, a commit of the translated content
and a patch to bitbake to make it use ":" instead of "_" for overrides.

I appreciate that translation like this isn't ideal, but it might just work
well enough for people to work with to allow us to move forward.

As to the actual syntax changes, one thing I would like to se is the
_append/_prepend/_remove operators actually turn into real operators.
I definitely do not want to see the same syntax for the operators as
for overrides (that is one of the most confusing things with the
current solution).
I very much want to improve things. The trouble is if we try and do too much 
at once, we'll lose the support of our users. There are several things we're
effectively trying to do:

a) Add more information to the metadata (which text *is* an override?)
b) Change the syntax to be more readable
c) Remove certain syntax/operations
d) Potentially change the behaviour of some of the syntax

In the original thread, I was mixing up several of these things. I've
tried to step back and try and just solve a) in a way we can agree on
as a first step. If we can't even do that...

Here is one suggestion:

PACKAGECONFIG_append = "foo" => PACKAGECONFIG{} .= "foo"
PACKAGECONFIG_append = " foo" => PACKAGECONFIG{} += "foo"
PACKAGECONFIG_prepend = "foo " => PACKAGECONFIG{} =. "foo"
PACKAGECONFIG_prepend = " foo " => PACKAGECONFIG{} =+ "foo"
PACKAGECONFIG_remove = "foo " => PACKAGECONFIG{} -= "foo"
How do you explain

PACKAGECONFIG{} += "foo"
vs
PACKAGECONFIG += "foo"

to a user? I suspect that is the kind of thing which will cause 
us problems.

[Just to be clear to everyone reading this, the problem is that .= and 
_append are not equivalent since the former applies to the default value, 
the latter to all possible values including overrides.]


PACKAGECONFIG_append_bar = "foo" => PACKAGECONFIG{bar} .= "foo"
PACKAGECONFIG_append_bar = " foo" => PACKAGECONFIG{bar} += "foo"
PACKAGECONFIG_prepend_bar = "foo " => PACKAGECONFIG{bar} =. "foo"
PACKAGECONFIG_prepend_bar = " foo " => PACKAGECONFIG{bar} =+ "foo"
PACKAGECONFIG_remove_bar = "foo " => PACKAGECONFIG{bar} -= "foo"

PACKAGECONFIG_append_bar_foo = "foo" => PACKAGECONFIG{bar&foo} .= "foo"
PACKAGECONFIG_append_bar_foo = " foo" => PACKAGECONFIG{bar&foo} += "foo"
PACKAGECONFIG_prepend_bar_foo = "foo " => PACKAGECONFIG{bar&foo} =. "foo"
PACKAGECONFIG_prepend_bar_foo = " foo " => PACKAGECONFIG{bar&foo} =+ "foo"
PACKAGECONFIG_remove_bar_foo = "foo " => PACKAGECONFIG{bar&foo} -= "foo"

This would not have a corresponding solution for, e.g.,
PACKAGECONFIG_bar_foo_append, but I actually think that is good.
Right, I think elsewhere I said this form was usually error prone 
and not correct. I was considering dropping it, *if* we can get to a point
where the parser can reliably detect it.

This would also allow for clearing out the pending appends for a
given override by doing:

PACKAGECONFIG{} = ""
I like this idea.

or:

PACKAGECONFIG{bar} = ""
However:

PACKAGECONFIG = "X"
PACKAGECONFIG_bar = ""

is a common usage.

And a totally different solution (and probably more controversial) would
be to add a more programmatical way to handle overrides, e.g.:

PACKAGECONFIG += "foo" if overrides("foo bar")

or:

if override("foo") && override("bar"): PACKAGECONFIG += "foo"

It doesn't help with whether the update to the variable is immediate or
deferred though.
I think these are getting too clever and steps too far beyond what
we can cope with both in bitbake and with the developers we have. I can't 
begin to think how these would work in the code :(.

As to RDEPENDS_${PN} & co, wouldn't it make sense to use RDEPENDS_pn-${PN},
since "pn-<package name>" already exists as an override? 
PN is really badly named since it is effectively the recipe name in many 
contexts. We're specifying one single thing in overrides with it rather than
a suite of packages a recipe generates. I can't see how using pn- like that
would work for the use cases we have for the package name override.

On that note,
I would like to see overrides being prefixed more (maybe even as a
requirement). Some suggestions:

pn- - Package name
df- - Distro feature
mf- - Machine feature
pf- - Package feature (can we rename/alias PACKAGECONFIG to
          PACKAGE_FEATURES while at it, for consistency?)
arch- - Architecture
dist- - Distribution

That way it would be more obvious what an override does. Especially the
use of the distro name as an override can have some unforeseen consequences
if the name is somewhat common).
We have tried to do this with newer overrides where it makes sense. It is 
a tradeoff between longer names and usability. I don't think adding something
for the package name would help. Having "distro-" and "machine-" could
help in some cases but it comes at the price of more complicated conversions
and it becomes a question of whether things become more readable. Worth thinking
about but the above list doesn't really make sense since features are in overrides.

(PACKAGECONFIG should really be RECIPE_FEATURES if we're going to talk names!)

Cheers,

Richard


Re: Should we change variable override formatting?

Peter Kjellerstedt
 

-----Original Message-----
From: openembedded-architecture@... <openembedded-
architecture@...> On Behalf Of Richard Purdie
Sent: den 16 juli 2021 12:09
To: Nicolas Dechesne <nicolas.dechesne@...>
Cc: openembedded-architecture <openembedded-
architecture@...>
Subject: Re: [Openembedded-architecture] Should we change variable
override formatting?

On Fri, 2021-07-16 at 10:12 +0200, Nicolas Dechesne wrote:
hey!

On Thu, Jul 15, 2021 at 3:56 PM Richard Purdie
<richard.purdie@...> wrote:
b) Some of the packaging code (at the very least) needs rewriting as
it accesses
   both RDEPENDS_${PN} and puts ${PN} in OVERRIDES and accesses
RDEPENDS. I'm not
   sure what else may be assuming this works.

Ouch.. I never realized that! I like Mark's suggestion to convert that
to variables. RDEPENDS is typically
used in users' layers, so that  way they wouldn't be impacted by an
override syntax change.
I have to admit I'm leaning the other way, make this explicit and hope
that
it actually helps users understand overrides a little better by making it
clear.


This change does buy us cleaner looking metadata and ultimately, a
faster
and cleaner internal bitbake that at least would use less memory. It
could set
the stage to allow the defval idea I mentioned in a previous mail to
happen.

It is a huge change and would need a lot of work to make it happen. Is
it worth
doing? Not sure. I'm putting it out there for discussion.

There is no doubt that the change is a good change and the project would
benefit
from it. However I think we must worry about our users, and even look
beyond the
layer maintainers that we know (e.g. the one on this list). There are
*way* more
layer maintainers that we don't know about in all the many companies who
are
successfully using YP to build their products. I am not worried about
all the main
layers at all, since we know all their maintainers, and we know they
will understand
why we make this change, and will make the effort to support it. But I
am very
worried about the hundreds (thousands??) of layer maintainers in all the
companies
using Yocto.

So if we do anything, I would really prefer if we find a way to do it in
a good way
*for them* , not *for us*. which means finding a way to support a soft
transition.
And of course we need to align this change with our LTS release cycle.
So perhaps
we can figure out how to support both syntaxes for some time (from one
LTS to the
next?), even if supporting both is impacting the performance. Or we can
have a layer
flag that indicates if it uses the old or the new syntax?
If we support both forms, firstly, few of the layers you're concerned
about will
change as it will break older compatibility and they have no incentive to
do so.

As I mentioned in another reply, there is a way some backwards
compatibility could
be added however I do worry about the horrible corner cases that may
result in.

Secondly, supporting both means we can't take advantage of any benefits
the change
brings to further improve for a period of time. Do we want to commit to
making this
change so that we can benefit from it in say four years time (a couple of
LTS cycles
you mention)?

I'm worried about the pain of transitioning, but I'm also very worried
that unless
we figure out how we can change something, we're not going to develop
beyond where
we are now.

I'm trying hard to find any way we can move forward with some of the
issues people
are raising and I'm not seeing many options. I'm not seeing any proposals
from anyone
else either, probably as there is so much inertia to overcome to make any
change :(.

Cheers,

Richard
This is a long answer, and responding to multiple subjects that have been
brought up in the different threads, so please bear with me.

I would love to see improvements to the bitbake syntax when it comes to
overrides. Unfortunately, I do not see how I would be able to ever update
to said version if it is done as a flag day release. The way we work is
to have our layers compatible with the version of bitbake we are using
(currently Hardknott). Then I (who basically do all work to build our
layers with master of Poky) have a special adaptation layer where I add
bbappends and some occasional backported bbclass to make our layers work
with Poky master. This work runs in parallel with the normal recipe updates
that the developers do so that when a new major release of Poky is ready,
so are we. However, if we would have to use a whole different version of
all our recipes, there is no way this process would work, and I can't stop
the normal development for a flag day upgrade of Poky. :(

The way I see it can be done would involve supporting both syntaxes in
one LTS-release. In the next non-LTS release, the support for the old
syntax can be dropped. That way there is always an intermediate release
that one can update to, update one's own recipes and then proceed to the
latest release. This of course means that the Yocto Project cannot reap
the benefits of the new and improved syntax immediately, but would have
to hold off for one release. On the other hand, this would mean the
transitioning path for all users of the Yocto Project would be trivial
(for some definition of trivial), compared to a major PITA.

Now, I don't know if supporting both syntaxes at once is feasible given
some of the comments given in the thread. However, most of them seem
related to being able to drop support for the old syntax and thereby
gaining the expected improvements to the internal data, which would not
be a goal for the release that actually has to support both syntaxes.
Its goal is compatibility and a way forward.


As to the actual syntax changes, one thing I would like to se is the
_append/_prepend/_remove operators actually turn into real operators.
I definitely do not want to see the same syntax for the operators as
for overrides (that is one of the most confusing things with the
current solution).

Here is one suggestion:

PACKAGECONFIG_append = "foo" => PACKAGECONFIG{} .= "foo"
PACKAGECONFIG_append = " foo" => PACKAGECONFIG{} += "foo"
PACKAGECONFIG_prepend = "foo " => PACKAGECONFIG{} =. "foo"
PACKAGECONFIG_prepend = " foo " => PACKAGECONFIG{} =+ "foo"
PACKAGECONFIG_remove = "foo " => PACKAGECONFIG{} -= "foo"

PACKAGECONFIG_append_bar = "foo" => PACKAGECONFIG{bar} .= "foo"
PACKAGECONFIG_append_bar = " foo" => PACKAGECONFIG{bar} += "foo"
PACKAGECONFIG_prepend_bar = "foo " => PACKAGECONFIG{bar} =. "foo"
PACKAGECONFIG_prepend_bar = " foo " => PACKAGECONFIG{bar} =+ "foo"
PACKAGECONFIG_remove_bar = "foo " => PACKAGECONFIG{bar} -= "foo"

PACKAGECONFIG_append_bar_foo = "foo" => PACKAGECONFIG{bar&foo} .= "foo"
PACKAGECONFIG_append_bar_foo = " foo" => PACKAGECONFIG{bar&foo} += "foo"
PACKAGECONFIG_prepend_bar_foo = "foo " => PACKAGECONFIG{bar&foo} =. "foo"
PACKAGECONFIG_prepend_bar_foo = " foo " => PACKAGECONFIG{bar&foo} =+ "foo"
PACKAGECONFIG_remove_bar_foo = "foo " => PACKAGECONFIG{bar&foo} -= "foo"

This would not have a corresponding solution for, e.g.,
PACKAGECONFIG_bar_foo_append, but I actually think that is good.

This would also allow for clearing out the pending appends for a
given override by doing:

PACKAGECONFIG{} = ""

or:

PACKAGECONFIG{bar} = ""

And a totally different solution (and probably more controversial) would
be to add a more programmatical way to handle overrides, e.g.:

PACKAGECONFIG += "foo" if overrides("foo bar")

or:

if override("foo") && override("bar"): PACKAGECONFIG += "foo"

It doesn't help with whether the update to the variable is immediate or
deferred though.


As to RDEPENDS_${PN} & co, wouldn't it make sense to use RDEPENDS_pn-${PN},
since "pn-<package name>" already exists as an override? On that note,
I would like to see overrides being prefixed more (maybe even as a
requirement). Some suggestions:

pn- - Package name
df- - Distro feature
mf- - Machine feature
pf- - Package feature (can we rename/alias PACKAGECONFIG to
PACKAGE_FEATURES while at it, for consistency?)
arch- - Architecture
dist- - Distribution

That way it would be more obvious what an override does. Especially the
use of the distro name as an override can have some unforeseen consequences
if the name is somewhat common).

//Peter


Debugging error in do_image functions

Mayank Agarwal
 

Hi,

I am getting error in do_image_rdk function while building the core-minimal-image for the rdk code.
Please let me know how can i debug the error and where to see the full error log that can help in understanding 
the details about the error.

Regards
Mayank


Re: Should we change variable override formatting?

Nicolas Dechesne
 



On Fri, Jul 16, 2021 at 12:09 PM Richard Purdie <richard.purdie@...> wrote:
On Fri, 2021-07-16 at 10:12 +0200, Nicolas Dechesne wrote:
> hey!
>
> On Thu, Jul 15, 2021 at 3:56 PM Richard Purdie <richard.purdie@...> wrote:
> > b) Some of the packaging code (at the very least) needs rewriting as it accesses
> >    both RDEPENDS_${PN} and puts ${PN} in OVERRIDES and accesses RDEPENDS. I'm not
> >    sure what else may be assuming this works.
> >
>
>
> Ouch.. I never realized that! I like Mark's suggestion to convert that to variables. RDEPENDS is typically
> used in users' layers, so that  way they wouldn't be impacted by an override syntax change.
>

I have to admit I'm leaning the other way, make this explicit and hope that
it actually helps users understand overrides a little better by making it clear.


> > This change does buy us cleaner looking metadata and ultimately, a faster
> > and cleaner internal bitbake that at least would use less memory. It could set
> > the stage to allow the defval idea I mentioned in a previous mail to happen.
> >
> > It is a huge change and would need a lot of work to make it happen. Is it worth 
> > doing? Not sure. I'm putting it out there for discussion.
> >
>
>
> There is no doubt that the change is a good change and the project would benefit 
> from it. However I think we must worry about our users, and even look beyond the 
> layer maintainers that we know (e.g. the one on this list). There are *way* more 
> layer maintainers that we don't know about in all the many companies who are 
> successfully using YP to build their products. I am not worried about all the main 
> layers at all, since we know all their maintainers, and we know they will understand 
> why we make this change, and will make the effort to support it. But I am very 
> worried about the hundreds (thousands??) of layer maintainers in all the companies 
> using Yocto. 
>
> So if we do anything, I would really prefer if we find a way to do it in a good way 
> *for them* , not *for us*. which means finding a way to support a soft transition. 
> And of course we need to align this change with our LTS release cycle. So perhaps 
> we can figure out how to support both syntaxes for some time (from one LTS to the 
> next?), even if supporting both is impacting the performance. Or we can have a layer 
> flag that indicates if it uses the old or the new syntax? 

If we support both forms, firstly, few of the layers you're concerned about will 
change as it will break older compatibility and they have no incentive to do so.

Well, I am not asking we support both forever, but that we offer a proper transition period. For most key (and public) layers, it might be an option to switch 'quickly' enough, but for developers involved in making projects they are lagging behind what the 'core' community is doing. So telling them in the next LTS everything breaks is not a very good message. 
 

As I mentioned in another reply, there is a way some backwards compatibility could
be added however I do worry about the horrible corner cases that may result in.

I definitely understand that.. I am just trying to emphasize that it's important to think about the large user base that is out of our reach.
 

Secondly, supporting both means we can't take advantage of any benefits the change
brings to further improve for a period of time. Do we want to commit to making this
change so that we can benefit from it in say four years time (a couple of LTS cycles 
you mention)?

I didn't not say a couple of LTS cycles ;) but from one LTS to the next. So perhaps a 2 year transition period. e.g. introduce the change in the next LTS, and deprecate the old syntax in the next-next LTS. 
 

I'm worried about the pain of transitioning, but I'm also very worried that unless
we figure out how we can change something, we're not going to develop beyond where
we are now.

I'm trying hard to find any way we can move forward with some of the issues people
are raising and I'm not seeing many options. I'm not seeing any proposals from anyone
else either, probably as there is so much inertia to overcome to make any change :(. 

Cheers,

Richard





Re: Should we change variable override formatting?

Richard Purdie
 

On Fri, 2021-07-16 at 10:12 +0200, Nicolas Dechesne wrote:
hey!

On Thu, Jul 15, 2021 at 3:56 PM Richard Purdie <richard.purdie@...> wrote:
b) Some of the packaging code (at the very least) needs rewriting as it accesses
   both RDEPENDS_${PN} and puts ${PN} in OVERRIDES and accesses RDEPENDS. I'm not
   sure what else may be assuming this works.

Ouch.. I never realized that! I like Mark's suggestion to convert that to variables. RDEPENDS is typically
used in users' layers, so that  way they wouldn't be impacted by an override syntax change.
I have to admit I'm leaning the other way, make this explicit and hope that
it actually helps users understand overrides a little better by making it clear.


This change does buy us cleaner looking metadata and ultimately, a faster
and cleaner internal bitbake that at least would use less memory. It could set
the stage to allow the defval idea I mentioned in a previous mail to happen.

It is a huge change and would need a lot of work to make it happen. Is it worth 
doing? Not sure. I'm putting it out there for discussion.

There is no doubt that the change is a good change and the project would benefit 
from it. However I think we must worry about our users, and even look beyond the 
layer maintainers that we know (e.g. the one on this list). There are *way* more 
layer maintainers that we don't know about in all the many companies who are 
successfully using YP to build their products. I am not worried about all the main 
layers at all, since we know all their maintainers, and we know they will understand 
why we make this change, and will make the effort to support it. But I am very 
worried about the hundreds (thousands??) of layer maintainers in all the companies 
using Yocto. 

So if we do anything, I would really prefer if we find a way to do it in a good way 
*for them* , not *for us*. which means finding a way to support a soft transition. 
And of course we need to align this change with our LTS release cycle. So perhaps 
we can figure out how to support both syntaxes for some time (from one LTS to the 
next?), even if supporting both is impacting the performance. Or we can have a layer 
flag that indicates if it uses the old or the new syntax? 
If we support both forms, firstly, few of the layers you're concerned about will 
change as it will break older compatibility and they have no incentive to do so.

As I mentioned in another reply, there is a way some backwards compatibility could
be added however I do worry about the horrible corner cases that may result in.

Secondly, supporting both means we can't take advantage of any benefits the change
brings to further improve for a period of time. Do we want to commit to making this
change so that we can benefit from it in say four years time (a couple of LTS cycles 
you mention)?

I'm worried about the pain of transitioning, but I'm also very worried that unless
we figure out how we can change something, we're not going to develop beyond where
we are now.

I'm trying hard to find any way we can move forward with some of the issues people
are raising and I'm not seeing many options. I'm not seeing any proposals from anyone
else either, probably as there is so much inertia to overcome to make any change :(.

Cheers,

Richard


Re: Should we change variable override formatting?

Martin Jansa
 

On Fri, Jul 16, 2021 at 10:12:57AM +0200, Nicolas Dechesne wrote:
hey!

On Thu, Jul 15, 2021 at 3:56 PM Richard Purdie <
richard.purdie@...> wrote:

Breaking things down a bit, one thing I keep running into with our current
codebase and metadata is that overrides are not clear. In my previous
email,
the example was:

do_configure_class-native

A human can usually spot "class-native" is and "configure" or
"configure_class-native" is not. Bitbake's parser struggles. It has huge
internal lists including variables like x86_64 where it has to track
whether "64" in an override.

One way of fixing this is to be explicit about overrides and use a
different
separator. See an example patch below I made to the quilt recipe using ":"
instead to see how it looks. Personally, I think this looks like an
improvement.

There are two challenges:

a) The work of migration. Do we migrate piecemeal or with a flag day? I'm
not
sure piecemeal is even possible unfortunately.

b) Some of the packaging code (at the very least) needs rewriting as it
accesses
both RDEPENDS_${PN} and puts ${PN} in OVERRIDES and accesses RDEPENDS.
I'm not
sure what else may be assuming this works.
Ouch.. I never realized that! I like Mark's suggestion to convert that to
variables. RDEPENDS is typically used in users' layers, so that way they
wouldn't be impacted by an override syntax change.


This change does buy us cleaner looking metadata and ultimately, a faster
and cleaner internal bitbake that at least would use less memory. It could
set
the stage to allow the defval idea I mentioned in a previous mail to
happen.

It is a huge change and would need a lot of work to make it happen. Is it
worth
doing? Not sure. I'm putting it out there for discussion.
There is no doubt that the change is a good change and the project would
benefit from it. However I think we must worry about our users, and even
look beyond the layer maintainers that we know (e.g. the one on this list).
There are *way* more layer maintainers that we don't know about in all the
many companies who are successfully using YP to build their products. I am
not worried about all the main layers at all, since we know all their
maintainers, and we know they will understand why we make this change, and
will make the effort to support it. But I am very worried about the
hundreds (thousands??) of layer maintainers in all the companies using
Yocto.

So if we do anything, I would really prefer if we find a way to do it in a
good way *for them* , not *for us*. which means finding a way to support a
soft transition. And of course we need to align this change with our LTS
release cycle. So perhaps we can figure out how to support both syntaxes
for some time (from one LTS to the next?), even if supporting both is
impacting the performance. Or we can have a layer flag that indicates if it
uses the old or the new syntax?
I agree.

I understand that this change would be needed to improve the metadata we
currently have, but I would like to stress that such big change in
syntax would be major PIA for some use cases.

E.g. with meta-ros https://github.com/ros/meta-ros which currently has
over 7000 recipes mostly generated from OSRF index by superflore
https://github.com/ros-infrastructure/superflore (there are a lot of
changes every few days).

Currently the work-flow is to regenerate the recipes for dunfell and
then migrate these changes to other branches as well. With different
syntax used in future LTS we would either need to teach superflore to
generate different syntax (but regenerate new recipes twice with
different syntax - and the regeneration takes quite a while) or reliable
script to migrate to new syntax (to generate with old syntax with
superflore and then migrate to new syntax before "cherry-pick" to
the next LTS branch).

So this is just one example that for some use-cases this change won't be
just one-time flag-day to migrate existing metadata, but ongoing PIA to
support multiple OE releases (e.g. dunfell LTS at the same time as the
next LTS).

Similarly for projects which do development e.g. on dunfell
branch, but maintain a master branch compatible with latest for others
to use, now if they need to spend significant time to migrate changes
from dunfell to master (with the same syntax simple git merge was often
good enough without any conflicts).

And last example could be meta-qt5 where the branches are usually
compatible with wide range of OE releases and projects tend to mismatch
them often (because they need specific version of Qt - e.g. due
licensing changes in newest - while using different OE release - e.g.
webOS OSE was using dunfell for all layers except meta-qt5 in warrior
branch to get Qt 5.12 with just small meta-qt5-compat shim layer to make
them compatible (recently migrated to meta-qt6 in
https://github.com/webosose/build-webos/commit/057477b7d8d95fee5714a468ed43defe085360a2)
With a new syntax I would need to update these older meta-qt5 branches
with the new syntax (like warrior-new-syntax branch for projects in this
situation - or even 5.12-new-syntax or <next-lts>-5.12 which could include
all compatibility tweaks directly).

And meta-qt6 will have similar issue as now it has
LAYERSERIES_COMPAT_qt6-layer = "zeus dunfell gatesgarth hardknott" in
https://code.qt.io/cgit/yocto/meta-qt6.git/tree/conf/layer.conf?h=6.2
but for future it will need 2 syntax variants.

Regards,


Re: Should we change variable override formatting?

Richard Purdie
 

On Thu, 2021-07-15 at 23:48 -0400, Denys Dmytriyenko wrote:
On Thu, Jul 15, 2021 at 02:56:38PM +0100, Richard Purdie wrote:
Breaking things down a bit, one thing I keep running into with our current 
codebase and metadata is that overrides are not clear. In my previous email,
the example was:

do_configure_class-native

A human can usually spot "class-native" is and "configure" or 
"configure_class-native" is not. Bitbake's parser struggles. It has huge 
internal lists including variables like x86_64 where it has to track 
whether "64" in an override.

One way of fixing this is to be explicit about overrides and use a different 
separator. See an example patch below I made to the quilt recipe using ":" 
instead to see how it looks. Personally, I think this looks like an improvement.

There are two challenges:

a) The work of migration. Do we migrate piecemeal or with a flag day? I'm not
   sure piecemeal is even possible unfortunately.

b) Some of the packaging code (at the very least) needs rewriting as it accesses
   both RDEPENDS_${PN} and puts ${PN} in OVERRIDES and accesses RDEPENDS. I'm not
   sure what else may be assuming this works.

This change does buy us cleaner looking metadata and ultimately, a faster
and cleaner internal bitbake that at least would use less memory. It could set
the stage to allow the defval idea I mentioned in a previous mail to happen.

It is a huge change and would need a lot of work to make it happen. Is it worth 
doing? Not sure. I'm putting it out there for discussion.

It is also going to be tempting that "if we're breaking things, lets break lots".
I want to be mindful that tempting as that may be, if users can't convert clearly,
it will be worlds of pain. The benefit of this change is that at least in the
general case, the transition should be clear. Taking steps rather than changing the
world may therefore be a better idea...
The idea is interesting, but it would certainly break layers out there that
only provide a single master branch and claim compatibility with several past
releases, including dunfell, or even older. Few come to mind - meta-browser,
meta-linux-mainline, meta-etherium, etc.
This is the dilemma. We hear a lot about usability and the syntax being confusing
and hard to use/understand. If we try and improve that, we are going to have to
break compatibility at some point/level.

We've not done it for a while but I'm not sure that is a good thing and we perhaps
do need to try and improve some things.

One possibility is that if we patched older bitbake's to substitute ":" for "_" in
variables coming into the data store, it might be able to handle metadata using 
the new delimiter to some level. I'm not entirely sure you could cover every possible
situation, it might work.

There would still end up being a flag day where beyond which we couldn't progress
without relying upon the new syntax. I'm a little reluctant to look too hard at that
as writing metadata to work with datastores with two different behaviours doesn't
sound like a great idea from a testing perspective and ultimately may hurt users more
that it would help.

Cheers,

Richard


Re: Should we change variable override formatting?

Richard Purdie
 

On Thu, 2021-07-15 at 09:26 -0500, Mark Hatle wrote:

On 7/15/21 8:56 AM, Richard Purdie wrote:
Breaking things down a bit, one thing I keep running into with our current 
codebase and metadata is that overrides are not clear. In my previous email,
the example was:

do_configure_class-native

A human can usually spot "class-native" is and "configure" or 
"configure_class-native" is not. Bitbake's parser struggles. It has huge 
internal lists including variables like x86_64 where it has to track 
whether "64" in an override.

One way of fixing this is to be explicit about overrides and use a different 
separator. See an example patch below I made to the quilt recipe using ":" 
instead to see how it looks. Personally, I think this looks like an improvement.

There are two challenges:

a) The work of migration. Do we migrate piecemeal or with a flag day? I'm not
   sure piecemeal is even possible unfortunately.

b) Some of the packaging code (at the very least) needs rewriting as it accesses
   both RDEPENDS_${PN} and puts ${PN} in OVERRIDES and accesses RDEPENDS. I'm not
   sure what else may be assuming this works.
Ya, this is all over the packaging code. Should be trivial to move to treating
it as a variable (and not override). I can help with this if you want.
I've been pondering this a bit more and my preference is to move this to use 
overrides more explicitly.

What I did realise is that d.getVar("RDEPENDS:${PN}") can actually split out
the override itself and apply it so I think the existing code can be made to
work just with the character change if we teach getVar how to handle it which
should be possible.

(It can of course also be moved to overrides, but that won't be as obvious to a
recipe creator, since they're not used to the package name being an 'override'.)
Whilst the instinct is to "hide" that, perhaps it could also make overrides more
obvious since people could see them in a different setting more clearly? I think
that is where I'm leaning right now.

This change does buy us cleaner looking metadata and ultimately, a faster
and cleaner internal bitbake that at least would use less memory. It could set
the stage to allow the defval idea I mentioned in a previous mail to happen.

It is a huge change and would need a lot of work to make it happen. Is it worth 
doing? Not sure. I'm putting it out there for discussion.
Is this something we do in parallel with master, and then flag day it? I could
easily see this work take a few weeks (if a bunch of people help).
I think we might be able to block convert much of the metadata with some kind of 
script. If that is possible, we'd have a flag day and a bunch of code tested in 
parallel before a final semi-automated conversion.

It is also going to be tempting that "if we're breaking things, lets break lots".
I want to be mindful that tempting as that may be, if users can't convert clearly,
it will be worlds of pain. The benefit of this change is that at least in the
general case, the transition should be clear. Taking steps rather than changing the
world may therefore be a better idea...
One of those things (variable syntax) I'm not sure 'steps' make sense. If we're
changing a fundamental syntax, it really becomes a new major version of bitbake
and thus a new version of OE.
Having lots of things change at once may make things much harder for layer maintainers
than having a controlled change to handle though?


 EXTRA_OECONF = "--with-perl='${USRBINPATH}/env perl' --with-patch=patch"
-EXTRA_OECONF_append_class-native = " --disable-nls"
+EXTRA_OECONF:append:class-native = " --disable-nls"
So operations (append) and overrides would both be delimited by ':'? It keeps
the ordering and makes it easier to read for sure, but I'm wondering if we need
to deliminate the operation differently. (No I have no suggestions, just a
thought. I don't object to the proposal at all.)
I wondered but this kind of felt neatest to me and it does work nicely with
the code backend handling it...



-RDEPENDS_${PN}-ptest = "make file sed gawk diffutils findutils ed perl \
+RDEPENDS:${PN}-ptest = "make file sed gawk diffutils findutils ed perl \
                         perl-module-filehandle perl-module-getopt-std \
                         perl-module-posix perl-module-file-temp \
                         perl-module-text-parsewords perl-module-overloading \
The above would work without code changes to the packaging class, but maybe we
really should make the change for readability. (I've often wondered if it
really should use 'flag' syntax instead, but lack of overrides make flag syntax
difficult to use.)

i.e.:

-FILES_${PN} = "${sysconfdir} ${datadir}/quilt \
+FILES[${PN}] = "${sysconfdir} ${datadir}/quilt \

or maybe something like? (This is a larger syntax change though)

-RDEPENDS_${PN} = "bash patch diffstat bzip2 util-linux less"
-RDEPENDS_${PN}_class-native = "diffstat-native patch-native bzip2-native"
+RDEPENDS[${PN}] = "bash patch diffstat bzip2 util-linux less"
+RDEPENDS[${PN}]:class-native = "diffstat-native patch-native bzip2-native"
'flags' in bitbake are really a poor cousin to the main variables and behave 
quite differently. Whether flags should have override support is a different 
question. It would hugely complicate the datastore code and I'm not sure the
above is more readable compared to the simpler ":". I've always seen flags as
supplemental data for a variable rather than anything like the above.

Cheers,

Richard


Re: Should we change variable override formatting?

Nicolas Dechesne
 

hey!

On Thu, Jul 15, 2021 at 3:56 PM Richard Purdie <richard.purdie@...> wrote:
Breaking things down a bit, one thing I keep running into with our current 
codebase and metadata is that overrides are not clear. In my previous email,
the example was:

do_configure_class-native

A human can usually spot "class-native" is and "configure" or 
"configure_class-native" is not. Bitbake's parser struggles. It has huge 
internal lists including variables like x86_64 where it has to track 
whether "64" in an override.

One way of fixing this is to be explicit about overrides and use a different 
separator. See an example patch below I made to the quilt recipe using ":" 
instead to see how it looks. Personally, I think this looks like an improvement.

There are two challenges:

a) The work of migration. Do we migrate piecemeal or with a flag day? I'm not
   sure piecemeal is even possible unfortunately.

b) Some of the packaging code (at the very least) needs rewriting as it accesses
   both RDEPENDS_${PN} and puts ${PN} in OVERRIDES and accesses RDEPENDS. I'm not
   sure what else may be assuming this works.

Ouch.. I never realized that! I like Mark's suggestion to convert that to variables. RDEPENDS is typically used in users' layers, so that  way they wouldn't be impacted by an override syntax change.


This change does buy us cleaner looking metadata and ultimately, a faster
and cleaner internal bitbake that at least would use less memory. It could set
the stage to allow the defval idea I mentioned in a previous mail to happen.

It is a huge change and would need a lot of work to make it happen. Is it worth 
doing? Not sure. I'm putting it out there for discussion.

There is no doubt that the change is a good change and the project would benefit from it. However I think we must worry about our users, and even look beyond the layer maintainers that we know (e.g. the one on this list). There are *way* more layer maintainers that we don't know about in all the many companies who are successfully using YP to build their products. I am not worried about all the main layers at all, since we know all their maintainers, and we know they will understand why we make this change, and will make the effort to support it. But I am very worried about the hundreds (thousands??) of layer maintainers in all the companies using Yocto. 

So if we do anything, I would really prefer if we find a way to do it in a good way *for them* , not *for us*. which means finding a way to support a soft transition. And of course we need to align this change with our LTS release cycle. So perhaps we can figure out how to support both syntaxes for some time (from one LTS to the next?), even if supporting both is impacting the performance. Or we can have a layer flag that indicates if it uses the old or the new syntax? 


It is also going to be tempting that "if we're breaking things, lets break lots".
I want to be mindful that tempting as that may be, if users can't convert clearly,
it will be worlds of pain. The benefit of this change is that at least in the
general case, the transition should be clear. Taking steps rather than changing the
world may therefore be a better idea...

Cheers,

Richard


diff --git a/meta/recipes-devtools/quilt/quilt.inc b/meta/recipes-devtools/quilt/quilt.inc
index d7ecda7aaa6..f85de384d26 100644
--- a/meta/recipes-devtools/quilt/quilt.inc
+++ b/meta/recipes-devtools/quilt/quilt.inc
@@ -14,36 +14,36 @@ SRC_URI = "${SAVANNAH_GNU_MIRROR}/quilt/quilt-${PV}.tar.gz \
         file://0001-tests-Allow-different-output-from-mv.patch \
 "

-SRC_URI_append_class-target = " file://gnu_patch_test_fix_target.patch"
+SRC_URI:append:class-target = " file://gnu_patch_test_fix_target.patch"

 SRC_URI[md5sum] = "6800c2404a2c0598ab2eff92a636ba70"
 SRC_URI[sha256sum] = "314b319a6feb13bf9d0f9ffa7ce6683b06919e734a41275087ea457cc9dc6e07"

 inherit autotools-brokensep ptest

-INHIBIT_AUTOTOOLS_DEPS_class-native = "1"
-PATCHTOOL_class-native = "patch"
+INHIBIT_AUTOTOOLS_DEPS:class-native = "1"
+PATCHTOOL:class-native = "patch"

 CLEANBROKEN = "1"

 EXTRA_OECONF = "--with-perl='${USRBINPATH}/env perl' --with-patch=patch"
-EXTRA_OECONF_append_class-native = " --disable-nls"
+EXTRA_OECONF:append:class-native = " --disable-nls"
 EXTRA_AUTORECONF += "--exclude=aclocal"

 CACHED_CONFIGUREVARS += "ac_cv_path_BASH=/bin/bash ac_cv_path_COLUMN=column"

 # Make sure we don't have "-w" in shebang lines: it breaks using
 # "/usr/bin/env perl" as parser
-do_configure_prepend () {
+do_configure:prepend () {
        find ${S} -name "*.in" -exec sed -i -e "1s,^#\!.*@PERL@ -w$,#\! @PERL@\nuse warnings;," {} \;
 }

 # Don't setup symlinks to host utilities, we don't need them
-do_configure_append () {
+do_configure:append () {
        sed -e 's,^COMPAT_SYMLINKS.*:=.*,COMPAT_SYMLINKS        :=,' -i ${S}/Makefile
 }

-do_configure_class-native () {
+do_configure:class-native () {
     oe_runconf
 }

@@ -54,7 +54,7 @@ do_install () {
        rm -rf ${D}/${datadir}/emacs
 }

-do_install_append_class-native () {
+do_install:append:class-native () {
     # Dummy quiltrc file for patch.bbclass
     install -d ${D}${sysconfdir}/
     touch ${D}${sysconfdir}/quiltrc
@@ -75,16 +75,16 @@ do_install_ptest() {

 PACKAGES += "guards guards-doc"

-FILES_${PN} = "${sysconfdir} ${datadir}/quilt \
+FILES:${PN} = "${sysconfdir} ${datadir}/quilt \
                ${bindir}/quilt ${libdir}/quilt"
-FILES_guards = "${bindir}/guards"
-FILES_${PN}-doc = "${mandir}/man1/quilt.1 ${docdir}/${BPN}"
-FILES_guards-doc = "${mandir}/man1/guards.1"
+FILES:guards = "${bindir}/guards"
+FILES:${PN}-doc = "${mandir}/man1/quilt.1 ${docdir}/${BPN}"
+FILES:guards-doc = "${mandir}/man1/guards.1"

-RDEPENDS_${PN} = "bash patch diffstat bzip2 util-linux less"
-RDEPENDS_${PN}_class-native = "diffstat-native patch-native bzip2-native"
+RDEPENDS:${PN} = "bash patch diffstat bzip2 util-linux less"
+RDEPENDS:${PN}:class-native = "diffstat-native patch-native bzip2-native"

-RDEPENDS_${PN}-ptest = "make file sed gawk diffutils findutils ed perl \
+RDEPENDS:${PN}-ptest = "make file sed gawk diffutils findutils ed perl \
                         perl-module-filehandle perl-module-getopt-std \
                         perl-module-posix perl-module-file-temp \
                         perl-module-text-parsewords perl-module-overloading \






Re: Should we change variable override formatting?

Denys Dmytriyenko
 

On Thu, Jul 15, 2021 at 02:56:38PM +0100, Richard Purdie wrote:
Breaking things down a bit, one thing I keep running into with our current 
codebase and metadata is that overrides are not clear. In my previous email,
the example was:

do_configure_class-native

A human can usually spot "class-native" is and "configure" or 
"configure_class-native" is not. Bitbake's parser struggles. It has huge 
internal lists including variables like x86_64 where it has to track 
whether "64" in an override.

One way of fixing this is to be explicit about overrides and use a different 
separator. See an example patch below I made to the quilt recipe using ":" 
instead to see how it looks. Personally, I think this looks like an improvement.

There are two challenges:

a) The work of migration. Do we migrate piecemeal or with a flag day? I'm not
sure piecemeal is even possible unfortunately.

b) Some of the packaging code (at the very least) needs rewriting as it accesses
both RDEPENDS_${PN} and puts ${PN} in OVERRIDES and accesses RDEPENDS. I'm not
sure what else may be assuming this works.

This change does buy us cleaner looking metadata and ultimately, a faster
and cleaner internal bitbake that at least would use less memory. It could set
the stage to allow the defval idea I mentioned in a previous mail to happen.

It is a huge change and would need a lot of work to make it happen. Is it worth 
doing? Not sure. I'm putting it out there for discussion.

It is also going to be tempting that "if we're breaking things, lets break lots".
I want to be mindful that tempting as that may be, if users can't convert clearly,
it will be worlds of pain. The benefit of this change is that at least in the
general case, the transition should be clear. Taking steps rather than changing the
world may therefore be a better idea...
The idea is interesting, but it would certainly break layers out there that
only provide a single master branch and claim compatibility with several past
releases, including dunfell, or even older. Few come to mind - meta-browser,
meta-linux-mainline, meta-etherium, etc.


Cheers,

Richard


diff --git a/meta/recipes-devtools/quilt/quilt.inc b/meta/recipes-devtools/quilt/quilt.inc
index d7ecda7aaa6..f85de384d26 100644
--- a/meta/recipes-devtools/quilt/quilt.inc
+++ b/meta/recipes-devtools/quilt/quilt.inc
@@ -14,36 +14,36 @@ SRC_URI = "${SAVANNAH_GNU_MIRROR}/quilt/quilt-${PV}.tar.gz \
file://0001-tests-Allow-different-output-from-mv.patch \
"

-SRC_URI_append_class-target = " file://gnu_patch_test_fix_target.patch"
+SRC_URI:append:class-target = " file://gnu_patch_test_fix_target.patch"

SRC_URI[md5sum] = "6800c2404a2c0598ab2eff92a636ba70"
SRC_URI[sha256sum] = "314b319a6feb13bf9d0f9ffa7ce6683b06919e734a41275087ea457cc9dc6e07"

inherit autotools-brokensep ptest

-INHIBIT_AUTOTOOLS_DEPS_class-native = "1"
-PATCHTOOL_class-native = "patch"
+INHIBIT_AUTOTOOLS_DEPS:class-native = "1"
+PATCHTOOL:class-native = "patch"

CLEANBROKEN = "1"

EXTRA_OECONF = "--with-perl='${USRBINPATH}/env perl' --with-patch=patch"
-EXTRA_OECONF_append_class-native = " --disable-nls"
+EXTRA_OECONF:append:class-native = " --disable-nls"
EXTRA_AUTORECONF += "--exclude=aclocal"

CACHED_CONFIGUREVARS += "ac_cv_path_BASH=/bin/bash ac_cv_path_COLUMN=column"

# Make sure we don't have "-w" in shebang lines: it breaks using
# "/usr/bin/env perl" as parser
-do_configure_prepend () {
+do_configure:prepend () {
find ${S} -name "*.in" -exec sed -i -e "1s,^#\!.*@PERL@ -w$,#\! @PERL@\nuse warnings;," {} \;
}

# Don't setup symlinks to host utilities, we don't need them
-do_configure_append () {
+do_configure:append () {
sed -e 's,^COMPAT_SYMLINKS.*:=.*,COMPAT_SYMLINKS :=,' -i ${S}/Makefile
}

-do_configure_class-native () {
+do_configure:class-native () {
oe_runconf
}

@@ -54,7 +54,7 @@ do_install () {
rm -rf ${D}/${datadir}/emacs
}

-do_install_append_class-native () {
+do_install:append:class-native () {
# Dummy quiltrc file for patch.bbclass
install -d ${D}${sysconfdir}/
touch ${D}${sysconfdir}/quiltrc
@@ -75,16 +75,16 @@ do_install_ptest() {

PACKAGES += "guards guards-doc"

-FILES_${PN} = "${sysconfdir} ${datadir}/quilt \
+FILES:${PN} = "${sysconfdir} ${datadir}/quilt \
${bindir}/quilt ${libdir}/quilt"
-FILES_guards = "${bindir}/guards"
-FILES_${PN}-doc = "${mandir}/man1/quilt.1 ${docdir}/${BPN}"
-FILES_guards-doc = "${mandir}/man1/guards.1"
+FILES:guards = "${bindir}/guards"
+FILES:${PN}-doc = "${mandir}/man1/quilt.1 ${docdir}/${BPN}"
+FILES:guards-doc = "${mandir}/man1/guards.1"

-RDEPENDS_${PN} = "bash patch diffstat bzip2 util-linux less"
-RDEPENDS_${PN}_class-native = "diffstat-native patch-native bzip2-native"
+RDEPENDS:${PN} = "bash patch diffstat bzip2 util-linux less"
+RDEPENDS:${PN}:class-native = "diffstat-native patch-native bzip2-native"

-RDEPENDS_${PN}-ptest = "make file sed gawk diffutils findutils ed perl \
+RDEPENDS:${PN}-ptest = "make file sed gawk diffutils findutils ed perl \
perl-module-filehandle perl-module-getopt-std \
perl-module-posix perl-module-file-temp \
perl-module-text-parsewords perl-module-overloading \




--
Regards,
Denys Dmytriyenko <denis@...>
PGP: 0x420902729A92C964 - https://denix.org/0x420902729A92C964
Fingerprint: 25FC E4A5 8A72 2F69 1186 6D76 4209 0272 9A92 C964


Re: Should we change variable override formatting?

Otavio Salvador
 

Hello Richard,

Em qui., 15 de jul. de 2021 às 10:56, Richard Purdie
<richard.purdie@...> escreveu:
a) The work of migration. Do we migrate piecemeal or with a flag day? I'm not
sure piecemeal is even possible unfortunately.
If we don't do as a flag day we cannot drop old code from bitbake, so
my vote goes to it and don't look back. Of course, this has some
inconvenience for layer maintainers and backward compatibility but I
believe we ought to do it sooner rather than later so we have some
time before next LTS release.

Generally, we could move existing BitBake code to a tool to help
converting existing metadata and reducing this burden.

--
Otavio Salvador O.S. Systems
http://www.ossystems.com.br http://code.ossystems.com.br
Mobile: +55 (53) 9 9981-7854 Mobile: +1 (347) 903-9750


Re: Should we change variable override formatting?

Mark Hatle
 

On 7/15/21 8:56 AM, Richard Purdie wrote:
Breaking things down a bit, one thing I keep running into with our current 
codebase and metadata is that overrides are not clear. In my previous email,
the example was:

do_configure_class-native

A human can usually spot "class-native" is and "configure" or 
"configure_class-native" is not. Bitbake's parser struggles. It has huge 
internal lists including variables like x86_64 where it has to track 
whether "64" in an override.

One way of fixing this is to be explicit about overrides and use a different 
separator. See an example patch below I made to the quilt recipe using ":" 
instead to see how it looks. Personally, I think this looks like an improvement.

There are two challenges:

a) The work of migration. Do we migrate piecemeal or with a flag day? I'm not
sure piecemeal is even possible unfortunately.

b) Some of the packaging code (at the very least) needs rewriting as it accesses
both RDEPENDS_${PN} and puts ${PN} in OVERRIDES and accesses RDEPENDS. I'm not
sure what else may be assuming this works.
Ya, this is all over the packaging code. Should be trivial to move to treating
it as a variable (and not override). I can help with this if you want.

(It can of course also be moved to overrides, but that won't be as obvious to a
recipe creator, since they're not used to the package name being an 'override'.)

This change does buy us cleaner looking metadata and ultimately, a faster
and cleaner internal bitbake that at least would use less memory. It could set
the stage to allow the defval idea I mentioned in a previous mail to happen.

It is a huge change and would need a lot of work to make it happen. Is it worth 
doing? Not sure. I'm putting it out there for discussion.
Is this something we do in parallel with master, and then flag day it? I could
easily see this work take a few weeks (if a bunch of people help).

It is also going to be tempting that "if we're breaking things, lets break lots".
I want to be mindful that tempting as that may be, if users can't convert clearly,
it will be worlds of pain. The benefit of this change is that at least in the
general case, the transition should be clear. Taking steps rather than changing the
world may therefore be a better idea...
One of those things (variable syntax) I'm not sure 'steps' make sense. If we're
changing a fundamental syntax, it really becomes a new major version of bitbake
and thus a new version of OE.

(a few more comments below)

Cheers,

Richard


diff --git a/meta/recipes-devtools/quilt/quilt.inc b/meta/recipes-devtools/quilt/quilt.inc
index d7ecda7aaa6..f85de384d26 100644
--- a/meta/recipes-devtools/quilt/quilt.inc
+++ b/meta/recipes-devtools/quilt/quilt.inc
@@ -14,36 +14,36 @@ SRC_URI = "${SAVANNAH_GNU_MIRROR}/quilt/quilt-${PV}.tar.gz \
file://0001-tests-Allow-different-output-from-mv.patch \
"

-SRC_URI_append_class-target = " file://gnu_patch_test_fix_target.patch"
+SRC_URI:append:class-target = " file://gnu_patch_test_fix_target.patch"

SRC_URI[md5sum] = "6800c2404a2c0598ab2eff92a636ba70"
SRC_URI[sha256sum] = "314b319a6feb13bf9d0f9ffa7ce6683b06919e734a41275087ea457cc9dc6e07"

inherit autotools-brokensep ptest

-INHIBIT_AUTOTOOLS_DEPS_class-native = "1"
-PATCHTOOL_class-native = "patch"
+INHIBIT_AUTOTOOLS_DEPS:class-native = "1"
+PATCHTOOL:class-native = "patch"

CLEANBROKEN = "1"

EXTRA_OECONF = "--with-perl='${USRBINPATH}/env perl' --with-patch=patch"
-EXTRA_OECONF_append_class-native = " --disable-nls"
+EXTRA_OECONF:append:class-native = " --disable-nls"
So operations (append) and overrides would both be delimited by ':'? It keeps
the ordering and makes it easier to read for sure, but I'm wondering if we need
to deliminate the operation differently. (No I have no suggestions, just a
thought. I don't object to the proposal at all.)



EXTRA_AUTORECONF += "--exclude=aclocal"

CACHED_CONFIGUREVARS += "ac_cv_path_BASH=/bin/bash ac_cv_path_COLUMN=column"

# Make sure we don't have "-w" in shebang lines: it breaks using
# "/usr/bin/env perl" as parser
-do_configure_prepend () {
+do_configure:prepend () {
find ${S} -name "*.in" -exec sed -i -e "1s,^#\!.*@PERL@ -w$,#\! @PERL@\nuse warnings;," {} \;
}

# Don't setup symlinks to host utilities, we don't need them
-do_configure_append () {
+do_configure:append () {
sed -e 's,^COMPAT_SYMLINKS.*:=.*,COMPAT_SYMLINKS :=,' -i ${S}/Makefile
}

-do_configure_class-native () {
+do_configure:class-native () {
oe_runconf
}

@@ -54,7 +54,7 @@ do_install () {
rm -rf ${D}/${datadir}/emacs
}

-do_install_append_class-native () {
+do_install:append:class-native () {
# Dummy quiltrc file for patch.bbclass
install -d ${D}${sysconfdir}/
touch ${D}${sysconfdir}/quiltrc
@@ -75,16 +75,16 @@ do_install_ptest() {

PACKAGES += "guards guards-doc"

-FILES_${PN} = "${sysconfdir} ${datadir}/quilt \
+FILES:${PN} = "${sysconfdir} ${datadir}/quilt \
${bindir}/quilt ${libdir}/quilt"
-FILES_guards = "${bindir}/guards"
-FILES_${PN}-doc = "${mandir}/man1/quilt.1 ${docdir}/${BPN}"
-FILES_guards-doc = "${mandir}/man1/guards.1"
+FILES:guards = "${bindir}/guards"
+FILES:${PN}-doc = "${mandir}/man1/quilt.1 ${docdir}/${BPN}"
+FILES:guards-doc = "${mandir}/man1/guards.1"

-RDEPENDS_${PN} = "bash patch diffstat bzip2 util-linux less"
-RDEPENDS_${PN}_class-native = "diffstat-native patch-native bzip2-native"
+RDEPENDS:${PN} = "bash patch diffstat bzip2 util-linux less"
+RDEPENDS:${PN}:class-native = "diffstat-native patch-native bzip2-native"

-RDEPENDS_${PN}-ptest = "make file sed gawk diffutils findutils ed perl \
+RDEPENDS:${PN}-ptest = "make file sed gawk diffutils findutils ed perl \
perl-module-filehandle perl-module-getopt-std \
perl-module-posix perl-module-file-temp \
perl-module-text-parsewords perl-module-overloading \
The above would work without code changes to the packaging class, but maybe we
really should make the change for readability. (I've often wondered if it
really should use 'flag' syntax instead, but lack of overrides make flag syntax
difficult to use.)

i.e.:

-FILES_${PN} = "${sysconfdir} ${datadir}/quilt \
+FILES[${PN}] = "${sysconfdir} ${datadir}/quilt \

or maybe something like? (This is a larger syntax change though)

-RDEPENDS_${PN} = "bash patch diffstat bzip2 util-linux less"
-RDEPENDS_${PN}_class-native = "diffstat-native patch-native bzip2-native"
+RDEPENDS[${PN}] = "bash patch diffstat bzip2 util-linux less"
+RDEPENDS[${PN}]:class-native = "diffstat-native patch-native bzip2-native"

--Mark