On Fri, 2021-07-16 at 14:35 +0000, Peter Kjellerstedt wrote:

This is a long answer, and responding to multiple subjects that have been
brought up in the different threads, so please bear with me.

I would love to see improvements to the bitbake syntax when it comes to
overrides. Unfortunately, I do not see how I would be able to ever update
to said version if it is done as a flag day release. The way we work is
to have our layers compatible with the version of bitbake we are using
(currently Hardknott). Then I (who basically do all work to build our
layers with master of Poky) have a special adaptation layer where I add
bbappends and some occasional backported bbclass to make our layers work
with Poky master. This work runs in parallel with the normal recipe updates
that the developers do so that when a new major release of Poky is ready,
so are we. However, if we would have to use a whole different version of
all our recipes, there is no way this process would work, and I can't stop
the normal development for a flag day upgrade of Poky. :(

The way I see it can be done would involve supporting both syntaxes in
one LTS-release. In the next non-LTS release, the support for the old
syntax can be dropped. That way there is always an intermediate release
that one can update to, update one's own recipes and then proceed to the
latest release. This of course means that the Yocto Project cannot reap
the benefits of the new and improved syntax immediately, but would have
to hold off for one release. On the other hand, this would mean the
transitioning path for all users of the Yocto Project would be trivial
(for some definition of trivial), compared to a major PITA.

Now, I don't know if supporting both syntaxes at once is feasible given
some of the comments given in the thread. However, most of them seem
related to being able to drop support for the old syntax and thereby
gaining the expected improvements to the internal data, which would not
be a goal for the release that actually has to support both syntaxes.
Its goal is compatibility and a way forward.

I'm not sure it is going to be feasible to take two years to make this kind
of transition. I've been looking for alternatives and one possibility may be
semi-automated translation.

By that, I mean that if you have a script with some knowledge of the layer
it is translating, it appears from some quick tests locally that you can
automate translation of a layer. My test was with poky (so bitbake, oe-core, 
meta-yocto and even the docs).

The kind of knowledge a script needs is the names of the overrides in use
and also function names or named parameter which contain keywords like
"prepend", "append" and "remove". Once you prime it with that infomation,
which a layer maintainer should have a good idea of, the script seems to
be able to handle the bulk of translation.

Code speaks volumes so:

http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=rpurdie/overrides-convert

which has a script to translate poky, a commit of the translated content
and a patch to bitbake to make it use ":" instead of "_" for overrides.

I appreciate that translation like this isn't ideal, but it might just work
well enough for people to work with to allow us to move forward.

As to the actual syntax changes, one thing I would like to se is the
_append/_prepend/_remove operators actually turn into real operators.
I definitely do not want to see the same syntax for the operators as
for overrides (that is one of the most confusing things with the
current solution).

I very much want to improve things. The trouble is if we try and do too much 
at once, we'll lose the support of our users. There are several things we're
effectively trying to do:

a) Add more information to the metadata (which text *is* an override?)
b) Change the syntax to be more readable
c) Remove certain syntax/operations
d) Potentially change the behaviour of some of the syntax

In the original thread, I was mixing up several of these things. I've
tried to step back and try and just solve a) in a way we can agree on
as a first step. If we can't even do that...

Here is one suggestion:

PACKAGECONFIG_append = "foo" => PACKAGECONFIG{} .= "foo"
PACKAGECONFIG_append = " foo" => PACKAGECONFIG{} += "foo"
PACKAGECONFIG_prepend = "foo " => PACKAGECONFIG{} =. "foo"
PACKAGECONFIG_prepend = " foo " => PACKAGECONFIG{} =+ "foo"
PACKAGECONFIG_remove = "foo " => PACKAGECONFIG{} -= "foo"

How do you explain

PACKAGECONFIG{} += "foo"
vs
PACKAGECONFIG += "foo"

to a user? I suspect that is the kind of thing which will cause 
us problems.

[Just to be clear to everyone reading this, the problem is that .= and 
_append are not equivalent since the former applies to the default value, 
the latter to all possible values including overrides.]

PACKAGECONFIG_append_bar = "foo" => PACKAGECONFIG{bar} .= "foo"
PACKAGECONFIG_append_bar = " foo" => PACKAGECONFIG{bar} += "foo"
PACKAGECONFIG_prepend_bar = "foo " => PACKAGECONFIG{bar} =. "foo"
PACKAGECONFIG_prepend_bar = " foo " => PACKAGECONFIG{bar} =+ "foo"
PACKAGECONFIG_remove_bar = "foo " => PACKAGECONFIG{bar} -= "foo"

PACKAGECONFIG_append_bar_foo = "foo" => PACKAGECONFIG{bar&foo} .= "foo"
PACKAGECONFIG_append_bar_foo = " foo" => PACKAGECONFIG{bar&foo} += "foo"
PACKAGECONFIG_prepend_bar_foo = "foo " => PACKAGECONFIG{bar&foo} =. "foo"
PACKAGECONFIG_prepend_bar_foo = " foo " => PACKAGECONFIG{bar&foo} =+ "foo"
PACKAGECONFIG_remove_bar_foo = "foo " => PACKAGECONFIG{bar&foo} -= "foo"

This would not have a corresponding solution for, e.g.,
PACKAGECONFIG_bar_foo_append, but I actually think that is good.

Right, I think elsewhere I said this form was usually error prone 
and not correct. I was considering dropping it, *if* we can get to a point
where the parser can reliably detect it.

This would also allow for clearing out the pending appends for a
given override by doing:

PACKAGECONFIG{} = ""

I like this idea.

or:

PACKAGECONFIG{bar} = ""

However:

PACKAGECONFIG = "X"
PACKAGECONFIG_bar = ""

is a common usage.

And a totally different solution (and probably more controversial) would
be to add a more programmatical way to handle overrides, e.g.:

PACKAGECONFIG += "foo" if overrides("foo bar")

or:

if override("foo") && override("bar"): PACKAGECONFIG += "foo"

It doesn't help with whether the update to the variable is immediate or
deferred though.

I think these are getting too clever and steps too far beyond what
we can cope with both in bitbake and with the developers we have. I can't 
begin to think how these would work in the code :(.

As to RDEPENDS_${PN} & co, wouldn't it make sense to use RDEPENDS_pn-${PN},
since "pn-<package name>" already exists as an override? 

PN is really badly named since it is effectively the recipe name in many 
contexts. We're specifying one single thing in overrides with it rather than
a suite of packages a recipe generates. I can't see how using pn- like that
would work for the use cases we have for the package name override.

On that note,
I would like to see overrides being prefixed more (maybe even as a
requirement). Some suggestions:

pn- - Package name
df- - Distro feature
mf- - Machine feature
pf- - Package feature (can we rename/alias PACKAGECONFIG to
          PACKAGE_FEATURES while at it, for consistency?)
arch- - Architecture
dist- - Distribution

That way it would be more obvious what an override does. Especially the
use of the distro name as an override can have some unforeseen consequences
if the name is somewhat common).

We have tried to do this with newer overrides where it makes sense. It is 
a tradeoff between longer names and usability. I don't think adding something
for the package name would help. Having "distro-" and "machine-" could
help in some cases but it comes at the price of more complicated conversions
and it becomes a question of whether things become more readable. Worth thinking
about but the above list doesn't really make sense since features are in overrides.

(PACKAGECONFIG should really be RECIPE_FEATURES if we're going to talk names!)

Cheers,

Richard