On 7/16/21 4:22 AM, Richard Purdie wrote:

On Thu, 2021-07-15 at 09:26 -0500, Mark Hatle wrote:

On 7/15/21 8:56 AM, Richard Purdie wrote:

Breaking things down a bit, one thing I keep running into with our current 
codebase and metadata is that overrides are not clear. In my previous email,
the example was:

do_configure_class-native

A human can usually spot "class-native" is and "configure" or 
"configure_class-native" is not. Bitbake's parser struggles. It has huge 
internal lists including variables like x86_64 where it has to track 
whether "64" in an override.

One way of fixing this is to be explicit about overrides and use a different 
separator. See an example patch below I made to the quilt recipe using ":" 
instead to see how it looks. Personally, I think this looks like an improvement.

There are two challenges:

a) The work of migration. Do we migrate piecemeal or with a flag day? I'm not
   sure piecemeal is even possible unfortunately.

b) Some of the packaging code (at the very least) needs rewriting as it accesses
   both RDEPENDS_${PN} and puts ${PN} in OVERRIDES and accesses RDEPENDS. I'm not
   sure what else may be assuming this works.

Ya, this is all over the packaging code. Should be trivial to move to treating
it as a variable (and not override). I can help with this if you want.

I've been pondering this a bit more and my preference is to move this to use 
overrides more explicitly.

The part of the code I worked on, the override was used as an optimization to avoid:

d.getVar("RDEPENDS_%s" % package) or d.getVar("RDEPENDS")

(and similar for all of the other variables that could be package specific.)

Overrides (at least to me) are system (or at least recipe) wide settings that
allow you to 'override' key things. The package items, based on that definitely
really aren't overrides -- they're package specific variables. (This is also
why I suggested RDEPENDS[package] might actually be a better syntax.)

What I did realise is that d.getVar("RDEPENDS:${PN}") can actually split out
the override itself and apply it so I think the existing code can be made to
work just with the character change if we teach getVar how to handle it which
should be possible.

(It can of course also be moved to overrides, but that won't be as obvious to a
recipe creator, since they're not used to the package name being an 'override'.)

Whilst the instinct is to "hide" that, perhaps it could also make overrides more
obvious since people could see them in a different setting more clearly? I think
that is where I'm leaning right now.

I'm all for making override usage more clear to people. But in this ONE case, I
don't think these are really overrides -- and worse they could cause confusion
because they are ONLY overrides when packaging. So if you use the override
chunk in something that doesn't affect a package specific setting it won't do
anything. It also won't do anything on the functions (I think), as those are
run before the override is loaded.

Specific code I'm talking about BTW is:

http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/classes/package_rpm.bbclass

starting at line 287. (srcname is PN-src BTW)

then the loop starting at 328. Iterates over PACKAGES and sets the override one
at a time to the package name to get (conffiles, dirnames, summary/description,
pkgv, pkgr, pkge, license, section, description, rdepends, rrecommends,
rsuggests, rprovides, rreplaces, rconflicts, and I suspect a few others...)

Note, I could make the above changes pretty quickly and just submit this as an
implementation change in a few hours. I'd be happy to do that and look for
other 'in-place' usages of d.setVar("OVERRIDE"... to see if there are other
things like this.

This change does buy us cleaner looking metadata and ultimately, a faster
and cleaner internal bitbake that at least would use less memory. It could set
the stage to allow the defval idea I mentioned in a previous mail to happen.

It is a huge change and would need a lot of work to make it happen. Is it worth 
doing? Not sure. I'm putting it out there for discussion.

Is this something we do in parallel with master, and then flag day it? I could
easily see this work take a few weeks (if a bunch of people help).

I think we might be able to block convert much of the metadata with some kind of 
script. If that is possible, we'd have a flag day and a bunch of code tested in 
parallel before a final semi-automated conversion.

Agreed. I'd expect a 90+% success rate on a conversion.

It is also going to be tempting that "if we're breaking things, lets break lots".
I want to be mindful that tempting as that may be, if users can't convert clearly,
it will be worlds of pain. The benefit of this change is that at least in the
general case, the transition should be clear. Taking steps rather than changing the
world may therefore be a better idea...

One of those things (variable syntax) I'm not sure 'steps' make sense. If we're
changing a fundamental syntax, it really becomes a new major version of bitbake
and thus a new version of OE.

Having lots of things change at once may make things much harder for layer maintainers
than having a controlled change to handle though?

(been reading the other replies)

I don't have a great answer to this except for personal experience. I greatly
prefer [when things pass a compatibility point] to maintain multiple layers. If
we make the conversion script? available to layer maintainers, then it becomes
easier for them to transition. Yes, there is absolutely still work if they have
to maintain multiple branches.

 EXTRA_OECONF = "--with-perl='${USRBINPATH}/env perl' --with-patch=patch"
-EXTRA_OECONF_append_class-native = " --disable-nls"
+EXTRA_OECONF:append:class-native = " --disable-nls"

So operations (append) and overrides would both be delimited by ':'? It keeps
the ordering and makes it easier to read for sure, but I'm wondering if we need
to deliminate the operation differently. (No I have no suggestions, just a
thought. I don't object to the proposal at all.)

I wondered but this kind of felt neatest to me and it does work nicely with
the code backend handling it...

-RDEPENDS_${PN}-ptest = "make file sed gawk diffutils findutils ed perl \
+RDEPENDS:${PN}-ptest = "make file sed gawk diffutils findutils ed perl \
                         perl-module-filehandle perl-module-getopt-std \
                         perl-module-posix perl-module-file-temp \
                         perl-module-text-parsewords perl-module-overloading \

The above would work without code changes to the packaging class, but maybe we
really should make the change for readability. (I've often wondered if it
really should use 'flag' syntax instead, but lack of overrides make flag syntax
difficult to use.)

i.e.:

-FILES_${PN} = "${sysconfdir} ${datadir}/quilt \
+FILES[${PN}] = "${sysconfdir} ${datadir}/quilt \

or maybe something like? (This is a larger syntax change though)

-RDEPENDS_${PN} = "bash patch diffstat bzip2 util-linux less"
-RDEPENDS_${PN}_class-native = "diffstat-native patch-native bzip2-native"
+RDEPENDS[${PN}] = "bash patch diffstat bzip2 util-linux less"
+RDEPENDS[${PN}]:class-native = "diffstat-native patch-native bzip2-native"

'flags' in bitbake are really a poor cousin to the main variables and behave 
quite differently. Whether flags should have override support is a different 
question. It would hugely complicate the datastore code and I'm not sure the
above is more readable compared to the simpler ":". I've always seen flags as
supplemental data for a variable rather than anything like the above.

This change would certainly allow flags to more easily have overrides as you
avoid the multiple "_" is it a literal _ or an override deliminator. But
supporting future override of flags would be possible -- I the more I think
about this, the more I agree that is a bad idea for a first pass at this.

So I amend what I said above to say, avoid flags and my suggestion would be
(because I don't like package name as an override):

-RDEPENDS_${PN} = "bash patch diffstat bzip2 util-linux less"
-RDEPENDS_${PN}_class-native = "diffstat-native patch-native bzip2-native"
+RDEPENDS_${PN} = "bash patch diffstat bzip2 util-linux less"
+RDEPENDS_${PN}:class-native = "diffstat-native patch-native bzip2-native"

With the limited number of variables that are used by "package" implementation
today, it should be easy to teach the conversion script to NOT convert
RDEPENDS_first_second_third_fourth to RDEPENDS:first, unless first is an
operation (append, prepend, etc.. which it really shouldn't be!)

--Mark

Cheers,

Richard