On Fri, 2021-07-16 at 10:12 +0200, Nicolas Dechesne wrote:

hey!

On Thu, Jul 15, 2021 at 3:56 PM Richard Purdie <richard.purdie@...> wrote:

b) Some of the packaging code (at the very least) needs rewriting as it accesses
   both RDEPENDS_${PN} and puts ${PN} in OVERRIDES and accesses RDEPENDS. I'm not
   sure what else may be assuming this works.

Ouch.. I never realized that! I like Mark's suggestion to convert that to variables. RDEPENDS is typically
used in users' layers, so that  way they wouldn't be impacted by an override syntax change.

I have to admit I'm leaning the other way, make this explicit and hope that
it actually helps users understand overrides a little better by making it clear.

This change does buy us cleaner looking metadata and ultimately, a faster
and cleaner internal bitbake that at least would use less memory. It could set
the stage to allow the defval idea I mentioned in a previous mail to happen.

It is a huge change and would need a lot of work to make it happen. Is it worth 
doing? Not sure. I'm putting it out there for discussion.

There is no doubt that the change is a good change and the project would benefit 
from it. However I think we must worry about our users, and even look beyond the 
layer maintainers that we know (e.g. the one on this list). There are *way* more 
layer maintainers that we don't know about in all the many companies who are 
successfully using YP to build their products. I am not worried about all the main 
layers at all, since we know all their maintainers, and we know they will understand 
why we make this change, and will make the effort to support it. But I am very 
worried about the hundreds (thousands??) of layer maintainers in all the companies 
using Yocto. 

So if we do anything, I would really prefer if we find a way to do it in a good way 
*for them* , not *for us*. which means finding a way to support a soft transition. 
And of course we need to align this change with our LTS release cycle. So perhaps 
we can figure out how to support both syntaxes for some time (from one LTS to the 
next?), even if supporting both is impacting the performance. Or we can have a layer 
flag that indicates if it uses the old or the new syntax? 

If we support both forms, firstly, few of the layers you're concerned about will 
change as it will break older compatibility and they have no incentive to do so.

As I mentioned in another reply, there is a way some backwards compatibility could
be added however I do worry about the horrible corner cases that may result in.

Secondly, supporting both means we can't take advantage of any benefits the change
brings to further improve for a period of time. Do we want to commit to making this
change so that we can benefit from it in say four years time (a couple of LTS cycles 
you mention)?

I'm worried about the pain of transitioning, but I'm also very worried that unless
we figure out how we can change something, we're not going to develop beyond where
we are now.

I'm trying hard to find any way we can move forward with some of the issues people
are raising and I'm not seeing many options. I'm not seeing any proposals from anyone
else either, probably as there is so much inertia to overcome to make any change :(.

Cheers,

Richard

On Fri, Jul 16, 2021 at 10:12:57AM +0200, Nicolas Dechesne wrote:

hey!

On Thu, Jul 15, 2021 at 3:56 PM Richard Purdie <
richard.purdie@...> wrote:

Breaking things down a bit, one thing I keep running into with our current
codebase and metadata is that overrides are not clear. In my previous
email,
the example was:

do_configure_class-native

A human can usually spot "class-native" is and "configure" or
"configure_class-native" is not. Bitbake's parser struggles. It has huge
internal lists including variables like x86_64 where it has to track
whether "64" in an override.

One way of fixing this is to be explicit about overrides and use a
different
separator. See an example patch below I made to the quilt recipe using ":"
instead to see how it looks. Personally, I think this looks like an
improvement.

There are two challenges:

a) The work of migration. Do we migrate piecemeal or with a flag day? I'm
not
sure piecemeal is even possible unfortunately.

b) Some of the packaging code (at the very least) needs rewriting as it
accesses
both RDEPENDS_${PN} and puts ${PN} in OVERRIDES and accesses RDEPENDS.
I'm not
sure what else may be assuming this works.

Ouch.. I never realized that! I like Mark's suggestion to convert that to
variables. RDEPENDS is typically used in users' layers, so that way they
wouldn't be impacted by an override syntax change.

This change does buy us cleaner looking metadata and ultimately, a faster
and cleaner internal bitbake that at least would use less memory. It could
set
the stage to allow the defval idea I mentioned in a previous mail to
happen.

It is a huge change and would need a lot of work to make it happen. Is it
worth
doing? Not sure. I'm putting it out there for discussion.

There is no doubt that the change is a good change and the project would
benefit from it. However I think we must worry about our users, and even
look beyond the layer maintainers that we know (e.g. the one on this list).
There are *way* more layer maintainers that we don't know about in all the
many companies who are successfully using YP to build their products. I am
not worried about all the main layers at all, since we know all their
maintainers, and we know they will understand why we make this change, and
will make the effort to support it. But I am very worried about the
hundreds (thousands??) of layer maintainers in all the companies using
Yocto.

So if we do anything, I would really prefer if we find a way to do it in a
good way *for them* , not *for us*. which means finding a way to support a
soft transition. And of course we need to align this change with our LTS
release cycle. So perhaps we can figure out how to support both syntaxes
for some time (from one LTS to the next?), even if supporting both is
impacting the performance. Or we can have a layer flag that indicates if it
uses the old or the new syntax?

I agree.

I understand that this change would be needed to improve the metadata we
currently have, but I would like to stress that such big change in
syntax would be major PIA for some use cases.

E.g. with meta-ros https://github.com/ros/meta-ros which currently has
over 7000 recipes mostly generated from OSRF index by superflore
https://github.com/ros-infrastructure/superflore (there are a lot of
changes every few days).

Currently the work-flow is to regenerate the recipes for dunfell and
then migrate these changes to other branches as well. With different
syntax used in future LTS we would either need to teach superflore to
generate different syntax (but regenerate new recipes twice with
different syntax - and the regeneration takes quite a while) or reliable
script to migrate to new syntax (to generate with old syntax with
superflore and then migrate to new syntax before "cherry-pick" to
the next LTS branch).

So this is just one example that for some use-cases this change won't be
just one-time flag-day to migrate existing metadata, but ongoing PIA to
support multiple OE releases (e.g. dunfell LTS at the same time as the
next LTS).

Similarly for projects which do development e.g. on dunfell
branch, but maintain a master branch compatible with latest for others
to use, now if they need to spend significant time to migrate changes
from dunfell to master (with the same syntax simple git merge was often
good enough without any conflicts).

And last example could be meta-qt5 where the branches are usually
compatible with wide range of OE releases and projects tend to mismatch
them often (because they need specific version of Qt - e.g. due
licensing changes in newest - while using different OE release - e.g.
webOS OSE was using dunfell for all layers except meta-qt5 in warrior
branch to get Qt 5.12 with just small meta-qt5-compat shim layer to make
them compatible (recently migrated to meta-qt6 in
https://github.com/webosose/build-webos/commit/057477b7d8d95fee5714a468ed43defe085360a2)
With a new syntax I would need to update these older meta-qt5 branches
with the new syntax (like warrior-new-syntax branch for projects in this
situation - or even 5.12-new-syntax or <next-lts>-5.12 which could include
all compatibility tweaks directly).

And meta-qt6 will have similar issue as now it has
LAYERSERIES_COMPAT_qt6-layer = "zeus dunfell gatesgarth hardknott" in
https://code.qt.io/cgit/yocto/meta-qt6.git/tree/conf/layer.conf?h=6.2
but for future it will need 2 syntax variants.

Regards,

On Thu, 2021-07-15 at 23:48 -0400, Denys Dmytriyenko wrote:

On Thu, Jul 15, 2021 at 02:56:38PM +0100, Richard Purdie wrote:

Breaking things down a bit, one thing I keep running into with our current 
codebase and metadata is that overrides are not clear. In my previous email,
the example was:

do_configure_class-native

A human can usually spot "class-native" is and "configure" or 
"configure_class-native" is not. Bitbake's parser struggles. It has huge 
internal lists including variables like x86_64 where it has to track 
whether "64" in an override.

One way of fixing this is to be explicit about overrides and use a different 
separator. See an example patch below I made to the quilt recipe using ":" 
instead to see how it looks. Personally, I think this looks like an improvement.

There are two challenges:

a) The work of migration. Do we migrate piecemeal or with a flag day? I'm not
   sure piecemeal is even possible unfortunately.

b) Some of the packaging code (at the very least) needs rewriting as it accesses
   both RDEPENDS_${PN} and puts ${PN} in OVERRIDES and accesses RDEPENDS. I'm not
   sure what else may be assuming this works.

This change does buy us cleaner looking metadata and ultimately, a faster
and cleaner internal bitbake that at least would use less memory. It could set
the stage to allow the defval idea I mentioned in a previous mail to happen.

It is a huge change and would need a lot of work to make it happen. Is it worth 
doing? Not sure. I'm putting it out there for discussion.

It is also going to be tempting that "if we're breaking things, lets break lots".
I want to be mindful that tempting as that may be, if users can't convert clearly,
it will be worlds of pain. The benefit of this change is that at least in the
general case, the transition should be clear. Taking steps rather than changing the
world may therefore be a better idea...

The idea is interesting, but it would certainly break layers out there that
only provide a single master branch and claim compatibility with several past
releases, including dunfell, or even older. Few come to mind - meta-browser,
meta-linux-mainline, meta-etherium, etc.

This is the dilemma. We hear a lot about usability and the syntax being confusing
and hard to use/understand. If we try and improve that, we are going to have to
break compatibility at some point/level.

We've not done it for a while but I'm not sure that is a good thing and we perhaps
do need to try and improve some things.

One possibility is that if we patched older bitbake's to substitute ":" for "_" in
variables coming into the data store, it might be able to handle metadata using 
the new delimiter to some level. I'm not entirely sure you could cover every possible
situation, it might work.

There would still end up being a flag day where beyond which we couldn't progress
without relying upon the new syntax. I'm a little reluctant to look too hard at that
as writing metadata to work with datastores with two different behaviours doesn't
sound like a great idea from a testing perspective and ultimately may hurt users more
that it would help.

Cheers,

Richard

On Thu, 2021-07-15 at 09:26 -0500, Mark Hatle wrote:

On 7/15/21 8:56 AM, Richard Purdie wrote:

Breaking things down a bit, one thing I keep running into with our current 
codebase and metadata is that overrides are not clear. In my previous email,
the example was:

do_configure_class-native

A human can usually spot "class-native" is and "configure" or 
"configure_class-native" is not. Bitbake's parser struggles. It has huge 
internal lists including variables like x86_64 where it has to track 
whether "64" in an override.

One way of fixing this is to be explicit about overrides and use a different 
separator. See an example patch below I made to the quilt recipe using ":" 
instead to see how it looks. Personally, I think this looks like an improvement.

There are two challenges:

a) The work of migration. Do we migrate piecemeal or with a flag day? I'm not
   sure piecemeal is even possible unfortunately.

b) Some of the packaging code (at the very least) needs rewriting as it accesses
   both RDEPENDS_${PN} and puts ${PN} in OVERRIDES and accesses RDEPENDS. I'm not
   sure what else may be assuming this works.

Ya, this is all over the packaging code. Should be trivial to move to treating
it as a variable (and not override). I can help with this if you want.

I've been pondering this a bit more and my preference is to move this to use 
overrides more explicitly.

What I did realise is that d.getVar("RDEPENDS:${PN}") can actually split out
the override itself and apply it so I think the existing code can be made to
work just with the character change if we teach getVar how to handle it which
should be possible.

(It can of course also be moved to overrides, but that won't be as obvious to a
recipe creator, since they're not used to the package name being an 'override'.)

Whilst the instinct is to "hide" that, perhaps it could also make overrides more
obvious since people could see them in a different setting more clearly? I think
that is where I'm leaning right now.

This change does buy us cleaner looking metadata and ultimately, a faster
and cleaner internal bitbake that at least would use less memory. It could set
the stage to allow the defval idea I mentioned in a previous mail to happen.

It is a huge change and would need a lot of work to make it happen. Is it worth 
doing? Not sure. I'm putting it out there for discussion.

Is this something we do in parallel with master, and then flag day it? I could
easily see this work take a few weeks (if a bunch of people help).

I think we might be able to block convert much of the metadata with some kind of 
script. If that is possible, we'd have a flag day and a bunch of code tested in 
parallel before a final semi-automated conversion.

It is also going to be tempting that "if we're breaking things, lets break lots".
I want to be mindful that tempting as that may be, if users can't convert clearly,
it will be worlds of pain. The benefit of this change is that at least in the
general case, the transition should be clear. Taking steps rather than changing the
world may therefore be a better idea...

One of those things (variable syntax) I'm not sure 'steps' make sense. If we're
changing a fundamental syntax, it really becomes a new major version of bitbake
and thus a new version of OE.

Having lots of things change at once may make things much harder for layer maintainers
than having a controlled change to handle though?

 EXTRA_OECONF = "--with-perl='${USRBINPATH}/env perl' --with-patch=patch"
-EXTRA_OECONF_append_class-native = " --disable-nls"
+EXTRA_OECONF:append:class-native = " --disable-nls"

So operations (append) and overrides would both be delimited by ':'? It keeps
the ordering and makes it easier to read for sure, but I'm wondering if we need
to deliminate the operation differently. (No I have no suggestions, just a
thought. I don't object to the proposal at all.)

I wondered but this kind of felt neatest to me and it does work nicely with
the code backend handling it...

-RDEPENDS_${PN}-ptest = "make file sed gawk diffutils findutils ed perl \
+RDEPENDS:${PN}-ptest = "make file sed gawk diffutils findutils ed perl \
                         perl-module-filehandle perl-module-getopt-std \
                         perl-module-posix perl-module-file-temp \
                         perl-module-text-parsewords perl-module-overloading \

The above would work without code changes to the packaging class, but maybe we
really should make the change for readability. (I've often wondered if it
really should use 'flag' syntax instead, but lack of overrides make flag syntax
difficult to use.)

i.e.:

-FILES_${PN} = "${sysconfdir} ${datadir}/quilt \
+FILES[${PN}] = "${sysconfdir} ${datadir}/quilt \

or maybe something like? (This is a larger syntax change though)

-RDEPENDS_${PN} = "bash patch diffstat bzip2 util-linux less"
-RDEPENDS_${PN}_class-native = "diffstat-native patch-native bzip2-native"
+RDEPENDS[${PN}] = "bash patch diffstat bzip2 util-linux less"
+RDEPENDS[${PN}]:class-native = "diffstat-native patch-native bzip2-native"

'flags' in bitbake are really a poor cousin to the main variables and behave 
quite differently. Whether flags should have override support is a different 
question. It would hugely complicate the datastore code and I'm not sure the
above is more readable compared to the simpler ":". I've always seen flags as
supplemental data for a variable rather than anything like the above.

Cheers,

Richard

On Thu, Jul 15, 2021 at 02:56:38PM +0100, Richard Purdie wrote:

Breaking things down a bit, one thing I keep running into with our current 
codebase and metadata is that overrides are not clear. In my previous email,
the example was:

do_configure_class-native

A human can usually spot "class-native" is and "configure" or 
"configure_class-native" is not. Bitbake's parser struggles. It has huge 
internal lists including variables like x86_64 where it has to track 
whether "64" in an override.

One way of fixing this is to be explicit about overrides and use a different 
separator. See an example patch below I made to the quilt recipe using ":" 
instead to see how it looks. Personally, I think this looks like an improvement.

There are two challenges:

a) The work of migration. Do we migrate piecemeal or with a flag day? I'm not
sure piecemeal is even possible unfortunately.

b) Some of the packaging code (at the very least) needs rewriting as it accesses
both RDEPENDS_${PN} and puts ${PN} in OVERRIDES and accesses RDEPENDS. I'm not
sure what else may be assuming this works.

This change does buy us cleaner looking metadata and ultimately, a faster
and cleaner internal bitbake that at least would use less memory. It could set
the stage to allow the defval idea I mentioned in a previous mail to happen.

It is a huge change and would need a lot of work to make it happen. Is it worth 
doing? Not sure. I'm putting it out there for discussion.

It is also going to be tempting that "if we're breaking things, lets break lots".
I want to be mindful that tempting as that may be, if users can't convert clearly,
it will be worlds of pain. The benefit of this change is that at least in the
general case, the transition should be clear. Taking steps rather than changing the
world may therefore be a better idea...

The idea is interesting, but it would certainly break layers out there that
only provide a single master branch and claim compatibility with several past
releases, including dunfell, or even older. Few come to mind - meta-browser,
meta-linux-mainline, meta-etherium, etc.

Cheers,

Richard


diff --git a/meta/recipes-devtools/quilt/quilt.inc b/meta/recipes-devtools/quilt/quilt.inc
index d7ecda7aaa6..f85de384d26 100644
--- a/meta/recipes-devtools/quilt/quilt.inc
+++ b/meta/recipes-devtools/quilt/quilt.inc
@@ -14,36 +14,36 @@ SRC_URI = "${SAVANNAH_GNU_MIRROR}/quilt/quilt-${PV}.tar.gz \
file://0001-tests-Allow-different-output-from-mv.patch \
"

-SRC_URI_append_class-target = " file://gnu_patch_test_fix_target.patch"
+SRC_URI:append:class-target = " file://gnu_patch_test_fix_target.patch"

SRC_URI[md5sum] = "6800c2404a2c0598ab2eff92a636ba70"
SRC_URI[sha256sum] = "314b319a6feb13bf9d0f9ffa7ce6683b06919e734a41275087ea457cc9dc6e07"

inherit autotools-brokensep ptest

-INHIBIT_AUTOTOOLS_DEPS_class-native = "1"
-PATCHTOOL_class-native = "patch"
+INHIBIT_AUTOTOOLS_DEPS:class-native = "1"
+PATCHTOOL:class-native = "patch"

CLEANBROKEN = "1"

EXTRA_OECONF = "--with-perl='${USRBINPATH}/env perl' --with-patch=patch"
-EXTRA_OECONF_append_class-native = " --disable-nls"
+EXTRA_OECONF:append:class-native = " --disable-nls"
EXTRA_AUTORECONF += "--exclude=aclocal"

CACHED_CONFIGUREVARS += "ac_cv_path_BASH=/bin/bash ac_cv_path_COLUMN=column"

# Make sure we don't have "-w" in shebang lines: it breaks using
# "/usr/bin/env perl" as parser
-do_configure_prepend () {
+do_configure:prepend () {
find ${S} -name "*.in" -exec sed -i -e "1s,^#\!.*@PERL@ -w$,#\! @PERL@\nuse warnings;," {} \;
}

# Don't setup symlinks to host utilities, we don't need them
-do_configure_append () {
+do_configure:append () {
sed -e 's,^COMPAT_SYMLINKS.*:=.*,COMPAT_SYMLINKS :=,' -i ${S}/Makefile
}

-do_configure_class-native () {
+do_configure:class-native () {
oe_runconf
}

@@ -54,7 +54,7 @@ do_install () {
rm -rf ${D}/${datadir}/emacs
}

-do_install_append_class-native () {
+do_install:append:class-native () {
# Dummy quiltrc file for patch.bbclass
install -d ${D}${sysconfdir}/
touch ${D}${sysconfdir}/quiltrc
@@ -75,16 +75,16 @@ do_install_ptest() {

PACKAGES += "guards guards-doc"

-FILES_${PN} = "${sysconfdir} ${datadir}/quilt \
+FILES:${PN} = "${sysconfdir} ${datadir}/quilt \
${bindir}/quilt ${libdir}/quilt"
-FILES_guards = "${bindir}/guards"
-FILES_${PN}-doc = "${mandir}/man1/quilt.1 ${docdir}/${BPN}"
-FILES_guards-doc = "${mandir}/man1/guards.1"
+FILES:guards = "${bindir}/guards"
+FILES:${PN}-doc = "${mandir}/man1/quilt.1 ${docdir}/${BPN}"
+FILES:guards-doc = "${mandir}/man1/guards.1"

-RDEPENDS_${PN} = "bash patch diffstat bzip2 util-linux less"
-RDEPENDS_${PN}_class-native = "diffstat-native patch-native bzip2-native"
+RDEPENDS:${PN} = "bash patch diffstat bzip2 util-linux less"
+RDEPENDS:${PN}:class-native = "diffstat-native patch-native bzip2-native"

-RDEPENDS_${PN}-ptest = "make file sed gawk diffutils findutils ed perl \
+RDEPENDS:${PN}-ptest = "make file sed gawk diffutils findutils ed perl \
perl-module-filehandle perl-module-getopt-std \
perl-module-posix perl-module-file-temp \
perl-module-text-parsewords perl-module-overloading \

--
Regards,
Denys Dmytriyenko <denis@...>
PGP: 0x420902729A92C964 - https://denix.org/0x420902729A92C964
Fingerprint: 25FC E4A5 8A72 2F69 1186 6D76 4209 0272 9A92 C964

Hello Richard,

Em qui., 15 de jul. de 2021 às 10:56, Richard Purdie
<richard.purdie@...> escreveu:

a) The work of migration. Do we migrate piecemeal or with a flag day? I'm not
sure piecemeal is even possible unfortunately.

If we don't do as a flag day we cannot drop old code from bitbake, so
my vote goes to it and don't look back. Of course, this has some
inconvenience for layer maintainers and backward compatibility but I
believe we ought to do it sooner rather than later so we have some
time before next LTS release.

Generally, we could move existing BitBake code to a tool to help
converting existing metadata and reducing this burden.

--
Otavio Salvador O.S. Systems
http://www.ossystems.com.br http://code.ossystems.com.br
Mobile: +55 (53) 9 9981-7854 Mobile: +1 (347) 903-9750

On 7/15/21 8:56 AM, Richard Purdie wrote:

Breaking things down a bit, one thing I keep running into with our current 
codebase and metadata is that overrides are not clear. In my previous email,
the example was:

do_configure_class-native

A human can usually spot "class-native" is and "configure" or 
"configure_class-native" is not. Bitbake's parser struggles. It has huge 
internal lists including variables like x86_64 where it has to track 
whether "64" in an override.

One way of fixing this is to be explicit about overrides and use a different 
separator. See an example patch below I made to the quilt recipe using ":" 
instead to see how it looks. Personally, I think this looks like an improvement.

There are two challenges:

a) The work of migration. Do we migrate piecemeal or with a flag day? I'm not
sure piecemeal is even possible unfortunately.

b) Some of the packaging code (at the very least) needs rewriting as it accesses
both RDEPENDS_${PN} and puts ${PN} in OVERRIDES and accesses RDEPENDS. I'm not
sure what else may be assuming this works.

Ya, this is all over the packaging code. Should be trivial to move to treating
it as a variable (and not override). I can help with this if you want.

(It can of course also be moved to overrides, but that won't be as obvious to a
recipe creator, since they're not used to the package name being an 'override'.)

This change does buy us cleaner looking metadata and ultimately, a faster
and cleaner internal bitbake that at least would use less memory. It could set
the stage to allow the defval idea I mentioned in a previous mail to happen.

It is a huge change and would need a lot of work to make it happen. Is it worth 
doing? Not sure. I'm putting it out there for discussion.

Is this something we do in parallel with master, and then flag day it? I could
easily see this work take a few weeks (if a bunch of people help).

It is also going to be tempting that "if we're breaking things, lets break lots".
I want to be mindful that tempting as that may be, if users can't convert clearly,
it will be worlds of pain. The benefit of this change is that at least in the
general case, the transition should be clear. Taking steps rather than changing the
world may therefore be a better idea...

One of those things (variable syntax) I'm not sure 'steps' make sense. If we're
changing a fundamental syntax, it really becomes a new major version of bitbake
and thus a new version of OE.

(a few more comments below)

Cheers,

Richard


diff --git a/meta/recipes-devtools/quilt/quilt.inc b/meta/recipes-devtools/quilt/quilt.inc
index d7ecda7aaa6..f85de384d26 100644
--- a/meta/recipes-devtools/quilt/quilt.inc
+++ b/meta/recipes-devtools/quilt/quilt.inc
@@ -14,36 +14,36 @@ SRC_URI = "${SAVANNAH_GNU_MIRROR}/quilt/quilt-${PV}.tar.gz \
file://0001-tests-Allow-different-output-from-mv.patch \
"

-SRC_URI_append_class-target = " file://gnu_patch_test_fix_target.patch"
+SRC_URI:append:class-target = " file://gnu_patch_test_fix_target.patch"

SRC_URI[md5sum] = "6800c2404a2c0598ab2eff92a636ba70"
SRC_URI[sha256sum] = "314b319a6feb13bf9d0f9ffa7ce6683b06919e734a41275087ea457cc9dc6e07"

inherit autotools-brokensep ptest

-INHIBIT_AUTOTOOLS_DEPS_class-native = "1"
-PATCHTOOL_class-native = "patch"
+INHIBIT_AUTOTOOLS_DEPS:class-native = "1"
+PATCHTOOL:class-native = "patch"

CLEANBROKEN = "1"

EXTRA_OECONF = "--with-perl='${USRBINPATH}/env perl' --with-patch=patch"
-EXTRA_OECONF_append_class-native = " --disable-nls"
+EXTRA_OECONF:append:class-native = " --disable-nls"

So operations (append) and overrides would both be delimited by ':'? It keeps
the ordering and makes it easier to read for sure, but I'm wondering if we need
to deliminate the operation differently. (No I have no suggestions, just a
thought. I don't object to the proposal at all.)

EXTRA_AUTORECONF += "--exclude=aclocal"

CACHED_CONFIGUREVARS += "ac_cv_path_BASH=/bin/bash ac_cv_path_COLUMN=column"

# Make sure we don't have "-w" in shebang lines: it breaks using
# "/usr/bin/env perl" as parser
-do_configure_prepend () {
+do_configure:prepend () {
find ${S} -name "*.in" -exec sed -i -e "1s,^#\!.*@PERL@ -w$,#\! @PERL@\nuse warnings;," {} \;
}

# Don't setup symlinks to host utilities, we don't need them
-do_configure_append () {
+do_configure:append () {
sed -e 's,^COMPAT_SYMLINKS.*:=.*,COMPAT_SYMLINKS :=,' -i ${S}/Makefile
}

-do_configure_class-native () {
+do_configure:class-native () {
oe_runconf
}

@@ -54,7 +54,7 @@ do_install () {
rm -rf ${D}/${datadir}/emacs
}

-do_install_append_class-native () {
+do_install:append:class-native () {
# Dummy quiltrc file for patch.bbclass
install -d ${D}${sysconfdir}/
touch ${D}${sysconfdir}/quiltrc
@@ -75,16 +75,16 @@ do_install_ptest() {

PACKAGES += "guards guards-doc"

-FILES_${PN} = "${sysconfdir} ${datadir}/quilt \
+FILES:${PN} = "${sysconfdir} ${datadir}/quilt \
${bindir}/quilt ${libdir}/quilt"
-FILES_guards = "${bindir}/guards"
-FILES_${PN}-doc = "${mandir}/man1/quilt.1 ${docdir}/${BPN}"
-FILES_guards-doc = "${mandir}/man1/guards.1"
+FILES:guards = "${bindir}/guards"
+FILES:${PN}-doc = "${mandir}/man1/quilt.1 ${docdir}/${BPN}"
+FILES:guards-doc = "${mandir}/man1/guards.1"

-RDEPENDS_${PN} = "bash patch diffstat bzip2 util-linux less"
-RDEPENDS_${PN}_class-native = "diffstat-native patch-native bzip2-native"
+RDEPENDS:${PN} = "bash patch diffstat bzip2 util-linux less"
+RDEPENDS:${PN}:class-native = "diffstat-native patch-native bzip2-native"

-RDEPENDS_${PN}-ptest = "make file sed gawk diffutils findutils ed perl \
+RDEPENDS:${PN}-ptest = "make file sed gawk diffutils findutils ed perl \
perl-module-filehandle perl-module-getopt-std \
perl-module-posix perl-module-file-temp \
perl-module-text-parsewords perl-module-overloading \

The above would work without code changes to the packaging class, but maybe we
really should make the change for readability. (I've often wondered if it
really should use 'flag' syntax instead, but lack of overrides make flag syntax
difficult to use.)

i.e.:

-FILES_${PN} = "${sysconfdir} ${datadir}/quilt \
+FILES[${PN}] = "${sysconfdir} ${datadir}/quilt \

or maybe something like? (This is a larger syntax change though)

-RDEPENDS_${PN} = "bash patch diffstat bzip2 util-linux less"
-RDEPENDS_${PN}_class-native = "diffstat-native patch-native bzip2-native"
+RDEPENDS[${PN}] = "bash patch diffstat bzip2 util-linux less"
+RDEPENDS[${PN}]:class-native = "diffstat-native patch-native bzip2-native"

--Mark

Breaking things down a bit, one thing I keep running into with our current 
codebase and metadata is that overrides are not clear. In my previous email,
the example was:

do_configure_class-native

A human can usually spot "class-native" is and "configure" or 
"configure_class-native" is not. Bitbake's parser struggles. It has huge 
internal lists including variables like x86_64 where it has to track 
whether "64" in an override.

One way of fixing this is to be explicit about overrides and use a different 
separator. See an example patch below I made to the quilt recipe using ":" 
instead to see how it looks. Personally, I think this looks like an improvement.

There are two challenges:

a) The work of migration. Do we migrate piecemeal or with a flag day? I'm not
sure piecemeal is even possible unfortunately.

b) Some of the packaging code (at the very least) needs rewriting as it accesses
both RDEPENDS_${PN} and puts ${PN} in OVERRIDES and accesses RDEPENDS. I'm not
sure what else may be assuming this works.

This change does buy us cleaner looking metadata and ultimately, a faster
and cleaner internal bitbake that at least would use less memory. It could set
the stage to allow the defval idea I mentioned in a previous mail to happen.

It is a huge change and would need a lot of work to make it happen. Is it worth 
doing? Not sure. I'm putting it out there for discussion.

It is also going to be tempting that "if we're breaking things, lets break lots".
I want to be mindful that tempting as that may be, if users can't convert clearly,
it will be worlds of pain. The benefit of this change is that at least in the
general case, the transition should be clear. Taking steps rather than changing the
world may therefore be a better idea...

Cheers,

Richard


diff --git a/meta/recipes-devtools/quilt/quilt.inc b/meta/recipes-devtools/quilt/quilt.inc
index d7ecda7aaa6..f85de384d26 100644
--- a/meta/recipes-devtools/quilt/quilt.inc
+++ b/meta/recipes-devtools/quilt/quilt.inc
@@ -14,36 +14,36 @@ SRC_URI = "${SAVANNAH_GNU_MIRROR}/quilt/quilt-${PV}.tar.gz \
file://0001-tests-Allow-different-output-from-mv.patch \
"

-SRC_URI_append_class-target = " file://gnu_patch_test_fix_target.patch"
+SRC_URI:append:class-target = " file://gnu_patch_test_fix_target.patch"

SRC_URI[md5sum] = "6800c2404a2c0598ab2eff92a636ba70"
SRC_URI[sha256sum] = "314b319a6feb13bf9d0f9ffa7ce6683b06919e734a41275087ea457cc9dc6e07"

inherit autotools-brokensep ptest

-INHIBIT_AUTOTOOLS_DEPS_class-native = "1"
-PATCHTOOL_class-native = "patch"
+INHIBIT_AUTOTOOLS_DEPS:class-native = "1"
+PATCHTOOL:class-native = "patch"

CLEANBROKEN = "1"

EXTRA_OECONF = "--with-perl='${USRBINPATH}/env perl' --with-patch=patch"
-EXTRA_OECONF_append_class-native = " --disable-nls"
+EXTRA_OECONF:append:class-native = " --disable-nls"
EXTRA_AUTORECONF += "--exclude=aclocal"

CACHED_CONFIGUREVARS += "ac_cv_path_BASH=/bin/bash ac_cv_path_COLUMN=column"

# Make sure we don't have "-w" in shebang lines: it breaks using
# "/usr/bin/env perl" as parser
-do_configure_prepend () {
+do_configure:prepend () {
find ${S} -name "*.in" -exec sed -i -e "1s,^#\!.*@PERL@ -w$,#\! @PERL@\nuse warnings;," {} \;
}

# Don't setup symlinks to host utilities, we don't need them
-do_configure_append () {
+do_configure:append () {
sed -e 's,^COMPAT_SYMLINKS.*:=.*,COMPAT_SYMLINKS :=,' -i ${S}/Makefile
}

-do_configure_class-native () {
+do_configure:class-native () {
oe_runconf
}

@@ -54,7 +54,7 @@ do_install () {
rm -rf ${D}/${datadir}/emacs
}

-do_install_append_class-native () {
+do_install:append:class-native () {
# Dummy quiltrc file for patch.bbclass
install -d ${D}${sysconfdir}/
touch ${D}${sysconfdir}/quiltrc
@@ -75,16 +75,16 @@ do_install_ptest() {

PACKAGES += "guards guards-doc"

-FILES_${PN} = "${sysconfdir} ${datadir}/quilt \
+FILES:${PN} = "${sysconfdir} ${datadir}/quilt \
${bindir}/quilt ${libdir}/quilt"
-FILES_guards = "${bindir}/guards"
-FILES_${PN}-doc = "${mandir}/man1/quilt.1 ${docdir}/${BPN}"
-FILES_guards-doc = "${mandir}/man1/guards.1"
+FILES:guards = "${bindir}/guards"
+FILES:${PN}-doc = "${mandir}/man1/quilt.1 ${docdir}/${BPN}"
+FILES:guards-doc = "${mandir}/man1/guards.1"

-RDEPENDS_${PN} = "bash patch diffstat bzip2 util-linux less"
-RDEPENDS_${PN}_class-native = "diffstat-native patch-native bzip2-native"
+RDEPENDS:${PN} = "bash patch diffstat bzip2 util-linux less"
+RDEPENDS:${PN}:class-native = "diffstat-native patch-native bzip2-native"

-RDEPENDS_${PN}-ptest = "make file sed gawk diffutils findutils ed perl \
+RDEPENDS:${PN}-ptest = "make file sed gawk diffutils findutils ed perl \
perl-module-filehandle perl-module-getopt-std \
perl-module-posix perl-module-file-temp \
perl-module-text-parsewords perl-module-overloading \

I've been quiet but have been thinking a lot about the syntax discussions.
I believe we do need to simplify things, at least so the majority of recipes
have simpler assignment syntax that is understandable to new users. There was
a recent question about whether it was preferred to use += or _append and it
is hard to answer. My worries about setting PACKAGECONFIG with ??=, ?= or =
are also still an open problem too.

One potential approach is we'd just map += and =+ to append/prepend operations 
behind the scenes and all would be good. The trouble is there is a difference
in behaviour. _append and _prepend apply to *any* variable value including an 
override value whilst += and =+ only apply to the "default" value and not any 
override value.

This behaviour is taken advantage of by the metadata. The first example you find 
in a build is quilt-native:

do_configure_prepend () {
A
}

do_configure_append () {
B
}

do_configure_class-native () {
C
}

which wants it's prepend/appends always applied but wants to override the
default value in the native case.

As mentioned on IRC, I've played with the idea of a couple of new override
levels, "defval" and "weakval" where:

A = "X"

would become A_defval within the datastore and then += and =+ can become 
prepends and appends to the defval. Sadly if we do that, we need to be able
to know that do_configure_class-native is an override rather than creating
do_configure_class-native_defval and we don't have a way to know that for
example, "configure" isn't an override to the variable "do" yet class-native
is.

Working backwards, I think the question is what we want this syntax to look
like. How do we say "append this to all variants of a variable" vs. "append 
this only to the default value" or "append this only to the overriden value"?
Can we offer a way to clear a certain subset of appends?

We have an immensely powerful syntax, the question is whether we can find a
way to present it which makes sense to more people...

If we did have clear marking of overrides, that could go some way to establishing
a defval override level, which imports our current syntax into the overrides model
which could then open up possibilities like allowing changing or clearing specific 
overrides explicitly and may allow dropping of the different assignment variants 
(ignoring flags :/).

Not sure quite where I'm going with this but wanted to document it a little and 
share...

Cheers,

Richard

Greetings,

On 7/7/21 5:22 PM, Mark Hatle wrote:

In the variables table:

PNBLACKLIST, I'd prefer PNEXCLUDELIST, or PNBLOCKLIST. (I think exclude better
captures what it does.)

Confused, where it says current name is whitelist, suggested rename is
'excludelist', isn't that the exact opposite?

SSTATE_DUPWHITELIST, this is really a list of things to ignore. So my
suggestion is SSTATE_DUPIGNORELIST.

CVE_CHECK_PN_WHITELIST, I think there is a typo in the table for the replacement
(missing an L). But like the above, I think ignore is a better word then skip
here. So my suggestion would be CVE_CHECK_PN_IGNORELIST.

Instead of "includelist" and "excludelist", what about just "includes"
and "excludes"? This terminology is already used in rsync's manual (for
example), and it seems lighter and clear enough.

For example, we could replace "CECC_USER_CLASS_BL" by
"ICECC_USER_CLASS_EXCLUDES" and "ICECC_USER_PACKAGE_BL"  by 
"ICECC_USER_PACKAGE_EXCLUDES".

Any thoughts? I mentioned this on the wiki page
(https://wiki.yoctoproject.org/wiki/Inclusive_language), but don't
hesitate to revert my edits.

Cheers,

Michael.

--
Michael Opdenacker, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

On Wed, Jul 7, 2021 at 7:16 PM Richard Purdie
<richard.purdie@...> wrote:

On Wed, 2021-07-07 at 10:22 -0500, Mark Hatle wrote:

On 7/7/21 7:30 AM, Armin Kuster wrote:

Hello all,

The Yocto Project TSC has created a wiki page to start making notes of
offending names and possible replacements. At some point, this wiki page
should include a plan on moving forward. Our hope is this will help
streamline the process when folks send patches to replace offending
language.
Everyone is welcome to find and make note of any offending names in the
tables on the wiki to help scope out this effort.

https://wiki.yoctoproject.org/wiki/Inclusive_language

In the variables table:

PNBLACKLIST, I'd prefer PNEXCLUDELIST, or PNBLOCKLIST. (I think exclude better
captures what it does.)

I'd note that it does actually raise "SkipRecipe" on the recipes in question so
SKIP may actually be more consistent and help with understanding in other code.

+1
I find the whole issue almost unbelievable but hey...
A.A.

Cheers,

Richard

On Wed, 2021-07-07 at 10:22 -0500, Mark Hatle wrote:

On 7/7/21 7:30 AM, Armin Kuster wrote:

Hello all,

The Yocto Project TSC has created a wiki page to start making notes of
offending names and possible replacements. At some point, this wiki page
should include a plan on moving forward.  Our hope is this will help
streamline the process when folks send patches to replace offending
language.
Everyone is welcome to find and make note of any offending names in the
tables on the wiki to help scope out this effort.

https://wiki.yoctoproject.org/wiki/Inclusive_language

In the variables table:

PNBLACKLIST, I'd prefer PNEXCLUDELIST, or PNBLOCKLIST. (I think exclude better
captures what it does.)

I'd note that it does actually raise "SkipRecipe" on the recipes in question so
SKIP may actually be more consistent and help with understanding in other code.

Cheers,

Richard

On 7/7/21 7:30 AM, Armin Kuster wrote:

Hello all,

The Yocto Project TSC has created a wiki page to start making notes of
offending names and possible replacements. At some point, this wiki page
should include a plan on moving forward.  Our hope is this will help
streamline the process when folks send patches to replace offending
language.
Everyone is welcome to find and make note of any offending names in the
tables on the wiki to help scope out this effort.

https://wiki.yoctoproject.org/wiki/Inclusive_language

In the variables table:

PNBLACKLIST, I'd prefer PNEXCLUDELIST, or PNBLOCKLIST. (I think exclude better
captures what it does.)

Confused, where it says current name is whitelist, suggested rename is
'excludelist', isn't that the exact opposite?

SSTATE_DUPWHITELIST, this is really a list of things to ignore. So my
suggestion is SSTATE_DUPIGNORELIST.

CVE_CHECK_PN_WHITELIST, I think there is a typo in the table for the replacement
(missing an L). But like the above, I think ignore is a better word then skip
here. So my suggestion would be CVE_CHECK_PN_IGNORELIST.

--Mark

see
https://lists.openembedded.org/g/openembedded-architecture/topic/inclusive_language_summary/75821819 
for background.

regards,
Armin
On behalf of The Yocto Project TSC.

Hello all,

The Yocto Project TSC has created a wiki page to start making notes of
offending names and possible replacements. At some point, this wiki page
should include a plan on moving forward.  Our hope is this will help
streamline the process when folks send patches to replace offending
language.
Everyone is welcome to find and make note of any offending names in the
tables on the wiki to help scope out this effort.

https://wiki.yoctoproject.org/wiki/Inclusive_language

see
https://lists.openembedded.org/g/openembedded-architecture/topic/inclusive_language_summary/75821819 
for background.

regards,
Armin
On behalf of The Yocto Project TSC.

Hello,
after a few days I've discovered the fallouts of my chat with RP.

Just to be precise the layer is   meta-handheld  and the patch uncovering the issue is:

https://git.openembedded.org/meta-handheld/commit/?id=4ef8f82e5db28f50901ce87f7ce786675aee6adf

Note that it was a lightheaded try to fix ubi build failures.
Yes, because of the inflaction in the years nowadays only core-image-minimal fits in the few PEBs.

Being there are pending pull requests I could just revert that patch (plus revert in meta-initramfs as well) and live in peace
I am really tempted to do so :)

Thanks to all for the reviews

FWIW my 2 cents are that we have enough operators.

Cheers
A.A.


P.S.
I think the easier fix is to have IMAGE_FSTYPES defined in collie.conf (MACHINE.conf) but I am sure there are more BSP around grouping the settings in an .inc so no, it's not just about this special case.

On Wed, Jun 16, 2021 at 2:22 AM Phil Blundell via
lists.openembedded.org <pb=pbcl.net@...> wrote:

On Tue, Jun 15, 2021 at 10:59:40PM +0100, Richard Purdie wrote:

It certainly could be a solution to the problem but I worry that having
variables that users should touch and not touch and variables shadowing
other variables is going to get complex rather quickly, even when you just
consider how you document or "enforce" it. Do we want to encourage what
could become many more intermiediate variables? It does sound like we're
effectively starting to write class "functions" with a list of variables
they operate on which may or may not be a good thing depending on how
common it became.

Yes, that's a fair point.  As you say, I kind of have the sense that our
classes are drifting towards being more like classes in a "real" programming
language, complete with an actual API, and possibly we ought to start
thinking about them more in those terms.  As in, "if you include image.bbclass,
this is the set of variables it takes as input, and you need to give them
appropriate values".

From descriptions from others it sounds like people have creatively worked
around this kind of issue with stronger overrides which I'd not considered
and would also work, it is just a little ugly in a different way. I'd consider
use of the forcevariable override in oe-core a metadata failing (much like
I view _remove similarly in core).

I agree.  Although it's true that _forcevariable probably would fix this
particular issue I think there's a fair chance that it will just end up creating
the opposite problem at some point in the future.  I don't honestly think there
is any situation where using _forcevariable in oe-core itself would be the
right thing to do.

I'm not sure I understand this. Why is using the _forcevariable
override somehow worse than inventing a new assignment operator to do
the same thing?

If we want to make a rule that nothing in oe-core should ever
forcefully override a setting provided by a BSP then that's OK... but
then isn't that an argument that a new assignment operator to
forcefully assign to variables shouldn't be used in oe-core?

I suppose the other obvious tactical answer to the problem at hand is for

meta-zaurus to stop using an override at all for what it's doing.  Although
possibly not quite so elegant, there's no reason that it couldn't do:

IMAGE_FSTYPES = "ext3"
# nb, include not require...
include conf/machine/zaurus/${MACHINE}-special.conf

and then in conf/machine/zaurus/collie-special.conf it could just set
IMAGE_FSTYPES = "squashfs"

directly without needing any kind of override.  That said I don't entirely
understand why meta-zaurus doesn't already have a collie.conf (assuming
MACHINE=collie) where it could be doing this...

Rewriting the BSP would work but it's sidestepping the issue and
doesn't really scale well. There will always be BSP layers which need
to support a collection of machines and the obvious way to do that is
to use default values and machine specific overrides within the BSP
layer. Making a rule that BSP layers can't use overrides for certain
variables will cause confusion and frustration.

I think the solution should be either that oe-core doesn't rely on
modifying variables set by the BSP (ie the initramfs images should be
able to signal to the image.bbclass that they want to use
INITRAMFS_FSTYPES via some other method than modifying IMAGE_FSTYPES)
or that we agree that oe-core can forcefully override the BSP
variables in certain well defined cases (and then the _forcevariable
override appears to be a good solution).

I think the initramfs issue is just an example. For this particular issue, we can come up with several different ways to solve it.

The point is, this example demonstrates a general problem, that is, we currently lack the ability to let the recipe make the final call on variable setting. I've stated this in detail in another email. Instead of repeating the points there, I'd like to provide another way to look at this problem.

We all know that we have feature_check.bbclass, which allow us to do DISTRO_FEATURES checking, etc. Why is this bbclass working well? Because DISTRO_FEATURES must be set in conf files. Now we have a mechanism for recipes to say that if some distro feature is not satisfied, I don't build. What if a recipe wants to say if some variable value is not satisfied, I don't build? We can't do this is some var_check.bbclass, because we can set var in recipes and we can set it in many different ways. This could only be achieved at bitbake level. The '!=!' is a good way to do this.

The above reason also put a strong restriction on the new '!=!' operator, that is, it MUST be limited to be used in recipes only.

P.S.
We have to be clear that _forcevariable is also only an override, it works because it's the last one in OVERRIDES.
But _forcevariable does not beat _append, it also does not beat _remove. So even if we use it in recipes in oe-core, which we should not, it still does not solve the general problem.

Best Regards,
Chen Qi

On Wed, Jun 16, 2021 at 11:26 AM Phil Blundell <pb@...> wrote:

On Wed, Jun 16, 2021 at 11:05:06AM -0700, Andre McCurdy wrote:

I'm not sure I understand this. Why is using the _forcevariable
override somehow worse than inventing a new assignment operator to do
the same thing?

Because _forcevariable isn't positional. It means "no matter what
else appears in the metadata, either before or after this assignment,
I want the variable to end up with this value". I don't think that's
ever an appropriate assertion for something in oe-core to be making.

Obviously if you have two _forcevariable overrides on the same
variable then one of them is going to lose so to that extent it's
making an impossible promise. But other than that, it just doesn't
play very well with others, not least because its effects can't
be undone. Which is more or less the original problem that we
started with: all we've done is take it up a notch, and this is
exactly the kind of arms race that I mentioned in my first email
on this subject.

Richard's !=! or whatever we want to call it at least has the merit
that it overrides what has gone before but can itself be overridden
by what follows.

I don't think users generally have a good grasp of the concept of
"what has gone before". They expect = to override ?= and _myoverride
to override = and if the ordering of those assignments can change
their effect then it's a bug rather than a feature.

I'm sure the advanced users reading this thread will be able to dream
up situations where an assignment operator which forcefully overrides
all overrides but can then be overridden again with another assignment
can be used in some creative hack somewhere... but it's not making OE
any better for the average user.

On Wed, Jun 16, 2021 at 11:05:06AM -0700, Andre McCurdy wrote:

I'm not sure I understand this. Why is using the _forcevariable
override somehow worse than inventing a new assignment operator to do
the same thing?

Because _forcevariable isn't positional. It means "no matter what
else appears in the metadata, either before or after this assignment,
I want the variable to end up with this value". I don't think that's
ever an appropriate assertion for something in oe-core to be making.

Obviously if you have two _forcevariable overrides on the same
variable then one of them is going to lose so to that extent it's
making an impossible promise. But other than that, it just doesn't
play very well with others, not least because its effects can't
be undone. Which is more or less the original problem that we
started with: all we've done is take it up a notch, and this is
exactly the kind of arms race that I mentioned in my first email
on this subject.

Richard's !=! or whatever we want to call it at least has the merit
that it overrides what has gone before but can itself be overridden
by what follows.

p.

On Wed, Jun 16, 2021 at 2:22 AM Phil Blundell via
lists.openembedded.org <pb=pbcl.net@...> wrote:

On Tue, Jun 15, 2021 at 10:59:40PM +0100, Richard Purdie wrote:

It certainly could be a solution to the problem but I worry that having
variables that users should touch and not touch and variables shadowing
other variables is going to get complex rather quickly, even when you just
consider how you document or "enforce" it. Do we want to encourage what
could become many more intermiediate variables? It does sound like we're
effectively starting to write class "functions" with a list of variables
they operate on which may or may not be a good thing depending on how
common it became.

Yes, that's a fair point. As you say, I kind of have the sense that our
classes are drifting towards being more like classes in a "real" programming
language, complete with an actual API, and possibly we ought to start
thinking about them more in those terms. As in, "if you include image.bbclass,
this is the set of variables it takes as input, and you need to give them
appropriate values".

From descriptions from others it sounds like people have creatively worked
around this kind of issue with stronger overrides which I'd not considered
and would also work, it is just a little ugly in a different way. I'd consider
use of the forcevariable override in oe-core a metadata failing (much like
I view _remove similarly in core).

I agree. Although it's true that _forcevariable probably would fix this
particular issue I think there's a fair chance that it will just end up creating
the opposite problem at some point in the future. I don't honestly think there
is any situation where using _forcevariable in oe-core itself would be the
right thing to do.

I'm not sure I understand this. Why is using the _forcevariable
override somehow worse than inventing a new assignment operator to do
the same thing?

If we want to make a rule that nothing in oe-core should ever
forcefully override a setting provided by a BSP then that's OK... but
then isn't that an argument that a new assignment operator to
forcefully assign to variables shouldn't be used in oe-core?

I suppose the other obvious tactical answer to the problem at hand is for

meta-zaurus to stop using an override at all for what it's doing. Although
possibly not quite so elegant, there's no reason that it couldn't do:

IMAGE_FSTYPES = "ext3"
# nb, include not require...
include conf/machine/zaurus/${MACHINE}-special.conf

and then in conf/machine/zaurus/collie-special.conf it could just set
IMAGE_FSTYPES = "squashfs"

directly without needing any kind of override. That said I don't entirely
understand why meta-zaurus doesn't already have a collie.conf (assuming
MACHINE=collie) where it could be doing this...

Rewriting the BSP would work but it's sidestepping the issue and
doesn't really scale well. There will always be BSP layers which need
to support a collection of machines and the obvious way to do that is
to use default values and machine specific overrides within the BSP
layer. Making a rule that BSP layers can't use overrides for certain
variables will cause confusion and frustration.

I think the solution should be either that oe-core doesn't rely on
modifying variables set by the BSP (ie the initramfs images should be
able to signal to the image.bbclass that they want to use
INITRAMFS_FSTYPES via some other method than modifying IMAGE_FSTYPES)
or that we agree that oe-core can forcefully override the BSP
variables in certain well defined cases (and then the _forcevariable
override appears to be a good solution).