Hi All,

I'm a little saddened/annoyed/frustrated that we're less that a week before
feature freeze and yet there is still no mechanism in bitbake to handle variable
deprecation/renaming. There has also been no real discussion about how this
could/should work. People have sent out odd patches changing one specific
variable but we still don't have any core well defined plan about how to do
this. I was recently asked for a "specification" on how this should work. This
just frustrates me even more since part of the work involved in this task was to
develop such a thing.

Change doesn't "just happen", it needs planning and thinking through the details
and this simply isn't happening here. A specification doesn't exist unless
someone proposes one.

I now face some choices and all of them appear bad. I can refuse the inclusive
language changes for the next LTS. This makes me and the project look bad as
we've talked about and planned them. I can try and make them happen and rush
them through. This creates a load of work I could do without and once again
leaves me supporting any fallout. It also risks alienating users and puts the
release at risk. I could just merge adhoc changes without plan. I suspect I
would get rather negative feedback about doing that. At this point I'm not sure
there is a solution where we can "win". I do not like being forced into this
position.

In the interests of trying to be productive and move forward, let me state some
assumptions:

a) We need to tell users if an old variable is used.

b) We cannot support automatic/dynamic conversion of the old usage to a new one.
This sounds like a nice idea until you find a case where both old and new syntax
is used and then how do you combine them?

c) There are two classes of old variable names, environment ones in bitbake and
standard variables.

d) It is probably helpful to show users the new variable name where we know it.

e) Due to b), we're not going to show warnings, they need to be errors.

I gave in and started to look at what we can do. I'm going to take it as a given
that we add warnonce and erroronce log levels and message handling to bitbake
and there are patches for that in the branch below.

Beyond that there are several ways we can handle things:

a) Add a RecipeParsed event handler and handle things in the metadata in OE-
Core. This keeps bitbake clean and is simple. It wouldn't support file/line
data. The overhead isn't great. It also doesn't help any non-OE usage of bitbake
and would mean knowledge of bitbake vars in the metadata.

b) Similar to a) but put the code into bitbake in the ast.py's finalize()
function. Scott had a version of this and I've tweaked it a little to put the
bitbake variable renames into bitbake and merge in OE additions to the list:

https://git.yoctoproject.org/poky-contrib/commit/?h=rpurdie/bitbake-renamevars&id=9e8c7e36cb6c9e06184f94d8670619d3361a0ffc

(needs a fragment of the patch below for bitbake_renamed_vars)

c) We could hook setVar and setVarFlag. This adds file/line info to anything set
by the core configuration which may be helpful to users.


https://git.yoctoproject.org/poky-contrib/commit/?h=rpurdie/t222&id=1cf65b3ce501e7927c6d319b7a4b422f44542a8e

This is more complex but adds some extra info and is possibly lower overhead. It
has the downside that parsing order matters, you'd need the renamed variables to
be early in the parsing. The order issues may mean it can miss things in
layer.conf files. The line/file info would not be shown for recipe level
settings since variable tracking is turned off for the main recipe parsing due
to the overhead it adds.


In all the above cases there are still the issues that:

1) showing errors doesn't make bitbake exit or stop the build
2) It won't handle variables from the shell environment. This will likely need
special code in bitbake.
3) There are probably some variables which are removed and no longer
used/supported which we should also tell the user about (show a message instead
of a rename?)
4) The current code doesn't handle overridden variables. This is easier to add
to c) but something for b) should be possible.

I had been wondering about c) above and keeping overhead down but I think we'll
just have to go back to b) and try and work through the issues above. I worry
that stopping the builds on error in particular is going to be problematic. I
felt I should at least share some of the complexities of this with people so
that if it doesn't end up happening, the complexity of the issue is at least
more visible.

Cheers,

Richard

On Tue, 2022-02-15 at 12:08 +0000, Richard Purdie via lists.openembedded.org
wrote:

In all the above cases there are still the issues that:

1) showing errors doesn't make bitbake exit or stop the build
2) It won't handle variables from the shell environment. This will likely need
special code in bitbake.
3) There are probably some variables which are removed and no longer
used/supported which we should also tell the user about (show a message instead
of a rename?)
4) The current code doesn't handle overridden variables. This is easier to add
to c) but something for b) should be possible.

I had been wondering about c) above and keeping overhead down but I think we'll
just have to go back to b) and try and work through the issues above. I worry
that stopping the builds on error in particular is going to be problematic. I
felt I should at least share some of the complexities of this with people so
that if it doesn't end up happening, the complexity of the issue is at least
more visible.

Let me follow up on where things are now at. I worked on the bitbake level
rename support and we have a patch which resolves some of the issues above. 2)
is fixed, 4) partially works and may still need tweaking. 1) was fixed but I
think may have regressed again as the autobuilder didn't stop builds the way I
expected it to. 3) still isn't done.

Joshua was able to fix the erroronce/warnonce log implementation, thanks.

Thanks to patches from Saul and Scott we have:

* a conversion script in master-next which converts metadata to match renames
* patches for bitbake and oe-core to transition to several of the new names

I was able to get the changes in master-next to run on the autobuilder with
unconverted layers being the failure cases.

The remaining things I'm aware of that need to be done are:

a) Resolve BB_DISKMON_DIRS in bitbake (last remaining bitbake rename)
b) Add some mechanism to show an error about now unused variables
c) Check builds really stop at parsing if errors are shown
d) Tweak the code for checking if overridden versions of variables are set
e) bump bitbake version and change the OE-Core minimum version requirement
f) consider changing the layer compatibility string to match for this
g) Handle ICECC_USER*/ICECC_SYSTEM* changes
h) Do something with the WHITELIST_(ANY LICENSE) variable
i) Scan over OE-Core for whitelist/blacklist variable name usage in python code
and propose patches for the issues
j) translate the names in the docs (the script should handle that)
k) document the conversion script and write the migration guide entry

We need to decide when and how to merge this, unfortunately there will be a flag
day for layers with it. I think we do need more of the issues above fixed before
we can do so, probably needing the majority of the items higher in the list to
be fixed first. Some of the later ones probably don't block merging. We do only
really want to have users need to look at the conversion script once for their
layer, not multiple passes.

At this point I'd like to take incremental patches against those in master-next
but for the final changes I will squash patches as appropriate.

Cheers,

Richard

On Thu, 2022-02-17 at 10:17 +0000, Richard Purdie via lists.openembedded.org
wrote:

On Tue, 2022-02-15 at 12:08 +0000, Richard Purdie via lists.openembedded.org
wrote:

In all the above cases there are still the issues that:

1) showing errors doesn't make bitbake exit or stop the build
2) It won't handle variables from the shell environment. This will likely need
special code in bitbake.
3) There are probably some variables which are removed and no longer
used/supported which we should also tell the user about (show a message instead
of a rename?)
4) The current code doesn't handle overridden variables. This is easier to add
to c) but something for b) should be possible.

I had been wondering about c) above and keeping overhead down but I think we'll
just have to go back to b) and try and work through the issues above. I worry
that stopping the builds on error in particular is going to be problematic. I
felt I should at least share some of the complexities of this with people so
that if it doesn't end up happening, the complexity of the issue is at least
more visible.

Let me follow up on where things are now at. I worked on the bitbake level
rename support and we have a patch which resolves some of the issues above. 2)
is fixed, 4) partially works and may still need tweaking. 1) was fixed but I
think may have regressed again as the autobuilder didn't stop builds the way I
expected it to. 3) still isn't done.

Joshua was able to fix the erroronce/warnonce log implementation, thanks.

Thanks to patches from Saul and Scott we have:

* a conversion script in master-next which converts metadata to match renames
* patches for bitbake and oe-core to transition to several of the new names

I was able to get the changes in master-next to run on the autobuilder with
unconverted layers being the failure cases.

The remaining things I'm aware of that need to be done are:

a) Resolve BB_DISKMON_DIRS in bitbake (last remaining bitbake rename)
b) Add some mechanism to show an error about now unused variables
c) Check builds really stop at parsing if errors are shown
d) Tweak the code for checking if overridden versions of variables are set
e) bump bitbake version and change the OE-Core minimum version requirement
f) consider changing the layer compatibility string to match for this
g) Handle ICECC_USER*/ICECC_SYSTEM* changes
h) Do something with the WHITELIST_(ANY LICENSE) variable
i) Scan over OE-Core for whitelist/blacklist variable name usage in python code
and propose patches for the issues
j) translate the names in the docs (the script should handle that)
k) document the conversion script and write the migration guide entry

b), d), e), f), g) are now in master-next but will need debugging and impact
other layers.

I think a), c) and h) probably block merging, the remainder can follow post
merge for core.

Cheers,

Richard

On 2/17/22 04:32, Richard Purdie wrote:

On Thu, 2022-02-17 at 10:17 +0000, Richard Purdie via lists.openembedded.org
wrote:

On Tue, 2022-02-15 at 12:08 +0000, Richard Purdie via lists.openembedded.org
wrote:

In all the above cases there are still the issues that:

1) showing errors doesn't make bitbake exit or stop the build
2) It won't handle variables from the shell environment. This will likely need
special code in bitbake.
3) There are probably some variables which are removed and no longer
used/supported which we should also tell the user about (show a message instead
of a rename?)
4) The current code doesn't handle overridden variables. This is easier to add
to c) but something for b) should be possible.

I had been wondering about c) above and keeping overhead down but I think we'll
just have to go back to b) and try and work through the issues above. I worry
that stopping the builds on error in particular is going to be problematic. I
felt I should at least share some of the complexities of this with people so
that if it doesn't end up happening, the complexity of the issue is at least
more visible.

Let me follow up on where things are now at. I worked on the bitbake level
rename support and we have a patch which resolves some of the issues above. 2)
is fixed, 4) partially works and may still need tweaking. 1) was fixed but I
think may have regressed again as the autobuilder didn't stop builds the way I
expected it to. 3) still isn't done.

Joshua was able to fix the erroronce/warnonce log implementation, thanks.

Thanks to patches from Saul and Scott we have:

* a conversion script in master-next which converts metadata to match renames
* patches for bitbake and oe-core to transition to several of the new names

I was able to get the changes in master-next to run on the autobuilder with
unconverted layers being the failure cases.

The remaining things I'm aware of that need to be done are:

a) Resolve BB_DISKMON_DIRS in bitbake (last remaining bitbake rename)
b) Add some mechanism to show an error about now unused variables
c) Check builds really stop at parsing if errors are shown
d) Tweak the code for checking if overridden versions of variables are set
e) bump bitbake version and change the OE-Core minimum version requirement
f) consider changing the layer compatibility string to match for this
g) Handle ICECC_USER*/ICECC_SYSTEM* changes
h) Do something with the WHITELIST_(ANY LICENSE) variable
i) Scan over OE-Core for whitelist/blacklist variable name usage in python code
and propose patches for the issues
j) translate the names in the docs (the script should handle that)
k) document the conversion script and write the migration guide entry

b), d), e), f), g) are now in master-next but will need debugging and impact
other layers.

Thank RP for all that you have done, I will have a patch to master-next for the script shortly.

I think a), c) and h) probably block merging, the remainder can follow post
merge for core.

I will start looking at the WHITELIST_<license> changes and review the old emails.

Sau!

Cheers,
Richard

--
Sau!

On Thu, 17 Feb 2022, Richard Purdie wrote:

On Thu, 2022-02-17 at 10:17 +0000, Richard Purdie via lists.openembedded.org
wrote:

On Tue, 2022-02-15 at 12:08 +0000, Richard Purdie via lists.openembedded.org
wrote:

In all the above cases there are still the issues that:

1) showing errors doesn't make bitbake exit or stop the build
2) It won't handle variables from the shell environment. This will likely need
special code in bitbake.
3) There are probably some variables which are removed and no longer
used/supported which we should also tell the user about (show a message instead
of a rename?)
4) The current code doesn't handle overridden variables. This is easier to add
to c) but something for b) should be possible.

I had been wondering about c) above and keeping overhead down but I think we'll
just have to go back to b) and try and work through the issues above. I worry
that stopping the builds on error in particular is going to be problematic. I
felt I should at least share some of the complexities of this with people so
that if it doesn't end up happening, the complexity of the issue is at least
more visible.

Let me follow up on where things are now at. I worked on the bitbake level
rename support and we have a patch which resolves some of the issues above. 2)
is fixed, 4) partially works and may still need tweaking. 1) was fixed but I
think may have regressed again as the autobuilder didn't stop builds the way I
expected it to. 3) still isn't done.

Joshua was able to fix the erroronce/warnonce log implementation, thanks.

Thanks to patches from Saul and Scott we have:

* a conversion script in master-next which converts metadata to match renames
* patches for bitbake and oe-core to transition to several of the new names

I was able to get the changes in master-next to run on the autobuilder with
unconverted layers being the failure cases.

The remaining things I'm aware of that need to be done are:

a) Resolve BB_DISKMON_DIRS in bitbake (last remaining bitbake rename)
b) Add some mechanism to show an error about now unused variables
c) Check builds really stop at parsing if errors are shown
d) Tweak the code for checking if overridden versions of variables are set
e) bump bitbake version and change the OE-Core minimum version requirement
f) consider changing the layer compatibility string to match for this
g) Handle ICECC_USER*/ICECC_SYSTEM* changes
h) Do something with the WHITELIST_(ANY LICENSE) variable
i) Scan over OE-Core for whitelist/blacklist variable name usage in python code
and propose patches for the issues
j) translate the names in the docs (the script should handle that)
k) document the conversion script and write the migration guide entry

b), d), e), f), g) are now in master-next but will need debugging and impact
other layers.

I think a), c) and h) probably block merging, the remainder can follow post
merge for core.

I'm working up something for (a) and at least the BitBake side of (i)
(and also replacing other "abort" usage). One issue is there some
variable names in Toaster code that look easy enough to change, but I'm
not entirely sure what the testing would need to be to vet it.

Scott

Just out of curiosity: is this all considered to be part of kirkstone release?
Lately I got the impression that neither the deprecation mechanism nor the list of potentially variable renamings are fully matured.

So from my perspective this shouldn't part of a potential LTS release, even if it's considered a big thing by some.

Or asking differently: just imagine this will be implemented for kirkstone release and after the release there will be additional findings... what's the take of the project on backporting vs. not backporting these changes?

On Fri, Feb 18, 2022 at 08:51:58PM +0100, Konrad Weihmann wrote:

Just out of curiosity: is this all considered to be part of
kirkstone release?

This was originally planned as one of the major "features" for the next LTS
release:
https://lists.openembedded.org/g/openembedded-architecture/topic/85488159

Lately I got the impression that neither the deprecation mechanism
nor the list of potentially variable renamings are fully matured.

https://lists.openembedded.org/g/openembedded-architecture/topic/75821819
https://lists.openembedded.org/g/openembedded-architecture/topic/84043114
https://wiki.yoctoproject.org/wiki/Inclusive_language
https://lists.openembedded.org/g/openembedded-core/topic/88650128
https://lists.openembedded.org/g/openembedded-architecture/topic/88899288

So from my perspective this shouldn't part of a potential LTS
release, even if it's considered a big thing by some.

Not having it in the next LTS will potentially be a huge PR nightmare for
the whole project (and its architect personally) for the next 2-4 years.

The Yocto TSC has even discussed the idea of slipping the code freeze and
potentially delaying the LTS release due to this not being fully ready yet.

Or asking differently: just imagine this will be implemented for
kirkstone release and after the release there will be additional
findings... what's the take of the project on backporting vs. not
backporting these changes?

Stable and LTS policies and procedures are outlined on the Wiki (specifically,
"Stable/LTS Patch Acceptance Policies" section) and should be followed:
https://wiki.yoctoproject.org/wiki/LTS

--
Denys

On 17.02.22 18:35, Saul Wold wrote:

On 2/17/22 04:32, Richard Purdie wrote:

On Thu, 2022-02-17 at 10:17 +0000, Richard Purdie via
lists.openembedded.org
wrote:

On Tue, 2022-02-15 at 12:08 +0000, Richard Purdie via
lists.openembedded.org
wrote:

In all the above cases there are still the issues that:

1) showing errors doesn't make bitbake exit or stop the build
2) It won't handle variables from the shell environment.
This will likely need
special code in bitbake.
3) There are probably some variables which are removed and no longer
used/supported which we should also tell the user about
(show a message instead
of a rename?)
4) The current code doesn't handle overridden variables.
This is easier to add
to c) but something for b) should be possible.

I had been wondering about c) above and keeping overhead
down but I think we'll
just have to go back to b) and try and work through the
issues above. I worry
that stopping the builds on error in particular is going to
be problematic. I
felt I should at least share some of the complexities of
this with people so
that if it doesn't end up happening, the complexity of the
issue is at least
more visible.

Let me follow up on where things are now at. I worked on the
bitbake level
rename support and we have a patch which resolves some of the
issues above. 2)
is fixed, 4) partially works and may still need tweaking. 1)
was fixed but I
think may have regressed again as the autobuilder didn't stop
builds the way I
expected it to. 3) still isn't done.

Joshua was able to fix the erroronce/warnonce log
implementation, thanks.

Thanks to patches from Saul and Scott we have:

* a conversion script in master-next which converts metadata
to match renames
* patches for bitbake and oe-core to transition to several of
the new names

I was able to get the changes in master-next to run on the
autobuilder with
unconverted layers being the failure cases.

The remaining things I'm aware of that need to be done are:

a) Resolve BB_DISKMON_DIRS in bitbake (last remaining bitbake rename)
b) Add some mechanism to show an error about now unused variables
c) Check builds really stop at parsing if errors are shown
d) Tweak the code for checking if overridden versions of
variables are set
e) bump bitbake version and change the OE-Core minimum version
requirement
f) consider changing the layer compatibility string to match for this
g) Handle ICECC_USER*/ICECC_SYSTEM* changes
h) Do something with the WHITELIST_(ANY LICENSE) variable
i) Scan over OE-Core for whitelist/blacklist variable name
usage in python code
and propose patches for the issues
j) translate the names in the docs (the script should handle that)
k) document the conversion script and write the migration guide entry

b), d), e), f), g) are now in master-next but will need
debugging and impact
other layers.

Thank RP for all that you have done, I will have a patch to
master-next for the script shortly.

I think a), c) and h) probably block merging, the remainder can
follow post
merge for core.

I will start looking at the WHITELIST_<license> changes and review
the old emails.

Sau!

Cheers,

Richard

On 18.02.22 21:44, Denys Dmytriyenko wrote:

On Fri, Feb 18, 2022 at 08:51:58PM +0100, Konrad Weihmann wrote:

Just out of curiosity: is this all considered to be part of
kirkstone release?

This was originally planned as one of the major "features" for the next LTS
release:
https://lists.openembedded.org/g/openembedded-architecture/topic/85488159

Lately I got the impression that neither the deprecation mechanism
nor the list of potentially variable renamings are fully matured.

https://lists.openembedded.org/g/openembedded-architecture/topic/75821819
https://lists.openembedded.org/g/openembedded-architecture/topic/84043114
https://wiki.yoctoproject.org/wiki/Inclusive_language
https://lists.openembedded.org/g/openembedded-core/topic/88650128
https://lists.openembedded.org/g/openembedded-architecture/topic/88899288

So from my perspective this shouldn't part of a potential LTS
release, even if it's considered a big thing by some.

Not having it in the next LTS will potentially be a huge PR nightmare for
the whole project (and its architect personally) for the next 2-4 years.

So the idea is to have rather something "premature" (intentional in quotes) in LTS than something that is technical sound and valid?

The Yocto TSC has even discussed the idea of slipping the code freeze and > potentially delaying the LTS release due to this not being fully

ready yet.
And even drop the proposed target deadline, just to "make it work"?

Or asking differently: just imagine this will be implemented for
kirkstone release and after the release there will be additional
findings... what's the take of the project on backporting vs. not
backporting these changes?

Stable and LTS policies and procedures are outlined on the Wiki (specifically,
"Stable/LTS Patch Acceptance Policies" section) and should be followed:
https://wiki.yoctoproject.org/wiki/LTS

The document doesn't even mention that very special case here, as there's technically speaking no reason for the applying any of the inclusive language patches on an once released branch - for me this is neither a bugfix, nor a security issue nor a technical necessity - so my take would be that none of the patches would be backported in any way.


---

Just to be clear, I'm not opposing the idea of having that non-technical feature, I just want to avoid of merging/releasing it prematurely (not to offend anyone, but SBOM feature for instance got a ton of patches after the release, not making the best impression to the people I talked to).
If merged in the state it is in right now I rather see loosing even more core contributors than one would gain

On Fri, 2022-02-18 at 22:56 +0100, Konrad Weihmann wrote:

On 18.02.22 21:44, Denys Dmytriyenko wrote:

On Fri, Feb 18, 2022 at 08:51:58PM +0100, Konrad Weihmann wrote:

Just out of curiosity: is this all considered to be part of
kirkstone release?

This was originally planned as one of the major "features" for the next LTS
release:
https://lists.openembedded.org/g/openembedded-architecture/topic/85488159

Lately I got the impression that neither the deprecation mechanism
nor the list of potentially variable renamings are fully matured.

https://lists.openembedded.org/g/openembedded-architecture/topic/75821819
https://lists.openembedded.org/g/openembedded-architecture/topic/84043114
https://wiki.yoctoproject.org/wiki/Inclusive_language
https://lists.openembedded.org/g/openembedded-core/topic/88650128
https://lists.openembedded.org/g/openembedded-architecture/topic/88899288

So from my perspective this shouldn't part of a potential LTS
release, even if it's considered a big thing by some.

Not having it in the next LTS will potentially be a huge PR nightmare for
the whole project (and its architect personally) for the next 2-4 years.

So the idea is to have rather something "premature" (intentional in
quotes) in LTS than something that is technical sound and valid?

In simple terms releases are time based or feature based, you do one or the
other. We're leaning towards getting this in as the objective, which means the
timing has to be flexible.

We were in a really bad position with this feature a week ago and I sounded the
alarm on it. Since then, partly through things I've done and partly through help
from several others we're in a much better position.

We're not there yet but I believe the remaining issues to be solvable in a
reasonable timeframe. I do not have a reputation of taking truly premature
changes and I do not intend to develop one now. The project would be in a better
position if this had been done earlier, yes.

The Yocto TSC has even discussed the idea of slipping the code freeze and 

potentially delaying the LTS release due to this not being fully
ready yet.

And even drop the proposed target deadline, just to "make it work"?

The feature or the deadline can move. We believe some flexibility in the
deadline might get the feature in.

Or asking differently: just imagine this will be implemented for
kirkstone release and after the release there will be additional
findings... what's the take of the project on backporting vs. not
backporting these changes?

Stable and LTS policies and procedures are outlined on the Wiki (specifically,
"Stable/LTS Patch Acceptance Policies" section) and should be followed:
https://wiki.yoctoproject.org/wiki/LTS

The document doesn't even mention that very special case here, as
there's technically speaking no reason for the applying any of the
inclusive language patches on an once released branch - for me this is
neither a bugfix, nor a security issue nor a technical necessity - so my
take would be that none of the patches would be backported in any way.

The changes do not qualify for backporting so they're in the release or they are
not.

Just to be clear, I'm not opposing the idea of having that non-technical
feature, I just want to avoid of merging/releasing it prematurely (not
to offend anyone, but SBOM feature for instance got a ton of patches
after the release, not making the best impression to the people I talked
to).

*Any* new feature ends up with a set of patches after release as people start to
use it. I think that has applied to most I can think of. It isn't any reflection
on the work on the feature, more that most people won't use or test anything
until the release. A sensible approach is to realise this and ensure there is a
stable policy which can quickly handle fixes.

If merged in the state it is in right now I rather see loosing even more
core contributors than one would gain

I respectfully disagree, particularly after your comments about issues with
previous features since you clearly haven't given thought to how these things
evolve in open source.

Pretty much every feature you use in the project evolves in this way, only once
people use things do you find and resolve the majority of the issues. I
therefore don't see this as a particularly strong reason not to put this into
the pending release.

I'd also note that this is the feature freeze date, not the final release date.
There are several weeks of testing between freeze and release to allow for
things to be tested and improved and this is unchanged. The fact few people
participate in that bug fixing is a source of frustration to me as it would help
solve the issue you mention above. The sad fact remains, most people won't test
it until the final release.

Cheers,

Richard