On 31/07/21 23:09, Richard Purdie wrote:

On Sat, 2021-07-31 at 15:29 +0000, Peter Kjellerstedt wrote:

-----Original Message-----
From: openembedded-architecture@... <openembedded-
architecture@...> On Behalf Of Richard Purdie
Sent: den 30 juli 2021 15:40
To: openembedded-architecture <openembedded-
architecture@...>
Subject: Re: [Openembedded-architecture] Overrides conversion plan

On Wed, 2021-07-28 at 16:43 +0100, Richard Purdie via
lists.openembedded.org wrote:


I have continued to work on this and I now think we're as ready as we'll
ever
be with the core. I have:

* submitted a section for the migration guide documenting the conversion
process
* increased the minimum bitbake version for OE-Core
* bumped the local.conf version to require new versions of the config file
* added an error to bitbake if it sees "_append"|"_prepend"|"_remove" in
  variable names which would suggest an unconverted layer. If you use
those
  in function names in the datastore that was never a good idea and is no
  longer supported.
* merged compatibility changes back to bitbake 1.50, 1.48 and 1.46
* merged the conversion script to OE-Core
* merged submitted tweaks to the conversion script (thanks Martin!)
* made OE-Core honister only, no longer supporting hardknott
* updated converted layers to be honister compatible
* converted autobuilder-helper to use the new syntax
* ensured all of a-quick builds on the autobuilder
* submitted patches for meta-yocto, meta-gplv2, meta-mingw and bitbake

I plan to merge these various things on Monday (2nd August). After that
time, unconverted layers will no longer work with master.

Cheers,

Richard

[ I am on summer vacation so I just happened to see a tweet about this. ]

I am sorry to say, but I think you are going too fast. AFAICT, the support
for using : instead of _ in overrides is not yet in any Dunfell or Hardknott
release, only on the branches. I can't speak for others of course, but at
least we will not pick it up until it is included in an actual release.

There are releases of hardknott and dunfell currently in QA so this won't
make it in until the ones after that.
I understand the concern however I don't think it is reasonable to wait that
long. You do have the options of pulling in the changes earlier, or
back porting them and if that isn't an option, it shouldn't be that long until
the next releases happen.
I am trying hard to find ways to allow things to operate over multiple releases
and I think we have come up with a good plan there for this change. This is not
something we have ever planned or committed to supporting though. I think it
is in fact dangerous as it is effectively tying the project into never changing
anything, or if it does, planning changes in multiple years time which isn't in
my view reasonable. I worry that the bar to making any change like this is so
high as to put off anyone from ever doing it. The master branch is a development
branch at the end of the day.
Cheers,
Richard

Hi Richard,
please apologize my question but I am not sure I have a clear vision of the global impact of this new feature across the branches because the answers above are confusing me.

From what I understood so far and I've see in the poky repo, the 'master' branch already has the switch done on 02/08/21 [1]

I don't understand if it is planned to do the same in 'dunfell' LTS or other branches (how and when).


[1] https://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=2abf8a699edd513405befbd1a0eafc8f55d6b514


Thank you
--
Marco

On Sat, 2021-07-31 at 15:29 +0000, Peter Kjellerstedt wrote:

-----Original Message-----
From: openembedded-architecture@... <openembedded-
architecture@...> On Behalf Of Richard Purdie
Sent: den 30 juli 2021 15:40
To: openembedded-architecture <openembedded-
architecture@...>
Subject: Re: [Openembedded-architecture] Overrides conversion plan

On Wed, 2021-07-28 at 16:43 +0100, Richard Purdie via
lists.openembedded.org wrote:


I have continued to work on this and I now think we're as ready as we'll
ever
be with the core. I have:

* submitted a section for the migration guide documenting the conversion
process
* increased the minimum bitbake version for OE-Core
* bumped the local.conf version to require new versions of the config file
* added an error to bitbake if it sees "_append"|"_prepend"|"_remove" in
  variable names which would suggest an unconverted layer. If you use
those
  in function names in the datastore that was never a good idea and is no
  longer supported.
* merged compatibility changes back to bitbake 1.50, 1.48 and 1.46
* merged the conversion script to OE-Core
* merged submitted tweaks to the conversion script (thanks Martin!)
* made OE-Core honister only, no longer supporting hardknott
* updated converted layers to be honister compatible
* converted autobuilder-helper to use the new syntax
* ensured all of a-quick builds on the autobuilder
* submitted patches for meta-yocto, meta-gplv2, meta-mingw and bitbake

I plan to merge these various things on Monday (2nd August). After that
time, unconverted layers will no longer work with master.

Cheers,

Richard

[ I am on summer vacation so I just happened to see a tweet about this. ]

I am sorry to say, but I think you are going too fast. AFAICT, the support
for using : instead of _ in overrides is not yet in any Dunfell or Hardknott
release, only on the branches. I can't speak for others of course, but at
least we will not pick it up until it is included in an actual release.

There are releases of hardknott and dunfell currently in QA so this won't
make it in until the ones after that.

I understand the concern however I don't think it is reasonable to wait that
long. You do have the options of pulling in the changes earlier, or 
back porting them and if that isn't an option, it shouldn't be that long until
the next releases happen.

I am trying hard to find ways to allow things to operate over multiple releases
and I think we have come up with a good plan there for this change. This is not 
something we have ever planned or committed to supporting though. I think it
is in fact dangerous as it is effectively tying the project into never changing
anything, or if it does, planning changes in multiple years time which isn't in
my view reasonable. I worry that the bar to making any change like this is so
high as to put off anyone from ever doing it. The master branch is a development
branch at the end of the day.

Cheers,

Richard

On Sat, Jul 31, 2021 at 5:29 AM Peter Kjellerstedt
<peter.kjellerstedt@...> wrote:

-----Original Message-----
From: openembedded-architecture@... <openembedded-
architecture@...> On Behalf Of Richard Purdie
Sent: den 30 juli 2021 15:40
To: openembedded-architecture <openembedded-
architecture@...>
Subject: Re: [Openembedded-architecture] Overrides conversion plan

On Wed, 2021-07-28 at 16:43 +0100, Richard Purdie via
lists.openembedded.org wrote:

I think the challenge is going to be the flag day issue for master

branches.

For example, there is code in devtool and other places which knows about

the

override character. If we allow mixing the different syntax for master

then

those tools need to complicate things by referencing both characters. To

try

and preserve what is left of my sanity, I'm starting to think we just

require

layers to migrate to the new syntax to continue to work with master. The

good

news is that those converted layers should work with dunfell and older

releases

where the layer already does that with the backported bitbake syntax

update.

If we accept that we need to have a flag day for master use, the

question is

when. We could pick some data well in the future or even post 3.4

however I'm

not sure this buys much and we probably may as well get on and do it.

Given these things, I therefore propose that we should start these

changes

and require it for master, probably relatively quickly within a couple

of

weeks?

I have continued to work on this and I now think we're as ready as we'll
ever
be with the core. I have:

* submitted a section for the migration guide documenting the conversion
process
* increased the minimum bitbake version for OE-Core
* bumped the local.conf version to require new versions of the config file
* added an error to bitbake if it sees "_append"|"_prepend"|"_remove" in
variable names which would suggest an unconverted layer. If you use
those
in function names in the datastore that was never a good idea and is no
longer supported.
* merged compatibility changes back to bitbake 1.50, 1.48 and 1.46
* merged the conversion script to OE-Core
* merged submitted tweaks to the conversion script (thanks Martin!)
* made OE-Core honister only, no longer supporting hardknott
* updated converted layers to be honister compatible
* converted autobuilder-helper to use the new syntax
* ensured all of a-quick builds on the autobuilder
* submitted patches for meta-yocto, meta-gplv2, meta-mingw and bitbake

I plan to merge these various things on Monday (2nd August). After that
time, unconverted layers will no longer work with master.

Cheers,

Richard

[ I am on summer vacation so I just happened to see a tweet about this. ]

I am sorry to say, but I think you are going too fast. AFAICT, the support
for using : instead of _ in overrides is not yet in any Dunfell or Hardknott
release, only on the branches. I can't speak for others of course, but at
least we will not pick it up until it is included in an actual release.

I am currently testing the change in dunfell and so far so good. If
no issues are found support will be present in the 3.1.11 release
currently scheduled for 2021/9/24

Steve

On Fri, Jul 30, 2021 at 3:40 PM Richard Purdie
<richard.purdie@...> wrote:

On Wed, 2021-07-28 at 16:43 +0100, Richard Purdie via lists.openembedded.org wrote:

I think the challenge is going to be the flag day issue for master branches.
For example, there is code in devtool and other places which knows about the
override character. If we allow mixing the different syntax for master then
those tools need to complicate things by referencing both characters. To try
and preserve what is left of my sanity, I'm starting to think we just require
layers to migrate to the new syntax to continue to work with master. The good
news is that those converted layers should work with dunfell and older releases
where the layer already does that with the backported bitbake syntax update.

If we accept that we need to have a flag day for master use, the question is
when. We could pick some data well in the future or even post 3.4 however I'm
not sure this buys much and we probably may as well get on and do it.

Given these things, I therefore propose that we should start these changes
and require it for master, probably relatively quickly within a couple of
weeks?

I have continued to work on this and I now think we're as ready as we'll ever
be with the core. I have:

* submitted a section for the migration guide documenting the conversion process
* increased the minimum bitbake version for OE-Core
* bumped the local.conf version to require new versions of the config file
* added an error to bitbake if it sees "_append"|"_prepend"|"_remove" in
variable names which would suggest an unconverted layer. If you use those
in function names in the datastore that was never a good idea and is no
longer supported.
* merged compatibility changes back to bitbake 1.50, 1.48 and 1.46
* merged the conversion script to OE-Core
* merged submitted tweaks to the conversion script (thanks Martin!)
* made OE-Core honister only, no longer supporting hardknott
* updated converted layers to be honister compatible
* converted autobuilder-helper to use the new syntax
* ensured all of a-quick builds on the autobuilder
* submitted patches for meta-yocto, meta-gplv2, meta-mingw and bitbake

I plan to merge these various things on Monday (2nd August). After that time,
unconverted layers will no longer work with master.

Cheers,

Richard

Thank you very much Martin and Richard.
Cheers
A.A.

On Wed, 2021-07-28 at 16:43 +0100, Richard Purdie via lists.openembedded.org wrote:

I think the challenge is going to be the flag day issue for master branches.
For example, there is code in devtool and other places which knows about the
override character. If we allow mixing the different syntax for master then
those tools need to complicate things by referencing both characters. To try
and preserve what is left of my sanity, I'm starting to think we just require
layers to migrate to the new syntax to continue to work with master. The good
news is that those converted layers should work with dunfell and older releases
where the layer already does that with the backported bitbake syntax update.

If we accept that we need to have a flag day for master use, the question is 
when. We could pick some data well in the future or even post 3.4 however I'm
not sure this buys much and we probably may as well get on and do it.

Given these things, I therefore propose that we should start these changes 
and require it for master, probably relatively quickly within a couple of 
weeks?

I have continued to work on this and I now think we're as ready as we'll ever
be with the core. I have:

* submitted a section for the migration guide documenting the conversion process
* increased the minimum bitbake version for OE-Core
* bumped the local.conf version to require new versions of the config file
* added an error to bitbake if it sees "_append"|"_prepend"|"_remove" in 
variable names which would suggest an unconverted layer. If you use those 
in function names in the datastore that was never a good idea and is no 
longer supported.
* merged compatibility changes back to bitbake 1.50, 1.48 and 1.46
* merged the conversion script to OE-Core
* merged submitted tweaks to the conversion script (thanks Martin!)
* made OE-Core honister only, no longer supporting hardknott
* updated converted layers to be honister compatible
* converted autobuilder-helper to use the new syntax
* ensured all of a-quick builds on the autobuilder
* submitted patches for meta-yocto, meta-gplv2, meta-mingw and bitbake

I plan to merge these various things on Monday (2nd August). After that time,
unconverted layers will no longer work with master.

Cheers,

Richard

Hi Marco,

On Thu, 2021-07-29 at 08:50 +0200, Marco Cavallini wrote:

IMHO it is important to produce a clear documentation about the new
features maybe providing some practical cases.

I agree it is important this is documented. I've sent a patch to the manuals 
to add a section to the 3.4 migration guide:

https://lists.yoctoproject.org/g/docs/message/1589

Does that help?

Hopefully that should start to document things and I'd welcome help in knowing
what other information we need to add there. It is a little more verbose
than some other migration guide sections since this one is something a lot
of people will run into.

Cheers,

Richard

On 7/28/21 10:43 AM, Richard Purdie wrote:

Hi All,

I've spent quite a bit of time seeing how to progress things. I now
have a script which can convert layers and seems to have reasonable success
on OE-Core, bitbake, yocto-docs, meta-yocto, meta-gplv2 and meta-mingw. I've
proposed this for OE-Core along with an example conversion patch for OE-Core.

The current patch status is I have conversions (including to bitbake's core)
in master-next branches. This nearly has q-quick builds passing on the 
autobuilder, there are a handful of oe-selftest cases which don't pass yet.

Things do get a little more complex than I'd hoped. We use OVERRIDES
in a number of interesting places, including within multilib configurations
for the tune-XXX variable suffixes. I did try making the code do "fallback"
variable lookups instead of expansion for tune- but I quickly gave up due
to the nesting. My view is that they are being used as overrides and we should 
therefore recognise them as such. Over time I think the tune code can be
improved to take explicit advantage of that.

As such, I'm proposing we make the package variables and tune variables
explicit overrides and the conversion scripts and patches above assume this.

I think the challenge is going to be the flag day issue for master branches.
For example, there is code in devtool and other places which knows about the
override character. If we allow mixing the different syntax for master then
those tools need to complicate things by referencing both characters. To try
and preserve what is left of my sanity, I'm starting to think we just require
layers to migrate to the new syntax to continue to work with master. The good
news is that those converted layers should work with dunfell and older releases
where the layer already does that with the backported bitbake syntax update.

If we accept that we need to have a flag day for master use, the question is 
when. We could pick some data well in the future or even post 3.4 however I'm
not sure this buys much and we probably may as well get on and do it.

Given these things, I therefore propose that we should start these changes 
and require it for master, probably relatively quickly within a couple of 
weeks?

Sooner the better is my view...

Thanks for all of this, I know it's going to make things better longer term no
matter when it switches over to the new format. (Also making the tune-XXX a
first class override makes sense!)

--Mark

Cheers,

Richard

On 28/07/21 17:43, Richard Purdie wrote:

Hi All,
I've spent quite a bit of time seeing how to progress things. I now
have a script which can convert layers and seems to have reasonable success
on OE-Core, bitbake, yocto-docs, meta-yocto, meta-gplv2 and meta-mingw. I've
proposed this for OE-Core along with an example conversion patch for OE-Core.
The current patch status is I have conversions (including to bitbake's core)
in master-next branches. This nearly has q-quick builds passing on the
autobuilder, there are a handful of oe-selftest cases which don't pass yet.
Things do get a little more complex than I'd hoped. We use OVERRIDES
in a number of interesting places, including within multilib configurations
for the tune-XXX variable suffixes. I did try making the code do "fallback"
variable lookups instead of expansion for tune- but I quickly gave up due
to the nesting. My view is that they are being used as overrides and we should
therefore recognise them as such. Over time I think the tune code can be
improved to take explicit advantage of that.
As such, I'm proposing we make the package variables and tune variables
explicit overrides and the conversion scripts and patches above assume this.
I think the challenge is going to be the flag day issue for master branches.
For example, there is code in devtool and other places which knows about the
override character. If we allow mixing the different syntax for master then
those tools need to complicate things by referencing both characters. To try
and preserve what is left of my sanity, I'm starting to think we just require
layers to migrate to the new syntax to continue to work with master. The good
news is that those converted layers should work with dunfell and older releases
where the layer already does that with the backported bitbake syntax update.
If we accept that we need to have a flag day for master use, the question is
when. We could pick some data well in the future or even post 3.4 however I'm
not sure this buys much and we probably may as well get on and do it.
Given these things, I therefore propose that we should start these changes
and require it for master, probably relatively quickly within a couple of
weeks?
Cheers,
Richard

Hi Richard,
thank you for your precious work.

IMHO it is important to produce a clear documentation about the new features maybe providing some practical cases.

Just my 2 cents,

Cheers,
--
Marco

Em qua., 28 de jul. de 2021 às 12:43, Richard Purdie
<richard.purdie@...> escreveu:

Given these things, I therefore propose that we should start these changes
and require it for master, probably relatively quickly within a couple of
weeks?

I believe your plan is good and given the new layers will work down to
dunfell I see no big problems in requiring it to work on master. In
fact, I think it is the most logical move as it allows master to
optimize for its new format.

--
Otavio Salvador O.S. Systems
http://www.ossystems.com.br http://code.ossystems.com.br
Mobile: +55 (53) 9 9981-7854 Mobile: +1 (347) 903-9750

Hi All,

I've spent quite a bit of time seeing how to progress things. I now
have a script which can convert layers and seems to have reasonable success
on OE-Core, bitbake, yocto-docs, meta-yocto, meta-gplv2 and meta-mingw. I've
proposed this for OE-Core along with an example conversion patch for OE-Core.

The current patch status is I have conversions (including to bitbake's core)
in master-next branches. This nearly has q-quick builds passing on the 
autobuilder, there are a handful of oe-selftest cases which don't pass yet.

Things do get a little more complex than I'd hoped. We use OVERRIDES
in a number of interesting places, including within multilib configurations
for the tune-XXX variable suffixes. I did try making the code do "fallback"
variable lookups instead of expansion for tune- but I quickly gave up due
to the nesting. My view is that they are being used as overrides and we should 
therefore recognise them as such. Over time I think the tune code can be
improved to take explicit advantage of that.

As such, I'm proposing we make the package variables and tune variables
explicit overrides and the conversion scripts and patches above assume this.

I think the challenge is going to be the flag day issue for master branches.
For example, there is code in devtool and other places which knows about the
override character. If we allow mixing the different syntax for master then
those tools need to complicate things by referencing both characters. To try
and preserve what is left of my sanity, I'm starting to think we just require
layers to migrate to the new syntax to continue to work with master. The good
news is that those converted layers should work with dunfell and older releases
where the layer already does that with the backported bitbake syntax update.

If we accept that we need to have a flag day for master use, the question is 
when. We could pick some data well in the future or even post 3.4 however I'm
not sure this buys much and we probably may as well get on and do it.

Given these things, I therefore propose that we should start these changes 
and require it for master, probably relatively quickly within a couple of 
weeks?

Cheers,

Richard