Date   

Re: Open Source Maintainers - An open letter/request

Armin Kuster
 

On 5/10/21 8:14 AM, Richard Purdie wrote:
TLDR: The project is seen as mature, employers don't prioritise maintaining
things and we're struggling for maintainers and help with day to day work


Open source projects survive, not just through development work and 
contributions of new features but through a whole load of "unglamorous" 
day to day "admin" work. This may be tracking down a regression, 
triaging failing builds, making a release of a component, reviewing a 
patch, documenting something or many other activities.

I love the fact we have active contributions, particularly for new features
but we are continuing to struggle in many of the other areas above. I am
extrememly grateful for the help we do receive with these tasks!

As a project we have automated an absolute ton of things, we can test
changes in ways we could only dream of a few years ago but maintaining
this automation, tracking down regressions and ensuring it all stays working
does have a cost.

I am worried, not just about the core of the project, but the wider layer
ecosystem since "layer maintainer" isn't seen as a particularly interesting
career enabling focus by employers and it seems a lot of this work isn't being
recognised. Internal business pressures are often continually being
prioritised over this.

The YP+OE ecosystem is becoming more mature and this means we have our 
experienced developers being pulled away to new things and few people
are replacing them so it feels like we're seeing a gradual skills drain/fade.

There are a few things companies can do to help:

a) Publicly acknowledge you use the project. 

I'm often asked where the project is being used but I find it hard to point
at companies using it, or products developed with it. It does help to be able
to point at real users rather than theoretical scenarios. We *know* it is used
in some interesting places but many won't let us say that publicly.

https://wiki.yoctoproject.org/wiki/Project_Users

b) Embrace employee's Open Source contributions, code and otherwise

If companies can find ways to recognise the value of having open source
experts/leaders working for them from a career development and reward 
perspective, that would encourage people to do the important work needed

c) Consider Yocto Project membership

https://www.yoctoproject.org/ecosystem/members/
https://www.yoctoproject.org/join/

We're finding that some infrastructure and roles need to be centrally funded
as the work is important but no one company is willing to commit people to it.
We're only able to to this through project membership which supports things
like the autobuilder, LTS, our build triage process and my own role.

d) Support employees in spending some time on open source projects

I hear quite often that employees get XX% time to spend on open source
projects. I also hear they get pulled onto mission critical product 
deliverables and can't prioritise that other project work. Finding ways
to ensure employees can spend time on open source projects including 
management support would help a lot.

e) Transition roles

If someone has a key role in a project but is moving to new things, help
them find a replacement and allow them time to train/transition to that
new person. Some companies do this really well, I'd call out NI and opkg
maintainership as a particularly good exmaple.



I appreciate these are difficult times, both for individuals and for 
businesses. I'd like to conclude by thanking everyone who does participate
and contribute. Whilst I do want/need to highlight the above (and have been
asked to do so that people have something they can point people at), the 
project is proving to be successful, going to interesting places and making
things possible we can all be proud of!
Thanks for summarizing all this.

So is the ask to forward this within the one's Employer?

-armin

Cheers,

Richard








Open Source Maintainers - An open letter/request

Richard Purdie
 

TLDR: The project is seen as mature, employers don't prioritise maintaining
things and we're struggling for maintainers and help with day to day work


Open source projects survive, not just through development work and 
contributions of new features but through a whole load of "unglamorous" 
day to day "admin" work. This may be tracking down a regression, 
triaging failing builds, making a release of a component, reviewing a 
patch, documenting something or many other activities.

I love the fact we have active contributions, particularly for new features
but we are continuing to struggle in many of the other areas above. I am
extrememly grateful for the help we do receive with these tasks!

As a project we have automated an absolute ton of things, we can test
changes in ways we could only dream of a few years ago but maintaining
this automation, tracking down regressions and ensuring it all stays working
does have a cost.

I am worried, not just about the core of the project, but the wider layer
ecosystem since "layer maintainer" isn't seen as a particularly interesting
career enabling focus by employers and it seems a lot of this work isn't being
recognised. Internal business pressures are often continually being
prioritised over this.

The YP+OE ecosystem is becoming more mature and this means we have our 
experienced developers being pulled away to new things and few people
are replacing them so it feels like we're seeing a gradual skills drain/fade.

There are a few things companies can do to help:

a) Publicly acknowledge you use the project. 

I'm often asked where the project is being used but I find it hard to point
at companies using it, or products developed with it. It does help to be able
to point at real users rather than theoretical scenarios. We *know* it is used
in some interesting places but many won't let us say that publicly.

https://wiki.yoctoproject.org/wiki/Project_Users

b) Embrace employee's Open Source contributions, code and otherwise

If companies can find ways to recognise the value of having open source
experts/leaders working for them from a career development and reward 
perspective, that would encourage people to do the important work needed

c) Consider Yocto Project membership

https://www.yoctoproject.org/ecosystem/members/
https://www.yoctoproject.org/join/

We're finding that some infrastructure and roles need to be centrally funded
as the work is important but no one company is willing to commit people to it.
We're only able to to this through project membership which supports things
like the autobuilder, LTS, our build triage process and my own role.

d) Support employees in spending some time on open source projects

I hear quite often that employees get XX% time to spend on open source
projects. I also hear they get pulled onto mission critical product 
deliverables and can't prioritise that other project work. Finding ways
to ensure employees can spend time on open source projects including 
management support would help a lot.

e) Transition roles

If someone has a key role in a project but is moving to new things, help
them find a replacement and allow them time to train/transition to that
new person. Some companies do this really well, I'd call out NI and opkg
maintainership as a particularly good exmaple.



I appreciate these are difficult times, both for individuals and for 
businesses. I'd like to conclude by thanking everyone who does participate
and contribute. Whilst I do want/need to highlight the above (and have been
asked to do so that people have something they can point people at), the 
project is proving to be successful, going to interesting places and making
things possible we can all be proud of!

Cheers,

Richard


Next Yocto Project LTS - April 2022

Richard Purdie
 

I'm pleased to be able to announce that the project is planning to have
the April 2022 release next year be our next LTS release.

This fits in with our original announced plan of 2 year cycles and 
recognises that the LTS has been well received by members and our 
community. It also aligns well with LTS releases of other projects
meaning we have "co-travellers".

The current dunfell LTS would be due to end at that time. There has
been discussion about extending it beyond the currently planned 
timeframe but no agreement/decision has been made on the extra finance 
that would require at this time. There is also some concern about the 
potential pressure on layer maintainers in wider ecosystem which we
plan to investigate further.

By confirming this now, we're hoping to give people a chance to align
strategies and plan for feature development to land into the LTS. There
is a clear need for any new/experimental changes to be planned/developed
now in order to ensure they're ready and stabilised for the LTS.

Cheers,

Richard


OpenEmbedded Developers Virtual meeting

Philip Balister
 

Part of the Yocto Project Summit (May 25-26, 2021), OpenEmbedded is
hosting a Virtual Developer meeting.
(https://pretalx.com/yocto-project-summit-2021/schedule/) Since this
will be a virtual event, we are going to try and set up the developer
meeting with more structure than previous meetings.

We've seeded a few topics and are looking for people to moderate them.
If I know you are active in certian areas, I'll be talking to you this
week. But please, we would like to get new voices involved solving some
of these high level issues. We also welcome topics that matter to end
users, so don't be afraid you aren't enough of a developer to
participate. As we get topics better defined and have moderators signed
up, we will add them to the summit schedule.

The format is described on the wiki (if you need an account, sign up and
message me to approve the account):

https://www.openembedded.org/wiki/OEDVM_2021

Philip


Re: OVERRIDES v2

Chris Laplante
 

I took the occasion to check what is the mechanism of overriding.

Below what myself found in Bitbake User Manual, ch. 3.3. Conditional Syntax
(Overrides) "You can use OVERRIDES to ... conditionally append or prepend
the value of a variable."
This is the point where I wonder why overriding is discussed in context of
append and prepend.
Which element of appending and prepending operations has the nature of
overriding other things?
Not sure if I understand the question, but maybe you're looking for this example (given in section 3.3.1):

KERNEL_FEATURES_append = " ${KERNEL_EXTRA_FEATURES}"
KERNEL_FEATURES_append_qemux86=" cfg/sound.scc cfg/paravirt_kvm.scc"
KERNEL_FEATURES_append_qemux86-64=" cfg/sound.scc cfg/paravirt_kvm.scc"

Thanks,
Chris


Re: OVERRIDES v2

keydi
 

I see for Bitbake appends and prepends are overrides flavor.
I still keep my position, generally there is no override component
in appending and prepending regardless which life area is considered.
Nothing get overridden by appending and prepending.
Things just get appended or prepended.
I see it to be unlucky these two to be treated in scope of overriding.
It is misleading in case of these to talk about overriding.

-----Original Message-----
From: openembedded-architecture@...
[mailto:openembedded-architecture@...] On Behalf
Of Denys Dmytriyenko
Sent: Montag, 26. April 2021 20:30
To: DUDZIAK Krzysztof <krzysztof.dudziak@...>
Cc: openembedded-architecture@...
Subject: Re: [Openembedded-architecture] OVERRIDES v2

On Mon, Apr 26, 2021 at 05:15:31PM +0000, keydi wrote:

Hi,

I took the occasion to check what is the mechanism of overriding.

Below what myself found in Bitbake User Manual, ch. 3.3. Conditional
Syntax
(Overrides) "You can use OVERRIDES to ... conditionally append or
prepend the value of a variable."
This is the point where I wonder why overriding is discussed in
context of append and prepend.
Which element of appending and prepending operations has the nature
of overriding other things?

BR
k.d.
If to use same chapter in order to continue my point regarding
intersection of overriding and appending/prepending I don't see in
example provided in bullet "Appending and Prepending:" any piece of
variable original value get literally overridden.
In contrary some further components get appended/prepended to original
content, myself doesn't understand the word "override" this way.
As _append, _prepend, _remove are special cases of the override
mechanism.
That was one of the reasons for 2015 discussion during OEDEM to see if
_append and _prepend need to be re-implemented to not confuse with
regular overrides.


My personal preference is to describe things as they are. Naming stuff is
also describing that stuff.
In case of append/prepend description seems to be misleading, it seems to
suggest non-existing behavior.
--
Regards,
Denys Dmytriyenko <denis@...>
PGP: 0x420902729A92C964 - https://denix.org/0x420902729A92C964
Fingerprint: 25FC E4A5 8A72 2F69 1186 6D76 4209 0272 9A92 C964


Re: OpenEmbedded Happy Hour April 28 9pm/2100 UTC

Denys Dmytriyenko
 

Reminder, next Happy Hour is in one day - everyone is welcome to meet with
fellow developers and chat about any interesting topics over Zoom. BYOB -
bring your own beverage.

On Wed, Apr 21, 2021 at 04:04:25PM -0400, Denys Dmytriyenko wrote:
Hi,

Please join us for the upcoming OpenEmbedded Happy Hour on April 28 for
Asia/Pacific timezones @ 2100/9pm UTC (5pm EDT):

https://www.openembedded.org/wiki/Calendar
https://www.timeanddate.com/worldclock/fixedtime.html?msg=OpenEmbedded+Happy+Hour+April+28&iso=20210428T21
--
Regards,
Denys Dmytriyenko <denis@...>
PGP: 0x420902729A92C964 - https://denix.org/0x420902729A92C964
Fingerprint: 25FC E4A5 8A72 2F69 1186 6D76 4209 0272 9A92 C964


Re: OVERRIDES v2

keydi
 


I took the occasion to check what is the mechanism of overriding.

Below what myself found in Bitbake User Manual, ch. 3.3. Conditional
Syntax
(Overrides) "You can use OVERRIDES to ... conditionally append or
prepend the value of a variable."
This is the point where I wonder why overriding is discussed in
context of append and prepend.
Which element of appending and prepending operations has the nature of
overriding other things?

BR
k.d.
I am also not comfortable with two first examples provided in chapter myself
referred previously too.
The first one, short one: why is "_nooverride = blabla" component
necessary?
Will variable default value not suffice?

The second one, longer one: It does not include the OVERRIDES = "blabla"
component.
Hence, the question is this override-code complete?
Regarding second point in my post above I see example for kernel seems to use
overrides-interface component, which however takes place in different code
thus not visible in shown example yet chapter didn't presented (even short reference)
in chapter prior to example presentation . That comment is obsolete.


Re: OVERRIDES v2

Denys Dmytriyenko
 

On Mon, Apr 26, 2021 at 12:19:58PM -0400, Trevor Woerner wrote:
On Mon 2021-04-26 @ 11:05:55 AM, Richard Purdie wrote:
On Mon, 2021-04-26 at 11:46 +0200, Quentin Schulz wrote:
Hi all,

I submitted a presentation about OVERRIDES, _append, +=, =. and others
for YP Summit 2021 in a month. While sharing the description with some
people in the Yocto community, I've been made aware that I'm missing
some (history) bits about OVERRIDES.

I've been told that it was added as a temporary measure/hack
Not sure who told you that but OVERRIDES has been around since bitbake 
(then oemake) was split out from openembedded which is probably around 2004.
I'm pretty sure Quentin is referring to me here ;-) and I'll be the first
person to tell you that I don't have the best memory going, so I apologize if
my poor memory causes a "fake news" incident ;-) But I left that meeting with
a very distinct impression that nobody felt that bitbake's OVERRIDE mechanism
was one of its best features. I thought the overall feeling was that OVERRIDES
was one of the biggest stumbling blocks for newbies. Although I wasn't around
when it was added, I seem to think it wasn't feature that was given much
thought; there was a need for something, this was proposed, and in it went.
Then, some years later, there was a feeling of "if we had known it was going
to get this complicated…"
I, for one, really like OVERRIDES mechanism! It may not be intuitive at first,
but once mastered, it's a very powerful mechanism and, as Chris mentioned,
allows doing layered (from generic to specific) alternatives very elegantly.


and that some had tried to get it removed/reconsidered back in 2015 (been 
given this link: https://www.openembedded.org/wiki/OEDEM_2015#Agenda) but it
was already largely (ab)used?
Reading that agenda item, I suspect I was the one who added and discussed it 
and it was less about removing OVERRIDES and more about considering whether 
there was some better operator/format to clearly differentiate between
a variable name and an override. It was a way to see if anyone had ideas, no
great replacement was identified (but was worth asking IMO).
I believe we discussed both a square bracket operator and a dot operator. The
square brackets were rejected because it was already being used for tasks and
PACKAGECONFIGs. Although the dot operator received a lot of support, in the
end I thought it came down to the difficulty of how invasive the changeover
would be (flag days etc).

A good idea, perhaps, but too much inertia otherwise.

Thanks for jumping in on this topic, Richard, and filling in the gaps. It's
nice to get this sort of information out of peoples' brains an onto paper.
--
Regards,
Denys Dmytriyenko <denis@...>
PGP: 0x420902729A92C964 - https://denix.org/0x420902729A92C964
Fingerprint: 25FC E4A5 8A72 2F69 1186 6D76 4209 0272 9A92 C964


Re: OVERRIDES v2

Denys Dmytriyenko
 

On Mon, Apr 26, 2021 at 05:15:31PM +0000, keydi wrote:

Hi,

I took the occasion to check what is the mechanism of overriding.

Below what myself found in Bitbake User Manual, ch. 3.3. Conditional Syntax
(Overrides) "You can use OVERRIDES to ... conditionally append or prepend
the value of a variable."
This is the point where I wonder why overriding is discussed in context of
append and prepend.
Which element of appending and prepending operations has the nature of
overriding other things?

BR
k.d.
If to use same chapter in order to continue my point regarding intersection
of overriding and appending/prepending
I don't see in example provided in bullet "Appending and Prepending:" any
piece of variable original value get literally overridden.
In contrary some further components get appended/prepended to original
content, myself doesn't understand
the word "override" this way.
As _append, _prepend, _remove are special cases of the override mechanism.
That was one of the reasons for 2015 discussion during OEDEM to see if _append
and _prepend need to be re-implemented to not confuse with regular overrides.


My personal preference is to describe things as they are. Naming stuff is
also describing that stuff.
In case of append/prepend description seems to be misleading, it seems to
suggest non-existing behavior.
--
Regards,
Denys Dmytriyenko <denis@...>
PGP: 0x420902729A92C964 - https://denix.org/0x420902729A92C964
Fingerprint: 25FC E4A5 8A72 2F69 1186 6D76 4209 0272 9A92 C964


Re: OVERRIDES v2

keydi
 

I took the occasion to check what is the mechanism of overriding.

Below what myself found in Bitbake User Manual, ch. 3.3. Conditional Syntax
(Overrides) "You can use OVERRIDES to ... conditionally append or prepend
the value of a variable."
This is the point where I wonder why overriding is discussed in context of
append and prepend.
Which element of appending and prepending operations has the nature of
overriding other things?
I use Bitbake user manual mentioned chapter as knowledge source to get more about override mechanism
as OVERRIDE variable description in YP Reference Manual refers to that document for more details.


Re: OVERRIDES v2

keydi
 


Hi,

I took the occasion to check what is the mechanism of overriding.

Below what myself found in Bitbake User Manual, ch. 3.3. Conditional Syntax
(Overrides) "You can use OVERRIDES to ... conditionally append or prepend
the value of a variable."
This is the point where I wonder why overriding is discussed in context of
append and prepend.
Which element of appending and prepending operations has the nature of
overriding other things?

BR
k.d.
If to use same chapter in order to continue my point regarding intersection of overriding and appending/prepending
I don't see in example provided in bullet "Appending and Prepending:" any piece of variable original value get literally overridden.
In contrary some further components get appended/prepended to original content, myself doesn't understand
the word "override" this way.

My personal preference is to describe things as they are. Naming stuff is also describing that stuff.
In case of append/prepend description seems to be misleading, it seems to suggest non-existing behavior.


Re: OVERRIDES v2

keydi
 

Hi,

I took the occasion to check what is the mechanism of overriding.

Below what myself found in Bitbake User Manual, ch. 3.3. Conditional Syntax
(Overrides) "You can use OVERRIDES to ... conditionally append or prepend
the value of a variable."
This is the point where I wonder why overriding is discussed in context of
append and prepend.
Which element of appending and prepending operations has the nature of
overriding other things?

BR
k.d.
I am also not comfortable with two first examples provided in chapter myself referred previously too.
The first one, short one: why is "_nooverride = blabla" component necessary?
Will variable default value not suffice?

The second one, longer one: It does not include the OVERRIDES = "blabla" component.
Hence, the question is this override-code complete?


Re: OVERRIDES v2

keydi
 

On Mon, 2021-04-26 at 11:46 +0200, Quentin Schulz wrote:
Hi all,

I submitted a presentation about OVERRIDES, _append, +=, =. and others
for YP Summit 2021 in a month. While sharing the description with some
people in the Yocto community, I've been made aware that I'm missing
some (history) bits about OVERRIDES.

I've been told that it was added as a temporary measure/hack
Not sure who told you that but OVERRIDES has been around since bitbake
(then oemake) was split out from openembedded which is probably around
2004.

and that some had tried to get it removed/reconsidered back in 2015
(been given this link:
https://www.openembedded.org/wiki/OEDEM_2015#Agenda) but it was
already largely (ab)used?

Reading that agenda item, I suspect I was the one who added and discussed
it and it was less about removing OVERRIDES and more about considering
whether there was some better operator/format to clearly differentiate
between a variable name and an override. It was a way to see if anyone had
ideas, no great replacement was identified (but was worth asking IMO).

So now, my questions:
  - why was OVERRIDES implemented in the first place? What was it
trying to resolve?
They're one of the ways OE tried to handle conditional configuration changes
which are needed on a machine or policy (distro) basis and as a way of
allowing customisation. I'd say on balance they've been highly successful at it.

  - why was it considered for removal/reimplementation back in 2015
(or even earlier?)?
Not removal, considered for reimplementation. Is there a better syntax
which could be used? It is an open question, we've not identified one.

  - what made you decide to not go for it? Already too widely used?
We'd need to demonstrate that there was an issue being solved, that there
was enough of a benefit for people to take any pain of migration. The fact
they are widely used is a big factor.

  - is there something you'd have done differently?
I suspect lots of things. Specifically with overrides, I've personally wondered
about whether a specific operator syntax would make sense.

  - what are you thoughts on OVERRIDES today?
  - is there a plan to code a new implementation of OVERRIDES or a
similar mechanism? If so, what are the ideas you have currently on how
to do it? (hence the clickbait title :) )
I think you might find there was actually a newly coded backend
implementation of overrides around the 2015 timeframe as I quite radically
changed the way the datastore worked behind the scenes. There were some
user visible changes but they were minor and mostly corrections making the
behaviour consistent.

What changed behind the scenes was the datastore became dynamic.
Previously you updated OVERRIDES and then called dd.data.update_data().
This was removed in:

http://git.yoctoproject.org/cgit.cgi/poky/commit/bitbake/lib/bb/data_smart
.py?id=e7ccd9071233d66afb0bc72774b0032fb8229fe4

We also changed the default on getVar to expand by default in 2016.

Reading the history of:

http://git.yoctoproject.org/cgit.cgi/poky/log/bitbake/lib/bb/data_smart.py

may give some insight.

Another big change was dependency tracking of variables allowing hash
calculation for signatures. We also mandated overrides by lower case, which
helps parsing as it becomes easier for the code to identify or ignore possible
overrides.

Basically, we needed those fixes for various reasons such as sstate hashes, or
to fix parsing performance issues but we were able to make the existing APIs
mostly work so we didn't change the user visible side. That removed much of
the pressure to change the user visible implementation.

Just FWIW, much of our parsing speed pain now is in the tons of anonymous
python people keep adding, thinking little of the overhead about trying to
parse it all for dependencies and run it all...

Whether my presentation is selected does not matter, still curious
nonetheless :)
Cheers,

Richard
Hi,

I took the occasion to check what is the mechanism of overriding.

Below what myself found in Bitbake User Manual, ch. 3.3. Conditional Syntax (Overrides) "You can use OVERRIDES to ... conditionally append or prepend the value of a variable."
This is the point where I wonder why overriding is discussed in context of append and prepend.
Which element of appending and prepending operations has the nature of overriding other things?

BR
k.d.


Re: OVERRIDES v2

Trevor Woerner
 

On Mon 2021-04-26 @ 11:05:55 AM, Richard Purdie wrote:
On Mon, 2021-04-26 at 11:46 +0200, Quentin Schulz wrote:
Hi all,

I submitted a presentation about OVERRIDES, _append, +=, =. and others
for YP Summit 2021 in a month. While sharing the description with some
people in the Yocto community, I've been made aware that I'm missing
some (history) bits about OVERRIDES.

I've been told that it was added as a temporary measure/hack
Not sure who told you that but OVERRIDES has been around since bitbake 
(then oemake) was split out from openembedded which is probably around 2004.
I'm pretty sure Quentin is referring to me here ;-) and I'll be the first
person to tell you that I don't have the best memory going, so I apologize if
my poor memory causes a "fake news" incident ;-) But I left that meeting with
a very distinct impression that nobody felt that bitbake's OVERRIDE mechanism
was one of its best features. I thought the overall feeling was that OVERRIDES
was one of the biggest stumbling blocks for newbies. Although I wasn't around
when it was added, I seem to think it wasn't feature that was given much
thought; there was a need for something, this was proposed, and in it went.
Then, some years later, there was a feeling of "if we had known it was going
to get this complicated…"


and that some had tried to get it removed/reconsidered back in 2015 (been 
given this link: https://www.openembedded.org/wiki/OEDEM_2015#Agenda) but it
was already largely (ab)used?
Reading that agenda item, I suspect I was the one who added and discussed it 
and it was less about removing OVERRIDES and more about considering whether 
there was some better operator/format to clearly differentiate between
a variable name and an override. It was a way to see if anyone had ideas, no
great replacement was identified (but was worth asking IMO).
I believe we discussed both a square bracket operator and a dot operator. The
square brackets were rejected because it was already being used for tasks and
PACKAGECONFIGs. Although the dot operator received a lot of support, in the
end I thought it came down to the difficulty of how invasive the changeover
would be (flag days etc).

A good idea, perhaps, but too much inertia otherwise.

Thanks for jumping in on this topic, Richard, and filling in the gaps. It's
nice to get this sort of information out of peoples' brains an onto paper.

Best regards,
Trevor


Re: OVERRIDES v2

Christopher Larson
 



On Mon, Apr 26, 2021 at 8:12 AM Richard Purdie <richard.purdie@...> wrote:
On Mon, 2021-04-26 at 14:44 +0000, chris.laplante@... wrote:
> > Just FWIW, much of our parsing speed pain now is in the tons of anonymous
> > python people keep adding, thinking little of the overhead about trying to
> > parse it all for dependencies and run it all...
>
> Is there a benchmark tool for measuring BitBake parse speed? I've been 
> digging into DataSmart. I see oe-build-perf-test but as the name implies 
> it seems like it's for build speed testing.

The perf test does include a parse speed test which is basically 
"time bitbake -p" with varying levels of cache being present.

You can get interesting profiling data with "bitbake -P" too.

As an alternative to just 'time', there are useful tools for basic statistical analysis for benchmarking of cli operations, such as hyperfine (https://github.com/sharkdp/hyperfine) nowadays.

I always liked to think of OVERRIDES as one of the mechanisms used to implement a layered hierarchy of metadata -- *not referring to oe layers here, but conceptual ones*, whereby our metadata ranges from most generic to most specific, generic, distro, arch, machine, forced/local, which is also implemented by the order of includes in bitbake.conf, adjusted for realities (can't include local after distro since local sets distro :).
--
Christopher Larson
kergoth at gmail dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Senior Software Engineer, Mentor Graphics


Re: OVERRIDES v2

Richard Purdie
 

On Mon, 2021-04-26 at 14:44 +0000, chris.laplante@... wrote:
Just FWIW, much of our parsing speed pain now is in the tons of anonymous
python people keep adding, thinking little of the overhead about trying to
parse it all for dependencies and run it all...
Is there a benchmark tool for measuring BitBake parse speed? I've been 
digging into DataSmart. I see oe-build-perf-test but as the name implies 
it seems like it's for build speed testing.
The perf test does include a parse speed test which is basically 
"time bitbake -p" with varying levels of cache being present.

You can get interesting profiling data with "bitbake -P" too.

Cheers,

Richard


Re: OVERRIDES v2

Chris Laplante
 

Hi Richard,

Just FWIW, much of our parsing speed pain now is in the tons of anonymous
python people keep adding, thinking little of the overhead about trying to
parse it all for dependencies and run it all...
Is there a benchmark tool for measuring BitBake parse speed? I've been digging into DataSmart. I see oe-build-perf-test but as the name implies it seems like it's for build speed testing.

Thanks,
Chris


Re: OVERRIDES v2

Richard Purdie
 

On Mon, 2021-04-26 at 11:46 +0200, Quentin Schulz wrote:
Hi all,

I submitted a presentation about OVERRIDES, _append, +=, =. and others
for YP Summit 2021 in a month. While sharing the description with some
people in the Yocto community, I've been made aware that I'm missing
some (history) bits about OVERRIDES.

I've been told that it was added as a temporary measure/hack
Not sure who told you that but OVERRIDES has been around since bitbake 
(then oemake) was split out from openembedded which is probably around 2004.

and that some had tried to get it removed/reconsidered back in 2015 (been 
given this link: https://www.openembedded.org/wiki/OEDEM_2015#Agenda) but it
was already largely (ab)used?
Reading that agenda item, I suspect I was the one who added and discussed it 
and it was less about removing OVERRIDES and more about considering whether 
there was some better operator/format to clearly differentiate between
a variable name and an override. It was a way to see if anyone had ideas, no
great replacement was identified (but was worth asking IMO).

So now, my questions:
  - why was OVERRIDES implemented in the first place? What was it trying
to resolve?
They're one of the ways OE tried to handle conditional configuration changes
which are needed on a machine or policy (distro) basis and as a way of allowing
customisation. I'd say on balance they've been highly successful at it.

  - why was it considered for removal/reimplementation back in 2015 (or
even earlier?)?
Not removal, considered for reimplementation. Is there a better syntax which
could be used? It is an open question, we've not identified one.

  - what made you decide to not go for it? Already too widely used?
We'd need to demonstrate that there was an issue being solved, that there was
enough of a benefit for people to take any pain of migration. The fact they are
widely used is a big factor.

  - is there something you'd have done differently?
I suspect lots of things. Specifically with overrides, I've personally wondered
about whether a specific operator syntax would make sense.

  - what are you thoughts on OVERRIDES today?
  - is there a plan to code a new implementation of OVERRIDES or a
similar mechanism? If so, what are the ideas you have currently on how
to do it? (hence the clickbait title :) )
I think you might find there was actually a newly coded backend implementation
of overrides around the 2015 timeframe as I quite radically changed the way the
datastore worked behind the scenes. There were some user visible changes but
they were minor and mostly corrections making the behaviour consistent.

What changed behind the scenes was the datastore became dynamic. Previously you
updated OVERRIDES and then called dd.data.update_data(). This was removed in:

http://git.yoctoproject.org/cgit.cgi/poky/commit/bitbake/lib/bb/data_smart.py?id=e7ccd9071233d66afb0bc72774b0032fb8229fe4

We also changed the default on getVar to expand by default in 2016.

Reading the history of:

http://git.yoctoproject.org/cgit.cgi/poky/log/bitbake/lib/bb/data_smart.py

may give some insight.

Another big change was dependency tracking of variables allowing hash calculation
for signatures. We also mandated overrides by lower case, which helps parsing as it
becomes easier for the code to identify or ignore possible overrides.

Basically, we needed those fixes for various reasons such as sstate hashes, or to
fix parsing performance issues but we were able to make the existing APIs mostly
work so we didn't change the user visible side. That removed much of the pressure
to change the user visible implementation.

Just FWIW, much of our parsing speed pain now is in the tons of anonymous python 
people keep adding, thinking little of the overhead about trying to parse it all
for dependencies and run it all...

Whether my presentation is selected does not matter, still curious
nonetheless :)
Cheers,

Richard


OVERRIDES v2

Quentin Schulz
 

Hi all,

I submitted a presentation about OVERRIDES, _append, +=, =. and others for YP Summit 2021 in a month. While sharing the description with some people in the Yocto community, I've been made aware that I'm missing some (history) bits about OVERRIDES.

I've been told that it was added as a temporary measure/hack and that some had tried to get it removed/reconsidered back in 2015 (been given this link: https://www.openembedded.org/wiki/OEDEM_2015#Agenda) but it was already largely (ab)used?

So now, my questions:
- why was OVERRIDES implemented in the first place? What was it trying to resolve?
- why was it considered for removal/reimplementation back in 2015 (or even earlier?)?
- what made you decide to not go for it? Already too widely used?
- is there something you'd have done differently?
- what are you thoughts on OVERRIDES today?
- is there a plan to code a new implementation of OVERRIDES or a similar mechanism? If so, what are the ideas you have currently on how to do it? (hence the clickbait title :) )

Whether my presentation is selected does not matter, still curious nonetheless :)

Cheers,
Quentin

441 - 460 of 1662