Date   

Re: OVERRIDES v2

Christopher Larson
 



On Mon, Apr 26, 2021 at 8:12 AM Richard Purdie <richard.purdie@...> wrote:
On Mon, 2021-04-26 at 14:44 +0000, chris.laplante@... wrote:
> > Just FWIW, much of our parsing speed pain now is in the tons of anonymous
> > python people keep adding, thinking little of the overhead about trying to
> > parse it all for dependencies and run it all...
>
> Is there a benchmark tool for measuring BitBake parse speed? I've been 
> digging into DataSmart. I see oe-build-perf-test but as the name implies 
> it seems like it's for build speed testing.

The perf test does include a parse speed test which is basically 
"time bitbake -p" with varying levels of cache being present.

You can get interesting profiling data with "bitbake -P" too.

As an alternative to just 'time', there are useful tools for basic statistical analysis for benchmarking of cli operations, such as hyperfine (https://github.com/sharkdp/hyperfine) nowadays.

I always liked to think of OVERRIDES as one of the mechanisms used to implement a layered hierarchy of metadata -- *not referring to oe layers here, but conceptual ones*, whereby our metadata ranges from most generic to most specific, generic, distro, arch, machine, forced/local, which is also implemented by the order of includes in bitbake.conf, adjusted for realities (can't include local after distro since local sets distro :).
--
Christopher Larson
kergoth at gmail dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Senior Software Engineer, Mentor Graphics


Re: OVERRIDES v2

Richard Purdie
 

On Mon, 2021-04-26 at 14:44 +0000, chris.laplante@... wrote:
Just FWIW, much of our parsing speed pain now is in the tons of anonymous
python people keep adding, thinking little of the overhead about trying to
parse it all for dependencies and run it all...
Is there a benchmark tool for measuring BitBake parse speed? I've been 
digging into DataSmart. I see oe-build-perf-test but as the name implies 
it seems like it's for build speed testing.
The perf test does include a parse speed test which is basically 
"time bitbake -p" with varying levels of cache being present.

You can get interesting profiling data with "bitbake -P" too.

Cheers,

Richard


Re: OVERRIDES v2

Chris Laplante
 

Hi Richard,

Just FWIW, much of our parsing speed pain now is in the tons of anonymous
python people keep adding, thinking little of the overhead about trying to
parse it all for dependencies and run it all...
Is there a benchmark tool for measuring BitBake parse speed? I've been digging into DataSmart. I see oe-build-perf-test but as the name implies it seems like it's for build speed testing.

Thanks,
Chris


Re: OVERRIDES v2

Richard Purdie
 

On Mon, 2021-04-26 at 11:46 +0200, Quentin Schulz wrote:
Hi all,

I submitted a presentation about OVERRIDES, _append, +=, =. and others
for YP Summit 2021 in a month. While sharing the description with some
people in the Yocto community, I've been made aware that I'm missing
some (history) bits about OVERRIDES.

I've been told that it was added as a temporary measure/hack
Not sure who told you that but OVERRIDES has been around since bitbake 
(then oemake) was split out from openembedded which is probably around 2004.

and that some had tried to get it removed/reconsidered back in 2015 (been 
given this link: https://www.openembedded.org/wiki/OEDEM_2015#Agenda) but it
was already largely (ab)used?
Reading that agenda item, I suspect I was the one who added and discussed it 
and it was less about removing OVERRIDES and more about considering whether 
there was some better operator/format to clearly differentiate between
a variable name and an override. It was a way to see if anyone had ideas, no
great replacement was identified (but was worth asking IMO).

So now, my questions:
  - why was OVERRIDES implemented in the first place? What was it trying
to resolve?
They're one of the ways OE tried to handle conditional configuration changes
which are needed on a machine or policy (distro) basis and as a way of allowing
customisation. I'd say on balance they've been highly successful at it.

  - why was it considered for removal/reimplementation back in 2015 (or
even earlier?)?
Not removal, considered for reimplementation. Is there a better syntax which
could be used? It is an open question, we've not identified one.

  - what made you decide to not go for it? Already too widely used?
We'd need to demonstrate that there was an issue being solved, that there was
enough of a benefit for people to take any pain of migration. The fact they are
widely used is a big factor.

  - is there something you'd have done differently?
I suspect lots of things. Specifically with overrides, I've personally wondered
about whether a specific operator syntax would make sense.

  - what are you thoughts on OVERRIDES today?
  - is there a plan to code a new implementation of OVERRIDES or a
similar mechanism? If so, what are the ideas you have currently on how
to do it? (hence the clickbait title :) )
I think you might find there was actually a newly coded backend implementation
of overrides around the 2015 timeframe as I quite radically changed the way the
datastore worked behind the scenes. There were some user visible changes but
they were minor and mostly corrections making the behaviour consistent.

What changed behind the scenes was the datastore became dynamic. Previously you
updated OVERRIDES and then called dd.data.update_data(). This was removed in:

http://git.yoctoproject.org/cgit.cgi/poky/commit/bitbake/lib/bb/data_smart.py?id=e7ccd9071233d66afb0bc72774b0032fb8229fe4

We also changed the default on getVar to expand by default in 2016.

Reading the history of:

http://git.yoctoproject.org/cgit.cgi/poky/log/bitbake/lib/bb/data_smart.py

may give some insight.

Another big change was dependency tracking of variables allowing hash calculation
for signatures. We also mandated overrides by lower case, which helps parsing as it
becomes easier for the code to identify or ignore possible overrides.

Basically, we needed those fixes for various reasons such as sstate hashes, or to
fix parsing performance issues but we were able to make the existing APIs mostly
work so we didn't change the user visible side. That removed much of the pressure
to change the user visible implementation.

Just FWIW, much of our parsing speed pain now is in the tons of anonymous python 
people keep adding, thinking little of the overhead about trying to parse it all
for dependencies and run it all...

Whether my presentation is selected does not matter, still curious
nonetheless :)
Cheers,

Richard


OVERRIDES v2

Quentin Schulz
 

Hi all,

I submitted a presentation about OVERRIDES, _append, +=, =. and others for YP Summit 2021 in a month. While sharing the description with some people in the Yocto community, I've been made aware that I'm missing some (history) bits about OVERRIDES.

I've been told that it was added as a temporary measure/hack and that some had tried to get it removed/reconsidered back in 2015 (been given this link: https://www.openembedded.org/wiki/OEDEM_2015#Agenda) but it was already largely (ab)used?

So now, my questions:
- why was OVERRIDES implemented in the first place? What was it trying to resolve?
- why was it considered for removal/reimplementation back in 2015 (or even earlier?)?
- what made you decide to not go for it? Already too widely used?
- is there something you'd have done differently?
- what are you thoughts on OVERRIDES today?
- is there a plan to code a new implementation of OVERRIDES or a similar mechanism? If so, what are the ideas you have currently on how to do it? (hence the clickbait title :) )

Whether my presentation is selected does not matter, still curious nonetheless :)

Cheers,
Quentin


Re: devtool modify docker failure

AnandJe
 

Hi,

 

I have made my yocto setup successfully.

The layers I have included is:

$ cat conf/bblayers.conf

  ${TOPDIR}/../poky/meta \

  ${TOPDIR}/../poky/meta-poky \

  ${TOPDIR}/../poky/meta-yocto-bsp \

  ${TOPDIR}/../layers/meta-openembedded/meta-filesystems \

  ${TOPDIR}/../layers/meta-openembedded/meta-networking \

  /${TOPDIR}/../layers/meta-openembedded/meta-oe \

  ${TOPDIR}/../layers/meta-openembedded/meta-perl \

  ${TOPDIR}/../layers/meta-openembedded/meta-python \

  ${TOPDIR}/../layers/meta-virtualization \

 

I want to work on docker source code so I thought of fetching the source code from devtool.

But while I'm using devtool I'm facing an issue.

 

$ devtool modify --no-same-dir --extract docker-ce workspace/sources/docker

ERROR: Command Error: 'sh -c 'PATCHFILE="0001-libnetwork-use-GO-instead-of-go.patch" git -c user.name="OpenEmbedded" -c user.email="oe.patch@oe" commit -F /tmp/tmps5pr6pui --author="Bruce Ashfield <bruce.ashfield@...>" --date="Fri, 6 Apr 2018 23:58:22 -0400"'' exited with 0  Output:

On branch devtool

Changes not staged for commit:

  (use "git add <file>..." to update what will be committed)

  (use "git restore <file>..." to discard changes in working directory)

  (commit or discard the untracked or modified content in submodules)

modified:   libnetwork (modified content)

 

no changes added to commit (use "git add" and/or "git commit -a")

ERROR: Logfile of failure stored in: ${TOPDIR}/build/tmp/work/core2-64-poky-linux/docker-ce/19.03.8-ce+gitafacb8b7f0d8d4f9d2a8e8736e9c993e672b41f3-r0/devtooltmp-lnnatl98/temp/log.do_patch.27786

 

Is there any workaround or solution to overcome this error?

 

Thanks & Regards

Anand


OpenEmbedded Happy Hour April 28 9pm/2100 UTC

Denys Dmytriyenko
 

Hi,

Please join us for the upcoming OpenEmbedded Happy Hour on April 28 for
Asia/Pacific timezones @ 2100/9pm UTC (5pm EDT):

https://www.openembedded.org/wiki/Calendar
https://www.timeanddate.com/worldclock/fixedtime.html?msg=OpenEmbedded+Happy+Hour+April+28&iso=20210428T21

--
Regards,
Denys Dmytriyenko <denis@...>
PGP: 0x420902729A92C964 - https://denix.org/0x420902729A92C964
Fingerprint: 25FC E4A5 8A72 2F69 1186 6D76 4209 0272 9A92 C964


Yocto Project Summit - registration open

Trevor Woerner
 

Registration is now open for the upcoming Yocto Project Summit!!

details: https://www.yoctoproject.org/yocto-project-virtual-summit-2021/
CfP: https://pretalx.com/yocto-project-summit-2021/cfp
registration: https://www.cvent.com/d/yjq4dr/4W?ct=868bfddd-ca91-46bb-aaa5-62d2b61b2501

PS: Don't forget to get in your talk proposals! The CfP closes April 25th


Yocto Project Virtual Summit 2021

Trevor Woerner
 

The Yocto Project Summit Planning Committee is happy to announce the
upcoming 3rd Yocto Project Summit to take place Tuesday and Wednesday
May 25-26 2021, virtually.

The 2-day event will run in 2 tracks including a virtual developers meeting,
beginner tutorial sessions, hands-on intermediate instruction, lightning
talks, regular talks, and social events.

The cost for all attendees will be $40USD. The event will run both days from
noon until 8pm GMT. Registration is not yet open but will be shortly, please
watch for further announcements.

The call for papers is now open and will close at 11:59 PM PST on Sunday,
April 25, 2021. To submit a proposal please visit:
https://pretalx.com/yocto-project-summit-2021/cfp

For more information please visit:
https://www.yoctoproject.org/yocto-project-virtual-summit-2021/

We look forward to seeing you at the conference!


OpenEmbedded Happy Hour March 31 5pm/1700 UTC

Denys Dmytriyenko
 

Hi,

Just a reminder about our upcoming OpenEmbedded Happy Hour on March 31 for
Europe/US timezones @ 1700/5pm UTC (1pm ET / 10am PT):

https://www.openembedded.org/wiki/Calendar
https://www.timeanddate.com/worldclock/fixedtime.html?msg=OpenEmbedded+Happy+Hour+March+31&iso=20210331T17

--
Regards,
Denys Dmytriyenko <denis@...>
PGP: 0x420902729A92C964 - https://denix.org/0x420902729A92C964
Fingerprint: 25FC E4A5 8A72 2F69 1186 6D76 4209 0272 9A92 C964


fitImage in not installed into the rootfs.

Ilies CHERGUI
 

Hi everyone,

I'm using Poky with a master branch, I'm enabling the fitImage. Unfortunately, the fitImage is not packaged and not installed into the root file system.
 
Here is my updates in the conf/local.conf file:
KERNEL_CLASSES += "kernel-fitimage"
KERNEL_IMAGETYPE = "fitImage"
INITRAMFS_IMAGE_BUNDLE = "1"

 
The fitImage is well generated and deployed in the deploy directory but it is not installed into the root file system and it is not packaged. the directory kernel-image-fitimage under the packages-split of the linux work directory is empty.
 
Please see the attached picture ?
 
Am I missing something?
Any help would be much appreciated.
 
Best regards,
Ilies


Re: This mailing-list the right location for newbie OE/Y/P user to ask questions?

keydi
 

Hi Nicolas,

Thank you for your quick support. I follow your kind instructions.

Regards
krzysiek

-----Original Message-----
From: Nicolas Dechesne [mailto:nicolas.dechesne@...]
Sent: Dienstag, 16. März 2021 14:11
To: DUDZIAK Krzysztof <krzysztof.dudziak@...>
Cc: openembedded-architecture <openembedded-
architecture@...>
Subject: Re: [Openembedded-architecture] This mailing-list the right location
for newbie OE/Y/P user to ask questions?

hi Krzysztof,


On Tue, Mar 16, 2021 at 1:28 PM DUDZIAK Krzysztof
<krzysztof.dudziak@...> wrote:

Hi, Myself needs to ask question regarding artifacts built along with
embedded Linux distro built using Yocto/Poky yet addressing software
composition of built distro/image. Is this list the right location to ask such
question? Is this list in general right location for newbies to asks question
regarding OE/Yocto/Poky build system usage?

Not really. This is more to discuss 'large' changes related to OE and its overall
architecture. Perhaps the best list would be:
https://lists.yoctoproject.org/g/yocto/topics

This is where we tend to see most of the new users' questions.



Re: This mailing-list the right location for newbie OE/Y/P user to ask questions?

Nicolas Dechesne
 

hi Krzysztof,


On Tue, Mar 16, 2021 at 1:28 PM DUDZIAK Krzysztof
<krzysztof.dudziak@...> wrote:

Hi, Myself needs to ask question regarding artifacts built along with embedded Linux distro built using Yocto/Poky yet addressing software composition of built distro/image. Is this list the right location to ask such question? Is this list in general right location for newbies to asks question regarding OE/Yocto/Poky build system usage?
Not really. This is more to discuss 'large' changes related to OE and
its overall architecture. Perhaps the best list would be:
https://lists.yoctoproject.org/g/yocto/topics

This is where we tend to see most of the new users' questions.



This mailing-list the right location for newbie OE/Y/P user to ask questions?

keydi
 

Hi, Myself needs to ask question regarding artifacts built along with embedded Linux distro built using Yocto/Poky yet addressing software composition of built distro/image. Is this list the right location to ask such question? Is this list in general right location for newbies to asks question regarding OE/Yocto/Poky build system usage?


Re: OEDAM 2021 Planning

Phil Blundell
 

On Wed, Feb 24, 2021 at 12:50:45AM +0100, Phil Blundell via lists.openembedded.org wrote:
On Tue, Feb 23, 2021 at 04:13:06PM -0500, Rich Persaud wrote:
Would YP or OE mailing list be preferred for planning the 2021 OEDAM (OpenEmbedded Developer Americas Meeting)? Newcomers don't make a distinction, but organizers can.
At the risk of stating the obvious... if it's got "OE" in the name
then I would have thought the OE list would be the right place.
Also, as I just discovered, it turns out that yocto-advocacy@ bounces
emails from non-yocto folk. So in order to facilitate reply-all, it
might be best not to cross-post to both that list and oe-architecture@.

Thanks

Phil


Re: OEDAM 2021 Planning

Phil Blundell
 

On Tue, Feb 23, 2021 at 04:13:06PM -0500, Rich Persaud wrote:
Would YP or OE mailing list be preferred for planning the 2021 OEDAM (OpenEmbedded Developer Americas Meeting)? Newcomers don't make a distinction, but organizers can.
At the risk of stating the obvious... if it's got "OE" in the name
then I would have thought the OE list would be the right place.

I guess if your target audience is Yocto dudes then you could change
the name to YGAM and organise it on the YP list :-)

p.


OEDAM 2021 Planning

Rich Persaud
 

Would YP or OE mailing list be preferred for planning the 2021 OEDAM (OpenEmbedded Developer Americas Meeting)? Newcomers don't make a distinction, but organizers can.

Rich


OpenEmbedded Happy Hour February 24 9pm/2100 UTC

Denys Dmytriyenko
 

Hi,

Please join us for the upcoming OpenEmbedded Happy Hour on February 24 for
Asia/Pacific timezones @ 2100/9pm UTC (4pm EST):

https://www.openembedded.org/wiki/Calendar
https://www.timeanddate.com/worldclock/fixedtime.html?msg=OpenEmbedded+Happy+Hour+February+24&iso=20210224T21

--
Regards,
Denys Dmytriyenko <denis@...>
PGP: 0x420902729A92C964 - https://denix.org/0x420902729A92C964
Fingerprint: 25FC E4A5 8A72 2F69 1186 6D76 4209 0272 9A92 C964


Default umask for tasks

Richard Purdie
 

In looking at some of the reproducibility failures, I realised that
several are from a common cause, basically tasks missing a umask
setting. Our common tasks have this but for example:

kernel-devsrc - copies files from ${S} which are created in special
kernel tasks which don't have umask set

quilt-ptest - contains files created by do_compile_ptest_base which has
no umask set.

Looking at the output for valgrind-ptest I suspect something similar.

We could continue to try and fix this per task, or we could set a
default umask for tasks. I was surprised that bitbake didn't have an
option for that so I added one, BB_DEFAULT_UMASK.

I'll send out patches for bitbake and OE-Core, I think/hope this change
shouldn't be controversial but I wanted people to be aware of the
change.

I'm using a value of 022 since that is what is used for most tasks
currently. We can get rid of quite a few umask variable flag settings
after this which is a nice cleanup.

Cheers,

Richard


Yocto 2.6, spdxscanner..

keydi
 

Hi,

1. Are any meta-spdxscanner alternative solutions provided by Yocto
for software bill of materials generation?
2. What level of maturity do artifacts generated by meta-spdxscanner
version used in YP 2.6 show? Any kind of artifacts needed for to OSS compliance
requirements still not collected by spdxscanner in mentioned YP version?
To put in other words: How safe is it to rely merely on mentioned compound
to be at the end of day compliant with OSS obligations?
Distribution comprised from more than 200 packages is built hence
we speak about potentially very wide range of open source licensing types
for packages used in distribution.
3. How might feasibility of backporting spdxscanner versions to Yocto 2.6 look like?

Regards
k.d.

461 - 480 of 1667