Re: New assignment operator?
Mark Hatle
On 6/15/21 9:15 AM, Richard Purdie wrote:
On Tue, 2021-06-15 at 16:50 +0300, Petr Nechaev wrote:I've hit this issue in the past (different context, but I believe the sameHi All,Consider this example where the BSP sets: issue). What I end up doing is (in the initramfs) is effectively: OVERRIDES .= ":ramfs" IMAGE_FSTYPES_ramfs = "cpio" doing this in the recipe (or a bbclass that should only be loaded by a recipe) ensures that we can set IMAGE_FSTYPE to a specific value, and avoids the specific issue mentioned here. (At least it worked well in the case I needed it for, not sure if it would here, but it may be what is needed.) --Mark Cheers,
|
|
Re: New assignment operator?
Tom Rini
On Tue, Jun 15, 2021 at 03:09:03PM +0100, Richard Purdie wrote:
On Tue, 2021-06-15 at 09:38 -0400, Tom Rini wrote:Ah! Just slightly before my starting time with the project :)Thanks for all the details here. Since this is stemming from a specificI haven't tried to obscure it, the machine name was in the original email The BSP layer in question is meta-handheld, MACHINE=collie.OK, thanks. Pondering time... -- Tom
|
|
Re: New assignment operator?
Richard Purdie
On Tue, 2021-06-15 at 16:50 +0300, Petr Nechaev wrote:
Hi All,Consider this example where the BSP sets: IMAGE_FSTYPES = "tar" IMAGE_FSTYPES_somemachine = "ext4" INITRAMFS_FSTYPES = "cpio" OVERRIDES = "${MACHINE}" then the initramfs image recipe does: IMAGE_FSTYPES = "${INITRAMFS_FSTYPES}" Given that config, if you set MACHINE = "somemachine", which image type would you expect the initramfs image to have? The answer is that it will try and generate an ext4 initramfs. What you'd expect/want is a cpio initramfs. The reason is that the machine override, i.e.: IMAGE_FSTYPES_somemachine = "ext4" cannot be undone. Cheers, Richard
|
|
Re: New assignment operator?
Richard Purdie
On Tue, 2021-06-15 at 09:38 -0400, Tom Rini wrote:
Thanks for all the details here. Since this is stemming from a specificI haven't tried to obscure it, the machine name was in the original email but I guess not everyone remembers Zarus models names like I do! :) The BSP layer in question is meta-handheld, MACHINE=collie. https://github.com/andrea-adami/meta-handheld Recipes which have the issue are in meta-initramfs in meta-oe, or there are several initramfs images in OE-Core which also reset IMAGE_FSTYPES. Cheers, Richard
|
|
Re: New assignment operator?
Petr Nechaev
Hi All, I've been using bitbake for >5 years now, however I do not understand the nature of the problem at all. Could you clarify what the problem is i.e. what's the difference between these lines: IMAGE_FSTYPES = "${INITRAMFS_FSTYPES}" VAR_A = "${VAR_B}" In both cases .. it seems .. I am prepared that VAR_C = "${VAR_A}" won't expand to VAR_B due to overrides, appends etc. What is special about initramfs image? Is there any special meaning to IMAGE_FSTYPES variable? If not ... are we trying to introduce a language feature to solve one particular problem of a specific image? --- Petr
On Tue, Jun 15, 2021 at 4:38 PM Tom Rini <trini@...> wrote: On Tue, Jun 15, 2021 at 02:29:16PM +0100, Richard Purdie wrote:
|
|
Re: New assignment operator?
Tom Rini
On Tue, Jun 15, 2021 at 02:29:16PM +0100, Richard Purdie wrote:
I should apologise for being a little grumpy in some of my replies,Thanks for all the details here. Since this is stemming from a specific BSP, I think at this point it might be good to share what exactly it is, and it wouldn't be seen as "shaming" that BSP at this point as it's exposed a rather, as you note, obtuse problem. I have another "could we just ..." idea on this, but I could answer that myself maybe with the exact problem laid out. -- Tom
|
|
Re: New assignment operator?
Richard Purdie
I should apologise for being a little grumpy in some of my replies,
it is fair to say that everything is getting to me a little as continual build failures and being continually asked for reasoned arguments for saying "no" to things is wearing me down. We have bugs in many core pieces of the system (pseudo, patchelf, prelink, ltp, oeqa, devtool, eSDK and so on) and currently it feels like I'm the only person with the domain knowledge to try and attempt to look into them. This shouldn't ripple out into emails though. The context of this issue is probably important and I didn't really mention it. I've been asked about a "bitbake bug" a lot on irc recently and asked for help in trying to resolve it. I spent quite a bit more time than expected (on my weekend) trying to understand the issue and it wasn't the issue as reported but a lot more subtle. In the emails here I've spelt out the problem but the way it becomes exposed to the end user is a lot more insidious. I don't think the BSP was doing anything wrong using a MACHINE override on a variable. The initramfs recipe was also not really doing anything wrong trying to set the fstypes to the initramfs ones. The interaction between the two things is rather unfortunate and in this case the BSP maintainer could not see why it was breaking and even me, with a few years experience with bitbake couldn't immediately understand what was wrong or how my own fix was going to break. Even now I think broken "fixes" are being spread around in attempts to try and work around the issue which swap on machine's breakage for another (collie works but qemux86 using image-live then doesn't). It does worry me a lot that the issue is so obtuse to debug and that whilst we can patch this one up, someone else can/will hit it again. The potential to hit it with some other variable also remains. I don't like issues that few people can "see" into and understand. For that reason I would like to change the initramfs recipe somehow to improve usability and ensure people don't hit this. Right now I can't see any way to do that other than to say "don't do that". I can't even add anything to tell the user there is a problem. This was the spirit the proposal was born from. I understand why people don't like any new operator, I'm not thrilled either but what I'm not seeing are alternatives to improve usability :/. Cheers, Richard
|
|
Re: New assignment operator?
Richard Purdie
On Tue, 2021-06-15 at 09:32 -0300, Otavio Salvador wrote:
Hello Richard,Think about this in the context of what the initramfs recipe is trying to do. It doesn't care about IMAGE_FSTYPES, it cares about INITRAMFS_FSTYPES only. If the BSP wants to change anything they'd be changing INITRAMFS_FSTYPES for the initramfs recipe. Cheers, Richard
|
|
Re: New assignment operator?
Otavio Salvador
Hello Richard,
Em ter., 15 de jun. de 2021 às 07:29, Richard Purdie <richard.purdie@...> escreveu: I wondered about making := clear existing overrides/appends/prepends/removesHaving it resetting it might fix this use case but what if the _collie use case is a valid one? how the BSP vendor could override it? -- Otavio Salvador O.S. Systems http://www.ossystems.com.br http://code.ossystems.com.br Mobile: +55 (53) 9 9981-7854 Mobile: +1 (347) 903-9750
|
|
Re: New assignment operator?
Richard Purdie
On Tue, 2021-06-15 at 13:55 +0200, Phil Blundell wrote:
On Tue, Jun 15, 2021 at 11:29:01AM +0100, Richard Purdie wrote:Sure, but then you just end up writing:a) is another assignment operator a good idea?I'm not terribly convinced it is. If the problem is that there's no unset IMAGE_FSTYPES IMAGE_FSTYPES = "x" The assignment wouldn't ignore them, it really would remove them from the knowledge of the variable. The difference is basically the difference between: setVar("IMAGE_FSTYPES", val, parsing=True) setVar("IMAGE_FSTYPES", val, parsing=False) since behind the scenes bitbake has two different behaviours for setVar, "parsing" mode and non-parsing mode. This was after we learnt we had two different expectations from setVar in anonymous code and from parsing metadata. So we'd be adding an operator that behaves like the default setVar(). Ultimately though this particular example just seems like a caseIt would solve a general case problem of not being able to actually reset a variable outside anonymous python. I'm also not sure I'd say the BSP was carelessly written. It used overrides, we recommend and support them, it just then interacted badly with the way a recipe was written (which has been copied to multiple places). Adding more and more special operators that defeat each other seemsMaybe, maybe not. The initramfs recipe can't really express what it needs/wants to do with the current syntax. If we addMaybe we make it a non-conf context operator? I don't know. I think it doesn't matter what I suggest these days, the answer will be "no, it is already complicated enough". As such we therefore accept nothing will change. I put together the proposal as I don't think the BSP or the image recipe are really at fault here and I can't give them a good way to handle the situation. Cheers, Richard
|
|
Re: New assignment operator?
Richard Purdie
On Tue, 2021-06-15 at 11:42 +0000, Quentin Schulz wrote:
Well, it seems like an ok thing for a BSP to do to me. There are probably other ways it could be done, sure. Resetting IMAGE_FSTYPES_collie to INITRAMFS_FSTYPES should be enoughNo, I don't see how that helps. The issue in the email is an example of a general problem where it isThe questions areI'm not sure the issue explained in this mail is enough to warrant a new effectively impossible to undo certain operations. That general issue bothers me more than this specific case. We can argue the BSP is broken. The BSP is arguing bitbake is broken. Failing that, there is an argument the initramfs recipes are broken. I can see the different sides but there is a general problem here we've pondered over for a while. I can already see vendors abusing this by always using this new operatorWe could ban the operator from conf files (enforced)? This would be even less fun to support on IRC or mailing lists.Maybe, maybe not. It depends how it gets used and if "abuse" occurs. Also... As opposed to other operators, what would happen if you use theThankfully that is quite clear and would clear any overrides set against "IMAGE_FSTYPES_collir" Except that half the behaviour of := doesn't apply (the expansion). :=b) if so, which form?Totally unbiased. The last two 😁 is immediate expansion *and* immediate assignment. Cheers, Richard
|
|
Re: New assignment operator?
Phil Blundell
On Tue, Jun 15, 2021 at 11:29:01AM +0100, Richard Purdie wrote:
a) is another assignment operator a good idea?I'm not terribly convinced it is. If the problem is that there's no way to clear the OVERRIDES, maybe we should just have a new verb to do that specifically rather than a new assignment operator that ignores them. Ultimately though this particular example just seems like a case where it's possible for a carelessly-written BSP to make life difficult for its users. I don't think this is the only such, and I'm not sure a new assignment operator would necessarily solve the general case. Adding more and more special operators that defeat each other seems like it's just going to result in a perpetual arms race. If we add a new !=! operator then, BSP authors being as ingenious as they are, they'll start using that all over the place and you suddenly won't be able to override anything any more. Then we'll need a new and even more powerful OVERRRRIDES that can't be suppressed, and we'll be back where we started :-) p.
|
|
Re: New assignment operator?
Quentin Schulz
On June 15, 2021 10:29:01 AM UTC, Richard Purdie <richard.purdie@...> wrote:
We have an interesting problem with initramfs images. They do something like:Is that an issue that needs to actually be fixed? This is just another example of weird BSP given by vendors to me, right? Resetting IMAGE_FSTYPES_collie to INITRAMFS_FSTYPES should be enough to fix this issue right? This is a long standing issue where in standard metadata, there is no wayI'm not sure the issue explained in this mail is enough to warrant a new assignment operator. I can already see vendors abusing this by always using this new operator instead of using overrides or whatever. This would be even less fun to support on IRC or mailing lists. Also... As opposed to other operators, what would happen if you use the following: IMAGE_FSTYPES_collir *:= "whatever". What would be the meaning and behavior, it might warrant a different handling compared to other operators to forbid any override (and append, prepend, remove). b) if so, which form?Totally unbiased. The last two 😁 It highlights that like := it is an immediate assignment operator. The asterisk indicates that it will apply to "everything". Cheers, Quentin
|
|
New assignment operator?
Richard Purdie
We have an interesting problem with initramfs images. They do something like:
IMAGE_FSTYPES = "${INITRAMFS_FSTYPES}" which looks innocent enough until a BSP does: IMAGE_FSTYPES_collie = "xxx" where collie is MACHINE, hence an override which causes confusion as the value of INITRAMFS_FSTYPES is never used for that machine. This is a long standing issue where in standard metadata, there is no way to undo an override (or append/prepend/remove). I tried to be clever and did: python () { d.setVar("IMAGE_FSTYPES", d.getVar("INITRAMFS_FSTYPES")) } and if you do that before the image class inherit (since anon python is executed in the order it is defined), it nearly works. It doesn't work with live images as IMAGE_FSTYPES is expanded for class inherits before the anonymous python executes. I also wondered about: IMAGE_FSTYPES := "${INITRAMFS_FSTYPES}" but that doesn't clear overrides either. In fact I can't actually see a way to do it other than some horrible thing like: def myfunc(d): d.setVar("IMAGE_FSTYPES", d.getVar("INITRAMFS_FSTYPES")) DUMMYVAR := "${@myfunc(d)}" Yes, we could do that but I don't think we should. I wondered about making := clear existing overrides/appends/prepends/removes but then I wondered if we should have a new assignment operator which would do that (and also not force the value to be expanded). I mentioned this on irc and there were a few suggestions: IMAGE_FSTYPES =!! "${INITRAMFS_FSTYPES}" IMAGE_FSTYPES !=! "${INITRAMFS_FSTYPES}" IMAGE_FSTYPES *:= "${INITRAMFS_FSTYPES}" IMAGE_FSTYPES :=* "${INITRAMFS_FSTYPES}" The questions are a) is another assignment operator a good idea? b) if so, which form? Implementation shouldn't be hard in bitbake, the APIs are already there to do this kind of operation. I am kind of preferring !=!. Cheers, Richard
|
|
eSDK & multiconfig
Mark Hatle
We're starting to hit a maturity level with our internal development using
multiconfigs that we need a way to export this into an eSDK. Currently we're working on the ability to build a system, it's configured similarly to: (local.conf) - (MACHINE=zynqmp DEFAULTTUNE=cortexa72-cortexa53) | --- microblaze-pmu - MACHINE=zynqmp - DEFAULTTUNE=microblaze - DISTRO=baremetal | --- cortexr5 - MACHINE=zynqmp - DEFAULTTUNE=cortexr5f - DISTRO=baremetal This will allow someone to build a zynqmp system, as well as associated microcontroller-like baremetal software. This software is then collected by the main Linux configuration, able to be packaged for field upgrades, as well as placement in /boot for pre-linux boot needs. We have all of this working using a pure Yocto Project workflow today. (For instance, standard 'bitbake core-image-minimal' should produce the resulting image we want.) The eSDK though can't deal with this type of configuration at present. To start with the eSDK simply won't build. It fails in copy_buildsystem when running devtool. So if this is going to work, I'm seeing a couple of work items that need to be done (but I'm not sure how exactly to do them!) 1) get the eSDK itself to generate, which will likely require some minor devtool changes to support this kind of setup. Does any one have any ideas what might be required here? 2) Once we have a working eSDK, we need to figure out how to work on things that are not in the primary (local.conf) configuration. Format wise, we could attempt to teach devtool about the 'mc:multiconfig:recipe' syntax. Is this really the right behavior to consider, and does anyone have any idea as to the complications on doing this? Any other thoughts on how this could be done? My goal is to try to get at least the eSDK (primary config) working for the fall release, so I know this isn't a huge amount of time for the implementation. --Mark
|
|
OpenEmbedded Happy Hour May 26 8pm/2000 UTC
Denys Dmytriyenko
Hi,
Due to the Yocto Project Summit running during our regular OpenEmbedded Happy Hour today, we decided to move it down by 3 hours and have Happy Hour at the end of the Summit, May 26 @ 2000/8pm UTC (4pm ET / 1pm PT). Please join us and socialize with fellow developers and have some good time. https://www.openembedded.org/wiki/Calendar https://www.timeanddate.com/worldclock/fixedtime.html?msg=OpenEmbedded+Happy+Hour+May+26&iso=20210526T20 -- Regards, Denys Dmytriyenko <denis@...> PGP: 0x420902729A92C964 - https://denix.org/0x420902729A92C964 Fingerprint: 25FC E4A5 8A72 2F69 1186 6D76 4209 0272 9A92 C964
|
|
[OEDVM 2021] Better BSPs
Rich Persaud
> https://www.openembedded.org/wiki/OEDVM_2021 > 1530 UTC on Zoom (link in the URL above) In preparation for the BSP discussion at today's OEDVM, the following background material may be helpful. Comments on BSP patterns, anti-patterns, incentives, constraints and other topics are welcome in this mailing list thread, so they can be organized ahead of the 30-minute session. 1. "How to write a really good BSP layer" -- Chris Simmonds (Feb 2020) 2. "State of BSPs" -- Beth Flanagan (Feb 2020) 3. "Tips for Yocto BSPs" -- Cliff Brake & Khem Raj (Feb 2021) https://tmpdir.org/009/ (patterns & anti-patterns) RESOURCES * Layer index: http://layers.openembedded.org/ * Dev guide: https://docs.yoctoproject.org/bsp-guide/index.html MINUTES OF PRIOR OE DEVELOPER MEETINGS NOV 2019 (LYON) BSPs: carrot versus stick 14:20 Crofton: How many use vendor BSPs? (show of hands) ~30% Andrey: Most fixes are not pushed back to the vendor BPSs. Scott: AGL is not prepared to get input via the layers Jan-Simon: That’s not really the case. Richard: Would it help if the BSPs were Yocto compatible? It should not change anything just when including the layer. Consensus: that would help. Mark H: Yocto compatibility is one part of the problem, the other is that the BSPs do not use the version of the kernel that he wants. Jon: The vendors also do not support old kernel releases in the Ubuntu-variant images. Mark A: They are also covering multiple boards with the same release, so the kernel is trailing. Richard: The compatibility program has not been pushed as hard as it could. Mark H: The layer index should show if a layer is Yocto Compatible. It should also be stated clearly what kernel version the BSP layer supports. Andrey: That would make it harder for the users. Marek: These BSP are more like a tech demo. Josef: They are also shipping arbitrary .bbappends. Jan-Simon: They cannot enable all BSPs at once. Friendly layers would help. Isolation is key. If they ship libc-headers, it will break a package feed. Richard: linux-libc-headers is just for building the libc, don’t use it for anything else. Scott: Some vendors with the bad BSP are yocto project members. William: The tools for checking compatibility have gotten a lot better. Scott: Koen has been talking about this for years. Richard: meta-oe doesn’t pass compatibility with Zeus right now. Mark H: Wants to use the linux-yocto recipe, for fragments and moving to a different board. Josef: There is a How to setup a BSP layer, and how to create a kernel recipe. Paul: The documentation on how to build a clean BSP layer is not understandable by the usual BSP builders. Mark H: The layerindex should show a checkmark for Yocto compatible layers. Mikko: Some vendor BSPs are not public. Mark H: Money speaks. Jan: Many vendors say that their BSP is just for evaluation purposes. OCT 2018 (EDINBURGH) Ndec: setup Google Analytics on layers.openembedded.org (and probably openembedded.org) Scott: we should run the existing checkers on known layers Marco Cavallini: Are there rules to submitting layers to the index? Some review, check that it’s reusable for other projects (yocto compatible) Crofton recommends to run yocto-check-layer (especially BSP layers) Richard: the layer-index can find problems easily which are not detected by check-layer Mark Hatle: Machine and distro checking works well Richard: check-layer uses the sstate checksums to verify that a layer affects only things that it should touch (machine/distro level) when enabled
|
|
Re: Open Source Maintainers - An open letter/request
Richard Purdie
On Mon, 2021-05-10 at 13:52 -0700, akuster808 wrote:
I think people will know what is appropriate in their own circumstances. There was an ask that some of these things be "documented" and I'm trying to do that as I think some of the things here are often overlooked or misunderstood. Cheers, Richard
|
|
Re: Open Source Maintainers - An open letter/request
On 5/10/21 8:14 AM, Richard Purdie wrote:
TLDR: The project is seen as mature, employers don't prioritise maintainingThanks for summarizing all this. So is the ask to forward this within the one's Employer? -armin
|
|
Open Source Maintainers - An open letter/request
Richard Purdie
TLDR: The project is seen as mature, employers don't prioritise maintaining
things and we're struggling for maintainers and help with day to day work Open source projects survive, not just through development work and contributions of new features but through a whole load of "unglamorous" day to day "admin" work. This may be tracking down a regression, triaging failing builds, making a release of a component, reviewing a patch, documenting something or many other activities. I love the fact we have active contributions, particularly for new features but we are continuing to struggle in many of the other areas above. I am extrememly grateful for the help we do receive with these tasks! As a project we have automated an absolute ton of things, we can test changes in ways we could only dream of a few years ago but maintaining this automation, tracking down regressions and ensuring it all stays working does have a cost. I am worried, not just about the core of the project, but the wider layer ecosystem since "layer maintainer" isn't seen as a particularly interesting career enabling focus by employers and it seems a lot of this work isn't being recognised. Internal business pressures are often continually being prioritised over this. The YP+OE ecosystem is becoming more mature and this means we have our experienced developers being pulled away to new things and few people are replacing them so it feels like we're seeing a gradual skills drain/fade. There are a few things companies can do to help: a) Publicly acknowledge you use the project. I'm often asked where the project is being used but I find it hard to point at companies using it, or products developed with it. It does help to be able to point at real users rather than theoretical scenarios. We *know* it is used in some interesting places but many won't let us say that publicly. https://wiki.yoctoproject.org/wiki/Project_Users b) Embrace employee's Open Source contributions, code and otherwise If companies can find ways to recognise the value of having open source experts/leaders working for them from a career development and reward perspective, that would encourage people to do the important work needed c) Consider Yocto Project membership https://www.yoctoproject.org/ecosystem/members/ https://www.yoctoproject.org/join/ We're finding that some infrastructure and roles need to be centrally funded as the work is important but no one company is willing to commit people to it. We're only able to to this through project membership which supports things like the autobuilder, LTS, our build triage process and my own role. d) Support employees in spending some time on open source projects I hear quite often that employees get XX% time to spend on open source projects. I also hear they get pulled onto mission critical product deliverables and can't prioritise that other project work. Finding ways to ensure employees can spend time on open source projects including management support would help a lot. e) Transition roles If someone has a key role in a project but is moving to new things, help them find a replacement and allow them time to train/transition to that new person. Some companies do this really well, I'd call out NI and opkg maintainership as a particularly good exmaple. I appreciate these are difficult times, both for individuals and for businesses. I'd like to conclude by thanking everyone who does participate and contribute. Whilst I do want/need to highlight the above (and have been asked to do so that people have something they can point people at), the project is proving to be successful, going to interesting places and making things possible we can all be proud of! Cheers, Richard
|
|