Re: New assignment operator?
Andre McCurdy
On Wed, Jun 16, 2021 at 2:22 AM Phil Blundell via
lists.openembedded.org <pb=pbcl.net@...> wrote:
override somehow worse than inventing a new assignment operator to do
the same thing?
If we want to make a rule that nothing in oe-core should ever
forcefully override a setting provided by a BSP then that's OK... but
then isn't that an argument that a new assignment operator to
forcefully assign to variables shouldn't be used in oe-core?
doesn't really scale well. There will always be BSP layers which need
to support a collection of machines and the obvious way to do that is
to use default values and machine specific overrides within the BSP
layer. Making a rule that BSP layers can't use overrides for certain
variables will cause confusion and frustration.
I think the solution should be either that oe-core doesn't rely on
modifying variables set by the BSP (ie the initramfs images should be
able to signal to the image.bbclass that they want to use
INITRAMFS_FSTYPES via some other method than modifying IMAGE_FSTYPES)
or that we agree that oe-core can forcefully override the BSP
variables in certain well defined cases (and then the _forcevariable
override appears to be a good solution).
lists.openembedded.org <pb=pbcl.net@...> wrote:
I'm not sure I understand this. Why is using the _forcevariable
On Tue, Jun 15, 2021 at 10:59:40PM +0100, Richard Purdie wrote:It certainly could be a solution to the problem but I worry that havingYes, that's a fair point. As you say, I kind of have the sense that our
variables that users should touch and not touch and variables shadowing
other variables is going to get complex rather quickly, even when you just
consider how you document or "enforce" it. Do we want to encourage what
could become many more intermiediate variables? It does sound like we're
effectively starting to write class "functions" with a list of variables
they operate on which may or may not be a good thing depending on how
common it became.
classes are drifting towards being more like classes in a "real" programming
language, complete with an actual API, and possibly we ought to start
thinking about them more in those terms. As in, "if you include image.bbclass,
this is the set of variables it takes as input, and you need to give them
appropriate values".From descriptions from others it sounds like people have creatively workedI agree. Although it's true that _forcevariable probably would fix this
around this kind of issue with stronger overrides which I'd not considered
and would also work, it is just a little ugly in a different way. I'd consider
use of the forcevariable override in oe-core a metadata failing (much like
I view _remove similarly in core).
particular issue I think there's a fair chance that it will just end up creating
the opposite problem at some point in the future. I don't honestly think there
is any situation where using _forcevariable in oe-core itself would be the
right thing to do.
override somehow worse than inventing a new assignment operator to do
the same thing?
If we want to make a rule that nothing in oe-core should ever
forcefully override a setting provided by a BSP then that's OK... but
then isn't that an argument that a new assignment operator to
forcefully assign to variables shouldn't be used in oe-core?
Rewriting the BSP would work but it's sidestepping the issue andI suppose the other obvious tactical answer to the problem at hand is formeta-zaurus to stop using an override at all for what it's doing. Although
possibly not quite so elegant, there's no reason that it couldn't do:
IMAGE_FSTYPES = "ext3"
# nb, include not require...
include conf/machine/zaurus/${MACHINE}-special.conf
and then in conf/machine/zaurus/collie-special.conf it could just set
IMAGE_FSTYPES = "squashfs"
directly without needing any kind of override. That said I don't entirely
understand why meta-zaurus doesn't already have a collie.conf (assuming
MACHINE=collie) where it could be doing this...
doesn't really scale well. There will always be BSP layers which need
to support a collection of machines and the obvious way to do that is
to use default values and machine specific overrides within the BSP
layer. Making a rule that BSP layers can't use overrides for certain
variables will cause confusion and frustration.
I think the solution should be either that oe-core doesn't rely on
modifying variables set by the BSP (ie the initramfs images should be
able to signal to the image.bbclass that they want to use
INITRAMFS_FSTYPES via some other method than modifying IMAGE_FSTYPES)
or that we agree that oe-core can forcefully override the BSP
variables in certain well defined cases (and then the _forcevariable
override appears to be a good solution).
Re: New assignment operator?
Otavio Salvador
Em qua., 16 de jun. de 2021 às 02:08, Chen Qi <Qi.Chen@...> escreveu:
--
Otavio Salvador O.S. Systems
http://www.ossystems.com.br http://code.ossystems.com.br
Mobile: +55 (53) 9 9981-7854 Mobile: +1 (347) 903-9750
Anyone proposed '==='? (I'm thinking about the 'always equal' sign in math.)I really like the === proposal.
--
Otavio Salvador O.S. Systems
http://www.ossystems.com.br http://code.ossystems.com.br
Mobile: +55 (53) 9 9981-7854 Mobile: +1 (347) 903-9750
Re: New assignment operator?
Phil Blundell
On Tue, Jun 15, 2021 at 10:59:40PM +0100, Richard Purdie wrote:
classes are drifting towards being more like classes in a "real" programming
language, complete with an actual API, and possibly we ought to start
thinking about them more in those terms. As in, "if you include image.bbclass,
this is the set of variables it takes as input, and you need to give them
appropriate values".
particular issue I think there's a fair chance that it will just end up creating
the opposite problem at some point in the future. I don't honestly think there
is any situation where using _forcevariable in oe-core itself would be the
right thing to do.
I suppose the other obvious tactical answer to the problem at hand is for
meta-zaurus to stop using an override at all for what it's doing. Although
possibly not quite so elegant, there's no reason that it couldn't do:
IMAGE_FSTYPES = "ext3"
# nb, include not require...
include conf/machine/zaurus/${MACHINE}-special.conf
and then in conf/machine/zaurus/collie-special.conf it could just set
IMAGE_FSTYPES = "squashfs"
directly without needing any kind of override. That said I don't entirely
understand why meta-zaurus doesn't already have a collie.conf (assuming
MACHINE=collie) where it could be doing this...
p.
It certainly could be a solution to the problem but I worry that havingYes, that's a fair point. As you say, I kind of have the sense that our
variables that users should touch and not touch and variables shadowing
other variables is going to get complex rather quickly, even when you just
consider how you document or "enforce" it. Do we want to encourage what
could become many more intermiediate variables? It does sound like we're
effectively starting to write class "functions" with a list of variables
they operate on which may or may not be a good thing depending on how
common it became.
classes are drifting towards being more like classes in a "real" programming
language, complete with an actual API, and possibly we ought to start
thinking about them more in those terms. As in, "if you include image.bbclass,
this is the set of variables it takes as input, and you need to give them
appropriate values".
From descriptions from others it sounds like people have creatively workedI agree. Although it's true that _forcevariable probably would fix this
around this kind of issue with stronger overrides which I'd not considered
and would also work, it is just a little ugly in a different way. I'd consider
use of the forcevariable override in oe-core a metadata failing (much like
I view _remove similarly in core).
particular issue I think there's a fair chance that it will just end up creating
the opposite problem at some point in the future. I don't honestly think there
is any situation where using _forcevariable in oe-core itself would be the
right thing to do.
I suppose the other obvious tactical answer to the problem at hand is for
meta-zaurus to stop using an override at all for what it's doing. Although
possibly not quite so elegant, there's no reason that it couldn't do:
IMAGE_FSTYPES = "ext3"
# nb, include not require...
include conf/machine/zaurus/${MACHINE}-special.conf
and then in conf/machine/zaurus/collie-special.conf it could just set
IMAGE_FSTYPES = "squashfs"
directly without needing any kind of override. That said I don't entirely
understand why meta-zaurus doesn't already have a collie.conf (assuming
MACHINE=collie) where it could be doing this...
p.
Re: New assignment operator?
Chen Qi
On 06/15/2021 06:29 PM, Richard Purdie
wrote:
We have an interesting problem with initramfs images. They do something like: IMAGE_FSTYPES = "${INITRAMFS_FSTYPES}" which looks innocent enough until a BSP does: IMAGE_FSTYPES_collie = "xxx" where collie is MACHINE, hence an override which causes confusion as the value of INITRAMFS_FSTYPES is never used for that machine. This is a long standing issue where in standard metadata, there is no way to undo an override (or append/prepend/remove). I tried to be clever and did: python () { d.setVar("IMAGE_FSTYPES", d.getVar("INITRAMFS_FSTYPES")) } and if you do that before the image class inherit (since anon python is executed in the order it is defined), it nearly works. It doesn't work with live images as IMAGE_FSTYPES is expanded for class inherits before the anonymous python executes. I also wondered about: IMAGE_FSTYPES := "${INITRAMFS_FSTYPES}" but that doesn't clear overrides either. In fact I can't actually see a way to do it other than some horrible thing like: def myfunc(d): d.setVar("IMAGE_FSTYPES", d.getVar("INITRAMFS_FSTYPES")) DUMMYVAR := "${@myfunc(d)}" Yes, we could do that but I don't think we should. I wondered about making := clear existing overrides/appends/prepends/removes but then I wondered if we should have a new assignment operator which would do that (and also not force the value to be expanded). I mentioned this on irc and there were a few suggestions: IMAGE_FSTYPES =!! "${INITRAMFS_FSTYPES}" IMAGE_FSTYPES !=! "${INITRAMFS_FSTYPES}" IMAGE_FSTYPES *:= "${INITRAMFS_FSTYPES}" IMAGE_FSTYPES :=* "${INITRAMFS_FSTYPES}" The questions are a) is another assignment operator a good idea?
I think so.
1) In the view of compatibility, it's acceptable.
Before the change, we have:
_remove > _append > OVERRIDES > normal assignments
After the change, we have:
!=! > _remove > _append > OVERRIDES > normal assignments
So it does not break anything
2) In the view of scope, it's necessary.
Recipe has smaller effective scope than conf file. Good design should be: it has higher priority than conf file.
The general principle is: smaller scope, higher priority.
For example, $HOME/.someconf beats /etc/someconf.
The current problem with bitbake/yocto/oe is that recipes cannot have the final call. They have smaller scope than conf files, yet are of the same priority regarding variable settings.
This problem can only be solved at the bitbake level. So adding a new assignment operator to let the recipe be able to make the final call is necessary.
b) if so, which form? Implementation shouldn't be hard in bitbake, the APIs are already there to do this kind of operation. I am kind of preferring !=!.
Any form is OK.
'!=!' seems a very natural choice for vi users. However, for users like me, it looks like 'never equals to' at a first glance.
Anyone proposed '==='? (I'm thinking about the 'always equal' sign in math.)
f(x) === C means no matter what x is, f always equals to C.
VAR(history) === C means no matter what the history is, the VAR always equals to C.
To clarify, I used 'VAR(history)' above to show that in bitbake a VAR's final value depends on its setting history.
Best Regards,
Chen Qi
Cheers, Richard
Re: New assignment operator?
Richard Purdie
On Tue, 2021-06-15 at 17:10 +0200, Phil Blundell wrote:
variables that users should touch and not touch and variables shadowing
other variables is going to get complex rather quickly, even when you just
consider how you document or "enforce" it. Do we want to encourage what
could become many more intermiediate variables? It does sound like we're
effectively starting to write class "functions" with a list of variables
they operate on which may or may not be a good thing depending on how
common it became.
From descriptions from others it sounds like people have creatively worked
around this kind of issue with stronger overrides which I'd not considered
and would also work, it is just a little ugly in a different way. I'd consider
use of the forcevariable override in oe-core a metadata failing (much like
I view _remove similarly in core).
Which does bring us back to whether adding something allowing the initramfs
recipes to do the right thing would be a better solution? An initramfs
class to abstract it whatever we do may also be worth considering but is
tangential.
Cheers,
Richard
On Tue, Jun 15, 2021 at 02:29:16PM +0100, Richard Purdie wrote:That is a good summary, yes.For that reason I would like to change the initramfs recipe somehow toThanks for the background, I understand the issue a bit better now.
improve usability and ensure people don't hit this. Right now I can't see
any way to do that other than to say "don't do that". I can't even add
anything to tell the user there is a problem. This was the spirit the
proposal was born from. I understand why people don't like any new operator,
I'm not thrilled either but what I'm not seeing are alternatives to improve
usability :/.
It seems fairly apparent that what's wrong here isn't fundamentally an issue
with OVERRIDES, it's more that the semantic layering (in the generic sense of
the word, not meaning OE layers) has just gone wrong with IMAGE_FSTYPES. So
you have this one variable that, on one hand, is a user-tweakable knob that
they can set to say "this is the type of filesystem image I want to generate
for my particular machine", and on the other hand is part of this shadow
interface contract between the initramfs code and the image-construction
backend. meta-zaurus is trying to change the first one and accidentally
clobbering the second one and I agree it can't really do much better with
the current infrastructure.
So, the simplest way of fixing the immediate problem seems to be to separateIt certainly could be a solution to the problem but I worry that having
the two things out: make image.bbclass look at some new variable
(IMAGE_GENERATE_FSTYPES or something), teach the initramfs code to set that
variable rather than IMAGE_FSTYPES, and provide a soft default of
IMAGE_GENERATE_FSTYPES ??= ${IMAGE_FSTYPES}.
In the more general case it seems like any given variable ought to be
classifiable as something that's either settable by the user configuration
(including the BSP in that) or that is set by some other class in oe-core,
but I think we ought to try to avoid having variables that fall into both
groups.
variables that users should touch and not touch and variables shadowing
other variables is going to get complex rather quickly, even when you just
consider how you document or "enforce" it. Do we want to encourage what
could become many more intermiediate variables? It does sound like we're
effectively starting to write class "functions" with a list of variables
they operate on which may or may not be a good thing depending on how
common it became.
From descriptions from others it sounds like people have creatively worked
around this kind of issue with stronger overrides which I'd not considered
and would also work, it is just a little ugly in a different way. I'd consider
use of the forcevariable override in oe-core a metadata failing (much like
I view _remove similarly in core).
Which does bring us back to whether adding something allowing the initramfs
recipes to do the right thing would be a better solution? An initramfs
class to abstract it whatever we do may also be worth considering but is
tangential.
Cheers,
Richard
Re: New assignment operator?
Andre McCurdy
On Tue, Jun 15, 2021 at 8:48 AM Mark Hatle
<mark.hatle@...> wrote:
initramfs recipe wants to forcefully over-ride a value of
IMAGE_FSTYPES set by a BSP, then isn't the existing _forcevariable
over-ride the way to do that?
<mark.hatle@...> wrote:
On 6/15/21 10:10 AM, Phil Blundell via lists.openembedded.org wrote:I skimmed some of the thread so may have missed a detail... but if theOn Tue, Jun 15, 2021 at 02:29:16PM +0100, Richard Purdie wrote:There is more then once I thought it would be good to add a flag for some sortFor that reason I would like to change the initramfs recipe somehow toThanks for the background, I understand the issue a bit better now.
improve usability and ensure people don't hit this. Right now I can't see
any way to do that other than to say "don't do that". I can't even add
anything to tell the user there is a problem. This was the spirit the
proposal was born from. I understand why people don't like any new operator,
I'm not thrilled either but what I'm not seeing are alternatives to improve
usability :/.
It seems fairly apparent that what's wrong here isn't fundamentally an issue
with OVERRIDES, it's more that the semantic layering (in the generic sense of
the word, not meaning OE layers) has just gone wrong with IMAGE_FSTYPES. So
you have this one variable that, on one hand, is a user-tweakable knob that
they can set to say "this is the type of filesystem image I want to generate
for my particular machine", and on the other hand is part of this shadow
interface contract between the initramfs code and the image-construction
backend. meta-zaurus is trying to change the first one and accidentally
clobbering the second one and I agree it can't really do much better with
the current infrastructure.
So, the simplest way of fixing the immediate problem seems to be to separate
the two things out: make image.bbclass look at some new variable
(IMAGE_GENERATE_FSTYPES or something), teach the initramfs code to set that
variable rather than IMAGE_FSTYPES, and provide a soft default of
IMAGE_GENERATE_FSTYPES ??= ${IMAGE_FSTYPES}.
In the more general case it seems like any given variable ought to be
classifiable as something that's either settable by the user configuration
(including the BSP in that) or that is set by some other class in oe-core,
but I think we ought to try to avoid having variables that fall into both
groups.
of variable type. Specific for this reason. "This variable is a MACHINE
variable, this is a DISTRO variable, this is a recipe variable, this is a class
variable, etc." Of course identifying what we're doing and warning may not be
so easy, but it could help. Along with this identify where an override is being
used, if it's a MACHINE override it probably shouldn't be in a DISTRO file for
instance.
The case where I was curious if we could do something like this very different.
I saw MACHINE files setting what are effectively distribution features, and
distro.conf files having machine overrides in them.
All tend to lead to problems like this in the long term, variables that get set
and can't easily be "unset".
initramfs recipe wants to forcefully over-ride a value of
IMAGE_FSTYPES set by a BSP, then isn't the existing _forcevariable
over-ride the way to do that?
Re: New assignment operator?
Mark Hatle
On 6/15/21 10:10 AM, Phil Blundell via lists.openembedded.org wrote:
of variable type. Specific for this reason. "This variable is a MACHINE
variable, this is a DISTRO variable, this is a recipe variable, this is a class
variable, etc." Of course identifying what we're doing and warning may not be
so easy, but it could help. Along with this identify where an override is being
used, if it's a MACHINE override it probably shouldn't be in a DISTRO file for
instance.
The case where I was curious if we could do something like this very different.
I saw MACHINE files setting what are effectively distribution features, and
distro.conf files having machine overrides in them.
All tend to lead to problems like this in the long term, variables that get set
and can't easily be "unset".
--Mark
On Tue, Jun 15, 2021 at 02:29:16PM +0100, Richard Purdie wrote:There is more then once I thought it would be good to add a flag for some sortFor that reason I would like to change the initramfs recipe somehow toThanks for the background, I understand the issue a bit better now.
improve usability and ensure people don't hit this. Right now I can't see
any way to do that other than to say "don't do that". I can't even add
anything to tell the user there is a problem. This was the spirit the
proposal was born from. I understand why people don't like any new operator,
I'm not thrilled either but what I'm not seeing are alternatives to improve
usability :/.
It seems fairly apparent that what's wrong here isn't fundamentally an issue
with OVERRIDES, it's more that the semantic layering (in the generic sense of
the word, not meaning OE layers) has just gone wrong with IMAGE_FSTYPES. So
you have this one variable that, on one hand, is a user-tweakable knob that
they can set to say "this is the type of filesystem image I want to generate
for my particular machine", and on the other hand is part of this shadow
interface contract between the initramfs code and the image-construction
backend. meta-zaurus is trying to change the first one and accidentally
clobbering the second one and I agree it can't really do much better with
the current infrastructure.
So, the simplest way of fixing the immediate problem seems to be to separate
the two things out: make image.bbclass look at some new variable
(IMAGE_GENERATE_FSTYPES or something), teach the initramfs code to set that
variable rather than IMAGE_FSTYPES, and provide a soft default of
IMAGE_GENERATE_FSTYPES ??= ${IMAGE_FSTYPES}.
In the more general case it seems like any given variable ought to be
classifiable as something that's either settable by the user configuration
(including the BSP in that) or that is set by some other class in oe-core,
but I think we ought to try to avoid having variables that fall into both
groups.
of variable type. Specific for this reason. "This variable is a MACHINE
variable, this is a DISTRO variable, this is a recipe variable, this is a class
variable, etc." Of course identifying what we're doing and warning may not be
so easy, but it could help. Along with this identify where an override is being
used, if it's a MACHINE override it probably shouldn't be in a DISTRO file for
instance.
The case where I was curious if we could do something like this very different.
I saw MACHINE files setting what are effectively distribution features, and
distro.conf files having machine overrides in them.
All tend to lead to problems like this in the long term, variables that get set
and can't easily be "unset".
--Mark
p.
Re: New assignment operator?
Phil Blundell
On Tue, Jun 15, 2021 at 02:29:16PM +0100, Richard Purdie wrote:
It seems fairly apparent that what's wrong here isn't fundamentally an issue
with OVERRIDES, it's more that the semantic layering (in the generic sense of
the word, not meaning OE layers) has just gone wrong with IMAGE_FSTYPES. So
you have this one variable that, on one hand, is a user-tweakable knob that
they can set to say "this is the type of filesystem image I want to generate
for my particular machine", and on the other hand is part of this shadow
interface contract between the initramfs code and the image-construction
backend. meta-zaurus is trying to change the first one and accidentally
clobbering the second one and I agree it can't really do much better with
the current infrastructure.
So, the simplest way of fixing the immediate problem seems to be to separate
the two things out: make image.bbclass look at some new variable
(IMAGE_GENERATE_FSTYPES or something), teach the initramfs code to set that
variable rather than IMAGE_FSTYPES, and provide a soft default of
IMAGE_GENERATE_FSTYPES ??= ${IMAGE_FSTYPES}.
In the more general case it seems like any given variable ought to be
classifiable as something that's either settable by the user configuration
(including the BSP in that) or that is set by some other class in oe-core,
but I think we ought to try to avoid having variables that fall into both
groups.
p.
For that reason I would like to change the initramfs recipe somehow toThanks for the background, I understand the issue a bit better now.
improve usability and ensure people don't hit this. Right now I can't see
any way to do that other than to say "don't do that". I can't even add
anything to tell the user there is a problem. This was the spirit the
proposal was born from. I understand why people don't like any new operator,
I'm not thrilled either but what I'm not seeing are alternatives to improve
usability :/.
It seems fairly apparent that what's wrong here isn't fundamentally an issue
with OVERRIDES, it's more that the semantic layering (in the generic sense of
the word, not meaning OE layers) has just gone wrong with IMAGE_FSTYPES. So
you have this one variable that, on one hand, is a user-tweakable knob that
they can set to say "this is the type of filesystem image I want to generate
for my particular machine", and on the other hand is part of this shadow
interface contract between the initramfs code and the image-construction
backend. meta-zaurus is trying to change the first one and accidentally
clobbering the second one and I agree it can't really do much better with
the current infrastructure.
So, the simplest way of fixing the immediate problem seems to be to separate
the two things out: make image.bbclass look at some new variable
(IMAGE_GENERATE_FSTYPES or something), teach the initramfs code to set that
variable rather than IMAGE_FSTYPES, and provide a soft default of
IMAGE_GENERATE_FSTYPES ??= ${IMAGE_FSTYPES}.
In the more general case it seems like any given variable ought to be
classifiable as something that's either settable by the user configuration
(including the BSP in that) or that is set by some other class in oe-core,
but I think we ought to try to avoid having variables that fall into both
groups.
p.
Re: New assignment operator?
Tom Rini
On Tue, Jun 15, 2021 at 11:29:01AM +0100, Richard Purdie wrote:
think.
This is because an initramfs can be more than one of:
- The only filesystem container we create for a MACHINE
- A filesystem container that gets bundled inside of the kernel vmlinux.
- A filesystem container that gets bundled inside of a container along
side a kernel (zImage, Image, etc), one or more device trees and
perhaps other blobs.
- A filesystem container that gets placed inside of another filesystem
via wic.
And once you start to add in the why for those use cases, things can get
racey too.
once you start having multiple cases from the list I made above in use.
When you aren't dealing with public layers, the fix is "easy" and it's
to adjust the machine confs and image recipes to not do what they were
doing.
What I want to suggest is that the problem is with how we treat
initramfs like it's another image, when it's not. What if for this
problem we treat INITRAMFS_FSTYPES separate from IMAGE_FSTYPES? Make
them "inherit initramfs" which in turn generates filesystem containers
based only on what INITRAMFS_FSTYPES evaluates out to. And soft-assign
that to cpio.gz by default.
last operation? One of:
FOO_remove = "bar"
FOO_forcevariable_remove = "baz"
really ought to do what I mean and take bar or baz out of FOO. Even if
some part of OVERRIDE prior to forcevariable puts it in.
--
Tom
We have an interesting problem with initramfs images.I'm going to intentionally avoid using "image" as gets overloaded I
think.
This is because an initramfs can be more than one of:
- The only filesystem container we create for a MACHINE
- A filesystem container that gets bundled inside of the kernel vmlinux.
- A filesystem container that gets bundled inside of a container along
side a kernel (zImage, Image, etc), one or more device trees and
perhaps other blobs.
- A filesystem container that gets placed inside of another filesystem
via wic.
And once you start to add in the why for those use cases, things can get
racey too.
They do something like:And this starts to be a lot more common of a problem than it might seem,
IMAGE_FSTYPES = "${INITRAMFS_FSTYPES}"
which looks innocent enough until a BSP does:
IMAGE_FSTYPES_collie = "xxx"
where collie is MACHINE, hence an override which causes confusion as
the value of INITRAMFS_FSTYPES is never used for that machine.
once you start having multiple cases from the list I made above in use.
When you aren't dealing with public layers, the fix is "easy" and it's
to adjust the machine confs and image recipes to not do what they were
doing.
What I want to suggest is that the problem is with how we treat
initramfs like it's another image, when it's not. What if for this
problem we treat INITRAMFS_FSTYPES separate from IMAGE_FSTYPES? Make
them "inherit initramfs" which in turn generates filesystem containers
based only on what INITRAMFS_FSTYPES evaluates out to. And soft-assign
that to cpio.gz by default.
This is a long standing issue where in standard metadata, there is no wayThis is a separate problem. Is there some way to make remove be the
to undo an override (or append/prepend/remove).
last operation? One of:
FOO_remove = "bar"
FOO_forcevariable_remove = "baz"
really ought to do what I mean and take bar or baz out of FOO. Even if
some part of OVERRIDE prior to forcevariable puts it in.
--
Tom
Re: New assignment operator?
Mark Hatle
On 6/15/21 9:15 AM, Richard Purdie wrote:
issue). What I end up doing is (in the initramfs) is effectively:
OVERRIDES .= ":ramfs"
IMAGE_FSTYPES_ramfs = "cpio"
doing this in the recipe (or a bbclass that should only be loaded by a recipe)
ensures that we can set IMAGE_FSTYPE to a specific value, and avoids the
specific issue mentioned here.
(At least it worked well in the case I needed it for, not sure if it would here,
but it may be what is needed.)
--Mark
On Tue, 2021-06-15 at 16:50 +0300, Petr Nechaev wrote:I've hit this issue in the past (different context, but I believe the sameHi All,Consider this example where the BSP sets:
I've been using bitbake for >5 years now, however I do not understand
the nature of the problem at all. Could you clarify what the problem is
i.e. what's the difference between these lines:
IMAGE_FSTYPES = "${INITRAMFS_FSTYPES}"
VAR_A = "${VAR_B}"
IMAGE_FSTYPES = "tar"
IMAGE_FSTYPES_somemachine = "ext4"
INITRAMFS_FSTYPES = "cpio"
OVERRIDES = "${MACHINE}"
then the initramfs image recipe does:
IMAGE_FSTYPES = "${INITRAMFS_FSTYPES}"
Given that config, if you set MACHINE = "somemachine", which image
type would you expect the initramfs image to have?
The answer is that it will try and generate an ext4 initramfs.
What you'd expect/want is a cpio initramfs.
The reason is that the machine override, i.e.:
IMAGE_FSTYPES_somemachine = "ext4"
cannot be undone.
issue). What I end up doing is (in the initramfs) is effectively:
OVERRIDES .= ":ramfs"
IMAGE_FSTYPES_ramfs = "cpio"
doing this in the recipe (or a bbclass that should only be loaded by a recipe)
ensures that we can set IMAGE_FSTYPE to a specific value, and avoids the
specific issue mentioned here.
(At least it worked well in the case I needed it for, not sure if it would here,
but it may be what is needed.)
--Mark
Cheers,
Richard
Re: New assignment operator?
Tom Rini
On Tue, Jun 15, 2021 at 03:09:03PM +0100, Richard Purdie wrote:
--
Tom
On Tue, 2021-06-15 at 09:38 -0400, Tom Rini wrote:Ah! Just slightly before my starting time with the project :)Thanks for all the details here. Since this is stemming from a specificI haven't tried to obscure it, the machine name was in the original email
BSP, I think at this point it might be good to share what exactly it is,
and it wouldn't be seen as "shaming" that BSP at this point as it's
exposed a rather, as you note, obtuse problem.
but I guess not everyone remembers Zarus models names like I do! :)
The BSP layer in question is meta-handheld, MACHINE=collie.OK, thanks. Pondering time...
https://github.com/andrea-adami/meta-handheld
Recipes which have the issue are in meta-initramfs in meta-oe, or there
are several initramfs images in OE-Core which also reset IMAGE_FSTYPES.
--
Tom
Re: New assignment operator?
Richard Purdie
On Tue, 2021-06-15 at 16:50 +0300, Petr Nechaev wrote:
IMAGE_FSTYPES = "tar"
IMAGE_FSTYPES_somemachine = "ext4"
INITRAMFS_FSTYPES = "cpio"
OVERRIDES = "${MACHINE}"
then the initramfs image recipe does:
IMAGE_FSTYPES = "${INITRAMFS_FSTYPES}"
Given that config, if you set MACHINE = "somemachine", which image
type would you expect the initramfs image to have?
The answer is that it will try and generate an ext4 initramfs.
What you'd expect/want is a cpio initramfs.
The reason is that the machine override, i.e.:
IMAGE_FSTYPES_somemachine = "ext4"
cannot be undone.
Cheers,
Richard
Hi All,Consider this example where the BSP sets:
I've been using bitbake for >5 years now, however I do not understand
the nature of the problem at all. Could you clarify what the problem is
i.e. what's the difference between these lines:
IMAGE_FSTYPES = "${INITRAMFS_FSTYPES}"
VAR_A = "${VAR_B}"
IMAGE_FSTYPES = "tar"
IMAGE_FSTYPES_somemachine = "ext4"
INITRAMFS_FSTYPES = "cpio"
OVERRIDES = "${MACHINE}"
then the initramfs image recipe does:
IMAGE_FSTYPES = "${INITRAMFS_FSTYPES}"
Given that config, if you set MACHINE = "somemachine", which image
type would you expect the initramfs image to have?
The answer is that it will try and generate an ext4 initramfs.
What you'd expect/want is a cpio initramfs.
The reason is that the machine override, i.e.:
IMAGE_FSTYPES_somemachine = "ext4"
cannot be undone.
Cheers,
Richard
Re: New assignment operator?
Richard Purdie
On Tue, 2021-06-15 at 09:38 -0400, Tom Rini wrote:
but I guess not everyone remembers Zarus models names like I do! :)
The BSP layer in question is meta-handheld, MACHINE=collie.
https://github.com/andrea-adami/meta-handheld
Recipes which have the issue are in meta-initramfs in meta-oe, or there
are several initramfs images in OE-Core which also reset IMAGE_FSTYPES.
Cheers,
Richard
Thanks for all the details here. Since this is stemming from a specificI haven't tried to obscure it, the machine name was in the original email
BSP, I think at this point it might be good to share what exactly it is,
and it wouldn't be seen as "shaming" that BSP at this point as it's
exposed a rather, as you note, obtuse problem.
but I guess not everyone remembers Zarus models names like I do! :)
The BSP layer in question is meta-handheld, MACHINE=collie.
https://github.com/andrea-adami/meta-handheld
Recipes which have the issue are in meta-initramfs in meta-oe, or there
are several initramfs images in OE-Core which also reset IMAGE_FSTYPES.
Cheers,
Richard
Re: New assignment operator?
Petr Nechaev
Hi All,
I've been using bitbake for >5 years now, however I do not understand the nature of the problem at all. Could you clarify what the problem is i.e. what's the difference between these lines:
IMAGE_FSTYPES = "${INITRAMFS_FSTYPES}"
VAR_A = "${VAR_B}"
In both cases .. it seems .. I am prepared that VAR_C = "${VAR_A}" won't expand to VAR_B due to overrides, appends etc. What is special about initramfs image? Is there any special meaning to IMAGE_FSTYPES variable?
If not ... are we trying to introduce a language feature to solve one particular problem of a specific image?
---
Petr
On Tue, Jun 15, 2021 at 4:38 PM Tom Rini <trini@...> wrote:
On Tue, Jun 15, 2021 at 02:29:16PM +0100, Richard Purdie wrote:
> I should apologise for being a little grumpy in some of my replies,
> it is fair to say that everything is getting to me a little as continual
> build failures and being continually asked for reasoned arguments for
> saying "no" to things is wearing me down. We have bugs in many core pieces
> of the system (pseudo, patchelf, prelink, ltp, oeqa, devtool, eSDK and so
> on) and currently it feels like I'm the only person with the domain
> knowledge to try and attempt to look into them. This shouldn't ripple
> out into emails though.
>
> The context of this issue is probably important and I didn't really
> mention it.
>
> I've been asked about a "bitbake bug" a lot on irc recently and asked
> for help in trying to resolve it. I spent quite a bit more time than
> expected (on my weekend) trying to understand the issue and it wasn't
> the issue as reported but a lot more subtle. In the emails here I've
> spelt out the problem but the way it becomes exposed to the end user
> is a lot more insidious.
>
> I don't think the BSP was doing anything wrong using a MACHINE override
> on a variable. The initramfs recipe was also not really doing anything
> wrong trying to set the fstypes to the initramfs ones.
>
> The interaction between the two things is rather unfortunate and in this
> case the BSP maintainer could not see why it was breaking and even me, with
> a few years experience with bitbake couldn't immediately understand what
> was wrong or how my own fix was going to break.
>
> Even now I think broken "fixes" are being spread around in attempts to try
> and work around the issue which swap on machine's breakage for another
> (collie works but qemux86 using image-live then doesn't).
>
> It does worry me a lot that the issue is so obtuse to debug and that whilst
> we can patch this one up, someone else can/will hit it again. The potential
> to hit it with some other variable also remains. I don't like issues that
> few people can "see" into and understand.
>
> For that reason I would like to change the initramfs recipe somehow to
> improve usability and ensure people don't hit this. Right now I can't see
> any way to do that other than to say "don't do that". I can't even add
> anything to tell the user there is a problem. This was the spirit the
> proposal was born from. I understand why people don't like any new operator,
> I'm not thrilled either but what I'm not seeing are alternatives to improve
> usability :/.
Thanks for all the details here. Since this is stemming from a specific
BSP, I think at this point it might be good to share what exactly it is,
and it wouldn't be seen as "shaming" that BSP at this point as it's
exposed a rather, as you note, obtuse problem.
I have another "could we just ..." idea on this, but I could answer that
myself maybe with the exact problem laid out.
--
Tom
Re: New assignment operator?
Tom Rini
On Tue, Jun 15, 2021 at 02:29:16PM +0100, Richard Purdie wrote:
BSP, I think at this point it might be good to share what exactly it is,
and it wouldn't be seen as "shaming" that BSP at this point as it's
exposed a rather, as you note, obtuse problem.
I have another "could we just ..." idea on this, but I could answer that
myself maybe with the exact problem laid out.
--
Tom
I should apologise for being a little grumpy in some of my replies,Thanks for all the details here. Since this is stemming from a specific
it is fair to say that everything is getting to me a little as continual
build failures and being continually asked for reasoned arguments for
saying "no" to things is wearing me down. We have bugs in many core pieces
of the system (pseudo, patchelf, prelink, ltp, oeqa, devtool, eSDK and so
on) and currently it feels like I'm the only person with the domain
knowledge to try and attempt to look into them. This shouldn't ripple
out into emails though.
The context of this issue is probably important and I didn't really
mention it.
I've been asked about a "bitbake bug" a lot on irc recently and asked
for help in trying to resolve it. I spent quite a bit more time than
expected (on my weekend) trying to understand the issue and it wasn't
the issue as reported but a lot more subtle. In the emails here I've
spelt out the problem but the way it becomes exposed to the end user
is a lot more insidious.
I don't think the BSP was doing anything wrong using a MACHINE override
on a variable. The initramfs recipe was also not really doing anything
wrong trying to set the fstypes to the initramfs ones.
The interaction between the two things is rather unfortunate and in this
case the BSP maintainer could not see why it was breaking and even me, with
a few years experience with bitbake couldn't immediately understand what
was wrong or how my own fix was going to break.
Even now I think broken "fixes" are being spread around in attempts to try
and work around the issue which swap on machine's breakage for another
(collie works but qemux86 using image-live then doesn't).
It does worry me a lot that the issue is so obtuse to debug and that whilst
we can patch this one up, someone else can/will hit it again. The potential
to hit it with some other variable also remains. I don't like issues that
few people can "see" into and understand.
For that reason I would like to change the initramfs recipe somehow to
improve usability and ensure people don't hit this. Right now I can't see
any way to do that other than to say "don't do that". I can't even add
anything to tell the user there is a problem. This was the spirit the
proposal was born from. I understand why people don't like any new operator,
I'm not thrilled either but what I'm not seeing are alternatives to improve
usability :/.
BSP, I think at this point it might be good to share what exactly it is,
and it wouldn't be seen as "shaming" that BSP at this point as it's
exposed a rather, as you note, obtuse problem.
I have another "could we just ..." idea on this, but I could answer that
myself maybe with the exact problem laid out.
--
Tom
Re: New assignment operator?
Richard Purdie
I should apologise for being a little grumpy in some of my replies,
it is fair to say that everything is getting to me a little as continual
build failures and being continually asked for reasoned arguments for
saying "no" to things is wearing me down. We have bugs in many core pieces
of the system (pseudo, patchelf, prelink, ltp, oeqa, devtool, eSDK and so
on) and currently it feels like I'm the only person with the domain
knowledge to try and attempt to look into them. This shouldn't ripple
out into emails though.
The context of this issue is probably important and I didn't really
mention it.
I've been asked about a "bitbake bug" a lot on irc recently and asked
for help in trying to resolve it. I spent quite a bit more time than
expected (on my weekend) trying to understand the issue and it wasn't
the issue as reported but a lot more subtle. In the emails here I've
spelt out the problem but the way it becomes exposed to the end user
is a lot more insidious.
I don't think the BSP was doing anything wrong using a MACHINE override
on a variable. The initramfs recipe was also not really doing anything
wrong trying to set the fstypes to the initramfs ones.
The interaction between the two things is rather unfortunate and in this
case the BSP maintainer could not see why it was breaking and even me, with
a few years experience with bitbake couldn't immediately understand what
was wrong or how my own fix was going to break.
Even now I think broken "fixes" are being spread around in attempts to try
and work around the issue which swap on machine's breakage for another
(collie works but qemux86 using image-live then doesn't).
It does worry me a lot that the issue is so obtuse to debug and that whilst
we can patch this one up, someone else can/will hit it again. The potential
to hit it with some other variable also remains. I don't like issues that
few people can "see" into and understand.
For that reason I would like to change the initramfs recipe somehow to
improve usability and ensure people don't hit this. Right now I can't see
any way to do that other than to say "don't do that". I can't even add
anything to tell the user there is a problem. This was the spirit the
proposal was born from. I understand why people don't like any new operator,
I'm not thrilled either but what I'm not seeing are alternatives to improve
usability :/.
Cheers,
Richard
it is fair to say that everything is getting to me a little as continual
build failures and being continually asked for reasoned arguments for
saying "no" to things is wearing me down. We have bugs in many core pieces
of the system (pseudo, patchelf, prelink, ltp, oeqa, devtool, eSDK and so
on) and currently it feels like I'm the only person with the domain
knowledge to try and attempt to look into them. This shouldn't ripple
out into emails though.
The context of this issue is probably important and I didn't really
mention it.
I've been asked about a "bitbake bug" a lot on irc recently and asked
for help in trying to resolve it. I spent quite a bit more time than
expected (on my weekend) trying to understand the issue and it wasn't
the issue as reported but a lot more subtle. In the emails here I've
spelt out the problem but the way it becomes exposed to the end user
is a lot more insidious.
I don't think the BSP was doing anything wrong using a MACHINE override
on a variable. The initramfs recipe was also not really doing anything
wrong trying to set the fstypes to the initramfs ones.
The interaction between the two things is rather unfortunate and in this
case the BSP maintainer could not see why it was breaking and even me, with
a few years experience with bitbake couldn't immediately understand what
was wrong or how my own fix was going to break.
Even now I think broken "fixes" are being spread around in attempts to try
and work around the issue which swap on machine's breakage for another
(collie works but qemux86 using image-live then doesn't).
It does worry me a lot that the issue is so obtuse to debug and that whilst
we can patch this one up, someone else can/will hit it again. The potential
to hit it with some other variable also remains. I don't like issues that
few people can "see" into and understand.
For that reason I would like to change the initramfs recipe somehow to
improve usability and ensure people don't hit this. Right now I can't see
any way to do that other than to say "don't do that". I can't even add
anything to tell the user there is a problem. This was the spirit the
proposal was born from. I understand why people don't like any new operator,
I'm not thrilled either but what I'm not seeing are alternatives to improve
usability :/.
Cheers,
Richard
Re: New assignment operator?
Richard Purdie
On Tue, 2021-06-15 at 09:32 -0300, Otavio Salvador wrote:
It doesn't care about IMAGE_FSTYPES, it cares about INITRAMFS_FSTYPES only.
If the BSP wants to change anything they'd be changing INITRAMFS_FSTYPES
for the initramfs recipe.
Cheers,
Richard
Hello Richard,Think about this in the context of what the initramfs recipe is trying to do.
Em ter., 15 de jun. de 2021 às 07:29, Richard Purdie
<richard.purdie@...> escreveu:I wondered about making := clear existing overrides/appends/prepends/removesHaving it resetting it might fix this use case but what if the _collie
but then I wondered if we should have a new assignment operator which
would do that (and also not force the value to be expanded). I mentioned this
on irc and there were a few suggestions:
use case is a valid one? how the BSP vendor could override it?
It doesn't care about IMAGE_FSTYPES, it cares about INITRAMFS_FSTYPES only.
If the BSP wants to change anything they'd be changing INITRAMFS_FSTYPES
for the initramfs recipe.
Cheers,
Richard
Re: New assignment operator?
Otavio Salvador
Hello Richard,
Em ter., 15 de jun. de 2021 às 07:29, Richard Purdie
<richard.purdie@...> escreveu:
use case is a valid one? how the BSP vendor could override it?
--
Otavio Salvador O.S. Systems
http://www.ossystems.com.br http://code.ossystems.com.br
Mobile: +55 (53) 9 9981-7854 Mobile: +1 (347) 903-9750
Em ter., 15 de jun. de 2021 às 07:29, Richard Purdie
<richard.purdie@...> escreveu:
I wondered about making := clear existing overrides/appends/prepends/removesHaving it resetting it might fix this use case but what if the _collie
but then I wondered if we should have a new assignment operator which
would do that (and also not force the value to be expanded). I mentioned this
on irc and there were a few suggestions:
use case is a valid one? how the BSP vendor could override it?
--
Otavio Salvador O.S. Systems
http://www.ossystems.com.br http://code.ossystems.com.br
Mobile: +55 (53) 9 9981-7854 Mobile: +1 (347) 903-9750
Re: New assignment operator?
Richard Purdie
On Tue, 2021-06-15 at 13:55 +0200, Phil Blundell wrote:
unset IMAGE_FSTYPES
IMAGE_FSTYPES = "x"
The assignment wouldn't ignore them, it really would remove them from
the knowledge of the variable. The difference is basically the difference
between:
setVar("IMAGE_FSTYPES", val, parsing=True)
setVar("IMAGE_FSTYPES", val, parsing=False)
since behind the scenes bitbake has two different behaviours for
setVar, "parsing" mode and non-parsing mode. This was after we learnt
we had two different expectations from setVar in anonymous code and
from parsing metadata.
So we'd be adding an operator that behaves like the default setVar().
reset a variable outside anonymous python. I'm also not sure I'd say
the BSP was carelessly written. It used overrides, we recommend and
support them, it just then interacted badly with the way a recipe was
written (which has been copied to multiple places).
needs/wants to do with the current syntax.
I don't know. I think it doesn't matter what I suggest these days,
the answer will be "no, it is already complicated enough". As such
we therefore accept nothing will change.
I put together the proposal as I don't think the BSP or the image
recipe are really at fault here and I can't give them a good way
to handle the situation.
Cheers,
Richard
On Tue, Jun 15, 2021 at 11:29:01AM +0100, Richard Purdie wrote:Sure, but then you just end up writing:a) is another assignment operator a good idea?I'm not terribly convinced it is. If the problem is that there's no
way to clear the OVERRIDES, maybe we should just have a new verb to
do that specifically rather than a new assignment operator that
ignores them.
unset IMAGE_FSTYPES
IMAGE_FSTYPES = "x"
The assignment wouldn't ignore them, it really would remove them from
the knowledge of the variable. The difference is basically the difference
between:
setVar("IMAGE_FSTYPES", val, parsing=True)
setVar("IMAGE_FSTYPES", val, parsing=False)
since behind the scenes bitbake has two different behaviours for
setVar, "parsing" mode and non-parsing mode. This was after we learnt
we had two different expectations from setVar in anonymous code and
from parsing metadata.
So we'd be adding an operator that behaves like the default setVar().
Ultimately though this particular example just seems like a caseIt would solve a general case problem of not being able to actually
where it's possible for a carelessly-written BSP to make life
difficult for its users. I don't think this is the only such,
and I'm not sure a new assignment operator would necessarily solve
the general case.
reset a variable outside anonymous python. I'm also not sure I'd say
the BSP was carelessly written. It used overrides, we recommend and
support them, it just then interacted badly with the way a recipe was
written (which has been copied to multiple places).
Adding more and more special operators that defeat each other seemsMaybe, maybe not. The initramfs recipe can't really express what it
like it's just going to result in a perpetual arms race.
needs/wants to do with the current syntax.
If we addMaybe we make it a non-conf context operator?
a new !=! operator then, BSP authors being as ingenious as they are,
they'll start using that all over the place and you suddenly won't
be able to override anything any more. Then we'll need a new and
even more powerful OVERRRRIDES that can't be suppressed, and we'll
be back where we started :-)
I don't know. I think it doesn't matter what I suggest these days,
the answer will be "no, it is already complicated enough". As such
we therefore accept nothing will change.
I put together the proposal as I don't think the BSP or the image
recipe are really at fault here and I can't give them a good way
to handle the situation.
Cheers,
Richard
Re: New assignment operator?
Richard Purdie
On Tue, 2021-06-15 at 11:42 +0000, Quentin Schulz wrote:
other ways it could be done, sure.
effectively impossible to undo certain operations. That general issue bothers
me more than this specific case.
We can argue the BSP is broken. The BSP is arguing bitbake is broken.
Failing that, there is an argument the initramfs recipes are broken. I
can see the different sides but there is a general problem here we've
pondered over for a while.
"IMAGE_FSTYPES_collir"
is immediate expansion *and* immediate assignment.
Cheers,
Richard
Well, it seems like an ok thing for a BSP to do to me. There are probably
On June 15, 2021 10:29:01 AM UTC, Richard Purdie <richard.purdie@...> wrote:We have an interesting problem with initramfs images. They do something like:Is that an issue that needs to actually be fixed? This is just another example
IMAGE_FSTYPES = "${INITRAMFS_FSTYPES}"
which looks innocent enough until a BSP does:
IMAGE_FSTYPES_collie = "xxx"
where collie is MACHINE, hence an override which causes confusion as
the value of INITRAMFS_FSTYPES is never used for that machine.
of weird BSP given by vendors to me, right?
other ways it could be done, sure.
Resetting IMAGE_FSTYPES_collie to INITRAMFS_FSTYPES should be enoughNo, I don't see how that helps.
to fix this issue right?
The issue in the email is an example of a general problem where it isThe questions areI'm not sure the issue explained in this mail is enough to warrant a new
a) is another assignment operator a good idea?
assignment operator.
effectively impossible to undo certain operations. That general issue bothers
me more than this specific case.
We can argue the BSP is broken. The BSP is arguing bitbake is broken.
Failing that, there is an argument the initramfs recipes are broken. I
can see the different sides but there is a general problem here we've
pondered over for a while.
I can already see vendors abusing this by always using this new operatorWe could ban the operator from conf files (enforced)?
instead of using overrides or whatever.
This would be even less fun to support on IRC or mailing lists.Maybe, maybe not. It depends how it gets used and if "abuse" occurs.
Also... As opposed to other operators, what would happen if you use theThankfully that is quite clear and would clear any overrides set against
following: IMAGE_FSTYPES_collir *:= "whatever". What would be the meaning
and behavior, it might warrant a different handling compared to other
operators to forbid any override (and append, prepend, remove).
"IMAGE_FSTYPES_collir"
Except that half the behaviour of := doesn't apply (the expansion). :=b) if so, which form?Totally unbiased. The last two 😁
It highlights that like := it is an immediate assignment operator.
The asterisk indicates that it will apply to "everything".
is immediate expansion *and* immediate assignment.
Cheers,
Richard