Hi All,

There has been some interest in some older bugs about empty packages
and bogus dependencies and what we should do about them. Yohan and
Fawzi were kind enough to post some patches to prompt some discussion
and this did indeed get me thinking.

I personally think that we shouldn't be generating an RDEPENDS or
RRECOMMENDS if we know that the package in question is never going to
exist. We have "auto dependency" code which does go a bit crazy with
these.

The easiest problem case to understand is an empty PN package. Based
upon their patches I can see we likely have this situation in a subset
of:

bind, bsd-headers, libssp-nonshared, newlib, libgcc, python3, libglu
mesa, libpthread-stubs, xtrans, xcb-proto, xorgproto, util-macros,
linux-libc-headers, make-mod-scripts, argp-standalone

Reasons vary but the easy to understand case is where the code is just
development headers and all ends up in ${PN}-dev.

We have https://bugzilla.yoctoproject.org/show_bug.cgi?id=6839

We did used to set an RDEPENDS between ${PN}-dev and ${PN} but that
turned out to be a bad idea and we relaxed it to a RRECOMENDS:

https://git.yoctoproject.org/poky/commit/?id=ddbd90a37c7e9089aaeccb42dd008cf47594099b

I did at least mark up the recipes with this issue:

https://git.yoctoproject.org/poky/commit/?id=73cf55cdde1f3d2773b42f52d2fcf57665102c81

which meant the worst of these dependencies were removed where the main
package didn't exist or didn't make sense.

As I see it there are a few options:

a) Do nothing and continue with the above
b) Generate an empty PN
c) Delete PN from PACKAGES
d) Move the contents of PN-dev to PN
e) Something else

I think I'm in favour of c), removing PN from PACKAGES and not even
pretending it exists. We could add an RPROVIDES:${PN}-dev += "${PN} so
that assumptions about naming work, not sure about that.


That handles one specific corner case but there are others as bug #8222
talks about for example
https://bugzilla.yoctoproject.org/show_bug.cgi?id=8222

Here, the proposal is to change the globally inserted RDEPENDS (now a
RRECOMMEND) from PN-dev to PN to become conditional upon (roughly hand
waving), there being libraries in other packages which PN-dev therefore
needs to depend upon. I think that idea is worthy of exploration. Are
there any cases where the -dev package depends on ${PN} for things
other than shared libraries?

Thinking out loud, our symlink dependency handling code in do_package
might even be enough to detect and handle injecting that?


So, turning this into a set of actions, my personal view is that we
should probably:

a) Add warnings for dependencies (RDEPENDS or RRECOMMENDS) on packages
which don't get created for some reason (either not in PACKAGES or not
ALLOW_EMPTY). We should be able to properly check this using
packagedata which never used to exist as it does now when these bugs
were first created! We can leave the warnings disabled at first until
we clean things up for core and our key layers

b) Work through those warnings to work out if we can silence them at
source. I think the dependency creation code for -dbg and -dev packages
will need to be taught to check if a target dependency exists to fix
the bulk of them.

c) Explore whether we should drop PN from PACKAGES in some of the above
cases.

d) Explore whether we can replace the current generic RRECOMMENDS from
PN-dev -> PN with something autogenerated based on the presence of
shared libs

e) For a long time I've wondered if we should split the -dbg packages
from one per recipe to one per package with binaries in it. This would
solve some of the problems that we'll likely encounter in b) but might 













new ones.

I've given this some thought but not thought through the ideas in full.

I appreciate some of this was set as a newcommer bug in bugzilla and in
hindsight this perhaps shouldn't have been the case. I suspect taking
d) in isolation was the idea and that should still be possible.

I'd note that we're at feature freeze so more invasive changes like
this would likely need to be post release unless any become bug fixes.

Cheers,

Richard

On 2/25/23 4:24 AM, Richard Purdie wrote:

Hi All,
There has been some interest in some older bugs about empty packages
and bogus dependencies and what we should do about them. Yohan and
Fawzi were kind enough to post some patches to prompt some discussion
and this did indeed get me thinking.
I personally think that we shouldn't be generating an RDEPENDS or
RRECOMMENDS if we know that the package in question is never going to
exist. We have "auto dependency" code which does go a bit crazy with
these.
The easiest problem case to understand is an empty PN package. Based
upon their patches I can see we likely have this situation in a subset
of:
bind, bsd-headers, libssp-nonshared, newlib, libgcc, python3, libglu
mesa, libpthread-stubs, xtrans, xcb-proto, xorgproto, util-macros,
linux-libc-headers, make-mod-scripts, argp-standalone
Reasons vary but the easy to understand case is where the code is just
development headers and all ends up in ${PN}-dev.
We have https://bugzilla.yoctoproject.org/show_bug.cgi?id=6839
We did used to set an RDEPENDS between ${PN}-dev and ${PN} but that
turned out to be a bad idea and we relaxed it to a RRECOMENDS:
https://git.yoctoproject.org/poky/commit/?id=ddbd90a37c7e9089aaeccb42dd008cf47594099b

Switching from recommends to depends, I think the biggest concern here is the symlinks .so -> implementation. However, if the code (dependency) is already capturing this and including them, I see no problem here.

What we want to avoid is installing a -dev package, and not getting the associated libraries at the same time, because a user doesn't have recommends enabled. (On-target development.. it's been pretty common for people I've worked with to turn off recommends on their systems.)

I did at least mark up the recipes with this issue:
https://git.yoctoproject.org/poky/commit/?id=73cf55cdde1f3d2773b42f52d2fcf57665102c81

I like the way this one is structured. Will there be any sort of helper to detect the older use of the RDEPENDS:${PN}-dev for conversion? I forsee a lot of people not understanding this and still doing it the old way.

which meant the worst of these dependencies were removed where the main
package didn't exist or didn't make sense.
As I see it there are a few options:
a) Do nothing and continue with the above
b) Generate an empty PN
c) Delete PN from PACKAGES
d) Move the contents of PN-dev to PN
e) Something else
I think I'm in favour of c), removing PN from PACKAGES and not even
pretending it exists. We could add an RPROVIDES:${PN}-dev += "${PN} so
that assumptions about naming work, not sure about that.

In my many cases install of the (empty) PN was treated like a package group, install the PN, and you get the 'collection' of runtime packages that may have different names. I'm not sure this is still true though, but it's really the only concern I have with C.

That handles one specific corner case but there are others as bug #8222
talks about for example
https://bugzilla.yoctoproject.org/show_bug.cgi?id=8222
Here, the proposal is to change the globally inserted RDEPENDS (now a
RRECOMMEND) from PN-dev to PN to become conditional upon (roughly hand
waving), there being libraries in other packages which PN-dev therefore
needs to depend upon. I think that idea is worthy of exploration. Are
there any cases where the -dev package depends on ${PN} for things
other than shared libraries?

Ya, I think this is more what is needed. If there are symlink .so in the -dev, we need to find out what is providing the targets and those should be the dependencies. In some case it may be ${PN}, but not all.

Thinking out loud, our symlink dependency handling code in do_package
might even be enough to detect and handle injecting that?

I think it is? I know that general symlink targeting was intended to do something like this, so there may not be any reason to special case .so symlinks. Any symlink really indicates a dependency on the provider of some kind.

So, turning this into a set of actions, my personal view is that we
should probably:
a) Add warnings for dependencies (RDEPENDS or RRECOMMENDS) on packages
which don't get created for some reason (either not in PACKAGES or not
ALLOW_EMPTY). We should be able to properly check this using
packagedata which never used to exist as it does now when these bugs
were first created! We can leave the warnings disabled at first until
we clean things up for core and our key layers
b) Work through those warnings to work out if we can silence them at
source. I think the dependency creation code for -dbg and -dev packages
will need to be taught to check if a target dependency exists to fix
the bulk of them.
c) Explore whether we should drop PN from PACKAGES in some of the above
cases.
d) Explore whether we can replace the current generic RRECOMMENDS from
PN-dev -> PN with something autogenerated based on the presence of
shared libs

I think a-d will provide us the largest benefit for the work required.

e) For a long time I've wondered if we should split the -dbg packages
from one per recipe to one per package with binaries in it. This would
solve some of the problems that we'll likely encounter in b) but might
new ones.

I've been seeing different distributions toy with this. I've got mixed feelings on it though.

A -dbg really shouldn't have dependencies on what it's providing debug for (IMHO) since the debug doesn't indicate a dependency, it indicates a 'I provide more info on...' So in a desktop world, I'd almost consider say /bin/bash to have a "debug dependency" on the bash-dbg components, as well as the debug components of the libraries it uses. (I know of no such way to indicate this without writing a new packaging system, but that is the mental model I have.)

What does breaking up the -dbg packages into sub packages buy us? I'm not sure. If I install a large -dbg (for things not installed) then I have extra files and more disk space used, but I'm not introducing a problem. (Assuming the -dbg doesn't have dependencies or recommends that were followed on install.)

This is much easier for the user to install a bigger package and ignore stuff they don't need.

But if I have smaller -dbg packages, then I can put in dependencies or recommends to complete the "what do I need to debug things?" set. But in either case, I'd be inclined to restrict -dbg package dependencies on other -dbg packages.

This is smaller and possibly better for an automated tool to only install matching -dbg for what might be on the target device. This alone may make breaking up the packages worth while for our use-cases.

I've given this some thought but not thought through the ideas in full.
I appreciate some of this was set as a newcommer bug in bugzilla and in
hindsight this perhaps shouldn't have been the case. I suspect taking
d) in isolation was the idea and that should still be possible.
I'd note that we're at feature freeze so more invasive changes like
this would likely need to be post release unless any become bug fixes.

e) seems like a good change for a future release. But I think it's going to require some consideration of debug use-cases. On-target, off-target, small devices, large devices, etc etc etc.

--Mark

Cheers,
Richard

On Sat, Feb 25, 2023 at 2:24 AM Richard Purdie
<richard.purdie@...> wrote:

Hi All,

There has been some interest in some older bugs about empty packages
and bogus dependencies and what we should do about them. Yohan and
Fawzi were kind enough to post some patches to prompt some discussion
and this did indeed get me thinking.

I personally think that we shouldn't be generating an RDEPENDS or
RRECOMMENDS if we know that the package in question is never going to
exist. We have "auto dependency" code which does go a bit crazy with
these.

The easiest problem case to understand is an empty PN package. Based
upon their patches I can see we likely have this situation in a subset
of:

bind, bsd-headers, libssp-nonshared, newlib, libgcc, python3, libglu
mesa, libpthread-stubs, xtrans, xcb-proto, xorgproto, util-macros,
linux-libc-headers, make-mod-scripts, argp-standalone

Reasons vary but the easy to understand case is where the code is just
development headers and all ends up in ${PN}-dev.

We have https://bugzilla.yoctoproject.org/show_bug.cgi?id=6839

We did used to set an RDEPENDS between ${PN}-dev and ${PN} but that
turned out to be a bad idea and we relaxed it to a RRECOMENDS:

https://git.yoctoproject.org/poky/commit/?id=ddbd90a37c7e9089aaeccb42dd008cf47594099b

I did at least mark up the recipes with this issue:

https://git.yoctoproject.org/poky/commit/?id=73cf55cdde1f3d2773b42f52d2fcf57665102c81

which meant the worst of these dependencies were removed where the main
package didn't exist or didn't make sense.

As I see it there are a few options:

a) Do nothing and continue with the above
b) Generate an empty PN
c) Delete PN from PACKAGES
d) Move the contents of PN-dev to PN
e) Something else

b) has the advantage that it "just works" for creating an sdk. i.e.
you always include PN in TOOLCHAIN_TARGET_TASK for your toolchain
recipe (or IMAGE_INSTALL if using an image recipe to define an sdk).
Solutions where you need to know that sometimes you add PN to an sdk
and sometimes PN-dev are more cumbersome.

On Sat, Feb 25, 2023 at 10:24:17AM +0000, Richard Purdie wrote:

Hi All,

<snip>

Reasons vary but the easy to understand case is where the code is just
development headers and all ends up in ${PN}-dev.

We have https://bugzilla.yoctoproject.org/show_bug.cgi?id=6839

We did used to set an RDEPENDS between ${PN}-dev and ${PN} but that
turned out to be a bad idea and we relaxed it to a RRECOMENDS:

https://git.yoctoproject.org/poky/commit/?id=ddbd90a37c7e9089aaeccb42dd008cf47594099b

I did at least mark up the recipes with this issue:

https://git.yoctoproject.org/poky/commit/?id=73cf55cdde1f3d2773b42f52d2fcf57665102c81

Independent of this discussion - does it make sense to backport the above 2
changes to Kirkstone?

--
Denys

On Tue, 2023-02-28 at 22:28 -0500, Denys Dmytriyenko wrote:

On Sat, Feb 25, 2023 at 10:24:17AM +0000, Richard Purdie wrote:

Hi All,

<snip>

Reasons vary but the easy to understand case is where the code is just
development headers and all ends up in ${PN}-dev.

We have https://bugzilla.yoctoproject.org/show_bug.cgi?id=6839

We did used to set an RDEPENDS between ${PN}-dev and ${PN} but that
turned out to be a bad idea and we relaxed it to a RRECOMENDS:

https://git.yoctoproject.org/poky/commit/?id=ddbd90a37c7e9089aaeccb42dd008cf47594099b

I did at least mark up the recipes with this issue:

https://git.yoctoproject.org/poky/commit/?id=73cf55cdde1f3d2773b42f52d2fcf57665102c81

Independent of this discussion - does it make sense to backport the above 2
changes to Kirkstone?

I suspect not since it is a significant change in behaviour which we're
not seeing a pressing need for from kirkstone users?

Cheers,

Richard