On Wed, 2017-01-18 at 11:57 -0600, Mark Hatle wrote:

On 1/18/17 11:01 AM, Richard Purdie wrote:
The one thing I keep coming back to on DEPENDS (vs a more specific
DEPENDS).  In the general case, when I've built recipes I have say 'I
depend on XYZ', thinking then that anything XYZ depends on will be
available for me to build.

That would include the native items.  Now, that of course does NOT
mean they are required to unpack/'install into the sysroot' those
items, just that the dependencies exist for my current build.

The sysroot requirements are actually quite varied. It really helps to
think about this from a task perspective which is how bitbake really
works.

In order to run do_fetch, there could be tools like git-native or
subversion-native needed. In order to run do_patch, we need quilt-
native. 

do_configure is special, we call its dependencies DEPENDS. 

There are special dependency requirements for do_package,
do_packagedata and RDEPENDS are actually only really mapped to
do_package_write_*.

So bitbake handles dependencies on a per task basis and our task
dependencies are very convoluted and complex.

We then added sstate into the mix. For any given task, we need to know
which set of dependencies are really needed.

A-native can link against B-native so we must install B-native if we
install A-native.

If we install target package C, which native dependencies might we
need?

The problem is we have to give the system some kind of hint as it
really can't figure it out today and we've only ever hacked it to work.

What I'm trying to wrap my head around are the cases where you don't
depend on something for build time (or don't want a dependency to be
inherited) that won't cause more confusion over time.

An easy example, if I run "bitbake core-image-sato -c rootfs", would
you expect it to extract from sstate gcc-cross, binutils-cross and so
on? They're large and I suspect you'd agree they're not needed to build
a rootfs from sstate. The system doesn't pull them in today, nor should
it. cmake-native? bison-native? gettext-native? They do mount up
significantly.

I can certainly see the optimization cases where I know I only need
XYZ for the install, but the rest of the DEPENDS (and their DEPENDS)
are not needed.

So if I'm understanding this right.

If we did all DEPENDS inherited from our DEPENDS, then this would be
working properly.  But that means then all dependencies would need to
be installed each time, causing a huge performance impact --
especially on native dependencies which are often assumed or so?

Yes, particularly as we don't know how to draw the line at things like
the complete cross toolchain if we follow DEPENDS.

By default all DEPENDS (not native) will still be loaded and
inherited, and we're really only concerned with adding missing
'native' things that a specific install script may need?

I'm going to say yes for the purposes of this discussion. Its way more
complicated but not relevant.

So for performance reasons, we want to not automatically install all
of the dependent referred native packages -- and specify them as
needed -- only where needed..  helping performance by further being
able to delay the extract installation to as late as possible (the
package write function -- or equivalent sstate-cache install
functions [pre/post/etc]).

We won't delay extraction but we would remove the need to fetch and
extract these things and we're talking hundreds of MB which is
significant, particularly if you only want to rebuild a rootfs (e.g. in
the eSDK).

I think where I'm getting confused by the name is, that
PACKAGE_WRITE_DEPS
(equivalent of do_package_write*) sounds to me like something
required in order
to write the package to the disk.  So rpm, deb, opkg that kind of
thing would go there.

Unfortunately that is one of its uses too. I can't find any less
confusing way to handle it though which doesn't have its own added and
worse complexities.

It's not clear that the package's install script would be affected by
this in any way (nor the sstate-cache versions.)

(I'm struggling to come up with any name better then
PACKAGE_WRITE_DEPENDS myself...)

Right, I'm open to ideas but trying to explain how we end up here...

Does that make it clearer why we need this?

I certainly understand it more.. I'm just struggling with how to
explain when
someone would need this or not, to someone who hasn't hit one of the
extraction
problems.   I think that is by far my biggest remaining concern, how
to make
sure people can understand they need to do this and when...

The rules in some ways are simple, it comes down to:

"""
If your postinstall can execute at rootfs creation time rather than on
target but depends on a native tool in order to execute, you need to
list that tool in PACKAGE_WRITE_DEPENDS.
"""

Cheers,

Richard