On 1/18/17 8:23 AM, Richard Purdie wrote:

One problem we're seeing with recipe specific sysroots (rss) is that
the dependencies package postinstalls have is badly represented.

For example, dbus adds a user and this needs tools from shadow-native.

If you run a do_rootfs build from an empty TMPDIR and have a populated
sstate cache, how does the system know it needs to install shadow-
native from sstate so that dbus' postinsts aren't deferred to on
target?

I have seen other systems declare these in different ways. Things like:

RPROVIDES_pn += "user(name)"

RREQUIRES_pn += "user(name)"

Automation can declare provides and dependencies users that are need. But of
course there will always be manual users/groups that are needed as well.

Theoretically we should already be doing some of this via dependencies today --
but it can be a complex problem.

Of course this doesn't handle other things postinsts may be needed for...

Currently its a bit of mess/hack. There is a list of hardcoded special
cases in sstate.bbclass but its incomplete. There are also lists of
special dependencies in the image.bbclass rootfs code to pull in things
like depmodwrapper-cross and ldconfig.

Some things are detected at build time like ldconfig dependencies so
there isn't much that can be done for them but most dependencies like
depmod, useradd, gtk-icon-cache, gdk-pixbuf, systemd and so on are
known in advance.

With rss, the problem gets more complicated since not only do we need
to ensure the tools get extracted from sstate but that they must get
into the image recipe's own native sysroot.

I've spent quite some time pondering how to do this. A normal DEPENDS
means something quite different so we need some kind of new markup.
Adding new kinds of dependencies to bitbake doesn't seem attractive
though.

I'm going to propose a new PACKAGE_WRITE_DEPS variable which gets added
to the [depends] flag of the do_package_write_XXX tasks.

What this means is that the dependency is indicated in the task graph
and its done in a generic way which the code can identify and act upon.
These tasks don't currently have much in the way of dependencies other
than tools to actually build packages.

The PACKAGE_WRITE_DEPS can either be in addition to DEPENDS if the
recipe needs these tools at build time, or if its only needed for the
postinst, PACKAGE_WRITE_DEPS can be sufficient and the DEPENDS can be
removed.

I'm a little worried this is making things more complicated then they may need
to be.

What is the downside of simply specifying those items in the existing DEPENDS as
we have it today? (Performance, they have to be installed before the package
can begin to build?)

The name itself doesn't make me understand (without reading your explanation)
why it's required. That also makes me suspect it might be difficult (over time)
to explain to people when they need this behavior and why.

--Mark

I've had a go at a quick proof of concept patch below. This includes
removing the hardcoded pieces from sstate.bbclass but I'm still in the
process of testing this.

Right now, I wanted to put this proposal out there and see if people
were ok with the general idea. There are perhaps some tweaks that could
be made to the final implementation. I am conscious the M2 deadline is
looming and if we want rss in this release, we need to be thinking
about merging code soon though.

Cheers,

Richard


From c8c37b97427e37852f2a6efc14b5337422489116 Mon Sep 17 00:00:00 2001
From: Richard Purdie <richard.purdie@...>
Date: Wed, 18 Jan 2017 12:33:16 +0000
Subject: PACKAGE_WRITE_DEPS first pass

Signed-off-by: Richard Purdie <richard.purdie@...>

diff --git a/meta/classes/gtk-icon-cache.bbclass b/meta/classes/gtk-icon-cache.bbclass
index c5d8d7c..1e75bde 100644
--- a/meta/classes/gtk-icon-cache.bbclass
+++ b/meta/classes/gtk-icon-cache.bbclass
@@ -2,6 +2,8 @@ FILES_${PN} += "${datadir}/icons/hicolor"

DEPENDS += "${@['hicolor-icon-theme', '']['${BPN}' == 'hicolor-icon-theme']} gtk-icon-utils-native"

+PACKAGE_WRITE_DEPS += "gtk-icon-utils-native:do_populate_sysroot gdk-pixbuf-native:do_populate_sysroot"
+
gtk_icon_cache_postinst() {
if [ "x$D" != "x" ]; then
$INTERCEPT_DIR/postinst_intercept update_icon_cache ${PKG} \
diff --git a/meta/classes/kernel-module-split.bbclass b/meta/classes/kernel-module-split.bbclass
index efe1b42..c7f4880 100644
--- a/meta/classes/kernel-module-split.bbclass
+++ b/meta/classes/kernel-module-split.bbclass
@@ -22,6 +22,8 @@ if [ x"$D" = "x" ]; then
fi
}

+PACKAGE_WRITE_DEPS += "kmod-native:do_populate_sysroot depmodwrapper-cross:do_populate_sysroot"
+
do_install_append() {
install -d ${D}${sysconfdir}/modules-load.d/ ${D}${sysconfdir}/modprobe.d/
}
diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
index 568b85c..0a56d1c 100644
--- a/meta/classes/package.bbclass
+++ b/meta/classes/package.bbclass
@@ -54,6 +54,8 @@ ALL_MULTILIB_PACKAGE_ARCHS = "${@all_multilib_tune_values(d, 'PACKAGE_ARCHS')}"
# rpm is used for the per-file dependency identification
PACKAGE_DEPENDS += "rpm-native"

+PACKAGE_WRITE_DEPS ??= ""
+
def legitimize_package_name(s):
"""
Make sure package names are legitimate strings
diff --git a/meta/classes/package_deb.bbclass b/meta/classes/package_deb.bbclass
index 68eca61..d4f9262 100644
--- a/meta/classes/package_deb.bbclass
+++ b/meta/classes/package_deb.bbclass
@@ -351,6 +351,7 @@ python do_package_write_deb () {
do_package_write_deb[dirs] = "${PKGWRITEDIRDEB}"
do_package_write_deb[cleandirs] = "${PKGWRITEDIRDEB}"
do_package_write_deb[umask] = "022"
+do_package_write_deb[depends] += "${PACKAGE_WRITE_DEPS}"
addtask package_write_deb after do_packagedata do_package


diff --git a/meta/classes/package_ipk.bbclass b/meta/classes/package_ipk.bbclass
index 7018a60..9def81b 100644
--- a/meta/classes/package_ipk.bbclass
+++ b/meta/classes/package_ipk.bbclass
@@ -291,6 +291,7 @@ python do_package_write_ipk () {
do_package_write_ipk[dirs] = "${PKGWRITEDIRIPK}"
do_package_write_ipk[cleandirs] = "${PKGWRITEDIRIPK}"
do_package_write_ipk[umask] = "022"
+do_package_write_ipk[depends] += "${PACKAGE_WRITE_DEPS}"
addtask package_write_ipk after do_packagedata do_package

PACKAGEINDEXDEPS += "opkg-utils-native:do_populate_sysroot"
diff --git a/meta/classes/package_rpm.bbclass b/meta/classes/package_rpm.bbclass
index b9f049e..d07ac51 100644
--- a/meta/classes/package_rpm.bbclass
+++ b/meta/classes/package_rpm.bbclass
@@ -772,6 +772,7 @@ python do_package_write_rpm () {
do_package_write_rpm[dirs] = "${PKGWRITEDIRRPM}"
do_package_write_rpm[cleandirs] = "${PKGWRITEDIRRPM}"
do_package_write_rpm[umask] = "022"
+do_package_write_rpm[depends] += "${PACKAGE_WRITE_DEPS}"
addtask package_write_rpm after do_packagedata do_package

PACKAGEINDEXDEPS += "rpm-native:do_populate_sysroot"
diff --git a/meta/classes/pixbufcache.bbclass b/meta/classes/pixbufcache.bbclass
index 0295a66..3f5b6ef 100644
--- a/meta/classes/pixbufcache.bbclass
+++ b/meta/classes/pixbufcache.bbclass
@@ -8,7 +8,7 @@ inherit qemu

PIXBUF_PACKAGES ??= "${PN}"

-do_package_write_rpm[depends] += "qemu-native:do_populate_sysroot gdk-pixbuf-native:do_populate_sysroot"
+PACKAGE_WRITE_DEPS += "qemu-native:do_populate_sysroot gdk-pixbuf-native:do_populate_sysroot"

pixbufcache_common() {
if [ "x$D" != "x" ]; then
diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
index e84e2d9..9500303 100644
--- a/meta/classes/sstate.bbclass
+++ b/meta/classes/sstate.bbclass
@@ -938,17 +938,14 @@ BB_SETSCENE_DEPVALID = "setscene_depvalid"
def setscene_depvalid(task, taskdependees, notneeded, d):
# taskdependees is a dict of tasks which depend on task, each being a 3 item list of [PN, TASKNAME, FILENAME]
# task is included in taskdependees too
+ # Return - False - We need this dependency
+ # - True - We can skip this dependency

bb.debug(2, "Considering setscene task: %s" % (str(taskdependees[task])))

def isNativeCross(x):
return x.endswith("-native") or "-cross-" in x or "-crosssdk" in x or x.endswith("-cross")

- def isPostInstDep(x):
- if x in ["qemu-native", "gdk-pixbuf-native", "qemuwrapper-cross", "depmodwrapper-cross", "systemd-systemctl-native", "gtk-icon-utils-native", "ca-certificates-native"]:
- return True
- return False
-
# We only need to trigger populate_lic through direct dependencies
if taskdependees[task][1] == "do_populate_lic":
return True
@@ -970,10 +967,11 @@ def setscene_depvalid(task, taskdependees, notneeded, d):
# do_package_write_* and do_package doesn't need do_package
if taskdependees[task][1] == "do_package" and taskdependees[dep][1] in ['do_package', 'do_package_write_deb', 'do_package_write_ipk', 'do_package_write_rpm', 'do_packagedata', 'do_package_qa']:
continue
- # do_package_write_* and do_package doesn't need do_populate_sysroot, unless is a postinstall dependency
- if taskdependees[task][1] == "do_populate_sysroot" and taskdependees[dep][1] in ['do_package', 'do_package_write_deb', 'do_package_write_ipk', 'do_package_write_rpm', 'do_packagedata', 'do_package_qa']:
- if isPostInstDep(taskdependees[task][0]) and taskdependees[dep][1] in ['do_package_write_deb', 'do_package_write_ipk', 'do_package_write_rpm']:
- return False
+ # do_package_write_* need do_populate_sysroot as they're mainly postinstall dependencies
+ if taskdependees[task][1] == "do_populate_sysroot" and taskdependees[dep][1] in ['do_package_write_deb', 'do_package_write_ipk', 'do_package_write_rpm']:
+ return False
+ # do_package/packagedata/package_qa don't need do_populate_sysroot
+ if taskdependees[task][1] == "do_populate_sysroot" and taskdependees[dep][1] in ['do_package', 'do_packagedata', 'do_package_qa']:
continue
# Native/Cross packages don't exist and are noexec anyway
if isNativeCross(taskdependees[dep][0]) and taskdependees[dep][1] in ['do_package_write_deb', 'do_package_write_ipk', 'do_package_write_rpm', 'do_packagedata', 'do_package', 'do_package_qa']:
diff --git a/meta/classes/systemd.bbclass b/meta/classes/systemd.bbclass
index 0b1c228..3e33456 100644
--- a/meta/classes/systemd.bbclass
+++ b/meta/classes/systemd.bbclass
@@ -17,6 +17,7 @@ python __anonymous() {
# files.
if bb.utils.contains('DISTRO_FEATURES', 'systemd', True, False, d):
d.appendVar("DEPENDS", " systemd-systemctl-native")
+ d.appendVar("PACKAGE_WRITE_DEPS", " systemd-systemctl-native:do_populate_sysroot")
if not bb.utils.contains('DISTRO_FEATURES', 'sysvinit', True, False, d):
d.setVar("INHIBIT_UPDATERCD_BBCLASS", "1")
}
diff --git a/meta/classes/useradd.bbclass b/meta/classes/useradd.bbclass
index 5bf81c4..7fdfca5 100644
--- a/meta/classes/useradd.bbclass
+++ b/meta/classes/useradd.bbclass
@@ -4,6 +4,7 @@ inherit useradd_base
# target sysroot, and shadow -native and -sysroot provide the utilities
# and support files needed to add and modify user and group accounts
DEPENDS_append_class-target = " base-files shadow-native shadow-sysroot shadow base-passwd"
+PACKAGE_WRITE_DEPS += "shadow-native:do_populate_sysroot"

# This preinstall function can be run in four different contexts:
#
diff --git a/meta/recipes-bsp/keymaps/keymaps_1.0.bb b/meta/recipes-bsp/keymaps/keymaps_1.0.bb
index 5793a76..adea233 100644
--- a/meta/recipes-bsp/keymaps/keymaps_1.0.bb
+++ b/meta/recipes-bsp/keymaps/keymaps_1.0.bb
@@ -37,7 +37,7 @@ do_install () {
fi
}

-DEPENDS_append = " ${@bb.utils.contains('DISTRO_FEATURES','systemd','systemd-systemctl-native','',d)}"
+PACKAGE_WRITE_DEPS_append = " ${@bb.utils.contains('DISTRO_FEATURES','systemd','systemd-systemctl-native:do_populate_sysroot','',d)}"
pkg_postinst_${PN} () {
if ${@bb.utils.contains('DISTRO_FEATURES','systemd sysvinit','true','false',d)}; then
if [ -n "$D" ]; then
diff --git a/meta/recipes-bsp/v86d/v86d_0.1.10.bb b/meta/recipes-bsp/v86d/v86d_0.1.10.bb
index 1046d63..8bd3700 100644
--- a/meta/recipes-bsp/v86d/v86d_0.1.10.bb
+++ b/meta/recipes-bsp/v86d/v86d_0.1.10.bb
@@ -60,7 +60,7 @@ python __anonymous() {

inherit update-rc.d

-DEPENDS_append = " ${@bb.utils.contains('DISTRO_FEATURES','systemd','systemd-systemctl-native','',d)}"
+PACKAGE_WRITE_DEPS_append = " ${@bb.utils.contains('DISTRO_FEATURES','systemd','systemd-systemctl-native:do_populate_sysroot','',d)}"
pkg_postinst_${PN} () {
if ${@bb.utils.contains('DISTRO_FEATURES','systemd sysvinit','true','false',d)}; then
if [ -n "$D" ]; then
diff --git a/meta/recipes-core/initscripts/initscripts_1.0.bb b/meta/recipes-core/initscripts/initscripts_1.0.bb
index 8f110b0..836e9eb 100644
--- a/meta/recipes-core/initscripts/initscripts_1.0.bb
+++ b/meta/recipes-core/initscripts/initscripts_1.0.bb
@@ -44,7 +44,7 @@ KERNEL_VERSION = ""

inherit update-alternatives
DEPENDS_append = " update-rc.d-native"
-DEPENDS_append = " ${@bb.utils.contains('DISTRO_FEATURES','systemd','systemd-systemctl-native','',d)}"
+PACKAGE_WRITE_DEPS_append = " ${@bb.utils.contains('DISTRO_FEATURES','systemd','systemd-systemctl-native:do_populate_sysroot','',d)}"

PACKAGES =+ "${PN}-functions"
RDEPENDS_${PN} = "${PN}-functions \
diff --git a/meta/recipes-core/psplash/psplash_git.bb b/meta/recipes-core/psplash/psplash_git.bb
index b0d6bb4..335593d 100644
--- a/meta/recipes-core/psplash/psplash_git.bb
+++ b/meta/recipes-core/psplash/psplash_git.bb
@@ -110,7 +110,7 @@ FILES_${PN} += "/mnt/.psplash"
INITSCRIPT_NAME = "psplash.sh"
INITSCRIPT_PARAMS = "start 0 S . stop 20 0 1 6 ."

-DEPENDS_append = " ${@bb.utils.contains('DISTRO_FEATURES','systemd','systemd-systemctl-native','',d)}"
+PACKAGE_WRITE_DEPS_append = " ${@bb.utils.contains('DISTRO_FEATURES','systemd','systemd-systemctl-native:do_populate_sysroot','',d)}"
pkg_postinst_${PN} () {
if ${@bb.utils.contains('DISTRO_FEATURES','systemd','true','false',d)}; then
if [ -n "$D" ]; then
diff --git a/meta/recipes-core/systemd/systemd-compat-units.bb b/meta/recipes-core/systemd/systemd-compat-units.bb
index 0f0876b..7655246 100644
--- a/meta/recipes-core/systemd/systemd-compat-units.bb
+++ b/meta/recipes-core/systemd/systemd-compat-units.bb
@@ -4,7 +4,7 @@ LICENSE = "MIT"

PR = "r29"

-DEPENDS = "systemd-systemctl-native"
+PACKAGE_WRITE_DEPS += "systemd-systemctl-native:do_populate_sysroot"

S = "${WORKDIR}"

diff --git a/meta/recipes-kernel/modutils-initscripts/modutils-initscripts.bb b/meta/recipes-kernel/modutils-initscripts/modutils-initscripts.bb
index db670cf..9b602a5 100644
--- a/meta/recipes-kernel/modutils-initscripts/modutils-initscripts.bb
+++ b/meta/recipes-kernel/modutils-initscripts/modutils-initscripts.bb
@@ -22,7 +22,7 @@ do_install () {
install -m 0755 ${WORKDIR}/modutils.sh ${D}${sysconfdir}/init.d/
}

-DEPENDS_append = " ${@bb.utils.contains('DISTRO_FEATURES','systemd','systemd-systemctl-native','',d)}"
+PACKAGE_WRITE_DEPS_append = " ${@bb.utils.contains('DISTRO_FEATURES','systemd','systemd-systemctl-native:do_populate_sysroot','',d)}"
pkg_postinst_${PN} () {
if ${@bb.utils.contains('DISTRO_FEATURES','systemd','true','false',d)}; then
if [ -n "$D" ]; then
diff --git a/meta/recipes-sato/matchbox-sato/matchbox-session-sato_0.1.bb b/meta/recipes-sato/matchbox-sato/matchbox-session-sato_0.1.bb
index 42c742f..bf0de36 100644
--- a/meta/recipes-sato/matchbox-sato/matchbox-session-sato_0.1.bb
+++ b/meta/recipes-sato/matchbox-sato/matchbox-session-sato_0.1.bb
@@ -6,7 +6,6 @@ LICENSE = "GPLv2.0+"
LIC_FILES_CHKSUM = "file://session;endline=3;md5=f8a5c5b9c279e52dc094d10e11c2be63"

SECTION = "x11"
-DEPENDS = "gconf-native"
RDEPENDS_${PN} = "formfactor matchbox-theme-sato matchbox-panel-2 matchbox-desktop matchbox-session gconf"
PR = "r30"

@@ -43,6 +42,7 @@ do_install() {
chmod +x ${D}/${sysconfdir}/matchbox/session
}

+PACKAGE_WRITE_DEPS += "gconf-native:do_populate_sysroot"
pkg_postinst_${PN} () {
set_value() {
#type, name, value
diff --git a/meta/recipes-support/ca-certificates/ca-certificates_20161130.bb b/meta/recipes-support/ca-certificates/ca-certificates_20161130.bb
index e0b2e41..1c70408 100644
--- a/meta/recipes-support/ca-certificates/ca-certificates_20161130.bb
+++ b/meta/recipes-support/ca-certificates/ca-certificates_20161130.bb
@@ -11,6 +11,7 @@ LIC_FILES_CHKSUM = "file://debian/copyright;md5=e7358b9541ccf3029e9705ed8de57968
DEPENDS = "ca-certificates-native"
DEPENDS_class-native = "openssl-native"
DEPENDS_class-nativesdk = "ca-certificates-native openssl-native"
+PACKAGE_WRITE_DEPS += "ca-certificates-native:do_populate_sysroot"

SRCREV = "61b70a1007dc269d56881a0d480fc841daacc77c"



_______________________________________________
Openembedded-architecture mailing list
Openembedded-architecture@...
http://lists.openembedded.org/mailman/listinfo/openembedded-architecture