I've gone around in circles on this for a long time. The background
isn't obvious at first glance.

"S" represents where our source is. There is code like do_unpack which
wants to clean S before rerunning the task. It can't just clean
everything in WORKDIR as there is T, recipe-sysroot* and more that need
to be left alone (e.g. recipe-sysroot-native may have
do_fetch/do_unpack dependencies).

S is usually defined as ${BPN}-${PV} via ${BP}.

Many SRC_URIs are tarballs and tarballs usually extract into a
directory, which often happens to be BPN-PV. The tarball itself
therefore creates the ${S} directory.

For git, the git fetcher extracts into git/ so git recipes end up
setting:

S = "${WORKDIR}/git"

This was done historically since using ${PV} for the source dir for a
git recipe was likely going to complicate the paths needlessly.

The problem comes with recipes like keymaps which only have local files
referenced in SRC_URI. These do:

S = "${WORKDIR}"

so if do_unpack cleans S, it cleans WORKDIR which is very bad. We have
horrible code to guard against that. This is the first thing it would
be nice to fix.

We also have a second problem which is that if do_fetch/do_unpack
rerun, a file installed by a SRC_URI in the first run but removed in
the second, will remain in ${WORKDIR}. This means you can change a
recipe and the rebuild might not be deterministic. Again, this is
rather horrible.

It is very hard for the code to know which files it should be removing
when it reruns do_unpack. This is the second issue I'd ideally like to
fix.

Ideas:

The first issue can be improved with code like:

diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index bdb3ac33c67..0da22b34e8a 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -161,8 +161,10 @@ python base_do_fetch() {
bb.fatal("Bitbake Fetcher Error: " + repr(e))
}

+UNPACK_DIR = "${WORKDIR}"
+
addtask unpack after do_fetch
-do_unpack[dirs] = "${WORKDIR}"
+do_unpack[dirs] = "${UNPACK_DIR}"

do_unpack[cleandirs] = "${@d.getVar('S') if os.path.normpath(d.getVar('S')) != os.path.normpath(d.getVar('WORKDIR')) else os.path.join('${S}', 'patches')}"

@@ -173,7 +175,7 @@ python base_do_unpack() {

try:
fetcher = bb.fetch2.Fetch(src_uri, d)
- fetcher.unpack(d.getVar('WORKDIR'))
+ fetcher.unpack(d.getVar('UNPACK_DIR'))
except bb.fetch2.BBFetchException as e:
bb.fatal("Bitbake Fetcher Error: " + repr(e))
}


i.e. we make UNPACK_DIR configurable, then a recipe like keymaps can
do:

diff --git a/meta/recipes-bsp/keymaps/keymaps_1.0.bb b/meta/recipes-bsp/keymaps/keymaps_1.0.bb
index 84d09cb965d..2927e7f1392 100644
--- a/meta/recipes-bsp/keymaps/keymaps_1.0.bb
+++ b/meta/recipes-bsp/keymaps/keymaps_1.0.bb
@@ -25,14 +25,14 @@ SRC_URI = "file://keymap.sh"
INITSCRIPT_NAME = "keymap.sh"
INITSCRIPT_PARAMS = "start 01 S ."

-S = "${WORKDIR}"
+UNPACK_DIR = "${S}"

do_install () {
# Only install the script if 'sysvinit' is in DISTRO_FEATURES
# THe ulitity this script provides could be achieved by systemd-vconsole-setup.service
if ${@bb.utils.contains('DISTRO_FEATURES','sysvinit','true','false',d)}; then
install -d ${D}${sysconfdir}/init.d/
- install -m 0755 ${WORKDIR}/keymap.sh ${D}${sysconfdir}/init.d/
+ install -m 0755 ${S}/keymap.sh ${D}${sysconfdir}/init.d/
fi
}


which I think then has the recipe make more logical sense.

For the second issue, the best idea I could come up with was to make
do_unpack extract to a subdirectory or WORKDIR, then symlink the files
into place in WORKDIR. do_unpack can then remove any symlink it finds
in WORKDIR when it reruns.

Whether either or both of these improvements are worthwhile and whether
there is the communuity demand and support to implement them and
migrate metadata to match, I don't know but I wanted to put the ideas
down here.

Cheers,

Richard