Date   

[hardknott][PATCH 04/12] gstreamer1.0-plugins-bad: 1.18.5 -> 1.18.6

Jose Quaresma
 

Signed-off-by: Jose Quaresma <quaresma.jose@...>
---
...plugins-bad_1.18.5.bb => gstreamer1.0-plugins-bad_1.18.6.bb} | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
rename meta/recipes-multimedia/gstreamer/{gstreamer1.0-plugins-bad_1.18.5.bb => gstreamer1.0-plugins-bad_1.18.6.bb} (98%)

diff --git a/meta/recipes-multimedia/gstreamer/gstreamer1.0-plugins-bad_1.18.5.bb b/meta/recipes-multimedia/gstreamer/gstreamer1.0-plugins-bad_1.18.6.bb
similarity index 98%
rename from meta/recipes-multimedia/gstreamer/gstreamer1.0-plugins-bad_1.18.5.bb
rename to meta/recipes-multimedia/gstreamer/gstreamer1.0-plugins-bad_1.18.6.bb
index 6209a62712..df148c0aff 100644
--- a/meta/recipes-multimedia/gstreamer/gstreamer1.0-plugins-bad_1.18.5.bb
+++ b/meta/recipes-multimedia/gstreamer/gstreamer1.0-plugins-bad_1.18.6.bb
@@ -11,7 +11,7 @@ SRC_URI = "https://gstreamer.freedesktop.org/src/gst-plugins-bad/gst-plugins-bad
file://0004-opencv-resolve-missing-opencv-data-dir-in-yocto-buil.patch \
file://0005-msdk-fix-includedir-path.patch \
"
-SRC_URI[sha256sum] = "a164923b94f0d08578a6fcaeaac6e0c05da788a46903a1086870e9ca45ad678e"
+SRC_URI[sha256sum] = "0b1b50ac6311f0c510248b6cd64d6d3c94369344828baa602db85ded5bc70ec9"

S = "${WORKDIR}/gst-plugins-bad-${PV}"

--
2.35.1


[hardknott][PATCH 03/12] gstreamer1.0-plugins-good: 1.18.5 -> 1.18.6

Jose Quaresma
 

Signed-off-by: Jose Quaresma <quaresma.jose@...>
---
...ugins-good_1.18.5.bb => gstreamer1.0-plugins-good_1.18.6.bb} | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
rename meta/recipes-multimedia/gstreamer/{gstreamer1.0-plugins-good_1.18.5.bb => gstreamer1.0-plugins-good_1.18.6.bb} (97%)

diff --git a/meta/recipes-multimedia/gstreamer/gstreamer1.0-plugins-good_1.18.5.bb b/meta/recipes-multimedia/gstreamer/gstreamer1.0-plugins-good_1.18.6.bb
similarity index 97%
rename from meta/recipes-multimedia/gstreamer/gstreamer1.0-plugins-good_1.18.5.bb
rename to meta/recipes-multimedia/gstreamer/gstreamer1.0-plugins-good_1.18.6.bb
index ade935df9e..a43b8095c5 100644
--- a/meta/recipes-multimedia/gstreamer/gstreamer1.0-plugins-good_1.18.5.bb
+++ b/meta/recipes-multimedia/gstreamer/gstreamer1.0-plugins-good_1.18.6.bb
@@ -8,7 +8,7 @@ SRC_URI = "https://gstreamer.freedesktop.org/src/gst-plugins-good/gst-plugins-go
file://0001-qt-include-ext-qt-gstqtgl.h-instead-of-gst-gl-gstglf.patch \
"

-SRC_URI[sha256sum] = "3aaeeea7765fbf8801acce4a503a9b05f73f04e8a35352e9d00232cfd555796b"
+SRC_URI[sha256sum] = "26723ac01fcb360ade1f41d168c7c322d8af4ceb7e55c8c12ed2690d06a76eed"

S = "${WORKDIR}/gst-plugins-good-${PV}"

--
2.35.1


[hardknott][PATCH 02/12] gstreamer1.0-plugins-base: 1.18.5 -> 1.18.6

Jose Quaresma
 

Signed-off-by: Jose Quaresma <quaresma.jose@...>
---
...ugins-base_1.18.5.bb => gstreamer1.0-plugins-base_1.18.6.bb} | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
rename meta/recipes-multimedia/gstreamer/{gstreamer1.0-plugins-base_1.18.5.bb => gstreamer1.0-plugins-base_1.18.6.bb} (97%)

diff --git a/meta/recipes-multimedia/gstreamer/gstreamer1.0-plugins-base_1.18.5.bb b/meta/recipes-multimedia/gstreamer/gstreamer1.0-plugins-base_1.18.6.bb
similarity index 97%
rename from meta/recipes-multimedia/gstreamer/gstreamer1.0-plugins-base_1.18.5.bb
rename to meta/recipes-multimedia/gstreamer/gstreamer1.0-plugins-base_1.18.6.bb
index 5e2cca3864..e20145ea8d 100644
--- a/meta/recipes-multimedia/gstreamer/gstreamer1.0-plugins-base_1.18.5.bb
+++ b/meta/recipes-multimedia/gstreamer/gstreamer1.0-plugins-base_1.18.6.bb
@@ -12,7 +12,7 @@ SRC_URI = "https://gstreamer.freedesktop.org/src/gst-plugins-base/gst-plugins-ba
file://0002-ssaparse-enhance-SSA-text-lines-parsing.patch \
file://0004-glimagesink-Downrank-to-marginal.patch \
"
-SRC_URI[sha256sum] = "960b7af4585700db0fdd5b843554e11e2564fed9e061f591fae88a7be6446fa3"
+SRC_URI[sha256sum] = "56a9ff2fe9e6603b9e658cf6897d412a173d2180829fe01e92568549c6bd0f5b"

S = "${WORKDIR}/gst-plugins-base-${PV}"

--
2.35.1


[hardknott][PATCH 01/12] gstreamer1.0: 1.18.5 -> 1.18.6

Jose Quaresma
 

Signed-off-by: Jose Quaresma <quaresma.jose@...>
---
.../{gstreamer1.0_1.18.5.bb => gstreamer1.0_1.18.6.bb} | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
rename meta/recipes-multimedia/gstreamer/{gstreamer1.0_1.18.5.bb => gstreamer1.0_1.18.6.bb} (97%)

diff --git a/meta/recipes-multimedia/gstreamer/gstreamer1.0_1.18.5.bb b/meta/recipes-multimedia/gstreamer/gstreamer1.0_1.18.6.bb
similarity index 97%
rename from meta/recipes-multimedia/gstreamer/gstreamer1.0_1.18.5.bb
rename to meta/recipes-multimedia/gstreamer/gstreamer1.0_1.18.6.bb
index 0d82dd338c..4bf89ed109 100644
--- a/meta/recipes-multimedia/gstreamer/gstreamer1.0_1.18.5.bb
+++ b/meta/recipes-multimedia/gstreamer/gstreamer1.0_1.18.6.bb
@@ -25,7 +25,7 @@ SRC_URI = "https://gstreamer.freedesktop.org/src/gstreamer/gstreamer-${PV}.tar.x
file://0006-tests-use-a-dictionaries-for-environment.patch \
file://0007-tests-install-the-environment-for-installed_tests.patch \
"
-SRC_URI[sha256sum] = "55862232a63459bbf56abebde3085ca9aec211b478e891dacea4d6df8cafe80a"
+SRC_URI[sha256sum] = "4ec816010dd4d3a93cf470ad0a6f25315f52b204eb1d71dfa70ab8a1c3bd06e6"

PACKAGECONFIG ??= "${@bb.utils.contains('PTEST_ENABLED', '1', 'tests', '', d)} \
check \
--
2.35.1


[PATCH 3/3] oeqa/selftest/bbtests: Update after license changes

Richard Purdie
 

Update the test to match the renamed license in the recipe.

Signed-off-by: Richard Purdie <richard.purdie@...>
---
meta/lib/oeqa/selftest/cases/bbtests.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/lib/oeqa/selftest/cases/bbtests.py b/meta/lib/oeqa/selftest/cases/bbtests.py
index e2b9290200a..ce72c4bcc69 100644
--- a/meta/lib/oeqa/selftest/cases/bbtests.py
+++ b/meta/lib/oeqa/selftest/cases/bbtests.py
@@ -228,8 +228,8 @@ INHERIT:remove = \"report-error\"
result = bitbake('selftest-ed', ignore_status=True)
self.assertEqual(result.status, 0, "Bitbake failed, exit code %s, output %s" % (result.status, result.output))
lic_dir = get_bb_var('LICENSE_DIRECTORY')
- self.assertFalse(os.path.isfile(os.path.join(lic_dir, 'selftest-ed/generic_GPLv3')))
- self.assertTrue(os.path.isfile(os.path.join(lic_dir, 'selftest-ed/generic_GPLv2')))
+ self.assertFalse(os.path.isfile(os.path.join(lic_dir, 'selftest-ed/generic_GPL-3.0-or-later')))
+ self.assertTrue(os.path.isfile(os.path.join(lic_dir, 'selftest-ed/generic_GPL-2.0-or-later')))

def test_setscene_only(self):
""" Bitbake option to restore from sstate only within a build (i.e. execute no real tasks, only setscene)"""
--
2.32.0


[PATCH 2/3] expat: Upgrade 2.4.4 -> 2.4.5

Richard Purdie
 

This is a security fix release containing fixes for CVE-2022-25235, CVE-2022-25236,
CVE-2022-25313, CVE-2022-25314 and CVE-2022-25315.

Signed-off-by: Richard Purdie <richard.purdie@...>
---
meta/recipes-core/expat/{expat_2.4.4.bb => expat_2.4.5.bb} | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
rename meta/recipes-core/expat/{expat_2.4.4.bb => expat_2.4.5.bb} (91%)

diff --git a/meta/recipes-core/expat/expat_2.4.4.bb b/meta/recipes-core/expat/expat_2.4.5.bb
similarity index 91%
rename from meta/recipes-core/expat/expat_2.4.4.bb
rename to meta/recipes-core/expat/expat_2.4.5.bb
index 14080220d2c..83ba3a23291 100644
--- a/meta/recipes-core/expat/expat_2.4.4.bb
+++ b/meta/recipes-core/expat/expat_2.4.5.bb
@@ -14,7 +14,7 @@ SRC_URI = "https://github.com/libexpat/libexpat/releases/download/R_${VERSION_TA

UPSTREAM_CHECK_URI = "https://github.com/libexpat/libexpat/releases/"

-SRC_URI[sha256sum] = "14c58c2a0b5b8b31836514dfab41bd191836db7aa7b84ae5c47bc0327a20d64a"
+SRC_URI[sha256sum] = "fbb430f964c7a2db2626452b6769e6a8d5d23593a453ccbc21701b74deabedff"

EXTRA_OECMAKE:class-native += "-DEXPAT_BUILD_DOCS=OFF"

--
2.32.0


[PATCH 1/3] sstate: Setup fetcher environment in advance

Richard Purdie
 

The threading code here can race as the fetcher changes the environment which is
shared between the threads. By setting it up in advance, it isn't changed and
therefore no longer races.

Signed-off-by: Richard Purdie <richard.purdie@...>
---
meta/classes/sstate.bbclass | 21 ++++++++++++---------
1 file changed, 12 insertions(+), 9 deletions(-)

diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
index 2e1d9428c27..787172b408d 100644
--- a/meta/classes/sstate.bbclass
+++ b/meta/classes/sstate.bbclass
@@ -1022,15 +1022,18 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True,
msg = "Checking sstate mirror object availability"
bb.event.fire(bb.event.ProcessStarted(msg, len(tasklist)), d)

- bb.event.enable_threadlock()
- pool = oe.utils.ThreadedPool(nproc, len(tasklist),
- worker_init=checkstatus_init, worker_end=checkstatus_end,
- name="sstate_checkhashes-")
- for t in tasklist:
- pool.add_task(checkstatus, t)
- pool.start()
- pool.wait_completion()
- bb.event.disable_threadlock()
+ # Have to setup the fetcher environment here rather than in each thread as it would race
+ fetcherenv = bb.fetch2.get_fetcher_environment(d)
+ with bb.utils.environment(**fetcherenv):
+ bb.event.enable_threadlock()
+ pool = oe.utils.ThreadedPool(nproc, len(tasklist),
+ worker_init=checkstatus_init, worker_end=checkstatus_end,
+ name="sstate_checkhashes-")
+ for t in tasklist:
+ pool.add_task(checkstatus, t)
+ pool.start()
+ pool.wait_completion()
+ bb.event.disable_threadlock()

if progress:
bb.event.fire(bb.event.ProcessFinished(msg), d)
--
2.32.0


Re: [honister][PATCH V3] Rust Oe-Selftest implementation

Richard Purdie
 

Hi,

On Sat, 2022-02-19 at 00:06 -0800, pgowda wrote:
The patch implements Oe-selftest framework for Rust test.
Some of the functions are as follows:-
setup_cargo_environment(): Build bootstrap and some early stage tools.
do_rust_setup_snapshot(): Install the snapshot version of rust binaries.
do_configure(): To generate config.toml
do_compile(): To build "remote-test-server" for qemutarget image.

The python file builds remote-test-server and executes rust testing
remotely using background ssh. It adds the necessary test environment
and variables to run the rust oe-selftest.
Print the results in case of failure of runCmd().

The patch has been run and tested on X86 and X86_64 targets on
Ubuntu-18 successfully.

There is an issue of loading libserde for master branch rust-1.58.1
sources as follows:-
"command did not execute successfully: oe-selftest/build-st/tmp/work
/core2-64-poky-linux/rust-testsuite/1.58.1-r0/rustc-1.58.1-src/build/
x86_64-unknown-linux-gnu/stage0-tools-bin/remote-test-client" "push"
"oe-selftest/build-st/tmp/work/core2-64-poky-linux/rust-testsuite/
1.58.1-r0/rustc-1.58.1-src/build/x86_64-unknown-linux-gnu/stage1/lib/
rustlib/x86_64-unknown-linux-gnu/lib/libserde_derive-804e1d2731595192.so"

thread 'main' panicked at 'io::copy(&mut file, dst) failed with
Connection reset by peer (os error 104)',
src/tools/remote-test-client/src/main.rs:353:5

The patch will be posted for master branch after fixing the issue.

Signed-off-by: pgowda <pgowda.cve@...>
Signed-off-by: Vinay Kumar <vinay.m.engg@...>
This looks interesting and is improving, thanks. I haven't done a full review on
it but as I glanced through it, I had some questions. See inline below.

---
meta/lib/oeqa/selftest/cases/rust.py | 53 ++
meta/recipes-devtools/rust/rust-testsuite.inc | 170 ++++
.../rust-testsuite/rust-oe-selftest.patch | 872 ++++++++++++++++++
.../rust/rust-testsuite_1.54.0.bb | 3 +
4 files changed, 1098 insertions(+)
create mode 100644 meta/lib/oeqa/selftest/cases/rust.py
create mode 100644 meta/recipes-devtools/rust/rust-testsuite.inc
create mode 100644 meta/recipes-devtools/rust/rust-testsuite/rust-oe-selftest.patch
create mode 100644 meta/recipes-devtools/rust/rust-testsuite_1.54.0.bb

diff --git a/meta/lib/oeqa/selftest/cases/rust.py b/meta/lib/oeqa/selftest/cases/rust.py
new file mode 100644
index 0000000000..ad28f7ab26
--- /dev/null
+++ b/meta/lib/oeqa/selftest/cases/rust.py
@@ -0,0 +1,53 @@
+# SPDX-License-Identifier: MIT
+import os
+import subprocess
+from oeqa.core.decorator import OETestTag
+from oeqa.core.case import OEPTestResultTestCase
+from oeqa.selftest.case import OESelftestTestCase
+from oeqa.utils.commands import runCmd, bitbake, get_bb_var, get_bb_vars, runqemu, Command
+from oeqa.utils.sshcontrol import SSHControl
+
+# Total time taken for testing is of about 2hr 20min, with PARALLEL_MAKE set to 40 number of jobs.
+class RustSelfTestBase(OESelftestTestCase, OEPTestResultTestCase):
+
+ def run_check_emulated(self, *args, **kwargs):
+ # build remote-test-server before image build
+ recipe = "rust-testsuite"
+ bitbake("{} -c compile".format(recipe))
+ builddir = get_bb_var("B", "rust-testsuite")
+ # build core-image-minimal with required packages
+ default_installed_packages = ["libgcc", "libstdc++", "libatomic", "libgomp"]
+ features = []
+ features.append('IMAGE_FEATURES += "ssh-server-openssh"')
+ features.append('CORE_IMAGE_EXTRA_INSTALL += "{0}"'.format(" ".join(default_installed_packages)))
+ self.write_config("\n".join(features))
+ bitbake("core-image-minimal")
+ # wrap the execution with a qemu instance
+ with runqemu("core-image-minimal", runqemuparams = "nographic", qemuparams = "-m 512") as qemu:
+ # Copy remote-test-server to image through scp
+ ssh = SSHControl(ip=qemu.ip, logfile=qemu.sshlog, user="root")
+ ssh.copy_to(builddir + "/" + "build/x86_64-unknown-linux-gnu/stage1-tools-bin/remote-test-server","~/")
+ # Execute remote-test-server on image through background ssh
+ command = '~/remote-test-server -v remote'
+ sshrun=subprocess.Popen(("ssh", '-o', 'UserKnownHostsFile=/dev/null', '-o', 'StrictHostKeyChecking=no', '-f', "root@%s" % qemu.ip, command),
+ shell=False,
+ stdout=subprocess.PIPE,
+ stderr=subprocess.PIPE)
+ # Get the values of variables.
+ targetsys = get_bb_var("TARGET_SYS", "rust-testsuite")
+ rustlibpath = get_bb_var("STAGING_LIBDIR_NATIVE", "rust-testsuite")
+ tmpdir = get_bb_var("TMPDIR", "rust-testsuite")
+ testargs = "--exclude compiler/rustc --exclude compiler/rustc_apfloat --exclude compiler/rustc_serialize --exclude src/tools/tidy --exclude src/tools/compiletest --no-fail-fast --bless"
Why do we patch some tests out but exclude others here? What is the difference?

+ # Set path for target-poky-linux-gcc, RUST_TARGET_PATH and hosttools.
+ cmd = " export PATH=%s/../bin:$PATH;" % rustlibpath
+ cmd = cmd + " export PATH=%s/../bin/%s:%s/hosttools:$PATH;" % (rustlibpath, targetsys, tmpdir)
+ cmd = cmd + " export RUST_TARGET_PATH=%s/rustlib;" % rustlibpath
+ # Trigger testing.
+ cmd = cmd + " export TEST_DEVICE_ADDR=\"%s:12345\";" % qemu.ip
+ cmd = cmd + " cd %s; python3 src/bootstrap/bootstrap.py -j 40 test %s --target %s ;" % (builddir, testargs, targetsys)
What does 40 mean here?

+ result = runCmd(cmd)
+
+@OETestTag("toolchain-system")
+class RustSelfTestSystemEmulated(RustSelfTestBase):
+ def test_rust(self):
+ self.run_check_emulated("rust")
diff --git a/meta/recipes-devtools/rust/rust-testsuite.inc b/meta/recipes-devtools/rust/rust-testsuite.inc
new file mode 100644
index 0000000000..8a6698a8e2
--- /dev/null
+++ b/meta/recipes-devtools/rust/rust-testsuite.inc
@@ -0,0 +1,170 @@
+SUMMARY = "Rust testing"
+HOMEPAGE = "https://rustc-dev-guide.rust-lang.org/tests/intro.html"
+SECTION = "test"
+LICENSE = "MIT | Apache-2.0"
+
+SRC_URI += "file://rust-oe-selftest.patch;striplevel=1"
+
+inherit rust
+inherit cargo_common
+
+DEPENDS += "file-native python3-native"
python3-native is normally through an inherit python3native so this could mean
you're not using it?

+EXCLUDE_FROM_WORLD = "1"
+
+S = "${RUSTSRC}"
+
+# Path of target specification file "target-poky-linux.json"
+export RUST_TARGET_PATH="${STAGING_LIBDIR_NATIVE}/rustlib"
+
+export FORCE_CRATE_HASH="${BB_TASKHASH}"
+
+# We don't want to use bitbakes vendoring because the rust sources do their
+# own vendoring.
+CARGO_DISABLE_BITBAKE_VENDORING = "1"
+
+# We can't use RUST_BUILD_SYS here because that may be "musl" if
+# TCLIBC="musl". Snapshots are always -unknown-linux-gnu
+SNAPSHOT_BUILD_SYS = "${BUILD_ARCH}-unknown-linux-gnu"
+setup_cargo_environment () {
+ # The first step is to build bootstrap and some early stage tools,
+ # these are build for the same target as the snapshot, e.g.
+ # x86_64-unknown-linux-gnu.
+ # Later stages are build for the native target (i.e. target.x86_64-linux)
+ cargo_common_do_configure
+
+ printf '[target.%s]\n' "${SNAPSHOT_BUILD_SYS}" >> ${CARGO_HOME}/config
+ printf "linker = '%s'\n" "${RUST_BUILD_CCLD}" >> ${CARGO_HOME}/config
+}
+
+include rust-common.inc
+
+do_rust_setup_snapshot () {
+ for installer in "${WORKDIR}/rust-snapshot-components/"*"/install.sh"; do
+ "${installer}" --prefix="${WORKDIR}/rust-snapshot" --disable-ldconfig
+ done
+
+ # Some versions of rust (e.g. 1.18.0) tries to find cargo in stage0/bin/cargo
+ # and fail without it there.
+ mkdir -p ${RUSTSRC}/build/${BUILD_SYS}
+ ln -sf ${WORKDIR}/rust-snapshot/ ${RUSTSRC}/build/${BUILD_SYS}/stage0
We're quite past 1.18.0 now so is that still needed?

+
+ # Need to use uninative's loader if enabled/present since the library paths
+ # are used internally by rust and result in symbol mismatches if we don't
+ if [ ! -z "${UNINATIVE_LOADER}" -a -e "${UNINATIVE_LOADER}" ]; then
+ for bin in cargo rustc rustdoc; do
+ patchelf-uninative ${WORKDIR}/rust-snapshot/bin/$bin --set-interpreter ${UNINATIVE_LOADER}
+ done
+ fi
+}
+addtask rust_setup_snapshot after do_unpack before do_configure
+do_rust_setup_snapshot[dirs] += "${WORKDIR}/rust-snapshot"
+
+python do_configure() {
+ import json
+ from distutils.version import LooseVersion
We've just removed most LooseVersion references from OE-Core since disutils is
deprecated. You can use bb.utils functions instead, like here:

https://git.yoctoproject.org/poky/commit/?id=f0627490711f29f0e308a0afc4fc4f8e54f58dec


+ try:
+ import configparser
+ except ImportError:
+ import ConfigParser as configparser
We don't need/want python 2 support here?

+
+ # toml is rather similar to standard ini like format except it likes values
+ # that look more JSON like. So for our purposes simply escaping all values
+ # as JSON seem to work fine.
+
+ e = lambda s: json.dumps(s)
+
+ config = configparser.RawConfigParser()
+
+ # [target.ARCH-unknown-linux-gnu] in case of x86_64 [target.ARCH-poky-linux]
+ target_section = "target.{}".format(d.getVar('TARGET_SYS', True))
+ config.add_section(target_section)
+
+ # Points to wrapper files which contain target specific compiler and linker commands.
+ config.set(target_section, "cxx", e(d.expand("${RUST_TARGET_CXX}")))
+ config.set(target_section, "cc", e(d.expand("${RUST_TARGET_CC}")))
+ config.set(target_section, "linker", e(d.expand("${RUST_TARGET_CCLD}")))
+
+ # If we don't do this rust-native will compile it's own llvm for BUILD.
+ # [target.${BUILD_ARCH}-unknown-linux-gnu]
+ target_section = "target.{}".format(d.getVar('SNAPSHOT_BUILD_SYS', True))
+ config.add_section(target_section)
+
+ # Wrapper scripts of build system.
+ config.set(target_section, "cxx", e(d.expand("${RUST_BUILD_CXX}")))
+ config.set(target_section, "cc", e(d.expand("${RUST_BUILD_CC}")))
+
+ # [llvm]
+ config.add_section("llvm")
+ config.set("llvm", "targets", e("ARM;AArch64;Mips;PowerPC;RISCV;X86"))
+ config.set("llvm", "ninja", e(False))
+
+ # [rust]
+ config.add_section("rust")
+ config.set("rust", "rpath", e(True))
+ config.set("rust", "channel", e("stable"))
+
+ if LooseVersion(d.getVar("PV")) < LooseVersion("1.32.0"):
+ config.set("rust", "use-jemalloc", e(False))
We're now a much later version of rust so do we need to support 1.32? That could
solve the LooseVersion problem too!

+
+ # Whether or not to optimize the compiler and standard library
+ config.set("rust", "optimize", e(True))
+
+ # Emits extraneous output from tests to ensure that failures of the test
+ # harness are debuggable just from logfiles
+ config.set("rust", "verbose-tests", e(True))
+
+ # Override default linker cc.
+ config.set("rust", "default-linker", e(d.expand("${RUST_BUILD_CCLD}")))
+
+ # [build]
+ config.add_section("build")
+ config.set("build", "submodules", e(False))
+ config.set("build", "docs", e(False))
+
+ rustc = d.expand("${WORKDIR}/rust-snapshot/bin/rustc")
+ config.set("build", "rustc", e(rustc))
+
+ cargo = d.expand("${WORKDIR}/rust-snapshot/bin/cargo")
+ config.set("build", "cargo", e(cargo))
+
+ config.set("build", "vendor", e(True))
+
+ targets = [d.getVar("TARGET_SYS", True)]
+ config.set("build", "target", e(targets))
+
+ hosts = [d.getVar("SNAPSHOT_BUILD_SYS", True)]
+ config.set("build", "host", e(hosts))
+
+ # We can't use BUILD_SYS since that is something the rust snapshot knows
+ # nothing about when trying to build some stage0 tools (like fabricate)
+ config.set("build", "build", e(d.getVar("SNAPSHOT_BUILD_SYS", True)))
+
+ with open("config.toml", "w") as f:
+ config.write(f)
+
+ # set up ${WORKDIR}/cargo_home
+ bb.build.exec_func("setup_cargo_environment", d)
+}
+
+
+rust_runx () {
+ echo "COMPILE ${PN}" "$@"
+
+ # CFLAGS, LDFLAGS, CXXFLAGS, CPPFLAGS are used by rust's build for a
+ # wide range of targets (not just TARGET). Yocto's settings for them will
s/Yocto/OE/

+ # be inappropriate, avoid using.
+ unset CFLAGS
+ unset LDFLAGS
+ unset CXXFLAGS
+ unset CPPFLAGS
+
+ oe_cargo_fix_env
+
+ python3 src/bootstrap/bootstrap.py ${@oe.utils.parallel_make_argument(d, '-j %d')} "$@" --verbose
+}
+rust_runx[vardepsexclude] += "PARALLEL_MAKE"
PARALLEL_MAKE isn't used above so I doubt this is needed now? (I realise the
oe.utils function uses it)

+
+do_compile () {
+
+ rust_runx build src/tools/remote-test-server --target "${TARGET_SYS}"
+}


[honister][PATCH V3] Rust Oe-Selftest implementation

Pgowda
 

The patch implements Oe-selftest framework for Rust test.
Some of the functions are as follows:-
setup_cargo_environment(): Build bootstrap and some early stage tools.
do_rust_setup_snapshot(): Install the snapshot version of rust binaries.
do_configure(): To generate config.toml
do_compile(): To build "remote-test-server" for qemutarget image.

The python file builds remote-test-server and executes rust testing
remotely using background ssh. It adds the necessary test environment
and variables to run the rust oe-selftest.
Print the results in case of failure of runCmd().

The patch has been run and tested on X86 and X86_64 targets on
Ubuntu-18 successfully.

There is an issue of loading libserde for master branch rust-1.58.1
sources as follows:-
"command did not execute successfully: oe-selftest/build-st/tmp/work
/core2-64-poky-linux/rust-testsuite/1.58.1-r0/rustc-1.58.1-src/build/
x86_64-unknown-linux-gnu/stage0-tools-bin/remote-test-client" "push"
"oe-selftest/build-st/tmp/work/core2-64-poky-linux/rust-testsuite/
1.58.1-r0/rustc-1.58.1-src/build/x86_64-unknown-linux-gnu/stage1/lib/
rustlib/x86_64-unknown-linux-gnu/lib/libserde_derive-804e1d2731595192.so"

thread 'main' panicked at 'io::copy(&mut file, dst) failed with
Connection reset by peer (os error 104)',
src/tools/remote-test-client/src/main.rs:353:5

The patch will be posted for master branch after fixing the issue.

Signed-off-by: pgowda <pgowda.cve@...>
Signed-off-by: Vinay Kumar <vinay.m.engg@...>
---
meta/lib/oeqa/selftest/cases/rust.py | 53 ++
meta/recipes-devtools/rust/rust-testsuite.inc | 170 ++++
.../rust-testsuite/rust-oe-selftest.patch | 872 ++++++++++++++++++
.../rust/rust-testsuite_1.54.0.bb | 3 +
4 files changed, 1098 insertions(+)
create mode 100644 meta/lib/oeqa/selftest/cases/rust.py
create mode 100644 meta/recipes-devtools/rust/rust-testsuite.inc
create mode 100644 meta/recipes-devtools/rust/rust-testsuite/rust-oe-selftest.patch
create mode 100644 meta/recipes-devtools/rust/rust-testsuite_1.54.0.bb

diff --git a/meta/lib/oeqa/selftest/cases/rust.py b/meta/lib/oeqa/selftest/cases/rust.py
new file mode 100644
index 0000000000..ad28f7ab26
--- /dev/null
+++ b/meta/lib/oeqa/selftest/cases/rust.py
@@ -0,0 +1,53 @@
+# SPDX-License-Identifier: MIT
+import os
+import subprocess
+from oeqa.core.decorator import OETestTag
+from oeqa.core.case import OEPTestResultTestCase
+from oeqa.selftest.case import OESelftestTestCase
+from oeqa.utils.commands import runCmd, bitbake, get_bb_var, get_bb_vars, runqemu, Command
+from oeqa.utils.sshcontrol import SSHControl
+
+# Total time taken for testing is of about 2hr 20min, with PARALLEL_MAKE set to 40 number of jobs.
+class RustSelfTestBase(OESelftestTestCase, OEPTestResultTestCase):
+
+ def run_check_emulated(self, *args, **kwargs):
+ # build remote-test-server before image build
+ recipe = "rust-testsuite"
+ bitbake("{} -c compile".format(recipe))
+ builddir = get_bb_var("B", "rust-testsuite")
+ # build core-image-minimal with required packages
+ default_installed_packages = ["libgcc", "libstdc++", "libatomic", "libgomp"]
+ features = []
+ features.append('IMAGE_FEATURES += "ssh-server-openssh"')
+ features.append('CORE_IMAGE_EXTRA_INSTALL += "{0}"'.format(" ".join(default_installed_packages)))
+ self.write_config("\n".join(features))
+ bitbake("core-image-minimal")
+ # wrap the execution with a qemu instance
+ with runqemu("core-image-minimal", runqemuparams = "nographic", qemuparams = "-m 512") as qemu:
+ # Copy remote-test-server to image through scp
+ ssh = SSHControl(ip=qemu.ip, logfile=qemu.sshlog, user="root")
+ ssh.copy_to(builddir + "/" + "build/x86_64-unknown-linux-gnu/stage1-tools-bin/remote-test-server","~/")
+ # Execute remote-test-server on image through background ssh
+ command = '~/remote-test-server -v remote'
+ sshrun=subprocess.Popen(("ssh", '-o', 'UserKnownHostsFile=/dev/null', '-o', 'StrictHostKeyChecking=no', '-f', "root@%s" % qemu.ip, command),
+ shell=False,
+ stdout=subprocess.PIPE,
+ stderr=subprocess.PIPE)
+ # Get the values of variables.
+ targetsys = get_bb_var("TARGET_SYS", "rust-testsuite")
+ rustlibpath = get_bb_var("STAGING_LIBDIR_NATIVE", "rust-testsuite")
+ tmpdir = get_bb_var("TMPDIR", "rust-testsuite")
+ testargs = "--exclude compiler/rustc --exclude compiler/rustc_apfloat --exclude compiler/rustc_serialize --exclude src/tools/tidy --exclude src/tools/compiletest --no-fail-fast --bless"
+ # Set path for target-poky-linux-gcc, RUST_TARGET_PATH and hosttools.
+ cmd = " export PATH=%s/../bin:$PATH;" % rustlibpath
+ cmd = cmd + " export PATH=%s/../bin/%s:%s/hosttools:$PATH;" % (rustlibpath, targetsys, tmpdir)
+ cmd = cmd + " export RUST_TARGET_PATH=%s/rustlib;" % rustlibpath
+ # Trigger testing.
+ cmd = cmd + " export TEST_DEVICE_ADDR=\"%s:12345\";" % qemu.ip
+ cmd = cmd + " cd %s; python3 src/bootstrap/bootstrap.py -j 40 test %s --target %s ;" % (builddir, testargs, targetsys)
+ result = runCmd(cmd)
+
+@OETestTag("toolchain-system")
+class RustSelfTestSystemEmulated(RustSelfTestBase):
+ def test_rust(self):
+ self.run_check_emulated("rust")
diff --git a/meta/recipes-devtools/rust/rust-testsuite.inc b/meta/recipes-devtools/rust/rust-testsuite.inc
new file mode 100644
index 0000000000..8a6698a8e2
--- /dev/null
+++ b/meta/recipes-devtools/rust/rust-testsuite.inc
@@ -0,0 +1,170 @@
+SUMMARY = "Rust testing"
+HOMEPAGE = "https://rustc-dev-guide.rust-lang.org/tests/intro.html"
+SECTION = "test"
+LICENSE = "MIT | Apache-2.0"
+
+SRC_URI += "file://rust-oe-selftest.patch;striplevel=1"
+
+inherit rust
+inherit cargo_common
+
+DEPENDS += "file-native python3-native"
+EXCLUDE_FROM_WORLD = "1"
+
+S = "${RUSTSRC}"
+
+# Path of target specification file "target-poky-linux.json"
+export RUST_TARGET_PATH="${STAGING_LIBDIR_NATIVE}/rustlib"
+
+export FORCE_CRATE_HASH="${BB_TASKHASH}"
+
+# We don't want to use bitbakes vendoring because the rust sources do their
+# own vendoring.
+CARGO_DISABLE_BITBAKE_VENDORING = "1"
+
+# We can't use RUST_BUILD_SYS here because that may be "musl" if
+# TCLIBC="musl". Snapshots are always -unknown-linux-gnu
+SNAPSHOT_BUILD_SYS = "${BUILD_ARCH}-unknown-linux-gnu"
+setup_cargo_environment () {
+ # The first step is to build bootstrap and some early stage tools,
+ # these are build for the same target as the snapshot, e.g.
+ # x86_64-unknown-linux-gnu.
+ # Later stages are build for the native target (i.e. target.x86_64-linux)
+ cargo_common_do_configure
+
+ printf '[target.%s]\n' "${SNAPSHOT_BUILD_SYS}" >> ${CARGO_HOME}/config
+ printf "linker = '%s'\n" "${RUST_BUILD_CCLD}" >> ${CARGO_HOME}/config
+}
+
+include rust-common.inc
+
+do_rust_setup_snapshot () {
+ for installer in "${WORKDIR}/rust-snapshot-components/"*"/install.sh"; do
+ "${installer}" --prefix="${WORKDIR}/rust-snapshot" --disable-ldconfig
+ done
+
+ # Some versions of rust (e.g. 1.18.0) tries to find cargo in stage0/bin/cargo
+ # and fail without it there.
+ mkdir -p ${RUSTSRC}/build/${BUILD_SYS}
+ ln -sf ${WORKDIR}/rust-snapshot/ ${RUSTSRC}/build/${BUILD_SYS}/stage0
+
+ # Need to use uninative's loader if enabled/present since the library paths
+ # are used internally by rust and result in symbol mismatches if we don't
+ if [ ! -z "${UNINATIVE_LOADER}" -a -e "${UNINATIVE_LOADER}" ]; then
+ for bin in cargo rustc rustdoc; do
+ patchelf-uninative ${WORKDIR}/rust-snapshot/bin/$bin --set-interpreter ${UNINATIVE_LOADER}
+ done
+ fi
+}
+addtask rust_setup_snapshot after do_unpack before do_configure
+do_rust_setup_snapshot[dirs] += "${WORKDIR}/rust-snapshot"
+
+python do_configure() {
+ import json
+ from distutils.version import LooseVersion
+ try:
+ import configparser
+ except ImportError:
+ import ConfigParser as configparser
+
+ # toml is rather similar to standard ini like format except it likes values
+ # that look more JSON like. So for our purposes simply escaping all values
+ # as JSON seem to work fine.
+
+ e = lambda s: json.dumps(s)
+
+ config = configparser.RawConfigParser()
+
+ # [target.ARCH-unknown-linux-gnu] in case of x86_64 [target.ARCH-poky-linux]
+ target_section = "target.{}".format(d.getVar('TARGET_SYS', True))
+ config.add_section(target_section)
+
+ # Points to wrapper files which contain target specific compiler and linker commands.
+ config.set(target_section, "cxx", e(d.expand("${RUST_TARGET_CXX}")))
+ config.set(target_section, "cc", e(d.expand("${RUST_TARGET_CC}")))
+ config.set(target_section, "linker", e(d.expand("${RUST_TARGET_CCLD}")))
+
+ # If we don't do this rust-native will compile it's own llvm for BUILD.
+ # [target.${BUILD_ARCH}-unknown-linux-gnu]
+ target_section = "target.{}".format(d.getVar('SNAPSHOT_BUILD_SYS', True))
+ config.add_section(target_section)
+
+ # Wrapper scripts of build system.
+ config.set(target_section, "cxx", e(d.expand("${RUST_BUILD_CXX}")))
+ config.set(target_section, "cc", e(d.expand("${RUST_BUILD_CC}")))
+
+ # [llvm]
+ config.add_section("llvm")
+ config.set("llvm", "targets", e("ARM;AArch64;Mips;PowerPC;RISCV;X86"))
+ config.set("llvm", "ninja", e(False))
+
+ # [rust]
+ config.add_section("rust")
+ config.set("rust", "rpath", e(True))
+ config.set("rust", "channel", e("stable"))
+
+ if LooseVersion(d.getVar("PV")) < LooseVersion("1.32.0"):
+ config.set("rust", "use-jemalloc", e(False))
+
+ # Whether or not to optimize the compiler and standard library
+ config.set("rust", "optimize", e(True))
+
+ # Emits extraneous output from tests to ensure that failures of the test
+ # harness are debuggable just from logfiles
+ config.set("rust", "verbose-tests", e(True))
+
+ # Override default linker cc.
+ config.set("rust", "default-linker", e(d.expand("${RUST_BUILD_CCLD}")))
+
+ # [build]
+ config.add_section("build")
+ config.set("build", "submodules", e(False))
+ config.set("build", "docs", e(False))
+
+ rustc = d.expand("${WORKDIR}/rust-snapshot/bin/rustc")
+ config.set("build", "rustc", e(rustc))
+
+ cargo = d.expand("${WORKDIR}/rust-snapshot/bin/cargo")
+ config.set("build", "cargo", e(cargo))
+
+ config.set("build", "vendor", e(True))
+
+ targets = [d.getVar("TARGET_SYS", True)]
+ config.set("build", "target", e(targets))
+
+ hosts = [d.getVar("SNAPSHOT_BUILD_SYS", True)]
+ config.set("build", "host", e(hosts))
+
+ # We can't use BUILD_SYS since that is something the rust snapshot knows
+ # nothing about when trying to build some stage0 tools (like fabricate)
+ config.set("build", "build", e(d.getVar("SNAPSHOT_BUILD_SYS", True)))
+
+ with open("config.toml", "w") as f:
+ config.write(f)
+
+ # set up ${WORKDIR}/cargo_home
+ bb.build.exec_func("setup_cargo_environment", d)
+}
+
+
+rust_runx () {
+ echo "COMPILE ${PN}" "$@"
+
+ # CFLAGS, LDFLAGS, CXXFLAGS, CPPFLAGS are used by rust's build for a
+ # wide range of targets (not just TARGET). Yocto's settings for them will
+ # be inappropriate, avoid using.
+ unset CFLAGS
+ unset LDFLAGS
+ unset CXXFLAGS
+ unset CPPFLAGS
+
+ oe_cargo_fix_env
+
+ python3 src/bootstrap/bootstrap.py ${@oe.utils.parallel_make_argument(d, '-j %d')} "$@" --verbose
+}
+rust_runx[vardepsexclude] += "PARALLEL_MAKE"
+
+do_compile () {
+
+ rust_runx build src/tools/remote-test-server --target "${TARGET_SYS}"
+}
diff --git a/meta/recipes-devtools/rust/rust-testsuite/rust-oe-selftest.patch b/meta/recipes-devtools/rust/rust-testsuite/rust-oe-selftest.patch
new file mode 100644
index 0000000000..78f061d028
--- /dev/null
+++ b/meta/recipes-devtools/rust/rust-testsuite/rust-oe-selftest.patch
@@ -0,0 +1,872 @@
+Rust testsuite outputs error even on a single testcase failure.
+Hence, some test runs are ignored as they fail with error messages.
+
+Signed-off-by: Pgowda <pgowda.cve@...>
+---
+diff -upr a/compiler/rustc_arena/Cargo.toml b/compiler/rustc_arena/Cargo.toml
+--- a/compiler/rustc_arena/Cargo.toml 2022-02-18 20:52:49.343261926 -0800
++++ b/compiler/rustc_arena/Cargo.toml 2022-02-18 20:22:40.745863059 -0800
+@@ -4,6 +4,10 @@ name = "rustc_arena"
+ version = "0.0.0"
+ edition = "2018"
+
++[lib]
++test = false
++doctest = false
++
+ [dependencies]
+ rustc_data_structures = { path = "../rustc_data_structures" }
+ smallvec = { version = "1.6.1", features = ["union", "may_dangle"] }
+diff -upr a/compiler/rustc_ast/Cargo.toml b/compiler/rustc_ast/Cargo.toml
+--- a/compiler/rustc_ast/Cargo.toml 2022-02-18 20:52:49.343261926 -0800
++++ b/compiler/rustc_ast/Cargo.toml 2022-02-18 20:22:40.745863059 -0800
+@@ -5,6 +5,7 @@ version = "0.0.0"
+ edition = "2018"
+
+ [lib]
++test = false
+ doctest = false
+
+ [dependencies]
+diff -upr a/compiler/rustc_ast_lowering/Cargo.toml b/compiler/rustc_ast_lowering/Cargo.toml
+--- a/compiler/rustc_ast_lowering/Cargo.toml 2022-02-18 20:52:49.343261926 -0800
++++ b/compiler/rustc_ast_lowering/Cargo.toml 2022-02-18 20:22:40.745863059 -0800
+@@ -5,6 +5,7 @@ version = "0.0.0"
+ edition = "2018"
+
+ [lib]
++test = false
+ doctest = false
+
+ [dependencies]
+diff -upr a/compiler/rustc_ast_passes/Cargo.toml b/compiler/rustc_ast_passes/Cargo.toml
+--- a/compiler/rustc_ast_passes/Cargo.toml 2022-02-18 20:52:49.343261926 -0800
++++ b/compiler/rustc_ast_passes/Cargo.toml 2022-02-18 20:22:40.745863059 -0800
+@@ -4,6 +4,10 @@ name = "rustc_ast_passes"
+ version = "0.0.0"
+ edition = "2018"
+
++[lib]
++test = false
++doctest = false
++
+ [dependencies]
+ itertools = "0.9"
+ tracing = "0.1"
+diff -upr a/compiler/rustc_ast_pretty/Cargo.toml b/compiler/rustc_ast_pretty/Cargo.toml
+--- a/compiler/rustc_ast_pretty/Cargo.toml 2022-02-18 20:52:49.343261926 -0800
++++ b/compiler/rustc_ast_pretty/Cargo.toml 2022-02-18 20:22:40.745863059 -0800
+@@ -5,6 +5,7 @@ version = "0.0.0"
+ edition = "2018"
+
+ [lib]
++test = false
+ doctest = false
+
+ [dependencies]
+diff -upr a/compiler/rustc_attr/Cargo.toml b/compiler/rustc_attr/Cargo.toml
+--- a/compiler/rustc_attr/Cargo.toml 2022-02-18 20:52:49.347261785 -0800
++++ b/compiler/rustc_attr/Cargo.toml 2022-02-18 20:22:40.745863059 -0800
+@@ -5,6 +5,7 @@ version = "0.0.0"
+ edition = "2018"
+
+ [lib]
++test = false
+ doctest = false
+
+ [dependencies]
+diff -upr a/compiler/rustc_builtin_macros/Cargo.toml b/compiler/rustc_builtin_macros/Cargo.toml
+--- a/compiler/rustc_builtin_macros/Cargo.toml 2022-02-18 20:52:49.347261785 -0800
++++ b/compiler/rustc_builtin_macros/Cargo.toml 2022-02-18 20:22:40.749862923 -0800
+@@ -5,6 +5,7 @@ version = "0.0.0"
+ edition = "2018"
+
+ [lib]
++test = false
+ doctest = false
+
+ [dependencies]
+diff -upr a/compiler/rustc_data_structures/Cargo.toml b/compiler/rustc_data_structures/Cargo.toml
+--- a/compiler/rustc_data_structures/Cargo.toml 2022-02-18 20:52:49.347261785 -0800
++++ b/compiler/rustc_data_structures/Cargo.toml 2022-02-18 20:22:40.749862923 -0800
+@@ -5,6 +5,7 @@ version = "0.0.0"
+ edition = "2018"
+
+ [lib]
++test = false
+ doctest = false
+
+ [dependencies]
+diff -upr a/compiler/rustc_driver/Cargo.toml b/compiler/rustc_driver/Cargo.toml
+--- a/compiler/rustc_driver/Cargo.toml 2022-02-18 20:52:49.347261785 -0800
++++ b/compiler/rustc_driver/Cargo.toml 2022-02-18 20:22:40.749862923 -0800
+@@ -6,6 +6,8 @@ edition = "2018"
+
+ [lib]
+ crate-type = ["dylib"]
++test = false
++doctest = false
+
+ [dependencies]
+ libc = "0.2"
+diff -upr a/compiler/rustc_error_codes/Cargo.toml b/compiler/rustc_error_codes/Cargo.toml
+--- a/compiler/rustc_error_codes/Cargo.toml 2022-02-18 20:52:49.347261785 -0800
++++ b/compiler/rustc_error_codes/Cargo.toml 2022-02-18 20:22:40.749862923 -0800
+@@ -3,3 +3,7 @@ authors = ["The Rust Project Developers"
+ name = "rustc_error_codes"
+ version = "0.0.0"
+ edition = "2018"
++
++[lib]
++test = false
++doctest = false
+diff -upr a/compiler/rustc_errors/Cargo.toml b/compiler/rustc_errors/Cargo.toml
+--- a/compiler/rustc_errors/Cargo.toml 2022-02-18 20:52:49.347261785 -0800
++++ b/compiler/rustc_errors/Cargo.toml 2022-02-18 20:22:40.749862923 -0800
+@@ -5,6 +5,7 @@ version = "0.0.0"
+ edition = "2018"
+
+ [lib]
++test = false
+ doctest = false
+
+ [dependencies]
+diff -upr a/compiler/rustc_expand/Cargo.toml b/compiler/rustc_expand/Cargo.toml
+--- a/compiler/rustc_expand/Cargo.toml 2022-02-18 20:52:49.347261785 -0800
++++ b/compiler/rustc_expand/Cargo.toml 2022-02-18 20:22:40.749862923 -0800
+@@ -6,6 +6,7 @@ edition = "2018"
+ build = false
+
+ [lib]
++test = false
+ doctest = false
+
+ [dependencies]
+diff -upr a/compiler/rustc_feature/Cargo.toml b/compiler/rustc_feature/Cargo.toml
+--- a/compiler/rustc_feature/Cargo.toml 2022-02-18 20:52:49.347261785 -0800
++++ b/compiler/rustc_feature/Cargo.toml 2022-02-18 20:22:40.749862923 -0800
+@@ -5,6 +5,7 @@ version = "0.0.0"
+ edition = "2018"
+
+ [lib]
++test = false
+ doctest = false
+
+ [dependencies]
+diff -upr a/compiler/rustc_fs_util/Cargo.toml b/compiler/rustc_fs_util/Cargo.toml
+--- a/compiler/rustc_fs_util/Cargo.toml 2022-02-18 20:52:49.351261644 -0800
++++ b/compiler/rustc_fs_util/Cargo.toml 2022-02-18 20:22:40.749862923 -0800
+@@ -3,3 +3,7 @@ authors = ["The Rust Project Developers"
+ name = "rustc_fs_util"
+ version = "0.0.0"
+ edition = "2018"
++
++[lib]
++test = false
++doctest = false
+diff -upr a/compiler/rustc_graphviz/Cargo.toml b/compiler/rustc_graphviz/Cargo.toml
+--- a/compiler/rustc_graphviz/Cargo.toml 2022-02-18 20:52:49.351261644 -0800
++++ b/compiler/rustc_graphviz/Cargo.toml 2022-02-18 20:22:40.749862923 -0800
+@@ -3,3 +3,7 @@ authors = ["The Rust Project Developers"
+ name = "rustc_graphviz"
+ version = "0.0.0"
+ edition = "2018"
++
++[lib]
++test = false
++doctest = false
+diff -upr a/compiler/rustc_hir/Cargo.toml b/compiler/rustc_hir/Cargo.toml
+--- a/compiler/rustc_hir/Cargo.toml 2022-02-18 20:52:49.351261644 -0800
++++ b/compiler/rustc_hir/Cargo.toml 2022-02-18 20:22:40.749862923 -0800
+@@ -5,6 +5,7 @@ version = "0.0.0"
+ edition = "2018"
+
+ [lib]
++test = false
+ doctest = false
+
+ [dependencies]
+diff -upr a/compiler/rustc_hir_pretty/Cargo.toml b/compiler/rustc_hir_pretty/Cargo.toml
+--- a/compiler/rustc_hir_pretty/Cargo.toml 2022-02-18 20:52:49.351261644 -0800
++++ b/compiler/rustc_hir_pretty/Cargo.toml 2022-02-18 20:22:40.749862923 -0800
+@@ -5,6 +5,7 @@ version = "0.0.0"
+ edition = "2018"
+
+ [lib]
++test = false
+ doctest = false
+
+ [dependencies]
+diff -upr a/compiler/rustc_incremental/Cargo.toml b/compiler/rustc_incremental/Cargo.toml
+--- a/compiler/rustc_incremental/Cargo.toml 2022-02-18 20:52:49.351261644 -0800
++++ b/compiler/rustc_incremental/Cargo.toml 2022-02-18 20:22:40.749862923 -0800
+@@ -5,6 +5,7 @@ version = "0.0.0"
+ edition = "2018"
+
+ [lib]
++test = false
+ doctest = false
+
+ [dependencies]
+diff -upr a/compiler/rustc_index/Cargo.toml b/compiler/rustc_index/Cargo.toml
+--- a/compiler/rustc_index/Cargo.toml 2022-02-18 20:52:49.351261644 -0800
++++ b/compiler/rustc_index/Cargo.toml 2022-02-18 20:22:40.749862923 -0800
+@@ -5,6 +5,7 @@ version = "0.0.0"
+ edition = "2018"
+
+ [lib]
++test = false
+ doctest = false
+
+ [dependencies]
+diff -upr a/compiler/rustc_infer/Cargo.toml b/compiler/rustc_infer/Cargo.toml
+--- a/compiler/rustc_infer/Cargo.toml 2022-02-18 20:52:49.351261644 -0800
++++ b/compiler/rustc_infer/Cargo.toml 2022-02-18 20:22:40.749862923 -0800
+@@ -5,6 +5,7 @@ version = "0.0.0"
+ edition = "2018"
+
+ [lib]
++test = false
+ doctest = false
+
+ [dependencies]
+diff -upr a/compiler/rustc_interface/Cargo.toml b/compiler/rustc_interface/Cargo.toml
+--- a/compiler/rustc_interface/Cargo.toml 2022-02-18 20:52:49.351261644 -0800
++++ b/compiler/rustc_interface/Cargo.toml 2022-02-18 20:22:40.753862788 -0800
+@@ -5,6 +5,7 @@ version = "0.0.0"
+ edition = "2018"
+
+ [lib]
++test = false
+ doctest = false
+
+ [dependencies]
+diff -upr a/compiler/rustc_lexer/Cargo.toml b/compiler/rustc_lexer/Cargo.toml
+--- a/compiler/rustc_lexer/Cargo.toml 2022-02-18 20:52:49.351261644 -0800
++++ b/compiler/rustc_lexer/Cargo.toml 2022-02-18 20:22:40.753862788 -0800
+@@ -13,6 +13,7 @@ Rust lexer used by rustc. No stability g
+ # Note: do not remove this blank `[lib]` section.
+ # This will be used when publishing this crate as `rustc-ap-rustc_lexer`.
+ [lib]
++test = false
+ doctest = false
+
+ # Note that this crate purposefully does not depend on other rustc crates
+diff -upr a/compiler/rustc_lint/Cargo.toml b/compiler/rustc_lint/Cargo.toml
+--- a/compiler/rustc_lint/Cargo.toml 2022-02-18 20:52:49.351261644 -0800
++++ b/compiler/rustc_lint/Cargo.toml 2022-02-18 20:22:40.753862788 -0800
+@@ -4,6 +4,10 @@ name = "rustc_lint"
+ version = "0.0.0"
+ edition = "2018"
+
++[lib]
++test = false
++doctest = false
++
+ [dependencies]
+ tracing = "0.1"
+ unicode-security = "0.0.5"
+diff -upr a/compiler/rustc_lint_defs/Cargo.toml b/compiler/rustc_lint_defs/Cargo.toml
+--- a/compiler/rustc_lint_defs/Cargo.toml 2022-02-18 20:52:49.351261644 -0800
++++ b/compiler/rustc_lint_defs/Cargo.toml 2022-02-18 20:22:40.753862788 -0800
+@@ -4,6 +4,10 @@ name = "rustc_lint_defs"
+ version = "0.0.0"
+ edition = "2018"
+
++[lib]
++test = false
++doctest = false
++
+ [dependencies]
+ log = { package = "tracing", version = "0.1" }
+ rustc_ast = { path = "../rustc_ast" }
+diff -upr a/compiler/rustc_llvm/Cargo.toml b/compiler/rustc_llvm/Cargo.toml
+--- a/compiler/rustc_llvm/Cargo.toml 2022-02-18 20:52:49.351261644 -0800
++++ b/compiler/rustc_llvm/Cargo.toml 2022-02-18 20:22:40.753862788 -0800
+@@ -4,6 +4,10 @@ name = "rustc_llvm"
+ version = "0.0.0"
+ edition = "2018"
+
++[lib]
++test = false
++doctest = false
++
+ [features]
+ static-libstdcpp = []
+ emscripten = []
+diff -upr a/compiler/rustc_macros/Cargo.toml b/compiler/rustc_macros/Cargo.toml
+--- a/compiler/rustc_macros/Cargo.toml 2022-02-18 20:52:49.355261504 -0800
++++ b/compiler/rustc_macros/Cargo.toml 2022-02-18 20:22:40.753862788 -0800
+@@ -6,6 +6,8 @@ edition = "2018"
+
+ [lib]
+ proc-macro = true
++test = false
++doctest = false
+
+ [dependencies]
+ synstructure = "0.12.1"
+diff -upr a/compiler/rustc_metadata/Cargo.toml b/compiler/rustc_metadata/Cargo.toml
+--- a/compiler/rustc_metadata/Cargo.toml 2022-02-18 20:52:49.355261504 -0800
++++ b/compiler/rustc_metadata/Cargo.toml 2022-02-18 20:22:40.753862788 -0800
+@@ -5,6 +5,7 @@ version = "0.0.0"
+ edition = "2018"
+
+ [lib]
++test = false
+ doctest = false
+
+ [dependencies]
+diff -upr a/compiler/rustc_middle/Cargo.toml b/compiler/rustc_middle/Cargo.toml
+--- a/compiler/rustc_middle/Cargo.toml 2022-02-18 20:52:49.355261504 -0800
++++ b/compiler/rustc_middle/Cargo.toml 2022-02-18 20:22:40.753862788 -0800
+@@ -5,6 +5,7 @@ version = "0.0.0"
+ edition = "2018"
+
+ [lib]
++test = false
+ doctest = false
+
+ [dependencies]
+diff -upr a/compiler/rustc_mir/Cargo.toml b/compiler/rustc_mir/Cargo.toml
+--- a/compiler/rustc_mir/Cargo.toml 2022-02-18 20:52:49.355261504 -0800
++++ b/compiler/rustc_mir/Cargo.toml 2022-02-18 20:22:40.753862788 -0800
+@@ -5,6 +5,7 @@ version = "0.0.0"
+ edition = "2018"
+
+ [lib]
++test = false
+ doctest = false
+
+ [dependencies]
+diff -upr a/compiler/rustc_mir/src/transform/coverage/test_macros/Cargo.toml b/compiler/rustc_mir/src/transform/coverage/test_macros/Cargo.toml
+--- a/compiler/rustc_mir/src/transform/coverage/test_macros/Cargo.toml 2022-02-18 20:52:49.355261504 -0800
++++ b/compiler/rustc_mir/src/transform/coverage/test_macros/Cargo.toml 2022-02-18 20:22:40.765862380 -0800
+@@ -6,4 +6,5 @@ edition = "2018"
+
+ [lib]
+ proc-macro = true
++test = false
+ doctest = false
+diff -upr a/compiler/rustc_mir_build/Cargo.toml b/compiler/rustc_mir_build/Cargo.toml
+--- a/compiler/rustc_mir_build/Cargo.toml 2022-02-18 20:52:49.355261504 -0800
++++ b/compiler/rustc_mir_build/Cargo.toml 2022-02-18 20:22:40.765862380 -0800
+@@ -5,6 +5,7 @@ version = "0.0.0"
+ edition = "2018"
+
+ [lib]
++test = false
+ doctest = false
+
+ [dependencies]
+diff -upr a/compiler/rustc_parse/Cargo.toml b/compiler/rustc_parse/Cargo.toml
+--- a/compiler/rustc_parse/Cargo.toml 2022-02-18 20:52:49.355261504 -0800
++++ b/compiler/rustc_parse/Cargo.toml 2022-02-18 20:22:40.765862380 -0800
+@@ -5,6 +5,7 @@ version = "0.0.0"
+ edition = "2018"
+
+ [lib]
++test = false
+ doctest = false
+
+ [dependencies]
+diff -upr a/compiler/rustc_parse_format/Cargo.toml b/compiler/rustc_parse_format/Cargo.toml
+--- a/compiler/rustc_parse_format/Cargo.toml 2022-02-18 20:52:49.355261504 -0800
++++ b/compiler/rustc_parse_format/Cargo.toml 2022-02-18 20:22:40.765862380 -0800
+@@ -4,6 +4,10 @@ name = "rustc_parse_format"
+ version = "0.0.0"
+ edition = "2018"
+
++[lib]
++test = false
++doctest = false
++
+ [dependencies]
+ rustc_span = { path = "../rustc_span" }
+ rustc_lexer = { path = "../rustc_lexer" }
+diff -upr a/compiler/rustc_passes/Cargo.toml b/compiler/rustc_passes/Cargo.toml
+--- a/compiler/rustc_passes/Cargo.toml 2022-02-18 20:52:49.355261504 -0800
++++ b/compiler/rustc_passes/Cargo.toml 2022-02-18 20:22:40.765862380 -0800
+@@ -4,6 +4,10 @@ name = "rustc_passes"
+ version = "0.0.0"
+ edition = "2018"
+
++[lib]
++test = false
++doctest = false
++
+ [dependencies]
+ tracing = "0.1"
+ rustc_middle = { path = "../rustc_middle" }
+diff -upr a/compiler/rustc_plugin_impl/Cargo.toml b/compiler/rustc_plugin_impl/Cargo.toml
+--- a/compiler/rustc_plugin_impl/Cargo.toml 2022-02-18 20:52:49.355261504 -0800
++++ b/compiler/rustc_plugin_impl/Cargo.toml 2022-02-18 20:22:40.765862380 -0800
+@@ -6,6 +6,7 @@ build = false
+ edition = "2018"
+
+ [lib]
++test = false
+ doctest = false
+
+ [dependencies]
+diff -upr a/compiler/rustc_privacy/Cargo.toml b/compiler/rustc_privacy/Cargo.toml
+--- a/compiler/rustc_privacy/Cargo.toml 2022-02-18 20:52:49.355261504 -0800
++++ b/compiler/rustc_privacy/Cargo.toml 2022-02-18 20:22:40.765862380 -0800
+@@ -4,6 +4,10 @@ name = "rustc_privacy"
+ version = "0.0.0"
+ edition = "2018"
+
++[lib]
++test = false
++doctest = false
++
+ [dependencies]
+ rustc_middle = { path = "../rustc_middle" }
+ rustc_attr = { path = "../rustc_attr" }
+diff -upr a/compiler/rustc_query_impl/Cargo.toml b/compiler/rustc_query_impl/Cargo.toml
+--- a/compiler/rustc_query_impl/Cargo.toml 2022-02-18 20:52:49.355261504 -0800
++++ b/compiler/rustc_query_impl/Cargo.toml 2022-02-18 20:22:40.765862380 -0800
+@@ -5,6 +5,7 @@ version = "0.0.0"
+ edition = "2018"
+
+ [lib]
++test = false
+ doctest = false
+
+ [dependencies]
+diff -upr a/compiler/rustc_query_system/Cargo.toml b/compiler/rustc_query_system/Cargo.toml
+--- a/compiler/rustc_query_system/Cargo.toml 2022-02-18 20:52:49.359261363 -0800
++++ b/compiler/rustc_query_system/Cargo.toml 2022-02-18 20:22:40.765862380 -0800
+@@ -5,6 +5,7 @@ version = "0.0.0"
+ edition = "2018"
+
+ [lib]
++test = false
+ doctest = false
+
+ [dependencies]
+diff -upr a/compiler/rustc_save_analysis/Cargo.toml b/compiler/rustc_save_analysis/Cargo.toml
+--- a/compiler/rustc_save_analysis/Cargo.toml 2022-02-18 20:52:49.359261363 -0800
++++ b/compiler/rustc_save_analysis/Cargo.toml 2022-02-18 20:22:40.765862380 -0800
+@@ -4,6 +4,10 @@ name = "rustc_save_analysis"
+ version = "0.0.0"
+ edition = "2018"
+
++[lib]
++test = false
++doctest = false
++
+ [dependencies]
+ tracing = "0.1"
+ rustc_middle = { path = "../rustc_middle" }
+diff -upr a/compiler/rustc_session/Cargo.toml b/compiler/rustc_session/Cargo.toml
+--- a/compiler/rustc_session/Cargo.toml 2022-02-18 20:52:49.359261363 -0800
++++ b/compiler/rustc_session/Cargo.toml 2022-02-18 20:22:40.765862380 -0800
+@@ -4,6 +4,10 @@ name = "rustc_session"
+ version = "0.0.0"
+ edition = "2018"
+
++[lib]
++test = false
++doctest = false
++
+ [dependencies]
+ bitflags = "1.2.1"
+ getopts = "0.2"
+diff -upr a/compiler/rustc_span/Cargo.toml b/compiler/rustc_span/Cargo.toml
+--- a/compiler/rustc_span/Cargo.toml 2022-02-18 20:52:49.359261363 -0800
++++ b/compiler/rustc_span/Cargo.toml 2022-02-18 20:22:40.765862380 -0800
+@@ -5,6 +5,7 @@ version = "0.0.0"
+ edition = "2018"
+
+ [lib]
++test = false
+ doctest = false
+
+ [dependencies]
+diff -upr a/compiler/rustc_symbol_mangling/Cargo.toml b/compiler/rustc_symbol_mangling/Cargo.toml
+--- a/compiler/rustc_symbol_mangling/Cargo.toml 2022-02-18 20:52:49.359261363 -0800
++++ b/compiler/rustc_symbol_mangling/Cargo.toml 2022-02-18 20:22:40.765862380 -0800
+@@ -5,6 +5,7 @@ version = "0.0.0"
+ edition = "2018"
+
+ [lib]
++test = false
+ doctest = false
+
+ [dependencies]
+diff -upr a/compiler/rustc_target/Cargo.toml b/compiler/rustc_target/Cargo.toml
+--- a/compiler/rustc_target/Cargo.toml 2022-02-18 20:52:49.359261363 -0800
++++ b/compiler/rustc_target/Cargo.toml 2022-02-18 20:22:40.769862244 -0800
+@@ -4,6 +4,10 @@ name = "rustc_target"
+ version = "0.0.0"
+ edition = "2018"
+
++[lib]
++test = false
++doctest = false
++
+ [dependencies]
+ bitflags = "1.2.1"
+ tracing = "0.1"
+diff -upr a/compiler/rustc_traits/Cargo.toml b/compiler/rustc_traits/Cargo.toml
+--- a/compiler/rustc_traits/Cargo.toml 2022-02-18 20:52:49.359261363 -0800
++++ b/compiler/rustc_traits/Cargo.toml 2022-02-18 20:22:40.769862244 -0800
+@@ -4,6 +4,10 @@ name = "rustc_traits"
+ version = "0.0.0"
+ edition = "2018"
+
++[lib]
++test = false
++doctest = false
++
+ [dependencies]
+ tracing = "0.1"
+ rustc_attr = { path = "../rustc_attr" }
+diff -upr a/compiler/rustc_trait_selection/Cargo.toml b/compiler/rustc_trait_selection/Cargo.toml
+--- a/compiler/rustc_trait_selection/Cargo.toml 2022-02-18 20:52:49.359261363 -0800
++++ b/compiler/rustc_trait_selection/Cargo.toml 2022-02-18 20:22:40.769862244 -0800
+@@ -5,6 +5,7 @@ version = "0.0.0"
+ edition = "2018"
+
+ [lib]
++test = false
+ doctest = false
+
+ [dependencies]
+diff -upr a/compiler/rustc_type_ir/Cargo.toml b/compiler/rustc_type_ir/Cargo.toml
+--- a/compiler/rustc_type_ir/Cargo.toml 2022-02-18 20:52:49.359261363 -0800
++++ b/compiler/rustc_type_ir/Cargo.toml 2022-02-18 20:22:40.769862244 -0800
+@@ -5,6 +5,7 @@ authors = ["The Rust Project Developers"
+ edition = "2018"
+
+ [lib]
++test = false
+ doctest = false
+
+ [dependencies]
+diff -upr a/compiler/rustc_ty_utils/Cargo.toml b/compiler/rustc_ty_utils/Cargo.toml
+--- a/compiler/rustc_ty_utils/Cargo.toml 2022-02-18 20:52:49.359261363 -0800
++++ b/compiler/rustc_ty_utils/Cargo.toml 2022-02-18 20:22:40.769862244 -0800
+@@ -4,6 +4,10 @@ name = "rustc_ty_utils"
+ version = "0.0.0"
+ edition = "2018"
+
++[lib]
++test = false
++doctest = false
++
+ [dependencies]
+ tracing = "0.1"
+ rustc_middle = { path = "../rustc_middle" }
+diff -upr a/library/alloc/Cargo.toml b/library/alloc/Cargo.toml
+--- a/library/alloc/Cargo.toml 2022-02-18 20:52:49.359261363 -0800
++++ b/library/alloc/Cargo.toml 2022-02-18 20:22:40.769862244 -0800
+@@ -24,7 +24,7 @@ path = "tests/lib.rs"
+ [[bench]]
+ name = "collectionsbenches"
+ path = "benches/lib.rs"
+-test = true
++test = false
+
+ [[bench]]
+ name = "vec_deque_append_bench"
+Only in a/: patches
+Only in a/: .pc
+diff -upr a/src/test/run-make/issue-36710/Makefile b/src/test/run-make/issue-36710/Makefile
+--- a/src/test/run-make/issue-36710/Makefile 2022-02-18 20:52:49.359261363 -0800
++++ b/src/test/run-make/issue-36710/Makefile 2022-02-18 20:22:40.769862244 -0800
+@@ -7,6 +7,8 @@
+ # ignore-nvptx64-nvidia-cuda FIXME: can't find crate for `std`
+ # ignore-musl FIXME: this makefile needs teaching how to use a musl toolchain
+ # (see dist-i586-gnu-i586-i686-musl Dockerfile)
++# ignore-windows-msvc
++# ignore-stage1
+
+ include ../../run-make-fulldeps/tools.mk
+
+diff -upr a/src/test/ui/abi/stack-probes-lto.rs b/src/test/ui/abi/stack-probes-lto.rs
+--- a/src/test/ui/abi/stack-probes-lto.rs 2022-02-18 20:52:49.291263757 -0800
++++ b/src/test/ui/abi/stack-probes-lto.rs 2022-02-18 20:58:59.706316652 -0800
+@@ -14,5 +14,6 @@
+ // ignore-pretty
+ // compile-flags: -C lto
+ // no-prefer-dynamic
++// ignore-stage1
+
+ include!("stack-probes.rs");
+diff -upr a/src/test/ui/abi/stack-probes.rs b/src/test/ui/abi/stack-probes.rs
+--- a/src/test/ui/abi/stack-probes.rs 2022-02-18 20:52:49.291263757 -0800
++++ b/src/test/ui/abi/stack-probes.rs 2022-02-18 20:59:12.777862379 -0800
+@@ -10,6 +10,7 @@
+ // ignore-wasm
+ // ignore-emscripten no processes
+ // ignore-sgx no processes
++// ignore-stage1
+
+ use std::env;
+ use std::mem::MaybeUninit;
+diff -upr a/src/test/ui/macros/restricted-shadowing-legacy.rs b/src/test/ui/macros/restricted-shadowing-legacy.rs
+--- a/src/test/ui/macros/restricted-shadowing-legacy.rs 2022-02-18 20:52:49.363261222 -0800
++++ b/src/test/ui/macros/restricted-shadowing-legacy.rs 2022-02-18 20:22:40.769862244 -0800
+@@ -74,6 +74,7 @@
+ // 62 | Unordered | Unordered | = | +? |
+ // 63 | Unordered | Unordered | > | +? |
+ // 64 | Unordered | Unordered | Unordered | + |
++// ignore-stage1
+
+ #![feature(decl_macro, rustc_attrs)]
+
+diff -upr a/src/test/ui/simd/simd-target-feature-mixup.rs b/src/test/ui/simd/simd-target-feature-mixup.rs
+--- a/src/test/ui/simd/simd-target-feature-mixup.rs 2022-02-18 20:52:49.363261222 -0800
++++ b/src/test/ui/simd/simd-target-feature-mixup.rs 2022-02-18 20:22:40.769862244 -0800
+@@ -1,4 +1,5 @@
+ // run-pass
++// ignore-stage1
+ #![allow(unused_variables)]
+ #![allow(stable_features)]
+ #![allow(overflowing_literals)]
+diff -upr a/src/test/ui-fulldeps/create-dir-all-bare.rs b/src/test/ui-fulldeps/create-dir-all-bare.rs
+--- a/src/test/ui-fulldeps/create-dir-all-bare.rs 2022-02-18 20:52:49.363261222 -0800
++++ b/src/test/ui-fulldeps/create-dir-all-bare.rs 2022-02-18 20:22:40.769862244 -0800
+@@ -1,4 +1,5 @@
+ // run-pass
++// ignore-stage1
+
+ use std::env;
+ use std::fs;
+diff -upr a/src/test/ui-fulldeps/deriving-encodable-decodable-box.rs b/src/test/ui-fulldeps/deriving-encodable-decodable-box.rs
+--- a/src/test/ui-fulldeps/deriving-encodable-decodable-box.rs 2022-02-18 20:52:49.363261222 -0800
++++ b/src/test/ui-fulldeps/deriving-encodable-decodable-box.rs 2022-02-18 20:22:40.769862244 -0800
+@@ -1,4 +1,5 @@
+ // run-pass
++// ignore-stage1
+
+ #![allow(unused_imports)]
+ #![feature(box_syntax)]
+diff -upr a/src/test/ui-fulldeps/deriving-encodable-decodable-cell-refcell.rs b/src/test/ui-fulldeps/deriving-encodable-decodable-cell-refcell.rs
+--- a/src/test/ui-fulldeps/deriving-encodable-decodable-cell-refcell.rs 2022-02-18 20:52:49.363261222 -0800
++++ b/src/test/ui-fulldeps/deriving-encodable-decodable-cell-refcell.rs 2022-02-18 20:22:40.769862244 -0800
+@@ -1,4 +1,5 @@
+ // run-pass
++// ignore-stage1
+
+ #![allow(unused_imports)]
+ // This briefly tests the capability of `Cell` and `RefCell` to implement the
+diff -upr a/src/test/ui-fulldeps/deriving-global.rs b/src/test/ui-fulldeps/deriving-global.rs
+--- a/src/test/ui-fulldeps/deriving-global.rs 2022-02-18 20:52:49.363261222 -0800
++++ b/src/test/ui-fulldeps/deriving-global.rs 2022-02-18 20:22:40.769862244 -0800
+@@ -1,4 +1,5 @@
+ // run-pass
++// ignore-stage1
+
+ #![feature(rustc_private)]
+
+diff -upr a/src/test/ui-fulldeps/deriving-hygiene.rs b/src/test/ui-fulldeps/deriving-hygiene.rs
+--- a/src/test/ui-fulldeps/deriving-hygiene.rs 2022-02-18 20:52:49.363261222 -0800
++++ b/src/test/ui-fulldeps/deriving-hygiene.rs 2022-02-18 20:22:40.769862244 -0800
+@@ -1,4 +1,5 @@
+ // run-pass
++// ignore-stage1
+
+ #![allow(non_upper_case_globals)]
+ #![feature(rustc_private)]
+diff -upr a/src/test/ui-fulldeps/dropck_tarena_sound_drop.rs b/src/test/ui-fulldeps/dropck_tarena_sound_drop.rs
+--- a/src/test/ui-fulldeps/dropck_tarena_sound_drop.rs 2022-02-18 20:52:49.363261222 -0800
++++ b/src/test/ui-fulldeps/dropck_tarena_sound_drop.rs 2022-02-18 20:22:40.769862244 -0800
+@@ -1,4 +1,5 @@
+ // run-pass
++// ignore-stage1
+
+ #![allow(unknown_lints)]
+ // Check that an arena (TypedArena) can carry elements whose drop
+diff -upr a/src/test/ui-fulldeps/empty-struct-braces-derive.rs b/src/test/ui-fulldeps/empty-struct-braces-derive.rs
+--- a/src/test/ui-fulldeps/empty-struct-braces-derive.rs 2022-02-18 20:52:49.363261222 -0800
++++ b/src/test/ui-fulldeps/empty-struct-braces-derive.rs 2022-02-18 20:22:40.769862244 -0800
+@@ -1,4 +1,5 @@
+ // run-pass
++// ignore-stage1
+ // `#[derive(Trait)]` works for empty structs/variants with braces or parens.
+
+ #![feature(rustc_private)]
+diff -upr a/src/test/ui-fulldeps/extern-mod-syntax.rs b/src/test/ui-fulldeps/extern-mod-syntax.rs
+--- a/src/test/ui-fulldeps/extern-mod-syntax.rs 2022-02-18 20:52:49.363261222 -0800
++++ b/src/test/ui-fulldeps/extern-mod-syntax.rs 2022-02-18 20:22:40.769862244 -0800
+@@ -1,4 +1,5 @@
+ // run-pass
++// ignore-stage1
+
+ #![allow(unused_imports)]
+ #![feature(rustc_private)]
+diff -upr a/src/test/ui-fulldeps/issue-11881.rs b/src/test/ui-fulldeps/issue-11881.rs
+--- a/src/test/ui-fulldeps/issue-11881.rs 2022-02-18 20:52:49.363261222 -0800
++++ b/src/test/ui-fulldeps/issue-11881.rs 2022-02-18 20:22:40.769862244 -0800
+@@ -1,4 +1,5 @@
+ // run-pass
++// ignore-stage1
+
+ #![allow(unused_must_use)]
+ #![allow(dead_code)]
+diff -upr a/src/test/ui-fulldeps/issue-13560.rs b/src/test/ui-fulldeps/issue-13560.rs
+--- a/src/test/ui-fulldeps/issue-13560.rs 2022-02-18 20:52:49.363261222 -0800
++++ b/src/test/ui-fulldeps/issue-13560.rs 2022-02-18 20:22:40.769862244 -0800
+@@ -1,4 +1,5 @@
+ // run-pass
++// ignore-stage1
+ // aux-build:issue-13560-1.rs
+ // aux-build:issue-13560-2.rs
+ // aux-build:issue-13560-3.rs
+diff -upr a/src/test/ui-fulldeps/issue-14021.rs b/src/test/ui-fulldeps/issue-14021.rs
+--- a/src/test/ui-fulldeps/issue-14021.rs 2022-02-18 20:52:49.363261222 -0800
++++ b/src/test/ui-fulldeps/issue-14021.rs 2022-02-18 20:22:40.769862244 -0800
+@@ -1,4 +1,5 @@
+ // run-pass
++// ignore-stage1
+
+ #![allow(unused_mut)]
+ #![allow(unused_imports)]
+diff -upr a/src/test/ui-fulldeps/issue-15149.rs b/src/test/ui-fulldeps/issue-15149.rs
+--- a/src/test/ui-fulldeps/issue-15149.rs 2022-02-18 20:52:49.363261222 -0800
++++ b/src/test/ui-fulldeps/issue-15149.rs 2022-02-18 20:22:40.773862108 -0800
+@@ -1,4 +1,5 @@
+ // run-pass
++// ignore-stage1
+
+ #![allow(unused_variables)]
+ // no-prefer-dynamic
+diff -upr a/src/test/ui-fulldeps/issue-15924.rs b/src/test/ui-fulldeps/issue-15924.rs
+--- a/src/test/ui-fulldeps/issue-15924.rs 2022-02-18 20:52:49.367261081 -0800
++++ b/src/test/ui-fulldeps/issue-15924.rs 2022-02-18 20:22:40.773862108 -0800
+@@ -1,4 +1,5 @@
+ // run-pass
++// ignore-stage1
+
+ #![allow(unused_imports)]
+ #![allow(unused_must_use)]
+diff -upr a/src/test/ui-fulldeps/issue-16822.rs b/src/test/ui-fulldeps/issue-16822.rs
+--- a/src/test/ui-fulldeps/issue-16822.rs 2022-02-18 20:52:49.367261081 -0800
++++ b/src/test/ui-fulldeps/issue-16822.rs 2022-02-18 20:22:40.773862108 -0800
+@@ -1,4 +1,5 @@
+ // run-pass
++// ignore-stage1
+ // aux-build:issue-16822.rs
+
+ extern crate issue_16822 as lib;
+diff -upr a/src/test/ui-fulldeps/issue-18502.rs b/src/test/ui-fulldeps/issue-18502.rs
+--- a/src/test/ui-fulldeps/issue-18502.rs 2022-02-18 20:52:49.367261081 -0800
++++ b/src/test/ui-fulldeps/issue-18502.rs 2022-02-18 20:22:40.773862108 -0800
+@@ -1,4 +1,5 @@
+ // run-pass
++// ignore-stage1
+ // aux-build:issue-18502.rs
+
+ extern crate issue_18502 as fmt;
+diff -upr a/src/test/ui-fulldeps/issue-24106.rs b/src/test/ui-fulldeps/issue-24106.rs
+--- a/src/test/ui-fulldeps/issue-24106.rs 2022-02-18 20:52:49.367261081 -0800
++++ b/src/test/ui-fulldeps/issue-24106.rs 2022-02-18 20:22:40.773862108 -0800
+@@ -1,4 +1,5 @@
+ // run-pass
++// ignore-stage1
+ // aux-build:issue-24106.rs
+
+ extern crate issue_24106;
+diff -upr a/src/test/ui-fulldeps/issue-24972.rs b/src/test/ui-fulldeps/issue-24972.rs
+--- a/src/test/ui-fulldeps/issue-24972.rs 2022-02-18 20:52:49.367261081 -0800
++++ b/src/test/ui-fulldeps/issue-24972.rs 2022-02-18 20:22:40.773862108 -0800
+@@ -1,4 +1,5 @@
+ // run-pass
++// ignore-stage1
+
+ #![allow(dead_code)]
+ #![feature(rustc_private)]
+diff -upr a/src/test/ui-fulldeps/issue-2804.rs b/src/test/ui-fulldeps/issue-2804.rs
+--- a/src/test/ui-fulldeps/issue-2804.rs 2022-02-18 20:52:49.367261081 -0800
++++ b/src/test/ui-fulldeps/issue-2804.rs 2022-02-18 20:22:40.773862108 -0800
+@@ -1,4 +1,5 @@
+ // run-pass
++// ignore-stage1
+
+ #![allow(non_camel_case_types)]
+ #![allow(dead_code)]
+diff -upr a/src/test/ui-fulldeps/issue-4016.rs b/src/test/ui-fulldeps/issue-4016.rs
+--- a/src/test/ui-fulldeps/issue-4016.rs 2022-02-18 20:52:49.367261081 -0800
++++ b/src/test/ui-fulldeps/issue-4016.rs 2022-02-18 20:22:40.773862108 -0800
+@@ -1,4 +1,5 @@
+ // run-pass
++// ignore-stage1
+
+ #![allow(dead_code)]
+ #![feature(rustc_private)]
+diff -upr a/src/test/ui-fulldeps/issue-4036.rs b/src/test/ui-fulldeps/issue-4036.rs
+--- a/src/test/ui-fulldeps/issue-4036.rs 2022-02-18 20:52:49.367261081 -0800
++++ b/src/test/ui-fulldeps/issue-4036.rs 2022-02-18 20:22:40.773862108 -0800
+@@ -1,4 +1,5 @@
+ // run-pass
++// ignore-stage1
+ // Issue #4036: Test for an issue that arose around fixing up type inference
+ // byproducts in vtable records.
+
+diff -upr a/src/test/ui-fulldeps/myriad-closures.rs b/src/test/ui-fulldeps/myriad-closures.rs
+--- a/src/test/ui-fulldeps/myriad-closures.rs 2022-02-18 20:52:49.367261081 -0800
++++ b/src/test/ui-fulldeps/myriad-closures.rs 2022-02-18 20:22:40.773862108 -0800
+@@ -1,4 +1,5 @@
+ // run-pass
++// ignore-stage1
+ // This test case tests whether we can handle code bases that contain a high
+ // number of closures, something that needs special handling in the MingGW
+ // toolchain.
+diff -upr a/src/test/ui-fulldeps/pprust-expr-roundtrip.rs b/src/test/ui-fulldeps/pprust-expr-roundtrip.rs
+--- a/src/test/ui-fulldeps/pprust-expr-roundtrip.rs 2022-02-18 20:52:49.367261081 -0800
++++ b/src/test/ui-fulldeps/pprust-expr-roundtrip.rs 2022-02-18 20:22:40.773862108 -0800
+@@ -1,4 +1,5 @@
+ // run-pass
++// ignore-stage1
+ // ignore-cross-compile
+
+ // The general idea of this test is to enumerate all "interesting" expressions and check that
+diff -upr a/src/test/ui-fulldeps/regions-mock-tcx.rs b/src/test/ui-fulldeps/regions-mock-tcx.rs
+--- a/src/test/ui-fulldeps/regions-mock-tcx.rs 2022-02-18 20:52:49.367261081 -0800
++++ b/src/test/ui-fulldeps/regions-mock-tcx.rs 2022-02-18 20:22:40.773862108 -0800
+@@ -1,4 +1,5 @@
+ // run-pass
++// ignore-stage1
+
+ #![allow(dead_code)]
+ #![allow(unused_imports)]
+diff -upr a/src/test/ui-fulldeps/rename-directory.rs b/src/test/ui-fulldeps/rename-directory.rs
+--- a/src/test/ui-fulldeps/rename-directory.rs 2022-02-18 20:52:49.367261081 -0800
++++ b/src/test/ui-fulldeps/rename-directory.rs 2022-02-18 20:22:40.773862108 -0800
+@@ -1,4 +1,5 @@
+ // run-pass
++// ignore-stage1
+
+ #![allow(unused_must_use)]
+ #![allow(unused_imports)]
+diff -upr a/src/test/ui-fulldeps/rustc_encodable_hygiene.rs b/src/test/ui-fulldeps/rustc_encodable_hygiene.rs
+--- a/src/test/ui-fulldeps/rustc_encodable_hygiene.rs 2022-02-18 20:52:49.367261081 -0800
++++ b/src/test/ui-fulldeps/rustc_encodable_hygiene.rs 2022-02-18 20:22:40.773862108 -0800
+@@ -1,4 +1,5 @@
+ // run-pass
++// ignore-stage1
+
+ #![feature(rustc_private)]
+
+diff -upr a/src/test/ui-fulldeps/stdio-from.rs b/src/test/ui-fulldeps/stdio-from.rs
+--- a/src/test/ui-fulldeps/stdio-from.rs 2022-02-18 20:52:49.367261081 -0800
++++ b/src/test/ui-fulldeps/stdio-from.rs 2022-02-18 20:22:40.773862108 -0800
+@@ -1,5 +1,6 @@
+ // run-pass
+ // ignore-cross-compile
++// ignore-stage1
+
+ use std::env;
+ use std::fs::File;
+diff -upr a/src/test/ui-fulldeps/switch-stdout.rs b/src/test/ui-fulldeps/switch-stdout.rs
+--- a/src/test/ui-fulldeps/switch-stdout.rs 2022-02-18 20:52:49.367261081 -0800
++++ b/src/test/ui-fulldeps/switch-stdout.rs 2022-02-18 20:22:40.773862108 -0800
+@@ -1,4 +1,5 @@
+ // run-pass
++// ignore-stage1
+
+ use std::env;
+ use std::fs::File;
diff --git a/meta/recipes-devtools/rust/rust-testsuite_1.54.0.bb b/meta/recipes-devtools/rust/rust-testsuite_1.54.0.bb
new file mode 100644
index 0000000000..ad758b71f4
--- /dev/null
+++ b/meta/recipes-devtools/rust/rust-testsuite_1.54.0.bb
@@ -0,0 +1,3 @@
+require rust-testsuite.inc
+require rust-source-${PV}.inc
+require rust-snapshot-${PV}.inc
--
2.31.1


Re: [oe] INCOMPATIBLE_LICENSES and WHITELIST_<license> usage

Peter Kjellerstedt
 

-----Original Message-----
From: Richard Purdie <richard.purdie@...>
Sent: den 19 februari 2022 01:45
To: Peter Kjellerstedt <peter.kjellerstedt@...>; Saul Wold
<Saul.Wold@...>; openembedded-
architecture@...; OE-core <openembedded-
core@...>; OpenEmbedded Devel List <openembedded-
devel@...>
Subject: Re: [oe] INCOMPATIBLE_LICENSES and WHITELIST_<license> usage

On Fri, 2022-02-18 at 23:34 +0000, Peter Kjellerstedt wrote:
Warning: wall of text ahead. Sorry about that, but I believe it is
important that we get this right if we are redesigning it.

TL;DR: see the end for the new suggested licensing variables.

-----Original Message-----
From: openembedded-devel@... <openembedded-
devel@...> On Behalf Of Richard Purdie
Sent: den 18 februari 2022 15:14
To: Saul Wold <Saul.Wold@...>; openembedded-
architecture@...; OE-core <openembedded-
core@...>; OpenEmbedded Devel List <openembedded-
devel@...>
Subject: Re: [oe] INCOMPATIBLE_LICENSES and WHITELIST_<license> usage

On Thu, 2022-02-17 at 15:01 -0800, Saul Wold wrote:
I am working on a proposal to re-write how INCOMPATIBLE_LICENSES is
used and processed to possibly include a COMPATIBLE_LICENSES variable
as well, see PeterK's email [0]

I am trying to determine the usage of WHITELIST_<license> which
would be used to override a license that might be listed in
INCOMPATIBLE_LICENSES variable.

Randy and I have done a quick and dirty survey of a 100 or so layers
(thanks Randy) and could not find any real usage other than what's
currently in OE-Core for WHITELIST_GPL-3.0.

If you are using WHITELIST_<license>, please let me reply with your
usage.

[0] https://lists.openembedded.org/g/openembedded-devel/message/95166
We need to be mindful that we need to resolve this to unblock the
other language changes and feature creep here is potentially
problematic. I do think it is worth trying to improve things rather
than blindly allowing the horrible syntax in this variable to
continue though.

The test case we have for this currently is:

WHITELIST_GPL-3.0:pn-core-image-minimal = "bash"

so I'd wondered about an alternative of:

INCOMPATIBLE_LICENSE_EXCEPTIONS:pn-core-image-minimal = "bash:GPL-3.0"
You do not really need the license here (more than possibly as
information to anyone reading the code). Specifying that a package is
allowed in an image regardless of its licenses can just as well be
done by allowing the variable to take a list of packages:

INCOMPATIBLE_LICENSE_EXCEPTIONS:pn-core-image-minimal = "bash readline"
I'm not sure I agree with that. I get the feeling some people want to
allow these packages as long as they're with license X. Licenses can
change over time and this allows a safety net that if it changes, the
exception has to be updated too.
Well, to complicate matters then, if you really want to be explicit
with the license exceptions you make for a recipe or a package, I will
argue that you need to do it for all incompatible licenses a recipe
uses. With today's code, as soon as you allow one incompatible license
for a recipe, the recipe is allowed to be built, even if it has
specified multiple incompatible licenses. E.g., if we take the gnupg
recipe, which specifies LICENSE = "GPLv3 & LGPLv3", and we have an
INCOMPATIBLE_LICENSE = "GPL-3.0-only LGPL-3.0-only", with your example
I would then expect the following:

INCOMPATIBLE_LICENSE_EXCEPTIONS:pn-core-image-minimal = "gnupg:GPL-3.0-only gnupg:LGPL-3.0-only"

to allow the gnupg package to be installed in the image. However, I
guess you could combine our suggestions by making the license part
of the value optional, i.e., the following would then also allow the
package to be included under the same conditions:

INCOMPATIBLE_LICENSE_EXCEPTIONS:pn-core-image-minimal = "gnupg"

An alternative (and in my opinion better) variable name would be:

IMAGE_ALLOW_PACKAGES:pn-core-image-minimal = "bash readline"

focusing on that this is a list of _packages_ allowed for a given
_image_. This does not explicitly mention licenses, but I do not
believe that is needed.
If you assume this isn't being done on license reasons, sure. Except
see above, I think this does need to account for licenses, at least
the way many use it.

After all there could be multiple reasons a package is not allowed in
an image and this variable would allow to
ignore them all because that is typically what you want to do:
specify that you really want that package in that image. I guess
this is in some sense the opposite of PACKAGE_EXCLUDE. And I guess
like :append vs :remove, if someone for some reason specifies a
package in both IMAGE_ALLOW_PACKAGES and PACKAGE_EXCLUDE, then the
later should win (to err on the side of caution).
How would IMAGE_ALLOW_PACKAGES work and be different to IMAGE_INSTALL?
Well, IMAGE_INSTALL determines what packages you want in your image,
while IMAGE_ALLOW_PACKAGES allows packages to be installed that would
otherwise render an error if you tried. To me it seems perfectly
natural, but I am probably a bit biased. ;)

This interface would likely confuse more users than it would help.
Well, it is really based on that only package names need to be specified
and not the licenses. If both are needed (as you believe), then this name
makes less sense.

For the case where you want to allow a recipe to be built despite
it using licenses that are otherwise not allowed you can simply use

INCOMPATIBLE_LICENSE:pn-foo = ""

and don't really need a separate variable for it.
That is a good point and perhaps should influence how an
INCOMPATIBLE_LICENSE_EXCEPTIONS should be package based rather than
recipe. I doubt the pn- override existed when that variable was
originally added.
I definitely believe that we need to separate the case for building
a recipe from the case for installing a package into an image. For the
recipe case, I believe that manipulating RECIPE_INCOMPATIBLE_LICENSES
(or whatever the variable ends up being called) using normal variable
manipulations should be enough, without having to introduce yet
another variable. I.e., my example above would correspond to giving
a blanket exception for all licenses for the given recipe, while a
case where a specific license is excepted would be something like
RECIPE_INCOMPATIBLE_LICENSES:remove:pn-foo = "GPL-3.0-only"

For the installing a package into an image case, I don't see a way
to avoid an extra variable as discussed above. However, the name
INCOMPATIBLE_LICENSE_EXCEPTIONS gives no indications that it is
image specific, which I think is bad. If we introduce the other
names I suggest below, then this would instead map to
IMAGE_INCOMPATIBLE_LICENSES_EXCEPTIONS (though I dislike that name
because it is so long, but maybe it is inevitable).

I'm still of the opinion the AVAILABLE_LICENSES variable is something
we should be aiming to remove ultimately too, it has horrible issues
with layers changing hashes for all recipes.
Since I was the one to introduce it, I will answer to that. It was
introduced so that it is possible to specify compatible licenses
instead of incompatible licenses by basically calculating the
incompatible licenses by taking the set of available licenses minus
the set of compatible licenses. If a mechanism to do that is
introduced, e.g., by adding support for a COMPATIBLE_LICENSES in
addition to INCOMPATIBLE_LICENSE, then I am of the opinion that we
can remove AVAILABLE_LICENSES again.

We also need this mechanism in the code for handling allowed vs
disallowed licenses because the current code really cannot handled a
list of many hundreds of incompatible licenses, which is what we got
after all SPDX licenses were added to OE-Core. The code is extremely
inefficient and evaluates the list of incompatible licenses over and
over again. In our case that meant the recipe parsing time tripled.
Ironically, part of the reason I want to change the design of
WHITELIST_XXX is that it forces code that doesn't perform well. You
may be surprised to find that this change actually improves that 3x
issue you've seen. If not, I think it would lead us in a direction
where it can certainly help.

That said, we really need two sets of variables. One for blocking
the building of recipes because of its license(s), and one for
preventing packages with disallowed licenses to be included in a
given image. These are very different use cases and they take
totally different lists of items (recipes vs packages). The current
mess where the same variables are used for both cases needs to be
resolved.

So why do we need both cases? The first case where recipes are
prevented from being built makes it possible to, e.g., prevent a
newer versioned recipe that uses GPL-3.0-or-later to be built and
instead build an older version that uses GPL-2.0-or-later (i.e.,
what meta-gplv2 can be used for today).

The second case allows to build code that uses disallowed licenses,
as long as the packages are not added to the image. Why is this
useful? Because many recipes are built for some packages that are
never used in the given image, and it is then much easier to allow
them to be built as long as their output is not used. E.g., very
many recipes depend on bash which is GPL-3.0-or-later, making it
near impossible to avoid having to build it. However, it is
perfectly possible to build production images that do not need bash
to be installed.

I believe for this second case we should have two variables,
IMAGE_COMPATIBLE_LICENSES and IMAGE_INCOMPATIBLE_LICENSES. And to
make the naming clear for the first case, I would suggest calling
those variables RECIPE_COMPATIBLE_LICENSES and
RECIPE_INCOMPATIBLE_LICENSES.

Also note that the use of RECIPE_COMPATIBLE_LICENSES and
RECIPE_INCOMPATIBLE_LICENSES is mutually exclusive, as is the use
of IMAGE_COMPATIBLE_LICENSES and IMAGE_INCOMPATIBLE_LICENSES. I.e.,
you will have to choose whether to specify what licenses to allow
or what licenses to disallow. You cannot do both. This is because
specifying a list of compatible licenses means that all other
licenses are by definition incompatible, and vice versa. However,
this also means that it makes perfect sense to be able to, e.g.,
specify a few RECIPE_INCOMPATIBLE_LICENSES together with a list of
IMAGE_COMPATIBLE_LICENSES.
I suspect there will be people who will want configurations where they
specify that they are happy with X and definitely don't want Y to work.
The unspecified licenses would effectively be compatible but not be
explicitly specified. It seems like an odd setup but I can imagine
people configuring it and wanting it to work.

I am very worried about the performance implications in this "every
license" in a compatible list. Obviously for the case your legal
department cares about, you have to do it but I'm not sure we want
to force it onto everyone (and AVAILABLE_LICENSES already heads in
that direction).
By splitting the list of licenses into two, to handle the opposite
use cases of allowed vs disallowed licenses, we avoid the need to know
the full list of potential licenses as we only need to know what is
allowed or disallowed and can exclude anything else.

Thus the expectation is that the lists in either *_COMPATIBLE_LICENSES
or *_INCOMPATIBLE_LICENSES will be relatively short, compared to lists
based on all licenses. In our case, we have 80 licenses in our list
of allowed licenses. That is something that I think even the current
code can handle reasonably well. And without pattern matching and
mappings of old license names, the code would basically have to do
something like:

compatible_licenses = (d.getVar('COMPATIBLE_LICENSES') or '').split()
incompatible_licenses = (d.getVar('INCOMPATIBLE_LICENSES') or '').split()
disallowed_licenses = set()
...
if compatible_licenses and license not in compatible_licenses or \
incompatible_licenses and license in incompatible_licenses:
disallowed_licenses.add(license)

Also, based on the assumption that in most cases you want to be able to
build as much as possible, I would guess that you would typically use
RECIPES_INCOMPATIBLE_LICENSES to exclude only a select few licenses that
cannot be used at all (e.g., GPL-3.0), and then use
IMAGE_COMPATIBLE_LICENSES or IMAGE_INCOMPATIBLE_LICENSES to limit what
goes into the images. Thus the longer lists would only affect the image
recipes compared to all recipes as it is today.

Oh, and another thing I would like to take the opportunity to raise
is whether we should continue to support patterns in these list, or
if we should change it so that the lists of licenses need to
explicitly specify all licenses. The latter would greatly simplify
the code, especially when combined with only allowing SPDX names.
If we decide to remove support for patterns, and based on the
assumption that the pattern is typically used to specify "*GPL-3.0*",
we could make available a variable or two that contain the typical
sets of GPL licenses. This would also allow us to remove the code
that handles how an or-later licenses specified as '<license>+'
should be treated in combination with patterns.
I'd love to remove it but it is something which people want and now
expect from the code. You might not like it, others do. How do we
please everyone? I don't think we'd be able to remove that, only
perhaps limit it a little more.
But if we redesign the license handling anyway, can't we change their
expectations at the same time? There really aren't that many groups
of SPDX licenses where it is useful to use pattern matching anyway
and that you are likely to put in these variables. Basically I think
it comes down to some set of GPL licenses. I.e., replacing "*GPL-3.0*"
with the corresponding licenses really doesn't yield a very long list.
And as I suggested above, we could even provide a variable or two with
the expanded lists of the most typical use cases. E.g., if we added a
GPL3_LICENSES variable, the INCOMPATIBLE_LICENSES = "*GPL-3.0*" would
become INCOMPATIBLE_LICENSES = "${GPL3_LICENSES}". Yes, it is a little
less flexible, but it would do wonders for the code...

Anyway, this is a minor gripe compared to the rest, so if we can't
get rid of it, so be it.

So, to reiterate, these are the variables that I suggest we adopt
to be able to handle the various use cases regarding licenses:

RECIPE_COMPATIBLE_LICENSES - list of compatible licenses for
recipes
RECIPE_INCOMPATIBLE_LICENSES - list of incompatible licenses for
recipes
IMAGE_COMPATIBLE_LICENSES - list of compatible licenses for
packages in images
IMAGE_INCOMPATIBLE_LICENSES - list of incompatible licenses for
packages in images
IMAGE_ALLOW_PACKAGES - list of packages allowed in the
image regardless of licenses and
other restrictions

This also means that the old WHITELIST_*, INCOMPATIBLE_LICENSE
and AVAILABLE_LICENSES variables are removed. I also suggest we
remove the support for patterns in the new variables.
Whilst I don't just want to map WHITELIST_* to something renamed, I think
the above is a bit too radical to get into this late in the release cycle
so I suspect we'll have to compromise. This discussion needed to happen
early in the cycle with people actively working on it.
And I really, really wish I had had the time then to do exactly that. :(
I know I even said I would after I noticed the problem with the prolonged
recipe parsing times due to the huge increase in available licenses, but
alas time wasn't on my side. So I did what anyone would do and made a
workaround and went on to handle more pressing problems. I am very sorry
about that.

Just to be really clear, in particular I detest IMAGE_ALLOW_PACKAGES as a
variable name. I think you're trying to be too wide in scope with "other
restrictions" and it will just confuse people.
Ouch, and I thought it was pretty ok. :( Especially compared to
INCOMPATIBLE_LICENSE_EXCEPTIONS, which I don't like very much (though that
is mostly based on its length). Anyway, this has been discussed at length
above, so I will not delve on it more here.

Cheers,

Richard
//Peter


Re: [dunfell][PATCH v2] openssl: upgrade to 1.1.1m for CVE-2021-4160

Steve Sakoman
 

On Fri, Feb 18, 2022 at 2:27 PM Tim Orling <ticotimo@...> wrote:



On Fri, Feb 18, 2022 at 3:36 PM Steve Sakoman <steve@...> wrote:

On Tue, Feb 15, 2022 at 5:59 PM Tim Orling <ticotimo@...> wrote:

Changes are only security and bug fixes.
I'm seeing ptest errors:

WARNING: core-image-sato-sdk-ptest-1.0-r0 do_testimage: There were
failing ptests.
Traceback (most recent call last):
File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py",
line 36, in wrapped_f
return func(*args, **kwargs)
File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py",
line 36, in wrapped_f
return func(*args, **kwargs)
File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py",
line 36, in wrapped_f
return func(*args, **kwargs)
File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/runtime/cases/ptest.py",
line 25, in test_ptestrunner_expectfail
self.do_ptestrunner()
File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/runtime/cases/ptest.py",
line 108, in do_ptestrunner
self.fail(failmsg)
AssertionError: Failed ptests:
{'openssl': ['test/recipes/30-test_evp_extra.t,_test_returned_1']}

I saw this on qemux86-64, but was not sure it was due to the upgrade or a one off infra issue. I’ll dig deeper and see what might be happening.
I re-ran the test and got the same error, so it doesn't seem to be intermittent.

Thanks!

Steve



Happens with both qemuarm64-ptest and qemux86-64-ptest:

https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/2863
https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/3124

Steve

https://www.openssl.org/news/cl111.txt
https://git.openssl.org/?p=openssl.git;a=log;h=refs/tags/OpenSSL_1_1_1m

CVE: CVE-2021-4160

https://nvd.nist.gov/vuln/detail/CVE-2021-4160

Signed-off-by: Tim Orling <tim.orling@...>
---
Changes in v2:
- drop SRC_URI[md5sum] that devtool snuck in.

.../openssl/{openssl_1.1.1l.bb => openssl_1.1.1m.bb} | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
rename meta/recipes-connectivity/openssl/{openssl_1.1.1l.bb => openssl_1.1.1m.bb} (98%)

diff --git a/meta/recipes-connectivity/openssl/openssl_1.1.1l.bb b/meta/recipes-connectivity/openssl/openssl_1.1.1m.bb
similarity index 98%
rename from meta/recipes-connectivity/openssl/openssl_1.1.1l.bb
rename to meta/recipes-connectivity/openssl/openssl_1.1.1m.bb
index bf7cd6527ef..c6f8499d4f5 100644
--- a/meta/recipes-connectivity/openssl/openssl_1.1.1l.bb
+++ b/meta/recipes-connectivity/openssl/openssl_1.1.1m.bb
@@ -24,7 +24,7 @@ SRC_URI_append_class-nativesdk = " \
file://environment.d-openssl.sh \
"

-SRC_URI[sha256sum] = "0b7a3e5e59c34827fe0c3a74b7ec8baef302b98fa80088d7f9153aa16fa76bd1"
+SRC_URI[sha256sum] = "f89199be8b23ca45fc7cb9f1d8d3ee67312318286ad030f5316aca6462db6c96"

inherit lib_package multilib_header multilib_script ptest
MULTILIB_SCRIPTS = "${PN}-bin:${bindir}/c_rehash"
--
2.30.2


Re: [PATCH] rust: Introduce RS_BUILD_ARCH to handle ppc64le

Andrew Jeffery
 

On Fri, 18 Feb 2022, at 16:43, Khem Raj wrote:
On Thu, Feb 17, 2022 at 9:43 PM Andrew Jeffery <andrew@...> wrote:

On modern Power systems `uname -m` reports 'ppc64le'. However, Rust's
toolchain names the architecture 'powerpc64le'.

Introduce RS_BUILD_ARCH to provide an indirection to fix this mismatch.

I've tested each of the generated URIs for Power using the output from
`bitbake -e rust-native` and could successfully fetch the artifacts.
thanks
can you also check if arch_to_rust_target_arch() in
meta/recipes-devtools/rust/rust-common.inc
is computing it right for ppc64le case ?
Yep, I'll poke at that.

Andrew


Re: [oe] INCOMPATIBLE_LICENSES and WHITELIST_<license> usage

Richard Purdie
 

On Fri, 2022-02-18 at 23:34 +0000, Peter Kjellerstedt wrote:
Warning: wall of text ahead. Sorry about that, but I believe it is
important that we get this right if we are redesigning it.

TL;DR: see the end for the new suggested licensing variables.

-----Original Message-----
From: openembedded-devel@... <openembedded-
devel@...> On Behalf Of Richard Purdie
Sent: den 18 februari 2022 15:14
To: Saul Wold <Saul.Wold@...>; openembedded-
architecture@...; OE-core <openembedded-
core@...>; OpenEmbedded Devel List <openembedded-
devel@...>
Subject: Re: [oe] INCOMPATIBLE_LICENSES and WHITELIST_<license> usage

On Thu, 2022-02-17 at 15:01 -0800, Saul Wold wrote:
I am working on a proposal to re-write how INCOMPATIBLE_LICENSES is used
and processed to possibly include a COMPATIBLE_LICENSES variable as
well, see PeterK's email [0]

I am trying to determine the usage of WHITELIST_<license> which would be
used to override a license that might be listed in INCOMPATIBLE_LICENSES
variable.

Randy and I have done a quick and dirty survey of a 100 or so layers
(thanks Randy) and could not find any real usage other than what's
currently in OE-Core for WHITELIST_GPL-3.0.

If you are using WHITELIST_<license>, please let me reply with your
usage.


[0] https://lists.openembedded.org/g/openembedded-devel/message/95166
We need to be mindful that we need to resolve this to unblock the other
language changes and feature creep here is potentially problematic. I do
think it is worth trying to improve things rather than blindly allowing
the horrible syntax in this variable to continue though.

The test case we have for this currently is:

WHITELIST_GPL-3.0:pn-core-image-minimal = "bash"

so I'd wondered about an alternative of:

INCOMPATIBLE_LICENSE_EXCEPTIONS:pn-core-image-minimal = "bash:GPL-3.0"
You do not really need the license here (more than possibly as
information to anyone reading the code). Specifying that a package is
allowed in an image regardless of its licenses can just as well be
done by allowing the variable to take a list of packages:

INCOMPATIBLE_LICENSE_EXCEPTIONS:pn-core-image-minimal = "bash readline"
I'm not sure I agree with that. I get the feeling some people want to allow
these packages as long as they're with license X. Licenses can change over time
and this allows a safety net that if it changes, the exception has to be updated
too.

An alternative (and in my opinion better) variable name would be:

IMAGE_ALLOW_PACKAGES:pn-core-image-minimal = "bash readline"

focusing on that this is a list of _packages_ allowed for a given
_image_. This does not explicitly mention licenses, but I do not
believe that is needed.
If you assume this isn't being done on license reasons, sure. Except see above,
I think this does need to account for licenses, at least the way many use it.

After all there could be multiple reasons a package is not allowed in 
an image and this variable would allow to
ignore them all because that is typically what you want to do:
specify that you really want that package in that image. I guess
this is in some sense the opposite of PACKAGE_EXCLUDE. And I guess
like :append vs :remove, if someone for some reason specifies a
package in both IMAGE_ALLOW_PACKAGES and PACKAGE_EXCLUDE, then the
later should win (to err on the side of caution).
How would IMAGE_ALLOW_PACKAGES work and be different to IMAGE_INSTALL? This
interface would likely confuse more users than it would help.

For the case where you want to allow a recipe to be built despite
it using licenses that are otherwise not allowed you can simply use

INCOMPATIBLE_LICENSE:pn-foo = ""

and don't really need a separate variable for it.
That is a good point and perhaps should influence how an
INCOMPATIBLE_LICENSE_EXCEPTIONS should be package based rather than recipe. I
doubt the pn- override existed when that variable was originally added.

I'm still of the opinion the AVAILABLE_LICENSES variable is something we
should be aiming to remove ultimately too, it has horrible issues with
layers changing hashes for all recipes.
Since I was the one to introduce it, I will answer to that. It was
introduced so that it is possible to specify compatible licenses
instead of incompatible licenses by basically calculating the
incompatible licenses by taking the set of available licenses minus
the set of compatible licenses. If a mechanism to do that is
introduced, e.g., by adding support for a COMPATIBLE_LICENSES in
addition to INCOMPATIBLE_LICENSE, then I am of the opinion that we
can remove AVAILABLE_LICENSES again.

We also need this mechanism in the code for handling allowed vs
disallowed licenses because the current code really cannot handled a
list of many hundreds of incompatible licenses, which is what we got
after all SPDX licenses were added to OE-Core. The code is extremely
inefficient and evaluates the list of incompatible licenses over and
over again. In our case that meant the recipe IMAGE_ALLOW_PACKAGESparsing time tripled.
Ironically, part of the reason I want to change the design of WHITELIST_XXX is
that it forces code that doesn't perform well. You may be surprised to find that
this change actually improves that 3x issue you've seen. If not, I think it
would lead us in a direction where it can certainly help.

That said, we really need two sets of variables. One for blocking
the building of recipes because of its license(s), and one for
preventing packages with disallowed licenses to be included in a
given image. These are very different use cases and they take
totally different lists of items (recipes vs packages). The current
mess where the same variables are used for both cases needs to be
resolved.

So why do we need both cases? The first case where recipes are
prevented from being built makes it possible to, e.g., prevent a
newer versioned recipe that uses GPL-3.0-or-later to be built and
instead build an older version that uses GPL-2.0-or-later (i.e.,
what meta-gplv2 can be used for today).

The second case allows to build code that uses disallowed licenses,
as long as the packages are not added to the image. Why is this
useful? Because many recipes are built for some packages that are
never used in the given image, and it is then much easier to allow
them to be built as long as their output is not used. E.g., very
many recipes depend on bash which is GPL-3.0-or-later, making it
near impossible to avoid having to build it. However, it is
perfectly possible to build production images that do not need bash
to be installed.

I believe for this second case we should have two variables,
IMAGE_COMPATIBLE_LICENSES and IMAGE_INCOMPATIBLE_LICENSES. And to
make the naming clear for the first case, I would suggest calling
those variables RECIPE_COMPATIBLE_LICENSES and
RECIPE_INCOMPATIBLE_LICENSES.

Also note that the use of RECIPE_COMPATIBLE_LICENSES and
RECIPE_INCOMPATIBLE_LICENSES is mutually exclusive, as is the use
of IMAGE_COMPATIBLE_LICENSES and IMAGE_INCOMPATIBLE_LICENSES. I.e.,
you will have to choose whether to specify what licenses to allow
or what licenses to disallow. You cannot do both. This is because
specifying a list of compatible licenses means that all other
licenses are by definition incompatible, and vice versa. However,
this also means that it makes perfect sense to be able to, e.g.,
specify a few RECIPE_INCOMPATIBLE_LICENSES together with a list of
IMAGE_COMPATIBLE_LICENSES.
I suspect there will be people who will want configurations where they specify
that they are happy with X and definitely don't want Y to work. The unspecified
licenses would effectively be compatible but not be explicitly specified. It
seems like an odd setup but I can imagine people configuring it and wanting it
to work.

I am very worried about the performance implications in this "every license" in
a compatible list. Obviously for the case your legal department cares about, you
have to do it but I'm not sure we want to force it onto everyone (and
AVAILABLE_LICENSES already heads in that direction).

Oh, and another thing I would like to take the opportunity to raise
is whether we should continue to support patterns in these list, or
if we should change it so that the lists of licenses need to
explicitly specify all licenses. The latter would greatly simplify
the code, especially when combined with only allowing SPDX names.
If we decide to remove support for patterns, and based on the
assumption that the pattern is typically used to specify "*GPL-3.0*",
we could make available a variable or two that contain the typical
sets of GPL licenses. This would also allow us to remove the code
that handles how an or-later licenses specified as '<license>+'
should be treated in combination with patterns.
I'd love to remove it but it is something which people want and now expect from
the code. You might not like it, others do. How do we please everyone? I don't
think we'd be able to remove that, only perhaps limit it a little more.

So, to reiterate, these are the variables that I suggest we adopt
to be able to handle the various use cases regarding licenses:

RECIPE_COMPATIBLE_LICENSES - list of compatible licenses for
recipes
RECIPE_INCOMPATIBLE_LICENSES - list of incompatible licenses for
recipes
IMAGE_COMPATIBLE_LICENSES - list of compatible licenses for
packages in images
IMAGE_INCOMPATIBLE_LICENSES - list of incompatible licenses for
packages in images
IMAGE_ALLOW_PACKAGES - list of packages allowed in the
image regardless of licenses and
other restrictions

This also means that the old WHITELIST_*, INCOMPATIBLE_LICENSE
and AVAILABLE_LICENSES variables are removed. I also suggest we
remove the support for patterns in the new variables.
Whilst I don't just want to map WHITELIST_* to something renamed, I think the
above is a bit too radical to get into this late in the release cycle so I
suspect we'll have to compromise. This discussion needed to happen early in the
cycle with people actively working on it.

Just to be really clear, in particular I detest IMAGE_ALLOW_PACKAGES as a
variable name. I think you're trying to be too wide in scope with "other
restrictions" and it will just confuse people.

Cheers,

Richard


Re: [dunfell][PATCH v2] openssl: upgrade to 1.1.1m for CVE-2021-4160

Tim Orling
 



On Fri, Feb 18, 2022 at 3:36 PM Steve Sakoman <steve@...> wrote:
On Tue, Feb 15, 2022 at 5:59 PM Tim Orling <ticotimo@...> wrote:
>
> Changes are only security and bug fixes.

I'm seeing ptest errors:

WARNING: core-image-sato-sdk-ptest-1.0-r0 do_testimage: There were
failing ptests.
Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py",
line 36, in wrapped_f
    return func(*args, **kwargs)
  File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py",
line 36, in wrapped_f
    return func(*args, **kwargs)
  File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py",
line 36, in wrapped_f
    return func(*args, **kwargs)
  File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/runtime/cases/ptest.py",
line 25, in test_ptestrunner_expectfail
    self.do_ptestrunner()
  File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/runtime/cases/ptest.py",
line 108, in do_ptestrunner
    self.fail(failmsg)
AssertionError: Failed ptests:
{'openssl': ['test/recipes/30-test_evp_extra.t,_test_returned_1']}

I saw this on qemux86-64, but was not sure it was due to the upgrade or a one off infra issue. I’ll dig deeper and see what might be happening.


Happens with both qemuarm64-ptest and qemux86-64-ptest:

https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/2863
https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/3124

Steve

> https://www.openssl.org/news/cl111.txt
> https://git.openssl.org/?p=openssl.git;a=log;h=refs/tags/OpenSSL_1_1_1m
>
> CVE: CVE-2021-4160
>
> https://nvd.nist.gov/vuln/detail/CVE-2021-4160
>
> Signed-off-by: Tim Orling <tim.orling@...>
> ---
> Changes in v2:
>  - drop SRC_URI[md5sum] that devtool snuck in.
>
>  .../openssl/{openssl_1.1.1l.bb => openssl_1.1.1m.bb}            | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>  rename meta/recipes-connectivity/openssl/{openssl_1.1.1l.bb => openssl_1.1.1m.bb} (98%)
>
> diff --git a/meta/recipes-connectivity/openssl/openssl_1.1.1l.bb b/meta/recipes-connectivity/openssl/openssl_1.1.1m.bb
> similarity index 98%
> rename from meta/recipes-connectivity/openssl/openssl_1.1.1l.bb
> rename to meta/recipes-connectivity/openssl/openssl_1.1.1m.bb
> index bf7cd6527ef..c6f8499d4f5 100644
> --- a/meta/recipes-connectivity/openssl/openssl_1.1.1l.bb
> +++ b/meta/recipes-connectivity/openssl/openssl_1.1.1m.bb
> @@ -24,7 +24,7 @@ SRC_URI_append_class-nativesdk = " \
>             file://environment.d-openssl.sh \
>             "
>
> -SRC_URI[sha256sum] = "0b7a3e5e59c34827fe0c3a74b7ec8baef302b98fa80088d7f9153aa16fa76bd1"
> +SRC_URI[sha256sum] = "f89199be8b23ca45fc7cb9f1d8d3ee67312318286ad030f5316aca6462db6c96"
>
>  inherit lib_package multilib_header multilib_script ptest
>  MULTILIB_SCRIPTS = "${PN}-bin:${bindir}/c_rehash"
> --
> 2.30.2
>


Re: [dunfell][PATCH v2] openssl: upgrade to 1.1.1m for CVE-2021-4160

Steve Sakoman
 

On Tue, Feb 15, 2022 at 5:59 PM Tim Orling <ticotimo@...> wrote:

Changes are only security and bug fixes.
I'm seeing ptest errors:

WARNING: core-image-sato-sdk-ptest-1.0-r0 do_testimage: There were
failing ptests.
Traceback (most recent call last):
File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py",
line 36, in wrapped_f
return func(*args, **kwargs)
File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py",
line 36, in wrapped_f
return func(*args, **kwargs)
File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py",
line 36, in wrapped_f
return func(*args, **kwargs)
File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/runtime/cases/ptest.py",
line 25, in test_ptestrunner_expectfail
self.do_ptestrunner()
File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/runtime/cases/ptest.py",
line 108, in do_ptestrunner
self.fail(failmsg)
AssertionError: Failed ptests:
{'openssl': ['test/recipes/30-test_evp_extra.t,_test_returned_1']}

Happens with both qemuarm64-ptest and qemux86-64-ptest:

https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/2863
https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/3124

Steve

https://www.openssl.org/news/cl111.txt
https://git.openssl.org/?p=openssl.git;a=log;h=refs/tags/OpenSSL_1_1_1m

CVE: CVE-2021-4160

https://nvd.nist.gov/vuln/detail/CVE-2021-4160

Signed-off-by: Tim Orling <tim.orling@...>
---
Changes in v2:
- drop SRC_URI[md5sum] that devtool snuck in.

.../openssl/{openssl_1.1.1l.bb => openssl_1.1.1m.bb} | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
rename meta/recipes-connectivity/openssl/{openssl_1.1.1l.bb => openssl_1.1.1m.bb} (98%)

diff --git a/meta/recipes-connectivity/openssl/openssl_1.1.1l.bb b/meta/recipes-connectivity/openssl/openssl_1.1.1m.bb
similarity index 98%
rename from meta/recipes-connectivity/openssl/openssl_1.1.1l.bb
rename to meta/recipes-connectivity/openssl/openssl_1.1.1m.bb
index bf7cd6527ef..c6f8499d4f5 100644
--- a/meta/recipes-connectivity/openssl/openssl_1.1.1l.bb
+++ b/meta/recipes-connectivity/openssl/openssl_1.1.1m.bb
@@ -24,7 +24,7 @@ SRC_URI_append_class-nativesdk = " \
file://environment.d-openssl.sh \
"

-SRC_URI[sha256sum] = "0b7a3e5e59c34827fe0c3a74b7ec8baef302b98fa80088d7f9153aa16fa76bd1"
+SRC_URI[sha256sum] = "f89199be8b23ca45fc7cb9f1d8d3ee67312318286ad030f5316aca6462db6c96"

inherit lib_package multilib_header multilib_script ptest
MULTILIB_SCRIPTS = "${PN}-bin:${bindir}/c_rehash"
--
2.30.2


Re: [oe] INCOMPATIBLE_LICENSES and WHITELIST_<license> usage

Peter Kjellerstedt
 

Warning: wall of text ahead. Sorry about that, but I believe it is
important that we get this right if we are redesigning it.

TL;DR: see the end for the new suggested licensing variables.

-----Original Message-----
From: openembedded-devel@... <openembedded-
devel@...> On Behalf Of Richard Purdie
Sent: den 18 februari 2022 15:14
To: Saul Wold <Saul.Wold@...>; openembedded-
architecture@...; OE-core <openembedded-
core@...>; OpenEmbedded Devel List <openembedded-
devel@...>
Subject: Re: [oe] INCOMPATIBLE_LICENSES and WHITELIST_<license> usage

On Thu, 2022-02-17 at 15:01 -0800, Saul Wold wrote:
I am working on a proposal to re-write how INCOMPATIBLE_LICENSES is used
and processed to possibly include a COMPATIBLE_LICENSES variable as
well, see PeterK's email [0]

I am trying to determine the usage of WHITELIST_<license> which would be
used to override a license that might be listed in INCOMPATIBLE_LICENSES
variable.

Randy and I have done a quick and dirty survey of a 100 or so layers
(thanks Randy) and could not find any real usage other than what's
currently in OE-Core for WHITELIST_GPL-3.0.

If you are using WHITELIST_<license>, please let me reply with your
usage.


[0] https://lists.openembedded.org/g/openembedded-devel/message/95166
We need to be mindful that we need to resolve this to unblock the other
language changes and feature creep here is potentially problematic. I do
think it is worth trying to improve things rather than blindly allowing
the horrible syntax in this variable to continue though.

The test case we have for this currently is:

WHITELIST_GPL-3.0:pn-core-image-minimal = "bash"

so I'd wondered about an alternative of:

INCOMPATIBLE_LICENSE_EXCEPTIONS:pn-core-image-minimal = "bash:GPL-3.0"
You do not really need the license here (more than possibly as
information to anyone reading the code). Specifying that a package is
allowed in an image regardless of its licenses can just as well be
done by allowing the variable to take a list of packages:

INCOMPATIBLE_LICENSE_EXCEPTIONS:pn-core-image-minimal = "bash readline"

An alternative (and in my opinion better) variable name would be:

IMAGE_ALLOW_PACKAGES:pn-core-image-minimal = "bash readline"

focusing on that this is a list of _packages_ allowed for a given
_image_. This does not explicitly mention licenses, but I do not
believe that is needed. After all there could be multiple reasons a
package is not allowed in an image and this variable would allow to
ignore them all because that is typically what you want to do:
specify that you really want that package in that image. I guess
this is in some sense the opposite of PACKAGE_EXCLUDE. And I guess
like :append vs :remove, if someone for some reason specifies a
package in both IMAGE_ALLOW_PACKAGES and PACKAGE_EXCLUDE, then the
later should win (to err on the side of caution).

For the case where you want to allow a recipe to be built despite
it using licenses that are otherwise not allowed you can simply use

INCOMPATIBLE_LICENSE:pn-foo = ""

and don't really need a separate variable for it.

which matches the current functionality, removes the issue that the name
of the variable is unknown without iterating every possible license name
and makes it clear where it is applying to.

I don't really like INCOMPATIBLE_LICENSE_ALLOWED_RECIPES since:

a) it is long
b) it refers to recipes when it works against packages

(INCOMPATIBLE_LICENSE_PACKAGE_EXCEPTIONS is more correct but still long)

I do like it mentioning INCOMPATIBLE_LICENSE in full since it works in
conjunction with that variable and that is definitely not clear from the
current WHITELIST_XXX until you look at the code.

I'm still of the opinion the AVAILABLE_LICENSES variable is something we
should be aiming to remove ultimately too, it has horrible issues with
layers changing hashes for all recipes.
Since I was the one to introduce it, I will answer to that. It was
introduced so that it is possible to specify compatible licenses
instead of incompatible licenses by basically calculating the
incompatible licenses by taking the set of available licenses minus
the set of compatible licenses. If a mechanism to do that is
introduced, e.g., by adding support for a COMPATIBLE_LICENSES in
addition to INCOMPATIBLE_LICENSE, then I am of the opinion that we
can remove AVAILABLE_LICENSES again.

We also need this mechanism in the code for handling allowed vs
disallowed licenses because the current code really cannot handled a
list of many hundreds of incompatible licenses, which is what we got
after all SPDX licenses were added to OE-Core. The code is extremely
inefficient and evaluates the list of incompatible licenses over and
over again. In our case that meant the recipe parsing time tripled.

Cheers,

Richard
That said, we really need two sets of variables. One for blocking
the building of recipes because of its license(s), and one for
preventing packages with disallowed licenses to be included in a
given image. These are very different use cases and they take
totally different lists of items (recipes vs packages). The current
mess where the same variables are used for both cases needs to be
resolved.

So why do we need both cases? The first case where recipes are
prevented from being built makes it possible to, e.g., prevent a
newer versioned recipe that uses GPL-3.0-or-later to be built and
instead build an older version that uses GPL-2.0-or-later (i.e.,
what meta-gplv2 can be used for today).

The second case allows to build code that uses disallowed licenses,
as long as the packages are not added to the image. Why is this
useful? Because many recipes are built for some packages that are
never used in the given image, and it is then much easier to allow
them to be built as long as their output is not used. E.g., very
many recipes depend on bash which is GPL-3.0-or-later, making it
near impossible to avoid having to build it. However, it is
perfectly possible to build production images that do not need bash
to be installed.

I believe for this second case we should have two variables,
IMAGE_COMPATIBLE_LICENSES and IMAGE_INCOMPATIBLE_LICENSES. And to
make the naming clear for the first case, I would suggest calling
those variables RECIPE_COMPATIBLE_LICENSES and
RECIPE_INCOMPATIBLE_LICENSES.

Also note that the use of RECIPE_COMPATIBLE_LICENSES and
RECIPE_INCOMPATIBLE_LICENSES is mutually exclusive, as is the use
of IMAGE_COMPATIBLE_LICENSES and IMAGE_INCOMPATIBLE_LICENSES. I.e.,
you will have to choose whether to specify what licenses to allow
or what licenses to disallow. You cannot do both. This is because
specifying a list of compatible licenses means that all other
licenses are by definition incompatible, and vice versa. However,
this also means that it makes perfect sense to be able to, e.g.,
specify a few RECIPE_INCOMPATIBLE_LICENSES together with a list of
IMAGE_COMPATIBLE_LICENSES.

Oh, and another thing I would like to take the opportunity to raise
is whether we should continue to support patterns in these list, or
if we should change it so that the lists of licenses need to
explicitly specify all licenses. The latter would greatly simplify
the code, especially when combined with only allowing SPDX names.
If we decide to remove support for patterns, and based on the
assumption that the pattern is typically used to specify "*GPL-3.0*",
we could make available a variable or two that contain the typical
sets of GPL licenses. This would also allow us to remove the code
that handles how an or-later licenses specified as '<license>+'
should be treated in combination with patterns.

So, to reiterate, these are the variables that I suggest we adopt
to be able to handle the various use cases regarding licenses:

RECIPE_COMPATIBLE_LICENSES - list of compatible licenses for
recipes
RECIPE_INCOMPATIBLE_LICENSES - list of incompatible licenses for
recipes
IMAGE_COMPATIBLE_LICENSES - list of compatible licenses for
packages in images
IMAGE_INCOMPATIBLE_LICENSES - list of incompatible licenses for
packages in images
IMAGE_ALLOW_PACKAGES - list of packages allowed in the
image regardless of licenses and
other restrictions

This also means that the old WHITELIST_*, INCOMPATIBLE_LICENSE
and AVAILABLE_LICENSES variables are removed. I also suggest we
remove the support for patterns in the new variables.

//Peter


[PATCH] libgit2: Upgrade to 1.4.1

Khem Raj
 

this is bugfix release
https://github.com/libgit2/libgit2/releases/tag/v1.4.1

Signed-off-by: Khem Raj <raj.khem@...>
---
.../libgit2/{libgit2_1.4.0.bb => libgit2_1.4.1.bb} | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
rename meta/recipes-support/libgit2/{libgit2_1.4.0.bb => libgit2_1.4.1.bb} (78%)

diff --git a/meta/recipes-support/libgit2/libgit2_1.4.0.bb b/meta/recipes-support/libgit2/libgit2_1.4.1.bb
similarity index 78%
rename from meta/recipes-support/libgit2/libgit2_1.4.0.bb
rename to meta/recipes-support/libgit2/libgit2_1.4.1.bb
index 23a195b43b3..da33893b97d 100644
--- a/meta/recipes-support/libgit2/libgit2_1.4.0.bb
+++ b/meta/recipes-support/libgit2/libgit2_1.4.1.bb
@@ -5,8 +5,8 @@ LIC_FILES_CHKSUM = "file://COPYING;md5=e5a9227de4cb6afb5d35ed7b0fdf480d"

DEPENDS = "curl openssl zlib libssh2 libgcrypt libpcre2"

-SRC_URI = "git://github.com/libgit2/libgit2.git;branch=main;protocol=https"
-SRCREV = "1d5b9bd86dccc7347aaadf5e3ab122eed9413404"
+SRC_URI = "git://github.com/libgit2/libgit2.git;branch=maint/v1.4;protocol=https"
+SRCREV = "fdd15bcfca6b2ec4b7ecad1aa11a396cb15bd064"

S = "${WORKDIR}/git"

--
2.35.1


Proposal: INCOMPATIBLE_LICENSE_EXCEPTION

Saul Wold
 

Folks,

As a follow-on to yesterday's email and replies, I would like to make the following proposal for dealing with the changes to INCOMPATIBLE_LICENSE and associated variables.

Current Usage:

INCOMPATIBLE_LICENSE is a list of licenses that are considered incompatible with a distro's requirements. This is used to compare against packages built by a given recipe.

A set of exception variables based on the license name (currently WHITELIST_<license>) that contains a list of recipes that will be checked against the current recipe (PN) being evaluated. If it's in that list then all packages in that recipe will be built and included and the rest of the evaluation will be skipped.

Otherwise, the packages (PKGS) from the recipe will be evaluated to see if any have a package specific license (LICENSE:<pkgname>). If a package has a license other than the INCOMPATIBLE_LICENSE the recipe will be built and any packages with the INCOMPATIBLE_LICENSE will be excluded from being packaged in package.bbclass via LICENSE_EXCLUSION-<pkgname> internal variable.

The exception is predominately used for GPLv3 related packages, based on the emails replies overnight.


Proposal:

Keep the existing INCOMPATIBLE_LICENSE variable with the same behavior. The values in INCOMPATIBLE_LICENSE should be SDPX normalized license strings.

As Richard has already suggested an alternative variable that is more meaningful: INCOMPATIBLE_LICENSE_EXCEPTION with an a <pkg>:<spdx_lic> value. Rename the LICENSE_EXCLUSION-<pkg> variable to make it clear that it is an internal variable. The usage of the _EXCEPTION variable should contain pkg names not recipe name. ** This would be an important change **

Clean up code as appropriate to ensure the exceptions are handled once and identified during parsing.

I will start working on the implementation, Monday is a holiday in the US, so this should give sometime for this to be reviewed for Tuesday.


--
Sau!


Re: [PATCH v2] convert-variables: Script for Inclusive Language variable renames

Enrico Scholz
 

Richard Purdie <richard.purdie@...> writes:

This script searches for a list of variable that have been renamed
and converts them to their more descriptive names.
s/descriptive/politically correct/
We did try and make some of the names better describe what the variables
do and make the variables more consistent.

For example, a reference to HASHBASE was changed to BASEHASH which
better matches every other reference to that thing in the code.
When you really feel that this Yoda style naming must be fixed; then ok.
But for me, the gain would not be large enough to break the api.


The BB_ENV variables were also traditionally very confusing. Once you
know, you know but for new users they weren't great.
The new naming is much more confusing. While this kind of operation
(exclude or include something) was previously named by "whitelist" +
"blacklist" more or less consistently, it is scattered around across
multiple variants now.

Especially for the hash related lists, searching for "whitelist" or
"blacklist" in the sources revealed often the right way how to do
things.


Also see the discussion about the license variable issues. I'm very
consciously blocking the "simple" change as I want to see things
improve.

Variable names were perfectly readable. A change should make things
better, not worse.
See above, there are at least some changes which do. I'm sad you don't
agree.
ok; some changes are ok. E.g.

| "PNBLACKLIST":"SKIP_RECIPE",

because former name is misleading (it is not really a list/set but has
some specially addressing by varflags).


Why is the rename done at all? It makes the product just worse due to a
less usable api and because it causes a lot of work for the users of the
api (I shudder when I think about updating my CI to work with new and
old branches).
For better or worse some of the terminology we have is offensive to
some people.
Is there any real-world evidence that this is really the case? Especially
the embedded sector is filled with blacklisted terms (i2c master/slave,
spi mosi/miso) so I do not think that somebody is really offended.

In the opposite, I feel offended when such changes are labeled as
(technical) improvements ("more descriptive names").


Obviously (and sadly) there will probably always be some element of
something which someone could be offended by but these issues have
come under a particular spotlight. People with far more experience of
this than us have produced guidelines on the key issues,
In my experience, such people have too much spare time and invented a
problem only to come up with a solution later.


We do have people wanting to try and improve things. Projects do
need to be open to and able to change. If we don't try and take this
input and help those people make such changes where appropriate,
we'll just alienate and frustrate more users even if those users
were not personally offended by the language, a kind of "rot" can
set in to our community. I do not want to see that
I see a much higher risk when significant changes are done due to
political reasons rather than technical ones.


I have advised caution all the way through this. Should the project
get this wrong, the potential for negative social media feedback
against the project is huge.
Does such negative social media feedback really exist resp. does it
exist in channels that are relevant to us?

Linux kernel has not "cleaned" its api either and just said that future
development should avoid certain terms. I can not remember any negative
feedback.

Just do the same here: avoid some terms in future development and when
there are really technical reasons to touch existing variables, then
rename them.


For me personally, there is also a huge risk should I "misstep" in
handling this. The potential to break things for users is also really
high, I realise that. I have done a lot of work to try and make sure
issues are at least clearly reported to users with some idea of how
to resolve them so the transition is less painful. I do have some
experience of making changes to the project and am trying my best to
use that to our benefit.
You are doing a great job, but the project evolved beautiful over the
last >15 years with the old identifier and I can not remember any
complaints about them.



Enrico


Re: Typo in comment (was: tool to list source files)

Richard Purdie
 

On Thu, 2022-02-17 at 17:39 -0600, Joseph Reynolds wrote:
There seems to a typo in a comment in archiver.bbclass; ARCHIVE_MODE
should be ARCHIVER_MODE for "mirror" (letter R is missing).

https://github.com/openembedded/openembedded-core/blob/master/meta/classes/archiver.bbclass#L8

Joseph

P.S. The archiver class is awesome.  Thank you!
I fixed that typo, thanks for reporting.

Cheers,

Richard

11981 - 12000 of 173941