|
[Yocto bug #14742]
The build shows below error while building for arm machines.
Exception: TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'
Detailed error info :
Steps to reproduce:
1. Set MACHINE ?= "qemuarm" in local.conf & add 'TOOLCHAIN_HOST_TASK:append = " packagegroup-rust-cross-canadian-${MACHINE}"'
2. bitbake core-image-minimal -cpopulate_sdk
Complete Error:
ERROR: rust-cross-canadian-arm-1.59.0-r0 do_rust_gen_targets: Error executing a python function in exec_func_python() autogenerated:
The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_func_python() autogenerated', lineno: 2, function: <module>
0001:
*** 0002:do_rust_gen_targets(d)
0003:
File: '/ala-lpggp31/skokkonda/yocto/poky/meta/recipes-devtools/rust/rust-cross-canadian-common.inc', lineno: 31, function: do_rust_gen_targets
0027:
0028:LLVM_TARGET[x86_64] = "${RUST_HOST_SYS}"
0029:python do_rust_gen_targets () {
0030: wd = d.getVar('WORKDIR') + '/targets/'
*** 0031: rust_gen_target(d, 'TARGET', wd, d.getVar('TARGET_LLVM_FEATURES') or "", d.getVar('TARGET_LLVM_CPU'), d.getVar('TARGET_ARCH'))
0032: rust_gen_target(d, 'HOST', wd, "", "generic", d.getVar('HOST_ARCH'))
0033: rust_gen_target(d, 'BUILD', wd, "", "generic", d.getVar('BUILD_ARCH'))
0034:}
0035:
File: '/ala-lpggp31/skokkonda/yocto/poky/meta/recipes-devtools/rust/rust-common.inc', lineno: 330, function: rust_gen_target
0326: # build tspec
0327: tspec = {}
0328: tspec['llvm-target'] = d.getVarFlag('LLVM_TARGET', arch_abi)
0329: tspec['data-layout'] = d.getVarFlag('DATA_LAYOUT', arch_abi)
*** 0330: tspec['max-atomic-width'] = int(d.getVarFlag('MAX_ATOMIC_WIDTH', arch_abi))
0331: tspec['target-pointer-width'] = d.getVarFlag('TARGET_POINTER_WIDTH', arch_abi)
0332: tspec['target-c-int-width'] = d.getVarFlag('TARGET_C_INT_WIDTH', arch_abi)
0333: tspec['target-endian'] = d.getVarFlag('TARGET_ENDIAN', arch_abi)
0334: tspec['arch'] = arch_to_rust_target_arch(rust_arch)
Exception: TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'
Below are the local variables from rust_gen_target function for arm and aarch64 targets. Refer below, the tspec varibles for 'arm' generated with NoneType.
(a) Locals at rust_gen_target for arm::
tspec['data-layout'] = None, Type of tspec['data-layout'] = <class 'NoneType'>
tspec['data-layout'] = None, Type of tspec['data-layout'] = <class 'NoneType'>
DEBUG: Python function do_rust_gen_targets finished
(b) Locals at rust_gen_target for aarch64::
tspec['data-layout'] = aarch64-unknown-linux-gnu, Type of tspec['data-layout'] = <class 'str'>
tspec['max-atomic-width'] = 128, Type of tspec['max-atomic-width'] = <class 'int'>
Reason for changing arm-eabi to arm: The earlier changes introduced this bug, so reverting the change 'arm-eabi' to 'arm' fixed the issue.
The 'rust_gen_targets' Task added with its dependent variable list.
Signed-off-by: Sundeep KOKKONDA <sundeep.kokkonda@...>
---
meta/recipes-devtools/rust/rust-common.inc | 16 +++++++++-------
1 file changed, 9 insertions(+), 7 deletions(-)
diff --git a/meta/recipes-devtools/rust/rust-common.inc b/meta/recipes-devtools/rust/rust-common.inc
index 310aecef22..984fe9099e 100644
--- a/meta/recipes-devtools/rust/rust-common.inc
+++ b/meta/recipes-devtools/rust/rust-common.inc
@@ -119,13 +119,13 @@ def llvm_features(d):
## arm-unknown-linux-gnueabihf
-DATA_LAYOUT[arm-eabi] = "e-m:e-p:32:32-i64:64-v128:64:128-a:0:32-n32-S64"
-LLVM_TARGET[arm-eabi] = "${RUST_TARGET_SYS}"
-TARGET_ENDIAN[arm-eabi] = "little"
-TARGET_POINTER_WIDTH[arm-eabi] = "32"
-TARGET_C_INT_WIDTH[arm-eabi] = "32"
-MAX_ATOMIC_WIDTH[arm-eabi] = "64"
-FEATURES[arm-eabi] = "+v6,+vfp2"
+DATA_LAYOUT[arm] = "e-m:e-p:32:32-i64:64-v128:64:128-a:0:32-n32-S64"
+LLVM_TARGET[arm] = "${RUST_TARGET_SYS}"
+TARGET_ENDIAN[arm] = "little"
+TARGET_POINTER_WIDTH[arm] = "32"
+TARGET_C_INT_WIDTH[arm] = "32"
+MAX_ATOMIC_WIDTH[arm] = "64"
+FEATURES[arm] = "+v6,+vfp2"
## armv7-unknown-linux-gnueabihf
DATA_LAYOUT[armv7-eabi] = "e-m:e-p:32:32-i64:64-v128:64:128-a:0:32-n32-S64"
@@ -360,6 +360,8 @@ def rust_gen_target(d, thing, wd, features, cpu, arch, abi=""):
with open(wd + sys + '.json', 'w') as f:
json.dump(tspec, f, indent=4)
+do_rust_gen_targets[vardeps] += "DATA_LAYOUT LLVM_TARGET TARGET_ENDIAN TARGET_POINTER_WIDTH TARGET_C_INT_WIDTH MAX_ATOMIC_WIDTH FEATURES"
+
python do_rust_gen_targets () {
wd = d.getVar('WORKDIR') + '/targets/'
build_arch = d.getVar('BUILD_ARCH')
--
2.35.1
|
|
|
On Thu, 2022-05-12 at 22:59 -0700, Sundeep KOKKONDA wrote: [Yocto bug #14742]
The build shows below error while building for arm machines.
Exception: TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'
Detailed error info :
Steps to reproduce:
1. Set MACHINE ?= "qemuarm" in local.conf & add 'TOOLCHAIN_HOST_TASK:append = " packagegroup-rust-cross-canadian-${MACHINE}"'
2. bitbake core-image-minimal -cpopulate_sdk
Complete Error:
ERROR: rust-cross-canadian-arm-1.59.0-r0 do_rust_gen_targets: Error executing a python function in exec_func_python() autogenerated:
The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_func_python() autogenerated', lineno: 2, function: <module>
0001:
*** 0002:do_rust_gen_targets(d)
0003:
File: '/ala-lpggp31/skokkonda/yocto/poky/meta/recipes-devtools/rust/rust-cross-canadian-common.inc', lineno: 31, function: do_rust_gen_targets
0027:
0028:LLVM_TARGET[x86_64] = "${RUST_HOST_SYS}"
0029:python do_rust_gen_targets () {
0030: wd = d.getVar('WORKDIR') + '/targets/'
*** 0031: rust_gen_target(d, 'TARGET', wd, d.getVar('TARGET_LLVM_FEATURES') or "", d.getVar('TARGET_LLVM_CPU'), d.getVar('TARGET_ARCH'))
0032: rust_gen_target(d, 'HOST', wd, "", "generic", d.getVar('HOST_ARCH'))
0033: rust_gen_target(d, 'BUILD', wd, "", "generic", d.getVar('BUILD_ARCH'))
0034:}
0035:
File: '/ala-lpggp31/skokkonda/yocto/poky/meta/recipes-devtools/rust/rust-common.inc', lineno: 330, function: rust_gen_target
0326: # build tspec
0327: tspec = {}
0328: tspec['llvm-target'] = d.getVarFlag('LLVM_TARGET', arch_abi)
0329: tspec['data-layout'] = d.getVarFlag('DATA_LAYOUT', arch_abi)
*** 0330: tspec['max-atomic-width'] = int(d.getVarFlag('MAX_ATOMIC_WIDTH', arch_abi))
0331: tspec['target-pointer-width'] = d.getVarFlag('TARGET_POINTER_WIDTH', arch_abi)
0332: tspec['target-c-int-width'] = d.getVarFlag('TARGET_C_INT_WIDTH', arch_abi)
0333: tspec['target-endian'] = d.getVarFlag('TARGET_ENDIAN', arch_abi)
0334: tspec['arch'] = arch_to_rust_target_arch(rust_arch)
Exception: TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'
Below are the local variables from rust_gen_target function for arm and aarch64 targets. Refer below, the tspec varibles for 'arm' generated with NoneType.
(a) Locals at rust_gen_target for arm::
tspec['data-layout'] = None, Type of tspec['data-layout'] = <class 'NoneType'>
tspec['data-layout'] = None, Type of tspec['data-layout'] = <class 'NoneType'>
DEBUG: Python function do_rust_gen_targets finished
(b) Locals at rust_gen_target for aarch64::
tspec['data-layout'] = aarch64-unknown-linux-gnu, Type of tspec['data-layout'] = <class 'str'>
tspec['max-atomic-width'] = 128, Type of tspec['max-atomic-width'] = <class 'int'>
Reason for changing arm-eabi to arm: The earlier changes introduced this bug, so reverting the change 'arm-eabi' to 'arm' fixed the issue.
The 'rust_gen_targets' Task added with its dependent variable list.
Signed-off-by: Sundeep KOKKONDA <sundeep.kokkonda@...>
---
meta/recipes-devtools/rust/rust-common.inc | 16 +++++++++-------
1 file changed, 9 insertions(+), 7 deletions(-)
diff --git a/meta/recipes-devtools/rust/rust-common.inc b/meta/recipes-devtools/rust/rust-common.inc
index 310aecef22..984fe9099e 100644
--- a/meta/recipes-devtools/rust/rust-common.inc
+++ b/meta/recipes-devtools/rust/rust-common.inc
@@ -119,13 +119,13 @@ def llvm_features(d):
## arm-unknown-linux-gnueabihf
-DATA_LAYOUT[arm-eabi] = "e-m:e-p:32:32-i64:64-v128:64:128-a:0:32-n32-S64"
-LLVM_TARGET[arm-eabi] = "${RUST_TARGET_SYS}"
-TARGET_ENDIAN[arm-eabi] = "little"
-TARGET_POINTER_WIDTH[arm-eabi] = "32"
-TARGET_C_INT_WIDTH[arm-eabi] = "32"
-MAX_ATOMIC_WIDTH[arm-eabi] = "64"
-FEATURES[arm-eabi] = "+v6,+vfp2"
+DATA_LAYOUT[arm] = "e-m:e-p:32:32-i64:64-v128:64:128-a:0:32-n32-S64"
+LLVM_TARGET[arm] = "${RUST_TARGET_SYS}"
+TARGET_ENDIAN[arm] = "little"
+TARGET_POINTER_WIDTH[arm] = "32"
+TARGET_C_INT_WIDTH[arm] = "32"
+MAX_ATOMIC_WIDTH[arm] = "64"
+FEATURES[arm] = "+v6,+vfp2"
## armv7-unknown-linux-gnueabihf
DATA_LAYOUT[armv7-eabi] = "e-m:e-p:32:32-i64:64-v128:64:128-a:0:32-n32-S64"
@@ -360,6 +360,8 @@ def rust_gen_target(d, thing, wd, features, cpu, arch, abi=""):
with open(wd + sys + '.json', 'w') as f:
json.dump(tspec, f, indent=4)
+do_rust_gen_targets[vardeps] += "DATA_LAYOUT LLVM_TARGET TARGET_ENDIAN TARGET_POINTER_WIDTH TARGET_C_INT_WIDTH MAX_ATOMIC_WIDTH FEATURES"
+
python do_rust_gen_targets () {
wd = d.getVar('WORKDIR') + '/targets/'
build_arch = d.getVar('BUILD_ARCH') We need to split this into two. One patch to fix the determinism issue, the other to fix the arm bug. The "good" news is that this patch now fails consistently on the autobuilder, for example: https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/3522sstatetests.SStateTests.test_sstate_allarch_samesigs is failing which you can rerun with: oe-selftest -r sstatetests.SStateTests.test_sstate_allarch_samesigs Now it fails reliably, we need to work out why the arm change breaks that. If you split the patches into two we can likely merge the reproducibility one, then onwards with the arm issue. In case it isn't clear, the failing signatures are for adwaita-icon- theme which depends on librsvg-native which depends on rust-native. What this means is that the signature of rust-native is changing when MACHINE is changed in that test. This shouldn't happen. We already knew that from our other discussion but now we can prove it. It would be nice if a test showed us the native hash was changing and that is the other issue we discussed but one step at a time! To be clear, rust-native (and it's signature) should be independent of MACHINE so we need to fix that. This means that currently rust-native varies and rebuilds when you change MACHINE (after the dependency fix in this patch). Cheers, Richard
|
|
|
Hello,
On Thu, 2022-05-12 at 22:59 -0700, Sundeep KOKKONDA wrote:
[Yocto bug #14742]
The build shows below error while building for arm machines.
Exception: TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'
Detailed error info :
Steps to reproduce:
1. Set MACHINE ?= "qemuarm" in local.conf & add 'TOOLCHAIN_HOST_TASK:append = " packagegroup-rust-cross-canadian-${MACHINE}"'
2. bitbake core-image-minimal -cpopulate_sdk
Complete Error:
ERROR: rust-cross-canadian-arm-1.59.0-r0 do_rust_gen_targets: Error executing a python function in exec_func_python() autogenerated:
The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_func_python() autogenerated', lineno: 2, function: <module>
0001:
*** 0002:do_rust_gen_targets(d)
0003:
File: '/ala-lpggp31/skokkonda/yocto/poky/meta/recipes-devtools/rust/rust-cross-canadian-common.inc', lineno: 31, function: do_rust_gen_targets
0027:
0028:LLVM_TARGET[x86_64] = "${RUST_HOST_SYS}"
0029:python do_rust_gen_targets () {
0030: wd = d.getVar('WORKDIR') + '/targets/'
*** 0031: rust_gen_target(d, 'TARGET', wd, d.getVar('TARGET_LLVM_FEATURES') or "", d.getVar('TARGET_LLVM_CPU'), d.getVar('TARGET_ARCH'))
0032: rust_gen_target(d, 'HOST', wd, "", "generic", d.getVar('HOST_ARCH'))
0033: rust_gen_target(d, 'BUILD', wd, "", "generic", d.getVar('BUILD_ARCH'))
0034:}
0035:
File: '/ala-lpggp31/skokkonda/yocto/poky/meta/recipes-devtools/rust/rust-common.inc', lineno: 330, function: rust_gen_target
0326: # build tspec
0327: tspec = {}
0328: tspec['llvm-target'] = d.getVarFlag('LLVM_TARGET', arch_abi)
0329: tspec['data-layout'] = d.getVarFlag('DATA_LAYOUT', arch_abi)
*** 0330: tspec['max-atomic-width'] = int(d.getVarFlag('MAX_ATOMIC_WIDTH', arch_abi))
0331: tspec['target-pointer-width'] = d.getVarFlag('TARGET_POINTER_WIDTH', arch_abi)
0332: tspec['target-c-int-width'] = d.getVarFlag('TARGET_C_INT_WIDTH', arch_abi)
0333: tspec['target-endian'] = d.getVarFlag('TARGET_ENDIAN', arch_abi)
0334: tspec['arch'] = arch_to_rust_target_arch(rust_arch)
Exception: TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'
Below are the local variables from rust_gen_target function for arm and aarch64 targets. Refer below, the tspec varibles for 'arm' generated with NoneType.
(a) Locals at rust_gen_target for arm::
tspec['data-layout'] = None, Type of tspec['data-layout'] = <class 'NoneType'>
tspec['data-layout'] = None, Type of tspec['data-layout'] = <class 'NoneType'>
DEBUG: Python function do_rust_gen_targets finished
(b) Locals at rust_gen_target for aarch64::
tspec['data-layout'] = aarch64-unknown-linux-gnu, Type of tspec['data-layout'] = <class 'str'>
tspec['max-atomic-width'] = 128, Type of tspec['max-atomic-width'] = <class 'int'>
Reason for changing arm-eabi to arm: The earlier changes introduced this bug, so reverting the change 'arm-eabi' to 'arm' fixed the issue.
The 'rust_gen_targets' Task added with its dependent variable list.
Signed-off-by: Sundeep KOKKONDA <sundeep.kokkonda@...>
---
meta/recipes-devtools/rust/rust-common.inc | 16 +++++++++-------
1 file changed, 9 insertions(+), 7 deletions(-)
diff --git a/meta/recipes-devtools/rust/rust-common.inc b/meta/recipes-devtools/rust/rust-common.inc
index 310aecef22..984fe9099e 100644
--- a/meta/recipes-devtools/rust/rust-common.inc
+++ b/meta/recipes-devtools/rust/rust-common.inc
@@ -119,13 +119,13 @@ def llvm_features(d):
## arm-unknown-linux-gnueabihf
-DATA_LAYOUT[arm-eabi] = "e-m:e-p:32:32-i64:64-v128:64:128-a:0:32-n32-S64"
-LLVM_TARGET[arm-eabi] = "${RUST_TARGET_SYS}"
-TARGET_ENDIAN[arm-eabi] = "little"
-TARGET_POINTER_WIDTH[arm-eabi] = "32"
-TARGET_C_INT_WIDTH[arm-eabi] = "32"
-MAX_ATOMIC_WIDTH[arm-eabi] = "64"
-FEATURES[arm-eabi] = "+v6,+vfp2"
+DATA_LAYOUT[arm] = "e-m:e-p:32:32-i64:64-v128:64:128-a:0:32-n32-S64"
+LLVM_TARGET[arm] = "${RUST_TARGET_SYS}"
+TARGET_ENDIAN[arm] = "little"
+TARGET_POINTER_WIDTH[arm] = "32"
+TARGET_C_INT_WIDTH[arm] = "32"
+MAX_ATOMIC_WIDTH[arm] = "64"
+FEATURES[arm] = "+v6,+vfp2"
## armv7-unknown-linux-gnueabihf
DATA_LAYOUT[armv7-eabi] = "e-m:e-p:32:32-i64:64-v128:64:128-a:0:32-n32-S64"
@@ -360,6 +360,8 @@ def rust_gen_target(d, thing, wd, features, cpu, arch, abi=""):
with open(wd + sys + '.json', 'w') as f:
json.dump(tspec, f, indent=4)
+do_rust_gen_targets[vardeps] += "DATA_LAYOUT LLVM_TARGET TARGET_ENDIAN TARGET_POINTER_WIDTH TARGET_C_INT_WIDTH MAX_ATOMIC_WIDTH FEATURES"
+
python do_rust_gen_targets () {
wd = d.getVar('WORKDIR') + '/targets/'
build_arch = d.getVar('BUILD_ARCH')
We need to split this into two. One patch to fix the determinism issue,
the other to fix the arm bug.
The "good" news is that this patch now fails consistently on the
autobuilder, for example:
https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/3522
sstatetests.SStateTests.test_sstate_allarch_samesigs is failing which
you can rerun with:
oe-selftest -r sstatetests.SStateTests.test_sstate_allarch_samesigs
Now it fails reliably, we need to work out why the arm change breaks
that.
If you split the patches into two we can likely merge the
reproducibility one, then onwards with the arm issue.
In case it isn't clear, the failing signatures are for adwaita-icon-
theme which depends on librsvg-native which depends on rust-native.
What this means is that the signature of rust-native is changing when
MACHINE is changed in that test. This shouldn't happen. We already knew
that from our other discussion but now we can prove it. It would be
nice if a test showed us the native hash was changing and that is the
other issue we discussed but one step at a time!
To be clear, rust-native (and it's signature) should be independent of
MACHINE so we need to fix that. This means that currently rust-native
varies and rebuilds when you change MACHINE (after the dependency fix
in this patch).
Hello Richard,
I analyzed the test failure and it is because of the newly added
dependency on variable LLVM_TARGET. In think the patch split will
not help here. We need to understand the reason for the introduced
change in first place from 'arm' to 'arm-eabi' & why the
rust_gen_target is not added dependency on these variables.
Hello Alex,
We need a few inputs from you regarding a 'arm' bug which is in
discussion in this patch.
The changes in file meta/recipes-devtools/rust/rust-common.inc
with commit ID 'd6b563710e6cc0857843433d85023d47f9f2037d'
are causing the build error with 'arm' architecture (the tspec
variables returning 'NoneType') when 'TOOLCHAIN_HOST_TASK:append
= " packagegroup-rust-cross-canadian-${MACHINE}" added in local.conf
(See the detailed build error at beginning of this email).
And to fix that error the changes for 'arm' architecture
are reverted (#arm-unknown-linux-gnueabihf variables reverted to 'arm'
from 'arm-eabi') and the build error is solved. But,
because no Task is depending on these variables, bitbake is
unable to detect these changes while rebuilding (In other words,
the Sstate-cache sigdata is not changed). So, the rust_gen_target
task dependency to these variables are added and then the Sstate
sigdata is updated with rebuild. However, the dependency (on
variable LLVM_TARGET) is causing the failure in oe-selftest
test_sstate_allarch_samesigs testcase.
Can you let us know is it the right thing to add these variable
dependencies for rust_gen_target task? And, to fix the 'arm
NoneType' error is it the correct fix to revert the 'arm-eabi'
to 'arm'?
Let me know if anything is not clear.
Below is the dependency list taken from bitbake-diffsigs
from the test_sstate_allarch_samesigs test sigdata for
x86_64 & qemuarm machines. The change in sigdata caused by
differed values for variable TUNE_FEATURES{callconvention-hard}.
=================
...
List of dependencies for variable LLVM_TARGET[aarch64] is
['RUST_TARGET_SYS']
List of dependencies for variable LLVM_TARGET[arm] is
['RUST_TARGET_SYS']
...
List of dependencies for variable RUST_TARGET_SYS is
['rust_base_triple']
...
List of dependencies for variable rust_base_triple is
['determine_libc', 'target_is_armv7']
...
List of dependencies for variable determine_libc is ['BUILD_SYS',
'HOST_SYS', 'RUST_LIBC', 'RUST_LIBC:class-native'] # The
determine_libc is using the TUNE_FEATURES
List of dependencies for variable target_is_armv7 is []
...
TUNE_FEATURES{callconvention-hard} = Unset #in x86_64
TUNE_FEATURES{callconvention-hard} = Set #in qemuarm
...
====================
Thanks,
Sundeep K.
Cheers,
Richard
|
|
|
Hello Sundeep, this looks like something that requires a bit of time to set up, reproduce and understand, and unfortunately I do not have the time this week or the next. If you need my assistance you have to wait. Alex On Thu, 19 May 2022 at 13:12, Sundeep KOKKONDA <sundeep.kokkonda@...> wrote:
Hello,
On 14-05-2022 12:46, richard.purdie@... wrote:
On Thu, 2022-05-12 at 22:59 -0700, Sundeep KOKKONDA wrote:
[Yocto bug #14742]
The build shows below error while building for arm machines.
Exception: TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'
Detailed error info :
Steps to reproduce:
1. Set MACHINE ?= "qemuarm" in local.conf & add 'TOOLCHAIN_HOST_TASK:append = " packagegroup-rust-cross-canadian-${MACHINE}"'
2. bitbake core-image-minimal -cpopulate_sdk
Complete Error:
ERROR: rust-cross-canadian-arm-1.59.0-r0 do_rust_gen_targets: Error executing a python function in exec_func_python() autogenerated:
The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_func_python() autogenerated', lineno: 2, function: <module>
0001:
*** 0002:do_rust_gen_targets(d)
0003:
File: '/ala-lpggp31/skokkonda/yocto/poky/meta/recipes-devtools/rust/rust-cross-canadian-common.inc', lineno: 31, function: do_rust_gen_targets
0027:
0028:LLVM_TARGET[x86_64] = "${RUST_HOST_SYS}"
0029:python do_rust_gen_targets () {
0030: wd = d.getVar('WORKDIR') + '/targets/'
*** 0031: rust_gen_target(d, 'TARGET', wd, d.getVar('TARGET_LLVM_FEATURES') or "", d.getVar('TARGET_LLVM_CPU'), d.getVar('TARGET_ARCH'))
0032: rust_gen_target(d, 'HOST', wd, "", "generic", d.getVar('HOST_ARCH'))
0033: rust_gen_target(d, 'BUILD', wd, "", "generic", d.getVar('BUILD_ARCH'))
0034:}
0035:
File: '/ala-lpggp31/skokkonda/yocto/poky/meta/recipes-devtools/rust/rust-common.inc', lineno: 330, function: rust_gen_target
0326: # build tspec
0327: tspec = {}
0328: tspec['llvm-target'] = d.getVarFlag('LLVM_TARGET', arch_abi)
0329: tspec['data-layout'] = d.getVarFlag('DATA_LAYOUT', arch_abi)
*** 0330: tspec['max-atomic-width'] = int(d.getVarFlag('MAX_ATOMIC_WIDTH', arch_abi))
0331: tspec['target-pointer-width'] = d.getVarFlag('TARGET_POINTER_WIDTH', arch_abi)
0332: tspec['target-c-int-width'] = d.getVarFlag('TARGET_C_INT_WIDTH', arch_abi)
0333: tspec['target-endian'] = d.getVarFlag('TARGET_ENDIAN', arch_abi)
0334: tspec['arch'] = arch_to_rust_target_arch(rust_arch)
Exception: TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'
Below are the local variables from rust_gen_target function for arm and aarch64 targets. Refer below, the tspec varibles for 'arm' generated with NoneType.
(a) Locals at rust_gen_target for arm::
tspec['data-layout'] = None, Type of tspec['data-layout'] = <class 'NoneType'>
tspec['data-layout'] = None, Type of tspec['data-layout'] = <class 'NoneType'>
DEBUG: Python function do_rust_gen_targets finished
(b) Locals at rust_gen_target for aarch64::
tspec['data-layout'] = aarch64-unknown-linux-gnu, Type of tspec['data-layout'] = <class 'str'>
tspec['max-atomic-width'] = 128, Type of tspec['max-atomic-width'] = <class 'int'>
Reason for changing arm-eabi to arm: The earlier changes introduced this bug, so reverting the change 'arm-eabi' to 'arm' fixed the issue.
The 'rust_gen_targets' Task added with its dependent variable list.
Signed-off-by: Sundeep KOKKONDA <sundeep.kokkonda@...>
---
meta/recipes-devtools/rust/rust-common.inc | 16 +++++++++-------
1 file changed, 9 insertions(+), 7 deletions(-)
diff --git a/meta/recipes-devtools/rust/rust-common.inc b/meta/recipes-devtools/rust/rust-common.inc
index 310aecef22..984fe9099e 100644
--- a/meta/recipes-devtools/rust/rust-common.inc
+++ b/meta/recipes-devtools/rust/rust-common.inc
@@ -119,13 +119,13 @@ def llvm_features(d):
## arm-unknown-linux-gnueabihf
-DATA_LAYOUT[arm-eabi] = "e-m:e-p:32:32-i64:64-v128:64:128-a:0:32-n32-S64"
-LLVM_TARGET[arm-eabi] = "${RUST_TARGET_SYS}"
-TARGET_ENDIAN[arm-eabi] = "little"
-TARGET_POINTER_WIDTH[arm-eabi] = "32"
-TARGET_C_INT_WIDTH[arm-eabi] = "32"
-MAX_ATOMIC_WIDTH[arm-eabi] = "64"
-FEATURES[arm-eabi] = "+v6,+vfp2"
+DATA_LAYOUT[arm] = "e-m:e-p:32:32-i64:64-v128:64:128-a:0:32-n32-S64"
+LLVM_TARGET[arm] = "${RUST_TARGET_SYS}"
+TARGET_ENDIAN[arm] = "little"
+TARGET_POINTER_WIDTH[arm] = "32"
+TARGET_C_INT_WIDTH[arm] = "32"
+MAX_ATOMIC_WIDTH[arm] = "64"
+FEATURES[arm] = "+v6,+vfp2"
## armv7-unknown-linux-gnueabihf
DATA_LAYOUT[armv7-eabi] = "e-m:e-p:32:32-i64:64-v128:64:128-a:0:32-n32-S64"
@@ -360,6 +360,8 @@ def rust_gen_target(d, thing, wd, features, cpu, arch, abi=""):
with open(wd + sys + '.json', 'w') as f:
json.dump(tspec, f, indent=4)
+do_rust_gen_targets[vardeps] += "DATA_LAYOUT LLVM_TARGET TARGET_ENDIAN TARGET_POINTER_WIDTH TARGET_C_INT_WIDTH MAX_ATOMIC_WIDTH FEATURES"
+
python do_rust_gen_targets () {
wd = d.getVar('WORKDIR') + '/targets/'
build_arch = d.getVar('BUILD_ARCH')
We need to split this into two. One patch to fix the determinism issue,
the other to fix the arm bug.
The "good" news is that this patch now fails consistently on the
autobuilder, for example:
https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/3522
sstatetests.SStateTests.test_sstate_allarch_samesigs is failing which
you can rerun with:
oe-selftest -r sstatetests.SStateTests.test_sstate_allarch_samesigs
Now it fails reliably, we need to work out why the arm change breaks
that.
If you split the patches into two we can likely merge the
reproducibility one, then onwards with the arm issue.
In case it isn't clear, the failing signatures are for adwaita-icon-
theme which depends on librsvg-native which depends on rust-native.
What this means is that the signature of rust-native is changing when
MACHINE is changed in that test. This shouldn't happen. We already knew
that from our other discussion but now we can prove it. It would be
nice if a test showed us the native hash was changing and that is the
other issue we discussed but one step at a time!
To be clear, rust-native (and it's signature) should be independent of
MACHINE so we need to fix that. This means that currently rust-native
varies and rebuilds when you change MACHINE (after the dependency fix
in this patch).
Hello Richard,
I analyzed the test failure and it is because of the newly added dependency on variable LLVM_TARGET. In think the patch split will not help here. We need to understand the reason for the introduced change in first place from 'arm' to 'arm-eabi' & why the rust_gen_target is not added dependency on these variables.
Hello Alex,
We need a few inputs from you regarding a 'arm' bug which is in discussion in this patch.
The changes in file meta/recipes-devtools/rust/rust-common.inc with commit ID 'd6b563710e6cc0857843433d85023d47f9f2037d' are causing the build error with 'arm' architecture (the tspec variables returning 'NoneType') when 'TOOLCHAIN_HOST_TASK:append = " packagegroup-rust-cross-canadian-${MACHINE}" added in local.conf (See the detailed build error at beginning of this email).
And to fix that error the changes for 'arm' architecture are reverted (#arm-unknown-linux-gnueabihf variables reverted to 'arm' from 'arm-eabi') and the build error is solved. But, because no Task is depending on these variables, bitbake is unable to detect these changes while rebuilding (In other words, the Sstate-cache sigdata is not changed). So, the rust_gen_target task dependency to these variables are added and then the Sstate sigdata is updated with rebuild. However, the dependency (on variable LLVM_TARGET) is causing the failure in oe-selftest test_sstate_allarch_samesigs testcase.
Can you let us know is it the right thing to add these variable dependencies for rust_gen_target task? And, to fix the 'arm NoneType' error is it the correct fix to revert the 'arm-eabi' to 'arm'?
Let me know if anything is not clear.
Below is the dependency list taken from bitbake-diffsigs from the test_sstate_allarch_samesigs test sigdata for x86_64 & qemuarm machines. The change in sigdata caused by differed values for variable TUNE_FEATURES{callconvention-hard}.
=================
...
List of dependencies for variable LLVM_TARGET[aarch64] is ['RUST_TARGET_SYS']
List of dependencies for variable LLVM_TARGET[arm] is ['RUST_TARGET_SYS']
...
List of dependencies for variable RUST_TARGET_SYS is ['rust_base_triple']
...
List of dependencies for variable rust_base_triple is ['determine_libc', 'target_is_armv7']
...
List of dependencies for variable determine_libc is ['BUILD_SYS', 'HOST_SYS', 'RUST_LIBC', 'RUST_LIBC:class-native'] # The determine_libc is using the TUNE_FEATURES
List of dependencies for variable target_is_armv7 is []
...
TUNE_FEATURES{callconvention-hard} = Unset #in x86_64
TUNE_FEATURES{callconvention-hard} = Set #in qemuarm
...
====================
Thanks,
Sundeep K.
Cheers,
Richard
|
|
|
Let me jump in and explain a bit about what I think we need to do here
to move forward.
With the patch in this series applied, "oe-selftest -r
sstatetests.SStateTests.test_sstate_allarch_samesigs" fails. Looking at
the test, it sets two different MACHINE values, "qemuarm" and "qemux86-
64". Lets see if we can get more information about what is going on.
With MACHINE = "qemux86-64", I run:
bitbake adwaita-icon-theme rust-native cargo-native -S none
(since adwaita-icon-theme is what is changing when it shouldn't and we
suspect a problem with rust-native and cargo-native).
This generates a locked-sigs.inc, lets move that to a different
location:
mv locked-sigs.inc locked-sigs2.inc
Now with MACHINE = "qemuarm", I run:
bitbake adwaita-icon-theme rust-native cargo-native -S none
and we get another locked-sigs.inc.
Comparing locked-sigs.inc with locked-sigs2.inc is revealing. I like:
meld locked-sigs.inc locked-sigs2.inc
but you could use diff too or whatever way you prefer to view
differences.
adwaita-icon-theme is changing, there are a load of changes from
SIGGEN_LOCKEDSIGS_t-core2-64 to SIGGEN_LOCKEDSIGS_t-cortexa15t2hf-neon
which isn't surprising. You can see binutils-cross-arm becoming
binutils-cross-x86_64 which again, isn't surprising. cargo-native
changes which is a worry. rust-native also changes and is the likely
cause of our issues, it is furthest down the dependency stack. The
tasks which change are:
rust-native:do_build
rust-native:do_compile
rust-native:do_install
rust-native:do_populate_sysroot
rust-native:do_rust_gen_targets
and since the patch is changing do_rust_gen_targets, that is likely our
problem.
Now we've proven that the signature of that task changes between these
two different configs, we need to investigate what the issue is there
and bitbake-diffsigs can likely help with that. The idea is that the
native recipes should be independent of the target machine choice so
that is the issue which now needs to be fixed.
I did notice that if you compare qemux86-64 with qemuarm64, you don't
see this issue, it only happens with qemuarm. This probably means that
some 32 vs 64 bit issue is happening in rust-native.
Note I did all of this without actually building anything, just some
recipe parses to dump the signatures.
Does that help move things forward with the issue?
Cheers,
Richard
|
|
|
On Fri, 2022-05-20 at 10:56 +0100, Richard Purdie via lists.openembedded.org wrote: Let me jump in and explain a bit about what I think we need to do here
to move forward.
With the patch in this series applied, "oe-selftest -r
sstatetests.SStateTests.test_sstate_allarch_samesigs" fails. Looking at
the test, it sets two different MACHINE values, "qemuarm" and "qemux86-
64". Lets see if we can get more information about what is going on.
With MACHINE = "qemux86-64", I run:
bitbake adwaita-icon-theme rust-native cargo-native -S none
(since adwaita-icon-theme is what is changing when it shouldn't and we
suspect a problem with rust-native and cargo-native).
This generates a locked-sigs.inc, lets move that to a different
location:
mv locked-sigs.inc locked-sigs2.inc
Now with MACHINE = "qemuarm", I run:
bitbake adwaita-icon-theme rust-native cargo-native -S none
and we get another locked-sigs.inc.
Comparing locked-sigs.inc with locked-sigs2.inc is revealing. I like:
meld locked-sigs.inc locked-sigs2.inc
but you could use diff too or whatever way you prefer to view
differences.
adwaita-icon-theme is changing, there are a load of changes from
SIGGEN_LOCKEDSIGS_t-core2-64 to SIGGEN_LOCKEDSIGS_t-cortexa15t2hf-neon
which isn't surprising. You can see binutils-cross-arm becoming
binutils-cross-x86_64 which again, isn't surprising. cargo-native
changes which is a worry. rust-native also changes and is the likely
cause of our issues, it is furthest down the dependency stack. The
tasks which change are:
rust-native:do_build
rust-native:do_compile
rust-native:do_install
rust-native:do_populate_sysroot
rust-native:do_rust_gen_targets
and since the patch is changing do_rust_gen_targets, that is likely our
problem.
Now we've proven that the signature of that task changes between these
two different configs, we need to investigate what the issue is there
and bitbake-diffsigs can likely help with that. The idea is that the
native recipes should be independent of the target machine choice so
that is the issue which now needs to be fixed.
I did notice that if you compare qemux86-64 with qemuarm64, you don't
see this issue, it only happens with qemuarm. This probably means that
some 32 vs 64 bit issue is happening in rust-native.
Note I did all of this without actually building anything, just some
recipe parses to dump the signatures.
Does that help move things forward with the issue? Since I now had the environment, I help but complete this (taking the hashes from the difference between the locked-sigs diff): $ bitbake-diffsigs tmp/stamps/x86_64-linux/rust-native/1.60.0-r0.do_rust_gen_targets.sigdata.63* tmp/stamps/x86_64-linux/rust-native/1.60.0-r0.do_rust_gen_targets.sigdata.34* NOTE: Starting bitbake server... basehash changed from 860b8f11b10182dc5b2737f62cdb697477f714adb63eeb4d4b932d67cac8eec2 to 9379e8b9df9696e8056fec7d1534661f34dda073f6d816e241b09a2dff76ae2d Variable rust_base_triple value changed: @@ -36,4 +36,4 @@ # In some cases uname and the toolchain differ on their idea of the arch name -TUNE_FEATURES{callconvention-hard} = Set +TUNE_FEATURES{callconvention-hard} = Unset The question now becomes how to fix that since I doubt rust-native really does depend on that. Cheers, Richard
|
|
|
On Fri, 2022-05-20 at 12:04 +0100, Richard Purdie via lists.openembedded.org wrote: On Fri, 2022-05-20 at 10:56 +0100, Richard Purdie via
lists.openembedded.org wrote:
Let me jump in and explain a bit about what I think we need to do here
to move forward.
With the patch in this series applied, "oe-selftest -r
sstatetests.SStateTests.test_sstate_allarch_samesigs" fails. Looking at
the test, it sets two different MACHINE values, "qemuarm" and "qemux86-
64". Lets see if we can get more information about what is going on.
With MACHINE = "qemux86-64", I run:
bitbake adwaita-icon-theme rust-native cargo-native -S none
(since adwaita-icon-theme is what is changing when it shouldn't and we
suspect a problem with rust-native and cargo-native).
This generates a locked-sigs.inc, lets move that to a different
location:
mv locked-sigs.inc locked-sigs2.inc
Now with MACHINE = "qemuarm", I run:
bitbake adwaita-icon-theme rust-native cargo-native -S none
and we get another locked-sigs.inc.
Comparing locked-sigs.inc with locked-sigs2.inc is revealing. I like:
meld locked-sigs.inc locked-sigs2.inc
but you could use diff too or whatever way you prefer to view
differences.
adwaita-icon-theme is changing, there are a load of changes from
SIGGEN_LOCKEDSIGS_t-core2-64 to SIGGEN_LOCKEDSIGS_t-cortexa15t2hf-neon
which isn't surprising. You can see binutils-cross-arm becoming
binutils-cross-x86_64 which again, isn't surprising. cargo-native
changes which is a worry. rust-native also changes and is the likely
cause of our issues, it is furthest down the dependency stack. The
tasks which change are:
rust-native:do_build
rust-native:do_compile
rust-native:do_install
rust-native:do_populate_sysroot
rust-native:do_rust_gen_targets
and since the patch is changing do_rust_gen_targets, that is likely our
problem.
Now we've proven that the signature of that task changes between these
two different configs, we need to investigate what the issue is there
and bitbake-diffsigs can likely help with that. The idea is that the
native recipes should be independent of the target machine choice so
that is the issue which now needs to be fixed.
I did notice that if you compare qemux86-64 with qemuarm64, you don't
see this issue, it only happens with qemuarm. This probably means that
some 32 vs 64 bit issue is happening in rust-native.
Note I did all of this without actually building anything, just some
recipe parses to dump the signatures.
Does that help move things forward with the issue? Since I now had the environment, I help but complete this (taking the
hashes from the difference between the locked-sigs diff):
$ bitbake-diffsigs tmp/stamps/x86_64-linux/rust-native/1.60.0-r0.do_rust_gen_targets.sigdata.63* tmp/stamps/x86_64-linux/rust-native/1.60.0-r0.do_rust_gen_targets.sigdata.34*
NOTE: Starting bitbake server...
basehash changed from 860b8f11b10182dc5b2737f62cdb697477f714adb63eeb4d4b932d67cac8eec2 to 9379e8b9df9696e8056fec7d1534661f34dda073f6d816e241b09a2dff76ae2d
Variable rust_base_triple value changed:
@@ -36,4 +36,4 @@
# In some cases uname and the toolchain differ on their idea of the arch name
-TUNE_FEATURES{callconvention-hard} = Set
+TUNE_FEATURES{callconvention-hard} = Unset
The question now becomes how to fix that since I doubt rust-native
really does depend on that. The fix is probably this: diff --git a/meta/classes/rust-common.bbclass b/meta/classes/rust-common.bbclass index 02a538258af..cb811ac5da7 100644 --- a/meta/classes/rust-common.bbclass +++ b/meta/classes/rust-common.bbclass @@ -117,8 +117,11 @@ RUST_BUILD_ARCH = "${@oe.rust.arch_to_rust_arch(d.getVar('BUILD_ARCH'))}" # its likely best to not use the triple suffix due to potential confusion. RUST_BUILD_SYS = "${@rust_base_triple(d, 'BUILD')}" +RUST_BUILD_SYS[vardepvalue] = "${RUST_BUILD_SYS}" RUST_HOST_SYS = "${@rust_base_triple(d, 'HOST')}" +RUST_HOST_SYS[vardepvalue] = "${RUST_HOST_SYS}" RUST_TARGET_SYS = "${@rust_base_triple(d, 'TARGET')}" +RUST_TARGET_SYS[vardepvalue] = "${RUST_TARGET_SYS}" # wrappers to get around the fact that Rust needs a single # binary but Yocto's compiler and linker commands have which makes the sstate signatures depend on the determined values rather than the ingredients in them (which is usually what we want but not here). Cheers, Richard
|
|
|
Hello Richard,
The patch you sent in rust-common.bbclass fixing the 'oe-selftest -r sstatetests.SStateTests.test_sstate_allarch_samesigs' test issue but the regression test shows that the 'sstatetests.SStateTests.test_sstate_32_64_same_hash' is getting failed. So, I think this change is not correct. With this patch, the diffsigs shows 'rust-native' sigdata is changed based on the changed value for RUST_TARGET_SYS variable (in x86_64 :: x86_64-unknown-linux-gnu & in i686 :: i686-unknown-linux-gnu). I tried several attempts to fix it but nothing worked out yet.
@Alex.
Did you get sometime to look into this 'arm' issue introduced with your commit?
Thanks,
Sundeep K.
|
|
|
On Tue, 2022-05-24 at 03:21 -0700, Sundeep KOKKONDA wrote: Hello Richard,
The patch you sent in rust-common.bbclass fixing the 'oe-selftest -r
sstatetests.SStateTests.test_sstate_allarch_samesigs' test issue but
the regression test shows that the
'sstatetests.SStateTests.test_sstate_32_64_same_hash' is getting
failed. So, I think this change is not correct. With this patch, the
diffsigs shows 'rust-native' sigdata is changed based on the changed
value for RUST_TARGET_SYS variable (in x86_64 :: x86_64-unknown-
linux-gnu & in i686 :: i686-unknown-linux-gnu). I tried several
attempts to fix it but nothing worked out yet. You should have seen a further couple of patches which fixed other issues and with current master, the issues should be resolved? Cheers, Richard
|
|
|
On 24-05-2022 16:34, richard.purdie@... wrote: On Tue, 2022-05-24 at 03:21 -0700, Sundeep KOKKONDA wrote:
Hello Richard,
The patch you sent in rust-common.bbclass fixing the 'oe-selftest -r
sstatetests.SStateTests.test_sstate_allarch_samesigs' test issue but
the regression test shows that the
'sstatetests.SStateTests.test_sstate_32_64_same_hash' is getting
failed. So, I think this change is not correct. With this patch, the
diffsigs shows 'rust-native' sigdata is changed based on the changed
value for RUST_TARGET_SYS variable (in x86_64 :: x86_64-unknown-
linux-gnu & in i686 :: i686-unknown-linux-gnu). I tried several
attempts to fix it but nothing worked out yet. You should have seen a further couple of patches which fixed other
issues and with current master, the issues should be resolved? I used old sources. Thanks for info. Issue is fixed. Thanks, Sundeep K.
Cheers,
Richard
|
|
|
