Re: [PATCH 6/7] native: Stop clearing PACKAGES


Martin Jansa
 

On Sun, Jan 31, 2021 at 10:34:48AM +0000, Richard Purdie wrote:
On Sun, 2021-01-31 at 10:55 +0100, Martin Jansa wrote:
On Sun, Jan 31, 2021 at 09:07:50AM +0000, Richard Purdie wrote:
On Sat, 2021-01-30 at 17:15 +0100, Martin Jansa wrote:
I'm asking because meta-python2 has the same issue in:
https://git.openembedded.org/meta-python2/tree/recipes-devtools/python/python-setuptools.inc#n49

and my fix I was planing to send was to replace it with:
RDEPENDS_${PN}_append_class-target = " ${PYTHON_PN}-pkg-resources"
and drop the RPROVIDES, because it unfortunately causes bitbake to
get stuck after reporting parsing error as:

ERROR: meta-python2/recipes-devtools/python/python-
setuptools_42.0.2.bb: QA Issue: meta-python2/recipes-
devtools/python/python-setuptools_42.0.2.bb: Variable RPROVIDES is
set as not being package specific, please fix this. [pkgvarcheck]
ERROR: meta-python2/recipes-devtools/python/python-
setuptools_42.0.2.bb: Fatal QA errors found, failing task.
ERROR: Failed to parse recipe: meta-python2/recipes-
devtools/python/python-setuptools_42.0.2.bb
Are you saying that it does that with the RPROVIDES removed? That seems
odd :/
No, this was before removing RPROVIDES in meta-python2, I was just
surprised that bitbake got stuck after reporting parsing failure, I'm
seeing bitbake getting stuck in dunfell builds quite often, but don't
remember seeing it with gatesgarth and newer (and was believing that you
have already fixed most if not all cases there). But it's also true that
our jenkins runs 1000x more builds based on dunfell, so it's much more
likely to happen there.
Its the first time I've heard a report of that and it sounds a little
worrying. What you say stuck, bitbake hangs? Is it using cpu? A process
tree of what it looks like when stuck would be interesting if it is
hanging and maybe the last things in any running or just finished task
logs. It could be some issue with the way a failing task is handled?

We've seen a lot of weird issues on the autobuilder but not that. Could
you have something enabled in your setup for logging/reporting whch
wouldn't be in ours and could be causing it?
It doesn't seem to happen very often and I guess autobuilder doesn't
include meta-python2 (which is currently the only reproducer in my list
of layers).

I've tried to reproduce it with just 3 layers included:
BBLAYERS = " \
/OE/build/oe-core/meta-python2 \
/OE/build/oe-core/meta-openembedded/meta-oe \
/OE/build/oe-core/openembedded-core/meta \
"

and it happened on 3rd try (cache, tmpdir, sstate-cache were removed
only before 1st try)

martin@jama:/OE/build/oe-core$ bitbake -k python-native
Loading cache: 100% | | ETA: --:--:--
Loaded 0 entries from dependency cache.
ERROR: /OE/build/oe-core/meta-python2/recipes-devtools/python/python-setuptools_42.0.2.bb: QA Issue: /OE/build/oe-core/meta-python2/recipes-devtools/python/python-setuptools_42.0.2.bb: Variable RPROVIDES is set as not being package specific, please fix this. [pkgvarcheck]
ERROR: /OE/build/oe-core/meta-python2/recipes-devtools/python/python-setuptools_42.0.2.bb: Fatal QA errors found, failing task.
ERROR: Failed to parse recipe: /OE/build/oe-core/meta-python2/recipes-devtools/python/python-setuptools_42.0.2.bb | ETA: 0:00:14

Summary: There were 3 ERROR messages shown, returning a non-zero exit code.
martin@jama:/OE/build/oe-core$ bitbake -k python-native
NOTE: Reconnecting to bitbake server...
Loading cache: 100% |###################################################################################################################################################################################################################################| Time: 0:00:00
Loaded 845 entries from dependency cache.
ERROR: /OE/build/oe-core/meta-python2/recipes-devtools/python/python-native_2.7.18.bb: Error executing a python function in <code>: | ETA: 0:01:10

The stack trace of python calls that resulted in this exception/failure was:
File: '<code>', lineno: 14, function: <module>
0010:__anon_35__OE_build_oe_core_openembedded_core_meta_classes_devshell_bbclass(d)
0011:__anon_151__OE_build_oe_core_openembedded_core_meta_classes_sstate_bbclass(d)
0012:__anon_20__OE_build_oe_core_openembedded_core_meta_classes_blacklist_bbclass(d)
0013:__anon_177__OE_build_oe_core_openembedded_core_meta_classes_siteinfo_bbclass(d)
*** 0014:__anon_90__OE_build_oe_core_meta_python2_recipes_devtools_python_python_native_2_7_18_bb(d)
File: '/OE/build/oe-core/meta-python2/recipes-devtools/python/python-native_2.7.18.bb', lineno: 76, function: __anon_90__OE_build_oe_core_meta_python2_recipes_devtools_python_python_native_2_7_18_bb
0072: manifest_file.seek(json_start)
0073: manifest_str = manifest_file.read()
0074: python_manifest = json.loads(manifest_str)
0075:
*** 0076: rprovides = d.getVar('RPROVIDES').split()
0077:
0078: # Hardcoded since it cant be python-native-foo, should be python-foo-native
0079: pn = 'python'
0080:
Exception: AttributeError: 'NoneType' object has no attribute 'split'

ERROR: /OE/build/oe-core/meta-python2/recipes-devtools/python/python-setuptools_42.0.2.bb: QA Issue: /OE/build/oe-core/meta-python2/recipes-devtools/python/python-setuptools_42.0.2.bb: Variable RPROVIDES is set as not being package specific, please fix this. [pkgvarcheck]
ERROR: /OE/build/oe-core/meta-python2/recipes-devtools/python/python-setuptools_42.0.2.bb: Fatal QA errors found, failing task.
ERROR: Failed to parse recipe: /OE/build/oe-core/meta-python2/recipes-devtools/python/python-native_2.7.18.bb | ETA: 0:00:30

Summary: There were 4 ERROR messages shown, returning a non-zero exit code.
martin@jama:/OE/build/oe-core$ bitbake -k python-native
NOTE: Reconnecting to bitbake server...
Loading cache: 100% |###################################################################################################################################################################################################################################| Time: 0:00:00
Loaded 1289 entries from dependency cache.
ERROR: /OE/build/oe-core/meta-python2/recipes-devtools/python/python-setuptools_42.0.2.bb: QA Issue: /OE/build/oe-core/meta-python2/recipes-devtools/python/python-setuptools_42.0.2.bb: Variable RPROVIDES is set as not being package specific, please fix this. [pkgvarcheck]
ERROR: /OE/build/oe-core/meta-python2/recipes-devtools/python/python-setuptools_42.0.2.bb: Fatal QA errors found, failing task.
ERROR: Failed to parse recipe: /OE/build/oe-core/meta-python2/recipes-devtools/python/python-setuptools_42.0.2.bb

and here it's stuck for around 1 hour, bitbake using very little cpu.

martin@jama:~$ ps aux | grep bitbake
martin 372784 0.0 0.0 11552 884 pts/1 S+ 13:43 0:00 grep --color=auto bitbake
martin 928962 0.1 0.1 219204 133704 ? Sl 12:42 0:04 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 943698 0.0 0.0 119900 29928 pts/8 Sl+ 12:43 0:01 python3 /OE/build/oe-core/bitbake/bin/bitbake -k python-native
martin 945507 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 945528 0.0 0.0 214364 124740 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 945532 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 945563 0.0 0.0 214364 124732 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 945577 0.0 0.0 214364 124740 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 945621 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 945632 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 945643 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 945653 0.0 0.0 214364 124740 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 945669 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 945685 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 945703 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 945736 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 945753 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 945774 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 945778 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 945819 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 945839 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 945860 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 945878 0.0 0.0 214364 124740 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 945894 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 945909 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 945915 0.0 0.0 214364 124804 ? Sl 12:43 0:00 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 945929 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 945950 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 945953 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 945964 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 945966 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 945989 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946005 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946016 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946037 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946038 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946064 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946068 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946080 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946103 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946109 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946122 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946135 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946137 0.0 0.0 214364 124736 ? Sl 12:43 0:00 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946152 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946167 0.0 0.0 214364 124736 ? Sl 12:43 0:00 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946168 0.0 0.0 214364 124736 ? Sl 12:43 0:00 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946185 0.0 0.0 214364 124736 ? Sl 12:43 0:00 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946197 0.0 0.0 214364 124736 ? Sl 12:43 0:00 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946214 0.0 0.0 214364 124736 ? Sl 12:43 0:00 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946228 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946243 0.0 0.0 214364 124728 ? Sl 12:43 0:00 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946249 0.0 0.0 214364 124736 ? Sl 12:43 0:00 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946265 0.0 0.0 214364 124736 ? Sl 12:43 0:00 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946298 0.0 0.0 214364 124740 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946345 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946364 0.0 0.0 214364 124736 ? Sl 12:43 0:00 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946386 0.0 0.0 214364 124736 ? Sl 12:43 0:00 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946389 0.0 0.0 214364 124736 ? Sl 12:43 0:00 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946405 0.0 0.0 214364 124736 ? Sl 12:43 0:00 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946410 0.0 0.0 214364 124736 ? Sl 12:43 0:00 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946442 0.0 0.0 214364 124736 ? Sl 12:43 0:00 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946458 0.0 0.0 214364 124612 ? Sl 12:43 0:00 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946463 0.0 0.0 214364 124652 ? Sl 12:43 0:00 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946481 0.0 0.0 214364 124736 ? Sl 12:43 0:00 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946501 0.0 0.0 214364 124736 ? Sl 12:43 0:00 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946518 0.0 0.0 214364 124736 ? Sl 12:43 0:00 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0

martin@jama:~$ sudo strace -p 928962
strace: Process 928962 attached
wait4(945507,

martin@jama:~$ kill 928962

adds an warning to the stuck build:

martin@jama:/OE/build/oe-core$ bitbake -k python-native
NOTE: Reconnecting to bitbake server...
Loading cache: 100% |###################################################################################################################################################################################################################################| Time: 0:00:00
Loaded 1289 entries from dependency cache.
ERROR: /OE/build/oe-core/meta-python2/recipes-devtools/python/python-setuptools_42.0.2.bb: QA Issue: /OE/build/oe-core/meta-python2/recipes-devtools/python/python-setuptools_42.0.2.bb: Variable RPROVIDES is set as not being package specific, please fix this. [pkgvarcheck]
ERROR: /OE/build/oe-core/meta-python2/recipes-devtools/python/python-setuptools_42.0.2.bb: Fatal QA errors found, failing task.
ERROR: Failed to parse recipe: /OE/build/oe-core/meta-python2/recipes-devtools/python/python-setuptools_42.0.2.bb | ETA: 0:00:50
WARNING: Cooker received SIGTERM, shutting down...

killing the main process:
martin@jama:~$ kill 943698

terminates the stuck build, but all bitbake-server processes are left behind.

If I try to kill all remaining bitbake-server processes, then only one is left behind:
martin 928962 0.1 0.1 285004 137688 ? S 12:42 0:05 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin@jama:/OE/build/oe-core$ sudo strace -p 928962
strace: Process 928962 attached
select(7, [6], [], [], {tv_sec=0, tv_usec=40768}) = 0 (Timeout)
poll([{fd=7, events=POLLIN}], 1, 0) = 0 (Timeout)
poll([{fd=8, events=POLLIN}], 1, 0) = 0 (Timeout)
select(7, [6], [], [], {tv_sec=0, tv_usec=100000}) = 0 (Timeout)
poll([{fd=7, events=POLLIN}], 1, 0) = 0 (Timeout)
poll([{fd=8, events=POLLIN}], 1, 0) = 0 (Timeout)

and then it dies as well after a while.

Here is bitbake-cookerdaemon.log, but doesn't look very interesting to me:

928962 12:42:56.099747 --- Starting bitbake server pid 928962 at 2021-01-31 12:42:56.099713 ---
928962 12:42:56.110782 Started bitbake server pid 928962
928962 12:42:56.111010 Entering server connection loop
928962 12:42:56.111833 Accepting [<socket.socket fd=6, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0, laddr=bitbake.sock>] ([])
928962 12:42:56.112382 Processing Client
928962 12:42:56.112428 Connecting Client
928962 12:42:56.112733 Running command ['setFeatures', [2]]
928962 12:42:56.113448 Command Completed
928962 12:42:56.113566 Running command ['updateConfig', {'abort': False, 'force': False, 'invalidate_stamp': None, 'dry_run': False, 'dump_signatures': [], 'extra_assume_provided': [], 'profile': False, 'prefile': [], 'postfile': [], 'server_timeout': None, 'nosetscene': False, 'setsceneonly': False, 'skipsetscene': False, 'runall': None, 'runonly': None, 'writeeventlog': None, 'build_verbose_shell': False, 'build_verbose_stdout': False, 'default_loglevel': 20, 'debug_domains': {}}, {'SHELL': '/bin/bash', 'SSH_AUTH_SOCK': '/run/user/1000/keyring/ssh', 'SSH_AGENT_PID': '3080', 'PWD': '/OE/build/oe-core', 'LOGNAME': 'martin', 'HOME': '/home/martin', 'MACHINE': 'qemux86-64', 'BB_ENV_EXTRAWHITE': 'MACHINE DISTRO http_proxy ftp_proxy https_proxy all_proxy ALL_PROXY no_proxy SSH_AGENT_PID SSH_AUTH_SOCK BB_SRCREV_POLICY SDKMACHINE BB_NUMBER_THREADS GIT_PROXY_COMMAND PSEUDO_DISABLED PSEUDO_BUILD', 'USER': 'martin', 'LC_ALL': 'en_US.UTF-8', 'PATH': '/OE/build/oe-core/openembedded-core/scripts:/OE/build/oe-core/bitbake/bin:/home/martin/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin', 'SESSION_MANAGER': 'local/jama:@/tmp/.ICE-unix/2943,unix/jama:/tmp/.ICE-unix/2943', 'WINDOWID': '79691779', 'QT_ACCESSIBILITY': '1', 'COLORTERM': 'truecolor', 'XDG_CONFIG_DIRS': '/etc/xdg/xdg-xubuntu:/etc/xdg:/etc/xdg', 'XDG_SESSION_PATH': '/org/freedesktop/DisplayManager/Session0', 'XDG_MENU_PREFIX': 'xfce-', 'CLUTTER_BACKEND': 'x11', 'LANGUAGE': 'en_US', 'TERMCAP': 'SC|screen.xterm-256color|VT 100/ANSI X3.64 virtual terminal:DO=\\E[%dB:LE=\\E[%dD:RI=\\E[%dC:UP=\\E[%dA:bs:bt=\\E[Z:cd=\\E[J:ce=\\E[K:cl=\\E[H\\E[J:cm=\\E[%i%d;%dH:ct=\\E[3g:do=^J:nd=\\E[C:pt:rc=\\E8:rs=\\Ec:sc=\\E7:st=\\EH:up=\\EM:le=^H:bl=^G:cr=^M:it#8:ho=\\E[H:nw=\\EE:ta=^I:is=\\E)0:li#51:co#263:am:xn:xv:LP:sr=\\EM:al=\\E[L:AL=\\E[%dL:cs=\\E[%i%d;%dr:dl=\\E[M:DL=\\E[%dM:dc=\\E[P:DC=\\E[%dP:im=\\E[4h:ei=\\E[4l:mi:IC=\\E[%d@:ks=\\E[?1h\\E=:ke=\\E[?1l\\E>:vi=\\E[?25l:ve=\\E[34h\\E[?25h:vs=\\E[34l:ti=\\E[?1049h:te=\\E[?1049l:us=\\E[4m:ue=\\E[24m:so=\\E[3m:se=\\E[23m:mb=\\E[5m:md=\\E[1m:mh=\\E[2m:mr=\\E[7m:me=\\E[m:ms:Co#8:pa#64:AF=\\E[3%dm:AB=\\E[4%dm:op=\\E[39;49m:AX:vb=\\Eg:G0:as=\\E(0:ae=\\E(B:ac=\\140\\140aaffggjjkkllmmnnooppqqrrssttuuvvwwxxyyzz{{||}}~~..--++,,hhII00:po=\\E[5i:pf=\\E[4i:Km=\\E[<:k0=\\E[10~:k1=\\EOP:k2=\\EOQ:k3=\\EOR:k4=\\EOS:k5=\\E[15~:k6=\\E[17~:k7=\\E[18~:k8=\\E[19~:k9=\\E[20~:k;=\\E[21~:F1=\\E[23~:F2=\\E[24~:kB=\\E[Z:kh=\\E[1~:@1=\\E[1~:kH=\\E[4~:@7=\\E[4~:kN=\\E[6~:kP=\\E[5~:kI=\\E[2~:kD=\\E[3~:ku=\\EOA:kd=\\EOB:kr=\\EOC:kl=\\EOD:km:', 'LC_ADDRESS': 'cs_CZ.UTF-8', 'LC_NAME': 'cs_CZ.UTF-8', 'WINDOW': '6', 'DESKTOP_SESSION': 'xubuntu', 'LC_MONETARY': 'cs_CZ.UTF-8', 'EDITOR': 'vim', 'XDG_SEAT': 'seat0', 'XDG_SESSION_DESKTOP': 'xubuntu', 'QT_QPA_PLATFORMTHEME': 'gtk2', 'XDG_SESSION_TYPE': 'x11', 'PANEL_GDK_CORE_DEVICE_EVENTS': '0', 'GPG_AGENT_INFO': '/run/user/1000/gnupg/S.gpg-agent:0:1', 'XAUTHORITY': '/home/martin/.Xauthority', 'XDG_GREETER_DATA_DIR': '/var/lib/lightdm-data/martin', 'GDM_LANG': 'en_US', 'LANG': 'C', 'LC_PAPER': 'cs_CZ.UTF-8', 'LS_COLORS': 'rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:', 'XDG_CURRENT_DESKTOP': 'XFCE', 'VTE_VERSION': '6200', 'XDG_SEAT_PATH': '/org/freedesktop/DisplayManager/Seat0', 'LESSCLOSE': '/usr/bin/lesspipe %s %s', 'XDG_SESSION_CLASS': 'user', 'LC_IDENTIFICATION': 'cs_CZ.UTF-8', 'TERM': 'screen.xterm-256color', 'GTK_OVERLAY_SCROLLING': '0', 'LESSOPEN': '| /usr/bin/lesspipe %s', 'DISPLAY': ':0.0', 'SHLVL': '2', 'LC_TELEPHONE': 'cs_CZ.UTF-8', 'LC_MEASUREMENT': 'cs_CZ.UTF-8', 'XDG_VTNR': '7', 'XDG_SESSION_ID': 'c2', 'LD_LIBRARY_PATH': '', 'XDG_RUNTIME_DIR': '/run/user/1000', 'LC_TIME': 'cs_CZ.UTF-8', 'XDG_DATA_DIRS': '/usr/share/xubuntu:/usr/share/xfce4:/usr/local/share:/usr/share:/var/lib/snapd/desktop:/usr/share', 'STY': '5368.x', 'GDMSESSION': 'xubuntu', 'DBUS_SESSION_BUS_ADDRESS': 'unix:path=/run/user/1000/bus', 'BUILDDIR': '/OE/build/oe-core', 'LC_NUMERIC': 'cs_CZ.UTF-8', 'OLDPWD': '/OE/build/oe-core/meta-python2', '_': '/OE/build/oe-core/bitbake/bin/bitbake'}, ['/OE/build/oe-core/bitbake/bin/bitbake', '-k', 'python-native']]
928962 12:42:56.450688 Command Completed
928962 12:42:56.450996 Running command ['getVariable', 'BBINCLUDELOGS']
928962 12:42:56.451159 Command Completed
928962 12:42:56.451239 Running command ['getVariable', 'BBINCLUDELOGS_LINES']
928962 12:42:56.451360 Command Completed
928962 12:42:56.451436 Running command ['getSetVariable', 'BB_CONSOLELOG']
928962 12:42:56.451656 Command Completed
928962 12:42:56.451740 Running command ['getSetVariable', 'BB_LOGCONFIG']
928962 12:42:56.453927 Command Completed
928962 12:42:56.454122 Running command ['getUIHandlerNum']
928962 12:42:56.454289 Command Completed
928962 12:42:56.454392 Running command ['setEventMask', 1, 20, {'BitBake.RunQueue.HashEquiv': 19, 'BitBake.SigGen.HashEquiv': 19}, ['bb.runqueue.runQueueExitWait', 'bb.event.LogExecTTY', 'logging.LogRecord', 'bb.build.TaskFailed', 'bb.build.TaskBase', 'bb.event.ParseStarted', 'bb.event.ParseProgress', 'bb.event.ParseCompleted', 'bb.event.CacheLoadStarted', 'bb.event.CacheLoadProgress', 'bb.event.CacheLoadCompleted', 'bb.command.CommandFailed', 'bb.command.CommandExit', 'bb.command.CommandCompleted', 'bb.cooker.CookerExit', 'bb.event.MultipleProviders', 'bb.event.NoProvider', 'bb.runqueue.sceneQueueTaskStarted', 'bb.runqueue.runQueueTaskStarted', 'bb.runqueue.runQueueTaskFailed', 'bb.runqueue.sceneQueueTaskFailed', 'bb.event.BuildBase', 'bb.build.TaskStarted', 'bb.build.TaskSucceeded', 'bb.build.TaskFailedSilent', 'bb.build.TaskProgress', 'bb.event.ProcessStarted', 'bb.event.ProcessProgress', 'bb.event.ProcessFinished']]
928962 12:42:56.454523 Command Completed
928962 12:42:56.454599 Running command ['getVariable', 'BB_DEFAULT_TASK']
928962 12:42:56.454741 Command Completed
928962 12:42:56.454816 Running command ['setConfig', 'cmd', 'build']
928962 12:42:56.454947 Command Completed
928962 12:42:56.455022 Running command ['buildTargets', ['python-native'], 'build']
928962 12:42:56.457778 Command Completed
928962 12:43:04.560968 Processing Client
928962 12:43:04.561063 Disconnecting Client
928962 12:43:08.544918 Accepting [<socket.socket fd=6, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0, laddr=bitbake.sock>] ([])
928962 12:43:08.557050 Processing Client
928962 12:43:08.557154 Connecting Client
928962 12:43:08.557787 Running command ['setFeatures', [2]]
928962 12:43:08.558183 Command Completed
928962 12:43:08.558952 Running command ['updateConfig', {'abort': False, 'force': False, 'invalidate_stamp': None, 'dry_run': False, 'dump_signatures': [], 'extra_assume_provided': [], 'profile': False, 'prefile': [], 'postfile': [], 'server_timeout': None, 'nosetscene': False, 'setsceneonly': False, 'skipsetscene': False, 'runall': None, 'runonly': None, 'writeeventlog': None, 'build_verbose_shell': False, 'build_verbose_stdout': False, 'default_loglevel': 20, 'debug_domains': {}}, {'SHELL': '/bin/bash', 'SSH_AUTH_SOCK': '/run/user/1000/keyring/ssh', 'SSH_AGENT_PID': '3080', 'PWD': '/OE/build/oe-core', 'LOGNAME': 'martin', 'HOME': '/home/martin', 'MACHINE': 'qemux86-64', 'BB_ENV_EXTRAWHITE': 'MACHINE DISTRO http_proxy ftp_proxy https_proxy all_proxy ALL_PROXY no_proxy SSH_AGENT_PID SSH_AUTH_SOCK BB_SRCREV_POLICY SDKMACHINE BB_NUMBER_THREADS GIT_PROXY_COMMAND PSEUDO_DISABLED PSEUDO_BUILD', 'USER': 'martin', 'LC_ALL': 'en_US.UTF-8', 'PATH': '/OE/build/oe-core/openembedded-core/scripts:/OE/build/oe-core/bitbake/bin:/home/martin/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin', 'SESSION_MANAGER': 'local/jama:@/tmp/.ICE-unix/2943,unix/jama:/tmp/.ICE-unix/2943', 'WINDOWID': '79691779', 'QT_ACCESSIBILITY': '1', 'COLORTERM': 'truecolor', 'XDG_CONFIG_DIRS': '/etc/xdg/xdg-xubuntu:/etc/xdg:/etc/xdg', 'XDG_SESSION_PATH': '/org/freedesktop/DisplayManager/Session0', 'XDG_MENU_PREFIX': 'xfce-', 'CLUTTER_BACKEND': 'x11', 'LANGUAGE': 'en_US', 'TERMCAP': 'SC|screen.xterm-256color|VT 100/ANSI X3.64 virtual terminal:DO=\\E[%dB:LE=\\E[%dD:RI=\\E[%dC:UP=\\E[%dA:bs:bt=\\E[Z:cd=\\E[J:ce=\\E[K:cl=\\E[H\\E[J:cm=\\E[%i%d;%dH:ct=\\E[3g:do=^J:nd=\\E[C:pt:rc=\\E8:rs=\\Ec:sc=\\E7:st=\\EH:up=\\EM:le=^H:bl=^G:cr=^M:it#8:ho=\\E[H:nw=\\EE:ta=^I:is=\\E)0:li#51:co#263:am:xn:xv:LP:sr=\\EM:al=\\E[L:AL=\\E[%dL:cs=\\E[%i%d;%dr:dl=\\E[M:DL=\\E[%dM:dc=\\E[P:DC=\\E[%dP:im=\\E[4h:ei=\\E[4l:mi:IC=\\E[%d@:ks=\\E[?1h\\E=:ke=\\E[?1l\\E>:vi=\\E[?25l:ve=\\E[34h\\E[?25h:vs=\\E[34l:ti=\\E[?1049h:te=\\E[?1049l:us=\\E[4m:ue=\\E[24m:so=\\E[3m:se=\\E[23m:mb=\\E[5m:md=\\E[1m:mh=\\E[2m:mr=\\E[7m:me=\\E[m:ms:Co#8:pa#64:AF=\\E[3%dm:AB=\\E[4%dm:op=\\E[39;49m:AX:vb=\\Eg:G0:as=\\E(0:ae=\\E(B:ac=\\140\\140aaffggjjkkllmmnnooppqqrrssttuuvvwwxxyyzz{{||}}~~..--++,,hhII00:po=\\E[5i:pf=\\E[4i:Km=\\E[<:k0=\\E[10~:k1=\\EOP:k2=\\EOQ:k3=\\EOR:k4=\\EOS:k5=\\E[15~:k6=\\E[17~:k7=\\E[18~:k8=\\E[19~:k9=\\E[20~:k;=\\E[21~:F1=\\E[23~:F2=\\E[24~:kB=\\E[Z:kh=\\E[1~:@1=\\E[1~:kH=\\E[4~:@7=\\E[4~:kN=\\E[6~:kP=\\E[5~:kI=\\E[2~:kD=\\E[3~:ku=\\EOA:kd=\\EOB:kr=\\EOC:kl=\\EOD:km:', 'LC_ADDRESS': 'cs_CZ.UTF-8', 'LC_NAME': 'cs_CZ.UTF-8', 'WINDOW': '6', 'DESKTOP_SESSION': 'xubuntu', 'LC_MONETARY': 'cs_CZ.UTF-8', 'EDITOR': 'vim', 'XDG_SEAT': 'seat0', 'XDG_SESSION_DESKTOP': 'xubuntu', 'QT_QPA_PLATFORMTHEME': 'gtk2', 'XDG_SESSION_TYPE': 'x11', 'PANEL_GDK_CORE_DEVICE_EVENTS': '0', 'GPG_AGENT_INFO': '/run/user/1000/gnupg/S.gpg-agent:0:1', 'XAUTHORITY': '/home/martin/.Xauthority', 'XDG_GREETER_DATA_DIR': '/var/lib/lightdm-data/martin', 'GDM_LANG': 'en_US', 'LANG': 'C', 'LC_PAPER': 'cs_CZ.UTF-8', 'LS_COLORS': 'rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:', 'XDG_CURRENT_DESKTOP': 'XFCE', 'VTE_VERSION': '6200', 'XDG_SEAT_PATH': '/org/freedesktop/DisplayManager/Seat0', 'LESSCLOSE': '/usr/bin/lesspipe %s %s', 'XDG_SESSION_CLASS': 'user', 'LC_IDENTIFICATION': 'cs_CZ.UTF-8', 'TERM': 'screen.xterm-256color', 'GTK_OVERLAY_SCROLLING': '0', 'LESSOPEN': '| /usr/bin/lesspipe %s', 'DISPLAY': ':0.0', 'SHLVL': '2', 'LC_TELEPHONE': 'cs_CZ.UTF-8', 'LC_MEASUREMENT': 'cs_CZ.UTF-8', 'XDG_VTNR': '7', 'XDG_SESSION_ID': 'c2', 'LD_LIBRARY_PATH': '', 'XDG_RUNTIME_DIR': '/run/user/1000', 'LC_TIME': 'cs_CZ.UTF-8', 'XDG_DATA_DIRS': '/usr/share/xubuntu:/usr/share/xfce4:/usr/local/share:/usr/share:/var/lib/snapd/desktop:/usr/share', 'STY': '5368.x', 'GDMSESSION': 'xubuntu', 'DBUS_SESSION_BUS_ADDRESS': 'unix:path=/run/user/1000/bus', 'BUILDDIR': '/OE/build/oe-core', 'LC_NUMERIC': 'cs_CZ.UTF-8', 'OLDPWD': '/OE/build/oe-core/meta-python2', '_': '/OE/build/oe-core/bitbake/bin/bitbake'}, ['/OE/build/oe-core/bitbake/bin/bitbake', '-k', 'python-native']]
928962 12:43:08.559567 Command Completed
928962 12:43:08.559798 Running command ['getVariable', 'BBINCLUDELOGS']
928962 12:43:08.812911 Command Completed
928962 12:43:08.813560 Running command ['getVariable', 'BBINCLUDELOGS_LINES']
928962 12:43:08.813765 Command Completed
928962 12:43:08.813879 Running command ['getSetVariable', 'BB_CONSOLELOG']
928962 12:43:08.814115 Command Completed
928962 12:43:08.814236 Running command ['getSetVariable', 'BB_LOGCONFIG']
928962 12:43:08.816210 Command Completed
928962 12:43:08.816451 Running command ['getUIHandlerNum']
928962 12:43:08.816661 Command Completed
928962 12:43:08.816799 Running command ['setEventMask', 2, 20, {'BitBake.RunQueue.HashEquiv': 19, 'BitBake.SigGen.HashEquiv': 19}, ['bb.runqueue.runQueueExitWait', 'bb.event.LogExecTTY', 'logging.LogRecord', 'bb.build.TaskFailed', 'bb.build.TaskBase', 'bb.event.ParseStarted', 'bb.event.ParseProgress', 'bb.event.ParseCompleted', 'bb.event.CacheLoadStarted', 'bb.event.CacheLoadProgress', 'bb.event.CacheLoadCompleted', 'bb.command.CommandFailed', 'bb.command.CommandExit', 'bb.command.CommandCompleted', 'bb.cooker.CookerExit', 'bb.event.MultipleProviders', 'bb.event.NoProvider', 'bb.runqueue.sceneQueueTaskStarted', 'bb.runqueue.runQueueTaskStarted', 'bb.runqueue.runQueueTaskFailed', 'bb.runqueue.sceneQueueTaskFailed', 'bb.event.BuildBase', 'bb.build.TaskStarted', 'bb.build.TaskSucceeded', 'bb.build.TaskFailedSilent', 'bb.build.TaskProgress', 'bb.event.ProcessStarted', 'bb.event.ProcessProgress', 'bb.event.ProcessFinished']]
928962 12:43:08.816980 Command Completed
928962 12:43:08.817101 Running command ['getVariable', 'BB_DEFAULT_TASK']
928962 12:43:08.817293 Command Completed
928962 12:43:08.817418 Running command ['setConfig', 'cmd', 'build']
928962 12:43:08.817604 Command Completed
928962 12:43:08.817732 Running command ['buildTargets', ['python-native'], 'build']
928962 12:43:08.820432 Command Completed
928962 12:43:18.569947 Processing Client
928962 12:43:18.570026 Disconnecting Client
928962 12:43:21.796010 Accepting [<socket.socket fd=6, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0, laddr=bitbake.sock>] ([])
928962 12:43:21.812637 Processing Client
928962 12:43:21.812758 Connecting Client
928962 12:43:21.816670 Running command ['setFeatures', [2]]
928962 12:43:21.817252 Command Completed
928962 12:43:21.817987 Running command ['updateConfig', {'abort': False, 'force': False, 'invalidate_stamp': None, 'dry_run': False, 'dump_signatures': [], 'extra_assume_provided': [], 'profile': False, 'prefile': [], 'postfile': [], 'server_timeout': None, 'nosetscene': False, 'setsceneonly': False, 'skipsetscene': False, 'runall': None, 'runonly': None, 'writeeventlog': None, 'build_verbose_shell': False, 'build_verbose_stdout': False, 'default_loglevel': 20, 'debug_domains': {}}, {'SHELL': '/bin/bash', 'SSH_AUTH_SOCK': '/run/user/1000/keyring/ssh', 'SSH_AGENT_PID': '3080', 'PWD': '/OE/build/oe-core', 'LOGNAME': 'martin', 'HOME': '/home/martin', 'MACHINE': 'qemux86-64', 'BB_ENV_EXTRAWHITE': 'MACHINE DISTRO http_proxy ftp_proxy https_proxy all_proxy ALL_PROXY no_proxy SSH_AGENT_PID SSH_AUTH_SOCK BB_SRCREV_POLICY SDKMACHINE BB_NUMBER_THREADS GIT_PROXY_COMMAND PSEUDO_DISABLED PSEUDO_BUILD', 'USER': 'martin', 'LC_ALL': 'en_US.UTF-8', 'PATH': '/OE/build/oe-core/openembedded-core/scripts:/OE/build/oe-core/bitbake/bin:/home/martin/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin', 'SESSION_MANAGER': 'local/jama:@/tmp/.ICE-unix/2943,unix/jama:/tmp/.ICE-unix/2943', 'WINDOWID': '79691779', 'QT_ACCESSIBILITY': '1', 'COLORTERM': 'truecolor', 'XDG_CONFIG_DIRS': '/etc/xdg/xdg-xubuntu:/etc/xdg:/etc/xdg', 'XDG_SESSION_PATH': '/org/freedesktop/DisplayManager/Session0', 'XDG_MENU_PREFIX': 'xfce-', 'CLUTTER_BACKEND': 'x11', 'LANGUAGE': 'en_US', 'TERMCAP': 'SC|screen.xterm-256color|VT 100/ANSI X3.64 virtual terminal:DO=\\E[%dB:LE=\\E[%dD:RI=\\E[%dC:UP=\\E[%dA:bs:bt=\\E[Z:cd=\\E[J:ce=\\E[K:cl=\\E[H\\E[J:cm=\\E[%i%d;%dH:ct=\\E[3g:do=^J:nd=\\E[C:pt:rc=\\E8:rs=\\Ec:sc=\\E7:st=\\EH:up=\\EM:le=^H:bl=^G:cr=^M:it#8:ho=\\E[H:nw=\\EE:ta=^I:is=\\E)0:li#51:co#263:am:xn:xv:LP:sr=\\EM:al=\\E[L:AL=\\E[%dL:cs=\\E[%i%d;%dr:dl=\\E[M:DL=\\E[%dM:dc=\\E[P:DC=\\E[%dP:im=\\E[4h:ei=\\E[4l:mi:IC=\\E[%d@:ks=\\E[?1h\\E=:ke=\\E[?1l\\E>:vi=\\E[?25l:ve=\\E[34h\\E[?25h:vs=\\E[34l:ti=\\E[?1049h:te=\\E[?1049l:us=\\E[4m:ue=\\E[24m:so=\\E[3m:se=\\E[23m:mb=\\E[5m:md=\\E[1m:mh=\\E[2m:mr=\\E[7m:me=\\E[m:ms:Co#8:pa#64:AF=\\E[3%dm:AB=\\E[4%dm:op=\\E[39;49m:AX:vb=\\Eg:G0:as=\\E(0:ae=\\E(B:ac=\\140\\140aaffggjjkkllmmnnooppqqrrssttuuvvwwxxyyzz{{||}}~~..--++,,hhII00:po=\\E[5i:pf=\\E[4i:Km=\\E[<:k0=\\E[10~:k1=\\EOP:k2=\\EOQ:k3=\\EOR:k4=\\EOS:k5=\\E[15~:k6=\\E[17~:k7=\\E[18~:k8=\\E[19~:k9=\\E[20~:k;=\\E[21~:F1=\\E[23~:F2=\\E[24~:kB=\\E[Z:kh=\\E[1~:@1=\\E[1~:kH=\\E[4~:@7=\\E[4~:kN=\\E[6~:kP=\\E[5~:kI=\\E[2~:kD=\\E[3~:ku=\\EOA:kd=\\EOB:kr=\\EOC:kl=\\EOD:km:', 'LC_ADDRESS': 'cs_CZ.UTF-8', 'LC_NAME': 'cs_CZ.UTF-8', 'WINDOW': '6', 'DESKTOP_SESSION': 'xubuntu', 'LC_MONETARY': 'cs_CZ.UTF-8', 'EDITOR': 'vim', 'XDG_SEAT': 'seat0', 'XDG_SESSION_DESKTOP': 'xubuntu', 'QT_QPA_PLATFORMTHEME': 'gtk2', 'XDG_SESSION_TYPE': 'x11', 'PANEL_GDK_CORE_DEVICE_EVENTS': '0', 'GPG_AGENT_INFO': '/run/user/1000/gnupg/S.gpg-agent:0:1', 'XAUTHORITY': '/home/martin/.Xauthority', 'XDG_GREETER_DATA_DIR': '/var/lib/lightdm-data/martin', 'GDM_LANG': 'en_US', 'LANG': 'C', 'LC_PAPER': 'cs_CZ.UTF-8', 'LS_COLORS': 'rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:', 'XDG_CURRENT_DESKTOP': 'XFCE', 'VTE_VERSION': '6200', 'XDG_SEAT_PATH': '/org/freedesktop/DisplayManager/Seat0', 'LESSCLOSE': '/usr/bin/lesspipe %s %s', 'XDG_SESSION_CLASS': 'user', 'LC_IDENTIFICATION': 'cs_CZ.UTF-8', 'TERM': 'screen.xterm-256color', 'GTK_OVERLAY_SCROLLING': '0', 'LESSOPEN': '| /usr/bin/lesspipe %s', 'DISPLAY': ':0.0', 'SHLVL': '2', 'LC_TELEPHONE': 'cs_CZ.UTF-8', 'LC_MEASUREMENT': 'cs_CZ.UTF-8', 'XDG_VTNR': '7', 'XDG_SESSION_ID': 'c2', 'LD_LIBRARY_PATH': '', 'XDG_RUNTIME_DIR': '/run/user/1000', 'LC_TIME': 'cs_CZ.UTF-8', 'XDG_DATA_DIRS': '/usr/share/xubuntu:/usr/share/xfce4:/usr/local/share:/usr/share:/var/lib/snapd/desktop:/usr/share', 'STY': '5368.x', 'GDMSESSION': 'xubuntu', 'DBUS_SESSION_BUS_ADDRESS': 'unix:path=/run/user/1000/bus', 'BUILDDIR': '/OE/build/oe-core', 'LC_NUMERIC': 'cs_CZ.UTF-8', 'OLDPWD': '/OE/build/oe-core/meta-python2', '_': '/OE/build/oe-core/bitbake/bin/bitbake'}, ['/OE/build/oe-core/bitbake/bin/bitbake', '-k', 'python-native']]
928962 12:43:21.818533 Command Completed
928962 12:43:21.818721 Running command ['getVariable', 'BBINCLUDELOGS']
928962 12:43:22.255470 Command Completed
928962 12:43:22.255835 Running command ['getVariable', 'BBINCLUDELOGS_LINES']
928962 12:43:22.256018 Command Completed
928962 12:43:22.256172 Running command ['getSetVariable', 'BB_CONSOLELOG']
928962 12:43:22.256346 Command Completed
928962 12:43:22.256468 Running command ['getSetVariable', 'BB_LOGCONFIG']
928962 12:43:22.256618 Command Completed
928962 12:43:22.264356 Running command ['getUIHandlerNum']
928962 12:43:22.264922 Command Completed
928962 12:43:22.265103 Running command ['setEventMask', 3, 20, {'BitBake.SigGen.HashEquiv': 19, 'BitBake.RunQueue.HashEquiv': 19}, ['bb.runqueue.runQueueExitWait', 'bb.event.LogExecTTY', 'logging.LogRecord', 'bb.build.TaskFailed', 'bb.build.TaskBase', 'bb.event.ParseStarted', 'bb.event.ParseProgress', 'bb.event.ParseCompleted', 'bb.event.CacheLoadStarted', 'bb.event.CacheLoadProgress', 'bb.event.CacheLoadCompleted', 'bb.command.CommandFailed', 'bb.command.CommandExit', 'bb.command.CommandCompleted', 'bb.cooker.CookerExit', 'bb.event.MultipleProviders', 'bb.event.NoProvider', 'bb.runqueue.sceneQueueTaskStarted', 'bb.runqueue.runQueueTaskStarted', 'bb.runqueue.runQueueTaskFailed', 'bb.runqueue.sceneQueueTaskFailed', 'bb.event.BuildBase', 'bb.build.TaskStarted', 'bb.build.TaskSucceeded', 'bb.build.TaskFailedSilent', 'bb.build.TaskProgress', 'bb.event.ProcessStarted', 'bb.event.ProcessProgress', 'bb.event.ProcessFinished']]
928962 12:43:22.265324 Command Completed
928962 12:43:22.265466 Running command ['getVariable', 'BB_DEFAULT_TASK']
928962 12:43:22.272249 Command Completed
928962 12:43:22.272677 Running command ['setConfig', 'cmd', 'build']
928962 12:43:22.273073 Command Completed
928962 12:43:22.273220 Running command ['buildTargets', ['python-native'], 'build']
928962 12:43:22.275823 Command Completed

It was reproduced on builder with 64 cores.

After changing BB_NUMBER_PARSE_THREADS from default 64 to 10, I wasn't able to get bitbake stuck again (in the same setup after cleaning cache, tmpdir, sstate again) by calling "bitbake python-native" 20+ times.

But after changing BB_NUMBER_PARSE_THREADS to 100 I wasn't able to reproduce it again (so maybe 3rd try with the default value was just me being lucky to reproduce it).

Then I've tried to change it to 1000, just to see what happens and it got stuck (but probably from different reasons):

martin@jama:/OE/build/oe-core$ bitbake -k python-native
Loading cache: 100% | | ETA: --:--:--
Loaded 0 entries from dependency cache.
ERROR: /OE/build/oe-core/meta-python2/recipes-devtools/python/python-native_2.7.18.bb: Error executing a python function in <code>: | ETA: --:--:--

The stack trace of python calls that resulted in this exception/failure was:
File: '<code>', lineno: 14, function: <module>
0010:__anon_35__OE_build_oe_core_openembedded_core_meta_classes_devshell_bbclass(d)
0011:__anon_151__OE_build_oe_core_openembedded_core_meta_classes_sstate_bbclass(d)
0012:__anon_20__OE_build_oe_core_openembedded_core_meta_classes_blacklist_bbclass(d)
0013:__anon_177__OE_build_oe_core_openembedded_core_meta_classes_siteinfo_bbclass(d)
*** 0014:__anon_90__OE_build_oe_core_meta_python2_recipes_devtools_python_python_native_2_7_18_bb(d)
File: '/OE/build/oe-core/meta-python2/recipes-devtools/python/python-native_2.7.18.bb', lineno: 76, function: __anon_90__OE_build_oe_core_meta_python2_recipes_devtools_python_python_native_2_7_18_bb
0072: manifest_file.seek(json_start)
0073: manifest_str = manifest_file.read()
0074: python_manifest = json.loads(manifest_str)
0075:
*** 0076: rprovides = d.getVar('RPROVIDES').split()
0077:
0078: # Hardcoded since it cant be python-native-foo, should be python-foo-native
0079: pn = 'python'
0080:
Exception: AttributeError: 'NoneType' object has no attribute 'split'

ERROR: Command execution failed: Traceback (most recent call last):
File "/OE/bitbake/lib/bb/command.py", line 114, in runAsyncCommand
File "/OE/bitbake/lib/bb/cooker.py", line 1605, in updateCache
File "/OE/bitbake/lib/bb/cooker.py", line 2087, in __init__
File "/OE/bitbake/lib/bb/cooker.py", line 2113, in start
File "/usr/lib/python3.8/multiprocessing/process.py", line 121, in start
File "/usr/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
File "/usr/lib/python3.8/multiprocessing/context.py", line 277, in _Popen
File "/usr/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
File "/usr/lib/python3.8/multiprocessing/popen_fork.py", line 69, in _launch
OSError: [Errno 24] Too many open files


Summary: There were 2 ERROR messages shown, returning a non-zero exit code.
martin@jama:/OE/build/oe-core$ bitbake -k python-native
NOTE: Reconnecting to bitbake server...
NOTE: No reply from server in 30s
NOTE: Retrying server connection (#1)...

On this builder I was using:
BB_SERVER_TIMEOUT = "60"
but that doesn't seem to be required to reproduce as the other builds I've seen stuck on jenkins weren't setting BB_SERVER_TIMEOUT at all.

Let me know if there is something else I should try next time it gets stuck, but right now
I'm not able to reliably reproduce it again (even after switching back to default BB_NUMBER_PARSE_THREADS)

Cheers,

Cheers,

Richard

Join openembedded-core@lists.openembedded.org to automatically receive all group messages.