Re: [PATCH 6/7] native: Stop clearing PACKAGES


Richard Purdie
 

On Sun, 2021-01-31 at 14:10 +0100, Martin Jansa wrote:
On Sun, Jan 31, 2021 at 10:34:48AM +0000, Richard Purdie wrote:
Its the first time I've heard a report of that and it sounds a little
worrying. What you say stuck, bitbake hangs? Is it using cpu? A process
tree of what it looks like when stuck would be interesting if it is
hanging and maybe the last things in any running or just finished task
logs. It could be some issue with the way a failing task is handled?

We've seen a lot of weird issues on the autobuilder but not that. Could
you have something enabled in your setup for logging/reporting whch
wouldn't be in ours and could be causing it?
It doesn't seem to happen very often and I guess autobuilder doesn't
include meta-python2 (which is currently the only reproducer in my list
of layers).
We don't use meta-python2 on the autobuilder so that could explain why
we've not seen this.

I've tried to reproduce it with just 3 layers included:
BBLAYERS = " \
  /OE/build/oe-core/meta-python2 \
  /OE/build/oe-core/meta-openembedded/meta-oe \
  /OE/build/oe-core/openembedded-core/meta \
"

and it happened on 3rd try (cache, tmpdir, sstate-cache were removed
only before 1st try)

martin@jama:/OE/build/oe-core$ bitbake -k python-native
Loading cache: 100% | | ETA: --:--:--
Loaded 0 entries from dependency cache.
ERROR: /OE/build/oe-core/meta-python2/recipes-devtools/python/python-setuptools_42.0.2.bb: QA Issue: /OE/build/oe-core/meta-python2/recipes-devtools/python/python-setuptools_42.0.2.bb: Variable RPROVIDES is set as not being package specific, please fix this. [pkgvarcheck]
ERROR: /OE/build/oe-core/meta-python2/recipes-devtools/python/python-setuptools_42.0.2.bb: Fatal QA errors found, failing task.
ERROR: Failed to parse recipe: /OE/build/oe-core/meta-python2/recipes-devtools/python/python-setuptools_42.0.2.bb | ETA: 0:00:14

Summary: There were 3 ERROR messages shown, returning a non-zero exit code.
martin@jama:/OE/build/oe-core$ bitbake -k python-native
NOTE: Reconnecting to bitbake server...
Loading cache: 100% |###################################################################################################################################################################################################################################| Time: 0:00:00
Loaded 845 entries from dependency cache.
ERROR: /OE/build/oe-core/meta-python2/recipes-devtools/python/python-native_2.7.18.bb: Error executing a python function in <code>: | ETA: 0:01:10

The stack trace of python calls that resulted in this exception/failure was:
File: '<code>', lineno: 14, function: <module>
     0010:__anon_35__OE_build_oe_core_openembedded_core_meta_classes_devshell_bbclass(d)
     0011:__anon_151__OE_build_oe_core_openembedded_core_meta_classes_sstate_bbclass(d)
     0012:__anon_20__OE_build_oe_core_openembedded_core_meta_classes_blacklist_bbclass(d)
     0013:__anon_177__OE_build_oe_core_openembedded_core_meta_classes_siteinfo_bbclass(d)
 *** 0014:__anon_90__OE_build_oe_core_meta_python2_recipes_devtools_python_python_native_2_7_18_bb(d)
File: '/OE/build/oe-core/meta-python2/recipes-devtools/python/python-native_2.7.18.bb', lineno: 76, function: __anon_90__OE_build_oe_core_meta_python2_recipes_devtools_python_python_native_2_7_18_bb
     0072: manifest_file.seek(json_start)
     0073: manifest_str = manifest_file.read()
     0074: python_manifest = json.loads(manifest_str)
     0075:
 *** 0076: rprovides = d.getVar('RPROVIDES').split()
     0077:
     0078: # Hardcoded since it cant be python-native-foo, should be python-foo-native
     0079: pn = 'python'
     0080:
Exception: AttributeError: 'NoneType' object has no attribute 'split'
This sounds like a similar issue to the one I fixed in the python3
recipe:

http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=cff7db890cdfab41cc8f74e3dc378660d9b6219e

ERROR: /OE/build/oe-core/meta-python2/recipes-devtools/python/python-setuptools_42.0.2.bb: QA Issue: /OE/build/oe-core/meta-python2/recipes-devtools/python/python-setuptools_42.0.2.bb: Variable RPROVIDES is set as not being package specific, please fix this. [pkgvarcheck]
ERROR: /OE/build/oe-core/meta-python2/recipes-devtools/python/python-setuptools_42.0.2.bb: Fatal QA errors found, failing task.
ERROR: Failed to parse recipe: /OE/build/oe-core/meta-python2/recipes-devtools/python/python-native_2.7.18.bb | ETA: 0:00:30
Which makes me wonder whether the real issue is a bug in bitbake when
parsing fails. We did have an issue like that a while back but I
thought it was fixed by changes both Joshua and I made.


Summary: There were 4 ERROR messages shown, returning a non-zero exit code.
martin@jama:/OE/build/oe-core$ bitbake -k python-native
NOTE: Reconnecting to bitbake server...
Loading cache: 100% |###################################################################################################################################################################################################################################| Time: 0:00:00
Loaded 1289 entries from dependency cache.
ERROR: /OE/build/oe-core/meta-python2/recipes-devtools/python/python-setuptools_42.0.2.bb: QA Issue: /OE/build/oe-core/meta-python2/recipes-devtools/python/python-setuptools_42.0.2.bb: Variable RPROVIDES is set as not being package specific, please fix this. [pkgvarcheck]
ERROR: /OE/build/oe-core/meta-python2/recipes-devtools/python/python-setuptools_42.0.2.bb: Fatal QA errors found, failing task.
ERROR: Failed to parse recipe: /OE/build/oe-core/meta-python2/recipes-devtools/python/python-setuptools_42.0.2.bb

and here it's stuck for around 1 hour, bitbake using very little cpu.

martin@jama:~$ ps aux | grep bitbake
martin 372784 0.0 0.0 11552 884 pts/1 S+ 13:43 0:00 grep --color=auto bitbake
martin 928962 0.1 0.1 219204 133704 ? Sl 12:42 0:04 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 943698 0.0 0.0 119900 29928 pts/8 Sl+ 12:43 0:01 python3 /OE/build/oe-core/bitbake/bin/bitbake -k python-native
martin 945507 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 945528 0.0 0.0 214364 124740 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 945532 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 945563 0.0 0.0 214364 124732 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 945577 0.0 0.0 214364 124740 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 945621 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 945632 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 945643 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 945653 0.0 0.0 214364 124740 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 945669 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 945685 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 945703 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 945736 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 945753 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 945774 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 945778 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 945819 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 945839 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 945860 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 945878 0.0 0.0 214364 124740 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 945894 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 945909 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 945915 0.0 0.0 214364 124804 ? Sl 12:43 0:00 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 945929 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 945950 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 945953 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 945964 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 945966 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 945989 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946005 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946016 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946037 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946038 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946064 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946068 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946080 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946103 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946109 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946122 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946135 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946137 0.0 0.0 214364 124736 ? Sl 12:43 0:00 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946152 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946167 0.0 0.0 214364 124736 ? Sl 12:43 0:00 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946168 0.0 0.0 214364 124736 ? Sl 12:43 0:00 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946185 0.0 0.0 214364 124736 ? Sl 12:43 0:00 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946197 0.0 0.0 214364 124736 ? Sl 12:43 0:00 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946214 0.0 0.0 214364 124736 ? Sl 12:43 0:00 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946228 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946243 0.0 0.0 214364 124728 ? Sl 12:43 0:00 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946249 0.0 0.0 214364 124736 ? Sl 12:43 0:00 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946265 0.0 0.0 214364 124736 ? Sl 12:43 0:00 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946298 0.0 0.0 214364 124740 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946345 0.0 0.0 214364 124736 ? Sl 12:43 0:01 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946364 0.0 0.0 214364 124736 ? Sl 12:43 0:00 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946386 0.0 0.0 214364 124736 ? Sl 12:43 0:00 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946389 0.0 0.0 214364 124736 ? Sl 12:43 0:00 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946405 0.0 0.0 214364 124736 ? Sl 12:43 0:00 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946410 0.0 0.0 214364 124736 ? Sl 12:43 0:00 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946442 0.0 0.0 214364 124736 ? Sl 12:43 0:00 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946458 0.0 0.0 214364 124612 ? Sl 12:43 0:00 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946463 0.0 0.0 214364 124652 ? Sl 12:43 0:00 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946481 0.0 0.0 214364 124736 ? Sl 12:43 0:00 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946501 0.0 0.0 214364 124736 ? Sl 12:43 0:00 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin 946518 0.0 0.0 214364 124736 ? Sl 12:43 0:00 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
Something is wrong here as bitbake shouldn't have this number of
processes like this :(

martin@jama:~$ sudo strace -p 928962
strace: Process 928962 attached
wait4(945507,

martin@jama:~$ kill 928962

adds an warning to the stuck build:

martin@jama:/OE/build/oe-core$ bitbake -k python-native
NOTE: Reconnecting to bitbake server...
Loading cache: 100% |###################################################################################################################################################################################################################################| Time: 0:00:00
Loaded 1289 entries from dependency cache.
ERROR: /OE/build/oe-core/meta-python2/recipes-devtools/python/python-setuptools_42.0.2.bb: QA Issue: /OE/build/oe-core/meta-python2/recipes-devtools/python/python-setuptools_42.0.2.bb: Variable RPROVIDES is set as not being package specific, please fix this. [pkgvarcheck]
ERROR: /OE/build/oe-core/meta-python2/recipes-devtools/python/python-setuptools_42.0.2.bb: Fatal QA errors found, failing task.
ERROR: Failed to parse recipe: /OE/build/oe-core/meta-python2/recipes-devtools/python/python-setuptools_42.0.2.bb | ETA: 0:00:50
WARNING: Cooker received SIGTERM, shutting down...

killing the main process:
martin@jama:~$ kill 943698

terminates the stuck build, but all bitbake-server processes are left behind.

If I try to kill all remaining bitbake-server processes, then only one is left behind:
martin 928962 0.1 0.1 285004 137688 ? S 12:42 0:05 bitbake-server /OE/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/oe-core/bitbake-cookerdaemon.log /OE/build/oe-core/bitbake.lock /OE/build/oe-core/bitbake.sock None None 0
martin@jama:/OE/build/oe-core$ sudo strace -p 928962
strace: Process 928962 attached
select(7, [6], [], [], {tv_sec=0, tv_usec=40768}) = 0 (Timeout)
poll([{fd=7, events=POLLIN}], 1, 0) = 0 (Timeout)
poll([{fd=8, events=POLLIN}], 1, 0) = 0 (Timeout)
select(7, [6], [], [], {tv_sec=0, tv_usec=100000}) = 0 (Timeout)
poll([{fd=7, events=POLLIN}], 1, 0) = 0 (Timeout)
poll([{fd=8, events=POLLIN}], 1, 0) = 0 (Timeout)

and then it dies as well after a while.
It sounds like the failed parsing is causing the parsing pool to fail
to clean up properly and its hanging at exit. I don't know why :(

Here is bitbake-cookerdaemon.log, but doesn't look very interesting to me:
You're right, nothing looked untoward there to me either.


It was reproduced on builder with 64 cores.

After changing BB_NUMBER_PARSE_THREADS from default 64 to 10, I
wasn't able to get bitbake stuck again (in the same setup after
cleaning cache, tmpdir, sstate again) by calling "bitbake python-
native" 20+ times.

But after changing BB_NUMBER_PARSE_THREADS to 100 I wasn't able to
reproduce it again (so maybe 3rd try with the default value was just
me being lucky to reproduce it).
I think after a certain number of tries, its not hitting the magic
combination where the parse failure breaks the parsing pool.



Then I've tried to change it to 1000, just to see what happens and it got stuck (but probably from different reasons):

martin@jama:/OE/build/oe-core$ bitbake -k python-native
Loading cache: 100% | | ETA: --:--:--
Loaded 0 entries from dependency cache.
ERROR: /OE/build/oe-core/meta-python2/recipes-devtools/python/python-native_2.7.18.bb: Error executing a python function in <code>: | ETA: --:--:--

The stack trace of python calls that resulted in this exception/failure was:
File: '<code>', lineno: 14, function: <module>
     0010:__anon_35__OE_build_oe_core_openembedded_core_meta_classes_devshell_bbclass(d)
     0011:__anon_151__OE_build_oe_core_openembedded_core_meta_classes_sstate_bbclass(d)
     0012:__anon_20__OE_build_oe_core_openembedded_core_meta_classes_blacklist_bbclass(d)
     0013:__anon_177__OE_build_oe_core_openembedded_core_meta_classes_siteinfo_bbclass(d)
 *** 0014:__anon_90__OE_build_oe_core_meta_python2_recipes_devtools_python_python_native_2_7_18_bb(d)
File: '/OE/build/oe-core/meta-python2/recipes-devtools/python/python-native_2.7.18.bb', lineno: 76, function: __anon_90__OE_build_oe_core_meta_python2_recipes_devtools_python_python_native_2_7_18_bb
     0072: manifest_file.seek(json_start)
     0073: manifest_str = manifest_file.read()
     0074: python_manifest = json.loads(manifest_str)
     0075:
 *** 0076: rprovides = d.getVar('RPROVIDES').split()
     0077:
     0078: # Hardcoded since it cant be python-native-foo, should be python-foo-native
     0079: pn = 'python'
     0080:
Exception: AttributeError: 'NoneType' object has no attribute 'split'
This one is one of the original errors above and can be fixed with a
patch.

ERROR: Command execution failed: Traceback (most recent call last):
  File "/OE/bitbake/lib/bb/command.py", line 114, in runAsyncCommand
  File "/OE/bitbake/lib/bb/cooker.py", line 1605, in updateCache
  File "/OE/bitbake/lib/bb/cooker.py", line 2087, in __init__
  File "/OE/bitbake/lib/bb/cooker.py", line 2113, in start
  File "/usr/lib/python3.8/multiprocessing/process.py", line 121, in start
  File "/usr/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
  File "/usr/lib/python3.8/multiprocessing/context.py", line 277, in _Popen
  File "/usr/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
  File "/usr/lib/python3.8/multiprocessing/popen_fork.py", line 69, in _launch
OSError: [Errno 24] Too many open files
This one is as it says, you exceeded the resource limits, so yes, it is
unrelated.

Summary: There were 2 ERROR messages shown, returning a non-zero exit code.
martin@jama:/OE/build/oe-core$ bitbake -k python-native
NOTE: Reconnecting to bitbake server...
NOTE: No reply from server in 30s
NOTE: Retrying server connection (#1)...

On this builder I was using:
BB_SERVER_TIMEOUT = "60"
but that doesn't seem to be required to reproduce as the other builds I've seen 
stuck on jenkins weren't setting BB_SERVER_TIMEOUT at all.
That did at least explain some of the logs as I was wondering!
BB_SERVER_TIMEOUT being set is known to cause problems on the
autobuilders too, we've not been able to adopt it yet. I believe we
opened bugs for the issues although I'm blanking on what they were
right now.

Let me know if there is something else I should try next time it gets
stuck, but right now
I'm not able to reliably reproduce it again (even after switching
back to default BB_NUMBER_PARSE_THREADS)
I think the reproducer will be some combination of parsing failures and
a need to parse a larger number of recipes. I will give debug some
further thought, I'm not entirely sure how to proceed on this one at
the moment :/.

I suspect if you're not hitting parse failures, things will calm down
though.

Cheers,

Richard

Join openembedded-core@lists.openembedded.org to automatically receive all group messages.