[OE-core] [PATCH 0/4] Restructure python2 and python3 packaging system

Alejandro Hernandez alejandro.hernandez at linux.intel.com
Wed Oct 18 21:07:36 UTC 2017

The reason we have a manifest file for python is that our goal is to
keep python-core as small as posible and add other python packages only
when the user needs them, hence why we split upstream python into several

There are many problems with our current implementation of the manifest file,
this patch tries to deal with all of them along with adding several other

This patch adds a new task to python recipes, which is meant to create a new
manifest file every release.

$ bitbake python -c create_manifest  

In a very simplistic way what this does is:                                                                                                      
Launch python and see specifically what is required for it to run at a minimum                                                                   
Go through the python-manifest file and launch a separate task for every single                                                                  
one of the files on each package, this task will check what was required for that                                                                
specific module to run, these modules will be called dependencies.                                                                               
The output of such task will be a list of the modules or dependencies that were                                                                  
found for that file.                                                                                                                             
Such output will be parsed by this script, we will look for each dependency on the                                                               
manifest and if we find that another package already includes it, then we will add                                                               
that package as an RDEPENDS to the package we are currently checking; in case we dont                                                            
find the current dependency on any other package we will add it to the current package                                                           
as part of FILES.                                                                                                                                
This way we will create a new manifest from the data structure that was built during                                                             
this process, ont this new manifest each package will contain specifically only                                                                  
what it needs to run, providing us with finer granularity.

There are some caveats which we try to deal with, such as repeated files on different                                                            
packages, packages that include folders, wildcards, and special packages.                                                                        
Its also important to note that this method only works for python files, and shared                                                              
libraries. Static libraries, header files and binaries need to be dealt with manually.

Using this script a new package can easily be added like this:
 - If a user wants to add a new package all that has to be done is
   modify the python2-manifest.json file, and add the required file(s)
   to the FILES list, the script should handle all the rest.
   Real example:
   "webbrowser": {
       "files": ["${libdir}/python2.7/lib-dynload/webbrowser.py"],
       "rdepends": [],
       "summary": "Python Web browser support"}

 Run bitbake python -c create_manifest and the resulting manifest
 should  be completed after a few seconds, showing something like:
   "webbrowser": {
      "files": ["${libdir}/python2.7/webbrowser.py"],
      "rdepends": ["core","fcntl","io","pickle","shell","subprocess"],
      "summary": "Python Web browser support"}

It also fixes several errors we didnt even know we had:
 - Fixes python-core for dependencies, e.g.
   When python is run on an image, it TRIES to import everything it needs,
   but it doesnt necessarily fails when it doesnt find something, so even if
   we didnt know, we had errors like (trimmed on purpose):
   # trying /usr/lib/python2.7/_locale.so
   # trying /usr/lib/python2.7/lib-dynload/_locale.so
   # trying /usr/lib/python2.7/_sysconfigdata.so

   while it didnt complain about _locale it should have imported it,
   after creating a new manifest with the automated script we get:

   # trying /usr/lib/python2.7/lib-dynload/_locale.so
   dlopen("/usr/lib/python2.7/lib-dynload/_locale.so", 2);
   import _locale # dynamically loaded from /usr/lib/python2.7/lib-dynload/_locale.so

The python2 and python3 versions differ on its core functionality in some bits:
 - Python3 handles precompiled bytecode files  (*.pyc) differently.
   for this reason and since we are cross compiling, wildcards couldnt be
   avoided on python3 (See PEP #3147 [1]).
   Both the manifest and the manifest creation script handle this
   differently, the manifest for python3 has an extra field for cached
   files, which is how it lets the user install the cached files or not
   via : INCLUDE_PYCS = "1" on their local.conf.
 - Shared libs nomenclature also changed on python3, so again, we use
   wildcards to deal with this issue ( See PEP #3149 [2]):
 - Fixes python3 manifest, python3-core should be base and everything
   should depend on it, hence several packages were deleted:
   python3-enum, re, gdbm, subprocess, signal, readline.

 - Rebased for 2.5 M1.

The following changes since commit 800b2856cf57d5d216d1403643d0b9f22c97bc90:

  mdadm: Fix build with gcc < 7 (2017-10-17 17:25:17 +0100)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib aehs29/pypack_rebase

Alejandro Hernandez (4):
  python: Restructure python packaging and replace it with autopackaging
  python3: Restructure python3 packaging and replace it with
  python3: fix RDEPENDS on several recipes, due to non-existent python3
  hwlatdetect: fix RDEPENDS to avoid QA failures

 meta/recipes-core/libxml/libxml2_2.9.5.bb          |    2 +-
 .../bootchart2/bootchart2_0.14.8.bb                |    2 +-
 meta/recipes-devtools/dnf/dnf_2.6.3.bb             |   26 +-
 meta/recipes-devtools/gdb/gdb-cross-canadian.inc   |    4 +-
 .../opkg-utils/opkg-utils_0.3.5.bb                 |    2 +-
 .../python-numpy/python3-numpy_1.13.3.bb           |    2 -
 .../python/python-2.7-manifest.inc                 |  287 -----
 .../python/python-3.5-manifest.inc                 |  283 -----
 .../python/python-native-3.5-manifest.inc          |   10 -
 .../python/python-native_2.7.13.bb                 |   24 +-
 .../python/python/create_manifest2.py              |  277 +++++
 .../python/python/get_module_deps2.py              |  112 ++
 .../python/python/python2-manifest.json            | 1033 ++++++++++++++++++
 .../python/python/sitecustomize.py                 |    8 -
 .../recipes-devtools/python/python3-async_0.6.2.bb |    2 +-
 meta/recipes-devtools/python/python3-git_2.1.7.bb  |    2 +-
 .../recipes-devtools/python/python3-gitdb_0.6.4.bb |    2 +-
 .../python/python3-native_3.5.3.bb                 |   30 +-
 meta/recipes-devtools/python/python3-pip_9.0.1.bb  |    1 -
 .../python/python3-pygobject_3.26.0.bb             |    2 +-
 .../python/python3-setuptools_36.5.0.bb            |    3 -
 .../recipes-devtools/python/python3-smmap_0.9.0.bb |    2 +-
 .../python/python3/create_manifest3.py             |  321 ++++++
 .../python/python3/get_module_deps3.py             |  146 +++
 .../python/python3/python3-manifest.json           | 1098 ++++++++++++++++++++
 meta/recipes-devtools/python/python3_3.5.3.bb      |   84 +-
 meta/recipes-devtools/python/python_2.7.13.bb      |   80 +-
 .../qemu/nativesdk-qemu-helper_1.0.bb              |    2 +-
 meta/recipes-graphics/piglit/piglit_git.bb         |    4 +-
 meta/recipes-rt/rt-tests/hwlatdetect_1.1.bb        |    2 +-
 meta/recipes-rt/rt-tests/rt-tests_1.1.bb           |    2 +-
 scripts/contrib/python/generate-manifest-2.7.py    |  421 --------
 scripts/contrib/python/generate-manifest-3.5.py    |  433 --------
 33 files changed, 3228 insertions(+), 1481 deletions(-)
 delete mode 100644 meta/recipes-devtools/python/python-2.7-manifest.inc
 delete mode 100644 meta/recipes-devtools/python/python-3.5-manifest.inc
 delete mode 100644 meta/recipes-devtools/python/python-native-3.5-manifest.inc
 create mode 100644 meta/recipes-devtools/python/python/create_manifest2.py
 create mode 100644 meta/recipes-devtools/python/python/get_module_deps2.py
 create mode 100644 meta/recipes-devtools/python/python/python2-manifest.json
 create mode 100644 meta/recipes-devtools/python/python3/create_manifest3.py
 create mode 100644 meta/recipes-devtools/python/python3/get_module_deps3.py
 create mode 100644 meta/recipes-devtools/python/python3/python3-manifest.json
 delete mode 100755 scripts/contrib/python/generate-manifest-2.7.py
 delete mode 100755 scripts/contrib/python/generate-manifest-3.5.py


More information about the Openembedded-core mailing list