Upstream Python is moving to pyproject.toml rather than setup.py and
wheels rather than eggs. The main place this is documented is in
PEP-517[1]. Installing packages with setup.py install is deprecated.

We have a choice with the 'kirkstone' LTS release of introducing this
series (and fixing any fall out during the stabilization period) or
living with deprecated Python packaging for the duration of the LTS (at
a minimum 2 years). This seems to be a compelling enough reason to bring
in this series, even if it results in a slight slip of the 'feature
freeze', which is officially tomorrow, Monday 21 Feb, 2022.

This series bootstraps python3-setuptools-native,
python3-flit-core-native, python3-wheel-native and python3-pip-native,
simply unzipping the wheel into PYTHON_SITEPACKAGES_DIR. This allows all
other recipes to be installed with pip, as intended by upstream.

Three new classes are introduced:
(1) flit_core which builds PEP-517 compliant packages which use
flit_core.buildapi as their build-backend.
(2) setuptools_build_meta which builds PEP-517 compliant packages which use
setuptools_build_meta as their build-backend.
(3) pip_install_wheel, which simply installs wheels built by other
classes.

The original setuptools3.bbclass do_compile is modified to
'setup.py bdist_wheel' (from python3-wheel-native) rather than the deprecated
'setup.py build' mechanism.

The original setuptools3.bbclass do_install is modified to
use the pip_install_wheel.bbclass to install wheels with pip rather than
the deprecated 'setup.py install' mechanism.

For pip install, we disable dependency checking (--no-deps) because it
proves to be brittle in our environment without providing full wheel
cache that seems like overhead that we do not need. We also disable
fetching wheels from pypi (--no-index). This is no different than our
normal workflow of requiring dependencies to be defined in a recipe.

Missing from this series is documentation of the new features, classes
and variables (this will be done under separate cover).

Also desired and previously discussed in YP meetings is a helper
function to warn that a recipe which currently inherits setuptools3
but has a pyproject.toml and defines a build-system.build-backend should
instead use of the new PEP-517 classes. This function is still in progress.

During testing, recipes which use python3-setuptools-scm are failing to
determine the version of the python package, For several recipes, the fix
was to use pyproject.toml and PEP-517 build backends. To be sure, other recipes
are still going to fail for the short term, until the root cause can be
determined. This was not seen again in v2 builds.

Another PEP-517 build-backend, poetry.core.masonry.api has been
identified in recipes in meta-python, but not in oe-core. The
python3-poetry-core recipe is WIP, but brings in a number of
dependencies that are perhaps not worth it to bring into oe-core at this
time. Rather we can support these recipes in meta-python?

Fixes in v2 include passing --root=${D} to pip install (rather than --target)
and also passing --prefix so that scripts are (usually) installed in the proper
location. pip install also 'helpfully' overwrites python3 intepreter with nativepython3
in #!, so add a fix to return the values to what is actually correct (similar to what is
done in setuptools3.bbclass). Also several recipes still needed PIP_INSTALL_PACKAGE defined.

Fixes in v3 include:
* passing --force-reinstall to pip install to ensure FILES can be
populated.
* passing --no-cache to pip install to avoid a (harmless) warning about
$HOME
* Fix check for ${D}${bindir}/pip in python3-pip-native (-e vs -f)
* Properly note that python3-wheel recipe is 0.37.1 not 0.37.0

[YOCTO #14638]

[1] https://www.python.org/dev/peps/pep-0517/

The following changes since commit 41ae26ecab731f2fb6593df143476c94183be209:

releases: update to include 3.4.2 (2022-02-21 16:46:22 +0000)

are available in the Git repository at:

git://git.yoctoproject.org/git/poky-contrib timo/bootstrap-wheels
http://git.yoctoproject.org/cgit.cgi/git/poky-contrib/log/?h=timo/bootstrap-wheels

Tim Orling (32):
python3-wheel: move 0.37.1 from meta-python
python3-flit-core: add recipe for 3.6.0
flit_core.bbclass: add helper for newer python packaging
python3-wheel: inherit flit_core
pip_install_wheel.bbclass: add helper class
setuptools_build_meta.bbclass: add helper class
python3-pip: inherit setuptools_build_meta
python3-attrs: inherit setuptools_build_meta
python3-git: inherit setuptools_build_meta
python3-pytest: inherit setuptools_build_meta
python3-setuptools-scm: inherit setuptools_build_meta
python3-zipp: inherit setuptools_build_meta
python3-iniconfig: inherit setuptools_build_meta
python3-py: inherit setuptools_build_meta
python3-pluggy: inherit setuptools_build_meta
python3-setuptools: inherit setuptools_base_meta
setuptools3.bbclass: refactor for wheels
python3-more-itertools: set PIP_INSTALL_PACKAGE
meson: inherit setuptools_build_meta
python3-libarchive-d: set PIP_INSTALL_PACKAGE
python3-smartypants: patch hash bang to python3
python3-scons: merge -native recipe
python3-subunit: merge inc; set PIP_INSTALL_PACKAGE
python3-magic: set PIP_INSTALL_PACKAGE
bmap-tools: set PIP_INSTALL_PACKAGE, BASEVER
asciidoc: set PIP_INSTALL_PACKAGE
gi-docgen: set PIP_INSTALL_PACKAGE
numpy: set PIP_INSTALL_PACKAGE
python3-dbusmock: set PIP_INSTALL_PACKAGE
python3-mako: inherit setuptools_build_meta
python3-packaging: inherit setuptools_build_meta
python3-nose: drop recipe

meta/classes/flit_core.bbclass | 16 +++
meta/classes/pip_install_wheel.bbclass | 39 +++++++
meta/classes/setuptools3.bbclass | 20 ++--
meta/classes/setuptools_build_meta.bbclass | 18 ++++
meta/conf/distro/include/maintainers.inc | 3 +-
meta/recipes-devtools/meson/meson_0.61.2.bb | 4 +-
meta/recipes-devtools/python/python-nose.inc | 18 ----
.../python/python-subunit.inc | 14 ---
.../python/python3-attrs_21.4.0.bb | 2 +-
.../python/python3-dbusmock_0.25.0.bb | 1 +
.../python/python3-flit-core_3.6.0.bb | 41 +++++++
.../python/python3-git_3.1.26.bb | 2 +-
.../python/python3-iniconfig_1.1.1.bb | 4 +-
.../python/python3-libarchive-c_4.0.bb | 2 +
.../python/python3-magic_0.4.25.bb | 1 +
.../python/python3-mako_1.1.6.bb | 2 +-
.../python/python3-more-itertools_8.12.0.bb | 2 +
.../python/python3-nose_1.3.7.bb | 2 -
.../python/python3-numpy_1.22.2.bb | 1 +
.../python/python3-packaging_21.3.bb | 2 +-
.../python/python3-pip_22.0.3.bb | 19 +++-
.../python/python3-pluggy_1.0.0.bb | 2 +-
.../python/python3-py_1.11.0.bb | 2 +-
.../python/python3-pytest_7.0.1.bb | 4 +-
.../python/python3-scons-native_4.3.0.bb | 7 --
.../python/python3-scons_4.3.0.bb | 13 ++-
.../python/python3-setuptools-scm_6.4.2.bb | 2 +-
.../python/python3-setuptools_59.5.0.bb | 19 +++-
.../0001-Change-hash-bang-to-python3.patch | 47 ++++++++
.../python/python3-smartypants_2.0.0.bb | 1 +
...001-setup.py-use-vendored-_distutils.patch | 27 +++++
.../python/python3-strict-rfc3339_0.7.bb | 4 +-
.../python/python3-subunit_1.4.0.bb | 17 ++-
...roject.toml-from-flit-backend-branch.patch | 100 ++++++++++++++++++
.../python/python3-wheel_0.37.1.bb | 24 +++++
.../python/python3-zipp_3.7.0.bb | 2 +-
.../asciidoc/asciidoc_10.1.1.bb | 1 +
meta/recipes-gnome/gi-docgen/gi-docgen_git.bb | 1 +
.../{bmap-tools_3.6.bb => bmap-tools_git.bb} | 6 +-
39 files changed, 414 insertions(+), 78 deletions(-)
create mode 100644 meta/classes/flit_core.bbclass
create mode 100644 meta/classes/pip_install_wheel.bbclass
create mode 100644 meta/classes/setuptools_build_meta.bbclass
delete mode 100644 meta/recipes-devtools/python/python-nose.inc
delete mode 100644 meta/recipes-devtools/python/python-subunit.inc
create mode 100644 meta/recipes-devtools/python/python3-flit-core_3.6.0.bb
delete mode 100644 meta/recipes-devtools/python/python3-nose_1.3.7.bb
delete mode 100644 meta/recipes-devtools/python/python3-scons-native_4.3.0.bb
create mode 100644 meta/recipes-devtools/python/python3-smartypants/0001-Change-hash-bang-to-python3.patch
create mode 100644 meta/recipes-devtools/python/python3-strict-rfc3339/0001-setup.py-use-vendored-_distutils.patch
create mode 100644 meta/recipes-devtools/python/python3-wheel/0001-Backport-pyproject.toml-from-flit-backend-branch.patch
create mode 100644 meta/recipes-devtools/python/python3-wheel_0.37.1.bb
rename meta/recipes-support/bmap-tools/{bmap-tools_3.6.bb => bmap-tools_git.bb} (86%)

--
2.30.2

I got a kind of general question about this patch series and all the followups: is this still considered to go into the next release?

I'm a bit worried about the fallout of this pretty invasive change - even though I see that at some point it needs to be done.

My understanding is that the "classic" way will stop with Python 3.12, which doesn't apply to next LTS release - as this will likely remain on 3.10.
Only downside will be that manual helper files for updates of packages that are lacking a setup.py needs to be provided (there are already a few examples how to do it) - not a big deal if you'd ask me.

So what's the stand of the project regarding this issue - also keeping in mind that I think it's already past feature freeze?

On Thu, 2022-02-24 at 16:52 +0100, Konrad Weihmann wrote:

I got a kind of general question about this patch series and all the
followups: is this still considered to go into the next release?

It is still being considered, yes.

I'm a bit worried about the fallout of this pretty invasive change -
even though I see that at some point it needs to be done.

My understanding is that the "classic" way will stop with Python 3.12,
which doesn't apply to next LTS release - as this will likely remain on
3.10.
Only downside will be that manual helper files for updates of packages
that are lacking a setup.py needs to be provided (there are already a
few examples how to do it) - not a big deal if you'd ask me.

So what's the stand of the project regarding this issue - also keeping
in mind that I think it's already past feature freeze?

This is a tough one to make a decision on and I am conflicted. The change was
flagged up a while ago and has been regularly talked about. It is also something
we all agree will have to happen at some point.

The change is late and has issues but there was a base patchset sent before the
freeze deadline.

This isn't the final release point, it is the point where we stop taking new
invasive changes and stabilise and I think it important to keep that in mind.

Stepping back and thinking about the big picture (and e.g. the ability to take
security fixes into the LTS), I'm leaning towards trying to get it in. One other
consideration is having large delta between the LTS and onging development and
I'd prefer to minimise this particular difference if it is practical to do so.

I believe we have identified and fixed the majority of the issues that have
shown up in automated testing.

I haven't made a final decision but I am keeping an open mind on it and would
really prefer to get it merged. There are other issues being worked in parallel
which also would block the M3 build which does give time to resolve this one.

Cheers,

Richard

On 25.02.22 14:16, Richard Purdie wrote:

On Thu, 2022-02-24 at 16:52 +0100, Konrad Weihmann wrote:

I got a kind of general question about this patch series and all the
followups: is this still considered to go into the next release?

It is still being considered, yes.

I'm a bit worried about the fallout of this pretty invasive change -
even though I see that at some point it needs to be done.

My understanding is that the "classic" way will stop with Python 3.12,
which doesn't apply to next LTS release - as this will likely remain on
3.10.
Only downside will be that manual helper files for updates of packages
that are lacking a setup.py needs to be provided (there are already a
few examples how to do it) - not a big deal if you'd ask me.

So what's the stand of the project regarding this issue - also keeping
in mind that I think it's already past feature freeze?

This is a tough one to make a decision on and I am conflicted. The change was
flagged up a while ago and has been regularly talked about. It is also something
we all agree will have to happen at some point.
The change is late and has issues but there was a base patchset sent before the
freeze deadline.
This isn't the final release point, it is the point where we stop taking new
invasive changes and stabilise and I think it important to keep that in mind.
Stepping back and thinking about the big picture (and e.g. the ability to take
security fixes into the LTS), I'm leaning towards trying to get it in. One other
consideration is having large delta between the LTS and onging development and
I'd prefer to minimise this particular difference if it is practical to do so.

Your argumentation does make sense, but I have to disagree on this particular point.
The using pip as the default installer and therefore wheels is something that will never (hopefully) get backported, so bringing this change in automatically builds up a huge delta to any other branch - thus here you would create a situation that (I agree) should be avoided.

Also moving around a few classes and recipe between core and meta-python, will either bind users to including meta-python in every setup or will create situation were people will try to work around these changes.

I see that this feature has been promised - and it might be bad for the project's reputation to drop it - still if one would ask me, I would prefer to delay it to the next release.
One potential option would be to offer that one (once mature and tested will a broad set of layers) as a mixin-layer, which then could be used with kirkstone LTS

I believe we have identified and fixed the majority of the issues that have
shown up in automated testing.
I haven't made a final decision but I am keeping an open mind on it and would
really prefer to get it merged. There are other issues being worked in parallel
which also would block the M3 build which does give time to resolve this one.
Cheers,
Richard

On Fri, 2022-02-25 at 14:27 +0100, Konrad Weihmann wrote:

On 25.02.22 14:16, Richard Purdie wrote:

On Thu, 2022-02-24 at 16:52 +0100, Konrad Weihmann wrote:

I got a kind of general question about this patch series and all the
followups: is this still considered to go into the next release?

It is still being considered, yes.

I'm a bit worried about the fallout of this pretty invasive change -
even though I see that at some point it needs to be done.

My understanding is that the "classic" way will stop with Python 3.12,
which doesn't apply to next LTS release - as this will likely remain on
3.10.
Only downside will be that manual helper files for updates of packages
that are lacking a setup.py needs to be provided (there are already a
few examples how to do it) - not a big deal if you'd ask me.

So what's the stand of the project regarding this issue - also keeping
in mind that I think it's already past feature freeze?

This is a tough one to make a decision on and I am conflicted. The change was
flagged up a while ago and has been regularly talked about. It is also something
we all agree will have to happen at some point.

The change is late and has issues but there was a base patchset sent before the
freeze deadline.

This isn't the final release point, it is the point where we stop taking new
invasive changes and stabilise and I think it important to keep that in mind.

Stepping back and thinking about the big picture (and e.g. the ability to take
security fixes into the LTS), I'm leaning towards trying to get it in. One other
consideration is having large delta between the LTS and onging development and
I'd prefer to minimise this particular difference if it is practical to do so.

Your argumentation does make sense, but I have to disagree on this
particular point.
The using pip as the default installer and therefore wheels is something
that will never (hopefully) get backported, so bringing this change in
automatically builds up a huge delta to any other branch - thus here you
would create a situation that (I agree) should be avoided.

Also moving around a few classes and recipe between core and
meta-python, will either bind users to including meta-python in every
setup or will create situation were people will try to work around these
changes.

I see that this feature has been promised - and it might be bad for the
project's reputation to drop it - still if one would ask me, I would
prefer to delay it to the next release.
One potential option would be to offer that one (once mature and tested
will a broad set of layers) as a mixin-layer, which then could be used
with kirkstone LTS

It is a tough call without a right answer and one I ultimately have to make
based on experience and judgement weighing up the pros and cons.

I did just merge the series as I do think it is now about as ready as things get
without merging and exposing to the wider userbase. We do have time before
release.

FWIW the distutils classes have been showing warnings for a while now so it
shouldn't surprise anyone that they've been removed, that was planned regardless
of the other changes.

Cheers,

Richard