On Sat, Jul 2, 2022 at 4:13 AM Richard Purdie
<richard.purdie@...> wrote:

I admit I've been putting off this topic for a long time. I actually
wrote up something about it about 5 years ago. I sent it to four people
to get some opinions. The six opinions I got back made me despair. i
have had a while to think about it though.

The challenge is that everyone has a workflow they like today and they
will tend to dislike anything that looks different. For that reason I
think we need a high level tool which can work with the different
approaches. Once we have that, I suspect we'll see some grow stronger
and some will wither off, which is probably as it should be through
natural evolution. The approach Alex has taken here does head in that
direction but I'm not sure it goes quite far enough to get the fraction
of users we need on board.

I'm not going to comment directly on Alex's proposal at this point, I'd
like to put some ideas out there and see what people think of them. I
think you'll see that I'm in agreement with the idea/direction but I
have a slightly bigger idea of how to do it, which will both be harder
to implement but also have a better chance of getting more people on
board.

So, let me go into a proposal of what I think the tool we need looks
like. I propose we create a new tool, lets call it "oe-setup". It is a
standalone project in it's own git repo and it's first command would be
"init" with a command line looking roughly like:

oe-setup init <project-name>
oe-setup init <project-name> <target-dir>
oe-setup init <project-name> <config-name> <target-dir>
oe-setup init <project-name> <config-url> <target-dir>
oe-setup init <config-url> <target-dir>

The idea being that the repository has some "pre-canned" idea of
certain project names, e.g. poky, yoe and whatever else we decide to
support "out the box" (criteria tbd). Those project names have a
default pre-canned config, or set of configs (e.g. taking branch/series
names) so I'd want to see these all work:

My proposal would be to try to reuse Linux kconfig. Users know how to
run "make oldconfig", "make menuconfig", etc and I believe all key
distro / machine configuration along with which meta layers should be
enabled, which branch to use, etc can all be captured within a kconfig
.config file. The task at hand would then become writing a tool to
translate from a .config file to a set of distro, machine, local.conf
config files (and an image recipe?) + drive whichever tool will be
used to fetch meta layer git repos.

The "pre-canned" knowledge of poky, yoe, etc then just becomes a set
of alternative defconfig files. Although if we can generate distro
configs on the fly from a defconfig file then separate meta layers to
hold distro configs starts to become redundant...

As a reminder, here's the Buildroot README:

https://github.com/buildroot/buildroot/blob/master/README

With those first 2 steps a Buildroot user can effectively create their
own custom distro and machine config... all from within the
user-friendly menuconfig environment. The OE equivalent (Yocto Quick
Start) is not only more complex and only tells a user how to build
someone else's preset configurations. Creating a custom distro and
machine config in OE is beyond the quick start and requires a full
knowledge of bitbake syntax etc, etc.

oe-setup init poky .
oe-setup init poky dunfell .
oe-setup init poky ./my-local-config.json .
oe-setup init poky http://someserver/my-remote-config.json .

We'd also allow something not in the default like of project names to
be used directly with a url, or maybe added with an "add-project"
command. This could work against a user local config file, a bit like
git does with global config and adding remotes to a repo.

You'll note I haven't made any mention of the tooling these use. The
reason is that we don't actually care. I'd propose we teach the tool
about a few common standards (kas, submodules, repo) in the form of
plugins and then hand off to those tools to do the setup. I'd also
propose we develop a "native" form where they perhaps aren't needed.
the nice thing is we can have several "native" forms if needed so if
one approach isn't working or we need to change it, we can.

We may also want to consider an optional "sub-config" parameter which
gets passed along to the underlying tool.

One of my conclusions after thinking about this for a long time is we
have a bootstrap problem. If everyone used git and git worked
everywhere, things would be easier. They don't and it doesn't. My
evidence? The bitbake fetcher. Much of the ugliness the bitbake fetcher
deals with applies to layers as well. Some people need proxy support,
some need mirror tarballs, some need floating revisions, some need
complete lock down and so on.

We could simplify the problem and just say those users "can manage". If
we do that, we're giving up at the first hurdle on the idea what we can
have a (mostly) universal tool. I'm not sure I'm quite ready to do
that.

Like most more seasoned developers, I dislike code duplication. We do
have the fetcher code, which knows something about these issues and it
even has a test suite with decent coverage. Most users already know how
to configure it too.

Whilst I don't like it particularly, I'd propose we include a chunk of
bitbake inside oe-setup (maybe just lib/bb?). How needs discussion. I
nearly wrote combo-layer here but people might think I'm serious :).
More seriously, git subtree might be a potential option. We could then
have bitbake master, or a chunk of it, available to the tool. This
would then allow us to potentially get out of a variety of different
firewall/mirror situations, if the appropriate backends were right.
This gives us the option for fix bugs in the fetcher and update oe-
setup and still use specific versions of bitbake for the different
releases as needed.

I'd propose oe-setup be a different kind of project to bitbake,
hopefully something we could make available via pip for example, again
to try and help the bootstrap issue for people behind firewalls etc.
although I would want it usable standalone too. It would be designed to
be a separate standalone tool installed somewhere once by a user rather
than associated with a particular build/metadata set like bitbake is.

Beyond init, the question is probably updating. There are probably two
options, one is an update command to oe-setup and the other is
deferring to the underlying tool and it should be possible to use
either depending on the underlying tool and the circumstances.

As for configuration of the build, I've wondered if we add a new level
of config file, basically dedicating a config file "slot" to the setup
tooling where config from the setup tool would land (i.e. like
auto.conf but dedicated to this). I need to think about this piece a
bit more.

The other side of this would be the generation of the config(s). I'm
less worried about this piece, it would be done by the people who can
deal with more complexity and I'm sure we could figure it out. I used
"json" in the config urls above but I'm aware that yaml is going to be
a discussion there too. From the oe-setup perspective, it may support
multiple options since once it knows the tool to pass it off to, it
doesn't really need to know more.

I'm sure there is more I could write on this topic but I'll stop there
and see what people think.

Cheers,

Richard