Re: [Automated-testing] RFC Linter in meta-openembedded
Thanks for looping back to this.
On Sat, 2022-04-16 at 13:28 +0200, Marius Kriegerowski wrote:
After some time I found the time and took a deeper dive into patchtest. It’s notI've wondered if we do need to re-invent it but for different reasons. It
currently integrates with patchwork but for example it may be better to have it
work off lore and it may be better using some of the more modern patchwork
tooling. One example I was pointed at was:
With regard to unitest vs pytest, please do use unittest. The reason I say that
is the rest of the project's testing runs within unitest so I'd prefer to have
one standard rather than two. You can see it here:
(there are selftests, runtime tests, sdk tests, eSDK tests, performance tests
and so on).
Patchtest is hardly documented which makes it hard to follow. Or is there anySadly there isn't :(.
There are selftests but they don’t work:That directory appears to contain a load of training/test data. I don't care so
much about the actual test mechanism but surely the old training data is a good
place to start to test any new instance that is brought up?
I’m assuming that in the CI context the `--json` flag is used to produce jsonIt looks like you're working on "patchtest-oe" from
https://git.yoctoproject.org/patchtest-oe/ . There is a second piece to the
puzzle, a harder piece unfortunately which is
The latter is the piece which is effectively the glue logic between patchwork
I'm not sure what you're aiming for as an end result. To recap, what used to
happen is that patchwork+patchtest would see new patches on the mailing list,
pick them up, run the tests in patchtest-oe against them, then reply on the
mailing list if there were any issues detected. We would like to get back to
Unfortunately we were running a forked/hacked up version of patchwork which we
couldn't update. We ended up switching to a vanilla upstream version of it so it
is now here: https://patchwork.yoctoproject.org/ however that broke patchtest.
What we'd like to do is:
a) Get patchtest back up and running against new patches on the mailing list in
some form. As I hinted at above, this may mean using a different approach to
what patchtest used to do.
b) Move the actual tests in patchtest-oe to meta/lib/oeqa/ in OE-Core as a new
set of tests.
c) Add a script in OE-Core so that you can run the tests in patchtest-oe (now in
OE-Core) against a patch before you send it to the mailing list
d) Anyone else could then reuse these tests as part of CI, e.g. on
I managed to make it run on my machine but even if one test fails, it returnsI suspect patchtest would have code to read and handle that.
I believe that patchtest had some minimal bitbake environment which would have
allowed some of the skipped tests to run. I'm not an expert on this and I don't
remember how the handoff happened with it though.
Hope this offers at least some guideance on where things were and where we need
to be, at least in my view! :)