On 25/12/2022 19:13:43-0500, Randy MacLeod wrote:

On 2022-11-25 08:00, Richard Purdie via lists.openembedded.org wrote:

Hi Qi,

On Fri, 2022-11-25 at 05:56 +0000, Chen Qi wrote:

The AB is actually running 'all' tests.

The '--skip-tests (-R)' option means 'Run all (unhidden) tests except
the ones specified.', according to its help message. This is also its
actual effect for now. You can see there's an implicit 'run all
tests' meaning in this option.
The '-T (--exclude-tag)' options means 'Exclude all (unhidden) tests
that match any of the specified tag(s)', according to its help
message.

So the AB's oe-selftest command means: execute all tests except the
ones tagged with 'machine' and 'toolchain-user', and also skip
'distrodata.Distrodata.test_checkpkg
buildoptions.SourceMirroring.test_yocto_source_mirror reproducible'.

You're right, a different build has -t machine and I'm confusing the
two, sorry.

I don't really like making changes which require lockstep changes to
the autobuilder configuration but I can see why we could do with doing
so here.

I was thinking there should be symettry between -r and -R like there is
with -T and -t but that also probably doesn't make sense when you think
about it more.

Alex: We'll probably have to work out how to make this change...

Nudge and Happy Holidays.

Yes, this is still on my radar.


--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

On 2022-11-25 08:00, Richard Purdie via lists.openembedded.org wrote:

Hi Qi,
On Fri, 2022-11-25 at 05:56 +0000, Chen Qi wrote:

The AB is actually running 'all' tests.

The '--skip-tests (-R)' option means 'Run all (unhidden) tests except
the ones specified.', according to its help message. This is also its
actual effect for now. You can see there's an implicit 'run all
tests' meaning in this option.
The '-T (--exclude-tag)' options means 'Exclude all (unhidden) tests
that match any of the specified tag(s)', according to its help
message.

So the AB's oe-selftest command means: execute all tests except the
ones tagged with 'machine' and 'toolchain-user', and also skip
'distrodata.Distrodata.test_checkpkg
buildoptions.SourceMirroring.test_yocto_source_mirror reproducible'.

You're right, a different build has -t machine and I'm confusing the
two, sorry.
I don't really like making changes which require lockstep changes to
the autobuilder configuration but I can see why we could do with doing
so here.
I was thinking there should be symettry between -r and -R like there is
with -T and -t but that also probably doesn't make sense when you think
about it more.
Alex: We'll probably have to work out how to make this change...

Nudge and Happy Holidays.

../Randy

Cheers,
Richard

--
# Randy MacLeod
# Wind River Linux

Hi Qi,

On Fri, 2022-11-25 at 05:56 +0000, Chen Qi wrote:

The AB is actually running 'all' tests.

The '--skip-tests (-R)' option means 'Run all (unhidden) tests except
the ones specified.', according to its help message. This is also its
actual effect for now. You can see there's an implicit 'run all
tests' meaning in this option.
The '-T (--exclude-tag)' options means 'Exclude all (unhidden) tests
that match any of the specified tag(s)', according to its help
message.

So the AB's oe-selftest command means: execute all tests except the
ones tagged with 'machine' and 'toolchain-user', and also skip
'distrodata.Distrodata.test_checkpkg
buildoptions.SourceMirroring.test_yocto_source_mirror reproducible'.

You're right, a different build has -t machine and I'm confusing the
two, sorry.

I don't really like making changes which require lockstep changes to
the autobuilder configuration but I can see why we could do with doing
so here.

I was thinking there should be symettry between -r and -R like there is
with -T and -t but that also probably doesn't make sense when you think
about it more.

Alex: We'll probably have to work out how to make this change...

Cheers,

Richard

Hi Richard,

The AB is actually running 'all' tests.

The '--skip-tests (-R)' option means 'Run all (unhidden) tests except the ones specified.', according to its help message. This is also its actual effect for now. You can see there's an implicit 'run all tests' meaning in this option.
The '-T (--exclude-tag)' options means 'Exclude all (unhidden) tests that match any of the specified tag(s)', according to its help message.

So the AB's oe-selftest command means: execute all tests except the ones tagged with 'machine' and 'toolchain-user', and also skip 'distrodata.Distrodata.test_checkpkg buildoptions.SourceMirroring.test_yocto_source_mirror reproducible'.

Regards,
Qi

On Thu, 2022-11-24 at 14:35 +0100, Alexandre Belloni via
lists.openembedded.org wrote:

On 24/11/2022 13:10:51+0000, Chen Qi wrote:

Hi Alexandre,

The command line is: oe-selftest --skip-tests distrodata.Distrodata.test_checkpkg buildoptions.SourceMirroring.test_yocto_source_mirror reproducible -T machine -T toolchain-user -T toolchain-system -j 15

The '--skip-tests' option is the long format of '-R'.

Right, sorry about that :) I actually checked we didn't have any -R
before sending the patch to the AB... I'll discuss that with RP.

The autobuilder is not running "all" tests though, it is running two
groups of tests, -T machine and -T toolchain-user. Adding -a here
doesn't really make sense.

Cheers,

Richard

On 24/11/2022 13:10:51+0000, Chen Qi wrote:

Hi Alexandre,

The command line is: oe-selftest --skip-tests distrodata.Distrodata.test_checkpkg buildoptions.SourceMirroring.test_yocto_source_mirror reproducible -T machine -T toolchain-user -T toolchain-system -j 15

The '--skip-tests' option is the long format of '-R'.

Right, sorry about that :) I actually checked we didn't have any -R
before sending the patch to the AB... I'll discuss that with RP.

Regards,
Qi

-----Original Message-----
From: Alexandre Belloni <alexandre.belloni@...>
Sent: Thursday, November 24, 2022 9:02 PM
To: Chen, Qi <Qi.Chen@...>
Cc: openembedded-core@...
Subject: Re: [OE-core][PATCH] selftest: allow '-R' and '-r' be used together

On 24/11/2022 12:24:38+0000, Chen Qi wrote:

Hi Alexandre,

Yes, I'm aware of this behavior regression.
With this patch, 'oe-selftest -R xxx' needs to be changed to 'oe-selftest -a -R xxx'.

Yes, my point is that there is no -R in the command line used by the AB

I'm not sure if this is useful for other people. Because how people use oe-selftest is unknown to me.
For me, I want to run something like 'oe-selftest -r A -R A.B.C'.

Regards,
Qi

-----Original Message-----
From: Alexandre Belloni <alexandre.belloni@...>
Sent: Thursday, November 24, 2022 7:35 PM
To: Chen, Qi <Qi.Chen@...>
Cc: openembedded-core@...
Subject: Re: [OE-core][PATCH] selftest: allow '-R' and '-r' be used
together

Hello,

On 23/11/2022 18:55:28-0800, Chen Qi wrote:

For now, there's no easy way to skip some specified tests when using
'oe-selftest -r'. This is because '-R' and '-r' are in the mutually
exclusive group. Looking at the codes, the codes have already
allowed running specified tests and skipping specified tests at the same time.
So move '-R' out of the mutually exclusive group and change its help
message.

After this change, the following command could be used.
oe-selftest -r A -R A.B.C

This does introduce a behavior regression.
Previous 'oe-selftest -R xxx' needs to now be changed to
'oe-selftest -a -R xxx'.

This fails on the autobuilders which uses:

oe-selftest --skip-tests distrodata.Distrodata.test_checkpkg
buildoptions.SourceMirroring.test_yocto_source_mirror reproducible -T
machine -T toolchain-user -T toolchain-system -j 15

https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/4395
/steps/14/logs/stdio

Signed-off-by: Chen Qi <Qi.Chen@...>
---
meta/lib/oeqa/selftest/context.py | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/meta/lib/oeqa/selftest/context.py
b/meta/lib/oeqa/selftest/context.py
index 78c7a467e2..c7dd03ce37 100644
--- a/meta/lib/oeqa/selftest/context.py
+++ b/meta/lib/oeqa/selftest/context.py
@@ -154,9 +154,6 @@ class OESelftestTestContextExecutor(OETestContextExecutor):
group.add_argument('-a', '--run-all-tests', default=False,
action="store_true", dest="run_all_tests",
help='Run all (unhidden) tests')
- group.add_argument('-R', '--skip-tests', required=False, action='store',
- nargs='+', dest="skips", default=None,
- help='Run all (unhidden) tests except the ones specified. Format should be <module>[.<class>[.<test_method>]]')
group.add_argument('-r', '--run-tests', required=False, action='store',
nargs='+', dest="run_tests", default=None,
help='Select what tests to run (modules, classes or
test methods). Format should be: <module>.<class>.<test_method>') @@ -171,6 +168,9 @@ class OESelftestTestContextExecutor(OETestContextExecutor):
action="store_true", default=False,
help='List all available tests.')

+ parser.add_argument('-R', '--skip-tests', required=False, action='store',
+ nargs='+', dest="skips", default=None,
+ help='Skip the tests specified. Format should be
+ <module>[.<class>[.<test_method>]]')
parser.add_argument('-j', '--num-processes', dest='processes', action='store',
type=int, help="number of processes to execute in
parallel with")

--
2.37.1

--
Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel
engineering https://bootlin.com

--
Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com

--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Hi Alexandre,

The command line is: oe-selftest --skip-tests distrodata.Distrodata.test_checkpkg buildoptions.SourceMirroring.test_yocto_source_mirror reproducible -T machine -T toolchain-user -T toolchain-system -j 15

The '--skip-tests' option is the long format of '-R'.

Regards,
Qi

On 24/11/2022 12:24:38+0000, Chen Qi wrote:

Hi Alexandre,

Yes, I'm aware of this behavior regression.
With this patch, 'oe-selftest -R xxx' needs to be changed to 'oe-selftest -a -R xxx'.

Yes, my point is that there is no -R in the command line used by the AB

I'm not sure if this is useful for other people. Because how people use oe-selftest is unknown to me.
For me, I want to run something like 'oe-selftest -r A -R A.B.C'.

Regards,
Qi

-----Original Message-----
From: Alexandre Belloni <alexandre.belloni@...>
Sent: Thursday, November 24, 2022 7:35 PM
To: Chen, Qi <Qi.Chen@...>
Cc: openembedded-core@...
Subject: Re: [OE-core][PATCH] selftest: allow '-R' and '-r' be used together

Hello,

On 23/11/2022 18:55:28-0800, Chen Qi wrote:

For now, there's no easy way to skip some specified tests when using
'oe-selftest -r'. This is because '-R' and '-r' are in the mutually
exclusive group. Looking at the codes, the codes have already allowed
running specified tests and skipping specified tests at the same time.
So move '-R' out of the mutually exclusive group and change its help
message.

After this change, the following command could be used.
oe-selftest -r A -R A.B.C

This does introduce a behavior regression.
Previous 'oe-selftest -R xxx' needs to now be changed to 'oe-selftest
-a -R xxx'.

This fails on the autobuilders which uses:

oe-selftest --skip-tests distrodata.Distrodata.test_checkpkg buildoptions.SourceMirroring.test_yocto_source_mirror reproducible -T machine -T toolchain-user -T toolchain-system -j 15

https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/4395/steps/14/logs/stdio

Signed-off-by: Chen Qi <Qi.Chen@...>
---
meta/lib/oeqa/selftest/context.py | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/meta/lib/oeqa/selftest/context.py
b/meta/lib/oeqa/selftest/context.py
index 78c7a467e2..c7dd03ce37 100644
--- a/meta/lib/oeqa/selftest/context.py
+++ b/meta/lib/oeqa/selftest/context.py
@@ -154,9 +154,6 @@ class OESelftestTestContextExecutor(OETestContextExecutor):
group.add_argument('-a', '--run-all-tests', default=False,
action="store_true", dest="run_all_tests",
help='Run all (unhidden) tests')
- group.add_argument('-R', '--skip-tests', required=False, action='store',
- nargs='+', dest="skips", default=None,
- help='Run all (unhidden) tests except the ones specified. Format should be <module>[.<class>[.<test_method>]]')
group.add_argument('-r', '--run-tests', required=False, action='store',
nargs='+', dest="run_tests", default=None,
help='Select what tests to run (modules, classes or
test methods). Format should be: <module>.<class>.<test_method>') @@ -171,6 +168,9 @@ class OESelftestTestContextExecutor(OETestContextExecutor):
action="store_true", default=False,
help='List all available tests.')

+ parser.add_argument('-R', '--skip-tests', required=False, action='store',
+ nargs='+', dest="skips", default=None,
+ help='Skip the tests specified. Format should be
+ <module>[.<class>[.<test_method>]]')
parser.add_argument('-j', '--num-processes', dest='processes', action='store',
type=int, help="number of processes to execute in
parallel with")

--
2.37.1

--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Hi Alexandre,

Yes, I'm aware of this behavior regression.
With this patch, 'oe-selftest -R xxx' needs to be changed to 'oe-selftest -a -R xxx'.

I'm not sure if this is useful for other people. Because how people use oe-selftest is unknown to me.
For me, I want to run something like 'oe-selftest -r A -R A.B.C'.

Regards,
Qi

Hello,

On 23/11/2022 18:55:28-0800, Chen Qi wrote:

For now, there's no easy way to skip some specified tests when
using 'oe-selftest -r'. This is because '-R' and '-r' are in the
mutually exclusive group. Looking at the codes, the codes have already
allowed running specified tests and skipping specified tests at
the same time. So move '-R' out of the mutually exclusive group
and change its help message.

After this change, the following command could be used.
oe-selftest -r A -R A.B.C

This does introduce a behavior regression.
Previous 'oe-selftest -R xxx' needs to now be changed to
'oe-selftest -a -R xxx'.

This fails on the autobuilders which uses:

oe-selftest --skip-tests distrodata.Distrodata.test_checkpkg buildoptions.SourceMirroring.test_yocto_source_mirror reproducible -T machine -T toolchain-user -T toolchain-system -j 15

https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/4395/steps/14/logs/stdio

Signed-off-by: Chen Qi <Qi.Chen@...>
---
meta/lib/oeqa/selftest/context.py | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/meta/lib/oeqa/selftest/context.py b/meta/lib/oeqa/selftest/context.py
index 78c7a467e2..c7dd03ce37 100644
--- a/meta/lib/oeqa/selftest/context.py
+++ b/meta/lib/oeqa/selftest/context.py
@@ -154,9 +154,6 @@ class OESelftestTestContextExecutor(OETestContextExecutor):
group.add_argument('-a', '--run-all-tests', default=False,
action="store_true", dest="run_all_tests",
help='Run all (unhidden) tests')
- group.add_argument('-R', '--skip-tests', required=False, action='store',
- nargs='+', dest="skips", default=None,
- help='Run all (unhidden) tests except the ones specified. Format should be <module>[.<class>[.<test_method>]]')
group.add_argument('-r', '--run-tests', required=False, action='store',
nargs='+', dest="run_tests", default=None,
help='Select what tests to run (modules, classes or test methods). Format should be: <module>.<class>.<test_method>')
@@ -171,6 +168,9 @@ class OESelftestTestContextExecutor(OETestContextExecutor):
action="store_true", default=False,
help='List all available tests.')

+ parser.add_argument('-R', '--skip-tests', required=False, action='store',
+ nargs='+', dest="skips", default=None,
+ help='Skip the tests specified. Format should be <module>[.<class>[.<test_method>]]')
parser.add_argument('-j', '--num-processes', dest='processes', action='store',
type=int, help="number of processes to execute in parallel with")

--
2.37.1

--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

For now, there's no easy way to skip some specified tests when
using 'oe-selftest -r'. This is because '-R' and '-r' are in the
mutually exclusive group. Looking at the codes, the codes have already
allowed running specified tests and skipping specified tests at
the same time. So move '-R' out of the mutually exclusive group
and change its help message.

After this change, the following command could be used.
oe-selftest -r A -R A.B.C

This does introduce a behavior regression.
Previous 'oe-selftest -R xxx' needs to now be changed to
'oe-selftest -a -R xxx'.

Signed-off-by: Chen Qi <Qi.Chen@...>
---
meta/lib/oeqa/selftest/context.py | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/meta/lib/oeqa/selftest/context.py b/meta/lib/oeqa/selftest/context.py
index 78c7a467e2..c7dd03ce37 100644
--- a/meta/lib/oeqa/selftest/context.py
+++ b/meta/lib/oeqa/selftest/context.py
@@ -154,9 +154,6 @@ class OESelftestTestContextExecutor(OETestContextExecutor):
group.add_argument('-a', '--run-all-tests', default=False,
action="store_true", dest="run_all_tests",
help='Run all (unhidden) tests')
- group.add_argument('-R', '--skip-tests', required=False, action='store',
- nargs='+', dest="skips", default=None,
- help='Run all (unhidden) tests except the ones specified. Format should be <module>[.<class>[.<test_method>]]')
group.add_argument('-r', '--run-tests', required=False, action='store',
nargs='+', dest="run_tests", default=None,
help='Select what tests to run (modules, classes or test methods). Format should be: <module>.<class>.<test_method>')
@@ -171,6 +168,9 @@ class OESelftestTestContextExecutor(OETestContextExecutor):
action="store_true", default=False,
help='List all available tests.')

+ parser.add_argument('-R', '--skip-tests', required=False, action='store',
+ nargs='+', dest="skips", default=None,
+ help='Skip the tests specified. Format should be <module>[.<class>[.<test_method>]]')
parser.add_argument('-j', '--num-processes', dest='processes', action='store',
type=int, help="number of processes to execute in parallel with")

--
2.37.1