QA and release process changes
Richard Purdie
Hi All,
There are a few different discussions I'd like to pull together and talk about
some QA and release process changes.
Firstly, we've just made some changes to the docs build process. This is now
deriving the version information from the git tags in the docs repository. This
means as long as the tags are present, the versioned docs should appear
correctly on the website.
This means changes to the release process but they should be simplifications and
make things easier and simpler for everyone involved. Simply tagging a point
release should enable all the right things to happen. A new release series
should just need changes to set_versions.py in master and then the correct
things should also "just happen". If there are any questions or issues let me
know.
The second thing we've been discussing is streamlining the release process where
it makes sense to, particularly around point releases. Since we started the
current processes the autobuilder has massively improved in test coverage and
we'd like to be more dynamic with point releases.
Experience tells us that the majority of blocking issues in a point release are
found on the autobuilder and that if the autobuilder build is green, the release
is likely going to go ahead. The "real hardware" testing QA does is extremely
valuable and important but in general hasn't been finding issues that block
releases.
As such we'd propose that for point releases:
* we proceed to release after builds pass as green on the autobuilder
or only show well known intermittent failures.
* 'real hardware' QA takes place alongside that release process
* the test results of that extra QA are merged into the release artefacts later
as additional test results
* any issues found in QA would be resolved by fast turnaround of a new release
* release approval would be sent to the TSC as usual but the TSC chair is able
to approve releases with no autobuilder issues flagged unless any of the TSC
raises a concern
In the case of any doubt, the TSC would discuss as currently done.
For milestone releases, the same process as above would apply but QA flagged
issues would usually block the next milestone as we'd want to see issues
resolved.
For main/final project major version changes/releases, the current process
applies with QA testing needing to be completed before the final release is made
and a TSC vote is required.
The intent of the changes is to allow the project to be dynamic where we can and
allow point releases to be sped up whilst still ensuring we do maintain project
quality and testing.
There will be changes needed to the release process to accommodate this but I'm
hoping those aren't too complex and can be figured out. Michael/Chee Yang, if
you do see any potential problems in this change please do let me know and we
can discuss them and try and figure out a solution. I'm also very open to other
ideas for improving things and making them more efficient if/where we can.
Cheers,
Richard
There are a few different discussions I'd like to pull together and talk about
some QA and release process changes.
Firstly, we've just made some changes to the docs build process. This is now
deriving the version information from the git tags in the docs repository. This
means as long as the tags are present, the versioned docs should appear
correctly on the website.
This means changes to the release process but they should be simplifications and
make things easier and simpler for everyone involved. Simply tagging a point
release should enable all the right things to happen. A new release series
should just need changes to set_versions.py in master and then the correct
things should also "just happen". If there are any questions or issues let me
know.
The second thing we've been discussing is streamlining the release process where
it makes sense to, particularly around point releases. Since we started the
current processes the autobuilder has massively improved in test coverage and
we'd like to be more dynamic with point releases.
Experience tells us that the majority of blocking issues in a point release are
found on the autobuilder and that if the autobuilder build is green, the release
is likely going to go ahead. The "real hardware" testing QA does is extremely
valuable and important but in general hasn't been finding issues that block
releases.
As such we'd propose that for point releases:
* we proceed to release after builds pass as green on the autobuilder
or only show well known intermittent failures.
* 'real hardware' QA takes place alongside that release process
* the test results of that extra QA are merged into the release artefacts later
as additional test results
* any issues found in QA would be resolved by fast turnaround of a new release
* release approval would be sent to the TSC as usual but the TSC chair is able
to approve releases with no autobuilder issues flagged unless any of the TSC
raises a concern
In the case of any doubt, the TSC would discuss as currently done.
For milestone releases, the same process as above would apply but QA flagged
issues would usually block the next milestone as we'd want to see issues
resolved.
For main/final project major version changes/releases, the current process
applies with QA testing needing to be completed before the final release is made
and a TSC vote is required.
The intent of the changes is to allow the project to be dynamic where we can and
allow point releases to be sped up whilst still ensuring we do maintain project
quality and testing.
There will be changes needed to the release process to accommodate this but I'm
hoping those aren't too complex and can be figured out. Michael/Chee Yang, if
you do see any potential problems in this change please do let me know and we
can discuss them and try and figure out a solution. I'm also very open to other
ideas for improving things and making them more efficient if/where we can.
Cheers,
Richard