Date   

Re: Yocto post-release CVE and package uprev policy - openssl, ffmpeg, etc.

Armin Kuster
 

On 01/26/2017 12:10 PM, Randy MacLeod wrote:

Yocto seems to have a policy not to update packages once a
release is generally available.
I believe this is already covered in the current policy.

(lifted from policy)
No recipe upgrades unless:
The new version contains a security patch or other critical bugfix that is too difficult to backport to the version already in the stable branch.

does that not cover this concern?

I think that rule should be
broken for certain packages that have been reviewed and tested
properly.
To me the responsibility is on the person who wants for issue fixed via a package update. They need to convince the package maintainer or branch maintainer that this is the better course of action than to back port a patch.

See:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=10707
for additional background.

For some packages, the upstream development team fixes CVE and
other bugs on their released version and by YP only cherry-picking back
specific fixes, we expose users to additional risk and incur higher
costs of maintenance. At least two packages that I know of have released
"bug fix only" updates to fix CVEs and other defects for packages
that are in morty:

- openssl 1.0.2j -> 1.0.2k
- ffmpeg 3.1.3 -> 3.1.5
Gcc, Glibc and kernel fall into this bucket too so where do you draw the line?

Should we continue to cherry-pick back only the CVEs fixes
or should we review, test, and release the full minor release?
My preference is to improve the guidelines rather than creating exceptions. There is the scenario that if a CVE fix exists and no new package version has been released, we should wait.

It not uncommon for a commit message to a package upgrade lacks information to help a stable branch maintainer make a decision for backporting.


I've done a review of openssl below but
before I proceed with more evaluation or sending the
uprev to the list for morty, I'd like to know if the upgrade
policy will block such a change. From my analysis, there's only
one change that seems like an upgrade blocker and I need help
to evaluate that since I'm not an openssl maintainer.


I've done the upgrade locally. It's just a few lines and builds
seem to be fine so far. I'll send the upgrade for master
at least once my builds complete and I've done some other
tests.
Then send patches and explain why. The maintainer of the stable branch appears to be some what reasonable and does appreciate all input on what should be back ported to the stable branches.

Regards,
Armin


../Randy


Review of openssl-1.0.2j->k.


Early next week, I'll check for an update on: 1.0.2j->k compatibility
here:
https://abi-laboratory.pro/tracker/timeline/openssl/
'k' hasn't been done as of Jan 26th.


I looked at the 78 changes to openssl-1.0.2j->k and
found that 4 header files had changed. Here's a list of
the header files and my conclusion/summary.

$ git diff OpenSSL_1_0_2j..OpenSSL_1_0_2k | \
diffstat| grep "\.h"
apps/apps.h | 4
--> Add: always call setup_engine
crypto/evp/evp.h | 6
--> +# define EVP_R_INVALID_KEY and whitespace
crypto/opensslv.h | 6
--> version update
ssl/ssl_locl.h | 2
--> api change but according to [1] it's an internal header
-int ssl_check_clienthello_tlsext_late(SSL *s);
+int ssl_check_clienthello_tlsext_late(SSL *s, int *al);


[1] Mr Burton claims this is (or was?) a private api:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=641711


The change to always call setup_engine() may be a problem
but I'm not familiar with the openssl code base so I'm
not sure how big a deal it is. Alex, are you familiar with
this part of openssl?


Here is a list of the commits:

$ git log --oneline OpenSSL_1_0_2j..OpenSSL_1_0_2k | wc -l
78

$ git log --oneline OpenSSL_1_0_2j..OpenSSL_1_0_2k

081314d Prepare for 1.0.2k release
06f87e9 Update CHANGES and NEWS for new release
918d8ea Better check of DH parameters in TLS data
760d043 bn/asm/x86_64-mont5.pl: fix carry bug in bn_sqr8x_internal.
51d0090 crypto/evp: harden RC4_MD5 cipher.
8957add Fix error handling in compute_key, BN_CTX_get can return NULL
cb00d4f Fix a ssl session leak due to OOM in lh_SSL_SESSION_insert
e203f49 Fix SSL_VERIFY_CLIENT_ONCE
149e98d Add missing va_end
16f013f Fix DSA parameter generation control error
52b703f Clean one unused variable, plus an useless one.
1f234f7 GH1986: Document -header flag.
0ecb682 Fix error handling in SSL_CTX_new
2045c58 Fix a memory leak in RSA_padding_add_PKCS1_OAEP_mgf1
18b8431 replace "will lookup up" by "will look up"
58c81e7 Reformat M_check_autoarg to match our coding style
222333c M_check_autoarg: sanity check the key
3fb9f87 Fix typo.
5bbedd3 zero pad DHE public key in ServerKeyExchange message for interop
70705b2 Fix ssl_cert_dup: change one 'return NULL' to 'goto err'
3b584ef Make 'err' lable in ssl_cert_dup unconditional
292bb56 Fix a bug in clienthello processing
7624a31 perlasm/x86_64-xlate.pl: refine sign extension in ea package.
10a5037 UI_OpenSSL()'s session opener fails on MacOS X
78a3e80 VMS UI_OpenSSL:
if the TT device isn't a tty, flag instead of error
fecd4c2 Check input length to pkey_rsa_verify()
5ae285e Remove extra bang
59ba83c UI code style cleanup
748a2d9 Revert "Fix heartbeat_test"
be3a7dd apps/speed.c: Fix crash when config loading fails
c477f8e INSTALL: clarify 386 and no-sse2 options.
f47201b modes/ctr128.c: fix false carry in counter increment procedure.
c4c7165 Clarify what X509_NAME_online does with
the given buffer and size
31b4307 Make SSL_read and SSL_write return
the old behaviour and document it.
09b894b Use consistent variable names
f4ef1c5 domd: Preserve Makefile time when it is unchanged
7a9d712 mklink: Do not needlessly overwrite linked files...
62f16de domd: Do not needlessly overwrite Makefiles
22cc44d mklink: Do not needlessly overwrite linked files...
ecc9551 Configure: Improve incremental build time
8ac70be Check return value of some BN functions.
3201a1d Solution proposal for issue #1647.
19e1de5 Update CHANGES and NEWS
57c4b9f bn/asm/x86_64-mont.pl: fix for CVE-2016-7055 (Low severity).
c210840 Makefile.org: clear APPS environment variable.
95873c5 Missed a mention of RT
563a34e Add a CHANGES entry for the unrecognised record type change
f118539 Fail if an unrecognised record type is received
ad69a30 Fix heartbeat_test
ba2bf83 Secure our notification email.
e022375 Fix grammar-o in CONTRIBUTING
787b2dc Add $(EX_LIBS) to the LIBDEPS for libgost.so,
just as for all other engines
0b9c5da Implement length checks as a macro
a520723 Ensure we have length checks for all extensions
83a1d4b Fix length check writing status request extension
57aa2f1 Fix a double free in ca command line
fa4c374 A zero return from BIO_read/BIO_write() could be retryable
31bf65c Fix typo (reported by Matthias St. Pierre)
0e46901 Fix leak of secrecy in ecdh_compute_key()
3ade92e Correctly find all critical CRL extensions
45f4761 remove redundant zero assignments
cdb203f %p takes void*, so make sure to cast arguments to void*
0df1caa apps: make setup_engine() and release_engine() available always
aa01b82 If an engine comes up explicitely,
it must also come down explicitely
10e60f2 Fix no-des
1c6aab6 Make 'openssl prime ""' not segfault
99c002b Fix strict-warnings build
b0161f6 Fix strict-warnings build
78ee64c Fix signatures of EVP_Digest{Sign,Verify}Update
02a0231 Ensure we handle len == 0 in ERR_err_string_n
6d69dc5 Degrade 3DES to MEDIUM in SSL2
e8e380c RT is put out to pasture
f1f9769 Add missing error string for SSL_R_TOO_MANY_WARN_ALERTS
53a71b7 apps/apps.c: initialize and de-initialize engine
around key loading
a269e5f Revert "Call ENGINE_init() before trying to use keys
from engine"
4badd2b Call ENGINE_init() before trying to use keys from engine
9702bf5 Fix NEWS error
f6e43fe Prepare for 1.0.2k-dev


I've look at any commits that *seem* like they could be more than
a bug fix or that might change the api. Aside from the two issues
related to header files, I didn't see anything to worry about.

$ git diff OpenSSL_1_0_2j..OpenSSL_1_0_2k | diffstat
.travis.yml | 2
CHANGES | 61 ++++++
CONTRIBUTING | 55 +----
Configure | 34 ++-
INSTALL | 69 +++---
Makefile.org | 3
NEWS | 8
README | 36 ---
apps/apps.c | 19 +
apps/apps.h | 4
apps/ca.c | 6
apps/cms.c | 5
apps/dgst.c | 1
apps/dh.c | 6
apps/dhparam.c | 8
apps/dsa.c | 7
apps/dsaparam.c | 8
apps/ec.c | 6
apps/ecparam.c | 10
apps/enc.c | 8
apps/gendh.c | 4
apps/gendsa.c | 8
apps/genpkey.c | 2
apps/genrsa.c | 7
apps/pkcs12.c | 7
apps/pkcs7.c | 8
apps/pkcs8.c | 5
apps/pkey.c | 5
apps/pkeyparam.c | 8
apps/pkeyutl.c | 1
apps/prime.c | 12 -
apps/rand.c | 8
apps/req.c | 5
apps/rsa.c | 5
apps/rsautl.c | 5
apps/s_cb.c | 4
apps/s_client.c | 7
apps/s_server.c | 7
apps/smime.c | 5
apps/speed.c | 14 -
apps/spkac.c | 5
apps/srp.c | 8
apps/verify.c | 5
apps/x509.c | 5
crypto/aes/asm/aes-s390x.pl | 8
crypto/asn1/p5_pbev2.c | 8
crypto/asn1/x_crl.c | 3
crypto/bn/asm/x86_64-mont.pl | 5
crypto/bn/asm/x86_64-mont5.pl | 16 -
crypto/bn/bn_exp.c | 5
crypto/bn/bn_mul.c | 5
crypto/bn/bn_prime.c | 3
crypto/bn/bn_sqr.c | 5
crypto/cms/cms_kari.c | 5
crypto/dh/dh_key.c | 2
crypto/dsa/dsa_pmeth.c | 2
crypto/ec/ec2_mult.c | 20 +
crypto/ecdh/ech_ossl.c | 4
crypto/err/err.c | 3
crypto/evp/e_aes.c | 4
crypto/evp/e_rc4_hmac_md5.c | 2
crypto/evp/evp.h | 6
crypto/evp/evp_err.c | 3
crypto/evp/pmeth_fn.c | 30 +-
crypto/evp/pmeth_lib.c | 28 --
crypto/modes/ctr128.c | 2
crypto/opensslv.h | 6
crypto/perlasm/x86_64-xlate.pl | 11 -
crypto/rsa/rsa_gen.c | 3
crypto/rsa/rsa_oaep.c | 8
crypto/rsa/rsa_pmeth.c | 4
crypto/s390xcap.c | 1
crypto/ui/ui_lib.c | 138 +++++++------
crypto/ui/ui_openssl.c | 59 +++--
demos/easy_tls/easy-tls.c | 1
doc/apps/ocsp.pod | 9
doc/crypto/EVP_DigestSignInit.pod | 2
doc/crypto/EVP_DigestVerifyInit.pod | 2
doc/crypto/RSA_generate_key.pod | 2
doc/crypto/X509_NAME_get_index_by_NID.pod | 3
doc/crypto/X509_NAME_print_ex.pod | 8
doc/ssl/SSL_CTX_set_session_cache_mode.pod | 2
doc/ssl/SSL_get_error.pod | 22 --
doc/ssl/SSL_read.pod | 32 +--
doc/ssl/SSL_write.pod | 19 -
engines/ccgost/Makefile | 2
openssl.spec | 2
ssl/bad_dtls_test.c | 5
ssl/s23_pkt.c | 12 -
ssl/s2_lib.c | 2
ssl/s2_pkt.c | 10
ssl/s3_clnt.c | 44 +++-
ssl/s3_pkt.c | 23 +-
ssl/s3_srvr.c | 33 ++-
ssl/ssl_cert.c | 4
ssl/ssl_err.c | 1
ssl/ssl_lib.c | 4
ssl/ssl_locl.h | 2
ssl/ssl_sess.c | 9
ssl/t1_lib.c | 291 ++++++++++++++++++-----------
util/domd | 11 -
util/mklink.pl | 8
102 files changed, 836 insertions(+), 634 deletions(-)




Re: Yocto post-release CVE and package uprev policy - openssl, ffmpeg, etc.

Martin Jansa
 

On Fri, Jan 27, 2017 at 09:43:42AM -0800, Khem Raj wrote:


On 1/26/17 8:04 PM, Paul Eggleton wrote:
On Thursday, 26 January 2017 3:10:57 PM NZDT Randy MacLeod wrote:
Yocto seems to have a policy not to update packages once a
release is generally available. I think that rule should be
broken for certain packages that have been reviewed and tested
properly.
We have made exceptions in the past for exactly the reasons you outline.
Unfortunately I know at least for OpenSSL when we have done so on at least one
occasion we have been bitten by compatibility issues :(

If we can introduce more rigorous runtime testing (and by that I don't just
mean tests for the package itself - runtime tests for functionality in other
applications that rely on that package would be preferred) then we would be in
a much better place. Being able to measure ABI compatibility is a good start
but doesn't cover any internal changes in behaviour that might be problematic.

See:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=10707
for additional background.

For some packages, the upstream development team fixes CVE and
other bugs on their released version and by YP only cherry-picking back
specific fixes, we expose users to additional risk and incur higher
costs of maintenance.

At least two packages that I know of have released
"bug fix only" updates to fix CVEs and other defects for packages
that are in morty:

- openssl 1.0.2j -> 1.0.2k
- ffmpeg 3.1.3 -> 3.1.5

Should we continue to cherry-pick back only the CVEs fixes
or should we review, test, and release the full minor release?


I've done a review of openssl below but
before I proceed with more evaluation or sending the
uprev to the list for morty, I'd like to know if the upgrade
policy will block such a change. From my analysis, there's only
one change that seems like an upgrade blocker and I need help
to evaluate that since I'm not an openssl maintainer.
I think we can contexualize it based upon amount of work, openssl has
violated ABI within same major release in past see 1.0.2f -> 1.0.2g

For complete list

https://abi-laboratory.pro/tracker/timeline/openssl/
And when the upstream didn't break the ABI, we did with seemingly safe
change of version-script.patch:

http://git.openembedded.org/openembedded-core/commit/?h=jethro&id=528541845df34843c14be5de62e9f53004d292ac
http://git.openembedded.org/openembedded-core/commit/?h=krogoth&id=08f85da10b3a7fc6165f163fd0f23784a2c9c8e4

So I agree that we should be very careful and make as few exceptions as
possible at least until there is strong testing for API/ABI
compatibility for release branches.

We may use that as guiding light to make a decision for backporting CVEs
or do a minor upgrade.


I would say we're not hard-blocked by the policy - we can make exceptions, but
we really do need to be careful, and I don't think we're prepared to make a
continuing exception for specific packages yet. The better idea we can get
that there won't be regressions, hopefully in an automated or semi-automated
manner, the safer position we'll be in.

Cheers,
Paul
_______________________________________________
Openembedded-architecture mailing list
Openembedded-architecture@...
http://lists.openembedded.org/mailman/listinfo/openembedded-architecture
--
Martin 'JaMa' Jansa jabber: Martin.Jansa@...


Re: Yocto post-release CVE and package uprev policy - openssl, ffmpeg, etc.

Khem Raj
 

On 1/26/17 8:04 PM, Paul Eggleton wrote:
On Thursday, 26 January 2017 3:10:57 PM NZDT Randy MacLeod wrote:
Yocto seems to have a policy not to update packages once a
release is generally available. I think that rule should be
broken for certain packages that have been reviewed and tested
properly.
We have made exceptions in the past for exactly the reasons you outline.
Unfortunately I know at least for OpenSSL when we have done so on at least one
occasion we have been bitten by compatibility issues :(

If we can introduce more rigorous runtime testing (and by that I don't just
mean tests for the package itself - runtime tests for functionality in other
applications that rely on that package would be preferred) then we would be in
a much better place. Being able to measure ABI compatibility is a good start
but doesn't cover any internal changes in behaviour that might be problematic.

See:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=10707
for additional background.

For some packages, the upstream development team fixes CVE and
other bugs on their released version and by YP only cherry-picking back
specific fixes, we expose users to additional risk and incur higher
costs of maintenance.

At least two packages that I know of have released
"bug fix only" updates to fix CVEs and other defects for packages
that are in morty:

- openssl 1.0.2j -> 1.0.2k
- ffmpeg 3.1.3 -> 3.1.5

Should we continue to cherry-pick back only the CVEs fixes
or should we review, test, and release the full minor release?


I've done a review of openssl below but
before I proceed with more evaluation or sending the
uprev to the list for morty, I'd like to know if the upgrade
policy will block such a change. From my analysis, there's only
one change that seems like an upgrade blocker and I need help
to evaluate that since I'm not an openssl maintainer.
I think we can contexualize it based upon amount of work, openssl has
violated ABI within same major release in past see 1.0.2f -> 1.0.2g

For complete list

https://abi-laboratory.pro/tracker/timeline/openssl/

We may use that as guiding light to make a decision for backporting CVEs
or do a minor upgrade.


I would say we're not hard-blocked by the policy - we can make exceptions, but
we really do need to be careful, and I don't think we're prepared to make a
continuing exception for specific packages yet. The better idea we can get
that there won't be regressions, hopefully in an automated or semi-automated
manner, the safer position we'll be in.

Cheers,
Paul


Re: Yocto post-release CVE and package uprev policy - openssl, ffmpeg, etc.

Alexander Kanavin <alexander.kanavin@...>
 

On 01/26/2017 10:10 PM, Randy MacLeod wrote:
I've done a review of openssl below but
before I proceed with more evaluation or sending the
uprev to the list for morty, I'd like to know if the upgrade
policy will block such a change. From my analysis, there's only
one change that seems like an upgrade blocker and I need help
to evaluate that since I'm not an openssl maintainer.

The change to always call setup_engine() may be a problem
but I'm not familiar with the openssl code base so I'm
not sure how big a deal it is. Alex, are you familiar with
this part of openssl?
Randy, you are starting with a wrong assumption: that people listed in maintainers.inc are real maintainers - real in the sense that they follow upstream development, take care of regular runtime testing (including writing the testsuite if upstream doesn't provide a good one), and understand the source code well to make informed decisions. That is simply not true. I have no idea about what goes on inside openssl, and I think no one else in Yocto does. My duty is updating openssl to the latest version in master and reacting to bug reports; there's simply no time for more.

We need to spread out recipe maintainership a lot more than is the case currently, then we can have a sane policy in place for stable releases.

Alex


Re: [OE-core] OpenEmbedded Stand at FOSDEM

Paul Barker <paul@...>
 

On Thu, 26 Jan 2017 22:39:17 +0100
Andreas Müller <schnitzeltony@...> wrote:

On Thu, Jan 5, 2017 at 4:30 PM, Philip Balister <philip@...>
wrote:
On 01/03/2017 08:13 PM, Andreas Müller wrote:
On Tue, Jan 3, 2017 at 4:32 PM, Philip Balister
<philip@...> wrote:
Every year since 2007, OpenEmbedded has a stand at FOSDEM
(http://www.fosdem.org)

From the first year:

https://www.flickr.com/photos/32615155@N00/405229708/in/album-72157594561002629/

Belen and I are sort of organizing this, but both of us are also
involved in devrooms, so we will need a lot of help manning the
stand and getting some demos together.

Demos should try and show how the project makes embedded work
easier, by showing tools and/or cool examples of devices using
Linux built with OpenEmbedded. In previous years, we've shown
toaster with data collected from demos on the table. Collections
of devices running images built from the same recipe and
interesting products using Linux by OpenEmbedded.

I'm happy to try and organize demos and staffing, but I could
really use some help this year, so if you are in a position to
tak ethe lead on operating the stand, that would be a huge help
to me and the rest of the project.

Thanks,

Philip
Nobody?

Andreas
There's 3 of us from Togán Labs who'll be able to help out manning the
stand. I know I can do all day Saturday and up to about 4pm on Sunday
and I'm happy to be at the stand all day (with the obvious caffeine
breaks!). Let me know if you need help setting things up and what time
you need people to turn up on Saturday morning.

If you need anything printing then send it to me.

I can bring 2x 4-way UK power strips plus 2x EU->UK power adaptors for
our gear. I've got 2 laptops plus a few embedded devices. I don't have
any EU power strips though.

I've updated the wiki page.

Thanks,
Paul


Re: Yocto post-release CVE and package uprev policy - openssl, ffmpeg, etc.

Paul Eggleton <paul.eggleton@...>
 

On Thursday, 26 January 2017 3:10:57 PM NZDT Randy MacLeod wrote:
Yocto seems to have a policy not to update packages once a
release is generally available. I think that rule should be
broken for certain packages that have been reviewed and tested
properly.
We have made exceptions in the past for exactly the reasons you outline.
Unfortunately I know at least for OpenSSL when we have done so on at least one
occasion we have been bitten by compatibility issues :(

If we can introduce more rigorous runtime testing (and by that I don't just
mean tests for the package itself - runtime tests for functionality in other
applications that rely on that package would be preferred) then we would be in
a much better place. Being able to measure ABI compatibility is a good start
but doesn't cover any internal changes in behaviour that might be problematic.

See:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=10707
for additional background.

For some packages, the upstream development team fixes CVE and
other bugs on their released version and by YP only cherry-picking back
specific fixes, we expose users to additional risk and incur higher
costs of maintenance.

At least two packages that I know of have released
"bug fix only" updates to fix CVEs and other defects for packages
that are in morty:

- openssl 1.0.2j -> 1.0.2k
- ffmpeg 3.1.3 -> 3.1.5

Should we continue to cherry-pick back only the CVEs fixes
or should we review, test, and release the full minor release?


I've done a review of openssl below but
before I proceed with more evaluation or sending the
uprev to the list for morty, I'd like to know if the upgrade
policy will block such a change. From my analysis, there's only
one change that seems like an upgrade blocker and I need help
to evaluate that since I'm not an openssl maintainer.
I would say we're not hard-blocked by the policy - we can make exceptions, but
we really do need to be careful, and I don't think we're prepared to make a
continuing exception for specific packages yet. The better idea we can get
that there won't be regressions, hopefully in an automated or semi-automated
manner, the safer position we'll be in.

Cheers,
Paul

--

Paul Eggleton
Intel Open Source Technology Centre


Re: Yocto post-release CVE and package uprev policy - openssl, ffmpeg, etc.

Khem Raj
 

On 1/26/17 12:10 PM, Randy MacLeod wrote:

Yocto seems to have a policy not to update packages once a
release is generally available. I think that rule should be
broken for certain packages that have been reviewed and tested
properly.

See:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=10707
for additional background.

For some packages, the upstream development team fixes CVE and
other bugs on their released version and by YP only cherry-picking back
specific fixes, we expose users to additional risk and incur higher
costs of maintenance.

At least two packages that I know of have released
"bug fix only" updates to fix CVEs and other defects for packages
that are in morty:

- openssl 1.0.2j -> 1.0.2k
- ffmpeg 3.1.3 -> 3.1.5

Should we continue to cherry-pick back only the CVEs fixes
or should we review, test, and release the full minor release?
If upstream claims j->k release to just have CVE changesets may be its
not a problem, however if there are more fixes that comes along with
CVEs then we need to understand closely what these fixes are what do
they break, IMO, this could be ok if we have some sort of API/ABI
testing to ensure that nothing else breaks, otherwise someone should sit
and bean the commits for what they are changing.


I've done a review of openssl below but
before I proceed with more evaluation or sending the
uprev to the list for morty, I'd like to know if the upgrade
policy will block such a change. From my analysis, there's only
one change that seems like an upgrade blocker and I need help
to evaluate that since I'm not an openssl maintainer.


I've done the upgrade locally. It's just a few lines and builds
seem to be fine so far. I'll send the upgrade for master
at least once my builds complete and I've done some other
tests.

../Randy


Review of openssl-1.0.2j->k.


Early next week, I'll check for an update on: 1.0.2j->k compatibility
here:
https://abi-laboratory.pro/tracker/timeline/openssl/
'k' hasn't been done as of Jan 26th.


I looked at the 78 changes to openssl-1.0.2j->k and
found that 4 header files had changed. Here's a list of
the header files and my conclusion/summary.

$ git diff OpenSSL_1_0_2j..OpenSSL_1_0_2k | \
diffstat| grep "\.h"
apps/apps.h | 4
--> Add: always call setup_engine
crypto/evp/evp.h | 6
--> +# define EVP_R_INVALID_KEY and whitespace
crypto/opensslv.h | 6
--> version update
ssl/ssl_locl.h | 2
--> api change but according to [1] it's an internal header
-int ssl_check_clienthello_tlsext_late(SSL *s);
+int ssl_check_clienthello_tlsext_late(SSL *s, int *al);


[1] Mr Burton claims this is (or was?) a private api:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=641711


The change to always call setup_engine() may be a problem
but I'm not familiar with the openssl code base so I'm
not sure how big a deal it is. Alex, are you familiar with
this part of openssl?


Here is a list of the commits:

$ git log --oneline OpenSSL_1_0_2j..OpenSSL_1_0_2k | wc -l
78

$ git log --oneline OpenSSL_1_0_2j..OpenSSL_1_0_2k

081314d Prepare for 1.0.2k release
06f87e9 Update CHANGES and NEWS for new release
918d8ea Better check of DH parameters in TLS data
760d043 bn/asm/x86_64-mont5.pl: fix carry bug in bn_sqr8x_internal.
51d0090 crypto/evp: harden RC4_MD5 cipher.
8957add Fix error handling in compute_key, BN_CTX_get can return NULL
cb00d4f Fix a ssl session leak due to OOM in lh_SSL_SESSION_insert
e203f49 Fix SSL_VERIFY_CLIENT_ONCE
149e98d Add missing va_end
16f013f Fix DSA parameter generation control error
52b703f Clean one unused variable, plus an useless one.
1f234f7 GH1986: Document -header flag.
0ecb682 Fix error handling in SSL_CTX_new
2045c58 Fix a memory leak in RSA_padding_add_PKCS1_OAEP_mgf1
18b8431 replace "will lookup up" by "will look up"
58c81e7 Reformat M_check_autoarg to match our coding style
222333c M_check_autoarg: sanity check the key
3fb9f87 Fix typo.
5bbedd3 zero pad DHE public key in ServerKeyExchange message for interop
70705b2 Fix ssl_cert_dup: change one 'return NULL' to 'goto err'
3b584ef Make 'err' lable in ssl_cert_dup unconditional
292bb56 Fix a bug in clienthello processing
7624a31 perlasm/x86_64-xlate.pl: refine sign extension in ea package.
10a5037 UI_OpenSSL()'s session opener fails on MacOS X
78a3e80 VMS UI_OpenSSL:
if the TT device isn't a tty, flag instead of error
fecd4c2 Check input length to pkey_rsa_verify()
5ae285e Remove extra bang
59ba83c UI code style cleanup
748a2d9 Revert "Fix heartbeat_test"
be3a7dd apps/speed.c: Fix crash when config loading fails
c477f8e INSTALL: clarify 386 and no-sse2 options.
f47201b modes/ctr128.c: fix false carry in counter increment procedure.
c4c7165 Clarify what X509_NAME_online does with
the given buffer and size
31b4307 Make SSL_read and SSL_write return
the old behaviour and document it.
09b894b Use consistent variable names
f4ef1c5 domd: Preserve Makefile time when it is unchanged
7a9d712 mklink: Do not needlessly overwrite linked files...
62f16de domd: Do not needlessly overwrite Makefiles
22cc44d mklink: Do not needlessly overwrite linked files...
ecc9551 Configure: Improve incremental build time
8ac70be Check return value of some BN functions.
3201a1d Solution proposal for issue #1647.
19e1de5 Update CHANGES and NEWS
57c4b9f bn/asm/x86_64-mont.pl: fix for CVE-2016-7055 (Low severity).
c210840 Makefile.org: clear APPS environment variable.
95873c5 Missed a mention of RT
563a34e Add a CHANGES entry for the unrecognised record type change
f118539 Fail if an unrecognised record type is received
ad69a30 Fix heartbeat_test
ba2bf83 Secure our notification email.
e022375 Fix grammar-o in CONTRIBUTING
787b2dc Add $(EX_LIBS) to the LIBDEPS for libgost.so,
just as for all other engines
0b9c5da Implement length checks as a macro
a520723 Ensure we have length checks for all extensions
83a1d4b Fix length check writing status request extension
57aa2f1 Fix a double free in ca command line
fa4c374 A zero return from BIO_read/BIO_write() could be retryable
31bf65c Fix typo (reported by Matthias St. Pierre)
0e46901 Fix leak of secrecy in ecdh_compute_key()
3ade92e Correctly find all critical CRL extensions
45f4761 remove redundant zero assignments
cdb203f %p takes void*, so make sure to cast arguments to void*
0df1caa apps: make setup_engine() and release_engine() available always
aa01b82 If an engine comes up explicitely,
it must also come down explicitely
10e60f2 Fix no-des
1c6aab6 Make 'openssl prime ""' not segfault
99c002b Fix strict-warnings build
b0161f6 Fix strict-warnings build
78ee64c Fix signatures of EVP_Digest{Sign,Verify}Update
02a0231 Ensure we handle len == 0 in ERR_err_string_n
6d69dc5 Degrade 3DES to MEDIUM in SSL2
e8e380c RT is put out to pasture
f1f9769 Add missing error string for SSL_R_TOO_MANY_WARN_ALERTS
53a71b7 apps/apps.c: initialize and de-initialize engine
around key loading
a269e5f Revert "Call ENGINE_init() before trying to use keys
from engine"
4badd2b Call ENGINE_init() before trying to use keys from engine
9702bf5 Fix NEWS error
f6e43fe Prepare for 1.0.2k-dev


I've look at any commits that *seem* like they could be more than
a bug fix or that might change the api. Aside from the two issues
related to header files, I didn't see anything to worry about.

$ git diff OpenSSL_1_0_2j..OpenSSL_1_0_2k | diffstat
.travis.yml | 2
CHANGES | 61 ++++++
CONTRIBUTING | 55 +----
Configure | 34 ++-
INSTALL | 69 +++---
Makefile.org | 3
NEWS | 8
README | 36 ---
apps/apps.c | 19 +
apps/apps.h | 4
apps/ca.c | 6
apps/cms.c | 5
apps/dgst.c | 1
apps/dh.c | 6
apps/dhparam.c | 8
apps/dsa.c | 7
apps/dsaparam.c | 8
apps/ec.c | 6
apps/ecparam.c | 10
apps/enc.c | 8
apps/gendh.c | 4
apps/gendsa.c | 8
apps/genpkey.c | 2
apps/genrsa.c | 7
apps/pkcs12.c | 7
apps/pkcs7.c | 8
apps/pkcs8.c | 5
apps/pkey.c | 5
apps/pkeyparam.c | 8
apps/pkeyutl.c | 1
apps/prime.c | 12 -
apps/rand.c | 8
apps/req.c | 5
apps/rsa.c | 5
apps/rsautl.c | 5
apps/s_cb.c | 4
apps/s_client.c | 7
apps/s_server.c | 7
apps/smime.c | 5
apps/speed.c | 14 -
apps/spkac.c | 5
apps/srp.c | 8
apps/verify.c | 5
apps/x509.c | 5
crypto/aes/asm/aes-s390x.pl | 8
crypto/asn1/p5_pbev2.c | 8
crypto/asn1/x_crl.c | 3
crypto/bn/asm/x86_64-mont.pl | 5
crypto/bn/asm/x86_64-mont5.pl | 16 -
crypto/bn/bn_exp.c | 5
crypto/bn/bn_mul.c | 5
crypto/bn/bn_prime.c | 3
crypto/bn/bn_sqr.c | 5
crypto/cms/cms_kari.c | 5
crypto/dh/dh_key.c | 2
crypto/dsa/dsa_pmeth.c | 2
crypto/ec/ec2_mult.c | 20 +
crypto/ecdh/ech_ossl.c | 4
crypto/err/err.c | 3
crypto/evp/e_aes.c | 4
crypto/evp/e_rc4_hmac_md5.c | 2
crypto/evp/evp.h | 6
crypto/evp/evp_err.c | 3
crypto/evp/pmeth_fn.c | 30 +-
crypto/evp/pmeth_lib.c | 28 --
crypto/modes/ctr128.c | 2
crypto/opensslv.h | 6
crypto/perlasm/x86_64-xlate.pl | 11 -
crypto/rsa/rsa_gen.c | 3
crypto/rsa/rsa_oaep.c | 8
crypto/rsa/rsa_pmeth.c | 4
crypto/s390xcap.c | 1
crypto/ui/ui_lib.c | 138 +++++++------
crypto/ui/ui_openssl.c | 59 +++--
demos/easy_tls/easy-tls.c | 1
doc/apps/ocsp.pod | 9
doc/crypto/EVP_DigestSignInit.pod | 2
doc/crypto/EVP_DigestVerifyInit.pod | 2
doc/crypto/RSA_generate_key.pod | 2
doc/crypto/X509_NAME_get_index_by_NID.pod | 3
doc/crypto/X509_NAME_print_ex.pod | 8
doc/ssl/SSL_CTX_set_session_cache_mode.pod | 2
doc/ssl/SSL_get_error.pod | 22 --
doc/ssl/SSL_read.pod | 32 +--
doc/ssl/SSL_write.pod | 19 -
engines/ccgost/Makefile | 2
openssl.spec | 2
ssl/bad_dtls_test.c | 5
ssl/s23_pkt.c | 12 -
ssl/s2_lib.c | 2
ssl/s2_pkt.c | 10
ssl/s3_clnt.c | 44 +++-
ssl/s3_pkt.c | 23 +-
ssl/s3_srvr.c | 33 ++-
ssl/ssl_cert.c | 4
ssl/ssl_err.c | 1
ssl/ssl_lib.c | 4
ssl/ssl_locl.h | 2
ssl/ssl_sess.c | 9
ssl/t1_lib.c | 291
++++++++++++++++++-----------
util/domd | 11 -
util/mklink.pl | 8
102 files changed, 836 insertions(+), 634 deletions(-)




Re: [OE-core] OpenEmbedded Stand at FOSDEM

Andreas Müller <schnitzeltony@...>
 

On Thu, Jan 5, 2017 at 4:30 PM, Philip Balister <philip@...> wrote:
On 01/03/2017 08:13 PM, Andreas Müller wrote:
On Tue, Jan 3, 2017 at 4:32 PM, Philip Balister <philip@...> wrote:
Every year since 2007, OpenEmbedded has a stand at FOSDEM
(http://www.fosdem.org)

From the first year:

https://www.flickr.com/photos/32615155@N00/405229708/in/album-72157594561002629/

Belen and I are sort of organizing this, but both of us are also
involved in devrooms, so we will need a lot of help manning the stand
and getting some demos together.

Demos should try and show how the project makes embedded work easier, by
showing tools and/or cool examples of devices using Linux built with
OpenEmbedded. In previous years, we've shown toaster with data collected
from demos on the table. Collections of devices running images built
from the same recipe and interesting products using Linux by OpenEmbedded.

I'm happy to try and organize demos and staffing, but I could really use
some help this year, so if you are in a position to tak ethe lead on
operating the stand, that would be a huge help to me and the rest of the
project.

Thanks,

Philip
Nobody?

Andreas


Yocto post-release CVE and package uprev policy - openssl, ffmpeg, etc.

Randy MacLeod
 

Yocto seems to have a policy not to update packages once a
release is generally available. I think that rule should be
broken for certain packages that have been reviewed and tested
properly.

See:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=10707
for additional background.

For some packages, the upstream development team fixes CVE and
other bugs on their released version and by YP only cherry-picking back
specific fixes, we expose users to additional risk and incur higher
costs of maintenance.

At least two packages that I know of have released
"bug fix only" updates to fix CVEs and other defects for packages
that are in morty:

- openssl 1.0.2j -> 1.0.2k
- ffmpeg 3.1.3 -> 3.1.5

Should we continue to cherry-pick back only the CVEs fixes
or should we review, test, and release the full minor release?


I've done a review of openssl below but
before I proceed with more evaluation or sending the
uprev to the list for morty, I'd like to know if the upgrade
policy will block such a change. From my analysis, there's only
one change that seems like an upgrade blocker and I need help
to evaluate that since I'm not an openssl maintainer.


I've done the upgrade locally. It's just a few lines and builds
seem to be fine so far. I'll send the upgrade for master
at least once my builds complete and I've done some other
tests.

../Randy


Review of openssl-1.0.2j->k.


Early next week, I'll check for an update on: 1.0.2j->k compatibility
here:
https://abi-laboratory.pro/tracker/timeline/openssl/
'k' hasn't been done as of Jan 26th.


I looked at the 78 changes to openssl-1.0.2j->k and
found that 4 header files had changed. Here's a list of
the header files and my conclusion/summary.

$ git diff OpenSSL_1_0_2j..OpenSSL_1_0_2k | \
diffstat| grep "\.h"
apps/apps.h | 4
--> Add: always call setup_engine
crypto/evp/evp.h | 6
--> +# define EVP_R_INVALID_KEY and whitespace
crypto/opensslv.h | 6
--> version update
ssl/ssl_locl.h | 2
--> api change but according to [1] it's an internal header
-int ssl_check_clienthello_tlsext_late(SSL *s);
+int ssl_check_clienthello_tlsext_late(SSL *s, int *al);


[1] Mr Burton claims this is (or was?) a private api:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=641711


The change to always call setup_engine() may be a problem
but I'm not familiar with the openssl code base so I'm
not sure how big a deal it is. Alex, are you familiar with
this part of openssl?


Here is a list of the commits:

$ git log --oneline OpenSSL_1_0_2j..OpenSSL_1_0_2k | wc -l
78

$ git log --oneline OpenSSL_1_0_2j..OpenSSL_1_0_2k

081314d Prepare for 1.0.2k release
06f87e9 Update CHANGES and NEWS for new release
918d8ea Better check of DH parameters in TLS data
760d043 bn/asm/x86_64-mont5.pl: fix carry bug in bn_sqr8x_internal.
51d0090 crypto/evp: harden RC4_MD5 cipher.
8957add Fix error handling in compute_key, BN_CTX_get can return NULL
cb00d4f Fix a ssl session leak due to OOM in lh_SSL_SESSION_insert
e203f49 Fix SSL_VERIFY_CLIENT_ONCE
149e98d Add missing va_end
16f013f Fix DSA parameter generation control error
52b703f Clean one unused variable, plus an useless one.
1f234f7 GH1986: Document -header flag.
0ecb682 Fix error handling in SSL_CTX_new
2045c58 Fix a memory leak in RSA_padding_add_PKCS1_OAEP_mgf1
18b8431 replace "will lookup up" by "will look up"
58c81e7 Reformat M_check_autoarg to match our coding style
222333c M_check_autoarg: sanity check the key
3fb9f87 Fix typo.
5bbedd3 zero pad DHE public key in ServerKeyExchange message for interop
70705b2 Fix ssl_cert_dup: change one 'return NULL' to 'goto err'
3b584ef Make 'err' lable in ssl_cert_dup unconditional
292bb56 Fix a bug in clienthello processing
7624a31 perlasm/x86_64-xlate.pl: refine sign extension in ea package.
10a5037 UI_OpenSSL()'s session opener fails on MacOS X
78a3e80 VMS UI_OpenSSL:
if the TT device isn't a tty, flag instead of error
fecd4c2 Check input length to pkey_rsa_verify()
5ae285e Remove extra bang
59ba83c UI code style cleanup
748a2d9 Revert "Fix heartbeat_test"
be3a7dd apps/speed.c: Fix crash when config loading fails
c477f8e INSTALL: clarify 386 and no-sse2 options.
f47201b modes/ctr128.c: fix false carry in counter increment procedure.
c4c7165 Clarify what X509_NAME_online does with
the given buffer and size
31b4307 Make SSL_read and SSL_write return
the old behaviour and document it.
09b894b Use consistent variable names
f4ef1c5 domd: Preserve Makefile time when it is unchanged
7a9d712 mklink: Do not needlessly overwrite linked files...
62f16de domd: Do not needlessly overwrite Makefiles
22cc44d mklink: Do not needlessly overwrite linked files...
ecc9551 Configure: Improve incremental build time
8ac70be Check return value of some BN functions.
3201a1d Solution proposal for issue #1647.
19e1de5 Update CHANGES and NEWS
57c4b9f bn/asm/x86_64-mont.pl: fix for CVE-2016-7055 (Low severity).
c210840 Makefile.org: clear APPS environment variable.
95873c5 Missed a mention of RT
563a34e Add a CHANGES entry for the unrecognised record type change
f118539 Fail if an unrecognised record type is received
ad69a30 Fix heartbeat_test
ba2bf83 Secure our notification email.
e022375 Fix grammar-o in CONTRIBUTING
787b2dc Add $(EX_LIBS) to the LIBDEPS for libgost.so,
just as for all other engines
0b9c5da Implement length checks as a macro
a520723 Ensure we have length checks for all extensions
83a1d4b Fix length check writing status request extension
57aa2f1 Fix a double free in ca command line
fa4c374 A zero return from BIO_read/BIO_write() could be retryable
31bf65c Fix typo (reported by Matthias St. Pierre)
0e46901 Fix leak of secrecy in ecdh_compute_key()
3ade92e Correctly find all critical CRL extensions
45f4761 remove redundant zero assignments
cdb203f %p takes void*, so make sure to cast arguments to void*
0df1caa apps: make setup_engine() and release_engine() available always
aa01b82 If an engine comes up explicitely,
it must also come down explicitely
10e60f2 Fix no-des
1c6aab6 Make 'openssl prime ""' not segfault
99c002b Fix strict-warnings build
b0161f6 Fix strict-warnings build
78ee64c Fix signatures of EVP_Digest{Sign,Verify}Update
02a0231 Ensure we handle len == 0 in ERR_err_string_n
6d69dc5 Degrade 3DES to MEDIUM in SSL2
e8e380c RT is put out to pasture
f1f9769 Add missing error string for SSL_R_TOO_MANY_WARN_ALERTS
53a71b7 apps/apps.c: initialize and de-initialize engine
around key loading
a269e5f Revert "Call ENGINE_init() before trying to use keys
from engine"
4badd2b Call ENGINE_init() before trying to use keys from engine
9702bf5 Fix NEWS error
f6e43fe Prepare for 1.0.2k-dev


I've look at any commits that *seem* like they could be more than
a bug fix or that might change the api. Aside from the two issues
related to header files, I didn't see anything to worry about.

$ git diff OpenSSL_1_0_2j..OpenSSL_1_0_2k | diffstat
.travis.yml | 2
CHANGES | 61 ++++++
CONTRIBUTING | 55 +----
Configure | 34 ++-
INSTALL | 69 +++---
Makefile.org | 3
NEWS | 8
README | 36 ---
apps/apps.c | 19 +
apps/apps.h | 4
apps/ca.c | 6
apps/cms.c | 5
apps/dgst.c | 1
apps/dh.c | 6
apps/dhparam.c | 8
apps/dsa.c | 7
apps/dsaparam.c | 8
apps/ec.c | 6
apps/ecparam.c | 10
apps/enc.c | 8
apps/gendh.c | 4
apps/gendsa.c | 8
apps/genpkey.c | 2
apps/genrsa.c | 7
apps/pkcs12.c | 7
apps/pkcs7.c | 8
apps/pkcs8.c | 5
apps/pkey.c | 5
apps/pkeyparam.c | 8
apps/pkeyutl.c | 1
apps/prime.c | 12 -
apps/rand.c | 8
apps/req.c | 5
apps/rsa.c | 5
apps/rsautl.c | 5
apps/s_cb.c | 4
apps/s_client.c | 7
apps/s_server.c | 7
apps/smime.c | 5
apps/speed.c | 14 -
apps/spkac.c | 5
apps/srp.c | 8
apps/verify.c | 5
apps/x509.c | 5
crypto/aes/asm/aes-s390x.pl | 8
crypto/asn1/p5_pbev2.c | 8
crypto/asn1/x_crl.c | 3
crypto/bn/asm/x86_64-mont.pl | 5
crypto/bn/asm/x86_64-mont5.pl | 16 -
crypto/bn/bn_exp.c | 5
crypto/bn/bn_mul.c | 5
crypto/bn/bn_prime.c | 3
crypto/bn/bn_sqr.c | 5
crypto/cms/cms_kari.c | 5
crypto/dh/dh_key.c | 2
crypto/dsa/dsa_pmeth.c | 2
crypto/ec/ec2_mult.c | 20 +
crypto/ecdh/ech_ossl.c | 4
crypto/err/err.c | 3
crypto/evp/e_aes.c | 4
crypto/evp/e_rc4_hmac_md5.c | 2
crypto/evp/evp.h | 6
crypto/evp/evp_err.c | 3
crypto/evp/pmeth_fn.c | 30 +-
crypto/evp/pmeth_lib.c | 28 --
crypto/modes/ctr128.c | 2
crypto/opensslv.h | 6
crypto/perlasm/x86_64-xlate.pl | 11 -
crypto/rsa/rsa_gen.c | 3
crypto/rsa/rsa_oaep.c | 8
crypto/rsa/rsa_pmeth.c | 4
crypto/s390xcap.c | 1
crypto/ui/ui_lib.c | 138 +++++++------
crypto/ui/ui_openssl.c | 59 +++--
demos/easy_tls/easy-tls.c | 1
doc/apps/ocsp.pod | 9
doc/crypto/EVP_DigestSignInit.pod | 2
doc/crypto/EVP_DigestVerifyInit.pod | 2
doc/crypto/RSA_generate_key.pod | 2
doc/crypto/X509_NAME_get_index_by_NID.pod | 3
doc/crypto/X509_NAME_print_ex.pod | 8
doc/ssl/SSL_CTX_set_session_cache_mode.pod | 2
doc/ssl/SSL_get_error.pod | 22 --
doc/ssl/SSL_read.pod | 32 +--
doc/ssl/SSL_write.pod | 19 -
engines/ccgost/Makefile | 2
openssl.spec | 2
ssl/bad_dtls_test.c | 5
ssl/s23_pkt.c | 12 -
ssl/s2_lib.c | 2
ssl/s2_pkt.c | 10
ssl/s3_clnt.c | 44 +++-
ssl/s3_pkt.c | 23 +-
ssl/s3_srvr.c | 33 ++-
ssl/ssl_cert.c | 4
ssl/ssl_err.c | 1
ssl/ssl_lib.c | 4
ssl/ssl_locl.h | 2
ssl/ssl_sess.c | 9
ssl/t1_lib.c | 291 ++++++++++++++++++-----------
util/domd | 11 -
util/mklink.pl | 8
102 files changed, 836 insertions(+), 634 deletions(-)




--
# Randy MacLeod. SMTS, Linux, Wind River
Direct: 613.963.1350 | 350 Terry Fox Drive, Suite 200, Ottawa, ON, Canada, K2K 2W5


Re: Recipe Specific Sysroots - An Update

Richard Purdie
 

On Sun, 2017-01-22 at 17:47 -0500, Denys Dmytriyenko wrote:
On Thu, Jan 19, 2017 at 04:38:14PM +0000, Richard Purdie wrote:
To delay this any further will mean less time for other layers to
adapt
to the changes and put the main release more at risk. I don't think
anyone would like to see this delayed until 2.4. I have given
people
plenty of warning this was coming and time to review the patch on
the
branch.
Thanks, Richard!

Will there be some sort of write-up detailing any work required for
other layers to adapt to this change? Thanks.
I did write up the specifics of the implementation details into the
main commit message for the change. For layer maintainers, the key
piece were these points:

* Recipes may fail with missing dependencies, particularly native 
  tools like gettext-native, glib-2.0-native and libxml2.0-native. 
  Some hosts have these installed and will mask these errors 

* Any recipe/class using SSTATEPOSTINSTFUNCS will need that code 
  rewriting into a postinst 

* Any postinst which expects native tools at rootfs time will need to 
  mark that dependency with PACKAGE_WRITE_DEPS.

* Relocation errors that existed before will become blocking problems 
  now.

I'm hoping this is main extent of the kinds of changes needed and that
most of the issues are limited to the core but at this point its going
to be a case of testing to know more.

FWIW everything is now in master-next for OE-Core and all the
prerequisites have merged into bitbake master. All autobuilder tests
are now green.

Cheers,

Richard


Re: Recipe Specific Sysroots - An Update

Denys Dmytriyenko
 

On Thu, Jan 19, 2017 at 04:38:14PM +0000, Richard Purdie wrote:
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=rpurdie/wip-rss

I've been quiet on this for a while but the patches are progressing and
I've been keeping this branch updated. Its proving a bit painful and
time consuming to fix issues as changes mean large rebuilds so the
debug cycles are getting long and oe-selftest takes several hours.

There was a bit of a breakthrough in making eSDK work again and I'm now
trying to work through the various test cases that are failing on the
autobuilders. I've nearly sent this email several times but the test
results are gradually improving and I think the series is looking
pretty reasonable now.

As things stand, of the ~250 oe-selftests, around 6 or so are failing.
We're having some issues with wic which I really appreciate the help
from Ed on. There have been musl, multilib, tiny, systemd and other
issues but I believe these to be resolved.

Thanks also go to Paul for fixing devtool to work with this.

The change has found dependency problems, mostly in missing tools like
glib-2.0-native and libxml2-native.

The biggest challenge we face is postinstall dependencies. There is a
separate email thread about that one and thanks to some help from
Jussi, I believe we should have a patchset to address this.

The bitbake changes have been posted and are in master-next, as are
most of the OE-Core ones which I could separate out.

M2 is due on Monday and my plan is probably to merge in Recipe Specific
Sysroots and then handle any fallout from there, even if it delays M2
as I don't think I'll be able to take this much further alone. That
implies I'll also take the postinstall changes as this depends on
those.

To delay this any further will mean less time for other layers to adapt
to the changes and put the main release more at risk. I don't think
anyone would like to see this delayed until 2.4. I have given people
plenty of warning this was coming and time to review the patch on the
branch.
Thanks, Richard!

Will there be some sort of write-up detailing any work required for other
layers to adapt to this change? Thanks.

--
Denys


Re: Package postinstall dependencies - Introduction of PACKAGE_WRITE_DEPS

Patrick Ohly
 

On Thu, 2017-01-19 at 18:17 -0200, Otavio Salvador wrote:
On Thu, Jan 19, 2017 at 5:47 PM, Patrick Ohly <patrick.ohly@...> wrote:
On Wed, 2017-01-18 at 21:26 +0000, Richard Purdie wrote:
On Wed, 2017-01-18 at 11:57 -0600, Mark Hatle wrote:
Does that make it clearer why we need this?
I certainly understand it more.. I'm just struggling with how to
explain when
someone would need this or not, to someone who hasn't hit one of the
extraction
problems. I think that is by far my biggest remaining concern, how
to make
sure people can understand they need to do this and when...
The rules in some ways are simple, it comes down to:

"""
If your postinstall can execute at rootfs creation time rather than on
target but depends on a native tool in order to execute, you need to
list that tool in PACKAGE_WRITE_DEPENDS.
"""
Looking at this description, it is not at all clear to me why the
variable is named PACKAGE_WRITE_DEPENDS. From the description, the
things listed in it are dependencies of do_rootfs, not of
do_package_write_*, even if it happens to be implemented that way.

I like PS_NATIVE_DEPENDS better. Just my 2 cents.
PACKAGE_SCRIPTS_DEPENDS maps better to what they are intended to be
used for, I think.
Agreed, that's even better.

--
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.


Re: Package postinstall dependencies - Introduction of PACKAGE_WRITE_DEPS

Bruce Ashfield
 



On Thu, Jan 19, 2017 at 3:17 PM, Otavio Salvador <otavio.salvador@...> wrote:
On Thu, Jan 19, 2017 at 5:47 PM, Patrick Ohly <patrick.ohly@...> wrote:
> On Wed, 2017-01-18 at 21:26 +0000, Richard Purdie wrote:
>> On Wed, 2017-01-18 at 11:57 -0600, Mark Hatle wrote:
>> > > Does that make it clearer why we need this?
>> > I certainly understand it more.. I'm just struggling with how to
>> > explain when
>> > someone would need this or not, to someone who hasn't hit one of the
>> > extraction
>> > problems.   I think that is by far my biggest remaining concern, how
>> > to make
>> > sure people can understand they need to do this and when...
>>
>> The rules in some ways are simple, it comes down to:
>>
>> """
>> If your postinstall can execute at rootfs creation time rather than on
>> target but depends on a native tool in order to execute, you need to
>> list that tool in PACKAGE_WRITE_DEPENDS.
>> """
>
> Looking at this description, it is not at all clear to me why the
> variable is named PACKAGE_WRITE_DEPENDS. From the description, the
> things listed in it are dependencies of do_rootfs, not of
> do_package_write_*, even if it happens to be implemented that way.
>
> I like PS_NATIVE_DEPENDS better. Just my 2 cents.

PACKAGE_SCRIPTS_DEPENDS maps better to what they are intended to be
used for, I think.

I tend to agree with the expanded "package", I keep reading "PS" as any
number of random things ... including "professional services".

Cheers,

Bruce
 


--
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750
_______________________________________________
Openembedded-architecture mailing list
Openembedded-architecture@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-architecture



--
"Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end"


Re: Package postinstall dependencies - Introduction of PACKAGE_WRITE_DEPS

Otavio Salvador
 

On Thu, Jan 19, 2017 at 5:47 PM, Patrick Ohly <patrick.ohly@...> wrote:
On Wed, 2017-01-18 at 21:26 +0000, Richard Purdie wrote:
On Wed, 2017-01-18 at 11:57 -0600, Mark Hatle wrote:
Does that make it clearer why we need this?
I certainly understand it more.. I'm just struggling with how to
explain when
someone would need this or not, to someone who hasn't hit one of the
extraction
problems. I think that is by far my biggest remaining concern, how
to make
sure people can understand they need to do this and when...
The rules in some ways are simple, it comes down to:

"""
If your postinstall can execute at rootfs creation time rather than on
target but depends on a native tool in order to execute, you need to
list that tool in PACKAGE_WRITE_DEPENDS.
"""
Looking at this description, it is not at all clear to me why the
variable is named PACKAGE_WRITE_DEPENDS. From the description, the
things listed in it are dependencies of do_rootfs, not of
do_package_write_*, even if it happens to be implemented that way.

I like PS_NATIVE_DEPENDS better. Just my 2 cents.
PACKAGE_SCRIPTS_DEPENDS maps better to what they are intended to be
used for, I think.


--
Otavio Salvador O.S. Systems
http://www.ossystems.com.br http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854 Mobile: +1 (347) 903-9750


Re: Package postinstall dependencies - Introduction of PACKAGE_WRITE_DEPS

Patrick Ohly
 

On Wed, 2017-01-18 at 21:26 +0000, Richard Purdie wrote:
On Wed, 2017-01-18 at 11:57 -0600, Mark Hatle wrote:
Does that make it clearer why we need this?
I certainly understand it more.. I'm just struggling with how to
explain when
someone would need this or not, to someone who hasn't hit one of the
extraction
problems. I think that is by far my biggest remaining concern, how
to make
sure people can understand they need to do this and when...
The rules in some ways are simple, it comes down to:

"""
If your postinstall can execute at rootfs creation time rather than on
target but depends on a native tool in order to execute, you need to
list that tool in PACKAGE_WRITE_DEPENDS.
"""
Looking at this description, it is not at all clear to me why the
variable is named PACKAGE_WRITE_DEPENDS. From the description, the
things listed in it are dependencies of do_rootfs, not of
do_package_write_*, even if it happens to be implemented that way.

I like PS_NATIVE_DEPENDS better. Just my 2 cents.

--
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.


Re: Package postinstall dependencies - Introduction of PACKAGE_WRITE_DEPS

Richard Purdie
 

On Thu, 2017-01-19 at 12:51 -0500, Randy MacLeod wrote:
Is there any chance that we could make 'do_populate_sysroot'
be the default task (with the possibility to over-ride)
so we'd have:

    PS_DEPENDS += "foo-native bar-native"

and since we've come this far... only -native dependencies are
sensible since this is running on the builder, so we could just have:

    PS_DEPENDS += "foo bar"

or if you want to spell it out:

    PS_NATIVE_DEPENDS += "foo bar"
I agree with stripping out the task piece, I was planning to work
something out with that. We could do with this cleanup in other places
too.

Not all have the form -native (we have some -cross) and I think that
removing that would be a step too far and make it much less clear what
these things were doing. 

Cheers,

Richard


Re: Package postinstall dependencies - Introduction of PACKAGE_WRITE_DEPS

Randy MacLeod
 

On 2017-01-19 12:14 PM, Otavio Salvador wrote:
On Thu, Jan 19, 2017 at 2:56 PM, Jussi Kukkonen
<jussi.kukkonen@...> wrote:
Regarding the naming discussion: It would be great if it was somehow made
obvious to a recipe writer who adds a postinstall script that they may need
to add PACKAGE_WRITE_DEPS as well... but I don't really have practical
suggestions here.
What about DEPENDS_PKG_SCRIPTS ?
or PS_DEPENDS with the manual defining PS
and users thinking that it means any of:
Package Script / Populate Sysroot / Post Script

Also, all of Jussi's package changes are of the form:

PS_DEPENDS += "foo-native:do_populate_sysroot \
bar-native:do_populate_sysroot"

Is there any chance that we could make 'do_populate_sysroot'
be the default task (with the possibility to over-ride)
so we'd have:

PS_DEPENDS += "foo-native bar-native"

and since we've come this far... only -native dependencies are
sensible since this is running on the builder, so we could just have:

PS_DEPENDS += "foo bar"

or if you want to spell it out:

PS_NATIVE_DEPENDS += "foo bar"


--
# Randy MacLeod. SMTS, Linux, Wind River
Direct: 613.963.1350 | 350 Terry Fox Drive, Suite 200, Ottawa, ON, Canada, K2K 2W5


















(where NATIVE is pronounced circus ... inside joke )


Re: Package postinstall dependencies - Introduction of PACKAGE_WRITE_DEPS

Otavio Salvador
 

On Thu, Jan 19, 2017 at 2:56 PM, Jussi Kukkonen
<jussi.kukkonen@...> wrote:
Regarding the naming discussion: It would be great if it was somehow made
obvious to a recipe writer who adds a postinstall script that they may need
to add PACKAGE_WRITE_DEPS as well... but I don't really have practical
suggestions here.
What about DEPENDS_PKG_SCRIPTS ?

--
Otavio Salvador O.S. Systems
http://www.ossystems.com.br http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854 Mobile: +1 (347) 903-9750


Re: Package postinstall dependencies - Introduction of PACKAGE_WRITE_DEPS

Jussi Kukkonen <jussi.kukkonen@...>
 

On 18 January 2017 at 16:23, Richard Purdie <richard.purdie@...> wrote:
One problem we're seeing with recipe specific sysroots (rss) is that
the dependencies package postinstalls have is badly represented.

For example, dbus adds a user and this needs tools from shadow-native.

If you run a do_rootfs build from an empty TMPDIR and have a populated
sstate cache, how does the system know it needs to install shadow-
native from sstate so that dbus' postinsts aren't deferred to on
target?

Currently its a bit of mess/hack. There is a list of hardcoded special
cases in sstate.bbclass but its incomplete. There are also lists of
special dependencies in the image.bbclass rootfs code to pull in things
like depmodwrapper-cross and ldconfig.

Some things are detected at build time like ldconfig dependencies so
there isn't much that can be done for them but most dependencies like
depmod, useradd, gtk-icon-cache, gdk-pixbuf, systemd and so on are
known in advance.

With rss, the problem gets more complicated since not only do we need
to ensure the tools get extracted from sstate but that they must get
into the image recipe's own native sysroot.

I've spent quite some time pondering how to do this. A normal DEPENDS
means something quite different so we need some kind of new markup.
Adding new kinds of dependencies to bitbake doesn't seem attractive
though.

I'm going to propose a new PACKAGE_WRITE_DEPS variable which gets added
to the [depends] flag of the do_package_write_XXX tasks.

What this means is that the dependency is indicated in the task graph
and its done in a generic way which the code can identify and act upon.
These tasks don't currently have much in the way of dependencies other
than tools to actually build packages.

The PACKAGE_WRITE_DEPS can either be in addition to DEPENDS if the
recipe needs these tools at build time, or if its only needed for the
postinst, PACKAGE_WRITE_DEPS can be sufficient and the DEPENDS can be
removed.

I've had a go at a quick proof of concept patch below. This includes
removing the hardcoded pieces from sstate.bbclass but I'm still in the
process of testing this.

Right now, I wanted to put this proposal out there and see if people
were ok with the general idea. There are perhaps some tweaks that could
be made to the final implementation. I am conscious the M2 deadline is
looming and if we want rss in this release, we need to be thinking
about merging code soon though.

Richard asked me to test this and look at the remaining unhandled postinstalls. I've now done that, patches for the missing PACKAGE_WRITE_DEPS are in the "jku/wip-rss" branch in poky-contrib http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=jku/wip-rss. The branch is based on Richards rpurdie/wip-rss -- but I did test this (Richards) patch  and mine on top of master first.

It all seems to work fine: I've not managed to break the dependency handling yet and things like running target tools via qemu in postinstall seem to do the right thing.  I'll continue testing though: especially with recipe-specific-sysroots there's quite a few new things to grasp so it's a little slower going than normal.


Regarding the naming discussion: It would be great if it was somehow made obvious to a recipe writer who adds a postinstall script that they may need to add PACKAGE_WRITE_DEPS as well... but I don't really have practical suggestions here.

Jussi



Re: Package postinstall dependencies - Introduction of PACKAGE_WRITE_DEPS

Burton, Ross <ross.burton@...>
 


On 19 January 2017 at 16:13, Trevor Woerner <twoerner@...> wrote:
Is there not already a mechanism for generically saying "I need <this> task of
<that> recipe to be completed before my <such-and-such> task can run" of which
DEPENDS and RDEPENDS are simply short-hand notations?

If "yes" (and I understand the discussion correctly) couldn't that be employed
here?

If "no" (and I understand the discussion correctly) wouldn't adding such a
mechanism be a possible solution. The use of which, hopefully, would make it
easier for a user to understand what's going on (without requiring ad-hoc
knowledge of the system)?

Yes, there is.  do_some_task[depends] += "some-recipe:some_other_task".

DEPENDS is just a neat way of setting [depends] on do_configure to the list of recipes in DEPENDS's do_populate_sysroot task.

The problem is that do_package_write doesn't exist: the task is do_package_write_deb ..._ipk and ..._rpm.  Thus the single variable that the classes and recipes write to, and the package classes do:

do_package_write_rpm[depends] += "${PACKAGE_WRITE_DEPS}"

Simples!  :)

IMHO, the current name is fine.  As the dependencies are added to do_package_write, it's the most obvious name.

Ross

1281 - 1300 of 1685