[Openembedded-architecture] [openembedded-architecture] sstate equivalency
Qi.Chen at windriver.com
Wed May 2 02:41:12 UTC 2018
On 05/02/2018 04:28 AM, Joshua Watt wrote:
> I have some interest it looking into the sstate equivalency as
> discussed in the Yocto Project Technical Meeting. I know there have
> been a few discussions about how to handle this problem, so please
> chime in with any information or background you may have.
> To start things off, I figured I would try to briefly summarize the
> problem so that we have a clear idea what we are trying to solve.
> My understanding is that there are many changes that can be made to a
> recipe that don't actually have any affect on what the recipe
> generates (comments in shell tasks(?), that sort of thing), but still
> figure into the sstate task hashes. When such changes are detected, it
> would be convenient to users if the build could detect that these task
> hashes are actually equivalent, and allow one sstate cache object to
> be used in place of the other (e.g. If you have sstate object A and
> your build calculates the hash as B, you can ask somewhere, "is A
> equivalent to B?", and if so, use A in lieu of rebuilding to produce
My focus is on user experience.
sstate is a successful mechanism because it has simple and clear rules.
From the user's view, it's easy to understand why something is rebuilt.
Adding such complex equivalency checking will surprise users, and then
So regardless of the solution, the goal itself is trying to break these
simple and clear rules.
> I think one of the biggest open questions I have is the mechanism for
> detecting if some new task hash is equivalent to an old one (known)
> one. I think there are two basic options:
> 1) Do an actual build and determine equivalence afterward. This seems
> the easiest option in my mind, but it does require at least one build
> of the new hash to determine that it is functionally equivalent to an
> old hash, and then a mechanism to publish this result so that
> hopefully no one else has to do it also.
This solution doesn't converge.
Say we have in database stating that recipe A's do_install would
consider the following changes to not affect the final output.
1) adding comment "# this is 1"
2) adding comment "# this is 2"
3) adding comment "# this is 3"
Then, when user adds a comment "# this is 1", A's do_install is not
rerun; when the user adds a comment "# this is 4", and the do_install is
rerun. It's not reasonable.
Basically we have infinite numbers of comments, so this solution does
> 2) Try to calculate if the new hash is going to be equivalent without
> doing a build. This seems *much* harder, IMHO... but also eliminates
> the need for a "central" database of equivalent hashes (because
> everyone can determine it themselves).
This is also not doable. Because python and shell codes' results cannot
be got by merely parsing them.
The above are my opinions on this issue.
> this is very doable. As a simplistic first approach, I think it should
> be entirely possible to piggyback on the reproducable build effort
> that has been on-going to solve the question of "when are two tasks
> identical". Essentially, if the outputs from two tasks are equivalent,
> the two input task hashes can be considered to be equivalent. You
> could of course perhaps get more intelligent than that, but I don't
> have a clear idea of what that would look like.
> After that, the other question I can think of right now is "how are
> results published". I'm a little less clear on what ideas people had
> there, so please chime in. I think you could go a simple as "symlinks
> in the sstate cache", and as complicated as a dedicated hierarchy of
> server with authentication and signing.
> I'm sure I'm glossing over a lot here....
> Joshua Watt
> Openembedded-architecture mailing list
> Openembedded-architecture at lists.openembedded.org
More information about the Openembedded-architecture