[Openembedded-architecture] [openembedded-architecture] sstate equivalency

ChenQi Qi.Chen at windriver.com
Wed May 2 02:41:12 UTC 2018


On 05/02/2018 04:28 AM, Joshua Watt wrote:
> All,
>
> I have some interest it looking into the sstate equivalency as
> discussed in the Yocto Project Technical Meeting. I know there have
> been a few discussions about how to handle this problem, so please
> chime in with any information or background you may have.
>
> To start things off, I figured I would try to briefly summarize the
> problem so that we have a clear idea what we are trying to solve.
>
> My understanding is that there are many changes that can be made to a
> recipe that don't actually have any affect on what the recipe
> generates (comments in shell tasks(?), that sort of thing), but still
> figure into the sstate task hashes. When such changes are detected, it
> would be convenient to users if the build could detect that these task
> hashes are actually equivalent, and allow one sstate cache object to
> be used in place of the other (e.g. If you have sstate object A and
> your build calculates the hash as B, you can ask somewhere, "is A
> equivalent to B?", and if so, use A in lieu of rebuilding to produce
> B).

Hi Joshua,

My focus is on user experience.
sstate is a successful mechanism because it has simple and clear rules. 
 From the user's view, it's easy to understand why something is rebuilt.
Adding such complex equivalency checking will surprise users, and then 
confuse them.
So regardless of the solution, the goal itself is trying to break these 
simple and clear rules.

> I think one of the biggest open questions I have is the mechanism for
> detecting if some new task hash is equivalent to an old one (known)
> one. I think there are two basic options:
>
> 1) Do an actual build and determine equivalence afterward. This seems
> the easiest option in my mind, but it does require at least one build
> of the new hash to determine that it is functionally equivalent to an
> old hash, and then a mechanism to publish this result so that
> hopefully no one else has to do it also.

This solution doesn't converge.
Say we have in database stating that recipe A's do_install would 
consider the following changes to not affect the final output.
1) adding comment "# this is 1"
2) adding comment "# this is 2"
3) adding comment "# this is 3"
Then, when user adds a comment "# this is 1", A's do_install is not 
rerun; when the user adds a comment "# this is 4", and the do_install is 
rerun. It's not reasonable.
Basically we have infinite numbers of comments, so this solution does 
not converge.

> 2) Try to calculate if the new hash is going to be equivalent without
> doing a build. This seems *much* harder, IMHO... but also eliminates
> the need for a "central" database of equivalent hashes (because
> everyone can determine it themselves).
>

This is also not doable. Because python and shell codes' results cannot 
be got by merely parsing them.

The above are my opinions on this issue.

Best Regards,
Chen Qi


> this is very doable. As a simplistic first approach, I think it should
> be entirely possible to piggyback on the reproducable build effort
> that has been on-going to solve the question of "when are two tasks
> identical". Essentially, if the outputs from two tasks are equivalent,
> the two input task hashes can be considered to be equivalent. You
> could of course perhaps get more intelligent than that, but I don't
> have a clear idea of what that would look like.
>
> After that, the other question I can think of right now is "how are
> results published". I'm a little less clear on what ideas people had
> there, so please chime in. I think you could go a simple as "symlinks
> in the sstate cache", and as complicated as a dedicated hierarchy of
> server with authentication and signing.
>
> I'm sure I'm glossing over a lot here....
>
> Thanks,
> Joshua Watt
> _______________________________________________
> Openembedded-architecture mailing list
> Openembedded-architecture at lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-architecture
>




More information about the Openembedded-architecture mailing list