Hi,

I've a few things I've been pondering for a while and given the summit
and OEDVM this week, I thought I'd sent out this email and some RFC
patches.

Export variable flag (and unexport/network)
-------------------------------------------

The patch sent as an RFC to the bitbake list has the details on this
one, do we make the export and network flags more useful/useable at the
cost of a 4% parsing speed hit? I'm torn, I just don't know.

addpylib directive
------------------

The easier one is probably the "addpylib" directive. This would move
the python module import code into a usable ABI within conf files such
as layer.conf. There are patches to OE-Core and Bitbake on the mailing
list as RFCs.

The reasons for doing this are:

* establish a clean API for adding python modules
* drop what was originally a workaround from base.bbclass
* allow potential to extend existing functionality to cover such 
modules accesses to the datastore in task hashes

I chose to make it append to sys.path in the new implementation and
stack on a first come, first in the queue approach so it will follow
bblayers.conf only. We have too many ways for things to break and don't
need that level of potential craziness IMO.

I added BBIMPORT as a way to list the namespaces to import into
bitbake. I did allow other namespaces to be imported as this is how
time, os and sys are accessed by our anonymous code. I suspect we
should perhaps only allow that from a different API but we can discuss
that or improve in due course.

There is a second patch, admittedly a bit thrown together which
implements variable dependency information from the python functions.
It basically runs the imported functions through our codeparser and
then merges that dependency information with the datastore task
signature/hash generation code. There are a number of things that need
to be improved with it but it does prove the concept can work. I don't
like the global variable, some of the functionality needs exporting to
a function and there are probably data lifecycle issues there but it is
good enough to start a discussion of the idea.

The big win if we do this is allowing large chunks of package.bbclass
and friends to be moved into lib/oe without having to worry about
variable accesses. This should then lead to better memory usage and
lower cache sizes.

Right now it doesn't invalidate the cache on changes to the functions
themselves, only the variable dependencies changing. That can be
changed easily enough if desired.

cache data changes
------------------

We now have the potential to add all of the data that makes up basehash
(and hence taskhash) to the bitbake cache file. The upside would be
much nicer information about why basehashes mismatch. The downsides
are:

* increased cache file size
* slower parsing speed
* slower load of the cache file back into memory

I have patches which pickle the cache all in one go at the expense of
the cache load progress bar. I did also develop a patch to support
efficient incremental pickling of the data and patches which compress
the cache file and add the basehash data. There is a tradeoff against
various speeds to be made and I'm torn on what to do. We could make the
data an option you could enable for debugging but often the situations
are hard to reproduce so that may not be as useful as it first seems.
These patches are in my poky-contrib branches.


There are further ideas I have floating around but these ones are the
ones there are patches out there for and that are probably ready/worthy
of discussion.

Cheers,

Richard