Date
1 - 20 of 28
[PATCH] base/patch: Disable network for unpack/patch/configure/compile/install
|
Richard Purdie
Use the newly added nonetwork task flag to disable network access where
possible in unpack/patch/configure/compile/install tasks. We can't disable networking in sstate tasks due to sstate downloads and also so we can report hash equivalence to the server. Signed-off-by: Richard Purdie <richard.purdie@...> --- meta/classes/base.bbclass | 4 ++++ meta/classes/patch.bbclass | 1 + 2 files changed, 5 insertions(+) diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass index b709777f243..e4c6c983b59 100644 --- a/meta/classes/base.bbclass +++ b/meta/classes/base.bbclass @@ -214,6 +214,7 @@ python create_source_date_epoch_stamp() { oe.reproducible.epochfile_write(source_date_epoch, d.getVar('SDE_FILE'), d) } do_unpack[postfuncs] += "create_source_date_epoch_stamp" +do_unpack[nonetwork] = "1" def get_source_date_epoch_value(d): return oe.reproducible.epochfile_read(d.getVar('SDE_FILE'), d) @@ -358,6 +359,7 @@ base_do_configure() { echo ${BB_TASKHASH} > ${CONFIGURESTAMPFILE} fi } +do_configure[nonetwork] = "1" addtask compile after do_configure do_compile[dirs] = "${B}" @@ -368,11 +370,13 @@ base_do_compile() { bbnote "nothing to compile" fi } +do_compile[nonetwork] = "1" addtask install after do_compile do_install[dirs] = "${B}" # Remove and re-create ${D} so that is it guaranteed to be empty do_install[cleandirs] = "${D}" +do_install[nonetwork] = "1" base_do_install() { : diff --git a/meta/classes/patch.bbclass b/meta/classes/patch.bbclass index 8de70254919..57aaf7c31d1 100644 --- a/meta/classes/patch.bbclass +++ b/meta/classes/patch.bbclass @@ -164,6 +164,7 @@ patch_do_patch[vardepsexclude] = "PATCHRESOLVE" addtask patch after do_unpack do_patch[dirs] = "${WORKDIR}" +do_patch[nonetwork] = "1" do_patch[depends] = "${PATCHDEPENDENCY}" EXPORT_FUNCTIONS do_patch -- 2.32.0 |
|
|
|
Alexander Kanavin
Should there be tests for this? Would be good to check that the network is indeed disabled in these tasks. Alex On Thu 23. Dec 2021 at 2.20, Richard Purdie <richard.purdie@...> wrote: Use the newly added nonetwork task flag to disable network access where |
|
|
|
Peter Kjellerstedt
toggle quoted message
Show quoted text
-----Original Message-----Since no tasks except fetch (and apparently the sstate tasks) are expected to use the network, wouldn't it make more sense to reverse this flag? I.e., add do_fetch[network] = "1" instead. That way you don't get away with adding some random task and using the network from it unless you explicitly state that you will. //Peter |
|
|
|
Konrad Weihmann <kweihmann@...>
On 23.12.21 11:49, Peter Kjellerstedt wrote:
This is actually a brilliant idea, which would also make it easier to control this behavior from a user's perspective-----Original Message-----Since no tasks except fetch (and apparently the sstate tasks) are expected //Peter |
|
|
|
Richard Purdie
On Thu, 2021-12-23 at 12:31 +0100, Konrad Weihmann wrote:
Part of me wonders if we really do want to make this "easy" for the user :/ Cheers, Richard |
|
|
|
Richard Purdie
On Thu, 2021-12-23 at 08:28 +0300, Alexander Kanavin wrote:
Should there be tests for this? Would be good to check that the network isI think Ross was looking at that. The challenge will be the list needed to know which distros this works on and which it does not (I know centos7, debian9 and debian10 don't work)... Cheers, Richard |
|
|
|
Konrad Weihmann <kweihmann@...>
On 23.12.21 14:11, Richard Purdie wrote:
On Thu, 2021-12-23 at 12:31 +0100, Konrad Weihmann wrote:"easier" may have sounded wrong - what I mean is that it would be then explicit, with no network access being the default.Part of me wonders if we really do want to make this "easy" for the user :/ So for instance a simple grep on a new layer could confirm whether a layer needs network access or not - while with the opt-out method of your patch things may be a bit more difficult to check. side note: this could also be put into the check-layer script (as I think the official badge shall just go to layers that do not need network access out of the scope defined by core) And yes I got a ton of additional tasks in the layers I work with and none of these would be affected by the opt-in model. Either way, I really love to see this feature finally happen Cheers, |
|
|
|
Andreas Müller
On Thu, Dec 23, 2021 at 2:19 PM Konrad Weihmann <kweihmann@...> wrote:
Out of curiosity (will see fallout at least on libreoffice): Why is this a feature to love? Andreas |
|
|
|
Konrad Weihmann <kweihmann@...>
On 23.12.21 15:52, Andreas Müller wrote:
On Thu, Dec 23, 2021 at 2:19 PM Konrad Weihmann <kweihmann@...> wrote:Simply said because it makes validation of changes way easier.Out of curiosity (will see fallout at least on libreoffice): Why is In before I had to build a change in a normal environment (w network access) to verify that it actually builds, then populate some mirror and rebuild the same change in an environment without network access. Now I only will need one run to be sure that it the same recipe will even build in 10yrs from now (assuming that a dl-mirror exists). Andreas |
|
|
|
Jose Quaresma
Peter Kjellerstedt <peter.kjellerstedt@...> escreveu no dia quinta, 23/12/2021 à(s) 10:49: > -----Original Message----- It is more safe and easy to check what are the tasks that need network access. Jose
Best regards, José Quaresma |
|
|
|
Alexander Kanavin
On Thu, 23 Dec 2021 at 14:11, Richard Purdie <richard.purdie@...> wrote: > This is actually a brilliant idea, which would also make it easier to If people are determined to subvert this, they're likely to do it by sandwiching additional tasks between configure/compile/install in both cases. It's better to explicitly force adding the 'network' flag to those extra tasks then, as it can be clearly seen, and questions can be asked in code review. Alex |
|
|
|
is this list of failures due to this patch ?
https://errors.yoctoproject.org/Errors/Build/137579/ On Wed, Dec 22, 2021 at 3:20 PM Richard Purdie <richard.purdie@...> wrote:
|
|
|
|
Richard Purdie
On Thu, 2021-12-23 at 22:00 -0800, Khem Raj wrote:
is this list of failures due to this patch ?Looks likely, yes. Recipes shouldn't be attempting downloads in do_compile :( Cheers, Richard |
|
|
|
Konrad Weihmann <kweihmann@...>
I had a look at the failures and most of them are fairly easy to fix - all but the go recipes like influxdb.
toggle quoted message
Show quoted text
I'm not sure how to deal with that, so there aren't that many options here. Either - inject all the needed with a fixed revision, but that would prevent devtool from catching updates to them properly - create recipes for all of them and pray that there won't be a circular dependency in any of them - allow go to work with network (the least favorable option IMO) in the case of influxdb for instance we are talking about github.com/peterh/liner golang.org/x/crypto github.com/influxdata/influxql github.com/influxdata/flux github.com/BurntSushi/toml github.com/influxdata/usage-client golang.org/x/text go.uber.org/zap github.com/klauspost/pgzip github.com/prometheus/client_golang github.com/jsternberg/zap-logfmt github.com/mattn/go-isatty collectd.org github.com/bmizerany/pat github.com/dgrijalva/jwt-go/v4 github.com/gogo/protobuf github.com/golang/snappy github.com/tinylib/msgp github.com/opentracing/opentracing-go golang.org/x/sync github.com/apache/arrow/go/arrow github.com/pkg/errors google.golang.org/grpc github.com/kraj/xxhash github.com/influxdata/roaring github.com/xlab/treeprint golang.org/x/time golang.org/x/sys github.com/jwilder/encoding github.com/dgryski/go-bitstream where especially the golang.org/x modules are known to have circular dependencies like x/a relies on x/b, while x/b requires x/c and x/c needs x/a for building. Not sure how to tackle this - any thoughts? On 24.12.21 09:30, Richard Purdie wrote:
On Thu, 2021-12-23 at 22:00 -0800, Khem Raj wrote:is this list of failures due to this patch ?Looks likely, yes. Recipes shouldn't be attempting downloads in do_compile :( |
|
|
|
Stefan Herbrechtsmeier
Hi Richard,
Am 24.12.21 um 09:30 schrieb Richard Purdie: On Thu, 2021-12-23 at 22:00 -0800, Khem Raj wrote:In this case the go and especially go-mod class is broken by design because it downloads all its dependencies after do_fetch.is this list of failures due to this patch ?Looks likely, yes. Recipes shouldn't be attempting downloads in do_compile :( Regards Stefan |
|
|
|
Stefan Herbrechtsmeier
Hi Konrad,
Am 24.12.21 um 11:36 schrieb Konrad Weihmann: I had a look at the failures and most of them are fairly easy to fix - all but the go recipes like influxdb.The go class doesn't work without network by default if the project doesn't provide a vendor folder. I'm not sure how to deal with that, so there aren't that many options here.This is a common problem for all language specific package managers (python / pip, Node.js / npm, Rust / Carge, go) and we need a common solution. EitherThis is possible if we extend recipetool to add the dependencies to the recipe. Rust and npm are using this option and I have a WIP to improve the npm solution. But in any case we loose a lot of advantages of OE. - create recipes for all of them and pray that there won't be a circular dependency in any of themThe circular dependency is only the least problem and could be fixed by splitting the dependency and code into separate recipes. The main problem is the huge increase of recipe count and parse time. Furthermore we need the possibility to test recipe updates of dependencies. Python is using this option. - allow go to work with network (the least favorable option IMO)This is how the current go and especially go-mod class work. in the case of influxdb for instance we are talking aboutIsn't the circular dependencies problem mainly a problem of the native packages? In many cases the dependency only exists at compile time of the main project and not at compile time of the dependency itself. In case of a native package all dependency sysroots need to be finished before the recipe sysroot itself. Regards Stefan |
|
|
|
On Fri, Dec 24, 2021 at 2:36 AM Konrad Weihmann <kweihmann@...> wrote:
right. Unless we can fix these modern language ecosystem use in OE this is going to be quite worrisome to enforce unless there is a tool for folks to convert these innumerable dependencies into DEPENDS or SRC_URIs, perhaps devtool could be enhanced. Asking folks to add so many entries by hand is a no go. Perhaps its better to apply this in stages, where it is enabled on C/C++ and some of other ecosystems where package management and build systems are not part of language ecosytem. I don't even have a formulated idea in my mind as to how to solve them for OE and bitbake method of fetching.
|
|
|
|
Alexander Kanavin
On Sat, 25 Dec 2021 at 20:32, Stefan Herbrechtsmeier <stefan@...> wrote: > I'm not sure how to deal with that, so there aren't that many options here. I tend to think that the best (and the hardest) option is to improve these tools so that they're usable inside do_fetch (e.g. fulfil the caching/reproducibility criteria for a bitbake fetcher), and the needed changes are acceptable to upstreams. Alex |
|
|
|
Konrad Weihmann <kweihmann@...>
What I so far don't really get is why increase in parsing time is such a big deal.
I admit when we're talking about npm it's some kind of a drastic increase in recipes one would have to maintain, just because some random project decides to use a trillion dependencies instead of writing two or three lines of code more. Still I come to think this might be actually beneficial, as it shows how broken npm is from a distribution perspective - as it may be that some users actually start to access the situation when they are actually aware what monstrosity of a dep tree they are inheriting by just a single npm module. So this is mind - and I don't want to sound radical - I would rather abandon npm and go support in core than to sacrifice closing the one main loophole in core that is preventing true accessibility at the moment. Which is uncontrolled fetching outside of the fetching task, as this invalidates everything you will get in the end in terms of licensing, quality reports that haven't been done as part of the build and even CVE checking becomes pretty much worthless if one is allowed to inject random code on the fly into the build - and in the end everything that can't be assessed outside of the build is pretty much non-existing for most of the assessments I had to work with. In the past I showed that npm and go can be made to work with these principles, even if they would introduce a different way of working and maybe the need for better tooling - *but* they can work with how oe-core works at the moment - to the expense of parsing time - and that's pretty much it from my point of view. Still the gains outweigh it by far in my opinion, as it would make all of that accessible by common tooling already in place - including true reproducibility and builtin quality reports. BTW I don't think rust and therefore cargo is heavily affected by it, as the cargo to bitbake scripting works kind in the way I would imagine for npm and go too - so far just npm and go are really really bad to handle in this case. I'm happy to help on such scripting and tooling, but I don't see much worth in either accepting the fact that "modern languages" have to include some not validatable "magic" (which then would mean allowing uncontrolled network access) or working around the fact that a trillion dependencies is still a trillion dependencies no matter how you put it ( :) ) Regards Konrad |
|
|
|
Konrad Weihmann <kweihmann@...>
On 25.12.21 20:32, Stefan Herbrechtsmeier wrote:
Isn't the circular dependencies problem mainly a problem of the native packages? In many cases the dependency only exists at compile time of the main project and not at compile time of the dependency itself. In case of a native package all dependency sysroots need to be finished before the recipe sysroot itself.Unfortunately it is not - it's happens that the go.mod already creates this circular dependency. The classic example is that golang.org/x/tools depends on parts of golang.org/x/text (and guess what) golang.org/x/text depends solely on golang.org/x/tools :facepalm: For go itself this isn't a big issue, as the compiler just pulls all the needed code into the compile workspace and build a binary out of it, which can be reused later on. But for bitbake it's a big issue, as either you pull in golang.org/x/text + golang.org/x/tools sources into the golang.org/x/text workspace, stripping you off the possibility to properly watch the revision of golang.org/x/tools (in terms of devtools and co) - or you simply end up with a circular dependency. I scratched my head on this one for weeks, till I found a way to actually use the latest greatest of each go module each coming as a recipe of its own, while avoiding above mentioned situations -- and it's kind of a huge break to how things are done right now. I'm pretty sure the same will happen in rust sooner or later. And I personally don't see how to break it as I don't know any way go distinguishes between compile and runtime dependency - afaik they all need to be present at compile time |
|
|