- [PATCH 1/2] sstate: don't limit the thread pool size when checking mirrors
Re: [PATCH 1/2] sstate: don't limit the thread pool size when checking mirrors
On Thu, 2021-12-02 at 22:05 +0000, Jose Quaresma wrote:
> This improves the time needed to check the mirrors,
> reducing it from 50s to 8s when building the core-image-minimal
> using the yocto upstream sstate and hashequivlance servers
> on a machine with 8 cpu cores.
> Tested with:
> SSTATE_MIRRORS = "file://.* http://sstate.yoctoproject.org/dev/PATH;downloadfilename=PATH"
> BB_HASHSERVE_UPSTREAM = "typhoon.yocto.io:8687"
> bitbake core-image-minimal
> Signed-off-by: Jose Quaresma <quaresma.jose@...>
> meta/classes/sstate.bbclass | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
> diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
> index 0326d27c74..5e404d7cd8 100644
> --- a/meta/classes/sstate.bbclass
> +++ b/meta/classes/sstate.bbclass
> @@ -1005,7 +1005,7 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True,
> tasklist.append((tid, sstatefile))
> if tasklist:
> - nproc = min(int(d.getVar("BB_NUMBER_THREADS")), len(tasklist))
> + nproc = len(tasklist)
> progress = len(tasklist) >= 100
> if progress:
No, we're not doing that. That opens a ton (thousands) of parallel connections
to the servers and isn't fair on the systems hosting the data. In many cases
that would look like a DoS attack.
Got it and I have to agree that it will launch a DoS to the server.
However there are some cases around that have the mirror hosted locally on the file system.
I'll try to find a solution that doesn't affect the http mirror.
Join firstname.lastname@example.org to automatically receive all group messages.