[Pkg-javascript-devel] Rebuilding of nodejs reverse dependencies

[Jérémy Lal]

On Thu, Mar 17, 2022 at 3:10 PM Paolo Greppi <paolo.greppi at libpf.com> wrote:

> Hi Jérémy,
>
> Il 17/03/22 13:26, Jérémy Lal ha scritto:
> > ...
> >
> >     So I ended up installing a gitlab-runner with nspawn-runner and a
> >     semi-active server I maintain.
> >     I documented the process here:
> >     https://wiki.debian.org/Salsa/Doc/CustomRunners/SystemdNspawnRunner
> >     <https://wiki.debian.org/Salsa/Doc/CustomRunners/SystemdNspawnRunner
> >
> >
> >     salsa's driving that runner to rebuild the thousands of packages now.
> >     Minimal build time is 1 minute, so it should take about two days to
> >     rebuild the whole.
> >
> >     After that, anyone knowing gitlab API and bugs.debian.org
> >     <http://bugs.debian.org> API could help to open FTBFS bugs ?
> >
> > I started this:
> > https://salsa.debian.org/kapouer/dance
> > <https://salsa.debian.org/kapouer/dance>
> >
> > to fetch logs of failed jobs, and also to be able to retry jobs that
> > failed because of external reasons
> > (runner configuration issue, for example).
> > When all jobs are finished, i'll start to use "dance" to send bug
> reports.
> >
> > Jérémy
>
> this is all super interesting. In the past I have done quite a few mass
> rebuilds of reverse dependencies using ratt
> (
> https://salsa.debian.org/debian/doxygen/-/wikis/ratt_doxygen_1.9.1-1_amd64%20partial%202
> )
>
> Your approach is cool because it can be easier to automate.
>
> To look at pipeline results programmatically I would have used this
> Python library:
> - https://packages.debian.org/bullseye/python3-gitlab
> -
>
> https://python-gitlab.readthedocs.io/en/stable/gl_objects/pipelines_and_jobs.html
>
> But I see you plan to use JS for "dance".
>

I'm just gluing existing solutions together, in js because i don't do
python.

Do you intend to trigger the dance from within the pipeline ? You'd then
> have a recursive pipeline ... who knows if it will finish in finite time
> ;-)

Anyway from my experience:
>

I don't plan to use this as CI, and instead it will only be manually
triggered.
(unfortunately I can't leave that runner open for all, and it would be great
if some sponsor could dedicate a group of runners for building reverse deps
using systemd-nspawn which is kind of simple and fast).

1. some rdeps reproducibly fail all the time, these should be excluded
>

Since it's not really Continuous, I don't mind some systematic failures (at
least for now).

2. ratt rebuilds failed rdeps against the previous version of the parent
> package (this filters out false positives), does your job definition do
> that as well ? (too lazy to check)
>

I suppose so, dance is not the one responsible for the schedule of the
rebuilds:
right now the list is generated here and added to its repository:
https://salsa.debian.org/js-team/nodejs/-/blob/master-14.x/debian/salsa-gen-transition-jobs
(with some ci variables to avoid triggering the pipeline by accident)

I plan to move that script (from Bernd Zeimetz) to "dance".

3. some rdeps take much longer to build than others, I ended up creating
> lists of "fast" and "slow" rdeps; if you only include "fast" ones your
> mass rebuild can run in hours instead of days, but may still be able to
> catch regressions
>

Totally ! I excluded
chromium firefox firefox-esr thunderbird nodejs ffmpeg emscripten
because they're just too slow to rebuild. Maybe I left some other slow
builds.
The quick builds, on the other hand, are very fast (~1 minute).

4. I used to create spreadsheets to compare build times and which rdep
> failed; for max fun it would be nice to have a database (!) where to
> store (stamp, parent_package, old_version, new_version, rdep, status,
> duration)


That kind of information is already somewhere in debian, but how to get it ?

Jérémy
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/pkg-javascript-devel/attachments/20220317/c46c571e/attachment.htm>