[Pkg-opencl-devel] Bug#1065395: spirv-llvm-translator-14: autopkgtest on s390x uses huge amount of disk space

Paul Gevers elbrus at debian.org
Sun Mar 3 19:52:31 GMT 2024 

Source: spirv-llvm-translator-14
Version: 14.0.0-10
Severity: serious
X-Debbugs-CC: debian-ci at lists.debian.org
User: debian-ci at lists.debian.org
Usertags: issue

Dear maintainers,

Since a couple of days, our workers on s390x are dying because some test
is filling up all disk space. Several days ago, I wrongly suspected 
src:fenics-dolfinx (bug #1064995) and added it to our reject-list. It 
didn't solve the issue, so today I spend more time on finding the 
culprit. Basically every spike above 40% in the graph [1] is a moment 
that we see issues like:

Feb 28 05:38:18 ci-worker-s390x-01 debci[1738391]: gzip:
/tmp/debci-worker-43383540-cNnbLE372K/autopkgtest-incoming/testing/s390x/f/fenics-dolfinx/43383540/log.gz: 

No space left on device
Feb 28 05:38:18 ci-worker-s390x-01 debci[1424101]: E: Test for package
fenics-dolfinx produced no exit code, aborting

One of the suspects started to be spirv-llvm-translator-14, so I ran its 
autopkgtest manually, while logging disk use every 10 seconds (I started 
slightly delayed because I monitored the wrong partition first). As you 
can see below, during the test it grows from 17 GB (at the end) to its 
peak at 179 GB. That's not acceptable on our infrastructure. One file I 
happened to spot on the way was 
build/test/test_output/DebugInfo/Generic/Output/two-cus-from-same-file.ll.tmp:
-rw-r--r-- 1 root root  41G Mar  3 19:18 two-cus-from-same-file.ll.tmp

I have added spirv-llvm-translator-14 to our reject-list on s390x.

As this seems to be a rather new issue, I'm wondering if it's due to:
* Add build-needed autopkgtest for spirv-headers compat check.

Or maybe something in the toolchain that broke on s390x?

Paul

[1]
https://ci.debian.net/munin/ci-worker-s390x-01/ci-worker-s390x-01/df.html

/dev/mapper/3600507630affd250000000000000004a  196G   40G  146G  22% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G   49G  138G  27% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G   57G  130G  31% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G   65G  122G  35% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G   66G  121G  36% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G   67G  120G  36% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G   70G  117G  38% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G   73G  114G  40% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G   76G  111G  41% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G   79G  108G  43% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G   83G  104G  45% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G   85G  101G  46% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G   88G   98G  48% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G   92G   95G  50% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G   95G   92G  51% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G   98G   89G  53% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G  101G   86G  54% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G  104G   83G  56% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G  107G   80G  58% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G   65G  122G  35% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G   65G  122G  35% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G   66G  121G  36% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G   68G  118G  37% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G   72G  115G  39% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G   75G  112G  41% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G   78G  109G  42% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G   81G  106G  44% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G   85G  102G  46% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G   87G   99G  47% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G   90G   96G  49% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G   94G   93G  51% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G   97G   90G  52% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G  100G   87G  54% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G  103G   84G  56% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G  106G   81G  57% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G  109G   78G  59% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G  112G   74G  61% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G  116G   71G  63% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G  119G   68G  64% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G  123G   64G  66% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G  126G   61G  68% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G  130G   57G  70% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G  133G   54G  72% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G  136G   50G  74% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G  140G   47G  76% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G  143G   43G  77% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G  147G   40G  79% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G  109G   78G  59% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G  111G   75G  60% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G  112G   75G  60% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G  114G   73G  62% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G  117G   70G  63% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G  120G   67G  65% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G  123G   64G  67% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G  127G   60G  69% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G  131G   56G  71% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G  134G   53G  72% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G  137G   50G  74% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G  141G   46G  76% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G  147G   40G  79% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G  155G   32G  84% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G  163G   23G  88% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G  171G   15G  93% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G  179G  7.0G  97% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G   58G  128G  32% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G   59G  128G  32% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G   61G  126G  33% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G   63G  124G  34% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G   65G  121G  35% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G   68G  119G  37% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G   71G  116G  38% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G   73G  113G  40% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G   76G  111G  41% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G   79G  108G  43% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G   81G  105G  44% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G   84G  103G  46% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G   87G  100G  47% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G   90G   97G  48% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G   92G   95G  50% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G   95G   92G  51% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G   98G   89G  53% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G  100G   87G  54% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G  103G   84G  56% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G  106G   80G  58% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G  110G   77G  59% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G  113G   74G  61% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G  116G   70G  63% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G  119G   67G  64% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G  122G   64G  66% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G  126G   61G  68% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G  129G   58G  70% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G  133G   54G  72% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G  136G   51G  73% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G  139G   48G  75% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G   17G  169G  10% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G   17G  169G  10% 
/scratch
/dev/mapper/3600507630affd250000000000000004a  196G   17G  169G  10% 
/scratch
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: OpenPGP digital signature
URL: <http://alioth-lists.debian.net/pipermail/pkg-opencl-devel/attachments/20240303/0f5b31a5/attachment.sig>

More information about the Pkg-opencl-devel mailing list