Bug#834961: 'perl Makefile.PL' sv_setpvn panic issues

Mon Aug 29 20:44:02 UTC 2016

clone 834961 -2 -3 -4
retitle -2 perl: EU::MM outputs UTF8 strings on an ASCII filehandle in non-UTF8 locales
reassign -2 perl 5.22.2-3
severity -2 normal
submitter -2 !
forwarded -2 https://rt.cpan.org/Public/Bug/Display.html?id=106461
retitle -3 libencode-perl: panic: sv_setpvn called with negative strlen
reassign -3 libencode-perl 2.84-2
forwarded -3 https://rt.cpan.org/Public/Bug/Display.html?id=65541
submitter -3 !
severity -3 important
retitle -4 perl: (Encode) panic: sv_setpvn called with negative strlen
reassign -4 perl 5.22.2-3
forwarded -4 https://rt.cpan.org/Public/Bug/Display.html?id=65541
block -4 with -3
severity -4 important
submitter -4 !
thanks

My current understanding of this bug is:

- ExtUtils::MakeMaker has a problem in non-UTF8 locales where it can
  write UTF8 metadata from for instance META.yml into Makefile through an
  ASCII-only file handle, causing the ""\x{00c2}" does not map to ascii"
  warnings. This is [rt.cpan.org #106461]; cloning a separate bug for it.

- Encode has an easily reproducible problem when coercing multibyte
  characters into a single byte file handle (or something to that
  effect). The IO layer buffering can get flushed in the middle of a
  multibyte character, corrupting the output. A test case is
    perl -e 'binmode(STDOUT, ":encoding(ascii)"); print(("A"x shift) . "ä\n")' 1023
  which outputs \x{fffd} when it shouldn't.

- More or less related to the above, Encode can also end up using
  uninitialized memory in similar circumstances, sometimes ending up
  with 'panic: sv_setpvn called with negative strlen'. This happens
  occasionally on current sid when building libvitacilina-perl. I'm
  appending valgrind output and a gdb stack trace. This together with
  the previous issue is [rt.cpan.org #65541], so I'm keeping them in
  one report for now (but cloning a separate one against perl which will
  only be updated after the separate package.)

The release critical part of this is probably easiest solved by either
building everything with UTF-8 metadata in a UTF-8 locale, or fixing the
ExtUtils::MakeMaker issue. I'm not sure how many packages are affected.

The upstream tickets need to be updated; I'll do that tomorrow or
so unless somebody wants to beat me to it.

Valgrind output with Encode-2.86 without optimization follows, fully
reproducible on current sid (perl 5.22.2-3), where it is sensitive to
these exact command line arguments.

% LC_ALL=C valgrind perl -I. Makefile.PL INSTALLDIRS=vendor create_packlist=0
==18958== Memcheck, a memory error detector
==18958== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==18958== Using Valgrind-3.12.0.SVN and LibVEX; rerun with -h for copyright info
==18958== Command: perl -I. Makefile.PL INSTALLDIRS=vendor create_packlist=0
==18958== 
'CREATE_PACKLIST' is not a known MakeMaker parameter name.
Generating a Unix-style Makefile
Writing Makefile for Vitacilina
"\x{00c2}" does not map to ascii at /usr/share/perl/5.22/ExtUtils/MakeMaker.pm line 1187.
"\x{00a1}" does not map to ascii at /usr/share/perl/5.22/ExtUtils/MakeMaker.pm line 1187.
"\x{00c3}" does not map to ascii at /usr/share/perl/5.22/ExtUtils/MakeMaker.pm line 1187.
"\x{00a9}" does not map to ascii at /usr/share/perl/5.22/ExtUtils/MakeMaker.pm line 1187.
"\x{00c2}" does not map to ascii at /usr/share/perl/5.22/ExtUtils/MakeMaker.pm line 1187.
"\x{00a1}" does not map to ascii at /usr/share/perl/5.22/ExtUtils/MakeMaker.pm line 1187.
==18958== Conditional jump or move depends on uninitialised value(s)
==18958==    at 0x52E2D2: Perl_utf8n_to_uvchr (utf8.c:588)
==18958==    by 0x755237B: encode_method (Encode.xs:193)
==18958==    by 0x7554C42: XS_Encode__XS_encode (Encode.xs:763)
==18958==    by 0x4BDAF9: Perl_pp_entersub (pp_hot.c:3272)
==18958==    by 0x4B69A5: Perl_runops_standard (run.c:41)
==18958==    by 0x43C212: Perl_call_sv (perl.c:2764)
==18958==    by 0x8F96E36: PerlIOEncode_flush (encoding.xs:425)
==18958==    by 0x53B74E: Perl_PerlIO_flush (perlio.c:1630)
==18958==    by 0x53C31B: PerlIOBuf_write (perlio.c:4165)
==18958==    by 0x518364: Perl_do_print (doio.c:1386)
==18958==    by 0x4B8815: Perl_pp_print (pp_hot.c:856)
==18958==    by 0x4B69A5: Perl_runops_standard (run.c:41)
==18958== 
"\x{fffd}" does not map to ascii at /usr/share/perl/5.22/ExtUtils/MakeMaker.pm line 1187.
"\x{fffd}" does not map to ascii at /usr/share/perl/5.22/ExtUtils/MakeMaker.pm line 1187.
"\x{00a9}" does not map to ascii at /usr/share/perl/5.22/ExtUtils/MakeMaker.pm line 1187.
Writing MYMETA.yml and MYMETA.json
==18958== 
==18958== HEAP SUMMARY:
==18958==     in use at exit: 14,378,715 bytes in 54,070 blocks
==18958==   total heap usage: 179,891 allocs, 125,821 frees, 32,136,429 bytes allocated
==18958== 
==18958== LEAK SUMMARY:
==18958==    definitely lost: 69,536 bytes in 29 blocks
==18958==    indirectly lost: 14,244,115 bytes in 53,915 blocks
==18958==      possibly lost: 51,952 bytes in 89 blocks
==18958==    still reachable: 13,112 bytes in 37 blocks
==18958==                       of which reachable via heuristic:
==18958==                         newarray           : 280 bytes in 8 blocks
==18958==         suppressed: 0 bytes in 0 blocks
==18958== Rerun with --leak-check=full to see details of leaked memory
==18958== 
==18958== For counts of detected and suppressed errors, rerun with: -v
==18958== Use --track-origins=yes to see where uninitialised values come from
==18958== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)

GDB stack trace (LC_ALL=C again) with a conditional breakpoint in
Perl_sv_setpvn(), triggered when len==-1 (so just before the panic). This
is not fully reproducible and usually needs a few runs inside the same
GDB session before triggering. As above, it is currently sensitive to
these command arguments.

Starting program: /usr/bin/perl -I. Makefile.PL INSTALLDIRS=vendor create_packlist=0
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
'CREATE_PACKLIST' is not a known MakeMaker parameter name.
Generating a Unix-style Makefile
Writing Makefile for Vitacilina
"\x{00c2}" does not map to ascii at /usr/share/perl/5.22/ExtUtils/MakeMaker.pm line 1187.
"\x{00a1}" does not map to ascii at /usr/share/perl/5.22/ExtUtils/MakeMaker.pm line 1187.
"\x{00c3}" does not map to ascii at /usr/share/perl/5.22/ExtUtils/MakeMaker.pm line 1187.
"\x{00a9}" does not map to ascii at /usr/share/perl/5.22/ExtUtils/MakeMaker.pm line 1187.
"\x{00c2}" does not map to ascii at /usr/share/perl/5.22/ExtUtils/MakeMaker.pm line 1187.
"\x{00a1}" does not map to ascii at /usr/share/perl/5.22/ExtUtils/MakeMaker.pm line 1187.
"\x{00e0}" does not map to ascii at /usr/share/perl/5.22/ExtUtils/MakeMaker.pm line 1187.

Breakpoint 1, Perl_sv_setpvn (my_perl=0x7d3010, sv=0x122d288, ptr=0x12d5731 "\247.\001", 
    len=18446744073709551615) at sv.c:4905
4905    in sv.c
(gdb) bt
#0  Perl_sv_setpvn (my_perl=0x7d3010, sv=0x122d288, ptr=0x12d5731 "\247.\001", len=18446744073709551615)
    at sv.c:4905
#1  0x00007ffff68b8905 in encode_method (my_perl=0x7d3010, enc=0x7ffff6ac0aa0 <ascii_encoding>, 
    dir=0x7ffff6ac02c0 <utf8_ascii>, src=0x122d288, check=2306, offset=0x0, term=0x0, retcode=0x0, 
    fallback_cb=0x7d3140) at Encode.xs:279
#2  0x00007ffff68bac43 in XS_Encode__XS_encode (my_perl=0x7d3010, cv=0xb2c3b0) at Encode.xs:763
#3  0x00000000004bdafa in Perl_pp_entersub (my_perl=0x7d3010) at pp_hot.c:3272
#4  0x00000000004b69a6 in Perl_runops_standard (my_perl=0x7d3010) at run.c:41
#5  0x000000000043c213 in Perl_call_sv (my_perl=0x7d3010, sv=0x12241b8, flags=<optimized out>) at perl.c:2764
#6  0x00007ffff5e7ae37 in PerlIOEncode_flush ()
   from /usr/lib/x86_64-linux-gnu/perl/5.22/auto/PerlIO/encoding/encoding.so
#7  0x000000000053b74f in Perl_PerlIO_flush (my_perl=0x7d3010, f=<optimized out>) at perlio.c:1630
#8  0x000000000053c31c in PerlIOBuf_write (my_perl=0x7d3010, f=0x7ecc70, vbuf=<optimized out>, count=1272)
    at perlio.c:4165
#9  0x0000000000518365 in Perl_do_print (my_perl=my_perl at entry=0x7d3010, sv=<optimized out>, 
    fp=fp at entry=0x7ecc70) at doio.c:1386
#10 0x00000000004b8816 in Perl_pp_print (my_perl=0x7d3010) at pp_hot.c:856
#11 0x00000000004b69a6 in Perl_runops_standard (my_perl=0x7d3010) at run.c:41
#12 0x000000000044409d in S_run_body (oldscope=<optimized out>, my_perl=<optimized out>) at perl.c:2453
#13 perl_run (my_perl=0x7d3010) at perl.c:2381
#14 0x000000000041cb6b in main (argc=5, argv=0x7fffffffe8d8, env=0x7fffffffe908) at perlmain.c:116

-- 
Niko Tyni   ntyni at debian.org