<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
On Sat, 5 Aug 2017 12:16:04 -0400 gregor herrmann <<a href="mailto:gregoa@debian.org" class="">gregoa@debian.org</a>> wrote:<br class="">> What helps is:<br class="">> - replace in lib/HTML/HTML5/Parser.pm<br class="">><span class="Apple-converted-space"> </span> $response->{decoded_content} with $response->{content}<br class="">><span class="Apple-converted-space"> </span> which feels a bit dangerous<br class="">> - or in lib/HTML/HTML5/Parser/UA.pm's get:<br class="">><span class="Apple-converted-space"> </span> move the<br class="">><span class="Apple-converted-space"> </span> if ($uri =~ /^file:/i)<br class="">><span class="Apple-converted-space"> </span> up so it's the first alternative and then _get_fs is used<br class="">> <br class="">> <br class="">> The latter change would be, as a diff:<br class="">> <br class="">> #v+<br class="">> --- a/lib/HTML/HTML5/Parser/UA.pm<br class="">> +++ b/lib/HTML/HTML5/Parser/UA.pm<br class="">> @@ -18,14 +18,14 @@ sub get<br class="">><span class="Apple-converted-space"> </span> {<br class="">><span class="Apple-converted-space"> </span> my ($class, $uri, $ua) = @_;<br class="">> <br class="">> +<span class="Apple-converted-space"> </span> if ($uri =~ /^file:/i)<br class="">> +<span class="Apple-converted-space"> </span> { goto \&_get_fs }<br class="">><span class="Apple-converted-space"> </span> if (ref $ua and $ua->isa('HTTP::Tiny') and $uri =~ /^https?:/i)<br class="">><span class="Apple-converted-space"> </span> { goto \&_get_tiny }<br class="">><span class="Apple-converted-space"> </span> if (ref $ua and $ua->isa('LWP::UserAgent'))<br class="">><span class="Apple-converted-space"> </span> { goto \&_get_lwp }<br class="">><span class="Apple-converted-space"> </span> if (UNIVERSAL::can('LWP::UserAgent', 'can') and not $NO_LWP)<br class="">><span class="Apple-converted-space"> </span> { goto \&_get_lwp }<br class="">> -<span class="Apple-converted-space"> </span> if ($uri =~ /^file:/i)<br class="">> -<span class="Apple-converted-space"> </span> { goto \&_get_fs }<br class="">> <br class="">
<div class="">> <br class=""><div class="">> </div><div class="">> While this helps for reading local files, I guess the _get_lwp() case</div><div class="">> might still be buggy.</div><br class="Apple-interchange-newline"><br class=""></div><div class="">I also looked into this and found another possible fix:</div><div class=""><br class=""></div><div class=""><pre style="box-sizing: border-box; font-family: SFMono-Regular, Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 11.899999618530273px; margin-top: 0px; line-height: 1.45; word-wrap: normal; padding: 16px; overflow: auto; background-color: rgb(246, 248, 250); border-top-left-radius: 3px; border-top-right-radius: 3px; border-bottom-right-radius: 3px; border-bottom-left-radius: 3px; color: rgb(36, 41, 46); margin-bottom: 0px !important;" class=""><code style="box-sizing: border-box; font-family: SFMono-Regular, Consolas, 'Liberation Mono', Menlo, Courier, monospace; padding: 0px; margin: 0px; background-color: transparent; border-top-left-radius: 3px; border-top-right-radius: 3px; border-bottom-right-radius: 3px; border-bottom-left-radius: 3px; word-break: normal; border: 0px; display: inline; overflow: visible; line-height: inherit; word-wrap: normal;" class="">diff -ru HTML-HTML5-Parser-0.301/lib/HTML/HTML5/Parser.pm HTML-HTML5-Parser-0.301-patched/lib/HTML/HTML5/Parser.pm
--- HTML-HTML5-Parser-0.301/lib/HTML/HTML5/Parser.pm 2013-07-08 07:12:25.000000000 -0700
+++ HTML-HTML5-Parser-0.301-patched/lib/HTML/HTML5/Parser.pm 2017-08-06 12:42:58.000000000 -0700
@@ -13,6 +13,7 @@
use HTML::HTML5::Parser::TagSoupParser;
use Scalar::Util qw(blessed);
use URI::file;
+use Encode qw(encode_utf8);
use XML::LibXML;
BEGIN {
@@ -102,6 +103,11 @@
{
# XXX AGAIN DO THIS TO STOP ENORMOUS MEMORY LEAKS
my ($errh, $errors) = @{$self}{qw(error_handler errors)};
+
+ if (utf8::is_utf8($text)) {
+ $text = encode_utf8($text);
+ }
+
$self->{parser}->parse_byte_string(
$opts->{'encoding'}, $text, $dom,
sub {</code></pre><div class=""><br class=""></div></div><div class=""><br class=""></div><div class="">Part of the underlying issue here is that many variables and methods in these modules are named in a confusing way, expecting/requiring encoded bytes, but using names which imply a desire for decoded strings.</div><div class=""><br class=""></div><div class="">The above patch should handle the LWP case which the previously suggest patch avoids. It still passes the test suite (which should probably be improved to verify this case), and also supports the test case detailed in this bug report (though I should mention that I believe the test script included by Vincent Lefevre includes a double-encoding bug as $doc->toString() actually returns utf8 encoded bytes, which the :encoding(UTF-8) PerlIO layer on stdout will attempt to encode a second time).</div><div class=""><br class=""></div><div class="">thanks,</div><div class="">.greg</div><div class=""><br class=""></div></body></html>