Bug#569191: libwww-perl: GET behavior changed in squeeze: URL stopped working

[Damyan Ivanov]

[debian-www, please CC the bugreport on replies; thanks]

-=| Andreas B. Mundt, Wed, Feb 10, 2010 at 06:16:03PM +0100 |=-
> Package: libwww-perl
> Version: 5.834-1
> Severity: important
> 
> we use GET to download a wikipage and further process the data to
> prepare the manual of Debian Edu. The command:
> 	GET "http://wiki.debian.org/DebianEdu/Documentation/Lenny/AllInOne?action=raw"
> works fine in Lenny, but stopped working in squeeze where "You are not
> allowed to access this!" is returned. If you remove "?action=raw" from
> the URL anything is fine. Is this inteded and we have to provide a
> header?

This is very strange. There is a slight difference in the User-Agent 
request header (as reported by tcpdump -s 1500 -X):

On Lenny (works)
================
User-Agent: lwp-request/0.810

On Sid (breaks)
===============
User-Agent: lwp-request/5.834 libwww-perl/5.834


So it seems there is some filter on wiki.debian.org that refuses some 
user agents.

HTTP/1.1 (section 14.43) explicitly allows multiple entries in this 
header:

    The field can contain multiple product tokens (section 3.8) and 
    comments identifying the agent and any subproducts which form 
    a significant part of the user agent

Cc-ing debian-www for comment.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: Digital signature
URL: <http://lists.alioth.debian.org/pipermail/pkg-perl-maintainers/attachments/20100210/a3f15bae/attachment.pgp>