Bug#762638: metaconfig source control/distribution and Debian's DFSG
H.Merijn Brand
h.m.brand at xs4all.nl
Sat Oct 4 18:23:10 UTC 2014
On Sat, 4 Oct 2014 17:50:51 +0100, Dominic Hargreaves <dom at earth.li>
wrote:
> Hi,
>
> I'm including Helmut, the submitter to the Debian bug report, as a
> CC, in case he can help provide additional context that I'm missing.
>
> On Thu, Sep 25, 2014 at 09:07:59PM -0400, Andy Dougherty wrote:
> > On Thu, Sep 25, 2014 at 10:16:32PM +0100, Dominic Hargreaves wrote:
> > > Hi,
> > >
> > > The Debian project has recently received a bug report indicating that
> > > the perl source package we distribute does not comply the the DFSG[1]
> > > since the preferred form of modification of Configure is not provided.
> > > You can see the bug report at [2].
> >
> > I'd quibble with that -- though I am no longer the primary Configure
> > maintainer, I actually did prefer most users to directly modify Configure.
> > It ended up being much easier than supporting users using metaconfig to
> > generate Configure. Users are of course welcome to fetch and install
> > metaconfig and use it, but I wouldn't say that was the "preferred"
> > form.
>
> I don't have the exact details, but Helmut suggests that this wasn't
> the case for him, as the changes would have had to be made in several
> places. In general I think a healthy free software project should aim to
> minimise cases where a casual hacker would be discouraged from doing
> things the same way a seasoned developer would.
>
> > Generating Configure requires both the 'dist' package (which contains
> > metaconfig) and the perl-specific metaconfig units. Historically, we have
> > often ended up using a patched 'dist', or a version that lagged behind
> > the main package. We definitely can't simply assume the version in the
> > Debian archive will work. Generating Configure also sometimes involves
> > patching or fixing the output of metaconfig in various ways. All the
> > information to do this should be in perl's metaconfig git repository.
> >
> > It is also sometimes difficult to check or merge the generated output.
> > The metaconfig units are assembled in order based on dependencies,
> > but those dependencies don't fully constrain the order. The result is
> > that the order of units can vary depending on who generates Configure.
> > Thus even a tiny patch to a single metaconfig unit can result in a
> > Configure patch with hundreds of lines that are just shuffling
> > parts around.
> >
> > That's another reason why modifying the metaconfig units is not
> > necessarily the "preferred" form of modification of Configure.
> >
> > Obviously, it's all a bit involved, and a fair amount of effort to
> > figure out.
> >
> > The way it's set up now, we encourage people to simply patch Configure.
> > If someone wants to go the metaconfig route instead, it's a lot of
> > extra work, but presumably the person deliberately chooses that route
> > knowing it's extra work.
>
> Again, there is an ideological point here. It *shouldn't* be a lot of
> extra work to do things in the way that upstream developers would.
> Clearly perl isn't going to be kicked out of Debian because of this,
> but a less important package might well be.
>
> > If we include all the units in the source tree
> > instead, I think end users will have an expectation of greater support.
> > They will expect it to work, and work easily, right out of the box.
> > Getting it to that point will require significant effort, and I really
> > don't see the benefit. Of course, if someone else wants to do it and
> > support it, that's fine with me.
>
> One benefit would be that we wouldn't this acknowledged bus factor
> issue with the metaconfig wrangling.
Disagree. Areas in the perl core are picked up by people that like that
area:
• Threading
• 64bit/32bit
• Regex
• COW
• Speed
• Debugging
• Toolchain
• Configuration
• …
It just happens that Configuration is NOT an area that appeals to
developers. It works, and they are happy it works the way it does.
Improving the configuration or the configuration process does NOT make
the language itself more appealing or more interesting, and eventually,
that is what the developers want (to achieve).
Yes, the bus-factor is (too) low, but it is not caused by
(un)availability of the pieces
> > > If we have to, we can make a tarball of that repository and include
> > > it with the perl source package, but it seems like it would we should
> > > explore the possibility of fixing this discrepancy upstream, rather
> > > than working around it in Debian. In fact, it's likely we'd also have
> > > to supply the patches between the current metaconfig output and what's
> > > actually in the perl release tarball, since Configure is explicitly
> > > allowed to be patched even though it's generated.
> >
> > To be complete, you would have to include both perl's metaconfig
> > repository and an appropriately patched 'dist' source tree, which would
> > likely differ somewhat from Debian's existing 'dist' tree. If the
> > intent is to actually use it, there are probably a bunch of installation
> > issues to work out. If you want to check your results, you'll have to
> > confront the reshuffling problem.
>
> > Short summary: I think it would be a lot of work for very little
> > gain.
>
> On Fri, Sep 26, 2014 at 08:34:58AM +0200, H.Merijn Brand wrote:
>
> > The main problem with generating Configure from meta is not the
> > availability of the metaunits used for perl, but the way meta/dist has
> > to be installed to make them work the way it does.
>
> Do you mean the fact that dist is being patched, or some other reason?
Just like learning a new language, the whole *process* is rather
complicated. It is not like most GNU tools "sh autogen.sh",
"configure", "make check", "make install", but - for people building
perl - "Configure", "make test", "make install"
The source code for autogen is widely available, but it works fine only
on Linux. (I'd invite you to the hell of using the auto* and (even
worse) libtool on OS's like AIX and HP-UX.
To make the developers' life of our Perl community *less* of a hell, we
are shipping "Configure", which is a autoconf/autogen/configure in one,
that runs on almost every *nix like platform (VMS and Windows have
special cased support).
For how many of the GNU tools that you ship, do you include the
autogen.sh stuff that is only used *before* the configure is included
in the source package? I only see the auto-stuff if I clone the source
repo for the tool itself.
The fact that we patched and extended meta is NO reason for it to be
more complex than using meta/dist from scratch. It is like installing a
patched version of autoconf. The changes are reasonable well
documented, and the source is widely and openly available.
> > As Andy already stated, it proves to be much easier to "backport" the
> > direct patches to Configure and friends to meta than it is to instruct
> > (new) users to patch metaunits the way it should. Even seasoned users
> > seem to have problems doing so. I had two sessions past year explaining
> > the ways and woes of how it works, sincerely hoping to get fresh blood
> > into the configuration process. The bus factor currently is about 1½
> >
> > If I stop maintaining perl, only Andy understands the *complete*
> > process right now, and Jarkko would be able to (reluctantly) step in.
> > (Andy still understands some of the steps way better than I do). I did
> > try to improve the docs, and I also made the meta/dist *as I use it*
> > available on my CPAN, but it still is so much different a process from
> > the rest of perl5 core development that it does not attrackt (new)
> > people to join in: it is not a focus area that attracks youngsters, as
> > NO END USER will see what your work involves, while when you improve
> > the regular expression engine, the whole of the perl user community
> > will notice.
>
> As noted above, these sound like excellent reasons to make it so that
> the process is understood by the developer community at large rather
> than just a handful of individuals. I know it's easier to say that
> rather than to do it (particularly from someone who doesn't yet understand
> all the complexities).
I have tried to talk about the process, both to individuals (who asked
to have it explained) and to groups, e.g. on workshops or conferences.
The size of the audience however proved to always be extremely small:
people do not seem to be interested in the pre-configuration steps and
I completely understand: it is not a rewarding job.
> > As the current maint for this process, I communicate with the
> > maintainer of dist/meta on a rather regular basis. I give feedback of
> > what we changed, and I look at the changes he makes to see if those
> > will be usable for perl.
> >
> > As part of the metaunits is still used from dist, part is a set of
> > (slightly) modified units from dist, and part is the units written only
> > for perl, keeping those in sync is a hell of a job. On simply cannot
> > stay in sync 100%
>
> Okay, so clearly from a pragmatic view we would need to ship our own version
> of dist along with the rest of the stuff from the metaconfig repository.
Done. It is now in our public git repo.
> Depressingly this violates another part of Debian policy relating to
> embedding copies of code not intended to be embedded, but the freedom
> to modify the code using the preferred form clearly overrides that in
> my mind.
That is not my area of expertise. sorry
> > > It is also sometimes difficult to check or merge the generated output.
> > > The metaconfig units are assembled in order based on dependencies,
> > > but those dependencies don't fully constrain the order. The result is
> > > that the order of units can vary depending on who generates Configure.
> >
> > Even if you rsync from one system to another and use exactly the same
> > tools, the order can change based on locale setting :(
> >
> > > Thus even a tiny patch to a single metaconfig unit can result in a
> > > Configure patch with hundreds of lines that are just shuffling
> > > parts around.
> > >
> > > That's another reason why modifying the metaconfig units is not
> > > necessarily the "preferred" form of modification of Configure.
> >
> > If possible, modify the hint files instead.
>
> So, one data point is whether Helmut's changes could in fact have
> been achieved this way instead.
Yes
> > > Obviously, it's all a bit involved, and a fair amount of effort to
> > > figure out.
> > >
> > > The way it's set up now, we encourage people to simply patch Configure.
> > > If someone wants to go the metaconfig route instead, it's a lot of
> > > extra work, but presumably the person deliberately chooses that route
> > > knowing it's extra work. If we include all the units in the source tree
> > > instead, I think end users will have an expectation of greater support.
> >
> > One extra note here: the generation process decides what units to
> > include based on the references used in the perl source tree. This
> > means that not all units will be used. Including all units in a
> > distribution might lead to extra confusion.
>
> Could you expand on this a bit more? It's not obvious to me what
> this means. Aside from the 'promise of support' issue, this seems
> to be the main reason against moving the units into the main perl source,
> which would itself be a huge step forward.
There are units that check for features that none of the perl core
parts touch or use. The generator looks at the perl source code to see
which defines, functions, symbols, and tests are used, and includes the
units that provide those tests in to generated include:
Locating units...
Extracting dependency lists from 877 units...
Extracting filenames (C and SH files) from MANIFEST.new...
Building a Wanted file...
Scanning .[chyl] .xs files for symbols...
Scanning .SH files for symbols...
*** Obsolete symbols found -- see file 'Obsolete' for a list.
Computing optimal dependency graph...
Building private make file...
Determining loadable units...
make: Circular perllibs <- perllibs dependency dropped.
Updating make file...
Determining the correct order for the units...
make: Circular perllibs <- perllibs dependency dropped.
Creating Configure...
Check if HAS_GETGROUPS (26) precedes Groups_t (97)
Check if BYTEORDER (135) precedes UVSIZE (487)
Check if LONGSIZE (300) precedes BYTEORDER (135)
Check if HAS_SETGROUPS (66) precedes Groups_t (97)
Check if HAS_QUAD (314) precedes I64TYPE (487)
Check if MULTIARCH (306) precedes BYTEORDER (136)
Check if MULTIARCH (136) precedes MEM_ALIGNBYTES (131)
Check if HAS_GETGROUPS (26) precedes Groups_t (97)
Check if BYTEORDER (137) precedes UVSIZE (487)
Check if LONGSIZE (136) precedes BYTEORDER (137)
Check if HAS_SETGROUPS (66) precedes Groups_t (97)
Check if HAS_QUAD (314) precedes I64TYPE (487)
Check if MULTIARCH (131) precedes BYTEORDER (137)
Check if MULTIARCH (131) precedes MEM_ALIGNBYTES (132)
Done.
> > > Short summary: I think it would be a lot of work for very little
> > > gain.
> >
> > 100% agree
> >
> > I'd rather put my free time in trying to get closer to dist-4.0 than we
> > are now (3.5+) than modifying the current process to enable
> > distribution of something that would work for all users that are
> > unlikely to use it anyway
>
> I'm genuinely confused here. Isn't making the perl build infrastructure
> less dependent on a small number of people a significant gain, even if
> you don't care about the ideological importance of the 'preferred form
> of modification' requirement from Debian?
Now *I* am confused. I do not understand the question.
I volunteered to do this work, and I have so far no reason to quit.
I don't see the maintenance of Configure from the units as a strain or
a burden. Only *IF* other/more people are *seriously* interested in
sharing in this process, modifying the process itself would be worth
considering. Only modifying the process to enable people that will not
show help in this area anyway is a waste of time.
> On Mon, Sep 29, 2014 at 01:33:41PM -0400, Andy Dougherty wrote:
> >
> > One final follow-up: Debian is certainly welcome to distribute the
> > files used to build Configure. The following two items would be needed:
> >
> > 1. The precise version of dist used. This can be obtained here:
> >
> > http://www.cpan.org/modules/by-authors/id/H/HM/HMBRAND/perl-meta-3.5-20.tgz
>
> Incidentally, this should be mentioned in the comments at the top of
> Configure, rather than the misleading reference to the 'normal' version
> of dist.
It is now obsolete. I still have to modify the docs to explain the new
state. I had no time yet.
> > 2. The custom metaconfig units used. These are available here:
> >
> > git://perl5.git.perl.org/metaconfig.git
> >
> > Rebuilding Configure would not be easy or automatic, but all the necessary
> > files would be available. Would that satisfy the Debian guidelines?
>
> I think it would satisfy the letter of the law, if not the full spirit.
> It's a starting point; I'd like it to not be the ending point, but
> I guess I would need to convince you and/or H.Merijn Brand of this in
> order to make significant progress.
>
> Having to pull in extra sources is a bit of a pain for the Debian package
> management, but it's doable. The main thing that seems to be missing is
> a way of knowing what version of that repository is to be used (or was
> used) for a given release of perl. That doesn't seem to be recorded
> in either the metaconfig repository or the perl repository.
The units and the source are now (in sync) in the same repo. After the
docs are changed, it should be clear(er) how to use the bunch
> The other option open to us is to convince Debian that the metaconfig
> units genuinely are external bits of tooling that aren't part of the
> build process, and that Configure really is the preferred form of
> modification. Working through Helmut's use case might help there.
That is still *my* preference!
> Thank you both for taking the time to work through some of these
> issues with me/Debian (and I see that things are already better in that
> the version of dist used is now readily available).
>
> Cheers,
> Dominic.
--
H.Merijn Brand http://tux.nl Perl Monger http://amsterdam.pm.org/
using perl5.00307 .. 5.19 porting perl5 on HP-UX, AIX, and openSUSE
http://mirrors.develooper.com/hpux/ http://www.test-smoke.org/
http://qa.perl.org http://www.goldmark.org/jeff/stupid-disclaimers/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 490 bytes
Desc: OpenPGP digital signature
URL: <http://lists.alioth.debian.org/pipermail/perl-maintainers/attachments/20141004/1d88246e/attachment-0001.sig>
More information about the Perl-maintainers
mailing list