[Pkg-crosswire-devel] Fwd: Module packaging, distribution, and management

[Matthew Talbert]

>Regarding the write-protected filesystems issue; surely this is at heart
>a design discussion for the sword library development team (if front
>ends use their install code); if the front ends do their own thing
>regarding module installers, then it is possible for any one such team
>to develop an appropriate specification, design docs, and then implement
>the improved installer... right?

>As I see it, this is *not* a packaging problem at all, and I confess
>that it somewhat irks me that Peter (and to some extent you) seem(s) to
>treat it as one, and lay it at the feet of a brand new and already busy
>packaging team.

No, I have never suggested that this is a packaging problem, except
with regards to the already acknowledged issues with bad module
dependencies.

>Maybe this will help you see this better: today, I can

>(A) Do a totally from source tarball install of sword-1.5.11, then read
>its documentation and see in the INSTALL file that /usr/share/sword is
>the official default place for shared modules, and I follow its
>directions on how to add a module, and so ...

>(B) At a shell prompt, I type:

 >wget http://crosswire.org/ftpmirror/pub/sword/packages/rawzip/KJV.zip
 >sudo unzip KJV.zip -d /usr/share/sword

>Now diatheke works just fine with this, I am using the recommended
>location and approach documented by the INSTALL file in the sword-1.5.11
>tarball.

>(C) Now I add to my system (again from its source tarball, no packaging
>involved still!) GnomeSword.

>And now, the GnomeSword module manager has (I am told) issues managing
>my installed KJV module.

Of course you have to "sudo" to get the module to unpack there, so
running GnomeSword with sudo will manage the modules with no issues.
Now we could, and probably should, enable the module manager to ask
for sudo permissions only when absolutely necessary rather than
forcing the user to run the entire program with sudo.

>How on *earth* is that a problem caused by apt-get or by any form of
>packaging whatsoever?!

>Do you understand?  You are apparently telling us packagers that a
>problem of your own making (module managers that don't use apt and don't
>know how to handle shared central module data well) is a packaging issue
>- and it just is not so.  It cannot possibly be so, since the problem
>can be fully demonstrated without using packaged copies of any part of
>the whole setup, and following Sword-supplied instructions on how to add
>modules.

>Blaming packagers or packaging systems for this, or asking them to fix
>it or work around it, when manifestly no packaging systems are involved
>in creating the problem, is simply not a logical response to this issue,
>in my view.  Illogical responses to issues are usually rather
>counter-productive, and do not generally lead to resolution of the
>issues concerned.

>I hope I have made my point!

>I'm quite willing to work with your developers on an improved module
>manager specification; I might even code some of it, if such a spec gets
>accepted, though C++ is not my preferred language at the moment.  But
>blaming packaging for this issue just drives me away.

We at GnomeSword would be particularly grateful for such help, because
apparently no one has known in the past how to elevate permissions in
the module manager. If you have experience in this, that would be
great. Note that we use primarily C with enough C++ to communicate
with Sword.

>>> It is not data like a game episode (a la Wesnoth).

>> libsword itself is perfectly usable without one of the modules.

>Yes.  In the same way, a desktop PC is perfectly usable without a screen
>or keyboard or mouse (you can use it via the serial port or USB port, or
>as a headless server across a network.  But very few PCs are used that way!

>> For example, you could use it for research purposes with modules you
>> build entirely yourself (and some do use it this way). So, it would
>> be more reasonable to recommend a Bible for the frontends rather than
>> the library, in my opinion.

>Yes, OK, I'm open to that.  Don't forget that both Suggests: and
>Recommends: are not enforced (though some package managers now default
>to installing all Recommends: items).  In reality, I think installing
>the sword library (only) could only really make sense if it included
>enough documentation to allow a research user to create modules from
>scratch; I don't think it currently does that, so in practice such a
>user will want at least one existing module to use as an example, I
>submit.  There is a reason that the INSTALL file for sword-1.5.11
>includes instructions on downloading and installing a module, after all
>-- doing so is apparently considered a part of the installation process.
> Those instructions even say:

Someone else proposed that Recommends be changes to Suggests, and I
second that. More to the point here, it is entirely possible that the
theoretical user of the sword lib will already have his own modules
and therefore not need any additional. Other libraries (and even other
tools, as has been pointed out several times) do not require that a
"sample" file exist, or even suggest such a file. At any rate, it's a
rather fine point, but my preference would still be to have the
Recommends or Suggests against the front-ends rather than the
libraries.

>  To be useful the software needs to find SWORD 'modules' installed
>  somewhere accessible.

> followed by:

>   In a default SWORD configuration, the module install process may
>   look like this:

>   [download a module with your favourite client]
 >  su
>   cd /usr/share/sword
>   unzip ~/KJV.zip

>So *that* is the default recommended way to install modules.  Per your
>own documentation.

Just to be clear, it is not *my* documentation. I am only a consumer
of the swordlib, not a developer.

>>> It apparently lacks fundamental functionality for handling per-system
>>> (or per set of systems on a LAN) module installations, as far as I can
>>> tell.  It just seems to naively assume that it can write to every
>>> location on every filesystem that it can see is a module location, and
>>> then gets upset or does not work as expected whenever that assumption is
>>> invalid.  This is a pretty serious bug, IMO, especially since one such
>>> system-shared location is clearly recommended by the documentation in
>>> the Sword tarball.

GnomeSword will not allow installation to a location it sees as
unwritable. It doesn't assume it can write everywhere, it checks
first. So it can easily read the modules installed in a system
location, and install modules in that location if it is writable, or
in ~/.sword. The only issues arise when trying to create indexes
(which could be solved by distributing them), and trying to delete
them (honestly I'm not sure what happens here, but I think that Sword
tries and fails silently). So, as the only two issues we have are
indexing and deleting, indexing could be solved by distributing them,
and deleting can be done via apt-get. I know of no other issues.

>> 1. Would it be possible to distribute the lucene indexes as part of
>> the package? This is an area of potential issues, because if modules
>> are in write-protected areas then GnomeSword will not be able to
>> create an index without being run as root. In my opinion, the indexes
>> are a valid part of the data (for some reason, the official Sword
>> position does not agree, though I've never heard a good explanation).

>That sounds almost reasonable to me.  Perhaps better would be for the
>packages to run a post-install script that builds those indexes, so we
>aren't transporting the (in some sense redundant, becase it is readily
>computable from the base modules) index data around on limited bandwidth
>links etc.  That is very doable, as long as there is a command-line tool
>which will build them.  We'd package that separately from the libraries
>(so libs of different versions can coexist, per my other thread), I'd
>think, and the modules would depend on it so they could be sure it was
>available for their postinstall script to use.

The index data is for general books around 100KB, for dictionaries
around 1MB, and for Bibles between 3MB and 7MB. I don't have a
preference whether it is distributed or done post-install, but I don't
think that bandwidth is a huge concern in this case.

>So could you write us a simple C or C++ command line program that is the
>exact equivalent of mkfastmod but which uses glib and so gets this
>right?  Or else persuade the Sword library developers to create such a
>tool?  mkturbomod, mkreallyfastmod, or mkfastandcorrectmod, perhaps???
>Actually, a name more like mkswordindex would probably be more descriptive.

While I know essentially how this needs to be done, I'm not sure I
could do it. I may be able to get someone who can.

>> I realize that I'm not proposing anything really simple now, but I
>> just want to say that if you have some suggestions for this, I would
>> really like to see the indexes distributed along with the modules.

>Fair enough.  That makes sense to me, too.  Deliver me source code for a
>working enhanced mkfastmod (that doesn't require ridiculously many Gnome
>dependencies in order to run, since it will get installed by all who
>install packaged sword data modules), and I think we can do this almost
>trivially in postinstall scripts.  I still think this is not really a
>packaging issue so much as a flawed or under-specified core module
>management interface... but I'll definitely work with you to resolve
>this need with postinstall scripts, if you can get me a commandline tool
>to build those indexes.  Deal?

>> 3. General comment: Sword and GnomeSword do allow the same module to
>> exist on the system. It is up to the front-end to decide how exactly
>> to handle this. GnomeSword does not show both modules but should
>> prefer the one in ~/.sword.

>Fine for naive end users, possibly bad for serious/advanced users who
>may actually want to be able to see the differences between those two
>modules!

I believe BibleTime allows this, but no "advanced user" has ever
requested such ability for GnomeSword.

>Ideally, of course, the module manager libraries recognize that this is
>a packaged-managed system and use apt to get and install the updated
>module (or RPM on Fedora, for example!)... but sadly you do not seem to
>be even considering upgrading your module manager in that way. Instead
>of working to get your code to fit in well with the environment that
>exists on Debian/Ubuntu machines, so conforming to Debian and Ubuntu
>community norms and expectations, you seek instead to vilify their
>package management and do your own.  But I doubt I can persuade you to
>change course on this and have your module installer cooperate with the
>Linux (or FreeBSD, or whatever) system package management approach.

As has been pointed out, the sword library works on a wide variety of
platforms, including most *nix's, Mac, Windows, Iphone, etc. and the
current system supports all of those platforms. What you propose does
not provide any benefit that I can see. With the Crosswire system,
users can get module updates immediately, they can get modules that
only Crosswire can distribute for legal reasons, they can get modules
from a private repository (quite trivial to set up, and useful for
cases where a church might want to share modules with just their
members), they can get modules from other repositories (we have 3
currently besides the main Crosswire repo, including one hosted by a
Bible publisher for their own materials), and this same system works
across platforms. Again, since it seems we are going to have this
discussion after all, could you explain what benefit your system
provides?

Matthew