[Neurodebian-devel] [RFC] Should we convert to use 'Modules' (http://modules.sourceforge.net/)?

[Yaroslav Halchenko]

Hi guys,

For packages (such as afni and fsl) which provide a multitude of command
line tools and environment variables so far we have been relying on
users to source /etc/X/X.sh to enrich their environment.   The problem
of defining per-tool/version environment is not new and quite
elderly http://modules.sourceforge.net  is being used on many systems,
especially in HPC setups.   It provides shell agnostic (although done
internally in TCL) way to augment (reversibly) environment to accomplish
the same goal.

I was thinking that we should start use 'modules' in place of our ad-hoc
shell (primarily bash) scripts.  Benefits would be

- ability to load AND unload
- support of users with non-POSIX shells
- unify analysis processing scripts with possible setups on HPC deployments
- uniformity among all the packages requiring such a functionality,
  while making it flexible enough for admins to provide customizations

Modules is now available in Debian (as environment-modules package,
thanks to Alastair McKinstry, CCed) and for which I have already
built backports for all releases (only not for ubuntu 10.04 which RIP
anyways in desktop and server expected to be discontinued only in April
2015).

I was thinking to provide modulesfile for upcoming update of ANTS
package, and upon reading and tinkering with it (not really using yet) I
came up with the following scheme and I hope you could provide me with
feedback either there are (other) shortcomings.  There is 1 major
corner-stone -- possibly existing per-user customizations, which I am
not yet sure what to do about them (see the last Q:):


a. within a package for tool X ship following files under
   /usr/share/modules/modulefiles/ directory:

  1.
   X/UVERSION

  where X is the name of the tool (e.g. afni) and UVERSION is upstream version
  (e.g. /usr/share/modules/modulefiles/ants/2.0.0).  this is the main modulefile
  for the UVERSION setting up the environment

  2.
   X/.version

   file specifying the default version (similar to our current
   /etc/fsl/fsl.sh -> 5.0/fsl.sh)

  3.
   For packages for which we maintain multiple major releases (e.g. FSL 4.1 and
   5.0), provide symlinks to the supported release line (similar to dynamic
   libraries setup). e.g. if for ants we decided to support both old 1.9
   and new 2.0: 2.0 -> 2.0.0,   1.9 -> 1.9.8

   this way users could explicitly load a specific 'release line', e.g.

    module load ants/1.9 (or module load fsl/5.0)

   while 

    module load ants

   would load the version we like them to use as the default

   Q: do you see any obvious shortcomings/deficiencies?

b. administrators would be welcome to add custom modulefiles
   pointing to local installations, happen they have them, and place them under
   
    /etc/environment-modules/modules 
    
   By doing so they might 'overload' definition of the default
   version specified in /usr/share/modules/modulefiles/X/.version thus it
   would be up to them either symlink that .version file or just have
   their possibly newer version being default

c. we then could replace custom /etc/X/X.sh scripts with simple 'module load X'
   (versioned or not), to avoid any duplication of the setup while staying compatible
   with existing deployments.  
   
   The only catch here which I do not know how to resolve yet
   without interaction with users is:

   $> tail -n 3 /etc/fsl/fsl.sh /etc/afni/afni.sh
    ==> /etc/fsl/fsl.sh <==
    if [ -f "${HOME}/.fslconf/fsl.sh" ] ; then
      . "${HOME}/.fslconf/fsl.sh" ;
    fi

    ==> /etc/afni/afni.sh <==
    if [ -f "${HOME}/.config/AFNI/afni.sh" ] ; then
      . "${HOME}/.config/AFNI/afni.sh" ;
    fi

   Since these are shell-specific they can't be just "sourced" anyhow within
   modulefile.  It should be possible to check and warn users though if such a
   file was found present, and advise to convert it to modulefiles but I am not
   yet sure on how easy that would be todo/enable.

   There can be ~/.modulerc file providing generic tune up, e.g.

    $> cat ~/.modulerc 
    #%Module1.0#######################################################################
    module-version ants/1.9 default

   would specify that I want to use 1.9 regardless of the system-wide
   defaults.  Thus question:
   
   Q: How we could enable per user per module specific settings?
   If someone had experience with Modules -- I would be great full for
   your help.

   Q: how often those per-user settings are used in your experience (I
   haven't used them)

PS. FYI: default installation looks for modulefiles (defining how to
augment environment) under following directories:

$> grep -e '^[^#]' /etc/environment-modules/modulespath 
/etc/environment-modules/modules
/usr/share/modules/versions             # location of version files
/usr/Modules/$MODULE_VERSION/modulefiles    # Module pkg modulefiles (if versioning)
/usr/share/modules/modulefiles              # General module files

-- 
Yaroslav O. Halchenko, Ph.D.
http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org
Senior Research Associate,     Psychological and Brain Sciences Dept.
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik