[Pkg-ofed-devel] IB on Squeeze, appropriate OFED packages and kernel support

[Bill Johnstone]

Hello all,

I'm trying to get an IB fabric working for an AMD64-based HPC cluster.  The nodes, which were previously on RHEL, are being replaced with Squeeze.  I'm keen on being able to pick and choose the pieces of OFED that I need installed, instead of being forced to install all of OFED, including its bundled and out-of-date MPI distribution, and things like OpenSM since my switch has an onboard SM.

However, I'm finding the documentation related to the IB stack on Debian to be a little out-of-date and lacking, and so would appreciate some help with a few questions.

1. Is there a particular reason ofa-kernel is not in the main Squeeze repo?  AFAICT, all the other meaningful packages of note from OFED are available individually?

2. With Squeeze, is it still appropriate to install ofa-kernel in the first place?  Without installing the ofa-kernel package, I have drivers for my hardware (mthca and mlx4), as well as the core protocols and such.  However, there doesn't seem to be a module for SDP by default, and I would like to have suport for it.  Does that mean I should be installing ofa-kernel?

By default, even if I load the appropriate hardware driver via /etc/modules , the permissions do not allow non-root access to the device nodes and there are no udev rules for me to alter the perms.  Presumably, the relevant udev rules would get installed with ofa-kernel?

3. Is the list of modules to load from http://pkg-ofed.alioth.debian.org/howto/infiniband-howto-4.html still current?  Shouldn't it also call for ib_ucm to be loaded?

4. This is more of a generic modules question, but if I've got an NFSROOT netbooting system image for machines that are otherwise similar, but have the two different IB HCAs mentioned above (mthca and mlx4), is there a way I can statically write /etc/modules or setup modprobe in such a way that only the appropriate kernel module is loaded for the particular IB HCA on each machine?

Right now, I just have both modules called out in /etc/modules, and it doesn't seem to be doing any harm, but the unused driver modules stay resident.

5. The Debian OFED packages all seem to be based on OFED 1.4 .  Is there a particular reason that OFED 1.5 is not being used as the basis for these packages?

Thanks a lot for any help.