[Pkg-ofed-commits] r343 - /trunk/howto/infiniband-howto.sgml
gmpc-guest at alioth.debian.org
gmpc-guest at alioth.debian.org
Mon Jun 1 14:19:37 UTC 2009
Author: gmpc-guest
Date: Mon Jun 1 14:19:37 2009
New Revision: 343
URL: http://svn.debian.org/wsvn/pkg-ofed/?sc=1&rev=343
Log:
Add MPI section. Fix spelling.
Modified:
trunk/howto/infiniband-howto.sgml
Modified: trunk/howto/infiniband-howto.sgml
URL: http://svn.debian.org/wsvn/pkg-ofed/trunk/howto/infiniband-howto.sgml?rev=343&op=diff
==============================================================================
--- trunk/howto/infiniband-howto.sgml (original)
+++ trunk/howto/infiniband-howto.sgml Mon Jun 1 14:19:37 2009
@@ -7,13 +7,13 @@
<sect>Introduction
<p>
-This document describes how to install and configure the OFED infiniband sotware on Debian. This document is intended
-to be a quickstart document on how to configure an Infiniband network as quickly as possible. It is not a replacment
+This document describes how to install and configure the OFED infiniband software on Debian. This document is intended
+to be a quick start document on how to configure an Infiniband network as quickly as possible. It is not a replacement
for the details documentation provided in the ofed-docs package!
<sect1>What is OFED?
<p>
-OFED (OpenFabric's Enterprise Distribution) is the defacto Infiniband software stack on linux. OFED
+OFED (OpenFabric's Enterprise Distribution) is the defacto Infiniband software stack on Linux. OFED
provides a consistent set of kernel modules and userspace libraries which have been tested together.
Further details of the Openfabrics Alliance and OFED can be found here <url url="http://www.openfabrics.org/"
@@ -26,17 +26,27 @@
You can choose to use the pre-build packages on alioth, or build your own packages straight from the alioth SVN repository.
<sect1>Installing prebuilt packages
<p>
-Download and install the packages at <url url="https://alioth.debian.org/frs/?group_id=100311"
-name ="https://alioth.debian.org/frs/?group_id=100311">. Packages are grouped by OFED release.
-Unless you know what you are doing, you should install all of the packages. Note that some OFED 1.4 packages are already
-in debian Lenny. You can install them from your usual repository.
-
+
+Add the following lines to your sources.list file:
+<tscreen>
+<verb>
+deb http://pkg-ofed.alioth.debian.org/apt/ofed ./
+deb-src http://pkg-ofed.alioth.debian.org/apt/ofed ./
+</verb>
+</tscreen>
+and run:
+<tscreen>
+<verb>
+aptitude update
+aptitude install ofed
+</verb>
+</tscreen>
<sect1>Building packages from source
<p>
If you wish to build the OFED packages from the alioth svn repository, use the following procedure.
-<sect2>Install the prerequities development packages
+<sect2>Install the prerequisites development packages
<p>
<tscreen>
<verb>
@@ -48,6 +58,7 @@
<tscreen>
svn co svn://svn.debian.org/pkg-ofed/
</tscreen>
+
<sect2>Install the upstream source (optional)
<p>
The upstream source tarballs need to be available if you
@@ -116,8 +127,8 @@
<p>
You now need to build a set of OFED kernel modules which match the version of the OFED software you have installed.
-The debian kernel contains a set of OFED infiniband drivers, but they may not match the OFED userspace version have installed.
-Consult the table below to determine what OFED version the debian kernel contains.
+The Debian kernel contains a set of OFED infiniband drivers, but they may not match the OFED userspace version have installed.
+Consult the table below to determine what OFED version the Debian kernel contains.
<tscreen>
<verb>
@@ -179,7 +190,7 @@
<sect1>Upgrade your Infiniband card and switch firmware
<p>
Before proceeding you should ensure that the firmware in your switches and infiniband cards is at the latest release.
-Older firmware versions may cause interoperbility and fabric stability issues. Do not assume that just because your
+Older firmware versions may cause interoperability and fabric stability issues. Do not assume that just because your
hardware has come fresh from the factory that it has the latest firmware on it.
You should follow the documentation from your vendor as to how the firmware should be updated.
@@ -191,7 +202,7 @@
<sect1>Choose a Subnet Manager
<p>
Each infiniband network requires a subnet manager. You can choose to run the OFED opensm subnet manager on one of the
-linux clients, or you may choose to use an embedded subnet manager running on one of the switches in your fabric. Note
+Linux clients, or you may choose to use an embedded subnet manager running on one of the switches in your fabric. Note
that not all switches come with a subnet manager; check your switch documentation.
@@ -214,6 +225,7 @@
#
# Protocol modules
# Common modules
+rdma_ucm
ib_umad
ib_uverbs
# IP over IB
@@ -372,13 +384,13 @@
<sect1>Testing RDMA performance
<p>
-You can test the latency and bandwith of a link with the ib_rdma_lat commands.
+You can test the latency and bandwidth of a link with the ib_rdma_lat commands.
To test the latency, start the server on a node:
<tscreen>
#ib_rdma_lat
</tscreen>
-and then start a client on anothe node, giving it the hostname of the server.
+and then start a client on another node, giving it the hostname of the server.
<tscreen>
<verb>
#ib_rdma_lat hostname-of-server
@@ -495,16 +507,119 @@
See the details in the ipoib_release_notes.txt document in the ofed-docs package.
-<sect>MPI
-<p>
-ADDME: How to run a test MPI application
-rdma_ucm
-
-
+<sect>OpenMPI
+<p>
+
+The section describes how to configure OpenMPI to use Infiniband.
+
+<sect1>Configure IPoIB
+<p>
+OpenMPI uses IPoIB for job startup and tear-down. You should configure IPoIB on all of your hosts.
+
+<sect1>Load the modules
+<p>
+Ensure the rdma_ucm module is loaded.
+<tscreen>
+modprobe rdma_ucm
+</tscreen>
+
+<sect1>Check permissions and limits
+<p>
+Uses who want to run MPI jobs will need to be placed into the rdma group to enable them to
+have write access to /dev/infiniband/uverbs* and /dev/infiniband/rdma_cm*.
+<p>
+OpenMPI will need to pin memory. Edit /etc/security/limits.conf and add the line:
+<tscreen>
+* hard memlock unlimited
+</tscreen>
+
+<sect1>Compile the test program
+<p>
+Install the openmpi development libraries:
+<tscreen>
+aptitude install openmpi-dev
+</tscreen>
-mpirun --mca btl_openib_verbose 1 --mca btl ^tcp -n 2 -hostfile ~/hostfile /nfs/acari/gmpc/work/infiniband/scripts/hello
-
- chmod a+rw /dev/infiniband/rdma_cm
+We will use a simple "hello world" program to test MPI functionality.
+<tscreen>
+<code>
+/* C Example */
+#include <stdio.h>
+#include <mpi.h>
+
+
+int main (argc, argv)
+ int argc;
+ char *argv[];
+{
+ int rank, size;
+ MPI_Init (&argc, &argv);/* starts MPI */
+ MPI_Comm_rank (MPI_COMM_WORLD, &rank);/* get current process id */
+ MPI_Comm_size (MPI_COMM_WORLD, &size);/* get number of processes */
+ printf( "Hello world from process %d of %d\n", rank, size );
+ MPI_Finalize();
+ return 0;
+}
+</code>
+</tscreen>
+
+Compile the hello-world.c program:
+
+<tscreen>
+mpicc hello-world.c -o hello-world
+</tscreen>
+
+Place the resulting binary on each node in you cluster. You can either copy it to a local filesystem on each node, or
+more conveniently, place it on a network filesystem visible to all the hosts.
+
+<sect1>Configure Hosts
+<p>
+OpenMPI uses ssh to spawn jobs on remote hosts. You should configure a public/private keypair to ensure that you
+can ssh between hosts without entering a password. You should also ensure that your login process is silent.
+
+Choose two hosts on which to test the program and put their hostnames into a file called hostfile:
+
+<tscreen>
+<verb>
+ hostA slots=1
+ hostB slots=1
+</verb>
+</tscreen>
+
+<sect1>Launch the program
+<p>
+
+Lauch the test program using the mpirun command. The -n option specifies the number of mpi processes to launch and
+the -h option points to the hostfile created above.
+For our
+testing we will also force mpi to ignore any TCP interfaces and make it extra-chatty during startup.
+(--mca btl_openib_verbose 1 --mca btl ^tcp options). These options are normally uncessesary, as openMPI will use
+any infiniband networks it detects in preference to any TCP networks.
+
+<verb>
+mpirun --mca btl_openib_verbose 1 --mca btl ^tcp -n 2 -hostfile /path/to/hostfile /path/to/hello-world
+</verb>
+
+If all goes well you should see debugging messages from both hosts, together with the job output.
+
+<tscreen>
+<verb>
+[hostA][[64185,1],0][btl_openib_ini.c:166:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 25418
+[hostA][[64185,1],0][btl_openib_ini.c:185:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon
+[hostA][[64185,1],0][btl_openib_ini.c:166:ompi_btl_openib_ini_query] Querying INI files for vendor 0x0000, part ID 0
+[hostA][[64185,1],0][btl_openib_ini.c:185:ompi_btl_openib_ini_query] Found corresponding INI values: default
+[hostB][[64185,1],1][btl_openib_ini.c:166:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 25418
+[hostB][[64185,1],1][btl_openib_ini.c:185:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon
+[hostB][[64185,1],1][btl_openib_ini.c:166:ompi_btl_openib_ini_query] Querying INI files for vendor 0x0000, part ID 0
+[hostB][[64185,1],1][btl_openib_ini.c:185:ompi_btl_openib_ini_query] Found corresponding INI values: default
+Hello world from process 0 of 2
+Hello world from process 1 of 2
+</verb>
+</tscreen>
+
+If you encounter any errors read the excellent OpenMPI troubleshooting guide. <url url="http://www.openmpi.org"
+name="http://www.openmpi.org">
+
<sect>SDP
<p>
@@ -598,9 +713,9 @@
modprobe ib_srp
</tscreen>
-<sect1>SRP deamon configuration
-<p>
-srp_deamon is responisble for discovering and connecting to SRP targets. The default configuration shipped with srp_daemon is to ignore all presented
+<sect1>SRP daemon configuration
+<p>
+srp_daemon is responsible for discovering and connecting to SRP targets. The default configuration shipped with srp_daemon is to ignore all presented
devices; this is a failsafe to prevent devices from being mounted by accident on the wrong hosts.
The srp_daemon config file /etc/srp_daemon.conf has a simply syntax, and is described in the srp_daemon(1) manpage. Each line in this file is a rule which can be either
@@ -629,7 +744,7 @@
d
</verb>
</tscreen>
-Restart the srp_deamon and the storage target should now become visible; check the kernel log to see if the disk has been detected.
+Restart the srp_daemon and the storage target should now become visible; check the kernel log to see if the disk has been detected.
<verb>
@@ -664,38 +779,100 @@
<p>
Lustre is a scalable cluster filesystem popular on high performance compute clusters. See <url url="http://www.lustre.org" name="http://www.lustre.org">
for more information. lustre can use infiniband as one of its network transports in order to increase performance. The section describes how to compile lustre
-against the OFED infinband stack.
+against the OFED infiniband stack.
<sect1>Check Compatibility
<p>
-<sect1>Install the OFED packages
-<p>
-<sect1>Install a set of kernel modules
-<p>
+Not all lustre versions are compatible with all OFED or kernel versions. Read the lustre release notes for which versions are supported.
+
+
+<sect1>Build a lustre patched kernel
+<p>
+Build a lustre patched kernel as per the instructions on the lustre wiki. Once you have build the kernel keep the configured source tree.
+It is required for the next step.
+
+
+<sect1>Build OFED modules for the lustre patched kernel.
+<p>
+Build OFED modules against the newly build lustre patched kernel.
+
+<tscreen>
+<verb>
+ module-assistant prepare
+ module-assistant clean ofa-kernel
+ module-assistant -k/path/to/lustre/patched/kernel build ofa-kernel
+</verb>
+</tscreen>
+
+Do not issue a "module-assistant clean" command after the build. The ofa-kernel-module source tree is needed for the
+next step.
+
+
<sect1>Configure lustre
<p>
+You can now configure lustre to build against the lustre patched kernel and the ofa-kernel-module sources.
+
+<tscreen>
+<verb>
+ cd lustre-source
+ ./configure --with-o2ib=/usr/src/modules/ofa-kernel --with-linux=/path/to/linux-2.6.22.19 \
+ ... --other-options
+</verb>
+</tscreen>
<sect>Network Troubleshooting
<p>
-Diags:
+<sect1>ibdiagnet
+<p>
+The ibdiagnet program can be used to troubleshoot potential issues with your infiniband fabric.
+
+<tscreen>
ibdiagnet -r
-
-
-Applications
-
- Lustre over IB.
-
- Example MPI application.
-
-
- openmpi-dev
-
-
-
-SDP
+</tscreen>
<sect>Further Information
<p>
+Extensive documentation on the OFED software is present in the ofed-docs package.
+
+The openfabrics alliance webpage can be found here:
+
+<url url="http://www.openfabrics.org/" name="http://www.openfabrics.org/">
+
+
+The following mailing lists are also useful:
+
+<url url="http://lists.alioth.debian.org/mailman/listinfo/pkg-ofed-devel" name="http://lists.alioth.debian.org/mailman/listinfo/pkg-ofed-devel">:
+pkg-ofed-devel: Discussion of debian specific problem or issues.
+
+
+<url url="http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general" name="http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general">:
+ofa-general: General discussion of the OFED software.
+
+Books:
+<verb>
+Infiniband Network Architecture
+by MindShare, Inc.; Tom Shanley
+Publisher: Addison-Wesley Professional
+Pub Date: October 31, 2002
+Print ISBN-10: 0-321-11765-4
+</verb>
</article>
+
+<!-- LocalWords: infiniband ofed svn ibping Pong RDMA ib rdma lat hostname bw
+ -->
+<!-- LocalWords: tx iters cma QPN PSN RKey VAddr sec KB Avg IP IPoIB SDP inet
+ -->
+<!-- LocalWords: IBoIP modprobe ipoib iface netmask ifup UD MTU txt OpenMPI
+ -->
+<!-- LocalWords: ucm MPI memlock openmpi dev mpicc filesystem ssh keypair mca
+ -->
+<!-- LocalWords: hostnames hostfile mpirun btl openib tcp sdp LD PRELOAD Mbps
+ -->
+<!-- LocalWords: config netpipe nodeA NPtcp nodeB usec SRP srp IDs ibsrpdm ff
+ -->
+<!-- LocalWords: ext ioc guid cmd scsi DCS PQ sd sdb DPO FUA sg Multipathing
+ -->
+<!-- LocalWords: LVM multipath multipathed wiki
+ -->
More information about the Pkg-ofed-commits
mailing list