<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Ah, many thanks for this info. The issue that prompted me to use the diagnostics was that when trying to get GlusterFS to run over IB, I got the same error (see below). I suspect they are using XRC in their startup routine to check the connection, etc. <div><br></div><div><div>root@ba7:~# glusterfs --debug -f /etc/glusterfs/glusterfsd.vol</div><div>[2009-08-03 12:17:51] D [glusterfsd.c:354:_get_specfp] glusterfs: loading volume file /etc/glusterfs/glusterfsd.vol</div><div>[2009-08-03 12:17:51] D [xlator.c:634:xlator_set_type] xlator: dlsym(notify) on /usr/lib/glusterfs/2.0.4/xlator/features/locks.so: undefined symbol: notify -- neglecting</div><div>[2009-08-03 12:17:51] D [xlator.c:634:xlator_set_type] xlator: dlsym(notify) on /usr/lib/glusterfs/2.0.4/xlator/performance/io-threads.so: undefined symbol: notify -- neglecting</div><div>================================================================================</div><div>Version : glusterfs 2.0.4 built on Aug 3 2009 08:33:26</div><div>TLA Revision : v2.0.4</div><div>Starting Time: 2009-08-03 12:17:51</div><div>Command line : glusterfs --debug -f /etc/glusterfs/glusterfsd.vol </div><div>PID : 9284</div><div>System name : Linux</div><div>Nodename : ba7</div><div>Kernel Release : 2.6.28-11-generic</div><div>Hardware Identifier: x86_64</div><div><br></div><div>Given volfile:</div><div>+------------------------------------------------------------------------------+</div><div> 1: # file: /etc/glusterfs/glusterfs-server.vol</div><div> 2: volume posix</div><div> 3: type storage/posix</div><div> 4: option directory /d2/gluster</div><div> 5: end-volume</div><div> 6: </div><div> 7: volume locks</div><div> 8: type features/locks</div><div> 9: option mandatory-locks on</div><div> 10: subvolumes posix</div><div> 11: end-volume</div><div> 12: </div><div> 13: volume brick</div><div> 14: type performance/io-threads</div><div> 15: option thread-count 8</div><div> 16: subvolumes locks</div><div> 17: end-volume</div><div> 18: </div><div> 19: volume server</div><div> 20: type protocol/server</div><div> 21: option transport-type ib-verbs/server</div><div> 22: option auth.addr.brick.allow 192.168.66.*</div><div> 23: subvolumes brick</div><div> 24: end-volume</div><div><br></div><div>+------------------------------------------------------------------------------+</div><div>[2009-08-03 12:17:51] D [glusterfsd.c:1205:main] glusterfs: running in pid 9284</div><div>[2009-08-03 12:17:51] D [io-threads.c:2280:init] brick: io-threads: Autoscaling: off, min_threads: 8, max_threads: 8</div><div>[2009-08-03 12:17:51] D [transport.c:141:transport_load] transport: attempt to load file /usr/lib/glusterfs/2.0.4/transport/ib-verbs.so</div><div>mlx4: There is a mismatch between the kernel and the userspace libraries: Kernel does not support XRC. Exiting.</div><div>[2009-08-03 12:17:51] E [ib-verbs.c:1382:ib_verbs_get_device] transport/ib-verbs: cannot open device `mlx4_0'</div><div>[2009-08-03 12:17:51] E [ib-verbs.c:1544:ib_verbs_init] transport/ib-verbs: could not create ib_verbs device for mlx4_0</div><div>[2009-08-03 12:17:51] E [ib-verbs.c:2369:init] server: Failed to initialize IB Device</div><div>[2009-08-03 12:17:51] E [transport.c:200:transport_load] transport: 'ib-verbs' initialization failed</div><div>[2009-08-03 12:17:51] E [server-protocol.c:7643:init] server: failed to load transport</div><div>[2009-08-03 12:17:51] E [xlator.c:736:xlator_init_rec] xlator: Initialization of volume 'server' failed, review your volfile again</div><div>[2009-08-03 12:17:51] E [glusterfsd.c:513:_xlator_graph_init] glusterfs: initializing translator failed</div><div>[2009-08-03 12:17:51] E [glusterfsd.c:1217:main] glusterfs: translator initialization failed. exiting</div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br><div><div>On Aug 3, 2009, at 12:12 PM, Guy Coates wrote:</div><blockquote type="cite"><div>Hi Mark,<br><br>Thanks for the installation report.<br><br>XRC support is currently not in the mainline kernel infiniband modules.<br><br>You have two options:<br><br>1) Not worry about it. The only package that actually uses XRC are the<br>benchmarking programs in the perftest package.<br><br><br>2) Build a set of OFED kernel modules.<br><br>On debian you can use module-assistant to build the OFED kernel for you.<br><br><a href="http://pkg-ofed.alioth.debian.org/howto/infiniband-howto-3.html">http://pkg-ofed.alioth.debian.org/howto/infiniband-howto-3.html</a><br><br>In theory, this should also work on ubuntu too. However, the ofa-kernel package<br>only contains patches for kernels <= 2.6.26, so I suspect that it will not build<br>correctly against the ubuntu 2.6.28. (I don't have an ubuntu system to test).<br><br>If you want to try anyhow, you will need to dig into the ofa-kernel package and<br>hack the ofed_scripts/ofed_patch.sh, which is responsible for guessing the<br>kernel version and building the appropriate modules.<br><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" color="#144FAE"><br></font></font></div></blockquote></div></div></div></body></html>