[Pkg-ofed-devel] ofed + ubuntu

[Mark J. Pearrow]

Ah, many thanks for this info. The issue that prompted me to use the  
diagnostics was that when trying to get GlusterFS to run over IB, I  
got the same error (see below). I suspect they are using XRC in their  
startup routine to check the connection, etc.

root at ba7:~# glusterfs --debug -f /etc/glusterfs/glusterfsd.vol
[2009-08-03 12:17:51] D [glusterfsd.c:354:_get_specfp] glusterfs:  
loading volume file /etc/glusterfs/glusterfsd.vol
[2009-08-03 12:17:51] D [xlator.c:634:xlator_set_type] xlator:  
dlsym(notify) on /usr/lib/glusterfs/2.0.4/xlator/features/locks.so:  
undefined symbol: notify -- neglecting
[2009-08-03 12:17:51] D [xlator.c:634:xlator_set_type] xlator:  
dlsym(notify) on /usr/lib/glusterfs/2.0.4/xlator/performance/io- 
threads.so: undefined symbol: notify -- neglecting
= 
= 
= 
= 
= 
= 
= 
= 
========================================================================
Version      : glusterfs 2.0.4 built on Aug  3 2009 08:33:26
TLA Revision : v2.0.4
Starting Time: 2009-08-03 12:17:51
Command line : glusterfs --debug -f /etc/glusterfs/glusterfsd.vol
PID          : 9284
System name  : Linux
Nodename     : ba7
Kernel Release : 2.6.28-11-generic
Hardware Identifier: x86_64

Given volfile:
+ 
------------------------------------------------------------------------------+
   1: # file: /etc/glusterfs/glusterfs-server.vol
   2: volume posix
   3:   type storage/posix
   4:   option directory /d2/gluster
   5: end-volume
   6:
   7: volume locks
   8:   type features/locks
   9:   option mandatory-locks on
  10:   subvolumes posix
  11: end-volume
  12:
  13: volume brick
  14:   type performance/io-threads
  15:   option thread-count 8
  16:   subvolumes locks
  17: end-volume
  18:
  19: volume server
  20:   type protocol/server
  21:   option transport-type ib-verbs/server
  22:   option auth.addr.brick.allow 192.168.66.*
  23:   subvolumes brick
  24: end-volume

+ 
------------------------------------------------------------------------------+
[2009-08-03 12:17:51] D [glusterfsd.c:1205:main] glusterfs: running in  
pid 9284
[2009-08-03 12:17:51] D [io-threads.c:2280:init] brick: io-threads:  
Autoscaling: off, min_threads: 8, max_threads: 8
[2009-08-03 12:17:51] D [transport.c:141:transport_load] transport:  
attempt to load file /usr/lib/glusterfs/2.0.4/transport/ib-verbs.so
mlx4: There is a mismatch between the kernel and the userspace  
libraries: Kernel does not support XRC. Exiting.
[2009-08-03 12:17:51] E [ib-verbs.c:1382:ib_verbs_get_device]  
transport/ib-verbs: cannot open device `mlx4_0'
[2009-08-03 12:17:51] E [ib-verbs.c:1544:ib_verbs_init] transport/ib- 
verbs: could not create ib_verbs device for mlx4_0
[2009-08-03 12:17:51] E [ib-verbs.c:2369:init] server: Failed to  
initialize IB Device
[2009-08-03 12:17:51] E [transport.c:200:transport_load] transport:  
'ib-verbs' initialization failed
[2009-08-03 12:17:51] E [server-protocol.c:7643:init] server: failed  
to load transport
[2009-08-03 12:17:51] E [xlator.c:736:xlator_init_rec] xlator:  
Initialization of volume 'server' failed, review your volfile again
[2009-08-03 12:17:51] E [glusterfsd.c:513:_xlator_graph_init]  
glusterfs: initializing translator failed
[2009-08-03 12:17:51] E [glusterfsd.c:1217:main] glusterfs: translator  
initialization failed.  exiting






On Aug 3, 2009, at 12:12 PM, Guy Coates wrote:
> Hi Mark,
>
> Thanks for the installation report.
>
> XRC support is currently not in the mainline kernel infiniband  
> modules.
>
> You have two options:
>
> 1) Not worry about it. The only package that actually uses XRC are the
> benchmarking programs in the perftest package.
>
>
> 2) Build a set of OFED kernel modules.
>
> On debian you can use module-assistant to build the OFED kernel for  
> you.
>
> http://pkg-ofed.alioth.debian.org/howto/infiniband-howto-3.html
>
> In theory, this should also work on ubuntu too. However, the ofa- 
> kernel package
> only contains patches for kernels <= 2.6.26, so I suspect that it  
> will not build
> correctly against the ubuntu 2.6.28. (I don't have an ubuntu system  
> to test).
>
> If you want to try anyhow, you will need to dig into the ofa-kernel  
> package and
> hack the ofed_scripts/ofed_patch.sh, which is responsible for  
> guessing the
> kernel version and building the appropriate modules.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.alioth.debian.org/pipermail/pkg-ofed-devel/attachments/20090803/aaaf0191/attachment.htm>