[Pkg-ofed-devel] Debian Squeeze - MT25408 Mellanox Infiniband card not working with Xen multiboot loaded.

[Ira Weiny]

I don't know much about Xen but...

On Thu, 7 Jul 2011 13:06:39 -0700
Krzysztof 'LANcaster' Kotkowicz <krzysztof.kotkowicz at gmail.com> wrote:

> Hello everyone :-)
> 
> I got problem with my MT25408 Mellanox card, on debian squeeze.
> When I load 2.6.32-5-xen-amd64 kernel, without GRUB multiboot option,
> the card is running well.
> But when I select start the Xen 4.1.1 the card status is:
> 
> 
> CA 'mlx4_0'
>        CA type: MT26428
>        Number of ports: 2
>        Firmware version: 2.8.0
>        Hardware version: a0
>        Node GUID: 0x0002c90300089dae
>        System image GUID: 0x0002c90300089db1
>        Port 1:poszło ;)
                ^^^^^^^^^
Not sure what this is?  Copy and paste error?

>                State: Initializing

This means your SM has not transitioned the port to Active.

>                Physical state: LinkUp
>                Rate: 40
>                Base lid: 0
>                LMC: 0
>                SM lid: 0
>                Capability mask: 0x02510868
>                Port GUID: 0x0002c90300089daf
>                Link layer: IB
>        Port 2:
>                State: Down
>                Physical state: Polling
>                Rate: 70
>                Base lid: 0
>                LMC: 0
>                SM lid: 0
>                Capability mask: 0x02510868
>                Port GUID: 0x0002c90300089db0
>                Link layer: IB
> 
> And card doesn't see the network.
> 
> root at genesis101:~# ibswitches
> src/query_smp.c:192; umad (DR path slid 0; dlid 0; 0,1 Attr 0x11:0)
> bad status 110; Connection timed out

This should work on a port which has not transitioned (Assuming you are connected to a switch).  Is the ibstat output above from the hypervisor or a client?  Again I don't know how Xen works but ibswitches should work on the hypervisor talking to real hardware.

You might want to try something like:

corni at root:smpquery nodeinfo -D 0,1
# Node info: DR path slid 65535; dlid 65535; 0,1
...
NodeType:........................Switch
...

To verify you are connected to a switch.

Ira

> 
> On default xen 4.0 installation, card is Active, but there is still
> problem with ib network connection.
> 
> Here is my dmesg: http://pastebin.com/ddrCnGZT
> 
> I will appreciate any help :-)
> 
> _______________________________________________
> Pkg-ofed-devel mailing list
> Pkg-ofed-devel at lists.alioth.debian.org
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-ofed-devel


-- 
Ira Weiny
Math Programmer/Computer Scientist
Lawrence Livermore National Lab
925-423-8008
weiny2 at llnl.gov