<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
On Sat, 16 Apr 2016 00:42:33 +0530 Ritesh Raj Sarraf
<a class="moz-txt-link-rfc2396E" href="mailto:rrs@debian.org"><rrs@debian.org></a> wrote:<br>
<span style="white-space: pre;">> On Sat, 2016-04-16 at 00:35 +0530, Ritesh Raj Sarraf wrote:
> > Hello Mike,
> >
> > All I can say is that I am able to see the behavior that you've mentioned (So
> > marking the bug as confirmed). THis is seen on my LIO iSCSI Setup running on 2
> > Guest VMs.
> >
> > In doing that, I was surprised that those dev/mapper/ entries are now
> > symlinks.
> > My storage knowledge has rusted for some time now, but from what I recollect,
> > it
> > used to by direct block node names, not symlinks.
>
> Maybe this could be of some help to you in debugging this further.
>
> Apr 16 00:34:11 debian-sanboot systemd[1]: Started Device-Mapper Multipath
> Device Controller.
> Apr 16 00:34:11 debian-sanboot systemd-udevd[949]: conflicting device node
> '/dev/mapper/sanroot' found, link to '/dev/dm-0' will not be created
>
>
> So I think my understanding was correct about Device Mapper's behavior. Now I'm
> not sure what 'systemd-udevd[949]' is reflecting here as. I doubt if that is the
> ID for the rules processing because then I'd have seen the respective rule.
>
> I hope this is of some use to you. My Debian time for today is running out :-)
></span><br>
<br>
Hi Ritesh,<br>
<br>
I am working with Mike on this problem.<br>
<br>
We have seen this issue using multiple transports (FCoE, FC, and<br>
iSCSI), although most of our testing has been using QLogic FC cards<br>
with 3Par FC storage. I don't think this is a transport problem, but<br>
is instead, some sort of udev/multipath interaction problem.<br>
<br>
Here is a test I ran with multipath-tools_0.5.0-7 and<br>
multipath-tools_0.5.0+git0.770e6d0d-1 on stretch using the same<br>
hardware and configuration that Mike has been using:<br>
<br>
Here is the storage:<br>
<br>
1:0:0:1] disk 3PARdata VV 3212 /dev/sdb <br>
[1:0:0:2] disk 3PARdata VV 3212 /dev/sdc <br>
[1:0:0:254] enclosu 3PARdata SES 3212 - <br>
[1:0:1:1] disk 3PARdata VV 3212 /dev/sdd <br>
[1:0:1:2] disk 3PARdata VV 3212 /dev/sde <br>
[1:0:1:254] enclosu 3PARdata SES 3212 - <br>
<br>
And udev monitor output (udevadm monitor -u -k):<br>
<br>
multipath-tools_0.5.0-7<br>
<br>
# udevadm monitor -k -u<br>
monitor will print the received events for:<br>
UDEV - the event which udev sends out after rule processing<br>
KERNEL - the kernel uevent<br>
<br>
KERNEL[90540.185998] change /devices/virtual/block/dm-0 (block)<br>
KERNEL[90540.187857] change /devices/virtual/block/dm-1 (block)<br>
UDEV [90540.265654] change /devices/virtual/block/dm-1 (block)<br>
UDEV [90540.266138] change /devices/virtual/block/dm-0 (block)<br>
<br>
# ls -l /dev/mapper<br>
total 0<br>
crw------- 1 root root 10, 236 Apr 14 14:31 control<br>
lrwxrwxrwx 1 root root 7 Apr 15 15:40 mpatha -> ../dm-0<br>
lrwxrwxrwx 1 root root 7 Apr 15 15:40 mpathb -> ../dm-1<br>
<br>
multipath-tools_0.5.0+git0.770e6d0d-1<br>
<br>
# udevadm monitor -k -u<br>
monitor will print the received events for:<br>
UDEV - the event which udev sends out after rule processing<br>
KERNEL - the kernel uevent<br>
<br>
KERNEL[91354.622867] change /devices/virtual/block/dm-0 (block)<br>
KERNEL[91354.625218] change /devices/virtual/block/dm-1 (block)<br>
UDEV [91354.699455] change /devices/virtual/block/dm-0 (block)<br>
UDEV [91354.701663] change /devices/virtual/block/dm-1 (block)<br>
<br>
So the udev events look identical. We think that perhaps we have a<br>
race and the new version of multipath is not waiting (or waiting
long<br>
enough) for udev to crate the device files and just goes and creates<br>
the required files.<br>
<br>
I am going to try and bisect the changes in the upstream multipath
code between<br>
0.5.0 and 770e6d0d to see where this breaks.<br>
<br>
This does not seem to that serious a problem on its own, since the<br>
block device that is created is identical to what the symlink would<br>
point to. But we are seeing other issue. But we are seeing other<br>
issues like device files reappearing after the running multipath -f
on<br>
it, removing the LUN, and then running multipath -r which we think
is<br>
related.<br>
<br>
-- <br>
Andrew Patterson<br>
Hewlett-Packard Enterprise<br>
<br>
<br>
</body>
</html>