Problem Scenario
Wanted to clone a pure storage volume on XFS filesystem for testing purpose on the same server.
- Took FlashRecover snapshot of the volume.
- Instantiated the snapshot by copying them into target volume.
- Attached the target volume to the same host and updated multipath.conf with the alias information.
- Tried to mount the partition from the target volume on the host but was not successful.
Error Messages on the host
[root@donald ~]# df -hT /x01
Filesystem Type Size Used Avail Use% Mounted on
/dev/mapper/fs_xfs1 xfs 2.0G 33M 2.0G 2% /x01
[root@donald ~]# ssh pureuser@10.21.34.3
pureuser@10.21.34.3's password:
pureuser@pure-m50-2-b12-35-ct1> purevol snap --suffix snap test_xfs
Name Size Source Created Serial
test_xfs.snap 2G test_xfs 2016-07-31 15:38:09 PDT 8C44AF151613837200011BF9
pureuser@pure-m50-2-b12-35-ct1> purevol copy test_xfs.snap cloned_test_xfs
Name Size Source Created Serial
cloned_test_xfs 2G test_xfs 2016-07-31 15:38:46 PDT 8C44AF151613837200011BEC
pureuser@pure-m50-2-b12-35-ct1> purehost connect --vol cloned_test_xfs donald
Name Vol LUN
donald cloned_test_xfs 3
pureuser@pure-m50-2-b12-35-ct1> exit
logout
Connection to 10.21.34.3 closed.
[root@donald ~]# vi /etc/multipath.conf
. . .
multipath {
wwid 3624a93708c44af151613837200011be8
alias fs_xfs
}
multipath {
wwid 3624a93708c44af151613837200011bec
alias fs_cloned_xfs
}
[root@donald ~]# systemctl restart multipathd.service
[root@donald ~]# rescan-scsi-bus.sh
Scanning SCSI subsystem for new devices
Scanning host 0 for all SCSI target IDs, all LUNs
Scanning for device 0 0 2 10 ...
OLD: Host: scsi0 Channel: 00 Id: 02 Lun: 10
Vendor: PURE Model: FlashArray Rev: 451
Type: Direct-Access ANSI SCSI revision: 06
. . . .
16 new or changed device(s) found.
. . . .
0 remapped or resized device(s) found.
0 device(s) removed.
[root@donald ~]# ls -ltr /dev/mapper/fs*xfs*
lrwxrwxrwx 1 root root 7 Jul 31 15:41 /dev/mapper/fs_xfs -> ../dm-2
lrwxrwxrwx 1 root root 7 Jul 31 15:41 /dev/mapper/fs_cloned_xfs -> ../dm-5
lrwxrwxrwx 1 root root 7 Jul 31 15:41 /dev/mapper/fs_xfs1 -> ../dm-8
lrwxrwxrwx 1 root root 8 Jul 31 15:41 /dev/mapper/fs_cloned_xfs1 -> ../dm-10
[root@donald ~]# mount -t xfs -o noatime /dev/mapper/fs_cloned_xfs1 /x02
mount: wrong fs type, bad option, bad superblock on /dev/mapper/fs_cloned_xfs1,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so.
Reason for the Error
Cloning a Pure FlashArray volume and mounting the filesystem of the cloned volume on the same Linux host as the source volume will throw “duplicate UUID” error or similar errors. The cloning of the volume reflects everything from the source at the point of time of the snapshot which also includes the UUID of the partitions within the volume. The UUID is the unique identifier pointing to the partition and hence it is not unique anymore with the cloned volume on the host.
This is generally not an issue when the cloned volume is attached to a different host than that of the source volume.
For EXT4 filesystems, even though /var/log/messages shows error about device appeared twice with different sysfs paths, it doesn’t throw the error when mounting the filesystem from the cloned volume.
Solution
The UUID of the partition should be changed before it is mounted. The commands vary based on the filesystem type (EXT4 vs XFS) but the idea is to change the UUID of the partition and mount. Run xfs_repair first on the partition, followed by xfs_admin to generate new UUID.
root@donald ~]# xfs_repair /dev/mapper/fs_cloned_xfs1 Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... - scan filesystem freespace and inode maps... - found root inode chunk Phase 3 - for each AG... - scan and clear agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 1 - agno = 2 - agno = 5 - agno = 3 - agno = 6 - agno = 4 - agno = 7 Phase 5 - rebuild AG headers and trees... - reset superblock... Phase 6 - check inode connectivity... - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify and correct link counts... done [root@donald ~]# xfs_admin -U generate /dev/mapper/fs_cloned_xfs1 Clearing log and setting UUID writing all SBs new UUID = 2f756348-e144-4569-b616-c2837d4ee944 [root@donald ~]# mount -t xfs -o noatime /dev/mapper/fs_cloned_xfs1 /x02 [root@donald ~]# df -h |grep x0 /dev/mapper/fs_xfs1 2.0G 33M 2.0G 2% /x01 /dev/mapper/fs_cloned_xfs1 2.0G 33M 2.0G 2% /x02
Excerpts from /var/log/messages
Jul 31 15:41:35 donald kernel: sd 0:0:4:3: [sdf] Attached SCSI disk Jul 31 15:41:35 donald systemd: Device dev-disk-by\x2did-scsi\x2d3624a93708c44af151613837200011be8.device appeared twice with different sysfs paths /sys/devices/pci0000:00/0000:00:02.0/0000:02:00.0/0000:03:00.0/0000:04:00.0/0000:05:01.0/0000:07:00.0/host0/rport-0:0-9/target0:0:3/0:0:3:1/block/sda and /sys/devices/pci0000:00/0000:00:02.0/0000:02:00.0/0000:03:00.0/0000:04:00.0/0000:05:01.0/0000:07:00.0/host0/rport-0:0-10/target0:0:4/0:0:4:1/block/sdd Jul 31 15:41:35 donald systemd: Device dev-disk-by\x2did-wwn\x2d0x624a93708c44af151613837200011be8.device appeared twice with different sysfs paths /sys/devices/pci0000:00/0000:00:02.0/0000:02:00.0/0000:03:00.0/0000:04:00.0/0000:05:01.0/0000:07:00.0/host0/rport-0:0-9/target0:0:3/0:0:3:1/block/sda and /sys/devices/pci0000:00/0000:00:02.0/0000:02:00.0/0000:03:00.0/0000:04:00.0/0000:05:01.0/0000:07:00.0/host0/rport-0:0-10/target0:0:4/0:0:4:1/block/sdd Jul 31 15:41:35 donald multipathd: sdd: add path (uevent) Jul 31 15:41:35 donald systemd-udevd: error: /dev/sdd1: No such file or directory Jul 31 15:41:35 donald systemd-udevd: inotify_add_watch(7, /dev/sdd1, 10) failed: No such file or directory Jul 31 15:41:35 donald multipathd: fs_xfs: load table [0 4194304 multipath 0 0 1 1 queue-length 0 2 1 8:0 1 8:48 1] Jul 31 15:41:35 donald multipathd: sdd [8:48]: path added to devmap fs_xfs Jul 31 15:41:35 donald multipathd: sdc: add path (uevent) Jul 31 15:41:35 donald systemd-udevd: error: /dev/sdc1: No such file or directory Jul 31 15:41:35 donald systemd-udevd: inotify_add_watch(7, /dev/sdc1, 10) failed: No such file or directory Jul 31 15:41:35 donald multipathd: fs_cloned_xfs: load table [0 4194304 multipath 0 0 1 1 queue-length 0 1 1 8:32 1] Jul 31 15:41:35 donald multipathd: fs_cloned_xfs: event checker started Jul 31 15:41:35 donald multipathd: sdc [8:32]: path added to devmap fs_cloned_xfs Jul 31 15:41:35 donald systemd: Device dev-disk-by\x2duuid-cd8db41d\x2dccb4\x2d4b1b\x2d8c42\x2d28241e1485ce.device appeared twice with different sysfs paths /sys/devices/virtual/block/dm-8 and /sys/devices/virtual/block/dm-10 . . . . . . . . Jul 31 15:45:13 donald kernel: XFS (dm-8): Mounting V4 Filesystem Jul 31 15:45:13 donald kernel: XFS (dm-8): Ending clean mount Jul 31 15:45:23 donald kernel: XFS (dm-10): Filesystem has duplicate UUID cd8db41d-ccb4-4b1b-8c42-28241e1485ce - can't mount . . . . . . . . Jul 31 15:50:08 donald kernel: XFS (dm-10): Mounting V4 Filesystem Jul 31 15:50:08 donald kernel: XFS (dm-10): Ending clean mount
