Duplicate UUID error on cloned volumes

Problem Scenario

Wanted to clone a pure storage volume on XFS filesystem for testing purpose on the same server.

  1. Took FlashRecover snapshot of the volume.  
  2. Instantiated the snapshot by copying them into target volume.
  3. Attached the target volume to the same host and updated multipath.conf with the alias information.
  4. Tried to mount the partition from the target volume on the host but was not successful.

Error Messages on the host

[root@donald ~]# df -hT /x01
Filesystem           Type  Size  Used  Avail  Use%  Mounted on
/dev/mapper/fs_xfs1  xfs   2.0G   33M   2.0G    2%  /x01

[root@donald ~]# ssh pureuser@10.21.34.3
pureuser@10.21.34.3's password:

pureuser@pure-m50-2-b12-35-ct1> purevol snap --suffix snap test_xfs
Name             Size Source    Created                 Serial
test_xfs.snap      2G test_xfs  2016-07-31 15:38:09 PDT 8C44AF151613837200011BF9

pureuser@pure-m50-2-b12-35-ct1> purevol copy test_xfs.snap cloned_test_xfs
Name             Size Source    Created                 Serial
cloned_test_xfs    2G test_xfs  2016-07-31 15:38:46 PDT 8C44AF151613837200011BEC

pureuser@pure-m50-2-b12-35-ct1> purehost connect --vol cloned_test_xfs donald
Name      Vol               LUN
donald    cloned_test_xfs   3

pureuser@pure-m50-2-b12-35-ct1> exit
logout
Connection to 10.21.34.3 closed.

[root@donald ~]# vi /etc/multipath.conf
. . .
 multipath {
   wwid 3624a93708c44af151613837200011be8
   alias fs_xfs
 }
 multipath {
   wwid 3624a93708c44af151613837200011bec
   alias fs_cloned_xfs
 }

[root@donald ~]# systemctl restart multipathd.service

[root@donald ~]# rescan-scsi-bus.sh
Scanning SCSI subsystem for new devices
Scanning host 0 for all SCSI target IDs, all LUNs
 Scanning for device 0 0 2 10 ...
OLD: Host: scsi0 Channel: 00 Id: 02 Lun: 10
 Vendor: PURE Model: FlashArray Rev: 451
 Type: Direct-Access ANSI SCSI revision: 06
. . . .
16 new or changed device(s) found.
. . . .
0 remapped or resized device(s) found.
0 device(s) removed.

[root@donald ~]# ls -ltr /dev/mapper/fs*xfs*
lrwxrwxrwx 1 root root 7 Jul 31 15:41 /dev/mapper/fs_xfs -> ../dm-2
lrwxrwxrwx 1 root root 7 Jul 31 15:41 /dev/mapper/fs_cloned_xfs -> ../dm-5
lrwxrwxrwx 1 root root 7 Jul 31 15:41 /dev/mapper/fs_xfs1 -> ../dm-8
lrwxrwxrwx 1 root root 8 Jul 31 15:41 /dev/mapper/fs_cloned_xfs1 -> ../dm-10

[root@donald ~]# mount -t xfs -o noatime /dev/mapper/fs_cloned_xfs1 /x02
mount: wrong fs type, bad option, bad superblock on /dev/mapper/fs_cloned_xfs1,
 missing codepage or helper program, or other error

 In some cases useful info is found in syslog - try
 dmesg | tail or so.

Reason for the Error

Cloning a Pure FlashArray volume and mounting the filesystem of the cloned volume on the same Linux host as the source volume will throw “duplicate UUID” error or similar errors.  The cloning of the volume reflects everything from the source at the point of time of the snapshot which also includes the UUID of the partitions within the volume.  The UUID is the unique identifier pointing to the partition and hence it is not unique anymore with the cloned volume on the host.

This is generally not an issue when the cloned volume is attached to different host than that of the source volume.  

For EXT4 filesystems, even though /var/log/messages shows error about device appeared twice with different sysfs paths, it doesn’t throw the error when mounting the filesystem from the cloned volume.

Solution

The UUID of the partition should be changed before it is mounted.  The commands vary based on the filesystem type (EXT4 vs XFS) but the idea is to change the UUID of the partition and mount.  Run xfs_repair first on the partition, followed by xfs_admin to generate new UUID.

root@donald ~]# xfs_repair /dev/mapper/fs_cloned_xfs1
Phase 1 - find and verify superblock...
Phase 2 - using internal log
 - zero log...
 - scan filesystem freespace and inode maps...
 - found root inode chunk
Phase 3 - for each AG...
 - scan and clear agi unlinked lists...
 - process known inodes and perform inode discovery...
 - agno = 0
 - agno = 1
 - agno = 2
 - agno = 3
 - agno = 4
 - agno = 5
 - agno = 6
 - agno = 7
 - process newly discovered inodes...
Phase 4 - check for duplicate blocks...
 - setting up duplicate extent list...
 - check for inodes claiming duplicate blocks...
 - agno = 0
 - agno = 1
 - agno = 2
 - agno = 5
 - agno = 3
 - agno = 6
 - agno = 4
 - agno = 7
Phase 5 - rebuild AG headers and trees...
 - reset superblock...
Phase 6 - check inode connectivity...
 - resetting contents of realtime bitmap and summary inodes
 - traversing filesystem ...
 - traversal finished ...
 - moving disconnected inodes to lost+found ...
Phase 7 - verify and correct link counts...
done

[root@donald ~]# xfs_admin -U generate /dev/mapper/fs_cloned_xfs1
Clearing log and setting UUID
writing all SBs
new UUID = 2f756348-e144-4569-b616-c2837d4ee944

[root@donald ~]# mount -t xfs -o noatime /dev/mapper/fs_cloned_xfs1 /x02

[root@donald ~]# df -h |grep x0
/dev/mapper/fs_xfs1          2.0G  33M  2.0G   2% /x01
/dev/mapper/fs_cloned_xfs1   2.0G  33M  2.0G   2% /x02

Excerpts from /var/log/messages

 
Jul 31 15:41:35 donald kernel: sd 0:0:4:3: [sdf] Attached SCSI disk
Jul 31 15:41:35 donald systemd: Device dev-disk-by\x2did-scsi\x2d3624a93708c44af151613837200011be8.device appeared twice with different sysfs paths /sys/devices/pci0000:00/0000:00:02.0/0000:02:00.0/0000:03:00.0/0000:04:00.0/0000:05:01.0/0000:07:00.0/host0/rport-0:0-9/target0:0:3/0:0:3:1/block/sda and /sys/devices/pci0000:00/0000:00:02.0/0000:02:00.0/0000:03:00.0/0000:04:00.0/0000:05:01.0/0000:07:00.0/host0/rport-0:0-10/target0:0:4/0:0:4:1/block/sdd
Jul 31 15:41:35 donald systemd: Device dev-disk-by\x2did-wwn\x2d0x624a93708c44af151613837200011be8.device appeared twice with different sysfs paths /sys/devices/pci0000:00/0000:00:02.0/0000:02:00.0/0000:03:00.0/0000:04:00.0/0000:05:01.0/0000:07:00.0/host0/rport-0:0-9/target0:0:3/0:0:3:1/block/sda and /sys/devices/pci0000:00/0000:00:02.0/0000:02:00.0/0000:03:00.0/0000:04:00.0/0000:05:01.0/0000:07:00.0/host0/rport-0:0-10/target0:0:4/0:0:4:1/block/sdd
Jul 31 15:41:35 donald multipathd: sdd: add path (uevent)
Jul 31 15:41:35 donald systemd-udevd: error: /dev/sdd1: No such file or directory
Jul 31 15:41:35 donald systemd-udevd: inotify_add_watch(7, /dev/sdd1, 10) failed: No such file or directory
Jul 31 15:41:35 donald multipathd: fs_xfs: load table [0 4194304 multipath 0 0 1 1 queue-length 0 2 1 8:0 1 8:48 1]
Jul 31 15:41:35 donald multipathd: sdd [8:48]: path added to devmap fs_xfs
Jul 31 15:41:35 donald multipathd: sdc: add path (uevent)
Jul 31 15:41:35 donald systemd-udevd: error: /dev/sdc1: No such file or directory
Jul 31 15:41:35 donald systemd-udevd: inotify_add_watch(7, /dev/sdc1, 10) failed: No such file or directory
Jul 31 15:41:35 donald multipathd: fs_cloned_xfs: load table [0 4194304 multipath 0 0 1 1 queue-length 0 1 1 8:32 1]
Jul 31 15:41:35 donald multipathd: fs_cloned_xfs: event checker started
Jul 31 15:41:35 donald multipathd: sdc [8:32]: path added to devmap fs_cloned_xfs
Jul 31 15:41:35 donald systemd: Device dev-disk-by\x2duuid-cd8db41d\x2dccb4\x2d4b1b\x2d8c42\x2d28241e1485ce.device appeared twice with different sysfs paths /sys/devices/virtual/block/dm-8 and /sys/devices/virtual/block/dm-10
. . . .
. . . .
Jul 31 15:45:13 donald kernel: XFS (dm-8): Mounting V4 Filesystem
Jul 31 15:45:13 donald kernel: XFS (dm-8): Ending clean mount
Jul 31 15:45:23 donald kernel: XFS (dm-10): Filesystem has duplicate UUID cd8db41d-ccb4-4b1b-8c42-28241e1485ce - can't mount
. . . .
. . . .
Jul 31 15:50:08 donald kernel: XFS (dm-10): Mounting V4 Filesystem
Jul 31 15:50:08 donald kernel: XFS (dm-10): Ending clean mount
Like it? Share ...Tweet about this on TwitterShare on LinkedIn

Leave a Reply

Your email address will not be published. Required fields are marked *