Friday, January 30, 2009

How to remove LUN devices from Veritas on Solaris and re-use them on a different server

Ever wanted to get rid of some of the old veritas disk groups from a Sun Solaris box in order to allocate that space onto a new server? Here is a short summary of steps which could be really helpful. As we all have faced these scenarios, I thought of documenting it.

To remove about 10 recently allocated storage LUN devices (out of available 18 devices in the veritas disk group) from the server and put them onto a new box.
Summary of work required

1. First of all, unmount all the new filesystems created after the recent addition of 10 devices in the disk group.

2. Stop the new volume within the disk group (If you miss it, it might come back to haunt you when you deport the volume).
vxvol -g datadg stopall ## in case you are removing the whole disk group
vxvol -g datadg stop {volume} ## in case you are removing only some volumes from the disk group.

3. Remove the new volume for freeing the disks.
vxassist remove volume {volume} ## Step for removing some volumes out of the disk group
If you want to remove the complete disk group you can use the following:-
vxdg deport datadg
vxdg destroy datadg

4. Once the volumes are removed, check if the associated LUN disk drives are empty.
vxdg free
It should report the OFFSET to be 0 for the devices, if not DO NOT remove that drive, rather try to shrink the volume. Since the new drives were added latest into the disk group, the resizing will free those up first.

5. If 4 holds true for you (for me it did :D) then go ahead and mark the disk drives offline.
Use vxdiskadm command for the same, its CLI is pretty nice.

6. Once the drives are marked offline, you can safely login to the Storage and unmap the associated drives.
Ideally, while using Sun storage, the devices appear in the order to generation/detection. Unlike Linux, where you can use devlabel, Solaris offers controller based device naming conventions. But the only issue is that this convention is not visible on OS. For Sun storage, all LUN devices have a WWN number, but the server only prefixes the HBA card's WWN number. Hence the order of adding new LUNs should be done one by one and not in batch ;).

7. After I unmasked the drives, I used format command to check if OS still sees those LUNs. Afterwards, follow the following steps to remove the device config from Veritas:-
vxdisk rm {lun}
devfsadm -C
I do not like to use cfgadm to unconfigure the path, this is a step which is required if format still shows the removed LUN devices
cfgadm -c unconfigure cx::dsk/cxtydz

8. Now the devices are out of Solaris/Veritas control. Run vxdctl enable to cross-check if the devices show up again.

Well, the hard/tricky part is over. Go ahead and destroy the unmapped LUNs and re-create the device on storage. Afterwards, map it to new server where it would be used finally. Sounds easy isn't it? Well if you are lucky enough, just like me, you will run into another weird stuff.

When I presented the newly created 300 GB device (6X50 GB old LUNs), the device showed up with veritas plexes/pre configuration on new server. In another words, the device data was still intact :(. And to my surprise it reported 50 GB device rather then the actual 300 GB.

So I had to finally use the good old "dd" companion for flushing out the first 128 bits on the storage LUN. This was enough for Veritas volume manager to treat the device as a new one.

If you run into this issue, better remove the LUN device, re-create it on storage, remove the config from solaris kernel and then add it to the OS.

Hope it helps,