Connect a Sun Solaris SPARC box to HP storage

My experience with SUN Sparc servers has been pretty good except for the fact that Sun charges $$ for anything which does not qualify as support and hence does not fall under master support contract.

I was once setting up a SPARC box V4** series and wanted to connect it to Sun Storage. I was told by Sun that it would cost us to connect the server to storage, as it was a new addition and not an existing support, fair enough. So instead we decided to connect it to another HP storage frame in our data center. Here are the detailed activities, which I did in order to connect the SUN box to HP frame.

1. Shutdown the box.
2. Insert the HBA card into the box.
3. Reboot the server.
4. Install the HBA packages and SUN packages for Storage management (SUNWqlc, SUNWqlcx, SUNWsmapi, SUNWssadx, SUNWssaop, OB2-CC, OB2-CORE, OB2-DA) available online and are opensource.
5. Install SFS_base_packages.S8 for Solaris 8 storage management.
6. After installing the packages, the server needs to be rebooted to load the libraries for HBA card detection.

7. Once the server is up, run the following commands to find out the path to HBA:-

bash-2.03# luxadm -e port

Found path to 5 HBA ports

/devices/pci@8,700000/SUNW,qlc@3/fp@0,0:devctl NOT CONNECTED
/devices/pci@8,700000/SUNW,qlc@3,1/fp@0,0:devctl NOT CONNECTED
/devices/pci@8,600000/SUNW,qlc@1/fp@0,0:devctl CONNECTED***
/devices/pci@8,600000/SUNW,qlc@1,1/fp@0,0:devctl NOT CONNECTED
/devices/pci@9,600000/SUNW,qlc@2/fp@0,0:devctl CONNECTED***

*** Indicates the HBA device ports available on the system, rest are the open PCI slots with no controller card.

Now confirm that the CONNECTED paths are HBA cards:-

# luxadm -e dump_map /devices/pci@8,600000/SUNW,qlc@1/fp@0,0:devctl
Pos Port_ID Hard_Addr Port WWN Node WWN Type
0 10e00 0 50001fe15000f3d8 50001fe15000f3d0 0xc (Array controller device)
1 10000 0 210000e08b07d5b4 200000e08b07d5b4 0x1f (Unknown Type,Host Bus Adapter)

Now check the dynamic configured devices

# cfgadm -al
Ap_Id Type Receptacle Occupant Condition
c0 scsi-bus connected configured unknown
c0::dsk/c0t0d0 CD-ROM connected configured unknown
c1 fc-private connected configured unknown
c1::500000e01129ff41 disk connected configured unknown
c1::500000e0112a0281 disk connected configured unknown
c2 fc connected unconfigured unknown
c3 fc connected unconfigured unknown
c4 fc-fabric connected configured unknown
c4::50001fe15000f3d8 array-ctrl connected unconfigured unknown
c5 fc connected unconfigured unknown

If the cfgadm command shows the HBA on controller 4 as unconfigured, connect the HBA to SAN through fiber cable, and run the command "cfgadm -o force_update -c configure c5" to configure the HBA channel. The good thing is that at this point you don’t have to actually present the vdisk, just ensure that the cable is in the HBA card so that link is UP..

# cfgadm -al
Ap_Id Type Receptacle Occupant Condition
c0 scsi-bus connected configured unknown
c0::dsk/c0t0d0 CD-ROM connected configured unknown
c1 fc-private connected configured unknown
c1::500000e01129ff41 disk connected configured unknown
c1::500000e0112a0281 disk connected configured unknown
c2 fc connected unconfigured unknown
c3 fc connected unconfigured unknown
c4 fc-fabric connected configured unknown
c4::50001fe15000f3d8 array-ctrl connected configured unknown
c5 fc connected unconfigured unknown

Important commands for HBA detection and configuration of LUNs are:-

luxadm -e port
luxadm -e dump_map
cfgadm -al
cfgadm -al -o show_FCP_dev
cfgadm -c configure c3
cfgadm -o force_update -c configure c3 ###Use this if the normal configure does not work.
luxadm probe
luxadm qlgc


8. Once the HBA is configured, create a vdisk on HP storage and assign it to the HBA card (using the WWN number).
9. Since my HP storage is connected to couple of switches, the switch needs to be configured to allocate the Path to HBA card.
10. After configuring the switch, enable the config on it (Note:- Never reboot the SAN switch, this is a common mistake and could lead to path revertion)
11. Once the config is enabled, the switch would be able to send out the LUNs to HBA card on the server.
12. Make sure that the Server Vdisk on the storage is set to "Path A/failover".
13. Begin detection of the disk drives on the server (following is the order of commands):-

luxadm probe ## initial probe for disk drives
luxadm qlgc ## initial probe of devices on Qlogic driver (HBA card vendor)
cfgadm -al ## Would display the new devices on the server

Sometime, cfgadm is unable to configure the controller so force it:-

"cfgadm -o force_update -c configure c3"

Once the controller is configured, the devices needs to be detected on the server:-

cfgadm -al -o show_FCP_dev ## this command will show all Fiber Channel Port devices coming from SAN.

14. Run the “format” command to see all the disk drives on the server.


Ideally, put in two HBA cards and then configure them. Afterwards, you could use the Volume manager software (like Veritas, it is free with Solaris 8 :D ). The LUNs multi-pathing can be done via DMP module of Veritas or any volume manager software.

Tryst with Dell PowerEdge servers (BCM or TG3 network Driver)

Thought of sharing this vital info and challenge which I faced while setting up Linux on Dell PowerEdge server. I am a firm admirer of HP hardware machines and there ease of deployment. The thing which I liked about Dell's motherboard was the presence of two onboard NIC cards compatible with Broadcomm redundancy module. Although not required but this could very well be a boon for NIC bonding for HA servers.

As long as the boot server setup and OS installation goes, things were looking good but the problem started to occur when the server was live. While trouble-shooting I came across issues which were unique to the hardware:-

We were experiencing intermittent loss of link and network connectivity that could not be restored. Nothing appeared on the NIC cards which indicated loss of packets or collisions.

Tackling the problem, I first went into check what hardware config was on the server.

# lspci | grep Ethernet
08:01.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5700 Gigabit Ethernet (rev 14)
08:02.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5700 Gigabit Ethernet (rev 14)

Indeed it was a 1000MB/s NIC but I doubt my switch would support that ;). So I manually set the auto-net to off and put it as 100MBPS full duplex.

To do the same, simply add this line to ifcfg-ethX file in sysconfig:-
ETHTOOL_OPTS="speed 100 duplex full autoneg off"

But even with this, the network packets were getting dropped. After a bit of googling :D, I figured out that the NetXtreme type NICs does not work well with the default tg3 driver in Redhat. So I decided to go with BCM drivers instead. Here is how to configure NICs with BCM instead of tg3 driver.

step 1:- Download the rpm bcm5700-7.1.22-1 and install it on the server.
step 2:- Edit the modules.conf file and change the enteries for NIC cards:-

alias eth0 bcm5700
alias eth1 bcm5700
options bcm5700 auto_flow_control=0,0 tx_flow_control=0,0 rx_flow_control=0,0

Load the modules correctly alongwith network services restart or just reboot the damn thing!!!

My take on this is that I feel its better to switch to BCM drivers ONLY if the NICs are of Nextreme type. I am not going into the details of whether tg3 is good or bcm. But in this case, BCM out-performed the tg3 driver for the older Dell box ;). I recently heard from one of my friend that even Nextreme is looking to go for tg3, not sure if its just a hoax.

Cheers!!!
Rahul.

Send attachments via mail command

At some point of our daily tasks, we come across a scenario where we would prefer to be able to send the attachments via mail command in Linux or Solaris. Here is a quick way to do that:-

# uuencode abc.txt abc.txt | mail -s "subject" email.address@domain.com

uuencode is a utility which is available in sharutils rpm for Linux or as a package in Solaris.

This is a quick/easy way in order for sending attachments. You might want to consider mutt, mpack etc. for advance options but again I like to keep things as simple as possible ;).

Enjoy!!
Rahul.

Recovering EMC power pseudo device config

This is actually a continuation to my earlier post. There was an instance in which I ran into an issue with pseudo devices. Here is a shorter version of what happened:-

We manage an Oracle 10g RAC cluster on two Linux boxes which have ~30 EMC devices. Out of which about ~25 devices are shared between both servers** and remaining ~5 devices are different on both servers, which means that there are about ~35 LUNs from EMC storage. One bad day, one of the server was removed from the storage zone (dont wanna go into the details) and we were left with an up-hill task to recover that node.

**25 devices are used for Oracle data while 5 different LUNs are being used as internal filesystems on both servers.

We managed to get our zoning config back and thankfully, the internal hard-drive was still intact. As soon as the devices were re-presented from the storage, someone ran the HP utility to scan all new LUNs. As soon as the powerpath services were started, the device mapping config changed. Now we were left with a situation where the older /dev/emcpowera device was actually mapped to /dev/emcpowerc :(. This happened as EMC powerpath software assigned the pseudo device IDs based on the new device config (as recognized by OS) and without noticing the difference, they issued powermt config.

One way to recover from such a scenario would have been to copy the device config from the good node to this node. So we did, and got our config back.

But much to my dismay, apart from the 25 devices which were same on both nodes, the 5 different devices IDs were skipped by Powerpath.

So effectively, I had /dev/emcpowers on one node representing the 26th device and /dev/emcpowerx on another (notice that due to config copy, powerpath reserved emcpowers till emcpowerw for the device IDs of good server and presented a new device ID for the recovered node.)

If you are fortunate enough to have Powerpath version 3.0 or earlier, you could manually edit the emcpower.conf file to change the device IDs. But if you the newer versions of powerpath, then the work-around is not there.

Riding on my good luck, I decided to restore the config files emcp_devicesDB.idx & emcp_devicesDB.dat and voila..it worked. I was now back to where we were before the node failure.



Here is how its done:-

For EMCpowerpath version 3.0.6 or earlier:-

Edit the emcpower.conf file and change the device IDs:-

name="emcpoweri" parent="pseudo" instance=8 vidb=00000000000000000000000011********************************* vids=******************************* dev=0x8,0x20,0x8,0x40;

to

name="emcpowers" parent="pseudo" instance=8 vidb=00000000000000000000000011********************************* vids=******************************* dev=0x8,0x20,0x8,0x40;

Now unmount all the filesystems and stop the volume manager (if any) to release the pseudo devices.

Afterwards, restart the powerpath software andd cross-check the device names via powermt command.




For EMCpowerpath version 4.** or later:-

If you don't have emcpower.conf file, you must be having a latest version. For this, do the following:-

1. Recover the following files for older device config from last backup:-
/etc/emcp_devicesDB.idx
/etc/emcp_devicesDB.dat

Or copy it from the good node (provided you do not have any exclusive LUN on either of the server)
2. Unmount the filesystem and stop the volume manager.
3. Restart the powerpath services.


Unmounting of filesystem/stop of volume manager is required, so that you don't run into the problem where the pseudo device is unavailable and hence server health is in question.

Do let me know if you need any clarification...

EMC Powerpath Software

I am having a lot of servers actually connected to various EMC storages like Symmetrix, Clariion. As with all storage softwares, the biggest challenge is to manage the multi-path devices and to ensure that devices remain on OS even in case of a Service processor failure or an HBA card failure.

As with all other storages like HP, SUN etc. EMC provides a software known as EMC Powerpath. This software allows one to map the OS devices to a pseudo block device without having to worry about the device paths.

For e.g. on a Linux system, HBA card 1 can give a device name as /dev/sda and the same device via HBA card 2 could have /dev/sdf or some other name. This setup can be really confusing if someone decides to use the devices on the basis of there SCSI device IDs. Lets say someone uses /dev/sda and due to power outage or service processor failure on SAN switch, the paths gets reversed. In that case,

/dev/sda will no longer be primary.
/dev/sdf will become primary.

But due to this path change, /etc/vfstab (Solaris) and /etc/fstab (Linux) configs would go foo-bar.

To save us from this hassel, EMC powerpath maps the OS block devices with the pseudo /dev/emcpower[a-z] devices.

This mapping can be checked using the command:-

powermt display dev=all


Powerpath software saves the device mapping info in a file /etc/opt/emcpower/emcpower.conf or /etc/emcp_devicesDB.idx & /etc/emcp_devicesDB.dat depending upon the versison of Powerpath software.

These mapped devices emcpower[a-z] can be partitioned & used for a filesystem (by running mkfs on them) or even as a rawdevice. It really boils down to how you wanna use them.

Always use emcpower pseudo devices in /etc/fstab or /etc/vfstab configurations or in /etc/sysconfig/rawdevices (For raw device mapping).

Rahul.

RDesktop - Powered by Linux for Windows

I came across to this nice utility for accessing RDP (Remote Desktop for Windows) through Linux/Unix based systems through X windows.

rdesktop is a client which is capable of communicating with Windows Terminal Services, thus bypassing the slow "mstsc" services of Windows. I prefer to connect to some of my Windows servers through a desktop Linux (run-level 5 as usual :) ).

The syntax is pretty easy and quick:-

# rdesktop serverIP

and thats it!!

For downloading the software visit http://www.rdesktop.org/ and download the commpatible version based on your kernel version.

P.S. I am running 1.6.0 on 2.4.** kernel.

The installation is quite simple as usual for any opensource software.

. configure or ./configure
make
make install

And you are all set to access your windows server while sitting on your desk enjoying your coffee.

Found it to be quite useful.

If you have installed it on a server, well export your display to any Linux desktop or to your laptop and enjoy!! Also do not forget to put the process in background or use "disown -h" command or use nohup instead.

Enjoy!!