Thought of sharing this vital info and challenge which I faced while setting up Linux on Dell PowerEdge server. I am a firm admirer of HP hardware machines and there ease of deployment. The thing which I liked about Dell's motherboard was the presence of two onboard NIC cards compatible with Broadcomm redundancy module. Although not required but this could very well be a boon for NIC bonding for HA servers.
As long as the boot server setup and OS installation goes, things were looking good but the problem started to occur when the server was live. While trouble-shooting I came across issues which were unique to the hardware:-
We were experiencing intermittent loss of link and network connectivity that could not be restored. Nothing appeared on the NIC cards which indicated loss of packets or collisions.
Tackling the problem, I first went into check what hardware config was on the server.
# lspci | grep Ethernet
08:01.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5700 Gigabit Ethernet (rev 14)
08:02.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5700 Gigabit Ethernet (rev 14)
Indeed it was a 1000MB/s NIC but I doubt my switch would support that ;). So I manually set the auto-net to off and put it as 100MBPS full duplex.
To do the same, simply add this line to ifcfg-ethX file in sysconfig:-
ETHTOOL_OPTS="speed 100 duplex full autoneg off"
But even with this, the network packets were getting dropped. After a bit of googling :D, I figured out that the NetXtreme type NICs does not work well with the default tg3 driver in Redhat. So I decided to go with BCM drivers instead. Here is how to configure NICs with BCM instead of tg3 driver.
step 1:- Download the rpm bcm5700-7.1.22-1 and install it on the server.
step 2:- Edit the modules.conf file and change the enteries for NIC cards:-
alias eth0 bcm5700
alias eth1 bcm5700
options bcm5700 auto_flow_control=0,0 tx_flow_control=0,0 rx_flow_control=0,0
Load the modules correctly alongwith network services restart or just reboot the damn thing!!!
My take on this is that I feel its better to switch to BCM drivers ONLY if the NICs are of Nextreme type. I am not going into the details of whether tg3 is good or bcm. But in this case, BCM out-performed the tg3 driver for the older Dell box ;). I recently heard from one of my friend that even Nextreme is looking to go for tg3, not sure if its just a hoax.
Cheers!!!
Rahul.
Tuesday, November 25, 2008
Tuesday, November 18, 2008
Send attachments via mail command
At some point of our daily tasks, we come across a scenario where we would prefer to be able to send the attachments via mail command in Linux or Solaris. Here is a quick way to do that:-
# uuencode abc.txt abc.txt |mail -s "subject" email.address@domain.com
uuencode is a utility which is available in sharutils rpm for Linux or as a package in Solaris.
This is a quick/easy way in order for sending attachments. You might want to consider mutt, mpack etc. for advance options but again I like to keep things as simple as possible ;).
Enjoy!!
Rahul.
# uuencode abc.txt abc.txt |
uuencode is a utility which is available in sharutils rpm for Linux or as a package in Solaris.
This is a quick/easy way in order for sending attachments. You might want to consider mutt, mpack etc. for advance options but again I like to keep things as simple as possible ;).
Enjoy!!
Rahul.
Recovering EMC power pseudo device config
This is actually a continuation to my earlier post. There was an instance in which I ran into an issue with pseudo devices. Here is a shorter version of what happened:-
We manage an Oracle 10g RAC cluster on two Linux boxes which have ~30 EMC devices. Out of which about ~25 devices are shared between both servers** and remaining ~5 devices are different on both servers, which means that there are about ~35 LUNs from EMC storage. One bad day, one of the server was removed from the storage zone (dont wanna go into the details) and we were left with an up-hill task to recover that node.
**25 devices are used for Oracle data while 5 different LUNs are being used as internal filesystems on both servers.
We managed to get our zoning config back and thankfully, the internal hard-drive was still intact. As soon as the devices were re-presented from the storage, someone ran the HP utility to scan all new LUNs. As soon as the powerpath services were started, the device mapping config changed. Now we were left with a situation where the older /dev/emcpowera device was actually mapped to /dev/emcpowerc :(. This happened as EMC powerpath software assigned the pseudo device IDs based on the new device config (as recognized by OS) and without noticing the difference, they issued powermt config.
One way to recover from such a scenario would have been to copy the device config from the good node to this node. So we did, and got our config back.
But much to my dismay, apart from the 25 devices which were same on both nodes, the 5 different devices IDs were skipped by Powerpath.
So effectively, I had /dev/emcpowers on one node representing the 26th device and /dev/emcpowerx on another (notice that due to config copy, powerpath reserved emcpowers till emcpowerw for the device IDs of good server and presented a new device ID for the recovered node.)
If you are fortunate enough to have Powerpath version 3.0 or earlier, you could manually edit the emcpower.conf file to change the device IDs. But if you the newer versions of powerpath, then the work-around is not there.
Riding on my good luck, I decided to restore the config files emcp_devicesDB.idx & emcp_devicesDB.dat and voila..it worked. I was now back to where we were before the node failure.
Here is how its done:-
For EMCpowerpath version 3.0.6 or earlier:-
Edit the emcpower.conf file and change the device IDs:-
name="emcpoweri" parent="pseudo" instance=8 vidb=00000000000000000000000011********************************* vids=******************************* dev=0x8,0x20,0x8,0x40;
to
name="emcpowers" parent="pseudo" instance=8 vidb=00000000000000000000000011********************************* vids=******************************* dev=0x8,0x20,0x8,0x40;
Now unmount all the filesystems and stop the volume manager (if any) to release the pseudo devices.
Afterwards, restart the powerpath software andd cross-check the device names via powermt command.
For EMCpowerpath version 4.** or later:-
If you don't have emcpower.conf file, you must be having a latest version. For this, do the following:-
1. Recover the following files for older device config from last backup:-
/etc/emcp_devicesDB.idx
/etc/emcp_devicesDB.dat
Or copy it from the good node (provided you do not have any exclusive LUN on either of the server)
2. Unmount the filesystem and stop the volume manager.
3. Restart the powerpath services.
Unmounting of filesystem/stop of volume manager is required, so that you don't run into the problem where the pseudo device is unavailable and hence server health is in question.
Do let me know if you need any clarification...
We manage an Oracle 10g RAC cluster on two Linux boxes which have ~30 EMC devices. Out of which about ~25 devices are shared between both servers** and remaining ~5 devices are different on both servers, which means that there are about ~35 LUNs from EMC storage. One bad day, one of the server was removed from the storage zone (dont wanna go into the details) and we were left with an up-hill task to recover that node.
**25 devices are used for Oracle data while 5 different LUNs are being used as internal filesystems on both servers.
We managed to get our zoning config back and thankfully, the internal hard-drive was still intact. As soon as the devices were re-presented from the storage, someone ran the HP utility to scan all new LUNs. As soon as the powerpath services were started, the device mapping config changed. Now we were left with a situation where the older /dev/emcpowera device was actually mapped to /dev/emcpowerc :(. This happened as EMC powerpath software assigned the pseudo device IDs based on the new device config (as recognized by OS) and without noticing the difference, they issued powermt config.
One way to recover from such a scenario would have been to copy the device config from the good node to this node. So we did, and got our config back.
But much to my dismay, apart from the 25 devices which were same on both nodes, the 5 different devices IDs were skipped by Powerpath.
So effectively, I had /dev/emcpowers on one node representing the 26th device and /dev/emcpowerx on another (notice that due to config copy, powerpath reserved emcpowers till emcpowerw for the device IDs of good server and presented a new device ID for the recovered node.)
If you are fortunate enough to have Powerpath version 3.0 or earlier, you could manually edit the emcpower.conf file to change the device IDs. But if you the newer versions of powerpath, then the work-around is not there.
Riding on my good luck, I decided to restore the config files emcp_devicesDB.idx & emcp_devicesDB.dat and voila..it worked. I was now back to where we were before the node failure.
Here is how its done:-
For EMCpowerpath version 3.0.6 or earlier:-
Edit the emcpower.conf file and change the device IDs:-
name="emcpoweri" parent="pseudo" instance=8 vidb=00000000000000000000000011********************************* vids=******************************* dev=0x8,0x20,0x8,0x40;
to
name="emcpowers" parent="pseudo" instance=8 vidb=00000000000000000000000011********************************* vids=******************************* dev=0x8,0x20,0x8,0x40;
Now unmount all the filesystems and stop the volume manager (if any) to release the pseudo devices.
Afterwards, restart the powerpath software andd cross-check the device names via powermt command.
For EMCpowerpath version 4.** or later:-
If you don't have emcpower.conf file, you must be having a latest version. For this, do the following:-
1. Recover the following files for older device config from last backup:-
/etc/emcp_devicesDB.idx
/etc/emcp_devicesDB.dat
Or copy it from the good node (provided you do not have any exclusive LUN on either of the server)
2. Unmount the filesystem and stop the volume manager.
3. Restart the powerpath services.
Unmounting of filesystem/stop of volume manager is required, so that you don't run into the problem where the pseudo device is unavailable and hence server health is in question.
Do let me know if you need any clarification...
EMC Powerpath Software
I am having a lot of servers actually connected to various EMC storages like Symmetrix, Clariion. As with all storage softwares, the biggest challenge is to manage the multi-path devices and to ensure that devices remain on OS even in case of a Service processor failure or an HBA card failure.
As with all other storages like HP, SUN etc. EMC provides a software known as EMC Powerpath. This software allows one to map the OS devices to a pseudo block device without having to worry about the device paths.
For e.g. on a Linux system, HBA card 1 can give a device name as /dev/sda and the same device via HBA card 2 could have /dev/sdf or some other name. This setup can be really confusing if someone decides to use the devices on the basis of there SCSI device IDs. Lets say someone uses /dev/sda and due to power outage or service processor failure on SAN switch, the paths gets reversed. In that case,
/dev/sda will no longer be primary.
/dev/sdf will become primary.
But due to this path change, /etc/vfstab (Solaris) and /etc/fstab (Linux) configs would go foo-bar.
To save us from this hassel, EMC powerpath maps the OS block devices with the pseudo /dev/emcpower[a-z] devices.
This mapping can be checked using the command:-
powermt display dev=all
Powerpath software saves the device mapping info in a file /etc/opt/emcpower/emcpower.conf or /etc/emcp_devicesDB.idx & /etc/emcp_devicesDB.dat depending upon the versison of Powerpath software.
These mapped devices emcpower[a-z] can be partitioned & used for a filesystem (by running mkfs on them) or even as a rawdevice. It really boils down to how you wanna use them.
Always use emcpower pseudo devices in /etc/fstab or /etc/vfstab configurations or in /etc/sysconfig/rawdevices (For raw device mapping).
Rahul.
As with all other storages like HP, SUN etc. EMC provides a software known as EMC Powerpath. This software allows one to map the OS devices to a pseudo block device without having to worry about the device paths.
For e.g. on a Linux system, HBA card 1 can give a device name as /dev/sda and the same device via HBA card 2 could have /dev/sdf or some other name. This setup can be really confusing if someone decides to use the devices on the basis of there SCSI device IDs. Lets say someone uses /dev/sda and due to power outage or service processor failure on SAN switch, the paths gets reversed. In that case,
/dev/sda will no longer be primary.
/dev/sdf will become primary.
But due to this path change, /etc/vfstab (Solaris) and /etc/fstab (Linux) configs would go foo-bar.
To save us from this hassel, EMC powerpath maps the OS block devices with the pseudo /dev/emcpower[a-z] devices.
This mapping can be checked using the command:-
powermt display dev=all
Powerpath software saves the device mapping info in a file /etc/opt/emcpower/emcpower.conf or /etc/emcp_devicesDB.idx & /etc/emcp_devicesDB.dat depending upon the versison of Powerpath software.
These mapped devices emcpower[a-z] can be partitioned & used for a filesystem (by running mkfs on them) or even as a rawdevice. It really boils down to how you wanna use them.
Always use emcpower pseudo devices in /etc/fstab or /etc/vfstab configurations or in /etc/sysconfig/rawdevices (For raw device mapping).
Rahul.
RDesktop - Powered by Linux for Windows
I came across to this nice utility for accessing RDP (Remote Desktop for Windows) through Linux/Unix based systems through X windows.
rdesktop is a client which is capable of communicating with Windows Terminal Services, thus bypassing the slow "mstsc" services of Windows. I prefer to connect to some of my Windows servers through a desktop Linux (run-level 5 as usual :) ).
The syntax is pretty easy and quick:-
# rdesktop serverIP
and thats it!!
For downloading the software visit http://www.rdesktop.org/ and download the commpatible version based on your kernel version.
P.S. I am running 1.6.0 on 2.4.** kernel.
The installation is quite simple as usual for any opensource software.
. configure or ./configure
make
make install
And you are all set to access your windows server while sitting on your desk enjoying your coffee.
Found it to be quite useful.
If you have installed it on a server, well export your display to any Linux desktop or to your laptop and enjoy!! Also do not forget to put the process in background or use "disown -h" command or use nohup instead.
Enjoy!!
rdesktop is a client which is capable of communicating with Windows Terminal Services, thus bypassing the slow "mstsc" services of Windows. I prefer to connect to some of my Windows servers through a desktop Linux (run-level 5 as usual :) ).
The syntax is pretty easy and quick:-
# rdesktop serverIP
and thats it!!
For downloading the software visit http://www.rdesktop.org/ and download the commpatible version based on your kernel version.
P.S. I am running 1.6.0 on 2.4.** kernel.
The installation is quite simple as usual for any opensource software.
. configure or ./configure
make
make install
And you are all set to access your windows server while sitting on your desk enjoying your coffee.
Found it to be quite useful.
If you have installed it on a server, well export your display to any Linux desktop or to your laptop and enjoy!! Also do not forget to put the process in background or use "disown -h" command or use nohup instead.
Enjoy!!
Subscribe to:
Posts (Atom)
