Monday, June 1, 2009

Network Teaming/bonding on Linux

Network Interface Bonding

Interface Bonding is one of the simpler tasks which ensures high availability on the network. Bonding/Teaming is a way through which two or more physical NIC cards could work in tandem in order to ensure high availability from network perspective. It saves us from a network switch failure, network cable failure as well as NIC card failure. Ideal configuration for a high availability server should be:-

n/w switch A n/w switch B

NIC card 1 NIC card 2

SERVER

The only requirement is to have all of the participating slave NICs/Network switches in same subnet/network.


Doing it on Linux is even simpler!! Here are the steps to setup the bonding:-

1. Ensure that your version of OS is natively supporting interface bonding. By default, Redhat does provide bonding as a loadable module within the OS itself. If not, there are a lot of source rpms which could be build and loaded into the kernel.

locate bonding ## run updatedb if slocate database

the above mentioned command will list all the available bonding modules, alternatively you could also do modprobe and then rmmod.

2. Setup the network config files for all ensalved interfaces (i.e. which will form the bonded interface)

vi /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
BOOTPROTO=none
HWADDR=xx:xx:xx:xx:xx:xx
ONBOOT=yes
MASTER=bond0
SLAVE=yes
USERCTL=no


vi /etc/sysconfig/network-scripts/ifcfg-eth1
DEVICE=eth1
BOOTPROTO=none
HWADDR=yy:yy:yy:yy:yy:yy
ONBOOT=yes
MASTER=bond0
SLAVE=yes
USERCTL=no

Boot protocol for NIC has been chosen to None as we are not assigning the IP address to slave NIC cards. HWADDR should NOT be edited. MASTER should be chosen to bond0 or any alias name (even apple0 will work ;)). As you could see above, the MASTER is same for both eth0, eth1. In case you want to ensalve three physical NICs, you need to keep the MASTER same for all three. SLAVE has been chosen as yes as eth0, eth1 will be slaves and will work depending upon the availability/config/requirement.

vi /etc/sysconfig/network-scripts/ifcfg-bond0 ##this file has to be created as its a logical interface
DEVICE=bond0
BOOTPROTO=none
USERCTL=no
IPADDR=.....
NETMASK=.....
GATEWAY=.....
NETWORK=.....
BROADCAST=.....
ONBOOT=yes

All .... entries need to be replaced with the ones which you would set for this logical/bonded interface.

3. vi /etc/modprobe.conf

Add the following lines in modprobe.conf file soon after the slave NICs configuration:-

alias bond0 bonding
options bonding mode=active-backup miimon=100

mode can be active-backup (1), balance-rr (0) etc. Here I am using it in active-backup mode where only one interface will be used for communication till the time a failure is detected on it. I could also use balance-rr which would load balance the packets based on Round Robin algo (sequential). miimon is the link detection interval which checks for the failure.



bonded Interface always have the MAC address of the first slave nic. It could, however, be changed via ifconfig command. Once the bonded logical NIC comes up, the ifconfig will report all slave/bonded NIC with same MAC address:-

# ip link show

2: eth0: mtu 1500 qdisc pfifo_fast master bond0 qlen 1000
link/ether 00:21:5a:d3:eb:b0 brd ff:ff:ff:ff:ff:ff
3: eth1: mtu 1500 qdisc pfifo_fast master bond0 qlen 1000
link/ether 00:21:5a:d3:eb:b0 brd ff:ff:ff:ff:ff:ff
4: bond0: mtu 1500 qdisc noqueue
link/ether 00:21:5a:d3:eb:b0 brd ff:ff:ff:ff:ff:ff

# ifconfig -a
bond0 Link encap:Ethernet HWaddr 00:21:5A:D3:EB:B0
....
....

eth0 Link encap:Ethernet HWaddr 00:21:5A:D3:EB:B0
....
....

eth1 Link encap:Ethernet HWaddr 00:21:5A:D3:EB:B0
.....
.....

Check either your ifconfig script or /proc filesystem for actual MAC address:-


# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.2.4 (January 28, 2008)

Bonding Mode: load balancing (round-robin)
MII Status: up
MII Polling Interval (ms): 500
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: eth0
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:21:5a:d3:eb:b0

Slave Interface: eth1
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:21:5a:d3:eb:b1

8 comments:

Unknown said...

Very Good Blog. It is like IPMP in solaris. Thanks for sharing these wonderful information.
One suggestion in that document is after creating alias , i think we restart the network service inorder to avail bond0 .

Rahul Khare said...

That is a correct statement but just remember:-
1. to load the bonding module before restarting the network..this step is not required in those flavors which natively support bonding (e.g. Redhat). The ones for which you compile and build the drivers would need to know the module before it could bring up the bonded interface.
2. Also remember to make sure that you have the entry listed in modprobe.conf or modules.conf. Or else it would not be persistent upon reboot.

teebs said...

Rahul, I have a question regarding bonding in RHEL 5.3.

I have built a server with kickstart profile where it creates bond1 and bond2 as I reserved bond0 for later infiniband configurations.

Once the server is up and running I can see an unconfigured bond0 info in the "ifconfig -a" output. Nothing is there in the network-scripts or modprobe.conf files regarding bond0. I understand may be when I tried to create bond1 and bond2, it might have given bond0(as bond1 can't exist without bond0). My assumption....

The question I have is, can I remove bond0? If so how? I don't see any bond0 from network GUI.

Your help is greatly appreciated and thank you for your time.

Rahul Khare said...

bond0 appeared due to the default behaviour of bonding driver. Upon loading, it auto created the bond0, bond1 & bond2. This happened as soon as the modprobe.conf file loaded the driver. I think you might have used max_bonds option with bonding.

You could try to unplumb bond0 for unconfiguring it.

ifconfig down unplumb bond0

if it does not help, email me your modprobe.conf and ifconfig -a output.

teebs said...

Rahul, unplumb didn't work. Errors out as unknown host.

Output of ifconfig -a:
[root@Rac5 ~]# ifconfig -a
bond0 Link encap:Ethernet HWaddr 00:00:00:00:00:00
BROADCAST MASTER MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)

bond1 Link encap:Ethernet HWaddr 00:21:28:6A:F8:18
inet addr:192.168.30.35 Bcast:192.168.30.255 Mask:255.255.255.0
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:242661 errors:0 dropped:0 overruns:0 frame:0
TX packets:2505006 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:19640400 (18.7 MiB) TX bytes:150495980 (143.5 MiB)

bond2 Link encap:Ethernet HWaddr 00:21:28:6A:F8:19
inet addr:192.168.31.35 Bcast:192.168.31.255 Mask:255.255.255.0
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:510291 errors:0 dropped:0 overruns:0 frame:0
TX packets:2631997 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:192054666 (183.1 MiB) TX bytes:154898394 (147.7 MiB)

eth0 Link encap:Ethernet HWaddr 00:21:28:6A:F8:18
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:127113 errors:0 dropped:0 overruns:0 frame:0
TX packets:1258538 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:10308544 (9.8 MiB) TX bytes:75856421 (72.3 MiB)
Memory:fa760000-fa780000

eth1 Link encap:Ethernet HWaddr 00:21:28:6A:F8:19
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:394896 errors:0 dropped:0 overruns:0 frame:0
TX packets:1399926 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:182732416 (174.2 MiB) TX bytes:80974206 (77.2 MiB)
Memory:fa7e0000-fa800000

eth2 Link encap:Ethernet HWaddr 00:21:28:6A:F8:1A
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:115548 errors:0 dropped:0 overruns:0 frame:0
TX packets:1246470 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:9331856 (8.8 MiB) TX bytes:74640091 (71.1 MiB)
Memory:fa860000-fa880000

eth3 Link encap:Ethernet HWaddr 00:21:28:6A:F8:1B
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:115395 errors:0 dropped:0 overruns:0 frame:0
TX packets:1232071 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:9322250 (8.8 MiB) TX bytes:73924188 (70.4 MiB)
Memory:fa8e0000-fa900000

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:2902 errors:0 dropped:0 overruns:0 frame:0
TX packets:2902 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:4173727 (3.9 MiB) TX bytes:4173727 (3.9 MiB)

[root@Rac5 ~]#

[root@Rac5 ~]# cat /etc/modprobe.conf
alias eth0 igb
alias eth1 igb
alias eth2 igb
alias eth3 igb
alias scsi_hostadapter aacraid
alias scsi_hostadapter1 ahci
alias scsi_hostadapter2 qla2xxx
alias scsi_hostadapter3 usb-storage
alias bond1 bonding
options bond1 miimon=100 mode=balance-alb
alias bond2 bonding
options bond2 miimon=100 mode=balance-alb
alias net-pf-10 off
alias ipv6 off
[root@Rac5 ~]#

Thank you

Rahul Khare said...

looks ok to me...ifconfig -a in Linux displays all the possible interfaces, including those which are present in the system but are not in configured state. You dont need to worry about the existent bond0. It does not exist, try simply ifconfig command. It does not have any MAC address assigned to it,, suggesting that its simply there due to bonding module.

This is not an issue for you, trust me.

tara said...

Thanks Rahul, great.

One question i configured it and it worked perfectly. I have eth0 and eth3 bonded as active/backup. Eth0 is the link to the main switch and when its disconnect eth3 takes over. But when eth0 is restored, it doesn't take back over and i'd need to main link to be restored. Any ideas?

Rahul Khare said...

Tara,

Append primary=eth0 to "options bond0 miimon=100 mode=active-backup" string.

Rahul.