Ultra Monkey- Heartbeat

Linux HA Solution

This one took a touch longer than we anticipated. Though the configurations were set, up and running, testing was a crucial aspect in implementing this H.A solution. Thanks to Nikhil for completing the ultramonkey project successfully. If you are looking to setup a high availability cluster using open source software, you can contact our team who has the skill and experience to implement this in any cluster.

Ultra Monkey is a project to create load balanced and highly available network services. For example a cluster of web servers that appear as a single web server to end-users. The service may be for end-users across the world connected via the internet, or for enterprise users connected via an intranet. Ultra Monkey makes use of the Linux operating system to provide a flexible solution that can be tailored to a wide range of needs. From small clusters of only two nodes to large systems serving thousands of connections per second. The advantage of using a load balancer compared to using round robin DNS is that it takes care of the load on the web server nodes and tries to direct requests to the node with less load, and it also takes care of connections/sessions. Many web applications (e.g. forum software, shopping carts, etc.) make use of sessions, and if you are in a session on Apache node 1, you would lose that session if suddenly node 2 served your requests. In addition to that, if one of the Apache nodes goes down, the load balancer realizes that and directs all incoming requests to the remaining node which would not be possible with round robin DNS.

Configuring UltraMonkey in Ubuntu 9.10

For this setup, we need four nodes (two Apache nodes and two load balancer nodes) and five IP addresses: one for each node and one virtual IP address that will be shared by the load balancer nodes and used for incoming HTTP requests.

* Apache node 1: apachenode1.com (webserver1) – IP address: 192.168.1.101; Apache document

root: /var/www

* Apache node 2: apacheode2.com (webserver2) – IP address: 192.168.1.102; Apache document root:

/var/www

* Load Balancer node 1: loadb1.com (loadb1) – IP address: 192.168.1.103

* Load Balancer node 2: loadb2.com (loadb2) – IP address: 192.168.1.104

* Virtual IP Address: 192.168.1.105 (used for incoming requests)

1 Enable IPVS On The Load Balancers

First we must enable IPVS on our load balancers. IPVS (IP Virtual Server) implements transport-layer load balancing inside the Linux kernel, so called Layer-4 switching

loadb1/loadb2:

echo ip_vs_dh >> /etc/modules

echo ip_vs_ftp >> /etc/modules

echo ip_vs >> /etc/modules

echo ip_vs_lblc >> /etc/modules

echo ip_vs_lblcr >> /etc/modules

echo ip_vs_lc >> /etc/modules

echo ip_vs_nq >> /etc/modules

echo ip_vs_rr >> /etc/modules

echo ip_vs_sed >> /etc/modules

echo ip_vs_sh >> /etc/modules

echo ip_vs_wlc >> /etc/modules

echo ip_vs_wrr >> /etc/modules

Then we do this:

loadb1/loadb2:

modprobe ip_vs_dh

modprobe ip_vs_ftp

modprobe ip_vs

modprobe ip_vs_lblc

modprobe ip_vs_lblcr

modprobe ip_vs_lc

modprobe ip_vs_nq

modprobe ip_vs_rr

modprobe ip_vs_sed

modprobe ip_vs_sh

modprobe ip_vs_wlc

modprobe ip_vs_wrr

If you get errors, then most probably your kernel wasn’t compiled with IPVS support, and you need to compile a new kernel with IPVS support (or install a kernel image with IPVS support) now.

2 Install Ultra Monkey On The Load Balancers

Loadb1/loadb2:

To install Ultra Monkey, we must edit /etc/apt/sources.list now and add these two lines :

vi /etc/apt/sources.list

deb http://www.ultramonkey.org/download/3/ sarge main

deb-src http://www.ultramonkey.org/download/3 sarge main

apt-get update

If you encounter update error due to the key. Installing the key with the following steps will fix the update error :

vi /home/key.asc

—–BEGIN PGP PUBLIC KEY BLOCK—–

Version: SKS 1.1.0

mQGiBEOBaH4RBACmZGWUpGxm6fOtwvrlOu38+cSh6iwdr9KMeNc9+qna8Ena3sEqV2FX9qxj

aALm/VnrfbtUu823VpjpiNy7rMjvOl17omDS8v4S6HSBRFuOFxmIoPi4BLDPlTWhfTYZJ3rH

Gx/2J3JFjvzkVoM1Htq1jbeZzzqPaHAvGJrnIY+6DwCg7aCaaKfOQzwaVeTq1jhu4zH0MMsD

/00cd61uRttV97JBGBDx4Zwp5CBzlnra24kZkHf43iQiexI7JzOx0xvqmbAhz2Aei3UcQXM7

c90EaA6FgFJplaRFUcwYJkdxu3+quK0R1EmKmDA50ugiUzu8qt30ODBoJav7ZM8/pq4DvcrT

mHLBK/hlKOcAhAVrAqwTztynILPdA/0YRs8GYGIqrAEKfX4/8mhBbCCrD6yJViohbSJ3lRTD

jKnMI3vNj/+s6eeT9B1vF8CyEuzZufmEBXL329CnhDRNWaVbTgpBl/GLI1FSXTiDSnDGdReP

mF7h5HHlL0IFwWTYl+oiXz5RF2hAOUALIxfX+RQTU6mH/Raz59VH2j7pW7QfU2ltb24gSG9y

bWFuIDxob3Jtc0BkZWJpYW4ub3JnPohGBBARAgAGBQJDgW2HAAoJENUjpuZgBiiEx2AAn0dk

OefzVqNYB+CmQgOgRjS5A/fnAJsHVdO3i/8NckSuQeJ9z/MyO6iCE4hGBBARAgAGBQJDgW+R

AAoJEBcFOQ7mbJuwAGQAoJDv1/chu75aQOIxK5DIQEZ09jRVAJwJtwDyqgdUnd1AEc//OrNZ

1kIuwohGBBARAgAGBQJDgXq1AAoJEFPlmVtRVTMKiCYAn1KlsPSPSdWFeAiHf3Qm54gXH9Yp

AJ9DIJ4jy1Z9IXfp9i1fVnwKHicZxohGBBARAgAGBQJIQuQaAAoJEPaNV/uq5zQnFQEAn0cZ

cG/XW3nZn7XJkmTQ7BhCct7wAKCFofIDm0ecW2LIku5W3ByRpwpTYIhGBBARAgAGBQJJt0yC

AAoJEBjYpOLQ7bZNKBsAn15fxX5vOVPxcCjG0MTkKq1KL+6uAJ9NBuT48l0lZrHoodrEK7HJ

yBATAIhGBBARAgAGBQJJt2+7AAoJEGykGndDuNbIWVsAnjgyEhiDo5Mje4EbPoX7BYQlYq+5

AJ47SpquAYY6zLB6u2lmzQ8DaHbm4IhMBBMRAgAMBQJDlAzMBYMSuV6yAAoJEAgFz2XePm2T

uMQAn1fWqSGo1EMJiglmWV+AQypVaz09AKCUH6rBS7sbPboBV2Xy+1I1oZdJMIhmBBMRAgAm

BQJDgWi5AhsDBQkSzAMABgsJCAcDAgQVAggDBBYCAwECHgECF4AACgkQA8ACPgVBDpew7wCg

273GZJS2311rS3/3C8VJa8cM9LwAn3QzO5DFhKN/3XOGI99vxaK+x0zCtCFTaW1vbiBIb3Jt

YW4gPGhvcm1zQHZlcmdlLm5ldC5hdT6IRgQQEQIABgUCQ4FthQAKCRDVI6bmYAYohM+7AJ4n

3bKIpEaejv/1pbHKLL11fse1DQCfXjFRojcLeRIXXkwvSWBDY/xpGvCIRgQQEQIABgUCQ4Fv

jQAKCRAXBTkO5mybsAjoAJ41mgdVAz23U51NJ8cgpWp+KaWYWwCfdE1q/K/DNbx+Pix8kkOL

NhQ2T6qIRgQQEQIABgUCQ4F6rwAKCRBT5ZlbUVUzCsoiAJ9GZNyA1XDcCdotS1qTFlp33tnA

OQCfWYzBttAL7etuutb4CDJqPwOh4j+IRgQQEQIABgUCSELkGgAKCRD2jVf7quc0J3OXAJ4/

nzcUCOYJKNtjYZtPaqVj0cW7BgCfb/Trs4uvS3vooJizh0cJyPQqjymIRgQQEQIABgUCSbdM

ggAKCRAY2KTi0O22TVM/AJ4yWpNFyqo67oYYOgr8OqQfMJnijACfbF0TqBE3xJYraCK3MIW5

Z1Yr+AuIRgQQEQIABgUCSbdvuwAKCRBspBp3Q7jWyEgjAJ98Zu4GfDiOylz+EDbm2T7P1BE7

kQCfbECHQARujttCnPxy2q3vXwL0yrWITAQTEQIADAUCQ5QMzAWDErlesgAKCRAIBc9l3j5t

kwVzAJ4wywabgeuYCDMWZuGAsC2hDSFlFQCfQbdeCT0VoUveXfh3e58f8GJV4h6IZgQTEQIA

JgUCQ4FofgIbAwUJEswDAAYLCQgHAwIEFQIIAwQWAgMBAh4BAheAAAoJEAPAAj4FQQ6XyuEA

n3jEhRhdOW2sftDoYVN4yxxwihw6AKDgTV6czvp1mp1n1/3IPRNdzHXqwIhpBBMRAgApAhsD

BQkSzAMABgsJCAcDAgQVAggDBBYCAwECHgECF4AFAkOBa6sCGQEACgkQA8ACPgVBDpeYsACg

6YLLgyAExRHb7/HoY9kHFsG1W+QAn3D/G7LJJgKv+Ak623nzyhhu/grVuQINBEOBaIoQCADP

7yaGkyu2a5A5clGb7sD6nDnKjz/rJrpFkluQ0Rwz73KxajjLzQeNGp2EV91UDRXbg1NCkf2E

F97maJO00SYR7ZbQdyBKORgQZuLU/aSJn8UR15ChsuWxe874p/QGylzwEea/JVJUltud3Aep

Dm8sB1nErPzz++iDZSFSdGqM5ltcv3geP0Ax+SDNqe7HWdmgv/EKaFgRUN62yS05V+HWylN+

wuj/SJJiKUhtiZQ26NiawQNDUcxUPgb8oYgGyZRL1GcGWs45RADiEUNSHifqm+CcRSkQFXex

GFHDDnHKFFnyVHKaWrwG6Ty9TFlx7+jo8gwDyphBLV0nsTYQcJvjAAMFCACSMCA/aewrLvVo

C63o90CKgftkpwt53lp/1vqBvRzuZlCTLQ5ijbA2Pn/9oqPfWUqeIObr2YCRlpwnw5jfOKah

wknlujA9nAzruqA2xwkDp2jpyEOoh1meDhbeaPa8lx4A2sxYwfSMCDDZ3Jwm/c0lFdhmlUEA

dm9tnXL024evFr5BJVmmgbQsHErSwfyI6CDWjVXbySO60j3K3GVIvTm55iB0rlsMjwhRD4Sy

5PO5aaCktmICqBtSDBHZjje7PWPKN92ZhHr24g11Xc3u42U1bQ+J9oNmuTJMI20NcR0lgJNR

d8oQvqHyU7B5mRMBalCKNDdtVLbTuh2Cs6vrS6jFiE8EGBECAA8FAkOBaIoCGwwFCRLMAwAA

CgkQA8ACPgVBDpfxKACeIfmJdm0wyb6FNyfP9/yYlSak1T8AoJjH/Pc+Uwq5T0kUEdcurSEm

MZd1

=5NNR

—–END PGP PUBLIC KEY BLOCK—–

Run the following command to fetch the missing public key

gpg –import /home/key.asc

and add the key to the keyring

apt-key add /root/.gnupg/pubring.gpg

Then do :

apt-key update

apt-get update

Install UltraMonkey

apt-get install ultramonkey

dpkg-reconfigure ipvsadm

Answer as follows:

Do you want to automatically load IPVS rules on boot?

No

Select a daemon method.

None

3 Enable Packet Forwarding On The Load Balancers

The load balancers must be able to route traffic to the Apache nodes. Therefore we must enable packet forwarding on the load balancers. Add the following lines to /etc/sysctl.conf:

vi /etc/sysctl.conf

# Enables packet forwarding

net.ipv4.ip_forward = 1

Then do this:

sysctl –p

4 Configure heartbeat And ldirectord

Now we have to create three configuration files for heartbeat. They must be identical on loadb1 and loadb2

vi /etc/ha.d/ha.cf

logfacility local0

bcast eth0 # Linux

mcast eth0 225.0.0.1 694 1 0

auto_failback off

node loadb1.com

node loadb2.com

respawn hacluster /usr/lib/heartbeat/ipfail

apiauth ipfail gid=haclient uid=hacluster

Important: As nodenames we must use the output of uname –n on loadb1 and loadb2.

vi /etc/ha.d/haresources

loadb1.com \

ldirectord::ldirectord.cf \

LVSSyncDaemonSwap::master \

IPaddr2::192.168.1.105/24/eth0/192.168.1.255

The first word is the output of uname –n on loadb1, no matter if you create the file on loadb1 or loadb2! After IPaddr2 we put our virtual IP address 192.168.1.105.

vi /etc/ha.d/authkeys

auth 3

3 md5 somerandomstring

somerandomstring is a password which the two heartbeat daemons on loadb1 and loadb2 use to authenticate against each other.

/etc/ha.d/authkeys should be readable by root only, therefore we do this:

Chmod 600 /etc/ha.d/authkeys

ldirectord is the actual load balancer. We are going to configure the two load balancers (loadb1.com and loadb2.com) in an active/passive setup, which means we have one active load balancer, and the other one is a hot-standby and becomes active if the active one fails. To make it work, we must create the ldirectord configuration file /etc/ha.d/ldirectord.cf which again must be identical on loadb1 and loadb2.

vi /etc/ha.d/ldirectord.cf

checktimeout=10

checkinterval=2

autoreload=no

logfile=”local0”

quiescent=yes

virtual=192.168.1.105:80

real=192.168.1.101:80 gate

real=192.168.1.102:80 gate

fallback=127.0.0.1:80 gate

service=http

request=”ldirector.html”

receive=”Test Page”

scheduler=rr

protocol=tcp

checktype=negotiate

Afterwards we create the system startup links for heartbeat and remove those of ldirectord because ldirectord will be started by the heartbeat daemon:

update-rc.d heartbeat start 75 2 3 4 5 . stop 05 0 1 6 .

update-rc.d -f ldirectord remove

Finally we start heartbeat (and with it ldirectord):

/etc/init.d/ldirectord stop

If you are getting the following error while stopping ldirectord service :

Can’t locate Socket6.pm in @INC (@INC contains: /etc/perl

/usr/local/lib/perl/5.10.0 /usr/local/share/perl/5.10.0 /usr/lib/perl5

/usr/share/perl5 /usr/lib/perl/5.10 /usr/share/perl/5.10

/usr/local/lib/site_perl .) at /usr/sbin/ldirectord line 815.

Run :

apt-get install libsocket6-perl

/etc/init.d/heartbeat start

5 Test the Load Balancers

ip addr sh eth0

The active load balancer should list the virtual IP address (192.168.1.105):

2: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 1000

link/ether 00:16:3e:40:18:e5 brd ff:ff:ff:ff:ff:ff

inet 192.168.1.103/24 brd 192.168.1.255 scope global eth0

inet 192.168.1.105/24 brd 192.168.1.255 scope global secondary eth0

The hot-standby should show this:

2: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 1000

link/ether 00:16:3e:50:e3:3a brd ff:ff:ff:ff:ff:ff

inet 192.168.1.104/24 brd 192.168.1.255 scope global eth0

ldirectord ldirectord.cf status

Output on the active load balancer:

ldirectord for /etc/ha.d/ldirectord.cf is running with pid: 1455

Output on the hot-standby:

ldirectord is stopped for /etc/ha.d/ldirectord.cf

ipvsadm -L –n

Output on the active load balancer:

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags

-> RemoteAddress:Port Forward Weight ActiveConn InActConn

TCP 192.168.0.105:80 rr

-> 192.168.0.101:80 Route 0 0 0

-> 192.168.0.102:80 Route 0 0 0

-> 127.0.0.1:80 Local 1 0 0

Output on the hot-standby:

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags

-> RemoteAddress:Port Forward Weight ActiveConn InActConn

/etc/ha.d/resource.d/LVSSyncDaemonSwap master status

Output on the active load balancer:

master running

(ipvs_syncmaster pid: 1591)

Output on the hot-standby:

master stopped

6 Configure The Two Apache Nodes

Finally we must configure our Apache cluster nodes apachenode1.com and apachenode1.com to accept requests on the virtual IP address 192.168.1.105.

apt-get install iproute

Add the following to /etc/sysctl.conf:

vi /etc/sysctl.conf

# Enable configuration of arp_ignore option

net.ipv4.conf.all.arp_ignore = 1

# When an arp request is received on eth0, only respond if that address is

# configured on eth0. In particular, do not respond if the address is

# configured on lo

net.ipv4.conf.eth0.arp_ignore = 1

# Ditto for eth1, add for all ARPing interfaces

#net.ipv4.conf.eth1.arp_ignore = 1

# Enable configuration of arp_announce option

net.ipv4.conf.all.arp_announce = 2

# When making an ARP request sent through eth0 Always use an address that

# is configured on eth0 as the source address of the ARP request. If this

# is not set, and packets are being sent out eth0 for an address that is on

# lo, and an arp request is required, then the address on lo will be used.

# As the source IP address of arp requests is entered into the ARP cache on

# the destination, it has the effect of announcing this address. This is

# not desirable in this case as adresses on lo on the real-servers should

# be announced only by the linux-director.

net.ipv4.conf.eth0.arp_announce = 2

# Ditto for eth1, add for all ARPing interfaces

#net.ipv4.conf.eth1.arp_announce = 2

Then run this:

sysctl -p

Add this section for the virtual IP address to /etc/network/interfaces:

vi /etc/network/interfaces

auto lo:0

iface lo:0 inet static

address 192.168.0.105

netmask 255.255.255.255

pre-up sysctl -p > /dev/null

Then run this:

ifup lo:0

Finally we must create the file ldirector.html. This file is requested by the two load balancer nodes repeatedly so that they can see if the two Apache nodes are still running. I assume that the document root of the main apache web site on webserver1 and webserver2 is /var/www, therefore we create the file /var/www/ldirector.html:

vi /var/www/ldirector.html

Test Page

7 Further Testing

You can now access the web site that is hosted by the two Apache nodes by typing http://192.168.1.105 in your browser.

Now stop the Apache on either webserver1 or webserver2. You should then still see the web site on http://192.168.1.105 because the load balancer directs requests to the working Apache node. Of course, if you stop both Apaches, then your request will fail.

Now let’s assume that loadb1 is our active load balancer, and loadb2 is the hot-standby. Now stop heartbeat on loadb1:

loadb1:

/etc/init.d/heartbeat stop

Wait a few seconds, and then try http://192.168.1.105 again in your browser. You should still see your web site because loadb2 has taken the active role now.

Now start heartbeat again on loadb1:

/etc/init.d/heartbeat start

loadb2 should still have the active role.