Tuning heartbeat and cluster parameters

Sending and receiving heartbeat messages among the nodes in the cluster is key for the cluster manager. If a cluster node does not receive heartbeat messages from the other node within the prescribed time, a cluster reformation is initiated. At the end of reformation, if a new set of nodes form a cluster, that information is passed to the package coordinator. Packages running on nodes that are no longer in the new cluster are transferred to their adoptive nodes.

A package is a collection of services, disk volumes and IP addresses managed by MC/ServiceGuard to ensure that they are available. Packages are units of failover by which MC/ServiceGuard starts and halts configured applications. The package coordinator decides when and where to run, halt or move packages. The package manager executes the user-defined control script to run and halt packages and package services, and react to changes in the monitored resources. A package starts up on the primary node when the cluster starts. In case of a service, network, or resource failure, a package failover takes place when the package coordinator initiates the start of a package on a new node. A package failover involves both halting the existing package and starting the new instance of the package. A package switch involves moving packages and their IP addresses to a new system. With package failovers, TCP connections are lost. TCP applications must reconnect to regain connectivity; this is not handled automatically.

The network manager detects and recovers from network card and cable failures so that network services remain highly available to clients. There are stationary IP addresses for each active network interface in each node. For example, in our case, we used 10.4.12.100, 10.5.12.200, etc. In order to perform transparent failover, we need to assign a relocatable IP address to database services so that clients will use the same IP address to connect to database servers after database services failover to another node. Our relocatable IP address for database servers is 10.5.12.80 or hpcluster.itso.ibm.com. The relocatable IP address is also known as a floating IP address or package IP address. When the database package starts up, the cmmodnet command in the package control script assigns this relocatable IP address (hpcluster.itso.ibm.com) to the primary LAN interface card in the primary node. Within the same node, both stationary and relocatable IP addresses will switch to a standby interface in the event of a LAN card failure. Only a relocatable IP address can be taken over by an adoptive node if control of the package is transferred. Therefore, WebSphere can access the database server via its relocatable IP address without knowing on which node the database server is running.

At regular intervals, MC/ServiceGuard polls all the network interface cards specified in the cluster configuration file. The polling interface sends LAN packets to all other interfaces in the node that are on the same bridged net and receives packets back from them. Once a network failure is detected, if there is a backup LAN card in the node, a local network switch is initiated and TCP/IP connections are not lost (except for IEEE 802.3 that does not have the rearp function). Otherwise, remote switch is initiated, TCP connections are lost, and database applications (WebSphere) must reconnect to regain connectivity. A remote switch involves moving packages and their associated IP address to a new system.

MC/ServiceGuard, together with other software, also monitors other resources and takes action when events occur. These resources are monitored by MC/ServiceGuard as a package dependency.

There are several parameters you can tune for performance and reliability: The heartbeat interval is the normal interval between the transmission of heartbeat messages from one node to the other in the cluster. The default value is 1 second, but you can set it to up to 30 seconds. We tested several different values, then used 2 seconds. The node timeout is the time after which a node may decide that the other node has become unavailable and initiate cluster reformation. The default value is 2 seconds, with a minimum of 2 x (heartbeat interval) and a maximum of 60 seconds. Small values of node timeout and heartbeat interval may increase the potential for spurious cluster reformation due to momentary system hangs or network load spikes. We tested several different values, then set 2 seconds of heartbeat interval and 4 seconds of node timeout. The network polling interval is the frequency at which the networks configured for MC/ServiceGuard are checked. The default value is 2 seconds. Changing this value can affect how quickly a network is detected, ranging from 1 to 30 seconds. Switching an IP address from a failed LAN card to a standby LAN card on the same physical subnet may take place if automatic switching is set to Enabled in SAM. You can define failover behavior and whether it will fall back automatically as soon as the primary node is available. The resource polling interval is the frequency of monitoring a configured package resource. The default value is 60 seconds, with a minimum value of 1 second.

  Prev | Home | Next

 

WebSphere is a trademark of the IBM Corporation in the United States, other countries, or both.

 

IBM is a trademark of the IBM Corporation in the United States, other countries, or both.