Core group discovery and failure detection protocols


 

+

Search Tips   |   Advanced Search

 

The Discovery and Failure Detection protocols, or an alternative protocol provider, start when core group members start, and handle...

 

The default Discovery Protocol

The default Discovery Protocol establishes network connectivity with the other members of the core group using a list of core group members and the associated network information from the WAS ND settings.

The Discovery Protocol then attempts to open network connections to all of the other core group members.

At periodic intervals, the Discovery Protocol recalculates the set of unconnected members and attempts to open connections to those members.

When a connection is made to another core group member, the Discovery Protocol notifies the View Synchrony Protocol, and logs this event as an informational message in SystemOut.log.

DCSV1032I: DCS Stack DefaultCoreGroup at Member MyCell\skyway\nodeagent: Connected a defined member MyCell\skywayCellManager\dmgr.

The Failure Detection Protocol...

  1. Detects connection failures
  2. Notifies the Discovery Protocol
  3. Discovery Protocol tries to opens a new network connection to that member at the next scheduled interval

The amount of CPU cycles that the Discovery Protocol task consumes is proportional to the number of core group members that are stopped or unreachable.

The CPU cycles that the Discovery Protocol task consumes is negligible at the default settings.

 

Default Failure Detection Protocol

The Failure Detection Protocol monitors the core group network connections that the Discovery Protocol establishes. When the Failure Detection Protocol detects a failed network connection, it reports the failure to the View Synchrony Protocol and the Discovery Protocol.

The View Synchrony Protocol adjusts the view to exclude the failed member.

The Discovery Protocol attempts to reestablish a network connection with the failed member. This task runs as long as the member is active.

The Failure Detection Protocol uses two distinct mechanisms to find failed members:

Core connections closed because the underlying socket was closed.

When a core group member normally stops in response to an administration command, the core group transport for that member also stops, and the socket that is associated with the transport closes. If a core group member terminates abnormally, the underlying operating system normally closes the sockets that the process opened and the socket associated with the core group transport is closed.

For either type of termination, core group members that have an open connection to the terminated member are notified that the connection is no longer usable. The core group member that receives the socket closed notification considers the terminated member a failed member.

When a failed member is detected because of the socket closing mechanism, the following messages are logged in SystemOut.log for the surviving members:

DCSV1113W: DCS Stack DefaultCoreGroup at Member skywayCell01\skywayCellManager01\dmgr:
Suspected another member because the outgoing connection to the other member was closed.
Suspected member is skywayCell01\nettuno\ServerB. DCS logical channel is View|Ptp.

DCSV1111W: DCS Stack DefaultCoreGroup at Member skywayCell01\skywayCellManager01\dmgr:
Suspected another member because the outgoing connection from the other member was closed.
Suspected members is skywayCell01\nettuno\ServerB. DCS logical channel is Connected|Ptp.

The closed socket mechanism is the way that failed members are typically discovered. TCP settings in the underlying operating system, such as FIN_WAIT, affect how quickly socket closing events are received.

Listens for active heartbeats from the core group members.

The active heart beating mechanism is analogous to the TCP keep alive function. At regularly scheduled intervals, each core group member sends a ping packet on every open core group connection. The rate or periodicity at which the packet is sent is called the heartbeat transmission period.

Each core group member expects to receive a packet on each open connection from the core group member on the other end of the connection. If no packets are received over an open connection within the time length specified for the heartbeat timeout period, then the member on the other end of the connection is marked as failed.

The heartbeat timeout period must be...

  • A whole number that is a multiple of the heartbeat transmission period.
  • At least twice as large as the heartbeat transmission period.

When a member is marked as failed, the following message is sent to the error log file:

DCSV1112W: DCS Stack DefaultCoreGroup at Member skywayCell01\skywayCellManager01\dmgr:
Suspected member skywayCell01\nettuno\ServerB because of heartbeat timeout.
Configured Timeout is 180000 milliseconds. DCS logical channel is Connected|Ptp.

Active heartbeats are most useful for detecting core group members that are unreachable because the network is stopped. Active heartbeats consume some CPU usage.

The amount of CPU usage that is consumed is proportional to the number of active members in the core group. The default configuration for active heartbeats is a balance of CPU usage and timely failed member detection.

Use the admin console or wsadmin to configure the heartbeat transmission and timeout periods.

 

Alternative protocol providers

New feature: Use an alternate protocol provider to manage communication between core group members. In general, alternate protocol providers, such as the z/OS Cross-system Coupling Facility (XCF)-based provider, uses less system resources than the default Discovery Protocol and Failure Detection Protocol, especially during times when the core group members are idle.

An alternate protocol provider generally use less system resources because it does not perform the member-to-member TCP/IP pinging that the default protocol providers use to determine if a core group member is still active.

Before reconfiguring a core group to use an alternative protocol provider, verify...

Currently, no alternative protocol providers are available for the i5/OS and distributed platforms.



 

Related concepts

Core groups (high availability domains)

 

Related tasks

Configure the default Discovery Protocol for a core group
Configure the default Failure Detection Protocol for a core group

 

Related

Core group custom properties