Core group discovery and failure detection protocols
When a core group member starts, no connections to other core group members exist. If a core group is configured to run with either the default Discovery and Failure Detection Protocols or an alternative protocol provider, either the discovery and failure detection tasks or the alternate protocol provider tasks start as part of the process startup procedure. These tasks establish connectivity to other core group members, monitor this connectivity and handle connectivity failures for this core group member, at regularly scheduled intervals, as long as the core group member is active.
The default Discovery Protocol
The default Discovery Protocol establishes network connectivity with the other members of the core group by retrieving the list of core group members and the associated network information from the product configuration settings. The Discovery Protocol then attempts to open network connections to all of the other core group members. At periodic intervals, the Discovery Protocol recalculates the set of unconnected members and attempts to open connections to those members.
When a connection is made to another core group member, the Discovery Protocol notifies the View Synchrony Protocol, and logs this event as an informational message in the SystemOut.log file.
DCSV1032I: DCS Stack DefaultCoreGroup at Member MyCell\anzio\nodeagent: Connected a defined member MyCell\anzioCellManager\dmgr.
Connections can fail at any time for a variety of reasons. The Failure Detection Protocol detects connection failures and notifies the Discovery Protocol. The Discovery Protocol then attempts to open a new network connection to that member at the next scheduled interval.
The amount of CPU cycles that the Discovery Protocol task consumes is proportional to the number of core group members that are stopped or unreachable. The CPU cycles that the Discovery Protocol task consumes is negligible at the default settings.
Default Failure Detection Protocol
The Failure Detection Protocol monitors the core group network connections that the Discovery Protocol establishes. When the Failure Detection Protocol detects a failed network connection, it reports the failure to the View Synchrony Protocol and the Discovery Protocol. The View Synchrony Protocol adjusts the view to exclude the failed member. The Discovery Protocol attempts to reestablish a network connection with the failed member. This task runs as long as the member is active.
The Failure Detection Protocol uses two distinct mechanisms to find failed members:
- It looks for connections that closed because the underlying socket was closed.
When a core group member normally stops in response to an administration command, the core group transport for that member also stops, and the socket associated with the transport closes. If a core group member terminates abnormally, the underlying operating system normally closes the sockets that the process opened and the socket associated with the core group transport. is closed.
For either type of termination, core group members that have an open connection to the terminated member are notified that the connection is no longer usable. The core group member that receives the socket closed notification considers the terminated member a failed member.
When a failed member is detected because of the socket closing mechanism, the following message is logged in the SystemOut.log file for the surviving members:
DCSV1115W: DCS Stack DefaultCoreGroup at Member anzioCell01\anzio\ServerD: Member anzioCell01\anzio\ServerC connection was closed. Member will be removed from view. DCS connection status is Discovery|Ptp, transmitter closed.
The closed socket mechanism is the way that failed members are typically discovered. TCP settings in the underlying operating system, such as FIN_WAIT, affect how quickly socket closing events are received.
- It listens for active heartbeats from the core group members.
The active heart beating mechanism is analogous to the TCP keep alive function. At regularly scheduled intervals, each core group member sends a ping packet on every open core group connection. The rate or periodicity at which the packet is sent is called the heartbeat transmission period.
Each core group member expects to receive a packet on each open connection from the core group member on the other end of the connection. If no packets are received over an open connection within the time length specified for the heartbeat timeout period, then the member on the other end of the connection is marked as failed.
The heartbeat timeout period must be a whole number that is a multiple of the heartbeat transmission period. The heartbeat timeout period must also be at least twice as large as the heartbeat transmission period.
When a member is marked as failed, the following message is sent to the error log file:
DCSV1112W: DCS Stack DefaultCoreGroup at Member anzioCell01\anzioCellManager01\dmgr: Suspected member anzioCell01\nettuno\ServerB because of heartbeat timeout. Configured Timeout is 180000 milliseconds. DCS logical channel is Connected|Ptp.
Active heartbeats are most useful for detecting core group members that are unreachable because the network is stopped. Active heartbeats consume some CPU usage. The amount of CPU usage that is consumed is proportional to the number of active members in the core group. The default configuration for active heartbeats is a balance of CPU usage and timely failed member detection.
Use the administrative console or the wsadmin tool to configure the heartbeat transmission period and heartbeat timeout period. See Configure the Failure Detection Protocol for a core group.
(iSeries) (Dist) Alternative protocol providers
Currently, no alternative protocol providers are available for the IBM i and distributed platforms.
Use an alternate protocol provider instead of the default Discovery Protocol and Failure Detection Protocol to monitor and manage communication between core group members. In general, alternate protocol providers, such as the z/OS Cross-system Coupling Facility (XCF)-based provider, uses less system resources than the default Discovery Protocol and Failure Detection Protocol, especially during times when the core group members are idle. An alternate protocol provider generally use less system resources because it does not perform the member-to-member TCP/IP pinging that the default protocol providers use to determine if a core group member is still active.
(ZOS) If we decide to use the z/OS Cross-system Coupling Facility (XCF)-based protocol provider, we should understand that at startup, the server process is joined, as a member, to an XCF group. The XCF group contains all of the active members for the core group. XCF provides notification to all of the members of this group whenever a member joins the group, and whenever a member can no longer be contacted because the server shutdown, or XCF determines that the server process has terminated. Whenever a connection between core group members is established, the z/OS Cross-system Coupling Facility (XCF)-based protocol provider notifies the View Synchrony Protocol, and logs this event as an informational message, similar to the following message, in the SystemOut.log file.
DCSV1032I: DCS Stack DefaultCoreGroup at Member MyCell\anzio\nodeagent: Connected a defined member MyCell\anzioCellManager\dmgr.
Before reconfiguring a specific core group to use an alternative protocol provider, verify that the core group meets the following requirements. If the core group does not meet all of these requirements, we must continue to use the default Discovery Protocol and the default Failure Detection Protocol with this core group.
- The core group is homogenous. This means that the core group processes must all reside on the same platform. For example, the core group cannot contain a mixture of z/OS and distributed processes.
(ZOS) If the core group contains non-z/OS processes, or if the core group is composed of members that are at different version levels of the product, then we cannot use XCF for this core group.
- If the core group needs to be bridged to another core group, using the core group bridge service, then all of the core groups that are bridged to this core group are also homogeneous core groups.
- All members of the core group must be at v7.x of the product. If any members of the core group are running at a v6.x level of the product, then we must update them to v7.x, before we can switch to the alternative protocol provider.
Configure the default Discovery Protocol for a core group Configure the default Failure Detection Protocol for a core group (ZOS) Select an alternate protocol provider for a core group Core group custom properties High Performance Extensible Logging