Core group protocols
The default discovery protocol, the default failure detection protocol or an alternative protocol provider start as part of the core group startup procedure, establishing connectivity to other core group members, monitoring this connectivity, and handling connectivity failures for this core group member, at regularly scheduled intervals.
The default discovery protocol
The default discovery protocol...
- Retrieves the list of core group members and associated network information from the product configuration settings.
- Attempts to open network connections to all of the other core group members.
- At periodic intervals recalculates the set of unconnected members and attempts to open connections to those members.
- When a connection is made to another core group member, the Discovery Protocol notifies the View Synchrony Protocol, and logs this event as an informational message in the SystemOut.log file.
DCSV1032I: DCS Stack DefaultCoreGroup at Member MyCell\foo\nodeagent: Connected a defined member MyCell\fooCellManager\dmgr.
The Failure Detection Protocol detects connection failures and notifies the Discovery Protocol. The Discovery Protocol then attempts to open a new network connection to that member at the next scheduled interval. The amount of CPU cycles that the Discovery Protocol task consumes is proportional to the number of core group members that are stopped or unreachable. The CPU cycles that the Discovery Protocol task consumes is negligible at the default settings.
Default Failure Detection Protocol
The Failure Detection Protocol monitors the core group network connections that the Discovery Protocol establishes. When the Failure Detection Protocol detects a failed network connection, it reports the failure to the View Synchrony Protocol and the Discovery Protocol. The View Synchrony Protocol adjusts the view to exclude the failed member. The Discovery Protocol attempts to reestablish a network connection with the failed member. This task runs as long as the member is active.
The Failure Detection Protocol uses two distinct mechanisms to find failed members:
- Looks for connections that closed because the underlying socket was closed.
When a core group member normally stops in response to an administration command, the core group transport for that member also stops, and the socket associated with the transport closes. If a core group member terminates abnormally, the underlying operating system normally closes the sockets that the process opened and the socket associated with the core group transport. is closed.
For either type of termination, core group members that have an open connection to the terminated member are notified that the connection is no longer usable. The core group member that receives the socket closed notification considers the terminated member a failed member.
When a failed member is detected because of the socket closing mechanism, the following message is logged in the SystemOut.log file for the surviving members:
DCSV1115W: DCS Stack DefaultCoreGroup at Member fooCell01\foo\ServerD: Member fooCell01\foo\ServerC connection was closed. Member will be removed from view. DCS connection status is Discovery|Ptp, transmitter closed.
The closed socket mechanism is the way that failed members are typically discovered. TCP settings in the underlying operating system, such as FIN_WAIT, affect how quickly socket closing events are received.
- It listens for active heartbeats from the core group members.
The active heart beating mechanism is analogous to the TCP keep alive function. At regularly scheduled intervals, each core group member sends a ping packet on every open core group connection. The rate or periodicity at which the packet is sent is called the heartbeat transmission period.
Each core group member expects to receive a packet on each open connection from the core group member on the other end of the connection. If no packets are received over an open connection within the time length specified for the heartbeat timeout period, then the member on the other end of the connection is marked as failed.
The heartbeat timeout period must be a whole number that is a multiple of the heartbeat transmission period. The heartbeat timeout period must also be at least twice as large as the heartbeat transmission period.
When a member is marked as failed, the following message is sent to the error log file:
DCSV1112W: DCS Stack DefaultCoreGroup at Member fooCell01\fooCellManager01\dmgr: Suspected member fooCell01\nettuno\ServerB because of heartbeat timeout. Configured Timeout is 180000 milliseconds. DCS logical channel is Connected|Ptp.
Active heartbeats are most useful for detecting core group members that are unreachable because the network is stopped. Active heartbeats consume some CPU usage. The amount of CPU usage that is consumed is proportional to the number of active members in the core group. The default configuration for active heartbeats is a balance of CPU usage and timely failed member detection.
Use the administrative console or wsadmin.sh to configure the heartbeat transmission period and heartbeat timeout period. See Configure the Failure Detection Protocol for a core group.
Alternative protocol providers
Currently, no alternative protocol providers are available for the IBM i and distributed platforms.
For z/OS an protocol provider is the Cross-system Coupling Facility (XCF)-based provider, which uses less system resources than the default discovery protocol and Failure Detection Protocol, especially during times when the core group members are idle. An alternate protocol provider generally use less system resources because it does not perform the member-to-member TCP/IP pinging that the default protocol providers use to determine if a core group member is still active.
Before reconfiguring a specific core group to use an alternative protocol provider, verify that the core group meets the following requirements. If the core group does not meet all of these requirements, we must continue to use the default discovery protocol and the default Failure Detection Protocol with this core group.
- The core group is homogenous. This means that the core group processes must all reside on the same platform. For example, the core group cannot contain a mixture of z/OS and distributed processes.
- If the core group needs to be bridged to another core group, using the core group bridge service, then all of the core groups that are bridged to this core group are also homogeneous core groups.
- All members of the core group must be at v7.x of the product. If any members of the core group are running at a v6.x level of the product, then update them to v7.x, before we can switch to the alternative protocol provider.
Configure the default discovery protocol for a core group Configure the default Failure Detection Protocol for a core group Core group custom properties High Performance Extensible Logging