Resolving a split-brain situation

There are situations where certain failure sequences in an HA group could lead to a split-brain situation being reported.

For example, say all three nodes lose connectivity. If both secondary nodes regain connectivity before the primary node, they form a new quorum and one of them runs the queue manager. When the original primary node regains connectivity, it is possible that a split-brain situation is reported.

In this situation, running rdqmstatus -m QMname on the original primary node shows the HA status as Inconsistent:

Node:                                   node1
Queue manager status:                   Running elsewhere
HA role:                                Secondary
HA status:                              Inconsistent
HA control:                             Enabled
HA current location:                    hanode2
HA preferred location:                  This node
HA floating IP interface:               None
HA floating IP address:                 None

Node:                                   node2
HA status:                              Inconsistent
HA out of sync data:                    8KB

Node:                                   node3
HA status:                              Inconsistent
HA out of sync data:                    8KB

In this instance, we should retain the data on the original secondary nodes (that formed the new quorum). Complete the following steps:

1. On the original primary node, as root, run the following command:

   drbdadm connect --discard-my-data QMname

2. On each of the secondary nodes, as root, run the following command:

   drbdadm connect QMname:first-node-name

drbdadm is a command provided by DRBD. It is installed as part of the drbd-utils package in /usr/sbin/drbdadm.

Parent topic: RDQM high availability