Configure Liberty collective replica sets

Replica sets provide highly available management capabilities for a Liberty administrative domain. A replica set is a set of collective controllers configured to work together. To create a replica set, we increase the number of collective controllers and configure them to communicate with each other. Each new collective controller we add is called a replica and has the same security configuration as the original controller. All information written to any one controller is automatically replicated to all other active controllers.

All collective controllers in the replica set can perform the same operations as the original controller. Each replica contains all the repository updates that the other replicas within the set have processed. Therefore, there is no need for a member or client to connect with a particular collective controller each time that it interacts with the collective. Any of the collective controllers configured in the replica set can provide the same data.

A replica set requires an odd number of replicas and the minimum number of controllers is three. This configuration makes it possible to maintain a consensus with a majority. All replicas and controllers in a set are co-equal. There are no primary or secondary replicas or controllers in a replica set.

We can add more replicas to an existing set. By default, The existing replicas do not require any updates to their configuration. Optionally, we can update their configurations in the server.xml file to more accurately reflect the replicas that form the replica set, but this update does not affect their behaviors.

It is not necessary to change the replicaSet value in the server.xml file of an existing replica in the set. No change to the configuration of an existing replica is required. To update the replicaSet values in the configurations of existing replicas in the set so that the configuration values are consistent across all replicas in the set, set the isInitialReplicaSet value of the existing replicas to false. After altering the replicaSet value, it describes a changed replica set rather than the initial replica set.

For an example of how to create a replica set that consists of three collective controllers on the same host, see Example: Create and activate a replica set .

Note: Whenever we refer to a replica, we must be consistent and always use the same host:port value. If a host name or IP address is used in one instance, it must be used in all instances.

Add more replicas to an existing set

Ensure that the existing replica set is running and that most of the replicas are available.
Create a server to be the new collective controller.

Replicate to transform the new server into a collective controller.

wlp/bin/collective replicate MyNewController \
   --host=host  \
   --port=https_port \
   --user=userName \
   --password=userPassword \
   --keystorePassword=password\
   --hostName=myHost.myCompany.com\
   --autoAcceptCertificates=true\
   --createConfigFile=c:/wlp/usr/servers/myNewController/collective-replicate-include.xml

Alternate syntax:

wlp/bin/collective replicate MyNewController \
    --controller=userName:password@host:https_port 
    --keystorePassword=password
    --hostName=myHost.myCompany.com
    --autoAcceptCertificates=true
    --createConfigFile=c:/wlp/usr/servers/myNewController/collective-replicate-include.xml

The XML output contains a configuration element, including public and private SSH key paths. If the collective-wide SSH keys used in the initial controller are auto-generated, the keys are copied to the replica server at the default location. Otherwise, the keys must be manually copied over to the replica server, and the paths must reflect the location where they were copied.

Include the outputted file in the collective configuration by adding an include statement to the server.xml file:
If the --createConfigFile=out_file_path parameter, is not specified, the replicate command writes XML output to a console screen. In this case, we must manually copy the output into the server.xml file.
Add host="*" into the httpEndpoint stanza of the new controller's server.xml file.
Verify these values are set in server.xml.
The XML output produced by the replicate command should contain these values:
Optional settings:
These settings have default values, which can be altered.
Here is an example of what we might add to a new replica server.xml file:
The XML printed by the replicate command requires updating the server's security configuration and specifying the collectiveRootKeys keystore password. The server's security configuration must be identical to the original collective controller's configuration. The collectiveRootKeys keystore password must be the password used for the original collective controller's collectiveRootKeys keystore password. If the replica was created from the controller created in Configure a Liberty collective, the new controller configuration must contain the appropriate information.
The new controller configuration must contain the following information:
If we are using the JKS keystore in version 19.0.0.2 and earlier, the new controller configuration must contain the following information:
Start the new replica by starting the new collective controller.
Confirm that the original collective controller successfully connected to the new replica.
Look for message CWWKX6009I in the messages.log files of both the original collective controller and the replica.
For scripts that run the replicate and addReplica commands, add a 10-second wait after the replicate command runs to ensure the original collective controller and replica have time to connect before the addReplica command runs.

Run the addReplica operation on the collective utility to activate the new replica.

wlp/bin/collective addReplica localhost:10012 
  --host=host
  --port=port
  --user=user
  --password=password
  --autoAcceptCertificates=true

Alternate syntax:

wlp/bin/collective addReplica localhost:10012 \
    --controller=user:password@host:port \
    --autoAcceptCertificates=true

To learn more about options, run...

collective help replicate

Example: Create and activate a replica set

This example describes how to create and activate a replica set. A replica set must have at least three replicas, preferably on different hosts, for high availability. This example has the replicas on the same host, which requires that we assign unique port numbers for the replicas. When replicas are on different hosts, the replicas can use the same port numbers.

Create a replica set.
1. If we do not have a collective controller, create one.
  See step 1 of Configure a Liberty collective.
2. Ensure that the existing collective controller is running.
  If not running, start it:
3. Create a server to be the new collective controller.
4. Replicate the existing collective controller configuration into the new collective controller.
  The new collective controller is called a replica.
  Run a replicate command that uses the administrative security domain configuration of the existing collective controller and sets a new keystore password for the replica. Review server.xml to find the values for the --host, --port, --user, and --password parameters. For the --keystorePassword parameter, set a value to use for the keystore, such as myController2.
  Alternate syntax:
  If prompted to accept the certificate chain, enter y (yes).
  If the --createConfigFile=output_file_path parameter is specified, the replicate command writes XML output to a file, rather than a console screen. In this case, include the output file in the collective configuration by adding an include statement to the server.xml file:
  If the --createConfigFile=output_file_path parameter is not specified, the replicate command writes XML output to a console screen. In this case, we must manually add the output into the server.xml file of the new replica.
  If the --hostName=myHost.myCompany.com parameter is specified using the collective replicate command, add the host="*" parameter into the httpEndpoint stanza in the new controller's server.xml file.
5. Edit the parameter values of the XML output from collective replicate command.
  - Set unique port values for the httpEndpoint element httpPort and httpsPort values.
    myController1...
    myController2...
  - Set RPC credentials for the replica in either of two ways:
    - Set hostAuthInfo RPC user and password values.
      Set rpcUser to an operating system login user ID for the host on which the replica resides, and set rpcUserPassword to the operating system login password for the user ID.
    - If the replica host is registered with the collective controller, set hostAuthInfo useHostCredentials to true for the replica to inherit RPC credentials from its host.
      See Overriding Liberty server host information
  - Because collective replicas share a host, set a port number for the new replica that is unique among the replica set. Set replicaSet sets host:port values that identify the replica set.
    For example, if the original controller named myController and the replica are both on the same localhost, change the values for myController2 from 10010...
    ...to 10011 for the replica port and 10010 for the replica set port:
  - Ensure that the security configuration sets the same values as those used by the original controller. For example, both myController and the myController2 replica use:
  - Ensure that the collective root signers keystore element sets the same password as that used by the original controller. For example, copy the collectiveRootKeys keystore password from the myController server.xml file and paste it into the myController2 replica server.xml file. This example shows a generated password:
    The following example shows a generated password if we are using the JKS keystore through 19.0.0.2:
6. Start the new replica by starting the new collective controller.
7. Verify that the original collective controller can communicate with the new replica.
  1. Open an editor on the original controller messages log, $WLP_USER_DIR/servers/myController/logs/messages.log.
  2. Look for the following message, which might have different IP addresses in the environment:
8. Verify that the new replica can communicate with the original collective controller.
  1. Open an editor on the replica messages log, $WLP_USER_DIR/servers/myController2/logs/messages.log.
  2. Look for the following message, which might have different IP addresses in the environment:
    Additionally, expect to see the following message as the controller is not active currently. We can ignore the message.
Activate the new replica.
Run an addReplica command that uses the administrative security domain configuration of the collective controller and specifies the endpoint of the replica to activate in the form replicaHost:replicaPort.
- Look at server.xml of the collective controller to find values for the --host, --port, --user, and --password.
- Look at server.xml of the replica to find values for the replicaHost:replicaPortparameters.
Alternate syntax:
If prompted to accept the certificate chain, enter y (yes).
Repeat steps 1 and 2 for any other replicas we want to create.
For example, add a third replica to the replica set. Name the new replica myController3 and specify replicaPort="10012". To ensure high availability, a replica set must have at least three replicas.After the third replica is added to the replica set, we can verify that the original collective controller and new replicas are synchronized successfully.
- Look for the following messages in the original controller messages log. The messages might have different IP addresses in the environment:
- Look for the following messages in the messages logs of the added replicas. The messages might have different IP addresses in the environment:

Parent topic: Set up the server-management environment for Liberty using collectives

Related information

Remove a controller from a Liberty collective replica set