Recovering a Failed Cluster
When a cluster fails, you must use the cluster configuration file and database data to recover the cluster.
Cluster failure can be caused by:
- A hardware or software failure that causes the cluster to be unresponsive to client requests or administrative actions.
- More than half of the cluster nodes lose connection with the cluster, resulting in quorum loss.
To recover a cluster and re-create it as it was before the failure you must restore the cluster configuration (ccs-redis.rdb) to the cluster nodes. To restore the data that was in the databases to databases in the new cluster you must restore the database persistence files (backup, AOF, or snapshot files) to the databases. These files are stored in the persistent storage location.
The cluster recovery process includes:
- Install RS on the nodes of the new cluster.
- Mount the persistent storage with the recovery files from the original cluster to the nodes of the new cluster.
- Recover the cluster configuration on the first node in the new cluster.
- Join the remaining nodes to the new cluster.
- Recover the databases.
- We recommend that you recover the cluster to clean nodes. If you use the original nodes, make sure there are no Redis processes running on any nodes in the new cluster.
- We recommend that you use clean persistent storage drives for the new cluster. If you use the original storage drives, make sure that you backup the files on the original storage drives to a safe location.
Recovering the Cluster
(Optional) If you want to recover the cluster to the original cluster nodes, uninstall RS from the nodes.
Install RS on the new cluster nodes.
Do not configure the cluster nodes (
rladmin cluster createin the CLI or Setup in the Web UI).
The new servers must have the same basic hardware and software configuration as the original servers, including:
Note -The cluster recovery can fail if these requirements are not met.
- The same number of nodes
- At least the same amount of memory
- The same RS version
Mount the persistent storage drives with the recovery files to the new nodes. These drives must contain the cluster configuration backup files and database persistence files.Note -Make sure that the user redislabs has permissions to access the storage location of the configuration and persistence files on each of the nodes.
If you use local persistent storage, place all of the recovery files on each of the cluster nodes.
To recover the cluster configuration from the original cluster to the first node in the new cluster, from the rladmin command-line interface (CLI):
cluster recover filename [ <persistent_path> | <ephemeral_path> ]<filename> node_uid <node_uid> rack_id <rack_id>Command syntax
<filename>- The full path of the old cluster configuration file in the persistent storage. The cluster configuration file is /css/ccs-redis.rdb. The file is located on the persistent storage drive for each node in the original cluster and all copies are identical.
<node_uid>- The id of the node, in this case
<persistent_path>(optional) - The location of the persistent storage in the new node.
<ephemeral_path>(optional) - The location of the ephemeral storage in the new node.
<rack_id>(optional) - If rack-zone awareness was enabled in the cluster, you can use this parameter to override the rack ID value that was set for the node with ID 1 with a new rack ID. Otherwise, the node gets the same rack ID as the original node.
rladmin cluster recover filename /tmp/persist/ccs/ccs-redis.rdb node_uid 1 rack_id 5
When the recovery command succeeds, this node is configured as the node from the old cluster that has ID 1.
To join the remaining servers to the new cluster, from the rladmin CLI of each new node run:
cluster join [ nodes <cluster_member_ip_address> | name <cluster_FQDN> ] username <username> password <password> replace_node <node_id>Command syntax
nodes- The IP address of a node in the cluster that this node is joining.
name- The FQDN name of the cluster this node is joining.
username- The email address of the cluster administrator.
password- The password of the cluster administrator.
replace_node- The ID of the node that this node replaces from the old cluster.
persistent_path(optional) - The location of the persistent storage in the new node.
ephemeral_path(optional) - The location of the ephemeral storage in the new node.
rack_id(optional) - If rack-zone awareness was enabled in the cluster, use this parameter to set the rack ID to be the same as the rack ID of the old node. You can also change the value of the rack ID by providing a different value and using the
rladmin cluster join nodes 10.142.0.4 username firstname.lastname@example.org password mysecret replace_node 2
You can run the
rladmin statuscommand to verify that the recovered nodes are now active, and that the databases are pending recovery.Note -Make sure that you update your DNS records with the IP addresses of the new nodes.
After the cluster is recovered, you must recover the databases.