When a cluster fails, you must use the cluster configuration file and database data to recover the cluster.

Note -
For cluster recovery in a Kubernetes Operator deployment, go to: Redis Enterprise Cluster Recovery for Kubernetes.

Cluster failure can be caused by:

  • A hardware or software failure that causes the cluster to be unresponsive to client requests or administrative actions.
  • More than half of the cluster nodes lose connection with the cluster, resulting in quorum loss.

To recover a cluster and re-create it as it was before the failure you must restore the cluster configuration (ccs-redis.rdb) to the cluster nodes. To restore the data that was in the databases to databases in the new cluster you must restore the database persistence files (backup, AOF, or snapshot files) to the databases. These files are stored in the persistent storage location.

The cluster recovery process includes:

  1. Install RS on the nodes of the new cluster.
  2. Mount the persistent storage with the recovery files from the original cluster to the nodes of the new cluster.
  3. Recover the cluster configuration on the first node in the new cluster.
  4. Join the remaining nodes to the new cluster.
  5. Recover the databases.

Prerequisites

  • We recommend that you recover the cluster to clean nodes. If you use the original nodes, make sure there are no Redis processes running on any nodes in the new cluster.
  • We recommend that you use clean persistent storage drives for the new cluster. If you use the original storage drives, make sure that you backup the files on the original storage drives to a safe location.

Recovering the Cluster

  1. (Optional) If you want to recover the cluster to the original cluster nodes, uninstall RS from the nodes.

  2. Install RS on the new cluster nodes.

    Do not configure the cluster nodes (rladmin cluster create in the CLI or Setup in the Web UI).

    The new servers must have the same basic hardware and software configuration as the original servers, including:

    • The same number of nodes
    • At least the same amount of memory
    • The same RS version
    Note -
    The cluster recovery can fail if these requirements are not met.
  3. Mount the persistent storage drives with the recovery files to the new nodes. These drives must contain the cluster configuration backup files and database persistence files.

    Note -
    Make sure that the user redislabs has permissions to access the storage location of the configuration and persistence files on each of the nodes.

    If you use local persistent storage, place all of the recovery files on each of the cluster nodes.

  4. To recover the cluster configuration from the original cluster to the first node in the new cluster, from the rladmin command-line interface (CLI):

    cluster recover filename [ <persistent_path> | <ephemeral_path> ]<filename> node_uid <node_uid> rack_id <rack_id>
    Command syntax

    For example:

    rladmin cluster recover filename /tmp/persist/ccs/ccs-redis.rdb node_uid 1 rack_id 5

    When the recovery command succeeds, this node is configured as the node from the old cluster that has ID 1.

  5. To join the remaining servers to the new cluster, from the rladmin CLI of each new node run:

    cluster join [ nodes <cluster_member_ip_address> | name <cluster_FQDN> ] username <username> password <password> replace_node <node_id>
    Command syntax

    For example:

    rladmin cluster join nodes 10.142.0.4 username admin@example.com password mysecret replace_node 2

    You can run the rladmin status command to verify that the recovered nodes are now active, and that the databases are pending recovery.

    Note -
    Make sure that you update your DNS records with the IP addresses of the new nodes.

After the cluster is recovered, you must recover the databases.