A Redis Stream is a data structure that acts like an append-only log. Each stream entry consists of:

  • A unique, monotonically increasing ID
  • A payload consisting of a series key-value pairs

You add entries to a stream with the XADD command. You access stream entries using the XRANGE, XREADGROUP, and XREAD commands (however, see the caveat about XREAD below).

Streams and Active-Active

Active-Active databases allow you to write to the same logical stream from more than one region. Streams are synchronized across the regions of an Active-Active database.

In the example below, we write to a stream concurrently from two regions. Notice that after syncing, both regions have identical streams:

Time Region 1 Region 2
t1 XADD messages * text hello XADD messages * text goodbye
t2 XRANGE messages - +
→ [1589929244828-1]
XRANGE messages - +
→ [1589929246795-2]
t3 — Sync — — Sync —
t4 XRANGE messages - +
→ [1589929244828-1, 1589929246795-2]
XRANGE messages - +
→ [1589929244828-1, 1589929246795-2]

Notice also that the synchronized streams contain no duplicate IDs. As long as you allow the database to generate your stream IDs, you’ll never have more than one stream entry with the same ID.

Note -
Open source Redis uses one radix tree (referred to as rax in the code base) to implement each stream. However, Active-Active databases implement a single logical stream using one rax per region. Each region adds entries only to its associated rax (but can remove entries from all rax trees). This means that XREAD and XREADGROUP iterate simultaneously over all rax trees and return the appropriate entry by comparing the entry IDs from each rax.

Conflict resolution

Active-Active databases use an “observed-remove” approach to automatically resolve potential conflicts.

With this approach, a delete only affects the locally observable data.

In the example below, a stream, x, is created at t1. At t3, the stream exists in two regions.

Time Region 1 Region 2
t1 XADD messages * text hello
t2 — Sync — — Sync —
t3 XRANGE messages - +
→ [1589929244828-1]
XRANGE messages - +
→ [1589929244828-2]
t4 DEL messages XADD messages * text goodbye
t5 — Sync — — Sync —
t6 XRANGE messages - +
→ [1589929246795-2]
XRANGE messages - +
→ [1589929246795-2]

At t4, the stream is deleted from Region 1. At the same time, an entry with ID ending in 3700 is added to the same stream at Region 2. After the sync, at t6, the entry with ID ending in 3700 exists in both regions. This is because that entry was not visible when the local stream was deleted at t4.

ID generation modes

Usually, you should allow Redis streams generate its own stream entry IDs. You do this by specifying * as the ID in calls to XADD. However, you can provide your own custom ID when adding entries to a stream.

Because Active-Active databases replicate asynchronously, providing your own IDs can create streams with duplicate IDs. This can occur when your write to the same stream from multiple regions.

Time Region 1 Region 2
t1 XADD x 100-1 f1 v1 XADD x 100-1 f1 v1
t2 — Sync — — Sync —
t3 XRANGE x - +
→ [100-1, 100-1]
XRANGE x - +
→ [100-1, 100-1]

In this scenario, two entries with the ID 100-1 are added at t1. After syncing, the stream x contains two entries with the same ID.

Note -
Stream IDs in open source Redis consist of two integers separated by a dash ('-'). When the server generates the ID, the first integer is the current time in milliseconds, and the second integer is a sequence number. So, the format for stream IDs is MS-SEQ.

To prevent duplicate IDs and to comply with the original Redis streams design, Active-Active databases provide three ID modes for XADD:

  1. Strict: In strict mode, XADD allows server-generated IDs (using the ‘*’ ID specifier) or IDs consisting only of the millisecond (MS) portion. When the millisecond portion of the ID is provided, the ID’s sequence number is calculated using the database’s region ID. This prevents duplicate IDs in the stream. Strict mode rejects full IDs (that is, IDs containing both milliseconds and a sequence number).
  2. Semi-strict: Semi-strict mode is just like strict mode except that it allows full IDs (MS-SEQ). Because it allows full IDs, duplicate IDs are possible in this mode.
  3. Liberal: XADD allows any monotonically ascending ID. When given the millisecond portion of the ID, the sequence number will be set to 0. This mode may also lead to duplicate IDs.

The default and recommended mode is strict, which prevents duplicate IDs

Warning -
Why do you want to prevent duplicate IDs? First, XDEL, XCLAIM, and other commands can affect more than one entry when duplicate IDs are present in a stream. Second, duplicate entries may be removed if a database is exported or renamed.

To change XADD’s ID generation mode, use the rladmin command-line utility:

Set strict mode:

rladmin> tune myCRDB crdb crdt_xadd_id_uniqueness_mode strict

Set semi-strict mode:

rladmin> tune myCRDB crdb crdt_xadd_id_uniqueness_mode semi-strict

Set liberal mode:

rladmin> tune myCRDB crdb crdt_xadd_id_uniqueness_mode liberal

Iterating a stream with XREAD

In open source Redis and in non-Active-Active databases, you can use XREAD to iterate over the entries in a Redis Stream. However, with an Active-Active database, XREAD may skip entries. This can happen when multiple regions write to the same stream.

In the example below, XREAD skips entry 115-2.

Time Region 1 Region 2
t1 XADD x 110 f1 v1 XADD x 115 f1 v1
t2 XADD x 120 f1 v1
t3 XADD x 130 f1 v1
t4 XREAD COUNT 2 STREAMS x 0
→ [110-1, 120-1]
t5 — Sync — — Sync —
t6 XREAD COUNT 2 STREAMS x 120
→ [130-1]
t7 XREAD STREAMS x 0
→[110-1, 115-2, 120-1, 130-1]
XREAD STREAMS x 0
→[110-1, 115-2, 120-1, 130-1]

You can use XREAD to reliably consume a stream only if all writes to the stream originate from a single region. Otherwise, you should use XREADGROUP, which always guarantees reliable stream consumption.

Consumer groups

Active-Active databases fully support consumer groups with Redis Streams. Here is an example of creating two consumer groups concurrently:

Time Region 1 Region 2
t1 XGROUP CREATE x g1 0 XGROUP CREATE x g2 0
t2 XINFO GROUPS x
→ [g1]
XINFO GROUPS x
→ [g2]
t3 — Sync — — Sync —
t4 XINFO GROUPS x
→ [g1, g2]
XINFO GROUPS x
→ [g1, g2]
Note -

Open source Redis uses one radix tree (rax) to hold the global pending entries list and another rax for each consumer’s PEL. The global PEL is a unification of all consumer PELs, which are disjoint.

An Active-Active database stream maintains a global PEL and a per-consumer PEL for each region.

When given an ID different from the special “>” ID, XREADGROUP iterates simultaneously over all of the PELs for all consumers. It returns the next entry by comparing entry IDs from the different PELs.

Conflict resolution

The “delete wins” approach is a way to automatically resolve conflicts with consumer groups. In case of concurrent consumer group operations, a delete will “win” over other concurrent operations on the same group.

In this example, the DEL at t4 deletes both the observed group1 and the non-observed group2:

Time Region 1 Region 2
t1 XGROUP CREATE x group1 0
t2 — Sync — — Sync —
t3 XINFO GROUPS x
→ [group1]
XINFO GROUPS x
→ [group1]
t4 DEL x | XGROUP CREATE x group2 0` |
t5 — Sync — — Sync —
t6 EXISTS x
→ 0
EXISTS x
→ 0

In this example, the XGROUP DESTROY at t4 affects both the observed g1 created in Region 1 and the non-observed g1 created in Region 3:

time Region 1 Region 2 Region 3
t1 XGROUP CREATE x g1 0
t2 — Sync — — Sync —
t3 XINFO GROUPS x
→ [g1]
XINFO GROUPS x
→ [g1]
XINFO GROUPS x
→ []
t4 XGROUP DESTROY x g1 XGROUP CREATE x g1 0
t5 — Sync — _— Sync — — Sync —
t6 EXISTS x
→ 0
EXISTS x
→ 0
EXISTS x
→ 0

Group replication

Calls to XREADGROUP and XACK change the state of a consumer group or consumer. However, it’s not efficient to replicate every change to a consumer or consumer group.

To maintain consumer groups in Active-Active databases with optimal performance:

  1. Group existence (CREATE/DESTROY) is replicated.
  2. Most XACK operations are replicated.
  3. Other operations, such as XGROUP, SETID, DELCONSUMER, are not replicated.

For example:

Time Region 1 Region 2
t1 XADD messages 110 text hello
t2 XGROUP CREATE messages group1 0
t3 XREADGROUP GROUP group1 Alice STREAMS messages >
→ [110-1]
t4 — Sync — — Sync —
t5 XRANGE messages - +
→ [110-1]
XRANGE messages - +
→ [110-1]
t6 XINFO GROUPS messages
→ [g1]
XINFO GROUPS messages
→ [g1]
t7 XINFO CONSUMERS messages
→ [Alice]
XINFO CONSUMERS messages
→ []
t8 XPENDING messages group1 - +
→ [110-1]
XPENDING messages group1 - +
→ []

Using XREADGROUP across regions can result in regions reading the same entries. This is due to the fact that Active-Active Streams is designed for at-least-once reads or a single consumer. As shown in the previous example, Region 2 is not aware of any consumer group activity, so redirecting the XREADGROUP traffic from Region 1 to Region 2 results in reading entries that have already been read.

Replication performance optimizations

Consumers acknowledge messages using the XACK command. Each ack effectively records the last consumed message. This can result in a lot of cross-region traffic. To reduce this traffic, we replicate XACK messages only when all of the read entries are acknowledged.

Time Region 1 Region 2 Explanation
t1 XADD x 110-0 f1 v1
t2 XADD x 120-0 f1 v1
t3 XADD x 130-0 f1 v1
t4 XGROUP CREATE x g1 0
t5 XREADGROUP GROUP g1 Alice STREAMS x >
→ [110-0, 120-0, 130-0]
t6 XACK g1 110-0
t7 — Sync — — Sync — 110-0 and its preceding entries (none) were acknowledged. We replicate an XACK effect for 110-0.
t8 XACK g1 130-0
t9 — Sync — — Sync — 130-0 was acknowledged, but not its preceding entries (120-0). We DO NOT replicate an XACK effect for 130-0
t10 XACK g1 120-0
t11 — Sync — — Sync — 120-0 and its preceding entries (110-0 through 130-0) were acknowledged. We replicate an XACK effect for 130-0.

In this scenario, if we redirect the XREADGROUP traffic from Region 1 to Region 2 we do not re-read entries 110-0, 120-0 and 130-0. This means that the XREADGROUP does not return already-acknowledged entries.

Guarantees

Unlike XREAD, XREADGOUP will never skip stream entries. In traffic redirection, XREADGROUP may return entries that have been read but not acknowledged. It may also even return entries that have already been acknowledged.

Summary

With Active-Active streams, you can write to the same logical stream from multiple regions. As a result, the behavior of Active-Active streams differs somewhat from the behavior you get with open source Redis. This is summarized below:

Stream commands

  1. When using the strict ID generation mode, XADD does not permit full stream entry IDs (that is, an ID containing both MS and SEQ).
  2. XREAD may skip entries when iterating a stream that is concurrently written to from more than one region. For reliable stream iteration, use XREADGROUP instead.
  3. XSETID fails when the new ID is less than current ID.

Consumer group notes

The following consumer group operations are replicated:

  1. Consecutive XACK operations
  2. Consumer group creation and deletion (that is, XGROUP CREATE and XGROUP DESTROY)

All other consumer group metadata is not replicated.

A few other notes:

  1. XGROUP SETID and DELCONSUMER are not replicated.
  2. Consumers exist locally (XREADGROUP creates a consumer implicitly).
  3. Renaming a stream (using RENAME) deletes all consumer group information.