by Example
Regional and Cross Region Streams (Cluster) in Use Cases
This example shows and alternative to the [supercluster][1] version. Rather than a supercluster, this relies on a nine node cluster spread out across three regions and three availability zones each.
$ nbe run use-cases/cross-region-streams-cluster/cliView the source code or learn how to run this example yourself
Code
#!/bin/bash
set -euo pipefail
NATS_URL="nats://localhost:4222"
Create a shared configuration which enables JetStream and defines the
unique_tag option which enforces all replicas for a given stream or
consumer to be placed on nodes with different availability zones (AZ).
cat <<- EOF > shared.conf
jetstream: {
unique_tag: "az:"
}
accounts: {
\$SYS: {
users: [{user: sys, password: sys}]
}
APP: {
jetstream: true
users: [{user: user, password: user}]
}
}
EOF
Create the shared clustered config defining the seed routes.
cat <<- EOF > cluster.conf
name: c1
routes: [
nats-route://127.0.0.1:6222,
nats-route://127.0.0.1:6223,
nats-route://127.0.0.1:6224,
]
EOF
Define nine server configurations modeling a cluster with three nodes
in each region across three AZs. NATS does not currently support
declaring tags with logical OR (or exclusive OR), so valid combinations
can be precomputed as tags and then used when creating streams. In this
case, the xr: tag encodes the cluster/AZ combination while still adhering
to the unique_tag requirement.
Each index corresponds to a region (e.g us-east) and the value at the
index corresponds to an AZ, e.g. us-east4.
Thus a tag xr:231 can be read as a cross-region stream where one replica
is placed in region 1 in AZ 2, another in region 2 in AZ 3, and the third
replica in region 3 in AZ 1.
In addition to the xr: tags, the standard rg: (region) and az: (AZ)
tags are set to support regional streams which is demonstrated below.
cat <<- EOF > "rg1-az1.conf"
server_name: rg1-az1
server_tags: [rg:1, az:1, xr:123, xr:132]
port: 4222
http_port: 8222
include shared.conf
cluster: {
port: 6222
include cluster.conf
}
jetstream: {
store_dir: /tmp/nats/storage/rg1-az1
}
EOF
cat <<- EOF > "rg1-az2.conf"
server_name: rg1-az2
server_tags: [rg:1, az:2, xr:213, xr:231]
port: 4223
include shared.conf
cluster: {
port: 6223
include cluster.conf
}
jetstream: {
store_dir: /tmp/nats/storage/rg1-az2
}
EOF
cat <<- EOF > "rg1-az3.conf"
server_name: rg1-az3
server_tags: [rg:1, az:3, xr:312, xr:321]
port: 4224
include shared.conf
cluster: {
port: 6224
include cluster.conf
}
jetstream: {
store_dir: /tmp/nats/storage/rg1-az3
}
EOF
cat <<- EOF > "rg2-az1.conf"
server_name: rg2-az1
server_tags: [rg:2, az:1, xr:213, xr:312]
port: 4225
include shared.conf
cluster: {
port: 6225
include cluster.conf
}
jetstream: {
store_dir: /tmp/nats/storage/rg2-az1
}
EOF
cat <<- EOF > "rg2-az2.conf"
server_name: rg2-az2
server_tags: [rg:2, az:2, xr:123, xr:321]
port: 4226
include shared.conf
cluster: {
port: 6226
include cluster.conf
}
jetstream: {
store_dir: /tmp/nats/storage/rg2-az2
}
EOF
cat <<- EOF > "rg2-az3.conf"
server_name: rg2-az3
server_tags: [rg:2, az:3, xr:132, xr:231]
port: 4227
include shared.conf
cluster: {
port: 6227
include cluster.conf
}
jetstream: {
store_dir: /tmp/nats/storage/rg2-az3
}
EOF
cat <<- EOF > "rg3-az1.conf"
server_name: rg3-az1
server_tags: [rg:3, az:1, xr:231, xr:321]
port: 4228
include shared.conf
cluster: {
port: 6228
include cluster.conf
}
jetstream: {
store_dir: /tmp/nats/storage/rg3-az1
}
EOF
cat <<- EOF > "rg3-az2.conf"
server_name: rg3-az2
server_tags: [rg:3, az:2, xr:132, xr:312]
port: 4229
include shared.conf
cluster: {
port: 6229
include cluster.conf
}
jetstream: {
store_dir: /tmp/nats/storage/rg3-az2
}
EOF
cat <<- EOF > "rg3-az3.conf"
server_name: rg3-az3
server_tags: [rg:3, az:3, xr:123, xr:213]
port: 4230
include shared.conf
cluster: {
port: 6230
include cluster.conf
}
jetstream: {
store_dir: /tmp/nats/storage/rg3-az3
}
EOF
Start a server for each configuration and sleep a second in between so the seeds can startup and get healthy.
for c in $(ls rg*.conf); do
echo "Starting server ${c%.*}"
nats-server -c $c > /dev/null 2>&1 &
sleep 1
done
Wait until the servers up and healthy.
echo 'Healthy?'
curl --fail --silent \
--retry 5 \
--retry-delay 1 \
http://localhost:8222/healthz; echo
Show the server lit and JetStream report.
nats --user sys --password sys server list
nats --user sys --password sys server report jetstream
Create a cross-region stream using one of the pre-defined xr: tags.
nats --user user --password user stream add \
--tag=xr:123 \
--retention=limits \
--storage=file \
--replicas=3 \
--discard=old \
--dupe-window=2m \
--max-age=-1 \
--max-msgs=-1 \
--max-bytes=-1 \
--max-msg-size=-1 \
--max-msgs-per-subject=-1 \
--max-consumers=-1 \
--allow-rollup \
--no-deny-delete \
--no-deny-purge \
--subjects="events.*" \
EVENTS
Create a regional stream which relies on the unique_tag to ensure
each replicas in a different AZ. This will create a stream in region 2
due to the rg:2 tag
nats --user user --password user stream add \
--tag=rg:2 \
--retention=limits \
--storage=file \
--replicas=3 \
--discard=old \
--dupe-window=2m \
--max-age=-1 \
--max-msgs=-1 \
--max-bytes=-1 \
--max-msg-size=-1 \
--max-msgs-per-subject=-1 \
--max-consumers=-1 \
--allow-rollup \
--no-deny-delete \
--no-deny-purge \
--subjects="orders.*" \
ORDERS
Report out the streams.
nats --user user --password user stream reportOutput
Starting server rg1-az1
Starting server rg1-az2
Starting server rg1-az3
Starting server rg2-az1
Starting server rg2-az2
Starting server rg2-az3
Starting server rg3-az1
Starting server rg3-az2
Starting server rg3-az3
Healthy?
{"status":"ok"}
╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Server Overview │
├─────────┬─────────┬──────┬─────────┬─────┬───────┬───────┬────────┬─────┬─────────┬───────┬───────┬──────┬────────┬─────┤
│ Name │ Cluster │ Host │ Version │ JS │ Conns │ Subs │ Routes │ GWs │ Mem │ CPU % │ Cores │ Slow │ Uptime │ RTT │
├─────────┼─────────┼──────┼─────────┼─────┼───────┼───────┼────────┼─────┼─────────┼───────┼───────┼──────┼────────┼─────┤
│ rg1-az1 │ c1 │ 0 │ 2.10.1 │ yes │ 1 │ 508 │ 32 │ 0 │ 18 MiB │ 3 │ 8 │ 0 │ 9.07s │ 1ms │
│ rg1-az3 │ c1 │ 0 │ 2.10.1 │ yes │ 0 │ 508 │ 32 │ 0 │ 17 MiB │ 3 │ 8 │ 0 │ 7.04s │ 2ms │
│ rg3-az2 │ c1 │ 0 │ 2.10.1 │ yes │ 0 │ 508 │ 32 │ 0 │ 17 MiB │ 4 │ 8 │ 0 │ 2.03s │ 2ms │
│ rg3-az1 │ c1 │ 0 │ 2.10.1 │ yes │ 0 │ 508 │ 32 │ 0 │ 17 MiB │ 2 │ 8 │ 0 │ 3.03s │ 2ms │
│ rg1-az2 │ c1 │ 0 │ 2.10.1 │ yes │ 0 │ 508 │ 32 │ 0 │ 17 MiB │ 1 │ 8 │ 0 │ 8.06s │ 2ms │
│ rg2-az1 │ c1 │ 0 │ 2.10.1 │ yes │ 0 │ 508 │ 32 │ 0 │ 17 MiB │ 3 │ 8 │ 0 │ 6.05s │ 2ms │
│ rg2-az3 │ c1 │ 0 │ 2.10.1 │ yes │ 0 │ 508 │ 32 │ 0 │ 17 MiB │ 2 │ 8 │ 0 │ 4.03s │ 2ms │
│ rg2-az2 │ c1 │ 0 │ 2.10.1 │ yes │ 0 │ 508 │ 32 │ 0 │ 17 MiB │ 2 │ 8 │ 0 │ 5.04s │ 2ms │
│ rg3-az3 │ c1 │ 0 │ 2.10.1 │ yes │ 0 │ 508 │ 32 │ 0 │ 16 MiB │ 6 │ 8 │ 0 │ 1.02s │ 2ms │
├─────────┼─────────┼──────┼─────────┼─────┼───────┼───────┼────────┼─────┼─────────┼───────┼───────┼──────┼────────┼─────┤
│ │ 1 │ 9 │ │ 9 │ 1 │ 4,572 │ │ │ 154 MIB │ │ │ 0 │ │ │
╰─────────┴─────────┴──────┴─────────┴─────┴───────┴───────┴────────┴─────┴─────────┴───────┴───────┴──────┴────────┴─────╯
╭────────────────────────────────────────────────────────────────────────────╮
│ Cluster Overview │
├─────────┬────────────┬───────────────────┬───────────────────┬─────────────┤
│ Cluster │ Node Count │ Outgoing Gateways │ Incoming Gateways │ Connections │
├─────────┼────────────┼───────────────────┼───────────────────┼─────────────┤
│ c1 │ 9 │ 0 │ 0 │ 1 │
├─────────┼────────────┼───────────────────┼───────────────────┼─────────────┤
│ │ 9 │ 0 │ 0 │ 1 │
╰─────────┴────────────┴───────────────────┴───────────────────┴─────────────╯
╭─────────────────────────────────────────────────────────────────────────────────────────────────╮
│ JetStream Summary │
├──────────┬─────────┬─────────┬───────────┬──────────┬───────┬────────┬──────┬─────────┬─────────┤
│ Server │ Cluster │ Streams │ Consumers │ Messages │ Bytes │ Memory │ File │ API Req │ API Err │
├──────────┼─────────┼─────────┼───────────┼──────────┼───────┼────────┼──────┼─────────┼─────────┤
│ rg1-az1* │ c1 │ 0 │ 0 │ 0 │ 0 B │ 0 B │ 0 B │ 0 │ 0 │
│ rg1-az2 │ c1 │ 0 │ 0 │ 0 │ 0 B │ 0 B │ 0 B │ 0 │ 0 │
│ rg1-az3 │ c1 │ 0 │ 0 │ 0 │ 0 B │ 0 B │ 0 B │ 0 │ 0 │
│ rg2-az1 │ c1 │ 0 │ 0 │ 0 │ 0 B │ 0 B │ 0 B │ 0 │ 0 │
│ rg2-az2 │ c1 │ 0 │ 0 │ 0 │ 0 B │ 0 B │ 0 B │ 0 │ 0 │
│ rg2-az3 │ c1 │ 0 │ 0 │ 0 │ 0 B │ 0 B │ 0 B │ 0 │ 0 │
│ rg3-az1 │ c1 │ 0 │ 0 │ 0 │ 0 B │ 0 B │ 0 B │ 0 │ 0 │
│ rg3-az2 │ c1 │ 0 │ 0 │ 0 │ 0 B │ 0 B │ 0 B │ 0 │ 0 │
│ rg3-az3 │ c1 │ 0 │ 0 │ 0 │ 0 B │ 0 B │ 0 B │ 0 │ 0 │
├──────────┼─────────┼─────────┼───────────┼──────────┼───────┼────────┼──────┼─────────┼─────────┤
│ │ │ 0 │ 0 │ 0 │ 0 B │ 0 B │ 0 B │ 0 │ 0 │
╰──────────┴─────────┴─────────┴───────────┴──────────┴───────┴────────┴──────┴─────────┴─────────╯
╭───────────────────────────────────────────────────────────────╮
│ RAFT Meta Group Information │
├─────────┬──────────┬────────┬─────────┬────────┬────────┬─────┤
│ Name │ ID │ Leader │ Current │ Online │ Active │ Lag │
├─────────┼──────────┼────────┼─────────┼────────┼────────┼─────┤
│ rg1-az1 │ 6s4h5MOv │ yes │ true │ true │ 0s │ 0 │
│ rg1-az2 │ X8wPeL6S │ │ true │ true │ 950ms │ 0 │
│ rg1-az3 │ FozYtzby │ │ true │ true │ 949ms │ 0 │
│ rg2-az1 │ 8ShHFKtZ │ │ true │ true │ 946ms │ 0 │
│ rg2-az2 │ Kx5WF0Q6 │ │ true │ true │ 950ms │ 0 │
│ rg2-az3 │ NSUWzTzT │ │ true │ true │ 946ms │ 0 │
│ rg3-az1 │ OEQTh4pP │ │ true │ true │ 950ms │ 0 │
│ rg3-az2 │ LJVb57VO │ │ true │ true │ 950ms │ 0 │
│ rg3-az3 │ uEtIi639 │ │ true │ true │ 950ms │ 0 │
╰─────────┴──────────┴────────┴─────────┴────────┴────────┴─────╯
Stream EVENTS was created
Information for Stream EVENTS created 2023-10-23 19:31:50
Subjects: events.*
Replicas: 3
Storage: File
Placement Tags: xr:123
Options:
Retention: Limits
Acknowledgments: true
Discard Policy: Old
Duplicate Window: 2m0s
Direct Get: true
Allows Msg Delete: true
Allows Purge: true
Allows Rollups: true
Limits:
Maximum Messages: unlimited
Maximum Per Subject: unlimited
Maximum Bytes: unlimited
Maximum Age: unlimited
Maximum Message Size: unlimited
Maximum Consumers: unlimited
Cluster Information:
Name: c1
Leader: rg2-az2
Replica: rg1-az1, current, seen 716µs ago
Replica: rg3-az3, current, seen 198µs ago
State:
Messages: 0
Bytes: 0 B
First Sequence: 0
Last Sequence: 0
Active Consumers: 0
Stream ORDERS was created
Information for Stream ORDERS created 2023-10-23 19:31:50
Subjects: orders.*
Replicas: 3
Storage: File
Placement Tags: rg:2
Options:
Retention: Limits
Acknowledgments: true
Discard Policy: Old
Duplicate Window: 2m0s
Direct Get: true
Allows Msg Delete: true
Allows Purge: true
Allows Rollups: true
Limits:
Maximum Messages: unlimited
Maximum Per Subject: unlimited
Maximum Bytes: unlimited
Maximum Age: unlimited
Maximum Message Size: unlimited
Maximum Consumers: unlimited
Cluster Information:
Name: c1
Leader: rg2-az3
Replica: rg2-az1, current, seen 45µs ago
Replica: rg2-az2, current, seen 1ms ago
State:
Messages: 0
Bytes: 0 B
First Sequence: 0
Last Sequence: 0
Active Consumers: 0
Obtaining Stream stats
╭──────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Stream Report │
├────────┬─────────┬──────────────┬───────────┬──────────┬───────┬──────┬─────────┬────────────────────────────┤
│ Stream │ Storage │ Placement │ Consumers │ Messages │ Bytes │ Lost │ Deleted │ Replicas │
├────────┼─────────┼──────────────┼───────────┼──────────┼───────┼──────┼─────────┼────────────────────────────┤
│ EVENTS │ File │ tags: xr:123 │ 0 │ 0 │ 0 B │ 0 │ 0 │ rg1-az1, rg2-az2*, rg3-az3 │
│ ORDERS │ File │ tags: rg:2 │ 0 │ 0 │ 0 B │ 0 │ 0 │ rg2-az1, rg2-az2, rg2-az3* │
╰────────┴─────────┴──────────────┴───────────┴──────────┴───────┴──────┴─────────┴────────────────────────────╯
Recording
Note, playback is half speed to make it a bit easier to follow.