by Example
Supercluster Arbiter in Topologies
This example demonstrates a lighter-weight alternative for satisfying the requirement of an odd number of servers (across clusters) described in the Supercluster with JetStream example.
Rather than deploying a full additional cluster, a single node cluster that is not JetStream-enabled can be deployed which connects to both clusters that are hosting the JetStream-enabled servers.
$ nbe run topologies/supercluster-arbiter/cliView the source code or learn how to run this example yourself
Code
#!/bin/bash
set -euo pipefail
NATS_URL="nats://localhost:4222"
Define the arbiter config which is a single node that defines gateway connections between each of the clusters.
cat <<- EOF > arbiter.conf
server_name: arbiter
port: 4228
http_port: 8224
jetstream: {
enabled: false
}
gateway {
name: arbiter
port: 7228
gateways: [
{name: east, urls: [
nats://localhost:7222,
nats://localhost:7223,
nats://localhost:7224,
]},
{name: west, urls: [
nats://localhost:7225,
nats://localhost:7226,
nats://localhost:7227,
]},
]
}
EOF
General shared config for each server in the clusters. JetStream enabled with the unique tag as well as basic accounts.
cat <<- EOF > cluster-shared.conf
jetstream: {
unique_tag: "az:"
}
accounts: {
\$SYS: {
users: [{user: sys, password: sys}]
}
APP: {
jetstream: true
users: [{user: user, password: user}]
}
}
EOF
Define the server configs for east cluster.
cat <<- EOF > "east-az1.conf"
server_name: east-az1
server_tags: [az:1]
port: 4222
http_port: 8222
include cluster-shared.conf
cluster: {
name: east
port: 6222
routes: [
nats-route://127.0.0.1:6222,
nats-route://127.0.0.1:6223,
nats-route://127.0.0.1:6224,
]
}
gateway {
name: east
port: 7222
}
EOF
cat <<- EOF > "east-az2.conf"
server_name: east-az2
server_tags: [az:2]
port: 4223
include cluster-shared.conf
cluster: {
name: east
port: 6223
routes: [
nats-route://127.0.0.1:6222,
nats-route://127.0.0.1:6223,
nats-route://127.0.0.1:6224,
]
}
gateway {
name: east
port: 7223
}
EOF
cat <<- EOF > "east-az3.conf"
server_name: east-az3
server_tags: [az:3]
port: 4224
include cluster-shared.conf
cluster: {
name: east
port: 6224
routes: [
nats-route://127.0.0.1:6222,
nats-route://127.0.0.1:6223,
nats-route://127.0.0.1:6224,
]
}
gateway {
name: east
port: 7224
}
EOF
Server configs for west cluster.
cat <<- EOF > "west-az1.conf"
server_name: west-az1
server_tags: [az:1]
port: 4225
http_port: 8223
include cluster-shared.conf
cluster: {
name: west
port: 6225
routes: [
nats-route://127.0.0.1:6225,
nats-route://127.0.0.1:6226,
nats-route://127.0.0.1:6227,
]
}
gateway {
name: west
port: 7225
}
EOF
cat <<- EOF > "west-az2.conf"
server_name: west-az2
server_tags: [az:2]
port: 4226
include cluster-shared.conf
cluster: {
name: west
port: 6226
routes: [
nats-route://127.0.0.1:6225,
nats-route://127.0.0.1:6226,
nats-route://127.0.0.1:6227,
]
}
gateway {
name: west
port: 7226
}
EOF
cat <<- EOF > "west-az3.conf"
server_name: west-az3
server_tags: [az:3]
port: 4227
include cluster-shared.conf
cluster: {
name: west
port: 6227
routes: [
nats-route://127.0.0.1:6225,
nats-route://127.0.0.1:6226,
nats-route://127.0.0.1:6227,
]
}
gateway {
name: west
port: 7227
}
EOF
Start all of the servers.
echo 'Starting the servers...'
nats-server -c arbiter.conf > /dev/null 2>&1 &
sleep 1
nats-server -c east-az1.conf > /dev/null 2>&1 &
sleep 1
nats-server -c east-az2.conf > /dev/null 2>&1 &
sleep 1
nats-server -c east-az3.conf > /dev/null 2>&1 &
sleep 1
nats-server -c west-az1.conf > /dev/null 2>&1 &
sleep 1
nats-server -c west-az2.conf > /dev/null 2>&1 &
sleep 1
nats-server -c west-az3.conf > /dev/null 2>&1 &
sleep 3
Wait until the servers up and healthy.
curl --fail --silent \
--retry 5 \
--retry-delay 1 \
http://localhost:8222/healthz > /dev/null
curl --fail --silent \
--retry 5 \
--retry-delay 1 \
http://localhost:8223/healthz > /dev/null
curl --fail --silent \
--retry 5 \
--retry-delay 1 \
http://localhost:8224/healthz > /dev/null
List the servers, show the JetStream report, and show the info of an arbitrary server in west.
nats --user sys --password sys server list
nats --user sys --password sys server report jetstream
nats --user sys --password sys server info west-az1
Connect to east (port 4222), but request that a stream be added to the west.
nats stream add \
--server nats://localhost:4222 \
--user user \
--password user \
--cluster=west \
--retention=limits \
--storage=file \
--replicas=3 \
--discard=old \
--dupe-window=2m \
--max-age=-1 \
--max-msgs=-1 \
--max-bytes=-1 \
--max-msg-size=-1 \
--max-msgs-per-subject=-1 \
--max-consumers=-1 \
--allow-rollup \
--no-deny-delete \
--no-deny-purge \
--subjects="orders.*" \
ORDERS
Report out the streams to confirm placement.
nats --user user --password user stream reportOutput
Starting the servers...
╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Server Overview │
├──────────┬─────────┬──────┬─────────┬─────┬───────┬──────┬────────┬─────┬─────────┬───────┬───────┬──────┬────────┬─────┤
│ Name │ Cluster │ Host │ Version │ JS │ Conns │ Subs │ Routes │ GWs │ Mem │ CPU % │ Cores │ Slow │ Uptime │ RTT │
├──────────┼─────────┼──────┼─────────┼─────┼───────┼──────┼────────┼─────┼─────────┼───────┼───────┼──────┼────────┼─────┤
│ east-az1 │ east │ 0 │ 2.9.15 │ yes │ 1 │ 154 │ 2 │ 2 │ 19 MiB │ 0 │ 8 │ 0 │ 8.03s │ 1ms │
│ east-az3 │ east │ 0 │ 2.9.15 │ yes │ 0 │ 154 │ 2 │ 2 │ 19 MiB │ 0 │ 8 │ 0 │ 6.02s │ 1ms │
│ arbiter │ arbiter │ 0 │ 2.9.15 │ no │ 0 │ 44 │ 0 │ 2 │ 16 MiB │ 0 │ 8 │ 0 │ 9.03s │ 1ms │
│ west-az1 │ west │ 0 │ 2.9.15 │ yes │ 0 │ 144 │ 2 │ 2 │ 17 MiB │ 0 │ 8 │ 0 │ 5.02s │ 1ms │
│ east-az2 │ east │ 0 │ 2.9.15 │ yes │ 0 │ 154 │ 2 │ 2 │ 17 MiB │ 0 │ 8 │ 0 │ 7.03s │ 1ms │
│ west-az3 │ west │ 0 │ 2.9.15 │ yes │ 0 │ 144 │ 2 │ 2 │ 19 MiB │ 1 │ 8 │ 0 │ 3.02s │ 1ms │
│ west-az2 │ west │ 0 │ 2.9.15 │ yes │ 0 │ 144 │ 2 │ 2 │ 18 MiB │ 0 │ 8 │ 0 │ 4.02s │ 1ms │
├──────────┼─────────┼──────┼─────────┼─────┼───────┼──────┼────────┼─────┼─────────┼───────┼───────┼──────┼────────┼─────┤
│ │ 3 │ 7 │ │ 6 │ 1 │ 938 │ │ │ 126 MiB │ │ │ 0 │ │ │
╰──────────┴─────────┴──────┴─────────┴─────┴───────┴──────┴────────┴─────┴─────────┴───────┴───────┴──────┴────────┴─────╯
╭────────────────────────────────────────────────────────────────────────────╮
│ Cluster Overview │
├─────────┬────────────┬───────────────────┬───────────────────┬─────────────┤
│ Cluster │ Node Count │ Outgoing Gateways │ Incoming Gateways │ Connections │
├─────────┼────────────┼───────────────────┼───────────────────┼─────────────┤
│ arbiter │ 1 │ 2 │ 6 │ 0 │
│ west │ 3 │ 6 │ 4 │ 0 │
│ east │ 3 │ 6 │ 4 │ 1 │
├─────────┼────────────┼───────────────────┼───────────────────┼─────────────┤
│ │ 7 │ 14 │ 14 │ 1 │
╰─────────┴────────────┴───────────────────┴───────────────────┴─────────────╯
╭──────────────────────────────────────────────────────────────────────────────────────────────────╮
│ JetStream Summary │
├───────────┬─────────┬─────────┬───────────┬──────────┬───────┬────────┬──────┬─────────┬─────────┤
│ Server │ Cluster │ Streams │ Consumers │ Messages │ Bytes │ Memory │ File │ API Req │ API Err │
├───────────┼─────────┼─────────┼───────────┼──────────┼───────┼────────┼──────┼─────────┼─────────┤
│ arbiter │ arbiter │ 0 │ 0 │ 0 │ 0 B │ 0 B │ 0 B │ 0 │ 0 │
│ east-az1* │ east │ 0 │ 0 │ 0 │ 0 B │ 0 B │ 0 B │ 0 │ 0 │
│ east-az2 │ east │ 0 │ 0 │ 0 │ 0 B │ 0 B │ 0 B │ 0 │ 0 │
│ east-az3 │ east │ 0 │ 0 │ 0 │ 0 B │ 0 B │ 0 B │ 0 │ 0 │
│ west-az3 │ west │ 0 │ 0 │ 0 │ 0 B │ 0 B │ 0 B │ 0 │ 0 │
│ west-az1 │ west │ 0 │ 0 │ 0 │ 0 B │ 0 B │ 0 B │ 0 │ 0 │
│ west-az2 │ west │ 0 │ 0 │ 0 │ 0 B │ 0 B │ 0 B │ 0 │ 0 │
├───────────┼─────────┼─────────┼───────────┼──────────┼───────┼────────┼──────┼─────────┼─────────┤
│ │ │ 0 │ 0 │ 0 │ 0 B │ 0 B │ 0 B │ 0 │ 0 │
╰───────────┴─────────┴─────────┴───────────┴──────────┴───────┴────────┴──────┴─────────┴─────────╯
╭────────────────────────────────────────────────────────────────╮
│ RAFT Meta Group Information │
├──────────┬──────────┬────────┬─────────┬────────┬────────┬─────┤
│ Name │ ID │ Leader │ Current │ Online │ Active │ Lag │
├──────────┼──────────┼────────┼─────────┼────────┼────────┼─────┤
│ east-az1 │ 6arG4jtP │ yes │ true │ true │ 0.00s │ 0 │
│ east-az2 │ Pl1TTxO3 │ │ true │ true │ 0.25s │ 0 │
│ east-az3 │ 5KQHX3nc │ │ true │ true │ 0.25s │ 0 │
│ west-az1 │ uZO7rd5U │ │ true │ true │ 0.25s │ 0 │
│ west-az2 │ hTiOv8JU │ │ true │ true │ 0.25s │ 0 │
│ west-az3 │ 8uI3CcsK │ │ true │ true │ 0.25s │ 0 │
╰──────────┴──────────┴────────┴─────────┴────────┴────────┴─────╯
Server information for west-az1 (NAIVZ6NMILTCUKGFW6ZPROMUZ4IKYKFFATLZGIFVPV6IEIMZAPT6LHQ2)
Process Details:
Version: 2.9.15
Git Commit:
Go Version: go1.19.4
Start Time: 2023-03-16 13:40:11.909535414 +0000 UTC
Uptime: 5s
Connection Details:
Auth Required: true
TLS Required: false
Host: 0.0.0.0:4225
Client URLs: 172.30.0.2:4225
172.30.0.2:4226
172.30.0.2:4227
JetStream:
Domain:
Storage Directory: /tmp/nats/jetstream
Max Memory: 23 GiB
Max File: 118 GiB
Active Acconts: 1
Memory In Use: 0 B
File In Use: 0 B
API Requests: 0
API Errors: 0
Limits:
Max Conn: 65536
Max Subs: 0
Max Payload: 1.0 MiB
TLS Timeout: 2s
Write Deadline: 10s
Statistics:
CPU Cores: 8 0.00%
Memory: 17 MiB
Connections: 0
Subscriptions: 144
Msgs: 67 in 108 out
Bytes: 50 KiB in 60 KiB out
Slow Consumers: 0
Cluster:
Name: west
Tags: az:1
Host: 0.0.0.0:6225
URLs: 127.0.0.1:6225
127.0.0.1:6226
127.0.0.1:6227
Stream ORDERS was created
Information for Stream ORDERS created 2023-03-16 13:40:17
Subjects: orders.*
Replicas: 3
Storage: File
Placement Cluster: west
Options:
Retention: Limits
Acknowledgements: true
Discard Policy: Old
Duplicate Window: 2m0s
Allows Msg Delete: true
Allows Purge: true
Allows Rollups: true
Limits:
Maximum Messages: unlimited
Maximum Per Subject: unlimited
Maximum Bytes: unlimited
Maximum Age: unlimited
Maximum Message Size: unlimited
Maximum Consumers: unlimited
Cluster Information:
Name: west
Leader: west-az3
Replica: west-az1, current, seen 0.00s ago
Replica: west-az2, current, seen 0.00s ago
State:
Messages: 0
Bytes: 0 B
FirstSeq: 0
LastSeq: 0
Active Consumers: 0
Obtaining Stream stats
╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Stream Report │
├────────┬─────────┬────────────────┬───────────┬──────────┬───────┬──────┬─────────┬───────────────────────────────┤
│ Stream │ Storage │ Placement │ Consumers │ Messages │ Bytes │ Lost │ Deleted │ Replicas │
├────────┼─────────┼────────────────┼───────────┼──────────┼───────┼──────┼─────────┼───────────────────────────────┤
│ ORDERS │ File │ cluster: west │ 0 │ 0 │ 0 B │ 0 │ 0 │ west-az1, west-az2, west-az3* │
╰────────┴─────────┴────────────────┴───────────┴──────────┴───────┴──────┴─────────┴───────────────────────────────╯
Recording
Note, playback is half speed to make it a bit easier to follow.