NATS Logo by Example

Supercluster Arbiter in Topologies

This example demonstrates a lighter-weight alternative for satisfying the requirement of an odd number of servers (across clusters) described in the Supercluster with JetStream example.

Rather than deploying a full additional cluster, a single node cluster that is not JetStream-enabled can be deployed which connects to both clusters that are hosting the JetStream-enabled servers.

CLI Go Python JavaScript Rust C# C#2 Java Ruby Elixir Crystal C
Jump to the output or the recording
$ nbe run topologies/supercluster-arbiter/cli
View the source code or learn how to run this example yourself

Code

#!/bin/bash


set -euo pipefail


NATS_URL="nats://localhost:4222"

Define the arbiter config which is a single node that defines gateway connections between each of the clusters.

cat <<- EOF > arbiter.conf
server_name: arbiter
port: 4228
http_port: 8224


jetstream: {
  enabled: false
}


gateway {
  name: arbiter
  port: 7228
  gateways: [
    {name: east, urls: [
      nats://localhost:7222,
      nats://localhost:7223,
      nats://localhost:7224,
    ]},
    {name: west, urls: [
      nats://localhost:7225,
      nats://localhost:7226,
      nats://localhost:7227,
    ]},
  ]
}
EOF

General shared config for each server in the clusters. JetStream enabled with the unique tag as well as basic accounts.

cat <<- EOF > cluster-shared.conf
jetstream: {
  unique_tag: "az:"
}
accounts: {
  \$SYS: {
    users: [{user: sys, password: sys}]
  }
  APP: {
    jetstream: true
    users: [{user: user, password: user}]
  }
}
EOF

Define the server configs for east cluster.

cat <<- EOF > "east-az1.conf"
server_name: east-az1
server_tags: [az:1]
port: 4222
http_port: 8222
include cluster-shared.conf
cluster: {
  name: east
  port: 6222
  routes: [
    nats-route://127.0.0.1:6222,
    nats-route://127.0.0.1:6223,
    nats-route://127.0.0.1:6224,
  ]
}
gateway {
  name: east
  port: 7222
}
EOF


cat <<- EOF > "east-az2.conf"
server_name: east-az2
server_tags: [az:2]
port: 4223
include cluster-shared.conf
cluster: {
  name: east
  port: 6223
  routes: [
    nats-route://127.0.0.1:6222,
    nats-route://127.0.0.1:6223,
    nats-route://127.0.0.1:6224,
  ]
}
gateway {
  name: east
  port: 7223
}
EOF


cat <<- EOF > "east-az3.conf"
server_name: east-az3
server_tags: [az:3]
port: 4224
include cluster-shared.conf
cluster: {
  name: east
  port: 6224
  routes: [
    nats-route://127.0.0.1:6222,
    nats-route://127.0.0.1:6223,
    nats-route://127.0.0.1:6224,
  ]
}
gateway {
  name: east
  port: 7224
}
EOF

Server configs for west cluster.

cat <<- EOF > "west-az1.conf"
server_name: west-az1
server_tags: [az:1]
port: 4225
http_port: 8223
include cluster-shared.conf
cluster: {
  name: west
  port: 6225
  routes: [
    nats-route://127.0.0.1:6225,
    nats-route://127.0.0.1:6226,
    nats-route://127.0.0.1:6227,
  ]
}
gateway {
  name: west
  port: 7225
}
EOF


cat <<- EOF > "west-az2.conf"
server_name: west-az2
server_tags: [az:2]
port: 4226
include cluster-shared.conf
cluster: {
  name: west
  port: 6226
  routes: [
    nats-route://127.0.0.1:6225,
    nats-route://127.0.0.1:6226,
    nats-route://127.0.0.1:6227,
  ]
}
gateway {
  name: west
  port: 7226
}
EOF


cat <<- EOF > "west-az3.conf"
server_name: west-az3
server_tags: [az:3]
port: 4227
include cluster-shared.conf
cluster: {
  name: west
  port: 6227
  routes: [
    nats-route://127.0.0.1:6225,
    nats-route://127.0.0.1:6226,
    nats-route://127.0.0.1:6227,
  ]
}
gateway {
  name: west
  port: 7227
}
EOF

Start all of the servers.

echo 'Starting the servers...'
nats-server -c arbiter.conf > /dev/null 2>&1 &
sleep 1


nats-server -c east-az1.conf > /dev/null 2>&1 &
sleep 1


nats-server -c east-az2.conf > /dev/null 2>&1 &
sleep 1


nats-server -c east-az3.conf > /dev/null 2>&1 &
sleep 1


nats-server -c west-az1.conf > /dev/null 2>&1 &
sleep 1


nats-server -c west-az2.conf > /dev/null 2>&1 &
sleep 1


nats-server -c west-az3.conf > /dev/null 2>&1 &
sleep 3

Wait until the servers up and healthy.

curl --fail --silent \
  --retry 5 \
  --retry-delay 1 \
  http://localhost:8222/healthz > /dev/null


curl --fail --silent \
  --retry 5 \
  --retry-delay 1 \
  http://localhost:8223/healthz > /dev/null


curl --fail --silent \
  --retry 5 \
  --retry-delay 1 \
  http://localhost:8224/healthz > /dev/null

List the servers, show the JetStream report, and show the info of an arbitrary server in west.

nats --user sys --password sys server list
nats --user sys --password sys server report jetstream
nats --user sys --password sys server info west-az1

Connect to east (port 4222), but request that a stream be added to the west.

nats stream add \
  --server nats://localhost:4222 \
  --user user \
  --password user \
  --cluster=west \
  --retention=limits \
  --storage=file \
  --replicas=3 \
  --discard=old \
  --dupe-window=2m \
  --max-age=-1 \
  --max-msgs=-1 \
  --max-bytes=-1 \
  --max-msg-size=-1 \
  --max-msgs-per-subject=-1 \
  --max-consumers=-1 \
  --allow-rollup \
  --no-deny-delete \
  --no-deny-purge \
  --subjects="orders.*" \
  ORDERS

Report out the streams to confirm placement.

nats --user user --password user stream report

Output

Starting the servers...
╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│                                                     Server Overview                                                     │
├──────────┬─────────┬──────┬─────────┬─────┬───────┬──────┬────────┬─────┬─────────┬───────┬───────┬──────┬────────┬─────┤
│ Name     │ Cluster │ Host │ Version │ JS  │ Conns │ Subs │ Routes │ GWs │ Mem     │ CPU % │ Cores │ Slow │ Uptime │ RTT │
├──────────┼─────────┼──────┼─────────┼─────┼───────┼──────┼────────┼─────┼─────────┼───────┼───────┼──────┼────────┼─────┤
│ east-az1 │ east    │ 0    │ 2.9.15  │ yes │ 1     │ 154  │ 2      │ 2   │ 19 MiB  │ 0     │ 8     │ 0    │ 8.03s  │ 1ms │
│ east-az3 │ east    │ 0    │ 2.9.15  │ yes │ 0     │ 154  │ 2      │ 2   │ 19 MiB  │ 0     │ 8     │ 0    │ 6.02s  │ 1ms │
│ arbiter  │ arbiter │ 0    │ 2.9.15  │ no  │ 0     │ 44   │ 0      │ 2   │ 16 MiB  │ 0     │ 8     │ 0    │ 9.03s  │ 1ms │
│ west-az1 │ west    │ 0    │ 2.9.15  │ yes │ 0     │ 144  │ 2      │ 2   │ 17 MiB  │ 0     │ 8     │ 0    │ 5.02s  │ 1ms │
│ east-az2 │ east    │ 0    │ 2.9.15  │ yes │ 0     │ 154  │ 2      │ 2   │ 17 MiB  │ 0     │ 8     │ 0    │ 7.03s  │ 1ms │
│ west-az3 │ west    │ 0    │ 2.9.15  │ yes │ 0     │ 144  │ 2      │ 2   │ 19 MiB  │ 1     │ 8     │ 0    │ 3.02s  │ 1ms │
│ west-az2 │ west    │ 0    │ 2.9.15  │ yes │ 0     │ 144  │ 2      │ 2   │ 18 MiB  │ 0     │ 8     │ 0    │ 4.02s  │ 1ms │
├──────────┼─────────┼──────┼─────────┼─────┼───────┼──────┼────────┼─────┼─────────┼───────┼───────┼──────┼────────┼─────┤
│          │ 3       │ 7    │         │ 6   │ 1     │ 938  │        │     │ 126 MiB │       │       │ 0    │        │     │
╰──────────┴─────────┴──────┴─────────┴─────┴───────┴──────┴────────┴─────┴─────────┴───────┴───────┴──────┴────────┴─────╯

╭────────────────────────────────────────────────────────────────────────────╮
│                              Cluster Overview                              │
├─────────┬────────────┬───────────────────┬───────────────────┬─────────────┤
│ Cluster │ Node Count │ Outgoing Gateways │ Incoming Gateways │ Connections │
├─────────┼────────────┼───────────────────┼───────────────────┼─────────────┤
│ arbiter │ 1          │ 2                 │ 6                 │ 0           │
│ west    │ 3          │ 6                 │ 4                 │ 0           │
│ east    │ 3          │ 6                 │ 4                 │ 1           │
├─────────┼────────────┼───────────────────┼───────────────────┼─────────────┤
│         │ 7          │ 14                │ 14                │ 1           │
╰─────────┴────────────┴───────────────────┴───────────────────┴─────────────╯
╭──────────────────────────────────────────────────────────────────────────────────────────────────╮
│                                        JetStream Summary                                         │
├───────────┬─────────┬─────────┬───────────┬──────────┬───────┬────────┬──────┬─────────┬─────────┤
│ Server    │ Cluster │ Streams │ Consumers │ Messages │ Bytes │ Memory │ File │ API Req │ API Err │
├───────────┼─────────┼─────────┼───────────┼──────────┼───────┼────────┼──────┼─────────┼─────────┤
│ arbiter   │ arbiter │ 0       │ 0         │ 0        │ 0 B   │ 0 B    │ 0 B  │ 0       │ 0       │
│ east-az1* │ east    │ 0       │ 0         │ 0        │ 0 B   │ 0 B    │ 0 B  │ 0       │ 0       │
│ east-az2  │ east    │ 0       │ 0         │ 0        │ 0 B   │ 0 B    │ 0 B  │ 0       │ 0       │
│ east-az3  │ east    │ 0       │ 0         │ 0        │ 0 B   │ 0 B    │ 0 B  │ 0       │ 0       │
│ west-az3  │ west    │ 0       │ 0         │ 0        │ 0 B   │ 0 B    │ 0 B  │ 0       │ 0       │
│ west-az1  │ west    │ 0       │ 0         │ 0        │ 0 B   │ 0 B    │ 0 B  │ 0       │ 0       │
│ west-az2  │ west    │ 0       │ 0         │ 0        │ 0 B   │ 0 B    │ 0 B  │ 0       │ 0       │
├───────────┼─────────┼─────────┼───────────┼──────────┼───────┼────────┼──────┼─────────┼─────────┤
│           │         │ 0       │ 0         │ 0        │ 0 B   │ 0 B    │ 0 B  │ 0       │ 0       │
╰───────────┴─────────┴─────────┴───────────┴──────────┴───────┴────────┴──────┴─────────┴─────────╯

╭────────────────────────────────────────────────────────────────╮
│                  RAFT Meta Group Information                   │
├──────────┬──────────┬────────┬─────────┬────────┬────────┬─────┤
│ Name     │ ID       │ Leader │ Current │ Online │ Active │ Lag │
├──────────┼──────────┼────────┼─────────┼────────┼────────┼─────┤
│ east-az1 │ 6arG4jtP │ yes    │ true    │ true   │ 0.00s  │ 0   │
│ east-az2 │ Pl1TTxO3 │        │ true    │ true   │ 0.25s  │ 0   │
│ east-az3 │ 5KQHX3nc │        │ true    │ true   │ 0.25s  │ 0   │
│ west-az1 │ uZO7rd5U │        │ true    │ true   │ 0.25s  │ 0   │
│ west-az2 │ hTiOv8JU │        │ true    │ true   │ 0.25s  │ 0   │
│ west-az3 │ 8uI3CcsK │        │ true    │ true   │ 0.25s  │ 0   │
╰──────────┴──────────┴────────┴─────────┴────────┴────────┴─────╯
Server information for west-az1 (NAIVZ6NMILTCUKGFW6ZPROMUZ4IKYKFFATLZGIFVPV6IEIMZAPT6LHQ2)

Process Details:

         Version: 2.9.15
      Git Commit: 
      Go Version: go1.19.4
      Start Time: 2023-03-16 13:40:11.909535414 +0000 UTC
          Uptime: 5s

Connection Details:

   Auth Required: true
    TLS Required: false
            Host: 0.0.0.0:4225
     Client URLs: 172.30.0.2:4225
                  172.30.0.2:4226
                  172.30.0.2:4227

JetStream:

              Domain: 
   Storage Directory: /tmp/nats/jetstream
          Max Memory: 23 GiB
            Max File: 118 GiB
      Active Acconts: 1
       Memory In Use: 0 B
         File In Use: 0 B
        API Requests: 0
          API Errors: 0

Limits:

        Max Conn: 65536
        Max Subs: 0
     Max Payload: 1.0 MiB
     TLS Timeout: 2s
  Write Deadline: 10s

Statistics:

       CPU Cores: 8 0.00%
          Memory: 17 MiB
     Connections: 0
   Subscriptions: 144
            Msgs: 67 in 108 out
           Bytes: 50 KiB in 60 KiB out
  Slow Consumers: 0

Cluster:

            Name: west
            Tags: az:1
            Host: 0.0.0.0:6225
            URLs: 127.0.0.1:6225
                  127.0.0.1:6226
                  127.0.0.1:6227

Stream ORDERS was created

Information for Stream ORDERS created 2023-03-16 13:40:17

             Subjects: orders.*
             Replicas: 3
              Storage: File
    Placement Cluster: west

Options:

            Retention: Limits
     Acknowledgements: true
       Discard Policy: Old
     Duplicate Window: 2m0s
    Allows Msg Delete: true
         Allows Purge: true
       Allows Rollups: true

Limits:

     Maximum Messages: unlimited
  Maximum Per Subject: unlimited
        Maximum Bytes: unlimited
          Maximum Age: unlimited
 Maximum Message Size: unlimited
    Maximum Consumers: unlimited


Cluster Information:

                 Name: west
               Leader: west-az3
              Replica: west-az1, current, seen 0.00s ago
              Replica: west-az2, current, seen 0.00s ago

State:

             Messages: 0
                Bytes: 0 B
             FirstSeq: 0
              LastSeq: 0
     Active Consumers: 0
Obtaining Stream stats

╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│                                                   Stream Report                                                   │
├────────┬─────────┬────────────────┬───────────┬──────────┬───────┬──────┬─────────┬───────────────────────────────┤
│ Stream │ Storage │ Placement      │ Consumers │ Messages │ Bytes │ Lost │ Deleted │ Replicas                      │
├────────┼─────────┼────────────────┼───────────┼──────────┼───────┼──────┼─────────┼───────────────────────────────┤
│ ORDERS │ File    │ cluster: west  │ 0         │ 0        │ 0 B   │ 0    │ 0       │ west-az1, west-az2, west-az3* │
╰────────┴─────────┴────────────────┴───────────┴──────────┴───────┴──────┴─────────┴───────────────────────────────╯

Recording

Note, playback is half speed to make it a bit easier to follow.