Appearance
Clustering
Clusto supports decentralized clustering where up to ~5 clustod nodes form a peer group with leaderless, eventually-consistent state replication. Every node is its own authority: it applies writes locally and instantly, then gossips them to all connected peers, which merge them. There is no leader, no election, and no quorum - the cluster stays usable with any number of nodes online (1..5, or even all-down then cold-restart).
Overview
- Leaderless replication: every node applies its own writes locally and gossips them to all peers. Peers merge incoming writes but never re-broadcast them (Clusto is a full mesh, so one hop reaches everyone).
- Conflict resolution: projects and teams use last-writer-wins by a per-object
updated_attimestamp; sessions use a generation-counter + tombstone-TTL scheme (owner-authoritative). Distinct objects are never lost on merge. - Anti-entropy on connect: when a peer connects, both sides exchange a fingerprint per record class. Only the classes that differ are compared record by record, and only the records one side lacks or holds older travel, so a node that was offline catches up the moment it reconnects, and a reconnect with nothing changed costs two small frames instead of the whole state (about 12 kB against about 15 MB on a large cluster). A restarting node dials its peers only after it has loaded its own state, so its first exchange never advertises an empty node. Join-time catch-up syncs (provider accounts, cluster feature settings, node commands) go only to the peer that joined, not to the whole cluster.
- Restarts are detected: each node process announces an instance id in the peer handshake. When a peer reconnects with a new instance id, requests still waiting on its previous process fail at once instead of timing out, and do not count against the new process.
- Client failover: clients connect to any node via the multi-node connection list; every node serves reads and writes locally.
- No process migration: agent/PTY processes are OS-local. A session is owned by the node running its process; if that node is offline the session shows as unreachable (its metadata is still replicated) until the owner returns.
Why leaderless? Clusto's availability requirement (any node may be offline at any time, often only 1-2 online) is an AP requirement. Consensus (Raft) is CP and cannot make progress in a minority/solo node by design, so leader election was the wrong tool. Going leaderless permanently removes the entire election bug class (split-vote livelock, catch-up wedge, phantom-leader, lonely-takeover).
How It Works
Node Roles
There are no distinct roles. Every node is a co-equal authority for its own writes. The per-peer role field in cluster status is vestigial and reports every node as Leader; there is no leader, term, or leader address.
Startup Sequence
- Node starts and is immediately its own authority (applies writes locally).
- Connects to configured peers over mutual TLS on the peer port (see Peer transport).
- On each peer connection (inbound or outbound), both sides send a state summary for anti-entropy and exchange only the records they are missing.
- Local writes apply instantly and gossip to whichever peers are connected; offline peers catch up via anti-entropy when they reconnect.
Peer transport
Node-to-node traffic on the peer port (default 9179) is always TLS 1.3 with mutual authentication: each node presents its identity key and the other side checks it. There is no plaintext option, so a peer link needs no certificate and works over any network, including the public internet or the Clusto Tunnel, which uses the same authentication. A node on another release, or anything that opens a plain HTTP connection to the peer port, is answered with 426 Upgrade Required.
After the TLS handshake each side proves it holds the cluster token without sending it: a keyed hash bound to both identities and the node id. The cluster token never crosses the network, and the proof cannot be replayed to a different node.
Each node pins its peers' identities in its persisted peer list and shares the pins with the rest of the cluster. A pin is learned from adoption, from the first contact that proves cluster membership, or from another node's roster. A pin learned from a roster only fills a gap: it never replaces a pin the node already holds, and a peer cannot change another node's pin. The cluster token is never synced between nodes; each node holds it from adoption or its own config. A peer that later presents a different key is refused and shown as offline with an Identity changed badge; that is never treated as a token mismatch, so it does not make the node leave the cluster. After a reinstall you did on purpose, use Trust new identity on the peer's card (see When an identity changes).
Who may control a node
By default every node of a cluster may control every other node: it can start and drive sessions, tasks and chats there, change its settings, and update or restart it. That suits one person's machines, but it means a laptop that joins the cluster can run code on your server. Directional trust lets a node limit who may control it, and it works in one direction only: a node that limits its controllers can still control the others.
Set it in the node's Configure Node dialog, May control this node: turn on Limit which nodes may control this node and tick the allowed nodes. The node itself always may. From a terminal use clusto node controllers, or set controllers in the node's [cluster] section (see Configuration). The list is stored on that node only and is never replicated, and a change applies at once without a restart. The default, no limit, is unchanged.
Once a node has a list, a node outside it can no longer:
- send terminal input, output or file subscriptions;
- rename, forget or remove nodes, update or restart the node, change its TLS or tunnel settings, or set its node commands and defaults;
- push any
Sync*message, including a forge account secret; - change the node's peers, pins, removals or tunnel through a cluster config sync (only the SSH vault key is still accepted);
- forward a request to it, except listing vaults, node metrics history, agent models and reading the controller list;
- replicate records that act on the node: sessions, projects and chats for this node or for no node, chat workspaces on this node, plugins installing here, Agent Docs assets and variables, lessons, and Factory records;
- sign in with the cluster token on the node's client port, including bearer HTTP.
In the other direction, the limited node no longer relays its task, task run, report and agent-idle updates to a node outside its list.
Because the cluster token is refused as a credential on a limited node, the app and the CLI must sign in to it with the node's own token (pair with it; see Node identity).
What a list does not stop:
- the node still receives replicated metadata: sessions, projects, SSH connection records and the SSH key, and time entries;
- a node outside the list can still rejoin the cluster with the cluster token;
- a flood of requests (denial of service);
- a controller that is itself compromised;
- time connectors and team records.
Trust is transitive. If an allowed controller can itself be controlled by a node outside your list, that node reaches yours through it. The dialog warns when this is the case: limit those nodes too.
A remote node refuses a list that leaves out the node you are connected to, and a node cannot remove itself from another node's list. To take the connected node out, change the setting on the target node itself.
Gossip and Anti-Entropy
apply_local_mutation (the single funnel every local write passes through) applies the mutation to local state and broadcasts the corresponding PeerMessage to all connected peers. Inbound replicated mutations (SessionUpserted / SessionRemoved / ProjectUpserted / ProjectDeleted / TeamUpserted / TeamRemoved) are applied to local state but not re-broadcast - this avoids gossip loops in the full mesh.
On PeerJoined, the node sends a StateSummary: one fingerprint per record class (sessions, projects, teams, chats, knowledge, time tracking, factory records and the other replicated classes). Because both sides send one, each compares it with its own. For every class whose fingerprint differs it sends a StateDigest, which lists id -> version for that class. The receiver of a digest answers with each record the sender lacks or holds older, one record per frame, as the same incremental message a live write uses (ProjectUpserted, ChatUpserted, ChatRemoved, and so on), and applies them the same way. After each batch the sender sends a StateSyncMark and waits for the StateSyncAck before the next, so a resync never outruns the node applying it. A chat travels without its transcript; the receiver pulls the content it lacks. Records are merged by the safe per-object rules below, never blindly overwritten. Equal fingerprints end the exchange for a class, so an unchanged reconnect costs one small frame each way.
Conflict Resolution (Safe Merge)
Two independent nodes each own their own state, so the per-node version counter is not comparable across nodes - the old "reject snapshot if its version is not newer" guard is gone. Merging works per object instead:
- Sessions: union. Incoming sessions honour the existing tombstone +
generationpunch-through rules (see Session Liveness Is Owner-Authoritative); a local session the peer lacks is kept. Tombstones are unioned (latestremoved_at, highest recorded generation). - Projects / Teams: union, last-writer-wins by
updated_aton a key collision; an exact-timestamp tie keeps the existing entry (deterministic). Every project/team create/update is stampedupdated_at = nowcentrally inapply_local_mutation, so a local write always wins over a stale gossip. IncrementalProjectUpserted/TeamUpsertedapply is LWW too - an out-of-order replicated upsert older than what we hold is dropped.
Practical-consistency note: for a single user, concurrent edits to the same object on two simultaneously-offline nodes are rare; in that rare case LWW silently drops one side. Distinct objects are always preserved.
State Replication
Replicated data includes:
- Session metadata (
SessionInfo: id, name, mode, working_dir, owner node, claim, timestamps) - Recent output history (for replay)
- Project settings (name, description, working_dir, pinned node, run-as users, default shell, timestamps,
updated_at) - Teams (members, status, results,
updated_at) - Chats, the SSH connection vault, Agent Docs, and approved knowledge. An Agent Docs deletion replicates as a tombstone record (last writer wins), and a restarted node seeds its Agent Docs records, tombstones, and template variables into cluster state from its database, so a stale peer snapshot cannot overwrite newer local records
Not replicated:
- Running agent/PTY processes (OS-local)
- Live broadcast channels (recreated on demand)
What replicates where
| Data | Reaches | Notes |
|---|---|---|
| Sessions, projects, teams | Every node | Metadata only; the process stays on its owner |
| Chats | Every node, except nodes a limited Notes vault leaves out | A chat follows the node it runs on |
| Notes vault | The nodes in its Kept on scope | Live task and run updates respect the same scope |
| Tasks and runs | The owning node, plus nodes that may see it | Not pushed to nodes a limited vault leaves out |
| Lessons (Knowledge) | Every node | Can be limited to nodes and OS accounts, so agents elsewhere do not see them |
| Reports, Agent Docs, SSH connections and credentials | Every node | Known limit: the vault boundary does not cover them |
| Forge account host lists | Set on the account's host | Changes made elsewhere are forwarded to the host |
Session Liveness Is Owner-Authoritative
Every session is owned by exactly one node - the node that holds the live PTY/SDK process. Only that owner may add, keep, or remove its own session:
- No node reaps a remote peer's sessions on transient signals. A dropped ownership announce, a brief link flap, or a disconnect does not remove the peer's sessions. Those processes keep running across a network blip; tearing their cluster state down on churn used to destroy live sessions. The session simply shows as unreachable until the owner returns.
- A departed node's sessions are pruned only by an explicit
remove_node. This trades lingering phantoms for a guarantee that churn never loses a live session. - Generation-stamped recovery. Each
SessionCreated/SessionUpdatedan owner emits carries a monotonicgeneration(seeded from wall-clock millis, so it increases across restarts too). ASessionRemovedrecords the removed generation as a tombstone. If a session is ever removed erroneously, the live owner re-asserts it with a higher generation that punches through the tombstone, restoring it to cluster state. Owners re-assert on reconnect, on receiving a snapshot, and on a periodic timer. - "Owner unknown" is not "dead." A node asked to deliver input to - or kill - a session it does not positively own (a PTY-less mirror, or an entry transiently absent from state) returns a soft, retryable error and never declares the session dead - so a routing hiccup cannot cascade into a cluster-wide removal. When a client claims a session before its ownership announcement has replicated, the connected node probes its connected peers and routes the claim to the peer holding the live process. This lets a just-created remote session become interactive without an
owner is not known yetretry loop. A kill request first waits briefly for ownership to settle (it can read as unknown during churn) before routing; only if it stays unknown does the soft error surface, which the client retries. Killing the local mirror in that window would emit aSessionRemovedthe live owner simply refuses and re-announces, falsely reporting success while the PTY survives.
Node Resource Metrics
Each daemon samples its own host resource usage - CPU %, memory, swap, and network throughput - every 10 seconds into a rolling 1-hour ring (360 samples, in memory; not persisted). Linux samples also include 1-minute load, runnable tasks, 10-second CPU/memory/I/O PSI, and the CPU and memory currently used by Clusto's workload cgroup. Collection runs on a dedicated OS thread so it never competes with the async runtimes.
Two derived views leave each node:
- A compact summary - the current sample plus 1-hour averages - is broadcast to peers every tick via
PeerMetricsAnnounce(aLossymessage on the replication lane) and carried in each node'sNodeInfo.metrics. A client gets a freshClusterStatuswith those readings on the same tick only while it watches them (WatchClusterMetrics): the app turns the feed on while the cluster view or the Settings cluster card is open, so the node cards show live CPU / memory / swap / network for every node, updating roughly every 10 seconds. An idle client, or one on another view, receivesClusterStatusonly when the cluster itself changes (a node joins, leaves, updates or changes its settings), not every 10 seconds. The sidebar shows Nodes online as n/m from that status instead of the age of the newest peer heartbeat. Linux cards additionally show load, runnable tasks, pressure, and Clusto workload usage. Windows cards omit the Linux PSI row rather than reporting a misleading zero. - The full per-sample series is fetched on demand with
GetNodeMetricsHistory { node_id }→NodeMetricsHistory. A request for a remote node is forwarded to that node (which owns its own ring) and the response relayed back. This keeps the hot path lean - only summaries ride the cluster continuously; the heavier series is paid for only when something is charting it.
The app's cluster view renders the live reading for each metric on each node card, with its sparkline underneath.
Charts show the past hour as soon as you open the app. Each node keeps its samples in a 1-hour in-memory ring and saves it to host-metrics.json in its data directory once a minute, then reloads it at startup (keeping only samples from the last hour). A daemon restart therefore loses at most about a minute of history. When the app connects it asks every online node for its saved hour and merges it with the live samples, and it asks again for a peer that comes back online after dropping offline. The sparklines no longer start empty on each connect or reconnect.
Node Failure and Recovery
Leaderless replication has no failover step - there is no leader to replace:
- When a node goes offline, the remaining nodes keep applying their own writes locally and gossiping to each other; the cluster stays fully usable.
- Sessions owned by the offline node show as unreachable (their OS-local processes are gone) but their replicated metadata is preserved.
- Clients connect to any node via the
--nodeslist; every node serves reads and writes locally. - When the node comes back, anti-entropy on reconnect compares the two nodes by record class and sends each side the records it missed; the per-object merge reconciles them.
- Even an all-down then cold restart converges: each node loads its own persisted state and re-merges as peers reconnect.
Setup Guide
Prerequisites
- Two or more machines, up to about five (or ports on one machine for testing)
- Same
cluster_tokenon all nodes - Same
token(client auth) on all nodes for seamless failover - Network connectivity between all nodes on the peer port - or Clusto Tunnel enabled on nodes that have no direct route (peer links fail over to the tunnel automatically)
Two-Node Cluster
Node A (10.0.0.1):
toml
bind = "0.0.0.0"
port = 9178
token = "client-auth-token"
claude_bin = "claude"
[cluster]
peers = ["10.0.0.2:9179"]
peer_port = 9179
cluster_token = "my-cluster-secret"
advertise_addresses = ["10.0.0.1:9178"]
display_name = "node-a"Node B (10.0.0.2):
toml
bind = "0.0.0.0"
port = 9178
token = "client-auth-token"
claude_bin = "claude"
[cluster]
peers = ["10.0.0.1:9179"]
peer_port = 9179
cluster_token = "my-cluster-secret"
advertise_addresses = ["10.0.0.2:9178"]
display_name = "node-b"These values are examples. A hand-set token or cluster_token shorter than 32 characters can be guessed offline, so the daemon logs a startup warning for it; leave them out of config.toml to get generated 64-character values, then copy the same cluster_token to every node.
Adopting a node
Instead of editing config.toml on each machine, add a node to a cluster from the app: Cluster > Adopt Node, or Discover Nodes to find candidates. Enter the remote node's address and either its pairing code or its token. A pairing code is the easier way to hand a node over without copying the token: create one on the new node under Configure Node > Pairing code (or clusto pairing generate), then paste it into Adopt Node. If you have a pairing link, paste it and the node identity is filled in, so the connection is verified against that key. See Node Identity & Pairing.
The node you adopt from makes the connection itself. It checks the remote node's identity first, never sends it a credential or its cluster token before that, and then gives it the cluster token and the list of nodes with their identities. Anyone who can redeem a pairing code gets full control of that node and its cluster, so use single-use codes (the default) and clear unused ones.
Local Testing (Single Machine)
Node A (port 9178/9179):
toml
bind = "127.0.0.1"
port = 9178
token = "test-token"
[cluster]
peers = ["127.0.0.1:9279"]
peer_port = 9179
cluster_token = "test-cluster"
advertise_addresses = ["127.0.0.1:9178"]
display_name = "local-a"Node B (port 9278/9279):
toml
bind = "127.0.0.1"
port = 9278
token = "test-token"
[cluster]
peers = ["127.0.0.1:9179"]
peer_port = 9279
cluster_token = "test-cluster"
advertise_addresses = ["127.0.0.1:9278"]
display_name = "local-b"Start each with separate config files or config directories. The loopback bind keeps each node's client API on the machine; the peer ports (9179 and 9279) still listen on all interfaces so the nodes can reach each other. Short hand-set tokens like these are fine for a local test, but the daemon logs a startup warning for any token under 32 characters.
Client Configuration
bash
export CLUSTO_TOKEN="client-auth-token"
export CLUSTO_NODES="10.0.0.1:9178,10.0.0.2:9178"
# Commands work transparently - the client connects to any node
clusto ls
clusto new --dir /project
clusto attach <session-id>
clusto cluster-status
clusto rename-node <node-id> "my-node" # rename a node at runtimeDesktop App
The Tauri desktop app provides a graphical Cluster Status view with:
- Connection state at a glance (every node is a co-equal authority - there is no leader/follower distinction in leaderless mode)
- One card layout for every node, the connected node included. The header has a status dot, the node's display name (or the start of its node id if unnamed), badges (This node, Offline, Identity changed, Browser TLS) and a ... menu holding Node settings, Enable browser TLS, Restart node and Remove node (Leave cluster on the connected node). Restart, remove and leave ask for confirmation first.
- Node settings opens the Configure Node dialog (node name, session defaults, shells; plus the pairing code on the connected node)
- A bottom action row with Connect here (peers) and Session as user on each node card
- A red Identity changed badge on a peer whose connections are refused because it presented a different identity key, with a Trust new identity button in the action row, even while that peer shows as offline (see Peer transport)
- Update status on the card (see URL updates from the Cluster view)
Node cards (including the connected node) are ordered by display name (falling back to node id), so a card's position is stable and predictable.
Known nodes are remembered across restarts. When a peer first connects, its identity (node id, name, address) is recorded in the daemon's persisted peer_nodes registry. A node that is configured but not currently connected stays in the view as an Offline card instead of disappearing - even after a fleet-wide update wipes every daemon's in-memory peer cache. Removing a node (Remove node in its card's ... menu) clears it from the registry; the connected node's Leave cluster menu item removes it from all peers and leaves the daemon running standalone. After the removal succeeds, its card disappears immediately, including when it was the final peer. A node that is only disconnected remains visible as Offline until it is explicitly removed.
A removal is permanent until you adopt the node again. Every daemon records it in [[cluster.removed_nodes]] and passes it on in the cluster config sync, so a removed node stays out after restarts and fleet updates, and a peer that was offline during the removal forgets the node the next time it connects. A removed node is refused in both directions, including under an address alias the removal did not know. Adopting the node again lifts the removal on every node.
Open the app, connect to any cluster node, and navigate to Cluster to see the full topology.
Session defaults per node
Each node card's ... menu has a Node settings item that opens the Configure Node dialog, whose Session Defaults section sets three values for that node:
- Default Shell - the command sessions start with when the request and the project both send none. Options are that node's registry entries (the same list the dialog's Shells section manages) plus Custom... for a raw command line; Auto-detect clears it. Stored as
default_pty_shell. - Default User - the OS account sessions, durable Chats and Notes run as when the request and the project both leave it unset. Stored as
default_session_user. On Unix a root-effective Chat launch is rejected, so pick a non-root account when the daemon runs as root. - Chat Folder - where chats with No workspace run on that node. It must be an absolute path; empty restores the default, a
clustofolder in the home of the OS user the chat runs as. Stored asdefault_chat_dir. Every no-workspace chat on the node shares the folder and its files do not replicate (see Chats). - Chat Workspaces Folder - where named chat workspaces and the workspaces of chats replicated from other nodes are created on that node. It must be an absolute path; empty restores the default, a
clusto-chatsfolder in the home of the OS user the chat runs as. Stored aschat_workspaces_dir. Existing chats keep the folder they already use.
All of these work for any node, not just the connected one: the request is relayed to the target node, which persists it to its own config.toml and announces the new values to the cluster, so every client's pickers show the change without a reconnect. The connected node applies a new default shell, chat folder or chat workspaces folder to the next session or chat immediately - no daemon restart. Use the same connected-node configuration dialog for a daemon installed on the desktop machine.
Sessions and chats never guess at these values: every picker that could inherit them names the one it will use (Node default (Zsh), Project default (ubuntu)), and selecting the inherit entry sends nothing so the daemon resolves the chain at spawn time. For a terminal session forwarded from another cluster node, the destination node performs that resolution. A request and project with no selected user therefore inherit the destination node's Default User, not the forwarding node's default.
Session as user
Each node card has a Session as user action in its bottom row. It opens a modal listing that node's non-system OS users (plus a shell picker, which offers the node's default shell as its first entry) and launches a single one-off terminal session that:
- runs as the selected user (via
runuseron Unix /clusto-runas.exeon Windows - the same mechanism a project'srun_as_usersmapping uses), and - starts in that user's home directory, so the session is grouped under the username in the workspace.
Unlike the per-project run-as mapping, this session is not tied to any project. The home directory comes from the node's OsUserInfo.home_dir (resolved from /etc/passwd on Unix); when no home directory is known the user is not selectable.
Connect here
Each node card carries a Connect here action that re-points the app's client connection at that specific node. It dials only the chosen node's own routes
- its advertised addresses plus its own Clusto Tunnel endpoint, which is often the only route a phone has. If none of them are reachable it reports an error and leaves you on the current node - it does not silently fall back to another node in the cluster, so "Connect here" always lands you where you asked (or tells you why it could not). This matters when you need to reach a node directly (for example to run a session on its own hardware) rather than through whichever peer happens to answer first.
The client stays on the node it is on
Every node in a cluster serves the same session and cluster views, so the client is free to connect to any of them - but it should not wander. The app remembers the node each connection profile is currently on and, on every reconnect and app launch, dials that node alone first (up to 8 seconds, cut short as soon as all of its routes have failed). Only when it cannot be reached does the rest of the profile race for the connection, and whichever node answers then becomes the new remembered one.
Without this the client raced every node's routes at once and kept whichever finished first. Tunnel dial time varies by seconds, far more than the small head start the preferred route gets, so a client - especially a mobile one on the tunnel - appeared to hop between nodes after most reconnects.
A node no longer periodically re-dials its own advertise address when that address is reflected back to it through cluster gossip. Earlier builds logged a repeating
rejecting self-connectionwarning loop from this; that noise is gone.
Cluster configuration in Settings
The Cluster Configuration card lives in Settings → Cluster, above Sync preferences with cluster. The Cluster Status view keeps a small card with an Open in Settings button that deep-links straight to it (/settings?tab=cluster).
The card shows the effective runtime config of the connected node - node ID, whether a pairing code and cluster token are configured, the Clusto Tunnel state, and every cluster node (self plus peers) with its addresses. Apart from the tunnel toggle the card is read-only: per-node settings live in each node card's Configure Node dialog (see Session defaults per node), and peers are added with Adopt Node or Discover Nodes on the Cluster page.
Each node entry also reports the absolute daemon data directory and the space used by all files beneath it. The line appears as Files: size - path and covers the node database, logs, attachments, staged update artifacts, and other node-local state. The connected node and online peers show Measuring storage... until their first measurement is available. An offline peer with no reported measurement shows Storage unavailable while offline; previously reported storage can remain visible with the node's Offline badge.
Where each cluster setting is changed:
- Clusto Tunnel (premium) - Enable for cluster / Disable for cluster turns Clusto Tunnel on or off on every node and restarts every node so the change applies; a node that is offline applies it when it reconnects. Enabling requires an active license. Once active, the card shows the node's tunnel endpoint id with a Copy id button
- Pairing code - the optional pairing code or password that lets the app or another node take over this node without its token is created in the connected node's Configure Node dialog (expiry, single use, or a custom password of 16 or more characters), or with
clusto pairing. The node shows its state (active, expired, locked, or reset required) but never the code after it was created; see Node Identity & Pairing - Advertise addresses and peers - peers are written by Adopt Node; there is no in-app editor for
advertise_addressesor the raw peer list, so edit the[cluster]section ofconfig.tomlfor those
Update settings - manifest URL, bearer token, auto-check toggle and interval - are not on this card. They live in Settings → Updates, which writes the same daemon-replicated [update] config and broadcasts it to peers; see Remote-URL Updates.
Verification
1. Connectivity Test
Start both nodes. Check logs (with log_filter = "info"; the default warn filter shows only the clusto_cluster peer lines such as the last one):
INFO starting Clusto daemon
INFO cluster peer server started peer_bind="0.0.0.0:9179"
INFO cluster mode enabled node_id=a1b2c3d4-...
...
INFO anti-entropy: in sync with peer, nothing to exchange peer=b2c3d4e5-...clusto cluster-status on either node lists both nodes as connected.
2. Replication Test
Create a session on either node:
bash
clusto new --dir /tmp/test --name "test-session"The session should appear in clusto ls on both nodes (it gossips to the peer).
3. Offline / Merge Test
bash
# Connect to the cluster from a client
clusto --nodes 10.0.0.1:9178,10.0.0.2:9178 ls
# Take one node offline, create a project on the still-up node, then bring the
# offline node back. Anti-entropy on reconnect merges the new project both ways.
clusto --nodes 10.0.0.1:9178,10.0.0.2:9178 ls
# Same sessions/projects visible from either nodeLimitations
- ~5 nodes - the full-mesh gossip is designed for small clusters.
- No process migration - agent/PTY processes are OS-local; a session is only interactive on its owning node, and is unreachable while that node is offline.
- Same release everywhere - there is no mixed-version operation. A node on another release cannot link: it shows as offline to the others until it updates, and Cluster > Update All upgrades the whole cluster in one step. Peer links, the apps and the CLI all rely on the same encrypted, key-pinned handshake.
- No automatic peering - Discover Nodes scans Tailscale, the LAN, localhost, Docker and the local subnet for candidate daemons, but each peer must still be adopted or configured explicitly.
- LWW drops one side of a same-object conflict - concurrent edits to the same project/team on two simultaneously-offline nodes resolve by
updated_at; the older edit is silently dropped (rare for a single user; distinct objects are never lost).
Tuning
Leaderless replication has no election or heartbeat timers to tune. The old heartbeat_interval_ms / election_timeout_* / min_quorum config keys are ignored when present and dropped from config.toml on the next save.