Appearance
Cluster Troubleshooting Runbook
This page covers the current operational checks for Clusto cluster deployments. It focuses on commands and config fields that exist in the current CLI and daemon.
1. Node Does Not Join the Cluster
Symptoms
clusto cluster-statusnever shows the new node- Logs show repeated peer connection failures
- The node starts, but no peer connections appear
Check
Verify the shared cluster secret on every node:
toml[cluster] cluster_token = "same-secret-on-every-node"Verify peer addresses:
toml[cluster] peers = ["10.0.0.2:9179", "10.0.0.3:9179"] peer_port = 9179Verify the node is advertising a reachable client address:
toml[cluster] advertise_addresses = ["10.0.0.1:9178"]Verify network reachability (the peer port speaks TLS only, so
ncproves the port is open, not that the handshake works):bashcurl http://10.0.0.1:9178/health nc -zv 10.0.0.1 9179Check that both nodes run the same Clusto release. A node on another release shows as offline to the others, and a plain HTTP request to its peer port is answered with
426 Upgrade Required(see section 17).A card marked Identity changed means the node's key no longer matches its pin (see section 18).
Fix
- Correct
cluster_token,peers,peer_port, oradvertise_addresses - Restart the daemon after changing cluster settings
2. Peers Connect but Sessions Do Not Replicate
Symptoms
- A node does not show sessions or projects created on a peer
- A node that was offline comes back but still shows stale state
Cause
Leaderless replication gossips each local write to all peers once and runs anti-entropy (both sides exchange a fingerprint per record class and then only the records that differ, merged per-object) on every reconnect. If a node is missing a peer's objects, the gossip message was lost while the peer was unreachable and the reconnect anti-entropy has not yet run - almost always a peer-connectivity problem, not a state-machine wedge.
Check
Confirm both nodes see each other as connected:
bashclusto cluster-statusInspect logs for the anti-entropy snapshot exchange on reconnect:
toml# in config.toml, then restart the service and follow the log log_filter = "clusto_cluster=debug,clusto_daemon=debug"Look for
sent anti-entropy snapshot to newly connected peerinjournalctl -u clustod(orclustod.login the data directory). Do not runclustodby hand on a node whose service is running: a second instance stops the live daemon and every session on it.Confirm each node can still reach the other on both the client and peer ports.
Fix
- Restore peer connectivity. Once the peer connection re-establishes, the reconnect anti-entropy exchange merges both sides' state automatically - no restart needed. Distinct objects are never lost; concurrent edits to the same object resolve last-writer-wins by
updated_at. - If a node still does not converge, re-check
peers,peer_port, andadvertise_addresseson both sides.
3. Client Auth Works on One Node but Fails on Another
Clusto uses two different tokens:
| Token | Purpose |
|---|---|
token | Client auth for CLI and app connections |
cluster_token | Peer auth between cluster nodes |
Check
- For CLI/app connection failures, verify
CLUSTO_TOKENmatches the node'stokenor cluster-wide access token you intend to use. - For node-to-node failures, verify every node has the same
cluster_token. - Inspect logs for
wrong token,auth rejected, or timeout messages. - A blank
CLUSTO_TOKEN, or an unexpanded${CLUSTO_TOKEN}, is reported with a message naming the cause rather than "invalid token". Export a real value or unset the variable so the CLI uses the local daemon's token.
Fix
- Update the client token you are using
- A hand-set
tokenorcluster_tokenshorter than 32 characters makes the daemon log a startup warning, because it can be guessed offline from the identity proof any caller can request. Removetokenfromconfig.toml(or set a long generated value) and use the same generatedcluster_tokenon every node - If the CLI says the node "could not prove it knows your token", the token you have is not the one that node holds (see section 18)
- On Windows, "authentication failed: invalid token" in chats or sessions run as another user is fixed: run-as chats and sessions now receive the daemon's
CLUSTO_*variables, which the run-as helper used to drop. Update to the latest version. - Or update the daemon config so all peer nodes share the same
cluster_token
4. A Node Goes Offline
What to expect
Leaderless clustering has no leader, no election, and no quorum, so there is no failover to wait on. Every remaining node keeps serving reads and writes for the state it already holds - a node going offline never stalls the rest of the cluster. The only thing lost while a node is offline is interactive access to the sessions it owns: agent and PTY processes are OS-local, so a session is only interactive on its owning node and is unreachable until that node returns.
When the node comes back, anti-entropy on reconnect merges any state that diverged while it was away (see section 2).
Symptoms that are NOT a cluster fault
clusto cluster-statusshows a peer as not connected - this is expected while that node is down; the rest of the cluster is unaffected.- Sessions owned by the offline node fail to load or accept input - expected; they resume when the owner returns.
Check
Confirm the still-up nodes see each other as connected:
bashclusto cluster-statusConfirm the offline node is actually down (vs. a peer-connectivity break) and bring it back or restore connectivity on
peer_port.
5. Node Drops Out After Restart or IP Change
Symptoms
- A node was healthy before restart but disappears afterward
- DHCP or Tailscale address changed
Check
- Confirm the node's current client-facing address.
- Update
advertise_addressesif the node should now be reached at a differenthost:port. - Update
peerson the other nodes if the peer endpoint changed.
Example:
toml
[cluster]
advertise_addresses = ["100.64.1.12:9178"]
peers = ["100.64.1.13:9179", "100.64.1.14:9179"]Fix
- Save the corrected addresses
- Restart the changed node
- If needed, restart the peers so they reconnect using the updated config
6. Remote Update From URL Fails
Check
Verify the manifest URL is reachable:
bashcurl -I https://updates.example.com/clusto/manifest.jsonVerify the manifest contains an artifact for the target platform.
For private endpoints, verify
CLUSTO_UPDATE_TOKENor--remote-token.
Current commands
bash
clusto update --from https://updates.example.com/clusto/manifest.json
clusto update-node --from https://updates.example.com/clusto/manifest.jsonFix
- Correct the manifest URL
- Publish the missing platform artifact
- Supply the correct bearer token
7. Daemon Lags Under Heavy CPU Load From Other Processes
Symptoms
- Peer-connection keepalive pings time out and peers mark this node offline while the host is otherwise busy (media transcoding, builds, backups)
- WebSocket sessions stutter or disconnect even though the network is fine
clusto cluster-statusfrom another node reports this node as flapping
What users now see in clients
- Recent CLI and app builds surface non-fatal daemon notices instead of leaving the slowdown silent.
- Remote-session attaches can show a notice that the live PTY is running on a different cluster node and Clusto is forwarding traffic there.
- If a viewer falls behind on terminal output, the client can show warnings such as
This client is falling behind on live session outputorClusto stopped one live output stream because this client could not keep up.
What clustod does automatically
- On every start,
clustodraises its own CPU and I/O priority -nice -10on Linux/macOS viasetpriority(2), best-effort I/O priority 0 on Linux, andHIGH_PRIORITY_CLASSon Windows viaSetPriorityClass. On Linux it also lowers its OOM score to make the daemon a last-resort kill target. This applies on fresh installs, self-updates, and manual restarts. A line likeraised CPU priority (nice = -10)appears in the daemon log on success; a warning is logged if the OS denied the request. - Spawned workload processes are pushed slightly below normal priority so Claude, shells, task commands, headless agents, and CPU-heavy builds do not compete with the daemon's control plane. On Linux they are set to
nice +2, keep the normalSCHED_OTHERscheduler, and use the default best-effort I/O priority. Initial Linux placement happens before the process executes, so wrappers such asrunusercannot pass daemon priority to the real shell. The margin is intentionally small: the cgroup split below is the actual CPU reservation, and a largenicevalue would only make sessions lose against unrelated nice-0 processes on the host (browsers, container runtimes, CI runners) without protecting the daemon any further. - A self-update keeps live PTYs running, so on start the daemon re-applies the current workload policy to every process already inside
workload/. Sessions spawned by an older build (which usednice +10andSCHED_BATCH) are healed without a respawn; the log line isre-applied workload scheduling policy to processes from a previous daemon. - On Linux, when the systemd unit delegates the cgroup (
Delegate=yes), clustod splits its service cgroup into adaemon/control-plane leaf and aworkload/subtree. The daemon leaf holds acpu.weight/io.weightof 10000 against the workload subtree's 1000, a 10:1 edge - more than enough for a socket-bound control plane that never saturates a core, without starving the sessions the node exists to run. Every PTY session and headless run gets its own child group, so a build with many compiler processes gets the same aggregate share as another interactive session instead of winning by process count. Descendants inherit the group, a guardian periodically returns marked descendants that escaped it, and empty per-session groups from ended sessions are pruned. This prevents one fan-out build from taking the control plane or another session with it. - Linux reserves 256 MiB with
memory.minand 512 MiB withmemory.lowfor the daemon. The workload subtree getsmemory.highat 80% of physical RAM, which asks the kernel to reclaim from workloads under pressure without imposing a hard cap. A line likecgroup workload isolation enabledappears on success;cgroup workload isolation unavailable(for example an older unit withoutDelegate=yes, or cgroup v1) means Clusto fell back to per-process nice and I/O priority. Isolation is best-effort and never blocks startup or session spawn. - On Windows, clustod runs at
HIGH_PRIORITY_CLASS, while each PTY session and headless run is assigned to its own inherited Job Object atNORMAL_PRIORITY_CLASSwith low CPU scheduling weight. Tools and builds launched by the agent inherit that Job Object. This contains the Clusto process tree without changing the priority of unrelated Windows applications. - The systemd units generated by the CLI, desktop app, and
clusto.ps1 deploysetNice=-10,IOSchedulingClass=best-effort,IOSchedulingPriority=0,CPUWeight=10000,IOWeight=10000,OOMScoreAdjust=-900,Delegate=yes,MemoryMin=256M, andMemoryLow=512Mso the elevated priority and cgroup delegation are in place before the binary's own startup code runs. - No application timer ever drops a client connection (SSH semantics). A widening pong gap is logged as
connection staleand held; output sheds via the resumable ring instead of timing out; genuine death is detected by the kernel (TCP keepalive +TCP_USER_TIMEOUTon the listener) and surfaces as a socket error. The only daemon-initiated teardown is the zombie reap of a connection with no attached sessions that has also stopped answering pings. See Keepalive. - If replicated state briefly loses a live PTY session's owner while output is still visible, the owner re-announces for several seconds and PTY input probes connected peers before surfacing
owner is not known yet. - The peer circuit breaker credits host-load stall before disconnecting. A node thrashing in swap cannot drain its own outbound channels in time and would otherwise time out every peer send at once and partition itself out of a healthy mesh. Instead, when at least 5 s of local scheduling stall is credited across a timeout streak, the link is held (
holding peer, not partitioning self) and recovers when load drops; a genuinely dead peer is still reaped by the per-peer receiver task and TCP keepalive. See Transport. - Nothing slow runs on a connection's select loop. Session-stream operations, claims, kills, history fetches, and every peer-forwarding request are dispatched to ordered off-loop workers or detached tasks, so a slow peer's RPC timeout can never stall this client's heartbeat and output while a different node is overloaded - the same protection long applied to keystrokes and resizes.
Check
Confirm clustod is actually running with raised priority:
bash# Linux ps -o pid,ni,cmd -C clustod # NI column should be -10 (or close to it)powershell# Windows Get-Process clustod | Select-Object Id, PriorityClass # PriorityClass should be HighOn Linux, if the NI value is
0, the daemon's self-elevation was denied - checkjournalctl -u clustod | grep "raise CPU priority"for the reason (usually capability/permission related).Confirm session children are not inheriting clustod's elevated nice value:
bashps -eo pid,ppid,ni,cmd | grep -E 'clustod|claude|codex|bash|zsh|pwsh' | grep -v grep # clustod should be around -10; session children should normally be +2.On Linux, inspect the workload hierarchy and current resource pressure:
bashsystemctl show clustod -p ControlGroup cat /proc/pressure/cpu /proc/pressure/memory /proc/pressure/ioThe service cgroup should contain
daemon/andworkload/; active sessions appear as children belowworkload/. The app's node card also shows Linux pressure, load, runnable tasks, Clusto workload CPU, and Clusto workload memory. Windows node cards continue to show CPU, memory, swap, and network, but do not show Linux PSI rows.On Windows, confirm the daemon remains elevated:
powershellGet-Process clustod | Select-Object Id, PriorityClass # PriorityClass should be High
Fix
- For a service install on Linux, ensure the unit was deployed via current
clusto.ps1 deployorsudo clusto installso theNice=andCPUWeight=directives are present. If editing an existing unit by hand, add the priority directives under[Service]andsudo systemctl daemon-reload && sudo systemctl restart clustod. - If a specific offending process (e.g. a wedged transcoder) is monopolising the CPU, cap or kill it rather than raising clustod further -
REALTIME_PRIORITY_CLASS/SCHED_RRare intentionally avoided because a hot loop in clustod under real-time scheduling can lock up the entire host.
8. A Long-Running Terminal Looks Corrupted After You Return
Symptoms
- A session runs for a long time while its terminal is in another workspace, tab, or backgrounded app.
- Returning shows scattered characters, duplicated TUI fragments, excessive blank space, or a layout that no longer resembles the running program.
- Reload full history or another forced redraw repairs the display.
Cause
- Older builds stopped parsing a terminal while it was hidden and rebuilt it from a daemon snapshot whenever it was shown again (and after every resize and app restore). Full-screen TUIs such as Claude Code repaint only the cells that changed, so any difference between the rebuilt screen and the real one stayed on screen: typed text landing on the status line, or leftover fragments of earlier frames, until a window resize forced a full repaint.
- Snapshots taken while a synchronized repaint was split across two PTY reads left out its first half, and snapshots of a full-screen program put its screen in the normal buffer instead of the alternate one.
- On native Windows, an older Tao event-loop path could also deadlock while holding its dispatcher lock across
PeekMessageW, freezing the whole app.
Fix
- Update every Clusto app and daemon in the fleet to the same current build.
- Terminals keep parsing their live output while hidden, so switching back shows the exact screen without any rebuild. Reconnects resume every stream from its seq cursor. See Switching terminals.
- Snapshots, used only for a terminal's first open or after output was really lost, include a repaint in progress, restore the alternate screen, and carry a split escape sequence across the boundary.
- Terminals share a bounded, foreground-priority render budget. Queue overflow triggers snapshot recovery instead of continuing with an incomplete escape stream, and recent input reduces background rendering work.
- The Windows app includes the upstream Tao event-loop deadlock fix.
Check
- Leave a busy terminal in another workspace long enough for it to produce substantial output.
- Return to it. The display should already show the current TUI state, with no history loading indicator.
- If an older build is still deployed, use Ctrl+Shift+H or right-click the terminal and choose Reload full history as a temporary recovery.
9. Copy Is Disabled After Selecting Text Inside a TUI
Symptoms
- Text can be selected and copied normally at a shell prompt
- Selecting inside a full-screen TUI such as Claude Code does not enable Copy in Clusto's terminal context menu
- Ctrl+C interrupts the TUI instead of copying the highlighted text
Cause
Some TUIs own their mouse selection instead of using the terminal emulator's native selection. The TUI asks the terminal to update the system clipboard with an OSC 52 write, so Clusto has no native selection to expose through its context menu.
Fix
Update to a build with OSC 52 clipboard support. A focused terminal that owns the session input claim accepts write-only OSC 52 requests and copies the text to the local system clipboard, including Unicode text. Passive viewers and unfocused terminals cannot change the clipboard.
For safety, Clusto rejects clipboard-read requests, non-system clipboard selections, malformed data, and decoded payloads larger than 1 MiB.
10. A Headless Agent (Team Member or Chat Session) Never Answers
Symptom: a team leader stays silent, its workers sit on Pending, or an agent chat shows no reply. The daemon log has the agent exiting a couple of seconds after spawn:
INFO clusto_daemon::session::sdk_backend: SDK process spawned provider="claude"
WARN clusto_daemon::session::sdk_backend: SDK process exited with non-zero code exit_code=1When the process dies this early the prompt cannot be written to it. Clusto reports the agent's own last stderr line (agent process exited: <reason>) rather than the bare SDK process stdin closed, so the error in the app usually names the cause directly.
Cause: a headless agent runs as an OS account, and the two common failures both come from that account:
It is
root. clustod runs as root under systemd, and the Claude CLI refuses--dangerously-skip-permissionsunder root or sudo, so abypass-mode team dies at startup. Clusto now rejects that combination when the team is created.The account does not exist on this node. Run-as names travel with a session (a project's
*mapping, another node'sdefault_session_user), so a team created against a directory owned by a different machine can be handed a username the spawning node has no account for.runuserthen exits before the agent starts and the only trace is one stderr line:runuser: user ubuntu does not exist or the user entry does not contain all the required fieldsClusto now refuses this at team creation and at spawn with
run-as user '<name>' does not exist on <host>.Its login expired. The CLI retries the 401 and gives up. Note that
claude auth statusstill reportsloggedIn: truefor a stale token, so it is not a reliable check; the daemon log line below is:OpenRouter has no API key. Native OpenRouter does not use a CLI login. It needs the cluster-wide
[openrouter].api_key, or a localOPENROUTER_API_KEYenvironment fallback when no cluster key exists. The affected turn closes as failed and shows the credential error instead of remaining in Starting.Team coordination is unavailable. A leader may still answer ordinary questions while lacking the tools needed to register workers. New managed leaders and workers reject this state before spawn if the colocated
clustoCLI is missing, does not match theclustodversion, or lacksclusto team mcp.
WARN clusto_daemon::session::agent_auth: agent authentication failed; raising AgentAuthRequiredFix:
- Give the project a Run as user (Projects > edit project), or set
default_session_userin the node config, so agents stop running as root. - If the account's login is stale, use the Sign in button on the banner in the session or team view. It opens a terminal as that account running
claude auth login. In the native desktop and mobile apps, the provider authorization link opens in the OS default browser, not Clusto's embedded webview. If Claude requests a verification code, return to Clusto after authorizing and enter it in the code-entry dialog, which remains open while the browser is active. - For OpenRouter, open Settings > Account > OpenRouter API, or use F1 > Configure OpenRouter API key. Save, replace, or clear the one write-only key for the whole cluster. Connected nodes apply the change to the next turn without restarting
clustod, and offline nodes receive the latest value when they reconnect. If no cluster key exists, each node can use its ownOPENROUTER_API_KEYenvironment fallback. Environment keys cannot be changed or cleared in the app. - Verify which account an agent used. This line is logged at
info, so setlog_filter = "info"(orRUST_LOG=info) first; the defaultwarnfilter hides it:
bash
journalctl -u clustod | grep "spawning SDK process"
# program=/usr/sbin/runuser ... run_as_user=Some("ubuntu") <- good
# program=claude ... run_as_user=None <- runs as the daemon user- If the leader says workers are ready but Clusto still shows zero workers, verify the coordination CLI on the daemon host:
bash
clustod --version
clusto --version
clusto team mcp --helpThe two versions must match. With an update source configured, clustod refreshes a missing or stale colocated CLI in the background; retry team creation after it finishes. If the affected team predates the update, recreate it so its leader receives the authenticated daemon address, token, TLS mode, MCP config, and current coordination instructions.
- Treat the app roster as authoritative. Claude Task subagents and files under
.claude/agentsare provider-native helpers, not Clusto workers. Use Add Worker in the Teams list or team detail view, or ask the leader to callteam_assign/team_handoffand confirm the returned session UUID withteam_status.
11. A Project Session Starts on the Wrong Node
Symptoms
- New Session or Quick New Session starts a project session on the connected node instead of the node selected for the project
- After opening a session owned by another node, Quick New Session fails with a shell error such as
zsh:1: command not found: PowerShell - Editing the project, reselecting its node, and saving appears to fix later sessions
- Claiming a just-created remote session repeatedly reports
session owner is not known yet
Cause
Older builds could create a project without a node ID when This node was submitted during app startup before cluster status supplied the connected node's UUID. Session creation then had no project node to target. Separately, a remote session's ownership announcement could arrive after the first claim. Older app builds could also choose a shell before resolving the quick session's target node. Switching between sessions on different operating systems could therefore send a Windows shell choice such as PowerShell to a Linux project node.
Fix
Update Clusto across the whole cluster. In current builds, a cluster daemon pins a project with no explicit node ID to itself. New Session and Quick New Session always target the selected project's pinned node and report a clear error if it is offline. Claim routing probes connected peers when ownership replication is delayed, so a newly created remote session can be claimed immediately. Quick New Session derives its target from the active project's node, or from the active session's owning node when needed, then resolves the shell choices available on that target node. It no longer carries a shell choice from one operating system to a session created on another.
12. Mouse Wheel Does Not Scroll in a Full-Screen TUI
Full-screen terminal apps use the alternate screen. If the app enables terminal mouse tracking, Clusto forwards mouse-wheel and touch-scroll input to it. Like native terminals, it sends one wheel report per line scrolled: a mouse-wheel notch scrolls three lines and a trackpad scrolls with its travel, so Claude Code's fullscreen view follows the gesture instead of moving one line per event. If it does not, Clusto converts scrolling to Page Up or Page Down so the TUI can still navigate. This covers GitHub Copilot CLI when its mouse integration is disabled or unavailable. The Page Up and Page Down keys continue to work directly.
If Claude Code's fullscreen view scrolls in jumps instead of a steady three lines per notch, it is accelerating the wheel: it multiplies the lines per report during fast scrolls (up to 6 on Linux, up to 36 on Windows, where it also starts at 3 lines per report). Each node's AI setup turns this off with the Claude Code smooth scrolling check, which writes "wheelScrollAccelerationEnabled": false and "env": { "CLAUDE_CODE_SCROLL_SPEED": "1" } to ~/.claude/settings.json when those keys are absent. Run Fix now on the AI setup card (or Settings > Node Doctor > Fix) if the node has not applied it yet, and check that the node's AI setup is on. A value you set yourself is kept, so look in that file if the jumps persist. See Knowledge.
13. Sidebar RTT Is High or Missing
The native app's sidebar shows the latest matched WebSocket heartbeat round-trip time, refreshed every 15 seconds. Earlier measurements used a cluster-status request and could include queued application work and frontend processing. See Sidebar RTT for the current measurement boundaries.
- After connecting or recovering from a stall, wait for the next matching heartbeat reply. The previous connection's sample is cleared.
- Compare several samples while using the app. One spike does not establish a sustained connection problem.
- Record the connected entry node and whether the app uses a direct address or Clusto Tunnel. RTT measures that connection, while a terminal hosted on a different node also depends on the link between nodes.
- For sustained high RTT, check network reachability, relay or tunnel connection errors, and CPU load on the client and entry node. A normal system
pingresult alone does not rule out transport buffering or runtime delays. - In the browser client, an absent RTT is expected: browser APIs do not expose WebSocket control-frame timing.
14. Browser or App Rejects the Node's TLS Certificate
Symptoms: the web client or app cannot connect over wss://, or the browser reports an expired certificate for the node.
- A daemon serving a
clusto cert(Tailscale) certificate renews it itself when less than a third of its lifetime is left, and hot-reloads it within the hour with no restart. Check the daemon log (log_filter = "info") for the renewal attempt and anyclusto certerror, such as Tailscale missing or HTTPS Certificates not enabled for the tailnet. - A certificate installed at a custom path is never renewed by the daemon. The daemon logs a warning when it is due; replace the file (the change is hot-reloaded) or run
sudo clusto certto switch to a managed certificate. - The desktop and mobile apps and the CLI do not use the certificate: they connect with TLS pinned to the node's identity key. Only browsers (the web client) and other plain TLS clients depend on it.
- Chat tools, the Clusto Assistant, team MCP servers and Stop hooks do not depend on the certificate: inside a session they use a plain loopback listener on
127.0.0.1. If only the browser is affected, the certificate is the cause. If those tools fail with a certificate error, the daemon is older than this behaviour; update it. - The first-time enable of TLS needs a restart:
sudo clusto cert --restart. See Web client.
15. Version Mismatch Notices
Clusto does not support mixed versions: the app, the CLI, and every node should run the same release. A mismatch is not repaired automatically, but it is now visible instead of failing silently.
Symptoms
- The app shows a warning toast "Node X runs A, app runs B - update the node" (or "update the app") with Update node and Check for update actions
- The CLI prints "this node runs X and does not understand the request from a client running Y; update the node" and exits non-zero
- Node logs show "peer NAME runs A, this node runs B: message not understood, update the older node" (rate limited)
Fix
Update the older side. Use Update node in the toast or Cluster view, clusto update-node, or Check for update for the app, then confirm with clustod --version and clusto --version. See Remote Updates.
16. "This Clusto node refuses unencrypted connections" (403)
Symptoms
- A script, browser page,
curl, Prometheus scrape or old app gets403with "This Clusto node refuses unencrypted connections from" followed by its address - The Android background notification check is paused or its polls are refused
Cause. The client port only accepts credentials (/ws, /notifications, /metrics, /clients/...) over an unencrypted connection from loopback, Tailscale, or a network the operator listed. Everything else gets 403 before a token is sent. /health and /identify always work. See Node Identity & Pairing.
Fix
Use the Clusto app or the CLI: both are always encrypted.
Connect over Tailscale or Clusto Tunnel.
For a script or scraper, use
https/wsswith the node's certificate (tls_certandtls_key), or list its network inconfig.toml:toml[transport] plaintext_networks = ["192.168.1.0/24"]The change is picked up without a restart. Remember a listed network sends the token in clear.
An old app on a LAN-only address cannot reach an updated node and cannot download its own update from it, because credential routes over unencrypted HTTP are refused from other networks. Update it from
clusto.app(Settings > Updates > Update from URL), or list the network in[transport] plaintext_networkson that node. See Node Identity & Pairing.
17. Peers show offline after an update (426 from a peer)
Symptoms
- After updating some nodes, the updated ones show the others as offline and the others show them offline
- The daemon log shows a peer answering
426 Upgrade Required, or a peer link failing the handshake - The app says "This node runs Clusto X, older than this app. Update the node or connect over the Clusto Tunnel."
Cause. Clusto does not run mixed versions. The peer port speaks only the current encrypted protocol, so a node on another release cannot link, and the app and CLI cannot dial an older node directly.
Fix. Update every node to the same release (Cluster > Update All, or clusto update-node on each machine), then the links come back on their own. Update the CLI with clusto update. Over the Clusto Tunnel an older node is still reachable, so you can update it from there.
18. "Node identity changed" or an "Identity changed" badge
Symptoms
- The app shows Node identity changed and stops reconnecting
- The CLI prints "presented identity ..., but ... is recorded in known_nodes.toml, and the node could not prove it knows your token" or "identity check ... failed"
- A peer in Cluster Status is offline with an Identity changed badge
- A toast says a node "has a new identity ... verified with your token"
Cause. The node presents a different key than the one remembered. The usual reasons are a reinstall, a restore without identity.key, a lost data directory, or a node that was replaced under the same name or address. It can also be an interception on that network, which is why nothing is sent until it is sorted out.
Check
- Did you reinstall or rebuild that node? If not, do not proceed: you may be talking to something else. Compare
clusto identity showrun on the node with the identity the client reports. - Is the token correct? A node that cannot prove it holds your token is refused.
Fix
- App: pair again with a pairing code or fix the token, using Edit profile in the dialog.
- CLI:
clusto identity forget <host:port>, then connect again. - Peer in the cluster: choose Trust new identity on its card and confirm, or adopt it again with a pairing code.
clusto doctor --fixon any other node offers the same trust step, and removes the stale entry a re-adopted node leaves behind. It also offers to remove a leftover roster address that no node owns (for example a retired machine's hostname), on every node. - If the identity changes again on its own, the node is losing
identity.key: check that only one daemon runs on that machine and that the service uses the same data directory each time (clusto doctorshows the service's config path). - Back up
identity.keywithconfig.tomlso reinstalls keep the identity. See Node Identity & Pairing.
19. Pairing is locked, expired or "reset required"
Symptoms
- "wrong pairing code or password", "the pairing code has expired", "pairing is not enabled on this node", or "the node's pairing secret is too weak and must be reset on the node"
- "pairing is locked after too many wrong attempts"
clusto pairing statusshows Expired, Locked or Disabled
Fix
- Run
clusto pairing status(or open Configure Node > Pairing code) on the node to see its state. - Expired or single use already spent: generate a new code with
clusto pairing generate. A single-use code works once. - Locked: ten wrong attempts in 15 minutes lock pairing for the whole node for 15 minutes, doubling per consecutive lockout up to 24 hours. Generating or setting a new code unlocks it; logging in with the token is unaffected.
- Reset required (an old
user_secretshorter than 16 characters was disabled when the node updated): set a new code or a password of at least 16 characters. - A code is typed without regard to case, dashes or spaces;
Ois read as0andI/Las1. A custom password is exact. - If the node itself is unreachable, check section 16 first.
20. "... is not permitted with a session credential"
An agent or script inside a session that runs as a non-root account holds a limited credential, not the admin token. Some requests are refused for it, or must run as the session's own account. Use the Clusto app or pass the node admin token for those. See Session Credentials, which lists each message.
21. The daemon will not start or an old Hive node is stuck
Run clusto doctor on the machine. It checks the clustod and legacy hived services, Hive-era state on disk and the cutover journal, then prints one diagnosis. clusto doctor --fix (Administrator terminal on Windows, sudo on Linux) starts a stopped clustod, stops a hived running beside it, removes a stale hived registration, upgrades a Hive-era node in place, and rolls back and retries a cutover that stopped part-way. See Moving from Hive to Clusto.
A user install (clusto install --user) that stops after you log out was installed without lingering. Check with loginctl show-user "$USER" -p Linger; an administrator enables it once with sudo loginctl enable-linger "$USER". If the installer said the user manager was not reachable, the daemon was started detached and will not return after a reboot: log in normally and run clusto install --user again. See Install without sudo.
22. Typing in a Terminal Feels Slow
Symptoms
- Characters appear late after you press a key
- The screen draws in pieces, or scrolling stutters
Check
Turn on the latency overlay: Settings > Terminal > Diagnostics > Show the terminal latency overlay, or run Show the terminal latency overlay from the
F1palette. It stays on this device.Type in a terminal. The overlay in the corner of every terminal shows p50 and p95 times in milliseconds for each stage a keystroke passes through:
- total - from the key press to the character painted on screen.
- key - the app handling the key press.
- invoke - handing the input to the app's native side.
- native - the native side sending it on to the node.
- echo - waiting for the node to send the echoed output back.
- parse - the terminal parsing the returned output.
- paint - drawing the result.
It also shows output events/s and KB/s, long tasks (stalls of the interface thread in the last 10 seconds) and the active renderer.
A large echo points at the network or the node. On a slow link the node sends a keystroke echo ahead of bulk traffic, so typing stays responsive while another terminal floods, and
Ctrl+Con a flooding terminal stops the old output within a second; if echo stays high anyway, the link or the node's CPU is the limit. The Keystroke echo and Frame gaps rows under Settings > Terminal > Diagnostics > Measured on this device show the same numbers without the overlay. Large parse, paint or long tasks values point at the device. Under Renderer in the same card, try WebGL or DOM instead of Automatic; it applies to terminals opened after the change.
Share the numbers
The app writes a terminal-latency line to its log every 50 keystrokes. Open the Logs view and export the log to attach those lines to a bug report.
Useful Commands
bash
# Check the local daemon install and AI setup
clusto doctor
# Show cluster topology
clusto cluster-status
# Discover Clusto nodes on Tailscale
clusto discover
# Read local daemon status
clusto daemon status
# Read local daemon config
clusto daemon get-config
# Read the credential this account holds (the admin token with sudo; a limited
# session credential for other accounts when that flag is on)
clusto daemon get-token
# This node's identity and pairing state
clusto identity show
clusto pairing status
# Check daemon health endpoint
curl http://localhost:9178/health
# Follow the daemon log (raise verbosity with log_filter in config.toml)
journalctl -u clustod -fConfig Paths
See Configuration for the full reference. Common paths:
- Linux:
~/.config/clusto/config.toml - macOS:
~/Library/Application Support/dev.vazac.Clusto/config.toml - Windows:
C:\Users\<user>\AppData\Roaming\vazac\Clusto\config\config.toml
clusto daemon status prints the path selected on the current machine.