---
url: https://clusto.app/docs/troubleshooting.md
description: >-
  Diagnose Clusto cluster, connection RTT, daemon, session, authentication,
  update, and headless team failures.
---

# Cluster Troubleshooting Runbook

This page covers the current operational checks for Clusto cluster deployments.
It focuses on commands and config fields that exist in the current CLI and
daemon.

## 1. Node Does Not Join the Cluster

**Symptoms**

* `clusto cluster-status` never shows the new node
* Logs show repeated peer connection failures
* The node starts, but no peer connections appear

**Check**

1. Verify the shared cluster secret on every node:

   ```toml
   [cluster]
   cluster_token = "same-secret-on-every-node"
   ```

2. Verify peer addresses:

   ```toml
   [cluster]
   peers = ["10.0.0.2:9179", "10.0.0.3:9179"]
   peer_port = 9179
   ```

3. Verify the node is advertising a reachable client address:

   ```toml
   [cluster]
   advertise_addresses = ["10.0.0.1:9178"]
   ```

4. Verify network reachability (the peer port speaks TLS only, so `nc` proves
   the port is open, not that the handshake works):

   ```bash
   curl http://10.0.0.1:9178/health
   nc -zv 10.0.0.1 9179
   ```

5. Check that both nodes run the same Clusto release. A node on another release
   shows as offline to the others, and a plain HTTP request to its peer port is
   answered with `426 Upgrade Required` (see [section 17](#_17-peers-show-offline-after-an-update-426-from-a-peer)).

6. A card marked **Identity changed** means the node's key no longer matches its
   pin (see [section 18](#_18-node-identity-changed-or-an-identity-changed-badge)).

**Fix**

* Correct `cluster_token`, `peers`, `peer_port`, or `advertise_addresses`
* Restart the daemon after changing cluster settings

## 2. Peers Connect but Sessions Do Not Replicate

**Symptoms**

* A node does not show sessions or projects created on a peer
* A node that was offline comes back but still shows stale state

**Cause**

Leaderless replication gossips each local write to all peers once and runs
anti-entropy (both sides exchange a fingerprint per record class and then only the records that differ, merged per-object) on
every reconnect. If a node is missing a peer's objects, the gossip message was
lost while the peer was unreachable and the reconnect anti-entropy has not yet
run - almost always a peer-connectivity problem, not a state-machine wedge.

**Check**

1. Confirm both nodes see each other as connected:

   ```bash
   clusto cluster-status
   ```

2. Inspect logs for the anti-entropy snapshot exchange on reconnect:

   ```toml
   # in config.toml, then restart the service and follow the log
   log_filter = "clusto_cluster=debug,clusto_daemon=debug"
   ```

   Look for `sent anti-entropy snapshot to newly connected peer` in
   `journalctl -u clustod` (or `clustod.log` in the data directory). Do not run
   `clustod` by hand on a node whose service is running: a second instance
   stops the live daemon and every session on it.

3. Confirm each node can still reach the other on both the client and peer ports.

**Fix**

* Restore peer connectivity. Once the peer connection re-establishes, the
  reconnect anti-entropy exchange merges both sides' state automatically - no
  restart needed. Distinct objects are never lost; concurrent edits to the
  *same* object resolve last-writer-wins by `updated_at`.
* If a node still does not converge, re-check `peers`, `peer_port`, and
  `advertise_addresses` on both sides.

## 3. Client Auth Works on One Node but Fails on Another

Clusto uses two different tokens:

| Token | Purpose |
|-------|---------|
| `token` | Client auth for CLI and app connections |
| `cluster_token` | Peer auth between cluster nodes |

**Check**

1. For CLI/app connection failures, verify `CLUSTO_TOKEN` matches the node's `token` or cluster-wide access token you intend to use.
2. For node-to-node failures, verify every node has the same `cluster_token`.
3. Inspect logs for `wrong token`, `auth rejected`, or timeout messages.
4. A blank `CLUSTO_TOKEN`, or an unexpanded `${CLUSTO_TOKEN}`, is reported with a
   message naming the cause rather than "invalid token". Export a real value or
   unset the variable so the CLI uses the local daemon's token.

**Fix**

* Update the client token you are using
* A hand-set `token` or `cluster_token` shorter than 32 characters makes the
  daemon log a startup warning, because it can be guessed offline from the
  identity proof any caller can request. Remove `token` from `config.toml` (or
  set a long generated value) and use the same generated `cluster_token` on every
  node
* If the CLI says the node "could not prove it knows your token", the token you
  have is not the one that node holds (see
  [section 18](#_18-node-identity-changed-or-an-identity-changed-badge))
* On Windows, "authentication failed: invalid token" in chats or sessions run as
  another user is fixed: run-as chats and sessions now receive the daemon's
  `CLUSTO_*` variables, which the run-as helper used to drop. Update to the
  latest version.
* Or update the daemon config so all peer nodes share the same `cluster_token`

## 4. A Node Goes Offline

**What to expect**

Leaderless clustering has no leader, no election, and no quorum, so there is no
failover to wait on. Every remaining node keeps serving reads and writes for the
state it already holds - a node going offline never stalls the rest of the
cluster. The only thing lost while a node is offline is interactive access to
the sessions it **owns**: agent and PTY processes are OS-local, so a session is
only interactive on its owning node and is unreachable until that node returns.

When the node comes back, anti-entropy on reconnect merges any state that
diverged while it was away (see section 2).

**Symptoms that are NOT a cluster fault**

* `clusto cluster-status` shows a peer as not connected - this is expected while
  that node is down; the rest of the cluster is unaffected.
* Sessions owned by the offline node fail to load or accept input - expected;
  they resume when the owner returns.

**Check**

1. Confirm the still-up nodes see each other as connected:

   ```bash
   clusto cluster-status
   ```

2. Confirm the offline node is actually down (vs. a peer-connectivity break) and
   bring it back or restore connectivity on `peer_port`.

## 5. Node Drops Out After Restart or IP Change

**Symptoms**

* A node was healthy before restart but disappears afterward
* DHCP or Tailscale address changed

**Check**

1. Confirm the node's current client-facing address.
2. Update `advertise_addresses` if the node should now be reached at a different `host:port`.
3. Update `peers` on the other nodes if the peer endpoint changed.

Example:

```toml
[cluster]
advertise_addresses = ["100.64.1.12:9178"]
peers = ["100.64.1.13:9179", "100.64.1.14:9179"]
```

**Fix**

* Save the corrected addresses
* Restart the changed node
* If needed, restart the peers so they reconnect using the updated config

## 6. Remote Update From URL Fails

**Check**

1. Verify the manifest URL is reachable:

   ```bash
   curl -I https://updates.example.com/clusto/manifest.json
   ```

2. Verify the manifest contains an artifact for the target platform.

3. For private endpoints, verify `CLUSTO_UPDATE_TOKEN` or `--remote-token`.

**Current commands**

```bash
clusto update --from https://updates.example.com/clusto/manifest.json
clusto update-node --from https://updates.example.com/clusto/manifest.json
```

**Fix**

* Correct the manifest URL
* Publish the missing platform artifact
* Supply the correct bearer token

## 7. Daemon Lags Under Heavy CPU Load From Other Processes

**Symptoms**

* Peer-connection keepalive pings time out and peers mark this node offline
  while the host is otherwise busy (media transcoding, builds, backups)
* WebSocket sessions stutter or disconnect even though the network is fine
* `clusto cluster-status` from another node reports this node as flapping

**What users now see in clients**

* Recent CLI and app builds surface non-fatal daemon notices instead of leaving
  the slowdown silent.
* Remote-session attaches can show a notice that the live PTY is running on a
  different cluster node and Clusto is forwarding traffic there.
* If a viewer falls behind on terminal output, the client can show warnings
  such as `This client is falling behind on live session output` or
  `Clusto stopped one live output stream because this client could not keep up`.

**What clustod does automatically**

* On every start, `clustod` raises its own CPU and I/O priority - `nice -10` on
  Linux/macOS via `setpriority(2)`, best-effort I/O priority 0 on Linux, and
  `HIGH_PRIORITY_CLASS` on Windows via `SetPriorityClass`. On Linux it also
  lowers its OOM score to make the daemon a last-resort kill target. This
  applies on fresh installs, self-updates, and manual restarts. A line like
  `raised CPU priority (nice = -10)` appears in the daemon log on success;
  a warning is logged if the OS denied the request.
* Spawned workload processes are pushed slightly below normal priority so
  Claude, shells, task commands, headless agents, and CPU-heavy builds do not
  compete with the daemon's control plane. On Linux they are set to `nice +2`,
  keep the normal `SCHED_OTHER` scheduler, and use the default best-effort I/O
  priority. Initial Linux placement happens before the process executes, so
  wrappers such as `runuser` cannot pass daemon priority to the real shell.
  The margin is intentionally small: the cgroup split below is the actual CPU
  reservation, and a large `nice` value would only make sessions lose against
  unrelated nice-0 processes on the host (browsers, container runtimes, CI
  runners) without protecting the daemon any further.
* A self-update keeps live PTYs running, so on start the daemon re-applies the
  current workload policy to every process already inside `workload/`. Sessions
  spawned by an older build (which used `nice +10` and `SCHED_BATCH`) are healed
  without a respawn; the log line is `re-applied workload scheduling policy to
  processes from a previous daemon`.
* On Linux, when the systemd unit delegates the cgroup (`Delegate=yes`), clustod
  splits its service cgroup into a `daemon/` control-plane leaf and a
  `workload/` subtree. The daemon leaf holds a `cpu.weight`/`io.weight` of
  10000 against the workload subtree's 1000, a 10:1 edge - more than enough for
  a socket-bound control plane that never saturates a core, without starving
  the sessions the node exists to run. Every PTY session and headless run gets
  its own child group, so a build with many compiler processes gets the same
  aggregate share as another interactive session instead of winning by process
  count. Descendants inherit the group, a guardian periodically returns marked
  descendants that escaped it, and empty per-session groups from ended sessions
  are pruned. This prevents one fan-out build from taking the control plane or
  another session with it.
* Linux reserves 256 MiB with `memory.min` and 512 MiB with `memory.low` for
  the daemon. The workload subtree gets `memory.high` at 80% of physical RAM,
  which asks the kernel to reclaim from workloads under pressure without
  imposing a hard cap. A line like `cgroup workload isolation enabled`
  appears on success; `cgroup workload isolation unavailable` (for example an
  older unit without `Delegate=yes`, or cgroup v1) means Clusto fell back to
  per-process nice and I/O priority. Isolation is best-effort and never blocks
  startup or session spawn.
* On Windows, clustod runs at `HIGH_PRIORITY_CLASS`, while each PTY session and
  headless run is assigned to its own inherited Job Object at
  `NORMAL_PRIORITY_CLASS` with low CPU scheduling weight. Tools and builds
  launched by the agent inherit that Job Object. This contains the Clusto process
  tree without changing the priority of unrelated Windows applications.
* The systemd units generated by the CLI, desktop app, and `clusto.ps1 deploy`
  set `Nice=-10`, `IOSchedulingClass=best-effort`,
  `IOSchedulingPriority=0`, `CPUWeight=10000`, `IOWeight=10000`,
  `OOMScoreAdjust=-900`, `Delegate=yes`, `MemoryMin=256M`, and `MemoryLow=512M`
  so the elevated priority and cgroup delegation are in place before the
  binary's own startup code runs.
* No application timer ever drops a client connection (SSH semantics). A
  widening pong gap is logged as `connection stale` and held; output sheds via
  the resumable ring instead of timing out; genuine death is detected by the
  kernel (TCP keepalive + `TCP_USER_TIMEOUT` on the listener) and surfaces as a
  socket error. The only daemon-initiated teardown is the zombie reap of a
  connection with no attached sessions that has also stopped answering pings.
  See [Keepalive](/transport#keepalive).
* If replicated state briefly loses a live PTY session's owner while output is
  still visible, the owner re-announces for several seconds and PTY input probes
  connected peers before surfacing `owner is not known yet`.
* The peer circuit breaker credits host-load stall before disconnecting. A node
  thrashing in swap cannot drain its own outbound channels in time and would
  otherwise time out every peer send at once and partition itself out of a
  healthy mesh. Instead, when at least 5 s of local scheduling stall is credited
  across a timeout streak, the link is held (`holding peer, not partitioning
  self`) and recovers when load drops; a genuinely dead peer is still reaped by
  the per-peer receiver task and TCP keepalive. See [Transport](/transport).
* Nothing slow runs on a connection's select loop. Session-stream operations,
  claims, kills, history fetches, and every peer-forwarding request are
  dispatched to ordered off-loop workers or detached tasks, so a slow peer's
  RPC timeout can never stall this client's heartbeat and output while a
  *different* node is overloaded - the same protection long applied to
  keystrokes and resizes.

**Check**

1. Confirm clustod is actually running with raised priority:

   ```bash
   # Linux
   ps -o pid,ni,cmd -C clustod
   # NI column should be -10 (or close to it)
   ```

   ```powershell
   # Windows
   Get-Process clustod | Select-Object Id, PriorityClass
   # PriorityClass should be High
   ```

2. On Linux, if the NI value is `0`, the daemon's self-elevation was denied -
   check `journalctl -u clustod | grep "raise CPU priority"` for the reason
   (usually capability/permission related).

3. Confirm session children are not inheriting clustod's elevated nice value:

   ```bash
   ps -eo pid,ppid,ni,cmd | grep -E 'clustod|claude|codex|bash|zsh|pwsh' | grep -v grep
   # clustod should be around -10; session children should normally be +2.
   ```

4. On Linux, inspect the workload hierarchy and current resource pressure:

   ```bash
   systemctl show clustod -p ControlGroup
   cat /proc/pressure/cpu /proc/pressure/memory /proc/pressure/io
   ```

   The service cgroup should contain `daemon/` and `workload/`; active sessions
   appear as children below `workload/`. The app's node card also shows Linux
   pressure, load, runnable tasks, Clusto workload CPU, and Clusto workload memory.
   Windows node cards continue to show CPU, memory, swap, and network, but do
   not show Linux PSI rows.

5. On Windows, confirm the daemon remains elevated:

   ```powershell
   Get-Process clustod | Select-Object Id, PriorityClass
   # PriorityClass should be High
   ```

**Fix**

* For a service install on Linux, ensure the unit was deployed via current
  `clusto.ps1 deploy` or `sudo clusto install` so the `Nice=` and
  `CPUWeight=` directives are present. If editing an existing unit by
  hand, add the priority directives under `[Service]` and `sudo systemctl daemon-reload &&
  sudo systemctl restart clustod`.
* If a specific offending process (e.g. a wedged transcoder) is monopolising
  the CPU, cap or kill it rather than raising clustod further - `REALTIME_PRIORITY_CLASS`
  / `SCHED_RR` are intentionally avoided because a hot loop in clustod under
  real-time scheduling can lock up the entire host.

## 8. A Long-Running Terminal Looks Corrupted After You Return

**Symptoms**

* A session runs for a long time while its terminal is in another workspace,
  tab, or backgrounded app.
* Returning shows scattered characters, duplicated TUI fragments, excessive
  blank space, or a layout that no longer resembles the running program.
* **Reload full history** or another forced redraw repairs the display.

**Cause**

* Older builds stopped parsing a terminal while it was hidden and rebuilt it
  from a daemon snapshot whenever it was shown again (and after every resize
  and app restore). Full-screen TUIs such as Claude Code repaint only the cells
  that changed, so any difference between the rebuilt screen and the real one
  stayed on screen: typed text landing on the status line, or leftover
  fragments of earlier frames, until a window resize forced a full repaint.
* Snapshots taken while a synchronized repaint was split across two PTY reads
  left out its first half, and snapshots of a full-screen program put its
  screen in the normal buffer instead of the alternate one.
* On native Windows, an older Tao event-loop path could also deadlock while
  holding its dispatcher lock across `PeekMessageW`, freezing the whole app.

**Fix**

* Update every Clusto app and daemon in the fleet to the same current build.
* Terminals keep parsing their live output while hidden, so switching back
  shows the exact screen without any rebuild. Reconnects resume every stream
  from its seq cursor. See [Switching terminals](/terminal-history#switching-terminals).
* Snapshots, used only for a terminal's first open or after output was really
  lost, include a repaint in progress, restore the alternate screen, and carry
  a split escape sequence across the boundary.
* Terminals share a bounded, foreground-priority render budget. Queue overflow
  triggers snapshot recovery instead of continuing with an incomplete escape
  stream, and recent input reduces background rendering work.
* The Windows app includes the upstream Tao event-loop deadlock fix.

**Check**

1. Leave a busy terminal in another workspace long enough for it to produce
   substantial output.
2. Return to it. The display should already show the current TUI state, with
   no history loading indicator.
3. If an older build is still deployed, use **Ctrl+Shift+H** or right-click the
   terminal and choose **Reload full history** as a temporary recovery.

## 9. Copy Is Disabled After Selecting Text Inside a TUI

**Symptoms**

* Text can be selected and copied normally at a shell prompt
* Selecting inside a full-screen TUI such as Claude Code does not enable
  **Copy** in Clusto's terminal context menu
* Ctrl+C interrupts the TUI instead of copying the highlighted text

**Cause**

Some TUIs own their mouse selection instead of using the terminal emulator's
native selection. The TUI asks the terminal to update the system clipboard with
an OSC 52 write, so Clusto has no native selection to expose through its context
menu.

**Fix**

Update to a build with OSC 52 clipboard support. A focused terminal that owns
the session input claim accepts write-only OSC 52 requests and copies the text
to the local system clipboard, including Unicode text. Passive viewers and
unfocused terminals cannot change the clipboard.

For safety, Clusto rejects clipboard-read requests, non-system clipboard
selections, malformed data, and decoded payloads larger than 1 MiB.

## 10. A Headless Agent (Team Member or Chat Session) Never Answers

**Symptom:** a team leader stays silent, its workers sit on `Pending`, or an
agent chat shows no reply. The daemon log has the agent exiting a couple of
seconds after spawn:

```
INFO  clusto_daemon::session::sdk_backend: SDK process spawned provider="claude"
WARN  clusto_daemon::session::sdk_backend: SDK process exited with non-zero code exit_code=1
```

When the process dies this early the prompt cannot be written to it. Clusto reports
the agent's own last stderr line (`agent process exited: <reason>`) rather than
the bare `SDK process stdin closed`, so the error in the app usually names the
cause directly.

**Cause:** a headless agent runs as an OS account, and the two common failures
both come from that account:

* **It is `root`.** clustod runs as root under systemd, and the Claude CLI refuses
  `--dangerously-skip-permissions` under root or sudo, so a `bypass`-mode team
  dies at startup. Clusto now rejects that combination when the team is created.
* **The account does not exist on this node.** Run-as names travel with a
  session (a project's `*` mapping, another node's `default_session_user`), so a
  team created against a directory owned by a different machine can be handed a
  username the spawning node has no account for. `runuser` then exits before the
  agent starts and the only trace is one stderr line:

  ```
  runuser: user ubuntu does not exist or the user entry does not contain all the required fields
  ```

  Clusto now refuses this at team creation and at spawn with `run-as user '<name>'
  does not exist on <host>`.
* **Its login expired.** The CLI retries the 401 and gives up. Note that
  `claude auth status` still reports `loggedIn: true` for a stale token, so it
  is not a reliable check; the daemon log line below is:
* **OpenRouter has no API key.** Native OpenRouter does not use a CLI login. It
  needs the cluster-wide `[openrouter].api_key`, or a local
  `OPENROUTER_API_KEY` environment fallback when no cluster key exists. The
  affected turn closes as failed and shows the credential error instead of
  remaining in **Starting**.
* **Team coordination is unavailable.** A leader may still answer ordinary
  questions while lacking the tools needed to register workers. New managed
  leaders and workers reject this state before spawn if the colocated `clusto` CLI is
  missing, does not match the `clustod` version, or lacks `clusto team mcp`.

```
WARN  clusto_daemon::session::agent_auth: agent authentication failed; raising AgentAuthRequired
```

**Fix:**

1. Give the project a **Run as** user (Projects > edit project), or set
   `default_session_user` in the node config, so agents stop running as root.
2. If the account's login is stale, use the **Sign in** button on the banner in
   the session or team view. It opens a terminal as that account running
   `claude auth login`. In the native desktop and mobile apps, the provider
   authorization link opens in the OS default browser, not Clusto's embedded
   webview. If Claude requests a verification code, return to Clusto after
   authorizing and enter it in the code-entry dialog, which remains open while
   the browser is active.
3. For OpenRouter, open **Settings > Account > OpenRouter API**, or use **F1 >
   Configure OpenRouter API key**. Save, replace, or clear the one write-only
   key for the whole cluster. Connected nodes apply the change to the next turn
   without restarting `clustod`, and offline nodes receive the latest value when
   they reconnect. If no cluster key exists, each node can use its own
   `OPENROUTER_API_KEY` environment fallback. Environment keys cannot be changed
   or cleared in the app.
4. Verify which account an agent used. This line is logged at `info`, so set
   `log_filter = "info"` (or `RUST_LOG=info`) first; the default `warn` filter
   hides it:

```bash
journalctl -u clustod | grep "spawning SDK process"
# program=/usr/sbin/runuser ... run_as_user=Some("ubuntu")   <- good
# program=claude ... run_as_user=None                        <- runs as the daemon user
```

5. If the leader says workers are ready but Clusto still shows zero workers,
   verify the coordination CLI on the daemon host:

```bash
clustod --version
clusto --version
clusto team mcp --help
```

The two versions must match. With an update source configured, clustod
refreshes a missing or stale colocated CLI in the background; retry team
creation after it finishes. If the affected team predates the update,
recreate it so its leader receives the authenticated daemon address, token,
TLS mode, MCP config, and current coordination instructions.

6. Treat the app roster as authoritative. Claude Task subagents and files under
   `.claude/agents` are provider-native helpers, not Clusto workers. Use
   **Add Worker** in the Teams list or team detail view, or ask the leader to
   call `team_assign` / `team_handoff` and confirm the returned session UUID
   with `team_status`.

## 11. A Project Session Starts on the Wrong Node

**Symptoms**

* **New Session** or **Quick New Session** starts a project session on the
  connected node instead of the node selected for the project
* After opening a session owned by another node, **Quick New Session** fails
  with a shell error such as `zsh:1: command not found: PowerShell`
* Editing the project, reselecting its node, and saving appears to fix later
  sessions
* Claiming a just-created remote session repeatedly reports `session owner is
  not known yet`

**Cause**

Older builds could create a project without a node ID when **This node** was
submitted during app startup before cluster status supplied the connected
node's UUID. Session creation then had no project node to target. Separately, a
remote session's ownership announcement could arrive after the first claim.
Older app builds could also choose a shell before resolving the quick session's
target node. Switching between sessions on different operating systems could
therefore send a Windows shell choice such as PowerShell to a Linux project
node.

**Fix**

Update Clusto across the whole cluster. In current builds, a cluster daemon pins
a project with no explicit node ID to itself. **New Session** and **Quick New
Session** always target the selected project's pinned node and report a clear
error if it is offline. Claim routing probes connected peers when ownership
replication is delayed, so a newly created remote session can be claimed
immediately. **Quick New Session** derives its target from the active project's
node, or from the active session's owning node when needed, then resolves the
shell choices available on that target node. It no longer carries a shell
choice from one operating system to a session created on another.

## 12. Mouse Wheel Does Not Scroll in a Full-Screen TUI

Full-screen terminal apps use the alternate screen. If the app enables terminal
mouse tracking, Clusto forwards mouse-wheel and touch-scroll input to it. Like
native terminals, it sends one wheel report per line scrolled: a mouse-wheel
notch scrolls three lines and a trackpad scrolls with its travel, so Claude Code's
fullscreen view follows the gesture instead of moving one line per event. If it
does not, Clusto converts scrolling to Page Up or Page Down so the TUI can still
navigate. This covers GitHub Copilot CLI when its mouse integration is disabled
or unavailable. The Page Up and Page Down keys continue to work directly.

If Claude Code's fullscreen view scrolls in jumps instead of a steady three
lines per notch, it is accelerating the wheel: it multiplies the lines per
report during fast scrolls (up to 6 on Linux, up to 36 on Windows, where it also
starts at 3 lines per report). Each node's AI setup turns this off with the
**Claude Code smooth scrolling** check, which writes
`"wheelScrollAccelerationEnabled": false` and `"env": { "CLAUDE_CODE_SCROLL_SPEED": "1" }`
to `~/.claude/settings.json` when those keys are absent. Run **Fix now** on the
AI setup card (or **Settings > Node Doctor > Fix**) if the node has not applied
it yet, and check that the node's AI setup is on. A value you set yourself is
kept, so look in that file if the jumps persist. See
[Knowledge](/knowledge#what-happens-on-each-node).

## 13. Sidebar RTT Is High or Missing

The native app's sidebar shows the latest matched WebSocket heartbeat round-trip
time, refreshed every 15 seconds. Earlier measurements used a cluster-status
request and could include queued application work and frontend processing. See
[Sidebar RTT](/transport#sidebar-rtt) for the current measurement boundaries.

* After connecting or recovering from a stall, wait for the next matching
  heartbeat reply. The previous connection's sample is cleared.
* Compare several samples while using the app. One spike does not establish a
  sustained connection problem.
* Record the connected entry node and whether the app uses a direct address or
  Clusto Tunnel. RTT measures that connection, while a terminal hosted on a
  different node also depends on the link between nodes.
* For sustained high RTT, check network reachability, relay or tunnel connection
  errors, and CPU load on the client and entry node. A normal system `ping`
  result alone does not rule out transport buffering or runtime delays.
* In the browser client, an absent RTT is expected: browser APIs do not expose
  WebSocket control-frame timing.

## 14. Browser or App Rejects the Node's TLS Certificate

Symptoms: the web client or app cannot connect over `wss://`, or the browser
reports an expired certificate for the node.

* A daemon serving a `clusto cert` (Tailscale) certificate renews it itself when
  less than a third of its lifetime is left, and hot-reloads it within the hour
  with no restart. Check the daemon log (`log_filter = "info"`) for the renewal
  attempt and any `clusto cert` error, such as Tailscale missing or HTTPS
  Certificates not enabled for the tailnet.
* A certificate installed at a custom path is never renewed by the daemon. The
  daemon logs a warning when it is due; replace the file (the change is
  hot-reloaded) or run `sudo clusto cert` to switch to a managed certificate.
* The desktop and mobile apps and the CLI do not use the certificate: they connect
  with TLS pinned to the node's identity key. Only browsers (the web client) and
  other plain TLS clients depend on it.
* Chat tools, the Clusto Assistant, team MCP servers and Stop hooks do not depend
  on the certificate: inside a session they use a plain loopback listener on
  `127.0.0.1`. If only the browser is affected, the certificate is the cause. If
  those tools fail with a certificate error, the daemon is older than this
  behaviour; update it.
* The first-time enable of TLS needs a restart: `sudo clusto cert --restart`. See
  [Web client](/web-client#getting-a-tls-domain-for-a-node).

## 15. Version Mismatch Notices

Clusto does not support mixed versions: the app, the CLI, and every node should
run the same release. A mismatch is not repaired automatically, but it is now
visible instead of failing silently.

**Symptoms**

* The app shows a warning toast "Node X runs A, app runs B - update the node"
  (or "update the app") with **Update node** and **Check for update** actions
* The CLI prints "this node runs X and does not understand the request from a
  client running Y; update the node" and exits non-zero
* Node logs show "peer NAME runs A, this node runs B: message not understood,
  update the older node" (rate limited)

**Fix**

Update the older side. Use **Update node** in the toast or Cluster view,
`clusto update-node`, or **Check for update** for the app, then confirm with
`clustod --version` and `clusto --version`. See [Remote Updates](/remote-updates).

## 16. "This Clusto node refuses unencrypted connections" (403)

**Symptoms**

* A script, browser page, `curl`, Prometheus scrape or old app gets `403` with
  "This Clusto node refuses unencrypted connections from" followed by its address
* The Android [background notification](/background-notifications) check is
  paused or its polls are refused

**Cause.** The client port only accepts credentials (`/ws`, `/notifications`,
`/metrics`, `/clients/...`) over an unencrypted connection from loopback,
Tailscale, or a network the operator listed. Everything else gets `403` before a
token is sent. `/health` and `/identify` always work. See
[Node Identity & Pairing](/node-identity#unencrypted-connections).

**Fix**

* Use the Clusto app or the CLI: both are always encrypted.
* Connect over Tailscale or [Clusto Tunnel](/clusto-tunnel).
* For a script or scraper, use `https`/`wss` with the node's certificate
  (`tls_cert` and `tls_key`), or list its network in `config.toml`:

  ```toml
  [transport]
  plaintext_networks = ["192.168.1.0/24"]
  ```

  The change is picked up without a restart. Remember a listed network sends the
  token in clear.
* An old app on a LAN-only address cannot reach an updated node and cannot
  download its own update from it, because credential routes over unencrypted
  HTTP are refused from other networks. Update it from `clusto.app` (Settings >
  Updates > Update from URL), or list the network in `[transport]
  plaintext_networks` on that node. See [Node Identity & Pairing](/node-identity#unencrypted-connections).

## 17. Peers show offline after an update (426 from a peer)

**Symptoms**

* After updating some nodes, the updated ones show the others as offline and the
  others show them offline
* The daemon log shows a peer answering `426 Upgrade Required`, or a peer link
  failing the handshake
* The app says "This node runs Clusto X, older than this app. Update the node or
  connect over the Clusto Tunnel."

**Cause.** Clusto does not run mixed versions. The peer port speaks only the
current encrypted protocol, so a node on another release cannot link, and the app
and CLI cannot dial an older node directly.

**Fix.** Update every node to the same release (**Cluster > Update All**, or
`clusto update-node` on each machine), then the links come back on their own.
Update the CLI with `clusto update`. Over the Clusto Tunnel an older node is
still reachable, so you can update it from there.

## 18. "Node identity changed" or an "Identity changed" badge

**Symptoms**

* The app shows **Node identity changed** and stops reconnecting
* The CLI prints "presented identity ..., but ... is recorded in known_nodes.toml,
  and the node could not prove it knows your token" or "identity check ... failed"
* A peer in Cluster Status is offline with an **Identity changed** badge
* A toast says a node "has a new identity ... verified with your token"

**Cause.** The node presents a different key than the one remembered. The usual
reasons are a reinstall, a restore without `identity.key`, a lost data directory,
or a node that was replaced under the same name or address. It can also be an
interception on that network, which is why nothing is sent until it is sorted out.

**Check**

1. Did you reinstall or rebuild that node? If not, do not proceed: you may be
   talking to something else. Compare `clusto identity show` run on the node with
   the identity the client reports.
2. Is the token correct? A node that cannot prove it holds your token is refused.

**Fix**

* App: pair again with a pairing code or fix the token, using **Edit profile** in
  the dialog.
* CLI: `clusto identity forget <host:port>`, then connect again.
* Peer in the cluster: choose **Trust new identity** on its card and confirm, or
  adopt it again with a pairing code. `clusto doctor --fix` on any other node
  offers the same trust step, and removes the stale entry a re-adopted node
  leaves behind. It also offers to remove a leftover roster address that no
  node owns (for example a retired machine's hostname), on every node.
* If the identity changes again on its own, the node is losing `identity.key`:
  check that only one daemon runs on that machine and that the service uses the
  same data directory each time (`clusto doctor` shows the service's config path).
* Back up `identity.key` with `config.toml` so reinstalls keep the identity. See
  [Node Identity & Pairing](/node-identity#when-an-identity-changes).

## 19. Pairing is locked, expired or "reset required"

**Symptoms**

* "wrong pairing code or password", "the pairing code has expired", "pairing is
  not enabled on this node", or "the node's pairing secret is too weak and must be
  reset on the node"
* "pairing is locked after too many wrong attempts"
* `clusto pairing status` shows Expired, Locked or Disabled

**Fix**

* Run `clusto pairing status` (or open **Configure Node > Pairing code**) on the
  node to see its state.
* **Expired** or **single use already spent**: generate a new code with
  `clusto pairing generate`. A single-use code works once.
* **Locked**: ten wrong attempts in 15 minutes lock pairing for the whole node
  for 15 minutes, doubling per consecutive lockout up to 24 hours. Generating or
  setting a new code unlocks it; logging in with the token is unaffected.
* **Reset required** (an old `user_secret` shorter than 16 characters was
  disabled when the node updated): set a new code or a password of at least 16
  characters.
* A code is typed without regard to case, dashes or spaces; `O` is read as `0`
  and `I`/`L` as `1`. A custom password is exact.
* If the node itself is unreachable, check section 16 first.

## 20. "... is not permitted with a session credential"

An agent or script inside a session that runs as a non-root account holds a
limited credential, not the admin token. Some requests are refused for it, or
must run as the session's own account. Use the Clusto app or pass the node admin
token for those. See [Session Credentials](/session-credentials), which lists
each message.

## 21. The daemon will not start or an old Hive node is stuck

Run `clusto doctor` on the machine. It checks the `clustod` and legacy `hived`
services, Hive-era state on disk and the cutover journal, then prints one
diagnosis. `clusto doctor --fix` (Administrator terminal on Windows, `sudo` on
Linux) starts a stopped `clustod`, stops a `hived` running beside it, removes a
stale `hived` registration, upgrades a Hive-era node in place, and rolls back
and retries a cutover that stopped part-way. See
[Moving from Hive to Clusto](/migrating-from-hive#repair-or-finish-a-stuck-node).

A user install (`clusto install --user`) that stops after you log out was
installed without lingering. Check with `loginctl show-user "$USER" -p Linger`;
an administrator enables it once with `sudo loginctl enable-linger "$USER"`.
If the installer said the user manager was not reachable, the daemon was started
detached and will not return after a reboot: log in normally and run
`clusto install --user` again. See
[Install without sudo](/downloads#install-without-sudo-or-administrator-rights).

## 22. Typing in a Terminal Feels Slow

**Symptoms**

* Characters appear late after you press a key
* The screen draws in pieces, or scrolling stutters

**Check**

1. Turn on the latency overlay: **Settings > Terminal > Diagnostics > Show the
   terminal latency overlay**, or run **Show the terminal latency overlay** from
   the `F1` palette. It stays on this device.
2. Type in a terminal. The overlay in the corner of every terminal shows p50 and
   p95 times in milliseconds for each stage a keystroke passes through:

   * **total** - from the key press to the character painted on screen.
   * **key** - the app handling the key press.
   * **invoke** - handing the input to the app's native side.
   * **native** - the native side sending it on to the node.
   * **echo** - waiting for the node to send the echoed output back.
   * **parse** - the terminal parsing the returned output.
   * **paint** - drawing the result.

   It also shows output **events/s** and **KB/s**, **long tasks** (stalls of the
   interface thread in the last 10 seconds) and the active **renderer**.
3. A large **echo** points at the network or the node. On a slow link the node
   sends a keystroke echo ahead of bulk traffic, so typing stays responsive while
   another terminal floods, and `Ctrl+C` on a flooding terminal stops the old
   output within a second; if echo stays high anyway, the link or the node's CPU
   is the limit. The **Keystroke echo** and **Frame gaps** rows under
   **Settings > Terminal > Diagnostics > Measured on this device** show the same
   numbers without the overlay. Large **parse**,
   **paint** or **long tasks** values point at the device. Under **Renderer**
   in the same card, try **WebGL** or **DOM** instead of **Automatic**; it
   applies to terminals opened after the change.

**Share the numbers**

The app writes a `terminal-latency` line to its log every 50 keystrokes. Open
the **Logs** view and export the log to attach those lines to a bug report.

## Useful Commands

```bash
# Check the local daemon install and AI setup
clusto doctor

# Show cluster topology
clusto cluster-status

# Discover Clusto nodes on Tailscale
clusto discover

# Read local daemon status
clusto daemon status

# Read local daemon config
clusto daemon get-config

# Read the credential this account holds (the admin token with sudo; a limited
# session credential for other accounts when that flag is on)
clusto daemon get-token

# This node's identity and pairing state
clusto identity show
clusto pairing status

# Check daemon health endpoint
curl http://localhost:9178/health

# Follow the daemon log (raise verbosity with log_filter in config.toml)
journalctl -u clustod -f
```

## Config Paths

See [Configuration](/configuration) for the full reference. Common paths:

* Linux: `~/.config/clusto/config.toml`
* macOS: `~/Library/Application Support/dev.vazac.Clusto/config.toml`
* Windows: `C:\Users\<user>\AppData\Roaming\vazac\Clusto\config\config.toml`

`clusto daemon status` prints the path selected on the current machine.
