Cluster Troubleshooting Runbook

This page covers the current operational checks for Hive cluster deployments. It focuses on commands and config fields that exist in the current CLI and daemon.

1. Node Does Not Join the Cluster

Symptoms

• hive cluster-status never shows the new node
• Logs show repeated peer connection failures
• The node starts, but no peer connections appear

Check

1. Verify the shared cluster secret on every node:

   [cluster]
   cluster_token = "same-secret-on-every-node"

2. Verify peer addresses:

   [cluster]
   peers = ["10.0.0.2:9179", "10.0.0.3:9179"]
   peer_port = 9179

3. Verify the node is advertising a reachable client address:

   [cluster]
   advertise_addresses = ["10.0.0.1:9178"]

4. Verify network reachability:

   curl http://10.0.0.1:9178/health
   nc -zv 10.0.0.1 9179

Fix

• Correct cluster_token, peers, peer_port, or advertise_addresses
• Restart the daemon after changing cluster settings

2. Peers Connect but Sessions Do Not Replicate

Symptoms

• A node does not show sessions or projects created on a peer
• A node that was offline comes back but still shows stale state

Cause

Leaderless replication gossips each local write to all peers once and runs anti-entropy (both sides exchange a full StateSnapshot, merged per-object) on every reconnect. If a node is missing a peer's objects, the gossip message was lost while the peer was unreachable and the reconnect anti-entropy has not yet run - almost always a peer-connectivity problem, not a state-machine wedge.

Check

1. Confirm both nodes see each other as connected:

   hive cluster-status

2. Inspect logs for the anti-entropy snapshot exchange on reconnect:

   RUST_LOG=clusto_cluster=debug,clusto_daemon=debug hived

   Look for sent anti-entropy snapshot to newly connected peer.

3. Confirm each node can still reach the other on both the client and peer ports.

Fix

• Restore peer connectivity. Once the peer connection re-establishes, the reconnect anti-entropy exchange merges both sides' state automatically - no restart needed. Distinct objects are never lost; concurrent edits to the same object resolve last-writer-wins by updated_at.
• If a node still does not converge, re-check peers, peer_port, and advertise_addresses on both sides.

3. Client Auth Works on One Node but Fails on Another

Hive uses two different tokens:

Token	Purpose
token	Client auth for CLI and app connections
cluster_token	Peer auth between cluster nodes

Check

1. For CLI/app connection failures, verify HIVE_TOKEN matches the node's token or cluster-wide access token you intend to use.
2. For node-to-node failures, verify every node has the same cluster_token.
3. Inspect logs for wrong token, auth rejected, or timeout messages.

Fix

• Update the client token you are using
• Or update the daemon config so all peer nodes share the same cluster_token

4. A Node Goes Offline

What to expect

Leaderless clustering has no leader, no election, and no quorum, so there is no failover to wait on. Every remaining node keeps serving reads and writes for the state it already holds - a node going offline never stalls the rest of the cluster. The only thing lost while a node is offline is interactive access to the sessions it owns: agent and PTY processes are OS-local, so a session is only interactive on its owning node and is unreachable until that node returns.

When the node comes back, anti-entropy on reconnect merges any state that diverged while it was away (see section 2).

Symptoms that are NOT a cluster fault

• hive cluster-status shows a peer as not connected - this is expected while that node is down; the rest of the cluster is unaffected.
• Sessions owned by the offline node fail to load or accept input - expected; they resume when the owner returns.

Check

1. Confirm the still-up nodes see each other as connected:

   hive cluster-status

2. Confirm the offline node is actually down (vs. a peer-connectivity break) and bring it back or restore connectivity on peer_port.

5. Node Drops Out After Restart or IP Change

Symptoms

• A node was healthy before restart but disappears afterward
• DHCP or Tailscale address changed

Check

1. Confirm the node's current client-facing address.
2. Update advertise_addresses if the node should now be reached at a different host:port.
3. Update peers on the other nodes if the peer endpoint changed.

Example:

[cluster]
advertise_addresses = ["100.64.1.12:9178"]
peers = ["100.64.1.13:9179", "100.64.1.14:9179"]

Fix

• Save the corrected addresses
• Restart the changed node
• If needed, restart the peers so they reconnect using the updated config

6. Remote Update From URL Fails

Check

1. Verify the manifest URL is reachable:

   curl -I https://updates.example.com/hive/manifest.json

2. Verify the manifest contains an artifact for the target platform.

3. For private endpoints, verify HIVE_UPDATE_TOKEN or --remote-token.

Current commands

hive update --from https://updates.example.com/hive/manifest.json
hive update-node --from https://updates.example.com/hive/manifest.json

Fix

• Correct the manifest URL
• Publish the missing platform artifact
• Supply the correct bearer token

7. Mobile App Is Very Slow When Several Sessions Are Already Open

Symptoms

• The app launches or resumes on Android, but the workspace stays sluggish for several seconds
• Switching into the workspace with many restored sessions causes delayed paints or input lag
• Daemon logs show a burst of GetSessionHistory requests immediately after reconnect

Cause

• The app restores background session subscriptions after reconnect.
• Older builds also hydrated history for every persisted pane, even when its terminal was not currently mounted or visible.
• On mobile, replaying several large histories at once can overwhelm the WebView and stall the first interactive paint.

Fix

• Update to a build that refreshes history eagerly only for terminals that are mounted and visible.
• Multiple visible terminals recover sequentially on mobile. Background panes still re-subscribe for live output, but defer history replay until opened.
• Mobile initially loads the newest 600 rows per terminal instead of 3000. Older rows remain available through scroll-to-top paging.

Check

1. Open Logs or the daemon log folder.
2. Reproduce an app launch/resume with several open sessions.
3. Confirm that reconnect shows history fetches for the visible terminals, one at a time, rather than every restored pane session.

8. Android Gboard Suggestion Taps Duplicate Earlier Text In Form Fields

Symptoms

• Accepting a suggested word in an app form field inserts the earlier text again
• The duplication is intermittent and happens more often with rapid mobile typing
• Plain typing usually works, but suggestion-chip picks can replay part or all of the existing buffer

Cause

• Android IME composition can keep an uncommitted in-progress value inside the input element.
• Older builds only stored the parent v-model value, so any rerender during composition wrote that stale value back into the DOM.
• Gboard then interpreted the stale write-back as a reset and re-sent the buffered text.

Fix

• Update to a build that keeps a local in-progress value inside shared app Input and Textarea components until composition ends.

Check

1. Open any form field in the app, such as session rename, task command, or project notes.
2. Type a few words with Gboard and accept a suggestion chip mid-sentence.
3. Confirm the field now keeps only the intended text instead of re-inserting the earlier buffer.

9. Daemon Lags Under Heavy CPU Load From Other Processes

Symptoms

• Peer-connection keepalive pings time out and peers mark this node offline while the host is otherwise busy (media transcoding, builds, backups)
• WebSocket sessions stutter or disconnect even though the network is fine
• hive cluster-status from another node reports this node as flapping

What users now see in clients

• Recent CLI and app builds surface non-fatal daemon notices instead of leaving the slowdown silent.
• Remote-session attaches can show a notice that the live PTY is running on a different cluster node and Hive is forwarding traffic there.
• If a viewer falls behind on terminal output, the client can show warnings such as This client is falling behind on live session output or Hive stopped one live output stream because this client could not keep up.

What hived does automatically

• On every start, hived raises its own CPU and I/O priority - nice -10 on Linux/macOS via setpriority(2), best-effort I/O priority 0 on Linux, and HIGH_PRIORITY_CLASS on Windows via SetPriorityClass. On Linux it also lowers its OOM score to make the daemon a last-resort kill target. This applies on fresh installs, self-updates, and manual restarts. A line like raised CPU priority (nice = -10) appears in the daemon log on success; a warning is logged if the OS denied the request.
• Spawned workload processes are pushed slightly below normal priority so Claude, shells, task commands, headless agents, and CPU-heavy builds do not compete with the daemon's control plane. On Linux they are set to nice +2, keep the normal SCHED_OTHER scheduler, and use the default best-effort I/O priority. Initial Linux placement happens before the process executes, so wrappers such as runuser cannot pass daemon priority to the real shell. The margin is intentionally small: the cgroup split below is the actual CPU reservation, and a large nice value would only make sessions lose against unrelated nice-0 processes on the host (browsers, container runtimes, CI runners) without protecting the daemon any further.
• A self-update keeps live PTYs running, so on start the daemon re-applies the current workload policy to every process already inside workload/. Sessions spawned by an older build (which used nice +10 and SCHED_BATCH) are healed without a respawn; the log line is re-applied workload scheduling policy to processes from a previous daemon.
• On Linux, when the systemd unit delegates the cgroup (Delegate=yes), hived splits its service cgroup into a daemon/ control-plane leaf and a workload/ subtree. The daemon leaf holds a cpu.weight/io.weight of 10000 against the workload subtree's 1000, a 10:1 edge - more than enough for a socket-bound control plane that never saturates a core, without starving the sessions the node exists to run. Every PTY session and headless run gets its own child group, so a build with many compiler processes gets the same aggregate share as another interactive session instead of winning by process count. Descendants inherit the group, a guardian periodically returns marked descendants that escaped it, and empty per-session groups from ended sessions are pruned. This prevents one fan-out build from taking the control plane or another session with it.
• Linux reserves 256 MiB with memory.min and 512 MiB with memory.low for the daemon. The workload subtree gets memory.high at 80% of physical RAM, which asks the kernel to reclaim from workloads under pressure without imposing a hard cap. A line like cgroup workload isolation enabled appears on success; cgroup workload isolation unavailable (for example an older unit without Delegate=yes, or cgroup v1) means Hive fell back to per-process nice and I/O priority. Isolation is best-effort and never blocks startup or session spawn.
• On Windows, hived runs at HIGH_PRIORITY_CLASS, while each PTY session and headless run is assigned to its own inherited Job Object at NORMAL_PRIORITY_CLASS with low CPU scheduling weight. Tools and builds launched by the agent inherit that Job Object. This contains the Hive process tree without changing the priority of unrelated Windows applications.
• The systemd units generated by the CLI, desktop app, and clusto.ps1 deploy set Nice=-10, IOSchedulingClass=best-effort, IOSchedulingPriority=0, CPUWeight=10000, IOWeight=10000, OOMScoreAdjust=-900, Delegate=yes, MemoryMin=256M, and MemoryLow=512M so the elevated priority and cgroup delegation are in place before the binary's own startup code runs.
• No application timer ever drops a client connection (SSH semantics). A widening pong gap is logged as connection stale and held; output sheds via the resumable ring instead of timing out; genuine death is detected by the kernel (TCP keepalive + TCP_USER_TIMEOUT on the listener) and surfaces as a socket error. The only daemon-initiated teardown is the zombie reap of a connection with no attached sessions that has also stopped answering pings. See docs/transport.md (Keepalive).
• If replicated state briefly loses a live PTY session's owner while output is still visible, the owner re-announces for several seconds and PTY input probes connected peers before surfacing owner is not known yet.
• The peer circuit breaker credits host-load stall before disconnecting. A node thrashing in swap cannot drain its own outbound channels in time and would otherwise time out every peer send at once and partition itself out of a healthy mesh. Instead, when at least 5 s of local scheduling stall is credited across a timeout streak, the link is held (holding peer, not partitioning self) and recovers when load drops; a genuinely dead peer is still reaped by the per-peer receiver task and TCP keepalive. See docs/transport.md.
• Nothing slow runs on a connection's select loop. Session-stream operations, claims, kills, history fetches, and every peer-forwarding request are dispatched to ordered off-loop workers or detached tasks, so a slow peer's RPC timeout can never stall this client's heartbeat and output while a different node is overloaded - the same protection long applied to keystrokes and resizes.

Check

1. Confirm hived is actually running with raised priority:

   # Linux
   ps -o pid,ni,cmd -C hived
   # NI column should be -10 (or close to it)

   # Windows
   Get-Process hived | Select-Object Id, PriorityClass
   # PriorityClass should be High

2. On Linux, if the NI value is 0, the daemon's self-elevation was denied - check journalctl -u hived | grep "raise CPU priority" for the reason (usually capability/permission related).

3. Confirm session children are not inheriting hived's elevated nice value:

   ps -eo pid,ppid,ni,cmd | grep -E 'hived|claude|codex|bash|zsh|pwsh' | grep -v grep
   # hived should be around -10; session children should normally be +10.

4. On Linux, inspect the workload hierarchy and current resource pressure:

   systemctl show hived -p ControlGroup
   cat /proc/pressure/cpu /proc/pressure/memory /proc/pressure/io

   The service cgroup should contain daemon/ and workload/; active sessions appear as children below workload/. The app's node card also shows Linux pressure, load, runnable tasks, Hive workload CPU, and Hive workload memory. Windows node cards continue to show CPU, memory, swap, and network, but do not show Linux PSI rows.

5. On Windows, confirm the daemon remains elevated:

   Get-Process hived | Select-Object Id, PriorityClass
   # PriorityClass should be High

Fix

• For a service install on Linux, ensure the unit was deployed via current clusto.ps1 deploy or hive daemon install so the Nice= and CPUWeight= directives are present. If editing an existing unit by hand, add the priority directives under [Service] and sudo systemctl daemon-reload && sudo systemctl restart hived.
• If a specific offending process (e.g. a wedged transcoder) is monopolising the CPU, cap or kill it rather than raising hived further - REALTIME_PRIORITY_CLASS / SCHED_RR are intentionally avoided because a hot loop in hived under real-time scheduling can lock up the entire host.

10. Terminal Cursor Leaves Blinking Copies While Claude or Codex Is Working

Symptoms

• While a Claude Code or Codex PTY session is actively repainting, the cursor appears to blink in old positions around the screen
• The text itself keeps updating, but stale cursor blocks briefly remain behind

Cause

• Some Chromium-backed canvas/WebGL terminal renderer paths can leave cursor blink artifacts behind during rapid TUI repaint bursts.

Fix

• Update to a build that suppresses cursor blinking while output is actively streaming, then restores it after the burst goes idle with a clean repaint.

11. Copy Is Disabled After Selecting Text Inside a TUI

Symptoms

• Text can be selected and copied normally at a shell prompt
• Selecting inside a full-screen TUI such as Claude Code does not enable Copy in Hive's terminal context menu
• Ctrl+C interrupts the TUI instead of copying the highlighted text

Cause

Some TUIs own their mouse selection instead of using the terminal emulator's native selection. The TUI asks the terminal to update the system clipboard with an OSC 52 write, so Hive has no native selection to expose through its context menu.

Fix

Update to a build with OSC 52 clipboard support. A focused terminal that owns the session input claim accepts write-only OSC 52 requests and copies the text to the local system clipboard, including Unicode text. Passive viewers and unfocused terminals cannot change the clipboard.

For safety, Hive rejects clipboard-read requests, non-system clipboard selections, malformed data, and decoded payloads larger than 1 MiB.

12. Ctrl+C Does Not Stop a Windows Foreground Command

Symptoms

• A Windows session is running PowerShell through the Hive service or a configured run-as user
• Pressing Ctrl+C while a foreground command such as npm run tauri dev is running either does nothing or closes the whole Hive session instead of stopping only that command

Fix

Update Hive on the Windows node. Current builds relay Ctrl+C across the Windows service-to-user session boundary, keeping the session helper alive while the foreground command is interrupted. The PowerShell prompt remains available in the same session.

13. A Headless Agent (Team Member or Chat Session) Never Answers

Symptom: a team leader stays silent, its workers sit on Pending, or an agent chat shows no reply. The daemon log has the agent exiting a couple of seconds after spawn:

INFO  hived::session::sdk_backend: claude SDK process spawned
WARN  hived::session::sdk_backend: SDK process exited with non-zero code exit_code=1

When the process dies this early the prompt cannot be written to it. Hive reports the agent's own last stderr line (agent process exited: <reason>) rather than the bare SDK process stdin closed, so the error in the app usually names the cause directly.

Cause: a headless agent runs as an OS account, and the two common failures both come from that account:

• It is root. hived runs as root under systemd, and the Claude CLI refuses --dangerously-skip-permissions under root or sudo, so a bypass-mode team dies at startup. Hive now rejects that combination when the team is created.

• The account does not exist on this node. Run-as names travel with a session (a project's * mapping, another node's default_session_user), so a team created against a directory owned by a different machine can be handed a username the spawning node has no account for. runuser then exits before the agent starts and the only trace is one stderr line:

  runuser: user ubuntu does not exist or the user entry does not contain all the required fields

  Hive now refuses this at team creation and at spawn with run-as user '<name>' does not exist on <host>.

• Its login expired. The CLI retries the 401 and gives up. Note that claude auth status still reports loggedIn: true for a stale token, so it is not a reliable check; the daemon log line below is:

• OpenRouter has no API key. Native OpenRouter does not use a CLI login. It needs OPENROUTER_API_KEY in the daemon environment or [openrouter].api_key in the node's private config.

• Team coordination is unavailable. A leader may still answer ordinary questions while lacking the tools needed to register workers. New Claude team sessions reject this state before spawn if the colocated hive CLI is missing, does not match the hived version, or lacks hive team mcp.

WARN  hived::session::agent_auth: agent authentication failed; raising AgentAuthRequired

Fix:

1. Give the project a Run as user (Projects > edit project), or set default_session_user in the node config, so agents stop running as root.
2. If the account's login is stale, use the Sign in button on the banner in the session or team view. It opens a terminal as that account running claude auth login. In the native desktop and mobile apps, the provider authorization link opens in the OS default browser, not Hive's embedded webview. If Claude requests a verification code, return to Hive after authorizing and enter it in the code-entry dialog, which remains open while the browser is active.
3. For OpenRouter, open Settings > Appearance > OpenRouter API, or use **F1

   Configure OpenRouter API key**. The card shows only whether the key is set and whether it comes from config or the environment. Save a key to create or replace the private config value, or clear a stored config key. Environment keys cannot be cleared in the app. The next turn uses the change immediately without restarting hived.

4. Verify which account an agent used:

journalctl -u hived | grep "spawning SDK process"
# program=/usr/sbin/runuser ... run_as_user=Some("ubuntu")   <- good
# program=claude ... run_as_user=None                        <- runs as the daemon user

5. If the leader says workers are ready but Hive still shows zero workers, verify the coordination CLI on the daemon host:

hived --version
hive --version
hive team mcp --help

The two versions must match. With an update channel configured, hived refreshes a missing or stale colocated CLI in the background; retry team creation after it finishes. If the affected team predates the update, recreate it so its leader receives the authenticated daemon address, token, TLS mode, MCP config, and current coordination instructions.

6. Treat the app roster as authoritative. Claude Task subagents and files under .claude/agents are provider-native helpers, not Hive workers. Use Add Worker in the Teams list or team detail view, or ask the leader to call team_assign / team_handoff and confirm the returned session UUID with team_status.

14. A Project Session Starts on the Wrong Node

Symptoms

• New Session or Quick New Session starts a project session on the connected node instead of the node selected for the project
• After opening a session owned by another node, Quick New Session fails with a shell error such as zsh:1: command not found: PowerShell
• Editing the project, reselecting its node, and saving appears to fix later sessions
• Claiming a just-created remote session repeatedly reports session owner is not known yet

Cause

Older builds could create a project without a node ID when This node was submitted during app startup before cluster status supplied the connected node's UUID. Session creation then had no project node to target. Separately, a remote session's ownership announcement could arrive after the first claim. Older app builds could also choose a shell before resolving the quick session's target node. Switching between sessions on different operating systems could therefore send a Windows shell choice such as PowerShell to a Linux project node.

Fix

Update Hive across the whole cluster. In current builds, a cluster daemon pins a project with no explicit node ID to itself. New Session and Quick New Session always target the selected project's pinned node and report a clear error if it is offline. Claim routing probes connected peers when ownership replication is delayed, so a newly created remote session can be claimed immediately. Quick New Session derives its target from the active project's node, or from the active session's owning node when needed, then resolves the shell choices available on that target node. It no longer carries a shell choice from one operating system to a session created on another.

15. Mouse Wheel Does Not Scroll in a Full-Screen TUI

Full-screen terminal apps use the alternate screen. If the app enables terminal mouse tracking, Hive forwards mouse-wheel and touch-scroll input to it. If it does not, Hive converts scrolling to Page Up or Page Down so the TUI can still navigate. This covers GitHub Copilot CLI when its mouse integration is disabled or unavailable. The Page Up and Page Down keys continue to work directly.

Useful Commands

# Show cluster topology
hive cluster-status

# Discover Hive nodes on Tailscale
hive discover

# Read local daemon status
hive daemon status

# Read local daemon config
hive daemon get-config

# Read local daemon token
hive daemon get-token

# Check daemon health endpoint
curl http://localhost:9178/health

# Run the daemon with verbose logs
RUST_LOG=clusto_cluster=debug,clusto_daemon=debug hived

Config Paths

See configuration.md for the full reference. Common paths:

• Linux: ~/.config/hive/Hive/config.toml
• macOS: ~/Library/Application Support/com.hive.Hive/config.toml
• Windows: C:\Users\<user>\AppData\Roaming\hive\Hive\config\config.toml