---
url: https://clusto.app/docs/transport.md
description: >-
  How Clusto transports terminal input and output, measures connection RTT,
  recovers gaps, and isolates latency across clients and cluster nodes.
---

# Terminal Transport & Resume

How session output reaches clients reliably over flaky links, and how a client
resumes a stream after a disconnect without a full history refetch or a screen
clear. This complements [Protocol](/protocol) (the full wire vocabulary) and
[Clustering](/clustering) (cross-node routing).

> Status: server- and cluster-side resume, the unified keepalive, host-load-aware
> liveness (CPU + memory-pressure) with soft-drop, per-session backpressure
> isolation, OS-level workload isolation, the cluster data/control lane
> split, the app's client-side resume, and history-snapshot gap recovery are
> implemented. Resume and snapshot recovery are **unconditional** - Clusto does
> not support mixed-version clusters or old clients; the whole fleet upgrades
> together. Remaining
> work (continuous viewer rendering / adaptive frame-rate, and a dedicated
> control-plane runtime) is in [Roadmap](#roadmap).

## The model

Session output is a per-session byte stream published as `Output { session_id,
seq, data }` frames. `seq` is a per-session monotonic counter assigned exactly
once, where the chunk is first emitted (`publish_chunk`), and it travels
end-to-end - through the cluster `SessionOutput` forwarding and into the
client's `Output` events - so it is comparable everywhere. Clients deduplicate
by dropping output and resize frames whose positive `seq` they have already
applied. Concurrent resume requests for the same session share one pending
request.

Sequence assignment, recording, replay-ring insertion, and broadcast share one
ordered publication boundary. Concurrent PTY output and resize markers cannot
overtake each other and cause a valid earlier frame to be discarded as old.

Replayed frames retain their individual sequence boundaries even when live
output is batched for transport. Overlapping replay and live delivery therefore
cannot merge duplicate text into a newer frame that would evade deduplication.
PTY decoding also consumes malformed UTF-8 as replacement characters while
retaining incomplete characters across reads, so one invalid byte cannot hold
back all subsequent output.

### Replay ring (`OutputRing`)

Every live session keeps a bounded in-memory ring of its most recently published
frames, keyed by `seq`. It is fed by
`publish_chunk` - the single point where a `seq` is assigned - so PTY output,
SDK output, and injected control frames are all retained uniformly. The ring is
bounded by **both** a byte budget (`OUTPUT_RING_MAX_BYTES`, 4 MiB) and a frame
count (`OUTPUT_RING_MAX_FRAMES`, 8192); the oldest frames are evicted first, so the
retained seqs are always a contiguous suffix `oldest..=last`.

`OutputRing::replay_from(from_seq)` returns either:

* `Frames(..)` - the buffered frames with `seq > from_seq`, in order (possibly
  empty if the caller is already current), or
* `TooOld { last_seq }` - the gap predates the retained window and cannot be
  replayed.

### Resume-from-seq

`SubscribeOutput` carries an optional `from_seq`. When present:

1. The daemon subscribes the client to the **live** broadcast first (so no frame
   published during replay is missed).
2. It then replays the ring from `from_seq`. Any overlap between the replayed
   frames and the live stream is harmless - the client dedupes by `seq`.
3. If the gap is `TooOld`, the daemon sends `ResyncRequired { session_id,
   last_seq }` and stops forwarding that incomplete stream. The native app,
   browser, and CLI fetch an authoritative `GetSessionHistory` snapshot, apply
   all its chunks, then subscribe from the snapshot's final sequence boundary.
   The notice's `last_seq` is not treated as an applied cursor.
4. A terminal (PTY) resume whose gap is larger than 64 KiB gets the same
   `ResyncRequired` even though the ring still holds the frames: one screen
   snapshot costs far less than megabytes of output that already scrolled
   past, which matters most on a slow link. The resume that follows that
   snapshot is replayed whatever its size, so a terminal that prints faster
   than the snapshot round trip is not sent back for another one. Agent (SDK)
   sessions always get the full replay.

The common reconnect replays a small tail from the ring with no screen clear
and no full-history refetch; the snapshot path covers a long outage on a busy
terminal.

A `from_seq` resume **always re-arms delivery**, even when an output task is
already registered for the session on that connection. Plain attach
(`from_seq` absent) keeps the idempotent fast-path - if a task exists it is left
alone - but an explicit cursor is a recovery request and must not be a no-op.
The task being replaced is typically *alive but wedged*: the fan-in shed a frame
and marked a gap, then the PTY went idle so the gap never refilled, leaving
`last_sent_seq` frozen. Such a task is not `is_finished()`, so
`prune_finished_output_task` cannot reap it, and re-acking `Attached` without
re-arming strands the client - its staleness resume loops forever at a frozen
seq and only a full reconnect recovers. So when `from_seq` is present the daemon
first runs `reset_output_task_for_resume` (abort + drop the registered task and
its remote subscriber id), then the normal subscribe path replays from the
client's cursor and spawns a fresh forwarder. Lossless (`replay_from` yields only
`seq > from_seq`; the forwarder dedupes `seq <= last_sent_seq`) and timer-free.

The daemon's local resume path (`resume_or_attach_local`) handles this; the
cross-node path mirrors it (below).

### History collapse (VT-rendered scrollback)

A full `GetSessionHistory` fetch does **not** return the raw recorded byte
stream for a PTY session. Terminal apps that redraw in place - spinners, status
bars, Claude Code's status block - emit thousands of cursor-relative redraw
frames (`\r`, cursor-up, erase-line). Replayed verbatim into a fresh xterm those
relative moves no longer land where they did live (a cursor-up at the top of a
near-empty replay buffer clamps at row 0), so every historical frame stacks as a
fresh stripe of debris and the scrollback turns to garbage. Because the debris is
in the persisted byte stream, a hard reconnect replays the same mess at any
width.

The daemon records the complete raw PTY stream while the session is active;
recording no longer stops after 10000 chunks. Existing cleanup on session
termination, PTY exit, and daemon restart is unchanged. This does not turn
terminal history into a permanent archive or restore output discarded by an
older version.

A history fetch captures its live tail, sequence boundary, durable recording
position, and owner geometry together. It reads only the durable prefix before
that captured live tail, so output persisted during the fetch cannot appear
twice or move the snapshot beyond its advertised boundary.

The daemon feeds this complete, ordered byte stream through a headless
virtual terminal (`alacritty_terminal`, sized to the PTY's authoritative width)
and serializes the reconstructed grid - scrollback and visible screen - back to
one clean ANSI line per grid row. The scrollback budget
counts line advances and wrapped text, including multiline output delivered in
one chunk. It has a floor of 10000 rows and a safety ceiling of 200000
scrollback rows, plus the visible screen. Rows older than that ceiling are not
included in reconstructed history.
This collapses the redraw frames to the exact grid the live terminal showed, and
the clean snapshot is what lands in the client's scrollback, so it survives
reconnects. The rendering runs off the async runtime (VT parsing is CPU work)
and applies only to PTY sessions - SDK/JSON sessions and the capped
`limit = Some(n)` fast path are served raw.

Every history response chunk is correlated to its request and carries the
captured owner's geometry. The app assembles concurrent responses separately,
ignores superseded replies, and applies snapshots at their recorded dimensions.
It queues newer output behind the replacement and retries recovery if a busy
stream leaves a gap while the response is in flight. Full reload preserves all
rows returned by the daemon; scroll-up paging commits its cursor only after
the page is rendered. See [Terminal History & Full Reload](/terminal-history)
for the initial tail limits and recovery controls.

No capability negotiation. Because the whole fleet runs one revision, resume and
snapshot recovery are always available - the app preserves its terminals and
re-subscribes with `from_seq` on reconnect, and recovers a too-old gap
via an authoritative history snapshot. There is no `client_caps`/`server_caps` handshake and no
legacy wipe-and-refetch path.

### Resume repaint nudge

The VT-collapsed snapshot (and a cursor-resume's replayed ring frames) is a
**static grid**. An in-place TUI like Claude Code only repaints on `SIGWINCH`,
so if the reconnecting client lands on the same dimensions the PTY already has,
the kernel sends no `SIGWINCH` and the process never redraws over that static
snapshot - the user sees a subtly misaligned screen (box borders, cursor row)
until they manually resize. The manual resize "fixed" it only because it changed
the size and so delivered a `SIGWINCH`.

To close this automatically, a `from_seq` resume subscribe fires
`SessionManager::nudge_pty_repaint` once the fresh output task is armed. On Unix
this sends `SIGWINCH` directly to the foreground process group without changing
the PTY rows or columns. It replaces the older hidden one-row shrink/restore
cycle, which could emit cursor-addressed output at an intermediate geometry
that attached clients did not know about. The nudge is gated on resume
(`from_seq.is_some()`) so ordinary attach subscriptions do not send a PTY
signal, and it is a no-op for non-PTY sessions. For a peer-owned session the
nudge fires on the **owner** node, where the PTY lives, because the forwarded
subscribe runs `resume_or_attach_local` there.

### Ordered PTY resize boundaries

A resize is part of the terminal stream, not just session metadata. The owner
publishes a resize marker immediately before applying the new PTY geometry.
The WebSocket output drain converts that marker into
`PtyResizeBoundary { session_id, seq, cols, rows, request_id }` and flushes any
older accumulated output before sending it. It does not merge output across
the marker.

The app keeps xterm at the old geometry until this boundary arrives. During a
window or pane resize it only CSS-scales the old grid for presentation. A
boundary queues behind older text and waits until xterm has finished parsing
that text. Only then does xterm change rows and columns before parsing output
from the new geometry. This ordering is preserved for local output, peer-forwarded
output, ring replay, and same-size convergence boundaries.

Initial terminal geometry is estimated from the available viewport without an
80x24 floor, which avoids an immediate shrink on narrow or short screens.
Session-creation estimates intentionally ignore the keyboard because the dialog
closes before navigating to the terminal. A terminal-mount fallback instead
subtracts the native keyboard inset, or uses the visual viewport when no native
inset is available, because that keyboard may remain visible. This mount seed is
local and does not resize an existing remote PTY.

Keyboard transitions never enter the ordered resize flow. The native IME inset
drives one local presentation update per animation frame: the persisted
`resize` mode, labeled **Move above keyboard**, applies one vertical transform
to the complete natural-size xterm surface. The transform is anchored on the
cursor or last content row so the live region stays in the band above the
keyboard. It moves xterm's text, cursor, selection, and accessibility layers
together instead of scrolling the outer container independently, which could
leave Android WebView GPU layers at different offsets and produce duplicated or
overlapping rows. **Fit to screen** locally scales the grid, and **Scroll**
leaves it natural-size and top-aligned. No mode sends a daemon request or makes
the TUI repaint, which removes the keyboard-path roundtrip, blink, and lag.

The client does not automatically clear and replace terminal history after a
row reduction. Output before and after the ordered boundary remains one live
stream, so the running TUI's repaint and existing scrollback are preserved.
Persisted terminal previews are still treated as caches and validated against
daemon history when the current app run does not already have a continuously
subscribed live buffer. An explicit **Reload full history** remains available
when a viewer wants a fresh collapsed snapshot.

A pooled terminal that is detached from a pane keeps parsing its live stream
and applying ordered resize boundaries while off-screen, so reopening it only
repaints the renderer. It is never reset or rewritten from a snapshot unless its
own write queue overflowed while it was hidden. Reconnects resume every
subscribed stream, including background ones, from the client's seq cursor.

Each collapsed snapshot also carries the terminal state that was active at its
generation: the alternate screen of a running full-screen program, the DEC
input and mouse-reporting modes, scrolling region, saved cursor, a pending line
wrap, and the prefix of an escape sequence split by the snapshot boundary. The
client applies that state immediately after the snapshot rows and before
replaying output with a newer sequence number. This ordering matters because a
newer live frame may disable or change a mode, or complete the split sequence.

Owner resize RPCs are correlated by `request_id`, trailing-edge coalesced, and
serialized with at most one request in flight. Cross-node requests use
`ForwardedRequest` and return `PtyResizeResult`; there is no separate
fire-and-forget `PeerResizePty` path.

### Backpressure isolation

The browser terminal adds its own parser boundary after transport delivery.
It submits at most one xterm write at a time and waits for that write's
completion callback before submitting the next display-frame batch. This keeps
a bursty TUI from building an unbounded parser queue even when the WebSocket
and application event loop are still keeping up.

A client that can't keep up never loses the connection and never holes the
stream - and shedding is **queue-driven, never timer-driven**. The per-client
output forwarder `try_send`s each frame into the connection's bounded fan-in;
when the fan-in is full it **drops that single frame** and marks a gap, then
replays the gap losslessly from the ring (`fill_gap_from_ring`) once the
consumer drains - resuming exactly where delivery stopped if the fan-in fills
mid-replay (`GapFill::Partial`). When the gap predates the ring, it requests a
snapshot resync and stops forwarding until the client resumes from that
snapshot. It never clears the screen and sends an arbitrary remaining delta.
There is deliberately no wall-clock drain
budget: a timer that fires late on a starved runtime used to drop frames of
perfectly healthy clients and trigger resync storms exactly when the host was
weakest; a full queue means real consumer backpressure, and a starved runtime
now produces delay, never loss.

The select loop couples the fan-in to the socket: past `WRITER_HIGH_WATER`
frames queued in the per-connection writer it stops draining output, so a
backed-up socket pushes shedding to the fan-in - where ring-replay gap repair
lives - instead of the writer's silent drop-oldest backstop (which remains only
as the memory bound). Control frames bypass the gate. A slow consumer therefore
degrades only its own session's fidelity and self-heals; other sessions on the
same connection are untouched.

### Client writer: lanes, bounded frames and pacing

Everything a slow link has already accepted sits in front of the next keystroke
echo, and one WebSocket message cannot be interleaved once it has started. Each
client connection therefore has a writer that keeps three things small
(see [Binary frames](/protocol#binary-frames) for the wire layout):

* **Frame size.** A JSON reply over 16 KiB goes out as Chunk v1 pieces and a
  large terminal chunk as Output-Part v1 pieces, 2 KiB to 256 KiB each depending
  on the link, so a latency-sensitive frame waits behind one bounded piece.
* **Bytes in flight.** Bulk frames are followed by a ping marker. The client's
  pong says how much arrived, and the writer keeps the bulk bytes that are sent
  but not confirmed near what the link carries in one round trip plus about
  150 ms of queue. A client that never answers a ping is not paced.
* **A session's backlog.** Output a connection cannot take in about two seconds
  is dropped whole (between 128 KiB and 4 MiB per session) and the client gets
  one `ResyncRequired`, then a screen snapshot, instead of minutes of stale
  bytes. `Ctrl+C` on a flooding terminal therefore stops the old output within
  a second.

Lanes are served in order: priority (pings, pongs, the resync notice);
interactive (a session whose whole backlog is 4 KiB or less, such as an echo or
a prompt, which skips the pacing window inside a 32 KiB token bucket refilling
at 16 KiB/s); then bulk, round robin between the JSON control stream and each
session with a backlog, one bounded frame per turn. Typing stays responsive on a
slow link while another terminal floods. A resume whose gap is over 64 KiB is
answered with `ResyncRequired` too (see [Resume-from-seq](#resume-from-seq)).

Keystrokes arrive as Input v1 binary frames (17 bytes of header plus the bytes)
and are applied by the connection's input task, not the output path.

### Take-over result

`ClaimSession` is answered with the session's `SessionUpdated` (naming the
current holder) followed by an explicit `ClaimResult { granted, error }` as soon
as the session's owner decided, or with `error` set when the owner was
unreachable or the session is gone. The app reads the outcome from that message
instead of inferring it from a timer, so a refused claim is reported as refused
and a failed one as failed, with no silent revert. A 20 s timeout remains only
for a connection that dropped with the claim in flight. Taking over a terminal
on another device does not redraw it.

### Cross-node resume

When the session is owned by another node, the node the client is connected to
forwards `from_seq` to the owner via the peer `SubscribeOutput`. The owner
replays from **its** ring before streaming live, sending the missed frames back
as `SessionOutput`. If the owner's ring cannot satisfy the gap it returns
`SessionResyncRequired { subscriber_id, session_id, last_seq }`, which the
requesting node surfaces to its local client as `ResyncRequired` (reusing the
in-band control-frame path). The owner stops that subscriber's live forwarding
until recovery establishes a new subscription. The
`seq` is the owner's throughout, so client dedup works identically whether the
client is attached locally or across the mesh.

## Keepalive

Both directions of every Clusto WebSocket follow one shared policy, defined once
in the shared protocol so the daemon, the app, and the CLI cannot drift apart:

* **Ping interval** - 15 s.
* **Pong timeout** - 45 s (three missed pings). This is a **staleness
  threshold, not a teardown deadline**: liveness is observational (SSH
  semantics). No application timer on either side ever closes a connection.

A host under load produces lag and catch-up, exactly like a loaded SSH host.
What each side does with a widening pong gap:

* **Daemon** - logs one `connection stale` warning per episode (and a recovery
  line when pongs resume). The only teardown it gates is the **zombie reap**: a
  connection with *no attached sessions* that has also stopped answering pings
  is reaped after `ZOMBIE_REAP_TIMEOUT` (10 minutes), so claim bookkeeping isn't pinned
  forever by sockets whose kernel still ACKs but whose application is gone.
* **App (Tauri)** - after two missed pings (30 s) emits a connection-stale
  event driving a pulsing yellow "Connection stalled Ns - waiting..." indicator;
  a connection-fresh event clears it the instant traffic resumes. The socket is
  never closed from the timer.
* **Browser shim** - browsers auto-pong invisibly, so it sends an app-level
  `keepalive` message every 15 s instead; the daemon replies `KeepaliveAck`
  (guaranteed inbound traffic on an idle session) and the shim tracks the gap
  since any inbound message, emitting the same stale/fresh events after 30 s.

Actual connection death is the **kernel's job**: TCP keepalive (30 s idle,
10 s probes, 3 retries) reaps dead idle sockets, and `TCP_USER_TIMEOUT` (60 s,
set on the daemon listener) bounds the dead-peer-with-in-flight-data case that
keepalive cannot see. A slow-but-ACKing peer is held indefinitely. When the
kernel kills the socket, the client's reader sees a stream error - the single
authoritative disconnect signal - and reconnects with buffers and
resume cursors intact, replaying only the gap from the ring.

Liveness is intentionally **decoupled from output flow**: pings ride the control
lane, so a client that is merely slow to drain a high-throughput ("flooding")
session is never reported stale by mistake.

### Sidebar RTT

The native desktop and mobile apps show the latest connection round-trip time
(RTT) from the 15-second WebSocket heartbeat. Each ping carries a unique value;
only its matching pong produces a sample. Timing starts when the native writer
begins sending the ping and ends when the native reader handles its pong, using
a monotonic clock. Cluster-status requests and frontend event processing do not
contribute to the measurement.

RTT covers the app's current connection to its entry node, including the tunnel
when used. It can still include transport buffering and runtime scheduling, so
it is not equivalent to an ICMP `ping` result or the response time of a session
on another cluster node. This measurement does not change connection routing or
make a slow connection faster.

The sample clears on disconnect, reconnect, or a stalled connection and appears
again after a matching heartbeat reply. Unrelated, duplicate, and superseded
replies cannot overwrite it. The browser client omits RTT because browser APIs
do not expose WebSocket control-frame timing.

### Host-load-aware liveness

The pong gap is wall-clock, which misfires when the *daemon itself* is
starved: a build pegging every core can keep the daemon's runtime off-CPU for
tens of seconds, so it neither sends pings nor reads the pongs that did arrive,
and a perfectly healthy client would be reported stale (or zombie-reaped) with
a multi-hundred-second "gap". The daemon corrects this. A watchdog task sleeps
for a fixed tick and, on each wake, credits the larger of two stall signals:

* **Timer lateness** - how much later than the tick it actually ran. That
  lateness *is* the runtime's scheduling stall, since a CPU-saturated runtime
  delays the watchdog's own timer by exactly the amount every other task was
  starved.
* **Memory-pressure stall** - the per-tick delta of Linux PSI `some` memory
  `total` (`/proc/pressure/memory`). Page-fault stalls under swap thrash hit
  tasks unevenly: a WS read task can be frozen reclaiming pages while the
  watchdog, its working set resident, ticks on time - so timer lateness alone
  misses it. PSI captures the stall regardless of which task ate it. Linux-only;
  absent elsewhere, crediting falls back to timer lateness with no change. The
  two are combined with `max`, not sum, so a stall both signals see is credited
  once, not doubled.

The stall accumulates into a shared `StallTracker`, and the staleness check
(`CreditedDeadline`) subtracts any stall accrued since the last pong before
reporting a client stale or reaping a session-less zombie. A starved daemon
thus never mis-reports a live client, while a genuinely silent peer (no stall)
still crosses the threshold normally. The watchdog runs on the data-plane
runtime - the one whose CPU stall must be measured; the PSI signal, being
host-wide, applies regardless.

### Host-load-aware peer breaker

The same crediting protects the *cluster* link, not just client connections. The
peer transport sends `Awaitable`
messages (RPC, claims, resize) with an 8 s timeout and, after six consecutive
timeouts to one peer (~48 s), trips a circuit breaker that disconnects it so
subsequent sends fast-fail and a clean reconnect resyncs only what changed. But that 8 s
measures the *local* node's ability to hand a message to its own outbound task -
not the peer's responsiveness. A node thrashing in swap (a Pi pushed into memory
pressure by other services, say) cannot drain its own channels in 8 s, so every
send to every peer times out, the breaker trips against all of them at once, and
the node **partitions itself out of a perfectly healthy mesh**.

The daemon injects its `StallTracker` into the transport (`set_stall_signal`) so
the breaker credits host-load stall before disconnecting, mirroring the client
soft-drop: at the threshold it compares the host stall accrued across the streak
against the 5 s credit floor, and if the local node was off-CPU or paging for
that much of it, the timeouts say nothing about the peer - the connection is
**held**, the streak resets, and a genuinely dead peer is still reaped by the
per-peer receiver task and TCP keepalive. A healthy node with a truly
unresponsive peer still trips normally. Logged as `holding peer, not
partitioning self` (held) vs `disconnecting peer` (genuine trip).

### Nothing slow on the select loop

The per-connection select loop only reads frames, routes them, drains the
output fan-in, and ticks pings. Anything that can park - a peer RPC, a
replication retry, heavy local work - runs off-loop, so a slow or starved node
can never freeze a healthy client's heartbeat and output streaming because some
*other* node is slow. Routing, by message class:

* **io task** (ordered FIFO, dedicated `clusto-io` runtime) -
  `SendInput`/`SendPtyInput`/`ResizePty`, plus `ClaimSession` so a claim
  immediately followed by typing gates in order.
* **stream task** (ordered FIFO) - `Attach`/`Detach`/`SubscribeOutput`/
  `UnsubscribeOutput`/`OpenSession`/`CloseSession`. Their registries live in a
  shared per-connection `ConnStreams` (short lock scopes; the FIFO itself
  serializes the idempotency checks).
* **Per-request detached tasks** - `KillSession`, `RenameSession`,
  `TagSession`, `SaveSessionFile`, `GetSessionHistory` (its drain fence stays
  on-loop ahead of the spawn), `ExportSession`, and a node-targeted
  `NewSession` (forwards up to `NODE_FORWARD_TIMEOUT`, 35 s).
* **request worker** (ordered FIFO) - every other request-shaped message
  (projects, teams, notes, git, worktrees, files, node-targeted diagnostics,
  `TeamWait`), preserving the old inline ordering semantics through a cloned
  writer handle.

Responses ride back through the push channel or the shared writer queue; the
loop's only awaits are channel operations and non-blocking writer enqueues.

### Android background keepalive

On mobile, a silent client is usually the OS *freezing the app process* when it
is backgrounded - the WebSocket lives in the Tauri/Tokio layer, and a frozen
process can no longer answer pings (the daemon holds the connection regardless,
but session claims and live streaming work best on a responsive socket). To keep
the process responsive across normal backgrounding, the Android app runs a
`dataSync` **foreground service** (`ClustoKeepAliveService`). While it holds an ongoing notification,
Android keeps the process unfrozen, so the Tokio runtime keeps answering pings
and output keeps flowing.

The service is driven by the app's connection lifecycle: it starts (via the
`ClustoAndroid` JS bridge) whenever the app holds a daemon connection and stops on
disconnect. It is a no-op on desktop/iOS, where the bridge is absent.

This is **not** an unlimited keepalive. Android 14+ budgets `dataSync`
foreground-service runtime, and Doze still throttles the network during long
screen-off periods. So it reliably covers app-switching and short screen-off
gaps; an overnight-backgrounded phone still drops and recovers through the
ordinary foreground resume path (eager reconnect + resume-from-seq).

### App reconnect loop

After an unintended drop the app retries on a backoff of 0.5, 1, 2, 4, 8, 15,
30, then 60 s. While the app is on screen the delay is capped at 3 s, so a phone
whose first dials failed before Wi-Fi, cellular, or Tailscale came back does not
wait a minute for the next try. An attempt still dialing after 12 s on a visible
app is replaced by a fresh race that dials every node at once, without the pinned
node's head start: a dial started before the network was usable can otherwise sit
until the 45 s race deadline. Returning to the app, the browser `online` event,
and on Android a network change (Wi-Fi to cellular handover) each trigger an
immediate retry; if the socket still reports connected, a network change probes it
with a Ping first and only reconnects when no Pong arrives within 2 s. The same
signals, and every unintended drop, also reset the Clusto Tunnel dialer so the
next tunnel dial does not wait on relay state from the previous network.

While the app is reconnecting, a **Reconnect now** button appears next to the
connection status (sidebar, mobile top bar, and mobile menu), and the
reconnecting banner shows **Retry now**. Either one dials every node
immediately instead of waiting out the backoff delay. **Reconnect now** is also
in the `F1` command palette.

Browser connection attempts race the preferred node's available routes, then
start other nodes after a 500 ms head start. Each route has a 5 s deadline for
opening the WebSocket and completing authentication. A route that opens but
never authenticates cannot block healthy alternatives. The first authenticated
route wins; losing sockets, delayed attempts, and superseded connection attempts
are cancelled.

## Input latency under CPU load

A terminal should stay crisp even when the host is CPU-saturated, the way an SSH
session does. clustod already runs at `nice -10` (more favored than `sshd`), so the
cause of felt lag under load was never CPU *starvation* of the daemon - it was
**work-per-byte and in-process scheduling**. Three changes close the gap:

* **Dedicated input runtime (`clusto-io`).** The per-connection ordered I/O task
  that applies keystrokes / pty input / resize is spawned on a separate
  4-worker tokio runtime, alongside the 2-worker control-plane runtime. Input therefore never waits for a
  data-plane worker that is busy serializing an output flood. A keystroke's path
  to the PTY is independent of how much output is in flight.
* **Per-session PTY writer thread.** Each session owns a dedicated writer thread
  that drains an input channel and does one
  `write_all` + `flush` per batch - symmetric with the reader thread. Sending
  input is now a non-blocking channel send; a stalled child
  backs up only its own thread, never a shared tokio blocking-pool worker, so the
  old per-write `spawn_blocking` + 5 s timeout/leak is gone.
* **Binary frames.** Output is sent as a raw binary WS frame rather than JSON,
  removing the escape-dense per-byte encode cost on the session's busy owner
  node, and keystrokes travel the other way as Input v1 binary frames (about 25
  bytes instead of 99 as JSON) - see [Binary frames](/protocol#binary-frames).

## Cluster request bounding

Forwarded peer requests (`ForwardedRequest`) carry a `hops` budget
(`FORWARD_INITIAL_HOPS`), decremented at each re-forward, so a misrouted or
phantom-owner request cannot bounce around the mesh indefinitely. A node that
receives `hops == 0` handles the request terminally.

## Peer link lanes and frames

Each peer link is one ordered stream of WebSocket messages drained by one writer
task, which interleaves **five lanes** frame by frame, highest priority first:

| Lane | Carries |
|------|---------|
| Interactive | forwarded PTY input, resize announcements, requests someone awaits |
| Output | forwarded terminal output (droppable) |
| Transfer | replies larger than one frame (a history fetch, a file) |
| Replication | live replication of local writes and the periodic announces |
| Bulk | anti-entropy digests and chat content pulls |

No frame carries more than 64 kB: a larger message is cut into chunk frames and
rebuilt per lane on the other side, so a keystroke waits behind at most one
frame of bulk traffic (about half a second at 1 Mbit/s). The lane is chosen by
message role and delivery policy, not by size alone: a `Lossy` keepalive or
announce frame still rides the replication lane ahead of bulk, because starving
it behind an output flood is exactly what used to produce false peer-death
detection. The framing is in [Peer link framing](/protocol#peer-link-framing).

**Liveness.** The writer pings the peer every 5 s by itself, ahead of every lane,
so the ping depends on no application tick and no queue. Any frame received from
the peer counts as a sign of life (a pong, a chunk of a large message, output,
replication), not only a pong, so a link busy with a large transfer is not
reset. A peer silent for 30 s is treated as wedged; on a slow link the threshold
scales to 20 measured round trips, up to 120 s. The host-stall credit above
still applies.

**Reconnect.** After a link drops, the redial backoff restarts from its base only
if the link lived at least two minutes; a flapping link is redialed ever less
often. On reconnect the nodes exchange state digests rather than full state (see
[Clustering](/clustering#gossip-and-anti-entropy)), which is why a reconnect now
moves about 12 kB where it used to move about 15 MB.

## OS-level workload isolation

Userspace backpressure cannot keep a daemon responsive if a build saturates the
host scheduler or pushes it into swap thrash. Clusto therefore separates the
control plane from the work it launches.

On Linux with cgroup v2 and a delegated systemd unit, clustod:

* moves itself into a high-weight `daemon/` cgroup;
* creates a separate child cgroup for each PTY session and headless agent run
  below `workload/`;
* places the initial process into its workload cgroup before it executes, so
  wrappers such as `runuser` cannot briefly inherit daemon priority;
* gives workload groups low CPU and I/O weights, while reserving 256 MiB with
  `memory.min` and 512 MiB with `memory.low` for the daemon;
* sets the workload subtree's `memory.high` to 80% of physical RAM. This is a
  soft reclaim threshold, not a hard memory cap, so large builds can continue
  when the host has capacity;
* runs a guardian that moves any marked descendant which escaped its workload
  group back into the correct per-session cgroup.

The generated systemd unit sets `Nice=-10`, `CPUWeight=10000`,
`IOWeight=10000`, `MemoryMin=256M`, `MemoryLow=512M`,
`OOMScoreAdjust=-900`, and `Delegate=yes`. Inside the service the daemon leaf
carries a `cpu.weight` and `io.weight` of 10000 against the workload subtree's
1000, and workloads additionally run at `nice +2` on the normal `SCHED_OTHER`
scheduler with the default best-effort I/O priority. On a host without cgroup v2
delegation, Clusto still applies those per-process priorities and logs that cgroup
isolation is unavailable.

Sessions are interactive, so the daemon's edge is deliberately bounded. The
cgroup split is the real CPU reservation; `nice` only breaks ties. Workloads are
never placed on `SCHED_BATCH`, which drops the scheduler's wakeup-preemption
bonus and shows up as terminal input lag when a shell woken by a keystroke has
to wait behind a running CPU hog.

On Windows, clustod runs at `HIGH_PRIORITY_CLASS`. Each session and headless run
gets its own inherited Job Object at `NORMAL_PRIORITY_CLASS` with a low CPU
scheduling weight. Descendant tools and builds inherit the Job Object, keeping
the daemon responsive without placing unrelated host applications into Clusto's
resource policy.

Peer links run over Tailscale, so on a busy host its packets used to wait behind
builds long enough for peers to reset the link. At start, clustod raises
`tailscaled.service` to `CPUWeight=1000` and `IOWeight=1000` through the drop-in
`/etc/systemd/system/tailscaled.service.d/zz-clusto-priority.conf`. It only
writes the file and never restarts tailscaled. If a node still shows flapping
links, [Node Doctor](/knowledge#node-doctor) reports them in the **Peer links**
check.

Nodes installed before the updated service directives keep their old unit until
the next deploy or `sudo clusto install` rewrites it. On Linux, verify the unit
with:

```bash
systemctl show clustod -p CPUWeight -p IOWeight -p MemoryMin -p MemoryLow -p Nice
```

## Verifying under synthetic load

The pass/fail procedure for "host stress must produce lag, never breakage"
(`stress-ng` is an external tool on the test node, not a dependency):

1. On the daemon host: `stress-ng --cpu $(nproc) --vm 2 --vm-bytes 90% --timeout 300s`
   (add `--iomix 2` for IO pressure).
2. From a client with a real attached session, type into a shell and run a
   flood (`yes | head -c 10000000`), watching keystroke echo and output flow.
3. Pass criteria: **zero** client disconnects and zero daemon
   "dropping/reaping connection" log lines for connections with sessions; the
   staleness indicator may appear and must clear on its own; terminal content
   stays intact (ring replay, no full-screen wipe unless the gap outran the
   ring, at most one `ResyncRequired` per session).
4. Socket-death matrix: `kill -STOP` the client app (kernel still ACKs) - the
   daemon must hold the connection indefinitely; pull the network with output
   in flight - TCP_USER_TIMEOUT errors the socket within ~60 s; an idle
   session-less zombie is reaped only by the `ZOMBIE_REAP_TIMEOUT` gate.

## Roadmap

* **Continuous viewer rendering / adaptive frame-rate** - go beyond snapshot
  recovery: render *all* viewers from the authoritative grid
  (killing width-mismatch debris for good), and for a sustained-slow client
  switch its stream from raw bytes to periodic screen diffs (Mosh's adaptive
  frame rate) instead of relying on ring replay.
