# Assistant Layer The assistant is a **separate process** that turns transcriptions into actions and conversation. The daemon stays a dumb, reliable publisher; the assistant owns all state. ## Topology - `stt-daemon` → publish-only event socket (`$XDG_RUNTIME_DIR/stt-daemon.sock`, mode `0600`). It never accepts input. - `stt-assistant` → subscribes to that stream and serves a control socket (`$XDG_RUNTIME_DIR/stt-assistant.sock`, mode `0600`). - `stt-tui` / `stt-ctl` → clients of the assistant. ## Running ### Simplest: single command with supervisor ```bash uv run stt-daemon up --with-tui ``` This starts daemon, assistant, and TUI as child processes under a supervisor. When you quit the TUI, everything stops cleanly. Daemon and assistant output is discarded while the TUI owns the screen; nothing is written to disk. ### Traditional: three terminals ```bash uv run stt-daemon # transcription + event bus uv run stt-assistant # controller: modes, LLM, tools, audit, control socket uv run stt-tui # viewer/controller TUI (or `stt-tui --dump 10` headless) ``` Or start daemon + assistant once, then connect viewers as needed: ```bash uv run stt-daemon up # daemon + assistant uv run stt-tui # optional: viewer in another terminal uv run stt-tui --dump 5 # optional: headless event dump (5 seconds) ``` ## Requirements A local llama.cpp server exposing an OpenAI-compatible `/v1/chat/completions` with a tool-calling model (default endpoint `http://localhost:9931/v1/chat/completions`). Set `[llm].enabled = false` to run without an LLM (commands then report "no executor configured"). ## Modes | mode | what it does | |---|---| | `idle` | listening; nothing is sent to the model | | `command_armed` | the next complete turn becomes a command; the text accumulates live | | `executing` | a command is running through the tool loop | | `chat` | every completed spoken turn is sent to the model; no tools | Command and chat modes are mutually exclusive. In chat mode, spoken transcripts can be captured automatically, but both `send transcripts` and `send to triage` default to **off**. The `send transcripts` checkbox (`{"type":"chat_capture","on":true}`) enables automatic capture of spoken turns; until it is on, only typed `chat-send` turns reach the model. Triage/brief still consume transcripts independently of this toggle. With the triage panel enabled, a second `send to triage` checkbox (`{"type":"triage_chat","on":true}`) mirrors chat turns into the triage window and brief. Because triage already hears spoken turns from the microphone, the toggle is content-aware: when `send transcripts` is on it adds only the assistant reply; when it is off (so chat turns are typed) it adds both your message and the assistant reply. The reply is always added, since it is independent of the transcriptions. ## The two-press command flow 1. Arm (`stt-ctl command-start`, or `a` in the TUI). 2. Speak; the transcript fills the buffer. 3. Optionally edit with `e` (freezes the buffer, so ASR mistakes can be corrected before the model sees them). 4. Execute (`stt-ctl command-execute` or `Enter`), or cancel (`Esc`). Executing clears the buffer immediately, before the command finishes. Execution is gated on **readiness**: no speech open, the last finalized segment's `segment_done` has arrived, and STT is healthy. With `require_viewer = true`, execute is refused unless a TUI is attached. ## Control protocol `stt-ctl` verbs (one message each): `command-start`, `command-execute`, `command-cancel`, `command-edit --text "..."`, `chat-toggle`, `chat-send --text "..."`, `clear`, `status`, `speaker-list`, `speaker-name `. Clients attach by sending `subscribe` first (`viewer: true` for the TUI, which makes the assistant treat it as a watcher for the `require_viewer` gate). Speaker control adds the inbound types `{"type":"speaker_list"}` and `{"type":"speaker_name","speaker_id":"spk_0001","name":"Efren"}` (the `speaker-list` and `speaker-name ` verbs). `speaker_list` replies `{"type":"speaker_list","speakers":[{"id","name","status","cumulative_s","sample_path","created_at"}...]}`; `speaker_name` replies `{"type":"ack","command":"speaker_name","ok":true}`. Errors: `{"type":"error","code":"unknown_speaker"}` when the id is not in the assistant's cached roster, and `{"type":"error","code":"bad_message"}` for an empty name. Assistant → clients: `mode`, `command_text`, `transcript`, `speech_start`/`speech_stop`, `tool_call`, `tool_result`, `chat_message`, `chat_capture`, `triage_chat`, `speakers`, `cleared`, `heartbeat`, `error`, plus the direct replies `ack` (command accepted), `status`, and `speaker_list`. `chat_capture` and `triage_chat` (inbound set, outbound broadcast) each carry `{"on": bool}`. `chat_capture` gates whether spoken transcripts are auto-fed into the chat session; `triage_chat` gates whether chat turns are mirrored into the triage window and brief (see above for the content rule). `status` echoes both as `chat_capture` and `triage_chat`. The `speakers` event carries `{"speakers":[...]}`. The daemon fires it on **mint / confirm / rename / prune only** (not on every adapt); the assistant caches that roster to answer `speaker-list` and validate `speaker-name`, and relays it to viewers. `transcript` events also carry `seg_id`, `speaker_id`, `speaker_status` (`matched|uncertain|unknown|none`), `speaker`, and `speaker_score`. The assistant resolves the display before relaying: the name when matched and named, otherwise the literal `"Unknown"`. The TUI renders it in the header between the language tag and duration (bold green when named, bold red otherwise). ## Audit Only **actions** are persisted, to `troubleshoot/audit/YYYY-MM-DD.jsonl`: ```json {"ts":"...","mode":"command","text":"what time is it","edited":false, "tool":"get_datetime","args":{},"side_effect":"read_only","requires_confirmation":false, "confirmed":null,"ok":true,"result":"...","latency_ms":412} ``` Renaming a speaker writes its own record shape: ```json {"ts":"...","mode":"speaker","action":"name","speaker_id":"spk_0001","name":"Efren"} ``` Commands and chat history are volatile — held in memory and discarded. ## Conversation memory (compaction) Chat history lives in memory only. To avoid overflowing the model's context, the assistant **compacts** rather than deletes: when the prompt reaches `compaction_trigger_ratio` of the derived budget (`max_context_tokens - max_tokens - compaction_safety_margin`), the oldest turns are summarized by the same endpoint (using `compaction_system_prompt`, capped at `compaction_max_tokens`) and replaced by one rolling summary, while the most recent `compaction_keep_turns` stay verbatim. Large evictions are summarized in chunks and folded. Each transcript enters the chat context prefixed with its resolved speaker display: `": "`, or `"Unknown: "` when the speaker is unknown. The summary is injected as a second `system` message, so the prefix (`system` + summary) is **stable between compactions** — a compaction invalidates the server's prefix cache *once*, and subsequent turns are append-only and cache-friendly. If summarization fails (error, empty reply, or a truncated summary) the assistant falls back to deleting the oldest turns in blocks (`trim_block_turns`); compaction is never the reason a request exceeds the window. Each compaction emits a `chat_message` with `role: "note"` so the pane shows why the context changed — the note is UI-only and never sent to the model. A reply cut off by the output limit (`finish_reason: "length"`) surfaces as `error{code:"truncated"}` instead of disappearing silently; an empty truncated turn is not stored. The chat engine keeps its own history and compacts it. The brief engine does **not**: it is stateless per call (a static system prompt plus one user turn carrying the code-owned fact ledger, the volatile narrative, and the new segment), so it has no history to compact, its prompt stays flat, and `[llm].brief_compaction_system_prompt` does not apply to it. Code owns the durable facts (see [TRIAGE.md](TRIAGE.md)). ## Inference queue Every LLM call — chat turns, command executions, the compaction summarizer, and triage brief updates — runs through a single worker thread, bounded by `inference_queue_size`. This serializes requests to the one llama.cpp server. Commands jump ahead of chat turns; when the queue is full an execute is refused with `error{code:"busy"}` and the oldest queued chat turn is dropped. An explicit `clear` discards queued chat turns; leaving and re-entering chat keeps the conversation, including anything still queued. Brief updates run on this same lane as non-evictable, single-outstanding jobs: a full queue refuses the brief rather than evicting a queued chat turn, and the refused turn's deltas stay buffered for the next attempt (see [TRIAGE.md](TRIAGE.md)). ## Triage An optional display-only panel that classifies recent speech and keeps a living plain-text brief. Off by default; enable `[triage]` and install the extra to use it (see [TRIAGE.md](TRIAGE.md)). The classifier runs **in-process** in the assistant on its own background thread (one worker plus a loader), never on the tick thread and never on the `InferenceQueue`. The Laya **multilingual** checkpoint is pinned for every call. This is the third single-worker lane: transcription (faster-whisper, not thread-safe), the LLM `InferenceQueue`, and the Laya triage lane. None of the three may be parallelized. On each settled turn in any mode (idle, chat, or command; `turn_ended()`), independent of chat, the lane classifies the last `window` transcribed utterances from the STT stream (not the chat history). Results flow through a `signals -> content` two-stage exchange: the signals stage always arrives first (per-category confidences), and, when `extract_enabled` is true and the fail-open gate on `substantive_update` passes, the assistant queues one `brief-update` job on the existing LLM lane. That job runs a stateless brief engine (`[llm].triage_system_prompt`) that sends one user turn carrying the code-owned fact ledger, the volatile narrative, and the pending speaker-labeled deltas plus the current signal words under a `Focus:` header; the model returns `NEW FACTS:` (only genuinely-new durable facts) and `BRIEF:` (the narrative), code appends facts to the ledger with conservative dedupe and renders the final brief, which arrives as the content stage and is rendered below the signals. On a Laya error it still runs with the deltas only. The job is token-guarded and single-outstanding: while one is in flight further turns are refused and their deltas stay buffered (bounded by `brief_pending_char_budget`), and a refusal, empty reply, or exception re-queues those deltas and clears the flag so the next turn retries. A successful update coalesces a flush when deltas remain. Code owns the durable facts, so a recorded fact can only disappear via a logged cap eviction (`ledger_max_facts`), never because the model dropped it. The deployment thresholds (`escalate_threshold`, `choice_min_confidence`) only drive the per-item `⚠` marker, not the gate and not the brief. The panel renders the last-triage clock (transcript time format/color) and both stages; it is display-only and triggers no modes, commands, or tools. Leaving chat mode does not clear the panel or its buffer; an explicit `clear` does (it also truncates the durable brief file). With `[brief].write = true` the latest brief is snapshotted to `//brief.md` inside the active conversation folder (gitignored, falling back to `brief/brief.md` when no session is active); with the default `write = false` it stays in the TUI only. A new conversation resets the brief, so each session folder gets a fresh one. With `dump_io = true` each triage also writes its I/O under `troubleshoot/triage/` (a coalesced update writes two files; see [TRIAGE.md](TRIAGE.md)). ## Speakers Opt-in per-utterance speaker identification runs in the daemon (see [SPEAKERS.md](SPEAKERS.md)). The assistant caches the daemon's `speakers` roster to answer `speaker-list` and validate `speaker-name`, relays the event to viewers, and resolves each transcript's display before broadcasting it. `delete` and `merge` are not supported in v1; the documented path is a manual purge (stop the daemon, remove the id from `speakers/speakers.json` and `prototypes.npz`, then restart, or wipe `speakers/`). The optional LLM tool `set_speaker_name` is deferred, not implemented. ## TUI keys `a` arm · `Enter` execute · `Esc` cancel · `e` edit · `t` cycle pane · `s` questions view · `c` clear · `q` quit `t` cycles the lower panes through command output, conversation, and hidden. Selecting the conversation turns chat mode on; cycling away turns it off. The conversation is kept across those transitions — only `c` discards it. Arming or executing a command (by key or typed input) leaves chat mode if it was on and reveals the command-output pane; this never happens from within chat itself, since a command always pre-empts it. Transcripts are not affected. The row below the transcript holds two inputs, split 50/50: the command buffer on the left and a **chat** box on the right. Typing a message into the chat box and pressing Enter turns chat mode on (if it was off) and shows the conversation pane, sends the message immediately (no turn-end wait), clears the box, and keeps it focused for the next message. Chat mode does not need to be entered first. To the right of the conversation pane sits a **send transcripts** checkbox, off by default: turning it on feeds spoken transcripts to the model while in chat (typed messages send regardless, and triage/brief are unaffected); it is clickable, or focusable with `Tab` outside the maximized view and toggled with Space. When the triage panel is enabled, a second **send to triage** checkbox mirrors chat turns into triage (assistant reply always, plus your message when `send transcripts` is off). Maximizing the conversation pane (`m`) hides the command buffer, keeps the chat box docked at the bottom of the screen, and docks the checkboxes beneath the pane; maximizing the command-output pane keeps the command buffer docked there instead. While maximized, `Tab` / `Shift+Tab` cycle focus between the pane and its docked input. `c` asks for confirmation, then wipes the conversation and command-output panes, hides whichever of them was showing, and resets the assistant's in-memory chat history (the LLM session), while staying in the current mode. Transcripts are not affected. `s` swaps the triage pane between the live signals view and a structured view of the loaded `questions.json` (one block per question with its id, type, instructions, and criteria); it is a no-op when `[triage].enabled = false`. See [TRIAGE.md](TRIAGE.md). **Live queue counters:** The pane borders show real-time queue depths. `transcripts [seg N]` displays the number of audio segments queued for transcription (the daemon's backlog). `conversation [inf N]` and `command output [inf N]` show the number of LLM inference jobs queued, with `· busy` appearing while one is running; command output (tool calls, results, and errors) lives in its own pane, separate from the conversation. The triage pane carries the same `[inf N]` depth and `· busy` marker, since brief updates share the LLM queue. These counters refresh about once per second, driven by the heartbeat cadence. **Selecting / copying text:** the TUI captures mouse events (Textual enables mouse reporting by default), so dragging does **not** select text the way it normally does in your terminal. Hold `Shift` while dragging to bypass the TUI and use your terminal's native selection and copy. ## Hotkeys See [`scripts/hotkeys/`](../scripts/hotkeys/README.md) for `sxhkd` and `xbindkeys` examples that bind those commands to global keys.