# Assistant Layer The assistant is a **separate process** that turns transcriptions into actions and conversation. The daemon stays a dumb, reliable publisher; the assistant owns all state. ## Topology - `stt-daemon` → publish-only event socket (`$XDG_RUNTIME_DIR/stt-daemon.sock`, mode `0600`). It never accepts input. - `stt-assistant` → subscribes to that stream and serves a control socket (`$XDG_RUNTIME_DIR/stt-assistant.sock`, mode `0600`). - `stt-tui` / `stt-ctl` → clients of the assistant. ## Running ### Simplest: single command with supervisor ```bash uv run stt-daemon up --with-tui ``` This starts daemon, assistant, and TUI as child processes under a supervisor. When you quit the TUI, everything stops cleanly. Daemon and assistant output is discarded while the TUI owns the screen; nothing is written to disk. ### Traditional: three terminals ```bash uv run stt-daemon # transcription + event bus uv run stt-assistant # controller: modes, LLM, tools, audit, control socket uv run stt-tui # viewer/controller TUI (or `stt-tui --dump 10` headless) ``` Or start daemon + assistant once, then connect viewers as needed: ```bash uv run stt-daemon up # daemon + assistant uv run stt-tui # optional: viewer in another terminal uv run stt-tui --dump 5 # optional: headless event dump (5 seconds) ``` ## Requirements A local llama.cpp server exposing an OpenAI-compatible `/v1/chat/completions` with a tool-calling model (default endpoint `http://localhost:9931/v1/chat/completions`). Set `[llm].enabled = false` to run without an LLM (commands then report "no executor configured"). ## Modes | mode | what it does | |---|---| | `idle` | listening; nothing is sent to the model | | `command_armed` | the next complete turn becomes a command; the text accumulates live | | `executing` | a command is running through the tool loop | | `chat` | every completed turn is sent to the model; no tools | Command and chat modes are mutually exclusive. ## The two-press command flow 1. Arm (`stt-ctl command-start`, or `a` in the TUI). 2. Speak; the transcript fills the buffer. 3. Optionally edit with `e` (freezes the buffer, so ASR mistakes can be corrected before the model sees them). 4. Execute (`stt-ctl command-execute` or `Enter`), or cancel (`Esc`). Execution is gated on **readiness**: no speech open, the last finalized segment's `segment_done` has arrived, and STT is healthy. With `require_viewer = true`, execute is refused unless a TUI is attached. ## Control protocol `stt-ctl` verbs (one message each): `command-start`, `command-execute`, `command-cancel`, `command-edit --text "..."`, `chat-toggle`, `clear`, `status`, `speaker-list`, `speaker-name `. Clients attach by sending `subscribe` first (`viewer: true` for the TUI, which makes the assistant treat it as a watcher for the `require_viewer` gate). Speaker control adds the inbound types `{"type":"speaker_list"}` and `{"type":"speaker_name","speaker_id":"spk_0001","name":"Efren"}` (the `speaker-list` and `speaker-name ` verbs). `speaker_list` replies `{"type":"speaker_list","speakers":[{"id","name","status","cumulative_s","sample_path","created_at"}...]}`; `speaker_name` replies `{"type":"ack","command":"speaker_name","ok":true}`. Errors: `{"type":"error","code":"unknown_speaker"}` when the id is not in the assistant's cached roster, and `{"type":"error","code":"bad_message"}` for an empty name. Assistant → clients: `mode`, `command_text`, `transcript`, `speech_start`/`speech_stop`, `tool_call`, `tool_result`, `chat_message`, `speakers`, `cleared`, `heartbeat`, `error`, plus the direct replies `ack` (command accepted), `status`, and `speaker_list`. The `speakers` event carries `{"speakers":[...]}`. The daemon fires it on **mint / confirm / rename / prune only** (not on every adapt); the assistant caches that roster to answer `speaker-list` and validate `speaker-name`, and relays it to viewers. `transcript` events also carry `seg_id`, `speaker_id`, `speaker_status` (`matched|uncertain|unknown|none`), `speaker`, and `speaker_score`. The assistant resolves the display before relaying: the name when matched and named, otherwise the literal `"Unknown"`. The TUI renders it in the header between the language tag and duration (bold green when named, bold red otherwise). ## Audit Only **actions** are persisted, to `audit/YYYY-MM-DD.jsonl`: ```json {"ts":"...","mode":"command","text":"what time is it","edited":false, "tool":"get_datetime","args":{},"side_effect":"read_only","requires_confirmation":false, "confirmed":null,"ok":true,"result":"...","latency_ms":412} ``` Renaming a speaker writes its own record shape: ```json {"ts":"...","mode":"speaker","action":"name","speaker_id":"spk_0001","name":"Efren"} ``` Commands and chat history are volatile — held in memory and discarded. ## Conversation memory (compaction) Chat history lives in memory only. To avoid overflowing the model's context, the assistant **compacts** rather than deletes: when the prompt reaches `compaction_trigger_ratio` of the derived budget (`max_context_tokens - max_tokens - compaction_safety_margin`), the oldest turns are summarized by the same endpoint (using `compaction_system_prompt`, capped at `compaction_max_tokens`) and replaced by one rolling summary, while the most recent `compaction_keep_turns` stay verbatim. Large evictions are summarized in chunks and folded. Each transcript enters the chat context prefixed with its resolved speaker display: `": "`, or `"Unknown: "` when the speaker is unknown. The summary is injected as a second `system` message, so the prefix (`system` + summary) is **stable between compactions** — a compaction invalidates the server's prefix cache *once*, and subsequent turns are append-only and cache-friendly. If summarization fails (error, empty reply, or a truncated summary) the assistant falls back to deleting the oldest turns in blocks (`trim_block_turns`); compaction is never the reason a request exceeds the window. Each compaction emits a `chat_message` with `role: "note"` so the pane shows why the context changed — the note is UI-only and never sent to the model. A reply cut off by the output limit (`finish_reason: "length"`) surfaces as `error{code:"truncated"}` instead of disappearing silently; an empty truncated turn is not stored. ## Inference queue Every LLM call — chat turns, command executions, and the compaction summarizer — runs through a single worker thread, bounded by `inference_queue_size`. This serializes requests to the one llama.cpp server. Commands jump ahead of chat turns; when the queue is full an execute is refused with `error{code:"busy"}` and the oldest queued chat turn is dropped. Leaving chat mode discards queued chat turns. ## Triage An optional display-only panel that classifies recent speech. Off by default; enable `[triage]` and install the extra to use it (see [TRIAGE.md](TRIAGE.md)). The classifier runs **in-process** in the assistant on its own background thread (one worker plus a loader), never on the tick thread and never on the `InferenceQueue`. The Laya **multilingual** checkpoint is pinned for every call. This is the third single-worker lane: transcription (faster-whisper, not thread-safe), the LLM `InferenceQueue`, and the Laya triage lane. None of the three may be parallelized. On each settled turn in any mode (idle, chat, or command; `turn_ended()`), independent of chat, the lane classifies the last `window` transcribed utterances from the STT stream (not the chat history). Results flow through a **`signals -> content`** cascade: the signals stage always arrives first (per-category confidences), and when they escalate the assistant submits one `triage-extract` job to the existing LLM lane for a structured JSON extraction, which arrives as the content stage and is merged over the signals. The panel renders both; it is display-only and triggers no modes, commands, or tools. Leaving chat mode does not clear the panel or its buffer; an explicit `clear` does. ## Speakers Opt-in per-utterance speaker identification runs in the daemon (see [SPEAKERS.md](SPEAKERS.md)). The assistant caches the daemon's `speakers` roster to answer `speaker-list` and validate `speaker-name`, relays the event to viewers, and resolves each transcript's display before broadcasting it. `delete` and `merge` are not supported in v1; the documented path is a manual purge (stop the daemon, remove the id from `speakers/speakers.json` and `prototypes.npz`, then restart, or wipe `speakers/`). The optional LLM tool `set_speaker_name` is deferred, not implemented. ## TUI keys `a` arm · `Enter` execute · `Esc` cancel · `e` edit · `t` chat toggle · `c` clear · `q` quit `c` asks for confirmation, then wipes the conversation / command-output pane and resets the assistant's in-memory chat history (the LLM session), while staying in the current mode. Transcripts are not affected. **Live queue counters:** The pane borders show real-time queue depths. `transcripts [seg N]` displays the number of audio segments queued for transcription (the daemon's backlog). `conversation / command output [inf N]` shows the number of LLM inference jobs queued, with `· busy` appearing while one is running. These counters refresh about once per second, driven by the heartbeat cadence. **Selecting / copying text:** the TUI captures mouse events (Textual enables mouse reporting by default), so dragging does **not** select text the way it normally does in your terminal. Hold `Shift` while dragging to bypass the TUI and use your terminal's native selection and copy. ## Hotkeys See [`scripts/hotkeys/`](../scripts/hotkeys/README.md) for `sxhkd` and `xbindkeys` examples that bind those commands to global keys.