STT-Assistant/docs/ASSISTANT.md

Assistant Layer

The assistant is a separate process that turns transcriptions into actions and conversation. The daemon stays a dumb, reliable publisher; the assistant owns all state.

Topology

• stt-daemon → publish-only event socket ($XDG_RUNTIME_DIR/stt-daemon.sock, mode 0600). It never accepts input.
• stt-assistant → subscribes to that stream and serves a control socket ($XDG_RUNTIME_DIR/stt-assistant.sock, mode 0600).
• stt-tui / stt-ctl → clients of the assistant.

Running

Simplest: single command with supervisor

uv run stt-daemon up --with-tui

This starts daemon, assistant, and TUI as child processes under a supervisor. When you quit the TUI, everything stops cleanly. Daemon and assistant output is discarded while the TUI owns the screen; nothing is written to disk.

Traditional: three terminals

uv run stt-daemon        # transcription + event bus
uv run stt-assistant     # controller: modes, LLM, tools, audit, control socket
uv run stt-tui           # viewer/controller TUI  (or `stt-tui --dump 10` headless)

Or start daemon + assistant once, then connect viewers as needed:

uv run stt-daemon up             # daemon + assistant
uv run stt-tui                   # optional: viewer in another terminal
uv run stt-tui --dump 5          # optional: headless event dump (5 seconds)

Requirements

A local llama.cpp server exposing an OpenAI-compatible /v1/chat/completions with a tool-calling model (default endpoint http://localhost:9931/v1/chat/completions). Set [llm].enabled = false to run without an LLM (commands then report "no executor configured").

Modes

mode	what it does
idle	listening; nothing is sent to the model
command_armed	the next complete turn becomes a command; the text accumulates live
executing	a command is running through the tool loop
chat	every completed turn is sent to the model; no tools

Command and chat modes are mutually exclusive.

The two-press command flow

1. Arm (stt-ctl command-start, or a in the TUI).
2. Speak; the transcript fills the buffer.
3. Optionally edit with e (freezes the buffer, so ASR mistakes can be corrected before the model sees them).
4. Execute (stt-ctl command-execute or Enter), or cancel (Esc). Executing clears the buffer immediately, before the command finishes.

Execution is gated on readiness: no speech open, the last finalized segment's segment_done has arrived, and STT is healthy. With require_viewer = true, execute is refused unless a TUI is attached.

Control protocol

stt-ctl verbs (one message each): command-start, command-execute, command-cancel, command-edit --text "...", chat-toggle, chat-send --text "...", clear, status, speaker-list, speaker-name <id> <name>.

Clients attach by sending subscribe first (viewer: true for the TUI, which makes the assistant treat it as a watcher for the require_viewer gate).

Speaker control adds the inbound types {"type":"speaker_list"} and {"type":"speaker_name","speaker_id":"spk_0001","name":"Efren"} (the speaker-list and speaker-name <id> <name> verbs). speaker_list replies {"type":"speaker_list","speakers":[{"id","name","status","cumulative_s","sample_path","created_at"}...]}; speaker_name replies {"type":"ack","command":"speaker_name","ok":true}. Errors: {"type":"error","code":"unknown_speaker"} when the id is not in the assistant's cached roster, and {"type":"error","code":"bad_message"} for an empty name.

Assistant → clients: mode, command_text, transcript, speech_start/speech_stop, tool_call, tool_result, chat_message, speakers, cleared, heartbeat, error, plus the direct replies ack (command accepted), status, and speaker_list.

The speakers event carries {"speakers":[...]}. The daemon fires it on mint / confirm / rename / prune only (not on every adapt); the assistant caches that roster to answer speaker-list and validate speaker-name, and relays it to viewers.

transcript events also carry seg_id, speaker_id, speaker_status (matched|uncertain|unknown|none), speaker, and speaker_score. The assistant resolves the display before relaying: the name when matched and named, otherwise the literal "Unknown". The TUI renders it in the header between the language tag and duration (bold green when named, bold red otherwise).

Audit

Only actions are persisted, to audit/YYYY-MM-DD.jsonl:

{"ts":"...","mode":"command","text":"what time is it","edited":false,
 "tool":"get_datetime","args":{},"side_effect":"read_only","requires_confirmation":false,
 "confirmed":null,"ok":true,"result":"...","latency_ms":412}

Renaming a speaker writes its own record shape:

{"ts":"...","mode":"speaker","action":"name","speaker_id":"spk_0001","name":"Efren"}

Commands and chat history are volatile — held in memory and discarded.

Conversation memory (compaction)

Chat history lives in memory only. To avoid overflowing the model's context, the assistant compacts rather than deletes: when the prompt reaches compaction_trigger_ratio of the derived budget (max_context_tokens - max_tokens - compaction_safety_margin), the oldest turns are summarized by the same endpoint (using compaction_system_prompt, capped at compaction_max_tokens) and replaced by one rolling summary, while the most recent compaction_keep_turns stay verbatim. Large evictions are summarized in chunks and folded.

Each transcript enters the chat context prefixed with its resolved speaker display: "<Name>: <text>", or "Unknown: <text>" when the speaker is unknown.

The summary is injected as a second system message, so the prefix (system + summary) is stable between compactions — a compaction invalidates the server's prefix cache once, and subsequent turns are append-only and cache-friendly. If summarization fails (error, empty reply, or a truncated summary) the assistant falls back to deleting the oldest turns in blocks (trim_block_turns); compaction is never the reason a request exceeds the window. Each compaction emits a chat_message with role: "note" so the pane shows why the context changed — the note is UI-only and never sent to the model.

A reply cut off by the output limit (finish_reason: "length") surfaces as error{code:"truncated"} instead of disappearing silently; an empty truncated turn is not stored.

The chat engine keeps its own history and compacts it. The brief engine does not: it is stateless per call (a static system prompt plus one user turn carrying the code-owned fact ledger, the volatile narrative, and the new segment), so it has no history to compact, its prompt stays flat, and [llm].brief_compaction_system_prompt does not apply to it. Code owns the durable facts (see TRIAGE.md).

Inference queue

Every LLM call — chat turns, command executions, the compaction summarizer, and triage brief updates — runs through a single worker thread, bounded by inference_queue_size. This serializes requests to the one llama.cpp server. Commands jump ahead of chat turns; when the queue is full an execute is refused with error{code:"busy"} and the oldest queued chat turn is dropped. An explicit clear discards queued chat turns; leaving and re-entering chat keeps the conversation, including anything still queued.

Brief updates run on this same lane as non-evictable, single-outstanding jobs: a full queue refuses the brief rather than evicting a queued chat turn, and the refused turn's deltas stay buffered for the next attempt (see TRIAGE.md).

Triage

An optional display-only panel that classifies recent speech and keeps a living plain-text brief. Off by default; enable [triage] and install the extra to use it (see TRIAGE.md).

The classifier runs in-process in the assistant on its own background thread (one worker plus a loader), never on the tick thread and never on the InferenceQueue. The Laya multilingual checkpoint is pinned for every call. This is the third single-worker lane: transcription (faster-whisper, not thread-safe), the LLM InferenceQueue, and the Laya triage lane. None of the three may be parallelized.

On each settled turn in any mode (idle, chat, or command; turn_ended()), independent of chat, the lane classifies the last window transcribed utterances from the STT stream (not the chat history). Results flow through a signals -> content two-stage exchange: the signals stage always arrives first (per-category confidences), and, when extract_enabled is true and the fail-open gate on substantive_update passes, the assistant queues one brief-update job on the existing LLM lane. That job runs a stateless brief engine ([llm].triage_system_prompt) that sends one user turn carrying the code-owned fact ledger, the volatile narrative, and the pending speaker-labeled deltas plus the current signal words under a Focus: header; the model returns NEW FACTS: (only genuinely-new durable facts) and BRIEF: (the narrative), code appends facts to the ledger with conservative dedupe and renders the final brief, which arrives as the content stage and is rendered below the signals. On a Laya error it still runs with the deltas only. The job is token-guarded and single-outstanding: while one is in flight further turns are refused and their deltas stay buffered (bounded by brief_pending_char_budget), and a refusal, empty reply, or exception re-queues those deltas and clears the flag so the next turn retries. A successful update coalesces a flush when deltas remain. Code owns the durable facts, so a recorded fact can only disappear via a logged cap eviction (ledger_max_facts), never because the model dropped it. The deployment thresholds (escalate_threshold, choice_min_confidence) only drive the per-item ⚠ marker, not the gate and not the brief. The panel renders the last-triage clock (transcript time format/color) and both stages; it is display-only and triggers no modes, commands, or tools. Leaving chat mode does not clear the panel or its buffer; an explicit clear does (it also truncates the durable brief file). With [brief].write = true the latest brief is snapshotted to <session_dir>/<id>/brief.md inside the active conversation folder (gitignored, falling back to brief/brief.md when no session is active); with the default write = false it stays in the TUI only. A new conversation resets the brief, so each session folder gets a fresh one. With dump_io = true each triage also writes its I/O under triage/ (a coalesced update writes two files; see TRIAGE.md).

Speakers

Opt-in per-utterance speaker identification runs in the daemon (see SPEAKERS.md). The assistant caches the daemon's speakers roster to answer speaker-list and validate speaker-name, relays the event to viewers, and resolves each transcript's display before broadcasting it.

delete and merge are not supported in v1; the documented path is a manual purge (stop the daemon, remove the id from speakers/speakers.json and prototypes.npz, then restart, or wipe speakers/). The optional LLM tool set_speaker_name is deferred, not implemented.

TUI keys

a arm · Enter execute · Esc cancel · e edit · t cycle pane · s questions view · c clear · q quit

t cycles the lower panes through command output, conversation, and hidden. Selecting the conversation turns chat mode on; cycling away turns it off. The conversation is kept across those transitions — only c discards it. Arming or executing a command (by key or typed input) leaves chat mode if it was on and reveals the command-output pane; this never happens from within chat itself, since a command always pre-empts it. Transcripts are not affected.

The row below the transcript holds two inputs, split 50/50: the command buffer on the left and a chat box on the right. Typing a message into the chat box and pressing Enter turns chat mode on (if it was off) and shows the conversation pane, sends the message immediately (no turn-end wait), clears the box, and keeps it focused for the next message. Chat mode does not need to be entered first. Maximizing the conversation pane (m) hides the command buffer and keeps the chat box docked at the bottom of the screen; maximizing the command-output pane keeps the command buffer docked there instead.

c asks for confirmation, then wipes the conversation and command-output panes, hides whichever of them was showing, and resets the assistant's in-memory chat history (the LLM session), while staying in the current mode. Transcripts are not affected.

s swaps the triage pane between the live signals view and a structured view of the loaded questions.json (one block per question with its id, type, instructions, and criteria); it is a no-op when [triage].enabled = false. See TRIAGE.md.

Live queue counters: The pane borders show real-time queue depths. transcripts [seg N] displays the number of audio segments queued for transcription (the daemon's backlog). conversation [inf N] and command output [inf N] show the number of LLM inference jobs queued, with · busy appearing while one is running; command output (tool calls, results, and errors) lives in its own pane, separate from the conversation. The triage pane carries the same [inf N] depth and · busy marker, since brief updates share the LLM queue. These counters refresh about once per second, driven by the heartbeat cadence.

Selecting / copying text: the TUI captures mouse events (Textual enables mouse reporting by default), so dragging does not select text the way it normally does in your terminal. Hold Shift while dragging to bypass the TUI and use your terminal's native selection and copy.

Hotkeys

See scripts/hotkeys/ for sxhkd and xbindkeys examples that bind those commands to global keys.