STT-Assistant

STT Daemon

An always-on speech-to-text daemon that continuously listens to a PulseAudio/PipeWire audio source, detects voice activity with webrtcvad, and transcribes utterances to Spanish/English text using Faster-Whisper.

Overview

This daemon is designed to:

• Capture audio continuously from PulseAudio/PipeWire sources
• Preserve the start of speech using a pre-roll (ring) buffer, so the first phoneme is never cut when voice detection fires
• Segment utterances with configurable silence thresholds
• Transcribe each utterance with Faster-Whisper (supports multilingual, Spanish, English, or auto-detect)
• Output to stdout (human-readable) and/or JSONL files (for logging/downstream processing)
• Run non-blocking with worker threads so audio keeps flowing during transcription

Architecture

Audio Source (PulseAudio)
    ↓
ParecCapture / SoundDeviceCapture
    ↓
Pre-roll Ring Buffer (VAD lookback)
    ↓
WebRTC-VAD (voice activity detection)
    ↓
FrameSegmenter (utterance boundaries)
    ↓
Queue (segment_queue)
    ↓
Faster-Whisper (transcription)
    ↓
TranscriptSink (stdout + transcripts/YYYY-MM-DD.jsonl) + EventPublisher
    ↓  publish-only UNIX socket ($XDG_RUNTIME_DIR/stt-daemon.sock)
stt-assistant  (subscribes; owns modes, command buffer, chat, LLM, tools)
    ↓  control socket ($XDG_RUNTIME_DIR/stt-assistant.sock)
stt-tui (viewer) / stt-ctl (hotkeys)              audit/YYYY-MM-DD.jsonl ←┘

The daemon never accepts input: it only transcribes and publishes. Everything stateful lives in the assistant, which is a separate process (see Assistant Layer).

Prerequisites

System packages

• PulseAudio or PipeWire-Pulse: pactl, parec command-line tools

  • On Ubuntu/Debian: apt-get install pulseaudio-utils or apt-get install pipewire pipewire-pulse
  • On Fedora: dnf install pulseaudio-utils
  • On macOS: Install via Homebrew (though PulseAudio is Linux-first; sounddevice backend recommended on macOS)

• FFmpeg (optional, used by Faster-Whisper for format conversion)

  • On Ubuntu/Debian: apt-get install ffmpeg

Python

• Python 3.11+ (tested on 3.13.5)
• uv package manager (https://docs.astral.sh/uv/)

Installation

Clone the repo and install with uv:

# From the repo root (wherever you cloned it)
cd /path/to/STT-TTS

# Sync dependencies (installs into .venv)
uv sync

# (Optional) If you want to use the sounddevice backend (requires portaudio):
uv sync --extra portaudio

Configuration

Quick Start

Copy the example config and edit as needed:

cp config.example.toml config.toml
# Edit config.toml to adjust settings (see section below)

Configuration Reference

The config file is TOML. Here are all available options with defaults. config.example.toml ships values only; this section is the annotated reference.

[audio]
backend = "parec"                     # "parec" (PulseAudio by name) | "sounddevice"
device  = "VirtualMicSink.monitor"    # PulseAudio source name (parec) or device index/name (sounddevice)
sample_rate = 16000                   # Hz
channels    = 1                       # Mono only
frame_ms    = 30                      # webrtcvad requires 10, 20, or 30
gain        = 1.0                     # Amplitude scaling (1.0 = no change, >1 = louder)

[vad]
aggressiveness   = 2                  # 0–3: higher = fewer false positives (speech misses more)
pre_roll_ms      = 400                # Ring buffer duration: keep this many ms of pre-speech
speech_start_ms  = 90                 # Consecutive speech frames to confirm speech start
silence_ms       = 700                # Consecutive silence to end a turn
min_utterance_ms = 300                # Discard utterances with less actual speech than this
max_utterance_s  = 30.0               # Hard cap: force-finalize a very long utterance
chunk_pause_ms   = 350                # Once soft_max_s is reached, end a chunk at a pause this long
soft_max_s       = 10.0               # After this much speech, chunk-ends become eligible at pauses
rms_floor        = 0.008              # Drop segments quieter than this RMS (noise-triggered blips)
trailing_trim_ms = 200                # Keep only this much trailing silence (less hallucination-prone padding)

[whisper]
model                   = "small"               # "tiny"/"base"/"small"/"medium"/"large" (multilingual; do NOT use `.en` variants)
device                  = "cpu"                 # "cpu" or "cuda"
compute_type            = "int8"                # "int8" (CPU) or "float16" (CUDA, slower but more accurate)
language                = "auto"                # "auto" (detect), "en", "es", or None/""/"none"
beam_size               = 1                     # 1 = fast greedy, >1 = slower beam search
vad_filter              = false                 # Use Whisper's built-in VAD (we already have webrtcvad)
cpu_threads             = 0                     # CPU inference threads (0 = CTranslate2's automatic choice).
                                                # On many-core CPUs, set this to the physical core count
                                                # (e.g. 16 on a Ryzen 9 8945HX) - measurably faster than auto.
no_speech_prob_max      = 0.6                   # Drop segments Whisper scores as non-speech above this
avg_logprob_min         = -1.0                  # Drop segments with confidence below this
language_strategy       = "static"              # "static" (detect per utterance) | "sticky" (detect once, pin)
sticky_redetect_logprob = -0.8                  # Unpin and re-detect if segment avg_logprob falls below this.
                                                # Segments below avg_logprob_min (-1.0) are dropped; between
                                                # -1.0 and -0.8 they are accepted but trigger re-detection.
                                                # No accepted segments also clears the pin for re-detection.

[sink]                                # Virtual audio sink configuration
sink_name             = "VirtualMicSink"       # null-sink name
remap_source_name     = "VirtualMic"           # Microphone source name
loopback_latency_msec = 1                      # Module latency (ms)
monitor_output        = false                  # Loop sink.monitor → default sink (OFF = no feedback risk)

[output]
stdout     = true                     # Print transcripts to stdout
jsonl      = false                    # Persist transcripts to JSONL files (off by default for agent-only use)
jsonl_dir  = "transcripts"            # JSONL destination (one file per day); only used if jsonl = true
save_audio = false                    # Also write each transcribed utterance as a 16 kHz mono WAV
audio_dir  = "audio"                  # WAV destination (YYYY-MM-DD subfolders); only used if save_audio

[daemon]
queue_size      = 50                        # Max utterances queued for transcription (oldest dropped when full)
backlog_warn_at = 5                         # Warn when the queue backs up this far (worker falling behind)

[events]                                    # Publish-only event bus
socket       = ""                           # Default: $XDG_RUNTIME_DIR/stt-daemon.sock
heartbeat_ms = 1000                         # Heartbeat interval (progress-driven, from the capture loop)
ring_size    = 200                          # Recent events replayed to a newly attached subscriber

[assistant]                                 # Controller process
control_socket        = ""                  # Default: $XDG_RUNTIME_DIR/stt-assistant.sock
stall_timeout_ms      = 3000                # No heartbeat for this long -> STT considered offline
worker_stall_ms       = 15000               # Heartbeats arriving but a transcription runs this long -> degraded
command_arm_timeout_s = 10.0                # Disarm a command if no turn is captured in this time
chat_idle_timeout_s   = 300.0               # Leave chat mode after this much silence
turn_gap_s            = 1.5                 # Quiet gap that ends a turn
require_viewer        = true                # Refuse to execute commands with no TUI attached
chat_exit_phrases     = ["end conversation", "that's all", "stop listening"]

[audit]                                     # Executed actions only (commands/chats are volatile)
enabled = true                              # audit/YYYY-MM-DD.jsonl
dir     = "audit"

[llm]                                       # local llama.cpp (OpenAI-compatible)
enabled                  = true
endpoint                 = "http://localhost:9931/v1/chat/completions"
temperature              = 0.7
max_tokens               = 1024             # max output per request; reasoning models spend tokens before the answer, so keep generous
timeout_s                = 120.0
cache_prompt             = true             # reuse the KV prefix (append-only history keeps it warm)
max_context_tokens       = 131072           # the server's --ctx-size (llama.cpp `GET /props` -> n_ctx); the prompt budget derives from it
compaction_safety_margin = 256              # tokens reserved for the token estimate error + template/tool overhead
compaction_enabled       = true             # summarize old turns instead of deleting them
compaction_trigger_ratio = 0.8              # compact when the prompt reaches this fraction of the budget
compaction_keep_turns    = 6                # recent turns kept verbatim
compaction_max_tokens    = 512              # summary length cap
compaction_timeout_s     = 30.0             # summarizer timeout, so a compaction cannot hold the inference lane
compaction_input_tokens  = 8192             # summarizer input cap before chunk-and-fold
compaction_system_prompt = "You compress a conversation. Preserve names, numbers, decisions, user preferences, and unresolved questions. Omit pleasantries. Output only the summary."
inference_queue_size     = 8                # bounded single-lane inference queue
trim_block_turns         = 4                # fallback block trimming, not per-turn, to limit re-prefill
max_tool_iterations      = 3
system_prompt_chat       = "You are a concise voice assistant. Reply in 1-3 sentences."
system_prompt_command    = "You translate a spoken request into a single tool call when one applies."

max_prompt_tokens was removed: the prompt budget is derived as max_context_tokens - max_tokens - compaction_safety_margin, so the window and the trim budget can never disagree. A legacy max_prompt_tokens key is ignored with a warning.

Audio Source Setup

By default, the daemon listens to VirtualMicSink.monitor (the monitor of the virtual sink). Apps route audio into VirtualMicSink, and the daemon captures the mix.

Option 1: Use the CLI with interactive setup (recommended)

# Show devices and create a virtual sink (interactive picker)
uv run stt-daemon setup

# Check status
uv run stt-daemon list

# In another terminal, start the daemon
uv run stt-daemon run

# The daemon auto-uses the provisioned sink's monitor as the capture device
# (no need to manually specify --device)

# Route audio:
#   - In pavucontrol (GUI), drag apps' "Recording" side to "VirtualMicSink"
#   - Or use pactl: pactl move-sink-input <app-index> VirtualMicSink

# Stop when done
uv run stt-daemon clear --yes

Option 2: Use the provided shell wrapper

For convenience, scripts/audio-sink.sh is a thin wrapper around the Python CLI:

scripts/audio-sink.sh start   # → uv run stt-daemon setup
scripts/audio-sink.sh status  # → uv run stt-daemon list
scripts/audio-sink.sh stop    # → uv run stt-daemon clear --yes

Option 3: Manual PulseAudio setup

If you prefer manual control or want to understand what's happening:

# Get your default microphone source
default_source=$(pactl get-default-source)
echo $default_source  # e.g., alsa_input.usb-...

# Create null sink
pactl load-module module-null-sink sink_name=VirtualMicSink \
    sink_properties=device.description=Virtual-Mic-Sink

# Expose the sink's monitor as a microphone
pactl load-module module-remap-source \
    master=VirtualMicSink.monitor \
    source_name=VirtualMic \
    source_properties=device.description=Virtual-Microphone

# Route your microphone into the sink
pactl load-module module-loopback source="$default_source" sink=VirtualMicSink latency_msec=1

# (Optional) Monitor the sink output on your speakers
pactl load-module module-loopback source=VirtualMicSink.monitor latency_msec=1

# Now tell the daemon to use it
uv run stt-daemon --device VirtualMicSink.monitor

# Clean up (unload modules and remove the file):
pactl unload-module <sink_id>
pactl unload-module <source_id>
pactl unload-module <loopback_id_1>
pactl unload-module <loopback_id_2>

Option 3: Use a physical microphone directly

If you have a single physical mic and don't need the virtual routing:

# List available sources
pactl list short sources

# Start daemon with your source
uv run stt-daemon --device alsa_input.usb-C-Media_Electronics_Inc._USB_Audio_Device-00.mono-fallback

Option 4: Use sounddevice backend

If you prefer not to use PulseAudio (e.g., on macOS or Windows):

uv sync --extra portaudio
uv run stt-daemon --backend sounddevice --device 0  # device 0 = default input

Running the Daemon

Subcommands

The stt-daemon CLI supports these subcommands. Every subcommand also accepts the common overrides --config, --device, --backend, --model, --language, and -v/--verbose (for example stt-daemon setup --device ...).

setup — Create virtual sink and loopbacks

Interactive setup (shows device picker):

uv run stt-daemon setup

Non-interactive setup (specify sources directly):

uv run stt-daemon setup --sources "alsa_input.usb-Foo,alsa_output.pci-0000.monitor"

This creates:

• A null sink (by default VirtualMicSink)
• A remap-source to expose the sink's monitor as a microphone (by default VirtualMic)
• Loopback modules routing each selected source into the sink
• A state file tracking all module IDs for later cleanup

run — Start the daemon (auto-provisions if needed)

uv run stt-daemon run

If no saved provisioning exists and the sink is missing, the daemon will:

1. Show the interactive device picker
2. Create the sink and loopbacks (if sources are selected)
3. Start listening and transcribing

You can also override settings:

uv run stt-daemon run --device "alsa_input.usb-..." --language es --model base
uv run stt-daemon run -v  # Verbose logging
uv run stt-daemon run --check  # Test audio capture + model, then exit

list — Show devices and provisioning status

uv run stt-daemon list

Output shows:

• Input devices (real microphones)
• Output devices (speakers/headphones; captured via their .monitor source)
• Current provisioning state (if a sink has been created)

clear — Remove virtual sink and loopbacks

uv run stt-daemon clear
# or without confirmation prompt:
uv run stt-daemon clear --yes

This unloads all tracked modules and removes the state file. If audio/audio.json is already gone (reboot, manual delete, crash), clear --yes still finds orphaned modules by name (VirtualMicSink/VirtualMic via pactl list short modules) and unloads them in reverse order (loopbacks → remap-source → null-sink), so the command is idempotent — safe to re-run and to recover from Sink 'VirtualMicSink' already exists but is not tracked.

up — Launch daemon, assistant, and optionally TUI

uv run stt-daemon up              # Start daemon + assistant (separate processes)
uv run stt-daemon up --with-tui   # Also launch the interactive TUI viewer

This subcommand runs the entire system as a supervised set of child processes:

• daemon — transcription engine (listens to audio, publishes transcriptions)
• assistant — controller (subscribes to transcriptions, serves the control socket)
• TUI (optional) — viewer/controller UI (connects to the assistant via control socket)

The supervisor manages their lifetimes: if any child exits or Ctrl+C is pressed, all remaining children are cleanly shut down. Without --with-tui, you can still run a separate uv run stt-tui in another terminal to view and control the assistant.

With --with-tui, the daemon and assistant output is discarded so it cannot corrupt the screen, while the TUI keeps the terminal. Without --with-tui, all output goes to the terminal. The supervisor writes nothing to disk.

Like all subcommands, up also accepts --config, --device, --backend, --model, --language, and -v (verbose) flags.

Example workflow:

# One-command startup with TUI (everything stops when you quit the TUI)
uv run stt-daemon up --with-tui

# Or, start daemon + assistant only
uv run stt-daemon up

# Then in another terminal, launch the TUI whenever you want
uv run stt-tui

(Legacy) --list-sources flag

uv run stt-daemon --list-sources
# Equivalent to: uv run stt-daemon list

(Legacy) --check flag

uv run stt-daemon --check
# Equivalent to: uv run stt-daemon run --check

Interactive Device Picker

When you run setup or run without --sources, the daemon shows an interactive picker:

• TTY mode (terminal): Uses questionary to show a checkbox UI. Pre-checked items are from the previously saved selection.
• Non-TTY mode (piped input): Shows a numbered menu; enter space/comma-separated indices, all, none, or blank (to keep previous selection).

Device labels:

• [IN] <description> (<name>) — Input device (real microphone)
• [OUT] <description> (<name>) — Output device (monitor of a sink)

Running the daemon

The simplest workflow:

# Create/setup the sink (interactive)
uv run stt-daemon setup

# Start the daemon in another terminal
# (auto-uses the provisioned sink)
uv run stt-daemon run

# Stop and remove the sink when done
uv run stt-daemon clear --yes

Output examples

Stdout (human-readable):

[14:23:45] (es) Hola, cómo estás?  [rms=0.041 end_of_turn]
[14:23:52] (en) What time is it?  [rms=0.038 end_of_turn]
[14:24:10] (es) Gracias por tu ayuda.  [rms=0.052 pause]

JSONL (one line per utterance; only written when jsonl = true):

{"ts": "2025-09-20T14:23:45.123456", "lang": "es", "lang_prob": 0.95, "duration_s": 2.5, "rms": 0.04123, "end_reason": "end_of_turn", "text": "Hola, cómo estás?", "transcribe_s": 0.42}
{"ts": "2025-09-20T14:23:52.654321", "lang": "en", "lang_prob": 0.98, "duration_s": 1.8, "rms": 0.03811, "end_reason": "pause", "text": "What time is it?", "transcribe_s": 0.31}

duration_s is the length of the utterance audio; transcribe_s is the wall-clock time the worker spent transcribing it (inference latency), rounded to milliseconds. Their ratio is the real-time factor. The field is omitted if the timing was not measured (e.g. emit() called directly without transcribe_s).

end_reason is one of end_of_turn (full silence_ms elapsed), pause (chunked at chunk_pause_ms because it passed soft_max_s), hard_cap (max_utterance_s reached), or flush (shutdown mid-utterance).

Audio files — when save_audio = true, each transcribed utterance is also written as a 16 kHz mono WAV under audio_dir/YYYY-MM-DD/, and the record gains an "audio" field with its path:

{..., "end_reason": "pause", "text": "Hola", "audio": "audio/2025-09-20/142345-123-pause.wav"}

Saving is best-effort: if a write fails (e.g. disk full), the transcript is still emitted and a warning is logged.

⚠️ Disk space — saved audio is never rotated or cleaned up automatically. At 16 kHz mono 16-bit it costs 32 KB/s, i.e. ~1.9 MB/min, ~115 MB/hour, ~2.8 GB/day of continuous speech. Silence is not written, so real usage is lower — but an always-on daemon in a talkative room still accumulates quickly. Point audio_dir at a volume with headroom and prune it yourself (e.g. a cron job removing files older than N days), and leave save_audio = false unless you actually need the clips for auditing.

Assistant Layer

The assistant is a separate process that turns transcriptions into actions and conversation. The daemon stays a dumb, reliable publisher; the assistant owns all state.

Topology

• stt-daemon → publish-only event socket ($XDG_RUNTIME_DIR/stt-daemon.sock, mode 0600). It never accepts input.
• stt-assistant → subscribes to that stream and serves a control socket ($XDG_RUNTIME_DIR/stt-assistant.sock, mode 0600).
• stt-tui / stt-ctl → clients of the assistant.

Running

Simplest: single command with supervisor

uv run stt-daemon up --with-tui

This starts daemon, assistant, and TUI as child processes under a supervisor. When you quit the TUI, everything stops cleanly. Daemon and assistant output is discarded while the TUI owns the screen; nothing is written to disk.

Traditional: three terminals

uv run stt-daemon        # transcription + event bus
uv run stt-assistant     # controller: modes, LLM, tools, audit, control socket
uv run stt-tui           # viewer/controller TUI  (or `stt-tui --dump 10` headless)

Or start daemon + assistant once, then connect viewers as needed:

uv run stt-daemon up             # daemon + assistant
uv run stt-tui                   # optional: viewer in another terminal
uv run stt-tui --dump 5          # optional: headless event dump (5 seconds)

Requirements

A local llama.cpp server exposing an OpenAI-compatible /v1/chat/completions with a tool-calling model (default endpoint http://localhost:9931/v1/chat/completions). Set [llm].enabled = false to run without an LLM (commands then report "no executor configured").

Modes

mode	what it does
idle	listening; nothing is sent to the model
command_armed	the next complete turn becomes a command; the text accumulates live
executing	a command is running through the tool loop
chat	every completed turn is sent to the model; no tools

Command and chat modes are mutually exclusive.

The two-press command flow

1. Arm (stt-ctl command-start, or a in the TUI).
2. Speak; the transcript fills the buffer.
3. Optionally edit with e (freezes the buffer, so ASR mistakes can be corrected before the model sees them).
4. Execute (stt-ctl command-execute or Enter), or cancel (Esc).

Execution is gated on readiness: no speech open, the last finalized segment's segment_done has arrived, and STT is healthy. With require_viewer = true, execute is refused unless a TUI is attached.

Control protocol

stt-ctl verbs (one message each): command-start, command-execute, command-cancel, command-edit --text "...", chat-toggle, clear, status.

Clients attach by sending subscribe first (viewer: true for the TUI, which makes the assistant treat it as a watcher for the require_viewer gate).

Assistant → clients: mode, command_text, transcript, speech_start/speech_stop, tool_call, tool_result, chat_message, cleared, heartbeat, error, plus the direct replies ack (command accepted) and status.

Audit

Only actions are persisted, to audit/YYYY-MM-DD.jsonl:

{"ts":"...","mode":"command","text":"what time is it","edited":false,
 "tool":"get_datetime","args":{},"side_effect":"read_only","requires_confirmation":false,
 "confirmed":null,"ok":true,"result":"...","latency_ms":412}

Commands and chat history are volatile — held in memory and discarded.

Conversation memory (compaction)

Chat history lives in memory only. To avoid overflowing the model's context, the assistant compacts rather than deletes: when the prompt reaches compaction_trigger_ratio of the derived budget (max_context_tokens - max_tokens - compaction_safety_margin), the oldest turns are summarized by the same endpoint (using compaction_system_prompt, capped at compaction_max_tokens) and replaced by one rolling summary, while the most recent compaction_keep_turns stay verbatim. Large evictions are summarized in chunks and folded.

The summary is injected as a second system message, so the prefix (system + summary) is stable between compactions — a compaction invalidates the server's prefix cache once, and subsequent turns are append-only and cache-friendly. If summarization fails (error, empty reply, or a truncated summary) the assistant falls back to deleting the oldest turns in blocks (trim_block_turns); compaction is never the reason a request exceeds the window. Each compaction emits a chat_message with role: "note" so the pane shows why the context changed — the note is UI-only and never sent to the model.

A reply cut off by the output limit (finish_reason: "length") surfaces as error{code:"truncated"} instead of disappearing silently; an empty truncated turn is not stored.

Inference queue

Every LLM call — chat turns, command executions, and the compaction summarizer — runs through a single worker thread, bounded by inference_queue_size. This serializes requests to the one llama.cpp server. Commands jump ahead of chat turns; when the queue is full an execute is refused with error{code:"busy"} and the oldest queued chat turn is dropped. Leaving chat mode discards queued chat turns.

TUI keys

a arm · Enter execute · Esc cancel · e edit · t chat toggle · c clear · q quit

c asks for confirmation, then wipes the conversation / command-output pane and resets the assistant's in-memory chat history (the LLM session), while staying in the current mode. Transcripts are not affected.

Live queue counters: The pane borders show real-time queue depths. transcripts [seg N] displays the number of audio segments queued for transcription (the daemon's backlog). conversation / command output [inf N] shows the number of LLM inference jobs queued, with · busy appearing while one is running. These counters refresh about once per second, driven by the heartbeat cadence.

Selecting / copying text: the TUI captures mouse events (Textual enables mouse reporting by default), so dragging does not select text the way it normally does in your terminal. Hold Shift while dragging to bypass the TUI and use your terminal's native selection and copy.

Hotkeys

See scripts/hotkeys/ for sxhkd and xbindkeys examples that bind those commands to global keys.

Virtual Sink Provisioning: How It Works

The daemon uses PulseAudio's module system to create a virtual microphone from multiple sources:

What Gets Created

1. Null Sink (module-null-sink, e.g., VirtualMicSink)

   • A sink that doesn't play anywhere; apps route audio into it instead.
     

2. Remap Source (module-remap-source, e.g., VirtualMic)

   • Exposes the null-sink's monitor as a microphone so the daemon can capture from it.
     

3. Loopback Modules (module-loopback, one per selected source)

   • Routes each selected input (microphone, speaker monitor, etc.) into the null-sink.

State Persistence

When you run uv run stt-daemon setup, the daemon:

• Creates all modules and records their numeric IDs
• Saves the state to audio/audio.json (or $STT_DAEMON_STATE)
• Allows run to re-provision if the sink is missing (e.g., after reboot)
• Allows clear to unload the modules it created — and falls back to name-based discovery (sink_name/remap_source_name in pactl list short modules) when audio.json is missing, so orphaned VirtualMicSink/VirtualMic sinks are still cleaned up (e.g., after a reboot or manual state-file delete)

The state file is JSON and can be edited manually if needed, but the daemon handles it automatically.

Feedback Risk

By default, monitor_output = false, so the sink's output is not looped back to the default speaker. If you enable it, speakers will play back everything captured in the sink—including your own voice—causing audio feedback. Keep it OFF unless you specifically want monitoring.

Pre-roll Buffer: Why It Matters

The pre-roll buffer is a ring buffer that keeps the last N milliseconds (default 400ms) of audio BEFORE speech is detected. When the VAD confirms speech has started, those pre-roll frames are prepended to the utterance.

Without pre-roll: The first phoneme of "hello" might be cut off—you'd get "llo" or "elo" because the VAD fires AFTER the first sound is already gone.

With pre-roll: The entire "hello" is captured, including the initial "h" and "e", because the ring buffer preserved them.

How it works (under the hood)

1. Silence phase: Frames are pushed into a ring buffer (maxlen = pre_roll_ms / frame_ms). Old frames fall off.
2. Speech detected: VAD returns True for consecutive speech_start_ms worth of frames.
3. Speech confirmed: _begin() is called. The entire ring buffer contents are copied into the segment, then regular frames append to it.
4. End of turn: after silence_ms of silence the segment is finalized and transcribed.

Long speech and silence

Two extra rules keep long monologues from becoming one huge blob, and keep noise from being transcribed:

• Pause-aligned chunking: once a segment reaches soft_max_s (10s) of speech, a shorter chunk_pause_ms (350ms) pause ends a chunk. The cut lands on a natural phrase boundary, so no word is split and no overlap/duplication occurs. max_utterance_s (30s) remains the last-resort cap for someone who never pauses.
• Noise suppression: min_utterance_ms drops blips, rms_floor drops segments too quiet to be speech, trailing_trim_ms strips hallucination-prone trailing silence before transcription, and Whisper segments below no_speech_prob_max / avg_logprob_min are discarded.

Troubleshooting

"Device not found"

Error: Device 'VirtualMicSink.monitor' not found. Available sources: [...]
To create VirtualMicSink, run: scripts/audio-sink.sh start

Fix:

scripts/audio-sink.sh start

Then route audio into VirtualMicSink and try again.

"Can't select or copy text in the TUI"

Cause: Textual enables terminal mouse reporting, so mouse events are consumed by the app instead of the terminal, and native drag-selection never starts.

Fix: hold Shift while dragging to force your terminal's native selection, then copy as usual. (Alternatively, launching the app with mouse support disabled — app.run(mouse=False) — restores plain selection but loses in-app scrolling/clicks.)

High latency / slow transcription

By default, the daemon uses compute_type = "int8" and device = "cpu", which trades speed for accuracy. To speed up:

1. Pin cpu_threads to the physical core count: cpu_threads = 16 in [whisper] (CTranslate2's automatic choice is often slower than an explicit count on many-core CPUs)
2. Reduce model size: --model base or --model tiny
3. Enable GPU (if you have CUDA): --device cuda with compute_type float16
4. Reduce beam_size: Edit config to beam_size = 1 (already default, very fast)

Example:

uv run stt-daemon --model base --device cuda

"Transcription backlog reached N of M"

This warning means the single transcription worker is falling behind: utterances are arriving faster than they are being transcribed, and once the queue is full the oldest ones get dropped. The worker is serial by design (faster-whisper's model isn't thread-safe), so fix it by making transcription faster rather than by queueing more:

1. Smaller/faster model: --model base or tiny
2. GPU: --device cuda with compute_type float16
3. Raise backlog_warn_at or queue_size in [daemon] only to make the warning/dropping less likely — it hides the symptom, not the cause.

Spanish / English / Other languages

The Faster-Whisper model is multilingual by default. It auto-detects language or can be forced:

uv run stt-daemon --language es   # Force Spanish
uv run stt-daemon --language en   # Force English
uv run stt-daemon --language auto # Auto-detect (default)

The output includes the detected language tag: [14:23:45] (es) ....

Recognizing Spanish vs. English better

• Force the language: Use --language es or --language en
• Use a larger model: --model medium or --model large (slower, more accurate)
• Tune VAD aggressiveness: Lower values (0–1) capture more, higher values (2–3) are stricter

Audio device not responding / parec hangs

If parec seems stuck:

1. Check if the source exists: pactl list short sources
2. Kill the daemon (Ctrl+C)
3. Verify PulseAudio is running: systemctl status pulseaudio or check journalctl
4. Restart PulseAudio if needed: systemctl restart pulseaudio
5. Re-run the daemon

Sounddevice backend issues

If using --backend sounddevice and getting import errors:

uv sync --extra portaudio

Then verify available devices:

python -c "import sounddevice as sd; print(sd.query_devices())"

Use the device index or name as --device 0 or --device "Microphone (...).

No transcriptions appearing

1. Check VAD is firing: uv run stt-daemon -v (verbose logging)
2. Verify audio is loud enough: uv run stt-daemon --check (check RMS)
3. Reduce aggressiveness (0–1 instead of 2–3) to detect quieter speech
4. Check min_utterance_ms isn't too high (default 300ms should be fine)

Whisper model not loading

Error loading model: ...

The model is downloaded on first run (~1.5 GB for small). Check:

1. Disk space: df -h (needs ~2 GB free)
2. Internet: Model is downloaded from Hugging Face
3. HF cache location: Models go to ~/.cache/huggingface/hub/ by default

You can override the cache:

export HF_HOME=/path/to/cache
uv run stt-daemon --model small

Testing

Run the test suite (no audio hardware required):

uv run pytest -v

Tests cover:

• Pre-roll buffer preservation: confirm the first frames are not cut
• Silence finalization: utterances end after N silence frames
• Min/max utterance duration: blips are discarded, long utterances are finalized
• Pause-aligned chunking: long speech splits at pauses (end_reason = "pause")
• Config loading and language normalization
• Provisioning: setup/ensure/teardown, orphan-name cleanup, and clear idempotency
• Event bus: envelope codec, ring replay, subscriber fan-out
• Assistant: mode machine, command readiness/timeouts, tool schema validation, LLM client/history trimming, audit log
• TUI: pure view-state reducer and control-socket client

Development

Code structure

src/stt_daemon/
├── __init__.py          - Package metadata
├── __main__.py          - Entry point for `python -m stt_daemon`
├── cli.py               - Argument parsing and entry point (`stt-daemon`)
├── ctl.py               - One-shot control client (`stt-ctl`, hotkeys)
├── config.py            - Config loading (TOML)
├── audio.py             - Audio capture (parec, sounddevice)
├── pulse.py             - PulseAudio/pactl wrapper (devices, modules, sink checks)
├── provisioner.py       - Virtual sink + loopback lifecycle (setup/ensure/teardown)
├── state.py             - Provisioning state file (audio/audio.json)
├── picker.py            - Interactive device picker (questionary / numbered menu)
├── segmenter.py         - VAD + pre-roll buffer + utterance segmentation (core)
├── transcriber.py       - Faster-Whisper wrapper
├── output.py            - Sinks (stdout, JSONL, optional WAV)
├── daemon.py            - Orchestrator (threads, queues, event emission)
├── events.py            - NDJSON event envelope + codec
├── bus.py               - Publish-only event bus (socket, ring replay)
├── assistant/
│   ├── modes.py         - Mode state machine (idle/command_armed/executing/chat)
│   ├── service.py       - Modes, command buffer, readiness, timeouts
│   ├── control.py       - Control socket (viewers, commands, broadcast)
│   ├── executor.py      - NullExecutor / LlmExecutor (tool loop)
│   ├── tools.py         - Tool registry + schema validation + get_datetime
│   ├── llm.py           - llama.cpp client + chat session (block trimming)
│   ├── audit.py         - Daily action audit log
│   ├── runner.py        - `stt-assistant` process
│   └── __main__.py      - Entry point for `python -m stt_daemon.assistant`
└── tui/
    ├── state.py         - Pure view-state reducer
    ├── client.py        - Control-socket client
    └── app.py           - Textual app + `stt-tui` entry point

tests/                    - 206 tests, no audio hardware or live LLM required

scripts/
├── audio-sink.sh        - Thin wrapper over `stt-daemon setup|clear|list`
└── hotkeys/             - sxhkd / xbindkeys examples

Running linters

uv run ruff check .

Building/packaging

# Build sdist + wheel (hatchling backend, installed in an isolated env by uv)
uv build

Limitations & Design Decisions

1. Mono only: Stereo support can be added, but speech is typically mono in transcription pipelines
2. Single language per transcription: Whisper detects one language per utterance; mixed-language speech is attributed to the dominant language
3. LLM is optional and separate: the daemon itself is transcription-only; the stt-assistant process adds chat/commands/tools on top, and can be disabled with [llm].enabled = false
4. PulseAudio/PipeWire-Pulse: The default parec backend assumes UNIX-like systems with PulseAudio. macOS/Windows should use sounddevice
5. Real-time: Latency depends on frame duration (30ms), silence threshold (default 700ms), and model size. Expect ~1–3 second latency with small model on CPU
6. Thread-safe queues: Reader thread (audio) is separate from worker thread (transcription) so audio never blocks waiting for transcription

References

• Faster-Whisper: https://github.com/SYSTRAN/faster-whisper
• WebRTC VAD: https://github.com/wiseman/py-webrtcvad
• PulseAudio: https://www.freedesktop.org/wiki/Software/PulseAudio/

License

This project is ISC-licensed. See LICENSE.md.

---

Made with ❤️ for Efren. Questions? Open an issue or reach out.