STT-Assistant

STT Daemon

An always-on speech-to-text daemon that listens to a PulseAudio/PipeWire source, detects voice activity, and transcribes utterances to Spanish/English text, with a separate voice-assistant layer for commands and chat on top.

Overview

• Capture audio continuously from PulseAudio/PipeWire sources
• Preserve the start of speech with a pre-roll (ring) buffer, so the first phoneme is never cut when voice detection fires
• Segment utterances with configurable silence thresholds
• Transcribe each utterance with Faster-Whisper (multilingual: Spanish, English, or auto-detect)
• Output to stdout (human-readable) and/or JSONL files
• Run non-blocking with worker threads so audio keeps flowing during transcription
• Act on speech through the separate stt-assistant process (modes, commands, chat, tools, audit)

Architecture

Audio Source (PulseAudio)
    ↓
ParecCapture / SoundDeviceCapture
    ↓
Pre-roll Ring Buffer (VAD lookback)
    ↓
WebRTC-VAD (voice activity detection)
    ↓
FrameSegmenter (utterance boundaries)
    ↓
Queue (segment_queue)
    ↓
Faster-Whisper (transcription)
    ↓
TranscriptSink (stdout + <session_dir>/<id>/transcripts.jsonl) + EventPublisher
    ↓  publish-only UNIX socket ($XDG_RUNTIME_DIR/stt-daemon.sock)
stt-assistant  (subscribes; owns modes, command buffer, chat, LLM, tools)
    ↓  control socket ($XDG_RUNTIME_DIR/stt-assistant.sock)
stt-tui (viewer) / stt-ctl (hotkeys)     troubleshoot/audit/YYYY-MM-DD.jsonl ←┘

The daemon never accepts input: it only transcribes and publishes. Everything stateful lives in the assistant, a separate process (see docs/ASSISTANT.md).

Prerequisites

System packages

• PulseAudio or PipeWire-Pulse: pactl, parec command-line tools

  • On Ubuntu/Debian: apt-get install pulseaudio-utils or apt-get install pipewire pipewire-pulse
  • On Fedora: dnf install pulseaudio-utils
  • On macOS: Install via Homebrew (though PulseAudio is Linux-first; the sounddevice backend is recommended on macOS)

• FFmpeg (optional, used by Faster-Whisper for format conversion)

  • On Ubuntu/Debian: apt-get install ffmpeg

Python

• Python 3.11+ (tested on 3.13.5)
• uv package manager (https://docs.astral.sh/uv/)

Installation

Clone the repo and install with uv:

# From the repo root (wherever you cloned it)
cd /path/to/STT-TTS

# Sync dependencies (installs into .venv)
uv sync

# (Optional) If you want to use the sounddevice backend (requires portaudio):
uv sync --extra portaudio

# (Optional) Enable the triage panel (installs the pinned in-process Laya lane):
uv sync --extra triage

# (Optional) Enable speaker identification (installs sherpa-onnx):
uv sync --extra speaker

AMD / ROCm hosts

uv sync installs the CUDA CTranslate2 wheel, which loads fine on a ROCm machine but silently runs on CPU. After the last sync, install the ROCm stack:

scripts/install-rocm.sh

Any later uv sync or plain uv run (it syncs implicitly) puts the CUDA wheels back. Once the ROCm stack is in place, launch with .venv/bin/stt-daemon ... or uv run --no-sync ..., and run tests with .venv/bin/python -m pytest. See docs/TROUBLESHOOTING.md.

The triage extra is opt-in: the default sync does not install Laya, and the test suite stays green without it.

The speaker extra is likewise opt-in and off by default. To enable speaker identification, run uv sync --extra speaker and place nemo_en_titanet_small.onnx under models/ (gitignored):

curl -L -o models/nemo_en_titanet_small.onnx \
  "https://github.com/k2-fsa/sherpa-onnx/releases/download/speaker-recongition-models/nemo_en_titanet_small.onnx"

Then set model_path and enabled = true under [speaker] in config.toml. Without the extra and model, the default sync is ONNX-free and speaker status stays "none". Note that uv sync --extra speaker is exclusive: it pins the environment to the default deps plus speaker, which drops the triage extra and its stack. To keep both, pass both flags: uv sync --extra speaker --extra triage. See docs/SPEAKERS.md for the full reference. You can also pre-enroll a known speaker from existing WAV files with stt-daemon speaker-enroll; see the pre-enrollment section of docs/SPEAKERS.md. For a guided flow, stt-daemon speaker-enroll-seed finds the clips closest to a seed you trust and lets you review each one before enrolling; see docs/SPEAKERS.md. Profiles can be tuned by re-running assisted enrollment (prototypes append, and the oldest is evicted at the cap), and the store can be wiped and rebuilt; see docs/SPEAKERS.md for both. Assisted review now flags near-duplicate and likely-other-speaker candidates and both enrollment commands strip near-duplicates from the profile automatically, while stt-daemon speaker-eval --labeled DIR scores the store against labeled clips; see docs/SPEAKERS.md.

Quick Start

Copy the example config and edit as needed:

cp config.example.toml config.toml

To turn on the triage panel, also copy the questions file and enable the section:

cp questions.example.json questions.json
# then set `enabled = true` under `[triage]` in config.toml

questions.json is private user state and is gitignored, like config.toml. See docs/TRIAGE.md for the panel and the questions format.

Then launch the whole system with one command:

uv run stt-daemon up --with-tui

That supervises the daemon, the assistant, and the TUI together; quitting the TUI stops everything cleanly. On an AMD/ROCm host, use .venv/bin/stt-daemon up --with-tui instead — see AMD / ROCm hosts. The manual flow is the same three steps without the supervisor:

uv run stt-daemon setup       # create the virtual sink (interactive picker)
uv run stt-daemon run         # start listening and transcribing
uv run stt-daemon clear --yes # tear down the sink when done

See docs/CLI.md for every subcommand and docs/CONFIGURATION.md for the config reference.

Documentation

• docs/CONFIGURATION.md — annotated TOML reference for every config key.
• docs/AUDIO-SETUP.md — virtual sink creation and audio routing options.
• docs/CLI.md — stt-daemon subcommands and common overrides.
• docs/ASSISTANT.md — the assistant layer: modes, command flow, LLM, tools, audit.
• docs/TRIAGE.md — the triage panel: Laya lane, the living brief, and questions file.
• docs/SPEAKERS.md — speaker identification: enrollment, thresholds, store, and commands.
• docs/OUTPUT.md — stdout, JSONL, and WAV output formats and fields.
• docs/TROUBLESHOOTING.md — common errors and how to fix them.
• docs/DEVELOPMENT.md — code structure, testing, linting, and packaging.
• docs/PYWHISPERCPP.md — building the pywhispercpp GPU backend from source.

Testing

Run the full suite with no audio hardware or live LLM required:

uv run pytest -v
# On an AMD/ROCm host, use `.venv/bin/python -m pytest -v` so the ROCm wheels are not reverted.

See docs/DEVELOPMENT.md for what the tests cover.

References

• Faster-Whisper: https://github.com/SYSTRAN/faster-whisper
• WebRTC VAD: https://github.com/wiseman/py-webrtcvad
• PulseAudio: https://www.freedesktop.org/wiki/Software/PulseAudio/

License

This project is ISC-licensed. See LICENSE.md.