STT-Assistant

STT Daemon

An always-on speech-to-text daemon that continuously listens to a PulseAudio/PipeWire audio source, detects voice activity with webrtcvad, and transcribes utterances to Spanish/English text using Faster-Whisper.

Overview

This daemon is designed to:

• Capture audio continuously from PulseAudio/PipeWire sources
• Preserve the start of speech using a pre-roll (ring) buffer, so the first phoneme is never cut when voice detection fires
• Segment utterances with configurable silence thresholds
• Transcribe each utterance with Faster-Whisper (supports multilingual, Spanish, English, or auto-detect)
• Output to stdout (human-readable) and/or JSONL files (for logging/downstream processing)
• Run non-blocking with worker threads so audio keeps flowing during transcription

Architecture

Audio Source (PulseAudio)
    ↓
ParecCapture / SoundDeviceCapture
    ↓
Pre-roll Ring Buffer (VAD lookback)
    ↓
WebRTC-VAD (voice activity detection)
    ↓
FrameSegmenter (utterance boundaries)
    ↓
Queue (segment_queue)
    ↓
Faster-Whisper (transcription)
    ↓
TranscriptSink (stdout + JSONL)

Prerequisites

System packages

• PulseAudio or PipeWire-Pulse: pactl, parec command-line tools

  • On Ubuntu/Debian: apt-get install pulseaudio-utils or apt-get install pipewire pipewire-pulse
  • On Fedora: dnf install pulseaudio-utils
  • On macOS: Install via Homebrew (though PulseAudio is Linux-first; sounddevice backend recommended on macOS)

• FFmpeg (optional, used by Faster-Whisper for format conversion)

  • On Ubuntu/Debian: apt-get install ffmpeg

Python

• Python 3.11+ (tested on 3.13.5)
• uv package manager (https://docs.astral.sh/uv/)

Installation

Clone the repo and install with uv:

cd /home/efren/Repositories/STT-TTS

# Sync dependencies (installs into .venv)
uv sync

# (Optional) If you want to use the sounddevice backend (requires portaudio):
uv sync --extra portaudio

Configuration

Quick Start

Copy the example config and edit as needed:

cp config.example.toml config.toml
# Edit config.toml to adjust settings (see section below)

Configuration Reference

The config file is TOML. Here are all available options with defaults:

[audio]
backend = "parec"                     # "parec" (PulseAudio by name) | "sounddevice"
device  = "VirtualMicSink.monitor"    # PulseAudio source name (parec) or device index/name (sounddevice)
sample_rate = 16000                   # Hz
channels    = 1                       # Mono only
frame_ms    = 30                      # webrtcvad requires 10, 20, or 30
gain        = 1.0                     # Amplitude scaling (1.0 = no change, >1 = louder)

[vad]
aggressiveness   = 2                  # 0–3: higher = fewer false positives (speech misses more)
pre_roll_ms      = 400                # Ring buffer duration: keep this many ms of pre-speech
speech_start_ms  = 90                 # Consecutive speech frames to confirm speech start
silence_ms       = 700                # Consecutive silence to end a turn
min_utterance_ms = 300                # Discard utterances with less actual speech than this
max_utterance_s  = 30.0               # Hard cap: force-finalize a very long utterance
chunk_pause_ms   = 350                # Once soft_max_s is reached, end a chunk at a pause this long
soft_max_s       = 10.0               # After this much speech, chunk-ends become eligible at pauses
rms_floor        = 0.008              # Drop segments quieter than this RMS (noise-triggered blips)
trailing_trim_ms = 200                # Keep only this much trailing silence (less hallucination-prone padding)

[whisper]
model        = "small"                # "tiny", "base", "small", "medium", "large"
device       = "cpu"                  # "cpu" or "cuda"
compute_type = "int8"                 # "int8" (CPU) or "float16" (CUDA, slower but more accurate)
language     = "auto"                 # "auto" (detect), "en", "es", or None/""/"none"
beam_size    = 1                      # 1 = fast greedy, >1 = slower beam search
vad_filter   = false                  # Use Whisper's built-in VAD (we already have webrtcvad)
no_speech_prob_max = 0.6              # Drop segments Whisper scores as non-speech above this
avg_logprob_min    = -1.0             # Drop segments with confidence below this

[sink]                                # Virtual audio sink configuration
sink_name             = "VirtualMicSink"       # null-sink name
remap_source_name     = "VirtualMic"           # Microphone source name
loopback_latency_msec = 1                      # Module latency (ms)
monitor_output        = false                  # Loop sink.monitor → default sink (OFF = no feedback risk)

[output]
stdout     = true                     # Print transcripts to stdout
jsonl_dir  = "transcripts"            # Write JSONL files (one per day) to this dir
save_audio = false                    # Also write each transcribed utterance as a 16 kHz mono WAV
audio_dir  = "audio"                  # WAV destination (YYYY-MM-DD subfolders); only used if save_audio

[sink]
sink_name             = "VirtualMicSink"    # Name of the null sink to create
remap_source_name     = "VirtualMic"        # Name of the exposed microphone
loopback_latency_msec = 1                   # Loopback latency in ms (lower = faster, higher = smoother)
monitor_output        = false               # Loop sink output back to default speaker (CAUTION: feedback risk!)

Audio Source Setup

By default, the daemon listens to VirtualMicSink.monitor (the monitor of the virtual sink). Apps route audio into VirtualMicSink, and the daemon captures the mix.

Option 1: Use the CLI with interactive setup (recommended)

# Show devices and create a virtual sink (interactive picker)
uv run stt-daemon setup

# Check status
uv run stt-daemon list

# In another terminal, start the daemon
uv run stt-daemon run

# The daemon auto-uses the provisioned sink's monitor as the capture device
# (no need to manually specify --device)

# Route audio:
#   - In pavucontrol (GUI), drag apps' "Recording" side to "VirtualMicSink"
#   - Or use pactl: pactl move-sink-input <app-index> VirtualMicSink

# Stop when done
uv run stt-daemon clear --yes

Option 2: Use the provided shell wrapper

For convenience, scripts/audio-sink.sh is a thin wrapper around the Python CLI:

scripts/audio-sink.sh start   # → uv run stt-daemon setup
scripts/audio-sink.sh status  # → uv run stt-daemon list
scripts/audio-sink.sh stop    # → uv run stt-daemon clear --yes

Option 3: Manual PulseAudio setup

If you prefer manual control or want to understand what's happening:

# Get your default microphone source
default_source=$(pactl get-default-source)
echo $default_source  # e.g., alsa_input.usb-...

# Create null sink
pactl load-module module-null-sink sink_name=VirtualMicSink \
    sink_properties=device.description=Virtual-Mic-Sink

# Expose the sink's monitor as a microphone
pactl load-module module-remap-source \
    master=VirtualMicSink.monitor \
    source_name=VirtualMic \
    source_properties=device.description=Virtual-Microphone

# Route your microphone into the sink
pactl load-module module-loopback source="$default_source" sink=VirtualMicSink latency_msec=1

# (Optional) Monitor the sink output on your speakers
pactl load-module module-loopback source=VirtualMicSink.monitor latency_msec=1

# Now tell the daemon to use it
uv run stt-daemon --device VirtualMicSink.monitor

# Clean up (unload modules and remove the file):
pactl unload-module <sink_id>
pactl unload-module <source_id>
pactl unload-module <loopback_id_1>
pactl unload-module <loopback_id_2>

Option 3: Use a physical microphone directly

If you have a single physical mic and don't need the virtual routing:

# List available sources
pactl list short sources

# Start daemon with your source
uv run stt-daemon --device alsa_input.usb-C-Media_Electronics_Inc._USB_Audio_Device-00.mono-fallback

Option 4: Use sounddevice backend

If you prefer not to use PulseAudio (e.g., on macOS or Windows):

uv sync --extra portaudio
uv run stt-daemon --backend sounddevice --device 0  # device 0 = default input

Running the Daemon

Subcommands

The stt-daemon CLI supports these subcommands:

setup — Create virtual sink and loopbacks

Interactive setup (shows device picker):

uv run stt-daemon setup

Non-interactive setup (specify sources directly):

uv run stt-daemon setup --sources "alsa_input.usb-Foo,alsa_output.pci-0000.monitor"

This creates:

• A null sink (by default VirtualMicSink)
• A remap-source to expose the sink's monitor as a microphone (by default VirtualMic)
• Loopback modules routing each selected source into the sink
• A state file tracking all module IDs for later cleanup

run — Start the daemon (auto-provisions if needed)

uv run stt-daemon run

If no saved provisioning exists and the sink is missing, the daemon will:

1. Show the interactive device picker
2. Create the sink and loopbacks (if sources are selected)
3. Start listening and transcribing

You can also override settings:

uv run stt-daemon run --device "alsa_input.usb-..." --language es --model base
uv run stt-daemon run -v  # Verbose logging
uv run stt-daemon run --check  # Test audio capture + model, then exit

list — Show devices and provisioning status

uv run stt-daemon list

Output shows:

• Input devices (real microphones)
• Output devices (speakers/headphones; captured via their .monitor source)
• Current provisioning state (if a sink has been created)

clear — Remove virtual sink and loopbacks

uv run stt-daemon clear
# or without confirmation prompt:
uv run stt-daemon clear --yes

This unloads all tracked modules and removes the state file.

(Legacy) --list-sources flag

uv run stt-daemon --list-sources
# Equivalent to: uv run stt-daemon list

(Legacy) --check flag

uv run stt-daemon --check
# Equivalent to: uv run stt-daemon run --check

Interactive Device Picker

When you run setup or run without --sources, the daemon shows an interactive picker:

• TTY mode (terminal): Uses questionary to show a checkbox UI. Pre-checked items are from the previously saved selection.
• Non-TTY mode (piped input): Shows a numbered menu; enter space/comma-separated indices, all, none, or blank (to keep previous selection).

Device labels:

• [IN] <description> (<name>) — Input device (real microphone)
• [OUT] <description> (<name>) — Output device (monitor of a sink)

Running the daemon

The simplest workflow:

# Create/setup the sink (interactive)
uv run stt-daemon setup

# Start the daemon in another terminal
# (auto-uses the provisioned sink)
uv run stt-daemon run

# Stop and remove the sink when done
uv run stt-daemon clear --yes

Output examples

Stdout (human-readable):

[14:23:45] (es) Hola, cómo estás?  [rms=0.041 end_of_turn]
[14:23:52] (en) What time is it?  [rms=0.038 end_of_turn]
[14:24:10] (es) Gracias por tu ayuda.  [rms=0.052 pause]

JSONL (one line per utterance):

{"ts": "2025-09-20T14:23:45.123456", "lang": "es", "lang_prob": 0.95, "duration_s": 2.5, "rms": 0.04123, "end_reason": "end_of_turn", "text": "Hola, cómo estás?"}
{"ts": "2025-09-20T14:23:52.654321", "lang": "en", "lang_prob": 0.98, "duration_s": 1.8, "rms": 0.03811, "end_reason": "pause", "text": "What time is it?"}

end_reason is one of end_of_turn (full silence_ms elapsed), pause (chunked at chunk_pause_ms because it passed soft_max_s), hard_cap (max_utterance_s reached), or flush (shutdown mid-utterance).

Audio files — when save_audio = true, each transcribed utterance is also written as a 16 kHz mono WAV under audio_dir/YYYY-MM-DD/, and the record gains an "audio" field with its path:

{..., "end_reason": "pause", "text": "Hola", "audio": "audio/2025-09-20/142345-123-pause.wav"}

16 kHz mono 16-bit is ~32 KB/s (~1.9 MB/minute), so enable this only if you need the clips for auditing.

Virtual Sink Provisioning: How It Works

The daemon uses PulseAudio's module system to create a virtual microphone from multiple sources:

What Gets Created

1. Null Sink (module-null-sink, e.g., VirtualMicSink)

   • A sink that doesn't play anywhere; apps route audio into it instead.
     

2. Remap Source (module-remap-source, e.g., VirtualMic)

   • Exposes the null-sink's monitor as a microphone so the daemon can capture from it.
     

3. Loopback Modules (module-loopback, one per selected source)

   • Routes each selected input (microphone, speaker monitor, etc.) into the null-sink.

State Persistence

When you run uv run stt-daemon setup, the daemon:

• Creates all modules and records their numeric IDs
• Saves the state to ~/.local/state/stt-daemon/audio.json (or $STT_DAEMON_STATE)
• Allows run to re-provision if the sink is missing (e.g., after reboot)
• Allows clear to unload only the modules it created

The state file is JSON and can be edited manually if needed, but the daemon handles it automatically.

Feedback Risk

By default, monitor_output = false, so the sink's output is not looped back to the default speaker. If you enable it, speakers will play back everything captured in the sink—including your own voice—causing audio feedback. Keep it OFF unless you specifically want monitoring.

Pre-roll Buffer: Why It Matters

The pre-roll buffer is a ring buffer that keeps the last N milliseconds (default 400ms) of audio BEFORE speech is detected. When the VAD confirms speech has started, those pre-roll frames are prepended to the utterance.

Without pre-roll: The first phoneme of "hello" might be cut off—you'd get "llo" or "elo" because the VAD fires AFTER the first sound is already gone.

With pre-roll: The entire "hello" is captured, including the initial "h" and "e", because the ring buffer preserved them.

How it works (under the hood)

1. Silence phase: Frames are pushed into a ring buffer (maxlen = pre_roll_ms / frame_ms). Old frames fall off.
2. Speech detected: VAD returns True for consecutive speech_start_ms worth of frames.
3. Speech confirmed: _begin() is called. The entire ring buffer contents are copied into the segment, then regular frames append to it.
4. End of turn: after silence_ms of silence the segment is finalized and transcribed.

Long speech and silence

Two extra rules keep long monologues from becoming one huge blob, and keep noise from being transcribed:

• Pause-aligned chunking: once a segment reaches soft_max_s (10s) of speech, a shorter chunk_pause_ms (350ms) pause ends a chunk. The cut lands on a natural phrase boundary, so no word is split and no overlap/duplication occurs. max_utterance_s (30s) remains the last-resort cap for someone who never pauses.
• Noise suppression: min_utterance_ms drops blips, rms_floor drops segments too quiet to be speech, trailing_trim_ms strips hallucination-prone trailing silence before transcription, and Whisper segments below no_speech_prob_max / avg_logprob_min are discarded.

Troubleshooting

"Device not found"

Error: Device 'VirtualMicSink.monitor' not found. Available sources: [...]
To create VirtualMicSink, run: scripts/audio-sink.sh start

Fix:

scripts/audio-sink.sh start

Then route audio into VirtualMicSink and try again.

High latency / slow transcription

By default, the daemon uses compute_type = "int8" and device = "cpu", which trades speed for accuracy. To speed up:

1. Reduce model size: --model base or --model tiny
2. Enable GPU (if you have CUDA): --device cuda with compute_type float16
3. Reduce beam_size: Edit config to beam_size = 1 (already default, very fast)

Example:

uv run stt-daemon --model base --device cuda

Spanish / English / Other languages

The Faster-Whisper model is multilingual by default. It auto-detects language or can be forced:

uv run stt-daemon --language es   # Force Spanish
uv run stt-daemon --language en   # Force English
uv run stt-daemon --language auto # Auto-detect (default)

The output includes the detected language tag: [14:23:45] (es) ....

Recognizing Spanish vs. English better

• Force the language: Use --language es or --language en
• Use a larger model: --model medium or --model large (slower, more accurate)
• Tune VAD aggressiveness: Lower values (0–1) capture more, higher values (2–3) are stricter

Audio device not responding / parec hangs

If parec seems stuck:

1. Check if the source exists: pactl list short sources
2. Kill the daemon (Ctrl+C)
3. Verify PulseAudio is running: systemctl status pulseaudio or check journalctl
4. Restart PulseAudio if needed: systemctl restart pulseaudio
5. Re-run the daemon

Sounddevice backend issues

If using --backend sounddevice and getting import errors:

uv sync --extra portaudio

Then verify available devices:

python -c "import sounddevice as sd; print(sd.query_devices())"

Use the device index or name as --device 0 or --device "Microphone (...).

No transcriptions appearing

1. Check VAD is firing: uv run stt-daemon -v (verbose logging)
2. Verify audio is loud enough: uv run stt-daemon --check (check RMS)
3. Reduce aggressiveness (0–1 instead of 2–3) to detect quieter speech
4. Check min_utterance_ms isn't too high (default 300ms should be fine)

Whisper model not loading

Error loading model: ...

The model is downloaded on first run (~1.5 GB for small). Check:

1. Disk space: df -h (needs ~2 GB free)
2. Internet: Model is downloaded from Hugging Face
3. HF cache location: Models go to ~/.cache/huggingface/hub/ by default

You can override the cache:

export HF_HOME=/path/to/cache
uv run stt-daemon --model small

Testing

Run the test suite (no audio hardware required):

uv run pytest -v

Tests cover:

• Pre-roll buffer preservation: confirm the first frames are not cut
• Silence finalization: utterances end after N silence frames
• Min/max utterance duration: blips are discarded, long utterances are finalized
• Config loading and language normalization

Development

Code structure

src/stt_daemon/
├── __init__.py          - Package metadata
├── __main__.py          - Entry point for `python -m stt_daemon`
├── cli.py               - Argument parsing and entry point
├── config.py            - Config loading (TOML)
├── audio.py             - Audio capture (parec, sounddevice)
├── segmenter.py         - VAD + pre-roll buffer + utterance segmentation (core)
├── transcriber.py       - Faster-Whisper wrapper
├── output.py            - Sinks (stdout, JSONL)
└── daemon.py            - Main orchestrator (threads, queues)

tests/
├── test_segmenter.py    - Pre-roll buffer, VAD, segmentation logic
└── test_config.py       - Config loading

scripts/
└── audio-sink.sh        - PulseAudio virtual sink manager

Running linters

uv run ruff check .

Building/packaging

# Install hatchling if needed
uv pip install hatchling

# Build wheel
uv run python -m build

Limitations & Design Decisions

1. Mono only: Stereo support can be added, but speech is typically mono in transcription pipelines
2. Single language per transcription: Whisper detects one language per utterance; mixed-language speech is attributed to the dominant language
3. No LLM integration: This is transcription-only; use the JSONL output to feed an LLM if needed
4. PulseAudio/PipeWire-Pulse: The default parec backend assumes UNIX-like systems with PulseAudio. macOS/Windows should use sounddevice
5. Real-time: Latency depends on frame duration (30ms), silence threshold (default 700ms), and model size. Expect ~1–3 second latency with small model on CPU
6. Thread-safe queues: Reader thread (audio) is separate from worker thread (transcription) so audio never blocks waiting for transcription

References

• Faster-Whisper: https://github.com/SYSTRAN/faster-whisper
• WebRTC VAD: https://github.com/wiseman/py-webrtcvad
• PulseAudio: https://www.freedesktop.org/wiki/Software/PulseAudio/

License

This project is MIT-licensed. See LICENSE file (if included).

---

Made with ❤️ for Efren. Questions? Open an issue or reach out.