STT-Assistant/docs/AUDIO-SETUP.md

Audio Setup

This document covers choosing and capturing an audio source, how the virtual sink is provisioned, and why the pre-roll buffer exists.

Audio Source Setup

By default, the daemon listens to VirtualMicSink.monitor (the monitor of the virtual sink). Apps route audio into VirtualMicSink, and the daemon captures the mix.

Option 1: Use the CLI with interactive setup (recommended)

# Show devices and create a virtual sink (interactive picker)
uv run stt-daemon setup

# Check status
uv run stt-daemon list

# In another terminal, start the daemon
uv run stt-daemon run

# The daemon auto-uses the provisioned sink's monitor as the capture device
# (no need to manually specify --device)

# Route audio:
#   - In pavucontrol (GUI), drag apps' "Recording" side to "VirtualMicSink"
#   - Or use pactl: pactl move-sink-input <app-index> VirtualMicSink

# Stop when done
uv run stt-daemon clear --yes

Option 2: Use the provided shell wrapper

For convenience, scripts/audio-sink.sh is a thin wrapper around the Python CLI:

scripts/audio-sink.sh start   # → uv run stt-daemon setup
scripts/audio-sink.sh status  # → uv run stt-daemon list
scripts/audio-sink.sh stop    # → uv run stt-daemon clear --yes

Option 3: Manual PulseAudio setup

If you prefer manual control or want to understand what's happening:

# Get your default microphone source
default_source=$(pactl get-default-source)
echo $default_source  # e.g., alsa_input.usb-...

# Create null sink
pactl load-module module-null-sink sink_name=VirtualMicSink \
    sink_properties=device.description=Virtual-Mic-Sink

# Expose the sink's monitor as a microphone
pactl load-module module-remap-source \
    master=VirtualMicSink.monitor \
    source_name=VirtualMic \
    source_properties=device.description=Virtual-Microphone

# Route your microphone into the sink
pactl load-module module-loopback source="$default_source" sink=VirtualMicSink latency_msec=1

# (Optional) Monitor the sink output on your speakers
pactl load-module module-loopback source=VirtualMicSink.monitor latency_msec=1

# Now tell the daemon to use it
uv run stt-daemon --device VirtualMicSink.monitor

# Clean up (unload modules and remove the file):
pactl unload-module <sink_id>
pactl unload-module <source_id>
pactl unload-module <loopback_id_1>
pactl unload-module <loopback_id_2>

Option 4: Use a physical microphone directly

If you have a single physical mic and don't need the virtual routing:

# List available sources
pactl list short sources

# Start daemon with your source
uv run stt-daemon --device alsa_input.usb-C-Media_Electronics_Inc._USB_Audio_Device-00.mono-fallback

Option 5: Use sounddevice backend

If you prefer not to use PulseAudio (e.g., on macOS or Windows):

uv sync --extra portaudio
uv run stt-daemon --backend sounddevice --device 0  # device 0 = default input

Virtual Sink Provisioning: How It Works

The daemon uses PulseAudio's module system to create a virtual microphone from multiple sources:

What Gets Created

1. Null Sink (module-null-sink, e.g., VirtualMicSink)

   • A sink that doesn't play anywhere; apps route audio into it instead.

2. Remap Source (module-remap-source, e.g., VirtualMic)

   • Exposes the null-sink's monitor as a microphone so the daemon can capture from it.

3. Loopback Modules (module-loopback, one per selected source)

   • Routes each selected input (microphone, speaker monitor, etc.) into the null-sink.

State Persistence

When you run uv run stt-daemon setup, the daemon:

• Creates all modules and records their numeric IDs
• Saves the state to audio/audio.json (or $STT_DAEMON_STATE)
• Allows run to re-provision if the sink is missing (e.g., after reboot)
• Allows clear to unload the modules it created — and falls back to name-based discovery (sink_name/remap_source_name in pactl list short modules) when audio.json is missing, so orphaned VirtualMicSink/VirtualMic sinks are still cleaned up (e.g., after a reboot or manual state-file delete)

The state file is JSON and can be edited manually if needed, but the daemon handles it automatically.

Feedback Risk

By default, monitor_output = false, so the sink's output is not looped back to the default speaker. If you enable it, speakers will play back everything captured in the sink—including your own voice—causing audio feedback. Keep it OFF unless you specifically want monitoring.

Pre-roll Buffer: Why It Matters

The pre-roll buffer is a ring buffer that keeps the last N milliseconds (default 400ms) of audio BEFORE speech is detected. When the VAD confirms speech has started, those pre-roll frames are prepended to the utterance.

Without pre-roll: The first phoneme of "hello" might be cut off—you'd get "llo" or "elo" because the VAD fires AFTER the first sound is already gone.

With pre-roll: The entire "hello" is captured, including the initial "h" and "e", because the ring buffer preserved them.

How it works (under the hood)

1. Silence phase: Frames are pushed into a ring buffer (maxlen = pre_roll_ms / frame_ms). Old frames fall off.
2. Speech detected: VAD returns True for consecutive speech_start_ms worth of frames.
3. Speech confirmed: _begin() is called. The entire ring buffer contents are copied into the segment, then regular frames append to it.
4. End of turn: after silence_ms of silence the segment is finalized and transcribed.

Long speech and silence

Two extra rules keep long monologues from becoming one huge blob, and keep noise from being transcribed:

• Pause-aligned chunking: once a segment reaches soft_max_s (10s) of speech, a shorter chunk_pause_ms (350ms) pause ends a chunk. The cut lands on a natural phrase boundary, so no word is split and no overlap/duplication occurs. max_utterance_s (30s) remains the last-resort cap for someone who never pauses.
• Noise suppression: min_utterance_ms drops blips, rms_floor drops segments too quiet to be speech, trailing_trim_ms strips hallucination-prone trailing silence before transcription, and Whisper segments below no_speech_prob_max / avg_logprob_min are discarded.