# STT Daemon

An always-on speech-to-text daemon that continuously listens to a PulseAudio/PipeWire audio source, detects voice activity with webrtcvad, and transcribes utterances to Spanish/English text using Faster-Whisper.

## Overview

This daemon is designed to:

- **Capture audio continuously** from PulseAudio/PipeWire sources
- **Preserve the start of speech** using a pre-roll (ring) buffer, so the first phoneme is never cut when voice detection fires
- **Segment utterances** with configurable silence thresholds
- **Transcribe** each utterance with Faster-Whisper (supports multilingual, Spanish, English, or auto-detect)
- **Output to stdout** (human-readable) and/or JSONL files (for logging/downstream processing)
- **Run non-blocking** with worker threads so audio keeps flowing during transcription

### Architecture

```
Audio Source (PulseAudio)
    ↓
ParecCapture / SoundDeviceCapture
    ↓
Pre-roll Ring Buffer (VAD lookback)
    ↓
WebRTC-VAD (voice activity detection)
    ↓
FrameSegmenter (utterance boundaries)
    ↓
Queue (segment_queue)
    ↓
Faster-Whisper (transcription)
    ↓
TranscriptSink (stdout + JSONL)
```

## Prerequisites

### System packages

- **PulseAudio or PipeWire-Pulse**: `pactl`, `parec` command-line tools
  - On Ubuntu/Debian: `apt-get install pulseaudio-utils` or `apt-get install pipewire pipewire-pulse`
  - On Fedora: `dnf install pulseaudio-utils`
  - On macOS: Install via Homebrew (though PulseAudio is Linux-first; `sounddevice` backend recommended on macOS)

- **FFmpeg** (optional, used by Faster-Whisper for format conversion)
  - On Ubuntu/Debian: `apt-get install ffmpeg`

### Python

- **Python 3.11+** (tested on 3.13.5)
- **uv** package manager (https://docs.astral.sh/uv/)

## Installation

Clone the repo and install with `uv`:

```bash
cd /home/efren/Repositories/STT-TTS

# Sync dependencies (installs into .venv)
uv sync

# (Optional) If you want to use the sounddevice backend (requires portaudio):
uv sync --extra portaudio
```

## Configuration

### Quick Start

Copy the example config and edit as needed:

```bash
cp config.example.toml config.toml
# Edit config.toml to adjust settings (see section below)
```

### Configuration Reference

The config file is TOML. Here are all available options with defaults:

```toml
[audio]
backend = "parec"                     # "parec" (PulseAudio by name) | "sounddevice"
device  = "VirtualMicSink.monitor"    # PulseAudio source name (parec) or device index/name (sounddevice)
sample_rate = 16000                   # Hz
channels    = 1                       # Mono only
frame_ms    = 30                      # webrtcvad requires 10, 20, or 30
gain        = 1.0                     # Amplitude scaling (1.0 = no change, >1 = louder)

[vad]
aggressiveness   = 2                  # 0–3: higher = fewer false positives (speech misses more)
pre_roll_ms      = 400                # Ring buffer duration: keep this many ms of pre-speech
speech_start_ms  = 90                 # Consecutive speech frames to confirm speech start
silence_ms       = 700                # Consecutive silence to end an utterance
min_utterance_ms = 300                # Discard utterances shorter than this
max_utterance_s  = 30.0               # Force-finalize if utterance exceeds this duration

[whisper]
model        = "small"                # "tiny", "base", "small", "medium", "large"
device       = "cpu"                  # "cpu" or "cuda"
compute_type = "int8"                 # "int8" (CPU) or "float16" (CUDA, slower but more accurate)
language     = "auto"                 # "auto" (detect), "en", "es", or None/""/"none"
beam_size    = 1                      # 1 = fast greedy, >1 = slower beam search
vad_filter   = false                  # Use Whisper's built-in VAD (we already have webrtcvad)

[sink]                                # Virtual audio sink configuration
sink_name             = "VirtualMicSink"       # null-sink name
remap_source_name     = "VirtualMic"           # Microphone source name
loopback_latency_msec = 1                      # Module latency (ms)
monitor_output        = false                  # Loop sink.monitor → default sink (OFF = no feedback risk)

[output]
stdout    = true                      # Print transcripts to stdout
jsonl_dir = "transcripts"             # Write JSONL files (one per day) to this dir

[sink]
sink_name             = "VirtualMicSink"    # Name of the null sink to create
remap_source_name     = "VirtualMic"        # Name of the exposed microphone
loopback_latency_msec = 1                   # Loopback latency in ms (lower = faster, higher = smoother)
monitor_output        = false               # Loop sink output back to default speaker (CAUTION: feedback risk!)
```

### Audio Source Setup

By default, the daemon listens to `VirtualMicSink.monitor` (the monitor of the virtual sink). Apps route audio into `VirtualMicSink`, and the daemon captures the mix.

#### Option 1: Use the CLI with interactive setup (recommended)

```bash
# Show devices and create a virtual sink (interactive picker)
uv run stt-daemon setup

# Check status
uv run stt-daemon list

# In another terminal, start the daemon
uv run stt-daemon run

# The daemon auto-uses the provisioned sink's monitor as the capture device
# (no need to manually specify --device)

# Route audio:
#   - In pavucontrol (GUI), drag apps' "Recording" side to "VirtualMicSink"
#   - Or use pactl: pactl move-sink-input <app-index> VirtualMicSink

# Stop when done
uv run stt-daemon clear --yes
```

#### Option 2: Use the provided shell wrapper

For convenience, `scripts/audio-sink.sh` is a thin wrapper around the Python CLI:

```bash
scripts/audio-sink.sh start   # → uv run stt-daemon setup
scripts/audio-sink.sh status  # → uv run stt-daemon list
scripts/audio-sink.sh stop    # → uv run stt-daemon clear --yes
```

#### Option 3: Manual PulseAudio setup

If you prefer manual control or want to understand what's happening:

```bash
# Get your default microphone source
default_source=$(pactl get-default-source)
echo $default_source  # e.g., alsa_input.usb-...

# Create null sink
pactl load-module module-null-sink sink_name=VirtualMicSink \
    sink_properties=device.description=Virtual-Mic-Sink

# Expose the sink's monitor as a microphone
pactl load-module module-remap-source \
    master=VirtualMicSink.monitor \
    source_name=VirtualMic \
    source_properties=device.description=Virtual-Microphone

# Route your microphone into the sink
pactl load-module module-loopback source="$default_source" sink=VirtualMicSink latency_msec=1

# (Optional) Monitor the sink output on your speakers
pactl load-module module-loopback source=VirtualMicSink.monitor latency_msec=1

# Now tell the daemon to use it
uv run stt-daemon --device VirtualMicSink.monitor

# Clean up (unload modules and remove the file):
pactl unload-module <sink_id>
pactl unload-module <source_id>
pactl unload-module <loopback_id_1>
pactl unload-module <loopback_id_2>
```

#### Option 3: Use a physical microphone directly

If you have a single physical mic and don't need the virtual routing:

```bash
# List available sources
pactl list short sources

# Start daemon with your source
uv run stt-daemon --device alsa_input.usb-C-Media_Electronics_Inc._USB_Audio_Device-00.mono-fallback
```

#### Option 4: Use sounddevice backend

If you prefer not to use PulseAudio (e.g., on macOS or Windows):

```bash
uv sync --extra portaudio
uv run stt-daemon --backend sounddevice --device 0  # device 0 = default input
```

## Running the Daemon

### Subcommands

The `stt-daemon` CLI supports these subcommands:

#### `setup` — Create virtual sink and loopbacks

Interactive setup (shows device picker):

```bash
uv run stt-daemon setup
```

Non-interactive setup (specify sources directly):

```bash
uv run stt-daemon setup --sources "alsa_input.usb-Foo,alsa_output.pci-0000.monitor"
```

This creates:
- A null sink (by default `VirtualMicSink`)
- A remap-source to expose the sink's monitor as a microphone (by default `VirtualMic`)
- Loopback modules routing each selected source into the sink
- A state file tracking all module IDs for later cleanup

#### `run` — Start the daemon (auto-provisions if needed)

```bash
uv run stt-daemon run
```

If no saved provisioning exists and the sink is missing, the daemon will:
1. Show the interactive device picker
2. Create the sink and loopbacks (if sources are selected)
3. Start listening and transcribing

You can also override settings:

```bash
uv run stt-daemon run --device "alsa_input.usb-..." --language es --model base
uv run stt-daemon run -v  # Verbose logging
uv run stt-daemon run --check  # Test audio capture + model, then exit
```

#### `list` — Show devices and provisioning status

```bash
uv run stt-daemon list
```

Output shows:
- Input devices (real microphones)
- Output devices (speakers/headphones; captured via their `.monitor` source)
- Current provisioning state (if a sink has been created)

#### `clear` — Remove virtual sink and loopbacks

```bash
uv run stt-daemon clear
# or without confirmation prompt:
uv run stt-daemon clear --yes
```

This unloads all tracked modules and removes the state file.

#### (Legacy) `--list-sources` flag

```bash
uv run stt-daemon --list-sources
# Equivalent to: uv run stt-daemon list
```

#### (Legacy) `--check` flag

```bash
uv run stt-daemon --check
# Equivalent to: uv run stt-daemon run --check
```

### Interactive Device Picker

When you run `setup` or `run` without `--sources`, the daemon shows an interactive picker:

- **TTY mode** (terminal): Uses `questionary` to show a checkbox UI. Pre-checked items are from the previously saved selection.
- **Non-TTY mode** (piped input): Shows a numbered menu; enter space/comma-separated indices, `all`, `none`, or blank (to keep previous selection).

Device labels:
- `[IN] <description> (<name>)` — Input device (real microphone)
- `[OUT] <description> (<name>)` — Output device (monitor of a sink)

### Running the daemon

The simplest workflow:

```bash
# Create/setup the sink (interactive)
uv run stt-daemon setup

# Start the daemon in another terminal
# (auto-uses the provisioned sink)
uv run stt-daemon run

# Stop and remove the sink when done
uv run stt-daemon clear --yes
```

### Output examples

**Stdout** (human-readable):
```
[14:23:45] (es) Hola, cómo estás?
[14:23:52] (en) What time is it?
[14:24:10] (es) Gracias por tu ayuda.
```

**JSONL** (one line per utterance):
```json
{"ts": "2025-09-20T14:23:45.123456", "lang": "es", "lang_prob": 0.95, "duration_s": 2.5, "text": "Hola, cómo estás?"}
{"ts": "2025-09-20T14:23:52.654321", "lang": "en", "lang_prob": 0.98, "duration_s": 1.8, "text": "What time is it?"}
```

## Virtual Sink Provisioning: How It Works

The daemon uses PulseAudio's module system to create a virtual microphone from multiple sources:

### What Gets Created

1. **Null Sink** (`module-null-sink`, e.g., `VirtualMicSink`)
   - A sink that doesn't play anywhere; apps route audio into it instead.
   
2. **Remap Source** (`module-remap-source`, e.g., `VirtualMic`)
   - Exposes the null-sink's monitor as a microphone so the daemon can capture from it.
   
3. **Loopback Modules** (`module-loopback`, one per selected source)
   - Routes each selected input (microphone, speaker monitor, etc.) into the null-sink.

### State Persistence

When you run `uv run stt-daemon setup`, the daemon:
- Creates all modules and records their numeric IDs
- Saves the state to `~/.local/state/stt-daemon/audio.json` (or `$STT_DAEMON_STATE`)
- Allows `run` to re-provision if the sink is missing (e.g., after reboot)
- Allows `clear` to unload only the modules it created

The state file is JSON and can be edited manually if needed, but the daemon handles it automatically.

### Feedback Risk

By default, `monitor_output = false`, so the sink's output is **not** looped back to the default speaker. If you enable it, speakers will play back everything captured in the sink—including your own voice—causing audio feedback. Keep it OFF unless you specifically want monitoring.

## Pre-roll Buffer: Why It Matters

The **pre-roll buffer** is a ring buffer that keeps the last N milliseconds (default 400ms) of audio BEFORE speech is detected. When the VAD confirms speech has started, those pre-roll frames are prepended to the utterance.

**Without pre-roll:** The first phoneme of "hello" might be cut off—you'd get "llo" or "elo" because the VAD fires AFTER the first sound is already gone.

**With pre-roll:** The entire "hello" is captured, including the initial "h" and "e", because the ring buffer preserved them.

### How it works (under the hood)

1. **Silence phase**: Frames are pushed into a ring buffer (maxlen = pre_roll_ms / frame_ms). Old frames fall off.
2. **Speech detected**: VAD returns True for consecutive `speech_start_ms` worth of frames.
3. **Speech confirmed**: `_begin()` is called. The entire ring buffer contents are copied into the segment, then regular frames append to it.
4. **Silence finalized**: After `silence_ms` of silence, the segment is converted to an Utterance and emitted.

## Troubleshooting

### "Device not found"

```
Error: Device 'VirtualMicSink.monitor' not found. Available sources: [...]
To create VirtualMicSink, run: scripts/audio-sink.sh start
```

**Fix:**
```bash
scripts/audio-sink.sh start
```

Then route audio into VirtualMicSink and try again.

### High latency / slow transcription

By default, the daemon uses `compute_type = "int8"` and `device = "cpu"`, which trades speed for accuracy. To speed up:

1. **Reduce model size**: `--model base` or `--model tiny`
2. **Enable GPU** (if you have CUDA): `--device cuda` with `compute_type float16`
3. **Reduce beam_size**: Edit config to `beam_size = 1` (already default, very fast)

Example:

```bash
uv run stt-daemon --model base --device cuda
```

### Spanish / English / Other languages

The Faster-Whisper model is **multilingual by default**. It auto-detects language or can be forced:

```bash
uv run stt-daemon --language es   # Force Spanish
uv run stt-daemon --language en   # Force English
uv run stt-daemon --language auto # Auto-detect (default)
```

The output includes the detected language tag: `[14:23:45] (es) ...`.

### Recognizing Spanish vs. English better

- **Force the language**: Use `--language es` or `--language en`
- **Use a larger model**: `--model medium` or `--model large` (slower, more accurate)
- **Tune VAD aggressiveness**: Lower values (0–1) capture more, higher values (2–3) are stricter

### Audio device not responding / parec hangs

If `parec` seems stuck:

1. Check if the source exists: `pactl list short sources`
2. Kill the daemon (Ctrl+C)
3. Verify PulseAudio is running: `systemctl status pulseaudio` or check journalctl
4. Restart PulseAudio if needed: `systemctl restart pulseaudio`
5. Re-run the daemon

### Sounddevice backend issues

If using `--backend sounddevice` and getting import errors:

```bash
uv sync --extra portaudio
```

Then verify available devices:

```bash
python -c "import sounddevice as sd; print(sd.query_devices())"
```

Use the device index or name as `--device 0` or `--device "Microphone (...)`.

### No transcriptions appearing

1. Check VAD is firing: `uv run stt-daemon -v` (verbose logging)
2. Verify audio is loud enough: `uv run stt-daemon --check` (check RMS)
3. Reduce `aggressiveness` (0–1 instead of 2–3) to detect quieter speech
4. Check `min_utterance_ms` isn't too high (default 300ms should be fine)

### Whisper model not loading

```
Error loading model: ...
```

The model is downloaded on first run (~1.5 GB for `small`). Check:

1. Disk space: `df -h` (needs ~2 GB free)
2. Internet: Model is downloaded from Hugging Face
3. HF cache location: Models go to `~/.cache/huggingface/hub/` by default

You can override the cache:

```bash
export HF_HOME=/path/to/cache
uv run stt-daemon --model small
```

## Testing

Run the test suite (no audio hardware required):

```bash
uv run pytest -v
```

Tests cover:

- **Pre-roll buffer** preservation: confirm the first frames are not cut
- **Silence finalization**: utterances end after N silence frames
- **Min/max utterance duration**: blips are discarded, long utterances are finalized
- **Config loading and language normalization**

## Development

### Code structure

```
src/stt_daemon/
├── __init__.py          - Package metadata
├── __main__.py          - Entry point for `python -m stt_daemon`
├── cli.py               - Argument parsing and entry point
├── config.py            - Config loading (TOML)
├── audio.py             - Audio capture (parec, sounddevice)
├── segmenter.py         - VAD + pre-roll buffer + utterance segmentation (core)
├── transcriber.py       - Faster-Whisper wrapper
├── output.py            - Sinks (stdout, JSONL)
└── daemon.py            - Main orchestrator (threads, queues)

tests/
├── test_segmenter.py    - Pre-roll buffer, VAD, segmentation logic
└── test_config.py       - Config loading

scripts/
└── audio-sink.sh        - PulseAudio virtual sink manager
```

### Running linters

```bash
uv run ruff check .
```

### Building/packaging

```bash
# Install hatchling if needed
uv pip install hatchling

# Build wheel
uv run python -m build
```

## Limitations & Design Decisions

1. **Mono only**: Stereo support can be added, but speech is typically mono in transcription pipelines
2. **Single language per transcription**: Whisper detects one language per utterance; mixed-language speech is attributed to the dominant language
3. **No LLM integration**: This is transcription-only; use the JSONL output to feed an LLM if needed
4. **PulseAudio/PipeWire-Pulse**: The default `parec` backend assumes UNIX-like systems with PulseAudio. macOS/Windows should use `sounddevice`
5. **Real-time**: Latency depends on frame duration (30ms), silence threshold (default 700ms), and model size. Expect ~1–3 second latency with `small` model on CPU
6. **Thread-safe queues**: Reader thread (audio) is separate from worker thread (transcription) so audio never blocks waiting for transcription

## References

- **Faster-Whisper**: https://github.com/SYSTRAN/faster-whisper
- **WebRTC VAD**: https://github.com/wiseman/py-webrtcvad
- **PulseAudio**: https://www.freedesktop.org/wiki/Software/PulseAudio/

## License

This project is MIT-licensed. See LICENSE file (if included).

---

Made with ❤️ for Efren. Questions? Open an issue or reach out.