Kezdőlap Felfedezés Súgó
Bejelentkezés
efren
/
STT-Assistant
1
0
Másolás 0
Fájlok Problémák 0 Beolvasztási kérések 0 Wiki
Fa: 12e3d3649a
Branch-ok Tag-ek
STT-Assistant
/
AGENTS.md

AGENTS.md 12 KB
Előzmények Nyers

AGENTS.md

Always-on speech-to-text daemon plus a separate voice-assistant layer. Python 3.11+ (dev env is 3.13, but requires-python = ">=3.11" and ruff targets py311 — don't use 3.12+ syntax), uv, src/ layout. stt_daemon/ is the only package; console scripts live in pyproject.toml (stt-daemon, stt-assistant, stt-ctl, stt-tui).

Commands

uv sync                                                  # install into .venv (see pywhispercpp below)
uv sync --extra portaudio                                # sounddevice capture backend only
uv sync --extra triage                                   # optional: in-process Laya triage lane
uv sync --extra speaker                                  # optional: in-process speaker embedding lane
# extras are exclusive under uv sync: pass both flags (--extra speaker --extra triage) to keep both
uv run pytest -v                                         # full suite (hardware- and network-free)
uv run pytest tests/test_segmenter.py::test_name -v      # single test
uv run ruff check .                                      # only static check: E,F,W, line-length 100, no typechecker
uv build                                                 # sdist + wheel (hatchling)

uv run stt-daemon up --with-tui                          # supervise all three processes (main launcher)
uv run stt-daemon setup && uv run stt-daemon run         # manual flow without supervisor

There is no CI, pre-commit, Makefile, or task runner. Run ruff + pytest manually before finishing.

ROCm environment (this host)

The repo's lock/vendor pins CUDA wheels (torch 2.14.0+cu130, vendored pywhispercpp). After any uv sync, reinstall the ROCm stack; otherwise the venv silently goes back to CUDA. One command:

scripts/install-rocm.sh

It fetches the torch ROCm wheels and the CTranslate2 ROCm wheel. The manual equivalent, if the script cannot run on this host:

TMPDIR=$HOME/.tmp .venv/bin/pip install --force-reinstall torch torchvision torchaudio \
  --index-url https://download.pytorch.org/whl/rocm7.2 \
  --extra-index-url https://pypi.org/simple
TMPDIR=$HOME/.tmp .venv/bin/pip install --force-reinstall ~/.tmp/ct2-rocm/ctranslate2-*-cp313-*.whl
  • Use the CTranslate2 ROCm wheel (release v4.8.4 rocm-python-wheels-Linux.zip; the script fetches it); PyPI's ctranslate2 is CUDA-only.
  • Plain uv run re-syncs the venv implicitly, so it also reverts these wheels (uv sync is not required to lose them). After installing the ROCm stack, run the suite as .venv/bin/python -m pytest (or uv run --no-sync ...), and re-run scripts/install-rocm.sh after any deliberate sync.
  • On HIP the default hipMallocAsync allocator silently corrupts CTranslate2 buffers (OpenNMT/CTranslate2 #2090): wrong language detection, empty transcripts, hangs, or segfaults. The transcriber sets CT2_CUDA_ALLOCATOR=cub_caching automatically on ROCm hosts (CTranslate2 4.8.4 makes it the HIP default); an explicit user-set value is respected.
  • STT backend: [whisper] backend = "faster-whisper", device = "auto", compute_type = "int8". auto resolves to the discrete R9700 (verified by VRAM delta) while the integrated GPU stays idle, and auto language detection works on ROCm once the allocator is configured. The vendored pywhispercpp wheel is CUDA-only and cannot import on this host.
  • Do not set HIP_VISIBLE_DEVICES=0 for CTranslate2 — it selects the wrong device and transcribes garbage. Leave both devices visible; device = "auto" picks the R9700.
  • TMPDIR=$HOME/.tmp for pip/uv: /tmp is a 1 GiB tmpfs and large wheels fail with ENOSPC.

Process architecture (don't break this)

Three processes wired by UNIX sockets under $XDG_RUNTIME_DIR (fallback /tmp):

  • stt-daemon (cli.py) — capture → pre-roll/VAD → segment → transcribe → stdout + JSONL. Publish-only on stt-daemon.sock: it never accepts inbound messages; lifecycle is signals only.
  • stt-assistant (assistant/runner.py) — subscribes to that bus, owns all state (modes, command buffer, chat history, LLM, tools, audit), serves stt-assistant.sock.
  • stt-tui / stt-ctl — clients of the assistant control socket only.

stt-daemon up [--with-tui] (supervisor.py) launches these as child processes. With --with-tui, TUI startup is mandatory (no silent headless fallback: socket timeout or early child exit kills all), and the supervisor restores the tty's ISIG on exit so a killed TUI doesn't leave Ctrl+C dead.

Stateful logic belongs in the assistant (the speaker embedding lane is the daemon-side exception: it owns the prototype store). Four single-worker-thread lanes exist by design, and none may be parallelized: transcription (faster-whisper isn't thread-safe), the speaker embedding lane (in-process in stt-daemon: src/stt_daemon/speakers.py + src/stt_daemon/embedder.py, owns the SpeakerStore, drains the rename inbox), all LLM calls (assistant/inference.py InferenceQueue; commands submit with priority -1 to preempt chat), and the Laya triage lane (assistant/laya_lane.py: one background thread plus one loader, running in-process in the assistant). Neither the speaker embedding lane nor the Laya lane is a process or a socket, and neither changes supervisor spawn counts (2 without the TUI, 3 with it); the daemon socket stays publish-only.

stt-daemon with no subcommand runs the daemon (legacy --check / --list-sources still work).

Post-conference reconstruction (session-summary)

After a session, the session-summary skill turns transcripts/<session>/ into the files worth sharing. It is an agent run, not a process: no console script, no socket, no effect on supervisor spawn counts. Full reference: docs/SESSION-SUMMARY.md.

  • Evidence order: transcripts.jsonl is the truth; brief.md is low-trust input that the skill rewrites. Never modify transcripts.jsonl or audio/, and work on one session directory only (no cross-session reads; shared knowledge lives in the skill's glossary, which is gitignored).
  • Outputs, all inside the session directory: session-summary.svg, session-summary.png, session-summary.content.json, the .es. trio, and a corrected brief.md. A run is done only when both SVGs and both PNGs exist; a ~1 MB PNG means quantization did not run.
  • scripts/render.py <session> <content.json> <out.svg> [--no-png] renders and validates the SVG and exports the PNG via Inkscape + ImageMagick, both required on PATH. --no-png is for drafts only — never hand-write or hand-export the SVG.
  • .opencode/ is gitignored except the session-summary skill, which is tracked repo tooling (see the progressive un-ignore in .gitignore) — with one exception inside it: references/glossary.md is private client vocabulary and stays ignored. Nothing the skill writes is committed: session directories stay ignored with transcripts/.

Config

  • Loaded from config.toml in the current working directory only (no env var, no XDG lookup); --config overrides. A missing file means dataclass defaults.
  • config.example.toml is the shipped template (values only); docs/CONFIGURATION.md is the annotated reference. Keep the dataclass, example, and docs in sync when adding a key.
  • Mapping is manual — no pydantic/dynaconf. Adding a key means editing the dataclass and load_config() in config.py. Unknown top-level sections warn and are ignored; unknown keys inside a known section are silently ignored (dict.get per field).
  • Env vars: STT_DAEMON_STATE (provisioning state; default audio.json in the cwd), XDG_RUNTIME_DIR (sockets), HF_HOME (Whisper cache, honored by faster-whisper/huggingface).
  • [triage] is the display-only signal panel, off by default (enabled = false). Its questions_file (questions.json in the cwd) is private user state and is gitignored, like config.toml. dump_io = true writes one JSON per triage under troubleshoot/triage/ (also gitignored); a coalesced brief update writes two files. The brief engine is driven by [llm].triage_system_prompt and returns two plain-text sections (NEW FACTS: + BRIEF:); code owns the append-only FactLedger (dedupe/eviction/render, ledger_max_facts), so the model never rewrites recorded facts. The optional fail-open substantive_update gate is brief_gate_enabled (default true); periodic_s re-runs triage every N seconds while speech accumulates without a settled turn.
  • [speaker] is opt-in per-utterance speaker identification, off by default, backed by the SpeakerConfig dataclass (12th known section; 14 keys: enabled, model_path, provider, threshold_id, threshold_new, min_enroll_s, rms_floor, margin, mint_cooldown_s, confirm_s, max_prototypes, provisional_ttl_h, store_dir, join_timeout_s). Needs uv sync --extra speaker plus a model under models/. Full reference: docs/SPEAKERS.md.
  • [brief] is opt-in durable-brief persistence, backed by the BriefConfig dataclass (13th known section; 1 key: write (default false, gates disk persistence only)). The writer defaults to brief/brief.md (cwd-relative, gitignored; directory/filename are BriefFile ctor defaults, not config keys) and set_directory() repoints it at the active session folder; with write = false the brief stays in the TUI only. Full reference: docs/TRIAGE.md.
  • Never treat the live config.toml as a source of defaults; it is gitignored user state.

Transcription backends

  • [whisper].backend picks faster-whisper (default) or pywhispercpp through build_transcriber() in transcriber.py. device/compute_type apply to faster-whisper only; pywhispercpp ignores them (GPU comes from the built wheel, quantization from the ggml model).
  • pywhispercpp is a local, uncommitted CUDA build: vendor/.gitignore is *, the cp313-only wheel is not tracked, and [tool.uv.sources] points at that path. A fresh clone on Python 3.13 fails uv sync until the wheel is rebuilt — see docs/PYWHISPERCPP.md. Never delete vendor/.

Testing

  • tests/ has 1038 tests, all hardware- and network-free. Preserve that: inject fakes instead of touching real pactl, audio devices, an LLM, or sherpa_onnx (the speaker lane uses a fake embedder; no ONNX in tests).
  • Key seams: Daemon(...) accepts capture_factory, vad_factory, transcriber_factory, embedder_factory, sink_factory; PulseAudio takes an injectable runner; AssistantService takes time_fn, executor, audit.
  • Speaker coverage lives in tests/test_speakers.py, tests/test_daemon_speaker.py, tests/test_assistant_speaker.py, tests/test_enroll_assist.py (the offline seed helper src/stt_daemon/enroll_assist.py), and tests/test_evaluate.py (the speaker-eval scorer src/stt_daemon/evaluate.py).
  • No conftest.py — per-file fakes are the convention. Tests are the behavioral spec; read them before changing behavior.
  • The default suite is green without the optional triage and speaker extras. Install triage (uv sync --extra triage) only to exercise the Laya measurement test, which otherwise skips; speaker (uv sync --extra speaker) is needed only for real ONNX embeddings, never the suite.

Conventions & gotchas

  • Never commit gitignored runtime output: config.toml, transcripts/, audio.json, audio/, troubleshoot/ (audit/debug/triage), brief/, speakers/ (biometric), models/, .omo/, questions.json, and .opencode/skills/session-summary/references/glossary.md (client vocabulary).
  • New assistant tools: register in assistant/tools.py. Model-supplied arguments are JSON-parsed and schema-validated before running; set side_effect and requires_confirmation for anything non-read-only.
  • Whisper models must be multilingual — do not use .en variants (defaults auto-detect es/en).
  • docs/ holds the full references (config, CLI, audio setup, assistant, speaker identification, output, troubleshooting, development); .omo/plans/ holds local design plans (gitignored).