# Speaker Identification Speaker identification labels each transcribed utterance with who spoke it. A dedicated in-process sherpa-onnx embedding lane maps the clip to a vector, and the vector is matched against a local store of per-speaker prototypes. The label then rides the transcript to the TUI header and into the JSONL record. There is no separate process, no socket, and no change to the supervisor's child count. The feature is off by default (`[speaker].enabled = false`), and the extra it needs is opt-in, so a plain `uv sync` and the existing test suite behave exactly as before. When it is off, or the model is missing, or the library fails to load, the feature no-ops: no lane is built, no store is touched, the daemon still runs, and every record carries `speaker_status = "none"`. ## What the transcript shows - **TUI header.** The speaker sits between the language tag and the duration, for example `10:01:03 EN Efren 3.2s pause`. Named speakers render in bold green; every other speaker, including provisionals and `Unknown`, renders in bold red. The literal `Unknown` is shown for unmatched turns. - **JSONL.** Existing records are enriched in place; there is one append-only file and no separate annotated file. Five additive, back-compatible fields are added: `seg_id`, `speaker_id`, `speaker_status`, `speaker`, and `speaker_score`. All are `null` or `"none"` when the feature is off or the turn could not be identified. See [OUTPUT.md](OUTPUT.md) for the exact field semantics. ## How matching works A speaker profile is a set of L2-normalized embedding prototypes, capped at `max_prototypes` (8). A turn's score is the maximum cosine similarity over all of that speaker's prototypes, never a centroid. The score lands the turn in one of four status bands: | status | condition | behavior | |---|---|---| | `matched` | score >= `threshold_id` (`0.60`) | Reports the matched speaker; may adapt the profile. | | `uncertain` | `threshold_new` (`0.45`) <= score < `0.60` | Reports the top-1 speaker with a `?` suffix; never enrolls and never adapts. | | `unknown` | score < `0.45` | No label; eligible for auto-enrollment after the guards pass. | | `none` | no embedding computed | Feature off, audio too short, or an error. | The two thresholds and `margin` are starting points, not measured values. They are **model- and channel-dependent**, so calibrate them on real far-field audio from the actual microphone and room before trusting the labels, and use the per-record `speaker_score` to tune. `speaker_status` is the coarse answer; the score is what you optimize against. ## Auto-enrollment and promotion An unknown voice is minted as a provisional `spk_NNNN` only after every guard passes: - Duration: the utterance is at least `min_enroll_s` (2.0 s). - Level: the utterance is at or above `rms_floor` (default `0.0`, i.e. no gate). - Hysteresis: two consecutive mutually-similar misses, so a single odd clip cannot mint. - Cooldown: no second mint within `mint_cooldown_s` (60 s). - Margin: the best prototype beats the second best by at least `margin` (0.04). A provisional profile is promoted to `confirmed` once it accumulates `confirm_s` (12 s) of speech. Until then it is labeled with its provisional id. A provisional that never confirms is pruned after `provisional_ttl_h` (24 h). `stt-ctl speaker-name` names a provisional, and naming it deletes the rolling sample that was kept for the enrollment decision. The name is then shown everywhere the speaker appears. ## Store layout and privacy Everything lives under `store_dir` (`speakers`), relative to the working directory, and is gitignored: ``` speakers/ speakers.json # roster: ids, names, status, cumulative speech seconds prototypes.npz # normalized embedding prototypes per speaker samples/ # rolling .wav samples held for enrollment decisions inbox/ # rename requests the daemon applies rejected/ # malformed requests ``` Directories are created `0700` and sample WAVs `0600`. Samples are deleted as soon as a profile is named or pruned. **Embeddings and samples are biometric data.** Treat `speakers/` as private user state, not as a shareable artifact. Identity here is a recoverable cache, not an archive: re-enrolling restores it. ## Commands List the roster (id, name, status, cumulative seconds, sample path): ```bash uv run stt-ctl speaker-list ``` Name a provisional speaker: ```bash uv run stt-ctl speaker-name spk_0001 Efren ``` Naming writes a rename request to `speakers/inbox/`, which the daemon applies on its next tick; an unknown id returns an error. The assistant caches the roster from `speakers` bus events, so a name that has not yet flowed through an event can be reported as `unknown_speaker` (see [TROUBLESHOOTING.md](TROUBLESHOOTING.md)). Renaming a speaker that already has a name overwrites it. There is no `delete` or `merge` in v1. ## Install and enable The dependency lives behind an optional extra: ```bash # 1. Install the speaker extra (adds sherpa-onnx) uv sync --extra speaker # 2. Download the embedding model into the gitignored models/ directory curl -L -o models/nemo_en_titanet_small.onnx \ "https://github.com/k2-fsa/sherpa-onnx/releases/download/speaker-recongition-models/nemo_en_titanet_small.onnx" # 3. Point config.toml at it and turn it on # [speaker] # enabled = true # model_path = "models/nemo_en_titanet_small.onnx" ``` The extra pulls two distributions, `sherpa-onnx` (the Python API) and `sherpa-onnx-core` (the wheel that bundles the native `libonnxruntime.so`). Extras are exclusive under `uv sync`: `uv sync --extra speaker` matches the default deps plus `speaker` exactly, so it removes the `triage` extra and its stack (`laya`, `torch`, `transformers`, ...). To keep both, request both: `uv sync --extra speaker --extra triage`. The model is `nemo_en_titanet_small.onnx`, 38.4 MB, 192-dimensional, CC-BY-4.0, and not gated. The `models/` directory is gitignored, so the download is local-only. Note the upstream release tag's spelling, `speaker-recongition-models`; the typo is in the URL and must be copied verbatim. The model is CPU by design (`provider = "cpu"`, `num_threads = 1`). `provider = "cuda"` requires a CUDA-enabled sherpa wheel; without one it warns and falls back to `cpu`. The installed `onnxruntime` exposes only `CPUExecutionProvider`, so `cuda` changes nothing without a different wheel. If the Spanish/English quality is not enough for your room, two larger sherpa models are drop-in alternatives: | model | size | dim | |---|---|---| | `3dspeaker_speech_campplus_sv_en_voxceleb_16k.onnx` | 28 MB | 192 | | `wespeaker_en_voxceleb_resnet34_LM.onnx` | 25 MB | 256 | Same download host and URL shape; only the filename changes. ## Pre-enroll from a WAV file Enrollment is reactive by default: the daemon mints a provisional `spk_NNNN` on an unknown live voice, and you name it afterwards with `stt-ctl speaker-name`. To register a known speaker up front from clips you already have, run `stt-daemon speaker-enroll` with one or more WAV files: ```bash uv run stt-daemon speaker-enroll --name Efren \ --wav transcripts/2026-09-28-14-30-22/audio/143022-123-pause.wav Repeat `--wav` to enroll from several clips; each file contributes one prototype, capped at `max_prototypes` with the oldest evicted: ```bash uv run stt-daemon speaker-enroll --name Efren \ --wav transcripts/2026-09-28-14-30-22/audio/143022-123-pause.wav \ --wav transcripts/2026-09-28-14-30-22/audio/150210-456-end_of_turn.wav ``` On success the command prints `enrolled 'Efren' as spk_0002 (2 clip(s), 7.4s)`. The `--name` and `--wav` flags are required; `--force` bypasses the running-daemon refusal, and `--config` is the same global override as every other subcommand. Each WAV must be **16 kHz mono 16-bit PCM**. No resampling is performed: stereo, a different sample rate, or a non-WAV file is rejected with a clear error. The clips the daemon saves under `//audio/` (default `transcripts/…`) when `[output].save_audio = true` are already in that format, so they need no conversion. Each clip must also be at least `[speaker].min_enroll_s` seconds (default `2.0`); shorter clips are rejected. This is the "is it enough audio?" check. The profile is created `confirmed` with the given name, and unlike a named provisional it stores no sample WAV; samples exist only for the listen-before-naming flow. Explicit enrollment bypasses the automatic mint guards (duration, level, consecutive-miss, cooldown, and margin), because you are asserting the identity rather than inferring it; the CLI's duration gate is the only length check. Re-enrolling an existing name appends a prototype to that same speaker rather than creating a duplicate. Enrollment does **not** need `[speaker].enabled = true`; you can pre-enroll before turning the feature on. It does need the model and the extra: `uv sync --extra speaker` and `[speaker].model_path` set. The store is single-writer and owned by the daemon. **Stop the daemon before enrolling**, or the in-memory store can overwrite your change on its next save. The command detects the daemon's event socket and refuses unless `--force` is passed. Storage is unchanged: `speakers/speakers.json` and `speakers/prototypes.npz` under `store_dir`, gitignored and biometric. ## Assisted enrollment from a seed clip Enrolling from one long clip produces weak, polluted prototypes: a long `pause` segment can hold several speakers, so its embedding lands somewhere between them. `speaker-enroll-seed` takes ONE clip you are certain is the speaker, finds the clips in the audio directory that are most similar to it, and lets you review each one before it joins the profile: ```bash uv run stt-daemon speaker-enroll-seed --name "Efren Yevale Varela" --seed clips/efren-seed.wav ``` Flags: - `--name` (required) — speaker display name. - `--seed` (required) — a 16 kHz mono 16-bit WAV known to be that speaker. - `--count N` (default `5`) — how many reviewed clips to accept. - `--min-score F` (default `0.55`) — minimum cosine to the seed for a clip to be offered. - `--audio-dir DIR` (default `[output].session_dir`, scanned recursively) — where to scan for `*.wav`. - `--max-scan N` (default `0` = all) — cap the number of files embedded, useful on large dirs. - `--dedupe-threshold F` (default `0.98`) — cosine at or above which a candidate counts as already covered by one of that speaker's prototypes. - `--other-threshold F` (default `0.45`) — cosine at or above which a candidate is flagged as scoring against a different enrolled speaker. - `--yes` — non-interactive: auto-accept the top `--count` candidates. - `--no-play` — do not attempt audio playback during review. - `--force` — enroll even when a daemon appears to be running. - `--config` — the same global override as every subcommand. The flow: 1. Loads config and refuses if a daemon appears to be running (its event socket exists) unless `--force` is passed. 2. Requires the model and `uv sync --extra speaker`, exactly like `speaker-enroll`. 3. Fails with a clear error if the stored prototype dimension differs from the model's: the store was built with a different model, so wipe `store_dir` and re-enroll. 4. Scans the audio directory, embeds each clip, ranks by cosine similarity to the seed, and keeps clips at least `[speaker].min_enroll_s` seconds long and at or above `--min-score`. 5. Reviews each candidate interactively (unless `--yes`), showing `[i/N] (score X.XX, Y.Ys)` with four actions: **Accept**, **Skip** (show the next candidate), **Play** (listen, then decide again), or **Quit** (stop, enroll nothing). 6. Enrolls the seed (when it is at least `min_enroll_s`) plus the accepted clips, then prints something like `enrolled 'Efren Yevale Varela' as spk_0001 from seed: 4 clip(s) (seed + 3 reviewed, 16.9s, 4 prototype(s))`. If the seed is shorter than `min_enroll_s` it is used only as the query, and a warning is printed. If nothing is accepted, or the store would exceed `max_prototypes` (an eviction warning), the command says so and no partial write happens. **Review warnings.** Each candidate is flagged during review: - `duplicate` — its embedding is already covered by one of that speaker's existing prototypes (cosine at or above `--dedupe-threshold`, default `0.98`). - `other-speaker:` — it scores highly against a **different** enrolled speaker's profile (cosine at or above `--other-threshold`, default `0.45`), which is a contamination hint. Choosing **Accept** on a `duplicate` is refused: it prints `skipping : already covered by this profile` and moves on without consuming the accept budget. An `other-speaker:` flag does not block the accept; it is a hint to listen harder. **Duplicate removal.** Both commands guarantee no duplicates are added. The incoming set (seed plus accepted clips) is greedily de-duplicated before enrolling, printed as `skipped N duplicate clip(s) already covered ...`, and after enrolling the command removes any pre-existing near-duplicate prototypes of that speaker, printed as `removed N duplicate prototype(s) from the ...`. Plain `speaker-enroll` removes pre-existing duplicates for the named speaker too, and reports it. The semantics are the same in both commands: for each speaker, prototypes are walked in insertion order and the **first of each cluster** is kept; a prototype is dropped when its cosine to an already-kept prototype of the **same** speaker is at or above the threshold. The first prototype is always kept. This is what cleans up the store when several near-identical clips were enrolled. **Abort semantics.** Pressing **Quit** or Ctrl+C during review writes nothing. The command prints the clips accepted so far and an equivalent manual `stt-daemon speaker-enroll --wav ...` command, so you can run the explicit form yourself. **Playback.** Review tries `paplay`, then `aplay`, then `ffplay`. If none exist, review continues without sound; `--no-play` skips playback entirely. **Non-interactive.** A non-TTY without `--yes` errors out; it never hangs waiting for input. Prefer short `end_of_turn` clips for enrollment. Long `pause` or `end_of_turn` segments can hold more than one speaker, which pollutes a prototype; that is exactly why the helper makes you listen and review each clip. The plain `speaker-enroll --wav ...` command still exists for enrolling an explicit clip list you already trust, as described in [Pre-enroll from a WAV file](#pre-enroll-from-a-wav-file). ## Tuning an existing profile Re-running `speaker-enroll-seed` or `speaker-enroll` for a name that already exists does **not** reset the profile. It reuses the same profile id (`spk_NNNN`) and **appends** the new clips as additional prototypes, and `cumulative_s` keeps growing. Tuning is therefore additive: you steer a profile by adding good clips, and you flush a bad one by adding enough new clips to push the oldest out. **Broaden coverage.** Add several diverse short `end_of_turn` clips, in the 1-5 s range you actually speak (enrollment still requires each clip to clear `min_enroll_s`, so the usable ones start at 2 s by default), varying phrase, level, and session. More spread across the ways you really sound means short live utterances score higher. This is what fixed a real case where a 1.74 s utterance landed below `threshold_new`: the profile held one narrow prototype, and a handful of varied short clips lifted the live score into range. **Flush a bad or polluted prototype.** Eviction is oldest-first (FIFO), so adding new clips pushes the oldest ones out. Each speaker holds at most `max_prototypes` (default `8`). To evict the oldest one, add enough clips to exceed the cap: at 7 of 8, adding **2** more drops the oldest, i.e. `max_prototypes - current + 1` additions; to evict the **N** oldest, add `max_prototypes - current + N`. There is no per-prototype delete and no replace. Duplicates, though, are handled for you: both enrollment commands drop near-duplicate prototypes automatically and the review flags them as `duplicate`, so flushing is the way to remove a bad prototype that is not a duplicate, short of wiping the store. **`max_prototypes` is itself a lever.** Raising it (e.g. `16`) keeps more prototypes before eviction, giving broader coverage at a marginal cost in false-match risk. Lowering it forces eviction sooner and keeps the profile tighter. **What to accept.** Listen with **Play**. Accept clean single-speaker clips; **skip anything with a second voice**, which is the whole point of the review. Long `pause` segments are the usual offenders. Near-duplicate clips are flagged `duplicate` and cannot be accepted, and both commands strip near-duplicates from the profile automatically, so you rarely need to worry about them. **What tuning cannot do.** It cannot delete a specific prototype or reorder them, and it does not change thresholds: `threshold_id` and `threshold_new` are config, not the helper. `--yes` skips the review entirely and auto-accepts the top `--count` candidates, so it cannot catch a clip containing a second voice; do **not** use `--yes` for tune-ups. **Verify.** Restart the daemon so it loads the updated store, then speak and watch the TUI header (your name vs `Unknown`). With `[output].jsonl = true`, check the `speaker`, `speaker_status`, and `speaker_score` fields. Repeatedly `uncertain` or `unknown` for your own voice means the profile needs more or more diverse clips. A confident **wrong** name usually means overlapping speakers in the enrollment clips, or a genuinely hard voice pair. ## Measuring accuracy (speaker-eval) Prototype geometry tells you the distances between your own enrolled clips, not whether the matcher gets real utterances right. `speaker-eval` scores the store against labeled clips and prints a plain-text report, so you get a real accuracy number instead of guessing: ```bash uv run stt-daemon speaker-eval --labeled labeled/ ``` `--labeled DIR` is required. Each immediate subdirectory is a speaker **display name** and holds that person's `*.wav` clips, 16 kHz mono 16-bit PCM like enrollment: ``` labeled/ Efren Yevale Varela/ 001.wav 002.wav Nombre Apellido/ 001.wav ``` The command embeds every clip, labels it with the same matcher the daemon uses, and prints `total`, `accuracy`, `correct`, `wrong`, `uncertain (top-1 correct/total)`, and `unknown`; a `per-speaker` table (`name | n | correct | acc`); the top `confusions` as `true -> predicted (count)`, up to 10; and `margins` with the `count`, `mean`, and `p10` of the top1 - top2 score gap over prototypes. The run is read-only: it never writes the store, so the daemon does not need to be stopped. It does need the model and `uv sync --extra speaker`, exactly like enrollment. If the store was built with a different embedding model (dimension mismatch), it fails clearly instead of scoring nonsense: wipe `store_dir` and re-enroll. To use it as a tuning gate, pass `--min-accuracy F`, which exits non-zero when measured accuracy is below `F`: ```bash uv run stt-daemon speaker-eval --labeled labeled/ --min-accuracy 0.9 ``` Label a handful of clips you are sure about per speaker. The accuracy and the confusion pairs are the fastest way to see whether a profile needs more diverse clips, whether two voices genuinely collide, or whether the thresholds need retuning. ## Wipe and rebuild Wipe the store when a profile is polluted beyond easy flushing, when you switch embedding models (the store must be rebuilt if the model dimension changes), or when the roster got messy. Wiping removes **all** speakers. To remove just one, use the manual purge below, or flush it with added clips as in [Tuning an existing profile](#tuning-an-existing-profile). 1. **Stop the daemon.** The store is single-writer. 2. **Delete the store:** ```bash rm -rf speakers/ ``` `speakers/` is gitignored and a recoverable cache; `models/` is untouched. 3. **Re-enroll each speaker** from one clip you are sure of, reviewing the suggestions: ```bash uv run stt-daemon speaker-enroll-seed --name "Nombre Apellido" --seed path/to/a-clip-you-trust.wav --count 5 ``` Repeat per speaker. The plain `speaker-enroll --wav ...` form works too when you have an explicit list. The next run rebuilds an empty store the moment it finds none. 4. **Restart the daemon** so it loads the fresh store. This is the broad version of the manual purge below, which removes a single speaker instead of the whole roster. ## Configuration reference All keys live under `[speaker]`. The defaults are what you get with no config file. `[speaker]` is the 12th known section; unknown keys inside it are silently ignored, like every other section. | key | default | meaning | |---|---|---| | `enabled` | `false` | Master switch. `false` means no lane, no store, and every record `speaker_status = "none"`. | | `model_path` | `null` | Path to the sherpa-onnx embedding `.onnx`. Unset or missing disables the feature. | | `provider` | `"cpu"` | `cpu` or `cuda`. `cuda` needs a CUDA-enabled sherpa wheel, else warns and falls back to `cpu`. | | `threshold_id` | `0.60` | `T_id`: cosine score at or above which a turn matches an existing prototype. | | `threshold_new` | `0.45` | `T_new`: cosine score below which a turn may mint a new provisional; the lower edge of `uncertain`. | | `min_enroll_s` | `2.0` | Mint/adapt duration gate: shorter utterances never create or adapt a profile. | | `rms_floor` | `0.0` | Mint/adapt level gate: quieter utterances never create or adapt a profile. | | `margin` | `0.04` | Best-2nd margin required before an adapt or mint decision. | | `mint_cooldown_s` | `60.0` | Minimum spacing between mints. | | `confirm_s` | `12.0` | Cumulative speech seconds a provisional needs to be promoted to `confirmed`. | | `max_prototypes` | `8` | Cap on stored prototypes per speaker. | | `provisional_ttl_h` | `24.0` | Prune provisionals that never confirm within this many hours, checked at startup. | | `store_dir` | `"speakers"` | Store root, cwd-relative; gitignored because it holds biometric artifacts. | | `join_timeout_s` | `2.0` | Embedding-lane join timeout before the transcript is emitted. | ## The one-label-per-segment limitation Identification runs on the whole VAD segment and produces **one speaker label per segment**. If two people speak within one segment, the clip yields a single blended embedding, which usually lands in `uncertain` or `unknown`; such clips are never enrolled and never adapted. Natural turn-taking, where a pause separates the speakers, splits into separate segments and labels each one correctly. If mixed-speaker segments are common in your audio, lower `[vad].silence_ms` (within reason) so pauses break turns sooner, or accept that overlapping speech is out of scope for v1. ## Manual purge `delete` and `merge` are not supported in v1. To remove a speaker, stop the daemon, remove the id from `speakers/speakers.json` and its rows from `speakers/prototypes.npz`, then restart. To start clean, stop the daemon and wipe `speakers/` entirely; the next run rebuilds an empty store. Because identity is a recoverable cache, re-enrollment restores it. ## Troubleshooting For a missing `sherpa-onnx` install, a missing model, everything showing `Unknown`, naming that reports `unknown_speaker`, or resetting the store, see the Speaker identification section of [TROUBLESHOOTING.md](TROUBLESHOOTING.md).