STT-Assistant/docs/SPEAKERS.md

Speaker Identification

Speaker identification labels each transcribed utterance with who spoke it. A dedicated in-process sherpa-onnx embedding lane maps the clip to a vector, and the vector is matched against a local store of per-speaker prototypes. The label then rides the transcript to the TUI header and into the JSONL record. There is no separate process, no socket, and no change to the supervisor's child count.

The feature is off by default ([speaker].enabled = false), and the extra it needs is opt-in, so a plain uv sync and the existing test suite behave exactly as before. When it is off, or the model is missing, or the library fails to load, the feature no-ops: no lane is built, no store is touched, the daemon still runs, and every record carries speaker_status = "none".

What the transcript shows

• TUI header. The speaker sits between the language tag and the duration, for example 10:01:03 EN Efren 3.2s pause. Named speakers render in bold green; every other speaker, including provisionals and Unknown, renders in bold red. The literal Unknown is shown for unmatched turns.
• JSONL. Existing records are enriched in place; there is one append-only file and no separate annotated file. Five additive, back-compatible fields are added: seg_id, speaker_id, speaker_status, speaker, and speaker_score. All are null or "none" when the feature is off or the turn could not be identified. See OUTPUT.md for the exact field semantics.

How matching works

A speaker profile is a set of L2-normalized embedding prototypes, capped at max_prototypes (8). A turn's score is the maximum cosine similarity over all of that speaker's prototypes, never a centroid. The score lands the turn in one of four status bands:

status	condition	behavior
matched	score >= threshold_id (0.60)	Reports the matched speaker; may adapt the profile.
uncertain	threshold_new (0.45) <= score < 0.60	Reports the top-1 speaker with a ? suffix; never enrolls and never adapts.
unknown	score < 0.45	No label; eligible for auto-enrollment after the guards pass.
none	no embedding computed	Feature off, audio too short, or an error.

The two thresholds and margin are starting points, not measured values. They are model- and channel-dependent, so calibrate them on real far-field audio from the actual microphone and room before trusting the labels, and use the per-record speaker_score to tune. speaker_status is the coarse answer; the score is what you optimize against.

Auto-enrollment and promotion

An unknown voice is minted as a provisional spk_NNNN only after every guard passes:

• Duration: the utterance is at least min_enroll_s (2.0 s).
• Level: the utterance is at or above rms_floor (default 0.0, i.e. no gate).
• Hysteresis: two consecutive mutually-similar misses, so a single odd clip cannot mint.
• Cooldown: no second mint within mint_cooldown_s (60 s).
• Margin: the best prototype beats the second best by at least margin (0.04).

A provisional profile is promoted to confirmed once it accumulates confirm_s (12 s) of speech. Until then it is labeled with its provisional id. A provisional that never confirms is pruned after provisional_ttl_h (24 h). stt-ctl speaker-name names a provisional, and naming it deletes the rolling sample that was kept for the enrollment decision. The name is then shown everywhere the speaker appears.

Store layout and privacy

Everything lives under store_dir (speakers), relative to the working directory, and is gitignored:

speakers/
  speakers.json          # roster: ids, names, status, cumulative speech seconds
  prototypes.npz         # normalized embedding prototypes per speaker
  samples/               # rolling <id>.wav samples held for enrollment decisions
  inbox/                 # rename requests the daemon applies
    rejected/            # malformed requests

Directories are created 0700 and sample WAVs 0600. Samples are deleted as soon as a profile is named or pruned. Embeddings and samples are biometric data. Treat speakers/ as private user state, not as a shareable artifact. Identity here is a recoverable cache, not an archive: re-enrolling restores it.

Commands

List the roster (id, name, status, cumulative seconds, sample path):

uv run stt-ctl speaker-list

Name a provisional speaker:

uv run stt-ctl speaker-name spk_0001 Efren

Naming writes a rename request to speakers/inbox/, which the daemon applies on its next tick; an unknown id returns an error. The assistant caches the roster from speakers bus events, so a name that has not yet flowed through an event can be reported as unknown_speaker (see TROUBLESHOOTING.md). Renaming a speaker that already has a name overwrites it. There is no delete or merge in v1.

Install and enable

The dependency lives behind an optional extra:

# 1. Install the speaker extra (adds sherpa-onnx)
uv sync --extra speaker

# 2. Download the embedding model into the gitignored models/ directory
curl -L -o models/nemo_en_titanet_small.onnx \
  "https://github.com/k2-fsa/sherpa-onnx/releases/download/speaker-recongition-models/nemo_en_titanet_small.onnx"

# 3. Point config.toml at it and turn it on
#    [speaker]
#    enabled = true
#    model_path = "models/nemo_en_titanet_small.onnx"

The extra pulls two distributions, sherpa-onnx (the Python API) and sherpa-onnx-core (the wheel that bundles the native libonnxruntime.so). Extras are exclusive under uv sync: uv sync --extra speaker matches the default deps plus speaker exactly, so it removes the triage extra and its stack (laya, torch, transformers, ...). To keep both, request both: uv sync --extra speaker --extra triage.

The model is nemo_en_titanet_small.onnx, 38.4 MB, 192-dimensional, CC-BY-4.0, and not gated. The models/ directory is gitignored, so the download is local-only. Note the upstream release tag's spelling, speaker-recongition-models; the typo is in the URL and must be copied verbatim.

The model is CPU by design (provider = "cpu", num_threads = 1). provider = "cuda" requires a CUDA-enabled sherpa wheel; without one it warns and falls back to cpu. The installed onnxruntime exposes only CPUExecutionProvider, so cuda changes nothing without a different wheel.

If the Spanish/English quality is not enough for your room, two larger sherpa models are drop-in alternatives:

model	size	dim
3dspeaker_speech_campplus_sv_en_voxceleb_16k.onnx	28 MB	192
wespeaker_en_voxceleb_resnet34_LM.onnx	25 MB	256

Same download host and URL shape; only the filename changes.

Pre-enroll from a WAV file

Enrollment is reactive by default: the daemon mints a provisional spk_NNNN on an unknown live voice, and you name it afterwards with stt-ctl speaker-name. To register a known speaker up front from clips you already have, run stt-daemon speaker-enroll with one or more WAV files:

uv run stt-daemon speaker-enroll --name Efren \
  --wav transcripts/2026-09-28-14-30-22/audio/143022-123-pause.wav

Repeat `--wav` to enroll from several clips; each file contributes one prototype, capped at
`max_prototypes` with the oldest evicted:

bash uv run stt-daemon speaker-enroll --name Efren \ --wav transcripts/2026-09-28-14-30-22/audio/143022-123-pause.wav \ --wav transcripts/2026-09-28-14-30-22/audio/150210-456-end_of_turn.wav

On success the command prints `enrolled 'Efren' as spk_0002 (2 clip(s), 7.4s)`. The `--name`
and `--wav` flags are required; `--force` bypasses the running-daemon refusal, and `--config`
is the same global override as every other subcommand.

Each WAV must be **16 kHz mono 16-bit PCM**. No resampling is performed: stereo, a different
sample rate, or a non-WAV file is rejected with a clear error. The clips the daemon saves
under `<session_dir>/<YYYY-MM-DD-HH-MM-SS>/audio/` (default `transcripts/…`) when
`[output].save_audio = true` are already in that format, so
they need no conversion. Each clip must also be at least `[speaker].min_enroll_s` seconds
(default `2.0`); shorter clips are rejected. This is the "is it enough audio?" check.

The profile is created `confirmed` with the given name, and unlike a named provisional it
stores no sample WAV; samples exist only for the listen-before-naming flow. Explicit
enrollment bypasses the automatic mint guards (duration, level, consecutive-miss, cooldown,
and margin), because you are asserting the identity rather than inferring it; the CLI's
duration gate is the only length check. Re-enrolling an existing name appends a prototype to
that same speaker rather than creating a duplicate.

Enrollment does **not** need `[speaker].enabled = true`; you can pre-enroll before turning
the feature on. It does need the model and the extra: `uv sync --extra speaker` and
`[speaker].model_path` set.

The store is single-writer and owned by the daemon. **Stop the daemon before enrolling**, or
the in-memory store can overwrite your change on its next save. The command detects the
daemon's event socket and refuses unless `--force` is passed. Storage is unchanged:
`speakers/speakers.json` and `speakers/prototypes.npz` under `store_dir`, gitignored and
biometric.

## Assisted enrollment from a seed clip

Enrolling from one long clip produces weak, polluted prototypes: a long `pause` segment can hold
several speakers, so its embedding lands somewhere between them. `speaker-enroll-seed` takes ONE
clip you are certain is the speaker, finds the clips in the audio directory that are most similar
to it, and lets you review each one before it joins the profile:

bash uv run stt-daemon speaker-enroll-seed --name "Efren Yevale Varela" --seed clips/efren-seed.wav

Flags:

- `--name` (required) — speaker display name.
- `--seed` (required) — a 16 kHz mono 16-bit WAV known to be that speaker.
- `--count N` (default `5`) — how many reviewed clips to accept.
- `--min-score F` (default `0.55`) — minimum cosine to the seed for a clip to be offered.
- `--audio-dir DIR` (default `[output].session_dir`, scanned recursively) — where to scan for `*.wav`.
- `--max-scan N` (default `0` = all) — cap the number of files embedded, useful on large dirs.
- `--dedupe-threshold F` (default `0.98`) — cosine at or above which a candidate counts as already
  covered by one of that speaker's prototypes.
- `--other-threshold F` (default `0.45`) — cosine at or above which a candidate is flagged as
  scoring against a different enrolled speaker.
- `--yes` — non-interactive: auto-accept the top `--count` candidates.
- `--no-play` — do not attempt audio playback during review.
- `--force` — enroll even when a daemon appears to be running.
- `--config` — the same global override as every subcommand.

The flow:

1. Loads config and refuses if a daemon appears to be running (its event socket exists) unless
   `--force` is passed.
2. Requires the model and `uv sync --extra speaker`, exactly like `speaker-enroll`.
3. Fails with a clear error if the stored prototype dimension differs from the model's: the store
   was built with a different model, so wipe `store_dir` and re-enroll.
4. Scans the audio directory, embeds each clip, ranks by cosine similarity to the seed, and keeps
   clips at least `[speaker].min_enroll_s` seconds long and at or above `--min-score`.
5. Reviews each candidate interactively (unless `--yes`), showing
   `[i/N] <filename> (score X.XX, Y.Ys)` with four actions: **Accept**, **Skip** (show the next
   candidate), **Play** (listen, then decide again), or **Quit** (stop, enroll nothing).
6. Enrolls the seed (when it is at least `min_enroll_s`) plus the accepted clips, then prints
   something like
   `enrolled 'Efren Yevale Varela' as spk_0001 from seed: 4 clip(s) (seed + 3 reviewed, 16.9s, 4 prototype(s))`.

If the seed is shorter than `min_enroll_s` it is used only as the query, and a warning is printed.
If nothing is accepted, or the store would exceed `max_prototypes` (an eviction warning), the
command says so and no partial write happens.

**Review warnings.** Each candidate is flagged during review:

- `duplicate` — its embedding is already covered by one of that speaker's existing prototypes
  (cosine at or above `--dedupe-threshold`, default `0.98`).
- `other-speaker:<name>` — it scores highly against a **different** enrolled speaker's profile
  (cosine at or above `--other-threshold`, default `0.45`), which is a contamination hint.

Choosing **Accept** on a `duplicate` is refused: it prints
`skipping <name>: already covered by this profile` and moves on without consuming the accept
budget. An `other-speaker:<name>` flag does not block the accept; it is a hint to listen harder.

**Duplicate removal.** Both commands guarantee no duplicates are added. The incoming set (seed plus
accepted clips) is greedily de-duplicated before enrolling, printed as
`skipped N duplicate clip(s) already covered ...`, and after enrolling the command removes any
pre-existing near-duplicate prototypes of that speaker, printed as
`removed N duplicate prototype(s) from the ...`. Plain `speaker-enroll` removes pre-existing
duplicates for the named speaker too, and reports it. The semantics are the same in both commands:
for each speaker, prototypes are walked in insertion order and the **first of each cluster** is
kept; a prototype is dropped when its cosine to an already-kept prototype of the **same** speaker
is at or above the threshold. The first prototype is always kept. This is what cleans up the store
when several near-identical clips were enrolled.

**Abort semantics.** Pressing **Quit** or Ctrl+C during review writes nothing. The command prints
the clips accepted so far and an equivalent manual `stt-daemon speaker-enroll --wav ...` command,
so you can run the explicit form yourself.

**Playback.** Review tries `paplay`, then `aplay`, then `ffplay`. If none exist, review continues
without sound; `--no-play` skips playback entirely.

**Non-interactive.** A non-TTY without `--yes` errors out; it never hangs waiting for input.

Prefer short `end_of_turn` clips for enrollment. Long `pause` or `end_of_turn` segments can hold
more than one speaker, which pollutes a prototype; that is exactly why the helper makes you listen
and review each clip. The plain `speaker-enroll --wav ...` command still exists for enrolling an
explicit clip list you already trust, as described in
[Pre-enroll from a WAV file](#pre-enroll-from-a-wav-file).

## Tuning an existing profile

Re-running `speaker-enroll-seed` or `speaker-enroll` for a name that already exists does **not**
reset the profile. It reuses the same profile id (`spk_NNNN`) and **appends** the new clips as
additional prototypes, and `cumulative_s` keeps growing. Tuning is therefore additive: you steer a
profile by adding good clips, and you flush a bad one by adding enough new clips to push the oldest
out.

**Broaden coverage.** Add several diverse short `end_of_turn` clips, in the 1-5 s range you actually
speak (enrollment still requires each clip to clear `min_enroll_s`, so the usable ones start at 2 s
by default), varying phrase, level, and session. More spread across the ways you really sound means
short live utterances score higher. This is what fixed a real case where a 1.74 s utterance landed
below `threshold_new`: the profile held one narrow prototype, and a handful of varied short clips
lifted the live score into range.

**Flush a bad or polluted prototype.** Eviction is oldest-first (FIFO), so adding new clips pushes
the oldest ones out. Each speaker holds at most `max_prototypes` (default `8`). To evict the oldest
one, add enough clips to exceed the cap: at 7 of 8, adding **2** more drops the oldest, i.e.
`max_prototypes - current + 1` additions; to evict the **N** oldest, add
`max_prototypes - current + N`. There is no per-prototype delete and no replace. Duplicates,
though, are handled for you: both enrollment commands drop near-duplicate prototypes
automatically and the review flags them as `duplicate`, so flushing is the way to remove a bad
prototype that is not a duplicate, short of wiping the store.

**`max_prototypes` is itself a lever.** Raising it (e.g. `16`) keeps more prototypes before eviction,
giving broader coverage at a marginal cost in false-match risk. Lowering it forces eviction sooner
and keeps the profile tighter.

**What to accept.** Listen with **Play**. Accept clean single-speaker clips; **skip anything with
a second voice**, which is the whole point of the review. Long `pause` segments are the usual
offenders. Near-duplicate clips are flagged `duplicate` and cannot be accepted, and both commands
strip near-duplicates from the profile automatically, so you rarely need to worry about them.

**What tuning cannot do.** It cannot delete a specific prototype or reorder them, and it does not
change thresholds: `threshold_id` and `threshold_new` are config, not the helper. `--yes` skips the
review entirely and auto-accepts the top `--count` candidates, so it cannot catch a clip containing
a second voice; do **not** use `--yes` for tune-ups.

**Verify.** Restart the daemon so it loads the updated store, then speak and watch the TUI header
(your name vs `Unknown`). With `[output].jsonl = true`, check the `speaker`, `speaker_status`, and
`speaker_score` fields. Repeatedly `uncertain` or `unknown` for your own voice means the profile
needs more or more diverse clips. A confident **wrong** name usually means overlapping speakers in
the enrollment clips, or a genuinely hard voice pair.

## Measuring accuracy (speaker-eval)

Prototype geometry tells you the distances between your own enrolled clips, not whether the matcher
gets real utterances right. `speaker-eval` scores the store against labeled clips and prints a
plain-text report, so you get a real accuracy number instead of guessing:

bash uv run stt-daemon speaker-eval --labeled labeled/

`--labeled DIR` is required. Each immediate subdirectory is a speaker **display name** and holds
that person's `*.wav` clips, 16 kHz mono 16-bit PCM like enrollment:

labeled/ Efren Yevale Varela/

001.wav
002.wav

Nombre Apellido/

001.wav

The command embeds every clip, labels it with the same matcher the daemon uses, and prints `total`,
`accuracy`, `correct`, `wrong`, `uncertain (top-1 correct/total)`, and `unknown`; a `per-speaker`
table (`name | n | correct | acc`); the top `confusions` as `true -> predicted (count)`, up to 10;
and `margins` with the `count`, `mean`, and `p10` of the top1 - top2 score gap over prototypes.

The run is read-only: it never writes the store, so the daemon does not need to be stopped. It does
need the model and `uv sync --extra speaker`, exactly like enrollment. If the store was built with
a different embedding model (dimension mismatch), it fails clearly instead of scoring nonsense:
wipe `store_dir` and re-enroll.

To use it as a tuning gate, pass `--min-accuracy F`, which exits non-zero when measured accuracy
is below `F`:

bash uv run stt-daemon speaker-eval --labeled labeled/ --min-accuracy 0.9

Label a handful of clips you are sure about per speaker. The accuracy and the confusion pairs are
the fastest way to see whether a profile needs more diverse clips, whether two voices genuinely
collide, or whether the thresholds need retuning.

## Wipe and rebuild

Wipe the store when a profile is polluted beyond easy flushing, when you switch embedding models
(the store must be rebuilt if the model dimension changes), or when the roster got messy. Wiping
removes **all** speakers. To remove just one, use the manual purge below, or flush it with added
clips as in [Tuning an existing profile](#tuning-an-existing-profile).

1. **Stop the daemon.** The store is single-writer.
2. **Delete the store:**

bash rm -rf speakers/

   `speakers/` is gitignored and a recoverable cache; `models/` is untouched.
3. **Re-enroll each speaker** from one clip you are sure of, reviewing the suggestions:

bash uv run stt-daemon speaker-enroll-seed --name "Nombre Apellido" --seed path/to/a-clip-you-trust.wav --count 5 ```

Repeat per speaker. The plain speaker-enroll --wav ... form works too when you have an explicit list. The next run rebuilds an empty store the moment it finds none.

1. Restart the daemon so it loads the fresh store.

This is the broad version of the manual purge below, which removes a single speaker instead of the whole roster.

Configuration reference

All keys live under [speaker]. The defaults are what you get with no config file. [speaker] is the 12th known section; unknown keys inside it are silently ignored, like every other section.

key	default	meaning
enabled	false	Master switch. false means no lane, no store, and every record speaker_status = "none".
model_path	null	Path to the sherpa-onnx embedding .onnx. Unset or missing disables the feature.
provider	"cpu"	cpu or cuda. cuda needs a CUDA-enabled sherpa wheel, else warns and falls back to cpu.
threshold_id	0.60	T_id: cosine score at or above which a turn matches an existing prototype.
threshold_new	0.45	T_new: cosine score below which a turn may mint a new provisional; the lower edge of uncertain.
min_enroll_s	2.0	Mint/adapt duration gate: shorter utterances never create or adapt a profile.
rms_floor	0.0	Mint/adapt level gate: quieter utterances never create or adapt a profile.
margin	0.04	Best-2nd margin required before an adapt or mint decision.
mint_cooldown_s	60.0	Minimum spacing between mints.
confirm_s	12.0	Cumulative speech seconds a provisional needs to be promoted to confirmed.
max_prototypes	8	Cap on stored prototypes per speaker.
provisional_ttl_h	24.0	Prune provisionals that never confirm within this many hours, checked at startup.
store_dir	"speakers"	Store root, cwd-relative; gitignored because it holds biometric artifacts.
join_timeout_s	2.0	Embedding-lane join timeout before the transcript is emitted.

The one-label-per-segment limitation

Identification runs on the whole VAD segment and produces one speaker label per segment. If two people speak within one segment, the clip yields a single blended embedding, which usually lands in uncertain or unknown; such clips are never enrolled and never adapted. Natural turn-taking, where a pause separates the speakers, splits into separate segments and labels each one correctly. If mixed-speaker segments are common in your audio, lower [vad].silence_ms (within reason) so pauses break turns sooner, or accept that overlapping speech is out of scope for v1.

Manual purge

delete and merge are not supported in v1. To remove a speaker, stop the daemon, remove the id from speakers/speakers.json and its rows from speakers/prototypes.npz, then restart. To start clean, stop the daemon and wipe speakers/ entirely; the next run rebuilds an empty store. Because identity is a recoverable cache, re-enrollment restores it.

Troubleshooting

For a missing sherpa-onnx install, a missing model, everything showing Unknown, naming that reports unknown_speaker, or resetting the store, see the Speaker identification section of TROUBLESHOOTING.md.