Speaker Identification

Speaker identification labels each transcribed utterance with who spoke it. A dedicated in-process sherpa-onnx embedding lane maps the clip to a vector, and the vector is matched against a local store of per-speaker prototypes. The label then rides the transcript to the TUI header and into the JSONL record. There is no separate process, no socket, and no change to the supervisor's child count.

The feature is off by default ([speaker].enabled = false), and the extra it needs is opt-in, so a plain uv sync and the existing test suite behave exactly as before. When it is off, or the model is missing, or the library fails to load, the feature no-ops: no lane is built, no store is touched, the daemon still runs, and every record carries speaker_status = "none".

What the transcript shows

TUI header. The speaker sits between the language tag and the duration, for example 10:01:03 EN Efren 3.2s pause. Named speakers render in bold green; every other speaker, including provisionals and Unknown, renders in bold red. The literal Unknown is shown for unmatched turns.
JSONL. Existing records are enriched in place; there is one append-only file and no separate annotated file. Five additive, back-compatible fields are added: seg_id, speaker_id, speaker_status, speaker, and speaker_score. All are null or "none" when the feature is off or the turn could not be identified. See OUTPUT.md for the exact field semantics.

How matching works

A speaker profile is a set of L2-normalized embedding prototypes, capped at max_prototypes (8). A turn's score is the maximum cosine similarity over all of that speaker's prototypes, never a centroid. The score lands the turn in one of four status bands:

status	condition	behavior
`matched`	score >= `threshold_id` (`0.60`)	Reports the matched speaker; may adapt the profile.
`uncertain`	`threshold_new` (`0.45`) <= score < `0.60`	Reports the top-1 speaker with a `?` suffix; never enrolls and never adapts.
`unknown`	score < `0.45`	No label; eligible for auto-enrollment after the guards pass.
`none`	no embedding computed	Feature off, audio too short, or an error.

The two thresholds and margin are starting points, not measured values. They are model- and channel-dependent, so calibrate them on real far-field audio from the actual microphone and room before trusting the labels, and use the per-record speaker_score to tune. speaker_status is the coarse answer; the score is what you optimize against.

Auto-enrollment and promotion

An unknown voice is minted as a provisional spk_NNNN only after every guard passes:

Duration: the utterance is at least min_enroll_s (2.0 s).
Level: the utterance is at or above rms_floor (default 0.0, i.e. no gate).
Hysteresis: two consecutive mutually-similar misses, so a single odd clip cannot mint.
Cooldown: no second mint within mint_cooldown_s (60 s).
Margin: the best prototype beats the second best by at least margin (0.04).

A provisional profile is promoted to confirmed once it accumulates confirm_s (12 s) of speech. Until then it is labeled with its provisional id. A provisional that never confirms is pruned after provisional_ttl_h (24 h). stt-ctl speaker-name names a provisional, and naming it deletes the rolling sample that was kept for the enrollment decision. The name is then shown everywhere the speaker appears.

Store layout and privacy

Everything lives under store_dir (speakers), relative to the working directory, and is gitignored:

speakers/
  speakers.json          # roster: ids, names, status, cumulative speech seconds
  prototypes.npz         # normalized embedding prototypes per speaker
  samples/               # rolling <id>.wav samples held for enrollment decisions
  inbox/                 # rename requests the daemon applies
    rejected/            # malformed requests

Directories are created 0700 and sample WAVs 0600. Samples are deleted as soon as a profile is named or pruned. Embeddings and samples are biometric data. Treat speakers/ as private user state, not as a shareable artifact. Identity here is a recoverable cache, not an archive: re-enrolling restores it.

Commands

List the roster (id, name, status, cumulative seconds, sample path):

uv run stt-ctl speaker-list

Name a provisional speaker:

uv run stt-ctl speaker-name spk_0001 Efren

Naming writes a rename request to speakers/inbox/, which the daemon applies on its next tick; an unknown id returns an error. The assistant caches the roster from speakers bus events, so a name that has not yet flowed through an event can be reported as unknown_speaker (see TROUBLESHOOTING.md). Renaming a speaker that already has a name overwrites it. There is no delete or merge in v1.

Install and enable

The dependency lives behind an optional extra:

# 1. Install the speaker extra (adds sherpa-onnx)
uv sync --extra speaker

# 2. Download the embedding model into the gitignored models/ directory
curl -L -o models/nemo_en_titanet_small.onnx \
  "https://github.com/k2-fsa/sherpa-onnx/releases/download/speaker-recongition-models/nemo_en_titanet_small.onnx"

# 3. Point config.toml at it and turn it on
#    [speaker]
#    enabled = true
#    model_path = "models/nemo_en_titanet_small.onnx"

The extra pulls two distributions, sherpa-onnx (the Python API) and sherpa-onnx-core (the wheel that bundles the native libonnxruntime.so). Extras are exclusive under uv sync: uv sync --extra speaker matches the default deps plus speaker exactly, so it removes the triage extra and its stack (laya, torch, transformers, ...). To keep both, request both: uv sync --extra speaker --extra triage.

The model is nemo_en_titanet_small.onnx, 38.4 MB, 192-dimensional, CC-BY-4.0, and not gated. The models/ directory is gitignored, so the download is local-only. Note the upstream release tag's spelling, speaker-recongition-models; the typo is in the URL and must be copied verbatim.

The model is CPU by design (provider = "cpu", num_threads = 1). provider = "cuda" requires a CUDA-enabled sherpa wheel; without one it warns and falls back to cpu. The installed onnxruntime exposes only CPUExecutionProvider, so cuda changes nothing without a different wheel.

If the Spanish/English quality is not enough for your room, two larger sherpa models are drop-in alternatives:

model	size	dim
`3dspeaker_speech_campplus_sv_en_voxceleb_16k.onnx`	28 MB	192
`wespeaker_en_voxceleb_resnet34_LM.onnx`	25 MB	256

Same download host and URL shape; only the filename changes.

Pre-enroll from a WAV file

Enrollment is reactive by default: the daemon mints a provisional spk_NNNN on an unknown live voice, and you name it afterwards with stt-ctl speaker-name. To register a known speaker up front from clips you already have, run stt-daemon speaker-enroll with one or more WAV files:

uv run stt-daemon speaker-enroll --name Efren --wav audio/2026-09-28/143022-123-pause.wav

Repeat --wav to enroll from several clips; each file contributes one prototype, capped at max_prototypes with the oldest evicted:

uv run stt-daemon speaker-enroll --name Efren \
  --wav audio/2026-09-28/143022-123-pause.wav \
  --wav audio/2026-09-28/150210-456-end_of_turn.wav

On success the command prints enrolled 'Efren' as spk_0002 (2 clip(s), 7.4s). The --name and --wav flags are required; --force bypasses the running-daemon refusal, and --config is the same global override as every other subcommand.

Each WAV must be 16 kHz mono 16-bit PCM. No resampling is performed: stereo, a different sample rate, or a non-WAV file is rejected with a clear error. The clips the daemon saves under audio/YYYY-MM-DD/ when [output].save_audio = true are already in that format, so they need no conversion. Each clip must also be at least [speaker].min_enroll_s seconds (default 2.0); shorter clips are rejected. This is the "is it enough audio?" check.

The profile is created confirmed with the given name, and unlike a named provisional it stores no sample WAV; samples exist only for the listen-before-naming flow. Explicit enrollment bypasses the automatic mint guards (duration, level, consecutive-miss, cooldown, and margin), because you are asserting the identity rather than inferring it; the CLI's duration gate is the only length check. Re-enrolling an existing name appends a prototype to that same speaker rather than creating a duplicate.

Enrollment does not need [speaker].enabled = true; you can pre-enroll before turning the feature on. It does need the model and the extra: uv sync --extra speaker and [speaker].model_path set.

The store is single-writer and owned by the daemon. Stop the daemon before enrolling, or the in-memory store can overwrite your change on its next save. The command detects the daemon's event socket and refuses unless --force is passed. Storage is unchanged: speakers/speakers.json and speakers/prototypes.npz under store_dir, gitignored and biometric.

Assisted enrollment from a seed clip

Enrolling from one long clip produces weak, polluted prototypes: a long pause segment can hold several speakers, so its embedding lands somewhere between them. speaker-enroll-seed takes ONE clip you are certain is the speaker, finds the clips in the audio directory that are most similar to it, and lets you review each one before it joins the profile:

uv run stt-daemon speaker-enroll-seed --name "Efren Yevale Varela" --seed audio/efren-seed.wav

Flags:

--name (required) — speaker display name.
--seed (required) — a 16 kHz mono 16-bit WAV known to be that speaker.
--count N (default 5) — how many reviewed clips to accept.
--min-score F (default 0.55) — minimum cosine to the seed for a clip to be offered.
--audio-dir DIR (default [output].audio_dir) — where to scan for *.wav.
--max-scan N (default 0 = all) — cap the number of files embedded, useful on large dirs.
--dedupe-threshold F (default 0.98) — cosine at or above which a candidate counts as already covered by one of that speaker's prototypes.
--other-threshold F (default 0.45) — cosine at or above which a candidate is flagged as scoring against a different enrolled speaker.
--yes — non-interactive: auto-accept the top --count candidates.
--no-play — do not attempt audio playback during review.
--force — enroll even when a daemon appears to be running.
--config — the same global override as every subcommand.

The flow:

Loads config and refuses if a daemon appears to be running (its event socket exists) unless --force is passed.
Requires the model and uv sync --extra speaker, exactly like speaker-enroll.
Fails with a clear error if the stored prototype dimension differs from the model's: the store was built with a different model, so wipe store_dir and re-enroll.
Scans the audio directory, embeds each clip, ranks by cosine similarity to the seed, and keeps clips at least [speaker].min_enroll_s seconds long and at or above --min-score.
Reviews each candidate interactively (unless --yes), showing [i/N] <filename> (score X.XX, Y.Ys) with four actions: Accept, Skip (show the next candidate), Play (listen, then decide again), or Quit (stop, enroll nothing).
Enrolls the seed (when it is at least min_enroll_s) plus the accepted clips, then prints something like enrolled 'Efren Yevale Varela' as spk_0001 from seed: 4 clip(s) (seed + 3 reviewed, 16.9s, 4 prototype(s)).

If the seed is shorter than min_enroll_s it is used only as the query, and a warning is printed. If nothing is accepted, or the store would exceed max_prototypes (an eviction warning), the command says so and no partial write happens.

Review warnings. Each candidate is flagged during review:

duplicate — its embedding is already covered by one of that speaker's existing prototypes (cosine at or above --dedupe-threshold, default 0.98).
other-speaker:<name> — it scores highly against a different enrolled speaker's profile (cosine at or above --other-threshold, default 0.45), which is a contamination hint.

Choosing Accept on a duplicate is refused: it prints skipping <name>: already covered by this profile and moves on without consuming the accept budget. An other-speaker:<name> flag does not block the accept; it is a hint to listen harder.

Duplicate removal. Both commands guarantee no duplicates are added. The incoming set (seed plus accepted clips) is greedily de-duplicated before enrolling, printed as skipped N duplicate clip(s) already covered ..., and after enrolling the command removes any pre-existing near-duplicate prototypes of that speaker, printed as removed N duplicate prototype(s) from the .... Plain speaker-enroll removes pre-existing duplicates for the named speaker too, and reports it. The semantics are the same in both commands: for each speaker, prototypes are walked in insertion order and the first of each cluster is kept; a prototype is dropped when its cosine to an already-kept prototype of the same speaker is at or above the threshold. The first prototype is always kept. This is what cleans up the store when several near-identical clips were enrolled.

Abort semantics. Pressing Quit or Ctrl+C during review writes nothing. The command prints the clips accepted so far and an equivalent manual stt-daemon speaker-enroll --wav ... command, so you can run the explicit form yourself.

Playback. Review tries paplay, then aplay, then ffplay. If none exist, review continues without sound; --no-play skips playback entirely.

Non-interactive. A non-TTY without --yes errors out; it never hangs waiting for input.

Prefer short end_of_turn clips for enrollment. Long pause or end_of_turn segments can hold more than one speaker, which pollutes a prototype; that is exactly why the helper makes you listen and review each clip. The plain speaker-enroll --wav ... command still exists for enrolling an explicit clip list you already trust, as described in Pre-enroll from a WAV file.

Tuning an existing profile

Re-running speaker-enroll-seed or speaker-enroll for a name that already exists does not reset the profile. It reuses the same profile id (spk_NNNN) and appends the new clips as additional prototypes, and cumulative_s keeps growing. Tuning is therefore additive: you steer a profile by adding good clips, and you flush a bad one by adding enough new clips to push the oldest out.

Broaden coverage. Add several diverse short end_of_turn clips, in the 1-5 s range you actually speak (enrollment still requires each clip to clear min_enroll_s, so the usable ones start at 2 s by default), varying phrase, level, and session. More spread across the ways you really sound means short live utterances score higher. This is what fixed a real case where a 1.74 s utterance landed below threshold_new: the profile held one narrow prototype, and a handful of varied short clips lifted the live score into range.

Flush a bad or polluted prototype. Eviction is oldest-first (FIFO), so adding new clips pushes the oldest ones out. Each speaker holds at most max_prototypes (default 8). To evict the oldest one, add enough clips to exceed the cap: at 7 of 8, adding 2 more drops the oldest, i.e. max_prototypes - current + 1 additions; to evict the N oldest, add max_prototypes - current + N. There is no per-prototype delete and no replace. Duplicates, though, are handled for you: both enrollment commands drop near-duplicate prototypes automatically and the review flags them as duplicate, so flushing is the way to remove a bad prototype that is not a duplicate, short of wiping the store.

max_prototypes is itself a lever. Raising it (e.g. 16) keeps more prototypes before eviction, giving broader coverage at a marginal cost in false-match risk. Lowering it forces eviction sooner and keeps the profile tighter.

What to accept. Listen with Play. Accept clean single-speaker clips; skip anything with a second voice, which is the whole point of the review. Long pause segments are the usual offenders. Near-duplicate clips are flagged duplicate and cannot be accepted, and both commands strip near-duplicates from the profile automatically, so you rarely need to worry about them.

What tuning cannot do. It cannot delete a specific prototype or reorder them, and it does not change thresholds: threshold_id and threshold_new are config, not the helper. --yes skips the review entirely and auto-accepts the top --count candidates, so it cannot catch a clip containing a second voice; do not use --yes for tune-ups.

Verify. Restart the daemon so it loads the updated store, then speak and watch the TUI header (your name vs Unknown). With [output].jsonl = true, check the speaker, speaker_status, and speaker_score fields. Repeatedly uncertain or unknown for your own voice means the profile needs more or more diverse clips. A confident wrong name usually means overlapping speakers in the enrollment clips, or a genuinely hard voice pair.

Measuring accuracy (speaker-eval)

Prototype geometry tells you the distances between your own enrolled clips, not whether the matcher gets real utterances right. speaker-eval scores the store against labeled clips and prints a plain-text report, so you get a real accuracy number instead of guessing:

uv run stt-daemon speaker-eval --labeled labeled/

--labeled DIR is required. Each immediate subdirectory is a speaker display name and holds that person's *.wav clips, 16 kHz mono 16-bit PCM like enrollment:

labeled/
  Efren Yevale Varela/
    001.wav
    002.wav
  Nombre Apellido/
    001.wav

The command embeds every clip, labels it with the same matcher the daemon uses, and prints total, accuracy, correct, wrong, uncertain (top-1 correct/total), and unknown; a per-speaker table (name | n | correct | acc); the top confusions as true -> predicted (count), up to 10; and margins with the count, mean, and p10 of the top1 - top2 score gap over prototypes.

The run is read-only: it never writes the store, so the daemon does not need to be stopped. It does need the model and uv sync --extra speaker, exactly like enrollment. If the store was built with a different embedding model (dimension mismatch), it fails clearly instead of scoring nonsense: wipe store_dir and re-enroll.

To use it as a tuning gate, pass --min-accuracy F, which exits non-zero when measured accuracy is below F:

uv run stt-daemon speaker-eval --labeled labeled/ --min-accuracy 0.9

Label a handful of clips you are sure about per speaker. The accuracy and the confusion pairs are the fastest way to see whether a profile needs more diverse clips, whether two voices genuinely collide, or whether the thresholds need retuning.

Wipe and rebuild

Wipe the store when a profile is polluted beyond easy flushing, when you switch embedding models (the store must be rebuilt if the model dimension changes), or when the roster got messy. Wiping removes all speakers. To remove just one, use the manual purge below, or flush it with added clips as in Tuning an existing profile.

Stop the daemon. The store is single-writer.
Delete the store:
```
rm -rf speakers/
```

speakers/ is gitignored and a recoverable cache; models/ is untouched.

Re-enroll each speaker from one clip you are sure of, reviewing the suggestions:

uv run stt-daemon speaker-enroll-seed --name "Nombre Apellido" --seed path/to/a-clip-you-trust.wav --count 5

Repeat per speaker. The plain speaker-enroll --wav ... form works too when you have an explicit list. The next run rebuilds an empty store the moment it finds none.

Restart the daemon so it loads the fresh store.

This is the broad version of the manual purge below, which removes a single speaker instead of the whole roster.

Configuration reference

All keys live under [speaker]. The defaults are what you get with no config file. [speaker] is the 12th known section; unknown keys inside it are silently ignored, like every other section.

key	default	meaning
`enabled`	`false`	Master switch. `false` means no lane, no store, and every record `speaker_status = "none"`.
`model_path`	`null`	Path to the sherpa-onnx embedding `.onnx`. Unset or missing disables the feature.
`provider`	`"cpu"`	`cpu` or `cuda`. `cuda` needs a CUDA-enabled sherpa wheel, else warns and falls back to `cpu`.
`threshold_id`	`0.60`	`T_id`: cosine score at or above which a turn matches an existing prototype.
`threshold_new`	`0.45`	`T_new`: cosine score below which a turn may mint a new provisional; the lower edge of `uncertain`.
`min_enroll_s`	`2.0`	Mint/adapt duration gate: shorter utterances never create or adapt a profile.
`rms_floor`	`0.0`	Mint/adapt level gate: quieter utterances never create or adapt a profile.
`margin`	`0.04`	Best-2nd margin required before an adapt or mint decision.
`mint_cooldown_s`	`60.0`	Minimum spacing between mints.
`confirm_s`	`12.0`	Cumulative speech seconds a provisional needs to be promoted to `confirmed`.
`max_prototypes`	`8`	Cap on stored prototypes per speaker.
`provisional_ttl_h`	`24.0`	Prune provisionals that never confirm within this many hours, checked at startup.
`store_dir`	`"speakers"`	Store root, cwd-relative; gitignored because it holds biometric artifacts.
`join_timeout_s`	`2.0`	Embedding-lane join timeout before the transcript is emitted.

The one-label-per-segment limitation

Identification runs on the whole VAD segment and produces one speaker label per segment. If two people speak within one segment, the clip yields a single blended embedding, which usually lands in uncertain or unknown; such clips are never enrolled and never adapted. Natural turn-taking, where a pause separates the speakers, splits into separate segments and labels each one correctly. If mixed-speaker segments are common in your audio, lower [vad].silence_ms (within reason) so pauses break turns sooner, or accept that overlapping speech is out of scope for v1.

Manual purge

delete and merge are not supported in v1. To remove a speaker, stop the daemon, remove the id from speakers/speakers.json and its rows from speakers/prototypes.npz, then restart. To start clean, stop the daemon and wipe speakers/ entirely; the next run rebuilds an empty store. Because identity is a recoverable cache, re-enrollment restores it.

Troubleshooting

For a missing sherpa-onnx install, a missing model, everything showing Unknown, naming that reports unknown_speaker, or resetting the store, see the Speaker identification section of TROUBLESHOOTING.md.

SPEAKERS.md 23 KB Histórico Raw