# pywhispercpp: GPU source rebuild

`pywhispercpp` is a Python binding for `whisper.cpp` (the ggml ASR engine, same
family as `llama.cpp`). It is an alternative transcription backend to
`faster-whisper`/CTranslate2, which on this machine cannot use the GPU (see
"Background" below).

## Background

- **Stock PyPI wheel:** `pywhispercpp 1.5.1` bundles `libwhisper.so.1.8.5` +
  ggml 0.13.1 with the **CPU backend only** (`libggml-cpu`; no
  `libggml-vulkan`, no `libggml-cuda`). The model-load log prints
  `whisper_backend_init_gpu: no GPU found`.
- **It works, on CPU:** the JFK sample transcribes correctly with `ggml-small`,
  at RTF ≈ 0.18 (11 s of audio in ≈ 2.0 s).
- **To use the GPU you must rebuild from source** with `GGML_VULKAN=1` or
  `GGML_CUDA=1`. The PyPI wheel cannot be reconfigured.

`faster-whisper`/CTranslate2 is hard-locked to CUDA 12 + cuDNN 9, and the CT2
PyPI wheels lack sm_100/sm_120 (Blackwell) kernels. The CUDA 13 toolkit and the
Vulkan stack are already installed, so a `whisper.cpp` build is the practical GPU
path.

## Verified result (this machine)

CUDA 13 build, cloned to `~/Repositories/pywhispercpp`, built into a vendored
wheel and installed into the project `.venv`. Runtime log:

```
whisper_init_with_params_no_state: use gpu    = 1
ggml_cuda_init: found 1 CUDA devices (Total VRAM: 15888 MiB):
  Device 0: NVIDIA RTX PRO 2000 Blackwell, compute capability 12.0
whisper_model_load:        CUDA0 total size =   487.01 MB
whisper_backend_init_gpu: using CUDA0 backend
```

JFK sample with `ggml-small`:

| backend       | transcribe (11 s audio) | RTF  |
| ------------- | ----------------------- | ---- |
| CPU wheel     | 2.00 s                  | 0.18 |
| CUDA 13 build | 0.15 s                  | 0.01 |

≈13× faster, transcript correct. sm_120 (Blackwell) kernels are present and run.

## Vendored wheel (locked into uv)

The GPU build is kept as a local wheel so `uv sync` restores it instead of
pruning it.

- **Wheel:** `vendor/pywhispercpp-1.5.1-cp313-cp313-linux_x86_64.whl` (~136 MB).
  `vendor/.gitignore` contains `*`, so it is **not committed** — it is a local
  artifact. If it is deleted, `uv sync` fails; rebuild it (step 2).
- **`pyproject.toml`:** the dependency is gated to the wheel's ABI, and the
  source is the local file:

  ```toml
  dependencies = [
      # ...
      "pywhispercpp>=1.5.1 ; python_full_version == '3.13.*'",
  ]

  [tool.uv.sources]
  pywhispercpp = { path = "vendor/pywhispercpp-1.5.1-cp313-cp313-linux_x86_64.whl" }
  ```

  The marker is required: the project supports Python `>=3.11`, but the wheel is
  `cp313`-only, so an unconditional dependency fails universal resolution. On
  Python 3.11/3.12 the package is simply not installed.
- **`uv.lock`** records `pywhispercpp` with `source = { path = "vendor/..." }` and
  its hash, so `uv sync` reinstalls the CUDA build. `uv sync --dry-run` reports
  "Would make no changes".

## Prerequisites (this machine)

- Debian 13 (trixie), `git`, `gcc`/`g++` 14.2
- **Vulkan:** `libvulkan-dev`, `glslc`, `glslang-tools`, `vulkan-tools`
- **CUDA 13.3** toolkit at `/usr/local/cuda-13.3` (`nvcc 13.3.73`, cuBLAS 13.6)

## 1. Source

The clone already exists at `~/Repositories/pywhispercpp` (recursive clone;
`pybind11` and `whisper.cpp` are submodules). If re-cloning, use
`git clone --recursive` with **full history** (the build uses `setuptools-scm`,
which needs git tags).

## 2. Build the CUDA wheel

Run from the clone, and build with the **project venv's Python** (the wheel must
be `cp313`, matching the venv):

```bash
cd ~/Repositories/pywhispercpp
CUDA_HOME=/usr/local/cuda-13.3 \
CUDACXX=/usr/local/cuda-13.3/bin/nvcc \
PATH="/usr/local/cuda-13.3/bin:$PATH" \
PYWHISPERCPP_VERSION=1.5.1 \
SETUPTOOLS_SCM_PRETEND_VERSION=1.5.1 \
GGML_CUDA=1 \
CMAKE_INSTALL_RPATH='$ORIGIN' \
CMAKE_BUILD_WITH_INSTALL_RPATH=ON \
uv build --wheel \
  --python /home/efren/Repositories/STT-TTS/.venv/bin/python \
  --out-dir /home/efren/Repositories/STT-TTS/vendor
```

- `GGML_CUDA=1` selects the CUDA backend; `setup.py` forwards **every** env var
  to CMake as `-DKEY=VALUE`, so `CMAKE_INSTALL_RPATH` /
  `CMAKE_BUILD_WITH_INSTALL_RPATH` can be set as plain env vars (no `CMAKE_ARGS`
  needed).
- `CMAKE_INSTALL_RPATH='$ORIGIN'` makes the wheel self-contained: the ggml libs
  resolve from the package directory, not the clone's `build/` tree.
- `--python <venv>/bin/python` is essential; without it the build uses whatever
  Python it finds and produces the wrong ABI tag (e.g. `cp312`).

Then install it through uv (already done; re-run only if the wheel changed):

```bash
cd /home/efren/Repositories/STT-TTS
uv lock && uv sync
```

## 3. Verify the GPU is used

```bash
# Self-contained: RUNPATH should be $ORIGIN, libs under site-packages
readelf -d .venv/lib/python3.13/site-packages/_pywhispercpp*.so | grep -i runpath
ldd .venv/lib/python3.13/site-packages/_pywhispercpp*.so | grep -iE 'ggml|cublas'

# Runtime init log (whisper.cpp logs to stderr by default)
uv run python -c "from pywhispercpp.model import Model; Model('small')"
```

Success: `ggml_cuda_init: found 1 CUDA devices` / `using CUDA0 backend`, and not
the `no GPU found` line. Note `redirect_whispercpp_logs_to` accepts `None`, a
filepath string, or a file-like object — **not** `True`.

## 4. Rollback

```bash
uv remove pywhispercpp     # drops the dependency + source entry, then re-locks
uv sync
```

## Gotchas

- **The wheel is `cp313`-only.** Any interpreter switch (or a `requires-python`
  change) breaks resolution; the dependency marker and the wheel tag must stay in
  sync.
- **The wheel is local-only** (`vendor/.gitignore` = `*`). A fresh clone has no
  wheel and `uv sync` fails until step 2 is re-run. The clone itself is *not*
  needed at runtime once the wheel is built (RUNPATH is `$ORIGIN`).
- **No `LD_LIBRARY_PATH` needed.** The CUDA libs (`libcudart.so.13`,
  `libcublas.so.13`, `libcublasLt.so.13`) resolve via `libggml-cuda`'s own
  link-path rpath, so a plain `import` works.
- **Vulkan device selection (if you switch to a Vulkan build):** ggml enumerates
  the NVIDIA GPU, the AMD 610M iGPU, and llvmpipe. It may pick the iGPU first;
  set `GGML_VULKAN_DEVICE=<index>` and confirm in the init log.
- **Build time:** several minutes.

## Integration

Set `[whisper] backend = "pywhispercpp"` in `config.toml` to use it. The default is
`faster-whisper`, so nothing changes unless you opt in.

The backend lives behind `build_transcriber()`, which returns either the
`faster-whisper` or the `pywhispercpp` implementation from the same config.
`device` and `compute_type` are ignored: the GPU support comes from the built
wheel and quantization is a property of the ggml model file, not a config key.
`vad_filter` is also ignored, with a warning (we already run webrtcvad upstream).

Everything else maps to the whisper.cpp equivalent: language auto-detect and
sticky pinning, `no_speech_prob_max` → `no_speech_thold`, `avg_logprob_min` →
segment probability gate, `beam_size` → beam search, and `cpu_threads`.

## When this is worth doing

The GPU build is verified ≈13× faster than the CPU wheel for the same `small`
model (RTF 0.01 vs 0.18). The remaining question is whether whisper.cpp's
transcript quality and latency beat the current `faster-whisper` CPU backend on
real utterances — compare the two on the same clip before wiring anything in.