pywhispercpp: GPU source rebuild

pywhispercpp is a Python binding for whisper.cpp (the ggml ASR engine, same family as llama.cpp). It is an alternative transcription backend to faster-whisper/CTranslate2, which on this machine cannot use the GPU (see "Background" below).

Background

Stock PyPI wheel: pywhispercpp 1.5.1 bundles libwhisper.so.1.8.5 + ggml 0.13.1 with the CPU backend only (libggml-cpu; no libggml-vulkan, no libggml-cuda). The model-load log prints whisper_backend_init_gpu: no GPU found.
It works, on CPU: the JFK sample transcribes correctly with ggml-small, at RTF ≈ 0.18 (11 s of audio in ≈ 2.0 s).
To use the GPU you must rebuild from source with GGML_VULKAN=1 or GGML_CUDA=1. The PyPI wheel cannot be reconfigured.

faster-whisper/CTranslate2 is hard-locked to CUDA 12 + cuDNN 9, and the CT2 PyPI wheels lack sm_100/sm_120 (Blackwell) kernels. The CUDA 13 toolkit and the Vulkan stack are already installed, so a whisper.cpp build is the practical GPU path.

Verified result (this machine)

CUDA 13 build, cloned to ~/Repositories/pywhispercpp, built into a vendored wheel and installed into the project .venv. Runtime log:

whisper_init_with_params_no_state: use gpu    = 1
ggml_cuda_init: found 1 CUDA devices (Total VRAM: 15888 MiB):
  Device 0: NVIDIA RTX PRO 2000 Blackwell, compute capability 12.0
whisper_model_load:        CUDA0 total size =   487.01 MB
whisper_backend_init_gpu: using CUDA0 backend

JFK sample with ggml-small:

backend	transcribe (11 s audio)	RTF
CPU wheel	2.00 s	0.18
CUDA 13 build	0.15 s	0.01

≈13× faster, transcript correct. sm_120 (Blackwell) kernels are present and run.

Vendored wheel (locked into uv)

The GPU build is kept as a local wheel so uv sync restores it instead of pruning it.

Wheel: vendor/pywhispercpp-1.5.1-cp313-cp313-linux_x86_64.whl (~136 MB). vendor/.gitignore contains *, so it is not committed — it is a local artifact. If it is deleted, uv sync fails; rebuild it (step 2).

pyproject.toml: the dependency is gated to the wheel's ABI, and the source is the local file:

dependencies = [
  # ...
  "pywhispercpp>=1.5.1 ; python_full_version == '3.13.*'",
]

[tool.uv.sources]
pywhispercpp = { path = "vendor/pywhispercpp-1.5.1-cp313-cp313-linux_x86_64.whl" }

The marker is required: the project supports Python >=3.11, but the wheel is cp313-only, so an unconditional dependency fails universal resolution. On Python 3.11/3.12 the package is simply not installed.

uv.lock records pywhispercpp with source = { path = "vendor/..." } and its hash, so uv sync reinstalls the CUDA build. uv sync --dry-run reports "Would make no changes".

Prerequisites (this machine)

Debian 13 (trixie), git, gcc/g++ 14.2
Vulkan: libvulkan-dev, glslc, glslang-tools, vulkan-tools
CUDA 13.3 toolkit at /usr/local/cuda-13.3 (nvcc 13.3.73, cuBLAS 13.6)

1. Source

The clone already exists at ~/Repositories/pywhispercpp (recursive clone; pybind11 and whisper.cpp are submodules). If re-cloning, use git clone --recursive with full history (the build uses setuptools-scm, which needs git tags).

2. Build the CUDA wheel

Run from the clone, and build with the project venv's Python (the wheel must be cp313, matching the venv):

cd ~/Repositories/pywhispercpp
CUDA_HOME=/usr/local/cuda-13.3 \
CUDACXX=/usr/local/cuda-13.3/bin/nvcc \
PATH="/usr/local/cuda-13.3/bin:$PATH" \
PYWHISPERCPP_VERSION=1.5.1 \
SETUPTOOLS_SCM_PRETEND_VERSION=1.5.1 \
GGML_CUDA=1 \
CMAKE_INSTALL_RPATH='$ORIGIN' \
CMAKE_BUILD_WITH_INSTALL_RPATH=ON \
uv build --wheel \
  --python /home/efren/Repositories/STT-TTS/.venv/bin/python \
  --out-dir /home/efren/Repositories/STT-TTS/vendor

GGML_CUDA=1 selects the CUDA backend; setup.py forwards every env var to CMake as -DKEY=VALUE, so CMAKE_INSTALL_RPATH / CMAKE_BUILD_WITH_INSTALL_RPATH can be set as plain env vars (no CMAKE_ARGS needed).
CMAKE_INSTALL_RPATH='$ORIGIN' makes the wheel self-contained: the ggml libs resolve from the package directory, not the clone's build/ tree.
--python <venv>/bin/python is essential; without it the build uses whatever Python it finds and produces the wrong ABI tag (e.g. cp312).

Then install it through uv (already done; re-run only if the wheel changed):

cd /home/efren/Repositories/STT-TTS
uv lock && uv sync

3. Verify the GPU is used

# Self-contained: RUNPATH should be $ORIGIN, libs under site-packages
readelf -d .venv/lib/python3.13/site-packages/_pywhispercpp*.so | grep -i runpath
ldd .venv/lib/python3.13/site-packages/_pywhispercpp*.so | grep -iE 'ggml|cublas'

# Runtime init log (whisper.cpp logs to stderr by default)
uv run python -c "from pywhispercpp.model import Model; Model('small')"

Success: ggml_cuda_init: found 1 CUDA devices / using CUDA0 backend, and not the no GPU found line. Note redirect_whispercpp_logs_to accepts None, a filepath string, or a file-like object — not True.

4. Rollback

uv remove pywhispercpp     # drops the dependency + source entry, then re-locks
uv sync

Gotchas

The wheel is cp313-only. Any interpreter switch (or a requires-python change) breaks resolution; the dependency marker and the wheel tag must stay in sync.
The wheel is local-only (vendor/.gitignore = *). A fresh clone has no wheel and uv sync fails until step 2 is re-run. The clone itself is not needed at runtime once the wheel is built (RUNPATH is $ORIGIN).
No LD_LIBRARY_PATH needed. The CUDA libs (libcudart.so.13, libcublas.so.13, libcublasLt.so.13) resolve via libggml-cuda's own link-path rpath, so a plain import works.
Vulkan device selection (if you switch to a Vulkan build): ggml enumerates the NVIDIA GPU, the AMD 610M iGPU, and llvmpipe. It may pick the iGPU first; set GGML_VULKAN_DEVICE=<index> and confirm in the init log.
Build time: several minutes.

Integration

Set [whisper] backend = "pywhispercpp" in config.toml to use it. The default is faster-whisper, so nothing changes unless you opt in.

The backend lives behind build_transcriber(), which returns either the faster-whisper or the pywhispercpp implementation from the same config. device and compute_type are ignored: the GPU support comes from the built wheel and quantization is a property of the ggml model file, not a config key. vad_filter is also ignored, with a warning (we already run webrtcvad upstream).

Everything else maps to the whisper.cpp equivalent: language auto-detect and sticky pinning, no_speech_prob_max → no_speech_thold, avg_logprob_min → segment probability gate, beam_size → beam search, and cpu_threads.

When this is worth doing

The GPU build is verified ≈13× faster than the CPU wheel for the same small model (RTF 0.01 vs 0.18). The remaining question is whether whisper.cpp's transcript quality and latency beat the current faster-whisper CPU backend on real utterances — compare the two on the same clip before wiring anything in.

PYWHISPERCPP.md 7.2 KB Lịch sử Raw