Skip to main content

CPU, CUDA, and performance

CPU​

CPU inference is the default and needs no backend argument.

tts = KittenTTS("KittenML/kitten-tts-nano-0.8", backend="cpu")

Omitting backend lets ONNX Runtime choose from its installed providers. Use an explicit value when reproducibility matters.

CUDA​

Install the repository's GPU dependency set in a clean environment:

git clone https://github.com/KittenML/KittenTTS.git
cd KittenTTS
pip install -r requirements_gpu.txt
pip install -e .

Then select the CUDA execution provider:

tts = KittenTTS(
    "KittenML/kitten-tts-mini-0.8",
    backend="cuda",
)

The GPU requirements target CUDA 12 libraries. Confirm that onnxruntime-gpu exposes CUDAExecutionProvider before benchmarking.

import onnxruntime as ort
print(ort.get_available_providers())

AMD ROCm​

The code accepts backend="amd_gpu" and requests ROCMExecutionProvider. Your Python environment must provide an ONNX Runtime build with that provider; the default wheel does not add it automatically.

Practical tuning​

  • Use Nano int8 or Nano for startup-sensitive CPU applications.
  • Load one KittenTTS instance and reuse it; model construction creates the ONNX session.
  • Use generate_stream() to reduce time to first audio for long input.
  • Warm the model with a short sentence before latency-sensitive work.
  • Keep the Hugging Face cache on fast local storage.
  • Measure end-to-end time including normalization, phonemization, inference, and file I/O.

Model size, CPU architecture, ONNX provider, text length, voice speed prior, and concurrent workloads all affect latency.

Last updated on