Pocket TTS

OverviewCopy Anchor Link

• PocketTTSService: synthesizes speech locally with Kyutai's Pocket TTS, a 100M-parameter streaming model that runs faster than real time on a CPU. No GPU, no account, no API key. Audio streams in 80 ms frames at 24 kHz.
• TTSModel (@pipedog/pocket-tts/pocket_tts): the engine. Upstream wraps the pocket_tts Python package (PyTorch); this one runs the same model on ONNX Runtime through @pipedog/onnx, with the tokenizer, text chunking, flow-LM loop, LSD sampler and streaming Mimi decoding reimplemented in TypeScript.

At temp: 0 it produces the same audio as pocket-tts 2.1.0 to within 2e-4 per sample. At the default temperature the noise comes from a different generator, so the audio differs sample by sample, as two seeds would.

InstallationCopy Anchor Link

bun add @pipedog/pocket-tts onnxruntime-node

onnxruntime-node is an optional peer of @pipedog/onnx, and it is required here: onnxruntime-web 1.30 can't read the graphs' empty state outputs.

import { PocketTTSService } from "@pipedog/pocket-tts/tts";

What you needCopy Anchor Link

Downloaded once, from Hugging Face, when the pipeline starts (or on the first utterance). No account or token is needed.

To use files you already have, put them in one directory and pass modelOptions: { modelDir }. Anything missing there is downloaded:

// Files you already have: bundle.json, the .onnx graphs, tokenizer.model and
// embeddings/<voice>.safetensors in one directory. Anything missing is downloaded.
const fromDisk = new PocketTTSService({
  modelOptions: { modelDir: "/models/pocket-tts/english" },
  settings: new PocketTTSService.Settings({ voice: "alba" }),
});

modelOptions.hfToken (default $HF_TOKEN) is sent to huggingface.co only, for private or gated files.

RuntimesCopy Anchor Link

Generated from pipecat.runtimes and pipecat.compat in packages/pocket-tts/package.json: ✅ means the compat smoke test passes on that runtime.

Cloudflare Workers is not supported: the flow-LM graph (76 MB to 1.2 GB) and its ~50 MB per-step KV-cache state don't fit a Worker's 128 MB of memory or its 10 MB bundle limit.

On an Apple M4 Pro with Bun and onnxruntime-node, English, 2.24 s of audio takes 0.37 s (6x real time), with the first 80 ms chunk after about 45 ms.

PocketTTSServiceCopy Anchor Link

ConfigurationCopy Anchor Link

modelOptions (pipedog only) takes TTSModel.loadModel's options: backend, sessionOptions (default { intraOpNumThreads: 4, interOpNumThreads: 1 }), shareSessions, modelDir, cacheDir, fetch, hfToken, voiceCloningModel, lsdDecodeSteps (1), noiseClamp (null), eosThreshold (-4) and random.

SettingsCopy Anchor Link

Runtime-updatable settings, passed as settings: new PocketTTSService.Settings({...}) (PocketTTSSettings). Any setting you leave out keeps the default below.

The 26 predefined voices: alba, marius, javert, jean, fantine, cosette, eponine, azelma, anna, vera, charles, paul, george, mary, jane, michael, eve, bill_boerst, peter_yearsley, stuart_bell, caro_davy, giovanni, lola, juergen, rafael, estelle.

UsageCopy Anchor Link

import { TTSUpdateSettingsFrame } from "@pipedog/core/frames/frames";
import { Language } from "@pipedog/core/transcriptions/language";
import { PocketTTSService } from "@pipedog/pocket-tts/tts";

// Downloads the English model (405 MB, or 180 MB with quantize) and the voice on first start.
const tts = new PocketTTSService({
  quantize: true,
  settings: new PocketTTSService.Settings({ voice: "alba", language: Language.EN }),
});

Change the voice mid-call with a TTSUpdateSettingsFrame:

// Queue it on the worker, or push it from a processor upstream of the service.
// The voice is re-derived on the next utterance; the language can't change at runtime.
const update = new TTSUpdateSettingsFrame({
  delta: new PocketTTSService.Settings({ voice: "marius" }),
});

The engine works on its own too:

import { TTSModel } from "@pipedog/pocket-tts/pocket_tts";

const model = await TTSModel.loadModel({ language: "english", temp: 0.7 });
const voice = await model.getStateForAudioPrompt("alba");
for await (const chunk of model.generateAudioStream(voice, "Hello world.")) {
  // chunk: a Float32Array of 1920 samples at model.sampleRate (24000 Hz)
}

NotesCopy Anchor Link

• The constructor rejects a missing voice or language. The model and the voice state load in start() (or on the first utterance).
• Voice cloning from a WAV file needs the Mimi encoder with Kyutai's speaker projection, which only exists in the gated kyutai/pocket-tts weights. It is not downloaded for you: accept Kyutai's terms, export the encoder to ONNX and pass it as modelOptions.voiceCloningModel. This path was not checked against upstream.
• TTSModel.loadModel also accepts german_24l, italian_24l, portuguese_24l and spanish_24l (same export, not checked numerically).

Differences from upstreamCopy Anchor Link

• The model and the voice state load in start(), not in the constructor.
• Sampling noise comes from Math.random (or the random option), not torch's generator.
• quantize uses ONNX Runtime's dynamic int8 quantization of the flow-LM transformer instead of torchao's, so quantized output is not bit-identical to upstream's.
• The exported flow LM has a fixed 1000-position KV cache. A text chunk that would overflow it is cut short with a warning; pocket-tts' 50-token chunks with the predefined voices stay far below it.
• Audio prompts are read from WAV only, and resampled to 24 kHz with soxr.
• TTSModel.loadModel has no config argument (custom YAML model configs).
• modelOptions is a pipedog option: pocket-tts downloads to its own cache.

LicensesCopy Anchor Link

• @pipedog/pocket-tts: BSD 2-Clause. The engine reimplements parts of pocket-tts (MIT) and of SentencePiece's Unigram encoder (Apache-2.0).
• Kyutai's Pocket TTS weights, tokenizers and voices, and the ONNX export: CC-BY-4.0 (attribution: Kyutai). They are downloaded at runtime, not shipped. Kyutai's terms forbid voice impersonation or cloning without consent, among other uses.