# Pocket TTS

> Local text-to-speech with PocketTTSService, running Kyutai's 100M-parameter Pocket TTS on ONNX Runtime, faster than real time on a CPU. No API key.

Source: https://polvallverdu.github.io/pipedog/docs/services/tts/pocket-tts/



## Overview [#overview]

* **`PocketTTSService`**: synthesizes speech locally with Kyutai's
  [Pocket TTS](https://github.com/kyutai-labs/pocket-tts), a 100M-parameter streaming model that
  runs faster than real time on a CPU. No GPU, no account, no API key. Audio streams in 80 ms
  frames at 24 kHz.
* **`TTSModel`** (`@pipedog/pocket-tts/pocket_tts`): the engine. Upstream wraps the `pocket_tts`
  Python package (PyTorch); this one runs the same model on ONNX Runtime through `@pipedog/onnx`,
  with the tokenizer, text chunking, flow-LM loop, LSD sampler and streaming Mimi decoding
  reimplemented in TypeScript.

At `temp: 0` it produces the same audio as pocket-tts 2.1.0 to within 2e-4 per sample. At the
default temperature the noise comes from a different generator, so the audio differs sample by
sample, as two seeds would.

<Cards>
  <Card title="Upstream page" href="https://docs.pipecat.ai/api-reference/server/services/tts/pocket-tts">
    Pipecat's documentation for the same service.
  </Card>

  <Card title="Source" href="https://github.com/polvallverdu/pipedog/tree/main/packages/pocket-tts/src">
    `packages/pocket-tts/src/tts.ts` and the engine, `pocket_tts.ts`.
  </Card>

  <Card title="Pocket TTS" href="https://github.com/kyutai-labs/pocket-tts">
    Kyutai's repository, with the voices and the terms of use.
  </Card>
</Cards>

## Installation [#installation]

```bash
bun add @pipedog/pocket-tts onnxruntime-node
```

`onnxruntime-node` is an optional peer of `@pipedog/onnx`, and it is required here:
`onnxruntime-web` 1.30 can't read the graphs' empty state outputs.

```ts
import { PocketTTSService } from "@pipedog/pocket-tts/tts";
```

## What you need [#what-you-need]

Downloaded once, from Hugging Face, when the pipeline starts (or on the first utterance). No
account or token is needed.

|                      |                                                                                                                                                                                                                                                                                           |
| -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Graphs               | The ONNX export of Kyutai's weights, [`KevinAHM/pocket-tts-onnx`](https://huggingface.co/KevinAHM/pocket-tts-onnx) (pinned revision): `text_conditioner.onnx`, `flow_lm_main.onnx` (or `flow_lm_main_int8.onnx` with `quantize`), `flow_lm_flow.onnx`, `mimi_decoder.onnx`, `bundle.json` |
| Tokenizer and voices | Kyutai's own files, from [`kyutai/pocket-tts-without-voice-cloning`](https://huggingface.co/kyutai/pocket-tts-without-voice-cloning), at the revisions pocket-tts 2.1.0 pins                                                                                                              |
| Sizes                | 6-layer languages: 405 MB, or 180 MB with `quantize`. `french_24l`: 1.31 GB, or 400 MB with `quantize`. About 6 MB per voice                                                                                                                                                              |
| Cache                | `$PIPEDOG_CACHE_DIR/pocket-tts`, else `$XDG_CACHE_HOME/pipedog/pocket-tts`, else `~/.cache/pipedog/pocket-tts`, or `modelOptions.cacheDir`. Files go in `<cacheDir>/<repo>/<revision>/<path>`                                                                                             |

To use files you already have, put them in one directory and pass `modelOptions: { modelDir }`.
Anything missing there is downloaded:

```ts
// Files you already have: bundle.json, the .onnx graphs, tokenizer.model and
// embeddings/<voice>.safetensors in one directory. Anything missing is downloaded.
const fromDisk = new PocketTTSService({
  modelOptions: { modelDir: "/models/pocket-tts/english" },
  settings: new PocketTTSService.Settings({ voice: "alba" }),
});
```

`modelOptions.hfToken` (default `$HF_TOKEN`) is sent to huggingface.co only, for private or gated
files.

## Runtimes [#runtimes]

| Runtime            | Works | What it needs                                                                                                                                                                                                                                                                                                                                                                                               |
| ------------------ | ----- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Bun                | ✅ yes | onnxruntime-node (onnxruntime-web 1.30 can't read the graphs' empty state outputs) and a filesystem for the model cache: the model downloads on first use from Hugging Face (6-layer languages \~405 MB fp32, \~180 MB with quantize; french_24l \~1.3 GB; \~6 MB per voice).                                                                                                                               |
| Node.js            | ✅ yes | onnxruntime-node (onnxruntime-web 1.30 can't read the graphs' empty state outputs) and a filesystem for the model cache: the model downloads on first use from Hugging Face (6-layer languages \~405 MB fp32, \~180 MB with quantize; french_24l \~1.3 GB; \~6 MB per voice).                                                                                                                               |
| Deno               | ✅ yes | onnxruntime-node (onnxruntime-web 1.30 can't read the graphs' empty state outputs) and a filesystem for the model cache: the model downloads on first use from Hugging Face (6-layer languages \~405 MB fp32, \~180 MB with quantize; french_24l \~1.3 GB; \~6 MB per voice).                                                                                                                               |
| Cloudflare Workers | ❌ no  | onnxruntime-node (onnxruntime-web 1.30 can't read the graphs' empty state outputs) and a filesystem for the model cache: the model downloads on first use from Hugging Face (6-layer languages \~405 MB fp32, \~180 MB with quantize; french_24l \~1.3 GB; \~6 MB per voice). Not supported: the flow LM graph (76-1210 MB) and its 50 MB KV-cache state per step exceed Workers' memory and bundle limits. |

Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/pocket-tts/package.json`: ✅ means the compat smoke test passes on that runtime.

Cloudflare Workers is not supported: the flow-LM graph (76 MB to 1.2 GB) and its \~50 MB per-step
KV-cache state don't fit a Worker's 128 MB of memory or its 10 MB bundle limit.

On an Apple M4 Pro with Bun and `onnxruntime-node`, English, 2.24 s of audio takes 0.37 s (6x real
time), with the first 80 ms chunk after about 45 ms.

## PocketTTSService [#pocketttsservice]

### Configuration [#configuration]

| Option | Type | Required | Description |
|---|---|---|---|
| `temp` | `number \| null \| undefined` | no | Sampling temperature for audio generation. Defaults to the pocket-tts model default. |
| `quantize` | `boolean \| undefined` | no | Quantize model weights for faster CPU inference (upstream: requires the `torchao` package; here: the int8 ONNX graph of the flow LM transformer). |
| `settings` | `PocketTTSSettings \| null \| undefined` | no | Runtime-updatable settings. Defaults to the `"alba"` voice and English. |
| `modelOptions` | `Omit<LoadModelOptions, "language" \| "quantize" \| "temp"> \| null \| undefined` | no | Engine options passed to `TTSModel.loadModel()`: ONNX backend, model directory, cache directory, `fetch`, voice-cloning encoder, ... DIVERGENCE: pipedog-specific; pocket-tts downloads to its own cache. |
| `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. |
| `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. |
| `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. |
| `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. |
| `textAggregationMode` | `TextAggregationMode \| null \| undefined` | no | How to aggregate incoming text before synthesis. `TextAggregationMode.SENTENCE` (default) buffers until sentence boundaries, `TextAggregationMode.TOKEN` streams tokens directly for lower latency. |
| `pushTextFrames` | `boolean \| undefined` | no | Whether to push TextFrames and LLMFullResponseEndFrames. |
| `pushStopFrames` | `boolean \| undefined` | no | Whether to automatically push TTSStoppedFrames. |
| `pushStartFrame` | `boolean \| undefined` | no | Whether to automatically create audio contexts and push TTSStartedFrames. When true, the base class handles `createAudioContext` and yields `TTSStartedFrame` before each synthesis call, so `runTts` implementations do not need to. |
| `stopFrameTimeoutS` | `number \| undefined` | no | Idle time before pushing TTSStoppedFrame when pushStopFrames is true. |
| `pushSilenceAfterStop` | `boolean \| undefined` | no | Whether to push silence audio after TTSStoppedFrame. |
| `silenceTimeS` | `number \| undefined` | no | Duration of silence to push when pushSilenceAfterStop is true. |
| `pauseFrameProcessing` | `boolean \| undefined` | no | Whether to hold incoming frames after a turn until its audio contexts end. Needed by services whose audio doesn't carry its request's context ID. |
| `maxConsecutiveZeroAudioContexts` | `number \| undefined` | no | How many consecutive TTS contexts may complete without producing any audio before the service is reported unable to do its job. Catches a provider that accepts requests and stays silent — an unknown voice ID, say — which no error ever surfaces. Every silent context reports an error the service can carry on from, whatever the limit is; on reaching the limit the service reports a permanent error instead, stops being given work, and the pipeline worker applies its `ProcessorUnusablePolicy`. Set to 0 to report silent contexts without ever writing the service off. |
| `appendTrailingSpace` | `boolean \| undefined` | no | Whether to append a trailing space to text before sending to TTS. This helps prevent some TTS services from vocalizing trailing punctuation (e.g., "dot"). Only applied in sentence aggregation mode; when streaming tokens, the incoming text's own whitespace is preserved. |
| `sampleRate` | `number \| null \| undefined` | no | Output sample rate for generated audio. |
| `skipTextTypes` | `string[] \| null \| undefined` | no | List of text types that should not be spoken. |
| `textTransforms` | `[string, TextTransform][] \| null \| undefined` | no | A list of TextTransforms to transform text just before sending it to TTS. Each TextTransform takes the aggregated text and its type, and returns the transformed text. To register, provide a list of tuples of `[textType \| "*", transformFunction]`. |
| `transportDestination` | `string \| null \| undefined` | no | Destination for generated audio frames. |
| `reuseContextIdWithinTurn` | `boolean \| undefined` | no | Whether the service should reuse context IDs within the same turn. |
| `audioCache` | `TTSAudioCacheStore \| null \| undefined` | no | Where to keep synthesized audio for reuse (e.g. a shared `TTSAudioCache`). Text whose audio is in the cache is served from it, with no request to the provider. Lines get there through `warmAudioCache()`, and also from live synthesis when `cacheSynthesizedAudio` allows it. Only services that yield their audio from `runTts` (HTTP and streaming-RPC services, see `supportsAudioCache`) use it, and only in sentence aggregation mode. Defaults to null (no cache). pipedog only. |
| `cacheSynthesizedAudio` | `boolean \| ((text: string) => boolean) \| undefined` | no | Whether audio synthesized live is stored in `audioCache`, or a function that decides per text. Defaults to false: only warmed lines are stored, since a generated reply can repeat what the user said. pipedog only. |
| `audioCacheNamespace` | `string \| undefined` | no | Extra part of the cache key, for constructor options that change the audio but are not in `settings`. pipedog only. |

`modelOptions` (pipedog only) takes `TTSModel.loadModel`'s options: `backend`, `sessionOptions`
(default `{ intraOpNumThreads: 4, interOpNumThreads: 1 }`), `shareSessions`, `modelDir`,
`cacheDir`, `fetch`, `hfToken`, `voiceCloningModel`, `lsdDecodeSteps` (1), `noiseClamp` (`null`),
`eosThreshold` (-4) and `random`.

### Settings [#settings]

Runtime-updatable settings, passed as `settings: new PocketTTSService.Settings({...})`
(`PocketTTSSettings`). Any setting you leave out keeps the default below.

| Setting    | Type       | Default       | Description                                                                                                                                                                                                                                    |
| ---------- | ---------- | ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `model`    | `null`     | `null`        | Not used.                                                                                                                                                                                                                                      |
| `voice`    | `string`   | `"alba"`      | A predefined voice, a `.safetensors` voice state, or a WAV file to clone (path, `https://` or `hf://` URL). Re-derived on the next utterance when it changes.                                                                                  |
| `language` | `Language` | `Language.EN` | `EN`, `DE`, `ES`, `IT`, `PT` or `FR` (and regional variants), mapped to `english`, `german`, `spanish`, `italian`, `portuguese` and `french_24l`. Stored but not applied after construction, as upstream: the weights are loaded per language. |

The 26 predefined voices: `alba`, `marius`, `javert`, `jean`, `fantine`, `cosette`, `eponine`,
`azelma`, `anna`, `vera`, `charles`, `paul`, `george`, `mary`, `jane`, `michael`, `eve`,
`bill_boerst`, `peter_yearsley`, `stuart_bell`, `caro_davy`, `giovanni`, `lola`, `juergen`,
`rafael`, `estelle`.

### Usage [#usage]

```ts title="tts.ts"
import { TTSUpdateSettingsFrame } from "@pipedog/core/frames/frames";
import { Language } from "@pipedog/core/transcriptions/language";
import { PocketTTSService } from "@pipedog/pocket-tts/tts";

// Downloads the English model (405 MB, or 180 MB with quantize) and the voice on first start.
const tts = new PocketTTSService({
  quantize: true,
  settings: new PocketTTSService.Settings({ voice: "alba", language: Language.EN }),
});
```

Change the voice mid-call with a `TTSUpdateSettingsFrame`:

```ts
// Queue it on the worker, or push it from a processor upstream of the service.
// The voice is re-derived on the next utterance; the language can't change at runtime.
const update = new TTSUpdateSettingsFrame({
  delta: new PocketTTSService.Settings({ voice: "marius" }),
});

```

The engine works on its own too:

```ts
import { TTSModel } from "@pipedog/pocket-tts/pocket_tts";

const model = await TTSModel.loadModel({ language: "english", temp: 0.7 });
const voice = await model.getStateForAudioPrompt("alba");
for await (const chunk of model.generateAudioStream(voice, "Hello world.")) {
  // chunk: a Float32Array of 1920 samples at model.sampleRate (24000 Hz)
}
```

### Notes [#notes]

* The constructor rejects a missing voice or language. The model and the voice state load in
  `start()` (or on the first utterance).
* Voice cloning from a WAV file needs the Mimi encoder with Kyutai's speaker projection, which only
  exists in the **gated** `kyutai/pocket-tts` weights. It is not downloaded for you: accept Kyutai's
  terms, export the encoder to ONNX and pass it as `modelOptions.voiceCloningModel`. This path was
  not checked against upstream.
* `TTSModel.loadModel` also accepts `german_24l`, `italian_24l`, `portuguese_24l` and
  `spanish_24l` (same export, not checked numerically).

## Differences from upstream [#differences-from-upstream]

* The model and the voice state load in `start()`, not in the constructor.
* Sampling noise comes from `Math.random` (or the `random` option), not torch's generator.
* `quantize` uses ONNX Runtime's dynamic int8 quantization of the flow-LM transformer instead of
  torchao's, so quantized output is not bit-identical to upstream's.
* The exported flow LM has a fixed 1000-position KV cache. A text chunk that would overflow it is
  cut short with a warning; pocket-tts' 50-token chunks with the predefined voices stay far below it.
* Audio prompts are read from WAV only, and resampled to 24 kHz with soxr.
* `TTSModel.loadModel` has no `config` argument (custom YAML model configs).
* `modelOptions` is a pipedog option: pocket-tts downloads to its own cache.

## Licenses [#licenses]

* `@pipedog/pocket-tts`: BSD 2-Clause. The engine reimplements parts of pocket-tts (MIT) and of
  SentencePiece's Unigram encoder (Apache-2.0).
* Kyutai's Pocket TTS weights, tokenizers and voices, and the ONNX export: CC-BY-4.0 (attribution:
  Kyutai). They are downloaded at runtime, not shipped. Kyutai's terms forbid voice impersonation
  or cloning without consent, among other uses.