# Kokoro

> Local text-to-speech with KokoroTTSService, running the Kokoro-82M ONNX model on your machine with eSpeak NG phonemization. No API key.

Source: https://polvallverdu.github.io/pipedog/docs/services/tts/kokoro/



## Overview [#overview]

* **`KokoroTTSService`**: synthesizes each sentence locally with
  [Kokoro-82M](https://huggingface.co/hexgrad/Kokoro-82M), an 82M-parameter model that runs on the
  CPU, and streams 24 kHz audio resampled to the pipeline's rate. No account, no API key.
* **`Kokoro`**: the engine behind it, a TypeScript reimplementation of the Python
  [kokoro-onnx](https://github.com/thewh1teagle/kokoro-onnx) package (0.6.1) on `@pipedog/onnx` and
  `@pipedog/espeak-ng`. Its audio is bit-identical to kokoro-onnx's for the int8 model.

Kokoro is one of upstream's keyless local defaults for [evals](https://polvallverdu.github.io/pipedog/docs/evals.md): it speaks the
simulated user's turns in audio scenarios (`service: kokoro`).

<Cards>
  <Card title="Upstream page" href="https://docs.pipecat.ai/api-reference/server/services/tts/kokoro">
    Pipecat's documentation for the same service.
  </Card>

  <Card title="Source" href="https://github.com/polvallverdu/pipedog/tree/main/packages/kokoro/src">
    `packages/kokoro/src/tts.ts` and the engine, `kokoro_onnx.ts`.
  </Card>

  <Card title="Kokoro-82M" href="https://huggingface.co/hexgrad/Kokoro-82M">
    The model card, with its voices and languages.
  </Card>
</Cards>

## Installation [#installation]

```bash
bun add @pipedog/kokoro onnxruntime-node @echogarden/espeak-ng-emscripten
```

* `onnxruntime-node` runs the model. `onnxruntime-web@1.30.0` (WebAssembly) works too, slower.
* `@echogarden/espeak-ng-emscripten` is eSpeak NG compiled to WebAssembly, for phonemization. It
  is GPL-3.0 (see [Licenses](#licenses)).

Both are optional peer dependencies, loaded on first use.

```ts
import { KokoroTTSService } from "@pipedog/kokoro/tts";
```

## What you need [#what-you-need]

The same files upstream uses, from kokoro-onnx's `model-files-v1.0` GitHub release. They are
downloaded once, when the pipeline starts, if they are missing:

| File                          | Size    | Default location           |
| ----------------------------- | ------- | -------------------------- |
| `kokoro-v1.0.onnx`            | 310 MiB | `<cache>/kokoro-v1.0.onnx` |
| `voices-v1.0.bin` (54 voices) | 27 MiB  | `<cache>/voices-v1.0.bin`  |

`<cache>` is `$PIPEDOG_CACHE_DIR/kokoro`, else `$XDG_CACHE_HOME/pipedog/kokoro`, else
`~/.cache/pipedog/kokoro`. Pass `cacheDir` to use another directory.

To use files you already have, pass `modelPath` and `voicesPath`. A path that exists is used as is;
a missing one is downloaded there from `modelUrl` / `voicesUrl`. The same release has smaller
variants of the model, such as `kokoro-v1.0.int8.onnx` (88 MiB):

```ts
// Files you already have, or the smaller int8 model from the same release.
const int8 = new KokoroTTSService({
  modelPath: "/models/kokoro-v1.0.int8.onnx", // used as is if present...
  modelUrl:
    "https://github.com/thewh1teagle/kokoro-onnx/releases/download/model-files-v1.0/kokoro-v1.0.int8.onnx", // ...else downloaded from here
  voicesPath: "/models/voices-v1.0.bin",
  settings: new KokoroTTSService.Settings({ voice: "af_heart" }),
});
```

eSpeak NG adds about 25 MB (the WebAssembly build and its voice data), loaded into memory once per
process. `@pipedog/espeak-ng` wraps it: `phonemize()` reproduces the Python `phonemizer` package
that kokoro-onnx uses, so the same text gives the same phonemes.

## Runtimes [#runtimes]

| Runtime            | Works | What it needs                                                                                                                                                                                                                                           |
| ------------------ | ----- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Bun                | ✅ yes | onnxruntime-node (or onnxruntime-web) and @echogarden/espeak-ng-emscripten; the Kokoro model (310 MB, or 88 MB int8) and voices (27 MB), downloaded on first use.                                                                                       |
| Node.js            | ✅ yes | onnxruntime-node (or onnxruntime-web) and @echogarden/espeak-ng-emscripten; the Kokoro model (310 MB, or 88 MB int8) and voices (27 MB), downloaded on first use.                                                                                       |
| Deno               | ✅ yes | onnxruntime-node (or onnxruntime-web) and @echogarden/espeak-ng-emscripten; the Kokoro model (310 MB, or 88 MB int8) and voices (27 MB), downloaded on first use.                                                                                       |
| Cloudflare Workers | ❌ no  | onnxruntime-node (or onnxruntime-web) and @echogarden/espeak-ng-emscripten; the Kokoro model (310 MB, or 88 MB int8) and voices (27 MB), downloaded on first use. Not supported: the model and eSpeak NG data exceed Workers' memory and bundle limits. |

Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/kokoro/package.json`: ✅ means the compat smoke test passes on that runtime.

Cloudflare Workers is not supported: the model (88-310 MiB), the voices and eSpeak NG's data don't
fit a Worker's 128 MB of memory or its 10 MB bundle limit.

## KokoroTTSService [#kokorottsservice]

### Configuration [#configuration]

| Option | Type | Required | Description |
|---|---|---|---|
| `textAggregationMode` | `TextAggregationMode \| null \| undefined` | no | How to aggregate incoming text before synthesis. `TextAggregationMode.SENTENCE` (default) buffers until sentence boundaries, `TextAggregationMode.TOKEN` streams tokens directly for lower latency. |
| `modelPath` | `string \| null \| undefined` | no | Path to the kokoro ONNX model file. Defaults to the auto-downloaded file. |
| `voicesPath` | `string \| null \| undefined` | no | Path to the voices binary file. Defaults to the auto-downloaded file. |
| `cacheDir` | `string \| null \| undefined` | no | Directory for the auto-downloaded files (default `kokoroCacheDir()`). Not upstream. |
| `modelUrl` | `string \| null \| undefined` | no | Where to download a missing model from (default `KOKORO_MODEL_URL`). Not upstream. |
| `voicesUrl` | `string \| null \| undefined` | no | Where to download a missing voices file from (default `KOKORO_VOICES_URL`). Not upstream. |
| `fetch` | `((input: RequestInfo \| URL, init?: RequestInit) => Promise<Response>) \| null \| undefined` | no | `fetch` for downloads (default: the global `fetch`). Not upstream. |
| `backend` | `OnnxBackend \| Promise<OnnxBackend> \| null \| undefined` | no | The ONNX backend (default: `createOnnxBackend()`). Not upstream. |
| `shareSession` | `boolean \| undefined` | no | Share the model session and voices across services (default `true`). Not upstream. |
| `settings` | `KokoroTTSSettings \| null \| undefined` | no | Runtime-updatable settings. |
| `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. |
| `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. |
| `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. |
| `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. |
| `pushTextFrames` | `boolean \| undefined` | no | Whether to push TextFrames and LLMFullResponseEndFrames. |
| `pushStopFrames` | `boolean \| undefined` | no | Whether to automatically push TTSStoppedFrames. |
| `pushStartFrame` | `boolean \| undefined` | no | Whether to automatically create audio contexts and push TTSStartedFrames. When true, the base class handles `createAudioContext` and yields `TTSStartedFrame` before each synthesis call, so `runTts` implementations do not need to. |
| `stopFrameTimeoutS` | `number \| undefined` | no | Idle time before pushing TTSStoppedFrame when pushStopFrames is true. |
| `pushSilenceAfterStop` | `boolean \| undefined` | no | Whether to push silence audio after TTSStoppedFrame. |
| `silenceTimeS` | `number \| undefined` | no | Duration of silence to push when pushSilenceAfterStop is true. |
| `pauseFrameProcessing` | `boolean \| undefined` | no | Whether to hold incoming frames after a turn until its audio contexts end. Needed by services whose audio doesn't carry its request's context ID. |
| `maxConsecutiveZeroAudioContexts` | `number \| undefined` | no | How many consecutive TTS contexts may complete without producing any audio before the service is reported unable to do its job. Catches a provider that accepts requests and stays silent — an unknown voice ID, say — which no error ever surfaces. Every silent context reports an error the service can carry on from, whatever the limit is; on reaching the limit the service reports a permanent error instead, stops being given work, and the pipeline worker applies its `ProcessorUnusablePolicy`. Set to 0 to report silent contexts without ever writing the service off. |
| `appendTrailingSpace` | `boolean \| undefined` | no | Whether to append a trailing space to text before sending to TTS. This helps prevent some TTS services from vocalizing trailing punctuation (e.g., "dot"). Only applied in sentence aggregation mode; when streaming tokens, the incoming text's own whitespace is preserved. |
| `sampleRate` | `number \| null \| undefined` | no | Output sample rate for generated audio. |
| `skipTextTypes` | `string[] \| null \| undefined` | no | List of text types that should not be spoken. |
| `textTransforms` | `[string, TextTransform][] \| null \| undefined` | no | A list of TextTransforms to transform text just before sending it to TTS. Each TextTransform takes the aggregated text and its type, and returns the transformed text. To register, provide a list of tuples of `[textType \| "*", transformFunction]`. |
| `transportDestination` | `string \| null \| undefined` | no | Destination for generated audio frames. |
| `reuseContextIdWithinTurn` | `boolean \| undefined` | no | Whether the service should reuse context IDs within the same turn. |
| `audioCache` | `TTSAudioCacheStore \| null \| undefined` | no | Where to keep synthesized audio for reuse (e.g. a shared `TTSAudioCache`). Text whose audio is in the cache is served from it, with no request to the provider. Lines get there through `warmAudioCache()`, and also from live synthesis when `cacheSynthesizedAudio` allows it. Only services that yield their audio from `runTts` (HTTP and streaming-RPC services, see `supportsAudioCache`) use it, and only in sentence aggregation mode. Defaults to null (no cache). pipedog only. |
| `cacheSynthesizedAudio` | `boolean \| ((text: string) => boolean) \| undefined` | no | Whether audio synthesized live is stored in `audioCache`, or a function that decides per text. Defaults to false: only warmed lines are stored, since a generated reply can repeat what the user said. pipedog only. |
| `audioCacheNamespace` | `string \| undefined` | no | Extra part of the cache key, for constructor options that change the audio but are not in `settings`. pipedog only. |

### Settings [#settings]

Runtime-updatable settings, passed as `settings: new KokoroTTSService.Settings({...})`
(`KokoroTTSSettings`). Any setting you leave out keeps the default below.

| Setting    | Type       | Default        | Description                                           |
| ---------- | ---------- | -------------- | ----------------------------------------------------- |
| `model`    | `null`     | `null`         | Not used: the model is `modelPath`.                   |
| `voice`    | `string`   | none, required | Voice from the voices file, for example `"af_heart"`. |
| `language` | `Language` | `Language.EN`  | Language, mapped to an espeak-ng code.                |
| `speed`    | `number`   | `1.0`          | Speech rate multiplier, 0.5 to 2.0.                   |

Languages: English (US and GB), Spanish, French, Hindi, Italian, Japanese, Portuguese (PT and BR)
and Mandarin. Pick a voice for the language: `af_*` / `am_*` American English, `bf_*` / `bm_*`
British, `ef_*` Spanish, `ff_*` French, `hf_*` Hindi, `if_*` Italian, `jf_*` Japanese, `pf_*`
Portuguese, `zf_*` Mandarin.

### Usage [#usage]

```ts title="tts.ts"
import { TTSUpdateSettingsFrame } from "@pipedog/core/frames/frames";
import { Language } from "@pipedog/core/transcriptions/language";
import { KokoroTTSService } from "@pipedog/kokoro/tts";

// Downloads kokoro-v1.0.onnx (310 MiB) and voices-v1.0.bin (27 MiB) on first start.
const tts = new KokoroTTSService({
  settings: new KokoroTTSService.Settings({ voice: "af_heart", language: Language.EN, speed: 1.0 }),
});
```

Change the voice, language or speed mid-call with a `TTSUpdateSettingsFrame`:

```ts
// Queue it on the worker, or push it from a processor upstream of the service.
const update = new TTSUpdateSettingsFrame({
  delta: new KokoroTTSService.Settings({ voice: "bf_emma", language: Language.EN_GB, speed: 1.1 }),
});

```

The engine works on its own too:

```ts
import { Kokoro } from "@pipedog/kokoro/kokoro_onnx";

const kokoro = await Kokoro.load("/models/kokoro-v1.0.onnx", "/models/voices-v1.0.bin");
for await (const [samples, sampleRate] of kokoro.createStream("Hello there!", {
  voice: "af_heart",
  lang: "en-us",
})) {
  // samples: a Float32Array at sampleRate (24000 Hz), one chunk per phoneme batch
}
```

### Notes [#notes]

* The model and voices load in `start()` (or on the first sentence). Services in one process
  share one model session and one copy of the voices; set `shareSession: false` to opt out.
* A missing or blank voice is rejected by the constructor. A voice that isn't in the voices file
  is reported when the model loads.
* Text is phonemized with espeak-ng, split into batches of at most 510 phonemes at sentence,
  clause and word boundaries, and the next batch is synthesized while the current one plays.
* `backend` takes any `OnnxBackend`, for example one built on `onnxruntime-web`.

## Differences from upstream [#differences-from-upstream]

* The model is downloaded and loaded in `start()`, not in the constructor, since loading is
  asynchronous.
* Default cache directory: `~/.cache/pipedog/kokoro`, where upstream uses
  `~/.cache/pipecat/kokoro-onnx`.
* `cacheDir`, `modelUrl`, `voicesUrl`, `fetch`, `backend` and `shareSession` are pipedog options.
* The deprecated `voice_id` and `params` (`InputParams`) arguments are not ported. Use `settings`.
* Not ported from kokoro-onnx: timed synthesis (`create_timed`), which needs a re-exported model.

## Licenses [#licenses]

* `@pipedog/kokoro`: BSD 2-Clause.
* Kokoro-82M model and voices: Apache-2.0 (hexgrad). The ONNX export and voice pack come from
  kokoro-onnx (MIT).
* eSpeak NG and `@echogarden/espeak-ng-emscripten`: **GPL-3.0**. Distributing an application that
  includes them may subject it to the GPL's terms. Upstream is in the same position: kokoro-onnx
  ships espeak-ng through espeakng-loader.