# Fish Audio

> Text-to-speech with FishAudioTTSService, Fish Audio's MessagePack WebSocket API, with cloned-voice support.

Source: https://polvallverdu.github.io/pipedog/docs/services/tts/fish/



## Overview [#overview]

Fish Audio turns text into speech, including with cloned voices. pipedog ports Pipecat's
**`FishAudioTTSService`**: it streams over Fish Audio's real-time WebSocket
(`wss://api.fish.audio/v1/tts/live`), speaking MessagePack, and reconnects when the user
interrupts the bot.

<Cards>
  <Card title="Upstream page" href="https://docs.pipecat.ai/api-reference/server/services/tts/fish">
    Pipecat's documentation for the same service.
  </Card>

  <Card title="Source" href="https://github.com/polvallverdu/pipedog/tree/main/packages/fish/src">
    `packages/fish/src/tts.ts`.
  </Card>

  <Card title="Fish Audio docs" href="https://docs.fish.audio/api-reference/endpoint/websocket/tts-live">
    The real-time WebSocket protocol, models and voices.
  </Card>
</Cards>

## Installation [#installation]

```bash
bun add @pipedog/fish
```

```ts
import { FishAudioTTSService } from "@pipedog/fish/tts";
```

## Prerequisites [#prerequisites]

* A Fish Audio account and an API key.
* A voice model's reference id, set as `voice`. It is sent as `reference_id`.
* Pass the key as `apiKey`. The examples read it from `FISH_API_KEY`.

## FishAudioTTSService [#fishaudiottsservice]

### Configuration [#configuration]

| Option | Type | Required | Description |
|---|---|---|---|
| `pushTextFrames` | `boolean \| undefined` | no | Whether to push TextFrames and LLMFullResponseEndFrames. |
| `pushStopFrames` | `boolean \| undefined` | no | Whether to automatically push TTSStoppedFrames. |
| `apiKey` | `string` | yes | Fish Audio API key for authentication. |
| `outputFormat` | `FishAudioOutputFormat \| undefined` | no | Audio output format. Defaults to "pcm". |
| `settings` | `FishAudioTTSSettings \| null \| undefined` | no | Runtime-updatable settings (defaults: model "s2.1-pro"). |
| `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. |
| `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. |
| `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. |
| `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. |
| `reconnectBackoffMinWait` | `number \| undefined` | no | Minimum time, in seconds, to wait between reconnection attempts. |
| `reconnectBackoffMaxWait` | `number \| undefined` | no | Maximum time, in seconds, to wait between reconnection attempts. |
| `reconnectOnError` | `boolean \| undefined` | no | Whether to automatically reconnect on connection errors. |
| `wsCloseTimeout` | `number \| undefined` | no | Maximum time, in seconds, to wait for the peer to acknowledge the websocket closing handshake before dropping the connection. Applied to connections opened through `_websocketConnect()`. Increase it for peers that need longer to complete a graceful close. |
| `wsSendTimeout` | `number \| null \| undefined` | no | Maximum time, in seconds, a single send may wait on a peer that has stopped reading before the connection is dropped and the send throws `ConnectionClosedError`. Applied to connections opened through `_websocketConnect()`. Pass `null` to wait indefinitely. Defaults to `WS_SEND_TIMEOUT`. |
| `textAggregationMode` | `TextAggregationMode \| null \| undefined` | no | How to aggregate incoming text before synthesis. `TextAggregationMode.SENTENCE` (default) buffers until sentence boundaries, `TextAggregationMode.TOKEN` streams tokens directly for lower latency. |
| `pushStartFrame` | `boolean \| undefined` | no | Whether to automatically create audio contexts and push TTSStartedFrames. When true, the base class handles `createAudioContext` and yields `TTSStartedFrame` before each synthesis call, so `runTts` implementations do not need to. |
| `stopFrameTimeoutS` | `number \| undefined` | no | Idle time before pushing TTSStoppedFrame when pushStopFrames is true. |
| `pushSilenceAfterStop` | `boolean \| undefined` | no | Whether to push silence audio after TTSStoppedFrame. |
| `silenceTimeS` | `number \| undefined` | no | Duration of silence to push when pushSilenceAfterStop is true. |
| `pauseFrameProcessing` | `boolean \| undefined` | no | Whether to hold incoming frames after a turn until its audio contexts end. Needed by services whose audio doesn't carry its request's context ID. |
| `maxConsecutiveZeroAudioContexts` | `number \| undefined` | no | How many consecutive TTS contexts may complete without producing any audio before the service is reported unable to do its job. Catches a provider that accepts requests and stays silent — an unknown voice ID, say — which no error ever surfaces. Every silent context reports an error the service can carry on from, whatever the limit is; on reaching the limit the service reports a permanent error instead, stops being given work, and the pipeline worker applies its `ProcessorUnusablePolicy`. Set to 0 to report silent contexts without ever writing the service off. |
| `appendTrailingSpace` | `boolean \| undefined` | no | Whether to append a trailing space to text before sending to TTS. This helps prevent some TTS services from vocalizing trailing punctuation (e.g., "dot"). Only applied in sentence aggregation mode; when streaming tokens, the incoming text's own whitespace is preserved. |
| `sampleRate` | `number \| null \| undefined` | no | Output sample rate for generated audio. |
| `skipTextTypes` | `string[] \| null \| undefined` | no | List of text types that should not be spoken. |
| `textTransforms` | `[string, TextTransform][] \| null \| undefined` | no | A list of TextTransforms to transform text just before sending it to TTS. Each TextTransform takes the aggregated text and its type, and returns the transformed text. To register, provide a list of tuples of `[textType \| "*", transformFunction]`. |
| `transportDestination` | `string \| null \| undefined` | no | Destination for generated audio frames. |
| `reuseContextIdWithinTurn` | `boolean \| undefined` | no | Whether the service should reuse context IDs within the same turn. |
| `audioCache` | `TTSAudioCacheStore \| null \| undefined` | no | Where to keep synthesized audio for reuse (e.g. a shared `TTSAudioCache`). Text whose audio is in the cache is served from it, with no request to the provider. Lines get there through `warmAudioCache()`, and also from live synthesis when `cacheSynthesizedAudio` allows it. Only services that yield their audio from `runTts` (HTTP and streaming-RPC services, see `supportsAudioCache`) use it, and only in sentence aggregation mode. Defaults to null (no cache). pipedog only. |
| `cacheSynthesizedAudio` | `boolean \| ((text: string) => boolean) \| undefined` | no | Whether audio synthesized live is stored in `audioCache`, or a function that decides per text. Defaults to false: only warmed lines are stored, since a generated reply can repeat what the user said. pipedog only. |
| `audioCacheNamespace` | `string \| undefined` | no | Extra part of the cache key, for constructor options that change the audio but are not in `settings`. pipedog only. |

### Settings [#settings]

Runtime-updatable settings, passed as `settings: new FishAudioTTSService.Settings({...})`.

| Setting                     | Type              | Default      | Description                                                                                      |
| --------------------------- | ----------------- | ------------ | ------------------------------------------------------------------------------------------------ |
| `model`                     | `string`          | `"s2.1-pro"` | Fish Audio model, sent as the `model` handshake header: `s2.1-pro`, `s2-pro`, `s1` or `drama-3`. |
| `voice`                     | `string \| null`  | `null`       | Voice model reference id.                                                                        |
| `latency`                   | `string \| null`  | `"balanced"` | `"normal"` or `"balanced"`.                                                                      |
| `normalize`                 | `boolean \| null` | `true`       | Normalize the text.                                                                              |
| `temperature`               | `number \| null`  | `null`       | Randomness, 0.0 to 1.0.                                                                          |
| `topP`                      | `number \| null`  | `null`       | Nucleus sampling, 0.0 to 1.0.                                                                    |
| `prosodySpeed`              | `number \| null`  | `1.0`        | Speech speed, 0.5 to 2.0.                                                                        |
| `prosodyVolume`             | `number \| null`  | `0`          | Volume in dB, -20 to 20.                                                                         |
| `prosodyNormalizeLoudness`  | `boolean \| null` | `null`       | Even out the perceived volume (S2 family and drama-3).                                           |
| `chunkLength`               | `number \| null`  | `null`       | Text segment size, 100 to 300.                                                                   |
| `minChunkLength`            | `number \| null`  | `null`       | Minimum characters before a new chunk, 0 to 100.                                                 |
| `conditionOnPreviousChunks` | `boolean \| null` | `null`       | Use earlier audio as context, for a consistent voice.                                            |

Settings left `null` are not sent, so Fish Audio applies its own defaults.
`Settings.fromMapping()` also accepts upstream's nested `prosody` dict.

### Usage [#usage]

```ts title="tts.ts"
import { TTSUpdateSettingsFrame } from "@pipedog/core/frames/frames";
import { FishAudioTTSService } from "@pipedog/fish/tts";

const tts = new FishAudioTTSService({
  apiKey: process.env.FISH_API_KEY ?? "",
  settings: new FishAudioTTSService.Settings({
    // A voice model's reference id from the Fish Audio library.
    voice: "your-reference-id",
    model: "s2.1-pro",
    latency: "balanced",
    prosodySpeed: 1.1,
  }),
});
```

Change settings mid-call with a `TTSUpdateSettingsFrame`:

```ts
// Queue it on the worker, or push it from a processor upstream of the service.
// A change to any setting reconnects, because they are all sent in the start message.
const update = new TTSUpdateSettingsFrame({
  delta: new FishAudioTTSService.Settings({ prosodySpeed: 0.9, prosodyVolume: 2 }),
});
```

### Event handlers [#event-handlers]

Inherited from `TTSService`:

| Event                 | Description                                     |
| --------------------- | ----------------------------------------------- |
| `on_connected`        | Connected to Fish Audio.                        |
| `on_disconnected`     | Disconnected from Fish Audio.                   |
| `on_connection_error` | The connection failed.                          |
| `on_tts_request`      | A text chunk is about to be sent for synthesis. |

### Notes [#notes]

* Each text chunk is sent with a `flush`, which makes Fish Audio generate the audio right away.
* The service reconnects on an interruption while the bot is speaking, and on any settings change.
* `outputFormat` is `"pcm"` by default. The service emits the audio as it arrives and does not
  produce word timestamps.

## Runtime notes [#runtime-notes]

* The socket is the global `WebSocket`. The `Authorization` and `model` headers go on the
  handshake, which works on Bun, Node and Deno, and on Cloudflare Workers, where the socket is
  opened through `fetch`.
* Upstream packs messages with `ormsgpack`. pipedog uses
  [`@msgpack/msgpack`](https://github.com/msgpack/msgpack-javascript), which is pure TypeScript.
  It writes a whole-number float such as `prosodySpeed: 1.0` as a MessagePack integer.
* Not ported: the deprecated `InputParams` and the `referenceId`, `modelId` and `params`
  constructor options. Use `settings`.

| Runtime            | Works | What it needs |
| ------------------ | ----- | ------------- |
| Bun                | ✅ yes | Nothing.      |
| Node.js            | ✅ yes | Nothing.      |
| Deno               | ✅ yes | Nothing.      |
| Cloudflare Workers | ✅ yes | Nothing.      |

Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/fish/package.json`: ✅ means the compat smoke test passes on that runtime.