# Cloudflare Workers AI

> Text-to-speech on Workers AI: Deepgram Aura streaming over a WebSocket or per sentence over HTTP, and MeloTTS, over the AI binding or the REST API.

Source: https://polvallverdu.github.io/pipedog/docs/services/tts/cloudflare/



## Overview [#overview]

`@pipedog/cloudflare` has three text-to-speech services on Workers AI:

* **`WorkersAIAuraTTSService`**: Deepgram Aura (`@cf/deepgram/aura-2-en`, `aura-2-es`, `aura-1`),
  streamed over a WebSocket. Workers AI carries Deepgram's `/v1/speak` protocol, so this is
  [`DeepgramTTSService`](https://polvallverdu.github.io/pipedog/docs/services/tts/deepgram.md) opening its socket through Workers AI.
* **`WorkersAIAuraHttpTTSService`**: Aura, one request per sentence, the PCM streamed back.
* **`WorkersAIMeloTTSService`**: MeloTTS (`@cf/myshell-ai/melotts`), one request per sentence. It
  answers in MP3, so it needs a decoder you pass in.

This package is pipedog's own: Pipecat has no Workers AI services.

<Cards>
  <Card title="Source" href="https://github.com/polvallverdu/pipedog/tree/main/packages/cloudflare/src">
    `packages/cloudflare/src/tts.ts` and `melotts.ts`.
  </Card>

  <Card title="Workers AI models" href="https://developers.cloudflare.com/workers-ai/models/">
    Speakers, inputs and pricing of each model.
  </Card>
</Cards>

## Installation [#installation]

```bash
bun add @pipedog/cloudflare
```

```ts
import { WorkersAIAuraHttpTTSService, WorkersAIAuraTTSService } from "@pipedog/cloudflare/tts";
import { WorkersAIMeloTTSService } from "@pipedog/cloudflare/melotts";
```

## Prerequisites [#prerequisites]

Inside a Worker, pass the AI binding as `ai: env.AI`. Elsewhere, pass `accountId` and `apiToken`
(an API token with the Workers AI permission). See
[the speech-to-text page](https://polvallverdu.github.io/pipedog/docs/services/stt/cloudflare/#prerequisites) for both paths.

## WorkersAIAuraTTSService [#workersaiaurattsservice]

### Configuration [#configuration]

| Option | Type | Required | Description |
|---|---|---|---|
| `settings` | `DeepgramTTSSettings \| null \| undefined` | no | Runtime-updatable settings. `model` is the Workers AI model ID (default `"@cf/deepgram/aura-2-en"`), `voice` the speaker (default: the model's). |
| `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. |
| `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. |
| `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. |
| `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. |
| `reconnectBackoffMinWait` | `number \| undefined` | no | Minimum time, in seconds, to wait between reconnection attempts. |
| `reconnectBackoffMaxWait` | `number \| undefined` | no | Maximum time, in seconds, to wait between reconnection attempts. |
| `reconnectOnError` | `boolean \| undefined` | no | Whether to automatically reconnect on connection errors. |
| `wsCloseTimeout` | `number \| undefined` | no | Maximum time, in seconds, to wait for the peer to acknowledge the websocket closing handshake before dropping the connection. Applied to connections opened through `_websocketConnect()`. Increase it for peers that need longer to complete a graceful close. |
| `wsSendTimeout` | `number \| null \| undefined` | no | Maximum time, in seconds, a single send may wait on a peer that has stopped reading before the connection is dropped and the send throws `ConnectionClosedError`. Applied to connections opened through `_websocketConnect()`. Pass `null` to wait indefinitely. Defaults to `WS_SEND_TIMEOUT`. |
| `textAggregationMode` | `TextAggregationMode \| null \| undefined` | no | How to aggregate incoming text before synthesis. `TextAggregationMode.SENTENCE` (default) buffers until sentence boundaries, `TextAggregationMode.TOKEN` streams tokens directly for lower latency. |
| `pushTextFrames` | `boolean \| undefined` | no | Whether to push TextFrames and LLMFullResponseEndFrames. |
| `pushStopFrames` | `boolean \| undefined` | no | Whether to automatically push TTSStoppedFrames. |
| `pushStartFrame` | `boolean \| undefined` | no | Whether to automatically create audio contexts and push TTSStartedFrames. When true, the base class handles `createAudioContext` and yields `TTSStartedFrame` before each synthesis call, so `runTts` implementations do not need to. |
| `stopFrameTimeoutS` | `number \| undefined` | no | Idle time before pushing TTSStoppedFrame when pushStopFrames is true. |
| `pushSilenceAfterStop` | `boolean \| undefined` | no | Whether to push silence audio after TTSStoppedFrame. |
| `silenceTimeS` | `number \| undefined` | no | Duration of silence to push when pushSilenceAfterStop is true. |
| `pauseFrameProcessing` | `boolean \| undefined` | no | Whether to hold incoming frames after a turn until its audio contexts end. Needed by services whose audio doesn't carry its request's context ID. |
| `maxConsecutiveZeroAudioContexts` | `number \| undefined` | no | How many consecutive TTS contexts may complete without producing any audio before the service is reported unable to do its job. Catches a provider that accepts requests and stays silent — an unknown voice ID, say — which no error ever surfaces. Every silent context reports an error the service can carry on from, whatever the limit is; on reaching the limit the service reports a permanent error instead, stops being given work, and the pipeline worker applies its `ProcessorUnusablePolicy`. Set to 0 to report silent contexts without ever writing the service off. |
| `appendTrailingSpace` | `boolean \| undefined` | no | Whether to append a trailing space to text before sending to TTS. This helps prevent some TTS services from vocalizing trailing punctuation (e.g., "dot"). Only applied in sentence aggregation mode; when streaming tokens, the incoming text's own whitespace is preserved. |
| `sampleRate` | `number \| null \| undefined` | no | Output sample rate for generated audio. |
| `skipTextTypes` | `string[] \| null \| undefined` | no | List of text types that should not be spoken. |
| `textTransforms` | `[string, TextTransform][] \| null \| undefined` | no | A list of TextTransforms to transform text just before sending it to TTS. Each TextTransform takes the aggregated text and its type, and returns the transformed text. To register, provide a list of tuples of `[textType \| "*", transformFunction]`. |
| `transportDestination` | `string \| null \| undefined` | no | Destination for generated audio frames. |
| `reuseContextIdWithinTurn` | `boolean \| undefined` | no | Whether the service should reuse context IDs within the same turn. |
| `audioCache` | `TTSAudioCacheStore \| null \| undefined` | no | Where to keep synthesized audio for reuse (e.g. a shared `TTSAudioCache`). Text whose audio is in the cache is served from it, with no request to the provider. Lines get there through `warmAudioCache()`, and also from live synthesis when `cacheSynthesizedAudio` allows it. Only services that yield their audio from `runTts` (HTTP and streaming-RPC services, see `supportsAudioCache`) use it, and only in sentence aggregation mode. Defaults to null (no cache). pipedog only. |
| `cacheSynthesizedAudio` | `boolean \| ((text: string) => boolean) \| undefined` | no | Whether audio synthesized live is stored in `audioCache`, or a function that decides per text. Defaults to false: only warmed lines are stored, since a generated reply can repeat what the user said. pipedog only. |
| `audioCacheNamespace` | `string \| undefined` | no | Extra part of the cache key, for constructor options that change the audio but are not in `settings`. pipedog only. |
| `ai` | `WorkersAIBinding \| null \| undefined` | no | The Workers AI binding (`env.AI`), inside a Worker. Takes precedence over the credentials. |
| `accountId` | `string \| null \| undefined` | no | Cloudflare account ID, for the REST API. |
| `apiToken` | `string \| null \| undefined` | no | Cloudflare API token with Workers AI access, for the REST API. |
| `baseUrl` | `string \| null \| undefined` | no | Base URL of Cloudflare's API. Defaults to `CLOUDFLARE_API_URL`. |
| `fetch` | `((input: RequestInfo \| URL, init?: RequestInit) => Promise<Response>) \| null \| undefined` | no | `fetch` used for REST requests. Defaults to the global `fetch`. |
| `webSocketFactory` | `((uri: string, options: { headers?: Record<string, string>; protocols?: string[]; }) => WebSocket \| Promise<WebSocket>) \| null \| undefined` | no | Creates the `WebSocket` for REST sessions, to open it some other way. Defaults to `websocketConnect`'s, which sends the token header on Bun, Node, Deno and Cloudflare Workers. |
| `client` | `WorkersAIClient \| null \| undefined` | no | A client to share between services. When set, the other connection options are ignored. |
| `encoding` | `string \| undefined` | no | Audio encoding format. Defaults to "linear16". Must be one of SUPPORTED_ENCODINGS. |

### Settings [#settings]

| Setting | Type     | Default                    | Description                                                                                                    |
| ------- | -------- | -------------------------- | -------------------------------------------------------------------------------------------------------------- |
| `model` | `string` | `"@cf/deepgram/aura-2-en"` | `WorkersAIAuraModel.AURA_2_EN`, `.AURA_2_ES` or `.AURA_1`.                                                     |
| `voice` | `string` | the model's default        | The `speaker`: `luna` (aura-2-en), `aquila` (aura-2-es), `angus` (aura-1). See the model pages for the others. |

### Usage [#usage]

```ts title="tts.ts"
import { WorkersAIAuraModel, WorkersAIAuraTTSService } from "@pipedog/cloudflare/tts";

const tts = new WorkersAIAuraTTSService({
  ai: env.AI,
  sampleRate: 24000,
  settings: new WorkersAIAuraTTSService.Settings({
    model: WorkersAIAuraModel.AURA_2_EN,
    voice: "asteria",
  }),
});
```

Over the REST API:

```ts
const ttsRest = new WorkersAIAuraTTSService({
  accountId: process.env.CLOUDFLARE_ACCOUNT_ID ?? "",
  apiToken: process.env.CLOUDFLARE_API_TOKEN ?? "",
  settings: new WorkersAIAuraTTSService.Settings({
    model: WorkersAIAuraModel.AURA_2_ES,
    voice: "celeste",
  }),
});

```

Change the voice mid-call with a `TTSUpdateSettingsFrame` (it reconnects):

```ts
// Queue it on the worker, or push it from a processor upstream of the service.
const update = new TTSUpdateSettingsFrame({
  delta: new WorkersAIAuraTTSService.Settings({ voice: "orion" }),
});
```

### Notes [#notes]

* Text is sent with `Speak` as it arrives and flushed (`Flush`) at the end of each LLM response; an
  interruption sends `Clear`. Output is 16-bit PCM at the service's sample rate.
* The session parameters are `speaker`, `encoding` and `sample_rate`. Deepgram's `speed` isn't in
  Workers AI's schema and isn't sent.

## WorkersAIAuraHttpTTSService [#workersaiaurahttpttsservice]

```ts title="tts.ts"
import { WorkersAIAuraHttpTTSService } from "@pipedog/cloudflare/tts";

const ttsHttp = new WorkersAIAuraHttpTTSService({ ai: env.AI, sampleRate: 24000 });

```

Each sentence is a request for `encoding: "linear16"`, `container: "none"` at the service's sample
rate, and the body is streamed back as audio frames. Use it when you'd rather not hold a socket open.

## WorkersAIMeloTTSService [#workersaimelottsservice]

### Configuration [#configuration-1]

| Option | Type | Required | Description |
|---|---|---|---|
| `decodeAudio` | `AudioDecoder \| null \| undefined` | no | Decodes MeloTTS's MP3 output. Required: without it every request fails with an error saying so. |
| `settings` | `WorkersAIMeloTTSSettings \| null \| undefined` | no | Runtime-updatable settings. |
| `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. |
| `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. |
| `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. |
| `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. |
| `textAggregationMode` | `TextAggregationMode \| null \| undefined` | no | How to aggregate incoming text before synthesis. `TextAggregationMode.SENTENCE` (default) buffers until sentence boundaries, `TextAggregationMode.TOKEN` streams tokens directly for lower latency. |
| `pushTextFrames` | `boolean \| undefined` | no | Whether to push TextFrames and LLMFullResponseEndFrames. |
| `pushStopFrames` | `boolean \| undefined` | no | Whether to automatically push TTSStoppedFrames. |
| `pushStartFrame` | `boolean \| undefined` | no | Whether to automatically create audio contexts and push TTSStartedFrames. When true, the base class handles `createAudioContext` and yields `TTSStartedFrame` before each synthesis call, so `runTts` implementations do not need to. |
| `stopFrameTimeoutS` | `number \| undefined` | no | Idle time before pushing TTSStoppedFrame when pushStopFrames is true. |
| `pushSilenceAfterStop` | `boolean \| undefined` | no | Whether to push silence audio after TTSStoppedFrame. |
| `silenceTimeS` | `number \| undefined` | no | Duration of silence to push when pushSilenceAfterStop is true. |
| `pauseFrameProcessing` | `boolean \| undefined` | no | Whether to hold incoming frames after a turn until its audio contexts end. Needed by services whose audio doesn't carry its request's context ID. |
| `maxConsecutiveZeroAudioContexts` | `number \| undefined` | no | How many consecutive TTS contexts may complete without producing any audio before the service is reported unable to do its job. Catches a provider that accepts requests and stays silent — an unknown voice ID, say — which no error ever surfaces. Every silent context reports an error the service can carry on from, whatever the limit is; on reaching the limit the service reports a permanent error instead, stops being given work, and the pipeline worker applies its `ProcessorUnusablePolicy`. Set to 0 to report silent contexts without ever writing the service off. |
| `appendTrailingSpace` | `boolean \| undefined` | no | Whether to append a trailing space to text before sending to TTS. This helps prevent some TTS services from vocalizing trailing punctuation (e.g., "dot"). Only applied in sentence aggregation mode; when streaming tokens, the incoming text's own whitespace is preserved. |
| `sampleRate` | `number \| null \| undefined` | no | Output sample rate for generated audio. |
| `skipTextTypes` | `string[] \| null \| undefined` | no | List of text types that should not be spoken. |
| `textTransforms` | `[string, TextTransform][] \| null \| undefined` | no | A list of TextTransforms to transform text just before sending it to TTS. Each TextTransform takes the aggregated text and its type, and returns the transformed text. To register, provide a list of tuples of `[textType \| "*", transformFunction]`. |
| `transportDestination` | `string \| null \| undefined` | no | Destination for generated audio frames. |
| `reuseContextIdWithinTurn` | `boolean \| undefined` | no | Whether the service should reuse context IDs within the same turn. |
| `audioCache` | `TTSAudioCacheStore \| null \| undefined` | no | Where to keep synthesized audio for reuse (e.g. a shared `TTSAudioCache`). Text whose audio is in the cache is served from it, with no request to the provider. Lines get there through `warmAudioCache()`, and also from live synthesis when `cacheSynthesizedAudio` allows it. Only services that yield their audio from `runTts` (HTTP and streaming-RPC services, see `supportsAudioCache`) use it, and only in sentence aggregation mode. Defaults to null (no cache). pipedog only. |
| `cacheSynthesizedAudio` | `boolean \| ((text: string) => boolean) \| undefined` | no | Whether audio synthesized live is stored in `audioCache`, or a function that decides per text. Defaults to false: only warmed lines are stored, since a generated reply can repeat what the user said. pipedog only. |
| `audioCacheNamespace` | `string \| undefined` | no | Extra part of the cache key, for constructor options that change the audio but are not in `settings`. pipedog only. |
| `ai` | `WorkersAIBinding \| null \| undefined` | no | The Workers AI binding (`env.AI`), inside a Worker. Takes precedence over the credentials. |
| `accountId` | `string \| null \| undefined` | no | Cloudflare account ID, for the REST API. |
| `apiToken` | `string \| null \| undefined` | no | Cloudflare API token with Workers AI access, for the REST API. |
| `baseUrl` | `string \| null \| undefined` | no | Base URL of Cloudflare's API. Defaults to `CLOUDFLARE_API_URL`. |
| `fetch` | `((input: RequestInfo \| URL, init?: RequestInit) => Promise<Response>) \| null \| undefined` | no | `fetch` used for REST requests. Defaults to the global `fetch`. |
| `webSocketFactory` | `((uri: string, options: { headers?: Record<string, string>; protocols?: string[]; }) => WebSocket \| Promise<WebSocket>) \| null \| undefined` | no | Creates the `WebSocket` for REST sessions, to open it some other way. Defaults to `websocketConnect`'s, which sends the token header on Bun, Node, Deno and Cloudflare Workers. |
| `client` | `WorkersAIClient \| null \| undefined` | no | A client to share between services. When set, the other connection options are ignored. |

### Usage [#usage-1]

```ts title="tts.ts"
import { WorkersAIMeloTTSService } from "@pipedog/cloudflare/melotts";

// Any MP3 decoder that returns per-channel Float32Arrays and a sample rate,
// e.g. mpg123-decoder's MPEGDecoder (bun add mpg123-decoder):
//   const decoder = new MPEGDecoder(); await decoder.ready;
//   decodeAudio: (mp3) => { decoder.reset(); return decoder.decode(mp3); }
declare function decodeMp3(mp3: Uint8Array): Promise<{
  channelData: Float32Array[];
  sampleRate: number;
}>;

const melo = new WorkersAIMeloTTSService({
  ai: env.AI,
  decodeAudio: decodeMp3,
  settings: new WorkersAIMeloTTSService.Settings({ language: Language.FR }),
});
```

### Notes [#notes-1]

* MeloTTS answers with MP3 only. pipedog ships no MP3 decoder (Workers have no Web Audio, and a
  WebAssembly decoder has to be imported as a module there), so pass `decodeAudio`. The decoded audio
  is mixed to mono and resampled to the service rate.
* `language` picks the voice: English, Spanish, French, Chinese, Japanese or Korean.

## Runtime notes [#runtime-notes]

| Runtime            | Works | What it needs |
| ------------------ | ----- | ------------- |
| Bun                | ✅ yes | Nothing.      |
| Node.js            | ✅ yes | Nothing.      |
| Deno               | ✅ yes | Nothing.      |
| Cloudflare Workers | ✅ yes | Nothing.      |

Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/cloudflare/package.json`: ✅ means the compat smoke test passes on that runtime.

* Tested against a local fake of Workers AI over both paths, not against Workers AI itself. Whether
  Aura's WebSocket opens through the binding (`websocket: true`) is documented for Flux but not for
  Aura: if it doesn't for you, use `WorkersAIAuraHttpTTSService` or the REST path.