# Cloudflare Workers AI

> Speech-to-text on Workers AI: Deepgram Flux with turn detection, Deepgram Nova-3 streaming, and Whisper, over the AI binding or the REST API.

Source: https://polvallverdu.github.io/pipedog/docs/services/stt/cloudflare/



## Overview [#overview]

Workers AI hosts speech-to-text models on Cloudflare's network. `@pipedog/cloudflare` has a service
for each kind:

* **`WorkersAIFluxSTTService`**: Deepgram Flux (`@cf/deepgram/flux`), conversational STT that also
  detects the end of the user's turn. It replaces VAD and Smart Turn.
* **`WorkersAINova3STTService`**: Deepgram Nova-3 (`@cf/deepgram/nova-3`) over a WebSocket, with
  interim results. Use it with a VAD, as with Deepgram.
* **`WorkersAIWhisperSTTService`**: Whisper (`@cf/openai/whisper-large-v3-turbo`, `@cf/openai/whisper`,
  `@cf/openai/whisper-tiny-en`), one request per speech segment.

This package is pipedog's own: Pipecat has no Workers AI services. The Flux and Nova-3 services are
the [Deepgram](https://polvallverdu.github.io/pipedog/docs/services/stt/deepgram.md) services with a different way to open the socket, since
Workers AI carries Deepgram's WebSocket protocol, so they behave and emit frames the same way.

<Cards>
  <Card title="Source" href="https://github.com/polvallverdu/pipedog/tree/main/packages/cloudflare/src">
    `packages/cloudflare/src/flux_stt.ts`, `stt.ts` and `whisper.ts`.
  </Card>

  <Card title="Workers AI models" href="https://developers.cloudflare.com/workers-ai/models/">
    Inputs, outputs and pricing of each model.
  </Card>

  <Card title="Voice agent fully on Cloudflare" href="/docs/deployment/cloudflare-workers-ai">
    Flux, a Workers AI LLM and Aura in a Durable Object.
  </Card>
</Cards>

## Installation [#installation]

```bash
bun add @pipedog/cloudflare
```

```ts
import { WorkersAIFluxSTTService } from "@pipedog/cloudflare/flux_stt";
import { WorkersAINova3STTService } from "@pipedog/cloudflare/stt";
import { WorkersAIWhisperSTTService } from "@pipedog/cloudflare/whisper";
```

## Prerequisites [#prerequisites]

Every Workers AI service reaches the models one of two ways:

* **The `AI` binding**, inside a Worker or Durable Object. Add `"ai": { "binding": "AI" }` to
  `wrangler.jsonc` and pass `ai: env.AI`. No token is needed. The realtime models open their socket
  with `env.AI.run(model, input, { websocket: true })`.
* **The REST API**, anywhere: pass `accountId` and `apiToken` (a [Cloudflare API
  token](https://dash.cloudflare.com/profile/api-tokens) with the Workers AI permission). Requests go to
  `https://api.cloudflare.com/client/v4/accounts/{accountId}/ai/run/{model}`, and the realtime models
  open `wss://` on the same URL, with the session parameters in the query string. The examples read
  them from `CLOUDFLARE_ACCOUNT_ID` and `CLOUDFLARE_API_TOKEN`.

```ts title="stt.ts"
// Anywhere else: an account ID and an API token with Workers AI access.
const fluxRest = new WorkersAIFluxSTTService({
  accountId: process.env.CLOUDFLARE_ACCOUNT_ID ?? "",
  apiToken: process.env.CLOUDFLARE_API_TOKEN ?? "",
});

```

To share one connection setup between services, build a `WorkersAIClient` and pass it as `client`:

```ts
import { WorkersAIClient } from "@pipedog/cloudflare/client";

// One client shared by several services.
const client = new WorkersAIClient({ ai: env.AI });
const fluxShared = new WorkersAIFluxSTTService({ client });

```

## WorkersAIFluxSTTService [#workersaifluxsttservice]

### Configuration [#configuration]

| Option | Type | Required | Description |
|---|---|---|---|
| `accountId` | `string \| null \| undefined` | no | Cloudflare account ID, for the REST API. |
| `apiToken` | `string \| null \| undefined` | no | Cloudflare API token with Workers AI access, for the REST API. |
| `baseUrl` | `string \| null \| undefined` | no | Base URL of Cloudflare's API. Defaults to `CLOUDFLARE_API_URL`. |
| `fetch` | `{ (input: RequestInfo \| URL, init?: RequestInit): Promise<Response>; (input: string \| URL \| Request, init?: RequestInit): Promise<Response>; } \| null \| undefined` | no | `fetch` used for REST requests. Defaults to the global `fetch`. |
| `settings` | `DeepgramFluxSTTSettings \| null \| undefined` | no | Runtime-updatable settings. `model` is the Workers AI model ID (default `"@cf/deepgram/flux"`). `languageHints`, `numerals`, `profanityFilter`, `redact` and `minConfidence` are Deepgram options that Workers AI does not document: they are sent as Deepgram sends them, and may be ignored. |
| `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. |
| `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. |
| `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. |
| `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. |
| `reconnectBackoffMinWait` | `number \| undefined` | no | Minimum time, in seconds, to wait between reconnection attempts. |
| `reconnectBackoffMaxWait` | `number \| undefined` | no | Maximum time, in seconds, to wait between reconnection attempts. |
| `reconnectOnError` | `boolean \| undefined` | no | Whether to automatically reconnect on connection errors. |
| `wsCloseTimeout` | `number \| undefined` | no | Maximum time, in seconds, to wait for the peer to acknowledge the websocket closing handshake before dropping the connection. Applied to connections opened through `_websocketConnect()`. Increase it for peers that need longer to complete a graceful close. |
| `wsSendTimeout` | `number \| null \| undefined` | no | Maximum time, in seconds, a single send may wait on a peer that has stopped reading before the connection is dropped and the send throws `ConnectionClosedError`. Applied to connections opened through `_websocketConnect()`. Pass `null` to wait indefinitely. Defaults to `WS_SEND_TIMEOUT`. |
| `audioPassthrough` | `boolean \| undefined` | no | Whether to pass audio frames downstream after processing. Defaults to true. |
| `sampleRate` | `number \| null \| undefined` | no | The sample rate for audio input. If null, will be determined from the start frame. |
| `sttTtfbTimeout` | `number \| undefined` | no | Time in seconds to wait after VAD stop before reporting TTFB. This delay allows the final transcription to arrive. Defaults to 2.0. |
| `ttfsP99Latency` | `number \| null \| undefined` | no | P99 latency from speech end to final transcript in seconds. Broadcast via STTMetadataFrame at pipeline start. |
| `keepaliveTimeout` | `number \| null \| undefined` | no | Seconds of no audio before sending silence to keep the connection alive. null disables keepalive. |
| `keepaliveInterval` | `number \| undefined` | no | Seconds between idle checks when keepalive is enabled. |
| `enableEagerEndOfTurn` | `boolean \| undefined` | no | Whether to answer a predicted end of turn ahead of the committed one. Off by default: it spends an inference on every prediction, including the ones the service withdraws. |
| `eagerMatchPolicy` | `EagerMatchPolicy \| null \| undefined` | no | Decides whether the committed transcript is close enough to the predicted one to keep the speculative response. Only consulted while eager end of turn is on. See EagerUserTurnStrategies. |
| `mipOptOut` | `boolean \| null \| undefined` | no | Opt out of the Deepgram Model Improvement Program. |
| `tag` | `string[] \| null \| undefined` | no | Tags to label requests for identification during usage reporting. |
| `shouldInterrupt` | `boolean \| undefined` | no | Whether to interrupt the bot when Flux detects that the user is speaking. |
| `watchdogMinTimeout` | `number \| undefined` | no | Minimum idle timeout before sending silence to prevent dangling turns. |
| `fluxEncoding` | `string \| undefined` | no | Audio encoding format required by Flux API. Must be "linear16". Raw signed little-endian 16-bit PCM encoding. |
| `ai` | `WorkersAIBinding \| null \| undefined` | no | The Workers AI binding (`env.AI`), inside a Worker. Takes precedence over the credentials. |
| `webSocketFactory` | `((uri: string, options: { headers?: Record<string, string>; protocols?: string[]; }) => WebSocket \| Promise<WebSocket>) \| null \| undefined` | no | Creates the `WebSocket` for REST sessions, to open it some other way. Defaults to `websocketConnect`'s, which sends the token header on Bun, Node, Deno and Cloudflare Workers. |
| `client` | `WorkersAIClient \| null \| undefined` | no | A client to share between services. When set, the other connection options are ignored. |

### Settings [#settings]

`WorkersAIFluxSTTService.Settings` is Deepgram Flux's settings class; see
[Deepgram Flux settings](https://polvallverdu.github.io/pipedog/docs/services/stt/deepgram/#deepgramfluxsttservice). The ones Workers AI
documents:

| Setting             | Type             | Default               | Description                                                  |
| ------------------- | ---------------- | --------------------- | ------------------------------------------------------------ |
| `model`             | `string`         | `"@cf/deepgram/flux"` | The Workers AI model ID.                                     |
| `eotThreshold`      | `number \| null` | `null` (Flux: 0.7)    | End-of-turn confidence that ends a turn, 0.5 to 0.9.         |
| `eagerEotThreshold` | `number \| null` | `null`                | Enables `EagerEndOfTurn` and `TurnResumed`, 0.3 to 0.9.      |
| `eotTimeoutMs`      | `number \| null` | `null` (Flux: 5000)   | Ends a turn this long after speech, whatever the confidence. |
| `keyterm`           | `string[]`       | `[]`                  | Terms to boost.                                              |

### Usage [#usage]

```ts title="stt.ts"
import { WorkersAIFluxSTTService } from "@pipedog/cloudflare/flux_stt";

// Inside a Worker or Durable Object: the AI binding, no token.
const flux = new WorkersAIFluxSTTService({
  ai: env.AI,
  settings: new WorkersAIFluxSTTService.Settings({
    eotThreshold: 0.75,
    keyterm: ["pipedog", "Cloudflare"],
  }),
});
```

Flux owns turn detection: use it with no VAD analyzer, and the user aggregator follows its turn
events, as with `DeepgramFluxSTTService` (see [Turn detection](https://polvallverdu.github.io/pipedog/docs/concepts/turn-detection.md)).

Change settings mid-call with an `STTUpdateSettingsFrame`:

```ts
// Queue it on the worker, or push it from a processor upstream of the service.
const update = new STTUpdateSettingsFrame({
  delta: new WorkersAIFluxSTTService.Settings({ eotThreshold: 0.85 }),
});
```

### Event handlers [#event-handlers]

The same as `DeepgramFluxSTTService`: `on_start_of_turn`, `on_end_of_turn`,
`on_eager_end_of_turn`, `on_turn_resumed`, `on_update`, plus `on_connected`, `on_disconnected` and
`on_connection_error`.

### Notes [#notes]

* The session parameters are the model's inputs (`encoding`, `sample_rate`, `eot_threshold`,
  `eager_eot_threshold`, `eot_timeout_ms`, `keyterm`, `mip_opt_out`, `tag`). The model is the URL
  path, not a `model` parameter.
* Workers AI's turn messages carry an `event` and no `type`; they are read as Deepgram's `TurnInfo`.
  Workers AI doesn't document Deepgram's `Connected` message, so an open socket counts as confirmed.
* Audio is 16-bit PCM (`linear16`) at the pipeline's input rate.

## WorkersAINova3STTService [#workersainova3sttservice]

### Configuration [#configuration-1]

| Option | Type | Required | Description |
|---|---|---|---|
| `settings` | `DeepgramSTTSettings \| null \| undefined` | no | Runtime-updatable settings (Deepgram's). `model` is the Workers AI model ID, `"@cf/deepgram/nova-3"` by default. |
| `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. |
| `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. |
| `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. |
| `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. |
| `audioPassthrough` | `boolean \| undefined` | no | Whether to pass audio frames downstream after processing. Defaults to true. |
| `sampleRate` | `number \| null \| undefined` | no | The sample rate for audio input. If null, will be determined from the start frame. |
| `sttTtfbTimeout` | `number \| undefined` | no | Time in seconds to wait after VAD stop before reporting TTFB. This delay allows the final transcription to arrive. Defaults to 2.0. |
| `ttfsP99Latency` | `number \| null \| undefined` | no | P99 latency from speech end to final transcript in seconds. Broadcast via STTMetadataFrame at pipeline start. |
| `keepaliveTimeout` | `number \| null \| undefined` | no | Seconds of no audio before sending silence to keep the connection alive. null disables keepalive. |
| `keepaliveInterval` | `number \| undefined` | no | Seconds between idle checks when keepalive is enabled. |
| `ai` | `WorkersAIBinding \| null \| undefined` | no | The Workers AI binding (`env.AI`), inside a Worker. Takes precedence over the credentials. |
| `accountId` | `string \| null \| undefined` | no | Cloudflare account ID, for the REST API. |
| `apiToken` | `string \| null \| undefined` | no | Cloudflare API token with Workers AI access, for the REST API. |
| `baseUrl` | `string \| null \| undefined` | no | Base URL of Cloudflare's API. Defaults to `CLOUDFLARE_API_URL`. |
| `fetch` | `{ (input: RequestInfo \| URL, init?: RequestInit): Promise<Response>; (input: string \| URL \| Request, init?: RequestInit): Promise<Response>; } \| null \| undefined` | no | `fetch` used for REST requests. Defaults to the global `fetch`. |
| `webSocketFactory` | `((uri: string, options: { headers?: Record<string, string>; protocols?: string[]; }) => WebSocket \| Promise<WebSocket>) \| null \| undefined` | no | Creates the `WebSocket` for REST sessions, to open it some other way. Defaults to `websocketConnect`'s, which sends the token header on Bun, Node, Deno and Cloudflare Workers. |
| `client` | `WorkersAIClient \| null \| undefined` | no | A client to share between services. When set, the other connection options are ignored. |
| `encoding` | `string \| undefined` | no | Audio encoding format. Defaults to "linear16". |
| `channels` | `number \| undefined` | no | Number of audio channels. Defaults to 1. |
| `multichannel` | `boolean \| undefined` | no | Transcribe each audio channel independently. Defaults to false. |
| `callback` | `string \| null \| undefined` | no | Callback URL for async transcription delivery. |
| `callbackMethod` | `string \| null \| undefined` | no | HTTP method for the callback ("GET" or "POST"). |
| `tag` | `any` | no | Custom billing tag. |
| `mipOptOut` | `boolean \| null \| undefined` | no | Opt out of Deepgram model improvement program. |
| `addons` | `Record<string, unknown> \| null \| undefined` | no | Additional Deepgram features to enable. |

### Settings [#settings-1]

`WorkersAINova3STTService.Settings` is Deepgram's settings class (see
[DeepgramSTTService settings](https://polvallverdu.github.io/pipedog/docs/services/stt/deepgram/#settings)), with `model` defaulting to
`"@cf/deepgram/nova-3"`.

### Usage [#usage-1]

```ts title="stt.ts"
import { WorkersAINova3STTService } from "@pipedog/cloudflare/stt";

const nova = new WorkersAINova3STTService({
  ai: env.AI,
  settings: new WorkersAINova3STTService.Settings({ language: Language.EN, smartFormat: true }),
});

```

### Notes [#notes-1]

* Every Deepgram parameter but the model is sent as a session parameter. Workers AI documents
  `interim_results`, `endpointing`, `vad_events`, `utterance_end_ms`, `smart_format`, `punctuate`,
  `keyterm`, `language` and more on the model page.

## WorkersAIWhisperSTTService [#workersaiwhispersttservice]

### Configuration [#configuration-2]

| Option | Type | Required | Description |
|---|---|---|---|
| `settings` | `WorkersAIWhisperSTTSettings \| null \| undefined` | no | Runtime-updatable settings. |
| `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. |
| `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. |
| `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. |
| `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. |
| `audioPassthrough` | `boolean \| undefined` | no | Whether to pass audio frames downstream after processing. Defaults to true. |
| `sampleRate` | `number \| null \| undefined` | no | The sample rate for audio input. If null, will be determined from the start frame. |
| `sttTtfbTimeout` | `number \| undefined` | no | Time in seconds to wait after VAD stop before reporting TTFB. This delay allows the final transcription to arrive. Defaults to 2.0. |
| `ttfsP99Latency` | `number \| null \| undefined` | no | P99 latency from speech end to final transcript in seconds. Broadcast via STTMetadataFrame at pipeline start. |
| `keepaliveTimeout` | `number \| null \| undefined` | no | Seconds of no audio before sending silence to keep the connection alive. null disables keepalive. |
| `keepaliveInterval` | `number \| undefined` | no | Seconds between idle checks when keepalive is enabled. |
| `trailingSilenceSecs` | `number \| undefined` | no | Seconds of silence appended to each segment before it is transcribed, so the model hears the end of speech and can finish the last word. Set to 0 to disable. Defaults to 0.5. |
| `ai` | `WorkersAIBinding \| null \| undefined` | no | The Workers AI binding (`env.AI`), inside a Worker. Takes precedence over the credentials. |
| `accountId` | `string \| null \| undefined` | no | Cloudflare account ID, for the REST API. |
| `apiToken` | `string \| null \| undefined` | no | Cloudflare API token with Workers AI access, for the REST API. |
| `baseUrl` | `string \| null \| undefined` | no | Base URL of Cloudflare's API. Defaults to `CLOUDFLARE_API_URL`. |
| `fetch` | `{ (input: RequestInfo \| URL, init?: RequestInit): Promise<Response>; (input: string \| URL \| Request, init?: RequestInit): Promise<Response>; } \| null \| undefined` | no | `fetch` used for REST requests. Defaults to the global `fetch`. |
| `webSocketFactory` | `((uri: string, options: { headers?: Record<string, string>; protocols?: string[]; }) => WebSocket \| Promise<WebSocket>) \| null \| undefined` | no | Creates the `WebSocket` for REST sessions, to open it some other way. Defaults to `websocketConnect`'s, which sends the token header on Bun, Node, Deno and Cloudflare Workers. |
| `client` | `WorkersAIClient \| null \| undefined` | no | A client to share between services. When set, the other connection options are ignored. |

### Settings [#settings-2]

| Setting     | Type               | Default                               | Description                                                                      |
| ----------- | ------------------ | ------------------------------------- | -------------------------------------------------------------------------------- |
| `model`     | `string`           | `"@cf/openai/whisper-large-v3-turbo"` | `WorkersAIWhisperModel.LARGE_V3_TURBO`, `.WHISPER` or `.TINY_EN`.                |
| `language`  | `Language \| null` | `null`                                | The audio's language, sent as its ISO 639-1 code; `null` detects it. Turbo only. |
| `prompt`    | `string \| null`   | `null`                                | Context for the model (`initial_prompt`). Turbo only.                            |
| `vadFilter` | `boolean \| null`  | `null`                                | Run Whisper's own VAD first (`vad_filter`). Turbo only.                          |

### Usage [#usage-2]

```ts title="stt.ts"
import { WorkersAIWhisperModel, WorkersAIWhisperSTTService } from "@pipedog/cloudflare/whisper";

const whisper = new WorkersAIWhisperSTTService({
  ai: env.AI,
  settings: new WorkersAIWhisperSTTService.Settings({
    model: WorkersAIWhisperModel.LARGE_V3_TURBO,
    language: Language.EN,
    prompt: "A conversation about Cloudflare Workers.",
  }),
});
```

### Notes [#notes-2]

* A segmented service: it needs a VAD analyzer. The audio between the VAD's start and stop is sent
  as a WAV file, base64 for turbo and an array of bytes for the original models.

## Runtime notes [#runtime-notes]

| Runtime            | Works | What it needs |
| ------------------ | ----- | ------------- |
| Bun                | ✅ yes | Nothing.      |
| Node.js            | ✅ yes | Nothing.      |
| Deno               | ✅ yes | Nothing.      |
| Cloudflare Workers | ✅ yes | Nothing.      |

Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/cloudflare/package.json`: ✅ means the compat smoke test passes on that runtime.

* No dependencies beyond pipedog's own packages. HTTP goes through `fetch` and WebSockets through
  the core `websocketConnect` (REST) or the binding's socket (Workers).
* Tested against a local fake of Workers AI (both paths), not against Workers AI itself. A live test
  (`packages/cloudflare/test/live_workers_ai.test.ts`) runs when `CLOUDFLARE_ACCOUNT_ID` and
  `CLOUDFLARE_API_TOKEN` are set.