# Voice agents on Workers and Durable Objects

> Run one voice agent per session on Cloudflare Durable Objects, and serve pipedog endpoints from any framework with fetch handlers.

Source: https://polvallverdu.github.io/pipedog/docs/deployment/cloudflare-workers/



A voice agent is a long-lived connection with state: a pipeline, a conversation, and a socket.
Cloudflare Durable Objects suit that shape. Each object is a single-threaded actor with its own
state, so one object can run one voice session without interfering with any other. Cloudflare places
the objects across its network, which lets many sessions run in parallel.

The complete example is
[`examples/workers-durable-voice-agent`](https://github.com/polvallverdu/pipedog/tree/main/examples/workers-durable-voice-agent).
This page explains the pattern. To run the models on Cloudflare too, see
[Voice agent fully on Cloudflare](https://polvallverdu.github.io/pipedog/docs/deployment/cloudflare-workers-ai.md).

## The shape [#the-shape]

```text
client ──WebSocket──► Worker ──getByName(id)──► VoiceSession (Durable Object)
                                                   │ FastAPIWebsocketTransport
                                                   │ OpenAI realtime transcription (turns)
                                                   │ chat, speech synthesis
```

* The **Worker** routes each session to a Durable Object by name. The same name always reaches the
  same object.
* The **Durable Object** accepts the WebSocket and runs the agent for as long as the socket is open.
* The **transport** is the WebSocket transport, carrying audio as protobuf frames in the Pipecat wire
  format. WebRTC needs UDP, which Workers don't have, so the Worker can't be a WebRTC peer itself.
  For browser or app users on WebRTC, let [Cloudflare Realtime](https://polvallverdu.github.io/pipedog/docs/transports/cloudflare-realtime.md)
  be the peer: its SFU converts the audio to PCM over WebSockets that the Durable Object accepts
  (see [`examples/workers-realtime-voice-agent`](https://github.com/polvallverdu/pipedog/tree/main/examples/workers-realtime-voice-agent)).

## The Durable Object [#the-durable-object]

`websocketFetchHandler` turns a connection into a fetch handler. The object returns its result from
`fetch`:

```ts
import { DurableObject } from "cloudflare:workers";
import { websocketFetchHandler } from "@pipedog/transport-websocket/fetch";

export class VoiceSession extends DurableObject<Env> {
  readonly #handler = websocketFetchHandler((connection) => runVoiceAgent(connection, this.env));

  override fetch(request: Request): Promise<Response> {
    return this.#handler(request);
  }
}
```

`runVoiceAgent` builds the pipeline on the connection with `FastAPIWebsocketTransport`. It is the
same pipeline as the other voice bot examples: speech in, speech-to-text, the user's turn, the chat
model, speech synthesis, and speech out.

## The Worker [#the-worker]

```ts
export default {
  async fetch(request: Request, env: Env): Promise<Response> {
    const id = new URL(request.url).pathname.split("/")[3];
    return env.VOICE_SESSION.getByName(id).fetch(request);
  },
} satisfies ExportedHandler<Env>;
```

## Turn detection [#turn-detection]

The agent detects turns on the provider's side: `OpenAIRealtimeSTTService` with server VAD, and no
VAD analyzer. So the Worker loads no local models and no ONNX runtime. That matters here:

| Turn detection                     | Worker size (gzipped) |
| ---------------------------------- | --------------------- |
| Provider (this example)            | 818 KB                |
| Silero VAD and Smart Turn, bundled | 13.7 MB               |

Cloudflare limits a Worker to 10 MB compressed on paid plans and 3 MB on the free plan, so the local
models don't fit. They also load about 100 MB of runtime and models, close to a Durable Object's
128 MB memory limit. See [Turn detection](https://polvallverdu.github.io/pipedog/docs/concepts/turn-detection.md) for the modes.

## Reconnection [#reconnection]

A client that drops and reconnects to the same session id within ten minutes continues the same
conversation. The Durable Object saves the conversation to its storage when a socket closes, loads it
when the next one opens, and skips the greeting. An alarm clears it once the ten minutes pass. If a
reconnect arrives while the old socket still looks open, the new one wins: the object closes the old
one and waits for its conversation to be saved first.

Pipecat's WebSocket transport doesn't resume sessions; this is the example's own logic, in
`src/voice_session.ts`.

## Tracing and alerts [#tracing-and-alerts]

Set `OTEL_EXPORTER_OTLP_ENDPOINT` and `ERROR_WEBHOOK_URL` in the Worker's environment, as in the
[voice bot guide](https://polvallverdu.github.io/pipedog/docs/voice-bot/#tracing-and-alerts). Spans are buffered, and a Worker doesn't export
them by itself after a request ends, so the agent flushes them when each session ends.

## Fetch handlers [#fetch-handlers]

pipedog's voice endpoints are fetch handlers: functions from a `Request` to a `Response`. That makes
them usable from any framework that routes requests this way.

* **WebSocket endpoints.** `websocketFetchHandler` upgrades from fetch on its own on Cloudflare
  Workers, Durable Objects and Deno. On Bun, pass it the server Bun gives `fetch` and serve with
  `bunWebSocket`; on Node, serve it with `serveFetch()`. The runner's fetch handler
  (`createRunnerApp()`) upgrades the same way, though it isn't tested on Workers yet.
* **WebRTC signaling.** `smallWebRTCOfferHandler` is a plain fetch handler. Its media runs over UDP,
  so the server has to be a long-lived Node, Bun or Deno process. Serverless hosts can't carry it.
  On Workers, use [Cloudflare Realtime](https://polvallverdu.github.io/pipedog/docs/transports/cloudflare-realtime.md) instead: its SFU
  carries the WebRTC media and hands the bot PCM over WebSockets.

Shapes for the other frameworks:

```ts
// Bun
Bun.serve({ fetch: (request, server) => handler(request, server), websocket: bunWebSocket });

// Node, Bun or Deno
await serveFetch(handler, { port: 8765 });

// Hono
app.post("/api/offer", (c) => offer(c.req.raw));

// Next.js app router (app/api/offer/route.ts)
export const POST = (request: Request) => offer(request);

// SvelteKit (src/routes/api/offer/+server.ts)
export const POST = ({ request }) => offer(request);
```

The Workers example runs in tests against workerd, through `wrangler dev`. The same handlers mounted
in each framework are tested in `examples/framework-snippets`: see [Frameworks](https://polvallverdu.github.io/pipedog/docs/frameworks.md).

## Run it [#run-it]

```sh
cd examples/workers-durable-voice-agent
echo "OPENAI_API_KEY=sk-..." > .dev.vars
bun run dev
```

Stream a recording to a session with the scripted user from the example, in a second terminal:

```sh
bun run client ws://127.0.0.1:8787/api/session/alice ../test-mocks/assets/capital_question.wav
```

Then deploy:

```sh
bunx wrangler secret put OPENAI_API_KEY
bun run deploy
```

## Limits to plan for [#limits-to-plan-for]

* **Duration.** A Durable Object is billed while it runs. A connected session keeps it running, so
  idle sessions cost time. Hibernation, where the object sleeps between messages, would save that, but
  the agent's timers would need to survive a sleep, and this example doesn't do that.
* **CPU.** Each session still decodes, resamples and serializes audio. Check the CPU limits for your
  plan under real load.
* **Transport.** The Worker can't be a WebRTC peer. Over the WebSocket transport, the client sends
  and receives protobuf frames. For WebRTC clients, put
  [Cloudflare Realtime](https://polvallverdu.github.io/pipedog/docs/transports/cloudflare-realtime.md) in between.