Voice agents on Workers and Durable Objects

A voice agent is a long-lived connection with state: a pipeline, a conversation, and a socket. Cloudflare Durable Objects suit that shape. Each object is a single-threaded actor with its own state, so one object can run one voice session without interfering with any other. Cloudflare places the objects across its network, which lets many sessions run in parallel.

The complete example is examples/workers-durable-voice-agent. This page explains the pattern. To run the models on Cloudflare too, see Voice agent fully on Cloudflare.

The shapeCopy Anchor Link

client ──WebSocket──► Worker ──getByName(id)──► VoiceSession (Durable Object)
                                                   │ FastAPIWebsocketTransport
                                                   │ OpenAI realtime transcription (turns)
                                                   │ chat, speech synthesis

• The Worker routes each session to a Durable Object by name. The same name always reaches the same object.
• The Durable Object accepts the WebSocket and runs the agent for as long as the socket is open.
• The transport is the WebSocket transport, carrying audio as protobuf frames in the Pipecat wire format. WebRTC needs UDP, which Workers don't have, so the Worker can't be a WebRTC peer itself. For browser or app users on WebRTC, let Cloudflare Realtime be the peer: its SFU converts the audio to PCM over WebSockets that the Durable Object accepts (see examples/workers-realtime-voice-agent).

The Durable ObjectCopy Anchor Link

websocketFetchHandler turns a connection into a fetch handler. The object returns its result from fetch:

import { DurableObject } from "cloudflare:workers";
import { websocketFetchHandler } from "@pipedog/transport-websocket/fetch";

export class VoiceSession extends DurableObject<Env> {
  readonly #handler = websocketFetchHandler((connection) => runVoiceAgent(connection, this.env));

  override fetch(request: Request): Promise<Response> {
    return this.#handler(request);
  }
}

runVoiceAgent builds the pipeline on the connection with FastAPIWebsocketTransport. It is the same pipeline as the other voice bot examples: speech in, speech-to-text, the user's turn, the chat model, speech synthesis, and speech out.

The WorkerCopy Anchor Link

export default {
  async fetch(request: Request, env: Env): Promise<Response> {
    const id = new URL(request.url).pathname.split("/")[3];
    return env.VOICE_SESSION.getByName(id).fetch(request);
  },
} satisfies ExportedHandler<Env>;

Turn detectionCopy Anchor Link

The agent detects turns on the provider's side: OpenAIRealtimeSTTService with server VAD, and no VAD analyzer. So the Worker loads no local models and no ONNX runtime. That matters here:

Cloudflare limits a Worker to 10 MB compressed on paid plans and 3 MB on the free plan, so the local models don't fit. They also load about 100 MB of runtime and models, close to a Durable Object's 128 MB memory limit. See Turn detection for the modes.

ReconnectionCopy Anchor Link

A client that drops and reconnects to the same session id within ten minutes continues the same conversation. The Durable Object saves the conversation to its storage when a socket closes, loads it when the next one opens, and skips the greeting. An alarm clears it once the ten minutes pass. If a reconnect arrives while the old socket still looks open, the new one wins: the object closes the old one and waits for its conversation to be saved first.

Pipecat's WebSocket transport doesn't resume sessions; this is the example's own logic, in src/voice_session.ts.

Tracing and alertsCopy Anchor Link

Set OTEL_EXPORTER_OTLP_ENDPOINT and ERROR_WEBHOOK_URL in the Worker's environment, as in the voice bot guide. Spans are buffered, and a Worker doesn't export them by itself after a request ends, so the agent flushes them when each session ends.

Fetch handlersCopy Anchor Link

pipedog's voice endpoints are fetch handlers: functions from a Request to a Response. That makes them usable from any framework that routes requests this way.

• WebSocket endpoints. websocketFetchHandler upgrades from fetch on its own on Cloudflare Workers, Durable Objects and Deno. On Bun, pass it the server Bun gives fetch and serve with bunWebSocket; on Node, serve it with serveFetch(). The runner's fetch handler (createRunnerApp()) upgrades the same way, though it isn't tested on Workers yet.
• WebRTC signaling. smallWebRTCOfferHandler is a plain fetch handler. Its media runs over UDP, so the server has to be a long-lived Node, Bun or Deno process. Serverless hosts can't carry it. On Workers, use Cloudflare Realtime instead: its SFU carries the WebRTC media and hands the bot PCM over WebSockets.

Shapes for the other frameworks:

// Bun
Bun.serve({ fetch: (request, server) => handler(request, server), websocket: bunWebSocket });

// Node, Bun or Deno
await serveFetch(handler, { port: 8765 });

// Hono
app.post("/api/offer", (c) => offer(c.req.raw));

// Next.js app router (app/api/offer/route.ts)
export const POST = (request: Request) => offer(request);

// SvelteKit (src/routes/api/offer/+server.ts)
export const POST = ({ request }) => offer(request);

The Workers example runs in tests against workerd, through wrangler dev. The same handlers mounted in each framework are tested in examples/framework-snippets: see Frameworks.

Run itCopy Anchor Link

cd examples/workers-durable-voice-agent
echo "OPENAI_API_KEY=sk-..." > .dev.vars
bun run dev

Stream a recording to a session with the scripted user from the example, in a second terminal:

bun run client ws://127.0.0.1:8787/api/session/alice ../test-mocks/assets/capital_question.wav

Then deploy:

bunx wrangler secret put OPENAI_API_KEY
bun run deploy

Limits to plan forCopy Anchor Link

• Duration. A Durable Object is billed while it runs. A connected session keeps it running, so idle sessions cost time. Hibernation, where the object sleeps between messages, would save that, but the agent's timers would need to survive a sleep, and this example doesn't do that.
• CPU. Each session still decodes, resamples and serializes audio. Check the CPU limits for your plan under real load.
• Transport. The Worker can't be a WebRTC peer. Over the WebSocket transport, the client sends and receives protobuf frames. For WebRTC clients, put Cloudflare Realtime in between.