Turn detection

A voice agent has to know when the user starts speaking, to stop talking, and when they've finished, to answer. pipedog does this in one of three ways. They differ in what runs on your server: two local models, one, or none.

Each mode is a choice of turn strategies on the user aggregator. This page covers the three common setups; the strategies page covers every strategy and option.

Local modelsCopy Anchor Link

Silero VAD hears speech; Smart Turn listens to the end of each utterance and decides whether the user is done or only paused. This is Pipecat's default, and pipedog's:

const aggregators = new LLMContextAggregatorPair(context, {
  userParams: new LLMUserAggregatorParams({ vadAnalyzer: new SileroVADAnalyzer() }),
});

The default turn strategies use Smart Turn from @pipedog/audio-smart-turn, so install it next to @pipedog/audio-silero. Without it, turns end on silence (the VAD-only mode), and pipedog logs a warning once. In a bundled app, pass the analyzer to the strategy yourself; see Bundling.

Both models share one inference session per process: the first call loads them, and later calls reuse them. Measured on Bun with onnxruntime-node, the first call takes about 100 MB for the runtime and both models, and each further call about 1.3 MB.

VAD onlyCopy Anchor Link

Leave out @pipedog/audio-smart-turn, or end turns on silence explicitly:

const userParams = new LLMUserAggregatorParams({
  vadAnalyzer: new SileroVADAnalyzer(),
  userTurnStrategies: new UserTurnStrategies({
    stop: [new SpeechTimeoutUserTurnStopStrategy()],
  }),
});

This answers sooner after a pause than Smart Turn does, so it interrupts users who pause to think more often.

Another VAD: TEN VADCopy Anchor Link

Silero is the default VAD. TEN VAD (TenVADAnalyzer, pipedog only) is an opt-in alternative that runs TEN's WebAssembly build, which you obtain from TEN and pass in, under TEN's own license. It takes 16 kHz audio only. Pass it as vadAnalyzer wherever Silero goes.

Remote Smart Turn over HTTPCopy Anchor Link

If you can't run the model on the bot's host (a Worker, a small container), run Smart Turn as a service and let the bot send it the audio. HttpSmartTurnAnalyzer posts each candidate turn to an endpoint and reads the verdict back. It needs only fetch, so it works on every runtime, Cloudflare Workers included.

Generated from pipecat.runtimes and pipecat.compat in packages/audio-smart-turn/package.json: ✅ means the compat smoke test passes on that runtime.

// Smart Turn as a service: each candidate turn is POSTed to the endpoint.
const userParams = new LLMUserAggregatorParams({
  vadAnalyzer: new SileroVADAnalyzer(),
  userTurnStrategies: new UserTurnStrategies({
    start: [new VADUserTurnStartStrategy(), new TranscriptionUserTurnStartStrategy()],
    stop: [
      new TurnAnalyzerUserTurnStopStrategy({
        turnAnalyzer: new HttpSmartTurnAnalyzer({
          url: "https://smart-turn.example.com/predict",
          headers: { Authorization: `Bearer ${process.env.SMART_TURN_KEY}` },
        }),
      }),
    ],
  }),
});

It goes where the local analyzer does, in a TurnAnalyzerUserTurnStopStrategy (see turn strategies); params: new SmartTurnParams({ stopSecs, ... }) takes the same settings. It makes the same request as Pipecat's: a POST of the audio (up to maxDurationSecs, as a NumPy .npy float32 vector at the analyzer's sample rate) with Content-Type: application/octet-stream, and expects JSON with a prediction (1 when the user is done) and a probability. Any failure, a non-200 status, a timeout after stopSecs or a body that isn't JSON, is logged and counts as "not done", as upstream does. fetch replaces the aiohttp session; pass your own as the fetch option.

Pipecat's older LocalSmartTurnAnalyzerV2 and the Core ML variant need PyTorch or Core ML and are not ported: Smart Turn v3, the default above, replaces them.

Smart Turn on Workers AICopy Anchor Link

Workers AI hosts Smart Turn v2 (@cf/pipecat-ai/smart-turn-v2). WorkersAISmartTurnAnalyzer, from @pipedog/cloudflare, works like HttpSmartTurnAnalyzer with the model on Cloudflare: through the Worker's AI binding, or anywhere with an account ID and API token. It sends the candidate turn as base64 float32 samples and reads Workers AI's is_complete and probability. A failed request counts as "not done"; one that takes longer than stopSecs ends the turn.

// Smart Turn v2 on Workers AI: the AI binding in a Worker, or accountId and apiToken elsewhere.
const userParams = new LLMUserAggregatorParams({
  vadAnalyzer: new SileroVADAnalyzer(),
  userTurnStrategies: new UserTurnStrategies({
    start: [new VADUserTurnStartStrategy(), new TranscriptionUserTurnStartStrategy()],
    stop: [
      new TurnAnalyzerUserTurnStopStrategy({
        turnAnalyzer: new WorkersAISmartTurnAnalyzer({ ai: env.AI }),
      }),
    ],
  }),
});

Smart Turn v2 was trained on 16 kHz audio, so run the pipeline input at 16 kHz. The local Smart Turn v3.2 above is newer and also runs on Workers (pass the model's bytes there), at the cost of the bundle size and memory described in Voice agents on Workers. On Workers AI, Deepgram Flux (WorkersAIFluxSTTService) detects turns itself and needs neither.

ProviderCopy Anchor Link

Some speech-to-text services detect turns themselves. Use one of them and pass no VAD analyzer: the user aggregator follows the service's turn signals, and nothing runs locally.

const stt = new OpenAIRealtimeSTTService({
  apiKey: process.env.OPENAI_API_KEY ?? "",
  turnDetection: { type: "server_vad" },
});
// No vadAnalyzer: the STT service's turn signals drive the turns.
const aggregators = new LLMContextAggregatorPair(context);

This is the mode for constrained hosts. The Durable Objects example uses it: its Worker is 818 KB gzipped, against 13.7 MB with the local models, which is over Cloudflare's 10 MB limit for a Worker.

Speech-to-text services with their own turn detection:

• OpenAIRealtimeSTTService (@pipedog/openai/stt) with turnDetection: { type: "server_vad" }.
• DeepgramFluxSTTService (@pipedog/deepgram/flux/stt): Deepgram's turn-taking model.
• CartesiaTurnsSTTService (@pipedog/cartesia/turns/stt).

Each of these recommends its own turn strategies to the user aggregator (ExternalUserTurnStrategies), so passing no VAD analyzer is all it takes. To keep the bot talking when the provider hears the user, pass shouldInterrupt: false to the service rather than configuring the strategies yourself.

Going furtherCopy Anchor Link