Turn strategies

The user aggregator decides when the user's turn starts and stops by asking a list of turn strategies. Start strategies watch for the user beginning to speak; stop strategies decide when they're done. You choose them with userTurnStrategies on LLMUserAggregatorParams.

import { UserTurnStrategies } from "@pipedog/core/turns/user_turn_strategies";
import { VADUserTurnStartStrategy /* , ... */ } from "@pipedog/core/turns/user_start/index";
import { SpeechTimeoutUserTurnStopStrategy /* , ... */ } from "@pipedog/core/turns/user_stop/index";

How a turn worksCopy Anchor Link

Every frame the user aggregator sees goes to every start and stop strategy. A strategy doesn't start or end the turn itself: it reports a decision (the user started, the user is done), and the turn controller combines the decisions of its list, following the list's StrategyMode.

Turn start. By default (startMode: StrategyMode.FIRST) the first start strategy that decides starts the turn: the aggregator broadcasts a UserStartedSpeakingFrame, broadcasts an InterruptionFrame if the strategy has interruptions enabled (the default), and emits on_user_turn_started. With StrategyMode.ALL the turn starts once every start strategy has decided.

Turn stop. By default (stopMode: StrategyMode.ALL) the turn ends once every stop strategy has decided the user is done; with StrategyMode.FIRST, on the first one. Then the aggregator broadcasts a UserStoppedSpeakingFrame, writes the user's transcript to the context, which runs the LLM, and emits on_user_turn_stopped. A list with one stop strategy behaves the same in either mode. The turn never ends while the user is audibly speaking: decisions made meanwhile wait until they stop.

Speaking again. When the user starts speaking again in an open turn, the stop decisions made so far are cleared. If the turn's answer had already started, it is voided: the aggregator interrupts it, or, for a turn that doesn't interrupt the bot, broadcasts a UserTurnResponseCancelFrame.

Stop timeout. A turn that sees no speech, transcript or other turn activity for userTurnStopTimeout seconds (default 5) ends anyway, after on_user_turn_stop_timeout, with what the decisions made so far said about it. It waits while a stop strategy has a decision of its own scheduled (decisionScheduled, such as the LLM's re-prompt timer). It is a safety net for a turn that never finalizes, not a way to tune responsiveness.

To end a turn, a stop strategy needs two things:

1. A transcript. Strategies that wait for the transcript (waitForTranscript, on by default) hold the turn until speech-to-text delivers. This is usually the largest share of the delay.
2. Its own criterion: a silence window, a turn model's verdict, a provider's signal, or the LLM's verdict.

To make the bot answer sooner or later, change the stop strategy (for example userSpeechTimeout). Leave the VAD's stopSecs (0.2 s by default) alone: it is a low-level threshold that the strategies already account for.

The VAD's volume gate is different: line noise and clicks often pass the model's confidence check, and VADParams.minVolume is what stops them. minVolume is a 0–1 value mapped from -110 LUFS to -10 LUFS (the default 0.6 is -50 LUFS). To tune it on a phone line, measure the noise floor in LUFS and set minVolumeDb a few dB above it instead (pipedog only):

new SileroVADAnalyzer({ params: new VADParams({ minVolumeDb: -35 }) });

Where the waiting time goesCopy Anchor Link

• VAD silence (stopSecs). The VAD only reports that speech stopped after this much silence.
• Transcription. Each STT service reports its p99 time to a final transcript (ttfsP99Latency); the stop strategies use it as a deadline counted from the end of speech, and stop waiting early as soon as the service marks a transcript as finalized. The VAD silence and the turn model's inference fall within that deadline rather than adding to it.
• The silence window of SpeechTimeoutUserTurnStopStrategy is the exception: it starts when the VAD reports the end of speech, so the shortest answer after a pause is stopSecs plus userSpeechTimeout (0.8 s by default).

The built-in p99 values assume stopSecs of 0.2. The strategies warn when stopSecs reaches the service's p99, because that leaves no time to wait for the transcript.

ConfigurationCopy Anchor Link

These are the defaults, spelled out:

const userParams = new LLMUserAggregatorParams({
  vadAnalyzer: new SileroVADAnalyzer(),
  userTurnStrategies: new UserTurnStrategies({
    // The first start strategy to fire opens the turn.
    start: [new VADUserTurnStartStrategy(), new TranscriptionUserTurnStartStrategy()],
    // Smart Turn decides when the user is done.
    stop: [
      new TurnAnalyzerUserTurnStopStrategy({ turnAnalyzer: createLocalSmartTurnAnalyzerV3() }),
    ],
  }),
  // The first start strategy to decide starts the turn; every stop strategy
  // has to agree before it ends.
  userTurnStrategiesParams: new UserTurnStrategiesParams({
    startMode: StrategyMode.FIRST,
    stopMode: StrategyMode.ALL,
  }),
  // A turn with no speech, transcript or turn activity for this long ends anyway.
  userTurnStopTimeout: 5.0,
});

new UserTurnStrategies() with no lists, or with empty ones, gets the same defaults: defaultUserTurnStartStrategies() (VAD, then transcription) and defaultUserTurnStopStrategies() (Smart Turn). Both functions are exported, so you can extend the defaults instead of repeating them.

How strategies combineCopy Anchor Link

userTurnStrategiesParams on LLMUserAggregatorParams (also an option of UserTurnProcessor and UserTurnController) takes a UserTurnStrategiesParams:

• startMode and stopMode take a StrategyMode: FIRST, the first strategy to decide wins, or ALL, every strategy has to agree. In an ALL list every strategy is a condition the turn needs, so a fallback such as TranscriptionUserTurnStartStrategy belongs in a FIRST list.
• startMergeMode and stopMergeMode take a MergeMode. A service with its own turn detection (an STT service, say) recommends strategies to the aggregator. With REPLACE, the default, your list is used on its own; with EXTEND, the strategies you gave are added to the service's (or to the defaults, when no service recommends any).

// Two stop strategies: by default (StrategyMode.ALL) both must agree.
// StrategyMode.FIRST ends the turn on whichever decides first.
const userParams = new LLMUserAggregatorParams({
  vadAnalyzer: new SileroVADAnalyzer(),
  userTurnStrategies: new UserTurnStrategies({
    stop: [
      new TurnAnalyzerUserTurnStopStrategy({ turnAnalyzer: createLocalSmartTurnAnalyzerV3() }),
      new SpeechTimeoutUserTurnStopStrategy({ userSpeechTimeout: 2.0 }),
    ],
  }),
  userTurnStrategiesParams: new UserTurnStrategiesParams({ stopMode: StrategyMode.FIRST }),
});

The other turn options on LLMUserAggregatorParams:

vadAnalyzer produces the VAD frames; VADUserTurnStartStrategy is what turns them into a turn start. They are separate things, and the VAD analyzer is set on the aggregator, not the transport. userMuteStrategies is covered in User muting and userIdleTimeout in Turn events & idle users.

Start strategiesCopy Anchor Link

Every start strategy accepts enableInterruptions (default true): whether starting a turn interrupts the bot. Set it to false and the bot keeps talking while the user speaks; what the user said is still transcribed, and answered after the bot finishes.

VADUserTurnStartStrategyCopy Anchor Link

Starts the turn as soon as the VAD hears speech (VADUserStartedSpeakingFrame). The most responsive option, and the first default. It needs a vadAnalyzer on the aggregator, and has no options of its own.

TranscriptionUserTurnStartStrategyCopy Anchor Link

Starts the turn when a transcript arrives. It is the second default, a fallback for speech the VAD misses (a soft voice, for example) but the STT service still transcribes.

MinWordsUserTurnStartStrategyCopy Anchor Link

Starts the turn only once the user has said minWords words, so a quick "okay" or "uh-huh" doesn't cut the bot off. The minimum applies only while the bot is speaking; when the bot is silent, one word starts a turn. Shorter transcripts during bot speech are discarded and never reach the context.

Use it instead of the VAD start strategy, which would open the turn on the first sound:

const userParams = new LLMUserAggregatorParams({
  vadAnalyzer: new SileroVADAnalyzer(),
  userTurnStrategies: new UserTurnStrategies({
    // No VAD start strategy: it would open the turn on the first sound.
    start: [new MinWordsUserTurnStartStrategy({ minWords: 3 })],
    stop: [new SpeechTimeoutUserTurnStopStrategy()],
  }),
});

Four options change upstream behavior, and all are off by default:

• startOnVadWhenBotQuiet starts the turn on VADUserStartedSpeakingFrame while the bot is silent, instead of waiting for the first transcript. A turn that starts with the transcript starts late: the end-of-turn analyzer may already have decided, gets reset, and the reply waits for the STT timeout. With this option the minimum only applies over the bot, and you don't need a VAD start strategy next to it.
• keepBotSpeakingOnTurnStart keeps the bot-speaking state when a turn starts. By default it is cleared, which assumes the turn interrupted the bot; if the bot goes on speaking (interruptions disabled, say), one word would then start the next turn.
• isBackchannel keeps acknowledgements from interrupting the bot, however many words they have: "yes, okay", "mm-hm, right". While the bot speaks, a transcript it matches is discarded like a short one. createBackchannelMatcher() builds a matcher from per-language phrase lists (BACKCHANNEL_PHRASES: English, Spanish, French, German, Italian, Portuguese and Catalan), picked by the transcript's language; pass your own lists to extend them. With the bot silent, a backchannel is an answer and starts the turn as usual.
• requireVadWithinSeconds counts words over the bot only if the VAD heard the user: they are speaking, or the VAD reported the end of their speech at most that many seconds ago. A TV in the background, or words the STT service hallucinates from clicks, then can't interrupt. It needs a VAD in the pipeline; tune what the VAD counts as speech with VADParams (see minVolumeDb above).

new MinWordsUserTurnStartStrategy({
  minWords: 3,
  isBackchannel: createBackchannelMatcher(),
  requireVadWithinSeconds: 1.0,
});

WakePhraseUserTurnStartStrategyCopy Anchor Link

Ignores the user until they say a wake phrase, then lets the conversation run until timeout seconds pass with no activity. It starts turns on its own: on the wake phrase, and, while awake, whenever the user starts speaking or a final transcript arrives (which covers speech the VAD missed). Transcripts that don't contain the phrase while it is asleep are dropped from the aggregation. Matching ignores case and punctuation ("Hey, pipedog!" matches "hey pipedog").

Use it as the only start strategy:

const wakePhrase = new WakePhraseUserTurnStartStrategy({
  phrases: ["hey pipedog", "ok pipedog"],
  timeout: 10.0, // seconds of inactivity before the wake phrase is needed again
});

wakePhrase.eventHandler("on_wake_phrase_detected", async (_strategy, phrase: string) => {
  console.log(`woken up by "${phrase}"`);
});
wakePhrase.eventHandler("on_wake_phrase_timeout", async () => {
  console.log("asleep again");
});

const userParams = new LLMUserAggregatorParams({
  vadAnalyzer: new SileroVADAnalyzer(),
  userTurnStrategies: new UserTurnStrategies({
    // It starts turns on its own once awake, so it is the only start strategy.
    start: [wakePhrase],
  }),
});

To combine it with other start conditions, such as MinWordsUserTurnStartStrategy, use startMode: StrategyMode.ALL, where it is one more condition a turn needs:

// Every start strategy has to agree: the wake phrase, and three words.
const userParams = new LLMUserAggregatorParams({
  vadAnalyzer: new SileroVADAnalyzer(),
  userTurnStrategies: new UserTurnStrategies({
    start: [
      new WakePhraseUserTurnStartStrategy({ phrases: ["hey pipedog"] }),
      new MinWordsUserTurnStartStrategy({ minWords: 3 }),
    ],
  }),
  userTurnStrategiesParams: new UserTurnStrategiesParams({ startMode: StrategyMode.ALL }),
});

It emits on_wake_phrase_detected(strategy, phrase) and on_wake_phrase_timeout(strategy), and exposes its current state ("idle" or "awake"). With singleActivation: true the phrase is needed before every turn: activity no longer extends the window, which closes timeout seconds after the phrase.

KrispVivaIPUserTurnStartStrategyCopy Anchor Link

Uses Krisp's Interruption Prediction model to tell a real interruption from a backchannel ("uh-huh", "yeah"). When the VAD hears speech it feeds the audio to the model, and starts a turn only if the model's probability exceeds threshold. Pair it with TranscriptionUserTurnStartStrategy as a fallback.

modelPath must end in .kef. It and apiKey fall back to the KRISP_VIVA_IP_MODEL_PATH and KRISP_VIVA_API_KEY environment variables.

ExternalUserTurnStartStrategyCopy Anchor Link

Lets another component decide when the turn starts. It understands two frames:

• ProposedUserStartedSpeakingFrame: a service with its own turn detection proposes a turn start. This strategy accepts it, pushes the UserStartedSpeakingFrame and interrupts the bot (if enableInterruptions). Subclass it to change when a proposal opens a turn.
• UserStartedSpeakingFrame: the turn was already decided and announced elsewhere, for example by a shared UserTurnProcessor feeding several aggregators. This strategy adopts it and emits nothing, so the turn isn't announced twice.

Stop strategiesCopy Anchor Link

TurnAnalyzerUserTurnStopStrategyCopy Anchor Link

The default. Feeds audio, VAD and transcription frames to a turn analyzer, Smart Turn v3 here, which predicts from the audio whether the user has finished or only paused. Once it says the turn is complete, the strategy waits for the transcript: it ends the turn on a finalized transcript, or when the STT service's p99 latency has passed since the end of speech.

const stop = new TurnAnalyzerUserTurnStopStrategy({
  turnAnalyzer: createLocalSmartTurnAnalyzerV3({ cpuCount: 2 }),
});

createLocalSmartTurnAnalyzerV3() from @pipedog/audio-smart-turn builds the analyzer with the ONNX runtime and the model shipped with that package. Its options:

params takes a SmartTurnParams (@pipedog/core/audio/turn/smart_turn/base_smart_turn): stopSecs (3 s, the longest silence before the analyzer gives up waiting and calls the turn complete), preSpeechMs (500 ms of audio kept from before speech starts) and maxDurationSecs (8 s, the longest segment the model sees).

waitForTranscript: false ends the turn as soon as the model says so, without waiting for text. Use it when the turn model drives a speech-to-speech service that hears the audio itself; LLMContextAggregatorPair sets it for you in realtime mode.

SpeechTimeoutUserTurnStopStrategyCopy Anchor Link

Ends the turn after a fixed silence. After the VAD reports the end of speech it runs two timers, and ends the turn when both have finished and a transcript has arrived:

• userSpeechTimeout (0.6 s): how long the user may pause and carry on. It always runs in full, and is the knob for responsiveness.
• The STT timer: the service's p99 time to a final transcript. A finalized transcript ends it early.

If the service finalizes its transcripts, the turn ends userSpeechTimeout after the VAD stop. If a transcript arrives without any VAD stop (no VAD analyzer, for instance), userSpeechTimeout measures the silence since the last transcript instead.

const stop = new SpeechTimeoutUserTurnStopStrategy({
  userSpeechTimeout: 0.8, // wait a little longer before answering
});

ExternalUserTurnStopStrategyCopy Anchor Link

The stop-side counterpart of ExternalUserTurnStartStrategy. On a ProposedUserStoppedSpeakingFrame it decides, and pushes the UserStoppedSpeakingFrame itself, holding the turn open until the transcript arrives if waitForTranscript. On a UserStoppedSpeakingFrame decided elsewhere it adopts the decision and emits nothing. It decides once per round: again only after new text, new speech from the service, or speech the VAD heard. To shift the timing, subclass it and override reportUserTurnStopped(), which both paths reach once they decide; pass its options through when the override reports:

/** Waits a grace period after the service says the user stopped. */
export class GracePeriodUserTurnStopStrategy extends ExternalUserTurnStopStrategy {
  #pending: Task<unknown> | null = null;

  /** Cancel a pending report when the user starts speaking again. */
  override async processFrame(frame: Frame): Promise<void> {
    if (frame instanceof ProposedUserStartedSpeakingFrame) {
      await this.#cancelPending();
    }
    await super.processFrame(frame);
  }

  // The base strategy calls this to report that the user is done, so this is
  // where a subclass adjusts the timing.
  override async reportUserTurnStopped(options: ReportUserTurnStoppedOptions = {}): Promise<void> {
    if (this.#pending) return;
    this.#pending = this.createTask(async () => {
      await sleep(0.5);
      this.#pending = null;
      await super.reportUserTurnStopped(options);
    }, `${this}::grace_period`);
  }

  override async handleUserTurnStopped(): Promise<void> {
    await this.#cancelPending();
    await super.handleUserTurnStopped();
  }

  override async cleanup(): Promise<void> {
    await this.#cancelPending();
    await super.cleanup();
  }

  async #cancelPending(): Promise<void> {
    if (this.#pending) {
      const task = this.#pending;
      this.#pending = null;
      await this.cancelTask(task);
    }
  }
}

EagerUserTurnStopStrategyCopy Anchor Link

For STT services that predict the end of a turn before committing to it. It starts generating a response on the prediction (an EagerTranscriptionFrame), so the gap before the committed end of turn is spent generating instead of waiting. The response is held back by the LLM service's speculation gate and discarded if the user resumes speaking, if the committed transcript doesn't match the predicted one, or if the prediction goes unresolved for speculationTimeout seconds. Nothing from a discarded speculation reaches the user or the context, and the context always records the committed transcript.

It extends ExternalUserTurnStopStrategy. Install it with EagerUserTurnStrategies rather than directly.

matchPolicy is a NormalizedMatch by default, which ignores case, punctuation and apostrophes, or an ExactMatch; subclass EagerMatchPolicy and implement matches(eager, final) for your own rule.

LLMTurnCompletionUserTurnStopStrategyCopy Anchor Link

Decides the user is done only when the LLM says so. At start it enables filterIncompleteUserTurns on the LLM (on every LLM behind an LLMSwitcher, too), which makes the LLM begin each response with a marker: ● complete, ◐ cut off mid-thought, ○ needs more time. The LLM service reports each marker upstream as a UserTurnCompletionFrame.

It waits for the turn's inference (waitsForInference) instead of listening to the user, so it needs other stop strategies next to it: once the strategies that listen to the user have decided, the controller runs the inference, and this strategy decides on its marker. ●, a tool call, or a response with no marker end the turn. On ◐ or ○ the turn stays open; if the user then stays quiet for incompleteShortTimeout or incompleteLongTimeout, it ends the turn and asks the LLM to re-engage them. This is what FilterIncompleteUserTurnStrategies installs.

Strategies that wait for the inferenceCopy Anchor Link

The controller runs the turn's inference itself. In a StrategyMode.ALL list it runs it once every stop strategy that listens to the user has decided; a stop strategy whose waitsForInference is true then decides on that inference, and the turn ends. In a StrategyMode.FIRST list the first decision runs the inference, and the turn ends on the decision of the strategy that waits for it. A list where every stop strategy waits for the inference has nothing to run it but the stop timeout, and logs a warning.

// Smart Turn decides the user stopped, which runs the inference; the LLM's
// ● marker then ends the turn.
const strategies = new FilterIncompleteUserTurnStrategies({
  config: new UserTurnCompletionConfig({
    incompleteShortTimeout: 5.0,
    incompleteLongTimeout: 10.0,
  }),
});

// The same strategies, built by hand:
const byHand = new UserTurnStrategies({
  stop: [
    new TurnAnalyzerUserTurnStopStrategy({ turnAnalyzer: createLocalSmartTurnAnalyzerV3() }),
    new LLMTurnCompletionUserTurnStopStrategy(),
  ],
});

// On top of the turn detection an STT service recommends:
const extending = new LLMUserAggregatorParams({
  userTurnStrategies: new FilterIncompleteUserTurnStrategies(),
  userTurnStrategiesParams: new UserTurnStrategiesParams({ stopMergeMode: MergeMode.EXTEND }),
});

PresetsCopy Anchor Link

ExternalUserTurnStrategiesCopy Anchor Link

An ExternalUserTurnStartStrategy and an ExternalUserTurnStopStrategy, for turns decided by a service with its own turn detection, a speech-to-speech service or a shared UserTurnProcessor. Services with turn detection (DeepgramFluxSTTService, CartesiaTurnsSTTService, OpenAIRealtimeSTTService with server VAD) recommend it to the aggregator on their own when you don't pass userTurnStrategies.

A VAD analyzer on the aggregator is optional here: it adds STT metrics but isn't needed for turns.

EagerUserTurnStrategiesCopy Anchor Link

ExternalUserTurnStrategies with an EagerUserTurnStopStrategy as the only stop strategy. The service owns turn detection, so this replaces the detector chain rather than extending it: a local detector alongside would run a second inference for the same turn.

const strategies = new EagerUserTurnStrategies({
  matchPolicy: new ExactMatch(), // default: NormalizedMatch
  speculationTimeout: 5.0,
});

STT services that predict turns take enableEagerEndOfTurn: true (off by default, since every prediction costs an inference, including the ones the service withdraws) and recommend this preset themselves.

FilterIncompleteUserTurnStrategiesCopy Anchor Link

Keeps the start strategies and the stop strategies (the defaults, or yours), and adds an LLMTurnCompletionUserTurnStopStrategy. Once the other stop strategies decide the user stopped, the inference runs, and on_user_turn_stopped fires once the LLM marks the turn complete. It works with either stopMode, and with an eager end of turn: the speculative answer goes out once the transcript commits and the LLM marks the turn complete, in either order, and ◐ or ○ withdraws it. With stopMergeMode: MergeMode.EXTEND, the LLM gate (and any stop strategies you gave) joins the ones a service recommends instead.

Turns with no transcriptCopy Anchor Link

A turn can start on the VAD and end without a transcript: a cough, background noise, speech the STT service couldn't make out. Nothing is written to the context, and by default:

• if the turn interrupted the bot (or came before the bot first finished speaking), the LLM runs once with a developer message, so the bot asks the user to repeat or picks up where it left off;
• if the bot was waiting for the user, the turn goes unanswered.

emptyUserTurn changes this; null leaves every such turn unanswered.

const userParams = new LLMUserAggregatorParams({
  vadAnalyzer: new SileroVADAnalyzer(),
  emptyUserTurn: new EmptyUserTurnConfig({
    idlePrompt: "The user made a sound but said nothing clear. Ask if they said something.",
    maxConsecutiveRecoveries: 2,
  }),
});

Recovery is skipped while the user is muted or a function call is pending, and with a realtime LLM service, which hears the audio itself.

Writing a strategyCopy Anchor Link

Subclass BaseUserTurnStartStrategy or BaseUserTurnStopStrategy and override processFrame(). It sees every frame that reaches the user aggregator and returns nothing. When the strategy decides, it reports the decision, and the turn controller decides when the turn starts or ends:

• start strategies: reportUserTurnStarted(), or triggerResetAggregation() to drop the text gathered so far (what MinWordsUserTurnStartStrategy does with a too-short transcript);
• stop strategies: reportUserTurnStopped() when the user is done. The controller runs the turn's inference itself. A strategy starts one only to answer ahead of the turn end, with startUserTurnSpeculation(), and withdraws it with cancelUserTurnSpeculation().

/** Decides the user started on a final transcript that ends with a question mark. */
export class QuestionUserTurnStartStrategy extends BaseUserTurnStartStrategy {
  override async processFrame(frame: Frame): Promise<void> {
    if (frame instanceof TranscriptionFrame && frame.text.trim().endsWith("?")) {
      // The turn controller starts the turn on this decision (StrategyMode.FIRST),
      // or once every start strategy has decided (StrategyMode.ALL).
      await this.reportUserTurnStarted();
    }
  }
}

A stop strategy that decides from the turn's answer rather than from the user returns true from waitsForInference, and one that is sure to decide soon on its own (a timer of its own) returns true from decisionScheduled meanwhile, so the stop timeout waits for it.

Override handleUserTurnStarted() and handleUserTurnStopped() to reset per-turn state; the controller calls them on every strategy at each turn boundary, whichever strategy caused it. setup() and cleanup() run with the aggregator, and this.taskManager creates tasks for timers.