User muting

User mute strategies decide when the user aggregator ignores the user. While the user is muted, their audio, VAD events, transcripts and interruptions are dropped at the aggregator: the bot can't be interrupted, and whatever the user says meanwhile never reaches the context. The user is not muted by default.

import {
  AlwaysUserMuteStrategy,
  BaseUserMuteStrategy,
  EndingUserMuteStrategy,
  FirstSpeechUserMuteStrategy,
  FunctionCallUserMuteStrategy,
  MuteUntilFirstBotCompleteUserMuteStrategy,
  OpeningUserMuteStrategy,
} from "@pipedog/core/turns/user_mute/index";

ConfigurationCopy Anchor Link

Pass the strategies as userMuteStrategies on LLMUserAggregatorParams:

const aggregators = new LLMContextAggregatorPair(context, {
  userParams: new LLMUserAggregatorParams({
    vadAnalyzer: new SileroVADAnalyzer(),
    userMuteStrategies: [
      // Let the greeting play out in full...
      new MuteUntilFirstBotCompleteUserMuteStrategy(),
      // ...and ignore the user while a tool runs.
      new FunctionCallUserMuteStrategy(),
    ],
  }),
});

Every strategy sees every frame, and the user is muted while any of them says so. The example above mutes from the start of the session until the bot has finished its greeting, and again whenever a function call runs.

Built-in strategiesCopy Anchor Link

OpeningUserMuteStrategy and EndingUserMuteStrategy are pipedog additions; the others are upstream's and take no options.

MuteUntilFirstBotCompleteUserMuteStrategyCopy Anchor Link

Mutes from the first frame, before the bot has said anything, and unmutes for good on the first BotStoppedSpeakingFrame. If the bot's first turn fails before it speaks (an ErrorFrame arrives before any BotStartedSpeakingFrame), it unmutes and logs a warning, so a failed greeting doesn't leave the user unable to talk.

OpeningUserMuteStrategyCopy Anchor Link

Like MuteUntilFirstBotCompleteUserMuteStrategy, but the first speech is only protected for maxSeconds from its BotStartedSpeakingFrame. On a phone call, that keeps the "hello?" of picking up from cutting the greeting or reaching the LLM as an answer, while the user can still talk over the rest of a long greeting. The window is checked as frames arrive (input audio keeps them coming). Pass clock to control time in tests.

new OpeningUserMuteStrategy({ maxSeconds: 3 });

FirstSpeechUserMuteStrategyCopy Anchor Link

Lets the user speak until the bot starts speaking, mutes them during that first speech, and unmutes for good when it ends. Later speech can be interrupted as usual.

FunctionCallUserMuteStrategyCopy Anchor Link

Mutes on FunctionCallsStartedFrame and tracks each call by its tool call ID; it unmutes once every call has a FunctionCallResultFrame or a FunctionCallCancelFrame.

EndingUserMuteStrategyCopy Anchor Link

Mutes, and stays muted, once it sees an EndWorkerFrame or StopWorkerFrame, once its isEnding() option returns true, or after end() is called. Strategies never see EndFrame, and an EndFrame queued after a goodbye arrives once the goodbye has been queued, so when the end is decided elsewhere (a tool that queues a goodbye and an EndFrame, say) tell the strategy with isEnding or end():

const ending = new EndingUserMuteStrategy();
// In the tool that ends the call:
ending.end();
await worker.queueFrames([new TTSSpeakFrame({ text: "Goodbye!" }), new EndFrame()]);

AlwaysUserMuteStrategyCopy Anchor Link

Mutes on BotStartedSpeakingFrame and unmutes on BotStoppedSpeakingFrame, for the whole session:

const userParams = new LLMUserAggregatorParams({
  vadAnalyzer: new SileroVADAnalyzer(),
  userMuteStrategies: [new AlwaysUserMuteStrategy()],
});

What muting dropsCopy Anchor Link

While muted, the user aggregator drops these frames instead of processing or forwarding them: InputAudioRawFrame, VADUserStartedSpeakingFrame, VADUserStoppedSpeakingFrame, UserStartedSpeakingFrame, UserStoppedSpeakingFrame, the Proposed... variants of those two, InterimTranscriptionFrame, TranscriptionFrame, EagerTranscriptionFrame, UserBackchannelFrame and InterruptionFrame. StartFrame, EndFrame and CancelFrame always pass, and are never handed to the strategies.

The mute state is updated after each frame: the frame that makes a strategy mute (a BotStartedSpeakingFrame, say) still passes, and muting starts with the next one.

Muting happens in the user aggregator, so processors before it (the STT service, for example) still receive the audio. To stop the speech-to-text service itself, push an STTMuteFrame.

Muting and interruptionsCopy Anchor Link

Muting and disabling interruptions both keep the bot talking, but they treat the user's words differently:

• Muted: what the user says over the bot is discarded. It is never transcribed into the context, and the bot never answers it.
• Interruptions disabled (enableInterruptions: false on the start strategies, see Interruptions): the user's turn still starts and is transcribed, and the bot answers it once it has finished speaking.

Use a mute strategy when the bot must finish and anything said over it should be ignored, and enableInterruptions: false when the bot should finish but still hear the user out.

EventsCopy Anchor Link

The user aggregator emits on_user_mute_started when the user becomes muted and on_user_mute_stopped when they are unmuted, whichever strategy caused it, and broadcasts a UserMuteStartedFrame or UserMuteStoppedFrame through the pipeline at the same moment:

const user = aggregators.user();

user.eventHandler("on_user_mute_started", async () => {
  console.log("user muted");
});
user.eventHandler("on_user_mute_stopped", async () => {
  console.log("user unmuted");
});

Writing a strategyCopy Anchor Link

Subclass BaseUserMuteStrategy (@pipedog/core/turns/user_mute/base_user_mute_strategy) and override processFrame(). Unlike turn strategies, which report their decisions through events, a mute strategy returns a boolean, true to mute. The user aggregator calls it with every frame it receives (except the lifecycle frames above), so it sees the frames that flow downstream to the aggregator and those broadcast upstream to it, such as the bot speaking frames, the function call frames and errors. Frames that only travel downstream after the aggregator, like the LLM's response frames, never reach it.

This one mutes the user while the bot speaks, but only when the application asks for it, such as before reading terms that must be heard in full:

/**
 * Mutes the user while the bot speaks, but only while `protecting` is on.
 * Turn it on before the bot reads something that must be heard in full.
 */
export class ProtectedSpeechUserMuteStrategy extends BaseUserMuteStrategy {
  protecting = false;
  protected _botSpeaking = false;

  override async processFrame(frame: Frame): Promise<boolean> {
    await super.processFrame(frame);

    if (frame instanceof BotStartedSpeakingFrame) {
      this._botSpeaking = true;
    } else if (frame instanceof BotStoppedSpeakingFrame) {
      this._botSpeaking = false;
      this.protecting = false; // one protected utterance at a time
    }

    // true mutes the user, starting with the next frame.
    return this.protecting && this._botSpeaking;
  }
}

const protectedSpeech = new ProtectedSpeechUserMuteStrategy();
const userParams = new LLMUserAggregatorParams({
  vadAnalyzer: new SileroVADAnalyzer(),
  userMuteStrategies: [protectedSpeech],
});

// Later, before the bot reads the terms:
protectedSpeech.protecting = true;
await worker.queueFrame(new TTSSpeakFrame({ text: "Before we continue, please listen..." }));

Mute strategies are BaseObjects, so override setup() and cleanup() (calling super) if the strategy needs tasks or resources; the aggregator calls both.