# User muting

> Ignore the user's audio and transcripts while the bot greets them, runs a tool, or speaks, with built-in or custom mute strategies.

Source: https://polvallverdu.github.io/pipedog/docs/concepts/user-muting/



**User mute strategies** decide when the user aggregator ignores the user. While the user is
muted, their audio, VAD events, transcripts and interruptions are dropped at the aggregator: the
bot can't be interrupted, and whatever the user says meanwhile never reaches the context. The user is
not muted by default.

```ts
import {
  AlwaysUserMuteStrategy,
  BaseUserMuteStrategy,
  EndingUserMuteStrategy,
  FirstSpeechUserMuteStrategy,
  FunctionCallUserMuteStrategy,
  MuteUntilFirstBotCompleteUserMuteStrategy,
  OpeningUserMuteStrategy,
} from "@pipedog/core/turns/user_mute/index";
```

## Configuration [#configuration]

Pass the strategies as `userMuteStrategies` on `LLMUserAggregatorParams`:

```ts
const aggregators = new LLMContextAggregatorPair(context, {
  userParams: new LLMUserAggregatorParams({
    vadAnalyzer: new SileroVADAnalyzer(),
    userMuteStrategies: [
      // Let the greeting play out in full...
      new MuteUntilFirstBotCompleteUserMuteStrategy(),
      // ...and ignore the user while a tool runs.
      new FunctionCallUserMuteStrategy(),
    ],
  }),
});
```

Every strategy sees every frame, and the user is muted while **any** of them says so. The example
above mutes from the start of the session until the bot has finished its greeting, and again
whenever a function call runs.

## Built-in strategies [#built-in-strategies]

| Strategy                                    | Mutes                                                                                | Use it to                                                     |
| ------------------------------------------- | ------------------------------------------------------------------------------------ | ------------------------------------------------------------- |
| `MuteUntilFirstBotCompleteUserMuteStrategy` | from the start until the bot first finishes speaking                                 | protect the greeting, including the silence before it         |
| `OpeningUserMuteStrategy`                   | from the start until the bot first finishes speaking, or has spoken for `maxSeconds` | protect the start of a long greeting without muting all of it |
| `FirstSpeechUserMuteStrategy`               | during the bot's first speech only                                                   | protect the greeting, but let the user speak before it starts |
| `FunctionCallUserMuteStrategy`              | while function calls run                                                             | keep the user from talking over a tool call                   |
| `AlwaysUserMuteStrategy`                    | whenever the bot speaks                                                              | turn off barge-in completely                                  |
| `EndingUserMuteStrategy`                    | for good, once the conversation is ending                                            | let the bot finish its goodbye                                |

`OpeningUserMuteStrategy` and `EndingUserMuteStrategy` are pipedog additions; the others are
upstream's and take no options.

### MuteUntilFirstBotCompleteUserMuteStrategy [#muteuntilfirstbotcompleteusermutestrategy]

Mutes from the first frame, before the bot has said anything, and unmutes for good on the first
`BotStoppedSpeakingFrame`. If the bot's first turn fails before it speaks (an `ErrorFrame` arrives
before any `BotStartedSpeakingFrame`), it unmutes and logs a warning, so a failed greeting doesn't
leave the user unable to talk.

### OpeningUserMuteStrategy [#openingusermutestrategy]

Like `MuteUntilFirstBotCompleteUserMuteStrategy`, but the first speech is only protected for
`maxSeconds` from its `BotStartedSpeakingFrame`. On a phone call, that keeps the "hello?" of picking
up from cutting the greeting or reaching the LLM as an answer, while the user can still talk over
the rest of a long greeting. The window is checked as frames arrive (input audio keeps them coming).
Pass `clock` to control time in tests.

```ts
new OpeningUserMuteStrategy({ maxSeconds: 3 });
```

### FirstSpeechUserMuteStrategy [#firstspeechusermutestrategy]

Lets the user speak until the bot starts speaking, mutes them during that first speech, and unmutes
for good when it ends. Later speech can be interrupted as usual.

### FunctionCallUserMuteStrategy [#functioncallusermutestrategy]

Mutes on `FunctionCallsStartedFrame` and tracks each call by its tool call ID; it unmutes once every
call has a `FunctionCallResultFrame` or a `FunctionCallCancelFrame`.

### EndingUserMuteStrategy [#endingusermutestrategy]

Mutes, and stays muted, once it sees an `EndWorkerFrame` or `StopWorkerFrame`, once its `isEnding()`
option returns true, or after `end()` is called. Strategies never see `EndFrame`, and an `EndFrame`
queued after a goodbye arrives once the goodbye has been queued, so when the end is decided
elsewhere (a tool that queues a goodbye and an `EndFrame`, say) tell the strategy with `isEnding` or
`end()`:

```ts
const ending = new EndingUserMuteStrategy();
// In the tool that ends the call:
ending.end();
await worker.queueFrames([new TTSSpeakFrame({ text: "Goodbye!" }), new EndFrame()]);
```

### AlwaysUserMuteStrategy [#alwaysusermutestrategy]

Mutes on `BotStartedSpeakingFrame` and unmutes on `BotStoppedSpeakingFrame`, for the whole session:

```ts
const userParams = new LLMUserAggregatorParams({
  vadAnalyzer: new SileroVADAnalyzer(),
  userMuteStrategies: [new AlwaysUserMuteStrategy()],
});
```

## What muting drops [#what-muting-drops]

While muted, the user aggregator drops these frames instead of processing or forwarding them:
`InputAudioRawFrame`, `VADUserStartedSpeakingFrame`, `VADUserStoppedSpeakingFrame`,
`UserStartedSpeakingFrame`, `UserStoppedSpeakingFrame`, the `Proposed...` variants of those two,
`InterimTranscriptionFrame`, `TranscriptionFrame`, `EagerTranscriptionFrame`, `UserBackchannelFrame`
and `InterruptionFrame`. `StartFrame`, `EndFrame` and `CancelFrame` always pass, and are never
handed to the strategies.

The mute state is updated after each frame: the frame that makes a strategy mute (a
`BotStartedSpeakingFrame`, say) still passes, and muting starts with the next one.

Muting happens in the user aggregator, so processors before it (the STT service, for example) still
receive the audio. To stop the speech-to-text service itself, push an `STTMuteFrame`.

## Muting and interruptions [#muting-and-interruptions]

Muting and disabling interruptions both keep the bot talking, but they treat the user's words
differently:

* **Muted:** what the user says over the bot is discarded. It is never transcribed into the
  context, and the bot never answers it.
* **Interruptions disabled** (`enableInterruptions: false` on the start strategies, see
  [Interruptions](https://polvallverdu.github.io/pipedog/docs/concepts/interruptions-and-cancellation/#controlling-interruptions)): the
  user's turn still starts and is transcribed, and the bot answers it once it has finished speaking.

Use a mute strategy when the bot must finish and anything said over it should be ignored, and
`enableInterruptions: false` when the bot should finish but still hear the user out.

## Events [#events]

The user aggregator emits `on_user_mute_started` when the user becomes muted and
`on_user_mute_stopped` when they are unmuted, whichever strategy caused it, and broadcasts a
`UserMuteStartedFrame` or `UserMuteStoppedFrame` through the pipeline at the same moment:

```ts
const user = aggregators.user();

user.eventHandler("on_user_mute_started", async () => {
  console.log("user muted");
});
user.eventHandler("on_user_mute_stopped", async () => {
  console.log("user unmuted");
});
```

## Writing a strategy [#writing-a-strategy]

Subclass `BaseUserMuteStrategy` (`@pipedog/core/turns/user_mute/base_user_mute_strategy`) and
override `processFrame()`. Unlike turn strategies, which report their decisions through events, a
mute strategy returns a boolean, `true` to mute. The user aggregator calls it with every
frame it receives (except the lifecycle frames above), so it sees the frames that flow downstream
to the aggregator and those broadcast upstream to it, such as the bot speaking frames, the function
call frames and errors. Frames that only travel downstream after the aggregator, like the LLM's
response frames, never reach it.

This one mutes the user while the bot speaks, but only when the application asks for it, such as
before reading terms that must be heard in full:

```ts
/**
 * Mutes the user while the bot speaks, but only while `protecting` is on.
 * Turn it on before the bot reads something that must be heard in full.
 */
export class ProtectedSpeechUserMuteStrategy extends BaseUserMuteStrategy {
  protecting = false;
  protected _botSpeaking = false;

  override async processFrame(frame: Frame): Promise<boolean> {
    await super.processFrame(frame);

    if (frame instanceof BotStartedSpeakingFrame) {
      this._botSpeaking = true;
    } else if (frame instanceof BotStoppedSpeakingFrame) {
      this._botSpeaking = false;
      this.protecting = false; // one protected utterance at a time
    }

    // true mutes the user, starting with the next frame.
    return this.protecting && this._botSpeaking;
  }
}

const protectedSpeech = new ProtectedSpeechUserMuteStrategy();
const userParams = new LLMUserAggregatorParams({
  vadAnalyzer: new SileroVADAnalyzer(),
  userMuteStrategies: [protectedSpeech],
});

// Later, before the bot reads the terms:
protectedSpeech.protecting = true;
await worker.queueFrame(new TTSSpeakFrame({ text: "Before we continue, please listen..." }));
```

Mute strategies are `BaseObject`s, so override `setup()` and `cleanup()` (calling `super`) if the
strategy needs tasks or resources; the aggregator calls both.