# Azure Voice Live

> Speech-to-speech with AzureVoiceLiveLLMService, over Azure's Voice Live API WebSocket.

Source: https://polvallverdu.github.io/pipedog/docs/services/s2s/azure-voice-live/



## Overview [#overview]

* **`AzureVoiceLiveLLMService`**: one WebSocket session with Azure's Voice Live API, which
  hears the user's audio and answers with audio. Voice Live puts speech recognition, a
  generative model (`gpt-4o-mini`, `gpt-realtime`, `azure-realtime` and others) and Azure
  text-to-speech behind a single realtime session, and adds noise suppression, echo cancellation
  and semantic end-of-turn detection. It replaces the STT, LLM and TTS services in a pipeline,
  and supports function calling and transcripts of both sides.

Voice Live is a different product from OpenAI Realtime deployments on Azure: the two speak
different event names and are not interchangeable. For those, see
[Azure OpenAI Realtime](https://polvallverdu.github.io/pipedog/docs/services/s2s/azure-realtime.md).

<Cards>
  <Card title="Upstream page" href="https://docs.pipecat.ai/api-reference/server/services/s2s/azure-voice-live">
    Pipecat's documentation for the same service.
  </Card>

  <Card title="Source" href="https://github.com/polvallverdu/pipedog/tree/main/packages/azure/src/voice_live">
    `packages/azure/src/voice_live/llm.ts` and `events.ts`.
  </Card>

  <Card title="Azure docs" href="https://learn.microsoft.com/en-us/azure/ai-services/speech-service/voice-live">
    The Voice Live API and its reference.
  </Card>
</Cards>

## Installation [#installation]

```bash
bun add @pipedog/azure
```

```ts
import { AzureVoiceLiveLLMService } from "@pipedog/azure/voice_live/llm";
import { SessionProperties } from "@pipedog/azure/voice_live/events";
```

`@pipedog/azure/voice_live/events` holds the session classes (`SessionProperties`,
`TurnDetection`, `InputAudioTranscription`, `InputAudioNoiseReduction`,
`InputAudioEchoCancellation`, the voice types `AzureStandardVoice`, `AzureCustomVoice`,
`AzurePersonalVoice`, `AzureRealtimeNativeVoice` and `OpenAIVoice`, ...) and the client and
server event models.

## Prerequisites [#prerequisites]

* A Microsoft Foundry (AI Services) resource with Voice Live, in a region that offers it.
* Its endpoint, as the Azure portal shows it: `https://<resource>.services.ai.azure.com` (a
  `cognitiveservices.azure.com` endpoint or a full `wss://.../voice-live/realtime` URL works
  too).
* Either the resource's API key, passed as `apiKey`, or Microsoft Entra ID: pass `tokenProvider`,
  an async function returning a bearer token for the `https://ai.azure.com/.default` scope (for
  example `getBearerTokenProvider()` from `@azure/identity`). One of the two is required.

## AzureVoiceLiveLLMService [#azurevoicelivellmservice]

### Configuration [#configuration]

| Option | Type | Required | Description |
|---|---|---|---|
| `endpoint` | `string` | yes | Voice Live endpoint for the Foundry resource. Accepts the resource endpoint as shown in the Azure portal (`https://<resource>.services.ai.azure.com`) or a full WebSocket URL; the scheme and `/voice-live/realtime` path are filled in when absent. |
| `apiKey` | `string \| null \| undefined` | no | API key for the resource. Required unless `tokenProvider` is given. |
| `tokenProvider` | `AzureTokenProvider \| null \| undefined` | no | Async callable supplying a Microsoft Entra ID bearer token, used instead of `apiKey` when given. Build one with `getBearerTokenProvider()` from `@azure/identity` and the `https://ai.azure.com/.default` scope. |
| `apiVersion` | `string \| undefined` | no | Voice Live API version to request. Defaults to `DEFAULT_API_VERSION`. |
| `settings` | `AzureVoiceLiveLLMSettings \| null \| undefined` | no | Runtime-updatable settings. `model` defaults to "gpt-4o-mini" and is sent as a query parameter on the connection, so it is fixed for the session. The default voice is "en-US-Ava:DragonHDLatestNeural", except for `azure-realtime`, which picks one of its own native voices. When `sessionProperties` is provided, it **replaces** all defaults wholesale — provide a complete `SessionProperties` in that case. |
| `startAudioPaused` | `boolean \| undefined` | no | Whether to start with audio input paused. Defaults to false. |
| `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. |
| `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. |
| `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. |
| `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. |
| `runInParallel` | `boolean \| undefined` | no | Whether to run function calls in parallel or sequentially. Defaults to true. |
| `groupParallelTools` | `boolean \| undefined` | no | Whether to group parallel function calls so the LLM is triggered exactly once after all calls in the batch complete. When false, each function call result triggers the LLM independently as it arrives. Defaults to true. |
| `functionCallTimeoutSecs` | `number \| null \| undefined` | no | Optional timeout in seconds for function calls. A call that runs past it is cancelled: its handler is cancelled (it gets a `CancelledError`), the call is settled as cancelled, and inference runs so the LLM can report that it didn't complete. |
| `fileResolver` | `FileResolver \| null \| undefined` | no | Resolves file URLs in the context that this provider can't fetch itself (see `FileResolver`). Pass one configured with a `fileStorage` to resolve uploaded-file URLs (`new FileResolver({ fileStorage: runnerArgs.fileStorage })` when a runner hosts the bot), and share one instance across services that may consume the same files (e.g. services switched between mid-session) so downloads are reused. Defaults to a resolver that handles `data:` and `http(s)` URLs only. |

### Settings [#settings]

Runtime-updatable settings, passed as `settings: new AzureVoiceLiveLLMService.Settings({...})`
(`AzureVoiceLiveLLMSettings`). Any setting you leave out keeps the default below.

| Setting                                                                    | Type                               | Default         | Description                                                                                                                                  |
| -------------------------------------------------------------------------- | ---------------------------------- | --------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| `model`                                                                    | `string`                           | `"gpt-4o-mini"` | Model backing the session. Fixed for the session: it goes in the connection URL.                                                             |
| `systemInstruction`                                                        | `string \| null`                   | `null`          | Instructions. Kept in sync with `sessionProperties.instructions`.                                                                            |
| `temperature`                                                              | `number \| null`                   | `null`          | Sampling temperature, copied into `sessionProperties.temperature`.                                                                           |
| `sessionProperties`                                                        | `SessionProperties`                | see below       | Session config: modalities, voice, turn detection, transcription, noise reduction, echo cancellation, tools, tool choice, max output tokens. |
| `maxTokens`, `topP`, `topK`, `frequencyPenalty`, `presencePenalty`, `seed` | `number \| null`                   | `null`          | Not used by Voice Live.                                                                                                                      |
| `filterIncompleteUserTurns`                                                | `boolean`                          | `false`         | Filter incomplete user turns.                                                                                                                |
| `userTurnCompletionConfig`                                                 | `UserTurnCompletionConfig \| null` | `null`          | Turn completion config used with `filterIncompleteUserTurns`.                                                                                |

The default `sessionProperties` responds with text and audio in the
`en-US-Ava:DragonHDLatestNeural` voice, runs `azure_semantic_vad` turn detection, transcribes the
caller with `azure-speech` and applies `azure_deep_noise_suppression`. With an `azure-realtime`
model the voice is left out, so the model picks one of its native voices.

A `sessionProperties` you pass replaces those defaults as a whole, so give a complete
configuration. Fields you leave out of it are not sent, and Voice Live's own defaults apply. Top-level
`model`, `systemInstruction` and `temperature` win over the same fields inside it.

### Usage [#usage]

```ts title="voice-live.ts"
import {
  AzureStandardVoice,
  type ConversationItem,
  InputAudioNoiseReduction,
  InputAudioTranscription,
  SessionProperties,
  TurnDetection,
} from "@pipedog/azure/voice_live/events";
import { AzureVoiceLiveLLMService } from "@pipedog/azure/voice_live/llm";
import { LLMUpdateSettingsFrame } from "@pipedog/core/frames/frames";

const llm = new AzureVoiceLiveLLMService({
  apiKey: process.env.AZURE_VOICE_LIVE_API_KEY ?? "",
  // The resource endpoint as the Azure portal shows it.
  endpoint: process.env.AZURE_VOICE_LIVE_ENDPOINT ?? "https://my-resource.services.ai.azure.com",
  settings: new AzureVoiceLiveLLMService.Settings({
    model: "gpt-4o-mini",
    systemInstruction: "You are a helpful assistant in a voice conversation.",
    // Replaces the default session properties as a whole.
    sessionProperties: new SessionProperties({
      modalities: ["text", "audio"],
      voice: new AzureStandardVoice({ name: "en-US-Ava:DragonHDLatestNeural" }),
      turnDetection: new TurnDetection({
        type: "azure_semantic_vad",
        silenceDurationMs: 500,
        removeFillerWords: true,
      }),
      inputAudioTranscription: new InputAudioTranscription({ model: "azure-speech" }),
      inputAudioNoiseReduction: new InputAudioNoiseReduction(),
    }),
  }),
});

llm.addEventHandler(
  "on_conversation_item_created",
  async (_service, itemId: string | null, _item: ConversationItem) => {
    console.log("conversation item created:", itemId);
  },
);
```

With Microsoft Entra ID instead of a key:

```ts
// Microsoft Entra ID instead of a key: any async function returning a bearer
// token for the https://ai.azure.com/.default scope, such as
// getBearerTokenProvider() from @azure/identity.
const entraLlm = new AzureVoiceLiveLLMService({
  endpoint: "https://my-resource.services.ai.azure.com",
  tokenProvider: async () => process.env.AZURE_ENTRA_TOKEN ?? "",
});
```

Change the instructions mid-call with an `LLMUpdateSettingsFrame`. A change to
`systemInstruction`, `temperature` or `sessionProperties` is sent as a `session.update`; other
fields (including `model`) are ignored with a warning:

```ts
// Sends a session.update to the open session.
const update = new LLMUpdateSettingsFrame({
  delta: new AzureVoiceLiveLLMService.Settings({
    systemInstruction: "Answer in Spanish from now on.",
  }),
});

```

### Event handlers [#event-handlers]

| Event                          | Description                                                                                                                  |
| ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------- |
| `on_conversation_item_created` | A conversation item was added. Called with the item id and the item.                                                         |
| `on_conversation_item_updated` | A conversation item changed (a transcript arrived, or a response finished). Called with the item id and the item, or `null`. |

### Notes [#notes]

* Consumes `InputAudioRawFrame`, `LLMContextFrame` and `LLMSetToolsFrame`. Emits
  `TTSAudioRawFrame`, `TTSTextFrame` and `LLMTextFrame` for the bot, and
  `InterimTranscriptionFrame` / `TranscriptionFrame` for the user. `LLMMessagesAppendFrame` is not
  supported yet and is ignored with a warning.
* Audio is 16-bit PCM both ways. The input rate is sent as `input_audio_sampling_rate`; the output
  rate picks the output format (`pcm16_8000hz`, `pcm16_16000hz` or `pcm16` at 24 kHz), and any
  other output rate falls back to 24 kHz. G.711 formats in `sessionProperties` are replaced with
  PCM, with a warning.
* With server-side turn detection (the default), the service broadcasts proposed user turn frames
  from Voice Live's speech events and asks the user aggregator for `ExternalUserTurnStrategies`,
  so no local VAD is needed. `LLMContextAggregatorPair` detects that this is a realtime service.
* For locally driven turns, set `turnDetection: null` (or `false`) in `sessionProperties` and add
  a VAD to the user aggregator. The service then commits the buffer and asks for a response when
  the user stops speaking; after an interruption it clears the input buffer and replays the last
  `startSecs` + 0.3 s of audio, so the start of the user's speech isn't lost. Leaving
  `turnDetection` out keeps Voice Live's own VAD running.
* Without `inputAudioTranscription` the caller's turns never reach the context, so the service
  warns when your `sessionProperties` leaves it out. Set it to `null` to turn transcription off
  without the warning.
* An interruption cancels the running response; whatever it still streams until `response.done`
  is dropped. A response asked for while another is running waits for it to finish.
* Tool results are sent as `function_call_output` items. Async tools
  (`cancelOnInterruption: false`) work, but streamed intermediate results don't: Voice Live has no
  channel for them, so they are dropped with an error.
* `resetConversation()` reconnects and sends the current context to the new session.
  `setAudioInputPaused()` pauses audio input, and `sendClientEvent()` sends any Voice Live client
  event.

## Runtime notes [#runtime-notes]

| Runtime            | Works | What it needs |
| ------------------ | ----- | ------------- |
| Bun                | ✅ yes | Nothing.      |
| Node.js            | ✅ yes | Nothing.      |
| Deno               | ✅ yes | Nothing.      |
| Cloudflare Workers | ✅ yes | Nothing.      |

Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/azure/package.json`: ✅ means the compat smoke test passes on that runtime.

* The socket is opened with `websocketConnect()` from core, with the key in an `api-key` header or
  the Entra token in an `Authorization: Bearer` header. Bun, Node and Deno send headers from the
  global `WebSocket`; on Cloudflare Workers core opens the socket through `fetch`. Browsers can't
  set WebSocket headers.
* The connection URL carries `api-version=2026-07-15` (the `apiVersion` option) and the model.