Azure Voice Live

OverviewCopy Anchor Link

• AzureVoiceLiveLLMService: one WebSocket session with Azure's Voice Live API, which hears the user's audio and answers with audio. Voice Live puts speech recognition, a generative model (gpt-4o-mini, gpt-realtime, azure-realtime and others) and Azure text-to-speech behind a single realtime session, and adds noise suppression, echo cancellation and semantic end-of-turn detection. It replaces the STT, LLM and TTS services in a pipeline, and supports function calling and transcripts of both sides.

Voice Live is a different product from OpenAI Realtime deployments on Azure: the two speak different event names and are not interchangeable. For those, see Azure OpenAI Realtime.

InstallationCopy Anchor Link

bun add @pipedog/azure

import { AzureVoiceLiveLLMService } from "@pipedog/azure/voice_live/llm";
import { SessionProperties } from "@pipedog/azure/voice_live/events";

@pipedog/azure/voice_live/events holds the session classes (SessionProperties, TurnDetection, InputAudioTranscription, InputAudioNoiseReduction, InputAudioEchoCancellation, the voice types AzureStandardVoice, AzureCustomVoice, AzurePersonalVoice, AzureRealtimeNativeVoice and OpenAIVoice, ...) and the client and server event models.

PrerequisitesCopy Anchor Link

• A Microsoft Foundry (AI Services) resource with Voice Live, in a region that offers it.
• Its endpoint, as the Azure portal shows it: https://<resource>.services.ai.azure.com (a cognitiveservices.azure.com endpoint or a full wss://.../voice-live/realtime URL works too).
• Either the resource's API key, passed as apiKey, or Microsoft Entra ID: pass tokenProvider, an async function returning a bearer token for the https://ai.azure.com/.default scope (for example getBearerTokenProvider() from @azure/identity). One of the two is required.

AzureVoiceLiveLLMServiceCopy Anchor Link

ConfigurationCopy Anchor Link

SettingsCopy Anchor Link

Runtime-updatable settings, passed as settings: new AzureVoiceLiveLLMService.Settings({...}) (AzureVoiceLiveLLMSettings). Any setting you leave out keeps the default below.

The default sessionProperties responds with text and audio in the en-US-Ava:DragonHDLatestNeural voice, runs azure_semantic_vad turn detection, transcribes the caller with azure-speech and applies azure_deep_noise_suppression. With an azure-realtime model the voice is left out, so the model picks one of its native voices.

A sessionProperties you pass replaces those defaults as a whole, so give a complete configuration. Fields you leave out of it are not sent, and Voice Live's own defaults apply. Top-level model, systemInstruction and temperature win over the same fields inside it.

UsageCopy Anchor Link

import {
  AzureStandardVoice,
  type ConversationItem,
  InputAudioNoiseReduction,
  InputAudioTranscription,
  SessionProperties,
  TurnDetection,
} from "@pipedog/azure/voice_live/events";
import { AzureVoiceLiveLLMService } from "@pipedog/azure/voice_live/llm";
import { LLMUpdateSettingsFrame } from "@pipedog/core/frames/frames";

const llm = new AzureVoiceLiveLLMService({
  apiKey: process.env.AZURE_VOICE_LIVE_API_KEY ?? "",
  // The resource endpoint as the Azure portal shows it.
  endpoint: process.env.AZURE_VOICE_LIVE_ENDPOINT ?? "https://my-resource.services.ai.azure.com",
  settings: new AzureVoiceLiveLLMService.Settings({
    model: "gpt-4o-mini",
    systemInstruction: "You are a helpful assistant in a voice conversation.",
    // Replaces the default session properties as a whole.
    sessionProperties: new SessionProperties({
      modalities: ["text", "audio"],
      voice: new AzureStandardVoice({ name: "en-US-Ava:DragonHDLatestNeural" }),
      turnDetection: new TurnDetection({
        type: "azure_semantic_vad",
        silenceDurationMs: 500,
        removeFillerWords: true,
      }),
      inputAudioTranscription: new InputAudioTranscription({ model: "azure-speech" }),
      inputAudioNoiseReduction: new InputAudioNoiseReduction(),
    }),
  }),
});

llm.addEventHandler(
  "on_conversation_item_created",
  async (_service, itemId: string | null, _item: ConversationItem) => {
    console.log("conversation item created:", itemId);
  },
);

With Microsoft Entra ID instead of a key:

// Microsoft Entra ID instead of a key: any async function returning a bearer
// token for the https://ai.azure.com/.default scope, such as
// getBearerTokenProvider() from @azure/identity.
const entraLlm = new AzureVoiceLiveLLMService({
  endpoint: "https://my-resource.services.ai.azure.com",
  tokenProvider: async () => process.env.AZURE_ENTRA_TOKEN ?? "",
});

Change the instructions mid-call with an LLMUpdateSettingsFrame. A change to systemInstruction, temperature or sessionProperties is sent as a session.update; other fields (including model) are ignored with a warning:

// Sends a session.update to the open session.
const update = new LLMUpdateSettingsFrame({
  delta: new AzureVoiceLiveLLMService.Settings({
    systemInstruction: "Answer in Spanish from now on.",
  }),
});

Event handlersCopy Anchor Link

NotesCopy Anchor Link

• Consumes InputAudioRawFrame, LLMContextFrame and LLMSetToolsFrame. Emits TTSAudioRawFrame, TTSTextFrame and LLMTextFrame for the bot, and InterimTranscriptionFrame / TranscriptionFrame for the user. LLMMessagesAppendFrame is not supported yet and is ignored with a warning.
• Audio is 16-bit PCM both ways. The input rate is sent as input_audio_sampling_rate; the output rate picks the output format (pcm16_8000hz, pcm16_16000hz or pcm16 at 24 kHz), and any other output rate falls back to 24 kHz. G.711 formats in sessionProperties are replaced with PCM, with a warning.
• With server-side turn detection (the default), the service broadcasts proposed user turn frames from Voice Live's speech events and asks the user aggregator for ExternalUserTurnStrategies, so no local VAD is needed. LLMContextAggregatorPair detects that this is a realtime service.
• For locally driven turns, set turnDetection: null (or false) in sessionProperties and add a VAD to the user aggregator. The service then commits the buffer and asks for a response when the user stops speaking; after an interruption it clears the input buffer and replays the last startSecs + 0.3 s of audio, so the start of the user's speech isn't lost. Leaving turnDetection out keeps Voice Live's own VAD running.
• Without inputAudioTranscription the caller's turns never reach the context, so the service warns when your sessionProperties leaves it out. Set it to null to turn transcription off without the warning.
• An interruption cancels the running response; whatever it still streams until response.done is dropped. A response asked for while another is running waits for it to finish.
• Tool results are sent as function_call_output items. Async tools (cancelOnInterruption: false) work, but streamed intermediate results don't: Voice Live has no channel for them, so they are dropped with an error.
• resetConversation() reconnects and sends the current context to the new session. setAudioInputPaused() pauses audio input, and sendClientEvent() sends any Voice Live client event.

Runtime notesCopy Anchor Link

Generated from pipecat.runtimes and pipecat.compat in packages/azure/package.json: ✅ means the compat smoke test passes on that runtime.

• The socket is opened with websocketConnect() from core, with the key in an api-key header or the Entra token in an Authorization: Bearer header. Bun, Node and Deno send headers from the global WebSocket; on Cloudflare Workers core opens the socket through fetch. Browsers can't set WebSocket headers.
• The connection URL carries api-version=2026-07-15 (the apiVersion option) and the model.