Azure OpenAI Realtime

OverviewCopy Anchor Link

• AzureRealtimeLLMService: OpenAIRealtimeLLMService pointed at an Azure OpenAI deployment of a realtime model (gpt-realtime and friends). The session, events, function calling, transcripts and turn handling are those of the OpenAI service; this class changes the URL and the authentication.

It speaks Azure's v1 realtime API (/openai/v1/realtime), whose protocol is OpenAI's. URLs with a dated api-version serve the older preview protocol, which this service doesn't implement: it logs a warning when given one.

For Azure's own speech-to-speech API, with Azure voices, avatars and noise suppression, see Azure Voice Live.

InstallationCopy Anchor Link

bun add @pipedog/azure

import { AzureRealtimeLLMService } from "@pipedog/azure/realtime/llm";
import { SessionProperties } from "@pipedog/openai/realtime/events";

The session classes come from @pipedog/openai/realtime/events, which @pipedog/azure installs.

PrerequisitesCopy Anchor Link

• An Azure OpenAI (or Azure AI Foundry) resource with a realtime model deployed.
• The resource's API key, or a Microsoft Entra ID identity with access to it. The examples read the key from AZURE_OPENAI_API_KEY.

AzureRealtimeLLMServiceCopy Anchor Link

ConfigurationCopy Anchor Link

SettingsCopy Anchor Link

AzureRealtimeLLMService.Settings (AzureRealtimeLLMSettings) has the same fields and defaults as OpenAIRealtimeLLMSettings. model must be your deployment's name: it is appended to baseUrl as ?model=<deployment>.

UsageCopy Anchor Link

import { AzureRealtimeLLMService } from "@pipedog/azure/realtime/llm";
import {
  AudioConfiguration,
  AudioInput,
  AudioOutput,
  InputAudioTranscription,
  SemanticTurnDetection,
  SessionProperties,
} from "@pipedog/openai/realtime/events";

const llm = new AzureRealtimeLLMService({
  apiKey: process.env.AZURE_OPENAI_API_KEY ?? "",
  baseUrl: "wss://my-resource.openai.azure.com/openai/v1/realtime",
  settings: new AzureRealtimeLLMService.Settings({
    // The deployment name, sent as ?model=<deployment>.
    model: "my-gpt-realtime-deployment",
    systemInstruction: "You are a helpful assistant in a voice conversation.",
    sessionProperties: new SessionProperties({
      audio: new AudioConfiguration({
        input: new AudioInput({
          transcription: new InputAudioTranscription(),
          turnDetection: new SemanticTurnDetection({ eagerness: "medium" }),
        }),
        output: new AudioOutput({ voice: "marin" }),
      }),
    }),
  }),
});

With Microsoft Entra ID instead of a key:

// For example getBearerTokenProvider(credential, "https://ai.azure.com/.default")
// from @azure/identity. It is called once per connection.
declare const tokenProvider: () => Promise<string>;

const withEntra = new AzureRealtimeLLMService({
  tokenProvider,
  baseUrl: "wss://my-resource.openai.azure.com/openai/v1/realtime",
  settings: new AzureRealtimeLLMService.Settings({ model: "my-gpt-realtime-deployment" }),
});

Event handlersCopy Anchor Link

The same as OpenAIRealtimeLLMService: on_conversation_item_created and on_conversation_item_updated.

NotesCopy Anchor Link

• With an API key the socket sends it as the api-key header. With tokenProvider, the token is fetched before each connection and sent as Authorization: Bearer <token>. One of the two is required. The service picks the header in _authHeaders(), the hook OpenAIRealtimeLLMService calls when it connects, and otherwise connects as that service does.
• A baseUrl that already has a query string is used as given, so you can name the deployment (or other parameters) yourself.
• Frames, turn detection, pre-roll and video input behave as in the OpenAI Realtime notes.

Runtime notesCopy Anchor Link

Generated from pipecat.runtimes and pipecat.compat in packages/azure/package.json: ✅ means the compat smoke test passes on that runtime.

• The socket is opened with websocketConnect() from core, which sends the auth header from the global WebSocket on Bun, Node and Deno, and through fetch on Cloudflare Workers.
• Browsers can't set WebSocket headers, so this service is for servers.