# Azure OpenAI Realtime

> Speech-to-speech with OpenAI's realtime models deployed on Azure, with AzureRealtimeLLMService.

Source: https://polvallverdu.github.io/pipedog/docs/services/s2s/azure-realtime/



## Overview [#overview]

* **`AzureRealtimeLLMService`**: [`OpenAIRealtimeLLMService`](https://polvallverdu.github.io/pipedog/docs/services/s2s/openai.md) pointed
  at an Azure OpenAI deployment of a realtime model (`gpt-realtime` and friends). The session,
  events, function calling, transcripts and turn handling are those of the OpenAI service; this
  class changes the URL and the authentication.

It speaks Azure's v1 realtime API (`/openai/v1/realtime`), whose protocol is OpenAI's. URLs with
a dated `api-version` serve the older preview protocol, which this service doesn't implement: it
logs a warning when given one.

For Azure's own speech-to-speech API, with Azure voices, avatars and noise suppression, see
[Azure Voice Live](https://polvallverdu.github.io/pipedog/docs/services/s2s/azure-voice-live.md).

<Cards>
  <Card title="Upstream page" href="https://docs.pipecat.ai/api-reference/server/services/s2s/azure">
    Pipecat's documentation for the same service.
  </Card>

  <Card title="Source" href="https://github.com/polvallverdu/pipedog/tree/main/packages/azure/src/realtime">
    `packages/azure/src/realtime/llm.ts`.
  </Card>

  <Card title="Azure docs" href="https://learn.microsoft.com/en-us/azure/ai-foundry/openai/how-to/realtime-audio-websockets">
    The realtime API over WebSocket on Azure.
  </Card>
</Cards>

## Installation [#installation]

```bash
bun add @pipedog/azure
```

```ts
import { AzureRealtimeLLMService } from "@pipedog/azure/realtime/llm";
import { SessionProperties } from "@pipedog/openai/realtime/events";
```

The session classes come from `@pipedog/openai/realtime/events`, which `@pipedog/azure`
installs.

## Prerequisites [#prerequisites]

* An Azure OpenAI (or Azure AI Foundry) resource with a realtime model deployed.
* The resource's API key, or a Microsoft Entra ID identity with access to it. The examples read
  the key from `AZURE_OPENAI_API_KEY`.

## AzureRealtimeLLMService [#azurerealtimellmservice]

### Configuration [#configuration]

| Option | Type | Required | Description |
|---|---|---|---|
| `baseUrl` | `string` | yes | The Azure Realtime WebSocket endpoint URL, on Azure's v1 API surface. The deployment is appended as `?model=<Settings.model>`, so `Settings.model` must name the deployment: wss://my-resource.openai.azure.com/openai/v1/realtime A URL that already carries a query string is used verbatim, leaving the deployment for the caller to name. |
| `apiKey` | `string \| null \| undefined` | no | The API key for the Azure OpenAI service. Required unless `tokenProvider` is given. |
| `tokenProvider` | `AzureTokenProvider \| null \| undefined` | no | Async callable supplying a Microsoft Entra ID bearer token, used instead of `apiKey`. |
| `settings` | `AzureRealtimeLLMSettings \| null \| undefined` | no | Runtime-updatable settings for this service. |
| `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. |
| `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. |
| `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. |
| `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. |
| `runInParallel` | `boolean \| undefined` | no | Whether to run function calls in parallel or sequentially. Defaults to true. |
| `groupParallelTools` | `boolean \| undefined` | no | Whether to group parallel function calls so the LLM is triggered exactly once after all calls in the batch complete. When false, each function call result triggers the LLM independently as it arrives. Defaults to true. |
| `functionCallTimeoutSecs` | `number \| null \| undefined` | no | Optional timeout in seconds for function calls. A call that runs past it is cancelled: its handler is cancelled (it gets a `CancelledError`), the call is settled as cancelled, and inference runs so the LLM can report that it didn't complete. |
| `fileResolver` | `FileResolver \| null \| undefined` | no | Resolves file URLs in the context that this provider can't fetch itself (see `FileResolver`). Pass one configured with a `fileStorage` to resolve uploaded-file URLs (`new FileResolver({ fileStorage: runnerArgs.fileStorage })` when a runner hosts the bot), and share one instance across services that may consume the same files (e.g. services switched between mid-session) so downloads are reused. Defaults to a resolver that handles `data:` and `http(s)` URLs only. |
| `startAudioPaused` | `boolean \| undefined` | no | Whether to start with audio input paused. Defaults to false. |
| `startVideoPaused` | `boolean \| undefined` | no | Whether to start with video input paused. Defaults to false. |
| `videoFrameDetail` | `ImageDetail \| undefined` | no | Detail level for video processing. Can be "auto", "low", or "high". This sets the image_detail parameter in the OpenAI Realtime API. "auto" lets the model decide, "low" is faster and uses fewer tokens, "high" provides more detail. Defaults to "auto". |
| `userAudioPrerollSecs` | `number \| null \| undefined` | no | In manual turn-detection mode (`turn_detection=False`, locally-driven turns), how much recent audio to replay after an interruption clears the input audio buffer, so the speech onset isn't lost. Defaults to null: auto-sized to the upstream VAD's `startSecs` plus a small margin, falling back to `DEFAULT_USER_AUDIO_PREROLL_SECS` when no VAD is present. Auto-sizing assumes VAD drives turn starts (the default `VADUserTurnStartStrategy`); set this explicitly if you use a non-VAD turn-start strategy. No effect when server-side turn detection is enabled. |

### Settings [#settings]

`AzureRealtimeLLMService.Settings` (`AzureRealtimeLLMSettings`) has the same fields and defaults
as [`OpenAIRealtimeLLMSettings`](https://polvallverdu.github.io/pipedog/docs/services/s2s/openai/#settings). `model` must be your
deployment's name: it is appended to `baseUrl` as `?model=<deployment>`.

### Usage [#usage]

```ts title="realtime.ts"
import { AzureRealtimeLLMService } from "@pipedog/azure/realtime/llm";
import {
  AudioConfiguration,
  AudioInput,
  AudioOutput,
  InputAudioTranscription,
  SemanticTurnDetection,
  SessionProperties,
} from "@pipedog/openai/realtime/events";

const llm = new AzureRealtimeLLMService({
  apiKey: process.env.AZURE_OPENAI_API_KEY ?? "",
  baseUrl: "wss://my-resource.openai.azure.com/openai/v1/realtime",
  settings: new AzureRealtimeLLMService.Settings({
    // The deployment name, sent as ?model=<deployment>.
    model: "my-gpt-realtime-deployment",
    systemInstruction: "You are a helpful assistant in a voice conversation.",
    sessionProperties: new SessionProperties({
      audio: new AudioConfiguration({
        input: new AudioInput({
          transcription: new InputAudioTranscription(),
          turnDetection: new SemanticTurnDetection({ eagerness: "medium" }),
        }),
        output: new AudioOutput({ voice: "marin" }),
      }),
    }),
  }),
});
```

With Microsoft Entra ID instead of a key:

```ts
// For example getBearerTokenProvider(credential, "https://ai.azure.com/.default")
// from @azure/identity. It is called once per connection.
declare const tokenProvider: () => Promise<string>;

const withEntra = new AzureRealtimeLLMService({
  tokenProvider,
  baseUrl: "wss://my-resource.openai.azure.com/openai/v1/realtime",
  settings: new AzureRealtimeLLMService.Settings({ model: "my-gpt-realtime-deployment" }),
});
```

### Event handlers [#event-handlers]

The same as [`OpenAIRealtimeLLMService`](https://polvallverdu.github.io/pipedog/docs/services/s2s/openai/#event-handlers):
`on_conversation_item_created` and `on_conversation_item_updated`.

### Notes [#notes]

* With an API key the socket sends it as the `api-key` header. With `tokenProvider`, the token
  is fetched before each connection and sent as `Authorization: Bearer <token>`. One of the two
  is required. The service picks the header in `_authHeaders()`, the hook
  `OpenAIRealtimeLLMService` calls when it connects, and otherwise connects as that service does.
* A `baseUrl` that already has a query string is used as given, so you can name the deployment
  (or other parameters) yourself.
* Frames, turn detection, pre-roll and video input behave as in the
  [OpenAI Realtime notes](https://polvallverdu.github.io/pipedog/docs/services/s2s/openai/#notes).

## Runtime notes [#runtime-notes]

| Runtime            | Works | What it needs |
| ------------------ | ----- | ------------- |
| Bun                | ✅ yes | Nothing.      |
| Node.js            | ✅ yes | Nothing.      |
| Deno               | ✅ yes | Nothing.      |
| Cloudflare Workers | ✅ yes | Nothing.      |

Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/azure/package.json`: ✅ means the compat smoke test passes on that runtime.

* The socket is opened with `websocketConnect()` from core, which sends the auth header from the
  global `WebSocket` on Bun, Node and Deno, and through `fetch` on Cloudflare Workers.
* Browsers can't set WebSocket headers, so this service is for servers.