# OpenAI Realtime > Speech-to-speech with OpenAIRealtimeLLMService, over OpenAI's Realtime API WebSocket. Source: https://polvallverdu.github.io/pipedog/docs/services/s2s/openai/ ## Overview [#overview] * **`OpenAIRealtimeLLMService`**: one WebSocket session with OpenAI's Realtime API, which hears the user's audio and answers with audio. It replaces the STT, LLM and TTS services in a pipeline. It supports function calling, transcripts of both sides, video input and the reasoning-capable Realtime models. For OpenAI's full-duplex live model, see [OpenAI Live](https://polvallverdu.github.io/pipedog/docs/services/s2s/openai-live.md). Pipecat's documentation for the same service. `packages/openai/src/realtime/llm.ts` and `events.ts`. The Realtime API over WebSocket. ## Installation [#installation] ```bash bun add @pipedog/openai ``` ```ts import { OpenAIRealtimeLLMService } from "@pipedog/openai/realtime/llm"; import { SessionProperties } from "@pipedog/openai/realtime/events"; ``` `@pipedog/openai/realtime/events` holds the session classes (`SessionProperties`, `AudioConfiguration`, `AudioInput`, `AudioOutput`, `InputAudioTranscription`, `TurnDetection`, `SemanticTurnDetection`, `Reasoning`, ...) and the client and server event models. ## Prerequisites [#prerequisites] * An OpenAI account and an API key from the [OpenAI dashboard](https://platform.openai.com/api-keys). * Pass the key as `apiKey` (required). The examples read it from `OPENAI_API_KEY`. ## OpenAIRealtimeLLMService [#openairealtimellmservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | OpenAI API key for authentication. | | `baseUrl` | `string \| undefined` | no | WebSocket base URL for the realtime API. Defaults to "wss://api.openai.com/v1/realtime". | | `settings` | `OpenAIRealtimeLLMSettings \| null \| undefined` | no | Runtime-updatable settings for this service. | | `startAudioPaused` | `boolean \| undefined` | no | Whether to start with audio input paused. Defaults to false. | | `startVideoPaused` | `boolean \| undefined` | no | Whether to start with video input paused. Defaults to false. | | `videoFrameDetail` | `events.ImageDetail \| undefined` | no | Detail level for video processing. Can be "auto", "low", or "high". This sets the image_detail parameter in the OpenAI Realtime API. "auto" lets the model decide, "low" is faster and uses fewer tokens, "high" provides more detail. Defaults to "auto". | | `userAudioPrerollSecs` | `number \| null \| undefined` | no | In manual turn-detection mode (`turn_detection=False`, locally-driven turns), how much recent audio to replay after an interruption clears the input audio buffer, so the speech onset isn't lost. Defaults to null: auto-sized to the upstream VAD's `startSecs` plus a small margin, falling back to `DEFAULT_USER_AUDIO_PREROLL_SECS` when no VAD is present. Auto-sizing assumes VAD drives turn starts (the default `VADUserTurnStartStrategy`); set this explicitly if you use a non-VAD turn-start strategy. No effect when server-side turn detection is enabled. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `runInParallel` | `boolean \| undefined` | no | Whether to run function calls in parallel or sequentially. Defaults to true. | | `groupParallelTools` | `boolean \| undefined` | no | Whether to group parallel function calls so the LLM is triggered exactly once after all calls in the batch complete. When false, each function call result triggers the LLM independently as it arrives. Defaults to true. | | `functionCallTimeoutSecs` | `number \| null \| undefined` | no | Optional timeout in seconds for function calls. A call that runs past it is cancelled: its handler is cancelled (it gets a `CancelledError`), the call is settled as cancelled, and inference runs so the LLM can report that it didn't complete. | | `fileResolver` | `FileResolver \| null \| undefined` | no | Resolves file URLs in the context that this provider can't fetch itself (see `FileResolver`). Pass one configured with a `fileStorage` to resolve uploaded-file URLs (`new FileResolver({ fileStorage: runnerArgs.fileStorage })` when a runner hosts the bot), and share one instance across services that may consume the same files (e.g. services switched between mid-session) so downloads are reused. Defaults to a resolver that handles `data:` and `http(s)` URLs only. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new OpenAIRealtimeLLMService.Settings({...})` (`OpenAIRealtimeLLMSettings`). Any setting you leave out keeps the default below. | Setting | Type | Default | Description | | ----------------------------------------------------------------------------------------- | ---------------------------------- | --------------------------- | ----------------------------------------------------------------------------------------------------------------------- | | `model` | `string` | `"gpt-realtime-2.1"` | Realtime model. Fixed for the session: it goes in the connection URL. | | `systemInstruction` | `string \| null` | `null` | Instructions. Kept in sync with `sessionProperties.instructions`. | | `sessionProperties` | `SessionProperties` | `new SessionProperties({})` | Session config: output modalities, audio input and output, tools, tool choice, max output tokens, reasoning, and so on. | | `temperature`, `maxTokens`, `topP`, `topK`, `frequencyPenalty`, `presencePenalty`, `seed` | `number \| null` | `null` | Not used by the Realtime API. | | `filterIncompleteUserTurns` | `boolean` | `false` | Filter incomplete user turns. | | `userTurnCompletionConfig` | `UserTurnCompletionConfig \| null` | `null` | Turn completion config used with `filterIncompleteUserTurns`. | A `sessionProperties` in an update replaces the stored one as a whole. Top-level `model` and `systemInstruction` win over the same fields inside it. ### Usage [#usage] ```ts title="realtime.ts" import { LLMUpdateSettingsFrame } from "@pipedog/core/frames/frames"; import { AudioConfiguration, AudioInput, AudioOutput, type ConversationItem, InputAudioTranscription, SemanticTurnDetection, SessionProperties, } from "@pipedog/openai/realtime/events"; import { OpenAIRealtimeLLMService } from "@pipedog/openai/realtime/llm"; const llm = new OpenAIRealtimeLLMService({ apiKey: process.env.OPENAI_API_KEY ?? "", settings: new OpenAIRealtimeLLMService.Settings({ model: "gpt-realtime-2.1", systemInstruction: "You are a helpful assistant in a voice conversation.", sessionProperties: new SessionProperties({ audio: new AudioConfiguration({ input: new AudioInput({ transcription: new InputAudioTranscription(), turnDetection: new SemanticTurnDetection({ eagerness: "medium" }), }), output: new AudioOutput({ voice: "marin" }), }), }), }), }); llm.addEventHandler( "on_conversation_item_created", async (_service, itemId: string, _item: ConversationItem) => { console.log("conversation item created:", itemId); }, ); ``` Change the instructions mid-call with an `LLMUpdateSettingsFrame`. A change to `systemInstruction` or `sessionProperties` is sent as a `session.update`; other fields are ignored with a warning: ```ts // Sends a session.update to the open session. const update = new LLMUpdateSettingsFrame({ delta: new OpenAIRealtimeLLMService.Settings({ systemInstruction: "Answer in Spanish from now on.", }), }); ``` ### Event handlers [#event-handlers] | Event | Description | | ------------------------------ | ----------------------------------------------------------------------------------------------------------------- | | `on_conversation_item_created` | A conversation item was added. Called with the item id and the item. | | `on_conversation_item_updated` | A conversation item changed (a transcript arrived, for example). Called with the item id and the item, or `null`. | ### Notes [#notes] * Consumes `InputAudioRawFrame` (resampled to 24 kHz), `InputImageRawFrame` (at most one per second, at the `videoFrameDetail` level), `LLMContextFrame`, `LLMMessagesAppendFrame` and `LLMSetToolsFrame`. Emits `TTSAudioRawFrame`, `TTSTextFrame` and `LLMTextFrame` for the bot, and `InterimTranscriptionFrame` / `TranscriptionFrame` for the user. * With OpenAI's server-side turn detection (the session default), the service broadcasts proposed user turn frames from the server's speech events and asks the user aggregator for `ExternalUserTurnStrategies`, so no local VAD is needed. `LLMContextAggregatorPair` detects that this is a realtime service. * For locally driven turns, set `turnDetection: false` in `AudioInput` and add a VAD to the user aggregator. After an interruption the input buffer is cleared and the last `userAudioPrerollSecs` of audio is replayed, so the start of the user's speech isn't lost. Don't run a local VAD with server-side turn detection on: both would send turn frames. * `setAudioInputPaused()`, `setVideoInputPaused()` and `setVideoFrameDetail()` change input at runtime. `sendClientEvent()` sends any Realtime client event, and `retrieveConversationItem()` fetches an item by id. * `reasoning` in `sessionProperties` is only sent to models recognized as reasoning-capable (`gpt-realtime-2`, `gpt-realtime-2.1`); otherwise it is dropped with a warning, since the server would reject the whole update. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/openai/package.json`: ✅ means the compat smoke test passes on that runtime. * The socket is opened with `websocketConnect()` from core and the key is sent as an `Authorization` header. Bun, Node and Deno send it from the global `WebSocket`; on Cloudflare Workers core opens the socket through `fetch`. A subclass that authenticates another way overrides `_authHeaders()`, as [Azure Realtime](https://polvallverdu.github.io/pipedog/docs/services/s2s/azure-realtime.md) does. * Video frames are encoded with core's `encodeImage()` (upstream uses Pillow). * The deprecated `model` and `sessionProperties` constructor options are not ported. Use `settings: new OpenAIRealtimeLLMService.Settings({ model, sessionProperties })`. * For realtime models deployed on Azure, see [`AzureRealtimeLLMService`](https://polvallverdu.github.io/pipedog/docs/services/s2s/azure-realtime.md).