OpenAI Realtime

OverviewCopy Anchor Link

• OpenAIRealtimeLLMService: one WebSocket session with OpenAI's Realtime API, which hears the user's audio and answers with audio. It replaces the STT, LLM and TTS services in a pipeline. It supports function calling, transcripts of both sides, video input and the reasoning-capable Realtime models.

For OpenAI's full-duplex live model, see OpenAI Live.

InstallationCopy Anchor Link

bun add @pipedog/openai

import { OpenAIRealtimeLLMService } from "@pipedog/openai/realtime/llm";
import { SessionProperties } from "@pipedog/openai/realtime/events";

@pipedog/openai/realtime/events holds the session classes (SessionProperties, AudioConfiguration, AudioInput, AudioOutput, InputAudioTranscription, TurnDetection, SemanticTurnDetection, Reasoning, ...) and the client and server event models.

PrerequisitesCopy Anchor Link

• An OpenAI account and an API key from the OpenAI dashboard.
• Pass the key as apiKey (required). The examples read it from OPENAI_API_KEY.

OpenAIRealtimeLLMServiceCopy Anchor Link

ConfigurationCopy Anchor Link

SettingsCopy Anchor Link

Runtime-updatable settings, passed as settings: new OpenAIRealtimeLLMService.Settings({...}) (OpenAIRealtimeLLMSettings). Any setting you leave out keeps the default below.

A sessionProperties in an update replaces the stored one as a whole. Top-level model and systemInstruction win over the same fields inside it.

UsageCopy Anchor Link

import { LLMUpdateSettingsFrame } from "@pipedog/core/frames/frames";
import {
  AudioConfiguration,
  AudioInput,
  AudioOutput,
  type ConversationItem,
  InputAudioTranscription,
  SemanticTurnDetection,
  SessionProperties,
} from "@pipedog/openai/realtime/events";
import { OpenAIRealtimeLLMService } from "@pipedog/openai/realtime/llm";

const llm = new OpenAIRealtimeLLMService({
  apiKey: process.env.OPENAI_API_KEY ?? "",
  settings: new OpenAIRealtimeLLMService.Settings({
    model: "gpt-realtime-2.1",
    systemInstruction: "You are a helpful assistant in a voice conversation.",
    sessionProperties: new SessionProperties({
      audio: new AudioConfiguration({
        input: new AudioInput({
          transcription: new InputAudioTranscription(),
          turnDetection: new SemanticTurnDetection({ eagerness: "medium" }),
        }),
        output: new AudioOutput({ voice: "marin" }),
      }),
    }),
  }),
});

llm.addEventHandler(
  "on_conversation_item_created",
  async (_service, itemId: string, _item: ConversationItem) => {
    console.log("conversation item created:", itemId);
  },
);

Change the instructions mid-call with an LLMUpdateSettingsFrame. A change to systemInstruction or sessionProperties is sent as a session.update; other fields are ignored with a warning:

// Sends a session.update to the open session.
const update = new LLMUpdateSettingsFrame({
  delta: new OpenAIRealtimeLLMService.Settings({
    systemInstruction: "Answer in Spanish from now on.",
  }),
});

Event handlersCopy Anchor Link

NotesCopy Anchor Link

• Consumes InputAudioRawFrame (resampled to 24 kHz), InputImageRawFrame (at most one per second, at the videoFrameDetail level), LLMContextFrame, LLMMessagesAppendFrame and LLMSetToolsFrame. Emits TTSAudioRawFrame, TTSTextFrame and LLMTextFrame for the bot, and InterimTranscriptionFrame / TranscriptionFrame for the user.
• With OpenAI's server-side turn detection (the session default), the service broadcasts proposed user turn frames from the server's speech events and asks the user aggregator for ExternalUserTurnStrategies, so no local VAD is needed. LLMContextAggregatorPair detects that this is a realtime service.
• For locally driven turns, set turnDetection: false in AudioInput and add a VAD to the user aggregator. After an interruption the input buffer is cleared and the last userAudioPrerollSecs of audio is replayed, so the start of the user's speech isn't lost. Don't run a local VAD with server-side turn detection on: both would send turn frames.
• setAudioInputPaused(), setVideoInputPaused() and setVideoFrameDetail() change input at runtime. sendClientEvent() sends any Realtime client event, and retrieveConversationItem() fetches an item by id.
• reasoning in sessionProperties is only sent to models recognized as reasoning-capable (gpt-realtime-2, gpt-realtime-2.1); otherwise it is dropped with a warning, since the server would reject the whole update.

Runtime notesCopy Anchor Link

Generated from pipecat.runtimes and pipecat.compat in packages/openai/package.json: ✅ means the compat smoke test passes on that runtime.

• The socket is opened with websocketConnect() from core and the key is sent as an Authorization header. Bun, Node and Deno send it from the global WebSocket; on Cloudflare Workers core opens the socket through fetch. A subclass that authenticates another way overrides _authHeaders(), as Azure Realtime does.
• Video frames are encoded with core's encodeImage() (upstream uses Pillow).
• The deprecated model and sessionProperties constructor options are not ported. Use settings: new OpenAIRealtimeLLMService.Settings({ model, sessionProperties }).
• For realtime models deployed on Azure, see AzureRealtimeLLMService.