OpenAI Live

OverviewCopy Anchor Link

• OpenAILiveLLMService: a session with OpenAI's Live API. The live model (gpt-live-1) listens and speaks at the same time. It decides when to answer, when to stop when the user talks over it, and when to delegate work (search, reasoning, tool use) to a backend text model while the conversation goes on. There is no client-side turn detection or response trigger: the pipeline streams audio in and plays audio out.

Where delegated work runs is set with delegation:

• ResponsesDelegation (OpenAILiveLLMService.ResponsesDelegation): OpenAI hosts the backend model on the Responses API. Its function calls run here, with the handlers registered for the tools in the pipeline's LLMContext.
• ClientDelegation (OpenAILiveLLMService.ClientDelegation): a BackendLLMWorker that runs any pipedog LLM service is the backend.
• null (the default): client delegation with no backend. Delegated requests are declined.

InstallationCopy Anchor Link

bun add @pipedog/openai

import { ClientDelegation, OpenAILiveLLMService, ResponsesDelegation } from "@pipedog/openai/live/llm";
import { BackendLLMWorker } from "@pipedog/core/workers/llm/backend_llm_worker";

PrerequisitesCopy Anchor Link

• An OpenAI account and a project API key from the OpenAI dashboard.
• Pass the key as apiKey (required). The examples read it from OPENAI_API_KEY.

OpenAILiveLLMServiceCopy Anchor Link

ConfigurationCopy Anchor Link

ResponsesDelegation options:

ClientDelegation options:

SettingsCopy Anchor Link

Settings passed as settings: new OpenAILiveLLMService.Settings({...}) (OpenAILiveLLMSettings). model, voice, store and systemInstruction are fixed once the session has started.

The backend model of a ResponsesDelegation takes an OpenAIResponsesLLMService.Settings (see OpenAI Responses). model is required. systemInstruction becomes the backend's instructions, maxCompletionTokens its max_output_tokens, reasoning is sent when set, and extra is merged into the delegation config. The sampling fields, topK, maxTokens and the user turn fields are dropped with a warning.

UsageCopy Anchor Link

import { LLMContext } from "@pipedog/core/processors/aggregators/llm_context";
import { BackendLLMWorker } from "@pipedog/core/workers/llm/backend_llm_worker";
import { OpenAILiveLLMService } from "@pipedog/openai/live/llm";
import { OpenAILLMService } from "@pipedog/openai/llm";
import { OpenAIResponsesLLMService } from "@pipedog/openai/responses/llm";

const llm = new OpenAILiveLLMService({
  apiKey: process.env.OPENAI_API_KEY ?? "",
  settings: new OpenAILiveLLMService.Settings({
    systemInstruction: "You are a helpful assistant in a voice conversation.",
    voice: "marin",
  }),
  delegation: new OpenAILiveLLMService.ResponsesDelegation({
    settings: new OpenAIResponsesLLMService.Settings({ model: "gpt-5.4-mini" }),
  }),
});

llm.addEventHandler("on_session_started", async (_service, session: unknown) => {
  console.log("live session started:", session);
});

Client delegation, with a pipedog LLM service as the backend:

// Any pipedog LLM service can be the backend. Run the pipeline under a WorkerRunner.
const backend = new BackendLLMWorker({
  llm: new OpenAILLMService({ apiKey: process.env.OPENAI_API_KEY ?? "" }),
  context: new LLMContext({
    messages: [{ role: "system", content: "You look things up for the assistant." }],
  }),
});

const live = new OpenAILiveLLMService({
  apiKey: process.env.OPENAI_API_KEY ?? "",
  delegation: new OpenAILiveLLMService.ClientDelegation({ backend, timeoutSecs: 60 }),
});

Event handlersCopy Anchor Link

NotesCopy Anchor Link

• The session starts on the first LLMContextFrame (usually queued as an LLMRunFrame). The context's leading system message (or systemInstruction) becomes the instructions, and the other text messages seed the session as prior conversation.
• Audio input is resampled to 24 kHz. The bot's audio comes out as SpeechOutputAudioRawFrames between TTSStartedFrame and TTSStoppedFrame, with LLMTextFrame / TTSTextFrame for its words and InterimTranscriptionFrame / TranscriptionFrame for the user's.
• The service proposes user turns from the API's transcript turns and asks for ExternalUserTurnStrategies({ enableInterruptions: false }). No InterruptionFrame is broadcast: the model handles being talked over itself, and a delegation keeps running. So every tool behaves as cancelOnInterruption: false; use cancellableByLlm: true for tools the model should be able to cancel.
• Unlike the Realtime services, it is not flagged as a realtime service for the aggregators. It closes each user turn itself once the user falls quiet, so the user aggregator writes the turn to the context before any tool calls it triggers.
• With ClientDelegation, each delegation is sent to the backend with the transcript since the previous one, and delegations run one at a time. A backend worker given as an object is added as a child of the pipeline worker at setup, so the pipeline must run under a WorkerRunner (see pipelines and workers). The backend's output is appended to the session as spoken commentary or as silent context, by its prefersSpoken flag.
• An LLMUpdateSettingsFrame is not sent to the session: the service logs a warning for each changed field. resetConversation() drops the session and opens a new one from the current context.
• Constructing a ResponsesDelegation without settings.model, or the service without a model, throws.

Runtime notesCopy Anchor Link

Generated from pipecat.runtimes and pipecat.compat in packages/openai/package.json: ✅ means the compat smoke test passes on that runtime.

• The socket is opened with websocketConnect() from core, and the key is sent as an Authorization header. Bun, Node and Deno send it from the global WebSocket; on Cloudflare Workers core opens the socket through fetch. webSocketFactory (a pipedog addition) lets you open it some other way.
• The backend model's OpenAIResponsesReasoningConfig is sent as its set fields (effort, summary, mode).