Ultravox

OverviewCopy Anchor Link

• UltravoxRealtimeLLMService: an Ultravox Realtime call. The service creates the call over the Ultravox REST API (or joins one by URL) and streams audio both ways over its WebSocket. Ultravox is an audio-native model: it hears the user directly and answers with audio or text. It replaces the STT, LLM and TTS services in a pipeline, and supports function calling, including async tools.

InstallationCopy Anchor Link

bun add @pipedog/ultravox

import {
  AgentInputParams,
  JoinUrlInputParams,
  OneShotInputParams,
  UltravoxRealtimeLLMService,
} from "@pipedog/ultravox/llm";

PrerequisitesCopy Anchor Link

• An Ultravox account and an API key from the Ultravox console.
• Pass the key in the params (apiKey); it is sent as the X-Api-Key header when the call is created. The examples read it from ULTRAVOX_API_KEY. Joining by URL needs no key.

UltravoxRealtimeLLMServiceCopy Anchor Link

ConfigurationCopy Anchor Link

params picks how the call starts:

• OneShotInputParams: a one-off call with its own systemPrompt, temperature, model, voice, metadata, outputMedium, maxDuration (seconds, 10 to 3600, default one hour) and extra call parameters. Tools come from oneShotSelectedTools.
• AgentInputParams: a call with an agent from the Ultravox console (agentId), with templateContext, metadata, outputMedium, maxDuration and extra.
• JoinUrlInputParams: join a call you created yourself, by its joinUrl.

SettingsCopy Anchor Link

Runtime-updatable settings, passed as settings: new UltravoxRealtimeLLMService.Settings({...}) (UltravoxRealtimeLLMSettings).

UsageCopy Anchor Link

import { FunctionSchema } from "@pipedog/core/adapters/llm/schemas/function_schema";
import { LLMUpdateSettingsFrame } from "@pipedog/core/frames/frames";
import type { FunctionCallParams } from "@pipedog/core/services/llm_service";
import {
  AgentInputParams,
  JoinUrlInputParams,
  OneShotInputParams,
  UltravoxRealtimeLLMService,
} from "@pipedog/ultravox/llm";

const getWeather = new FunctionSchema({
  name: "get_weather",
  description: "Get the current weather for a city.",
  properties: { city: { type: "string", description: "The city name." } },
  required: ["city"],
});

// A one-off call: the system prompt, voice and tools are set here.
const llm = new UltravoxRealtimeLLMService({
  params: new OneShotInputParams({
    apiKey: process.env.ULTRAVOX_API_KEY ?? "",
    systemPrompt: "You are a helpful assistant in a voice conversation.",
    temperature: 0.3,
    maxDuration: 600, // seconds
  }),
  oneShotSelectedTools: [getWeather],
});

llm.registerFunction("get_weather", async (params: FunctionCallParams) => {
  await params.resultCallback({ city: params.arguments.city, temperature: 21 });
});

With an agent, or joining a call:

// A call with an agent configured in the Ultravox console.
const agent = new UltravoxRealtimeLLMService({
  params: new AgentInputParams({
    apiKey: process.env.ULTRAVOX_API_KEY ?? "",
    agentId: "your-agent-id",
    templateContext: { customerName: "Ada" },
  }),
});

// Or join a call created elsewhere, by its join URL.
const joined = new UltravoxRealtimeLLMService({
  params: new JoinUrlInputParams({ joinUrl: "wss://your-join-url" }),
});

Switch the reply medium mid-call:

// Switch the model's replies to text (sends set_output_medium).
const update = new LLMUpdateSettingsFrame({
  delta: new UltravoxRealtimeLLMService.Settings({ outputMedium: "text" }),
});

NotesCopy Anchor Link

• Consumes InputAudioRawFrame (resampled to 48 kHz and sent as binary), LLMContextFrame (new user text goes in as user_text_message) and InterruptionFrame. Emits TTSAudioRawFrame at 48 kHz, TTSTextFrame and LLMTextFrame for the bot, and TranscriptionFrame for the user. Transcripts of the user are Ultravox's and may not match what the audio-native model understood.
• When Ultravox clears its playback buffer (the user barged in), the service broadcasts an InterruptionFrame.
• It does not emit UserStartedSpeakingFrame / UserStoppedSpeakingFrame; Ultravox decides turns on the server. To get local turn frames, add a VAD to the user aggregator. LLMContextAggregatorPair detects that this is a realtime service either way.
• An async tool (cancelOnInterruption: false) gets a placeholder result as soon as Ultravox calls it; its real result, and any intermediate results, go to the model as user text.

Runtime notesCopy Anchor Link

Generated from pipecat.runtimes and pipecat.compat in packages/ultravox/package.json: ✅ means the compat smoke test passes on that runtime.

• Calls are created with fetch (upstream uses aiohttp) and the socket is opened with websocketConnect() from core, so the service runs on Bun, Node, Deno and Cloudflare Workers.
• maxDuration is in seconds (upstream takes a timedelta); out-of-range values throw a RangeError from the params constructor.