# Ultravox

> Speech-to-speech with UltravoxRealtimeLLMService, Ultravox's audio-native realtime model.

Source: https://polvallverdu.github.io/pipedog/docs/services/s2s/ultravox/



## Overview [#overview]

* **`UltravoxRealtimeLLMService`**: an Ultravox Realtime call. The service creates the call over
  the Ultravox REST API (or joins one by URL) and streams audio both ways over its WebSocket.
  Ultravox is an audio-native model: it hears the user directly and answers with audio or text.
  It replaces the STT, LLM and TTS services in a pipeline, and supports function calling,
  including async tools.

<Cards>
  <Card title="Upstream page" href="https://docs.pipecat.ai/api-reference/server/services/s2s/ultravox">
    Pipecat's documentation for the same service.
  </Card>

  <Card title="Source" href="https://github.com/polvallverdu/pipedog/tree/main/packages/ultravox/src">
    `packages/ultravox/src/llm.ts`.
  </Card>

  <Card title="Ultravox docs" href="https://docs.ultravox.ai/api-reference/calls/calls-post">
    Creating calls, and the call parameters `extra` can carry.
  </Card>
</Cards>

## Installation [#installation]

```bash
bun add @pipedog/ultravox
```

```ts
import {
  AgentInputParams,
  JoinUrlInputParams,
  OneShotInputParams,
  UltravoxRealtimeLLMService,
} from "@pipedog/ultravox/llm";
```

## Prerequisites [#prerequisites]

* An Ultravox account and an API key from the [Ultravox console](https://app.ultravox.ai).
* Pass the key in the params (`apiKey`); it is sent as the `X-Api-Key` header when the call is
  created. The examples read it from `ULTRAVOX_API_KEY`. Joining by URL needs no key.

## UltravoxRealtimeLLMService [#ultravoxrealtimellmservice]

### Configuration [#configuration]

| Option | Type | Required | Description |
|---|---|---|---|
| `params` | `AgentInputParams \| JoinUrlInputParams \| OneShotInputParams` | yes | Configuration parameters for the model. |
| `settings` | `UltravoxRealtimeLLMSettings \| null \| undefined` | no | Ultravox Realtime LLM settings. If provided, the `settings` values take precedence over default values. |
| `oneShotSelectedTools` | `(FunctionSchema \| DirectFunction)[] \| ToolsSchema \| null \| undefined` | no | Tools to use with this call: a `ToolsSchema` or a plain list of direct functions and/or `FunctionSchema` objects (handlers auto-register). May only be set with `OneShotInputParams`. |
| `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. |
| `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. |
| `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. |
| `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. |
| `runInParallel` | `boolean \| undefined` | no | Whether to run function calls in parallel or sequentially. Defaults to true. |
| `groupParallelTools` | `boolean \| undefined` | no | Whether to group parallel function calls so the LLM is triggered exactly once after all calls in the batch complete. When false, each function call result triggers the LLM independently as it arrives. Defaults to true. |
| `functionCallTimeoutSecs` | `number \| null \| undefined` | no | Optional timeout in seconds for function calls. A call that runs past it is cancelled: its handler is cancelled (it gets a `CancelledError`), the call is settled as cancelled, and inference runs so the LLM can report that it didn't complete. |
| `fileResolver` | `FileResolver \| null \| undefined` | no | Resolves file URLs in the context that this provider can't fetch itself (see `FileResolver`). Pass one configured with a `fileStorage` to resolve uploaded-file URLs (`new FileResolver({ fileStorage: runnerArgs.fileStorage })` when a runner hosts the bot), and share one instance across services that may consume the same files (e.g. services switched between mid-session) so downloads are reused. Defaults to a resolver that handles `data:` and `http(s)` URLs only. |

`params` picks how the call starts:

* **`OneShotInputParams`**: a one-off call with its own `systemPrompt`, `temperature`, `model`,
  `voice`, `metadata`, `outputMedium`, `maxDuration` (seconds, 10 to 3600, default one hour) and
  `extra` call parameters. Tools come from `oneShotSelectedTools`.
* **`AgentInputParams`**: a call with an agent from the Ultravox console (`agentId`), with
  `templateContext`, `metadata`, `outputMedium`, `maxDuration` and `extra`.
* **`JoinUrlInputParams`**: join a call you created yourself, by its `joinUrl`.

| Option | Type | Required | Description |
|---|---|---|---|
| `apiKey` | `string` | yes | Ultravox API key for authentication. |
| `systemPrompt` | `string \| null \| undefined` | no | System prompt to guide the model's behavior. Defaults to null. |
| `temperature` | `number \| undefined` | no | Sampling temperature for response generation (0 to 1). Defaults to 0. |
| `model` | `string \| null \| undefined` | no | Model identifier to use, e.g. "ultravox-v0.7". Defaults to null, which lets Ultravox pick its current default model. |
| `voice` | `string \| null \| undefined` | no | Voice identifier for speech generation. Defaults to null. |
| `metadata` | `Record<string, string> \| undefined` | no | Metadata to attach to the call. Defaults to `{}`. |
| `outputMedium` | `"text" \| "voice" \| null \| undefined` | no | The initial output medium for the agent. Use "text" for text responses or "voice" for audio responses. Defaults to null (voice). |
| `maxDuration` | `number \| undefined` | no | The maximum duration of the call, in seconds (10 s to 1 h). Defaults to one hour. |
| `extra` | `Record<string, unknown> \| undefined` | no | Extra parameters to include in the call creation request. Defaults to `{}`. See the Ultravox API documentation for valid arguments: https://docs.ultravox.ai/api-reference/calls/calls-post |

### Settings [#settings]

Runtime-updatable settings, passed as `settings: new UltravoxRealtimeLLMService.Settings({...})`
(`UltravoxRealtimeLLMSettings`).

| Setting                                          | Type             | Default | Description                                                                  |
| ------------------------------------------------ | ---------------- | ------- | ---------------------------------------------------------------------------- |
| `outputMedium`                                   | `string \| null` | `null`  | `"voice"` or `"text"`. A change is sent to the call as `set_output_medium`.  |
| `model`, `systemInstruction`, `temperature`, ... |                  | `null`  | Set in `params` instead; changing them at runtime is ignored with a warning. |

### Usage [#usage]

```ts title="realtime.ts"
import { FunctionSchema } from "@pipedog/core/adapters/llm/schemas/function_schema";
import { LLMUpdateSettingsFrame } from "@pipedog/core/frames/frames";
import type { FunctionCallParams } from "@pipedog/core/services/llm_service";
import {
  AgentInputParams,
  JoinUrlInputParams,
  OneShotInputParams,
  UltravoxRealtimeLLMService,
} from "@pipedog/ultravox/llm";

const getWeather = new FunctionSchema({
  name: "get_weather",
  description: "Get the current weather for a city.",
  properties: { city: { type: "string", description: "The city name." } },
  required: ["city"],
});

// A one-off call: the system prompt, voice and tools are set here.
const llm = new UltravoxRealtimeLLMService({
  params: new OneShotInputParams({
    apiKey: process.env.ULTRAVOX_API_KEY ?? "",
    systemPrompt: "You are a helpful assistant in a voice conversation.",
    temperature: 0.3,
    maxDuration: 600, // seconds
  }),
  oneShotSelectedTools: [getWeather],
});

llm.registerFunction("get_weather", async (params: FunctionCallParams) => {
  await params.resultCallback({ city: params.arguments.city, temperature: 21 });
});
```

With an agent, or joining a call:

```ts
// A call with an agent configured in the Ultravox console.
const agent = new UltravoxRealtimeLLMService({
  params: new AgentInputParams({
    apiKey: process.env.ULTRAVOX_API_KEY ?? "",
    agentId: "your-agent-id",
    templateContext: { customerName: "Ada" },
  }),
});

// Or join a call created elsewhere, by its join URL.
const joined = new UltravoxRealtimeLLMService({
  params: new JoinUrlInputParams({ joinUrl: "wss://your-join-url" }),
});
```

Switch the reply medium mid-call:

```ts
// Switch the model's replies to text (sends set_output_medium).
const update = new LLMUpdateSettingsFrame({
  delta: new UltravoxRealtimeLLMService.Settings({ outputMedium: "text" }),
});
```

### Notes [#notes]

* Consumes `InputAudioRawFrame` (resampled to 48 kHz and sent as binary), `LLMContextFrame`
  (new user text goes in as `user_text_message`) and `InterruptionFrame`. Emits
  `TTSAudioRawFrame` at 48 kHz, `TTSTextFrame` and `LLMTextFrame` for the bot, and
  `TranscriptionFrame` for the user. Transcripts of the user are Ultravox's and may not match
  what the audio-native model understood.
* When Ultravox clears its playback buffer (the user barged in), the service broadcasts an
  `InterruptionFrame`.
* It does not emit `UserStartedSpeakingFrame` / `UserStoppedSpeakingFrame`; Ultravox decides
  turns on the server. To get local turn frames, add a VAD to the user aggregator.
  `LLMContextAggregatorPair` detects that this is a realtime service either way.
* An async tool (`cancelOnInterruption: false`) gets a placeholder result as soon as Ultravox
  calls it; its real result, and any intermediate results, go to the model as user text.

## Runtime notes [#runtime-notes]

| Runtime            | Works | What it needs |
| ------------------ | ----- | ------------- |
| Bun                | ✅ yes | Nothing.      |
| Node.js            | ✅ yes | Nothing.      |
| Deno               | ✅ yes | Nothing.      |
| Cloudflare Workers | ✅ yes | Nothing.      |

Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/ultravox/package.json`: ✅ means the compat smoke test passes on that runtime.

* Calls are created with `fetch` (upstream uses aiohttp) and the socket is opened with
  `websocketConnect()` from core, so the service runs on Bun, Node, Deno and Cloudflare Workers.
* `maxDuration` is in seconds (upstream takes a `timedelta`); out-of-range values throw a
  `RangeError` from the params constructor.