# Inworld Realtime

> Speech-to-speech with InworldRealtimeLLMService, over Inworld's Realtime API WebSocket.

Source: https://polvallverdu.github.io/pipedog/docs/services/s2s/inworld/



## Overview [#overview]

* **`InworldRealtimeLLMService`**: one WebSocket session with Inworld's Realtime API, which hears
  the user's audio and answers with audio. Under the hood Inworld runs STT, an LLM of your choice
  (for example `openai/gpt-4.1-mini`) and Inworld TTS, with semantic server-side turn detection.
  It replaces the STT, LLM and TTS services in a pipeline, and supports function calling and
  transcripts of both sides.

<Cards>
  <Card title="Upstream page" href="https://docs.pipecat.ai/api-reference/server/services/s2s/inworld">
    Pipecat's documentation for the same service.
  </Card>

  <Card title="Source" href="https://github.com/polvallverdu/pipedog/tree/main/packages/inworld/src/realtime">
    `packages/inworld/src/realtime/llm.ts` and `events.ts`.
  </Card>

  <Card title="Inworld docs" href="https://docs.inworld.ai/api-reference/realtimeAPI/realtime/realtime-websocket">
    The Realtime API over WebSocket.
  </Card>
</Cards>

## Installation [#installation]

```bash
bun add @pipedog/inworld
```

```ts
import { InworldRealtimeLLMService } from "@pipedog/inworld/realtime/llm";
import { SessionProperties } from "@pipedog/inworld/realtime/events";
```

`@pipedog/inworld/realtime/events` holds the session classes (`SessionProperties`,
`AudioConfiguration`, `AudioInput`, `AudioOutput`, `InputTranscription`, `TurnDetection`,
`PCMAudioFormat`, ...) and the client and server event models.

## Prerequisites [#prerequisites]

* An Inworld account and an API key from the [Inworld portal](https://platform.inworld.ai).
* Pass the key as `apiKey` (required). With the default `authType: "basic"` it is sent as
  `Authorization: Basic <apiKey>` (the Base64 key Inworld gives you). Use `authType: "bearer"`
  for a client-side JWT. The examples read it from `INWORLD_API_KEY`.

## InworldRealtimeLLMService [#inworldrealtimellmservice]

### Configuration [#configuration]

| Option | Type | Required | Description |
|---|---|---|---|
| `apiKey` | `string` | yes | Inworld API key for authentication. |
| `llmModel` | `string \| null \| undefined` | no | LLM model to use (e.g. "openai/gpt-4.1-mini"). Shorthand for `sessionProperties.model`. |
| `voice` | `string \| null \| undefined` | no | Voice ID for TTS output (e.g. "Sarah", "Clive"). Shorthand for `sessionProperties.audio.output.voice`. |
| `ttsModel` | `string \| null \| undefined` | no | TTS model to use (e.g. "inworld-tts-2"). Shorthand for `sessionProperties.audio.output.model`. |
| `sttModel` | `string \| null \| undefined` | no | STT model for input transcription (e.g. "inworld/inworld-stt-1"). Defaults to "inworld/inworld-stt-1". Shorthand for `sessionProperties.audio.input.transcription.model`. |
| `baseUrl` | `string \| undefined` | no | WebSocket base URL for the realtime API. Defaults to "wss://api.inworld.ai/api/v1/realtime/session". |
| `authType` | `"basic" \| "bearer" \| undefined` | no | Authentication type. "basic" for server-side API key auth, "bearer" for client-side JWT auth. Defaults to "basic". |
| `settings` | `InworldRealtimeLLMSettings \| null \| undefined` | no | Full settings for fine-grained control. When `sessionProperties` is provided in settings, it **replaces** all defaults wholesale — provide a complete `SessionProperties` in that case. |
| `startAudioPaused` | `boolean \| undefined` | no | Whether to start with audio input paused. Defaults to false. |
| `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. |
| `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. |
| `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. |
| `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. |
| `runInParallel` | `boolean \| undefined` | no | Whether to run function calls in parallel or sequentially. Defaults to true. |
| `groupParallelTools` | `boolean \| undefined` | no | Whether to group parallel function calls so the LLM is triggered exactly once after all calls in the batch complete. When false, each function call result triggers the LLM independently as it arrives. Defaults to true. |
| `functionCallTimeoutSecs` | `number \| null \| undefined` | no | Optional timeout in seconds for function calls. A call that runs past it is cancelled: its handler is cancelled (it gets a `CancelledError`), the call is settled as cancelled, and inference runs so the LLM can report that it didn't complete. |
| `fileResolver` | `FileResolver \| null \| undefined` | no | Resolves file URLs in the context that this provider can't fetch itself (see `FileResolver`). Pass one configured with a `fileStorage` to resolve uploaded-file URLs (`new FileResolver({ fileStorage: runnerArgs.fileStorage })` when a runner hosts the bot), and share one instance across services that may consume the same files (e.g. services switched between mid-session) so downloads are reused. Defaults to a resolver that handles `data:` and `http(s)` URLs only. |

### Settings [#settings]

Runtime-updatable settings, passed as `settings: new InworldRealtimeLLMService.Settings({...})`
(`InworldRealtimeLLMSettings`). Any setting you leave out keeps the default below.

| Setting                                                                    | Type                               | Default                                | Description                                                              |
| -------------------------------------------------------------------------- | ---------------------------------- | -------------------------------------- | ------------------------------------------------------------------------ |
| `model`                                                                    | `string`                           | `llmModel`, or `"openai/gpt-4.1-mini"` | The LLM Inworld runs. Kept in sync with `sessionProperties.model`.       |
| `systemInstruction`                                                        | `string \| null`                   | `null`                                 | Instructions. Kept in sync with `sessionProperties.instructions`.        |
| `temperature`                                                              | `number \| null`                   | `null`                                 | Sampling temperature. Kept in sync with `sessionProperties.temperature`. |
| `sessionProperties`                                                        | `SessionProperties`                | see below                              | Session config: output modalities, audio input and output, tools.        |
| `maxTokens`, `topP`, `topK`, `frequencyPenalty`, `presencePenalty`, `seed` | `number \| null`                   | `null`                                 | Not used by the Realtime API.                                            |
| `filterIncompleteUserTurns`                                                | `boolean`                          | `false`                                | Filter incomplete user turns.                                            |
| `userTurnCompletionConfig`                                                 | `UserTurnCompletionConfig \| null` | `null`                                 | Turn completion config used with `filterIncompleteUserTurns`.            |

The default session has output modalities `["audio", "text"]`, PCM audio both ways (the rates
follow the pipeline's), transcription with `sttModel` (default `"inworld/inworld-stt-1"`),
semantic turn detection with low eagerness that creates and interrupts responses, and the
`voice` (default `"Clive"`) and `ttsModel` (default `"inworld-tts-2"`) for output.

A `sessionProperties` you pass **replaces** those defaults as a whole, so give a complete config.
Top-level `model`, `systemInstruction` and `temperature` win over the same fields inside it.

### Usage [#usage]

```ts title="realtime.ts"
import { LLMUpdateSettingsFrame } from "@pipedog/core/frames/frames";
import {
  AudioConfiguration,
  AudioInput,
  AudioOutput,
  type ConversationItem,
  InputTranscription,
  PCMAudioFormat,
  SessionProperties,
  TurnDetection,
} from "@pipedog/inworld/realtime/events";
import { InworldRealtimeLLMService } from "@pipedog/inworld/realtime/llm";

// The shorthand options fill in the default session.
const llm = new InworldRealtimeLLMService({
  apiKey: process.env.INWORLD_API_KEY ?? "",
  llmModel: "openai/gpt-4.1-mini",
  voice: "Sarah",
  ttsModel: "inworld-tts-2",
  settings: new InworldRealtimeLLMService.Settings({
    systemInstruction: "You are a helpful assistant in a voice conversation.",
  }),
});

llm.addEventHandler(
  "on_conversation_item_created",
  async (_service, itemId: string, _item: ConversationItem) => {
    console.log("conversation item created:", itemId);
  },
);
```

With full control over the session:

```ts
// A full SessionProperties replaces the defaults wholesale: give a complete config.
const custom = new InworldRealtimeLLMService({
  apiKey: process.env.INWORLD_API_KEY ?? "",
  settings: new InworldRealtimeLLMService.Settings({
    sessionProperties: new SessionProperties({
      model: "openai/gpt-4.1-mini",
      temperature: 0.7,
      outputModalities: ["audio", "text"],
      audio: new AudioConfiguration({
        input: new AudioInput({
          format: new PCMAudioFormat({ rate: 24000 }),
          transcription: new InputTranscription({ model: "inworld/inworld-stt-1" }),
          turnDetection: new TurnDetection({ type: "semantic_vad", eagerness: "low" }),
        }),
        output: new AudioOutput({
          format: new PCMAudioFormat({ rate: 24000 }),
          voice: "Sarah",
          model: "inworld-tts-2",
        }),
      }),
    }),
  }),
});
```

Change the instructions mid-call with an `LLMUpdateSettingsFrame`. A change to
`sessionProperties`, `systemInstruction`, `model` or `temperature` is sent as a `session.update`;
other fields are ignored with a warning:

```ts
// Sends a session.update to the open session.
const update = new LLMUpdateSettingsFrame({
  delta: new InworldRealtimeLLMService.Settings({
    systemInstruction: "Answer in Spanish from now on.",
    temperature: 0.3,
  }),
});
```

### Event handlers [#event-handlers]

| Event                          | Description                                                                                                                    |
| ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------ |
| `on_conversation_item_created` | A conversation item was added. Called with the item id and the item.                                                           |
| `on_conversation_item_updated` | A conversation item changed (a transcript arrived, or the response finished). Called with the item id and the item, or `null`. |

### Notes [#notes]

* Consumes `InputAudioRawFrame` (sent in about 60 ms chunks), `LLMContextFrame` and
  `LLMSetToolsFrame`. Emits `TTSAudioRawFrame`, `TTSTextFrame` and `LLMTextFrame` for the bot,
  and `InterimTranscriptionFrame` / `TranscriptionFrame` for the user.
* With Inworld's server-side turn detection (the default), the service broadcasts proposed user
  turn frames from the server's speech events and asks the user aggregator for
  `ExternalUserTurnStrategies`, so no local VAD is needed. `LLMContextAggregatorPair` detects
  that this is a realtime service.
* For locally driven turns, set `turnDetection: null` in `AudioInput` and add a VAD to the user
  aggregator; the service then commits the audio and asks for a response when the user stops
  speaking. Don't run a local VAD with server-side turn detection on: both would send turn frames.
* Tool results are sent as `function_call_output` items. Intermediate results
  (`isFinal: false`) are dropped with a warning: Inworld's model calls the tool again instead of
  relaying them. `cancelOnInterruption: false` isn't reliably supported by Inworld, and the
  service reports an error when it sees one. `LLMMessagesAppendFrame` isn't supported yet.
* `setAudioInputPaused()` pauses input at runtime, `sendClientEvent()` sends any Realtime client
  event, and `resetConversation()` reconnects with a fresh server-side conversation.

## Runtime notes [#runtime-notes]

| Runtime            | Works | What it needs |
| ------------------ | ----- | ------------- |
| Bun                | ✅ yes | Nothing.      |
| Node.js            | ✅ yes | Nothing.      |
| Deno               | ✅ yes | Nothing.      |
| Cloudflare Workers | ✅ yes | Nothing.      |

Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/inworld/package.json`: ✅ means the compat smoke test passes on that runtime.

* The socket is opened with `websocketConnect()` from core, with the key in the `Authorization`
  header and a fresh session `key` in the URL. Bun, Node and Deno send the header from the global
  `WebSocket`; on Cloudflare Workers core opens the socket through `fetch`.
* A failed response whose `status_details` isn't a string is reported as JSON (upstream prints a
  Python dict).