Inworld Realtime

OverviewCopy Anchor Link

• InworldRealtimeLLMService: one WebSocket session with Inworld's Realtime API, which hears the user's audio and answers with audio. Under the hood Inworld runs STT, an LLM of your choice (for example openai/gpt-4.1-mini) and Inworld TTS, with semantic server-side turn detection. It replaces the STT, LLM and TTS services in a pipeline, and supports function calling and transcripts of both sides.

InstallationCopy Anchor Link

bun add @pipedog/inworld

import { InworldRealtimeLLMService } from "@pipedog/inworld/realtime/llm";
import { SessionProperties } from "@pipedog/inworld/realtime/events";

@pipedog/inworld/realtime/events holds the session classes (SessionProperties, AudioConfiguration, AudioInput, AudioOutput, InputTranscription, TurnDetection, PCMAudioFormat, ...) and the client and server event models.

PrerequisitesCopy Anchor Link

• An Inworld account and an API key from the Inworld portal.
• Pass the key as apiKey (required). With the default authType: "basic" it is sent as Authorization: Basic <apiKey> (the Base64 key Inworld gives you). Use authType: "bearer" for a client-side JWT. The examples read it from INWORLD_API_KEY.

InworldRealtimeLLMServiceCopy Anchor Link

ConfigurationCopy Anchor Link

SettingsCopy Anchor Link

Runtime-updatable settings, passed as settings: new InworldRealtimeLLMService.Settings({...}) (InworldRealtimeLLMSettings). Any setting you leave out keeps the default below.

The default session has output modalities ["audio", "text"], PCM audio both ways (the rates follow the pipeline's), transcription with sttModel (default "inworld/inworld-stt-1"), semantic turn detection with low eagerness that creates and interrupts responses, and the voice (default "Clive") and ttsModel (default "inworld-tts-2") for output.

A sessionProperties you pass replaces those defaults as a whole, so give a complete config. Top-level model, systemInstruction and temperature win over the same fields inside it.

UsageCopy Anchor Link

import { LLMUpdateSettingsFrame } from "@pipedog/core/frames/frames";
import {
  AudioConfiguration,
  AudioInput,
  AudioOutput,
  type ConversationItem,
  InputTranscription,
  PCMAudioFormat,
  SessionProperties,
  TurnDetection,
} from "@pipedog/inworld/realtime/events";
import { InworldRealtimeLLMService } from "@pipedog/inworld/realtime/llm";

// The shorthand options fill in the default session.
const llm = new InworldRealtimeLLMService({
  apiKey: process.env.INWORLD_API_KEY ?? "",
  llmModel: "openai/gpt-4.1-mini",
  voice: "Sarah",
  ttsModel: "inworld-tts-2",
  settings: new InworldRealtimeLLMService.Settings({
    systemInstruction: "You are a helpful assistant in a voice conversation.",
  }),
});

llm.addEventHandler(
  "on_conversation_item_created",
  async (_service, itemId: string, _item: ConversationItem) => {
    console.log("conversation item created:", itemId);
  },
);

With full control over the session:

// A full SessionProperties replaces the defaults wholesale: give a complete config.
const custom = new InworldRealtimeLLMService({
  apiKey: process.env.INWORLD_API_KEY ?? "",
  settings: new InworldRealtimeLLMService.Settings({
    sessionProperties: new SessionProperties({
      model: "openai/gpt-4.1-mini",
      temperature: 0.7,
      outputModalities: ["audio", "text"],
      audio: new AudioConfiguration({
        input: new AudioInput({
          format: new PCMAudioFormat({ rate: 24000 }),
          transcription: new InputTranscription({ model: "inworld/inworld-stt-1" }),
          turnDetection: new TurnDetection({ type: "semantic_vad", eagerness: "low" }),
        }),
        output: new AudioOutput({
          format: new PCMAudioFormat({ rate: 24000 }),
          voice: "Sarah",
          model: "inworld-tts-2",
        }),
      }),
    }),
  }),
});

Change the instructions mid-call with an LLMUpdateSettingsFrame. A change to sessionProperties, systemInstruction, model or temperature is sent as a session.update; other fields are ignored with a warning:

// Sends a session.update to the open session.
const update = new LLMUpdateSettingsFrame({
  delta: new InworldRealtimeLLMService.Settings({
    systemInstruction: "Answer in Spanish from now on.",
    temperature: 0.3,
  }),
});

Event handlersCopy Anchor Link

NotesCopy Anchor Link

• Consumes InputAudioRawFrame (sent in about 60 ms chunks), LLMContextFrame and LLMSetToolsFrame. Emits TTSAudioRawFrame, TTSTextFrame and LLMTextFrame for the bot, and InterimTranscriptionFrame / TranscriptionFrame for the user.
• With Inworld's server-side turn detection (the default), the service broadcasts proposed user turn frames from the server's speech events and asks the user aggregator for ExternalUserTurnStrategies, so no local VAD is needed. LLMContextAggregatorPair detects that this is a realtime service.
• For locally driven turns, set turnDetection: null in AudioInput and add a VAD to the user aggregator; the service then commits the audio and asks for a response when the user stops speaking. Don't run a local VAD with server-side turn detection on: both would send turn frames.
• Tool results are sent as function_call_output items. Intermediate results (isFinal: false) are dropped with a warning: Inworld's model calls the tool again instead of relaying them. cancelOnInterruption: false isn't reliably supported by Inworld, and the service reports an error when it sees one. LLMMessagesAppendFrame isn't supported yet.
• setAudioInputPaused() pauses input at runtime, sendClientEvent() sends any Realtime client event, and resetConversation() reconnects with a fresh server-side conversation.

Runtime notesCopy Anchor Link

Generated from pipecat.runtimes and pipecat.compat in packages/inworld/package.json: ✅ means the compat smoke test passes on that runtime.

• The socket is opened with websocketConnect() from core, with the key in the Authorization header and a fresh session key in the URL. Bun, Node and Deno send the header from the global WebSocket; on Cloudflare Workers core opens the socket through fetch.
• A failed response whose status_details isn't a string is reported as JSON (upstream prints a Python dict).