xAI Grok Realtime

OverviewCopy Anchor Link

• GrokRealtimeLLMService: one WebSocket session with xAI's Grok Voice Agent API, which hears the user's audio and answers with audio. It replaces the STT, LLM and TTS services in a pipeline, and supports function calling, xAI's built-in tools (web search, X search, file search, remote MCP), transcripts of both sides, DTMF on SIP sessions and fixed spoken lines.

InstallationCopy Anchor Link

bun add @pipedog/xai

import { GrokRealtimeLLMService } from "@pipedog/xai/realtime/llm";
import { SessionProperties } from "@pipedog/xai/realtime/events";

@pipedog/xai/realtime/events holds the session classes (SessionProperties, AudioConfiguration, AudioInput, AudioOutput, InputAudioTranscription, TurnDetection, Reasoning, SessionResumption), the tools (WebSearchTool, XSearchTool, FileSearchTool, FunctionTool, McpTool) and the client and server event models.

PrerequisitesCopy Anchor Link

• An xAI account and an API key from the xAI console.
• Pass the key as apiKey (required); it is sent as a bearer token. The examples read it from XAI_API_KEY.

GrokRealtimeLLMServiceCopy Anchor Link

ConfigurationCopy Anchor Link

SettingsCopy Anchor Link

Runtime-updatable settings, passed as settings: new GrokRealtimeLLMService.Settings({...}) (GrokRealtimeLLMSettings). Any setting you leave out keeps the default below.

A sessionProperties in an update replaces the stored one as a whole. The top-level systemInstruction wins over instructions inside it. Audio formats default to PCM at the pipeline's rates.

UsageCopy Anchor Link

import { LLMUpdateSettingsFrame } from "@pipedog/core/frames/frames";
import {
  AudioConfiguration,
  AudioInput,
  type ConversationItem,
  InputAudioTranscription,
  SessionProperties,
  TurnDetection,
  WebSearchTool,
} from "@pipedog/xai/realtime/events";
import { GrokRealtimeLLMService } from "@pipedog/xai/realtime/llm";

const llm = new GrokRealtimeLLMService({
  apiKey: process.env.XAI_API_KEY ?? "",
  settings: new GrokRealtimeLLMService.Settings({
    systemInstruction: "You are a helpful assistant in a voice conversation.",
    sessionProperties: new SessionProperties({
      voice: "rex",
      turnDetection: new TurnDetection({ type: "server_vad", silenceDurationMs: 500 }),
      audio: new AudioConfiguration({
        input: new AudioInput({
          transcription: new InputAudioTranscription({ model: "grok-transcribe" }),
        }),
      }),
      tools: [new WebSearchTool()],
    }),
  }),
});

llm.addEventHandler(
  "on_conversation_item_created",
  async (_service, itemId: string, _item: ConversationItem) => {
    console.log("conversation item created:", itemId);
  },
);

Speak a fixed line without involving the model:

// Speak a fixed line without the model (no response.create follows).
await llm.forceMessage("This call is being recorded.");

Change the instructions mid-call with an LLMUpdateSettingsFrame. A change to systemInstruction or sessionProperties is sent as a session.update; other fields are ignored with a warning:

// Sends a session.update to the open session.
const update = new LLMUpdateSettingsFrame({
  delta: new GrokRealtimeLLMService.Settings({
    systemInstruction: "Answer in Spanish from now on.",
  }),
});

Event handlersCopy Anchor Link

NotesCopy Anchor Link

• Consumes InputAudioRawFrame (sent once the session is configured), LLMContextFrame, LLMMessagesAppendFrame and LLMSetToolsFrame. Emits TTSAudioRawFrame, TTSTextFrame and LLMTextFrame for the bot, InterimTranscriptionFrame (with grok-transcribe) and TranscriptionFrame for the user, and InputDTMFFrame upstream for DTMF digits.
• With server-side VAD (the default), the service broadcasts proposed user turn frames from the server's speech events and asks the user aggregator for ExternalUserTurnStrategies, so no local VAD is needed. On interruption it cancels the response and truncates the assistant audio to what was played.
• For locally driven turns, set turnDetection: null in SessionProperties and add a VAD to the user aggregator; the service then commits the audio and asks for a response when the user stops speaking.
• Intermediate tool results (isFinal: false) go in as conversation messages beside the open call; the final result answers the call.
• forceMessage(), deleteConversationItem(), setAudioInputPaused(), sendClientEvent() and resetConversation() are available at runtime.

Runtime notesCopy Anchor Link

Generated from pipecat.runtimes and pipecat.compat in packages/xai/package.json: ✅ means the compat smoke test passes on that runtime.

• The socket is opened with websocketConnect() from core, with the key in the Authorization header. Bun, Node and Deno send it from the global WebSocket; on Cloudflare Workers core opens the socket through fetch.
• The deprecated sessionProperties constructor option and the pipecat.services.grok.realtime aliases are not ported: import from @pipedog/xai/realtime/* and pass sessionProperties in settings.
• A failed response whose status_details isn't a string is reported as JSON (upstream prints a Python dict).