# xAI Grok Realtime > Speech-to-speech with GrokRealtimeLLMService, over xAI's Grok Voice Agent WebSocket. Source: https://polvallverdu.github.io/pipedog/docs/services/s2s/grok/ ## Overview [#overview] * **`GrokRealtimeLLMService`**: one WebSocket session with xAI's Grok Voice Agent API, which hears the user's audio and answers with audio. It replaces the STT, LLM and TTS services in a pipeline, and supports function calling, xAI's built-in tools (web search, X search, file search, remote MCP), transcripts of both sides, DTMF on SIP sessions and fixed spoken lines. Pipecat's documentation for the same service. `packages/xai/src/realtime/llm.ts` and `events.ts`. The Grok Voice Agent API, voices and tools. ## Installation [#installation] ```bash bun add @pipedog/xai ``` ```ts import { GrokRealtimeLLMService } from "@pipedog/xai/realtime/llm"; import { SessionProperties } from "@pipedog/xai/realtime/events"; ``` `@pipedog/xai/realtime/events` holds the session classes (`SessionProperties`, `AudioConfiguration`, `AudioInput`, `AudioOutput`, `InputAudioTranscription`, `TurnDetection`, `Reasoning`, `SessionResumption`), the tools (`WebSearchTool`, `XSearchTool`, `FileSearchTool`, `FunctionTool`, `McpTool`) and the client and server event models. ## Prerequisites [#prerequisites] * An xAI account and an API key from the [xAI console](https://console.x.ai). * Pass the key as `apiKey` (required); it is sent as a bearer token. The examples read it from `XAI_API_KEY`. ## GrokRealtimeLLMService [#grokrealtimellmservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | xAI API key for authentication. | | `baseUrl` | `string \| undefined` | no | WebSocket base URL for the realtime API. Defaults to "wss://api.x.ai/v1/realtime". | | `settings` | `GrokRealtimeLLMSettings \| null \| undefined` | no | Runtime-updatable settings for this service. Set the voice and the rest of the session with `sessionProperties`, e.g. `new SessionProperties({ voice: "rex" })`. Built-in voice IDs are documented by xAI (https://docs.x.ai/docs/guides/voice/agent); a custom voice ID from the Custom Voices API may be used too. | | `startAudioPaused` | `boolean \| undefined` | no | Whether to start with audio input paused. Defaults to false. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `runInParallel` | `boolean \| undefined` | no | Whether to run function calls in parallel or sequentially. Defaults to true. | | `groupParallelTools` | `boolean \| undefined` | no | Whether to group parallel function calls so the LLM is triggered exactly once after all calls in the batch complete. When false, each function call result triggers the LLM independently as it arrives. Defaults to true. | | `functionCallTimeoutSecs` | `number \| null \| undefined` | no | Optional timeout in seconds for function calls. A call that runs past it is cancelled: its handler is cancelled (it gets a `CancelledError`), the call is settled as cancelled, and inference runs so the LLM can report that it didn't complete. | | `fileResolver` | `FileResolver \| null \| undefined` | no | Resolves file URLs in the context that this provider can't fetch itself (see `FileResolver`). Pass one configured with a `fileStorage` to resolve uploaded-file URLs (`new FileResolver({ fileStorage: runnerArgs.fileStorage })` when a runner hosts the bot), and share one instance across services that may consume the same files (e.g. services switched between mid-session) so downloads are reused. Defaults to a resolver that handles `data:` and `http(s)` URLs only. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new GrokRealtimeLLMService.Settings({...})` (`GrokRealtimeLLMSettings`). Any setting you leave out keeps the default below. | Setting | Type | Default | Description | | ----------------------------------------------------------------------------------------- | ---------------------------------- | ------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | | `model` | `string` | `"grok-voice-latest"` | Voice model. Fixed for the session: it goes in the connection URL. | | `systemInstruction` | `string \| null` | `null` | Instructions. Kept in sync with `sessionProperties.instructions`. | | `sessionProperties` | `SessionProperties` | `new SessionProperties()` | Session config: `voice` (default `"eve"`), `turnDetection` (default server VAD), audio, tools, reasoning, resumption, pronunciation `replace` map. | | `temperature`, `maxTokens`, `topP`, `topK`, `frequencyPenalty`, `presencePenalty`, `seed` | `number \| null` | `null` | Not used by the Voice Agent API. | | `filterIncompleteUserTurns` | `boolean` | `false` | Filter incomplete user turns. | | `userTurnCompletionConfig` | `UserTurnCompletionConfig \| null` | `null` | Turn completion config used with `filterIncompleteUserTurns`. | A `sessionProperties` in an update replaces the stored one as a whole. The top-level `systemInstruction` wins over `instructions` inside it. Audio formats default to PCM at the pipeline's rates. ### Usage [#usage] ```ts title="realtime.ts" import { LLMUpdateSettingsFrame } from "@pipedog/core/frames/frames"; import { AudioConfiguration, AudioInput, type ConversationItem, InputAudioTranscription, SessionProperties, TurnDetection, WebSearchTool, } from "@pipedog/xai/realtime/events"; import { GrokRealtimeLLMService } from "@pipedog/xai/realtime/llm"; const llm = new GrokRealtimeLLMService({ apiKey: process.env.XAI_API_KEY ?? "", settings: new GrokRealtimeLLMService.Settings({ systemInstruction: "You are a helpful assistant in a voice conversation.", sessionProperties: new SessionProperties({ voice: "rex", turnDetection: new TurnDetection({ type: "server_vad", silenceDurationMs: 500 }), audio: new AudioConfiguration({ input: new AudioInput({ transcription: new InputAudioTranscription({ model: "grok-transcribe" }), }), }), tools: [new WebSearchTool()], }), }), }); llm.addEventHandler( "on_conversation_item_created", async (_service, itemId: string, _item: ConversationItem) => { console.log("conversation item created:", itemId); }, ); ``` Speak a fixed line without involving the model: ```ts // Speak a fixed line without the model (no response.create follows). await llm.forceMessage("This call is being recorded."); ``` Change the instructions mid-call with an `LLMUpdateSettingsFrame`. A change to `systemInstruction` or `sessionProperties` is sent as a `session.update`; other fields are ignored with a warning: ```ts // Sends a session.update to the open session. const update = new LLMUpdateSettingsFrame({ delta: new GrokRealtimeLLMService.Settings({ systemInstruction: "Answer in Spanish from now on.", }), }); ``` ### Event handlers [#event-handlers] | Event | Description | | -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | | `on_conversation_item_created` | A conversation item was added. Called with the item id and the item. | | `on_conversation_item_updated` | A conversation item changed (a transcript arrived, or the response finished). Called with the item id and the item, or `null`. | | `on_conversation_item_deleted` | The server confirmed a deletion. Called with the item id. | | `on_conversation_item_truncated` | The server confirmed a truncation. Called with the item id, content index and audio end (ms). | | `on_idle_timeout` | `turnDetection.idleTimeoutMs` fired and the server is checking in. Called with the item id. | | `on_dtmf_received` | A DTMF digit arrived on a SIP session. Called with the `KeypadEntry`. | | `on_mcp_event` | An MCP tool discovery or call event. Called with the event type and the event. | ### Notes [#notes] * Consumes `InputAudioRawFrame` (sent once the session is configured), `LLMContextFrame`, `LLMMessagesAppendFrame` and `LLMSetToolsFrame`. Emits `TTSAudioRawFrame`, `TTSTextFrame` and `LLMTextFrame` for the bot, `InterimTranscriptionFrame` (with `grok-transcribe`) and `TranscriptionFrame` for the user, and `InputDTMFFrame` upstream for DTMF digits. * With server-side VAD (the default), the service broadcasts proposed user turn frames from the server's speech events and asks the user aggregator for `ExternalUserTurnStrategies`, so no local VAD is needed. On interruption it cancels the response and truncates the assistant audio to what was played. * For locally driven turns, set `turnDetection: null` in `SessionProperties` and add a VAD to the user aggregator; the service then commits the audio and asks for a response when the user stops speaking. * Intermediate tool results (`isFinal: false`) go in as conversation messages beside the open call; the final result answers the call. * `forceMessage()`, `deleteConversationItem()`, `setAudioInputPaused()`, `sendClientEvent()` and `resetConversation()` are available at runtime. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/xai/package.json`: ✅ means the compat smoke test passes on that runtime. * The socket is opened with `websocketConnect()` from core, with the key in the `Authorization` header. Bun, Node and Deno send it from the global `WebSocket`; on Cloudflare Workers core opens the socket through `fetch`. * The deprecated `sessionProperties` constructor option and the `pipecat.services.grok.realtime` aliases are not ported: import from `@pipedog/xai/realtime/*` and pass `sessionProperties` in `settings`. * A failed response whose `status_details` isn't a string is reported as JSON (upstream prints a Python dict).