# xAI

> Realtime speech-to-text with XAISTTService, over xAI's WebSocket API.

Source: https://polvallverdu.github.io/pipedog/docs/services/stt/xai/



## Overview [#overview]

* **`XAISTTService`**: streams audio over a persistent WebSocket to xAI's speech-to-text endpoint. Interim results and chunk finals become `InterimTranscriptionFrame`s; the utterance final becomes a `TranscriptionFrame`.

<Cards>
  <Card title="Upstream page" href="https://docs.pipecat.ai/api-reference/server/services/stt/xai">
    Pipecat's documentation for the same service.
  </Card>

  <Card title="Source" href="https://github.com/polvallverdu/pipedog/tree/main/packages/xai/src">
    `packages/xai/src/stt.ts`.
  </Card>

  <Card title="xAI docs" href="https://docs.x.ai/developers/rest-api-reference/inference/voice">
    The voice API reference.
  </Card>
</Cards>

## Installation [#installation]

```bash
bun add @pipedog/xai
```

```ts
import { XAISTTService } from "@pipedog/xai/stt";
```

## Prerequisites [#prerequisites]

* An xAI account and an API key from the [xAI console](https://console.x.ai/).
* Pass the key as `apiKey`. The examples read it from `XAI_API_KEY`.

## XAISTTService [#xaisttservice]

Connects to `wss://api.x.ai/v1/stt`, with the key as a Bearer token on the handshake.

### Configuration [#configuration]

| Option | Type | Required | Description |
|---|---|---|---|
| `apiKey` | `string` | yes | xAI API key (used as Bearer for the WebSocket handshake). |
| `wsUrl` | `string \| undefined` | no | WebSocket endpoint URL. Defaults to "wss://api.x.ai/v1/stt". |
| `encoding` | `string \| undefined` | no | Audio encoding. One of "pcm" (signed 16-bit LE), "mulaw", or "alaw". Defaults to "pcm". |
| `settings` | `XAISTTSettings \| null \| undefined` | no | Runtime-updatable settings overriding defaults. |
| `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. |
| `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. |
| `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. |
| `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. |
| `reconnectBackoffMinWait` | `number \| undefined` | no | Minimum time, in seconds, to wait between reconnection attempts. |
| `reconnectBackoffMaxWait` | `number \| undefined` | no | Maximum time, in seconds, to wait between reconnection attempts. |
| `wsCloseTimeout` | `number \| undefined` | no | Maximum time, in seconds, to wait for the peer to acknowledge the websocket closing handshake before dropping the connection. Applied to connections opened through `_websocketConnect()`. Increase it for peers that need longer to complete a graceful close. |
| `wsSendTimeout` | `number \| null \| undefined` | no | Maximum time, in seconds, a single send may wait on a peer that has stopped reading before the connection is dropped and the send throws `ConnectionClosedError`. Applied to connections opened through `_websocketConnect()`. Pass `null` to wait indefinitely. Defaults to `WS_SEND_TIMEOUT`. |
| `audioPassthrough` | `boolean \| undefined` | no | Whether to pass audio frames downstream after processing. Defaults to true. |
| `sampleRate` | `number \| null \| undefined` | no | The sample rate for audio input. If null, will be determined from the start frame. |
| `sttTtfbTimeout` | `number \| undefined` | no | Time in seconds to wait after VAD stop before reporting TTFB. This delay allows the final transcription to arrive. Defaults to 2.0. |
| `ttfsP99Latency` | `number \| null \| undefined` | no | P99 latency from speech end to final transcript in seconds. Broadcast via STTMetadataFrame at pipeline start. |
| `keepaliveTimeout` | `number \| null \| undefined` | no | Seconds of no audio before sending silence to keep the connection alive. null disables keepalive. |
| `keepaliveInterval` | `number \| undefined` | no | Seconds between idle checks when keepalive is enabled. |
| `reconnectOnError` | `boolean \| undefined` | no | Whether to automatically reconnect on websocket errors. Defaults to true. |

### Settings [#settings]

Runtime-updatable settings, passed as `settings: new XAISTTService.Settings({...})`
(`XAISTTSettings`). Any setting you leave out keeps the default below.

| Setting          | Type               | Default                       | Description                                                                    |
| ---------------- | ------------------ | ----------------------------- | ------------------------------------------------------------------------------ |
| `model`          | `string`           | `"grok-voice-transcribe-2.0"` | Transcription model.                                                           |
| `language`       | `Language \| null` | `Language.EN`                 | Language of the audio, sent as a two-letter code.                              |
| `interimResults` | `boolean`          | `true`                        | Partial transcripts about every 500 ms.                                        |
| `endpointing`    | `number \| null`   | `null`                        | Silence, in ms (0 to 5000), that ends an utterance. The server default is 400. |
| `multichannel`   | `boolean \| null`  | `null`                        | Transcribe each interleaved channel on its own. Needs `channels` of 2 or more. |
| `channels`       | `number \| null`   | `null`                        | Number of interleaved channels (2 to 8).                                       |
| `diarize`        | `boolean \| null`  | `null`                        | Attach a `speaker` to each word.                                               |

### Usage [#usage]

```ts title="stt.ts"
import { STTUpdateSettingsFrame } from "@pipedog/core/frames/frames";
import { XAISTTService } from "@pipedog/xai/stt";

const stt = new XAISTTService({
  apiKey: process.env.XAI_API_KEY ?? "",
  settings: new XAISTTService.Settings({ endpointing: 300 }),
});
```

Change settings mid-call with an `STTUpdateSettingsFrame`:

```ts
// Queue it on the worker, or push it from a processor upstream of the service.
const update = new STTUpdateSettingsFrame({
  delta: new XAISTTService.Settings({ endpointing: 500 }),
});
```

### Event handlers [#event-handlers]

| Event             | Description              |
| ----------------- | ------------------------ |
| `on_connected`    | The WebSocket connected. |
| `on_disconnected` | The WebSocket closed.    |

### Notes [#notes]

* `encoding` is `"pcm"` (16-bit little-endian) by default; `"mulaw"` and `"alaw"` also work. `sampleRate` defaults to the pipeline's input rate.
* Settings are query parameters of the connection, so any settings change reconnects.
* Audio is only sent once the server reports the session ready (`transcript.created`).

## Runtime notes [#runtime-notes]

| Runtime            | Works | What it needs |
| ------------------ | ----- | ------------- |
| Bun                | ✅ yes | Nothing.      |
| Node.js            | ✅ yes | Nothing.      |
| Deno               | ✅ yes | Nothing.      |
| Cloudflare Workers | ✅ yes | Nothing.      |

Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/xai/package.json`: ✅ means the compat smoke test passes on that runtime.

* The WebSocket goes through the core `websocketConnect`, with headers on the handshake. It works on Bun, Node, Deno and Cloudflare Workers.
* Tested against a local WebSocket server, as upstream's tests do, not against xAI itself.