# Mistral

> Realtime speech-to-text with MistralSTTService, Mistral's Voxtral over WebSocket.

Source: https://polvallverdu.github.io/pipedog/docs/services/stt/mistral/



## Overview [#overview]

* **`MistralSTTService`**: streams audio to Mistral's Voxtral realtime transcription over a WebSocket, pushes the text deltas as interim transcriptions, and the final transcript when the user stops speaking.

<Cards>
  <Card title="Upstream page" href="https://docs.pipecat.ai/api-reference/server/services/stt/mistral">
    Pipecat's documentation for the same service.
  </Card>

  <Card title="Source" href="https://github.com/polvallverdu/pipedog/tree/main/packages/mistral/src">
    `packages/mistral/src/stt.ts`.
  </Card>

  <Card title="Mistral docs" href="https://docs.mistral.ai/">
    Voxtral and the audio APIs.
  </Card>
</Cards>

## Installation [#installation]

```bash
bun add @pipedog/mistral
```

```ts
import { MistralSTTService } from "@pipedog/mistral/stt";
```

## Prerequisites [#prerequisites]

* A Mistral account and an API key from [Mistral](https://docs.mistral.ai/api/).
* Pass the key as `apiKey`. The examples read it from `MISTRAL_API_KEY`.
* A VAD in the pipeline: the end of the user's speech flushes the audio buffer.

## MistralSTTService [#mistralsttservice]

Connects to `wss://api.mistral.ai/v1/audio/transcriptions/realtime`.

### Configuration [#configuration]

| Option | Type | Required | Description |
|---|---|---|---|
| `apiKey` | `string \| null \| undefined` | no | Mistral API key for authentication. |
| `baseUrl` | `string \| null \| undefined` | no | Custom API endpoint URL. Defaults to `"https://api.mistral.ai"`. |
| `targetStreamingDelayMs` | `number \| null \| undefined` | no | Streaming delay for accuracy/latency tradeoff. Higher values may improve accuracy at the cost of latency. |
| `settings` | `MistralSTTSettings \| null \| undefined` | no | Runtime-updatable settings. |
| `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. |
| `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. |
| `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. |
| `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. |
| `audioPassthrough` | `boolean \| undefined` | no | Whether to pass audio frames downstream after processing. Defaults to true. |
| `sampleRate` | `number \| null \| undefined` | no | The sample rate for audio input. If null, will be determined from the start frame. |
| `sttTtfbTimeout` | `number \| undefined` | no | Time in seconds to wait after VAD stop before reporting TTFB. This delay allows the final transcription to arrive. Defaults to 2.0. |
| `ttfsP99Latency` | `number \| null \| undefined` | no | P99 latency from speech end to final transcript in seconds. Broadcast via STTMetadataFrame at pipeline start. |
| `keepaliveTimeout` | `number \| null \| undefined` | no | Seconds of no audio before sending silence to keep the connection alive. null disables keepalive. |
| `keepaliveInterval` | `number \| undefined` | no | Seconds between idle checks when keepalive is enabled. |

### Settings [#settings]

Runtime-updatable settings, passed as `settings: new MistralSTTService.Settings({...})`
(`MistralSTTSettings`). Any setting you leave out keeps the default below.

| Setting    | Type               | Default                                   | Description                                                                        |
| ---------- | ------------------ | ----------------------------------------- | ---------------------------------------------------------------------------------- |
| `model`    | `string`           | `"voxtral-mini-transcribe-realtime-2602"` | Realtime transcription model.                                                      |
| `language` | `Language \| null` | `null`                                    | Not sent: Voxtral detects the language, which is reported on the final transcript. |

### Usage [#usage]

```ts title="stt.ts"
import { STTUpdateSettingsFrame } from "@pipedog/core/frames/frames";
import { MistralSTTService } from "@pipedog/mistral/stt";

const stt = new MistralSTTService({
  apiKey: process.env.MISTRAL_API_KEY ?? "",
});
```

Change settings mid-call with an `STTUpdateSettingsFrame`:

```ts
// Queue it on the worker, or push it from a processor upstream of the service.
const update = new STTUpdateSettingsFrame({
  delta: new MistralSTTService.Settings({ model: "voxtral-mini-transcribe-realtime-2602" }),
});
```

### Event handlers [#event-handlers]

| Event                 | Description              |
| --------------------- | ------------------------ |
| `on_connected`        | The session was created. |
| `on_disconnected`     | The WebSocket closed.    |
| `on_connection_error` | The connection failed.   |

### Notes [#notes]

* `targetStreamingDelayMs` trades latency for accuracy, when set. It is sent in the
  `session.update` message (`target_streaming_delay_ms`); Mistral accepts 80 to 1200, or 2400,
  and recommends 480.
* The service waits for `session.created` before it sends the session settings, as Mistral's
  SDK does.
* Any settings change reconnects.

## Runtime notes [#runtime-notes]

| Runtime            | Works | What it needs |
| ------------------ | ----- | ------------- |
| Bun                | ✅ yes | Nothing.      |
| Node.js            | ✅ yes | Nothing.      |
| Deno               | ✅ yes | Nothing.      |
| Cloudflare Workers | ✅ yes | Nothing.      |

Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/mistral/package.json`: ✅ means the compat smoke test passes on that runtime.

* The WebSocket goes through the core `websocketConnect` (upstream uses the `mistralai` SDK's realtime connection). It works on Bun, Node, Deno and Cloudflare Workers.
* The protocol follows Mistral's SDK (`mistralai` 3.3.0 and `@mistralai/mistralai` 3.1.0,
  `extra/realtime`) and the [realtime transcription docs](https://docs.mistral.ai/studio/audio/speech-to-text/transcribe-in-realtime).
  `input_audio.flush` and `transcription.language` are in the SDK but not in the docs.
* Tested against a local fake that follows that protocol, not against Mistral itself.