# Gradium

> Realtime speech-to-text with GradiumSTTService, over Gradium's WebSocket API, with optional server-side turn detection.

Source: https://polvallverdu.github.io/pipedog/docs/services/stt/gradium/



## Overview [#overview]

* **`GradiumSTTService`**: streams audio over a WebSocket to Gradium and pushes interim and final transcriptions. By default the pipeline's VAD closes each utterance by flushing the server. With `enableTurnDetection`, Gradium's end-pointing signal decides when turns start and end instead.

<Cards>
  <Card title="Upstream page" href="https://docs.pipecat.ai/api-reference/server/services/stt/gradium">
    Pipecat's documentation for the same service.
  </Card>

  <Card title="Source" href="https://github.com/polvallverdu/pipedog/tree/main/packages/gradium/src">
    `packages/gradium/src/stt.ts`.
  </Card>

  <Card title="Gradium docs" href="https://docs.gradium.ai/api-reference/endpoint/stt-websocket">
    The speech-to-text WebSocket reference.
  </Card>
</Cards>

## Installation [#installation]

```bash
bun add @pipedog/gradium
```

```ts
import { GradiumSTTService } from "@pipedog/gradium/stt";
```

## Prerequisites [#prerequisites]

* A Gradium account and an API key from [Gradium](https://gradium.ai/).
* Pass the key as `apiKey`. The examples read it from `GRADIUM_API_KEY`.

## GradiumSTTService [#gradiumsttservice]

Connects to `wss://api.gradium.ai/api/speech/asr`, with the key in the `x-api-key` header.

### Configuration [#configuration]

| Option | Type | Required | Description |
|---|---|---|---|
| `apiKey` | `string` | yes | Gradium API key for authentication. |
| `apiEndpointBaseUrl` | `string \| undefined` | no | WebSocket endpoint URL. Defaults to "wss://api.gradium.ai/api/speech/asr". |
| `encoding` | `string \| undefined` | no | Base audio encoding type. One of "pcm", "wav", or "opus". For PCM, the sample rate is appended automatically from the pipeline's audio_in_sample_rate (e.g., "pcm" becomes "pcm_16000"). Audio at a rate Gradium does not accept is resampled to one it does. Defaults to "pcm". |
| `enableTurnDetection` | `boolean \| undefined` | no | Whether the server's end-pointing signal decides when user turns start and end, instead of the pipeline's VAD. Off by default. |
| `settings` | `GradiumSTTSettings \| null \| undefined` | no | Runtime-updatable settings. |
| `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. |
| `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. |
| `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. |
| `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. |
| `reconnectBackoffMinWait` | `number \| undefined` | no | Minimum time, in seconds, to wait between reconnection attempts. |
| `reconnectBackoffMaxWait` | `number \| undefined` | no | Maximum time, in seconds, to wait between reconnection attempts. |
| `wsCloseTimeout` | `number \| undefined` | no | Maximum time, in seconds, to wait for the peer to acknowledge the websocket closing handshake before dropping the connection. Applied to connections opened through `_websocketConnect()`. Increase it for peers that need longer to complete a graceful close. |
| `wsSendTimeout` | `number \| null \| undefined` | no | Maximum time, in seconds, a single send may wait on a peer that has stopped reading before the connection is dropped and the send throws `ConnectionClosedError`. Applied to connections opened through `_websocketConnect()`. Pass `null` to wait indefinitely. Defaults to `WS_SEND_TIMEOUT`. |
| `audioPassthrough` | `boolean \| undefined` | no | Whether to pass audio frames downstream after processing. Defaults to true. |
| `sampleRate` | `number \| null \| undefined` | no | The sample rate for audio input. If null, will be determined from the start frame. |
| `sttTtfbTimeout` | `number \| undefined` | no | Time in seconds to wait after VAD stop before reporting TTFB. This delay allows the final transcription to arrive. Defaults to 2.0. |
| `ttfsP99Latency` | `number \| null \| undefined` | no | P99 latency from speech end to final transcript in seconds. Broadcast via STTMetadataFrame at pipeline start. |
| `keepaliveTimeout` | `number \| null \| undefined` | no | Seconds of no audio before sending silence to keep the connection alive. null disables keepalive. |
| `keepaliveInterval` | `number \| undefined` | no | Seconds between idle checks when keepalive is enabled. |
| `reconnectOnError` | `boolean \| undefined` | no | Whether to automatically reconnect on websocket errors. Defaults to true. |

### Settings [#settings]

Runtime-updatable settings, passed as `settings: new GradiumSTTService.Settings({...})`
(`GradiumSTTSettings`). The `eot*` and `postFlushCooldownFrames` fields tune turn detection and
are `null` unless `enableTurnDetection` is `true`, which gives them the defaults below.

| Setting                   | Type                | Default       | Description                                                                                                     |
| ------------------------- | ------------------- | ------------- | --------------------------------------------------------------------------------------------------------------- |
| `language`                | `Language \| "any"` | `Language.EN` | English, German, Spanish, French or Portuguese; `"any"` has Gradium detect it.                                  |
| `delayInFrames`           | `number \| null`    | `12` (960 ms) | Delay, in 80 ms frames, before text is generated. One of 7, 8, 10, 12, 14, 16, 20, 24, 36, 48. Lower is faster. |
| `eotHorizonS`             | `number \| null`    | `3.0`         | The end-pointing horizon that drives turn decisions.                                                            |
| `eotThreshold`            | `number \| null`    | `0.5`         | Inactivity probability at or above which an open turn ends.                                                     |
| `postFlushCooldownFrames` | `number \| null`    | `8`           | Step messages to ignore after a flush before the next turn can start or end.                                    |

### Usage [#usage]

```ts title="stt.ts"
import { Language } from "@pipedog/core/transcriptions/language";
import { GradiumSTTService } from "@pipedog/gradium/stt";

const stt = new GradiumSTTService({
  apiKey: process.env.GRADIUM_API_KEY ?? "",
  settings: new GradiumSTTService.Settings({ language: Language.EN }),
});
```

With the server's turn detection:

```ts
const withTurns = new GradiumSTTService({
  apiKey: process.env.GRADIUM_API_KEY ?? "",
  enableTurnDetection: true,
  settings: new GradiumSTTService.Settings({ eotHorizonS: 3.0, eotThreshold: 0.5 }),
});
```

### Event handlers [#event-handlers]

| Event             | Description                                                  |
| ----------------- | ------------------------------------------------------------ |
| `on_connected`    | The WebSocket connected.                                     |
| `on_disconnected` | The WebSocket closed.                                        |
| `on_turn_start`   | The end-pointing signal opened a turn (turn detection on).   |
| `on_turn_end`     | The end-pointing signal closed the turn (turn detection on). |

### Notes [#notes]

* `encoding` is `"pcm"` (the sample rate is appended, for example `pcm_16000`), `"wav"` or `"opus"`. Audio at a rate Gradium doesn't accept is resampled.
* With `enableTurnDetection`, local VAD frames are ignored and the service recommends `ExternalUserTurnStrategies`, which resolve its `ProposedUserStartedSpeakingFrame` and `ProposedUserStoppedSpeakingFrame` into user turns.

| Runtime            | Works | What it needs |
| ------------------ | ----- | ------------- |
| Bun                | ✅ yes | Nothing.      |
| Node.js            | ✅ yes | Nothing.      |
| Deno               | ✅ yes | Nothing.      |
| Cloudflare Workers | ✅ yes | Nothing.      |

Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/gradium/package.json`: ✅ means the compat smoke test passes on that runtime.

## Runtime notes [#runtime-notes]

* The WebSocket goes through the core `websocketConnect`, with headers on the handshake.
* The deprecated `InputParams`, `params` and `json_config` options are not ported.
* Tested against local WebSocket servers, as upstream's tests do, not against Gradium itself.