# Deepgram

> Streaming speech-to-text with DeepgramSTTService (Nova) and DeepgramFluxSTTService (Flux, with model-side turn detection).

Source: https://polvallverdu.github.io/pipedog/docs/services/stt/deepgram/



## Overview [#overview]

Deepgram streams transcripts over a WebSocket. pipedog ports both of Pipecat's Deepgram
speech-to-text services:

* **`DeepgramSTTService`**: Deepgram's `/v1/listen` API (Nova models), with interim results,
  smart formatting, keyterms and the other Deepgram query options.
* **`DeepgramFluxSTTService`**: Deepgram Flux (`/v2/listen`), which also detects the end of the
  user's turn, with optional eager end-of-turn events.

<Cards>
  <Card title="Upstream page" href="https://docs.pipecat.ai/api-reference/server/services/stt/deepgram">
    Pipecat's documentation for the same services.
  </Card>

  <Card title="Source" href="https://github.com/polvallverdu/pipedog/tree/main/packages/deepgram/src">
    `packages/deepgram/src/stt.ts` and `flux/stt.ts`.
  </Card>

  <Card title="Deepgram docs" href="https://developers.deepgram.com/docs">
    Models, languages and query parameters.
  </Card>
</Cards>

## Installation [#installation]

```bash
bun add @pipedog/deepgram
```

```ts
import { DeepgramSTTService } from "@pipedog/deepgram/stt";
import { DeepgramFluxSTTService } from "@pipedog/deepgram/flux/stt";
```

## Prerequisites [#prerequisites]

* A Deepgram account and an API key from the [Deepgram console](https://console.deepgram.com).
* Pass the key as `apiKey`. The examples read it from `DEEPGRAM_API_KEY`.

## DeepgramSTTService [#deepgramsttservice]

### Configuration [#configuration]

| Option | Type | Required | Description |
|---|---|---|---|
| `apiKey` | `string` | yes | Deepgram API key for authentication. |
| `baseUrl` | `string \| undefined` | no | Custom Deepgram API base URL. |
| `encoding` | `string \| undefined` | no | Audio encoding format. Defaults to "linear16". |
| `channels` | `number \| undefined` | no | Number of audio channels. Defaults to 1. |
| `multichannel` | `boolean \| undefined` | no | Transcribe each audio channel independently. Defaults to false. |
| `callback` | `string \| null \| undefined` | no | Callback URL for async transcription delivery. |
| `callbackMethod` | `string \| null \| undefined` | no | HTTP method for the callback ("GET" or "POST"). |
| `tag` | `any` | no | Custom billing tag. |
| `mipOptOut` | `boolean \| null \| undefined` | no | Opt out of Deepgram model improvement program. |
| `addons` | `Record<string, unknown> \| null \| undefined` | no | Additional Deepgram features to enable. |
| `settings` | `DeepgramSTTSettings \| null \| undefined` | no | Runtime-updatable settings. |
| `webSocketFactory` | `((uri: string, options: { headers?: Record<string, string>; protocols?: string[]; }) => WebSocket \| Promise<WebSocket>) \| null \| undefined` | no | Creates the underlying `WebSocket`, to open it some other way (through a proxy, say). Defaults to `websocketConnect`'s, which sends the header on every runtime, Cloudflare Workers included. Not in upstream. |
| `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. |
| `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. |
| `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. |
| `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. |
| `audioPassthrough` | `boolean \| undefined` | no | Whether to pass audio frames downstream after processing. Defaults to true. |
| `sampleRate` | `number \| null \| undefined` | no | The sample rate for audio input. If null, will be determined from the start frame. |
| `sttTtfbTimeout` | `number \| undefined` | no | Time in seconds to wait after VAD stop before reporting TTFB. This delay allows the final transcription to arrive. Defaults to 2.0. |
| `ttfsP99Latency` | `number \| null \| undefined` | no | P99 latency from speech end to final transcript in seconds. Broadcast via STTMetadataFrame at pipeline start. |
| `keepaliveTimeout` | `number \| null \| undefined` | no | Seconds of no audio before sending silence to keep the connection alive. null disables keepalive. |
| `keepaliveInterval` | `number \| undefined` | no | Seconds between idle checks when keepalive is enabled. |

### Settings [#settings]

Runtime-updatable settings, passed as `settings: new DeepgramSTTService.Settings({...})`. Any
setting you leave out keeps the default below. Other Deepgram query parameters can go in `extra`.

| Setting           | Type                         | Default            | Description                                           |
| ----------------- | ---------------------------- | ------------------ | ----------------------------------------------------- |
| `model`           | `string`                     | `"nova-3-general"` | Deepgram model.                                       |
| `language`        | `Language`                   | `Language.EN`      | Transcription language.                               |
| `interimResults`  | `boolean`                    | `true`             | Emit `InterimTranscriptionFrame`s.                    |
| `punctuate`       | `boolean`                    | `true`             | Add punctuation.                                      |
| `smartFormat`     | `boolean`                    | `false`            | Smart formatting.                                     |
| `endpointing`     | `number \| false \| null`    | `null`             | Endpointing sensitivity in ms, or `false` to disable. |
| `utteranceEndMs`  | `number \| null`             | `null`             | Silence in ms before an utterance-end event.          |
| `keyterm`         | `string \| string[] \| null` | `null`             | Keyterms to boost.                                    |
| `keywords`        | `string \| string[] \| null` | `null`             | Keywords to boost.                                    |
| `diarize`         | `boolean`                    | `false`            | Speaker diarization.                                  |
| `detectEntities`  | `boolean`                    | `false`            | Named entity detection.                               |
| `dictation`       | `boolean`                    | `false`            | Dictation mode.                                       |
| `numerals`        | `boolean`                    | `false`            | Spoken numbers as numerals.                           |
| `profanityFilter` | `boolean \| null`            | `null`             | Filter profanity.                                     |
| `redact`          | `string \| string[] \| null` | `null`             | Redaction types.                                      |
| `replace`         | `string \| string[] \| null` | `null`             | Word replacement rules.                               |
| `search`          | `string \| string[] \| null` | `null`             | Search terms to highlight.                            |
| `version`         | `string \| null`             | `null`             | Model version; `null` leaves Deepgram on `"latest"`.  |

### Usage [#usage]

```ts title="stt.ts"
import { STTUpdateSettingsFrame } from "@pipedog/core/frames/frames";
import { Language } from "@pipedog/core/transcriptions/language";
import { DeepgramSTTService } from "@pipedog/deepgram/stt";

const stt = new DeepgramSTTService({
  apiKey: process.env.DEEPGRAM_API_KEY ?? "",
  settings: new DeepgramSTTService.Settings({
    model: "nova-3-general",
    language: Language.EN,
    smartFormat: true,
    keyterm: ["pipedog", "Pipecat"],
  }),
});
```

Change settings mid-call with an `STTUpdateSettingsFrame`. The service reconnects with the new
settings:

```ts
// Queue it on the worker, or push it from a processor upstream of the service.
const update = new STTUpdateSettingsFrame({
  delta: new DeepgramSTTService.Settings({ language: Language.ES }),
});
```

### Notes [#notes]

* Emits `InterimTranscriptionFrame` while the user speaks and `TranscriptionFrame` for final
  results. A `VADUserStoppedSpeakingFrame` from the user aggregator's VAD makes it finalize early.
* `sampleRate` defaults to the pipeline's input rate.
* If the connection keeps failing quickly (an invalid key, say), the service stops retrying and
  pushes an error.

## DeepgramFluxSTTService [#deepgramfluxsttservice]

### Configuration [#configuration-1]

| Option | Type | Required | Description |
|---|---|---|---|
| `apiKey` | `string` | yes | Deepgram API key for authentication. Required for API access. |
| `url` | `string \| undefined` | no | WebSocket URL for the Deepgram Flux API. Defaults to Deepgram's Flux endpoint, "wss://api.deepgram.com/v2/listen". |
| `fluxEncoding` | `string \| undefined` | no | Audio encoding format required by Flux API. Must be "linear16". Raw signed little-endian 16-bit PCM encoding. |
| `settings` | `DeepgramFluxSTTSettings \| null \| undefined` | no | Runtime-updatable settings. |
| `webSocketFactory` | `((uri: string, options: { headers?: Record<string, string>; protocols?: string[]; }) => WebSocket \| Promise<WebSocket>) \| null \| undefined` | no | Creates the underlying `WebSocket`, to open it some other way (through a proxy, say). Defaults to `websocketConnect`'s, which sends the header on every runtime, Cloudflare Workers included. Not in upstream. |
| `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. |
| `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. |
| `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. |
| `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. |
| `reconnectBackoffMinWait` | `number \| undefined` | no | Minimum time, in seconds, to wait between reconnection attempts. |
| `reconnectBackoffMaxWait` | `number \| undefined` | no | Maximum time, in seconds, to wait between reconnection attempts. |
| `reconnectOnError` | `boolean \| undefined` | no | Whether to automatically reconnect on connection errors. |
| `wsCloseTimeout` | `number \| undefined` | no | Maximum time, in seconds, to wait for the peer to acknowledge the websocket closing handshake before dropping the connection. Applied to connections opened through `_websocketConnect()`. Increase it for peers that need longer to complete a graceful close. |
| `wsSendTimeout` | `number \| null \| undefined` | no | Maximum time, in seconds, a single send may wait on a peer that has stopped reading before the connection is dropped and the send throws `ConnectionClosedError`. Applied to connections opened through `_websocketConnect()`. Pass `null` to wait indefinitely. Defaults to `WS_SEND_TIMEOUT`. |
| `audioPassthrough` | `boolean \| undefined` | no | Whether to pass audio frames downstream after processing. Defaults to true. |
| `sampleRate` | `number \| null \| undefined` | no | The sample rate for audio input. If null, will be determined from the start frame. |
| `sttTtfbTimeout` | `number \| undefined` | no | Time in seconds to wait after VAD stop before reporting TTFB. This delay allows the final transcription to arrive. Defaults to 2.0. |
| `ttfsP99Latency` | `number \| null \| undefined` | no | P99 latency from speech end to final transcript in seconds. Broadcast via STTMetadataFrame at pipeline start. |
| `keepaliveTimeout` | `number \| null \| undefined` | no | Seconds of no audio before sending silence to keep the connection alive. null disables keepalive. |
| `keepaliveInterval` | `number \| undefined` | no | Seconds between idle checks when keepalive is enabled. |
| `enableEagerEndOfTurn` | `boolean \| undefined` | no | Whether to answer a predicted end of turn ahead of the committed one. Off by default: it spends an inference on every prediction, including the ones the service withdraws. |
| `eagerMatchPolicy` | `EagerMatchPolicy \| null \| undefined` | no | Decides whether the committed transcript is close enough to the predicted one to keep the speculative response. Only consulted while eager end of turn is on. See EagerUserTurnStrategies. |
| `mipOptOut` | `boolean \| null \| undefined` | no | Opt out of the Deepgram Model Improvement Program. |
| `tag` | `string[] \| null \| undefined` | no | Tags to label requests for identification during usage reporting. |
| `shouldInterrupt` | `boolean \| undefined` | no | Whether to interrupt the bot when Flux detects that the user is speaking. |
| `watchdogMinTimeout` | `number \| undefined` | no | Minimum idle timeout before sending silence to prevent dangling turns. |

### Settings [#settings-1]

| Setting             | Type                 | Default                 | Description                                                     |
| ------------------- | -------------------- | ----------------------- | --------------------------------------------------------------- |
| `model`             | `string`             | `"flux-general-en"`     | Flux model; `"flux-general-multi"` for multilingual.            |
| `eagerEotThreshold` | `number \| null`     | `null` (off)            | Threshold for eager end-of-turn and turn-resumed events.        |
| `eotThreshold`      | `number \| null`     | `null` (Deepgram: 0.7)  | End-of-turn confidence to finish a turn.                        |
| `eotTimeoutMs`      | `number \| null`     | `null` (Deepgram: 5000) | Finish a turn this long after speech, regardless of confidence. |
| `keyterm`           | `string[]`           | `[]`                    | Keyterms to boost.                                              |
| `minConfidence`     | `number \| null`     | `null`                  | Minimum confidence to emit a `TranscriptionFrame`.              |
| `numerals`          | `boolean \| null`    | `null`                  | Spoken numbers as numerals.                                     |
| `profanityFilter`   | `boolean \| null`    | `null`                  | Mask profanity.                                                 |
| `redact`            | `string \| null`     | `null`                  | `"numbers"` or `"aggressive_numbers"`. Changing it reconnects.  |
| `languageHints`     | `Language[] \| null` | `null`                  | Bias detection (`flux-general-multi` only).                     |

### Usage [#usage-1]

```ts title="flux.ts"
import { Language } from "@pipedog/core/transcriptions/language";
import { DeepgramFluxSTTService } from "@pipedog/deepgram/flux/stt";

const stt = new DeepgramFluxSTTService({
  apiKey: process.env.DEEPGRAM_API_KEY ?? "",
  settings: new DeepgramFluxSTTService.Settings({
    model: "flux-general-en",
    eagerEotThreshold: 0.5,
    eotThreshold: 0.8,
  }),
});

stt.addEventHandler("on_end_of_turn", async (_service, transcript: string) => {
  console.log("end of turn:", transcript);
});
```

Multilingual, with language hints:

```ts
const multilingual = new DeepgramFluxSTTService({
  apiKey: process.env.DEEPGRAM_API_KEY ?? "",
  settings: new DeepgramFluxSTTService.Settings({
    model: "flux-general-multi",
    languageHints: [Language.EN, Language.ES],
  }),
});
```

### Event handlers [#event-handlers]

| Event                  | Description                                                        |
| ---------------------- | ------------------------------------------------------------------ |
| `on_start_of_turn`     | Deepgram detected the start of speech.                             |
| `on_end_of_turn`       | Deepgram detected the end of the turn.                             |
| `on_eager_end_of_turn` | Deepgram predicted the end of the turn (with `eagerEotThreshold`). |
| `on_turn_resumed`      | The user kept speaking after an eager end of turn.                 |
| `on_update`            | A transcript update for the turn in progress.                      |

### Notes [#notes-1]

* Emits an `InterimTranscriptionFrame` on every Flux update and a `TranscriptionFrame` at the end
  of the turn.
* Flux needs `linear16` audio, which is what pipedog transports produce.

## Runtime notes [#runtime-notes]

| Runtime            | Works | What it needs |
| ------------------ | ----- | ------------- |
| Bun                | ✅ yes | Nothing.      |
| Node.js            | ✅ yes | Nothing.      |
| Deno               | ✅ yes | Nothing.      |
| Cloudflare Workers | ✅ yes | Nothing.      |

Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/deepgram/package.json`: ✅ means the compat smoke test passes on that runtime.

Both services use the global `WebSocket` and send the key as an `Authorization` header on the
handshake. That works on Bun, Node, Deno and Cloudflare Workers (where the socket is opened
through `fetch`). Pass `webSocketFactory` to open the socket some other way, through a proxy for
example. A standard `WebSocket` doesn't expose the HTTP status of a refused handshake, so
`DeepgramApiError` is only raised when the runtime's error carries one.