# AssemblyAI

> Streaming speech-to-text with AssemblyAI Universal-Streaming (v3), plus the sync HTTP service.

Source: https://polvallverdu.github.io/pipedog/docs/services/stt/assemblyai/



## Overview [#overview]

pipedog ports both of Pipecat's AssemblyAI speech-to-text services:

* **`AssemblyAISTTService`**: Universal-Streaming v3 over a WebSocket, with turn detection, keyterms, diarization, language detection and hot-updatable settings.
* **`AssemblyAISyncSTTService`**: AssemblyAI's sync HTTP API for segmented (VAD-bounded) audio, with optional pre-warming and conversation context.

The same package has AssemblyAI's [text-to-speech](https://polvallverdu.github.io/pipedog/docs/services/tts/assemblyai.md) service.

<Cards>
  <Card title="Upstream page" href="https://docs.pipecat.ai/api-reference/server/services/stt/assemblyai">
    Pipecat's documentation for the same service.
  </Card>

  <Card title="Source" href="https://github.com/polvallverdu/pipedog/tree/main/packages/assemblyai/src">
    `packages/assemblyai/src`.
  </Card>
</Cards>

## Installation [#installation]

```bash
bun add @pipedog/assemblyai
```

## Prerequisites [#prerequisites]

* An [AssemblyAI](https://www.assemblyai.com) account and API key, passed as `apiKey` (the examples read `ASSEMBLYAI_API_KEY`).

## AssemblyAISTTService [#assemblyaisttservice]

### Configuration [#configuration]

| Option | Type | Required | Description |
|---|---|---|---|
| `apiKey` | `string` | yes | AssemblyAI API key for authentication. |
| `apiEndpointBaseUrl` | `string \| undefined` | no | WebSocket endpoint URL. Defaults to "wss://streaming.assemblyai.com/v3/ws". |
| `encoding` | `string \| undefined` | no | Audio encoding format. Defaults to "pcm_s16le". |
| `vadForceTurnEndpoint` | `boolean \| undefined` | no | Controls turn detection mode. When true (Pipecat mode, default), AssemblyAI is forced to return finals ASAP so Pipecat's turn detection decides when the user is done. When false (AssemblyAI turn detection mode, U3 Pro models only), AssemblyAI's model controls turn endings and the service proposes turn starts and stops. |
| `shouldInterrupt` | `boolean \| undefined` | no | Whether to interrupt the bot when the user starts speaking in AssemblyAI turn detection mode (`vadForceTurnEndpoint: false`). Defaults to true. |
| `speakerFormat` | `string \| null \| undefined` | no | Format string for speaker labels when diarization is enabled; use `{speaker}` and `{text}`. Example: "<{speaker}>{text}</{speaker}>". |
| `settings` | `AssemblyAISTTSettings \| null \| undefined` | no | Runtime-updatable settings. |
| `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. |
| `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. |
| `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. |
| `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. |
| `reconnectBackoffMinWait` | `number \| undefined` | no | Minimum time, in seconds, to wait between reconnection attempts. |
| `reconnectBackoffMaxWait` | `number \| undefined` | no | Maximum time, in seconds, to wait between reconnection attempts. |
| `wsCloseTimeout` | `number \| undefined` | no | Maximum time, in seconds, to wait for the peer to acknowledge the websocket closing handshake before dropping the connection. Applied to connections opened through `_websocketConnect()`. Increase it for peers that need longer to complete a graceful close. |
| `wsSendTimeout` | `number \| null \| undefined` | no | Maximum time, in seconds, a single send may wait on a peer that has stopped reading before the connection is dropped and the send throws `ConnectionClosedError`. Applied to connections opened through `_websocketConnect()`. Pass `null` to wait indefinitely. Defaults to `WS_SEND_TIMEOUT`. |
| `audioPassthrough` | `boolean \| undefined` | no | Whether to pass audio frames downstream after processing. Defaults to true. |
| `sampleRate` | `number \| null \| undefined` | no | The sample rate for audio input. If null, will be determined from the start frame. |
| `sttTtfbTimeout` | `number \| undefined` | no | Time in seconds to wait after VAD stop before reporting TTFB. This delay allows the final transcription to arrive. Defaults to 2.0. |
| `ttfsP99Latency` | `number \| null \| undefined` | no | P99 latency from speech end to final transcript in seconds. Broadcast via STTMetadataFrame at pipeline start. |
| `keepaliveTimeout` | `number \| null \| undefined` | no | Seconds of no audio before sending silence to keep the connection alive. null disables keepalive. |
| `keepaliveInterval` | `number \| undefined` | no | Seconds between idle checks when keepalive is enabled. |
| `reconnectOnError` | `boolean \| undefined` | no | Whether to automatically reconnect on websocket errors. Defaults to true. |

### Settings [#settings]

Runtime-updatable settings, passed as `settings: new AssemblyAISTTService.Settings({...})`. Settings you leave out keep their defaults.

| Setting                                       | Type                                 | Default               | Description                                                        |
| --------------------------------------------- | ------------------------------------ | --------------------- | ------------------------------------------------------------------ |
| `model`                                       | `string`                             | `"universal-3-6-pro"` | Speech model.                                                      |
| `language` / `languageCode` / `languageCodes` | `Language` / `string` / `Language[]` |                       | Language selection.                                                |
| `formatTurns`                                 | `boolean`                            |                       | Return formatted final turns.                                      |
| `endOfTurnConfidenceThreshold`                | `number`                             |                       | End-of-turn confidence.                                            |
| `minTurnSilence` / `maxTurnSilence`           | `number`                             |                       | Silence bounds, in ms.                                             |
| `keytermsPrompt` / `prompt`                   | `string[]` / `string`                |                       | Vocabulary boosting and prompting.                                 |
| `speakerLabels`                               | `boolean`                            |                       | Speaker diarization.                                               |
| `vadThreshold`                                | `number`                             |                       | Server VAD threshold.                                              |
| `agentContext`, `previousContextNTurns`       |                                      |                       | Conversation context (`updateAgentContext(text)` updates it live). |

More (`languageDetection`, `domain`, `voiceFocus`, `mode`, ...) are listed on `AssemblyAISTTSettings`.

### Usage [#usage]

```ts title="stt.ts"
import { AssemblyAISTTService } from "@pipedog/assemblyai/stt";
import { STTUpdateSettingsFrame } from "@pipedog/core/frames/frames";
import { Language } from "@pipedog/core/transcriptions/language";

const stt = new AssemblyAISTTService({
  apiKey: process.env.ASSEMBLYAI_API_KEY ?? "",
  settings: new AssemblyAISTTService.Settings({
    model: "universal-3-6-pro",
    keytermsPrompt: ["pipedog", "Pipecat"],
    languageCodes: [Language.EN, Language.ES],
  }),
});
```

### Notes [#notes]

* Emits `InterimTranscriptionFrame` for partial turns and `TranscriptionFrame` for final turns; fires `on_end_of_turn`.
* With `vadForceTurnEndpoint` (default), a VAD stop sends `ForceEndpoint`.

Update settings mid-call; fields AssemblyAI supports are sent as `UpdateConfiguration` without reconnecting:

```ts
// Hot-updatable settings apply without reconnecting.
const update = new STTUpdateSettingsFrame({
  delta: new AssemblyAISTTService.Settings({ languageCodes: [Language.ES] }),
});
```

## AssemblyAISyncSTTService [#assemblyaisyncsttservice]

### Configuration [#configuration-1]

| Option | Type | Required | Description |
|---|---|---|---|
| `apiKey` | `string` | yes | AssemblyAI API key for authentication. |
| `fetch` | `((input: RequestInfo \| URL, init?: RequestInit) => Promise<Response>) \| null \| undefined` | no | `fetch` used for HTTP requests. Defaults to the global `fetch`. (Upstream takes an aiohttp session here.) |
| `baseUrl` | `string \| undefined` | no | Base URL for the Sync API. Override for a data-residency endpoint (e.g. `https://sync.us.assemblyai.com` or `https://sync.eu.assemblyai.com`). |
| `enablePrewarming` | `boolean \| undefined` | no | Whether to open the connection when the user starts speaking so the transcription request avoids the connection handshake. Defaults to true. |
| `maxContextTurns` | `number \| undefined` | no | Number of prior conversation turns (user transcripts and agent replies together) automatically carried as `conversation_context`. 0 disables it. Ignored when `conversationContext` is set explicitly. Defaults to 5. |
| `maxContextChars` | `number \| undefined` | no | Character budget for the same buffer; oldest turns drop first. Defaults to 1500. |
| `settings` | `AssemblyAISyncSTTSettings \| null \| undefined` | no | Runtime-updatable settings. |
| `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. |
| `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. |
| `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. |
| `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. |
| `audioPassthrough` | `boolean \| undefined` | no | Whether to pass audio frames downstream after processing. Defaults to true. |
| `sampleRate` | `number \| null \| undefined` | no | The sample rate for audio input. If null, will be determined from the start frame. |
| `sttTtfbTimeout` | `number \| undefined` | no | Time in seconds to wait after VAD stop before reporting TTFB. This delay allows the final transcription to arrive. Defaults to 2.0. |
| `ttfsP99Latency` | `number \| null \| undefined` | no | P99 latency from speech end to final transcript in seconds. Broadcast via STTMetadataFrame at pipeline start. |
| `keepaliveTimeout` | `number \| null \| undefined` | no | Seconds of no audio before sending silence to keep the connection alive. null disables keepalive. |
| `keepaliveInterval` | `number \| undefined` | no | Seconds between idle checks when keepalive is enabled. |
| `trailingSilenceSecs` | `number \| undefined` | no | Seconds of silence appended to each segment before it is transcribed, so the model hears the end of speech and can finish the last word. Set to 0 to disable. Defaults to 0.5. |

### Settings [#settings-1]

Runtime-updatable settings, passed as `settings: new AssemblyAISyncSTTService.Settings({...})`. Settings you leave out keep their defaults.

| Setting                    | Description                        |
| -------------------------- | ---------------------------------- |
| `prompt`, `keytermsPrompt` | Prompting and vocabulary boosting. |
| `conversationContext`      | Prior turns sent as context.       |
| `languageCodes`            | Language selection.                |
| `timestamps`               | Request word timestamps.           |

### Notes [#notes-1]

* Pass `fetch` to use your own HTTP client; `warm()` and `enablePrewarming` call `/v1/warm`.

## Runtimes [#runtimes]

| Runtime            | Works | What it needs |
| ------------------ | ----- | ------------- |
| Bun                | ✅ yes | Nothing.      |
| Node.js            | ✅ yes | Nothing.      |
| Deno               | ✅ yes | Nothing.      |
| Cloudflare Workers | ✅ yes | Nothing.      |

Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/assemblyai/package.json`: ✅ means the compat smoke test passes on that runtime.

The service uses `fetch` and the global `WebSocket` (through core's `websocketConnect`, which sends the auth header on the handshake on Cloudflare Workers too). Pass no SDK; nothing is Node-only. The Sync service uses `fetch` only.