# Zoom AI

> Streaming speech-to-text with Zoom's Scribe Live, which ends each turn with its own voice activity detection.

Source: https://polvallverdu.github.io/pipedog/docs/services/stt/zoom-ai/



## Overview [#overview]

`ZoomAISTTService` streams 16 kHz PCM audio to Zoom's Scribe Live over a WebSocket. Scribe's own
voice activity detection decides when the user starts and stops speaking, and each completed turn
comes back as one final transcript. It is a port of Pipecat's `ZoomAISTTService`.

<Cards>
  <Card title="Upstream page" href="https://docs.pipecat.ai/api-reference/server/services/stt/zoom-ai">
    Pipecat's documentation for the same service.
  </Card>

  <Card title="Source" href="https://github.com/polvallverdu/pipedog/tree/main/packages/zoom-ai/src">
    `packages/zoom-ai/src/stt.ts`.
  </Card>

  <Card title="Zoom Scribe" href="https://developers.zoom.us/docs/ai-services/scribe/">
    The Scribe API: features, availability and enablement.
  </Card>
</Cards>

## Installation [#installation]

```bash
bun add @pipedog/zoom-ai
```

## Prerequisites [#prerequisites]

* A Zoom AI Services API key or JWT, passed as `apiKey` (the examples read `ZOOM_AI_API_KEY`).
  It is sent on every connect and reconnect, so it must stay valid for as long as the service
  runs.

## ZoomAISTTService [#zoomaisttservice]

### Configuration [#configuration]

| Option | Type | Required | Description |
|---|---|---|---|
| `apiKey` | `string` | yes | Zoom AI Services API key or JWT. It is sent on every request and reconnect, so it must stay valid for as long as the service runs. |
| `url` | `string \| undefined` | no | Scribe Live WebSocket URL. A plaintext `ws://` URL to a host other than localhost logs a warning. |
| `sampleRate` | `number \| null \| undefined` | no | Input audio sample rate. Audio is resampled to 16 kHz. |
| `shouldInterrupt` | `boolean \| undefined` | no | Whether a user turn started by Scribe interrupts the bot. Defaults to true. |
| `settings` | `ZoomAISTTSettings \| null \| undefined` | no | Runtime-updatable settings. |
| `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. |
| `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. |
| `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. |
| `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. |
| `reconnectBackoffMinWait` | `number \| undefined` | no | Minimum time, in seconds, to wait between reconnection attempts. |
| `reconnectBackoffMaxWait` | `number \| undefined` | no | Maximum time, in seconds, to wait between reconnection attempts. |
| `wsCloseTimeout` | `number \| undefined` | no | Maximum time, in seconds, to wait for the peer to acknowledge the websocket closing handshake before dropping the connection. Applied to connections opened through `_websocketConnect()`. Increase it for peers that need longer to complete a graceful close. |
| `wsSendTimeout` | `number \| null \| undefined` | no | Maximum time, in seconds, a single send may wait on a peer that has stopped reading before the connection is dropped and the send throws `ConnectionClosedError`. Applied to connections opened through `_websocketConnect()`. Pass `null` to wait indefinitely. Defaults to `WS_SEND_TIMEOUT`. |
| `audioPassthrough` | `boolean \| undefined` | no | Whether to pass audio frames downstream after processing. Defaults to true. |
| `sttTtfbTimeout` | `number \| undefined` | no | Time in seconds to wait after VAD stop before reporting TTFB. This delay allows the final transcription to arrive. Defaults to 2.0. |
| `ttfsP99Latency` | `number \| null \| undefined` | no | P99 latency from speech end to final transcript in seconds. Broadcast via STTMetadataFrame at pipeline start. |
| `keepaliveTimeout` | `number \| null \| undefined` | no | Seconds of no audio before sending silence to keep the connection alive. null disables keepalive. |
| `keepaliveInterval` | `number \| undefined` | no | Seconds between idle checks when keepalive is enabled. |
| `reconnectOnError` | `boolean \| undefined` | no | Whether to automatically reconnect on websocket errors. Defaults to true. |

The default URL is `wss://api.zoom.us/v2/aiservices/scribe/live`. A plaintext `ws://` URL to a
host other than localhost logs a warning.

### Settings [#settings]

Runtime-updatable settings, passed as `settings: new ZoomAISTTService.Settings({...})`
(`ZoomAISTTSettings`). Settings you leave out keep their defaults. Changing a setting reconnects
with a new Scribe session.

| Setting             | Type               | Default          | Description                                                                                                                                                          |
| ------------------- | ------------------ | ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `language`          | `Language`         | `Language.EN_US` | The Scribe locale. Regional variants Scribe doesn't list resolve to their base language's locale.                                                                    |
| `silenceDurationMs` | `number \| null`   | `350`            | Silence that ends a turn, tuned for voice agents. For transcription without an agent, a longer value such as 600 can improve accuracy. `null` uses Scribe's default. |
| `diarization`       | `boolean`          | `false`          | Label each transcript with a speaker. The labels are in the Scribe event attached as `TranscriptionFrame.result`, and the speaker tags are removed from the text.    |
| `languageDetection` | `string \| null`   | `null`           | Scribe's language detection mode (`config.language_detection.mode`). `null` turns it off.                                                                            |
| `vocabulary`        | `string[] \| null` | `null`           | Words and phrases that customize recognition.                                                                                                                        |
| `textPolishing`     | `boolean`          | `false`          | Scribe's text polishing. The polished text is the transcript, and the recognized text stays in `TranscriptionFrame.result`.                                          |
| `extra`             | `object`           | `{}`             | Scribe request fields with no setting, shaped like the request. Merged into the `session.update` over the settings above.                                            |

### Usage [#usage]

```ts title="stt.ts"
import { STTUpdateSettingsFrame } from "@pipedog/core/frames/frames";
import { Language } from "@pipedog/core/transcriptions/language";
import { ZoomAISTTService } from "@pipedog/zoom-ai/stt";

const stt = new ZoomAISTTService({
  apiKey: process.env.ZOOM_AI_API_KEY ?? "",
  settings: new ZoomAISTTService.Settings({
    language: Language.EN_US,
    vocabulary: ["pipedog", "Pipecat"],
  }),
});
```

Turn on Scribe's other features:

```ts
// Speaker labels, text polishing, and a Scribe field the settings don't cover, through `extra`.
const transcriber = new ZoomAISTTService({
  apiKey: process.env.ZOOM_AI_API_KEY ?? "",
  settings: new ZoomAISTTService.Settings({
    silenceDurationMs: 600,
    diarization: true,
    textPolishing: true,
    extra: { config: { profanity_filter: true } },
  }),
});
```

Change settings mid-call with an `STTUpdateSettingsFrame`:

```ts
// A settings change reconnects with a new Scribe session.
const update = new STTUpdateSettingsFrame({
  delta: new ZoomAISTTService.Settings({ language: Language.ES }),
});
```

### Notes [#notes]

* Scribe's `speech_started` event broadcasts a `ProposedUserStartedSpeakingFrame`, and each
  completed transcript is pushed as a `TranscriptionFrame` followed by a
  `ProposedUserStoppedSpeakingFrame`. The service recommends `ExternalUserTurnStrategies`, so the
  user aggregator follows Scribe's turns instead of running a local VAD.
* Only final transcripts are pushed. There are no interim transcriptions, and no time to final
  segment (TTFS) metric, since Scribe sets the turn boundary.
* `shouldInterrupt` (default `true`) decides whether a user turn started by Scribe interrupts the
  bot.
* Audio at another sample rate is resampled to 16 kHz. When input stops mid-turn (the user
  muted, say), the service sends enough silence after a second to end the turn.
* When Zoom refuses the connection, the error carries Zoom's reason when the WebSocket error
  exposes the handshake response. A standard `WebSocket` doesn't, so the reason may be missing.

## Runtimes [#runtimes]

| Runtime            | Works          | What it needs |
| ------------------ | -------------- | ------------- |
| Bun                | ✅ yes          | Nothing.      |
| Node.js            | ✅ yes          | Nothing.      |
| Deno               | — not verified | Nothing.      |
| Cloudflare Workers | ✅ yes          | Nothing.      |

Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/zoom-ai/package.json`: ✅ means the compat smoke test passes on that runtime.

The service uses the global `WebSocket` (through core's `websocketConnect`, which sends the auth
headers on the handshake on Cloudflare Workers too). No Zoom SDK is needed.