# ElevenLabs

> Speech-to-text with ElevenLabsSTTService (segmented uploads) and ElevenLabsRealtimeSTTService (WebSocket streaming).

Source: https://polvallverdu.github.io/pipedog/docs/services/stt/elevenlabs/



## Overview [#overview]

ElevenLabs Scribe transcribes speech. pipedog ports both of Pipecat's ElevenLabs speech-to-text
services:

* **`ElevenLabsSTTService`**: uploads each speech segment, cut by the pipeline's VAD, to
  ElevenLabs' `/v1/speech-to-text` endpoint and pushes the transcript it gets back.
* **`ElevenLabsRealtimeSTTService`**: streams audio over ElevenLabs' realtime WebSocket API, with
  partial and committed transcripts. Segments are committed by the pipeline's VAD or by
  ElevenLabs' own.

<Cards>
  <Card title="Upstream page" href="https://docs.pipecat.ai/api-reference/server/services/stt/elevenlabs">
    Pipecat's documentation for the same services.
  </Card>

  <Card title="Source" href="https://github.com/polvallverdu/pipedog/tree/main/packages/elevenlabs/src">
    `packages/elevenlabs/src/stt.ts`.
  </Card>

  <Card title="ElevenLabs docs" href="https://elevenlabs.io/docs/capabilities/speech-to-text">
    Scribe models, languages and the realtime API.
  </Card>
</Cards>

## Installation [#installation]

```bash
bun add @pipedog/elevenlabs
```

```ts
import {
  CommitStrategy,
  ElevenLabsRealtimeSTTService,
  ElevenLabsSTTService,
} from "@pipedog/elevenlabs/stt";
```

## Prerequisites [#prerequisites]

* An ElevenLabs account and an API key from the
  [ElevenLabs dashboard](https://elevenlabs.io/app/settings/api-keys).
* Pass the key as `apiKey`. The examples read it from `ELEVENLABS_API_KEY`.

## ElevenLabsSTTService [#elevenlabssttservice]

### Configuration [#configuration]

| Option | Type | Required | Description |
|---|---|---|---|
| `apiKey` | `string` | yes | ElevenLabs API key for authentication. |
| `fetch` | `((input: RequestInfo \| URL, init?: RequestInit) => Promise<Response>) \| null \| undefined` | no | `fetch` used for HTTP requests. Defaults to the global `fetch`. DIVERGENCE: replaces upstream's required `aiohttp_session`. |
| `baseUrl` | `string \| undefined` | no | Base URL for ElevenLabs API. Defaults to "https://api.elevenlabs.io". |
| `settings` | `ElevenLabsSTTSettings \| null \| undefined` | no | Runtime-updatable settings. Its given fields override the defaults (model "scribe_v2", language English). |
| `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. |
| `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. |
| `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. |
| `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. |
| `audioPassthrough` | `boolean \| undefined` | no | Whether to pass audio frames downstream after processing. Defaults to true. |
| `sampleRate` | `number \| null \| undefined` | no | The sample rate for audio input. If null, will be determined from the start frame. |
| `sttTtfbTimeout` | `number \| undefined` | no | Time in seconds to wait after VAD stop before reporting TTFB. This delay allows the final transcription to arrive. Defaults to 2.0. |
| `ttfsP99Latency` | `number \| null \| undefined` | no | P99 latency from speech end to final transcript in seconds. Broadcast via STTMetadataFrame at pipeline start. |
| `keepaliveTimeout` | `number \| null \| undefined` | no | Seconds of no audio before sending silence to keep the connection alive. null disables keepalive. |
| `keepaliveInterval` | `number \| undefined` | no | Seconds between idle checks when keepalive is enabled. |
| `trailingSilenceSecs` | `number \| undefined` | no | Seconds of silence appended to each segment before it is transcribed, so the model hears the end of speech and can finish the last word. Set to 0 to disable. Defaults to 0.5. |

### Settings [#settings]

Runtime-updatable settings, passed as `settings: new ElevenLabsSTTService.Settings({...})`. Any
setting you leave out keeps the default below. `null` means the field is not sent, so ElevenLabs
uses its own default.

| Setting          | Type               | Default       | Description                                                              |
| ---------------- | ------------------ | ------------- | ------------------------------------------------------------------------ |
| `model`          | `string`           | `"scribe_v2"` | Scribe model.                                                            |
| `language`       | `Language \| null` | `Language.EN` | Transcription language. `null` lets ElevenLabs detect it.                |
| `tagAudioEvents` | `boolean \| null`  | `null`        | Include audio events such as (laughter) in the transcript.               |
| `keyterms`       | `string[] \| null` | `null`        | Terms to bias transcription towards.                                     |
| `noVerbatim`     | `boolean \| null`  | `null`        | Drop filler words, false starts and non-speech sounds. `scribe_v2` only. |

### Usage [#usage]

```ts title="stt.ts"
import { STTUpdateSettingsFrame } from "@pipedog/core/frames/frames";
import { Language } from "@pipedog/core/transcriptions/language";
import {
  CommitStrategy,
  ElevenLabsRealtimeSTTService,
  ElevenLabsSTTService,
} from "@pipedog/elevenlabs/stt";

const stt = new ElevenLabsSTTService({
  apiKey: process.env.ELEVENLABS_API_KEY ?? "",
  settings: new ElevenLabsSTTService.Settings({
    model: "scribe_v2",
    language: Language.EN,
    keyterms: ["pipedog", "Pipecat"],
  }),
});
```

### Notes [#notes]

* A segmented service: it needs VAD in the pipeline (on the user aggregator). Audio between
  the VAD's start and stop is sent as one WAV upload, padded with a little trailing silence.
* Transcription runs in the background, so audio keeps flowing while a segment is transcribed.
* Emits one `TranscriptionFrame` per segment, with the language code ElevenLabs returns.
  There are no interim results.

## ElevenLabsRealtimeSTTService [#elevenlabsrealtimesttservice]

### Configuration [#configuration-1]

| Option | Type | Required | Description |
|---|---|---|---|
| `reconnectBackoffMinWait` | `number \| undefined` | no | Minimum time, in seconds, to wait between reconnection attempts. |
| `reconnectBackoffMaxWait` | `number \| undefined` | no | Maximum time, in seconds, to wait between reconnection attempts. |
| `apiKey` | `string` | yes | ElevenLabs API key for authentication. |
| `baseUrl` | `string \| undefined` | no | Base URL for ElevenLabs WebSocket API. Defaults to "api.elevenlabs.io". |
| `commitStrategy` | `CommitStrategy \| undefined` | no | How to segment speech — `CommitStrategy.MANUAL` (Pipecat VAD) or `CommitStrategy.VAD` (ElevenLabs VAD). Defaults to `CommitStrategy.MANUAL`. |
| `includeTimestamps` | `boolean \| undefined` | no | Whether to include word-level timestamps in transcripts. |
| `enableLogging` | `boolean \| undefined` | no | Whether to enable logging on ElevenLabs' side. |
| `includeLanguageDetection` | `boolean \| undefined` | no | Whether to include language detection in transcripts. |
| `settings` | `ElevenLabsRealtimeSTTSettings \| null \| undefined` | no | Runtime-updatable settings. Its given fields override the defaults (model "scribe_v2_realtime", language auto-detected). |
| `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. |
| `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. |
| `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. |
| `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. |
| `wsCloseTimeout` | `number \| undefined` | no | Maximum time, in seconds, to wait for the peer to acknowledge the websocket closing handshake before dropping the connection. Applied to connections opened through `_websocketConnect()`. Increase it for peers that need longer to complete a graceful close. |
| `wsSendTimeout` | `number \| null \| undefined` | no | Maximum time, in seconds, a single send may wait on a peer that has stopped reading before the connection is dropped and the send throws `ConnectionClosedError`. Applied to connections opened through `_websocketConnect()`. Pass `null` to wait indefinitely. Defaults to `WS_SEND_TIMEOUT`. |
| `audioPassthrough` | `boolean \| undefined` | no | Whether to pass audio frames downstream after processing. Defaults to true. |
| `sampleRate` | `number \| null \| undefined` | no | The sample rate for audio input. If null, will be determined from the start frame. |
| `sttTtfbTimeout` | `number \| undefined` | no | Time in seconds to wait after VAD stop before reporting TTFB. This delay allows the final transcription to arrive. Defaults to 2.0. |
| `ttfsP99Latency` | `number \| null \| undefined` | no | P99 latency from speech end to final transcript in seconds. Broadcast via STTMetadataFrame at pipeline start. |
| `keepaliveTimeout` | `number \| null \| undefined` | no | Seconds of no audio before sending silence to keep the connection alive. null disables keepalive. |
| `keepaliveInterval` | `number \| undefined` | no | Seconds between idle checks when keepalive is enabled. |
| `reconnectOnError` | `boolean \| undefined` | no | Whether to automatically reconnect on websocket errors. Defaults to true. |

### Settings [#settings-1]

| Setting                   | Type               | Default                | Description                                                                       |
| ------------------------- | ------------------ | ---------------------- | --------------------------------------------------------------------------------- |
| `model`                   | `string`           | `"scribe_v2_realtime"` | Scribe realtime model.                                                            |
| `language`                | `Language \| null` | `null` (auto-detect)   | Transcription language.                                                           |
| `keyterms`                | `string[] \| null` | `null`                 | Terms to bias transcription towards.                                              |
| `noVerbatim`              | `boolean \| null`  | `null`                 | Drop filler words, false starts and non-speech sounds.                            |
| `vadSilenceThresholdSecs` | `number \| null`   | `null`                 | Silence before ElevenLabs' VAD commits, 0.3 to 3.0 s. `CommitStrategy.VAD` only.  |
| `vadThreshold`            | `number \| null`   | `null`                 | VAD sensitivity, 0.1 to 0.9 (lower is more sensitive). `CommitStrategy.VAD` only. |
| `minSpeechDurationMs`     | `number \| null`   | `null`                 | Minimum speech duration, 50 to 2000 ms. `CommitStrategy.VAD` only.                |
| `minSilenceDurationMs`    | `number \| null`   | `null`                 | Minimum silence duration, 50 to 2000 ms. `CommitStrategy.VAD` only.               |
| `filterBackgroundAudio`   | `boolean \| null`  | `null`                 | Filter background audio before transcription.                                     |

### Usage [#usage-1]

```ts title="realtime.ts"
const realtime = new ElevenLabsRealtimeSTTService({
  apiKey: process.env.ELEVENLABS_API_KEY ?? "",
  settings: new ElevenLabsRealtimeSTTService.Settings({
    language: Language.EN,
    keyterms: ["pipedog"],
  }),
});
```

With ElevenLabs' VAD committing segments:

```ts
// Let ElevenLabs decide when a segment ends instead of the pipeline's VAD.
const serverVad = new ElevenLabsRealtimeSTTService({
  apiKey: process.env.ELEVENLABS_API_KEY ?? "",
  commitStrategy: CommitStrategy.VAD,
  includeTimestamps: true,
  settings: new ElevenLabsRealtimeSTTService.Settings({
    vadSilenceThresholdSecs: 0.8,
    vadThreshold: 0.4,
  }),
});
```

Change settings mid-call with an `STTUpdateSettingsFrame`. The service reconnects with the new
settings:

```ts
// Queue it on the worker, or push it from a processor upstream of the service.
const update = new STTUpdateSettingsFrame({
  delta: new ElevenLabsRealtimeSTTService.Settings({ language: Language.ES }),
});
```

### Event handlers [#event-handlers]

Inherited from `STTService`:

| Event                 | Description                   |
| --------------------- | ----------------------------- |
| `on_connected`        | Connected to ElevenLabs.      |
| `on_disconnected`     | Disconnected from ElevenLabs. |
| `on_connection_error` | The connection failed.        |

### Notes [#notes-1]

* Emits `InterimTranscriptionFrame` for partial transcripts and `TranscriptionFrame` for
  committed ones.
* With `CommitStrategy.MANUAL` (the default), a `VADUserStoppedSpeakingFrame` makes the service
  send a commit, so the pipeline's VAD decides where segments end. With `CommitStrategy.VAD`,
  ElevenLabs decides, using the `vad*` and `min*DurationMs` settings.
* The audio format is chosen from the sample rate: 8000, 16000, 22050, 24000, 44100 or 48000 Hz.
  Any other rate falls back to `pcm_16000` with a warning.
* The service sends silence as a keepalive after 10 s without audio (checked every 5 s).
* Error messages from ElevenLabs (authentication, quota, rate limit, throttled commits, session
  time limit and others) are pushed as errors.
* `baseUrl` is a bare host and defaults to `"api.elevenlabs.io"`.

## Runtime notes [#runtime-notes]

| Runtime            | Works | What it needs |
| ------------------ | ----- | ------------- |
| Bun                | ✅ yes | Nothing.      |
| Node.js            | ✅ yes | Nothing.      |
| Deno               | ✅ yes | Nothing.      |
| Cloudflare Workers | ✅ yes | Nothing.      |

Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/elevenlabs/package.json`: ✅ means the compat smoke test passes on that runtime.

* `ElevenLabsSTTService` uploads with `fetch` and `FormData`. Upstream's required
  `aiohttp_session` is replaced by an optional `fetch` option (the global `fetch` by default).
* `ElevenLabsRealtimeSTTService` uses the global `WebSocket` and sends the key as an
  `xi-api-key` header on the handshake. That works on Bun, Node and Deno, and on Cloudflare
  Workers, where the socket is opened through `fetch`.
* Not ported: the deprecated `InputParams` classes and the `model` / `params` constructor
  options. Use `settings`.