ElevenLabs

OverviewCopy Anchor Link

ElevenLabs Scribe transcribes speech. pipedog ports both of Pipecat's ElevenLabs speech-to-text services:

• ElevenLabsSTTService: uploads each speech segment, cut by the pipeline's VAD, to ElevenLabs' /v1/speech-to-text endpoint and pushes the transcript it gets back.
• ElevenLabsRealtimeSTTService: streams audio over ElevenLabs' realtime WebSocket API, with partial and committed transcripts. Segments are committed by the pipeline's VAD or by ElevenLabs' own.

InstallationCopy Anchor Link

bun add @pipedog/elevenlabs

import {
  CommitStrategy,
  ElevenLabsRealtimeSTTService,
  ElevenLabsSTTService,
} from "@pipedog/elevenlabs/stt";

PrerequisitesCopy Anchor Link

• An ElevenLabs account and an API key from the ElevenLabs dashboard.
• Pass the key as apiKey. The examples read it from ELEVENLABS_API_KEY.

ElevenLabsSTTServiceCopy Anchor Link

ConfigurationCopy Anchor Link

SettingsCopy Anchor Link

Runtime-updatable settings, passed as settings: new ElevenLabsSTTService.Settings({...}). Any setting you leave out keeps the default below. null means the field is not sent, so ElevenLabs uses its own default.

UsageCopy Anchor Link

import { STTUpdateSettingsFrame } from "@pipedog/core/frames/frames";
import { Language } from "@pipedog/core/transcriptions/language";
import {
  CommitStrategy,
  ElevenLabsRealtimeSTTService,
  ElevenLabsSTTService,
} from "@pipedog/elevenlabs/stt";

const stt = new ElevenLabsSTTService({
  apiKey: process.env.ELEVENLABS_API_KEY ?? "",
  settings: new ElevenLabsSTTService.Settings({
    model: "scribe_v2",
    language: Language.EN,
    keyterms: ["pipedog", "Pipecat"],
  }),
});

NotesCopy Anchor Link

• A segmented service: it needs VAD in the pipeline (on the user aggregator). Audio between the VAD's start and stop is sent as one WAV upload, padded with a little trailing silence.
• Transcription runs in the background, so audio keeps flowing while a segment is transcribed.
• Emits one TranscriptionFrame per segment, with the language code ElevenLabs returns. There are no interim results.

ElevenLabsRealtimeSTTServiceCopy Anchor Link

ConfigurationCopy Anchor Link

SettingsCopy Anchor Link

UsageCopy Anchor Link

const realtime = new ElevenLabsRealtimeSTTService({
  apiKey: process.env.ELEVENLABS_API_KEY ?? "",
  settings: new ElevenLabsRealtimeSTTService.Settings({
    language: Language.EN,
    keyterms: ["pipedog"],
  }),
});

With ElevenLabs' VAD committing segments:

// Let ElevenLabs decide when a segment ends instead of the pipeline's VAD.
const serverVad = new ElevenLabsRealtimeSTTService({
  apiKey: process.env.ELEVENLABS_API_KEY ?? "",
  commitStrategy: CommitStrategy.VAD,
  includeTimestamps: true,
  settings: new ElevenLabsRealtimeSTTService.Settings({
    vadSilenceThresholdSecs: 0.8,
    vadThreshold: 0.4,
  }),
});

Change settings mid-call with an STTUpdateSettingsFrame. The service reconnects with the new settings:

// Queue it on the worker, or push it from a processor upstream of the service.
const update = new STTUpdateSettingsFrame({
  delta: new ElevenLabsRealtimeSTTService.Settings({ language: Language.ES }),
});

Event handlersCopy Anchor Link

Inherited from STTService:

NotesCopy Anchor Link

• Emits InterimTranscriptionFrame for partial transcripts and TranscriptionFrame for committed ones.
• With CommitStrategy.MANUAL (the default), a VADUserStoppedSpeakingFrame makes the service send a commit, so the pipeline's VAD decides where segments end. With CommitStrategy.VAD, ElevenLabs decides, using the vad* and min*DurationMs settings.
• The audio format is chosen from the sample rate: 8000, 16000, 22050, 24000, 44100 or 48000 Hz. Any other rate falls back to pcm_16000 with a warning.
• The service sends silence as a keepalive after 10 s without audio (checked every 5 s).
• Error messages from ElevenLabs (authentication, quota, rate limit, throttled commits, session time limit and others) are pushed as errors.
• baseUrl is a bare host and defaults to "api.elevenlabs.io".

Runtime notesCopy Anchor Link

Generated from pipecat.runtimes and pipecat.compat in packages/elevenlabs/package.json: ✅ means the compat smoke test passes on that runtime.

• ElevenLabsSTTService uploads with fetch and FormData. Upstream's required aiohttp_session is replaced by an optional fetch option (the global fetch by default).
• ElevenLabsRealtimeSTTService uses the global WebSocket and sends the key as an xi-api-key header on the handshake. That works on Bun, Node and Deno, and on Cloudflare Workers, where the socket is opened through fetch.
• Not ported: the deprecated InputParams classes and the model / params constructor options. Use settings.