Whisper

OverviewCopy Anchor Link

@pipedog/whisper is a port of Pipecat's pipecat.services.whisper (base_stt, stt, utils):

• WhisperSTTService (@pipedog/whisper/stt): transcribes each speech segment, cut by the pipeline's VAD, with a Whisper model on your machine. No account, no API key. Upstream wraps faster-whisper (CTranslate2); this one runs faster-whisper's algorithm on ONNX Runtime (@pipedog/whisper/faster_whisper), with ONNX exports of the same Whisper checkpoints.
• WhisperSTTServiceMLX (@pipedog/whisper/stt): upstream's MLX Whisper service, for Apple Silicon. It is not MLX here: it keeps upstream's class, settings, model names and filtering, and runs the same ONNX engine, decoding like mlx_whisper. It works on any platform.
• BaseWhisperSTTService (@pipedog/whisper/base_stt): the base for services that call a Whisper-style transcription API (OpenAI's /v1/audio/transcriptions and compatible servers). OpenAISTTService and GroqSTTService extend it.
• Probability helpers (@pipedog/whisper/utils): extractWhisperProbability, extractOpenaiGpt4oProbability and extractDeepgramProbability read a confidence from a TranscriptionFrame's result.

Local Whisper is one of upstream's keyless local defaults for evals: set service: whisper to transcribe the bot's speech with it instead of Moonshine.

InstallationCopy Anchor Link

bun add @pipedog/whisper
# for the local services, also an ONNX runtime:
bun add onnxruntime-node

onnxruntime-node (native, fastest on Bun, Node and Deno) or onnxruntime-web@1.30.0 (WebAssembly) are optional peers of @pipedog/onnx. The tokenizer, @huggingface/tokenizers (pure JS), is a dependency.

import { Model, WhisperSTTService, WhisperSTTServiceMLX } from "@pipedog/whisper/stt";
import { BaseWhisperSTTService } from "@pipedog/whisper/base_stt";
import { extractWhisperProbability } from "@pipedog/whisper/utils";

The package root re-exports base_stt and utils only. Import the local services from @pipedog/whisper/stt: they pull in the ONNX runtime and the tokenizer, which the API services don't need.

What you needCopy Anchor Link

Local services: the model files, downloaded once from Hugging Face when the pipeline starts, into $PIPEDOG_CACHE_DIR/whisper, else $XDG_CACHE_HOME/pipedog/whisper, else ~/.cache/pipedog/whisper (one folder per repo). Input must be 16 kHz (the pipeline's input sample rate), as upstream: segments are not resampled. And a VAD in the pipeline.

Upstream's model names map to the ONNX export of the same checkpoint:

Plus the tokenizer (2.5 MB) and configs. Every other faster-whisper name works too (tiny.en, base.en, small.en, medium.en, large-v1, large-v2, distil-small.en, distil-large-v3, distil-large-v3.5, large-v3-turbo, turbo), as does any Hugging Face repo with the same ONNX layout (onnx/encoder_model*.onnx, onnx/decoder_model_merged*.onnx).

To control where the files come from, use modelOptions:

• downloadRoot: another cache directory.
• localFilesOnly: true: never download.
• A local directory with the same layout as a model name, as the model setting.
• files: every file explicitly, as bytes, paths or URLs (config.json, generation_config.json, preprocessor_config.json, tokenizer.json, the encoder and the decoder).

// Never download: read the files from a cache you filled beforehand.
const offline = new WhisperSTTService({
  modelOptions: { downloadRoot: "/models/whisper", localFilesOnly: true },
  settings: new WhisperSTTService.Settings({ model: Model.LARGE_V3_TURBO, language: Language.ES }),
});

computeType (faster-whisper's compute_type) picks the weights: default is an fp32 encoder and an int8 (q8) decoder, int8* is q8 for both, float32 is fp32 for both, or pass an ONNX dtype (fp32, q8, int8, uint8, q4, bnb4) or modelOptions: { dtype: { encoder, decoder } }. The distil-whisper/* exports have only fp32 and q8, and fp16 isn't supported. The large fp32 models keep their weights in a separate .onnx_data file, which needs onnxruntime-node: use int8 with onnxruntime-web.

BaseWhisperSTTService: a Whisper-compatible transcription endpoint and its API key, if it needs one (the examples read it from WHISPER_API_KEY), and a VAD in the pipeline.

RuntimesCopy Anchor Link

Generated from pipecat.runtimes and pipecat.compat in packages/whisper/package.json: ✅ means the compat smoke test passes on that runtime.

BaseWhisperSTTService and the probability helpers run on Cloudflare Workers. The local services don't: Whisper models (41 MB to 3.7 GB) don't fit a Worker's 128 MB of memory or its 10 MB bundle limit, and there is no model cache. Their code is smoke-tested there with a fake engine.

WhisperSTTServiceCopy Anchor Link

ConfigurationCopy Anchor Link

SettingsCopy Anchor Link

Runtime-updatable settings, passed as settings: new WhisperSTTService.Settings({...}) (WhisperSTTSettings). Any setting you leave out keeps the default below.

UsageCopy Anchor Link

import { Model, WhisperSTTService } from "@pipedog/whisper/stt";

// Downloads distil-whisper/distil-medium.en (402 MB with int8) on first start.
const local = new WhisperSTTService({
  computeType: "int8", // smaller download, faster on CPU
  settings: new WhisperSTTService.Settings({
    model: Model.DISTIL_MEDIUM_EN,
    language: Language.EN,
    hotwords: "Pipecat pipedog",
  }),
});

The engine works on its own too, with faster-whisper's API:

import { WhisperModel } from "@pipedog/whisper/faster_whisper";

const model = new WhisperModel("tiny", { computeType: "int8" });
const audio = new Float32Array(16000); // 16 kHz mono audio in [-1, 1]
const [segments, info] = await model.transcribe(audio, { language: null });
for await (const s of segments) console.log(info.language, s.start, s.end, s.text, s.noSpeechProb);

NotesCopy Anchor Link

• Emits one TranscriptionFrame per segment. There are no interim results.
• Decoding follows faster-whisper: the same log-mel features, prompt, hotwords and initialPrompt, no_speech_prob, token suppression, timestamp rules, beam search (beam 5), the temperature fallback with best-of-5 sampling, segment splitting and language detection. With fp32 weights, whisper tiny gives the same text, avg_logprob and no_speech_prob (to 4 decimals) as faster-whisper on the test clip.
• The model files load in start(). A failed load is pushed as an error and retried on the next segment.
• device: "cpu" forces the CPU provider. cuda can't be selected.

WhisperSTTServiceMLXCopy Anchor Link

The MLX service's class, settings, model names and filtering, on the same ONNX engine. Each MLXModel selects the ONNX export of the same checkpoint (LARGE_V3_TURBO_Q4 selects its 4-bit q4 weights). It decodes like mlx_whisper: greedy, at the configured temperature only, and drops likely hallucinations. The model loads on the first segment.

ConfigurationCopy Anchor Link

SettingsCopy Anchor Link

Runtime-updatable settings, passed as settings: new WhisperSTTServiceMLX.Settings({...}) (WhisperMLXSTTSettings). Any setting you leave out keeps the default below.

UsageCopy Anchor Link

import { MLXModel, WhisperSTTServiceMLX } from "@pipedog/whisper/stt";

// Not MLX: the same ONNX engine, with MLX Whisper's model names and greedy decoding.
const mlx = new WhisperSTTServiceMLX({
  settings: new WhisperSTTServiceMLX.Settings({
    model: MLXModel.LARGE_V3_TURBO_Q4,
    language: Language.EN,
    temperature: 0.0,
  }),
});

BaseWhisperSTTServiceCopy Anchor Link

ConfigurationCopy Anchor Link

SettingsCopy Anchor Link

Runtime-updatable settings, passed as settings: new BaseWhisperSTTService.Settings({...}). The base leaves every field unset; subclasses set their own defaults (OpenAISTTService uses "gpt-transcribe" and Language.EN).

UsageCopy Anchor Link

The base class cannot be used on its own: its _transcribe() throws. To point at a Whisper-compatible endpoint, pass baseUrl to OpenAISTTService:

import { Language } from "@pipedog/core/transcriptions/language";
import { OpenAISTTService } from "@pipedog/openai/stt";
import { BaseWhisperSTTService, type Transcription } from "@pipedog/whisper/base_stt";

// A server that speaks OpenAI's /v1/audio/transcriptions API.
const stt = new OpenAISTTService({
  apiKey: process.env.WHISPER_API_KEY ?? "",
  baseUrl: "http://localhost:8000/v1",
  settings: new OpenAISTTService.Settings({
    model: "whisper-1",
    language: Language.EN,
  }),
});

For an endpoint that needs a different request, subclass the base and implement _transcribe() with the OpenAI client it built from apiKey, baseUrl and httpClient:

/** A provider whose Whisper-compatible endpoint needs its own request shape. */
class MyWhisperSTTService extends BaseWhisperSTTService {
  // Fail at construction without a key, instead of falling back to OPENAI_API_KEY.
  protected static override _requiresApiKey = true;

  protected override async _transcribe(audio: Uint8Array): Promise<Transcription> {
    return await this._client.audio.transcriptions.create({
      file: new File([audio.slice()], "audio.wav", { type: "audio/wav" }),
      model: String(this._settings.model),
      language: String(this._settings.language),
      response_format: "json",
    });
  }
}

const custom = new MyWhisperSTTService({
  apiKey: process.env.WHISPER_API_KEY ?? "",
  baseUrl: "https://whisper.example.com/v1",
  settings: new MyWhisperSTTService.Settings({ model: "whisper-large-v3", language: Language.EN }),
});

NotesCopy Anchor Link

• Emits one TranscriptionFrame per segment, with the API response as result. There are no interim results.
• Empty transcripts are logged and dropped. With pushEmptyTranscripts: true they are pushed, so the bot can move on when the VAD fired but nobody spoke.
• A transcription error is pushed as an ErrorFrame.
• Languages map to Whisper's codes. A language not in the map falls back to its base code (en from en-US) with a warning.
• Set the static _requiresApiKey to true in a subclass for a provider other than OpenAI. Without it, a missing key makes the OpenAI client fall back to OPENAI_API_KEY and send it to that provider.
• The HTTP client is the openai JS SDK over fetch. httpClient replaces its fetch, for a longer timeout for example. Upstream takes an httpx client, so its Timeout and connection limits do not carry over.

Probability helpersCopy Anchor Link

Each reads a confidence (0 to 1) from a TranscriptionFrame's result, or returns null:

import { TranscriptionFrame } from "@pipedog/core/frames/frames";
import { extractWhisperProbability } from "@pipedog/whisper/utils";

// In a frame processor after an STT service created with includeProbMetrics: true.
function logConfidence(frame: unknown): void {
  if (frame instanceof TranscriptionFrame) {
    const prob = extractWhisperProbability(frame);
    if (prob !== null) console.log(`Transcription confidence: ${(prob * 100).toFixed(2)}%`);
  }
}

Differences from upstreamCopy Anchor Link

• The local services run on ONNX exports of the same checkpoints, not CTranslate2 conversions. int8 quantization differs from CTranslate2's, so quantized outputs differ slightly.
• WhisperSTTServiceMLX is not MLX (above). Its features are faster-whisper's, not mlx_whisper's (which pads 30 s of silence before computing them).
• Model files load in start(); upstream loads them in the constructor, blocking. The English-only language check still runs in the constructor for faster-whisper's names, and after loading for custom models.
• There are no threads: segments are consumed on the event loop, and each decoder step awaits the runtime (onnxruntime-node runs it on its own threads).
• Sampling at temperature above 0 uses Math.random (no seed).
• Not implemented (faster-whisper options the services never use): vad_filter, word_timestamps, hallucination_silence_threshold, clip_timestamps, repetition_penalty, no_repeat_ngram_size, the batched pipeline.
• modelOptions is a pipedog option.
• extractOpenaiGpt4oProbability reads each entry's logprob, since the API returns { token, logprob, bytes } objects that upstream's sum() can't add.
• The deprecated constructor arguments (model, no_speech_prob, language, temperature, prompt) are not ported. Use settings.

LicensesCopy Anchor Link

• @pipedog/whisper: BSD 2-Clause. faster_whisper.ts follows faster-whisper (MIT, Copyright (c) 2023 SYSTRAN) and OpenAI Whisper (MIT).
• Models: OpenAI's Whisper weights and the distil-whisper models are MIT. The ONNX exports are by the Hugging Face onnx-community, Xenova and distil-whisper accounts. They are downloaded at runtime, not shipped.
• @huggingface/tokenizers: Apache-2.0.