OpenAI

OverviewCopy Anchor Link

pipedog ports both of Pipecat's OpenAI speech-to-text services:

• OpenAISTTService: transcribes each speech segment with the Audio API (audio.transcriptions). It waits for the user to stop speaking, then sends the segment as a WAV file. Works with the GPT transcription models and Whisper (whisper-1).
• OpenAIRealtimeSTTService: streams audio over a WebSocket to the Realtime API in transcription-only mode, with interim results. Turns come from a local VAD (the default) or from OpenAI's server-side VAD.

InstallationCopy Anchor Link

bun add @pipedog/openai

import { OpenAIRealtimeSTTService, OpenAISTTService } from "@pipedog/openai/stt";

OpenAISTTService builds on BaseWhisperSTTService from @pipedog/whisper, which @pipedog/openai depends on.

PrerequisitesCopy Anchor Link

• An OpenAI account and an API key from the OpenAI dashboard.
• Pass the key as apiKey. OpenAISTTService falls back to OPENAI_API_KEY when you leave it out; OpenAIRealtimeSTTService requires it. The examples read it from OPENAI_API_KEY.

OpenAISTTServiceCopy Anchor Link

ConfigurationCopy Anchor Link

SettingsCopy Anchor Link

Runtime-updatable settings, passed as settings: new OpenAISTTService.Settings({...}) (OpenAISTTSettings). Any setting you leave out keeps the default below.

UsageCopy Anchor Link

import { Language } from "@pipedog/core/transcriptions/language";
import { OpenAISTTService } from "@pipedog/openai/stt";

const stt = new OpenAISTTService({
  apiKey: process.env.OPENAI_API_KEY ?? "",
  settings: new OpenAISTTService.Settings({
    model: "gpt-transcribe",
    language: Language.EN,
    keywords: ["pipedog", "Pipecat"],
  }),
});

Whisper, keeping empty transcripts:

const whisper = new OpenAISTTService({
  apiKey: process.env.OPENAI_API_KEY ?? "",
  settings: new OpenAISTTService.Settings({ model: "whisper-1", temperature: 0 }),
  pushEmptyTranscripts: true,
});

NotesCopy Anchor Link

• A segmented service: it needs a VAD (in the user aggregator) to mark where each segment ends, then pushes one TranscriptionFrame per segment. There are no interim results.
• pushEmptyTranscripts: true pushes a TranscriptionFrame even when nothing was transcribed, so the bot can go on when the VAD fired on noise.
• With includeProbMetrics: true, GPT transcription models report per-token logprobs and Whisper models per-segment logprobs. Diarization models report none, with a warning.
• httpClient replaces the SDK's fetch, for example to set a longer request timeout.

OpenAIRealtimeSTTServiceCopy Anchor Link

ConfigurationCopy Anchor Link

SettingsCopy Anchor Link

Runtime-updatable settings, passed as settings: new OpenAIRealtimeSTTService.Settings({...}) (OpenAIRealtimeSTTSettings).

UsageCopy Anchor Link

import { Language } from "@pipedog/core/transcriptions/language";
import { OpenAIRealtimeSTTService } from "@pipedog/openai/stt";

// Local VAD (the default): a VAD in the user aggregator ends each segment.
const stt = new OpenAIRealtimeSTTService({
  apiKey: process.env.OPENAI_API_KEY ?? "",
  settings: new OpenAIRealtimeSTTService.Settings({
    model: "gpt-realtime-whisper",
    language: Language.EN,
    noiseReduction: "near_field",
    delay: "low",
  }),
});

Server-side VAD. OpenAI detects the turns, so the pipeline needs no local VAD:

// OpenAI detects the turns. Use a model with turn detection, and no local VAD.
const serverVad = new OpenAIRealtimeSTTService({
  apiKey: process.env.OPENAI_API_KEY ?? "",
  turnDetection: { type: "server_vad", threshold: 0.5 },
  settings: new OpenAIRealtimeSTTService.Settings({ model: "gpt-transcribe" }),
});

Settings sent mid-call with an STTUpdateSettingsFrame are applied with a session.update on the open session.

Event handlersCopy Anchor Link

NotesCopy Anchor Link

• Emits InterimTranscriptionFrames as text arrives and a TranscriptionFrame for each completed segment.
• turnDetection defaults to false: a local VAD drives turns, and each VADUserStoppedSpeakingFrame commits the audio buffer so the server transcribes the segment.
• With turnDetection: { type: "server_vad" }, the service broadcasts ProposedUserStartedSpeakingFrame and ProposedUserStoppedSpeakingFrame from the server's speech events, and asks the user aggregator for ExternalUserTurnStrategies (unless you pass your own userTurnStrategies). shouldInterrupt (default true) decides whether those turns interrupt the bot. Server VAD needs a model with turn detection: gpt-transcribe has it, gpt-realtime-whisper does not. See turn detection.
• Audio is sent as 24 kHz 16-bit mono PCM. Other input rates are resampled.
• The Cloudflare Workers example uses this service with server VAD, so the Worker loads no local VAD model.

Runtime notesCopy Anchor Link

Generated from pipecat.runtimes and pipecat.compat in packages/openai/package.json: ✅ means the compat smoke test passes on that runtime.

• OpenAISTTService uses the openai JS SDK over fetch. It works on Bun, Node, Deno and Cloudflare Workers.
• OpenAIRealtimeSTTService sends the key as an Authorization header on the WebSocket handshake. Bun, Node and Deno send it from the global WebSocket; on Cloudflare Workers core opens the socket through fetch.
• The deprecated model, language, prompt, temperature and noiseReduction constructor options are not ported. Use settings.