OpenAI

OverviewCopy Anchor Link

• OpenAITTSService: sends each sentence to OpenAI's speech API (audio.speech) and streams the reply back as raw 24 kHz PCM. Supports the gpt-4o-mini-tts instructions and speed controls.

InstallationCopy Anchor Link

bun add @pipedog/openai

import { OpenAITTSService } from "@pipedog/openai/tts";

PrerequisitesCopy Anchor Link

• An OpenAI account and an API key from the OpenAI dashboard.
• Pass the key as apiKey. If you leave it out, the OpenAI SDK reads OPENAI_API_KEY. The examples read it from OPENAI_API_KEY.

OpenAITTSServiceCopy Anchor Link

ConfigurationCopy Anchor Link

SettingsCopy Anchor Link

Runtime-updatable settings, passed as settings: new OpenAITTSService.Settings({...}) (OpenAITTSSettings). Any setting you leave out keeps the default below.

UsageCopy Anchor Link

import { TTSUpdateSettingsFrame } from "@pipedog/core/frames/frames";
import { OpenAITTSService } from "@pipedog/openai/tts";

const tts = new OpenAITTSService({
  apiKey: process.env.OPENAI_API_KEY ?? "",
  sampleRate: 24000,
  settings: new OpenAITTSService.Settings({
    model: "gpt-4o-mini-tts",
    voice: "marin",
    instructions: "Speak in a calm, friendly tone.",
  }),
});

Change the voice or speed mid-call with a TTSUpdateSettingsFrame:

const update = new TTSUpdateSettingsFrame({
  delta: new OpenAITTSService.Settings({ voice: "cedar", speed: 1.2 }),
});

Event handlersCopy Anchor Link

This comes from the TTSService base class.

NotesCopy Anchor Link

• Requests response_format: "pcm" and emits TTSAudioRawFrames (mono, chunkSize bytes each) between TTSStartedFrame and TTSStoppedFrame.
• OpenAI only returns 24 kHz audio. Set sampleRate: 24000 (or a 24 kHz pipeline output rate); any other rate logs a warning, and the frames are still labelled with that rate.
• An unknown voice, or a request OpenAI rejects, produces an ErrorFrame instead of audio.
• httpClient replaces the SDK's fetch, for example to set a longer request timeout.

Runtime notesCopy Anchor Link

Generated from pipecat.runtimes and pipecat.compat in packages/openai/package.json: ✅ means the compat smoke test passes on that runtime.

• The client is the openai JS SDK over fetch. The audio is read from the response body stream, so it works on Bun, Node, Deno and Cloudflare Workers.
• httpClient takes a fetch function where upstream takes an httpx client.
• The deprecated OpenAITTSService.InputParams and the voice, model, instructions, speed and params constructor options are not ported. Use settings.