Azure Speech

OverviewCopy Anchor Link

AzureSTTService streams the user's audio to Azure AI Speech and turns the results into transcription frames: an InterimTranscriptionFrame for each hypothesis while the user speaks, and a TranscriptionFrame (with finalized: true) for each recognized phrase.

Upstream drives Azure through the native Speech SDK. pipedog speaks the Speech service's own WebSocket protocol (the one the SDKs use) on the continuous "conversation" endpoint, so no SDK is needed and the service runs on every runtime.

InstallationCopy Anchor Link

bun add @pipedog/azure

import { AzureSTTService } from "@pipedog/azure/stt";

PrerequisitesCopy Anchor Link

• An Azure AI Speech (or Azure AI Foundry) resource, with its key and region from the portal's Keys and Endpoint page.
• Pass the key as apiKey and the region (e.g. "eastus") as region. The examples read them from AZURE_SPEECH_API_KEY and AZURE_SPEECH_REGION.

ConfigurationCopy Anchor Link

SettingsCopy Anchor Link

Runtime-updatable settings, passed as settings: new AzureSTTService.Settings({...}). Any setting you leave out keeps the default below.

UsageCopy Anchor Link

import { AzureSTTService } from "@pipedog/azure/stt";
import { STTUpdateSettingsFrame } from "@pipedog/core/frames/frames";
import { Language } from "@pipedog/core/transcriptions/language";

const stt = new AzureSTTService({
  apiKey: process.env.AZURE_SPEECH_API_KEY ?? "",
  region: process.env.AZURE_SPEECH_REGION ?? "eastus",
  settings: new AzureSTTService.Settings({
    language: Language.EN_US,
    profanity: "raw",
    segmentationSilenceTimeoutMs: 800,
  }),
});

Change settings mid-call with an STTUpdateSettingsFrame. A change to language, profanity or segmentationSilenceTimeoutMs reconnects with the new settings:

// Queue it on the worker, or push it from a processor upstream of the service.
// The service reconnects with the new language.
const update = new STTUpdateSettingsFrame({
  delta: new AzureSTTService.Settings({ language: Language.ES_ES }),
});

A Private Link endpoint or custom domain replaces the region, and endpointId selects a Custom Speech model:

const privateStt = new AzureSTTService({
  apiKey: process.env.AZURE_SPEECH_API_KEY ?? "",
  privateEndpoint: "wss://my-speech.cognitiveservices.azure.com/",
  endpointId: process.env.AZURE_CUSTOM_SPEECH_ENDPOINT_ID ?? null,
});

NotesCopy Anchor Link

• The connection opens when the pipeline starts, at the pipeline's input sample rate (16-bit mono PCM). If Azure closes it (its session limit, an error), the next audio reopens it, and an error close is reported as an ErrorFrame.
• TranscriptionFrames carry finalized: true: Azure's phrase result is the final one for the utterance, so turn-stop strategies can act on it without waiting for VAD.
• Results are requested in Azure's simple format; the raw result is on the frame's result.

Runtime notesCopy Anchor Link

Generated from pipecat.runtimes and pipecat.compat in packages/azure/package.json: ✅ means the compat smoke test passes on that runtime.

AzureSTTService opens its WebSocket with core's websocketConnect and works on Bun, Node, Deno and Cloudflare Workers. The key is sent both as the Ocp-Apim-Subscription-Key handshake header and as a query parameter, as the Speech SDK does, so it also authenticates where a WebSocket can't send headers. Pass webSocketFactory to open the socket some other way.

Differences from upstream: there is no Speech SDK SpeechConfig; _speechConfig is a small record of the language, profanity option and properties, and the connection opens in start() rather than setup(). The deprecated language constructor option isn't ported; use settings.language.