# Azure Speech

> Continuous speech recognition from Azure AI Speech with AzureSTTService, over the Speech service's WebSocket protocol.

Source: https://polvallverdu.github.io/pipedog/docs/services/stt/azure/



## Overview [#overview]

**`AzureSTTService`** streams the user's audio to Azure AI Speech and turns the results into
transcription frames: an `InterimTranscriptionFrame` for each hypothesis while the user speaks,
and a `TranscriptionFrame` (with `finalized: true`) for each recognized phrase.

Upstream drives Azure through the native Speech SDK. pipedog speaks the Speech service's own
WebSocket protocol (the one the SDKs use) on the continuous "conversation" endpoint, so no SDK is
needed and the service runs on every runtime.

<Cards>
  <Card title="Upstream page" href="https://docs.pipecat.ai/api-reference/server/services/stt/azure">
    Pipecat's documentation for the same service.
  </Card>

  <Card title="Source" href="https://github.com/polvallverdu/pipedog/tree/main/packages/azure/src">
    `packages/azure/src/stt.ts` and `speech_protocol.ts`.
  </Card>

  <Card title="Azure docs" href="https://learn.microsoft.com/en-us/azure/ai-services/speech-service/how-to-recognize-speech">
    Recognition languages, segmentation and custom models.
  </Card>
</Cards>

## Installation [#installation]

```bash
bun add @pipedog/azure
```

```ts
import { AzureSTTService } from "@pipedog/azure/stt";
```

## Prerequisites [#prerequisites]

* An Azure AI Speech (or Azure AI Foundry) resource, with its key and region from the portal's
  **Keys and Endpoint** page.
* Pass the key as `apiKey` and the region (e.g. `"eastus"`) as `region`. The examples read them
  from `AZURE_SPEECH_API_KEY` and `AZURE_SPEECH_REGION`.

## Configuration [#configuration]

| Option | Type | Required | Description |
|---|---|---|---|
| `apiKey` | `string` | yes | Azure Cognitive Services subscription key. |
| `region` | `string \| null \| undefined` | no | Azure region for the Speech service (e.g., 'eastus'). Required unless `privateEndpoint` is provided. |
| `privateEndpoint` | `string \| null \| undefined` | no | Private endpoint for STT behind firewall. See https://learn.microsoft.com/en-us/azure/ai-services/speech-service/speech-services-private-link?tabs=portal |
| `endpointId` | `string \| null \| undefined` | no | Custom model endpoint id. |
| `settings` | `AzureSTTSettings \| null \| undefined` | no | Runtime-updatable settings. |
| `webSocketFactory` | `((uri: string, options: { headers?: Record<string, string>; protocols?: string[]; }) => WebSocket \| Promise<WebSocket>) \| null \| undefined` | no | Creates the underlying `WebSocket`, to open it some other way. Defaults to `websocketConnect`'s. Not in upstream. |
| `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. |
| `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. |
| `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. |
| `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. |
| `audioPassthrough` | `boolean \| undefined` | no | Whether to pass audio frames downstream after processing. Defaults to true. |
| `sampleRate` | `number \| null \| undefined` | no | The sample rate for audio input. If null, will be determined from the start frame. |
| `sttTtfbTimeout` | `number \| undefined` | no | Time in seconds to wait after VAD stop before reporting TTFB. This delay allows the final transcription to arrive. Defaults to 2.0. |
| `ttfsP99Latency` | `number \| null \| undefined` | no | P99 latency from speech end to final transcript in seconds. Broadcast via STTMetadataFrame at pipeline start. |
| `keepaliveTimeout` | `number \| null \| undefined` | no | Seconds of no audio before sending silence to keep the connection alive. null disables keepalive. |
| `keepaliveInterval` | `number \| undefined` | no | Seconds between idle checks when keepalive is enabled. |

## Settings [#settings]

Runtime-updatable settings, passed as `settings: new AzureSTTService.Settings({...})`. Any
setting you leave out keeps the default below.

| Setting                        | Type                                     | Default                    | Description                                                                                                                                      |
| ------------------------------ | ---------------------------------------- | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ |
| `language`                     | `Language \| string`                     | `Language.EN_US`           | Recognition language, sent as an Azure locale (`"en-US"`).                                                                                       |
| `profanity`                    | `"raw" \| "masked" \| "removed" \| null` | `null` (Azure: `"masked"`) | How profanity appears in transcripts. `"raw"` helps non-English deployments where Azure masks ordinary words. An unknown value throws.           |
| `segmentationSilenceTimeoutMs` | `number \| null`                         | `null` (Azure: 500)        | Silence, in ms (100 to 5000), inside a phrase before Azure finalizes it. Lower finalizes sooner; higher keeps hesitant speech in one transcript. |

## Usage [#usage]

```ts title="stt.ts"
import { AzureSTTService } from "@pipedog/azure/stt";
import { STTUpdateSettingsFrame } from "@pipedog/core/frames/frames";
import { Language } from "@pipedog/core/transcriptions/language";

const stt = new AzureSTTService({
  apiKey: process.env.AZURE_SPEECH_API_KEY ?? "",
  region: process.env.AZURE_SPEECH_REGION ?? "eastus",
  settings: new AzureSTTService.Settings({
    language: Language.EN_US,
    profanity: "raw",
    segmentationSilenceTimeoutMs: 800,
  }),
});
```

Change settings mid-call with an `STTUpdateSettingsFrame`. A change to `language`, `profanity` or
`segmentationSilenceTimeoutMs` reconnects with the new settings:

```ts
// Queue it on the worker, or push it from a processor upstream of the service.
// The service reconnects with the new language.
const update = new STTUpdateSettingsFrame({
  delta: new AzureSTTService.Settings({ language: Language.ES_ES }),
});
```

A Private Link endpoint or custom domain replaces the region, and `endpointId` selects a Custom
Speech model:

```ts
const privateStt = new AzureSTTService({
  apiKey: process.env.AZURE_SPEECH_API_KEY ?? "",
  privateEndpoint: "wss://my-speech.cognitiveservices.azure.com/",
  endpointId: process.env.AZURE_CUSTOM_SPEECH_ENDPOINT_ID ?? null,
});
```

## Notes [#notes]

* The connection opens when the pipeline starts, at the pipeline's input sample rate (16-bit mono
  PCM). If Azure closes it (its session limit, an error), the next audio reopens it, and an error
  close is reported as an `ErrorFrame`.
* `TranscriptionFrame`s carry `finalized: true`: Azure's phrase result is the final one for the
  utterance, so turn-stop strategies can act on it without waiting for VAD.
* Results are requested in Azure's `simple` format; the raw result is on the frame's `result`.

## Runtime notes [#runtime-notes]

| Runtime            | Works | What it needs |
| ------------------ | ----- | ------------- |
| Bun                | ✅ yes | Nothing.      |
| Node.js            | ✅ yes | Nothing.      |
| Deno               | ✅ yes | Nothing.      |
| Cloudflare Workers | ✅ yes | Nothing.      |

Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/azure/package.json`: ✅ means the compat smoke test passes on that runtime.

`AzureSTTService` opens its WebSocket with core's `websocketConnect` and works on
Bun, Node, Deno and Cloudflare Workers. The key is sent both as the `Ocp-Apim-Subscription-Key`
handshake header and as a query parameter, as the Speech SDK does, so it also authenticates where
a `WebSocket` can't send headers. Pass `webSocketFactory` to open the socket some other way.

Differences from upstream: there is no Speech SDK `SpeechConfig`; `_speechConfig` is a small
record of the language, profanity option and properties, and the connection opens in `start()`
rather than `setup()`. The deprecated `language` constructor option isn't ported; use
`settings.language`.