# Azure Speech

> Neural voices from Azure AI Speech with AzureTTSService (streaming, with word timestamps) and AzureHttpTTSService (REST).

Source: https://polvallverdu.github.io/pipedog/docs/services/tts/azure/



## Overview [#overview]

pipedog ports both of Pipecat's Azure text-to-speech services:

* **`AzureTTSService`**: streams each sentence's audio over the Speech service's WebSocket
  synthesis protocol, with word boundaries turned into word-timed `TTSTextFrame`s.
* **`AzureHttpTTSService`**: one `POST` to the REST endpoint per sentence, returning the whole
  sentence's audio.

Both build the same SSML from their settings (voice, style, prosody, emphasis, HD voice parameters,
locale pinning). Upstream drives Azure through the native Speech SDK; pipedog speaks the service's
WebSocket and REST protocols directly, so no SDK is needed.

<Cards>
  <Card title="Upstream page" href="https://docs.pipecat.ai/api-reference/server/services/tts/azure">
    Pipecat's documentation for the same services.
  </Card>

  <Card title="Source" href="https://github.com/polvallverdu/pipedog/tree/main/packages/azure/src">
    `packages/azure/src/tts.ts` and `speech_protocol.ts`.
  </Card>

  <Card title="Azure docs" href="https://learn.microsoft.com/en-us/azure/ai-services/speech-service/speech-synthesis-markup">
    Voices, SSML styles and prosody.
  </Card>
</Cards>

## Installation [#installation]

```bash
bun add @pipedog/azure
```

```ts
import { AzureHttpTTSService, AzureTTSService } from "@pipedog/azure/tts";
```

## Prerequisites [#prerequisites]

* An Azure AI Speech (or Azure AI Foundry) resource, with its key and region from the portal's
  **Keys and Endpoint** page.
* Pass the key as `apiKey` and the region (e.g. `"eastus"`) as `region`, or a Private Link or
  custom-domain URL as `privateEndpoint`. The examples read them from `AZURE_SPEECH_API_KEY` and
  `AZURE_SPEECH_REGION`.

## AzureTTSService [#azurettsservice]

### Configuration [#configuration]

| Option | Type | Required | Description |
|---|---|---|---|
| `webSocketFactory` | `((uri: string, options: { headers?: Record<string, string>; protocols?: string[]; }) => WebSocket \| Promise<WebSocket>) \| null \| undefined` | no | Creates the underlying `WebSocket`, to open it some other way. Defaults to `websocketConnect`'s. Not in upstream. |
| `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. |
| `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. |
| `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. |
| `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. |
| `textAggregationMode` | `TextAggregationMode \| null \| undefined` | no | How to aggregate incoming text before synthesis. `TextAggregationMode.SENTENCE` (default) buffers until sentence boundaries, `TextAggregationMode.TOKEN` streams tokens directly for lower latency. |
| `pushTextFrames` | `boolean \| undefined` | no | Whether to push TextFrames and LLMFullResponseEndFrames. |
| `pushStopFrames` | `boolean \| undefined` | no | Whether to automatically push TTSStoppedFrames. |
| `pushStartFrame` | `boolean \| undefined` | no | Whether to automatically create audio contexts and push TTSStartedFrames. When true, the base class handles `createAudioContext` and yields `TTSStartedFrame` before each synthesis call, so `runTts` implementations do not need to. |
| `stopFrameTimeoutS` | `number \| undefined` | no | Idle time before pushing TTSStoppedFrame when pushStopFrames is true. |
| `pushSilenceAfterStop` | `boolean \| undefined` | no | Whether to push silence audio after TTSStoppedFrame. |
| `silenceTimeS` | `number \| undefined` | no | Duration of silence to push when pushSilenceAfterStop is true. |
| `pauseFrameProcessing` | `boolean \| undefined` | no | Whether to hold incoming frames after a turn until its audio contexts end. Needed by services whose audio doesn't carry its request's context ID. |
| `maxConsecutiveZeroAudioContexts` | `number \| undefined` | no | How many consecutive TTS contexts may complete without producing any audio before the service is reported unable to do its job. Catches a provider that accepts requests and stays silent — an unknown voice ID, say — which no error ever surfaces. Every silent context reports an error the service can carry on from, whatever the limit is; on reaching the limit the service reports a permanent error instead, stops being given work, and the pipeline worker applies its `ProcessorUnusablePolicy`. Set to 0 to report silent contexts without ever writing the service off. |
| `appendTrailingSpace` | `boolean \| undefined` | no | Whether to append a trailing space to text before sending to TTS. This helps prevent some TTS services from vocalizing trailing punctuation (e.g., "dot"). Only applied in sentence aggregation mode; when streaming tokens, the incoming text's own whitespace is preserved. |
| `sampleRate` | `number \| null \| undefined` | no | Output sample rate for generated audio. |
| `skipTextTypes` | `string[] \| null \| undefined` | no | List of text types that should not be spoken. |
| `textTransforms` | `[string, TextTransform][] \| null \| undefined` | no | A list of TextTransforms to transform text just before sending it to TTS. Each TextTransform takes the aggregated text and its type, and returns the transformed text. To register, provide a list of tuples of `[textType \| "*", transformFunction]`. |
| `transportDestination` | `string \| null \| undefined` | no | Destination for generated audio frames. |
| `reuseContextIdWithinTurn` | `boolean \| undefined` | no | Whether the service should reuse context IDs within the same turn. |
| `audioCache` | `TTSAudioCacheStore \| null \| undefined` | no | Where to keep synthesized audio for reuse (e.g. a shared `TTSAudioCache`). Text whose audio is in the cache is served from it, with no request to the provider. Lines get there through `warmAudioCache()`, and also from live synthesis when `cacheSynthesizedAudio` allows it. Only services that yield their audio from `runTts` (HTTP and streaming-RPC services, see `supportsAudioCache`) use it, and only in sentence aggregation mode. Defaults to null (no cache). pipedog only. |
| `cacheSynthesizedAudio` | `boolean \| ((text: string) => boolean) \| undefined` | no | Whether audio synthesized live is stored in `audioCache`, or a function that decides per text. Defaults to false: only warmed lines are stored, since a generated reply can repeat what the user said. pipedog only. |
| `audioCacheNamespace` | `string \| undefined` | no | Extra part of the cache key, for constructor options that change the audio but are not in `settings`. pipedog only. |
| `apiKey` | `string` | yes | Azure Cognitive Services subscription key. |
| `region` | `string \| null \| undefined` | no | Azure region identifier (e.g., "eastus", "westus2"). Required unless `privateEndpoint` is provided. |
| `privateEndpoint` | `string \| null \| undefined` | no | Custom endpoint URL for Azure Speech Services (e.g., "https://my-resource.cognitiveservices.azure.com/"). Use this when connecting via Private Link or a custom domain. See https://learn.microsoft.com/en-us/azure/ai-services/speech-service/speech-services-private-link |
| `settings` | `AzureTTSSettings \| null \| undefined` | no | Runtime-updatable settings. |
| `fetch` | `{ (input: RequestInfo \| URL, init?: RequestInit): Promise<Response>; (input: string \| URL \| Request, init?: RequestInit): Promise<Response>; } \| null \| undefined` | no | `fetch` used for HTTP requests (the REST synthesis, and the custom-domain redirect lookup). Defaults to the global `fetch`. Not in upstream. |

### Settings [#settings]

Runtime-updatable settings, passed as `settings: new AzureTTSService.Settings({...})`. Any
setting you leave out keeps the default below. `AzureHttpTTSService` uses the same settings class.

| Setting           | Type                 | Default              | Description                                                                                                |
| ----------------- | -------------------- | -------------------- | ---------------------------------------------------------------------------------------------------------- |
| `voice`           | `string`             | `"en-US-SaraNeural"` | Azure voice name.                                                                                          |
| `language`        | `Language \| string` | `"en-US"`            | `xml:lang` of the SSML.                                                                                    |
| `forceLocale`     | `boolean`            | `false`              | Wrap the text in `<lang xml:lang>` so a multilingual voice speaks in `language` instead of auto-detecting. |
| `style`           | `string \| null`     | `null`               | Speaking style (`"cheerful"`, `"sad"`, ...), as `<mstts:express-as>`.                                      |
| `styleDegree`     | `string \| null`     | `null`               | Style intensity, 0.01 to 2.0.                                                                              |
| `role`            | `string \| null`     | `null`               | Role-play voice (`"YoungAdultFemale"`, ...).                                                               |
| `rate`            | `string \| null`     | `null`               | Prosody rate (`"1.25"`, `"slow"`).                                                                         |
| `pitch`           | `string \| null`     | `null`               | Prosody pitch (`"+10%"`, `"-5Hz"`).                                                                        |
| `volume`          | `string \| null`     | `null`               | Prosody volume (`"+20%"`, `"loud"`).                                                                       |
| `emphasis`        | `string \| null`     | `null`               | `"strong"`, `"moderate"` or `"reduced"`.                                                                   |
| `effect`          | `string \| null`     | `null`               | Device effect: `"eq_car"`, or `"eq_telecomhp8k"` with an 8 kHz pipeline.                                   |
| `voiceParameters` | `string \| null`     | `null`               | HD voice parameters (`"temperature=0.7;top_p=0.8"`).                                                       |

### Usage [#usage]

```ts title="tts.ts"
import { AzureHttpTTSService, AzureTTSService } from "@pipedog/azure/tts";
import { TTSUpdateSettingsFrame } from "@pipedog/core/frames/frames";

const tts = new AzureTTSService({
  apiKey: process.env.AZURE_SPEECH_API_KEY ?? "",
  region: process.env.AZURE_SPEECH_REGION ?? "eastus",
  settings: new AzureTTSService.Settings({
    voice: "en-US-AvaMultilingualNeural",
    language: "en-US",
    style: "cheerful",
    rate: "1.1",
  }),
});
```

Change settings mid-call with a `TTSUpdateSettingsFrame`. The SSML is built per sentence, so the
change applies from the next one:

```ts
// Queue it on the worker, or push it from a processor upstream of the service.
// SSML settings apply from the next sentence.
const update = new TTSUpdateSettingsFrame({
  delta: new AzureTTSService.Settings({ language: "en-GB", forceLocale: true }),
});
```

### Notes [#notes]

* The synthesis connection opens on the first sentence and is reused. Audio is raw 16-bit mono PCM
  at the pipeline's output rate (8, 16, 22.05, 24, 44.1 or 48 kHz; other rates fall back to
  24 kHz).
* Word boundaries become `TTSTextFrame`s timed against the audio. Trailing punctuation is merged
  into the word before it, and Chinese, Japanese and Korean characters are merged up to the next
  punctuation or space. The last word always arrives before the `TTSStoppedFrame`.
* On an interruption the connection is closed, which stops Azure from sending more audio for the
  interrupted sentence; the next sentence opens a new one. Upstream calls the SDK's
  `stop_speaking_async()` instead.

## AzureHttpTTSService [#azurehttpttsservice]

### Configuration [#configuration-1]

| Option | Type | Required | Description |
|---|---|---|---|
| `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. |
| `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. |
| `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. |
| `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. |
| `textAggregationMode` | `TextAggregationMode \| null \| undefined` | no | How to aggregate incoming text before synthesis. `TextAggregationMode.SENTENCE` (default) buffers until sentence boundaries, `TextAggregationMode.TOKEN` streams tokens directly for lower latency. |
| `pushTextFrames` | `boolean \| undefined` | no | Whether to push TextFrames and LLMFullResponseEndFrames. |
| `pushStopFrames` | `boolean \| undefined` | no | Whether to automatically push TTSStoppedFrames. |
| `pushStartFrame` | `boolean \| undefined` | no | Whether to automatically create audio contexts and push TTSStartedFrames. When true, the base class handles `createAudioContext` and yields `TTSStartedFrame` before each synthesis call, so `runTts` implementations do not need to. |
| `stopFrameTimeoutS` | `number \| undefined` | no | Idle time before pushing TTSStoppedFrame when pushStopFrames is true. |
| `pushSilenceAfterStop` | `boolean \| undefined` | no | Whether to push silence audio after TTSStoppedFrame. |
| `silenceTimeS` | `number \| undefined` | no | Duration of silence to push when pushSilenceAfterStop is true. |
| `pauseFrameProcessing` | `boolean \| undefined` | no | Whether to hold incoming frames after a turn until its audio contexts end. Needed by services whose audio doesn't carry its request's context ID. |
| `maxConsecutiveZeroAudioContexts` | `number \| undefined` | no | How many consecutive TTS contexts may complete without producing any audio before the service is reported unable to do its job. Catches a provider that accepts requests and stays silent — an unknown voice ID, say — which no error ever surfaces. Every silent context reports an error the service can carry on from, whatever the limit is; on reaching the limit the service reports a permanent error instead, stops being given work, and the pipeline worker applies its `ProcessorUnusablePolicy`. Set to 0 to report silent contexts without ever writing the service off. |
| `appendTrailingSpace` | `boolean \| undefined` | no | Whether to append a trailing space to text before sending to TTS. This helps prevent some TTS services from vocalizing trailing punctuation (e.g., "dot"). Only applied in sentence aggregation mode; when streaming tokens, the incoming text's own whitespace is preserved. |
| `sampleRate` | `number \| null \| undefined` | no | Output sample rate for generated audio. |
| `skipTextTypes` | `string[] \| null \| undefined` | no | List of text types that should not be spoken. |
| `textTransforms` | `[string, TextTransform][] \| null \| undefined` | no | A list of TextTransforms to transform text just before sending it to TTS. Each TextTransform takes the aggregated text and its type, and returns the transformed text. To register, provide a list of tuples of `[textType \| "*", transformFunction]`. |
| `transportDestination` | `string \| null \| undefined` | no | Destination for generated audio frames. |
| `reuseContextIdWithinTurn` | `boolean \| undefined` | no | Whether the service should reuse context IDs within the same turn. |
| `audioCache` | `TTSAudioCacheStore \| null \| undefined` | no | Where to keep synthesized audio for reuse (e.g. a shared `TTSAudioCache`). Text whose audio is in the cache is served from it, with no request to the provider. Lines get there through `warmAudioCache()`, and also from live synthesis when `cacheSynthesizedAudio` allows it. Only services that yield their audio from `runTts` (HTTP and streaming-RPC services, see `supportsAudioCache`) use it, and only in sentence aggregation mode. Defaults to null (no cache). pipedog only. |
| `cacheSynthesizedAudio` | `boolean \| ((text: string) => boolean) \| undefined` | no | Whether audio synthesized live is stored in `audioCache`, or a function that decides per text. Defaults to false: only warmed lines are stored, since a generated reply can repeat what the user said. pipedog only. |
| `audioCacheNamespace` | `string \| undefined` | no | Extra part of the cache key, for constructor options that change the audio but are not in `settings`. pipedog only. |
| `apiKey` | `string` | yes | Azure Cognitive Services subscription key. |
| `region` | `string \| null \| undefined` | no | Azure region identifier (e.g., "eastus", "westus2"). Required unless `privateEndpoint` is provided. |
| `privateEndpoint` | `string \| null \| undefined` | no | Custom endpoint URL for Azure Speech Services (e.g., "https://my-resource.cognitiveservices.azure.com/"). Use this when connecting via Private Link or a custom domain. See https://learn.microsoft.com/en-us/azure/ai-services/speech-service/speech-services-private-link |
| `settings` | `AzureTTSSettings \| null \| undefined` | no | Runtime-updatable settings. |
| `fetch` | `{ (input: RequestInfo \| URL, init?: RequestInit): Promise<Response>; (input: string \| URL \| Request, init?: RequestInit): Promise<Response>; } \| null \| undefined` | no | `fetch` used for HTTP requests (the REST synthesis, and the custom-domain redirect lookup). Defaults to the global `fetch`. Not in upstream. |

### Usage [#usage-1]

```ts title="http.ts"
const httpTts = new AzureHttpTTSService({
  apiKey: process.env.AZURE_SPEECH_API_KEY ?? "",
  region: process.env.AZURE_SPEECH_REGION ?? "eastus",
  sampleRate: 24000,
});
```

### Notes [#notes-1]

* Posts the SSML to `https://<region>.tts.speech.microsoft.com/cognitiveservices/v1` (or
  `<privateEndpoint>/tts/cognitiveservices/v1`) and yields the sentence as one audio frame.
* It requests a `raw-*-16bit-mono-pcm` format, which has no WAV header. Upstream strips the 44
  header bytes of the SDK's RIFF result instead.
* Pass `fetch` to send the request through your own client.

## Runtime notes [#runtime-notes]

| Runtime            | Works | What it needs |
| ------------------ | ----- | ------------- |
| Bun                | ✅ yes | Nothing.      |
| Node.js            | ✅ yes | Nothing.      |
| Deno               | ✅ yes | Nothing.      |
| Cloudflare Workers | ✅ yes | Nothing.      |

Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/azure/package.json`: ✅ means the compat smoke test passes on that runtime.

Both services work on Bun, Node, Deno and Cloudflare Workers: `AzureTTSService` uses core's
`websocketConnect`, `AzureHttpTTSService` uses `fetch`. The key is sent as the
`Ocp-Apim-Subscription-Key` header, and `AzureTTSService` also sends it as a query parameter, as
the Speech SDK does, for runtimes whose `WebSocket` can't send headers. With a custom-domain
`privateEndpoint`, `AzureTTSService` first asks the endpoint for its regional redirect, as the SDK
does.

The deprecated `voice`, `params` (`InputParams`) and `aggregateSentences` options aren't ported;
use `settings` and `textAggregationMode`.