# Smallest AI > Realtime text-to-speech with SmallestTTSService, over Smallest AI's WebSocket API, with word timestamps. Source: https://polvallverdu.github.io/pipedog/docs/services/tts/smallest/ ## Overview [#overview] * **`SmallestTTSService`**: streams text over a WebSocket and pushes audio as it arrives. Fragments of one LLM turn share a continuation context, so the voice keeps its prosody across them. An interruption reconnects the socket. Pipecat's documentation for the same service. `packages/smallest/src/tts.ts`. The text-to-speech API reference. ## Installation [#installation] ```bash bun add @pipedog/smallest ``` ```ts import { SmallestTTSService } from "@pipedog/smallest/tts"; ``` ## Prerequisites [#prerequisites] * A Smallest AI account and an API key from [Smallest AI](https://smallest.ai/). * Pass the key as `apiKey`. The examples read it from `SMALLEST_API_KEY`. ## SmallestTTSService [#smallestttsservice] Connects to `wss://api.smallest.ai`, with the key as a Bearer token on the handshake. ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `textAggregationMode` | `TextAggregationMode \| null \| undefined` | no | How to aggregate incoming text before synthesis. `TextAggregationMode.SENTENCE` (default) buffers until sentence boundaries, `TextAggregationMode.TOKEN` streams tokens directly for lower latency. | | `apiKey` | `string` | yes | Smallest AI API key for authentication. | | `baseUrl` | `string \| undefined` | no | Base WebSocket URL for the Smallest API. Defaults to "wss://api.smallest.ai". | | `outputFormat` | `string \| undefined` | no | Audio format returned by the API. One of `pcm`, `mp3`, `wav`, `ulaw`, `alaw`. Defaults to `pcm`, which is what the pipeline expects internally. Fixed at init time. | | `wordTimestamps` | `boolean \| undefined` | no | Whether to request per-word timing events, enabled by default. When true, the server interleaves `word_timestamp` messages and the service emits aligned per-word `TTSTextFrame`s. Supported on base-queue English + Hindi voices (`meher`, `devansh`, `kartik`, `maithili`, `liam`, `avery`); other voices silently emit no word events, so leaving this on is safe regardless of voice. Fixed at init time because it determines whether text frames are produced from word timing or pushed whole. | | `maxBufferDelayMs` | `number \| null \| undefined` | no | Upper bound, in milliseconds (0-5000), on how long the server buffers a continuation fragment before speaking it. If null, the server's default (3000ms) applies. | | `settings` | `SmallestTTSSettings \| null \| undefined` | no | Runtime-updatable settings for the TTS service. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `reconnectBackoffMinWait` | `number \| undefined` | no | Minimum time, in seconds, to wait between reconnection attempts. | | `reconnectBackoffMaxWait` | `number \| undefined` | no | Maximum time, in seconds, to wait between reconnection attempts. | | `reconnectOnError` | `boolean \| undefined` | no | Whether to automatically reconnect on connection errors. | | `wsCloseTimeout` | `number \| undefined` | no | Maximum time, in seconds, to wait for the peer to acknowledge the websocket closing handshake before dropping the connection. Applied to connections opened through `_websocketConnect()`. Increase it for peers that need longer to complete a graceful close. | | `wsSendTimeout` | `number \| null \| undefined` | no | Maximum time, in seconds, a single send may wait on a peer that has stopped reading before the connection is dropped and the send throws `ConnectionClosedError`. Applied to connections opened through `_websocketConnect()`. Pass `null` to wait indefinitely. Defaults to `WS_SEND_TIMEOUT`. | | `pushTextFrames` | `boolean \| undefined` | no | Whether to push TextFrames and LLMFullResponseEndFrames. | | `pushStopFrames` | `boolean \| undefined` | no | Whether to automatically push TTSStoppedFrames. | | `pushStartFrame` | `boolean \| undefined` | no | Whether to automatically create audio contexts and push TTSStartedFrames. When true, the base class handles `createAudioContext` and yields `TTSStartedFrame` before each synthesis call, so `runTts` implementations do not need to. | | `stopFrameTimeoutS` | `number \| undefined` | no | Idle time before pushing TTSStoppedFrame when pushStopFrames is true. | | `pushSilenceAfterStop` | `boolean \| undefined` | no | Whether to push silence audio after TTSStoppedFrame. | | `silenceTimeS` | `number \| undefined` | no | Duration of silence to push when pushSilenceAfterStop is true. | | `pauseFrameProcessing` | `boolean \| undefined` | no | Whether to hold incoming frames after a turn until its audio contexts end. Needed by services whose audio doesn't carry its request's context ID. | | `maxConsecutiveZeroAudioContexts` | `number \| undefined` | no | How many consecutive TTS contexts may complete without producing any audio before the service is reported unable to do its job. Catches a provider that accepts requests and stays silent — an unknown voice ID, say — which no error ever surfaces. Every silent context reports an error the service can carry on from, whatever the limit is; on reaching the limit the service reports a permanent error instead, stops being given work, and the pipeline worker applies its `ProcessorUnusablePolicy`. Set to 0 to report silent contexts without ever writing the service off. | | `appendTrailingSpace` | `boolean \| undefined` | no | Whether to append a trailing space to text before sending to TTS. This helps prevent some TTS services from vocalizing trailing punctuation (e.g., "dot"). Only applied in sentence aggregation mode; when streaming tokens, the incoming text's own whitespace is preserved. | | `sampleRate` | `number \| null \| undefined` | no | Output sample rate for generated audio. | | `skipTextTypes` | `string[] \| null \| undefined` | no | List of text types that should not be spoken. | | `textTransforms` | `[string, TextTransform][] \| null \| undefined` | no | A list of TextTransforms to transform text just before sending it to TTS. Each TextTransform takes the aggregated text and its type, and returns the transformed text. To register, provide a list of tuples of `[textType \| "*", transformFunction]`. | | `transportDestination` | `string \| null \| undefined` | no | Destination for generated audio frames. | | `reuseContextIdWithinTurn` | `boolean \| undefined` | no | Whether the service should reuse context IDs within the same turn. | | `audioCache` | `TTSAudioCacheStore \| null \| undefined` | no | Where to keep synthesized audio for reuse (e.g. a shared `TTSAudioCache`). Text whose audio is in the cache is served from it, with no request to the provider. Lines get there through `warmAudioCache()`, and also from live synthesis when `cacheSynthesizedAudio` allows it. Only services that yield their audio from `runTts` (HTTP and streaming-RPC services, see `supportsAudioCache`) use it, and only in sentence aggregation mode. Defaults to null (no cache). pipedog only. | | `cacheSynthesizedAudio` | `boolean \| ((text: string) => boolean) \| undefined` | no | Whether audio synthesized live is stored in `audioCache`, or a function that decides per text. Defaults to false: only warmed lines are stored, since a generated reply can repeat what the user said. pipedog only. | | `audioCacheNamespace` | `string \| undefined` | no | Extra part of the cache key, for constructor options that change the audio but are not in `settings`. pipedog only. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new SmallestTTSService.Settings({...})` (`SmallestTTSSettings`). Any setting you leave out keeps the default below. | Setting | Type | Default | Description | | --------------- | -------------------------------------------------- | ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `model` | `"lightning_v3.1" \| "lightning_v3.1_pro"` | `"lightning_v3.1_pro"` | Speech model. | | `voice` | `string` | `"meher"` (`"sophia"` for `lightning_v3.1`) | Voice. | | `language` | `Language` | `Language.EN` | Language. | | `speed` | `number \| null` | `null` | Speed multiplier, 0.5 to 2.0. `null` uses the API default. | | `mathNotation` | `boolean \| null` | `null` | Read operators between digits as words ("2 + 2" as "two plus two"). | | `contentFilter` | `"reject" \| "flag" \| "redact" \| string \| null` | `null` | Check the text against a profanity list; `"reject"` fails the request, `"flag"` synthesizes and records the match, `"redact"` drops the matched term and speaks the rest. | | `dateFormat` | `"EU" \| "US" \| string \| null` | `null` | Read all-numeric dates day-first (`"EU"`) or month-first (`"US"`). | ### Usage [#usage] ```ts title="tts.ts" import { TTSUpdateSettingsFrame } from "@pipedog/core/frames/frames"; import { Language } from "@pipedog/core/transcriptions/language"; import { SmallestTTSService } from "@pipedog/smallest/tts"; const tts = new SmallestTTSService({ apiKey: process.env.SMALLEST_API_KEY ?? "", settings: new SmallestTTSService.Settings({ voice: "sophia", language: Language.EN }), }); ``` Change settings mid-call with a `TTSUpdateSettingsFrame`: ```ts // Queue it on the worker, or push it from a processor upstream of the service. const update = new TTSUpdateSettingsFrame({ delta: new SmallestTTSService.Settings({ speed: 1.1 }), }); ``` ### Notes [#notes] * `outputFormat` is `"pcm"` by default; it is fixed when the service is created. * `wordTimestamps` (on by default) makes the service push aligned per-word `TTSTextFrame`s. It works on `meher`, `devansh`, `kartik`, `maithili`, `liam` and `avery`; other voices send no word events, so leaving it on is safe. * `maxBufferDelayMs` (0 to 5000) bounds how long the server holds a continuation fragment. `null` keeps the server's 3000 ms. * The same package has Smallest AI's [LLM](https://polvallverdu.github.io/pipedog/docs/services/llm/smallest.md) and [speech-to-text](https://polvallverdu.github.io/pipedog/docs/services/stt/smallest.md) services. | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/smallest/package.json`: ✅ means the compat smoke test passes on that runtime. ## Runtime notes [#runtime-notes] * The WebSocket goes through the core `websocketConnect`, with headers on the handshake. * Tested against local WebSocket servers, as upstream's tests do, not against Smallest AI itself.