# Google > Text-to-speech with GoogleTTSService (Cloud TTS streaming), GoogleHttpTTSService (Cloud TTS with SSML) and GeminiTTSService (Gemini TTS models). Source: https://polvallverdu.github.io/pipedog/docs/services/tts/google/ ## Overview [#overview] pipedog ports Pipecat's three Google text-to-speech services: * **`GoogleTTSService`**: Google Cloud Text-to-Speech streaming synthesis, for low latency. Supports Chirp 3 HD and Journey voices, and Chirp 3 custom voices through a voice cloning key. * **`GoogleHttpTTSService`**: Cloud Text-to-Speech unary synthesis. Works with every Google voice (Standard, WaveNet, Neural2, ...) and builds SSML for pitch, rate, volume, emphasis, gender and style on voices that accept it. * **`GeminiTTSService`**: Gemini TTS models, with style prompts, expressive tags and multi-speaker output. Runs against Cloud Text-to-Speech (service account) or the Gemini API (API key). Pipecat's documentation for the same services. `packages/google/src/tts.ts`. Voices, SSML and Gemini TTS on Google Cloud. ## Installation [#installation] ```bash bun add @pipedog/google ``` The Cloud Text-to-Speech classes (and the Cloud backend of `GeminiTTSService`) also need the Node SDK, an optional peer dependency: ```bash bun add @google-cloud/text-to-speech ``` ```ts import { GeminiTTSService, GoogleHttpTTSService, GoogleTTSService } from "@pipedog/google/tts"; ``` ## Prerequisites [#prerequisites] * **Cloud Text-to-Speech**: a Google Cloud project with the Text-to-Speech API enabled, and a service account key (file or JSON string) or Application Default Credentials (ADC). The examples read `GOOGLE_APPLICATION_CREDENTIALS` (a key file path) or `GOOGLE_CREDENTIALS_JSON` (the key itself). * **Gemini API backend of `GeminiTTSService`**: a Gemini API key from [Google AI Studio](https://aistudio.google.com/apikey), read from `GOOGLE_API_KEY`. ## GoogleTTSService [#googlettsservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `credentials` | `string \| null \| undefined` | no | JSON string containing Google Cloud service account credentials. | | `credentialsPath` | `string \| null \| undefined` | no | Path to Google Cloud service account JSON file. | | `location` | `string \| null \| undefined` | no | Google Cloud location for regional endpoint (e.g., "us-central1"). | | `voiceCloningKey` | `string \| null \| undefined` | no | The voice cloning key for Chirp 3 custom voices. | | `clientOptions` | `Record \| null \| undefined` | no | Extra options for the `@google-cloud/text-to-speech` `TextToSpeechClient` (e.g. `apiEndpoint`, `port`, `sslCreds`), merged over the ones derived from `location` and the credentials. DIVERGENCE: pipedog-only. Upstream builds its client from `location` and the credentials alone. | | `textToSpeechModule` | `GoogleTextToSpeechModule \| null \| undefined` | no | The `@google-cloud/text-to-speech` module, for a bundled app: see `GoogleCloudTTSClientOptions.module`. | | `settings` | `GoogleTTSSettings \| null \| undefined` | no | Runtime-updatable settings. | | `client` | `GoogleCloudTTSClient \| null \| undefined` | no | A Cloud Text-to-Speech client to use instead of building one from the credentials, `location` and `clientOptions` (which are then ignored). Share one client (e.g. `createGoogleCloudTTSClient(...)`) across services to skip the per-service credential lookup, TLS handshake and token fetch on the first sentence. DIVERGENCE: pipedog-only. | | `sharedClient` | `boolean \| undefined` | no | Use the process-wide client for this endpoint, credentials and `clientOptions` (`getSharedGoogleCloudTTSClient`) instead of building one per service, so a server running one pipeline per call opens one gRPC channel and fetches tokens once. Pair it with `prewarm()` at start-up. Default false: each service has its own client (upstream's behavior), since one channel carries every concurrent stream of the process. Ignored when `client` is given. DIVERGENCE: pipedog-only. | | `bufferOnlyStartup` | `boolean \| undefined` | no | Buffer only the start of each utterance: once `chunkSize` bytes have arrived, audio is passed on as it is received instead of being re-cut into `chunkSize` pieces (which holds back up to one chunk throughout). Default false (upstream's behavior). DIVERGENCE: pipedog-only. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `textAggregationMode` | `TextAggregationMode \| null \| undefined` | no | How to aggregate incoming text before synthesis. `TextAggregationMode.SENTENCE` (default) buffers until sentence boundaries, `TextAggregationMode.TOKEN` streams tokens directly for lower latency. | | `pushTextFrames` | `boolean \| undefined` | no | Whether to push TextFrames and LLMFullResponseEndFrames. | | `pushStopFrames` | `boolean \| undefined` | no | Whether to automatically push TTSStoppedFrames. | | `pushStartFrame` | `boolean \| undefined` | no | Whether to automatically create audio contexts and push TTSStartedFrames. When true, the base class handles `createAudioContext` and yields `TTSStartedFrame` before each synthesis call, so `runTts` implementations do not need to. | | `stopFrameTimeoutS` | `number \| undefined` | no | Idle time before pushing TTSStoppedFrame when pushStopFrames is true. | | `pushSilenceAfterStop` | `boolean \| undefined` | no | Whether to push silence audio after TTSStoppedFrame. | | `silenceTimeS` | `number \| undefined` | no | Duration of silence to push when pushSilenceAfterStop is true. | | `pauseFrameProcessing` | `boolean \| undefined` | no | Whether to hold incoming frames after a turn until its audio contexts end. Needed by services whose audio doesn't carry its request's context ID. | | `maxConsecutiveZeroAudioContexts` | `number \| undefined` | no | How many consecutive TTS contexts may complete without producing any audio before the service is reported unable to do its job. Catches a provider that accepts requests and stays silent — an unknown voice ID, say — which no error ever surfaces. Every silent context reports an error the service can carry on from, whatever the limit is; on reaching the limit the service reports a permanent error instead, stops being given work, and the pipeline worker applies its `ProcessorUnusablePolicy`. Set to 0 to report silent contexts without ever writing the service off. | | `appendTrailingSpace` | `boolean \| undefined` | no | Whether to append a trailing space to text before sending to TTS. This helps prevent some TTS services from vocalizing trailing punctuation (e.g., "dot"). Only applied in sentence aggregation mode; when streaming tokens, the incoming text's own whitespace is preserved. | | `sampleRate` | `number \| null \| undefined` | no | Output sample rate for generated audio. | | `skipTextTypes` | `string[] \| null \| undefined` | no | List of text types that should not be spoken. | | `textTransforms` | `[string, TextTransform][] \| null \| undefined` | no | A list of TextTransforms to transform text just before sending it to TTS. Each TextTransform takes the aggregated text and its type, and returns the transformed text. To register, provide a list of tuples of `[textType \| "*", transformFunction]`. | | `transportDestination` | `string \| null \| undefined` | no | Destination for generated audio frames. | | `reuseContextIdWithinTurn` | `boolean \| undefined` | no | Whether the service should reuse context IDs within the same turn. | | `audioCache` | `TTSAudioCacheStore \| null \| undefined` | no | Where to keep synthesized audio for reuse (e.g. a shared `TTSAudioCache`). Text whose audio is in the cache is served from it, with no request to the provider. Lines get there through `warmAudioCache()`, and also from live synthesis when `cacheSynthesizedAudio` allows it. Only services that yield their audio from `runTts` (HTTP and streaming-RPC services, see `supportsAudioCache`) use it, and only in sentence aggregation mode. Defaults to null (no cache). pipedog only. | | `cacheSynthesizedAudio` | `boolean \| ((text: string) => boolean) \| undefined` | no | Whether audio synthesized live is stored in `audioCache`, or a function that decides per text. Defaults to false: only warmed lines are stored, since a generated reply can repeat what the user said. pipedog only. | | `audioCacheNamespace` | `string \| undefined` | no | Extra part of the cache key, for constructor options that change the audio but are not in `settings`. pipedog only. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new GoogleTTSService.Settings({...})`. | Setting | Type | Default | Description | | -------------- | -------------------- | -------------------------- | ---------------------------------------------------------------------------- | | `model` | `string \| null` | `null` | Not used by the streaming request. | | `voice` | `string` | `"en-US-Chirp3-HD-Charon"` | Chirp 3 HD or Journey voice name. | | `language` | `Language \| string` | `"en-US"` | Voice language. | | `speakingRate` | `number \| null` | `null` | Speaking rate, 0.25 to 2.0. Out-of-range updates are ignored with a warning. | ### Usage [#usage] ```ts title="tts.ts" import { TTSUpdateSettingsFrame } from "@pipedog/core/frames/frames"; import { Language } from "@pipedog/core/transcriptions/language"; import { GoogleHttpTTSService, GoogleTTSService } from "@pipedog/google/tts"; const tts = new GoogleTTSService({ credentialsPath: process.env.GOOGLE_APPLICATION_CREDENTIALS ?? "", settings: new GoogleTTSService.Settings({ voice: "en-US-Chirp3-HD-Charon", language: Language.EN_US, speakingRate: 1.1, }), }); ``` Change settings mid-call with a `TTSUpdateSettingsFrame`: ```ts const update = new TTSUpdateSettingsFrame({ delta: new GoogleTTSService.Settings({ voice: "en-US-Chirp3-HD-Aoede" }), }); ``` ### Notes [#notes] * With `voiceCloningKey`, the request uses that Chirp 3 custom voice instead of `voice`. * Audio is requested as PCM at the pipeline's output sample rate. ## GoogleHttpTTSService [#googlehttpttsservice] ### Configuration [#configuration-1] | Option | Type | Required | Description | |---|---|---|---| | `credentials` | `string \| null \| undefined` | no | JSON string containing Google Cloud service account credentials. | | `credentialsPath` | `string \| null \| undefined` | no | Path to Google Cloud service account JSON file. | | `location` | `string \| null \| undefined` | no | Google Cloud location for regional endpoint (e.g., "us-central1"). | | `clientOptions` | `Record \| null \| undefined` | no | Extra options for the `@google-cloud/text-to-speech` `TextToSpeechClient` (e.g. `apiEndpoint`, `port`, `sslCreds`), merged over the ones derived from `location` and the credentials. DIVERGENCE: pipedog-only. Upstream builds its client from `location` and the credentials alone. | | `textToSpeechModule` | `GoogleTextToSpeechModule \| null \| undefined` | no | The `@google-cloud/text-to-speech` module, for a bundled app: see `GoogleCloudTTSClientOptions.module`. | | `settings` | `GoogleHttpTTSSettings \| null \| undefined` | no | Runtime-updatable settings. | | `sharedClient` | `boolean \| undefined` | no | See `GoogleTTSServiceOptions.sharedClient`. DIVERGENCE: pipedog-only. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `textAggregationMode` | `TextAggregationMode \| null \| undefined` | no | How to aggregate incoming text before synthesis. `TextAggregationMode.SENTENCE` (default) buffers until sentence boundaries, `TextAggregationMode.TOKEN` streams tokens directly for lower latency. | | `pushTextFrames` | `boolean \| undefined` | no | Whether to push TextFrames and LLMFullResponseEndFrames. | | `pushStopFrames` | `boolean \| undefined` | no | Whether to automatically push TTSStoppedFrames. | | `pushStartFrame` | `boolean \| undefined` | no | Whether to automatically create audio contexts and push TTSStartedFrames. When true, the base class handles `createAudioContext` and yields `TTSStartedFrame` before each synthesis call, so `runTts` implementations do not need to. | | `stopFrameTimeoutS` | `number \| undefined` | no | Idle time before pushing TTSStoppedFrame when pushStopFrames is true. | | `pushSilenceAfterStop` | `boolean \| undefined` | no | Whether to push silence audio after TTSStoppedFrame. | | `silenceTimeS` | `number \| undefined` | no | Duration of silence to push when pushSilenceAfterStop is true. | | `pauseFrameProcessing` | `boolean \| undefined` | no | Whether to hold incoming frames after a turn until its audio contexts end. Needed by services whose audio doesn't carry its request's context ID. | | `maxConsecutiveZeroAudioContexts` | `number \| undefined` | no | How many consecutive TTS contexts may complete without producing any audio before the service is reported unable to do its job. Catches a provider that accepts requests and stays silent — an unknown voice ID, say — which no error ever surfaces. Every silent context reports an error the service can carry on from, whatever the limit is; on reaching the limit the service reports a permanent error instead, stops being given work, and the pipeline worker applies its `ProcessorUnusablePolicy`. Set to 0 to report silent contexts without ever writing the service off. | | `appendTrailingSpace` | `boolean \| undefined` | no | Whether to append a trailing space to text before sending to TTS. This helps prevent some TTS services from vocalizing trailing punctuation (e.g., "dot"). Only applied in sentence aggregation mode; when streaming tokens, the incoming text's own whitespace is preserved. | | `sampleRate` | `number \| null \| undefined` | no | Output sample rate for generated audio. | | `skipTextTypes` | `string[] \| null \| undefined` | no | List of text types that should not be spoken. | | `textTransforms` | `[string, TextTransform][] \| null \| undefined` | no | A list of TextTransforms to transform text just before sending it to TTS. Each TextTransform takes the aggregated text and its type, and returns the transformed text. To register, provide a list of tuples of `[textType \| "*", transformFunction]`. | | `transportDestination` | `string \| null \| undefined` | no | Destination for generated audio frames. | | `reuseContextIdWithinTurn` | `boolean \| undefined` | no | Whether the service should reuse context IDs within the same turn. | | `audioCache` | `TTSAudioCacheStore \| null \| undefined` | no | Where to keep synthesized audio for reuse (e.g. a shared `TTSAudioCache`). Text whose audio is in the cache is served from it, with no request to the provider. Lines get there through `warmAudioCache()`, and also from live synthesis when `cacheSynthesizedAudio` allows it. Only services that yield their audio from `runTts` (HTTP and streaming-RPC services, see `supportsAudioCache`) use it, and only in sentence aggregation mode. Defaults to null (no cache). pipedog only. | | `cacheSynthesizedAudio` | `boolean \| ((text: string) => boolean) \| undefined` | no | Whether audio synthesized live is stored in `audioCache`, or a function that decides per text. Defaults to false: only warmed lines are stored, since a generated reply can repeat what the user said. pipedog only. | | `audioCacheNamespace` | `string \| undefined` | no | Extra part of the cache key, for constructor options that change the audio but are not in `settings`. pipedog only. | ### Settings [#settings-1] | Setting | Type | Default | Description | | -------------- | ---------------------------------------------------------------------- | -------------------------- | -------------------------------------------------------------- | | `model` | `string \| null` | `null` | Not used by the request. | | `voice` | `string` | `"en-US-Chirp3-HD-Charon"` | Any Google voice name. | | `language` | `Language \| string` | `"en-US"` | Voice language. | | `pitch` | `string \| null` | `null` | SSML pitch, e.g. `"+2st"`, `"-50%"`. | | `rate` | `string \| null` | `null` | SSML prosody rate, e.g. `"slow"`, `"125%"` (non-Chirp voices). | | `speakingRate` | `number \| null` | `null` | Audio config rate, 0.25 to 2.0 (Chirp and Journey voices). | | `volume` | `string \| null` | `null` | SSML volume, e.g. `"loud"`, `"+6dB"`. | | `emphasis` | `"strong" \| "moderate" \| "reduced" \| "none" \| null` | `null` | SSML emphasis. | | `gender` | `"male" \| "female" \| "neutral" \| null` | `null` | Voice gender. | | `googleStyle` | `"apologetic" \| "calm" \| "empathetic" \| "firm" \| "lively" \| null` | `null` | Google voice style. | ### Usage [#usage-1] ```ts title="http.ts" // Any Google voice. Voices other than Chirp and Journey get SSML prosody. const httpTts = new GoogleHttpTTSService({ credentials: process.env.GOOGLE_CREDENTIALS_JSON ?? "", settings: new GoogleHttpTTSService.Settings({ voice: "en-US-Neural2-F", language: Language.EN_US, pitch: "+2st", rate: "110%", volume: "+2dB", }), }); ``` ### Notes [#notes-1] * Chirp and Journey voices (matched by name) don't accept SSML, so the service sends plain text and the SSML settings have no effect, including on the default voice. Use `speakingRate` for them. Other voices get SSML, with the text XML-escaped. ## GeminiTTSService [#geminittsservice] ### Configuration [#configuration-2] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string \| null \| undefined` | no | Google AI API key for authentication when using GenAI. | | `credentials` | `string \| null \| undefined` | no | JSON string containing Google Cloud service account credentials. | | `credentialsPath` | `string \| null \| undefined` | no | Path to Google Cloud service account JSON file. | | `location` | `string \| null \| undefined` | no | Google Cloud location for regional endpoint (e.g., "us-central1"). | | `clientOptions` | `Record \| null \| undefined` | no | Extra options for the `@google-cloud/text-to-speech` `TextToSpeechClient` (e.g. `apiEndpoint`, `port`, `sslCreds`), merged over the ones derived from `location` and the credentials. DIVERGENCE: pipedog-only. Upstream builds its client from `location` and the credentials alone. | | `textToSpeechModule` | `GoogleTextToSpeechModule \| null \| undefined` | no | The `@google-cloud/text-to-speech` module, for a bundled app: see `GoogleCloudTTSClientOptions.module`. | | `settings` | `GeminiTTSSettings \| null \| undefined` | no | Runtime-updatable settings. | | `useGenai` | `boolean \| null \| undefined` | no | Force use of the google-genai client if true. | | `httpOptions` | `HttpOptions \| null \| undefined` | no | HTTP client options for the google-genai client. | | `client` | `GoogleCloudTTSClient \| null \| undefined` | no | A Cloud Text-to-Speech client for the GCP backend, instead of building one from the credentials, `location` and `clientOptions`. See `GoogleTTSServiceOptions.client`. Ignored by the GenAI backend. DIVERGENCE: pipedog-only. | | `sharedClient` | `boolean \| undefined` | no | See `GoogleTTSServiceOptions.sharedClient` (GCP backend; the GenAI backend talks HTTP and ignores it). DIVERGENCE: pipedog-only. | | `bufferOnlyStartup` | `boolean \| undefined` | no | See `GoogleTTSServiceOptions.bufferOnlyStartup`. DIVERGENCE: pipedog-only. | | `retryOnBlocked` | `boolean \| GeminiTTSRetryOnBlockedOptions \| null \| undefined` | no | Retry a sentence that Google blocks for its usage guidelines instead of failing it: `true` for one retry, or `GeminiTTSRetryOnBlockedOptions`. Default off: a blocked sentence yields an `ErrorFrame` (upstream's behavior). DIVERGENCE: pipedog-only. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `textAggregationMode` | `TextAggregationMode \| null \| undefined` | no | How to aggregate incoming text before synthesis. `TextAggregationMode.SENTENCE` (default) buffers until sentence boundaries, `TextAggregationMode.TOKEN` streams tokens directly for lower latency. | | `pushTextFrames` | `boolean \| undefined` | no | Whether to push TextFrames and LLMFullResponseEndFrames. | | `pushStopFrames` | `boolean \| undefined` | no | Whether to automatically push TTSStoppedFrames. | | `pushStartFrame` | `boolean \| undefined` | no | Whether to automatically create audio contexts and push TTSStartedFrames. When true, the base class handles `createAudioContext` and yields `TTSStartedFrame` before each synthesis call, so `runTts` implementations do not need to. | | `stopFrameTimeoutS` | `number \| undefined` | no | Idle time before pushing TTSStoppedFrame when pushStopFrames is true. | | `pushSilenceAfterStop` | `boolean \| undefined` | no | Whether to push silence audio after TTSStoppedFrame. | | `silenceTimeS` | `number \| undefined` | no | Duration of silence to push when pushSilenceAfterStop is true. | | `pauseFrameProcessing` | `boolean \| undefined` | no | Whether to hold incoming frames after a turn until its audio contexts end. Needed by services whose audio doesn't carry its request's context ID. | | `maxConsecutiveZeroAudioContexts` | `number \| undefined` | no | How many consecutive TTS contexts may complete without producing any audio before the service is reported unable to do its job. Catches a provider that accepts requests and stays silent — an unknown voice ID, say — which no error ever surfaces. Every silent context reports an error the service can carry on from, whatever the limit is; on reaching the limit the service reports a permanent error instead, stops being given work, and the pipeline worker applies its `ProcessorUnusablePolicy`. Set to 0 to report silent contexts without ever writing the service off. | | `appendTrailingSpace` | `boolean \| undefined` | no | Whether to append a trailing space to text before sending to TTS. This helps prevent some TTS services from vocalizing trailing punctuation (e.g., "dot"). Only applied in sentence aggregation mode; when streaming tokens, the incoming text's own whitespace is preserved. | | `sampleRate` | `number \| null \| undefined` | no | Output sample rate for generated audio. | | `skipTextTypes` | `string[] \| null \| undefined` | no | List of text types that should not be spoken. | | `textTransforms` | `[string, TextTransform][] \| null \| undefined` | no | A list of TextTransforms to transform text just before sending it to TTS. Each TextTransform takes the aggregated text and its type, and returns the transformed text. To register, provide a list of tuples of `[textType \| "*", transformFunction]`. | | `transportDestination` | `string \| null \| undefined` | no | Destination for generated audio frames. | | `reuseContextIdWithinTurn` | `boolean \| undefined` | no | Whether the service should reuse context IDs within the same turn. | | `audioCache` | `TTSAudioCacheStore \| null \| undefined` | no | Where to keep synthesized audio for reuse (e.g. a shared `TTSAudioCache`). Text whose audio is in the cache is served from it, with no request to the provider. Lines get there through `warmAudioCache()`, and also from live synthesis when `cacheSynthesizedAudio` allows it. Only services that yield their audio from `runTts` (HTTP and streaming-RPC services, see `supportsAudioCache`) use it, and only in sentence aggregation mode. Defaults to null (no cache). pipedog only. | | `cacheSynthesizedAudio` | `boolean \| ((text: string) => boolean) \| undefined` | no | Whether audio synthesized live is stored in `audioCache`, or a function that decides per text. Defaults to false: only warmed lines are stored, since a generated reply can repeat what the user said. pipedog only. | | `audioCacheNamespace` | `string \| undefined` | no | Extra part of the cache key, for constructor options that change the audio but are not in `settings`. pipedog only. | ### Settings [#settings-2] | Setting | Type | Default | Description | | ---------------- | ------------------------------- | -------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | | `model` | `string` | `"gemini-3.1-flash-tts-preview"` | Gemini TTS model. This default works on both backends. | | `voice` | `string` | `"Kore"` | A Gemini voice (`GeminiTTSService.AVAILABLE_VOICES`) or a custom `voice_...` ID. | | `language` | `Language \| string` | `"en-US"` | Voice language. | | `prompt` | `string \| null` | `null` | Style instructions. See the backend notes below. | | `multiSpeaker` | `boolean` | `false` | Multi-speaker output (Cloud backend only). | | `speakerConfigs` | `GeminiSpeakerConfig[] \| null` | `null` | `{ speaker_alias, speaker_id? }` entries for multi-speaker mode. | | `speakingRate` | `number \| null` | `null` | Speaking rate, 0.25 to 2.0, sent in the streaming audio config (Cloud backend only, pipedog only). Out-of-range updates are ignored with a warning. | ### Usage [#usage-2] With the Gemini API backend: ```ts title="gemini-tts.ts" import { Language } from "@pipedog/core/transcriptions/language"; import { GeminiTTSService } from "@pipedog/google/tts"; // An apiKey selects the Gemini API (GenAI) backend. const tts = new GeminiTTSService({ apiKey: process.env.GOOGLE_API_KEY ?? "", settings: new GeminiTTSService.Settings({ model: "gemini-3.1-flash-tts-preview", voice: "Kore", language: Language.EN_US, }), }); ``` With the Cloud Text-to-Speech backend and a style prompt: ```ts // Service account credentials (and no apiKey) select Cloud Text-to-Speech. const styled = new GeminiTTSService({ credentialsPath: process.env.GOOGLE_APPLICATION_CREDENTIALS ?? "", settings: new GeminiTTSService.Settings({ voice: "Charon", prompt: "Say this in a calm, friendly tone.", }), }); ``` Multi-speaker (Cloud backend only): ```ts // Cloud backend only. Mark up the text with the speaker aliases. const dialogue = new GeminiTTSService({ credentialsPath: process.env.GOOGLE_APPLICATION_CREDENTIALS ?? "", settings: new GeminiTTSService.Settings({ multiSpeaker: true, speakerConfigs: [ { speaker_alias: "Host", speaker_id: "Kore" }, { speaker_alias: "Guest", speaker_id: "Charon" }, ], }), }); ``` ### Notes [#notes-2] * **Backend selection**: `useGenai` wins when set. Otherwise passing `apiKey` selects the Gemini API backend and leaving it out selects Cloud Text-to-Speech. With the Gemini API backend and no `apiKey`, the key falls back to `GOOGLE_API_KEY`. The env var alone never switches the backend. * **Gemini API backend**: no multi-speaker output (a warning is logged and one speaker is used). `prompt` works only with Gemini 3.8 TTS models, sent as a `speechMetadata.style` annotation; other models ignore it with a warning. * **Model names differ by backend.** Cloud Text-to-Speech takes `gemini-3.1-flash-tts-preview`, `gemini-2.5-flash-tts`, `gemini-2.5-flash-lite-preview-tts` and `gemini-2.5-pro-tts`. The Gemini API takes `gemini-3.8-flash-tts`, `gemini-3.8-flash-lite-tts`, `gemini-3.1-flash-tts-preview`, `gemini-2.5-flash-preview-tts` and `gemini-2.5-pro-preview-tts`. * Gemini 3.8 TTS models treat the text as a verbatim transcript. Put momentary vocal events inline as angle-bracket tags (``, ``, ``) and sustained style in `prompt`. * Gemini TTS outputs 24 kHz audio (`GeminiTTSService.GOOGLE_SAMPLE_RATE`). Any other sample rate logs a warning. * A voice that is neither in `AVAILABLE_VOICES` nor a `voice_...` ID logs a warning and is used anyway. ### Retrying blocked sentences [#retrying-blocked-sentences] Gemini TTS sometimes refuses a harmless sentence as a possible violation of Google's usage guidelines, at times after part of its audio has played, and the same request usually goes through on the next try. By default (as upstream) the sentence is lost and an `ErrorFrame` goes upstream. With `retryOnBlocked` (pipedog only), the service asks for the sentence again: ```ts const tts = new GeminiTTSService({ credentialsPath: process.env.GOOGLE_APPLICATION_CREDENTIALS, settings: new GeminiTTSSettings({ voice: "Kore", prompt: "Warm and calm." }), retryOnBlocked: { retries: 1, // the default; `retryOnBlocked: true` is this alone // Only when part of the sentence already played: it starts over with this text. restartText: (text) => `Sorry, ${text}`, // Still blocked after the retries: one last attempt with these settings. fallback: new GeminiTTSSettings({ prompt: null }), }, }); ``` * Only the usage-guidelines block is retried (`isGeminiTTSBlockedError()`): Cloud Text-to-Speech's INVALID_ARGUMENT that says so, the Gemini API's 400 that says so, or a Gemini API response blocked with no audio. Other errors fail the sentence as before. * Audio that already played can't be taken back, so a sentence blocked midway starts over from the beginning, as `restartText(text)` if given, instead of continuing from the middle. * `fallback` is either a `GeminiTTSSettings` delta used for that last attempt only (without the style prompt, or on the Cloud backend a Chirp 3 HD voice: `new GeminiTTSSettings({ model: null, prompt: null, voice: "en-US-Chirp3-HD-Kore" })`), or a function `(text, contextId) => AsyncIterable`, such as another TTS service's `runTts` at the same sample rate. * If every attempt is blocked, the sentence fails with one `ErrorFrame`, as without the option. Restarted and fallback audio is never stored in the audio cache. ## Sharing a client and streaming options [#sharing-a-client-and-streaming-options] These options are pipedog additions: * `sharedClient` (the three Cloud TTS services, and the Cloud backend of `GeminiTTSService`): use one client per process for the same endpoint (`location`), credentials and `clientOptions`, instead of one per service. Building a client costs a credential lookup, a TLS handshake and a token fetch on the first sentence; a server running one pipeline per call pays that once. Call `prewarm()` at start-up to pay it before the first call: it loads the SDK, opens the gRPC channel and fetches a token, and returns `false` (with a warning) instead of throwing if that fails, in which case the first sentence connects as usual. ```ts const ttsOptions = { location: "eu", sharedClient: true }; // At server start-up: a throwaway service warms the shared client. await new GeminiTTSService(ttsOptions).prewarm(); // Per call: every service uses that client. const tts = new GeminiTTSService({ ...ttsOptions, settings: new GeminiTTSSettings({ voice: "Kore" }) }); ``` It is off by default (each service builds its own client, as upstream) because a shared client puts every concurrent stream of the process on one gRPC channel. Services never close a client, so one call ending doesn't affect the others. `getSharedGoogleCloudTTSClient(options)` returns the same shared client. The Gemini API backend talks HTTP with an API key, so it has nothing to share or prewarm. * `client` (`GoogleTTSService` and the Cloud backend of `GeminiTTSService`): a `GoogleCloudTTSClient` to use instead of building one, for full control over sharing: ```ts const client = createGoogleCloudTTSClient({ apiEndpoint: "eu-texttospeech.googleapis.com" }); const tts = new GeminiTTSService({ client, settings: new GeminiTTSSettings({ voice: "Kore" }) }); ``` With `client`, the credentials, `location`, `clientOptions` and `sharedClient` are not used. * `bufferOnlyStartup` (`GoogleTTSService`, `GeminiTTSService`): upstream re-cuts the audio into `chunkSize` pieces (0.5 s) for the whole utterance, which holds back up to one piece throughout. With `bufferOnlyStartup: true`, only the start is buffered; after the first `chunkSize` bytes, audio goes on as it arrives (whole samples). Both streaming services treat a gRPC CANCELLED error, which is what an interrupted stream reports, as a quiet end of synthesis instead of an error. ## Pre-synthesized lines [#pre-synthesized-lines] All three services (and both backends of `GeminiTTSService`) work with the [TTS audio cache](https://polvallverdu.github.io/pipedog/docs/concepts/tts-audio-cache.md): give them an `audioCache` and call `warmAudioCache()` for fixed lines such as a greeting or a filler, and those sentences play from memory with no request to Google. ```ts const audioCache = new TTSAudioCache(); // one per process const tts = new GeminiTTSService({ location: "eu", audioCache }); // While an outbound call rings, before the pipeline starts: await tts.warmAudioCache("Hi! This is the clinic calling about your appointment.", { sampleRate: 24000, // the rate the service will run at }); ``` The cache key covers the text, the settings (model, voice, language, `prompt`, `speakingRate` and the rest), the sample rate, and what changes the audio outside the settings: the backend of `GeminiTTSService` and the `voiceCloningKey` of `GoogleTTSService` (hashed). A settings change therefore misses the cache instead of playing the old voice. Warming works before the pipeline starts, since the client is created with the service; pass `sampleRate` unless the service has one. ## Event handlers [#event-handlers] All three services inherit the `TTSService` events, for example `on_tts_request` (`contextId`, `text`), called for each text sent for synthesis. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/google/package.json`: ✅ means the compat smoke test passes on that runtime. Cloud Text-to-Speech streaming is gRPC only, so `GoogleTTSService`, `GoogleHttpTTSService` and the Cloud backend of `GeminiTTSService` use the Node SDK `@google-cloud/text-to-speech`. It is imported on the first request, so importing the module is safe everywhere, but these classes need Node or Bun. The SDK sits behind the `GoogleCloudTTSClient` interface; pass `client` (or override `_createClient()`) to supply another client. `clientOptions` (pipedog only) is merged into the SDK's `TextToSpeechClient` options. The Gemini API backend of `GeminiTTSService` uses `@google/genai` over `fetch` and runs on Bun, Node, Deno and Cloudflare Workers. `credentials` is parsed in the constructor, as upstream does. With `credentialsPath` or ADC, the SDK resolves credentials on the first request, so a missing credential surfaces as a synthesis error rather than as upstream's constructor error. The deprecated upstream `InputParams` classes, the `params`, `voiceId` and (Gemini) `model` constructor options, and the `GoogleStreamTTSSettings` alias are not ported.