# Google
> Speech-to-text with GeminiSTTService (Gemini Live transcription models) and GoogleSTTService (Google Cloud Speech-to-Text V2).
Source: https://polvallverdu.github.io/pipedog/docs/services/stt/google/
## Overview [#overview]
pipedog ports both of Pipecat's Google speech-to-text services. They use different APIs and
different credentials:
* **`GeminiSTTService`**: streams audio to a Gemini Live transcription model
(`gemini-3.5-transcribe-live` by default) over the Gemini Live API. Authenticates with a
Gemini API key, or with Google Cloud credentials on Vertex AI. Supports automatic language
detection, language hints, adaptation phrases and verbatim or smart transcription.
* **`GoogleSTTService`**: Google Cloud Speech-to-Text V2 streaming recognition. Authenticates
with a Google Cloud service account or Application Default Credentials (ADC). Supports
multiple languages, phrase-set adaptation, denoising and automatic stream renewal.
Pipecat's documentation for the same services.
`packages/google/src/gemini_live/stt.ts` and `stt.ts`.
Models, languages and adaptation for Speech-to-Text V2.
## Installation [#installation]
```bash
bun add @pipedog/google
```
`GoogleSTTService` also needs the Cloud Speech Node SDK, an optional peer dependency:
```bash
bun add @google-cloud/speech
```
```ts
import { GeminiSTTService } from "@pipedog/google/gemini_live/stt";
import { GoogleSTTService } from "@pipedog/google/stt";
```
## Prerequisites [#prerequisites]
* **`GeminiSTTService`**: a Gemini API key from [Google AI Studio](https://aistudio.google.com/apikey),
passed as `apiKey`. The examples read it from `GOOGLE_API_KEY`. On Vertex AI
(`vertexai: true`), a Google Cloud project with the Vertex AI API enabled, and a service
account key or ADC instead.
* **`GoogleSTTService`**: a Google Cloud project with the Speech-to-Text API enabled, and a
service account key (as a file or a JSON string) or ADC. The examples read
`GOOGLE_APPLICATION_CREDENTIALS` (a key file path) or `GOOGLE_CREDENTIALS_JSON` (the key
itself). The project ID comes from the credentials.
## GeminiSTTService [#geministtservice]
### Configuration [#configuration]
| Option | Type | Required | Description |
|---|---|---|---|
| `apiKey` | `string \| undefined` | no | Google AI (Gemini) API key. Required unless `vertexai` is true. |
| `vertexai` | `boolean \| undefined` | no | Connect through Vertex AI instead of the Gemini Developer API, with Google Cloud credentials instead of an API key. Requires `location` (pipedog only). |
| `location` | `string \| undefined` | no | Vertex AI location, e.g. `"eu"` or `"us-central1"`. Required with `vertexai` (pipedog only). |
| `projectId` | `string \| null \| undefined` | no | Google Cloud project ID, with `vertexai`. Defaults to the `GOOGLE_CLOUD_PROJECT` environment variable (pipedog only). |
| `credentials` | `string \| null \| undefined` | no | JSON string of service account credentials, with `vertexai`. Without it (or `credentialsPath`), Application Default Credentials are used (pipedog only). |
| `credentialsPath` | `string \| null \| undefined` | no | Path to the service account JSON file, with `vertexai` (pipedog only). |
| `httpOptions` | `GoogleClientHttpOptions \| null \| undefined` | no | Optional HTTP options passed to the google-genai client. |
| `settings` | `GeminiSTTSettings \| null \| undefined` | no | Runtime-updatable settings. Defaults to the `gemini-3.5-transcribe-live` model with automatic language detection. |
| `reportDetectedLanguage` | `boolean \| undefined` | no | Put the language the model reports on transcription frames. Defaults to true. When false, frames carry the configured language if exactly one is configured, and `null` otherwise (pipedog only). |
| `awaitConnectionOnStart` | `boolean \| undefined` | no | Hold the pipeline's StartFrame until the Live session is open, as upstream does. Defaults to false: the session is opened in the background and audio is buffered until it is live (pipedog only). |
| `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. |
| `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. |
| `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. |
| `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. |
| `audioPassthrough` | `boolean \| undefined` | no | Whether to pass audio frames downstream after processing. Defaults to true. |
| `sampleRate` | `number \| null \| undefined` | no | The sample rate for audio input. If null, will be determined from the start frame. |
| `sttTtfbTimeout` | `number \| undefined` | no | Time in seconds to wait after VAD stop before reporting TTFB. This delay allows the final transcription to arrive. Defaults to 2.0. |
| `ttfsP99Latency` | `number \| null \| undefined` | no | P99 latency from speech end to final transcript in seconds. Broadcast via STTMetadataFrame at pipeline start. |
| `keepaliveTimeout` | `number \| null \| undefined` | no | Seconds of no audio before sending silence to keep the connection alive. null disables keepalive. |
| `keepaliveInterval` | `number \| undefined` | no | Seconds between idle checks when keepalive is enabled. |
### Settings [#settings]
Runtime-updatable settings, passed as `settings: new GeminiSTTService.Settings({...})`.
| Setting | Type | Default | Description |
| ------------------- | ------------------------------- | ------------------------------ | --------------------------------------------------------------------------------- |
| `model` | `string` | `"gemini-3.5-transcribe-live"` | Gemini Live transcription model. |
| `language` | `Language \| string \| null` | `null` | A single language hint. |
| `languages` | `(Language \| string)[]` | `[]` | Language hints. Take precedence over `language`. |
| `languageAuto` | `boolean \| null` | `null` | Automatic detection. `null` detects unless hints are given; `false` turns it off. |
| `adaptationPhrases` | `string[] \| null` | `null` | Phrases to bias recognition toward. |
| `mode` | `"VERBATIM" \| "SMART" \| null` | `null` (model: `"VERBATIM"`) | `"SMART"` removes disfluencies and applies light cleanup. |
### Usage [#usage]
```ts title="gemini-stt.ts"
import { Language } from "@pipedog/core/transcriptions/language";
import { GeminiSTTService } from "@pipedog/google/gemini_live/stt";
const stt = new GeminiSTTService({
apiKey: process.env.GOOGLE_API_KEY ?? "",
});
stt.addEventHandler("on_connected", async () => {
console.log("connected to the Gemini Live API");
});
```
With language hints, adaptation phrases and smart transcription:
```ts
const hinted = new GeminiSTTService({
apiKey: process.env.GOOGLE_API_KEY ?? "",
settings: new GeminiSTTService.Settings({
languages: [Language.ES_ES, Language.EN_US],
adaptationPhrases: ["pipedog", "oatmilk"],
mode: "SMART",
}),
});
```
### Vertex AI [#vertex-ai]
`vertexai: true` (pipedog only) connects through Vertex AI instead of the Gemini Developer API.
It takes `location` (required, e.g. `"eu"` for data residency), `projectId` (defaults to
`GOOGLE_CLOUD_PROJECT`), and `credentials` (a service account JSON string) or `credentialsPath`,
falling back to Application Default Credentials, as the Vertex LLM services do. `apiKey` is
rejected in this mode.
```ts
const vertex = new GeminiSTTService({
vertexai: true,
location: "eu",
projectId: process.env.GOOGLE_CLOUD_PROJECT ?? "",
// credentials: "", or credentialsPath; otherwise ADC.
reportDetectedLanguage: false,
settings: new GeminiSTTService.Settings({ languages: [Language.ES_ES] }),
});
```
Vertex's Live API rejects the `languageAuto` and `languageHints` fields, so in this mode the
language codes are sent as `inputAudioTranscription.languageCodes`, and no codes means automatic
detection. `languageAuto: false` has no Vertex equivalent and is ignored.
### Reported language [#reported-language]
Each transcription frame carries the language the model reports. That value can be wrong,
especially on short utterances. With `reportDetectedLanguage: false` (pipedog only), frames carry
the configured language when exactly one is configured, and `null` otherwise.
### Changing languages during a session [#changing-languages-during-a-session]
`setLanguageCodes(codes)` (pipedog only) sets the language codes to transcribe, for example when
the bot switches language. The codes are sent as-is and take precedence over the `language` and
`languages` settings until one of those is updated. An empty list means automatic detection. Pass
a function (`() => string[]`) to have the codes read each time they are needed.
```ts
// For example from a function the LLM calls when the conversation changes language.
// The session reconnects with the new codes when the bot next starts speaking.
await vertex.setLanguageCodes(["en-US"]);
```
The session does not reconnect at once. When the codes differ from the ones it was opened with,
the service reconnects on the next `BotStartedSpeakingFrame`. If the user is mid-turn then, it
waits until the turn ends, so no utterance is cut off. Audio that arrives while it reconnects is
buffered and sent to the new session. `GoogleSTTService` has the same method.
### Event handlers [#event-handlers]
| Event | Description |
| --------------------- | ------------------------------------------------------------------------------------------------------ |
| `on_connected` | Connected to the Gemini Live API. |
| `on_disconnected` | Disconnected from the Gemini Live API. |
| `on_connection_error` | The first connection failed, or three connection attempts failed in a row; the service stops retrying. |
### Notes [#notes]
* Emits `InterimTranscriptionFrame`s as results arrive and a `TranscriptionFrame` for each
finished utterance.
* On a `VADUserStoppedSpeakingFrame`, the service sends an audio-stream-end signal so the model
finalizes the utterance at once. Without an upstream VAD, the model finalizes on its own
schedule.
* Language hints and automatic detection are mutually exclusive. If both are set, hints win.
* Any settings change reconnects the session.
* The Live session is opened in the background when the `StartFrame` arrives, so the
`StartFrame` (and the bot's first words) don't wait for it. Audio is buffered until the session
is live. Upstream holds the `StartFrame` until the session is open; `awaitConnectionOnStart:
true` (pipedog only) does the same.
* Audio is sent at the pipeline's input sample rate. The model works best with 16 kHz mono PCM.
## GoogleSTTService [#googlesttservice]
### Configuration [#configuration-1]
| Option | Type | Required | Description |
|---|---|---|---|
| `credentials` | `string \| null \| undefined` | no | JSON string containing Google Cloud service account credentials. |
| `credentialsPath` | `string \| null \| undefined` | no | Path to service account credentials JSON file. |
| `location` | `string \| undefined` | no | Google Cloud location (e.g., "global", "us-central1"). |
| `settings` | `GoogleSTTSettings \| null \| undefined` | no | Runtime-updatable settings. |
| `client` | `GoogleSTTClient \| null \| undefined` | no | A Speech V2 client to use instead of `@google-cloud/speech`. When given, credentials are not read; the project ID comes from `credentials` or from the client's `getProjectId()`. |
| `clientOptions` | `Record \| null \| undefined` | no | Extra options for the `@google-cloud/speech` v2 `SpeechClient` (e.g. `apiEndpoint`, `port`, `sslCreds`, `fallback`), merged over the ones the service derives from `location` and the credentials. DIVERGENCE: pipedog-only. Upstream builds its client from `location` and the credentials alone. |
| `speechModule` | `GoogleSpeechModule \| null \| undefined` | no | The `@google-cloud/speech` module (`import * as speech from "@google-cloud/speech"`), instead of importing it by name on first connect. A bundler (`bun build`) cannot see an import by name, so a bundled app imports the SDK itself and passes it here. DIVERGENCE: pipedog-only. |
| `sharedClient` | `boolean \| undefined` | no | Use the process-wide Speech client for this location, credentials, `clientOptions` and `speechModule` instead of building one per service, so a server running one pipeline per call opens one gRPC channel and fetches tokens once. Pair it with `prewarm()` at start-up. Default false: each service has its own client (upstream's behavior), since one channel carries every concurrent stream of the process. Services never close clients. Ignored when `client` is given. DIVERGENCE: pipedog-only. |
| `finalizeOnVadStop` | `boolean \| undefined` | no | Finalize an utterance as soon as VAD reports the user stopped speaking. Google only sends a final result once its own endpointing decides the utterance is over (about 1.5 s after speech ends). With this option the service half-closes the request stream on `VADUserStoppedSpeakingFrame`, so Google returns the final for the audio it has right away (about 0.6 s), and opens a new stream for the next audio, which waits in the queue meanwhile. The finals of the closing stream are pushed as one finalized `TranscriptionFrame`. Requires a VAD upstream. Default false. DIVERGENCE: pipedog-only. |
| `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. |
| `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. |
| `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. |
| `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. |
| `audioPassthrough` | `boolean \| undefined` | no | Whether to pass audio frames downstream after processing. Defaults to true. |
| `sampleRate` | `number \| null \| undefined` | no | The sample rate for audio input. If null, will be determined from the start frame. |
| `sttTtfbTimeout` | `number \| undefined` | no | Time in seconds to wait after VAD stop before reporting TTFB. This delay allows the final transcription to arrive. Defaults to 2.0. |
| `ttfsP99Latency` | `number \| null \| undefined` | no | P99 latency from speech end to final transcript in seconds. Broadcast via STTMetadataFrame at pipeline start. |
| `keepaliveTimeout` | `number \| null \| undefined` | no | Seconds of no audio before sending silence to keep the connection alive. null disables keepalive. |
| `keepaliveInterval` | `number \| undefined` | no | Seconds between idle checks when keepalive is enabled. |
### Settings [#settings-1]
Runtime-updatable settings, passed as `settings: new GoogleSTTService.Settings({...})`.
| Setting | Type | Default | Description |
| ---------------------------------- | -------------------------- | ------------------ | ------------------------------------------------------------------------------------------------------------------------ |
| `model` | `string` | `"latest_long"` | Speech V2 model. |
| `languages` | `Language[]` | `[Language.EN_US]` | Recognition languages. |
| `languageCodes` | `string[] \| null` | `null` | Google language codes sent as-is, instead of `languages`, e.g. `["auto"]` for Chirp's language detection (pipedog only). |
| `language` | `Language \| null` | `null` | Setting it replaces `languages` (and `languageCodes`) with that one language. |
| `useSeparateRecognitionPerChannel` | `boolean` | `false` | Recognize each channel separately. |
| `enableAutomaticPunctuation` | `boolean` | `true` | Add punctuation. |
| `enableSpokenPunctuation` | `boolean` | `false` | Transcribe spoken punctuation. |
| `enableSpokenEmojis` | `boolean` | `false` | Transcribe spoken emojis. |
| `profanityFilter` | `boolean` | `false` | Filter profanity. |
| `enableWordTimeOffsets` | `boolean` | `false` | Word timings. |
| `enableWordConfidence` | `boolean` | `false` | Word confidence scores. |
| `enableInterimResults` | `boolean` | `true` | Emit interim results. |
| `enableVoiceActivityEvents` | `boolean` | `false` | Voice activity events. |
| `adaptation` | `SpeechAdaptation \| null` | `null` | Phrase sets that bias recognition. |
| `denoiserConfig` | `DenoiserConfig \| null` | `null` | Background-noise removal (Chirp 3 models only), e.g. `{ denoise_audio: true }`. |
`adaptation` and `denoiserConfig` are plain objects. Keys may be snake_case, as Google documents
them, or camelCase. A `phrase_sets` entry is a phrase set resource name or an inline phrase set.
Referenced phrase sets must live in the same location as the service.
### Usage [#usage-1]
```ts title="google-stt.ts"
import { STTUpdateSettingsFrame } from "@pipedog/core/frames/frames";
import { Language } from "@pipedog/core/transcriptions/language";
import { GoogleSTTService } from "@pipedog/google/stt";
const stt = new GoogleSTTService({
credentialsPath: process.env.GOOGLE_APPLICATION_CREDENTIALS ?? "",
location: "global",
settings: new GoogleSTTService.Settings({
model: "latest_long",
languages: [Language.EN_US],
enableInterimResults: true,
}),
});
```
Speech adaptation:
```ts
// Bias recognition with a stored phrase set and an inline one.
const biased = new GoogleSTTService({
credentials: process.env.GOOGLE_CREDENTIALS_JSON ?? "",
settings: new GoogleSTTService.Settings({
adaptation: {
phrase_sets: [
"projects/my-project/locations/global/phraseSets/catalog",
{ phrases: [{ value: "pipedog", boost: 15.0 }] },
],
},
}),
});
```
Change settings mid-call with an `STTUpdateSettingsFrame`:
```ts
// Changing settings reconnects the stream.
const update = new STTUpdateSettingsFrame({
delta: new GoogleSTTService.Settings({ languages: [Language.ES_ES, Language.EN_US] }),
});
```
### Event handlers [#event-handlers-1]
| Event | Description |
| ----------------- | ------------------------- |
| `on_connected` | The service connected. |
| `on_disconnected` | The service disconnected. |
### Notes [#notes-1]
* Emits `InterimTranscriptionFrame`s (with `enableInterimResults`) and `TranscriptionFrame`s.
* Google limits a streaming call to a few minutes. The service renews the stream every 4
minutes (`GoogleSTTService.STREAMING_LIMIT`, in ms) and carries the unfinished audio over.
* A settings change reconnects the stream.
* The `telephony` model rejects adaptation; the service logs a warning and transcribes without
it.
* With inline `credentials`, a missing `project_id` throws in the constructor.
* With `languageCodes: ["auto"]`, transcripts carry the language Google detected.
* A gRPC CANCELLED error (the call was cancelled on our side) ends that stream quietly instead
of being reported as an error.
### Finalizing on VAD stop [#finalizing-on-vad-stop]
Google sends a final result only once its own endpointing decides the utterance is over, about
1.5 s after speech ends. With `finalizeOnVadStop: true` (pipedog only), the service half-closes
the request stream when a `VADUserStoppedSpeakingFrame` arrives, so Google returns the final for
the audio it has right away (about 0.6 s), and opens a new stream for the next audio, which waits
in the queue meanwhile. The finals of the closed stream are pushed as one finalized
`TranscriptionFrame`, so the turn can end without waiting for the STT timeout. Put a VAD before
the service.
```ts
const stt = new GoogleSTTService({
location: "eu",
finalizeOnVadStop: true,
settings: new GoogleSTTService.Settings({ model: "chirp_3", languageCodes: ["auto"] }),
});
```
### Switching languages mid-call [#switching-languages-mid-call]
`setLanguageCodes(codes)` (pipedog only, the same API as `GeminiSTTService`) changes the
`languageCodes` without cutting off the user: when the codes differ from the current ones, the
stream reconnects on the next `BotStartedSpeakingFrame`, or once the user's turn ends if they are
speaking over the bot. Pass a function to have the codes read again each time the bot starts
speaking; it stays in effect until the `language`, `languages` or `languageCodes` setting is
updated.
```ts
await stt.setLanguageCodes(["es-ES", "en-US"]);
// or follow the bot's language:
await stt.setLanguageCodes(() => [currentLanguageCode()]);
```
### Sharing a client and prewarming [#sharing-a-client-and-prewarming]
With `sharedClient: true` (pipedog only), services with the same `location`, credentials,
`clientOptions` and `speechModule` use one Speech client per process instead of one each, so a
server running one pipeline per call opens one gRPC channel and fetches tokens once.
`prewarm()` connects it ahead of time (it loads the SDK, resolves the project ID, opens the
channel and fetches a token) and returns `false` with a warning instead of throwing if that
fails; the first stream then connects as usual.
```ts
const sttOptions = { location: "eu", sharedClient: true };
await new GoogleSTTService(sttOptions).prewarm(); // at server start-up
// per call:
const stt = new GoogleSTTService({ ...sttOptions, settings: new GoogleSTTService.Settings({ model: "chirp_3" }) });
```
It is off by default, as upstream builds a client per service, because a shared client puts
every concurrent stream of the process on one gRPC channel. Services never close their client,
so one call ending doesn't affect the others.
## Runtime notes [#runtime-notes]
| Runtime | Works | What it needs |
| ------------------ | ----- | ------------- |
| Bun | ✅ yes | Nothing. |
| Node.js | ✅ yes | Nothing. |
| Deno | ✅ yes | Nothing. |
| Cloudflare Workers | ✅ yes | Nothing. |
Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/google/package.json`: ✅ means the compat smoke test passes on that runtime.
`GeminiSTTService` uses `GoogleGenAI` from `@google/genai`, which opens the Live API over the
global `WebSocket` and runs on Bun, Node, Deno and Cloudflare Workers. In Vertex AI mode it
authenticates through google-auth-library, which only the SDK's Node build supports, so that mode
runs on Bun, Node and Deno, not on Workers. The client is created on first connect, so a subclass
that overrides `_createClient()` can read its own fields there.
Cloud Speech-to-Text V2 streaming is gRPC only, so `GoogleSTTService` loads the Node SDK
`@google-cloud/speech` lazily on first connect. That needs a Node-compatible runtime, not
Workers. To use another transport, pass a `client` that implements `GoogleSTTClient`
(`streamingRecognize()` and, optionally, `getProjectId()`). `clientOptions` (pipedog only) is
merged into the SDK's `SpeechClient` options.
Upstream resolves credentials and the project ID in the constructor. Here, with
`credentialsPath` or ADC, they are resolved on first connect (in `setup()`), so "No valid
credentials provided." and "Project ID not found in credentials" are raised there. Inline
`credentials` are still checked in the constructor.
The deprecated upstream `InputParams` class, the `params` option, `setLanguages()` and
`updateOptions()` are not ported. `GoogleSTTSettings.languageCodes`, deprecated upstream, is kept
for codes `Language` cannot express, such as `"auto"`.