Google

OverviewCopy Anchor Link

pipedog ports both of Pipecat's Google speech-to-text services. They use different APIs and different credentials:

• GeminiSTTService: streams audio to a Gemini Live transcription model (gemini-3.5-transcribe-live by default) over the Gemini Live API. Authenticates with a Gemini API key, or with Google Cloud credentials on Vertex AI. Supports automatic language detection, language hints, adaptation phrases and verbatim or smart transcription.
• GoogleSTTService: Google Cloud Speech-to-Text V2 streaming recognition. Authenticates with a Google Cloud service account or Application Default Credentials (ADC). Supports multiple languages, phrase-set adaptation, denoising and automatic stream renewal.

InstallationCopy Anchor Link

bun add @pipedog/google

GoogleSTTService also needs the Cloud Speech Node SDK, an optional peer dependency:

bun add @google-cloud/speech

import { GeminiSTTService } from "@pipedog/google/gemini_live/stt";
import { GoogleSTTService } from "@pipedog/google/stt";

PrerequisitesCopy Anchor Link

• GeminiSTTService: a Gemini API key from Google AI Studio, passed as apiKey. The examples read it from GOOGLE_API_KEY. On Vertex AI (vertexai: true), a Google Cloud project with the Vertex AI API enabled, and a service account key or ADC instead.
• GoogleSTTService: a Google Cloud project with the Speech-to-Text API enabled, and a service account key (as a file or a JSON string) or ADC. The examples read GOOGLE_APPLICATION_CREDENTIALS (a key file path) or GOOGLE_CREDENTIALS_JSON (the key itself). The project ID comes from the credentials.

GeminiSTTServiceCopy Anchor Link

ConfigurationCopy Anchor Link

SettingsCopy Anchor Link

Runtime-updatable settings, passed as settings: new GeminiSTTService.Settings({...}).

UsageCopy Anchor Link

import { Language } from "@pipedog/core/transcriptions/language";
import { GeminiSTTService } from "@pipedog/google/gemini_live/stt";

const stt = new GeminiSTTService({
  apiKey: process.env.GOOGLE_API_KEY ?? "",
});

stt.addEventHandler("on_connected", async () => {
  console.log("connected to the Gemini Live API");
});

With language hints, adaptation phrases and smart transcription:

const hinted = new GeminiSTTService({
  apiKey: process.env.GOOGLE_API_KEY ?? "",
  settings: new GeminiSTTService.Settings({
    languages: [Language.ES_ES, Language.EN_US],
    adaptationPhrases: ["pipedog", "oatmilk"],
    mode: "SMART",
  }),
});

Vertex AICopy Anchor Link

vertexai: true (pipedog only) connects through Vertex AI instead of the Gemini Developer API. It takes location (required, e.g. "eu" for data residency), projectId (defaults to GOOGLE_CLOUD_PROJECT), and credentials (a service account JSON string) or credentialsPath, falling back to Application Default Credentials, as the Vertex LLM services do. apiKey is rejected in this mode.

const vertex = new GeminiSTTService({
  vertexai: true,
  location: "eu",
  projectId: process.env.GOOGLE_CLOUD_PROJECT ?? "",
  // credentials: "<service account JSON>", or credentialsPath; otherwise ADC.
  reportDetectedLanguage: false,
  settings: new GeminiSTTService.Settings({ languages: [Language.ES_ES] }),
});

Vertex's Live API rejects the languageAuto and languageHints fields, so in this mode the language codes are sent as inputAudioTranscription.languageCodes, and no codes means automatic detection. languageAuto: false has no Vertex equivalent and is ignored.

Reported languageCopy Anchor Link

Each transcription frame carries the language the model reports. That value can be wrong, especially on short utterances. With reportDetectedLanguage: false (pipedog only), frames carry the configured language when exactly one is configured, and null otherwise.

Changing languages during a sessionCopy Anchor Link

setLanguageCodes(codes) (pipedog only) sets the language codes to transcribe, for example when the bot switches language. The codes are sent as-is and take precedence over the language and languages settings until one of those is updated. An empty list means automatic detection. Pass a function (() => string[]) to have the codes read each time they are needed.

// For example from a function the LLM calls when the conversation changes language.
// The session reconnects with the new codes when the bot next starts speaking.
await vertex.setLanguageCodes(["en-US"]);

The session does not reconnect at once. When the codes differ from the ones it was opened with, the service reconnects on the next BotStartedSpeakingFrame. If the user is mid-turn then, it waits until the turn ends, so no utterance is cut off. Audio that arrives while it reconnects is buffered and sent to the new session. GoogleSTTService has the same method.

Event handlersCopy Anchor Link

NotesCopy Anchor Link

• Emits InterimTranscriptionFrames as results arrive and a TranscriptionFrame for each finished utterance.
• On a VADUserStoppedSpeakingFrame, the service sends an audio-stream-end signal so the model finalizes the utterance at once. Without an upstream VAD, the model finalizes on its own schedule.
• Language hints and automatic detection are mutually exclusive. If both are set, hints win.
• Any settings change reconnects the session.
• The Live session is opened in the background when the StartFrame arrives, so the StartFrame (and the bot's first words) don't wait for it. Audio is buffered until the session is live. Upstream holds the StartFrame until the session is open; awaitConnectionOnStart: true (pipedog only) does the same.
• Audio is sent at the pipeline's input sample rate. The model works best with 16 kHz mono PCM.

GoogleSTTServiceCopy Anchor Link

ConfigurationCopy Anchor Link

SettingsCopy Anchor Link

Runtime-updatable settings, passed as settings: new GoogleSTTService.Settings({...}).

adaptation and denoiserConfig are plain objects. Keys may be snake_case, as Google documents them, or camelCase. A phrase_sets entry is a phrase set resource name or an inline phrase set. Referenced phrase sets must live in the same location as the service.

UsageCopy Anchor Link

import { STTUpdateSettingsFrame } from "@pipedog/core/frames/frames";
import { Language } from "@pipedog/core/transcriptions/language";
import { GoogleSTTService } from "@pipedog/google/stt";

const stt = new GoogleSTTService({
  credentialsPath: process.env.GOOGLE_APPLICATION_CREDENTIALS ?? "",
  location: "global",
  settings: new GoogleSTTService.Settings({
    model: "latest_long",
    languages: [Language.EN_US],
    enableInterimResults: true,
  }),
});

Speech adaptation:

// Bias recognition with a stored phrase set and an inline one.
const biased = new GoogleSTTService({
  credentials: process.env.GOOGLE_CREDENTIALS_JSON ?? "",
  settings: new GoogleSTTService.Settings({
    adaptation: {
      phrase_sets: [
        "projects/my-project/locations/global/phraseSets/catalog",
        { phrases: [{ value: "pipedog", boost: 15.0 }] },
      ],
    },
  }),
});

Change settings mid-call with an STTUpdateSettingsFrame:

// Changing settings reconnects the stream.
const update = new STTUpdateSettingsFrame({
  delta: new GoogleSTTService.Settings({ languages: [Language.ES_ES, Language.EN_US] }),
});

Event handlersCopy Anchor Link

NotesCopy Anchor Link

• Emits InterimTranscriptionFrames (with enableInterimResults) and TranscriptionFrames.
• Google limits a streaming call to a few minutes. The service renews the stream every 4 minutes (GoogleSTTService.STREAMING_LIMIT, in ms) and carries the unfinished audio over.
• A settings change reconnects the stream.
• The telephony model rejects adaptation; the service logs a warning and transcribes without it.
• With inline credentials, a missing project_id throws in the constructor.
• With languageCodes: ["auto"], transcripts carry the language Google detected.
• A gRPC CANCELLED error (the call was cancelled on our side) ends that stream quietly instead of being reported as an error.

Finalizing on VAD stopCopy Anchor Link

Google sends a final result only once its own endpointing decides the utterance is over, about 1.5 s after speech ends. With finalizeOnVadStop: true (pipedog only), the service half-closes the request stream when a VADUserStoppedSpeakingFrame arrives, so Google returns the final for the audio it has right away (about 0.6 s), and opens a new stream for the next audio, which waits in the queue meanwhile. The finals of the closed stream are pushed as one finalized TranscriptionFrame, so the turn can end without waiting for the STT timeout. Put a VAD before the service.

const stt = new GoogleSTTService({
  location: "eu",
  finalizeOnVadStop: true,
  settings: new GoogleSTTService.Settings({ model: "chirp_3", languageCodes: ["auto"] }),
});

Switching languages mid-callCopy Anchor Link

setLanguageCodes(codes) (pipedog only, the same API as GeminiSTTService) changes the languageCodes without cutting off the user: when the codes differ from the current ones, the stream reconnects on the next BotStartedSpeakingFrame, or once the user's turn ends if they are speaking over the bot. Pass a function to have the codes read again each time the bot starts speaking; it stays in effect until the language, languages or languageCodes setting is updated.

await stt.setLanguageCodes(["es-ES", "en-US"]);
// or follow the bot's language:
await stt.setLanguageCodes(() => [currentLanguageCode()]);

Sharing a client and prewarmingCopy Anchor Link

With sharedClient: true (pipedog only), services with the same location, credentials, clientOptions and speechModule use one Speech client per process instead of one each, so a server running one pipeline per call opens one gRPC channel and fetches tokens once. prewarm() connects it ahead of time (it loads the SDK, resolves the project ID, opens the channel and fetches a token) and returns false with a warning instead of throwing if that fails; the first stream then connects as usual.

const sttOptions = { location: "eu", sharedClient: true };
await new GoogleSTTService(sttOptions).prewarm(); // at server start-up
// per call:
const stt = new GoogleSTTService({ ...sttOptions, settings: new GoogleSTTService.Settings({ model: "chirp_3" }) });

It is off by default, as upstream builds a client per service, because a shared client puts every concurrent stream of the process on one gRPC channel. Services never close their client, so one call ending doesn't affect the others.

Runtime notesCopy Anchor Link

Generated from pipecat.runtimes and pipecat.compat in packages/google/package.json: ✅ means the compat smoke test passes on that runtime.

GeminiSTTService uses GoogleGenAI from @google/genai, which opens the Live API over the global WebSocket and runs on Bun, Node, Deno and Cloudflare Workers. In Vertex AI mode it authenticates through google-auth-library, which only the SDK's Node build supports, so that mode runs on Bun, Node and Deno, not on Workers. The client is created on first connect, so a subclass that overrides _createClient() can read its own fields there.

Cloud Speech-to-Text V2 streaming is gRPC only, so GoogleSTTService loads the Node SDK @google-cloud/speech lazily on first connect. That needs a Node-compatible runtime, not Workers. To use another transport, pass a client that implements GoogleSTTClient (streamingRecognize() and, optionally, getProjectId()). clientOptions (pipedog only) is merged into the SDK's SpeechClient options.

The deprecated upstream InputParams class, the params option, setLanguages() and updateOptions() are not ported. GoogleSTTSettings.languageCodes, deprecated upstream, is kept for codes Language cannot express, such as "auto".