Gemini Live on Vertex AI

OverviewCopy Anchor Link

GeminiLiveVertexLLMService runs Gemini Live through Vertex AI. It extends GeminiLiveLLMService, so audio, transcripts, VAD, tools, thinking and compression work the same way. Authentication, the endpoint and a few features differ.

• GeminiLiveVertexLLMService: Gemini Live on Vertex AI, with a Google Cloud project, a region and service account credentials or Application Default Credentials (ADC).

InstallationCopy Anchor Link

bun add @pipedog/google

import { GeminiLiveVertexLLMService } from "@pipedog/google/gemini_live/vertex/llm";

PrerequisitesCopy Anchor Link

• A Google Cloud project with the Vertex AI API enabled.
• A service account with access to Vertex AI, as a key file or a JSON string, or ADC (gcloud auth application-default login, or the attached service account on GCP).
• The examples read GOOGLE_CLOUD_PROJECT_ID, GOOGLE_CLOUD_LOCATION, and GOOGLE_APPLICATION_CREDENTIALS (a key file path) or GOOGLE_CREDENTIALS_JSON (the key itself).

GeminiLiveVertexLLMServiceCopy Anchor Link

ConfigurationCopy Anchor Link

SettingsCopy Anchor Link

GeminiLiveVertexLLMService.Settings has the same fields as GeminiLiveLLMService.Settings, with the same defaults except model, which is "google/gemini-3.8-live" (Vertex AI model identifiers carry the google/ publisher prefix).

UsageCopy Anchor Link

import { GeminiLiveVertexLLMService } from "@pipedog/google/gemini_live/vertex/llm";

const llm = new GeminiLiveVertexLLMService({
  credentialsPath: process.env.GOOGLE_APPLICATION_CREDENTIALS ?? "",
  projectId: process.env.GOOGLE_CLOUD_PROJECT_ID ?? "",
  location: process.env.GOOGLE_CLOUD_LOCATION ?? "us-east4",
  systemInstruction: "You are a friendly voice assistant.",
  settings: new GeminiLiveVertexLLMService.Settings({
    model: "google/gemini-3.8-live",
    voice: "Charon",
  }),
});

With the service account key as a JSON string:

// A service account key as a JSON string. Leave out both credentials options
// to use Application Default Credentials.
const fromJson = new GeminiLiveVertexLLMService({
  credentials: process.env.GOOGLE_CREDENTIALS_JSON ?? "",
  projectId: process.env.GOOGLE_CLOUD_PROJECT_ID ?? "",
  location: "us-east4",
});

AuthenticationCopy Anchor Link

The service picks credentials in this order:

1. credentials: a service account key as a JSON string. It is parsed in the constructor.
2. credentialsPath: the path to a key file, read by the client.
3. Neither: Application Default Credentials.

All three go to @google/genai as googleAuthOptions with the cloud-platform scope, and google-auth-library inside the SDK loads the key file, finds ADC and refreshes the access token as it expires.

MethodsCopy Anchor Link

setAudioInputPaused(), setVideoInputPaused(), setModelModalities() and setLanguage() work as on Gemini Live.

NotesCopy Anchor Link

• Passing apiKey throws: Vertex AI takes credentials, credentialsPath or ADC.
• projectId and location (a region such as "us-east4") are required.
• The Gemini File API isn't available: fileApi throws NotImplementedError. Use Google Cloud Storage for files.
• Vertex's Live endpoint doesn't support NON_BLOCKING tool calls. Functions registered with cancelOnInterruption: false run synchronously (the conversation pauses while they run), and the first time one is used the service logs an error and pushes an error frame.
• historyConfig is not sent on Vertex AI.

Runtime notesCopy Anchor Link

Generated from pipecat.runtimes and pipecat.compat in packages/google/package.json: ✅ means the compat smoke test passes on that runtime.

Authentication goes through google-auth-library, which only the Node build of @google/genai supports. This service runs on Bun, Node and Deno, not on Cloudflare Workers. On Workers, use GeminiLiveLLMService with a Gemini API key.

The deprecated upstream model, voiceId and params constructor options are not ported; use settings.