# Gemini Live on Vertex AI

> Real-time speech-to-speech with GeminiLiveVertexLLMService, the Gemini Live API through Vertex AI with service account or ADC authentication.

Source: https://polvallverdu.github.io/pipedog/docs/services/s2s/gemini-live-vertex/



## Overview [#overview]

`GeminiLiveVertexLLMService` runs Gemini Live through Vertex AI. It extends
[`GeminiLiveLLMService`](https://polvallverdu.github.io/pipedog/docs/services/s2s/gemini-live.md), so audio, transcripts, VAD, tools,
thinking and compression work the same way. Authentication, the endpoint and a few features
differ.

* **`GeminiLiveVertexLLMService`**: Gemini Live on Vertex AI, with a Google Cloud project, a
  region and service account credentials or Application Default Credentials (ADC).

<Cards>
  <Card title="Upstream page" href="https://docs.pipecat.ai/api-reference/server/services/s2s/gemini-live-vertex">
    Pipecat's documentation for the same service.
  </Card>

  <Card title="Source" href="https://github.com/polvallverdu/pipedog/tree/main/packages/google/src/gemini_live/vertex">
    `packages/google/src/gemini_live/vertex/llm.ts`.
  </Card>

  <Card title="Vertex AI Live API docs" href="https://cloud.google.com/vertex-ai/generative-ai/docs/live-api">
    The Live API on Vertex AI.
  </Card>
</Cards>

## Installation [#installation]

```bash
bun add @pipedog/google
```

```ts
import { GeminiLiveVertexLLMService } from "@pipedog/google/gemini_live/vertex/llm";
```

## Prerequisites [#prerequisites]

* A Google Cloud project with the Vertex AI API enabled.
* A service account with access to Vertex AI, as a key file or a JSON string, or ADC
  (`gcloud auth application-default login`, or the attached service account on GCP).
* The examples read `GOOGLE_CLOUD_PROJECT_ID`, `GOOGLE_CLOUD_LOCATION`, and
  `GOOGLE_APPLICATION_CREDENTIALS` (a key file path) or `GOOGLE_CREDENTIALS_JSON` (the key
  itself).

## GeminiLiveVertexLLMService [#geminilivevertexllmservice]

### Configuration [#configuration]

| Option | Type | Required | Description |
|---|---|---|---|
| `credentials` | `string \| null \| undefined` | no | JSON string of service account credentials. |
| `credentialsPath` | `string \| null \| undefined` | no | Path to the service account JSON file. |
| `location` | `string` | yes | GCP region for Vertex AI endpoint (e.g., "us-east4"). |
| `projectId` | `string` | yes | Google Cloud project ID. |
| `settings` | `GeminiLiveVertexLLMSettings \| null \| undefined` | no | Gemini Live LLM settings. |
| `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. |
| `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. |
| `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. |
| `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. |
| `runInParallel` | `boolean \| undefined` | no | Whether to run function calls in parallel or sequentially. Defaults to true. |
| `groupParallelTools` | `boolean \| undefined` | no | Whether to group parallel function calls so the LLM is triggered exactly once after all calls in the batch complete. When false, each function call result triggers the LLM independently as it arrives. Defaults to true. |
| `functionCallTimeoutSecs` | `number \| null \| undefined` | no | Optional timeout in seconds for function calls. A call that runs past it is cancelled: its handler is cancelled (it gets a `CancelledError`), the call is settled as cancelled, and inference runs so the LLM can report that it didn't complete. |
| `fileResolver` | `FileResolver \| null \| undefined` | no | Resolves file URLs in the context that this provider can't fetch itself (see `FileResolver`). Pass one configured with a `fileStorage` to resolve uploaded-file URLs (`new FileResolver({ fileStorage: runnerArgs.fileStorage })` when a runner hosts the bot), and share one instance across services that may consume the same files (e.g. services switched between mid-session) so downloads are reused. Defaults to a resolver that handles `data:` and `http(s)` URLs only. |
| `startAudioPaused` | `boolean \| undefined` | no | Whether to start with audio input paused. Defaults to false. |
| `startVideoPaused` | `boolean \| undefined` | no | Whether to start with video input paused. Defaults to false. |
| `systemInstruction` | `string \| null \| undefined` | no | System prompt for the model. Defaults to null. |
| `tools` | `Record<string, unknown>[] \| (FunctionSchema \| DirectFunction)[] \| ToolsSchema \| null \| undefined` | no | Tools available to the model: a `ToolsSchema`, a plain list of direct functions and/or `FunctionSchema` objects (handlers auto-register), or a list of provider-native tool dicts. Defaults to null. |
| `inferenceOnContextInitialization` | `boolean \| undefined` | no | Whether to generate a response when context is first set. Defaults to true. |
| `userAudioPrerollSecs` | `number \| null \| undefined` | no | In server-VAD-disabled (locally-driven turns) mode, how much recent audio to replay after activity_start so the speech onset isn't clipped. |
| `fileApiBaseUrl` | `string \| undefined` | no | Base URL for the Gemini File API. Defaults to the official endpoint. |
| `httpOptions` | `GoogleClientHttpOptions \| null \| undefined` | no | HTTP options for the client. |

### Settings [#settings]

`GeminiLiveVertexLLMService.Settings` has the same fields as
[`GeminiLiveLLMService.Settings`](https://polvallverdu.github.io/pipedog/docs/services/s2s/gemini-live/#settings), with the same
defaults except `model`, which is `"google/gemini-3.8-live"` (Vertex AI model identifiers carry
the `google/` publisher prefix).

### Usage [#usage]

```ts title="gemini-live-vertex.ts"
import { GeminiLiveVertexLLMService } from "@pipedog/google/gemini_live/vertex/llm";

const llm = new GeminiLiveVertexLLMService({
  credentialsPath: process.env.GOOGLE_APPLICATION_CREDENTIALS ?? "",
  projectId: process.env.GOOGLE_CLOUD_PROJECT_ID ?? "",
  location: process.env.GOOGLE_CLOUD_LOCATION ?? "us-east4",
  systemInstruction: "You are a friendly voice assistant.",
  settings: new GeminiLiveVertexLLMService.Settings({
    model: "google/gemini-3.8-live",
    voice: "Charon",
  }),
});
```

With the service account key as a JSON string:

```ts
// A service account key as a JSON string. Leave out both credentials options
// to use Application Default Credentials.
const fromJson = new GeminiLiveVertexLLMService({
  credentials: process.env.GOOGLE_CREDENTIALS_JSON ?? "",
  projectId: process.env.GOOGLE_CLOUD_PROJECT_ID ?? "",
  location: "us-east4",
});
```

### Authentication [#authentication]

The service picks credentials in this order:

1. `credentials`: a service account key as a JSON string. It is parsed in the constructor.
2. `credentialsPath`: the path to a key file, read by the client.
3. Neither: Application Default Credentials.

All three go to `@google/genai` as `googleAuthOptions` with the `cloud-platform` scope, and
google-auth-library inside the SDK loads the key file, finds ADC and refreshes the access token
as it expires.

<Callout type="info">
  Upstream loads the credentials and refreshes a token in the constructor, so missing ADC fails
  there with "No valid credentials provided.". Here those steps are asynchronous and happen on
  first connect, so missing credentials surface as a connection error.
</Callout>

### Methods [#methods]

`setAudioInputPaused()`, `setVideoInputPaused()`, `setModelModalities()` and `setLanguage()`
work as on [Gemini Live](https://polvallverdu.github.io/pipedog/docs/services/s2s/gemini-live/#methods).

### Notes [#notes]

* Passing `apiKey` throws: Vertex AI takes `credentials`, `credentialsPath` or ADC.
* `projectId` and `location` (a region such as `"us-east4"`) are required.
* The Gemini File API isn't available: `fileApi` throws `NotImplementedError`. Use Google Cloud
  Storage for files.
* Vertex's Live endpoint doesn't support `NON_BLOCKING` tool calls. Functions registered with
  `cancelOnInterruption: false` run synchronously (the conversation pauses while they run), and
  the first time one is used the service logs an error and pushes an error frame.
* `historyConfig` is not sent on Vertex AI.

## Runtime notes [#runtime-notes]

| Runtime            | Works | What it needs |
| ------------------ | ----- | ------------- |
| Bun                | ✅ yes | Nothing.      |
| Node.js            | ✅ yes | Nothing.      |
| Deno               | ✅ yes | Nothing.      |
| Cloudflare Workers | ✅ yes | Nothing.      |

Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/google/package.json`: ✅ means the compat smoke test passes on that runtime.

Authentication goes through google-auth-library, which only the Node build of `@google/genai`
supports. This service runs on Bun, Node and Deno, not on Cloudflare Workers. On Workers, use
[`GeminiLiveLLMService`](https://polvallverdu.github.io/pipedog/docs/services/s2s/gemini-live.md) with a Gemini API key.

The deprecated upstream `model`, `voiceId` and `params` constructor options are not ported;
use `settings`.