# Gemini Live

> Real-time speech-to-speech with GeminiLiveLLMService over the Gemini Live API, with server or local VAD, tools, thinking and the Gemini File API.

Source: https://polvallverdu.github.io/pipedog/docs/services/s2s/gemini-live/



## Overview [#overview]

`GeminiLiveLLMService` connects to the Gemini Live API over a WebSocket. It takes user audio
(and text and video), and returns spoken audio, transcripts and tool calls, so one service
replaces the STT, LLM and TTS stages of a pipeline.

* **`GeminiLiveLLMService`**: the Live session, with server-side or local VAD, function calling,
  thinking, context window compression, session resumption and Google Search grounding.
* **`GeminiVADParams`** and **`ContextWindowCompressionParams`**: settings objects for VAD and
  compression.
* **`GeminiFileAPI`**: a client for the Gemini File API (upload, get, list, delete), available
  as `llm.fileApi`.

For Vertex AI, see [Gemini Live on Vertex AI](https://polvallverdu.github.io/pipedog/docs/services/s2s/gemini-live-vertex.md).

<Cards>
  <Card title="Upstream page" href="https://docs.pipecat.ai/api-reference/server/services/s2s/gemini-live">
    Pipecat's documentation for the same service.
  </Card>

  <Card title="Source" href="https://github.com/polvallverdu/pipedog/tree/main/packages/google/src/gemini_live">
    `packages/google/src/gemini_live/llm.ts` and `file_api.ts`.
  </Card>

  <Card title="Gemini Live API docs" href="https://ai.google.dev/gemini-api/docs/live">
    Models, voices, VAD and session management.
  </Card>
</Cards>

## Installation [#installation]

```bash
bun add @pipedog/google
```

```ts
import {
  ContextWindowCompressionParams,
  GeminiLiveLLMService,
  GeminiVADParams,
} from "@pipedog/google/gemini_live/llm";
import { GeminiFileAPI } from "@pipedog/google/gemini_live/file_api";
```

## Prerequisites [#prerequisites]

* A Gemini API key from [Google AI Studio](https://aistudio.google.com/apikey).
* Pass the key as `apiKey`. The examples read it from `GOOGLE_API_KEY`.

## GeminiLiveLLMService [#geminilivellmservice]

### Configuration [#configuration]

| Option | Type | Required | Description |
|---|---|---|---|
| `apiKey` | `string` | yes | Google AI API key for authentication. |
| `startAudioPaused` | `boolean \| undefined` | no | Whether to start with audio input paused. Defaults to false. |
| `startVideoPaused` | `boolean \| undefined` | no | Whether to start with video input paused. Defaults to false. |
| `systemInstruction` | `string \| null \| undefined` | no | System prompt for the model. Defaults to null. |
| `tools` | `Record<string, unknown>[] \| (FunctionSchema \| DirectFunction)[] \| ToolsSchema \| null \| undefined` | no | Tools available to the model: a `ToolsSchema`, a plain list of direct functions and/or `FunctionSchema` objects (handlers auto-register), or a list of provider-native tool dicts. Defaults to null. |
| `settings` | `GeminiLiveLLMSettings \| null \| undefined` | no | Gemini Live LLM settings. |
| `inferenceOnContextInitialization` | `boolean \| undefined` | no | Whether to generate a response when context is first set. Defaults to true. |
| `userAudioPrerollSecs` | `number \| null \| undefined` | no | In server-VAD-disabled (locally-driven turns) mode, how much recent audio to replay after activity_start so the speech onset isn't clipped. |
| `fileApiBaseUrl` | `string \| undefined` | no | Base URL for the Gemini File API. Defaults to the official endpoint. |
| `httpOptions` | `GoogleClientHttpOptions \| null \| undefined` | no | HTTP options for the client. |
| `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. |
| `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. |
| `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. |
| `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. |
| `runInParallel` | `boolean \| undefined` | no | Whether to run function calls in parallel or sequentially. Defaults to true. |
| `groupParallelTools` | `boolean \| undefined` | no | Whether to group parallel function calls so the LLM is triggered exactly once after all calls in the batch complete. When false, each function call result triggers the LLM independently as it arrives. Defaults to true. |
| `functionCallTimeoutSecs` | `number \| null \| undefined` | no | Optional timeout in seconds for function calls. A call that runs past it is cancelled: its handler is cancelled (it gets a `CancelledError`), the call is settled as cancelled, and inference runs so the LLM can report that it didn't complete. |
| `fileResolver` | `FileResolver \| null \| undefined` | no | Resolves file URLs in the context that this provider can't fetch itself (see `FileResolver`). Pass one configured with a `fileStorage` to resolve uploaded-file URLs (`new FileResolver({ fileStorage: runnerArgs.fileStorage })` when a runner hosts the bot), and share one instance across services that may consume the same files (e.g. services switched between mid-session) so downloads are reused. Defaults to a resolver that handles `data:` and `http(s)` URLs only. |

### Settings [#settings]

Passed as `settings: new GeminiLiveLLMService.Settings({...})`. They are sent when the session
connects. Changing them later stores the new values but doesn't apply them to the open session
(a warning names the settings that didn't take effect).

| Setting                               | Type                             | Default                             | Description                                                                                |
| ------------------------------------- | -------------------------------- | ----------------------------------- | ------------------------------------------------------------------------------------------ |
| `model`                               | `string`                         | `"models/gemini-3.8-live"`          | Gemini Live model.                                                                         |
| `voice`                               | `string`                         | `"Charon"`                          | Voice name.                                                                                |
| `language`                            | `Language \| string`             | `"en-US"`                           | Language. A `Language` value is sent as Gemini's locale code (`Language.ES` as `"es-ES"`). |
| `modalities`                          | `GeminiModalities`               | `GeminiModalities.AUDIO`            | Response modality. `TEXT` logs a warning: recent models may not support it.                |
| `mediaResolution`                     | `GeminiMediaResolution`          | `GeminiMediaResolution.UNSPECIFIED` | Video input resolution.                                                                    |
| `vad`                                 | `GeminiVADParams \| null`        | `null` (server VAD on)              | Voice activity detection.                                                                  |
| `turnCoverage`                        | `TurnCoverage \| null`           | `null` (model default)              | Which realtime input a user turn covers.                                                   |
| `contextWindowCompression`            | `ContextWindowCompressionParams` | `{}` (off)                          | Context window compression.                                                                |
| `thinking`                            | `ThinkingConfig`                 | `{}`                                | Thinking configuration. Thinking models get `thinkingLevel: "LOW"` when none is set.       |
| `enableAffectiveDialog`               | `boolean`                        | `false`                             | Affective dialog.                                                                          |
| `proactivity`                         | `ProactivityConfig`              | `{}`                                | Proactivity configuration.                                                                 |
| `maxTokens`                           | `number \| null`                 | `4096`                              | Maximum output tokens.                                                                     |
| `temperature`, `topK`, `topP`, `seed` | `number \| null`                 | `null`                              | Sampling settings. Gemini treats `seed` as best effort.                                    |

`GeminiVADParams`:

| Option | Type | Required | Description |
|---|---|---|---|
| `disabled` | `boolean \| null \| undefined` | no | Whether to disable VAD. Defaults to null (server-side VAD is enabled). |
| `startSensitivity` | `StartSensitivity \| null \| undefined` | no | Sensitivity for speech start detection. Defaults to null. |
| `endSensitivity` | `EndSensitivity \| null \| undefined` | no | Sensitivity for speech end detection. Defaults to null. |
| `prefixPaddingMs` | `number \| null \| undefined` | no | Prefix padding in milliseconds. Defaults to null. |
| `silenceDurationMs` | `number \| null \| undefined` | no | Silence duration threshold in milliseconds. Defaults to null. |

`ContextWindowCompressionParams`:

| Option | Type | Required | Description |
|---|---|---|---|
| `enabled` | `boolean \| undefined` | no | Whether compression is enabled. Defaults to false. |
| `triggerTokens` | `number \| null \| undefined` | no | Token count to trigger compression. null uses 80% of context window. |

### Usage [#usage]

```ts title="gemini-live.ts"
import { LLMContext } from "@pipedog/core/processors/aggregators/llm_context";
import { Language } from "@pipedog/core/transcriptions/language";
import {
  ContextWindowCompressionParams,
  GeminiLiveLLMService,
  GeminiVADParams,
} from "@pipedog/google/gemini_live/llm";

const llm = new GeminiLiveLLMService({
  apiKey: process.env.GOOGLE_API_KEY ?? "",
  systemInstruction: "You are a friendly voice assistant. Keep answers short.",
  settings: new GeminiLiveLLMService.Settings({
    model: "models/gemini-3.8-live",
    voice: "Puck",
    language: Language.EN_US,
  }),
});
```

Compression and thinking:

```ts
const tuned = new GeminiLiveLLMService({
  apiKey: process.env.GOOGLE_API_KEY ?? "",
  settings: new GeminiLiveLLMService.Settings({
    temperature: 0.6,
    contextWindowCompression: new ContextWindowCompressionParams({
      enabled: true,
      triggerTokens: 8000,
    }),
    thinking: { includeThoughts: true },
  }),
});
```

Local VAD. Disable server-side VAD and configure a VAD analyzer on the user aggregator; the
service then sends activity start and end signals to Gemini and replays a short pre-roll of
audio (`userAudioPrerollSecs`) so the start of speech isn't clipped:

```ts
// Turn off server-side VAD. A local VAD in the user aggregator then drives
// turns, and the service sends activity start/end signals to Gemini.
const locallyDriven = new GeminiLiveLLMService({
  apiKey: process.env.GOOGLE_API_KEY ?? "",
  settings: new GeminiLiveLLMService.Settings({
    vad: new GeminiVADParams({ disabled: true }),
  }),
});
```

### Methods [#methods]

| Method                           | Description                                            |
| -------------------------------- | ------------------------------------------------------ |
| `setAudioInputPaused(paused)`    | Stop (`true`) or resume (`false`) sending input audio. |
| `setVideoInputPaused(paused)`    | Stop or resume sending video frames.                   |
| `setModelModalities(modalities)` | Set the response modality.                             |
| `setLanguage(language)`          | Set the language.                                      |
| `fileApi`                        | The `GeminiFileAPI` client, created on first access.   |

### Gemini File API [#gemini-file-api]

Upload a file, then reference its `uri` in a `file_data` content part of a context message:

```ts
// A path is read with node:fs/promises (Bun, Node, Deno). On any runtime,
// including Workers, pass the contents instead (bytes, a Blob or a stream):
// llm.fileApi.uploadFile(new TextEncoder().encode(menu), "menu.txt").
const uploaded = await llm.fileApi.uploadFile("menu.txt", "Menu");
const file = uploaded.file as { uri: string };

const context = new LLMContext({
  messages: [
    {
      role: "user",
      content: [
        { type: "text", text: "What's on the menu today?" },
        { type: "file_data", file_data: { mime_type: "text/plain", file_uri: file.uri } },
      ],
    },
  ],
});
```

`GeminiFileAPI` has `uploadFile(file, displayName?, { mimeType? })`, `getFile(name)`,
`listFiles(pageSize?, pageToken?)` and `deleteFile(name)`. Each returns the parsed JSON response
(or `true` for `deleteFile`) and throws when the API returns an error. To build one yourself:

| Option | Type | Required | Description |
|---|---|---|---|
| `apiKey` | `string` | yes | Google AI API key. |
| `baseUrl` | `string \| null \| undefined` | no | Base URL for the Gemini File API (default is the v1beta endpoint). |
| `fs` | `GeminiFileAPIFileSystem \| null \| undefined` | no | Reads paths given to `uploadFile` (`node:fs/promises` satisfies it). Not needed when uploading contents. |
| `fetch` | `{ (input: RequestInfo \| URL, init?: RequestInit): Promise<Response>; (input: string \| URL \| Request, init?: RequestInit): Promise<Response>; } \| null \| undefined` | no | The `fetch` implementation to use. Defaults to the global `fetch`. |

### Event handlers [#event-handlers]

Inherited from `LLMService`.

| Event                         | Description                                        |
| ----------------------------- | -------------------------------------------------- |
| `on_function_calls_started`   | Function calls were received and are about to run. |
| `on_function_calls_cancelled` | In-progress function calls were cancelled.         |

### Notes [#notes]

* Bot audio comes out as `TTSStartedFrame`, `TTSAudioRawFrame`s and `TTSStoppedFrame`, with the
  bot's transcript as `LLMTextFrame`s and `TTSTextFrame`s. User transcripts come out as
  `TranscriptionFrame`s, aggregated into sentences with a 0.5 second fallback timeout.
* With server-side VAD, the service emits no `UserStartedSpeakingFrame` or
  `UserStoppedSpeakingFrame`. Processors that depend on them (RTVI user speaking events, user
  idle and mute handling, turn recording) do nothing unless the user aggregator has a VAD
  analyzer. `LLMContextAggregatorPair` detects this service, so context writes are correct
  either way.
* Tools in the context take precedence over `tools` given at construction. Tools can't change
  mid-session: `LLMSetToolsFrame` is dropped, with a warning.
* Functions registered with `cancelOnInterruption: false` use Gemini's `NON_BLOCKING` tool
  behavior on models that support it (3.8 and later, and pre-3.0). On Gemini 3.x models before
  3.8, the service logs an error and pushes an error the first time one is used.
* Grounding metadata (for example from a Google Search tool) produces an
  `LLMSearchResponseFrame`, as with [`GoogleLLMService`](https://polvallverdu.github.io/pipedog/docs/services/llm/google/#notes).
* Video frames are throttled to one per second and sent as JPEG.
* The service reconnects after a connection error, resuming the session with the last
  resumption handle. After three consecutive failures it reports an error and stops.
* An `EndFrame` waits for the bot's turn to finish, for at most 30 seconds.
* On thinking models, a turn that Gemini reports as still in progress is held open, so a reply
  spanning several `turn_complete` messages is one assistant turn.
* A new system instruction or tools from the context make the service reconnect the session.

## Runtime notes [#runtime-notes]

| Runtime            | Works | What it needs |
| ------------------ | ----- | ------------- |
| Bun                | ✅ yes | Nothing.      |
| Node.js            | ✅ yes | Nothing.      |
| Deno               | ✅ yes | Nothing.      |
| Cloudflare Workers | ✅ yes | Nothing.      |

Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/google/package.json`: ✅ means the compat smoke test passes on that runtime.

The client is `GoogleGenAI` from `@google/genai`, which opens the Live API over the global
`WebSocket`. Server messages arrive through the SDK's callbacks and are fed to the service's
receive task, so socket errors go through the same reconnect path as upstream.

`GeminiFileAPI` uses the global `fetch` (injectable with `fetch`). `uploadFile()` takes a path,
as upstream does, or the file's contents (a `Uint8Array`, an `ArrayBuffer`, a `Blob` or a
`ReadableStream`). Contents work on every runtime, Cloudflare Workers included. A path is read
with the injected `fs`, or with `node:fs/promises` imported on first use (Bun, Node, Deno); on a
runtime without it, a path throws an error that says to pass the contents. The MIME type comes
from `mimeType`, else the path's extension, else a `Blob`'s type, else `displayName`'s
extension, else `application/octet-stream`.

`historyConfig` is set on the connect config as upstream does, but `@google/genai` 2.27 does
not forward it in the setup message.

The deprecated upstream `InputParams` class and the `model`, `voiceId` and `params`
constructor options are not ported; use `settings`.