Gemini Live

OverviewCopy Anchor Link

GeminiLiveLLMService connects to the Gemini Live API over a WebSocket. It takes user audio (and text and video), and returns spoken audio, transcripts and tool calls, so one service replaces the STT, LLM and TTS stages of a pipeline.

• GeminiLiveLLMService: the Live session, with server-side or local VAD, function calling, thinking, context window compression, session resumption and Google Search grounding.
• GeminiVADParams and ContextWindowCompressionParams: settings objects for VAD and compression.
• GeminiFileAPI: a client for the Gemini File API (upload, get, list, delete), available as llm.fileApi.

For Vertex AI, see Gemini Live on Vertex AI.

InstallationCopy Anchor Link

bun add @pipedog/google

import {
  ContextWindowCompressionParams,
  GeminiLiveLLMService,
  GeminiVADParams,
} from "@pipedog/google/gemini_live/llm";
import { GeminiFileAPI } from "@pipedog/google/gemini_live/file_api";

PrerequisitesCopy Anchor Link

• A Gemini API key from Google AI Studio.
• Pass the key as apiKey. The examples read it from GOOGLE_API_KEY.

GeminiLiveLLMServiceCopy Anchor Link

ConfigurationCopy Anchor Link

SettingsCopy Anchor Link

Passed as settings: new GeminiLiveLLMService.Settings({...}). They are sent when the session connects. Changing them later stores the new values but doesn't apply them to the open session (a warning names the settings that didn't take effect).

GeminiVADParams:

ContextWindowCompressionParams:

UsageCopy Anchor Link

import { LLMContext } from "@pipedog/core/processors/aggregators/llm_context";
import { Language } from "@pipedog/core/transcriptions/language";
import {
  ContextWindowCompressionParams,
  GeminiLiveLLMService,
  GeminiVADParams,
} from "@pipedog/google/gemini_live/llm";

const llm = new GeminiLiveLLMService({
  apiKey: process.env.GOOGLE_API_KEY ?? "",
  systemInstruction: "You are a friendly voice assistant. Keep answers short.",
  settings: new GeminiLiveLLMService.Settings({
    model: "models/gemini-3.8-live",
    voice: "Puck",
    language: Language.EN_US,
  }),
});

Compression and thinking:

const tuned = new GeminiLiveLLMService({
  apiKey: process.env.GOOGLE_API_KEY ?? "",
  settings: new GeminiLiveLLMService.Settings({
    temperature: 0.6,
    contextWindowCompression: new ContextWindowCompressionParams({
      enabled: true,
      triggerTokens: 8000,
    }),
    thinking: { includeThoughts: true },
  }),
});

Local VAD. Disable server-side VAD and configure a VAD analyzer on the user aggregator; the service then sends activity start and end signals to Gemini and replays a short pre-roll of audio (userAudioPrerollSecs) so the start of speech isn't clipped:

// Turn off server-side VAD. A local VAD in the user aggregator then drives
// turns, and the service sends activity start/end signals to Gemini.
const locallyDriven = new GeminiLiveLLMService({
  apiKey: process.env.GOOGLE_API_KEY ?? "",
  settings: new GeminiLiveLLMService.Settings({
    vad: new GeminiVADParams({ disabled: true }),
  }),
});

MethodsCopy Anchor Link

Gemini File APICopy Anchor Link

Upload a file, then reference its uri in a file_data content part of a context message:

// A path is read with node:fs/promises (Bun, Node, Deno). On any runtime,
// including Workers, pass the contents instead (bytes, a Blob or a stream):
// llm.fileApi.uploadFile(new TextEncoder().encode(menu), "menu.txt").
const uploaded = await llm.fileApi.uploadFile("menu.txt", "Menu");
const file = uploaded.file as { uri: string };

const context = new LLMContext({
  messages: [
    {
      role: "user",
      content: [
        { type: "text", text: "What's on the menu today?" },
        { type: "file_data", file_data: { mime_type: "text/plain", file_uri: file.uri } },
      ],
    },
  ],
});

GeminiFileAPI has uploadFile(file, displayName?, { mimeType? }), getFile(name), listFiles(pageSize?, pageToken?) and deleteFile(name). Each returns the parsed JSON response (or true for deleteFile) and throws when the API returns an error. To build one yourself:

Event handlersCopy Anchor Link

Inherited from LLMService.

NotesCopy Anchor Link

• Bot audio comes out as TTSStartedFrame, TTSAudioRawFrames and TTSStoppedFrame, with the bot's transcript as LLMTextFrames and TTSTextFrames. User transcripts come out as TranscriptionFrames, aggregated into sentences with a 0.5 second fallback timeout.
• With server-side VAD, the service emits no UserStartedSpeakingFrame or UserStoppedSpeakingFrame. Processors that depend on them (RTVI user speaking events, user idle and mute handling, turn recording) do nothing unless the user aggregator has a VAD analyzer. LLMContextAggregatorPair detects this service, so context writes are correct either way.
• Tools in the context take precedence over tools given at construction. Tools can't change mid-session: LLMSetToolsFrame is dropped, with a warning.
• Functions registered with cancelOnInterruption: false use Gemini's NON_BLOCKING tool behavior on models that support it (3.8 and later, and pre-3.0). On Gemini 3.x models before 3.8, the service logs an error and pushes an error the first time one is used.
• Grounding metadata (for example from a Google Search tool) produces an LLMSearchResponseFrame, as with GoogleLLMService.
• Video frames are throttled to one per second and sent as JPEG.
• The service reconnects after a connection error, resuming the session with the last resumption handle. After three consecutive failures it reports an error and stops.
• An EndFrame waits for the bot's turn to finish, for at most 30 seconds.
• On thinking models, a turn that Gemini reports as still in progress is held open, so a reply spanning several turn_complete messages is one assistant turn.
• A new system instruction or tools from the context make the service reconnect the session.

Runtime notesCopy Anchor Link

Generated from pipecat.runtimes and pipecat.compat in packages/google/package.json: ✅ means the compat smoke test passes on that runtime.

The client is GoogleGenAI from @google/genai, which opens the Live API over the global WebSocket. Server messages arrive through the SDK's callbacks and are fed to the service's receive task, so socket errors go through the same reconnect path as upstream.

GeminiFileAPI uses the global fetch (injectable with fetch). uploadFile() takes a path, as upstream does, or the file's contents (a Uint8Array, an ArrayBuffer, a Blob or a ReadableStream). Contents work on every runtime, Cloudflare Workers included. A path is read with the injected fs, or with node:fs/promises imported on first use (Bun, Node, Deno); on a runtime without it, a path throws an error that says to pass the contents. The MIME type comes from mimeType, else the path's extension, else a Blob's type, else displayName's extension, else application/octet-stream.

historyConfig is set on the connect config as upstream does, but @google/genai 2.27 does not forward it in the setup message.

The deprecated upstream InputParams class and the model, voiceId and params constructor options are not ported; use settings.