Google Gemini

OverviewCopy Anchor Link

GoogleLLMService runs Gemini models through the Gemini API (Google AI Studio keys). It converts an LLMContext to Gemini's Content/Part format, streams the reply and handles function calls.

• GoogleLLMService: streaming Gemini inference with tools, thinking (GoogleThinkingConfig), safety settings, image output and search grounding.
• LLMSearchResponseFrame (frames.ts): pushed when a reply carries grounding metadata, for example from the Google Search tool.
• GoogleRTVIProcessor / GoogleRTVIObserver (rtvi.ts): an RTVI processor whose observer also forwards LLMSearchResponseFrames to the client.

For Vertex AI, see Google Vertex AI. For speech-to-speech, see Gemini Live.

InstallationCopy Anchor Link

bun add @pipedog/google

import { GoogleLLMService, GoogleThinkingConfig } from "@pipedog/google/llm";
import { LLMSearchResponseFrame } from "@pipedog/google/frames";
import { GoogleRTVIProcessor } from "@pipedog/google/rtvi";

PrerequisitesCopy Anchor Link

• A Gemini API key from Google AI Studio.
• Pass the key as apiKey. The examples read it from GOOGLE_API_KEY.

GoogleLLMServiceCopy Anchor Link

ConfigurationCopy Anchor Link

SettingsCopy Anchor Link

Runtime-updatable settings, passed as settings: new GoogleLLMService.Settings({...}). Any setting you leave out keeps the default below. Extra Gemini GenerateContentConfig fields can go in extra, with the Python SDK's snake_case keys (they are converted to camelCase when the request is sent).

GoogleThinkingConfig takes thinkingLevel (Gemini 3 and later: "minimal", "low", "medium", "high"), thinkingBudget (Gemini 2.5: a token count, 0 off, -1 dynamic) and includeThoughts. Set a level or a budget, not both.

When thinking is null, the service sends the lowest thinking setting Google documents for the model, for the fastest replies, and logs what it sent. For example gemini-2.5-flash gets thinking_budget: 0, gemini-2.5-pro gets 128, gemini-3.6-flash gets thinking_level: "minimal" and gemini-3.7-flash gets "low". Models not in that table (including aliases such as gemini-flash-latest) run at Google's default.

UsageCopy Anchor Link

import { LLMUpdateSettingsFrame } from "@pipedog/core/frames/frames";
import { GoogleLLMService, GoogleThinkingConfig } from "@pipedog/google/llm";
import { GoogleRTVIProcessor } from "@pipedog/google/rtvi";

const llm = new GoogleLLMService({
  apiKey: process.env.GOOGLE_API_KEY ?? "",
  settings: new GoogleLLMService.Settings({
    model: "gemini-3.6-flash",
    systemInstruction: "You are a helpful voice assistant. Keep answers short.",
    temperature: 0.7,
  }),
});

Thinking, for Gemini 3 and Gemini 2.5 models:

// Gemini 3 models take a thinking level.
const gemini3 = new GoogleLLMService({
  apiKey: process.env.GOOGLE_API_KEY ?? "",
  settings: new GoogleLLMService.Settings({
    model: "gemini-3.6-flash",
    thinking: new GoogleThinkingConfig({ thinkingLevel: "low", includeThoughts: true }),
  }),
});

// Gemini 2.5 models take a token budget (0 turns thinking off).
const gemini25 = new GoogleLLMService({
  apiKey: process.env.GOOGLE_API_KEY ?? "",
  settings: new GoogleLLMService.Settings({
    model: "gemini-2.5-flash",
    thinking: new GoogleThinkingConfig({ thinkingBudget: 1024 }),
  }),
});

Google Search grounding, with the RTVI processor that forwards search results to the client:

// Tools in the SDK's camelCase shape are sent as given.
const grounded = new GoogleLLMService({
  apiKey: process.env.GOOGLE_API_KEY ?? "",
  tools: [{ googleSearch: {} }],
});

// Pass it to PipelineWorker as `rtviProcessor` to forward each
// LLMSearchResponseFrame to the client as a "bot-llm-search-response" message.
const rtvi = new GoogleRTVIProcessor();

Change settings mid-call with an LLMUpdateSettingsFrame:

// Queue it on the worker, or push it from a processor upstream of the service.
const update = new LLMUpdateSettingsFrame({
  delta: new GoogleLLMService.Settings({ temperature: 0.2, maxTokens: 1024 }),
});

Event handlersCopy Anchor Link

Inherited from LLMService.

NotesCopy Anchor Link

• Emits LLMFullResponseStartFrame, LLMTextFrames and LLMFullResponseEndFrame. Thoughts (with includeThoughts) arrive whole, each as an LLMThoughtStartFrame, LLMThoughtTextFrame and LLMThoughtEndFrame.
• When a reply has grounding metadata, the service pushes an LLMSearchResponseFrame with the reply text (searchResult), the search entry point's renderedContent, and origins (site URI, title and the supported text segments with confidence scores). GoogleRTVIObserver sends it to the client as a bot-llm-search-response message.
• Gemini thought signatures are saved into the context (through an LLMMessagesAppendFrame), so they go back to the model on the next turn.
• Inline image output becomes an AssistantImageRawFrame, but only with an imageDecoder. The default decoder throws, so without one an image reply surfaces as a pushed error.
• streamIdleTimeoutSecs (default 20) bounds the gap between chunks, including the wait for the first one. With retryOnTimeout, a first chunk that doesn't arrive within retryTimeoutSecs (default 5) re-issues the request once.
• Newer models reject a request that ends with a model turn. For any model whose name doesn't start with gemini-2., gemini-3-, gemini-3.1- or gemini-pro-latest, the service adds a trailing user message when needed.
• A reply that stops for a reason other than STOP (safety, recitation, token limit, ...) logs a warning with the finish reason. The text that did arrive is still pushed.
• Setting thinkingBudget on a Gemini 3, Gemini 4 or Gemma 4 model logs a warning: use thinkingLevel there.
• runInference() accepts a responseSchema (JSON schema) on Gemini 2.5 and later.
• An INVALID_ARGUMENT rejection removes a pending invalid file message from the context and is reported as an application error.
• Token usage includes tool-use prompt tokens in the prompt count and thinking tokens in the completion count.

Runtime notesCopy Anchor Link

Generated from pipecat.runtimes and pipecat.compat in packages/google/package.json: ✅ means the compat smoke test passes on that runtime.

The client is GoogleGenAI from @google/genai, which uses fetch, so the service runs on Bun, Node, Deno and Cloudflare Workers. Subclasses can override createClient(). Image decoding (upstream uses Pillow) is not ported: pass an imageDecoder built on sharp, @jsquash/* or similar to handle image output.

The deprecated upstream InputParams class and the model, params and systemInstruction constructor options are not ported; use settings.