# Vercel AI SDK

> Run any Vercel AI SDK language model (OpenAI, Google, Anthropic, Ollama, the AI Gateway, ...) as the pipeline's LLM with AISDKLLMService.

Source: https://polvallverdu.github.io/pipedog/docs/services/llm/ai-sdk/



## Overview [#overview]

`AISDKLLMService` runs a [Vercel AI SDK](https://ai-sdk.dev) language model as the pipeline's LLM
service. The AI SDK gives one interface to many providers, so one service covers all of them:
`@ai-sdk/openai`, `@ai-sdk/google`, `@ai-sdk/anthropic`, `@ai-sdk/mistral`, `@ai-sdk/amazon-bedrock`,
Ollama and the other community providers, and model ID strings routed through the AI Gateway.

* **`AISDKLLMService`**: an `LLMService` that streams the model's reply with the AI SDK's
  `streamText()`. It behaves like pipedog's native LLM services: it consumes `LLMContextFrame`s,
  pushes `LLMFullResponseStartFrame`, `LLMTextFrame`s and `LLMFullResponseEndFrame`, runs function
  calls through pipedog's function calling, and reports TTFB, processing time and token usage.

This is a pipedog addition: Pipecat (Python) has no AI SDK service.

<Cards>
  <Card title="Source" href="https://github.com/polvallverdu/pipedog/tree/main/packages/ai-sdk/src">
    `packages/ai-sdk/src/llm.ts` and the message adapter, `ai_sdk_adapter.ts`.
  </Card>

  <Card title="AI SDK providers" href="https://ai-sdk.dev/providers/ai-sdk-providers">
    The official and community providers, and their model IDs and options.
  </Card>
</Cards>

## Installation [#installation]

`ai` (version 7) is a peer dependency, so you pick its version. Add the provider package you use,
and `zod`, which `ai` requires.

```bash
bun add @pipedog/ai-sdk ai zod @ai-sdk/google
```

```ts
import { AISDKLLMService } from "@pipedog/ai-sdk";
```

## Prerequisites [#prerequisites]

* Whatever the provider needs. `@ai-sdk/google` reads `GOOGLE_GENERATIVE_AI_API_KEY`, or takes
  `createGoogleGenerativeAI({ apiKey })`. A model ID string uses the AI Gateway, which reads
  `AI_GATEWAY_API_KEY`.

## AISDKLLMService [#aisdkllmservice]

### Configuration [#configuration]

| Option | Type | Required | Description |
|---|---|---|---|
| `model` | `LanguageModel` | yes | The AI SDK language model, e.g. `openai("gpt-4.1-mini")`, `google("gemini-2.5-flash")`, `anthropic("claude-sonnet-4-5")`, or a model ID string for the AI SDK's global provider (the AI Gateway by default). |
| `provider` | `((modelId: string) => LanguageModel) \| null \| undefined` | no | Builds a model from a model ID, e.g. the provider instance (`google`, `openai`, `createOllama(...)`). With it, changing `settings.model` at runtime (`LLMUpdateSettingsFrame`) switches the model. |
| `settings` | `AISDKLLMSettings \| null \| undefined` | no | Runtime-updatable settings. Given fields override the defaults. |
| `tools` | `ToolSet \| null \| undefined` | no | AI SDK tools sent on every request alongside the context's tools, such as provider-defined tools (`google.tools.googleSearch({})`, `openai.tools.webSearch({})`). Your own functions belong in the context's `ToolsSchema`, so pipedog runs them. |
| `maxRetries` | `number \| null \| undefined` | no | Maximum retries of a failed request (the AI SDK's `maxRetries`). Defaults to 2. |
| `headers` | `Record<string, string> \| null \| undefined` | no | Additional HTTP headers sent with each request. |
| `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. |
| `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. |
| `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. |
| `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. |
| `runInParallel` | `boolean \| undefined` | no | Whether to run function calls in parallel or sequentially. Defaults to true. |
| `groupParallelTools` | `boolean \| undefined` | no | Whether to group parallel function calls so the LLM is triggered exactly once after all calls in the batch complete. When false, each function call result triggers the LLM independently as it arrives. Defaults to true. |
| `functionCallTimeoutSecs` | `number \| null \| undefined` | no | Optional timeout in seconds for function calls. A call that runs past it is cancelled: its handler is cancelled (it gets a `CancelledError`), the call is settled as cancelled, and inference runs so the LLM can report that it didn't complete. |
| `fileResolver` | `FileResolver \| null \| undefined` | no | Resolves file URLs in the context that this provider can't fetch itself (see `FileResolver`). Pass one configured with a `fileStorage` to resolve uploaded-file URLs (`new FileResolver({ fileStorage: runnerArgs.fileStorage })` when a runner hosts the bot), and share one instance across services that may consume the same files (e.g. services switched between mid-session) so downloads are reused. Defaults to a resolver that handles `data:` and `http(s)` URLs only. |

### Settings [#settings]

`AISDKLLMService.Settings` (`AISDKLLMSettings`) holds the runtime-updatable settings. The common
fields map to the AI SDK's call settings: `temperature`, `maxTokens` (`maxOutputTokens`), `topP`,
`topK`, `frequencyPenalty`, `presencePenalty` and `seed`. `null`, the default, leaves a field to the
provider.

| Option | Type | Required | Description |
|---|---|---|---|
| `stopSequences` | `string[] \| NotGiven \| null \| undefined` | no | Sequences that stop generation. |
| `providerOptions` | `NotGiven \| SharedV4ProviderOptions \| null \| undefined` | no | Provider-specific options, keyed by provider (the AI SDK's `providerOptions`), e.g. `{ google: { thinkingConfig: { thinkingBudget: 0 } } }`. |
| `model` | `string \| NotGiven \| null \| undefined` | no | AI model identifier (e.g. `"gpt-4o"`, `"eleven_flash_v2_5"`). Defaults to `NOT_GIVEN` for delta mode. In store mode, set to a model string or `null` if the service has no model concept. |
| `extra` | `Record<string, unknown> \| undefined` | no | Catch-all for service-specific keys that have no declared field. |
| `systemInstruction` | `string \| NotGiven \| null \| undefined` | no | System instruction/prompt for the model. |
| `temperature` | `number \| NotGiven \| null \| undefined` | no | Sampling temperature. |
| `maxTokens` | `number \| NotGiven \| null \| undefined` | no | Maximum tokens to generate. |
| `topP` | `number \| NotGiven \| null \| undefined` | no | Nucleus sampling probability. |
| `topK` | `number \| NotGiven \| null \| undefined` | no | Top-k sampling parameter. |
| `frequencyPenalty` | `number \| NotGiven \| null \| undefined` | no | Frequency penalty. |
| `presencePenalty` | `number \| NotGiven \| null \| undefined` | no | Presence penalty. |
| `seed` | `number \| NotGiven \| null \| undefined` | no | Random seed for reproducibility. |
| `filterIncompleteUserTurns` | `boolean \| NotGiven \| null \| undefined` | no | Enable LLM-based turn completion detection to suppress bot responses when the user was cut off mid-thought. Set by `LLMTurnCompletionUserTurnStopStrategy`, which is also what waits on the resulting verdict. Not meant to be set by application code; use `userTurnStrategies: new FilterIncompleteUserTurnStrategies()` on `LLMUserAggregatorParams` instead. |
| `userTurnCompletionConfig` | `NotGiven \| UserTurnCompletionConfig \| null \| undefined` | no | Configuration for turn completion behavior when `filterIncompleteUserTurns` is enabled. Set by the user turn strategy; pass the config to `FilterIncompleteUserTurnStrategies({ config })`. |
| `useProviderThinkingDefault` | `boolean \| NotGiven \| undefined` | no | Whether a model with no thinking configured runs at its provider's default. Services that otherwise send each model its lowest thinking setting, for the fastest responses, send none. `BackendLLMWorker` sets it on its LLM. Defaults to false. |

`systemInstruction` is sent as the AI SDK's `instructions`. `extra` is merged into the
`streamText()` call as is.

### Usage [#usage]

```ts title="llm.ts"
import { google } from "@ai-sdk/google";
import { AISDKLLMService } from "@pipedog/ai-sdk";

const llm = new AISDKLLMService({
  model: google("gemini-3.5-flash-lite"),
  settings: new AISDKLLMService.Settings({
    systemInstruction: "You are a helpful voice assistant. Keep answers short.",
    temperature: 0.7,
    maxTokens: 1024,
  }),
});

```

### Function calling [#function-calling]

Tools come from the context's `ToolsSchema`, as with every LLM service. Each `FunctionSchema`
becomes an AI SDK tool with a JSON schema and no `execute`, so the AI SDK hands the model's calls
back to pipedog, which runs the registered handler (or the schema's own handler, or a direct
function), adds the result to the context and asks the model again.

```ts title="tools.ts"
import { FunctionSchema } from "@pipedog/core/adapters/llm/schemas/function_schema";
import { ToolsSchema } from "@pipedog/core/adapters/llm/schemas/tools_schema";
import { LLMContext } from "@pipedog/core/processors/aggregators/llm_context";
import type { FunctionCallParams } from "@pipedog/core/services/llm_service";

const weather = new FunctionSchema({
  name: "get_weather",
  description: "Get the current weather for a city",
  properties: { city: { type: "string" } },
  required: ["city"],
});

// pipedog runs the handler and sends the result back to the model.
llm.registerFunction("get_weather", async (params: FunctionCallParams) => {
  await params.resultCallback({ conditions: "sunny", celsius: 21 });
});

const context = new LLMContext({ tools: new ToolsSchema({ standardTools: [weather] }) });

```

Provider-defined tools, such as Google Search grounding or OpenAI web search, go in the `tools`
option. They are sent on every request and run on the provider's side.

### Switching models [#switching-models]

```ts title="switch.ts"
import { LLMUpdateSettingsFrame } from "@pipedog/core/frames/frames";

// With `provider`, a model ID in a settings update switches the model.
const switchable = new AISDKLLMService({
  model: google("gemini-3.5-flash-lite"),
  provider: google,
});
const update = new LLMUpdateSettingsFrame({
  delta: new AISDKLLMService.Settings({ model: "gemini-3.5-flash" }),
});

// Or switch to a model from any provider directly.
switchable.setLanguageModel(google("gemini-3.5-flash"));
```

Without `provider`, a model ID in a settings update can't be turned into a model, so it is ignored
with a warning (unless the model was given as an ID string in the first place).

### Provider options [#provider-options]

Provider-specific options go in `providerOptions`, keyed by provider, as in the AI SDK:

```ts title="provider-options.ts"
const thinkingOff = new AISDKLLMService({
  model: google("gemini-3.5-flash-lite"),
  settings: new AISDKLLMService.Settings({
    providerOptions: { google: { thinkingConfig: { thinkingBudget: 0 } } },
  }),
});
```

### Notes [#notes]

* Consumes `LLMContextFrame`. For each response it pushes `LLMFullResponseStartFrame`, one
  `LLMTextFrame` per streamed text delta and `LLMFullResponseEndFrame`, even when the request fails.
* Reasoning is pushed as `LLMThoughtStartFrame` / `LLMThoughtTextFrame` / `LLMThoughtEndFrame`
  and kept in the context, so providers that need it back (Anthropic thinking blocks) get it.
* Provider metadata that comes with a tool call or the reply text, such as Gemini's thought
  signatures, is kept in the context and sent back with that output on later turns.
* An initial `"system"` context message becomes the instructions when the service has no
  `systemInstruction`. Later `"system"` and `"developer"` messages are sent as user messages.
* Images (`image_url`), files (`file_url`, `file_base64`) and audio (`input_audio`) in user
  messages become AI SDK image and file parts. File URLs are fetched by pipedog's file resolver
  and sent inline.
* A failed request is pushed upstream as an `ErrorFrame` (`Error during completion: ...`). The AI
  SDK retries failed requests itself (`maxRetries`, 2 by default).
* `runInference()` runs a one-shot `generateText()`. A `responseSchema` is passed as
  `Output.object()`, so the provider enforces it where it can.

## Runtime notes [#runtime-notes]

| Runtime            | Works | What it needs |
| ------------------ | ----- | ------------- |
| Bun                | ✅ yes | Nothing.      |
| Node.js            | ✅ yes | Nothing.      |
| Deno               | ✅ yes | Nothing.      |
| Cloudflare Workers | ✅ yes | Nothing.      |

Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/ai-sdk/package.json`: ✅ means the compat smoke test passes on that runtime.

* An interruption cancels the response: the request's `abortSignal` is aborted and each stream
  part is a cancellation point.
* The service uses no runtime-specific APIs. Whether a provider package runs on a given runtime is
  up to that package; the official ones use `fetch`.