# Cloudflare Workers AI

> WorkersAILLMService: Workers AI chat models through Cloudflare's OpenAI-compatible endpoint.

Source: https://polvallverdu.github.io/pipedog/docs/services/llm/cloudflare/



## Overview [#overview]

**`WorkersAILLMService`** runs Workers AI's chat models (Llama, GLM, Kimi, Qwen, gpt-oss and others)
through Workers AI's OpenAI-compatible endpoint. It is an `OpenAILLMService` subclass, like the other
[OpenAI-compatible providers](https://polvallverdu.github.io/pipedog/docs/services/llm/openai.md): streaming, function calling and usage
metrics work the same way.

This package is pipedog's own: Pipecat has no Workers AI services.

<Cards>
  <Card title="Source" href="https://github.com/polvallverdu/pipedog/tree/main/packages/cloudflare/src/llm.ts">
    `packages/cloudflare/src/llm.ts`.
  </Card>

  <Card title="OpenAI compatibility" href="https://developers.cloudflare.com/workers-ai/configuration/open-ai-compatibility/">
    Workers AI's `/v1/chat/completions` endpoint.
  </Card>

  <Card title="Models" href="https://developers.cloudflare.com/workers-ai/models/">
    The text generation models, and which take function calls.
  </Card>
</Cards>

## Installation [#installation]

```bash
bun add @pipedog/cloudflare
```

```ts
import { WorkersAILLMService } from "@pipedog/cloudflare/llm";
```

## Prerequisites [#prerequisites]

* A Cloudflare account ID and an [API token](https://dash.cloudflare.com/profile/api-tokens) with the
  Workers AI permission. The examples read them from `CLOUDFLARE_ACCOUNT_ID` and
  `CLOUDFLARE_API_TOKEN`.
* The service needs them inside a Worker too: the `AI` binding answers chat requests in Workers AI's
  own format, not OpenAI's, so the service calls the REST endpoint with `fetch`, which works from a
  Worker like from any other runtime.

## Configuration [#configuration]

| Option | Type | Required | Description |
|---|---|---|---|
| `accountId` | `string \| null \| undefined` | no | Cloudflare account ID. Required unless `client` has one. |
| `apiToken` | `string \| null \| undefined` | no | Cloudflare API token with Workers AI access. Required unless `client` has one. |
| `client` | `WorkersAIClient \| null \| undefined` | no | A REST `WorkersAIClient` to take the account ID, token and base URL from. |
| `baseUrl` | `string \| null \| undefined` | no | The full OpenAI-compatible base URL. Defaults to `https://api.cloudflare.com/client/v4/accounts/{accountId}/ai/v1`; set it to go through an AI Gateway, for instance. |
| `settings` | `OpenAILLMSettings \| null \| undefined` | no | Runtime-updatable settings; values override the built-in defaults. |
| `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. |
| `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. |
| `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. |
| `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. |
| `runInParallel` | `boolean \| undefined` | no | Whether to run function calls in parallel or sequentially. Defaults to true. |
| `groupParallelTools` | `boolean \| undefined` | no | Whether to group parallel function calls so the LLM is triggered exactly once after all calls in the batch complete. When false, each function call result triggers the LLM independently as it arrives. Defaults to true. |
| `functionCallTimeoutSecs` | `number \| null \| undefined` | no | Optional timeout in seconds for function calls. A call that runs past it is cancelled: its handler is cancelled (it gets a `CancelledError`), the call is settled as cancelled, and inference runs so the LLM can report that it didn't complete. |
| `fileResolver` | `FileResolver \| null \| undefined` | no | Resolves file URLs in the context that this provider can't fetch itself (see `FileResolver`). Pass one configured with a `fileStorage` to resolve uploaded-file URLs (`new FileResolver({ fileStorage: runnerArgs.fileStorage })` when a runner hosts the bot), and share one instance across services that may consume the same files (e.g. services switched between mid-session) so downloads are reused. Defaults to a resolver that handles `data:` and `http(s)` URLs only. |
| `organization` | `string \| null \| undefined` | no | OpenAI organization ID. |
| `project` | `string \| null \| undefined` | no | OpenAI project ID. |
| `defaultHeaders` | `Record<string, string> \| null \| undefined` | no | Additional HTTP headers. |
| `serviceTier` | `string \| null \| undefined` | no | Service tier to use (e.g., "auto", "flex", "priority"). |
| `retryTimeoutSecs` | `number \| null \| undefined` | no | How long an inference may go without producing output before it is abandoned and re-issued, when `retryOnTimeout` is set. Defaults to 5.0 seconds. |
| `retryOnTimeout` | `boolean \| null \| undefined` | no | Whether to re-issue the request once if the first attempt produces no output within `retryTimeoutSecs`. The retry is unbounded. |

The base URL is `https://api.cloudflare.com/client/v4/accounts/{accountId}/ai/v1`. A REST
`WorkersAIClient` can be passed as `client` instead of `accountId` and `apiToken`.

## Settings [#settings]

The [OpenAI settings](https://polvallverdu.github.io/pipedog/docs/services/llm/openai/#settings) (`model`, `systemInstruction`,
`temperature`, `maxTokens`, ...), passed as `settings: new WorkersAILLMService.Settings({...})`.
`model` defaults to `"@cf/meta/llama-3.3-70b-instruct-fp8-fast"`. Others to try:
`"@cf/zai-org/glm-4.7-flash"`, `"@cf/meta/llama-4-scout-17b-16e-instruct"`, `"@cf/openai/gpt-oss-20b"`.

## Usage [#usage]

```ts title="llm.ts"
import { WorkersAILLMService } from "@pipedog/cloudflare/llm";
import { LLMUpdateSettingsFrame } from "@pipedog/core/frames/frames";

const llm = new WorkersAILLMService({
  accountId: process.env.CLOUDFLARE_ACCOUNT_ID ?? "",
  apiToken: process.env.CLOUDFLARE_API_TOKEN ?? "",
  settings: new WorkersAILLMService.Settings({
    model: "@cf/meta/llama-3.3-70b-instruct-fp8-fast",
    systemInstruction: "You are a helpful assistant in a voice conversation.",
    temperature: 0.7,
  }),
});
```

Through an [AI Gateway](https://developers.cloudflare.com/ai-gateway/), for caching, logs and rate
limits, set `baseUrl` to the gateway's Workers AI URL:

```ts
// Through an AI Gateway: the gateway's Workers AI OpenAI-compatible URL.
const viaGateway = new WorkersAILLMService({
  apiToken: process.env.CLOUDFLARE_API_TOKEN ?? "",
  baseUrl: `https://gateway.ai.cloudflare.com/v1/${process.env.CLOUDFLARE_ACCOUNT_ID}/my-gateway/workers-ai/v1`,
});
```

Change settings mid-call with an `LLMUpdateSettingsFrame`:

```ts
// Queue it on the worker, or push it from a processor upstream of the service.
const update = new LLMUpdateSettingsFrame({
  delta: new WorkersAILLMService.Settings({ temperature: 0.2, maxTokens: 300 }),
});
```

## Notes [#notes]

* Not every Workers AI model takes function calls; check the model page before giving it tools.
* The same package has Workers AI [speech-to-text](https://polvallverdu.github.io/pipedog/docs/services/stt/cloudflare.md),
  [text-to-speech](https://polvallverdu.github.io/pipedog/docs/services/tts/cloudflare.md) and Smart Turn v2
  ([turn detection](https://polvallverdu.github.io/pipedog/docs/concepts/turn-detection/#smart-turn-on-workers-ai)). For a whole agent on
  Cloudflare, see [Voice agent fully on Cloudflare](https://polvallverdu.github.io/pipedog/docs/deployment/cloudflare-workers-ai.md).

## Runtime notes [#runtime-notes]

| Runtime            | Works | What it needs |
| ------------------ | ----- | ------------- |
| Bun                | ✅ yes | Nothing.      |
| Node.js            | ✅ yes | Nothing.      |
| Deno               | ✅ yes | Nothing.      |
| Cloudflare Workers | ✅ yes | Nothing.      |

Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/cloudflare/package.json`: ✅ means the compat smoke test passes on that runtime.

* The client is the official `openai` JS SDK (through `@pipedog/openai`), which talks through
  `fetch`. It works on Bun, Node, Deno and Cloudflare Workers.
* Tested against a local OpenAI-compatible mock server, not against Workers AI itself.