# OpenAI

> Streaming chat completions with OpenAILLMService, the base for every OpenAI-compatible API.

Source: https://polvallverdu.github.io/pipedog/docs/services/llm/openai/



## Overview [#overview]

`OpenAILLMService` streams replies from OpenAI's Chat Completions API, with function calling,
token usage metrics and out-of-band inference (`runInference`).

* **`OpenAILLMService`**: the OpenAI chat service. It extends `BaseOpenAILLMService`, which
  builds the client and the request. Point `baseUrl` at any server that speaks the Chat
  Completions API (Ollama, vLLM, OpenRouter, and so on) to use it with that provider.

For the Responses API, see [OpenAI Responses](https://polvallverdu.github.io/pipedog/docs/services/llm/openai-responses.md).

<Cards>
  <Card title="Upstream page" href="https://docs.pipecat.ai/api-reference/server/services/llm/openai">
    Pipecat's documentation for the same service.
  </Card>

  <Card title="Source" href="https://github.com/polvallverdu/pipedog/tree/main/packages/openai/src">
    `packages/openai/src/llm.ts` and `base_llm.ts`.
  </Card>

  <Card title="OpenAI docs" href="https://platform.openai.com/docs/api-reference/chat">
    The Chat Completions API reference.
  </Card>
</Cards>

## Installation [#installation]

```bash
bun add @pipedog/openai
```

```ts
import { OpenAILLMService } from "@pipedog/openai/llm";
import { OpenAILLMSettings } from "@pipedog/openai/base_llm";
```

## Prerequisites [#prerequisites]

* An OpenAI account and an API key from the [OpenAI dashboard](https://platform.openai.com/api-keys).
* Pass the key as `apiKey`. If you leave it out, the OpenAI SDK reads `OPENAI_API_KEY`. The
  examples read it from `OPENAI_API_KEY`.

## OpenAILLMService [#openaillmservice]

### Configuration [#configuration]

| Option | Type | Required | Description |
|---|---|---|---|
| `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. |
| `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. |
| `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. |
| `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. |
| `runInParallel` | `boolean \| undefined` | no | Whether to run function calls in parallel or sequentially. Defaults to true. |
| `groupParallelTools` | `boolean \| undefined` | no | Whether to group parallel function calls so the LLM is triggered exactly once after all calls in the batch complete. When false, each function call result triggers the LLM independently as it arrives. Defaults to true. |
| `functionCallTimeoutSecs` | `number \| null \| undefined` | no | Optional timeout in seconds for function calls. A call that runs past it is cancelled: its handler is cancelled (it gets a `CancelledError`), the call is settled as cancelled, and inference runs so the LLM can report that it didn't complete. |
| `fileResolver` | `FileResolver \| null \| undefined` | no | Resolves file URLs in the context that this provider can't fetch itself (see `FileResolver`). Pass one configured with a `fileStorage` to resolve uploaded-file URLs (`new FileResolver({ fileStorage: runnerArgs.fileStorage })` when a runner hosts the bot), and share one instance across services that may consume the same files (e.g. services switched between mid-session) so downloads are reused. Defaults to a resolver that handles `data:` and `http(s)` URLs only. |
| `apiKey` | `string \| null \| undefined` | no | OpenAI API key. If null, the SDK reads `OPENAI_API_KEY`. |
| `baseUrl` | `string \| null \| undefined` | no | Custom base URL for the API. |
| `organization` | `string \| null \| undefined` | no | OpenAI organization ID. |
| `project` | `string \| null \| undefined` | no | OpenAI project ID. |
| `defaultHeaders` | `Record<string, string> \| null \| undefined` | no | Additional HTTP headers. |
| `serviceTier` | `string \| null \| undefined` | no | Service tier to use (e.g., "auto", "flex", "priority"). |
| `settings` | `OpenAILLMSettings \| null \| undefined` | no | Runtime-updatable settings. When provided, its given fields override the service's defaults. |
| `retryTimeoutSecs` | `number \| null \| undefined` | no | How long an inference may go without producing output before it is abandoned and re-issued, when `retryOnTimeout` is set. Defaults to 5.0 seconds. |
| `retryOnTimeout` | `boolean \| null \| undefined` | no | Whether to re-issue the request once if the first attempt produces no output within `retryTimeoutSecs`. The retry is unbounded. |

The client options (`apiKey`, `baseUrl`, `organization`, `project`, `defaultHeaders`) come from
`OpenAIClientOptions`:

| Option | Type | Required | Description |
|---|---|---|---|
| `apiKey` | `string \| null \| undefined` | no | OpenAI API key. If null, the SDK reads `OPENAI_API_KEY`. |
| `baseUrl` | `string \| null \| undefined` | no | Custom base URL for the API. |
| `organization` | `string \| null \| undefined` | no | OpenAI organization ID. |
| `project` | `string \| null \| undefined` | no | OpenAI project ID. |
| `defaultHeaders` | `Record<string, string> \| null \| undefined` | no | Additional HTTP headers. |

### Settings [#settings]

Runtime-updatable settings, passed as `settings: new OpenAILLMService.Settings({...})`
(`OpenAILLMSettings`). Any setting you leave out keeps the default below. Fields marked
"not sent" hold the OpenAI SDK's sentinel and are left out of the request, so OpenAI's own
default applies.

| Setting                     | Type                               | Default     | Description                                                                          |
| --------------------------- | ---------------------------------- | ----------- | ------------------------------------------------------------------------------------ |
| `model`                     | `string`                           | `"gpt-4.1"` | Chat model.                                                                          |
| `systemInstruction`         | `string \| null`                   | `null`      | System prompt, added ahead of the context's messages.                                |
| `temperature`               | `number \| null`                   | not sent    | Sampling temperature.                                                                |
| `topP`                      | `number \| null`                   | not sent    | Nucleus sampling.                                                                    |
| `maxTokens`                 | `number \| null`                   | not sent    | `max_tokens`.                                                                        |
| `maxCompletionTokens`       | `number \| null`                   | not sent    | `max_completion_tokens`.                                                             |
| `frequencyPenalty`          | `number \| null`                   | not sent    | Frequency penalty.                                                                   |
| `presencePenalty`           | `number \| null`                   | not sent    | Presence penalty.                                                                    |
| `seed`                      | `number \| null`                   | not sent    | Sampling seed.                                                                       |
| `topK`                      | `number \| null`                   | `null`      | Not part of the OpenAI request.                                                      |
| `filterIncompleteUserTurns` | `boolean`                          | `false`     | Filter incomplete user turns.                                                        |
| `userTurnCompletionConfig`  | `UserTurnCompletionConfig \| null` | `null`      | Turn completion config used with `filterIncompleteUserTurns`.                        |
| `extra`                     | `Record<string, unknown>`          | `{}`        | Extra request fields, merged into the request last (they override the fields above). |

### Usage [#usage]

```ts title="llm.ts"
import { LLMUpdateSettingsFrame } from "@pipedog/core/frames/frames";
import { OpenAILLMService } from "@pipedog/openai/llm";

const llm = new OpenAILLMService({
  apiKey: process.env.OPENAI_API_KEY ?? "",
  settings: new OpenAILLMService.Settings({
    model: "gpt-4.1",
    systemInstruction: "You are a helpful assistant in a voice conversation.",
    temperature: 0.7,
  }),
});
```

An OpenAI-compatible server, through `baseUrl`:

```ts
// Any server that speaks the Chat Completions API, here a local Ollama.
const local = new OpenAILLMService({
  apiKey: "ollama",
  baseUrl: "http://localhost:11434/v1",
  settings: new OpenAILLMService.Settings({ model: "llama3.2" }),
});
```

Re-issue a request that produces no output in time:

```ts
const withRetry = new OpenAILLMService({
  apiKey: process.env.OPENAI_API_KEY ?? "",
  retryOnTimeout: true,
  retryTimeoutSecs: 3,
});
```

Change settings mid-call with an `LLMUpdateSettingsFrame`. The next completion uses them:

```ts
// Queue it on the worker, or push it from a processor upstream of the service.
const update = new LLMUpdateSettingsFrame({
  delta: new OpenAILLMService.Settings({ temperature: 0.2, maxCompletionTokens: 300 }),
});
```

### Event handlers [#event-handlers]

| Event                         | Description                                     |
| ----------------------------- | ----------------------------------------------- |
| `on_completion_timeout`       | A completion timed out.                         |
| `on_function_calls_started`   | The model asked for one or more function calls. |
| `on_function_calls_cancelled` | Function calls were cancelled.                  |

These come from the `LLMService` base class.

### Notes [#notes]

* Runs a completion for each `LLMContextFrame`, between an `LLMFullResponseStartFrame` and an
  `LLMFullResponseEndFrame`, and streams the reply as `LLMTextFrame`s. Other frames pass
  through.
* Requests always stream, with `stream_options: { include_usage: true }`, so token usage metrics
  are reported.
* `serviceTier` is sent as `service_tier` when set.
* `retryOnTimeout` is off by default. With it on, an attempt that produces nothing within
  `retryTimeoutSecs` (5 seconds by default) is abandoned and re-issued once.
* Models older than `gpt-4o-mini` and `gpt-4o-2024-08-06` can't enforce a response schema, so
  `runInference` ignores a `responseSchema` for them, with a warning.
* OpenAI-compatible subclasses override `createClient()` to build the client their own way, and
  can set `supportsDeveloperRole = false` to send "developer" messages as "user" messages.

## Runtime notes [#runtime-notes]

| Runtime            | Works | What it needs |
| ------------------ | ----- | ------------- |
| Bun                | ✅ yes | Nothing.      |
| Node.js            | ✅ yes | Nothing.      |
| Deno               | ✅ yes | Nothing.      |
| Cloudflare Workers | ✅ yes | Nothing.      |

Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/openai/package.json`: ✅ means the compat smoke test passes on that runtime.

* The client is the official `openai` JS SDK, which talks through `fetch`. It works on Bun,
  Node, Deno and Cloudflare Workers.
* Upstream builds the client with an httpx connection pool. `fetch` has no such object, so
  connection reuse is left to the runtime.
* The JS SDK has no `extra_body` option. Provider-specific `extra_body` fields are merged into
  the request JSON just before it is sent, as the Python client does.
* The deprecated `model` and `params` constructor options and `BaseOpenAILLMService.InputParams`
  are not ported. Use `settings`.
* For Azure OpenAI, use [`AzureLLMService`](https://polvallverdu.github.io/pipedog/docs/services/llm/azure.md). Other OpenAI-compatible
  providers have their own subclasses where ported (see [LLM](https://polvallverdu.github.io/pipedog/docs/services/llm.md)); for the
  rest, `OpenAILLMService` with `baseUrl` covers most of them.