# OpenAI

> Speech-to-text with OpenAISTTService (segmented, Audio API) and OpenAIRealtimeSTTService (streaming, Realtime API, optional server VAD).

Source: https://polvallverdu.github.io/pipedog/docs/services/stt/openai/



## Overview [#overview]

pipedog ports both of Pipecat's OpenAI speech-to-text services:

* **`OpenAISTTService`**: transcribes each speech segment with the Audio API
  (`audio.transcriptions`). It waits for the user to stop speaking, then sends the segment as a
  WAV file. Works with the GPT transcription models and Whisper (`whisper-1`).
* **`OpenAIRealtimeSTTService`**: streams audio over a WebSocket to the Realtime API in
  transcription-only mode, with interim results. Turns come from a local VAD (the default) or
  from OpenAI's server-side VAD.

<Cards>
  <Card title="Upstream page" href="https://docs.pipecat.ai/api-reference/server/services/stt/openai">
    Pipecat's documentation for the same services.
  </Card>

  <Card title="Source" href="https://github.com/polvallverdu/pipedog/tree/main/packages/openai/src">
    `packages/openai/src/stt.ts`.
  </Card>

  <Card title="OpenAI docs" href="https://platform.openai.com/docs/guides/speech-to-text">
    Transcription models and languages.
  </Card>
</Cards>

## Installation [#installation]

```bash
bun add @pipedog/openai
```

```ts
import { OpenAIRealtimeSTTService, OpenAISTTService } from "@pipedog/openai/stt";
```

`OpenAISTTService` builds on `BaseWhisperSTTService` from `@pipedog/whisper`, which
`@pipedog/openai` depends on.

## Prerequisites [#prerequisites]

* An OpenAI account and an API key from the [OpenAI dashboard](https://platform.openai.com/api-keys).
* Pass the key as `apiKey`. `OpenAISTTService` falls back to `OPENAI_API_KEY` when you leave it
  out; `OpenAIRealtimeSTTService` requires it. The examples read it from `OPENAI_API_KEY`.

## OpenAISTTService [#openaisttservice]

### Configuration [#configuration]

| Option | Type | Required | Description |
|---|---|---|---|
| `settings` | `OpenAISTTSettings \| null \| undefined` | no | Runtime-updatable settings. |
| `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. |
| `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. |
| `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. |
| `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. |
| `audioPassthrough` | `boolean \| undefined` | no | Whether to pass audio frames downstream after processing. Defaults to true. |
| `sampleRate` | `number \| null \| undefined` | no | The sample rate for audio input. If null, will be determined from the start frame. |
| `sttTtfbTimeout` | `number \| undefined` | no | Time in seconds to wait after VAD stop before reporting TTFB. This delay allows the final transcription to arrive. Defaults to 2.0. |
| `ttfsP99Latency` | `number \| null \| undefined` | no | P99 latency from speech end to final transcript in seconds. Broadcast via STTMetadataFrame at pipeline start. |
| `keepaliveTimeout` | `number \| null \| undefined` | no | Seconds of no audio before sending silence to keep the connection alive. null disables keepalive. |
| `keepaliveInterval` | `number \| undefined` | no | Seconds between idle checks when keepalive is enabled. |
| `trailingSilenceSecs` | `number \| undefined` | no | Seconds of silence appended to each segment before it is transcribed, so the model hears the end of speech and can finish the last word. Set to 0 to disable. Defaults to 0.5. |
| `apiKey` | `string \| null \| undefined` | no | Service API key. Defaults to null. |
| `baseUrl` | `string \| null \| undefined` | no | Service API base URL. Defaults to null. |
| `httpClient` | `((input: RequestInfo \| URL, init?: RequestInit) => Promise<Response>) \| null \| undefined` | no | Custom `fetch` for API requests, e.g. one with a longer request timeout. Defaults to null, which lets the SDK use the global `fetch`. |
| `includeProbMetrics` | `boolean \| undefined` | no | If true, enables probability metrics in API response. Each service implements this differently (see child classes). Defaults to false. |
| `pushEmptyTranscripts` | `boolean \| undefined` | no | If true, allow empty `TranscriptionFrame` frames to be pushed downstream instead of discarding them. This is intended for situations where VAD fires even though the user did not speak. In these cases, it is useful to know that nothing was transcribed so that the agent can resume speaking, instead of waiting longer for a transcription. Defaults to false. |

### Settings [#settings]

Runtime-updatable settings, passed as `settings: new OpenAISTTService.Settings({...})`
(`OpenAISTTSettings`). Any setting you leave out keeps the default below.

| Setting       | Type               | Default            | Description                                                               |
| ------------- | ------------------ | ------------------ | ------------------------------------------------------------------------- |
| `model`       | `string`           | `"gpt-transcribe"` | Transcription model, for example `"whisper-1"`.                           |
| `language`    | `Language`         | `Language.EN`      | Transcription language.                                                   |
| `prompt`      | `string \| null`   | `null`             | Text that guides the style, or continues a previous segment.              |
| `temperature` | `number \| null`   | `null`             | Sampling temperature, 0 to 1.                                             |
| `keywords`    | `string[] \| null` | `null`             | Words or phrases to guide transcription. Supported by `"gpt-transcribe"`. |

### Usage [#usage]

```ts title="stt.ts"
import { Language } from "@pipedog/core/transcriptions/language";
import { OpenAISTTService } from "@pipedog/openai/stt";

const stt = new OpenAISTTService({
  apiKey: process.env.OPENAI_API_KEY ?? "",
  settings: new OpenAISTTService.Settings({
    model: "gpt-transcribe",
    language: Language.EN,
    keywords: ["pipedog", "Pipecat"],
  }),
});
```

Whisper, keeping empty transcripts:

```ts
const whisper = new OpenAISTTService({
  apiKey: process.env.OPENAI_API_KEY ?? "",
  settings: new OpenAISTTService.Settings({ model: "whisper-1", temperature: 0 }),
  pushEmptyTranscripts: true,
});
```

### Notes [#notes]

* A segmented service: it needs a VAD (in the user aggregator) to mark where each segment ends,
  then pushes one `TranscriptionFrame` per segment. There are no interim results.
* `pushEmptyTranscripts: true` pushes a `TranscriptionFrame` even when nothing was transcribed,
  so the bot can go on when the VAD fired on noise.
* With `includeProbMetrics: true`, GPT transcription models report per-token logprobs and
  Whisper models per-segment logprobs. Diarization models report none, with a warning.
* `httpClient` replaces the SDK's `fetch`, for example to set a longer request timeout.

## OpenAIRealtimeSTTService [#openairealtimesttservice]

### Configuration [#configuration-1]

| Option | Type | Required | Description |
|---|---|---|---|
| `apiKey` | `string` | yes | OpenAI API key for authentication. |
| `baseUrl` | `string \| undefined` | no | WebSocket base URL for the Realtime API. Defaults to "wss://api.openai.com/v1/realtime". |
| `turnDetection` | `false \| Dict \| null \| undefined` | no | Server-side VAD configuration. Defaults to `false` (disabled), which relies on a local VAD processor in the pipeline. Pass an object to enable it — at minimum `{ type: "server_vad" }`, optionally with custom settings (e.g. `{ type: "server_vad", threshold: 0.5 }`). Requires a model that supports turn detection. `null` omits the field entirely, leaving the session's own default in place. |
| `shouldInterrupt` | `boolean \| undefined` | no | Whether to interrupt bot output when speech is detected by server-side VAD. Only applies when turn detection is enabled. Passed along to the user turn strategies this service recommends, which own the interruption; a user-supplied `userTurnStrategies` overrides the recommendation and this setting with it. Defaults to true. |
| `settings` | `OpenAIRealtimeSTTSettings \| null \| undefined` | no | Runtime-updatable settings. |
| `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. |
| `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. |
| `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. |
| `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. |
| `reconnectBackoffMinWait` | `number \| undefined` | no | Minimum time, in seconds, to wait between reconnection attempts. |
| `reconnectBackoffMaxWait` | `number \| undefined` | no | Maximum time, in seconds, to wait between reconnection attempts. |
| `wsCloseTimeout` | `number \| undefined` | no | Maximum time, in seconds, to wait for the peer to acknowledge the websocket closing handshake before dropping the connection. Applied to connections opened through `_websocketConnect()`. Increase it for peers that need longer to complete a graceful close. |
| `wsSendTimeout` | `number \| null \| undefined` | no | Maximum time, in seconds, a single send may wait on a peer that has stopped reading before the connection is dropped and the send throws `ConnectionClosedError`. Applied to connections opened through `_websocketConnect()`. Pass `null` to wait indefinitely. Defaults to `WS_SEND_TIMEOUT`. |
| `audioPassthrough` | `boolean \| undefined` | no | Whether to pass audio frames downstream after processing. Defaults to true. |
| `sampleRate` | `number \| null \| undefined` | no | The sample rate for audio input. If null, will be determined from the start frame. |
| `sttTtfbTimeout` | `number \| undefined` | no | Time in seconds to wait after VAD stop before reporting TTFB. This delay allows the final transcription to arrive. Defaults to 2.0. |
| `ttfsP99Latency` | `number \| null \| undefined` | no | P99 latency from speech end to final transcript in seconds. Broadcast via STTMetadataFrame at pipeline start. |
| `keepaliveTimeout` | `number \| null \| undefined` | no | Seconds of no audio before sending silence to keep the connection alive. null disables keepalive. |
| `keepaliveInterval` | `number \| undefined` | no | Seconds between idle checks when keepalive is enabled. |
| `reconnectOnError` | `boolean \| undefined` | no | Whether to automatically reconnect on websocket errors. Defaults to true. |

### Settings [#settings-1]

Runtime-updatable settings, passed as `settings: new OpenAIRealtimeSTTService.Settings({...})`
(`OpenAIRealtimeSTTSettings`).

| Setting          | Type                                                          | Default                  | Description                                                                                                              |
| ---------------- | ------------------------------------------------------------- | ------------------------ | ------------------------------------------------------------------------------------------------------------------------ |
| `model`          | `string`                                                      | `"gpt-realtime-whisper"` | Transcription model.                                                                                                     |
| `language`       | `Language`                                                    | `Language.EN`            | Transcription language, sent as its ISO-639-1 code.                                                                      |
| `prompt`         | `string \| null`                                              | `null`                   | Text that guides the style. Not supported by `"gpt-realtime-whisper"`, so it is dropped (with a warning) for that model. |
| `noiseReduction` | `"near_field" \| "far_field" \| null`                         | `null`                   | Noise reduction for close or distant microphones. `null` turns it off.                                                   |
| `delay`          | `"minimal" \| "low" \| "medium" \| "high" \| "xhigh" \| null` | `null`                   | How long the model waits before emitting text, trading latency for accuracy. Only `"gpt-realtime-whisper"`.              |

### Usage [#usage-1]

```ts title="realtime.ts"
import { Language } from "@pipedog/core/transcriptions/language";
import { OpenAIRealtimeSTTService } from "@pipedog/openai/stt";

// Local VAD (the default): a VAD in the user aggregator ends each segment.
const stt = new OpenAIRealtimeSTTService({
  apiKey: process.env.OPENAI_API_KEY ?? "",
  settings: new OpenAIRealtimeSTTService.Settings({
    model: "gpt-realtime-whisper",
    language: Language.EN,
    noiseReduction: "near_field",
    delay: "low",
  }),
});
```

Server-side VAD. OpenAI detects the turns, so the pipeline needs no local VAD:

```ts
// OpenAI detects the turns. Use a model with turn detection, and no local VAD.
const serverVad = new OpenAIRealtimeSTTService({
  apiKey: process.env.OPENAI_API_KEY ?? "",
  turnDetection: { type: "server_vad", threshold: 0.5 },
  settings: new OpenAIRealtimeSTTService.Settings({ model: "gpt-transcribe" }),
});
```

Settings sent mid-call with an `STTUpdateSettingsFrame` are applied with a `session.update` on
the open session.

### Event handlers [#event-handlers]

| Event                 | Description              |
| --------------------- | ------------------------ |
| `on_connected`        | The WebSocket connected. |
| `on_disconnected`     | The WebSocket closed.    |
| `on_connection_error` | The connection failed.   |

### Notes [#notes-1]

* Emits `InterimTranscriptionFrame`s as text arrives and a `TranscriptionFrame` for each
  completed segment.
* `turnDetection` defaults to `false`: a local VAD drives turns, and each
  `VADUserStoppedSpeakingFrame` commits the audio buffer so the server transcribes the segment.
* With `turnDetection: { type: "server_vad" }`, the service broadcasts
  `ProposedUserStartedSpeakingFrame` and `ProposedUserStoppedSpeakingFrame` from the server's
  speech events, and asks the user aggregator for `ExternalUserTurnStrategies` (unless you pass
  your own `userTurnStrategies`). `shouldInterrupt` (default `true`) decides whether those turns
  interrupt the bot. Server VAD needs a model with turn detection: `gpt-transcribe` has it,
  `gpt-realtime-whisper` does not. See [turn detection](https://polvallverdu.github.io/pipedog/docs/concepts/turn-detection.md).
* Audio is sent as 24 kHz 16-bit mono PCM. Other input rates are resampled.
* The [Cloudflare Workers example](https://polvallverdu.github.io/pipedog/docs/deployment/cloudflare-workers.md) uses this service with
  server VAD, so the Worker loads no local VAD model.

## Runtime notes [#runtime-notes]

| Runtime            | Works | What it needs |
| ------------------ | ----- | ------------- |
| Bun                | ✅ yes | Nothing.      |
| Node.js            | ✅ yes | Nothing.      |
| Deno               | ✅ yes | Nothing.      |
| Cloudflare Workers | ✅ yes | Nothing.      |

Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/openai/package.json`: ✅ means the compat smoke test passes on that runtime.

* `OpenAISTTService` uses the `openai` JS SDK over `fetch`. It works on Bun, Node, Deno and
  Cloudflare Workers.
* `OpenAIRealtimeSTTService` sends the key as an `Authorization` header on the WebSocket
  handshake. Bun, Node and Deno send it from the global `WebSocket`; on Cloudflare Workers core
  opens the socket through `fetch`.
* The deprecated `model`, `language`, `prompt`, `temperature` and `noiseReduction` constructor
  options are not ported. Use `settings`.