# Cartesia

> Streaming speech-to-text with CartesiaSTTService (Ink) and CartesiaTurnsSTTService (Ink-2, with server-side turn detection and eager end of turn).

Source: https://polvallverdu.github.io/pipedog/docs/services/stt/cartesia/



## Overview [#overview]

Cartesia streams transcripts over a WebSocket. pipedog ports both of Pipecat's Cartesia
speech-to-text services:

* **`CartesiaSTTService`**: Cartesia's live STT API (`/stt/websocket`), with interim and final
  transcripts, runtime model and language switching, and keyterms on ink-2 models.
* **`CartesiaTurnsSTTService`**: Cartesia's turn-based Streaming ASR v2 API
  (`/stt/turns/websocket`, Ink-2). The server decides when the user's turn starts and ends, with
  optional eager end-of-turn predictions.

<Cards>
  <Card title="Upstream page" href="https://docs.pipecat.ai/api-reference/server/services/stt/cartesia">
    Pipecat's documentation for the same services.
  </Card>

  <Card title="Source" href="https://github.com/polvallverdu/pipedog/tree/main/packages/cartesia/src">
    `packages/cartesia/src/stt.ts` and `turns/stt.ts`.
  </Card>

  <Card title="Cartesia docs" href="https://docs.cartesia.ai/api-reference/stt/stt">
    STT models, keyterms and turn detection.
  </Card>
</Cards>

## Installation [#installation]

```bash
bun add @pipedog/cartesia
```

```ts
import { CartesiaSTTService } from "@pipedog/cartesia/stt";
import { CartesiaTurnsSTTService } from "@pipedog/cartesia/turns/stt";
```

## Prerequisites [#prerequisites]

* A Cartesia account and an API key from the [Cartesia playground](https://play.cartesia.ai).
* Pass the key as `apiKey`. The examples read it from `CARTESIA_API_KEY`.

## CartesiaSTTService [#cartesiasttservice]

### Configuration [#configuration]

| Option | Type | Required | Description |
|---|---|---|---|
| `apiKey` | `string` | yes | Authentication key for Cartesia API. |
| `baseUrl` | `string \| undefined` | no | Custom API endpoint, either a bare host ("api.cartesia.ai") or a URL with a scheme ("ws://localhost:8000"). If empty, uses default. |
| `encoding` | `string \| undefined` | no | Audio encoding format. Defaults to "pcm_s16le". |
| `extraHeaders` | `Record<string, string> \| null \| undefined` | no | Additional headers to send on the websocket handshake. Merged after the default headers, so it can override them. |
| `settings` | `CartesiaSTTSettings \| null \| undefined` | no | Runtime-updatable settings. |
| `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. |
| `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. |
| `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. |
| `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. |
| `reconnectBackoffMinWait` | `number \| undefined` | no | Minimum time, in seconds, to wait between reconnection attempts. |
| `reconnectBackoffMaxWait` | `number \| undefined` | no | Maximum time, in seconds, to wait between reconnection attempts. |
| `wsCloseTimeout` | `number \| undefined` | no | Maximum time, in seconds, to wait for the peer to acknowledge the websocket closing handshake before dropping the connection. Applied to connections opened through `_websocketConnect()`. Increase it for peers that need longer to complete a graceful close. |
| `wsSendTimeout` | `number \| null \| undefined` | no | Maximum time, in seconds, a single send may wait on a peer that has stopped reading before the connection is dropped and the send throws `ConnectionClosedError`. Applied to connections opened through `_websocketConnect()`. Pass `null` to wait indefinitely. Defaults to `WS_SEND_TIMEOUT`. |
| `audioPassthrough` | `boolean \| undefined` | no | Whether to pass audio frames downstream after processing. Defaults to true. |
| `sampleRate` | `number \| null \| undefined` | no | The sample rate for audio input. If null, will be determined from the start frame. |
| `sttTtfbTimeout` | `number \| undefined` | no | Time in seconds to wait after VAD stop before reporting TTFB. This delay allows the final transcription to arrive. Defaults to 2.0. |
| `ttfsP99Latency` | `number \| null \| undefined` | no | P99 latency from speech end to final transcript in seconds. Broadcast via STTMetadataFrame at pipeline start. |
| `keepaliveTimeout` | `number \| null \| undefined` | no | Seconds of no audio before sending silence to keep the connection alive. null disables keepalive. |
| `keepaliveInterval` | `number \| undefined` | no | Seconds between idle checks when keepalive is enabled. |
| `reconnectOnError` | `boolean \| undefined` | no | Whether to automatically reconnect on websocket errors. Defaults to true. |

### Settings [#settings]

Runtime-updatable settings, passed as `settings: new CartesiaSTTService.Settings({...})`. Any
setting you leave out keeps the default below.

| Setting    | Type               | Default         | Description                                                                          |
| ---------- | ------------------ | --------------- | ------------------------------------------------------------------------------------ |
| `model`    | `string`           | `"ink-whisper"` | Cartesia STT model.                                                                  |
| `language` | `Language`         | `Language.EN`   | Transcription language.                                                              |
| `keyterm`  | `string[] \| null` | `null`          | Terms to bias transcription towards. Only `ink-2` and `ink-preview` models use them. |

### Usage [#usage]

```ts title="stt.ts"
import { CartesiaSTTService } from "@pipedog/cartesia/stt";
import { STTUpdateSettingsFrame } from "@pipedog/core/frames/frames";
import { Language } from "@pipedog/core/transcriptions/language";

const stt = new CartesiaSTTService({
  apiKey: process.env.CARTESIA_API_KEY ?? "",
  settings: new CartesiaSTTService.Settings({
    model: "ink-whisper",
    language: Language.EN,
  }),
});
```

Keyterms, on an ink-2 model:

```ts
// Keyterms only work on ink-2 and ink-preview models.
const withKeyterms = new CartesiaSTTService({
  apiKey: process.env.CARTESIA_API_KEY ?? "",
  settings: new CartesiaSTTService.Settings({
    model: "ink-2",
    keyterm: ["pipedog", "Pipecat"],
  }),
});
```

Change settings mid-call with an `STTUpdateSettingsFrame`. The service reconnects with the new
settings:

```ts
// Queue it on the worker, or push it from a processor upstream of the service.
const update = new STTUpdateSettingsFrame({
  delta: new CartesiaSTTService.Settings({ language: Language.ES }),
});
```

### Event handlers [#event-handlers]

Inherited from `STTService`:

| Event                 | Description                 |
| --------------------- | --------------------------- |
| `on_connected`        | Connected to Cartesia.      |
| `on_disconnected`     | Disconnected from Cartesia. |
| `on_connection_error` | The connection failed.      |

### Notes [#notes]

* Emits `InterimTranscriptionFrame` while the user speaks and `TranscriptionFrame` for final
  results. A `VADUserStoppedSpeakingFrame` makes it send `finalize`, which flushes a final result.
* Cartesia closes a socket after 3 minutes without messages. The service sends silence as a
  keepalive (after 120 s idle, checked every 30 s).
* A settings change reconnects. While the user is speaking, the reconnect waits until they stop.
* Keyterms set for a model other than `ink-2` or `ink-preview` are ignored with a warning.
  Cartesia accepts at most 100 keyterms totaling 1200 characters; extra ones are dropped with a
  warning.
* `baseUrl` takes a bare host (`"api.cartesia.ai"`) or a URL with a scheme
  (`"ws://localhost:8000"`).

## CartesiaTurnsSTTService [#cartesiaturnssttservice]

### Configuration [#configuration-1]

| Option | Type | Required | Description |
|---|---|---|---|
| `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. |
| `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. |
| `apiKey` | `string` | yes | Cartesia API key. |
| `url` | `string \| undefined` | no | WebSocket URL for the Cartesia Streaming ASR v2 endpoint. |
| `shouldInterrupt` | `boolean \| undefined` | no | Whether to interrupt the bot when the server signals the start of a new turn. Passed along to the user turn strategies this service recommends, which own the interruption; a user-supplied `userTurnStrategies` overrides the recommendation and this setting with it. |
| `watchdogMinTimeout` | `number \| undefined` | no | Minimum idle timeout before sending silence to prevent dangling turns. The actual threshold is `max(chunkDuration * 2, watchdogMinTimeout)`. Defaults to 0.5. |
| `extraHeaders` | `Record<string, string> \| null \| undefined` | no | Optional additional HTTP headers to send with the WebSocket handshake. |
| `settings` | `CartesiaTurnsSTTSettings \| null \| undefined` | no | Runtime-updatable settings. The ink-2 family does not support runtime model or language switching; attempts to update either field will be reported as unhandled. |
| `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. |
| `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. |
| `reconnectBackoffMinWait` | `number \| undefined` | no | Minimum time, in seconds, to wait between reconnection attempts. |
| `reconnectBackoffMaxWait` | `number \| undefined` | no | Maximum time, in seconds, to wait between reconnection attempts. |
| `wsCloseTimeout` | `number \| undefined` | no | Maximum time, in seconds, to wait for the peer to acknowledge the websocket closing handshake before dropping the connection. Applied to connections opened through `_websocketConnect()`. Increase it for peers that need longer to complete a graceful close. |
| `wsSendTimeout` | `number \| null \| undefined` | no | Maximum time, in seconds, a single send may wait on a peer that has stopped reading before the connection is dropped and the send throws `ConnectionClosedError`. Applied to connections opened through `_websocketConnect()`. Pass `null` to wait indefinitely. Defaults to `WS_SEND_TIMEOUT`. |
| `audioPassthrough` | `boolean \| undefined` | no | Whether to pass audio frames downstream after processing. Defaults to true. |
| `sampleRate` | `number \| null \| undefined` | no | The sample rate for audio input. If null, will be determined from the start frame. |
| `sttTtfbTimeout` | `number \| undefined` | no | Time in seconds to wait after VAD stop before reporting TTFB. This delay allows the final transcription to arrive. Defaults to 2.0. |
| `ttfsP99Latency` | `number \| null \| undefined` | no | P99 latency from speech end to final transcript in seconds. Broadcast via STTMetadataFrame at pipeline start. |
| `keepaliveTimeout` | `number \| null \| undefined` | no | Seconds of no audio before sending silence to keep the connection alive. null disables keepalive. |
| `keepaliveInterval` | `number \| undefined` | no | Seconds between idle checks when keepalive is enabled. |
| `reconnectOnError` | `boolean \| undefined` | no | Whether to automatically reconnect on websocket errors. Defaults to true. |
| `enableEagerEndOfTurn` | `boolean \| undefined` | no | Whether to answer a predicted end of turn ahead of the committed one. Off by default: it spends an inference on every prediction, including the ones the service withdraws. |
| `eagerMatchPolicy` | `EagerMatchPolicy \| null \| undefined` | no | Decides whether the committed transcript is close enough to the predicted one to keep the speculative response. Only consulted while eager end of turn is on. See EagerUserTurnStrategies. |

### Settings [#settings-1]

| Setting                 | Type               | Default                 | Description                                                                 |
| ----------------------- | ------------------ | ----------------------- | --------------------------------------------------------------------------- |
| `model`                 | `string`           | `"ink-2"`               | Cartesia model. Cannot change at runtime.                                   |
| `language`              | `Language \| null` | `null`                  | Not used: ink-2 is English-only. Emitted frames carry `Language.EN`.        |
| `keyterm`               | `string[] \| null` | `null`                  | Terms to bias transcription towards.                                        |
| `turnStartThreshold`    | `number \| null`   | `null` (Cartesia: 0.8)  | Likelihood above which the server emits `turn.start`. Range 0.5 to 0.9.     |
| `turnEagerEndThreshold` | `number \| null`   | `null` (Cartesia: 0.6)  | Likelihood below which the server emits `turn.eager_end`. Range 0.3 to 0.8. |
| `turnEndThreshold`      | `number \| null`   | `null` (Cartesia: 0.3)  | Likelihood below which the server emits `turn.end`. Range 0.05 to 0.5.      |
| `turnEndTimeoutMs`      | `number \| null`   | `null` (Cartesia: 5600) | Wait after the user stops speaking before `turn.end`. Range 640 to 11200.   |

Keep the thresholds in order: `turnStartThreshold` > `turnEagerEndThreshold` > `turnEndThreshold`.
Changing `keyterm` or a `turn*` setting at runtime reconnects. A `model` or `language` update is
reported as unhandled.

### Usage [#usage-1]

```ts title="turns.ts"
import { CartesiaTurnsSTTService } from "@pipedog/cartesia/turns/stt";

const stt = new CartesiaTurnsSTTService({
  apiKey: process.env.CARTESIA_API_KEY ?? "",
  settings: new CartesiaTurnsSTTService.Settings({
    keyterm: ["pipedog"],
    turnEndTimeoutMs: 2000,
  }),
});

stt.addEventHandler("on_turn_end", async (_service, transcript: string) => {
  console.log("end of turn:", transcript);
});
```

With eager end of turn:

```ts
// Answer the server's predicted end of turn ahead of the committed one.
const eager = new CartesiaTurnsSTTService({
  apiKey: process.env.CARTESIA_API_KEY ?? "",
  enableEagerEndOfTurn: true,
  settings: new CartesiaTurnsSTTService.Settings({
    turnStartThreshold: 0.8,
    turnEagerEndThreshold: 0.6,
    turnEndThreshold: 0.3,
  }),
});

eager.addEventHandler("on_turn_eager_end", async (_service, transcript: string) => {
  console.log("predicted end of turn:", transcript);
});
eager.addEventHandler("on_turn_resume", async () => {
  console.log("user kept speaking");
});
```

### Event handlers [#event-handlers-1]

| Event               | Description                                   |
| ------------------- | --------------------------------------------- |
| `on_turn_start`     | The server detected the start of a turn.      |
| `on_turn_update`    | A transcript update for the turn in progress. |
| `on_turn_eager_end` | The server predicted the end of the turn.     |
| `on_turn_resume`    | The user kept speaking after an eager end.    |
| `on_turn_end`       | The final transcript for the turn.            |

`on_connected`, `on_disconnected` and `on_connection_error` are inherited from `STTService`.

### Notes [#notes-1]

* The server drives turns: `turn.start` broadcasts a `ProposedUserStartedSpeakingFrame`,
  `turn.update` pushes an `InterimTranscriptionFrame`, and `turn.end` pushes a
  `TranscriptionFrame` and broadcasts a `ProposedUserStoppedSpeakingFrame`. Transcripts are
  cumulative per turn. There is no `finalize` command.
* The service recommends `ExternalUserTurnStrategies` to the user aggregator, or
  `EagerUserTurnStrategies` with `enableEagerEndOfTurn`, so no local VAD or smart-turn model is
  needed. Passing your own `userTurnStrategies` overrides this. See
  [Turn detection](https://polvallverdu.github.io/pipedog/docs/concepts/turn-detection.md).
* With `enableEagerEndOfTurn: true`, `turn.eager_end` pushes an `EagerTranscriptionFrame` that
  starts a response early. `turn.resume` pushes a `UserTurnResponseCancelFrame`. The early
  response is dropped if the user resumes or the final transcript differs. It is off by default
  because every prediction costs an inference.
* `shouldInterrupt` (default `true`) decides whether the start of a turn interrupts the bot.
* If audio stops after a `turn.start`, a watchdog sends silence so the turn does not hang. The
  idle threshold is `max(chunkDuration * 2, watchdogMinTimeout)`, with `watchdogMinTimeout`
  defaulting to 0.5 s.
* The service waits up to 10 s for Cartesia's `connected` message after the handshake.

## Runtime notes [#runtime-notes]

| Runtime            | Works | What it needs |
| ------------------ | ----- | ------------- |
| Bun                | ✅ yes | Nothing.      |
| Node.js            | ✅ yes | Nothing.      |
| Deno               | ✅ yes | Nothing.      |
| Cloudflare Workers | ✅ yes | Nothing.      |

Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/cartesia/package.json`: ✅ means the compat smoke test passes on that runtime.

Both services use the global `WebSocket` and send the key as an `X-API-Key` header on the
handshake. The Bun, Node and Deno `WebSocket` support handshake headers, and on Cloudflare
Workers the socket is opened through `fetch` with an `Upgrade` header. A standard `WebSocket`
does not expose the handshake response, so the `X-Request-ID` header that upstream logs on
connect is always logged as `None`. The deprecated `CartesiaLiveOptions` class and the
`liveOptions` constructor option are not ported; use `settings`.