# Together AI

> Realtime speech-to-text with TogetherSTTService, over Together AI's WebSocket API.

Source: https://polvallverdu.github.io/pipedog/docs/services/stt/together/



## Overview [#overview]

* **`TogetherSTTService`**: streams audio over a WebSocket to Together's realtime transcription endpoint and pushes interim and final transcriptions.

<Cards>
  <Card title="Upstream page" href="https://docs.pipecat.ai/api-reference/server/services/stt/together">
    Pipecat's documentation for the same service.
  </Card>

  <Card title="Source" href="https://github.com/polvallverdu/pipedog/tree/main/packages/together/src">
    `packages/together/src/stt.ts`.
  </Card>

  <Card title="Together AI docs" href="https://docs.together.ai/">
    Speech-to-text models and the realtime API.
  </Card>
</Cards>

## Installation [#installation]

```bash
bun add @pipedog/together
```

```ts
import { TogetherSTTService } from "@pipedog/together/stt";
```

## Prerequisites [#prerequisites]

* A Together AI account and an API key from [Together AI](https://together.ai/).
* Pass the key as `apiKey`. The examples read it from `TOGETHER_API_KEY`.

## TogetherSTTService [#togethersttservice]

Connects to `wss://api.together.ai/v1/realtime?intent=transcription`, with `Authorization: Bearer <key>`.

### Configuration [#configuration]

| Option | Type | Required | Description |
|---|---|---|---|
| `ttfsP99Latency` | `number \| null \| undefined` | no | P99 latency from speech end to final transcript in seconds. Broadcast via STTMetadataFrame at pipeline start. |
| `apiKey` | `string` | yes | Together AI API key for authentication. |
| `baseUrl` | `string \| undefined` | no | The URL of the Together AI WebSocket API. Defaults to "wss://api.together.ai/v1". |
| `settings` | `TogetherSTTSettings \| null \| undefined` | no | Runtime-updatable settings for the STT service. |
| `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. |
| `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. |
| `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. |
| `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. |
| `reconnectBackoffMinWait` | `number \| undefined` | no | Minimum time, in seconds, to wait between reconnection attempts. |
| `reconnectBackoffMaxWait` | `number \| undefined` | no | Maximum time, in seconds, to wait between reconnection attempts. |
| `wsCloseTimeout` | `number \| undefined` | no | Maximum time, in seconds, to wait for the peer to acknowledge the websocket closing handshake before dropping the connection. Applied to connections opened through `_websocketConnect()`. Increase it for peers that need longer to complete a graceful close. |
| `wsSendTimeout` | `number \| null \| undefined` | no | Maximum time, in seconds, a single send may wait on a peer that has stopped reading before the connection is dropped and the send throws `ConnectionClosedError`. Applied to connections opened through `_websocketConnect()`. Pass `null` to wait indefinitely. Defaults to `WS_SEND_TIMEOUT`. |
| `audioPassthrough` | `boolean \| undefined` | no | Whether to pass audio frames downstream after processing. Defaults to true. |
| `sampleRate` | `number \| null \| undefined` | no | The sample rate for audio input. If null, will be determined from the start frame. |
| `sttTtfbTimeout` | `number \| undefined` | no | Time in seconds to wait after VAD stop before reporting TTFB. This delay allows the final transcription to arrive. Defaults to 2.0. |
| `keepaliveTimeout` | `number \| null \| undefined` | no | Seconds of no audio before sending silence to keep the connection alive. null disables keepalive. |
| `keepaliveInterval` | `number \| undefined` | no | Seconds between idle checks when keepalive is enabled. |
| `reconnectOnError` | `boolean \| undefined` | no | Whether to automatically reconnect on websocket errors. Defaults to true. |

### Settings [#settings]

Runtime-updatable settings, passed as `settings: new TogetherSTTService.Settings({...})`
(`TogetherSTTSettings`). Any setting you leave out keeps the default below.

| Setting    | Type               | Default                     | Description            |
| ---------- | ------------------ | --------------------------- | ---------------------- |
| `model`    | `string`           | `"openai/whisper-large-v3"` | Transcription model.   |
| `language` | `Language \| null` | `Language.EN`               | Language of the audio. |

### Usage [#usage]

```ts title="stt.ts"
import { STTUpdateSettingsFrame } from "@pipedog/core/frames/frames";
import { TogetherSTTService } from "@pipedog/together/stt";

const stt = new TogetherSTTService({
  apiKey: process.env.TOGETHER_API_KEY ?? "",
  settings: new TogetherSTTService.Settings({ model: "openai/whisper-large-v3" }),
});
```

Change settings mid-call with an `STTUpdateSettingsFrame`:

```ts
// Queue it on the worker, or push it from a processor upstream of the service.
const update = new STTUpdateSettingsFrame({
  delta: new TogetherSTTService.Settings({ model: "openai/whisper-large-v3" }),
});
```

### Event handlers [#event-handlers]

| Event             | Description              |
| ----------------- | ------------------------ |
| `on_connected`    | The WebSocket connected. |
| `on_disconnected` | The WebSocket closed.    |

### Notes [#notes]

* Audio is sent as base64 16-bit PCM. When the user stops speaking, the buffer is committed so the final transcript arrives.
* Any settings change reconnects, since the session is configured in the connection URL.
* If the connection drops, the base class reconnects.

## Runtime notes [#runtime-notes]

| Runtime            | Works | What it needs |
| ------------------ | ----- | ------------- |
| Bun                | ✅ yes | Nothing.      |
| Node.js            | ✅ yes | Nothing.      |
| Deno               | ✅ yes | Nothing.      |
| Cloudflare Workers | ✅ yes | Nothing.      |

Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/together/package.json`: ✅ means the compat smoke test passes on that runtime.

* The WebSocket goes through the core `websocketConnect`, with headers on the handshake. It works on Bun, Node, Deno and Cloudflare Workers.
* Tested against a local WebSocket fake, not against Together AI itself.