# Smallest AI

> Realtime speech-to-text with SmallestSTTService, over Smallest AI's Pulse WebSocket API.

Source: https://polvallverdu.github.io/pipedog/docs/services/stt/smallest/



## Overview [#overview]

* **`SmallestSTTService`**: streams audio over a persistent WebSocket to Smallest AI's Pulse speech-to-text endpoint. Interim transcripts become `InterimTranscriptionFrame`s; when the VAD sees the user stop, the service sends `finalize` and the final becomes a `TranscriptionFrame`.

<Cards>
  <Card title="Upstream page" href="https://docs.pipecat.ai/api-reference/server/services/stt/smallest">
    Pipecat's documentation for the same service.
  </Card>

  <Card title="Source" href="https://github.com/polvallverdu/pipedog/tree/main/packages/smallest/src">
    `packages/smallest/src/stt.ts`.
  </Card>

  <Card title="Smallest AI docs" href="https://docs.smallest.ai/waves/documentation/speech-to-text-pulse/realtime-web-socket">
    The realtime WebSocket reference.
  </Card>
</Cards>

## Installation [#installation]

```bash
bun add @pipedog/smallest
```

```ts
import { SmallestSTTService } from "@pipedog/smallest/stt";
```

## Prerequisites [#prerequisites]

* A Smallest AI account and an API key from [Smallest AI](https://smallest.ai/).
* Pass the key as `apiKey`. The examples read it from `SMALLEST_API_KEY`.

## SmallestSTTService [#smalleststtservice]

Connects to `wss://api.smallest.ai/waves/v1/stt/live`, with the key as a Bearer token on the handshake.

### Configuration [#configuration]

| Option | Type | Required | Description |
|---|---|---|---|
| `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. |
| `apiKey` | `string` | yes | Smallest AI API key for authentication. |
| `baseUrl` | `string \| undefined` | no | Base WebSocket URL for the Smallest API. Defaults to "wss://api.smallest.ai". |
| `encoding` | `string \| undefined` | no | Audio encoding format. Defaults to "linear16". |
| `settings` | `SmallestSTTSettings \| null \| undefined` | no | Runtime-updatable settings for the STT service. |
| `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. |
| `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. |
| `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. |
| `reconnectBackoffMinWait` | `number \| undefined` | no | Minimum time, in seconds, to wait between reconnection attempts. |
| `reconnectBackoffMaxWait` | `number \| undefined` | no | Maximum time, in seconds, to wait between reconnection attempts. |
| `wsCloseTimeout` | `number \| undefined` | no | Maximum time, in seconds, to wait for the peer to acknowledge the websocket closing handshake before dropping the connection. Applied to connections opened through `_websocketConnect()`. Increase it for peers that need longer to complete a graceful close. |
| `wsSendTimeout` | `number \| null \| undefined` | no | Maximum time, in seconds, a single send may wait on a peer that has stopped reading before the connection is dropped and the send throws `ConnectionClosedError`. Applied to connections opened through `_websocketConnect()`. Pass `null` to wait indefinitely. Defaults to `WS_SEND_TIMEOUT`. |
| `audioPassthrough` | `boolean \| undefined` | no | Whether to pass audio frames downstream after processing. Defaults to true. |
| `sampleRate` | `number \| null \| undefined` | no | The sample rate for audio input. If null, will be determined from the start frame. |
| `sttTtfbTimeout` | `number \| undefined` | no | Time in seconds to wait after VAD stop before reporting TTFB. This delay allows the final transcription to arrive. Defaults to 2.0. |
| `ttfsP99Latency` | `number \| null \| undefined` | no | P99 latency from speech end to final transcript in seconds. Broadcast via STTMetadataFrame at pipeline start. |
| `keepaliveTimeout` | `number \| null \| undefined` | no | Seconds of no audio before sending silence to keep the connection alive. null disables keepalive. |
| `keepaliveInterval` | `number \| undefined` | no | Seconds between idle checks when keepalive is enabled. |
| `reconnectOnError` | `boolean \| undefined` | no | Whether to automatically reconnect on websocket errors. Defaults to true. |

### Settings [#settings]

Runtime-updatable settings, passed as `settings: new SmallestSTTService.Settings({...})`
(`SmallestSTTSettings`). Any setting you leave out keeps the default below.

| Setting              | Type                   | Default       | Description                                                                                     |
| -------------------- | ---------------------- | ------------- | ----------------------------------------------------------------------------------------------- |
| `model`              | `"pulse" \| "pulse-2"` | `"pulse"`     | `pulse-2` is English-only and adds per-utterance emotion and gender.                            |
| `language`           | `Language`             | `Language.EN` | Language of the audio. A non-English language with `pulse-2` throws.                            |
| `wordTimestamps`     | `boolean`              | `false`       | Word-level timestamps.                                                                          |
| `fullTranscript`     | `boolean`              | `false`       | Include the cumulative transcript.                                                              |
| `sentenceTimestamps` | `boolean`              | `false`       | Sentence-level timestamps.                                                                      |
| `redactPii`          | `boolean`              | `false`       | Redact personally identifiable information.                                                     |
| `redactPci`          | `boolean`              | `false`       | Redact payment card information.                                                                |
| `numerals`           | `string`               | `"auto"`      | Convert spoken numerals to digits.                                                              |
| `diarize`            | `boolean`              | `false`       | Speaker diarization.                                                                            |
| `endpointing`        | `boolean`              | `true`        | Finalize promptly on trailing silence. No effect on `pulse-2`.                                  |
| `keywords`           | `string`               | `""`          | Comma-separated `KEYWORD:INTENSIFIER` pairs, such as `"NVIDIA:2"`.                              |
| `format`             | `boolean`              | `true`        | Punctuation and capitalization.                                                                 |
| `emotionDetection`   | `boolean`              | `true`        | `pulse-2` only: a six-class `emotion` on each final.                                            |
| `genderDetection`    | `boolean`              | `true`        | `pulse-2` only: a `gender` estimate on each final.                                              |
| `itnNormalize`       | `boolean`              | `false`       | Inverse text normalization ("twenty five" becomes "25").                                        |
| `noFormatDelay`      | `boolean`              | `false`       | With `itnNormalize`, emit finals at once instead of holding back a number still being dictated. |

### Usage [#usage]

```ts title="stt.ts"
import { Language } from "@pipedog/core/transcriptions/language";
import { SmallestSTTService } from "@pipedog/smallest/stt";

const stt = new SmallestSTTService({
  apiKey: process.env.SMALLEST_API_KEY ?? "",
  settings: new SmallestSTTService.Settings({ language: Language.EN, wordTimestamps: true }),
});
```

Pulse 2.0, English only:

```ts
const pulse2 = new SmallestSTTService({
  apiKey: process.env.SMALLEST_API_KEY ?? "",
  settings: new SmallestSTTService.Settings({ model: "pulse-2" }),
});

```

Change settings mid-call with an `STTUpdateSettingsFrame`:

```ts
import { STTUpdateSettingsFrame } from "@pipedog/core/frames/frames";

// Queue it on the worker, or push it from a processor upstream of the service.
const update = new STTUpdateSettingsFrame({
  delta: new SmallestSTTService.Settings({ redactPii: true }),
});
```

### Event handlers [#event-handlers]

| Event             | Description              |
| ----------------- | ------------------------ |
| `on_connected`    | The WebSocket connected. |
| `on_disconnected` | The WebSocket closed.    |

### Notes [#notes]

* `encoding` is `"linear16"` by default. `sampleRate` defaults to the pipeline's input rate.
* Settings are query parameters of the connection, so changing one reconnects.
* The same package has Smallest AI's [LLM](https://polvallverdu.github.io/pipedog/docs/services/llm/smallest.md) and [text-to-speech](https://polvallverdu.github.io/pipedog/docs/services/tts/smallest.md) services.

| Runtime            | Works | What it needs |
| ------------------ | ----- | ------------- |
| Bun                | ✅ yes | Nothing.      |
| Node.js            | ✅ yes | Nothing.      |
| Deno               | ✅ yes | Nothing.      |
| Cloudflare Workers | ✅ yes | Nothing.      |

Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/smallest/package.json`: ✅ means the compat smoke test passes on that runtime.

## Runtime notes [#runtime-notes]

* The WebSocket goes through the core `websocketConnect`, with headers on the handshake.
* Upstream sets a keepalive interval without implementing the keepalive message, so an idle connection logs a warning every few seconds. The port behaves the same way.
* Tested against local WebSocket servers, as upstream's tests do, not against Smallest AI itself.