# Sarvam AI

> Speech-to-text for Indian languages with SarvamSTTService and SarvamRealtimeSTTService, over Sarvam's WebSocket APIs.

Source: https://polvallverdu.github.io/pipedog/docs/services/stt/sarvam/



## Overview [#overview]

* **`SarvamSTTService`**: streams audio to Sarvam's speech-to-text WebSocket (`saaras:v4` by default, or `saaras:v3`; see `MODEL_CONFIGS`). Sarvam's VAD signals can drive the turn.
* **`SarvamRealtimeSTTService`**: Sarvam's realtime endpoint (`saaras:v3-realtime` or `saaras:v4`), with the server or the pipeline marking turn boundaries, and reconnects.

<Cards>
  <Card title="Upstream page" href="https://docs.pipecat.ai/api-reference/server/services/stt/sarvam">
    Pipecat's documentation for the same service.
  </Card>

  <Card title="Source" href="https://github.com/polvallverdu/pipedog/tree/main/packages/sarvam/src">
    `packages/sarvam/src/stt.ts`.
  </Card>

  <Card title="Sarvam docs" href="https://docs.sarvam.ai/">
    The speech-to-text API reference.
  </Card>
</Cards>

## Installation [#installation]

```bash
bun add @pipedog/sarvam
```

```ts
import { SarvamRealtimeSTTService, SarvamSTTService } from "@pipedog/sarvam/stt";
```

## Prerequisites [#prerequisites]

* A Sarvam account and an API key from the [Sarvam dashboard](https://dashboard.sarvam.ai/).
* Pass the key as `apiKey`. The examples read it from `SARVAM_API_KEY`.

## SarvamSTTService [#sarvamsttservice]

Connects to `wss://api.sarvam.ai/speech-to-text/ws`, with the key in the `Api-Subscription-Key` header.

### Configuration [#configuration]

| Option | Type | Required | Description |
|---|---|---|---|
| `apiKey` | `string` | yes | Sarvam API key for authentication. |
| `mode` | `SarvamMode \| null \| undefined` | no | Mode of operation. Options: transcribe, translate, verbatim, translit, codemix. Only applicable to models that support it. Defaults to the model's default mode. |
| `sampleRate` | `number \| null \| undefined` | no | Audio sample rate. Defaults to the pipeline's input rate if not specified. |
| `inputAudioCodec` | `string \| undefined` | no | Audio codec/format of the input file. Defaults to "wav". |
| `keyterms` | `string[] \| null \| undefined` | no | Connection-time recognition hints for saaras:v4. At most 50 terms of 64 characters each; put each phrase in a separate item. |
| `baseUrl` | `string \| undefined` | no | WebSocket endpoint. Defaults to "wss://api.sarvam.ai/speech-to-text/ws". |
| `settings` | `SarvamSTTSettings \| null \| undefined` | no | Runtime-updatable settings. |
| `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. |
| `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. |
| `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. |
| `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. |
| `audioPassthrough` | `boolean \| undefined` | no | Whether to pass audio frames downstream after processing. Defaults to true. |
| `sttTtfbTimeout` | `number \| undefined` | no | Time in seconds to wait after VAD stop before reporting TTFB. This delay allows the final transcription to arrive. Defaults to 2.0. |
| `ttfsP99Latency` | `number \| null \| undefined` | no | P99 latency from speech end to final transcript in seconds. Broadcast via STTMetadataFrame at pipeline start. |
| `keepaliveTimeout` | `number \| null \| undefined` | no | Seconds of no audio before sending silence to keep the connection alive. null disables keepalive. |
| `keepaliveInterval` | `number \| undefined` | no | Seconds between idle checks when keepalive is enabled. |

### Settings [#settings]

Runtime-updatable settings, passed as `settings: new SarvamSTTService.Settings({...})`
(`SarvamSTTSettings`). They are the base STT settings (`model`, `language`) plus the VAD tuning
fields below, all `null` (the server's default) unless you set them.

| Setting                      | Type               | Default       | Description                                                                   |
| ---------------------------- | ------------------ | ------------- | ----------------------------------------------------------------------------- |
| `model`                      | `string`           | `"saaras:v4"` | Model. Its entry in `MODEL_CONFIGS` decides the modes and options that apply. |
| `language`                   | `Language \| null` | `null`        | Language of the audio. `null` lets models that can detect it do so.           |
| `vadSignals`                 | `boolean \| null`  | `null`        | Have Sarvam return VAD signals.                                               |
| `highVadSensitivity`         | `boolean \| null`  | `null`        | High VAD sensitivity.                                                         |
| `positiveSpeechThreshold`    | `number \| null`   | `null`        | Probability (0 to 1) above which a frame counts as speech.                    |
| `negativeSpeechThreshold`    | `number \| null`   | `null`        | Probability (0 to 1) below which a frame counts as silence.                   |
| `minSpeechFrames`            | `number \| null`   | `null`        | Consecutive speech frames that start a segment.                               |
| `firstTurnMinSpeechFrames`   | `number \| null`   | `null`        | The same, for the first turn.                                                 |
| `negativeFramesCount`        | `number \| null`   | `null`        | Silence frames within the window that end a segment.                          |
| `negativeFramesWindow`       | `number \| null`   | `null`        | Window, in frames, for counting silence frames.                               |
| `startSpeechVolumeThreshold` | `number \| null`   | `null`        | Volume (dB) below which audio is too quiet to be speech.                      |
| `interruptMinSpeechFrames`   | `number \| null`   | `null`        | Speech frames needed to register a barge-in.                                  |
| `preSpeechPadFrames`         | `number \| null`   | `null`        | Frames to prepend before the detected start of speech.                        |
| `numInitialIgnoredFrames`    | `number \| null`   | `null`        | Leading frames to skip at connection start.                                   |

### Usage [#usage]

```ts title="stt.ts"
import { Language } from "@pipedog/core/transcriptions/language";
import { SarvamSTTService } from "@pipedog/sarvam/stt";

const stt = new SarvamSTTService({
  apiKey: process.env.SARVAM_API_KEY ?? "",
  settings: new SarvamSTTService.Settings({ language: Language.HI_IN }),
});

```

### Event handlers [#event-handlers]

| Event                 | Description              |
| --------------------- | ------------------------ |
| `on_connected`        | The WebSocket connected. |
| `on_disconnected`     | The WebSocket closed.    |
| `on_connection_error` | The connection failed.   |

### Notes [#notes]

* `mode` is one of `transcribe`, `translate`, `verbatim`, `translit` and `codemix`, for the models that support it; it defaults to the model's own default.
* `keyterms` (at most 50 terms of 64 characters each) are connection-time recognition hints for `saaras:v4`.
* Only 8000 and 16000 Hz audio is accepted.
* Settings that go in the connection query string reconnect the socket when they change.

## SarvamRealtimeSTTService [#sarvamrealtimesttservice]

Connects to Sarvam's realtime speech-to-text WebSocket and maps its VAD and transcript events to frames. A dropped socket is reopened with backoff unless `reconnectOnError` is `false`.

### Configuration [#configuration-1]

| Option | Type | Required | Description |
|---|---|---|---|
| `reconnectOnError` | `boolean \| undefined` | no | Whether to automatically reconnect on websocket errors. Defaults to true. |
| `apiKey` | `string` | yes | Sarvam API key. |
| `baseUrl` | `string \| undefined` | no | Realtime STT websocket endpoint. |
| `endpointing` | `"manual" \| "vad" \| undefined` | no | Which side detects turn boundaries: `vad` for Sarvam's own detection, or `manual` for the pipeline's. Decides the turn strategies this service asks the user aggregator to run, so it is fixed for the life of the service. Defaults to `vad`. |
| `sampleRate` | `number \| null \| undefined` | no | Declared input audio sample rate, 8000 or 16000. `null` adopts the pipeline's input rate. |
| `returnTimestamps` | `boolean \| undefined` | no | Whether final transcripts should include segment offsets. Defaults to false. |
| `keyterms` | `string[] \| null \| undefined` | no | Connection-time recognition hints for saaras:v4. At most 50 terms of 64 characters each; put each phrase in a separate item. |
| `prefixPaddingMs` | `number \| null \| undefined` | no | Optional VAD prefix padding, used only under `endpointing="vad"`. |
| `settings` | `SarvamRealtimeSTTSettings \| null \| undefined` | no | Runtime-updatable realtime settings. |
| `shouldInterrupt` | `boolean \| undefined` | no | Determine whether the bot should be interrupted when Sarvam detects user speech. Passed along to the user turn strategies this service recommends, which own the interruption; a user-supplied `userTurnStrategies` overrides the recommendation and this setting with it. Defaults to true. |
| `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. |
| `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. |
| `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. |
| `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. |
| `reconnectBackoffMinWait` | `number \| undefined` | no | Minimum time, in seconds, to wait between reconnection attempts. |
| `reconnectBackoffMaxWait` | `number \| undefined` | no | Maximum time, in seconds, to wait between reconnection attempts. |
| `wsCloseTimeout` | `number \| undefined` | no | Maximum time, in seconds, to wait for the peer to acknowledge the websocket closing handshake before dropping the connection. Applied to connections opened through `_websocketConnect()`. Increase it for peers that need longer to complete a graceful close. |
| `wsSendTimeout` | `number \| null \| undefined` | no | Maximum time, in seconds, a single send may wait on a peer that has stopped reading before the connection is dropped and the send throws `ConnectionClosedError`. Applied to connections opened through `_websocketConnect()`. Pass `null` to wait indefinitely. Defaults to `WS_SEND_TIMEOUT`. |
| `audioPassthrough` | `boolean \| undefined` | no | Whether to pass audio frames downstream after processing. Defaults to true. |
| `sttTtfbTimeout` | `number \| undefined` | no | Time in seconds to wait after VAD stop before reporting TTFB. This delay allows the final transcription to arrive. Defaults to 2.0. |
| `ttfsP99Latency` | `number \| null \| undefined` | no | P99 latency from speech end to final transcript in seconds. Broadcast via STTMetadataFrame at pipeline start. |
| `keepaliveTimeout` | `number \| null \| undefined` | no | Seconds of no audio before sending silence to keep the connection alive. null disables keepalive. |
| `keepaliveInterval` | `number \| undefined` | no | Seconds between idle checks when keepalive is enabled. |

### Settings [#settings-1]

Passed as `settings: new SarvamRealtimeSTTService.Settings({...})` (`SarvamRealtimeSTTSettings`).

| Setting               | Type                                  | Default                | Description                                               |
| --------------------- | ------------------------------------- | ---------------------- | --------------------------------------------------------- |
| `model`               | `string`                              | `"saaras:v3-realtime"` | `saaras:v3-realtime` or `saaras:v4`.                      |
| `language`            | `Language \| null`                    | `Language.EN_IN`       | Language; `null` has Sarvam detect it for each utterance. |
| `streamType`          | `"fast" \| "balanced" \| "simulated"` | `"balanced"`           | Streaming cadence.                                        |
| `mode`                | `SarvamMode`                          | `"transcribe"`         | Realtime task mode.                                       |
| `prompt`              | `string \| null`                      | `null`                 | Decoding prompt.                                          |
| `threshold`           | `number \| null`                      | `null`                 | VAD sensitivity.                                          |
| `silenceDurationMs`   | `number \| null`                      | `null`                 | Silence that ends speech.                                 |
| `minSpeechDurationMs` | `number \| null`                      | `null`                 | Shortest speech that counts.                              |

### Usage [#usage-1]

```ts
import { SarvamRealtimeSTTService } from "@pipedog/sarvam/stt";

const realtime = new SarvamRealtimeSTTService({
  apiKey: process.env.SARVAM_API_KEY ?? "",
  endpointing: "vad",
  settings: new SarvamRealtimeSTTService.Settings({ language: Language.HI_IN }),
});

```

Change settings mid-call with an `STTUpdateSettingsFrame`:

```ts
import { STTUpdateSettingsFrame } from "@pipedog/core/frames/frames";

// Queue it on the worker, or push it from a processor upstream of the service.
const update = new STTUpdateSettingsFrame({
  delta: new SarvamRealtimeSTTService.Settings({ streamType: "fast" }),
});
```

### Notes [#notes-1]

* `endpointing: "vad"` (the default) lets the server mark turn boundaries. With `"manual"` the pipeline does, so it needs a turn strategy that emits `VADUserStartedSpeakingFrame` and `VADUserStoppedSpeakingFrame`; without one Sarvam never emits a final transcript. It's an option, not a setting, because it decides which turn strategies the user aggregator runs.
* A VAD analyzer is needed in both modes: it times the transcription latency.
* `sampleRate` is 8000 or 16000; `null` adopts the pipeline's input rate.

| Runtime            | Works | What it needs |
| ------------------ | ----- | ------------- |
| Bun                | ✅ yes | Nothing.      |
| Node.js            | ✅ yes | Nothing.      |
| Deno               | ✅ yes | Nothing.      |
| Cloudflare Workers | ✅ yes | Nothing.      |

Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/sarvam/package.json`: ✅ means the compat smoke test passes on that runtime.

## Runtime notes [#runtime-notes]

* Upstream uses the `sarvamai` Python SDK for `SarvamSTTService`. Here it is a plain WebSocket through the core `websocketConnect` (which sends the auth headers on Workers too), so no SDK is needed.
* The deprecated `model` and `params` constructor options and `Settings.language_code` are not ported.
* Tested against local WebSocket servers, as upstream's tests do, not against Sarvam itself.