# FunASR

> Local multilingual speech-to-text with FunASRSTTService, running SenseVoiceSmall on ONNX Runtime. Strong on Chinese, with emotion and audio-event tags. No API key.

Source: https://polvallverdu.github.io/pipedog/docs/services/stt/funasr/



## Overview [#overview]

* **`FunASRSTTService`**: a segmented STT service that transcribes each speech segment, cut by the
  pipeline's VAD, with [SenseVoiceSmall](https://huggingface.co/FunAudioLLM/SenseVoiceSmall) on
  your machine. SenseVoice is multilingual (Chinese, Cantonese, English, Japanese, Korean), with
  strong Chinese accuracy and fast non-autoregressive inference. No account, no API key.
* **`AutoModel`** (`@pipedog/funasr/funasr`): the engine. Upstream runs the model with the `funasr`
  Python package (PyTorch). This one runs the same weights on ONNX Runtime through
  `@pipedog/onnx`, and mirrors the part of `funasr` the service uses: `AutoModel`, `generate()` and
  `richTranscriptionPostprocess()`. Its transcripts match sherpa-onnx's for the same model.

<Cards>
  <Card title="Upstream page" href="https://docs.pipecat.ai/api-reference/server/services/stt/funasr">
    Pipecat's documentation for the same service.
  </Card>

  <Card title="Source" href="https://github.com/polvallverdu/pipedog/tree/main/packages/funasr/src">
    `packages/funasr/src/stt.ts` and the engine, `funasr.ts`.
  </Card>

  <Card title="SenseVoice" href="https://github.com/FunAudioLLM/SenseVoice">
    The model, its languages and tags.
  </Card>
</Cards>

## Installation [#installation]

```bash
bun add @pipedog/funasr onnxruntime-node
```

`onnxruntime-node` is an optional peer of `@pipedog/onnx`. `onnxruntime-web@1.30.0` (WebAssembly)
works too, about 10x slower (1.2 s instead of 0.11 s for a 7 s clip on an M4 Pro).

```ts
import { FunASRSTTService } from "@pipedog/funasr/stt";
```

## What you need [#what-you-need]

|          |                                                                                                                                                                                                                                                                                                                                                                                          |
| -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Model    | sherpa-onnx's ONNX export of `iic/SenseVoiceSmall`, from Hugging Face [`csukuangfj/sherpa-onnx-sense-voice-zh-en-ja-ko-yue-2024-07-17`](https://huggingface.co/csukuangfj/sherpa-onnx-sense-voice-zh-en-ja-ko-yue-2024-07-17) (pinned revision): `model.int8.onnx` (239 MB, the default) or `model.onnx` (938 MB, with `modelOptions: { quantized: false }`), plus `tokens.txt` (316 KB) |
| Download | Automatic, once, when the pipeline starts (or on the first transcription). No account or token needed                                                                                                                                                                                                                                                                                    |
| Cache    | `$PIPEDOG_CACHE_DIR/funasr`, else `$XDG_CACHE_HOME/pipedog/funasr`, else `~/.cache/pipedog/funasr`, or `modelOptions: { cacheDir }`                                                                                                                                                                                                                                                      |
| Input    | 16 kHz, 16-bit mono PCM, and a VAD in the pipeline                                                                                                                                                                                                                                                                                                                                       |

To use local files instead, set the `model` setting to a directory holding `model.int8.onnx` (or
`model.onnx`) and `tokens.txt`, or to the `.onnx` file's path with `tokens.txt` next to it, or pass
explicit files:

```ts
// A directory holding model.int8.onnx (or model.onnx) and tokens.txt...
const fromDir = new FunASRSTTService({
  settings: new FunASRSTTService.Settings({ model: "/models/sense-voice" }),
});
// ...or explicit files.
const fromFiles = new FunASRSTTService({
  modelOptions: { modelFile: "/models/model.int8.onnx", tokens: "/models/tokens.txt" },
});
```

## Runtimes [#runtimes]

| Runtime            | Works | What it needs                                                                                                                                                                                                                                                     |
| ------------------ | ----- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Bun                | ✅ yes | onnxruntime-node (or onnxruntime-web) and the SenseVoiceSmall ONNX export (model.int8.onnx 239 MB + tokens.txt), downloaded once from Hugging Face to \~/.cache/pipedog/funasr.                                                                                   |
| Node.js            | ✅ yes | onnxruntime-node (or onnxruntime-web) and the SenseVoiceSmall ONNX export (model.int8.onnx 239 MB + tokens.txt), downloaded once from Hugging Face to \~/.cache/pipedog/funasr.                                                                                   |
| Deno               | ✅ yes | onnxruntime-node (or onnxruntime-web) and the SenseVoiceSmall ONNX export (model.int8.onnx 239 MB + tokens.txt), downloaded once from Hugging Face to \~/.cache/pipedog/funasr.                                                                                   |
| Cloudflare Workers | ❌ no  | onnxruntime-node (or onnxruntime-web) and the SenseVoiceSmall ONNX export (model.int8.onnx 239 MB + tokens.txt), downloaded once from Hugging Face to \~/.cache/pipedog/funasr. Not supported: the model (239 MB int8) exceeds Workers' memory and bundle limits. |

Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/funasr/package.json`: ✅ means the compat smoke test passes on that runtime.

Cloudflare Workers is not supported: the model (239 MB as int8) doesn't fit a Worker's 128 MB of
memory or its 10 MB bundle limit.

## FunASRSTTService [#funasrsttservice]

### Configuration [#configuration]

| Option | Type | Required | Description |
|---|---|---|---|
| `device` | `string \| undefined` | no | Inference device, `"cpu"` or `"cuda"`. |
| `settings` | `FunASRSTTSettings \| null \| undefined` | no | Runtime-updatable settings (`model`, `language`, `useItn`). |
| `modelOptions` | `Partial<Omit<AutoModelOptions, "device" \| "disableUpdate" \| "model">> \| undefined` | no | Extra `AutoModel` options (pipedog only). |
| `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. |
| `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. |
| `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. |
| `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. |
| `audioPassthrough` | `boolean \| undefined` | no | Whether to pass audio frames downstream after processing. Defaults to true. |
| `sampleRate` | `number \| null \| undefined` | no | The sample rate for audio input. If null, will be determined from the start frame. |
| `sttTtfbTimeout` | `number \| undefined` | no | Time in seconds to wait after VAD stop before reporting TTFB. This delay allows the final transcription to arrive. Defaults to 2.0. |
| `ttfsP99Latency` | `number \| null \| undefined` | no | P99 latency from speech end to final transcript in seconds. Broadcast via STTMetadataFrame at pipeline start. |
| `keepaliveTimeout` | `number \| null \| undefined` | no | Seconds of no audio before sending silence to keep the connection alive. null disables keepalive. |
| `keepaliveInterval` | `number \| undefined` | no | Seconds between idle checks when keepalive is enabled. |
| `trailingSilenceSecs` | `number \| undefined` | no | Seconds of silence appended to each segment before it is transcribed, so the model hears the end of speech and can finish the last word. Set to 0 to disable. Defaults to 0.5. |

`modelOptions` (pipedog only) takes `cacheDir`, `quantized`, `modelFile`, `tokens`, `backend`,
`shareSession` and `fetch`.

### Settings [#settings]

Runtime-updatable settings, passed as `settings: new FunASRSTTService.Settings({...})`
(`FunASRSTTSettings`). Any setting you leave out keeps the default below.

| Setting    | Type       | Default                 | Description                                                                                                             |
| ---------- | ---------- | ----------------------- | ----------------------------------------------------------------------------------------------------------------------- |
| `model`    | `string`   | `"iic/SenseVoiceSmall"` | `"iic/SenseVoiceSmall"` or `"FunAudioLLM/SenseVoiceSmall"` to download the model, or a local directory or `.onnx` path. |
| `language` | `Language` | `Language.EN`           | `ZH`, `EN`, `JA`, `KO` or `YUE`. Any other language means auto-detect.                                                  |
| `useItn`   | `boolean`  | `true`                  | Inverse text normalization: punctuation and digits ("nine" becomes "9").                                                |

### Usage [#usage]

```ts title="stt.ts"
import { STTUpdateSettingsFrame } from "@pipedog/core/frames/frames";
import { Language } from "@pipedog/core/transcriptions/language";
import { FunASRSTTService } from "@pipedog/funasr/stt";

// Downloads SenseVoiceSmall (model.int8.onnx, 239 MB, and tokens.txt) on first start.
const stt = new FunASRSTTService({
  settings: new FunASRSTTService.Settings({ language: Language.ZH, useItn: true }),
});
```

Change the language mid-call with an `STTUpdateSettingsFrame`:

```ts
// Queue it on the worker, or push it from a processor upstream of the service.
const update = new STTUpdateSettingsFrame({
  delta: new FunASRSTTService.Settings({ language: Language.JA }),
});

```

The engine works on its own too. `generate()` returns FunASR's raw text with its `<|...|>` language,
emotion and event tags, and `richTranscriptionPostprocess()` strips them:

```ts
import { AutoModel, richTranscriptionPostprocess } from "@pipedog/funasr/funasr";

const model = new AutoModel({ model: "iic/SenseVoiceSmall" });
const samples = new Float32Array(16000); // 16 kHz mono audio in [-1, 1]
const [result] = await model.generate({ input: samples, language: "auto", useItn: true });
if (result) {
  console.log(result.text); // "<|en|><|NEUTRAL|><|Speech|><|withitn|>..."
  console.log(richTranscriptionPostprocess(result.text)); // the text without the tags
}
```

### Notes [#notes]

* Emits one `TranscriptionFrame` per segment, with the detected language. There are no interim
  results.
* The front end computes FunASR's `WavFrontend` features (Kaldi fbank, LFR, CMVN from the model's
  metadata), within 2e-4 of kaldi-native-fbank's, then decodes with CTC greedy search, as FunASR
  does.
* FunASR's PyTorch checkpoints and other ModelScope model ids are not supported.

## Differences from upstream [#differences-from-upstream]

* The model runs on onnxruntime on the CPU, not PyTorch. `device` is accepted, but `"cuda"`
  doesn't select a GPU.
* The model downloads and loads in `start()` (or on the first transcription), not in the
  constructor.
* No dither in the front end (FunASR's `WavFrontend` adds dither 1.0 by default), so results are
  deterministic.
* `modelOptions` is a pipedog option, and the engine's options are camelCase (`disableUpdate`,
  `useItn`).

## Licenses [#licenses]

* `@pipedog/funasr`: BSD 2-Clause. `postprocess_utils.ts` is adapted from
  [FunASR](https://github.com/modelscope/FunASR) (MIT, Copyright (c) 2022 Alibaba).
* SenseVoiceSmall weights: the [FunASR model license](https://github.com/modelscope/FunASR/blob/main/MODEL_LICENSE)
  (attribution required; check its terms for your use). The ONNX export is by k2-fsa/sherpa-onnx.