FunASR

OverviewCopy Anchor Link

• FunASRSTTService: a segmented STT service that transcribes each speech segment, cut by the pipeline's VAD, with SenseVoiceSmall on your machine. SenseVoice is multilingual (Chinese, Cantonese, English, Japanese, Korean), with strong Chinese accuracy and fast non-autoregressive inference. No account, no API key.
• AutoModel (@pipedog/funasr/funasr): the engine. Upstream runs the model with the funasr Python package (PyTorch). This one runs the same weights on ONNX Runtime through @pipedog/onnx, and mirrors the part of funasr the service uses: AutoModel, generate() and richTranscriptionPostprocess(). Its transcripts match sherpa-onnx's for the same model.

InstallationCopy Anchor Link

bun add @pipedog/funasr onnxruntime-node

onnxruntime-node is an optional peer of @pipedog/onnx. onnxruntime-web@1.30.0 (WebAssembly) works too, about 10x slower (1.2 s instead of 0.11 s for a 7 s clip on an M4 Pro).

import { FunASRSTTService } from "@pipedog/funasr/stt";

What you needCopy Anchor Link

To use local files instead, set the model setting to a directory holding model.int8.onnx (or model.onnx) and tokens.txt, or to the .onnx file's path with tokens.txt next to it, or pass explicit files:

// A directory holding model.int8.onnx (or model.onnx) and tokens.txt...
const fromDir = new FunASRSTTService({
  settings: new FunASRSTTService.Settings({ model: "/models/sense-voice" }),
});
// ...or explicit files.
const fromFiles = new FunASRSTTService({
  modelOptions: { modelFile: "/models/model.int8.onnx", tokens: "/models/tokens.txt" },
});

RuntimesCopy Anchor Link

Generated from pipecat.runtimes and pipecat.compat in packages/funasr/package.json: ✅ means the compat smoke test passes on that runtime.

Cloudflare Workers is not supported: the model (239 MB as int8) doesn't fit a Worker's 128 MB of memory or its 10 MB bundle limit.

FunASRSTTServiceCopy Anchor Link

ConfigurationCopy Anchor Link

modelOptions (pipedog only) takes cacheDir, quantized, modelFile, tokens, backend, shareSession and fetch.

SettingsCopy Anchor Link

Runtime-updatable settings, passed as settings: new FunASRSTTService.Settings({...}) (FunASRSTTSettings). Any setting you leave out keeps the default below.

UsageCopy Anchor Link

import { STTUpdateSettingsFrame } from "@pipedog/core/frames/frames";
import { Language } from "@pipedog/core/transcriptions/language";
import { FunASRSTTService } from "@pipedog/funasr/stt";

// Downloads SenseVoiceSmall (model.int8.onnx, 239 MB, and tokens.txt) on first start.
const stt = new FunASRSTTService({
  settings: new FunASRSTTService.Settings({ language: Language.ZH, useItn: true }),
});

Change the language mid-call with an STTUpdateSettingsFrame:

// Queue it on the worker, or push it from a processor upstream of the service.
const update = new STTUpdateSettingsFrame({
  delta: new FunASRSTTService.Settings({ language: Language.JA }),
});

The engine works on its own too. generate() returns FunASR's raw text with its <|...|> language, emotion and event tags, and richTranscriptionPostprocess() strips them:

import { AutoModel, richTranscriptionPostprocess } from "@pipedog/funasr/funasr";

const model = new AutoModel({ model: "iic/SenseVoiceSmall" });
const samples = new Float32Array(16000); // 16 kHz mono audio in [-1, 1]
const [result] = await model.generate({ input: samples, language: "auto", useItn: true });
if (result) {
  console.log(result.text); // "<|en|><|NEUTRAL|><|Speech|><|withitn|>..."
  console.log(richTranscriptionPostprocess(result.text)); // the text without the tags
}

NotesCopy Anchor Link

• Emits one TranscriptionFrame per segment, with the detected language. There are no interim results.
• The front end computes FunASR's WavFrontend features (Kaldi fbank, LFR, CMVN from the model's metadata), within 2e-4 of kaldi-native-fbank's, then decodes with CTC greedy search, as FunASR does.
• FunASR's PyTorch checkpoints and other ModelScope model ids are not supported.

Differences from upstreamCopy Anchor Link

• The model runs on onnxruntime on the CPU, not PyTorch. device is accepted, but "cuda" doesn't select a GPU.
• The model downloads and loads in start() (or on the first transcription), not in the constructor.
• No dither in the front end (FunASR's WavFrontend adds dither 1.0 by default), so results are deterministic.
• modelOptions is a pipedog option, and the engine's options are camelCase (disableUpdate, useItn).

LicensesCopy Anchor Link

• @pipedog/funasr: BSD 2-Clause. postprocess_utils.ts is adapted from FunASR (MIT, Copyright (c) 2022 Alibaba).
• SenseVoiceSmall weights: the FunASR model license (attribution required; check its terms for your use). The ONNX export is by k2-fsa/sherpa-onnx.