Evals

@pipedog/evals is the port of Pipecat's pipecat.evals. It tests a running bot end to end. The harness connects to the bot as an RTVI client and plays the user's side of a conversation, in text or in synthesized speech. It listens to what the bot says and checks it against the scenario, with exact checks and an LLM judge.

A scenario is one check, written in YAML. There are two kinds:

• A scripted scenario writes out the user's turns and what the bot should do after each one. You can check text, events, function calls and timing, and send criteria to the judge.
• A simulated scenario gives an LLM a persona and a goal, and lets it talk to the bot. A judge then reads the whole conversation and decides whether the bot did its job.

InstallationCopy Anchor Link

bun add -d @pipedog/evals

The harness runs on Bun and Node. The pipedog-eval command runs scenarios, judges saved conversations and runs suites:

bunx pipedog-eval --help    # or: npx pipedog-eval --help

The same commands are pipedog eval in @pipedog/cli, which loads this package when you run it. A project made with pipedog init --eval has the eval transport, a starter scenario and both packages.

Make the bot reachableCopy Anchor Link

The harness connects to the development runner's /ws route. Its first message, eval-connect, tells createTransport() to build the eval transport, so the bot needs nothing more than an eval entry in its transport parameters:

import { EvalTransportParams } from "@pipedog/evals/transport";
import type { RunnerArguments } from "@pipedog/runner/types";
import { createTransport } from "@pipedog/runner/utils";

const transportParams = {
  // Used when an eval harness connects to the runner's /ws and sends `eval-connect`.
  eval: () => new EvalTransportParams({ audioInEnabled: true, audioOutEnabled: true }),
};

export async function bot(runnerArgs: RunnerArguments) {
  const transport = await createTransport(runnerArgs, transportParams);
  // build and run the pipeline with transport.input() and transport.output()
  void transport;
}

Start it as usual, e.g. bun bot.ts, and it listens on ws://localhost:7860/ws.

Write a scenarioCopy Anchor Link

# scenarios/capital.yaml
name: capital
judge:
  eval:
    factory: ./my_evals.ts.llm   # the judge's LLM, built by your code (below)
    model: gpt-4.1-mini
scenarios:
  - name: capital_question
    turns:
      # Wait for the bot's greeting before speaking.
      - expect:
          - event: response
            eval: "the bot opens the conversation (a greeting, an offer to help)"
      - user: "What is the capital of Germany?"
        expect:
          - event: response
            text_contains: Berlin
            within_ms: 10000
            eval: "the response says the capital of Germany is Berlin"

Each turn sends user: (or nothing, to wait for the bot) and checks the bot's events in order. response is the bot's reply: the LLM's text in text modality, the default, or a transcription of its speech in audio modality. Other events include user_transcription, bot_started_speaking, function_call (with calls:), llm_marker, image and expression. text_contains and text_excludes check the text exactly, text_matches takes one regular expression or a list that the text must each match (Python re.search semantics), within_ms sets a latency budget, and eval: asks the judge. A missed text_matches pattern fails the expectation with the text_unmatched failure kind, naming the first pattern that didn't match. The module docs of script.ts and simulation.ts describe every key, as upstream's do.

      - user: "Read me my order number."
        expect:
          - event: response
            text_matches:
              - "^(Sure|Okay)"   # how the reply must open
              - "\\d{6}"         # it has a six-digit number

expression is the expression the bot's TTS performed in expressive conversation (alpha), written as the LLM writes its tags: <<excited>> where the delivery of the spoken text changes (<<neutral>> back to the baseline), [[laughter]] for a sound, [[pause 600]] for a pause. A tag the TTS can't perform is never reported, so a scenario can check that what the LLM wrote was performed. It takes text_contains, text_matches and eval: like a response.

A simulation swaps turns: for a persona:

scenarios:
  - name: capital_curious
    persona: |
      A curious, polite traveler who asks one thing at a time.
    goal: "Find out what the capital of Germany is, then say goodbye."
    success: "the bot told the caller that the capital of Germany is Berlin"
    simulator:
      factory: ./my_evals.ts.llm   # must support function calling (the persona hangs up with a tool)
    metrics:
      - name: politeness
        criterion: "the bot stayed courteous throughout"
        min_score: 1
    max_turns: 10

Services: the judge, the user's voice and the transcriberCopy Anchor Link

The harness builds its own services from the scenario. A factory: names a module and one of its exports, joined by a dot. The module is a path relative to the working directory, a package or a file: URL. The export is a function that takes the config block and returns the service:

// my_evals.ts: the services a scenario names with `factory: ./my_evals.ts.<export>`
import { OpenAILLMSettings } from "@pipedog/openai/base_llm";
import { OpenAILLMService } from "@pipedog/openai/llm";
import { OpenAISTTService } from "@pipedog/openai/stt";
import { OpenAITTSService } from "@pipedog/openai/tts";

type Config = Record<string, unknown>;

/** `judge.eval` and `simulator`: an OpenAI-compatible LLM. */
export function llm(config: Config) {
  return new OpenAILLMService({
    apiKey: process.env.OPENAI_API_KEY ?? null,
    settings: new OpenAILLMSettings({ model: String(config.model ?? "gpt-4.1-mini") }),
  });
}

/** `user.speech`: the voice the harness speaks the user's turns with. */
export function voice(_config: Config) {
  return new OpenAITTSService({ apiKey: process.env.OPENAI_API_KEY ?? null });
}

/** `judge.transcription`: transcribes the bot's speech for `response` checks. */
export function transcriber(_config: Config) {
  return new OpenAISTTService({ apiKey: process.env.OPENAI_API_KEY ?? null });
}

• judge.eval (and a simulation's simulator): an OpenAI-compatible LLM, or any BaseClassifier. Without a factory it is Ollama (service: ollama) running gemma4:12b locally. Set explainer: false to turn off the explainer that gives reasons for a no.
• user.speech: the voice the user's turns are spoken in, when user.modality: audio. Synthesized turns are cached on disk.
• judge.transcription: the transcriber for the bot's speech, when judge.modality: audio.

Audio scenarios use upstream's keyless local defaults: Kokoro (service: kokoro) for the user's voice, and Moonshine (the default) or local Whisper (service: whisper) for the transcriber. They are optional peer dependencies of @pipedog/evals: install @pipedog/kokoro, @pipedog/moonshine or @pipedog/whisper with onnxruntime-node (Kokoro also needs @echogarden/espeak-ng-emscripten). Models download on first use into ~/.cache/pipedog.

Run itCopy Anchor Link

bunx pipedog-eval run scenarios/capital.yaml --bot-url ws://localhost:7860/ws
bunx pipedog-eval run scenarios/ -v          # every scenario in a folder, verbose

The command prints each scenario's configuration, then live status and timings. Each scenario's <scenario>.eval.log goes in --logs-dir. The exit code is 1 if any scenario failed.

pipedog-eval suite manifest.yaml runs a manifest of bots and their scenarios: it spawns each bot on its own port (with the current runtime, {python} {bot} --port {port} by default), runs the scenarios concurrently, and can repeat them to measure flakiness. pipedog-eval judge judges saved conversations against a simulation's criteria.

From codeCopy Anchor Link

The same run, without the CLI:

import {
  EvalScenarioFile,
  EvalScriptResult,
  EvalSession,
  EvalSessionParams,
  type EvalSimulationResult,
} from "@pipedog/evals";

const file = await EvalScenarioFile.load("scenarios/capital.yaml");
for (const scenario of file) {
  const session: EvalSession<EvalScriptResult | EvalSimulationResult> =
    await EvalSession.fromScenario(scenario, "ws://localhost:7860/ws", {
      params: new EvalSessionParams({ defaultTimeoutMs: 30_000 }),
    });
  session.eventHandler("on_progress", async (_session: unknown, progress: unknown) => {
    console.log(progress);
  });
  const result = await session.run();
  console.log(scenario.name, result.passed ? "PASS" : "FAIL");
  if (result instanceof EvalScriptResult) {
    for (const failure of result.failures) console.log(`  ${failure}`);
  }
}

EvalSession.fromScenario() also takes judge, userTts, botStt and personaLlm, which override what the scenario configures. That's how a test can pass in fakes.

Differences from upstreamCopy Anchor Link

• python -m pipecat.evals and pipecat eval are the pipedog-eval bin. A plain-ANSI table replaces rich's live dashboard. .env is loaded by a small built-in loader.
• A factory: is <module>.<export>, imported with import(), rather than a dotted Python path. The service builders and EvalScenarioFile.load() are async.
• Each suite run is a child process running _session_subprocess with the current runtime.
• The deprecated pieces are left out: the EvalTransport server, pipecat.evals.harness, flat scenario files, trigger_disconnect, stop_bot, and service: openai / service: cartesia (use a factory).