# Evals

> Test a bot end to end with scripted or LLM-simulated users, judged by an LLM, from YAML scenarios and the pipedog-eval CLI.

Source: https://polvallverdu.github.io/pipedog/docs/evals/



`@pipedog/evals` is the port of Pipecat's `pipecat.evals`. It tests a running bot end to end.
The harness connects to the bot as an RTVI client and plays the user's side of a conversation,
in text or in synthesized speech. It listens to what the bot says and checks it against the
scenario, with exact checks and an LLM judge.

A **scenario** is one check, written in YAML. There are two kinds:

* A **scripted** scenario writes out the user's turns and what the bot should do after each one.
  You can check text, events, function calls and timing, and send criteria to the judge.
* A **simulated** scenario gives an LLM a persona and a goal, and lets it talk to the bot. A judge
  then reads the whole conversation and decides whether the bot did its job.

<Cards>
  <Card title="Upstream" href="https://github.com/pipecat-ai/pipecat/tree/main/evals">
    Pipecat's eval suites and their scenarios, which use the same YAML format.
  </Card>

  <Card title="Source" href="https://github.com/polvallverdu/pipedog/tree/main/packages/evals/src">
    `packages/evals/src`: `session.ts`, `scenario.ts`, `script.ts`, `judge.ts`, `suite.ts`, `cli.ts`.
  </Card>
</Cards>

## Installation [#installation]

```bash
bun add -d @pipedog/evals
```

The harness runs on Bun and Node. The `pipedog-eval` command runs scenarios, judges saved
conversations and runs suites:

```bash
bunx pipedog-eval --help    # or: npx pipedog-eval --help
```

The same commands are `pipedog eval` in [`@pipedog/cli`](https://polvallverdu.github.io/pipedog/docs/cli.md), which loads this package when
you run it. A project made with `pipedog init --eval` has the `eval` transport, a starter scenario
and both packages.

## Make the bot reachable [#make-the-bot-reachable]

The harness connects to the [development runner](https://polvallverdu.github.io/pipedog/docs/runner.md)'s `/ws` route. Its first message,
`eval-connect`, tells `createTransport()` to build the eval transport, so the bot needs nothing
more than an `eval` entry in its transport parameters:

```ts title="bot.ts"
import { EvalTransportParams } from "@pipedog/evals/transport";
import type { RunnerArguments } from "@pipedog/runner/types";
import { createTransport } from "@pipedog/runner/utils";

const transportParams = {
  // Used when an eval harness connects to the runner's /ws and sends `eval-connect`.
  eval: () => new EvalTransportParams({ audioInEnabled: true, audioOutEnabled: true }),
};

export async function bot(runnerArgs: RunnerArguments) {
  const transport = await createTransport(runnerArgs, transportParams);
  // build and run the pipeline with transport.input() and transport.output()
  void transport;
}

```

Start it as usual, e.g. `bun bot.ts`, and it listens on `ws://localhost:7860/ws`.

## Write a scenario [#write-a-scenario]

```yaml
# scenarios/capital.yaml
name: capital
judge:
  eval:
    factory: ./my_evals.ts.llm   # the judge's LLM, built by your code (below)
    model: gpt-4.1-mini
scenarios:
  - name: capital_question
    turns:
      # Wait for the bot's greeting before speaking.
      - expect:
          - event: response
            eval: "the bot opens the conversation (a greeting, an offer to help)"
      - user: "What is the capital of Germany?"
        expect:
          - event: response
            text_contains: Berlin
            within_ms: 10000
            eval: "the response says the capital of Germany is Berlin"
```

Each turn sends `user:` (or nothing, to wait for the bot) and checks the bot's events in order.
`response` is the bot's reply: the LLM's text in text modality, the default, or a transcription of
its speech in audio modality. Other events include `user_transcription`, `bot_started_speaking`,
`function_call` (with `calls:`), `llm_marker`, `image` and `expression`. `text_contains` and
`text_excludes` check the text exactly, `text_matches` takes one regular expression or a list that
the text must each match (Python `re.search` semantics), `within_ms` sets a latency budget, and
`eval:` asks the judge. A missed `text_matches` pattern fails the expectation with the
`text_unmatched` failure kind, naming the first pattern that didn't match. The module docs of
`script.ts` and `simulation.ts` describe every key, as upstream's do.

```yaml
      - user: "Read me my order number."
        expect:
          - event: response
            text_matches:
              - "^(Sure|Okay)"   # how the reply must open
              - "\\d{6}"         # it has a six-digit number
```

`expression` is the expression the bot's TTS performed in expressive conversation (alpha), written
as the LLM writes its tags: `<<excited>>` where the delivery of the spoken text changes
(`<<neutral>>` back to the baseline), `[[laughter]]` for a sound, `[[pause 600]]` for a pause. A
tag the TTS can't perform is never reported, so a scenario can check that what the LLM wrote was
performed. It takes `text_contains`, `text_matches` and `eval:` like a response.

A simulation swaps `turns:` for a persona:

```yaml
scenarios:
  - name: capital_curious
    persona: |
      A curious, polite traveler who asks one thing at a time.
    goal: "Find out what the capital of Germany is, then say goodbye."
    success: "the bot told the caller that the capital of Germany is Berlin"
    simulator:
      factory: ./my_evals.ts.llm   # must support function calling (the persona hangs up with a tool)
    metrics:
      - name: politeness
        criterion: "the bot stayed courteous throughout"
        min_score: 1
    max_turns: 10
```

## Services: the judge, the user's voice and the transcriber [#services-the-judge-the-users-voice-and-the-transcriber]

The harness builds its own services from the scenario. A `factory:` names a module and one of
its exports, joined by a dot. The module is a path relative to the working directory, a package
or a `file:` URL. The export is a function that takes the config block and returns the service:

```ts title="my_evals.ts"
// my_evals.ts: the services a scenario names with `factory: ./my_evals.ts.<export>`
import { OpenAILLMSettings } from "@pipedog/openai/base_llm";
import { OpenAILLMService } from "@pipedog/openai/llm";
import { OpenAISTTService } from "@pipedog/openai/stt";
import { OpenAITTSService } from "@pipedog/openai/tts";

type Config = Record<string, unknown>;

/** `judge.eval` and `simulator`: an OpenAI-compatible LLM. */
export function llm(config: Config) {
  return new OpenAILLMService({
    apiKey: process.env.OPENAI_API_KEY ?? null,
    settings: new OpenAILLMSettings({ model: String(config.model ?? "gpt-4.1-mini") }),
  });
}

/** `user.speech`: the voice the harness speaks the user's turns with. */
export function voice(_config: Config) {
  return new OpenAITTSService({ apiKey: process.env.OPENAI_API_KEY ?? null });
}

/** `judge.transcription`: transcribes the bot's speech for `response` checks. */
export function transcriber(_config: Config) {
  return new OpenAISTTService({ apiKey: process.env.OPENAI_API_KEY ?? null });
}

```

* `judge.eval` (and a simulation's `simulator`): an OpenAI-compatible LLM, or any
  `BaseClassifier`. Without a factory it is Ollama (`service: ollama`) running `gemma4:12b`
  locally. Set `explainer: false` to turn off the explainer that gives reasons for a `no`.
* `user.speech`: the voice the user's turns are spoken in, when `user.modality: audio`. Synthesized
  turns are cached on disk.
* `judge.transcription`: the transcriber for the bot's speech, when `judge.modality: audio`.

Audio scenarios use upstream's keyless local defaults: [Kokoro](https://polvallverdu.github.io/pipedog/docs/services/tts/kokoro.md)
(`service: kokoro`) for the user's voice, and [Moonshine](https://polvallverdu.github.io/pipedog/docs/services/stt/moonshine.md) (the default)
or [local Whisper](https://polvallverdu.github.io/pipedog/docs/services/stt/whisper.md) (`service: whisper`) for the transcriber. They are
optional peer dependencies of `@pipedog/evals`: install `@pipedog/kokoro`, `@pipedog/moonshine` or
`@pipedog/whisper` with `onnxruntime-node` (Kokoro also needs `@echogarden/espeak-ng-emscripten`).
Models download on first use into `~/.cache/pipedog`.

<Callout>
  WAV files work as `audio:` recordings; other formats need a decoder.
</Callout>

## Run it [#run-it]

```bash
bunx pipedog-eval run scenarios/capital.yaml --bot-url ws://localhost:7860/ws
bunx pipedog-eval run scenarios/ -v          # every scenario in a folder, verbose
```

The command prints each scenario's configuration, then live status and timings. Each scenario's
`<scenario>.eval.log` goes in `--logs-dir`. The exit code is 1 if any scenario failed.

`pipedog-eval suite manifest.yaml` runs a manifest of bots and their scenarios: it spawns each bot
on its own port (with the current runtime, `{python} {bot} --port {port}` by default), runs the
scenarios concurrently, and can repeat them to measure flakiness. `pipedog-eval judge` judges
saved conversations against a simulation's criteria.

## From code [#from-code]

The same run, without the CLI:

```ts title="run.ts"
import {
  EvalScenarioFile,
  EvalScriptResult,
  EvalSession,
  EvalSessionParams,
  type EvalSimulationResult,
} from "@pipedog/evals";

const file = await EvalScenarioFile.load("scenarios/capital.yaml");
for (const scenario of file) {
  const session: EvalSession<EvalScriptResult | EvalSimulationResult> =
    await EvalSession.fromScenario(scenario, "ws://localhost:7860/ws", {
      params: new EvalSessionParams({ defaultTimeoutMs: 30_000 }),
    });
  session.eventHandler("on_progress", async (_session: unknown, progress: unknown) => {
    console.log(progress);
  });
  const result = await session.run();
  console.log(scenario.name, result.passed ? "PASS" : "FAIL");
  if (result instanceof EvalScriptResult) {
    for (const failure of result.failures) console.log(`  ${failure}`);
  }
}
```

`EvalSession.fromScenario()` also takes `judge`, `userTts`, `botStt` and `personaLlm`, which
override what the scenario configures. That's how a test can pass in fakes.

## Differences from upstream [#differences-from-upstream]

* `python -m pipecat.evals` and `pipecat eval` are the `pipedog-eval` bin. A plain-ANSI table
  replaces rich's live dashboard. `.env` is loaded by a small built-in loader.
* A `factory:` is `<module>.<export>`, imported with `import()`, rather than a dotted Python path.
  The service builders and `EvalScenarioFile.load()` are async.
* Each suite run is a child process running `_session_subprocess` with the current runtime.
* The deprecated pieces are left out: the `EvalTransport` server, `pipecat.evals.harness`, flat
  scenario files, `trigger_disconnect`, `stop_bot`, and `service: openai` / `service: cartesia`
  (use a factory).