# RTVI client

> RTVIClientSerializer lets a pipedog pipeline act as an RTVI client, turning a bot's RTVI server messages into frames.

Source: https://polvallverdu.github.io/pipedog/docs/serializers/rtvi-client/



## Overview [#overview]

Bots usually speak RTVI as the server: `RTVIObserver` turns their pipeline frames into RTVI
server messages for a browser client. **`RTVIClientSerializer`** is the inverse. It lets a
pipedog pipeline sit on the client side of that connection:

* Incoming RTVI server messages from the bot (`bot-llm-text`, `user-transcription`,
  `bot-started-speaking` and so on) become pipeline frames.
* Outgoing audio becomes RTVI `raw-audio` messages, and RTVI client messages you build are sent
  as they are.

It is the default serializer of [`RTVIClientTransport`](https://polvallverdu.github.io/pipedog/docs/transports/websocket/#rtviclienttransport),
which pairs it with a WebSocket client and the `client-ready` / `bot-ready` handshake. The two are
the base for driving a bot from a scripted user, as eval harnesses do.

<Cards>
  <Card title="Upstream source" href="https://github.com/pipecat-ai/pipecat/blob/main/src/pipecat/serializers/rtvi_client.py">
    `pipecat/serializers/rtvi_client.py`. Upstream has no docs page for this class.
  </Card>

  <Card title="Source" href="https://github.com/polvallverdu/pipedog/tree/main/packages/serializer-rtvi-client/src">
    `packages/serializer-rtvi-client/src/rtvi_client.ts`.
  </Card>

  <Card title="RTVI" href="https://docs.pipecat.ai/client/rtvi-standard">
    The RTVI message standard.
  </Card>
</Cards>

## Installation [#installation]

```bash
bun add @pipedog/serializer-rtvi-client
```

```ts
import { RTVIClientSerializer } from "@pipedog/serializer-rtvi-client/rtvi_client";
```

`@pipedog/transport-websocket` already depends on it.

## Prerequisites [#prerequisites]

A bot that runs an RTVI server transport and is reachable over a WebSocket.

## RTVIClientSerializer [#rtviclientserializer]

### Configuration [#configuration]

`new RTVIClientSerializer(options?)`:

| Option | Type | Required | Description |
|---|---|---|---|
| `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. |
| `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. |

There are no serializer params: the constructor always sets `ignoreRtviMessages` to `false`,
since RTVI messages are what it sends.

### Usage [#usage]

```ts title="client.ts"
import { OutputTransportMessageFrame } from "@pipedog/core/frames/frames";
import { RTVIClientSerializer } from "@pipedog/serializer-rtvi-client/rtvi_client";
import {
  WebsocketClientParams,
  WebsocketClientTransport,
} from "@pipedog/transport-websocket/client";

// RTVIClientTransport uses this serializer by default. With the plain client
// transport, pass it yourself and turn off WAV headers.
const transport = new WebsocketClientTransport(
  "ws://localhost:7860",
  new WebsocketClientParams({
    audioInEnabled: true,
    audioOutEnabled: true,
    addWavHeader: false,
    serializer: new RTVIClientSerializer(),
  }),
);
```

Send an RTVI client message by pushing a transport message frame toward the output:

```ts
// An RTVI client message, built as a transport message frame. The serializer
// sends any message labelled "rtvi-ai" as it is.
const sendText = new OutputTransportMessageFrame({
  message: {
    label: "rtvi-ai",
    type: "send-text",
    id: "1",
    data: { content: "Hello, bot" },
  },
});
```

### Notes [#notes]

Incoming messages map to frames as follows. Anything else, `bot-ready` and metrics included,
returns `null`.

| RTVI message                    | Frame                                                                           |
| ------------------------------- | ------------------------------------------------------------------------------- |
| `bot-llm-started`               | `LLMFullResponseStartFrame`                                                     |
| `bot-llm-text`                  | `LLMTextFrame`                                                                  |
| `bot-llm-marker`                | `LLMMarkerResponseFrame`                                                        |
| `bot-llm-stopped`               | `LLMFullResponseEndFrame`                                                       |
| `bot-tts-text`                  | `TTSTextFrame` (text type `SENTENCE`; the wire does not carry it)               |
| `bot-started-speaking`          | `BotStartedSpeakingFrame`                                                       |
| `bot-stopped-speaking`          | `BotStoppedSpeakingFrame`                                                       |
| `bot-interrupted`               | `InterruptionFrame`                                                             |
| `llm-function-call-in-progress` | `FunctionCallInProgressFrame`                                                   |
| `llm-function-call-stopped`     | `FunctionCallCancelFrame` if `cancelled`, else `FunctionCallResultFrame`        |
| `user-transcription`            | `TranscriptionFrame` if `final` (the default), else `InterimTranscriptionFrame` |

* Outgoing: an `OutputAudioRawFrame` becomes a `raw-audio` message with base64 PCM, its sample
  rate and channel count, and a counter as `id`. An `OutputTransportMessageFrame` (or urgent) is
  sent only if its message has `label: "rtvi-ai"`. Every other frame returns `null`.
* A message that is not JSON, or not labelled `rtvi-ai`, is dropped. Bytes are decoded as strict
  UTF-8.
* Bot audio and metrics messages are not mapped yet, upstream as here.

## Runtime notes [#runtime-notes]

| Runtime            | Works | What it needs |
| ------------------ | ----- | ------------- |
| Bun                | ✅ yes | Nothing.      |
| Node.js            | ✅ yes | Nothing.      |
| Deno               | ✅ yes | Nothing.      |
| Cloudflare Workers | ✅ yes | Nothing.      |

Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/serializer-rtvi-client/package.json`: ✅ means the compat smoke test passes on that runtime.

Pure TypeScript with no dependencies beyond `@pipedog/core`. Runs on Bun, Node, Deno and
Cloudflare Workers.