RTVI client

OverviewCopy Anchor Link

Bots usually speak RTVI as the server: RTVIObserver turns their pipeline frames into RTVI server messages for a browser client. RTVIClientSerializer is the inverse. It lets a pipedog pipeline sit on the client side of that connection:

• Incoming RTVI server messages from the bot (bot-llm-text, user-transcription, bot-started-speaking and so on) become pipeline frames.
• Outgoing audio becomes RTVI raw-audio messages, and RTVI client messages you build are sent as they are.

It is the default serializer of RTVIClientTransport, which pairs it with a WebSocket client and the client-ready / bot-ready handshake. The two are the base for driving a bot from a scripted user, as eval harnesses do.

InstallationCopy Anchor Link

bun add @pipedog/serializer-rtvi-client

import { RTVIClientSerializer } from "@pipedog/serializer-rtvi-client/rtvi_client";

@pipedog/transport-websocket already depends on it.

PrerequisitesCopy Anchor Link

A bot that runs an RTVI server transport and is reachable over a WebSocket.

RTVIClientSerializerCopy Anchor Link

ConfigurationCopy Anchor Link

new RTVIClientSerializer(options?):

There are no serializer params: the constructor always sets ignoreRtviMessages to false, since RTVI messages are what it sends.

UsageCopy Anchor Link

import { OutputTransportMessageFrame } from "@pipedog/core/frames/frames";
import { RTVIClientSerializer } from "@pipedog/serializer-rtvi-client/rtvi_client";
import {
  WebsocketClientParams,
  WebsocketClientTransport,
} from "@pipedog/transport-websocket/client";

// RTVIClientTransport uses this serializer by default. With the plain client
// transport, pass it yourself and turn off WAV headers.
const transport = new WebsocketClientTransport(
  "ws://localhost:7860",
  new WebsocketClientParams({
    audioInEnabled: true,
    audioOutEnabled: true,
    addWavHeader: false,
    serializer: new RTVIClientSerializer(),
  }),
);

Send an RTVI client message by pushing a transport message frame toward the output:

// An RTVI client message, built as a transport message frame. The serializer
// sends any message labelled "rtvi-ai" as it is.
const sendText = new OutputTransportMessageFrame({
  message: {
    label: "rtvi-ai",
    type: "send-text",
    id: "1",
    data: { content: "Hello, bot" },
  },
});

NotesCopy Anchor Link

Incoming messages map to frames as follows. Anything else, bot-ready and metrics included, returns null.

• Outgoing: an OutputAudioRawFrame becomes a raw-audio message with base64 PCM, its sample rate and channel count, and a counter as id. An OutputTransportMessageFrame (or urgent) is sent only if its message has label: "rtvi-ai". Every other frame returns null.
• A message that is not JSON, or not labelled rtvi-ai, is dropped. Bytes are decoded as strict UTF-8.
• Bot audio and metrics messages are not mapped yet, upstream as here.

Runtime notesCopy Anchor Link

Generated from pipecat.runtimes and pipecat.compat in packages/serializer-rtvi-client/package.json: ✅ means the compat smoke test passes on that runtime.

Pure TypeScript with no dependencies beyond @pipedog/core. Runs on Bun, Node, Deno and Cloudflare Workers.