Frames

Frames are the fundamental units of data in pipedog. Every piece of information that moves through a pipeline (audio, text, images, control signals) is wrapped in a frame. Frame processors receive frames, act on them, and push new or modified frames to the next processor.

All frames extend the base Frame class from @pipedog/core/frames/frames, the port of Pipecat's pipecat.frames.frames. Where Pipecat frames are Python dataclasses, pipedog frames are classes whose constructor takes one options object:

import { TextFrame, TranscriptionFrame } from "@pipedog/core/frames/frames";

const text = new TextFrame({ text: "hello" });
const transcription = new TranscriptionFrame({
  text: "hi there",
  userId: "user-1",
  timestamp: new Date().toISOString(),
});

This is true even where upstream tests pass positional arguments: TextFrame("hi") is always new TextFrame({ text: "hi" }).

Frame categoriesCopy Anchor Link

There are three base frame types, each with different processing behavior:

DataFrameCopy Anchor Link

Data frames carry the main content flowing through a pipeline: audio chunks, text, images and LLM messages. They are queued and processed in order with other data and control frames. If the user interrupts (starts speaking while the bot is responding), pending data frames are discarded so the new input can be handled immediately.

Examples: TextFrame, OutputAudioRawFrame, LLMMessagesAppendFrame, TTSSpeakFrame.

ControlFrameCopy Anchor Link

Control frames signal processing boundaries and configuration changes: response start/end markers, settings updates and state transitions. They are processed in order alongside data frames and, like them, are discarded on interruption unless their interruptible field is false.

Examples: EndFrame, LLMFullResponseStartFrame, TTSStartedFrame.

SystemFrameCopy Anchor Link

System frames are high-priority signals that must always be delivered: interruptions, user input, errors and pipeline lifecycle. They are processed in order with other system frames, ahead of any queued data or control frame, and are never discarded by an interruption.

Examples: StartFrame, CancelFrame, InterruptionFrame, UserStartedSpeakingFrame, InputAudioRawFrame, ErrorFrame, MetricsFrame.

export function describe(f: Frame): string {
  if (f instanceof SystemFrame) return "system: high priority, never dropped by interruptions";
  if (f.interruptible) return "data/control: queued in order, dropped by interruptions";
  return "data/control: queued in order, survives interruptions";
}

Processing orderCopy Anchor Link

Frames are processed in a guaranteed order within their lane. Each processor has a priority input queue: the StartFrame first, then system frames, then data and control frames, each tier in arrival order. System frames are handled as they arrive; data and control frames go to a second queue processed one at a time, so the work for one finishes before the next starts. That is what makes this reliable: speak first, then end the pipeline once the goodbye is flushed.

await worker.queueFrames([new TTSSpeakFrame({ text: "Goodbye!" }), new EndFrame()]);

Frame propertiesCopy Anchor Link

Every frame has these fields, set automatically (see the Frames API):

Interruptible framesCopy Anchor Link

Occasionally a data or control frame is too important to lose when the user interrupts. A frame class that must survive interruptions overrides the interruptible field. In Python that is interruptible: bool = field(default=False, init=False); in TypeScript:

/** Like AsteriskCommandFrame, but survives user interruptions. */
export class CriticalCommandFrame extends DataFrame {
  override interruptible = false;
  command: string;

  constructor(options: AsteriskCommandFrameOptions) {
    super(options);
    this.command = options.command;
  }
}

You can also set it on a single frame before pushing it:

const frame = new TextFrame({ text: "Important" });
frame.interruptible = false; // This specific frame survives interruption.

Uninterruptible frames include EndFrame, StopFrame, EndWorkerFrame, StopWorkerFrame and FunctionCallResultFrame: terminal frames must survive interruption so the pipeline still shuts down properly. The deprecated UninterruptibleFrame mixin is not ported; use the field.

Media mixinsCopy Anchor Link

Pipecat shares audio, image and file fields between frames through dataclass mixins. pipedog ports them as class-factory mixins (defineMixin), so instanceof keeps working:

• AudioRawFrame: audio (Uint8Array, 16-bit little-endian PCM, never Buffer), sampleRate, numChannels, and the computed numFrames.
• ImageRawFrame: image (Uint8Array), size ([width, height]), format.
• FileRawFrame: file, type ("bytes" or "url"), filename, format.

import { AudioRawFrame, InputAudioRawFrame } from "@pipedog/core/frames/frames";

const frame = new InputAudioRawFrame({
  audio: new Uint8Array(320),
  sampleRate: 16000,
  numChannels: 1,
});
console.log(frame instanceof AudioRawFrame, frame.numFrames); // true 160

Creating a custom frameCopy Anchor Link

Subclass one of the three base categories, picking it by how the frame should be handled:

• DataFrame for ordered data such as text, audio or your own payload. Dropped by interruptions.
• ControlFrame for ordered control signals that flush in line with the data.
• SystemFrame for high-priority signals that skip the queue and are not affected by interruptions.

import { DataFrame, type Frame, SystemFrame, TextFrame } from "@pipedog/core/frames/frames";

export interface AsteriskCommandFrameOptions {
  /** The command to send to the telephony transport. */
  command: string;
}

/** A custom command sent downstream to a telephony transport. */
export class AsteriskCommandFrame extends DataFrame {
  command: string;

  constructor(options: AsteriskCommandFrameOptions) {
    super(options);
    this.command = options.command;
  }
}

Name the options interface <Class>Options and pass the whole object to super(options), as every built-in frame does.

Common patternsCopy Anchor Link

pipedog, like Pipecat, prefers pushing frames over calling methods between processors. Routing data through the pipeline as frames keeps processing in order, which real-time use cases depend on.

• End a conversation: push EndWorkerFrame downstream, so frames queued ahead of it (a goodbye TTSSpeakFrame, say) are flushed before the pipeline ends.
• Interrupt the bot yourself: from a processor, await this.broadcastInterruption(); from code holding the worker, await worker.queueFrame(new InterruptionWorkerFrame()).
• React to LLM response boundaries: watch for LLMFullResponseStartFrame and LLMFullResponseEndFrame in a custom processor (see Frame processors).