HeyGen

OverviewCopy Anchor Link

HeyGen renders a talking avatar from your bot's speech. The session is controlled over HTTP and a WebSocket, and the avatar's video and audio travel through a LiveKit room. There are two ways to use it:

• HeyGenVideoService (@pipedog/heygen): a pipeline service placed after your TTS. It sends the TTS audio to the avatar and pushes the avatar's video (OutputImageRawFrame) and audio (SpeechOutputAudioRawFrame) downstream, so any output transport can deliver them to the user.
• HeyGenTransport (@pipedog/transport-heygen): the bot joins the avatar's LiveKit room as a participant. The user joins the same room and sees the avatar directly; the bot hears the user and sends its speech to the avatar.

Both talk to LiveAvatar (ServiceType.LIVE_AVATAR, api.liveavatar.com) or to HeyGen's older Interactive Avatar streaming API (ServiceType.INTERACTIVE_AVATAR, api.heygen.com, the default for backwards compatibility).

RuntimesCopy Anchor Link

The session APIs use only fetch and the control channel only WebSocket, so they have no runtime requirements of their own, but the packages aren't tested on Workers. The media goes through LiveKit's native Node SDK, @livekit/rtc-node, which loads a native N-API addon. Workers can't load native code, so the service and the transport don't run there. Importing the packages works everywhere: the addon is only loaded when the client sets up.

InstallationCopy Anchor Link

bun add @pipedog/heygen @livekit/rtc-node
# and, for the transport:
bun add @pipedog/transport-heygen

import { HeyGenVideoService } from "@pipedog/heygen/video";
import { HeyGenTransport, HeyGenParams } from "@pipedog/transport-heygen/transport";
import { LiveAvatarNewSessionRequest } from "@pipedog/heygen/api_liveavatar";
import { ServiceType } from "@pipedog/heygen/client";

@livekit/rtc-node is an optional peer dependency: install it yourself (version 1.1.0 or newer in the 1.x line). It ships prebuilt binaries for these platforms only:

There is no build for musl (Alpine Linux): use a Debian- or Ubuntu-based image. Without the package, or on another platform, joining the room fails with an error that says how to install it.

PrerequisitesCopy Anchor Link

• A LiveAvatar account and its API key (for ServiceType.LIVE_AVATAR; see LiveAvatar's docs), or a HeyGen account and API key for the Interactive Avatar API. Pass it as apiKey; the samples read HEYGEN_LIVE_AVATAR_API_KEY.
• An avatar ID (avatarId). Without a session request, the client uses upstream's default public avatar for the chosen API.
• No LiveKit server of your own: HeyGen hosts the room and returns its URL and tokens. To use your own LiveKit server or LiveKit Cloud instead, pass a CustomSDKLiveKitConfig as livekitConfig.
• A long-running Bun, Node or Deno process on one of the platforms above.

HeyGenVideoServiceCopy Anchor Link

ConfigurationCopy Anchor Link

The LiveAvatar session request:

UsageCopy Anchor Link

import { Pipeline } from "@pipedog/core/pipeline/pipeline";
import { PipelineParams, PipelineWorker } from "@pipedog/core/pipeline/worker";
import {
  LiveAvatarNewSessionRequest,
  VideoEncoding,
  VideoSettings,
} from "@pipedog/heygen/api_liveavatar";
import { ServiceType } from "@pipedog/heygen/client";
import { HeyGenVideoService } from "@pipedog/heygen/video";

const heygen = new HeyGenVideoService({
  apiKey: process.env.HEYGEN_LIVE_AVATAR_API_KEY ?? "",
  serviceType: ServiceType.LIVE_AVATAR,
  sessionRequest: new LiveAvatarNewSessionRequest({
    avatarId: "<your LiveAvatar avatar id>",
    videoSettings: new VideoSettings({ encoding: VideoEncoding.H264 }),
  }),
});

// The avatar speaks the TTS audio; its video and audio come back as frames that
// the output transport sends to the user.
const worker = new PipelineWorker(
  new Pipeline([
    transport.input(),
    stt,
    userAggregator,
    llm,
    tts,
    heygen,
    transport.output(),
    assistantAggregator,
  ]),
  { params: new PipelineParams({ audioInSampleRate: 16000, audioOutSampleRate: 24000 }) },
);

NotesCopy Anchor Link

• TTS audio is resampled to 24 kHz and sent as agent.speak messages: a 400 ms first chunk, then 1 s chunks. When no audio arrives for 0.35 s, the rest is flushed and agent.speak_end sent.
• A UserStartedSpeakingFrame interrupts the avatar (agent.interrupt) and starts its listening animation; UserStoppedSpeakingFrame stops it.
• The service joins the room with the user's token (connectAsUser), so HeyGen doesn't end the session after five minutes without a user.

HeyGenTransportCopy Anchor Link

ConfigurationCopy Anchor Link

HeyGenParams is a TransportParams with audioInEnabled and audioOutEnabled on by default.

EventsCopy Anchor Link

UsageCopy Anchor Link

import { HeyGenParams, HeyGenTransport } from "@pipedog/transport-heygen/transport";

const heygenTransport = new HeyGenTransport({
  apiKey: process.env.HEYGEN_LIVE_AVATAR_API_KEY ?? "",
  serviceType: ServiceType.LIVE_AVATAR,
  sessionRequest: new LiveAvatarNewSessionRequest({ avatarId: "<your avatar id>" }),
  params: new HeyGenParams({ audioInEnabled: true, audioOutEnabled: true }),
});

heygenTransport.eventHandler("on_client_connected", async (_transport, participant) => {
  console.log(`user joined the avatar's room: ${participant}`);
});

const avatarWorker = new PipelineWorker(
  new Pipeline([
    heygenTransport.input(),
    stt,
    userAggregator,
    llm,
    tts,
    heygenTransport.output(),
    assistantAggregator,
  ]),
);

The session's room link (https://meet.livekit.io/custom?...) is logged at info level when the session starts; open it to join as the user.

Runtime notesCopy Anchor Link

Generated from pipecat.runtimes and pipecat.compat in packages/heygen/package.json: ✅ means the compat smoke test passes on that runtime.

Generated from pipecat.runtimes and pipecat.compat in packages/transport-heygen/package.json: ✅ means the compat smoke test passes on that runtime.

• The session APIs use fetch; pass fetch to replace the global one (upstream takes an aiohttp session).
• The control WebSocket uses the global WebSocket. clientOptions.websocketOptions takes a webSocketFactory for runtimes that open sockets differently.
• LiveAvatar's API documents livekit_agent_token and ws_url as optional. Upstream requires them; here a session without a WebSocket URL runs without avatar control (logged as a warning), and one without an agent token joins the room with the client token.
• Callbacks given to HeyGenClient.captureParticipantAudio and captureParticipantVideo receive the frame fields (audio, sampleRate, numChannels / image, size, format) rather than an AudioRawFrame or ImageRawFrame instance.