# HeyGen

> HeyGen video avatars (LiveAvatar and Interactive Avatar) with HeyGenVideoService and HeyGenTransport.

Source: https://polvallverdu.github.io/pipedog/docs/services/video/heygen/



## Overview [#overview]

HeyGen renders a talking avatar from your bot's speech. The session is controlled over HTTP and a
WebSocket, and the avatar's video and audio travel through a LiveKit room. There are two ways to
use it:

* **`HeyGenVideoService`** (`@pipedog/heygen`): a pipeline service placed after your TTS. It sends
  the TTS audio to the avatar and pushes the avatar's video (`OutputImageRawFrame`) and audio
  (`SpeechOutputAudioRawFrame`) downstream, so any output transport can deliver them to the user.
* **`HeyGenTransport`** (`@pipedog/transport-heygen`): the bot joins the avatar's LiveKit room as
  a participant. The user joins the same room and sees the avatar directly; the bot hears the user
  and sends its speech to the avatar.

Both talk to **LiveAvatar** (`ServiceType.LIVE_AVATAR`, `api.liveavatar.com`) or to HeyGen's older
**Interactive Avatar** streaming API (`ServiceType.INTERACTIVE_AVATAR`, `api.heygen.com`, the
default for backwards compatibility).

<Cards>
  <Card title="Upstream page" href="https://docs.pipecat.ai/api-reference/server/services/video/heygen">
    Pipecat's documentation for the same service.
  </Card>

  <Card title="Source" href="https://github.com/polvallverdu/pipedog/tree/main/packages/heygen/src">
    `video.ts`, `client.ts`, `api_liveavatar.ts` and `api_interactive_avatar.ts`, plus
    `packages/transport-heygen/src/transport.ts`.
  </Card>

  <Card title="LiveAvatar docs" href="https://docs.liveavatar.com">
    Sessions, avatars and API keys.
  </Card>
</Cards>

## Runtimes [#runtimes]

|                                                        | Bun | Node ≥ 22 | Deno              | Cloudflare Workers |
| ------------------------------------------------------ | --- | --------- | ----------------- | ------------------ |
| `LiveAvatarApi`, `HeyGenApi` (session HTTP APIs)       | ✅   | ✅         | ✅                 | not verified       |
| `HeyGenClient` control WebSocket                       | ✅   | ✅         | ✅                 | not verified       |
| `HeyGenVideoService`, `HeyGenTransport` (avatar media) | ✅   | ✅         | ✅ (`--allow-all`) | ❌                  |

The session APIs use only `fetch` and the control channel only `WebSocket`, so they have no
runtime requirements of their own, but the packages aren't tested on Workers. The media goes
through LiveKit's native Node SDK, `@livekit/rtc-node`, which loads a native N-API addon. Workers
can't load native code, so the service and the transport don't run there. Importing the packages
works everywhere: the addon is only loaded when the client sets up.

## Installation [#installation]

```bash
bun add @pipedog/heygen @livekit/rtc-node
# and, for the transport:
bun add @pipedog/transport-heygen
```

```ts
import { HeyGenVideoService } from "@pipedog/heygen/video";
import { HeyGenTransport, HeyGenParams } from "@pipedog/transport-heygen/transport";
import { LiveAvatarNewSessionRequest } from "@pipedog/heygen/api_liveavatar";
import { ServiceType } from "@pipedog/heygen/client";
```

`@livekit/rtc-node` is an optional peer dependency: install it yourself (version 1.1.0 or newer in
the 1.x line). It ships prebuilt binaries for these platforms only:

| OS               | Architectures                                    |
| ---------------- | ------------------------------------------------ |
| macOS            | arm64 (`darwin-arm64`), x64 (`darwin-x64`)       |
| Linux with glibc | x64 (`linux-x64-gnu`), arm64 (`linux-arm64-gnu`) |
| Windows          | x64 (`win32-x64-msvc`)                           |

There is no build for musl (Alpine Linux): use a Debian- or Ubuntu-based image. Without the
package, or on another platform, joining the room fails with an error that says how to install
it.

## Prerequisites [#prerequisites]

* A **LiveAvatar account** and its API key (for `ServiceType.LIVE_AVATAR`; see
  [LiveAvatar's docs](https://docs.liveavatar.com)), or a **HeyGen account** and API key for the
  Interactive Avatar API. Pass it as `apiKey`; the samples read `HEYGEN_LIVE_AVATAR_API_KEY`.
* An avatar ID (`avatarId`). Without a session request, the client uses upstream's default public
  avatar for the chosen API.
* No LiveKit server of your own: HeyGen hosts the room and returns its URL and tokens. To use your
  own LiveKit server or LiveKit Cloud instead, pass a `CustomSDKLiveKitConfig` as `livekitConfig`.
* A long-running Bun, Node or Deno process on one of the platforms above.

## HeyGenVideoService [#heygenvideoservice]

### Configuration [#configuration]

| Option | Type | Required | Description |
|---|---|---|---|
| `apiKey` | `string` | yes | HeyGen API key for authentication. |
| `fetch` | `{ (input: RequestInfo \| URL, init?: RequestInit): Promise<Response>; (input: string \| URL \| Request, init?: RequestInit): Promise<Response>; } \| null \| undefined` | no | `fetch` used for API requests. Defaults to the global `fetch`. DIVERGENCE: replaces upstream's required aiohttp `session`. |
| `sessionRequest` | `LiveAvatarNewSessionRequest \| NewSessionRequest \| null \| undefined` | no | Configuration for the HeyGen session. |
| `serviceType` | `ServiceType \| null \| undefined` | no | Service type for the avatar session. |
| `settings` | `HeyGenVideoSettings \| null \| undefined` | no | Runtime-updatable settings. HeyGen has no model concept, so this is primarily used for the `extra` dict. |
| `clientOptions` | `Partial<HeyGenClientOptions> \| null \| undefined` | no | Extra `HeyGenClient` options (e.g. `websocketOptions`, `api`). pipedog addition. |
| `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. |
| `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. |
| `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. |
| `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. |

The LiveAvatar session request:

| Option | Type | Required | Description |
|---|---|---|---|
| `mode` | `string \| undefined` | no | Session mode (default: "LITE"). |
| `avatarId` | `string` | yes | Unique identifier for the avatar. |
| `videoSettings` | `VideoSettings \| null \| undefined` | no | Video encoding settings (default: H264, high quality). |
| `isSandbox` | `boolean \| null \| undefined` | no | Enable sandbox mode (default: false). |
| `avatarPersona` | `AvatarPersona \| null \| undefined` | no | Avatar persona configuration. |
| `livekitConfig` | `CustomSDKLiveKitConfig \| null \| undefined` | no | Custom LiveKit configuration. |
| `maxSessionDuration` | `number \| null \| undefined` | no | Maximum session duration in seconds. The ceiling is set by the LiveAvatar subscription tier, which also supplies the default when this is unset. |

### Usage [#usage]

```ts title="bot.ts"
import { Pipeline } from "@pipedog/core/pipeline/pipeline";
import { PipelineParams, PipelineWorker } from "@pipedog/core/pipeline/worker";
import {
  LiveAvatarNewSessionRequest,
  VideoEncoding,
  VideoSettings,
} from "@pipedog/heygen/api_liveavatar";
import { ServiceType } from "@pipedog/heygen/client";
import { HeyGenVideoService } from "@pipedog/heygen/video";

const heygen = new HeyGenVideoService({
  apiKey: process.env.HEYGEN_LIVE_AVATAR_API_KEY ?? "",
  serviceType: ServiceType.LIVE_AVATAR,
  sessionRequest: new LiveAvatarNewSessionRequest({
    avatarId: "<your LiveAvatar avatar id>",
    videoSettings: new VideoSettings({ encoding: VideoEncoding.H264 }),
  }),
});

// The avatar speaks the TTS audio; its video and audio come back as frames that
// the output transport sends to the user.
const worker = new PipelineWorker(
  new Pipeline([
    transport.input(),
    stt,
    userAggregator,
    llm,
    tts,
    heygen,
    transport.output(),
    assistantAggregator,
  ]),
  { params: new PipelineParams({ audioInSampleRate: 16000, audioOutSampleRate: 24000 }) },
);

```

### Notes [#notes]

* TTS audio is resampled to 24 kHz and sent as `agent.speak` messages: a 400 ms first chunk, then
  1 s chunks. When no audio arrives for 0.35 s, the rest is flushed and `agent.speak_end` sent.
* A `UserStartedSpeakingFrame` interrupts the avatar (`agent.interrupt`) and starts its listening
  animation; `UserStoppedSpeakingFrame` stops it.
* The service joins the room with the user's token (`connectAsUser`), so HeyGen doesn't end the
  session after five minutes without a user.

## HeyGenTransport [#heygentransport]

### Configuration [#configuration-1]

| Option | Type | Required | Description |
|---|---|---|---|
| `fetch` | `{ (input: RequestInfo \| URL, init?: RequestInit): Promise<Response>; (input: string \| URL \| Request, init?: RequestInit): Promise<Response>; } \| null \| undefined` | no | `fetch` used for API requests. Defaults to the global `fetch`. DIVERGENCE: replaces upstream's required aiohttp `session`. |
| `apiKey` | `string` | yes | HeyGen API key for authentication. |
| `params` | `HeyGenParams \| null \| undefined` | no | HeyGen-specific configuration parameters (default: new HeyGenParams()). |
| `inputName` | `string \| null \| undefined` | no | Optional custom name for the input transport. |
| `outputName` | `string \| null \| undefined` | no | Optional custom name for the output transport. |
| `sessionRequest` | `LiveAvatarNewSessionRequest \| NewSessionRequest \| null \| undefined` | no | Configuration for the HeyGen session. |
| `serviceType` | `ServiceType \| null \| undefined` | no | Service type for the avatar session. |
| `clientOptions` | `Partial<HeyGenClientOptions> \| null \| undefined` | no | Extra `HeyGenClient` options (e.g. `websocketOptions`, `api`). pipedog addition. |

`HeyGenParams` is a `TransportParams` with `audioInEnabled` and `audioOutEnabled` on by default.

### Events [#events]

| Event                    | Arguments                  | When                                                                 |
| ------------------------ | -------------------------- | -------------------------------------------------------------------- |
| `on_connected`           | `(transport)`              | The bot joined the avatar's room.                                    |
| `on_client_connected`    | `(transport, participant)` | A participant other than the avatar joined. Their audio is captured. |
| `on_client_disconnected` | `(transport, participant)` | That participant left.                                               |

### Usage [#usage-1]

```ts title="bot.ts"
import { HeyGenParams, HeyGenTransport } from "@pipedog/transport-heygen/transport";

const heygenTransport = new HeyGenTransport({
  apiKey: process.env.HEYGEN_LIVE_AVATAR_API_KEY ?? "",
  serviceType: ServiceType.LIVE_AVATAR,
  sessionRequest: new LiveAvatarNewSessionRequest({ avatarId: "<your avatar id>" }),
  params: new HeyGenParams({ audioInEnabled: true, audioOutEnabled: true }),
});

heygenTransport.eventHandler("on_client_connected", async (_transport, participant) => {
  console.log(`user joined the avatar's room: ${participant}`);
});

const avatarWorker = new PipelineWorker(
  new Pipeline([
    heygenTransport.input(),
    stt,
    userAggregator,
    llm,
    tts,
    heygenTransport.output(),
    assistantAggregator,
  ]),
);

```

The session's room link (`https://meet.livekit.io/custom?...`) is logged at `info` level when the
session starts; open it to join as the user.

## Runtime notes [#runtime-notes]

| Runtime            | Works | What it needs |
| ------------------ | ----- | ------------- |
| Bun                | ✅ yes | Nothing.      |
| Node.js            | ✅ yes | Nothing.      |
| Deno               | ✅ yes | Nothing.      |
| Cloudflare Workers | ❌ no  | Nothing.      |

Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/heygen/package.json`: ✅ means the compat smoke test passes on that runtime.

| Runtime            | Works | What it needs |
| ------------------ | ----- | ------------- |
| Bun                | ✅ yes | Nothing.      |
| Node.js            | ✅ yes | Nothing.      |
| Deno               | ✅ yes | Nothing.      |
| Cloudflare Workers | ❌ no  | Nothing.      |

Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/transport-heygen/package.json`: ✅ means the compat smoke test passes on that runtime.

* The session APIs use `fetch`; pass `fetch` to replace the global one (upstream takes an aiohttp
  session).
* The control WebSocket uses the global `WebSocket`. `clientOptions.websocketOptions` takes a
  `webSocketFactory` for runtimes that open sockets differently.
* LiveAvatar's API documents `livekit_agent_token` and `ws_url` as optional. Upstream requires
  them; here a session without a WebSocket URL runs without avatar control (logged as a warning),
  and one without an agent token joins the room with the client token.
* Callbacks given to `HeyGenClient.captureParticipantAudio` and `captureParticipantVideo` receive
  the frame fields (`audio`, `sampleRate`, `numChannels` / `image`, `size`, `format`) rather than
  an `AudioRawFrame` or `ImageRawFrame` instance.