# LiveKit

> WebRTC rooms on a LiveKit server or LiveKit Cloud, with LiveKitTransport, LiveKitRESTHelper and the runner's token helpers.

Source: https://polvallverdu.github.io/pipedog/docs/transports/livekit/



## Overview [#overview]

`LiveKitTransport` joins a LiveKit room as a participant. Every other participant in the room (a
browser with LiveKit's client SDK, a phone call through LiveKit SIP, another bot) can talk to the
bot. LiveKit's server routes the media, so the bot needs no public UDP ports of its own.

* **`LiveKitTransport`**: audio in and out (one `pipecat-audio` microphone track out, each
  participant's audio in), optional video out (one `pipecat-video` camera track) and video in
  (camera and screen share), data messages (RTVI travels over them), participant metadata, and
  SIP DTMF in and out.
* **`LiveKitRESTHelper`**: LiveKit's room API over `fetch` (`deleteRoomByName`).
* **`@pipedog/runner/livekit`**: token helpers (`generateToken`, `generateTokenWithAgent`,
  `generateSessionTokens`, `configure`) and the development runner's `-t livekit`.

<Cards>
  <Card title="Upstream page" href="https://docs.pipecat.ai/api-reference/server/services/transport/livekit">
    Pipecat's documentation for the same transport.
  </Card>

  <Card title="Source" href="https://github.com/polvallverdu/pipedog/tree/main/packages/transport-livekit/src">
    `transport.ts`, `utils.ts` and `rtc.ts`.
  </Card>

  <Card title="LiveKit docs" href="https://docs.livekit.io">
    Servers, LiveKit Cloud, tokens and client SDKs.
  </Card>
</Cards>

## Runtimes [#runtimes]

| Runtime            | Works | What it needs |
| ------------------ | ----- | ------------- |
| Bun                | ✅ yes | Nothing.      |
| Node.js            | ✅ yes | Nothing.      |
| Deno               | ✅ yes | Nothing.      |
| Cloudflare Workers | ❌ no  | Nothing.      |

Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/transport-livekit/package.json`: ✅ means the compat smoke test passes on that runtime.

|                                                                   | Bun | Node ≥ 22 | Deno              | Cloudflare Workers |
| ----------------------------------------------------------------- | --- | --------- | ----------------- | ------------------ |
| `LiveKitTransport` (media)                                        | ✅   | ✅         | ✅ (`--allow-all`) | ❌                  |
| `LiveKitRESTHelper`, `@pipedog/runner/livekit` (tokens, room API) | ✅   | ✅         | ✅                 | not verified       |

The transport's media goes through LiveKit's Node SDK, `@livekit/rtc-node`, which loads a native
N-API addon. The end-to-end test joins a real room on Bun 1.4, Node 24 and Deno 2.9 and checks data
messages and audio both ways. Workers can't load native code, so the transport doesn't run there.
Importing the package works everywhere: the addon is only loaded when the transport is set up.

The token helpers sign with `livekit-server-sdk`, which is plain JavaScript on WebCrypto and
`fetch`.

## Installation [#installation]

```bash
bun add @pipedog/transport-livekit @livekit/rtc-node
```

```ts
import { LiveKitParams, LiveKitTransport } from "@pipedog/transport-livekit/transport";
import { LiveKitRESTHelper } from "@pipedog/transport-livekit/utils";
import { configure, generateSessionTokens } from "@pipedog/runner/livekit";
```

`@livekit/rtc-node` is an optional peer dependency, so install it yourself (1.1.0 or newer in the
1.x line). It ships prebuilt binaries for these platforms only:

| OS               | Architectures                                    |
| ---------------- | ------------------------------------------------ |
| macOS            | arm64 (`darwin-arm64`), x64 (`darwin-x64`)       |
| Linux with glibc | x64 (`linux-x64-gnu`), arm64 (`linux-arm64-gnu`) |
| Windows          | x64 (`win32-x64-msvc`)                           |

There is no build for musl (Alpine Linux), so use a Debian or Ubuntu based image. Without the
package, or on another platform, setting up the transport fails with an error that says how to
install it.

## Prerequisites [#prerequisites]

* **A LiveKit server.** Either [LiveKit Cloud](https://cloud.livekit.io) or your own
  `livekit-server` (for local development: `docker run --rm -p 7880:7880 -p 7881:7881
  -p 7882:7882/udp livekit/livekit-server --dev --node-ip 127.0.0.1`, whose key is `devkey` and
  secret `secret`).
* **Its API key and secret**, to sign access tokens: `LIVEKIT_API_KEY` and `LIVEKIT_API_SECRET`.
  The server URL goes in `LIVEKIT_URL` (`wss://<project>.livekit.cloud`, or `ws://localhost:7880`).
* **A client** in the same room: LiveKit's [Agents Playground](https://agents-playground.livekit.io),
  [Meet](https://meet.livekit.io), or your own app with a LiveKit client SDK.
* A long-running Bun, Node or Deno process on one of the platforms above.

## LiveKitTransport [#livekittransport]

### Configuration [#configuration]

`new LiveKitTransport({ url, token, roomName, params?, inputName?, outputName? })`. The token is
an access token whose identity is the bot's (see [Tokens](#tokens)).

| Option | Type | Required | Description |
|---|---|---|---|
| `url` | `string` | yes | LiveKit server URL to connect to. |
| `token` | `string` | yes | Authentication token for the room. |
| `roomName` | `string` | yes | Name of the LiveKit room to join. |
| `params` | `LiveKitParams \| null \| undefined` | no | Configuration parameters for the transport. |
| `inputName` | `string \| null \| undefined` | no | Optional name for the input transport. |
| `outputName` | `string \| null \| undefined` | no | Optional name for the output transport. |

`LiveKitParams` is a `TransportParams` with three more fields:

| Option | Type | Required | Description |
|---|---|---|---|
| `audioInUserTracks` | `boolean \| undefined` | no | Receive each participant's audio as its own stream of `UserAudioRawFrame`s tagged with their identity (the default). All of these streams enter the pipeline through the one input transport, so a pipeline that runs a single STT and VAD should only receive one of them. When false, the participants' audio is mixed into a single stream of `InputAudioRawFrame`s. |
| `audioOutQueueSizeMs` | `number \| undefined` | no | Buffer size of the outgoing audio source, in milliseconds (LiveKit's default is 1000). |
| `videoOutMaxBitrate` | `number \| null \| undefined` | no | Maximum bitrate of the published video track, in bits per second, capped at `videoOutFramerate`. LiveKit chooses the encoding from the track resolution when unset. |
| `audioOutEnabled` | `boolean \| undefined` | no | Enable audio output streaming. |
| `audioOutSampleRate` | `number \| null \| undefined` | no | Output audio sample rate in Hz. |
| `audioOutChannels` | `number \| undefined` | no | Number of output audio channels. |
| `audioOutBitrate` | `number \| undefined` | no | Output audio bitrate in bits per second. |
| `audioOut10msChunks` | `number \| undefined` | no | Number of 10ms chunks to buffer for output. |
| `audioOutFilter` | `BaseAudioFilter \| null \| undefined` | no | Audio filter to apply to output audio of the default destination, before the mixer. Audio the transport generates itself (mixer audio, end silence, DTMF tones) is not filtered. |
| `audioOutVolume` | `number \| undefined` | no | Volume of the output audio, as a multiplier on its samples, applied after the filter and before the mixer. A `VolumeFrame` changes it while the transport runs, and a `VolumeGainFrame` adjusts it for some audio. |
| `audioOutMixer` | `AudioOutMixer \| null \| undefined` | no | Audio mixer instance or destination mapping. |
| `audioOutDestinations` | `string[] \| undefined` | no | List of audio output destination identifiers. |
| `audioOutEndSilenceSecs` | `number \| undefined` | no | How much silence (seconds) to send after an EndFrame (0 for no silence). |
| `audioOutAutoSilence` | `boolean \| undefined` | no | Insert silence frames when the audio output queue is empty. When false, the transport will wait for audio data instead of inserting silence. |
| `audioOutWriteTimeoutSecs` | `number \| undefined` | no | How long (seconds) a single write to the transport may take before the peer is considered gone. A client that stops reading leaves the write waiting with nothing to fail, so it would otherwise never return. |
| `audioInEnabled` | `boolean \| undefined` | no | Enable audio input streaming. |
| `audioInSampleRate` | `number \| null \| undefined` | no | Input audio sample rate in Hz. |
| `audioInChannels` | `number \| undefined` | no | Number of input audio channels. |
| `audioInFilter` | `BaseAudioFilter \| null \| undefined` | no | Audio filter to apply to input audio. |
| `audioInStreamOnStart` | `boolean \| undefined` | no | Start audio streaming immediately on transport start. |
| `audioInPassthrough` | `boolean \| undefined` | no | Pass through input audio frames downstream. |
| `videoInEnabled` | `boolean \| undefined` | no | Enable video input streaming. |
| `videoInSources` | `Record<string, VideoInSourceParams \| VideoInSourceParamsOptions> \| undefined` | no | Video sources to capture from each user as they connect, keyed by source (`"camera"`, `"screenVideo"`, or a transport-specific custom source). Sources not listed here are captured only when the application asks for them, e.g. with the transport's `captureParticipantVideo()`. Requires `videoInEnabled`. Transports that don't support it log a warning and ignore it, so check the logs if video doesn't arrive as configured. |
| `videoOutEnabled` | `boolean \| undefined` | no | Enable video output streaming. |
| `videoOutIsLive` | `boolean \| undefined` | no | Enable real-time video output streaming. |
| `videoOutWidth` | `number \| undefined` | no | Video output width in pixels. |
| `videoOutHeight` | `number \| undefined` | no | Video output height in pixels. |
| `videoOutFramerate` | `number \| undefined` | no | Video output frame rate in FPS. |
| `videoOutColorFormat` | `string \| undefined` | no | Video output color format string. |
| `videoOutCodec` | `string \| null \| undefined` | no | Preferred video codec for output (e.g., 'VP8', 'H264', 'H265'). |
| `videoOutDestinations` | `string[] \| undefined` | no | List of video output destination identifiers. |

### Usage [#usage]

```ts title="bot.ts"
import { Pipeline } from "@pipedog/core/pipeline/pipeline";
import { PipelineWorker } from "@pipedog/core/pipeline/worker";
import { WorkerRunner } from "@pipedog/core/workers/runner";
import { configure } from "@pipedog/runner/livekit";
import { LiveKitParams, LiveKitTransport } from "@pipedog/transport-livekit/transport";

// LIVEKIT_URL, LIVEKIT_API_KEY, LIVEKIT_API_SECRET and LIVEKIT_ROOM_NAME (or -r/--room).
// configure() mints an agent token for the bot and logs one for a user.
const [url, token, roomName] = await configure();

const transport = new LiveKitTransport({
  url,
  token,
  roomName,
  params: new LiveKitParams({ audioInEnabled: true, audioOutEnabled: true }),
});

const pipeline = new Pipeline([
  transport.input(),
  // ... STT, LLM context aggregators, LLM, TTS
  transport.output(),
]);
const worker = new PipelineWorker(pipeline);

transport.eventHandler(
  "on_first_participant_joined",
  async (_t: unknown, participantId: string) => {
    console.log(`${participantId} joined`);
  },
);
transport.eventHandler("on_participant_disconnected", async () => {
  await worker.cancel();
});

const runner = new WorkerRunner();
await runner.addWorkers(worker);
await runner.run();

```

### Event handlers [#event-handlers]

Every `participantId` is the participant's LiveKit **identity**, the one set in its token, not its
session id.

| Event                                                      | Arguments                                            | When                                                                                  |
| ---------------------------------------------------------- | ---------------------------------------------------- | ------------------------------------------------------------------------------------- |
| `on_connected`                                             | `(transport)`                                        | The bot joined the room.                                                              |
| `on_disconnected`                                          | `(transport)`                                        | The bot left the room, or the server disconnected it.                                 |
| `on_before_disconnect`                                     | `(transport)`                                        | Sync, just before the bot leaves.                                                     |
| `on_first_participant_joined`                              | `(transport, participantId)`                         | The first participant joined (or was already in the room).                            |
| `on_participant_connected`                                 | `(transport, participantId)`                         | A participant joined.                                                                 |
| `on_participant_disconnected`                              | `(transport, participantId)`                         | A participant left.                                                                   |
| `on_participant_left`                                      | `(transport, participantId, reason)`                 | Same, with `"disconnected"`.                                                          |
| `on_client_connected`, `on_client_disconnected`            | `(transport, { id })`                                | Aliases with the shape other transports use.                                          |
| `on_audio_track_subscribed`, `on_audio_track_unsubscribed` | `(transport, participantId)`                         | A participant's audio started or stopped.                                             |
| `on_video_track_subscribed`, `on_video_track_unsubscribed` | `(transport, participantId)`                         | A participant's video started or stopped.                                             |
| `on_app_message`                                           | `(transport, message, participantId)`                | A JSON object arrived as a data message (RTVI uses this).                             |
| `on_data_received`                                         | `(transport, data, participantId)`                   | Any data message, as bytes. `participantId` is `null` for messages a server SDK sent. |
| `on_dtmf_event`                                            | `(transport, { tone, digit, code, participant_id })` | A SIP participant pressed a key. An `InputDTMFFrame` is pushed too.                   |
| `on_active_speaker_changed`                                | `(transport, { id })`                                | The room's loudest speaker changed.                                                   |

### Methods [#methods]

* `getParticipants()`, `getParticipantMetadata(participantId)`, `setMetadata(metadata)`.
* `muteParticipant(participantId)` / `unmuteParticipant(participantId)`: stop or resume receiving
  a participant's audio (LiveKit doesn't let one participant mute another's microphone).
* `sendMessage(message, participantId?)` / `sendMessageUrgent(...)`: a data message to everyone or
  to one participant. `OutputTransportMessageFrame`s in the pipeline go out the same way.
* `captureParticipantVideo(participantId, framerate?, videoSource?, { onRequestOnly })`.

### Notes [#notes]

* Audio comes in at 48 kHz from LiveKit and is resampled to the pipeline's rate. With
  `audioInUserTracks` (the default) each participant's audio is a stream of `UserAudioRawFrame`s
  with their identity; set it to `false` to mix everyone into one stream.
* `OutputDTMFFrame` is sent as a native DTMF event (`publishDtmf`), one key per frame.
* An `InterruptionFrame` clears the audio already queued in LiveKit's audio source, so the bot
  stops talking at once.
* Connecting is retried 3 times, 4 seconds apart.

## Tokens [#tokens]

The bot and each user need their own token for the room, signed with the API key and secret.
`@pipedog/runner/livekit` signs them with `livekit-server-sdk`:

* `generateTokenWithAgent(room, identity, key, secret)`: the bot's token (`agent: true`, so
  LiveKit's clients show it as an agent).
* `generateToken(room, identity, key, secret)`: a user's token.
* `generateSessionTokens(room, sessionId, key, secret)`: both, with identities suffixed by the
  session id so sessions sharing a room don't evict each other.
* `configure()`: reads `LIVEKIT_URL`, `LIVEKIT_API_KEY`, `LIVEKIT_API_SECRET` and the room
  (`-r/--room` or `LIVEKIT_ROOM_NAME`), and returns `[url, botToken, roomName]`. It logs a user
  token you can paste into a client.

All of them are async (signing uses WebCrypto).

## With the development runner [#with-the-development-runner]

`bun bot.ts -t livekit` serves `POST /start`. Each call creates a session: it mints a bot token and
a user token for `LIVEKIT_ROOM_NAME` (or a new `pipecat-xxxxxxxx` room), starts your `bot()` with
`LiveKitRunnerArguments`, and answers `{ url, token, sessionId }` for the user. `createTransport`
builds the transport from those arguments:

```ts title="bot.ts"
import type { RunnerArguments } from "@pipedog/runner";
import { createTransport } from "@pipedog/runner/utils";

/** The bot the development runner starts for each session (`bun bot.ts -t livekit`). */
export async function bot(runnerArgs: RunnerArguments): Promise<void> {
  const transport = await createTransport(runnerArgs, {
    livekit: () => new LiveKitParams({ audioInEnabled: true, audioOutEnabled: true }),
  });
  // ... build the pipeline around transport.input() and transport.output()
  void transport;
}
```

See [Runner](https://polvallverdu.github.io/pipedog/docs/runner.md).

## LiveKitRESTHelper [#livekitresthelper]

`new LiveKitRESTHelper({ apiKey, apiSecret, apiUrl, fetch? })`. `deleteRoomByName(roomName)`
deletes a room and disconnects everyone in it. `apiUrl` is the server's HTTP URL
(`https://<project>.livekit.cloud`). `fetch` replaces the global one (upstream takes an aiohttp
session).