# @pipedog/asyncio > Tasks, cooperative cancellation, timeouts, queues and events with Python asyncio semantics. Source: https://polvallverdu.github.io/pipedog/docs/api/asyncio/ ```ts import { createTask, Event, Queue, sleep, Task, waitFor } from "@pipedog/asyncio"; ``` See [Interruptions & cancellation](https://polvallverdu.github.io/pipedog/docs/concepts/interruptions-and-cancellation.md) for how cancellation points work. ## Task [#task] A unit of concurrent work, like `asyncio.Task`. Created with `createTask()`. Awaiting a task (it is thenable) is a cancellation point, and cancelling the awaiting task also cancels the awaited one. Only `await` links the two: `task.then(cb)`, `task.promise` and `addDoneCallback` observe the task without suspending the caller, so cancelling the caller cancels the caller alone. | Option | Type | Required | Description | |---|---|---|---| | `name` | `string` | yes | | | `getName` | `() => string` | yes | asyncio Task.get_name() | | `setName` | `(name: string) => void` | yes | | | `signal` | `AbortSignal` | yes | Aborted when cancellation is requested. Pass it to fetch() and friends. Once every request is withdrawn with `uncancel()`, a fresh signal is used. | | `cancelling` | `() => number` | yes | Number of pending cancellation requests (asyncio Task.cancelling()). | | `uncancel` | `() => number` | yes | | | `setResult` | `(_value: T) => void` | yes | | | `setException` | `(_error: unknown) => void` | yes | | | `cancel` | `(message?: string) => boolean` | yes | Request cancellation. Returns false if the task is already done. | | `checkpoint` | `() => void` | yes | Throw CancelledError if a cancellation request is pending. Primitives call this on their fast paths, so a task that resumes from a foreign await after being cancelled stops at the next one. | | `promise` | `Promise` | yes | A promise that settles with the task, without being a cancellation point. | | `toString` | `() => string` | yes | Returns a string representation of an object. | | `done` | `() => boolean` | yes | | | `cancelled` | `() => boolean` | yes | | | `result` | `() => T` | yes | | | `exception` | `() => unknown` | yes | | | `addDoneCallback` | `(callback: (future: Task) => void) => void` | yes | | | `removeDoneCallback` | `(callback: (future: Task) => void) => number` | yes | | | `then` | `(onfulfilled?: ((value: T) => R1 \| PromiseLike) \| null \| undefined, onrejected?: ((reason: unknown) => R2 \| PromiseLike) \| null \| undefined) => Promise` | yes | | ## Task functions [#task-functions] | Option | Type | Required | Description | |---|---|---|---| | `createTask` | `(fn: TaskFn, name?: string) => Task` | yes | | | `currentTask` | `() => Task \| undefined` | yes | | | `currentSignal` | `() => AbortSignal` | yes | | | `abortable` | `(promise: PromiseLike) => Promise` | yes | | | `checkCancelled` | `() => void` | yes | | | `suspend` | `(register: (resolve: (value: T) => void, reject: (error: unknown) => void) => () => void, onCancel?: () => void) => Promise` | yes | | | `rethrowIfCancelled` | `(e: unknown) => void` | yes | | | `isCancelledError` | `(e: unknown) => e is CancelledError` | yes | | ## Timeouts and composition [#timeouts-and-composition] | Option | Type | Required | Description | |---|---|---|---| | `sleep` | `(seconds: number) => Promise` | yes | | | `waitFor` | `(aw: Awaitable, timeout: number \| null \| undefined) => Promise` | yes | | | `timeout` | `(seconds: number \| null, fn: TaskFn) => Promise` | yes | | | `gather` | `(...aws: { [K in keyof T]: Awaitable; }) => Promise` | yes | | | `gatherSettled` | `(...aws: Awaitable[]) => Promise` | yes | | | `wait` | `>(tasks: Iterable, options?: { timeout?: number \| null; returnWhen?: ReturnWhen; }) => Promise<[Set, Set]>` | yes | | | `shield` | `(aw: Awaitable) => Promise` | yes | | | `time` | `() => number` | yes | | | `monotonic` | `() => number` | yes | | | `monotonicNs` | `() => number` | yes | | `waitFor(aw, timeout)` takes an `Awaitable`: pass a **function** so it runs as a child task that is cancelled on timeout, raising `TimeoutError`. `timeout` is in seconds; `null` waits forever. ## Queue [#queue] `asyncio.Queue`. `LifoQueue` and `PriorityQueue` subclass it by overriding the storage hooks. `get()` and `put()` are cancellation points. | Option | Type | Required | Description | |---|---|---|---| | `maxsize` | `number` | yes | | | `_items` | `T[]` | yes | | | `_head` | `number` | yes | | | `_init` | `() => void` | yes | | | `_put` | `(item: T) => void` | yes | | | `_get` | `() => T` | yes | | | `_size` | `() => number` | yes | | | `_queued` | `() => readonly T[]` | yes | The queued items in storage order (for subclasses that inspect them). | | `qsize` | `() => number` | yes | | | `empty` | `() => boolean` | yes | | | `full` | `() => boolean` | yes | | | `put` | `(item: T) => Promise` | yes | | | `putNowait` | `(item: T) => void` | yes | | | `get` | `() => Promise` | yes | | | `getNowait` | `() => T` | yes | | | `taskDone` | `() => void` | yes | | | `join` | `() => Promise` | yes | | | `shutdown` | `(immediate?: boolean) => void` | yes | | ## Event [#event] `asyncio.Event`. `wait()` is a cancellation point. | Option | Type | Required | Description | |---|---|---|---| | `isSet` | `() => boolean` | yes | | | `set` | `() => void` | yes | | | `clear` | `() => void` | yes | | | `wait` | `() => Promise` | yes | | ## Lock and Semaphore [#lock-and-semaphore] `async with lock:` is `await lock.run(async () => { ... })`. | Option | Type | Required | Description | |---|---|---|---| | `locked` | `() => boolean` | yes | | | `acquire` | `() => Promise` | yes | | | `release` | `() => void` | yes | | | `run` | `(fn: () => Promise \| R) => Promise` | yes | `async with lock:` → `await lock.run(async () => {...})` | | Option | Type | Required | Description | |---|---|---|---| | `locked` | `() => boolean` | yes | Due to state, or FIFO rules (must allow others to run first). | | `acquire` | `() => Promise` | yes | | | `release` | `() => void` | yes | | | `run` | `(fn: () => Promise \| R) => Promise` | yes | | ## Errors [#errors] | Class | Python | | ------------------------------------------ | -------------------------------------------------- | | `CancelledError` | `asyncio.CancelledError` | | `TimeoutError` | `asyncio.TimeoutError` / `TimeoutError` | | `InvalidStateError` | `asyncio.InvalidStateError` | | `QueueEmpty`, `QueueFull`, `QueueShutDown` | `asyncio.QueueEmpty`, `QueueFull`, `QueueShutDown` | # FrameProcessor > FrameProcessor options, FrameProcessorSetup and the FrameProcessor class, from @pipedog/core/processors/frame_processor. Source: https://polvallverdu.github.io/pipedog/docs/api/frame-processor/ ```ts import { FrameDirection, FrameProcessor, type FrameProcessorOptions, FrameProcessorSetup, } from "@pipedog/core/processors/frame_processor"; ``` See [Frame processors](https://polvallverdu.github.io/pipedog/docs/concepts/frame-processors.md) for a guide. ## FrameProcessorOptions [#frameprocessoroptions] The options object every processor constructor accepts (and passes to `super(options)`). | Option | Type | Required | Description | |---|---|---|---| | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | It extends `BaseObjectOptions` from `@pipedog/core/utils/base_object`: | Option | Type | Required | Description | |---|---|---|---| | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | ## FrameDirection [#framedirection] ```ts export const FrameDirection = { DOWNSTREAM: 1, UPSTREAM: 2 } as const; export type FrameDirection = (typeof FrameDirection)[keyof typeof FrameDirection]; ``` ## FrameProcessorSetup [#frameprocessorsetup] Passed to `setup()` by the pipeline before any frame flows. Built from these options: | Option | Type | Required | Description | |---|---|---|---| | `clock` | `BaseClock` | yes | | | `taskManager` | `BaseTaskManager` | yes | | | `pipelineWorker` | `PipelineWorker \| null` | yes | | | `audioInSampleRate` | `number \| undefined` | no | | | `audioOutSampleRate` | `number \| undefined` | no | | | `enableExpressiveConversation` | `boolean \| undefined` | no | | | `enableMetrics` | `boolean \| undefined` | no | | | `enableTracing` | `boolean \| undefined` | no | | | `enableUsageMetrics` | `boolean \| undefined` | no | | | `expressionPolicy` | `ExpressionPolicy \| null \| undefined` | no | | | `observer` | `BaseObserver \| null \| undefined` | no | | | `reportOnlyInitialTtfb` | `boolean \| undefined` | no | | | `tracingContext` | `TracingContext \| null \| undefined` | no | | ## FrameProcessor [#frameprocessor] The public members of the class (including those inherited from `BaseObject`): | Option | Type | Required | Description | |---|---|---|---| | `_prev` | `FrameProcessor \| null` | yes | | | `_next` | `FrameProcessor \| null` | yes | | | `_enableDirectMode` | `boolean` | yes | | | `_setup` | `FrameProcessorSetup \| null` | yes | | | `_cancelling` | `boolean` | yes | | | `_metrics` | `FrameProcessorMetrics` | yes | | | `_isUsable` | `boolean` | yes | | | `isUsable` | `boolean` | yes | Whether this processor can still do its job. A processor stays usable through failures it might recover from, and becomes unusable once its work can no longer succeed: a provider has rejected its API key, model or voice, or it has failed enough times to stop trying. Services stop accepting work and stop reconnecting once this is false. Errors set this as they are reported, so an error handler reading `frame.processor.isUsable` sees the verdict that came with the error. | | `setUsable` | `(isUsable: boolean) => Promise` | yes | Set whether this processor can be given work. Call this to bring back a processor that became unusable, once whatever stopped it working has been dealt with. | | `processors` | `FrameProcessor[]` | yes | The sub-processors contained within this processor. Only compound processors (e.g. pipelines and parallel pipelines) have sub-processors. | | `entryProcessors` | `FrameProcessor[]` | yes | The entry processors for this processor: the first processors in a compound processor. Non-compound processors return an empty list. | | `next` | `FrameProcessor \| null` | yes | The next processor, or null. | | `previous` | `FrameProcessor \| null` | yes | The previous processor, or null. | | `processorSetup` | `FrameProcessorSetup` | yes | The configuration this processor was set up with. | | `metricsEnabled` | `boolean` | yes | Whether metrics collection is enabled. | | `usageMetricsEnabled` | `boolean` | yes | Whether usage metrics collection is enabled. | | `reportOnlyInitialTtfb` | `boolean` | yes | Whether only the initial TTFB should be reported. | | `pipelineWorker` | `PipelineWorker` | yes | The PipelineWorker this processor is running in. Gives access to worker-scoped state, e.g. `this.pipelineWorker.appResources`. | | `workerRunner` | `WorkerRunner` | yes | The WorkerRunner hosting this processor's worker. Use it to reach another worker by name, e.g. `this.workerRunner.getWorker("ui-jobs")`. | | `processorsWithMetrics` | `() => FrameProcessor[]` | yes | Processors that can generate metrics, including nested ones. | | `canGenerateMetrics` | `() => boolean` | yes | Whether this processor can generate metrics. | | `setCoreMetricsData` | `(data: MetricsData) => void` | yes | Set core metrics data for this processor. | | `startTtfbMetrics` | `(options?: { startTime?: number \| null; }) => Promise` | yes | Start time-to-first-byte metrics collection. | | `cancelTtfbMetrics` | `() => Promise` | yes | Abandon the current time-to-first-byte measurement without reporting it. | | `stopTtfbMetrics` | `(options?: { endTime?: number \| null; }) => Promise` | yes | Stop time-to-first-byte metrics collection and push results. | | `processTtfaMetrics` | `(frame: TTSAudioRawFrame) => Promise` | yes | Scan a TTS audio frame for the first audible sample and push TTFA. Call for every audio frame until a measurement is produced. | | `stopTtfatMetrics` | `(options?: { endTime?: number \| null; }) => Promise` | yes | Stop time-to-first-answer-token metrics collection and push results. | | `startProcessingMetrics` | `(options?: { startTime?: number \| null; }) => Promise` | yes | Start processing metrics collection. | | `stopProcessingMetrics` | `(options?: { endTime?: number \| null; }) => Promise` | yes | Stop processing metrics collection and push results. | | `startLlmUsageMetrics` | `(tokens: LLMTokenUsage) => Promise` | yes | Start LLM usage metrics collection. | | `startSttUsageMetrics` | `(usage: STTUsage) => Promise` | yes | Start STT usage metrics collection. | | `startTtsUsageMetrics` | `(text: string) => Promise` | yes | Start TTS usage metrics collection. | | `startTextAggregationMetrics` | `() => Promise` | yes | Start text aggregation time metrics collection. | | `stopTextAggregationMetrics` | `() => Promise` | yes | Stop text aggregation time metrics collection and push results. | | `stopAllMetrics` | `() => Promise` | yes | Stop all active metrics collection. | | `setup` | `(setup: FrameProcessorSetup) => Promise` | yes | Set up the processor with required components. This is where a processor connects and does its other slow start-up work. A pipeline sets its processors up concurrently, so a resource shared with another processor needs guarding (see `utils/shared`). | | `_setupStarted` | `() => void` | yes | Hold the StartFrame until `_setupFinished` is called. Called by the pipeline before setup. | | `_setupFinished` | `() => void` | yes | Let this processor start, once its own setup() has returned. | | `cleanup` | `() => Promise` | yes | Release this processor's resources at teardown. This base implementation cancels only the processor's internal input/process tasks; tasks created via `createTask` are released by an override. Runs concurrently with every other processor's cleanup. | | `link` | `(processor: FrameProcessor) => void` | yes | Link this processor to the next processor in the pipeline. | | `getClock` | `() => BaseClock` | yes | The clock used by this processor. | | `getEventLoop` | `() => EventLoop` | yes | The event loop used by this processor. | | `queueFrame` | `(frame: Frame, direction?: FrameDirection, callback?: FrameCallback \| null) => Promise` | yes | Queue a frame for processing. | | `pauseProcessingFrames` | `() => Promise` | yes | Pause processing of queued frames. | | `pauseProcessingSystemFrames` | `() => Promise` | yes | Pause processing of queued system frames. | | `resumeProcessingFrames` | `() => Promise` | yes | Resume processing of queued frames. | | `resumeProcessingSystemFrames` | `() => Promise` | yes | Resume processing of queued system frames. | | `pauseProcessingAllFramesUntil` | `(ready: () => Promise, options?: { timeout?: number; }) => Promise` | yes | Hold frames arriving at this processor until `ready` resolves. Useful for a processor that cannot act on frames until some condition holds, such as one that establishes a connection in the background. Frames wait in the processor's queues and are delivered in order once the condition resolves, so nothing is lost. The frame being processed when this is called is unaffected: the pause takes hold from the next frame onwards. Both queues are held, so the pause is always lifted: when `ready` resolves, when `timeout` (seconds) elapses, or at teardown. | | `processFrame` | `(frame: Frame, direction: FrameDirection) => Promise` | yes | Process a frame. Subclasses override this and call `await super.processFrame(frame, direction)` first. | | `_classifyError` | `(_exception: unknown) => ErrorCategory \| null` | yes | Classify an exception this processor knows the shape of. Override for providers that signal failures through SDK-specific errors rather than an HTTP status code. Return null to fall back to `classifyHttpException`. | | `pushError` | `(errorMsg: string, options?: { exception?: unknown; category?: ErrorCategory \| null; forceTreatAsPermanent?: boolean; }) => Promise` | yes | Create and push an ErrorFrame upstream, with context about which processor generated the error. | | `pushErrorFrame` | `(error: ErrorFrame, options?: { forceTreatAsPermanent?: boolean; }) => Promise` | yes | Push an error frame upstream. | | `pushFrame` | `(frame: Frame, direction?: FrameDirection) => Promise` | yes | Push a frame to the next (downstream) or previous (upstream) processor. | | `broadcastInterruption` | `() => Promise` | yes | Broadcast an InterruptionFrame both upstream and downstream. | | `broadcastFrame` | `(frameCls: new (options: O) => Frame, options?: O) => Promise` | yes | Broadcast a frame of the specified class upstream and downstream. Creates two instances of the given frame class from the provided options (without deep-copying them) and pushes them upstream and downstream. | | `broadcastFrameInstance` | `(frame: Frame) => Promise` | yes | Broadcast a frame instance upstream and downstream. Creates two new frames shallow-copying all fields from the original frame except `id` and `name`, which get fresh values. Prefer `broadcastFrame()` when you are the creator of the frame. | | `_startInterruption` | `() => Promise` | yes | Start handling an interruption by cancelling current tasks. | | `_cancelFrameInProgress` | `(frame: Frame) => Promise` | yes | Stop processing a frame if it is the one in progress, leaving the queue as it is. Unlike `_startInterruption`, this drops nothing else: the frames queued behind the cancelled one, interruptible or not, are processed next. The cancelled frame's callback, if any, is not called. Call it from a system frame's processing, as interruptions are, so it runs on the input task and never overlaps an interruption. | | `hasQueuedFrame` | `(frameType: FrameClass) => boolean` | yes | Whether a frame of the given type is waiting in the processing queue. | | `_id` | `number` | yes | | | `_name` | `string` | yes | | | `_taskManager` | `BaseTaskManager \| null` | yes | | | `_eventHandlers` | `Map` | yes | | | `_eventTasks` | `Set<[string, Task]>` | yes | | | `id` | `number` | yes | The unique identifier for this object. | | `name` | `string` | yes | The object's name, either custom-provided or auto-generated. | | `taskManager` | `BaseTaskManager` | yes | The task manager. Throws if not initialized. | | `createTask` | `(coroutine: TaskFn, name?: string \| null) => Task` | yes | Create a new task managed by this object's task manager. | | `cancelTask` | `(task: Task, timeout?: number \| null) => Promise` | yes | Cancel a task managed by this object's task manager. A default timeout of 1 second avoids freezes caused by code that swallows cancellation. | | `eventHandler` | `>(eventName: string, handler: F) => F` | yes | Register a handler for an event. Equivalent of Python's `@obj.event_handler("on_x")` decorator; returns the handler. | | `addEventHandler` | `(eventName: string, handler: EventHandlerFn) => void` | yes | Add an event handler for the specified event. Can be sync or async. | | `removeEventHandler` | `(eventName: string, handler: EventHandlerFn) => void` | yes | Remove a previously added event handler. No-op if unknown. | | `_registerEventHandler` | `(eventName: string, sync?: boolean) => void` | yes | Register an event handler type. | | `_hasEventHandlers` | `(eventName: string) => boolean` | yes | Whether any handler is registered for `eventName`. Not in upstream: code that fires an event for every frame checks it first, so an event nobody listens to costs no await. | | `_callEventHandler` | `(eventName: string, ...args: any[]) => Promise` | yes | | | `_runHandler` | `(eventName: string, handler: EventHandlerFn, ...args: any[]) => Promise` | yes | | | `toString` | `() => string` | yes | | # Frames > The Frame base class, frame categories and common frame options, from @pipedog/core/frames/frames. Source: https://polvallverdu.github.io/pipedog/docs/api/frames/ ```ts import { ControlFrame, DataFrame, Frame, SystemFrame } from "@pipedog/core/frames/frames"; ``` See [Frames](https://polvallverdu.github.io/pipedog/docs/concepts/frames.md) for how the categories are processed. ## Frame [#frame] The base class of every frame. Its fields are set automatically. | Option | Type | Required | Description | |---|---|---|---| | `id` | `number` | yes | | | `name` | `string` | yes | | | `pts` | `number \| null` | yes | | | `broadcastSiblingId` | `number \| null` | yes | | | `metadata` | `Record` | yes | | | `transportSource` | `string \| null` | yes | | | `transportDestination` | `string \| null` | yes | | | `interruptible` | `boolean` | yes | | | `toString` | `() => string` | yes | | ## SystemFrame, DataFrame, ControlFrame [#systemframe-dataframe-controlframe] The three base categories add no fields: they only decide how frames are queued and whether interruptions drop them. `SystemFrame`s are processed with high priority and never dropped; `DataFrame`s and `ControlFrame`s are processed in order and dropped on interruption unless `interruptible` is `false`. ## TextFrame [#textframe] | Option | Type | Required | Description | |---|---|---|---| | `text` | `string` | yes | | | `skipTts` | `boolean \| null` | yes | | | `includesInterFrameSpaces` | `boolean` | yes | | | `appendToContext` | `boolean` | yes | | | `toString` | `() => string` | yes | | | `id` | `number` | yes | | | `name` | `string` | yes | | | `pts` | `number \| null` | yes | | | `broadcastSiblingId` | `number \| null` | yes | | | `metadata` | `Record` | yes | | | `transportSource` | `string \| null` | yes | | | `transportDestination` | `string \| null` | yes | | | `interruptible` | `boolean` | yes | | Constructor options: | Option | Type | Required | Description | |---|---|---|---| | `text` | `string` | yes | The text content. | ## Audio frames [#audio-frames] `InputAudioRawFrame` (a system frame) and `OutputAudioRawFrame` (a data frame) share the `AudioRawFrame` mixin. Their options: | Option | Type | Required | Description | |---|---|---|---| | `audio` | `Uint8Array` | yes | Raw audio bytes in PCM format (16-bit little endian). | | `sampleRate` | `number` | yes | Audio sample rate in Hz. | | `numChannels` | `number` | yes | Number of audio channels. | ## Lifecycle frames [#lifecycle-frames] `StartFrame` starts the pipeline. `EndFrame` (control, uninterruptible) ends it gracefully, `CancelFrame` (system) stops it immediately; both take an optional `reason`: | Option | Type | Required | Description | |---|---|---|---| | `reason` | `unknown` | no | Optional reason for pushing an end frame. | ## ErrorFrame [#errorframe] | Option | Type | Required | Description | |---|---|---|---| | `error` | `string` | yes | Description of the error that occurred. | | `processor` | `FrameProcessor \| null \| undefined` | no | The frame processor that generated the error. | | `exception` | `unknown` | no | The exception that occurred. | | `category` | `ErrorCategory \| null \| undefined` | no | What kind of failure this was. | ## MetricsFrame [#metricsframe] | Option | Type | Required | Description | |---|---|---|---| | `data` | `MetricsData[]` | yes | List of metrics data collected by the processor. | # Overview > API reference generated from the pipedog TypeScript sources. Source: https://polvallverdu.github.io/pipedog/docs/api/ The tables in this section are generated at build time with [fumadocs-typescript](https://fumadocs.dev/docs/ui/typescript) (`AutoTypeTable`), straight from `packages/core/src` and `packages/asyncio/src`, so they always match the code. Descriptions come from the JSDoc in the sources, which keeps upstream's docstrings. `Frame`, the three base categories, and common frame options. Processor options, setup, and the `FrameProcessor` class. `PipelineParams`, worker options, runner options, `runTest`. `Task`, `createTask`, `waitFor`, `Queue`, `Event` and friends. Click a row's type to expand its full TypeScript type and documentation. # PipelineWorker & WorkerRunner > PipelineParams, PipelineWorker options, WorkerRunner options and runTest options. Source: https://polvallverdu.github.io/pipedog/docs/api/pipeline-worker/ ```ts import { Pipeline } from "@pipedog/core/pipeline/pipeline"; import { PipelineParams, PipelineWorker } from "@pipedog/core/pipeline/worker"; import { WorkerRunner } from "@pipedog/core/workers/runner"; ``` See [Pipelines & workers](https://polvallverdu.github.io/pipedog/docs/concepts/pipelines-and-workers.md) for a guide. ## PipelineParams [#pipelineparams] `new PipelineParams(options)`: pipeline-wide parameters, passed to every processor through `FrameProcessorSetup`. | Option | Type | Required | Description | |---|---|---|---| | `audioInSampleRate` | `number` | yes | | | `audioOutSampleRate` | `number` | yes | | | `enableExpressiveConversation` | `boolean` | yes | | | `expressionPolicy` | `ExpressionPolicy \| null` | yes | | | `enableHeartbeats` | `boolean` | yes | | | `enableMetrics` | `boolean` | yes | | | `enableUsageMetrics` | `boolean` | yes | | | `heartbeatsPeriodSecs` | `number` | yes | | | `heartbeatsMonitorSecs` | `number` | yes | | | `reportOnlyInitialTtfb` | `boolean` | yes | | | `sendInitialEmptyMetrics` | `boolean` | yes | | | `startMetadata` | `Record` | yes | | ## PipelineWorkerOptions [#pipelineworkeroptions] `new PipelineWorker(pipeline, options)`. | Option | Type | Required | Description | |---|---|---|---| | `active` | `boolean \| undefined` | no | Whether the worker starts active. Forwarded to `BaseWorker`. | | `additionalSpanAttributes` | `Record \| null \| undefined` | no | Optional attributes to propagate as OpenTelemetry conversation span attributes. | | `appResources` | `unknown` | no | Optional application-defined bag of anything your application code may want to share across this session (DB handles, HTTP clients, etc.), passed by reference. Pipecat passes it through untouched and exposes it on the worker itself as `worker.appResources` and passes it to tool handlers as `FunctionCallParams.appResources`. The framework never copies or clears this object; the caller retains their handle and can read any mutations after the worker finishes. | | `bridged` | `readonly string[] \| null \| undefined` | no | Bridge configuration. `null` means the pipeline is not bridged. An empty array `[]` wraps the pipeline with bus edge processors that accept frames from all bridges. An array of names like `["voice"]` accepts only frames from those bridges. The bus comes from `attach()` (called by the runner). | | `cancelOnIdleTimeout` | `boolean \| undefined` | no | Whether reaching the idle timeout should cancel the pipeline worker. When false, the idle event still fires `on_idle_timeout` but the worker is left alone (and `cancelRunnerOnIdleTimeout` is ignored too: opting out of local cancellation also opts out of the runner-wide cancel). | | `cancelRunnerOnIdleTimeout` | `boolean \| undefined` | no | When `cancelOnIdleTimeout` is also true, whether reaching the idle timeout should also cancel the entire `WorkerRunner`. The worker is always cancelled first; when this is true the worker also emits a `BusCancelMessage` so the runner broadcasts cancellation to every other root worker. Defaults to true so a multi-worker bot's helpers shut down with the main pipeline; set to false for a sidecar `PipelineWorker` that should self-cancel on idle without bringing down its peers. | | `cancelTimeoutSecs` | `number \| undefined` | no | Timeout (in seconds) to wait for cancellation to happen cleanly. | | `capabilities` | `BotCapabilities \| null \| undefined` | no | What the bot does, for fields the pipeline can't show. Its known fields replace the ones derived from the pipeline's transports and parameters. See `capabilities`. | | `checkDanglingTasks` | `boolean \| undefined` | no | Whether to warn about tasks left running when the worker finishes. Only applies when the worker owns its task manager; otherwise the runner reports dangling tasks. | | `clock` | `BaseClock \| null \| undefined` | no | Clock implementation for timing operations. | | `conversationId` | `string \| null \| undefined` | no | Optional custom ID for the conversation. | | `enableTracing` | `boolean \| undefined` | no | Whether to enable tracing. | | `enableTurnTracking` | `boolean \| undefined` | no | Whether to enable turn tracking. | | `handleFlushFrame` | `boolean \| null \| undefined` | no | Whether this worker answers a flush probe, bouncing it at the sink and completing it at the source. Defaults to whether the pipeline is unbridged, so a worker wired into someone else's topology leaves the probe to travel on and be completed by the pipeline that owns the transport. A bridged worker with no such peer never completes a flush and every `flushPipeline` call waits out its timeout. | | `enableRtvi` | `boolean \| null \| undefined` | no | Whether to automatically add RTVI support to the pipeline. `null`, the default, adds it unless the pipeline is bridged: a bridged worker has no client of its own, and its RTVI would report every frame a second time as it crosses the bridge. | | `excludeFrames` | `readonly FrameClass[] \| null \| undefined` | no | When `bridged` is set, extra frame types that should not cross the bus (lifecycle frames are always excluded). | | `idleTimeoutFrames` | `readonly FrameClass[] \| undefined` | no | The frames that should trigger an idle timeout if not received within `idleTimeoutSecs`. The default pairs the VAD-only `UserSpeakingFrame` with the turn and transcription frames a provider-driven pipeline reports instead. | | `idleTimeoutSecs` | `number \| null \| undefined` | no | Timeout (in seconds) to consider pipeline idle, or null. If a pipeline is idle the pipeline worker will be cancelled automatically. | | `name` | `string \| null \| undefined` | no | Optional worker name (used for worker-style addressing on the bus). | | `observers` | `BaseObserver[] \| null \| undefined` | no | Observers for monitoring pipeline execution. | | `processorUnusablePolicy` | `ProcessorUnusablePolicy \| undefined` | no | What to do when a processor reports an error that leaves it unable to do its job, such as a service whose API key was rejected. Defaults to `ProcessorUnusablePolicy.CONTINUE`, leaving the decision to `on_pipeline_error` handlers. | | `params` | `PipelineParams \| null \| undefined` | no | Configuration parameters for the pipeline. | | `rtviProcessor` | `RTVIProcessor \| null \| undefined` | no | The RTVI processor to add if RTVI is enabled. | | `rtviObserverParams` | `RTVIObserverParams \| null \| undefined` | no | The RTVI observer parameters to use if RTVI is enabled. | | `setupTimeoutSecs` | `number \| undefined` | no | Timeout (in seconds) to wait for every processor to be set up. Processors connect while they are set up, so one that blocks connecting never lets the pipeline start, and the worker is torn down instead of waiting forever. The `StartFrame` is pushed while processors are still being set up, and each processor holds it only until its own setup is done, so processors ahead of a slow one start without it. | | `startTimeoutSecs` | `number \| undefined` | no | Timeout (in seconds) to wait for the `StartFrame` to reach the end of the pipeline, counted from when every processor is set up. A processor that blocks while handling it never lets the pipeline start, so the worker is torn down instead of waiting forever. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Optional task manager for handling tasks. | ## PipelineOptions [#pipelineoptions] `new Pipeline(processors, options)`. | Option | Type | Required | Description | |---|---|---|---| | `source` | `FrameProcessor \| null \| undefined` | no | An optional pipeline source processor. | | `sink` | `FrameProcessor \| null \| undefined` | no | An optional pipeline sink processor. | ## WorkerRunnerOptions [#workerrunneroptions] `new WorkerRunner(options)`. | Option | Type | Required | Description | |---|---|---|---| | `name` | `string \| null \| undefined` | no | Optional name for the runner instance. Defaults to a UUID-based name. Must be unique across runners in a distributed setup. | | `bus` | `WorkerBus \| null \| undefined` | no | Optional `WorkerBus`. Defaults to a new in-process `AsyncQueueBus`. | | `handleSigint` | `boolean \| undefined` | no | Whether to automatically handle SIGINT signals. | | `handleSigterm` | `boolean \| undefined` | no | Whether to automatically handle SIGTERM signals. | | `forceGc` | `boolean \| undefined` | no | Whether to force garbage collection once every worker has been torn down. | | `checkDanglingTasks` | `boolean \| undefined` | no | Whether to warn about tasks left running on the shared task manager once every worker has finished. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Optional task manager for handling tasks. | `runner.run(options)`: | Option | Type | Required | Description | |---|---|---|---| | `autoEnd` | `boolean \| undefined` | no | When true (the default), the runner ends once every root worker has finished. When false, the runner blocks until `end()` or `cancel()` is called. | ## RunTestOptions [#runtestoptions] `runTest(processor, options)` from `@pipedog/core/tests/utils`. | Option | Type | Required | Description | |---|---|---|---| | `enableRtvi` | `boolean \| undefined` | no | Whether RTVI should be enabled in this test. | | `expectedDownFrames` | `readonly FrameType[] \| null \| undefined` | no | Expected frame types flowing downstream (optional). | | `expectedUpFrames` | `readonly FrameType[] \| null \| undefined` | no | Expected frame types flowing upstream (optional). | | `framesToSend` | `readonly Frame[]` | yes | Sequence of frames to send through the processor. | | `framesToSendDirection` | `FrameDirection \| undefined` | no | Direction to send `framesToSend`. Downstream frames are pushed from the beginning of the pipeline, upstream frames from the end. Defaults to DOWNSTREAM. | | `ignoreStart` | `boolean \| undefined` | no | Whether to ignore StartFrames in frame validation. | | `observers` | `BaseObserver[] \| null \| undefined` | no | Optional list of observers to attach to the pipeline. | | `pipelineParams` | `PipelineParams \| null \| undefined` | no | Optional pipeline parameters. | | `sendEndFrame` | `boolean \| undefined` | no | Whether to send an EndFrame at the end of the test. | | `startTimeout` | `number \| undefined` | no | How long to wait, in seconds, for the pipeline to start before giving up. | # ai-coustics > ai-coustics speech enhancement (AICFilter) and the Quail voice activity detector (AICQuailVADAnalyzer). Deferred in pipedog: not ported yet. Source: https://polvallverdu.github.io/pipedog/docs/audio/aic-filter/ {/* Generated by scripts/generate-providers.ts from data/providers.ts. */} ai-coustics is deferred: it is possible, but it isn't ready to port. **Why.** `@ai-coustics/aic-sdk` is a native addon for darwin and linux, or a browser WebAssembly build, and it needs a license key. Neither fits every pipedog runtime. **What would unblock it.** Deferred until there is a package we can test without a license key, or someone contributes it as an optional package for Node and Bun. Sources: [ai-coustics Node.js SDK](https://docs.ai-coustics.com/sdk/bindings/nodejs). ## What it is [#what-it-is] ai-coustics speech enhancement (AICFilter) and the Quail voice activity detector (AICQuailVADAnalyzer). Upstream, it's these classes: `AICFilter`, `AICQuailVADAnalyzer`, in [`audio/filters/aic_filter.py`](https://github.com/pipecat-ai/pipecat/blob/main/src/pipecat/audio/filters/aic_filter.py) and [`audio/vad/aic_quail_vad.py`](https://github.com/pipecat-ai/pipecat/blob/main/src/pipecat/audio/vad/aic_quail_vad.py). Its pipedog package will be `@pipedog/audio-aic-filter`. * [Pipecat's ai-coustics page](https://docs.pipecat.ai/api-reference/server/services/audio-filters/aic-filter) ## Get it ported [#get-it-ported] * [Search the issues](https://github.com/polvallverdu/pipedog/issues?q=is%3Aissue+ai-coustics) for a request, and upvote it with a reaction. * If there's none, [open a port request](https://github.com/polvallverdu/pipedog/issues/new?template=feature_request.yml\&title=Port+ai-coustics+%28AICFilter%2C+AICQuailVADAnalyzer%29\&kind=Port+an+upstream+module\&upstream=src%2Fpipecat%2Faudio%2Ffilters%2Faic_filter.py%2C+src%2Fpipecat%2Faudio%2Fvad%2Faic_quail_vad.py). * Or port it yourself: the [contributing guide](https://polvallverdu.github.io/pipedog/docs/contributing.md) explains how modules are ported, test by test. The [porting status](https://polvallverdu.github.io/pipedog/docs/contributing/status.md) lists what's done. Until then, see what is ported in [Audio filters & mixers](https://polvallverdu.github.io/pipedog/docs/audio.md). # Ducking > Lower the bot's voice while the user talks over it, with AudioDuckingFilter, on any transport and every runtime. Source: https://polvallverdu.github.io/pipedog/docs/audio/ducking/ ## Overview [#overview] When the user starts talking while the bot speaks, the bot normally talks on at full volume until the overlap counts as an interruption (or never, for a short "uh-huh"). `AudioDuckingFilter` lowers the bot's voice while the user's voice activity lasts, so it sounds like the bot heard them, and brings it back when they stop. This is pipedog only: upstream Pipecat has no ducking. The output transport queues the bot's audio well before it plays, so the volume can't be changed where the audio is produced. The filter is an **output audio filter** (`audioOutFilter`): the transport runs it on each chunk as the chunk leaves, so the change is heard at once, whatever the transport or serializer (WebRTC, WebSocket, Twilio, Telnyx...). `ducking_filter.ts`. ## Installation [#installation] ```bash bun add @pipedog/audio-ducking ``` ```ts import { AudioDuckingFilter } from "@pipedog/audio-ducking"; ``` ## Runtimes [#runtimes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------------- | | Bun | ✅ yes | Nothing to install. | | Node.js | ✅ yes | Nothing to install. | | Deno | ✅ yes | Nothing to install. | | Cloudflare Workers | ✅ yes | Nothing to install. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/audio-ducking/package.json`: ✅ means the compat smoke test passes on that runtime. ## Usage [#usage] Set the filter as the transport's `audioOutFilter`, and add its `observer` to the pipeline worker: ```ts import { AudioDuckingFilter } from "@pipedog/audio-ducking"; import { PipelineWorker } from "@pipedog/core/pipeline/worker"; import { TransportParams } from "@pipedog/core/transports/base_transport"; const ducking = new AudioDuckingFilter({ duckedGain: 0.35 }); const params = new TransportParams({ audioOutEnabled: true, audioOutFilter: ducking }); // ... the transport and pipeline ... const worker = new PipelineWorker(pipeline, { observers: [ducking.observer] }); ``` The observer watches the pipeline's frames, so no processor goes in the pipeline: | Frame | Effect | | ----------------------------- | --------------------------------------------------------------------------------- | | `VADUserStartedSpeakingFrame` | Duck. | | `VADUserStoppedSpeakingFrame` | Restore: the user stopped without interrupting. | | `InterruptionFrame` | Restore: the bot's queued speech was flushed, its next words play at full volume. | | `BotStoppedSpeakingFrame` | Restore. | The gain moves over the first chunk after a change (a linear ramp from the previous gain), so there is no click. You can also call `ducking.duck()` and `ducking.restore()` yourself. At runtime, change the gain or turn ducking off with filter frames: ```ts import { FilterEnableFrame, FilterUpdateSettingsFrame } from "@pipedog/core/frames/frames"; await worker.queueFrame(new FilterUpdateSettingsFrame({ settings: { duckedGain: 0.2 } })); await worker.queueFrame(new FilterEnableFrame({ enable: false })); ``` ## Configuration [#configuration] `new AudioDuckingFilter(options)`: | Option | Type | Default | Description | | ------------ | --------- | ------- | ------------------------------- | | `duckedGain` | `number` | `0.35` | Gain while ducked, 0.0 to 1.0. | | `enabled` | `boolean` | `true` | Whether ducking starts enabled. | The transport has one `audioOutFilter`. To combine ducking with another output filter, call one filter's `filter()` from the other's. # Audio filters & mixers > Clean up the user's audio before it reaches VAD and speech-to-text, and mix background sound into the bot's audio. Source: https://polvallverdu.github.io/pipedog/docs/audio/ {/* Generated by scripts/generate-providers.ts from data/providers.ts. */} An audio filter cleans up the user's audio before it reaches VAD and speech-to-text (set it as `audioInFilter` in the transport's params). An audio mixer blends background sound into the bot's audio (`audioOutMixer`). | Provider | Classes | Package | Status | | ---------------------------------------------- | ---------------------------------------------------------- | -------------------------------- | ------------ | | [RNNoise filter](https://polvallverdu.github.io/pipedog/docs/audio/rnnoise-filter.md) | `RNNoiseFilter` | `@pipedog/audio-rnnoise-filter` | Ported | | [Soundfile mixer](https://polvallverdu.github.io/pipedog/docs/audio/soundfile-mixer.md) | `SoundfileMixer` | `@pipedog/audio-soundfile-mixer` | Ported | | [Ducking](https://polvallverdu.github.io/pipedog/docs/audio/ducking.md) | `AudioDuckingFilter`, `AudioDuckingObserver` | `@pipedog/audio-ducking` | pipedog only | | [TEN VAD](https://polvallverdu.github.io/pipedog/docs/audio/ten-vad.md) | `TenVADAnalyzer` | `@pipedog/audio-ten-vad` | pipedog only | | [ai-coustics](https://polvallverdu.github.io/pipedog/docs/audio/aic-filter.md) | `AICFilter`, `AICQuailVADAnalyzer` | | Deferred | | [Koala](https://polvallverdu.github.io/pipedog/docs/audio/koala-filter.md) | `KoalaFilter` | | Deferred | | [Krisp VIVA](https://polvallverdu.github.io/pipedog/docs/audio/krisp-viva-filter.md) | `KrispVivaFilter`, `KrispVivaVadAnalyzer`, `KrispVivaTurn` | | Deferred | Pipecat's [audio filters & mixers reference](https://docs.pipecat.ai/api-reference/server/services/audio-filters) lists every upstream provider. Those not in this table are on [Other upstream providers](https://polvallverdu.github.io/pipedog/docs/services/other-upstream.md). # Koala > Picovoice Koala noise suppression (KoalaFilter). Deferred in pipedog: not ported yet. Source: https://polvallverdu.github.io/pipedog/docs/audio/koala-filter/ {/* Generated by scripts/generate-providers.ts from data/providers.ts. */} Koala is deferred: it is possible, but it isn't ready to port. **Why.** Picovoice has no Node build of Koala. There is only `@picovoice/koala-web`, for the browser, and every build needs an AccessKey. **What would unblock it.** Deferred until Picovoice ships a Node or server-side WebAssembly build, or someone ports it on top of `@picovoice/koala-web`. Sources: [Koala for web](https://picovoice.ai/docs/quick-start/koala-web). ## What it is [#what-it-is] Picovoice Koala noise suppression (KoalaFilter). Upstream, it's this class: `KoalaFilter`, in [`audio/filters/koala_filter.py`](https://github.com/pipecat-ai/pipecat/blob/main/src/pipecat/audio/filters/koala_filter.py). Its pipedog package will be `@pipedog/audio-koala-filter`. * [Pipecat's Koala page](https://docs.pipecat.ai/api-reference/server/services/audio-filters/koala-filter) ## Get it ported [#get-it-ported] * [Search the issues](https://github.com/polvallverdu/pipedog/issues?q=is%3Aissue+Koala) for a request, and upvote it with a reaction. * If there's none, [open a port request](https://github.com/polvallverdu/pipedog/issues/new?template=feature_request.yml\&title=Port+Koala+%28KoalaFilter%29\&kind=Port+an+upstream+module\&upstream=src%2Fpipecat%2Faudio%2Ffilters%2Fkoala_filter.py). * Or port it yourself: the [contributing guide](https://polvallverdu.github.io/pipedog/docs/contributing.md) explains how modules are ported, test by test. The [porting status](https://polvallverdu.github.io/pipedog/docs/contributing/status.md) lists what's done. Until then, see what is ported in [Audio filters & mixers](https://polvallverdu.github.io/pipedog/docs/audio.md). # Krisp VIVA > Krisp's voice isolation filter, VAD analyzer and turn analyzer. Deferred in pipedog: not ported yet. Source: https://polvallverdu.github.io/pipedog/docs/audio/krisp-viva-filter/ {/* Generated by scripts/generate-providers.ts from data/providers.ts. */} Krisp VIVA is deferred: it is possible, but it isn't ready to port. **Why.** The Krisp VIVA SDK is proprietary. It isn't on public npm, and it needs a license and the model files from Krisp. **What would unblock it.** Bring your own SDK: core already has the `KrispAudioSdk` seam (`audio/krisp_instance`), and `setKrispAudioSdk()` takes the SDK you obtained from Krisp. An optional package that wires it up is planned. Sources: [Krisp VIVA in Pipecat](https://docs.pipecat.ai/pipecat/features/krisp-viva). ## What it is [#what-it-is] Krisp's voice isolation filter, VAD analyzer and turn analyzer. Upstream, it's these classes: `KrispVivaFilter`, `KrispVivaVadAnalyzer`, `KrispVivaTurn`, in [`audio/filters/krisp_viva_filter.py`](https://github.com/pipecat-ai/pipecat/blob/main/src/pipecat/audio/filters/krisp_viva_filter.py) and [`audio/vad/krisp_viva_vad.py`](https://github.com/pipecat-ai/pipecat/blob/main/src/pipecat/audio/vad/krisp_viva_vad.py) and [`audio/turn/krisp_viva_turn.py`](https://github.com/pipecat-ai/pipecat/blob/main/src/pipecat/audio/turn/krisp_viva_turn.py). Its pipedog package will be `@pipedog/audio-krisp-viva-filter`. * [Pipecat's Krisp VIVA page](https://docs.pipecat.ai/api-reference/server/services/audio-filters/krisp-viva-filter) ## Get it ported [#get-it-ported] * [Search the issues](https://github.com/polvallverdu/pipedog/issues?q=is%3Aissue+Krisp+VIVA) for a request, and upvote it with a reaction. * If there's none, [open a port request](https://github.com/polvallverdu/pipedog/issues/new?template=feature_request.yml\&title=Port+Krisp+VIVA+%28KrispVivaFilter%2C+KrispVivaVadAnalyzer%2C+KrispVivaTurn%29\&kind=Port+an+upstream+module\&upstream=src%2Fpipecat%2Faudio%2Ffilters%2Fkrisp_viva_filter.py%2C+src%2Fpipecat%2Faudio%2Fvad%2Fkrisp_viva_vad.py%2C+src%2Fpipecat%2Faudio%2Fturn%2Fkrisp_viva_turn.py). * Or port it yourself: the [contributing guide](https://polvallverdu.github.io/pipedog/docs/contributing.md) explains how modules are ported, test by test. The [porting status](https://polvallverdu.github.io/pipedog/docs/contributing/status.md) lists what's done. Until then, see what is ported in [Audio filters & mixers](https://polvallverdu.github.io/pipedog/docs/audio.md). # RNNoise filter > Suppress background noise in the user's audio with RNNoise, compiled to WebAssembly so it runs on Bun, Node, Deno and Cloudflare Workers. Source: https://polvallverdu.github.io/pipedog/docs/audio/rnnoise-filter/ ## Overview [#overview] `RNNoiseFilter` removes background noise from the user's audio before it reaches VAD and speech-to-text, using [RNNoise](https://gitlab.xiph.org/xiph/rnnoise), a small recurrent neural network. It is a port of Pipecat's `RNNoiseFilter`. Upstream calls the native `pyrnnoise` library; pipedog runs RNNoise 0.2 as **WebAssembly**, shipped inside the package, so there is nothing to install and it works on Cloudflare Workers too. Pipecat's documentation for the same filter. `rnnoise_filter.ts`, `rnnoise.ts` and `engine.ts`. ## Installation [#installation] ```bash bun add @pipedog/audio-rnnoise-filter ``` ```ts import { RNNoiseFilter } from "@pipedog/audio-rnnoise-filter"; ``` ## Runtimes [#runtimes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------------------------------------------------------------------------------------------------------------------------------------ | | Bun | ✅ yes | Nothing to install: the WebAssembly module ships in the package. | | Node.js | ✅ yes | Nothing to install: the WebAssembly module ships in the package. | | Deno | ✅ yes | Nothing to install: the WebAssembly module ships in the package. | | Cloudflare Workers | ✅ yes | Nothing to install: the WebAssembly module ships in the package. The bundler must import .wasm files as modules (wrangler's default rule). | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/audio-rnnoise-filter/package.json`: ✅ means the compat smoke test passes on that runtime. ## Usage [#usage] Set the filter as the transport's `audioInFilter`. It starts and stops with the transport. ```ts import { RNNoiseFilter } from "@pipedog/audio-rnnoise-filter"; import { TransportParams } from "@pipedog/core/transports/base_transport"; const params = new TransportParams({ audioInEnabled: true, audioInFilter: new RNNoiseFilter(), }); ``` RNNoise works on 480-sample frames (10 ms) at 48 kHz. The filter buffers incoming audio into frames and, when the transport is not at 48 kHz, resamples in and out with the soxr-style stream resampler (`@pipedog/resampler-wasm`). A call returns the frames that completed, so the first calls can be empty, and the output lags the input by less than one frame. Send a `FilterEnableFrame` to switch the filter on and off while the bot runs: ```ts import { FilterEnableFrame } from "@pipedog/core/frames/frames"; await worker.queueFrame(new FilterEnableFrame({ enable: false })); ``` ## Configuration [#configuration] `new RNNoiseFilter(options?)`: | Option | Type | Default | Description | | ------------------ | --------------------------------------- | ---------------------- | ------------------------------------------------------------------------------------ | | `resamplerQuality` | `"VHQ" \| "HQ" \| "MQ" \| "LQ" \| "QQ"` | `"QQ"` | Resampler preset when the transport is not at 48 kHz. `"QQ"` has the lowest latency. | | `rnnoise` | `({ sampleRate }) => RNNoiseLike` | the bundled WASM build | Create your own denoiser, for example a fake in tests. | Lower-level pieces are exported too: `initRnnoise(source?)` loads the module from bytes, a `WebAssembly.Module`, a URL or a `Response` (on Workers only a module), `RNNoise` mirrors pyrnnoise's `denoise_chunk`, and `RnnoiseEngine.createState()` gives the raw 480-sample `processFrame`. ## Cloudflare Workers [#cloudflare-workers] workerd refuses to compile WebAssembly from bytes, so on Workers the package's `"workerd"` export condition imports `rnnoise.wasm` as a pre-compiled module. Wrangler's default rule for `.wasm` files does this; nothing else is needed. The module is 1.4 MB. ## Differences from upstream [#differences-from-upstream] * **WebAssembly instead of `pyrnnoise`.** The model is RNNoise 0.2, so the denoising is the same kind of output, but it is not bit-identical. The module is the one built by [`@jitsi/rnnoise-wasm`](https://github.com/jitsi/rnnoise-wasm) (Apache-2.0). * The `rnnoise` option and a protected `_createResampler()` hook replace the module-level patching upstream's tests do. See the [package README](https://github.com/polvallverdu/pipedog/tree/main/packages/audio-rnnoise-filter) for the licenses of what it bundles. # Soundfile mixer > Mix WAV background sound into the bot's output audio with SoundfileMixer, in pure TypeScript on every runtime. Source: https://polvallverdu.github.io/pipedog/docs/audio/soundfile-mixer/ ## Overview [#overview] `SoundfileMixer` blends sound from files (office noise, hold music, a keyboard clatter) into the audio the transport sends to the user. It is a port of Pipecat's `SoundfileMixer`. Upstream reads the files with libsndfile; pipedog parses **WAV** in pure TypeScript, so the mixer runs on Bun, Node, Deno and Cloudflare Workers with nothing to install. Pipecat's documentation for the same mixer. `soundfile_mixer.ts` and the WAV reader `wav.ts`. ## Installation [#installation] ```bash bun add @pipedog/audio-soundfile-mixer ``` ```ts import { SoundfileMixer } from "@pipedog/audio-soundfile-mixer"; ``` ## Runtimes [#runtimes] | Runtime | Works | What it needs | | ------------------ | ----- | -------------------------------------------------------------------------------------------------------------- | | Bun | ✅ yes | Nothing to install. WAV files only. | | Node.js | ✅ yes | Nothing to install. WAV files only. | | Deno | ✅ yes | Nothing to install. WAV files only. | | Cloudflare Workers | ✅ yes | Nothing to install. WAV files only. Pass sounds as bytes or a URL: there is no filesystem to read a path from. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/audio-soundfile-mixer/package.json`: ✅ means the compat smoke test passes on that runtime. ## Usage [#usage] Give each sound a name, pick the default, and set the mixer as the transport's `audioOutMixer`: ```ts import { SoundfileMixer } from "@pipedog/audio-soundfile-mixer"; import { TransportParams } from "@pipedog/core/transports/base_transport"; const mixer = new SoundfileMixer({ soundFiles: { office: "./assets/office.wav", // a path (Bun, Node, Deno) hold: "https://example.com/hold.wav", // a URL // bytes work everywhere, Workers included: office: new Uint8Array(await bucket.get(...).arrayBuffer()) }, defaultSound: "office", volume: 0.4, }); const params = new TransportParams({ audioOutEnabled: true, audioOutMixer: mixer }); ``` The files must be mono and have the output transport's sample rate (for example 16000 Hz). A file with another rate, or one that can't be read, is skipped with a log message, and audio passes through unchanged while its sound is missing. Change the sound, volume or looping while the bot runs, and turn mixing on and off, with frames: ```ts import { MixerEnableFrame, MixerUpdateSettingsFrame } from "@pipedog/core/frames/frames"; await worker.queueFrame(new MixerUpdateSettingsFrame({ settings: { sound: "hold", volume: 0.2, loop: false } })); await worker.queueFrame(new MixerEnableFrame({ enable: false })); ``` ## Configuration [#configuration] `new SoundfileMixer(options)`: | Option | Type | Default | Description | | -------------- | ----------------------------- | ------------------ | -------------------------------------------------------------------------------------------------- | | `soundFiles` | `Record` | required | Sound name to source: WAV bytes, a `URL`, an `http(s):`, `file:` or `data:` URL string, or a path. | | `defaultSound` | `string` | required | The sound to play first. | | `volume` | `number` | `0.4` | Mixing volume, 0.0 to 1.0. | | `mixing` | `boolean` | `true` | Whether mixing starts enabled. | | `loop` | `boolean` | `true` | Start the sound again when it ends. With `false`, audio passes through once it has played. | | `fileSystem` | `{ readFile(path) }` | `node:fs/promises` | How paths are read. Imported lazily, so Workers can use the mixer with bytes or URLs. | ## Supported files [#supported-files] WAV only: PCM 8, 16, 24 and 32 bit, IEEE float 32 and 64 bit, A-law and mu-law, including `WAVE_FORMAT_EXTENSIBLE` and RF64 headers. Samples are converted to 16-bit the way libsndfile does. FLAC, Ogg Vorbis and Opus, MP3, AIFF and the other formats libsndfile reads are rejected with an error that names the format. Convert them first: ```bash ffmpeg -i office.mp3 -ar 16000 -ac 1 office.wav ``` ## Differences from upstream [#differences-from-upstream] * **WAV only** (above), where upstream accepts every libsndfile format. * **Sources can be bytes or URLs**, not only paths, and files are read with `await`, not `asyncio.to_thread`. * A multi-channel file is accepted with a warning, and its interleaved samples play back as one channel, as upstream's flat view of the array does. # TEN VAD > Detect the user's voice with TEN VAD's WebAssembly build, which you obtain from TEN and pass in, as an opt-in alternative to Silero. Source: https://polvallverdu.github.io/pipedog/docs/audio/ten-vad/ ## Overview [#overview] `TenVADAnalyzer` is a VAD analyzer backed by [TEN VAD](https://github.com/TEN-framework/ten-vad), a small voice activity detector from the TEN framework. It goes wherever a VAD analyzer goes (the user aggregator's `vadAnalyzer`, `VADProcessor`), in place of [Silero](https://polvallverdu.github.io/pipedog/docs/concepts/turn-detection.md). This is pipedog only: upstream Pipecat has no TEN VAD. TEN VAD is licensed under Apache-2.0 **with additional conditions** (among them a non-compete clause regarding Agora, and that derivative works of TEN VAD stay under TEN's license). pipedog is BSD-2-Clause, so `@pipedog/audio-ten-vad` contains **none** of TEN's code, JavaScript glue, WebAssembly or model: it only calls TEN's documented WebAssembly interface. You download TEN's prebuilt files yourself and pass them in, and TEN's conditions apply to your use of them. [Read TEN VAD's license](https://github.com/TEN-framework/ten-vad/blob/main/LICENSE) before you ship, and consider crediting it in your app ("Powered by ten-vad"). `ten_vad.ts`. TEN's repository, its WebAssembly build and its license. ## Installation [#installation] ```bash bun add @pipedog/audio-ten-vad ``` ```ts import { loadTenVadModule, TenVADAnalyzer } from "@pipedog/audio-ten-vad"; ``` Then get TEN's WebAssembly build from [TEN's repository](https://github.com/TEN-framework/ten-vad): `lib/Web/ten_vad.js` (the Emscripten loader, whose default export is `createVADModule`) and `lib/Web/ten_vad.wasm`. Copy both into your app, for example as `ten-vad/`. ## Runtimes [#runtimes] | Runtime | Works | What it needs | | ------------------ | -------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Bun | ✅ yes | TEN VAD's WebAssembly build (ten_vad.js and ten_vad.wasm from TEN's lib/Web), obtained from TEN under TEN's license and passed in at runtime. pipedog ships none of it. | | Node.js | ✅ yes | TEN VAD's WebAssembly build (ten_vad.js and ten_vad.wasm from TEN's lib/Web), obtained from TEN under TEN's license and passed in at runtime. pipedog ships none of it. | | Deno | — not verified | TEN VAD's WebAssembly build (ten_vad.js and ten_vad.wasm from TEN's lib/Web), obtained from TEN under TEN's license and passed in at runtime. pipedog ships none of it. | | Cloudflare Workers | ✅ yes | TEN VAD's WebAssembly build (ten_vad.js and ten_vad.wasm from TEN's lib/Web), obtained from TEN under TEN's license and passed in at runtime. pipedog ships none of it. Import ten_vad.wasm as a module and pass it as wasmModule (Workers can't compile WebAssembly from bytes). | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/audio-ten-vad/package.json`: ✅ means the compat smoke test passes on that runtime. The table covers pipedog's analyzer, which the compat test runs against a stand-in module. Whether TEN's `ten_vad.js` loads on a runtime is up to TEN's build (it targets browsers and web workers). ## Usage [#usage] `loadTenVadModule()` calls TEN's factory with the Emscripten options for where the `.wasm` comes from. Load it once per process and pass the promise to every analyzer: each one creates its own TEN VAD instance in the shared module. On **Bun, Node and Deno**, read the `.wasm` bytes: ```ts title="ten_vad.ts" // import createVADModule from "./ten-vad/ten_vad.js"; // TEN's lib/Web/ten_vad.js // Load TEN's module once and share it: each analyzer creates its own TEN VAD instance in it. const tenVad = loadTenVadModule({ factory: createVADModule, wasmBinary: await readFile("./ten-vad/ten_vad.wasm"), }); const vad = new TenVADAnalyzer({ module: tenVad, sampleRate: 16000 }); ``` In **browsers**, TEN's loader can fetch `ten_vad.wasm` from next to `ten_vad.js`: ```ts // import createVADModule from "/ten-vad/ten_vad.js"; // TEN's glue fetches ten_vad.wasm from next to ten_vad.js (or pass `wasmBinary`). const vad = new TenVADAnalyzer({ module: loadTenVadModule({ factory: createVADModule }) }); ``` On **Cloudflare Workers**, which can't compile WebAssembly from bytes, import the `.wasm` file as a module (Wrangler's default rule for `.wasm` gives a `WebAssembly.Module`) and pass it as `wasmModule`. It goes through Emscripten's `instantiateWasm` hook: ```ts // import createVADModule from "./ten-vad/ten_vad.js"; // import tenVadWasm from "./ten-vad/ten_vad.wasm"; // Wrangler gives a WebAssembly.Module const tenVad = loadTenVadModule({ factory: createVADModule, wasmModule: tenVadWasm }); const vad = new TenVADAnalyzer({ module: tenVad }); ``` Then use the analyzer like Silero: ```ts const context = new LLMContext(); const vad = new TenVADAnalyzer({ module: tenVad, hopSize: 160, // 10 ms frames (default: 256, 16 ms) // TEN's speech probability is the confidence: VADParams decides, as for Silero. params: new VADParams({ confidence: 0.7, startSecs: 0.2, stopSecs: 0.2 }), }); const aggregators = new LLMContextAggregatorPair(context, { userParams: new LLMUserAggregatorParams({ vadAnalyzer: vad }), }); ``` TEN VAD is created once the module has loaded: `await vad.ready()` waits for it, otherwise the first `analyzeAudio` does. After that every frame is scored synchronously. `cleanup()` destroys the instance and frees its memory. ### 16 kHz only [#16-khz-only] TEN VAD takes 16 kHz mono 16-bit audio, so the analyzer throws for any other sample rate. Run the transport's input at 16 kHz (`audioInSampleRate: 16000`), or resample before the VAD. ## Configuration [#configuration] `new TenVADAnalyzer(options)`: | Option | Type | Default | Description | | ------------ | --------------------------------------- | ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `module` | `TenVADModule \| Promise` | required | TEN VAD's WebAssembly module, from `loadTenVadModule()`. | | `hopSize` | `160 \| 256` | `256` | Samples per frame: 256 (16 ms) or 160 (10 ms), the sizes TEN VAD is optimized for. | | `threshold` | `number` | `0.5` | TEN's own speech-flag threshold. The analyzer uses TEN's speech **probability** as the confidence, so `params.confidence` decides what counts as speech; this only sets the flag TEN computes alongside. | | `sampleRate` | `number` | set later | Must be 16000. | | `params` | `VADParams` | `new VADParams()` | Confidence, start and stop times, and the volume gate, as for every VAD analyzer. | `loadTenVadModule(options)`: | Option | Type | Description | | ----------------- | --------------------------- | -------------------------------------------------------- | | `factory` | `TenVADModuleFactory` | The default export of TEN's `ten_vad.js`. | | `wasmBinary` | `ArrayBuffer \| Uint8Array` | The `.wasm` bytes. | | `wasmModule` | `WebAssembly.Module` | The `.wasm` compiled ahead of time (Cloudflare Workers). | | `moduleOverrides` | `Record` | Further Emscripten options, such as `locateFile`. | Pass `wasmBinary` or `wasmModule`, or neither to let TEN's loader find the file. ## TEN VAD or Silero [#ten-vad-or-silero] Both are small neural VADs; which one works better depends on your audio. In short: | | TEN VAD | Silero | | -------------------------------- | ---------------------------------------------------------- | ------------------------------------------------------ | | Package | `@pipedog/audio-ten-vad` | `@pipedog/audio-silero` | | License of the model and runtime | Apache-2.0 with TEN's additional conditions | MIT | | Files | you obtain and pass TEN's `.js` and `.wasm` (about 300 KB) | the model ships with the package (2.3 MB) | | Runtime | WebAssembly | ONNX Runtime (`onnxruntime-node` or `onnxruntime-web`) | | Sample rates | 16 kHz | 8 and 16 kHz | | Frame | 10 or 16 ms | 32 ms | TEN reports lower per-frame cost and faster speech-end detection than Silero in its own benchmarks. Evaluate both on recordings of your own users (line quality, noise, languages) before switching, and tune `VADParams` for the one you pick. # CLI > Scaffold a TypeScript voice bot with pipedog init, set up your coding agent with the docs MCP server, and run evals, with the pipedog command. Source: https://polvallverdu.github.io/pipedog/docs/cli/ `@pipedog/cli` is the port of Pipecat's `pipecat` command line. Its `pipedog init` writes a new bot project: a `bot.ts` on the [development runner](https://polvallverdu.github.io/pipedog/docs/runner.md) with the services you pick, a `package.json` with the right packages, a client, a README, and the guide files and MCP server config that set up your coding agent. `pipedog eval` runs [evals](https://polvallverdu.github.io/pipedog/docs/evals.md). Pipecat's CLI source: `pipecat init` and `pipecat eval`. `packages/cli/src`: `init.ts`, `scaffold.ts`, `project.ts`, the registry in `service_metadata.ts`. ## Create a bot [#create-a-bot] Run it without installing anything: ```bash bunx @pipedog/cli init ``` ```bash npx @pipedog/cli init ``` ```bash pnpm dlx @pipedog/cli init ``` On a terminal, `init` asks for a directory and how you want to build: **with a coding agent** (it writes the guide files and registers the docs MCP server, then you open your agent there) or **scaffold a runnable bot now** (the wizard asks for the bot type, transport, pipeline mode, services and features). Then: ```bash cd my-bot/server bun install cp .env.example .env # add your API keys bun bot.ts # open http://localhost:7860 and click Connect ``` The fastest start is the canned quickstart: SmallWebRTC, Deepgram, OpenAI and Cartesia, in `./pipedog-quickstart`. ```bash bunx @pipedog/cli init quickstart ``` ### From flags [#from-flags] Any scaffold flag turns the wizard off, so scripts and coding agents never wait on a prompt. The directory is `.` or a path (created if missing), and the project name defaults to its name: ```bash bunx @pipedog/cli init my-bot \ --transport smallwebrtc --mode cascade \ --stt deepgram_stt --llm openai_llm --tts cartesia_tts \ --eval ``` The bot type is inferred from the transports: a phone transport makes it a telephony bot, anything else a web bot. Repeat `--transport` for several at once (`-t twilio -t smallwebrtc` is a phone bot you can also try from a browser). The values of every flag are in `pipedog init --list-options`, as JSON, and `--dry-run` prints the config the flags resolve to, without writing anything. A mistake lists every problem at once, before any file is written. | Flag | | | ---------------------------------- | ----------------------------------------------------------------------------------------------------------- | | `-n, --name` | The project name (default: the directory's). | | `-b, --bot-type` | `web` or `telephony` (default: inferred from `--transport`). | | `-t, --transport` | `smallwebrtc`, `websocket`, `livekit`, `twilio`, `telnyx`, `plivo` or `exotel`. Repeatable. | | `-m, --mode` | `cascade` (STT, LLM, TTS) or `realtime` (speech-to-speech). | | `--stt`, `--llm`, `--tts` | The services of a cascade bot, such as `deepgram_stt`. | | `--realtime` | The service of a realtime bot, such as `openai_realtime`. | | `--video` | A video avatar service (`heygen_video`). Web bots only. | | `--client-framework` | `vanilla` for a Vite client in `client/`, or `none` (the default: the runner serves a client at `/client`). | | `--runtime` | `bun` (the default) or `node`. | | `--recording`, `--transcription` | Save the call's audio as WAV files; log the transcript. | | `--video-input`, `--video-output` | Camera in, video out (web bots). | | `--observability` | [OpenTelemetry tracing](https://polvallverdu.github.io/pipedog/docs/concepts/metrics-and-tracing.md), on when `OTEL_EXPORTER_OTLP_ENDPOINT` is set. | | `--eval` | An `eval` transport and a starter scenario, so `pipedog eval` can drive the bot. | | `--dockerfile` | A Dockerfile for Bun or Node. | | `-c, --config` | A JSON file with the same settings (below). Flags win over the file. | | `--mcp / --no-mcp`, `--mcp-client` | Register the docs MCP server, on by default; for one client only (repeatable). | | `--overwrite-guide` | Replace `AGENTS.md`, `CLAUDE.md`, `GETTING_STARTED.md` and a differing MCP entry. | Every toggle has a `--no-` form, and the last of `--recording --no-recording` wins. The config file takes upstream's keys (`project_name`, `bot_type`, `transports`, `mode`, `stt_service`, `llm_service`, `tts_service`, `realtime_service`, ...); `--dry-run` prints them, so a file can be made from one. ## What it writes [#what-it-writes] ```text my-bot/ ├── server/ │ ├── bot.ts # the bot: services, pipeline, event handlers; main({ bot }) │ ├── package.json # @pipedog/* packages at the CLI's release, and what they need │ ├── tsconfig.json │ ├── .env.example # the keys your services read │ ├── evals/ # starter_text.yaml, with --eval (cascade bots) │ └── Dockerfile # with --dockerfile ├── client/ # a Vite client, with --client-framework vanilla ├── AGENTS.md, CLAUDE.md # a guide for coding agents ├── .mcp.json, .cursor/mcp.json, .vscode/mcp.json ├── README.md # how to run it, and the phone setup of your provider └── .gitignore ``` `bot.ts` is a bot written the way [the runner's docs](https://polvallverdu.github.io/pipedog/docs/runner/#a-bot) show: it exports `bot(runnerArgs)`, builds its transport with `createTransport()`, and starts the runner with `main({ bot })` when it is the entry file (on Bun, Deno and Node alike). Its services come from the registry, with the keys read from the environment, and it refuses to start without the variables its services need, naming the missing ones. Run it with `bun bot.ts`, or `node bot.ts` (Node 22.18 or later runs the TypeScript itself) with `--runtime node`; the runner's flags (`-t twilio --proxy my.ngrok.io`, `--port`) work as in its docs. The project is a TypeScript project to keep working in: `npm run typecheck` is `tsc --noEmit` against the real packages. ### The services [#the-services] The list isn't maintained by hand against the docs: a test compares it with the provider catalog the [Services](https://polvallverdu.github.io/pipedog/docs/services.md) pages are generated from, so every ported speech, LLM, realtime and video service is either offered or explained, and no service that isn't ported is offered. `pipedog init --list-options` prints what your version offers: the web transports (SmallWebRTC, WebSocket, LiveKit), the phone transports (Twilio, Telnyx, Plivo, Exotel), speech-to-text, LLMs (including every OpenAI-compatible provider), text-to-speech, realtime services and HeyGen. What isn't offered: Daily, Pipecat's services that aren't ported yet, and the HTTP variants and processors that have no one-line setup (see the [Differences](#differences-from-pipecat)). ## Coding agents [#coding-agents] `init` writes `AGENTS.md` (which most coding agents read) and a `CLAUDE.md` that imports it. The guide tells the agent to scaffold with flags rather than by hand, how a pipedog bot is built, where to find current facts, and how to verify its work with evals. It points at two sources: the [`llms.txt`](https://polvallverdu.github.io/pipedog/llms.txt) index of these docs, and [`@pipedog/mcp`](https://polvallverdu.github.io/pipedog/docs/guides/llms/#mcp-server), the local docs server. A bare `init` also writes `GETTING_STARTED.md`, for you: how to write the first prompt. `init` registers the MCP server in the project, in the config format of each client from [Docs for AI tools](https://polvallverdu.github.io/pipedog/docs/guides/llms/#mcp-server): | Client | File | | ----------- | ------------------ | | Claude Code | `.mcp.json` | | Cursor | `.cursor/mcp.json` | | VS Code | `.vscode/mcp.json` | It adds the stanza to the file when there is one, leaves the other servers alone, and does nothing when it is already there. A file it can't read as plain JSON (VS Code allows comments) is left untouched, and `init` prints the stanza to paste. Clients load MCP servers when a session starts, so open yours in the project after `init`. To pin the server to your pipedog release, change `@pipedog/mcp` to `@pipedog/mcp@`. Existing guide files are kept, unless you pass `--overwrite-guide`. Each carries a footer with the CLI's version, so a later `init` can tell you the guide is out of date (and, on a terminal, offer to refresh it). ## Evals [#evals] `pipedog eval` is [`pipedog-eval`](https://polvallverdu.github.io/pipedog/docs/evals.md): `pipedog eval run`, `judge` and `suite` are the same code, which lives in `@pipedog/evals`, loaded when you run the command. It's an optional peer dependency: a project created with `--eval` has it, and `bunx pipedog eval` runs from there. Without it the command says how to install it. ```bash bun bot.ts # one terminal bunx pipedog eval run evals/starter_text.yaml -v # another ``` ## Differences from Pipecat [#differences-from-pipecat] * **A TypeScript project.** The generated bot is `bot.ts` on `@pipedog/runner`, with `bun` or `node` instead of `uv`; `.env.example` and the README are written for it. * **Only what's ported is offered.** There is no Daily, so no Daily PSTN or Twilio + Daily SIP, and none of Pipecat's services that aren't ported (NVIDIA, Simli, fal, ...). Their values are errors that list the valid ones. Services whose HTTP variants, base classes or processors have no one-line setup (`OpenAILiveLLMService`, `LangchainProcessor`, ...) are left out on purpose, and the registry's `NOT_OFFERED` says why. * **No Pipecat Cloud.** `--deploy-to-cloud` and `--enable-krisp` are errors that say so, and there is no `pcc-deploy.toml`; `--dockerfile` writes a plain Bun or Node image. Deployment is up to you: [Servers](https://polvallverdu.github.io/pipedog/docs/deployment/servers.md), [Vercel](https://polvallverdu.github.io/pipedog/docs/deployment/vercel.md). * **Cloudflare Workers has no template.** A voice agent on Workers needs a Durable Object, a `wrangler` config and provider-side turn detection: start from [`examples/workers-durable-voice-agent`](https://github.com/polvallverdu/pipedog/tree/main/examples/workers-durable-voice-agent) and [Cloudflare Workers](https://polvallverdu.github.io/pipedog/docs/deployment/cloudflare-workers.md). * **One client.** `--client-framework vanilla` is Pipecat's vanilla JS client; the React clients (Vite and Next.js, with the Pipecat UI components) aren't ported. The runner serves a React client at `/client` for WebRTC, and the Pipecat browser SDKs work unchanged. * **The docs MCP server instead of the Context Hub.** There is no index to build: `@pipedog/mcp` bundles the docs of its release, and `init` only writes the config. * **The plugin system isn't ported** (`pipecat cloud` and other extensions), nor the `pc` alias or a `create-pipedog` command: that would be its own npm package. * **Prompts** are [`@clack/prompts`](https://www.npmjs.com/package/@clack/prompts). `init` never prompts without a terminal. The CLI itself runs on Bun, Node 22 or later and Deno. # Frame processors > Write a custom FrameProcessor - handle frames, push them on, pick a direction, set up resources and listen to events. Source: https://polvallverdu.github.io/pipedog/docs/concepts/frame-processors/ Frame processors are the workers of a pipeline. Each has a specific job (transcribe audio, generate text, play audio, log metrics...). They: * **receive frames** from the previous processor, * **process the data**, * **create new frames** with their output, * **pass frames along** to the next processor. pipedog ports Pipecat's built-in processors (aggregators, filters, observers, and services as their packages land). Most applications also need some custom frame processing logic; this page shows how to write it. ## Example: MetricsFrame logger [#example-metricsframe-logger] This custom processor formats and logs every `MetricsFrame`: ```ts function formatMetrics(metrics: MetricsData[]): string { return metrics .map((metric) => { const fields = Object.entries(metric) .map(([key, value]) => `\t${key}=${JSON.stringify(value)}`) .join("\n"); return `${metric.constructor.name}\n${fields}`; }) .join("\n"); } /** Formats and logs every MetricsFrame. */ export class MetricsFrameLogger extends FrameProcessor { override async processFrame(frame: Frame, direction: FrameDirection): Promise { await super.processFrame(frame, direction); if (frame instanceof MetricsFrame) { logger.info(`${frame.name}\n${formatMetrics(frame.data)}`); } // SUPER IMPORTANT: always push every frame! await this.pushFrame(frame, direction); } } ``` Add it to a pipeline wherever it should see the frames, for example at the very end so it receives every `MetricsFrame` the services produce: ```ts const pipeline = new Pipeline([ transportInput, stt, userAggregator, llm, tts, transportOutput, assistantAggregator, new MetricsFrameLogger(), // our custom processor ]); ``` ## Key requirements [#key-requirements] Custom processors extend `FrameProcessor` from `@pipedog/core/processors/frame_processor`. The base class handles `StartFrame`, `CancelFrame`, `InterruptionFrame` and pause / resume frames for you, and provides `processFrame()` and `pushFrame()`: * **`processFrame(frame, direction)`** receives every frame and is where your conditional logic goes. * **`pushFrame(frame, direction?)`** sends a frame on. Normally downstream, but you can push upstream, or both ways. Two calls are essential: 1. **`super(options)`** in the constructor, passing the whole options object (that is how Python's `**kwargs` flows to the base class). 2. **`await super.processFrame(frame, direction)`** first thing in `processFrame()`. ```ts export interface MyCustomProcessorOptions extends FrameProcessorOptions { greeting?: string; } export class MyCustomProcessor extends FrameProcessor { protected _greeting: string; constructor(options: MyCustomProcessorOptions = {}) { super(options); // ✅ Required: forwards name, enableDirectMode, metrics... this._greeting = options.greeting ?? "hello"; } override async processFrame(frame: Frame, direction: FrameDirection): Promise { await super.processFrame(frame, direction); // ✅ Required if (frame instanceof TranscriptionFrame) { console.log(`${this._greeting}, user ${frame.userId} said: ${frame.text}`); } await this.pushFrame(frame, direction); // ✅ Required: pass the frame through } } ``` Constructor options extend `FrameProcessorOptions` (see the [FrameProcessor API](https://polvallverdu.github.io/pipedog/docs/api/frame-processor.md)): `name`, `enableDirectMode` (skip the internal queues and process frames right away) and `metrics`. ## Critical responsibility: frame forwarding [#critical-responsibility-frame-forwarding] Processors receive **all** frames pushed through the pipeline, which gives them a lot of power and a great responsibility: they must push every frame on. A processor that drops frames it doesn't care about blocks the pipeline, and your application stops working in confusing ways. Note the unconditional `await this.pushFrame(frame, direction)` at the end of every example here. ## Frame direction [#frame-direction] Frames flow in one of two directions: ```ts export const FrameDirection = { DOWNSTREAM: 1, UPSTREAM: 2 } as const; ``` **Downstream** (the default) goes from input to output: audio enters at the transport input, gets transcribed, runs through the LLM, turns into speech and reaches the transport output. **Upstream** sends information back towards the start: errors travel upstream, and the assistant context aggregator at the end pushes context frames upstream to the LLM. ```ts // Push downstream (the default) await this.pushFrame(frame, FrameDirection.DOWNSTREAM); // Push upstream, back towards the start of the pipeline await this.pushFrame(new TextFrame({ text: "hi" }), FrameDirection.UPSTREAM); // Create two instances, push one each way, linked by `broadcastSiblingId` await this.broadcastFrame(UserStartedSpeakingFrame); // Copy a frame you didn't create and push it both ways await this.broadcastFrameInstance(frame); ``` Prefer `broadcastFrame(FrameClass, options)` when you create the frame; use `broadcastFrameInstance(frame)` for a frame you received, which copies every field except `id` and `name`. ## Setup and cleanup [#setup-and-cleanup] A processor is set up before any frame reaches it, and cleaned up when the pipeline tears down. That is where a processor that needs a connection, a client, or the pipeline's audio configuration gets it: ```ts export class ConnectedProcessor extends FrameProcessor { protected _sampleRate = 0; protected _client: Client | null = null; override async setup(setup: FrameProcessorSetup): Promise { await super.setup(setup); // ✅ Required this._sampleRate = setup.audioOutSampleRate; this._client = await connect(); } override async cleanup(): Promise { await super.cleanup(); // ✅ Required await this._client?.close(); } } ``` `FrameProcessorSetup` carries `clock`, `taskManager`, `pipelineWorker`, `audioInSampleRate` (default 16000), `audioOutSampleRate` (default 24000), `enableMetrics`, `enableTracing`, `enableUsageMetrics`, `observer`, `reportOnlyInitialTtfb` and `tracingContext`. The processor also exposes `metricsEnabled`, `usageMetricsEnabled`, `reportOnlyInitialTtfb` and, once set up, `processorSetup`. Don't read this configuration off the `StartFrame`: upstream deprecated those `StartFrame` fields and pipedog doesn't have them. Processors are set up **concurrently**. A processor that throws while setting up pushes an `ErrorFrame` upstream, so the application learns its pipeline came up degraded; one that throws while being cleaned up is logged and doesn't stop the others from being released. ### Holding frames until a condition is met [#holding-frames-until-a-condition-is-met] A processor that connects during setup may receive frames before it is ready. `pauseProcessingAllFramesUntil(ready, { timeout })` holds everything arriving until `ready()` resolves, then delivers it in order. The pause takes hold from the frame *after* the one being processed, and is always lifted: when `ready` resolves, when `timeout` seconds (default 5) elapse, or at cleanup. ```ts export class SlowToConnect extends FrameProcessor { protected _ready = new Event(); override async setup(setup: FrameProcessorSetup): Promise { await super.setup(setup); // Hold every frame until connected, for at most 10 seconds. await this.pauseProcessingAllFramesUntil(() => this._ready.wait(), { timeout: 10 }); this.createTask(async () => { await connect(); this._ready.set(); }, "connect"); } } ``` ## Errors [#errors] Report errors with `pushError()`, which builds an `ErrorFrame`, classifies it, fires `on_error` and pushes it upstream. An error that can't be recovered from (a rejected API key, say) makes the processor unusable (`isUsable` becomes `false`) and the worker applies its `processorUnusablePolicy`. ```ts try { await riskyOperation(); } catch (e) { // A recoverable error: report it and keep going. await this.pushError("Failed to process audio chunk, skipping", { exception: e }); } // An error that leaves this processor unable to do any more work. await this.pushError("Critical operation failed", { forceTreatAsPermanent: true }); ``` Every broad `catch` in a processor should start with `rethrowIfCancelled(e)`; see [Interruptions & cancellation](https://polvallverdu.github.io/pipedog/docs/concepts/interruptions-and-cancellation/#rethrowifcancelled). ## Event handlers [#event-handlers] Every frame processor, including services and transport inputs/outputs, emits these events. Event names are the same strings as in Pipecat. Python's `@obj.event_handler("on_x")` decorator becomes `obj.eventHandler("on_x", handler)` (which returns the handler) or `obj.addEventHandler("on_x", handler)`. The first argument of every handler is the processor. | Event | Arguments | Description | | ------------------------- | -------------------------------- | -------------------------------------------------------------------------------- | | `on_error` | `(processor, error: ErrorFrame)` | An error occurred in this processor, before its `ErrorFrame` is pushed upstream. | | `on_usable_changed` | `(processor, isUsable: boolean)` | The processor stopped (or started again) being able to do its job. | | `on_before_process_frame` | `(processor, frame)` | A frame is about to be processed. | | `on_after_process_frame` | `(processor, frame)` | A frame has just been processed. | | `on_before_push_frame` | `(processor, frame)` | A frame is about to be pushed. | | `on_after_push_frame` | `(processor, frame)` | A frame has just been pushed. | ```ts const tts = new MyCustomProcessor({ name: "tts" }); // Like Python's `@tts.event_handler("on_error")` decorator. tts.eventHandler("on_error", async (processor: FrameProcessor, error: ErrorFrame) => { console.log(`Error in ${processor}: ${error.error}`); if (error.exception) { console.log("Exception:", error.exception); } if (error.processor && !error.processor.isUsable) { console.log("This processor can no longer do its job"); } }); tts.eventHandler("on_usable_changed", async (processor: FrameProcessor, isUsable: boolean) => { if (!isUsable) console.warn(`${processor} is down`); }); // The same handler on several processors. async function handleServiceError(processor: FrameProcessor, error: ErrorFrame): Promise { logger.error(`Service error in ${processor.name}: ${error.error}`); } const stt = new MyCustomProcessor({ name: "stt" }); const llm = new MyCustomProcessor({ name: "llm" }); for (const service of [stt, llm, tts]) { service.addEventHandler("on_error", handleServiceError); } // Frame hooks run inline: keep them fast. llm.eventHandler("on_before_push_frame", async (processor: FrameProcessor, frame: Frame) => { console.log(`${processor} is about to push: ${frame}`); }); ``` All of these except `on_usable_changed` are **synchronous**: handlers run inline and block the pipeline until they complete. Use them for fast work like logging, counting or setting flags, not for I/O. `on_usable_changed` runs in its own task. ## Example: response logger [#example-response-logger] Watching for frames that bracket a piece of work is a common pattern. `LLMFullResponseStartFrame` and `LLMFullResponseEndFrame` bracket every LLM response: ```ts export class ResponseLogger extends FrameProcessor { override async processFrame(frame: Frame, direction: FrameDirection): Promise { await super.processFrame(frame, direction); if (frame instanceof LLMFullResponseStartFrame) { logger.info("LLM response started"); } else if (frame instanceof LLMFullResponseEndFrame) { logger.info("LLM response finished"); } await this.pushFrame(frame, direction); } } ``` ## Testing a processor [#testing-a-processor] `runTest()` from `@pipedog/core/tests/utils` (the port of `pipecat.tests.utils.run_test`) runs a processor in a real pipeline, sends frames, and checks the frame types that come out: ```ts import { expect, test } from "bun:test"; import { type Frame, TextFrame } from "@pipedog/core/frames/frames"; import { type FrameDirection, FrameProcessor } from "@pipedog/core/processors/frame_processor"; import { runTest } from "@pipedog/core/tests/utils"; class UpperCaseProcessor extends FrameProcessor { override async processFrame(frame: Frame, direction: FrameDirection): Promise { await super.processFrame(frame, direction); if (frame instanceof TextFrame) { await this.pushFrame(new TextFrame({ text: frame.text.toUpperCase() }), direction); } else { await this.pushFrame(frame, direction); } } } test("test_upper_case", async () => { const [down] = await runTest(new UpperCaseProcessor(), { framesToSend: [new TextFrame({ text: "hello" })], expectedDownFrames: [TextFrame], }); expect((down[0] as TextFrame).text).toEqual("HELLO"); }); ``` ## Best practices [#best-practices] 1. **Always call the parent**: `super(options)` and `await super.processFrame(frame, direction)`. 2. **Forward all frames** with `await this.pushFrame(frame, direction)`. 3. **Handle frames conditionally** with `instanceof` checks. 4. **Handle errors properly**: `rethrowIfCancelled(e)` first in every `catch`, then `pushError()`. 5. **Position carefully**: put the processor where it receives the frames it needs. 6. **Track your tasks**: start background work with `this.createTask()`, never a floating promise. # Frames > Frame categories, processing order, interruption behavior and custom frames. Source: https://polvallverdu.github.io/pipedog/docs/concepts/frames/ Frames are the fundamental units of data in pipedog. Every piece of information that moves through a pipeline (audio, text, images, control signals) is wrapped in a frame. Frame processors receive frames, act on them, and push new or modified frames to the next processor. All frames extend the base `Frame` class from `@pipedog/core/frames/frames`, the port of Pipecat's `pipecat.frames.frames`. Where Pipecat frames are Python dataclasses, pipedog frames are classes whose constructor takes **one options object**: ```ts twoslash import { TextFrame, TranscriptionFrame } from "@pipedog/core/frames/frames"; const text = new TextFrame({ text: "hello" }); const transcription = new TranscriptionFrame({ text: "hi there", userId: "user-1", timestamp: new Date().toISOString(), }); ``` This is true even where upstream tests pass positional arguments: `TextFrame("hi")` is always `new TextFrame({ text: "hi" })`. ## Frame categories [#frame-categories] There are three base frame types, each with different processing behavior: | Base type | Processing | Interruption behavior | | -------------- | ------------------------------------------------------------ | -------------------------------------- | | `DataFrame` | Queued, processed in order with other non-system frames | Dropped on user interruption | | `ControlFrame` | Queued, processed in order with other non-system frames | Dropped on user interruption | | `SystemFrame` | Higher priority, processed in order with other system frames | **Never** dropped on user interruption | ### DataFrame [#dataframe] Data frames carry the main content flowing through a pipeline: audio chunks, text, images and LLM messages. They are queued and processed in order with other data and control frames. If the user interrupts (starts speaking while the bot is responding), pending data frames are discarded so the new input can be handled immediately. Examples: `TextFrame`, `OutputAudioRawFrame`, `LLMMessagesAppendFrame`, `TTSSpeakFrame`. ### ControlFrame [#controlframe] Control frames signal processing boundaries and configuration changes: response start/end markers, settings updates and state transitions. They are processed in order alongside data frames and, like them, are discarded on interruption unless their `interruptible` field is `false`. Examples: `EndFrame`, `LLMFullResponseStartFrame`, `TTSStartedFrame`. ### SystemFrame [#systemframe] System frames are high-priority signals that must always be delivered: interruptions, user input, errors and pipeline lifecycle. They are processed in order with other system frames, ahead of any queued data or control frame, and are never discarded by an interruption. Examples: `StartFrame`, `CancelFrame`, `InterruptionFrame`, `UserStartedSpeakingFrame`, `InputAudioRawFrame`, `ErrorFrame`, `MetricsFrame`. ```ts export function describe(f: Frame): string { if (f instanceof SystemFrame) return "system: high priority, never dropped by interruptions"; if (f.interruptible) return "data/control: queued in order, dropped by interruptions"; return "data/control: queued in order, survives interruptions"; } ``` ### Processing order [#processing-order] Frames are processed in a guaranteed order **within their lane**. Each processor has a priority input queue: the `StartFrame` first, then system frames, then data and control frames, each tier in arrival order. System frames are handled as they arrive; data and control frames go to a second queue processed one at a time, so the work for one finishes before the next starts. That is what makes this reliable: speak first, then end the pipeline once the goodbye is flushed. ```ts await worker.queueFrames([new TTSSpeakFrame({ text: "Goodbye!" }), new EndFrame()]); ``` ## Frame properties [#frame-properties] Every frame has these fields, set automatically (see the [Frames API](https://polvallverdu.github.io/pipedog/docs/api/frames.md)): | Field | Type | Description | | ---------------------- | ------------------------- | ------------------------------------------------------------------------- | | `id` | `number` | Unique identifier for the frame instance. | | `name` | `string` | Class name and instance count, e.g. `TextFrame#3`. Handy for debugging. | | `pts` | `number \| null` | Presentation timestamp in **nanoseconds**, for audio/video sync. | | `broadcastSiblingId` | `number \| null` | ID of the paired frame when broadcast in both directions. | | `metadata` | `Record` | Arbitrary frame metadata. | | `transportSource` | `string \| null` | Name of the transport source that created the frame. | | `transportDestination` | `string \| null` | Name of the transport destination (multi-track output). | | `interruptible` | `boolean` | Whether an interruption may drop or cancel this frame. `true` by default. | ## Interruptible frames [#interruptible-frames] Occasionally a data or control frame is too important to lose when the user interrupts. A frame class that must survive interruptions overrides the `interruptible` field. In Python that is `interruptible: bool = field(default=False, init=False)`; in TypeScript: ```ts /** Like AsteriskCommandFrame, but survives user interruptions. */ export class CriticalCommandFrame extends DataFrame { override interruptible = false; command: string; constructor(options: AsteriskCommandFrameOptions) { super(options); this.command = options.command; } } ``` You can also set it on a single frame before pushing it: ```ts const frame = new TextFrame({ text: "Important" }); frame.interruptible = false; // This specific frame survives interruption. ``` Uninterruptible frames include `EndFrame`, `StopFrame`, `EndWorkerFrame`, `StopWorkerFrame` and `FunctionCallResultFrame`: terminal frames must survive interruption so the pipeline still shuts down properly. The deprecated `UninterruptibleFrame` mixin is not ported; use the field. ## Media mixins [#media-mixins] Pipecat shares audio, image and file fields between frames through dataclass mixins. pipedog ports them as class-factory mixins (`defineMixin`), so `instanceof` keeps working: * `AudioRawFrame`: `audio` (`Uint8Array`, 16-bit little-endian PCM, never `Buffer`), `sampleRate`, `numChannels`, and the computed `numFrames`. * `ImageRawFrame`: `image` (`Uint8Array`), `size` (`[width, height]`), `format`. * `FileRawFrame`: `file`, `type` (`"bytes"` or `"url"`), `filename`, `format`. ```ts twoslash import { AudioRawFrame, InputAudioRawFrame } from "@pipedog/core/frames/frames"; const frame = new InputAudioRawFrame({ audio: new Uint8Array(320), sampleRate: 16000, numChannels: 1, }); console.log(frame instanceof AudioRawFrame, frame.numFrames); // true 160 ``` ## Creating a custom frame [#creating-a-custom-frame] Subclass one of the three base categories, picking it by how the frame should be handled: * `DataFrame` for ordered data such as text, audio or your own payload. Dropped by interruptions. * `ControlFrame` for ordered control signals that flush in line with the data. * `SystemFrame` for high-priority signals that skip the queue and are not affected by interruptions. ```ts import { DataFrame, type Frame, SystemFrame, TextFrame } from "@pipedog/core/frames/frames"; export interface AsteriskCommandFrameOptions { /** The command to send to the telephony transport. */ command: string; } /** A custom command sent downstream to a telephony transport. */ export class AsteriskCommandFrame extends DataFrame { command: string; constructor(options: AsteriskCommandFrameOptions) { super(options); this.command = options.command; } } ``` Name the options interface `Options` and pass the whole object to `super(options)`, as every built-in frame does. ## Common patterns [#common-patterns] pipedog, like Pipecat, prefers pushing frames over calling methods between processors. Routing data through the pipeline as frames keeps processing in order, which real-time use cases depend on. * **End a conversation**: push `EndWorkerFrame` **downstream**, so frames queued ahead of it (a goodbye `TTSSpeakFrame`, say) are flushed before the pipeline ends. * **Interrupt the bot yourself**: from a processor, `await this.broadcastInterruption()`; from code holding the worker, `await worker.queueFrame(new InterruptionWorkerFrame())`. * **React to LLM response boundaries**: watch for `LLMFullResponseStartFrame` and `LLMFullResponseEndFrame` in a custom processor (see [Frame processors](https://polvallverdu.github.io/pipedog/docs/concepts/frame-processors/#example-response-logger)). Patterns that need services (injecting prompts with `LLMMessagesAppendFrame`, updating tools with `LLMSetToolsFrame`, changing settings at runtime) depend on the service: each provider page says which of these frames it accepts, for example [OpenAI Realtime](https://polvallverdu.github.io/pipedog/docs/services/s2s/openai.md) and [Gemini Live](https://polvallverdu.github.io/pipedog/docs/services/s2s/gemini-live.md). The frames are in `@pipedog/core/frames/frames`. # Interruptions & cancellation > How the bot stops when the user speaks, and how @pipedog/asyncio gives JavaScript Python's asyncio cancellation. Source: https://polvallverdu.github.io/pipedog/docs/concepts/interruptions-and-cancellation/ Interruptions (barge-in) let the user talk over the bot: the bot stops right away, in-flight work is cancelled, and the pipeline is ready for the new input. Making that work in JavaScript needs Python's `asyncio` cancellation semantics, which is what `@pipedog/asyncio` provides. ## What happens on an interruption [#what-happens-on-an-interruption] When a user turn starts with interruptions enabled, the user aggregator **broadcasts an `InterruptionFrame`** upstream and downstream. Then: **Every processor cancels its current work.** `InterruptionFrame` is a system frame, so each processor handles it immediately instead of waiting behind queued frames. The processor cancels the task processing its current data/control frame and discards queued ones. Frames with `interruptible = false` (like `FunctionCallResultFrame` and `EndFrame`) are kept and still processed. **The LLM stops generating**: the in-flight completion is cancelled mid-stream. **TTS clears its buffers** and drops pending output. **The transport flushes unplayed audio**, so the bot goes silent within about one audio write. Only the words actually spoken end up in the LLM context: the output transport pushes `TTSTextFrame`s in sync with playback, and the assistant aggregator commits the spoken-so-far text when interrupted. ### Triggering an interruption yourself [#triggering-an-interruption-yourself] From inside a custom processor: ```ts await this.broadcastInterruption(); ``` From code holding the worker, queue an `InterruptionWorkerFrame`; the worker turns it into an `InterruptionFrame` sent through the whole pipeline: ```ts await worker.queueFrame(new InterruptionWorkerFrame()); ``` A processor sees the interruption in `processFrame()` (after `super.processFrame()` has already cancelled the work in progress), and can override `protected async _startInterruption()` (calling `super._startInterruption()` first) for cleanup that must happen at that moment. ### Turns with no transcript [#turns-with-no-transcript] If the user interrupts but nothing is transcribed (a cough, say), the bot has stopped and has nothing to answer. By default the LLM then runs once, so the bot asks the user to repeat or picks up where it left off; `emptyUserTurn` on `LLMUserAggregatorParams` changes that (see [Turns with no transcript](https://polvallverdu.github.io/pipedog/docs/concepts/turn-strategies/#turns-with-no-transcript)). ## Controlling interruptions [#controlling-interruptions] Interruptions are on by default. What starts one is the [start strategy](https://polvallverdu.github.io/pipedog/docs/concepts/turn-strategies.md) that opens the user's turn, and [mute strategies](https://polvallverdu.github.io/pipedog/docs/concepts/user-muting.md) decide whether the user is heard at all: | Goal | How | | -------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- | | Stop the bot as soon as the user speaks | the default (`VADUserTurnStartStrategy`) | | Never interrupt, but answer after the bot finishes | `enableInterruptions: false` on the start strategies | | Ignore short acknowledgements like "okay" | `MinWordsUserTurnStartStrategy` | | Tell interruptions from backchannels with a model | `KrispVivaIPUserTurnStartStrategy` (bring your own Krisp SDK) | | Discard what the user says while the bot talks | `AlwaysUserMuteStrategy` | | Protect only the greeting, or tool calls | `MuteUntilFirstBotCompleteUserMuteStrategy`, `FirstSpeechUserMuteStrategy`, `FunctionCallUserMuteStrategy` | | Keep a provider's turn detection from interrupting | `shouldInterrupt: false` on the STT service | To turn interruptions off, set `enableInterruptions: false` on every start strategy, since any of them can start a turn: ```ts const userParams = new LLMUserAggregatorParams({ vadAnalyzer: new SileroVADAnalyzer(), userTurnStrategies: new UserTurnStrategies({ start: [ new VADUserTurnStartStrategy({ enableInterruptions: false }), new TranscriptionUserTurnStartStrategy({ enableInterruptions: false }), ], }), }); ``` **Disabling interruptions is not muting.** With interruptions off the user's turn still starts and is transcribed: the bot finishes speaking and then answers. A muted user's audio and transcripts are dropped at the user aggregator, so nothing they say over the bot reaches the context or gets an answer. The user aggregator also drops `InterruptionFrame`s while the user is muted, so an interruption you queue on the worker doesn't get past it then; one broadcast by a processor after the aggregator still does. With an STT service that detects turns itself, set `shouldInterrupt: false` on the service rather than on the strategies: the service hands the setting to the strategies it recommends. Speech-to-speech services detect interruptions on their side, and pipedog follows their speaking events and still flushes the local audio, so the bot goes quiet at once. ## Why JavaScript needs help [#why-javascript-needs-help] Python delivers `CancelledError` **into** a coroutine, at the `await` it is suspended on. That is how an interruption stops a processor mid-work: its processing task is cancelled, and whatever it was awaiting raises. JavaScript has no such thing. A promise cannot be interrupted from outside, and `catch (e)` catches everything, including what Python would consider a cancellation. `@pipedog/asyncio` emulates asyncio on top of promises and `AsyncLocalStorage`, which Bun, Node, Deno and Cloudflare Workers (`nodejs_compat`) all provide: | Python | `@pipedog/asyncio` | | -------------------------------------------------------------- | ----------------------------------------------------- | | `asyncio.create_task(coro())` | `createTask(() => coro())` | | `task.cancel()` / `await task` | `task.cancel()` / `await task` (a `Task` is thenable) | | `asyncio.sleep(s)` | `sleep(s)` (seconds) | | `asyncio.wait_for(coro(), timeout=t)` | `waitFor(() => coro(), t)` | | `asyncio.gather(a(), b())` | `gather(() => a(), () => b())` | | `asyncio.Event`, `Queue`, `PriorityQueue`, `Lock`, `Semaphore` | same names | | `asyncio.CancelledError`, `asyncio.TimeoutError` | `CancelledError`, `TimeoutError` | | `asyncio.current_task()` | `currentTask()` | | `async with lock:` | `await lock.run(async () => { ... })` | ```ts import { abortable, CancelledError, createTask, currentSignal, Event, gather, Queue, rethrowIfCancelled, sleep, TimeoutError, waitFor, } from "@pipedog/asyncio"; const task = createTask(async () => { await sleep(10); // a cancellation point return "done"; }, "slow-task"); task.cancel(); // like asyncio: the pending sleep() rejects with CancelledError try { await task; } catch (e) { console.log(e instanceof CancelledError); // true } ``` ## Cancellation points [#cancellation-points] When a task is cancelled: * if it is **suspended on a cancellation point**, that await rejects with `CancelledError` right away (the request is consumed, as in asyncio); * otherwise the request stays pending, and the **next cancellation point** the task reaches throws it; * `task.signal`, an `AbortSignal`, is aborted. The cancellation points are the awaits on this package's primitives: `sleep()`, `Event.wait()`, `Queue.get()` / `Queue.put()`, `waitFor()`, `gather()`, `wait()`, `abortable()`, and awaiting a `Task`. Cancelling a task that awaits another task cancels the awaited one too, as in asyncio. ```ts try { // Pass a function, so it runs as a child task that can be cancelled on timeout. await waitFor(() => sleep(5), 0.1); } catch (e) { if (!(e instanceof TimeoutError)) throw e; } const [x, y] = await gather( async () => 1, async () => "two", ); ``` Always pass `waitFor()` a **function** rather than a promise, so it can run as a child task and be cancelled on timeout. Queues and events behave like their asyncio counterparts: ```ts const queue = new Queue(); const ready = new Event(); const consumer = createTask(async () => { await ready.wait(); for (;;) { const item = await queue.get(); // a cancellation point console.log("got", item); queue.taskDone(); } }); ready.set(); await queue.put("hello"); await queue.join(); consumer.cancel(); ``` ## rethrowIfCancelled [#rethrowifcancelled] Python's `except Exception:` does **not** catch `CancelledError` (it derives from `BaseException`). JavaScript's `catch (e)` does. A ported `except Exception:` that swallowed a cancellation would keep a cancelled task running, so **every broad `catch` starts with `rethrowIfCancelled(e)`**: ```ts async function callProvider(url: string): Promise { try { // fetch is not a cancellation point. Give it the current task's signal // so cancelling the task aborts the request too. const response = await fetch(url, { signal: currentSignal() }); return await response.json(); } catch (e) { rethrowIfCancelled(e); // Python's `except Exception` never catches CancelledError console.error("provider failed", e); return null; } } ``` Catching the cancellation on purpose (Python's `except asyncio.CancelledError:`) looks like `if (e instanceof CancelledError) { cleanup(); throw e; }`. ## Foreign awaits: fetch, WebSockets, SDKs [#foreign-awaits-fetch-websockets-sdks] `fetch`, a WebSocket, or a provider SDK is **not** a cancellation point: a task cancelled while awaiting one keeps waiting until it settles. Two tools fix that: * **Pass `currentSignal()`** (the current task's `AbortSignal`, or a never-aborting one outside tasks) to anything that accepts a signal, such as `fetch(url, { signal: currentSignal() })`, as in the sample above. Cancelling the task then aborts the request itself. * **Wrap the await in `abortable(promise)`** to make it a cancellation point. The foreign operation keeps running in the background unless it was also given the signal. ```ts async function withSdk(sdkCall: () => Promise): Promise { // An SDK that takes no AbortSignal: turn the await into a cancellation point. // The SDK call itself keeps running in the background if we are cancelled. return await abortable(sdkCall()); } ``` ## Tasks in processors [#tasks-in-processors] Processors (and every `BaseObject`) create tasks through their task manager with `this.createTask(fn, name)`, and cancel them with `await this.cancelTask(task)` (which waits up to 1 second for the task to finish). Tasks created this way are tracked and cancelled when the pipeline cleans up, so nothing is left floating. Never fire and forget a bare promise. ```ts twoslash import { rethrowIfCancelled, sleep, type Task } from "@pipedog/asyncio"; import { type Frame, InterruptionFrame, StartFrame, TextFrame } from "@pipedog/core/frames/frames"; import { type FrameDirection, FrameProcessor } from "@pipedog/core/processors/frame_processor"; /** Says "tick" every second until the pipeline ends. */ export class Ticker extends FrameProcessor { #tickTask: Task | null = null; override async processFrame(frame: Frame, direction: FrameDirection): Promise { await super.processFrame(frame, direction); if (frame instanceof StartFrame) { // Tracked by the task manager: cancelled at cleanup, never left floating. this.#tickTask = this.createTask(() => this.#tick(), "tick"); } else if (frame instanceof InterruptionFrame) { console.log("interrupted: stop what we were saying"); } await this.pushFrame(frame, direction); } async #tick(): Promise { for (;;) { await sleep(1.0); // seconds, never raw setTimeout try { await this.pushFrame(new TextFrame({ text: "tick" })); } catch (e) { rethrowIfCancelled(e); await this.pushError("tick failed", { exception: e }); } } } override async cleanup(): Promise { await super.cleanup(); if (this.#tickTask) { await this.cancelTask(this.#tickTask); this.#tickTask = null; } } } ``` `await task` waits for the task as asyncio does: if your task is cancelled while it waits, `task` is cancelled too. To react to a task's end without waiting for it, use `task.addDoneCallback(cb)`, `task.promise.then(cb)` or `task.then(cb)`. These don't tie the two tasks together, so cancelling yours leaves `task` running until it is cancelled itself. (In 0.10.0 and earlier, `task.then(cb)` could leave the calling task impossible to cancel and hang `cleanup()`.) ## Rules of thumb [#rules-of-thumb] 1. Every broad `catch` begins with `rethrowIfCancelled(e)`. 2. Use `sleep(seconds)` from `@pipedog/asyncio`, never a raw `setTimeout`. 3. Pass `currentSignal()` to `fetch` and other APIs that take an `AbortSignal`; wrap the rest in `abortable()`. 4. Never leave a promise floating: use `this.createTask()` in processors, `createTask()` elsewhere. 5. Never `return` a `Task` (or any thenable) from an async function unless you mean to await it. The [asyncio API reference](https://polvallverdu.github.io/pipedog/docs/api/asyncio.md) lists every export. # Metrics & tracing > Performance and usage metrics as MetricsFrames, and OpenTelemetry tracing of conversations, turns and services. Source: https://polvallverdu.github.io/pipedog/docs/concepts/metrics-and-tracing/ When building real-time AI applications two things matter to monitor: **performance** (latency), which shapes the user experience, and **usage** (LLM tokens, TTS characters), which drives cost. pipedog ports Pipecat's built-in metrics for both. ## Enabling metrics [#enabling-metrics] Metrics are switched on through `PipelineParams` when creating the worker: ```ts const worker = new PipelineWorker(pipeline, { params: new PipelineParams({ enableMetrics: true, // TTFB, processing time, ... enableUsageMetrics: true, // LLM tokens, TTS characters, ... reportOnlyInitialTtfb: true, // only the first TTFB per service sendInitialEmptyMetrics: false, }), observers: [ // Log only LLM and TTS usage. new MetricsLogObserver({ includeMetrics: new Set([LLMUsageMetricsData, TTSUsageMetricsData]), }), ], }); ``` With `enableMetrics`, services report: | Metric | Description | | ---------------- | ---------------------------------------------------------------------------- | | TTFB | Time to first byte, in seconds. | | TTFA | Time to first audio, in seconds (TTS services). | | TTFAT | Time to first answer token, in seconds (LLM services). | | Processing time | Time the service took to respond, in seconds. | | Text aggregation | Time from the first LLM token to the first complete sentence (TTS services). | With `enableUsageMetrics`, services report LLM prompt/completion tokens and TTS characters. Usage metrics are per interaction, not running totals. * `reportOnlyInitialTtfb: true` reports only the first TTFB measurement of each service. * By default the worker sends an initial `MetricsFrame` with zero values for every service when the pipeline starts; `sendInitialEmptyMetrics: false` turns that off. ## Capturing metrics data [#capturing-metrics-data] Every measurement travels as a `MetricsFrame` (a system frame) whose `data` is a list of `MetricsData` objects from `@pipedog/core/metrics/metrics`: * `TTFBMetricsData`, `TTFAMetricsData`, `TTFATMetricsData` * `ProcessingMetricsData`, `TextAggregationMetricsData` * `LLMUsageMetricsData` (its `value` is an `LLMTokenUsage`: `promptTokens`, `completionTokens`, `totalTokens`, cache and reasoning token counts) * `STTUsageMetricsData`, `TTSUsageMetricsData` * `TurnMetricsData` All of them have `processor` (the name of the processor that produced the metric) and an optional `model`. Read them with an **observer** such as `MetricsLogObserver` (above), or with a custom processor placed after the services: ```ts export class MetricsCollector extends FrameProcessor { override async processFrame(frame: Frame, direction: FrameDirection): Promise { await super.processFrame(frame, direction); if (frame instanceof MetricsFrame) { for (const data of frame.data) { if (data instanceof TTFBMetricsData) { console.log(`${data.processor} TTFB: ${data.value}s`); } else if (data instanceof ProcessingMetricsData) { console.log(`${data.processor} processing: ${data.value}s`); } else if (data instanceof TTFAMetricsData) { console.log( `${data.processor} TTFA: ${data.ttfa}s (leading silence ${data.leadingSilence}s)`, ); } else if (data instanceof LLMUsageMetricsData) { const { promptTokens, completionTokens } = data.value; console.log(`${data.processor} prompt: ${promptTokens}, completion: ${completionTokens}`); } else if (data instanceof TTSUsageMetricsData) { console.log(`${data.processor} characters: ${data.value}`); } } } await this.pushFrame(frame, direction); } } ``` Processors that generate metrics use the helpers on `FrameProcessor`: `startTtfbMetrics()` / `stopTtfbMetrics()`, `startProcessingMetrics()` / `stopProcessingMetrics()`, `startLlmUsageMetrics(tokens)`, `startTtsUsageMetrics(text)` and so on. They only report when metrics are enabled, and `canGenerateMetrics()` tells the worker which processors to include in the initial empty metrics. To send TTFB and processing time to Sentry as transactions, see [Sentry metrics](https://polvallverdu.github.io/pipedog/docs/concepts/sentry-metrics.md). ## Tracing (OpenTelemetry) [#tracing-opentelemetry] pipedog ports Pipecat's OpenTelemetry tracing: one trace per conversation, a span per turn, and spans for the service calls in each turn. Span names, attributes and their hierarchy are the same as upstream's, so dashboards and LLM-observability tools built for Pipecat traces (Langfuse, Honeycomb, Grafana Tempo, Jaeger, ...) work unchanged. Like upstream's `pipecat-ai[tracing]` extra, OpenTelemetry is optional: * `@pipedog/core` has `@opentelemetry/api` as an **optional peer dependency**. Without it, core runs exactly the same, `isTracingAvailable()` is false and every tracing helper is a no-op. * `@pipedog/tracing` brings the OpenTelemetry SDK, an OTLP/HTTP exporter and `setupTracing()`. ```bash bun add @pipedog/tracing @opentelemetry/api ``` ### Setup [#setup] Call `setupTracing()` once at startup, then turn tracing on per worker with `enableTracing`: ```ts // Once, at startup (upstream: setup_tracing("my-voice-bot", exporter=...)). setupTracing({ serviceName: "my-voice-bot", exporter: new OTLPTraceExporter({ url: "http://localhost:4318/v1/traces", // any OTLP/HTTP collector }), consoleExport: false, // true also prints every span (debugging) }); // Then per conversation: const worker = new PipelineWorker(pipeline, { enableTracing: true, conversationId: "customer-123", // optional; a UUID otherwise additionalSpanAttributes: { "app.tenant": "acme" }, // added to the conversation span }); ``` `setupTracing({ serviceName, exporter, consoleExport })` matches upstream's `setup_tracing(service_name, exporter, console_export)`. It sets the resource attributes `service.name`, `service.instance.id` (`$HOSTNAME`) and `deployment.environment` (`$ENVIRONMENT`, default `development`), adds a batch span processor for each exporter, and registers the global tracer provider (which, as in Python, can only be set once per process). It returns `false` if setup failed. `@pipedog/tracing` re-exports `OTLPTraceExporter` (from `@opentelemetry/exporter-trace-otlp-http`, which sends OTLP/HTTP JSON with `fetch` or `node:http`), `ConsoleSpanExporter`, `InMemorySpanExporter` and the span processors. Any OpenTelemetry `SpanExporter` works. Pointing it at a hosted backend is a matter of URL and headers: ```ts // Langfuse (OTLP endpoint, Basic auth with your public/secret key pair) const langfuse = new OTLPTraceExporter({ url: "https://cloud.langfuse.com/api/public/otel/v1/traces", headers: { Authorization: `Basic ${btoa("pk-lf-...:sk-lf-...")}` }, }); // Honeycomb const honeycomb = new OTLPTraceExporter({ url: "https://api.honeycomb.io/v1/traces", headers: { "x-honeycomb-team": "YOUR_API_KEY" }, }); ``` ### Names: pipedog by default, pipecat for compatibility [#names-pipedog-by-default-pipecat-for-compatibility] Span names and attribute keys are upstream's (see below). Two names that say which library made the trace differ, so a pipedog bot can be told apart from a Pipecat one: | | pipedog | Pipecat | | ---------------------------------------------------------------- | -------------- | -------------- | | Tracer (instrumentation scope, `otel.scope.name`): service spans | `pipedog` | `pipecat` | | Tracer: conversation and turn spans | `pipedog.turn` | `pipecat.turn` | | Default `service.name` (when you don't pass `serviceName`) | `pipedog` | `pipecat` | Always set your own `serviceName` in production: it should name your app, not the library. **If anything filters on Pipecat's tracer names** (a dashboard, an alert, a query on `otel.scope.name`, or a Python and a TypeScript bot reporting to the same place), switch them back with `setTracerName("pipecat")`, or `tracerName: "pipecat"` in `setupTracing`. Do it at startup, before a pipeline is created, because the turn tracer is fetched when its observer is built. Span names (`conversation`, `turn`, `stt`, `llm`, `tts`) and attribute keys (`gen_ai.*`, `metrics.ttfb`, ...) don't depend on this: they are the same either way. ```ts // Tracers are called "pipedog" and "pipedog.turn" by default. For dashboards, alerts or queries // that filter on Pipecat's names ("pipecat" and "pipecat.turn"), switch them back at startup, // before any pipeline is created: setTracerName("pipecat"); // or in setupTracing, which does the same: setupTracing({ serviceName: "my-voice-bot", tracerName: "pipecat", exporter: new OTLPTraceExporter({ url: "http://localhost:4318/v1/traces" }), }); ``` ### Spans and attributes [#spans-and-attributes] ``` conversation conversation.id, conversation.type = "voice", + additionalSpanAttributes └── turn turn.number, turn.type, conversation.id, turn.duration_seconds, │ turn.was_interrupted, turn.user_bot_latency_seconds ├── stt gen_ai.provider.name, gen_ai.request.model, gen_ai.operation.name, │ transcript, is_final, language, vad_enabled, metrics.ttfb, │ metrics.audio_seconds, settings.* ├── llm gen_ai.provider.name, gen_ai.request.model, gen_ai.operation.name = "chat", │ stream, input, output, tools, tool_count, tool_choice, │ gen_ai.system_instructions, gen_ai.usage.input_tokens, │ gen_ai.usage.output_tokens, metrics.ttfb, gen_ai.request.*, param.* └── tts gen_ai.provider.name, gen_ai.request.model, gen_ai.output.type = "speech", voice_id, text, metrics.character_count, metrics.ttfb, settings.* ``` * The **conversation** span starts with the pipeline (`StartFrame`) and ends when the worker finishes; its id is `conversationId`, or a random UUID. * A **turn** span covers one user turn and the bot's answer, as tracked by `TurnTrackingObserver`. * **Service** spans are children of the current turn (or of the conversation, between turns). They come from the `tracedStt` / `tracedLlm` / `tracedTts` decorators (and `tracedGeminiLive`, `tracedOpenaiRealtime` for speech-to-speech services, whose spans are named after the operation: `llm_setup`, `llm_request`, `llm_response`, `llm_tool_call`, `llm_tool_result`) in `@pipedog/core/utils/tracing/service_decorators`. Provider packages apply them as upstream's services do, so a service shows up in traces once its package is ported. The metrics described above are recorded on these spans (`metrics.ttfb`, token usage, character counts), not as separate OpenTelemetry metrics: like Pipecat, pipedog has no OpenTelemetry metrics exporter. Read `MetricsFrame`s with an observer if you need them as time series. ### Keeping content out of traces [#keeping-content-out-of-traces] Like Pipecat's, the spans carry what people said and what the model answered: `transcript`, `text`, `input`, `output`, `gen_ai.system_instructions`, tool definitions. With sensitive data (health, payments, anything personal) pass `redactContent` and the spans keep timings, ids and metadata only: ```ts import { setupTracing } from "@pipedog/tracing"; setupTracing({ serviceName: "my-voice-bot", exporter, redactContent: true }); ``` * String attributes are dropped unless their key is allowed. The default list (`DEFAULT_REDACTION_ALLOWED_ATTRIBUTES`) holds the non-content keys pipedog writes: the model, provider and operation (`gen_ai.*`), `voice_id`, `language`, `settings.model` / `voice` / `language`, `conversation.id`, `conversation.type`, `turn.type`, tool names. Numbers and booleans (`metrics.*`, token counts, `turn.*` durations) are kept on any key. * Events keep their name and only `exception.type`; a recorded exception keeps its type but not its message or stack; a status keeps its code but not its message. * `redactContent: { extraAllowedAttributes: ["app.tenant"] }` adds your own keys (for example from `additionalSpanAttributes`), `allowedAttributes` replaces the list, and `isAllowed(key, value)` decides on its own. Redaction happens as each attribute is written, on the SDK's own span, so nothing reaches a span processor or exporter. If you set up OpenTelemetry yourself, call `registerWithCore({ redactContent: true })` instead; `redactTracer`, `redactTracerProvider` and `redactSpan` apply it to your own spans. Not in upstream. ### Cloudflare Workers [#cloudflare-workers] `@pipedog/core` and `@pipedog/tracing` run on Workers (see the [compatibility table](https://polvallverdu.github.io/pipedog/docs/guides/runtimes.md)). The OTLP exporter posts spans with `fetch`, and spans are batched, so flush them before the request ends: ```ts export default { async fetch(_request: Request, _env: unknown, ctx: { waitUntil(p: Promise): void }) { const worker = new PipelineWorker(new Pipeline([]), { enableTracing: true }); const runner = new WorkerRunner({ handleSigint: false }); await runner.addWorkers(worker); await runner.run(); // Spans are batched: export what is left before the request ends. ctx.waitUntil(forceFlushTracing()); return new Response("ok"); }, }; ``` `forceFlushTracing()` and `shutdownTracing()` are pipedog additions (a Python process flushes on exit). ### Without `@pipedog/tracing` [#without-pipedogtracing] If you already set up the OpenTelemetry SDK yourself (for example with `@opentelemetry/sdk-node`), you only need `@opentelemetry/api` installed: core imports it on its own, and a worker with `enableTracing: true` traces whenever it is found, wherever and whenever the worker was built: ```ts // Without @pipedog/tracing: set up the OpenTelemetry SDK your way (it registers // the global tracer provider), then enable tracing on the worker as usual. Core // imports @opentelemetry/api on its own. const traced = new PipelineWorker(pipeline, { enableTracing: true }); // Optional: check whether OpenTelemetry was found... console.log(await loadOpenTelemetry(), isTracingAvailable()); // ...or hand in a specific copy of the API yourself. registerOpenTelemetry({ api }); ``` # Pipelines & workers > How a Pipeline links processors, how a PipelineWorker runs it, and how a WorkerRunner runs workers. Source: https://polvallverdu.github.io/pipedog/docs/concepts/pipelines-and-workers/ Three classes take a pipeline from a list of processors to a running bot: | Class | Module | Job | | ---------------- | --------------------------------- | ----------------------------------------------------------------------------------------------------------- | | `Pipeline` | `@pipedog/core/pipeline/pipeline` | Links frame processors in sequence. It is itself a processor, so pipelines nest. | | `PipelineWorker` | `@pipedog/core/pipeline/worker` | Runs one pipeline: start, frame queueing, lifecycle events, idle and heartbeat monitoring, observers, RTVI. | | `WorkerRunner` | `@pipedog/core/workers/runner` | Runs workers to completion: signal handling, the shared bus, the worker registry. | `PipelineTask` and `PipelineRunner` are deprecated aliases upstream (of `PipelineWorker` and `WorkerRunner`) and are not ported. Use the `Worker` names. ## Pipeline [#pipeline] A `Pipeline` takes an array of processors and connects them in order, adding a source before the first and a sink after the last: ```ts const pipeline = new Pipeline([a, b, c]); ``` A typical voice pipeline (once the service packages are ported) looks exactly like Pipecat's: ```ts const pipeline = new Pipeline([ transport.input(), // receives user audio stt, // speech to text userAggregator, // collects user turns into the context llm, // generates responses tts, // text to speech transport.output(), // sends audio to the user assistantAggregator, // collects assistant turns ]); ``` **Order matters**: each processor must sit after the processors that produce the frames it needs. And **processors push frames on** rather than consuming them, so several processors can use the same audio or text without interfering. ### Parallel pipelines [#parallel-pipelines] `ParallelPipeline` creates branches that each receive every frame; their output is merged back. Branches are usually paired with filters so each handles only some frames. Like Python, each branch is a separate array argument: ```ts // Every branch receives every frame; their output is merged downstream. const parallel = new Pipeline([a, new ParallelPipeline([b], [c]), d]); ``` ## PipelineWorker [#pipelineworker] `PipelineWorker` runs a pipeline. It takes the pipeline and an options object (`PipelineWorkerOptions`), including the pipeline-wide `PipelineParams`: ```ts const worker = new PipelineWorker(pipeline, { params: new PipelineParams({ audioInSampleRate: 16000, audioOutSampleRate: 24000, enableMetrics: true, enableUsageMetrics: true, }), idleTimeoutSecs: 300, // seconds, as upstream cancelOnIdleTimeout: true, processorUnusablePolicy: ProcessorUnusablePolicy.CONTINUE, enableRtvi: false, }); ``` `PipelineParams` is passed to every processor through `FrameProcessorSetup`. Its fields (see the [PipelineWorker API](https://polvallverdu.github.io/pipedog/docs/api/pipeline-worker.md)): | Field | Default | Description | | ------------------------- | ------- | ---------------------------------------------------------- | | `audioInSampleRate` | `16000` | Input audio sample rate in Hz. | | `audioOutSampleRate` | `24000` | Output audio sample rate in Hz. | | `enableMetrics` | `false` | Collect performance metrics (TTFB, processing time). | | `enableUsageMetrics` | `false` | Collect usage metrics (LLM tokens, TTS characters). | | `reportOnlyInitialTtfb` | `false` | Report only the first TTFB per service. | | `sendInitialEmptyMetrics` | `true` | Send a zeroed `MetricsFrame` when the pipeline starts. | | `enableHeartbeats` | `false` | Push `HeartbeatFrame`s to detect a stalled pipeline. | | `heartbeatsPeriodSecs` | `1.0` | Seconds between heartbeats. | | `heartbeatsMonitorSecs` | `10.0` | Seconds without a heartbeat before `on_heartbeat_timeout`. | | `startMetadata` | `{}` | Extra metadata for pipeline start. | Other worker behavior is set on the worker options: `idleTimeoutSecs` (default 300) and `idleTimeoutFrames`, `cancelOnIdleTimeout`, `cancelTimeoutSecs`, `setupTimeoutSecs` and `startTimeoutSecs` (20 s each), `observers`, `enableRtvi`, `enableTurnTracking`, `enableTracing`, `conversationId`, `appResources`, `processorUnusablePolicy`, `bridged` and more. ### Event handlers [#event-handlers] | Event | Arguments | Fires when | | ----------------------------- | ----------------------------- | ------------------------------------------------------------------------------------- | | `on_pipeline_started` | `(worker, frame: StartFrame)` | The `StartFrame` reached the end of the pipeline. | | `on_pipeline_finished` | `(worker, frame)` | The pipeline reached a terminal state: `StopFrame`, `EndFrame` or `CancelFrame`. | | `on_pipeline_error` | `(worker, frame: ErrorFrame)` | An `ErrorFrame` reached the source. Check `frame.processor.isUsable`. | | `on_pipeline_timeout` | `(worker, frame)` | A frame the worker waited on (a `StartFrame` or `CancelFrame`) never reached the end. | | `on_setup_timeout` | `(worker)` | Setting the processors up took longer than `setupTimeoutSecs`. | | `on_idle_timeout` | `(worker)` | None of `idleTimeoutFrames` arrived within `idleTimeoutSecs`. | | `on_heartbeat_timeout` | `(worker)` | No heartbeat within `heartbeatsMonitorSecs` (repeats while stalled). | | `on_frame_reached_upstream` | `(worker, frame)` | A frame matching the upstream filter reached the source. | | `on_frame_reached_downstream` | `(worker, frame)` | A frame matching the downstream filter reached the sink. | ```ts worker.eventHandler("on_pipeline_started", async (_worker: PipelineWorker, _frame: Frame) => { console.log("pipeline started"); }); worker.eventHandler("on_pipeline_finished", async (_worker: PipelineWorker, frame: Frame) => { // StopFrame, EndFrame or CancelFrame: inspect it to tell the cases apart. console.log(`pipeline finished: ${frame}`); }); worker.eventHandler("on_pipeline_error", async (_worker: PipelineWorker, frame: ErrorFrame) => { if (frame.processor && !frame.processor.isUsable) { await worker.cancel({ reason: `${frame.processor} is no longer usable` }); } }); worker.eventHandler("on_idle_timeout", async (_worker: PipelineWorker) => { console.log("nobody said anything for a while"); }); ``` Set the frame types `on_frame_reached_*` fire for with `setReachedUpstreamFilter([...])` / `addReachedDownstreamFilter([...])`. ### Queueing frames [#queueing-frames] Frames queued on the worker enter at the start of the pipeline (or, upstream, at the end). They are processed in order within their lane, so a goodbye is spoken before the `EndFrame` ends the pipeline: ```ts // Executed in order: speak first, then end the pipeline once that is flushed. await worker.queueFrames([ new TTSSpeakFrame({ text: "Goodbye!" }), new EndFrame({ reason: "user said goodbye" }), ]); // Interrupt whatever the bot is doing. await worker.queueFrame(new InterruptionWorkerFrame()); // Push from the end of the pipeline, towards the start. await worker.queueFrame(new TextFrame({ text: "upstream" }), FrameDirection.UPSTREAM); ``` ## Ending a pipeline [#ending-a-pipeline] | Frame | Job | How to push it | | ------------------- | ------------------------------------------------------ | ------------------------------------------------------------------------------ | | `EndFrame` | Graceful shutdown, after pending frames are processed. | From outside: `worker.queueFrame(new EndFrame())`, or `worker.stopWhenDone()`. | | `CancelFrame` | Immediate shutdown, discarding pending frames. | `worker.cancel()`. | | `EndWorkerFrame` | Graceful shutdown requested from inside the pipeline. | `this.pushFrame(new EndWorkerFrame(), FrameDirection.DOWNSTREAM)`. | | `CancelWorkerFrame` | Immediate shutdown requested from inside the pipeline. | `this.pushFrame(new CancelWorkerFrame())`. | ```ts await worker.stopWhenDone(); // queue an EndFrame behind everything already queued await worker.end({ reason: "done" }); // request a graceful end, draining first await worker.cancel({ reason: "shutting down" }); // stop now, dropping queued frames ``` From inside a processor, push the worker frames. Push `EndWorkerFrame` **downstream**, so it queues behind the work ahead of it instead of jumping past it: ```ts /** Ends the conversation when it sees a "goodbye" TextFrame. */ export class GoodbyeDetector extends FrameProcessor { override async processFrame(frame: Frame, direction: FrameDirection): Promise { await super.processFrame(frame, direction); await this.pushFrame(frame, direction); if (frame instanceof TextFrame && frame.text.toLowerCase().includes("goodbye")) { await this.pushFrame(new TTSSpeakFrame({ text: "Have a nice day!" })); // Downstream (the default), so the goodbye is flushed before the pipeline ends. await this.pushFrame(new EndWorkerFrame(), FrameDirection.DOWNSTREAM); } else if (frame instanceof TextFrame && frame.text === "abort") { // Ends right away, abandoning queued frames. await this.pushFrame(new CancelWorkerFrame()); } } } ``` ## WorkerRunner [#workerrunner] `WorkerRunner` runs workers until they finish. For a single-pipeline bot, add the worker and run: ```ts const runner = new WorkerRunner({ handleSigint: true, // the default: Ctrl-C cancels the runner (Bun, Node, Deno) }); await runner.addWorkers(worker); await runner.run(); // returns once every root worker has finished ``` * `run()` resolves once every root worker has finished, so a one-pipeline bot ends when its pipeline does. Pass `run({ autoEnd: false })` for a long-lived host (an HTTP server, say) that adds and removes workers over many sessions; it then runs until `runner.end()` or `runner.cancel()`. * With `handleSigint` (default `true`) and `handleSigterm` (default `false`) the runner cancels itself on those signals, on runtimes with process signals (Bun, Node, Deno). On Cloudflare Workers there are none, and it is a no-op. * `runner.end(reason)` ends every worker gracefully; `runner.cancel(reason)` cancels them. * Multi-worker bots add several workers with `addWorkers(a, b, ...)`; they talk over the runner's `bus` (an in-process `AsyncQueueBus` by default) and are found with `runner.getWorker(name)`. # Sentry metrics > Report TTFB and processing time of any service to Sentry as transactions with SentryMetrics, using the Sentry SDK for Bun, Node, Deno or Cloudflare Workers. Source: https://polvallverdu.github.io/pipedog/docs/concepts/sentry-metrics/ ## Overview [#overview] `SentryMetrics` (`@pipedog/sentry`, a port of `pipecat.processors.metrics.sentry`) extends [`FrameProcessorMetrics`](https://polvallverdu.github.io/pipedog/docs/concepts/metrics-and-tracing.md). Wherever a service measures **time to first byte** or **processing time**, `SentryMetrics` also starts a Sentry transaction and ends it when the measurement stops: | Measurement | Transaction `op` | Transaction name | | ----------- | ---------------- | --------------------------------- | | TTFB | `ttfb` | `TTFB for ` | | Processing | `processing` | `Processing for ` | The usual `MetricsFrame`s are produced as before, so observers and RTVI clients see no difference. At cleanup it ends any pending transaction and flushes Sentry for up to 5 seconds. Pipecat's `sentry.py`. Setup guides per runtime. ## Installation [#installation] ```bash bun add @pipedog/sentry ``` `@pipedog/sentry` depends on no Sentry package. You install and initialise the Sentry SDK for your runtime, and pass its namespace in. See [Choosing the SDK](#choosing-the-sdk). ## Runtimes [#runtimes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/sentry/package.json`: ✅ means the compat smoke test passes on that runtime. The smoke test on every runtime drives `SentryMetrics` with `@sentry/core`'s tracing functions and a recording transport, and checks the transactions that come out. ## Usage [#usage] ```ts import { OpenAILLMService } from "@pipedog/openai/llm"; import { SentryMetrics } from "@pipedog/sentry"; import * as Sentry from "@sentry/core"; // Sentry.init({ dsn, tracesSampleRate: 1.0 }) has already run, with the SDK for your runtime. const llm = new OpenAILLMService({ apiKey: process.env.OPENAI_API_KEY ?? null, // One SentryMetrics per processor: each tracks its own transactions. metrics: new SentryMetrics({ sentry: Sentry }), }); ``` ```ts // Sentry transactions are only started when metrics are enabled. const worker = new PipelineWorker(new Pipeline([llm]), { params: new PipelineParams({ enableMetrics: true }), }); ``` Metrics have to be on (`enableMetrics`) for a service to measure anything, and so for Sentry to see anything. Pass `SentryMetrics` as the `metrics` option of a service (or any `FrameProcessor`), one instance per processor. ### Options [#options] | Option | Type | Required | Description | |---|---|---|---| | `sentry` | `SentryClient` | yes | The Sentry SDK to report to, for example `import * as Sentry from "@sentry/node"`. | If the SDK has not been initialised (`isInitialized()` is false when the `SentryMetrics` is constructed), a warning is logged and Sentry reporting is switched off. The ordinary metrics keep working. ## Choosing the SDK [#choosing-the-sdk] Sentry's JavaScript SDKs are one per runtime, and all re-export the same tracing functions from `@sentry/core`. `SentryMetrics` calls only three of them, described by the `SentryClient` interface: ```ts interface SentryClient { isInitialized(): boolean; startInactiveSpan(options: { op: string; name: string; forceTransaction: boolean }): { end(): void; }; flush(timeout?: number): PromiseLike; // milliseconds } ``` So `import * as Sentry from ""` satisfies it, and so does any object of your own with those three methods. | Runtime | SDK | Init | | ------------------ | -------------------- | ------------------------------------------------------------------------------- | | Node | `@sentry/node` | `Sentry.init({ dsn, tracesSampleRate: 1.0 })` at startup, before other imports. | | Bun | `@sentry/bun` | `Sentry.init({ dsn, tracesSampleRate: 1.0 })` at startup. | | Deno | `@sentry/deno` | `Sentry.init({ dsn, tracesSampleRate: 1.0 })` at startup. | | Cloudflare Workers | `@sentry/cloudflare` | Wrap the Worker so the SDK is initialised per request (see below). | ### Cloudflare Workers [#cloudflare-workers] `@sentry/cloudflare` initialises per request, so it needs the Worker wrapped: either the Vite plugin (`sentryCloudflareVitePlugin()` from `@sentry/cloudflare/vite`, with options in an `instrument.server.ts` that default-exports `defineCloudflareOptions((env) => ({ dsn, tracesSampleRate }))`) or `withSentry`. The Worker needs the `nodejs_compat` flag and a `compatibility_date` of `2024-09-23` or later. Then build the `SentryMetrics` inside the request handler (the WebSocket upgrade that starts the bot), so `isInitialized()` is true when it is created: ```ts import * as Sentry from "@sentry/cloudflare"; import { SentryMetrics } from "@pipedog/sentry"; // inside the handler that starts the pipeline: const metrics = new SentryMetrics({ sentry: Sentry }); ``` Two Workers caveats from Sentry's documentation: span durations show 0 ms where only `performance.now()` and `Date.now()` are available, because the Workers clock advances only after I/O, and the SDK needs the request's lifetime to send events (the cleanup flush happens in the bot's own request). Treat the durations in Sentry on Workers as indicative; the `MetricsFrame` values come from the same clock and have the same limit. ## Differences from upstream [#differences-from-upstream] * **The SDK is passed in** (`sentry` option) instead of importing `sentry_sdk`, so one package serves every runtime. Upstream has one optional extra for the Python SDK. * **Transactions are root spans.** The JavaScript SDKs since v8 have no `startTransaction`; the equivalent is `startInactiveSpan({ forceTransaction: true })`. Sentry's docs say `forceTransaction` is not available in stream mode, which is the default trace lifecycle of `@sentry/core` 11. Set `traceLifecycle: "static"` in `Sentry.init` to get transactions as upstream sends them. With the default (stream) lifecycle the spans are sent as streamed segment spans, which we have not verified with a real Sentry backend. * **Ending a span runs inline** in the background task. Upstream hands `tx.finish` to an executor thread; ending a JavaScript span only records its end time. * **`flush` takes milliseconds** in Sentry's JS SDKs; `SentryMetrics` passes `5000`. One upstream behaviour is kept as is: with `reportOnlyInitialTtfb: true` the base class turns reporting off before `SentryMetrics` checks it, so that mode starts no TTFB transaction at all. ## Testing [#testing] Upstream has no test for this module. The package's own tests use a fake `SentryClient` (spans, flush, the not-initialised case, a span whose `end` throws) and the real `@sentry/core` with a transport that records envelopes. Nothing has been sent to Sentry itself, and `@sentry/cloudflare`, `@sentry/node`, `@sentry/bun` and `@sentry/deno` were not run; they share the functions tested here. # Pipecat Tone > Read how the user meant each turn, and voice the bot's plain replies with expression tags, using Pipecat Tone's small local models. Source: https://polvallverdu.github.io/pipedog/docs/concepts/tone/ Pipecat Tone is a pair of small models for expressive conversation (alpha): a **reader** for how the user spoke and a **tagger** for how the bot speaks. Both run on the CPU, through ONNX Runtime, in milliseconds. pipedog ports them as two processors in `@pipedog/core`, with the models in `@pipedog/tone`: * **`PipecatToneReader`** goes between the STT service and the user aggregator. It reads each final transcription, together with the bot's reply before it, and sets the reading as the transcription's `perception`: the user's two likeliest `emotion`s with their confidences, and `sarcasm`, the chance the turn is sarcastic. * **`PipecatToneTagger`** goes between the LLM and the TTS service. The LLM writes plain words, and the tagger labels each sentence as it streams in: a delivery, slow spans for details, stressed words, a sound before or after, and with `ExpressionGuidance.BREATHE` an occasional breath. It writes the labels as the expression tags the TTS service performs. Expressive conversation is alpha upstream, and these types are likely to change between releases. `packages/core/src/processors/tone` and `packages/tone/src`. `pipecat-ai/tone-models` on Hugging Face. ## Installation [#installation] ```bash bun add @pipedog/core @pipedog/tone onnxruntime-node ``` The processors are in `@pipedog/core`, which loads the models from `@pipedog/tone` when it is installed. `@pipedog/tone` runs them on [`@pipedog/onnx`](https://polvallverdu.github.io/pipedog/docs/deployment/bundling.md), with `onnxruntime-node` or `onnxruntime-web` as an optional peer. ```ts import { PipecatToneReader } from "@pipedog/core/processors/tone/reader/index"; import { PipecatToneTagger } from "@pipedog/core/processors/tone/tagger/index"; ``` ## Usage [#usage] Turn on expressive conversation in the `PipelineParams`, and place the reader after the STT service and the tagger after the LLM. The `ExpressionPolicy`'s presets steer the tagger, and your own guidance lines tell the LLM what to do with a reading: ```ts title="bot.ts" import { SileroVADAnalyzer } from "@pipedog/audio-silero"; import { formatPerception } from "@pipedog/core/expressive/perception"; import { ExpressionGuidance, ExpressionPolicy } from "@pipedog/core/expressive/policy"; import { Pipeline } from "@pipedog/core/pipeline/pipeline"; import { PipelineParams, PipelineWorker } from "@pipedog/core/pipeline/worker"; import { LLMContext } from "@pipedog/core/processors/aggregators/llm_context"; import { LLMContextAggregatorPair, LLMUserAggregatorParams, type UserTurnStoppedMessage, } from "@pipedog/core/processors/aggregators/llm_response_universal"; import { PipecatToneReader } from "@pipedog/core/processors/tone/reader/index"; import { PipecatToneTagger } from "@pipedog/core/processors/tone/tagger/index"; import type { BaseTransport } from "@pipedog/core/transports/base_transport"; import type { BaseUserTurnStopStrategy } from "@pipedog/core/turns/user_stop/index"; import { DeepgramSTTService } from "@pipedog/deepgram/stt"; import { ElevenLabsDialogueTTSService } from "@pipedog/elevenlabs/dialogue/tts"; import { OpenAILLMService } from "@pipedog/openai/llm"; declare const transport: BaseTransport; const stt = new DeepgramSTTService({ apiKey: process.env.DEEPGRAM_API_KEY ?? "" }); // Reads how each user turn was meant. Its model downloads from Hugging Face at start. const toneReader = new PipecatToneReader(); // Voices the LLM's plain replies, a sentence at a time. const toneTagger = new PipecatToneTagger(); const llm = new OpenAILLMService({ apiKey: process.env.OPENAI_API_KEY ?? "", settings: new OpenAILLMService.Settings({ systemInstruction: "You are a warm assistant in a voice conversation. Keep replies brief.", }), }); // Eleven v3 and v4 perform audio tags. const tts = new ElevenLabsDialogueTTSService({ apiKey: process.env.ELEVENLABS_API_KEY ?? "", settings: new ElevenLabsDialogueTTSService.Settings({ voice: process.env.ELEVENLABS_VOICE_ID ?? "", }), }); const context = new LLMContext(); const aggregators = new LLMContextAggregatorPair(context, { userParams: new LLMUserAggregatorParams({ vadAnalyzer: new SileroVADAnalyzer() }), }); const userAggregator = aggregators.user(); const pipeline = new Pipeline([ transport.input(), stt, toneReader, // how each user turn was meant userAggregator, llm, toneTagger, // how each sentence is voiced tts, transport.output(), aggregators.assistant(), ]); const worker = new PipelineWorker(pipeline, { params: new PipelineParams({ enableExpressiveConversation: true, expressionPolicy: new ExpressionPolicy({ guidance: [ ExpressionGuidance.MATCH_STYLE, ExpressionGuidance.SLOW_FOR_DETAILS, ExpressionGuidance.BREATHE, "If the user sounds angry at 0.7 or above, apologize once, then focus on helping.", ], }), }), }); userAggregator.eventHandler( "on_user_turn_stopped", async (_aggregator, _strategy: BaseUserTurnStopStrategy, message: UserTurnStoppedMessage) => { if (message.perception) console.log("tone:", formatPerception(message.perception)); }, ); ``` With a tagger in the pipeline, the LLM is taught to read the user's tone but no tags, and the context keeps its replies plain. The user aggregator writes each turn's reading at the head of the user's message, such as `([emotion frustrated=0.86, neutral=0.10],[sarcasm value=0.03]) Where is my order?`, and `on_user_turn_stopped` gets it in `UserTurnStoppedMessage.perception`. Each processor works without the other: with the reader alone, the LLM writes the tags itself (when the TTS service performs them), and the tagger alone voices replies without knowing how the user sounded. ## Options [#options] | Processor | Option | Default | Description | | ------------------- | ------------ | -------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | | `PipecatToneReader` | `revision` | `"reader-rc1"` | The model's release on Hugging Face. | | | `modelDir` | `null` | A folder (or an `http(s):` or `file:` URL) to load the model from instead of downloading it. | | | `precision` | `"fp16"` | `"fp16"` gives the same readings at half the size; or `"fp32"`. | | | `threads` | `4` | CPU threads for one read. | | `PipecatToneTagger` | `size` | `"17m"` | `"17m"`, or `"32m"`, which agrees with its teacher a little more often and takes nearly three times as long a sentence. | | | `revision` | `"tagger-rc1"` | The model's release on Hugging Face. | | | `modelDir` | `null` | A folder to load the model from, laid out as a release's `tagger/17m`. | | | `threads` | `1` | CPU threads for one sentence. | | | `vocabulary` | `null` | Names to write first for a delivery, sound or kind of breath, such as `{ [ExpressionDelivery.WARM]: ["cheerful"] }`, where the TTS service performs them. | Both also take `openModel`, a function that opens the model your own way. ```ts // A larger tagger, and a reader loaded from a folder of your own instead of Hugging Face. const largerTagger = new PipecatToneTagger({ size: "32m", threads: 2 }); const localReader = new PipecatToneReader({ modelDir: "/models/tone/reader", precision: "fp32" }); ``` ## Notes [#notes] * The model loads while the pipeline is set up, once a process, and every reader (or tagger) shares it. The first load downloads it from Hugging Face into `~/.cache/pipedog/tone` (or `$PIPEDOG_CACHE_DIR/tone`), which holds the pipeline's start, so download it ahead, such as when building the image. * A model that can't be loaded is reported as an error that leaves the processor unusable: transcriptions pass through unread, and replies pass through as the LLM wrote them, with the LLM writing the tags itself. Set the worker's `processorUnusablePolicy` to end the pipeline instead. * The tagger was trained with the Cartesia, ElevenLabs, Fish Audio and MiniMax TTS services, and reads any other as a generic one. It only tags with expressive conversation on. * Each read or sentence waits for the model: a few milliseconds a turn for the reader, about 15 milliseconds a sentence for the 17m tagger. ## Runtimes [#runtimes] | Runtime | Works | What it needs | | ------------------ | -------------- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | — not verified | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/tone/package.json`: ✅ means the compat smoke test passes on that runtime. The download cache needs a file system. On Cloudflare Workers, host the model files and pass their `https:` URL as `modelDir`. Upstream runs the models off the event loop in a thread. Here the ONNX runtime runs them asynchronously, and the model's `tag()` returns a promise. # TTS audio cache > Keep the audio of fixed lines (a greeting, a disclaimer) and play it without asking the TTS provider again, with TTSAudioCache and warmAudioCache(). Source: https://polvallverdu.github.io/pipedog/docs/concepts/tts-audio-cache/ Fixed lines, such as a greeting, a recording notice or a hold message, are synthesized again on every call that says them. That costs a request and its time to first byte each time. With an **audio cache**, a TTS service that already has the audio for a line plays it at once, without a request to the provider. This is pipedog only: upstream Pipecat has no TTS cache. ```ts import { TTSAudioCache } from "@pipedog/core/services/tts_audio_cache"; import { OpenAITTSService } from "@pipedog/openai/tts"; // One cache for the process, shared by every service and session. const audioCache = new TTSAudioCache({ maxBytes: 32 * 1024 * 1024 }); const tts = new OpenAITTSService({ apiKey, audioCache }); // Ahead of time (for example while an outbound call rings): await tts.warmAudioCache("Hi! This is the clinic calling about your appointment.", { sampleRate: 8000, // the rate the pipeline will run at }); ``` When the bot later says that text, each of its sentences is served from the cache: the service pushes `TTSStartedFrame`, the audio as `TTSAudioRawFrame`s of `chunkSize` bytes (half a second) with the context ID of the request, and `TTSStoppedFrame`, as for live synthesis. TTFB is reported (it is near zero) and no TTS usage metrics are, since the provider wasn't asked. ## What is cached [#what-is-cached] * **Only warmed lines, by default.** A generated reply can repeat what the user said, so live synthesis is not stored unless you set `cacheSynthesizedAudio`: `true` stores every line, and a function `(text) => boolean` decides per line. * **Lines, as the service asks for them.** LLM text is cut into sentences before synthesis, so `warmAudioCache(text)` cuts the text the same way (`tts.spokenSentences(text)` shows the result) and runs it through the service's text transforms. A `TTSSpeakFrame` is spoken whole: warm its text with `{ split: false }`. * **The key** is the text plus the service class, its settings (voice, model, language and the rest of `settings`), the sample rate and `audioCacheNamespace`. Change the voice and the cached audio no longer matches. Pass `audioCacheNamespace` when a constructor option outside `settings` changes the audio, or override `_audioCacheKey(text)` in a subclass. * **Whole syntheses only.** A synthesis that is interrupted, reports an error, or yields audio at another sample rate is not stored. `TTSAudioCache` keeps up to `maxBytes` of audio in memory and drops the least recently used first. To keep audio elsewhere (KV, Redis), pass any object with `get(key)` and `set(key, audio)` (`TTSAudioCacheStore`); both may return promises. ## Which services [#which-services] The cache works with services that yield their audio from `runTts`: HTTP services and streaming-RPC services such as Google's. Services that receive audio on their own WebSocket (`supportsAudioCache` is false) can't tell which sentence a piece of audio belongs to, so they ignore the cache, and so does any service in token aggregation mode. `warmAudioCache()` calls `runTts` outside the pipeline. Before the pipeline starts, pass `sampleRate` (or give the service a `sampleRate` option), and it must equal the rate the pipeline runs at. A service that only creates its client when the pipeline starts can only warm once it has started. # Turn detection > Decide when the user has finished speaking, with local models, with VAD only, or on the provider's side. Source: https://polvallverdu.github.io/pipedog/docs/concepts/turn-detection/ A voice agent has to know when the user starts speaking, to stop talking, and when they've finished, to answer. pipedog does this in one of three ways. They differ in what runs on your server: two local models, one, or none. | Mode | Start of turn | End of turn | Runs locally | | ---------------------- | --------------- | --------------- | ------------------------ | | Local models (default) | Silero VAD | Smart Turn | ONNX runtime, two models | | VAD only | Silero VAD | silence | ONNX runtime, one model | | Provider | the STT service | the STT service | nothing | Each mode is a choice of [turn strategies](https://polvallverdu.github.io/pipedog/docs/concepts/turn-strategies.md) on the user aggregator. This page covers the three common setups; the strategies page covers every strategy and option. ## Local models [#local-models] Silero VAD hears speech; Smart Turn listens to the end of each utterance and decides whether the user is done or only paused. This is Pipecat's default, and pipedog's: ```ts const aggregators = new LLMContextAggregatorPair(context, { userParams: new LLMUserAggregatorParams({ vadAnalyzer: new SileroVADAnalyzer() }), }); ``` The default turn strategies use Smart Turn from `@pipedog/audio-smart-turn`, so install it next to `@pipedog/audio-silero`. Without it, turns end on silence (the VAD-only mode), and pipedog logs a warning once. In a bundled app, pass the analyzer to the strategy yourself; see [Bundling](https://polvallverdu.github.io/pipedog/docs/deployment/bundling.md). Both models share one inference session per process: the first call loads them, and later calls reuse them. Measured on Bun with `onnxruntime-node`, the first call takes about 100 MB for the runtime and both models, and each further call about 1.3 MB. ## VAD only [#vad-only] Leave out `@pipedog/audio-smart-turn`, or end turns on silence explicitly: ```ts const userParams = new LLMUserAggregatorParams({ vadAnalyzer: new SileroVADAnalyzer(), userTurnStrategies: new UserTurnStrategies({ stop: [new SpeechTimeoutUserTurnStopStrategy()], }), }); ``` This answers sooner after a pause than Smart Turn does, so it interrupts users who pause to think more often. ## Another VAD: TEN VAD [#another-vad-ten-vad] Silero is the default VAD. [TEN VAD](https://polvallverdu.github.io/pipedog/docs/audio/ten-vad.md) (`TenVADAnalyzer`, pipedog only) is an opt-in alternative that runs TEN's WebAssembly build, which you obtain from TEN and pass in, under TEN's own license. It takes 16 kHz audio only. Pass it as `vadAnalyzer` wherever Silero goes. ## Remote Smart Turn over HTTP [#remote-smart-turn-over-http] If you can't run the model on the bot's host (a Worker, a small container), run Smart Turn as a service and let the bot send it the audio. `HttpSmartTurnAnalyzer` posts each candidate turn to an endpoint and reads the verdict back. It needs only `fetch`, so it works on every runtime, Cloudflare Workers included. | Runtime | Works | What it needs | | ------------------ | ----- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Bun | ✅ yes | Local Smart Turn v3: onnxruntime-node or onnxruntime-web. HttpSmartTurnAnalyzer: only fetch and a Smart Turn endpoint. | | Node.js | ✅ yes | Local Smart Turn v3: onnxruntime-node or onnxruntime-web. HttpSmartTurnAnalyzer: only fetch and a Smart Turn endpoint. | | Deno | ✅ yes | Local Smart Turn v3: onnxruntime-node or onnxruntime-web. HttpSmartTurnAnalyzer: only fetch and a Smart Turn endpoint. | | Cloudflare Workers | ✅ yes | Local Smart Turn v3: onnxruntime-node or onnxruntime-web. HttpSmartTurnAnalyzer: only fetch and a Smart Turn endpoint. Local Smart Turn: pass the model's bytes (there is no filesystem). HttpSmartTurnAnalyzer needs nothing extra. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/audio-smart-turn/package.json`: ✅ means the compat smoke test passes on that runtime. ```ts // Smart Turn as a service: each candidate turn is POSTed to the endpoint. const userParams = new LLMUserAggregatorParams({ vadAnalyzer: new SileroVADAnalyzer(), userTurnStrategies: new UserTurnStrategies({ start: [new VADUserTurnStartStrategy(), new TranscriptionUserTurnStartStrategy()], stop: [ new TurnAnalyzerUserTurnStopStrategy({ turnAnalyzer: new HttpSmartTurnAnalyzer({ url: "https://smart-turn.example.com/predict", headers: { Authorization: `Bearer ${process.env.SMART_TURN_KEY}` }, }), }), ], }), }); ``` It goes where the local analyzer does, in a `TurnAnalyzerUserTurnStopStrategy` (see [turn strategies](https://polvallverdu.github.io/pipedog/docs/concepts/turn-strategies.md)); `params: new SmartTurnParams({ stopSecs, ... })` takes the same settings. It makes the same request as Pipecat's: a `POST` of the audio (up to `maxDurationSecs`, as a NumPy `.npy` float32 vector at the analyzer's sample rate) with `Content-Type: application/octet-stream`, and expects JSON with a `prediction` (1 when the user is done) and a `probability`. Any failure, a non-200 status, a timeout after `stopSecs` or a body that isn't JSON, is logged and counts as "not done", as upstream does. `fetch` replaces the `aiohttp` session; pass your own as the `fetch` option. Pipecat's older `LocalSmartTurnAnalyzerV2` and the Core ML variant need PyTorch or Core ML and are not ported: Smart Turn v3, the default above, replaces them. ## Smart Turn on Workers AI [#smart-turn-on-workers-ai] Workers AI hosts Smart Turn v2 (`@cf/pipecat-ai/smart-turn-v2`). `WorkersAISmartTurnAnalyzer`, from `@pipedog/cloudflare`, works like `HttpSmartTurnAnalyzer` with the model on Cloudflare: through the Worker's `AI` binding, or anywhere with an account ID and API token. It sends the candidate turn as base64 float32 samples and reads Workers AI's `is_complete` and `probability`. A failed request counts as "not done"; one that takes longer than `stopSecs` ends the turn. ```ts // Smart Turn v2 on Workers AI: the AI binding in a Worker, or accountId and apiToken elsewhere. const userParams = new LLMUserAggregatorParams({ vadAnalyzer: new SileroVADAnalyzer(), userTurnStrategies: new UserTurnStrategies({ start: [new VADUserTurnStartStrategy(), new TranscriptionUserTurnStartStrategy()], stop: [ new TurnAnalyzerUserTurnStopStrategy({ turnAnalyzer: new WorkersAISmartTurnAnalyzer({ ai: env.AI }), }), ], }), }); ``` Smart Turn v2 was trained on 16 kHz audio, so run the pipeline input at 16 kHz. The local Smart Turn v3.2 above is newer and also runs on Workers (pass the model's bytes there), at the cost of the bundle size and memory described in [Voice agents on Workers](https://polvallverdu.github.io/pipedog/docs/deployment/cloudflare-workers/#turn-detection). On Workers AI, Deepgram Flux ([`WorkersAIFluxSTTService`](https://polvallverdu.github.io/pipedog/docs/services/stt/cloudflare.md)) detects turns itself and needs neither. ## Provider [#provider] Some speech-to-text services detect turns themselves. Use one of them and pass no VAD analyzer: the user aggregator follows the service's turn signals, and nothing runs locally. ```ts const stt = new OpenAIRealtimeSTTService({ apiKey: process.env.OPENAI_API_KEY ?? "", turnDetection: { type: "server_vad" }, }); // No vadAnalyzer: the STT service's turn signals drive the turns. const aggregators = new LLMContextAggregatorPair(context); ``` This is the mode for constrained hosts. The [Durable Objects example](https://polvallverdu.github.io/pipedog/docs/deployment/cloudflare-workers.md) uses it: its Worker is 818 KB gzipped, against 13.7 MB with the local models, which is over Cloudflare's 10 MB limit for a Worker. Speech-to-text services with their own turn detection: * `OpenAIRealtimeSTTService` (`@pipedog/openai/stt`) with `turnDetection: { type: "server_vad" }`. * `DeepgramFluxSTTService` (`@pipedog/deepgram/flux/stt`): Deepgram's turn-taking model. * `CartesiaTurnsSTTService` (`@pipedog/cartesia/turns/stt`). Each of these recommends its own turn strategies to the user aggregator ([`ExternalUserTurnStrategies`](https://polvallverdu.github.io/pipedog/docs/concepts/turn-strategies/#externaluserturnstrategies)), so passing no VAD analyzer is all it takes. To keep the bot talking when the provider hears the user, pass `shouldInterrupt: false` to the service rather than configuring the strategies yourself. ## Going further [#going-further] Every start and stop strategy, the presets, and how to write your own. Ignore the user during the greeting, during tool calls, or whenever the bot speaks. React to turns, prompt a silent user, and measure turns and response latency. What happens when the user talks over the bot, and how to control it. # Turn events & idle users > The events the context aggregators emit around each turn, prompting a user who has gone quiet, and the observers that track turns and response latency. Source: https://polvallverdu.github.io/pipedog/docs/concepts/turn-events/ The context aggregators emit an event at each step of a conversation turn. User events come from the user aggregator (`aggregators.user()`), assistant events from the assistant aggregator (`aggregators.assistant()`), both from an `LLMContextAggregatorPair`. Register handlers with `eventHandler()`; each handler receives the aggregator first. ```ts const aggregators = new LLMContextAggregatorPair(context, { userParams: new LLMUserAggregatorParams({ vadAnalyzer: new SileroVADAnalyzer() }), }); const user = aggregators.user(); const assistant = aggregators.assistant(); user.eventHandler( "on_user_turn_started", async (_aggregator, strategy: BaseUserTurnStartStrategy) => { console.log(`user turn started (${strategy.name})`); }, ); user.eventHandler( "on_user_turn_stopped", async (_aggregator, strategy: BaseUserTurnStopStrategy, message: UserTurnStoppedMessage) => { console.log(`user said "${message.content}" (${strategy.name}, from ${message.timestamp})`); }, ); // The text written to the context, in cascade and realtime mode alike. user.eventHandler( "on_user_turn_message_added", async (_aggregator, message: UserTurnMessageAddedMessage) => { console.log(`added to context: ${message.content}`); }, ); assistant.eventHandler( "on_assistant_turn_stopped", async (_aggregator, message: AssistantTurnStoppedMessage) => { console.log(`bot said "${message.content}" (interrupted: ${message.interrupted})`); }, ); ``` ## User events [#user-events] | Event | Arguments after the aggregator | Fires when | | ---------------------------------- | --------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `on_user_turn_started` | `strategy` | a start strategy starts a user turn | | `on_user_turn_inference_triggered` | `strategy` | the turn's inference runs. Usually together with `on_user_turn_stopped`; before it, possibly more than once, when the turn end waits for a decision on the inference (a [strategy that waits for it](https://polvallverdu.github.io/pipedog/docs/concepts/turn-strategies/#strategies-that-wait-for-the-inference)), or for a speculative inference with [`EagerUserTurnStrategies`](https://polvallverdu.github.io/pipedog/docs/concepts/turn-strategies/#eageruserturnstrategies) to be confirmed | | `on_user_turn_stopped` | `strategy`, `message: UserTurnStoppedMessage` | the stop strategies end the turn. `message.content` is the transcript, `null` in realtime mode | | `on_user_turn_message_added` | `message: UserTurnMessageAddedMessage` | the user's text is written to the context. In realtime mode that happens when the assistant starts answering, so use this event there for the user's text | | `on_user_turn_stop_timeout` | | the stop timeout ends a turn its stop strategies didn't end, after `userTurnStopTimeout` seconds without activity; `on_user_turn_stopped` follows | | `on_user_turn_idle` | | the user stays silent for `userIdleTimeout` seconds after the bot finishes ([below](#idle-users)) | | `on_user_mute_started` | | the user becomes [muted](https://polvallverdu.github.io/pipedog/docs/concepts/user-muting.md) | | `on_user_mute_stopped` | | the user is unmuted | | `on_user_backchannel` | `message: UserBackchannelMessage` | the user acknowledges the bot ("mhm") without interrupting it. The bot keeps talking and the text stays out of the context. Not while muted | `strategy` is the `BaseUserTurnStartStrategy` or `BaseUserTurnStopStrategy` that decided; for a turn ended by the stop timeout it is `null`. ## Assistant events [#assistant-events] | Event | Arguments after the aggregator | Fires when | | --------------------------- | ------------------------------------------ | -------------------------------------------------------------------------------------------------------- | | `on_assistant_turn_started` | | the assistant starts generating a response | | `on_assistant_turn_stopped` | `message: AssistantTurnStoppedMessage` | the response is complete, interrupted, or the pipeline ends. Fires even when `content` is empty | | `on_assistant_thought` | `message: AssistantThoughtMessage` | a reasoning model produces a thought | | `on_assistant_backchannel` | `message: AssistantBackchannelMessage` | the assistant starts saying a backchannel ("Mm-hmm"), which stays out of the context and the turn events | | `on_summary_applied` | `summarizer`, `event: SummaryAppliedEvent` | context summarization succeeds | ## Messages [#messages] The message classes are exported from `@pipedog/core/processors/aggregators/llm_response_universal`. Timestamps are ISO 8601 strings. | Class | Fields | | ----------------------------- | ---------------------------------------------------------------------------------------------------- | | `UserTurnStoppedMessage` | `content: string \| null`, `timestamp` (turn start), `userId: string \| null` | | `UserTurnMessageAddedMessage` | `content: string`, `timestamp` (turn start), `userId` | | `UserBackchannelMessage` | `content`, `timestamp` (when it was said), `userId` | | `AssistantTurnStoppedMessage` | `content` (what was actually spoken, may be empty), `interrupted: boolean`, `timestamp` (turn start) | | `AssistantThoughtMessage` | `content`, `timestamp` (thought start) | | `AssistantBackchannelMessage` | `content`, `timestamp` | Because the assistant aggregator only commits what was spoken, `AssistantTurnStoppedMessage.content` for an interrupted turn holds the words the user actually heard. ## Idle users [#idle-users] Set `userIdleTimeout` (seconds; 0, the default, turns it off) and the user aggregator emits `on_user_turn_idle` when the user hasn't spoken for that long after the bot finished: * The timer starts when the bot stops speaking (`BotStoppedSpeakingFrame`). * It is cancelled when the user or the bot starts speaking. * It doesn't run while a user turn is in progress (an interruption also stops the bot) or while function calls are pending. * It starts again after a user turn that ends with nothing to answer, such as background noise with no transcript. * It fires again after each further timeout, so a handler can escalate. What to do is up to you. Prompting the LLM with a developer message lets the bot ask in its own words; after a few tries, say goodbye and end the session: ```ts const aggregators = new LLMContextAggregatorPair(context, { userParams: new LLMUserAggregatorParams({ vadAnalyzer: new SileroVADAnalyzer(), userIdleTimeout: 8.0, // seconds of silence after the bot stops speaking }), }); const user = aggregators.user(); let idleCount = 0; user.eventHandler("on_user_turn_idle", async (aggregator: LLMUserAggregator) => { idleCount += 1; if (idleCount === 1) { await aggregator.pushFrame( new LLMMessagesAppendFrame({ messages: [ { role: "developer", content: "The user has been quiet. Ask if they're still there." }, ], runLlm: true, }), ); } else if (idleCount === 2) { await aggregator.pushFrame( new LLMMessagesAppendFrame({ messages: [ { role: "developer", content: "The user is still quiet. Ask whether they'd like to continue.", }, ], runLlm: true, }), ); } else { await aggregator.pushFrame(new TTSSpeakFrame({ text: "I'll let you go. Goodbye!" })); await aggregator.pushFrame(new EndWorkerFrame()); } }); // Any user turn resets the count. user.eventHandler("on_user_turn_started", async () => { idleCount = 0; }); ``` Resetting the count in `on_user_turn_started` keeps the escalation per silence. Timeouts of 5 to 10 seconds suit most voice conversations. To change the timeout while the pipeline runs, queue a `UserIdleTimeoutUpdateFrame`: ```ts await worker.queueFrame(new UserIdleTimeoutUpdateFrame({ timeout: 15.0 })); // change or enable await worker.queueFrame(new UserIdleTimeoutUpdateFrame({ timeout: 0 })); // disable ``` The new timeout applies at once: a running timer restarts with it, and going from 0 to a positive value arms the timer right away if the bot is already waiting for the user. `UserTurnProcessor` (`@pipedog/core/turns/user_turn_processor`), the standalone processor that runs turn strategies outside an aggregator, takes the same `userIdleTimeout` and emits the same event. Not to be confused with the worker's `idleTimeoutSecs`, which cancels a pipeline that has stopped receiving frames altogether (see [Pipelines & workers](https://polvallverdu.github.io/pipedog/docs/concepts/pipelines-and-workers.md)). ## Turn tracking [#turn-tracking] `TurnTrackingObserver` (`@pipedog/core/observers/turn_tracking_observer`) follows the conversation turn by turn, a turn being one user utterance and the bot's answer: * The first turn starts with the pipeline; later turns start when the user starts speaking. * A turn ends when the bot has stopped speaking and either the user speaks again or `turnEndTimeoutSecs` (2.5 s) passes without more bot speech. * If the user speaks while the bot is talking, the turn ends as interrupted and a new one starts. * If the pipeline ends mid-turn, that turn ends as interrupted. Every `PipelineWorker` adds one unless `enableTurnTracking` is `false`, and exposes it as `worker.turnTrackingObserver`: ```ts const worker = new PipelineWorker(pipeline); // enableTurnTracking defaults to true worker.turnTrackingObserver?.eventHandler( "on_turn_started", async (_observer, turnNumber: number) => { console.log(`turn ${turnNumber} started`); }, ); worker.turnTrackingObserver?.eventHandler( "on_turn_ended", async (_observer, turnNumber: number, duration: number, wasInterrupted: boolean) => { console.log(`turn ${turnNumber}: ${duration.toFixed(2)}s, interrupted: ${wasInterrupted}`); }, ); ``` | Event | Arguments after the observer | | ----------------- | ----------------------------------------------------------------------------- | | `on_turn_started` | `turnNumber: number` | | `on_turn_ended` | `turnNumber: number`, `duration: number` (seconds), `wasInterrupted: boolean` | To change the timeout, add your own and turn the default off: ```ts const turns = new TurnTrackingObserver({ turnEndTimeoutSecs: 3.5 }); const worker = new PipelineWorker(pipeline, { observers: [turns], enableTurnTracking: false, // or the worker adds a second, default observer }); ``` | Option | Type | Required | Description | |---|---|---|---| | `turnEndTimeoutSecs` | `number \| undefined` | no | Timeout in seconds after bot stops speaking before automatically ending the turn. Defaults to 2.5. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | With tracing on, the worker builds its turn spans from its own turn tracking observer (see [Metrics & tracing](https://polvallverdu.github.io/pipedog/docs/concepts/metrics-and-tracing.md)), so `enableTurnTracking: false` also turns off the turn spans. ## Response latency [#response-latency] `UserBotLatencyObserver` (`@pipedog/core/observers/user_bot_latency_observer`) measures how long the user waits: from the user going silent (`VADUserStoppedSpeakingFrame`) to the bot starting to speak (`BotStartedSpeakingFrame`). ```ts const latency = new UserBotLatencyObserver(); latency.eventHandler("on_latency_measured", async (_observer, seconds: number) => { console.log(`user stopped -> bot started: ${seconds.toFixed(3)}s`); }); latency.eventHandler("on_first_bot_speech_latency", async (_observer, seconds: number) => { console.log(`client connected -> first bot speech: ${seconds.toFixed(3)}s`); }); // The per-service figures need enableMetrics. latency.eventHandler("on_latency_breakdown", async (_observer, breakdown: LatencyBreakdown) => { for (const line of breakdown.turnContributionLines()) console.log(line); }); const worker = new PipelineWorker(pipeline, { params: new PipelineParams({ enableMetrics: true }), observers: [latency], }); ``` | Event | Arguments after the observer | Fires | | ----------------------------- | ----------------------------- | ---------------------------------------------------------- | | `on_latency_measured` | `seconds: number` | at each bot response to a user turn | | `on_first_bot_speech_latency` | `seconds: number` | once, from the client connecting to the bot's first speech | | `on_latency_breakdown` | `breakdown: LatencyBreakdown` | with either of the above | A `LatencyBreakdown` names where the time went: `contributions` splits the interval into the endpointing wait, transcription, LLM inference, speech synthesis and so on, `ttfb` lists each service's time to first byte, `userTurnSecs` how long turn detection held the turn after the user went silent, and `functionCalls` the time spent in tools. The per-service figures come from the services' metrics, so turn on `enableMetrics` in `PipelineParams`. `turnContributionLines()` formats it: ``` 0.200s endpointing wait [config: VAD stop_secs] 0.125s transcription [DeepgramSTTService#0] 0.336s LLM inference [OpenAILLMService#0] 0.359s speech synthesis [CartesiaTTSService#0] 1.020s TOTAL ``` | Option | Type | Required | Description | |---|---|---|---| | `minContributionSecs` | `number \| undefined` | no | Contributions shorter than this are rolled into the single pipeline entry rather than listed. Pass 0 to list every one, including individual frame hops. | | `timeSource` | `(() => number) \| undefined` | no | Reads the current time in seconds. Supplying one lets a test drive a cycle without waiting out the intervals it describes. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `observeEveryPush` | `boolean \| undefined` | no | Whether to observe every push of a frame, rather than only its first. Defaults to true. | With tracing enabled, the worker creates its own `UserBotLatencyObserver` for the turn spans; add another to the `observers` list for your own handlers, as above. # Turn strategies > The start and stop strategies that decide when a user turn begins and ends, the presets that bundle them, and how to write your own. Source: https://polvallverdu.github.io/pipedog/docs/concepts/turn-strategies/ The user aggregator decides when the user's turn starts and stops by asking a list of **turn strategies**. Start strategies watch for the user beginning to speak; stop strategies decide when they're done. You choose them with `userTurnStrategies` on `LLMUserAggregatorParams`. ```ts import { UserTurnStrategies } from "@pipedog/core/turns/user_turn_strategies"; import { VADUserTurnStartStrategy /* , ... */ } from "@pipedog/core/turns/user_start/index"; import { SpeechTimeoutUserTurnStopStrategy /* , ... */ } from "@pipedog/core/turns/user_stop/index"; ``` ## How a turn works [#how-a-turn-works] Every frame the user aggregator sees goes to every start and stop strategy. A strategy doesn't start or end the turn itself: it **reports a decision** (the user started, the user is done), and the turn controller combines the decisions of its list, following the list's `StrategyMode`. **Turn start.** By default (`startMode: StrategyMode.FIRST`) the first start strategy that decides starts the turn: the aggregator broadcasts a `UserStartedSpeakingFrame`, broadcasts an `InterruptionFrame` if the strategy has interruptions enabled (the default), and emits `on_user_turn_started`. With `StrategyMode.ALL` the turn starts once every start strategy has decided. **Turn stop.** By default (`stopMode: StrategyMode.ALL`) the turn ends once every stop strategy has decided the user is done; with `StrategyMode.FIRST`, on the first one. Then the aggregator broadcasts a `UserStoppedSpeakingFrame`, writes the user's transcript to the context, which runs the LLM, and emits `on_user_turn_stopped`. A list with one stop strategy behaves the same in either mode. The turn never ends while the user is audibly speaking: decisions made meanwhile wait until they stop. **Speaking again.** When the user starts speaking again in an open turn, the stop decisions made so far are cleared. If the turn's answer had already started, it is voided: the aggregator interrupts it, or, for a turn that doesn't interrupt the bot, broadcasts a `UserTurnResponseCancelFrame`. **Stop timeout.** A turn that sees no speech, transcript or other turn activity for `userTurnStopTimeout` seconds (default 5) ends anyway, after `on_user_turn_stop_timeout`, with what the decisions made so far said about it. It waits while a stop strategy has a decision of its own scheduled (`decisionScheduled`, such as the LLM's re-prompt timer). It is a safety net for a turn that never finalizes, not a way to tune responsiveness. To end a turn, a stop strategy needs two things: 1. **A transcript.** Strategies that wait for the transcript (`waitForTranscript`, on by default) hold the turn until speech-to-text delivers. This is usually the largest share of the delay. 2. **Its own criterion**: a silence window, a turn model's verdict, a provider's signal, or the LLM's verdict. | Stop strategy | Ends the turn when | | -------------------------------------------- | --------------------------------------------------- | | `TurnAnalyzerUserTurnStopStrategy` (default) | Smart Turn predicts the user is done | | `SpeechTimeoutUserTurnStopStrategy` | a fixed silence after VAD reports the end of speech | | `ExternalUserTurnStopStrategy` | the STT service or another processor says so | | `FilterIncompleteUserTurnStrategies` | the LLM judges the utterance complete | | Your own | your own logic | To make the bot answer sooner or later, change the stop strategy (for example `userSpeechTimeout`). Leave the VAD's `stopSecs` (0.2 s by default) alone: it is a low-level threshold that the strategies already account for. The VAD's volume gate is different: line noise and clicks often pass the model's confidence check, and `VADParams.minVolume` is what stops them. `minVolume` is a 0–1 value mapped from -110 LUFS to -10 LUFS (the default 0.6 is -50 LUFS). To tune it on a phone line, measure the noise floor in LUFS and set `minVolumeDb` a few dB above it instead (pipedog only): ```ts new SileroVADAnalyzer({ params: new VADParams({ minVolumeDb: -35 }) }); ``` ### Where the waiting time goes [#where-the-waiting-time-goes] * **VAD silence (`stopSecs`).** The VAD only reports that speech stopped after this much silence. * **Transcription.** Each STT service reports its p99 time to a final transcript (`ttfsP99Latency`); the stop strategies use it as a deadline counted from the end of speech, and stop waiting early as soon as the service marks a transcript as finalized. The VAD silence and the turn model's inference fall within that deadline rather than adding to it. * **The silence window** of `SpeechTimeoutUserTurnStopStrategy` is the exception: it starts when the VAD reports the end of speech, so the shortest answer after a pause is `stopSecs` plus `userSpeechTimeout` (0.8 s by default). The built-in p99 values assume `stopSecs` of 0.2. The strategies warn when `stopSecs` reaches the service's p99, because that leaves no time to wait for the transcript. ## Configuration [#configuration] These are the defaults, spelled out: ```ts const userParams = new LLMUserAggregatorParams({ vadAnalyzer: new SileroVADAnalyzer(), userTurnStrategies: new UserTurnStrategies({ // The first start strategy to fire opens the turn. start: [new VADUserTurnStartStrategy(), new TranscriptionUserTurnStartStrategy()], // Smart Turn decides when the user is done. stop: [ new TurnAnalyzerUserTurnStopStrategy({ turnAnalyzer: createLocalSmartTurnAnalyzerV3() }), ], }), // The first start strategy to decide starts the turn; every stop strategy // has to agree before it ends. userTurnStrategiesParams: new UserTurnStrategiesParams({ startMode: StrategyMode.FIRST, stopMode: StrategyMode.ALL, }), // A turn with no speech, transcript or turn activity for this long ends anyway. userTurnStopTimeout: 5.0, }); ``` `new UserTurnStrategies()` with no lists, or with empty ones, gets the same defaults: `defaultUserTurnStartStrategies()` (VAD, then transcription) and `defaultUserTurnStopStrategies()` (Smart Turn). Both functions are exported, so you can extend the defaults instead of repeating them. | Option | Type | Required | Description | |---|---|---|---| | `start` | `BaseUserTurnStartStrategy[] \| null \| undefined` | no | A list of user turn start strategies used to detect when the user starts speaking. | | `stop` | `BaseUserTurnStopStrategy[] \| null \| undefined` | no | A list of user turn stop strategies used to decide when the user stops speaking. | ### How strategies combine [#how-strategies-combine] `userTurnStrategiesParams` on `LLMUserAggregatorParams` (also an option of `UserTurnProcessor` and `UserTurnController`) takes a `UserTurnStrategiesParams`: | Option | Type | Required | Description | |---|---|---|---| | `startMergeMode` | `MergeMode \| undefined` | no | How the bot's start strategies combine with the ones a service recommends. Defaults to `MergeMode.REPLACE`. | | `stopMergeMode` | `MergeMode \| undefined` | no | How the bot's stop strategies combine with the ones a service recommends. Defaults to `MergeMode.REPLACE`. | | `startMode` | `StrategyMode \| undefined` | no | Whether the turn starts on the first start strategy that decides the user started, or once every start strategy has. Defaults to `StrategyMode.FIRST`. | | `stopMode` | `StrategyMode \| undefined` | no | Whether the turn ends once every stop strategy has decided the user is done, or on the first one that does. Defaults to `StrategyMode.ALL`. | * `startMode` and `stopMode` take a `StrategyMode`: `FIRST`, the first strategy to decide wins, or `ALL`, every strategy has to agree. In an `ALL` list every strategy is a condition the turn needs, so a fallback such as `TranscriptionUserTurnStartStrategy` belongs in a `FIRST` list. * `startMergeMode` and `stopMergeMode` take a `MergeMode`. A service with its own turn detection (an STT service, say) recommends strategies to the aggregator. With `REPLACE`, the default, your list is used on its own; with `EXTEND`, the strategies you gave are added to the service's (or to the defaults, when no service recommends any). ```ts // Two stop strategies: by default (StrategyMode.ALL) both must agree. // StrategyMode.FIRST ends the turn on whichever decides first. const userParams = new LLMUserAggregatorParams({ vadAnalyzer: new SileroVADAnalyzer(), userTurnStrategies: new UserTurnStrategies({ stop: [ new TurnAnalyzerUserTurnStopStrategy({ turnAnalyzer: createLocalSmartTurnAnalyzerV3() }), new SpeechTimeoutUserTurnStopStrategy({ userSpeechTimeout: 2.0 }), ], }), userTurnStrategiesParams: new UserTurnStrategiesParams({ stopMode: StrategyMode.FIRST }), }); ``` The other turn options on `LLMUserAggregatorParams`: | Option | Type | Required | Description | |---|---|---|---| | `addToolChangeMessages` | `boolean \| undefined` | no | When true, on each `LLMSetToolsFrame` the aggregator computes the diff against the currently advertised tools and appends a developer-role message to the context describing additions/removals. Helps the LLM stay coherent across mid-conversation tool changes, mitigating several flavors of tool-call-related hallucination: calling tools that have been removed, avoiding tools that have been re-added, and hallucinating output (made-up answers or tool-call-shaped non-tool-calls) when tools are unavailable. Only standard tools are diffed; custom (LLM-specific) tools are ignored. When using `LLMContextAggregatorPair`, prefer setting this via its `addToolChangeMessages` argument instead. Defaults to false. | | `audioIdleTimeout` | `number \| undefined` | no | Timeout in seconds to force speech stop when no audio frames are received while in SPEAKING state (e.g. user mutes mic mid-speech). Set to 0 to disable. Defaults to 1.0. | | `userTurnStrategies` | `UserTurnStrategies \| null \| undefined` | no | User turn start and stop strategies. | | `userTurnStrategiesParams` | `UserTurnStrategiesParams \| undefined` | no | How the user turn strategies combine with the ones a service recommends, and how they decide. Defaults to `new UserTurnStrategiesParams()`. | | `userMuteStrategies` | `BaseUserMuteStrategy[] \| undefined` | no | List of user mute strategies. | | `userTurnStopTimeout` | `number \| undefined` | no | Seconds without user activity before an open user turn is ended, unless a stop strategy has a decision scheduled. | | `userIdleTimeout` | `number \| undefined` | no | Timeout in seconds for detecting user idle state. The aggregator will emit an `on_user_turn_idle` event when the user has been idle (not speaking) for this duration. Set to 0 to disable idle detection. | | `vadAnalyzer` | `VADAnalyzer \| null \| undefined` | no | Voice Activity Detection analyzer instance. | | `emptyUserTurn` | `EmptyUserTurnConfig \| null \| undefined` | no | How to respond to a user turn that ends with no transcript. By default, the bot answers such a turn when it interrupted the bot, and leaves it unanswered otherwise. `null` leaves every such turn unanswered. Ignored with a realtime LLM service, which hears the user's audio directly. | `vadAnalyzer` produces the VAD frames; `VADUserTurnStartStrategy` is what turns them into a turn start. They are separate things, and the VAD analyzer is set on the aggregator, not the transport. `userMuteStrategies` is covered in [User muting](https://polvallverdu.github.io/pipedog/docs/concepts/user-muting.md) and `userIdleTimeout` in [Turn events & idle users](https://polvallverdu.github.io/pipedog/docs/concepts/turn-events/#idle-users). ## Start strategies [#start-strategies] Every start strategy accepts `enableInterruptions` (default `true`): whether starting a turn interrupts the bot. Set it to `false` and the bot keeps talking while the user speaks; what the user said is still transcribed, and answered after the bot finishes. | Option | Type | Required | Description | |---|---|---|---| | `enableInterruptions` | `boolean \| undefined` | no | If true, the user aggregator will emit an interruption frame when the user turn starts. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | ### VADUserTurnStartStrategy [#vaduserturnstartstrategy] Starts the turn as soon as the VAD hears speech (`VADUserStartedSpeakingFrame`). The most responsive option, and the first default. It needs a `vadAnalyzer` on the aggregator, and has no options of its own. ### TranscriptionUserTurnStartStrategy [#transcriptionuserturnstartstrategy] Starts the turn when a transcript arrives. It is the second default, a fallback for speech the VAD misses (a soft voice, for example) but the STT service still transcribes. | Option | Type | Required | Description | |---|---|---|---| | `useInterim` | `boolean \| undefined` | no | Whether interim transcriptions also start a turn. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableInterruptions` | `boolean \| undefined` | no | If true, the user aggregator will emit an interruption frame when the user turn starts. | ### MinWordsUserTurnStartStrategy [#minwordsuserturnstartstrategy] Starts the turn only once the user has said `minWords` words, so a quick "okay" or "uh-huh" doesn't cut the bot off. The minimum applies only while the bot is speaking; when the bot is silent, one word starts a turn. Shorter transcripts during bot speech are discarded and never reach the context. Use it **instead of** the VAD start strategy, which would open the turn on the first sound: ```ts const userParams = new LLMUserAggregatorParams({ vadAnalyzer: new SileroVADAnalyzer(), userTurnStrategies: new UserTurnStrategies({ // No VAD start strategy: it would open the turn on the first sound. start: [new MinWordsUserTurnStartStrategy({ minWords: 3 })], stop: [new SpeechTimeoutUserTurnStopStrategy()], }), }); ``` | Option | Type | Required | Description | |---|---|---|---| | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableInterruptions` | `boolean \| undefined` | no | If true, the user aggregator will emit an interruption frame when the user turn starts. | | `minWords` | `number` | yes | Minimum number of spoken words required to trigger the start of a user turn. | | `useInterim` | `boolean \| undefined` | no | Whether to consider interim transcription frames for earlier detection. | | `startOnVadWhenBotQuiet` | `boolean \| undefined` | no | Whether to start the turn as soon as VAD detects speech while the bot is not speaking, instead of waiting for the first transcript. Defaults to false (upstream behavior). The word count exists to keep short noises from interrupting the bot, so it only matters while the bot speaks. Waiting for a transcript when the bot is quiet starts the turn late: by then an end-of-turn analyzer may already have decided, the late start resets it, and the reply falls back to the STT timeout; a pause mid-answer can also split it in two. With this option the turn starts with the speech, as with `VADUserTurnStartStrategy`, and the minimum applies only over the bot. | | `keepBotSpeakingOnTurnStart` | `boolean \| undefined` | no | Whether to keep the bot-speaking state when a user turn starts. Defaults to false (upstream behavior: the state is cleared, so the next turn counts from one word). Clearing it assumes the bot stopped because it was interrupted. When it goes on speaking (interruptions disabled, or the turn started some other way), the strategy would then treat a single word over the bot as a new turn. With this option the state only follows `BotStartedSpeakingFrame` and `BotStoppedSpeakingFrame`. | | `isBackchannel` | `BackchannelMatcher \| null \| undefined` | no | Decides whether a transcript is a backchannel ("mm-hm", "yes, okay", "right"): the user acknowledging the bot, not taking the turn. While the bot speaks, a backchannel never starts a turn, whatever its word count. `createBackchannelMatcher()` builds one from per-language phrase lists (`BACKCHANNEL_PHRASES` by default). Defaults to null (upstream behavior). DIVERGENCE: pipedog-only. | | `requireVadWithinSeconds` | `number \| null \| undefined` | no | While the bot speaks, count a transcript's words only if VAD heard the user within this many seconds: the user is speaking, or VAD reported the end of their speech at most this long ago. Transcripts with no speech behind them (a TV in the background, STT hallucinating words from line clicks) then can't interrupt the bot. VAD already gates on volume and confidence, so tune it with `VADParams`. Requires a VAD in the pipeline. Defaults to null (no check, upstream behavior). DIVERGENCE: pipedog-only. | Four options change upstream behavior, and all are off by default: * `startOnVadWhenBotQuiet` starts the turn on `VADUserStartedSpeakingFrame` while the bot is silent, instead of waiting for the first transcript. A turn that starts with the transcript starts late: the end-of-turn analyzer may already have decided, gets reset, and the reply waits for the STT timeout. With this option the minimum only applies over the bot, and you don't need a VAD start strategy next to it. * `keepBotSpeakingOnTurnStart` keeps the bot-speaking state when a turn starts. By default it is cleared, which assumes the turn interrupted the bot; if the bot goes on speaking (interruptions disabled, say), one word would then start the next turn. * `isBackchannel` keeps acknowledgements from interrupting the bot, however many words they have: "yes, okay", "mm-hm, right". While the bot speaks, a transcript it matches is discarded like a short one. `createBackchannelMatcher()` builds a matcher from per-language phrase lists (`BACKCHANNEL_PHRASES`: English, Spanish, French, German, Italian, Portuguese and Catalan), picked by the transcript's language; pass your own lists to extend them. With the bot silent, a backchannel is an answer and starts the turn as usual. * `requireVadWithinSeconds` counts words over the bot only if the VAD heard the user: they are speaking, or the VAD reported the end of their speech at most that many seconds ago. A TV in the background, or words the STT service hallucinates from clicks, then can't interrupt. It needs a VAD in the pipeline; tune what the VAD counts as speech with `VADParams` (see `minVolumeDb` above). ```ts new MinWordsUserTurnStartStrategy({ minWords: 3, isBackchannel: createBackchannelMatcher(), requireVadWithinSeconds: 1.0, }); ``` ### WakePhraseUserTurnStartStrategy [#wakephraseuserturnstartstrategy] Ignores the user until they say a wake phrase, then lets the conversation run until `timeout` seconds pass with no activity. It starts turns on its own: on the wake phrase, and, while awake, whenever the user starts speaking or a final transcript arrives (which covers speech the VAD missed). Transcripts that don't contain the phrase while it is asleep are dropped from the aggregation. Matching ignores case and punctuation ("Hey, pipedog!" matches "hey pipedog"). Use it as the only start strategy: ```ts const wakePhrase = new WakePhraseUserTurnStartStrategy({ phrases: ["hey pipedog", "ok pipedog"], timeout: 10.0, // seconds of inactivity before the wake phrase is needed again }); wakePhrase.eventHandler("on_wake_phrase_detected", async (_strategy, phrase: string) => { console.log(`woken up by "${phrase}"`); }); wakePhrase.eventHandler("on_wake_phrase_timeout", async () => { console.log("asleep again"); }); const userParams = new LLMUserAggregatorParams({ vadAnalyzer: new SileroVADAnalyzer(), userTurnStrategies: new UserTurnStrategies({ // It starts turns on its own once awake, so it is the only start strategy. start: [wakePhrase], }), }); ``` To combine it with other start conditions, such as `MinWordsUserTurnStartStrategy`, use `startMode: StrategyMode.ALL`, where it is one more condition a turn needs: ```ts // Every start strategy has to agree: the wake phrase, and three words. const userParams = new LLMUserAggregatorParams({ vadAnalyzer: new SileroVADAnalyzer(), userTurnStrategies: new UserTurnStrategies({ start: [ new WakePhraseUserTurnStartStrategy({ phrases: ["hey pipedog"] }), new MinWordsUserTurnStartStrategy({ minWords: 3 }), ], }), userTurnStrategiesParams: new UserTurnStrategiesParams({ startMode: StrategyMode.ALL }), }); ``` | Option | Type | Required | Description | |---|---|---|---| | `phrases` | `string[]` | yes | List of wake phrases to detect. | | `timeout` | `number \| undefined` | no | Inactivity timeout in seconds before returning to IDLE. In timeout mode, the timer resets on activity. In single activation mode, acts as a keepalive window after wake phrase detection. | | `singleActivation` | `boolean \| undefined` | no | If true, the wake phrase is required before every turn. The strategy returns to IDLE after each turn completes. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableInterruptions` | `boolean \| undefined` | no | If true, the user aggregator will emit an interruption frame when the user turn starts. | It emits `on_wake_phrase_detected(strategy, phrase)` and `on_wake_phrase_timeout(strategy)`, and exposes its current `state` (`"idle"` or `"awake"`). With `singleActivation: true` the phrase is needed before every turn: activity no longer extends the window, which closes `timeout` seconds after the phrase. ### KrispVivaIPUserTurnStartStrategy [#krispvivaipuserturnstartstrategy] Uses Krisp's Interruption Prediction model to tell a real interruption from a backchannel ("uh-huh", "yeah"). When the VAD hears speech it feeds the audio to the model, and starts a turn only if the model's probability exceeds `threshold`. Pair it with `TranscriptionUserTurnStartStrategy` as a fallback. Krisp ships a native Python SDK and no JavaScript binding, so pipedog ports the strategy's logic but not the model runtime. Out of the box the constructor throws, because the Krisp SDK isn't there. To use it, register a binding with `setKrispAudioSdk()` from `@pipedog/core/audio/krisp_instance` and subclass the strategy to override `_createIpSession()`, which returns a `KrispIpSession`. | Option | Type | Required | Description | |---|---|---|---| | `modelPath` | `string \| null \| undefined` | no | Path to the Krisp VIVA IP model file (.kef). If null, uses the KRISP_VIVA_IP_MODEL_PATH environment variable. | | `threshold` | `number \| undefined` | no | IP probability threshold (0.0 to 1.0). When the model's output exceeds this value, the speech is classified as a genuine interruption. | | `frameDurationMs` | `number \| undefined` | no | Frame duration in milliseconds for IP processing. Supported values: 10, 15, 20, 30, 32. | | `apiKey` | `string \| undefined` | no | Krisp SDK API key. If empty, falls back to the KRISP_VIVA_API_KEY environment variable. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableInterruptions` | `boolean \| undefined` | no | If true, the user aggregator will emit an interruption frame when the user turn starts. | `modelPath` must end in `.kef`. It and `apiKey` fall back to the `KRISP_VIVA_IP_MODEL_PATH` and `KRISP_VIVA_API_KEY` environment variables. ### ExternalUserTurnStartStrategy [#externaluserturnstartstrategy] Lets another component decide when the turn starts. It understands two frames: * `ProposedUserStartedSpeakingFrame`: a service with its own turn detection proposes a turn start. This strategy accepts it, pushes the `UserStartedSpeakingFrame` and interrupts the bot (if `enableInterruptions`). Subclass it to change when a proposal opens a turn. * `UserStartedSpeakingFrame`: the turn was already decided and announced elsewhere, for example by a shared `UserTurnProcessor` feeding several aggregators. This strategy adopts it and emits nothing, so the turn isn't announced twice. | Option | Type | Required | Description | |---|---|---|---| | `enableInterruptions` | `boolean \| undefined` | no | If true, broadcast an interruption when a proposal opens a turn. Ignored on the adopt path, where the emitter has already broadcast one. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | ## Stop strategies [#stop-strategies] ### TurnAnalyzerUserTurnStopStrategy [#turnanalyzeruserturnstopstrategy] The default. Feeds audio, VAD and transcription frames to a turn analyzer, Smart Turn v3 here, which predicts from the audio whether the user has finished or only paused. Once it says the turn is complete, the strategy waits for the transcript: it ends the turn on a finalized transcript, or when the STT service's p99 latency has passed since the end of speech. ```ts const stop = new TurnAnalyzerUserTurnStopStrategy({ turnAnalyzer: createLocalSmartTurnAnalyzerV3({ cpuCount: 2 }), }); ``` | Option | Type | Required | Description | |---|---|---|---| | `turnAnalyzer` | `BaseTurnAnalyzer` | yes | The turn detection analyzer instance to detect end of user turn. | | `waitForTranscript` | `boolean \| undefined` | no | Whether to require a transcript before triggering end-of-turn. When true (default), turn-end fires only after the turn analyzer reports COMPLETE *and* either a finalized transcript arrives or the STT safety-net timeout elapses with text in hand. When false, the strategy signals turn-end as soon as the turn analyzer reports COMPLETE — independent of transcripts. Set this to false when local turn detection is the intended driver of the conversation (e.g. with a realtime LLM service consuming audio directly), so transcripts are off the latency critical path. `LLMContextAggregatorPair` flips this for you when `realtimeServiceMode: true`. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | `createLocalSmartTurnAnalyzerV3()` from `@pipedog/audio-smart-turn` builds the analyzer with the ONNX runtime and the model shipped with that package. Its options: | Option | Type | Required | Description | |---|---|---|---| | `model` | `OnnxModelSource \| undefined` | no | The Smart Turn model: bytes, a path or a URL. Defaults to `smartTurnModelPath` if set, else the model shipped with this package. Required on Cloudflare Workers. | | `backend` | `OnnxBackend \| Promise \| undefined` | no | The ONNX backend (default: `createOnnxBackend()`). | | `shareSession` | `boolean \| undefined` | no | Share the inference session across analyzers (default `true`). | | `sampleRate` | `number \| null \| undefined` | no | Optional initial sample rate for audio processing. If provided, this will be used as the fixed sample rate. | | `params` | `SmartTurnParams \| null \| undefined` | no | Configuration parameters for turn analysis behavior. | | `smartTurnModelPath` | `string \| null \| undefined` | no | Path/URL to the ONNX model file. If not set, the upstream bundled model name (smart-turn-v3.2-cpu.onnx) is used. | | `cpuCount` | `number \| undefined` | no | The number of CPUs to use for inference. Defaults to 1. | `params` takes a `SmartTurnParams` (`@pipedog/core/audio/turn/smart_turn/base_smart_turn`): `stopSecs` (3 s, the longest silence before the analyzer gives up waiting and calls the turn complete), `preSpeechMs` (500 ms of audio kept from before speech starts) and `maxDurationSecs` (8 s, the longest segment the model sees). `waitForTranscript: false` ends the turn as soon as the model says so, without waiting for text. Use it when the turn model drives a speech-to-speech service that hears the audio itself; `LLMContextAggregatorPair` sets it for you in realtime mode. ### SpeechTimeoutUserTurnStopStrategy [#speechtimeoutuserturnstopstrategy] Ends the turn after a fixed silence. After the VAD reports the end of speech it runs two timers, and ends the turn when both have finished and a transcript has arrived: * `userSpeechTimeout` (0.6 s): how long the user may pause and carry on. It always runs in full, and is the knob for responsiveness. * The STT timer: the service's p99 time to a final transcript. A finalized transcript ends it early. If the service finalizes its transcripts, the turn ends `userSpeechTimeout` after the VAD stop. If a transcript arrives without any VAD stop (no VAD analyzer, for instance), `userSpeechTimeout` measures the silence since the last transcript instead. ```ts const stop = new SpeechTimeoutUserTurnStopStrategy({ userSpeechTimeout: 0.8, // wait a little longer before answering }); ``` | Option | Type | Required | Description | |---|---|---|---| | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `userSpeechTimeout` | `number \| undefined` | no | Time to wait for the user to potentially say more after they pause speaking. Defaults to 0.6 seconds. | | `waitForTranscript` | `boolean \| undefined` | no | Whether to require at least one transcript before triggering end-of-turn. When true (default), turn-end fires only after the user-speech timer expires *and* at least one transcript has been received. When false, the strategy signals turn-end as soon as VAD reports end of speech and the user-speech timer has elapsed — independent of transcripts. Set this to false when local turn detection is the intended driver of the conversation (e.g. with a realtime LLM service consuming audio directly), so transcripts are off the latency critical path. `LLMContextAggregatorPair` flips this for you when `realtimeServiceMode: true`. | ### ExternalUserTurnStopStrategy [#externaluserturnstopstrategy] The stop-side counterpart of `ExternalUserTurnStartStrategy`. On a `ProposedUserStoppedSpeakingFrame` it decides, and pushes the `UserStoppedSpeakingFrame` itself, holding the turn open until the transcript arrives if `waitForTranscript`. On a `UserStoppedSpeakingFrame` decided elsewhere it adopts the decision and emits nothing. It decides once per round: again only after new text, new speech from the service, or speech the VAD heard. To shift the timing, subclass it and override `reportUserTurnStopped()`, which both paths reach once they decide; pass its options through when the override reports: ```ts /** Waits a grace period after the service says the user stopped. */ export class GracePeriodUserTurnStopStrategy extends ExternalUserTurnStopStrategy { #pending: Task | null = null; /** Cancel a pending report when the user starts speaking again. */ override async processFrame(frame: Frame): Promise { if (frame instanceof ProposedUserStartedSpeakingFrame) { await this.#cancelPending(); } await super.processFrame(frame); } // The base strategy calls this to report that the user is done, so this is // where a subclass adjusts the timing. override async reportUserTurnStopped(options: ReportUserTurnStoppedOptions = {}): Promise { if (this.#pending) return; this.#pending = this.createTask(async () => { await sleep(0.5); this.#pending = null; await super.reportUserTurnStopped(options); }, `${this}::grace_period`); } override async handleUserTurnStopped(): Promise { await this.#cancelPending(); await super.handleUserTurnStopped(); } override async cleanup(): Promise { await this.#cancelPending(); await super.cleanup(); } async #cancelPending(): Promise { if (this.#pending) { const task = this.#pending; this.#pending = null; await this.cancelTask(task); } } } ``` | Option | Type | Required | Description | |---|---|---|---| | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `timeout` | `number \| undefined` | no | How long (seconds) to wait after a final transcription for more of them, so a turn whose transcription arrives late or in pieces is decided once the text has settled. | | `waitForTranscript` | `boolean \| undefined` | no | When true (default), turn-stop signaling waits for transcript text to arrive after the external stop signal. When false, the strategy signals turn-stop as soon as that signal arrives — independent of transcripts. Set this to false when local turn detection is the intended driver of the conversation (e.g. with a realtime LLM service consuming audio directly), so transcripts are off the latency critical path. `LLMContextAggregatorPair` flips this for you when `realtimeServiceMode: true`. | ### EagerUserTurnStopStrategy [#eageruserturnstopstrategy] For STT services that predict the end of a turn before committing to it. It starts generating a response on the prediction (an `EagerTranscriptionFrame`), so the gap before the committed end of turn is spent generating instead of waiting. The response is held back by the LLM service's speculation gate and discarded if the user resumes speaking, if the committed transcript doesn't match the predicted one, or if the prediction goes unresolved for `speculationTimeout` seconds. Nothing from a discarded speculation reaches the user or the context, and the context always records the committed transcript. It extends `ExternalUserTurnStopStrategy`. Install it with [`EagerUserTurnStrategies`](#eageruserturnstrategies) rather than directly. | Option | Type | Required | Description | |---|---|---|---| | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `matchPolicy` | `EagerMatchPolicy \| null \| undefined` | no | Decides whether the committed transcript is close enough to the eager one to keep the speculative response. Defaults to `NormalizedMatch`, which ignores the capitalization and punctuation services commonly add when they commit a transcript. Pass `ExactMatch` to require the two to be identical. | | `speculationTimeout` | `number \| undefined` | no | Seconds a prediction may go unresolved before it is withdrawn. A service that stops sending turn signals mid-speculation would otherwise leave the response held and the bot silent for the rest of the session. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `timeout` | `number \| undefined` | no | How long (seconds) to wait after a final transcription for more of them, so a turn whose transcription arrives late or in pieces is decided once the text has settled. | | `waitForTranscript` | `boolean \| undefined` | no | When true (default), turn-stop signaling waits for transcript text to arrive after the external stop signal. When false, the strategy signals turn-stop as soon as that signal arrives — independent of transcripts. Set this to false when local turn detection is the intended driver of the conversation (e.g. with a realtime LLM service consuming audio directly), so transcripts are off the latency critical path. `LLMContextAggregatorPair` flips this for you when `realtimeServiceMode: true`. | `matchPolicy` is a `NormalizedMatch` by default, which ignores case, punctuation and apostrophes, or an `ExactMatch`; subclass `EagerMatchPolicy` and implement `matches(eager, final)` for your own rule. ### LLMTurnCompletionUserTurnStopStrategy [#llmturncompletionuserturnstopstrategy] Decides the user is done only when the LLM says so. At start it enables `filterIncompleteUserTurns` on the LLM (on every LLM behind an `LLMSwitcher`, too), which makes the LLM begin each response with a marker: `●` complete, `◐` cut off mid-thought, `○` needs more time. The LLM service reports each marker upstream as a `UserTurnCompletionFrame`. It **waits for the turn's inference** (`waitsForInference`) instead of listening to the user, so it needs other stop strategies next to it: once the strategies that listen to the user have decided, the controller runs the inference, and this strategy decides on its marker. `●`, a tool call, or a response with no marker end the turn. On `◐` or `○` the turn stays open; if the user then stays quiet for `incompleteShortTimeout` or `incompleteLongTimeout`, it ends the turn and asks the LLM to re-engage them. This is what [`FilterIncompleteUserTurnStrategies`](#filterincompleteuserturnstrategies) installs. | Option | Type | Required | Description | |---|---|---|---| | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `config` | `UserTurnCompletionConfig \| null \| undefined` | no | Markers, instructions, timeouts and re-prompts applied to the LLM. Defaults to `new UserTurnCompletionConfig()`. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | ### Strategies that wait for the inference [#strategies-that-wait-for-the-inference] The controller runs the turn's inference itself. In a `StrategyMode.ALL` list it runs it once every stop strategy that listens to the user has decided; a stop strategy whose `waitsForInference` is `true` then decides on that inference, and the turn ends. In a `StrategyMode.FIRST` list the first decision runs the inference, and the turn ends on the decision of the strategy that waits for it. A list where every stop strategy waits for the inference has nothing to run it but the stop timeout, and logs a warning. ```ts // Smart Turn decides the user stopped, which runs the inference; the LLM's // ● marker then ends the turn. const strategies = new FilterIncompleteUserTurnStrategies({ config: new UserTurnCompletionConfig({ incompleteShortTimeout: 5.0, incompleteLongTimeout: 10.0, }), }); // The same strategies, built by hand: const byHand = new UserTurnStrategies({ stop: [ new TurnAnalyzerUserTurnStopStrategy({ turnAnalyzer: createLocalSmartTurnAnalyzerV3() }), new LLMTurnCompletionUserTurnStopStrategy(), ], }); // On top of the turn detection an STT service recommends: const extending = new LLMUserAggregatorParams({ userTurnStrategies: new FilterIncompleteUserTurnStrategies(), userTurnStrategiesParams: new UserTurnStrategiesParams({ stopMergeMode: MergeMode.EXTEND }), }); ``` ## Presets [#presets] ### ExternalUserTurnStrategies [#externaluserturnstrategies] An `ExternalUserTurnStartStrategy` and an `ExternalUserTurnStopStrategy`, for turns decided by a service with its own turn detection, a speech-to-speech service or a shared `UserTurnProcessor`. Services with turn detection (`DeepgramFluxSTTService`, `CartesiaTurnsSTTService`, `OpenAIRealtimeSTTService` with server VAD) recommend it to the aggregator on their own when you don't pass `userTurnStrategies`. | Option | Type | Required | Description | |---|---|---|---| | `enableInterruptions` | `boolean \| undefined` | no | Whether to broadcast an interruption when a proposal starts a turn. Services route their `should_interrupt` setting here. | | `start` | `BaseUserTurnStartStrategy[] \| null \| undefined` | no | A list of user turn start strategies used to detect when the user starts speaking. | | `stop` | `BaseUserTurnStopStrategy[] \| null \| undefined` | no | A list of user turn stop strategies used to decide when the user stops speaking. | A service routes its `shouldInterrupt` option to the strategies it recommends. If you set `shouldInterrupt: false` on the service but also pass `new ExternalUserTurnStrategies()` yourself, interruptions are back on. Leave `userTurnStrategies` out, or pass `enableInterruptions: false`. A VAD analyzer on the aggregator is optional here: it adds STT metrics but isn't needed for turns. ### EagerUserTurnStrategies [#eageruserturnstrategies] `ExternalUserTurnStrategies` with an [`EagerUserTurnStopStrategy`](#eageruserturnstopstrategy) as the only stop strategy. The service owns turn detection, so this replaces the detector chain rather than extending it: a local detector alongside would run a second inference for the same turn. ```ts const strategies = new EagerUserTurnStrategies({ matchPolicy: new ExactMatch(), // default: NormalizedMatch speculationTimeout: 5.0, }); ``` | Option | Type | Required | Description | |---|---|---|---| | `matchPolicy` | `EagerMatchPolicy \| null \| undefined` | no | Decides whether the committed transcript is close enough to the eager one to keep the speculative response. Defaults to NormalizedMatch. | | `speculationTimeout` | `number \| undefined` | no | Seconds a prediction may go unresolved before it is withdrawn and the turn falls back to answering the committed transcript. | | `start` | `BaseUserTurnStartStrategy[] \| null \| undefined` | no | A list of user turn start strategies used to detect when the user starts speaking. | | `stop` | `BaseUserTurnStopStrategy[] \| null \| undefined` | no | A list of user turn stop strategies used to decide when the user stops speaking. | | `enableInterruptions` | `boolean \| undefined` | no | Whether to broadcast an interruption when a proposal starts a turn. Services route their `should_interrupt` setting here. | STT services that predict turns take `enableEagerEndOfTurn: true` (off by default, since every prediction costs an inference, including the ones the service withdraws) and recommend this preset themselves. ### FilterIncompleteUserTurnStrategies [#filterincompleteuserturnstrategies] Keeps the start strategies and the stop strategies (the defaults, or yours), and adds an `LLMTurnCompletionUserTurnStopStrategy`. Once the other stop strategies decide the user stopped, the inference runs, and `on_user_turn_stopped` fires once the LLM marks the turn complete. It works with either `stopMode`, and with an eager end of turn: the speculative answer goes out once the transcript commits and the LLM marks the turn complete, in either order, and `◐` or `○` withdraws it. With `stopMergeMode: MergeMode.EXTEND`, the LLM gate (and any stop strategies you gave) joins the ones a service recommends instead. | Option | Type | Required | Description | |---|---|---|---| | `config` | `UserTurnCompletionConfig \| null \| undefined` | no | Optional configuration applied to the LLM via the `filter_incomplete_user_turns` setting. Customizes the turn-completion instructions, incomplete-turn timeouts, and re-prompts. If null, defaults from UserTurnCompletionConfig are used. | | `start` | `BaseUserTurnStartStrategy[] \| null \| undefined` | no | A list of user turn start strategies used to detect when the user starts speaking. | | `stop` | `BaseUserTurnStopStrategy[] \| null \| undefined` | no | A list of user turn stop strategies used to decide when the user stops speaking. | | Option | Type | Required | Description | |---|---|---|---| | `instructions` | `string \| null \| undefined` | no | Custom instructions for turn completion. If not provided, the default instructions are rendered from the configured markers. | | `completeMarker` | `string \| undefined` | no | Marker the LLM emits when the user turn is complete. It is generated before any speakable text, so prefer a character that is a single token in the model's tokenizer. | | `incompleteShortMarker` | `string \| undefined` | no | Marker for a turn cut off mid-thought. | | `incompleteLongMarker` | `string \| undefined` | no | Marker for a user who needs more time. | | `incompleteShortTimeout` | `number \| undefined` | no | Seconds to wait after a short incomplete before prompting. | | `incompleteLongTimeout` | `number \| undefined` | no | Seconds to wait after a long incomplete before prompting. | | `incompleteShortPrompt` | `string \| null \| undefined` | no | Custom prompt when short timeout expires. | | `incompleteLongPrompt` | `string \| null \| undefined` | no | Custom prompt when long timeout expires. | ## Turns with no transcript [#turns-with-no-transcript] A turn can start on the VAD and end without a transcript: a cough, background noise, speech the STT service couldn't make out. Nothing is written to the context, and by default: * if the turn **interrupted the bot** (or came before the bot first finished speaking), the LLM runs once with a developer message, so the bot asks the user to repeat or picks up where it left off; * if the bot **was waiting** for the user, the turn goes unanswered. `emptyUserTurn` changes this; `null` leaves every such turn unanswered. ```ts const userParams = new LLMUserAggregatorParams({ vadAnalyzer: new SileroVADAnalyzer(), emptyUserTurn: new EmptyUserTurnConfig({ idlePrompt: "The user made a sound but said nothing clear. Ask if they said something.", maxConsecutiveRecoveries: 2, }), }); ``` | Option | Type | Required | Description | |---|---|---|---| | `interruptedPrompt` | `string \| null \| undefined` | no | Developer message for an empty turn that interrupted the bot. `null` leaves such turns unanswered. | | `idlePrompt` | `string \| null \| undefined` | no | Developer message for an empty turn while the bot was idle. `null` (the default) leaves such turns unanswered. | | `maxConsecutiveRecoveries` | `number \| undefined` | no | How many empty turns in a row are answered. Further ones are left unanswered until the user says something that is transcribed. | Recovery is skipped while the user is muted or a function call is pending, and with a realtime LLM service, which hears the audio itself. ## Writing a strategy [#writing-a-strategy] Subclass `BaseUserTurnStartStrategy` or `BaseUserTurnStopStrategy` and override `processFrame()`. It sees every frame that reaches the user aggregator and returns nothing. When the strategy decides, it reports the decision, and the turn controller decides when the turn starts or ends: * start strategies: `reportUserTurnStarted()`, or `triggerResetAggregation()` to drop the text gathered so far (what `MinWordsUserTurnStartStrategy` does with a too-short transcript); * stop strategies: `reportUserTurnStopped()` when the user is done. The controller runs the turn's inference itself. A strategy starts one only to answer ahead of the turn end, with `startUserTurnSpeculation()`, and withdraws it with `cancelUserTurnSpeculation()`. ```ts /** Decides the user started on a final transcript that ends with a question mark. */ export class QuestionUserTurnStartStrategy extends BaseUserTurnStartStrategy { override async processFrame(frame: Frame): Promise { if (frame instanceof TranscriptionFrame && frame.text.trim().endsWith("?")) { // The turn controller starts the turn on this decision (StrategyMode.FIRST), // or once every start strategy has decided (StrategyMode.ALL). await this.reportUserTurnStarted(); } } } ``` A stop strategy that decides from the turn's answer rather than from the user returns `true` from `waitsForInference`, and one that is sure to decide soon on its own (a timer of its own) returns `true` from `decisionScheduled` meanwhile, so the stop timeout waits for it. Override `handleUserTurnStarted()` and `handleUserTurnStopped()` to reset per-turn state; the controller calls them on every strategy at each turn boundary, whichever strategy caused it. `setup()` and `cleanup()` run with the aggregator, and `this.taskManager` creates tasks for timers. # Typed tools > Define LLM function calls with a Zod, Valibot or ArkType schema (Standard Schema) and get the JSON schema, inferred handler types and argument validation. Source: https://polvallverdu.github.io/pipedog/docs/concepts/typed-tools/ `defineTool` builds a function-calling tool from a [Standard Schema](https://standardschema.dev) object: Zod, Valibot, ArkType and any other library that implements the spec. One definition gives you: * the JSON schema the LLM needs, generated from the schema; * a handler whose `args` are typed from it; * validation of the arguments the LLM sends, before your handler runs. This is pipedog's own addition, not part of Pipecat. It sits next to the ported ways to define tools (`FunctionSchema`, direct functions, the `tool` decorator), which keep working unchanged. Unlike direct functions, it never reads parameter names from function source, so it survives minification. ```ts import { defineTool, registerTools } from "@pipedog/core/adapters/llm/schemas/standard_tool"; ``` ## Define a tool [#define-a-tool] ```ts const getWeather = defineTool({ name: "get_weather", description: "Get the current weather for a location.", parameters: z.object({ location: z.string().describe("City name, e.g. 'Paris'"), unit: z.enum(["celsius", "fahrenheit"]).default("celsius"), }), // `args` is { location: string; unit: "celsius" | "fahrenheit" }, already validated. handler: async ({ location, unit }) => { return { location, unit, temperature: 21, conditions: "sunny" }; }, }); ``` `defineTool` returns a `FunctionSchema` that carries the handler, so it works with every provider adapter. The handler's return value is sent to the LLM as the function result. To report the result yourself, for example with `FunctionCallResultProperties`, call `params.resultCallback` (the second handler argument); the return value is then ignored. Pass the tool's call options (`cancelOnInterruption`, `timeoutSecs`, `cancellableByLlm`) to `defineTool` directly, as you would to `toolOptions`. ## Register it [#register-it] Advertise the tool in a `ToolsSchema`. The LLM service registers its handler automatically: ```ts // Advertising the tool registers its handler on the LLM service automatically. const context = new LLMContext({ tools: new ToolsSchema({ standardTools: [getWeather] }), }); ``` Or register explicitly: ```ts // Or register explicitly, with `registerFunction` under the hood. registerTools(llm, [getWeather]); ``` ## Invalid arguments [#invalid-arguments] The arguments are checked with the schema's `~standard.validate`. If they don't pass, your handler is **not** called. A warning is logged, and the issues go back to the LLM as the function result so it can retry: ```json { "error": "Invalid arguments. Fix the issues and call the function again.", "issues": [{ "message": "Invalid input: expected string, received undefined", "path": "location" }] } ``` Valid arguments reach the handler as the schema's output, so defaults and transforms are applied. ## Where the JSON schema comes from [#where-the-json-schema-comes-from] `defineTool` asks the schema for it, using the companion spec [Standard JSON Schema](https://standardschema.dev/json-schema) (`~standard.jsonSchema.input`, target `draft-07`; change it with `target`). The LLM supplies the input side, so a field with a default is optional in the schema it sees. Zod 4.2+ and ArkType 2.1.28+ implement it on their schemas; Valibot does through `toStandardJsonSchema` from `@valibot/to-json-schema`. A schema that doesn't implement it needs a `jsonSchema` option: the JSON schema object, or a converter function. ```ts const search = defineTool({ name: "search", description: "Search the knowledge base.", parameters: v.object({ query: v.string() }), // This schema doesn't provide a JSON schema itself: pass a converter (or the object). jsonSchema: (schema) => toJsonSchema(schema) as Record, handler: async ({ query }) => ({ hits: [query] }), }); ``` `defineTool` throws when there is no JSON schema, or when it isn't an object schema. `$ref`s into `$defs` are inlined; recursive schemas are rejected. A `FunctionSchema` carries only `properties` and `required`, so other top-level keywords such as `additionalProperties` are not sent to the LLM. ```ts // The handler's argument types are inferred; the JSON schema the LLM sees comes from the schema too. const schemaForLlm = getWeather.toDefaultDict(); ``` # User muting > Ignore the user's audio and transcripts while the bot greets them, runs a tool, or speaks, with built-in or custom mute strategies. Source: https://polvallverdu.github.io/pipedog/docs/concepts/user-muting/ **User mute strategies** decide when the user aggregator ignores the user. While the user is muted, their audio, VAD events, transcripts and interruptions are dropped at the aggregator: the bot can't be interrupted, and whatever the user says meanwhile never reaches the context. The user is not muted by default. ```ts import { AlwaysUserMuteStrategy, BaseUserMuteStrategy, EndingUserMuteStrategy, FirstSpeechUserMuteStrategy, FunctionCallUserMuteStrategy, MuteUntilFirstBotCompleteUserMuteStrategy, OpeningUserMuteStrategy, } from "@pipedog/core/turns/user_mute/index"; ``` ## Configuration [#configuration] Pass the strategies as `userMuteStrategies` on `LLMUserAggregatorParams`: ```ts const aggregators = new LLMContextAggregatorPair(context, { userParams: new LLMUserAggregatorParams({ vadAnalyzer: new SileroVADAnalyzer(), userMuteStrategies: [ // Let the greeting play out in full... new MuteUntilFirstBotCompleteUserMuteStrategy(), // ...and ignore the user while a tool runs. new FunctionCallUserMuteStrategy(), ], }), }); ``` Every strategy sees every frame, and the user is muted while **any** of them says so. The example above mutes from the start of the session until the bot has finished its greeting, and again whenever a function call runs. ## Built-in strategies [#built-in-strategies] | Strategy | Mutes | Use it to | | ------------------------------------------- | ------------------------------------------------------------------------------------ | ------------------------------------------------------------- | | `MuteUntilFirstBotCompleteUserMuteStrategy` | from the start until the bot first finishes speaking | protect the greeting, including the silence before it | | `OpeningUserMuteStrategy` | from the start until the bot first finishes speaking, or has spoken for `maxSeconds` | protect the start of a long greeting without muting all of it | | `FirstSpeechUserMuteStrategy` | during the bot's first speech only | protect the greeting, but let the user speak before it starts | | `FunctionCallUserMuteStrategy` | while function calls run | keep the user from talking over a tool call | | `AlwaysUserMuteStrategy` | whenever the bot speaks | turn off barge-in completely | | `EndingUserMuteStrategy` | for good, once the conversation is ending | let the bot finish its goodbye | `OpeningUserMuteStrategy` and `EndingUserMuteStrategy` are pipedog additions; the others are upstream's and take no options. ### MuteUntilFirstBotCompleteUserMuteStrategy [#muteuntilfirstbotcompleteusermutestrategy] Mutes from the first frame, before the bot has said anything, and unmutes for good on the first `BotStoppedSpeakingFrame`. If the bot's first turn fails before it speaks (an `ErrorFrame` arrives before any `BotStartedSpeakingFrame`), it unmutes and logs a warning, so a failed greeting doesn't leave the user unable to talk. ### OpeningUserMuteStrategy [#openingusermutestrategy] Like `MuteUntilFirstBotCompleteUserMuteStrategy`, but the first speech is only protected for `maxSeconds` from its `BotStartedSpeakingFrame`. On a phone call, that keeps the "hello?" of picking up from cutting the greeting or reaching the LLM as an answer, while the user can still talk over the rest of a long greeting. The window is checked as frames arrive (input audio keeps them coming). Pass `clock` to control time in tests. ```ts new OpeningUserMuteStrategy({ maxSeconds: 3 }); ``` ### FirstSpeechUserMuteStrategy [#firstspeechusermutestrategy] Lets the user speak until the bot starts speaking, mutes them during that first speech, and unmutes for good when it ends. Later speech can be interrupted as usual. ### FunctionCallUserMuteStrategy [#functioncallusermutestrategy] Mutes on `FunctionCallsStartedFrame` and tracks each call by its tool call ID; it unmutes once every call has a `FunctionCallResultFrame` or a `FunctionCallCancelFrame`. ### EndingUserMuteStrategy [#endingusermutestrategy] Mutes, and stays muted, once it sees an `EndWorkerFrame` or `StopWorkerFrame`, once its `isEnding()` option returns true, or after `end()` is called. Strategies never see `EndFrame`, and an `EndFrame` queued after a goodbye arrives once the goodbye has been queued, so when the end is decided elsewhere (a tool that queues a goodbye and an `EndFrame`, say) tell the strategy with `isEnding` or `end()`: ```ts const ending = new EndingUserMuteStrategy(); // In the tool that ends the call: ending.end(); await worker.queueFrames([new TTSSpeakFrame({ text: "Goodbye!" }), new EndFrame()]); ``` ### AlwaysUserMuteStrategy [#alwaysusermutestrategy] Mutes on `BotStartedSpeakingFrame` and unmutes on `BotStoppedSpeakingFrame`, for the whole session: ```ts const userParams = new LLMUserAggregatorParams({ vadAnalyzer: new SileroVADAnalyzer(), userMuteStrategies: [new AlwaysUserMuteStrategy()], }); ``` ## What muting drops [#what-muting-drops] While muted, the user aggregator drops these frames instead of processing or forwarding them: `InputAudioRawFrame`, `VADUserStartedSpeakingFrame`, `VADUserStoppedSpeakingFrame`, `UserStartedSpeakingFrame`, `UserStoppedSpeakingFrame`, the `Proposed...` variants of those two, `InterimTranscriptionFrame`, `TranscriptionFrame`, `EagerTranscriptionFrame`, `UserBackchannelFrame` and `InterruptionFrame`. `StartFrame`, `EndFrame` and `CancelFrame` always pass, and are never handed to the strategies. The mute state is updated after each frame: the frame that makes a strategy mute (a `BotStartedSpeakingFrame`, say) still passes, and muting starts with the next one. Muting happens in the user aggregator, so processors before it (the STT service, for example) still receive the audio. To stop the speech-to-text service itself, push an `STTMuteFrame`. ## Muting and interruptions [#muting-and-interruptions] Muting and disabling interruptions both keep the bot talking, but they treat the user's words differently: * **Muted:** what the user says over the bot is discarded. It is never transcribed into the context, and the bot never answers it. * **Interruptions disabled** (`enableInterruptions: false` on the start strategies, see [Interruptions](https://polvallverdu.github.io/pipedog/docs/concepts/interruptions-and-cancellation/#controlling-interruptions)): the user's turn still starts and is transcribed, and the bot answers it once it has finished speaking. Use a mute strategy when the bot must finish and anything said over it should be ignored, and `enableInterruptions: false` when the bot should finish but still hear the user out. ## Events [#events] The user aggregator emits `on_user_mute_started` when the user becomes muted and `on_user_mute_stopped` when they are unmuted, whichever strategy caused it, and broadcasts a `UserMuteStartedFrame` or `UserMuteStoppedFrame` through the pipeline at the same moment: ```ts const user = aggregators.user(); user.eventHandler("on_user_mute_started", async () => { console.log("user muted"); }); user.eventHandler("on_user_mute_stopped", async () => { console.log("user unmuted"); }); ``` ## Writing a strategy [#writing-a-strategy] Subclass `BaseUserMuteStrategy` (`@pipedog/core/turns/user_mute/base_user_mute_strategy`) and override `processFrame()`. Unlike turn strategies, which report their decisions through events, a mute strategy returns a boolean, `true` to mute. The user aggregator calls it with every frame it receives (except the lifecycle frames above), so it sees the frames that flow downstream to the aggregator and those broadcast upstream to it, such as the bot speaking frames, the function call frames and errors. Frames that only travel downstream after the aggregator, like the LLM's response frames, never reach it. This one mutes the user while the bot speaks, but only when the application asks for it, such as before reading terms that must be heard in full: ```ts /** * Mutes the user while the bot speaks, but only while `protecting` is on. * Turn it on before the bot reads something that must be heard in full. */ export class ProtectedSpeechUserMuteStrategy extends BaseUserMuteStrategy { protecting = false; protected _botSpeaking = false; override async processFrame(frame: Frame): Promise { await super.processFrame(frame); if (frame instanceof BotStartedSpeakingFrame) { this._botSpeaking = true; } else if (frame instanceof BotStoppedSpeakingFrame) { this._botSpeaking = false; this.protecting = false; // one protected utterance at a time } // true mutes the user, starting with the next frame. return this.protecting && this._botSpeaking; } } const protectedSpeech = new ProtectedSpeechUserMuteStrategy(); const userParams = new LLMUserAggregatorParams({ vadAnalyzer: new SileroVADAnalyzer(), userMuteStrategies: [protectedSpeech], }); // Later, before the bot reads the terms: protectedSpeech.protecting = true; await worker.queueFrame(new TTSSpeakFrame({ text: "Before we continue, please listen..." })); ``` Mute strategies are `BaseObject`s, so override `setup()` and `cleanup()` (calling `super`) if the strategy needs tasks or resources; the aggregator calls both. # Porting workflow > How pipedog tracks upstream Pipecat, picks the next work, ports modules and tests, and moves to a new upstream commit. Source: https://polvallverdu.github.io/pipedog/docs/contributing/ pipedog is ported from a **pinned upstream commit** of Pipecat, module by module and test by test. Everything about that process is tracked in the repository so humans and porting agents can share the work. Read `docs/CONVENTIONS.md` before porting or reviewing anything. The [Coming from Pipecat](https://polvallverdu.github.io/pipedog/docs/guides/coming-from-pipecat.md) page summarizes it. ## The pieces [#the-pieces] | Path | What | | ----------------------------- | ----------------------------------------------------------------------------------------------- | | `upstream.json` | The upstream repository and pinned commit. | | `.upstream/pipecat` | A checkout of that commit (`bun run upstream` restores it). | | `porting/manifest.json` | Every upstream module and test: target package and TS path, dependencies, layer, status, notes. | | `porting/STATUS.md` | Generated progress report ([rendered here](https://polvallverdu.github.io/pipedog/docs/contributing/status.md)). Never edited by hand. | | `tools/port/graph.py` | Builds the manifest from the upstream checkout. | | `tools/port/next.ts` | Selects the next work and packs it into work units. | | `tools/port/status.ts` | Changes a status and regenerates `STATUS.md`. | | `tools/port/check-headers.ts` | Checks license headers (part of `bun run lint`). | | `tools/compat/` | Runtime compatibility harness (Bun, Node, Deno, workerd). | | `.claude/workflows/port.js` | The `port` workflow: batch porting with agents. | ## The manifest [#the-manifest] `bun run port:plan` (`python3 -I tools/port/graph.py`) walks the upstream checkout and writes `porting/manifest.json`. For every upstream module it records: * `upstream` (e.g. `src/pipecat/processors/frame_processor.py`), `package` and `ts` (the target file, at the same path with the same snake_case name); * `deps`, its pipecat imports, and a topological `layer`: modules in the same layer can be ported in parallel; * the upstream `tests` that exercise it and its third-party imports, which decide whether it can live in core or needs its own package; * `status` (`todo`, `in_progress`, `ported`, `tested`, `skipped`), `portedAt` (the upstream commit it was ported from) and `notes` (deprecated APIs dropped, divergences, missing dependencies). Existing statuses and notes are preserved, so re-running after an upstream bump only refreshes the analysis. Change a status with the status tool, never by hand-editing: ```bash bun tools/port/status.ts --set [notes] bun run port:status # regenerate porting/STATUS.md ``` ## Picking the next work [#picking-the-next-work] `tools/port/next.ts` prints JSON work units: ```bash # Modules, packed into units of related files (same directory first) up to --max-lines of Python bun tools/port/next.ts modules --mvp bun tools/port/next.ts modules --package @pipedog/deepgram --max-lines 3500 bun tools/port/next.ts modules --modules pipecat.services.deepgram.stt # Upstream test files whose imported modules (and their dependencies) are all ported bun tools/port/next.ts tests --package @pipedog/core --limit 40 ``` ## The `port` workflow [#the-port-workflow] `.claude/workflows/port.js` runs a whole batch with Claude Code agents. Its phases: 1. **Plan**: pack pending modules into work units (`tools/port/next.ts`). 2. **Port**: one porter per unit, Python to TypeScript per `docs/CONVENTIONS.md`. Porters import modules other agents are porting concurrently by their deterministic TS path, and read the Python source of a dependency to learn its API rather than inventing one. 3. **Review**: an adversarial parity review of each unit against upstream, fixing divergences. 4. **Integrate**: typecheck and lint the package, fixing cross-unit mismatches. 5. **Tests**: port the upstream tests that are now ready, 1:1, and make them pass. 6. **Record**: update `porting/manifest.json` and `STATUS.md`. Arguments select the work, for example `{ "select": "--package @pipedog/deepgram" }`, or `{ "select": "--mvp" }` (the default). Other options are `maxLines`, `review`, `tests`, `testLimit` and `testsOnly`. ## Porting by hand [#porting-by-hand] 1. Pick a module (`bun tools/port/next.ts modules ...`) and mark it `in_progress`. 2. Create the TS file at its manifest `ts` path, starting with the license header: ```ts keep-header // Copyright (c) 2024-2026, Daily // Copyright (c) 2026, pipedog contributors // SPDX-License-Identifier: BSD-2-Clause // // Ported from pipecat: src/pipecat/.py ``` Files with no upstream counterpart omit the Daily and "Ported from" lines. 3. Port the code following the conventions. Use the hand-ported reference files as models: `packages/asyncio/src/*`, `packages/core/src/processors/frame_processor.ts`, `packages/core/src/utils/base_object.ts`, the top of `packages/core/src/frames/frames.ts`, and `packages/core/src/utils/frame_queue.ts`. 4. Port its upstream tests 1:1 (`tests/test_x.py` to `packages//test/test_x.test.ts`). A test blocked by an unported dependency is `test.todo` naming the missing module; a deliberate behavior difference is `test.skip` with a `// DIVERGENCE:` comment. Never weaken an assertion. 5. Run the checks, then set the status to `ported` (or `tested`) with notes. ```bash bun test bun run typecheck bun run lint # biome + license headers bun run compat # smoke tests on bun, node, deno and workerd; --write records the results ``` ## Moving to a new upstream commit [#moving-to-a-new-upstream-commit] ```bash bun run upstream bump # to origin/main bun run upstream bump # or to a tag / commit ``` This moves the pin in `upstream.json`, checks out the new commit, and lists the ported modules and tests whose upstream source changed. Those are reset to `todo` with a note, so the next `port` workflow run picks them up again. It then re-runs `tools/port/graph.py` to refresh the dependency analysis; run `bun run port:status` afterwards to regenerate `STATUS.md`. ## Documentation [#documentation] These docs live in `apps/docs` (Fumadocs on Next.js). Code samples live in `apps/docs/snippets` and are type-checked against the real sources on every docs build; API tables are generated from the sources with AutoTypeTable. ```bash bun run docs:dev # http://localhost:3000 bun run docs:build ``` # Porting status > Progress per package and runtime compatibility, generated from porting/manifest.json. Source: https://polvallverdu.github.io/pipedog/docs/contributing/status/ This page renders `porting/STATUS.md` from the repository at build time. It is generated by `bun run port:status` from `porting/manifest.json` and each package's `pipecat.runtimes`, so it is as current as the last docs build. # Bundling > Ship a pipedog bot as one file with bun build, with the models and WebAssembly files inside, and know what has to stay outside. Source: https://polvallverdu.github.io/pipedog/docs/deployment/bundling/ `bun build` turns a bot and its dependencies into one file (or one executable, with `--compile`), so a server needs no `node_modules`. pipedog bundles, with a few things you have to hand over yourself, because a bundler can follow an `import` but not a path built at run time. This page lists them, with a recipe for each. Every recipe here runs in CI: `examples/bundle-check` bundles a sample bot per configuration with `bun build` (and `--compile`), and runs it from a directory with no `node_modules`, against mock providers. The runs use Bun 1.4. ## The build [#the-build] ```sh bun build --target=bun --outdir dist src/bot.ts bun dist/bot.js ``` * **Use `--outdir`, not `--outfile`.** Models and WebAssembly files imported with `with { type: "file" }` are copied next to the bundle; Bun refuses `--outfile` when there are any. Deploy the whole `dist` directory. * **`bun build --compile --outfile bot src/bot.ts`** puts the bundle, the runtime and every file import in one executable. Nothing is copied next to it. * Minifying works. Add `--keep-names` to keep processor names readable in the logs. * **Run it with `bun --no-install`** when you test the bundle in a directory without `node_modules`: without it Bun fetches missing packages from npm while it runs, which hides what the bundle lacks. ## What breaks, and what to do [#what-breaks-and-what-to-do] | What you see | Cause | What to do | | -------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------- | | `ENOENT ... data/silero_vad.onnx` (now an error that names the `model` option) | Silero and Smart Turn read their model from `data/` next to the package's source. A bundle has no such directory. | [Pass the model](#the-model-and-webassembly-files) as a `with { type: "file" }` import. | | `Failed to initialize RNNoise: ENOENT ... rnnoise.wasm` | Same: the module is read next to the source. | Call `initRnnoise()` with the file. | | `No ONNX runtime available` | `onnxruntime-node` and `onnxruntime-web` are imported by name at run time, which the bundle doesn't contain. | Install `onnxruntime-node` next to the bundle, or [bundle `onnxruntime-web`](#onnxruntime). | | `Smart Turn is unavailable, so user turns end when the VAD hears silence` | Core finds `@pipedog/audio-smart-turn` by name at run time, for the default turn strategies. | Pass `createLocalSmartTurnAnalyzerV3()` to the turn strategy yourself. | | `Missing module: @google-cloud/text-to-speech` or `.../speech` | The services import the gRPC SDKs by name. | [Import them and pass them in](#google), or install them next to the bundle. | | `/ws` answers 404 from `@pipedog/runner`, or `WebRTC routes disabled` in its log | The runner checked that its transports resolve by name. Fixed: they are now always counted as there. SmallWebRTC's `reflect-metadata` import is also kept in order. | Update `@pipedog/runner` and `@pipedog/transport-smallwebrtc`. | SmallWebRTC (werift, with Opus compiled to WebAssembly in `libopus-wasm`), the WebSocket transports, the OpenAI and Gemini services, the resampler and the telephony serializers bundle without any option. `reflect-metadata`, which werift's X.509 code needs, is loaded by `@pipedog/transport-smallwebrtc` itself, first: you don't mark it external or import it. ## The model and WebAssembly files [#the-model-and-webassembly-files] `with { type: "file" }` makes Bun copy the file and give you its path. Turn the path into a URL with `new URL(path, import.meta.url)`: it points at the file next to the bundle (or inside the executable) wherever the process was started from. `Bun.file(path)` alone is relative to the working directory, so it fails when a service manager starts the bot from another one. **Silero VAD** (`@pipedog/audio-silero/silero_vad.onnx`, 2.3 MB), and **Smart Turn** (`@pipedog/audio-smart-turn/smart-turn-v3.2-cpu.onnx`, 8.7 MB): ```ts title="bundling_silero.ts" import { SileroVADAnalyzer } from "@pipedog/audio-silero"; import sileroPath from "@pipedog/audio-silero/silero_vad.onnx" with { type: "file" }; // `new URL(path, import.meta.url)` is where the bundler put the file, wherever the bot is // started from. (`Bun.file(sileroPath)` alone is relative to the working directory.) export const vad = new SileroVADAnalyzer({ model: new URL(sileroPath, import.meta.url) }); ``` ```ts title="bundling_smart_turn.ts" import { createLocalSmartTurnAnalyzerV3 } from "@pipedog/audio-smart-turn"; import smartTurnPath from "@pipedog/audio-smart-turn/smart-turn-v3.2-cpu.onnx" with { type: "file", }; import { TurnAnalyzerUserTurnStopStrategy } from "@pipedog/core/turns/user_stop/turn_analyzer_user_turn_stop_strategy"; import { UserTurnStrategies } from "@pipedog/core/turns/user_turn_strategies"; // Pass the analyzer to the strategy yourself. By default core finds @pipedog/audio-smart-turn // by name at run time, which a bundler can't see. export const turnStrategies = new UserTurnStrategies({ stop: [ new TurnAnalyzerUserTurnStopStrategy({ turnAnalyzer: createLocalSmartTurnAnalyzerV3({ model: new URL(smartTurnPath, import.meta.url), }), }), ], }); ``` **RNNoise** (`@pipedog/audio-rnnoise-filter/rnnoise.wasm`, 1.4 MB): ```ts title="bundling_rnnoise.ts" import { initRnnoise, RNNoiseFilter } from "@pipedog/audio-rnnoise-filter"; import rnnoisePath from "@pipedog/audio-rnnoise-filter/rnnoise.wasm" with { type: "file" }; // Load the module before the filter starts; the filter then finds it ready. await initRnnoise(new URL(rnnoisePath, import.meta.url)); export const audioInFilter = new RNNoiseFilter(); ``` **The resampler** (`@pipedog/resampler-wasm/resampler.wasm`, 39 KB) needs nothing: the package embeds a copy of the module in its JavaScript and uses it when the file isn't there. To load the file instead: ```ts title="bundling_resampler.ts" import { initResampler } from "@pipedog/resampler-wasm"; import resamplerPath from "@pipedog/resampler-wasm/resampler.wasm" with { type: "file" }; // Optional: without this the resampler falls back to a copy of the module embedded in the // package's JavaScript (38 KB), so a bundle needs nothing. await initResampler(new URL(resamplerPath, import.meta.url)); ``` The model options also take bytes (`new Uint8Array(await Bun.file(url).arrayBuffer())`) or a path. The models are not embedded in the JavaScript: they are 2 to 9 MB. ## onnxruntime [#onnxruntime] Silero and Smart Turn run on `onnxruntime-node` (a native addon) when it is installed, and on `onnxruntime-web` (WebAssembly) otherwise. Pick the one that fits how you ship: * **`onnxruntime-node`** is the fastest, and it can't be bundled: it is a native addon with a binary per platform. Build with `--external onnxruntime-node` and install it next to the bundle (`bun add onnxruntime-node`). It doesn't work with `bun build --compile`: an executable can't load an external package. * **`onnxruntime-web`** bundles, and works with `--compile`. Import it yourself and give it its two files (the loader and the module, 14 MB, copied next to the bundle): ```ts title="bundling_onnx_web.ts" import { SileroVADAnalyzer } from "@pipedog/audio-silero"; import sileroPath from "@pipedog/audio-silero/silero_vad.onnx" with { type: "file" }; import { createLocalSmartTurnAnalyzerV3 } from "@pipedog/audio-smart-turn"; import smartTurnPath from "@pipedog/audio-smart-turn/smart-turn-v3.2-cpu.onnx" with { type: "file", }; import { createOnnxBackend } from "@pipedog/onnx"; import * as ort from "onnxruntime-web"; import ortLoaderPath from "onnxruntime-web/ort-wasm-simd-threaded.mjs" with { type: "file" }; import ortWasmPath from "onnxruntime-web/ort-wasm-simd-threaded.wasm" with { type: "file" }; const here = (asset: string) => new URL(asset, import.meta.url); // One WebAssembly backend for every model, with everything it loads inside the bundle. const backend = createOnnxBackend({ prefer: "web", web: { ort, wasmPaths: { mjs: here(ortLoaderPath), wasm: here(ortWasmPath) } }, }); export const vad = new SileroVADAnalyzer({ model: here(sileroPath), backend }); export const turnAnalyzer = createLocalSmartTurnAnalyzerV3({ model: here(smartTurnPath), backend }); ``` Install `onnxruntime-web@1.30.0` (`bun add onnxruntime-web@1.30.0`) as a dependency of your app, at the version `@pipedog/onnx` is tested with. Create the backend once and pass it to every model, so they share it. The cost is speed. Measured on Bun 1.4 (Apple M4 Pro, one thread, `bun run bench`): | Backend | Silero, per 32 ms chunk | Smart Turn inference | Smart Turn per prediction (features and inference) | | ------------------ | ----------------------: | -------------------: | -------------------------------------------------: | | `onnxruntime-node` | 0.135 ms | 72 ms | 208 ms | | `onnxruntime-web` | 0.199 ms | 166 ms | 225 ms | Silero is 0.6% of a real-time chunk either way. Smart Turn runs once each time the user pauses, and the web backend adds about 17 ms to that prediction. ## Google [#google] `@google/genai` (Gemini) is a dependency of `@pipedog/google` and bundles. The Cloud Speech-to-Text and Text-to-Speech clients are optional peers that talk gRPC; the services import them by name. Import them in your bot and pass the modules, which puts them in the bundle (grpc-js, google-gax and their protos add about 4 MB): ```ts title="bundling_google.ts" import * as speechModule from "@google-cloud/speech"; import * as textToSpeechModule from "@google-cloud/text-to-speech"; import { GoogleSTTService } from "@pipedog/google/stt"; import { GoogleHttpTTSService } from "@pipedog/google/tts"; // Import the SDKs yourself and hand them over: the services import them by name otherwise. export const stt = new GoogleSTTService({ speechModule }); export const tts = new GoogleHttpTTSService({ textToSpeechModule }); ``` `GoogleTTSService` and `GeminiTTSService` take `textToSpeechModule` too. ## What stays outside [#what-stays-outside] | Package | When | Why | | ----------------------------------------------------------------------------- | --------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | | `onnxruntime-node` | You use the native ONNX backend. | A native addon: its `.node` binaries are loaded from `node_modules`. Not possible with `--compile`; use `onnxruntime-web`. | | `ws` | You serve WebSockets on Node. | An optional peer, imported by name. Bun doesn't need it: `Bun.serve` upgrades the socket. | | `@pipedog/transport-livekit`, `@pipedog/transport-whatsapp`, `@pipedog/evals` | You use them through `@pipedog/runner`. | Optional peers that the runner looks up by name. The bundle check doesn't cover them. | Nothing else the bundle check runs has to stay out: the Google clients, `reflect-metadata`, werift, `@peculiar/x509`, `tsyringe`, `libopus-wasm` and the models are all in the bundle, or next to it as files. ## The runner [#the-runner] `@pipedog/runner`'s `main()` also serves the prebuilt browser client from a directory next to its source, which a bundle doesn't have, and logs `Prebuilt frontend not available`. The telephony and WebSocket routes work without it, and the WebRTC routes stay enabled. See [The runner](https://polvallverdu.github.io/pipedog/docs/runner.md). # Voice agent fully on Cloudflare > Run the whole voice stack on Cloudflare, with Deepgram Flux speech-to-text, a Workers AI LLM and Aura text-to-speech in a Durable Object. Source: https://polvallverdu.github.io/pipedog/docs/deployment/cloudflare-workers-ai/ [Voice agents on Workers and Durable Objects](https://polvallverdu.github.io/pipedog/docs/deployment/cloudflare-workers.md) runs one agent per Durable Object, with the models at OpenAI. This page swaps the models for Workers AI, so the transport, the agent and the models all run on Cloudflare's network, and the only secrets are a Cloudflare account ID and API token: | Step | Service | Workers AI model | | --------------------------------- | ------------------------- | -------------------------------------------------------------- | | Speech-to-text and turn detection | `WorkersAIFluxSTTService` | `@cf/deepgram/flux` | | LLM | `WorkersAILLMService` | `@cf/meta/llama-3.3-70b-instruct-fp8-fast` (or any chat model) | | Text-to-speech | `WorkersAIAuraTTSService` | `@cf/deepgram/aura-2-en` | ```text client ──WebSocket──► Worker ──getByName(id)──► VoiceSession (Durable Object) │ FastAPIWebsocketTransport │ Flux (env.AI, WebSocket): transcripts and turns │ Workers AI LLM (OpenAI-compatible REST) │ Aura (env.AI, WebSocket): speech ``` ## Install [#install] ```bash bun add @pipedog/core @pipedog/asyncio @pipedog/cloudflare \ @pipedog/transport-websocket @pipedog/serializer-protobuf ``` ## Configure the Worker [#configure-the-worker] Add the AI binding next to the Durable Object in `wrangler.jsonc`: ```jsonc { "name": "voice-agent", "main": "src/worker.ts", "compatibility_date": "2026-10-01", "compatibility_flags": ["nodejs_compat"], "ai": { "binding": "AI" }, "durable_objects": { "bindings": [{ "name": "VOICE_SESSION", "class_name": "VoiceSession" }] }, "migrations": [{ "tag": "v1", "new_classes": ["VoiceSession"] }] } ``` The speech services use the binding, with no token. The LLM uses Workers AI's OpenAI-compatible REST endpoint (the binding answers chat requests in Workers AI's own format), so it needs the account ID and an API token with the Workers AI permission: ```sh bunx wrangler secret put CLOUDFLARE_ACCOUNT_ID bunx wrangler secret put CLOUDFLARE_API_TOKEN ``` ```ts title="env.ts" import type { WorkersAIBinding } from "@pipedog/cloudflare/client"; export interface Env { /** The Workers AI binding: `"ai": { "binding": "AI" }` in wrangler.jsonc. */ AI: WorkersAIBinding; /** For the LLM, which uses Workers AI's OpenAI-compatible REST endpoint. */ CLOUDFLARE_ACCOUNT_ID: string; CLOUDFLARE_API_TOKEN: string; } ``` ## The agent [#the-agent] The pipeline is the same as in any voice bot. What changes is the services: ```ts title="bot.ts" import { WorkersAIFluxSTTService } from "@pipedog/cloudflare/flux_stt"; import { WorkersAILLMService } from "@pipedog/cloudflare/llm"; import { WorkersAIAuraTTSService } from "@pipedog/cloudflare/tts"; import { LLMRunFrame } from "@pipedog/core/frames/frames"; import { Pipeline } from "@pipedog/core/pipeline/pipeline"; import { PipelineParams, PipelineWorker } from "@pipedog/core/pipeline/worker"; import { LLMContext } from "@pipedog/core/processors/aggregators/llm_context"; import { LLMContextAggregatorPair } from "@pipedog/core/processors/aggregators/llm_response_universal"; import { WorkerRunner } from "@pipedog/core/workers/runner"; import { ProtobufFrameSerializer } from "@pipedog/serializer-protobuf/protobuf"; import { FastAPIWebsocketParams, FastAPIWebsocketTransport, } from "@pipedog/transport-websocket/fastapi"; import type { WebsocketConnection } from "@pipedog/transport-websocket/socket"; /** Flux and the client both use 16 kHz audio; Aura answers at 24 kHz. */ const INPUT_SAMPLE_RATE = 16000; const OUTPUT_SAMPLE_RATE = 24000; export async function runVoiceAgent(connection: WebsocketConnection, env: Env): Promise { const transport = new FastAPIWebsocketTransport( connection, new FastAPIWebsocketParams({ audioInEnabled: true, audioOutEnabled: true, serializer: new ProtobufFrameSerializer(), }), ); // Flux detects the start and end of each turn, so there is no VAD analyzer // and no Smart Turn model to load: the user aggregator follows Flux. const stt = new WorkersAIFluxSTTService({ ai: env.AI }); const llm = new WorkersAILLMService({ accountId: env.CLOUDFLARE_ACCOUNT_ID, apiToken: env.CLOUDFLARE_API_TOKEN, settings: new WorkersAILLMService.Settings({ model: "@cf/meta/llama-3.3-70b-instruct-fp8-fast", systemInstruction: "You are a helpful assistant in a voice conversation. Answer briefly, in plain sentences.", }), }); const tts = new WorkersAIAuraTTSService({ ai: env.AI, sampleRate: OUTPUT_SAMPLE_RATE, settings: new WorkersAIAuraTTSService.Settings({ voice: "luna" }), }); const context = new LLMContext(); const aggregators = new LLMContextAggregatorPair(context); const pipeline = new Pipeline([ transport.input(), stt, aggregators.user(), llm, tts, transport.output(), aggregators.assistant(), ]); const worker = new PipelineWorker(pipeline, { cancelOnIdleTimeout: false, params: new PipelineParams({ audioInSampleRate: INPUT_SAMPLE_RATE, audioOutSampleRate: OUTPUT_SAMPLE_RATE, enableMetrics: true, enableUsageMetrics: true, }), }); const runner = new WorkerRunner({ handleSigint: false, handleSigterm: false }); await runner.addWorkers(worker); transport.eventHandler("on_client_connected", async () => { context.addMessage({ role: "developer", content: "Please introduce yourself to the user." }); await worker.queueFrames([new LLMRunFrame()]); }); transport.eventHandler("on_client_disconnected", async () => { await runner.cancel(); }); await runner.run(); } ``` Flux detects turns on Cloudflare's side and the user aggregator follows its turn events, so there is no VAD analyzer and no Smart Turn model: the Worker loads no ONNX runtime and stays small (see [turn detection on Workers](https://polvallverdu.github.io/pipedog/docs/deployment/cloudflare-workers/#turn-detection)). ## The Durable Object and the Worker [#the-durable-object-and-the-worker] They are those of [Voice agents on Workers](https://polvallverdu.github.io/pipedog/docs/deployment/cloudflare-workers.md), with `runVoiceAgent` from above: ```ts import { DurableObject } from "cloudflare:workers"; import { websocketFetchHandler } from "@pipedog/transport-websocket/fetch"; export class VoiceSession extends DurableObject { readonly #handler = websocketFetchHandler((connection) => runVoiceAgent(connection, this.env)); override fetch(request: Request): Promise { return this.#handler(request); } } export default { async fetch(request: Request, env: Env & { VOICE_SESSION: DurableObjectNamespace }) { const id = new URL(request.url).pathname.split("/")[3]; return env.VOICE_SESSION.getByName(id).fetch(request); }, }; ``` The [example](https://github.com/polvallverdu/pipedog/tree/main/examples/workers-durable-voice-agent) adds reconnection, tracing and alerts on top, and runs the same way: replace its `bot.ts` services with the ones above and add the AI binding. ## Variations [#variations] * **Nova-3 or Whisper instead of Flux.** `WorkersAINova3STTService` (streaming) and `WorkersAIWhisperSTTService` (one request per utterance) need a VAD and a turn analyzer. On Workers, Smart Turn v2 runs on Workers AI too (`WorkersAISmartTurnAnalyzer`, see [turn detection](https://polvallverdu.github.io/pipedog/docs/concepts/turn-detection/#smart-turn-on-workers-ai)); the newer local Smart Turn v3.2 also runs on Workers but makes the bundle much larger. * **Aura over HTTP.** `WorkersAIAuraHttpTTSService` makes one request per sentence instead of holding a WebSocket. * **Outside Workers.** Every service takes `accountId` and `apiToken` instead of `ai`, and works on Bun, Node and Deno over the REST API. ## Limits to plan for [#limits-to-plan-for] * The Durable Object limits of [Voice agents on Workers](https://polvallverdu.github.io/pipedog/docs/deployment/cloudflare-workers/#limits-to-plan-for) apply: billed duration while a session is connected, CPU per session, WebSocket audio only. * Workers AI bills Flux and Aura per audio minute and per character; see each [model page](https://developers.cloudflare.com/workers-ai/models/). * The services are tested against a local fake of Workers AI (binding and REST), not against Workers AI itself. # Voice agents on Workers and Durable Objects > Run one voice agent per session on Cloudflare Durable Objects, and serve pipedog endpoints from any framework with fetch handlers. Source: https://polvallverdu.github.io/pipedog/docs/deployment/cloudflare-workers/ A voice agent is a long-lived connection with state: a pipeline, a conversation, and a socket. Cloudflare Durable Objects suit that shape. Each object is a single-threaded actor with its own state, so one object can run one voice session without interfering with any other. Cloudflare places the objects across its network, which lets many sessions run in parallel. The complete example is [`examples/workers-durable-voice-agent`](https://github.com/polvallverdu/pipedog/tree/main/examples/workers-durable-voice-agent). This page explains the pattern. To run the models on Cloudflare too, see [Voice agent fully on Cloudflare](https://polvallverdu.github.io/pipedog/docs/deployment/cloudflare-workers-ai.md). ## The shape [#the-shape] ```text client ──WebSocket──► Worker ──getByName(id)──► VoiceSession (Durable Object) │ FastAPIWebsocketTransport │ OpenAI realtime transcription (turns) │ chat, speech synthesis ``` * The **Worker** routes each session to a Durable Object by name. The same name always reaches the same object. * The **Durable Object** accepts the WebSocket and runs the agent for as long as the socket is open. * The **transport** is the WebSocket transport, carrying audio as protobuf frames in the Pipecat wire format. WebRTC needs UDP, which Workers don't have, so the Worker can't be a WebRTC peer itself. For browser or app users on WebRTC, let [Cloudflare Realtime](https://polvallverdu.github.io/pipedog/docs/transports/cloudflare-realtime.md) be the peer: its SFU converts the audio to PCM over WebSockets that the Durable Object accepts (see [`examples/workers-realtime-voice-agent`](https://github.com/polvallverdu/pipedog/tree/main/examples/workers-realtime-voice-agent)). ## The Durable Object [#the-durable-object] `websocketFetchHandler` turns a connection into a fetch handler. The object returns its result from `fetch`: ```ts import { DurableObject } from "cloudflare:workers"; import { websocketFetchHandler } from "@pipedog/transport-websocket/fetch"; export class VoiceSession extends DurableObject { readonly #handler = websocketFetchHandler((connection) => runVoiceAgent(connection, this.env)); override fetch(request: Request): Promise { return this.#handler(request); } } ``` `runVoiceAgent` builds the pipeline on the connection with `FastAPIWebsocketTransport`. It is the same pipeline as the other voice bot examples: speech in, speech-to-text, the user's turn, the chat model, speech synthesis, and speech out. ## The Worker [#the-worker] ```ts export default { async fetch(request: Request, env: Env): Promise { const id = new URL(request.url).pathname.split("/")[3]; return env.VOICE_SESSION.getByName(id).fetch(request); }, } satisfies ExportedHandler; ``` ## Turn detection [#turn-detection] The agent detects turns on the provider's side: `OpenAIRealtimeSTTService` with server VAD, and no VAD analyzer. So the Worker loads no local models and no ONNX runtime. That matters here: | Turn detection | Worker size (gzipped) | | ---------------------------------- | --------------------- | | Provider (this example) | 818 KB | | Silero VAD and Smart Turn, bundled | 13.7 MB | Cloudflare limits a Worker to 10 MB compressed on paid plans and 3 MB on the free plan, so the local models don't fit. They also load about 100 MB of runtime and models, close to a Durable Object's 128 MB memory limit. See [Turn detection](https://polvallverdu.github.io/pipedog/docs/concepts/turn-detection.md) for the modes. ## Reconnection [#reconnection] A client that drops and reconnects to the same session id within ten minutes continues the same conversation. The Durable Object saves the conversation to its storage when a socket closes, loads it when the next one opens, and skips the greeting. An alarm clears it once the ten minutes pass. If a reconnect arrives while the old socket still looks open, the new one wins: the object closes the old one and waits for its conversation to be saved first. Pipecat's WebSocket transport doesn't resume sessions; this is the example's own logic, in `src/voice_session.ts`. ## Tracing and alerts [#tracing-and-alerts] Set `OTEL_EXPORTER_OTLP_ENDPOINT` and `ERROR_WEBHOOK_URL` in the Worker's environment, as in the [voice bot guide](https://polvallverdu.github.io/pipedog/docs/voice-bot/#tracing-and-alerts). Spans are buffered, and a Worker doesn't export them by itself after a request ends, so the agent flushes them when each session ends. ## Fetch handlers [#fetch-handlers] pipedog's voice endpoints are fetch handlers: functions from a `Request` to a `Response`. That makes them usable from any framework that routes requests this way. * **WebSocket endpoints.** `websocketFetchHandler` upgrades from fetch on its own on Cloudflare Workers, Durable Objects and Deno. On Bun, pass it the server Bun gives `fetch` and serve with `bunWebSocket`; on Node, serve it with `serveFetch()`. The runner's fetch handler (`createRunnerApp()`) upgrades the same way, though it isn't tested on Workers yet. * **WebRTC signaling.** `smallWebRTCOfferHandler` is a plain fetch handler. Its media runs over UDP, so the server has to be a long-lived Node, Bun or Deno process. Serverless hosts can't carry it. On Workers, use [Cloudflare Realtime](https://polvallverdu.github.io/pipedog/docs/transports/cloudflare-realtime.md) instead: its SFU carries the WebRTC media and hands the bot PCM over WebSockets. Shapes for the other frameworks: ```ts // Bun Bun.serve({ fetch: (request, server) => handler(request, server), websocket: bunWebSocket }); // Node, Bun or Deno await serveFetch(handler, { port: 8765 }); // Hono app.post("/api/offer", (c) => offer(c.req.raw)); // Next.js app router (app/api/offer/route.ts) export const POST = (request: Request) => offer(request); // SvelteKit (src/routes/api/offer/+server.ts) export const POST = ({ request }) => offer(request); ``` The Workers example runs in tests against workerd, through `wrangler dev`. The same handlers mounted in each framework are tested in `examples/framework-snippets`: see [Frameworks](https://polvallverdu.github.io/pipedog/docs/frameworks.md). ## Run it [#run-it] ```sh cd examples/workers-durable-voice-agent echo "OPENAI_API_KEY=sk-..." > .dev.vars bun run dev ``` Stream a recording to a session with the scripted user from the example, in a second terminal: ```sh bun run client ws://127.0.0.1:8787/api/session/alice ../test-mocks/assets/capital_question.wav ``` Then deploy: ```sh bunx wrangler secret put OPENAI_API_KEY bun run deploy ``` ## Limits to plan for [#limits-to-plan-for] * **Duration.** A Durable Object is billed while it runs. A connected session keeps it running, so idle sessions cost time. Hibernation, where the object sleeps between messages, would save that, but the agent's timers would need to survive a sleep, and this example doesn't do that. * **CPU.** Each session still decodes, resamples and serializes audio. Check the CPU limits for your plan under real load. * **Transport.** The Worker can't be a WebRTC peer. Over the WebSocket transport, the client sends and receives protobuf frames. For WebRTC clients, put [Cloudflare Realtime](https://polvallverdu.github.io/pipedog/docs/transports/cloudflare-realtime.md) in between. # Deployment overview > Where a pipedog voice agent can run in production, and which transport each host allows. Source: https://polvallverdu.github.io/pipedog/docs/deployment/ A voice agent is a long-lived session: it holds a pipeline, a conversation and a connection for the whole call. That shapes where it can run. The transport decides most of it: * **SmallWebRTC** sends media over UDP. It needs a long-lived server process with UDP reachable: Bun, Node or Deno on a VM, a container or a bare-metal host. * **WebSocket** sends audio frames over TCP. It runs on the same servers, and also on Cloudflare Workers and Durable Objects, and on Vercel's Bun runtime. * **Cloudflare Realtime** lets users connect over WebRTC to Cloudflare's SFU, which streams their audio to the bot as PCM over WebSockets. The bot can then run in a Worker or Durable Object (see [Cloudflare Realtime](https://polvallverdu.github.io/pipedog/docs/transports/cloudflare-realtime.md)). | Host | Runtime | SmallWebRTC | WebSocket | Page | | --------------------------------------- | --------------- | --------------------------------------------------------------------------------------------- | --------- | -------------------------------------------------------------------------- | | VM, container, bare metal | Bun, Node, Deno | Yes | Yes | [Servers](https://polvallverdu.github.io/pipedog/docs/deployment/servers.md), [Bundling](https://polvallverdu.github.io/pipedog/docs/deployment/bundling.md) | | Cloudflare Workers + Durable Objects | workerd | No (no UDP); WebRTC users through [Cloudflare Realtime](https://polvallverdu.github.io/pipedog/docs/transports/cloudflare-realtime.md) | Yes | [Cloudflare Workers](https://polvallverdu.github.io/pipedog/docs/deployment/cloudflare-workers.md) | | Vercel Functions | Bun runtime | No (no UDP) | Yes | [Vercel](https://polvallverdu.github.io/pipedog/docs/deployment/vercel.md) | | Serverless functions without WebSockets | any | No | No | Signalling only | On a server, [`@pipedog/runner`](https://polvallverdu.github.io/pipedog/docs/runner.md) gives you Pipecat's runner routes (the prebuilt client, WebRTC signalling, telephony webhooks and WebSockets) as one fetch handler: `createRunnerApp()` builds it and `serve()` listens on Bun, Node or Deno. See [Servers](https://polvallverdu.github.io/pipedog/docs/deployment/servers/#the-development-runner). Pipecat's own deployment guides (Fly.io, Cerebrium, Modal, Pipecat Cloud) assume a Python process per session. The same patterns apply to a pipedog server: one process (or one Durable Object) per session, and a cap on concurrent sessions. ## Before you deploy [#before-you-deploy] * **Authenticate the endpoints.** pipedog's endpoints don't authenticate. Put a check in front of them, as in [Frameworks: Authentication](https://polvallverdu.github.io/pipedog/docs/frameworks/#authentication). * **Provider keys stay on the server.** Services read their keys from options you pass; read them from the environment or the host's secret store, never from the client. * **Turn detection and size.** Local Silero VAD and Smart Turn load ONNX models (about 100 MB of runtime and models). On Workers, use provider-side turn detection instead. See [Turn detection](https://polvallverdu.github.io/pipedog/docs/concepts/turn-detection.md). * **Tracing.** Set `OTEL_EXPORTER_OTLP_ENDPOINT` to export spans. See [Metrics & tracing](https://polvallverdu.github.io/pipedog/docs/concepts/metrics-and-tracing.md). * **Runtimes.** The [compatibility table](https://polvallverdu.github.io/pipedog/docs/guides/runtimes.md) lists where each package is verified. Bun, Node or Deno, with SmallWebRTC and WebSocket. One Durable Object per session, over WebSocket. Flux, a Workers AI LLM and Aura, with the agent in a Durable Object. WebSocket sessions on Vercel Functions with the Bun runtime. # Node, Bun and Deno servers > Run a pipedog voice agent as a long-lived server process, with SmallWebRTC or WebSocket sessions. Source: https://polvallverdu.github.io/pipedog/docs/deployment/servers/ A server process is the most flexible host: it runs both transports, local turn detection models, and any framework. The [voice bot guide](https://polvallverdu.github.io/pipedog/docs/voice-bot.md) builds one with SmallWebRTC, and [Frameworks](https://polvallverdu.github.io/pipedog/docs/frameworks.md) mounts the same endpoints in SvelteKit, Hono, Next.js and others. ## The process [#the-process] Each session runs in the same process, as its own pipeline and `PipelineWorker`. Start one per connection, from the endpoint handler, and end it when the client disconnects: ```ts transport.eventHandler("on_client_disconnected", async () => { await runner.cancel(); }); ``` The shared agent in [`examples/framework-snippets/shared/agent.ts`](https://github.com/polvallverdu/pipedog/blob/main/examples/framework-snippets/shared/agent.ts) does exactly this. Pass `handleSigint: false` and `handleSigterm: false` to each session's `WorkerRunner`, so one session doesn't take the process's signals; handle shutdown in the server. ## Runtimes [#runtimes] * **Bun** runs the agent and upgrades WebSockets in `Bun.serve`: pass the server Bun gives `fetch` to `websocketFetchHandler`'s handler and serve with `bunWebSocket`. See [Bun](https://polvallverdu.github.io/pipedog/docs/frameworks/bun.md). * **Node.js** 22 or later. Use a framework's Node adapter (Express, Hono's `@hono/node-server`, SvelteKit's `adapter-node`), or `nodeRequestListener` from `@pipedog/transport-smallwebrtc/http` for a plain `node:http` server. For WebSocket endpoints, `serveFetch()` from `@pipedog/transport-websocket/serve` serves a fetch handler with upgrades (with the `ws` package). * **Deno** serves both endpoints with `Deno.serve`. See [Deno](https://polvallverdu.github.io/pipedog/docs/frameworks/deno.md). `serveFetch(handler, { port })` listens with the same fetch handler on all three runtimes, so one `server.ts` runs on Bun, Node and Deno; `examples/twilio-phone-bot` is served this way. ## The development runner [#the-development-runner] [`@pipedog/runner`](https://polvallverdu.github.io/pipedog/docs/runner.md) is Pipecat's development runner. `main()` serves your `bot()` with the prebuilt browser client, SmallWebRTC signalling and the telephony webhooks and WebSockets, from flags (`bun bot.ts -t twilio --proxy ...`). For a server you deploy, build the same app with `createRunnerApp()` and listen with `serve()`, or mount its `fetch` in your own server: ```ts import { createRunnerApp } from "@pipedog/runner/run"; import { serve } from "@pipedog/runner/serve"; const runner = await createRunnerApp({ bot, transport: "twilio", proxy: "bot.example.com" }); await serve(runner, { host: "0.0.0.0", port: 7860 }); ``` `serve()` runs on Bun, Node (with `ws`) and Deno. The runner's routes don't authenticate, as upstream's don't; put your checks in front of them. See [Mounting the runner](https://polvallverdu.github.io/pipedog/docs/runner/#mounting-the-runner-in-your-own-server). ## SmallWebRTC in production [#smallwebrtc-in-production] SmallWebRTC is a peer-to-peer WebRTC connection between the browser and your server. The media runs over UDP, so: * the server needs UDP reachable from clients, on the ports the WebRTC stack picks; * behind NAT, give the transport ICE servers (STUN, and TURN for restrictive networks); * one server holds every session's media, so scale by adding servers and routing sessions to them. See [SmallWebRTC](https://polvallverdu.github.io/pipedog/docs/transports/smallwebrtc.md) for the transport's options. ## WebSocket in production [#websocket-in-production] WebSocket sessions need only TCP, so they pass through ordinary load balancers and proxies. Keep idle timeouts on the proxy longer than a call, or send keepalives. Each socket stays on the server that accepted it for its lifetime. See [WebSocket](https://polvallverdu.github.io/pipedog/docs/transports/websocket.md). ## Containers [#containers] A container image needs the runtime and your app. If you use local turn detection (`@pipedog/audio-silero`, `@pipedog/audio-smart-turn`), the ONNX models load at startup from a path or URL you give them; bake them into the image or download them on start. See [Turn detection](https://polvallverdu.github.io/pipedog/docs/concepts/turn-detection.md). ## Sizing [#sizing] Each session decodes, resamples and encodes audio, and keeps provider connections open. Measure CPU and memory per session under real load, and cap concurrent sessions per process. Provider rate limits often bind before CPU does. # Vercel > Run WebSocket voice sessions on Vercel Functions with the Bun runtime, and what Vercel can't host. Source: https://polvallverdu.github.io/pipedog/docs/deployment/vercel/ Vercel Functions can run a `server.ts` with the Bun runtime. With it, a function accepts WebSocket upgrades and runs one voice session per socket. The code is the [Vercel framework page](https://polvallverdu.github.io/pipedog/docs/frameworks/vercel.md), tested in [`examples/framework-snippets/vercel`](https://github.com/polvallverdu/pipedog/tree/main/examples/framework-snippets/vercel). Set the runtime in `vercel.json`: ```json title="vercel.json" { "bunVersion": "1.x" } ``` ## What works [#what-works] * **WebSocket sessions.** The `/ws` endpoint upgrades in `fetch` and runs the agent on the socket, with protobuf audio frames. * **WebRTC signalling.** The `/api/offer` route answers the SDP offer, but the media can't follow: see below. ## Limits [#limits] * **No WebRTC media.** SmallWebRTC needs UDP, which Vercel Functions don't carry. Use the WebSocket transport, or run SmallWebRTC on a [server](https://polvallverdu.github.io/pipedog/docs/deployment/servers.md). * **One instance per socket.** Each WebSocket is pinned to one function instance for its lifetime, and messages don't cross instances. Keep shared state in a database. * **Duration.** A function's maximum duration also ends a long session. Check your plan's limit against your longest calls. * **Vercel Daemons** are always-on workers with no inbound HTTP or TCP traffic, so they can't serve these endpoints. They suit background workers that pull their own work. * **Next.js on Vercel.** A Next.js route handler can answer the offer, but it can't upgrade a WebSocket. See [Next.js](https://polvallverdu.github.io/pipedog/docs/frameworks/nextjs.md). # Evals > Test a bot end to end with scripted or LLM-simulated users, judged by an LLM, from YAML scenarios and the pipedog-eval CLI. Source: https://polvallverdu.github.io/pipedog/docs/evals/ `@pipedog/evals` is the port of Pipecat's `pipecat.evals`. It tests a running bot end to end. The harness connects to the bot as an RTVI client and plays the user's side of a conversation, in text or in synthesized speech. It listens to what the bot says and checks it against the scenario, with exact checks and an LLM judge. A **scenario** is one check, written in YAML. There are two kinds: * A **scripted** scenario writes out the user's turns and what the bot should do after each one. You can check text, events, function calls and timing, and send criteria to the judge. * A **simulated** scenario gives an LLM a persona and a goal, and lets it talk to the bot. A judge then reads the whole conversation and decides whether the bot did its job. Pipecat's eval suites and their scenarios, which use the same YAML format. `packages/evals/src`: `session.ts`, `scenario.ts`, `script.ts`, `judge.ts`, `suite.ts`, `cli.ts`. ## Installation [#installation] ```bash bun add -d @pipedog/evals ``` The harness runs on Bun and Node. The `pipedog-eval` command runs scenarios, judges saved conversations and runs suites: ```bash bunx pipedog-eval --help # or: npx pipedog-eval --help ``` The same commands are `pipedog eval` in [`@pipedog/cli`](https://polvallverdu.github.io/pipedog/docs/cli.md), which loads this package when you run it. A project made with `pipedog init --eval` has the `eval` transport, a starter scenario and both packages. ## Make the bot reachable [#make-the-bot-reachable] The harness connects to the [development runner](https://polvallverdu.github.io/pipedog/docs/runner.md)'s `/ws` route. Its first message, `eval-connect`, tells `createTransport()` to build the eval transport, so the bot needs nothing more than an `eval` entry in its transport parameters: ```ts title="bot.ts" import { EvalTransportParams } from "@pipedog/evals/transport"; import type { RunnerArguments } from "@pipedog/runner/types"; import { createTransport } from "@pipedog/runner/utils"; const transportParams = { // Used when an eval harness connects to the runner's /ws and sends `eval-connect`. eval: () => new EvalTransportParams({ audioInEnabled: true, audioOutEnabled: true }), }; export async function bot(runnerArgs: RunnerArguments) { const transport = await createTransport(runnerArgs, transportParams); // build and run the pipeline with transport.input() and transport.output() void transport; } ``` Start it as usual, e.g. `bun bot.ts`, and it listens on `ws://localhost:7860/ws`. ## Write a scenario [#write-a-scenario] ```yaml # scenarios/capital.yaml name: capital judge: eval: factory: ./my_evals.ts.llm # the judge's LLM, built by your code (below) model: gpt-4.1-mini scenarios: - name: capital_question turns: # Wait for the bot's greeting before speaking. - expect: - event: response eval: "the bot opens the conversation (a greeting, an offer to help)" - user: "What is the capital of Germany?" expect: - event: response text_contains: Berlin within_ms: 10000 eval: "the response says the capital of Germany is Berlin" ``` Each turn sends `user:` (or nothing, to wait for the bot) and checks the bot's events in order. `response` is the bot's reply: the LLM's text in text modality, the default, or a transcription of its speech in audio modality. Other events include `user_transcription`, `bot_started_speaking`, `function_call` (with `calls:`), `llm_marker`, `image` and `expression`. `text_contains` and `text_excludes` check the text exactly, `text_matches` takes one regular expression or a list that the text must each match (Python `re.search` semantics), `within_ms` sets a latency budget, and `eval:` asks the judge. A missed `text_matches` pattern fails the expectation with the `text_unmatched` failure kind, naming the first pattern that didn't match. The module docs of `script.ts` and `simulation.ts` describe every key, as upstream's do. ```yaml - user: "Read me my order number." expect: - event: response text_matches: - "^(Sure|Okay)" # how the reply must open - "\\d{6}" # it has a six-digit number ``` `expression` is the expression the bot's TTS performed in expressive conversation (alpha), written as the LLM writes its tags: `<>` where the delivery of the spoken text changes (`<>` back to the baseline), `[[laughter]]` for a sound, `[[pause 600]]` for a pause. A tag the TTS can't perform is never reported, so a scenario can check that what the LLM wrote was performed. It takes `text_contains`, `text_matches` and `eval:` like a response. A simulation swaps `turns:` for a persona: ```yaml scenarios: - name: capital_curious persona: | A curious, polite traveler who asks one thing at a time. goal: "Find out what the capital of Germany is, then say goodbye." success: "the bot told the caller that the capital of Germany is Berlin" simulator: factory: ./my_evals.ts.llm # must support function calling (the persona hangs up with a tool) metrics: - name: politeness criterion: "the bot stayed courteous throughout" min_score: 1 max_turns: 10 ``` ## Services: the judge, the user's voice and the transcriber [#services-the-judge-the-users-voice-and-the-transcriber] The harness builds its own services from the scenario. A `factory:` names a module and one of its exports, joined by a dot. The module is a path relative to the working directory, a package or a `file:` URL. The export is a function that takes the config block and returns the service: ```ts title="my_evals.ts" // my_evals.ts: the services a scenario names with `factory: ./my_evals.ts.` import { OpenAILLMSettings } from "@pipedog/openai/base_llm"; import { OpenAILLMService } from "@pipedog/openai/llm"; import { OpenAISTTService } from "@pipedog/openai/stt"; import { OpenAITTSService } from "@pipedog/openai/tts"; type Config = Record; /** `judge.eval` and `simulator`: an OpenAI-compatible LLM. */ export function llm(config: Config) { return new OpenAILLMService({ apiKey: process.env.OPENAI_API_KEY ?? null, settings: new OpenAILLMSettings({ model: String(config.model ?? "gpt-4.1-mini") }), }); } /** `user.speech`: the voice the harness speaks the user's turns with. */ export function voice(_config: Config) { return new OpenAITTSService({ apiKey: process.env.OPENAI_API_KEY ?? null }); } /** `judge.transcription`: transcribes the bot's speech for `response` checks. */ export function transcriber(_config: Config) { return new OpenAISTTService({ apiKey: process.env.OPENAI_API_KEY ?? null }); } ``` * `judge.eval` (and a simulation's `simulator`): an OpenAI-compatible LLM, or any `BaseClassifier`. Without a factory it is Ollama (`service: ollama`) running `gemma4:12b` locally. Set `explainer: false` to turn off the explainer that gives reasons for a `no`. * `user.speech`: the voice the user's turns are spoken in, when `user.modality: audio`. Synthesized turns are cached on disk. * `judge.transcription`: the transcriber for the bot's speech, when `judge.modality: audio`. Audio scenarios use upstream's keyless local defaults: [Kokoro](https://polvallverdu.github.io/pipedog/docs/services/tts/kokoro.md) (`service: kokoro`) for the user's voice, and [Moonshine](https://polvallverdu.github.io/pipedog/docs/services/stt/moonshine.md) (the default) or [local Whisper](https://polvallverdu.github.io/pipedog/docs/services/stt/whisper.md) (`service: whisper`) for the transcriber. They are optional peer dependencies of `@pipedog/evals`: install `@pipedog/kokoro`, `@pipedog/moonshine` or `@pipedog/whisper` with `onnxruntime-node` (Kokoro also needs `@echogarden/espeak-ng-emscripten`). Models download on first use into `~/.cache/pipedog`. WAV files work as `audio:` recordings; other formats need a decoder. ## Run it [#run-it] ```bash bunx pipedog-eval run scenarios/capital.yaml --bot-url ws://localhost:7860/ws bunx pipedog-eval run scenarios/ -v # every scenario in a folder, verbose ``` The command prints each scenario's configuration, then live status and timings. Each scenario's `.eval.log` goes in `--logs-dir`. The exit code is 1 if any scenario failed. `pipedog-eval suite manifest.yaml` runs a manifest of bots and their scenarios: it spawns each bot on its own port (with the current runtime, `{python} {bot} --port {port}` by default), runs the scenarios concurrently, and can repeat them to measure flakiness. `pipedog-eval judge` judges saved conversations against a simulation's criteria. ## From code [#from-code] The same run, without the CLI: ```ts title="run.ts" import { EvalScenarioFile, EvalScriptResult, EvalSession, EvalSessionParams, type EvalSimulationResult, } from "@pipedog/evals"; const file = await EvalScenarioFile.load("scenarios/capital.yaml"); for (const scenario of file) { const session: EvalSession = await EvalSession.fromScenario(scenario, "ws://localhost:7860/ws", { params: new EvalSessionParams({ defaultTimeoutMs: 30_000 }), }); session.eventHandler("on_progress", async (_session: unknown, progress: unknown) => { console.log(progress); }); const result = await session.run(); console.log(scenario.name, result.passed ? "PASS" : "FAIL"); if (result instanceof EvalScriptResult) { for (const failure of result.failures) console.log(` ${failure}`); } } ``` `EvalSession.fromScenario()` also takes `judge`, `userTts`, `botStt` and `personaLlm`, which override what the scenario configures. That's how a test can pass in fakes. ## Differences from upstream [#differences-from-upstream] * `python -m pipecat.evals` and `pipecat eval` are the `pipedog-eval` bin. A plain-ANSI table replaces rich's live dashboard. `.env` is loaded by a small built-in loader. * A `factory:` is `.`, imported with `import()`, rather than a dotted Python path. The service builders and `EvalScenarioFile.load()` are async. * Each suite run is a child process running `_session_subprocess` with the current runtime. * The deprecated pieces are left out: the `EvalTransport` server, `pipecat.evals.harness`, flat scenario files, `trigger_disconnect`, `stop_bot`, and `service: openai` / `service: cartesia` (use a factory). # Examples > Runnable voice bots in the repository, over WebRTC, over a WebSocket, on the phone, on Cloudflare Durable Objects, and mounted in each major framework. Source: https://polvallverdu.github.io/pipedog/docs/examples/ Each example is a complete project in the repository's [`examples`](https://github.com/polvallverdu/pipedog/tree/main/examples) folder, with an end-to-end test that runs it against mock provider APIs. Clone the repository and run `bun install` once, then start any of them from the repository root. | Example | Run it | What it is | | ------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [`smallwebrtc-voice-bot`](https://github.com/polvallverdu/pipedog/tree/main/examples/smallwebrtc-voice-bot) | `OPENAI_API_KEY=sk-... bun run --cwd examples/smallwebrtc-voice-bot start` | A voice assistant in the browser over SmallWebRTC, with OpenAI or Google Gemini services. | | [`websocket-voice-bot`](https://github.com/polvallverdu/pipedog/tree/main/examples/websocket-voice-bot) | `OPENAI_API_KEY=sk-... bun run --cwd examples/websocket-voice-bot start` | The same assistant over a WebSocket, with protobuf frames for `@pipecat-ai/websocket-transport`. | | [`twilio-phone-bot`](https://github.com/polvallverdu/pipedog/tree/main/examples/twilio-phone-bot) | `OPENAI_API_KEY=sk-... PUBLIC_URL=https://... bun run --cwd examples/twilio-phone-bot start` (or `start:deno`) | A phone bot on Twilio Media Streams: the TwiML webhook and the media WebSocket as one fetch handler, served with `serveFetch` on Bun or Node, or on Deno and Workers. See the [Twilio quickstart](https://polvallverdu.github.io/pipedog/docs/telephony/twilio-quickstart.md). | | [`workers-durable-voice-agent`](https://github.com/polvallverdu/pipedog/tree/main/examples/workers-durable-voice-agent) | `bun run --cwd examples/workers-durable-voice-agent dev` | Parallel voice agents on Cloudflare Workers, one Durable Object per session, over WebSocket. | | [`workers-realtime-voice-agent`](https://github.com/polvallverdu/pipedog/tree/main/examples/workers-realtime-voice-agent) | `bun run --cwd examples/workers-realtime-voice-agent deploy` | A browser voice agent over WebRTC on Cloudflare Workers: Cloudflare Realtime carries the audio, a Durable Object per call runs the bot. | | [`landing-demo`](https://github.com/polvallverdu/pipedog/tree/main/examples/landing-demo) | `bun run --cwd examples/landing-demo dev:fakes` | The live demo on the landing page: Workers AI Flux and Aura, Claude Haiku 5.5 with typed tools, Turnstile and a session budget, over WebSocket or WebRTC, with a SvelteKit page served by the same Worker. | | [`framework-snippets`](https://github.com/polvallverdu/pipedog/tree/main/examples/framework-snippets) | `bun test examples/framework-snippets` | The same voice endpoints mounted in Bun, Deno, Elysia, Express, Hono, Next.js, Nuxt, SvelteKit, TanStack Start and Vercel. See [Frameworks](https://polvallverdu.github.io/pipedog/docs/frameworks.md). | The mock provider APIs and the recorded question the tests use are in [`test-mocks`](https://github.com/polvallverdu/pipedog/tree/main/examples/test-mocks). The tests need no API keys; running an example against the real providers does. The two browser bots serve their page on [http://localhost:7860](http://localhost:7860). Set `VOICE_PROVIDER=google` and `GOOGLE_API_KEY` to use Gemini instead of OpenAI. The Workers example reads `OPENAI_API_KEY` from `.dev.vars`; its README shows how to stream a recording to a session. ## WebRTC or WebSocket? [#webrtc-or-websocket] The two browser bots are the same pipeline with a different transport. Pick by where your audio comes from: * **WebRTC** copes better with lossy networks: it runs over UDP, so a lost packet doesn't hold up the audio behind it. Browser users get lower latency, the browser's echo cancellation, and Opus compression. See the [voice bot guide](https://polvallverdu.github.io/pipedog/docs/voice-bot.md). * **WebSocket** is simpler: no ICE or TURN servers, it passes through HTTP proxies and load balancers, and it runs on Cloudflare Workers, where WebRTC can't. It is also the way to go for server-to-server audio and phone bridges. See the [WebSocket transport](https://polvallverdu.github.io/pipedog/docs/transports/websocket.md). # Actions > Run speech, hang-ups and your own code before or after the LLM responds at a node. Source: https://polvallverdu.github.io/pipedog/docs/flows/actions/ Actions are steps a node runs around the LLM's turn. List them in `preActions` (before the LLM responds) or `postActions` (after the bot has finished speaking). Each is an `ActionConfig` with a `type`. | Type | Does | Fields | | ------------------ | ------------------------------------------------ | -------------------------------------- | | `tts_say` | Speaks a fixed line. | `text`, optional `appendTextToContext` | | `end_conversation` | Ends the pipeline, optionally after a last line. | optional `text` | | `function` | Runs your handler inline in the pipeline. | `handler` | | anything else | A custom action you register. | `handler` and any extra fields | ```ts const actionsNode: NodeConfig = { taskMessages: [{ role: "developer", content: "Confirm the booking." }], preActions: [{ type: "tts_say", text: "One moment while I check." }], postActions: [ { // A custom action: the handler receives the action and the flow manager. type: "notify", channel: "#bookings", handler: async (action, fm) => { console.log(`notify ${String(action.channel)}: ${JSON.stringify(fm.state.slot)}`); }, }, { type: "end_conversation", text: "Goodbye!" }, ], }; ``` A handler is `(action, flowManager) => void | Promise` and receives the whole action object, so extra fields such as `channel` above reach it. ## Registering custom actions [#registering-custom-actions] A custom action needs a handler. Put it on the action, or register it once on the manager: ```ts flowManager.registerAction("notify", async (action, fm) => { console.log(`notify ${String(action.channel)}: ${JSON.stringify(fm.state)}`); }); ``` Post-actions wait until the bot has finished speaking, so with a TTS service in the pipeline they run after the response is heard. `end_conversation` stops the remaining actions. # Context strategies > Choose whether a node keeps, resets or summarizes the conversation history on entry. Source: https://polvallverdu.github.io/pipedog/docs/flows/context-strategies/ When the conversation enters a node, its **context strategy** decides what the LLM remembers. | Strategy | Effect | | ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `ContextStrategy.APPEND` | Keep the history and add the node's messages. The default. | | `ContextStrategy.RESET` | Drop the history; the LLM sees only the new node's messages (and the role message). | | `ContextStrategy.RESET_WITH_SUMMARY` | Reset, but include an LLM-written summary of the earlier conversation. Needs a `summaryPrompt`. Deprecated upstream: it still works and logs a warning once. | Set one on a node, or give the `FlowManager` a default for nodes that do not: ```ts const resetNode: NodeConfig = { taskMessages: [{ role: "developer", content: "Start the payment step." }], // Drop the earlier conversation; keep only this node's messages. contextStrategy: new ContextStrategyConfig({ strategy: ContextStrategy.RESET }), }; const managerWithDefault = new FlowManager({ worker, llm, contextAggregator: aggregators, // The default for nodes that do not set their own. contextStrategy: new ContextStrategyConfig({ strategy: ContextStrategy.APPEND }), }); ``` Reset when an earlier phase no longer matters (and would only cost tokens or confuse the model), such as moving from small talk to a payment step. Keep the history when the later steps need what the user said. In a [flow config](https://polvallverdu.github.io/pipedog/docs/flows/flow-configs.md), write `context_strategy: reset` on the node. State in `flowManager.state` is never reset by a context strategy, so values the conversation learned stay available to handlers and placeholders. # Dynamic flows > Build each node at runtime from what the conversation has learned. Source: https://polvallverdu.github.io/pipedog/docs/flows/dynamic-flows/ In a dynamic flow, nodes are functions that return a `NodeConfig`, and a tool's handler returns the node to go to next. Because the next node is built when the tool is called, it can depend on arguments, state or anything else you can compute. This flow books a slot in three steps. The hour node is built for the day the caller picked, and the last node confirms the slot and ends the call: ```ts interface Slot { day: string; hour: number; } function createDayNode(): NodeConfig { return { name: "day", taskMessages: [{ role: "developer", content: "Ask which weekday the caller prefers." }], functions: [ new FlowsFunctionSchema({ name: "choose_day", description: "Record the preferred weekday.", properties: { day: { type: "string" } }, required: ["day"], handler: async (args, fm) => { fm.state.day = args.day; // Build the next node from what the caller just said. return [{ day: args.day }, createHourNode(String(args.day))]; }, }), ], }; } function createHourNode(day: string): NodeConfig { return { name: "hour", taskMessages: [{ role: "developer", content: `Offer times on ${day} and ask which one.` }], functions: [ new FlowsFunctionSchema({ name: "choose_hour", description: "Record the chosen hour.", properties: { hour: { type: "integer" } }, required: ["hour"], handler: async (args, fm) => { const slot: Slot = { day, hour: Number(args.hour) }; fm.state.slot = slot; return [slot, createDoneNode(slot)]; }, }), ], }; } function createDoneNode(slot: Slot): NodeConfig { return { name: "done", taskMessages: [ { role: "developer", content: `Confirm ${slot.day} at ${slot.hour}:00, then say goodbye.` }, ], postActions: [{ type: "end_conversation" }], }; } await flowManager.setNodeFromConfig(createDayNode()); ``` Each handler returns `[result, nextNode]`. The result goes back to the LLM as the tool result; the manager then enters the next node, which swaps in its task messages and tools and asks the LLM to respond. A handler can return `null` as the node to stay put, or `NO_RESPONSE` to stay quiet. Start it with `flowManager.initialize(createDayNode())`, or enter it later with `setNodeFromConfig`. Use [flow configs](https://polvallverdu.github.io/pipedog/docs/flows/flow-configs.md) instead when the graph is fixed and you want it reviewable as data. # Flow configs > Describe a flow as YAML or JSON and join it to your handlers with Flow. Source: https://polvallverdu.github.io/pipedog/docs/flows/flow-configs/ A `FlowConfig` describes the graph as data: the nodes, the tools each offers, and where each tool leads. Your code only implements what the tools *do*. ```ts const YAML = ` initial_node: greeting nodes: greeting: role_message: You work for {{ clinic }}. task_messages: - role: developer content: Greet the caller and ask for their name. functions: - name: record_name transition_to: ask_reason ask_reason: task_messages: - role: developer content: Ask why {{ name }} is calling. context_strategy: reset `; async function record_name(fm: FlowManager, name: string) { fm.state.name = name; // The config decides the next node. return [{ status: "ok" }, TRANSITION_IN_YAML] as const; } Object.assign(record_name, { description: "Record the caller's name.", parameters: { name: { type: "string" } }, }); const flow = new Flow(FlowConfig.fromYaml(YAML), { handlers: { record_name } }); const configured = new FlowManager({ worker, llm, contextAggregator: aggregators, globalFunctions: flow.globalFunctions, }); configured.state.clinic = "Riverside Dental"; await configured.initialize(flow.initialNode); ``` `Flow` joins the config to the handlers it names. It checks every reference at construction (each tool exists and is a valid direct function, each action handler is callable) and reports all problems together in one `FlowReferenceError`. It then hands the manager ready-made nodes: `flow.initialNode`, `flow.node(name)` and `flow.globalFunctions`. A tool in a config returns `[result, TRANSITION_IN_YAML]`; the config's `transition_to` picks the next node. `transition_to` can also branch on a field of the result: ```yaml functions: - name: select_pizza_order transition_to: field: status cases: ok: confirm unavailable: choose_pizza default: confirm ``` A `transition_only` function needs no handler at all: give it a `description` and a `transition_to`, and it just moves the conversation. ## Loading [#loading] `FlowConfig.fromYaml(text)`, `fromJson(text)` and `modelValidate(object)` take text or parsed data. `fromFile` reads through a function you provide, which keeps the package runtime-agnostic: ```ts const fromDisk = FlowConfig.fromFile("flow.yaml", { readFile: (path) => readFileSync(path, "utf8"), }); ``` `!include` in YAML uses the same `readFile`, resolved against a `baseDir`. It can read any file the process can read, so use it only for configs you ship; load untrusted text with `fromJson`, or with `fromYaml` and no `baseDir`. Invalid configs throw `ValidationError` with every problem found, not only the first. `FlowConfig.modelJsonSchema()` returns the JSON Schema of the file format, for editor validation. ## Placeholders [#placeholders] Prompt text may contain `{{ key }}` placeholders: a node's `role_message`, the `content` of its `task_messages`, and the `text` of `tts_say` actions. The manager fills them from `flowManager.state` every time it enters the node. `{{ order.size }}` reads into a stored object, and `\{{ key }}` shows a literal `{{ key }}`. A key missing from state throws `FlowError` on entry. # Functions > Tools the LLM calls at a node, written as direct functions or explicit schemas, and how they choose the next node. Source: https://polvallverdu.github.io/pipedog/docs/flows/functions/ A function is a tool the LLM can call at a node. Its handler does work and returns `[result, nextNode]`: * `result` is delivered to the LLM as the tool result (any JSON value, or `null`). * `nextNode` is the `NodeConfig` to move to, `null` to stay on the node and let the LLM respond, or `NO_RESPONSE` to stay and **not** respond (the next reply is triggered by the user). ## Schemas [#schemas] The explicit form is a `FlowsFunctionSchema`, whose handler receives the arguments object and the flow manager. The [node example](https://polvallverdu.github.io/pipedog/docs/flows/nodes.md) uses it. ## Direct functions [#direct-functions] A **direct function** is a plain async function whose first parameter is `flowManager`; the other parameters are the tool's arguments, matched by name. JavaScript has no docstrings or type hints at runtime, so attach the description and the parameter schemas as properties: ```ts /** * A direct function: the first parameter is the flow manager, the rest are * the tool's arguments. Its description and parameter schemas are attached * as metadata, since JavaScript has no docstrings or type hints at runtime. */ async function bookSlot(fm: FlowManager, day: string, hour: number) { fm.state.slot = { day, hour }; return [{ booked: true }, createAskReasonNode()] as const; } Object.assign(bookSlot, { description: "Book an appointment slot.", parameters: { day: { type: "string", description: "Weekday name." }, hour: { type: "integer", description: "Hour of the day, 0-23." }, }, }); // Per-tool call options, as a decorator-style wrapper. const slowLookup = flowsToolOptions({ cancelOnInterruption: false, timeoutSecs: 30 })( Object.assign( async (fm: FlowManager, query: string) => { void fm; return [{ matches: [query] }, NO_RESPONSE] as const; }, { description: "Search the records.", parameters: { query: { type: "string" } } }, ), ); const node: NodeConfig = { taskMessages: [{ role: "developer", content: "Help the caller book a slot." }], functions: [bookSlot as any, slowLookup as any], }; ``` `flowsToolOptions` sets per-tool `cancelOnInterruption` (default `false` in Flows) and `timeoutSecs`. ## Global functions [#global-functions] Tools available at every node go in `globalFunctions` on the `FlowManager`. They are registered once and offered alongside each node's own functions. ## Errors [#errors] If a handler throws, the LLM receives an error result and the conversation stays on the current node. A tool built from a [flow config](https://polvallverdu.github.io/pipedog/docs/flows/flow-configs.md) must return `[result, TRANSITION_IN_YAML]` (the config chooses the destination); returning a node or a node name there is an error reported to the LLM. # Flows > Build structured conversations from nodes, functions and actions with @pipedog/flows, a port of Pipecat Flows. Source: https://polvallverdu.github.io/pipedog/docs/flows/ **Flows** keeps a conversation on rails. Instead of one long prompt with every tool available at every moment, you describe the conversation as a graph of **nodes**. Each node gives the LLM one job (its task messages), the few tools it may call there, and optional actions to run on the way in or out. When the LLM calls a tool, the tool does its work and can send the conversation to the next node. `@pipedog/flows` ports `pipecat.flows`. It works with any LLM service that does function calling: OpenAI, Anthropic and Google are covered by the package's end-to-end tests. ```bash bun add @pipedog/flows ``` ## Concepts [#concepts] | Concept | What it is | | -------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | | [Node](https://polvallverdu.github.io/pipedog/docs/flows/nodes.md) | One step of the conversation: a role, task messages, functions, actions and a context strategy. | | [Function](https://polvallverdu.github.io/pipedog/docs/flows/functions.md) | A tool the LLM can call at a node. It does work, and may name the next node. | | [Action](https://polvallverdu.github.io/pipedog/docs/flows/actions.md) | Something that happens around the LLM's turn at a node: speaking a line, ending the call, or your own code. | | [Context strategy](https://polvallverdu.github.io/pipedog/docs/flows/context-strategies.md) | What happens to the conversation history when a node is entered. | | Flow manager | `FlowManager` runs the graph: it holds the current node and the shared `state`, registers each node's tools with the LLM service, and performs transitions. | | Flow config | A flow described as YAML or JSON data, joined to your handlers with `Flow`. | ## Wiring it up [#wiring-it-up] `FlowManager` needs the pipeline worker (to queue frames), the LLM service, and the context aggregators: ```ts const flowManager = new FlowManager({ worker, llm, contextAggregator: aggregators, }); ``` Then start in a node with `flowManager.initialize(node)`. See [Nodes](https://polvallverdu.github.io/pipedog/docs/flows/nodes.md) for what a node contains, or [Dynamic flows](https://polvallverdu.github.io/pipedog/docs/flows/dynamic-flows.md) for a graph built at runtime. ## Two ways to define a flow [#two-ways-to-define-a-flow] * **In code.** Node functions return `NodeConfig` objects, and a tool returns the next node. Nothing is fixed ahead of time, so the next node can depend on what the user said. See [Dynamic flows](https://polvallverdu.github.io/pipedog/docs/flows/dynamic-flows.md). * **As data.** A `FlowConfig` lists the nodes and the edges between them, in YAML or JSON. Your code supplies only the tools' behavior. See [Flow configs](https://polvallverdu.github.io/pipedog/docs/flows/flow-configs.md). Both run on the same `FlowManager`. ## State [#state] `flowManager.state` is a plain object shared by every node and handler. Store what the conversation learns (a name, a booking) and read it back anywhere, including in [`{{ placeholders }}`](https://polvallverdu.github.io/pipedog/docs/flows/flow-configs/#placeholders) in a config. ## Differences from Pipecat [#differences-from-pipecat] The behavior matches upstream; the API follows TypeScript conventions: * `NodeConfig` and `ActionConfig` keys are camelCase (`taskMessages`, `preActions`). The YAML/JSON `FlowConfig` file format keeps upstream's snake_case keys. * A tool returns a two-element array, `[result, nextNode]`, where Python returns a tuple. * Handlers are called as `(args, flowManager)`. * A direct function's first parameter is named `flowManager`, and its description and parameter schemas are attached as metadata (JavaScript has no docstrings or type hints at runtime). * `FlowConfig.fromFile` and `!include` read files through a `readFile` function you pass in, so the package runs on every runtime. * Config validation is hand-written and throws `ValidationError`, listing every problem found. * Placeholders render with `String()` (objects as JSON) instead of Python's `str()`. * Deprecated upstream APIs are not ported. `RESET_WITH_SUMMARY` is kept and logs a deprecation warning once. # Nodes > A node is one step of a flow, with a role, task messages, functions and actions. Source: https://polvallverdu.github.io/pipedog/docs/flows/nodes/ A node is a `NodeConfig`: | Field | Meaning | | ---------------------------- | -------------------------------------------------------------------------------------------------------------------- | | `taskMessages` | Required. What the LLM should do at this node, as messages (usually `role: "developer"`). | | `name` | Optional. Shown in logs and used as `flowManager.currentNode`; generated if omitted. | | `roleMessage` | The bot's role or personality. Sent as the LLM's system instruction and kept across nodes until a node sets another. | | `functions` | Tools the LLM may call at this node, in addition to the manager's global functions. | | `preActions` / `postActions` | [Actions](https://polvallverdu.github.io/pipedog/docs/flows/actions.md) before and after the LLM's response. | | `contextStrategy` | How [the conversation history](https://polvallverdu.github.io/pipedog/docs/flows/context-strategies.md) changes on entering the node. | | `respondImmediately` | Whether the LLM answers as soon as the node is entered. Defaults to `true`; set `false` to wait for the user. | ```ts function createGreetingNode(): NodeConfig { return { name: "greeting", roleMessage: "You are a friendly receptionist at a dental clinic. Speak briefly.", taskMessages: [{ role: "developer", content: "Greet the caller and ask for their name." }], functions: [ new FlowsFunctionSchema({ name: "record_name", description: "Record the caller's name.", properties: { name: { type: "string", description: "The caller's name." } }, required: ["name"], // Handlers receive (args, flowManager) and return [result, nextNode]. handler: async (args, fm) => { fm.state.name = args.name; return [{ status: "ok" }, createAskReasonNode()]; }, }), ], }; } function createAskReasonNode(): NodeConfig { return { name: "ask_reason", taskMessages: [{ role: "developer", content: "Ask why they are calling." }], }; } await flowManager.initialize(createGreetingNode()); ``` ## Role and task messages [#role-and-task-messages] Keep the persona in `roleMessage` (set once, usually on the first node) and the step's job in `taskMessages`. A node's task messages are what makes the LLM stay on one topic: say what to do, what to collect, and when to call the node's functions. Only the functions of the **current** node (plus global functions) are offered to the LLM. Entering a node replaces the tool set. ## Entering a node [#entering-a-node] `flowManager.initialize(node)` enters the first node. A function enters the next one by returning it. You can also enter a node yourself with `flowManager.setNodeFromConfig(node)`, for example from an event handler. On entry the manager runs the node's pre-actions, applies the context strategy, registers the node's functions, asks the LLM to respond (unless `respondImmediately` is `false`), and finally runs the post-actions. `flowManager.currentNode` is the name of the active node. # Bun > Serve the pipedog WebRTC offer and WebSocket endpoints with Bun.serve. Source: https://polvallverdu.github.io/pipedog/docs/frameworks/bun/ `Bun.serve` routes both endpoints. `socket` is a fetch handler that upgrades `/ws` with the server Bun passes to `fetch`, and the `bunWebSocket` handlers deliver each socket's messages and close to the agent's `WebsocketConnection`. ## Install [#install] ```bash bun add @pipedog/core @pipedog/asyncio @pipedog/transport-smallwebrtc @pipedog/transport-websocket @pipedog/serializer-protobuf @pipedog/openai @pipedog/audio-silero @pipedog/audio-smart-turn ``` `@pipedog/audio-smart-turn` isn't imported anywhere: the default turn strategies load it. To run the snippet from a checkout instead, see [Installation](https://polvallverdu.github.io/pipedog/docs/installation/#from-source). ## Files [#files] The endpoints come from the shared module on the [overview](https://polvallverdu.github.io/pipedog/docs/frameworks/#the-two-endpoints). `voice.ts` re-exports it: ```ts title="voice.ts" export { offer, runSocketAgent, socket } from "../shared/voice.ts"; ``` ```ts title="server.ts" import { bunWebSocket, type UpgradingServer } from "@pipedog/transport-websocket/fetch"; import { offer, socket } from "./voice.ts"; const config = { fetch(request: Request, server: UpgradingServer) { const { pathname } = new URL(request.url); if (pathname === "/api/offer") return offer(request); // `socket` upgrades with the server Bun passes to `fetch`. if (pathname === "/ws") return socket(request, server); return new Response("not found", { status: 404 }); }, // Hands each upgraded socket's messages and close to its connection. websocket: bunWebSocket, }; // `bun server.ts` serves the default export, on port 3000 or $PORT. export default config; ``` Bun upgrades on its server, not from the `Request` alone, so pass `server` on to `socket` (or to `upgradeWebSocket(request, { server })` from `@pipedog/transport-websocket/fetch`) and serve with `websocket: bunWebSocket`. To serve any fetch handler with WebSocket upgrades on Bun, Node or Deno without this wiring, use `serveFetch()` from `@pipedog/transport-websocket/serve`; see [WebSocket](https://polvallverdu.github.io/pipedog/docs/transports/websocket.md). ## Run it [#run-it] ```bash OPENAI_API_KEY=sk-... bun server.ts ``` Bun serves the default export on port 3000, or on `$PORT`. The offer is at `POST /api/offer` and the WebSocket at `/ws`. ## Deploy [#deploy] Run it as a long-lived Bun process on a host that allows UDP traffic, for the offer's media. To serve the same config on Vercel Functions, see [Vercel](https://polvallverdu.github.io/pipedog/docs/frameworks/vercel.md). ## Authentication [#authentication] Neither route has auth. Check the session in `fetch` before `offer` and `socket` run, as in [Authentication](https://polvallverdu.github.io/pipedog/docs/frameworks/#authentication). # Cloudflare Workers > Serve the pipedog WebSocket endpoint from a Worker, with one Durable Object per session. Source: https://polvallverdu.github.io/pipedog/docs/frameworks/cloudflare-workers/ Workers upgrade a WebSocket from fetch, so `websocketFetchHandler` works there as it is. Workers have no UDP, so the WebRTC offer doesn't: audio goes over the WebSocket, as protobuf frames. For WebRTC users, [Cloudflare Realtime](https://polvallverdu.github.io/pipedog/docs/transports/cloudflare-realtime.md) carries the media and hands the bot PCM over WebSockets. A voice session is a long-lived connection with state, so each one runs in its own Durable Object. The Worker routes each request to an object by session name. The full guide, with turn detection, reconnection and limits, is [Voice agents on Workers and Durable Objects](https://polvallverdu.github.io/pipedog/docs/deployment/cloudflare-workers.md). ## Install [#install] ```bash bun add @pipedog/core @pipedog/asyncio @pipedog/transport-websocket @pipedog/serializer-protobuf @pipedog/openai ``` To run it from a checkout instead, see [Installation](https://polvallverdu.github.io/pipedog/docs/installation/#from-source). The Worker needs the `nodejs_compat` compatibility flag. ## The shape [#the-shape] ```text client ──WebSocket──► Worker ──getByName(id)──► VoiceSession (Durable Object) │ FastAPIWebsocketTransport │ OpenAI realtime transcription (turns) │ chat, speech synthesis ``` The Durable Object turns a connection into a fetch handler and returns its result from `fetch`: ```ts import { DurableObject } from "cloudflare:workers"; import { websocketFetchHandler } from "@pipedog/transport-websocket/fetch"; export class VoiceSession extends DurableObject { readonly #handler = websocketFetchHandler((connection) => runVoiceAgent(connection, this.env)); override fetch(request: Request): Promise { return this.#handler(request); } } ``` The Worker routes each session to an object by name. The same name always reaches the same object: ```ts export default { async fetch(request: Request, env: Env): Promise { const id = new URL(request.url).pathname.split("/")[3]; return env.VOICE_SESSION.getByName(id).fetch(request); }, } satisfies ExportedHandler; ``` `runVoiceAgent` builds the pipeline on the connection with `FastAPIWebsocketTransport`. It detects turns on the provider's side, not with Silero and Smart Turn: the local models don't fit in a Worker's size limit. ## Run it [#run-it] The complete example is [`examples/workers-durable-voice-agent`](https://github.com/polvallverdu/pipedog/tree/main/examples/workers-durable-voice-agent): ```bash cd examples/workers-durable-voice-agent echo "OPENAI_API_KEY=sk-..." > .dev.vars bun run dev ``` Its tests run against workerd through `wrangler dev`. To deploy, see [Run it](https://polvallverdu.github.io/pipedog/docs/deployment/cloudflare-workers/#run-it) in the full guide. ## Authentication [#authentication] The Worker has no auth. Check the session in the Worker's `fetch` before it reaches the Durable Object, as in [Authentication](https://polvallverdu.github.io/pipedog/docs/frameworks/#authentication). Browsers can't set headers on a WebSocket, so take the token from the URL. # Deno > Serve the pipedog WebRTC offer and WebSocket endpoints with Deno.serve. Source: https://polvallverdu.github.io/pipedog/docs/frameworks/deno/ `Deno.serve` routes both endpoints. `socket` is a fetch handler, and Deno upgrades a WebSocket from fetch, so the upgrade uses `Deno.upgradeWebSocket` under the hood. ## Install [#install] ```bash deno add npm:@pipedog/core npm:@pipedog/asyncio npm:@pipedog/transport-smallwebrtc npm:@pipedog/transport-websocket npm:@pipedog/serializer-protobuf npm:@pipedog/openai npm:@pipedog/audio-silero npm:@pipedog/audio-smart-turn ``` `@pipedog/audio-smart-turn` isn't imported anywhere: the default turn strategies load it. To run the snippet from a checkout instead, see [Installation](https://polvallverdu.github.io/pipedog/docs/installation/#from-source). ## Files [#files] The endpoints come from the shared module on the [overview](https://polvallverdu.github.io/pipedog/docs/frameworks/#the-two-endpoints). `voice.ts` re-exports it: ```ts title="voice.ts" export { offer, runSocketAgent, socket } from "../shared/voice.ts"; ``` ```ts title="main.ts" import { offer, socket } from "./voice.ts"; // `deno run -A main.ts` starts it on port 3000, or on $PORT. Deno.serve({ port: Number(Deno.env.get("PORT") ?? 3000) }, (request) => { const { pathname } = new URL(request.url); if (pathname === "/api/offer") return offer(request); if (pathname === "/ws") return socket(request); return new Response("not found", { status: 404 }); }); ``` ## Run it [#run-it] ```bash OPENAI_API_KEY=sk-... deno run -A main.ts ``` It listens on port 3000, or on `$PORT`. The offer is at `POST /api/offer` and the WebSocket at `/ws`. ## Deploy [#deploy] Run it as a long-lived Deno process on a host that allows UDP traffic, for the offer's media. A host that runs Deno without UDP can still serve `socket`. ## Authentication [#authentication] Neither route has auth. Check the session in the `Deno.serve` handler before either endpoint runs, as in [Authentication](https://polvallverdu.github.io/pipedog/docs/frameworks/#authentication). # Elysia > Mount the pipedog WebRTC offer endpoint in an Elysia app on Bun. Source: https://polvallverdu.github.io/pipedog/docs/frameworks/elysia/ Elysia runs on Bun. The offer is a route that passes Elysia's web `Request` to `offer`. This snippet mounts only the offer. ## Install [#install] ```bash bun add @pipedog/core @pipedog/asyncio @pipedog/transport-smallwebrtc @pipedog/transport-websocket @pipedog/serializer-protobuf @pipedog/openai @pipedog/audio-silero @pipedog/audio-smart-turn elysia ``` `@pipedog/audio-smart-turn` isn't imported anywhere: the default turn strategies load it. To run the snippet from a checkout instead, see [Installation](https://polvallverdu.github.io/pipedog/docs/installation/#from-source). ## Files [#files] The endpoints come from the shared module on the [overview](https://polvallverdu.github.io/pipedog/docs/frameworks/#the-two-endpoints). `voice.ts` re-exports it: ```ts title="voice.ts" export { offer, runSocketAgent, socket } from "../shared/voice.ts"; ``` ```ts title="server.ts" import { Elysia } from "elysia"; import { offer } from "./voice.ts"; export const app = new Elysia().post("/api/offer", ({ request }) => offer(request)); // `bun server.ts` starts it on port 3000. if (import.meta.main) { app.listen(3000); } ``` ## Run it [#run-it] ```bash OPENAI_API_KEY=sk-... bun server.ts ``` It listens on port 3000, at `POST /api/offer`. ## Deploy [#deploy] Run it as a long-lived Bun process on a host that allows UDP traffic, for the offer's media. ## WebSocket [#websocket] Elysia has its own WebSocket route API (`.ws()`), which this snippet doesn't use. To serve WebSocket sessions, wire `.ws()` to a `WebsocketConnection` the way the [Bun](https://polvallverdu.github.io/pipedog/docs/frameworks/bun.md) snippet does, or run the Bun snippet next to this app. ## Authentication [#authentication] The route has no auth. Wrap `offer` with a session check before it runs, or use an Elysia `beforeHandle` hook, as in [Authentication](https://polvallverdu.github.io/pipedog/docs/frameworks/#authentication). # Express > Mount the pipedog WebRTC offer endpoint in an Express 5 app. Source: https://polvallverdu.github.io/pipedog/docs/frameworks/express/ Express 5 mounts the offer through `nodeRequestListener`, which turns a fetch handler into a Node request listener. This snippet mounts only the offer. ## Install [#install] ```bash bun add @pipedog/core @pipedog/asyncio @pipedog/transport-smallwebrtc @pipedog/transport-websocket @pipedog/serializer-protobuf @pipedog/openai @pipedog/audio-silero @pipedog/audio-smart-turn express ``` `@pipedog/audio-smart-turn` isn't imported anywhere: the default turn strategies load it. To run the snippet from a checkout instead, see [Installation](https://polvallverdu.github.io/pipedog/docs/installation/#from-source). ## Files [#files] The endpoints come from the shared module on the [overview](https://polvallverdu.github.io/pipedog/docs/frameworks/#the-two-endpoints). `voice.ts` re-exports it: ```ts title="voice.ts" export { offer, runSocketAgent, socket } from "../shared/voice.ts"; ``` ```ts title="server.ts" import { nodeRequestListener } from "@pipedog/transport-smallwebrtc/http"; import express, { type Express } from "express"; import { offer } from "./voice.ts"; const app: Express = express(); // Don't add express.json(): the offer handler reads the body itself. app.post("/api/offer", nodeRequestListener(offer)); export default app; // `node server.ts` starts it on port 3000. if (import.meta.main) { app.listen(3000); } ``` Don't add `express.json()` in front of the route: the offer reads the body itself, and a body parser would consume it first. ## Run it [#run-it] ```bash OPENAI_API_KEY=sk-... node server.ts ``` It listens on port 3000. `bun server.ts` works too. Node runs the TypeScript with its built-in type stripping, in a workspace; see [Runtimes](https://polvallverdu.github.io/pipedog/docs/guides/runtimes.md). ## Deploy [#deploy] Run it as a long-lived Node or Bun process on a host that allows UDP traffic, for the offer's media. ## WebSocket [#websocket] This snippet mounts only the offer. To serve WebSocket sessions too, serve `socket` on a second port with `serveFetch(socket, { port })` from `@pipedog/transport-websocket/serve`, which upgrades WebSockets on Node (with the `ws` package), Bun and Deno, or run a [Bun](https://polvallverdu.github.io/pipedog/docs/frameworks/bun.md) or [Deno](https://polvallverdu.github.io/pipedog/docs/frameworks/deno.md) server for it. See [WebSocket](https://polvallverdu.github.io/pipedog/docs/transports/websocket.md). ## Authentication [#authentication] The route has no auth. Wrap `offer` with a session check before `nodeRequestListener`, or add an Express middleware in front of the route, as in [Authentication](https://polvallverdu.github.io/pipedog/docs/frameworks/#authentication). # Hono > Mount the pipedog WebRTC offer and WebSocket endpoints in a Hono app on Bun. Source: https://polvallverdu.github.io/pipedog/docs/frameworks/hono/ Hono routes the offer, and upgrades `/ws` with its own WebSocket helper and pipedog's Hono adapter, `honoWebsocket`. This snippet runs on Bun. ## Install [#install] ```bash bun add @pipedog/core @pipedog/asyncio @pipedog/transport-smallwebrtc @pipedog/transport-websocket @pipedog/serializer-protobuf @pipedog/openai @pipedog/audio-silero @pipedog/audio-smart-turn hono ``` `@pipedog/audio-smart-turn` isn't imported anywhere: the default turn strategies load it. To run the snippet from a checkout instead, see [Installation](https://polvallverdu.github.io/pipedog/docs/installation/#from-source). ## Files [#files] The endpoints come from the shared module on the [overview](https://polvallverdu.github.io/pipedog/docs/frameworks/#the-two-endpoints). `voice.ts` re-exports it: ```ts title="voice.ts" export { offer, runSocketAgent, socket } from "../shared/voice.ts"; ``` ```ts title="app.ts" import { honoWebsocket } from "@pipedog/transport-websocket/hono"; import { Hono } from "hono"; import { upgradeWebSocket, websocket } from "hono/bun"; import { offer, runSocketAgent } from "./voice.ts"; const app = new Hono(); app.post("/api/offer", (c) => offer(c.req.raw)); app.get( "/ws", upgradeWebSocket(() => honoWebsocket(runSocketAgent)), ); // Bun serves the app; on Node, use @hono/node-server with the same routes. export default { fetch: app.fetch, websocket }; ``` ## Run it [#run-it] ```bash OPENAI_API_KEY=sk-... bun app.ts ``` Bun serves the default export on port 3000, or on `$PORT`. The offer is at `POST /api/offer` and the WebSocket at `/ws`. ## Other runtimes [#other-runtimes] * **Node.** Serve the app with `@hono/node-server`, with the same routes. `hono/bun`'s `upgradeWebSocket` is Bun only, so use Hono's Node WebSocket helper (`@hono/node-ws`) with `honoWebsocket` for `/ws`, or serve `socket` on its own with `serveFetch(socket, { port })` from `@pipedog/transport-websocket/serve`, which upgrades on Node with the `ws` package. * **Cloudflare Workers.** `socket` is a fetch handler that Workers can upgrade, so mount it directly, without the Bun helper: `app.get("/ws", (c) => socket(c.req.raw))`. Leave the offer, and the WebRTC transport, out of a Workers build: Workers have no UDP. A session that should outlive one request belongs in a Durable Object; see [Cloudflare Workers](https://polvallverdu.github.io/pipedog/docs/frameworks/cloudflare-workers.md). ## Deploy [#deploy] Run it as a long-lived Bun process. The offer's media runs over UDP, so a serverless host can't carry it. ## Authentication [#authentication] Neither route has auth. Wrap `offer` and the socket route with a session check before they run, as in [Authentication](https://polvallverdu.github.io/pipedog/docs/frameworks/#authentication). Hono middleware on both routes works too. # Overview > Mount a pipedog voice agent in SvelteKit, Hono, Bun, Next.js, TanStack Start, Cloudflare Workers, Express, Elysia, Nuxt, Deno or Vercel. Source: https://polvallverdu.github.io/pipedog/docs/frameworks/ pipedog's voice endpoints are fetch handlers: a function from a `Request` to a `Response`. Each framework only has to route a request to them. Every snippet in this section is in [`examples/framework-snippets`](https://github.com/polvallverdu/pipedog/tree/main/examples/framework-snippets), and the test suite there mounts each one and checks it. ## The two endpoints [#the-two-endpoints] * **`offer`**: WebRTC signalling for [SmallWebRTC](https://polvallverdu.github.io/pipedog/docs/transports/smallwebrtc.md). The browser POSTs its SDP offer and gets the answer. The media then runs over UDP, so the server has to be a long-lived process, not a serverless function. * **`socket`**: a WebSocket with protobuf audio frames, for clients that stream audio over a socket (see [WebSocket](https://polvallverdu.github.io/pipedog/docs/transports/websocket.md)). It works wherever the runtime can upgrade a request from fetch. Where it can't, the framework's page says where to mount it instead. The agent behind both is the same: speech in, OpenAI transcription, chat and speech synthesis, speech out, as in the [voice bot guide](https://polvallverdu.github.io/pipedog/docs/voice-bot.md). It needs `OPENAI_API_KEY` in the environment, and it reads `OPENAI_BASE_URL` if you set one. Silero detects speech, and the default turn strategies end the turn with Smart Turn (see [Turn detection](https://polvallverdu.github.io/pipedog/docs/concepts/turn-detection.md)). Every framework page imports the endpoints from this module: ```ts title="shared/voice.ts" // The two voice endpoints, as fetch handlers. Each framework's snippet mounts // them; see the framework folders. import { createTask } from "@pipedog/asyncio"; import { TransportParams } from "@pipedog/core/transports/base_transport"; import { ProtobufFrameSerializer } from "@pipedog/serializer-protobuf/protobuf"; import { smallWebRTCOfferHandler } from "@pipedog/transport-smallwebrtc/http"; import { SmallWebRTCRequestHandler } from "@pipedog/transport-smallwebrtc/request_handler"; import { SmallWebRTCTransport } from "@pipedog/transport-smallwebrtc/transport"; import { FastAPIWebsocketParams, FastAPIWebsocketTransport, } from "@pipedog/transport-websocket/fastapi"; import { websocketFetchHandler } from "@pipedog/transport-websocket/fetch"; import type { WebsocketConnection } from "@pipedog/transport-websocket/socket"; import { runAgent } from "./agent.ts"; /** * WebRTC signalling: POST an SDP offer, get the answer. The media runs over UDP, * so this needs a long-lived server, not serverless functions. */ export const offer = smallWebRTCOfferHandler(new SmallWebRTCRequestHandler(), (connection) => { createTask( () => runAgent( new SmallWebRTCTransport( connection, new TransportParams({ audioInEnabled: true, audioOutEnabled: true }), ), ), "voice-agent", ); }); /** Runs one agent on a WebSocket connection, with protobuf audio frames. */ export function runSocketAgent(connection: WebsocketConnection): Promise { return runAgent( new FastAPIWebsocketTransport( connection, new FastAPIWebsocketParams({ audioInEnabled: true, audioOutEnabled: true, serializer: new ProtobufFrameSerializer(), }), ), ); } /** A WebSocket endpoint as a fetch handler, where the runtime upgrades from fetch. */ export const socket = websocketFetchHandler(runSocketAgent); ``` The module builds the agent with `runAgent` from `shared/agent.ts`, which is the pipeline above. ## Authentication [#authentication] pipedog's endpoints don't authenticate: anyone who can reach them can start a session, on your provider keys. That's the same split as Pipecat, whose transports don't authenticate either; its development runner mints one-time tokens for its WebSocket endpoints, and production auth comes from the application or Pipecat Cloud. Put your own check in front of each endpoint, in the framework, so it runs before a session starts: ```ts /** Rejects requests without a valid session token before the endpoint runs. */ function requireSession(endpoint: (request: Request) => Promise) { return async (request: Request): Promise => { const url = new URL(request.url); // Browsers can't set headers on a WebSocket, so a socket brings its token in the URL. const token = request.headers.get("authorization")?.replace(/^Bearer /, "") ?? url.searchParams.get("token"); if (!token || !(await verifySessionToken(token))) { return new Response("unauthorized", { status: 401 }); } return endpoint(request); }; } export const authedOffer = requireSession(offer); export const authedSocket = requireSession(socket); ``` `verifySessionToken` is yours: a signed token your app issues when it creates the session, short-lived and used once, as Pipecat's runner does. Add a cap on concurrent sessions and on session length there too, since each session spends provider credits. ## Compare [#compare] | Framework | Runtime(s) | `offer` (WebRTC) | `socket` (WebSocket) | Serverless? | Tested how | | --------------------------------------------------------- | ------------------------------------------- | --------------------------------- | ------------------------ | ----------- | ----------------------------------------------- | | [SvelteKit](https://polvallverdu.github.io/pipedog/docs/frameworks/sveltekit.md) | Node, Bun, Deno (`adapter-node` or similar) | Yes | Not in the snippet | No | Route handler called directly | | [Hono](https://polvallverdu.github.io/pipedog/docs/frameworks/hono.md) | Bun (Node and Workers with other adapters) | Yes | Yes, with `hono/bun` | No | Served with `Bun.serve`, both routes | | [Bun](https://polvallverdu.github.io/pipedog/docs/frameworks/bun.md) | Bun | Yes | Yes | No | Served with `Bun.serve`, both routes | | [Next.js](https://polvallverdu.github.io/pipedog/docs/frameworks/nextjs.md) | Node | Yes | No, mount it elsewhere | No | Route handler called directly | | [TanStack Start](https://polvallverdu.github.io/pipedog/docs/frameworks/tanstack-start.md) | Node, Bun | Yes | No, mount it elsewhere | No | Server route handler called directly | | [Cloudflare Workers](https://polvallverdu.github.io/pipedog/docs/frameworks/cloudflare-workers.md) | workerd | No, no UDP | Yes, in a Durable Object | Yes | Its own example, run against workerd | | [Express](https://polvallverdu.github.io/pipedog/docs/frameworks/express.md) | Node, Bun | Yes | Not in the snippet | No | `app.listen`, then fetch | | [Elysia](https://polvallverdu.github.io/pipedog/docs/frameworks/elysia.md) | Bun | Yes | Not in the snippet | No | `app.handle` | | [Nuxt](https://polvallverdu.github.io/pipedog/docs/frameworks/nuxt.md) | Node, Bun, Deno (Nitro presets) | Yes | Not in the snippet | No | Handler mounted on h3 | | [Deno](https://polvallverdu.github.io/pipedog/docs/frameworks/deno.md) | Deno | Yes | Yes | No | `deno run`, both routes, when Deno is installed | | [Vercel](https://polvallverdu.github.io/pipedog/docs/frameworks/vercel.md) | Bun runtime on Vercel Functions | Route mounted, UDP media untested | Yes | Yes | Served with `Bun.serve`, both routes | ## What is tested [#what-is-tested] The test suite runs on Bun, and runs the Deno snippet with Deno when it's installed. It checks two things for each snippet: the offer route reaches pipedog's handler (a framework's own 404 wouldn't return pipedog's message), and the WebSocket route accepts an upgrade where the snippet mounts one. The Next.js, TanStack Start, Nuxt and SvelteKit route files are called as the handlers they export; the others run as servers. The Cloudflare Workers example has its own tests, which run against workerd through `wrangler dev`. Not tested here: a deployment on any of these hosts, a full TanStack Start, Next.js, Nuxt or SvelteKit build, and the WebRTC media on the serverless hosts (Vercel, Next.js on Vercel, Nuxt's serverless presets), which the UDP requirement rules out. ## Frameworks [#frameworks] A `+server.ts` route for the offer. The offer and a WebSocket route, on Bun. `Bun.serve` with the offer and a WebSocket upgrade. A route handler on the Node.js runtime. A server route for the offer. A WebSocket agent per session, in a Durable Object. The offer through `nodeRequestListener`. The offer as an Elysia route. A Nitro server route for the offer. `Deno.serve` with both endpoints. A Bun server on Vercel Functions. # Next.js > Mount the pipedog WebRTC offer endpoint in a Next.js route handler on the Node.js runtime. Source: https://polvallverdu.github.io/pipedog/docs/frameworks/nextjs/ An App Router route handler handles `POST /api/offer` on the Node.js runtime. It passes the web `Request` to `offer` unchanged. ## Install [#install] ```bash bun add @pipedog/core @pipedog/asyncio @pipedog/transport-smallwebrtc @pipedog/transport-websocket @pipedog/serializer-protobuf @pipedog/openai @pipedog/audio-silero @pipedog/audio-smart-turn ``` `@pipedog/audio-smart-turn` isn't imported anywhere: the default turn strategies load it. To run the snippet from a checkout instead, see [Installation](https://polvallverdu.github.io/pipedog/docs/installation/#from-source). ## Files [#files] The endpoints come from the shared module on the [overview](https://polvallverdu.github.io/pipedog/docs/frameworks/#the-two-endpoints). `@/lib/voice` re-exports it: ```ts title="lib/voice.ts" export { offer, runSocketAgent, socket } from "../../shared/voice.ts"; ``` ```ts title="app/api/offer/route.ts" import { offer } from "@/lib/voice"; // The Node.js runtime, not the edge: the media runs over UDP, and the server has // to stay up for the call. export const runtime = "nodejs"; export const POST = (request: Request) => offer(request); ``` ## Run it [#run-it] ```bash OPENAI_API_KEY=sk-... bunx next dev ``` The browser then POSTs its SDP offer to `/api/offer`. ## Deploy [#deploy] The route sets `runtime = "nodejs"`, not the edge, because the offer's media runs over UDP and the server has to stay up for the call. Run it with `next start` on a long-lived server. Vercel's serverless functions don't carry the UDP media. ## WebSocket [#websocket] A Next.js route handler can't upgrade a WebSocket. Mount `socket` on a server that can upgrade: a [Bun](https://polvallverdu.github.io/pipedog/docs/frameworks/bun.md), Node or [Deno](https://polvallverdu.github.io/pipedog/docs/frameworks/deno.md) server next to the Next.js app, or [Cloudflare Workers](https://polvallverdu.github.io/pipedog/docs/frameworks/cloudflare-workers.md). ## Authentication [#authentication] The route has no auth. Wrap `offer` with a session check before it runs, as in [Authentication](https://polvallverdu.github.io/pipedog/docs/frameworks/#authentication). # Nuxt > Mount the pipedog WebRTC offer endpoint in a Nuxt (Nitro) server route. Source: https://polvallverdu.github.io/pipedog/docs/frameworks/nuxt/ A Nitro server route handles `POST /api/offer`. The `.post.ts` suffix limits it to POST. `toWebRequest` turns the h3 event into a web `Request` for `offer`. ## Install [#install] ```bash bun add @pipedog/core @pipedog/asyncio @pipedog/transport-smallwebrtc @pipedog/transport-websocket @pipedog/serializer-protobuf @pipedog/openai @pipedog/audio-silero @pipedog/audio-smart-turn ``` `@pipedog/audio-smart-turn` isn't imported anywhere: the default turn strategies load it. To run the snippet from a checkout instead, see [Installation](https://polvallverdu.github.io/pipedog/docs/installation/#from-source). ## Files [#files] The endpoints come from the shared module on the [overview](https://polvallverdu.github.io/pipedog/docs/frameworks/#the-two-endpoints). `~/server/utils/voice` re-exports it: ```ts title="server/utils/voice.ts" export { offer, runSocketAgent, socket } from "../../../shared/voice.ts"; ``` ```ts title="server/api/offer.post.ts" import { offer } from "~/server/utils/voice"; // Nitro's Node, Bun or Deno preset: the media runs over UDP, so a serverless // preset won't carry it. `toWebRequest` is auto-imported from h3. export default defineEventHandler((event) => offer(toWebRequest(event))); ``` Nuxt auto-imports `defineEventHandler` and `toWebRequest`, so the handler doesn't import them. The snippet ships `server/auto-imports.d.ts` so it typechecks without a Nuxt project. A Nuxt app generates those declarations itself. ## Run it [#run-it] ```bash OPENAI_API_KEY=sk-... bunx nuxt dev ``` The browser then POSTs its SDP offer to `/api/offer`. ## Deploy [#deploy] Use a Nitro preset that runs a Node, Bun or Deno server, such as `node-server` (the default), `bun` or `deno-server`. The serverless presets don't carry the UDP media. ## WebSocket [#websocket] This snippet mounts only the offer. Nitro has its own WebSocket handlers, which this snippet doesn't use. To serve WebSocket sessions, run a [Bun](https://polvallverdu.github.io/pipedog/docs/frameworks/bun.md) or [Deno](https://polvallverdu.github.io/pipedog/docs/frameworks/deno.md) server next to the Nuxt app, or serve `socket` with `serveFetch()` from `@pipedog/transport-websocket/serve`, which upgrades on Node, Bun and Deno. ## Authentication [#authentication] The route has no auth. Wrap `offer` with a session check before it runs, as in [Authentication](https://polvallverdu.github.io/pipedog/docs/frameworks/#authentication). A Nitro server middleware works too. # SvelteKit > Mount the pipedog WebRTC offer endpoint in a SvelteKit +server.ts route. Source: https://polvallverdu.github.io/pipedog/docs/frameworks/sveltekit/ A `+server.ts` route handles `POST /api/offer`. SvelteKit hands the route a web `Request`, so the route passes it to `offer` unchanged. ## Install [#install] ```bash bun add @pipedog/core @pipedog/asyncio @pipedog/transport-smallwebrtc @pipedog/transport-websocket @pipedog/serializer-protobuf @pipedog/openai @pipedog/audio-silero @pipedog/audio-smart-turn @sveltejs/kit ``` `@pipedog/audio-smart-turn` isn't imported anywhere: the default turn strategies load it. To run the snippet from a checkout instead, see [Installation](https://polvallverdu.github.io/pipedog/docs/installation/#from-source). ## Files [#files] The endpoints come from the shared module on the [overview](https://polvallverdu.github.io/pipedog/docs/frameworks/#the-two-endpoints). `$lib/voice` re-exports it: ```ts title="src/lib/voice.ts" export { offer, runSocketAgent, socket } from "../../../shared/voice.ts"; ``` ```ts title="src/routes/api/offer/+server.ts" import type { RequestHandler } from "@sveltejs/kit"; import { offer } from "$lib/voice"; // Use adapter-node (or any Node, Bun or Deno server): the media runs over UDP, // which serverless adapters don't carry. export const POST: RequestHandler = ({ request }) => offer(request); ``` In your app, put `shared/voice.ts` and `shared/agent.ts` in `src/lib/` and drop the re-export. ## Run it [#run-it] ```bash OPENAI_API_KEY=sk-... bunx vite dev ``` The browser then POSTs its SDP offer to `/api/offer`. ## Deploy [#deploy] Use an adapter that runs a Node, Bun or Deno server, such as `@sveltejs/adapter-node` (`bun add -d @sveltejs/adapter-node`, then set it in `svelte.config.js`). The serverless adapters (Vercel, Netlify, Cloudflare) don't carry the UDP media. ## WebSocket [#websocket] This snippet mounts only the offer. To serve `socket` too, run it on a server that upgrades WebSockets next to the SvelteKit app: [Bun](https://polvallverdu.github.io/pipedog/docs/frameworks/bun.md), [Deno](https://polvallverdu.github.io/pipedog/docs/frameworks/deno.md), or `serveFetch(socket, { port })` from `@pipedog/transport-websocket/serve` on Node, Bun or Deno. ## Authentication [#authentication] The route has no auth. Wrap `offer` with a session check before it runs, as in [Authentication](https://polvallverdu.github.io/pipedog/docs/frameworks/#authentication). # TanStack Start > Mount the pipedog WebRTC offer endpoint in a TanStack Start server route. Source: https://polvallverdu.github.io/pipedog/docs/frameworks/tanstack-start/ A server route handles `POST /api/offer`. It is a file route with `server.handlers` and no component. TanStack Start passes each handler the web `Request`, and the handler passes it to `offer` unchanged. The snippet is checked against `@tanstack/react-start` 1.168.60 and `@tanstack/react-router` 1.170.41. ## Install [#install] ```bash bun add @pipedog/core @pipedog/asyncio @pipedog/transport-smallwebrtc @pipedog/transport-websocket @pipedog/serializer-protobuf @pipedog/openai @pipedog/audio-silero @pipedog/audio-smart-turn @tanstack/react-router ``` `@pipedog/audio-smart-turn` isn't imported anywhere: the default turn strategies load it. To run the snippet from a checkout instead, see [Installation](https://polvallverdu.github.io/pipedog/docs/installation/#from-source). `@tanstack/react-router` is already in a TanStack Start project. ## Files [#files] The endpoints come from the shared module on the [overview](https://polvallverdu.github.io/pipedog/docs/frameworks/#the-two-endpoints). `~/lib/voice` re-exports it. The snippet's tsconfig maps `~/lib/*` to `src/lib/*`. Use a relative import if your project has no such alias. ```ts title="src/lib/voice.ts" export { offer, runSocketAgent, socket } from "../../../shared/voice.ts"; ``` ```ts title="src/routes/api/offer.ts" import { createFileRoute } from "@tanstack/react-router"; import { offer } from "~/lib/voice"; // A server route: `server.handlers` and no component. Deploy to a Node, Bun or // Deno server: the media runs over UDP, which serverless targets don't carry. export const Route = createFileRoute("/api/offer")({ server: { handlers: { POST: ({ request }) => offer(request), }, }, }); ``` The snippet also ships `src/routeTree.gen.d.ts`. In a real project the router plugin generates `src/routeTree.gen.ts`, which registers the route's path for `createFileRoute` and loads the types of the `server` option. The snippet has no generator, so the declaration stands in for it. Don't copy it into your app. ## Run it [#run-it] ```bash OPENAI_API_KEY=sk-... bunx vite dev ``` The router plugin picks up `src/routes/api/offer.ts` and serves it at `/api/offer`. ## Deploy [#deploy] Build for a long-lived Node or Bun server, not a serverless or edge target: the offer's media runs over UDP, and the server has to stay up for the call. ## WebSocket [#websocket] A TanStack Start server route returns a `Response` and has no API to upgrade a WebSocket. Mount `socket` on a server that can upgrade: a [Bun](https://polvallverdu.github.io/pipedog/docs/frameworks/bun.md), Node or [Deno](https://polvallverdu.github.io/pipedog/docs/frameworks/deno.md) server next to the app, or [Cloudflare Workers](https://polvallverdu.github.io/pipedog/docs/frameworks/cloudflare-workers.md). ## Authentication [#authentication] The route has no auth. Wrap `offer` with a session check before it runs, as in [Authentication](https://polvallverdu.github.io/pipedog/docs/frameworks/#authentication). A request middleware on the route (`server.middleware`) works too. ## What is tested [#what-is-tested] The test imports the route module and calls `Route.options.server.handlers.POST` with a request that has no SDP offer, and checks that pipedog's own 400 comes back. It doesn't build a TanStack Start app or run its dev server. # Vercel > Serve the pipedog endpoints from a Bun server on Vercel Functions. Source: https://polvallverdu.github.io/pipedog/docs/frameworks/vercel/ Vercel runs a `server.ts` with the Bun runtime, so the [Bun](https://polvallverdu.github.io/pipedog/docs/frameworks/bun.md) snippet's shape works, with a `routes` map for static responses. ## Install [#install] ```bash bun add @pipedog/core @pipedog/asyncio @pipedog/transport-smallwebrtc @pipedog/transport-websocket @pipedog/serializer-protobuf @pipedog/openai @pipedog/audio-silero @pipedog/audio-smart-turn ``` `@pipedog/audio-smart-turn` isn't imported anywhere: the default turn strategies load it. To run the snippet from a checkout instead, see [Installation](https://polvallverdu.github.io/pipedog/docs/installation/#from-source). ## Files [#files] Set the runtime in `vercel.json`: ```json title="vercel.json" { "bunVersion": "1.x" } ``` The endpoints come from the shared module on the [overview](https://polvallverdu.github.io/pipedog/docs/frameworks/#the-two-endpoints). `voice.ts` re-exports it: ```ts title="voice.ts" export { offer, runSocketAgent, socket } from "../shared/voice.ts"; ``` ```ts title="server.ts" import { bunWebSocket, type UpgradingServer } from "@pipedog/transport-websocket/fetch"; import { offer, socket } from "./voice.ts"; // Vercel Functions run this server with the Bun runtime (`"bunVersion": "1.x"` in // vercel.json). Each WebSocket is pinned to one function instance for its lifetime, // and messages don't cross instances: keep shared state in a database. const config = { routes: { "/": new Response("voice agent", { headers: { "content-type": "text/plain" } }), }, fetch(request: Request, server: UpgradingServer) { const { pathname } = new URL(request.url); if (pathname === "/api/offer") return offer(request); // `socket` upgrades with the server Bun passes to `fetch`. if (pathname === "/ws") return socket(request, server); return new Response("not found", { status: 404 }); }, websocket: bunWebSocket, }; export default config; ``` ## Run it [#run-it] Locally, Bun serves the default export: ```bash OPENAI_API_KEY=sk-... bun server.ts ``` Then deploy with `vercel deploy`, and set `OPENAI_API_KEY` in the project's environment variables. ## Caveats [#caveats] * **WebRTC media.** The offer route is mounted, but its media runs over UDP, which a function isn't expected to carry. It's untested here. Use the WebSocket for audio on Vercel. * **One instance per socket.** Each WebSocket is pinned to one function instance for its lifetime, and messages don't cross instances. Keep shared state in a database. * **Duration.** A function's duration limit also ends a long session. * **Daemons.** Vercel Daemons are always-on workers with no inbound HTTP or TCP traffic, so they can't serve these endpoints. They suit background workers that pull their own work. ## Authentication [#authentication] Neither route has auth. Check the session in `fetch` before `offer` and `socket` run, as in [Authentication](https://polvallverdu.github.io/pipedog/docs/frameworks/#authentication). # Coming from Pipecat (Python) > How Pipecat's Python API maps to pipedog - naming, options objects, enums, async and cancellation, frames, and what is not ported. Source: https://polvallverdu.github.io/pipedog/docs/guides/coming-from-pipecat/ pipedog is designed so that someone who knows Pipecat can read pipedog code without a dictionary. Class names, frame names, event names, module paths and wire formats are the same; what changes is what TypeScript forces to change. This page is the short version of the porting rulebook, `docs/CONVENTIONS.md` in the repository. ## A side-by-side example [#a-side-by-side-example] ```python from pipecat.frames.frames import Frame, TextFrame from pipecat.processors.frame_processor import FrameDirection, FrameProcessor class Shout(FrameProcessor): def __init__(self, *, suffix: str = "!", **kwargs): super().__init__(**kwargs) self._suffix = suffix async def process_frame(self, frame: Frame, direction: FrameDirection): await super().process_frame(frame, direction) if isinstance(frame, TextFrame): frame = TextFrame(text=frame.text.upper() + self._suffix) await self.push_frame(frame, direction) shout = Shout(suffix="!!", name="shout") @shout.event_handler("on_error") async def on_error(processor, error): print(error.error) ``` ```ts twoslash import { type ErrorFrame, type Frame, TextFrame } from "@pipedog/core/frames/frames"; import { type FrameDirection, FrameProcessor, type FrameProcessorOptions, } from "@pipedog/core/processors/frame_processor"; export interface ShoutOptions extends FrameProcessorOptions { suffix?: string; } class Shout extends FrameProcessor { protected _suffix: string; constructor(options: ShoutOptions = {}) { super(options); this._suffix = options.suffix ?? "!"; } override async processFrame(frame: Frame, direction: FrameDirection): Promise { await super.processFrame(frame, direction); if (frame instanceof TextFrame) { frame = new TextFrame({ text: frame.text.toUpperCase() + this._suffix }); } await this.pushFrame(frame, direction); } } const shout = new Shout({ suffix: "!!", name: "shout" }); shout.eventHandler("on_error", async (_processor: FrameProcessor, error: ErrorFrame) => { console.log(error.error); }); ``` ## Modules and imports [#modules-and-imports] One Python module is one TypeScript file at the same path, with the same snake_case name: | Python | TypeScript | | --------------------------------------------------------------- | --------------------------------------------------------------------------- | | `from pipecat.processors.frame_processor import FrameProcessor` | `import { FrameProcessor } from "@pipedog/core/processors/frame_processor"` | | `from pipecat.services.deepgram.stt import DeepgramSTTService` | `import { DeepgramSTTService } from "@pipedog/deepgram/stt"` (once ported) | | `pipecat.transports.websocket` | `@pipedog/transport-websocket` | | `pipecat.serializers.twilio` | `@pipedog/serializer-twilio` (base serializers stay in core) | | `__init__.py` that re-exports | `index.ts` | The core import closure (frames, processors base, pipeline, workers, bus, observers, RTVI, base transports and services, turns, utils) is one package, `@pipedog/core`. Each provider, transport, serializer and audio model is its own package, like Pipecat's extras. ## Naming [#naming] | Python | TypeScript | | ------------------------------------------------------------- | ------------------------------------------------------------- | | `ClassName` | `ClassName` (unchanged) | | `def snake_case()`, `snake_case` attribute | `camelCase` | | `_protected` | `_protected` (camelCase, keeps the underscore) or `protected` | | `__private` | `#private` | | `CONSTANT` | `CONSTANT` | | event names `"on_pipeline_started"` | **unchanged strings** | | dict keys and JSON on the wire | **unchanged** (a protocol is not an API) | | acronyms: `LLMService`, `start_llm_usage_metrics`, `tts_text` | `LLMService`, `startLlmUsageMetrics`, `ttsText` | ## Constructors and parameters [#constructors-and-parameters] * **Keyword arguments become one options object**, named `Options` and extending the parent's. The whole object is passed to `super(options)`: that is how `**kwargs` flows. * **Required positional parameters of functions stay positional**: `push_frame(frame, direction=DOWNSTREAM)` is `pushFrame(frame, direction = FrameDirection.DOWNSTREAM)`. * **Keyword-only or optional groups become a trailing options object**: `push_error(msg, exception=e)` is `pushError(msg, { exception: e })`. * **`None` becomes `null`**, and defaults use `??` (never `||`, which swallows `0`, `""` and `false`). * **Pydantic params classes** (`PipelineParams`, `InputParams`) become classes with an options constructor and defaults: `new PipelineParams({ enableMetrics: true })`. * **Dataclasses** become classes with fields and an options constructor. ```python worker = PipelineWorker( pipeline, params=PipelineParams(enable_metrics=True, audio_out_sample_rate=24000), idle_timeout_secs=300, ) ``` ```ts const worker = new PipelineWorker(pipeline, { params: new PipelineParams({ enableMetrics: true, audioOutSampleRate: 24000 }), idleTimeoutSecs: 300, }); ``` ## Time, bytes and units [#time-bytes-and-units] * **Durations and timeouts stay in seconds** (floats), as upstream, so configs and docs carry over: `sleep(0.5)`, `idleTimeoutSecs: 300`, `waitFor(fn, 5)`. * **Clock values stay in nanoseconds** as `number` (`frame.pts`). * **`bytes` is `Uint8Array`**, never `Buffer`. PCM is 16-bit little-endian; read samples with `new Int16Array(u8.buffer, u8.byteOffset, u8.length / 2)` (copy first if `byteOffset` is odd). * `time.time()`, `time.monotonic()`, `time.monotonic_ns()` are `time()`, `monotonic()` and `monotonicNs()` from `@pipedog/asyncio`. ## Enums [#enums] Plain enums become an `as const` object plus a type of the same name, keeping upstream's values. `X.FOO.value` is just `X.FOO`: ```ts export const FrameDirection = { DOWNSTREAM: 1, UPSTREAM: 2 } as const; export type FrameDirection = (typeof FrameDirection)[keyof typeof FrameDirection]; ``` Enums with methods or properties (like `ErrorCategory.is_permanent`) become classes with static instances, a `value` field and getters (`ErrorCategory.AUTHENTICATION.isPermanent`), plus `values()` and `fromValue(v)`. `StrEnum`s are `as const` objects with string values. There is no TypeScript `enum` or `namespace` anywhere: the code is erasable so Node can run it directly. ## Async and cancellation [#async-and-cancellation] Python's asyncio semantics come from `@pipedog/asyncio`. The rules that matter most are in [Interruptions & cancellation](https://polvallverdu.github.io/pipedog/docs/concepts/interruptions-and-cancellation.md). | Python | TypeScript | | ---------------------------------------------------------- | ------------------------------------------------------------------------------------- | | `asyncio.create_task(coro())` | `createTask(() => coro())`; in a processor, `this.createTask(() => this.x(), "name")` | | `await self.cancel_task(task)` | `await this.cancelTask(task)` | | `asyncio.sleep(s)` | `sleep(s)` | | `asyncio.Event / Queue / PriorityQueue / Lock / Semaphore` | same names from `@pipedog/asyncio` | | `asyncio.wait_for(coro(), timeout=t)` | `waitFor(() => coro(), t)` (pass a function, so it can be cancelled) | | `asyncio.gather(a(), b())` | `gather(() => a(), () => b())` | | `asyncio.TimeoutError` / `TimeoutError` | `TimeoutError` from `@pipedog/asyncio` | | `asyncio.CancelledError` | `CancelledError` | | `asyncio.current_task()` | `currentTask()` | | `async with lock:` | `await lock.run(async () => { ... })` | | `async for x in aiter` | `for await (const x of aiter)` | | `async def gen(): yield` | `async function* gen()` | | `loop.run_in_executor(None, fn)` | inline if cheap, otherwise an injectable async function (no threads on Workers) | And three rules: 1. **`except Exception:` does not catch `CancelledError`**, but `catch (e)` catches everything. Every broad `catch` starts with `rethrowIfCancelled(e)`. 2. **Foreign awaits** (fetch, an SDK, a WebSocket) are not cancellation points. Pass `currentSignal()` to APIs that take an `AbortSignal`, and/or wrap the await in `abortable()`. 3. **Never leave a promise floating.** Where upstream fires and forgets a task, use `this.createTask(...)` so it is tracked and cancelled at cleanup. ## Classes and inheritance [#classes-and-inheritance] | Python | TypeScript | | -------------------------------------- | ----------------------------------------------------------------- | | abstract methods | `abstract` methods on an `abstract class` | | `@property` (and setter) | `get x()` (and `set x(v)`) | | `@staticmethod`, `@classmethod` | `static` | | `isinstance(x, (A, B))` | `x instanceof A \|\| x instanceof B` | | `type(x).__name__` | `x.constructor.name` | | `__str__` / `__repr__` | `toString()` | | `@obj.event_handler("on_x")` decorator | `obj.eventHandler("on_x", handler)` or `obj.addEventHandler(...)` | **Multiple inheritance**: a second base that is a mixin becomes a class-factory mixin (`class C(A, M)` is `class C extends M(A)`, and `instanceof M` still works). Two concrete bases in the same hierarchy (such as a frame that is both a system frame and a text frame) extend the first and register the second as a virtual base, so `instanceof` works for both. ## Frames [#frames] * Frame constructors **always** take one options object: `TextFrame("hi")` is `new TextFrame({ text: "hi" })`, even where upstream tests pass positional arguments. * `init=True` dataclass fields come from the options; `init=False` fields are initializers. * Frames that are uninterruptible by default declare `override interruptible = false`. * `broadcast_frame(FrameCls, **kwargs)` is `broadcastFrame(FrameCls, options)`. * Field order follows upstream, and `__post_init__` logic runs in the constructor after `super()`. ## Logging [#logging] `from loguru import logger` becomes `import { logger } from "@pipedog/core/utils/logger"`, with the same level methods (`trace`, `debug`, `info`, `warning`, `error`, `critical`). `logger.exception(msg)` is `logger.error(msg, e)`. ## The development runner [#the-development-runner] Pipecat's `pipecat.runner` is `@pipedog/runner`, with the same entry point: export a `bot(runnerArgs)` function and call `main()`. The flags (`-t webrtc`, `-t twilio`, `--proxy`, `--host`, `--port`, ...) are the same. | Python | TypeScript | | ------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | | `from pipecat.runner.run import main` | `import { main } from "@pipedog/runner/run"` | | `if __name__ == "__main__": main()` | `if (import.meta.main) await main();` | | `async def bot(runner_args: RunnerArguments)` | `export async function bot(runnerArgs: RunnerArguments)` | | `await create_transport(runner_args, transport_params)` | `await createTransport(runnerArgs, transportParams)`, from `@pipedog/runner/utils` | | `parse_telephony_websocket(websocket)` | `parseTelephonyWebsocket(websocket)` | | `from pipecat.runner.run import app; @app.get(...)` | `import { app } from "@pipedog/runner/run"; app.get(...)` | | `uvicorn.run(app)` | `serve(app)` from `@pipedog/runner/serve`, or `createRunnerApp()` and its `fetch` in your server | | `python bot.py -t webrtc` | `bun bot.ts -t webrtc` | The FastAPI app is a fetch handler, so the runner mounts in any framework. Daily, MOQ and the Vonage Video Connector are recognized but fail with a clear message: their transports aren't ported yet. See [Development runner](https://polvallverdu.github.io/pipedog/docs/runner.md) for the full mapping. ## What isn't ported yet [#what-isnt-ported-yet] The core, Pipecat Flows, evals, the CLI and most providers are ported; Daily and the other room transports, the Krisp, Koala and AIC filters, the Redis and pgmq buses and a few providers are not. The [roadmap](https://polvallverdu.github.io/pipedog/docs/roadmap/#whats-not-ported-yet) lists the gaps with their upstream module counts. ## Deprecated APIs are not ported [#deprecated-apis-are-not-ported] Anything upstream marks deprecated (with `@deprecated`, `warn_deprecated`, or "will be removed in 2.0.0") is left out of pipedog, including: * the `PipelineTask` / `PipelineTaskParams` / `PipelineRunner` aliases (use `PipelineWorker`, `PipelineParams` and `WorkerRunner`); * `EndTaskFrame` / `CancelTaskFrame` (use `EndWorkerFrame` / `CancelWorkerFrame`); * the `UninterruptibleFrame` mixin (use `interruptible = false`); * deprecated parameters such as `push_error(fatal=)` and `ErrorFrame.fatal` (check `processor.isUsable`, or pass `forceTreatAsPermanent`); * the deprecated `StartFrame` configuration fields (read `FrameProcessorSetup` in `setup()`). What was dropped from each module is recorded in its `notes` in `porting/manifest.json`. ## Tests [#tests] Upstream tests are ported 1:1 with `bun:test`, keeping their names so the two suites can be diffed: `class TestX(unittest.IsolatedAsyncioTestCase)` is `describe("TestX", ...)`, `async def test_y` is `test("test_y", ...)`, and `run_test(...)` is `runTest(...)` from `@pipedog/core/tests/utils`. # Differences from upstream > Where pipedog behaves differently from Pipecat (Python), and why. Source: https://polvallverdu.github.io/pipedog/docs/guides/differences-from-upstream/ pipedog aims for 1:1 behavior with Pipecat: the same frames in the same order, the same lifecycle and the same wire formats. Where JavaScript or the runtimes force a difference, the code marks it with a `DIVERGENCE` comment. This page collects them. It covers behavior. For how names and APIs map (camelCase, options objects, enums and so on), see [Coming from Pipecat](https://polvallverdu.github.io/pipedog/docs/guides/coming-from-pipecat.md). ## Across the codebase [#across-the-codebase] These differences apply everywhere, not to one module. They come from the porting rules in `docs/CONVENTIONS.md` and the "Porting notes" at the top of the provider source files. | Area | Upstream | pipedog | Why | | -------------------------- | ------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Threads | `ThreadPoolExecutor` and `run_in_executor` run blocking work in a thread. | The work runs inline (async) or in an injected runner. | There are no threads on Cloudflare Workers. Keep per-call CPU low there. | | asyncio | Python's `asyncio`. | `@pipedog/asyncio` keeps asyncio's semantics, scheduling included. | So ported code interleaves exactly as upstream. See [Interruptions & cancellation](https://polvallverdu.github.io/pipedog/docs/concepts/interruptions-and-cancellation.md). | | Context variables | `contextvars` carry state into tasks. | Core has none (no `node:async_hooks` outside `@pipedog/asyncio`). State is recorded per task instead. | Core stays runtime-agnostic. One visible effect is listed under [LLM tools](#llm-tools-and-direct-functions). | | HTTP clients | `aiohttp` sessions and `httpx` clients. | The global `fetch`, or a `fetch` option. | `fetch` exists on every runtime. | | Provider SDKs | The provider's Python SDK. | The provider's JS SDK when it works over `fetch` or `WebSocket` (for example `openai`, `@google/genai`, `@langchain/core`). Otherwise a small client over `fetch` or the global `WebSocket`. Node SDKs only where the provider offers nothing but gRPC streaming. | So each service runs on as many runtimes as the provider allows. | | Bytes | `bytes`. | `Uint8Array`, never `Buffer`. | `Buffer` is a Node API. Core imports no `node:*` modules except `node:async_hooks`. | | Filesystem | `aiofiles`, `pathlib`. | An injected reader or `fs` object (`node:fs/promises` satisfies it). | Core never imports `node:fs`. | | WebSocket handshake | The `websockets` package exposes the handshake response. | A standard `WebSocket` does not, so response headers and the HTTP status of a refused handshake are not available. | The standard API has no access to them. Affected services are listed below. | | Deprecated APIs | Kept until removal, with warnings. | Not ported: deprecated classes, aliases, parameters and fields are left out. | pipedog starts without the deprecated surface. Each module's dropped items are in its `notes` in `porting/manifest.json`. Moved modules exist only at their new path: the LLM adapters are under `adapters/llm` (pipecat 1.14 keeps the old `pipecat.adapters.*` paths as deprecated shims). | | Pillow | Decodes and encodes images. | Built-in or injectable decoders and encoders (for example `utils/jpeg_encoder.ts`, an `imageDecoder` option). | No native dependencies. | | `GRPC_ENABLE_FORK_SUPPORT` | Set to false at import by the Google services. | Not set. | There is no fork in JS. | ## Core [#core] ### Utils [#utils] * **`NOT_GIVEN` is truthy.** Upstream's sentinel is falsy, so `value or default` treats a missing value as absent. JavaScript objects are always truthy, so use `isGiven(value) ? value : fallback`. `packages/core/src/utils/types.ts` * **Timeout exceptions.** Upstream lists httpx `TimeoutException`. Here `TIMEOUT_EXCEPTIONS` holds the asyncio `TimeoutError`, and `isTimeoutException` also recognizes the `DOMException` named "TimeoutError" that a `fetch` aborted by `AbortSignal.timeout()` rejects with. `packages/core/src/utils/http.ts` * **Connection limits.** Upstream's `connection_limits` returns an httpx `Limits` instance. `fetch` has no such object, so `connectionLimits` returns the plain options for an SDK or agent that reads them field by field. `packages/core/src/utils/http.ts` ### TTS [#tts] * **Audio cache (pipedog only).** `TTSService` takes an optional `audioCache`, and `warmAudioCache()` synthesizes fixed lines ahead of time, so they play without a request to the provider. Off by default. See [TTS audio cache](https://polvallverdu.github.io/pipedog/docs/concepts/tts-audio-cache.md). `packages/core/src/services/tts_audio_cache.ts` ### Expressive conversation [#expressive-conversation] * **Text offsets.** The expression formatter and `SpokenExpression` count offsets in UTF-16 code units (JavaScript string indices), not code points, so they differ from upstream only after an astral character such as an emoji. A `[[pause N]]` tag takes ASCII digits only (Python's `\d` also accepts other Unicode digits). `packages/core/src/expressive/formatter.ts` * **`ExpressionGuidance`.** Upstream's `StrEnum` is a class with static instances (`value`, `values()`, `fromValue()`) rather than an `as const` object, so `instanceof` can still tell a preset from a developer line with the same text, which upstream's `isinstance` check relies on. `packages/core/src/expressive/policy.ts` ### Logging [#logging] * **Secrets are masked in warnings and errors.** Upstream masks credentials only in `ErrorFrame`. pipedog masks the message and attached error of every `WARNING` or higher log record, for every sink, because these often quote what failed, such as a URL with `?key=...`. `packages/core/src/utils/logger.ts` ### Tracing [#tracing] * **Tracer names and the default service name say pipedog.** Upstream's tracers are `pipecat` and `pipecat.turn`, and `setup_tracing` defaults `service_name` to `"pipecat"`. Here they are `pipedog`, `pipedog.turn` and `"pipedog"`, so traces from the two can be told apart. Span names and attribute keys are upstream's. `setTracerName("pipecat")` (or `setupTracing({ tracerName: "pipecat" })`) brings the upstream tracer names back, for dashboards and queries that filter on them: see [Metrics and tracing](https://polvallverdu.github.io/pipedog/docs/concepts/metrics-and-tracing.md). `packages/core/src/utils/tracing/setup.ts` * **`forceFlushTracing()` waits for exports already in flight.** The SDK's batch processor only flushes the spans still queued, so a flush that landed just after its 5 s timer started an export returned before the collector had the spans. `packages/tracing/src/index.ts` ### asyncio [#asyncio] * **Only `await` suspends on a task.** A `Task` is thenable so it can be awaited, but `then()` is also an ordinary method. Calling `task.then(cb)` yourself (Python has no equivalent) observes the task without making the caller wait on it: cancelling the caller doesn't cancel `task`. `await`, `Promise.all` and `Promise.resolve(task)` still suspend the caller, as in asyncio. `packages/asyncio/src/task.ts` ### asyncio on Cloudflare Workers [#asyncio-on-cloudflare-workers] * **Callbacks run in the I/O context that scheduled them.** One workerd isolate serves many requests and Durable Objects, and I/O must happen in the context that owns it. On Workers, `@pipedog/asyncio` has no shared ready queue: each callback runs in the context that scheduled it, still in FIFO order. That is what lets several Durable Objects run voice sessions in one isolate at once. Bun, Node and Deno keep a single ready queue, like CPython. `packages/asyncio/src/loop.ts` ### Workers and runner [#workers-and-runner] * **Signal handlers.** SIGINT and SIGTERM handlers are installed through the host's `process.on` when there is one (Bun, Node, Deno). On runtimes without process signals (Workers) they are a no-op. `packages/core/src/workers/runner.ts` * **Garbage collection.** JS has no `gc.collect()` count or `gc.garbage`. The runner calls the global `gc()` when the runtime exposes one (for example Node or Deno with `--expose-gc`), and skips it otherwise. `packages/core/src/workers/runner.ts` * **Backend capabilities.** Upstream overrides `PipelineWorker.capabilities` with a plain dict. TypeScript does not allow an override of a different type, so the backend's capabilities are `backendCapabilities`, and `PipelineWorker.capabilities` keeps its meaning. `packages/core/src/workers/llm/backend_llm_worker.ts` ### LLM tools and direct functions [#llm-tools-and-direct-functions] * **Direct function schemas.** JavaScript has no runtime type hints or docstrings. Parameter names, and which ones are required (no default value), are read from the function's source. The description and parameter types come from optional `DirectFunctionMetadata` attached to the function. Parameters without metadata get an any-type schema. Minified functions lose their parameter names, so direct functions must ship unminified. `packages/core/src/adapters/llm/schemas/direct_function.ts` * **Collected tools are unbound.** `_collectTools` returns the methods found on the prototype chain, not bound methods, because a bound function does not expose the source its schema is read from. `LLMWorker` binds each one when it registers it. `packages/core/src/workers/llm/tool_decorator.ts` * **Tool-call state does not reach spawned tasks.** Upstream's `_IN_TOOL_CALL` is a context variable. Here the worker running a tool is recorded per task, so a task the tool handler spawns (with `gather` or `createTask`) does not inherit it, and frames queued from such a task are not held. `packages/core/src/workers/llm/llm_worker.ts` ### Turn analyzers and VAD [#turn-analyzers-and-vad] * **Smart Turn loads lazily.** Upstream loads the model when the analyzer is created. pipedog loads it on the first prediction, so an unused analyzer loads nothing. Without the inference runtime package (or where it can't load), turns end on the VAD's silence instead of failing. A factory passed in explicitly still fails loudly. `packages/core/src/audio/turn/smart_turn/local_smart_turn_v3.ts` See [Turn detection](https://polvallverdu.github.io/pipedog/docs/concepts/turn-detection.md). * **Silero sessions are shared.** Upstream loads one session per model. pipedog shares sessions by default, since a bot builds one VAD per call. Pass `shareSession: false` for a session of its own. `packages/audio-silero/src/silero.ts` * **`minVolumeDb` (pipedog only).** `VADParams` takes the volume gate as a loudness in LUFS as an alternative to the 0–1 `minVolume`, converted with the same mapping (`volumeFromDb`, `volumeToDb`). Setting both throws. `packages/core/src/audio/vad/vad_analyzer.ts` * **Backchannels and VAD-backed words (pipedog only).** `MinWordsUserTurnStartStrategy` takes `isBackchannel` (with `createBackchannelMatcher()` and per-language `BACKCHANNEL_PHRASES`), so an acknowledgement over the bot never starts a turn, and `requireVadWithinSeconds`, so words over the bot count only if the VAD heard the user recently. Both are off by default. `packages/core/src/turns/user_start/min_words_user_turn_start_strategy.ts`, `packages/core/src/turns/user_start/backchannel.ts` * **A wake phrase doesn't gate a `StrategyMode.FIRST` list.** Upstream's `WakePhraseUserTurnStartStrategy`, while asleep, still keeps the start strategies after it in a `FIRST` list from seeing frames, through the deprecated `ProcessFrameResult.STOP`. pipedog doesn't port `ProcessFrameResult`, so every strategy sees every frame: use the wake phrase strategy alone, or combine it with other conditions in a `StrategyMode.ALL` list (both work the same upstream). `packages/core/src/turns/user_start/wake_phrase_user_turn_start_strategy.ts` ## Providers [#providers] ### Anthropic [#anthropic] * **Explicit `null` sampling values are omitted.** The Python SDK's `NOT_GIVEN` has no JS counterpart, so `temperature`, `topK`, `topP` and `thinking` store `null` for "not sent". An explicit `null` is left out of the request, where upstream would send JSON `null`. `packages/anthropic/src/llm.ts` ### AWS [#aws] * **No botocore credential chain.** Keys come from the options or the `AWS_*` environment variables, as upstream, and beyond that from a `credentialsProvider` function you pass (called per request or per session, so it can renew temporary credentials). Instance profiles, IRSA, ECS roles, SSO and credential files are not read for you. `packages/aws/src/utils.ts` * **Nova Sonic over WebSocket.** Upstream streams over HTTP/2 with the experimental Python SDK. Full-duplex HTTP/2 isn't available to `fetch` or on Workers, so the default client speaks the same API over Bedrock's WebSocket endpoint (the transport the AWS JS SDK uses in browsers), on every runtime. `transport: "http2"` uses `@aws-sdk/client-bedrock-runtime` on Node and Bun. `packages/aws/src/nova_sonic/llm.ts` * **SageMaker streaming needs HTTP/2.** `SageMakerBidiClient` loads the optional `@aws-sdk/client-sagemaker-runtime-http2` peer, which runs on Node 22+ and Bun, not on Deno or Workers. Its events use the JS SDK's shapes. `packages/aws/src/sagemaker/bidi_client.ts` * **No botocore retries.** Bedrock's `client_config` is not ported: no client-side retries of throttled or 5xx requests. Pass a `fetch` to add them. `packages/aws/src/llm.ts` * **Every Polly failure is an `ErrorFrame`.** Upstream catches botocore errors only; here a non-2xx response, a network error or missing credentials all become an `ErrorFrame`. `packages/aws/src/tts.ts` ### Azure [#azure] * **No Speech SDK.** Upstream uses `azure-cognitiveservices-speech`, a native library. Azure Speech STT and `AzureTTSService` speak the Speech service's WebSocket protocol directly (the one the SDKs use), and `AzureHttpTTSService` calls the REST endpoint with `fetch`, so all of them run on Bun, Node, Deno and Workers. `_speechConfig` is a small stand-in for the SDK's `SpeechConfig`. `packages/azure/src/stt.ts`, `packages/azure/src/tts.ts` * **STT connects in `start()`.** The connection opens once the input sample rate is known, and the next audio reopens it after the service closes it. `packages/azure/src/stt.ts` * **Interruptions close the TTS connection.** Upstream calls the SDK's `stop_speaking_async()`; here the synthesis connection is closed and reopens on the next synthesis. `packages/azure/src/tts.ts` * **`fetch` replaces `aiohttp_session`** in `AzureImageGenServiceREST`, with an injectable image decoder instead of Pillow. `packages/azure/src/image.ts` ### Cartesia [#cartesia] * **`fetch` replaces `aiohttp_session`.** `CartesiaHttpTTSService` posts with the global `fetch`, or the `fetch` option. `packages/cartesia/src/tts.ts` * **No handshake headers.** The standard `WebSocket` does not expose the handshake response, so the `X-Request-ID` header upstream logs on connect is always reported as `None`. `packages/cartesia/src/stt.ts` ### Deepgram [#deepgram] * **`fetch` replaces `aiohttp_session`.** `DeepgramHttpTTSService` takes an optional `fetch` where upstream requires a shared `aiohttp_session`. `packages/deepgram/src/tts.ts` * **No Python SDK.** STT uses a minimal `DeepgramClient` that speaks the same `/v1/listen` WebSocket protocol. `DeepgramApiError` is raised only when the runtime's error carries the handshake status; otherwise a refused handshake is retried until the quick-failure tracker gives up. `packages/deepgram/src/stt.ts` * **No `dg-*` headers.** The TTS service cannot log the `dg-*` response headers upstream logs on connect. Query values use JS number formatting, so `speed=1.0` is written `speed=1`. `packages/deepgram/src/tts.ts` ### ElevenLabs [#elevenlabs] * **`fetch` replaces `aiohttp_session`.** `ElevenLabsHttpTTSService` and `ElevenLabsSTTService` take an optional `fetch` where upstream requires an `aiohttp_session`. `packages/elevenlabs/src/tts.ts`, `packages/elevenlabs/src/stt.ts` * **No message size setting.** Upstream sets a 16 MB max WebSocket message size for large audio responses. The standard `WebSocket` has no message size limit to set. `packages/elevenlabs/src/tts_base.ts` ### Google [#google] * **Credentials resolve later.** Upstream resolves credentials (and, for STT, the project ID) in the constructor. Reading a key file, finding Application Default Credentials and refreshing a token are asynchronous in JS, so they happen on first connect or first request. A missing credential surfaces then (as "No valid credentials provided.", or for Cloud TTS as a synthesis error). Inline `credentials` are still checked in the constructor. `packages/google/src/stt.ts`, `packages/google/src/vertex/llm.ts`, `packages/google/src/gemini_live/vertex/llm.ts`, `packages/google/src/tts.ts` * **`clientOptions` (pipedog-only).** The Cloud STT and TTS services accept extra client options, merged over the ones derived from `location` and the credentials. Upstream builds its client from those alone. `packages/google/src/stt.ts`, `packages/google/src/tts.ts` * **gRPC services need a server runtime.** Cloud Speech streaming and Cloud TTS go through the Node SDKs (`@google-cloud/speech`, `@google-cloud/text-to-speech`), loaded lazily; they run on Bun, Node and Deno, not Workers. Vertex services authenticate through google-auth-library and run on Bun, Node and Deno, not Workers. See [Runtimes](https://polvallverdu.github.io/pipedog/docs/guides/runtimes.md). * **Image decoding is injectable.** Upstream decodes inline images with Pillow. Here that is the `imageDecoder` option; the default throws "not ported yet", so without one an image response surfaces as an error. `packages/google/src/llm.ts`, `packages/google/src/image.ts` * **Gemini STT on Vertex AI (pipedog-only).** `GeminiSTTService` takes `vertexai: true` with `location`, `projectId`, and `credentials` / `credentialsPath` (or ADC) instead of `apiKey`, as the Vertex LLM services do. Vertex's Live API rejects `languageAuto` and `languageHints`, so in that mode the codes go in `inputAudioTranscription.languageCodes` and none means automatic detection. The client is created on first connect rather than in the constructor, so a subclass's `_createClient()` can read its own fields. `packages/google/src/gemini_live/stt.ts` * **Gemini STT language options (pipedog-only).** `reportDetectedLanguage: false` puts the configured language (when exactly one is set) or `null` on transcription frames instead of the language the model reports. `setLanguageCodes(codes | () => codes)` changes the languages during a session; the service reconnects on the next `BotStartedSpeakingFrame`, or after the user's turn if they are speaking, buffering audio meanwhile. `packages/google/src/gemini_live/stt.ts` * **Gemini STT connects in the background.** Upstream opens the Live session in `setup()`, so the `StartFrame` waits for it. Here it is opened in a background task on `StartFrame` and audio is buffered until the session is live; a failure is reported through `on_connection_error`. `awaitConnectionOnStart: true` (pipedog-only) restores the blocking behaviour. `packages/google/src/gemini_live/stt.ts` * **`retryOnBlocked` on `GeminiTTSService` (pipedog-only).** Gemini TTS sometimes refuses a harmless sentence for Google's usage guidelines, and the same request usually succeeds on the next try. With `retryOnBlocked` the sentence is requested again (`retries`, default 1); if part of its audio already played, it starts over (optionally as `restartText(text)`, e.g. "Sorry, …") rather than resuming mid-sentence. A `fallback`, a `GeminiTTSSettings` delta or a function yielding frames, gets one last attempt. Only the usage-guidelines block is retried (`isGeminiTTSBlockedError`), and restarted or fallback audio is never cached. Off by default, so a blocked sentence fails with an `ErrorFrame` as upstream. `packages/google/src/tts.ts` * **Shared Cloud clients and `prewarm()` (pipedog-only).** `sharedClient: true` on `GoogleSTTService`, `GoogleTTSService`, `GoogleHttpTTSService` and `GeminiTTSService` (Cloud backend) uses one gRPC client per process for the same endpoint, credentials and client options (`getSharedGoogleCloudTTSClient` for TTS). `prewarm()` loads the SDK, opens the channel and fetches a token at start-up, and returns `false` instead of throwing. Opt-in, because one shared channel carries every concurrent stream. Upstream builds a client per service. `packages/google/src/tts.ts`, `packages/google/src/stt.ts` * **`GoogleSTTService.setLanguageCodes()` (pipedog-only).** Changes the `languageCodes` mid-call (an array, or a function re-read on each `BotStartedSpeakingFrame`). The stream reconnects on the next `BotStartedSpeakingFrame`, or after the user's turn if they are speaking over the bot, so no utterance is cut. Same API as `GeminiSTTService`. `packages/google/src/stt.ts` * **Audio cache key for Google TTS.** With the pipedog-only TTS audio cache, the key also covers what changes the audio outside `settings`: `GoogleTTSService`'s `voiceCloningKey` (hashed) and the backend of `GeminiTTSService`. `packages/google/src/tts.ts` ### Inworld and xAI realtime [#inworld-and-xai-realtime] * **Status details are JSON.** When a realtime response fails, upstream reports `str(status_details)`, a Python dict repr. pipedog renders it as JSON. `packages/inworld/src/realtime/llm.ts`, `packages/xai/src/realtime/llm.ts` * **`fetch` replaces `aiohttp_session`** in the Inworld and xAI HTTP TTS services. `packages/inworld/src/tts.ts`, `packages/xai/src/tts.ts` ### LangChain [#langchain] * **List content is JSON.** When a chunk's `content` is a list, upstream stringifies it with Python's `str()`. pipedog uses JSON. `packages/langchain/src/langchain.ts` ### OpenAI [#openai] * **`fetch` replaces `aiohttp_session`.** `OpenAIImageGenService` downloads hosted results with the global `fetch`, or the `fetch` option. `packages/openai/src/image.ts` * **`decodeImage` (pipedog-only).** Upstream decodes with Pillow. PNGs are decoded with `fast-png` by default, and `decodeImage` can decode other formats. `packages/openai/src/image.ts` * **`httpClient` is a `fetch`.** Upstream takes an httpx client. Here `httpClient` is the SDK's `fetch` option, so httpx connection limits do not carry over. `packages/openai/src/tts.ts`, `packages/whisper/src/base_stt.ts` * **Tool message content is JSON.** In the Responses adapter, non-string tool content is written as JSON for objects, where upstream uses Python's `str()` (a Python repr). `packages/openai/src/open_ai_responses_adapter.ts` ## Runner [#runner] * **A fetch handler instead of FastAPI.** The runner's server is a fetch handler (`app.fetch`), so it runs on Bun, Node and Deno through `serve()` and mounts in any framework. Startup hooks are `lifespan` functions that may return a cleanup. `packages/runner/src/run.ts` * **The prebuilt client ships in the package.** Upstream installs it from the `pipecat-ai-prebuilt` wheel; `@pipedog/runner` bundles the same built client. `packages/runner/src/prebuilt.ts` * **Unported transports fail clearly.** Daily (with `--direct` and the dial-in webhook), MOQ and the Vonage Video Connector are recognized, and fail with a message naming them. WhatsApp (`--whatsapp`) is ported. An eval harness's `eval-connect` message is not detected. `packages/runner/src/run.ts`, `packages/runner/src/utils.ts` * **Hang-ups during the handshake are typed.** A caller who hangs up before the provider's `start` message is a `TelephonyCallEndedError` (a subclass of upstream's error), logged at info level. `packages/runner/src/utils.ts` * **Async helpers.** The WebSocket token helpers are async, because WebCrypto's HMAC is. `packages/runner/src/run.ts` ## Telephony serializers [#telephony-serializers] The Twilio, Telnyx, Plivo, Exotel, Vonage and Genesys serializers keep upstream's wire formats and behavior. Hang-ups through the provider's REST API use `fetch` instead of `aiohttp`, so they work on every runtime, Workers included. ## Transports [#transports] * **SmallWebRTC completes pending writes on stop.** Upstream leaves queued writes pending when a track stops, so a write in progress when the peer connection is replaced (`restart_pc`) waits for the output transport's write timeout and the pipeline fails. pipedog completes them. `packages/transport-smallwebrtc/src/transport.ts` * **SmallWebRTC uses werift.** Upstream is written against aiortc. pipedog wraps werift, a pure TypeScript WebRTC stack, in the subset of aiortc's API upstream uses. `packages/transport-smallwebrtc/src/rtc.ts` * **WebSocket transports bring their own socket.** Upstream sits on the `websockets` package and Starlette's `WebSocket`. pipedog's transports sit on a runtime-agnostic `WebsocketConnection`. The FastAPI transport takes any socket your server accepted, on any runtime. Workers can't listen on a port, so use `FastAPIWebsocketTransport` with a `WebSocketPair` there. Fetch-handler helpers (`websocketFetchHandler`, `upgradeWebSocket`, `serveFetch`) accept the socket on each runtime: from the request on Deno and Workers, with the server on Bun and Node. `packages/transport-websocket/src/socket.ts`, `packages/transport-websocket/src/fastapi.ts` ## Other packages [#other-packages] Most newer packages list their own differences next to their usage, on the page named below. The main ones, in short: * **Flows** behave as upstream; the API is camelCase, a tool returns `[result, nextNode]`, and files are read through a `readFile` function you pass in. See [Flows](https://polvallverdu.github.io/pipedog/docs/flows/#differences-from-pipecat). * **Evals** are the `pipedog-eval` bin, `factory:` entries are imported with `import()`, each suite run is a child process, and the deprecated `EvalTransport` server and `harness` are left out. See [Evals](https://polvallverdu.github.io/pipedog/docs/evals/#differences-from-upstream). * **The CLI** is a TypeScript project scaffold: no Pipecat Cloud, no Daily, no plugin system, and the docs MCP server in place of the Context Hub. See [CLI](https://polvallverdu.github.io/pipedog/docs/cli/#differences-from-pipecat). * **The MCP client** follows a server's `tools/list` cursor, where upstream reads the first page, and on Workers does not validate a result against the tool's output schema (the validator needs `new Function`). See [MCP client](https://polvallverdu.github.io/pipedog/docs/services/mcp-client/#differences-from-pipecat). * **LiveKit and HeyGen** take a `fetch` option in place of upstream's `aiohttp` session, and the token helpers are async (WebCrypto signs the JWT). HeyGen's LiveAvatar fields that its API reference documents as optional (`livekitAgentToken`, `wsUrl`) are optional here, where upstream's model rejects a session without them. See [LiveKit](https://polvallverdu.github.io/pipedog/docs/transports/livekit.md) and [HeyGen](https://polvallverdu.github.io/pipedog/docs/services/video/heygen.md). * **WhatsApp, local audio, Sentry, RNNoise and the soundfile mixer** each have a section: [WhatsApp](https://polvallverdu.github.io/pipedog/docs/transports/whatsapp/#differences-from-upstream), [local audio](https://polvallverdu.github.io/pipedog/docs/transports/local-audio/#differences-from-upstream), [Sentry metrics](https://polvallverdu.github.io/pipedog/docs/concepts/sentry-metrics/#differences-from-upstream), [RNNoise](https://polvallverdu.github.io/pipedog/docs/audio/rnnoise-filter/#differences-from-upstream) and [soundfile mixer](https://polvallverdu.github.io/pipedog/docs/audio/soundfile-mixer/#differences-from-upstream). * **Ducking** (`@pipedog/audio-ducking`) is pipedog only: an output audio filter that lowers the bot's voice while the user talks over it. See [Ducking](https://polvallverdu.github.io/pipedog/docs/audio/ducking.md). * **TEN VAD** (`@pipedog/audio-ten-vad`) is pipedog only: `TenVADAnalyzer`, an opt-in alternative to Silero that drives TEN VAD's WebAssembly build. pipedog ships none of TEN's files; you obtain them from TEN, under TEN's license, and pass them in. See [TEN VAD](https://polvallverdu.github.io/pipedog/docs/audio/ten-vad.md). * **Local speech models** (Whisper, Moonshine, FunASR, Kokoro, Piper, Pocket TTS) run on ONNX Runtime instead of the Python libraries. Each page has a Differences section, for example [Kokoro](https://polvallverdu.github.io/pipedog/docs/services/tts/kokoro/#differences-from-upstream). * **The speech providers** (AssemblyAI, Speechmatics, Gladia, Soniox, Rime, Fish, Hume, Sarvam, Smallest, Gradium, Meta, Neuphonic, Speechify, MiniMax, Resemble AI, Camb.ai, Async, Bland) follow the rules in [Across the codebase](#across-the-codebase): a `fetch` option in place of the `aiohttp` session or the provider's SDK client, a direct WebSocket client where upstream uses a Python SDK (Sarvam, Camb.ai), and no deprecated `InputParams`. Where a service sends floats as JSON or MessagePack, a whole number such as `speed: 1.0` is written `1`, because JavaScript has one number type. Each page's notes list what it leaves out. ## Reporting a difference [#reporting-a-difference] If pipedog behaves differently from Pipecat and this page doesn't list it, please [open an issue](https://github.com/polvallverdu/pipedog/issues) with the upstream behavior, what pipedog does, and a small reproduction. See [Contributing](https://polvallverdu.github.io/pipedog/docs/contributing.md) to send a fix. # Effect > Use pipedog from Effect with @pipedog/effect. Run a pipeline as a scoped resource, stream its frames, feed it from a Stream, and write frame processors as Effect functions with services from Layers. Source: https://polvallverdu.github.io/pipedog/docs/guides/effect/ [Effect](https://effect.website) apps can drive pipedog with `@pipedog/effect`, a small adapter that maps pipedog's asyncio-style tasks onto Effect's fibers and scopes. Pipelines, processors and services stay the same; the adapter covers starting and stopping a pipeline, passing frames in and out, and writing processors as Effects. It is pipedog-only: Pipecat has no counterpart. ## Installation [#installation] ```bash bun add @pipedog/effect effect ``` `effect` is a peer dependency (version 4). ## Runtimes [#runtimes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/effect/package.json`: ✅ means the compat smoke test passes on that runtime. The smoke test on every runtime runs a pipeline with an Effect processor from end to end, interrupts a running pipeline and checks that no task is left, and cancels a pipeline while a processor's Effect is running and checks that the Effect is interrupted. ## The API [#the-api] | Export | What it does | | --------------------------------- | ------------------------------------------------------------------------------------------------------- | | `startPipeline(worker, options?)` | Runs a `PipelineWorker` for the lifetime of the current `Scope` and returns a `PipelineHandle`. | | `runPipeline(worker, options?)` | `startPipeline` and `join` in a scope of their own. | | `makeProcessor(fn, options?)` | Builds a `FrameProcessor` from `(frame, direction, ctx) => Effect`. Its services come from the context. | | `PipelineError` | The tagged error a failed run fails with. | A `PipelineHandle` has: | Member | | | ----------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- | | `join` | Waits for the pipeline to end. See [How a run ends](#how-a-run-ends). | | `queueFrame(frame, direction?)` | `worker.queueFrame` as an Effect. | | `queueFrames(framesOrStream, direction?)` | Queues an iterable, or every frame of a `Stream` until it ends. | | `frames(...types)` | A scoped subscription to the frames of those classes that reach the end of the pipeline, as a `Stream` that ends with the pipeline. | | `stopWhenDone` | Queues an `EndFrame`. | | `cancel(reason?)` | Cancels the pipeline. | | `worker` | The worker. | `frames()` subscribes when it runs, not when the stream is read, so subscribe before queueing the frames you want to see. Frames are buffered without bound until the stream reads them. ### Options [#options] | Option | Type | Required | Description | |---|---|---|---| | `runner` | `WorkerRunnerOptions \| undefined` | no | Options for the `WorkerRunner` that runs the worker. Signal handling is off by default (`handleSigint: false`): an Effect app handles signals with its runtime (e.g. `NodeRuntime.runMain`), whose interruption cancels the pipeline. | | `failOnError` | `boolean \| undefined` | no | Fail with a `PipelineError` on the first `ErrorFrame` that reaches the worker, cancelling the pipeline. Default false: errors are reported by the pipeline as usual and the run fails only if the pipeline stops because of an unusable processor. | | `cancelReason` | `string \| undefined` | no | Reason given to the runner when closing the scope cancels a running pipeline. | ## A voice agent with Effect [#a-voice-agent-with-effect] The example below is the [voice bot](https://polvallverdu.github.io/pipedog/docs/voice-bot.md) with one addition, written in Effect: every final transcription goes through a moderation service before it reaches the LLM. The service is an ordinary Effect service, with a live Layer that calls OpenAI's moderation endpoint: ```ts import * as Context from "effect/Context"; import * as Data from "effect/Data"; import * as Effect from "effect/Effect"; import * as Layer from "effect/Layer"; import * as Schedule from "effect/Schedule"; class ModerationError extends Data.TaggedError("ModerationError")<{ cause: unknown }> {} /** Decides whether a user's words may go to the LLM. */ class Moderation extends Context.Service< Moderation, { readonly isAllowed: (text: string) => Effect.Effect } >()("Moderation") {} /** OpenAI's moderation endpoint. */ const ModerationLive = Layer.succeed(Moderation, { isAllowed: (text) => Effect.tryPromise({ try: async (signal) => { const response = await fetch("https://api.openai.com/v1/moderations", { method: "POST", headers: { authorization: `Bearer ${process.env.OPENAI_API_KEY}`, "content-type": "application/json", }, body: JSON.stringify({ model: "omni-moderation-latest", input: text }), signal, }); const body = (await response.json()) as { results: { flagged: boolean }[] }; return !body.results[0]?.flagged; }, catch: (cause) => new ModerationError({ cause }), }), }); ``` The processor is an Effect function. It sees every frame, as `processFrame` does, and passes on what should go on with `ctx.push`. `makeProcessor` returns an Effect that needs `Moderation`, so the processor gets whichever implementation the program is given: ```ts import { TranscriptionFrame, TTSSpeakFrame } from "@pipedog/core/frames/frames"; import { FrameDirection } from "@pipedog/core/processors/frame_processor"; import { makeProcessor } from "@pipedog/effect"; /** Lets a final transcription through only if Moderation allows it. */ const moderateTranscriptions = makeProcessor((frame, direction, ctx) => Effect.gen(function* () { if (!(frame instanceof TranscriptionFrame) || direction !== FrameDirection.DOWNSTREAM) { return yield* ctx.push(frame, direction); } const moderation = yield* Moderation; const allowed = yield* moderation.isAllowed(frame.text).pipe( Effect.retry({ times: 2, schedule: Schedule.exponential("100 millis") }), Effect.timeout("2 seconds"), // Fail open: a slow or failing moderation call must not stall the conversation. Effect.orElseSucceed(() => true), ); if (allowed) { yield* ctx.push(frame); } else { yield* ctx.push(new TTSSpeakFrame({ text: "Sorry, I can't help with that." })); } }), ); ``` A session builds the pipeline and runs it for as long as its scope is open: ```ts import { startPipeline } from "@pipedog/effect"; /** One voice session on `transport`, for as long as the scope is open. */ const session = (transport: BaseTransport) => Effect.gen(function* () { const context = new LLMContext(); const aggregators = new LLMContextAggregatorPair(context, { userParams: new LLMUserAggregatorParams({ vadAnalyzer: new SileroVADAnalyzer() }), }); const worker = new PipelineWorker( new Pipeline([ transport.input(), new OpenAISTTService(), yield* moderateTranscriptions, aggregators.user(), new OpenAILLMService({ settings: new OpenAILLMSettings({ systemInstruction: "You are a helpful assistant." }), }), new OpenAITTSService({ sampleRate: 24000 }), transport.output(), aggregators.assistant(), ]), { cancelOnIdleTimeout: false, params: new PipelineParams({ audioInSampleRate: 16000, audioOutSampleRate: 24000 }), }, ); // A hang-up cancels the pipeline, which interrupts this fiber's `join`. transport.eventHandler("on_client_disconnected", async () => { await worker.cancel({ reason: "client disconnected" }); }); const pipeline = yield* startPipeline(worker); yield* Effect.logInfo(`session ${worker.name} started`); yield* pipeline.join; }).pipe(Effect.scoped); ``` Your server forks one fiber per connection, with the live services: ```ts import * as Fiber from "effect/Fiber"; /** Called by your server for each connected client. */ export function onConnection(transport: BaseTransport): () => void { const fiber = Effect.runFork(session(transport).pipe(Effect.provide(ModerationLive))); // Interrupting the fiber cancels the pipeline and waits until it has stopped. return () => void Effect.runFork(Fiber.interrupt(fiber)); } ``` In a test, swap the Layer and look at what reaches the end of the pipeline: ```ts import { EndFrame, TextFrame } from "@pipedog/core/frames/frames"; import * as Stream from "effect/Stream"; const BlockEverything = Layer.succeed(Moderation, { isAllowed: () => Effect.succeed(false) }); const moderationTest = Effect.gen(function* () { const worker = new PipelineWorker(new Pipeline([yield* moderateTranscriptions]), { cancelOnIdleTimeout: false, }); const pipeline = yield* startPipeline(worker); const spoken = yield* pipeline.frames(TTSSpeakFrame); const passed = yield* pipeline.frames(TextFrame); yield* pipeline.queueFrames([ new TranscriptionFrame({ text: "something awful", userId: "u1", timestamp: "" }), new EndFrame(), ]); yield* pipeline.join; return { spoken: yield* Stream.runCollect(spoken), passed: yield* Stream.runCollect(passed) }; }).pipe(Effect.scoped, Effect.provide(BlockEverything)); ``` ## How a run ends [#how-a-run-ends] | The pipeline… | `join` | | ------------------------------------------------------------------------------------------------------- | --------------------------------------------------- | | ends with an `EndFrame` or `StopFrame` | succeeds | | is cancelled: `cancel()`, a `CancelWorkerFrame`, a processor that cancels it, or the scope closing | is interrupted | | is cancelled after an error from a processor that can no longer work (`ProcessorUnusablePolicy.CANCEL`) | fails with `PipelineError`, its `frame` set | | gets any `ErrorFrame`, with `failOnError: true` | is cancelled, and `join` fails with `PipelineError` | | throws from the worker | fails with `PipelineError`, its `cause` set | Without `failOnError`, `ErrorFrame`s are handled as usual: logged, passed to `on_pipeline_error`, and the pipeline goes on. ## Cancellation [#cancellation] Effect interrupts fibers; pipedog cancels tasks, as asyncio does (see [Interruptions and cancellation](https://polvallverdu.github.io/pipedog/docs/concepts/interruptions-and-cancellation.md)). The adapter maps one onto the other in both directions, and never lets either side outlive the other: * **Interrupting the Effect cancels the pipeline.** `startPipeline` is an `acquireRelease`: closing the scope (because the fiber was interrupted, timed out, or simply finished) cancels a pipeline that is still running and waits until the runner has finished, its processors cleaned up and its tasks done. Release is uninterruptible, as in any Effect finalizer. * **Cancelling the pipeline interrupts the Effect.** `join` is interrupted when the pipeline is cancelled from its side. A processor's Effect is interrupted when its task is cancelled, which happens when the pipeline is cancelled and when an `InterruptionFrame` (a user barging in) cancels the processing of an interruptible frame. The processor waits for the Effect's finalizers before it lets the cancellation through. * **Every call into the pipeline runs in a task of its own.** `queueFrame`, `stopWhenDone`, `cancel` and the rest start an asyncio task and wait for it: interrupting the Effect cancels that task and waits for it; the task ending cancelled interrupts the Effect. * **Frames a processor's Effect pushes are pushed by the processor's own task**, in order, as a Python processor pushes from `process_frame`. `startPipeline` turns the runner's signal handling off: an Effect app handles `SIGINT` with its runtime (`NodeRuntime.runMain` and friends interrupt the main fiber), and that interruption cancels the pipeline. A failure or defect of a processor's Effect becomes `pushError` on the processor: an `ErrorFrame` upstream, with the error as its `exception`. An Effect that interrupts itself drops the frame. # Docs for AI tools > Read these docs from an LLM or coding agent, with llms.txt, llms-full.txt and a Markdown version of every page. Source: https://polvallverdu.github.io/pipedog/docs/guides/llms/ Every page of these docs is also available as plain Markdown, for LLMs and coding agents. The Markdown is generated at build time from the same source as the HTML, so it is always in sync. ## llms.txt [#llmstxt] [`/llms.txt`](https://polvallverdu.github.io/pipedog/llms.txt) is the index: every page, by section, with its title, its description and a link to its Markdown. It follows the [llms.txt](https://llmstxt.org) convention. Give an agent this URL first, so it can pick the pages it needs. [`/llms-full.txt`](https://polvallverdu.github.io/pipedog/llms-full.txt) is every page in one file, for tools that take one big context. ## A page as Markdown [#a-page-as-markdown] Add `.md` to a page's URL: | Page | Markdown | | -------------------------------------- | ---------------------------------------- | | `/pipedog/docs/` | `/pipedog/docs.md` | | `/pipedog/docs/concepts/frames/` | `/pipedog/docs/concepts/frames.md` | | `/pipedog/docs/services/stt/deepgram/` | `/pipedog/docs/services/stt/deepgram.md` | Each file starts with the page's title, description and URL. Option tables are rendered as Markdown tables, and code samples are inlined from the type-checked snippets. ## On every page [#on-every-page] The buttons under each page title: * **Copy page** copies the page's Markdown to the clipboard. * **Open** has the Markdown view, the page's source on GitHub, and links that open ChatGPT, Claude, Cursor or Scira with a prompt to read the page. ## MCP server [#mcp-server] [`@pipedog/mcp`](https://www.npmjs.com/package/@pipedog/mcp) is a local MCP server for these docs. Coding agents (Claude Code, Cursor, VS Code, Windsurf and any other MCP client) start it over stdio and get four tools: | Tool | What it does | | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `list_pages(section?)` | Every page's slug, title and description, by section. `section` is a sidebar section (`Services`), a group (`Speech-to-text`) or a slug prefix (`services/stt`). | | `search_docs(query, limit?)` | Keyword search (BM25) over the page sections: the slug, title, section heading and a snippet. | | `read_page(slug)` | A page's full Markdown, e.g. `services/stt/deepgram`. | | `get_api(symbol)` | An exported class, function or type of the `@pipedog/*` packages: its import path, its constructor and options interface, its declaration from the `.d.ts` and the pages that document it. | Each page is also an MCP resource, `pipedog://docs/`, and llms.txt is `pipedog://llms.txt`. The docs ship inside the package: each release bundles the docs and the API of that release, so the server works offline and matches the pipedog version it came with. To keep the two together, pin it to your pipedog version: `npx -y @pipedog/mcp@`. With `--live` (or `PIPEDOG_MCP_LIVE=1`) it fetches the current docs from this site instead, and falls back to the bundled ones if the site doesn't answer within 10 seconds (`--timeout `). It needs Node.js 20 or later. ### From `pipedog init` [#from-pipedog-init] [`pipedog init`](https://polvallverdu.github.io/pipedog/docs/cli.md) writes the entries below into a new project's `.mcp.json`, `.cursor/mcp.json` and `.vscode/mcp.json`, next to an `AGENTS.md` that tells the agent to use them. `--no-mcp` skips it. ### Claude Code [#claude-code] ```bash claude mcp add pipedog -- npx -y @pipedog/mcp ``` Add `--scope project` to write it to the project's `.mcp.json`, for everyone on the repository. ### Cursor [#cursor] `.cursor/mcp.json` in the project, or `~/.cursor/mcp.json` for every project: ```json { "mcpServers": { "pipedog": { "command": "npx", "args": ["-y", "@pipedog/mcp"] } } } ``` ### VS Code [#vs-code] `.vscode/mcp.json` in the project (VS Code uses `servers`, not `mcpServers`): ```json { "servers": { "pipedog": { "type": "stdio", "command": "npx", "args": ["-y", "@pipedog/mcp"] } } } ``` ### Windsurf [#windsurf] `~/.codeium/windsurf/mcp_config.json` (in Devin Desktop, Windsurf's successor, `~/.config/devin/mcp_config.json`): ```json { "mcpServers": { "pipedog": { "command": "npx", "args": ["-y", "@pipedog/mcp"] } } } ``` ### Options [#options] | Option | Environment variable | | | ------------------ | ------------------------ | ---------------------------------------------------------------------- | | `--live` | `PIPEDOG_MCP_LIVE=1` | Read the current docs from the site, falling back to the bundled ones. | | `--docs-url ` | `PIPEDOG_DOCS_URL` | The docs site for `--live` (default: this site). | | `--timeout ` | `PIPEDOG_MCP_TIMEOUT_MS` | How long `--live` waits for the site (default 10000). | With Bun, `bunx @pipedog/mcp` works too. For live mode, add `"--live"` to `args`, or `-- npx -y @pipedog/mcp --live` for Claude Code. # Runtimes & compatibility > Running pipedog on Bun, Node, Deno and Cloudflare Workers, and which packages are verified where. Source: https://polvallverdu.github.io/pipedog/docs/guides/runtimes/ `@pipedog/core` and `@pipedog/asyncio` are runtime-agnostic: they run unchanged on **Bun**, **Node.js** (22 or later), **Deno** and **Cloudflare Workers** (workerd). Code that only works on some runtimes lives in its own package, and every package records where it is verified. ## Compatibility table [#compatibility-table] Generated at build time from the `pipecat.runtimes` field of each `packages/*/package.json`: `runtimes` lists only what is verified: the compat harness runs each package's smoke test (its `compat/smoke.ts`, or a default that imports every public entry) on each runtime, and `bun run compat --write` records the runtimes that passed. It is never edited by hand. An empty row means the package has not been run through the harness yet. A package that cannot work on a runtime (SmallWebRTC on Workers, which has no UDP) lists it in `pipecat.compat.unsupportedRuntimes`; the harness skips it there. Optional peers a package's users are expected to install go in `pipecat.compat.usesOptionalPeers`; all other optional peers are made unresolvable during the run, so a package that silently depends on one fails. Every package is verified on all four runtimes except `@pipedog/transport-smallwebrtc` (no UDP on Workers), `@pipedog/transport-livekit`, `@pipedog/heygen` and `@pipedog/transport-heygen` (LiveKit's media SDK is a native addon, which Workers can't load), the local models `@pipedog/kokoro`, `@pipedog/moonshine`, `@pipedog/funasr`, `@pipedog/pocket-tts` and the eSpeak NG phonemizer `@pipedog/espeak-ng` (model files and memory exceed what Workers allow), `@pipedog/transport-local` (it runs sox or ffmpeg on your machine), `@pipedog/mcp` (a stdio tool for Bun and Node) and `@pipedog/cli` (a command line tool: Bun, Node and Deno). The harness checks that each package loads and runs its smoke test; inside a verified package, a few entry points still need a server runtime: | Entry point | Runs on | Why | | ------------------------------------------------------------------------------------------------------- | ----------------------------- | --------------------------------------------------------------------------------------------------- | | Google Cloud Speech-to-Text and Text-to-Speech, and Gemini TTS on GCP (`@pipedog/google`) | Bun, Node, Deno | gRPC over HTTP/2 through Google's Node SDKs. The Gemini API services run on Workers too. | | `SageMakerBidiClient` (`@pipedog/aws`) | Bun, Node 22+ (Deno untested) | Full-duplex HTTP/2 through the AWS SDK. | | Nova Sonic with `transport: "http2"` (`@pipedog/aws`) | Bun, Node (Deno untested) | The default WebSocket transport runs everywhere, Workers included. | | `main()`, `serve()` (`@pipedog/runner`) | Bun, Node (with `ws`), Deno | They listen on a port. On Workers, use the fetch handler from `createRunnerApp()` (untested there). | | `serveFetch`, `websocketServe`, `SingleClientWebsocketServerTransport` (`@pipedog/transport-websocket`) | Bun, Node (with `ws`), Deno | They listen on a port. On Workers, use `websocketFetchHandler`. | Local turn detection (`@pipedog/audio-silero`, `@pipedog/audio-smart-turn`) runs on Workers with `onnxruntime-web`, but the models and runtime are large; provider-side turn detection is the usual choice there. See [Turn detection](https://polvallverdu.github.io/pipedog/docs/concepts/turn-detection.md). ## What core avoids [#what-core-avoids] To stay portable, `@pipedog/core` and `@pipedog/asyncio`: * import no `node:*` modules except `node:async_hooks` (for `AsyncLocalStorage`, in asyncio); * use `Uint8Array` for bytes, never `Buffer` (PCM is 16-bit little-endian); * use no Bun- or Deno-specific APIs; * use the global `fetch` and `WebSocket` for network I/O; * treat process signals as optional (`WorkerRunner` handles SIGINT/SIGTERM only where the runtime has them). ## Bun, Node and Deno [#bun-node-and-deno] The npm packages ship compiled JavaScript, and from a checkout Bun and Deno run the TypeScript sources directly (Node with its built-in type stripping). `WorkerRunner` installs a SIGINT handler by default, so Ctrl-C cancels the bot cleanly. Node needs version 22 or later. On Bun and Node, a WebSocket is upgraded by the server rather than from the `Request`: `serveFetch()` from `@pipedog/transport-websocket/serve` serves a fetch handler with upgrades on Bun, Node (with the `ws` package) and Deno, and on Bun you can also pass the server `Bun.serve` gives `fetch` to `websocketFetchHandler`'s handler. See [WebSocket](https://polvallverdu.github.io/pipedog/docs/transports/websocket.md). ## Cloudflare Workers [#cloudflare-workers] Workers are a good fit for pipedog bots that talk to clients over WebSockets, with a few constraints: * **Enable `nodejs_compat`.** Task tracking uses `AsyncLocalStorage` from `node:async_hooks`, which workerd provides under the `nodejs_compat` compatibility flag. ```jsonc title="wrangler.jsonc" { "compatibility_flags": ["nodejs_compat"] } ``` * **No WebRTC.** Workers can't run WebRTC, so the WebRTC-based transports (Daily, SmallWebRTC, LiveKit) don't work there. Use a **WebSocket transport** instead; WebSocket-based transports and serializers must work on Workers. Pipecat clients can connect over WebSocket with RTVI as usual. For WebRTC users, [Cloudflare Realtime](https://polvallverdu.github.io/pipedog/docs/transports/cloudflare-realtime.md) is the WebRTC peer and streams the audio to the bot over WebSockets. * **WASM is imported as a module.** workerd refuses to compile WebAssembly from bytes at runtime (`WebAssembly.compile(buffer)`), so a `.wasm` file must be imported. Packages with WASM ship two entries selected by the **`workerd` export condition**, which wrangler resolves automatically: ```json title="package.json" { "exports": { ".": { "types": "./src/index.ts", "workerd": "./src/workerd.ts", "default": "./src/index.ts" } } } ``` The `workerd` entry does `import wasm from "../wasm/x.wasm"` and instantiates that `WebAssembly.Module`; the default entry loads the bytes (a `fetch` of `new URL(..., import.meta.url)`). Both accept an explicit `WebAssembly.Module | BufferSource` so apps can supply their own. `@pipedog/resampler-wasm` is built this way. * **Models are never bundled.** Packages that need a model (Silero VAD, Smart Turn) load it from a path or URL you give them, with the upstream file name as the default. On Workers, serve it from R2, KV, static assets or a `fetch`. * **No threads, limited CPU.** Upstream work done in a thread pool (`run_in_executor`) runs inline or in an injected runner. Keep per-call CPU low: Workers have CPU-time limits. * **No process signals.** `handleSigint` is a no-op; end the runner from your Worker code (for example when the WebSocket closes) with `runner.end()` or `runner.cancel()`. * **Many sessions per isolate.** One isolate serves many requests and Durable Objects. `@pipedog/asyncio` runs each callback in the I/O context that scheduled it, so parallel sessions in separate Durable Objects don't do each other's I/O. `examples/workers-durable-voice-agent` tests this with parallel sessions on workerd. ## Native and ML dependencies [#native-and-ml-dependencies] Where Pipecat uses native modules (onnxruntime, Krisp, Koala, soxr), the pipedog package takes the model as an external file or URL and accepts an injected runtime: `onnxruntime-node` on Bun/Node, `onnxruntime-web` (WASM) on Workers. Pure-algorithm fallbacks, such as a linear resampler, may live in core. # Introduction > pipedog is an unofficial TypeScript port of Pipecat, the framework for real-time voice and multimodal conversational AI. Source: https://polvallverdu.github.io/pipedog/docs/ pipedog is an **unofficial** TypeScript port of [Pipecat](https://github.com/pipecat-ai/pipecat), the open source Python framework (by [Daily](https://www.daily.co), BSD-2-Clause) for building voice and multimodal AI agents. Pipecat orchestrates AI services, network transports and audio processing for ultra-low latency conversations; pipedog brings the same architecture to **Bun, Node, Deno and Cloudflare Workers**. pipedog is a community port. It is not affiliated with or endorsed by Daily or the Pipecat project. The core runtime, most providers, Pipecat Flows, evals and the CLI are ported; Daily and a few providers (NVIDIA, Simli, Tavus) are not yet. Everything is tested against mocks and recorded protocol vectors, not yet against the live providers. See the [roadmap](https://polvallverdu.github.io/pipedog/docs/roadmap.md). Run a pipeline with a custom frame processor in a few lines. How Pipecat's Python API maps to pipedog. Frames, frame processors, pipelines, workers and cancellation. What runs where: Bun, Node, Deno, Cloudflare Workers. LLM, speech-to-text, text-to-speech and realtime providers, one page each. Mount a voice agent in SvelteKit, Hono, Bun, Next.js, TanStack Start and more. Pipecat's `main()` runner: the prebuilt client, WebRTC and phone calls. Phone bots on Twilio, Telnyx, Plivo, Exotel, Vonage and Genesys. ## How it works [#how-it-works] Like Pipecat, pipedog orchestrates work in a **pipeline**: a series of **frame processors** that handle real-time audio, text and video **frames**. In a typical voice conversation: 1. A **transport** receives audio from the user (browser, phone, ...). 2. **Speech recognition** converts speech to text in real time. 3. An **LLM** generates a response from the conversation context. 4. **Speech synthesis** converts the response back to speech. 5. The **transport** streams the audio back to the user. A `PipelineWorker` runs a pipeline and a `WorkerRunner` runs workers: one for a simple bot, or many that talk to each other over a shared message bus for multi-agent setups. ## Relation to Pipecat [#relation-to-pipecat] pipedog tracks a pinned upstream Pipecat commit and ports it module by module, test by test. The goal is that someone who knows Pipecat can read pipedog code (and these docs) without a dictionary: * **Same names.** Classes, frames, module paths and file names are the same as upstream: `src/pipecat/processors/frame_processor.py` becomes `@pipedog/core/processors/frame_processor`. * **Same event names.** `"on_pipeline_started"`, `"on_client_ready"`, ... are unchanged strings. * **camelCase members.** `push_frame()` is `pushFrame()`, `audio_in_sample_rate` is `audioInSampleRate`. * **Options objects.** Python keyword arguments become one options object: `new TextFrame({ text: "hi" })`. * **Seconds.** Durations and timeouts stay in seconds, so upstream configs carry over. * **asyncio semantics.** Tasks, cancellation, queues and events behave like Python's `asyncio`, through [`@pipedog/asyncio`](https://polvallverdu.github.io/pipedog/docs/concepts/interruptions-and-cancellation.md). * **Deprecated APIs are not ported.** Things upstream marks as deprecated (for example the `PipelineTask` / `PipelineRunner` aliases) are left out. The upstream test suite is ported 1:1, and is how behavioral parity is proven. ## Compatibility with Pipecat clients (RTVI) [#compatibility-with-pipecat-clients-rtvi] pipedog keeps Pipecat's **wire protocols** unchanged: RTVI messages, serializers and dict keys on the wire are byte-for-byte what Pipecat sends. A protocol is not an API, so nothing there is camelCased. That means Pipecat's client SDKs (`@pipecat-ai/client-js`, the React voice-ui-kit, the prebuilt UIs) can talk to a pipedog bot unchanged. `PipelineWorker` adds RTVI support automatically unless you pass `enableRtvi: false`. ## Status [#status] The core runtime (frames, processors, pipelines, workers, bus, observers, RTVI, base services and transports) is ported in full, with its upstream test suite. So are the main providers: OpenAI, Anthropic, Google, Azure, AWS (including Nova Sonic), Deepgram, Cartesia, ElevenLabs, Inworld, Ultravox, xAI, about 30 more speech providers (AssemblyAI, Speechmatics, Gladia, Soniox, Rime, Fish, Hume, Sarvam and others) and 16 OpenAI-compatible providers, with the WebSocket, SmallWebRTC, LiveKit, WhatsApp and local audio transports, HeyGen avatars, six telephony serializers (Twilio, Telnyx, Plivo, Exotel, Vonage, Genesys) and the [development runner](https://polvallverdu.github.io/pipedog/docs/runner.md). Pipecat Flows, evals, the [`pipedog` CLI](https://polvallverdu.github.io/pipedog/docs/cli.md), the RNNoise filter, the soundfile mixer, HTTP Smart Turn, the [MCP client](https://polvallverdu.github.io/pipedog/docs/services/mcp-client.md) and the keyless local speech models (Whisper, Moonshine, FunASR, Kokoro, Piper, Pocket TTS) are ported too. In all, 622 of upstream's 667 modules (93%) and 305 of its 340 test files are ported, across 87 packages. Provider services are tested against mock servers and recorded protocol vectors, and the examples run end to end against mock APIs. Testing against the real providers is the next phase, so treat pipedog as 0.x software: try it, and report what breaks. [Services](https://polvallverdu.github.io/pipedog/docs/services.md) lists every provider, ported or coming soon, and the [roadmap](https://polvallverdu.github.io/pipedog/docs/roadmap.md) says what isn't ported yet. The generated [porting status](https://polvallverdu.github.io/pipedog/docs/contributing/status.md) (`porting/STATUS.md`) lists every package, how much of it is ported, and which runtimes it is verified on. ## License [#license] pipedog is BSD-2-Clause, like Pipecat. The original Pipecat code is Copyright (c) 2024–2026 Daily; the port is Copyright (c) 2026 pipedog contributors. Parts of these docs are adapted from the [Pipecat documentation](https://github.com/pipecat-ai/docs) (Copyright (c) 2024–2025 Daily, BSD-2-Clause), with every Python example rewritten in TypeScript against the pipedog API. # Installation > Add pipedog to a Bun, Node, Deno or Cloudflare Workers project. Source: https://polvallverdu.github.io/pipedog/docs/installation/ pipedog is split into packages, like Pipecat's optional extras. They are published on npm under the `@pipedog` scope and released together at one version: upgrade them together. | Package | What | | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `@pipedog/asyncio` | asyncio semantics for JavaScript: tasks, cooperative cancellation, queues, events | | `@pipedog/core` | frames, processors, pipelines, workers, bus, observers, RTVI, base services, transports and serializers | | `@pipedog/tracing` | OpenTelemetry setup (`setupTracing()`, OTLP and console exporters), like `pipecat-ai[tracing]` | | `@pipedog/sentry` | [Sentry metrics](https://polvallverdu.github.io/pipedog/docs/concepts/sentry-metrics.md): `SentryMetrics` reports TTFB and processing time as Sentry transactions, with the Sentry SDK of your runtime | | `@pipedog/audio-silero`, `@pipedog/audio-smart-turn`, `@pipedog/onnx` | Silero VAD and Smart Turn (local v3, and an HTTP analyzer for a remote endpoint), and the ONNX backend they run on | | `@pipedog/audio-ten-vad` | [TEN VAD](https://polvallverdu.github.io/pipedog/docs/audio/ten-vad.md), an opt-in alternative to Silero on TEN's WebAssembly build, which you obtain from TEN (under TEN's license) and pass in | | `@pipedog/audio-rnnoise-filter`, `@pipedog/audio-soundfile-mixer`, `@pipedog/audio-ducking` | RNNoise noise suppression (WebAssembly), background sound from WAV files mixed into the bot's audio, and ducking (the bot's voice drops while the user talks over it) | | `@pipedog/openai`, `@pipedog/google`, `@pipedog/anthropic`, `@pipedog/azure`, `@pipedog/aws` | LLM, speech, realtime and image services | | `@pipedog/deepgram`, `@pipedog/cartesia`, `@pipedog/elevenlabs`, `@pipedog/inworld` | speech-to-text and text-to-speech (Inworld also realtime) | | `@pipedog/cloudflare` | [Cloudflare Workers AI](https://polvallverdu.github.io/pipedog/docs/deployment/cloudflare-workers-ai.md): Flux, Nova-3 and Whisper speech-to-text, Aura and MeloTTS text-to-speech, LLMs and Smart Turn v2, over the `AI` binding or the REST API | | `@pipedog/whisper`, `@pipedog/moonshine`, `@pipedog/funasr` | local speech-to-text on ONNX Runtime, no API key: [Whisper](https://polvallverdu.github.io/pipedog/docs/services/stt/whisper.md) (also the base for Whisper-compatible APIs), [Moonshine](https://polvallverdu.github.io/pipedog/docs/services/stt/moonshine.md), [FunASR SenseVoice](https://polvallverdu.github.io/pipedog/docs/services/stt/funasr.md) | | `@pipedog/kokoro`, `@pipedog/piper`, `@pipedog/pocket-tts`, `@pipedog/espeak-ng` | local text-to-speech on ONNX Runtime, no API key: [Kokoro](https://polvallverdu.github.io/pipedog/docs/services/tts/kokoro.md), [Piper](https://polvallverdu.github.io/pipedog/docs/services/tts/piper.md) (also over HTTP), [Pocket TTS](https://polvallverdu.github.io/pipedog/docs/services/tts/pocket-tts.md), and the eSpeak NG phonemizer Kokoro and Piper use | | `@pipedog/rime`, `@pipedog/fish`, `@pipedog/hume` | text-to-speech | | `@pipedog/assemblyai`, `@pipedog/speechmatics`, `@pipedog/gladia`, `@pipedog/soniox`, `@pipedog/zoom-ai` | speech-to-text (AssemblyAI, Speechmatics and Soniox also text-to-speech) | | `@pipedog/neuphonic`, `@pipedog/speechify`, `@pipedog/minimax`, `@pipedog/resembleai`, `@pipedog/camb`, `@pipedog/asyncai`, `@pipedog/bland` | text-to-speech, one package per provider | | `@pipedog/ultravox` | Ultravox realtime speech-to-speech | | `@pipedog/groq`, `@pipedog/mistral`, `@pipedog/together`, `@pipedog/xai` | OpenAI-compatible LLMs with the provider's STT and TTS (xAI also realtime); `@pipedog/grok` is the deprecated alias of `@pipedog/xai` | | `@pipedog/baseten`, `@pipedog/cerebras`, `@pipedog/crusoe`, `@pipedog/deepseek`, `@pipedog/fireworks`, `@pipedog/nebius`, `@pipedog/novita`, `@pipedog/ollama`, `@pipedog/openrouter`, `@pipedog/qwen`, `@pipedog/sambanova` | OpenAI-compatible LLMs, one package per provider | | `@pipedog/sarvam`, `@pipedog/smallest`, `@pipedog/gradium`, `@pipedog/meta`, `@pipedog/inception` | Sarvam (LLM, STT, TTS), Smallest AI (LLM, STT, TTS), Gradium (STT, TTS), Meta (STT) and Inception (LLM) | | `@pipedog/nvidia` | [NVIDIA](https://polvallverdu.github.io/pipedog/docs/services/llm/nvidia.md) NIM LLMs, and Nemotron Omni, an LLM that hears the user's speech | | `@pipedog/tone` | the models for [Pipecat Tone](https://polvallverdu.github.io/pipedog/docs/concepts/tone.md)'s reader and tagger processors (expressive conversation, alpha) | | `@pipedog/perplexity` | [Perplexity](https://polvallverdu.github.io/pipedog/docs/services/llm/perplexity.md)'s Agent API: web-search answers over the OpenAI Responses protocol | | `@pipedog/langchain` | the LangChain processor | | `@pipedog/ai-sdk` | [Vercel AI SDK](https://polvallverdu.github.io/pipedog/docs/services/llm/ai-sdk.md): any AI SDK language model as the LLM service | | `@pipedog/flows` | [Flows](https://polvallverdu.github.io/pipedog/docs/flows.md): structured conversations from nodes, functions and actions | | `@pipedog/transport-websocket`, `@pipedog/transport-smallwebrtc` | transports | | `@pipedog/transport-cloudflare-realtime` | [Cloudflare Realtime](https://polvallverdu.github.io/pipedog/docs/transports/cloudflare-realtime.md): WebRTC users through the Realtime SFU, for bots in Workers and Durable Objects | | `@pipedog/transport-livekit` | [LiveKit](https://polvallverdu.github.io/pipedog/docs/transports/livekit.md) rooms (needs `@livekit/rtc-node`) | | `@pipedog/transport-whatsapp` | [WhatsApp](https://polvallverdu.github.io/pipedog/docs/transports/whatsapp.md) Business voice calls, on SmallWebRTC | | `@pipedog/transport-local` | [local audio](https://polvallverdu.github.io/pipedog/docs/transports/local-audio.md): microphone and speakers through sox or ffmpeg | | `@pipedog/heygen`, `@pipedog/transport-heygen` | [HeyGen](https://polvallverdu.github.io/pipedog/docs/services/video/heygen.md) avatars, as a video service and as a transport | | `@pipedog/serializer-protobuf`, `@pipedog/serializer-rtvi-client` | serializers for Pipecat clients | | `@pipedog/serializer-twilio`, `@pipedog/serializer-telnyx`, `@pipedog/serializer-plivo`, `@pipedog/serializer-exotel`, `@pipedog/serializer-vonage`, `@pipedog/serializer-genesys` | telephony serializers (see [Telephony](https://polvallverdu.github.io/pipedog/docs/telephony.md)) | | `@pipedog/runner` | the [development runner](https://polvallverdu.github.io/pipedog/docs/runner.md): `main()`, `createTransport()`, the prebuilt client | | `@pipedog/extensions` | [telephony extensions](https://polvallverdu.github.io/pipedog/docs/telephony/extensions.md): `IVRNavigator` for phone menus and `VoicemailDetector` for outbound calls | | `@pipedog/evals` | [evals](https://polvallverdu.github.io/pipedog/docs/evals.md): end-to-end tests of a bot with scripted or simulated users and an LLM judge, and the `pipedog-eval` CLI | | `@pipedog/cli` | the [`pipedog` command](https://polvallverdu.github.io/pipedog/docs/cli.md): `pipedog init` scaffolds a bot project and sets up coding agents, `pipedog eval` runs evals | | `@pipedog/resampler-wasm`, `@pipedog/sentencex-wasm` | high-quality resampling and sentence segmentation, compiled to WebAssembly | | `@pipedog/effect` | the [Effect adapter](https://polvallverdu.github.io/pipedog/docs/guides/effect.md): run a pipeline as a scoped Effect, frames in and out as `Stream`s, processors as Effect functions | | `@pipedog/mcp-client` | the [MCP client](https://polvallverdu.github.io/pipedog/docs/services/mcp-client.md): `MCPClient` connects to an MCP server over stdio, SSE or streamable HTTP and gives its tools to an LLM service | | `@pipedog/mcp` | the docs as a local MCP server for coding agents (see [Docs for AI tools](https://polvallverdu.github.io/pipedog/docs/guides/llms/#mcp-server)) | To start a project from a template instead, run `bunx @pipedog/cli init` (see [CLI](https://polvallverdu.github.io/pipedog/docs/cli.md)): it writes a bot with the packages below already in its `package.json`. Where Python installs `pipecat-ai[deepgram,openai]`, you add `@pipedog/core` plus `@pipedog/deepgram` and `@pipedog/openai`. [Services](https://polvallverdu.github.io/pipedog/docs/services.md) lists every provider and its package, and [Transports](https://polvallverdu.github.io/pipedog/docs/transports.md) the transports. ## Package managers [#package-managers] ```bash bun add @pipedog/core @pipedog/asyncio ``` ```bash npm install @pipedog/core @pipedog/asyncio ``` ```bash pnpm add @pipedog/core @pipedog/asyncio ``` ```bash deno add npm:@pipedog/core npm:@pipedog/asyncio ``` ## Requirements [#requirements] * **Bun**, **Node.js** 22 or later, **Deno**, or **Cloudflare Workers** (with the `nodejs_compat` compatibility flag, which provides `AsyncLocalStorage`). * The npm packages ship compiled JavaScript (ES modules) with `.d.ts` types, so every runtime and bundler loads them as they are. * From source, the packages are TypeScript written with erasable syntax only (no `enum`, `namespace` or parameter properties), so Bun and Deno run them directly and Node runs them with its built-in type stripping. If you type-check against the sources with `tsc`, use `"moduleResolution": "Bundler"` (or `"NodeNext"`) and `"allowImportingTsExtensions": true`. ## Imports [#imports] Modules are imported by their upstream path, so a Pipecat import translates directly: ```python from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.worker import PipelineParams, PipelineWorker from pipecat.workers.runner import WorkerRunner ``` ```ts import { Pipeline } from "@pipedog/core/pipeline/pipeline"; import { PipelineParams, PipelineWorker } from "@pipedog/core/pipeline/worker"; import { WorkerRunner } from "@pipedog/core/workers/runner"; ``` ## From source [#from-source] To work on pipedog itself, or to run unreleased changes: pipedog is a Bun monorepo. Clone it, install, and add your app as a workspace (for example under `examples/`), depending on the packages with `workspace:*`: ```bash bun install bun test # the ported upstream test suite bun run typecheck # tsc over every package ``` ```json title="examples/my-bot/package.json" { "name": "my-bot", "private": true, "type": "module", "dependencies": { "@pipedog/asyncio": "workspace:*", "@pipedog/core": "workspace:*" } } ``` Then continue with the [Quickstart](https://polvallverdu.github.io/pipedog/docs/quickstart.md). # Performance > pipedog on Bun, Node, Deno and Cloudflare Workers against Pipecat on CPython, measured on the same machine with the same scenarios. Source: https://polvallverdu.github.io/pipedog/docs/performance/ pipedog and Pipecat run the same pipelines: a frame goes through the same processors, the same queues and the same `PipelineWorker`. This page measures what that costs on each runtime, offline (the framework alone) and against Gemini (a real service, where the network dominates). Every scenario is written twice, once against each framework, in the same shape: [`tools/bench/perf/pipedog.ts`](https://github.com/polvallverdu/pipedog/blob/main/tools/bench/perf/pipedog.ts) and [`tools/bench/perf/pipecat_bench.py`](https://github.com/polvallverdu/pipedog/blob/main/tools/bench/perf/pipecat_bench.py). Pipecat runs from the upstream commit pipedog is ported from. Both log at `WARNING`. pipedog runs from a bundle made with `bun build`, as it is deployed. The runtimes take turns within each round, so a slow minute on the network or on the machine falls on all of them. ## Results [#results] {/* perf:start */} ### Frame processing [#frame-processing] 10 pass-through processors in a `PipelineWorker`. **Throughput**: 20,000 `TextFrame`s queued at once. **Latency**: one frame at a time, queued to arrival at the end of the pipeline. In brackets: compared with Pipecat (higher is better). | | Throughput (frames/s) | Latency per frame (median) | p90 | Peak memory | | ------------------------------ | --------------------: | -------------------------: | -----: | ----------: | | Pipecat · CPython 3.13.13 | 9,294 (1.0×) | 572 µs (1.0×) | 578 µs | 144 MB | | pipedog · Bun 1.4.2 | 44,394 (4.8×) | 52 µs (11.0×) | 60 µs | 184 MB | | pipedog · Node 24.14.0 | 52,350 (5.6×) | 41 µs (13.8×) | 58 µs | 278 MB | | pipedog · Deno 2.9.5 | 53,288 (5.7×) | 59 µs (9.7×) | 74 µs | 221 MB | | pipedog · workerd 1.20261005.1 | 45,198 (4.9×) | 180 µs (3.2×) | 239 µs | – | ### Cold start [#cold-start] From starting the process to a pipeline that has started and ended, with the Google services loaded (for workerd: from starting `workerd serve` to the response). *Runtime alone* is the same for an empty program, so the difference is what loading the framework and running the pipeline cost. | | Cold start | Runtime alone | Framework | Peak memory | Bundle | | ----------------- | ---------: | ------------: | --------: | ----------: | -----: | | Pipecat · CPython | 396 ms | 13 ms | 383 ms | 114 MB | – | | pipedog · Bun | 44 ms | 6 ms | 38 ms | 51 MB | 3.5 MB | | pipedog · Node | 66 ms | 21 ms | 45 ms | 87 MB | 3.8 MB | | pipedog · Deno | 47 ms | 20 ms | 27 ms | 90 MB | 3.1 MB | | pipedog · workerd | 42 ms | 11 ms | 31 ms | – | 1.7 MB | ### Gemini LLM: time to first token [#gemini-llm-time-to-first-token] `GoogleLLMService` with `gemini-3.5-flash-lite` (thinking at its minimum, as the service sends by default), from queueing an `LLMContextFrame` to the first `LLMTextFrame` at the end of the pipeline. *SDK alone* is the same request made with the Google GenAI SDK of the same language, in the same process, alternating with the pipeline. | | Through the pipeline (median) | SDK alone (median) | Difference, paired (median) | Middle half of differences | Requests | | ----------------- | ----------------------------: | -----------------: | --------------------------: | -------------------------: | -------: | | Pipecat · CPython | 529 ms | 491 ms | -6 ms | -41 to 104 ms | 36 | | pipedog · Bun | 497 ms | 452 ms | 15 ms | -24 to 116 ms | 36 | | pipedog · Node | 516 ms | 500 ms | -21 ms | -173 to 166 ms | 36 | | pipedog · Deno | 527 ms | 513 ms | 40 ms | -30 to 93 ms | 36 | | pipedog · workerd | 496 ms | 467 ms | 51 ms | -68 to 129 ms | 36 | ### Gemini TTS: time to first audio [#gemini-tts-time-to-first-audio] `GeminiTTSService` with `gemini-3.8-flash-tts`, from queueing a `TTSSpeakFrame` to the first `TTSAudioRawFrame` at the end of the pipeline, and the same request made with the SDK. | | Through the pipeline (median) | SDK alone (median) | Difference, paired (median) | Middle half of differences | Requests | | ----------------- | ----------------------------: | -----------------: | --------------------------: | -------------------------: | -------: | | Pipecat · CPython | 758 ms | 664 ms | 127 ms | 15 to 204 ms | 24 | | pipedog · Bun | 741 ms | 626 ms | 127 ms | 43 to 192 ms | 24 | | pipedog · Node | 753 ms | 727 ms | 30 ms | -42 to 117 ms | 24 | | pipedog · Deno | 786 ms | 803 ms | 84 ms | -126 to 253 ms | 24 | | pipedog · workerd | 876 ms | 642 ms | 170 ms | 92 to 294 ms | 24 | Measured 2026-10-22 on Apple M4 Pro (12 cores, 24 GB), darwin 25.6.0: pipedog 0.8.0, Pipecat 5558d26dca (pinned upstream) on CPython 3.13.13, Bun 1.4.2, Node 24.14.0, Deno 2.9.5, workerd 1.20261005.1. 6 rounds, runtimes taking turns. {/* perf:end */} ## Reading the numbers [#reading-the-numbers] **Frame processing** is the framework alone: no I/O, no models. It's what a pipeline spends moving frames between processors, so it sets the floor under every response. In a voice bot each audio chunk (every 10–20 ms) and each LLM token goes through it. **Workers** run pipedog's event loop the slowest of the JavaScript runtimes. Its timers and `setImmediate` wait about a millisecond, so `@pipedog/asyncio` yields through a body read instead, and a frame takes several such yields across 10 processors. That is still well under a millisecond per frame. **Peak memory** in the JavaScript runtimes is mostly garbage the collector hasn't reclaimed yet, not memory in use. With 20,000 frames sent through, the live heap after a forced collection grows by about 5 MB on Node, and drops back once the pipeline ends. CPython frees objects as soon as the last reference goes, so its peak stays closer to what it holds. Under memory pressure V8 collects earlier; `--max-old-space-size` caps it. **Cold start** matters for serverless (a Worker or a function that starts per call) and for scaling out. Pipecat's is mostly importing Python modules (here with the Google services). A bundle loads in one read. **Gemini** times are dominated by the network and the model, which vary by hundreds of milliseconds from one request to the next. The *SDK alone* column makes the same request with Google's SDK, in the same process, right before or after each pipeline request. The paired difference is the framework's share, and is mostly within the noise for both frameworks. TTS services send audio downstream in chunks of a fixed size (as Pipecat does), so the first audio frame waits for the first full chunk, while the SDK column stops the clock at the first byte. ## Telephony audio [#telephony-audio] On a phone call, the bot's speech goes through the same steps on every call: the text-to-speech audio (usually 24 kHz) is resampled to the provider's rate (8 kHz for Twilio, Telnyx and others), μ-law encoded, and sent over a WebSocket in small messages paced in real time. These steps decide what the caller hears, so they are measured directly against Pipecat's. ### Resampling and μ-law [#resampling-and-μ-law] Pipecat resamples with libsoxr at its highest quality (VHQ). pipedog can't load native code on every runtime, so it uses a windowed-sinc resampler compiled to WebAssembly (`@pipedog/resampler-wasm`), with a pure-TypeScript fallback. Both get the same test signal: three tones they must keep (300 Hz, 1 kHz, 2.5 kHz) and three above 4 kHz that a resampler going down to 8 kHz must remove. μ-law is checked bit for bit against Pipecat's (Python's `audioop`). {/* audio:start */} | Resampling | Pipecat (libsoxr, VHQ) | pipedog (WebAssembly) | pipedog (TypeScript fallback) | 20 ms chunks vs one go: Pipecat / pipedog | | --------------- | ---------------------: | --------------------: | ----------------------------: | ----------------------------------------: | | 24 kHz → 8 kHz | -83 / -104 dB | -88 / -98 dB | -88 / -95 dB | -80 dB / identical | | 24 kHz → 16 kHz | -83 / -105 dB | -87 / -100 dB | -88 / -100 dB | -81 dB / identical | | 16 kHz → 8 kHz | -83 / -102 dB | -87 / -100 dB | -87 / -97 dB | -80 dB / identical | | 8 kHz → 16 kHz | -82 / -97 dB | -86 / -98 dB | -86 / -96 dB | -80 dB / identical | | 24 kHz → 48 kHz | -82 / -100 dB | -85 / -99 dB | -86 / -101 dB | -81 dB / identical | | 48 kHz → 16 kHz | -83 / -107 dB | -87 / -100 dB | -88 / -100 dB | -81 dB / identical | Each cell: everything besides the tones (noise, aliasing, distortion) / the largest single artefact, relative to the tones. Lower is cleaner; both are far below what can be heard. **μ-law:** all 65,536 16-bit values encoded: identical to Pipecat's; all 256 codes decoded: identical. {/* audio:end */} ### Pacing [#pacing] A bot plays two 6-second utterances through the WebSocket server transport and the Telnyx serializer, and a separate process standing in for Telnyx records when each audio message arrives. The *dropouts* are what a playback buffer of that size would run out of: silences a caller would hear. Each case runs with the bot's event loop idle and kept busy by other work, standing in for VAD, other calls and LLM streaming. Both frameworks use the same pacing code; the message sizes differ because libsoxr's stream resampler returns audio in blocks, while pipedog's returns it sample by sample, so pipedog sends a small message about every 40 ms and Pipecat a larger one every 120–160 ms. Telephony providers buffer more than 40 ms, so neither drops audio in practice. {/* pacing:start */} | | Event loop busy | Gap between messages, median / worst | Dropouts, 20 ms buffer | Dropouts, 40 ms buffer | Dropouts, 80 ms buffer | | ----------------- | --------------: | -----------------------------------: | ---------------------: | ---------------------: | ---------------------: | | Pipecat · CPython | 0% | 121 / 163 ms | 8 (27 ms of silence) | none | none | | pipedog · Bun | 0% | 40 / 48 ms | none | none | none | | pipedog · Node | 0% | 40 / 44 ms | none | none | none | | pipedog · Deno | 0% | 40 / 47 ms | none | none | none | | Pipecat · CPython | 40% | 122 / 163 ms | 8 (28 ms of silence) | none | none | | pipedog · Bun | 40% | 40 / 60 ms | none | none | none | | pipedog · Node | 40% | 41 / 53 ms | none | none | none | | pipedog · Deno | 40% | 39 / 50 ms | none | none | none | | Pipecat · CPython | 70% | 125 / 167 ms | 7 (36 ms of silence) | none | none | | pipedog · Bun | 70% | 41 / 43 ms | none | none | none | | pipedog · Node | 70% | 41 / 43 ms | none | none | none | | pipedog · Deno | 70% | 38 / 47 ms | none | none | none | {/* pacing:end */} ## Running it [#running-it] ```sh bun install GOOGLE_API_KEY=... bun run bench:perf --write # without a key, the Gemini scenarios are skipped bun run bench:perf --runtime bun,python --scenario hop --rounds 1 bun run bench:audio --write # resampling and μ-law against Pipecat's bun run bench:pacing --write # telephony pacing (--runtime, --load to narrow it down) ``` The first run creates a Python 3.13 environment with [uv](https://docs.astral.sh/uv/) and installs Pipecat from the pinned upstream checkout (`bun run upstream` restores it). `--write` saves the raw samples to [`tools/bench/perf/results.json`](https://github.com/polvallverdu/pipedog/blob/main/tools/bench/perf/results.json) and the tables above. Model inference (Silero VAD, Smart Turn) has its own benchmark, `bun run bench`; its results are in the [`@pipedog/onnx` README](https://github.com/polvallverdu/pipedog/tree/main/packages/onnx#readme). # Quickstart > Build and run your first pipedog pipeline with a custom frame processor. Source: https://polvallverdu.github.io/pipedog/docs/quickstart/ This quickstart builds the smallest useful pipedog program: a pipeline of two custom frame processors, run by a `PipelineWorker` inside a `WorkerRunner`. No API keys, no network. The [voice bot guide](https://polvallverdu.github.io/pipedog/docs/voice-bot.md) runs an assistant you talk to in the browser, with OpenAI speech-to-text, chat and text-to-speech over SmallWebRTC. [Examples](https://polvallverdu.github.io/pipedog/docs/examples.md) lists every runnable example, including the same bot over a WebSocket. ### Set up a project [#set-up-a-project] Follow [Installation](https://polvallverdu.github.io/pipedog/docs/installation.md) to get `@pipedog/core` and `@pipedog/asyncio` into a project. Any of Bun, Node, Deno or Cloudflare Workers works; this page uses Bun. ### Write the bot [#write-the-bot] Create `quickstart.ts`. It defines two processors: one upper-cases every `TextFrame`, the other prints the text frames that reach it. Both pass every other frame through untouched. ```ts twoslash title="quickstart.ts" import { EndFrame, type Frame, TextFrame } from "@pipedog/core/frames/frames"; import { Pipeline } from "@pipedog/core/pipeline/pipeline"; import { PipelineWorker } from "@pipedog/core/pipeline/worker"; import { type FrameDirection, FrameProcessor } from "@pipedog/core/processors/frame_processor"; import { WorkerRunner } from "@pipedog/core/workers/runner"; /** Upper-cases every TextFrame and passes every other frame through. */ class UpperCaseProcessor extends FrameProcessor { override async processFrame(frame: Frame, direction: FrameDirection): Promise { await super.processFrame(frame, direction); if (frame instanceof TextFrame) { await this.pushFrame(new TextFrame({ text: frame.text.toUpperCase() }), direction); } else { await this.pushFrame(frame, direction); } } } /** Prints the text frames that reach it. */ class PrintProcessor extends FrameProcessor { override async processFrame(frame: Frame, direction: FrameDirection): Promise { await super.processFrame(frame, direction); if (frame instanceof TextFrame) { console.log(`${this.name}: ${frame.text}`); } await this.pushFrame(frame, direction); } } const pipeline = new Pipeline([new UpperCaseProcessor(), new PrintProcessor()]); const worker = new PipelineWorker(pipeline, { enableRtvi: false }); worker.eventHandler("on_pipeline_started", async () => { await worker.queueFrames([ new TextFrame({ text: "hello" }), new TextFrame({ text: "from pipedog" }), new EndFrame(), ]); }); worker.eventHandler("on_pipeline_finished", async (_worker: PipelineWorker, frame: Frame) => { console.log(`pipeline finished with ${frame.name}`); }); const runner = new WorkerRunner({ handleSigint: false }); await runner.addWorkers(worker); await runner.run(); ``` Hover over the code to see the types: this sample is type-checked against the real `@pipedog/core` sources when the docs are built. ### Run it [#run-it] ```bash bun quickstart.ts ``` Among the debug logs you will see: ```text PrintProcessor#0: HELLO PrintProcessor#0: FROM PIPEDOG pipeline finished with EndFrame#0 ``` ## What just happened [#what-just-happened] 1. `new Pipeline([...])` linked the processors in order, between a source and a sink. 2. `new PipelineWorker(pipeline, ...)` wrapped the pipeline with lifecycle management. When the runner started it, it pushed a `StartFrame` through the pipeline; once that reached the end, `on_pipeline_started` fired. 3. The handler queued two `TextFrame`s and an `EndFrame`. The `EndFrame` is a control frame, so it is processed in order, after the text frames, and ends the pipeline gracefully once everything ahead of it is flushed. 4. `runner.run()` returned once its only worker finished. `enableRtvi: false` turns off the RTVI processor the worker adds by default, since this bot has no client. `handleSigint: false` keeps the runner from installing a Ctrl-C handler; leave it at its default (`true`) in a real bot. ## Next steps [#next-steps] Runnable voice bots over WebRTC, WebSocket and Durable Objects. System, data and control frames, and how interruptions treat them. `processFrame`, `pushFrame`, directions, setup and event handlers. `PipelineParams`, worker events, ending and cancelling. How `@pipedog/asyncio` gives JavaScript asyncio's cancellation. # Roadmap & status > What is ported from Pipecat, what isn't yet, and how to ask for the next port. Source: https://polvallverdu.github.io/pipedog/docs/roadmap/ pipedog ports Pipecat module by module, against a pinned upstream commit, and ports each module's tests with it. The generated status file, [`porting/STATUS.md`](https://github.com/polvallverdu/pipedog/blob/main/porting/STATUS.md), is the source of truth: it lists every package, the share of its upstream modules and tests that are ported, and the runtimes it is verified on. The [porting status](https://polvallverdu.github.io/pipedog/docs/contributing/status.md) page renders the same file. ## At a glance [#at-a-glance] Against the pinned upstream commit (counts from `porting/manifest.json`): | | Ported | Upstream | Share | | ------------------------------ | ------------------ | -------- | ---------------------- | | Modules | 622 | 667 | 93% (90% of the lines) | | Test files | 305 | 340 | 90% | | Core modules (`@pipedog/core`) | 361 | 361 | 100% | | Core test files | 150 | 150 | 100% | | Target packages | 81 started, 24 not | 105 | | pipedog publishes 87 packages: 80 of those 81 (`@pipedog/xtts`'s one module is skipped because it is deprecated upstream), plus seven with no upstream counterpart (`@pipedog/asyncio`, `@pipedog/espeak-ng`, `@pipedog/mcp`, `@pipedog/onnx`, `@pipedog/resampler-wasm`, `@pipedog/sentencex-wasm`, `@pipedog/tracing`). Ported code runs against its ported upstream tests, mock provider servers and recorded protocol vectors, and the examples run end to end against mock APIs (in headless Chromium, on Deno and on workerd). Tests that need real API keys are skipped by default, as upstream does. Testing against the real providers is the next phase; until then, expect provider-specific bugs that mocks can't catch, and please report them. ## Ported [#ported] * **Core runtime**: frames, processors, pipelines, workers and the runner, the bus, observers, RTVI, base services and transports, turn management. See [Core concepts](https://polvallverdu.github.io/pipedog/docs/concepts/frames.md). * **Local audio models and filters**: Silero VAD and Smart Turn (local v3, and `HttpSmartTurnAnalyzer` for a remote endpoint), the RNNoise noise filter and the soundfile mixer. See [Turn detection](https://polvallverdu.github.io/pipedog/docs/concepts/turn-detection.md). * **Services**: OpenAI, Google, Anthropic, Deepgram, Cartesia, ElevenLabs, Whisper and LangChain. See [Supported services](https://polvallverdu.github.io/pipedog/docs/services.md). * **Local speech models**, keyless, on ONNX Runtime: [Whisper](https://polvallverdu.github.io/pipedog/docs/services/stt/whisper.md) (`WhisperSTTService`, `WhisperSTTServiceMLX`), [Moonshine](https://polvallverdu.github.io/pipedog/docs/services/stt/moonshine.md), [FunASR SenseVoice](https://polvallverdu.github.io/pipedog/docs/services/stt/funasr.md), [Kokoro](https://polvallverdu.github.io/pipedog/docs/services/tts/kokoro.md), [Piper](https://polvallverdu.github.io/pipedog/docs/services/tts/piper.md) (also over HTTP) and [Pocket TTS](https://polvallverdu.github.io/pipedog/docs/services/tts/pocket-tts.md). * **Cloud platforms**: Azure (Azure OpenAI LLM, Realtime and image, Voice Live, Azure Speech STT and TTS) and AWS (Bedrock LLM, Transcribe STT, Polly TTS, Nova Sonic, AgentCore and the SageMaker streaming client). * **Realtime speech-to-speech**: OpenAI Realtime, Gemini Live, Azure Voice Live, AWS Nova Sonic, xAI Grok Voice Agent, Inworld Realtime and Ultravox. * **OpenAI-compatible providers**: Baseten, Cerebras, Crusoe, DeepSeek, Fireworks AI, Groq, Mistral, Nebius, Novita AI, Ollama, OpenRouter, Qwen, SambaNova, Smallest AI, Together AI and xAI (plus `@pipedog/grok`, the deprecated alias), one small package each. Groq, Mistral, Smallest AI, Together AI and xAI come with their speech-to-text and text-to-speech services. Perplexity's Agent API runs over the OpenAI Responses protocol instead. * **Speech providers**: Inworld TTS, AssemblyAI, Speechmatics, Gladia, Soniox, Rime, Fish Audio, Hume, Sarvam AI, Smallest AI, Gradium, Meta, Inception, Neuphonic, Speechify, MiniMax, Resemble AI, Camb.ai, Async and Bland, alongside the Deepgram, Cartesia, ElevenLabs, Google, Azure and AWS speech services above. See [Supported services](https://polvallverdu.github.io/pipedog/docs/services.md). * **Transports**: [WebSocket](https://polvallverdu.github.io/pipedog/docs/transports/websocket.md) (server, client, fetch handlers, and `serveFetch` for WebSocket upgrades on Bun, Node and Deno), [SmallWebRTC](https://polvallverdu.github.io/pipedog/docs/transports/smallwebrtc.md), [LiveKit](https://polvallverdu.github.io/pipedog/docs/transports/livekit.md), [WhatsApp](https://polvallverdu.github.io/pipedog/docs/transports/whatsapp.md) calling and [local audio](https://polvallverdu.github.io/pipedog/docs/transports/local-audio.md) (microphone and speakers). * **Video avatars**: [HeyGen](https://polvallverdu.github.io/pipedog/docs/services/video/heygen.md) (LiveAvatar and Interactive Avatar), as a service and as a transport. * **Serializers**: [Protobuf](https://polvallverdu.github.io/pipedog/docs/serializers/protobuf.md), [RTVI client](https://polvallverdu.github.io/pipedog/docs/serializers/rtvi-client.md), and the telephony serializers for [Twilio](https://polvallverdu.github.io/pipedog/docs/serializers/twilio.md), [Telnyx](https://polvallverdu.github.io/pipedog/docs/serializers/telnyx.md), [Plivo](https://polvallverdu.github.io/pipedog/docs/serializers/plivo.md), [Exotel](https://polvallverdu.github.io/pipedog/docs/serializers/exotel.md), [Vonage](https://polvallverdu.github.io/pipedog/docs/serializers/vonage.md) and [Genesys](https://polvallverdu.github.io/pipedog/docs/serializers/genesys.md). See [Telephony](https://polvallverdu.github.io/pipedog/docs/telephony.md). * **Runner**: the [development runner](https://polvallverdu.github.io/pipedog/docs/runner.md) (`main()`, the prebuilt client, WebRTC and telephony routes, `serve()` and `createRunnerApp()`) and `createTransport`, in `@pipedog/runner`. * **Evals**: [Pipecat's evals](https://polvallverdu.github.io/pipedog/docs/evals.md) in `@pipedog/evals`: scripted and simulated scenarios, the LLM judge, suites, and the `pipedog-eval` CLI (`pipecat eval`). * **CLI**: the [`pipedog` command](https://polvallverdu.github.io/pipedog/docs/cli.md) in `@pipedog/cli`: `pipedog init` (project scaffolding for a registry of the ported services, the config validator, coding-agent guides and the docs MCP server) and `pipedog eval`. The Context Hub's freshness check is skipped. * **Flows**: [Pipecat Flows](https://polvallverdu.github.io/pipedog/docs/flows.md) in `@pipedog/flows`: nodes, functions, actions, and YAML/JSON configs. * **Extensions**: the IVR navigator and voicemail detector in `@pipedog/extensions`. See [Telephony extensions](https://polvallverdu.github.io/pipedog/docs/telephony/extensions.md). * **MCP client**: [`@pipedog/mcp-client`](https://polvallverdu.github.io/pipedog/docs/services/mcp-client.md), which exposes an MCP server's tools to the LLM. * **Sentry**: [Sentry metrics](https://polvallverdu.github.io/pipedog/docs/concepts/sentry-metrics.md) in `@pipedog/sentry`. * **Tracing**: OpenTelemetry, in `@pipedog/tracing`. See [Metrics & tracing](https://polvallverdu.github.io/pipedog/docs/concepts/metrics-and-tracing.md). ## What's not ported yet [#whats-not-ported-yet] 45 upstream modules (10% of the lines) and 35 test files are still to do. Every upstream provider that isn't ported has a "coming soon" page, or a row on [Other upstream providers](https://polvallverdu.github.io/pipedog/docs/services/other-upstream.md). The main gaps, with their remaining upstream modules: | Gap | Modules | What it is | | ---------------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Transports | 9 | [Daily](https://polvallverdu.github.io/pipedog/docs/transports/daily.md) (with its runner routes and dial-in), the [Vonage Video Connector](https://polvallverdu.github.io/pipedog/docs/transports/vonage.md), and the Tavus, LemonSlice and MoQ transports. | | Audio filters and analyzers | 7 | Noise filters (Krisp VIVA, Koala, AIC), and the Krisp and AIC VAD and turn models. Smart Turn v3, RNNoise, the soundfile mixer and HTTP Smart Turn are ported. | | Variants of ported providers | 9 | Deepgram and NVIDIA speech services on SageMaker. The `SageMakerBidiClient` under them is ported. | | Providers with a page | 5 | NVIDIA (LLM, Riva STT and TTS), and the Simli and Tavus video avatars. | | Long-tail providers | 5 | Four smaller providers: fal, Keenable, Mem0 and Moondream, listed on [Other upstream providers](https://polvallverdu.github.io/pipedog/docs/services/other-upstream.md). XTTS is deprecated upstream, and skipped. | | Message buses | 3 | The Redis and pgmq buses (`@pipedog/bus-redis`, `@pipedog/bus-pgmq`). | | Other | 7 | The OpenClaw gateway, Strands Agents, the WebSocket worker proxy server, GStreamer. | There are no dates. Ports happen in the order people need them. ## Blocked or deferred [#blocked-or-deferred] Some upstream pieces can't be ported today, or aren't worth porting yet. Each has a reason, and a condition that would change that. "Blocked" means there is no viable JS path today. "Deferred" means it is possible, but needs vendor SDKs or keys, or waits on a moving spec. Last assessed on 2026-10-21. | Item | Status | Why | What would unblock it | Source | | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [Daily](https://polvallverdu.github.io/pipedog/docs/transports/daily.md) transport, plus [Tavus](https://polvallverdu.github.io/pipedog/docs/services/video/tavus.md) and [LemonSlice](https://polvallverdu.github.io/pipedog/docs/transports/lemonslice.md), which run on Daily rooms | Blocked | Upstream uses the native `daily-python` SDK. Daily publishes no headless or server-side JS SDK, and `@daily-co/daily-js` is browser-only. | Daily ships a Node or server SDK, or someone builds a headless media path. The REST helpers for rooms and tokens are portable now, and a pull request would be accepted. | [daily-js quickstart](https://docs.daily.co/docs/daily-js/quickstart), [LemonSlice](https://docs.pipecat.ai/server/services/transport/lemonslice), [Tavus](https://docs.pipecat.ai/api-reference/server/services/transport/tavus) | | [Vonage Video Connector](https://polvallverdu.github.io/pipedog/docs/transports/vonage.md) | Blocked | Upstream uses a native, Python-only SDK. Vonage lists "Node.js coming soon". The Vonage voice (Audio Connector) [serializer](https://polvallverdu.github.io/pipedog/docs/serializers/vonage.md) is already ported. | Vonage ships its Node SDK. | [Vonage AI connectors](https://developer.vonage.com/en/video/ai-connectors/overview) | | [MoQ](https://polvallverdu.github.io/pipedog/docs/transports/moq.md) transport | Deferred | The protocol is still changing. The JS library, `@moq/net`, falls back to WebSocket on servers and only has experimental WebTransport. | The spec and server-side JS support settle. | [MoQ JS docs](https://doc.moq.dev/js/) | | Krisp VIVA (filter, VAD, turn) | Deferred | The proprietary SDK isn't on public npm, and it needs a Krisp license and models. The hook `setKrispAudioSdk()` (in `@pipedog/core`) is ported, so you can plug in your own copy of the SDK. | A bring-your-own-SDK package is planned. | Krisp's SDK terms | | ai-coustics (AIC) filter and Quail VAD | Deferred | `@ai-coustics/aic-sdk` is native (darwin and linux), and the other build is WASM, aimed at browsers. Both need a license key. | Planned as an optional package. | `@ai-coustics/aic-sdk` on npm | | Koala (Picovoice) noise suppression | Deferred | There is no Node build, only `@picovoice/koala-web` (WASM, for browsers), and it needs an AccessKey. | It will be ported if users ask: [open a request](https://github.com/polvallverdu/pipedog/issues/new?template=feature_request.yml). | `@picovoice/koala-web` on npm | | Tk local transport, Smart Turn v2 and Smart Turn CoreML | Skipped | Skipped on purpose. Tk is a desktop GUI, and both Smart Turn variants are superseded by v3, which is ported. | Nothing planned. Open a request if you have a use for one. | | Contributions are welcome on any of these, especially the portable parts (the REST helpers for Daily, Tavus and LemonSlice) and the bring-your-own-SDK packages. Open a request with the [port request template](https://github.com/polvallverdu/pipedog/issues/new?template=feature_request.yml) first, so the work isn't duplicated. ## Ask for a port [#ask-for-a-port] 1. Look for the provider on its coming-soon page, which links to a search of the issues. 2. Upvote an existing request with a reaction, or open one with the [port request template](https://github.com/polvallverdu/pipedog/issues/new?template=feature_request.yml). 3. Or port it yourself: [Contributing](https://polvallverdu.github.io/pipedog/docs/contributing.md) explains the workflow, from `bun tools/port/next.ts` to the ported tests. # Development runner > Run a bot with WebRTC in the browser and phone calls over Twilio, Telnyx, Plivo or Exotel, with the same bot() entry point as Pipecat's runner. Source: https://polvallverdu.github.io/pipedog/docs/runner/ `@pipedog/runner` is the port of Pipecat's `pipecat.runner`: `createTransport()` picks the transport from the runner arguments, and `main()` starts a development server that serves the prebuilt client, the SmallWebRTC signalling routes, the telephony WebSocket and webhook, and `/start`. A bot written against Pipecat's runner keeps its shape: one exported `bot(runnerArgs)` function, one `main()` call. Pipecat's documentation for the development runner. `packages/runner/src` (`run.ts`, `utils.ts`, `types.ts`, `serve.ts`). ## Installation [#installation] ```bash bun add @pipedog/runner ``` It brings the SmallWebRTC and WebSocket transports and the Telnyx, Twilio, Plivo and Exotel serializers. On Node, also install `ws` for the WebSocket routes (`npm install ws`); Bun and Deno need nothing else. For `-t livekit`, add `@pipedog/transport-livekit` and `@livekit/rtc-node` (see [LiveKit](https://polvallverdu.github.io/pipedog/docs/transports/livekit.md)). For `--whatsapp`, add `@pipedog/transport-whatsapp` (see [WhatsApp](https://polvallverdu.github.io/pipedog/docs/transports/whatsapp.md)). ## A bot [#a-bot] ```ts // bot.ts import { TransportParams } from "@pipedog/core/transports/base_transport"; import { FastAPIWebsocketParams } from "@pipedog/transport-websocket/fastapi"; import { createTransport } from "@pipedog/runner/utils"; import type { RunnerArguments } from "@pipedog/runner/types"; const transportParams = { webrtc: () => new TransportParams({ audioInEnabled: true, audioOutEnabled: true }), telnyx: () => new FastAPIWebsocketParams({ audioInEnabled: true, audioOutEnabled: true }), }; export async function bot(runnerArgs: RunnerArguments) { const transport = await createTransport(runnerArgs, transportParams); // build the pipeline with transport.input() and transport.output(), then run it } if (import.meta.main) { const { main } = await import("@pipedog/runner/run"); await main(); } ``` ```bash bun bot.ts -t webrtc # open http://localhost:7860 bun bot.ts -t telnyx --proxy mybot.ngrok.io # point the Telnyx number at https://mybot.ngrok.io/ bun bot.ts -t livekit # POST /start returns a LiveKit room URL and user token bun bot.ts --whatsapp # WhatsApp calls: webhook at /whatsapp (needs the WHATSAPP_* variables) bun bot.ts # every ported transport at once ``` `main()` finds `bot` the way upstream does: on the script that was run, then in `bot.ts` (or `.js`) in the current directory, then in the first script there that exports it. Pass `main({ bot })` to skip the lookup. ## Routes [#routes] | Route | What it does | | ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `GET /` | Redirects to `/client/`. | | `GET /client/` | The prebuilt Pipecat client (see below). | | `POST /start` | Starts a session: `{"transport": "webrtc", "enableDefaultIceServers": true, "body": {...}}`. Returns `sessionId` and `iceConfig` for WebRTC, `url`, `token` and `sessionId` for LiveKit, `wsUrl` (and `token` with `--ws-auth token`) for telephony and plain WebSocket. | | `GET /status` | The transports this runner serves. | | `POST`, `PATCH /api/offer` | SmallWebRTC offers and ICE candidates; a known `pc_id` renegotiates (and `restart_pc` restarts) the existing peer connection. | | `/sessions/{id}/api/offer` | The same, for a session from `/start` (Pipecat Cloud's proxy shape); its `/start` body becomes `runnerArgs.body`. | | `POST /` | With `-t twilio`, `telnyx` or `plivo`: the TwiML or TeXML that connects the call to `wss:///ws`. With `-t exotel`: the WebSocket URL to configure. | | `GET /whatsapp` | With `--whatsapp`: Meta's webhook verification. Returns the `hub.challenge` number, or 403 for a wrong `hub.verify_token`. | | `POST /whatsapp` | With `--whatsapp`: call events. Checks `X-Hub-Signature-256` against the app secret, answers a `connect` event's SDP offer over SmallWebRTC and starts the bot with `SmallWebRTCRunnerArguments` (`runnerArgs.body` is the call: `from`, `id`, `direction`, ...). 400 for another `object` or no supported event, 422 for a malformed body, 500 for failures. | | `/ws`, `/ws/{token}` | The telephony media stream. | | `/ws-client`, `/ws-client/{token}` | A plain WebSocket session (`transportType: "websocket"`). | | `POST /files`, `POST /sessions/{id}/files` | Client uploads, with `-u/--uploads-folder` or a `createFileStorage()` export. | | `GET /files/{name}` | Downloads from `-f/--downloads-folder`. | ## Phone calls and hang-ups [#phone-calls-and-hang-ups] For a telephony connection, `createTransport()` reads the provider's first messages (Telnyx sends `connected`, then `start`), finds the provider, and builds its serializer with the stream ID and call ID. The handshake is then on the runner arguments: ```ts if (runnerArgs instanceof WebSocketRunnerArguments) { runnerArgs.transportType; // "telnyx" runnerArgs.callData?.callId; // Telnyx's call_control_id runnerArgs.callData?.streamId; runnerArgs.callData?.fromNumber; } ``` The serializer hangs up the call when the pipeline ends, with credentials it reads from the environment: | Provider | Environment variables | | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Twilio | `TWILIO_ACCOUNT_SID`, and `TWILIO_AUTH_TOKEN` or an API key (`TWILIO_API_KEY` and `TWILIO_API_SECRET`, without `TWILIO_AUTH_TOKEN`). `TWILIO_REGION` and `TWILIO_EDGE`, set together, hang up through a regional API host. | | Telnyx | `TELNYX_API_KEY` | | Plivo | `PLIVO_AUTH_ID`, `PLIVO_AUTH_TOKEN` | | Exotel | none: Exotel's serializer doesn't hang up | See [Twilio](https://polvallverdu.github.io/pipedog/docs/serializers/twilio.md) for how the credentials are checked. A caller can hang up while the call is connecting, before the provider's `start` message. That is a typed outcome rather than a stray exception: `createTransport()` throws `TelephonyCallEndedError` (a subclass of the `TelephonyHandshakeError` upstream raises), and `runnerArgs.handshake` is `{ status: "closed", messagesReceived: 0 | 1 }`. ```ts import { TelephonyCallEndedError } from "@pipedog/runner/utils"; let transport; try { transport = await createTransport(runnerArgs, transportParams); } catch (e) { if (e instanceof TelephonyCallEndedError) return; // nothing to talk to throw e; } ``` If the bot doesn't catch it, the runner logs the hang-up at info level instead of as an error. `telephonyHandshake(websocket)` gives the same outcome without creating a transport. ## Startup hook [#startup-hook] Upstream composes FastAPI lifespans. Here a lifespan is a function that runs before the server listens and may return a cleanup: ```ts await main({ lifespan: async () => { await warmUpClients(); return async () => await closeClients(); }, }); ``` ## Mounting the runner in your own server [#mounting-the-runner-in-your-own-server] The runner is a fetch handler. `createRunnerApp()` builds it from the same options as the CLI flags; `serve()` listens on Bun, Node or Deno. ```ts import { createRunnerApp } from "@pipedog/runner/run"; import { serve } from "@pipedog/runner/serve"; const runner = await createRunnerApp({ bot, transport: "telnyx", proxy: "mybot.ngrok.io" }); await serve(runner, { port: 7860 }); ``` In Hono, SvelteKit or anything else built on `Request` and `Response`, call `runner.startup()` once and route requests to `runner.fetch`. `runner.fetch` also upgrades the WebSocket routes: on its own on Deno (and Cloudflare Workers, untested), and with the upgrading server as its second argument on Bun (`runner.fetch(request, server)`, served with `bunWebSocket` from `@pipedog/transport-websocket/fetch`) and on Node (through `serveFetch`, which `serve()` uses). A framework that upgrades sockets itself can match the request with `runner.matchWebSocket(request)`, upgrade it, and run the returned function with the connection (for example a `WebsocketConnection` from `honoWebsocket`). ```ts const runner = await createRunnerApp({ bot }); await runner.startup(); app.all("*", (c) => runner.fetch(c.req.raw)); // Hono ``` Upstream lets you add routes to the runner's FastAPI `app`; the module-level `app` here works the same way: `app.get("/health", () => ({ ok: true }))` before `main()`. ## The prebuilt client [#the-prebuilt-client] Upstream mounts `PipecatPrebuiltUI` from the `pipecat-ai-prebuilt` Python package, the React client from [pipecat-ai/pipecat-prebuilt](https://github.com/pipecat-ai/pipecat-prebuilt) (formerly `small-webrtc-prebuilt`). Its build is only published inside that Python wheel: there is no npm package or CDN build of it (`@pipecat-ai/voice-ui-kit` is the component library it is made from, not a built app). `@pipedog/runner` ships the wheel's built client unmodified (release 1.3.0, BSD-2-Clause, in `prebuilt/dist`), so `/client` works offline once installed. Pass `createRunnerApp({ prebuilt: false })` to leave it out, or a directory or reader function to serve another build. ## Coming from Pipecat's runner [#coming-from-pipecats-runner] | Python | TypeScript | | ------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- | | `from pipecat.runner.run import main` | `import { main } from "@pipedog/runner/run"` | | `if __name__ == "__main__": main()` | `if (import.meta.main) await main();` | | `async def bot(runner_args: RunnerArguments)` | `export async function bot(runnerArgs: RunnerArguments)` | | `from pipecat.runner.utils import create_transport` | `import { createTransport } from "@pipedog/runner/utils"` | | `await create_transport(runner_args, transport_params)` | `await createTransport(runnerArgs, transportParams)` | | `"webrtc": lambda: TransportParams(audio_in_enabled=True)` | `webrtc: () => new TransportParams({ audioInEnabled: true })` | | `"telnyx": lambda: FastAPIWebsocketParams(...)` | `telnyx: () => new FastAPIWebsocketParams({ ... })` | | `SmallWebRTCRunnerArguments`, `WebSocketRunnerArguments`, `LiveKitRunnerArguments` | same names, from `@pipedog/runner/types` | | `DailyRunnerArguments`, `VonageRunnerArguments`, `MOQRunnerArguments` | same names (types only; the transports are not ported) | | `from pipecat.runner.livekit import configure, generate_session_tokens` | `import { configure, generateSessionTokens } from "@pipedog/runner/livekit"` (async) | | `runner_args.call_data.call_id` / `call_data["call_id"]` | `runnerArgs.callData?.callId` / `callData.getItem("call_id")` | | `runner_args.transport_type` | `runnerArgs.transportType` | | `runner_args.body`, `session_id`, `handle_sigint`, `cli_args` | `body`, `sessionId`, `handleSigint`, `cliArgs` | | `parse_telephony_websocket(websocket)` | `parseTelephonyWebsocket(websocket)` | | `ValueError` when the call closes before the handshake | `TelephonyCallEndedError`, and `runnerArgs.handshake.status === "closed"` | | `main(parser)` with an `argparse.ArgumentParser` | `main({ parser })` with `ArgumentParser` from `@pipedog/runner/run`; values in `runnerArgs.cliArgs` | | `from pipecat.runner.run import app; @app.get(...)` | `import { app } from "@pipedog/runner/run"; app.get(...)` | | FastAPI lifespan | `main({ lifespan })` / `app.addLifespan()` | | `uvicorn.run(app)` | `serve(app)` from `@pipedog/runner/serve`, or `app.fetch` in your server | | `python bot.py -t webrtc` | `bun bot.ts -t webrtc` | | `-t twilio -x mybot.ngrok.io` | the same flags | | `--host`, `--port`, `--esp32`, `--ice-servers`, `--ws-auth`, `--allowed-origins`, `-u`, `-f`, `--runner-body`, `-v` | the same flags | | `-t livekit` | the same: `/start` mints tokens from `LIVEKIT_URL`, `LIVEKIT_API_KEY`, `LIVEKIT_API_SECRET` (and `LIVEKIT_ROOM_NAME`) | | `--whatsapp` | the same: mounts `/whatsapp` (below) | | `-t daily`, `-d`, `-t moq`, `-t vonage` | recognized; fail with "not ported to pipedog yet" | | `.env` loaded with `load_dotenv(override=True)` | the same | ## A migrated bot: bot.py to bot.ts [#a-migrated-bot-botpy-to-botts] A bot that talks over WebRTC during development and answers Telnyx calls in production. ```python # bot.py from loguru import logger from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.worker import PipelineParams, PipelineWorker from pipecat.processors.aggregators.llm_context import LLMContext from pipecat.processors.aggregators.llm_response_universal import LLMContextAggregatorPair from pipecat.runner.types import RunnerArguments, WebSocketRunnerArguments from pipecat.runner.utils import create_transport from pipecat.services.openai.llm import OpenAILLMService from pipecat.services.openai.stt import OpenAISTTService from pipecat.services.openai.tts import OpenAITTSService from pipecat.transports.base_transport import TransportParams from pipecat.transports.websocket.fastapi import FastAPIWebsocketParams from pipecat.workers.runner import WorkerRunner transport_params = { "webrtc": lambda: TransportParams(audio_in_enabled=True, audio_out_enabled=True), "telnyx": lambda: FastAPIWebsocketParams(audio_in_enabled=True, audio_out_enabled=True), } async def bot(runner_args: RunnerArguments): transport = await create_transport(runner_args, transport_params) if isinstance(runner_args, WebSocketRunnerArguments): logger.info(f"Call from {runner_args.call_data.from_number}") stt = OpenAISTTService() llm = OpenAILLMService() tts = OpenAITTSService() context = LLMContext() aggregators = LLMContextAggregatorPair(context) pipeline = Pipeline([ transport.input(), stt, aggregators.user(), llm, tts, transport.output(), aggregators.assistant(), ]) worker = PipelineWorker(pipeline, params=PipelineParams(enable_metrics=True)) @transport.event_handler("on_client_disconnected") async def on_client_disconnected(transport, client): await worker.cancel() runner = WorkerRunner(handle_sigint=runner_args.handle_sigint) await runner.add_workers(worker) await runner.run() if __name__ == "__main__": from pipecat.runner.run import main main() ``` ```ts // bot.ts import { Pipeline } from "@pipedog/core/pipeline/pipeline"; import { PipelineParams, PipelineWorker } from "@pipedog/core/pipeline/worker"; import { LLMContext } from "@pipedog/core/processors/aggregators/llm_context"; import { LLMContextAggregatorPair } from "@pipedog/core/processors/aggregators/llm_response_universal"; import { TransportParams } from "@pipedog/core/transports/base_transport"; import { logger } from "@pipedog/core/utils/logger"; import { WorkerRunner } from "@pipedog/core/workers/runner"; import { OpenAILLMService } from "@pipedog/openai/llm"; import { OpenAISTTService } from "@pipedog/openai/stt"; import { OpenAITTSService } from "@pipedog/openai/tts"; import { type RunnerArguments, WebSocketRunnerArguments } from "@pipedog/runner/types"; import { createTransport, TelephonyCallEndedError } from "@pipedog/runner/utils"; import { FastAPIWebsocketParams } from "@pipedog/transport-websocket/fastapi"; const transportParams = { webrtc: () => new TransportParams({ audioInEnabled: true, audioOutEnabled: true }), telnyx: () => new FastAPIWebsocketParams({ audioInEnabled: true, audioOutEnabled: true }), }; export async function bot(runnerArgs: RunnerArguments) { let transport; try { transport = await createTransport(runnerArgs, transportParams); } catch (e) { if (e instanceof TelephonyCallEndedError) return; // hung up while connecting throw e; } if (runnerArgs instanceof WebSocketRunnerArguments) { logger.info(`Call from ${runnerArgs.callData?.fromNumber}`); } const stt = new OpenAISTTService(); const llm = new OpenAILLMService(); const tts = new OpenAITTSService(); const context = new LLMContext(); const aggregators = new LLMContextAggregatorPair(context); const pipeline = new Pipeline([ transport.input(), stt, aggregators.user(), llm, tts, transport.output(), aggregators.assistant(), ]); const worker = new PipelineWorker(pipeline, { params: new PipelineParams({ enableMetrics: true }), }); transport.eventHandler("on_client_disconnected", async () => { await worker.cancel(); }); const runner = new WorkerRunner({ handleSigint: runnerArgs.handleSigint }); await runner.addWorkers(worker); await runner.run(); } if (import.meta.main) { const { main } = await import("@pipedog/runner/run"); await main(); } ``` ```bash bun bot.ts -t webrtc # development, in the browser bun bot.ts -t telnyx --proxy mybot.ngrok.io # phone calls through an ngrok tunnel ``` ## WhatsApp [#whatsapp] `--whatsapp` mounts the webhook for WhatsApp Business calls at `/whatsapp`. It is a flag, not a `-t` transport: combine it with whatever else you serve. The bot runs over SmallWebRTC, so `createTransport()` builds a `SmallWebRTCTransport` from the `SmallWebRTCRunnerArguments` the webhook passes in (there is no WhatsApp runner argument type, as upstream). `runnerArgs.body` is the call's metadata from Meta (`from`, `id`, `direction`, `timestamp`). ```bash bun add @pipedog/transport-whatsapp # an optional peer of the runner export WHATSAPP_APP_SECRET=... # Meta app secret, to check X-Hub-Signature-256 export WHATSAPP_PHONE_NUMBER_ID=... # the business phone number ID export WHATSAPP_TOKEN=... # Graph API access token export WHATSAPP_WEBHOOK_VERIFICATION_TOKEN=... # the verify token you set in the Meta dashboard bun bot.ts -t webrtc --whatsapp ``` Point the webhook of your Meta app (subscribed to `calls`) at `https:///whatsapp`. If a variable is missing, or `@pipedog/transport-whatsapp` is not installed, the runner logs an error and mounts no WhatsApp routes. On shutdown it ends the calls that are still open. See [WhatsApp](https://polvallverdu.github.io/pipedog/docs/transports/whatsapp.md) for the transport. ## Not ported [#not-ported] * Daily (`-t daily`, `-d/--direct`, `/daily`, the dial-in webhook), MOQ and the Vonage Video Connector (`-t vonage`): their transports are not ported. The argument classes exist for type parity; selecting these fails with a clear message. * `-t eval`: deprecated upstream, so not ported. The evals themselves are ported, in [`@pipedog/evals`](https://polvallverdu.github.io/pipedog/docs/evals.md). * The MOQ CLI flags and the deprecated `--folder` flag. # Exotel > Phone calls over Exotel's Voicebot stream with ExotelFrameSerializer, used with FastAPIWebsocketTransport. Source: https://polvallverdu.github.io/pipedog/docs/serializers/exotel/ ## Overview [#overview] **`ExotelFrameSerializer`** speaks the WebSocket protocol of Exotel's Stream and Voicebot applets. Paired with [`FastAPIWebsocketTransport`](https://polvallverdu.github.io/pipedog/docs/transports/websocket.md), it lets a bot answer phone calls: * Caller audio (16-bit PCM at 8 kHz, base64 in `media` events) becomes `InputAudioRawFrame`s at the pipeline's input rate. * Bot audio is resampled to 8 kHz and sent as `media` events. * Touch-tone digits become `InputDTMFFrame`s. * An `InterruptionFrame` sends `clear`. There is no hang-up REST call: the call ends when the stream closes. See [How phone calls connect](https://polvallverdu.github.io/pipedog/docs/telephony.md) for the overall flow. Pipecat's documentation for the same serializer. `packages/serializer-exotel/src/exotel.ts`. The Stream and Voicebot applets. ## Installation [#installation] ```bash bun add @pipedog/serializer-exotel @pipedog/transport-websocket @pipedog/runner ``` ```ts import { ExotelFrameSerializer } from "@pipedog/serializer-exotel/exotel"; ``` ## Prerequisites [#prerequisites] * An Exotel account with a call flow whose Voicebot applet points at your WebSocket URL. * The stream SID (and call SID), which Exotel sends in its `start` message. ## ExotelFrameSerializer [#exotelframeserializer] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `streamSid` | `string` | yes | The Exotel Media Stream SID. | | `callSid` | `string \| null \| undefined` | no | The associated Exotel Call SID (optional, not used in this implementation). | | `params` | `ExotelFrameSerializerInputParams \| null \| undefined` | no | Configuration parameters. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | ### InputParams [#inputparams] `params` is an `ExotelFrameSerializer.InputParams` (`ExotelFrameSerializerInputParams`): | Param | Type | Default | Description | | ------------------------- | ---------------- | ------- | ---------------------------------------------------------------------------------------------- | | `exotelSampleRate` | `number` | `8000` | Sample rate Exotel uses, in Hz. | | `sampleRate` | `number \| null` | `null` | Override for the pipeline input rate. `null` uses the pipeline's. | | `resamplerClearAfterSecs` | `number \| null` | `0.2` | Seconds of silence after which the stream resamplers clear their history. `null` never clears. | | `ignoreRtviMessages` | `boolean` | `true` | Drop RTVI messages instead of sending them to Exotel. | ### Usage [#usage] ```ts title="exotel.ts" import { parseTelephonyWebsocket } from "@pipedog/runner/utils"; import { ExotelFrameSerializer } from "@pipedog/serializer-exotel/exotel"; import { FastAPIWebsocketParams, FastAPIWebsocketTransport, } from "@pipedog/transport-websocket/fastapi"; import { websocketFetchHandler } from "@pipedog/transport-websocket/fetch"; import type { WebsocketConnection } from "@pipedog/transport-websocket/socket"; async function runCall(connection: WebsocketConnection): Promise { // Exotel sends "connected", then "start" with the stream and call SIDs. const [, callData] = await parseTelephonyWebsocket(connection); const serializer = new ExotelFrameSerializer({ streamSid: callData.streamId ?? "", callSid: callData.callId, }); const transport = new FastAPIWebsocketTransport( connection, new FastAPIWebsocketParams({ audioInEnabled: true, audioOutEnabled: true, addWavHeader: false, serializer, }), ); // ... build and run the pipeline with transport.input() and transport.output() } // The Voicebot applet's WebSocket URL points here. const handler = websocketFetchHandler(runCall); ``` ### Notes [#notes] * `callSid` is stored but not used: Exotel has no hang-up call here. * The audio is linear PCM, not μ-law: only resampled, never companded. * An invalid DTMF digit is logged at info level and ignored. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/serializer-exotel/package.json`: ✅ means the compat smoke test passes on that runtime. Runs on Bun, Node, Deno and Cloudflare Workers. The package's golden tests replay vectors recorded from upstream: at 8 kHz the wire output is byte-identical. # Genesys > Genesys Cloud AudioHook sessions with GenesysAudioHookSerializer, used with FastAPIWebsocketTransport. Source: https://polvallverdu.github.io/pipedog/docs/serializers/genesys/ ## Overview [#overview] **`GenesysAudioHookSerializer`** speaks Genesys Cloud's AudioHook protocol, so a contact-center call flow can hand a caller to a bot and get results back: * Binary μ-law audio at 8 kHz both ways, resampled to and from the pipeline's rate. In stereo (`external` and `internal`), only the caller's (left) channel reaches the pipeline. * The handshake is answered for you: `open` → `opened`, `ping` → `pong`, `close` → `closed`, each returned from `deserialize` as an `OutputTransportMessageUrgentFrame` the transport sends back. * `pause` stops audio until you send `createResumedResponse()`; `update` refreshes the participant; `dtmf` becomes an `InputDTMFFrame`. * Architect variables in (`inputVariables`, from `open`) and out (`setOutputVariables()`, sent in `closed` and `disconnect`). * An `InterruptionFrame` sends a barge-in event; an `EndFrame` or `CancelFrame` sends `disconnect`. Pipecat's documentation for the same serializer. `packages/serializer-genesys/src/genesys.ts`. The AudioHook protocol. ## Installation [#installation] ```bash bun add @pipedog/serializer-genesys @pipedog/transport-websocket ``` ```ts import { AudioHookChannel, GenesysAudioHookSerializer, } from "@pipedog/serializer-genesys/genesys"; ``` ## Prerequisites [#prerequisites] * A Genesys Cloud org with an AudioHook Monitor or AudioConnector integration whose connection URI is your WebSocket URL, and an Architect flow that uses it. ## GenesysAudioHookSerializer [#genesysaudiohookserializer] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `params` | `GenesysAudioHookSerializerInputParams \| null \| undefined` | no | Configuration parameters. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | ### InputParams [#inputparams] `params` is a `GenesysAudioHookSerializer.InputParams` (`GenesysAudioHookSerializerInputParams`): | Param | Type | Default | Description | | ------------------------- | ---------------------- | ---------- | ------------------------------------------------------------------------------------------------------------------- | | `genesysSampleRate` | `number` | `8000` | Sample rate Genesys uses, in Hz. | | `sampleRate` | `number \| null` | `null` | Override for the pipeline input rate. `null` uses the pipeline's. | | `channel` | `AudioHookChannel` | `EXTERNAL` | Channels to offer: `EXTERNAL`, `INTERNAL` or `BOTH`. Updated from Genesys's `open`. | | `mediaFormat` | `AudioHookMediaFormat` | `PCMU` | Audio format. Only `PCMU` is implemented, as upstream. | | `processExternal` | `boolean` | `true` | Process the caller's audio. | | `processInternal` | `boolean` | `false` | Process the agent's audio. | | `supportedLanguages` | `string[] \| null` | `null` | Languages sent in `opened`. | | `selectedLanguage` | `string \| null` | `null` | Language selected in `opened`. | | `startPaused` | `boolean` | `false` | Ask Genesys to start the session paused. | | `resamplerClearAfterSecs` | `number \| null` | `0.2` | Seconds of silence after which the resamplers clear their history. `null` (never) suits Genesys's irregular chunks. | | `ignoreRtviMessages` | `boolean` | `true` | Drop RTVI messages. | ### Usage [#usage] ```ts title="genesys.ts" import { AudioHookChannel, GenesysAudioHookSerializer } from "@pipedog/serializer-genesys/genesys"; import { FastAPIWebsocketParams, FastAPIWebsocketTransport, } from "@pipedog/transport-websocket/fastapi"; import { websocketFetchHandler } from "@pipedog/transport-websocket/fetch"; import type { WebsocketConnection } from "@pipedog/transport-websocket/socket"; async function runSession(connection: WebsocketConnection): Promise { const serializer = new GenesysAudioHookSerializer({ params: new GenesysAudioHookSerializer.InputParams({ channel: AudioHookChannel.EXTERNAL, supportedLanguages: ["en-US", "es-ES"], selectedLanguage: "en-US", }), }); const transport = new FastAPIWebsocketTransport( connection, new FastAPIWebsocketParams({ audioInEnabled: true, audioOutEnabled: true, serializer, // Genesys rate-limits (HTTP 429) many small audio messages. fixedAudioPacketSize: 1600, }), ); // ... build and run the pipeline with transport.input() and transport.output() } // The AudioHook integration's connection URI points here. const handler = websocketFetchHandler(runSession); ``` Read the caller's details and return variables to the flow: ```ts const serializer = new GenesysAudioHookSerializer(); serializer.eventHandler("on_open", async () => { // From Genesys's "open" message. const ani = serializer.participant?.ani; const queue = serializer.inputVariables?.queue_name; console.log(`call from ${ani} for ${queue}`); }); // Returned to the Architect flow in "closed" (and "disconnect"). serializer.setOutputVariables({ intent: "billing", resolved: true }); ``` ### Events [#events] | Event | When | | ----------- | ----------------------------------------------------- | | `on_open` | Genesys opened the session (before `opened` is sent). | | `on_close` | Genesys closed the session. | | `on_ping` | A `ping` arrived. | | `on_pause` | Genesys paused the stream (e.g. on hold). | | `on_update` | An `update` arrived. | | `on_error` | Genesys reported an error. | | `on_dtmf` | A digit was pressed. | ### Notes [#notes] * Audio is dropped until the session is open, and while it is paused. * Only transport messages that are AudioHook messages (with a `version` field) are sent; others are dropped. * Positions are ISO 8601 durations with millisecond precision (`PT12.500S`), echoed from the last client message. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/serializer-genesys/package.json`: ✅ means the compat smoke test passes on that runtime. Runs on Bun, Node, Deno and Cloudflare Workers. The package's golden tests replay AudioHook sessions recorded from upstream, message by message. # Serializers > Convert frames to and from a wire format, for WebSocket and telephony transports. Source: https://polvallverdu.github.io/pipedog/docs/serializers/ {/* Generated by scripts/generate-providers.ts from data/providers.ts. */} A serializer converts frames to and from a wire format, so a WebSocket transport can speak a client's or a telephony provider's protocol. | Provider | Classes | Package | Status | | -------------------------------------------- | ---------------------------- | --------------------------------- | ------ | | [Protobuf](https://polvallverdu.github.io/pipedog/docs/serializers/protobuf.md) | `ProtobufFrameSerializer` | `@pipedog/serializer-protobuf` | Ported | | [RTVI client](https://polvallverdu.github.io/pipedog/docs/serializers/rtvi-client.md) | `RTVIClientSerializer` | `@pipedog/serializer-rtvi-client` | Ported | | [Telnyx](https://polvallverdu.github.io/pipedog/docs/serializers/telnyx.md) | `TelnyxFrameSerializer` | `@pipedog/serializer-telnyx` | Ported | | [Twilio](https://polvallverdu.github.io/pipedog/docs/serializers/twilio.md) | `TwilioFrameSerializer` | `@pipedog/serializer-twilio` | Ported | | [Plivo](https://polvallverdu.github.io/pipedog/docs/serializers/plivo.md) | `PlivoFrameSerializer` | `@pipedog/serializer-plivo` | Ported | | [Exotel](https://polvallverdu.github.io/pipedog/docs/serializers/exotel.md) | `ExotelFrameSerializer` | `@pipedog/serializer-exotel` | Ported | | [Vonage](https://polvallverdu.github.io/pipedog/docs/serializers/vonage.md) | `VonageFrameSerializer` | `@pipedog/serializer-vonage` | Ported | | [Genesys](https://polvallverdu.github.io/pipedog/docs/serializers/genesys.md) | `GenesysAudioHookSerializer` | `@pipedog/serializer-genesys` | Ported | Pipecat's [serializers reference](https://docs.pipecat.ai/api-reference/server/services/serializers) lists every upstream provider. Those not in this table are on [Other upstream providers](https://polvallverdu.github.io/pipedog/docs/services/other-upstream.md). # Plivo > Phone calls over Plivo audio streaming with PlivoFrameSerializer, used with FastAPIWebsocketTransport. Source: https://polvallverdu.github.io/pipedog/docs/serializers/plivo/ ## Overview [#overview] **`PlivoFrameSerializer`** speaks Plivo's audio streaming WebSocket protocol. Paired with [`FastAPIWebsocketTransport`](https://polvallverdu.github.io/pipedog/docs/transports/websocket.md), it lets a bot answer phone calls: * Caller audio (`media` events, μ-law at 8 kHz) becomes `InputAudioRawFrame`s at the pipeline's input rate. * Bot audio is resampled to 8 kHz, μ-law encoded and sent as `playAudio` events. * Touch-tone digits become `InputDTMFFrame`s. * An `InterruptionFrame` sends `clearAudio`. * An `EndFrame` or `CancelFrame` hangs up the call through Plivo's REST API (`autoHangUp`). See [How phone calls connect](https://polvallverdu.github.io/pipedog/docs/telephony.md) for the overall flow. Pipecat's documentation for the same serializer. `packages/serializer-plivo/src/plivo.ts`. The Stream XML element and audio streaming. ## Installation [#installation] ```bash bun add @pipedog/serializer-plivo @pipedog/transport-websocket @pipedog/runner ``` ```ts import { PlivoFrameSerializer } from "@pipedog/serializer-plivo/plivo"; ``` ## Prerequisites [#prerequisites] * A Plivo account and a number whose answer XML has a bidirectional `wss://your.host/ws`. * Your auth ID and auth token, for auto hang-up. The examples read them from `PLIVO_AUTH_ID` and `PLIVO_AUTH_TOKEN`. * The stream ID and call ID, which Plivo sends in its `start` message. ## PlivoFrameSerializer [#plivoframeserializer] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `streamId` | `string` | yes | The Plivo Stream ID. | | `callId` | `string \| null \| undefined` | no | The associated Plivo Call ID (optional, but required for auto hang-up). | | `authId` | `string \| null \| undefined` | no | Plivo auth ID (required for auto hang-up). | | `authToken` | `string \| null \| undefined` | no | Plivo auth token (required for auto hang-up). | | `params` | `PlivoFrameSerializerInputParams \| null \| undefined` | no | Configuration parameters. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | ### InputParams [#inputparams] `params` is a `PlivoFrameSerializer.InputParams` (`PlivoFrameSerializerInputParams`): | Param | Type | Default | Description | | ------------------------- | ---------------- | ------- | ---------------------------------------------------------------------------------------------- | | `plivoSampleRate` | `number` | `8000` | Sample rate Plivo uses, in Hz. | | `sampleRate` | `number \| null` | `null` | Override for the pipeline input rate. `null` uses the pipeline's. | | `autoHangUp` | `boolean` | `true` | Hang up the call on `EndFrame` or `CancelFrame`. | | `resamplerClearAfterSecs` | `number \| null` | `0.2` | Seconds of silence after which the stream resamplers clear their history. `null` never clears. | | `ignoreRtviMessages` | `boolean` | `true` | Drop RTVI messages instead of sending them to Plivo. | ### Usage [#usage] ```ts title="plivo.ts" import { parseTelephonyWebsocket } from "@pipedog/runner/utils"; import { PlivoFrameSerializer } from "@pipedog/serializer-plivo/plivo"; import { FastAPIWebsocketParams, FastAPIWebsocketTransport, } from "@pipedog/transport-websocket/fastapi"; import { websocketFetchHandler } from "@pipedog/transport-websocket/fetch"; import type { WebsocketConnection } from "@pipedog/transport-websocket/socket"; async function runCall(connection: WebsocketConnection): Promise { // Plivo's "start" message carries the stream and call IDs. const [, callData] = await parseTelephonyWebsocket(connection); const serializer = new PlivoFrameSerializer({ streamId: callData.streamId ?? "", callId: callData.callId, authId: process.env.PLIVO_AUTH_ID, authToken: process.env.PLIVO_AUTH_TOKEN, }); const transport = new FastAPIWebsocketTransport( connection, new FastAPIWebsocketParams({ audioInEnabled: true, audioOutEnabled: true, addWavHeader: false, serializer, }), ); // ... build and run the pipeline with transport.input() and transport.output() } // The in your Plivo XML points here. const handler = websocketFetchHandler(runCall); ``` Without auto hang-up, only the stream ID is needed: ```ts const noHangUp = new PlivoFrameSerializer({ streamId: "stream-id", params: new PlivoFrameSerializer.InputParams({ autoHangUp: false }), }); ``` ### Notes [#notes] * With `autoHangUp` on (the default), the constructor throws if `callId`, `authId` or `authToken` is missing. The hang-up is a `DELETE` of `https://api.plivo.com/v1/Account/{authId}/Call/{callId}/` with Basic auth, attempted once. 204 and 404 (already ended) are logged at debug level; other failures are logged as errors and don't throw. * Outbound audio is `playAudio` with `contentType: "audio/x-mulaw"` and the Plivo sample rate. * Set `addWavHeader: false` on the transport params. * An invalid DTMF digit is logged as a warning and ignored. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/serializer-plivo/package.json`: ✅ means the compat smoke test passes on that runtime. Runs on Bun, Node, Deno and Cloudflare Workers (`fetch` for the hang-up, core's WASM resampler for the audio). The package's golden tests replay vectors recorded from upstream: at 8 kHz the wire output is byte-identical. # Protobuf > Binary frame serialization for WebSocket transports with ProtobufFrameSerializer, wire-compatible with Pipecat's frames.proto. Source: https://polvallverdu.github.io/pipedog/docs/serializers/protobuf/ ## Overview [#overview] A serializer turns frames into the messages a transport sends, and incoming messages back into frames. **`ProtobufFrameSerializer`** encodes frames as Protocol Buffers with Pipecat's `frames.proto` schema, so a pipedog bot talks to Pipecat's WebSocket clients and to Pipecat bots over the same wire format. It is the default serializer of `WebsocketClientTransport`, and the usual choice for `FastAPIWebsocketTransport` and `SingleClientWebsocketServerTransport` when the other end is a Pipecat client. Pipecat's serializer overview. Upstream has no page for this class. `packages/serializer-protobuf/src/protobuf.ts`, with the schema codec in `packages/core/src/frames/protobufs/frames_pb2.ts`. The transports this serializer is used with. ## Installation [#installation] ```bash bun add @pipedog/serializer-protobuf ``` ```ts import { ProtobufFrameSerializer } from "@pipedog/serializer-protobuf/protobuf"; ``` `@pipedog/transport-websocket` already depends on it. ## Prerequisites [#prerequisites] None. No account or key is needed. The other end must speak the same schema: a Pipecat client SDK's WebSocket transport with its protobuf serializer, or another Pipecat or pipedog bot. ## ProtobufFrameSerializer [#protobufframeserializer] ### Configuration [#configuration] `new ProtobufFrameSerializer(options?)`. The options are the base serializer's: | Option | Type | Required | Description | |---|---|---|---| | `params` | `FrameSerializerInputParams \| null \| undefined` | no | Configuration parameters. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | ### Usage [#usage] ```ts title="transport.ts" import { TextFrame } from "@pipedog/core/frames/frames"; import { ProtobufFrameSerializer } from "@pipedog/serializer-protobuf/protobuf"; import { FastAPIWebsocketParams, FastAPIWebsocketTransport, } from "@pipedog/transport-websocket/fastapi"; import type { WebsocketConnection } from "@pipedog/transport-websocket/socket"; function createTransport(connection: WebsocketConnection): FastAPIWebsocketTransport { return new FastAPIWebsocketTransport( connection, new FastAPIWebsocketParams({ audioInEnabled: true, audioOutEnabled: true, serializer: new ProtobufFrameSerializer(), }), ); } ``` The transports call `serialize()` and `deserialize()` for you. Called directly: ```ts // The transports call these for you; this shows the wire form of one frame. const serializer = new ProtobufFrameSerializer(); const bytes = await serializer.serialize(new TextFrame({ text: "hello" })); const frame = bytes ? await serializer.deserialize(bytes) : null; ``` ### Notes [#notes] The schema has five frame types: | Proto field | Serialized from | Deserialized to | | --------------- | ------------------------------------------------------------------ | ---------------------------- | | `text` | `TextFrame` | `TextFrame` | | `audio` | `OutputAudioRawFrame` | `InputAudioRawFrame` | | `transcription` | `TranscriptionFrame` | `TranscriptionFrame` | | `message` | `OutputTransportMessageFrame`, `OutputTransportMessageUrgentFrame` | `InputTransportMessageFrame` | | `interruption` | `InterruptionFrame` | `InterruptionFrame` | * Transport messages go on the wire as a `message` frame whose `data` is the JSON-encoded message. On the way in, `data` is parsed back to an object; invalid JSON logs an error and returns `null`. * Any other frame type logs a warning and serializes to `null`, which the transport skips. * RTVI messages are always sent. The base serializer drops them by default to keep them off telephony streams, but this constructor sets `ignoreRtviMessages` to `false`, even if you pass params: WebSocket transports are how RTVI messages reach the client. * `deserialize()` takes bytes. A text message throws a `TypeError`, as protobuf's `FromString` does upstream. * The frame's `id`, `name` and `pts` (for audio) are carried across when set. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/serializer-protobuf/package.json`: ✅ means the compat smoke test passes on that runtime. Upstream's `frames_pb2` is generated by `protoc` and needs the `protobuf` runtime. pipedog's core has no third-party dependencies, so `frames_pb2.ts` is a hand-written proto3 codec for the same schema, with the same wire format. `uint64` fields are JS numbers, exact up to 2^53. The serializer runs on Bun, Node, Deno and Cloudflare Workers. The other serializers are ported too: [RTVI client](https://polvallverdu.github.io/pipedog/docs/serializers/rtvi-client.md), and the telephony serializers for [Twilio](https://polvallverdu.github.io/pipedog/docs/serializers/twilio.md), [Telnyx](https://polvallverdu.github.io/pipedog/docs/serializers/telnyx.md), [Plivo](https://polvallverdu.github.io/pipedog/docs/serializers/plivo.md), [Exotel](https://polvallverdu.github.io/pipedog/docs/serializers/exotel.md), [Vonage](https://polvallverdu.github.io/pipedog/docs/serializers/vonage.md) and [Genesys](https://polvallverdu.github.io/pipedog/docs/serializers/genesys.md). # RTVI client > RTVIClientSerializer lets a pipedog pipeline act as an RTVI client, turning a bot's RTVI server messages into frames. Source: https://polvallverdu.github.io/pipedog/docs/serializers/rtvi-client/ ## Overview [#overview] Bots usually speak RTVI as the server: `RTVIObserver` turns their pipeline frames into RTVI server messages for a browser client. **`RTVIClientSerializer`** is the inverse. It lets a pipedog pipeline sit on the client side of that connection: * Incoming RTVI server messages from the bot (`bot-llm-text`, `user-transcription`, `bot-started-speaking` and so on) become pipeline frames. * Outgoing audio becomes RTVI `raw-audio` messages, and RTVI client messages you build are sent as they are. It is the default serializer of [`RTVIClientTransport`](https://polvallverdu.github.io/pipedog/docs/transports/websocket/#rtviclienttransport), which pairs it with a WebSocket client and the `client-ready` / `bot-ready` handshake. The two are the base for driving a bot from a scripted user, as eval harnesses do. `pipecat/serializers/rtvi_client.py`. Upstream has no docs page for this class. `packages/serializer-rtvi-client/src/rtvi_client.ts`. The RTVI message standard. ## Installation [#installation] ```bash bun add @pipedog/serializer-rtvi-client ``` ```ts import { RTVIClientSerializer } from "@pipedog/serializer-rtvi-client/rtvi_client"; ``` `@pipedog/transport-websocket` already depends on it. ## Prerequisites [#prerequisites] A bot that runs an RTVI server transport and is reachable over a WebSocket. ## RTVIClientSerializer [#rtviclientserializer] ### Configuration [#configuration] `new RTVIClientSerializer(options?)`: | Option | Type | Required | Description | |---|---|---|---| | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | There are no serializer params: the constructor always sets `ignoreRtviMessages` to `false`, since RTVI messages are what it sends. ### Usage [#usage] ```ts title="client.ts" import { OutputTransportMessageFrame } from "@pipedog/core/frames/frames"; import { RTVIClientSerializer } from "@pipedog/serializer-rtvi-client/rtvi_client"; import { WebsocketClientParams, WebsocketClientTransport, } from "@pipedog/transport-websocket/client"; // RTVIClientTransport uses this serializer by default. With the plain client // transport, pass it yourself and turn off WAV headers. const transport = new WebsocketClientTransport( "ws://localhost:7860", new WebsocketClientParams({ audioInEnabled: true, audioOutEnabled: true, addWavHeader: false, serializer: new RTVIClientSerializer(), }), ); ``` Send an RTVI client message by pushing a transport message frame toward the output: ```ts // An RTVI client message, built as a transport message frame. The serializer // sends any message labelled "rtvi-ai" as it is. const sendText = new OutputTransportMessageFrame({ message: { label: "rtvi-ai", type: "send-text", id: "1", data: { content: "Hello, bot" }, }, }); ``` ### Notes [#notes] Incoming messages map to frames as follows. Anything else, `bot-ready` and metrics included, returns `null`. | RTVI message | Frame | | ------------------------------- | ------------------------------------------------------------------------------- | | `bot-llm-started` | `LLMFullResponseStartFrame` | | `bot-llm-text` | `LLMTextFrame` | | `bot-llm-marker` | `LLMMarkerResponseFrame` | | `bot-llm-stopped` | `LLMFullResponseEndFrame` | | `bot-tts-text` | `TTSTextFrame` (text type `SENTENCE`; the wire does not carry it) | | `bot-started-speaking` | `BotStartedSpeakingFrame` | | `bot-stopped-speaking` | `BotStoppedSpeakingFrame` | | `bot-interrupted` | `InterruptionFrame` | | `llm-function-call-in-progress` | `FunctionCallInProgressFrame` | | `llm-function-call-stopped` | `FunctionCallCancelFrame` if `cancelled`, else `FunctionCallResultFrame` | | `user-transcription` | `TranscriptionFrame` if `final` (the default), else `InterimTranscriptionFrame` | * Outgoing: an `OutputAudioRawFrame` becomes a `raw-audio` message with base64 PCM, its sample rate and channel count, and a counter as `id`. An `OutputTransportMessageFrame` (or urgent) is sent only if its message has `label: "rtvi-ai"`. Every other frame returns `null`. * A message that is not JSON, or not labelled `rtvi-ai`, is dropped. Bytes are decoded as strict UTF-8. * Bot audio and metrics messages are not mapped yet, upstream as here. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/serializer-rtvi-client/package.json`: ✅ means the compat smoke test passes on that runtime. Pure TypeScript with no dependencies beyond `@pipedog/core`. Runs on Bun, Node, Deno and Cloudflare Workers. # Telnyx > Phone calls over Telnyx media streaming with TelnyxFrameSerializer, used with FastAPIWebsocketTransport. Source: https://polvallverdu.github.io/pipedog/docs/serializers/telnyx/ ## Overview [#overview] **`TelnyxFrameSerializer`** speaks Telnyx's WebSocket media streaming protocol. Paired with [`FastAPIWebsocketTransport`](https://polvallverdu.github.io/pipedog/docs/transports/websocket.md), it lets a bot answer phone calls: * Caller audio (PCMU or PCMA at 8 kHz) becomes `InputAudioRawFrame`s at the pipeline's input rate. * Bot audio is resampled to 8 kHz, encoded and sent as `media` events. * Touch-tone digits become `InputDTMFFrame`s. * An `EndFrame` or `CancelFrame` hangs up the call through Telnyx's REST API (`autoHangUp`). Pipecat's documentation for the same serializer. `packages/serializer-telnyx/src/telnyx.ts`. Voice API and media streaming. ## Installation [#installation] ```bash bun add @pipedog/serializer-telnyx @pipedog/transport-websocket ``` ```ts import { TelnyxFrameSerializer } from "@pipedog/serializer-telnyx/telnyx"; ``` ## Prerequisites [#prerequisites] * A Telnyx account, a phone number with voice, and media streaming pointed at your WebSocket endpoint. * A Telnyx API key for auto hang-up. The examples read it from `TELNYX_API_KEY`. * The stream ID and call control ID, which Telnyx sends in its `start` message when the stream opens. ## TelnyxFrameSerializer [#telnyxframeserializer] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `streamId` | `string` | yes | The Stream ID for Telnyx. | | `outboundEncoding` | `string` | yes | The encoding type for outbound audio (e.g., "PCMU"). | | `inboundEncoding` | `string` | yes | The encoding type for inbound audio (e.g., "PCMU"). | | `callControlId` | `string \| null \| undefined` | no | The Call Control ID for the Telnyx call (optional, but required for auto hang-up). | | `apiKey` | `string \| null \| undefined` | no | Your Telnyx API key (required for auto hang-up). | | `params` | `TelnyxFrameSerializerInputParams \| null \| undefined` | no | Configuration parameters. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | ### InputParams [#inputparams] `params` is a `TelnyxFrameSerializer.InputParams` (`TelnyxFrameSerializerInputParams`): | Param | Type | Default | Description | | ------------------------- | ---------------- | -------- | ---------------------------------------------------------------------------------------------- | | `telnyxSampleRate` | `number` | `8000` | Sample rate Telnyx uses, in Hz. | | `sampleRate` | `number \| null` | `null` | Override for the pipeline input rate. `null` uses the pipeline's. | | `inboundEncoding` | `string` | `"PCMU"` | Encoding of audio sent to Telnyx. Overwritten by the constructor's `inboundEncoding`. | | `outboundEncoding` | `string` | `"PCMU"` | Encoding of audio received from Telnyx. Overwritten by the constructor's `outboundEncoding`. | | `autoHangUp` | `boolean` | `true` | Hang up the call on `EndFrame` or `CancelFrame`. | | `resamplerClearAfterSecs` | `number \| null` | `0.2` | Seconds of silence after which the stream resamplers clear their history. `null` never clears. | | `ignoreRtviMessages` | `boolean` | `true` | Drop RTVI messages instead of sending them to Telnyx. | ### Usage [#usage] Read Telnyx's first two messages to get the call details, then build the serializer and the transport: ```ts title="telnyx.ts" import { TelnyxFrameSerializer } from "@pipedog/serializer-telnyx/telnyx"; import { FastAPIWebsocketParams, FastAPIWebsocketTransport, } from "@pipedog/transport-websocket/fastapi"; import { websocketFetchHandler } from "@pipedog/transport-websocket/fetch"; import type { WebsocketConnection } from "@pipedog/transport-websocket/socket"; async function runCall(connection: WebsocketConnection): Promise { // Telnyx sends a "connected" message, then "start" with the call details. await connection.recv(); const start = JSON.parse(String(await connection.recv())); const serializer = new TelnyxFrameSerializer({ streamId: start.stream_id, callControlId: start.start.call_control_id, outboundEncoding: start.start.media_format.encoding, inboundEncoding: "PCMU", apiKey: process.env.TELNYX_API_KEY ?? "", }); const transport = new FastAPIWebsocketTransport( connection, new FastAPIWebsocketParams({ audioInEnabled: true, audioOutEnabled: true, addWavHeader: false, serializer, }), ); // ... build and run the pipeline with transport.input() and transport.output() } // The WebSocket URL you give Telnyx's media streaming points here. const handler = websocketFetchHandler(runCall); ``` Without auto hang-up, the call control ID and API key are not needed: ```ts const noHangUp = new TelnyxFrameSerializer({ streamId: "stream-id", outboundEncoding: "PCMU", inboundEncoding: "PCMU", params: new TelnyxFrameSerializer.InputParams({ autoHangUp: false }), }); ``` ### Notes [#notes] * With `autoHangUp` on (the default), the constructor throws if `callControlId` or `apiKey` is missing. The hang-up is a `POST` to `https://api.telnyx.com/v2/calls/{callControlId}/actions/hangup`, attempted once. A 422 with error code 90018 (call already ended) is logged at debug level; other failures are logged as errors and don't throw. * Only `PCMU` and `PCMA` are supported. Another encoding throws when the first audio is sent or received. * An `InterruptionFrame` sends a `clear` event, so Telnyx drops the audio it has buffered. * `OutputTransportMessageFrame`s are sent as JSON, except RTVI messages while `ignoreRtviMessages` is on. * Set `addWavHeader: false` on the transport params: Telnyx takes raw encoded audio. * A DTMF event whose digit is not a keypad key is ignored. ## TeXML helpers [#texml-helpers] `@pipedog/serializer-telnyx/texml` holds the HTTP side of a Telnyx bot: answering the call, checking webhooks, and placing outbound calls. Not in upstream (Pipecat leaves it to the app); it only uses `fetch` and WebCrypto, so it runs on Bun, Node, Deno and Workers. ```ts import { answerTexml, parseCallCost, parseStatusCallback, TelnyxTexmlClient, verifyTelnyxSignature, } from "@pipedog/serializer-telnyx/texml"; // The TeXML application's voice URL: connect the call to the bot's WebSocket. async function answer(request: Request): Promise { const body = await request.text(); const ok = await verifyTelnyxSignature({ body, signature: request.headers.get("telnyx-signature-ed25519"), timestamp: request.headers.get("telnyx-timestamp"), publicKey: env.TELNYX_PUBLIC_KEY, // Mission Control → Keys & Credentials }); if (!ok) return new Response("bad signature", { status: 401 }); return new Response(answerTexml("wss://bot.example.com/telnyx", { parameters: { lang: "en" } }), { headers: { "content-type": "text/xml" }, }); } // Outbound calls through the TeXML REST API. const telnyx = new TelnyxTexmlClient({ apiKey: env.TELNYX_API_KEY, accountSid: env.TELNYX_ACCOUNT_SID, applicationSid: env.TELNYX_APPLICATION_SID, from: "+15550000000", }); const { callSid } = await telnyx.dial({ to: "+15551234567", url: "https://bot.example.com/answer", statusCallback: "https://bot.example.com/status", }); ``` * `verifyTelnyxSignature()` checks the Ed25519 signature over `` `${timestamp}|${body}` `` and refuses timestamps more than 5 minutes off (`tolerance`). Pass the raw body, before parsing it. * `answerTexml(url, { parameters, codec, track })` returns ``, with the URL and parameters XML-escaped. Parameters reach the bot in the `start` message's `custom_parameters`. * `parseStatusCallback(body)` reads a form-encoded status callback (`callSid`, `status`, all `fields`), and `parseCallCost(body)` Telnyx's `call.cost` webhook (total, currency, billed seconds and cost by part). Both return `null` for anything else. * `TelnyxTexmlClient.dial()` throws `TelnyxCallRejectedError` when Telnyx refuses the call (a 4xx, or a destination that is not E.164); its `invalidDestination` says whether the number itself is unusable, so a retry won't help. A 5xx throws a plain `Error`, since the call may have been placed anyway. `getCall(callSid)` looks a call up. Pass `fetch` to the client for tests or a proxy. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/serializer-telnyx/package.json`: ✅ means the compat smoke test passes on that runtime. The serializer uses the global `fetch` for the hang-up and runs on Bun, Node, Deno and Cloudflare Workers. On Workers, accept the Telnyx WebSocket with `websocketFetchHandler`; see [WebSocket](https://polvallverdu.github.io/pipedog/docs/transports/websocket.md). Resampling between 8 kHz and the pipeline rate goes through core's resampler engine (`@pipedog/resampler-wasm` by default, a pure TypeScript fallback if it can't load) in place of libsoxr, so the samples differ slightly from upstream's. `parseTelephonyWebsocket` from `@pipedog/runner` detects Telnyx and parses its handshake for you (`callData.streamId`, `callData.callId` for the call control ID, and `outboundEncoding`); see [How phone calls connect](https://polvallverdu.github.io/pipedog/docs/telephony/#reading-the-handshake). The example above reads the `start` message by hand instead. # Twilio > Phone calls over Twilio Media Streams with TwilioFrameSerializer, used with FastAPIWebsocketTransport. Source: https://polvallverdu.github.io/pipedog/docs/serializers/twilio/ ## Overview [#overview] **`TwilioFrameSerializer`** speaks Twilio's Media Streams WebSocket protocol. Paired with [`FastAPIWebsocketTransport`](https://polvallverdu.github.io/pipedog/docs/transports/websocket.md), it lets a bot answer phone calls: * Caller audio (μ-law at 8 kHz) becomes `InputAudioRawFrame`s at the pipeline's input rate. * Bot audio is resampled to 8 kHz, μ-law encoded and sent as `media` events. * Touch-tone digits become `InputDTMFFrame`s. * An `InterruptionFrame` sends `clear`, so Twilio drops the audio it has buffered. * An `EndFrame` or `CancelFrame` hangs up the call through Twilio's REST API (`autoHangUp`). New to phone bots? Start with [How phone calls connect](https://polvallverdu.github.io/pipedog/docs/telephony.md) and the [Twilio quickstart](https://polvallverdu.github.io/pipedog/docs/telephony/twilio-quickstart.md). Pipecat's documentation for the same serializer. `packages/serializer-twilio/src/twilio.ts`. Media Streams WebSocket messages. ## Installation [#installation] ```bash bun add @pipedog/serializer-twilio @pipedog/transport-websocket @pipedog/runner ``` ```ts import { TwilioFrameSerializer } from "@pipedog/serializer-twilio/twilio"; ``` ## Prerequisites [#prerequisites] * A Twilio account and a phone number with voice, whose incoming-call webhook returns TwiML with ``. * For auto hang-up, your account SID and one credential: the auth token, or an API key (its SID, which starts with `SK`, and its secret). The examples read them from `TWILIO_ACCOUNT_SID`, `TWILIO_AUTH_TOKEN`, `TWILIO_API_KEY` and `TWILIO_API_SECRET`. Each Twilio region has its own auth token and API keys, so use the ones that belong to the region you hang up through. * The stream SID and call SID, which Twilio sends in its `start` message. [`parseTelephonyWebsocket`](https://polvallverdu.github.io/pipedog/docs/telephony/#reading-the-handshake) reads them for you. ## TwilioFrameSerializer [#twilioframeserializer] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `streamSid` | `string` | yes | The Twilio Media Stream SID. | | `callSid` | `string \| null \| undefined` | no | The associated Twilio Call SID (optional, but required for auto hang-up). | | `accountSid` | `string \| null \| undefined` | no | Twilio account SID (required for auto hang-up). | | `authToken` | `string \| null \| undefined` | no | Twilio Auth Token for auto hang-up. Mutually exclusive with apiKeySid/apiKeySecret. | | `apiKeySid` | `string \| null \| undefined` | no | Twilio API key SID (starts with "SK"). Alternative to authToken for auto hang-up. Must be provided with apiKeySecret. | | `apiKeySecret` | `string \| null \| undefined` | no | Twilio API key secret. Must be provided with apiKeySid. | | `region` | `string \| null \| undefined` | no | Twilio region (e.g., "au1", "ie1"). Must be specified with edge. | | `edge` | `string \| null \| undefined` | no | Twilio edge location (e.g., "sydney", "dublin"). Must be specified with region. | | `baseUrl` | `string \| null \| undefined` | no | Optional REST API base URL (scheme + host, e.g. `https://api.twilio.com`) used for auto hang-up. Defaults to the host derived from region/edge. Set this to target a Twilio-API-compatible backend or a self-hosted server. | | `params` | `TwilioFrameSerializerInputParams \| null \| undefined` | no | Configuration parameters. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | ### InputParams [#inputparams] `params` is a `TwilioFrameSerializer.InputParams` (`TwilioFrameSerializerInputParams`): | Param | Type | Default | Description | | ------------------------- | ---------------- | ------- | ---------------------------------------------------------------------------------------------- | | `twilioSampleRate` | `number` | `8000` | Sample rate Twilio uses, in Hz. | | `sampleRate` | `number \| null` | `null` | Override for the pipeline input rate. `null` uses the pipeline's. | | `autoHangUp` | `boolean` | `true` | Hang up the call on `EndFrame` or `CancelFrame`. | | `resamplerClearAfterSecs` | `number \| null` | `0.2` | Seconds of silence after which the stream resamplers clear their history. `null` never clears. | | `ignoreRtviMessages` | `boolean` | `true` | Drop RTVI messages instead of sending them to Twilio. | ### Usage [#usage] Read Twilio's handshake to get the call details, then build the serializer and the transport: ```ts title="twilio.ts" import { parseTelephonyWebsocket } from "@pipedog/runner/utils"; import { TwilioFrameSerializer } from "@pipedog/serializer-twilio/twilio"; import { FastAPIWebsocketParams, FastAPIWebsocketTransport, } from "@pipedog/transport-websocket/fastapi"; import { websocketFetchHandler } from "@pipedog/transport-websocket/fetch"; import type { WebsocketConnection } from "@pipedog/transport-websocket/socket"; async function runCall(connection: WebsocketConnection): Promise { // Twilio sends "connected", then "start" with the stream and call SIDs. const [, callData] = await parseTelephonyWebsocket(connection); const serializer = new TwilioFrameSerializer({ streamSid: callData.streamId ?? "", callSid: callData.callId, accountSid: process.env.TWILIO_ACCOUNT_SID, authToken: process.env.TWILIO_AUTH_TOKEN, }); const transport = new FastAPIWebsocketTransport( connection, new FastAPIWebsocketParams({ audioInEnabled: true, audioOutEnabled: true, addWavHeader: false, serializer, }), ); // ... build and run the pipeline with transport.input() and transport.output() } // The in your TwiML points here. const handler = websocketFetchHandler(runCall); ``` To hang up with an API key instead of the auth token, pass `apiKeySid` and `apiKeySecret`: ```ts const withApiKey = new TwilioFrameSerializer({ streamSid: "MZ...", callSid: "CA...", accountSid: process.env.TWILIO_ACCOUNT_SID, // An API key replaces the auth token. Pass both parts, and no authToken. apiKeySid: process.env.TWILIO_API_KEY, apiKeySecret: process.env.TWILIO_API_SECRET, }); ``` To hang up through a regional API host, pass both `region` and `edge`: ```ts const regional = new TwilioFrameSerializer({ streamSid: "MZ...", callSid: "CA...", accountSid: "AC...", authToken: "...", // Hang up through api.dublin.ie1.twilio.com. Both are required together. region: "ie1", edge: "dublin", }); ``` Without auto hang-up, only the stream SID is needed: ```ts const noHangUp = new TwilioFrameSerializer({ streamSid: "MZ...", params: new TwilioFrameSerializer.InputParams({ autoHangUp: false }), }); ``` ### Notes [#notes] * With `autoHangUp` on (the default), the constructor throws if `callSid` or `accountSid` is missing, if there is neither an `authToken` nor an API key, if both an `authToken` and any part of an API key are given, if only one of `apiKeySid` and `apiKeySecret` is set, or if only one of `region` and `edge` is set (unless `baseUrl` is). * The hang-up is a `POST` of `Status=completed` to `https://api.twilio.com/2010-04-01/Accounts/{accountSid}/Calls/{callSid}.json` with Basic auth (`accountSid` and `authToken`, or `apiKeySid` and `apiKeySecret`), attempted once. A 404 with code 20404 (call already ended) is logged at debug level; other failures are logged as errors and don't throw. `baseUrl` replaces the host, for a Twilio-compatible API. * When the caller hangs up first, the socket is already closed, so no hang-up request is made. * Set `addWavHeader: false` on the transport params: Twilio takes raw μ-law. * `OutputTransportMessageFrame`s are sent as JSON, except RTVI messages while `ignoreRtviMessages` is on. * A DTMF event whose digit is not a keypad key is ignored. Malformed JSON and `media` events without a payload are ignored too. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/serializer-twilio/package.json`: ✅ means the compat smoke test passes on that runtime. The serializer uses the global `fetch` for the hang-up and runs on Bun, Node, Deno and Cloudflare Workers. Accept the Twilio WebSocket with `websocketFetchHandler` (Deno, Workers) or any of the [WebSocket](https://polvallverdu.github.io/pipedog/docs/transports/websocket.md) server adapters. Resampling between 8 kHz and the pipeline rate goes through core's resampler engine (`@pipedog/resampler-wasm` by default) in place of libsoxr, so resampled samples differ slightly from upstream's. At 8 kHz the wire output is byte-identical; the package's golden tests replay vectors recorded from upstream. # Vonage > Vonage Audio Connector streams with VonageFrameSerializer, used with FastAPIWebsocketTransport. Source: https://polvallverdu.github.io/pipedog/docs/serializers/vonage/ ## Overview [#overview] **`VonageFrameSerializer`** speaks the WebSocket protocol of Vonage's Audio Connector, which streams a Vonage Video session's audio (or a Voice API call's) to your server: * Binary messages carry raw 16-bit PCM (16 kHz by default), resampled to the pipeline's input rate as `InputAudioRawFrame`s. * Bot audio is resampled to the Vonage rate and sent as binary messages. * JSON text events: `websocket:connected`, `websocket:cleared` and `websocket:notify` are logged, and `websocket:dtmf` becomes an `InputDTMFFrame`. * An `InterruptionFrame` sends `{"action": "clear"}`. Pipecat's documentation for the same serializer. `packages/serializer-vonage/src/vonage.ts`. The Audio Connector. ## Installation [#installation] ```bash bun add @pipedog/serializer-vonage @pipedog/transport-websocket ``` ```ts import { VonageFrameSerializer } from "@pipedog/serializer-vonage/vonage"; ``` ## Prerequisites [#prerequisites] * A Vonage application, and an Audio Connector (or a Voice API `websocket` endpoint) pointed at your WebSocket URL, with `audio/l16` at the rate you configure. ## VonageFrameSerializer [#vonageframeserializer] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `params` | `VonageFrameSerializerInputParams \| null \| undefined` | no | Configuration parameters. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | ### InputParams [#inputparams] `params` is a `VonageFrameSerializer.InputParams` (`VonageFrameSerializerInputParams`): | Param | Type | Default | Description | | ------------------------- | ---------------- | ------- | ---------------------------------------------------------------------------------------------- | | `vonageSampleRate` | `number` | `16000` | Sample rate of the Vonage stream, in Hz (8000, 16000 or 24000). | | `sampleRate` | `number \| null` | `null` | Override for the pipeline input rate. `null` uses the pipeline's. | | `resamplerClearAfterSecs` | `number \| null` | `0.2` | Seconds of silence after which the stream resamplers clear their history. `null` never clears. | | `ignoreRtviMessages` | `boolean` | `true` | Drop RTVI messages instead of sending them to Vonage. | ### Usage [#usage] ```ts title="vonage.ts" import { VonageFrameSerializer } from "@pipedog/serializer-vonage/vonage"; import { FastAPIWebsocketParams, FastAPIWebsocketTransport, } from "@pipedog/transport-websocket/fastapi"; import { websocketFetchHandler } from "@pipedog/transport-websocket/fetch"; import type { WebsocketConnection } from "@pipedog/transport-websocket/socket"; async function runSession(connection: WebsocketConnection): Promise { const serializer = new VonageFrameSerializer({ // The rate you gave the Audio Connector (audio/l16;rate=16000). params: new VonageFrameSerializer.InputParams({ vonageSampleRate: 16000 }), }); const transport = new FastAPIWebsocketTransport( connection, new FastAPIWebsocketParams({ audioInEnabled: true, audioOutEnabled: true, addWavHeader: false, serializer, }), ); // ... build and run the pipeline with transport.input() and transport.output() } // The Audio Connector's websocket.uri points here. const handler = websocketFetchHandler(runSession); ``` ### Notes [#notes] * DTMF digits may arrive as strings or numbers, under `digit` or `dtmf.digit`; both work. * `OutputTransportMessageFrame`s are sent as JSON text (e.g. a `notify` action), except RTVI messages while `ignoreRtviMessages` is on. * There is no hang-up call: end the session from Vonage's side. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/serializer-vonage/package.json`: ✅ means the compat smoke test passes on that runtime. Runs on Bun, Node, Deno and Cloudflare Workers. The package's golden tests replay vectors recorded from upstream: at 16 kHz the audio is byte-identical. # Azure OpenAI > Image generation through Azure OpenAI's REST API, with AzureImageGenServiceREST. Source: https://polvallverdu.github.io/pipedog/docs/services/image/azure/ ## Overview [#overview] * **`AzureImageGenServiceREST`**: turns each `TextFrame` it receives into a prompt for Azure OpenAI's asynchronous image generation API. It submits the job, polls it once a second until it succeeds, downloads the image and pushes it as a `URLImageRawFrame` of RGB pixels. Pipecat's documentation for the same service. `packages/azure/src/image.ts`. Image generation on Azure OpenAI. ## Installation [#installation] ```bash bun add @pipedog/azure ``` ```ts import { AzureImageGenServiceREST } from "@pipedog/azure/image"; ``` ## Prerequisites [#prerequisites] * An Azure OpenAI resource whose region offers image generation, and its API key. The examples read it from `AZURE_OPENAI_API_KEY`. * The resource endpoint, ending in `/` (the service appends `openai/images/...` to it). ## AzureImageGenServiceREST [#azureimagegenservicerest] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | Azure OpenAI API key for authentication. | | `endpoint` | `string` | yes | Azure OpenAI endpoint URL, ending in `/`. | | `apiVersion` | `string \| undefined` | no | Azure API version string. Defaults to "2023-06-01-preview". | | `fetch` | `((input: RequestInfo \| URL, init?: RequestInit) => Promise) \| null \| undefined` | no | `fetch` used for the requests. Defaults to the global `fetch`. DIVERGENCE: replaces upstream's required `aiohttp_session`. | | `decodeImage` | `ImageDecoder \| null \| undefined` | no | Decodes the generated image into RGB pixels. Defaults to `decodePngToRgb`. DIVERGENCE: pipedog-only. Upstream decodes with Pillow. | | `settings` | `AzureImageGenSettings \| null \| undefined` | no | Runtime-updatable settings. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new AzureImageGenServiceREST.Settings({...})` (`AzureImageGenSettings`). Any setting you leave out keeps the default below. | Setting | Type | Default | Description | | ----------- | ---------------- | ------- | ----------------------------------------------------------------- | | `model` | `string \| null` | `null` | Not sent: the endpoint decides the model. | | `imageSize` | `string \| null` | `null` | Sent as `size` (for example `"1024x1024"`). `null` leaves it out. | ### Usage [#usage] ```ts title="image.ts" import { AzureImageGenServiceREST } from "@pipedog/azure/image"; const imageGen = new AzureImageGenServiceREST({ apiKey: process.env.AZURE_OPENAI_API_KEY ?? "", // The resource endpoint, with its trailing slash. endpoint: "https://my-resource.openai.azure.com/", settings: new AzureImageGenServiceREST.Settings({ imageSize: "1024x1024" }), }); ``` ### Notes [#notes] * The request is `POST {endpoint}openai/images/generations:submit?api-version=...` with the key in the `api-key` header and `{ prompt, n: 1, size }` as the body. The job's status is read from the `operation-location` URL the submission returns. * Polling gives up after 119 attempts, about two minutes, with an `ErrorFrame` (`"Image generation timed out"`). A finished job without an image URL gives `"Image generation failed"`. * A submission without an `operation-location` header (a 429, for example) throws, as upstream. * `apiVersion` defaults to `"2023-06-01-preview"`, the version this asynchronous API was published under. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/azure/package.json`: ✅ means the compat smoke test passes on that runtime. * All requests go through `fetch` (the global one, or the `fetch` option), which replaces upstream's required `aiohttp_session`. The service works on Bun, Node, Deno and Cloudflare Workers. * The image is decoded to RGB with `decodePngToRgb` from `@pipedog/openai/image`, a pure JS PNG decoder, and the frame's `format` is always `"RGB"`. Upstream decodes with Pillow and reports Pillow's mode. Pass `decodeImage` to decode other formats. * The deprecated `imageSize` and `model` constructor options are not ported. Use `settings`. # Google > Text-to-image generation with GoogleImageGenService and Gemini image models. Source: https://polvallverdu.github.io/pipedog/docs/services/image/google/ ## Overview [#overview] `GoogleImageGenService` generates images from text prompts with Gemini image models through the Gemini API. Gemini returns each image inline in the response, and the service decodes it to RGB pixels for the output transport. * **`GoogleImageGenService`**: one `generateContent` request per image, with aspect ratio and size settings. Pipecat's documentation for the same service. `packages/google/src/image.ts` and `image_decoder.ts`. Image models, aspect ratios and sizes. ## Installation [#installation] ```bash bun add @pipedog/google ``` ```ts import { GoogleImageGenService } from "@pipedog/google/image"; import type { ImageDecoder } from "@pipedog/google/image_decoder"; ``` ## Prerequisites [#prerequisites] * A Gemini API key from [Google AI Studio](https://aistudio.google.com/apikey). * Pass the key as `apiKey`. The examples read it from `GOOGLE_API_KEY`. * An image decoder (see below). ## GoogleImageGenService [#googleimagegenservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | Google AI API key for authentication. | | `httpOptions` | `HttpOptions \| null \| undefined` | no | HTTP options for the client. | | `settings` | `GoogleImageGenSettings \| null \| undefined` | no | Runtime-updatable settings. | | `client` | `GoogleImageGenClient \| null \| undefined` | no | A GenAI client to use instead of creating one. | | `imageDecoder` | `ImageDecoder \| null \| undefined` | no | Decodes the returned image to RGB pixels (upstream: Pillow). | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new GoogleImageGenService.Settings({...})`. | Setting | Type | Default | Description | | ---------------- | ------------------------------ | -------------------------- | --------------------------------------------------------------------------- | | `model` | `string` | `"gemini-3.1-flash-image"` | Gemini image model. | | `numberOfImages` | `number` | `1` | Images per prompt. | | `aspectRatio` | `string \| null` | `"1:1"` | Aspect ratio, e.g. `"16:9"`. `null` leaves the model default. | | `imageSize` | `"1K" \| "2K" \| "4K" \| null` | `null` | Resolution. `null` leaves the model default. Supported sizes vary by model. | ### Usage [#usage] ```ts title="image.ts" import { GoogleImageGenService } from "@pipedog/google/image"; import type { ImageDecoder } from "@pipedog/google/image_decoder"; // Your decoder from PNG/JPEG/WebP bytes to RGB pixels (sharp, @jsquash/*, ...). declare const decodeImage: ImageDecoder; const imageGen = new GoogleImageGenService({ apiKey: process.env.GOOGLE_API_KEY ?? "", imageDecoder: decodeImage, settings: new GoogleImageGenService.Settings({ model: "gemini-3.1-flash-image", aspectRatio: "16:9", imageSize: "2K", }), }); ``` ### Image decoder [#image-decoder] Upstream decodes the returned PNG, JPEG or WebP with Pillow. pipedog has no edge-compatible decoder yet, so the service takes an `imageDecoder`: a function from the encoded bytes and MIME type to `{ image, size, mode }` (raw pixels, `[width, height]` and a Pillow-style mode such as `"RGB"`). Build one on `sharp`, `@jsquash/*` or similar. The default decoder throws "not ported yet". Without an `imageDecoder`, every generated image yields an `ErrorFrame`. ### Notes [#notes] * The service generates an image for each `TextFrame` it receives and emits a `URLImageRawFrame` per image, with `url: null` since Gemini returns the image inline. * With `numberOfImages` above 1, the images are generated one after another, one request each. * A response with no image, or a failed request, yields an `ErrorFrame`. * Supports TTFB metrics. * Pass `client` to use your own GenAI client (anything with `models.generateContent`). ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/google/package.json`: ✅ means the compat smoke test passes on that runtime. The client is `GoogleGenAI` from `@google/genai`, which uses `fetch`, so the service runs on Bun, Node, Deno and Cloudflare Workers, as long as your `imageDecoder` does too. The deprecated upstream `InputParams` class, the `params` option and the `negativePrompt` setting are not ported. # Image generation > Generate images from text prompts. Source: https://polvallverdu.github.io/pipedog/docs/services/image/ {/* Generated by scripts/generate-providers.ts from data/providers.ts. */} An image generation service takes a text prompt and pushes the generated image as a frame. | Provider | Classes | Package | Status | | ------------------------------------------ | -------------------------- | ----------------- | ------ | | [OpenAI](https://polvallverdu.github.io/pipedog/docs/services/image/openai.md) | `OpenAIImageGenService` | `@pipedog/openai` | Ported | | [Google](https://polvallverdu.github.io/pipedog/docs/services/image/google.md) | `GoogleImageGenService` | `@pipedog/google` | Ported | | [Azure OpenAI](https://polvallverdu.github.io/pipedog/docs/services/image/azure.md) | `AzureImageGenServiceREST` | `@pipedog/azure` | Ported | Pipecat's [image generation reference](https://docs.pipecat.ai/api-reference/server/services/image-generation) lists every upstream provider. Those not in this table are on [Other upstream providers](https://polvallverdu.github.io/pipedog/docs/services/other-upstream.md). # OpenAI > Image generation with OpenAIImageGenService, using OpenAI's GPT Image models. Source: https://polvallverdu.github.io/pipedog/docs/services/image/openai/ ## Overview [#overview] * **`OpenAIImageGenService`**: turns each `TextFrame` it receives into a prompt for OpenAI's Images API (`images.generate`) and pushes the result as a `URLImageRawFrame` of RGB pixels. Pipecat's documentation for the same service. `packages/openai/src/image.ts`. Image models, sizes and quality. ## Installation [#installation] ```bash bun add @pipedog/openai ``` ```ts import { OpenAIImageGenService } from "@pipedog/openai/image"; ``` ## Prerequisites [#prerequisites] * An OpenAI account and an API key from the [OpenAI dashboard](https://platform.openai.com/api-keys). * Pass the key as `apiKey` (required). The examples read it from `OPENAI_API_KEY`. ## OpenAIImageGenService [#openaiimagegenservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | OpenAI API key for authentication. | | `baseUrl` | `string \| null \| undefined` | no | Custom base URL for OpenAI API. If null, uses default. | | `fetch` | `((input: RequestInfo \| URL, init?: RequestInit) => Promise) \| null \| undefined` | no | `fetch` used to download hosted results. Defaults to the global `fetch`. DIVERGENCE: replaces upstream's required `aiohttp_session`. | | `decodeImage` | `ImageDecoder \| null \| undefined` | no | Decodes the generated image into RGB pixels. Defaults to `decodePngToRgb`. DIVERGENCE: pipedog-only. Upstream decodes with Pillow. | | `settings` | `OpenAIImageGenSettings \| null \| undefined` | no | Runtime-updatable settings. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new OpenAIImageGenService.Settings({...})` (`OpenAIImageGenSettings`). Any setting you leave out keeps the default below. | Setting | Type | Default | Description | | ------------ | ----------------------------------------- | ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------- | | `model` | `string` | `"gpt-image-2.5-flare"` | Image model. | | `imageSize` | `OpenAIImageSize \| string \| null` | `null` | `"auto"`, `"1024x1024"`, `"1536x1024"`, `"1024x1536"`, ... `null` leaves it to OpenAI. | | `quality` | `OpenAIImageQuality \| string \| null` | `null` | `"auto"`, `"low"`, `"medium"`, `"high"`, ... Lower is faster. `null` lets OpenAI pick. | | `moderation` | `OpenAIImageModeration \| string \| null` | `null` | Content-moderation level for generated images: `"low"` for less restrictive filtering, or `"auto"`. `null` leaves OpenAI's default. | ### Usage [#usage] ```ts title="image.ts" import { OpenAIImageGenService } from "@pipedog/openai/image"; const imageGen = new OpenAIImageGenService({ apiKey: process.env.OPENAI_API_KEY ?? "", settings: new OpenAIImageGenService.Settings({ model: "gpt-image-2.5-flare", imageSize: "1024x1024", quality: "low", }), }); ``` ### Notes [#notes] * Each `TextFrame` is pushed on downstream, then used as the prompt. One image is generated per prompt (`n: 1`). * The GPT Image models return the image inline (base64). When the response carries a URL instead, the image is downloaded with `fetch` (the global one, or the `fetch` option) and the URL is set on the frame. * The image is decoded to RGB (alpha dropped) with `decodePngToRgb`, a pure JS PNG decoder. Pass `decodeImage` to decode other formats. * A response with no image data produces an `ErrorFrame`. * As upstream, the constructor passes only `settings` to the base service, so the inherited frame processor options (such as `name`) have no effect. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/openai/package.json`: ✅ means the compat smoke test passes on that runtime. * The client is the `openai` JS SDK over `fetch`, and decoding is pure JS (`fast-png`), so it works on Bun, Node, Deno and Cloudflare Workers. * The `fetch` option replaces upstream's required `aiohttp_session`. `decodeImage` is a pipedog addition; upstream decodes with Pillow. * The deprecated `imageSize` and `model` constructor options are not ported. Use `settings`. * For Azure OpenAI image generation, see [`AzureImageGenServiceREST`](https://polvallverdu.github.io/pipedog/docs/services/image/azure.md). # Supported services > Every pipedog service by category: ported, and coming soon from Pipecat. Source: https://polvallverdu.github.io/pipedog/docs/services/ {/* Generated by scripts/generate-providers.ts from data/providers.ts. */} Services are frame processors that call an AI provider. pipedog ports them from Pipecat with the same class names and options (in camelCase), one package per provider: `@pipedog/openai`, `@pipedog/deepgram` and so on. 102 service pages are ported so far. The "coming soon" pages link to Pipecat's docs and to a port request. Each service page follows the same layout as Pipecat's: overview, installation, configuration options, settings, usage, and runtime notes for Bun, Node, Deno and Cloudflare Workers. ## LLM [#llm] Chat models that turn the conversation context into a response. [All llm services](https://polvallverdu.github.io/pipedog/docs/services/llm.md). | Provider | Classes | Package | Status | | ------------------------------------------------------- | ------------------------------------------------------------ | --------------------- | ------ | | [OpenAI](https://polvallverdu.github.io/pipedog/docs/services/llm/openai.md) | `OpenAILLMService` | `@pipedog/openai` | Ported | | [OpenAI Responses](https://polvallverdu.github.io/pipedog/docs/services/llm/openai-responses.md) | `OpenAIResponsesLLMService`, `OpenAIResponsesHttpLLMService` | `@pipedog/openai` | Ported | | [Azure OpenAI](https://polvallverdu.github.io/pipedog/docs/services/llm/azure.md) | `AzureLLMService` | `@pipedog/azure` | Ported | | [Anthropic](https://polvallverdu.github.io/pipedog/docs/services/llm/anthropic.md) | `AnthropicLLMService` | `@pipedog/anthropic` | Ported | | [Google Gemini](https://polvallverdu.github.io/pipedog/docs/services/llm/google.md) | `GoogleLLMService` | `@pipedog/google` | Ported | | [Google Vertex AI](https://polvallverdu.github.io/pipedog/docs/services/llm/google-vertex.md) | `GoogleVertexLLMService` | `@pipedog/google` | Ported | | [AWS Bedrock](https://polvallverdu.github.io/pipedog/docs/services/llm/aws.md) | `AWSBedrockLLMService` | `@pipedog/aws` | Ported | | [AWS AgentCore](https://polvallverdu.github.io/pipedog/docs/services/llm/aws-agentcore.md) | `AWSAgentCoreProcessor` | `@pipedog/aws` | Ported | | [LangChain](https://polvallverdu.github.io/pipedog/docs/services/llm/langchain.md) | `LangchainProcessor` | `@pipedog/langchain` | Ported | | [Vercel AI SDK](https://polvallverdu.github.io/pipedog/docs/services/llm/ai-sdk.md) | `AISDKLLMService` | `@pipedog/ai-sdk` | Ported | | [Cloudflare Workers AI](https://polvallverdu.github.io/pipedog/docs/services/llm/cloudflare.md) | `WorkersAILLMService` | `@pipedog/cloudflare` | Ported | | [Perplexity](https://polvallverdu.github.io/pipedog/docs/services/llm/perplexity.md) | `PerplexityAgentHttpLLMService` | `@pipedog/perplexity` | Ported | | [NVIDIA](https://polvallverdu.github.io/pipedog/docs/services/llm/nvidia.md) | `NvidiaLLMService`, `NvidiaOmniLLMService` | `@pipedog/nvidia` | Ported | | [Baseten](https://polvallverdu.github.io/pipedog/docs/services/llm/baseten.md) | `BasetenLLMService` | `@pipedog/baseten` | Ported | | [Cerebras](https://polvallverdu.github.io/pipedog/docs/services/llm/cerebras.md) | `CerebrasLLMService` | `@pipedog/cerebras` | Ported | | [Crusoe](https://polvallverdu.github.io/pipedog/docs/services/llm/crusoe.md) | `CrusoeLLMService` | `@pipedog/crusoe` | Ported | | [DeepSeek](https://polvallverdu.github.io/pipedog/docs/services/llm/deepseek.md) | `DeepSeekLLMService` | `@pipedog/deepseek` | Ported | | [Fireworks AI](https://polvallverdu.github.io/pipedog/docs/services/llm/fireworks.md) | `FireworksLLMService` | `@pipedog/fireworks` | Ported | | [Groq](https://polvallverdu.github.io/pipedog/docs/services/llm/groq.md) | `GroqLLMService` | `@pipedog/groq` | Ported | | [Inception](https://polvallverdu.github.io/pipedog/docs/services/llm/inception.md) | `InceptionLLMService` | `@pipedog/inception` | Ported | | [Mistral](https://polvallverdu.github.io/pipedog/docs/services/llm/mistral.md) | `MistralLLMService` | `@pipedog/mistral` | Ported | | [Nebius](https://polvallverdu.github.io/pipedog/docs/services/llm/nebius.md) | `NebiusLLMService` | `@pipedog/nebius` | Ported | | [Novita AI](https://polvallverdu.github.io/pipedog/docs/services/llm/novita.md) | `NovitaLLMService` | `@pipedog/novita` | Ported | | [Ollama](https://polvallverdu.github.io/pipedog/docs/services/llm/ollama.md) | `OLLamaLLMService` | `@pipedog/ollama` | Ported | | [OpenRouter](https://polvallverdu.github.io/pipedog/docs/services/llm/openrouter.md) | `OpenRouterLLMService` | `@pipedog/openrouter` | Ported | | [Qwen](https://polvallverdu.github.io/pipedog/docs/services/llm/qwen.md) | `QwenLLMService` | `@pipedog/qwen` | Ported | | [SambaNova](https://polvallverdu.github.io/pipedog/docs/services/llm/sambanova.md) | `SambaNovaLLMService` | `@pipedog/sambanova` | Ported | | [Sarvam AI](https://polvallverdu.github.io/pipedog/docs/services/llm/sarvam.md) | `SarvamLLMService` | `@pipedog/sarvam` | Ported | | [Smallest AI](https://polvallverdu.github.io/pipedog/docs/services/llm/smallest.md) | `SmallestLLMService` | `@pipedog/smallest` | Ported | | [Together AI](https://polvallverdu.github.io/pipedog/docs/services/llm/together.md) | `TogetherLLMService` | `@pipedog/together` | Ported | | [xAI Grok](https://polvallverdu.github.io/pipedog/docs/services/llm/grok.md) | `GrokLLMService` | `@pipedog/xai` | Ported | ## Speech-to-text [#speech-to-text] Transcribe the user's speech into TranscriptionFrames. [All speech-to-text services](https://polvallverdu.github.io/pipedog/docs/services/stt.md). | Provider | Classes | Package | Status | | --------------------------------------------------------------- | ----------------------------------------------------------------------------------- | ----------------------- | ----------- | | [AssemblyAI](https://polvallverdu.github.io/pipedog/docs/services/stt/assemblyai.md) | `AssemblyAISTTService`, `AssemblyAISyncSTTService` | `@pipedog/assemblyai` | Ported | | [Gladia](https://polvallverdu.github.io/pipedog/docs/services/stt/gladia.md) | `GladiaSTTService` | `@pipedog/gladia` | Ported | | [Speechmatics](https://polvallverdu.github.io/pipedog/docs/services/stt/speechmatics.md) | `SpeechmaticsSTTService` | `@pipedog/speechmatics` | Ported | | [Soniox](https://polvallverdu.github.io/pipedog/docs/services/stt/soniox.md) | `SonioxSTTService` | `@pipedog/soniox` | Ported | | [Deepgram](https://polvallverdu.github.io/pipedog/docs/services/stt/deepgram.md) | `DeepgramSTTService`, `DeepgramFluxSTTService` | `@pipedog/deepgram` | Ported | | [Cloudflare Workers AI](https://polvallverdu.github.io/pipedog/docs/services/stt/cloudflare.md) | `WorkersAIFluxSTTService`, `WorkersAINova3STTService`, `WorkersAIWhisperSTTService` | `@pipedog/cloudflare` | Ported | | [OpenAI](https://polvallverdu.github.io/pipedog/docs/services/stt/openai.md) | `OpenAISTTService`, `OpenAIRealtimeSTTService` | `@pipedog/openai` | Ported | | [Google](https://polvallverdu.github.io/pipedog/docs/services/stt/google.md) | `GoogleSTTService`, `GeminiSTTService` | `@pipedog/google` | Ported | | [Cartesia](https://polvallverdu.github.io/pipedog/docs/services/stt/cartesia.md) | `CartesiaSTTService`, `CartesiaTurnsSTTService` | `@pipedog/cartesia` | Ported | | [ElevenLabs](https://polvallverdu.github.io/pipedog/docs/services/stt/elevenlabs.md) | `ElevenLabsSTTService`, `ElevenLabsRealtimeSTTService` | `@pipedog/elevenlabs` | Ported | | [AWS Transcribe](https://polvallverdu.github.io/pipedog/docs/services/stt/aws.md) | `AWSTranscribeSTTService` | `@pipedog/aws` | Ported | | [AWS SageMaker (bidi client)](https://polvallverdu.github.io/pipedog/docs/services/stt/aws-sagemaker.md) | `SageMakerBidiClient` | `@pipedog/aws` | Ported | | [Azure Speech](https://polvallverdu.github.io/pipedog/docs/services/stt/azure.md) | `AzureSTTService` | `@pipedog/azure` | Ported | | [Whisper](https://polvallverdu.github.io/pipedog/docs/services/stt/whisper.md) | `WhisperSTTService`, `WhisperSTTServiceMLX`, `BaseWhisperSTTService` | `@pipedog/whisper` | Ported | | [Moonshine (local)](https://polvallverdu.github.io/pipedog/docs/services/stt/moonshine.md) | `MoonshineSTTService` | `@pipedog/moonshine` | Ported | | [FunASR SenseVoice (local)](https://polvallverdu.github.io/pipedog/docs/services/stt/funasr.md) | `FunASRSTTService` | `@pipedog/funasr` | Ported | | [Groq](https://polvallverdu.github.io/pipedog/docs/services/stt/groq.md) | `GroqSTTService` | `@pipedog/groq` | Ported | | [Mistral](https://polvallverdu.github.io/pipedog/docs/services/stt/mistral.md) | `MistralSTTService` | `@pipedog/mistral` | Ported | | [Together AI](https://polvallverdu.github.io/pipedog/docs/services/stt/together.md) | `TogetherSTTService` | `@pipedog/together` | Ported | | [xAI](https://polvallverdu.github.io/pipedog/docs/services/stt/xai.md) | `XAISTTService` | `@pipedog/xai` | Ported | | [Sarvam AI](https://polvallverdu.github.io/pipedog/docs/services/stt/sarvam.md) | `SarvamSTTService`, `SarvamRealtimeSTTService` | `@pipedog/sarvam` | Ported | | [Smallest AI](https://polvallverdu.github.io/pipedog/docs/services/stt/smallest.md) | `SmallestSTTService` | `@pipedog/smallest` | Ported | | [Gradium](https://polvallverdu.github.io/pipedog/docs/services/stt/gradium.md) | `GradiumSTTService` | `@pipedog/gradium` | Ported | | [Meta](https://polvallverdu.github.io/pipedog/docs/services/stt/meta.md) | `MetaSTTService` | `@pipedog/meta` | Ported | | [Zoom AI](https://polvallverdu.github.io/pipedog/docs/services/stt/zoom-ai.md) | `ZoomAISTTService` | `@pipedog/zoom-ai` | Ported | | [NVIDIA Riva](https://polvallverdu.github.io/pipedog/docs/services/stt/nvidia.md) | `NvidiaSTTService`, `NvidiaSegmentedSTTService` | | Coming soon | ## Text-to-speech [#text-to-speech] Turn the bot's text into audio frames. [All text-to-speech services](https://polvallverdu.github.io/pipedog/docs/services/tts.md). | Provider | Classes | Package | Status | | ------------------------------------------------------ | ----------------------------------------------------------------------------------- | ----------------------- | ----------- | | [OpenAI](https://polvallverdu.github.io/pipedog/docs/services/tts/openai.md) | `OpenAITTSService` | `@pipedog/openai` | Ported | | [Google](https://polvallverdu.github.io/pipedog/docs/services/tts/google.md) | `GoogleTTSService`, `GoogleHttpTTSService`, `GeminiTTSService` | `@pipedog/google` | Ported | | [Deepgram](https://polvallverdu.github.io/pipedog/docs/services/tts/deepgram.md) | `DeepgramTTSService`, `DeepgramHttpTTSService`, `DeepgramFluxTTSService` | `@pipedog/deepgram` | Ported | | [Cloudflare Workers AI](https://polvallverdu.github.io/pipedog/docs/services/tts/cloudflare.md) | `WorkersAIAuraTTSService`, `WorkersAIAuraHttpTTSService`, `WorkersAIMeloTTSService` | `@pipedog/cloudflare` | Ported | | [Speechmatics](https://polvallverdu.github.io/pipedog/docs/services/tts/speechmatics.md) | `SpeechmaticsTTSService` | `@pipedog/speechmatics` | Ported | | [Soniox](https://polvallverdu.github.io/pipedog/docs/services/tts/soniox.md) | `SonioxTTSService` | `@pipedog/soniox` | Ported | | [Cartesia](https://polvallverdu.github.io/pipedog/docs/services/tts/cartesia.md) | `CartesiaTTSService`, `CartesiaHttpTTSService` | `@pipedog/cartesia` | Ported | | [ElevenLabs](https://polvallverdu.github.io/pipedog/docs/services/tts/elevenlabs.md) | `ElevenLabsTTSService`, `ElevenLabsHttpTTSService`, `ElevenLabsDialogueTTSService` | `@pipedog/elevenlabs` | Ported | | [Groq](https://polvallverdu.github.io/pipedog/docs/services/tts/groq.md) | `GroqTTSService` | `@pipedog/groq` | Ported | | [Mistral](https://polvallverdu.github.io/pipedog/docs/services/tts/mistral.md) | `MistralTTSService` | `@pipedog/mistral` | Ported | | [Together AI](https://polvallverdu.github.io/pipedog/docs/services/tts/together.md) | `TogetherTTSService` | `@pipedog/together` | Ported | | [xAI](https://polvallverdu.github.io/pipedog/docs/services/tts/xai.md) | `XAITTSService`, `XAIHttpTTSService` | `@pipedog/xai` | Ported | | [Sarvam AI](https://polvallverdu.github.io/pipedog/docs/services/tts/sarvam.md) | `SarvamTTSService`, `SarvamHttpTTSService` | `@pipedog/sarvam` | Ported | | [Smallest AI](https://polvallverdu.github.io/pipedog/docs/services/tts/smallest.md) | `SmallestTTSService` | `@pipedog/smallest` | Ported | | [Gradium](https://polvallverdu.github.io/pipedog/docs/services/tts/gradium.md) | `GradiumTTSService` | `@pipedog/gradium` | Ported | | [Inworld](https://polvallverdu.github.io/pipedog/docs/services/tts/inworld.md) | `InworldTTSService`, `InworldHttpTTSService` | `@pipedog/inworld` | Ported | | [AWS Polly](https://polvallverdu.github.io/pipedog/docs/services/tts/aws.md) | `AWSPollyTTSService` | `@pipedog/aws` | Ported | | [Azure Speech](https://polvallverdu.github.io/pipedog/docs/services/tts/azure.md) | `AzureTTSService`, `AzureHttpTTSService` | `@pipedog/azure` | Ported | | [Kokoro (local)](https://polvallverdu.github.io/pipedog/docs/services/tts/kokoro.md) | `KokoroTTSService` | `@pipedog/kokoro` | Ported | | [Piper](https://polvallverdu.github.io/pipedog/docs/services/tts/piper.md) | `PiperTTSService`, `PiperHttpTTSService` | `@pipedog/piper` | Ported | | [Pocket TTS (local)](https://polvallverdu.github.io/pipedog/docs/services/tts/pocket-tts.md) | `PocketTTSService` | `@pipedog/pocket-tts` | Ported | | [Rime](https://polvallverdu.github.io/pipedog/docs/services/tts/rime.md) | `RimeTTSService`, `RimeHttpTTSService`, `RimeV1TTSService` | `@pipedog/rime` | Ported | | [Fish Audio](https://polvallverdu.github.io/pipedog/docs/services/tts/fish.md) | `FishAudioTTSService` | `@pipedog/fish` | Ported | | [Hume](https://polvallverdu.github.io/pipedog/docs/services/tts/hume.md) | `HumeTTSService` | `@pipedog/hume` | Ported | | [Neuphonic](https://polvallverdu.github.io/pipedog/docs/services/tts/neuphonic.md) | `NeuphonicTTSService`, `NeuphonicHttpTTSService` | `@pipedog/neuphonic` | Ported | | [Speechify](https://polvallverdu.github.io/pipedog/docs/services/tts/speechify.md) | `SpeechifyHttpTTSService` | `@pipedog/speechify` | Ported | | [MiniMax](https://polvallverdu.github.io/pipedog/docs/services/tts/minimax.md) | `MiniMaxHttpTTSService` | `@pipedog/minimax` | Ported | | [Resemble AI](https://polvallverdu.github.io/pipedog/docs/services/tts/resembleai.md) | `ResembleAITTSService` | `@pipedog/resembleai` | Ported | | [Camb.ai](https://polvallverdu.github.io/pipedog/docs/services/tts/camb.md) | `CambTTSService` | `@pipedog/camb` | Ported | | [Async](https://polvallverdu.github.io/pipedog/docs/services/tts/asyncai.md) | `AsyncAITTSService`, `AsyncAIHttpTTSService` | `@pipedog/asyncai` | Ported | | [Bland](https://polvallverdu.github.io/pipedog/docs/services/tts/bland.md) | `BlandTTSService`, `BlandHttpTTSService` | `@pipedog/bland` | Ported | | [AssemblyAI](https://polvallverdu.github.io/pipedog/docs/services/tts/assemblyai.md) | `AssemblyAITTSService` | `@pipedog/assemblyai` | Ported | | [NVIDIA Riva](https://polvallverdu.github.io/pipedog/docs/services/tts/nvidia.md) | `NvidiaTTSService` | | Coming soon | ## Speech-to-speech [#speech-to-speech] Realtime models that take audio in and give audio out, in one service. [All speech-to-speech services](https://polvallverdu.github.io/pipedog/docs/services/s2s.md). | Provider | Classes | Package | Status | | ----------------------------------------------------------------- | ---------------------------- | ------------------- | ------ | | [OpenAI Realtime](https://polvallverdu.github.io/pipedog/docs/services/s2s/openai.md) | `OpenAIRealtimeLLMService` | `@pipedog/openai` | Ported | | [OpenAI Live](https://polvallverdu.github.io/pipedog/docs/services/s2s/openai-live.md) | `OpenAILiveLLMService` | `@pipedog/openai` | Ported | | [Gemini Live](https://polvallverdu.github.io/pipedog/docs/services/s2s/gemini-live.md) | `GeminiLiveLLMService` | `@pipedog/google` | Ported | | [Gemini Live on Vertex AI](https://polvallverdu.github.io/pipedog/docs/services/s2s/gemini-live-vertex.md) | `GeminiLiveVertexLLMService` | `@pipedog/google` | Ported | | [Inworld Realtime](https://polvallverdu.github.io/pipedog/docs/services/s2s/inworld.md) | `InworldRealtimeLLMService` | `@pipedog/inworld` | Ported | | [Ultravox](https://polvallverdu.github.io/pipedog/docs/services/s2s/ultravox.md) | `UltravoxRealtimeLLMService` | `@pipedog/ultravox` | Ported | | [xAI Grok Realtime](https://polvallverdu.github.io/pipedog/docs/services/s2s/grok.md) | `GrokRealtimeLLMService` | `@pipedog/xai` | Ported | | [AWS Nova Sonic](https://polvallverdu.github.io/pipedog/docs/services/s2s/aws.md) | `AWSNovaSonicLLMService` | `@pipedog/aws` | Ported | | [Azure Voice Live](https://polvallverdu.github.io/pipedog/docs/services/s2s/azure-voice-live.md) | `AzureVoiceLiveLLMService` | `@pipedog/azure` | Ported | | [Azure OpenAI Realtime](https://polvallverdu.github.io/pipedog/docs/services/s2s/azure-realtime.md) | `AzureRealtimeLLMService` | `@pipedog/azure` | Ported | ## Image generation [#image-generation] Generate images from text prompts. [All image generation services](https://polvallverdu.github.io/pipedog/docs/services/image.md). | Provider | Classes | Package | Status | | ------------------------------------------ | -------------------------- | ----------------- | ------ | | [OpenAI](https://polvallverdu.github.io/pipedog/docs/services/image/openai.md) | `OpenAIImageGenService` | `@pipedog/openai` | Ported | | [Google](https://polvallverdu.github.io/pipedog/docs/services/image/google.md) | `GoogleImageGenService` | `@pipedog/google` | Ported | | [Azure OpenAI](https://polvallverdu.github.io/pipedog/docs/services/image/azure.md) | `AzureImageGenServiceREST` | `@pipedog/azure` | Ported | ## Video avatars [#video-avatars] Animated avatars that speak the bot's audio. [All video avatars services](https://polvallverdu.github.io/pipedog/docs/services/video.md). | Provider | Classes | Package | Status | | ------------------------------------- | --------------------------------------- | ----------------- | ----------- | | [HeyGen](https://polvallverdu.github.io/pipedog/docs/services/video/heygen.md) | `HeyGenVideoService`, `HeyGenTransport` | `@pipedog/heygen` | Ported | | [Simli](https://polvallverdu.github.io/pipedog/docs/services/video/simli.md) | `SimliVideoService` | | Coming soon | | [Tavus](https://polvallverdu.github.io/pipedog/docs/services/video/tavus.md) | `TavusVideoService`, `TavusTransport` | | Blocked | ## Tools [#tools] Give the bot the tools of an MCP server (stdio, SSE or streamable HTTP) as function-call tools. | Service | Classes | Package | Status | | --------------------------------------- | ----------- | --------------------- | ------ | | [MCP client](https://polvallverdu.github.io/pipedog/docs/services/mcp-client.md) | `MCPClient` | `@pipedog/mcp-client` | Ported | ## Not listed here [#not-listed-here] Pipecat has more providers than this page lists. See [Other upstream providers](https://polvallverdu.github.io/pipedog/docs/services/other-upstream.md) for the rest, and the [porting status](https://polvallverdu.github.io/pipedog/docs/contributing/status.md) for module-level detail. # Vercel AI SDK > Run any Vercel AI SDK language model (OpenAI, Google, Anthropic, Ollama, the AI Gateway, ...) as the pipeline's LLM with AISDKLLMService. Source: https://polvallverdu.github.io/pipedog/docs/services/llm/ai-sdk/ ## Overview [#overview] `AISDKLLMService` runs a [Vercel AI SDK](https://ai-sdk.dev) language model as the pipeline's LLM service. The AI SDK gives one interface to many providers, so one service covers all of them: `@ai-sdk/openai`, `@ai-sdk/google`, `@ai-sdk/anthropic`, `@ai-sdk/mistral`, `@ai-sdk/amazon-bedrock`, Ollama and the other community providers, and model ID strings routed through the AI Gateway. * **`AISDKLLMService`**: an `LLMService` that streams the model's reply with the AI SDK's `streamText()`. It behaves like pipedog's native LLM services: it consumes `LLMContextFrame`s, pushes `LLMFullResponseStartFrame`, `LLMTextFrame`s and `LLMFullResponseEndFrame`, runs function calls through pipedog's function calling, and reports TTFB, processing time and token usage. This is a pipedog addition: Pipecat (Python) has no AI SDK service. `packages/ai-sdk/src/llm.ts` and the message adapter, `ai_sdk_adapter.ts`. The official and community providers, and their model IDs and options. ## Installation [#installation] `ai` (version 7) is a peer dependency, so you pick its version. Add the provider package you use, and `zod`, which `ai` requires. ```bash bun add @pipedog/ai-sdk ai zod @ai-sdk/google ``` ```ts import { AISDKLLMService } from "@pipedog/ai-sdk"; ``` ## Prerequisites [#prerequisites] * Whatever the provider needs. `@ai-sdk/google` reads `GOOGLE_GENERATIVE_AI_API_KEY`, or takes `createGoogleGenerativeAI({ apiKey })`. A model ID string uses the AI Gateway, which reads `AI_GATEWAY_API_KEY`. ## AISDKLLMService [#aisdkllmservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `model` | `LanguageModel` | yes | The AI SDK language model, e.g. `openai("gpt-4.1-mini")`, `google("gemini-2.5-flash")`, `anthropic("claude-sonnet-4-5")`, or a model ID string for the AI SDK's global provider (the AI Gateway by default). | | `provider` | `((modelId: string) => LanguageModel) \| null \| undefined` | no | Builds a model from a model ID, e.g. the provider instance (`google`, `openai`, `createOllama(...)`). With it, changing `settings.model` at runtime (`LLMUpdateSettingsFrame`) switches the model. | | `settings` | `AISDKLLMSettings \| null \| undefined` | no | Runtime-updatable settings. Given fields override the defaults. | | `tools` | `ToolSet \| null \| undefined` | no | AI SDK tools sent on every request alongside the context's tools, such as provider-defined tools (`google.tools.googleSearch({})`, `openai.tools.webSearch({})`). Your own functions belong in the context's `ToolsSchema`, so pipedog runs them. | | `maxRetries` | `number \| null \| undefined` | no | Maximum retries of a failed request (the AI SDK's `maxRetries`). Defaults to 2. | | `headers` | `Record \| null \| undefined` | no | Additional HTTP headers sent with each request. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `runInParallel` | `boolean \| undefined` | no | Whether to run function calls in parallel or sequentially. Defaults to true. | | `groupParallelTools` | `boolean \| undefined` | no | Whether to group parallel function calls so the LLM is triggered exactly once after all calls in the batch complete. When false, each function call result triggers the LLM independently as it arrives. Defaults to true. | | `functionCallTimeoutSecs` | `number \| null \| undefined` | no | Optional timeout in seconds for function calls. A call that runs past it is cancelled: its handler is cancelled (it gets a `CancelledError`), the call is settled as cancelled, and inference runs so the LLM can report that it didn't complete. | | `fileResolver` | `FileResolver \| null \| undefined` | no | Resolves file URLs in the context that this provider can't fetch itself (see `FileResolver`). Pass one configured with a `fileStorage` to resolve uploaded-file URLs (`new FileResolver({ fileStorage: runnerArgs.fileStorage })` when a runner hosts the bot), and share one instance across services that may consume the same files (e.g. services switched between mid-session) so downloads are reused. Defaults to a resolver that handles `data:` and `http(s)` URLs only. | ### Settings [#settings] `AISDKLLMService.Settings` (`AISDKLLMSettings`) holds the runtime-updatable settings. The common fields map to the AI SDK's call settings: `temperature`, `maxTokens` (`maxOutputTokens`), `topP`, `topK`, `frequencyPenalty`, `presencePenalty` and `seed`. `null`, the default, leaves a field to the provider. | Option | Type | Required | Description | |---|---|---|---| | `stopSequences` | `string[] \| NotGiven \| null \| undefined` | no | Sequences that stop generation. | | `providerOptions` | `NotGiven \| SharedV4ProviderOptions \| null \| undefined` | no | Provider-specific options, keyed by provider (the AI SDK's `providerOptions`), e.g. `{ google: { thinkingConfig: { thinkingBudget: 0 } } }`. | | `model` | `string \| NotGiven \| null \| undefined` | no | AI model identifier (e.g. `"gpt-4o"`, `"eleven_flash_v2_5"`). Defaults to `NOT_GIVEN` for delta mode. In store mode, set to a model string or `null` if the service has no model concept. | | `extra` | `Record \| undefined` | no | Catch-all for service-specific keys that have no declared field. | | `systemInstruction` | `string \| NotGiven \| null \| undefined` | no | System instruction/prompt for the model. | | `temperature` | `number \| NotGiven \| null \| undefined` | no | Sampling temperature. | | `maxTokens` | `number \| NotGiven \| null \| undefined` | no | Maximum tokens to generate. | | `topP` | `number \| NotGiven \| null \| undefined` | no | Nucleus sampling probability. | | `topK` | `number \| NotGiven \| null \| undefined` | no | Top-k sampling parameter. | | `frequencyPenalty` | `number \| NotGiven \| null \| undefined` | no | Frequency penalty. | | `presencePenalty` | `number \| NotGiven \| null \| undefined` | no | Presence penalty. | | `seed` | `number \| NotGiven \| null \| undefined` | no | Random seed for reproducibility. | | `filterIncompleteUserTurns` | `boolean \| NotGiven \| null \| undefined` | no | Enable LLM-based turn completion detection to suppress bot responses when the user was cut off mid-thought. Set by `LLMTurnCompletionUserTurnStopStrategy`, which is also what waits on the resulting verdict. Not meant to be set by application code; use `userTurnStrategies: new FilterIncompleteUserTurnStrategies()` on `LLMUserAggregatorParams` instead. | | `userTurnCompletionConfig` | `NotGiven \| UserTurnCompletionConfig \| null \| undefined` | no | Configuration for turn completion behavior when `filterIncompleteUserTurns` is enabled. Set by the user turn strategy; pass the config to `FilterIncompleteUserTurnStrategies({ config })`. | | `useProviderThinkingDefault` | `boolean \| NotGiven \| undefined` | no | Whether a model with no thinking configured runs at its provider's default. Services that otherwise send each model its lowest thinking setting, for the fastest responses, send none. `BackendLLMWorker` sets it on its LLM. Defaults to false. | `systemInstruction` is sent as the AI SDK's `instructions`. `extra` is merged into the `streamText()` call as is. ### Usage [#usage] ```ts title="llm.ts" import { google } from "@ai-sdk/google"; import { AISDKLLMService } from "@pipedog/ai-sdk"; const llm = new AISDKLLMService({ model: google("gemini-3.5-flash-lite"), settings: new AISDKLLMService.Settings({ systemInstruction: "You are a helpful voice assistant. Keep answers short.", temperature: 0.7, maxTokens: 1024, }), }); ``` ### Function calling [#function-calling] Tools come from the context's `ToolsSchema`, as with every LLM service. Each `FunctionSchema` becomes an AI SDK tool with a JSON schema and no `execute`, so the AI SDK hands the model's calls back to pipedog, which runs the registered handler (or the schema's own handler, or a direct function), adds the result to the context and asks the model again. ```ts title="tools.ts" import { FunctionSchema } from "@pipedog/core/adapters/llm/schemas/function_schema"; import { ToolsSchema } from "@pipedog/core/adapters/llm/schemas/tools_schema"; import { LLMContext } from "@pipedog/core/processors/aggregators/llm_context"; import type { FunctionCallParams } from "@pipedog/core/services/llm_service"; const weather = new FunctionSchema({ name: "get_weather", description: "Get the current weather for a city", properties: { city: { type: "string" } }, required: ["city"], }); // pipedog runs the handler and sends the result back to the model. llm.registerFunction("get_weather", async (params: FunctionCallParams) => { await params.resultCallback({ conditions: "sunny", celsius: 21 }); }); const context = new LLMContext({ tools: new ToolsSchema({ standardTools: [weather] }) }); ``` Provider-defined tools, such as Google Search grounding or OpenAI web search, go in the `tools` option. They are sent on every request and run on the provider's side. ### Switching models [#switching-models] ```ts title="switch.ts" import { LLMUpdateSettingsFrame } from "@pipedog/core/frames/frames"; // With `provider`, a model ID in a settings update switches the model. const switchable = new AISDKLLMService({ model: google("gemini-3.5-flash-lite"), provider: google, }); const update = new LLMUpdateSettingsFrame({ delta: new AISDKLLMService.Settings({ model: "gemini-3.5-flash" }), }); // Or switch to a model from any provider directly. switchable.setLanguageModel(google("gemini-3.5-flash")); ``` Without `provider`, a model ID in a settings update can't be turned into a model, so it is ignored with a warning (unless the model was given as an ID string in the first place). ### Provider options [#provider-options] Provider-specific options go in `providerOptions`, keyed by provider, as in the AI SDK: ```ts title="provider-options.ts" const thinkingOff = new AISDKLLMService({ model: google("gemini-3.5-flash-lite"), settings: new AISDKLLMService.Settings({ providerOptions: { google: { thinkingConfig: { thinkingBudget: 0 } } }, }), }); ``` ### Notes [#notes] * Consumes `LLMContextFrame`. For each response it pushes `LLMFullResponseStartFrame`, one `LLMTextFrame` per streamed text delta and `LLMFullResponseEndFrame`, even when the request fails. * Reasoning is pushed as `LLMThoughtStartFrame` / `LLMThoughtTextFrame` / `LLMThoughtEndFrame` and kept in the context, so providers that need it back (Anthropic thinking blocks) get it. * Provider metadata that comes with a tool call or the reply text, such as Gemini's thought signatures, is kept in the context and sent back with that output on later turns. * An initial `"system"` context message becomes the instructions when the service has no `systemInstruction`. Later `"system"` and `"developer"` messages are sent as user messages. * Images (`image_url`), files (`file_url`, `file_base64`) and audio (`input_audio`) in user messages become AI SDK image and file parts. File URLs are fetched by pipedog's file resolver and sent inline. * A failed request is pushed upstream as an `ErrorFrame` (`Error during completion: ...`). The AI SDK retries failed requests itself (`maxRetries`, 2 by default). * `runInference()` runs a one-shot `generateText()`. A `responseSchema` is passed as `Output.object()`, so the provider enforces it where it can. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/ai-sdk/package.json`: ✅ means the compat smoke test passes on that runtime. * An interruption cancels the response: the request's `abortSignal` is aborted and each stream part is a cancellation point. * The service uses no runtime-specific APIs. Whether a provider package runs on a given runtime is up to that package; the official ones use `fetch`. # Anthropic > Claude models with streaming, function calling, thinking and prompt caching through AnthropicLLMService. Source: https://polvallverdu.github.io/pipedog/docs/services/llm/anthropic/ ## Overview [#overview] `AnthropicLLMService` runs Claude models through the Anthropic Messages API. It streams text, calls functions, supports thinking and prompt caching, and accepts any client with the same `beta.messages.create()` shape. * **`AnthropicLLMService`**: the LLM service, with `AnthropicLLMSettings` for runtime-updatable settings and `AnthropicThinkingConfig` for thinking. Pipecat's documentation for the same service. `packages/anthropic/src/llm.ts` and `anthropic_adapter.ts`. The Messages API, models, thinking and prompt caching. ## Installation [#installation] ```bash bun add @pipedog/anthropic ``` ```ts import { AnthropicLLMService, AnthropicThinkingConfig } from "@pipedog/anthropic/llm"; ``` The package depends on the official `@anthropic-ai/sdk`, which is installed with it. ## Prerequisites [#prerequisites] * An Anthropic account and an API key from the [Anthropic Console](https://console.anthropic.com). * Pass the key as `apiKey`. The examples read it from `ANTHROPIC_API_KEY`. ## AnthropicLLMService [#anthropicllmservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | Anthropic API key for authentication. | | `settings` | `AnthropicLLMSettings \| null \| undefined` | no | Runtime-updatable settings for this service. Values given here override the service defaults. | | `client` | `AnthropicClient \| null \| undefined` | no | Optional custom Anthropic client instance. | | `retryTimeoutSecs` | `number \| null \| undefined` | no | Request timeout in seconds for retry logic. Defaults to 5.0. | | `retryOnTimeout` | `boolean \| null \| undefined` | no | Whether to retry the request once if it times out. Defaults to false. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `runInParallel` | `boolean \| undefined` | no | Whether to run function calls in parallel or sequentially. Defaults to true. | | `groupParallelTools` | `boolean \| undefined` | no | Whether to group parallel function calls so the LLM is triggered exactly once after all calls in the batch complete. When false, each function call result triggers the LLM independently as it arrives. Defaults to true. | | `functionCallTimeoutSecs` | `number \| null \| undefined` | no | Optional timeout in seconds for function calls. A call that runs past it is cancelled: its handler is cancelled (it gets a `CancelledError`), the call is settled as cancelled, and inference runs so the LLM can report that it didn't complete. | | `fileResolver` | `FileResolver \| null \| undefined` | no | Resolves file URLs in the context that this provider can't fetch itself (see `FileResolver`). Pass one configured with a `fileStorage` to resolve uploaded-file URLs (`new FileResolver({ fileStorage: runnerArgs.fileStorage })` when a runner hosts the bot), and share one instance across services that may consume the same files (e.g. services switched between mid-session) so downloads are reused. Defaults to a resolver that handles `data:` and `http(s)` URLs only. | `apiKey` is required even when you pass your own `client`, which doesn't use it. ### Settings [#settings] Runtime-updatable settings, passed as `settings: new AnthropicLLMService.Settings({...})`. Any setting you leave out keeps the default below. `null` means the field is not sent, so Anthropic uses its own default. | Setting | Type | Default | Description | | ---------------------------- | --------------------------------------------------------- | ----------------------------- | ----------------------------------------------------------------------- | | `model` | `string` | `"claude-sonnet-4-6"` | Claude model id. | | `systemInstruction` | `string \| null` | `null` | System prompt. | | `maxTokens` | `number \| null` | `4096` | Maximum tokens to generate. | | `temperature` | `number \| null` | `null` | Sampling temperature. Sent through `extra_body`. | | `topK` | `number \| null` | `null` | Top-k sampling. Sent through `extra_body`. | | `topP` | `number \| null` | `null` | Top-p sampling. Sent through `extra_body`. | | `enablePromptCaching` | `boolean` | `false` | Add cache-control markers to the system prompt and recent messages. | | `systemPromptCacheTtl` | `"5m" \| "1h" \| null` | `null` (Anthropic: 5 minutes) | Lifetime of the system prompt's cache entry when caching is on. | | `thinking` | `AnthropicThinkingConfig \| null` | `null` | Thinking configuration. See below. | | `effort` | `"low" \| "medium" \| "high" \| "xhigh" \| "max" \| null` | `null` | Sent as `output_config.effort`. `null` uses the model's default. | | `useProviderThinkingDefault` | `boolean` | `false` | Run at Anthropic's default when neither `thinking` nor `effort` is set. | | `extra` | `Record` | `{}` | Extra request parameters, merged into the request. | `frequencyPenalty`, `presencePenalty` and `seed` exist on the base `LLMSettings` and default to `null`; the Anthropic API does not use them. ### AnthropicThinkingConfig [#anthropicthinkingconfig] | Option | Type | Required | Description | |---|---|---|---| | `type` | `"adaptive" \| "between_tools" \| "disabled" \| "enabled" \| (string & {})` | yes | Thinking mode. "adaptive" lets the model decide when and how deeply to think; prefer it. "enabled" is legacy manual thinking, sized by `budgetTokens`: Claude 4.7 and later reject it, and Claude 4.5 and earlier accept only it. "disabled" turns thinking off on the models that allow it. "between_tools" is how Claude Sonnet 5.5, which rejects "disabled", runs without up-front thinking; no other model accepts it. | | `budgetTokens` | `number \| null \| undefined` | no | Maximum number of tokens for thinking. With today's models, the minimum is 1024. Required when type is "enabled", not allowed otherwise. | | `display` | `"omitted" \| "summarized" \| (string & {}) \| null \| undefined` | no | How thinking text comes back: "summarized" for readable thinking, which is what `LLMThoughtTextFrame` carries, or "omitted" for thinking blocks whose text is empty. Claude 4.7 and later default to "omitted", so set "summarized" there to keep those frames carrying text. Not allowed when type is "disabled". | When the model thinks, the service emits `LLMThoughtStartFrame`, `LLMThoughtTextFrame` and `LLMThoughtEndFrame`. Claude 4.7 and later return empty thinking text unless `display` is `"summarized"`. ### Usage [#usage] ```ts title="llm.ts" import { AnthropicLLMService, AnthropicThinkingConfig } from "@pipedog/anthropic/llm"; import { LLMUpdateSettingsFrame } from "@pipedog/core/frames/frames"; const llm = new AnthropicLLMService({ apiKey: process.env.ANTHROPIC_API_KEY ?? "", settings: new AnthropicLLMService.Settings({ model: "claude-sonnet-4-6", systemInstruction: "You are a helpful voice assistant. Keep answers short.", maxTokens: 1024, enablePromptCaching: true, }), }); llm.addEventHandler("on_completion_timeout", async () => { console.log("LLM completion timed out"); }); ``` Adaptive thinking with readable thinking text: ```ts const thinkingLlm = new AnthropicLLMService({ apiKey: process.env.ANTHROPIC_API_KEY ?? "", settings: new AnthropicLLMService.Settings({ model: "claude-opus-4-7", maxTokens: 16384, thinking: new AnthropicThinkingConfig({ type: "adaptive", display: "summarized" }), }), }); ``` Change settings mid-call with an `LLMUpdateSettingsFrame`: ```ts // Queue it on the worker, or push it from a processor upstream of the service. const update = new LLMUpdateSettingsFrame({ delta: new AnthropicLLMService.Settings({ temperature: 0.3, maxTokens: 512 }), }); ``` ### Event handlers [#event-handlers] Inherited from `LLMService`: | Event | Description | | ----------------------------- | -------------------------------------------------- | | `on_completion_timeout` | A completion request timed out. | | `on_function_calls_started` | Function calls arrived and are about to run. | | `on_function_calls_cancelled` | One or more running function calls were cancelled. | ### Notes [#notes] * Consumes `LLMContextFrame` and emits `LLMFullResponseStartFrame`, `LLMTextFrame`s and `LLMFullResponseEndFrame`. An `LLMEnablePromptCachingFrame` turns prompt caching on or off. * When neither `thinking` nor `effort` is set, the service sends the lowest thinking setting Anthropic documents for the model, for the fastest responses: `thinking: { type: "disabled" }` on most Claude 4 and 5 models, `"between_tools"` on Claude Sonnet 5.5, and `effort: "low"` on Claude Opus 5.5 and Fable models. It logs what it sent. A Claude model not in that list runs at Anthropic's default, with a warning. Set `useProviderThinkingDefault: true` to skip this. * Bedrock and Vertex model ids (`us.anthropic.claude-sonnet-5-...`, `claude-sonnet-5@...`) and dated snapshots are matched to their model's entry. * Models from the 4.6 generation on reject assistant prefill. For those, the service adds a trailing user message when the context ends with an assistant message. * `runInference` accepts a `responseSchema`. It is sent as `output_config.format` and needs a 4.5 model or later. * With `retryOnTimeout: true`, the first attempt is limited to `retryTimeoutSecs` (5 by default). If it times out, the request is retried once with no timeout. * The service logs a warning when Claude stops for a reason other than a normal end, such as `max_tokens` or a refusal. The text that arrived is still passed on. * Token usage metrics include cache creation, cache read and reasoning tokens. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/anthropic/package.json`: ✅ means the compat smoke test passes on that runtime. * `@anthropic-ai/sdk` is fetch-based and runs on Bun, Node, Deno and Cloudflare Workers. * `client` takes any object with a `beta.messages.create(params, { signal })` method, such as `AnthropicBedrock` or `AnthropicVertex` from Anthropic's Bedrock and Vertex SDKs (not dependencies of this package). * The Python SDK's `NOT_GIVEN` has no JS counterpart: sampling settings and `thinking` use `null` for "not sent". An explicit `null` sampling value is left out of the request, where upstream sends JSON `null`. * The JS SDK has no `extra_body` option. pipedog merges `extra_body` into the request JSON itself, as the Python SDK does. * The JS SDK has no `RequestTooLargeError`; a 413 is recognized by its status. * Not ported: the deprecated `AnthropicLLMService.InputParams` and the `model` / `params` constructor arguments. Use `settings`. # AWS AgentCore > Run an Amazon Bedrock AgentCore agent in the pipeline, in place of an LLM, with AWSAgentCoreProcessor. Source: https://polvallverdu.github.io/pipedog/docs/services/llm/aws-agentcore/ ## Overview [#overview] `AWSAgentCoreProcessor` sends the conversation to an agent hosted on Amazon Bedrock AgentCore Runtime and pushes the agent's replies as `LLMTextFrame`s, so it sits where an LLM service would: after the user context aggregator, before TTS. * On each `LLMContextFrame` it builds a payload from the context and calls `InvokeAgentRuntime`. * Each line the agent streams back is turned into text and pushed as an `LLMTextFrame`, between `LLMFullResponseStartFrame` and `LLMFullResponseEndFrame`. A response is closed when no new text arrives for 1 second. `packages/aws/src/agent_core.ts`. Deploying and invoking AgentCore Runtime agents. ## Installation [#installation] ```bash bun add @pipedog/aws ``` ```ts import { AWSAgentCoreProcessor } from "@pipedog/aws/agent_core"; ``` ## Prerequisites [#prerequisites] * An agent deployed to AgentCore Runtime, and its ARN. * An IAM identity allowed `bedrock-agentcore:InvokeAgentRuntime`. * Keys, in this order of precedence: 1. `awsAccessKey`, `awsSecretKey` and `awsSessionToken`. 2. The `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` and `AWS_SESSION_TOKEN` environment variables. 3. A `credentialsProvider` function, called for every request. The region comes from `awsRegion`, else `AWS_REGION`, else `us-east-1`. ## AWSAgentCoreProcessor [#awsagentcoreprocessor] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `agentArn` | `string` | yes | The Amazon Web Services Resource Name (ARN) of the agent. | | `awsAccessKey` | `string \| null \| undefined` | no | AWS access key ID. If null, falls back to environment variables and `credentialsProvider`. | | `awsSecretKey` | `string \| null \| undefined` | no | AWS secret access key. Same fallback behaviour as `awsAccessKey`. | | `awsSessionToken` | `string \| null \| undefined` | no | AWS session token for temporary credentials. | | `awsRegion` | `string \| null \| undefined` | no | AWS region. | | `contextToPayloadTransformer` | `((context: LLMContext) => string \| null) \| null \| undefined` | no | Optional function to transform LLMContext into AgentCore payload string. If null, uses `defaultContextToPayloadTransformer`. | | `responseToOutputTransformer` | `((responseLine: string) => string \| null) \| null \| undefined` | no | Optional function to extract output text from AgentCore response. If null, uses `defaultResponseToOutputTransformer`. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `credentialsProvider` | `AWSCredentialsProvider \| null \| undefined` | no | Keys to sign with when none are passed in or set in the environment. Called for every request or connection, so it can renew temporary credentials. | | `fetch` | `FetchLike \| null \| undefined` | no | `fetch` used for HTTP calls (defaults to the global `fetch`). | | `endpointUrl` | `string \| null \| undefined` | no | Override the service endpoint (e.g. a VPC endpoint or a local fake), `https://host[:port]`. | ### Usage [#usage] ```ts title="agent.ts" import { AWSAgentCoreProcessor } from "@pipedog/aws/agent_core"; import type { LLMContext } from "@pipedog/core/processors/aggregators/llm_context"; const agent = new AWSAgentCoreProcessor({ agentArn: "arn:aws:bedrock-agentcore:us-east-1:123456789012:runtime/my-agent", awsRegion: "us-east-1", }); // new Pipeline([transport.input(), stt, userAggregator, agent, tts, transport.output(), ...]) ``` By default the payload is `{"prompt": ""}`, from the last message when it is a user or system message, and each response line is read as `{"response": ""}`. Change both with the transformers: ```ts const customAgent = new AWSAgentCoreProcessor({ agentArn: "arn:aws:bedrock-agentcore:us-east-1:123456789012:runtime/my-agent", // Send the whole conversation instead of the last user message. contextToPayloadTransformer: (context: LLMContext) => JSON.stringify({ messages: context.getMessages() }), // The agent streams `data: {"text": "..."}` lines; the processor strips `data: `. responseToOutputTransformer: (line: string) => { try { return (JSON.parse(line) as { text?: string }).text ?? null; } catch { return null; } }, }); ``` A transformer that returns `null` skips the call (for the payload) or the line (for the output). ### Notes [#notes] * Each call gets a new `X-Amzn-Bedrock-AgentCore-Runtime-Session-Id`, as botocore generates one. * A streamed `text/event-stream` response is split into lines as they arrive, joining a line split across two chunks. Lines must start with `data: `, which is stripped before the output transformer sees them. Any other response is read whole and passed to the transformer once. * The `LLMContextFrame` is consumed. Other frames pass through unchanged. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/aws/package.json`: ✅ means the compat smoke test passes on that runtime. * Runs on Bun, Node, Deno and Cloudflare Workers: `POST /runtimes/{arn}/invocations` with `fetch`, signed with SigV4 on WebCrypto. * There is no default AWS credential chain. Use keys, the environment, or `credentialsProvider`. * `endpointUrl` and `fetch` replace the AgentCore host and the global `fetch`. # AWS Bedrock > Foundation models on Amazon Bedrock (Claude, Nova, Llama and others) through AWSBedrockLLMService and the Converse API. Source: https://polvallverdu.github.io/pipedog/docs/services/llm/aws/ ## Overview [#overview] `AWSBedrockLLMService` runs any Bedrock model that supports the Converse API. It streams text with ConverseStream, calls functions, supports prompt caching and reasoning effort, and reports token usage, including cache tokens. * **`AWSBedrockLLMService`**: the LLM service, with `AWSBedrockLLMSettings` for runtime-updatable settings. * **`AWSBedrockLLMAdapter`**: converts the universal `LLMContext` and tools to Converse messages, `system` blocks and `toolConfig`. Pipecat's documentation for the same service. `packages/aws/src/llm.ts` and `bedrock_adapter.ts`. The Converse API, models and prompt caching. ## Installation [#installation] ```bash bun add @pipedog/aws ``` ```ts import { AWSBedrockLLMService } from "@pipedog/aws/llm"; ``` No AWS SDK is needed: the service calls the Bedrock REST API with `fetch`, signed with SigV4. ## Prerequisites [#prerequisites] * An AWS account with access to the model in the Bedrock console, in the region you use. * An IAM identity allowed `bedrock:InvokeModel` and `bedrock:InvokeModelWithResponseStream`. * Keys, in this order of precedence: 1. `awsAccessKey`, `awsSecretKey` and `awsSessionToken`. 2. The `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` and `AWS_SESSION_TOKEN` environment variables. 3. A `credentialsProvider` function, called for every request. The region comes from `awsRegion`, else `AWS_REGION`, else `us-east-1`. ## AWSBedrockLLMService [#awsbedrockllmservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `awsAccessKey` | `string \| null \| undefined` | no | AWS access key ID. If null, falls back to the `AWS_ACCESS_KEY_ID` environment variable. | | `awsSecretKey` | `string \| null \| undefined` | no | AWS secret access key. Same fallback behaviour as `awsAccessKey`. | | `awsSessionToken` | `string \| null \| undefined` | no | AWS session token for temporary credentials. | | `awsRegion` | `string \| null \| undefined` | no | AWS region for the Bedrock service. | | `settings` | `AWSBedrockLLMSettings \| null \| undefined` | no | Runtime-updatable settings for this service. Values given here override the service defaults. | | `retryTimeoutSecs` | `number \| null \| undefined` | no | Request timeout in seconds for retry logic. Defaults to 5.0. | | `retryOnTimeout` | `boolean \| null \| undefined` | no | Whether to retry the request once if it times out. Defaults to false. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `runInParallel` | `boolean \| undefined` | no | Whether to run function calls in parallel or sequentially. Defaults to true. | | `groupParallelTools` | `boolean \| undefined` | no | Whether to group parallel function calls so the LLM is triggered exactly once after all calls in the batch complete. When false, each function call result triggers the LLM independently as it arrives. Defaults to true. | | `functionCallTimeoutSecs` | `number \| null \| undefined` | no | Optional timeout in seconds for function calls. A call that runs past it is cancelled: its handler is cancelled (it gets a `CancelledError`), the call is settled as cancelled, and inference runs so the LLM can report that it didn't complete. | | `fileResolver` | `FileResolver \| null \| undefined` | no | Resolves file URLs in the context that this provider can't fetch itself (see `FileResolver`). Pass one configured with a `fileStorage` to resolve uploaded-file URLs (`new FileResolver({ fileStorage: runnerArgs.fileStorage })` when a runner hosts the bot), and share one instance across services that may consume the same files (e.g. services switched between mid-session) so downloads are reused. Defaults to a resolver that handles `data:` and `http(s)` URLs only. | | `credentialsProvider` | `AWSCredentialsProvider \| null \| undefined` | no | Keys to sign with when none are passed in or set in the environment. Called for every request or connection, so it can renew temporary credentials. | | `fetch` | `FetchLike \| null \| undefined` | no | `fetch` used for HTTP calls (defaults to the global `fetch`). | | `endpointUrl` | `string \| null \| undefined` | no | Override the service endpoint (e.g. a VPC endpoint or a local fake), `https://host[:port]`. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new AWSBedrockLLMService.Settings({...})`. Any setting you leave out keeps the default below. `null` means the field is not sent. | Setting | Type | Default | Description | | ------------------------------ | ----------------------------------- | ---------------------------- | ---------------------------------------------------------------- | | `model` | `string` | `"us.amazon.nova-lite-v1:0"` | Bedrock model ID or inference profile. | | `systemInstruction` | `string \| null` | `null` | System prompt, sent as a `system` block. | | `maxTokens` | `number \| null` | `null` | `inferenceConfig.maxTokens`. | | `temperature` | `number \| null` | `null` | `inferenceConfig.temperature`. | | `topP` | `number \| null` | `null` | `inferenceConfig.topP`. | | `stopSequences` | `string[] \| null` | `null` | `inferenceConfig.stopSequences`. | | `latency` | `"standard" \| "optimized" \| null` | `null` | `performanceConfig.latency`. | | `effort` | `string \| null` | `null` | `outputConfig.effort` for reasoning models (`"low"` to `"max"`). | | `enablePromptCaching` | `boolean` | `false` | Add `cachePoint` markers to the system prompt and tools. | | `additionalModelRequestFields` | `Record` | `{}` | Model-specific fields, sent as `additionalModelRequestFields`. | ### Usage [#usage] ```ts title="llm.ts" import { AWSBedrockLLMService } from "@pipedog/aws/llm"; import { LLMUpdateSettingsFrame } from "@pipedog/core/frames/frames"; const llm = new AWSBedrockLLMService({ // Or leave these out: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, // AWS_SESSION_TOKEN and AWS_REGION are read from the environment. awsAccessKey: process.env.AWS_ACCESS_KEY_ID, awsSecretKey: process.env.AWS_SECRET_ACCESS_KEY, awsRegion: "us-east-1", settings: new AWSBedrockLLMService.Settings({ model: "us.anthropic.claude-haiku-4-5-20251001-v1:0", systemInstruction: "You are a helpful voice assistant. Keep answers short.", maxTokens: 1024, enablePromptCaching: true, }), }); ``` With temporary credentials from a role, SSO or the instance metadata service, pass a provider. `fromNodeProviderChain()` from `@aws-sdk/credential-providers` (Node only) caches and renews them: ```ts // Temporary credentials that renew themselves, e.g. from the AWS SDK's // provider chain: the function is called for every request. declare function fromNodeProviderChain(): () => Promise<{ accessKeyId: string; secretAccessKey: string; sessionToken?: string; }>; const roleLlm = new AWSBedrockLLMService({ awsRegion: "eu-west-1", credentialsProvider: fromNodeProviderChain(), }); ``` Change settings mid-call with an `LLMUpdateSettingsFrame`: ```ts // Queue it on the worker, or push it from a processor upstream of the service. const update = new LLMUpdateSettingsFrame({ delta: new AWSBedrockLLMService.Settings({ temperature: 0.3, maxTokens: 512 }), }); ``` ### Event handlers [#event-handlers] Inherited from `LLMService`: | Event | Description | | ----------------------------- | -------------------------------------------------- | | `on_completion_timeout` | A completion request timed out. | | `on_function_calls_started` | Function calls arrived and are about to run. | | `on_function_calls_cancelled` | One or more running function calls were cancelled. | ### Notes [#notes] * Consumes `LLMContextFrame` and emits `LLMFullResponseStartFrame`, `LLMTextFrame`s and `LLMFullResponseEndFrame`. * Reasoning the model returns in `reasoningContent` (Claude with extended thinking, Nova 2 with reasoning enabled) comes out as `LLMThoughtStartFrame`, `LLMThoughtTextFrame` and `LLMThoughtEndFrame`. Signed reasoning, such as Claude's, is kept in the context and sent back on later turns, so Claude keeps its reasoning across tool calls. * Amazon Nova (v1) models write their reasoning into the answer, in `...`, before a tool call. A `` block at the start of the response, or of any text between tool calls, is emitted as thought frames instead of `LLMTextFrame`s, so it isn't spoken. * When the context has tool calls in its history but no tools are configured, the service sends a no-op tool, because Bedrock rejects tool messages without a `toolConfig`. * For Claude models that reject assistant prefill, a trailing user message is added when the context ends with an assistant message. * `runInference` uses Converse (not streaming). Converse has no reply schema, so a `responseSchema` is accepted but not enforced. * With `retryOnTimeout: true`, the first attempt is limited to `retryTimeoutSecs` (5 by default). If it times out, the request is retried once with no timeout. * Errors are `AWSServiceError`s with the AWS error code (`ThrottlingException`, `ValidationException`, …) and the HTTP status. An exception sent inside the event stream has status 0. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/aws/package.json`: ✅ means the compat smoke test passes on that runtime. * Runs on Bun, Node, Deno and Cloudflare Workers: requests are signed with SigV4 on WebCrypto, and the `application/vnd.amazon.eventstream` response is decoded in the package. * There is no default AWS credential chain (profiles, IMDS, ECS roles, SSO). Use the keys, the environment, or `credentialsProvider`. * botocore's `client_config` is not ported, so there are no client-side retries of throttled or 5xx requests and no 300 s timeouts. Pass your own `fetch` to add them. * `endpointUrl` sends requests to another host (a VPC endpoint or a local fake); `fetch` replaces the global `fetch`. * Not ported: the deprecated `AWSBedrockLLMService.InputParams` and the `model`, `params` and `stopSequences` constructor arguments. Use `settings`. # Azure OpenAI > Streaming chat completions from OpenAI models deployed on Azure, with AzureLLMService. Source: https://polvallverdu.github.io/pipedog/docs/services/llm/azure/ ## Overview [#overview] * **`AzureLLMService`**: OpenAI chat models deployed in Azure AI Foundry (Azure OpenAI). It extends [`OpenAILLMService`](https://polvallverdu.github.io/pipedog/docs/services/llm/openai.md), so streaming, function calling, token usage metrics and `runInference` work the same way; only the client differs. The shape of `endpoint` picks the API surface: * An endpoint ending in **`/openai/v1`** uses Azure's v1 API, which tracks new features and needs no `api-version`. This is the one to use. * Any other endpoint (for example `https://my-resource.openai.azure.com`) goes through the dated API, with `api-version=2025-04-01-preview`, the last dated version Azure issued. Both surfaces accept an API key or a Microsoft Entra ID token. Pipecat's documentation for the same service. `packages/azure/src/llm.ts`. The v1 API and the dated API versions. ## Installation [#installation] ```bash bun add @pipedog/azure ``` ```ts import { AzureLLMService } from "@pipedog/azure/llm"; ``` `@pipedog/azure` depends on `@pipedog/openai` and the `openai` SDK, which it installs for you. ## Prerequisites [#prerequisites] * An Azure OpenAI (or Azure AI Foundry) resource with a chat model deployed. The deployment name is what you pass as `model`. * The resource's endpoint, and either its API key or a Microsoft Entra ID identity with access to it. The examples read the key from `AZURE_OPENAI_API_KEY`. ## AzureLLMService [#azurellmservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `endpoint` | `string` | yes | The Azure endpoint URL. Ending it in `/openai/v1` selects the v1 API surface. | | `tokenProvider` | `AzureTokenProvider \| null \| undefined` | no | Async callable supplying a Microsoft Entra ID bearer token, used instead of `apiKey`. | | `settings` | `OpenAILLMSettings \| null \| undefined` | no | Runtime-updatable settings. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `runInParallel` | `boolean \| undefined` | no | Whether to run function calls in parallel or sequentially. Defaults to true. | | `groupParallelTools` | `boolean \| undefined` | no | Whether to group parallel function calls so the LLM is triggered exactly once after all calls in the batch complete. When false, each function call result triggers the LLM independently as it arrives. Defaults to true. | | `functionCallTimeoutSecs` | `number \| null \| undefined` | no | Optional timeout in seconds for function calls. A call that runs past it is cancelled: its handler is cancelled (it gets a `CancelledError`), the call is settled as cancelled, and inference runs so the LLM can report that it didn't complete. | | `fileResolver` | `FileResolver \| null \| undefined` | no | Resolves file URLs in the context that this provider can't fetch itself (see `FileResolver`). Pass one configured with a `fileStorage` to resolve uploaded-file URLs (`new FileResolver({ fileStorage: runnerArgs.fileStorage })` when a runner hosts the bot), and share one instance across services that may consume the same files (e.g. services switched between mid-session) so downloads are reused. Defaults to a resolver that handles `data:` and `http(s)` URLs only. | | `apiKey` | `string \| null \| undefined` | no | OpenAI API key. If null, the SDK reads `OPENAI_API_KEY`. | | `baseUrl` | `string \| null \| undefined` | no | Custom base URL for the API. | | `organization` | `string \| null \| undefined` | no | OpenAI organization ID. | | `project` | `string \| null \| undefined` | no | OpenAI project ID. | | `defaultHeaders` | `Record \| null \| undefined` | no | Additional HTTP headers. | | `serviceTier` | `string \| null \| undefined` | no | Service tier to use (e.g., "auto", "flex", "priority"). | | `retryTimeoutSecs` | `number \| null \| undefined` | no | How long an inference may go without producing output before it is abandoned and re-issued, when `retryOnTimeout` is set. Defaults to 5.0 seconds. | | `retryOnTimeout` | `boolean \| null \| undefined` | no | Whether to re-issue the request once if the first attempt produces no output within `retryTimeoutSecs`. The retry is unbounded. | The other options (`serviceTier`, `retryOnTimeout`, `retryTimeoutSecs`, and the frame processor options) are the same as [`OpenAILLMService`](https://polvallverdu.github.io/pipedog/docs/services/llm/openai/#configuration). `baseUrl` is ignored: the endpoint sets it. ### Settings [#settings] `AzureLLMService.Settings` (`AzureLLMSettings`) has the same fields and defaults as [`OpenAILLMSettings`](https://polvallverdu.github.io/pipedog/docs/services/llm/openai/#settings). `model` defaults to `"gpt-4.1"`; on Azure it must be your deployment's name. ### Usage [#usage] ```ts title="llm.ts" import { AzureLLMService } from "@pipedog/azure/llm"; const llm = new AzureLLMService({ apiKey: process.env.AZURE_OPENAI_API_KEY ?? "", // Ending the endpoint in /openai/v1 selects Azure's v1 API surface. endpoint: "https://my-resource.openai.azure.com/openai/v1", settings: new AzureLLMService.Settings({ // On Azure the model is the deployment name. model: "my-gpt-4.1-deployment", systemInstruction: "You are a helpful assistant in a voice conversation.", temperature: 0.7, }), }); ``` With Microsoft Entra ID instead of a key. The token provider is called for every request: ```ts // Any async function that returns a Microsoft Entra ID bearer token for the // https://ai.azure.com/.default scope, for example // getBearerTokenProvider(new DefaultAzureCredential(), "https://ai.azure.com/.default") // from @azure/identity. declare const tokenProvider: () => Promise; const withEntra = new AzureLLMService({ tokenProvider, endpoint: "https://my-resource.openai.azure.com/openai/v1", settings: new AzureLLMService.Settings({ model: "my-gpt-4.1-deployment" }), }); ``` Through the dated API: ```ts // Any other endpoint shape goes through the dated API (api-version 2025-04-01-preview). const dated = new AzureLLMService({ apiKey: process.env.AZURE_OPENAI_API_KEY ?? "", endpoint: "https://my-resource.openai.azure.com", settings: new AzureLLMService.Settings({ model: "my-gpt-4.1-deployment" }), }); ``` ### Notes [#notes] * On the v1 surface the client is the plain `OpenAI` client pointed at the endpoint, with the key (or the token provider) as its API key. On the dated surface it is the SDK's `AzureOpenAI` client, which sends the key in the `api-key` header, adds `api-version`, and routes to `/openai/deployments//...`. * `apiKey` or `tokenProvider` is required; the constructor throws without either. With both, the token provider is used. * Event handlers, frames and metrics are those of `OpenAILLMService`. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/azure/package.json`: ✅ means the compat smoke test passes on that runtime. * The client is the official `openai` JS SDK over `fetch`, so the service works on Bun, Node, Deno and Cloudflare Workers. * `AzureTokenProvider` is any `() => Promise`. `getBearerTokenProvider()` from `@azure/identity` returns one; on Workers, where `@azure/identity`'s credentials don't run, fetch the token yourself (for example through a client-credentials request to Entra ID). * The deprecated `model` and `apiVersion` constructor options are not ported. Use `settings` for the model, and an `/openai/v1` endpoint instead of a dated version. # Baseten > BasetenLLMService streams chat completions from Baseten through its OpenAI-compatible API. Source: https://polvallverdu.github.io/pipedog/docs/services/llm/baseten/ {/* Generated by scripts/generate-providers.ts from data/providers.ts. */} ## Overview [#overview] Baseten's Model APIs serve open-weights models (GLM, Kimi, DeepSeek, Nemotron, gpt-oss) over an OpenAI-compatible API, and dedicated deployments run your own model on your own GPUs. * **`BasetenLLMService`**: an `OpenAILLMService` subclass with Baseten's base URL and default model. Pipecat's documentation for the same service. `packages/baseten/src/llm.ts`. Model APIs and deployments. ## Installation [#installation] ```bash bun add @pipedog/baseten ``` It depends on `@pipedog/openai`, which is installed with it. ```ts import { BasetenLLMService } from "@pipedog/baseten/llm"; ``` ## Prerequisites [#prerequisites] * A Baseten account and an API key from the [Baseten console](https://app.baseten.co/). * Pass the key as `apiKey`. It is required, and the service never falls back to `OPENAI_API_KEY`. The examples read it from `BASETEN_API_KEY`. ## BasetenLLMService [#basetenllmservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | The API key for accessing Baseten's inference API. | | `baseUrl` | `string \| null \| undefined` | no | The base URL for the API. Defaults to `"https://inference.baseten.co/v1"`. | | `settings` | `BasetenLLMSettings \| null \| undefined` | no | Runtime-updatable settings; values override the built-in defaults. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `runInParallel` | `boolean \| undefined` | no | Whether to run function calls in parallel or sequentially. Defaults to true. | | `groupParallelTools` | `boolean \| undefined` | no | Whether to group parallel function calls so the LLM is triggered exactly once after all calls in the batch complete. When false, each function call result triggers the LLM independently as it arrives. Defaults to true. | | `functionCallTimeoutSecs` | `number \| null \| undefined` | no | Optional timeout in seconds for function calls. A call that runs past it is cancelled: its handler is cancelled (it gets a `CancelledError`), the call is settled as cancelled, and inference runs so the LLM can report that it didn't complete. | | `fileResolver` | `FileResolver \| null \| undefined` | no | Resolves file URLs in the context that this provider can't fetch itself (see `FileResolver`). Pass one configured with a `fileStorage` to resolve uploaded-file URLs (`new FileResolver({ fileStorage: runnerArgs.fileStorage })` when a runner hosts the bot), and share one instance across services that may consume the same files (e.g. services switched between mid-session) so downloads are reused. Defaults to a resolver that handles `data:` and `http(s)` URLs only. | | `organization` | `string \| null \| undefined` | no | OpenAI organization ID. | | `project` | `string \| null \| undefined` | no | OpenAI project ID. | | `defaultHeaders` | `Record \| null \| undefined` | no | Additional HTTP headers. | | `serviceTier` | `string \| null \| undefined` | no | Service tier to use (e.g., "auto", "flex", "priority"). | | `retryTimeoutSecs` | `number \| null \| undefined` | no | How long an inference may go without producing output before it is abandoned and re-issued, when `retryOnTimeout` is set. Defaults to 5.0 seconds. | | `retryOnTimeout` | `boolean \| null \| undefined` | no | Whether to re-issue the request once if the first attempt produces no output within `retryTimeoutSecs`. The retry is unbounded. | The other options (`organization`, `project`, `defaultHeaders`, `serviceTier`, `retryOnTimeout`, `retryTimeoutSecs`, function calling) are `OpenAILLMService`'s. See [OpenAI](https://polvallverdu.github.io/pipedog/docs/services/llm/openai/#configuration). | | | | ------------- | ----------------------------------------------------------- | | Base URL | `https://inference.baseten.co/v1` | | Default model | `zai-org/GLM-5.3` | | Models to try | `zai-org/GLM-5.3`. See the provider's catalog for the rest. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new BasetenLLMService.Settings({...})` (`BasetenLLMSettings`). They are [OpenAI's settings](https://polvallverdu.github.io/pipedog/docs/services/llm/openai/#settings), with these defaults: | Setting | Type | Default | Description | | ------- | -------- | ------------------- | ----------- | | `model` | `string` | `"zai-org/GLM-5.3"` | Model name. | Every other setting keeps OpenAI's default. Unset sampling settings are left out of the request, so the provider's own defaults apply. `extra` carries provider-specific request fields. ### Usage [#usage] ```ts title="llm.ts" import { LLMUpdateSettingsFrame } from "@pipedog/core/frames/frames"; const llm = new BasetenLLMService({ apiKey: process.env.BASETEN_API_KEY ?? "", settings: new BasetenLLMService.Settings({ model: "zai-org/GLM-5.3", systemInstruction: "You are a helpful assistant in a voice conversation.", temperature: 0.7, }), }); ``` A dedicated deployment: point `baseUrl` at its `/sync/v1` URL and set `model` to the model it serves. ```ts // Replace {model_id} with your deployment's ID. const dedicated = new BasetenLLMService({ apiKey: process.env.BASETEN_API_KEY ?? "", baseUrl: "https://model-{model_id}.api.baseten.co/environments/production/sync/v1", settings: new BasetenLLMService.Settings({ model: "Qwen/Qwen2.5-3B-Instruct" }), }); ``` Change settings mid-call with an `LLMUpdateSettingsFrame`. The next completion uses them: ```ts // Queue it on the worker, or push it from a processor upstream of the service. const update = new LLMUpdateSettingsFrame({ delta: new BasetenLLMService.Settings({ temperature: 0.2, maxTokens: 300 }), }); ``` ### Notes [#notes] * Runs a completion for each `LLMContextFrame`, between an `LLMFullResponseStartFrame` and an `LLMFullResponseEndFrame`, and streams the reply as `LLMTextFrame`s, as [`OpenAILLMService`](https://polvallverdu.github.io/pipedog/docs/services/llm/openai.md) does. Function calls, token usage metrics and `runInference` work the same way. * "developer" messages are sent as "user" messages, since the API doesn't support that role (`supportsDeveloperRole` is `false`). * Baseten's message roles are system, user, assistant and tool, so "developer" messages are sent as "user" messages. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/baseten/package.json`: ✅ means the compat smoke test passes on that runtime. * The client is the official `openai` JS SDK (through `@pipedog/openai`), which talks through `fetch`. It works on Bun, Node, Deno and Cloudflare Workers. * The deprecated `model` constructor option is not ported. Use `settings`. * Tested against a local OpenAI-compatible mock server, not against Baseten itself. # Cerebras > CerebrasLLMService streams chat completions from Cerebras through its OpenAI-compatible API. Source: https://polvallverdu.github.io/pipedog/docs/services/llm/cerebras/ {/* Generated by scripts/generate-providers.ts from data/providers.ts. */} ## Overview [#overview] Cerebras serves open models with fast inference on its own hardware. * **`CerebrasLLMService`**: an `OpenAILLMService` subclass with Cerebras's base URL and default model. Pipecat's documentation for the same service. `packages/cerebras/src/llm.ts`. The chat completions API reference. ## Installation [#installation] ```bash bun add @pipedog/cerebras ``` It depends on `@pipedog/openai`, which is installed with it. ```ts import { CerebrasLLMService } from "@pipedog/cerebras/llm"; ``` ## Prerequisites [#prerequisites] * A Cerebras account and an API key from the [Cerebras platform](https://cloud.cerebras.ai/). * Pass the key as `apiKey`. It is required, and the service never falls back to `OPENAI_API_KEY`. The examples read it from `CEREBRAS_API_KEY`. ## CerebrasLLMService [#cerebrasllmservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `apiKey` | `string` | yes | The API key for accessing Cerebras's API. | | `baseUrl` | `string \| null \| undefined` | no | The base URL for the API. Defaults to `"https://api.cerebras.ai/v1"`. | | `settings` | `CerebrasLLMSettings \| null \| undefined` | no | Runtime-updatable settings; values override the built-in defaults. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `runInParallel` | `boolean \| undefined` | no | Whether to run function calls in parallel or sequentially. Defaults to true. | | `groupParallelTools` | `boolean \| undefined` | no | Whether to group parallel function calls so the LLM is triggered exactly once after all calls in the batch complete. When false, each function call result triggers the LLM independently as it arrives. Defaults to true. | | `functionCallTimeoutSecs` | `number \| null \| undefined` | no | Optional timeout in seconds for function calls. A call that runs past it is cancelled: its handler is cancelled (it gets a `CancelledError`), the call is settled as cancelled, and inference runs so the LLM can report that it didn't complete. | | `fileResolver` | `FileResolver \| null \| undefined` | no | Resolves file URLs in the context that this provider can't fetch itself (see `FileResolver`). Pass one configured with a `fileStorage` to resolve uploaded-file URLs (`new FileResolver({ fileStorage: runnerArgs.fileStorage })` when a runner hosts the bot), and share one instance across services that may consume the same files (e.g. services switched between mid-session) so downloads are reused. Defaults to a resolver that handles `data:` and `http(s)` URLs only. | | `organization` | `string \| null \| undefined` | no | OpenAI organization ID. | | `project` | `string \| null \| undefined` | no | OpenAI project ID. | | `defaultHeaders` | `Record \| null \| undefined` | no | Additional HTTP headers. | | `serviceTier` | `string \| null \| undefined` | no | Service tier to use (e.g., "auto", "flex", "priority"). | | `retryTimeoutSecs` | `number \| null \| undefined` | no | How long an inference may go without producing output before it is abandoned and re-issued, when `retryOnTimeout` is set. Defaults to 5.0 seconds. | | `retryOnTimeout` | `boolean \| null \| undefined` | no | Whether to re-issue the request once if the first attempt produces no output within `retryTimeoutSecs`. The retry is unbounded. | The other options (`organization`, `project`, `defaultHeaders`, `serviceTier`, `retryOnTimeout`, `retryTimeoutSecs`, function calling) are `OpenAILLMService`'s. See [OpenAI](https://polvallverdu.github.io/pipedog/docs/services/llm/openai/#configuration). | | | | ------------- | -------------------------------------------------------- | | Base URL | `https://api.cerebras.ai/v1` | | Default model | `gpt-oss-120b` | | Models to try | `gpt-oss-120b`. See the provider's catalog for the rest. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new CerebrasLLMService.Settings({...})` (`CerebrasLLMSettings`). They are [OpenAI's settings](https://polvallverdu.github.io/pipedog/docs/services/llm/openai/#settings), with these defaults: | Setting | Type | Default | Description | | ------- | -------- | ---------------- | ----------- | | `model` | `string` | `"gpt-oss-120b"` | Model name. | Every other setting keeps OpenAI's default. Unset sampling settings are left out of the request, so the provider's own defaults apply. `extra` carries provider-specific request fields. ### Usage [#usage] ```ts title="llm.ts" import { LLMUpdateSettingsFrame } from "@pipedog/core/frames/frames"; const llm = new CerebrasLLMService({ apiKey: process.env.CEREBRAS_API_KEY ?? "", settings: new CerebrasLLMService.Settings({ model: "gpt-oss-120b", systemInstruction: "You are a helpful assistant in a voice conversation.", temperature: 0.7, }), }); ``` Cerebras-only request fields, such as `reasoning_effort`, go in `extra`: ```ts const reasoning = new CerebrasLLMService({ apiKey: process.env.CEREBRAS_API_KEY ?? "", settings: new CerebrasLLMService.Settings({ extra: { reasoning_effort: "low" } }), }); ``` Change settings mid-call with an `LLMUpdateSettingsFrame`. The next completion uses them: ```ts // Queue it on the worker, or push it from a processor upstream of the service. const update = new LLMUpdateSettingsFrame({ delta: new CerebrasLLMService.Settings({ temperature: 0.2, maxTokens: 300 }), }); ``` ### Notes [#notes] * Runs a completion for each `LLMContextFrame`, between an `LLMFullResponseStartFrame` and an `LLMFullResponseEndFrame`, and streams the reply as `LLMTextFrame`s, as [`OpenAILLMService`](https://polvallverdu.github.io/pipedog/docs/services/llm/openai.md) does. Function calls, token usage metrics and `runInference` work the same way. * "developer" messages are sent as is: the API supports that role. * Cerebras honors `max_tokens`, `max_completion_tokens` and the penalties, so they are sent when set. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/cerebras/package.json`: ✅ means the compat smoke test passes on that runtime. * The client is the official `openai` JS SDK (through `@pipedog/openai`), which talks through `fetch`. It works on Bun, Node, Deno and Cloudflare Workers. * The deprecated `model` constructor option is not ported. Use `settings`. * Tested against a local OpenAI-compatible mock server, not against Cerebras itself. # Cloudflare Workers AI > WorkersAILLMService: Workers AI chat models through Cloudflare's OpenAI-compatible endpoint. Source: https://polvallverdu.github.io/pipedog/docs/services/llm/cloudflare/ ## Overview [#overview] **`WorkersAILLMService`** runs Workers AI's chat models (Llama, GLM, Kimi, Qwen, gpt-oss and others) through Workers AI's OpenAI-compatible endpoint. It is an `OpenAILLMService` subclass, like the other [OpenAI-compatible providers](https://polvallverdu.github.io/pipedog/docs/services/llm/openai.md): streaming, function calling and usage metrics work the same way. This package is pipedog's own: Pipecat has no Workers AI services. `packages/cloudflare/src/llm.ts`. Workers AI's `/v1/chat/completions` endpoint. The text generation models, and which take function calls. ## Installation [#installation] ```bash bun add @pipedog/cloudflare ``` ```ts import { WorkersAILLMService } from "@pipedog/cloudflare/llm"; ``` ## Prerequisites [#prerequisites] * A Cloudflare account ID and an [API token](https://dash.cloudflare.com/profile/api-tokens) with the Workers AI permission. The examples read them from `CLOUDFLARE_ACCOUNT_ID` and `CLOUDFLARE_API_TOKEN`. * The service needs them inside a Worker too: the `AI` binding answers chat requests in Workers AI's own format, not OpenAI's, so the service calls the REST endpoint with `fetch`, which works from a Worker like from any other runtime. ## Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `accountId` | `string \| null \| undefined` | no | Cloudflare account ID. Required unless `client` has one. | | `apiToken` | `string \| null \| undefined` | no | Cloudflare API token with Workers AI access. Required unless `client` has one. | | `client` | `WorkersAIClient \| null \| undefined` | no | A REST `WorkersAIClient` to take the account ID, token and base URL from. | | `baseUrl` | `string \| null \| undefined` | no | The full OpenAI-compatible base URL. Defaults to `https://api.cloudflare.com/client/v4/accounts/{accountId}/ai/v1`; set it to go through an AI Gateway, for instance. | | `settings` | `OpenAILLMSettings \| null \| undefined` | no | Runtime-updatable settings; values override the built-in defaults. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `runInParallel` | `boolean \| undefined` | no | Whether to run function calls in parallel or sequentially. Defaults to true. | | `groupParallelTools` | `boolean \| undefined` | no | Whether to group parallel function calls so the LLM is triggered exactly once after all calls in the batch complete. When false, each function call result triggers the LLM independently as it arrives. Defaults to true. | | `functionCallTimeoutSecs` | `number \| null \| undefined` | no | Optional timeout in seconds for function calls. A call that runs past it is cancelled: its handler is cancelled (it gets a `CancelledError`), the call is settled as cancelled, and inference runs so the LLM can report that it didn't complete. | | `fileResolver` | `FileResolver \| null \| undefined` | no | Resolves file URLs in the context that this provider can't fetch itself (see `FileResolver`). Pass one configured with a `fileStorage` to resolve uploaded-file URLs (`new FileResolver({ fileStorage: runnerArgs.fileStorage })` when a runner hosts the bot), and share one instance across services that may consume the same files (e.g. services switched between mid-session) so downloads are reused. Defaults to a resolver that handles `data:` and `http(s)` URLs only. | | `organization` | `string \| null \| undefined` | no | OpenAI organization ID. | | `project` | `string \| null \| undefined` | no | OpenAI project ID. | | `defaultHeaders` | `Record \| null \| undefined` | no | Additional HTTP headers. | | `serviceTier` | `string \| null \| undefined` | no | Service tier to use (e.g., "auto", "flex", "priority"). | | `retryTimeoutSecs` | `number \| null \| undefined` | no | How long an inference may go without producing output before it is abandoned and re-issued, when `retryOnTimeout` is set. Defaults to 5.0 seconds. | | `retryOnTimeout` | `boolean \| null \| undefined` | no | Whether to re-issue the request once if the first attempt produces no output within `retryTimeoutSecs`. The retry is unbounded. | The base URL is `https://api.cloudflare.com/client/v4/accounts/{accountId}/ai/v1`. A REST `WorkersAIClient` can be passed as `client` instead of `accountId` and `apiToken`. ## Settings [#settings] The [OpenAI settings](https://polvallverdu.github.io/pipedog/docs/services/llm/openai/#settings) (`model`, `systemInstruction`, `temperature`, `maxTokens`, ...), passed as `settings: new WorkersAILLMService.Settings({...})`. `model` defaults to `"@cf/meta/llama-3.3-70b-instruct-fp8-fast"`. Others to try: `"@cf/zai-org/glm-4.7-flash"`, `"@cf/meta/llama-4-scout-17b-16e-instruct"`, `"@cf/openai/gpt-oss-20b"`. ## Usage [#usage] ```ts title="llm.ts" import { WorkersAILLMService } from "@pipedog/cloudflare/llm"; import { LLMUpdateSettingsFrame } from "@pipedog/core/frames/frames"; const llm = new WorkersAILLMService({ accountId: process.env.CLOUDFLARE_ACCOUNT_ID ?? "", apiToken: process.env.CLOUDFLARE_API_TOKEN ?? "", settings: new WorkersAILLMService.Settings({ model: "@cf/meta/llama-3.3-70b-instruct-fp8-fast", systemInstruction: "You are a helpful assistant in a voice conversation.", temperature: 0.7, }), }); ``` Through an [AI Gateway](https://developers.cloudflare.com/ai-gateway/), for caching, logs and rate limits, set `baseUrl` to the gateway's Workers AI URL: ```ts // Through an AI Gateway: the gateway's Workers AI OpenAI-compatible URL. const viaGateway = new WorkersAILLMService({ apiToken: process.env.CLOUDFLARE_API_TOKEN ?? "", baseUrl: `https://gateway.ai.cloudflare.com/v1/${process.env.CLOUDFLARE_ACCOUNT_ID}/my-gateway/workers-ai/v1`, }); ``` Change settings mid-call with an `LLMUpdateSettingsFrame`: ```ts // Queue it on the worker, or push it from a processor upstream of the service. const update = new LLMUpdateSettingsFrame({ delta: new WorkersAILLMService.Settings({ temperature: 0.2, maxTokens: 300 }), }); ``` ## Notes [#notes] * Not every Workers AI model takes function calls; check the model page before giving it tools. * The same package has Workers AI [speech-to-text](https://polvallverdu.github.io/pipedog/docs/services/stt/cloudflare.md), [text-to-speech](https://polvallverdu.github.io/pipedog/docs/services/tts/cloudflare.md) and Smart Turn v2 ([turn detection](https://polvallverdu.github.io/pipedog/docs/concepts/turn-detection/#smart-turn-on-workers-ai)). For a whole agent on Cloudflare, see [Voice agent fully on Cloudflare](https://polvallverdu.github.io/pipedog/docs/deployment/cloudflare-workers-ai.md). ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/cloudflare/package.json`: ✅ means the compat smoke test passes on that runtime. * The client is the official `openai` JS SDK (through `@pipedog/openai`), which talks through `fetch`. It works on Bun, Node, Deno and Cloudflare Workers. * Tested against a local OpenAI-compatible mock server, not against Workers AI itself. # Crusoe > CrusoeLLMService streams chat completions from Crusoe through its OpenAI-compatible API. Source: https://polvallverdu.github.io/pipedog/docs/services/llm/crusoe/ {/* Generated by scripts/generate-providers.ts from data/providers.ts. */} ## Overview [#overview] Crusoe Cloud's Managed Inference serves open models through an OpenAI-compatible API. * **`CrusoeLLMService`**: an `OpenAILLMService` subclass with Crusoe's base URL and default model. Pipecat's documentation for the same service. `packages/crusoe/src/llm.ts`. Crusoe Cloud and its docs. ## Installation [#installation] ```bash bun add @pipedog/crusoe ``` It depends on `@pipedog/openai`, which is installed with it. ```ts import { CrusoeLLMService } from "@pipedog/crusoe/llm"; ``` ## Prerequisites [#prerequisites] * A Crusoe Cloud account and an API key from [Crusoe](https://crusoe.ai/). * Pass the key as `apiKey`. It is required, and the service never falls back to `OPENAI_API_KEY`. The examples read it from `CRUSOE_API_KEY`. ## CrusoeLLMService [#crusoellmservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | The API key for accessing Crusoe's Managed Inference API. | | `baseUrl` | `string \| null \| undefined` | no | The base URL for the API. Defaults to `"https://api.inference.crusoecloud.com/v1/"`. | | `settings` | `CrusoeLLMSettings \| null \| undefined` | no | Runtime-updatable settings; values override the built-in defaults. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `runInParallel` | `boolean \| undefined` | no | Whether to run function calls in parallel or sequentially. Defaults to true. | | `groupParallelTools` | `boolean \| undefined` | no | Whether to group parallel function calls so the LLM is triggered exactly once after all calls in the batch complete. When false, each function call result triggers the LLM independently as it arrives. Defaults to true. | | `functionCallTimeoutSecs` | `number \| null \| undefined` | no | Optional timeout in seconds for function calls. A call that runs past it is cancelled: its handler is cancelled (it gets a `CancelledError`), the call is settled as cancelled, and inference runs so the LLM can report that it didn't complete. | | `fileResolver` | `FileResolver \| null \| undefined` | no | Resolves file URLs in the context that this provider can't fetch itself (see `FileResolver`). Pass one configured with a `fileStorage` to resolve uploaded-file URLs (`new FileResolver({ fileStorage: runnerArgs.fileStorage })` when a runner hosts the bot), and share one instance across services that may consume the same files (e.g. services switched between mid-session) so downloads are reused. Defaults to a resolver that handles `data:` and `http(s)` URLs only. | | `organization` | `string \| null \| undefined` | no | OpenAI organization ID. | | `project` | `string \| null \| undefined` | no | OpenAI project ID. | | `defaultHeaders` | `Record \| null \| undefined` | no | Additional HTTP headers. | | `serviceTier` | `string \| null \| undefined` | no | Service tier to use (e.g., "auto", "flex", "priority"). | | `retryTimeoutSecs` | `number \| null \| undefined` | no | How long an inference may go without producing output before it is abandoned and re-issued, when `retryOnTimeout` is set. Defaults to 5.0 seconds. | | `retryOnTimeout` | `boolean \| null \| undefined` | no | Whether to re-issue the request once if the first attempt produces no output within `retryTimeoutSecs`. The retry is unbounded. | The other options (`organization`, `project`, `defaultHeaders`, `serviceTier`, `retryOnTimeout`, `retryTimeoutSecs`, function calling) are `OpenAILLMService`'s. See [OpenAI](https://polvallverdu.github.io/pipedog/docs/services/llm/openai/#configuration). | | | | ------------- | --------------------------------------------------------------- | | Base URL | `https://api.inference.crusoecloud.com/v1/` | | Default model | `openai/gpt-oss-120b` | | Models to try | `openai/gpt-oss-120b`. See the provider's catalog for the rest. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new CrusoeLLMService.Settings({...})` (`CrusoeLLMSettings`). They are [OpenAI's settings](https://polvallverdu.github.io/pipedog/docs/services/llm/openai/#settings), with these defaults: | Setting | Type | Default | Description | | ------- | -------- | ----------------------- | ----------- | | `model` | `string` | `"openai/gpt-oss-120b"` | Model name. | Every other setting keeps OpenAI's default. Unset sampling settings are left out of the request, so the provider's own defaults apply. `extra` carries provider-specific request fields. ### Usage [#usage] ```ts title="llm.ts" import { LLMUpdateSettingsFrame } from "@pipedog/core/frames/frames"; import { CrusoeLLMService } from "@pipedog/crusoe/llm"; const llm = new CrusoeLLMService({ apiKey: process.env.CRUSOE_API_KEY ?? "", settings: new CrusoeLLMService.Settings({ model: "openai/gpt-oss-120b", systemInstruction: "You are a helpful assistant in a voice conversation.", temperature: 0.7, }), }); ``` Change settings mid-call with an `LLMUpdateSettingsFrame`. The next completion uses them: ```ts // Queue it on the worker, or push it from a processor upstream of the service. const update = new LLMUpdateSettingsFrame({ delta: new CrusoeLLMService.Settings({ temperature: 0.2, maxTokens: 300 }), }); ``` ### Notes [#notes] * Runs a completion for each `LLMContextFrame`, between an `LLMFullResponseStartFrame` and an `LLMFullResponseEndFrame`, and streams the reply as `LLMTextFrame`s, as [`OpenAILLMService`](https://polvallverdu.github.io/pipedog/docs/services/llm/openai.md) does. Function calls, token usage metrics and `runInference` work the same way. * "developer" messages are sent as "user" messages, since the API doesn't support that role (`supportsDeveloperRole` is `false`). ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/crusoe/package.json`: ✅ means the compat smoke test passes on that runtime. * The client is the official `openai` JS SDK (through `@pipedog/openai`), which talks through `fetch`. It works on Bun, Node, Deno and Cloudflare Workers. * The deprecated `model` constructor option is not ported. Use `settings`. * Tested against a local OpenAI-compatible mock server, not against Crusoe itself. # DeepSeek > DeepSeekLLMService streams chat completions from DeepSeek through its OpenAI-compatible API. Source: https://polvallverdu.github.io/pipedog/docs/services/llm/deepseek/ {/* Generated by scripts/generate-providers.ts from data/providers.ts. */} ## Overview [#overview] DeepSeek's chat models, with control over their thinking mode. * **`DeepSeekLLMService`**: an `OpenAILLMService` subclass with DeepSeek's base URL and default model. It uses `DeepSeekLLMAdapter`, which adds an empty `reasoning_content` to assistant messages that have none, which DeepSeek requires in thinking mode once a tool call is involved. Pipecat's documentation for the same service. `packages/deepseek/src/llm.ts` and `deepseek_adapter.ts`. The chat completion API reference. ## Installation [#installation] ```bash bun add @pipedog/deepseek ``` It depends on `@pipedog/openai`, which is installed with it. ```ts import { DeepSeekLLMService } from "@pipedog/deepseek/llm"; import { DeepSeekLLMAdapter } from "@pipedog/deepseek/deepseek_adapter"; ``` ## Prerequisites [#prerequisites] * A DeepSeek account and an API key from the [DeepSeek platform](https://platform.deepseek.com/). * Pass the key as `apiKey`. It is required, and the service never falls back to `OPENAI_API_KEY`. The examples read it from `DEEPSEEK_API_KEY`. ## DeepSeekLLMService [#deepseekllmservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | The API key for accessing DeepSeek's API. | | `baseUrl` | `string \| null \| undefined` | no | The base URL for DeepSeek API. Defaults to `"https://api.deepseek.com/v1"`. | | `settings` | `DeepSeekLLMSettings \| null \| undefined` | no | Runtime-updatable settings; values override the built-in defaults. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `runInParallel` | `boolean \| undefined` | no | Whether to run function calls in parallel or sequentially. Defaults to true. | | `groupParallelTools` | `boolean \| undefined` | no | Whether to group parallel function calls so the LLM is triggered exactly once after all calls in the batch complete. When false, each function call result triggers the LLM independently as it arrives. Defaults to true. | | `functionCallTimeoutSecs` | `number \| null \| undefined` | no | Optional timeout in seconds for function calls. A call that runs past it is cancelled: its handler is cancelled (it gets a `CancelledError`), the call is settled as cancelled, and inference runs so the LLM can report that it didn't complete. | | `fileResolver` | `FileResolver \| null \| undefined` | no | Resolves file URLs in the context that this provider can't fetch itself (see `FileResolver`). Pass one configured with a `fileStorage` to resolve uploaded-file URLs (`new FileResolver({ fileStorage: runnerArgs.fileStorage })` when a runner hosts the bot), and share one instance across services that may consume the same files (e.g. services switched between mid-session) so downloads are reused. Defaults to a resolver that handles `data:` and `http(s)` URLs only. | | `organization` | `string \| null \| undefined` | no | OpenAI organization ID. | | `project` | `string \| null \| undefined` | no | OpenAI project ID. | | `defaultHeaders` | `Record \| null \| undefined` | no | Additional HTTP headers. | | `serviceTier` | `string \| null \| undefined` | no | Service tier to use (e.g., "auto", "flex", "priority"). | | `retryTimeoutSecs` | `number \| null \| undefined` | no | How long an inference may go without producing output before it is abandoned and re-issued, when `retryOnTimeout` is set. Defaults to 5.0 seconds. | | `retryOnTimeout` | `boolean \| null \| undefined` | no | Whether to re-issue the request once if the first attempt produces no output within `retryTimeoutSecs`. The retry is unbounded. | The other options (`organization`, `project`, `defaultHeaders`, `serviceTier`, `retryOnTimeout`, `retryTimeoutSecs`, function calling) are `OpenAILLMService`'s. See [OpenAI](https://polvallverdu.github.io/pipedog/docs/services/llm/openai/#configuration). | | | | ------------- | ---------------------------------------------------------- | | Base URL | `https://api.deepseek.com/v1` | | Default model | `deepseek-flash` | | Models to try | `deepseek-flash`. See the provider's catalog for the rest. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new DeepSeekLLMService.Settings({...})` (`DeepSeekLLMSettings`). They are [OpenAI's settings](https://polvallverdu.github.io/pipedog/docs/services/llm/openai/#settings), with these defaults: | Setting | Type | Default | Description | | ---------- | -------------------------------- | ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | | `model` | `string` | `"deepseek-flash"` | Model name. | | `thinking` | `DeepSeekThinkingConfig \| null` | `{ type: "disabled" }` | Thinking mode, sent as `thinking` in the request body. `null` leaves it to DeepSeek's default. A plain `{ type }` object is converted. | Every other setting keeps OpenAI's default. Unset sampling settings are left out of the request, so the provider's own defaults apply. `extra` carries provider-specific request fields. ### Usage [#usage] ```ts title="llm.ts" import { LLMUpdateSettingsFrame } from "@pipedog/core/frames/frames"; import { DeepSeekLLMService } from "@pipedog/deepseek/llm"; const llm = new DeepSeekLLMService({ apiKey: process.env.DEEPSEEK_API_KEY ?? "", settings: new DeepSeekLLMService.Settings({ model: "deepseek-flash", systemInstruction: "You are a helpful assistant in a voice conversation.", temperature: 0.7, }), }); ``` Thinking is off by default, because DeepSeek's V4 models otherwise reason before every answer, which delays the first spoken token. Turn it on with `DeepSeekLLMService.ThinkingConfig`: ```ts const thinking = new DeepSeekLLMService({ apiKey: process.env.DEEPSEEK_API_KEY ?? "", settings: new DeepSeekLLMService.Settings({ thinking: new DeepSeekLLMService.ThinkingConfig({ type: "enabled" }), }), }); ``` Change settings mid-call with an `LLMUpdateSettingsFrame`. The next completion uses them: ```ts // Queue it on the worker, or push it from a processor upstream of the service. const update = new LLMUpdateSettingsFrame({ delta: new DeepSeekLLMService.Settings({ temperature: 0.2, maxTokens: 300 }), }); ``` ### Notes [#notes] * Runs a completion for each `LLMContextFrame`, between an `LLMFullResponseStartFrame` and an `LLMFullResponseEndFrame`, and streams the reply as `LLMTextFrame`s, as [`OpenAILLMService`](https://polvallverdu.github.io/pipedog/docs/services/llm/openai.md) does. Function calls, token usage metrics and `runInference` work the same way. * "developer" messages are sent as "user" messages, since the API doesn't support that role (`supportsDeveloperRole` is `false`). * Messages go through `DeepSeekLLMAdapter`, which adds an empty `reasoning_content` to assistant messages that have none, which DeepSeek requires in thinking mode once a tool call is involved. * `seed`, `max_completion_tokens` and `service_tier` are not sent: DeepSeek doesn't support them. * DeepSeek can't enforce a response schema (`supportsResponseSchema` is `false`), so `runInference` ignores `responseSchema`, with a warning. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/deepseek/package.json`: ✅ means the compat smoke test passes on that runtime. * The client is the official `openai` JS SDK (through `@pipedog/openai`), which talks through `fetch`. It works on Bun, Node, Deno and Cloudflare Workers. * The deprecated `model` constructor option is not ported. Use `settings`. * Tested against a local OpenAI-compatible mock server, not against DeepSeek itself. # Fireworks AI > FireworksLLMService streams chat completions from Fireworks AI through its OpenAI-compatible API. Source: https://polvallverdu.github.io/pipedog/docs/services/llm/fireworks/ {/* Generated by scripts/generate-providers.ts from data/providers.ts. */} ## Overview [#overview] Fireworks AI serves open and proprietary models through an OpenAI-compatible API. * **`FireworksLLMService`**: an `OpenAILLMService` subclass with Fireworks AI's base URL and default model. Pipecat's documentation for the same service. `packages/fireworks/src/llm.ts`. The chat completions API reference. ## Installation [#installation] ```bash bun add @pipedog/fireworks ``` It depends on `@pipedog/openai`, which is installed with it. ```ts import { FireworksLLMService } from "@pipedog/fireworks/llm"; ``` ## Prerequisites [#prerequisites] * A Fireworks AI account and an API key from [Fireworks](https://fireworks.ai/). * Pass the key as `apiKey`. It is required, and the service never falls back to `OPENAI_API_KEY`. The examples read it from `FIREWORKS_API_KEY`. ## FireworksLLMService [#fireworksllmservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `apiKey` | `string` | yes | The API key for accessing Fireworks AI. | | `baseUrl` | `string \| null \| undefined` | no | The base URL for Fireworks API. Defaults to `"https://api.fireworks.ai/inference/v1"`. | | `settings` | `FireworksLLMSettings \| null \| undefined` | no | Runtime-updatable settings; values override the built-in defaults. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `runInParallel` | `boolean \| undefined` | no | Whether to run function calls in parallel or sequentially. Defaults to true. | | `groupParallelTools` | `boolean \| undefined` | no | Whether to group parallel function calls so the LLM is triggered exactly once after all calls in the batch complete. When false, each function call result triggers the LLM independently as it arrives. Defaults to true. | | `functionCallTimeoutSecs` | `number \| null \| undefined` | no | Optional timeout in seconds for function calls. A call that runs past it is cancelled: its handler is cancelled (it gets a `CancelledError`), the call is settled as cancelled, and inference runs so the LLM can report that it didn't complete. | | `fileResolver` | `FileResolver \| null \| undefined` | no | Resolves file URLs in the context that this provider can't fetch itself (see `FileResolver`). Pass one configured with a `fileStorage` to resolve uploaded-file URLs (`new FileResolver({ fileStorage: runnerArgs.fileStorage })` when a runner hosts the bot), and share one instance across services that may consume the same files (e.g. services switched between mid-session) so downloads are reused. Defaults to a resolver that handles `data:` and `http(s)` URLs only. | | `organization` | `string \| null \| undefined` | no | OpenAI organization ID. | | `project` | `string \| null \| undefined` | no | OpenAI project ID. | | `defaultHeaders` | `Record \| null \| undefined` | no | Additional HTTP headers. | | `serviceTier` | `string \| null \| undefined` | no | Service tier to use (e.g., "auto", "flex", "priority"). | | `retryTimeoutSecs` | `number \| null \| undefined` | no | How long an inference may go without producing output before it is abandoned and re-issued, when `retryOnTimeout` is set. Defaults to 5.0 seconds. | | `retryOnTimeout` | `boolean \| null \| undefined` | no | Whether to re-issue the request once if the first attempt produces no output within `retryTimeoutSecs`. The retry is unbounded. | The other options (`organization`, `project`, `defaultHeaders`, `serviceTier`, `retryOnTimeout`, `retryTimeoutSecs`, function calling) are `OpenAILLMService`'s. See [OpenAI](https://polvallverdu.github.io/pipedog/docs/services/llm/openai/#configuration). | | | | ------------- | -------------------------------------------------------------------------------------------- | | Base URL | `https://api.fireworks.ai/inference/v1` | | Default model | `accounts/fireworks/models/nemotron-3-ultra-nvfp4` | | Models to try | `accounts/fireworks/models/nemotron-3-ultra-nvfp4`. See the provider's catalog for the rest. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new FireworksLLMService.Settings({...})` (`FireworksLLMSettings`). They are [OpenAI's settings](https://polvallverdu.github.io/pipedog/docs/services/llm/openai/#settings), with these defaults: | Setting | Type | Default | Description | | ------- | -------- | ---------------------------------------------------- | ----------- | | `model` | `string` | `"accounts/fireworks/models/nemotron-3-ultra-nvfp4"` | Model name. | Every other setting keeps OpenAI's default. Unset sampling settings are left out of the request, so the provider's own defaults apply. `extra` carries provider-specific request fields. ### Usage [#usage] ```ts title="llm.ts" import { LLMUpdateSettingsFrame } from "@pipedog/core/frames/frames"; import { FireworksLLMService } from "@pipedog/fireworks/llm"; const llm = new FireworksLLMService({ apiKey: process.env.FIREWORKS_API_KEY ?? "", settings: new FireworksLLMService.Settings({ model: "accounts/fireworks/models/nemotron-3-ultra-nvfp4", systemInstruction: "You are a helpful assistant in a voice conversation.", temperature: 0.7, }), }); ``` Change settings mid-call with an `LLMUpdateSettingsFrame`. The next completion uses them: ```ts // Queue it on the worker, or push it from a processor upstream of the service. const update = new LLMUpdateSettingsFrame({ delta: new FireworksLLMService.Settings({ temperature: 0.2, maxTokens: 300 }), }); ``` ### Notes [#notes] * Runs a completion for each `LLMContextFrame`, between an `LLMFullResponseStartFrame` and an `LLMFullResponseEndFrame`, and streams the reply as `LLMTextFrame`s, as [`OpenAILLMService`](https://polvallverdu.github.io/pipedog/docs/services/llm/openai.md) does. Function calls, token usage metrics and `runInference` work the same way. * "developer" messages are sent as is: the API supports that role. * Model names use the `accounts/fireworks/models/` prefix. * `seed`, `max_completion_tokens`, `stream_options` and `service_tier` are not sent. Without `stream_options`, token usage is reported only when Fireworks includes it anyway. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/fireworks/package.json`: ✅ means the compat smoke test passes on that runtime. * The client is the official `openai` JS SDK (through `@pipedog/openai`), which talks through `fetch`. It works on Bun, Node, Deno and Cloudflare Workers. * The deprecated `model` constructor option is not ported. Use `settings`. * Tested against a local OpenAI-compatible mock server, not against Fireworks AI itself. # Google Vertex AI > Gemini models on Vertex AI with GoogleVertexLLMService, authenticated with a service account or Application Default Credentials. Source: https://polvallverdu.github.io/pipedog/docs/services/llm/google-vertex/ ## Overview [#overview] `GoogleVertexLLMService` runs Gemini models through Vertex AI instead of the Gemini API. It extends [`GoogleLLMService`](https://polvallverdu.github.io/pipedog/docs/services/llm/google.md), so streaming, tools, thinking, safety settings and grounding work the same way. Only authentication and the endpoint differ. * **`GoogleVertexLLMService`**: Gemini on Vertex AI, with a Google Cloud project, a location and service account credentials or Application Default Credentials (ADC). Pipecat's documentation for the same service. `packages/google/src/vertex/llm.ts`. The Vertex AI generative inference API. ## Installation [#installation] ```bash bun add @pipedog/google ``` ```ts import { GoogleVertexLLMService } from "@pipedog/google/vertex/llm"; ``` ## Prerequisites [#prerequisites] * A Google Cloud project with the Vertex AI API enabled. * A service account with access to Vertex AI, as a key file or a JSON string, or ADC (`gcloud auth application-default login`, or the attached service account on GCP). * The examples read `GOOGLE_CLOUD_PROJECT_ID`, and `GOOGLE_APPLICATION_CREDENTIALS` (a key file path) or `GOOGLE_CREDENTIALS_JSON` (the key itself). ## GoogleVertexLLMService [#googlevertexllmservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `credentials` | `string \| null \| undefined` | no | JSON string of service account credentials. | | `credentialsPath` | `string \| null \| undefined` | no | Path to the service account JSON file. | | `location` | `string \| undefined` | no | Location for the Vertex AI endpoint. Defaults to "global", the only location that serves the Gemini 3 series; the regional endpoints serve the 2.5 series. | | `projectId` | `string` | yes | Google Cloud project ID. | | `settings` | `GoogleVertexLLMSettings \| null \| undefined` | no | Runtime-updatable settings for this service. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `runInParallel` | `boolean \| undefined` | no | Whether to run function calls in parallel or sequentially. Defaults to true. | | `groupParallelTools` | `boolean \| undefined` | no | Whether to group parallel function calls so the LLM is triggered exactly once after all calls in the batch complete. When false, each function call result triggers the LLM independently as it arrives. Defaults to true. | | `functionCallTimeoutSecs` | `number \| null \| undefined` | no | Optional timeout in seconds for function calls. A call that runs past it is cancelled: its handler is cancelled (it gets a `CancelledError`), the call is settled as cancelled, and inference runs so the LLM can report that it didn't complete. | | `fileResolver` | `FileResolver \| null \| undefined` | no | Resolves file URLs in the context that this provider can't fetch itself (see `FileResolver`). Pass one configured with a `fileStorage` to resolve uploaded-file URLs (`new FileResolver({ fileStorage: runnerArgs.fileStorage })` when a runner hosts the bot), and share one instance across services that may consume the same files (e.g. services switched between mid-session) so downloads are reused. Defaults to a resolver that handles `data:` and `http(s)` URLs only. | | `tools` | `Record[] \| null \| undefined` | no | List of available tools/functions. | | `toolConfig` | `Record \| null \| undefined` | no | Configuration for tool usage. | | `httpOptions` | `HttpOptions \| null \| undefined` | no | HTTP options for the client. | | `streamIdleTimeoutSecs` | `number \| null \| undefined` | no | Gap allowed between streamed chunks, in seconds. `null` waits forever. Default 20. | | `retryTimeoutSecs` | `number \| null \| undefined` | no | First-chunk wait before re-issuing, in seconds. Default 5. | | `retryOnTimeout` | `boolean \| null \| undefined` | no | Whether to re-issue the request once on a first-chunk timeout. Default false. | | `imageDecoder` | `ImageDecoder \| null \| undefined` | no | Decodes inline images to pixels (upstream: Pillow). | ### Settings [#settings] `GoogleVertexLLMService.Settings` has the same fields and defaults as [`GoogleLLMService.Settings`](https://polvallverdu.github.io/pipedog/docs/services/llm/google/#settings): the default model is `"gemini-3.6-flash"`, `maxTokens` is `4096`, and the other settings start as `null`. ### Usage [#usage] ```ts title="vertex.ts" import { GoogleVertexLLMService } from "@pipedog/google/vertex/llm"; const llm = new GoogleVertexLLMService({ credentialsPath: process.env.GOOGLE_APPLICATION_CREDENTIALS ?? "", projectId: process.env.GOOGLE_CLOUD_PROJECT_ID ?? "", settings: new GoogleVertexLLMService.Settings({ model: "gemini-3.6-flash", systemInstruction: "You are a helpful voice assistant.", }), }); ``` With the service account key as a JSON string: ```ts // A service account key as a JSON string, e.g. from a secret store. const fromJson = new GoogleVertexLLMService({ credentials: process.env.GOOGLE_CREDENTIALS_JSON ?? "", projectId: process.env.GOOGLE_CLOUD_PROJECT_ID ?? "", }); ``` With Application Default Credentials and a regional endpoint: ```ts // No credentials option: Application Default Credentials, resolved on the // first request. Gemini 2.5 models are served from regional endpoints. const regional = new GoogleVertexLLMService({ projectId: process.env.GOOGLE_CLOUD_PROJECT_ID ?? "", location: "us-east4", settings: new GoogleVertexLLMService.Settings({ model: "gemini-2.5-flash" }), }); ``` ### Authentication [#authentication] The service picks credentials in this order: 1. `credentials`: a service account key as a JSON string. It is parsed in the constructor. 2. `credentialsPath`: the path to a key file, read by the client. 3. Neither: Application Default Credentials. All three go to `@google/genai` as `googleAuthOptions` with the `cloud-platform` scope, and google-auth-library inside the SDK loads the key file, finds ADC and refreshes the access token as it expires. Upstream loads the credentials and refreshes a token in the constructor, so missing ADC fails there with "No valid credentials provided.". Here loading a file, finding ADC and refreshing a token are asynchronous, so they happen on the first request, and missing credentials surface as an error from that request. ### Event handlers [#event-handlers] The same as [`GoogleLLMService`](https://polvallverdu.github.io/pipedog/docs/services/llm/google/#event-handlers). ### Notes [#notes] * Passing `apiKey` throws: Vertex AI takes `credentials`, `credentialsPath` or ADC. * `projectId` is required. `location` defaults to `"global"`, the only location that serves the Gemini 3 series. Regional endpoints (`"us-east4"`, `"us-central1"`, ...) serve the Gemini 2.5 series. * The Vertex adapter also accepts `gs://` file URLs in the context: Vertex reads them through its own IAM. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/google/package.json`: ✅ means the compat smoke test passes on that runtime. Authentication goes through google-auth-library, which only the Node build of `@google/genai` supports. `GoogleVertexLLMService` runs on Bun, Node and Deno, not on Cloudflare Workers. On Workers, use [`GoogleLLMService`](https://polvallverdu.github.io/pipedog/docs/services/llm/google.md) with a Gemini API key. The deprecated upstream `model`, `params` and `systemInstruction` constructor options are not ported; use `settings`. # Google Gemini > Streaming Gemini chat completions with GoogleLLMService, with thinking, safety settings and Google Search grounding. Source: https://polvallverdu.github.io/pipedog/docs/services/llm/google/ ## Overview [#overview] `GoogleLLMService` runs Gemini models through the Gemini API (Google AI Studio keys). It converts an `LLMContext` to Gemini's `Content`/`Part` format, streams the reply and handles function calls. * **`GoogleLLMService`**: streaming Gemini inference with tools, thinking (`GoogleThinkingConfig`), safety settings, image output and search grounding. * **`LLMSearchResponseFrame`** (`frames.ts`): pushed when a reply carries grounding metadata, for example from the Google Search tool. * **`GoogleRTVIProcessor` / `GoogleRTVIObserver`** (`rtvi.ts`): an RTVI processor whose observer also forwards `LLMSearchResponseFrame`s to the client. For Vertex AI, see [Google Vertex AI](https://polvallverdu.github.io/pipedog/docs/services/llm/google-vertex.md). For speech-to-speech, see [Gemini Live](https://polvallverdu.github.io/pipedog/docs/services/s2s/gemini-live.md). Pipecat's documentation for the same service. `packages/google/src/llm.ts`, `frames.ts` and `rtvi.ts`. Models, thinking, safety settings and grounding. ## Installation [#installation] ```bash bun add @pipedog/google ``` ```ts import { GoogleLLMService, GoogleThinkingConfig } from "@pipedog/google/llm"; import { LLMSearchResponseFrame } from "@pipedog/google/frames"; import { GoogleRTVIProcessor } from "@pipedog/google/rtvi"; ``` ## Prerequisites [#prerequisites] * A Gemini API key from [Google AI Studio](https://aistudio.google.com/apikey). * Pass the key as `apiKey`. The examples read it from `GOOGLE_API_KEY`. ## GoogleLLMService [#googlellmservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `runInParallel` | `boolean \| undefined` | no | Whether to run function calls in parallel or sequentially. Defaults to true. | | `apiKey` | `string` | yes | Google AI API key for authentication. | | `settings` | `GoogleLLMSettings \| null \| undefined` | no | Runtime-updatable settings for this service. | | `tools` | `Record[] \| null \| undefined` | no | List of available tools/functions. | | `toolConfig` | `Record \| null \| undefined` | no | Configuration for tool usage. | | `httpOptions` | `HttpOptions \| null \| undefined` | no | HTTP options for the client. | | `streamIdleTimeoutSecs` | `number \| null \| undefined` | no | Gap allowed between streamed chunks, in seconds. `null` waits forever. Default 20. | | `retryTimeoutSecs` | `number \| null \| undefined` | no | First-chunk wait before re-issuing, in seconds. Default 5. | | `retryOnTimeout` | `boolean \| null \| undefined` | no | Whether to re-issue the request once on a first-chunk timeout. Default false. | | `imageDecoder` | `ImageDecoder \| null \| undefined` | no | Decodes inline images to pixels (upstream: Pillow). | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `groupParallelTools` | `boolean \| undefined` | no | Whether to group parallel function calls so the LLM is triggered exactly once after all calls in the batch complete. When false, each function call result triggers the LLM independently as it arrives. Defaults to true. | | `functionCallTimeoutSecs` | `number \| null \| undefined` | no | Optional timeout in seconds for function calls. A call that runs past it is cancelled: its handler is cancelled (it gets a `CancelledError`), the call is settled as cancelled, and inference runs so the LLM can report that it didn't complete. | | `fileResolver` | `FileResolver \| null \| undefined` | no | Resolves file URLs in the context that this provider can't fetch itself (see `FileResolver`). Pass one configured with a `fileStorage` to resolve uploaded-file URLs (`new FileResolver({ fileStorage: runnerArgs.fileStorage })` when a runner hosts the bot), and share one instance across services that may consume the same files (e.g. services switched between mid-session) so downloads are reused. Defaults to a resolver that handles `data:` and `http(s)` URLs only. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new GoogleLLMService.Settings({...})`. Any setting you leave out keeps the default below. Extra Gemini `GenerateContentConfig` fields can go in `extra`, with the Python SDK's snake_case keys (they are converted to camelCase when the request is sent). | Setting | Type | Default | Description | | ---------------------------- | ------------------------------ | -------------------- | ------------------------------------------------------------------------------ | | `model` | `string` | `"gemini-3.6-flash"` | Gemini model. | | `systemInstruction` | `string \| null` | `null` | System prompt. | | `maxTokens` | `number \| null` | `4096` | Sent as `maxOutputTokens`. | | `temperature` | `number \| null` | `null` | Sampling temperature. | | `topK` | `number \| null` | `null` | Top-k sampling. | | `topP` | `number \| null` | `null` | Nucleus sampling. | | `seed` | `number \| null` | `null` | Random seed. Gemini treats it as best effort. | | `frequencyPenalty` | `number \| null` | `null` | Frequency penalty. Not every Gemini model accepts it. | | `presencePenalty` | `number \| null` | `null` | Presence penalty. Not every Gemini model accepts it. | | `thinking` | `GoogleThinkingConfig \| null` | `null` | Thinking configuration. See below. | | `safetySettings` | `SafetySetting[] \| null` | `null` | Content safety filters. Categories you leave out keep the Gemini API defaults. | | `useProviderThinkingDefault` | `boolean` | `false` | Send no thinking default; run at Google's default. | `GoogleThinkingConfig` takes `thinkingLevel` (Gemini 3 and later: `"minimal"`, `"low"`, `"medium"`, `"high"`), `thinkingBudget` (Gemini 2.5: a token count, `0` off, `-1` dynamic) and `includeThoughts`. Set a level or a budget, not both. When `thinking` is `null`, the service sends the lowest thinking setting Google documents for the model, for the fastest replies, and logs what it sent. For example `gemini-2.5-flash` gets `thinking_budget: 0`, `gemini-2.5-pro` gets `128`, `gemini-3.6-flash` gets `thinking_level: "minimal"` and `gemini-3.7-flash` gets `"low"`. Models not in that table (including aliases such as `gemini-flash-latest`) run at Google's default. ### Usage [#usage] ```ts title="llm.ts" import { LLMUpdateSettingsFrame } from "@pipedog/core/frames/frames"; import { GoogleLLMService, GoogleThinkingConfig } from "@pipedog/google/llm"; import { GoogleRTVIProcessor } from "@pipedog/google/rtvi"; const llm = new GoogleLLMService({ apiKey: process.env.GOOGLE_API_KEY ?? "", settings: new GoogleLLMService.Settings({ model: "gemini-3.6-flash", systemInstruction: "You are a helpful voice assistant. Keep answers short.", temperature: 0.7, }), }); ``` Thinking, for Gemini 3 and Gemini 2.5 models: ```ts // Gemini 3 models take a thinking level. const gemini3 = new GoogleLLMService({ apiKey: process.env.GOOGLE_API_KEY ?? "", settings: new GoogleLLMService.Settings({ model: "gemini-3.6-flash", thinking: new GoogleThinkingConfig({ thinkingLevel: "low", includeThoughts: true }), }), }); // Gemini 2.5 models take a token budget (0 turns thinking off). const gemini25 = new GoogleLLMService({ apiKey: process.env.GOOGLE_API_KEY ?? "", settings: new GoogleLLMService.Settings({ model: "gemini-2.5-flash", thinking: new GoogleThinkingConfig({ thinkingBudget: 1024 }), }), }); ``` Google Search grounding, with the RTVI processor that forwards search results to the client: ```ts // Tools in the SDK's camelCase shape are sent as given. const grounded = new GoogleLLMService({ apiKey: process.env.GOOGLE_API_KEY ?? "", tools: [{ googleSearch: {} }], }); // Pass it to PipelineWorker as `rtviProcessor` to forward each // LLMSearchResponseFrame to the client as a "bot-llm-search-response" message. const rtvi = new GoogleRTVIProcessor(); ``` Change settings mid-call with an `LLMUpdateSettingsFrame`: ```ts // Queue it on the worker, or push it from a processor upstream of the service. const update = new LLMUpdateSettingsFrame({ delta: new GoogleLLMService.Settings({ temperature: 0.2, maxTokens: 1024 }), }); ``` ### Event handlers [#event-handlers] Inherited from `LLMService`. | Event | Description | | ----------------------------- | ---------------------------------------------------------------------- | | `on_completion_timeout` | The stream stalled past `streamIdleTimeoutSecs` (or the retry window). | | `on_function_calls_started` | Function calls were received and are about to run. | | `on_function_calls_cancelled` | In-progress function calls were cancelled. | ### Notes [#notes] * Emits `LLMFullResponseStartFrame`, `LLMTextFrame`s and `LLMFullResponseEndFrame`. Thoughts (with `includeThoughts`) arrive whole, each as an `LLMThoughtStartFrame`, `LLMThoughtTextFrame` and `LLMThoughtEndFrame`. * When a reply has grounding metadata, the service pushes an `LLMSearchResponseFrame` with the reply text (`searchResult`), the search entry point's `renderedContent`, and `origins` (site URI, title and the supported text segments with confidence scores). `GoogleRTVIObserver` sends it to the client as a `bot-llm-search-response` message. * Gemini thought signatures are saved into the context (through an `LLMMessagesAppendFrame`), so they go back to the model on the next turn. * Inline image output becomes an `AssistantImageRawFrame`, but only with an `imageDecoder`. The default decoder throws, so without one an image reply surfaces as a pushed error. * `streamIdleTimeoutSecs` (default 20) bounds the gap between chunks, including the wait for the first one. With `retryOnTimeout`, a first chunk that doesn't arrive within `retryTimeoutSecs` (default 5) re-issues the request once. * Newer models reject a request that ends with a model turn. For any model whose name doesn't start with `gemini-2.`, `gemini-3-`, `gemini-3.1-` or `gemini-pro-latest`, the service adds a trailing user message when needed. * A reply that stops for a reason other than `STOP` (safety, recitation, token limit, ...) logs a warning with the finish reason. The text that did arrive is still pushed. * Setting `thinkingBudget` on a Gemini 3, Gemini 4 or Gemma 4 model logs a warning: use `thinkingLevel` there. * `runInference()` accepts a `responseSchema` (JSON schema) on Gemini 2.5 and later. * An `INVALID_ARGUMENT` rejection removes a pending invalid file message from the context and is reported as an application error. * Token usage includes tool-use prompt tokens in the prompt count and thinking tokens in the completion count. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/google/package.json`: ✅ means the compat smoke test passes on that runtime. The client is `GoogleGenAI` from `@google/genai`, which uses `fetch`, so the service runs on Bun, Node, Deno and Cloudflare Workers. Subclasses can override `createClient()`. Image decoding (upstream uses Pillow) is not ported: pass an `imageDecoder` built on `sharp`, `@jsquash/*` or similar to handle image output. The deprecated upstream `InputParams` class and the `model`, `params` and `systemInstruction` constructor options are not ported; use `settings`. # xAI Grok > GrokLLMService streams chat completions from xAI Grok through its OpenAI-compatible API. Source: https://polvallverdu.github.io/pipedog/docs/services/llm/grok/ {/* Generated by scripts/generate-providers.ts from data/providers.ts. */} ## Overview [#overview] xAI's Grok models. * **`GrokLLMService`**: an `OpenAILLMService` subclass with xAI Grok's base URL and default model. Pipecat's documentation for the same service. `packages/xai/src/llm.ts`. The chat completions API reference. ## Installation [#installation] ```bash bun add @pipedog/xai ``` It depends on `@pipedog/openai`, which is installed with it. ```ts import { GrokLLMService } from "@pipedog/xai/llm"; ``` ## Prerequisites [#prerequisites] * An xAI account and an API key from the [xAI console](https://console.x.ai/). * Pass the key as `apiKey`. It is required, and the service never falls back to `OPENAI_API_KEY`. The examples read it from `XAI_API_KEY`. ## GrokLLMService [#grokllmservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | The API key for accessing Grok's API. | | `baseUrl` | `string \| null \| undefined` | no | The base URL for Grok API. Defaults to `"https://api.x.ai/v1"`. | | `settings` | `GrokLLMSettings \| null \| undefined` | no | Runtime-updatable settings; values override the built-in defaults. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `runInParallel` | `boolean \| undefined` | no | Whether to run function calls in parallel or sequentially. Defaults to true. | | `groupParallelTools` | `boolean \| undefined` | no | Whether to group parallel function calls so the LLM is triggered exactly once after all calls in the batch complete. When false, each function call result triggers the LLM independently as it arrives. Defaults to true. | | `functionCallTimeoutSecs` | `number \| null \| undefined` | no | Optional timeout in seconds for function calls. A call that runs past it is cancelled: its handler is cancelled (it gets a `CancelledError`), the call is settled as cancelled, and inference runs so the LLM can report that it didn't complete. | | `fileResolver` | `FileResolver \| null \| undefined` | no | Resolves file URLs in the context that this provider can't fetch itself (see `FileResolver`). Pass one configured with a `fileStorage` to resolve uploaded-file URLs (`new FileResolver({ fileStorage: runnerArgs.fileStorage })` when a runner hosts the bot), and share one instance across services that may consume the same files (e.g. services switched between mid-session) so downloads are reused. Defaults to a resolver that handles `data:` and `http(s)` URLs only. | | `organization` | `string \| null \| undefined` | no | OpenAI organization ID. | | `project` | `string \| null \| undefined` | no | OpenAI project ID. | | `defaultHeaders` | `Record \| null \| undefined` | no | Additional HTTP headers. | | `serviceTier` | `string \| null \| undefined` | no | Service tier to use (e.g., "auto", "flex", "priority"). | | `retryTimeoutSecs` | `number \| null \| undefined` | no | How long an inference may go without producing output before it is abandoned and re-issued, when `retryOnTimeout` is set. Defaults to 5.0 seconds. | | `retryOnTimeout` | `boolean \| null \| undefined` | no | Whether to re-issue the request once if the first attempt produces no output within `retryTimeoutSecs`. The retry is unbounded. | The other options (`organization`, `project`, `defaultHeaders`, `serviceTier`, `retryOnTimeout`, `retryTimeoutSecs`, function calling) are `OpenAILLMService`'s. See [OpenAI](https://polvallverdu.github.io/pipedog/docs/services/llm/openai/#configuration). | | | | ------------- | ------------------------------------------------------------------------------- | | Base URL | `https://api.x.ai/v1` | | Default model | `grok-4.20-non-reasoning` | | Models to try | `grok-4.20-non-reasoning`, `grok-4.6`. See the provider's catalog for the rest. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new GrokLLMService.Settings({...})` (`GrokLLMSettings`). They are [OpenAI's settings](https://polvallverdu.github.io/pipedog/docs/services/llm/openai/#settings), with these defaults: | Setting | Type | Default | Description | | ----------------- | ---------------------------------------------------------- | --------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | | `model` | `string` | `"grok-4.20-non-reasoning"` | Model name. | | `reasoningEffort` | `"none" \| "low" \| "medium" \| "high" \| "xhigh" \| null` | `null` | Sent as `reasoning_effort` when set, unless `extra` already has one. The non-reasoning models, including the default, reject it. | Every other setting keeps OpenAI's default. Unset sampling settings are left out of the request, so the provider's own defaults apply. `extra` carries provider-specific request fields. ### Usage [#usage] ```ts title="llm.ts" import { LLMUpdateSettingsFrame } from "@pipedog/core/frames/frames"; import { GrokLLMService } from "@pipedog/xai/llm"; const llm = new GrokLLMService({ apiKey: process.env.XAI_API_KEY ?? "", settings: new GrokLLMService.Settings({ model: "grok-4.20-non-reasoning", systemInstruction: "You are a helpful assistant in a voice conversation.", temperature: 0.7, }), }); ``` A reasoning model, with its reasoning effort: ```ts const reasoning = new GrokLLMService({ apiKey: process.env.XAI_API_KEY ?? "", settings: new GrokLLMService.Settings({ model: "grok-4.6", reasoningEffort: "low" }), }); ``` Change settings mid-call with an `LLMUpdateSettingsFrame`. The next completion uses them: ```ts // Queue it on the worker, or push it from a processor upstream of the service. const update = new LLMUpdateSettingsFrame({ delta: new GrokLLMService.Settings({ temperature: 0.2, maxTokens: 300 }), }); ``` ### Notes [#notes] * Runs a completion for each `LLMContextFrame`, between an `LLMFullResponseStartFrame` and an `LLMFullResponseEndFrame`, and streams the reply as `LLMTextFrame`s, as [`OpenAILLMService`](https://polvallverdu.github.io/pipedog/docs/services/llm/openai.md) does. Function calls, token usage metrics and `runInference` work the same way. * "developer" messages are sent as is: the API supports that role. * xAI reports reasoning tokens apart from `completion_tokens`, so they are added to the completion count in token usage metrics when the total says so. * `@pipedog/grok` re-exports `GrokLLMService` from `@pipedog/xai/llm`, as upstream's deprecated `pipecat.services.grok.llm` does. Import from `@pipedog/xai/llm`. * The same package has xAI's [speech-to-text](https://polvallverdu.github.io/pipedog/docs/services/stt/xai.md) and [text-to-speech](https://polvallverdu.github.io/pipedog/docs/services/tts/xai.md) services, and [Grok Voice Agent](https://polvallverdu.github.io/pipedog/docs/services/s2s/grok.md), the realtime (speech-to-speech) service. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/xai/package.json`: ✅ means the compat smoke test passes on that runtime. * The client is the official `openai` JS SDK (through `@pipedog/openai`), which talks through `fetch`. It works on Bun, Node, Deno and Cloudflare Workers. * The deprecated `model` constructor option is not ported. Use `settings`. * Tested against a local OpenAI-compatible mock server, not against xAI Grok itself. # Groq > GroqLLMService streams chat completions from Groq through its OpenAI-compatible API. Source: https://polvallverdu.github.io/pipedog/docs/services/llm/groq/ {/* Generated by scripts/generate-providers.ts from data/providers.ts. */} ## Overview [#overview] Groq serves open models with fast inference on its LPUs. * **`GroqLLMService`**: an `OpenAILLMService` subclass with Groq's base URL and default model. Pipecat's documentation for the same service. `packages/groq/src/llm.ts`. The chat completions API reference. ## Installation [#installation] ```bash bun add @pipedog/groq ``` It depends on `@pipedog/openai`, which is installed with it. ```ts import { GroqLLMService } from "@pipedog/groq/llm"; ``` ## Prerequisites [#prerequisites] * A Groq account and an API key from the [Groq console](https://console.groq.com/). * Pass the key as `apiKey`. It is required, and the service never falls back to `OPENAI_API_KEY`. The examples read it from `GROQ_API_KEY`. ## GroqLLMService [#groqllmservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | The API key for accessing Groq's API. | | `baseUrl` | `string \| null \| undefined` | no | The base URL for Groq API. Defaults to `"https://api.groq.com/openai/v1"`. | | `settings` | `GroqLLMSettings \| null \| undefined` | no | Runtime-updatable settings; values override the built-in defaults. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `runInParallel` | `boolean \| undefined` | no | Whether to run function calls in parallel or sequentially. Defaults to true. | | `groupParallelTools` | `boolean \| undefined` | no | Whether to group parallel function calls so the LLM is triggered exactly once after all calls in the batch complete. When false, each function call result triggers the LLM independently as it arrives. Defaults to true. | | `functionCallTimeoutSecs` | `number \| null \| undefined` | no | Optional timeout in seconds for function calls. A call that runs past it is cancelled: its handler is cancelled (it gets a `CancelledError`), the call is settled as cancelled, and inference runs so the LLM can report that it didn't complete. | | `fileResolver` | `FileResolver \| null \| undefined` | no | Resolves file URLs in the context that this provider can't fetch itself (see `FileResolver`). Pass one configured with a `fileStorage` to resolve uploaded-file URLs (`new FileResolver({ fileStorage: runnerArgs.fileStorage })` when a runner hosts the bot), and share one instance across services that may consume the same files (e.g. services switched between mid-session) so downloads are reused. Defaults to a resolver that handles `data:` and `http(s)` URLs only. | | `organization` | `string \| null \| undefined` | no | OpenAI organization ID. | | `project` | `string \| null \| undefined` | no | OpenAI project ID. | | `defaultHeaders` | `Record \| null \| undefined` | no | Additional HTTP headers. | | `serviceTier` | `string \| null \| undefined` | no | Service tier to use (e.g., "auto", "flex", "priority"). | | `retryTimeoutSecs` | `number \| null \| undefined` | no | How long an inference may go without producing output before it is abandoned and re-issued, when `retryOnTimeout` is set. Defaults to 5.0 seconds. | | `retryOnTimeout` | `boolean \| null \| undefined` | no | Whether to re-issue the request once if the first attempt produces no output within `retryTimeoutSecs`. The retry is unbounded. | The other options (`organization`, `project`, `defaultHeaders`, `serviceTier`, `retryOnTimeout`, `retryTimeoutSecs`, function calling) are `OpenAILLMService`'s. See [OpenAI](https://polvallverdu.github.io/pipedog/docs/services/llm/openai/#configuration). | | | | ------------- | --------------------------------------------------------------- | | Base URL | `https://api.groq.com/openai/v1` | | Default model | `openai/gpt-oss-120b` | | Models to try | `openai/gpt-oss-120b`. See the provider's catalog for the rest. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new GroqLLMService.Settings({...})` (`GroqLLMSettings`). They are [OpenAI's settings](https://polvallverdu.github.io/pipedog/docs/services/llm/openai/#settings), with these defaults: | Setting | Type | Default | Description | | ----------------- | ------------------------------------------------------------ | ----------------------- | ------------------------------------------------------------------------------------------------------------- | | `model` | `string` | `"openai/gpt-oss-120b"` | Model name. | | `reasoningEffort` | `"none" \| "default" \| "low" \| "medium" \| "high" \| null` | `null` | Sent as `reasoning_effort` when set. Which values a model accepts varies; the non-reasoning models reject it. | Every other setting keeps OpenAI's default. Unset sampling settings are left out of the request, so the provider's own defaults apply. `extra` carries provider-specific request fields. ### Usage [#usage] ```ts title="llm.ts" import { LLMUpdateSettingsFrame } from "@pipedog/core/frames/frames"; import { GroqLLMService } from "@pipedog/groq/llm"; const llm = new GroqLLMService({ apiKey: process.env.GROQ_API_KEY ?? "", settings: new GroqLLMService.Settings({ model: "openai/gpt-oss-120b", systemInstruction: "You are a helpful assistant in a voice conversation.", temperature: 0.7, }), }); ``` Keep a Qwen model's reasoning out of the spoken reply with `reasoningEffort: "none"`: ```ts const qwen = new GroqLLMService({ apiKey: process.env.GROQ_API_KEY ?? "", settings: new GroqLLMService.Settings({ model: "qwen/qwen3.8-27b", reasoningEffort: "none" }), }); ``` Change settings mid-call with an `LLMUpdateSettingsFrame`. The next completion uses them: ```ts // Queue it on the worker, or push it from a processor upstream of the service. const update = new LLMUpdateSettingsFrame({ delta: new GroqLLMService.Settings({ temperature: 0.2, maxTokens: 300 }), }); ``` ### Notes [#notes] * Runs a completion for each `LLMContextFrame`, between an `LLMFullResponseStartFrame` and an `LLMFullResponseEndFrame`, and streams the reply as `LLMTextFrame`s, as [`OpenAILLMService`](https://polvallverdu.github.io/pipedog/docs/services/llm/openai.md) does. Function calls, token usage metrics and `runInference` work the same way. * "developer" messages are sent as is: the API supports that role. * The same package has Groq's [speech-to-text](https://polvallverdu.github.io/pipedog/docs/services/stt/groq.md) and [text-to-speech](https://polvallverdu.github.io/pipedog/docs/services/tts/groq.md) services. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/groq/package.json`: ✅ means the compat smoke test passes on that runtime. * The client is the official `openai` JS SDK (through `@pipedog/openai`), which talks through `fetch`. It works on Bun, Node, Deno and Cloudflare Workers. * The deprecated `model` constructor option is not ported. Use `settings`. * Tested against a local OpenAI-compatible mock server, not against Groq itself. # Inception > InceptionLLMService streams chat completions from Inception through its OpenAI-compatible API. Source: https://polvallverdu.github.io/pipedog/docs/services/llm/inception/ {/* Generated by scripts/generate-providers.ts from data/providers.ts. */} ## Overview [#overview] Inception serves Mercury-3, a diffusion-based reasoning language model, over an OpenAI-compatible API. * **`InceptionLLMService`**: an `OpenAILLMService` subclass with Inception's base URL and default model. Pipecat's documentation for the same service. `packages/inception/src/llm.ts`. Models and the API reference. ## Installation [#installation] ```bash bun add @pipedog/inception ``` It depends on `@pipedog/openai`, which is installed with it. ```ts import { InceptionLLMService } from "@pipedog/inception/llm"; ``` ## Prerequisites [#prerequisites] * An Inception account and an API key from [Inception](https://docs.inceptionlabs.ai/). * Pass the key as `apiKey`. It is required, and the service never falls back to `OPENAI_API_KEY`. The examples read it from `INCEPTION_API_KEY`. ## InceptionLLMService [#inceptionllmservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | The API key for accessing Inception's API. | | `baseUrl` | `string \| null \| undefined` | no | The base URL for Inception API. Defaults to `"https://api.inceptionlabs.ai/v1"`. | | `settings` | `InceptionLLMSettings \| null \| undefined` | no | Runtime-updatable settings; values override the built-in defaults. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `runInParallel` | `boolean \| undefined` | no | Whether to run function calls in parallel or sequentially. Defaults to true. | | `groupParallelTools` | `boolean \| undefined` | no | Whether to group parallel function calls so the LLM is triggered exactly once after all calls in the batch complete. When false, each function call result triggers the LLM independently as it arrives. Defaults to true. | | `functionCallTimeoutSecs` | `number \| null \| undefined` | no | Optional timeout in seconds for function calls. A call that runs past it is cancelled: its handler is cancelled (it gets a `CancelledError`), the call is settled as cancelled, and inference runs so the LLM can report that it didn't complete. | | `fileResolver` | `FileResolver \| null \| undefined` | no | Resolves file URLs in the context that this provider can't fetch itself (see `FileResolver`). Pass one configured with a `fileStorage` to resolve uploaded-file URLs (`new FileResolver({ fileStorage: runnerArgs.fileStorage })` when a runner hosts the bot), and share one instance across services that may consume the same files (e.g. services switched between mid-session) so downloads are reused. Defaults to a resolver that handles `data:` and `http(s)` URLs only. | | `organization` | `string \| null \| undefined` | no | OpenAI organization ID. | | `project` | `string \| null \| undefined` | no | OpenAI project ID. | | `defaultHeaders` | `Record \| null \| undefined` | no | Additional HTTP headers. | | `serviceTier` | `string \| null \| undefined` | no | Service tier to use (e.g., "auto", "flex", "priority"). | | `retryTimeoutSecs` | `number \| null \| undefined` | no | How long an inference may go without producing output before it is abandoned and re-issued, when `retryOnTimeout` is set. Defaults to 5.0 seconds. | | `retryOnTimeout` | `boolean \| null \| undefined` | no | Whether to re-issue the request once if the first attempt produces no output within `retryTimeoutSecs`. The retry is unbounded. | The other options (`organization`, `project`, `defaultHeaders`, `serviceTier`, `retryOnTimeout`, `retryTimeoutSecs`, function calling) are `OpenAILLMService`'s. See [OpenAI](https://polvallverdu.github.io/pipedog/docs/services/llm/openai/#configuration). | | | | ------------- | ----------------------------------------------------- | | Base URL | `https://api.inceptionlabs.ai/v1` | | Default model | `mercury-3` | | Models to try | `mercury-3`. See the provider's catalog for the rest. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new InceptionLLMService.Settings({...})` (`InceptionLLMSettings`). They are [OpenAI's settings](https://polvallverdu.github.io/pipedog/docs/services/llm/openai/#settings), with these defaults: | Setting | Type | Default | Description | | ----------------- | -------------------------------------------------- | ------------- | ----------------------------------------------------------------------------------------- | | `model` | `string` | `"mercury-3"` | Model name. | | `reasoningEffort` | `"instant" \| "low" \| "medium" \| "high" \| null` | `null` | Sent as `reasoning_effort` when set. When unset, Inception's server-side default applies. | | `realtime` | `boolean \| null` | `true` | Sent as `realtime` in the request body, to cut the time to the first diffusion block. | Every other setting keeps OpenAI's default. Unset sampling settings are left out of the request, so the provider's own defaults apply. `extra` carries provider-specific request fields. ### Usage [#usage] ```ts title="llm.ts" import { LLMUpdateSettingsFrame } from "@pipedog/core/frames/frames"; import { InceptionLLMService } from "@pipedog/inception/llm"; const llm = new InceptionLLMService({ apiKey: process.env.INCEPTION_API_KEY ?? "", settings: new InceptionLLMService.Settings({ model: "mercury-3", systemInstruction: "You are a helpful assistant in a voice conversation.", }), }); ``` Trade reasoning for speed with `reasoningEffort`: ```ts const fast = new InceptionLLMService({ apiKey: process.env.INCEPTION_API_KEY ?? "", settings: new InceptionLLMService.Settings({ reasoningEffort: "instant" }), }); ``` Change settings mid-call with an `LLMUpdateSettingsFrame`. The next completion uses them: ```ts // Queue it on the worker, or push it from a processor upstream of the service. const update = new LLMUpdateSettingsFrame({ delta: new InceptionLLMService.Settings({ reasoningEffort: "low" }), }); ``` ### Notes [#notes] * Runs a completion for each `LLMContextFrame`, between an `LLMFullResponseStartFrame` and an `LLMFullResponseEndFrame`, and streams the reply as `LLMTextFrame`s, as [`OpenAILLMService`](https://polvallverdu.github.io/pipedog/docs/services/llm/openai.md) does. Function calls, token usage metrics and `runInference` work the same way. * "developer" messages are sent as "user" messages, since the API doesn't support that role (`supportsDeveloperRole` is `false`). * `realtime` and `reasoningEffort` go out in the request's `extra_body`. A field you put in `settings.extra` under `extra_body` wins over the service's own. * The default model was `mercury-2` before. Inception retires Mercury 2 and 2.5 on November 15, 2026, and they are closed to new signups. Until then, `settings: new InceptionLLMService.Settings({ model: "mercury-2" })` keeps it. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/inception/package.json`: ✅ means the compat smoke test passes on that runtime. * The client is the official `openai` JS SDK (through `@pipedog/openai`), which talks through `fetch`. It works on Bun, Node, Deno and Cloudflare Workers. * The deprecated `model` constructor option is not ported. Use `settings`. * Tested against a local OpenAI-compatible mock server, not against Inception itself. # LLM > Chat models that turn the conversation context into a response. Source: https://polvallverdu.github.io/pipedog/docs/services/llm/ {/* Generated by scripts/generate-providers.ts from data/providers.ts. */} An LLM service takes the conversation context (an `LLMContext`, through the context aggregators) and streams the model's response as text frames, with function calls. | Provider | Classes | Package | Status | | ------------------------------------------------------- | ------------------------------------------------------------ | --------------------- | ------ | | [OpenAI](https://polvallverdu.github.io/pipedog/docs/services/llm/openai.md) | `OpenAILLMService` | `@pipedog/openai` | Ported | | [OpenAI Responses](https://polvallverdu.github.io/pipedog/docs/services/llm/openai-responses.md) | `OpenAIResponsesLLMService`, `OpenAIResponsesHttpLLMService` | `@pipedog/openai` | Ported | | [Azure OpenAI](https://polvallverdu.github.io/pipedog/docs/services/llm/azure.md) | `AzureLLMService` | `@pipedog/azure` | Ported | | [Anthropic](https://polvallverdu.github.io/pipedog/docs/services/llm/anthropic.md) | `AnthropicLLMService` | `@pipedog/anthropic` | Ported | | [Google Gemini](https://polvallverdu.github.io/pipedog/docs/services/llm/google.md) | `GoogleLLMService` | `@pipedog/google` | Ported | | [Google Vertex AI](https://polvallverdu.github.io/pipedog/docs/services/llm/google-vertex.md) | `GoogleVertexLLMService` | `@pipedog/google` | Ported | | [AWS Bedrock](https://polvallverdu.github.io/pipedog/docs/services/llm/aws.md) | `AWSBedrockLLMService` | `@pipedog/aws` | Ported | | [AWS AgentCore](https://polvallverdu.github.io/pipedog/docs/services/llm/aws-agentcore.md) | `AWSAgentCoreProcessor` | `@pipedog/aws` | Ported | | [LangChain](https://polvallverdu.github.io/pipedog/docs/services/llm/langchain.md) | `LangchainProcessor` | `@pipedog/langchain` | Ported | | [Vercel AI SDK](https://polvallverdu.github.io/pipedog/docs/services/llm/ai-sdk.md) | `AISDKLLMService` | `@pipedog/ai-sdk` | Ported | | [Cloudflare Workers AI](https://polvallverdu.github.io/pipedog/docs/services/llm/cloudflare.md) | `WorkersAILLMService` | `@pipedog/cloudflare` | Ported | | [Perplexity](https://polvallverdu.github.io/pipedog/docs/services/llm/perplexity.md) | `PerplexityAgentHttpLLMService` | `@pipedog/perplexity` | Ported | | [NVIDIA](https://polvallverdu.github.io/pipedog/docs/services/llm/nvidia.md) | `NvidiaLLMService`, `NvidiaOmniLLMService` | `@pipedog/nvidia` | Ported | | [Baseten](https://polvallverdu.github.io/pipedog/docs/services/llm/baseten.md) | `BasetenLLMService` | `@pipedog/baseten` | Ported | | [Cerebras](https://polvallverdu.github.io/pipedog/docs/services/llm/cerebras.md) | `CerebrasLLMService` | `@pipedog/cerebras` | Ported | | [Crusoe](https://polvallverdu.github.io/pipedog/docs/services/llm/crusoe.md) | `CrusoeLLMService` | `@pipedog/crusoe` | Ported | | [DeepSeek](https://polvallverdu.github.io/pipedog/docs/services/llm/deepseek.md) | `DeepSeekLLMService` | `@pipedog/deepseek` | Ported | | [Fireworks AI](https://polvallverdu.github.io/pipedog/docs/services/llm/fireworks.md) | `FireworksLLMService` | `@pipedog/fireworks` | Ported | | [Groq](https://polvallverdu.github.io/pipedog/docs/services/llm/groq.md) | `GroqLLMService` | `@pipedog/groq` | Ported | | [Inception](https://polvallverdu.github.io/pipedog/docs/services/llm/inception.md) | `InceptionLLMService` | `@pipedog/inception` | Ported | | [Mistral](https://polvallverdu.github.io/pipedog/docs/services/llm/mistral.md) | `MistralLLMService` | `@pipedog/mistral` | Ported | | [Nebius](https://polvallverdu.github.io/pipedog/docs/services/llm/nebius.md) | `NebiusLLMService` | `@pipedog/nebius` | Ported | | [Novita AI](https://polvallverdu.github.io/pipedog/docs/services/llm/novita.md) | `NovitaLLMService` | `@pipedog/novita` | Ported | | [Ollama](https://polvallverdu.github.io/pipedog/docs/services/llm/ollama.md) | `OLLamaLLMService` | `@pipedog/ollama` | Ported | | [OpenRouter](https://polvallverdu.github.io/pipedog/docs/services/llm/openrouter.md) | `OpenRouterLLMService` | `@pipedog/openrouter` | Ported | | [Qwen](https://polvallverdu.github.io/pipedog/docs/services/llm/qwen.md) | `QwenLLMService` | `@pipedog/qwen` | Ported | | [SambaNova](https://polvallverdu.github.io/pipedog/docs/services/llm/sambanova.md) | `SambaNovaLLMService` | `@pipedog/sambanova` | Ported | | [Sarvam AI](https://polvallverdu.github.io/pipedog/docs/services/llm/sarvam.md) | `SarvamLLMService` | `@pipedog/sarvam` | Ported | | [Smallest AI](https://polvallverdu.github.io/pipedog/docs/services/llm/smallest.md) | `SmallestLLMService` | `@pipedog/smallest` | Ported | | [Together AI](https://polvallverdu.github.io/pipedog/docs/services/llm/together.md) | `TogetherLLMService` | `@pipedog/together` | Ported | | [xAI Grok](https://polvallverdu.github.io/pipedog/docs/services/llm/grok.md) | `GrokLLMService` | `@pipedog/xai` | Ported | Pipecat's [llm reference](https://docs.pipecat.ai/api-reference/server/services/llm) lists every upstream provider. Those not in this table are on [Other upstream providers](https://polvallverdu.github.io/pipedog/docs/services/other-upstream.md). # LangChain > Run a LangChain.js runnable as the pipeline's LLM step with LangchainProcessor. Source: https://polvallverdu.github.io/pipedog/docs/services/llm/langchain/ ## Overview [#overview] `LangchainProcessor` puts a [LangChain.js](https://js.langchain.com) runnable where an LLM service would sit in the pipeline. It takes the latest user message from the context, streams it through the runnable and pushes the output as text frames. * **`LangchainProcessor`**: a frame processor that wraps any `Runnable` from `@langchain/core`: a prompt piped into a chat model, a chain with message history, or a compiled LangGraph graph. Pipecat's `LangchainProcessor`. Upstream has no service page for it. `packages/langchain/src/langchain.ts`. Runnables, chat models and LangGraph. ## Installation [#installation] `@langchain/core` is a peer dependency (`^1.0.0`): `@pipedog/langchain` imports from it but does not install it, so you pick the version. Add the chat model integration you use as well. ```bash bun add @pipedog/langchain @langchain/core @langchain/openai ``` ```ts import { LangchainProcessor } from "@pipedog/langchain/langchain"; ``` ## Prerequisites [#prerequisites] * Whatever your runnable needs. With `ChatOpenAI` that is an OpenAI API key in `OPENAI_API_KEY`. * The processor itself needs no key. ## LangchainProcessor [#langchainprocessor] ### Configuration [#configuration] The constructor is `new LangchainProcessor(chain, options)`. `chain` is a `Runnable` from `@langchain/core/runnables`. The processor calls only its `stream(input, config)` method. | Option | Type | Required | Description | |---|---|---|---| | `transcriptKey` | `string \| undefined` | no | The key to use when passing input to the chain. Defaults to `"input"`. | What the runnable receives: * **input**: `{ [transcriptKey]: text }`, where `text` is the trimmed content of the last context message. With the default key that is `{ input: "..." }`, so a prompt template should use an `{input}` variable. * **config**: `{ configurable: { session_id }, signal }`. `session_id` is the value passed to `setParticipantId()` (`null` until then), which is what `RunnableWithMessageHistory` reads. `signal` aborts the stream when the pipeline is interrupted or cancelled. What it must yield: strings or `AIMessageChunk`s. A chunk's string `content` becomes the frame text. List content (multimodal) is turned into text with `JSON.stringify`. Any other value becomes an empty `TextFrame`. ### Usage [#usage] Build the runnable with LangChain: ```ts title="chain.ts" import { ChatPromptTemplate } from "@langchain/core/prompts"; import { ChatOpenAI } from "@langchain/openai"; const prompt = ChatPromptTemplate.fromMessages([ ["system", "You are a helpful voice assistant. Keep answers short."], ["human", "{input}"], ]); export const chain = prompt.pipe(new ChatOpenAI({ model: "gpt-4o-mini" })); ``` Then wrap it in the processor, in place of an LLM service: ```ts title="langchain.ts" import { LangchainProcessor } from "@pipedog/langchain/langchain"; // Any @langchain/core Runnable: a prompt piped into a chat model, a compiled LangGraph, ... declare const chain: ConstructorParameters[0]; const langchain = new LangchainProcessor(chain, { transcriptKey: "input" }); // Sent to the chain as `config.configurable.session_id`, for message history. langchain.setParticipantId("participant-123"); ``` The block that builds the chain is not type-checked by the docs build, since the docs app does not install `@langchain/core`. The processor block is. ### Notes [#notes] * Consumes `LLMContextFrame`. Every other frame passes through. * The last message must be a standard message with string content. A provider-specific message (`LLMSpecificMessage`) or list content is not sent to the chain: the context frame is passed on instead. * For each run it pushes `LLMFullResponseStartFrame`, one `TextFrame` per streamed chunk (with `includesInterFrameSpaces` set) and `LLMFullResponseEndFrame`. The end frame is pushed even when the chain throws. * A chain error is pushed upstream as an `ErrorFrame` (`Unknown error occurred: ...`). * A compiled LangGraph graph is a `Runnable`, so it works here too. Its default `stream()` yields state updates, not text, so wrap it (for example in a `RunnableLambda` that returns the final message) or stream with `streamMode: "messages"`. The package tests show both. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/langchain/package.json`: ✅ means the compat smoke test passes on that runtime. * Each step of the stream is a cancellation point, so an interruption stops the chain mid-stream. The iterator is closed in a `finally` block. * Upstream logs a warning on Python's `GeneratorExit`. That branch has no JS equivalent and is not ported. * The package targets Bun, Node, Deno and Cloudflare Workers, and uses no runtime-specific APIs. Whether a chain runs on a given runtime depends on the LangChain integrations it uses. # Mistral > MistralLLMService streams chat completions from Mistral through its OpenAI-compatible API. Source: https://polvallverdu.github.io/pipedog/docs/services/llm/mistral/ {/* Generated by scripts/generate-providers.ts from data/providers.ts. */} ## Overview [#overview] Mistral AI's chat models. * **`MistralLLMService`**: an `OpenAILLMService` subclass with Mistral's base URL and default model. It uses `MistralLLMAdapter`, which rewrites the history to Mistral's rules: an assistant message after every tool message, system messages only at the start, and `prefix: true` on a trailing assistant message. Pipecat's documentation for the same service. `packages/mistral/src/llm.ts` and `mistral_adapter.ts`. The API reference. ## Installation [#installation] ```bash bun add @pipedog/mistral ``` It depends on `@pipedog/openai`, which is installed with it. ```ts import { MistralLLMService } from "@pipedog/mistral/llm"; import { MistralLLMAdapter } from "@pipedog/mistral/mistral_adapter"; ``` ## Prerequisites [#prerequisites] * A Mistral account and an API key from [Mistral](https://docs.mistral.ai/api/). * Pass the key as `apiKey`. It is required, and the service never falls back to `OPENAI_API_KEY`. The examples read it from `MISTRAL_API_KEY`. ## MistralLLMService [#mistralllmservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `apiKey` | `string` | yes | The API key for accessing Mistral's API. | | `baseUrl` | `string \| null \| undefined` | no | The base URL for Mistral API. Defaults to `"https://api.mistral.ai/v1"`. | | `settings` | `MistralLLMSettings \| null \| undefined` | no | Runtime-updatable settings; values override the built-in defaults. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `runInParallel` | `boolean \| undefined` | no | Whether to run function calls in parallel or sequentially. Defaults to true. | | `groupParallelTools` | `boolean \| undefined` | no | Whether to group parallel function calls so the LLM is triggered exactly once after all calls in the batch complete. When false, each function call result triggers the LLM independently as it arrives. Defaults to true. | | `functionCallTimeoutSecs` | `number \| null \| undefined` | no | Optional timeout in seconds for function calls. A call that runs past it is cancelled: its handler is cancelled (it gets a `CancelledError`), the call is settled as cancelled, and inference runs so the LLM can report that it didn't complete. | | `fileResolver` | `FileResolver \| null \| undefined` | no | Resolves file URLs in the context that this provider can't fetch itself (see `FileResolver`). Pass one configured with a `fileStorage` to resolve uploaded-file URLs (`new FileResolver({ fileStorage: runnerArgs.fileStorage })` when a runner hosts the bot), and share one instance across services that may consume the same files (e.g. services switched between mid-session) so downloads are reused. Defaults to a resolver that handles `data:` and `http(s)` URLs only. | | `organization` | `string \| null \| undefined` | no | OpenAI organization ID. | | `project` | `string \| null \| undefined` | no | OpenAI project ID. | | `defaultHeaders` | `Record \| null \| undefined` | no | Additional HTTP headers. | | `serviceTier` | `string \| null \| undefined` | no | Service tier to use (e.g., "auto", "flex", "priority"). | | `retryTimeoutSecs` | `number \| null \| undefined` | no | How long an inference may go without producing output before it is abandoned and re-issued, when `retryOnTimeout` is set. Defaults to 5.0 seconds. | | `retryOnTimeout` | `boolean \| null \| undefined` | no | Whether to re-issue the request once if the first attempt produces no output within `retryTimeoutSecs`. The retry is unbounded. | The other options (`organization`, `project`, `defaultHeaders`, `serviceTier`, `retryOnTimeout`, `retryTimeoutSecs`, function calling) are `OpenAILLMService`'s. See [OpenAI](https://polvallverdu.github.io/pipedog/docs/services/llm/openai/#configuration). | | | | ------------- | ---------------------------------------------------------------------------------------- | | Base URL | `https://api.mistral.ai/v1` | | Default model | `mistral-small-latest` | | Models to try | `mistral-small-latest`, `mistral-large-latest`. See the provider's catalog for the rest. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new MistralLLMService.Settings({...})` (`MistralLLMSettings`). They are [OpenAI's settings](https://polvallverdu.github.io/pipedog/docs/services/llm/openai/#settings), with these defaults: | Setting | Type | Default | Description | | ------- | -------- | ------------------------ | ----------- | | `model` | `string` | `"mistral-small-latest"` | Model name. | Every other setting keeps OpenAI's default. Unset sampling settings are left out of the request, so the provider's own defaults apply. `extra` carries provider-specific request fields. ### Usage [#usage] ```ts title="llm.ts" import { LLMUpdateSettingsFrame } from "@pipedog/core/frames/frames"; import { MistralLLMService } from "@pipedog/mistral/llm"; const llm = new MistralLLMService({ apiKey: process.env.MISTRAL_API_KEY ?? "", settings: new MistralLLMService.Settings({ model: "mistral-small-latest", systemInstruction: "You are a helpful assistant in a voice conversation.", temperature: 0.7, }), }); ``` Change settings mid-call with an `LLMUpdateSettingsFrame`. The next completion uses them: ```ts // Queue it on the worker, or push it from a processor upstream of the service. const update = new LLMUpdateSettingsFrame({ delta: new MistralLLMService.Settings({ temperature: 0.2, maxTokens: 300 }), }); ``` ### Notes [#notes] * Runs a completion for each `LLMContextFrame`, between an `LLMFullResponseStartFrame` and an `LLMFullResponseEndFrame`, and streams the reply as `LLMTextFrame`s, as [`OpenAILLMService`](https://polvallverdu.github.io/pipedog/docs/services/llm/openai.md) does. Function calls, token usage metrics and `runInference` work the same way. * "developer" messages are sent as "user" messages, since the API doesn't support that role (`supportsDeveloperRole` is `false`). * Messages go through `MistralLLMAdapter`, which rewrites the history to Mistral's rules: an assistant message after every tool message, system messages only at the start, and `prefix: true` on a trailing assistant message. * `seed` is sent as Mistral's `random_seed`. `stream_options`, `max_completion_tokens` and `service_tier` are not sent. * Mistral detects function calls from the whole history, so calls that already have a tool result in the context are skipped instead of run twice. `on_function_calls_started` may still fire twice. * The same package has Mistral's [speech-to-text](https://polvallverdu.github.io/pipedog/docs/services/stt/mistral.md) and [text-to-speech](https://polvallverdu.github.io/pipedog/docs/services/tts/mistral.md) services. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/mistral/package.json`: ✅ means the compat smoke test passes on that runtime. * The client is the official `openai` JS SDK (through `@pipedog/openai`), which talks through `fetch`. It works on Bun, Node, Deno and Cloudflare Workers. * The deprecated `model` constructor option is not ported. Use `settings`. * Tested against a local OpenAI-compatible mock server, not against Mistral itself. # Nebius > NebiusLLMService streams chat completions from Nebius through its OpenAI-compatible API. Source: https://polvallverdu.github.io/pipedog/docs/services/llm/nebius/ {/* Generated by scripts/generate-providers.ts from data/providers.ts. */} ## Overview [#overview] Nebius Token Factory serves open models through an OpenAI-compatible API. * **`NebiusLLMService`**: an `OpenAILLMService` subclass with Nebius's base URL and default model. Pipecat's documentation for the same service. `packages/nebius/src/llm.ts`. Models and API keys. ## Installation [#installation] ```bash bun add @pipedog/nebius ``` It depends on `@pipedog/openai`, which is installed with it. ```ts import { NebiusLLMService } from "@pipedog/nebius/llm"; ``` ## Prerequisites [#prerequisites] * A Nebius account and an API key from [Nebius Token Factory](https://tokenfactory.nebius.com/). * Pass the key as `apiKey`. It is required, and the service never falls back to `OPENAI_API_KEY`. The examples read it from `NEBIUS_API_KEY`. ## NebiusLLMService [#nebiusllmservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | The API key for accessing Nebius's API. | | `baseUrl` | `string \| null \| undefined` | no | The base URL for the API. Defaults to `"https://api.tokenfactory.nebius.com/v1/"`. | | `settings` | `NebiusLLMSettings \| null \| undefined` | no | Runtime-updatable settings; values override the built-in defaults. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `runInParallel` | `boolean \| undefined` | no | Whether to run function calls in parallel or sequentially. Defaults to true. | | `groupParallelTools` | `boolean \| undefined` | no | Whether to group parallel function calls so the LLM is triggered exactly once after all calls in the batch complete. When false, each function call result triggers the LLM independently as it arrives. Defaults to true. | | `functionCallTimeoutSecs` | `number \| null \| undefined` | no | Optional timeout in seconds for function calls. A call that runs past it is cancelled: its handler is cancelled (it gets a `CancelledError`), the call is settled as cancelled, and inference runs so the LLM can report that it didn't complete. | | `fileResolver` | `FileResolver \| null \| undefined` | no | Resolves file URLs in the context that this provider can't fetch itself (see `FileResolver`). Pass one configured with a `fileStorage` to resolve uploaded-file URLs (`new FileResolver({ fileStorage: runnerArgs.fileStorage })` when a runner hosts the bot), and share one instance across services that may consume the same files (e.g. services switched between mid-session) so downloads are reused. Defaults to a resolver that handles `data:` and `http(s)` URLs only. | | `organization` | `string \| null \| undefined` | no | OpenAI organization ID. | | `project` | `string \| null \| undefined` | no | OpenAI project ID. | | `defaultHeaders` | `Record \| null \| undefined` | no | Additional HTTP headers. | | `serviceTier` | `string \| null \| undefined` | no | Service tier to use (e.g., "auto", "flex", "priority"). | | `retryTimeoutSecs` | `number \| null \| undefined` | no | How long an inference may go without producing output before it is abandoned and re-issued, when `retryOnTimeout` is set. Defaults to 5.0 seconds. | | `retryOnTimeout` | `boolean \| null \| undefined` | no | Whether to re-issue the request once if the first attempt produces no output within `retryTimeoutSecs`. The retry is unbounded. | The other options (`organization`, `project`, `defaultHeaders`, `serviceTier`, `retryOnTimeout`, `retryTimeoutSecs`, function calling) are `OpenAILLMService`'s. See [OpenAI](https://polvallverdu.github.io/pipedog/docs/services/llm/openai/#configuration). | | | | ------------- | ---------------------------------------------------------------------------- | | Base URL | `https://api.tokenfactory.nebius.com/v1/` | | Default model | `Qwen/Qwen3-30B-A3B-Instruct-2507` | | Models to try | `Qwen/Qwen3-30B-A3B-Instruct-2507`. See the provider's catalog for the rest. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new NebiusLLMService.Settings({...})` (`NebiusLLMSettings`). They are [OpenAI's settings](https://polvallverdu.github.io/pipedog/docs/services/llm/openai/#settings), with these defaults: | Setting | Type | Default | Description | | ------- | -------- | ------------------------------------ | ----------- | | `model` | `string` | `"Qwen/Qwen3-30B-A3B-Instruct-2507"` | Model name. | Every other setting keeps OpenAI's default. Unset sampling settings are left out of the request, so the provider's own defaults apply. `extra` carries provider-specific request fields. ### Usage [#usage] ```ts title="llm.ts" import { LLMUpdateSettingsFrame } from "@pipedog/core/frames/frames"; import { NebiusLLMService } from "@pipedog/nebius/llm"; const llm = new NebiusLLMService({ apiKey: process.env.NEBIUS_API_KEY ?? "", settings: new NebiusLLMService.Settings({ model: "Qwen/Qwen3-30B-A3B-Instruct-2507", systemInstruction: "You are a helpful assistant in a voice conversation.", temperature: 0.7, }), }); ``` Change settings mid-call with an `LLMUpdateSettingsFrame`. The next completion uses them: ```ts // Queue it on the worker, or push it from a processor upstream of the service. const update = new LLMUpdateSettingsFrame({ delta: new NebiusLLMService.Settings({ temperature: 0.2, maxTokens: 300 }), }); ``` ### Notes [#notes] * Runs a completion for each `LLMContextFrame`, between an `LLMFullResponseStartFrame` and an `LLMFullResponseEndFrame`, and streams the reply as `LLMTextFrame`s, as [`OpenAILLMService`](https://polvallverdu.github.io/pipedog/docs/services/llm/openai.md) does. Function calls, token usage metrics and `runInference` work the same way. * "developer" messages are sent as "user" messages, since the API doesn't support that role (`supportsDeveloperRole` is `false`). ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/nebius/package.json`: ✅ means the compat smoke test passes on that runtime. * The client is the official `openai` JS SDK (through `@pipedog/openai`), which talks through `fetch`. It works on Bun, Node, Deno and Cloudflare Workers. * The deprecated `model` constructor option is not ported. Use `settings`. * Tested against a local OpenAI-compatible mock server, not against Nebius itself. # Novita AI > NovitaLLMService streams chat completions from Novita AI through its OpenAI-compatible API. Source: https://polvallverdu.github.io/pipedog/docs/services/llm/novita/ {/* Generated by scripts/generate-providers.ts from data/providers.ts. */} ## Overview [#overview] Novita AI serves open models through an OpenAI-compatible API. * **`NovitaLLMService`**: an `OpenAILLMService` subclass with Novita AI's base URL and default model. Pipecat's documentation for the same service. `packages/novita/src/llm.ts`. Models and the API. ## Installation [#installation] ```bash bun add @pipedog/novita ``` It depends on `@pipedog/openai`, which is installed with it. ```ts import { NovitaLLMService } from "@pipedog/novita/llm"; ``` ## Prerequisites [#prerequisites] * A Novita AI account and an API key from [Novita AI](https://novita.ai). * Pass the key as `apiKey`. It is required, and the service never falls back to `OPENAI_API_KEY`. The examples read it from `NOVITA_API_KEY`. ## NovitaLLMService [#novitallmservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | The API key for accessing Novita AI's API. | | `baseUrl` | `string \| null \| undefined` | no | The base URL for the API. Defaults to `"https://api.novita.ai/openai"`. | | `settings` | `NovitaLLMSettings \| null \| undefined` | no | Runtime-updatable settings; values override the built-in defaults. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `runInParallel` | `boolean \| undefined` | no | Whether to run function calls in parallel or sequentially. Defaults to true. | | `groupParallelTools` | `boolean \| undefined` | no | Whether to group parallel function calls so the LLM is triggered exactly once after all calls in the batch complete. When false, each function call result triggers the LLM independently as it arrives. Defaults to true. | | `functionCallTimeoutSecs` | `number \| null \| undefined` | no | Optional timeout in seconds for function calls. A call that runs past it is cancelled: its handler is cancelled (it gets a `CancelledError`), the call is settled as cancelled, and inference runs so the LLM can report that it didn't complete. | | `fileResolver` | `FileResolver \| null \| undefined` | no | Resolves file URLs in the context that this provider can't fetch itself (see `FileResolver`). Pass one configured with a `fileStorage` to resolve uploaded-file URLs (`new FileResolver({ fileStorage: runnerArgs.fileStorage })` when a runner hosts the bot), and share one instance across services that may consume the same files (e.g. services switched between mid-session) so downloads are reused. Defaults to a resolver that handles `data:` and `http(s)` URLs only. | | `organization` | `string \| null \| undefined` | no | OpenAI organization ID. | | `project` | `string \| null \| undefined` | no | OpenAI project ID. | | `defaultHeaders` | `Record \| null \| undefined` | no | Additional HTTP headers. | | `serviceTier` | `string \| null \| undefined` | no | Service tier to use (e.g., "auto", "flex", "priority"). | | `retryTimeoutSecs` | `number \| null \| undefined` | no | How long an inference may go without producing output before it is abandoned and re-issued, when `retryOnTimeout` is set. Defaults to 5.0 seconds. | | `retryOnTimeout` | `boolean \| null \| undefined` | no | Whether to re-issue the request once if the first attempt produces no output within `retryTimeoutSecs`. The retry is unbounded. | The other options (`organization`, `project`, `defaultHeaders`, `serviceTier`, `retryOnTimeout`, `retryTimeoutSecs`, function calling) are `OpenAILLMService`'s. See [OpenAI](https://polvallverdu.github.io/pipedog/docs/services/llm/openai/#configuration). | | | | ------------- | --------------------------------------------------------------------------------------- | | Base URL | `https://api.novita.ai/openai` | | Default model | `moonshotai/kimi-k2.5` | | Models to try | `moonshotai/kimi-k2.5`, `openai/gpt-oss-120b`. See the provider's catalog for the rest. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new NovitaLLMService.Settings({...})` (`NovitaLLMSettings`). They are [OpenAI's settings](https://polvallverdu.github.io/pipedog/docs/services/llm/openai/#settings), with these defaults: | Setting | Type | Default | Description | | ------- | -------- | ------------------------ | ----------- | | `model` | `string` | `"moonshotai/kimi-k2.5"` | Model name. | Every other setting keeps OpenAI's default. Unset sampling settings are left out of the request, so the provider's own defaults apply. `extra` carries provider-specific request fields. ### Usage [#usage] ```ts title="llm.ts" import { LLMUpdateSettingsFrame } from "@pipedog/core/frames/frames"; import { NovitaLLMService } from "@pipedog/novita/llm"; const llm = new NovitaLLMService({ apiKey: process.env.NOVITA_API_KEY ?? "", settings: new NovitaLLMService.Settings({ model: "moonshotai/kimi-k2.5", systemInstruction: "You are a helpful assistant in a voice conversation.", temperature: 0.7, }), }); ``` Change settings mid-call with an `LLMUpdateSettingsFrame`. The next completion uses them: ```ts // Queue it on the worker, or push it from a processor upstream of the service. const update = new LLMUpdateSettingsFrame({ delta: new NovitaLLMService.Settings({ temperature: 0.2, maxTokens: 300 }), }); ``` ### Notes [#notes] * Runs a completion for each `LLMContextFrame`, between an `LLMFullResponseStartFrame` and an `LLMFullResponseEndFrame`, and streams the reply as `LLMTextFrame`s, as [`OpenAILLMService`](https://polvallverdu.github.io/pipedog/docs/services/llm/openai.md) does. Function calls, token usage metrics and `runInference` work the same way. * "developer" messages are sent as is: the API supports that role. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/novita/package.json`: ✅ means the compat smoke test passes on that runtime. * The client is the official `openai` JS SDK (through `@pipedog/openai`), which talks through `fetch`. It works on Bun, Node, Deno and Cloudflare Workers. * The deprecated `model` constructor option is not ported. Use `settings`. * Tested against a local OpenAI-compatible mock server, not against Novita AI itself. # NVIDIA > NvidiaLLMService streams chat completions from NVIDIA NIM, and NvidiaOmniLLMService hears the user's speech with Nemotron Omni. Source: https://polvallverdu.github.io/pipedog/docs/services/llm/nvidia/ ## Overview [#overview] NVIDIA NIM serves models such as Nemotron over an OpenAI-compatible API, on NVIDIA's cloud or on your own hardware. pipedog ports both of Pipecat's NVIDIA LLM services: * **`NvidiaLLMService`**: an [`OpenAILLMService`](https://polvallverdu.github.io/pipedog/docs/services/llm/openai.md) subclass with NIM's base URL and a default Nemotron model. It turns a model's reasoning into `LLMThought*Frame`s, from a `reasoning_content` field or a leading `` block, so TTS never reads it. * **`NvidiaOmniLLMService`**: Nemotron Omni, which takes speech, images and video as well as text, and replies with text. By default it hears the user directly: it transcribes and answers each spoken turn in one request, in place of an STT service. Upstream's NVIDIA package also has Riva services: `NvidiaSTTService`, `NvidiaSegmentedSTTService` and `NvidiaTTSService`, and SageMaker variants of them. They aren't ported yet. See [NVIDIA Riva speech-to-text](https://polvallverdu.github.io/pipedog/docs/services/stt/nvidia.md) and [NVIDIA Riva text-to-speech](https://polvallverdu.github.io/pipedog/docs/services/tts/nvidia.md). Use another TTS service with Omni. Pipecat's documentation for the same service. `packages/nvidia/src/llm.ts`, `omni/llm.ts` and `nvidia_omni_adapter.ts`. Models and the API reference. The Omni model card: supported inputs and deployment options. ## Installation [#installation] ```bash bun add @pipedog/nvidia ``` It depends on `@pipedog/openai`, which is installed with it. ```ts import { NvidiaLLMService } from "@pipedog/nvidia/llm"; import { NvidiaOmniLLMService } from "@pipedog/nvidia/omni/llm"; ``` ## Prerequisites [#prerequisites] * For NVIDIA's cloud endpoint, an API key from [build.nvidia.com](https://build.nvidia.com/), passed as `apiKey`. The examples read it from `NVIDIA_API_KEY`. The service logs a warning when it uses the cloud endpoint without a key. * A local NIM deployment needs no key: pass its address as `baseUrl`. ## NvidiaLLMService [#nvidiallmservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string \| null \| undefined` | no | NVIDIA API key for authentication. Required when using the cloud endpoint. Not needed for local NIM deployments. | | `baseUrl` | `string \| null \| undefined` | no | The base URL for NIM API. Defaults to NVIDIA's cloud endpoint. For local deployments, pass the local address (e.g. `http://localhost:8000/v1`). | | `settings` | `NvidiaLLMSettings \| null \| undefined` | no | Runtime-updatable settings. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `runInParallel` | `boolean \| undefined` | no | Whether to run function calls in parallel or sequentially. Defaults to true. | | `groupParallelTools` | `boolean \| undefined` | no | Whether to group parallel function calls so the LLM is triggered exactly once after all calls in the batch complete. When false, each function call result triggers the LLM independently as it arrives. Defaults to true. | | `functionCallTimeoutSecs` | `number \| null \| undefined` | no | Optional timeout in seconds for function calls. A call that runs past it is cancelled: its handler is cancelled (it gets a `CancelledError`), the call is settled as cancelled, and inference runs so the LLM can report that it didn't complete. | | `fileResolver` | `FileResolver \| null \| undefined` | no | Resolves file URLs in the context that this provider can't fetch itself (see `FileResolver`). Pass one configured with a `fileStorage` to resolve uploaded-file URLs (`new FileResolver({ fileStorage: runnerArgs.fileStorage })` when a runner hosts the bot), and share one instance across services that may consume the same files (e.g. services switched between mid-session) so downloads are reused. Defaults to a resolver that handles `data:` and `http(s)` URLs only. | | `organization` | `string \| null \| undefined` | no | OpenAI organization ID. | | `project` | `string \| null \| undefined` | no | OpenAI project ID. | | `defaultHeaders` | `Record \| null \| undefined` | no | Additional HTTP headers. | | `serviceTier` | `string \| null \| undefined` | no | Service tier to use (e.g., "auto", "flex", "priority"). | | `retryTimeoutSecs` | `number \| null \| undefined` | no | How long an inference may go without producing output before it is abandoned and re-issued, when `retryOnTimeout` is set. Defaults to 5.0 seconds. | | `retryOnTimeout` | `boolean \| null \| undefined` | no | Whether to re-issue the request once if the first attempt produces no output within `retryTimeoutSecs`. The retry is unbounded. | The other options (`organization`, `project`, `defaultHeaders`, `serviceTier`, `retryOnTimeout`, `retryTimeoutSecs`, function calling) are `OpenAILLMService`'s. See [OpenAI](https://polvallverdu.github.io/pipedog/docs/services/llm/openai/#configuration). | | | | ------------- | ---------------------------------------- | | Base URL | `https://integrate.api.nvidia.com/v1` | | Default model | `nvidia/nemotron-3.5-super-vl-120b-a12b` | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new NvidiaLLMService.Settings({...})` (`NvidiaLLMSettings`). They are [OpenAI's settings](https://polvallverdu.github.io/pipedog/docs/services/llm/openai/#settings), with `model` defaulting to `"nvidia/nemotron-3.5-super-vl-120b-a12b"`. ### Usage [#usage] ```ts title="llm.ts" import { SileroVADAnalyzer } from "@pipedog/audio-silero"; import { CartesiaTTSService } from "@pipedog/cartesia/tts"; import { LLMUpdateSettingsFrame } from "@pipedog/core/frames/frames"; import { Pipeline } from "@pipedog/core/pipeline/pipeline"; import { LLMContext } from "@pipedog/core/processors/aggregators/llm_context"; import { LLMContextAggregatorPair, LLMUserAggregatorParams, type UserTurnMessageAddedMessage, } from "@pipedog/core/processors/aggregators/llm_response_universal"; import type { BaseTransport } from "@pipedog/core/transports/base_transport"; import { NvidiaLLMService } from "@pipedog/nvidia/llm"; import { NvidiaOmniLLMService } from "@pipedog/nvidia/omni/llm"; const llm = new NvidiaLLMService({ apiKey: process.env.NVIDIA_API_KEY ?? "", settings: new NvidiaLLMService.Settings({ model: "nvidia/nemotron-3.5-super-vl-120b-a12b", systemInstruction: "You are a helpful assistant in a voice conversation. Keep replies short.", }), }); ``` Point it at a NIM you run yourself: ```ts // A NIM running on your own machine takes no API key. const local = new NvidiaLLMService({ baseUrl: "http://localhost:8000/v1" }); ``` Change settings mid-call with an `LLMUpdateSettingsFrame`. The next completion uses them: ```ts // Queue it on the worker, or push it from a processor upstream of the service. const update = new LLMUpdateSettingsFrame({ delta: new NvidiaLLMService.Settings({ temperature: 0.4 }), }); ``` ### Notes [#notes] * Runs a completion for each `LLMContextFrame` and streams the reply as `LLMTextFrame`s, as `OpenAILLMService` does. Function calls, token usage metrics and `runInference` work the same way. * Reasoning sent in the delta's `reasoning_content` (or `reasoning`) field, as Nemotron Nano models do, is pushed as `LLMThoughtStartFrame`, `LLMThoughtTextFrame` and `LLMThoughtEndFrame`. * A reply that starts with a `...` block, as DeepSeek-R1 and some Nemotron models write, has the block pushed as thought frames and only the rest sent on as text. ## NvidiaOmniLLMService [#nvidiaomnillmservice] ### Configuration [#configuration-1] | Option | Type | Required | Description | |---|---|---|---| | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `retryOnTimeout` | `boolean \| null \| undefined` | no | Whether to re-issue the request once if the first attempt produces no output within `retryTimeoutSecs`. The retry is unbounded. | | `apiKey` | `string \| null \| undefined` | no | NVIDIA API key. Required for the cloud endpoint, not for local deployments. | | `baseUrl` | `string \| null \| undefined` | no | OpenAI-compatible endpoint base URL. For local deployments, pass the local address (e.g. `http://localhost:8000/v1`). | | `audioInput` | `boolean \| undefined` | no | Whether Omni hears the user directly, with no STT service in the pipeline. `false` makes it a text LLM for a cascaded pipeline. Defaults to `true`. | | `context` | `LLMContext \| null \| undefined` | no | Context that audio turns are answered against until the first `LLMContextFrame` supplies one. | | `settings` | `NvidiaOmniLLMSettings \| null \| undefined` | no | Runtime-updatable settings. | | `requestTimeoutSecs` | `number \| undefined` | no | HTTP request timeout, in seconds. Defaults to 120. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `runInParallel` | `boolean \| undefined` | no | Whether to run function calls in parallel or sequentially. Defaults to true. | | `groupParallelTools` | `boolean \| undefined` | no | Whether to group parallel function calls so the LLM is triggered exactly once after all calls in the batch complete. When false, each function call result triggers the LLM independently as it arrives. Defaults to true. | | `functionCallTimeoutSecs` | `number \| null \| undefined` | no | Optional timeout in seconds for function calls. A call that runs past it is cancelled: its handler is cancelled (it gets a `CancelledError`), the call is settled as cancelled, and inference runs so the LLM can report that it didn't complete. | | `fileResolver` | `FileResolver \| null \| undefined` | no | Resolves file URLs in the context that this provider can't fetch itself (see `FileResolver`). Pass one configured with a `fileStorage` to resolve uploaded-file URLs (`new FileResolver({ fileStorage: runnerArgs.fileStorage })` when a runner hosts the bot), and share one instance across services that may consume the same files (e.g. services switched between mid-session) so downloads are reused. Defaults to a resolver that handles `data:` and `http(s)` URLs only. | | `organization` | `string \| null \| undefined` | no | OpenAI organization ID. | | `project` | `string \| null \| undefined` | no | OpenAI project ID. | | `defaultHeaders` | `Record \| null \| undefined` | no | Additional HTTP headers. | | `serviceTier` | `string \| null \| undefined` | no | Service tier to use (e.g., "auto", "flex", "priority"). | | `retryTimeoutSecs` | `number \| null \| undefined` | no | How long an inference may go without producing output before it is abandoned and re-issued, when `retryOnTimeout` is set. Defaults to 5.0 seconds. | | | | | --------------- | ----------------------------------------------- | | Base URL | `https://integrate.api.nvidia.com/v1` | | Default model | `nvidia/nemotron-3-nano-omni-30b-a3b-reasoning` | | Request timeout | 120 seconds (`requestTimeoutSecs`) | ### Settings [#settings-1] Passed as `settings: new NvidiaOmniLLMService.Settings({...})` (`NvidiaOmniLLMSettings`): the `NvidiaLLMService` settings, with these defaults and additions. | Setting | Type | Default | Description | | -------------------------- | ---------------- | ------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `model` | `string` | `"nvidia/nemotron-3-nano-omni-30b-a3b-reasoning"` | Model name. | | `temperature` | `number` | `0.6` | Sampling temperature. | | `topP` | `number` | `0.95` | Nucleus sampling. | | `audioResponseInstruction` | `string \| null` | `null` | Instruction sent with the user's speech in audio input mode. `null` uses `DEFAULT_AUDIO_RESPONSE_INSTRUCTION`, which asks for a `` and a `` section. A custom one should keep asking for both. | | `minUserAudioSecs` | `number` | `0.3` | Shortest buffered utterance that starts a turn. | | `preSpeechBufferSecs` | `number` | `0.2` | Audio kept from before the user started speaking, so the start of an utterance isn't clipped. | ### Usage [#usage-1] In audio input mode (the default), Omni takes the place of the STT service. The user aggregator still needs a VAD analyzer, to mark the turns: ```ts title="omni.ts" // Omni hears the user's speech itself, so there is no STT service. const omni = new NvidiaOmniLLMService({ apiKey: process.env.NVIDIA_API_KEY ?? "", settings: new NvidiaOmniLLMService.Settings({ systemInstruction: "You are a helpful assistant in a voice conversation. Keep replies short.", }), }); const tts = new CartesiaTTSService({ apiKey: process.env.CARTESIA_API_KEY ?? "" }); const context = new LLMContext(); const aggregators = new LLMContextAggregatorPair(context, { userParams: new LLMUserAggregatorParams({ vadAnalyzer: new SileroVADAnalyzer() }), }); const userAggregator = aggregators.user(); const pipeline = new Pipeline([ transport.input(), userAggregator, omni, // hears the user and answers tts, transport.output(), aggregators.assistant(), ]); // What the user said is known once Omni has transcribed it. userAggregator.eventHandler( "on_user_turn_message_added", async (_aggregator, message: UserTurnMessageAddedMessage) => { console.log("user said:", message.content); }, ); ``` With `audioInput: false`, Omni is a text LLM for a cascaded pipeline, with an STT service in front of it. Media placed in the context is sent in either mode: ```ts // With an STT service in the pipeline, Omni is a text LLM that also reads media in the context. const textOmni = new NvidiaOmniLLMService({ apiKey: process.env.NVIDIA_API_KEY ?? "", audioInput: false, }); ``` `runMultimodalInference()` runs one request outside the pipeline, for a worker that analyzes media. It returns the text, the reasoning and the finish reason, and can stream deltas to `onTextDelta` and `onReasoningDelta`: ```ts // Analyze a video outside the pipeline, with a cap on the reasoning. async function describeVideo(base64Mp4: string): Promise { const media = new LLMContext({ messages: [ await LLMContext.createFileMessage({ type: "bytes", format: "video/mp4", file: `data:video/mp4;base64,${base64Mp4}`, text: "Describe what happens in this clip.", }), ], }); const result = await textOmni.runMultimodalInference(media, { reasoningBudget: 512, maxTokens: 1024, }); return result.text; } ``` ### Notes [#notes-1] * In audio input mode the service announces itself as a realtime service, so the user aggregator works in realtime mode. It buffers the user's audio between turn boundaries and answers each spoken turn once it ends, after any turn still running. * Omni writes what the user said in a `` section. The service pushes it upstream as a `TranscriptionFrame`, which the user aggregator writes to the context, and sends only the `` section on to TTS. Read the user's text from `on_user_turn_message_added`, not `on_user_turn_stopped`. * The speech of a turn interrupted before Omni transcribed it is answered together with the user's next utterance. * Context frames that need an answer (tool results, developer messages, typed user input) still start a completion, so the `LLMRunFrame` greeting works in both modes. * Messages go through `NvidiaOmniLLMAdapter` (`@pipedog/nvidia/nvidia_omni_adapter`), which sends audio from `LLMContext.createAudioMessage()` and audio and video files from `LLMContext.createFileMessage()` as `audio_url` and `video_url` data URLs. "developer" messages are sent as "user" messages. * `pipedog init` offers `NvidiaLLMService`, not the Omni service, as upstream's CLI does. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | -------------- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | — not verified | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/nvidia/package.json`: ✅ means the compat smoke test passes on that runtime. * The client is the official `openai` JS SDK (through `@pipedog/openai`), which talks through `fetch`. * The deprecated `model` constructor option is not ported. Use `settings`. * Upstream keeps an Omni turn's audio in a Python `ContextVar`. pipedog keeps it per task instead, so tasks started from a turn don't inherit it. Only the turn's own request sends it, as upstream. * Unit-tested with mocked clients (the ported upstream tests), not against NVIDIA itself. # Ollama > OLLamaLLMService streams chat completions from Ollama through its OpenAI-compatible API. Source: https://polvallverdu.github.io/pipedog/docs/services/llm/ollama/ {/* Generated by scripts/generate-providers.ts from data/providers.ts. */} ## Overview [#overview] Ollama runs open models on your own machine, behind an OpenAI-compatible API. * **`OLLamaLLMService`**: an `OpenAILLMService` subclass with Ollama's base URL and default model. Pipecat's documentation for the same service. `packages/ollama/src/llm.ts`. The models you can pull. ## Installation [#installation] ```bash bun add @pipedog/ollama ``` It depends on `@pipedog/openai`, which is installed with it. ```ts import { OLLamaLLMService } from "@pipedog/ollama/llm"; ``` ## Prerequisites [#prerequisites] * [Ollama](https://ollama.com/download) running locally, with a model pulled (`ollama pull llama2`). No API key is needed; `apiKey` defaults to `"ollama"`, which a local Ollama ignores. ## OLLamaLLMService [#ollamallmservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `baseUrl` | `string \| null \| undefined` | no | The base URL for the OLLama API endpoint. Defaults to `"http://localhost:11434/v1"`. | | `apiKey` | `string \| null \| undefined` | no | API key for an Ollama server that requires one, such as one behind an authenticating proxy. A local Ollama ignores it. Defaults to `"ollama"`. | | `settings` | `OllamaLLMSettings \| null \| undefined` | no | Runtime-updatable settings; values override the built-in defaults. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `runInParallel` | `boolean \| undefined` | no | Whether to run function calls in parallel or sequentially. Defaults to true. | | `groupParallelTools` | `boolean \| undefined` | no | Whether to group parallel function calls so the LLM is triggered exactly once after all calls in the batch complete. When false, each function call result triggers the LLM independently as it arrives. Defaults to true. | | `functionCallTimeoutSecs` | `number \| null \| undefined` | no | Optional timeout in seconds for function calls. A call that runs past it is cancelled: its handler is cancelled (it gets a `CancelledError`), the call is settled as cancelled, and inference runs so the LLM can report that it didn't complete. | | `fileResolver` | `FileResolver \| null \| undefined` | no | Resolves file URLs in the context that this provider can't fetch itself (see `FileResolver`). Pass one configured with a `fileStorage` to resolve uploaded-file URLs (`new FileResolver({ fileStorage: runnerArgs.fileStorage })` when a runner hosts the bot), and share one instance across services that may consume the same files (e.g. services switched between mid-session) so downloads are reused. Defaults to a resolver that handles `data:` and `http(s)` URLs only. | | `organization` | `string \| null \| undefined` | no | OpenAI organization ID. | | `project` | `string \| null \| undefined` | no | OpenAI project ID. | | `defaultHeaders` | `Record \| null \| undefined` | no | Additional HTTP headers. | | `serviceTier` | `string \| null \| undefined` | no | Service tier to use (e.g., "auto", "flex", "priority"). | | `retryTimeoutSecs` | `number \| null \| undefined` | no | How long an inference may go without producing output before it is abandoned and re-issued, when `retryOnTimeout` is set. Defaults to 5.0 seconds. | | `retryOnTimeout` | `boolean \| null \| undefined` | no | Whether to re-issue the request once if the first attempt produces no output within `retryTimeoutSecs`. The retry is unbounded. | The other options (`organization`, `project`, `defaultHeaders`, `serviceTier`, `retryOnTimeout`, `retryTimeoutSecs`, function calling) are `OpenAILLMService`'s. See [OpenAI](https://polvallverdu.github.io/pipedog/docs/services/llm/openai/#configuration). | | | | ------------- | ------------------------------------------------------------- | | Base URL | `http://localhost:11434/v1` | | Default model | `llama2` | | Models to try | `llama2`, `mistral`. See the provider's catalog for the rest. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new OLLamaLLMService.Settings({...})` (`OllamaLLMSettings`). They are [OpenAI's settings](https://polvallverdu.github.io/pipedog/docs/services/llm/openai/#settings), with these defaults: | Setting | Type | Default | Description | | ------- | -------- | ---------- | ----------- | | `model` | `string` | `"llama2"` | Model name. | Every other setting keeps OpenAI's default. Unset sampling settings are left out of the request, so the provider's own defaults apply. `extra` carries provider-specific request fields. ### Usage [#usage] ```ts title="llm.ts" import { LLMUpdateSettingsFrame } from "@pipedog/core/frames/frames"; import { OLLamaLLMService } from "@pipedog/ollama/llm"; const llm = new OLLamaLLMService({ settings: new OLLamaLLMService.Settings({ model: "llama2", systemInstruction: "You are a helpful assistant in a voice conversation.", temperature: 0.7, }), }); ``` An Ollama server on another host, behind a proxy that checks a key: ```ts const remote = new OLLamaLLMService({ baseUrl: "https://ollama.example.com/v1", apiKey: process.env.OLLAMA_PROXY_KEY ?? "", settings: new OLLamaLLMService.Settings({ model: "mistral" }), }); ``` Change settings mid-call with an `LLMUpdateSettingsFrame`. The next completion uses them: ```ts // Queue it on the worker, or push it from a processor upstream of the service. const update = new LLMUpdateSettingsFrame({ delta: new OLLamaLLMService.Settings({ temperature: 0.2, maxTokens: 300 }), }); ``` ### Notes [#notes] * Runs a completion for each `LLMContextFrame`, between an `LLMFullResponseStartFrame` and an `LLMFullResponseEndFrame`, and streams the reply as `LLMTextFrame`s, as [`OpenAILLMService`](https://polvallverdu.github.io/pipedog/docs/services/llm/openai.md) does. Function calls, token usage metrics and `runInference` work the same way. * "developer" messages are sent as "user" messages, since the API doesn't support that role (`supportsDeveloperRole` is `false`). * `OLLamaLLMService` doesn't require an API key, unlike the other OpenAI-compatible services, and never falls back to `OPENAI_API_KEY`. * Ollama seems to ignore "developer" messages, so they are sent as "user" messages. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/ollama/package.json`: ✅ means the compat smoke test passes on that runtime. * The client is the official `openai` JS SDK (through `@pipedog/openai`), which talks through `fetch`. It works on Bun, Node, Deno and Cloudflare Workers. * The deprecated `model` constructor option is not ported. Use `settings`. * Tested against a local OpenAI-compatible mock server, not against Ollama itself. # OpenAI Responses > OpenAI's Responses API with OpenAIResponsesLLMService (WebSocket) and OpenAIResponsesHttpLLMService (HTTP), including reasoning models. Source: https://polvallverdu.github.io/pipedog/docs/services/llm/openai-responses/ ## Overview [#overview] The Responses API is OpenAI's newer interface for text generation, tools and reasoning models. pipedog ports both of Pipecat's Responses services: * **`OpenAIResponsesLLMService`**: keeps a WebSocket open to `wss://api.openai.com/v1/responses`. It sends only the new input items with `previous_response_id` when it can, which lowers latency in long, tool-heavy conversations. This is the recommended variant for voice. * **`OpenAIResponsesHttpLLMService`**: the same service over HTTP streaming, through the OpenAI SDK. It sends the full context on every turn. Both store nothing on OpenAI's side (`store: false`). For Chat Completions, see [OpenAI](https://polvallverdu.github.io/pipedog/docs/services/llm/openai.md). Pipecat's documentation for the same services. `packages/openai/src/responses/llm.ts`. The Responses API reference. ## Installation [#installation] ```bash bun add @pipedog/openai ``` ```ts import { OpenAIResponsesHttpLLMService, OpenAIResponsesLLMService, OpenAIResponsesReasoningConfig, } from "@pipedog/openai/responses/llm"; ``` ## Prerequisites [#prerequisites] * An OpenAI account and an API key from the [OpenAI dashboard](https://platform.openai.com/api-keys). * Pass the key as `apiKey`. If you leave it out, both services read `OPENAI_API_KEY`. The examples read it from `OPENAI_API_KEY`. ## OpenAIResponsesLLMService [#openairesponsesllmservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `settings` | `OpenAIResponsesLLMSettings \| null \| undefined` | no | Runtime-updatable settings. | | `wsUrl` | `string \| undefined` | no | WebSocket endpoint URL. Defaults to `wss://api.openai.com/v1/responses`. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `reconnectBackoffMinWait` | `number \| undefined` | no | Minimum time, in seconds, to wait between reconnection attempts. | | `reconnectBackoffMaxWait` | `number \| undefined` | no | Maximum time, in seconds, to wait between reconnection attempts. | | `reconnectOnError` | `boolean \| undefined` | no | Whether to automatically reconnect on connection errors. | | `wsCloseTimeout` | `number \| undefined` | no | Maximum time, in seconds, to wait for the peer to acknowledge the websocket closing handshake before dropping the connection. Applied to connections opened through `_websocketConnect()`. Increase it for peers that need longer to complete a graceful close. | | `wsSendTimeout` | `number \| null \| undefined` | no | Maximum time, in seconds, a single send may wait on a peer that has stopped reading before the connection is dropped and the send throws `ConnectionClosedError`. Applied to connections opened through `_websocketConnect()`. Pass `null` to wait indefinitely. Defaults to `WS_SEND_TIMEOUT`. | | `runInParallel` | `boolean \| undefined` | no | Whether to run function calls in parallel or sequentially. Defaults to true. | | `groupParallelTools` | `boolean \| undefined` | no | Whether to group parallel function calls so the LLM is triggered exactly once after all calls in the batch complete. When false, each function call result triggers the LLM independently as it arrives. Defaults to true. | | `functionCallTimeoutSecs` | `number \| null \| undefined` | no | Optional timeout in seconds for function calls. A call that runs past it is cancelled: its handler is cancelled (it gets a `CancelledError`), the call is settled as cancelled, and inference runs so the LLM can report that it didn't complete. | | `fileResolver` | `FileResolver \| null \| undefined` | no | Resolves file URLs in the context that this provider can't fetch itself (see `FileResolver`). Pass one configured with a `fileStorage` to resolve uploaded-file URLs (`new FileResolver({ fileStorage: runnerArgs.fileStorage })` when a runner hosts the bot), and share one instance across services that may consume the same files (e.g. services switched between mid-session) so downloads are reused. Defaults to a resolver that handles `data:` and `http(s)` URLs only. | | `apiKey` | `string \| null \| undefined` | no | OpenAI API key. | | `baseUrl` | `string \| null \| undefined` | no | Custom base URL for the API. | | `organization` | `string \| null \| undefined` | no | OpenAI organization ID. | | `project` | `string \| null \| undefined` | no | OpenAI project ID. | | `defaultHeaders` | `Record \| null \| undefined` | no | Additional HTTP headers. | | `serviceTier` | `string \| null \| undefined` | no | Service tier to use: "auto", "default", "flex", "scale", "fast", "priority" or "ultrafast". "fast" is OpenAI's low-latency tier, the name that replaced "priority"; both values are accepted. "ultrafast" is the access-controlled Ultrafast Processing tier, which shortens the time between output tokens on the models that offer it. | | `retryTimeoutSecs` | `number \| null \| undefined` | no | How long an inference may go without producing output before it is abandoned and re-issued, when `retryOnTimeout` is set. Defaults to 5.0 seconds. | | `retryOnTimeout` | `boolean \| null \| undefined` | no | Whether to re-issue the request once if the first attempt produces no output within `retryTimeoutSecs`. The retry is unbounded. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new OpenAIResponsesLLMService.Settings({...})` (`OpenAIResponsesLLMSettings`). Both services share them. Any setting you leave out keeps the default below. Fields marked "not sent" are left out of the request, so OpenAI's own default applies. | Setting | Type | Default | Description | | ------------------------------------------------------------------ | ---------------------------------------- | ----------- | ------------------------------------------------------------- | | `model` | `string` | `"gpt-4.1"` | Model. | | `systemInstruction` | `string \| null` | `null` | Sent as `instructions`. | | `temperature` | `number` | not sent | Sampling temperature. | | `topP` | `number` | not sent | Nucleus sampling. | | `maxCompletionTokens` | `number` | not sent | Sent as `max_output_tokens`. | | `reasoning` | `OpenAIResponsesReasoningConfig \| null` | `null` | Reasoning config, for reasoning models. See below. | | `frequencyPenalty`, `presencePenalty`, `seed`, `topK`, `maxTokens` | `number \| null` | `null` | Not part of the Responses request. | | `filterIncompleteUserTurns` | `boolean` | `false` | Filter incomplete user turns. | | `userTurnCompletionConfig` | `UserTurnCompletionConfig \| null` | `null` | Turn completion config used with `filterIncompleteUserTurns`. | | `extra` | `Record` | `{}` | Extra request fields, merged in after the fields above. | `OpenAIResponsesReasoningConfig` (also `OpenAIResponsesLLMService.ReasoningConfig`) takes: | Field | Type | Default | Description | | --------- | ------------------------------------------------------------------------------------------ | ------- | -------------------------------------------------- | | `effort` | `"none" \| "minimal" \| "low" \| "medium" \| "high" \| "xhigh" \| "max" \| string \| null` | `null` | Reasoning effort. `null` leaves it to the model. | | `summary` | `"auto" \| "concise" \| "detailed" \| string \| null` | `null` | Reasoning summary to return. `null` asks for none. | | `mode` | `"standard" \| "pro" \| string \| null` | `null` | Reasoning mode, for models that have one. | ### Usage [#usage] ```ts title="responses.ts" import { OpenAIResponsesHttpLLMService, OpenAIResponsesLLMService, OpenAIResponsesReasoningConfig, } from "@pipedog/openai/responses/llm"; const llm = new OpenAIResponsesLLMService({ apiKey: process.env.OPENAI_API_KEY ?? "", settings: new OpenAIResponsesLLMService.Settings({ model: "gpt-4.1", systemInstruction: "You are a helpful assistant in a voice conversation.", }), }); ``` A reasoning model, with a reasoning summary: ```ts const reasoning = new OpenAIResponsesLLMService({ apiKey: process.env.OPENAI_API_KEY ?? "", settings: new OpenAIResponsesLLMService.Settings({ model: "gpt-5.4-mini", reasoning: new OpenAIResponsesReasoningConfig({ effort: "low", summary: "auto" }), }), }); ``` ### Event handlers [#event-handlers] | Event | Description | | ----------------------------- | ----------------------------------------------- | | `on_completion_timeout` | A completion timed out. | | `on_function_calls_started` | The model asked for one or more function calls. | | `on_function_calls_cancelled` | Function calls were cancelled. | | `on_connection_error` | The WebSocket failed (WebSocket variant only). | These come from the `LLMService` and `WebsocketLLMService` base classes. ### Notes [#notes] * Runs a response for each `LLMContextFrame`, between an `LLMFullResponseStartFrame` and an `LLMFullResponseEndFrame`, and streams the reply as `LLMTextFrame`s. * `wsUrl` sets the WebSocket endpoint. `baseUrl` only affects the HTTP client the service uses for `runInference`. * When the context still starts with what was sent last time, only the new items go out, with `previous_response_id`. After a reconnect, or when the server no longer has the previous response, the service sends the full context again. It also reconnects and retries when it hits OpenAI's 60 minute connection limit. * With `reasoning` left `null`, the service sends the lowest effort OpenAI documents for a known reasoning model (for example `"none"` for `gpt-5.4` and `gpt-6.1-terra`, `"low"` for `o3`), for the fastest replies, and logs what it sent. Set `useProviderThinkingDefault` to keep OpenAI's default. `gpt-4.1`, the default model, does not reason. * Reasoning summaries come out as `LLMThoughtStartFrame`, `LLMThoughtTextFrame` and `LLMThoughtEndFrame` (the assistant aggregator's `on_assistant_thought` event). The encrypted reasoning is kept in the context and sent back on later turns. * `serviceTier` accepts `"auto"`, `"default"`, `"flex"`, `"scale"`, `"fast"`, `"priority"` and `"ultrafast"`. ## OpenAIResponsesHttpLLMService [#openairesponseshttpllmservice] ### Configuration [#configuration-1] | Option | Type | Required | Description | |---|---|---|---| | `serviceTier` | `string \| null \| undefined` | no | Service tier to use: "auto", "default", "flex", "scale", "fast", "priority" or "ultrafast". "fast" is OpenAI's low-latency tier, the name that replaced "priority"; both values are accepted. "ultrafast" is the access-controlled Ultrafast Processing tier, which shortens the time between output tokens on the models that offer it. | | `settings` | `OpenAIResponsesLLMSettings \| null \| undefined` | no | Runtime-updatable settings. | | `retryTimeoutSecs` | `number \| null \| undefined` | no | How long an inference may go without producing output before it is abandoned and re-issued, when `retryOnTimeout` is set. Defaults to 5.0 seconds. | | `retryOnTimeout` | `boolean \| null \| undefined` | no | Whether to re-issue the request once if the first attempt produces no output within `retryTimeoutSecs`. The retry is unbounded. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `runInParallel` | `boolean \| undefined` | no | Whether to run function calls in parallel or sequentially. Defaults to true. | | `groupParallelTools` | `boolean \| undefined` | no | Whether to group parallel function calls so the LLM is triggered exactly once after all calls in the batch complete. When false, each function call result triggers the LLM independently as it arrives. Defaults to true. | | `functionCallTimeoutSecs` | `number \| null \| undefined` | no | Optional timeout in seconds for function calls. A call that runs past it is cancelled: its handler is cancelled (it gets a `CancelledError`), the call is settled as cancelled, and inference runs so the LLM can report that it didn't complete. | | `fileResolver` | `FileResolver \| null \| undefined` | no | Resolves file URLs in the context that this provider can't fetch itself (see `FileResolver`). Pass one configured with a `fileStorage` to resolve uploaded-file URLs (`new FileResolver({ fileStorage: runnerArgs.fileStorage })` when a runner hosts the bot), and share one instance across services that may consume the same files (e.g. services switched between mid-session) so downloads are reused. Defaults to a resolver that handles `data:` and `http(s)` URLs only. | | `apiKey` | `string \| null \| undefined` | no | OpenAI API key. | | `baseUrl` | `string \| null \| undefined` | no | Custom base URL for the API. | | `organization` | `string \| null \| undefined` | no | OpenAI organization ID. | | `project` | `string \| null \| undefined` | no | OpenAI project ID. | | `defaultHeaders` | `Record \| null \| undefined` | no | Additional HTTP headers. | ### Settings [#settings-1] Same as `OpenAIResponsesLLMService` above. ### Usage [#usage-1] ```ts title="responses-http.ts" const http = new OpenAIResponsesHttpLLMService({ apiKey: process.env.OPENAI_API_KEY ?? "", settings: new OpenAIResponsesHttpLLMService.Settings({ model: "gpt-4.1", systemInstruction: "You are a helpful assistant.", }), }); ``` ### Notes [#notes-1] * Each turn sends the whole context. Over HTTP, `previous_response_id` needs `store: true`, which turns on OpenAI-side conversation storage, so this variant doesn't use it. * `retryOnTimeout` and `retryTimeoutSecs` work as in [OpenAI](https://polvallverdu.github.io/pipedog/docs/services/llm/openai.md): one re-issue when the first attempt produces nothing in time. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/openai/package.json`: ✅ means the compat smoke test passes on that runtime. * `OpenAIResponsesHttpLLMService` uses the `openai` JS SDK over `fetch`, which works on Bun, Node, Deno and Cloudflare Workers. * `OpenAIResponsesLLMService` sends the key as an `Authorization` header on the WebSocket handshake. Bun, Node and Deno send it from the global `WebSocket`; on Cloudflare Workers core opens the socket through `fetch`. * Upstream's mixin base `_BaseOpenAIResponsesLLMService` is a class-factory mixin here (`BaseOpenAIResponsesLLMServiceMixin`). `instanceof _BaseOpenAIResponsesLLMService` holds for both services. # OpenAI > Streaming chat completions with OpenAILLMService, the base for every OpenAI-compatible API. Source: https://polvallverdu.github.io/pipedog/docs/services/llm/openai/ ## Overview [#overview] `OpenAILLMService` streams replies from OpenAI's Chat Completions API, with function calling, token usage metrics and out-of-band inference (`runInference`). * **`OpenAILLMService`**: the OpenAI chat service. It extends `BaseOpenAILLMService`, which builds the client and the request. Point `baseUrl` at any server that speaks the Chat Completions API (Ollama, vLLM, OpenRouter, and so on) to use it with that provider. For the Responses API, see [OpenAI Responses](https://polvallverdu.github.io/pipedog/docs/services/llm/openai-responses.md). Pipecat's documentation for the same service. `packages/openai/src/llm.ts` and `base_llm.ts`. The Chat Completions API reference. ## Installation [#installation] ```bash bun add @pipedog/openai ``` ```ts import { OpenAILLMService } from "@pipedog/openai/llm"; import { OpenAILLMSettings } from "@pipedog/openai/base_llm"; ``` ## Prerequisites [#prerequisites] * An OpenAI account and an API key from the [OpenAI dashboard](https://platform.openai.com/api-keys). * Pass the key as `apiKey`. If you leave it out, the OpenAI SDK reads `OPENAI_API_KEY`. The examples read it from `OPENAI_API_KEY`. ## OpenAILLMService [#openaillmservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `runInParallel` | `boolean \| undefined` | no | Whether to run function calls in parallel or sequentially. Defaults to true. | | `groupParallelTools` | `boolean \| undefined` | no | Whether to group parallel function calls so the LLM is triggered exactly once after all calls in the batch complete. When false, each function call result triggers the LLM independently as it arrives. Defaults to true. | | `functionCallTimeoutSecs` | `number \| null \| undefined` | no | Optional timeout in seconds for function calls. A call that runs past it is cancelled: its handler is cancelled (it gets a `CancelledError`), the call is settled as cancelled, and inference runs so the LLM can report that it didn't complete. | | `fileResolver` | `FileResolver \| null \| undefined` | no | Resolves file URLs in the context that this provider can't fetch itself (see `FileResolver`). Pass one configured with a `fileStorage` to resolve uploaded-file URLs (`new FileResolver({ fileStorage: runnerArgs.fileStorage })` when a runner hosts the bot), and share one instance across services that may consume the same files (e.g. services switched between mid-session) so downloads are reused. Defaults to a resolver that handles `data:` and `http(s)` URLs only. | | `apiKey` | `string \| null \| undefined` | no | OpenAI API key. If null, the SDK reads `OPENAI_API_KEY`. | | `baseUrl` | `string \| null \| undefined` | no | Custom base URL for the API. | | `organization` | `string \| null \| undefined` | no | OpenAI organization ID. | | `project` | `string \| null \| undefined` | no | OpenAI project ID. | | `defaultHeaders` | `Record \| null \| undefined` | no | Additional HTTP headers. | | `serviceTier` | `string \| null \| undefined` | no | Service tier to use (e.g., "auto", "flex", "priority"). | | `settings` | `OpenAILLMSettings \| null \| undefined` | no | Runtime-updatable settings. When provided, its given fields override the service's defaults. | | `retryTimeoutSecs` | `number \| null \| undefined` | no | How long an inference may go without producing output before it is abandoned and re-issued, when `retryOnTimeout` is set. Defaults to 5.0 seconds. | | `retryOnTimeout` | `boolean \| null \| undefined` | no | Whether to re-issue the request once if the first attempt produces no output within `retryTimeoutSecs`. The retry is unbounded. | The client options (`apiKey`, `baseUrl`, `organization`, `project`, `defaultHeaders`) come from `OpenAIClientOptions`: | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string \| null \| undefined` | no | OpenAI API key. If null, the SDK reads `OPENAI_API_KEY`. | | `baseUrl` | `string \| null \| undefined` | no | Custom base URL for the API. | | `organization` | `string \| null \| undefined` | no | OpenAI organization ID. | | `project` | `string \| null \| undefined` | no | OpenAI project ID. | | `defaultHeaders` | `Record \| null \| undefined` | no | Additional HTTP headers. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new OpenAILLMService.Settings({...})` (`OpenAILLMSettings`). Any setting you leave out keeps the default below. Fields marked "not sent" hold the OpenAI SDK's sentinel and are left out of the request, so OpenAI's own default applies. | Setting | Type | Default | Description | | --------------------------- | ---------------------------------- | ----------- | ------------------------------------------------------------------------------------ | | `model` | `string` | `"gpt-4.1"` | Chat model. | | `systemInstruction` | `string \| null` | `null` | System prompt, added ahead of the context's messages. | | `temperature` | `number \| null` | not sent | Sampling temperature. | | `topP` | `number \| null` | not sent | Nucleus sampling. | | `maxTokens` | `number \| null` | not sent | `max_tokens`. | | `maxCompletionTokens` | `number \| null` | not sent | `max_completion_tokens`. | | `frequencyPenalty` | `number \| null` | not sent | Frequency penalty. | | `presencePenalty` | `number \| null` | not sent | Presence penalty. | | `seed` | `number \| null` | not sent | Sampling seed. | | `topK` | `number \| null` | `null` | Not part of the OpenAI request. | | `filterIncompleteUserTurns` | `boolean` | `false` | Filter incomplete user turns. | | `userTurnCompletionConfig` | `UserTurnCompletionConfig \| null` | `null` | Turn completion config used with `filterIncompleteUserTurns`. | | `extra` | `Record` | `{}` | Extra request fields, merged into the request last (they override the fields above). | ### Usage [#usage] ```ts title="llm.ts" import { LLMUpdateSettingsFrame } from "@pipedog/core/frames/frames"; import { OpenAILLMService } from "@pipedog/openai/llm"; const llm = new OpenAILLMService({ apiKey: process.env.OPENAI_API_KEY ?? "", settings: new OpenAILLMService.Settings({ model: "gpt-4.1", systemInstruction: "You are a helpful assistant in a voice conversation.", temperature: 0.7, }), }); ``` An OpenAI-compatible server, through `baseUrl`: ```ts // Any server that speaks the Chat Completions API, here a local Ollama. const local = new OpenAILLMService({ apiKey: "ollama", baseUrl: "http://localhost:11434/v1", settings: new OpenAILLMService.Settings({ model: "llama3.2" }), }); ``` Re-issue a request that produces no output in time: ```ts const withRetry = new OpenAILLMService({ apiKey: process.env.OPENAI_API_KEY ?? "", retryOnTimeout: true, retryTimeoutSecs: 3, }); ``` Change settings mid-call with an `LLMUpdateSettingsFrame`. The next completion uses them: ```ts // Queue it on the worker, or push it from a processor upstream of the service. const update = new LLMUpdateSettingsFrame({ delta: new OpenAILLMService.Settings({ temperature: 0.2, maxCompletionTokens: 300 }), }); ``` ### Event handlers [#event-handlers] | Event | Description | | ----------------------------- | ----------------------------------------------- | | `on_completion_timeout` | A completion timed out. | | `on_function_calls_started` | The model asked for one or more function calls. | | `on_function_calls_cancelled` | Function calls were cancelled. | These come from the `LLMService` base class. ### Notes [#notes] * Runs a completion for each `LLMContextFrame`, between an `LLMFullResponseStartFrame` and an `LLMFullResponseEndFrame`, and streams the reply as `LLMTextFrame`s. Other frames pass through. * Requests always stream, with `stream_options: { include_usage: true }`, so token usage metrics are reported. * `serviceTier` is sent as `service_tier` when set. * `retryOnTimeout` is off by default. With it on, an attempt that produces nothing within `retryTimeoutSecs` (5 seconds by default) is abandoned and re-issued once. * Models older than `gpt-4o-mini` and `gpt-4o-2024-08-06` can't enforce a response schema, so `runInference` ignores a `responseSchema` for them, with a warning. * OpenAI-compatible subclasses override `createClient()` to build the client their own way, and can set `supportsDeveloperRole = false` to send "developer" messages as "user" messages. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/openai/package.json`: ✅ means the compat smoke test passes on that runtime. * The client is the official `openai` JS SDK, which talks through `fetch`. It works on Bun, Node, Deno and Cloudflare Workers. * Upstream builds the client with an httpx connection pool. `fetch` has no such object, so connection reuse is left to the runtime. * The JS SDK has no `extra_body` option. Provider-specific `extra_body` fields are merged into the request JSON just before it is sent, as the Python client does. * The deprecated `model` and `params` constructor options and `BaseOpenAILLMService.InputParams` are not ported. Use `settings`. * For Azure OpenAI, use [`AzureLLMService`](https://polvallverdu.github.io/pipedog/docs/services/llm/azure.md). Other OpenAI-compatible providers have their own subclasses where ported (see [LLM](https://polvallverdu.github.io/pipedog/docs/services/llm.md)); for the rest, `OpenAILLMService` with `baseUrl` covers most of them. # OpenRouter > OpenRouterLLMService streams chat completions from OpenRouter through its OpenAI-compatible API. Source: https://polvallverdu.github.io/pipedog/docs/services/llm/openrouter/ {/* Generated by scripts/generate-providers.ts from data/providers.ts. */} ## Overview [#overview] OpenRouter puts one OpenAI-compatible API in front of many model providers, and routes each request to one of them. * **`OpenRouterLLMService`**: an `OpenAILLMService` subclass with OpenRouter's base URL and default model. Pipecat's documentation for the same service. `packages/openrouter/src/llm.ts`. The chat completion API reference. The provider preferences and their semantics. ## Installation [#installation] ```bash bun add @pipedog/openrouter ``` It depends on `@pipedog/openai`, which is installed with it. ```ts import { OpenRouterLLMService } from "@pipedog/openrouter/llm"; ``` ## Prerequisites [#prerequisites] * An OpenRouter account, with credits, and an API key from [OpenRouter](https://openrouter.ai/). * Pass the key as `apiKey`. It is required, and the service never falls back to `OPENAI_API_KEY`. The examples read it from `OPENROUTER_API_KEY`. ## OpenRouterLLMService [#openrouterllmservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | The API key for accessing OpenRouter's API. | | `baseUrl` | `string \| null \| undefined` | no | The base URL for OpenRouter API. Defaults to `"https://openrouter.ai/api/v1"`. | | `settings` | `OpenRouterLLMSettings \| null \| undefined` | no | Runtime-updatable settings; values override the built-in defaults. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `runInParallel` | `boolean \| undefined` | no | Whether to run function calls in parallel or sequentially. Defaults to true. | | `groupParallelTools` | `boolean \| undefined` | no | Whether to group parallel function calls so the LLM is triggered exactly once after all calls in the batch complete. When false, each function call result triggers the LLM independently as it arrives. Defaults to true. | | `functionCallTimeoutSecs` | `number \| null \| undefined` | no | Optional timeout in seconds for function calls. A call that runs past it is cancelled: its handler is cancelled (it gets a `CancelledError`), the call is settled as cancelled, and inference runs so the LLM can report that it didn't complete. | | `fileResolver` | `FileResolver \| null \| undefined` | no | Resolves file URLs in the context that this provider can't fetch itself (see `FileResolver`). Pass one configured with a `fileStorage` to resolve uploaded-file URLs (`new FileResolver({ fileStorage: runnerArgs.fileStorage })` when a runner hosts the bot), and share one instance across services that may consume the same files (e.g. services switched between mid-session) so downloads are reused. Defaults to a resolver that handles `data:` and `http(s)` URLs only. | | `organization` | `string \| null \| undefined` | no | OpenAI organization ID. | | `project` | `string \| null \| undefined` | no | OpenAI project ID. | | `defaultHeaders` | `Record \| null \| undefined` | no | Additional HTTP headers. | | `serviceTier` | `string \| null \| undefined` | no | Service tier to use (e.g., "auto", "flex", "priority"). | | `retryTimeoutSecs` | `number \| null \| undefined` | no | How long an inference may go without producing output before it is abandoned and re-issued, when `retryOnTimeout` is set. Defaults to 5.0 seconds. | | `retryOnTimeout` | `boolean \| null \| undefined` | no | Whether to re-issue the request once if the first attempt produces no output within `retryTimeoutSecs`. The retry is unbounded. | The other options (`organization`, `project`, `defaultHeaders`, `serviceTier`, `retryOnTimeout`, `retryTimeoutSecs`, function calling) are `OpenAILLMService`'s. See [OpenAI](https://polvallverdu.github.io/pipedog/docs/services/llm/openai/#configuration). | | | | ------------- | --------------------------------------------------------------------------------------- | | Base URL | `https://openrouter.ai/api/v1` | | Default model | `openai/gpt-4.1` | | Models to try | `openai/gpt-4.1`, `anthropic/claude-sonnet-4`. See the provider's catalog for the rest. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new OpenRouterLLMService.Settings({...})` (`OpenRouterLLMSettings`). They are [OpenAI's settings](https://polvallverdu.github.io/pipedog/docs/services/llm/openai/#settings), with these defaults: | Setting | Type | Default | Description | | ---------- | --------------------------------------- | ------------------ | -------------------------------------------------------------------------------------------------------------------- | | `model` | `string` | `"openai/gpt-4.1"` | Model name. | | `provider` | `OpenRouterProviderPreferences \| null` | `null` | Which upstream providers may serve the request, sent as `provider` in the request body. A plain object is converted. | Every other setting keeps OpenAI's default. Unset sampling settings are left out of the request, so the provider's own defaults apply. `extra` carries provider-specific request fields. ### Usage [#usage] ```ts title="llm.ts" import { LLMUpdateSettingsFrame } from "@pipedog/core/frames/frames"; import { OpenRouterLLMService } from "@pipedog/openrouter/llm"; const llm = new OpenRouterLLMService({ apiKey: process.env.OPENROUTER_API_KEY ?? "", settings: new OpenRouterLLMService.Settings({ model: "openai/gpt-4.1", systemInstruction: "You are a helpful assistant in a voice conversation.", temperature: 0.7, }), }); ``` Constrain which providers serve the request with `OpenRouterLLMService.ProviderPreferences`. Its options are camelCase (`allowFallbacks`, `preferredMaxLatency`, ...); options it doesn't know are sent as given: ```ts const routed = new OpenRouterLLMService({ apiKey: process.env.OPENROUTER_API_KEY ?? "", settings: new OpenRouterLLMService.Settings({ model: "anthropic/claude-sonnet-4", provider: new OpenRouterLLMService.ProviderPreferences({ sort: "latency", preferredMaxLatency: { p90: 1.5 }, allowFallbacks: true, }), }), }); ``` Change settings mid-call with an `LLMUpdateSettingsFrame`. The next completion uses them: ```ts // Queue it on the worker, or push it from a processor upstream of the service. const update = new LLMUpdateSettingsFrame({ delta: new OpenRouterLLMService.Settings({ temperature: 0.2, maxTokens: 300 }), }); ``` ### Notes [#notes] * Runs a completion for each `LLMContextFrame`, between an `LLMFullResponseStartFrame` and an `LLMFullResponseEndFrame`, and streams the reply as `LLMTextFrame`s, as [`OpenAILLMService`](https://polvallverdu.github.io/pipedog/docs/services/llm/openai.md) does. Function calls, token usage metrics and `runInference` work the same way. * "developer" messages are sent as "user" messages, since the API doesn't support that role (`supportsDeveloperRole` is `false`). * For Gemini models, every system message after the first is sent as a user message. * A `provider` in `extra.extra_body` wins over the `provider` setting. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/openrouter/package.json`: ✅ means the compat smoke test passes on that runtime. * The client is the official `openai` JS SDK (through `@pipedog/openai`), which talks through `fetch`. It works on Bun, Node, Deno and Cloudflare Workers. * The deprecated `model` constructor option is not ported. Use `settings`. * Tested against a local OpenAI-compatible mock server, not against OpenRouter itself. # Perplexity > PerplexityAgentHttpLLMService answers with web search through Perplexity's Agent API, over the OpenAI Responses protocol. Source: https://polvallverdu.github.io/pipedog/docs/services/llm/perplexity/ ## Overview [#overview] Perplexity's Agent API answers with web search, using models from Perplexity and from other providers (`openai/...`, `anthropic/...` and so on). It speaks the OpenAI Responses protocol over HTTP. * **`PerplexityAgentHttpLLMService`**: an [`OpenAIResponsesHttpLLMService`](https://polvallverdu.github.io/pipedog/docs/services/llm/openai-responses/#openairesponseshttpllmservice) subclass with Perplexity's base URL, a default model, and Perplexity's own settings (`preset`, `webSearch`, `maxSteps`). Function calling, multi-turn context, token usage metrics and `runInference` work as they do with the OpenAI Responses services. Upstream's `PerplexityLLMService`, for the Sonar chat completions API, is deprecated since Pipecat 1.13.0, as Perplexity no longer supports that API. pipedog doesn't port it. Use `PerplexityAgentHttpLLMService`. Pipecat's documentation for the same service. `packages/perplexity/src/llm.ts`. The Agent API, its presets and its models. ## Installation [#installation] ```bash bun add @pipedog/perplexity ``` It depends on `@pipedog/openai`, which is installed with it. ```ts import { PerplexityAgentHttpLLMService } from "@pipedog/perplexity/llm"; ``` ## Prerequisites [#prerequisites] * A Perplexity account and an API key from [Perplexity](https://www.perplexity.ai/). * Pass the key as `apiKey`. The examples read it from `PERPLEXITY_API_KEY`. If you leave it out, the OpenAI client falls back to `OPENAI_API_KEY`, as with the OpenAI Responses services, which is not a Perplexity key. ## PerplexityAgentHttpLLMService [#perplexityagenthttpllmservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string \| null \| undefined` | no | The API key for accessing Perplexity's API. | | `baseUrl` | `string \| null \| undefined` | no | The base URL for Perplexity's API. Defaults to `"https://api.perplexity.ai/v1"`. | | `settings` | `PerplexityAgentLLMSettings \| null \| undefined` | no | Runtime-updatable settings. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `runInParallel` | `boolean \| undefined` | no | Whether to run function calls in parallel or sequentially. Defaults to true. | | `groupParallelTools` | `boolean \| undefined` | no | Whether to group parallel function calls so the LLM is triggered exactly once after all calls in the batch complete. When false, each function call result triggers the LLM independently as it arrives. Defaults to true. | | `functionCallTimeoutSecs` | `number \| null \| undefined` | no | Optional timeout in seconds for function calls. A call that runs past it is cancelled: its handler is cancelled (it gets a `CancelledError`), the call is settled as cancelled, and inference runs so the LLM can report that it didn't complete. | | `fileResolver` | `FileResolver \| null \| undefined` | no | Resolves file URLs in the context that this provider can't fetch itself (see `FileResolver`). Pass one configured with a `fileStorage` to resolve uploaded-file URLs (`new FileResolver({ fileStorage: runnerArgs.fileStorage })` when a runner hosts the bot), and share one instance across services that may consume the same files (e.g. services switched between mid-session) so downloads are reused. Defaults to a resolver that handles `data:` and `http(s)` URLs only. | | `organization` | `string \| null \| undefined` | no | OpenAI organization ID. | | `project` | `string \| null \| undefined` | no | OpenAI project ID. | | `defaultHeaders` | `Record \| null \| undefined` | no | Additional HTTP headers. | | `serviceTier` | `string \| null \| undefined` | no | Service tier to use: "auto", "default", "flex", "scale", "fast", "priority" or "ultrafast". "fast" is OpenAI's low-latency tier, the name that replaced "priority"; both values are accepted. "ultrafast" is the access-controlled Ultrafast Processing tier, which shortens the time between output tokens on the models that offer it. | | `retryTimeoutSecs` | `number \| null \| undefined` | no | How long an inference may go without producing output before it is abandoned and re-issued, when `retryOnTimeout` is set. Defaults to 5.0 seconds. | | `retryOnTimeout` | `boolean \| null \| undefined` | no | Whether to re-issue the request once if the first attempt produces no output within `retryTimeoutSecs`. The retry is unbounded. | The other options (`organization`, `project`, `defaultHeaders`, `serviceTier`, `retryOnTimeout`, `retryTimeoutSecs`) are `OpenAIResponsesHttpLLMService`'s. See [OpenAI Responses](https://polvallverdu.github.io/pipedog/docs/services/llm/openai-responses/#openairesponseshttpllmservice). | | | | ------------- | ------------------------------ | | Base URL | `https://api.perplexity.ai/v1` | | Default model | `openai/gpt-5.6-luna` | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new PerplexityAgentHttpLLMService.Settings({...})` (`PerplexityAgentLLMSettings`). They are the [OpenAI Responses settings](https://polvallverdu.github.io/pipedog/docs/services/llm/openai-responses/#settings), with these defaults and additions: | Setting | Type | Default | Description | | ----------- | ---------------------------------------- | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `model` | `string \| null` | `"openai/gpt-5.6-luna"` | Model, as `/`. With a `preset`, it replaces the preset's model; set it to `null` to use the preset's. | | `preset` | `string \| null` | `null` | Perplexity preset, such as `"fast"`, `"low"`, `"medium"`, `"high"` or `"xhigh"`. It sets the model, its instructions, its search budget and how much it reasons. Sent as `preset` in the request body. | | `webSearch` | `boolean` | `true` | Offer the `web_search` tool on every request, alongside any tools from the context. A preset searches the web whatever this is set to. | | `maxSteps` | `number \| null` | `null` | Maximum number of search and reasoning steps per response, sent as `max_steps`. `null` leaves Perplexity's default. | | `reasoning` | `OpenAIResponsesReasoningConfig \| null` | `null` | Reasoning config (`PerplexityAgentHttpLLMService.ReasoningConfig`). See the default below. | Every other setting (`systemInstruction`, `temperature`, `topP`, `maxCompletionTokens`, `extra`, ...) keeps its OpenAI Responses default. The settings need a `model`, a `preset`, or both: the constructor throws when they leave neither, and a settings update that leaves neither pushes an error. ### Usage [#usage] ```ts title="llm.ts" import { LLMUpdateSettingsFrame } from "@pipedog/core/frames/frames"; import { PerplexityAgentHttpLLMService } from "@pipedog/perplexity/llm"; const llm = new PerplexityAgentHttpLLMService({ apiKey: process.env.PERPLEXITY_API_KEY ?? "", settings: new PerplexityAgentHttpLLMService.Settings({ systemInstruction: "You are a helpful assistant in a voice conversation. Keep replies short.", }), }); ``` Let a preset choose the model: ```ts // The "high" preset chooses the model, its instructions, its search budget and its reasoning. // `model: null` clears the default model, so the preset's own is used. const thorough = new PerplexityAgentHttpLLMService({ apiKey: process.env.PERPLEXITY_API_KEY ?? "", settings: new PerplexityAgentHttpLLMService.Settings({ preset: "high", model: null }), }); ``` Pin a model, cap the search steps and set the reasoning effort yourself: ```ts // Another provider's model, with web search capped at three steps and a set reasoning effort. const pinned = new PerplexityAgentHttpLLMService({ apiKey: process.env.PERPLEXITY_API_KEY ?? "", settings: new PerplexityAgentHttpLLMService.Settings({ model: "openai/gpt-5.5", maxSteps: 3, reasoning: new PerplexityAgentHttpLLMService.ReasoningConfig({ effort: "low" }), }), }); ``` Change settings mid-call with an `LLMUpdateSettingsFrame`. The next request uses them: ```ts // Queue it on the worker, or push it from a processor upstream of the service. const update = new LLMUpdateSettingsFrame({ delta: new PerplexityAgentHttpLLMService.Settings({ webSearch: false, maxSteps: 1 }), }); ``` ### Notes [#notes] * Runs a response for each `LLMContextFrame`, between an `LLMFullResponseStartFrame` and an `LLMFullResponseEndFrame`, and streams the reply as `LLMTextFrame`s. Each turn sends the whole context, as `OpenAIResponsesHttpLLMService` does. * When neither `preset` nor `reasoning` is set, the service sends Perplexity's `"fast"` preset, its lowest-latency configuration, with your model in place of the preset's own, and logs it on the first request for each model. A preset or `reasoning` you set (in the settings, or in `extra`) is sent as it is. Set `useProviderThinkingDefault` to send no default and run at Perplexity's. * Perplexity runs its web search on the server. The search shows up in the stream as a function call, which the service drops, so only your own functions are run. * Answers can carry citation markers such as `[1]`. Filter them out before TTS if you don't want them read aloud. * Anthropic models need `maxCompletionTokens` set. * Tokens written to the prompt cache are reported as `cacheCreationInputTokens` in token usage metrics. * `PerplexityLLMAdapter` (`@pipedog/perplexity/perplexity_adapter`) is still exported, as upstream keeps it for the deprecated Sonar service, but `PerplexityAgentHttpLLMService` doesn't use it: its messages go through the OpenAI Responses adapter. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/perplexity/package.json`: ✅ means the compat smoke test passes on that runtime. * The client is the official `openai` JS SDK (through `@pipedog/openai`), which talks through `fetch`. It works on Bun, Node, Deno and Cloudflare Workers. * The JS SDK has no `extra_body` request option, so the service's client merges the request's `extra_body` (where `preset` and `max_steps` go) into the request JSON. * Unit-tested on the requests it builds and on scripted event streams, not against Perplexity itself. # Qwen > QwenLLMService streams chat completions from Qwen through its OpenAI-compatible API. Source: https://polvallverdu.github.io/pipedog/docs/services/llm/qwen/ {/* Generated by scripts/generate-providers.ts from data/providers.ts. */} ## Overview [#overview] Alibaba Cloud's Qwen models, through DashScope's OpenAI-compatible mode. * **`QwenLLMService`**: an `OpenAILLMService` subclass with Qwen's base URL and default model. Pipecat's documentation for the same service. `packages/qwen/src/llm.ts`. Calling Qwen through the API. ## Installation [#installation] ```bash bun add @pipedog/qwen ``` It depends on `@pipedog/openai`, which is installed with it. ```ts import { QwenLLMService } from "@pipedog/qwen/llm"; ``` ## Prerequisites [#prerequisites] * An Alibaba Cloud account and a DashScope API key from [Alibaba Cloud Model Studio](https://www.alibabacloud.com/). * Pass the key as `apiKey`. It is required, and the service never falls back to `OPENAI_API_KEY`. The examples read it from `QWEN_API_KEY`. ## QwenLLMService [#qwenllmservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `apiKey` | `string` | yes | The API key for accessing Qwen's API (DashScope API key). | | `baseUrl` | `string \| null \| undefined` | no | The base URL for the API. Defaults to `"https://dashscope-intl.aliyuncs.com/compatible-mode/v1"`. | | `settings` | `QwenLLMSettings \| null \| undefined` | no | Runtime-updatable settings; values override the built-in defaults. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `runInParallel` | `boolean \| undefined` | no | Whether to run function calls in parallel or sequentially. Defaults to true. | | `groupParallelTools` | `boolean \| undefined` | no | Whether to group parallel function calls so the LLM is triggered exactly once after all calls in the batch complete. When false, each function call result triggers the LLM independently as it arrives. Defaults to true. | | `functionCallTimeoutSecs` | `number \| null \| undefined` | no | Optional timeout in seconds for function calls. A call that runs past it is cancelled: its handler is cancelled (it gets a `CancelledError`), the call is settled as cancelled, and inference runs so the LLM can report that it didn't complete. | | `fileResolver` | `FileResolver \| null \| undefined` | no | Resolves file URLs in the context that this provider can't fetch itself (see `FileResolver`). Pass one configured with a `fileStorage` to resolve uploaded-file URLs (`new FileResolver({ fileStorage: runnerArgs.fileStorage })` when a runner hosts the bot), and share one instance across services that may consume the same files (e.g. services switched between mid-session) so downloads are reused. Defaults to a resolver that handles `data:` and `http(s)` URLs only. | | `organization` | `string \| null \| undefined` | no | OpenAI organization ID. | | `project` | `string \| null \| undefined` | no | OpenAI project ID. | | `defaultHeaders` | `Record \| null \| undefined` | no | Additional HTTP headers. | | `serviceTier` | `string \| null \| undefined` | no | Service tier to use (e.g., "auto", "flex", "priority"). | | `retryTimeoutSecs` | `number \| null \| undefined` | no | How long an inference may go without producing output before it is abandoned and re-issued, when `retryOnTimeout` is set. Defaults to 5.0 seconds. | | `retryOnTimeout` | `boolean \| null \| undefined` | no | Whether to re-issue the request once if the first attempt produces no output within `retryTimeoutSecs`. The retry is unbounded. | The other options (`organization`, `project`, `defaultHeaders`, `serviceTier`, `retryOnTimeout`, `retryTimeoutSecs`, function calling) are `OpenAILLMService`'s. See [OpenAI](https://polvallverdu.github.io/pipedog/docs/services/llm/openai/#configuration). | | | | ------------- | ------------------------------------------------------------------------------- | | Base URL | `https://dashscope-intl.aliyuncs.com/compatible-mode/v1` | | Default model | `qwen-plus` | | Models to try | `qwen-plus`, `qwen-max`, `qwen-turbo`. See the provider's catalog for the rest. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new QwenLLMService.Settings({...})` (`QwenLLMSettings`). They are [OpenAI's settings](https://polvallverdu.github.io/pipedog/docs/services/llm/openai/#settings), with these defaults: | Setting | Type | Default | Description | | ------- | -------- | ------------- | ----------- | | `model` | `string` | `"qwen-plus"` | Model name. | Every other setting keeps OpenAI's default. Unset sampling settings are left out of the request, so the provider's own defaults apply. `extra` carries provider-specific request fields. ### Usage [#usage] ```ts title="llm.ts" import { LLMUpdateSettingsFrame } from "@pipedog/core/frames/frames"; import { QwenLLMService } from "@pipedog/qwen/llm"; const llm = new QwenLLMService({ apiKey: process.env.QWEN_API_KEY ?? "", settings: new QwenLLMService.Settings({ model: "qwen-plus", systemInstruction: "You are a helpful assistant in a voice conversation.", temperature: 0.7, }), }); ``` Change settings mid-call with an `LLMUpdateSettingsFrame`. The next completion uses them: ```ts // Queue it on the worker, or push it from a processor upstream of the service. const update = new LLMUpdateSettingsFrame({ delta: new QwenLLMService.Settings({ temperature: 0.2, maxTokens: 300 }), }); ``` ### Notes [#notes] * Runs a completion for each `LLMContextFrame`, between an `LLMFullResponseStartFrame` and an `LLMFullResponseEndFrame`, and streams the reply as `LLMTextFrame`s, as [`OpenAILLMService`](https://polvallverdu.github.io/pipedog/docs/services/llm/openai.md) does. Function calls, token usage metrics and `runInference` work the same way. * "developer" messages are sent as "user" messages, since the API doesn't support that role (`supportsDeveloperRole` is `false`). * The default `baseUrl` is DashScope's international endpoint. Pass your region's endpoint as `baseUrl` if your key belongs to another one. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/qwen/package.json`: ✅ means the compat smoke test passes on that runtime. * The client is the official `openai` JS SDK (through `@pipedog/openai`), which talks through `fetch`. It works on Bun, Node, Deno and Cloudflare Workers. * The deprecated `model` constructor option is not ported. Use `settings`. * Tested against a local OpenAI-compatible mock server, not against Qwen itself. # SambaNova > SambaNovaLLMService streams chat completions from SambaNova through its OpenAI-compatible API. Source: https://polvallverdu.github.io/pipedog/docs/services/llm/sambanova/ {/* Generated by scripts/generate-providers.ts from data/providers.ts. */} ## Overview [#overview] SambaNova Cloud serves open models with fast inference on its own hardware. * **`SambaNovaLLMService`**: an `OpenAILLMService` subclass with SambaNova's base URL and default model. Pipecat's documentation for the same service. `packages/sambanova/src/llm.ts`. Models and the API. ## Installation [#installation] ```bash bun add @pipedog/sambanova ``` It depends on `@pipedog/openai`, which is installed with it. ```ts import { SambaNovaLLMService } from "@pipedog/sambanova/llm"; ``` ## Prerequisites [#prerequisites] * A SambaNova Cloud account and an API key from [SambaNova](https://docs.sambanova.ai/). * Pass the key as `apiKey`. It is required, and the service never falls back to `OPENAI_API_KEY`. The examples read it from `SAMBANOVA_API_KEY`. ## SambaNovaLLMService [#sambanovallmservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | The API key for accessing SambaNova API. | | `baseUrl` | `string \| null \| undefined` | no | The base URL for SambaNova API. Defaults to `"https://api.sambanova.ai/v1"`. | | `settings` | `SambaNovaLLMSettings \| null \| undefined` | no | Runtime-updatable settings; values override the built-in defaults. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `runInParallel` | `boolean \| undefined` | no | Whether to run function calls in parallel or sequentially. Defaults to true. | | `groupParallelTools` | `boolean \| undefined` | no | Whether to group parallel function calls so the LLM is triggered exactly once after all calls in the batch complete. When false, each function call result triggers the LLM independently as it arrives. Defaults to true. | | `functionCallTimeoutSecs` | `number \| null \| undefined` | no | Optional timeout in seconds for function calls. A call that runs past it is cancelled: its handler is cancelled (it gets a `CancelledError`), the call is settled as cancelled, and inference runs so the LLM can report that it didn't complete. | | `fileResolver` | `FileResolver \| null \| undefined` | no | Resolves file URLs in the context that this provider can't fetch itself (see `FileResolver`). Pass one configured with a `fileStorage` to resolve uploaded-file URLs (`new FileResolver({ fileStorage: runnerArgs.fileStorage })` when a runner hosts the bot), and share one instance across services that may consume the same files (e.g. services switched between mid-session) so downloads are reused. Defaults to a resolver that handles `data:` and `http(s)` URLs only. | | `organization` | `string \| null \| undefined` | no | OpenAI organization ID. | | `project` | `string \| null \| undefined` | no | OpenAI project ID. | | `defaultHeaders` | `Record \| null \| undefined` | no | Additional HTTP headers. | | `serviceTier` | `string \| null \| undefined` | no | Service tier to use (e.g., "auto", "flex", "priority"). | | `retryTimeoutSecs` | `number \| null \| undefined` | no | How long an inference may go without producing output before it is abandoned and re-issued, when `retryOnTimeout` is set. Defaults to 5.0 seconds. | | `retryOnTimeout` | `boolean \| null \| undefined` | no | Whether to re-issue the request once if the first attempt produces no output within `retryTimeoutSecs`. The retry is unbounded. | The other options (`organization`, `project`, `defaultHeaders`, `serviceTier`, `retryOnTimeout`, `retryTimeoutSecs`, function calling) are `OpenAILLMService`'s. See [OpenAI](https://polvallverdu.github.io/pipedog/docs/services/llm/openai/#configuration). | | | | ------------- | ----------------------------------------------------------------------- | | Base URL | `https://api.sambanova.ai/v1` | | Default model | `Meta-Llama-3.3-70B-Instruct` | | Models to try | `Meta-Llama-3.3-70B-Instruct`. See the provider's catalog for the rest. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new SambaNovaLLMService.Settings({...})` (`SambaNovaLLMSettings`). They are [OpenAI's settings](https://polvallverdu.github.io/pipedog/docs/services/llm/openai/#settings), with these defaults: | Setting | Type | Default | Description | | ------- | -------- | ------------------------------- | ----------- | | `model` | `string` | `"Meta-Llama-3.3-70B-Instruct"` | Model name. | Every other setting keeps OpenAI's default. Unset sampling settings are left out of the request, so the provider's own defaults apply. `extra` carries provider-specific request fields. ### Usage [#usage] ```ts title="llm.ts" import { LLMUpdateSettingsFrame } from "@pipedog/core/frames/frames"; import { SambaNovaLLMService } from "@pipedog/sambanova/llm"; const llm = new SambaNovaLLMService({ apiKey: process.env.SAMBANOVA_API_KEY ?? "", settings: new SambaNovaLLMService.Settings({ model: "Meta-Llama-3.3-70B-Instruct", systemInstruction: "You are a helpful assistant in a voice conversation.", temperature: 0.7, }), }); ``` Change settings mid-call with an `LLMUpdateSettingsFrame`. The next completion uses them: ```ts // Queue it on the worker, or push it from a processor upstream of the service. const update = new LLMUpdateSettingsFrame({ delta: new SambaNovaLLMService.Settings({ temperature: 0.2, maxTokens: 300 }), }); ``` ### Notes [#notes] * Runs a completion for each `LLMContextFrame`, between an `LLMFullResponseStartFrame` and an `LLMFullResponseEndFrame`, and streams the reply as `LLMTextFrame`s, as [`OpenAILLMService`](https://polvallverdu.github.io/pipedog/docs/services/llm/openai.md) does. Function calls, token usage metrics and `runInference` work the same way. * "developer" messages are sent as "user" messages, since the API doesn't support that role (`supportsDeveloperRole` is `false`). * `service_tier` is not sent: SambaNova doesn't support it. * SambaNova's tool calls may come without an index. The service skips tool calls whose arguments are empty, and runs function calls only when the last one has arguments. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/sambanova/package.json`: ✅ means the compat smoke test passes on that runtime. * The client is the official `openai` JS SDK (through `@pipedog/openai`), which talks through `fetch`. It works on Bun, Node, Deno and Cloudflare Workers. * The deprecated `model` constructor option is not ported. Use `settings`. * Tested against a local OpenAI-compatible mock server, not against SambaNova itself. # Sarvam AI > SarvamLLMService streams chat completions from Sarvam AI through its OpenAI-compatible API. Source: https://polvallverdu.github.io/pipedog/docs/services/llm/sarvam/ {/* Generated by scripts/generate-providers.ts from data/providers.ts. */} ## Overview [#overview] Sarvam AI's models for Indian languages, over an OpenAI-compatible API (`sarvam-105b`, plus a few hosted open models). * **`SarvamLLMService`**: an `OpenAILLMService` subclass with Sarvam AI's base URL and default model. Pipecat's documentation for the same service. `packages/sarvam/src/llm.ts`. Models and the API reference. ## Installation [#installation] ```bash bun add @pipedog/sarvam ``` It depends on `@pipedog/openai`, which is installed with it. ```ts import { SarvamLLMService } from "@pipedog/sarvam/llm"; ``` ## Prerequisites [#prerequisites] * A Sarvam account and an API key from the [Sarvam dashboard](https://dashboard.sarvam.ai/). * Pass the key as `apiKey`. It is required, and the service never falls back to `OPENAI_API_KEY`. The examples read it from `SARVAM_API_KEY`. ## SarvamLLMService [#sarvamllmservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | Sarvam API key used for both OpenAI auth and Sarvam subscription header. | | `baseUrl` | `string \| null \| undefined` | no | Sarvam OpenAI-compatible base URL. When null, resolved from the model: `/v1` for `sarvam-105b-conversations`, `/v2` for all other models. | | `settings` | `SarvamLLMSettings \| null \| undefined` | no | Runtime-updatable settings. | | `defaultHeaders` | `Record \| null \| undefined` | no | Additional HTTP headers to include in requests. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `runInParallel` | `boolean \| undefined` | no | Whether to run function calls in parallel or sequentially. Defaults to true. | | `groupParallelTools` | `boolean \| undefined` | no | Whether to group parallel function calls so the LLM is triggered exactly once after all calls in the batch complete. When false, each function call result triggers the LLM independently as it arrives. Defaults to true. | | `functionCallTimeoutSecs` | `number \| null \| undefined` | no | Optional timeout in seconds for function calls. A call that runs past it is cancelled: its handler is cancelled (it gets a `CancelledError`), the call is settled as cancelled, and inference runs so the LLM can report that it didn't complete. | | `fileResolver` | `FileResolver \| null \| undefined` | no | Resolves file URLs in the context that this provider can't fetch itself (see `FileResolver`). Pass one configured with a `fileStorage` to resolve uploaded-file URLs (`new FileResolver({ fileStorage: runnerArgs.fileStorage })` when a runner hosts the bot), and share one instance across services that may consume the same files (e.g. services switched between mid-session) so downloads are reused. Defaults to a resolver that handles `data:` and `http(s)` URLs only. | | `organization` | `string \| null \| undefined` | no | OpenAI organization ID. | | `project` | `string \| null \| undefined` | no | OpenAI project ID. | | `serviceTier` | `string \| null \| undefined` | no | Service tier to use (e.g., "auto", "flex", "priority"). | | `retryTimeoutSecs` | `number \| null \| undefined` | no | How long an inference may go without producing output before it is abandoned and re-issued, when `retryOnTimeout` is set. Defaults to 5.0 seconds. | | `retryOnTimeout` | `boolean \| null \| undefined` | no | Whether to re-issue the request once if the first attempt produces no output within `retryTimeoutSecs`. The retry is unbounded. | The other options (`organization`, `project`, `defaultHeaders`, `serviceTier`, `retryOnTimeout`, `retryTimeoutSecs`, function calling) are `OpenAILLMService`'s. See [OpenAI](https://polvallverdu.github.io/pipedog/docs/services/llm/openai/#configuration). | | | | ------------- | ------------------------------------------------------------------------------------------------------------------------------ | | Base URL | `https://api.sarvam.ai/v2` | | Default model | `sarvam-105b` | | Models to try | `sarvam-105b`, `sarvam-105b-conversations`, `gemma4`, `glm5.3`, `deepseekv4.1-flash`. See the provider's catalog for the rest. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new SarvamLLMService.Settings({...})` (`SarvamLLMSettings`). They are [OpenAI's settings](https://polvallverdu.github.io/pipedog/docs/services/llm/openai/#settings), with these defaults: | Setting | Type | Default | Description | | ----------------- | ------------------------------------- | --------------- | ------------------------------------------------------------------------------ | | `model` | `string` | `"sarvam-105b"` | Model name. | | `wikiGrounding` | `boolean \| null` | `null` | Sent as `wiki_grounding` when set. Only `sarvam-105b` and `gemma4` support it. | | `reasoningEffort` | `"low" \| "medium" \| "high" \| null` | `null` | Sent as `reasoning_effort` when set, on the models that reason. | Every other setting keeps OpenAI's default. Unset sampling settings are left out of the request, so the provider's own defaults apply. `extra` carries provider-specific request fields. ### Usage [#usage] ```ts title="llm.ts" import { LLMUpdateSettingsFrame } from "@pipedog/core/frames/frames"; import { SarvamLLMService } from "@pipedog/sarvam/llm"; const llm = new SarvamLLMService({ apiKey: process.env.SARVAM_API_KEY ?? "", settings: new SarvamLLMService.Settings({ model: "sarvam-105b", systemInstruction: "You are a helpful assistant in a voice conversation.", }), }); ``` Ground answers in Wikipedia and set the reasoning effort: ```ts const grounded = new SarvamLLMService({ apiKey: process.env.SARVAM_API_KEY ?? "", settings: new SarvamLLMService.Settings({ model: "sarvam-105b", wikiGrounding: true, reasoningEffort: "low", }), }); ``` Change settings mid-call with an `LLMUpdateSettingsFrame`. The next completion uses them: ```ts // Queue it on the worker, or push it from a processor upstream of the service. const update = new LLMUpdateSettingsFrame({ delta: new SarvamLLMService.Settings({ temperature: 0.2 }), }); ``` ### Notes [#notes] * Runs a completion for each `LLMContextFrame`, between an `LLMFullResponseStartFrame` and an `LLMFullResponseEndFrame`, and streams the reply as `LLMTextFrame`s, as [`OpenAILLMService`](https://polvallverdu.github.io/pipedog/docs/services/llm/openai.md) does. Function calls, token usage metrics and `runInference` work the same way. * "developer" messages are sent as "user" messages, since the API doesn't support that role (`supportsDeveloperRole` is `false`). * Only `deepseekv4.1-flash`, `gemma4`, `glm5.3`, `sarvam-105b` and `sarvam-105b-conversations` are accepted as the model. Any other raises when the service is constructed. * When you don't pass `baseUrl`, it is chosen from the model: `/v1` for `sarvam-105b-conversations` and `/v2` for the rest. * A context that a model can't serve (an image for a model without vision, say) fails with a non-fatal `ErrorFrame`, not an exception, so an `LLMSwitcher` can fall back to another service. * The same package has Sarvam's [speech-to-text](https://polvallverdu.github.io/pipedog/docs/services/stt/sarvam.md) and [text-to-speech](https://polvallverdu.github.io/pipedog/docs/services/tts/sarvam.md) services. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/sarvam/package.json`: ✅ means the compat smoke test passes on that runtime. * The client is the official `openai` JS SDK (through `@pipedog/openai`), which talks through `fetch`. It works on Bun, Node, Deno and Cloudflare Workers. * The deprecated `model` constructor option is not ported. Use `settings`. * Tested against a local OpenAI-compatible mock server, not against Sarvam AI itself. # Smallest AI > SmallestLLMService streams chat completions from Smallest AI through its OpenAI-compatible API. Source: https://polvallverdu.github.io/pipedog/docs/services/llm/smallest/ {/* Generated by scripts/generate-providers.ts from data/providers.ts. */} ## Overview [#overview] Smallest AI serves Electron, its own LLM, through the Waves API's OpenAI-compatible chat completions endpoint. * **`SmallestLLMService`**: an `OpenAILLMService` subclass with Smallest AI's base URL and default model. Pipecat's documentation for the same service. `packages/smallest/src/llm.ts`. Models and the Waves API. ## Installation [#installation] ```bash bun add @pipedog/smallest ``` It depends on `@pipedog/openai`, which is installed with it. ```ts import { SmallestLLMService } from "@pipedog/smallest/llm"; ``` ## Prerequisites [#prerequisites] * A Smallest AI account and an API key from [Smallest AI](https://smallest.ai/). * Pass the key as `apiKey`. It is required, and the service never falls back to `OPENAI_API_KEY`. The examples read it from `SMALLEST_API_KEY`. ## SmallestLLMService [#smallestllmservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `apiKey` | `string` | yes | The API key for accessing Smallest AI's API. | | `baseUrl` | `string \| null \| undefined` | no | The base URL for Smallest AI's API. Defaults to `"https://api.smallest.ai/waves/v1"`. | | `settings` | `SmallestLLMSettings \| null \| undefined` | no | Runtime-updatable settings. Defaults to Electron, Smallest AI's LLM, via `settings: new SmallestLLMService.Settings({ model: "electron" })`. | | `defaultHeaders` | `Record \| null \| undefined` | no | Additional HTTP headers to include in requests. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `runInParallel` | `boolean \| undefined` | no | Whether to run function calls in parallel or sequentially. Defaults to true. | | `groupParallelTools` | `boolean \| undefined` | no | Whether to group parallel function calls so the LLM is triggered exactly once after all calls in the batch complete. When false, each function call result triggers the LLM independently as it arrives. Defaults to true. | | `functionCallTimeoutSecs` | `number \| null \| undefined` | no | Optional timeout in seconds for function calls. A call that runs past it is cancelled: its handler is cancelled (it gets a `CancelledError`), the call is settled as cancelled, and inference runs so the LLM can report that it didn't complete. | | `fileResolver` | `FileResolver \| null \| undefined` | no | Resolves file URLs in the context that this provider can't fetch itself (see `FileResolver`). Pass one configured with a `fileStorage` to resolve uploaded-file URLs (`new FileResolver({ fileStorage: runnerArgs.fileStorage })` when a runner hosts the bot), and share one instance across services that may consume the same files (e.g. services switched between mid-session) so downloads are reused. Defaults to a resolver that handles `data:` and `http(s)` URLs only. | | `organization` | `string \| null \| undefined` | no | OpenAI organization ID. | | `project` | `string \| null \| undefined` | no | OpenAI project ID. | | `serviceTier` | `string \| null \| undefined` | no | Service tier to use (e.g., "auto", "flex", "priority"). | | `retryTimeoutSecs` | `number \| null \| undefined` | no | How long an inference may go without producing output before it is abandoned and re-issued, when `retryOnTimeout` is set. Defaults to 5.0 seconds. | | `retryOnTimeout` | `boolean \| null \| undefined` | no | Whether to re-issue the request once if the first attempt produces no output within `retryTimeoutSecs`. The retry is unbounded. | The other options (`organization`, `project`, `defaultHeaders`, `serviceTier`, `retryOnTimeout`, `retryTimeoutSecs`, function calling) are `OpenAILLMService`'s. See [OpenAI](https://polvallverdu.github.io/pipedog/docs/services/llm/openai/#configuration). | | | | ------------- | ---------------------------------------------------- | | Base URL | `https://api.smallest.ai/waves/v1` | | Default model | `electron` | | Models to try | `electron`. See the provider's catalog for the rest. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new SmallestLLMService.Settings({...})` (`SmallestLLMSettings`). They are [OpenAI's settings](https://polvallverdu.github.io/pipedog/docs/services/llm/openai/#settings), with these defaults: | Setting | Type | Default | Description | | ------- | -------- | ------------ | ----------- | | `model` | `string` | `"electron"` | Model name. | Every other setting keeps OpenAI's default. Unset sampling settings are left out of the request, so the provider's own defaults apply. `extra` carries provider-specific request fields. ### Usage [#usage] ```ts title="llm.ts" import { LLMUpdateSettingsFrame } from "@pipedog/core/frames/frames"; import { SmallestLLMService } from "@pipedog/smallest/llm"; const llm = new SmallestLLMService({ apiKey: process.env.SMALLEST_API_KEY ?? "", settings: new SmallestLLMService.Settings({ model: "electron", systemInstruction: "You are a helpful assistant in a voice conversation.", temperature: 0.7, }), }); ``` Change settings mid-call with an `LLMUpdateSettingsFrame`. The next completion uses them: ```ts // Queue it on the worker, or push it from a processor upstream of the service. const update = new LLMUpdateSettingsFrame({ delta: new SmallestLLMService.Settings({ temperature: 0.2, maxTokens: 300 }), }); ``` ### Notes [#notes] * Runs a completion for each `LLMContextFrame`, between an `LLMFullResponseStartFrame` and an `LLMFullResponseEndFrame`, and streams the reply as `LLMTextFrame`s, as [`OpenAILLMService`](https://polvallverdu.github.io/pipedog/docs/services/llm/openai.md) does. Function calls, token usage metrics and `runInference` work the same way. * "developer" messages are sent as "user" messages, since the API doesn't support that role (`supportsDeveloperRole` is `false`). * Every request carries `X-Source: pipecat` and `X-Pipecat-Version` headers (the version is pipedog's), merged over any `defaultHeaders` you pass. * The same package has Smallest AI's [speech-to-text](https://polvallverdu.github.io/pipedog/docs/services/stt/smallest.md) and [text-to-speech](https://polvallverdu.github.io/pipedog/docs/services/tts/smallest.md) services. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/smallest/package.json`: ✅ means the compat smoke test passes on that runtime. * The client is the official `openai` JS SDK (through `@pipedog/openai`), which talks through `fetch`. It works on Bun, Node, Deno and Cloudflare Workers. * The deprecated `model` constructor option is not ported. Use `settings`. * Tested against a local OpenAI-compatible mock server, not against Smallest AI itself. # Together AI > TogetherLLMService streams chat completions from Together AI through its OpenAI-compatible API. Source: https://polvallverdu.github.io/pipedog/docs/services/llm/together/ {/* Generated by scripts/generate-providers.ts from data/providers.ts. */} ## Overview [#overview] Together AI serves open models through an OpenAI-compatible API. * **`TogetherLLMService`**: an `OpenAILLMService` subclass with Together AI's base URL and default model. Pipecat's documentation for the same service. `packages/together/src/llm.ts`. The chat completions API reference. ## Installation [#installation] ```bash bun add @pipedog/together ``` It depends on `@pipedog/openai`, which is installed with it. ```ts import { TogetherLLMService } from "@pipedog/together/llm"; ``` ## Prerequisites [#prerequisites] * A Together AI account and an API key from [Together AI](https://together.ai/). * Pass the key as `apiKey`. It is required, and the service never falls back to `OPENAI_API_KEY`. The examples read it from `TOGETHER_API_KEY`. ## TogetherLLMService [#togetherllmservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | The API key for accessing Together.ai's API. | | `baseUrl` | `string \| null \| undefined` | no | The base URL for Together.ai API. Defaults to `"https://api.together.xyz/v1"`. | | `settings` | `TogetherLLMSettings \| null \| undefined` | no | Runtime-updatable settings; values override the built-in defaults. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `runInParallel` | `boolean \| undefined` | no | Whether to run function calls in parallel or sequentially. Defaults to true. | | `groupParallelTools` | `boolean \| undefined` | no | Whether to group parallel function calls so the LLM is triggered exactly once after all calls in the batch complete. When false, each function call result triggers the LLM independently as it arrives. Defaults to true. | | `functionCallTimeoutSecs` | `number \| null \| undefined` | no | Optional timeout in seconds for function calls. A call that runs past it is cancelled: its handler is cancelled (it gets a `CancelledError`), the call is settled as cancelled, and inference runs so the LLM can report that it didn't complete. | | `fileResolver` | `FileResolver \| null \| undefined` | no | Resolves file URLs in the context that this provider can't fetch itself (see `FileResolver`). Pass one configured with a `fileStorage` to resolve uploaded-file URLs (`new FileResolver({ fileStorage: runnerArgs.fileStorage })` when a runner hosts the bot), and share one instance across services that may consume the same files (e.g. services switched between mid-session) so downloads are reused. Defaults to a resolver that handles `data:` and `http(s)` URLs only. | | `organization` | `string \| null \| undefined` | no | OpenAI organization ID. | | `project` | `string \| null \| undefined` | no | OpenAI project ID. | | `defaultHeaders` | `Record \| null \| undefined` | no | Additional HTTP headers. | | `serviceTier` | `string \| null \| undefined` | no | Service tier to use (e.g., "auto", "flex", "priority"). | | `retryTimeoutSecs` | `number \| null \| undefined` | no | How long an inference may go without producing output before it is abandoned and re-issued, when `retryOnTimeout` is set. Defaults to 5.0 seconds. | | `retryOnTimeout` | `boolean \| null \| undefined` | no | Whether to re-issue the request once if the first attempt produces no output within `retryTimeoutSecs`. The retry is unbounded. | The other options (`organization`, `project`, `defaultHeaders`, `serviceTier`, `retryOnTimeout`, `retryTimeoutSecs`, function calling) are `OpenAILLMService`'s. See [OpenAI](https://polvallverdu.github.io/pipedog/docs/services/llm/openai/#configuration). | | | | ------------- | ----------------------------------------------------------- | | Base URL | `https://api.together.xyz/v1` | | Default model | `zai-org/GLM-5.2` | | Models to try | `zai-org/GLM-5.2`. See the provider's catalog for the rest. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new TogetherLLMService.Settings({...})` (`TogetherLLMSettings`). They are [OpenAI's settings](https://polvallverdu.github.io/pipedog/docs/services/llm/openai/#settings), with these defaults: | Setting | Type | Default | Description | | ----------------- | -------------------------------------------------------- | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `model` | `string` | `"zai-org/GLM-5.2"` | Model name. | | `reasoning` | `Record \| null` | `null` | Together's reasoning toggle, e.g. `{ enabled: false }`, sent as `reasoning` in the request body. `null` leaves it to the model. | | `reasoningEffort` | `"low" \| "medium" \| "high" \| "max" \| string \| null` | `null` | How much the model reasons, sent as `reasoning_effort` when set, unless `extra` already has one. Values vary by model (`"max"` on `zai-org/GLM-5.3`). It reaches models that ignore `reasoning`, such as GLM-5.3. `null` leaves Together's per-model default. | Every other setting keeps OpenAI's default. Unset sampling settings are left out of the request, so the provider's own defaults apply. `extra` carries provider-specific request fields. ### Usage [#usage] ```ts title="llm.ts" import { LLMUpdateSettingsFrame } from "@pipedog/core/frames/frames"; import { TogetherLLMService } from "@pipedog/together/llm"; const llm = new TogetherLLMService({ apiKey: process.env.TOGETHER_API_KEY ?? "", settings: new TogetherLLMService.Settings({ model: "zai-org/GLM-5.2", systemInstruction: "You are a helpful assistant in a voice conversation.", temperature: 0.7, }), }); ``` Turn off a reasoning model's thinking pass, which otherwise delays the first spoken token: ```ts const noThinking = new TogetherLLMService({ apiKey: process.env.TOGETHER_API_KEY ?? "", settings: new TogetherLLMService.Settings({ reasoning: { enabled: false } }), }); ``` Models that ignore the toggle, such as `zai-org/GLM-5.3`, still take a reasoning effort: ```ts const lowEffort = new TogetherLLMService({ apiKey: process.env.TOGETHER_API_KEY ?? "", settings: new TogetherLLMService.Settings({ model: "zai-org/GLM-5.3", reasoningEffort: "low", }), }); ``` Change settings mid-call with an `LLMUpdateSettingsFrame`. The next completion uses them: ```ts // Queue it on the worker, or push it from a processor upstream of the service. const update = new LLMUpdateSettingsFrame({ delta: new TogetherLLMService.Settings({ temperature: 0.2, maxTokens: 300 }), }); ``` ### Notes [#notes] * Runs a completion for each `LLMContextFrame`, between an `LLMFullResponseStartFrame` and an `LLMFullResponseEndFrame`, and streams the reply as `LLMTextFrame`s, as [`OpenAILLMService`](https://polvallverdu.github.io/pipedog/docs/services/llm/openai.md) does. Function calls, token usage metrics and `runInference` work the same way. * "developer" messages are sent as "user" messages, since the API doesn't support that role (`supportsDeveloperRole` is `false`). * The same package has Together's [speech-to-text](https://polvallverdu.github.io/pipedog/docs/services/stt/together.md) and [text-to-speech](https://polvallverdu.github.io/pipedog/docs/services/tts/together.md) services. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/together/package.json`: ✅ means the compat smoke test passes on that runtime. * The client is the official `openai` JS SDK (through `@pipedog/openai`), which talks through `fetch`. It works on Bun, Node, Deno and Cloudflare Workers. * The deprecated `model` constructor option is not ported. Use `settings`. * Tested against a local OpenAI-compatible mock server, not against Together AI itself. # MCP client > Give the bot the tools of an MCP server (stdio, SSE or streamable HTTP) with MCPClient. Source: https://polvallverdu.github.io/pipedog/docs/services/mcp-client/ ## Overview [#overview] `MCPClient` connects to a [Model Context Protocol](https://modelcontextprotocol.io) server, lists its tools, and gives them to the LLM as function-call tools. When the model calls one, the client runs it on the server and returns the text to the model. * **`MCPClient`**: the client. It keeps one persistent session with the server, opened by the first call to `tools()` (or by `start()`). * **`StreamableHttpParameters`, `SseServerParameters`, `StdioServerParameters`**: how to reach the server. This is the MCP *client*. For the MCP server that serves these docs to coding agents, see [`@pipedog/mcp`](https://polvallverdu.github.io/pipedog/docs/guides/llms/#mcp-server). Pipecat's `MCPClient` (`services/mcp_service.py`). `packages/mcp-client/src/mcp_service.ts`. The protocol, and servers to connect to. ## Installation [#installation] ```bash bun add @pipedog/mcp-client ``` ```ts import { MCPClient, StreamableHttpParameters } from "@pipedog/mcp-client"; ``` It runs on the official [`@modelcontextprotocol/sdk`](https://github.com/modelcontextprotocol/typescript-sdk), which is a dependency. ## Runtimes [#runtimes] | Runtime | Works | What it needs | | ------------------ | --------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Bun | ✅ yes | An MCP server to connect to: a URL (streamable HTTP or SSE), or a command (stdio). Stdio servers start as child processes. | | Node.js | ✅ yes | An MCP server to connect to: a URL (streamable HTTP or SSE), or a command (stdio). Stdio servers start as child processes. | | Deno | ✅ yes | An MCP server to connect to: a URL (streamable HTTP or SSE), or a command (stdio). Stdio servers start as child processes: run with --allow-run. HTTP servers need --allow-net. | | Cloudflare Workers | ⚠️ partly | An MCP server to connect to: a URL (streamable HTTP or SSE), or a command (stdio). Streamable HTTP and SSE only. Workers cannot start processes, so a stdio server is refused with an error. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/mcp-client/package.json`: ✅ means the compat smoke test passes on that runtime. | Transport | Bun | Node | Deno | Cloudflare Workers | | --------------- | --- | ---- | ---- | ------------------ | | Streamable HTTP | ✅ | ✅ | ✅ | ✅ | | SSE | ✅ | ✅ | ✅ | ✅ | | stdio | ✅ | ✅ | ✅ | no | Streamable HTTP and SSE go through `fetch`. A stdio server is a child process, which Workers cannot start: there `tools()` rejects with an error that says to use an HTTP server. The stdio code is a separate module that the `workerd` export condition replaces with that error, so a Worker bundle does not include the SDK's process spawner. ## Prerequisites [#prerequisites] * An MCP server to connect to: its URL (streamable HTTP or SSE), or the command that starts it (stdio). Pass credentials as `headers` (HTTP) or `env` (stdio). ## MCPClient [#mcpclient] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `serverParams` | `ServerParameters` | yes | Server connection parameters (stdio, SSE, or streamable HTTP). | | `toolsFilter` | `string[] \| null \| undefined` | no | Optional list of tool names to register. If null, all tools are registered. | | `toolsOutputFilters` | `Record any> \| null \| undefined` | no | Optional map from tool names to filter functions that process tool outputs. Each filter receives the tool's text output and returns the processed output. | | `toolsArguments` | `Record> \| null \| undefined` | no | Optional map from tool names to fixed arguments that are merged into every call of that tool (overriding any model-supplied values). The fixed parameter names are removed from the advertised tool schema so the LLM never sees them. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | Server parameters: | Option | Type | Required | Description | |---|---|---|---| | `url` | `string \| URL` | yes | The MCP endpoint of the server. | | `headers` | `Record \| null \| undefined` | no | Extra HTTP headers, for example `Authorization`. | | `timeout` | `number \| undefined` | no | Seconds to wait for the `initialize` handshake. Defaults to 30. | | `sseReadTimeout` | `number \| undefined` | no | Seconds to wait for the response headers of a request. Defaults to 300. | | `terminateOnClose` | `boolean \| undefined` | no | Whether closing the client also ends the server's session (an HTTP DELETE). Defaults to true. | | `fetch` | `{ (input: RequestInfo \| URL, init?: RequestInit): Promise; (input: string \| URL \| Request, init?: RequestInit): Promise; } \| null \| undefined` | no | The `fetch` to use, instead of the global one. DIVERGENCE: not in upstream (which uses httpx). It lets a test, or a Worker with a service binding, serve the connection in process. | | Option | Type | Required | Description | |---|---|---|---| | `url` | `string \| URL` | yes | The SSE endpoint of the server. | | `headers` | `Record \| null \| undefined` | no | Extra HTTP headers, for example `Authorization`. | | `timeout` | `number \| undefined` | no | Seconds to wait for the `initialize` handshake. Defaults to 5. | | `sseReadTimeout` | `number \| undefined` | no | Seconds to wait for the response headers of a request. Defaults to 300. | | `fetch` | `{ (input: RequestInfo \| URL, init?: RequestInit): Promise; (input: string \| URL \| Request, init?: RequestInit): Promise; } \| null \| undefined` | no | The `fetch` to use, instead of the global one. DIVERGENCE: not in upstream (which uses httpx). It lets a test, or a Worker with a service binding, serve the connection in process. | | Option | Type | Required | Description | |---|---|---|---| | `command` | `string` | yes | The executable to run to start the server. | | `args` | `string[] \| undefined` | no | Command line arguments to pass to the executable. | | `env` | `Record \| null \| undefined` | no | The environment to use when spawning the process. It is merged over a default environment of safe variables (`PATH`, `HOME` and so on), not over the whole of `process.env`. | | `cwd` | `string \| null \| undefined` | no | The working directory to use when spawning the process. | ### Usage [#usage] Create the client, call `tools()`, and put the result in the context. There is no `registerFunction` call: each schema carries its handler, and the LLM service registers it when the context advertises the tools. ```ts title="mcp-client.ts" import { LLMContext } from "@pipedog/core/processors/aggregators/llm_context"; import { LLMContextAggregatorPair } from "@pipedog/core/processors/aggregators/llm_response_universal"; import { MCPClient, StreamableHttpParameters } from "@pipedog/mcp-client"; import { OpenAILLMService } from "@pipedog/openai/llm"; const mcp = new MCPClient({ serverParams: new StreamableHttpParameters({ url: "https://mcp.example.com/mcp", headers: { authorization: `Bearer ${process.env.MCP_TOKEN ?? ""}` }, }), }); const llm = new OpenAILLMService({ apiKey: process.env.OPENAI_API_KEY ?? "" }); // tools() connects, and returns schemas that carry their handlers. The LLM service registers them // when the context advertises them, and closes the connection when the pipeline is torn down. const context = new LLMContext({ messages: [{ role: "system", content: "You are a helpful voice assistant." }], tools: await mcp.tools(), }); const aggregators = new LLMContextAggregatorPair(context); ``` The connection closes by itself when the pipeline is torn down, because the handlers tell the LLM service to close the client at cleanup. ### Transports [#transports] ```ts title="transports.ts" import { SseServerParameters, StdioServerParameters } from "@pipedog/mcp-client"; // A server that is a child process (Bun, Node and Deno). const stdio = new MCPClient({ serverParams: new StdioServerParameters({ command: "npx", args: ["-y", "@modelcontextprotocol/server-filesystem", "/tmp"], env: { LOG_LEVEL: "warn" }, }), }); // The older HTTP transport of MCP. const sse = new MCPClient({ serverParams: new SseServerParameters({ url: "https://mcp.example.com/sse" }), }); ``` ### Filtering tools [#filtering-tools] `toolsFilter` limits which tools are registered. `toolsOutputFilters` processes the text a tool returns. `toolsArguments` fixes arguments: they are removed from the schema the model sees, merged into every call, and win over anything the model sends. That is how to pin a setting, or pass an identifier the model should not choose. ```ts title="filters.ts" const filtered = new MCPClient({ serverParams: new StreamableHttpParameters({ url: "https://mcp.example.com/mcp" }), // Register only these tools. toolsFilter: ["search", "get_page"], // Shorten what a tool returns before the model reads it. toolsOutputFilters: { get_page: (text: string) => text.slice(0, 2000) }, // Fixed arguments: removed from the schema the model sees, and sent on every call. toolsArguments: { search: { mode: "realtime" } }, }); ``` ### Sessions and errors [#sessions-and-errors] One session serves every call, as in Pipecat. * A call that returns an error returns the line `Error calling mcp tool : ` to the model, cut to 200 characters. An empty result returns `Sorry, could not call the mcp tool`. * A call that finds the connection gone returns its error to the model and drops the session. The next call connects again. * `close()` releases the connection early. A closed client stays closed until `start()` or `tools()`; a tool call on it throws (it is a wiring mistake, not something for the model). * `start()` and `close()` can be called from different tasks, and from several at once. ```ts title="lifecycle.ts" // Open the connection ahead of the first call, so the first user turn does not pay for it. await mcp.start(); // Release it earlier than pipeline teardown, for example when the client disconnects. await mcp.close(); // Or scope it: `await using` closes the client at the end of the block. { await using scoped = new MCPClient({ serverParams: new StreamableHttpParameters({ url: "https://mcp.example.com/mcp" }), }); await scoped.start(); } ``` ### Serving a connection in process [#serving-a-connection-in-process] `StreamableHttpParameters` and `SseServerParameters` take a `fetch`, so a test, or a Worker with a service binding, can serve the connection without a network. Pipecat has no such option. ```ts title="fetch.ts" // Serve the connection from a function, with no network: a test, or a service binding on Workers. declare const handler: (request: Request) => Promise; const inProcess = new MCPClient({ serverParams: new StreamableHttpParameters({ url: "http://in.process/mcp", fetch: ((input, init) => handler(new Request(input, init))) as typeof fetch, }), }); ``` ## Differences from Pipecat [#differences-from-pipecat] * **The constructor takes one options object**: `new MCPClient({ serverParams, toolsFilter, ... })`. * **`async with` is `await using`**, or `await mcp.scoped(async (mcp) => { ... })`. * **Deprecated methods are not ported**: `register_tools`, `get_tools_schema` and `register_tools_schema`. Use `tools()`. * **Timeouts.** With `fetch` there is no separate connect timeout. `timeout` bounds the `initialize` handshake and `sseReadTimeout` the wait for the response headers of a request; an idle SSE stream is not timed out. * **`tools()` follows the server's pagination cursor.** Pipecat reads the first page. * **Lost connections** are recognised from what the SDK raises (`McpError` with code `ConnectionClosed`, or `Not connected`), where Pipecat reads anyio's stream errors. A request that fails at the network level is reported to the model and keeps the session. * **The SDK's `MCP error -32000:` prefix is left off** the line the model reads. * **On Workers the SDK does not check a result against the tool's output schema.** The check needs code generation, which Workers forbid. ## Runtime notes [#runtime-notes] * Calls are cancellation points: when a function call is cancelled (an interruption, a timeout), the request to the server is cancelled with it. * The SDK's own request timeout is off, as Pipecat's session has none, so a slow tool is waited for. Set `functionCallTimeoutSecs` on the LLM service if the bot should not wait. * On Deno, stdio servers need `--allow-run`, and HTTP ones `--allow-net`. # Other upstream providers > Pipecat services that aren't ported to pipedog yet and have no page of their own here. Source: https://polvallverdu.github.io/pipedog/docs/services/other-upstream/ {/* Generated by scripts/generate-providers.ts from data/providers.ts. */} These Pipecat services aren't ported yet. Each will keep its upstream class names when it is. To ask for one, [open a port request](https://github.com/polvallverdu/pipedog/issues/new?template=feature_request.yml) or upvote an existing one in the [issues](https://github.com/polvallverdu/pipedog/issues). | Provider | Kind | Upstream classes | Upstream source | | -------------------------- | ---------- | ------------------------------------- | ------------------------------------------------------------------------------------------------------ | | fal | STT, Image | `FalSTTService`, `FalImageGenService` | [`services/fal`](https://github.com/pipecat-ai/pipecat/blob/main/src/pipecat/services/fal) | | Keenable | Search | `KeenableWebSearch` | [`services/keenable`](https://github.com/pipecat-ai/pipecat/blob/main/src/pipecat/services/keenable) | | Mem0 | Memory | `Mem0MemoryService` | [`services/mem0`](https://github.com/pipecat-ai/pipecat/blob/main/src/pipecat/services/mem0) | | Moondream | Vision | `MoondreamService` | [`services/moondream`](https://github.com/pipecat-ai/pipecat/blob/main/src/pipecat/services/moondream) | | OpenClaw | Agents | `OpenClawGatewayService` | [`services/openclaw`](https://github.com/pipecat-ai/pipecat/blob/main/src/pipecat/services/openclaw) | | XTTS (deprecated upstream) | TTS | `XTTSService` | [`services/xtts`](https://github.com/pipecat-ai/pipecat/blob/main/src/pipecat/services/xtts) | ## Variants of ported providers [#variants-of-ported-providers] These providers are ported, but these classes aren't yet: | Variant | Upstream classes | Upstream source | | --------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Deepgram on SageMaker | `DeepgramSageMakerSTTService`, `DeepgramSageMakerTTSService`, `DeepgramFluxSageMakerSTTService`, `DeepgramFluxSageMakerTTSService` | [`services/deepgram/sagemaker/`](https://github.com/pipecat-ai/pipecat/blob/main/src/pipecat/services/deepgram/sagemaker/), [`services/deepgram/flux/sagemaker/`](https://github.com/pipecat-ai/pipecat/blob/main/src/pipecat/services/deepgram/flux/sagemaker/) | | NVIDIA on SageMaker | `NvidiaSageMakerSTTService`, `NvidiaSageMakerTTSService` | [`services/nvidia/sagemaker/`](https://github.com/pipecat-ai/pipecat/blob/main/src/pipecat/services/nvidia/sagemaker/) | # AWS Nova Sonic > Real-time speech-to-speech with Amazon Nova Sonic on Bedrock through AWSNovaSonicLLMService, with tools and session continuation. Source: https://polvallverdu.github.io/pipedog/docs/services/s2s/aws/ ## Overview [#overview] `AWSNovaSonicLLMService` streams the user's audio to Amazon Nova Sonic on Bedrock (`InvokeModelWithBidirectionalStream`) and returns spoken audio, transcripts and tool calls, so one service replaces the STT, LLM and TTS stages of a pipeline. * **`AWSNovaSonicLLMService`**: the session, with `AWSNovaSonicLLMSettings` and `AudioConfig`. * **`AWSNovaSonicLLMAdapter`**: converts the `LLMContext` history and tools to Nova Sonic's events. * **`SessionContinuationParams`**: hands the conversation to a new session before Nova Sonic's session time limit, buffering audio so nothing is lost. Pipecat's documentation for the same service. `packages/aws/src/nova_sonic/llm.ts`, `session_continuation.ts` and `bidirectional_stream.ts`. Voices, events, tools and limits. ## Installation [#installation] ```bash bun add @pipedog/aws ``` ```ts import { AudioConfig, AWSNovaSonicLLMService } from "@pipedog/aws/nova_sonic/llm"; ``` The default transport needs no AWS SDK. For `transport: "http2"`, also install `@aws-sdk/client-bedrock-runtime` (an optional peer dependency). ## Prerequisites [#prerequisites] * Access to Nova Sonic in the Bedrock console, in a region that offers it. * An IAM identity allowed `bedrock:InvokeModelWithBidirectionalStream`. * Keys, in this order of precedence: 1. `accessKeyId`, `secretAccessKey` and `sessionToken`. 2. The `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` and `AWS_SESSION_TOKEN` environment variables. 3. A `credentialsProvider` function, called for every new session. The region comes from `region`, else `AWS_REGION`, else `us-east-1`. ## AWSNovaSonicLLMService [#awsnovasonicllmservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `secretAccessKey` | `string \| null \| undefined` | no | AWS secret access key for authentication. Falls back to `AWS_SECRET_ACCESS_KEY`. | | `accessKeyId` | `string \| null \| undefined` | no | AWS access key ID for authentication. Falls back to `AWS_ACCESS_KEY_ID`. | | `sessionToken` | `string \| null \| undefined` | no | AWS session token for authentication. Falls back to `AWS_SESSION_TOKEN`. | | `region` | `string \| null \| undefined` | no | AWS region where the service is hosted. Supported regions: "us-east-1", "us-west-2", "eu-north-1", "ap-northeast-1". Falls back to `AWS_REGION`. | | `credentialsProvider` | `AWSCredentialsProvider \| null \| undefined` | no | Keys to use when none are passed or set in the environment (pipedog addition). | | `audioConfig` | `AudioConfig \| null \| undefined` | no | Audio configuration (sample rates, sample sizes, channel counts). If not provided, defaults are used. | | `settings` | `AWSNovaSonicLLMSettings \| null \| undefined` | no | AWS Nova Sonic LLM settings. | | `tools` | `(FunctionSchema \| DirectFunction)[] \| ToolsSchema \| null \| undefined` | no | Available tools for the model: a `ToolsSchema` or a plain list of direct functions and/or `FunctionSchema` objects (handlers auto-register). | | `sessionContinuation` | `SessionContinuationParams \| null \| undefined` | no | Configuration for automatic session continuation. When enabled (the default), sessions are seamlessly rotated before the AWS time limit (~8 minutes) with no user-perceptible interruption. | | `transport` | `NovaSonicTransport \| undefined` | no | Transport of the default client (pipedog addition): `"websocket"` (default, every runtime) or `"http2"` (`@aws-sdk/client-bedrock-runtime`, Node and Bun). | | `endpointUrl` | `string \| null \| undefined` | no | Endpoint override, `https://host[:port]` (pipedog addition). | | `clientFactory` | `((service: AWSNovaSonicLLMService) => NovaSonicStreamClient) \| null \| undefined` | no | Creates the Bedrock client (pipedog addition; replaces `createClient()`'s default). | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `runInParallel` | `boolean \| undefined` | no | Whether to run function calls in parallel or sequentially. Defaults to true. | | `groupParallelTools` | `boolean \| undefined` | no | Whether to group parallel function calls so the LLM is triggered exactly once after all calls in the batch complete. When false, each function call result triggers the LLM independently as it arrives. Defaults to true. | | `functionCallTimeoutSecs` | `number \| null \| undefined` | no | Optional timeout in seconds for function calls. A call that runs past it is cancelled: its handler is cancelled (it gets a `CancelledError`), the call is settled as cancelled, and inference runs so the LLM can report that it didn't complete. | | `fileResolver` | `FileResolver \| null \| undefined` | no | Resolves file URLs in the context that this provider can't fetch itself (see `FileResolver`). Pass one configured with a `fileStorage` to resolve uploaded-file URLs (`new FileResolver({ fileStorage: runnerArgs.fileStorage })` when a runner hosts the bot), and share one instance across services that may consume the same files (e.g. services switched between mid-session) so downloads are reused. Defaults to a resolver that handles `data:` and `http(s)` URLs only. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new AWSNovaSonicLLMService.Settings({...})`. | Setting | Type | Default | Description | | ------------------------ | ------------------------------------- | ------------------------- | --------------------------------------------------------- | | `model` | `string` | `"amazon.nova-2-5-sonic"` | Bedrock model ID. | | `voice` | `string` | `"matthew"` | Voice ID. | | `systemInstruction` | `string \| null` | `null` | System prompt, sent when the session starts. | | `temperature` | `number` | `0.7` | Sampling temperature. | | `maxTokens` | `number` | `1024` | Maximum tokens per response. | | `topP` | `number` | `0.9` | Top-p sampling. | | `endpointingSensitivity` | `"LOW" \| "MEDIUM" \| "HIGH" \| null` | `null` | How quickly Nova Sonic decides the user stopped speaking. | ### AudioConfig [#audioconfig] | Option | Type | Required | Description | |---|---|---|---| | `inputSampleRate` | `number \| undefined` | no | | | `inputSampleSize` | `number \| undefined` | no | | | `inputChannelCount` | `number \| undefined` | no | | | `outputSampleRate` | `number \| undefined` | no | | | `outputSampleSize` | `number \| undefined` | no | | | `outputChannelCount` | `number \| undefined` | no | | Defaults: 16 kHz 16-bit mono in, 24 kHz 16-bit mono out. ### Usage [#usage] ```ts title="s2s.ts" import { AudioConfig, AWSNovaSonicLLMService } from "@pipedog/aws/nova_sonic/llm"; import { SessionContinuationParams } from "@pipedog/aws/nova_sonic/session_continuation"; const llm = new AWSNovaSonicLLMService({ // Or leave these out to read the AWS_* environment variables. accessKeyId: process.env.AWS_ACCESS_KEY_ID, secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY, region: "us-east-1", settings: new AWSNovaSonicLLMService.Settings({ voice: "tiffany", systemInstruction: "You are a friendly voice assistant. Keep answers short.", endpointingSensitivity: "MEDIUM", }), audioConfig: new AudioConfig({ inputSampleRate: 16000, outputSampleRate: 24000 }), // Hand the conversation to a fresh session before Nova Sonic's time limit. sessionContinuation: new SessionContinuationParams({ enabled: true }), }); ``` To use HTTP/2 through the AWS SDK, as upstream does: ```ts // Upstream's transport: HTTP/2 through the AWS SDK (Node and Bun only). // Needs `@aws-sdk/client-bedrock-runtime`. const http2Llm = new AWSNovaSonicLLMService({ region: "us-east-1", transport: "http2", }); ``` ### Notes [#notes] * Nova Sonic detects turns itself. User and bot transcripts are pushed as transcription and `LLMTextFrame`s, and audio as `TTSAudioRawFrame`s. * Tools from the context (or `tools`) are sent when the session starts. Tool calls run through the usual function-call flow, and results go back as `toolResult` events. * With `sessionContinuation`, a new session is opened ahead of the time limit (`transitionThresholdSeconds`) and takes over at the next turn boundary, with the conversation history and buffered user audio. * The JSON events on the wire are upstream's, character for character. Float settings use JS formatting, so `1.0` is sent as `1`. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/aws/package.json`: ✅ means the compat smoke test passes on that runtime. * The default transport is Bedrock's WebSocket endpoint for `InvokeModelWithBidirectionalStream`, the one the AWS SDK for JavaScript uses in browsers. pipedog presigns the URL and signs each event with SigV4 on WebCrypto, so it runs on Bun, Node, Deno and Cloudflare Workers. * `transport: "http2"` loads `@aws-sdk/client-bedrock-runtime` on connect and uses Node's `http2`, as upstream does: Node and Bun only. * `clientFactory` replaces the stream client entirely (for tests or another transport), and `endpointUrl` changes the host. * Upstream requires the keys; here they are optional, with the environment and `credentialsProvider` as fallbacks. * Not ported: the deprecated `Params` class and the `model`, `voiceId`, `params` and `systemInstruction` constructor options. Use `settings` and `audioConfig`. # Azure OpenAI Realtime > Speech-to-speech with OpenAI's realtime models deployed on Azure, with AzureRealtimeLLMService. Source: https://polvallverdu.github.io/pipedog/docs/services/s2s/azure-realtime/ ## Overview [#overview] * **`AzureRealtimeLLMService`**: [`OpenAIRealtimeLLMService`](https://polvallverdu.github.io/pipedog/docs/services/s2s/openai.md) pointed at an Azure OpenAI deployment of a realtime model (`gpt-realtime` and friends). The session, events, function calling, transcripts and turn handling are those of the OpenAI service; this class changes the URL and the authentication. It speaks Azure's v1 realtime API (`/openai/v1/realtime`), whose protocol is OpenAI's. URLs with a dated `api-version` serve the older preview protocol, which this service doesn't implement: it logs a warning when given one. For Azure's own speech-to-speech API, with Azure voices, avatars and noise suppression, see [Azure Voice Live](https://polvallverdu.github.io/pipedog/docs/services/s2s/azure-voice-live.md). Pipecat's documentation for the same service. `packages/azure/src/realtime/llm.ts`. The realtime API over WebSocket on Azure. ## Installation [#installation] ```bash bun add @pipedog/azure ``` ```ts import { AzureRealtimeLLMService } from "@pipedog/azure/realtime/llm"; import { SessionProperties } from "@pipedog/openai/realtime/events"; ``` The session classes come from `@pipedog/openai/realtime/events`, which `@pipedog/azure` installs. ## Prerequisites [#prerequisites] * An Azure OpenAI (or Azure AI Foundry) resource with a realtime model deployed. * The resource's API key, or a Microsoft Entra ID identity with access to it. The examples read the key from `AZURE_OPENAI_API_KEY`. ## AzureRealtimeLLMService [#azurerealtimellmservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `baseUrl` | `string` | yes | The Azure Realtime WebSocket endpoint URL, on Azure's v1 API surface. The deployment is appended as `?model=`, so `Settings.model` must name the deployment: wss://my-resource.openai.azure.com/openai/v1/realtime A URL that already carries a query string is used verbatim, leaving the deployment for the caller to name. | | `apiKey` | `string \| null \| undefined` | no | The API key for the Azure OpenAI service. Required unless `tokenProvider` is given. | | `tokenProvider` | `AzureTokenProvider \| null \| undefined` | no | Async callable supplying a Microsoft Entra ID bearer token, used instead of `apiKey`. | | `settings` | `AzureRealtimeLLMSettings \| null \| undefined` | no | Runtime-updatable settings for this service. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `runInParallel` | `boolean \| undefined` | no | Whether to run function calls in parallel or sequentially. Defaults to true. | | `groupParallelTools` | `boolean \| undefined` | no | Whether to group parallel function calls so the LLM is triggered exactly once after all calls in the batch complete. When false, each function call result triggers the LLM independently as it arrives. Defaults to true. | | `functionCallTimeoutSecs` | `number \| null \| undefined` | no | Optional timeout in seconds for function calls. A call that runs past it is cancelled: its handler is cancelled (it gets a `CancelledError`), the call is settled as cancelled, and inference runs so the LLM can report that it didn't complete. | | `fileResolver` | `FileResolver \| null \| undefined` | no | Resolves file URLs in the context that this provider can't fetch itself (see `FileResolver`). Pass one configured with a `fileStorage` to resolve uploaded-file URLs (`new FileResolver({ fileStorage: runnerArgs.fileStorage })` when a runner hosts the bot), and share one instance across services that may consume the same files (e.g. services switched between mid-session) so downloads are reused. Defaults to a resolver that handles `data:` and `http(s)` URLs only. | | `startAudioPaused` | `boolean \| undefined` | no | Whether to start with audio input paused. Defaults to false. | | `startVideoPaused` | `boolean \| undefined` | no | Whether to start with video input paused. Defaults to false. | | `videoFrameDetail` | `ImageDetail \| undefined` | no | Detail level for video processing. Can be "auto", "low", or "high". This sets the image_detail parameter in the OpenAI Realtime API. "auto" lets the model decide, "low" is faster and uses fewer tokens, "high" provides more detail. Defaults to "auto". | | `userAudioPrerollSecs` | `number \| null \| undefined` | no | In manual turn-detection mode (`turn_detection=False`, locally-driven turns), how much recent audio to replay after an interruption clears the input audio buffer, so the speech onset isn't lost. Defaults to null: auto-sized to the upstream VAD's `startSecs` plus a small margin, falling back to `DEFAULT_USER_AUDIO_PREROLL_SECS` when no VAD is present. Auto-sizing assumes VAD drives turn starts (the default `VADUserTurnStartStrategy`); set this explicitly if you use a non-VAD turn-start strategy. No effect when server-side turn detection is enabled. | ### Settings [#settings] `AzureRealtimeLLMService.Settings` (`AzureRealtimeLLMSettings`) has the same fields and defaults as [`OpenAIRealtimeLLMSettings`](https://polvallverdu.github.io/pipedog/docs/services/s2s/openai/#settings). `model` must be your deployment's name: it is appended to `baseUrl` as `?model=`. ### Usage [#usage] ```ts title="realtime.ts" import { AzureRealtimeLLMService } from "@pipedog/azure/realtime/llm"; import { AudioConfiguration, AudioInput, AudioOutput, InputAudioTranscription, SemanticTurnDetection, SessionProperties, } from "@pipedog/openai/realtime/events"; const llm = new AzureRealtimeLLMService({ apiKey: process.env.AZURE_OPENAI_API_KEY ?? "", baseUrl: "wss://my-resource.openai.azure.com/openai/v1/realtime", settings: new AzureRealtimeLLMService.Settings({ // The deployment name, sent as ?model=. model: "my-gpt-realtime-deployment", systemInstruction: "You are a helpful assistant in a voice conversation.", sessionProperties: new SessionProperties({ audio: new AudioConfiguration({ input: new AudioInput({ transcription: new InputAudioTranscription(), turnDetection: new SemanticTurnDetection({ eagerness: "medium" }), }), output: new AudioOutput({ voice: "marin" }), }), }), }), }); ``` With Microsoft Entra ID instead of a key: ```ts // For example getBearerTokenProvider(credential, "https://ai.azure.com/.default") // from @azure/identity. It is called once per connection. declare const tokenProvider: () => Promise; const withEntra = new AzureRealtimeLLMService({ tokenProvider, baseUrl: "wss://my-resource.openai.azure.com/openai/v1/realtime", settings: new AzureRealtimeLLMService.Settings({ model: "my-gpt-realtime-deployment" }), }); ``` ### Event handlers [#event-handlers] The same as [`OpenAIRealtimeLLMService`](https://polvallverdu.github.io/pipedog/docs/services/s2s/openai/#event-handlers): `on_conversation_item_created` and `on_conversation_item_updated`. ### Notes [#notes] * With an API key the socket sends it as the `api-key` header. With `tokenProvider`, the token is fetched before each connection and sent as `Authorization: Bearer `. One of the two is required. The service picks the header in `_authHeaders()`, the hook `OpenAIRealtimeLLMService` calls when it connects, and otherwise connects as that service does. * A `baseUrl` that already has a query string is used as given, so you can name the deployment (or other parameters) yourself. * Frames, turn detection, pre-roll and video input behave as in the [OpenAI Realtime notes](https://polvallverdu.github.io/pipedog/docs/services/s2s/openai/#notes). ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/azure/package.json`: ✅ means the compat smoke test passes on that runtime. * The socket is opened with `websocketConnect()` from core, which sends the auth header from the global `WebSocket` on Bun, Node and Deno, and through `fetch` on Cloudflare Workers. * Browsers can't set WebSocket headers, so this service is for servers. # Azure Voice Live > Speech-to-speech with AzureVoiceLiveLLMService, over Azure's Voice Live API WebSocket. Source: https://polvallverdu.github.io/pipedog/docs/services/s2s/azure-voice-live/ ## Overview [#overview] * **`AzureVoiceLiveLLMService`**: one WebSocket session with Azure's Voice Live API, which hears the user's audio and answers with audio. Voice Live puts speech recognition, a generative model (`gpt-4o-mini`, `gpt-realtime`, `azure-realtime` and others) and Azure text-to-speech behind a single realtime session, and adds noise suppression, echo cancellation and semantic end-of-turn detection. It replaces the STT, LLM and TTS services in a pipeline, and supports function calling and transcripts of both sides. Voice Live is a different product from OpenAI Realtime deployments on Azure: the two speak different event names and are not interchangeable. For those, see [Azure OpenAI Realtime](https://polvallverdu.github.io/pipedog/docs/services/s2s/azure-realtime.md). Pipecat's documentation for the same service. `packages/azure/src/voice_live/llm.ts` and `events.ts`. The Voice Live API and its reference. ## Installation [#installation] ```bash bun add @pipedog/azure ``` ```ts import { AzureVoiceLiveLLMService } from "@pipedog/azure/voice_live/llm"; import { SessionProperties } from "@pipedog/azure/voice_live/events"; ``` `@pipedog/azure/voice_live/events` holds the session classes (`SessionProperties`, `TurnDetection`, `InputAudioTranscription`, `InputAudioNoiseReduction`, `InputAudioEchoCancellation`, the voice types `AzureStandardVoice`, `AzureCustomVoice`, `AzurePersonalVoice`, `AzureRealtimeNativeVoice` and `OpenAIVoice`, ...) and the client and server event models. ## Prerequisites [#prerequisites] * A Microsoft Foundry (AI Services) resource with Voice Live, in a region that offers it. * Its endpoint, as the Azure portal shows it: `https://.services.ai.azure.com` (a `cognitiveservices.azure.com` endpoint or a full `wss://.../voice-live/realtime` URL works too). * Either the resource's API key, passed as `apiKey`, or Microsoft Entra ID: pass `tokenProvider`, an async function returning a bearer token for the `https://ai.azure.com/.default` scope (for example `getBearerTokenProvider()` from `@azure/identity`). One of the two is required. ## AzureVoiceLiveLLMService [#azurevoicelivellmservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `endpoint` | `string` | yes | Voice Live endpoint for the Foundry resource. Accepts the resource endpoint as shown in the Azure portal (`https://.services.ai.azure.com`) or a full WebSocket URL; the scheme and `/voice-live/realtime` path are filled in when absent. | | `apiKey` | `string \| null \| undefined` | no | API key for the resource. Required unless `tokenProvider` is given. | | `tokenProvider` | `AzureTokenProvider \| null \| undefined` | no | Async callable supplying a Microsoft Entra ID bearer token, used instead of `apiKey` when given. Build one with `getBearerTokenProvider()` from `@azure/identity` and the `https://ai.azure.com/.default` scope. | | `apiVersion` | `string \| undefined` | no | Voice Live API version to request. Defaults to `DEFAULT_API_VERSION`. | | `settings` | `AzureVoiceLiveLLMSettings \| null \| undefined` | no | Runtime-updatable settings. `model` defaults to "gpt-4o-mini" and is sent as a query parameter on the connection, so it is fixed for the session. The default voice is "en-US-Ava:DragonHDLatestNeural", except for `azure-realtime`, which picks one of its own native voices. When `sessionProperties` is provided, it **replaces** all defaults wholesale — provide a complete `SessionProperties` in that case. | | `startAudioPaused` | `boolean \| undefined` | no | Whether to start with audio input paused. Defaults to false. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `runInParallel` | `boolean \| undefined` | no | Whether to run function calls in parallel or sequentially. Defaults to true. | | `groupParallelTools` | `boolean \| undefined` | no | Whether to group parallel function calls so the LLM is triggered exactly once after all calls in the batch complete. When false, each function call result triggers the LLM independently as it arrives. Defaults to true. | | `functionCallTimeoutSecs` | `number \| null \| undefined` | no | Optional timeout in seconds for function calls. A call that runs past it is cancelled: its handler is cancelled (it gets a `CancelledError`), the call is settled as cancelled, and inference runs so the LLM can report that it didn't complete. | | `fileResolver` | `FileResolver \| null \| undefined` | no | Resolves file URLs in the context that this provider can't fetch itself (see `FileResolver`). Pass one configured with a `fileStorage` to resolve uploaded-file URLs (`new FileResolver({ fileStorage: runnerArgs.fileStorage })` when a runner hosts the bot), and share one instance across services that may consume the same files (e.g. services switched between mid-session) so downloads are reused. Defaults to a resolver that handles `data:` and `http(s)` URLs only. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new AzureVoiceLiveLLMService.Settings({...})` (`AzureVoiceLiveLLMSettings`). Any setting you leave out keeps the default below. | Setting | Type | Default | Description | | -------------------------------------------------------------------------- | ---------------------------------- | --------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | | `model` | `string` | `"gpt-4o-mini"` | Model backing the session. Fixed for the session: it goes in the connection URL. | | `systemInstruction` | `string \| null` | `null` | Instructions. Kept in sync with `sessionProperties.instructions`. | | `temperature` | `number \| null` | `null` | Sampling temperature, copied into `sessionProperties.temperature`. | | `sessionProperties` | `SessionProperties` | see below | Session config: modalities, voice, turn detection, transcription, noise reduction, echo cancellation, tools, tool choice, max output tokens. | | `maxTokens`, `topP`, `topK`, `frequencyPenalty`, `presencePenalty`, `seed` | `number \| null` | `null` | Not used by Voice Live. | | `filterIncompleteUserTurns` | `boolean` | `false` | Filter incomplete user turns. | | `userTurnCompletionConfig` | `UserTurnCompletionConfig \| null` | `null` | Turn completion config used with `filterIncompleteUserTurns`. | The default `sessionProperties` responds with text and audio in the `en-US-Ava:DragonHDLatestNeural` voice, runs `azure_semantic_vad` turn detection, transcribes the caller with `azure-speech` and applies `azure_deep_noise_suppression`. With an `azure-realtime` model the voice is left out, so the model picks one of its native voices. A `sessionProperties` you pass replaces those defaults as a whole, so give a complete configuration. Fields you leave out of it are not sent, and Voice Live's own defaults apply. Top-level `model`, `systemInstruction` and `temperature` win over the same fields inside it. ### Usage [#usage] ```ts title="voice-live.ts" import { AzureStandardVoice, type ConversationItem, InputAudioNoiseReduction, InputAudioTranscription, SessionProperties, TurnDetection, } from "@pipedog/azure/voice_live/events"; import { AzureVoiceLiveLLMService } from "@pipedog/azure/voice_live/llm"; import { LLMUpdateSettingsFrame } from "@pipedog/core/frames/frames"; const llm = new AzureVoiceLiveLLMService({ apiKey: process.env.AZURE_VOICE_LIVE_API_KEY ?? "", // The resource endpoint as the Azure portal shows it. endpoint: process.env.AZURE_VOICE_LIVE_ENDPOINT ?? "https://my-resource.services.ai.azure.com", settings: new AzureVoiceLiveLLMService.Settings({ model: "gpt-4o-mini", systemInstruction: "You are a helpful assistant in a voice conversation.", // Replaces the default session properties as a whole. sessionProperties: new SessionProperties({ modalities: ["text", "audio"], voice: new AzureStandardVoice({ name: "en-US-Ava:DragonHDLatestNeural" }), turnDetection: new TurnDetection({ type: "azure_semantic_vad", silenceDurationMs: 500, removeFillerWords: true, }), inputAudioTranscription: new InputAudioTranscription({ model: "azure-speech" }), inputAudioNoiseReduction: new InputAudioNoiseReduction(), }), }), }); llm.addEventHandler( "on_conversation_item_created", async (_service, itemId: string | null, _item: ConversationItem) => { console.log("conversation item created:", itemId); }, ); ``` With Microsoft Entra ID instead of a key: ```ts // Microsoft Entra ID instead of a key: any async function returning a bearer // token for the https://ai.azure.com/.default scope, such as // getBearerTokenProvider() from @azure/identity. const entraLlm = new AzureVoiceLiveLLMService({ endpoint: "https://my-resource.services.ai.azure.com", tokenProvider: async () => process.env.AZURE_ENTRA_TOKEN ?? "", }); ``` Change the instructions mid-call with an `LLMUpdateSettingsFrame`. A change to `systemInstruction`, `temperature` or `sessionProperties` is sent as a `session.update`; other fields (including `model`) are ignored with a warning: ```ts // Sends a session.update to the open session. const update = new LLMUpdateSettingsFrame({ delta: new AzureVoiceLiveLLMService.Settings({ systemInstruction: "Answer in Spanish from now on.", }), }); ``` ### Event handlers [#event-handlers] | Event | Description | | ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------- | | `on_conversation_item_created` | A conversation item was added. Called with the item id and the item. | | `on_conversation_item_updated` | A conversation item changed (a transcript arrived, or a response finished). Called with the item id and the item, or `null`. | ### Notes [#notes] * Consumes `InputAudioRawFrame`, `LLMContextFrame` and `LLMSetToolsFrame`. Emits `TTSAudioRawFrame`, `TTSTextFrame` and `LLMTextFrame` for the bot, and `InterimTranscriptionFrame` / `TranscriptionFrame` for the user. `LLMMessagesAppendFrame` is not supported yet and is ignored with a warning. * Audio is 16-bit PCM both ways. The input rate is sent as `input_audio_sampling_rate`; the output rate picks the output format (`pcm16_8000hz`, `pcm16_16000hz` or `pcm16` at 24 kHz), and any other output rate falls back to 24 kHz. G.711 formats in `sessionProperties` are replaced with PCM, with a warning. * With server-side turn detection (the default), the service broadcasts proposed user turn frames from Voice Live's speech events and asks the user aggregator for `ExternalUserTurnStrategies`, so no local VAD is needed. `LLMContextAggregatorPair` detects that this is a realtime service. * For locally driven turns, set `turnDetection: null` (or `false`) in `sessionProperties` and add a VAD to the user aggregator. The service then commits the buffer and asks for a response when the user stops speaking; after an interruption it clears the input buffer and replays the last `startSecs` + 0.3 s of audio, so the start of the user's speech isn't lost. Leaving `turnDetection` out keeps Voice Live's own VAD running. * Without `inputAudioTranscription` the caller's turns never reach the context, so the service warns when your `sessionProperties` leaves it out. Set it to `null` to turn transcription off without the warning. * An interruption cancels the running response; whatever it still streams until `response.done` is dropped. A response asked for while another is running waits for it to finish. * Tool results are sent as `function_call_output` items. Async tools (`cancelOnInterruption: false`) work, but streamed intermediate results don't: Voice Live has no channel for them, so they are dropped with an error. * `resetConversation()` reconnects and sends the current context to the new session. `setAudioInputPaused()` pauses audio input, and `sendClientEvent()` sends any Voice Live client event. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/azure/package.json`: ✅ means the compat smoke test passes on that runtime. * The socket is opened with `websocketConnect()` from core, with the key in an `api-key` header or the Entra token in an `Authorization: Bearer` header. Bun, Node and Deno send headers from the global `WebSocket`; on Cloudflare Workers core opens the socket through `fetch`. Browsers can't set WebSocket headers. * The connection URL carries `api-version=2026-07-15` (the `apiVersion` option) and the model. # Gemini Live on Vertex AI > Real-time speech-to-speech with GeminiLiveVertexLLMService, the Gemini Live API through Vertex AI with service account or ADC authentication. Source: https://polvallverdu.github.io/pipedog/docs/services/s2s/gemini-live-vertex/ ## Overview [#overview] `GeminiLiveVertexLLMService` runs Gemini Live through Vertex AI. It extends [`GeminiLiveLLMService`](https://polvallverdu.github.io/pipedog/docs/services/s2s/gemini-live.md), so audio, transcripts, VAD, tools, thinking and compression work the same way. Authentication, the endpoint and a few features differ. * **`GeminiLiveVertexLLMService`**: Gemini Live on Vertex AI, with a Google Cloud project, a region and service account credentials or Application Default Credentials (ADC). Pipecat's documentation for the same service. `packages/google/src/gemini_live/vertex/llm.ts`. The Live API on Vertex AI. ## Installation [#installation] ```bash bun add @pipedog/google ``` ```ts import { GeminiLiveVertexLLMService } from "@pipedog/google/gemini_live/vertex/llm"; ``` ## Prerequisites [#prerequisites] * A Google Cloud project with the Vertex AI API enabled. * A service account with access to Vertex AI, as a key file or a JSON string, or ADC (`gcloud auth application-default login`, or the attached service account on GCP). * The examples read `GOOGLE_CLOUD_PROJECT_ID`, `GOOGLE_CLOUD_LOCATION`, and `GOOGLE_APPLICATION_CREDENTIALS` (a key file path) or `GOOGLE_CREDENTIALS_JSON` (the key itself). ## GeminiLiveVertexLLMService [#geminilivevertexllmservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `credentials` | `string \| null \| undefined` | no | JSON string of service account credentials. | | `credentialsPath` | `string \| null \| undefined` | no | Path to the service account JSON file. | | `location` | `string` | yes | GCP region for Vertex AI endpoint (e.g., "us-east4"). | | `projectId` | `string` | yes | Google Cloud project ID. | | `settings` | `GeminiLiveVertexLLMSettings \| null \| undefined` | no | Gemini Live LLM settings. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `runInParallel` | `boolean \| undefined` | no | Whether to run function calls in parallel or sequentially. Defaults to true. | | `groupParallelTools` | `boolean \| undefined` | no | Whether to group parallel function calls so the LLM is triggered exactly once after all calls in the batch complete. When false, each function call result triggers the LLM independently as it arrives. Defaults to true. | | `functionCallTimeoutSecs` | `number \| null \| undefined` | no | Optional timeout in seconds for function calls. A call that runs past it is cancelled: its handler is cancelled (it gets a `CancelledError`), the call is settled as cancelled, and inference runs so the LLM can report that it didn't complete. | | `fileResolver` | `FileResolver \| null \| undefined` | no | Resolves file URLs in the context that this provider can't fetch itself (see `FileResolver`). Pass one configured with a `fileStorage` to resolve uploaded-file URLs (`new FileResolver({ fileStorage: runnerArgs.fileStorage })` when a runner hosts the bot), and share one instance across services that may consume the same files (e.g. services switched between mid-session) so downloads are reused. Defaults to a resolver that handles `data:` and `http(s)` URLs only. | | `startAudioPaused` | `boolean \| undefined` | no | Whether to start with audio input paused. Defaults to false. | | `startVideoPaused` | `boolean \| undefined` | no | Whether to start with video input paused. Defaults to false. | | `systemInstruction` | `string \| null \| undefined` | no | System prompt for the model. Defaults to null. | | `tools` | `Record[] \| (FunctionSchema \| DirectFunction)[] \| ToolsSchema \| null \| undefined` | no | Tools available to the model: a `ToolsSchema`, a plain list of direct functions and/or `FunctionSchema` objects (handlers auto-register), or a list of provider-native tool dicts. Defaults to null. | | `inferenceOnContextInitialization` | `boolean \| undefined` | no | Whether to generate a response when context is first set. Defaults to true. | | `userAudioPrerollSecs` | `number \| null \| undefined` | no | In server-VAD-disabled (locally-driven turns) mode, how much recent audio to replay after activity_start so the speech onset isn't clipped. | | `fileApiBaseUrl` | `string \| undefined` | no | Base URL for the Gemini File API. Defaults to the official endpoint. | | `httpOptions` | `GoogleClientHttpOptions \| null \| undefined` | no | HTTP options for the client. | ### Settings [#settings] `GeminiLiveVertexLLMService.Settings` has the same fields as [`GeminiLiveLLMService.Settings`](https://polvallverdu.github.io/pipedog/docs/services/s2s/gemini-live/#settings), with the same defaults except `model`, which is `"google/gemini-3.8-live"` (Vertex AI model identifiers carry the `google/` publisher prefix). ### Usage [#usage] ```ts title="gemini-live-vertex.ts" import { GeminiLiveVertexLLMService } from "@pipedog/google/gemini_live/vertex/llm"; const llm = new GeminiLiveVertexLLMService({ credentialsPath: process.env.GOOGLE_APPLICATION_CREDENTIALS ?? "", projectId: process.env.GOOGLE_CLOUD_PROJECT_ID ?? "", location: process.env.GOOGLE_CLOUD_LOCATION ?? "us-east4", systemInstruction: "You are a friendly voice assistant.", settings: new GeminiLiveVertexLLMService.Settings({ model: "google/gemini-3.8-live", voice: "Charon", }), }); ``` With the service account key as a JSON string: ```ts // A service account key as a JSON string. Leave out both credentials options // to use Application Default Credentials. const fromJson = new GeminiLiveVertexLLMService({ credentials: process.env.GOOGLE_CREDENTIALS_JSON ?? "", projectId: process.env.GOOGLE_CLOUD_PROJECT_ID ?? "", location: "us-east4", }); ``` ### Authentication [#authentication] The service picks credentials in this order: 1. `credentials`: a service account key as a JSON string. It is parsed in the constructor. 2. `credentialsPath`: the path to a key file, read by the client. 3. Neither: Application Default Credentials. All three go to `@google/genai` as `googleAuthOptions` with the `cloud-platform` scope, and google-auth-library inside the SDK loads the key file, finds ADC and refreshes the access token as it expires. Upstream loads the credentials and refreshes a token in the constructor, so missing ADC fails there with "No valid credentials provided.". Here those steps are asynchronous and happen on first connect, so missing credentials surface as a connection error. ### Methods [#methods] `setAudioInputPaused()`, `setVideoInputPaused()`, `setModelModalities()` and `setLanguage()` work as on [Gemini Live](https://polvallverdu.github.io/pipedog/docs/services/s2s/gemini-live/#methods). ### Notes [#notes] * Passing `apiKey` throws: Vertex AI takes `credentials`, `credentialsPath` or ADC. * `projectId` and `location` (a region such as `"us-east4"`) are required. * The Gemini File API isn't available: `fileApi` throws `NotImplementedError`. Use Google Cloud Storage for files. * Vertex's Live endpoint doesn't support `NON_BLOCKING` tool calls. Functions registered with `cancelOnInterruption: false` run synchronously (the conversation pauses while they run), and the first time one is used the service logs an error and pushes an error frame. * `historyConfig` is not sent on Vertex AI. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/google/package.json`: ✅ means the compat smoke test passes on that runtime. Authentication goes through google-auth-library, which only the Node build of `@google/genai` supports. This service runs on Bun, Node and Deno, not on Cloudflare Workers. On Workers, use [`GeminiLiveLLMService`](https://polvallverdu.github.io/pipedog/docs/services/s2s/gemini-live.md) with a Gemini API key. The deprecated upstream `model`, `voiceId` and `params` constructor options are not ported; use `settings`. # Gemini Live > Real-time speech-to-speech with GeminiLiveLLMService over the Gemini Live API, with server or local VAD, tools, thinking and the Gemini File API. Source: https://polvallverdu.github.io/pipedog/docs/services/s2s/gemini-live/ ## Overview [#overview] `GeminiLiveLLMService` connects to the Gemini Live API over a WebSocket. It takes user audio (and text and video), and returns spoken audio, transcripts and tool calls, so one service replaces the STT, LLM and TTS stages of a pipeline. * **`GeminiLiveLLMService`**: the Live session, with server-side or local VAD, function calling, thinking, context window compression, session resumption and Google Search grounding. * **`GeminiVADParams`** and **`ContextWindowCompressionParams`**: settings objects for VAD and compression. * **`GeminiFileAPI`**: a client for the Gemini File API (upload, get, list, delete), available as `llm.fileApi`. For Vertex AI, see [Gemini Live on Vertex AI](https://polvallverdu.github.io/pipedog/docs/services/s2s/gemini-live-vertex.md). Pipecat's documentation for the same service. `packages/google/src/gemini_live/llm.ts` and `file_api.ts`. Models, voices, VAD and session management. ## Installation [#installation] ```bash bun add @pipedog/google ``` ```ts import { ContextWindowCompressionParams, GeminiLiveLLMService, GeminiVADParams, } from "@pipedog/google/gemini_live/llm"; import { GeminiFileAPI } from "@pipedog/google/gemini_live/file_api"; ``` ## Prerequisites [#prerequisites] * A Gemini API key from [Google AI Studio](https://aistudio.google.com/apikey). * Pass the key as `apiKey`. The examples read it from `GOOGLE_API_KEY`. ## GeminiLiveLLMService [#geminilivellmservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | Google AI API key for authentication. | | `startAudioPaused` | `boolean \| undefined` | no | Whether to start with audio input paused. Defaults to false. | | `startVideoPaused` | `boolean \| undefined` | no | Whether to start with video input paused. Defaults to false. | | `systemInstruction` | `string \| null \| undefined` | no | System prompt for the model. Defaults to null. | | `tools` | `Record[] \| (FunctionSchema \| DirectFunction)[] \| ToolsSchema \| null \| undefined` | no | Tools available to the model: a `ToolsSchema`, a plain list of direct functions and/or `FunctionSchema` objects (handlers auto-register), or a list of provider-native tool dicts. Defaults to null. | | `settings` | `GeminiLiveLLMSettings \| null \| undefined` | no | Gemini Live LLM settings. | | `inferenceOnContextInitialization` | `boolean \| undefined` | no | Whether to generate a response when context is first set. Defaults to true. | | `userAudioPrerollSecs` | `number \| null \| undefined` | no | In server-VAD-disabled (locally-driven turns) mode, how much recent audio to replay after activity_start so the speech onset isn't clipped. | | `fileApiBaseUrl` | `string \| undefined` | no | Base URL for the Gemini File API. Defaults to the official endpoint. | | `httpOptions` | `GoogleClientHttpOptions \| null \| undefined` | no | HTTP options for the client. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `runInParallel` | `boolean \| undefined` | no | Whether to run function calls in parallel or sequentially. Defaults to true. | | `groupParallelTools` | `boolean \| undefined` | no | Whether to group parallel function calls so the LLM is triggered exactly once after all calls in the batch complete. When false, each function call result triggers the LLM independently as it arrives. Defaults to true. | | `functionCallTimeoutSecs` | `number \| null \| undefined` | no | Optional timeout in seconds for function calls. A call that runs past it is cancelled: its handler is cancelled (it gets a `CancelledError`), the call is settled as cancelled, and inference runs so the LLM can report that it didn't complete. | | `fileResolver` | `FileResolver \| null \| undefined` | no | Resolves file URLs in the context that this provider can't fetch itself (see `FileResolver`). Pass one configured with a `fileStorage` to resolve uploaded-file URLs (`new FileResolver({ fileStorage: runnerArgs.fileStorage })` when a runner hosts the bot), and share one instance across services that may consume the same files (e.g. services switched between mid-session) so downloads are reused. Defaults to a resolver that handles `data:` and `http(s)` URLs only. | ### Settings [#settings] Passed as `settings: new GeminiLiveLLMService.Settings({...})`. They are sent when the session connects. Changing them later stores the new values but doesn't apply them to the open session (a warning names the settings that didn't take effect). | Setting | Type | Default | Description | | ------------------------------------- | -------------------------------- | ----------------------------------- | ------------------------------------------------------------------------------------------ | | `model` | `string` | `"models/gemini-3.8-live"` | Gemini Live model. | | `voice` | `string` | `"Charon"` | Voice name. | | `language` | `Language \| string` | `"en-US"` | Language. A `Language` value is sent as Gemini's locale code (`Language.ES` as `"es-ES"`). | | `modalities` | `GeminiModalities` | `GeminiModalities.AUDIO` | Response modality. `TEXT` logs a warning: recent models may not support it. | | `mediaResolution` | `GeminiMediaResolution` | `GeminiMediaResolution.UNSPECIFIED` | Video input resolution. | | `vad` | `GeminiVADParams \| null` | `null` (server VAD on) | Voice activity detection. | | `turnCoverage` | `TurnCoverage \| null` | `null` (model default) | Which realtime input a user turn covers. | | `contextWindowCompression` | `ContextWindowCompressionParams` | `{}` (off) | Context window compression. | | `thinking` | `ThinkingConfig` | `{}` | Thinking configuration. Thinking models get `thinkingLevel: "LOW"` when none is set. | | `enableAffectiveDialog` | `boolean` | `false` | Affective dialog. | | `proactivity` | `ProactivityConfig` | `{}` | Proactivity configuration. | | `maxTokens` | `number \| null` | `4096` | Maximum output tokens. | | `temperature`, `topK`, `topP`, `seed` | `number \| null` | `null` | Sampling settings. Gemini treats `seed` as best effort. | `GeminiVADParams`: | Option | Type | Required | Description | |---|---|---|---| | `disabled` | `boolean \| null \| undefined` | no | Whether to disable VAD. Defaults to null (server-side VAD is enabled). | | `startSensitivity` | `StartSensitivity \| null \| undefined` | no | Sensitivity for speech start detection. Defaults to null. | | `endSensitivity` | `EndSensitivity \| null \| undefined` | no | Sensitivity for speech end detection. Defaults to null. | | `prefixPaddingMs` | `number \| null \| undefined` | no | Prefix padding in milliseconds. Defaults to null. | | `silenceDurationMs` | `number \| null \| undefined` | no | Silence duration threshold in milliseconds. Defaults to null. | `ContextWindowCompressionParams`: | Option | Type | Required | Description | |---|---|---|---| | `enabled` | `boolean \| undefined` | no | Whether compression is enabled. Defaults to false. | | `triggerTokens` | `number \| null \| undefined` | no | Token count to trigger compression. null uses 80% of context window. | ### Usage [#usage] ```ts title="gemini-live.ts" import { LLMContext } from "@pipedog/core/processors/aggregators/llm_context"; import { Language } from "@pipedog/core/transcriptions/language"; import { ContextWindowCompressionParams, GeminiLiveLLMService, GeminiVADParams, } from "@pipedog/google/gemini_live/llm"; const llm = new GeminiLiveLLMService({ apiKey: process.env.GOOGLE_API_KEY ?? "", systemInstruction: "You are a friendly voice assistant. Keep answers short.", settings: new GeminiLiveLLMService.Settings({ model: "models/gemini-3.8-live", voice: "Puck", language: Language.EN_US, }), }); ``` Compression and thinking: ```ts const tuned = new GeminiLiveLLMService({ apiKey: process.env.GOOGLE_API_KEY ?? "", settings: new GeminiLiveLLMService.Settings({ temperature: 0.6, contextWindowCompression: new ContextWindowCompressionParams({ enabled: true, triggerTokens: 8000, }), thinking: { includeThoughts: true }, }), }); ``` Local VAD. Disable server-side VAD and configure a VAD analyzer on the user aggregator; the service then sends activity start and end signals to Gemini and replays a short pre-roll of audio (`userAudioPrerollSecs`) so the start of speech isn't clipped: ```ts // Turn off server-side VAD. A local VAD in the user aggregator then drives // turns, and the service sends activity start/end signals to Gemini. const locallyDriven = new GeminiLiveLLMService({ apiKey: process.env.GOOGLE_API_KEY ?? "", settings: new GeminiLiveLLMService.Settings({ vad: new GeminiVADParams({ disabled: true }), }), }); ``` ### Methods [#methods] | Method | Description | | -------------------------------- | ------------------------------------------------------ | | `setAudioInputPaused(paused)` | Stop (`true`) or resume (`false`) sending input audio. | | `setVideoInputPaused(paused)` | Stop or resume sending video frames. | | `setModelModalities(modalities)` | Set the response modality. | | `setLanguage(language)` | Set the language. | | `fileApi` | The `GeminiFileAPI` client, created on first access. | ### Gemini File API [#gemini-file-api] Upload a file, then reference its `uri` in a `file_data` content part of a context message: ```ts // A path is read with node:fs/promises (Bun, Node, Deno). On any runtime, // including Workers, pass the contents instead (bytes, a Blob or a stream): // llm.fileApi.uploadFile(new TextEncoder().encode(menu), "menu.txt"). const uploaded = await llm.fileApi.uploadFile("menu.txt", "Menu"); const file = uploaded.file as { uri: string }; const context = new LLMContext({ messages: [ { role: "user", content: [ { type: "text", text: "What's on the menu today?" }, { type: "file_data", file_data: { mime_type: "text/plain", file_uri: file.uri } }, ], }, ], }); ``` `GeminiFileAPI` has `uploadFile(file, displayName?, { mimeType? })`, `getFile(name)`, `listFiles(pageSize?, pageToken?)` and `deleteFile(name)`. Each returns the parsed JSON response (or `true` for `deleteFile`) and throws when the API returns an error. To build one yourself: | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | Google AI API key. | | `baseUrl` | `string \| null \| undefined` | no | Base URL for the Gemini File API (default is the v1beta endpoint). | | `fs` | `GeminiFileAPIFileSystem \| null \| undefined` | no | Reads paths given to `uploadFile` (`node:fs/promises` satisfies it). Not needed when uploading contents. | | `fetch` | `{ (input: RequestInfo \| URL, init?: RequestInit): Promise; (input: string \| URL \| Request, init?: RequestInit): Promise; } \| null \| undefined` | no | The `fetch` implementation to use. Defaults to the global `fetch`. | ### Event handlers [#event-handlers] Inherited from `LLMService`. | Event | Description | | ----------------------------- | -------------------------------------------------- | | `on_function_calls_started` | Function calls were received and are about to run. | | `on_function_calls_cancelled` | In-progress function calls were cancelled. | ### Notes [#notes] * Bot audio comes out as `TTSStartedFrame`, `TTSAudioRawFrame`s and `TTSStoppedFrame`, with the bot's transcript as `LLMTextFrame`s and `TTSTextFrame`s. User transcripts come out as `TranscriptionFrame`s, aggregated into sentences with a 0.5 second fallback timeout. * With server-side VAD, the service emits no `UserStartedSpeakingFrame` or `UserStoppedSpeakingFrame`. Processors that depend on them (RTVI user speaking events, user idle and mute handling, turn recording) do nothing unless the user aggregator has a VAD analyzer. `LLMContextAggregatorPair` detects this service, so context writes are correct either way. * Tools in the context take precedence over `tools` given at construction. Tools can't change mid-session: `LLMSetToolsFrame` is dropped, with a warning. * Functions registered with `cancelOnInterruption: false` use Gemini's `NON_BLOCKING` tool behavior on models that support it (3.8 and later, and pre-3.0). On Gemini 3.x models before 3.8, the service logs an error and pushes an error the first time one is used. * Grounding metadata (for example from a Google Search tool) produces an `LLMSearchResponseFrame`, as with [`GoogleLLMService`](https://polvallverdu.github.io/pipedog/docs/services/llm/google/#notes). * Video frames are throttled to one per second and sent as JPEG. * The service reconnects after a connection error, resuming the session with the last resumption handle. After three consecutive failures it reports an error and stops. * An `EndFrame` waits for the bot's turn to finish, for at most 30 seconds. * On thinking models, a turn that Gemini reports as still in progress is held open, so a reply spanning several `turn_complete` messages is one assistant turn. * A new system instruction or tools from the context make the service reconnect the session. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/google/package.json`: ✅ means the compat smoke test passes on that runtime. The client is `GoogleGenAI` from `@google/genai`, which opens the Live API over the global `WebSocket`. Server messages arrive through the SDK's callbacks and are fed to the service's receive task, so socket errors go through the same reconnect path as upstream. `GeminiFileAPI` uses the global `fetch` (injectable with `fetch`). `uploadFile()` takes a path, as upstream does, or the file's contents (a `Uint8Array`, an `ArrayBuffer`, a `Blob` or a `ReadableStream`). Contents work on every runtime, Cloudflare Workers included. A path is read with the injected `fs`, or with `node:fs/promises` imported on first use (Bun, Node, Deno); on a runtime without it, a path throws an error that says to pass the contents. The MIME type comes from `mimeType`, else the path's extension, else a `Blob`'s type, else `displayName`'s extension, else `application/octet-stream`. `historyConfig` is set on the connect config as upstream does, but `@google/genai` 2.27 does not forward it in the setup message. The deprecated upstream `InputParams` class and the `model`, `voiceId` and `params` constructor options are not ported; use `settings`. # xAI Grok Realtime > Speech-to-speech with GrokRealtimeLLMService, over xAI's Grok Voice Agent WebSocket. Source: https://polvallverdu.github.io/pipedog/docs/services/s2s/grok/ ## Overview [#overview] * **`GrokRealtimeLLMService`**: one WebSocket session with xAI's Grok Voice Agent API, which hears the user's audio and answers with audio. It replaces the STT, LLM and TTS services in a pipeline, and supports function calling, xAI's built-in tools (web search, X search, file search, remote MCP), transcripts of both sides, DTMF on SIP sessions and fixed spoken lines. Pipecat's documentation for the same service. `packages/xai/src/realtime/llm.ts` and `events.ts`. The Grok Voice Agent API, voices and tools. ## Installation [#installation] ```bash bun add @pipedog/xai ``` ```ts import { GrokRealtimeLLMService } from "@pipedog/xai/realtime/llm"; import { SessionProperties } from "@pipedog/xai/realtime/events"; ``` `@pipedog/xai/realtime/events` holds the session classes (`SessionProperties`, `AudioConfiguration`, `AudioInput`, `AudioOutput`, `InputAudioTranscription`, `TurnDetection`, `Reasoning`, `SessionResumption`), the tools (`WebSearchTool`, `XSearchTool`, `FileSearchTool`, `FunctionTool`, `McpTool`) and the client and server event models. ## Prerequisites [#prerequisites] * An xAI account and an API key from the [xAI console](https://console.x.ai). * Pass the key as `apiKey` (required); it is sent as a bearer token. The examples read it from `XAI_API_KEY`. ## GrokRealtimeLLMService [#grokrealtimellmservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | xAI API key for authentication. | | `baseUrl` | `string \| undefined` | no | WebSocket base URL for the realtime API. Defaults to "wss://api.x.ai/v1/realtime". | | `settings` | `GrokRealtimeLLMSettings \| null \| undefined` | no | Runtime-updatable settings for this service. Set the voice and the rest of the session with `sessionProperties`, e.g. `new SessionProperties({ voice: "rex" })`. Built-in voice IDs are documented by xAI (https://docs.x.ai/docs/guides/voice/agent); a custom voice ID from the Custom Voices API may be used too. | | `startAudioPaused` | `boolean \| undefined` | no | Whether to start with audio input paused. Defaults to false. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `runInParallel` | `boolean \| undefined` | no | Whether to run function calls in parallel or sequentially. Defaults to true. | | `groupParallelTools` | `boolean \| undefined` | no | Whether to group parallel function calls so the LLM is triggered exactly once after all calls in the batch complete. When false, each function call result triggers the LLM independently as it arrives. Defaults to true. | | `functionCallTimeoutSecs` | `number \| null \| undefined` | no | Optional timeout in seconds for function calls. A call that runs past it is cancelled: its handler is cancelled (it gets a `CancelledError`), the call is settled as cancelled, and inference runs so the LLM can report that it didn't complete. | | `fileResolver` | `FileResolver \| null \| undefined` | no | Resolves file URLs in the context that this provider can't fetch itself (see `FileResolver`). Pass one configured with a `fileStorage` to resolve uploaded-file URLs (`new FileResolver({ fileStorage: runnerArgs.fileStorage })` when a runner hosts the bot), and share one instance across services that may consume the same files (e.g. services switched between mid-session) so downloads are reused. Defaults to a resolver that handles `data:` and `http(s)` URLs only. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new GrokRealtimeLLMService.Settings({...})` (`GrokRealtimeLLMSettings`). Any setting you leave out keeps the default below. | Setting | Type | Default | Description | | ----------------------------------------------------------------------------------------- | ---------------------------------- | ------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | | `model` | `string` | `"grok-voice-latest"` | Voice model. Fixed for the session: it goes in the connection URL. | | `systemInstruction` | `string \| null` | `null` | Instructions. Kept in sync with `sessionProperties.instructions`. | | `sessionProperties` | `SessionProperties` | `new SessionProperties()` | Session config: `voice` (default `"eve"`), `turnDetection` (default server VAD), audio, tools, reasoning, resumption, pronunciation `replace` map. | | `temperature`, `maxTokens`, `topP`, `topK`, `frequencyPenalty`, `presencePenalty`, `seed` | `number \| null` | `null` | Not used by the Voice Agent API. | | `filterIncompleteUserTurns` | `boolean` | `false` | Filter incomplete user turns. | | `userTurnCompletionConfig` | `UserTurnCompletionConfig \| null` | `null` | Turn completion config used with `filterIncompleteUserTurns`. | A `sessionProperties` in an update replaces the stored one as a whole. The top-level `systemInstruction` wins over `instructions` inside it. Audio formats default to PCM at the pipeline's rates. ### Usage [#usage] ```ts title="realtime.ts" import { LLMUpdateSettingsFrame } from "@pipedog/core/frames/frames"; import { AudioConfiguration, AudioInput, type ConversationItem, InputAudioTranscription, SessionProperties, TurnDetection, WebSearchTool, } from "@pipedog/xai/realtime/events"; import { GrokRealtimeLLMService } from "@pipedog/xai/realtime/llm"; const llm = new GrokRealtimeLLMService({ apiKey: process.env.XAI_API_KEY ?? "", settings: new GrokRealtimeLLMService.Settings({ systemInstruction: "You are a helpful assistant in a voice conversation.", sessionProperties: new SessionProperties({ voice: "rex", turnDetection: new TurnDetection({ type: "server_vad", silenceDurationMs: 500 }), audio: new AudioConfiguration({ input: new AudioInput({ transcription: new InputAudioTranscription({ model: "grok-transcribe" }), }), }), tools: [new WebSearchTool()], }), }), }); llm.addEventHandler( "on_conversation_item_created", async (_service, itemId: string, _item: ConversationItem) => { console.log("conversation item created:", itemId); }, ); ``` Speak a fixed line without involving the model: ```ts // Speak a fixed line without the model (no response.create follows). await llm.forceMessage("This call is being recorded."); ``` Change the instructions mid-call with an `LLMUpdateSettingsFrame`. A change to `systemInstruction` or `sessionProperties` is sent as a `session.update`; other fields are ignored with a warning: ```ts // Sends a session.update to the open session. const update = new LLMUpdateSettingsFrame({ delta: new GrokRealtimeLLMService.Settings({ systemInstruction: "Answer in Spanish from now on.", }), }); ``` ### Event handlers [#event-handlers] | Event | Description | | -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | | `on_conversation_item_created` | A conversation item was added. Called with the item id and the item. | | `on_conversation_item_updated` | A conversation item changed (a transcript arrived, or the response finished). Called with the item id and the item, or `null`. | | `on_conversation_item_deleted` | The server confirmed a deletion. Called with the item id. | | `on_conversation_item_truncated` | The server confirmed a truncation. Called with the item id, content index and audio end (ms). | | `on_idle_timeout` | `turnDetection.idleTimeoutMs` fired and the server is checking in. Called with the item id. | | `on_dtmf_received` | A DTMF digit arrived on a SIP session. Called with the `KeypadEntry`. | | `on_mcp_event` | An MCP tool discovery or call event. Called with the event type and the event. | ### Notes [#notes] * Consumes `InputAudioRawFrame` (sent once the session is configured), `LLMContextFrame`, `LLMMessagesAppendFrame` and `LLMSetToolsFrame`. Emits `TTSAudioRawFrame`, `TTSTextFrame` and `LLMTextFrame` for the bot, `InterimTranscriptionFrame` (with `grok-transcribe`) and `TranscriptionFrame` for the user, and `InputDTMFFrame` upstream for DTMF digits. * With server-side VAD (the default), the service broadcasts proposed user turn frames from the server's speech events and asks the user aggregator for `ExternalUserTurnStrategies`, so no local VAD is needed. On interruption it cancels the response and truncates the assistant audio to what was played. * For locally driven turns, set `turnDetection: null` in `SessionProperties` and add a VAD to the user aggregator; the service then commits the audio and asks for a response when the user stops speaking. * Intermediate tool results (`isFinal: false`) go in as conversation messages beside the open call; the final result answers the call. * `forceMessage()`, `deleteConversationItem()`, `setAudioInputPaused()`, `sendClientEvent()` and `resetConversation()` are available at runtime. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/xai/package.json`: ✅ means the compat smoke test passes on that runtime. * The socket is opened with `websocketConnect()` from core, with the key in the `Authorization` header. Bun, Node and Deno send it from the global `WebSocket`; on Cloudflare Workers core opens the socket through `fetch`. * The deprecated `sessionProperties` constructor option and the `pipecat.services.grok.realtime` aliases are not ported: import from `@pipedog/xai/realtime/*` and pass `sessionProperties` in `settings`. * A failed response whose `status_details` isn't a string is reported as JSON (upstream prints a Python dict). # Speech-to-speech > Realtime models that take audio in and give audio out, in one service. Source: https://polvallverdu.github.io/pipedog/docs/services/s2s/ {/* Generated by scripts/generate-providers.ts from data/providers.ts. */} A speech-to-speech service replaces the STT, LLM and TTS trio with one realtime model: audio in, audio and text out, with function calls. | Provider | Classes | Package | Status | | ----------------------------------------------------------------- | ---------------------------- | ------------------- | ------ | | [OpenAI Realtime](https://polvallverdu.github.io/pipedog/docs/services/s2s/openai.md) | `OpenAIRealtimeLLMService` | `@pipedog/openai` | Ported | | [OpenAI Live](https://polvallverdu.github.io/pipedog/docs/services/s2s/openai-live.md) | `OpenAILiveLLMService` | `@pipedog/openai` | Ported | | [Gemini Live](https://polvallverdu.github.io/pipedog/docs/services/s2s/gemini-live.md) | `GeminiLiveLLMService` | `@pipedog/google` | Ported | | [Gemini Live on Vertex AI](https://polvallverdu.github.io/pipedog/docs/services/s2s/gemini-live-vertex.md) | `GeminiLiveVertexLLMService` | `@pipedog/google` | Ported | | [Inworld Realtime](https://polvallverdu.github.io/pipedog/docs/services/s2s/inworld.md) | `InworldRealtimeLLMService` | `@pipedog/inworld` | Ported | | [Ultravox](https://polvallverdu.github.io/pipedog/docs/services/s2s/ultravox.md) | `UltravoxRealtimeLLMService` | `@pipedog/ultravox` | Ported | | [xAI Grok Realtime](https://polvallverdu.github.io/pipedog/docs/services/s2s/grok.md) | `GrokRealtimeLLMService` | `@pipedog/xai` | Ported | | [AWS Nova Sonic](https://polvallverdu.github.io/pipedog/docs/services/s2s/aws.md) | `AWSNovaSonicLLMService` | `@pipedog/aws` | Ported | | [Azure Voice Live](https://polvallverdu.github.io/pipedog/docs/services/s2s/azure-voice-live.md) | `AzureVoiceLiveLLMService` | `@pipedog/azure` | Ported | | [Azure OpenAI Realtime](https://polvallverdu.github.io/pipedog/docs/services/s2s/azure-realtime.md) | `AzureRealtimeLLMService` | `@pipedog/azure` | Ported | Pipecat's [speech-to-speech reference](https://docs.pipecat.ai/api-reference/server/services/s2s) lists every upstream provider. Those not in this table are on [Other upstream providers](https://polvallverdu.github.io/pipedog/docs/services/other-upstream.md). # Inworld Realtime > Speech-to-speech with InworldRealtimeLLMService, over Inworld's Realtime API WebSocket. Source: https://polvallverdu.github.io/pipedog/docs/services/s2s/inworld/ ## Overview [#overview] * **`InworldRealtimeLLMService`**: one WebSocket session with Inworld's Realtime API, which hears the user's audio and answers with audio. Under the hood Inworld runs STT, an LLM of your choice (for example `openai/gpt-4.1-mini`) and Inworld TTS, with semantic server-side turn detection. It replaces the STT, LLM and TTS services in a pipeline, and supports function calling and transcripts of both sides. Pipecat's documentation for the same service. `packages/inworld/src/realtime/llm.ts` and `events.ts`. The Realtime API over WebSocket. ## Installation [#installation] ```bash bun add @pipedog/inworld ``` ```ts import { InworldRealtimeLLMService } from "@pipedog/inworld/realtime/llm"; import { SessionProperties } from "@pipedog/inworld/realtime/events"; ``` `@pipedog/inworld/realtime/events` holds the session classes (`SessionProperties`, `AudioConfiguration`, `AudioInput`, `AudioOutput`, `InputTranscription`, `TurnDetection`, `PCMAudioFormat`, ...) and the client and server event models. ## Prerequisites [#prerequisites] * An Inworld account and an API key from the [Inworld portal](https://platform.inworld.ai). * Pass the key as `apiKey` (required). With the default `authType: "basic"` it is sent as `Authorization: Basic ` (the Base64 key Inworld gives you). Use `authType: "bearer"` for a client-side JWT. The examples read it from `INWORLD_API_KEY`. ## InworldRealtimeLLMService [#inworldrealtimellmservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | Inworld API key for authentication. | | `llmModel` | `string \| null \| undefined` | no | LLM model to use (e.g. "openai/gpt-4.1-mini"). Shorthand for `sessionProperties.model`. | | `voice` | `string \| null \| undefined` | no | Voice ID for TTS output (e.g. "Sarah", "Clive"). Shorthand for `sessionProperties.audio.output.voice`. | | `ttsModel` | `string \| null \| undefined` | no | TTS model to use (e.g. "inworld-tts-2"). Shorthand for `sessionProperties.audio.output.model`. | | `sttModel` | `string \| null \| undefined` | no | STT model for input transcription (e.g. "inworld/inworld-stt-1"). Defaults to "inworld/inworld-stt-1". Shorthand for `sessionProperties.audio.input.transcription.model`. | | `baseUrl` | `string \| undefined` | no | WebSocket base URL for the realtime API. Defaults to "wss://api.inworld.ai/api/v1/realtime/session". | | `authType` | `"basic" \| "bearer" \| undefined` | no | Authentication type. "basic" for server-side API key auth, "bearer" for client-side JWT auth. Defaults to "basic". | | `settings` | `InworldRealtimeLLMSettings \| null \| undefined` | no | Full settings for fine-grained control. When `sessionProperties` is provided in settings, it **replaces** all defaults wholesale — provide a complete `SessionProperties` in that case. | | `startAudioPaused` | `boolean \| undefined` | no | Whether to start with audio input paused. Defaults to false. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `runInParallel` | `boolean \| undefined` | no | Whether to run function calls in parallel or sequentially. Defaults to true. | | `groupParallelTools` | `boolean \| undefined` | no | Whether to group parallel function calls so the LLM is triggered exactly once after all calls in the batch complete. When false, each function call result triggers the LLM independently as it arrives. Defaults to true. | | `functionCallTimeoutSecs` | `number \| null \| undefined` | no | Optional timeout in seconds for function calls. A call that runs past it is cancelled: its handler is cancelled (it gets a `CancelledError`), the call is settled as cancelled, and inference runs so the LLM can report that it didn't complete. | | `fileResolver` | `FileResolver \| null \| undefined` | no | Resolves file URLs in the context that this provider can't fetch itself (see `FileResolver`). Pass one configured with a `fileStorage` to resolve uploaded-file URLs (`new FileResolver({ fileStorage: runnerArgs.fileStorage })` when a runner hosts the bot), and share one instance across services that may consume the same files (e.g. services switched between mid-session) so downloads are reused. Defaults to a resolver that handles `data:` and `http(s)` URLs only. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new InworldRealtimeLLMService.Settings({...})` (`InworldRealtimeLLMSettings`). Any setting you leave out keeps the default below. | Setting | Type | Default | Description | | -------------------------------------------------------------------------- | ---------------------------------- | -------------------------------------- | ------------------------------------------------------------------------ | | `model` | `string` | `llmModel`, or `"openai/gpt-4.1-mini"` | The LLM Inworld runs. Kept in sync with `sessionProperties.model`. | | `systemInstruction` | `string \| null` | `null` | Instructions. Kept in sync with `sessionProperties.instructions`. | | `temperature` | `number \| null` | `null` | Sampling temperature. Kept in sync with `sessionProperties.temperature`. | | `sessionProperties` | `SessionProperties` | see below | Session config: output modalities, audio input and output, tools. | | `maxTokens`, `topP`, `topK`, `frequencyPenalty`, `presencePenalty`, `seed` | `number \| null` | `null` | Not used by the Realtime API. | | `filterIncompleteUserTurns` | `boolean` | `false` | Filter incomplete user turns. | | `userTurnCompletionConfig` | `UserTurnCompletionConfig \| null` | `null` | Turn completion config used with `filterIncompleteUserTurns`. | The default session has output modalities `["audio", "text"]`, PCM audio both ways (the rates follow the pipeline's), transcription with `sttModel` (default `"inworld/inworld-stt-1"`), semantic turn detection with low eagerness that creates and interrupts responses, and the `voice` (default `"Clive"`) and `ttsModel` (default `"inworld-tts-2"`) for output. A `sessionProperties` you pass **replaces** those defaults as a whole, so give a complete config. Top-level `model`, `systemInstruction` and `temperature` win over the same fields inside it. ### Usage [#usage] ```ts title="realtime.ts" import { LLMUpdateSettingsFrame } from "@pipedog/core/frames/frames"; import { AudioConfiguration, AudioInput, AudioOutput, type ConversationItem, InputTranscription, PCMAudioFormat, SessionProperties, TurnDetection, } from "@pipedog/inworld/realtime/events"; import { InworldRealtimeLLMService } from "@pipedog/inworld/realtime/llm"; // The shorthand options fill in the default session. const llm = new InworldRealtimeLLMService({ apiKey: process.env.INWORLD_API_KEY ?? "", llmModel: "openai/gpt-4.1-mini", voice: "Sarah", ttsModel: "inworld-tts-2", settings: new InworldRealtimeLLMService.Settings({ systemInstruction: "You are a helpful assistant in a voice conversation.", }), }); llm.addEventHandler( "on_conversation_item_created", async (_service, itemId: string, _item: ConversationItem) => { console.log("conversation item created:", itemId); }, ); ``` With full control over the session: ```ts // A full SessionProperties replaces the defaults wholesale: give a complete config. const custom = new InworldRealtimeLLMService({ apiKey: process.env.INWORLD_API_KEY ?? "", settings: new InworldRealtimeLLMService.Settings({ sessionProperties: new SessionProperties({ model: "openai/gpt-4.1-mini", temperature: 0.7, outputModalities: ["audio", "text"], audio: new AudioConfiguration({ input: new AudioInput({ format: new PCMAudioFormat({ rate: 24000 }), transcription: new InputTranscription({ model: "inworld/inworld-stt-1" }), turnDetection: new TurnDetection({ type: "semantic_vad", eagerness: "low" }), }), output: new AudioOutput({ format: new PCMAudioFormat({ rate: 24000 }), voice: "Sarah", model: "inworld-tts-2", }), }), }), }), }); ``` Change the instructions mid-call with an `LLMUpdateSettingsFrame`. A change to `sessionProperties`, `systemInstruction`, `model` or `temperature` is sent as a `session.update`; other fields are ignored with a warning: ```ts // Sends a session.update to the open session. const update = new LLMUpdateSettingsFrame({ delta: new InworldRealtimeLLMService.Settings({ systemInstruction: "Answer in Spanish from now on.", temperature: 0.3, }), }); ``` ### Event handlers [#event-handlers] | Event | Description | | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------ | | `on_conversation_item_created` | A conversation item was added. Called with the item id and the item. | | `on_conversation_item_updated` | A conversation item changed (a transcript arrived, or the response finished). Called with the item id and the item, or `null`. | ### Notes [#notes] * Consumes `InputAudioRawFrame` (sent in about 60 ms chunks), `LLMContextFrame` and `LLMSetToolsFrame`. Emits `TTSAudioRawFrame`, `TTSTextFrame` and `LLMTextFrame` for the bot, and `InterimTranscriptionFrame` / `TranscriptionFrame` for the user. * With Inworld's server-side turn detection (the default), the service broadcasts proposed user turn frames from the server's speech events and asks the user aggregator for `ExternalUserTurnStrategies`, so no local VAD is needed. `LLMContextAggregatorPair` detects that this is a realtime service. * For locally driven turns, set `turnDetection: null` in `AudioInput` and add a VAD to the user aggregator; the service then commits the audio and asks for a response when the user stops speaking. Don't run a local VAD with server-side turn detection on: both would send turn frames. * Tool results are sent as `function_call_output` items. Intermediate results (`isFinal: false`) are dropped with a warning: Inworld's model calls the tool again instead of relaying them. `cancelOnInterruption: false` isn't reliably supported by Inworld, and the service reports an error when it sees one. `LLMMessagesAppendFrame` isn't supported yet. * `setAudioInputPaused()` pauses input at runtime, `sendClientEvent()` sends any Realtime client event, and `resetConversation()` reconnects with a fresh server-side conversation. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/inworld/package.json`: ✅ means the compat smoke test passes on that runtime. * The socket is opened with `websocketConnect()` from core, with the key in the `Authorization` header and a fresh session `key` in the URL. Bun, Node and Deno send the header from the global `WebSocket`; on Cloudflare Workers core opens the socket through `fetch`. * A failed response whose `status_details` isn't a string is reported as JSON (upstream prints a Python dict). # OpenAI Live > Full-duplex speech-to-speech with OpenAILiveLLMService, which delegates work to a backend model through ResponsesDelegation or ClientDelegation. Source: https://polvallverdu.github.io/pipedog/docs/services/s2s/openai-live/ ## Overview [#overview] * **`OpenAILiveLLMService`**: a session with OpenAI's Live API. The live model (`gpt-live-1`) listens and speaks at the same time. It decides when to answer, when to stop when the user talks over it, and when to delegate work (search, reasoning, tool use) to a backend text model while the conversation goes on. There is no client-side turn detection or response trigger: the pipeline streams audio in and plays audio out. Where delegated work runs is set with `delegation`: * **`ResponsesDelegation`** (`OpenAILiveLLMService.ResponsesDelegation`): OpenAI hosts the backend model on the Responses API. Its function calls run here, with the handlers registered for the tools in the pipeline's `LLMContext`. * **`ClientDelegation`** (`OpenAILiveLLMService.ClientDelegation`): a `BackendLLMWorker` that runs any pipedog LLM service is the backend. * `null` (the default): client delegation with no backend. Delegated requests are declined. Pipecat's documentation for the same service. `packages/openai/src/live/llm.ts` and `events.ts`. The Live API guide. ## Installation [#installation] ```bash bun add @pipedog/openai ``` ```ts import { ClientDelegation, OpenAILiveLLMService, ResponsesDelegation } from "@pipedog/openai/live/llm"; import { BackendLLMWorker } from "@pipedog/core/workers/llm/backend_llm_worker"; ``` ## Prerequisites [#prerequisites] * An OpenAI account and a project API key from the [OpenAI dashboard](https://platform.openai.com/api-keys). * Pass the key as `apiKey` (required). The examples read it from `OPENAI_API_KEY`. ## OpenAILiveLLMService [#openailivellmservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | OpenAI project API key. | | `baseUrl` | `string \| undefined` | no | WebSocket base URL of the Live API. | | `settings` | `OpenAILiveLLMSettings \| null \| undefined` | no | Runtime-updatable settings. `model`, `voice`, `store` and `systemInstruction` are fixed once the session has started. | | `delegation` | `ClientDelegation \| ResponsesDelegation \| null \| undefined` | no | Where the model's delegated work runs. `null` selects client delegation with no backend configured. | | `webSocketFactory` | `((uri: string, options: { headers?: Record; protocols?: string[]; }) => WebSocket \| Promise) \| null \| undefined` | no | Creates the underlying `WebSocket` (see `WebsocketConnectOptions`), for runtimes other than Bun, Node, Deno and Cloudflare Workers, whose `WebSocket` can't send the `Authorization` header. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `runInParallel` | `boolean \| undefined` | no | Whether to run function calls in parallel or sequentially. Defaults to true. | | `groupParallelTools` | `boolean \| undefined` | no | Whether to group parallel function calls so the LLM is triggered exactly once after all calls in the batch complete. When false, each function call result triggers the LLM independently as it arrives. Defaults to true. | | `functionCallTimeoutSecs` | `number \| null \| undefined` | no | Optional timeout in seconds for function calls. A call that runs past it is cancelled: its handler is cancelled (it gets a `CancelledError`), the call is settled as cancelled, and inference runs so the LLM can report that it didn't complete. | | `fileResolver` | `FileResolver \| null \| undefined` | no | Resolves file URLs in the context that this provider can't fetch itself (see `FileResolver`). Pass one configured with a `fileStorage` to resolve uploaded-file URLs (`new FileResolver({ fileStorage: runnerArgs.fileStorage })` when a runner hosts the bot), and share one instance across services that may consume the same files (e.g. services switched between mid-session) so downloads are reused. Defaults to a resolver that handles `data:` and `http(s)` URLs only. | `ResponsesDelegation` options: | Option | Type | Required | Description | |---|---|---|---| | `settings` | `OpenAIResponsesLLMSettings` | yes | Request settings for the backend model — the same object `OpenAIResponsesLLMService` takes. `model` is required. | | `serviceTier` | `string \| null \| undefined` | no | Responses API service tier for delegated requests (`auto`, `default`, `flex` or `priority`). | `ClientDelegation` options: | Option | Type | Required | Description | |---|---|---|---| | `backend` | `string \| BackendLLMWorker` | yes | The worker that runs delegated work — normally a `BackendLLMWorker` wrapping any LLM service, or the name of one registered elsewhere. | | `timeoutSecs` | `number \| undefined` | no | How long a delegation may take before it is abandoned. Defaults to 120. | ### Settings [#settings] Settings passed as `settings: new OpenAILiveLLMService.Settings({...})` (`OpenAILiveLLMSettings`). `model`, `voice`, `store` and `systemInstruction` are fixed once the session has started. | Setting | Type | Default | Description | | ----------------------------------------------------------------------------------------- | ---------------------------------- | -------------- | --------------------------------------------------------------------------------------------------------------------------------- | | `model` | `string` | `"gpt-live-1"` | Live model. | | `systemInstruction` | `string \| null` | `null` | Instructions, used when the context has no leading system message. | | `voice` | `string \| null` | `null` | Output voice, for example `"marin"` or `"cedar"`. `null` leaves it to the API. | | `store` | `boolean \| null` | `null` | Keep the session for 30 days (recording and forking). The API default is `false`; it needs a project whose data policy allows it. | | `temperature`, `maxTokens`, `topP`, `topK`, `frequencyPenalty`, `presencePenalty`, `seed` | `number \| null` | `null` | Not used by the live model. | | `filterIncompleteUserTurns` | `boolean` | `false` | Filter incomplete user turns. | | `userTurnCompletionConfig` | `UserTurnCompletionConfig \| null` | `null` | Turn completion config used with `filterIncompleteUserTurns`. | The backend model of a `ResponsesDelegation` takes an `OpenAIResponsesLLMService.Settings` (see [OpenAI Responses](https://polvallverdu.github.io/pipedog/docs/services/llm/openai-responses.md)). `model` is required. `systemInstruction` becomes the backend's instructions, `maxCompletionTokens` its `max_output_tokens`, `reasoning` is sent when set, and `extra` is merged into the delegation config. The sampling fields, `topK`, `maxTokens` and the user turn fields are dropped with a warning. ### Usage [#usage] ```ts title="live.ts" import { LLMContext } from "@pipedog/core/processors/aggregators/llm_context"; import { BackendLLMWorker } from "@pipedog/core/workers/llm/backend_llm_worker"; import { OpenAILiveLLMService } from "@pipedog/openai/live/llm"; import { OpenAILLMService } from "@pipedog/openai/llm"; import { OpenAIResponsesLLMService } from "@pipedog/openai/responses/llm"; const llm = new OpenAILiveLLMService({ apiKey: process.env.OPENAI_API_KEY ?? "", settings: new OpenAILiveLLMService.Settings({ systemInstruction: "You are a helpful assistant in a voice conversation.", voice: "marin", }), delegation: new OpenAILiveLLMService.ResponsesDelegation({ settings: new OpenAIResponsesLLMService.Settings({ model: "gpt-5.4-mini" }), }), }); llm.addEventHandler("on_session_started", async (_service, session: unknown) => { console.log("live session started:", session); }); ``` Client delegation, with a pipedog LLM service as the backend: ```ts // Any pipedog LLM service can be the backend. Run the pipeline under a WorkerRunner. const backend = new BackendLLMWorker({ llm: new OpenAILLMService({ apiKey: process.env.OPENAI_API_KEY ?? "" }), context: new LLMContext({ messages: [{ role: "system", content: "You look things up for the assistant." }], }), }); const live = new OpenAILiveLLMService({ apiKey: process.env.OPENAI_API_KEY ?? "", delegation: new OpenAILiveLLMService.ClientDelegation({ backend, timeoutSecs: 60 }), }); ``` ### Event handlers [#event-handlers] | Event | Description | | ----------------------- | ----------------------------------------------------------------- | | `on_session_started` | The session is ready for audio. Called with the session resource. | | `on_delegation_created` | The model delegated work. Called with the delegation item. | ### Notes [#notes] * The session starts on the first `LLMContextFrame` (usually queued as an `LLMRunFrame`). The context's leading system message (or `systemInstruction`) becomes the instructions, and the other text messages seed the session as prior conversation. * Audio input is resampled to 24 kHz. The bot's audio comes out as `SpeechOutputAudioRawFrame`s between `TTSStartedFrame` and `TTSStoppedFrame`, with `LLMTextFrame` / `TTSTextFrame` for its words and `InterimTranscriptionFrame` / `TranscriptionFrame` for the user's. * The service proposes user turns from the API's transcript turns and asks for `ExternalUserTurnStrategies({ enableInterruptions: false })`. No `InterruptionFrame` is broadcast: the model handles being talked over itself, and a delegation keeps running. So every tool behaves as `cancelOnInterruption: false`; use `cancellableByLlm: true` for tools the model should be able to cancel. * Unlike the Realtime services, it is not flagged as a realtime service for the aggregators. It closes each user turn itself once the user falls quiet, so the user aggregator writes the turn to the context before any tool calls it triggers. * With `ClientDelegation`, each delegation is sent to the backend with the transcript since the previous one, and delegations run one at a time. A backend worker given as an object is added as a child of the pipeline worker at setup, so the pipeline must run under a `WorkerRunner` (see [pipelines and workers](https://polvallverdu.github.io/pipedog/docs/concepts/pipelines-and-workers.md)). The backend's output is appended to the session as spoken commentary or as silent context, by its `prefersSpoken` flag. * An `LLMUpdateSettingsFrame` is not sent to the session: the service logs a warning for each changed field. `resetConversation()` drops the session and opens a new one from the current context. * Constructing a `ResponsesDelegation` without `settings.model`, or the service without a `model`, throws. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/openai/package.json`: ✅ means the compat smoke test passes on that runtime. * The socket is opened with `websocketConnect()` from core, and the key is sent as an `Authorization` header. Bun, Node and Deno send it from the global `WebSocket`; on Cloudflare Workers core opens the socket through `fetch`. `webSocketFactory` (a pipedog addition) lets you open it some other way. * The backend model's `OpenAIResponsesReasoningConfig` is sent as its set fields (`effort`, `summary`, `mode`). # OpenAI Realtime > Speech-to-speech with OpenAIRealtimeLLMService, over OpenAI's Realtime API WebSocket. Source: https://polvallverdu.github.io/pipedog/docs/services/s2s/openai/ ## Overview [#overview] * **`OpenAIRealtimeLLMService`**: one WebSocket session with OpenAI's Realtime API, which hears the user's audio and answers with audio. It replaces the STT, LLM and TTS services in a pipeline. It supports function calling, transcripts of both sides, video input and the reasoning-capable Realtime models. For OpenAI's full-duplex live model, see [OpenAI Live](https://polvallverdu.github.io/pipedog/docs/services/s2s/openai-live.md). Pipecat's documentation for the same service. `packages/openai/src/realtime/llm.ts` and `events.ts`. The Realtime API over WebSocket. ## Installation [#installation] ```bash bun add @pipedog/openai ``` ```ts import { OpenAIRealtimeLLMService } from "@pipedog/openai/realtime/llm"; import { SessionProperties } from "@pipedog/openai/realtime/events"; ``` `@pipedog/openai/realtime/events` holds the session classes (`SessionProperties`, `AudioConfiguration`, `AudioInput`, `AudioOutput`, `InputAudioTranscription`, `TurnDetection`, `SemanticTurnDetection`, `Reasoning`, ...) and the client and server event models. ## Prerequisites [#prerequisites] * An OpenAI account and an API key from the [OpenAI dashboard](https://platform.openai.com/api-keys). * Pass the key as `apiKey` (required). The examples read it from `OPENAI_API_KEY`. ## OpenAIRealtimeLLMService [#openairealtimellmservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | OpenAI API key for authentication. | | `baseUrl` | `string \| undefined` | no | WebSocket base URL for the realtime API. Defaults to "wss://api.openai.com/v1/realtime". | | `settings` | `OpenAIRealtimeLLMSettings \| null \| undefined` | no | Runtime-updatable settings for this service. | | `startAudioPaused` | `boolean \| undefined` | no | Whether to start with audio input paused. Defaults to false. | | `startVideoPaused` | `boolean \| undefined` | no | Whether to start with video input paused. Defaults to false. | | `videoFrameDetail` | `events.ImageDetail \| undefined` | no | Detail level for video processing. Can be "auto", "low", or "high". This sets the image_detail parameter in the OpenAI Realtime API. "auto" lets the model decide, "low" is faster and uses fewer tokens, "high" provides more detail. Defaults to "auto". | | `userAudioPrerollSecs` | `number \| null \| undefined` | no | In manual turn-detection mode (`turn_detection=False`, locally-driven turns), how much recent audio to replay after an interruption clears the input audio buffer, so the speech onset isn't lost. Defaults to null: auto-sized to the upstream VAD's `startSecs` plus a small margin, falling back to `DEFAULT_USER_AUDIO_PREROLL_SECS` when no VAD is present. Auto-sizing assumes VAD drives turn starts (the default `VADUserTurnStartStrategy`); set this explicitly if you use a non-VAD turn-start strategy. No effect when server-side turn detection is enabled. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `runInParallel` | `boolean \| undefined` | no | Whether to run function calls in parallel or sequentially. Defaults to true. | | `groupParallelTools` | `boolean \| undefined` | no | Whether to group parallel function calls so the LLM is triggered exactly once after all calls in the batch complete. When false, each function call result triggers the LLM independently as it arrives. Defaults to true. | | `functionCallTimeoutSecs` | `number \| null \| undefined` | no | Optional timeout in seconds for function calls. A call that runs past it is cancelled: its handler is cancelled (it gets a `CancelledError`), the call is settled as cancelled, and inference runs so the LLM can report that it didn't complete. | | `fileResolver` | `FileResolver \| null \| undefined` | no | Resolves file URLs in the context that this provider can't fetch itself (see `FileResolver`). Pass one configured with a `fileStorage` to resolve uploaded-file URLs (`new FileResolver({ fileStorage: runnerArgs.fileStorage })` when a runner hosts the bot), and share one instance across services that may consume the same files (e.g. services switched between mid-session) so downloads are reused. Defaults to a resolver that handles `data:` and `http(s)` URLs only. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new OpenAIRealtimeLLMService.Settings({...})` (`OpenAIRealtimeLLMSettings`). Any setting you leave out keeps the default below. | Setting | Type | Default | Description | | ----------------------------------------------------------------------------------------- | ---------------------------------- | --------------------------- | ----------------------------------------------------------------------------------------------------------------------- | | `model` | `string` | `"gpt-realtime-2.1"` | Realtime model. Fixed for the session: it goes in the connection URL. | | `systemInstruction` | `string \| null` | `null` | Instructions. Kept in sync with `sessionProperties.instructions`. | | `sessionProperties` | `SessionProperties` | `new SessionProperties({})` | Session config: output modalities, audio input and output, tools, tool choice, max output tokens, reasoning, and so on. | | `temperature`, `maxTokens`, `topP`, `topK`, `frequencyPenalty`, `presencePenalty`, `seed` | `number \| null` | `null` | Not used by the Realtime API. | | `filterIncompleteUserTurns` | `boolean` | `false` | Filter incomplete user turns. | | `userTurnCompletionConfig` | `UserTurnCompletionConfig \| null` | `null` | Turn completion config used with `filterIncompleteUserTurns`. | A `sessionProperties` in an update replaces the stored one as a whole. Top-level `model` and `systemInstruction` win over the same fields inside it. ### Usage [#usage] ```ts title="realtime.ts" import { LLMUpdateSettingsFrame } from "@pipedog/core/frames/frames"; import { AudioConfiguration, AudioInput, AudioOutput, type ConversationItem, InputAudioTranscription, SemanticTurnDetection, SessionProperties, } from "@pipedog/openai/realtime/events"; import { OpenAIRealtimeLLMService } from "@pipedog/openai/realtime/llm"; const llm = new OpenAIRealtimeLLMService({ apiKey: process.env.OPENAI_API_KEY ?? "", settings: new OpenAIRealtimeLLMService.Settings({ model: "gpt-realtime-2.1", systemInstruction: "You are a helpful assistant in a voice conversation.", sessionProperties: new SessionProperties({ audio: new AudioConfiguration({ input: new AudioInput({ transcription: new InputAudioTranscription(), turnDetection: new SemanticTurnDetection({ eagerness: "medium" }), }), output: new AudioOutput({ voice: "marin" }), }), }), }), }); llm.addEventHandler( "on_conversation_item_created", async (_service, itemId: string, _item: ConversationItem) => { console.log("conversation item created:", itemId); }, ); ``` Change the instructions mid-call with an `LLMUpdateSettingsFrame`. A change to `systemInstruction` or `sessionProperties` is sent as a `session.update`; other fields are ignored with a warning: ```ts // Sends a session.update to the open session. const update = new LLMUpdateSettingsFrame({ delta: new OpenAIRealtimeLLMService.Settings({ systemInstruction: "Answer in Spanish from now on.", }), }); ``` ### Event handlers [#event-handlers] | Event | Description | | ------------------------------ | ----------------------------------------------------------------------------------------------------------------- | | `on_conversation_item_created` | A conversation item was added. Called with the item id and the item. | | `on_conversation_item_updated` | A conversation item changed (a transcript arrived, for example). Called with the item id and the item, or `null`. | ### Notes [#notes] * Consumes `InputAudioRawFrame` (resampled to 24 kHz), `InputImageRawFrame` (at most one per second, at the `videoFrameDetail` level), `LLMContextFrame`, `LLMMessagesAppendFrame` and `LLMSetToolsFrame`. Emits `TTSAudioRawFrame`, `TTSTextFrame` and `LLMTextFrame` for the bot, and `InterimTranscriptionFrame` / `TranscriptionFrame` for the user. * With OpenAI's server-side turn detection (the session default), the service broadcasts proposed user turn frames from the server's speech events and asks the user aggregator for `ExternalUserTurnStrategies`, so no local VAD is needed. `LLMContextAggregatorPair` detects that this is a realtime service. * For locally driven turns, set `turnDetection: false` in `AudioInput` and add a VAD to the user aggregator. After an interruption the input buffer is cleared and the last `userAudioPrerollSecs` of audio is replayed, so the start of the user's speech isn't lost. Don't run a local VAD with server-side turn detection on: both would send turn frames. * `setAudioInputPaused()`, `setVideoInputPaused()` and `setVideoFrameDetail()` change input at runtime. `sendClientEvent()` sends any Realtime client event, and `retrieveConversationItem()` fetches an item by id. * `reasoning` in `sessionProperties` is only sent to models recognized as reasoning-capable (`gpt-realtime-2`, `gpt-realtime-2.1`); otherwise it is dropped with a warning, since the server would reject the whole update. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/openai/package.json`: ✅ means the compat smoke test passes on that runtime. * The socket is opened with `websocketConnect()` from core and the key is sent as an `Authorization` header. Bun, Node and Deno send it from the global `WebSocket`; on Cloudflare Workers core opens the socket through `fetch`. A subclass that authenticates another way overrides `_authHeaders()`, as [Azure Realtime](https://polvallverdu.github.io/pipedog/docs/services/s2s/azure-realtime.md) does. * Video frames are encoded with core's `encodeImage()` (upstream uses Pillow). * The deprecated `model` and `sessionProperties` constructor options are not ported. Use `settings: new OpenAIRealtimeLLMService.Settings({ model, sessionProperties })`. * For realtime models deployed on Azure, see [`AzureRealtimeLLMService`](https://polvallverdu.github.io/pipedog/docs/services/s2s/azure-realtime.md). # Ultravox > Speech-to-speech with UltravoxRealtimeLLMService, Ultravox's audio-native realtime model. Source: https://polvallverdu.github.io/pipedog/docs/services/s2s/ultravox/ ## Overview [#overview] * **`UltravoxRealtimeLLMService`**: an Ultravox Realtime call. The service creates the call over the Ultravox REST API (or joins one by URL) and streams audio both ways over its WebSocket. Ultravox is an audio-native model: it hears the user directly and answers with audio or text. It replaces the STT, LLM and TTS services in a pipeline, and supports function calling, including async tools. Pipecat's documentation for the same service. `packages/ultravox/src/llm.ts`. Creating calls, and the call parameters `extra` can carry. ## Installation [#installation] ```bash bun add @pipedog/ultravox ``` ```ts import { AgentInputParams, JoinUrlInputParams, OneShotInputParams, UltravoxRealtimeLLMService, } from "@pipedog/ultravox/llm"; ``` ## Prerequisites [#prerequisites] * An Ultravox account and an API key from the [Ultravox console](https://app.ultravox.ai). * Pass the key in the params (`apiKey`); it is sent as the `X-Api-Key` header when the call is created. The examples read it from `ULTRAVOX_API_KEY`. Joining by URL needs no key. ## UltravoxRealtimeLLMService [#ultravoxrealtimellmservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `params` | `AgentInputParams \| JoinUrlInputParams \| OneShotInputParams` | yes | Configuration parameters for the model. | | `settings` | `UltravoxRealtimeLLMSettings \| null \| undefined` | no | Ultravox Realtime LLM settings. If provided, the `settings` values take precedence over default values. | | `oneShotSelectedTools` | `(FunctionSchema \| DirectFunction)[] \| ToolsSchema \| null \| undefined` | no | Tools to use with this call: a `ToolsSchema` or a plain list of direct functions and/or `FunctionSchema` objects (handlers auto-register). May only be set with `OneShotInputParams`. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `runInParallel` | `boolean \| undefined` | no | Whether to run function calls in parallel or sequentially. Defaults to true. | | `groupParallelTools` | `boolean \| undefined` | no | Whether to group parallel function calls so the LLM is triggered exactly once after all calls in the batch complete. When false, each function call result triggers the LLM independently as it arrives. Defaults to true. | | `functionCallTimeoutSecs` | `number \| null \| undefined` | no | Optional timeout in seconds for function calls. A call that runs past it is cancelled: its handler is cancelled (it gets a `CancelledError`), the call is settled as cancelled, and inference runs so the LLM can report that it didn't complete. | | `fileResolver` | `FileResolver \| null \| undefined` | no | Resolves file URLs in the context that this provider can't fetch itself (see `FileResolver`). Pass one configured with a `fileStorage` to resolve uploaded-file URLs (`new FileResolver({ fileStorage: runnerArgs.fileStorage })` when a runner hosts the bot), and share one instance across services that may consume the same files (e.g. services switched between mid-session) so downloads are reused. Defaults to a resolver that handles `data:` and `http(s)` URLs only. | `params` picks how the call starts: * **`OneShotInputParams`**: a one-off call with its own `systemPrompt`, `temperature`, `model`, `voice`, `metadata`, `outputMedium`, `maxDuration` (seconds, 10 to 3600, default one hour) and `extra` call parameters. Tools come from `oneShotSelectedTools`. * **`AgentInputParams`**: a call with an agent from the Ultravox console (`agentId`), with `templateContext`, `metadata`, `outputMedium`, `maxDuration` and `extra`. * **`JoinUrlInputParams`**: join a call you created yourself, by its `joinUrl`. | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | Ultravox API key for authentication. | | `systemPrompt` | `string \| null \| undefined` | no | System prompt to guide the model's behavior. Defaults to null. | | `temperature` | `number \| undefined` | no | Sampling temperature for response generation (0 to 1). Defaults to 0. | | `model` | `string \| null \| undefined` | no | Model identifier to use, e.g. "ultravox-v0.7". Defaults to null, which lets Ultravox pick its current default model. | | `voice` | `string \| null \| undefined` | no | Voice identifier for speech generation. Defaults to null. | | `metadata` | `Record \| undefined` | no | Metadata to attach to the call. Defaults to `{}`. | | `outputMedium` | `"text" \| "voice" \| null \| undefined` | no | The initial output medium for the agent. Use "text" for text responses or "voice" for audio responses. Defaults to null (voice). | | `maxDuration` | `number \| undefined` | no | The maximum duration of the call, in seconds (10 s to 1 h). Defaults to one hour. | | `extra` | `Record \| undefined` | no | Extra parameters to include in the call creation request. Defaults to `{}`. See the Ultravox API documentation for valid arguments: https://docs.ultravox.ai/api-reference/calls/calls-post | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new UltravoxRealtimeLLMService.Settings({...})` (`UltravoxRealtimeLLMSettings`). | Setting | Type | Default | Description | | ------------------------------------------------ | ---------------- | ------- | ---------------------------------------------------------------------------- | | `outputMedium` | `string \| null` | `null` | `"voice"` or `"text"`. A change is sent to the call as `set_output_medium`. | | `model`, `systemInstruction`, `temperature`, ... | | `null` | Set in `params` instead; changing them at runtime is ignored with a warning. | ### Usage [#usage] ```ts title="realtime.ts" import { FunctionSchema } from "@pipedog/core/adapters/llm/schemas/function_schema"; import { LLMUpdateSettingsFrame } from "@pipedog/core/frames/frames"; import type { FunctionCallParams } from "@pipedog/core/services/llm_service"; import { AgentInputParams, JoinUrlInputParams, OneShotInputParams, UltravoxRealtimeLLMService, } from "@pipedog/ultravox/llm"; const getWeather = new FunctionSchema({ name: "get_weather", description: "Get the current weather for a city.", properties: { city: { type: "string", description: "The city name." } }, required: ["city"], }); // A one-off call: the system prompt, voice and tools are set here. const llm = new UltravoxRealtimeLLMService({ params: new OneShotInputParams({ apiKey: process.env.ULTRAVOX_API_KEY ?? "", systemPrompt: "You are a helpful assistant in a voice conversation.", temperature: 0.3, maxDuration: 600, // seconds }), oneShotSelectedTools: [getWeather], }); llm.registerFunction("get_weather", async (params: FunctionCallParams) => { await params.resultCallback({ city: params.arguments.city, temperature: 21 }); }); ``` With an agent, or joining a call: ```ts // A call with an agent configured in the Ultravox console. const agent = new UltravoxRealtimeLLMService({ params: new AgentInputParams({ apiKey: process.env.ULTRAVOX_API_KEY ?? "", agentId: "your-agent-id", templateContext: { customerName: "Ada" }, }), }); // Or join a call created elsewhere, by its join URL. const joined = new UltravoxRealtimeLLMService({ params: new JoinUrlInputParams({ joinUrl: "wss://your-join-url" }), }); ``` Switch the reply medium mid-call: ```ts // Switch the model's replies to text (sends set_output_medium). const update = new LLMUpdateSettingsFrame({ delta: new UltravoxRealtimeLLMService.Settings({ outputMedium: "text" }), }); ``` ### Notes [#notes] * Consumes `InputAudioRawFrame` (resampled to 48 kHz and sent as binary), `LLMContextFrame` (new user text goes in as `user_text_message`) and `InterruptionFrame`. Emits `TTSAudioRawFrame` at 48 kHz, `TTSTextFrame` and `LLMTextFrame` for the bot, and `TranscriptionFrame` for the user. Transcripts of the user are Ultravox's and may not match what the audio-native model understood. * When Ultravox clears its playback buffer (the user barged in), the service broadcasts an `InterruptionFrame`. * It does not emit `UserStartedSpeakingFrame` / `UserStoppedSpeakingFrame`; Ultravox decides turns on the server. To get local turn frames, add a VAD to the user aggregator. `LLMContextAggregatorPair` detects that this is a realtime service either way. * An async tool (`cancelOnInterruption: false`) gets a placeholder result as soon as Ultravox calls it; its real result, and any intermediate results, go to the model as user text. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/ultravox/package.json`: ✅ means the compat smoke test passes on that runtime. * Calls are created with `fetch` (upstream uses aiohttp) and the socket is opened with `websocketConnect()` from core, so the service runs on Bun, Node, Deno and Cloudflare Workers. * `maxDuration` is in seconds (upstream takes a `timedelta`); out-of-range values throw a `RangeError` from the params constructor. # AssemblyAI > Streaming speech-to-text with AssemblyAI Universal-Streaming (v3), plus the sync HTTP service. Source: https://polvallverdu.github.io/pipedog/docs/services/stt/assemblyai/ ## Overview [#overview] pipedog ports both of Pipecat's AssemblyAI speech-to-text services: * **`AssemblyAISTTService`**: Universal-Streaming v3 over a WebSocket, with turn detection, keyterms, diarization, language detection and hot-updatable settings. * **`AssemblyAISyncSTTService`**: AssemblyAI's sync HTTP API for segmented (VAD-bounded) audio, with optional pre-warming and conversation context. The same package has AssemblyAI's [text-to-speech](https://polvallverdu.github.io/pipedog/docs/services/tts/assemblyai.md) service. Pipecat's documentation for the same service. `packages/assemblyai/src`. ## Installation [#installation] ```bash bun add @pipedog/assemblyai ``` ## Prerequisites [#prerequisites] * An [AssemblyAI](https://www.assemblyai.com) account and API key, passed as `apiKey` (the examples read `ASSEMBLYAI_API_KEY`). ## AssemblyAISTTService [#assemblyaisttservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | AssemblyAI API key for authentication. | | `apiEndpointBaseUrl` | `string \| undefined` | no | WebSocket endpoint URL. Defaults to "wss://streaming.assemblyai.com/v3/ws". | | `encoding` | `string \| undefined` | no | Audio encoding format. Defaults to "pcm_s16le". | | `vadForceTurnEndpoint` | `boolean \| undefined` | no | Controls turn detection mode. When true (Pipecat mode, default), AssemblyAI is forced to return finals ASAP so Pipecat's turn detection decides when the user is done. When false (AssemblyAI turn detection mode, U3 Pro models only), AssemblyAI's model controls turn endings and the service proposes turn starts and stops. | | `shouldInterrupt` | `boolean \| undefined` | no | Whether to interrupt the bot when the user starts speaking in AssemblyAI turn detection mode (`vadForceTurnEndpoint: false`). Defaults to true. | | `speakerFormat` | `string \| null \| undefined` | no | Format string for speaker labels when diarization is enabled; use `{speaker}` and `{text}`. Example: "<{speaker}>{text}". | | `settings` | `AssemblyAISTTSettings \| null \| undefined` | no | Runtime-updatable settings. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `reconnectBackoffMinWait` | `number \| undefined` | no | Minimum time, in seconds, to wait between reconnection attempts. | | `reconnectBackoffMaxWait` | `number \| undefined` | no | Maximum time, in seconds, to wait between reconnection attempts. | | `wsCloseTimeout` | `number \| undefined` | no | Maximum time, in seconds, to wait for the peer to acknowledge the websocket closing handshake before dropping the connection. Applied to connections opened through `_websocketConnect()`. Increase it for peers that need longer to complete a graceful close. | | `wsSendTimeout` | `number \| null \| undefined` | no | Maximum time, in seconds, a single send may wait on a peer that has stopped reading before the connection is dropped and the send throws `ConnectionClosedError`. Applied to connections opened through `_websocketConnect()`. Pass `null` to wait indefinitely. Defaults to `WS_SEND_TIMEOUT`. | | `audioPassthrough` | `boolean \| undefined` | no | Whether to pass audio frames downstream after processing. Defaults to true. | | `sampleRate` | `number \| null \| undefined` | no | The sample rate for audio input. If null, will be determined from the start frame. | | `sttTtfbTimeout` | `number \| undefined` | no | Time in seconds to wait after VAD stop before reporting TTFB. This delay allows the final transcription to arrive. Defaults to 2.0. | | `ttfsP99Latency` | `number \| null \| undefined` | no | P99 latency from speech end to final transcript in seconds. Broadcast via STTMetadataFrame at pipeline start. | | `keepaliveTimeout` | `number \| null \| undefined` | no | Seconds of no audio before sending silence to keep the connection alive. null disables keepalive. | | `keepaliveInterval` | `number \| undefined` | no | Seconds between idle checks when keepalive is enabled. | | `reconnectOnError` | `boolean \| undefined` | no | Whether to automatically reconnect on websocket errors. Defaults to true. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new AssemblyAISTTService.Settings({...})`. Settings you leave out keep their defaults. | Setting | Type | Default | Description | | --------------------------------------------- | ------------------------------------ | --------------------- | ------------------------------------------------------------------ | | `model` | `string` | `"universal-3-6-pro"` | Speech model. | | `language` / `languageCode` / `languageCodes` | `Language` / `string` / `Language[]` | | Language selection. | | `formatTurns` | `boolean` | | Return formatted final turns. | | `endOfTurnConfidenceThreshold` | `number` | | End-of-turn confidence. | | `minTurnSilence` / `maxTurnSilence` | `number` | | Silence bounds, in ms. | | `keytermsPrompt` / `prompt` | `string[]` / `string` | | Vocabulary boosting and prompting. | | `speakerLabels` | `boolean` | | Speaker diarization. | | `vadThreshold` | `number` | | Server VAD threshold. | | `agentContext`, `previousContextNTurns` | | | Conversation context (`updateAgentContext(text)` updates it live). | More (`languageDetection`, `domain`, `voiceFocus`, `mode`, ...) are listed on `AssemblyAISTTSettings`. ### Usage [#usage] ```ts title="stt.ts" import { AssemblyAISTTService } from "@pipedog/assemblyai/stt"; import { STTUpdateSettingsFrame } from "@pipedog/core/frames/frames"; import { Language } from "@pipedog/core/transcriptions/language"; const stt = new AssemblyAISTTService({ apiKey: process.env.ASSEMBLYAI_API_KEY ?? "", settings: new AssemblyAISTTService.Settings({ model: "universal-3-6-pro", keytermsPrompt: ["pipedog", "Pipecat"], languageCodes: [Language.EN, Language.ES], }), }); ``` ### Notes [#notes] * Emits `InterimTranscriptionFrame` for partial turns and `TranscriptionFrame` for final turns; fires `on_end_of_turn`. * With `vadForceTurnEndpoint` (default), a VAD stop sends `ForceEndpoint`. Update settings mid-call; fields AssemblyAI supports are sent as `UpdateConfiguration` without reconnecting: ```ts // Hot-updatable settings apply without reconnecting. const update = new STTUpdateSettingsFrame({ delta: new AssemblyAISTTService.Settings({ languageCodes: [Language.ES] }), }); ``` ## AssemblyAISyncSTTService [#assemblyaisyncsttservice] ### Configuration [#configuration-1] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | AssemblyAI API key for authentication. | | `fetch` | `((input: RequestInfo \| URL, init?: RequestInit) => Promise) \| null \| undefined` | no | `fetch` used for HTTP requests. Defaults to the global `fetch`. (Upstream takes an aiohttp session here.) | | `baseUrl` | `string \| undefined` | no | Base URL for the Sync API. Override for a data-residency endpoint (e.g. `https://sync.us.assemblyai.com` or `https://sync.eu.assemblyai.com`). | | `enablePrewarming` | `boolean \| undefined` | no | Whether to open the connection when the user starts speaking so the transcription request avoids the connection handshake. Defaults to true. | | `maxContextTurns` | `number \| undefined` | no | Number of prior conversation turns (user transcripts and agent replies together) automatically carried as `conversation_context`. 0 disables it. Ignored when `conversationContext` is set explicitly. Defaults to 5. | | `maxContextChars` | `number \| undefined` | no | Character budget for the same buffer; oldest turns drop first. Defaults to 1500. | | `settings` | `AssemblyAISyncSTTSettings \| null \| undefined` | no | Runtime-updatable settings. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `audioPassthrough` | `boolean \| undefined` | no | Whether to pass audio frames downstream after processing. Defaults to true. | | `sampleRate` | `number \| null \| undefined` | no | The sample rate for audio input. If null, will be determined from the start frame. | | `sttTtfbTimeout` | `number \| undefined` | no | Time in seconds to wait after VAD stop before reporting TTFB. This delay allows the final transcription to arrive. Defaults to 2.0. | | `ttfsP99Latency` | `number \| null \| undefined` | no | P99 latency from speech end to final transcript in seconds. Broadcast via STTMetadataFrame at pipeline start. | | `keepaliveTimeout` | `number \| null \| undefined` | no | Seconds of no audio before sending silence to keep the connection alive. null disables keepalive. | | `keepaliveInterval` | `number \| undefined` | no | Seconds between idle checks when keepalive is enabled. | | `trailingSilenceSecs` | `number \| undefined` | no | Seconds of silence appended to each segment before it is transcribed, so the model hears the end of speech and can finish the last word. Set to 0 to disable. Defaults to 0.5. | ### Settings [#settings-1] Runtime-updatable settings, passed as `settings: new AssemblyAISyncSTTService.Settings({...})`. Settings you leave out keep their defaults. | Setting | Description | | -------------------------- | ---------------------------------- | | `prompt`, `keytermsPrompt` | Prompting and vocabulary boosting. | | `conversationContext` | Prior turns sent as context. | | `languageCodes` | Language selection. | | `timestamps` | Request word timestamps. | ### Notes [#notes-1] * Pass `fetch` to use your own HTTP client; `warm()` and `enablePrewarming` call `/v1/warm`. ## Runtimes [#runtimes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/assemblyai/package.json`: ✅ means the compat smoke test passes on that runtime. The service uses `fetch` and the global `WebSocket` (through core's `websocketConnect`, which sends the auth header on the handshake on Cloudflare Workers too). Pass no SDK; nothing is Node-only. The Sync service uses `fetch` only. # AWS SageMaker (bidirectional streaming) > SageMakerBidiClient, the HTTP/2 bidirectional streaming client for speech models hosted on Amazon SageMaker endpoints. Source: https://polvallverdu.github.io/pipedog/docs/services/stt/aws-sagemaker/ ## Overview [#overview] `SageMakerBidiClient` streams data to and from a SageMaker real-time endpoint with `InvokeEndpointWithBidirectionalStream`. Upstream, it is the transport under the SageMaker variants of the Deepgram and NVIDIA speech services. Those services are not ported yet, but the client is, so you can talk to a model on your own endpoint. * `startSession()` opens the stream, retrying throttling and transient failures with jittered backoff (not 401 or 403). * `sendAudioChunk()`, `sendText()`, `sendJson()` and `sendData()` send payload parts. * `receiveResponse()` returns the next response event, or `null` once the stream has closed. * `closeSession()` ends the input stream. * `classifySagemakerBidiError()` sorts an error into an `ErrorCategory`, so a service can tell bad credentials from a busy endpoint. `packages/aws/src/sagemaker/bidi_client.ts`. Invoking real-time endpoints, including bidirectional streams. ## Installation [#installation] ```bash bun add @pipedog/aws @aws-sdk/client-sagemaker-runtime-http2 ``` ```ts import { SageMakerBidiClient } from "@pipedog/aws/sagemaker/bidi_client"; ``` `@aws-sdk/client-sagemaker-runtime-http2` is an optional peer dependency, loaded on the first `startSession()`. It is needed only by this client. ## Prerequisites [#prerequisites] * A SageMaker endpoint serving a model that supports bidirectional streaming (for example Deepgram from the AWS Marketplace). * An IAM identity allowed `sagemaker:InvokeEndpointWithBidirectionalStream`. * Keys from `awsAccessKeyId`, `awsSecretAccessKey` and `awsSessionToken`, else the `AWS_*` environment variables, else a `credentialsProvider` function. ## SageMakerBidiClient [#sagemakerbidiclient] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `endpointName` | `string` | yes | Name of the SageMaker endpoint to connect to. | | `region` | `string` | yes | AWS region where the endpoint is deployed. | | `modelInvocationPath` | `string \| null \| undefined` | no | API path for the model invocation (e.g., "v1/listen"). | | `modelQueryString` | `string \| null \| undefined` | no | Query string parameters for the model (e.g., "model=nova-3"). | | `maxConnectAttempts` | `number \| undefined` | no | How many times `startSession` attempts the session when SageMaker throttles it or fails transiently. The AWS SDK does not retry bidirectional streams itself. | | `awsAccessKeyId` | `string \| null \| undefined` | no | AWS access key ID (pipedog addition; defaults to `AWS_ACCESS_KEY_ID`). | | `awsSecretAccessKey` | `string \| null \| undefined` | no | AWS secret access key (pipedog addition; defaults to `AWS_SECRET_ACCESS_KEY`). | | `awsSessionToken` | `string \| null \| undefined` | no | AWS session token (pipedog addition; defaults to `AWS_SESSION_TOKEN`). | | `client` | `SageMakerRuntimeHTTP2ClientLike \| null \| undefined` | no | SDK client to use (pipedog addition; defaults to `SdkSageMakerRuntimeHTTP2Client`). | | `credentialsProvider` | `AWSCredentialsProvider \| null \| undefined` | no | Keys to sign with when none are passed in or set in the environment. Called for every request or connection, so it can renew temporary credentials. | | `fetch` | `FetchLike \| null \| undefined` | no | `fetch` used for HTTP calls (defaults to the global `fetch`). | | `endpointUrl` | `string \| null \| undefined` | no | Override the service endpoint (e.g. a VPC endpoint or a local fake), `https://host[:port]`. | ### Usage [#usage] ```ts title="sagemaker.ts" import { SageMakerBidiClient } from "@pipedog/aws/sagemaker/bidi_client"; const client = new SageMakerBidiClient({ endpointName: "my-deepgram-endpoint", region: "us-east-2", modelInvocationPath: "v1/listen", modelQueryString: "model=nova-3&language=en", }); async function transcribe(audio: Uint8Array): Promise { await client.startSession(); await client.sendAudioChunk(audio); await client.sendJson({ type: "CloseStream" }); while (client.isActive) { const event = await client.receiveResponse(); if (event === null) break; const bytes = event.PayloadPart?.Bytes; if (bytes) console.log(new TextDecoder().decode(bytes)); } await client.closeSession(); } ``` ### Notes [#notes] * Requests go to `https://runtime.sagemaker..amazonaws.com:8443`, or `endpointUrl`. * Events use the JS SDK's shapes: requests are `{ PayloadPart: { Bytes, DataType } }`, and `receiveResponse()` returns the SDK's `ResponseStreamEvent`. * A failed start throws `SageMakerBidiSessionError`, with the AWS error ID, the HTTP status and the number of attempts. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/aws/package.json`: ✅ means the compat smoke test passes on that runtime. * The API needs a full-duplex HTTP/2 stream, which `fetch` cannot do and Cloudflare Workers do not offer. The default client is the AWS SDK on Node's `http2`: it runs on Node and Bun. Deno is untested, and Workers are not supported. * Importing the package never loads the SDK, so the rest of `@pipedog/aws` keeps working on Workers. * Pass `client` (any object with `invokeEndpointWithBidirectionalStream(input)`) to use another transport, or a fake in tests. * Not ported yet: the SageMaker services that use it (`DeepgramSageMakerSTTService`, `DeepgramSageMakerTTSService`, the Flux variants, `NvidiaSageMakerSTTService` and `NvidiaSageMakerTTSService`). # AWS Transcribe > Streaming speech-to-text from Amazon Transcribe with AWSTranscribeSTTService, over Transcribe's WebSocket API. Source: https://polvallverdu.github.io/pipedog/docs/services/stt/aws/ ## Overview [#overview] `AWSTranscribeSTTService` streams the user's audio to Amazon Transcribe and emits interim and final transcripts. It connects to Transcribe's WebSocket API with a SigV4-presigned URL and sends audio as event-stream `AudioEvent` messages, as upstream does. Pipecat's documentation for the same service. `packages/aws/src/stt.ts` and `utils.ts`. Streaming over WebSocket, languages and partial-result stabilization. ## Installation [#installation] ```bash bun add @pipedog/aws ``` ```ts import { AWSTranscribeSTTService } from "@pipedog/aws/stt"; ``` ## Prerequisites [#prerequisites] * An IAM identity allowed `transcribe:StartStreamTranscriptionWebSocket`. * Keys, in this order of precedence: 1. `awsAccessKeyId`, `apiKey` (the secret access key) and `awsSessionToken`. 2. The `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` and `AWS_SESSION_TOKEN` environment variables. 3. A `credentialsProvider` function, called for every connection, so each URL is signed with current keys. The region comes from `region`, else `AWS_REGION`, else `us-east-1`. ## AWSTranscribeSTTService [#awstranscribesttservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string \| null \| undefined` | no | AWS secret access key. If null, falls back to environment variables and `credentialsProvider`. | | `awsAccessKeyId` | `string \| null \| undefined` | no | AWS access key ID. Same fallback behaviour as `apiKey`. | | `awsSessionToken` | `string \| null \| undefined` | no | AWS session token for temporary credentials. | | `region` | `string \| null \| undefined` | no | AWS region for the service. | | `settings` | `AWSTranscribeSTTSettings \| null \| undefined` | no | Runtime-updatable settings. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `reconnectBackoffMinWait` | `number \| undefined` | no | Minimum time, in seconds, to wait between reconnection attempts. | | `reconnectBackoffMaxWait` | `number \| undefined` | no | Maximum time, in seconds, to wait between reconnection attempts. | | `wsCloseTimeout` | `number \| undefined` | no | Maximum time, in seconds, to wait for the peer to acknowledge the websocket closing handshake before dropping the connection. Applied to connections opened through `_websocketConnect()`. Increase it for peers that need longer to complete a graceful close. | | `wsSendTimeout` | `number \| null \| undefined` | no | Maximum time, in seconds, a single send may wait on a peer that has stopped reading before the connection is dropped and the send throws `ConnectionClosedError`. Applied to connections opened through `_websocketConnect()`. Pass `null` to wait indefinitely. Defaults to `WS_SEND_TIMEOUT`. | | `audioPassthrough` | `boolean \| undefined` | no | Whether to pass audio frames downstream after processing. Defaults to true. | | `sampleRate` | `number \| null \| undefined` | no | The sample rate for audio input. If null, will be determined from the start frame. | | `sttTtfbTimeout` | `number \| undefined` | no | Time in seconds to wait after VAD stop before reporting TTFB. This delay allows the final transcription to arrive. Defaults to 2.0. | | `ttfsP99Latency` | `number \| null \| undefined` | no | P99 latency from speech end to final transcript in seconds. Broadcast via STTMetadataFrame at pipeline start. | | `keepaliveTimeout` | `number \| null \| undefined` | no | Seconds of no audio before sending silence to keep the connection alive. null disables keepalive. | | `keepaliveInterval` | `number \| undefined` | no | Seconds between idle checks when keepalive is enabled. | | `reconnectOnError` | `boolean \| undefined` | no | Whether to automatically reconnect on websocket errors. Defaults to true. | | `credentialsProvider` | `AWSCredentialsProvider \| null \| undefined` | no | Keys to sign with when none are passed in or set in the environment. Called for every request or connection, so it can renew temporary credentials. | | `fetch` | `FetchLike \| null \| undefined` | no | `fetch` used for HTTP calls (defaults to the global `fetch`). | | `endpointUrl` | `string \| null \| undefined` | no | Override the service endpoint (e.g. a VPC endpoint or a local fake), `https://host[:port]`. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new AWSTranscribeSTTService.Settings({...})`. Transcribe binds them to the connection, so changing one reconnects. | Setting | Type | Default | Description | | ------------------------- | ----------------------------- | ------------- | -------------------------------------------------------------------------- | | `language` | `Language` | `Language.EN` | Transcription language, mapped to a Transcribe code (`en-US`, `es-ES`, …). | | `partialResultsStability` | `"high" \| "medium" \| "low"` | `"high"` | Stability of interim results: `"high"` is fastest, `"low"` most accurate. | | `model` | `string \| null` | `null` | Not used by Transcribe. | ### Usage [#usage] ```ts title="stt.ts" import { AWSTranscribeSTTService } from "@pipedog/aws/stt"; import { STTUpdateSettingsFrame } from "@pipedog/core/frames/frames"; import { Language } from "@pipedog/core/transcriptions/language"; const stt = new AWSTranscribeSTTService({ // Or leave these out to read AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, // AWS_SESSION_TOKEN and AWS_REGION from the environment. awsAccessKeyId: process.env.AWS_ACCESS_KEY_ID, apiKey: process.env.AWS_SECRET_ACCESS_KEY, region: "us-east-1", sampleRate: 16000, settings: new AWSTranscribeSTTService.Settings({ language: Language.EN_US, partialResultsStability: "medium", }), }); ``` Change the language mid-call with an `STTUpdateSettingsFrame`: ```ts // Queue it on the worker, or push it from a processor upstream of the service. // Transcribe binds the language to the connection, so this reconnects. const update = new STTUpdateSettingsFrame({ delta: new AWSTranscribeSTTService.Settings({ language: Language.ES_ES }), }); ``` ### Event handlers [#event-handlers] | Event | Description | | --------------------- | ---------------------------------------------- | | `on_connected` | The WebSocket to Transcribe is open. | | `on_disconnected` | The WebSocket to Transcribe closed. | | `on_connection_error` | Connecting failed. Receives the error message. | ### Notes [#notes] * Emits `InterimTranscriptionFrame`s for partial results and a `TranscriptionFrame` for each final result. * Transcribe accepts 8000 or 16000 Hz PCM. Another `sampleRate` (or pipeline rate) is replaced by 16000 Hz on connect, with a warning. * Partial-result stabilization is always on; `partialResultsStability` sets its level. * On disconnect the service sends upstream's end-of-stream message before closing the socket. * A 401 or 403 from Transcribe is reported as a connectivity error, as upstream does. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/aws/package.json`: ✅ means the compat smoke test passes on that runtime. * Runs on Bun, Node, Deno and Cloudflare Workers. The URL is presigned with SigV4 on WebCrypto, and the socket is the global `WebSocket` (opened through `fetch` on Workers). * Only the `Origin` handshake header is sent; the runtime's `WebSocket` sets the others. * There is no default AWS credential chain (profiles, IMDS, ECS roles, SSO). Use keys, the environment, or `credentialsProvider`. * `endpointUrl` replaces `wss://transcribestreaming..amazonaws.com:8443`, for a proxy or a local fake. The URL is still signed for the AWS host. * Not ported: the deprecated `language` constructor option. Use `settings.language`. # Azure Speech > Continuous speech recognition from Azure AI Speech with AzureSTTService, over the Speech service's WebSocket protocol. Source: https://polvallverdu.github.io/pipedog/docs/services/stt/azure/ ## Overview [#overview] **`AzureSTTService`** streams the user's audio to Azure AI Speech and turns the results into transcription frames: an `InterimTranscriptionFrame` for each hypothesis while the user speaks, and a `TranscriptionFrame` (with `finalized: true`) for each recognized phrase. Upstream drives Azure through the native Speech SDK. pipedog speaks the Speech service's own WebSocket protocol (the one the SDKs use) on the continuous "conversation" endpoint, so no SDK is needed and the service runs on every runtime. Pipecat's documentation for the same service. `packages/azure/src/stt.ts` and `speech_protocol.ts`. Recognition languages, segmentation and custom models. ## Installation [#installation] ```bash bun add @pipedog/azure ``` ```ts import { AzureSTTService } from "@pipedog/azure/stt"; ``` ## Prerequisites [#prerequisites] * An Azure AI Speech (or Azure AI Foundry) resource, with its key and region from the portal's **Keys and Endpoint** page. * Pass the key as `apiKey` and the region (e.g. `"eastus"`) as `region`. The examples read them from `AZURE_SPEECH_API_KEY` and `AZURE_SPEECH_REGION`. ## Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | Azure Cognitive Services subscription key. | | `region` | `string \| null \| undefined` | no | Azure region for the Speech service (e.g., 'eastus'). Required unless `privateEndpoint` is provided. | | `privateEndpoint` | `string \| null \| undefined` | no | Private endpoint for STT behind firewall. See https://learn.microsoft.com/en-us/azure/ai-services/speech-service/speech-services-private-link?tabs=portal | | `endpointId` | `string \| null \| undefined` | no | Custom model endpoint id. | | `settings` | `AzureSTTSettings \| null \| undefined` | no | Runtime-updatable settings. | | `webSocketFactory` | `((uri: string, options: { headers?: Record; protocols?: string[]; }) => WebSocket \| Promise) \| null \| undefined` | no | Creates the underlying `WebSocket`, to open it some other way. Defaults to `websocketConnect`'s. Not in upstream. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `audioPassthrough` | `boolean \| undefined` | no | Whether to pass audio frames downstream after processing. Defaults to true. | | `sampleRate` | `number \| null \| undefined` | no | The sample rate for audio input. If null, will be determined from the start frame. | | `sttTtfbTimeout` | `number \| undefined` | no | Time in seconds to wait after VAD stop before reporting TTFB. This delay allows the final transcription to arrive. Defaults to 2.0. | | `ttfsP99Latency` | `number \| null \| undefined` | no | P99 latency from speech end to final transcript in seconds. Broadcast via STTMetadataFrame at pipeline start. | | `keepaliveTimeout` | `number \| null \| undefined` | no | Seconds of no audio before sending silence to keep the connection alive. null disables keepalive. | | `keepaliveInterval` | `number \| undefined` | no | Seconds between idle checks when keepalive is enabled. | ## Settings [#settings] Runtime-updatable settings, passed as `settings: new AzureSTTService.Settings({...})`. Any setting you leave out keeps the default below. | Setting | Type | Default | Description | | ------------------------------ | ---------------------------------------- | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | | `language` | `Language \| string` | `Language.EN_US` | Recognition language, sent as an Azure locale (`"en-US"`). | | `profanity` | `"raw" \| "masked" \| "removed" \| null` | `null` (Azure: `"masked"`) | How profanity appears in transcripts. `"raw"` helps non-English deployments where Azure masks ordinary words. An unknown value throws. | | `segmentationSilenceTimeoutMs` | `number \| null` | `null` (Azure: 500) | Silence, in ms (100 to 5000), inside a phrase before Azure finalizes it. Lower finalizes sooner; higher keeps hesitant speech in one transcript. | ## Usage [#usage] ```ts title="stt.ts" import { AzureSTTService } from "@pipedog/azure/stt"; import { STTUpdateSettingsFrame } from "@pipedog/core/frames/frames"; import { Language } from "@pipedog/core/transcriptions/language"; const stt = new AzureSTTService({ apiKey: process.env.AZURE_SPEECH_API_KEY ?? "", region: process.env.AZURE_SPEECH_REGION ?? "eastus", settings: new AzureSTTService.Settings({ language: Language.EN_US, profanity: "raw", segmentationSilenceTimeoutMs: 800, }), }); ``` Change settings mid-call with an `STTUpdateSettingsFrame`. A change to `language`, `profanity` or `segmentationSilenceTimeoutMs` reconnects with the new settings: ```ts // Queue it on the worker, or push it from a processor upstream of the service. // The service reconnects with the new language. const update = new STTUpdateSettingsFrame({ delta: new AzureSTTService.Settings({ language: Language.ES_ES }), }); ``` A Private Link endpoint or custom domain replaces the region, and `endpointId` selects a Custom Speech model: ```ts const privateStt = new AzureSTTService({ apiKey: process.env.AZURE_SPEECH_API_KEY ?? "", privateEndpoint: "wss://my-speech.cognitiveservices.azure.com/", endpointId: process.env.AZURE_CUSTOM_SPEECH_ENDPOINT_ID ?? null, }); ``` ## Notes [#notes] * The connection opens when the pipeline starts, at the pipeline's input sample rate (16-bit mono PCM). If Azure closes it (its session limit, an error), the next audio reopens it, and an error close is reported as an `ErrorFrame`. * `TranscriptionFrame`s carry `finalized: true`: Azure's phrase result is the final one for the utterance, so turn-stop strategies can act on it without waiting for VAD. * Results are requested in Azure's `simple` format; the raw result is on the frame's `result`. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/azure/package.json`: ✅ means the compat smoke test passes on that runtime. `AzureSTTService` opens its WebSocket with core's `websocketConnect` and works on Bun, Node, Deno and Cloudflare Workers. The key is sent both as the `Ocp-Apim-Subscription-Key` handshake header and as a query parameter, as the Speech SDK does, so it also authenticates where a `WebSocket` can't send headers. Pass `webSocketFactory` to open the socket some other way. Differences from upstream: there is no Speech SDK `SpeechConfig`; `_speechConfig` is a small record of the language, profanity option and properties, and the connection opens in `start()` rather than `setup()`. The deprecated `language` constructor option isn't ported; use `settings.language`. # Cartesia > Streaming speech-to-text with CartesiaSTTService (Ink) and CartesiaTurnsSTTService (Ink-2, with server-side turn detection and eager end of turn). Source: https://polvallverdu.github.io/pipedog/docs/services/stt/cartesia/ ## Overview [#overview] Cartesia streams transcripts over a WebSocket. pipedog ports both of Pipecat's Cartesia speech-to-text services: * **`CartesiaSTTService`**: Cartesia's live STT API (`/stt/websocket`), with interim and final transcripts, runtime model and language switching, and keyterms on ink-2 models. * **`CartesiaTurnsSTTService`**: Cartesia's turn-based Streaming ASR v2 API (`/stt/turns/websocket`, Ink-2). The server decides when the user's turn starts and ends, with optional eager end-of-turn predictions. Pipecat's documentation for the same services. `packages/cartesia/src/stt.ts` and `turns/stt.ts`. STT models, keyterms and turn detection. ## Installation [#installation] ```bash bun add @pipedog/cartesia ``` ```ts import { CartesiaSTTService } from "@pipedog/cartesia/stt"; import { CartesiaTurnsSTTService } from "@pipedog/cartesia/turns/stt"; ``` ## Prerequisites [#prerequisites] * A Cartesia account and an API key from the [Cartesia playground](https://play.cartesia.ai). * Pass the key as `apiKey`. The examples read it from `CARTESIA_API_KEY`. ## CartesiaSTTService [#cartesiasttservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | Authentication key for Cartesia API. | | `baseUrl` | `string \| undefined` | no | Custom API endpoint, either a bare host ("api.cartesia.ai") or a URL with a scheme ("ws://localhost:8000"). If empty, uses default. | | `encoding` | `string \| undefined` | no | Audio encoding format. Defaults to "pcm_s16le". | | `extraHeaders` | `Record \| null \| undefined` | no | Additional headers to send on the websocket handshake. Merged after the default headers, so it can override them. | | `settings` | `CartesiaSTTSettings \| null \| undefined` | no | Runtime-updatable settings. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `reconnectBackoffMinWait` | `number \| undefined` | no | Minimum time, in seconds, to wait between reconnection attempts. | | `reconnectBackoffMaxWait` | `number \| undefined` | no | Maximum time, in seconds, to wait between reconnection attempts. | | `wsCloseTimeout` | `number \| undefined` | no | Maximum time, in seconds, to wait for the peer to acknowledge the websocket closing handshake before dropping the connection. Applied to connections opened through `_websocketConnect()`. Increase it for peers that need longer to complete a graceful close. | | `wsSendTimeout` | `number \| null \| undefined` | no | Maximum time, in seconds, a single send may wait on a peer that has stopped reading before the connection is dropped and the send throws `ConnectionClosedError`. Applied to connections opened through `_websocketConnect()`. Pass `null` to wait indefinitely. Defaults to `WS_SEND_TIMEOUT`. | | `audioPassthrough` | `boolean \| undefined` | no | Whether to pass audio frames downstream after processing. Defaults to true. | | `sampleRate` | `number \| null \| undefined` | no | The sample rate for audio input. If null, will be determined from the start frame. | | `sttTtfbTimeout` | `number \| undefined` | no | Time in seconds to wait after VAD stop before reporting TTFB. This delay allows the final transcription to arrive. Defaults to 2.0. | | `ttfsP99Latency` | `number \| null \| undefined` | no | P99 latency from speech end to final transcript in seconds. Broadcast via STTMetadataFrame at pipeline start. | | `keepaliveTimeout` | `number \| null \| undefined` | no | Seconds of no audio before sending silence to keep the connection alive. null disables keepalive. | | `keepaliveInterval` | `number \| undefined` | no | Seconds between idle checks when keepalive is enabled. | | `reconnectOnError` | `boolean \| undefined` | no | Whether to automatically reconnect on websocket errors. Defaults to true. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new CartesiaSTTService.Settings({...})`. Any setting you leave out keeps the default below. | Setting | Type | Default | Description | | ---------- | ------------------ | --------------- | ------------------------------------------------------------------------------------ | | `model` | `string` | `"ink-whisper"` | Cartesia STT model. | | `language` | `Language` | `Language.EN` | Transcription language. | | `keyterm` | `string[] \| null` | `null` | Terms to bias transcription towards. Only `ink-2` and `ink-preview` models use them. | ### Usage [#usage] ```ts title="stt.ts" import { CartesiaSTTService } from "@pipedog/cartesia/stt"; import { STTUpdateSettingsFrame } from "@pipedog/core/frames/frames"; import { Language } from "@pipedog/core/transcriptions/language"; const stt = new CartesiaSTTService({ apiKey: process.env.CARTESIA_API_KEY ?? "", settings: new CartesiaSTTService.Settings({ model: "ink-whisper", language: Language.EN, }), }); ``` Keyterms, on an ink-2 model: ```ts // Keyterms only work on ink-2 and ink-preview models. const withKeyterms = new CartesiaSTTService({ apiKey: process.env.CARTESIA_API_KEY ?? "", settings: new CartesiaSTTService.Settings({ model: "ink-2", keyterm: ["pipedog", "Pipecat"], }), }); ``` Change settings mid-call with an `STTUpdateSettingsFrame`. The service reconnects with the new settings: ```ts // Queue it on the worker, or push it from a processor upstream of the service. const update = new STTUpdateSettingsFrame({ delta: new CartesiaSTTService.Settings({ language: Language.ES }), }); ``` ### Event handlers [#event-handlers] Inherited from `STTService`: | Event | Description | | --------------------- | --------------------------- | | `on_connected` | Connected to Cartesia. | | `on_disconnected` | Disconnected from Cartesia. | | `on_connection_error` | The connection failed. | ### Notes [#notes] * Emits `InterimTranscriptionFrame` while the user speaks and `TranscriptionFrame` for final results. A `VADUserStoppedSpeakingFrame` makes it send `finalize`, which flushes a final result. * Cartesia closes a socket after 3 minutes without messages. The service sends silence as a keepalive (after 120 s idle, checked every 30 s). * A settings change reconnects. While the user is speaking, the reconnect waits until they stop. * Keyterms set for a model other than `ink-2` or `ink-preview` are ignored with a warning. Cartesia accepts at most 100 keyterms totaling 1200 characters; extra ones are dropped with a warning. * `baseUrl` takes a bare host (`"api.cartesia.ai"`) or a URL with a scheme (`"ws://localhost:8000"`). ## CartesiaTurnsSTTService [#cartesiaturnssttservice] ### Configuration [#configuration-1] | Option | Type | Required | Description | |---|---|---|---| | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `apiKey` | `string` | yes | Cartesia API key. | | `url` | `string \| undefined` | no | WebSocket URL for the Cartesia Streaming ASR v2 endpoint. | | `shouldInterrupt` | `boolean \| undefined` | no | Whether to interrupt the bot when the server signals the start of a new turn. Passed along to the user turn strategies this service recommends, which own the interruption; a user-supplied `userTurnStrategies` overrides the recommendation and this setting with it. | | `watchdogMinTimeout` | `number \| undefined` | no | Minimum idle timeout before sending silence to prevent dangling turns. The actual threshold is `max(chunkDuration * 2, watchdogMinTimeout)`. Defaults to 0.5. | | `extraHeaders` | `Record \| null \| undefined` | no | Optional additional HTTP headers to send with the WebSocket handshake. | | `settings` | `CartesiaTurnsSTTSettings \| null \| undefined` | no | Runtime-updatable settings. The ink-2 family does not support runtime model or language switching; attempts to update either field will be reported as unhandled. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `reconnectBackoffMinWait` | `number \| undefined` | no | Minimum time, in seconds, to wait between reconnection attempts. | | `reconnectBackoffMaxWait` | `number \| undefined` | no | Maximum time, in seconds, to wait between reconnection attempts. | | `wsCloseTimeout` | `number \| undefined` | no | Maximum time, in seconds, to wait for the peer to acknowledge the websocket closing handshake before dropping the connection. Applied to connections opened through `_websocketConnect()`. Increase it for peers that need longer to complete a graceful close. | | `wsSendTimeout` | `number \| null \| undefined` | no | Maximum time, in seconds, a single send may wait on a peer that has stopped reading before the connection is dropped and the send throws `ConnectionClosedError`. Applied to connections opened through `_websocketConnect()`. Pass `null` to wait indefinitely. Defaults to `WS_SEND_TIMEOUT`. | | `audioPassthrough` | `boolean \| undefined` | no | Whether to pass audio frames downstream after processing. Defaults to true. | | `sampleRate` | `number \| null \| undefined` | no | The sample rate for audio input. If null, will be determined from the start frame. | | `sttTtfbTimeout` | `number \| undefined` | no | Time in seconds to wait after VAD stop before reporting TTFB. This delay allows the final transcription to arrive. Defaults to 2.0. | | `ttfsP99Latency` | `number \| null \| undefined` | no | P99 latency from speech end to final transcript in seconds. Broadcast via STTMetadataFrame at pipeline start. | | `keepaliveTimeout` | `number \| null \| undefined` | no | Seconds of no audio before sending silence to keep the connection alive. null disables keepalive. | | `keepaliveInterval` | `number \| undefined` | no | Seconds between idle checks when keepalive is enabled. | | `reconnectOnError` | `boolean \| undefined` | no | Whether to automatically reconnect on websocket errors. Defaults to true. | | `enableEagerEndOfTurn` | `boolean \| undefined` | no | Whether to answer a predicted end of turn ahead of the committed one. Off by default: it spends an inference on every prediction, including the ones the service withdraws. | | `eagerMatchPolicy` | `EagerMatchPolicy \| null \| undefined` | no | Decides whether the committed transcript is close enough to the predicted one to keep the speculative response. Only consulted while eager end of turn is on. See EagerUserTurnStrategies. | ### Settings [#settings-1] | Setting | Type | Default | Description | | ----------------------- | ------------------ | ----------------------- | --------------------------------------------------------------------------- | | `model` | `string` | `"ink-2"` | Cartesia model. Cannot change at runtime. | | `language` | `Language \| null` | `null` | Not used: ink-2 is English-only. Emitted frames carry `Language.EN`. | | `keyterm` | `string[] \| null` | `null` | Terms to bias transcription towards. | | `turnStartThreshold` | `number \| null` | `null` (Cartesia: 0.8) | Likelihood above which the server emits `turn.start`. Range 0.5 to 0.9. | | `turnEagerEndThreshold` | `number \| null` | `null` (Cartesia: 0.6) | Likelihood below which the server emits `turn.eager_end`. Range 0.3 to 0.8. | | `turnEndThreshold` | `number \| null` | `null` (Cartesia: 0.3) | Likelihood below which the server emits `turn.end`. Range 0.05 to 0.5. | | `turnEndTimeoutMs` | `number \| null` | `null` (Cartesia: 5600) | Wait after the user stops speaking before `turn.end`. Range 640 to 11200. | Keep the thresholds in order: `turnStartThreshold` > `turnEagerEndThreshold` > `turnEndThreshold`. Changing `keyterm` or a `turn*` setting at runtime reconnects. A `model` or `language` update is reported as unhandled. ### Usage [#usage-1] ```ts title="turns.ts" import { CartesiaTurnsSTTService } from "@pipedog/cartesia/turns/stt"; const stt = new CartesiaTurnsSTTService({ apiKey: process.env.CARTESIA_API_KEY ?? "", settings: new CartesiaTurnsSTTService.Settings({ keyterm: ["pipedog"], turnEndTimeoutMs: 2000, }), }); stt.addEventHandler("on_turn_end", async (_service, transcript: string) => { console.log("end of turn:", transcript); }); ``` With eager end of turn: ```ts // Answer the server's predicted end of turn ahead of the committed one. const eager = new CartesiaTurnsSTTService({ apiKey: process.env.CARTESIA_API_KEY ?? "", enableEagerEndOfTurn: true, settings: new CartesiaTurnsSTTService.Settings({ turnStartThreshold: 0.8, turnEagerEndThreshold: 0.6, turnEndThreshold: 0.3, }), }); eager.addEventHandler("on_turn_eager_end", async (_service, transcript: string) => { console.log("predicted end of turn:", transcript); }); eager.addEventHandler("on_turn_resume", async () => { console.log("user kept speaking"); }); ``` ### Event handlers [#event-handlers-1] | Event | Description | | ------------------- | --------------------------------------------- | | `on_turn_start` | The server detected the start of a turn. | | `on_turn_update` | A transcript update for the turn in progress. | | `on_turn_eager_end` | The server predicted the end of the turn. | | `on_turn_resume` | The user kept speaking after an eager end. | | `on_turn_end` | The final transcript for the turn. | `on_connected`, `on_disconnected` and `on_connection_error` are inherited from `STTService`. ### Notes [#notes-1] * The server drives turns: `turn.start` broadcasts a `ProposedUserStartedSpeakingFrame`, `turn.update` pushes an `InterimTranscriptionFrame`, and `turn.end` pushes a `TranscriptionFrame` and broadcasts a `ProposedUserStoppedSpeakingFrame`. Transcripts are cumulative per turn. There is no `finalize` command. * The service recommends `ExternalUserTurnStrategies` to the user aggregator, or `EagerUserTurnStrategies` with `enableEagerEndOfTurn`, so no local VAD or smart-turn model is needed. Passing your own `userTurnStrategies` overrides this. See [Turn detection](https://polvallverdu.github.io/pipedog/docs/concepts/turn-detection.md). * With `enableEagerEndOfTurn: true`, `turn.eager_end` pushes an `EagerTranscriptionFrame` that starts a response early. `turn.resume` pushes a `UserTurnResponseCancelFrame`. The early response is dropped if the user resumes or the final transcript differs. It is off by default because every prediction costs an inference. * `shouldInterrupt` (default `true`) decides whether the start of a turn interrupts the bot. * If audio stops after a `turn.start`, a watchdog sends silence so the turn does not hang. The idle threshold is `max(chunkDuration * 2, watchdogMinTimeout)`, with `watchdogMinTimeout` defaulting to 0.5 s. * The service waits up to 10 s for Cartesia's `connected` message after the handshake. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/cartesia/package.json`: ✅ means the compat smoke test passes on that runtime. Both services use the global `WebSocket` and send the key as an `X-API-Key` header on the handshake. The Bun, Node and Deno `WebSocket` support handshake headers, and on Cloudflare Workers the socket is opened through `fetch` with an `Upgrade` header. A standard `WebSocket` does not expose the handshake response, so the `X-Request-ID` header that upstream logs on connect is always logged as `None`. The deprecated `CartesiaLiveOptions` class and the `liveOptions` constructor option are not ported; use `settings`. # Cloudflare Workers AI > Speech-to-text on Workers AI: Deepgram Flux with turn detection, Deepgram Nova-3 streaming, and Whisper, over the AI binding or the REST API. Source: https://polvallverdu.github.io/pipedog/docs/services/stt/cloudflare/ ## Overview [#overview] Workers AI hosts speech-to-text models on Cloudflare's network. `@pipedog/cloudflare` has a service for each kind: * **`WorkersAIFluxSTTService`**: Deepgram Flux (`@cf/deepgram/flux`), conversational STT that also detects the end of the user's turn. It replaces VAD and Smart Turn. * **`WorkersAINova3STTService`**: Deepgram Nova-3 (`@cf/deepgram/nova-3`) over a WebSocket, with interim results. Use it with a VAD, as with Deepgram. * **`WorkersAIWhisperSTTService`**: Whisper (`@cf/openai/whisper-large-v3-turbo`, `@cf/openai/whisper`, `@cf/openai/whisper-tiny-en`), one request per speech segment. This package is pipedog's own: Pipecat has no Workers AI services. The Flux and Nova-3 services are the [Deepgram](https://polvallverdu.github.io/pipedog/docs/services/stt/deepgram.md) services with a different way to open the socket, since Workers AI carries Deepgram's WebSocket protocol, so they behave and emit frames the same way. `packages/cloudflare/src/flux_stt.ts`, `stt.ts` and `whisper.ts`. Inputs, outputs and pricing of each model. Flux, a Workers AI LLM and Aura in a Durable Object. ## Installation [#installation] ```bash bun add @pipedog/cloudflare ``` ```ts import { WorkersAIFluxSTTService } from "@pipedog/cloudflare/flux_stt"; import { WorkersAINova3STTService } from "@pipedog/cloudflare/stt"; import { WorkersAIWhisperSTTService } from "@pipedog/cloudflare/whisper"; ``` ## Prerequisites [#prerequisites] Every Workers AI service reaches the models one of two ways: * **The `AI` binding**, inside a Worker or Durable Object. Add `"ai": { "binding": "AI" }` to `wrangler.jsonc` and pass `ai: env.AI`. No token is needed. The realtime models open their socket with `env.AI.run(model, input, { websocket: true })`. * **The REST API**, anywhere: pass `accountId` and `apiToken` (a [Cloudflare API token](https://dash.cloudflare.com/profile/api-tokens) with the Workers AI permission). Requests go to `https://api.cloudflare.com/client/v4/accounts/{accountId}/ai/run/{model}`, and the realtime models open `wss://` on the same URL, with the session parameters in the query string. The examples read them from `CLOUDFLARE_ACCOUNT_ID` and `CLOUDFLARE_API_TOKEN`. ```ts title="stt.ts" // Anywhere else: an account ID and an API token with Workers AI access. const fluxRest = new WorkersAIFluxSTTService({ accountId: process.env.CLOUDFLARE_ACCOUNT_ID ?? "", apiToken: process.env.CLOUDFLARE_API_TOKEN ?? "", }); ``` To share one connection setup between services, build a `WorkersAIClient` and pass it as `client`: ```ts import { WorkersAIClient } from "@pipedog/cloudflare/client"; // One client shared by several services. const client = new WorkersAIClient({ ai: env.AI }); const fluxShared = new WorkersAIFluxSTTService({ client }); ``` ## WorkersAIFluxSTTService [#workersaifluxsttservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `accountId` | `string \| null \| undefined` | no | Cloudflare account ID, for the REST API. | | `apiToken` | `string \| null \| undefined` | no | Cloudflare API token with Workers AI access, for the REST API. | | `baseUrl` | `string \| null \| undefined` | no | Base URL of Cloudflare's API. Defaults to `CLOUDFLARE_API_URL`. | | `fetch` | `{ (input: RequestInfo \| URL, init?: RequestInit): Promise; (input: string \| URL \| Request, init?: RequestInit): Promise; } \| null \| undefined` | no | `fetch` used for REST requests. Defaults to the global `fetch`. | | `settings` | `DeepgramFluxSTTSettings \| null \| undefined` | no | Runtime-updatable settings. `model` is the Workers AI model ID (default `"@cf/deepgram/flux"`). `languageHints`, `numerals`, `profanityFilter`, `redact` and `minConfidence` are Deepgram options that Workers AI does not document: they are sent as Deepgram sends them, and may be ignored. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `reconnectBackoffMinWait` | `number \| undefined` | no | Minimum time, in seconds, to wait between reconnection attempts. | | `reconnectBackoffMaxWait` | `number \| undefined` | no | Maximum time, in seconds, to wait between reconnection attempts. | | `reconnectOnError` | `boolean \| undefined` | no | Whether to automatically reconnect on connection errors. | | `wsCloseTimeout` | `number \| undefined` | no | Maximum time, in seconds, to wait for the peer to acknowledge the websocket closing handshake before dropping the connection. Applied to connections opened through `_websocketConnect()`. Increase it for peers that need longer to complete a graceful close. | | `wsSendTimeout` | `number \| null \| undefined` | no | Maximum time, in seconds, a single send may wait on a peer that has stopped reading before the connection is dropped and the send throws `ConnectionClosedError`. Applied to connections opened through `_websocketConnect()`. Pass `null` to wait indefinitely. Defaults to `WS_SEND_TIMEOUT`. | | `audioPassthrough` | `boolean \| undefined` | no | Whether to pass audio frames downstream after processing. Defaults to true. | | `sampleRate` | `number \| null \| undefined` | no | The sample rate for audio input. If null, will be determined from the start frame. | | `sttTtfbTimeout` | `number \| undefined` | no | Time in seconds to wait after VAD stop before reporting TTFB. This delay allows the final transcription to arrive. Defaults to 2.0. | | `ttfsP99Latency` | `number \| null \| undefined` | no | P99 latency from speech end to final transcript in seconds. Broadcast via STTMetadataFrame at pipeline start. | | `keepaliveTimeout` | `number \| null \| undefined` | no | Seconds of no audio before sending silence to keep the connection alive. null disables keepalive. | | `keepaliveInterval` | `number \| undefined` | no | Seconds between idle checks when keepalive is enabled. | | `enableEagerEndOfTurn` | `boolean \| undefined` | no | Whether to answer a predicted end of turn ahead of the committed one. Off by default: it spends an inference on every prediction, including the ones the service withdraws. | | `eagerMatchPolicy` | `EagerMatchPolicy \| null \| undefined` | no | Decides whether the committed transcript is close enough to the predicted one to keep the speculative response. Only consulted while eager end of turn is on. See EagerUserTurnStrategies. | | `mipOptOut` | `boolean \| null \| undefined` | no | Opt out of the Deepgram Model Improvement Program. | | `tag` | `string[] \| null \| undefined` | no | Tags to label requests for identification during usage reporting. | | `shouldInterrupt` | `boolean \| undefined` | no | Whether to interrupt the bot when Flux detects that the user is speaking. | | `watchdogMinTimeout` | `number \| undefined` | no | Minimum idle timeout before sending silence to prevent dangling turns. | | `fluxEncoding` | `string \| undefined` | no | Audio encoding format required by Flux API. Must be "linear16". Raw signed little-endian 16-bit PCM encoding. | | `ai` | `WorkersAIBinding \| null \| undefined` | no | The Workers AI binding (`env.AI`), inside a Worker. Takes precedence over the credentials. | | `webSocketFactory` | `((uri: string, options: { headers?: Record; protocols?: string[]; }) => WebSocket \| Promise) \| null \| undefined` | no | Creates the `WebSocket` for REST sessions, to open it some other way. Defaults to `websocketConnect`'s, which sends the token header on Bun, Node, Deno and Cloudflare Workers. | | `client` | `WorkersAIClient \| null \| undefined` | no | A client to share between services. When set, the other connection options are ignored. | ### Settings [#settings] `WorkersAIFluxSTTService.Settings` is Deepgram Flux's settings class; see [Deepgram Flux settings](https://polvallverdu.github.io/pipedog/docs/services/stt/deepgram/#deepgramfluxsttservice). The ones Workers AI documents: | Setting | Type | Default | Description | | ------------------- | ---------------- | --------------------- | ------------------------------------------------------------ | | `model` | `string` | `"@cf/deepgram/flux"` | The Workers AI model ID. | | `eotThreshold` | `number \| null` | `null` (Flux: 0.7) | End-of-turn confidence that ends a turn, 0.5 to 0.9. | | `eagerEotThreshold` | `number \| null` | `null` | Enables `EagerEndOfTurn` and `TurnResumed`, 0.3 to 0.9. | | `eotTimeoutMs` | `number \| null` | `null` (Flux: 5000) | Ends a turn this long after speech, whatever the confidence. | | `keyterm` | `string[]` | `[]` | Terms to boost. | ### Usage [#usage] ```ts title="stt.ts" import { WorkersAIFluxSTTService } from "@pipedog/cloudflare/flux_stt"; // Inside a Worker or Durable Object: the AI binding, no token. const flux = new WorkersAIFluxSTTService({ ai: env.AI, settings: new WorkersAIFluxSTTService.Settings({ eotThreshold: 0.75, keyterm: ["pipedog", "Cloudflare"], }), }); ``` Flux owns turn detection: use it with no VAD analyzer, and the user aggregator follows its turn events, as with `DeepgramFluxSTTService` (see [Turn detection](https://polvallverdu.github.io/pipedog/docs/concepts/turn-detection.md)). Change settings mid-call with an `STTUpdateSettingsFrame`: ```ts // Queue it on the worker, or push it from a processor upstream of the service. const update = new STTUpdateSettingsFrame({ delta: new WorkersAIFluxSTTService.Settings({ eotThreshold: 0.85 }), }); ``` ### Event handlers [#event-handlers] The same as `DeepgramFluxSTTService`: `on_start_of_turn`, `on_end_of_turn`, `on_eager_end_of_turn`, `on_turn_resumed`, `on_update`, plus `on_connected`, `on_disconnected` and `on_connection_error`. ### Notes [#notes] * The session parameters are the model's inputs (`encoding`, `sample_rate`, `eot_threshold`, `eager_eot_threshold`, `eot_timeout_ms`, `keyterm`, `mip_opt_out`, `tag`). The model is the URL path, not a `model` parameter. * Workers AI's turn messages carry an `event` and no `type`; they are read as Deepgram's `TurnInfo`. Workers AI doesn't document Deepgram's `Connected` message, so an open socket counts as confirmed. * Audio is 16-bit PCM (`linear16`) at the pipeline's input rate. ## WorkersAINova3STTService [#workersainova3sttservice] ### Configuration [#configuration-1] | Option | Type | Required | Description | |---|---|---|---| | `settings` | `DeepgramSTTSettings \| null \| undefined` | no | Runtime-updatable settings (Deepgram's). `model` is the Workers AI model ID, `"@cf/deepgram/nova-3"` by default. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `audioPassthrough` | `boolean \| undefined` | no | Whether to pass audio frames downstream after processing. Defaults to true. | | `sampleRate` | `number \| null \| undefined` | no | The sample rate for audio input. If null, will be determined from the start frame. | | `sttTtfbTimeout` | `number \| undefined` | no | Time in seconds to wait after VAD stop before reporting TTFB. This delay allows the final transcription to arrive. Defaults to 2.0. | | `ttfsP99Latency` | `number \| null \| undefined` | no | P99 latency from speech end to final transcript in seconds. Broadcast via STTMetadataFrame at pipeline start. | | `keepaliveTimeout` | `number \| null \| undefined` | no | Seconds of no audio before sending silence to keep the connection alive. null disables keepalive. | | `keepaliveInterval` | `number \| undefined` | no | Seconds between idle checks when keepalive is enabled. | | `ai` | `WorkersAIBinding \| null \| undefined` | no | The Workers AI binding (`env.AI`), inside a Worker. Takes precedence over the credentials. | | `accountId` | `string \| null \| undefined` | no | Cloudflare account ID, for the REST API. | | `apiToken` | `string \| null \| undefined` | no | Cloudflare API token with Workers AI access, for the REST API. | | `baseUrl` | `string \| null \| undefined` | no | Base URL of Cloudflare's API. Defaults to `CLOUDFLARE_API_URL`. | | `fetch` | `{ (input: RequestInfo \| URL, init?: RequestInit): Promise; (input: string \| URL \| Request, init?: RequestInit): Promise; } \| null \| undefined` | no | `fetch` used for REST requests. Defaults to the global `fetch`. | | `webSocketFactory` | `((uri: string, options: { headers?: Record; protocols?: string[]; }) => WebSocket \| Promise) \| null \| undefined` | no | Creates the `WebSocket` for REST sessions, to open it some other way. Defaults to `websocketConnect`'s, which sends the token header on Bun, Node, Deno and Cloudflare Workers. | | `client` | `WorkersAIClient \| null \| undefined` | no | A client to share between services. When set, the other connection options are ignored. | | `encoding` | `string \| undefined` | no | Audio encoding format. Defaults to "linear16". | | `channels` | `number \| undefined` | no | Number of audio channels. Defaults to 1. | | `multichannel` | `boolean \| undefined` | no | Transcribe each audio channel independently. Defaults to false. | | `callback` | `string \| null \| undefined` | no | Callback URL for async transcription delivery. | | `callbackMethod` | `string \| null \| undefined` | no | HTTP method for the callback ("GET" or "POST"). | | `tag` | `any` | no | Custom billing tag. | | `mipOptOut` | `boolean \| null \| undefined` | no | Opt out of Deepgram model improvement program. | | `addons` | `Record \| null \| undefined` | no | Additional Deepgram features to enable. | ### Settings [#settings-1] `WorkersAINova3STTService.Settings` is Deepgram's settings class (see [DeepgramSTTService settings](https://polvallverdu.github.io/pipedog/docs/services/stt/deepgram/#settings)), with `model` defaulting to `"@cf/deepgram/nova-3"`. ### Usage [#usage-1] ```ts title="stt.ts" import { WorkersAINova3STTService } from "@pipedog/cloudflare/stt"; const nova = new WorkersAINova3STTService({ ai: env.AI, settings: new WorkersAINova3STTService.Settings({ language: Language.EN, smartFormat: true }), }); ``` ### Notes [#notes-1] * Every Deepgram parameter but the model is sent as a session parameter. Workers AI documents `interim_results`, `endpointing`, `vad_events`, `utterance_end_ms`, `smart_format`, `punctuate`, `keyterm`, `language` and more on the model page. ## WorkersAIWhisperSTTService [#workersaiwhispersttservice] ### Configuration [#configuration-2] | Option | Type | Required | Description | |---|---|---|---| | `settings` | `WorkersAIWhisperSTTSettings \| null \| undefined` | no | Runtime-updatable settings. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `audioPassthrough` | `boolean \| undefined` | no | Whether to pass audio frames downstream after processing. Defaults to true. | | `sampleRate` | `number \| null \| undefined` | no | The sample rate for audio input. If null, will be determined from the start frame. | | `sttTtfbTimeout` | `number \| undefined` | no | Time in seconds to wait after VAD stop before reporting TTFB. This delay allows the final transcription to arrive. Defaults to 2.0. | | `ttfsP99Latency` | `number \| null \| undefined` | no | P99 latency from speech end to final transcript in seconds. Broadcast via STTMetadataFrame at pipeline start. | | `keepaliveTimeout` | `number \| null \| undefined` | no | Seconds of no audio before sending silence to keep the connection alive. null disables keepalive. | | `keepaliveInterval` | `number \| undefined` | no | Seconds between idle checks when keepalive is enabled. | | `trailingSilenceSecs` | `number \| undefined` | no | Seconds of silence appended to each segment before it is transcribed, so the model hears the end of speech and can finish the last word. Set to 0 to disable. Defaults to 0.5. | | `ai` | `WorkersAIBinding \| null \| undefined` | no | The Workers AI binding (`env.AI`), inside a Worker. Takes precedence over the credentials. | | `accountId` | `string \| null \| undefined` | no | Cloudflare account ID, for the REST API. | | `apiToken` | `string \| null \| undefined` | no | Cloudflare API token with Workers AI access, for the REST API. | | `baseUrl` | `string \| null \| undefined` | no | Base URL of Cloudflare's API. Defaults to `CLOUDFLARE_API_URL`. | | `fetch` | `{ (input: RequestInfo \| URL, init?: RequestInit): Promise; (input: string \| URL \| Request, init?: RequestInit): Promise; } \| null \| undefined` | no | `fetch` used for REST requests. Defaults to the global `fetch`. | | `webSocketFactory` | `((uri: string, options: { headers?: Record; protocols?: string[]; }) => WebSocket \| Promise) \| null \| undefined` | no | Creates the `WebSocket` for REST sessions, to open it some other way. Defaults to `websocketConnect`'s, which sends the token header on Bun, Node, Deno and Cloudflare Workers. | | `client` | `WorkersAIClient \| null \| undefined` | no | A client to share between services. When set, the other connection options are ignored. | ### Settings [#settings-2] | Setting | Type | Default | Description | | ----------- | ------------------ | ------------------------------------- | -------------------------------------------------------------------------------- | | `model` | `string` | `"@cf/openai/whisper-large-v3-turbo"` | `WorkersAIWhisperModel.LARGE_V3_TURBO`, `.WHISPER` or `.TINY_EN`. | | `language` | `Language \| null` | `null` | The audio's language, sent as its ISO 639-1 code; `null` detects it. Turbo only. | | `prompt` | `string \| null` | `null` | Context for the model (`initial_prompt`). Turbo only. | | `vadFilter` | `boolean \| null` | `null` | Run Whisper's own VAD first (`vad_filter`). Turbo only. | ### Usage [#usage-2] ```ts title="stt.ts" import { WorkersAIWhisperModel, WorkersAIWhisperSTTService } from "@pipedog/cloudflare/whisper"; const whisper = new WorkersAIWhisperSTTService({ ai: env.AI, settings: new WorkersAIWhisperSTTService.Settings({ model: WorkersAIWhisperModel.LARGE_V3_TURBO, language: Language.EN, prompt: "A conversation about Cloudflare Workers.", }), }); ``` ### Notes [#notes-2] * A segmented service: it needs a VAD analyzer. The audio between the VAD's start and stop is sent as a WAV file, base64 for turbo and an array of bytes for the original models. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/cloudflare/package.json`: ✅ means the compat smoke test passes on that runtime. * No dependencies beyond pipedog's own packages. HTTP goes through `fetch` and WebSockets through the core `websocketConnect` (REST) or the binding's socket (Workers). * Tested against a local fake of Workers AI (both paths), not against Workers AI itself. A live test (`packages/cloudflare/test/live_workers_ai.test.ts`) runs when `CLOUDFLARE_ACCOUNT_ID` and `CLOUDFLARE_API_TOKEN` are set. # Deepgram > Streaming speech-to-text with DeepgramSTTService (Nova) and DeepgramFluxSTTService (Flux, with model-side turn detection). Source: https://polvallverdu.github.io/pipedog/docs/services/stt/deepgram/ ## Overview [#overview] Deepgram streams transcripts over a WebSocket. pipedog ports both of Pipecat's Deepgram speech-to-text services: * **`DeepgramSTTService`**: Deepgram's `/v1/listen` API (Nova models), with interim results, smart formatting, keyterms and the other Deepgram query options. * **`DeepgramFluxSTTService`**: Deepgram Flux (`/v2/listen`), which also detects the end of the user's turn, with optional eager end-of-turn events. Pipecat's documentation for the same services. `packages/deepgram/src/stt.ts` and `flux/stt.ts`. Models, languages and query parameters. ## Installation [#installation] ```bash bun add @pipedog/deepgram ``` ```ts import { DeepgramSTTService } from "@pipedog/deepgram/stt"; import { DeepgramFluxSTTService } from "@pipedog/deepgram/flux/stt"; ``` ## Prerequisites [#prerequisites] * A Deepgram account and an API key from the [Deepgram console](https://console.deepgram.com). * Pass the key as `apiKey`. The examples read it from `DEEPGRAM_API_KEY`. ## DeepgramSTTService [#deepgramsttservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | Deepgram API key for authentication. | | `baseUrl` | `string \| undefined` | no | Custom Deepgram API base URL. | | `encoding` | `string \| undefined` | no | Audio encoding format. Defaults to "linear16". | | `channels` | `number \| undefined` | no | Number of audio channels. Defaults to 1. | | `multichannel` | `boolean \| undefined` | no | Transcribe each audio channel independently. Defaults to false. | | `callback` | `string \| null \| undefined` | no | Callback URL for async transcription delivery. | | `callbackMethod` | `string \| null \| undefined` | no | HTTP method for the callback ("GET" or "POST"). | | `tag` | `any` | no | Custom billing tag. | | `mipOptOut` | `boolean \| null \| undefined` | no | Opt out of Deepgram model improvement program. | | `addons` | `Record \| null \| undefined` | no | Additional Deepgram features to enable. | | `settings` | `DeepgramSTTSettings \| null \| undefined` | no | Runtime-updatable settings. | | `webSocketFactory` | `((uri: string, options: { headers?: Record; protocols?: string[]; }) => WebSocket \| Promise) \| null \| undefined` | no | Creates the underlying `WebSocket`, to open it some other way (through a proxy, say). Defaults to `websocketConnect`'s, which sends the header on every runtime, Cloudflare Workers included. Not in upstream. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `audioPassthrough` | `boolean \| undefined` | no | Whether to pass audio frames downstream after processing. Defaults to true. | | `sampleRate` | `number \| null \| undefined` | no | The sample rate for audio input. If null, will be determined from the start frame. | | `sttTtfbTimeout` | `number \| undefined` | no | Time in seconds to wait after VAD stop before reporting TTFB. This delay allows the final transcription to arrive. Defaults to 2.0. | | `ttfsP99Latency` | `number \| null \| undefined` | no | P99 latency from speech end to final transcript in seconds. Broadcast via STTMetadataFrame at pipeline start. | | `keepaliveTimeout` | `number \| null \| undefined` | no | Seconds of no audio before sending silence to keep the connection alive. null disables keepalive. | | `keepaliveInterval` | `number \| undefined` | no | Seconds between idle checks when keepalive is enabled. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new DeepgramSTTService.Settings({...})`. Any setting you leave out keeps the default below. Other Deepgram query parameters can go in `extra`. | Setting | Type | Default | Description | | ----------------- | ---------------------------- | ------------------ | ----------------------------------------------------- | | `model` | `string` | `"nova-3-general"` | Deepgram model. | | `language` | `Language` | `Language.EN` | Transcription language. | | `interimResults` | `boolean` | `true` | Emit `InterimTranscriptionFrame`s. | | `punctuate` | `boolean` | `true` | Add punctuation. | | `smartFormat` | `boolean` | `false` | Smart formatting. | | `endpointing` | `number \| false \| null` | `null` | Endpointing sensitivity in ms, or `false` to disable. | | `utteranceEndMs` | `number \| null` | `null` | Silence in ms before an utterance-end event. | | `keyterm` | `string \| string[] \| null` | `null` | Keyterms to boost. | | `keywords` | `string \| string[] \| null` | `null` | Keywords to boost. | | `diarize` | `boolean` | `false` | Speaker diarization. | | `detectEntities` | `boolean` | `false` | Named entity detection. | | `dictation` | `boolean` | `false` | Dictation mode. | | `numerals` | `boolean` | `false` | Spoken numbers as numerals. | | `profanityFilter` | `boolean \| null` | `null` | Filter profanity. | | `redact` | `string \| string[] \| null` | `null` | Redaction types. | | `replace` | `string \| string[] \| null` | `null` | Word replacement rules. | | `search` | `string \| string[] \| null` | `null` | Search terms to highlight. | | `version` | `string \| null` | `null` | Model version; `null` leaves Deepgram on `"latest"`. | ### Usage [#usage] ```ts title="stt.ts" import { STTUpdateSettingsFrame } from "@pipedog/core/frames/frames"; import { Language } from "@pipedog/core/transcriptions/language"; import { DeepgramSTTService } from "@pipedog/deepgram/stt"; const stt = new DeepgramSTTService({ apiKey: process.env.DEEPGRAM_API_KEY ?? "", settings: new DeepgramSTTService.Settings({ model: "nova-3-general", language: Language.EN, smartFormat: true, keyterm: ["pipedog", "Pipecat"], }), }); ``` Change settings mid-call with an `STTUpdateSettingsFrame`. The service reconnects with the new settings: ```ts // Queue it on the worker, or push it from a processor upstream of the service. const update = new STTUpdateSettingsFrame({ delta: new DeepgramSTTService.Settings({ language: Language.ES }), }); ``` ### Notes [#notes] * Emits `InterimTranscriptionFrame` while the user speaks and `TranscriptionFrame` for final results. A `VADUserStoppedSpeakingFrame` from the user aggregator's VAD makes it finalize early. * `sampleRate` defaults to the pipeline's input rate. * If the connection keeps failing quickly (an invalid key, say), the service stops retrying and pushes an error. ## DeepgramFluxSTTService [#deepgramfluxsttservice] ### Configuration [#configuration-1] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | Deepgram API key for authentication. Required for API access. | | `url` | `string \| undefined` | no | WebSocket URL for the Deepgram Flux API. Defaults to Deepgram's Flux endpoint, "wss://api.deepgram.com/v2/listen". | | `fluxEncoding` | `string \| undefined` | no | Audio encoding format required by Flux API. Must be "linear16". Raw signed little-endian 16-bit PCM encoding. | | `settings` | `DeepgramFluxSTTSettings \| null \| undefined` | no | Runtime-updatable settings. | | `webSocketFactory` | `((uri: string, options: { headers?: Record; protocols?: string[]; }) => WebSocket \| Promise) \| null \| undefined` | no | Creates the underlying `WebSocket`, to open it some other way (through a proxy, say). Defaults to `websocketConnect`'s, which sends the header on every runtime, Cloudflare Workers included. Not in upstream. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `reconnectBackoffMinWait` | `number \| undefined` | no | Minimum time, in seconds, to wait between reconnection attempts. | | `reconnectBackoffMaxWait` | `number \| undefined` | no | Maximum time, in seconds, to wait between reconnection attempts. | | `reconnectOnError` | `boolean \| undefined` | no | Whether to automatically reconnect on connection errors. | | `wsCloseTimeout` | `number \| undefined` | no | Maximum time, in seconds, to wait for the peer to acknowledge the websocket closing handshake before dropping the connection. Applied to connections opened through `_websocketConnect()`. Increase it for peers that need longer to complete a graceful close. | | `wsSendTimeout` | `number \| null \| undefined` | no | Maximum time, in seconds, a single send may wait on a peer that has stopped reading before the connection is dropped and the send throws `ConnectionClosedError`. Applied to connections opened through `_websocketConnect()`. Pass `null` to wait indefinitely. Defaults to `WS_SEND_TIMEOUT`. | | `audioPassthrough` | `boolean \| undefined` | no | Whether to pass audio frames downstream after processing. Defaults to true. | | `sampleRate` | `number \| null \| undefined` | no | The sample rate for audio input. If null, will be determined from the start frame. | | `sttTtfbTimeout` | `number \| undefined` | no | Time in seconds to wait after VAD stop before reporting TTFB. This delay allows the final transcription to arrive. Defaults to 2.0. | | `ttfsP99Latency` | `number \| null \| undefined` | no | P99 latency from speech end to final transcript in seconds. Broadcast via STTMetadataFrame at pipeline start. | | `keepaliveTimeout` | `number \| null \| undefined` | no | Seconds of no audio before sending silence to keep the connection alive. null disables keepalive. | | `keepaliveInterval` | `number \| undefined` | no | Seconds between idle checks when keepalive is enabled. | | `enableEagerEndOfTurn` | `boolean \| undefined` | no | Whether to answer a predicted end of turn ahead of the committed one. Off by default: it spends an inference on every prediction, including the ones the service withdraws. | | `eagerMatchPolicy` | `EagerMatchPolicy \| null \| undefined` | no | Decides whether the committed transcript is close enough to the predicted one to keep the speculative response. Only consulted while eager end of turn is on. See EagerUserTurnStrategies. | | `mipOptOut` | `boolean \| null \| undefined` | no | Opt out of the Deepgram Model Improvement Program. | | `tag` | `string[] \| null \| undefined` | no | Tags to label requests for identification during usage reporting. | | `shouldInterrupt` | `boolean \| undefined` | no | Whether to interrupt the bot when Flux detects that the user is speaking. | | `watchdogMinTimeout` | `number \| undefined` | no | Minimum idle timeout before sending silence to prevent dangling turns. | ### Settings [#settings-1] | Setting | Type | Default | Description | | ------------------- | -------------------- | ----------------------- | --------------------------------------------------------------- | | `model` | `string` | `"flux-general-en"` | Flux model; `"flux-general-multi"` for multilingual. | | `eagerEotThreshold` | `number \| null` | `null` (off) | Threshold for eager end-of-turn and turn-resumed events. | | `eotThreshold` | `number \| null` | `null` (Deepgram: 0.7) | End-of-turn confidence to finish a turn. | | `eotTimeoutMs` | `number \| null` | `null` (Deepgram: 5000) | Finish a turn this long after speech, regardless of confidence. | | `keyterm` | `string[]` | `[]` | Keyterms to boost. | | `minConfidence` | `number \| null` | `null` | Minimum confidence to emit a `TranscriptionFrame`. | | `numerals` | `boolean \| null` | `null` | Spoken numbers as numerals. | | `profanityFilter` | `boolean \| null` | `null` | Mask profanity. | | `redact` | `string \| null` | `null` | `"numbers"` or `"aggressive_numbers"`. Changing it reconnects. | | `languageHints` | `Language[] \| null` | `null` | Bias detection (`flux-general-multi` only). | ### Usage [#usage-1] ```ts title="flux.ts" import { Language } from "@pipedog/core/transcriptions/language"; import { DeepgramFluxSTTService } from "@pipedog/deepgram/flux/stt"; const stt = new DeepgramFluxSTTService({ apiKey: process.env.DEEPGRAM_API_KEY ?? "", settings: new DeepgramFluxSTTService.Settings({ model: "flux-general-en", eagerEotThreshold: 0.5, eotThreshold: 0.8, }), }); stt.addEventHandler("on_end_of_turn", async (_service, transcript: string) => { console.log("end of turn:", transcript); }); ``` Multilingual, with language hints: ```ts const multilingual = new DeepgramFluxSTTService({ apiKey: process.env.DEEPGRAM_API_KEY ?? "", settings: new DeepgramFluxSTTService.Settings({ model: "flux-general-multi", languageHints: [Language.EN, Language.ES], }), }); ``` ### Event handlers [#event-handlers] | Event | Description | | ---------------------- | ------------------------------------------------------------------ | | `on_start_of_turn` | Deepgram detected the start of speech. | | `on_end_of_turn` | Deepgram detected the end of the turn. | | `on_eager_end_of_turn` | Deepgram predicted the end of the turn (with `eagerEotThreshold`). | | `on_turn_resumed` | The user kept speaking after an eager end of turn. | | `on_update` | A transcript update for the turn in progress. | ### Notes [#notes-1] * Emits an `InterimTranscriptionFrame` on every Flux update and a `TranscriptionFrame` at the end of the turn. * Flux needs `linear16` audio, which is what pipedog transports produce. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/deepgram/package.json`: ✅ means the compat smoke test passes on that runtime. Both services use the global `WebSocket` and send the key as an `Authorization` header on the handshake. That works on Bun, Node, Deno and Cloudflare Workers (where the socket is opened through `fetch`). Pass `webSocketFactory` to open the socket some other way, through a proxy for example. A standard `WebSocket` doesn't expose the HTTP status of a refused handshake, so `DeepgramApiError` is only raised when the runtime's error carries one. # ElevenLabs > Speech-to-text with ElevenLabsSTTService (segmented uploads) and ElevenLabsRealtimeSTTService (WebSocket streaming). Source: https://polvallverdu.github.io/pipedog/docs/services/stt/elevenlabs/ ## Overview [#overview] ElevenLabs Scribe transcribes speech. pipedog ports both of Pipecat's ElevenLabs speech-to-text services: * **`ElevenLabsSTTService`**: uploads each speech segment, cut by the pipeline's VAD, to ElevenLabs' `/v1/speech-to-text` endpoint and pushes the transcript it gets back. * **`ElevenLabsRealtimeSTTService`**: streams audio over ElevenLabs' realtime WebSocket API, with partial and committed transcripts. Segments are committed by the pipeline's VAD or by ElevenLabs' own. Pipecat's documentation for the same services. `packages/elevenlabs/src/stt.ts`. Scribe models, languages and the realtime API. ## Installation [#installation] ```bash bun add @pipedog/elevenlabs ``` ```ts import { CommitStrategy, ElevenLabsRealtimeSTTService, ElevenLabsSTTService, } from "@pipedog/elevenlabs/stt"; ``` ## Prerequisites [#prerequisites] * An ElevenLabs account and an API key from the [ElevenLabs dashboard](https://elevenlabs.io/app/settings/api-keys). * Pass the key as `apiKey`. The examples read it from `ELEVENLABS_API_KEY`. ## ElevenLabsSTTService [#elevenlabssttservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | ElevenLabs API key for authentication. | | `fetch` | `((input: RequestInfo \| URL, init?: RequestInit) => Promise) \| null \| undefined` | no | `fetch` used for HTTP requests. Defaults to the global `fetch`. DIVERGENCE: replaces upstream's required `aiohttp_session`. | | `baseUrl` | `string \| undefined` | no | Base URL for ElevenLabs API. Defaults to "https://api.elevenlabs.io". | | `settings` | `ElevenLabsSTTSettings \| null \| undefined` | no | Runtime-updatable settings. Its given fields override the defaults (model "scribe_v2", language English). | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `audioPassthrough` | `boolean \| undefined` | no | Whether to pass audio frames downstream after processing. Defaults to true. | | `sampleRate` | `number \| null \| undefined` | no | The sample rate for audio input. If null, will be determined from the start frame. | | `sttTtfbTimeout` | `number \| undefined` | no | Time in seconds to wait after VAD stop before reporting TTFB. This delay allows the final transcription to arrive. Defaults to 2.0. | | `ttfsP99Latency` | `number \| null \| undefined` | no | P99 latency from speech end to final transcript in seconds. Broadcast via STTMetadataFrame at pipeline start. | | `keepaliveTimeout` | `number \| null \| undefined` | no | Seconds of no audio before sending silence to keep the connection alive. null disables keepalive. | | `keepaliveInterval` | `number \| undefined` | no | Seconds between idle checks when keepalive is enabled. | | `trailingSilenceSecs` | `number \| undefined` | no | Seconds of silence appended to each segment before it is transcribed, so the model hears the end of speech and can finish the last word. Set to 0 to disable. Defaults to 0.5. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new ElevenLabsSTTService.Settings({...})`. Any setting you leave out keeps the default below. `null` means the field is not sent, so ElevenLabs uses its own default. | Setting | Type | Default | Description | | ---------------- | ------------------ | ------------- | ------------------------------------------------------------------------ | | `model` | `string` | `"scribe_v2"` | Scribe model. | | `language` | `Language \| null` | `Language.EN` | Transcription language. `null` lets ElevenLabs detect it. | | `tagAudioEvents` | `boolean \| null` | `null` | Include audio events such as (laughter) in the transcript. | | `keyterms` | `string[] \| null` | `null` | Terms to bias transcription towards. | | `noVerbatim` | `boolean \| null` | `null` | Drop filler words, false starts and non-speech sounds. `scribe_v2` only. | ### Usage [#usage] ```ts title="stt.ts" import { STTUpdateSettingsFrame } from "@pipedog/core/frames/frames"; import { Language } from "@pipedog/core/transcriptions/language"; import { CommitStrategy, ElevenLabsRealtimeSTTService, ElevenLabsSTTService, } from "@pipedog/elevenlabs/stt"; const stt = new ElevenLabsSTTService({ apiKey: process.env.ELEVENLABS_API_KEY ?? "", settings: new ElevenLabsSTTService.Settings({ model: "scribe_v2", language: Language.EN, keyterms: ["pipedog", "Pipecat"], }), }); ``` ### Notes [#notes] * A segmented service: it needs VAD in the pipeline (on the user aggregator). Audio between the VAD's start and stop is sent as one WAV upload, padded with a little trailing silence. * Transcription runs in the background, so audio keeps flowing while a segment is transcribed. * Emits one `TranscriptionFrame` per segment, with the language code ElevenLabs returns. There are no interim results. ## ElevenLabsRealtimeSTTService [#elevenlabsrealtimesttservice] ### Configuration [#configuration-1] | Option | Type | Required | Description | |---|---|---|---| | `reconnectBackoffMinWait` | `number \| undefined` | no | Minimum time, in seconds, to wait between reconnection attempts. | | `reconnectBackoffMaxWait` | `number \| undefined` | no | Maximum time, in seconds, to wait between reconnection attempts. | | `apiKey` | `string` | yes | ElevenLabs API key for authentication. | | `baseUrl` | `string \| undefined` | no | Base URL for ElevenLabs WebSocket API. Defaults to "api.elevenlabs.io". | | `commitStrategy` | `CommitStrategy \| undefined` | no | How to segment speech — `CommitStrategy.MANUAL` (Pipecat VAD) or `CommitStrategy.VAD` (ElevenLabs VAD). Defaults to `CommitStrategy.MANUAL`. | | `includeTimestamps` | `boolean \| undefined` | no | Whether to include word-level timestamps in transcripts. | | `enableLogging` | `boolean \| undefined` | no | Whether to enable logging on ElevenLabs' side. | | `includeLanguageDetection` | `boolean \| undefined` | no | Whether to include language detection in transcripts. | | `settings` | `ElevenLabsRealtimeSTTSettings \| null \| undefined` | no | Runtime-updatable settings. Its given fields override the defaults (model "scribe_v2_realtime", language auto-detected). | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `wsCloseTimeout` | `number \| undefined` | no | Maximum time, in seconds, to wait for the peer to acknowledge the websocket closing handshake before dropping the connection. Applied to connections opened through `_websocketConnect()`. Increase it for peers that need longer to complete a graceful close. | | `wsSendTimeout` | `number \| null \| undefined` | no | Maximum time, in seconds, a single send may wait on a peer that has stopped reading before the connection is dropped and the send throws `ConnectionClosedError`. Applied to connections opened through `_websocketConnect()`. Pass `null` to wait indefinitely. Defaults to `WS_SEND_TIMEOUT`. | | `audioPassthrough` | `boolean \| undefined` | no | Whether to pass audio frames downstream after processing. Defaults to true. | | `sampleRate` | `number \| null \| undefined` | no | The sample rate for audio input. If null, will be determined from the start frame. | | `sttTtfbTimeout` | `number \| undefined` | no | Time in seconds to wait after VAD stop before reporting TTFB. This delay allows the final transcription to arrive. Defaults to 2.0. | | `ttfsP99Latency` | `number \| null \| undefined` | no | P99 latency from speech end to final transcript in seconds. Broadcast via STTMetadataFrame at pipeline start. | | `keepaliveTimeout` | `number \| null \| undefined` | no | Seconds of no audio before sending silence to keep the connection alive. null disables keepalive. | | `keepaliveInterval` | `number \| undefined` | no | Seconds between idle checks when keepalive is enabled. | | `reconnectOnError` | `boolean \| undefined` | no | Whether to automatically reconnect on websocket errors. Defaults to true. | ### Settings [#settings-1] | Setting | Type | Default | Description | | ------------------------- | ------------------ | ---------------------- | --------------------------------------------------------------------------------- | | `model` | `string` | `"scribe_v2_realtime"` | Scribe realtime model. | | `language` | `Language \| null` | `null` (auto-detect) | Transcription language. | | `keyterms` | `string[] \| null` | `null` | Terms to bias transcription towards. | | `noVerbatim` | `boolean \| null` | `null` | Drop filler words, false starts and non-speech sounds. | | `vadSilenceThresholdSecs` | `number \| null` | `null` | Silence before ElevenLabs' VAD commits, 0.3 to 3.0 s. `CommitStrategy.VAD` only. | | `vadThreshold` | `number \| null` | `null` | VAD sensitivity, 0.1 to 0.9 (lower is more sensitive). `CommitStrategy.VAD` only. | | `minSpeechDurationMs` | `number \| null` | `null` | Minimum speech duration, 50 to 2000 ms. `CommitStrategy.VAD` only. | | `minSilenceDurationMs` | `number \| null` | `null` | Minimum silence duration, 50 to 2000 ms. `CommitStrategy.VAD` only. | | `filterBackgroundAudio` | `boolean \| null` | `null` | Filter background audio before transcription. | ### Usage [#usage-1] ```ts title="realtime.ts" const realtime = new ElevenLabsRealtimeSTTService({ apiKey: process.env.ELEVENLABS_API_KEY ?? "", settings: new ElevenLabsRealtimeSTTService.Settings({ language: Language.EN, keyterms: ["pipedog"], }), }); ``` With ElevenLabs' VAD committing segments: ```ts // Let ElevenLabs decide when a segment ends instead of the pipeline's VAD. const serverVad = new ElevenLabsRealtimeSTTService({ apiKey: process.env.ELEVENLABS_API_KEY ?? "", commitStrategy: CommitStrategy.VAD, includeTimestamps: true, settings: new ElevenLabsRealtimeSTTService.Settings({ vadSilenceThresholdSecs: 0.8, vadThreshold: 0.4, }), }); ``` Change settings mid-call with an `STTUpdateSettingsFrame`. The service reconnects with the new settings: ```ts // Queue it on the worker, or push it from a processor upstream of the service. const update = new STTUpdateSettingsFrame({ delta: new ElevenLabsRealtimeSTTService.Settings({ language: Language.ES }), }); ``` ### Event handlers [#event-handlers] Inherited from `STTService`: | Event | Description | | --------------------- | ----------------------------- | | `on_connected` | Connected to ElevenLabs. | | `on_disconnected` | Disconnected from ElevenLabs. | | `on_connection_error` | The connection failed. | ### Notes [#notes-1] * Emits `InterimTranscriptionFrame` for partial transcripts and `TranscriptionFrame` for committed ones. * With `CommitStrategy.MANUAL` (the default), a `VADUserStoppedSpeakingFrame` makes the service send a commit, so the pipeline's VAD decides where segments end. With `CommitStrategy.VAD`, ElevenLabs decides, using the `vad*` and `min*DurationMs` settings. * The audio format is chosen from the sample rate: 8000, 16000, 22050, 24000, 44100 or 48000 Hz. Any other rate falls back to `pcm_16000` with a warning. * The service sends silence as a keepalive after 10 s without audio (checked every 5 s). * Error messages from ElevenLabs (authentication, quota, rate limit, throttled commits, session time limit and others) are pushed as errors. * `baseUrl` is a bare host and defaults to `"api.elevenlabs.io"`. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/elevenlabs/package.json`: ✅ means the compat smoke test passes on that runtime. * `ElevenLabsSTTService` uploads with `fetch` and `FormData`. Upstream's required `aiohttp_session` is replaced by an optional `fetch` option (the global `fetch` by default). * `ElevenLabsRealtimeSTTService` uses the global `WebSocket` and sends the key as an `xi-api-key` header on the handshake. That works on Bun, Node and Deno, and on Cloudflare Workers, where the socket is opened through `fetch`. * Not ported: the deprecated `InputParams` classes and the `model` / `params` constructor options. Use `settings`. # FunASR > Local multilingual speech-to-text with FunASRSTTService, running SenseVoiceSmall on ONNX Runtime. Strong on Chinese, with emotion and audio-event tags. No API key. Source: https://polvallverdu.github.io/pipedog/docs/services/stt/funasr/ ## Overview [#overview] * **`FunASRSTTService`**: a segmented STT service that transcribes each speech segment, cut by the pipeline's VAD, with [SenseVoiceSmall](https://huggingface.co/FunAudioLLM/SenseVoiceSmall) on your machine. SenseVoice is multilingual (Chinese, Cantonese, English, Japanese, Korean), with strong Chinese accuracy and fast non-autoregressive inference. No account, no API key. * **`AutoModel`** (`@pipedog/funasr/funasr`): the engine. Upstream runs the model with the `funasr` Python package (PyTorch). This one runs the same weights on ONNX Runtime through `@pipedog/onnx`, and mirrors the part of `funasr` the service uses: `AutoModel`, `generate()` and `richTranscriptionPostprocess()`. Its transcripts match sherpa-onnx's for the same model. Pipecat's documentation for the same service. `packages/funasr/src/stt.ts` and the engine, `funasr.ts`. The model, its languages and tags. ## Installation [#installation] ```bash bun add @pipedog/funasr onnxruntime-node ``` `onnxruntime-node` is an optional peer of `@pipedog/onnx`. `onnxruntime-web@1.30.0` (WebAssembly) works too, about 10x slower (1.2 s instead of 0.11 s for a 7 s clip on an M4 Pro). ```ts import { FunASRSTTService } from "@pipedog/funasr/stt"; ``` ## What you need [#what-you-need] | | | | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Model | sherpa-onnx's ONNX export of `iic/SenseVoiceSmall`, from Hugging Face [`csukuangfj/sherpa-onnx-sense-voice-zh-en-ja-ko-yue-2024-07-17`](https://huggingface.co/csukuangfj/sherpa-onnx-sense-voice-zh-en-ja-ko-yue-2024-07-17) (pinned revision): `model.int8.onnx` (239 MB, the default) or `model.onnx` (938 MB, with `modelOptions: { quantized: false }`), plus `tokens.txt` (316 KB) | | Download | Automatic, once, when the pipeline starts (or on the first transcription). No account or token needed | | Cache | `$PIPEDOG_CACHE_DIR/funasr`, else `$XDG_CACHE_HOME/pipedog/funasr`, else `~/.cache/pipedog/funasr`, or `modelOptions: { cacheDir }` | | Input | 16 kHz, 16-bit mono PCM, and a VAD in the pipeline | To use local files instead, set the `model` setting to a directory holding `model.int8.onnx` (or `model.onnx`) and `tokens.txt`, or to the `.onnx` file's path with `tokens.txt` next to it, or pass explicit files: ```ts // A directory holding model.int8.onnx (or model.onnx) and tokens.txt... const fromDir = new FunASRSTTService({ settings: new FunASRSTTService.Settings({ model: "/models/sense-voice" }), }); // ...or explicit files. const fromFiles = new FunASRSTTService({ modelOptions: { modelFile: "/models/model.int8.onnx", tokens: "/models/tokens.txt" }, }); ``` ## Runtimes [#runtimes] | Runtime | Works | What it needs | | ------------------ | ----- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Bun | ✅ yes | onnxruntime-node (or onnxruntime-web) and the SenseVoiceSmall ONNX export (model.int8.onnx 239 MB + tokens.txt), downloaded once from Hugging Face to \~/.cache/pipedog/funasr. | | Node.js | ✅ yes | onnxruntime-node (or onnxruntime-web) and the SenseVoiceSmall ONNX export (model.int8.onnx 239 MB + tokens.txt), downloaded once from Hugging Face to \~/.cache/pipedog/funasr. | | Deno | ✅ yes | onnxruntime-node (or onnxruntime-web) and the SenseVoiceSmall ONNX export (model.int8.onnx 239 MB + tokens.txt), downloaded once from Hugging Face to \~/.cache/pipedog/funasr. | | Cloudflare Workers | ❌ no | onnxruntime-node (or onnxruntime-web) and the SenseVoiceSmall ONNX export (model.int8.onnx 239 MB + tokens.txt), downloaded once from Hugging Face to \~/.cache/pipedog/funasr. Not supported: the model (239 MB int8) exceeds Workers' memory and bundle limits. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/funasr/package.json`: ✅ means the compat smoke test passes on that runtime. Cloudflare Workers is not supported: the model (239 MB as int8) doesn't fit a Worker's 128 MB of memory or its 10 MB bundle limit. ## FunASRSTTService [#funasrsttservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `device` | `string \| undefined` | no | Inference device, `"cpu"` or `"cuda"`. | | `settings` | `FunASRSTTSettings \| null \| undefined` | no | Runtime-updatable settings (`model`, `language`, `useItn`). | | `modelOptions` | `Partial> \| undefined` | no | Extra `AutoModel` options (pipedog only). | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `audioPassthrough` | `boolean \| undefined` | no | Whether to pass audio frames downstream after processing. Defaults to true. | | `sampleRate` | `number \| null \| undefined` | no | The sample rate for audio input. If null, will be determined from the start frame. | | `sttTtfbTimeout` | `number \| undefined` | no | Time in seconds to wait after VAD stop before reporting TTFB. This delay allows the final transcription to arrive. Defaults to 2.0. | | `ttfsP99Latency` | `number \| null \| undefined` | no | P99 latency from speech end to final transcript in seconds. Broadcast via STTMetadataFrame at pipeline start. | | `keepaliveTimeout` | `number \| null \| undefined` | no | Seconds of no audio before sending silence to keep the connection alive. null disables keepalive. | | `keepaliveInterval` | `number \| undefined` | no | Seconds between idle checks when keepalive is enabled. | | `trailingSilenceSecs` | `number \| undefined` | no | Seconds of silence appended to each segment before it is transcribed, so the model hears the end of speech and can finish the last word. Set to 0 to disable. Defaults to 0.5. | `modelOptions` (pipedog only) takes `cacheDir`, `quantized`, `modelFile`, `tokens`, `backend`, `shareSession` and `fetch`. ### Settings [#settings] Runtime-updatable settings, passed as `settings: new FunASRSTTService.Settings({...})` (`FunASRSTTSettings`). Any setting you leave out keeps the default below. | Setting | Type | Default | Description | | ---------- | ---------- | ----------------------- | ----------------------------------------------------------------------------------------------------------------------- | | `model` | `string` | `"iic/SenseVoiceSmall"` | `"iic/SenseVoiceSmall"` or `"FunAudioLLM/SenseVoiceSmall"` to download the model, or a local directory or `.onnx` path. | | `language` | `Language` | `Language.EN` | `ZH`, `EN`, `JA`, `KO` or `YUE`. Any other language means auto-detect. | | `useItn` | `boolean` | `true` | Inverse text normalization: punctuation and digits ("nine" becomes "9"). | ### Usage [#usage] ```ts title="stt.ts" import { STTUpdateSettingsFrame } from "@pipedog/core/frames/frames"; import { Language } from "@pipedog/core/transcriptions/language"; import { FunASRSTTService } from "@pipedog/funasr/stt"; // Downloads SenseVoiceSmall (model.int8.onnx, 239 MB, and tokens.txt) on first start. const stt = new FunASRSTTService({ settings: new FunASRSTTService.Settings({ language: Language.ZH, useItn: true }), }); ``` Change the language mid-call with an `STTUpdateSettingsFrame`: ```ts // Queue it on the worker, or push it from a processor upstream of the service. const update = new STTUpdateSettingsFrame({ delta: new FunASRSTTService.Settings({ language: Language.JA }), }); ``` The engine works on its own too. `generate()` returns FunASR's raw text with its `<|...|>` language, emotion and event tags, and `richTranscriptionPostprocess()` strips them: ```ts import { AutoModel, richTranscriptionPostprocess } from "@pipedog/funasr/funasr"; const model = new AutoModel({ model: "iic/SenseVoiceSmall" }); const samples = new Float32Array(16000); // 16 kHz mono audio in [-1, 1] const [result] = await model.generate({ input: samples, language: "auto", useItn: true }); if (result) { console.log(result.text); // "<|en|><|NEUTRAL|><|Speech|><|withitn|>..." console.log(richTranscriptionPostprocess(result.text)); // the text without the tags } ``` ### Notes [#notes] * Emits one `TranscriptionFrame` per segment, with the detected language. There are no interim results. * The front end computes FunASR's `WavFrontend` features (Kaldi fbank, LFR, CMVN from the model's metadata), within 2e-4 of kaldi-native-fbank's, then decodes with CTC greedy search, as FunASR does. * FunASR's PyTorch checkpoints and other ModelScope model ids are not supported. ## Differences from upstream [#differences-from-upstream] * The model runs on onnxruntime on the CPU, not PyTorch. `device` is accepted, but `"cuda"` doesn't select a GPU. * The model downloads and loads in `start()` (or on the first transcription), not in the constructor. * No dither in the front end (FunASR's `WavFrontend` adds dither 1.0 by default), so results are deterministic. * `modelOptions` is a pipedog option, and the engine's options are camelCase (`disableUpdate`, `useItn`). ## Licenses [#licenses] * `@pipedog/funasr`: BSD 2-Clause. `postprocess_utils.ts` is adapted from [FunASR](https://github.com/modelscope/FunASR) (MIT, Copyright (c) 2022 Alibaba). * SenseVoiceSmall weights: the [FunASR model license](https://github.com/modelscope/FunASR/blob/main/MODEL_LICENSE) (attribution required; check its terms for your use). The ONNX export is by k2-fsa/sherpa-onnx. # Gladia > Realtime multilingual speech-to-text with Gladia. Source: https://polvallverdu.github.io/pipedog/docs/services/stt/gladia/ ## Overview [#overview] `GladiaSTTService` opens a Gladia live session over HTTP (`POST /v2/live`), then streams base64 audio over the returned WebSocket, with code-switching, translation, custom vocabulary and realtime processing options (see `@pipedog/gladia/config`). Pipecat's documentation for the same service. `packages/gladia/src`. ## Installation [#installation] ```bash bun add @pipedog/gladia ``` ## Prerequisites [#prerequisites] * A [Gladia](https://www.gladia.io) account and API key, passed as `apiKey` (the examples read `GLADIA_API_KEY`). ## GladiaSTTService [#gladiasttservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | Gladia API key for authentication. | | `region` | `"eu-west" \| "us-west" \| null \| undefined` | no | Region used to process audio. eu-west or us-west. Defaults to eu-west. | | `url` | `string \| undefined` | no | Gladia API URL. Defaults to "https://api.gladia.io/v2/live". | | `encoding` | `string \| undefined` | no | Audio encoding format. Defaults to `"wav/pcm"`. | | `bitDepth` | `number \| undefined` | no | Audio bit depth. Defaults to 16. | | `channels` | `number \| undefined` | no | Number of audio channels. Defaults to 1. | | `maxBufferSize` | `number \| undefined` | no | Maximum size of audio buffer in bytes. Defaults to 20MB. | | `shouldInterrupt` | `boolean \| undefined` | no | Determine whether the bot should be interrupted when Gladia VAD detects user speech. Passed along to the user turn strategies this service recommends, which own the interruption; a user-supplied `userTurnStrategies` overrides the recommendation and this setting with it. Defaults to true. | | `settings` | `GladiaSTTSettings \| null \| undefined` | no | Runtime-updatable settings. | | `fetch` | `((input: RequestInfo \| URL, init?: RequestInit) => Promise) \| undefined` | no | `fetch` used for the session init request. Defaults to the global one. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `reconnectBackoffMinWait` | `number \| undefined` | no | Minimum time, in seconds, to wait between reconnection attempts. | | `reconnectBackoffMaxWait` | `number \| undefined` | no | Maximum time, in seconds, to wait between reconnection attempts. | | `wsCloseTimeout` | `number \| undefined` | no | Maximum time, in seconds, to wait for the peer to acknowledge the websocket closing handshake before dropping the connection. Applied to connections opened through `_websocketConnect()`. Increase it for peers that need longer to complete a graceful close. | | `wsSendTimeout` | `number \| null \| undefined` | no | Maximum time, in seconds, a single send may wait on a peer that has stopped reading before the connection is dropped and the send throws `ConnectionClosedError`. Applied to connections opened through `_websocketConnect()`. Pass `null` to wait indefinitely. Defaults to `WS_SEND_TIMEOUT`. | | `audioPassthrough` | `boolean \| undefined` | no | Whether to pass audio frames downstream after processing. Defaults to true. | | `sampleRate` | `number \| null \| undefined` | no | The sample rate for audio input. If null, will be determined from the start frame. | | `sttTtfbTimeout` | `number \| undefined` | no | Time in seconds to wait after VAD stop before reporting TTFB. This delay allows the final transcription to arrive. Defaults to 2.0. | | `ttfsP99Latency` | `number \| null \| undefined` | no | P99 latency from speech end to final transcript in seconds. Broadcast via STTMetadataFrame at pipeline start. | | `keepaliveTimeout` | `number \| null \| undefined` | no | Seconds of no audio before sending silence to keep the connection alive. null disables keepalive. | | `keepaliveInterval` | `number \| undefined` | no | Seconds between idle checks when keepalive is enabled. | | `reconnectOnError` | `boolean \| undefined` | no | Whether to automatically reconnect on websocket errors. Defaults to true. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new GladiaSTTService.Settings({...})`. Settings you leave out keep their defaults. | Setting | Default | Description | | ------------------------------------------------------- | ------------- | --------------------------------------------- | | `model` | `"solaria-1"` | Gladia model. | | `language` | | Single language. | | `languageConfig` | | `LanguageConfig` (languages, code switching). | | `endpointing` | | Endpointing, in seconds. | | `maximumDurationWithoutEndpointing` | `5` | Seconds. | | `preProcessing`, `realtimeProcessing`, `messagesConfig` | | Config classes from `@pipedog/gladia/config`. | | `enableVad` | | Gladia-side VAD. | | `customMetadata` | | Metadata attached to the session. | ### Usage [#usage] ```ts title="stt.ts" import { LanguageConfig } from "@pipedog/gladia/config"; import { GladiaSTTService } from "@pipedog/gladia/stt"; const stt = new GladiaSTTService({ apiKey: process.env.GLADIA_API_KEY ?? "", region: "eu-west", settings: new GladiaSTTService.Settings({ languageConfig: new LanguageConfig({ languages: ["en", "fr"], codeSwitching: true }), }), }); ``` ### Notes [#notes] * `region` is `"us-west"` or `"eu-west"`. Pass `fetch` to customize the session request. ## Runtimes [#runtimes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/gladia/package.json`: ✅ means the compat smoke test passes on that runtime. The service uses `fetch` and the global `WebSocket` (through core's `websocketConnect`, which sends the auth header on the handshake on Cloudflare Workers too). Pass no SDK; nothing is Node-only. # Google > Speech-to-text with GeminiSTTService (Gemini Live transcription models) and GoogleSTTService (Google Cloud Speech-to-Text V2). Source: https://polvallverdu.github.io/pipedog/docs/services/stt/google/ ## Overview [#overview] pipedog ports both of Pipecat's Google speech-to-text services. They use different APIs and different credentials: * **`GeminiSTTService`**: streams audio to a Gemini Live transcription model (`gemini-3.5-transcribe-live` by default) over the Gemini Live API. Authenticates with a Gemini API key, or with Google Cloud credentials on Vertex AI. Supports automatic language detection, language hints, adaptation phrases and verbatim or smart transcription. * **`GoogleSTTService`**: Google Cloud Speech-to-Text V2 streaming recognition. Authenticates with a Google Cloud service account or Application Default Credentials (ADC). Supports multiple languages, phrase-set adaptation, denoising and automatic stream renewal. Pipecat's documentation for the same services. `packages/google/src/gemini_live/stt.ts` and `stt.ts`. Models, languages and adaptation for Speech-to-Text V2. ## Installation [#installation] ```bash bun add @pipedog/google ``` `GoogleSTTService` also needs the Cloud Speech Node SDK, an optional peer dependency: ```bash bun add @google-cloud/speech ``` ```ts import { GeminiSTTService } from "@pipedog/google/gemini_live/stt"; import { GoogleSTTService } from "@pipedog/google/stt"; ``` ## Prerequisites [#prerequisites] * **`GeminiSTTService`**: a Gemini API key from [Google AI Studio](https://aistudio.google.com/apikey), passed as `apiKey`. The examples read it from `GOOGLE_API_KEY`. On Vertex AI (`vertexai: true`), a Google Cloud project with the Vertex AI API enabled, and a service account key or ADC instead. * **`GoogleSTTService`**: a Google Cloud project with the Speech-to-Text API enabled, and a service account key (as a file or a JSON string) or ADC. The examples read `GOOGLE_APPLICATION_CREDENTIALS` (a key file path) or `GOOGLE_CREDENTIALS_JSON` (the key itself). The project ID comes from the credentials. ## GeminiSTTService [#geministtservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string \| undefined` | no | Google AI (Gemini) API key. Required unless `vertexai` is true. | | `vertexai` | `boolean \| undefined` | no | Connect through Vertex AI instead of the Gemini Developer API, with Google Cloud credentials instead of an API key. Requires `location` (pipedog only). | | `location` | `string \| undefined` | no | Vertex AI location, e.g. `"eu"` or `"us-central1"`. Required with `vertexai` (pipedog only). | | `projectId` | `string \| null \| undefined` | no | Google Cloud project ID, with `vertexai`. Defaults to the `GOOGLE_CLOUD_PROJECT` environment variable (pipedog only). | | `credentials` | `string \| null \| undefined` | no | JSON string of service account credentials, with `vertexai`. Without it (or `credentialsPath`), Application Default Credentials are used (pipedog only). | | `credentialsPath` | `string \| null \| undefined` | no | Path to the service account JSON file, with `vertexai` (pipedog only). | | `httpOptions` | `GoogleClientHttpOptions \| null \| undefined` | no | Optional HTTP options passed to the google-genai client. | | `settings` | `GeminiSTTSettings \| null \| undefined` | no | Runtime-updatable settings. Defaults to the `gemini-3.5-transcribe-live` model with automatic language detection. | | `reportDetectedLanguage` | `boolean \| undefined` | no | Put the language the model reports on transcription frames. Defaults to true. When false, frames carry the configured language if exactly one is configured, and `null` otherwise (pipedog only). | | `awaitConnectionOnStart` | `boolean \| undefined` | no | Hold the pipeline's StartFrame until the Live session is open, as upstream does. Defaults to false: the session is opened in the background and audio is buffered until it is live (pipedog only). | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `audioPassthrough` | `boolean \| undefined` | no | Whether to pass audio frames downstream after processing. Defaults to true. | | `sampleRate` | `number \| null \| undefined` | no | The sample rate for audio input. If null, will be determined from the start frame. | | `sttTtfbTimeout` | `number \| undefined` | no | Time in seconds to wait after VAD stop before reporting TTFB. This delay allows the final transcription to arrive. Defaults to 2.0. | | `ttfsP99Latency` | `number \| null \| undefined` | no | P99 latency from speech end to final transcript in seconds. Broadcast via STTMetadataFrame at pipeline start. | | `keepaliveTimeout` | `number \| null \| undefined` | no | Seconds of no audio before sending silence to keep the connection alive. null disables keepalive. | | `keepaliveInterval` | `number \| undefined` | no | Seconds between idle checks when keepalive is enabled. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new GeminiSTTService.Settings({...})`. | Setting | Type | Default | Description | | ------------------- | ------------------------------- | ------------------------------ | --------------------------------------------------------------------------------- | | `model` | `string` | `"gemini-3.5-transcribe-live"` | Gemini Live transcription model. | | `language` | `Language \| string \| null` | `null` | A single language hint. | | `languages` | `(Language \| string)[]` | `[]` | Language hints. Take precedence over `language`. | | `languageAuto` | `boolean \| null` | `null` | Automatic detection. `null` detects unless hints are given; `false` turns it off. | | `adaptationPhrases` | `string[] \| null` | `null` | Phrases to bias recognition toward. | | `mode` | `"VERBATIM" \| "SMART" \| null` | `null` (model: `"VERBATIM"`) | `"SMART"` removes disfluencies and applies light cleanup. | ### Usage [#usage] ```ts title="gemini-stt.ts" import { Language } from "@pipedog/core/transcriptions/language"; import { GeminiSTTService } from "@pipedog/google/gemini_live/stt"; const stt = new GeminiSTTService({ apiKey: process.env.GOOGLE_API_KEY ?? "", }); stt.addEventHandler("on_connected", async () => { console.log("connected to the Gemini Live API"); }); ``` With language hints, adaptation phrases and smart transcription: ```ts const hinted = new GeminiSTTService({ apiKey: process.env.GOOGLE_API_KEY ?? "", settings: new GeminiSTTService.Settings({ languages: [Language.ES_ES, Language.EN_US], adaptationPhrases: ["pipedog", "oatmilk"], mode: "SMART", }), }); ``` ### Vertex AI [#vertex-ai] `vertexai: true` (pipedog only) connects through Vertex AI instead of the Gemini Developer API. It takes `location` (required, e.g. `"eu"` for data residency), `projectId` (defaults to `GOOGLE_CLOUD_PROJECT`), and `credentials` (a service account JSON string) or `credentialsPath`, falling back to Application Default Credentials, as the Vertex LLM services do. `apiKey` is rejected in this mode. ```ts const vertex = new GeminiSTTService({ vertexai: true, location: "eu", projectId: process.env.GOOGLE_CLOUD_PROJECT ?? "", // credentials: "", or credentialsPath; otherwise ADC. reportDetectedLanguage: false, settings: new GeminiSTTService.Settings({ languages: [Language.ES_ES] }), }); ``` Vertex's Live API rejects the `languageAuto` and `languageHints` fields, so in this mode the language codes are sent as `inputAudioTranscription.languageCodes`, and no codes means automatic detection. `languageAuto: false` has no Vertex equivalent and is ignored. ### Reported language [#reported-language] Each transcription frame carries the language the model reports. That value can be wrong, especially on short utterances. With `reportDetectedLanguage: false` (pipedog only), frames carry the configured language when exactly one is configured, and `null` otherwise. ### Changing languages during a session [#changing-languages-during-a-session] `setLanguageCodes(codes)` (pipedog only) sets the language codes to transcribe, for example when the bot switches language. The codes are sent as-is and take precedence over the `language` and `languages` settings until one of those is updated. An empty list means automatic detection. Pass a function (`() => string[]`) to have the codes read each time they are needed. ```ts // For example from a function the LLM calls when the conversation changes language. // The session reconnects with the new codes when the bot next starts speaking. await vertex.setLanguageCodes(["en-US"]); ``` The session does not reconnect at once. When the codes differ from the ones it was opened with, the service reconnects on the next `BotStartedSpeakingFrame`. If the user is mid-turn then, it waits until the turn ends, so no utterance is cut off. Audio that arrives while it reconnects is buffered and sent to the new session. `GoogleSTTService` has the same method. ### Event handlers [#event-handlers] | Event | Description | | --------------------- | ------------------------------------------------------------------------------------------------------ | | `on_connected` | Connected to the Gemini Live API. | | `on_disconnected` | Disconnected from the Gemini Live API. | | `on_connection_error` | The first connection failed, or three connection attempts failed in a row; the service stops retrying. | ### Notes [#notes] * Emits `InterimTranscriptionFrame`s as results arrive and a `TranscriptionFrame` for each finished utterance. * On a `VADUserStoppedSpeakingFrame`, the service sends an audio-stream-end signal so the model finalizes the utterance at once. Without an upstream VAD, the model finalizes on its own schedule. * Language hints and automatic detection are mutually exclusive. If both are set, hints win. * Any settings change reconnects the session. * The Live session is opened in the background when the `StartFrame` arrives, so the `StartFrame` (and the bot's first words) don't wait for it. Audio is buffered until the session is live. Upstream holds the `StartFrame` until the session is open; `awaitConnectionOnStart: true` (pipedog only) does the same. * Audio is sent at the pipeline's input sample rate. The model works best with 16 kHz mono PCM. ## GoogleSTTService [#googlesttservice] ### Configuration [#configuration-1] | Option | Type | Required | Description | |---|---|---|---| | `credentials` | `string \| null \| undefined` | no | JSON string containing Google Cloud service account credentials. | | `credentialsPath` | `string \| null \| undefined` | no | Path to service account credentials JSON file. | | `location` | `string \| undefined` | no | Google Cloud location (e.g., "global", "us-central1"). | | `settings` | `GoogleSTTSettings \| null \| undefined` | no | Runtime-updatable settings. | | `client` | `GoogleSTTClient \| null \| undefined` | no | A Speech V2 client to use instead of `@google-cloud/speech`. When given, credentials are not read; the project ID comes from `credentials` or from the client's `getProjectId()`. | | `clientOptions` | `Record \| null \| undefined` | no | Extra options for the `@google-cloud/speech` v2 `SpeechClient` (e.g. `apiEndpoint`, `port`, `sslCreds`, `fallback`), merged over the ones the service derives from `location` and the credentials. DIVERGENCE: pipedog-only. Upstream builds its client from `location` and the credentials alone. | | `speechModule` | `GoogleSpeechModule \| null \| undefined` | no | The `@google-cloud/speech` module (`import * as speech from "@google-cloud/speech"`), instead of importing it by name on first connect. A bundler (`bun build`) cannot see an import by name, so a bundled app imports the SDK itself and passes it here. DIVERGENCE: pipedog-only. | | `sharedClient` | `boolean \| undefined` | no | Use the process-wide Speech client for this location, credentials, `clientOptions` and `speechModule` instead of building one per service, so a server running one pipeline per call opens one gRPC channel and fetches tokens once. Pair it with `prewarm()` at start-up. Default false: each service has its own client (upstream's behavior), since one channel carries every concurrent stream of the process. Services never close clients. Ignored when `client` is given. DIVERGENCE: pipedog-only. | | `finalizeOnVadStop` | `boolean \| undefined` | no | Finalize an utterance as soon as VAD reports the user stopped speaking. Google only sends a final result once its own endpointing decides the utterance is over (about 1.5 s after speech ends). With this option the service half-closes the request stream on `VADUserStoppedSpeakingFrame`, so Google returns the final for the audio it has right away (about 0.6 s), and opens a new stream for the next audio, which waits in the queue meanwhile. The finals of the closing stream are pushed as one finalized `TranscriptionFrame`. Requires a VAD upstream. Default false. DIVERGENCE: pipedog-only. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `audioPassthrough` | `boolean \| undefined` | no | Whether to pass audio frames downstream after processing. Defaults to true. | | `sampleRate` | `number \| null \| undefined` | no | The sample rate for audio input. If null, will be determined from the start frame. | | `sttTtfbTimeout` | `number \| undefined` | no | Time in seconds to wait after VAD stop before reporting TTFB. This delay allows the final transcription to arrive. Defaults to 2.0. | | `ttfsP99Latency` | `number \| null \| undefined` | no | P99 latency from speech end to final transcript in seconds. Broadcast via STTMetadataFrame at pipeline start. | | `keepaliveTimeout` | `number \| null \| undefined` | no | Seconds of no audio before sending silence to keep the connection alive. null disables keepalive. | | `keepaliveInterval` | `number \| undefined` | no | Seconds between idle checks when keepalive is enabled. | ### Settings [#settings-1] Runtime-updatable settings, passed as `settings: new GoogleSTTService.Settings({...})`. | Setting | Type | Default | Description | | ---------------------------------- | -------------------------- | ------------------ | ------------------------------------------------------------------------------------------------------------------------ | | `model` | `string` | `"latest_long"` | Speech V2 model. | | `languages` | `Language[]` | `[Language.EN_US]` | Recognition languages. | | `languageCodes` | `string[] \| null` | `null` | Google language codes sent as-is, instead of `languages`, e.g. `["auto"]` for Chirp's language detection (pipedog only). | | `language` | `Language \| null` | `null` | Setting it replaces `languages` (and `languageCodes`) with that one language. | | `useSeparateRecognitionPerChannel` | `boolean` | `false` | Recognize each channel separately. | | `enableAutomaticPunctuation` | `boolean` | `true` | Add punctuation. | | `enableSpokenPunctuation` | `boolean` | `false` | Transcribe spoken punctuation. | | `enableSpokenEmojis` | `boolean` | `false` | Transcribe spoken emojis. | | `profanityFilter` | `boolean` | `false` | Filter profanity. | | `enableWordTimeOffsets` | `boolean` | `false` | Word timings. | | `enableWordConfidence` | `boolean` | `false` | Word confidence scores. | | `enableInterimResults` | `boolean` | `true` | Emit interim results. | | `enableVoiceActivityEvents` | `boolean` | `false` | Voice activity events. | | `adaptation` | `SpeechAdaptation \| null` | `null` | Phrase sets that bias recognition. | | `denoiserConfig` | `DenoiserConfig \| null` | `null` | Background-noise removal (Chirp 3 models only), e.g. `{ denoise_audio: true }`. | `adaptation` and `denoiserConfig` are plain objects. Keys may be snake_case, as Google documents them, or camelCase. A `phrase_sets` entry is a phrase set resource name or an inline phrase set. Referenced phrase sets must live in the same location as the service. ### Usage [#usage-1] ```ts title="google-stt.ts" import { STTUpdateSettingsFrame } from "@pipedog/core/frames/frames"; import { Language } from "@pipedog/core/transcriptions/language"; import { GoogleSTTService } from "@pipedog/google/stt"; const stt = new GoogleSTTService({ credentialsPath: process.env.GOOGLE_APPLICATION_CREDENTIALS ?? "", location: "global", settings: new GoogleSTTService.Settings({ model: "latest_long", languages: [Language.EN_US], enableInterimResults: true, }), }); ``` Speech adaptation: ```ts // Bias recognition with a stored phrase set and an inline one. const biased = new GoogleSTTService({ credentials: process.env.GOOGLE_CREDENTIALS_JSON ?? "", settings: new GoogleSTTService.Settings({ adaptation: { phrase_sets: [ "projects/my-project/locations/global/phraseSets/catalog", { phrases: [{ value: "pipedog", boost: 15.0 }] }, ], }, }), }); ``` Change settings mid-call with an `STTUpdateSettingsFrame`: ```ts // Changing settings reconnects the stream. const update = new STTUpdateSettingsFrame({ delta: new GoogleSTTService.Settings({ languages: [Language.ES_ES, Language.EN_US] }), }); ``` ### Event handlers [#event-handlers-1] | Event | Description | | ----------------- | ------------------------- | | `on_connected` | The service connected. | | `on_disconnected` | The service disconnected. | ### Notes [#notes-1] * Emits `InterimTranscriptionFrame`s (with `enableInterimResults`) and `TranscriptionFrame`s. * Google limits a streaming call to a few minutes. The service renews the stream every 4 minutes (`GoogleSTTService.STREAMING_LIMIT`, in ms) and carries the unfinished audio over. * A settings change reconnects the stream. * The `telephony` model rejects adaptation; the service logs a warning and transcribes without it. * With inline `credentials`, a missing `project_id` throws in the constructor. * With `languageCodes: ["auto"]`, transcripts carry the language Google detected. * A gRPC CANCELLED error (the call was cancelled on our side) ends that stream quietly instead of being reported as an error. ### Finalizing on VAD stop [#finalizing-on-vad-stop] Google sends a final result only once its own endpointing decides the utterance is over, about 1.5 s after speech ends. With `finalizeOnVadStop: true` (pipedog only), the service half-closes the request stream when a `VADUserStoppedSpeakingFrame` arrives, so Google returns the final for the audio it has right away (about 0.6 s), and opens a new stream for the next audio, which waits in the queue meanwhile. The finals of the closed stream are pushed as one finalized `TranscriptionFrame`, so the turn can end without waiting for the STT timeout. Put a VAD before the service. ```ts const stt = new GoogleSTTService({ location: "eu", finalizeOnVadStop: true, settings: new GoogleSTTService.Settings({ model: "chirp_3", languageCodes: ["auto"] }), }); ``` ### Switching languages mid-call [#switching-languages-mid-call] `setLanguageCodes(codes)` (pipedog only, the same API as `GeminiSTTService`) changes the `languageCodes` without cutting off the user: when the codes differ from the current ones, the stream reconnects on the next `BotStartedSpeakingFrame`, or once the user's turn ends if they are speaking over the bot. Pass a function to have the codes read again each time the bot starts speaking; it stays in effect until the `language`, `languages` or `languageCodes` setting is updated. ```ts await stt.setLanguageCodes(["es-ES", "en-US"]); // or follow the bot's language: await stt.setLanguageCodes(() => [currentLanguageCode()]); ``` ### Sharing a client and prewarming [#sharing-a-client-and-prewarming] With `sharedClient: true` (pipedog only), services with the same `location`, credentials, `clientOptions` and `speechModule` use one Speech client per process instead of one each, so a server running one pipeline per call opens one gRPC channel and fetches tokens once. `prewarm()` connects it ahead of time (it loads the SDK, resolves the project ID, opens the channel and fetches a token) and returns `false` with a warning instead of throwing if that fails; the first stream then connects as usual. ```ts const sttOptions = { location: "eu", sharedClient: true }; await new GoogleSTTService(sttOptions).prewarm(); // at server start-up // per call: const stt = new GoogleSTTService({ ...sttOptions, settings: new GoogleSTTService.Settings({ model: "chirp_3" }) }); ``` It is off by default, as upstream builds a client per service, because a shared client puts every concurrent stream of the process on one gRPC channel. Services never close their client, so one call ending doesn't affect the others. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/google/package.json`: ✅ means the compat smoke test passes on that runtime. `GeminiSTTService` uses `GoogleGenAI` from `@google/genai`, which opens the Live API over the global `WebSocket` and runs on Bun, Node, Deno and Cloudflare Workers. In Vertex AI mode it authenticates through google-auth-library, which only the SDK's Node build supports, so that mode runs on Bun, Node and Deno, not on Workers. The client is created on first connect, so a subclass that overrides `_createClient()` can read its own fields there. Cloud Speech-to-Text V2 streaming is gRPC only, so `GoogleSTTService` loads the Node SDK `@google-cloud/speech` lazily on first connect. That needs a Node-compatible runtime, not Workers. To use another transport, pass a `client` that implements `GoogleSTTClient` (`streamingRecognize()` and, optionally, `getProjectId()`). `clientOptions` (pipedog only) is merged into the SDK's `SpeechClient` options. Upstream resolves credentials and the project ID in the constructor. Here, with `credentialsPath` or ADC, they are resolved on first connect (in `setup()`), so "No valid credentials provided." and "Project ID not found in credentials" are raised there. Inline `credentials` are still checked in the constructor. The deprecated upstream `InputParams` class, the `params` option, `setLanguages()` and `updateOptions()` are not ported. `GoogleSTTSettings.languageCodes`, deprecated upstream, is kept for codes `Language` cannot express, such as `"auto"`. # Gradium > Realtime speech-to-text with GradiumSTTService, over Gradium's WebSocket API, with optional server-side turn detection. Source: https://polvallverdu.github.io/pipedog/docs/services/stt/gradium/ ## Overview [#overview] * **`GradiumSTTService`**: streams audio over a WebSocket to Gradium and pushes interim and final transcriptions. By default the pipeline's VAD closes each utterance by flushing the server. With `enableTurnDetection`, Gradium's end-pointing signal decides when turns start and end instead. Pipecat's documentation for the same service. `packages/gradium/src/stt.ts`. The speech-to-text WebSocket reference. ## Installation [#installation] ```bash bun add @pipedog/gradium ``` ```ts import { GradiumSTTService } from "@pipedog/gradium/stt"; ``` ## Prerequisites [#prerequisites] * A Gradium account and an API key from [Gradium](https://gradium.ai/). * Pass the key as `apiKey`. The examples read it from `GRADIUM_API_KEY`. ## GradiumSTTService [#gradiumsttservice] Connects to `wss://api.gradium.ai/api/speech/asr`, with the key in the `x-api-key` header. ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | Gradium API key for authentication. | | `apiEndpointBaseUrl` | `string \| undefined` | no | WebSocket endpoint URL. Defaults to "wss://api.gradium.ai/api/speech/asr". | | `encoding` | `string \| undefined` | no | Base audio encoding type. One of "pcm", "wav", or "opus". For PCM, the sample rate is appended automatically from the pipeline's audio_in_sample_rate (e.g., "pcm" becomes "pcm_16000"). Audio at a rate Gradium does not accept is resampled to one it does. Defaults to "pcm". | | `enableTurnDetection` | `boolean \| undefined` | no | Whether the server's end-pointing signal decides when user turns start and end, instead of the pipeline's VAD. Off by default. | | `settings` | `GradiumSTTSettings \| null \| undefined` | no | Runtime-updatable settings. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `reconnectBackoffMinWait` | `number \| undefined` | no | Minimum time, in seconds, to wait between reconnection attempts. | | `reconnectBackoffMaxWait` | `number \| undefined` | no | Maximum time, in seconds, to wait between reconnection attempts. | | `wsCloseTimeout` | `number \| undefined` | no | Maximum time, in seconds, to wait for the peer to acknowledge the websocket closing handshake before dropping the connection. Applied to connections opened through `_websocketConnect()`. Increase it for peers that need longer to complete a graceful close. | | `wsSendTimeout` | `number \| null \| undefined` | no | Maximum time, in seconds, a single send may wait on a peer that has stopped reading before the connection is dropped and the send throws `ConnectionClosedError`. Applied to connections opened through `_websocketConnect()`. Pass `null` to wait indefinitely. Defaults to `WS_SEND_TIMEOUT`. | | `audioPassthrough` | `boolean \| undefined` | no | Whether to pass audio frames downstream after processing. Defaults to true. | | `sampleRate` | `number \| null \| undefined` | no | The sample rate for audio input. If null, will be determined from the start frame. | | `sttTtfbTimeout` | `number \| undefined` | no | Time in seconds to wait after VAD stop before reporting TTFB. This delay allows the final transcription to arrive. Defaults to 2.0. | | `ttfsP99Latency` | `number \| null \| undefined` | no | P99 latency from speech end to final transcript in seconds. Broadcast via STTMetadataFrame at pipeline start. | | `keepaliveTimeout` | `number \| null \| undefined` | no | Seconds of no audio before sending silence to keep the connection alive. null disables keepalive. | | `keepaliveInterval` | `number \| undefined` | no | Seconds between idle checks when keepalive is enabled. | | `reconnectOnError` | `boolean \| undefined` | no | Whether to automatically reconnect on websocket errors. Defaults to true. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new GradiumSTTService.Settings({...})` (`GradiumSTTSettings`). The `eot*` and `postFlushCooldownFrames` fields tune turn detection and are `null` unless `enableTurnDetection` is `true`, which gives them the defaults below. | Setting | Type | Default | Description | | ------------------------- | ------------------- | ------------- | --------------------------------------------------------------------------------------------------------------- | | `language` | `Language \| "any"` | `Language.EN` | English, German, Spanish, French or Portuguese; `"any"` has Gradium detect it. | | `delayInFrames` | `number \| null` | `12` (960 ms) | Delay, in 80 ms frames, before text is generated. One of 7, 8, 10, 12, 14, 16, 20, 24, 36, 48. Lower is faster. | | `eotHorizonS` | `number \| null` | `3.0` | The end-pointing horizon that drives turn decisions. | | `eotThreshold` | `number \| null` | `0.5` | Inactivity probability at or above which an open turn ends. | | `postFlushCooldownFrames` | `number \| null` | `8` | Step messages to ignore after a flush before the next turn can start or end. | ### Usage [#usage] ```ts title="stt.ts" import { Language } from "@pipedog/core/transcriptions/language"; import { GradiumSTTService } from "@pipedog/gradium/stt"; const stt = new GradiumSTTService({ apiKey: process.env.GRADIUM_API_KEY ?? "", settings: new GradiumSTTService.Settings({ language: Language.EN }), }); ``` With the server's turn detection: ```ts const withTurns = new GradiumSTTService({ apiKey: process.env.GRADIUM_API_KEY ?? "", enableTurnDetection: true, settings: new GradiumSTTService.Settings({ eotHorizonS: 3.0, eotThreshold: 0.5 }), }); ``` ### Event handlers [#event-handlers] | Event | Description | | ----------------- | ------------------------------------------------------------ | | `on_connected` | The WebSocket connected. | | `on_disconnected` | The WebSocket closed. | | `on_turn_start` | The end-pointing signal opened a turn (turn detection on). | | `on_turn_end` | The end-pointing signal closed the turn (turn detection on). | ### Notes [#notes] * `encoding` is `"pcm"` (the sample rate is appended, for example `pcm_16000`), `"wav"` or `"opus"`. Audio at a rate Gradium doesn't accept is resampled. * With `enableTurnDetection`, local VAD frames are ignored and the service recommends `ExternalUserTurnStrategies`, which resolve its `ProposedUserStartedSpeakingFrame` and `ProposedUserStoppedSpeakingFrame` into user turns. | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/gradium/package.json`: ✅ means the compat smoke test passes on that runtime. ## Runtime notes [#runtime-notes] * The WebSocket goes through the core `websocketConnect`, with headers on the handshake. * The deprecated `InputParams`, `params` and `json_config` options are not ported. * Tested against local WebSocket servers, as upstream's tests do, not against Gradium itself. # Groq > Speech-to-text with GroqSTTService, Whisper models served by Groq. Source: https://polvallverdu.github.io/pipedog/docs/services/stt/groq/ ## Overview [#overview] * **`GroqSTTService`**: a `BaseWhisperSTTService` subclass. Each utterance (segmented by VAD) is sent to Groq's OpenAI-compatible transcription endpoint, and the text comes back as a `TranscriptionFrame`. Pipecat's documentation for the same service. `packages/groq/src/stt.ts`. Speech-to-text models and limits. ## Installation [#installation] ```bash bun add @pipedog/groq ``` ```ts import { GroqSTTService } from "@pipedog/groq/stt"; ``` ## Prerequisites [#prerequisites] * A Groq account and an API key from the [Groq console](https://console.groq.com/). * Pass the key as `apiKey`. It is required, and the service never falls back to `OPENAI_API_KEY`. The examples read it from `GROQ_API_KEY`. * A VAD in the pipeline, since this is a segmented service: it transcribes each utterance once the user stops speaking. ## GroqSTTService [#groqsttservice] Base URL `https://api.groq.com/openai/v1`. ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `apiKey` | `string` | yes | Groq API key. | | `baseUrl` | `string \| null \| undefined` | no | API base URL. Defaults to `"https://api.groq.com/openai/v1"`. | | `settings` | `GroqSTTSettings \| null \| undefined` | no | Runtime-updatable settings. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `audioPassthrough` | `boolean \| undefined` | no | Whether to pass audio frames downstream after processing. Defaults to true. | | `sampleRate` | `number \| null \| undefined` | no | The sample rate for audio input. If null, will be determined from the start frame. | | `sttTtfbTimeout` | `number \| undefined` | no | Time in seconds to wait after VAD stop before reporting TTFB. This delay allows the final transcription to arrive. Defaults to 2.0. | | `ttfsP99Latency` | `number \| null \| undefined` | no | P99 latency from speech end to final transcript in seconds. Broadcast via STTMetadataFrame at pipeline start. | | `keepaliveTimeout` | `number \| null \| undefined` | no | Seconds of no audio before sending silence to keep the connection alive. null disables keepalive. | | `keepaliveInterval` | `number \| undefined` | no | Seconds between idle checks when keepalive is enabled. | | `trailingSilenceSecs` | `number \| undefined` | no | Seconds of silence appended to each segment before it is transcribed, so the model hears the end of speech and can finish the last word. Set to 0 to disable. Defaults to 0.5. | | `httpClient` | `{ (input: RequestInfo \| URL, init?: RequestInit): Promise; (input: string \| URL \| Request, init?: RequestInit): Promise; } \| null \| undefined` | no | Custom `fetch` for API requests, e.g. one with a longer request timeout. Defaults to null, which lets the SDK use the global `fetch`. | | `includeProbMetrics` | `boolean \| undefined` | no | If true, enables probability metrics in API response. Each service implements this differently (see child classes). Defaults to false. | | `pushEmptyTranscripts` | `boolean \| undefined` | no | If true, allow empty `TranscriptionFrame` frames to be pushed downstream instead of discarding them. This is intended for situations where VAD fires even though the user did not speak. In these cases, it is useful to know that nothing was transcribed so that the agent can resume speaking, instead of waiting longer for a transcription. Defaults to false. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new GroqSTTService.Settings({...})` (`GroqSTTSettings`). Any setting you leave out keeps the default below. | Setting | Type | Default | Description | | ------------- | ------------------ | -------------------------- | ----------------------------------------------------------- | | `model` | `string` | `"whisper-large-v3-turbo"` | Whisper model. | | `language` | `Language \| null` | `Language.EN` | Language of the audio. | | `prompt` | `string \| null` | `null` | Text to guide the style, or to continue a previous segment. | | `temperature` | `number \| null` | `null` | Sampling temperature, 0 to 1. | ### Usage [#usage] ```ts title="stt.ts" import { STTUpdateSettingsFrame } from "@pipedog/core/frames/frames"; import { GroqSTTService } from "@pipedog/groq/stt"; const stt = new GroqSTTService({ apiKey: process.env.GROQ_API_KEY ?? "", settings: new GroqSTTService.Settings({ model: "whisper-large-v3-turbo" }), }); ``` Change settings mid-call with an `STTUpdateSettingsFrame`: ```ts // Queue it on the worker, or push it from a processor upstream of the service. const update = new STTUpdateSettingsFrame({ delta: new GroqSTTService.Settings({ prompt: "Pipecat, pipedog" }), }); ``` ### Notes [#notes] * Uploads each utterance as a WAV file, with `response_format: "json"` (`"verbose_json"` when `includeProbMetrics` is on, for per-segment probabilities). * `httpClient` replaces the SDK's `fetch`, for example to set a longer request timeout. * `pushEmptyTranscripts` pushes empty transcriptions instead of dropping them. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/groq/package.json`: ✅ means the compat smoke test passes on that runtime. * The client is the `openai` JS SDK (through `@pipedog/whisper`) over `fetch`. It works on Bun, Node, Deno and Cloudflare Workers. * The deprecated `model`, `language`, `prompt` and `temperature` constructor options are not ported. Use `settings`. * Tested against fakes, not against Groq itself. # Speech-to-text > Transcribe the user's speech into TranscriptionFrames. Source: https://polvallverdu.github.io/pipedog/docs/services/stt/ {/* Generated by scripts/generate-providers.ts from data/providers.ts. */} A speech-to-text service takes the user's audio frames and pushes `TranscriptionFrame`s (and `InterimTranscriptionFrame`s while the user speaks). | Provider | Classes | Package | Status | | --------------------------------------------------------------- | ----------------------------------------------------------------------------------- | ----------------------- | ----------- | | [AssemblyAI](https://polvallverdu.github.io/pipedog/docs/services/stt/assemblyai.md) | `AssemblyAISTTService`, `AssemblyAISyncSTTService` | `@pipedog/assemblyai` | Ported | | [Gladia](https://polvallverdu.github.io/pipedog/docs/services/stt/gladia.md) | `GladiaSTTService` | `@pipedog/gladia` | Ported | | [Speechmatics](https://polvallverdu.github.io/pipedog/docs/services/stt/speechmatics.md) | `SpeechmaticsSTTService` | `@pipedog/speechmatics` | Ported | | [Soniox](https://polvallverdu.github.io/pipedog/docs/services/stt/soniox.md) | `SonioxSTTService` | `@pipedog/soniox` | Ported | | [Deepgram](https://polvallverdu.github.io/pipedog/docs/services/stt/deepgram.md) | `DeepgramSTTService`, `DeepgramFluxSTTService` | `@pipedog/deepgram` | Ported | | [Cloudflare Workers AI](https://polvallverdu.github.io/pipedog/docs/services/stt/cloudflare.md) | `WorkersAIFluxSTTService`, `WorkersAINova3STTService`, `WorkersAIWhisperSTTService` | `@pipedog/cloudflare` | Ported | | [OpenAI](https://polvallverdu.github.io/pipedog/docs/services/stt/openai.md) | `OpenAISTTService`, `OpenAIRealtimeSTTService` | `@pipedog/openai` | Ported | | [Google](https://polvallverdu.github.io/pipedog/docs/services/stt/google.md) | `GoogleSTTService`, `GeminiSTTService` | `@pipedog/google` | Ported | | [Cartesia](https://polvallverdu.github.io/pipedog/docs/services/stt/cartesia.md) | `CartesiaSTTService`, `CartesiaTurnsSTTService` | `@pipedog/cartesia` | Ported | | [ElevenLabs](https://polvallverdu.github.io/pipedog/docs/services/stt/elevenlabs.md) | `ElevenLabsSTTService`, `ElevenLabsRealtimeSTTService` | `@pipedog/elevenlabs` | Ported | | [AWS Transcribe](https://polvallverdu.github.io/pipedog/docs/services/stt/aws.md) | `AWSTranscribeSTTService` | `@pipedog/aws` | Ported | | [AWS SageMaker (bidi client)](https://polvallverdu.github.io/pipedog/docs/services/stt/aws-sagemaker.md) | `SageMakerBidiClient` | `@pipedog/aws` | Ported | | [Azure Speech](https://polvallverdu.github.io/pipedog/docs/services/stt/azure.md) | `AzureSTTService` | `@pipedog/azure` | Ported | | [Whisper](https://polvallverdu.github.io/pipedog/docs/services/stt/whisper.md) | `WhisperSTTService`, `WhisperSTTServiceMLX`, `BaseWhisperSTTService` | `@pipedog/whisper` | Ported | | [Moonshine (local)](https://polvallverdu.github.io/pipedog/docs/services/stt/moonshine.md) | `MoonshineSTTService` | `@pipedog/moonshine` | Ported | | [FunASR SenseVoice (local)](https://polvallverdu.github.io/pipedog/docs/services/stt/funasr.md) | `FunASRSTTService` | `@pipedog/funasr` | Ported | | [Groq](https://polvallverdu.github.io/pipedog/docs/services/stt/groq.md) | `GroqSTTService` | `@pipedog/groq` | Ported | | [Mistral](https://polvallverdu.github.io/pipedog/docs/services/stt/mistral.md) | `MistralSTTService` | `@pipedog/mistral` | Ported | | [Together AI](https://polvallverdu.github.io/pipedog/docs/services/stt/together.md) | `TogetherSTTService` | `@pipedog/together` | Ported | | [xAI](https://polvallverdu.github.io/pipedog/docs/services/stt/xai.md) | `XAISTTService` | `@pipedog/xai` | Ported | | [Sarvam AI](https://polvallverdu.github.io/pipedog/docs/services/stt/sarvam.md) | `SarvamSTTService`, `SarvamRealtimeSTTService` | `@pipedog/sarvam` | Ported | | [Smallest AI](https://polvallverdu.github.io/pipedog/docs/services/stt/smallest.md) | `SmallestSTTService` | `@pipedog/smallest` | Ported | | [Gradium](https://polvallverdu.github.io/pipedog/docs/services/stt/gradium.md) | `GradiumSTTService` | `@pipedog/gradium` | Ported | | [Meta](https://polvallverdu.github.io/pipedog/docs/services/stt/meta.md) | `MetaSTTService` | `@pipedog/meta` | Ported | | [Zoom AI](https://polvallverdu.github.io/pipedog/docs/services/stt/zoom-ai.md) | `ZoomAISTTService` | `@pipedog/zoom-ai` | Ported | | [NVIDIA Riva](https://polvallverdu.github.io/pipedog/docs/services/stt/nvidia.md) | `NvidiaSTTService`, `NvidiaSegmentedSTTService` | | Coming soon | Pipecat's [speech-to-text reference](https://docs.pipecat.ai/api-reference/server/services/stt) lists every upstream provider. Those not in this table are on [Other upstream providers](https://polvallverdu.github.io/pipedog/docs/services/other-upstream.md). # Meta > Realtime speech-to-text with MetaSTTService, over Meta's Muse Voice WebSocket API. Source: https://polvallverdu.github.io/pipedog/docs/services/stt/meta/ ## Overview [#overview] * **`MetaSTTService`**: streams audio over a WebSocket to Meta's realtime transcription endpoint. Cumulative partials become `InterimTranscriptionFrame`s; the post-processed text of each segment becomes a `TranscriptionFrame`. Pipecat's documentation for the same service. `packages/meta/src/stt.ts`. The realtime voice API reference. ## Installation [#installation] ```bash bun add @pipedog/meta ``` ```ts import { MetaSTTService } from "@pipedog/meta/stt"; ``` ## Prerequisites [#prerequisites] * A Meta Model API key from [Meta's developer site](https://dev.meta.ai/). * Pass the key as `apiKey`. The examples read it from `META_API_KEY`. ## MetaSTTService [#metasttservice] Connects to `wss://api.meta.ai/v1/asr/realtime`. The key travels in the handshake message as a Bearer token, not in an HTTP header, and the session's configuration is fixed once the server accepts it, so a settings change reconnects. ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | Meta Model API key, sent as a Bearer token in the handshake. | | `url` | `string \| undefined` | no | WebSocket endpoint URL. Defaults to `wss://api.meta.ai/v1/asr/realtime`. | | `settings` | `MetaSTTSettings \| null \| undefined` | no | Runtime-updatable settings overriding defaults. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `reconnectBackoffMinWait` | `number \| undefined` | no | Minimum time, in seconds, to wait between reconnection attempts. | | `reconnectBackoffMaxWait` | `number \| undefined` | no | Maximum time, in seconds, to wait between reconnection attempts. | | `wsCloseTimeout` | `number \| undefined` | no | Maximum time, in seconds, to wait for the peer to acknowledge the websocket closing handshake before dropping the connection. Applied to connections opened through `_websocketConnect()`. Increase it for peers that need longer to complete a graceful close. | | `wsSendTimeout` | `number \| null \| undefined` | no | Maximum time, in seconds, a single send may wait on a peer that has stopped reading before the connection is dropped and the send throws `ConnectionClosedError`. Applied to connections opened through `_websocketConnect()`. Pass `null` to wait indefinitely. Defaults to `WS_SEND_TIMEOUT`. | | `audioPassthrough` | `boolean \| undefined` | no | Whether to pass audio frames downstream after processing. Defaults to true. | | `sampleRate` | `number \| null \| undefined` | no | The sample rate for audio input. If null, will be determined from the start frame. | | `sttTtfbTimeout` | `number \| undefined` | no | Time in seconds to wait after VAD stop before reporting TTFB. This delay allows the final transcription to arrive. Defaults to 2.0. | | `ttfsP99Latency` | `number \| null \| undefined` | no | P99 latency from speech end to final transcript in seconds. Broadcast via STTMetadataFrame at pipeline start. | | `keepaliveTimeout` | `number \| null \| undefined` | no | Seconds of no audio before sending silence to keep the connection alive. null disables keepalive. | | `keepaliveInterval` | `number \| undefined` | no | Seconds between idle checks when keepalive is enabled. | | `reconnectOnError` | `boolean \| undefined` | no | Whether to automatically reconnect on websocket errors. Defaults to true. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new MetaSTTService.Settings({...})` (`MetaSTTSettings`). Any setting you leave out keeps the default below. | Setting | Type | Default | Description | | ------------------- | -------------------- | ----------------------------- | --------------------------------------------------------------------------------------------------------- | | `model` | `string` | `"muse-voice-transcribe-1.0"` | Transcription model. | | `language` | `Language \| null` | `Language.EN` | Language to bias recognition toward. Leave it and `languageBias` unset to auto-detect. | | `languageBias` | `Language[] \| null` | `null` | Languages to bias toward, for speech that switches between them. Replaces `language` when set. | | `mode` | `string` | `"ENDPOINTING"` | `"ENDPOINTING"` (the model marks turn boundaries), `"PUSH_TO_TALK"` or `"DIARIZATION"`. | | `keywords` | `string[] \| null` | `null` | Terms to bias recognition toward. Biasing doesn't guarantee a spelling. | | `emitAudioProgress` | `boolean` | `false` | Have the server report progress for every processed audio chunk. | | `zdrOverride` | `boolean \| null` | `null` | `true` forces metadata-only logging; `false` allows content retention; `null` uses the account's setting. | ### Usage [#usage] ```ts title="stt.ts" import { Language } from "@pipedog/core/transcriptions/language"; import { MetaSTTService } from "@pipedog/meta/stt"; const stt = new MetaSTTService({ apiKey: process.env.META_API_KEY ?? "", settings: new MetaSTTService.Settings({ language: Language.EN, keywords: ["pipedog"] }), }); ``` ### Event handlers [#event-handlers] | Event | Description | | ----------------- | ------------------------ | | `on_connected` | The WebSocket connected. | | `on_disconnected` | The WebSocket closed. | ### Notes [#notes] * Audio is sent at 24000 Hz, or 16000 Hz for any other pipeline rate (resampled), the only two rates the API accepts. * The server's turn boundaries only segment transcripts. The VAD and turn strategies still decide when the user's turn ends. * Sessions are capped at 60 minutes. The server then closes the socket, and the service reconnects into a fresh one. | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/meta/package.json`: ✅ means the compat smoke test passes on that runtime. ## Runtime notes [#runtime-notes] * The WebSocket goes through the core `websocketConnect`. It works on Bun, Node, Deno and Cloudflare Workers. * Tested against local WebSocket servers, as upstream's tests do, not against Meta itself. # Mistral > Realtime speech-to-text with MistralSTTService, Mistral's Voxtral over WebSocket. Source: https://polvallverdu.github.io/pipedog/docs/services/stt/mistral/ ## Overview [#overview] * **`MistralSTTService`**: streams audio to Mistral's Voxtral realtime transcription over a WebSocket, pushes the text deltas as interim transcriptions, and the final transcript when the user stops speaking. Pipecat's documentation for the same service. `packages/mistral/src/stt.ts`. Voxtral and the audio APIs. ## Installation [#installation] ```bash bun add @pipedog/mistral ``` ```ts import { MistralSTTService } from "@pipedog/mistral/stt"; ``` ## Prerequisites [#prerequisites] * A Mistral account and an API key from [Mistral](https://docs.mistral.ai/api/). * Pass the key as `apiKey`. The examples read it from `MISTRAL_API_KEY`. * A VAD in the pipeline: the end of the user's speech flushes the audio buffer. ## MistralSTTService [#mistralsttservice] Connects to `wss://api.mistral.ai/v1/audio/transcriptions/realtime`. ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string \| null \| undefined` | no | Mistral API key for authentication. | | `baseUrl` | `string \| null \| undefined` | no | Custom API endpoint URL. Defaults to `"https://api.mistral.ai"`. | | `targetStreamingDelayMs` | `number \| null \| undefined` | no | Streaming delay for accuracy/latency tradeoff. Higher values may improve accuracy at the cost of latency. | | `settings` | `MistralSTTSettings \| null \| undefined` | no | Runtime-updatable settings. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `audioPassthrough` | `boolean \| undefined` | no | Whether to pass audio frames downstream after processing. Defaults to true. | | `sampleRate` | `number \| null \| undefined` | no | The sample rate for audio input. If null, will be determined from the start frame. | | `sttTtfbTimeout` | `number \| undefined` | no | Time in seconds to wait after VAD stop before reporting TTFB. This delay allows the final transcription to arrive. Defaults to 2.0. | | `ttfsP99Latency` | `number \| null \| undefined` | no | P99 latency from speech end to final transcript in seconds. Broadcast via STTMetadataFrame at pipeline start. | | `keepaliveTimeout` | `number \| null \| undefined` | no | Seconds of no audio before sending silence to keep the connection alive. null disables keepalive. | | `keepaliveInterval` | `number \| undefined` | no | Seconds between idle checks when keepalive is enabled. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new MistralSTTService.Settings({...})` (`MistralSTTSettings`). Any setting you leave out keeps the default below. | Setting | Type | Default | Description | | ---------- | ------------------ | ----------------------------------------- | ---------------------------------------------------------------------------------- | | `model` | `string` | `"voxtral-mini-transcribe-realtime-2602"` | Realtime transcription model. | | `language` | `Language \| null` | `null` | Not sent: Voxtral detects the language, which is reported on the final transcript. | ### Usage [#usage] ```ts title="stt.ts" import { STTUpdateSettingsFrame } from "@pipedog/core/frames/frames"; import { MistralSTTService } from "@pipedog/mistral/stt"; const stt = new MistralSTTService({ apiKey: process.env.MISTRAL_API_KEY ?? "", }); ``` Change settings mid-call with an `STTUpdateSettingsFrame`: ```ts // Queue it on the worker, or push it from a processor upstream of the service. const update = new STTUpdateSettingsFrame({ delta: new MistralSTTService.Settings({ model: "voxtral-mini-transcribe-realtime-2602" }), }); ``` ### Event handlers [#event-handlers] | Event | Description | | --------------------- | ------------------------ | | `on_connected` | The session was created. | | `on_disconnected` | The WebSocket closed. | | `on_connection_error` | The connection failed. | ### Notes [#notes] * `targetStreamingDelayMs` trades latency for accuracy, when set. It is sent in the `session.update` message (`target_streaming_delay_ms`); Mistral accepts 80 to 1200, or 2400, and recommends 480. * The service waits for `session.created` before it sends the session settings, as Mistral's SDK does. * Any settings change reconnects. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/mistral/package.json`: ✅ means the compat smoke test passes on that runtime. * The WebSocket goes through the core `websocketConnect` (upstream uses the `mistralai` SDK's realtime connection). It works on Bun, Node, Deno and Cloudflare Workers. * The protocol follows Mistral's SDK (`mistralai` 3.3.0 and `@mistralai/mistralai` 3.1.0, `extra/realtime`) and the [realtime transcription docs](https://docs.mistral.ai/studio/audio/speech-to-text/transcribe-in-realtime). `input_audio.flush` and `transcription.language` are in the SDK but not in the docs. * Tested against a local fake that follows that protocol, not against Mistral itself. # Moonshine > Local speech-to-text with MoonshineSTTService, running Moonshine's small, fast ASR models on the CPU. No API key. Source: https://polvallverdu.github.io/pipedog/docs/services/stt/moonshine/ ## Overview [#overview] * **`MoonshineSTTService`**: a segmented STT service that transcribes each speech segment, cut by the pipeline's VAD, with a [Moonshine](https://github.com/moonshine-ai/moonshine) model on your machine. Moonshine is a small, fast ASR family that runs on the CPU: no GPU, no account, no API key. * **`Transcriber`** (`@pipedog/moonshine/moonshine_voice`): the engine. Upstream wraps the `moonshine_voice` Python package (a C++ core on ONNX Runtime). This one runs the **same quantized `.ort` model files** through `@pipedog/onnx`, with the part of Moonshine's C++ core that `transcribe_without_streaming` runs ported to TypeScript. Moonshine is the default transcriber of upstream's keyless local defaults for [evals](https://polvallverdu.github.io/pipedog/docs/evals.md): it transcribes the bot's speech in audio scenarios. Pipecat's documentation for the same service. `packages/moonshine/src/stt.ts` and the engine, `moonshine_voice.ts`. The models, languages and licenses. ## Installation [#installation] ```bash bun add @pipedog/moonshine onnxruntime-node ``` `onnxruntime-node` is an optional peer of `@pipedog/onnx`. `onnxruntime-web@1.30.0` (WebAssembly) loads the same `.ort` files and gives the same transcripts, about 3x slower. ```ts import { Model, MoonshineSTTService } from "@pipedog/moonshine/stt"; ``` ## What you need [#what-you-need] | | | | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Models | Moonshine's quantized ORT-format models from `https://download.moonshine.ai/model/...`, the files and versions `moonshine_voice` 0.1.5 uses. Each download is checked against Moonshine's published size and CRC32C | | Sizes | English: `tiny` 44 MB, `tiny-streaming` 45 MB, `base` 141 MB, `small-streaming` 142 MB (the default), `medium-streaming` 269 MB. Other languages: `tiny-streaming` 32 MB, `small-streaming` 122 MB, `base` 65-141 MB, `tiny` (ja, ko) 72 MB | | Download | Automatic, once, when the pipeline starts (or on the first transcription) | | Cache | `$PIPEDOG_CACHE_DIR/moonshine`, else `$XDG_CACHE_HOME/pipedog/moonshine`, else `~/.cache/pipedog/moonshine`, or the `cacheRoot` option | | VAD | Silero VAD, the `silero_vad.onnx` shipped with `@pipedog/audio-silero` (a dependency), for the segmentation Moonshine applies inside each batch | | Input | 16 kHz, 16-bit mono PCM, and a VAD in the pipeline | Files go in `/download.moonshine.ai/model//...`, the layout `moonshine_voice` uses, so you can copy a `moonshine_voice` cache over. To use files you already have, pass `cacheRoot`: ```ts // Models already on disk, in /download.moonshine.ai/model//... // (the layout moonshine_voice uses, so a moonshine_voice cache can be copied over). const fromDisk = new MoonshineSTTService({ cacheRoot: "/models/moonshine", settings: new MoonshineSTTService.Settings({ model: Model.TINY }), }); ``` Or give the engine any model directory directly (see [Usage](#usage)), or the model files in memory with `transcriberOptions.files`. `downloadModel(language, arch?)` downloads a model ahead of time. ## Runtimes [#runtimes] | Runtime | Works | What it needs | | ------------------ | ----- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Bun | ✅ yes | onnxruntime-node (or onnxruntime-web) and a filesystem for the model cache: models download on first use from download.moonshine.ai (tiny 44 MB; the default, small-streaming, 142 MB). | | Node.js | ✅ yes | onnxruntime-node (or onnxruntime-web) and a filesystem for the model cache: models download on first use from download.moonshine.ai (tiny 44 MB; the default, small-streaming, 142 MB). | | Deno | ✅ yes | onnxruntime-node (or onnxruntime-web) and a filesystem for the model cache: models download on first use from download.moonshine.ai (tiny 44 MB; the default, small-streaming, 142 MB). | | Cloudflare Workers | ❌ no | onnxruntime-node (or onnxruntime-web) and a filesystem for the model cache: models download on first use from download.moonshine.ai (tiny 44 MB; the default, small-streaming, 142 MB). Not supported: the models (32-269 MB) and their inference sessions exceed Workers' memory and bundle limits. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/moonshine/package.json`: ✅ means the compat smoke test passes on that runtime. Cloudflare Workers is not supported: the models (32-269 MB) and their inference sessions don't fit a Worker's 128 MB of memory or its 10 MB bundle limit. For the 11 s JFK clip (4 segments), on an Apple M4 Pro with Bun and `onnxruntime-node`: `tiny` 0.1 s, `base` 0.12 s, `tiny-streaming` 0.2 s, `small-streaming` 0.35 s. ## MoonshineSTTService [#moonshinesttservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `settings` | `MoonshineSTTSettings \| null \| undefined` | no | Runtime-updatable settings (`model`, `language`). | | `transcriberOptions` | `TranscriberOptions \| undefined` | no | `Transcriber` options (pipedog only). | | `cacheRoot` | `string \| undefined` | no | Where models are downloaded (pipedog only). | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `audioPassthrough` | `boolean \| undefined` | no | Whether to pass audio frames downstream after processing. Defaults to true. | | `sampleRate` | `number \| null \| undefined` | no | The sample rate for audio input. If null, will be determined from the start frame. | | `sttTtfbTimeout` | `number \| undefined` | no | Time in seconds to wait after VAD stop before reporting TTFB. This delay allows the final transcription to arrive. Defaults to 2.0. | | `ttfsP99Latency` | `number \| null \| undefined` | no | P99 latency from speech end to final transcript in seconds. Broadcast via STTMetadataFrame at pipeline start. | | `keepaliveTimeout` | `number \| null \| undefined` | no | Seconds of no audio before sending silence to keep the connection alive. null disables keepalive. | | `keepaliveInterval` | `number \| undefined` | no | Seconds between idle checks when keepalive is enabled. | | `trailingSilenceSecs` | `number \| undefined` | no | Seconds of silence appended to each segment before it is transcribed, so the model hears the end of speech and can finish the last word. Set to 0 to disable. Defaults to 0.5. | `transcriberOptions` (pipedog only) takes the `Transcriber`'s options: `backend` (any `OnnxBackend`), `fetch` for downloads, `files` (model files in memory), `vadModel`, `shareSessions`, and Moonshine's `maxTokensPerSecond` (6.5), `vadThreshold` (0.5), `vadWindowDuration` (0.5 s), `vadHopSize` (512), `vadLookBehindSampleCount` (8192) and `vadMaxSegmentDuration` (15 s). ### Settings [#settings] Runtime-updatable settings, passed as `settings: new MoonshineSTTService.Settings({...})` (`MoonshineSTTSettings`). Any setting you leave out keeps the default below. A change reloads the model, and keeps the old one if the new one fails to load. | Setting | Type | Default | Description | | ---------- | ----------------- | ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `model` | `Model \| string` | `Model.SMALL_STREAMING` | Architecture: `TINY`, `BASE`, `TINY_STREAMING`, `BASE_STREAMING`, `SMALL_STREAMING` or `MEDIUM_STREAMING` (or the string, such as `"small-streaming"`). | | `language` | `Language` | `Language.EN` | Arabic, Chinese, English, German, Japanese, Korean, Spanish, Tagalog, Ukrainian or Vietnamese. Regional variants resolve to the base code. An architecture a language doesn't publish falls back to that language's default, as upstream. | ### Usage [#usage] ```ts title="stt.ts" import { STTUpdateSettingsFrame } from "@pipedog/core/frames/frames"; import { Language } from "@pipedog/core/transcriptions/language"; import { Model, MoonshineSTTService } from "@pipedog/moonshine/stt"; // Downloads small-streaming (142 MB) from download.moonshine.ai on first start. const stt = new MoonshineSTTService({ settings: new MoonshineSTTService.Settings({ model: Model.SMALL_STREAMING, language: Language.EN, }), }); ``` Change the language or model mid-call with an `STTUpdateSettingsFrame`: ```ts // Queue it on the worker, or push it from a processor upstream of the service. // A model or language change reloads the model. const update = new STTUpdateSettingsFrame({ delta: new MoonshineSTTService.Settings({ language: Language.ES }), }); ``` The engine works on its own, with any model directory: ```ts import { ModelArch, Transcriber } from "@pipedog/moonshine/moonshine_voice"; const transcriber = new Transcriber("/models/tiny-en", ModelArch.TINY); const samples = new Float32Array(16000); // mono audio in [-1, 1] const transcript = await transcriber.transcribeWithoutStreaming(samples, 16000); console.log(transcript.lines.map((line) => line.text).join(" ")); ``` ### Notes [#notes] * Emits one `TranscriptionFrame` per segment. There are no interim results. * Every model, streaming ones included, is run in one batch per segment (`transcribeWithoutStreaming`). * The constructor rejects a missing model or language, a model that names no architecture, and a language Moonshine publishes no model for. ## Differences from upstream [#differences-from-upstream] * The model is resolved in the constructor but downloaded and loaded in `start()` (or on the first transcription). Upstream downloads and loads it in the constructor. * Upstream also prefetches the optional English spelling model, which `transcribe_without_streaming` never uses. It is not downloaded. * Only what `transcribe_without_streaming` needs is ported: no streaming API (`create_stream`), word timestamps, speaker diarization, key-term biasing or spelling mode. * Inference runs on onnxruntime 1.30, where Moonshine bundles 1.23.2: the same graphs, so the same transcripts up to floating-point rounding. * `cacheRoot` and `transcriberOptions` are pipedog options. ## Licenses [#licenses] * `@pipedog/moonshine`: BSD 2-Clause. The engine ports parts of Moonshine (MIT). * Models: MIT in every language and size, except the non-streaming models (`tiny`, `base`) for languages other than English, which are under the non-commercial [Moonshine Community License](https://www.moonshine.ai/license). They are downloaded at runtime, not shipped. * Silero VAD (through `@pipedog/audio-silero`): MIT. # NVIDIA Riva > Speech recognition from NVIDIA Riva. Not ported to pipedog yet. Source: https://polvallverdu.github.io/pipedog/docs/services/stt/nvidia/ {/* Generated by scripts/generate-providers.ts from data/providers.ts. */} NVIDIA Riva isn't ported to pipedog yet. It is available in Pipecat (Python), and its pipedog port will keep the same class names and options. [Ask for it, or upvote the request](https://github.com/polvallverdu/pipedog/issues?q=is%3Aissue+NVIDIA+Riva). ## What it is [#what-it-is] Speech recognition from NVIDIA Riva. Upstream, it's these classes: `NvidiaSTTService`, `NvidiaSegmentedSTTService`, in [`services/nvidia/stt.py`](https://github.com/pipecat-ai/pipecat/blob/main/src/pipecat/services/nvidia/stt.py). Its pipedog package will be `@pipedog/nvidia`. * [Pipecat's NVIDIA Riva page](https://docs.pipecat.ai/api-reference/server/services/stt/nvidia) ## Get it ported [#get-it-ported] * [Search the issues](https://github.com/polvallverdu/pipedog/issues?q=is%3Aissue+NVIDIA+Riva) for a request, and upvote it with a reaction. * If there's none, [open a port request](https://github.com/polvallverdu/pipedog/issues/new?template=feature_request.yml\&title=Port+NVIDIA+Riva+%28NvidiaSTTService%2C+NvidiaSegmentedSTTService%29\&kind=Port+an+upstream+module\&upstream=src%2Fpipecat%2Fservices%2Fnvidia%2Fstt.py). * Or port it yourself: the [contributing guide](https://polvallverdu.github.io/pipedog/docs/contributing.md) explains how modules are ported, test by test. The [porting status](https://polvallverdu.github.io/pipedog/docs/contributing/status.md) lists what's done. Until then, see what is ported in [Speech-to-text](https://polvallverdu.github.io/pipedog/docs/services/stt.md). # OpenAI > Speech-to-text with OpenAISTTService (segmented, Audio API) and OpenAIRealtimeSTTService (streaming, Realtime API, optional server VAD). Source: https://polvallverdu.github.io/pipedog/docs/services/stt/openai/ ## Overview [#overview] pipedog ports both of Pipecat's OpenAI speech-to-text services: * **`OpenAISTTService`**: transcribes each speech segment with the Audio API (`audio.transcriptions`). It waits for the user to stop speaking, then sends the segment as a WAV file. Works with the GPT transcription models and Whisper (`whisper-1`). * **`OpenAIRealtimeSTTService`**: streams audio over a WebSocket to the Realtime API in transcription-only mode, with interim results. Turns come from a local VAD (the default) or from OpenAI's server-side VAD. Pipecat's documentation for the same services. `packages/openai/src/stt.ts`. Transcription models and languages. ## Installation [#installation] ```bash bun add @pipedog/openai ``` ```ts import { OpenAIRealtimeSTTService, OpenAISTTService } from "@pipedog/openai/stt"; ``` `OpenAISTTService` builds on `BaseWhisperSTTService` from `@pipedog/whisper`, which `@pipedog/openai` depends on. ## Prerequisites [#prerequisites] * An OpenAI account and an API key from the [OpenAI dashboard](https://platform.openai.com/api-keys). * Pass the key as `apiKey`. `OpenAISTTService` falls back to `OPENAI_API_KEY` when you leave it out; `OpenAIRealtimeSTTService` requires it. The examples read it from `OPENAI_API_KEY`. ## OpenAISTTService [#openaisttservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `settings` | `OpenAISTTSettings \| null \| undefined` | no | Runtime-updatable settings. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `audioPassthrough` | `boolean \| undefined` | no | Whether to pass audio frames downstream after processing. Defaults to true. | | `sampleRate` | `number \| null \| undefined` | no | The sample rate for audio input. If null, will be determined from the start frame. | | `sttTtfbTimeout` | `number \| undefined` | no | Time in seconds to wait after VAD stop before reporting TTFB. This delay allows the final transcription to arrive. Defaults to 2.0. | | `ttfsP99Latency` | `number \| null \| undefined` | no | P99 latency from speech end to final transcript in seconds. Broadcast via STTMetadataFrame at pipeline start. | | `keepaliveTimeout` | `number \| null \| undefined` | no | Seconds of no audio before sending silence to keep the connection alive. null disables keepalive. | | `keepaliveInterval` | `number \| undefined` | no | Seconds between idle checks when keepalive is enabled. | | `trailingSilenceSecs` | `number \| undefined` | no | Seconds of silence appended to each segment before it is transcribed, so the model hears the end of speech and can finish the last word. Set to 0 to disable. Defaults to 0.5. | | `apiKey` | `string \| null \| undefined` | no | Service API key. Defaults to null. | | `baseUrl` | `string \| null \| undefined` | no | Service API base URL. Defaults to null. | | `httpClient` | `((input: RequestInfo \| URL, init?: RequestInit) => Promise) \| null \| undefined` | no | Custom `fetch` for API requests, e.g. one with a longer request timeout. Defaults to null, which lets the SDK use the global `fetch`. | | `includeProbMetrics` | `boolean \| undefined` | no | If true, enables probability metrics in API response. Each service implements this differently (see child classes). Defaults to false. | | `pushEmptyTranscripts` | `boolean \| undefined` | no | If true, allow empty `TranscriptionFrame` frames to be pushed downstream instead of discarding them. This is intended for situations where VAD fires even though the user did not speak. In these cases, it is useful to know that nothing was transcribed so that the agent can resume speaking, instead of waiting longer for a transcription. Defaults to false. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new OpenAISTTService.Settings({...})` (`OpenAISTTSettings`). Any setting you leave out keeps the default below. | Setting | Type | Default | Description | | ------------- | ------------------ | ------------------ | ------------------------------------------------------------------------- | | `model` | `string` | `"gpt-transcribe"` | Transcription model, for example `"whisper-1"`. | | `language` | `Language` | `Language.EN` | Transcription language. | | `prompt` | `string \| null` | `null` | Text that guides the style, or continues a previous segment. | | `temperature` | `number \| null` | `null` | Sampling temperature, 0 to 1. | | `keywords` | `string[] \| null` | `null` | Words or phrases to guide transcription. Supported by `"gpt-transcribe"`. | ### Usage [#usage] ```ts title="stt.ts" import { Language } from "@pipedog/core/transcriptions/language"; import { OpenAISTTService } from "@pipedog/openai/stt"; const stt = new OpenAISTTService({ apiKey: process.env.OPENAI_API_KEY ?? "", settings: new OpenAISTTService.Settings({ model: "gpt-transcribe", language: Language.EN, keywords: ["pipedog", "Pipecat"], }), }); ``` Whisper, keeping empty transcripts: ```ts const whisper = new OpenAISTTService({ apiKey: process.env.OPENAI_API_KEY ?? "", settings: new OpenAISTTService.Settings({ model: "whisper-1", temperature: 0 }), pushEmptyTranscripts: true, }); ``` ### Notes [#notes] * A segmented service: it needs a VAD (in the user aggregator) to mark where each segment ends, then pushes one `TranscriptionFrame` per segment. There are no interim results. * `pushEmptyTranscripts: true` pushes a `TranscriptionFrame` even when nothing was transcribed, so the bot can go on when the VAD fired on noise. * With `includeProbMetrics: true`, GPT transcription models report per-token logprobs and Whisper models per-segment logprobs. Diarization models report none, with a warning. * `httpClient` replaces the SDK's `fetch`, for example to set a longer request timeout. ## OpenAIRealtimeSTTService [#openairealtimesttservice] ### Configuration [#configuration-1] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | OpenAI API key for authentication. | | `baseUrl` | `string \| undefined` | no | WebSocket base URL for the Realtime API. Defaults to "wss://api.openai.com/v1/realtime". | | `turnDetection` | `false \| Dict \| null \| undefined` | no | Server-side VAD configuration. Defaults to `false` (disabled), which relies on a local VAD processor in the pipeline. Pass an object to enable it — at minimum `{ type: "server_vad" }`, optionally with custom settings (e.g. `{ type: "server_vad", threshold: 0.5 }`). Requires a model that supports turn detection. `null` omits the field entirely, leaving the session's own default in place. | | `shouldInterrupt` | `boolean \| undefined` | no | Whether to interrupt bot output when speech is detected by server-side VAD. Only applies when turn detection is enabled. Passed along to the user turn strategies this service recommends, which own the interruption; a user-supplied `userTurnStrategies` overrides the recommendation and this setting with it. Defaults to true. | | `settings` | `OpenAIRealtimeSTTSettings \| null \| undefined` | no | Runtime-updatable settings. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `reconnectBackoffMinWait` | `number \| undefined` | no | Minimum time, in seconds, to wait between reconnection attempts. | | `reconnectBackoffMaxWait` | `number \| undefined` | no | Maximum time, in seconds, to wait between reconnection attempts. | | `wsCloseTimeout` | `number \| undefined` | no | Maximum time, in seconds, to wait for the peer to acknowledge the websocket closing handshake before dropping the connection. Applied to connections opened through `_websocketConnect()`. Increase it for peers that need longer to complete a graceful close. | | `wsSendTimeout` | `number \| null \| undefined` | no | Maximum time, in seconds, a single send may wait on a peer that has stopped reading before the connection is dropped and the send throws `ConnectionClosedError`. Applied to connections opened through `_websocketConnect()`. Pass `null` to wait indefinitely. Defaults to `WS_SEND_TIMEOUT`. | | `audioPassthrough` | `boolean \| undefined` | no | Whether to pass audio frames downstream after processing. Defaults to true. | | `sampleRate` | `number \| null \| undefined` | no | The sample rate for audio input. If null, will be determined from the start frame. | | `sttTtfbTimeout` | `number \| undefined` | no | Time in seconds to wait after VAD stop before reporting TTFB. This delay allows the final transcription to arrive. Defaults to 2.0. | | `ttfsP99Latency` | `number \| null \| undefined` | no | P99 latency from speech end to final transcript in seconds. Broadcast via STTMetadataFrame at pipeline start. | | `keepaliveTimeout` | `number \| null \| undefined` | no | Seconds of no audio before sending silence to keep the connection alive. null disables keepalive. | | `keepaliveInterval` | `number \| undefined` | no | Seconds between idle checks when keepalive is enabled. | | `reconnectOnError` | `boolean \| undefined` | no | Whether to automatically reconnect on websocket errors. Defaults to true. | ### Settings [#settings-1] Runtime-updatable settings, passed as `settings: new OpenAIRealtimeSTTService.Settings({...})` (`OpenAIRealtimeSTTSettings`). | Setting | Type | Default | Description | | ---------------- | ------------------------------------------------------------- | ------------------------ | ------------------------------------------------------------------------------------------------------------------------ | | `model` | `string` | `"gpt-realtime-whisper"` | Transcription model. | | `language` | `Language` | `Language.EN` | Transcription language, sent as its ISO-639-1 code. | | `prompt` | `string \| null` | `null` | Text that guides the style. Not supported by `"gpt-realtime-whisper"`, so it is dropped (with a warning) for that model. | | `noiseReduction` | `"near_field" \| "far_field" \| null` | `null` | Noise reduction for close or distant microphones. `null` turns it off. | | `delay` | `"minimal" \| "low" \| "medium" \| "high" \| "xhigh" \| null` | `null` | How long the model waits before emitting text, trading latency for accuracy. Only `"gpt-realtime-whisper"`. | ### Usage [#usage-1] ```ts title="realtime.ts" import { Language } from "@pipedog/core/transcriptions/language"; import { OpenAIRealtimeSTTService } from "@pipedog/openai/stt"; // Local VAD (the default): a VAD in the user aggregator ends each segment. const stt = new OpenAIRealtimeSTTService({ apiKey: process.env.OPENAI_API_KEY ?? "", settings: new OpenAIRealtimeSTTService.Settings({ model: "gpt-realtime-whisper", language: Language.EN, noiseReduction: "near_field", delay: "low", }), }); ``` Server-side VAD. OpenAI detects the turns, so the pipeline needs no local VAD: ```ts // OpenAI detects the turns. Use a model with turn detection, and no local VAD. const serverVad = new OpenAIRealtimeSTTService({ apiKey: process.env.OPENAI_API_KEY ?? "", turnDetection: { type: "server_vad", threshold: 0.5 }, settings: new OpenAIRealtimeSTTService.Settings({ model: "gpt-transcribe" }), }); ``` Settings sent mid-call with an `STTUpdateSettingsFrame` are applied with a `session.update` on the open session. ### Event handlers [#event-handlers] | Event | Description | | --------------------- | ------------------------ | | `on_connected` | The WebSocket connected. | | `on_disconnected` | The WebSocket closed. | | `on_connection_error` | The connection failed. | ### Notes [#notes-1] * Emits `InterimTranscriptionFrame`s as text arrives and a `TranscriptionFrame` for each completed segment. * `turnDetection` defaults to `false`: a local VAD drives turns, and each `VADUserStoppedSpeakingFrame` commits the audio buffer so the server transcribes the segment. * With `turnDetection: { type: "server_vad" }`, the service broadcasts `ProposedUserStartedSpeakingFrame` and `ProposedUserStoppedSpeakingFrame` from the server's speech events, and asks the user aggregator for `ExternalUserTurnStrategies` (unless you pass your own `userTurnStrategies`). `shouldInterrupt` (default `true`) decides whether those turns interrupt the bot. Server VAD needs a model with turn detection: `gpt-transcribe` has it, `gpt-realtime-whisper` does not. See [turn detection](https://polvallverdu.github.io/pipedog/docs/concepts/turn-detection.md). * Audio is sent as 24 kHz 16-bit mono PCM. Other input rates are resampled. * The [Cloudflare Workers example](https://polvallverdu.github.io/pipedog/docs/deployment/cloudflare-workers.md) uses this service with server VAD, so the Worker loads no local VAD model. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/openai/package.json`: ✅ means the compat smoke test passes on that runtime. * `OpenAISTTService` uses the `openai` JS SDK over `fetch`. It works on Bun, Node, Deno and Cloudflare Workers. * `OpenAIRealtimeSTTService` sends the key as an `Authorization` header on the WebSocket handshake. Bun, Node and Deno send it from the global `WebSocket`; on Cloudflare Workers core opens the socket through `fetch`. * The deprecated `model`, `language`, `prompt`, `temperature` and `noiseReduction` constructor options are not ported. Use `settings`. # Sarvam AI > Speech-to-text for Indian languages with SarvamSTTService and SarvamRealtimeSTTService, over Sarvam's WebSocket APIs. Source: https://polvallverdu.github.io/pipedog/docs/services/stt/sarvam/ ## Overview [#overview] * **`SarvamSTTService`**: streams audio to Sarvam's speech-to-text WebSocket (`saaras:v4` by default, or `saaras:v3`; see `MODEL_CONFIGS`). Sarvam's VAD signals can drive the turn. * **`SarvamRealtimeSTTService`**: Sarvam's realtime endpoint (`saaras:v3-realtime` or `saaras:v4`), with the server or the pipeline marking turn boundaries, and reconnects. Pipecat's documentation for the same service. `packages/sarvam/src/stt.ts`. The speech-to-text API reference. ## Installation [#installation] ```bash bun add @pipedog/sarvam ``` ```ts import { SarvamRealtimeSTTService, SarvamSTTService } from "@pipedog/sarvam/stt"; ``` ## Prerequisites [#prerequisites] * A Sarvam account and an API key from the [Sarvam dashboard](https://dashboard.sarvam.ai/). * Pass the key as `apiKey`. The examples read it from `SARVAM_API_KEY`. ## SarvamSTTService [#sarvamsttservice] Connects to `wss://api.sarvam.ai/speech-to-text/ws`, with the key in the `Api-Subscription-Key` header. ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | Sarvam API key for authentication. | | `mode` | `SarvamMode \| null \| undefined` | no | Mode of operation. Options: transcribe, translate, verbatim, translit, codemix. Only applicable to models that support it. Defaults to the model's default mode. | | `sampleRate` | `number \| null \| undefined` | no | Audio sample rate. Defaults to the pipeline's input rate if not specified. | | `inputAudioCodec` | `string \| undefined` | no | Audio codec/format of the input file. Defaults to "wav". | | `keyterms` | `string[] \| null \| undefined` | no | Connection-time recognition hints for saaras:v4. At most 50 terms of 64 characters each; put each phrase in a separate item. | | `baseUrl` | `string \| undefined` | no | WebSocket endpoint. Defaults to "wss://api.sarvam.ai/speech-to-text/ws". | | `settings` | `SarvamSTTSettings \| null \| undefined` | no | Runtime-updatable settings. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `audioPassthrough` | `boolean \| undefined` | no | Whether to pass audio frames downstream after processing. Defaults to true. | | `sttTtfbTimeout` | `number \| undefined` | no | Time in seconds to wait after VAD stop before reporting TTFB. This delay allows the final transcription to arrive. Defaults to 2.0. | | `ttfsP99Latency` | `number \| null \| undefined` | no | P99 latency from speech end to final transcript in seconds. Broadcast via STTMetadataFrame at pipeline start. | | `keepaliveTimeout` | `number \| null \| undefined` | no | Seconds of no audio before sending silence to keep the connection alive. null disables keepalive. | | `keepaliveInterval` | `number \| undefined` | no | Seconds between idle checks when keepalive is enabled. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new SarvamSTTService.Settings({...})` (`SarvamSTTSettings`). They are the base STT settings (`model`, `language`) plus the VAD tuning fields below, all `null` (the server's default) unless you set them. | Setting | Type | Default | Description | | ---------------------------- | ------------------ | ------------- | ----------------------------------------------------------------------------- | | `model` | `string` | `"saaras:v4"` | Model. Its entry in `MODEL_CONFIGS` decides the modes and options that apply. | | `language` | `Language \| null` | `null` | Language of the audio. `null` lets models that can detect it do so. | | `vadSignals` | `boolean \| null` | `null` | Have Sarvam return VAD signals. | | `highVadSensitivity` | `boolean \| null` | `null` | High VAD sensitivity. | | `positiveSpeechThreshold` | `number \| null` | `null` | Probability (0 to 1) above which a frame counts as speech. | | `negativeSpeechThreshold` | `number \| null` | `null` | Probability (0 to 1) below which a frame counts as silence. | | `minSpeechFrames` | `number \| null` | `null` | Consecutive speech frames that start a segment. | | `firstTurnMinSpeechFrames` | `number \| null` | `null` | The same, for the first turn. | | `negativeFramesCount` | `number \| null` | `null` | Silence frames within the window that end a segment. | | `negativeFramesWindow` | `number \| null` | `null` | Window, in frames, for counting silence frames. | | `startSpeechVolumeThreshold` | `number \| null` | `null` | Volume (dB) below which audio is too quiet to be speech. | | `interruptMinSpeechFrames` | `number \| null` | `null` | Speech frames needed to register a barge-in. | | `preSpeechPadFrames` | `number \| null` | `null` | Frames to prepend before the detected start of speech. | | `numInitialIgnoredFrames` | `number \| null` | `null` | Leading frames to skip at connection start. | ### Usage [#usage] ```ts title="stt.ts" import { Language } from "@pipedog/core/transcriptions/language"; import { SarvamSTTService } from "@pipedog/sarvam/stt"; const stt = new SarvamSTTService({ apiKey: process.env.SARVAM_API_KEY ?? "", settings: new SarvamSTTService.Settings({ language: Language.HI_IN }), }); ``` ### Event handlers [#event-handlers] | Event | Description | | --------------------- | ------------------------ | | `on_connected` | The WebSocket connected. | | `on_disconnected` | The WebSocket closed. | | `on_connection_error` | The connection failed. | ### Notes [#notes] * `mode` is one of `transcribe`, `translate`, `verbatim`, `translit` and `codemix`, for the models that support it; it defaults to the model's own default. * `keyterms` (at most 50 terms of 64 characters each) are connection-time recognition hints for `saaras:v4`. * Only 8000 and 16000 Hz audio is accepted. * Settings that go in the connection query string reconnect the socket when they change. ## SarvamRealtimeSTTService [#sarvamrealtimesttservice] Connects to Sarvam's realtime speech-to-text WebSocket and maps its VAD and transcript events to frames. A dropped socket is reopened with backoff unless `reconnectOnError` is `false`. ### Configuration [#configuration-1] | Option | Type | Required | Description | |---|---|---|---| | `reconnectOnError` | `boolean \| undefined` | no | Whether to automatically reconnect on websocket errors. Defaults to true. | | `apiKey` | `string` | yes | Sarvam API key. | | `baseUrl` | `string \| undefined` | no | Realtime STT websocket endpoint. | | `endpointing` | `"manual" \| "vad" \| undefined` | no | Which side detects turn boundaries: `vad` for Sarvam's own detection, or `manual` for the pipeline's. Decides the turn strategies this service asks the user aggregator to run, so it is fixed for the life of the service. Defaults to `vad`. | | `sampleRate` | `number \| null \| undefined` | no | Declared input audio sample rate, 8000 or 16000. `null` adopts the pipeline's input rate. | | `returnTimestamps` | `boolean \| undefined` | no | Whether final transcripts should include segment offsets. Defaults to false. | | `keyterms` | `string[] \| null \| undefined` | no | Connection-time recognition hints for saaras:v4. At most 50 terms of 64 characters each; put each phrase in a separate item. | | `prefixPaddingMs` | `number \| null \| undefined` | no | Optional VAD prefix padding, used only under `endpointing="vad"`. | | `settings` | `SarvamRealtimeSTTSettings \| null \| undefined` | no | Runtime-updatable realtime settings. | | `shouldInterrupt` | `boolean \| undefined` | no | Determine whether the bot should be interrupted when Sarvam detects user speech. Passed along to the user turn strategies this service recommends, which own the interruption; a user-supplied `userTurnStrategies` overrides the recommendation and this setting with it. Defaults to true. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `reconnectBackoffMinWait` | `number \| undefined` | no | Minimum time, in seconds, to wait between reconnection attempts. | | `reconnectBackoffMaxWait` | `number \| undefined` | no | Maximum time, in seconds, to wait between reconnection attempts. | | `wsCloseTimeout` | `number \| undefined` | no | Maximum time, in seconds, to wait for the peer to acknowledge the websocket closing handshake before dropping the connection. Applied to connections opened through `_websocketConnect()`. Increase it for peers that need longer to complete a graceful close. | | `wsSendTimeout` | `number \| null \| undefined` | no | Maximum time, in seconds, a single send may wait on a peer that has stopped reading before the connection is dropped and the send throws `ConnectionClosedError`. Applied to connections opened through `_websocketConnect()`. Pass `null` to wait indefinitely. Defaults to `WS_SEND_TIMEOUT`. | | `audioPassthrough` | `boolean \| undefined` | no | Whether to pass audio frames downstream after processing. Defaults to true. | | `sttTtfbTimeout` | `number \| undefined` | no | Time in seconds to wait after VAD stop before reporting TTFB. This delay allows the final transcription to arrive. Defaults to 2.0. | | `ttfsP99Latency` | `number \| null \| undefined` | no | P99 latency from speech end to final transcript in seconds. Broadcast via STTMetadataFrame at pipeline start. | | `keepaliveTimeout` | `number \| null \| undefined` | no | Seconds of no audio before sending silence to keep the connection alive. null disables keepalive. | | `keepaliveInterval` | `number \| undefined` | no | Seconds between idle checks when keepalive is enabled. | ### Settings [#settings-1] Passed as `settings: new SarvamRealtimeSTTService.Settings({...})` (`SarvamRealtimeSTTSettings`). | Setting | Type | Default | Description | | --------------------- | ------------------------------------- | ---------------------- | --------------------------------------------------------- | | `model` | `string` | `"saaras:v3-realtime"` | `saaras:v3-realtime` or `saaras:v4`. | | `language` | `Language \| null` | `Language.EN_IN` | Language; `null` has Sarvam detect it for each utterance. | | `streamType` | `"fast" \| "balanced" \| "simulated"` | `"balanced"` | Streaming cadence. | | `mode` | `SarvamMode` | `"transcribe"` | Realtime task mode. | | `prompt` | `string \| null` | `null` | Decoding prompt. | | `threshold` | `number \| null` | `null` | VAD sensitivity. | | `silenceDurationMs` | `number \| null` | `null` | Silence that ends speech. | | `minSpeechDurationMs` | `number \| null` | `null` | Shortest speech that counts. | ### Usage [#usage-1] ```ts import { SarvamRealtimeSTTService } from "@pipedog/sarvam/stt"; const realtime = new SarvamRealtimeSTTService({ apiKey: process.env.SARVAM_API_KEY ?? "", endpointing: "vad", settings: new SarvamRealtimeSTTService.Settings({ language: Language.HI_IN }), }); ``` Change settings mid-call with an `STTUpdateSettingsFrame`: ```ts import { STTUpdateSettingsFrame } from "@pipedog/core/frames/frames"; // Queue it on the worker, or push it from a processor upstream of the service. const update = new STTUpdateSettingsFrame({ delta: new SarvamRealtimeSTTService.Settings({ streamType: "fast" }), }); ``` ### Notes [#notes-1] * `endpointing: "vad"` (the default) lets the server mark turn boundaries. With `"manual"` the pipeline does, so it needs a turn strategy that emits `VADUserStartedSpeakingFrame` and `VADUserStoppedSpeakingFrame`; without one Sarvam never emits a final transcript. It's an option, not a setting, because it decides which turn strategies the user aggregator runs. * A VAD analyzer is needed in both modes: it times the transcription latency. * `sampleRate` is 8000 or 16000; `null` adopts the pipeline's input rate. | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/sarvam/package.json`: ✅ means the compat smoke test passes on that runtime. ## Runtime notes [#runtime-notes] * Upstream uses the `sarvamai` Python SDK for `SarvamSTTService`. Here it is a plain WebSocket through the core `websocketConnect` (which sends the auth headers on Workers too), so no SDK is needed. * The deprecated `model` and `params` constructor options and `Settings.language_code` are not ported. * Tested against local WebSocket servers, as upstream's tests do, not against Sarvam itself. # Smallest AI > Realtime speech-to-text with SmallestSTTService, over Smallest AI's Pulse WebSocket API. Source: https://polvallverdu.github.io/pipedog/docs/services/stt/smallest/ ## Overview [#overview] * **`SmallestSTTService`**: streams audio over a persistent WebSocket to Smallest AI's Pulse speech-to-text endpoint. Interim transcripts become `InterimTranscriptionFrame`s; when the VAD sees the user stop, the service sends `finalize` and the final becomes a `TranscriptionFrame`. Pipecat's documentation for the same service. `packages/smallest/src/stt.ts`. The realtime WebSocket reference. ## Installation [#installation] ```bash bun add @pipedog/smallest ``` ```ts import { SmallestSTTService } from "@pipedog/smallest/stt"; ``` ## Prerequisites [#prerequisites] * A Smallest AI account and an API key from [Smallest AI](https://smallest.ai/). * Pass the key as `apiKey`. The examples read it from `SMALLEST_API_KEY`. ## SmallestSTTService [#smalleststtservice] Connects to `wss://api.smallest.ai/waves/v1/stt/live`, with the key as a Bearer token on the handshake. ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `apiKey` | `string` | yes | Smallest AI API key for authentication. | | `baseUrl` | `string \| undefined` | no | Base WebSocket URL for the Smallest API. Defaults to "wss://api.smallest.ai". | | `encoding` | `string \| undefined` | no | Audio encoding format. Defaults to "linear16". | | `settings` | `SmallestSTTSettings \| null \| undefined` | no | Runtime-updatable settings for the STT service. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `reconnectBackoffMinWait` | `number \| undefined` | no | Minimum time, in seconds, to wait between reconnection attempts. | | `reconnectBackoffMaxWait` | `number \| undefined` | no | Maximum time, in seconds, to wait between reconnection attempts. | | `wsCloseTimeout` | `number \| undefined` | no | Maximum time, in seconds, to wait for the peer to acknowledge the websocket closing handshake before dropping the connection. Applied to connections opened through `_websocketConnect()`. Increase it for peers that need longer to complete a graceful close. | | `wsSendTimeout` | `number \| null \| undefined` | no | Maximum time, in seconds, a single send may wait on a peer that has stopped reading before the connection is dropped and the send throws `ConnectionClosedError`. Applied to connections opened through `_websocketConnect()`. Pass `null` to wait indefinitely. Defaults to `WS_SEND_TIMEOUT`. | | `audioPassthrough` | `boolean \| undefined` | no | Whether to pass audio frames downstream after processing. Defaults to true. | | `sampleRate` | `number \| null \| undefined` | no | The sample rate for audio input. If null, will be determined from the start frame. | | `sttTtfbTimeout` | `number \| undefined` | no | Time in seconds to wait after VAD stop before reporting TTFB. This delay allows the final transcription to arrive. Defaults to 2.0. | | `ttfsP99Latency` | `number \| null \| undefined` | no | P99 latency from speech end to final transcript in seconds. Broadcast via STTMetadataFrame at pipeline start. | | `keepaliveTimeout` | `number \| null \| undefined` | no | Seconds of no audio before sending silence to keep the connection alive. null disables keepalive. | | `keepaliveInterval` | `number \| undefined` | no | Seconds between idle checks when keepalive is enabled. | | `reconnectOnError` | `boolean \| undefined` | no | Whether to automatically reconnect on websocket errors. Defaults to true. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new SmallestSTTService.Settings({...})` (`SmallestSTTSettings`). Any setting you leave out keeps the default below. | Setting | Type | Default | Description | | -------------------- | ---------------------- | ------------- | ----------------------------------------------------------------------------------------------- | | `model` | `"pulse" \| "pulse-2"` | `"pulse"` | `pulse-2` is English-only and adds per-utterance emotion and gender. | | `language` | `Language` | `Language.EN` | Language of the audio. A non-English language with `pulse-2` throws. | | `wordTimestamps` | `boolean` | `false` | Word-level timestamps. | | `fullTranscript` | `boolean` | `false` | Include the cumulative transcript. | | `sentenceTimestamps` | `boolean` | `false` | Sentence-level timestamps. | | `redactPii` | `boolean` | `false` | Redact personally identifiable information. | | `redactPci` | `boolean` | `false` | Redact payment card information. | | `numerals` | `string` | `"auto"` | Convert spoken numerals to digits. | | `diarize` | `boolean` | `false` | Speaker diarization. | | `endpointing` | `boolean` | `true` | Finalize promptly on trailing silence. No effect on `pulse-2`. | | `keywords` | `string` | `""` | Comma-separated `KEYWORD:INTENSIFIER` pairs, such as `"NVIDIA:2"`. | | `format` | `boolean` | `true` | Punctuation and capitalization. | | `emotionDetection` | `boolean` | `true` | `pulse-2` only: a six-class `emotion` on each final. | | `genderDetection` | `boolean` | `true` | `pulse-2` only: a `gender` estimate on each final. | | `itnNormalize` | `boolean` | `false` | Inverse text normalization ("twenty five" becomes "25"). | | `noFormatDelay` | `boolean` | `false` | With `itnNormalize`, emit finals at once instead of holding back a number still being dictated. | ### Usage [#usage] ```ts title="stt.ts" import { Language } from "@pipedog/core/transcriptions/language"; import { SmallestSTTService } from "@pipedog/smallest/stt"; const stt = new SmallestSTTService({ apiKey: process.env.SMALLEST_API_KEY ?? "", settings: new SmallestSTTService.Settings({ language: Language.EN, wordTimestamps: true }), }); ``` Pulse 2.0, English only: ```ts const pulse2 = new SmallestSTTService({ apiKey: process.env.SMALLEST_API_KEY ?? "", settings: new SmallestSTTService.Settings({ model: "pulse-2" }), }); ``` Change settings mid-call with an `STTUpdateSettingsFrame`: ```ts import { STTUpdateSettingsFrame } from "@pipedog/core/frames/frames"; // Queue it on the worker, or push it from a processor upstream of the service. const update = new STTUpdateSettingsFrame({ delta: new SmallestSTTService.Settings({ redactPii: true }), }); ``` ### Event handlers [#event-handlers] | Event | Description | | ----------------- | ------------------------ | | `on_connected` | The WebSocket connected. | | `on_disconnected` | The WebSocket closed. | ### Notes [#notes] * `encoding` is `"linear16"` by default. `sampleRate` defaults to the pipeline's input rate. * Settings are query parameters of the connection, so changing one reconnects. * The same package has Smallest AI's [LLM](https://polvallverdu.github.io/pipedog/docs/services/llm/smallest.md) and [text-to-speech](https://polvallverdu.github.io/pipedog/docs/services/tts/smallest.md) services. | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/smallest/package.json`: ✅ means the compat smoke test passes on that runtime. ## Runtime notes [#runtime-notes] * The WebSocket goes through the core `websocketConnect`, with headers on the handshake. * Upstream sets a keepalive interval without implementing the keepalive message, so an idle connection logs a warning every few seconds. The port behaves the same way. * Tested against local WebSocket servers, as upstream's tests do, not against Smallest AI itself. # Soniox > Realtime multilingual speech-to-text with Soniox. Source: https://polvallverdu.github.io/pipedog/docs/services/stt/soniox/ ## Overview [#overview] `SonioxSTTService` streams audio to Soniox's real-time WebSocket API, with endpoint detection, speaker diarization, language identification and context. Pipecat's documentation for the same service. `packages/soniox/src`. ## Installation [#installation] ```bash bun add @pipedog/soniox ``` ## Prerequisites [#prerequisites] * A [Soniox](https://soniox.com) account and API key, passed as `apiKey` (the examples read `SONIOX_API_KEY`). ## SonioxSTTService [#sonioxsttservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | Soniox API key. | | `url` | `string \| undefined` | no | Soniox WebSocket API URL. | | `audioFormat` | `string \| undefined` | no | Audio format for transcription. Defaults to `"pcm_s16le"`. | | `numChannels` | `number \| undefined` | no | Number of audio channels. Defaults to 1. | | `vadForceTurnEndpoint` | `boolean \| undefined` | no | Controls turn detection mode. When true (pipedog mode, default): Soniox endpoint detection is disabled and a `VADUserStoppedSpeakingFrame` sends a finalize message to Soniox. When false (Soniox turn detection mode): Soniox endpoint detection is enabled and controls turn endings. Proposes a turn start on the local VAD signal when a VAD analyzer is configured (most responsive) or on the first transcript token otherwise, and a turn stop when the endpoint is detected. | | `shouldInterrupt` | `boolean \| undefined` | no | Whether to interrupt the bot when the user starts speaking in Soniox turn detection mode (`vadForceTurnEndpoint: false`). Passed along to the user turn strategies this service recommends, which own the interruption; a user-supplied `userTurnStrategies` overrides the recommendation and this setting with it. Defaults to true. | | `settings` | `STTSettings \| null \| undefined` | no | Runtime-updatable settings. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `reconnectBackoffMinWait` | `number \| undefined` | no | Minimum time, in seconds, to wait between reconnection attempts. | | `reconnectBackoffMaxWait` | `number \| undefined` | no | Maximum time, in seconds, to wait between reconnection attempts. | | `wsCloseTimeout` | `number \| undefined` | no | Maximum time, in seconds, to wait for the peer to acknowledge the websocket closing handshake before dropping the connection. Applied to connections opened through `_websocketConnect()`. Increase it for peers that need longer to complete a graceful close. | | `wsSendTimeout` | `number \| null \| undefined` | no | Maximum time, in seconds, a single send may wait on a peer that has stopped reading before the connection is dropped and the send throws `ConnectionClosedError`. Applied to connections opened through `_websocketConnect()`. Pass `null` to wait indefinitely. Defaults to `WS_SEND_TIMEOUT`. | | `audioPassthrough` | `boolean \| undefined` | no | Whether to pass audio frames downstream after processing. Defaults to true. | | `sampleRate` | `number \| null \| undefined` | no | The sample rate for audio input. If null, will be determined from the start frame. | | `sttTtfbTimeout` | `number \| undefined` | no | Time in seconds to wait after VAD stop before reporting TTFB. This delay allows the final transcription to arrive. Defaults to 2.0. | | `ttfsP99Latency` | `number \| null \| undefined` | no | P99 latency from speech end to final transcript in seconds. Broadcast via STTMetadataFrame at pipeline start. | | `keepaliveTimeout` | `number \| null \| undefined` | no | Seconds of no audio before sending silence to keep the connection alive. null disables keepalive. | | `keepaliveInterval` | `number \| undefined` | no | Seconds between idle checks when keepalive is enabled. | | `reconnectOnError` | `boolean \| undefined` | no | Whether to automatically reconnect on websocket errors. Defaults to true. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new SonioxSTTService.Settings({...})`. Settings you leave out keep their defaults. | Setting | Default | Description | | ----------------------------------------------------------------------------- | ------------- | ---------------------------- | | `model` | `"stt-rt-v5"` | Soniox model. | | `language` | | Language. | | `languageHints`, `languageHintsStrict` | | Bias language detection. | | `context` | | A `SonioxContextObject`. | | `enableSpeakerDiarization`, `enableLanguageIdentification` | | Diarization and language ID. | | `maxEndpointDelayMs`, `endpointSensitivity`, `endpointLatencyAdjustmentLevel` | | Endpointing. | | `clientReferenceId` | | Your own reference. | ### Usage [#usage] ```ts title="stt.ts" import { Language } from "@pipedog/core/transcriptions/language"; import { SonioxSTTService } from "@pipedog/soniox/stt"; const stt = new SonioxSTTService({ apiKey: process.env.SONIOX_API_KEY ?? "", settings: new SonioxSTTService.Settings({ languageHints: [Language.EN, Language.ES] }), }); ``` ### Notes [#notes] * With `vadForceTurnEndpoint` (default), a VAD stop sends a finalize message. ## Runtimes [#runtimes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/soniox/package.json`: ✅ means the compat smoke test passes on that runtime. The service uses `fetch` and the global `WebSocket` (through core's `websocketConnect`, which sends the auth header on the handshake on Cloudflare Workers too). Pass no SDK; nothing is Node-only. # Speechmatics > Realtime speech-to-text with Speechmatics, with diarization and server-side turn detection. Source: https://polvallverdu.github.io/pipedog/docs/services/stt/speechmatics/ ## Overview [#overview] `SpeechmaticsSTTService` speaks Speechmatics' Agent STT WebSocket protocol (`/v2/agent`). pipedog implements the protocol directly (`src/agent_stt.ts`) instead of depending on the Python-only `speechmatics-rt` SDK. Pipecat's documentation for the same service. `packages/speechmatics/src`. ## Installation [#installation] ```bash bun add @pipedog/speechmatics ``` ## Prerequisites [#prerequisites] * A [Speechmatics](https://www.speechmatics.com) API key, passed as `apiKey` or read from `SPEECHMATICS_API_KEY`. `baseUrl` defaults to `wss://eu2.rt.speechmatics.com/v2/agent` (or `SPEECHMATICS_RT_URL`). ## SpeechmaticsSTTService [#speechmaticssttservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string \| null \| undefined` | no | Speechmatics API key for authentication. Uses environment variable `SPEECHMATICS_API_KEY` if not provided. | | `baseUrl` | `string \| null \| undefined` | no | Base URL for Speechmatics API. Uses environment variable `SPEECHMATICS_RT_URL` or defaults to `wss://eu2.rt.speechmatics.com/v2/agent`. | | `encoding` | `AudioEncoding \| undefined` | no | Audio encoding format. Defaults to `AudioEncoding.PCM_S16LE`. | | `shouldInterrupt` | `boolean \| undefined` | no | Determine whether the bot should be interrupted when Speechmatics `turnDetectionMode` is configured to detect user speech. Passed along to the user turn strategies this service recommends, which own the interruption; a user-supplied `userTurnStrategies` overrides the recommendation and this setting with it. Defaults to true. | | `settings` | `SpeechmaticsSTTSettings \| null \| undefined` | no | Runtime-updatable settings. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `audioPassthrough` | `boolean \| undefined` | no | Whether to pass audio frames downstream after processing. Defaults to true. | | `sampleRate` | `number \| null \| undefined` | no | The sample rate for audio input. If null, will be determined from the start frame. | | `sttTtfbTimeout` | `number \| undefined` | no | Time in seconds to wait after VAD stop before reporting TTFB. This delay allows the final transcription to arrive. Defaults to 2.0. | | `ttfsP99Latency` | `number \| null \| undefined` | no | P99 latency from speech end to final transcript in seconds. Broadcast via STTMetadataFrame at pipeline start. | | `keepaliveTimeout` | `number \| null \| undefined` | no | Seconds of no audio before sending silence to keep the connection alive. null disables keepalive. | | `keepaliveInterval` | `number \| undefined` | no | Seconds between idle checks when keepalive is enabled. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new SpeechmaticsSTTService.Settings({...})`. Settings you leave out keep their defaults. | Setting | Default | Description | | --------------------------------------------------------------------------------------------------------- | ------------ | ----------------------------------------------------------------------- | | `model` | `"linden-1"` | Speechmatics model. | | `language`, `domain` | | Language and domain. | | `turnDetectionMode` | `EXTERNAL` | `VAD` hands turn detection to the server. | | `speakerActiveFormat` | `"{text}"` | Template with `{speaker_id}`, `{text}`, `{ts}`, `{lang}`. Applies live. | | `enableDiarization`, `speakerSensitivity`, `maxSpeakers`, `preferCurrentSpeaker`, `knownSpeakers` | | Diarization. | | `additionalVocab`, `enablePartials`, `emitSentences`, `punctuationOverrides`, `transcriptFilteringConfig` | | Transcript shaping. | Changing any setting other than `speakerActiveFormat` reconnects. ### Usage [#usage] ```ts title="stt.ts" import { Language } from "@pipedog/core/transcriptions/language"; import { SpeechmaticsSTTService, TurnDetectionMode } from "@pipedog/speechmatics/stt"; const stt = new SpeechmaticsSTTService({ apiKey: process.env.SPEECHMATICS_API_KEY ?? "", settings: new SpeechmaticsSTTService.Settings({ language: Language.EN, turnDetectionMode: TurnDetectionMode.EXTERNAL, enableDiarization: true, speakerActiveFormat: "@{speaker_id}: {text}", }), }); ``` ### Notes [#notes] * Event: `on_speakers_result`. * A refused handshake status is hidden by the standard `WebSocket` on Bun, Node and Deno, so an invalid key is retried 3 times and then reported as a permanent failure. ## Runtimes [#runtimes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/speechmatics/package.json`: ✅ means the compat smoke test passes on that runtime. The service uses `fetch` and the global `WebSocket` (through core's `websocketConnect`, which sends the auth header on the handshake on Cloudflare Workers too). Pass no SDK; nothing is Node-only. # Together AI > Realtime speech-to-text with TogetherSTTService, over Together AI's WebSocket API. Source: https://polvallverdu.github.io/pipedog/docs/services/stt/together/ ## Overview [#overview] * **`TogetherSTTService`**: streams audio over a WebSocket to Together's realtime transcription endpoint and pushes interim and final transcriptions. Pipecat's documentation for the same service. `packages/together/src/stt.ts`. Speech-to-text models and the realtime API. ## Installation [#installation] ```bash bun add @pipedog/together ``` ```ts import { TogetherSTTService } from "@pipedog/together/stt"; ``` ## Prerequisites [#prerequisites] * A Together AI account and an API key from [Together AI](https://together.ai/). * Pass the key as `apiKey`. The examples read it from `TOGETHER_API_KEY`. ## TogetherSTTService [#togethersttservice] Connects to `wss://api.together.ai/v1/realtime?intent=transcription`, with `Authorization: Bearer `. ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `ttfsP99Latency` | `number \| null \| undefined` | no | P99 latency from speech end to final transcript in seconds. Broadcast via STTMetadataFrame at pipeline start. | | `apiKey` | `string` | yes | Together AI API key for authentication. | | `baseUrl` | `string \| undefined` | no | The URL of the Together AI WebSocket API. Defaults to "wss://api.together.ai/v1". | | `settings` | `TogetherSTTSettings \| null \| undefined` | no | Runtime-updatable settings for the STT service. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `reconnectBackoffMinWait` | `number \| undefined` | no | Minimum time, in seconds, to wait between reconnection attempts. | | `reconnectBackoffMaxWait` | `number \| undefined` | no | Maximum time, in seconds, to wait between reconnection attempts. | | `wsCloseTimeout` | `number \| undefined` | no | Maximum time, in seconds, to wait for the peer to acknowledge the websocket closing handshake before dropping the connection. Applied to connections opened through `_websocketConnect()`. Increase it for peers that need longer to complete a graceful close. | | `wsSendTimeout` | `number \| null \| undefined` | no | Maximum time, in seconds, a single send may wait on a peer that has stopped reading before the connection is dropped and the send throws `ConnectionClosedError`. Applied to connections opened through `_websocketConnect()`. Pass `null` to wait indefinitely. Defaults to `WS_SEND_TIMEOUT`. | | `audioPassthrough` | `boolean \| undefined` | no | Whether to pass audio frames downstream after processing. Defaults to true. | | `sampleRate` | `number \| null \| undefined` | no | The sample rate for audio input. If null, will be determined from the start frame. | | `sttTtfbTimeout` | `number \| undefined` | no | Time in seconds to wait after VAD stop before reporting TTFB. This delay allows the final transcription to arrive. Defaults to 2.0. | | `keepaliveTimeout` | `number \| null \| undefined` | no | Seconds of no audio before sending silence to keep the connection alive. null disables keepalive. | | `keepaliveInterval` | `number \| undefined` | no | Seconds between idle checks when keepalive is enabled. | | `reconnectOnError` | `boolean \| undefined` | no | Whether to automatically reconnect on websocket errors. Defaults to true. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new TogetherSTTService.Settings({...})` (`TogetherSTTSettings`). Any setting you leave out keeps the default below. | Setting | Type | Default | Description | | ---------- | ------------------ | --------------------------- | ---------------------- | | `model` | `string` | `"openai/whisper-large-v3"` | Transcription model. | | `language` | `Language \| null` | `Language.EN` | Language of the audio. | ### Usage [#usage] ```ts title="stt.ts" import { STTUpdateSettingsFrame } from "@pipedog/core/frames/frames"; import { TogetherSTTService } from "@pipedog/together/stt"; const stt = new TogetherSTTService({ apiKey: process.env.TOGETHER_API_KEY ?? "", settings: new TogetherSTTService.Settings({ model: "openai/whisper-large-v3" }), }); ``` Change settings mid-call with an `STTUpdateSettingsFrame`: ```ts // Queue it on the worker, or push it from a processor upstream of the service. const update = new STTUpdateSettingsFrame({ delta: new TogetherSTTService.Settings({ model: "openai/whisper-large-v3" }), }); ``` ### Event handlers [#event-handlers] | Event | Description | | ----------------- | ------------------------ | | `on_connected` | The WebSocket connected. | | `on_disconnected` | The WebSocket closed. | ### Notes [#notes] * Audio is sent as base64 16-bit PCM. When the user stops speaking, the buffer is committed so the final transcript arrives. * Any settings change reconnects, since the session is configured in the connection URL. * If the connection drops, the base class reconnects. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/together/package.json`: ✅ means the compat smoke test passes on that runtime. * The WebSocket goes through the core `websocketConnect`, with headers on the handshake. It works on Bun, Node, Deno and Cloudflare Workers. * Tested against a local WebSocket fake, not against Together AI itself. # Whisper > Speech-to-text with Whisper. WhisperSTTService and WhisperSTTServiceMLX run Whisper locally on ONNX Runtime, and BaseWhisperSTTService is the base for Whisper-compatible APIs. Source: https://polvallverdu.github.io/pipedog/docs/services/stt/whisper/ ## Overview [#overview] `@pipedog/whisper` is a port of Pipecat's `pipecat.services.whisper` (`base_stt`, `stt`, `utils`): * **`WhisperSTTService`** (`@pipedog/whisper/stt`): transcribes each speech segment, cut by the pipeline's VAD, with a Whisper model on your machine. No account, no API key. Upstream wraps faster-whisper (CTranslate2); this one runs faster-whisper's algorithm on ONNX Runtime (`@pipedog/whisper/faster_whisper`), with ONNX exports of the same Whisper checkpoints. * **`WhisperSTTServiceMLX`** (`@pipedog/whisper/stt`): upstream's MLX Whisper service, for Apple Silicon. It is **not MLX** here: it keeps upstream's class, settings, model names and filtering, and runs the same ONNX engine, decoding like mlx_whisper. It works on any platform. * **`BaseWhisperSTTService`** (`@pipedog/whisper/base_stt`): the base for services that call a Whisper-style transcription API (OpenAI's `/v1/audio/transcriptions` and compatible servers). [`OpenAISTTService`](https://polvallverdu.github.io/pipedog/docs/services/stt/openai.md) and [`GroqSTTService`](https://polvallverdu.github.io/pipedog/docs/services/stt/groq.md) extend it. * **Probability helpers** (`@pipedog/whisper/utils`): `extractWhisperProbability`, `extractOpenaiGpt4oProbability` and `extractDeepgramProbability` read a confidence from a `TranscriptionFrame`'s `result`. Local Whisper is one of upstream's keyless local defaults for [evals](https://polvallverdu.github.io/pipedog/docs/evals.md): set `service: whisper` to transcribe the bot's speech with it instead of Moonshine. Pipecat's documentation for the same services. `stt.ts`, the engine `faster_whisper.ts`, `base_stt.ts` and `utils.ts`. The decoding algorithm the local services follow. ## Installation [#installation] ```bash bun add @pipedog/whisper # for the local services, also an ONNX runtime: bun add onnxruntime-node ``` `onnxruntime-node` (native, fastest on Bun, Node and Deno) or `onnxruntime-web@1.30.0` (WebAssembly) are optional peers of `@pipedog/onnx`. The tokenizer, [`@huggingface/tokenizers`](https://www.npmjs.com/package/@huggingface/tokenizers) (pure JS), is a dependency. ```ts import { Model, WhisperSTTService, WhisperSTTServiceMLX } from "@pipedog/whisper/stt"; import { BaseWhisperSTTService } from "@pipedog/whisper/base_stt"; import { extractWhisperProbability } from "@pipedog/whisper/utils"; ``` The package root re-exports `base_stt` and `utils` only. Import the local services from `@pipedog/whisper/stt`: they pull in the ONNX runtime and the tokenizer, which the API services don't need. ## What you need [#what-you-need] **Local services**: the model files, downloaded once from Hugging Face when the pipeline starts, into `$PIPEDOG_CACHE_DIR/whisper`, else `$XDG_CACHE_HOME/pipedog/whisper`, else `~/.cache/pipedog/whisper` (one folder per repo). Input must be 16 kHz (the pipeline's input sample rate), as upstream: segments are not resampled. And a VAD in the pipeline. Upstream's model names map to the ONNX export of the same checkpoint: | `Model` / name | ONNX repo | Default download (fp32 encoder, int8 decoder) | `computeType: "int8"` | | -------------------------------- | --------------------------------------- | --------------------------------------------- | --------------------- | | `TINY` (`tiny`) | `onnx-community/whisper-tiny` | 64 MB | 41 MB | | `BASE` (`base`) | `onnx-community/whisper-base` | 136 MB | 77 MB | | `SMALL` (`small`) | `onnx-community/whisper-small` | 510 MB | 249 MB | | `MEDIUM` (`medium`) | `onnx-community/whisper-medium-ONNX` | 1.9 GB | 986 MB | | `LARGE` (`large-v3`) | `onnx-community/whisper-large-v3-ONNX` | 3.7 GB | 1.8 GB | | `LARGE_V3_TURBO` | `onnx-community/whisper-large-v3-turbo` | 3.0 GB | 1.1 GB | | `DISTIL_LARGE_V2` | `distil-whisper/distil-large-v2` | 2.7 GB | 767 MB | | `DISTIL_MEDIUM_EN` (the default) | `distil-whisper/distil-medium.en` | 1.3 GB | 402 MB | Plus the tokenizer (2.5 MB) and configs. Every other faster-whisper name works too (`tiny.en`, `base.en`, `small.en`, `medium.en`, `large-v1`, `large-v2`, `distil-small.en`, `distil-large-v3`, `distil-large-v3.5`, `large-v3-turbo`, `turbo`), as does any Hugging Face repo with the same ONNX layout (`onnx/encoder_model*.onnx`, `onnx/decoder_model_merged*.onnx`). To control where the files come from, use `modelOptions`: * `downloadRoot`: another cache directory. * `localFilesOnly: true`: never download. * A local directory with the same layout as a model name, as the `model` setting. * `files`: every file explicitly, as bytes, paths or URLs (`config.json`, `generation_config.json`, `preprocessor_config.json`, `tokenizer.json`, the encoder and the decoder). ```ts // Never download: read the files from a cache you filled beforehand. const offline = new WhisperSTTService({ modelOptions: { downloadRoot: "/models/whisper", localFilesOnly: true }, settings: new WhisperSTTService.Settings({ model: Model.LARGE_V3_TURBO, language: Language.ES }), }); ``` `computeType` (faster-whisper's `compute_type`) picks the weights: `default` is an fp32 encoder and an int8 (`q8`) decoder, `int8*` is `q8` for both, `float32` is fp32 for both, or pass an ONNX dtype (`fp32`, `q8`, `int8`, `uint8`, `q4`, `bnb4`) or `modelOptions: { dtype: { encoder, decoder } }`. The `distil-whisper/*` exports have only `fp32` and `q8`, and fp16 isn't supported. The large fp32 models keep their weights in a separate `.onnx_data` file, which needs `onnxruntime-node`: use `int8` with `onnxruntime-web`. **`BaseWhisperSTTService`**: a Whisper-compatible transcription endpoint and its API key, if it needs one (the examples read it from `WHISPER_API_KEY`), and a VAD in the pipeline. ## Runtimes [#runtimes] | Runtime | Works | What it needs | | ------------------ | ----- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Bun | ✅ yes | BaseWhisperSTTService and utils: only fetch. WhisperSTTService / WhisperSTTServiceMLX (@pipedog/whisper/stt): onnxruntime-node or onnxruntime-web, a filesystem for the model cache (or the model files passed as bytes), and 16 kHz input. | | Node.js | ✅ yes | BaseWhisperSTTService and utils: only fetch. WhisperSTTService / WhisperSTTServiceMLX (@pipedog/whisper/stt): onnxruntime-node or onnxruntime-web, a filesystem for the model cache (or the model files passed as bytes), and 16 kHz input. | | Deno | ✅ yes | BaseWhisperSTTService and utils: only fetch. WhisperSTTService / WhisperSTTServiceMLX (@pipedog/whisper/stt): onnxruntime-node or onnxruntime-web, a filesystem for the model cache (or the model files passed as bytes), and 16 kHz input. | | Cloudflare Workers | ✅ yes | BaseWhisperSTTService and utils: only fetch. WhisperSTTService / WhisperSTTServiceMLX (@pipedog/whisper/stt): onnxruntime-node or onnxruntime-web, a filesystem for the model cache (or the model files passed as bytes), and 16 kHz input. BaseWhisperSTTService and utils. The local services' code runs there (smoke-tested with a fake engine), but real Whisper models don't fit a Worker's memory and there is no model cache: not supported. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/whisper/package.json`: ✅ means the compat smoke test passes on that runtime. `BaseWhisperSTTService` and the probability helpers run on Cloudflare Workers. The local services don't: Whisper models (41 MB to 3.7 GB) don't fit a Worker's 128 MB of memory or its 10 MB bundle limit, and there is no model cache. Their code is smoke-tested there with a fake engine. ## WhisperSTTService [#whispersttservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `device` | `string \| undefined` | no | The device to run inference on ('cpu', 'cuda', or 'auto'). Defaults to "auto". | | `computeType` | `string \| undefined` | no | The compute type for inference ('default', 'int8', 'int8_float16', 'float32', ... or an ONNX dtype: 'fp32', 'q8', 'int8', 'uint8', 'q4', 'bnb4'). Defaults to "default" (fp32 encoder, int8 decoder). | | `settings` | `WhisperSTTSettings \| null \| undefined` | no | Runtime-updatable settings. When provided, its values take precedence. | | `modelOptions` | `Omit \| undefined` | no | Options for the ONNX engine (`WhisperModel`). Not in upstream. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `audioPassthrough` | `boolean \| undefined` | no | Whether to pass audio frames downstream after processing. Defaults to true. | | `sampleRate` | `number \| null \| undefined` | no | The sample rate for audio input. If null, will be determined from the start frame. | | `sttTtfbTimeout` | `number \| undefined` | no | Time in seconds to wait after VAD stop before reporting TTFB. This delay allows the final transcription to arrive. Defaults to 2.0. | | `ttfsP99Latency` | `number \| null \| undefined` | no | P99 latency from speech end to final transcript in seconds. Broadcast via STTMetadataFrame at pipeline start. | | `keepaliveTimeout` | `number \| null \| undefined` | no | Seconds of no audio before sending silence to keep the connection alive. null disables keepalive. | | `keepaliveInterval` | `number \| undefined` | no | Seconds between idle checks when keepalive is enabled. | | `trailingSilenceSecs` | `number \| undefined` | no | Seconds of silence appended to each segment before it is transcribed, so the model hears the end of speech and can finish the last word. Set to 0 to disable. Defaults to 0.5. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new WhisperSTTService.Settings({...})` (`WhisperSTTSettings`). Any setting you leave out keeps the default below. | Setting | Type | Default | Description | | --------------- | ------------------ | ------------------------ | ----------------------------------------------------------------------------------------- | | `model` | `string` | `Model.DISTIL_MEDIUM_EN` | A `Model`, a faster-whisper name, a Hugging Face repo or a local directory. | | `language` | `Language \| null` | `Language.EN` | Transcription language. English-only models reject other languages. | | `noSpeechProb` | `number` | `0.4` | Segments whose no-speech probability is at or above this are dropped. | | `hotwords` | `string \| null` | `null` | Words or phrases to bias the transcription towards, space-separated. | | `initialPrompt` | `string \| null` | `null` | Text given to the decoder as preceding context, to steer style, punctuation and spelling. | ### Usage [#usage] ```ts title="stt.ts" import { Model, WhisperSTTService } from "@pipedog/whisper/stt"; // Downloads distil-whisper/distil-medium.en (402 MB with int8) on first start. const local = new WhisperSTTService({ computeType: "int8", // smaller download, faster on CPU settings: new WhisperSTTService.Settings({ model: Model.DISTIL_MEDIUM_EN, language: Language.EN, hotwords: "Pipecat pipedog", }), }); ``` The engine works on its own too, with faster-whisper's API: ```ts import { WhisperModel } from "@pipedog/whisper/faster_whisper"; const model = new WhisperModel("tiny", { computeType: "int8" }); const audio = new Float32Array(16000); // 16 kHz mono audio in [-1, 1] const [segments, info] = await model.transcribe(audio, { language: null }); for await (const s of segments) console.log(info.language, s.start, s.end, s.text, s.noSpeechProb); ``` ### Notes [#notes] * Emits one `TranscriptionFrame` per segment. There are no interim results. * Decoding follows faster-whisper: the same log-mel features, prompt, `hotwords` and `initialPrompt`, `no_speech_prob`, token suppression, timestamp rules, beam search (beam 5), the temperature fallback with best-of-5 sampling, segment splitting and language detection. With fp32 weights, whisper tiny gives the same text, `avg_logprob` and `no_speech_prob` (to 4 decimals) as faster-whisper on the test clip. * The model files load in `start()`. A failed load is pushed as an error and retried on the next segment. * `device: "cpu"` forces the CPU provider. `cuda` can't be selected. ## WhisperSTTServiceMLX [#whispersttservicemlx] The MLX service's class, settings, model names and filtering, on the same ONNX engine. Each `MLXModel` selects the ONNX export of the same checkpoint (`LARGE_V3_TURBO_Q4` selects its 4-bit `q4` weights). It decodes like mlx_whisper: greedy, at the configured temperature only, and drops likely hallucinations. The model loads on the first segment. ### Configuration [#configuration-1] | Option | Type | Required | Description | |---|---|---|---| | `settings` | `WhisperMLXSTTSettings \| null \| undefined` | no | Runtime-updatable settings. When provided, its values take precedence. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `audioPassthrough` | `boolean \| undefined` | no | Whether to pass audio frames downstream after processing. Defaults to true. | | `sampleRate` | `number \| null \| undefined` | no | The sample rate for audio input. If null, will be determined from the start frame. | | `sttTtfbTimeout` | `number \| undefined` | no | Time in seconds to wait after VAD stop before reporting TTFB. This delay allows the final transcription to arrive. Defaults to 2.0. | | `ttfsP99Latency` | `number \| null \| undefined` | no | P99 latency from speech end to final transcript in seconds. Broadcast via STTMetadataFrame at pipeline start. | | `keepaliveTimeout` | `number \| null \| undefined` | no | Seconds of no audio before sending silence to keep the connection alive. null disables keepalive. | | `keepaliveInterval` | `number \| undefined` | no | Seconds between idle checks when keepalive is enabled. | | `trailingSilenceSecs` | `number \| undefined` | no | Seconds of silence appended to each segment before it is transcribed, so the model hears the end of speech and can finish the last word. Set to 0 to disable. Defaults to 0.5. | | `modelOptions` | `Omit \| undefined` | no | Options for the ONNX engine (`WhisperModel`). Not in upstream. | ### Settings [#settings-1] Runtime-updatable settings, passed as `settings: new WhisperSTTServiceMLX.Settings({...})` (`WhisperMLXSTTSettings`). Any setting you leave out keeps the default below. | Setting | Type | Default | Description | | -------------- | ------------------ | --------------- | ---------------------------------------------------------------------------------------------------------------- | | `model` | `string` | `MLXModel.TINY` | `TINY`, `MEDIUM`, `LARGE_V3`, `LARGE_V3_TURBO`, `LARGE_V3_TURBO_Q4` or `DISTIL_LARGE_V3`, or another model name. | | `language` | `Language \| null` | `Language.EN` | Transcription language. | | `noSpeechProb` | `number` | `0.6` | Segments whose no-speech probability is at or above this are dropped. | | `temperature` | `number` | `0.0` | Sampling temperature, 0 to 1. | | `engine` | `string` | `"mlx"` | Kept from upstream; it has no effect here. | ### Usage [#usage-1] ```ts import { MLXModel, WhisperSTTServiceMLX } from "@pipedog/whisper/stt"; // Not MLX: the same ONNX engine, with MLX Whisper's model names and greedy decoding. const mlx = new WhisperSTTServiceMLX({ settings: new WhisperSTTServiceMLX.Settings({ model: MLXModel.LARGE_V3_TURBO_Q4, language: Language.EN, temperature: 0.0, }), }); ``` ## BaseWhisperSTTService [#basewhispersttservice] ### Configuration [#configuration-2] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string \| null \| undefined` | no | Service API key. Defaults to null. | | `baseUrl` | `string \| null \| undefined` | no | Service API base URL. Defaults to null. | | `httpClient` | `((input: RequestInfo \| URL, init?: RequestInit) => Promise) \| null \| undefined` | no | Custom `fetch` for API requests, e.g. one with a longer request timeout. Defaults to null, which lets the SDK use the global `fetch`. | | `includeProbMetrics` | `boolean \| undefined` | no | If true, enables probability metrics in API response. Each service implements this differently (see child classes). Defaults to false. | | `pushEmptyTranscripts` | `boolean \| undefined` | no | If true, allow empty `TranscriptionFrame` frames to be pushed downstream instead of discarding them. This is intended for situations where VAD fires even though the user did not speak. In these cases, it is useful to know that nothing was transcribed so that the agent can resume speaking, instead of waiting longer for a transcription. Defaults to false. | | `settings` | `BaseWhisperSTTSettings \| null \| undefined` | no | Runtime-updatable settings. When provided, its values take precedence. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `audioPassthrough` | `boolean \| undefined` | no | Whether to pass audio frames downstream after processing. Defaults to true. | | `sampleRate` | `number \| null \| undefined` | no | The sample rate for audio input. If null, will be determined from the start frame. | | `sttTtfbTimeout` | `number \| undefined` | no | Time in seconds to wait after VAD stop before reporting TTFB. This delay allows the final transcription to arrive. Defaults to 2.0. | | `ttfsP99Latency` | `number \| null \| undefined` | no | P99 latency from speech end to final transcript in seconds. Broadcast via STTMetadataFrame at pipeline start. | | `keepaliveTimeout` | `number \| null \| undefined` | no | Seconds of no audio before sending silence to keep the connection alive. null disables keepalive. | | `keepaliveInterval` | `number \| undefined` | no | Seconds between idle checks when keepalive is enabled. | | `trailingSilenceSecs` | `number \| undefined` | no | Seconds of silence appended to each segment before it is transcribed, so the model hears the end of speech and can finish the last word. Set to 0 to disable. Defaults to 0.5. | ### Settings [#settings-2] Runtime-updatable settings, passed as `settings: new BaseWhisperSTTService.Settings({...})`. The base leaves every field unset; subclasses set their own defaults (`OpenAISTTService` uses `"gpt-transcribe"` and `Language.EN`). | Setting | Type | Default | Description | | ------------- | ------------------ | ------- | ----------------------------------------------------------- | | `model` | `string \| null` | `null` | Transcription model. | | `language` | `Language \| null` | `null` | Transcription language. | | `prompt` | `string \| null` | `null` | Text to guide the style, or to continue a previous segment. | | `temperature` | `number \| null` | `null` | Sampling temperature, 0 to 1. | ### Usage [#usage-2] The base class cannot be used on its own: its `_transcribe()` throws. To point at a Whisper-compatible endpoint, pass `baseUrl` to `OpenAISTTService`: ```ts title="whisper.ts" import { Language } from "@pipedog/core/transcriptions/language"; import { OpenAISTTService } from "@pipedog/openai/stt"; import { BaseWhisperSTTService, type Transcription } from "@pipedog/whisper/base_stt"; // A server that speaks OpenAI's /v1/audio/transcriptions API. const stt = new OpenAISTTService({ apiKey: process.env.WHISPER_API_KEY ?? "", baseUrl: "http://localhost:8000/v1", settings: new OpenAISTTService.Settings({ model: "whisper-1", language: Language.EN, }), }); ``` For an endpoint that needs a different request, subclass the base and implement `_transcribe()` with the OpenAI client it built from `apiKey`, `baseUrl` and `httpClient`: ```ts /** A provider whose Whisper-compatible endpoint needs its own request shape. */ class MyWhisperSTTService extends BaseWhisperSTTService { // Fail at construction without a key, instead of falling back to OPENAI_API_KEY. protected static override _requiresApiKey = true; protected override async _transcribe(audio: Uint8Array): Promise { return await this._client.audio.transcriptions.create({ file: new File([audio.slice()], "audio.wav", { type: "audio/wav" }), model: String(this._settings.model), language: String(this._settings.language), response_format: "json", }); } } const custom = new MyWhisperSTTService({ apiKey: process.env.WHISPER_API_KEY ?? "", baseUrl: "https://whisper.example.com/v1", settings: new MyWhisperSTTService.Settings({ model: "whisper-large-v3", language: Language.EN }), }); ``` ### Notes [#notes-1] * Emits one `TranscriptionFrame` per segment, with the API response as `result`. There are no interim results. * Empty transcripts are logged and dropped. With `pushEmptyTranscripts: true` they are pushed, so the bot can move on when the VAD fired but nobody spoke. * A transcription error is pushed as an `ErrorFrame`. * Languages map to Whisper's codes. A language not in the map falls back to its base code (`en` from `en-US`) with a warning. * Set the static `_requiresApiKey` to `true` in a subclass for a provider other than OpenAI. Without it, a missing key makes the OpenAI client fall back to `OPENAI_API_KEY` and send it to that provider. * The HTTP client is the `openai` JS SDK over `fetch`. `httpClient` replaces its `fetch`, for a longer timeout for example. Upstream takes an httpx client, so its `Timeout` and connection limits do not carry over. ## Probability helpers [#probability-helpers] Each reads a confidence (0 to 1) from a `TranscriptionFrame`'s `result`, or returns `null`: | Function | Reads | For | | ------------------------------- | ----------------------------------------------------------- | ------------------------------------------------------------------------------------------ | | `extractWhisperProbability` | `segments[0].avg_logprob` (verbose JSON) | `OpenAISTTService` and `GroqSTTService` with Whisper models and `includeProbMetrics: true` | | `extractOpenaiGpt4oProbability` | the average of `logprobs` | `OpenAISTTService` with GPT transcription models and `includeProbMetrics: true` | | `extractDeepgramProbability` | the first alternative's `confidence`, or its words' average | `DeepgramSTTService` | ```ts import { TranscriptionFrame } from "@pipedog/core/frames/frames"; import { extractWhisperProbability } from "@pipedog/whisper/utils"; // In a frame processor after an STT service created with includeProbMetrics: true. function logConfidence(frame: unknown): void { if (frame instanceof TranscriptionFrame) { const prob = extractWhisperProbability(frame); if (prob !== null) console.log(`Transcription confidence: ${(prob * 100).toFixed(2)}%`); } } ``` ## Differences from upstream [#differences-from-upstream] * The local services run on ONNX exports of the same checkpoints, not CTranslate2 conversions. int8 quantization differs from CTranslate2's, so quantized outputs differ slightly. * `WhisperSTTServiceMLX` is not MLX (above). Its features are faster-whisper's, not mlx_whisper's (which pads 30 s of silence before computing them). * Model files load in `start()`; upstream loads them in the constructor, blocking. The English-only language check still runs in the constructor for faster-whisper's names, and after loading for custom models. * There are no threads: segments are consumed on the event loop, and each decoder step awaits the runtime (`onnxruntime-node` runs it on its own threads). * Sampling at temperature above 0 uses `Math.random` (no seed). * Not implemented (faster-whisper options the services never use): `vad_filter`, `word_timestamps`, `hallucination_silence_threshold`, `clip_timestamps`, `repetition_penalty`, `no_repeat_ngram_size`, the batched pipeline. * `modelOptions` is a pipedog option. * `extractOpenaiGpt4oProbability` reads each entry's `logprob`, since the API returns `{ token, logprob, bytes }` objects that upstream's `sum()` can't add. * The deprecated constructor arguments (`model`, `no_speech_prob`, `language`, `temperature`, `prompt`) are not ported. Use `settings`. ## Licenses [#licenses] * `@pipedog/whisper`: BSD 2-Clause. `faster_whisper.ts` follows faster-whisper (MIT, Copyright (c) 2023 SYSTRAN) and OpenAI Whisper (MIT). * Models: OpenAI's Whisper weights and the distil-whisper models are MIT. The ONNX exports are by the Hugging Face `onnx-community`, `Xenova` and `distil-whisper` accounts. They are downloaded at runtime, not shipped. * `@huggingface/tokenizers`: Apache-2.0. # xAI > Realtime speech-to-text with XAISTTService, over xAI's WebSocket API. Source: https://polvallverdu.github.io/pipedog/docs/services/stt/xai/ ## Overview [#overview] * **`XAISTTService`**: streams audio over a persistent WebSocket to xAI's speech-to-text endpoint. Interim results and chunk finals become `InterimTranscriptionFrame`s; the utterance final becomes a `TranscriptionFrame`. Pipecat's documentation for the same service. `packages/xai/src/stt.ts`. The voice API reference. ## Installation [#installation] ```bash bun add @pipedog/xai ``` ```ts import { XAISTTService } from "@pipedog/xai/stt"; ``` ## Prerequisites [#prerequisites] * An xAI account and an API key from the [xAI console](https://console.x.ai/). * Pass the key as `apiKey`. The examples read it from `XAI_API_KEY`. ## XAISTTService [#xaisttservice] Connects to `wss://api.x.ai/v1/stt`, with the key as a Bearer token on the handshake. ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | xAI API key (used as Bearer for the WebSocket handshake). | | `wsUrl` | `string \| undefined` | no | WebSocket endpoint URL. Defaults to "wss://api.x.ai/v1/stt". | | `encoding` | `string \| undefined` | no | Audio encoding. One of "pcm" (signed 16-bit LE), "mulaw", or "alaw". Defaults to "pcm". | | `settings` | `XAISTTSettings \| null \| undefined` | no | Runtime-updatable settings overriding defaults. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `reconnectBackoffMinWait` | `number \| undefined` | no | Minimum time, in seconds, to wait between reconnection attempts. | | `reconnectBackoffMaxWait` | `number \| undefined` | no | Maximum time, in seconds, to wait between reconnection attempts. | | `wsCloseTimeout` | `number \| undefined` | no | Maximum time, in seconds, to wait for the peer to acknowledge the websocket closing handshake before dropping the connection. Applied to connections opened through `_websocketConnect()`. Increase it for peers that need longer to complete a graceful close. | | `wsSendTimeout` | `number \| null \| undefined` | no | Maximum time, in seconds, a single send may wait on a peer that has stopped reading before the connection is dropped and the send throws `ConnectionClosedError`. Applied to connections opened through `_websocketConnect()`. Pass `null` to wait indefinitely. Defaults to `WS_SEND_TIMEOUT`. | | `audioPassthrough` | `boolean \| undefined` | no | Whether to pass audio frames downstream after processing. Defaults to true. | | `sampleRate` | `number \| null \| undefined` | no | The sample rate for audio input. If null, will be determined from the start frame. | | `sttTtfbTimeout` | `number \| undefined` | no | Time in seconds to wait after VAD stop before reporting TTFB. This delay allows the final transcription to arrive. Defaults to 2.0. | | `ttfsP99Latency` | `number \| null \| undefined` | no | P99 latency from speech end to final transcript in seconds. Broadcast via STTMetadataFrame at pipeline start. | | `keepaliveTimeout` | `number \| null \| undefined` | no | Seconds of no audio before sending silence to keep the connection alive. null disables keepalive. | | `keepaliveInterval` | `number \| undefined` | no | Seconds between idle checks when keepalive is enabled. | | `reconnectOnError` | `boolean \| undefined` | no | Whether to automatically reconnect on websocket errors. Defaults to true. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new XAISTTService.Settings({...})` (`XAISTTSettings`). Any setting you leave out keeps the default below. | Setting | Type | Default | Description | | ---------------- | ------------------ | ----------------------------- | ------------------------------------------------------------------------------ | | `model` | `string` | `"grok-voice-transcribe-2.0"` | Transcription model. | | `language` | `Language \| null` | `Language.EN` | Language of the audio, sent as a two-letter code. | | `interimResults` | `boolean` | `true` | Partial transcripts about every 500 ms. | | `endpointing` | `number \| null` | `null` | Silence, in ms (0 to 5000), that ends an utterance. The server default is 400. | | `multichannel` | `boolean \| null` | `null` | Transcribe each interleaved channel on its own. Needs `channels` of 2 or more. | | `channels` | `number \| null` | `null` | Number of interleaved channels (2 to 8). | | `diarize` | `boolean \| null` | `null` | Attach a `speaker` to each word. | ### Usage [#usage] ```ts title="stt.ts" import { STTUpdateSettingsFrame } from "@pipedog/core/frames/frames"; import { XAISTTService } from "@pipedog/xai/stt"; const stt = new XAISTTService({ apiKey: process.env.XAI_API_KEY ?? "", settings: new XAISTTService.Settings({ endpointing: 300 }), }); ``` Change settings mid-call with an `STTUpdateSettingsFrame`: ```ts // Queue it on the worker, or push it from a processor upstream of the service. const update = new STTUpdateSettingsFrame({ delta: new XAISTTService.Settings({ endpointing: 500 }), }); ``` ### Event handlers [#event-handlers] | Event | Description | | ----------------- | ------------------------ | | `on_connected` | The WebSocket connected. | | `on_disconnected` | The WebSocket closed. | ### Notes [#notes] * `encoding` is `"pcm"` (16-bit little-endian) by default; `"mulaw"` and `"alaw"` also work. `sampleRate` defaults to the pipeline's input rate. * Settings are query parameters of the connection, so any settings change reconnects. * Audio is only sent once the server reports the session ready (`transcript.created`). ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/xai/package.json`: ✅ means the compat smoke test passes on that runtime. * The WebSocket goes through the core `websocketConnect`, with headers on the handshake. It works on Bun, Node, Deno and Cloudflare Workers. * Tested against a local WebSocket server, as upstream's tests do, not against xAI itself. # Zoom AI > Streaming speech-to-text with Zoom's Scribe Live, which ends each turn with its own voice activity detection. Source: https://polvallverdu.github.io/pipedog/docs/services/stt/zoom-ai/ ## Overview [#overview] `ZoomAISTTService` streams 16 kHz PCM audio to Zoom's Scribe Live over a WebSocket. Scribe's own voice activity detection decides when the user starts and stops speaking, and each completed turn comes back as one final transcript. It is a port of Pipecat's `ZoomAISTTService`. Pipecat's documentation for the same service. `packages/zoom-ai/src/stt.ts`. The Scribe API: features, availability and enablement. ## Installation [#installation] ```bash bun add @pipedog/zoom-ai ``` ## Prerequisites [#prerequisites] * A Zoom AI Services API key or JWT, passed as `apiKey` (the examples read `ZOOM_AI_API_KEY`). It is sent on every connect and reconnect, so it must stay valid for as long as the service runs. ## ZoomAISTTService [#zoomaisttservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | Zoom AI Services API key or JWT. It is sent on every request and reconnect, so it must stay valid for as long as the service runs. | | `url` | `string \| undefined` | no | Scribe Live WebSocket URL. A plaintext `ws://` URL to a host other than localhost logs a warning. | | `sampleRate` | `number \| null \| undefined` | no | Input audio sample rate. Audio is resampled to 16 kHz. | | `shouldInterrupt` | `boolean \| undefined` | no | Whether a user turn started by Scribe interrupts the bot. Defaults to true. | | `settings` | `ZoomAISTTSettings \| null \| undefined` | no | Runtime-updatable settings. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `reconnectBackoffMinWait` | `number \| undefined` | no | Minimum time, in seconds, to wait between reconnection attempts. | | `reconnectBackoffMaxWait` | `number \| undefined` | no | Maximum time, in seconds, to wait between reconnection attempts. | | `wsCloseTimeout` | `number \| undefined` | no | Maximum time, in seconds, to wait for the peer to acknowledge the websocket closing handshake before dropping the connection. Applied to connections opened through `_websocketConnect()`. Increase it for peers that need longer to complete a graceful close. | | `wsSendTimeout` | `number \| null \| undefined` | no | Maximum time, in seconds, a single send may wait on a peer that has stopped reading before the connection is dropped and the send throws `ConnectionClosedError`. Applied to connections opened through `_websocketConnect()`. Pass `null` to wait indefinitely. Defaults to `WS_SEND_TIMEOUT`. | | `audioPassthrough` | `boolean \| undefined` | no | Whether to pass audio frames downstream after processing. Defaults to true. | | `sttTtfbTimeout` | `number \| undefined` | no | Time in seconds to wait after VAD stop before reporting TTFB. This delay allows the final transcription to arrive. Defaults to 2.0. | | `ttfsP99Latency` | `number \| null \| undefined` | no | P99 latency from speech end to final transcript in seconds. Broadcast via STTMetadataFrame at pipeline start. | | `keepaliveTimeout` | `number \| null \| undefined` | no | Seconds of no audio before sending silence to keep the connection alive. null disables keepalive. | | `keepaliveInterval` | `number \| undefined` | no | Seconds between idle checks when keepalive is enabled. | | `reconnectOnError` | `boolean \| undefined` | no | Whether to automatically reconnect on websocket errors. Defaults to true. | The default URL is `wss://api.zoom.us/v2/aiservices/scribe/live`. A plaintext `ws://` URL to a host other than localhost logs a warning. ### Settings [#settings] Runtime-updatable settings, passed as `settings: new ZoomAISTTService.Settings({...})` (`ZoomAISTTSettings`). Settings you leave out keep their defaults. Changing a setting reconnects with a new Scribe session. | Setting | Type | Default | Description | | ------------------- | ------------------ | ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `language` | `Language` | `Language.EN_US` | The Scribe locale. Regional variants Scribe doesn't list resolve to their base language's locale. | | `silenceDurationMs` | `number \| null` | `350` | Silence that ends a turn, tuned for voice agents. For transcription without an agent, a longer value such as 600 can improve accuracy. `null` uses Scribe's default. | | `diarization` | `boolean` | `false` | Label each transcript with a speaker. The labels are in the Scribe event attached as `TranscriptionFrame.result`, and the speaker tags are removed from the text. | | `languageDetection` | `string \| null` | `null` | Scribe's language detection mode (`config.language_detection.mode`). `null` turns it off. | | `vocabulary` | `string[] \| null` | `null` | Words and phrases that customize recognition. | | `textPolishing` | `boolean` | `false` | Scribe's text polishing. The polished text is the transcript, and the recognized text stays in `TranscriptionFrame.result`. | | `extra` | `object` | `{}` | Scribe request fields with no setting, shaped like the request. Merged into the `session.update` over the settings above. | ### Usage [#usage] ```ts title="stt.ts" import { STTUpdateSettingsFrame } from "@pipedog/core/frames/frames"; import { Language } from "@pipedog/core/transcriptions/language"; import { ZoomAISTTService } from "@pipedog/zoom-ai/stt"; const stt = new ZoomAISTTService({ apiKey: process.env.ZOOM_AI_API_KEY ?? "", settings: new ZoomAISTTService.Settings({ language: Language.EN_US, vocabulary: ["pipedog", "Pipecat"], }), }); ``` Turn on Scribe's other features: ```ts // Speaker labels, text polishing, and a Scribe field the settings don't cover, through `extra`. const transcriber = new ZoomAISTTService({ apiKey: process.env.ZOOM_AI_API_KEY ?? "", settings: new ZoomAISTTService.Settings({ silenceDurationMs: 600, diarization: true, textPolishing: true, extra: { config: { profanity_filter: true } }, }), }); ``` Change settings mid-call with an `STTUpdateSettingsFrame`: ```ts // A settings change reconnects with a new Scribe session. const update = new STTUpdateSettingsFrame({ delta: new ZoomAISTTService.Settings({ language: Language.ES }), }); ``` ### Notes [#notes] * Scribe's `speech_started` event broadcasts a `ProposedUserStartedSpeakingFrame`, and each completed transcript is pushed as a `TranscriptionFrame` followed by a `ProposedUserStoppedSpeakingFrame`. The service recommends `ExternalUserTurnStrategies`, so the user aggregator follows Scribe's turns instead of running a local VAD. * Only final transcripts are pushed. There are no interim transcriptions, and no time to final segment (TTFS) metric, since Scribe sets the turn boundary. * `shouldInterrupt` (default `true`) decides whether a user turn started by Scribe interrupts the bot. * Audio at another sample rate is resampled to 16 kHz. When input stops mid-turn (the user muted, say), the service sends enough silence after a second to end the turn. * When Zoom refuses the connection, the error carries Zoom's reason when the WebSocket error exposes the handshake response. A standard `WebSocket` doesn't, so the reason may be missing. ## Runtimes [#runtimes] | Runtime | Works | What it needs | | ------------------ | -------------- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | — not verified | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/zoom-ai/package.json`: ✅ means the compat smoke test passes on that runtime. The service uses the global `WebSocket` (through core's `websocketConnect`, which sends the auth headers on the handshake on Cloudflare Workers too). No Zoom SDK is needed. # AssemblyAI > Streaming text-to-speech with per-word timings over AssemblyAI's Streaming TTS WebSocket API. Source: https://polvallverdu.github.io/pipedog/docs/services/tts/assemblyai/ ## Overview [#overview] `AssemblyAITTSService` streams text to AssemblyAI's Streaming TTS WebSocket API over one session and plays the audio as it arrives, with per-word timings for word-level `TTSTextFrame`s. It is a port of Pipecat's `AssemblyAITTSService`. AssemblyAI's speech-to-text services, in the same package, are on the [AssemblyAI speech-to-text](https://polvallverdu.github.io/pipedog/docs/services/stt/assemblyai.md) page. Pipecat's documentation for the same service. `packages/assemblyai/src/tts.ts`. ## Installation [#installation] ```bash bun add @pipedog/assemblyai ``` ## Prerequisites [#prerequisites] * An [AssemblyAI](https://www.assemblyai.com) account and API key, passed as `apiKey` (the examples read `ASSEMBLYAI_API_KEY`). A temporary token opens a single session, so it can't carry the service's reconnects and session renewals. ## AssemblyAITTSService [#assemblyaittsservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | AssemblyAI API key. A temporary token opens a single session, so it can't carry the service's reconnects and session renewals. | | `url` | `string \| undefined` | no | Streaming TTS WebSocket URL. Use `wss://tts.eu.assemblyai.com/v1/ws` to process text and audio in the EU. A cloned voice works only on the host of the zone it was created in. | | `sampleRate` | `number \| null \| undefined` | no | Output sample rate in Hz. If null, uses the pipeline's output rate. | | `settings` | `AssemblyAITTSSettings \| null \| undefined` | no | Runtime-updatable settings. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `reconnectBackoffMinWait` | `number \| undefined` | no | Minimum time, in seconds, to wait between reconnection attempts. | | `reconnectBackoffMaxWait` | `number \| undefined` | no | Maximum time, in seconds, to wait between reconnection attempts. | | `reconnectOnError` | `boolean \| undefined` | no | Whether to automatically reconnect on connection errors. | | `wsCloseTimeout` | `number \| undefined` | no | Maximum time, in seconds, to wait for the peer to acknowledge the websocket closing handshake before dropping the connection. Applied to connections opened through `_websocketConnect()`. Increase it for peers that need longer to complete a graceful close. | | `wsSendTimeout` | `number \| null \| undefined` | no | Maximum time, in seconds, a single send may wait on a peer that has stopped reading before the connection is dropped and the send throws `ConnectionClosedError`. Applied to connections opened through `_websocketConnect()`. Pass `null` to wait indefinitely. Defaults to `WS_SEND_TIMEOUT`. | | `textAggregationMode` | `TextAggregationMode \| null \| undefined` | no | How to aggregate incoming text before synthesis. `TextAggregationMode.SENTENCE` (default) buffers until sentence boundaries, `TextAggregationMode.TOKEN` streams tokens directly for lower latency. | | `pushTextFrames` | `boolean \| undefined` | no | Whether to push TextFrames and LLMFullResponseEndFrames. | | `pushStopFrames` | `boolean \| undefined` | no | Whether to automatically push TTSStoppedFrames. | | `pushStartFrame` | `boolean \| undefined` | no | Whether to automatically create audio contexts and push TTSStartedFrames. When true, the base class handles `createAudioContext` and yields `TTSStartedFrame` before each synthesis call, so `runTts` implementations do not need to. | | `stopFrameTimeoutS` | `number \| undefined` | no | Idle time before pushing TTSStoppedFrame when pushStopFrames is true. | | `pushSilenceAfterStop` | `boolean \| undefined` | no | Whether to push silence audio after TTSStoppedFrame. | | `silenceTimeS` | `number \| undefined` | no | Duration of silence to push when pushSilenceAfterStop is true. | | `pauseFrameProcessing` | `boolean \| undefined` | no | Whether to hold incoming frames after a turn until its audio contexts end. Needed by services whose audio doesn't carry its request's context ID. | | `maxConsecutiveZeroAudioContexts` | `number \| undefined` | no | How many consecutive TTS contexts may complete without producing any audio before the service is reported unable to do its job. Catches a provider that accepts requests and stays silent — an unknown voice ID, say — which no error ever surfaces. Every silent context reports an error the service can carry on from, whatever the limit is; on reaching the limit the service reports a permanent error instead, stops being given work, and the pipeline worker applies its `ProcessorUnusablePolicy`. Set to 0 to report silent contexts without ever writing the service off. | | `appendTrailingSpace` | `boolean \| undefined` | no | Whether to append a trailing space to text before sending to TTS. This helps prevent some TTS services from vocalizing trailing punctuation (e.g., "dot"). Only applied in sentence aggregation mode; when streaming tokens, the incoming text's own whitespace is preserved. | | `skipTextTypes` | `string[] \| null \| undefined` | no | List of text types that should not be spoken. | | `textTransforms` | `[string, TextTransform][] \| null \| undefined` | no | A list of TextTransforms to transform text just before sending it to TTS. Each TextTransform takes the aggregated text and its type, and returns the transformed text. To register, provide a list of tuples of `[textType \| "*", transformFunction]`. | | `transportDestination` | `string \| null \| undefined` | no | Destination for generated audio frames. | | `reuseContextIdWithinTurn` | `boolean \| undefined` | no | Whether the service should reuse context IDs within the same turn. | | `audioCache` | `TTSAudioCacheStore \| null \| undefined` | no | Where to keep synthesized audio for reuse (e.g. a shared `TTSAudioCache`). Text whose audio is in the cache is served from it, with no request to the provider. Lines get there through `warmAudioCache()`, and also from live synthesis when `cacheSynthesizedAudio` allows it. Only services that yield their audio from `runTts` (HTTP and streaming-RPC services, see `supportsAudioCache`) use it, and only in sentence aggregation mode. Defaults to null (no cache). pipedog only. | | `cacheSynthesizedAudio` | `boolean \| ((text: string) => boolean) \| undefined` | no | Whether audio synthesized live is stored in `audioCache`, or a function that decides per text. Defaults to false: only warmed lines are stored, since a generated reply can repeat what the user said. pipedog only. | | `audioCacheNamespace` | `string \| undefined` | no | Extra part of the cache key, for constructor options that change the audio but are not in `settings`. pipedog only. | The default URL is `wss://tts.us.assemblyai.com/v1/ws`. Use `wss://tts.eu.assemblyai.com/v1/ws` to process text and audio in the EU. A cloned voice works only on the host of the zone it was created in. ### Settings [#settings] Runtime-updatable settings, passed as `settings: new AssemblyAITTSService.Settings({...})` (`AssemblyAITTSSettings`). Settings you leave out keep their defaults. The session is configured when it connects, so changing any of them reconnects. | Setting | Type | Default | Description | | ---------- | ------------------ | ------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `voice` | `string \| null` | `"49c443a4-8a1d-5f7e-b70b-98bdbd9823c3"` (Audrey) | Sent as `voice_id`: any voice by its ID, an AssemblyAI voice by its library slug such as `aai-jane`, or a cloned voice by the slug the dashboard shows. `null` leaves the voice to the server. | | `language` | `Language \| null` | `null` | The language the session speaks: German, English, Spanish, French, Italian or Portuguese. Unset, the voice's own language. | | `model` | `string \| null` | `null` | The model, such as `assemblyai-tts-1`. Unset, the language's default model. | ### Usage [#usage] ```ts title="tts.ts" import { AssemblyAITTSService } from "@pipedog/assemblyai/tts"; import { TTSUpdateSettingsFrame } from "@pipedog/core/frames/frames"; import { Language } from "@pipedog/core/transcriptions/language"; // Speaks in Audrey, the default voice. const tts = new AssemblyAITTSService({ apiKey: process.env.ASSEMBLYAI_API_KEY ?? "" }); ``` Pick a voice and a region: ```ts // A library voice by its slug, processed in the EU. const eu = new AssemblyAITTSService({ apiKey: process.env.ASSEMBLYAI_API_KEY ?? "", url: "wss://tts.eu.assemblyai.com/v1/ws", settings: new AssemblyAITTSService.Settings({ voice: "aai-jane", language: Language.EN }), }); ``` Change settings mid-call with a `TTSUpdateSettingsFrame`: ```ts // The session is configured at connect time, so a new voice reconnects. const update = new TTSUpdateSettingsFrame({ delta: new AssemblyAITTSService.Settings({ voice: "987db62e-a2c2-569a-9c4d-82460cdea1bd" }), }); ``` ### Notes [#notes] * Each sentence is sent as its own request (a `Generate`, then a `Flush`). With `TextAggregationMode.TOKEN`, the turn's text streams as it is generated and is flushed when the turn ends. Text longer than the session's limit is split across `Generate` messages. * Interruptions send `Cancel`, which stops synthesis in flight and keeps the session open. * The service keeps an idle session open with a `KeepAlive` every 30 seconds, replaces a session between requests before its one-hour limit, and waits as long as an error asks before reconnecting. * `sampleRate` left out uses the pipeline's output rate. ## Runtimes [#runtimes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/assemblyai/package.json`: ✅ means the compat smoke test passes on that runtime. The service uses the global `WebSocket` (through core's `websocketConnect`, which sends the `Authorization` header on the handshake on Cloudflare Workers too). No AssemblyAI SDK is needed. # Async > Text-to-speech with Async: AsyncAITTSService over WebSocket and AsyncAIHttpTTSService over HTTP streaming. Source: https://polvallverdu.github.io/pipedog/docs/services/tts/asyncai/ ## Overview [#overview] * **`AsyncAITTSService`**: keeps one WebSocket open and sends each sentence as a forced transcript on its own context, closing the context when its audio has played or the bot is interrupted. * **`AsyncAIHttpTTSService`**: one HTTP request per sentence. The audio is collected and pushed as one frame. Pipecat's documentation for the same service. `packages/asyncai/src/tts.ts`. The WebSocket API reference. ## Installation [#installation] ```bash bun add @pipedog/asyncai ``` ```ts import { AsyncAIHttpTTSService, AsyncAITTSService } from "@pipedog/asyncai/tts"; ``` ## Prerequisites [#prerequisites] * An Async account, an API key and the UUID of a voice. * Pass the key as `apiKey`. The examples read it from `ASYNC_API_KEY`. ## AsyncAITTSService [#asyncaittsservice] Connects to `wss://api.async.com/text_to_speech/websocket/ws`, with the key and API version as query parameters. ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | Async API key. | | `version` | `string \| undefined` | no | Async API version. | | `url` | `string \| undefined` | no | WebSocket URL for Async TTS API. | | `encoding` | `string \| undefined` | no | Audio encoding format. | | `container` | `string \| undefined` | no | Audio container format. | | `settings` | `AsyncAITTSSettings \| null \| undefined` | no | Runtime-updatable settings. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `reconnectBackoffMinWait` | `number \| undefined` | no | Minimum time, in seconds, to wait between reconnection attempts. | | `reconnectBackoffMaxWait` | `number \| undefined` | no | Maximum time, in seconds, to wait between reconnection attempts. | | `reconnectOnError` | `boolean \| undefined` | no | Whether to automatically reconnect on connection errors. | | `wsCloseTimeout` | `number \| undefined` | no | Maximum time, in seconds, to wait for the peer to acknowledge the websocket closing handshake before dropping the connection. Applied to connections opened through `_websocketConnect()`. Increase it for peers that need longer to complete a graceful close. | | `wsSendTimeout` | `number \| null \| undefined` | no | Maximum time, in seconds, a single send may wait on a peer that has stopped reading before the connection is dropped and the send throws `ConnectionClosedError`. Applied to connections opened through `_websocketConnect()`. Pass `null` to wait indefinitely. Defaults to `WS_SEND_TIMEOUT`. | | `textAggregationMode` | `TextAggregationMode \| null \| undefined` | no | How to aggregate incoming text before synthesis. `TextAggregationMode.SENTENCE` (default) buffers until sentence boundaries, `TextAggregationMode.TOKEN` streams tokens directly for lower latency. | | `pushTextFrames` | `boolean \| undefined` | no | Whether to push TextFrames and LLMFullResponseEndFrames. | | `pushStopFrames` | `boolean \| undefined` | no | Whether to automatically push TTSStoppedFrames. | | `pushStartFrame` | `boolean \| undefined` | no | Whether to automatically create audio contexts and push TTSStartedFrames. When true, the base class handles `createAudioContext` and yields `TTSStartedFrame` before each synthesis call, so `runTts` implementations do not need to. | | `stopFrameTimeoutS` | `number \| undefined` | no | Idle time before pushing TTSStoppedFrame when pushStopFrames is true. | | `pushSilenceAfterStop` | `boolean \| undefined` | no | Whether to push silence audio after TTSStoppedFrame. | | `silenceTimeS` | `number \| undefined` | no | Duration of silence to push when pushSilenceAfterStop is true. | | `pauseFrameProcessing` | `boolean \| undefined` | no | Whether to hold incoming frames after a turn until its audio contexts end. Needed by services whose audio doesn't carry its request's context ID. | | `maxConsecutiveZeroAudioContexts` | `number \| undefined` | no | How many consecutive TTS contexts may complete without producing any audio before the service is reported unable to do its job. Catches a provider that accepts requests and stays silent — an unknown voice ID, say — which no error ever surfaces. Every silent context reports an error the service can carry on from, whatever the limit is; on reaching the limit the service reports a permanent error instead, stops being given work, and the pipeline worker applies its `ProcessorUnusablePolicy`. Set to 0 to report silent contexts without ever writing the service off. | | `appendTrailingSpace` | `boolean \| undefined` | no | Whether to append a trailing space to text before sending to TTS. This helps prevent some TTS services from vocalizing trailing punctuation (e.g., "dot"). Only applied in sentence aggregation mode; when streaming tokens, the incoming text's own whitespace is preserved. | | `sampleRate` | `number \| null \| undefined` | no | Output sample rate for generated audio. | | `skipTextTypes` | `string[] \| null \| undefined` | no | List of text types that should not be spoken. | | `textTransforms` | `[string, TextTransform][] \| null \| undefined` | no | A list of TextTransforms to transform text just before sending it to TTS. Each TextTransform takes the aggregated text and its type, and returns the transformed text. To register, provide a list of tuples of `[textType \| "*", transformFunction]`. | | `transportDestination` | `string \| null \| undefined` | no | Destination for generated audio frames. | | `reuseContextIdWithinTurn` | `boolean \| undefined` | no | Whether the service should reuse context IDs within the same turn. | | `audioCache` | `TTSAudioCacheStore \| null \| undefined` | no | Where to keep synthesized audio for reuse (e.g. a shared `TTSAudioCache`). Text whose audio is in the cache is served from it, with no request to the provider. Lines get there through `warmAudioCache()`, and also from live synthesis when `cacheSynthesizedAudio` allows it. Only services that yield their audio from `runTts` (HTTP and streaming-RPC services, see `supportsAudioCache`) use it, and only in sentence aggregation mode. Defaults to null (no cache). pipedog only. | | `cacheSynthesizedAudio` | `boolean \| ((text: string) => boolean) \| undefined` | no | Whether audio synthesized live is stored in `audioCache`, or a function that decides per text. Defaults to false: only warmed lines are stored, since a generated reply can repeat what the user said. pipedog only. | | `audioCacheNamespace` | `string \| undefined` | no | Extra part of the cache key, for constructor options that change the audio but are not in `settings`. pipedog only. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new AsyncAITTSService.Settings({...})` (`AsyncAITTSSettings`). Any setting you leave out keeps the default below. A change to `model`, `voice` or `language` opens a new session, because they are sent once, in the init message. | Setting | Type | Default | Description | | ---------- | ------------------ | -------------------- | ------------------------------------ | | `model` | `string` | `"async_flash_v1.5"` | Speech model. | | `voice` | `string \| null` | `null` | Voice UUID. | | `language` | `Language \| null` | `null` | Language, sent as a two-letter code. | ### Usage [#usage] ```ts title="tts.ts" import { AsyncAIHttpTTSService, AsyncAITTSService } from "@pipedog/asyncai/tts"; import { TTSUpdateSettingsFrame } from "@pipedog/core/frames/frames"; const tts = new AsyncAITTSService({ apiKey: process.env.ASYNC_API_KEY ?? "", settings: new AsyncAITTSService.Settings({ model: "async_flash_v1.5", voice: "your-voice-uuid", language: "en", }), }); ``` Change settings mid-call with a `TTSUpdateSettingsFrame`: ```ts // Opens a new session, since the voice is part of the init message. const update = new TTSUpdateSettingsFrame({ delta: new AsyncAITTSService.Settings({ voice: "another-voice-uuid" }), }); ``` ### Event handlers [#event-handlers] | Event | Description | | --------------------- | ------------------------ | | `on_connected` | The WebSocket connected. | | `on_disconnected` | The WebSocket closed. | | `on_connection_error` | The connection failed. | ## AsyncAIHttpTTSService [#asyncaihttpttsservice] Posts each sentence to `https://api.async.com/text_to_speech/streaming`. ### Configuration [#configuration-1] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | Async API key. | | `fetch` | `{ (input: RequestInfo \| URL, init?: RequestInit): Promise; (input: string \| URL \| Request, init?: RequestInit): Promise; } \| null \| undefined` | no | `fetch` used for HTTP requests. Defaults to the global `fetch`. DIVERGENCE: upstream takes a required shared `aiohttp_session`. | | `url` | `string \| undefined` | no | Base URL for Async API. | | `version` | `string \| undefined` | no | API version string for Async API. | | `encoding` | `string \| undefined` | no | Audio encoding format. | | `container` | `string \| undefined` | no | Audio container format. | | `settings` | `AsyncAITTSSettings \| null \| undefined` | no | Runtime-updatable settings. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `textAggregationMode` | `TextAggregationMode \| null \| undefined` | no | How to aggregate incoming text before synthesis. `TextAggregationMode.SENTENCE` (default) buffers until sentence boundaries, `TextAggregationMode.TOKEN` streams tokens directly for lower latency. | | `pushTextFrames` | `boolean \| undefined` | no | Whether to push TextFrames and LLMFullResponseEndFrames. | | `pushStopFrames` | `boolean \| undefined` | no | Whether to automatically push TTSStoppedFrames. | | `pushStartFrame` | `boolean \| undefined` | no | Whether to automatically create audio contexts and push TTSStartedFrames. When true, the base class handles `createAudioContext` and yields `TTSStartedFrame` before each synthesis call, so `runTts` implementations do not need to. | | `stopFrameTimeoutS` | `number \| undefined` | no | Idle time before pushing TTSStoppedFrame when pushStopFrames is true. | | `pushSilenceAfterStop` | `boolean \| undefined` | no | Whether to push silence audio after TTSStoppedFrame. | | `silenceTimeS` | `number \| undefined` | no | Duration of silence to push when pushSilenceAfterStop is true. | | `pauseFrameProcessing` | `boolean \| undefined` | no | Whether to hold incoming frames after a turn until its audio contexts end. Needed by services whose audio doesn't carry its request's context ID. | | `maxConsecutiveZeroAudioContexts` | `number \| undefined` | no | How many consecutive TTS contexts may complete without producing any audio before the service is reported unable to do its job. Catches a provider that accepts requests and stays silent — an unknown voice ID, say — which no error ever surfaces. Every silent context reports an error the service can carry on from, whatever the limit is; on reaching the limit the service reports a permanent error instead, stops being given work, and the pipeline worker applies its `ProcessorUnusablePolicy`. Set to 0 to report silent contexts without ever writing the service off. | | `appendTrailingSpace` | `boolean \| undefined` | no | Whether to append a trailing space to text before sending to TTS. This helps prevent some TTS services from vocalizing trailing punctuation (e.g., "dot"). Only applied in sentence aggregation mode; when streaming tokens, the incoming text's own whitespace is preserved. | | `sampleRate` | `number \| null \| undefined` | no | Output sample rate for generated audio. | | `skipTextTypes` | `string[] \| null \| undefined` | no | List of text types that should not be spoken. | | `textTransforms` | `[string, TextTransform][] \| null \| undefined` | no | A list of TextTransforms to transform text just before sending it to TTS. Each TextTransform takes the aggregated text and its type, and returns the transformed text. To register, provide a list of tuples of `[textType \| "*", transformFunction]`. | | `transportDestination` | `string \| null \| undefined` | no | Destination for generated audio frames. | | `reuseContextIdWithinTurn` | `boolean \| undefined` | no | Whether the service should reuse context IDs within the same turn. | | `audioCache` | `TTSAudioCacheStore \| null \| undefined` | no | Where to keep synthesized audio for reuse (e.g. a shared `TTSAudioCache`). Text whose audio is in the cache is served from it, with no request to the provider. Lines get there through `warmAudioCache()`, and also from live synthesis when `cacheSynthesizedAudio` allows it. Only services that yield their audio from `runTts` (HTTP and streaming-RPC services, see `supportsAudioCache`) use it, and only in sentence aggregation mode. Defaults to null (no cache). pipedog only. | | `cacheSynthesizedAudio` | `boolean \| ((text: string) => boolean) \| undefined` | no | Whether audio synthesized live is stored in `audioCache`, or a function that decides per text. Defaults to false: only warmed lines are stored, since a generated reply can repeat what the user said. pipedog only. | | `audioCacheNamespace` | `string \| undefined` | no | Extra part of the cache key, for constructor options that change the audio but are not in `settings`. pipedog only. | It takes the same settings as the WebSocket service. ### Usage [#usage-1] ```ts const httpTts = new AsyncAIHttpTTSService({ apiKey: process.env.ASYNC_API_KEY ?? "", settings: new AsyncAIHttpTTSService.Settings({ voice: "your-voice-uuid" }), }); ``` ## Notes [#notes] * Audio is raw `pcm_s16le` by default (`encoding` and `container` change that) at the pipeline's sample rate. * The WebSocket sends a blank transcript every 10 seconds to keep the connection open. ## Runtime notes [#runtime-notes] * The WebSocket opens through the core `websocketConnect`. Async takes the key in the URL, so no header is needed. The HTTP service uses the global `fetch`; pass `fetch` to use another one (upstream takes an aiohttp session). Both work on Bun, Node, Deno and Cloudflare Workers. * The deprecated `params` (`InputParams`), `voiceId`, `model` and `aggregateSentences` options are not ported. Use `settings` and `textAggregationMode`. * Tested against local servers, as upstream's tests do for the settings updates, not against Async itself. | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/asyncai/package.json`: ✅ means the compat smoke test passes on that runtime. # AWS Polly > Text-to-speech from Amazon Polly with AWSPollyTTSService, with SSML prosody, engines and lexicons. Source: https://polvallverdu.github.io/pipedog/docs/services/tts/aws/ ## Overview [#overview] `AWSPollyTTSService` synthesizes each sentence with Amazon Polly's `SynthesizeSpeech` API. It wraps the text in SSML (with prosody when `rate`, `pitch` or `volume` are set), requests 16 kHz PCM and resamples it to the pipeline's output rate. Pipecat's documentation for the same service. `packages/aws/src/tts.ts`. Voices, engines and SSML. ## Installation [#installation] ```bash bun add @pipedog/aws ``` ```ts import { AWSPollyTTSService } from "@pipedog/aws/tts"; ``` ## Prerequisites [#prerequisites] * An IAM identity allowed `polly:SynthesizeSpeech`. * Keys, in this order of precedence: 1. `awsAccessKeyId`, `apiKey` (the secret access key) and `awsSessionToken`. 2. The `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` and `AWS_SESSION_TOKEN` environment variables. 3. A `credentialsProvider` function, called for every request. The region comes from `region`, else `AWS_REGION`, else `us-east-1`. ## AWSPollyTTSService [#awspollyttsservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string \| null \| undefined` | no | AWS secret access key. If null, falls back to environment variables and `credentialsProvider`. | | `awsAccessKeyId` | `string \| null \| undefined` | no | AWS access key ID. Same fallback behaviour as `apiKey`. | | `awsSessionToken` | `string \| null \| undefined` | no | AWS session token for temporary credentials. | | `region` | `string \| null \| undefined` | no | AWS region for Polly service. Defaults to 'us-east-1'. | | `settings` | `AWSPollyTTSSettings \| null \| undefined` | no | Runtime-updatable settings. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `textAggregationMode` | `TextAggregationMode \| null \| undefined` | no | How to aggregate incoming text before synthesis. `TextAggregationMode.SENTENCE` (default) buffers until sentence boundaries, `TextAggregationMode.TOKEN` streams tokens directly for lower latency. | | `pushTextFrames` | `boolean \| undefined` | no | Whether to push TextFrames and LLMFullResponseEndFrames. | | `pushStopFrames` | `boolean \| undefined` | no | Whether to automatically push TTSStoppedFrames. | | `pushStartFrame` | `boolean \| undefined` | no | Whether to automatically create audio contexts and push TTSStartedFrames. When true, the base class handles `createAudioContext` and yields `TTSStartedFrame` before each synthesis call, so `runTts` implementations do not need to. | | `stopFrameTimeoutS` | `number \| undefined` | no | Idle time before pushing TTSStoppedFrame when pushStopFrames is true. | | `pushSilenceAfterStop` | `boolean \| undefined` | no | Whether to push silence audio after TTSStoppedFrame. | | `silenceTimeS` | `number \| undefined` | no | Duration of silence to push when pushSilenceAfterStop is true. | | `pauseFrameProcessing` | `boolean \| undefined` | no | Whether to hold incoming frames after a turn until its audio contexts end. Needed by services whose audio doesn't carry its request's context ID. | | `maxConsecutiveZeroAudioContexts` | `number \| undefined` | no | How many consecutive TTS contexts may complete without producing any audio before the service is reported unable to do its job. Catches a provider that accepts requests and stays silent — an unknown voice ID, say — which no error ever surfaces. Every silent context reports an error the service can carry on from, whatever the limit is; on reaching the limit the service reports a permanent error instead, stops being given work, and the pipeline worker applies its `ProcessorUnusablePolicy`. Set to 0 to report silent contexts without ever writing the service off. | | `appendTrailingSpace` | `boolean \| undefined` | no | Whether to append a trailing space to text before sending to TTS. This helps prevent some TTS services from vocalizing trailing punctuation (e.g., "dot"). Only applied in sentence aggregation mode; when streaming tokens, the incoming text's own whitespace is preserved. | | `sampleRate` | `number \| null \| undefined` | no | Output sample rate for generated audio. | | `skipTextTypes` | `string[] \| null \| undefined` | no | List of text types that should not be spoken. | | `textTransforms` | `[string, TextTransform][] \| null \| undefined` | no | A list of TextTransforms to transform text just before sending it to TTS. Each TextTransform takes the aggregated text and its type, and returns the transformed text. To register, provide a list of tuples of `[textType \| "*", transformFunction]`. | | `transportDestination` | `string \| null \| undefined` | no | Destination for generated audio frames. | | `reuseContextIdWithinTurn` | `boolean \| undefined` | no | Whether the service should reuse context IDs within the same turn. | | `audioCache` | `TTSAudioCacheStore \| null \| undefined` | no | Where to keep synthesized audio for reuse (e.g. a shared `TTSAudioCache`). Text whose audio is in the cache is served from it, with no request to the provider. Lines get there through `warmAudioCache()`, and also from live synthesis when `cacheSynthesizedAudio` allows it. Only services that yield their audio from `runTts` (HTTP and streaming-RPC services, see `supportsAudioCache`) use it, and only in sentence aggregation mode. Defaults to null (no cache). pipedog only. | | `cacheSynthesizedAudio` | `boolean \| ((text: string) => boolean) \| undefined` | no | Whether audio synthesized live is stored in `audioCache`, or a function that decides per text. Defaults to false: only warmed lines are stored, since a generated reply can repeat what the user said. pipedog only. | | `audioCacheNamespace` | `string \| undefined` | no | Extra part of the cache key, for constructor options that change the audio but are not in `settings`. pipedog only. | | `credentialsProvider` | `AWSCredentialsProvider \| null \| undefined` | no | Keys to sign with when none are passed in or set in the environment. Called for every request or connection, so it can renew temporary credentials. | | `fetch` | `FetchLike \| null \| undefined` | no | `fetch` used for HTTP calls (defaults to the global `fetch`). | | `endpointUrl` | `string \| null \| undefined` | no | Override the service endpoint (e.g. a VPC endpoint or a local fake), `https://host[:port]`. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new AWSPollyTTSService.Settings({...})`. | Setting | Type | Default | Description | | -------------- | ------------------ | ---------- | --------------------------------------------------------------------------------------- | | `voice` | `string` | `"Joanna"` | Polly voice ID. | | `language` | `string` | `"en-US"` | Polly language code. A `Language` is mapped with `languageToAwsLanguage`. | | `engine` | `string \| null` | `null` | `"standard"`, `"neural"`, `"long-form"` or `"generative"`. `null` uses Polly's default. | | `rate` | `string \| null` | `null` | SSML prosody rate, e.g. `"110%"` or `"slow"`. | | `pitch` | `string \| null` | `null` | SSML prosody pitch (standard engine only). | | `volume` | `string \| null` | `null` | SSML prosody volume, e.g. `"loud"` or `"+3dB"`. | | `lexiconNames` | `string[] \| null` | `null` | Pronunciation lexicons to apply. | ### Usage [#usage] ```ts title="tts.ts" import { AWSPollyTTSService } from "@pipedog/aws/tts"; import { TTSUpdateSettingsFrame } from "@pipedog/core/frames/frames"; const tts = new AWSPollyTTSService({ // Or leave these out to read the AWS_* environment variables. awsAccessKeyId: process.env.AWS_ACCESS_KEY_ID, apiKey: process.env.AWS_SECRET_ACCESS_KEY, region: "us-east-1", sampleRate: 24000, settings: new AWSPollyTTSService.Settings({ voice: "Joanna", engine: "generative", language: "en-US", rate: "105%", }), }); ``` Change the voice mid-call with a `TTSUpdateSettingsFrame`: ```ts // Queue it on the worker, or push it from a processor upstream of the service. const update = new TTSUpdateSettingsFrame({ delta: new AWSPollyTTSService.Settings({ voice: "Matthew" }), }); ``` ### Notes [#notes] * One HTTP request per sentence. The audio is pushed as `TTSAudioRawFrame`s of `chunkSize` bytes once the response has arrived. * Text is XML-escaped and sent as SSML (`TextType: "ssml"`). * Any failed request (an HTTP error, a network error, missing credentials) becomes an `ErrorFrame`. Upstream does this for botocore errors only. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/aws/package.json`: ✅ means the compat smoke test passes on that runtime. * Runs on Bun, Node, Deno and Cloudflare Workers: `POST /v1/speech` with `fetch`, signed with SigV4 on WebCrypto. * There is no default AWS credential chain (profiles, IMDS, ECS roles, SSO). Use keys, the environment, or `credentialsProvider`. * `endpointUrl` and `fetch` replace the Polly host and the global `fetch`. * Not ported: the deprecated `InputParams`, and the `voiceId` and `params` constructor options. Use `settings`. # Azure Speech > Neural voices from Azure AI Speech with AzureTTSService (streaming, with word timestamps) and AzureHttpTTSService (REST). Source: https://polvallverdu.github.io/pipedog/docs/services/tts/azure/ ## Overview [#overview] pipedog ports both of Pipecat's Azure text-to-speech services: * **`AzureTTSService`**: streams each sentence's audio over the Speech service's WebSocket synthesis protocol, with word boundaries turned into word-timed `TTSTextFrame`s. * **`AzureHttpTTSService`**: one `POST` to the REST endpoint per sentence, returning the whole sentence's audio. Both build the same SSML from their settings (voice, style, prosody, emphasis, HD voice parameters, locale pinning). Upstream drives Azure through the native Speech SDK; pipedog speaks the service's WebSocket and REST protocols directly, so no SDK is needed. Pipecat's documentation for the same services. `packages/azure/src/tts.ts` and `speech_protocol.ts`. Voices, SSML styles and prosody. ## Installation [#installation] ```bash bun add @pipedog/azure ``` ```ts import { AzureHttpTTSService, AzureTTSService } from "@pipedog/azure/tts"; ``` ## Prerequisites [#prerequisites] * An Azure AI Speech (or Azure AI Foundry) resource, with its key and region from the portal's **Keys and Endpoint** page. * Pass the key as `apiKey` and the region (e.g. `"eastus"`) as `region`, or a Private Link or custom-domain URL as `privateEndpoint`. The examples read them from `AZURE_SPEECH_API_KEY` and `AZURE_SPEECH_REGION`. ## AzureTTSService [#azurettsservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `webSocketFactory` | `((uri: string, options: { headers?: Record; protocols?: string[]; }) => WebSocket \| Promise) \| null \| undefined` | no | Creates the underlying `WebSocket`, to open it some other way. Defaults to `websocketConnect`'s. Not in upstream. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `textAggregationMode` | `TextAggregationMode \| null \| undefined` | no | How to aggregate incoming text before synthesis. `TextAggregationMode.SENTENCE` (default) buffers until sentence boundaries, `TextAggregationMode.TOKEN` streams tokens directly for lower latency. | | `pushTextFrames` | `boolean \| undefined` | no | Whether to push TextFrames and LLMFullResponseEndFrames. | | `pushStopFrames` | `boolean \| undefined` | no | Whether to automatically push TTSStoppedFrames. | | `pushStartFrame` | `boolean \| undefined` | no | Whether to automatically create audio contexts and push TTSStartedFrames. When true, the base class handles `createAudioContext` and yields `TTSStartedFrame` before each synthesis call, so `runTts` implementations do not need to. | | `stopFrameTimeoutS` | `number \| undefined` | no | Idle time before pushing TTSStoppedFrame when pushStopFrames is true. | | `pushSilenceAfterStop` | `boolean \| undefined` | no | Whether to push silence audio after TTSStoppedFrame. | | `silenceTimeS` | `number \| undefined` | no | Duration of silence to push when pushSilenceAfterStop is true. | | `pauseFrameProcessing` | `boolean \| undefined` | no | Whether to hold incoming frames after a turn until its audio contexts end. Needed by services whose audio doesn't carry its request's context ID. | | `maxConsecutiveZeroAudioContexts` | `number \| undefined` | no | How many consecutive TTS contexts may complete without producing any audio before the service is reported unable to do its job. Catches a provider that accepts requests and stays silent — an unknown voice ID, say — which no error ever surfaces. Every silent context reports an error the service can carry on from, whatever the limit is; on reaching the limit the service reports a permanent error instead, stops being given work, and the pipeline worker applies its `ProcessorUnusablePolicy`. Set to 0 to report silent contexts without ever writing the service off. | | `appendTrailingSpace` | `boolean \| undefined` | no | Whether to append a trailing space to text before sending to TTS. This helps prevent some TTS services from vocalizing trailing punctuation (e.g., "dot"). Only applied in sentence aggregation mode; when streaming tokens, the incoming text's own whitespace is preserved. | | `sampleRate` | `number \| null \| undefined` | no | Output sample rate for generated audio. | | `skipTextTypes` | `string[] \| null \| undefined` | no | List of text types that should not be spoken. | | `textTransforms` | `[string, TextTransform][] \| null \| undefined` | no | A list of TextTransforms to transform text just before sending it to TTS. Each TextTransform takes the aggregated text and its type, and returns the transformed text. To register, provide a list of tuples of `[textType \| "*", transformFunction]`. | | `transportDestination` | `string \| null \| undefined` | no | Destination for generated audio frames. | | `reuseContextIdWithinTurn` | `boolean \| undefined` | no | Whether the service should reuse context IDs within the same turn. | | `audioCache` | `TTSAudioCacheStore \| null \| undefined` | no | Where to keep synthesized audio for reuse (e.g. a shared `TTSAudioCache`). Text whose audio is in the cache is served from it, with no request to the provider. Lines get there through `warmAudioCache()`, and also from live synthesis when `cacheSynthesizedAudio` allows it. Only services that yield their audio from `runTts` (HTTP and streaming-RPC services, see `supportsAudioCache`) use it, and only in sentence aggregation mode. Defaults to null (no cache). pipedog only. | | `cacheSynthesizedAudio` | `boolean \| ((text: string) => boolean) \| undefined` | no | Whether audio synthesized live is stored in `audioCache`, or a function that decides per text. Defaults to false: only warmed lines are stored, since a generated reply can repeat what the user said. pipedog only. | | `audioCacheNamespace` | `string \| undefined` | no | Extra part of the cache key, for constructor options that change the audio but are not in `settings`. pipedog only. | | `apiKey` | `string` | yes | Azure Cognitive Services subscription key. | | `region` | `string \| null \| undefined` | no | Azure region identifier (e.g., "eastus", "westus2"). Required unless `privateEndpoint` is provided. | | `privateEndpoint` | `string \| null \| undefined` | no | Custom endpoint URL for Azure Speech Services (e.g., "https://my-resource.cognitiveservices.azure.com/"). Use this when connecting via Private Link or a custom domain. See https://learn.microsoft.com/en-us/azure/ai-services/speech-service/speech-services-private-link | | `settings` | `AzureTTSSettings \| null \| undefined` | no | Runtime-updatable settings. | | `fetch` | `{ (input: RequestInfo \| URL, init?: RequestInit): Promise; (input: string \| URL \| Request, init?: RequestInit): Promise; } \| null \| undefined` | no | `fetch` used for HTTP requests (the REST synthesis, and the custom-domain redirect lookup). Defaults to the global `fetch`. Not in upstream. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new AzureTTSService.Settings({...})`. Any setting you leave out keeps the default below. `AzureHttpTTSService` uses the same settings class. | Setting | Type | Default | Description | | ----------------- | -------------------- | -------------------- | ---------------------------------------------------------------------------------------------------------- | | `voice` | `string` | `"en-US-SaraNeural"` | Azure voice name. | | `language` | `Language \| string` | `"en-US"` | `xml:lang` of the SSML. | | `forceLocale` | `boolean` | `false` | Wrap the text in `` so a multilingual voice speaks in `language` instead of auto-detecting. | | `style` | `string \| null` | `null` | Speaking style (`"cheerful"`, `"sad"`, ...), as ``. | | `styleDegree` | `string \| null` | `null` | Style intensity, 0.01 to 2.0. | | `role` | `string \| null` | `null` | Role-play voice (`"YoungAdultFemale"`, ...). | | `rate` | `string \| null` | `null` | Prosody rate (`"1.25"`, `"slow"`). | | `pitch` | `string \| null` | `null` | Prosody pitch (`"+10%"`, `"-5Hz"`). | | `volume` | `string \| null` | `null` | Prosody volume (`"+20%"`, `"loud"`). | | `emphasis` | `string \| null` | `null` | `"strong"`, `"moderate"` or `"reduced"`. | | `effect` | `string \| null` | `null` | Device effect: `"eq_car"`, or `"eq_telecomhp8k"` with an 8 kHz pipeline. | | `voiceParameters` | `string \| null` | `null` | HD voice parameters (`"temperature=0.7;top_p=0.8"`). | ### Usage [#usage] ```ts title="tts.ts" import { AzureHttpTTSService, AzureTTSService } from "@pipedog/azure/tts"; import { TTSUpdateSettingsFrame } from "@pipedog/core/frames/frames"; const tts = new AzureTTSService({ apiKey: process.env.AZURE_SPEECH_API_KEY ?? "", region: process.env.AZURE_SPEECH_REGION ?? "eastus", settings: new AzureTTSService.Settings({ voice: "en-US-AvaMultilingualNeural", language: "en-US", style: "cheerful", rate: "1.1", }), }); ``` Change settings mid-call with a `TTSUpdateSettingsFrame`. The SSML is built per sentence, so the change applies from the next one: ```ts // Queue it on the worker, or push it from a processor upstream of the service. // SSML settings apply from the next sentence. const update = new TTSUpdateSettingsFrame({ delta: new AzureTTSService.Settings({ language: "en-GB", forceLocale: true }), }); ``` ### Notes [#notes] * The synthesis connection opens on the first sentence and is reused. Audio is raw 16-bit mono PCM at the pipeline's output rate (8, 16, 22.05, 24, 44.1 or 48 kHz; other rates fall back to 24 kHz). * Word boundaries become `TTSTextFrame`s timed against the audio. Trailing punctuation is merged into the word before it, and Chinese, Japanese and Korean characters are merged up to the next punctuation or space. The last word always arrives before the `TTSStoppedFrame`. * On an interruption the connection is closed, which stops Azure from sending more audio for the interrupted sentence; the next sentence opens a new one. Upstream calls the SDK's `stop_speaking_async()` instead. ## AzureHttpTTSService [#azurehttpttsservice] ### Configuration [#configuration-1] | Option | Type | Required | Description | |---|---|---|---| | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `textAggregationMode` | `TextAggregationMode \| null \| undefined` | no | How to aggregate incoming text before synthesis. `TextAggregationMode.SENTENCE` (default) buffers until sentence boundaries, `TextAggregationMode.TOKEN` streams tokens directly for lower latency. | | `pushTextFrames` | `boolean \| undefined` | no | Whether to push TextFrames and LLMFullResponseEndFrames. | | `pushStopFrames` | `boolean \| undefined` | no | Whether to automatically push TTSStoppedFrames. | | `pushStartFrame` | `boolean \| undefined` | no | Whether to automatically create audio contexts and push TTSStartedFrames. When true, the base class handles `createAudioContext` and yields `TTSStartedFrame` before each synthesis call, so `runTts` implementations do not need to. | | `stopFrameTimeoutS` | `number \| undefined` | no | Idle time before pushing TTSStoppedFrame when pushStopFrames is true. | | `pushSilenceAfterStop` | `boolean \| undefined` | no | Whether to push silence audio after TTSStoppedFrame. | | `silenceTimeS` | `number \| undefined` | no | Duration of silence to push when pushSilenceAfterStop is true. | | `pauseFrameProcessing` | `boolean \| undefined` | no | Whether to hold incoming frames after a turn until its audio contexts end. Needed by services whose audio doesn't carry its request's context ID. | | `maxConsecutiveZeroAudioContexts` | `number \| undefined` | no | How many consecutive TTS contexts may complete without producing any audio before the service is reported unable to do its job. Catches a provider that accepts requests and stays silent — an unknown voice ID, say — which no error ever surfaces. Every silent context reports an error the service can carry on from, whatever the limit is; on reaching the limit the service reports a permanent error instead, stops being given work, and the pipeline worker applies its `ProcessorUnusablePolicy`. Set to 0 to report silent contexts without ever writing the service off. | | `appendTrailingSpace` | `boolean \| undefined` | no | Whether to append a trailing space to text before sending to TTS. This helps prevent some TTS services from vocalizing trailing punctuation (e.g., "dot"). Only applied in sentence aggregation mode; when streaming tokens, the incoming text's own whitespace is preserved. | | `sampleRate` | `number \| null \| undefined` | no | Output sample rate for generated audio. | | `skipTextTypes` | `string[] \| null \| undefined` | no | List of text types that should not be spoken. | | `textTransforms` | `[string, TextTransform][] \| null \| undefined` | no | A list of TextTransforms to transform text just before sending it to TTS. Each TextTransform takes the aggregated text and its type, and returns the transformed text. To register, provide a list of tuples of `[textType \| "*", transformFunction]`. | | `transportDestination` | `string \| null \| undefined` | no | Destination for generated audio frames. | | `reuseContextIdWithinTurn` | `boolean \| undefined` | no | Whether the service should reuse context IDs within the same turn. | | `audioCache` | `TTSAudioCacheStore \| null \| undefined` | no | Where to keep synthesized audio for reuse (e.g. a shared `TTSAudioCache`). Text whose audio is in the cache is served from it, with no request to the provider. Lines get there through `warmAudioCache()`, and also from live synthesis when `cacheSynthesizedAudio` allows it. Only services that yield their audio from `runTts` (HTTP and streaming-RPC services, see `supportsAudioCache`) use it, and only in sentence aggregation mode. Defaults to null (no cache). pipedog only. | | `cacheSynthesizedAudio` | `boolean \| ((text: string) => boolean) \| undefined` | no | Whether audio synthesized live is stored in `audioCache`, or a function that decides per text. Defaults to false: only warmed lines are stored, since a generated reply can repeat what the user said. pipedog only. | | `audioCacheNamespace` | `string \| undefined` | no | Extra part of the cache key, for constructor options that change the audio but are not in `settings`. pipedog only. | | `apiKey` | `string` | yes | Azure Cognitive Services subscription key. | | `region` | `string \| null \| undefined` | no | Azure region identifier (e.g., "eastus", "westus2"). Required unless `privateEndpoint` is provided. | | `privateEndpoint` | `string \| null \| undefined` | no | Custom endpoint URL for Azure Speech Services (e.g., "https://my-resource.cognitiveservices.azure.com/"). Use this when connecting via Private Link or a custom domain. See https://learn.microsoft.com/en-us/azure/ai-services/speech-service/speech-services-private-link | | `settings` | `AzureTTSSettings \| null \| undefined` | no | Runtime-updatable settings. | | `fetch` | `{ (input: RequestInfo \| URL, init?: RequestInit): Promise; (input: string \| URL \| Request, init?: RequestInit): Promise; } \| null \| undefined` | no | `fetch` used for HTTP requests (the REST synthesis, and the custom-domain redirect lookup). Defaults to the global `fetch`. Not in upstream. | ### Usage [#usage-1] ```ts title="http.ts" const httpTts = new AzureHttpTTSService({ apiKey: process.env.AZURE_SPEECH_API_KEY ?? "", region: process.env.AZURE_SPEECH_REGION ?? "eastus", sampleRate: 24000, }); ``` ### Notes [#notes-1] * Posts the SSML to `https://.tts.speech.microsoft.com/cognitiveservices/v1` (or `/tts/cognitiveservices/v1`) and yields the sentence as one audio frame. * It requests a `raw-*-16bit-mono-pcm` format, which has no WAV header. Upstream strips the 44 header bytes of the SDK's RIFF result instead. * Pass `fetch` to send the request through your own client. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/azure/package.json`: ✅ means the compat smoke test passes on that runtime. Both services work on Bun, Node, Deno and Cloudflare Workers: `AzureTTSService` uses core's `websocketConnect`, `AzureHttpTTSService` uses `fetch`. The key is sent as the `Ocp-Apim-Subscription-Key` header, and `AzureTTSService` also sends it as a query parameter, as the Speech SDK does, for runtimes whose `WebSocket` can't send headers. With a custom-domain `privateEndpoint`, `AzureTTSService` first asks the endpoint for its regional redirect, as the SDK does. The deprecated `voice`, `params` (`InputParams`) and `aggregateSentences` options aren't ported; use `settings` and `textAggregationMode`. # Bland > Text-to-speech with Bland: BlandTTSService streams LLM tokens over a realtime WebSocket, and BlandHttpTTSService posts whole texts. Source: https://polvallverdu.github.io/pipedog/docs/services/tts/bland/ ## Overview [#overview] * **`BlandTTSService`**: one WebSocket for the whole conversation. LLM tokens go to Bland as they arrive (Bland chooses its own synthesis boundaries), audio comes back as binary frames, and a barge-in sends `cancel` without dropping the connection. * **`BlandHttpTTSService`**: one request per text to `/v2/tts`. Voice agents should use the WebSocket service. Pipecat's documentation for the same service. `packages/bland/src/tts.ts`. The realtime WebSocket reference. ## Installation [#installation] ```bash bun add @pipedog/bland ``` ```ts import { BlandHttpTTSService, BlandTTSService } from "@pipedog/bland/tts"; ``` ## Prerequisites [#prerequisites] * A Bland account and an API key. * Pass the key as `apiKey`. The examples read it from `BLAND_API_KEY`. An empty key throws. ## BlandTTSService [#blandttsservice] Connects to `wss://api.bland.ai/v2/tts/ws` and waits for the session's `ready` message, so a session Bland refuses fails at connect, not on the first turn. ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | Bland API key for authentication. | | `url` | `string \| undefined` | no | WebSocket URL for the Bland realtime TTS API. Defaults to `wss://api.bland.ai/v2/tts/ws`. | | `textAggregationMode` | `TextAggregationMode \| null \| undefined` | no | How to aggregate incoming text before sending. Defaults to `TextAggregationMode.TOKEN`, streaming LLM tokens straight to Bland for the lowest latency. | | `settings` | `BlandTTSSettings \| null \| undefined` | no | Runtime-updatable settings. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `reconnectBackoffMinWait` | `number \| undefined` | no | Minimum time, in seconds, to wait between reconnection attempts. | | `reconnectBackoffMaxWait` | `number \| undefined` | no | Maximum time, in seconds, to wait between reconnection attempts. | | `reconnectOnError` | `boolean \| undefined` | no | Whether to automatically reconnect on connection errors. | | `wsCloseTimeout` | `number \| undefined` | no | Maximum time, in seconds, to wait for the peer to acknowledge the websocket closing handshake before dropping the connection. Applied to connections opened through `_websocketConnect()`. Increase it for peers that need longer to complete a graceful close. | | `wsSendTimeout` | `number \| null \| undefined` | no | Maximum time, in seconds, a single send may wait on a peer that has stopped reading before the connection is dropped and the send throws `ConnectionClosedError`. Applied to connections opened through `_websocketConnect()`. Pass `null` to wait indefinitely. Defaults to `WS_SEND_TIMEOUT`. | | `pushTextFrames` | `boolean \| undefined` | no | Whether to push TextFrames and LLMFullResponseEndFrames. | | `pushStopFrames` | `boolean \| undefined` | no | Whether to automatically push TTSStoppedFrames. | | `pushStartFrame` | `boolean \| undefined` | no | Whether to automatically create audio contexts and push TTSStartedFrames. When true, the base class handles `createAudioContext` and yields `TTSStartedFrame` before each synthesis call, so `runTts` implementations do not need to. | | `stopFrameTimeoutS` | `number \| undefined` | no | Idle time before pushing TTSStoppedFrame when pushStopFrames is true. | | `pushSilenceAfterStop` | `boolean \| undefined` | no | Whether to push silence audio after TTSStoppedFrame. | | `silenceTimeS` | `number \| undefined` | no | Duration of silence to push when pushSilenceAfterStop is true. | | `pauseFrameProcessing` | `boolean \| undefined` | no | Whether to hold incoming frames after a turn until its audio contexts end. Needed by services whose audio doesn't carry its request's context ID. | | `maxConsecutiveZeroAudioContexts` | `number \| undefined` | no | How many consecutive TTS contexts may complete without producing any audio before the service is reported unable to do its job. Catches a provider that accepts requests and stays silent — an unknown voice ID, say — which no error ever surfaces. Every silent context reports an error the service can carry on from, whatever the limit is; on reaching the limit the service reports a permanent error instead, stops being given work, and the pipeline worker applies its `ProcessorUnusablePolicy`. Set to 0 to report silent contexts without ever writing the service off. | | `appendTrailingSpace` | `boolean \| undefined` | no | Whether to append a trailing space to text before sending to TTS. This helps prevent some TTS services from vocalizing trailing punctuation (e.g., "dot"). Only applied in sentence aggregation mode; when streaming tokens, the incoming text's own whitespace is preserved. | | `sampleRate` | `number \| null \| undefined` | no | Output sample rate for generated audio. | | `skipTextTypes` | `string[] \| null \| undefined` | no | List of text types that should not be spoken. | | `textTransforms` | `[string, TextTransform][] \| null \| undefined` | no | A list of TextTransforms to transform text just before sending it to TTS. Each TextTransform takes the aggregated text and its type, and returns the transformed text. To register, provide a list of tuples of `[textType \| "*", transformFunction]`. | | `transportDestination` | `string \| null \| undefined` | no | Destination for generated audio frames. | | `reuseContextIdWithinTurn` | `boolean \| undefined` | no | Whether the service should reuse context IDs within the same turn. | | `audioCache` | `TTSAudioCacheStore \| null \| undefined` | no | Where to keep synthesized audio for reuse (e.g. a shared `TTSAudioCache`). Text whose audio is in the cache is served from it, with no request to the provider. Lines get there through `warmAudioCache()`, and also from live synthesis when `cacheSynthesizedAudio` allows it. Only services that yield their audio from `runTts` (HTTP and streaming-RPC services, see `supportsAudioCache`) use it, and only in sentence aggregation mode. Defaults to null (no cache). pipedog only. | | `cacheSynthesizedAudio` | `boolean \| ((text: string) => boolean) \| undefined` | no | Whether audio synthesized live is stored in `audioCache`, or a function that decides per text. Defaults to false: only warmed lines are stored, since a generated reply can repeat what the user said. pipedog only. | | `audioCacheNamespace` | `string \| undefined` | no | Extra part of the cache key, for constructor options that change the audio but are not in `settings`. pipedog only. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new BlandTTSService.Settings({...})` (`BlandTTSSettings`). Any setting you leave out keeps the default below. A change to `voice`, `expressiveness`, `stability` or `autoFormatting` opens a new session, because the session's `init` message fixes them. | Setting | Type | Default | Description | | ---------------- | ----------------- | ---------------------------------------- | ----------------------------------------------------------------------------------------- | | `voice` | `string` | `"29158307-9893-4149-8a75-bc9ce313d64e"` | Voice UUID. The voice sets the model. | | `expressiveness` | `number \| null` | `null` | 0.0 to 1.0. Higher values vary the intonation more. Calibrated for `BTTS_V3`. | | `stability` | `number \| null` | `null` | 0.0 to 1.0. Higher values keep delivery more consistent. | | `autoFormatting` | `boolean \| null` | `null` | Rewrite numbers into the form the voice reads best, such as phone numbers digit by digit. | ### Usage [#usage] ```ts title="tts.ts" import { BlandHttpTTSService, BlandTTSService } from "@pipedog/bland/tts"; import { TTSUpdateSettingsFrame } from "@pipedog/core/frames/frames"; const tts = new BlandTTSService({ apiKey: process.env.BLAND_API_KEY ?? "", settings: new BlandTTSService.Settings({ voice: "29158307-9893-4149-8a75-bc9ce313d64e", expressiveness: 0.6, stability: 0.5, }), }); ``` Change settings mid-call with a `TTSUpdateSettingsFrame`: ```ts // Opens a new session, since the voice and controls are part of the init message. const update = new TTSUpdateSettingsFrame({ delta: new BlandTTSService.Settings({ stability: 0.8 }), }); ``` ### Event handlers [#event-handlers] | Event | Description | | --------------------- | ---------------------- | | `on_connected` | The session is ready. | | `on_disconnected` | The WebSocket closed. | | `on_connection_error` | The connection failed. | ## BlandHttpTTSService [#blandhttpttsservice] Posts to `https://api.bland.ai/v2/tts` and streams the raw PCM response into frames. ### Configuration [#configuration-1] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | Bland API key for authentication. | | `baseUrl` | `string \| undefined` | no | Base URL for the Bland API. Defaults to `https://api.bland.ai/v2`. | | `fetch` | `{ (input: RequestInfo \| URL, init?: RequestInit): Promise; (input: string \| URL \| Request, init?: RequestInit): Promise; } \| null \| undefined` | no | `fetch` used for HTTP requests. Defaults to the global `fetch`. DIVERGENCE: upstream takes an optional shared `aiohttp_session`, and creates and closes one itself when none is given. | | `settings` | `BlandTTSSettings \| null \| undefined` | no | Runtime-updatable settings. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `textAggregationMode` | `TextAggregationMode \| null \| undefined` | no | How to aggregate incoming text before synthesis. `TextAggregationMode.SENTENCE` (default) buffers until sentence boundaries, `TextAggregationMode.TOKEN` streams tokens directly for lower latency. | | `pushTextFrames` | `boolean \| undefined` | no | Whether to push TextFrames and LLMFullResponseEndFrames. | | `pushStopFrames` | `boolean \| undefined` | no | Whether to automatically push TTSStoppedFrames. | | `pushStartFrame` | `boolean \| undefined` | no | Whether to automatically create audio contexts and push TTSStartedFrames. When true, the base class handles `createAudioContext` and yields `TTSStartedFrame` before each synthesis call, so `runTts` implementations do not need to. | | `stopFrameTimeoutS` | `number \| undefined` | no | Idle time before pushing TTSStoppedFrame when pushStopFrames is true. | | `pushSilenceAfterStop` | `boolean \| undefined` | no | Whether to push silence audio after TTSStoppedFrame. | | `silenceTimeS` | `number \| undefined` | no | Duration of silence to push when pushSilenceAfterStop is true. | | `pauseFrameProcessing` | `boolean \| undefined` | no | Whether to hold incoming frames after a turn until its audio contexts end. Needed by services whose audio doesn't carry its request's context ID. | | `maxConsecutiveZeroAudioContexts` | `number \| undefined` | no | How many consecutive TTS contexts may complete without producing any audio before the service is reported unable to do its job. Catches a provider that accepts requests and stays silent — an unknown voice ID, say — which no error ever surfaces. Every silent context reports an error the service can carry on from, whatever the limit is; on reaching the limit the service reports a permanent error instead, stops being given work, and the pipeline worker applies its `ProcessorUnusablePolicy`. Set to 0 to report silent contexts without ever writing the service off. | | `appendTrailingSpace` | `boolean \| undefined` | no | Whether to append a trailing space to text before sending to TTS. This helps prevent some TTS services from vocalizing trailing punctuation (e.g., "dot"). Only applied in sentence aggregation mode; when streaming tokens, the incoming text's own whitespace is preserved. | | `sampleRate` | `number \| null \| undefined` | no | Output sample rate for generated audio. | | `skipTextTypes` | `string[] \| null \| undefined` | no | List of text types that should not be spoken. | | `textTransforms` | `[string, TextTransform][] \| null \| undefined` | no | A list of TextTransforms to transform text just before sending it to TTS. Each TextTransform takes the aggregated text and its type, and returns the transformed text. To register, provide a list of tuples of `[textType \| "*", transformFunction]`. | | `transportDestination` | `string \| null \| undefined` | no | Destination for generated audio frames. | | `reuseContextIdWithinTurn` | `boolean \| undefined` | no | Whether the service should reuse context IDs within the same turn. | | `audioCache` | `TTSAudioCacheStore \| null \| undefined` | no | Where to keep synthesized audio for reuse (e.g. a shared `TTSAudioCache`). Text whose audio is in the cache is served from it, with no request to the provider. Lines get there through `warmAudioCache()`, and also from live synthesis when `cacheSynthesizedAudio` allows it. Only services that yield their audio from `runTts` (HTTP and streaming-RPC services, see `supportsAudioCache`) use it, and only in sentence aggregation mode. Defaults to null (no cache). pipedog only. | | `cacheSynthesizedAudio` | `boolean \| ((text: string) => boolean) \| undefined` | no | Whether audio synthesized live is stored in `audioCache`, or a function that decides per text. Defaults to false: only warmed lines are stored, since a generated reply can repeat what the user said. pipedog only. | | `audioCacheNamespace` | `string \| undefined` | no | Extra part of the cache key, for constructor options that change the audio but are not in `settings`. pipedog only. | It takes the same settings as the WebSocket service. ### Usage [#usage-1] ```ts const httpTts = new BlandHttpTTSService({ apiKey: process.env.BLAND_API_KEY ?? "", settings: new BlandHttpTTSService.Settings({ autoFormatting: true }), }); ``` ## Notes [#notes] * `textAggregationMode` defaults to `TextAggregationMode.TOKEN` for the WebSocket service. Pass `TextAggregationMode.SENTENCE` to send whole sentences, with a trailing space added. * Bland renders 8000, 16000, 24000, 44100 and 48000 Hz. For another pipeline rate the service asks for 48000 Hz, tags its audio frames with that rate and logs a warning, and the output transport resamples. * Bland ends a session after 60 seconds without a client message. The service opens a new one without reporting an error, and a turn that was half sent continues on the new session as a new turn. * At shutdown the service sends `close` and waits up to five seconds for `done`, so Bland settles the usage. * A turn that Bland refuses or fails is reported once as an error frame, and its audio context is closed. ## Runtime notes [#runtime-notes] * The WebSocket opens through the core `websocketConnect`, with the key in an `Authorization` header (through `fetch` on Cloudflare Workers). The HTTP service uses the global `fetch`; pass `fetch` to use another one (upstream takes an optional aiohttp session, and creates and closes one itself when none is given). Both work on Bun, Node, Deno and Cloudflare Workers. * Tested against local servers that follow Bland's documented message flow, not against Bland itself. Bland's HTTP reference doesn't list `auto_formatting`; the service sends it, as upstream does. | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/bland/package.json`: ✅ means the compat smoke test passes on that runtime. # Camb.ai > Text-to-speech with Camb.ai's MARS models: CambTTSService streams PCM audio over HTTP. Source: https://polvallverdu.github.io/pipedog/docs/services/tts/camb/ ## Overview [#overview] `CambTTSService` posts each sentence to Camb.ai's `tts-stream` endpoint and pushes the streamed 16-bit PCM, kept aligned to whole samples. Models: `mars-flash` (fast, 22.05 kHz, the default), `mars-pro` (high quality, 48 kHz) and `mars-instruct` (22.05 kHz, takes instructions for tone and style). Pipecat's documentation for the same service. `packages/camb/src/tts.ts`. The streaming TTS endpoint reference. ## Installation [#installation] ```bash bun add @pipedog/camb ``` ```ts import { CambTTSService } from "@pipedog/camb/tts"; ``` ## Prerequisites [#prerequisites] * A Camb.ai account and an API key. * Pass the key as `apiKey`. The examples read it from `CAMB_API_KEY`. ## Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `textAggregationMode` | `TextAggregationMode \| null \| undefined` | no | How to aggregate incoming text before synthesis. `TextAggregationMode.SENTENCE` (default) buffers until sentence boundaries, `TextAggregationMode.TOKEN` streams tokens directly for lower latency. | | `pushTextFrames` | `boolean \| undefined` | no | Whether to push TextFrames and LLMFullResponseEndFrames. | | `pushStopFrames` | `boolean \| undefined` | no | Whether to automatically push TTSStoppedFrames. | | `apiKey` | `string` | yes | Camb.ai API key for authentication. | | `timeout` | `number \| undefined` | no | Request timeout in seconds. Defaults to 60.0 (minimum recommended by Camb.ai). Applies to the wait for the response and to the wait for each chunk of audio. | | `baseUrl` | `string \| undefined` | no | Base URL of the Camb.ai API. Defaults to "https://client.camb.ai/apis". DIVERGENCE: not an upstream option. It stands for the SDK's `base_url` / `environment`. | | `fetch` | `((input: RequestInfo \| URL, init?: RequestInit) => Promise) \| null \| undefined` | no | `fetch` used for HTTP requests. Defaults to the global `fetch`. DIVERGENCE: upstream calls Camb's Python SDK, which brings its own HTTP client. | | `settings` | `CambTTSSettings \| null \| undefined` | no | Runtime-updatable settings. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `pushStartFrame` | `boolean \| undefined` | no | Whether to automatically create audio contexts and push TTSStartedFrames. When true, the base class handles `createAudioContext` and yields `TTSStartedFrame` before each synthesis call, so `runTts` implementations do not need to. | | `stopFrameTimeoutS` | `number \| undefined` | no | Idle time before pushing TTSStoppedFrame when pushStopFrames is true. | | `pushSilenceAfterStop` | `boolean \| undefined` | no | Whether to push silence audio after TTSStoppedFrame. | | `silenceTimeS` | `number \| undefined` | no | Duration of silence to push when pushSilenceAfterStop is true. | | `pauseFrameProcessing` | `boolean \| undefined` | no | Whether to hold incoming frames after a turn until its audio contexts end. Needed by services whose audio doesn't carry its request's context ID. | | `maxConsecutiveZeroAudioContexts` | `number \| undefined` | no | How many consecutive TTS contexts may complete without producing any audio before the service is reported unable to do its job. Catches a provider that accepts requests and stays silent — an unknown voice ID, say — which no error ever surfaces. Every silent context reports an error the service can carry on from, whatever the limit is; on reaching the limit the service reports a permanent error instead, stops being given work, and the pipeline worker applies its `ProcessorUnusablePolicy`. Set to 0 to report silent contexts without ever writing the service off. | | `appendTrailingSpace` | `boolean \| undefined` | no | Whether to append a trailing space to text before sending to TTS. This helps prevent some TTS services from vocalizing trailing punctuation (e.g., "dot"). Only applied in sentence aggregation mode; when streaming tokens, the incoming text's own whitespace is preserved. | | `sampleRate` | `number \| null \| undefined` | no | Output sample rate for generated audio. | | `skipTextTypes` | `string[] \| null \| undefined` | no | List of text types that should not be spoken. | | `textTransforms` | `[string, TextTransform][] \| null \| undefined` | no | A list of TextTransforms to transform text just before sending it to TTS. Each TextTransform takes the aggregated text and its type, and returns the transformed text. To register, provide a list of tuples of `[textType \| "*", transformFunction]`. | | `transportDestination` | `string \| null \| undefined` | no | Destination for generated audio frames. | | `reuseContextIdWithinTurn` | `boolean \| undefined` | no | Whether the service should reuse context IDs within the same turn. | | `audioCache` | `TTSAudioCacheStore \| null \| undefined` | no | Where to keep synthesized audio for reuse (e.g. a shared `TTSAudioCache`). Text whose audio is in the cache is served from it, with no request to the provider. Lines get there through `warmAudioCache()`, and also from live synthesis when `cacheSynthesizedAudio` allows it. Only services that yield their audio from `runTts` (HTTP and streaming-RPC services, see `supportsAudioCache`) use it, and only in sentence aggregation mode. Defaults to null (no cache). pipedog only. | | `cacheSynthesizedAudio` | `boolean \| ((text: string) => boolean) \| undefined` | no | Whether audio synthesized live is stored in `audioCache`, or a function that decides per text. Defaults to false: only warmed lines are stored, since a generated reply can repeat what the user said. pipedog only. | | `audioCacheNamespace` | `string \| undefined` | no | Extra part of the cache key, for constructor options that change the audio but are not in `settings`. pipedog only. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new CambTTSService.Settings({...})` (`CambTTSSettings`). Any setting you leave out keeps the default below. | Setting | Type | Default | Description | | ------------------ | -------------------- | -------------- | ------------------------------------------------------------------------------------------ | | `model` | `string` | `"mars-flash"` | `mars-flash`, `mars-pro` or `mars-instruct`. | | `voice` | `number` | `147320` | Voice id. Camb.ai uses integer ids. | | `language` | `Language \| string` | `"en-us"` | Language, as a lowercase BCP-47 tag. | | `userInstructions` | `string \| null` | `null` | Instructions for `mars-instruct` only, up to 1000 characters. Ignored by the other models. | ## Usage [#usage] ```ts title="tts.ts" import { CambTTSService } from "@pipedog/camb/tts"; import { TTSUpdateSettingsFrame } from "@pipedog/core/frames/frames"; // mars-flash (22.05 kHz) is the default; mars-pro renders at 48 kHz. const tts = new CambTTSService({ apiKey: process.env.CAMB_API_KEY ?? "", settings: new CambTTSService.Settings({ model: "mars-pro", voice: 147320 }), }); ``` Change settings mid-call with a `TTSUpdateSettingsFrame`: ```ts // Queue it on the worker, or push it from a processor upstream of the service. const update = new TTSUpdateSettingsFrame({ delta: new CambTTSService.Settings({ voice: 12345 }), }); ``` ## Notes [#notes] * Without a `sampleRate`, the service uses the model's: 22050 Hz for `mars-flash` and `mars-instruct`, 48000 Hz for `mars-pro`. A `sampleRate` that differs from the model's logs a warning. * Text over 3000 characters is truncated. * `timeout` (60 seconds by default, the minimum Camb.ai recommends) limits the wait for the response and the wait for each chunk of audio. ## Runtime notes [#runtime-notes] * Upstream uses Camb's Python SDK. That SDK is a thin client over one endpoint, so this port sends the same request (`POST /tts-stream` with an `x-api-key` header) with the global `fetch`, which works on Bun, Node, Deno and Cloudflare Workers. `baseUrl` and `fetch` are not upstream options. * The deprecated `params` (`InputParams`), `voiceId` and `model` options are not ported. Use `settings`. * Upstream has no Camb.ai tests. Ours run against a local fake server, not against Camb.ai itself. | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/camb/package.json`: ✅ means the compat smoke test passes on that runtime. # Cartesia > Text-to-speech with CartesiaTTSService (WebSocket, word timestamps) and CartesiaHttpTTSService (HTTP), configured with GenerationConfig. Source: https://polvallverdu.github.io/pipedog/docs/services/tts/cartesia/ ## Overview [#overview] Cartesia's Sonic models turn text into speech. pipedog ports both of Pipecat's Cartesia text-to-speech services: * **`CartesiaTTSService`**: streams over Cartesia's WebSocket API. Audio arrives as it is generated, with word-level timestamps that keep the spoken text in step with playback. * **`CartesiaHttpTTSService`**: posts each request to Cartesia's `/tts/bytes` HTTP endpoint and gets the audio back in one piece, without word timestamps. * **`GenerationConfig`**: volume, speed and emotion for sonic-3 series models, shared by both. Pipecat's documentation for the same services. `packages/cartesia/src/tts.ts`. Voices, models, SSML tags and pronunciation dictionaries. ## Installation [#installation] ```bash bun add @pipedog/cartesia ``` ```ts import { CartesiaEmotion, CartesiaHttpTTSService, CartesiaTTSService, GenerationConfig, } from "@pipedog/cartesia/tts"; ``` ## Prerequisites [#prerequisites] * A Cartesia account and an API key from the [Cartesia playground](https://play.cartesia.ai). * A voice id from the Cartesia voice library. There is no default voice. * Pass the key as `apiKey`. The examples read it from `CARTESIA_API_KEY`. ## CartesiaTTSService [#cartesiattsservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | Cartesia API key for authentication. | | `url` | `string \| undefined` | no | WebSocket URL for Cartesia TTS API. | | `encoding` | `string \| undefined` | no | Audio encoding format. | | `container` | `string \| undefined` | no | Audio container format. | | `maxBufferDelayMs` | `number \| null \| undefined` | no | Server-side buffering window before generation starts. `0` disables server buffering (custom buffering); any value in (0, 5000] enables managed buffering. If null, derived from `textAggregationMode`: `0` for `SENTENCE` (avoids stacking client and server buffering), unset for `TOKEN` (uses Cartesia's 3000ms default). | | `extraHeaders` | `Record \| null \| undefined` | no | Optional additional HTTP headers to send with the WebSocket handshake. | | `settings` | `CartesiaTTSSettings \| null \| undefined` | no | Runtime-updatable settings. | | `textAggregationMode` | `TextAggregationMode \| null \| undefined` | no | How to aggregate incoming text before synthesis. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `reconnectBackoffMinWait` | `number \| undefined` | no | Minimum time, in seconds, to wait between reconnection attempts. | | `reconnectBackoffMaxWait` | `number \| undefined` | no | Maximum time, in seconds, to wait between reconnection attempts. | | `reconnectOnError` | `boolean \| undefined` | no | Whether to automatically reconnect on connection errors. | | `wsCloseTimeout` | `number \| undefined` | no | Maximum time, in seconds, to wait for the peer to acknowledge the websocket closing handshake before dropping the connection. Applied to connections opened through `_websocketConnect()`. Increase it for peers that need longer to complete a graceful close. | | `wsSendTimeout` | `number \| null \| undefined` | no | Maximum time, in seconds, a single send may wait on a peer that has stopped reading before the connection is dropped and the send throws `ConnectionClosedError`. Applied to connections opened through `_websocketConnect()`. Pass `null` to wait indefinitely. Defaults to `WS_SEND_TIMEOUT`. | | `pushTextFrames` | `boolean \| undefined` | no | Whether to push TextFrames and LLMFullResponseEndFrames. | | `pushStopFrames` | `boolean \| undefined` | no | Whether to automatically push TTSStoppedFrames. | | `pushStartFrame` | `boolean \| undefined` | no | Whether to automatically create audio contexts and push TTSStartedFrames. When true, the base class handles `createAudioContext` and yields `TTSStartedFrame` before each synthesis call, so `runTts` implementations do not need to. | | `stopFrameTimeoutS` | `number \| undefined` | no | Idle time before pushing TTSStoppedFrame when pushStopFrames is true. | | `pushSilenceAfterStop` | `boolean \| undefined` | no | Whether to push silence audio after TTSStoppedFrame. | | `silenceTimeS` | `number \| undefined` | no | Duration of silence to push when pushSilenceAfterStop is true. | | `pauseFrameProcessing` | `boolean \| undefined` | no | Whether to hold incoming frames after a turn until its audio contexts end. Needed by services whose audio doesn't carry its request's context ID. | | `maxConsecutiveZeroAudioContexts` | `number \| undefined` | no | How many consecutive TTS contexts may complete without producing any audio before the service is reported unable to do its job. Catches a provider that accepts requests and stays silent — an unknown voice ID, say — which no error ever surfaces. Every silent context reports an error the service can carry on from, whatever the limit is; on reaching the limit the service reports a permanent error instead, stops being given work, and the pipeline worker applies its `ProcessorUnusablePolicy`. Set to 0 to report silent contexts without ever writing the service off. | | `appendTrailingSpace` | `boolean \| undefined` | no | Whether to append a trailing space to text before sending to TTS. This helps prevent some TTS services from vocalizing trailing punctuation (e.g., "dot"). Only applied in sentence aggregation mode; when streaming tokens, the incoming text's own whitespace is preserved. | | `sampleRate` | `number \| null \| undefined` | no | Output sample rate for generated audio. | | `skipTextTypes` | `string[] \| null \| undefined` | no | List of text types that should not be spoken. | | `textTransforms` | `[string, TextTransform][] \| null \| undefined` | no | A list of TextTransforms to transform text just before sending it to TTS. Each TextTransform takes the aggregated text and its type, and returns the transformed text. To register, provide a list of tuples of `[textType \| "*", transformFunction]`. | | `transportDestination` | `string \| null \| undefined` | no | Destination for generated audio frames. | | `reuseContextIdWithinTurn` | `boolean \| undefined` | no | Whether the service should reuse context IDs within the same turn. | | `audioCache` | `TTSAudioCacheStore \| null \| undefined` | no | Where to keep synthesized audio for reuse (e.g. a shared `TTSAudioCache`). Text whose audio is in the cache is served from it, with no request to the provider. Lines get there through `warmAudioCache()`, and also from live synthesis when `cacheSynthesizedAudio` allows it. Only services that yield their audio from `runTts` (HTTP and streaming-RPC services, see `supportsAudioCache`) use it, and only in sentence aggregation mode. Defaults to null (no cache). pipedog only. | | `cacheSynthesizedAudio` | `boolean \| ((text: string) => boolean) \| undefined` | no | Whether audio synthesized live is stored in `audioCache`, or a function that decides per text. Defaults to false: only warmed lines are stored, since a generated reply can repeat what the user said. pipedog only. | | `audioCacheNamespace` | `string \| undefined` | no | Extra part of the cache key, for constructor options that change the audio but are not in `settings`. pipedog only. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new CartesiaTTSService.Settings({...})`. Both services use the same settings class. | Setting | Type | Default | Description | | --------------------- | -------------------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------- | | `model` | `string` | `"sonic-3.6"` | Cartesia TTS model. | | `voice` | `string \| null` | `null` | Voice id. Set one: there is no default. | | `language` | `Language` | `Language.EN` | Synthesis language. | | `accent` | `string \| null` | `null` | Accent a multilingual voice speaks the language with, from the `accents` of Cartesia's Get Voice response. `null` lets Cartesia pick. | | `generationConfig` | `GenerationConfig \| null` | `null` | Volume, speed and emotion. See below. | | `pronunciationDictId` | `string \| null` | `null` | Cartesia pronunciation dictionary id. | ### GenerationConfig [#generationconfig] | Option | Type | Required | Description | |---|---|---|---| | `volume` | `number \| null \| undefined` | no | Volume multiplier for generated speech. Valid range: [0.5, 2.0]. Default is 1.0. | | `speed` | `number \| null \| undefined` | no | Speed multiplier for generated speech. Valid range: [0.6, 1.5]. Default is 1.0. | | `emotion` | `string \| null \| undefined` | no | Single emotion string to guide the emotional tone. Examples include neutral, angry, excited, content, sad, scared. Over 60 emotions are supported. For best results, use with recommended voices: Leo, Jace, Kyle, Gavin, Maya, Tessa, Dana, and Marian. | Fields left `null` are not sent. `CartesiaEmotion` lists the emotion names Cartesia supports. ### Usage [#usage] ```ts title="tts.ts" import { CartesiaEmotion, CartesiaHttpTTSService, CartesiaTTSService, GenerationConfig, } from "@pipedog/cartesia/tts"; import { TTSUpdateSettingsFrame } from "@pipedog/core/frames/frames"; import { TextAggregationMode } from "@pipedog/core/services/tts_service"; import { Language } from "@pipedog/core/transcriptions/language"; const tts = new CartesiaTTSService({ apiKey: process.env.CARTESIA_API_KEY ?? "", settings: new CartesiaTTSService.Settings({ voice: "your-voice-id", model: "sonic-3.6", language: Language.EN, }), }); ``` Token streaming and a generation config: ```ts const expressive = new CartesiaTTSService({ apiKey: process.env.CARTESIA_API_KEY ?? "", // Stream LLM tokens straight to Cartesia instead of waiting for whole sentences. textAggregationMode: TextAggregationMode.TOKEN, settings: new CartesiaTTSService.Settings({ voice: "your-voice-id", generationConfig: new GenerationConfig({ speed: 1.1, emotion: CartesiaEmotion.EXCITED }), }), }); ``` SSML tag helpers and IPA pronunciations: ```ts // Helpers that build Cartesia SSML tags, for use in text transforms. const spelled = CartesiaTTSService.SPELL("A1B2"); const tagged = `${CartesiaTTSService.EMOTION_TAG(CartesiaEmotion.CALM)} One moment.${CartesiaTTSService.PAUSE_TAG(0.5)}`; // Say words as IPA describes. Register it last in textTransforms. const pronounce = CartesiaTTSService.pronunciationTransformIpa({ Metformin: "mɛtˈfɔɹmɪn" }); const withPronunciations = new CartesiaTTSService({ apiKey: process.env.CARTESIA_API_KEY ?? "", settings: new CartesiaTTSService.Settings({ voice: "your-voice-id" }), textTransforms: [["*", pronounce]], }); ``` Change settings mid-call with a `TTSUpdateSettingsFrame`: ```ts // Queue it on the worker, or push it from a processor upstream of the service. const update = new TTSUpdateSettingsFrame({ delta: new CartesiaTTSService.Settings({ voice: "another-voice-id", language: Language.ES }), }); ``` ### Event handlers [#event-handlers] Inherited from `TTSService`: | Event | Description | | --------------------- | ----------------------------------------------- | | `on_connected` | Connected to Cartesia. | | `on_disconnected` | Disconnected from Cartesia. | | `on_connection_error` | The connection failed. | | `on_tts_request` | A text chunk is about to be sent for synthesis. | ### Notes [#notes] * The service does not push the LLM's text frames itself. It builds `TTSTextFrame`s from Cartesia's word timestamps, so the context only records words the user actually heard. * Text is aggregated into sentences by default. `TextAggregationMode.TOKEN` streams tokens directly for lower latency. * `maxBufferDelayMs` sets Cartesia's server-side buffer. Left `null`, it is `0` in sentence mode (no double buffering) and unset in token mode (Cartesia's 3000 ms default). * A `voice`, `model` or `language` change takes effect from the next sentence, which opens a new Cartesia context. * Static helpers build Cartesia SSML tags for text transforms: `SPELL`, `EMOTION_TAG`, `PAUSE_TAG`, `VOLUME_TAG` and `SPEED_TAG`. Text inside `` tags is never split across aggregates. * `pronunciationTransformIpa()` turns IPA into Cartesia inline phonemes. Register it last in `textTransforms` so later transforms don't rewrite its markup. ## CartesiaHttpTTSService [#cartesiahttpttsservice] ### Configuration [#configuration-1] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | Cartesia API key for authentication. | | `baseUrl` | `string \| undefined` | no | Base URL for Cartesia HTTP API. | | `fetch` | `{ (input: RequestInfo \| URL, init?: RequestInit): Promise; (input: string \| URL \| Request, init?: RequestInit): Promise; } \| null \| undefined` | no | `fetch` used for HTTP requests. If not provided, the global `fetch` is used, set up in `setup()` and released at teardown. DIVERGENCE: replaces upstream's `aiohttp_session`. | | `encoding` | `string \| undefined` | no | Audio encoding format. | | `container` | `string \| undefined` | no | Audio container format. | | `extraHeaders` | `Record \| null \| undefined` | no | Optional additional HTTP headers to send with each synthesis request. | | `settings` | `CartesiaTTSSettings \| null \| undefined` | no | Runtime-updatable settings. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `textAggregationMode` | `TextAggregationMode \| null \| undefined` | no | How to aggregate incoming text before synthesis. `TextAggregationMode.SENTENCE` (default) buffers until sentence boundaries, `TextAggregationMode.TOKEN` streams tokens directly for lower latency. | | `pushTextFrames` | `boolean \| undefined` | no | Whether to push TextFrames and LLMFullResponseEndFrames. | | `pushStopFrames` | `boolean \| undefined` | no | Whether to automatically push TTSStoppedFrames. | | `pushStartFrame` | `boolean \| undefined` | no | Whether to automatically create audio contexts and push TTSStartedFrames. When true, the base class handles `createAudioContext` and yields `TTSStartedFrame` before each synthesis call, so `runTts` implementations do not need to. | | `stopFrameTimeoutS` | `number \| undefined` | no | Idle time before pushing TTSStoppedFrame when pushStopFrames is true. | | `pushSilenceAfterStop` | `boolean \| undefined` | no | Whether to push silence audio after TTSStoppedFrame. | | `silenceTimeS` | `number \| undefined` | no | Duration of silence to push when pushSilenceAfterStop is true. | | `pauseFrameProcessing` | `boolean \| undefined` | no | Whether to hold incoming frames after a turn until its audio contexts end. Needed by services whose audio doesn't carry its request's context ID. | | `maxConsecutiveZeroAudioContexts` | `number \| undefined` | no | How many consecutive TTS contexts may complete without producing any audio before the service is reported unable to do its job. Catches a provider that accepts requests and stays silent — an unknown voice ID, say — which no error ever surfaces. Every silent context reports an error the service can carry on from, whatever the limit is; on reaching the limit the service reports a permanent error instead, stops being given work, and the pipeline worker applies its `ProcessorUnusablePolicy`. Set to 0 to report silent contexts without ever writing the service off. | | `appendTrailingSpace` | `boolean \| undefined` | no | Whether to append a trailing space to text before sending to TTS. This helps prevent some TTS services from vocalizing trailing punctuation (e.g., "dot"). Only applied in sentence aggregation mode; when streaming tokens, the incoming text's own whitespace is preserved. | | `sampleRate` | `number \| null \| undefined` | no | Output sample rate for generated audio. | | `skipTextTypes` | `string[] \| null \| undefined` | no | List of text types that should not be spoken. | | `textTransforms` | `[string, TextTransform][] \| null \| undefined` | no | A list of TextTransforms to transform text just before sending it to TTS. Each TextTransform takes the aggregated text and its type, and returns the transformed text. To register, provide a list of tuples of `[textType \| "*", transformFunction]`. | | `transportDestination` | `string \| null \| undefined` | no | Destination for generated audio frames. | | `reuseContextIdWithinTurn` | `boolean \| undefined` | no | Whether the service should reuse context IDs within the same turn. | | `audioCache` | `TTSAudioCacheStore \| null \| undefined` | no | Where to keep synthesized audio for reuse (e.g. a shared `TTSAudioCache`). Text whose audio is in the cache is served from it, with no request to the provider. Lines get there through `warmAudioCache()`, and also from live synthesis when `cacheSynthesizedAudio` allows it. Only services that yield their audio from `runTts` (HTTP and streaming-RPC services, see `supportsAudioCache`) use it, and only in sentence aggregation mode. Defaults to null (no cache). pipedog only. | | `cacheSynthesizedAudio` | `boolean \| ((text: string) => boolean) \| undefined` | no | Whether audio synthesized live is stored in `audioCache`, or a function that decides per text. Defaults to false: only warmed lines are stored, since a generated reply can repeat what the user said. pipedog only. | | `audioCacheNamespace` | `string \| undefined` | no | Extra part of the cache key, for constructor options that change the audio but are not in `settings`. pipedog only. | ### Settings [#settings-1] Same as [`CartesiaTTSService`](#settings), with the same defaults. ### Usage [#usage-1] ```ts title="http.ts" const httpTts = new CartesiaHttpTTSService({ apiKey: process.env.CARTESIA_API_KEY ?? "", settings: new CartesiaHttpTTSService.Settings({ voice: "your-voice-id" }), }); ``` ### Notes [#notes-1] * Each request returns its audio in one piece. Use it when streaming does not matter. * `baseUrl` defaults to `"https://api.cartesia.ai"`. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/cartesia/package.json`: ✅ means the compat smoke test passes on that runtime. * `CartesiaTTSService` uses the global `WebSocket` and sends the key as an `X-API-Key` header on the handshake. The Bun, Node and Deno `WebSocket` support handshake headers, and on Cloudflare Workers the socket is opened through `fetch`. * `CartesiaHttpTTSService` uses the global `fetch` unless you pass `fetch`. That option replaces upstream's `aiohttp_session`. * The Cartesia API version header is pinned to `2026-03-01`. Upstream's `cartesia_version` option is not ported. * Not ported: the deprecated `voiceId`, `model`, `params` (`InputParams`) and `aggregateSentences` constructor options. Use `settings` and `textAggregationMode`. # Cloudflare Workers AI > Text-to-speech on Workers AI: Deepgram Aura streaming over a WebSocket or per sentence over HTTP, and MeloTTS, over the AI binding or the REST API. Source: https://polvallverdu.github.io/pipedog/docs/services/tts/cloudflare/ ## Overview [#overview] `@pipedog/cloudflare` has three text-to-speech services on Workers AI: * **`WorkersAIAuraTTSService`**: Deepgram Aura (`@cf/deepgram/aura-2-en`, `aura-2-es`, `aura-1`), streamed over a WebSocket. Workers AI carries Deepgram's `/v1/speak` protocol, so this is [`DeepgramTTSService`](https://polvallverdu.github.io/pipedog/docs/services/tts/deepgram.md) opening its socket through Workers AI. * **`WorkersAIAuraHttpTTSService`**: Aura, one request per sentence, the PCM streamed back. * **`WorkersAIMeloTTSService`**: MeloTTS (`@cf/myshell-ai/melotts`), one request per sentence. It answers in MP3, so it needs a decoder you pass in. This package is pipedog's own: Pipecat has no Workers AI services. `packages/cloudflare/src/tts.ts` and `melotts.ts`. Speakers, inputs and pricing of each model. ## Installation [#installation] ```bash bun add @pipedog/cloudflare ``` ```ts import { WorkersAIAuraHttpTTSService, WorkersAIAuraTTSService } from "@pipedog/cloudflare/tts"; import { WorkersAIMeloTTSService } from "@pipedog/cloudflare/melotts"; ``` ## Prerequisites [#prerequisites] Inside a Worker, pass the AI binding as `ai: env.AI`. Elsewhere, pass `accountId` and `apiToken` (an API token with the Workers AI permission). See [the speech-to-text page](https://polvallverdu.github.io/pipedog/docs/services/stt/cloudflare/#prerequisites) for both paths. ## WorkersAIAuraTTSService [#workersaiaurattsservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `settings` | `DeepgramTTSSettings \| null \| undefined` | no | Runtime-updatable settings. `model` is the Workers AI model ID (default `"@cf/deepgram/aura-2-en"`), `voice` the speaker (default: the model's). | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `reconnectBackoffMinWait` | `number \| undefined` | no | Minimum time, in seconds, to wait between reconnection attempts. | | `reconnectBackoffMaxWait` | `number \| undefined` | no | Maximum time, in seconds, to wait between reconnection attempts. | | `reconnectOnError` | `boolean \| undefined` | no | Whether to automatically reconnect on connection errors. | | `wsCloseTimeout` | `number \| undefined` | no | Maximum time, in seconds, to wait for the peer to acknowledge the websocket closing handshake before dropping the connection. Applied to connections opened through `_websocketConnect()`. Increase it for peers that need longer to complete a graceful close. | | `wsSendTimeout` | `number \| null \| undefined` | no | Maximum time, in seconds, a single send may wait on a peer that has stopped reading before the connection is dropped and the send throws `ConnectionClosedError`. Applied to connections opened through `_websocketConnect()`. Pass `null` to wait indefinitely. Defaults to `WS_SEND_TIMEOUT`. | | `textAggregationMode` | `TextAggregationMode \| null \| undefined` | no | How to aggregate incoming text before synthesis. `TextAggregationMode.SENTENCE` (default) buffers until sentence boundaries, `TextAggregationMode.TOKEN` streams tokens directly for lower latency. | | `pushTextFrames` | `boolean \| undefined` | no | Whether to push TextFrames and LLMFullResponseEndFrames. | | `pushStopFrames` | `boolean \| undefined` | no | Whether to automatically push TTSStoppedFrames. | | `pushStartFrame` | `boolean \| undefined` | no | Whether to automatically create audio contexts and push TTSStartedFrames. When true, the base class handles `createAudioContext` and yields `TTSStartedFrame` before each synthesis call, so `runTts` implementations do not need to. | | `stopFrameTimeoutS` | `number \| undefined` | no | Idle time before pushing TTSStoppedFrame when pushStopFrames is true. | | `pushSilenceAfterStop` | `boolean \| undefined` | no | Whether to push silence audio after TTSStoppedFrame. | | `silenceTimeS` | `number \| undefined` | no | Duration of silence to push when pushSilenceAfterStop is true. | | `pauseFrameProcessing` | `boolean \| undefined` | no | Whether to hold incoming frames after a turn until its audio contexts end. Needed by services whose audio doesn't carry its request's context ID. | | `maxConsecutiveZeroAudioContexts` | `number \| undefined` | no | How many consecutive TTS contexts may complete without producing any audio before the service is reported unable to do its job. Catches a provider that accepts requests and stays silent — an unknown voice ID, say — which no error ever surfaces. Every silent context reports an error the service can carry on from, whatever the limit is; on reaching the limit the service reports a permanent error instead, stops being given work, and the pipeline worker applies its `ProcessorUnusablePolicy`. Set to 0 to report silent contexts without ever writing the service off. | | `appendTrailingSpace` | `boolean \| undefined` | no | Whether to append a trailing space to text before sending to TTS. This helps prevent some TTS services from vocalizing trailing punctuation (e.g., "dot"). Only applied in sentence aggregation mode; when streaming tokens, the incoming text's own whitespace is preserved. | | `sampleRate` | `number \| null \| undefined` | no | Output sample rate for generated audio. | | `skipTextTypes` | `string[] \| null \| undefined` | no | List of text types that should not be spoken. | | `textTransforms` | `[string, TextTransform][] \| null \| undefined` | no | A list of TextTransforms to transform text just before sending it to TTS. Each TextTransform takes the aggregated text and its type, and returns the transformed text. To register, provide a list of tuples of `[textType \| "*", transformFunction]`. | | `transportDestination` | `string \| null \| undefined` | no | Destination for generated audio frames. | | `reuseContextIdWithinTurn` | `boolean \| undefined` | no | Whether the service should reuse context IDs within the same turn. | | `audioCache` | `TTSAudioCacheStore \| null \| undefined` | no | Where to keep synthesized audio for reuse (e.g. a shared `TTSAudioCache`). Text whose audio is in the cache is served from it, with no request to the provider. Lines get there through `warmAudioCache()`, and also from live synthesis when `cacheSynthesizedAudio` allows it. Only services that yield their audio from `runTts` (HTTP and streaming-RPC services, see `supportsAudioCache`) use it, and only in sentence aggregation mode. Defaults to null (no cache). pipedog only. | | `cacheSynthesizedAudio` | `boolean \| ((text: string) => boolean) \| undefined` | no | Whether audio synthesized live is stored in `audioCache`, or a function that decides per text. Defaults to false: only warmed lines are stored, since a generated reply can repeat what the user said. pipedog only. | | `audioCacheNamespace` | `string \| undefined` | no | Extra part of the cache key, for constructor options that change the audio but are not in `settings`. pipedog only. | | `ai` | `WorkersAIBinding \| null \| undefined` | no | The Workers AI binding (`env.AI`), inside a Worker. Takes precedence over the credentials. | | `accountId` | `string \| null \| undefined` | no | Cloudflare account ID, for the REST API. | | `apiToken` | `string \| null \| undefined` | no | Cloudflare API token with Workers AI access, for the REST API. | | `baseUrl` | `string \| null \| undefined` | no | Base URL of Cloudflare's API. Defaults to `CLOUDFLARE_API_URL`. | | `fetch` | `((input: RequestInfo \| URL, init?: RequestInit) => Promise) \| null \| undefined` | no | `fetch` used for REST requests. Defaults to the global `fetch`. | | `webSocketFactory` | `((uri: string, options: { headers?: Record; protocols?: string[]; }) => WebSocket \| Promise) \| null \| undefined` | no | Creates the `WebSocket` for REST sessions, to open it some other way. Defaults to `websocketConnect`'s, which sends the token header on Bun, Node, Deno and Cloudflare Workers. | | `client` | `WorkersAIClient \| null \| undefined` | no | A client to share between services. When set, the other connection options are ignored. | | `encoding` | `string \| undefined` | no | Audio encoding format. Defaults to "linear16". Must be one of SUPPORTED_ENCODINGS. | ### Settings [#settings] | Setting | Type | Default | Description | | ------- | -------- | -------------------------- | -------------------------------------------------------------------------------------------------------------- | | `model` | `string` | `"@cf/deepgram/aura-2-en"` | `WorkersAIAuraModel.AURA_2_EN`, `.AURA_2_ES` or `.AURA_1`. | | `voice` | `string` | the model's default | The `speaker`: `luna` (aura-2-en), `aquila` (aura-2-es), `angus` (aura-1). See the model pages for the others. | ### Usage [#usage] ```ts title="tts.ts" import { WorkersAIAuraModel, WorkersAIAuraTTSService } from "@pipedog/cloudflare/tts"; const tts = new WorkersAIAuraTTSService({ ai: env.AI, sampleRate: 24000, settings: new WorkersAIAuraTTSService.Settings({ model: WorkersAIAuraModel.AURA_2_EN, voice: "asteria", }), }); ``` Over the REST API: ```ts const ttsRest = new WorkersAIAuraTTSService({ accountId: process.env.CLOUDFLARE_ACCOUNT_ID ?? "", apiToken: process.env.CLOUDFLARE_API_TOKEN ?? "", settings: new WorkersAIAuraTTSService.Settings({ model: WorkersAIAuraModel.AURA_2_ES, voice: "celeste", }), }); ``` Change the voice mid-call with a `TTSUpdateSettingsFrame` (it reconnects): ```ts // Queue it on the worker, or push it from a processor upstream of the service. const update = new TTSUpdateSettingsFrame({ delta: new WorkersAIAuraTTSService.Settings({ voice: "orion" }), }); ``` ### Notes [#notes] * Text is sent with `Speak` as it arrives and flushed (`Flush`) at the end of each LLM response; an interruption sends `Clear`. Output is 16-bit PCM at the service's sample rate. * The session parameters are `speaker`, `encoding` and `sample_rate`. Deepgram's `speed` isn't in Workers AI's schema and isn't sent. ## WorkersAIAuraHttpTTSService [#workersaiaurahttpttsservice] ```ts title="tts.ts" import { WorkersAIAuraHttpTTSService } from "@pipedog/cloudflare/tts"; const ttsHttp = new WorkersAIAuraHttpTTSService({ ai: env.AI, sampleRate: 24000 }); ``` Each sentence is a request for `encoding: "linear16"`, `container: "none"` at the service's sample rate, and the body is streamed back as audio frames. Use it when you'd rather not hold a socket open. ## WorkersAIMeloTTSService [#workersaimelottsservice] ### Configuration [#configuration-1] | Option | Type | Required | Description | |---|---|---|---| | `decodeAudio` | `AudioDecoder \| null \| undefined` | no | Decodes MeloTTS's MP3 output. Required: without it every request fails with an error saying so. | | `settings` | `WorkersAIMeloTTSSettings \| null \| undefined` | no | Runtime-updatable settings. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `textAggregationMode` | `TextAggregationMode \| null \| undefined` | no | How to aggregate incoming text before synthesis. `TextAggregationMode.SENTENCE` (default) buffers until sentence boundaries, `TextAggregationMode.TOKEN` streams tokens directly for lower latency. | | `pushTextFrames` | `boolean \| undefined` | no | Whether to push TextFrames and LLMFullResponseEndFrames. | | `pushStopFrames` | `boolean \| undefined` | no | Whether to automatically push TTSStoppedFrames. | | `pushStartFrame` | `boolean \| undefined` | no | Whether to automatically create audio contexts and push TTSStartedFrames. When true, the base class handles `createAudioContext` and yields `TTSStartedFrame` before each synthesis call, so `runTts` implementations do not need to. | | `stopFrameTimeoutS` | `number \| undefined` | no | Idle time before pushing TTSStoppedFrame when pushStopFrames is true. | | `pushSilenceAfterStop` | `boolean \| undefined` | no | Whether to push silence audio after TTSStoppedFrame. | | `silenceTimeS` | `number \| undefined` | no | Duration of silence to push when pushSilenceAfterStop is true. | | `pauseFrameProcessing` | `boolean \| undefined` | no | Whether to hold incoming frames after a turn until its audio contexts end. Needed by services whose audio doesn't carry its request's context ID. | | `maxConsecutiveZeroAudioContexts` | `number \| undefined` | no | How many consecutive TTS contexts may complete without producing any audio before the service is reported unable to do its job. Catches a provider that accepts requests and stays silent — an unknown voice ID, say — which no error ever surfaces. Every silent context reports an error the service can carry on from, whatever the limit is; on reaching the limit the service reports a permanent error instead, stops being given work, and the pipeline worker applies its `ProcessorUnusablePolicy`. Set to 0 to report silent contexts without ever writing the service off. | | `appendTrailingSpace` | `boolean \| undefined` | no | Whether to append a trailing space to text before sending to TTS. This helps prevent some TTS services from vocalizing trailing punctuation (e.g., "dot"). Only applied in sentence aggregation mode; when streaming tokens, the incoming text's own whitespace is preserved. | | `sampleRate` | `number \| null \| undefined` | no | Output sample rate for generated audio. | | `skipTextTypes` | `string[] \| null \| undefined` | no | List of text types that should not be spoken. | | `textTransforms` | `[string, TextTransform][] \| null \| undefined` | no | A list of TextTransforms to transform text just before sending it to TTS. Each TextTransform takes the aggregated text and its type, and returns the transformed text. To register, provide a list of tuples of `[textType \| "*", transformFunction]`. | | `transportDestination` | `string \| null \| undefined` | no | Destination for generated audio frames. | | `reuseContextIdWithinTurn` | `boolean \| undefined` | no | Whether the service should reuse context IDs within the same turn. | | `audioCache` | `TTSAudioCacheStore \| null \| undefined` | no | Where to keep synthesized audio for reuse (e.g. a shared `TTSAudioCache`). Text whose audio is in the cache is served from it, with no request to the provider. Lines get there through `warmAudioCache()`, and also from live synthesis when `cacheSynthesizedAudio` allows it. Only services that yield their audio from `runTts` (HTTP and streaming-RPC services, see `supportsAudioCache`) use it, and only in sentence aggregation mode. Defaults to null (no cache). pipedog only. | | `cacheSynthesizedAudio` | `boolean \| ((text: string) => boolean) \| undefined` | no | Whether audio synthesized live is stored in `audioCache`, or a function that decides per text. Defaults to false: only warmed lines are stored, since a generated reply can repeat what the user said. pipedog only. | | `audioCacheNamespace` | `string \| undefined` | no | Extra part of the cache key, for constructor options that change the audio but are not in `settings`. pipedog only. | | `ai` | `WorkersAIBinding \| null \| undefined` | no | The Workers AI binding (`env.AI`), inside a Worker. Takes precedence over the credentials. | | `accountId` | `string \| null \| undefined` | no | Cloudflare account ID, for the REST API. | | `apiToken` | `string \| null \| undefined` | no | Cloudflare API token with Workers AI access, for the REST API. | | `baseUrl` | `string \| null \| undefined` | no | Base URL of Cloudflare's API. Defaults to `CLOUDFLARE_API_URL`. | | `fetch` | `((input: RequestInfo \| URL, init?: RequestInit) => Promise) \| null \| undefined` | no | `fetch` used for REST requests. Defaults to the global `fetch`. | | `webSocketFactory` | `((uri: string, options: { headers?: Record; protocols?: string[]; }) => WebSocket \| Promise) \| null \| undefined` | no | Creates the `WebSocket` for REST sessions, to open it some other way. Defaults to `websocketConnect`'s, which sends the token header on Bun, Node, Deno and Cloudflare Workers. | | `client` | `WorkersAIClient \| null \| undefined` | no | A client to share between services. When set, the other connection options are ignored. | ### Usage [#usage-1] ```ts title="tts.ts" import { WorkersAIMeloTTSService } from "@pipedog/cloudflare/melotts"; // Any MP3 decoder that returns per-channel Float32Arrays and a sample rate, // e.g. mpg123-decoder's MPEGDecoder (bun add mpg123-decoder): // const decoder = new MPEGDecoder(); await decoder.ready; // decodeAudio: (mp3) => { decoder.reset(); return decoder.decode(mp3); } declare function decodeMp3(mp3: Uint8Array): Promise<{ channelData: Float32Array[]; sampleRate: number; }>; const melo = new WorkersAIMeloTTSService({ ai: env.AI, decodeAudio: decodeMp3, settings: new WorkersAIMeloTTSService.Settings({ language: Language.FR }), }); ``` ### Notes [#notes-1] * MeloTTS answers with MP3 only. pipedog ships no MP3 decoder (Workers have no Web Audio, and a WebAssembly decoder has to be imported as a module there), so pass `decodeAudio`. The decoded audio is mixed to mono and resampled to the service rate. * `language` picks the voice: English, Spanish, French, Chinese, Japanese or Korean. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/cloudflare/package.json`: ✅ means the compat smoke test passes on that runtime. * Tested against a local fake of Workers AI over both paths, not against Workers AI itself. Whether Aura's WebSocket opens through the binding (`websocket: true`) is documented for Flux but not for Aura: if it doesn't for you, use `WorkersAIAuraHttpTTSService` or the REST path. # Deepgram > Text-to-speech with DeepgramTTSService (Aura over WebSocket), DeepgramHttpTTSService (Aura over HTTP) and DeepgramFluxTTSService (Flux). Source: https://polvallverdu.github.io/pipedog/docs/services/tts/deepgram/ ## Overview [#overview] pipedog ports three of Pipecat's Deepgram text-to-speech services: * **`DeepgramTTSService`**: Deepgram Aura over a WebSocket (`/v1/speak`). Streams audio as text arrives and clears Deepgram's buffer on interruption. * **`DeepgramHttpTTSService`**: Deepgram Aura over HTTP. One `POST` per sentence, with the audio streamed back in the response body. * **`DeepgramFluxTTSService`**: Deepgram Flux over a WebSocket (`/v2/speak`). Streams LLM tokens as they arrive and keeps acoustic state across turns on one connection. Pipecat's documentation for the same services. `packages/deepgram/src/tts.ts`, `flux/tts.ts` and `flux/tts_base.ts`. Aura and Flux voices and voice controls. ## Installation [#installation] ```bash bun add @pipedog/deepgram ``` ```ts import { DeepgramHttpTTSService, DeepgramTTSService } from "@pipedog/deepgram/tts"; import { DeepgramFluxTTSService } from "@pipedog/deepgram/flux/tts"; ``` ## Prerequisites [#prerequisites] * A Deepgram account and an API key from the [Deepgram console](https://console.deepgram.com). Flux needs a key with Flux TTS access. * Pass the key as `apiKey`. The examples read it from `DEEPGRAM_API_KEY`. ## DeepgramTTSService [#deepgramttsservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | Deepgram API key for authentication. | | `baseUrl` | `string \| undefined` | no | WebSocket base URL for Deepgram API. Defaults to "wss://api.deepgram.com". | | `encoding` | `string \| undefined` | no | Audio encoding format. Defaults to "linear16". Must be one of SUPPORTED_ENCODINGS. | | `mipOptOut` | `boolean \| null \| undefined` | no | Opt out of the Deepgram Model Improvement Program. See https://dpgr.am/deepgram-mip for pricing impacts before setting to true. | | `settings` | `DeepgramTTSSettings \| null \| undefined` | no | Runtime-updatable settings. | | `webSocketFactory` | `((uri: string, options: { headers?: Record; protocols?: string[]; }) => WebSocket \| Promise) \| null \| undefined` | no | Creates the underlying `WebSocket`, to open it some other way (through a proxy, say). Defaults to `websocketConnect`'s, which sends the header on every runtime, Cloudflare Workers included. Not in upstream. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `reconnectBackoffMinWait` | `number \| undefined` | no | Minimum time, in seconds, to wait between reconnection attempts. | | `reconnectBackoffMaxWait` | `number \| undefined` | no | Maximum time, in seconds, to wait between reconnection attempts. | | `reconnectOnError` | `boolean \| undefined` | no | Whether to automatically reconnect on connection errors. | | `wsCloseTimeout` | `number \| undefined` | no | Maximum time, in seconds, to wait for the peer to acknowledge the websocket closing handshake before dropping the connection. Applied to connections opened through `_websocketConnect()`. Increase it for peers that need longer to complete a graceful close. | | `wsSendTimeout` | `number \| null \| undefined` | no | Maximum time, in seconds, a single send may wait on a peer that has stopped reading before the connection is dropped and the send throws `ConnectionClosedError`. Applied to connections opened through `_websocketConnect()`. Pass `null` to wait indefinitely. Defaults to `WS_SEND_TIMEOUT`. | | `textAggregationMode` | `TextAggregationMode \| null \| undefined` | no | How to aggregate incoming text before synthesis. `TextAggregationMode.SENTENCE` (default) buffers until sentence boundaries, `TextAggregationMode.TOKEN` streams tokens directly for lower latency. | | `pushTextFrames` | `boolean \| undefined` | no | Whether to push TextFrames and LLMFullResponseEndFrames. | | `pushStopFrames` | `boolean \| undefined` | no | Whether to automatically push TTSStoppedFrames. | | `pushStartFrame` | `boolean \| undefined` | no | Whether to automatically create audio contexts and push TTSStartedFrames. When true, the base class handles `createAudioContext` and yields `TTSStartedFrame` before each synthesis call, so `runTts` implementations do not need to. | | `stopFrameTimeoutS` | `number \| undefined` | no | Idle time before pushing TTSStoppedFrame when pushStopFrames is true. | | `pushSilenceAfterStop` | `boolean \| undefined` | no | Whether to push silence audio after TTSStoppedFrame. | | `silenceTimeS` | `number \| undefined` | no | Duration of silence to push when pushSilenceAfterStop is true. | | `pauseFrameProcessing` | `boolean \| undefined` | no | Whether to hold incoming frames after a turn until its audio contexts end. Needed by services whose audio doesn't carry its request's context ID. | | `maxConsecutiveZeroAudioContexts` | `number \| undefined` | no | How many consecutive TTS contexts may complete without producing any audio before the service is reported unable to do its job. Catches a provider that accepts requests and stays silent — an unknown voice ID, say — which no error ever surfaces. Every silent context reports an error the service can carry on from, whatever the limit is; on reaching the limit the service reports a permanent error instead, stops being given work, and the pipeline worker applies its `ProcessorUnusablePolicy`. Set to 0 to report silent contexts without ever writing the service off. | | `appendTrailingSpace` | `boolean \| undefined` | no | Whether to append a trailing space to text before sending to TTS. This helps prevent some TTS services from vocalizing trailing punctuation (e.g., "dot"). Only applied in sentence aggregation mode; when streaming tokens, the incoming text's own whitespace is preserved. | | `sampleRate` | `number \| null \| undefined` | no | Output sample rate for generated audio. | | `skipTextTypes` | `string[] \| null \| undefined` | no | List of text types that should not be spoken. | | `textTransforms` | `[string, TextTransform][] \| null \| undefined` | no | A list of TextTransforms to transform text just before sending it to TTS. Each TextTransform takes the aggregated text and its type, and returns the transformed text. To register, provide a list of tuples of `[textType \| "*", transformFunction]`. | | `transportDestination` | `string \| null \| undefined` | no | Destination for generated audio frames. | | `reuseContextIdWithinTurn` | `boolean \| undefined` | no | Whether the service should reuse context IDs within the same turn. | | `audioCache` | `TTSAudioCacheStore \| null \| undefined` | no | Where to keep synthesized audio for reuse (e.g. a shared `TTSAudioCache`). Text whose audio is in the cache is served from it, with no request to the provider. Lines get there through `warmAudioCache()`, and also from live synthesis when `cacheSynthesizedAudio` allows it. Only services that yield their audio from `runTts` (HTTP and streaming-RPC services, see `supportsAudioCache`) use it, and only in sentence aggregation mode. Defaults to null (no cache). pipedog only. | | `cacheSynthesizedAudio` | `boolean \| ((text: string) => boolean) \| undefined` | no | Whether audio synthesized live is stored in `audioCache`, or a function that decides per text. Defaults to false: only warmed lines are stored, since a generated reply can repeat what the user said. pipedog only. | | `audioCacheNamespace` | `string \| undefined` | no | Extra part of the cache key, for constructor options that change the audio but are not in `settings`. pipedog only. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new DeepgramTTSService.Settings({...})`. Any setting you leave out keeps the default below. `DeepgramHttpTTSService` uses the same settings class. | Setting | Type | Default | Description | | ---------- | ------------------ | -------------------- | ------------------------------------------------------------------------------------------------------------------------ | | `voice` | `string` | `"aura-2-helena-en"` | Aura voice. Sent to Deepgram as the `model` query parameter. | | `model` | `string \| null` | `null` | Kept in sync with `voice` when the voice changes, for metrics. | | `language` | `Language \| null` | `null` | Used to split text into sentences. Not sent to Deepgram. | | `speed` | `number \| null` | `null` | Speech-rate multiplier, from 0.7 to 1.5. `null` leaves Aura at its default rate. Aura-2 English and Spanish voices only. | ### Usage [#usage] ```ts title="tts.ts" import { TTSUpdateSettingsFrame } from "@pipedog/core/frames/frames"; import { TextAggregationMode } from "@pipedog/core/services/tts_service"; import { DeepgramFluxTTSService } from "@pipedog/deepgram/flux/tts"; import { DeepgramHttpTTSService, DeepgramTTSService } from "@pipedog/deepgram/tts"; const tts = new DeepgramTTSService({ apiKey: process.env.DEEPGRAM_API_KEY ?? "", settings: new DeepgramTTSService.Settings({ voice: "aura-2-helena-en", speed: 1.1, }), }); ``` Change settings mid-call with a `TTSUpdateSettingsFrame`. Any change closes the WebSocket and reconnects with the new settings: ```ts // Queue it on the worker, or push it from a processor upstream of the service. // Any change reconnects the WebSocket with the new settings. const update = new TTSUpdateSettingsFrame({ delta: new DeepgramTTSService.Settings({ voice: "aura-2-thalia-en" }), }); ``` ### Event handlers [#event-handlers] | Event | Description | | --------------------- | ------------------------------------------------------------------------ | | `on_connected` | The WebSocket to Deepgram is open. | | `on_disconnected` | The WebSocket to Deepgram closed. | | `on_connection_error` | Connecting failed. Receives the error message. | | `on_tts_request` | A TTS request is about to be made. Receives the context ID and the text. | ### Notes [#notes] * `encoding` must be `"linear16"`, `"mulaw"` or `"alaw"`; anything else throws in the constructor. The received bytes go into `TTSAudioRawFrame`s as they are, so only `"linear16"` plays correctly in a pipedog pipeline. * Text is sent with `Speak` messages. The service sends `Flush` when the LLM response ends, and emits `TTSStoppedFrame` when Deepgram answers with `Flushed`. * On interruption it sends `Clear`, so Deepgram drops its text buffer and stops sending audio. * `sampleRate` defaults to the pipeline's output rate. * Aura-2 reads inline pronunciations. `formatPronunciation(word, ipa)` renders one as Deepgram's inline object, and returns `null` when the IPA is empty or too long for the word (over 128 characters, or over 10 times the word's length with a floor of 15). ## DeepgramHttpTTSService [#deepgramhttpttsservice] ### Configuration [#configuration-1] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | Deepgram API key for authentication. | | `fetch` | `{ (input: RequestInfo \| URL, init?: RequestInit): Promise; (input: string \| URL \| Request, init?: RequestInit): Promise; } \| null \| undefined` | no | `fetch` used for HTTP requests. Defaults to the global `fetch`. DIVERGENCE: upstream requires a shared `aiohttp_session`. | | `baseUrl` | `string \| undefined` | no | Custom base URL for Deepgram API. Defaults to "https://api.deepgram.com". | | `encoding` | `string \| undefined` | no | Audio encoding format. Defaults to "linear16". | | `mipOptOut` | `boolean \| null \| undefined` | no | Opt out of the Deepgram Model Improvement Program. See https://dpgr.am/deepgram-mip for pricing impacts before setting to true. | | `settings` | `DeepgramTTSSettings \| null \| undefined` | no | Runtime-updatable settings. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `textAggregationMode` | `TextAggregationMode \| null \| undefined` | no | How to aggregate incoming text before synthesis. `TextAggregationMode.SENTENCE` (default) buffers until sentence boundaries, `TextAggregationMode.TOKEN` streams tokens directly for lower latency. | | `pushTextFrames` | `boolean \| undefined` | no | Whether to push TextFrames and LLMFullResponseEndFrames. | | `pushStopFrames` | `boolean \| undefined` | no | Whether to automatically push TTSStoppedFrames. | | `pushStartFrame` | `boolean \| undefined` | no | Whether to automatically create audio contexts and push TTSStartedFrames. When true, the base class handles `createAudioContext` and yields `TTSStartedFrame` before each synthesis call, so `runTts` implementations do not need to. | | `stopFrameTimeoutS` | `number \| undefined` | no | Idle time before pushing TTSStoppedFrame when pushStopFrames is true. | | `pushSilenceAfterStop` | `boolean \| undefined` | no | Whether to push silence audio after TTSStoppedFrame. | | `silenceTimeS` | `number \| undefined` | no | Duration of silence to push when pushSilenceAfterStop is true. | | `pauseFrameProcessing` | `boolean \| undefined` | no | Whether to hold incoming frames after a turn until its audio contexts end. Needed by services whose audio doesn't carry its request's context ID. | | `maxConsecutiveZeroAudioContexts` | `number \| undefined` | no | How many consecutive TTS contexts may complete without producing any audio before the service is reported unable to do its job. Catches a provider that accepts requests and stays silent — an unknown voice ID, say — which no error ever surfaces. Every silent context reports an error the service can carry on from, whatever the limit is; on reaching the limit the service reports a permanent error instead, stops being given work, and the pipeline worker applies its `ProcessorUnusablePolicy`. Set to 0 to report silent contexts without ever writing the service off. | | `appendTrailingSpace` | `boolean \| undefined` | no | Whether to append a trailing space to text before sending to TTS. This helps prevent some TTS services from vocalizing trailing punctuation (e.g., "dot"). Only applied in sentence aggregation mode; when streaming tokens, the incoming text's own whitespace is preserved. | | `sampleRate` | `number \| null \| undefined` | no | Output sample rate for generated audio. | | `skipTextTypes` | `string[] \| null \| undefined` | no | List of text types that should not be spoken. | | `textTransforms` | `[string, TextTransform][] \| null \| undefined` | no | A list of TextTransforms to transform text just before sending it to TTS. Each TextTransform takes the aggregated text and its type, and returns the transformed text. To register, provide a list of tuples of `[textType \| "*", transformFunction]`. | | `transportDestination` | `string \| null \| undefined` | no | Destination for generated audio frames. | | `reuseContextIdWithinTurn` | `boolean \| undefined` | no | Whether the service should reuse context IDs within the same turn. | | `audioCache` | `TTSAudioCacheStore \| null \| undefined` | no | Where to keep synthesized audio for reuse (e.g. a shared `TTSAudioCache`). Text whose audio is in the cache is served from it, with no request to the provider. Lines get there through `warmAudioCache()`, and also from live synthesis when `cacheSynthesizedAudio` allows it. Only services that yield their audio from `runTts` (HTTP and streaming-RPC services, see `supportsAudioCache`) use it, and only in sentence aggregation mode. Defaults to null (no cache). pipedog only. | | `cacheSynthesizedAudio` | `boolean \| ((text: string) => boolean) \| undefined` | no | Whether audio synthesized live is stored in `audioCache`, or a function that decides per text. Defaults to false: only warmed lines are stored, since a generated reply can repeat what the user said. pipedog only. | | `audioCacheNamespace` | `string \| undefined` | no | Extra part of the cache key, for constructor options that change the audio but are not in `settings`. pipedog only. | ### Settings [#settings-1] Same settings and defaults as [`DeepgramTTSService`](#settings). ### Usage [#usage-1] ```ts title="http.ts" const httpTts = new DeepgramHttpTTSService({ apiKey: process.env.DEEPGRAM_API_KEY ?? "", settings: new DeepgramHttpTTSService.Settings({ voice: "aura-2-helena-en" }), }); ``` ### Notes [#notes-1] * Each request is a `POST` to `{baseUrl}/v1/speak` with `container=none`. A status other than 200 becomes an `ErrorFrame` with the response body. * The response body is read as a stream and cut into sample-aligned `TTSAudioRawFrame`s, so audio starts before the whole response has arrived. * Upstream requires an `aiohttp_session`. Here `fetch` is optional and defaults to the global `fetch`. ## DeepgramFluxTTSService [#deepgramfluxttsservice] ### Configuration [#configuration-2] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | Deepgram API key for authentication. | | `url` | `string \| undefined` | no | WebSocket URL for the Flux TTS API. Defaults to "wss://api.deepgram.com/v2/speak". | | `settings` | `DeepgramFluxTTSSettings \| null \| undefined` | no | Runtime-updatable settings. | | `webSocketFactory` | `((uri: string, options: { headers?: Record; protocols?: string[]; }) => WebSocket \| Promise) \| null \| undefined` | no | Creates the underlying `WebSocket`, to open it some other way (through a proxy, say). Defaults to `websocketConnect`'s, which sends the header on every runtime, Cloudflare Workers included. Not in upstream. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `reconnectBackoffMinWait` | `number \| undefined` | no | Minimum time, in seconds, to wait between reconnection attempts. | | `reconnectBackoffMaxWait` | `number \| undefined` | no | Maximum time, in seconds, to wait between reconnection attempts. | | `reconnectOnError` | `boolean \| undefined` | no | Whether to automatically reconnect on connection errors. | | `wsCloseTimeout` | `number \| undefined` | no | Maximum time, in seconds, to wait for the peer to acknowledge the websocket closing handshake before dropping the connection. Applied to connections opened through `_websocketConnect()`. Increase it for peers that need longer to complete a graceful close. | | `wsSendTimeout` | `number \| null \| undefined` | no | Maximum time, in seconds, a single send may wait on a peer that has stopped reading before the connection is dropped and the send throws `ConnectionClosedError`. Applied to connections opened through `_websocketConnect()`. Pass `null` to wait indefinitely. Defaults to `WS_SEND_TIMEOUT`. | | `pushTextFrames` | `boolean \| undefined` | no | Whether to push TextFrames and LLMFullResponseEndFrames. | | `pushStopFrames` | `boolean \| undefined` | no | Whether to automatically push TTSStoppedFrames. | | `pushStartFrame` | `boolean \| undefined` | no | Whether to automatically create audio contexts and push TTSStartedFrames. When true, the base class handles `createAudioContext` and yields `TTSStartedFrame` before each synthesis call, so `runTts` implementations do not need to. | | `stopFrameTimeoutS` | `number \| undefined` | no | Idle time before pushing TTSStoppedFrame when pushStopFrames is true. | | `pushSilenceAfterStop` | `boolean \| undefined` | no | Whether to push silence audio after TTSStoppedFrame. | | `silenceTimeS` | `number \| undefined` | no | Duration of silence to push when pushSilenceAfterStop is true. | | `pauseFrameProcessing` | `boolean \| undefined` | no | Whether to hold incoming frames after a turn until its audio contexts end. Needed by services whose audio doesn't carry its request's context ID. | | `maxConsecutiveZeroAudioContexts` | `number \| undefined` | no | How many consecutive TTS contexts may complete without producing any audio before the service is reported unable to do its job. Catches a provider that accepts requests and stays silent — an unknown voice ID, say — which no error ever surfaces. Every silent context reports an error the service can carry on from, whatever the limit is; on reaching the limit the service reports a permanent error instead, stops being given work, and the pipeline worker applies its `ProcessorUnusablePolicy`. Set to 0 to report silent contexts without ever writing the service off. | | `appendTrailingSpace` | `boolean \| undefined` | no | Whether to append a trailing space to text before sending to TTS. This helps prevent some TTS services from vocalizing trailing punctuation (e.g., "dot"). Only applied in sentence aggregation mode; when streaming tokens, the incoming text's own whitespace is preserved. | | `sampleRate` | `number \| null \| undefined` | no | Output sample rate for generated audio. | | `skipTextTypes` | `string[] \| null \| undefined` | no | List of text types that should not be spoken. | | `textTransforms` | `[string, TextTransform][] \| null \| undefined` | no | A list of TextTransforms to transform text just before sending it to TTS. Each TextTransform takes the aggregated text and its type, and returns the transformed text. To register, provide a list of tuples of `[textType \| "*", transformFunction]`. | | `transportDestination` | `string \| null \| undefined` | no | Destination for generated audio frames. | | `reuseContextIdWithinTurn` | `boolean \| undefined` | no | Whether the service should reuse context IDs within the same turn. | | `audioCache` | `TTSAudioCacheStore \| null \| undefined` | no | Where to keep synthesized audio for reuse (e.g. a shared `TTSAudioCache`). Text whose audio is in the cache is served from it, with no request to the provider. Lines get there through `warmAudioCache()`, and also from live synthesis when `cacheSynthesizedAudio` allows it. Only services that yield their audio from `runTts` (HTTP and streaming-RPC services, see `supportsAudioCache`) use it, and only in sentence aggregation mode. Defaults to null (no cache). pipedog only. | | `cacheSynthesizedAudio` | `boolean \| ((text: string) => boolean) \| undefined` | no | Whether audio synthesized live is stored in `audioCache`, or a function that decides per text. Defaults to false: only warmed lines are stored, since a generated reply can repeat what the user said. pipedog only. | | `audioCacheNamespace` | `string \| undefined` | no | Extra part of the cache key, for constructor options that change the audio but are not in `settings`. pipedog only. | | `mipOptOut` | `boolean \| null \| undefined` | no | Opt out of the Deepgram Model Improvement Program. See https://dpgr.am/deepgram-mip for pricing impacts before setting to true. | | `tag` | `string[] \| null \| undefined` | no | Tags to label requests for identification during usage reporting. | | `textAggregationMode` | `TextAggregationMode \| undefined` | no | How to aggregate incoming text before synthesis. Defaults to `TextAggregationMode.TOKEN`, streaming LLM tokens straight to Flux for the lowest latency. | ### Settings [#settings-2] Runtime-updatable settings, passed as `settings: new DeepgramFluxTTSService.Settings({...})`. | Setting | Type | Default | Description | | -------------- | --------------------------------- | ------------------- | -------------------------------------------------------------------------------------------- | | `voice` | `string` | `"flux-heather-en"` | Flux voice, `flux-{voice}-{language}`. Sent as the `model` query parameter. | | `model` | `string \| null` | synced to `voice` | Always set to `voice`, for metrics. | | `language` | `Language \| null` | `null` | Used to split text into sentences. Not sent to Deepgram. | | `speed` | `number \| null` | `null` | Speech-rate multiplier, 0.5 to 1.5 in steps of 0.05. `null` leaves Flux at its default rate. | | `expressivity` | `-2 \| -1 \| 0 \| 1 \| 2 \| null` | `null` | Expressive range, calm to animated. `null` leaves Flux at its default range. | ### Usage [#usage-2] ```ts title="flux.ts" const fluxTts = new DeepgramFluxTTSService({ apiKey: process.env.DEEPGRAM_API_KEY ?? "", settings: new DeepgramFluxTTSService.Settings({ voice: "flux-alexis-en", speed: 1.05, expressivity: 1, }), }); fluxTts.addEventHandler("on_connection_error", async (_service, error: string) => { console.error("Flux connection error:", error); }); ``` Flux streams LLM tokens by default (`TextAggregationMode.TOKEN`). To aggregate text into sentences first: ```ts // Aggregate text into sentences instead of streaming LLM tokens. const fluxSentences = new DeepgramFluxTTSService({ apiKey: process.env.DEEPGRAM_API_KEY ?? "", textAggregationMode: TextAggregationMode.SENTENCE, }); ``` A `speed` change is applied to the open connection with a Flux `Configure` message, so the cross-turn acoustic state is kept. Any other change (`voice`, `expressivity`, `language`) reconnects: ```ts // A speed change is sent on the open connection with a Configure message. const fluxSpeed = new TTSUpdateSettingsFrame({ delta: new DeepgramFluxTTSService.Settings({ speed: 0.9 }), }); ``` ### Event handlers [#event-handlers-1] | Event | Description | | --------------------- | ------------------------------------------------------------------------------------- | | `on_connected` | The WebSocket to Flux is open. | | `on_disconnected` | The WebSocket to Flux closed. | | `on_connection_error` | Connecting failed, or the receive loop reported an error. Receives the error message. | | `on_tts_request` | A TTS request is about to be made. Receives the context ID and the text. | ### Notes [#notes-2] * Audio is always requested as `linear16`. Supported sample rates are 8000, 16000, 24000, 32000, 44100 and 48000 Hz. Another rate only logs a warning on connect. * On interruption the service sends `Interrupt`, which ends the active turn without closing the connection. `Flush` ends a turn when the LLM response is done. * Clearing `speed` (setting it to `null`) sends `Configure` with `speed: 1.0`, Flux's default. * Disconnecting closes the socket without a `Close` message: in Flux, `Close` asks the server to finish the active turn's audio first. * `formatPronunciation` is left at the base implementation, so words are spoken as written. Flux rejects inline pronunciations on a connection whose speed is not 1.0. * `tag` adds one `tag` query parameter per entry, for Deepgram usage reports. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/deepgram/package.json`: ✅ means the compat smoke test passes on that runtime. `DeepgramTTSService` and `DeepgramFluxTTSService` use the global `WebSocket` and send the key as an `Authorization: Token ` header on the handshake. That works on Bun, Node, Deno and Cloudflare Workers (where the socket is opened through `fetch`). Pass `webSocketFactory` to open the socket some other way, through a proxy for example. A standard `WebSocket` does not expose the handshake's response headers, so the `dg-*` headers upstream logs on connect are not available. `DeepgramHttpTTSService` only needs `fetch`, available on all four runtimes. Query values use JS number formatting, so `speed: 1.0` is sent as `speed=1`. The deprecated `voice` constructor option is not ported; use `settings.voice`. Not ported yet: `DeepgramSageMakerTTSService` and `DeepgramFluxSageMakerTTSService` (AWS SageMaker endpoints). # ElevenLabs > Text-to-speech with ElevenLabsTTSService (multi-context WebSocket), ElevenLabsHttpTTSService (HTTP streaming) and ElevenLabsDialogueTTSService (Text to Dialogue), all with word timestamps. Source: https://polvallverdu.github.io/pipedog/docs/services/tts/elevenlabs/ ## Overview [#overview] ElevenLabs turns text into speech with its Flash, Turbo, Multilingual and v3 models. pipedog ports all three of Pipecat's ElevenLabs text-to-speech services: * **`ElevenLabsTTSService`**: streams over ElevenLabs' multi-context WebSocket API (`multi-stream-input`), with word-level timestamps and per-turn audio contexts. * **`ElevenLabsHttpTTSService`**: posts each request to the `stream/with-timestamps` HTTP endpoint and streams the audio and alignment back. * **`ElevenLabsDialogueTTSService`**: streams over ElevenLabs' multi-context Text to Dialogue WebSocket, which serves the Eleven v3 and v4 dialogue models. Pipecat's documentation for the same services. `packages/elevenlabs/src/tts.ts`, `dialogue/tts.ts` and `tts_base.ts`. Voices, models and voice settings. ## Installation [#installation] ```bash bun add @pipedog/elevenlabs ``` ```ts import { ElevenLabsDialogueTTSService } from "@pipedog/elevenlabs/dialogue/tts"; import { ElevenLabsHttpTTSService, ElevenLabsTTSService } from "@pipedog/elevenlabs/tts"; ``` ## Prerequisites [#prerequisites] * An ElevenLabs account and an API key from the [ElevenLabs dashboard](https://elevenlabs.io/app/settings/api-keys). * A voice id from the ElevenLabs voice library. There is no default voice. * Pass the key as `apiKey`. The examples read it from `ELEVENLABS_API_KEY`. ## ElevenLabsTTSService [#elevenlabsttsservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `url` | `string \| undefined` | no | WebSocket URL for ElevenLabs TTS API. Defaults to "wss://api.elevenlabs.io". | | `autoMode` | `boolean \| null \| undefined` | no | Whether to enable ElevenLabs' auto mode, which reduces latency by disabling server-side chunk scheduling and buffering. Recommended when sending complete sentences or phrases. When null (default), auto mode is enabled for `SENTENCE` aggregation and disabled for `TOKEN` aggregation — because token streaming relies on the server-side chunk scheduler to accumulate enough text for natural-sounding synthesis. | | `enableSsmlParsing` | `boolean \| null \| undefined` | no | Whether to parse SSML tags in text. | | `settings` | `ElevenLabsTTSSettings \| null \| undefined` | no | Runtime-updatable settings. Its given fields override the defaults (model "eleven_flash_v2_5"). | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `reconnectBackoffMinWait` | `number \| undefined` | no | Minimum time, in seconds, to wait between reconnection attempts. | | `reconnectBackoffMaxWait` | `number \| undefined` | no | Maximum time, in seconds, to wait between reconnection attempts. | | `reconnectOnError` | `boolean \| undefined` | no | Whether to automatically reconnect on connection errors. | | `wsCloseTimeout` | `number \| undefined` | no | Maximum time, in seconds, to wait for the peer to acknowledge the websocket closing handshake before dropping the connection. Applied to connections opened through `_websocketConnect()`. Increase it for peers that need longer to complete a graceful close. | | `wsSendTimeout` | `number \| null \| undefined` | no | Maximum time, in seconds, a single send may wait on a peer that has stopped reading before the connection is dropped and the send throws `ConnectionClosedError`. Applied to connections opened through `_websocketConnect()`. Pass `null` to wait indefinitely. Defaults to `WS_SEND_TIMEOUT`. | | `textAggregationMode` | `TextAggregationMode \| null \| undefined` | no | How to aggregate incoming text before synthesis. `TextAggregationMode.SENTENCE` (default) buffers until sentence boundaries, `TextAggregationMode.TOKEN` streams tokens directly for lower latency. | | `pushTextFrames` | `boolean \| undefined` | no | Whether to push TextFrames and LLMFullResponseEndFrames. | | `pushStopFrames` | `boolean \| undefined` | no | Whether to automatically push TTSStoppedFrames. | | `pushStartFrame` | `boolean \| undefined` | no | Whether to automatically create audio contexts and push TTSStartedFrames. When true, the base class handles `createAudioContext` and yields `TTSStartedFrame` before each synthesis call, so `runTts` implementations do not need to. | | `stopFrameTimeoutS` | `number \| undefined` | no | Idle time before pushing TTSStoppedFrame when pushStopFrames is true. | | `pushSilenceAfterStop` | `boolean \| undefined` | no | Whether to push silence audio after TTSStoppedFrame. | | `silenceTimeS` | `number \| undefined` | no | Duration of silence to push when pushSilenceAfterStop is true. | | `pauseFrameProcessing` | `boolean \| undefined` | no | Whether to hold incoming frames after a turn until its audio contexts end. Needed by services whose audio doesn't carry its request's context ID. | | `maxConsecutiveZeroAudioContexts` | `number \| undefined` | no | How many consecutive TTS contexts may complete without producing any audio before the service is reported unable to do its job. Catches a provider that accepts requests and stays silent — an unknown voice ID, say — which no error ever surfaces. Every silent context reports an error the service can carry on from, whatever the limit is; on reaching the limit the service reports a permanent error instead, stops being given work, and the pipeline worker applies its `ProcessorUnusablePolicy`. Set to 0 to report silent contexts without ever writing the service off. | | `appendTrailingSpace` | `boolean \| undefined` | no | Whether to append a trailing space to text before sending to TTS. This helps prevent some TTS services from vocalizing trailing punctuation (e.g., "dot"). Only applied in sentence aggregation mode; when streaming tokens, the incoming text's own whitespace is preserved. | | `sampleRate` | `number \| null \| undefined` | no | Output sample rate for generated audio. | | `skipTextTypes` | `string[] \| null \| undefined` | no | List of text types that should not be spoken. | | `textTransforms` | `[string, TextTransform][] \| null \| undefined` | no | A list of TextTransforms to transform text just before sending it to TTS. Each TextTransform takes the aggregated text and its type, and returns the transformed text. To register, provide a list of tuples of `[textType \| "*", transformFunction]`. | | `transportDestination` | `string \| null \| undefined` | no | Destination for generated audio frames. | | `reuseContextIdWithinTurn` | `boolean \| undefined` | no | Whether the service should reuse context IDs within the same turn. | | `audioCache` | `TTSAudioCacheStore \| null \| undefined` | no | Where to keep synthesized audio for reuse (e.g. a shared `TTSAudioCache`). Text whose audio is in the cache is served from it, with no request to the provider. Lines get there through `warmAudioCache()`, and also from live synthesis when `cacheSynthesizedAudio` allows it. Only services that yield their audio from `runTts` (HTTP and streaming-RPC services, see `supportsAudioCache`) use it, and only in sentence aggregation mode. Defaults to null (no cache). pipedog only. | | `cacheSynthesizedAudio` | `boolean \| ((text: string) => boolean) \| undefined` | no | Whether audio synthesized live is stored in `audioCache`, or a function that decides per text. Defaults to false: only warmed lines are stored, since a generated reply can repeat what the user said. pipedog only. | | `audioCacheNamespace` | `string \| undefined` | no | Extra part of the cache key, for constructor options that change the audio but are not in `settings`. pipedog only. | | `apiKey` | `string` | yes | ElevenLabs API key for authentication. | | `enableLogging` | `boolean \| null \| undefined` | no | Whether to enable ElevenLabs server-side logging. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new ElevenLabsTTSService.Settings({...})`. Any setting you leave out keeps the default below. `null` means the field is not sent, so ElevenLabs uses its own default. | Setting | Type | Default | Description | | ------------------------ | --------------------------------- | --------------------- | ------------------------------------------------------------------ | | `model` | `string` | `"eleven_flash_v2_5"` | ElevenLabs model id. | | `voice` | `string \| null` | `null` | Voice id. Set one: there is no default. | | `language` | `Language \| null` | `null` | Synthesis language. Sent only to models that take a language code. | | `stability` | `number \| null` | `null` | Voice stability, 0.0 to 1.0. | | `similarityBoost` | `number \| null` | `null` | Similarity boost, 0.0 to 1.0. | | `style` | `number \| null` | `null` | Style exaggeration, 0.0 to 1.0. | | `useSpeakerBoost` | `boolean \| null` | `null` | Speaker boost. | | `speed` | `number \| null` | `null` | Speaking speed, 0.7 to 1.2. | | `applyTextNormalization` | `"auto" \| "on" \| "off" \| null` | `null` | Text normalization mode. | ### Usage [#usage] ```ts title="tts.ts" import { TTSUpdateSettingsFrame } from "@pipedog/core/frames/frames"; import { TextAggregationMode } from "@pipedog/core/services/tts_service"; import { Language } from "@pipedog/core/transcriptions/language"; import { ElevenLabsHttpTTSService, ElevenLabsTTSService } from "@pipedog/elevenlabs/tts"; const tts = new ElevenLabsTTSService({ apiKey: process.env.ELEVENLABS_API_KEY ?? "", settings: new ElevenLabsTTSService.Settings({ voice: "your-voice-id", model: "eleven_flash_v2_5", language: Language.EN, stability: 0.5, similarityBoost: 0.75, }), }); ``` Token streaming: ```ts // Stream LLM tokens instead of whole sentences. autoMode then defaults to false. const streaming = new ElevenLabsTTSService({ apiKey: process.env.ELEVENLABS_API_KEY ?? "", textAggregationMode: TextAggregationMode.TOKEN, settings: new ElevenLabsTTSService.Settings({ voice: "your-voice-id" }), }); ``` Change settings mid-call with a `TTSUpdateSettingsFrame`: ```ts // Queue it on the worker, or push it from a processor upstream of the service. // A voice-character change (stability, speed, ...) starts a new audio context; // a voice, model or language change reconnects. const update = new TTSUpdateSettingsFrame({ delta: new ElevenLabsTTSService.Settings({ speed: 1.1 }), }); ``` ### Event handlers [#event-handlers] Inherited from `TTSService`: | Event | Description | | --------------------- | ----------------------------------------------- | | `on_connected` | Connected to ElevenLabs. | | `on_disconnected` | Disconnected from ElevenLabs. | | `on_connection_error` | The connection failed. | | `on_tts_request` | A text chunk is about to be sent for synthesis. | ### Notes [#notes] * The service does not push the LLM's text frames itself. It builds `TTSTextFrame`s from ElevenLabs' word alignment, so the context only records words the user actually heard. * Text is aggregated into sentences by default. `autoMode` left `null` is `true` in sentence mode and `false` in token mode, where ElevenLabs' chunk scheduler needs to gather enough text. * Changing `voice`, `model` or `language` reconnects, since they are part of the WebSocket URL. Changing `stability`, `similarityBoost`, `style`, `useSpeakerBoost` or `speed` closes the current audio context, so the next one opens with the new voice settings. * `language` is checked against the model's language list. A model that takes no language code, or doesn't cover the language, gets none, and this is logged. * The output format is chosen from the sample rate: 8000, 16000, 22050, 24000, 32000, 44100 or 48000 Hz. Any other rate falls back to `pcm_24000` with a warning. * `pronunciationTransformIpa()` renders IPA as SSML `` tags. Only `eleven_flash_v2` and `eleven_turbo_v2` read them, and this service also needs `enableSsmlParsing: true`. Otherwise the transform is skipped. * Other settings changes (such as `applyTextNormalization`) are not applied mid-call and are logged as unhandled, unless a URL change reconnects at the same time. ## ElevenLabsHttpTTSService [#elevenlabshttpttsservice] ### Configuration [#configuration-1] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | ElevenLabs API key for authentication. | | `fetch` | `((input: RequestInfo \| URL, init?: RequestInit) => Promise) \| null \| undefined` | no | `fetch` used for HTTP requests. Defaults to the global `fetch`. DIVERGENCE: replaces upstream's required `aiohttp_session`. | | `baseUrl` | `string \| undefined` | no | Base URL for ElevenLabs HTTP API. Defaults to "https://api.elevenlabs.io". | | `enableLogging` | `boolean \| null \| undefined` | no | Whether to enable ElevenLabs server-side logging. Set to false for zero retention mode (enterprise only). | | `settings` | `ElevenLabsHttpTTSSettings \| null \| undefined` | no | Runtime-updatable settings. Its given fields override the defaults (model "eleven_flash_v2_5"). | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `textAggregationMode` | `TextAggregationMode \| null \| undefined` | no | How to aggregate incoming text before synthesis. `TextAggregationMode.SENTENCE` (default) buffers until sentence boundaries, `TextAggregationMode.TOKEN` streams tokens directly for lower latency. | | `pushTextFrames` | `boolean \| undefined` | no | Whether to push TextFrames and LLMFullResponseEndFrames. | | `pushStopFrames` | `boolean \| undefined` | no | Whether to automatically push TTSStoppedFrames. | | `pushStartFrame` | `boolean \| undefined` | no | Whether to automatically create audio contexts and push TTSStartedFrames. When true, the base class handles `createAudioContext` and yields `TTSStartedFrame` before each synthesis call, so `runTts` implementations do not need to. | | `stopFrameTimeoutS` | `number \| undefined` | no | Idle time before pushing TTSStoppedFrame when pushStopFrames is true. | | `pushSilenceAfterStop` | `boolean \| undefined` | no | Whether to push silence audio after TTSStoppedFrame. | | `silenceTimeS` | `number \| undefined` | no | Duration of silence to push when pushSilenceAfterStop is true. | | `pauseFrameProcessing` | `boolean \| undefined` | no | Whether to hold incoming frames after a turn until its audio contexts end. Needed by services whose audio doesn't carry its request's context ID. | | `maxConsecutiveZeroAudioContexts` | `number \| undefined` | no | How many consecutive TTS contexts may complete without producing any audio before the service is reported unable to do its job. Catches a provider that accepts requests and stays silent — an unknown voice ID, say — which no error ever surfaces. Every silent context reports an error the service can carry on from, whatever the limit is; on reaching the limit the service reports a permanent error instead, stops being given work, and the pipeline worker applies its `ProcessorUnusablePolicy`. Set to 0 to report silent contexts without ever writing the service off. | | `appendTrailingSpace` | `boolean \| undefined` | no | Whether to append a trailing space to text before sending to TTS. This helps prevent some TTS services from vocalizing trailing punctuation (e.g., "dot"). Only applied in sentence aggregation mode; when streaming tokens, the incoming text's own whitespace is preserved. | | `sampleRate` | `number \| null \| undefined` | no | Output sample rate for generated audio. | | `skipTextTypes` | `string[] \| null \| undefined` | no | List of text types that should not be spoken. | | `textTransforms` | `[string, TextTransform][] \| null \| undefined` | no | A list of TextTransforms to transform text just before sending it to TTS. Each TextTransform takes the aggregated text and its type, and returns the transformed text. To register, provide a list of tuples of `[textType \| "*", transformFunction]`. | | `transportDestination` | `string \| null \| undefined` | no | Destination for generated audio frames. | | `reuseContextIdWithinTurn` | `boolean \| undefined` | no | Whether the service should reuse context IDs within the same turn. | | `audioCache` | `TTSAudioCacheStore \| null \| undefined` | no | Where to keep synthesized audio for reuse (e.g. a shared `TTSAudioCache`). Text whose audio is in the cache is served from it, with no request to the provider. Lines get there through `warmAudioCache()`, and also from live synthesis when `cacheSynthesizedAudio` allows it. Only services that yield their audio from `runTts` (HTTP and streaming-RPC services, see `supportsAudioCache`) use it, and only in sentence aggregation mode. Defaults to null (no cache). pipedog only. | | `cacheSynthesizedAudio` | `boolean \| ((text: string) => boolean) \| undefined` | no | Whether audio synthesized live is stored in `audioCache`, or a function that decides per text. Defaults to false: only warmed lines are stored, since a generated reply can repeat what the user said. pipedog only. | | `audioCacheNamespace` | `string \| undefined` | no | Extra part of the cache key, for constructor options that change the audio but are not in `settings`. pipedog only. | ### Settings [#settings-1] | Setting | Type | Default | Description | | -------------------------- | --------------------------------- | --------------------- | --------------------------------------- | | `model` | `string` | `"eleven_flash_v2_5"` | ElevenLabs model id. | | `voice` | `string \| null` | `null` | Voice id. Set one: there is no default. | | `language` | `Language \| null` | `null` | Synthesis language. | | `optimizeStreamingLatency` | `number \| null` | `null` | Latency optimization level, 0 to 4. | | `stability` | `number \| null` | `null` | Voice stability, 0.0 to 1.0. | | `similarityBoost` | `number \| null` | `null` | Similarity boost, 0.0 to 1.0. | | `style` | `number \| null` | `null` | Style exaggeration, 0.0 to 1.0. | | `useSpeakerBoost` | `boolean \| null` | `null` | Speaker boost. | | `speed` | `number \| null` | `null` | Speaking speed, 0.25 to 4.0. | | `applyTextNormalization` | `"auto" \| "on" \| "off" \| null` | `null` | Text normalization mode. | ### Usage [#usage-1] ```ts title="http.ts" const httpTts = new ElevenLabsHttpTTSService({ apiKey: process.env.ELEVENLABS_API_KEY ?? "", settings: new ElevenLabsHttpTTSService.Settings({ voice: "your-voice-id", optimizeStreamingLatency: 2, speed: 1.1, }), }); ``` ### Notes [#notes-1] * Word timestamps come from the `with-timestamps` endpoint, so this service also emits `TTSTextFrame`s aligned with playback. * Earlier text in the turn is sent as `previous_text`, for continuity between requests. * `enableLogging: false` turns on ElevenLabs' zero retention mode (enterprise only). * IPA pronunciations (`pronunciationTransformIpa()`) work with `eleven_flash_v2` and `eleven_turbo_v2` only. ## ElevenLabsDialogueTTSService [#elevenlabsdialoguettsservice] ### Configuration [#configuration-2] | Option | Type | Required | Description | |---|---|---|---| | `url` | `string \| undefined` | no | WebSocket URL for the ElevenLabs API. Defaults to "wss://api.elevenlabs.io". | | `seed` | `number \| null \| undefined` | no | Seed for reproducible generation. | | `settings` | `ElevenLabsDialogueTTSSettings \| null \| undefined` | no | Runtime-updatable settings. Its given fields override the defaults (model "eleven_v4_turbo"). | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `reconnectBackoffMinWait` | `number \| undefined` | no | Minimum time, in seconds, to wait between reconnection attempts. | | `reconnectBackoffMaxWait` | `number \| undefined` | no | Maximum time, in seconds, to wait between reconnection attempts. | | `reconnectOnError` | `boolean \| undefined` | no | Whether to automatically reconnect on connection errors. | | `wsCloseTimeout` | `number \| undefined` | no | Maximum time, in seconds, to wait for the peer to acknowledge the websocket closing handshake before dropping the connection. Applied to connections opened through `_websocketConnect()`. Increase it for peers that need longer to complete a graceful close. | | `wsSendTimeout` | `number \| null \| undefined` | no | Maximum time, in seconds, a single send may wait on a peer that has stopped reading before the connection is dropped and the send throws `ConnectionClosedError`. Applied to connections opened through `_websocketConnect()`. Pass `null` to wait indefinitely. Defaults to `WS_SEND_TIMEOUT`. | | `textAggregationMode` | `TextAggregationMode \| null \| undefined` | no | How to aggregate incoming text before synthesis. `TextAggregationMode.SENTENCE` (default) buffers until sentence boundaries, `TextAggregationMode.TOKEN` streams tokens directly for lower latency. | | `pushTextFrames` | `boolean \| undefined` | no | Whether to push TextFrames and LLMFullResponseEndFrames. | | `pushStopFrames` | `boolean \| undefined` | no | Whether to automatically push TTSStoppedFrames. | | `pushStartFrame` | `boolean \| undefined` | no | Whether to automatically create audio contexts and push TTSStartedFrames. When true, the base class handles `createAudioContext` and yields `TTSStartedFrame` before each synthesis call, so `runTts` implementations do not need to. | | `stopFrameTimeoutS` | `number \| undefined` | no | Idle time before pushing TTSStoppedFrame when pushStopFrames is true. | | `pushSilenceAfterStop` | `boolean \| undefined` | no | Whether to push silence audio after TTSStoppedFrame. | | `silenceTimeS` | `number \| undefined` | no | Duration of silence to push when pushSilenceAfterStop is true. | | `pauseFrameProcessing` | `boolean \| undefined` | no | Whether to hold incoming frames after a turn until its audio contexts end. Needed by services whose audio doesn't carry its request's context ID. | | `maxConsecutiveZeroAudioContexts` | `number \| undefined` | no | How many consecutive TTS contexts may complete without producing any audio before the service is reported unable to do its job. Catches a provider that accepts requests and stays silent — an unknown voice ID, say — which no error ever surfaces. Every silent context reports an error the service can carry on from, whatever the limit is; on reaching the limit the service reports a permanent error instead, stops being given work, and the pipeline worker applies its `ProcessorUnusablePolicy`. Set to 0 to report silent contexts without ever writing the service off. | | `appendTrailingSpace` | `boolean \| undefined` | no | Whether to append a trailing space to text before sending to TTS. This helps prevent some TTS services from vocalizing trailing punctuation (e.g., "dot"). Only applied in sentence aggregation mode; when streaming tokens, the incoming text's own whitespace is preserved. | | `sampleRate` | `number \| null \| undefined` | no | Output sample rate for generated audio. | | `skipTextTypes` | `string[] \| null \| undefined` | no | List of text types that should not be spoken. | | `textTransforms` | `[string, TextTransform][] \| null \| undefined` | no | A list of TextTransforms to transform text just before sending it to TTS. Each TextTransform takes the aggregated text and its type, and returns the transformed text. To register, provide a list of tuples of `[textType \| "*", transformFunction]`. | | `transportDestination` | `string \| null \| undefined` | no | Destination for generated audio frames. | | `reuseContextIdWithinTurn` | `boolean \| undefined` | no | Whether the service should reuse context IDs within the same turn. | | `audioCache` | `TTSAudioCacheStore \| null \| undefined` | no | Where to keep synthesized audio for reuse (e.g. a shared `TTSAudioCache`). Text whose audio is in the cache is served from it, with no request to the provider. Lines get there through `warmAudioCache()`, and also from live synthesis when `cacheSynthesizedAudio` allows it. Only services that yield their audio from `runTts` (HTTP and streaming-RPC services, see `supportsAudioCache`) use it, and only in sentence aggregation mode. Defaults to null (no cache). pipedog only. | | `cacheSynthesizedAudio` | `boolean \| ((text: string) => boolean) \| undefined` | no | Whether audio synthesized live is stored in `audioCache`, or a function that decides per text. Defaults to false: only warmed lines are stored, since a generated reply can repeat what the user said. pipedog only. | | `audioCacheNamespace` | `string \| undefined` | no | Extra part of the cache key, for constructor options that change the audio but are not in `settings`. pipedog only. | | `apiKey` | `string` | yes | ElevenLabs API key for authentication. | | `enableLogging` | `boolean \| null \| undefined` | no | Whether to enable ElevenLabs server-side logging. | ### Settings [#settings-2] Runtime-updatable settings, passed as `settings: new ElevenLabsDialogueTTSService.Settings({...})`. | Setting | Type | Default | Description | | ----------- | ------------------ | ------------------- | --------------------------------------------------------------------------- | | `model` | `string` | `"eleven_v4_turbo"` | An `eleven_v3` or `eleven_v4` model. Any other model logs a warning. | | `voice` | `string \| null` | `null` | Voice id. Registered per context, so a change only opens a new context. | | `language` | `Language \| null` | `null` | Synthesis language, sent as `language_code` where the model supports it. | | `stability` | `number \| null` | `null` | Voice stability, 0.0 to 1.0. The only voice setting Text to Dialogue reads. | A `model` or `language` change reconnects the WebSocket. A `voice` or `stability` change closes the current audio context, and the next one registers the new values. ### Usage [#usage-2] ```ts title="dialogue.ts" import { ElevenLabsDialogueTTSService } from "@pipedog/elevenlabs/dialogue/tts"; // Eleven v3 and v4 dialogue models. Needs a voice id. const dialogue = new ElevenLabsDialogueTTSService({ apiKey: process.env.ELEVENLABS_API_KEY ?? "", settings: new ElevenLabsDialogueTTSService.Settings({ voice: "your-voice-id", model: "eleven_v4_turbo", stability: 0.5, }), }); ``` ### Notes [#notes-2] * Text is always aggregated by sentence, and a trailing space is added to each input, because the server joins inputs verbatim. A `textAggregationMode` other than `SENTENCE` is ignored with a warning. * Eleven v3 and v4 perform inline audio tags such as `[laughs]`. They come back as spoken characters in the alignment, so they reach the LLM context unless a text transform removes them. * A keepalive context is registered when the socket opens, because the server drops a connection that is idle for 20 seconds and rejects a keepalive that names no registered context. Set a `voice` so it can be registered. * `pronunciationTransformIpa()` renders pronunciations as IPA between slashes, which Eleven v3 reads. * `seed` makes generation reproducible. ## Runtime notes [#runtime-notes] * `ElevenLabsTTSService` uses the global `WebSocket` and sends the key as an `xi-api-key` header on the handshake. That works on Bun, Node and Deno, and on Cloudflare Workers, where the socket is opened through `fetch`. Upstream sets a 16 MB message size limit; the standard `WebSocket` has none. * `ElevenLabsDialogueTTSService` uses the same WebSocket and `xi-api-key` header as `ElevenLabsTTSService`, so it runs wherever that does. * `ElevenLabsHttpTTSService` uses `fetch`. Upstream's required `aiohttp_session` is replaced by an optional `fetch` option (the global `fetch` by default). * Not ported: the deprecated `PronunciationDictionaryLocator`, the `InputParams` classes and the `voiceId`, `model`, `params`, `pronunciationDictionaryLocators` and `aggregateSentences` constructor options. Use `settings` and `textAggregationMode`. | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/elevenlabs/package.json`: ✅ means the compat smoke test passes on that runtime. # Fish Audio > Text-to-speech with FishAudioTTSService, Fish Audio's MessagePack WebSocket API, with cloned-voice support. Source: https://polvallverdu.github.io/pipedog/docs/services/tts/fish/ ## Overview [#overview] Fish Audio turns text into speech, including with cloned voices. pipedog ports Pipecat's **`FishAudioTTSService`**: it streams over Fish Audio's real-time WebSocket (`wss://api.fish.audio/v1/tts/live`), speaking MessagePack, and reconnects when the user interrupts the bot. Pipecat's documentation for the same service. `packages/fish/src/tts.ts`. The real-time WebSocket protocol, models and voices. ## Installation [#installation] ```bash bun add @pipedog/fish ``` ```ts import { FishAudioTTSService } from "@pipedog/fish/tts"; ``` ## Prerequisites [#prerequisites] * A Fish Audio account and an API key. * A voice model's reference id, set as `voice`. It is sent as `reference_id`. * Pass the key as `apiKey`. The examples read it from `FISH_API_KEY`. ## FishAudioTTSService [#fishaudiottsservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `pushTextFrames` | `boolean \| undefined` | no | Whether to push TextFrames and LLMFullResponseEndFrames. | | `pushStopFrames` | `boolean \| undefined` | no | Whether to automatically push TTSStoppedFrames. | | `apiKey` | `string` | yes | Fish Audio API key for authentication. | | `outputFormat` | `FishAudioOutputFormat \| undefined` | no | Audio output format. Defaults to "pcm". | | `settings` | `FishAudioTTSSettings \| null \| undefined` | no | Runtime-updatable settings (defaults: model "s2.1-pro"). | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `reconnectBackoffMinWait` | `number \| undefined` | no | Minimum time, in seconds, to wait between reconnection attempts. | | `reconnectBackoffMaxWait` | `number \| undefined` | no | Maximum time, in seconds, to wait between reconnection attempts. | | `reconnectOnError` | `boolean \| undefined` | no | Whether to automatically reconnect on connection errors. | | `wsCloseTimeout` | `number \| undefined` | no | Maximum time, in seconds, to wait for the peer to acknowledge the websocket closing handshake before dropping the connection. Applied to connections opened through `_websocketConnect()`. Increase it for peers that need longer to complete a graceful close. | | `wsSendTimeout` | `number \| null \| undefined` | no | Maximum time, in seconds, a single send may wait on a peer that has stopped reading before the connection is dropped and the send throws `ConnectionClosedError`. Applied to connections opened through `_websocketConnect()`. Pass `null` to wait indefinitely. Defaults to `WS_SEND_TIMEOUT`. | | `textAggregationMode` | `TextAggregationMode \| null \| undefined` | no | How to aggregate incoming text before synthesis. `TextAggregationMode.SENTENCE` (default) buffers until sentence boundaries, `TextAggregationMode.TOKEN` streams tokens directly for lower latency. | | `pushStartFrame` | `boolean \| undefined` | no | Whether to automatically create audio contexts and push TTSStartedFrames. When true, the base class handles `createAudioContext` and yields `TTSStartedFrame` before each synthesis call, so `runTts` implementations do not need to. | | `stopFrameTimeoutS` | `number \| undefined` | no | Idle time before pushing TTSStoppedFrame when pushStopFrames is true. | | `pushSilenceAfterStop` | `boolean \| undefined` | no | Whether to push silence audio after TTSStoppedFrame. | | `silenceTimeS` | `number \| undefined` | no | Duration of silence to push when pushSilenceAfterStop is true. | | `pauseFrameProcessing` | `boolean \| undefined` | no | Whether to hold incoming frames after a turn until its audio contexts end. Needed by services whose audio doesn't carry its request's context ID. | | `maxConsecutiveZeroAudioContexts` | `number \| undefined` | no | How many consecutive TTS contexts may complete without producing any audio before the service is reported unable to do its job. Catches a provider that accepts requests and stays silent — an unknown voice ID, say — which no error ever surfaces. Every silent context reports an error the service can carry on from, whatever the limit is; on reaching the limit the service reports a permanent error instead, stops being given work, and the pipeline worker applies its `ProcessorUnusablePolicy`. Set to 0 to report silent contexts without ever writing the service off. | | `appendTrailingSpace` | `boolean \| undefined` | no | Whether to append a trailing space to text before sending to TTS. This helps prevent some TTS services from vocalizing trailing punctuation (e.g., "dot"). Only applied in sentence aggregation mode; when streaming tokens, the incoming text's own whitespace is preserved. | | `sampleRate` | `number \| null \| undefined` | no | Output sample rate for generated audio. | | `skipTextTypes` | `string[] \| null \| undefined` | no | List of text types that should not be spoken. | | `textTransforms` | `[string, TextTransform][] \| null \| undefined` | no | A list of TextTransforms to transform text just before sending it to TTS. Each TextTransform takes the aggregated text and its type, and returns the transformed text. To register, provide a list of tuples of `[textType \| "*", transformFunction]`. | | `transportDestination` | `string \| null \| undefined` | no | Destination for generated audio frames. | | `reuseContextIdWithinTurn` | `boolean \| undefined` | no | Whether the service should reuse context IDs within the same turn. | | `audioCache` | `TTSAudioCacheStore \| null \| undefined` | no | Where to keep synthesized audio for reuse (e.g. a shared `TTSAudioCache`). Text whose audio is in the cache is served from it, with no request to the provider. Lines get there through `warmAudioCache()`, and also from live synthesis when `cacheSynthesizedAudio` allows it. Only services that yield their audio from `runTts` (HTTP and streaming-RPC services, see `supportsAudioCache`) use it, and only in sentence aggregation mode. Defaults to null (no cache). pipedog only. | | `cacheSynthesizedAudio` | `boolean \| ((text: string) => boolean) \| undefined` | no | Whether audio synthesized live is stored in `audioCache`, or a function that decides per text. Defaults to false: only warmed lines are stored, since a generated reply can repeat what the user said. pipedog only. | | `audioCacheNamespace` | `string \| undefined` | no | Extra part of the cache key, for constructor options that change the audio but are not in `settings`. pipedog only. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new FishAudioTTSService.Settings({...})`. | Setting | Type | Default | Description | | --------------------------- | ----------------- | ------------ | ------------------------------------------------------------------------------------------------ | | `model` | `string` | `"s2.1-pro"` | Fish Audio model, sent as the `model` handshake header: `s2.1-pro`, `s2-pro`, `s1` or `drama-3`. | | `voice` | `string \| null` | `null` | Voice model reference id. | | `latency` | `string \| null` | `"balanced"` | `"normal"` or `"balanced"`. | | `normalize` | `boolean \| null` | `true` | Normalize the text. | | `temperature` | `number \| null` | `null` | Randomness, 0.0 to 1.0. | | `topP` | `number \| null` | `null` | Nucleus sampling, 0.0 to 1.0. | | `prosodySpeed` | `number \| null` | `1.0` | Speech speed, 0.5 to 2.0. | | `prosodyVolume` | `number \| null` | `0` | Volume in dB, -20 to 20. | | `prosodyNormalizeLoudness` | `boolean \| null` | `null` | Even out the perceived volume (S2 family and drama-3). | | `chunkLength` | `number \| null` | `null` | Text segment size, 100 to 300. | | `minChunkLength` | `number \| null` | `null` | Minimum characters before a new chunk, 0 to 100. | | `conditionOnPreviousChunks` | `boolean \| null` | `null` | Use earlier audio as context, for a consistent voice. | Settings left `null` are not sent, so Fish Audio applies its own defaults. `Settings.fromMapping()` also accepts upstream's nested `prosody` dict. ### Usage [#usage] ```ts title="tts.ts" import { TTSUpdateSettingsFrame } from "@pipedog/core/frames/frames"; import { FishAudioTTSService } from "@pipedog/fish/tts"; const tts = new FishAudioTTSService({ apiKey: process.env.FISH_API_KEY ?? "", settings: new FishAudioTTSService.Settings({ // A voice model's reference id from the Fish Audio library. voice: "your-reference-id", model: "s2.1-pro", latency: "balanced", prosodySpeed: 1.1, }), }); ``` Change settings mid-call with a `TTSUpdateSettingsFrame`: ```ts // Queue it on the worker, or push it from a processor upstream of the service. // A change to any setting reconnects, because they are all sent in the start message. const update = new TTSUpdateSettingsFrame({ delta: new FishAudioTTSService.Settings({ prosodySpeed: 0.9, prosodyVolume: 2 }), }); ``` ### Event handlers [#event-handlers] Inherited from `TTSService`: | Event | Description | | --------------------- | ----------------------------------------------- | | `on_connected` | Connected to Fish Audio. | | `on_disconnected` | Disconnected from Fish Audio. | | `on_connection_error` | The connection failed. | | `on_tts_request` | A text chunk is about to be sent for synthesis. | ### Notes [#notes] * Each text chunk is sent with a `flush`, which makes Fish Audio generate the audio right away. * The service reconnects on an interruption while the bot is speaking, and on any settings change. * `outputFormat` is `"pcm"` by default. The service emits the audio as it arrives and does not produce word timestamps. ## Runtime notes [#runtime-notes] * The socket is the global `WebSocket`. The `Authorization` and `model` headers go on the handshake, which works on Bun, Node and Deno, and on Cloudflare Workers, where the socket is opened through `fetch`. * Upstream packs messages with `ormsgpack`. pipedog uses [`@msgpack/msgpack`](https://github.com/msgpack/msgpack-javascript), which is pure TypeScript. It writes a whole-number float such as `prosodySpeed: 1.0` as a MessagePack integer. * Not ported: the deprecated `InputParams` and the `referenceId`, `modelId` and `params` constructor options. Use `settings`. | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/fish/package.json`: ✅ means the compat smoke test passes on that runtime. # Google > Text-to-speech with GoogleTTSService (Cloud TTS streaming), GoogleHttpTTSService (Cloud TTS with SSML) and GeminiTTSService (Gemini TTS models). Source: https://polvallverdu.github.io/pipedog/docs/services/tts/google/ ## Overview [#overview] pipedog ports Pipecat's three Google text-to-speech services: * **`GoogleTTSService`**: Google Cloud Text-to-Speech streaming synthesis, for low latency. Supports Chirp 3 HD and Journey voices, and Chirp 3 custom voices through a voice cloning key. * **`GoogleHttpTTSService`**: Cloud Text-to-Speech unary synthesis. Works with every Google voice (Standard, WaveNet, Neural2, ...) and builds SSML for pitch, rate, volume, emphasis, gender and style on voices that accept it. * **`GeminiTTSService`**: Gemini TTS models, with style prompts, expressive tags and multi-speaker output. Runs against Cloud Text-to-Speech (service account) or the Gemini API (API key). Pipecat's documentation for the same services. `packages/google/src/tts.ts`. Voices, SSML and Gemini TTS on Google Cloud. ## Installation [#installation] ```bash bun add @pipedog/google ``` The Cloud Text-to-Speech classes (and the Cloud backend of `GeminiTTSService`) also need the Node SDK, an optional peer dependency: ```bash bun add @google-cloud/text-to-speech ``` ```ts import { GeminiTTSService, GoogleHttpTTSService, GoogleTTSService } from "@pipedog/google/tts"; ``` ## Prerequisites [#prerequisites] * **Cloud Text-to-Speech**: a Google Cloud project with the Text-to-Speech API enabled, and a service account key (file or JSON string) or Application Default Credentials (ADC). The examples read `GOOGLE_APPLICATION_CREDENTIALS` (a key file path) or `GOOGLE_CREDENTIALS_JSON` (the key itself). * **Gemini API backend of `GeminiTTSService`**: a Gemini API key from [Google AI Studio](https://aistudio.google.com/apikey), read from `GOOGLE_API_KEY`. ## GoogleTTSService [#googlettsservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `credentials` | `string \| null \| undefined` | no | JSON string containing Google Cloud service account credentials. | | `credentialsPath` | `string \| null \| undefined` | no | Path to Google Cloud service account JSON file. | | `location` | `string \| null \| undefined` | no | Google Cloud location for regional endpoint (e.g., "us-central1"). | | `voiceCloningKey` | `string \| null \| undefined` | no | The voice cloning key for Chirp 3 custom voices. | | `clientOptions` | `Record \| null \| undefined` | no | Extra options for the `@google-cloud/text-to-speech` `TextToSpeechClient` (e.g. `apiEndpoint`, `port`, `sslCreds`), merged over the ones derived from `location` and the credentials. DIVERGENCE: pipedog-only. Upstream builds its client from `location` and the credentials alone. | | `textToSpeechModule` | `GoogleTextToSpeechModule \| null \| undefined` | no | The `@google-cloud/text-to-speech` module, for a bundled app: see `GoogleCloudTTSClientOptions.module`. | | `settings` | `GoogleTTSSettings \| null \| undefined` | no | Runtime-updatable settings. | | `client` | `GoogleCloudTTSClient \| null \| undefined` | no | A Cloud Text-to-Speech client to use instead of building one from the credentials, `location` and `clientOptions` (which are then ignored). Share one client (e.g. `createGoogleCloudTTSClient(...)`) across services to skip the per-service credential lookup, TLS handshake and token fetch on the first sentence. DIVERGENCE: pipedog-only. | | `sharedClient` | `boolean \| undefined` | no | Use the process-wide client for this endpoint, credentials and `clientOptions` (`getSharedGoogleCloudTTSClient`) instead of building one per service, so a server running one pipeline per call opens one gRPC channel and fetches tokens once. Pair it with `prewarm()` at start-up. Default false: each service has its own client (upstream's behavior), since one channel carries every concurrent stream of the process. Ignored when `client` is given. DIVERGENCE: pipedog-only. | | `bufferOnlyStartup` | `boolean \| undefined` | no | Buffer only the start of each utterance: once `chunkSize` bytes have arrived, audio is passed on as it is received instead of being re-cut into `chunkSize` pieces (which holds back up to one chunk throughout). Default false (upstream's behavior). DIVERGENCE: pipedog-only. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `textAggregationMode` | `TextAggregationMode \| null \| undefined` | no | How to aggregate incoming text before synthesis. `TextAggregationMode.SENTENCE` (default) buffers until sentence boundaries, `TextAggregationMode.TOKEN` streams tokens directly for lower latency. | | `pushTextFrames` | `boolean \| undefined` | no | Whether to push TextFrames and LLMFullResponseEndFrames. | | `pushStopFrames` | `boolean \| undefined` | no | Whether to automatically push TTSStoppedFrames. | | `pushStartFrame` | `boolean \| undefined` | no | Whether to automatically create audio contexts and push TTSStartedFrames. When true, the base class handles `createAudioContext` and yields `TTSStartedFrame` before each synthesis call, so `runTts` implementations do not need to. | | `stopFrameTimeoutS` | `number \| undefined` | no | Idle time before pushing TTSStoppedFrame when pushStopFrames is true. | | `pushSilenceAfterStop` | `boolean \| undefined` | no | Whether to push silence audio after TTSStoppedFrame. | | `silenceTimeS` | `number \| undefined` | no | Duration of silence to push when pushSilenceAfterStop is true. | | `pauseFrameProcessing` | `boolean \| undefined` | no | Whether to hold incoming frames after a turn until its audio contexts end. Needed by services whose audio doesn't carry its request's context ID. | | `maxConsecutiveZeroAudioContexts` | `number \| undefined` | no | How many consecutive TTS contexts may complete without producing any audio before the service is reported unable to do its job. Catches a provider that accepts requests and stays silent — an unknown voice ID, say — which no error ever surfaces. Every silent context reports an error the service can carry on from, whatever the limit is; on reaching the limit the service reports a permanent error instead, stops being given work, and the pipeline worker applies its `ProcessorUnusablePolicy`. Set to 0 to report silent contexts without ever writing the service off. | | `appendTrailingSpace` | `boolean \| undefined` | no | Whether to append a trailing space to text before sending to TTS. This helps prevent some TTS services from vocalizing trailing punctuation (e.g., "dot"). Only applied in sentence aggregation mode; when streaming tokens, the incoming text's own whitespace is preserved. | | `sampleRate` | `number \| null \| undefined` | no | Output sample rate for generated audio. | | `skipTextTypes` | `string[] \| null \| undefined` | no | List of text types that should not be spoken. | | `textTransforms` | `[string, TextTransform][] \| null \| undefined` | no | A list of TextTransforms to transform text just before sending it to TTS. Each TextTransform takes the aggregated text and its type, and returns the transformed text. To register, provide a list of tuples of `[textType \| "*", transformFunction]`. | | `transportDestination` | `string \| null \| undefined` | no | Destination for generated audio frames. | | `reuseContextIdWithinTurn` | `boolean \| undefined` | no | Whether the service should reuse context IDs within the same turn. | | `audioCache` | `TTSAudioCacheStore \| null \| undefined` | no | Where to keep synthesized audio for reuse (e.g. a shared `TTSAudioCache`). Text whose audio is in the cache is served from it, with no request to the provider. Lines get there through `warmAudioCache()`, and also from live synthesis when `cacheSynthesizedAudio` allows it. Only services that yield their audio from `runTts` (HTTP and streaming-RPC services, see `supportsAudioCache`) use it, and only in sentence aggregation mode. Defaults to null (no cache). pipedog only. | | `cacheSynthesizedAudio` | `boolean \| ((text: string) => boolean) \| undefined` | no | Whether audio synthesized live is stored in `audioCache`, or a function that decides per text. Defaults to false: only warmed lines are stored, since a generated reply can repeat what the user said. pipedog only. | | `audioCacheNamespace` | `string \| undefined` | no | Extra part of the cache key, for constructor options that change the audio but are not in `settings`. pipedog only. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new GoogleTTSService.Settings({...})`. | Setting | Type | Default | Description | | -------------- | -------------------- | -------------------------- | ---------------------------------------------------------------------------- | | `model` | `string \| null` | `null` | Not used by the streaming request. | | `voice` | `string` | `"en-US-Chirp3-HD-Charon"` | Chirp 3 HD or Journey voice name. | | `language` | `Language \| string` | `"en-US"` | Voice language. | | `speakingRate` | `number \| null` | `null` | Speaking rate, 0.25 to 2.0. Out-of-range updates are ignored with a warning. | ### Usage [#usage] ```ts title="tts.ts" import { TTSUpdateSettingsFrame } from "@pipedog/core/frames/frames"; import { Language } from "@pipedog/core/transcriptions/language"; import { GoogleHttpTTSService, GoogleTTSService } from "@pipedog/google/tts"; const tts = new GoogleTTSService({ credentialsPath: process.env.GOOGLE_APPLICATION_CREDENTIALS ?? "", settings: new GoogleTTSService.Settings({ voice: "en-US-Chirp3-HD-Charon", language: Language.EN_US, speakingRate: 1.1, }), }); ``` Change settings mid-call with a `TTSUpdateSettingsFrame`: ```ts const update = new TTSUpdateSettingsFrame({ delta: new GoogleTTSService.Settings({ voice: "en-US-Chirp3-HD-Aoede" }), }); ``` ### Notes [#notes] * With `voiceCloningKey`, the request uses that Chirp 3 custom voice instead of `voice`. * Audio is requested as PCM at the pipeline's output sample rate. ## GoogleHttpTTSService [#googlehttpttsservice] ### Configuration [#configuration-1] | Option | Type | Required | Description | |---|---|---|---| | `credentials` | `string \| null \| undefined` | no | JSON string containing Google Cloud service account credentials. | | `credentialsPath` | `string \| null \| undefined` | no | Path to Google Cloud service account JSON file. | | `location` | `string \| null \| undefined` | no | Google Cloud location for regional endpoint (e.g., "us-central1"). | | `clientOptions` | `Record \| null \| undefined` | no | Extra options for the `@google-cloud/text-to-speech` `TextToSpeechClient` (e.g. `apiEndpoint`, `port`, `sslCreds`), merged over the ones derived from `location` and the credentials. DIVERGENCE: pipedog-only. Upstream builds its client from `location` and the credentials alone. | | `textToSpeechModule` | `GoogleTextToSpeechModule \| null \| undefined` | no | The `@google-cloud/text-to-speech` module, for a bundled app: see `GoogleCloudTTSClientOptions.module`. | | `settings` | `GoogleHttpTTSSettings \| null \| undefined` | no | Runtime-updatable settings. | | `sharedClient` | `boolean \| undefined` | no | See `GoogleTTSServiceOptions.sharedClient`. DIVERGENCE: pipedog-only. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `textAggregationMode` | `TextAggregationMode \| null \| undefined` | no | How to aggregate incoming text before synthesis. `TextAggregationMode.SENTENCE` (default) buffers until sentence boundaries, `TextAggregationMode.TOKEN` streams tokens directly for lower latency. | | `pushTextFrames` | `boolean \| undefined` | no | Whether to push TextFrames and LLMFullResponseEndFrames. | | `pushStopFrames` | `boolean \| undefined` | no | Whether to automatically push TTSStoppedFrames. | | `pushStartFrame` | `boolean \| undefined` | no | Whether to automatically create audio contexts and push TTSStartedFrames. When true, the base class handles `createAudioContext` and yields `TTSStartedFrame` before each synthesis call, so `runTts` implementations do not need to. | | `stopFrameTimeoutS` | `number \| undefined` | no | Idle time before pushing TTSStoppedFrame when pushStopFrames is true. | | `pushSilenceAfterStop` | `boolean \| undefined` | no | Whether to push silence audio after TTSStoppedFrame. | | `silenceTimeS` | `number \| undefined` | no | Duration of silence to push when pushSilenceAfterStop is true. | | `pauseFrameProcessing` | `boolean \| undefined` | no | Whether to hold incoming frames after a turn until its audio contexts end. Needed by services whose audio doesn't carry its request's context ID. | | `maxConsecutiveZeroAudioContexts` | `number \| undefined` | no | How many consecutive TTS contexts may complete without producing any audio before the service is reported unable to do its job. Catches a provider that accepts requests and stays silent — an unknown voice ID, say — which no error ever surfaces. Every silent context reports an error the service can carry on from, whatever the limit is; on reaching the limit the service reports a permanent error instead, stops being given work, and the pipeline worker applies its `ProcessorUnusablePolicy`. Set to 0 to report silent contexts without ever writing the service off. | | `appendTrailingSpace` | `boolean \| undefined` | no | Whether to append a trailing space to text before sending to TTS. This helps prevent some TTS services from vocalizing trailing punctuation (e.g., "dot"). Only applied in sentence aggregation mode; when streaming tokens, the incoming text's own whitespace is preserved. | | `sampleRate` | `number \| null \| undefined` | no | Output sample rate for generated audio. | | `skipTextTypes` | `string[] \| null \| undefined` | no | List of text types that should not be spoken. | | `textTransforms` | `[string, TextTransform][] \| null \| undefined` | no | A list of TextTransforms to transform text just before sending it to TTS. Each TextTransform takes the aggregated text and its type, and returns the transformed text. To register, provide a list of tuples of `[textType \| "*", transformFunction]`. | | `transportDestination` | `string \| null \| undefined` | no | Destination for generated audio frames. | | `reuseContextIdWithinTurn` | `boolean \| undefined` | no | Whether the service should reuse context IDs within the same turn. | | `audioCache` | `TTSAudioCacheStore \| null \| undefined` | no | Where to keep synthesized audio for reuse (e.g. a shared `TTSAudioCache`). Text whose audio is in the cache is served from it, with no request to the provider. Lines get there through `warmAudioCache()`, and also from live synthesis when `cacheSynthesizedAudio` allows it. Only services that yield their audio from `runTts` (HTTP and streaming-RPC services, see `supportsAudioCache`) use it, and only in sentence aggregation mode. Defaults to null (no cache). pipedog only. | | `cacheSynthesizedAudio` | `boolean \| ((text: string) => boolean) \| undefined` | no | Whether audio synthesized live is stored in `audioCache`, or a function that decides per text. Defaults to false: only warmed lines are stored, since a generated reply can repeat what the user said. pipedog only. | | `audioCacheNamespace` | `string \| undefined` | no | Extra part of the cache key, for constructor options that change the audio but are not in `settings`. pipedog only. | ### Settings [#settings-1] | Setting | Type | Default | Description | | -------------- | ---------------------------------------------------------------------- | -------------------------- | -------------------------------------------------------------- | | `model` | `string \| null` | `null` | Not used by the request. | | `voice` | `string` | `"en-US-Chirp3-HD-Charon"` | Any Google voice name. | | `language` | `Language \| string` | `"en-US"` | Voice language. | | `pitch` | `string \| null` | `null` | SSML pitch, e.g. `"+2st"`, `"-50%"`. | | `rate` | `string \| null` | `null` | SSML prosody rate, e.g. `"slow"`, `"125%"` (non-Chirp voices). | | `speakingRate` | `number \| null` | `null` | Audio config rate, 0.25 to 2.0 (Chirp and Journey voices). | | `volume` | `string \| null` | `null` | SSML volume, e.g. `"loud"`, `"+6dB"`. | | `emphasis` | `"strong" \| "moderate" \| "reduced" \| "none" \| null` | `null` | SSML emphasis. | | `gender` | `"male" \| "female" \| "neutral" \| null` | `null` | Voice gender. | | `googleStyle` | `"apologetic" \| "calm" \| "empathetic" \| "firm" \| "lively" \| null` | `null` | Google voice style. | ### Usage [#usage-1] ```ts title="http.ts" // Any Google voice. Voices other than Chirp and Journey get SSML prosody. const httpTts = new GoogleHttpTTSService({ credentials: process.env.GOOGLE_CREDENTIALS_JSON ?? "", settings: new GoogleHttpTTSService.Settings({ voice: "en-US-Neural2-F", language: Language.EN_US, pitch: "+2st", rate: "110%", volume: "+2dB", }), }); ``` ### Notes [#notes-1] * Chirp and Journey voices (matched by name) don't accept SSML, so the service sends plain text and the SSML settings have no effect, including on the default voice. Use `speakingRate` for them. Other voices get SSML, with the text XML-escaped. ## GeminiTTSService [#geminittsservice] ### Configuration [#configuration-2] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string \| null \| undefined` | no | Google AI API key for authentication when using GenAI. | | `credentials` | `string \| null \| undefined` | no | JSON string containing Google Cloud service account credentials. | | `credentialsPath` | `string \| null \| undefined` | no | Path to Google Cloud service account JSON file. | | `location` | `string \| null \| undefined` | no | Google Cloud location for regional endpoint (e.g., "us-central1"). | | `clientOptions` | `Record \| null \| undefined` | no | Extra options for the `@google-cloud/text-to-speech` `TextToSpeechClient` (e.g. `apiEndpoint`, `port`, `sslCreds`), merged over the ones derived from `location` and the credentials. DIVERGENCE: pipedog-only. Upstream builds its client from `location` and the credentials alone. | | `textToSpeechModule` | `GoogleTextToSpeechModule \| null \| undefined` | no | The `@google-cloud/text-to-speech` module, for a bundled app: see `GoogleCloudTTSClientOptions.module`. | | `settings` | `GeminiTTSSettings \| null \| undefined` | no | Runtime-updatable settings. | | `useGenai` | `boolean \| null \| undefined` | no | Force use of the google-genai client if true. | | `httpOptions` | `HttpOptions \| null \| undefined` | no | HTTP client options for the google-genai client. | | `client` | `GoogleCloudTTSClient \| null \| undefined` | no | A Cloud Text-to-Speech client for the GCP backend, instead of building one from the credentials, `location` and `clientOptions`. See `GoogleTTSServiceOptions.client`. Ignored by the GenAI backend. DIVERGENCE: pipedog-only. | | `sharedClient` | `boolean \| undefined` | no | See `GoogleTTSServiceOptions.sharedClient` (GCP backend; the GenAI backend talks HTTP and ignores it). DIVERGENCE: pipedog-only. | | `bufferOnlyStartup` | `boolean \| undefined` | no | See `GoogleTTSServiceOptions.bufferOnlyStartup`. DIVERGENCE: pipedog-only. | | `retryOnBlocked` | `boolean \| GeminiTTSRetryOnBlockedOptions \| null \| undefined` | no | Retry a sentence that Google blocks for its usage guidelines instead of failing it: `true` for one retry, or `GeminiTTSRetryOnBlockedOptions`. Default off: a blocked sentence yields an `ErrorFrame` (upstream's behavior). DIVERGENCE: pipedog-only. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `textAggregationMode` | `TextAggregationMode \| null \| undefined` | no | How to aggregate incoming text before synthesis. `TextAggregationMode.SENTENCE` (default) buffers until sentence boundaries, `TextAggregationMode.TOKEN` streams tokens directly for lower latency. | | `pushTextFrames` | `boolean \| undefined` | no | Whether to push TextFrames and LLMFullResponseEndFrames. | | `pushStopFrames` | `boolean \| undefined` | no | Whether to automatically push TTSStoppedFrames. | | `pushStartFrame` | `boolean \| undefined` | no | Whether to automatically create audio contexts and push TTSStartedFrames. When true, the base class handles `createAudioContext` and yields `TTSStartedFrame` before each synthesis call, so `runTts` implementations do not need to. | | `stopFrameTimeoutS` | `number \| undefined` | no | Idle time before pushing TTSStoppedFrame when pushStopFrames is true. | | `pushSilenceAfterStop` | `boolean \| undefined` | no | Whether to push silence audio after TTSStoppedFrame. | | `silenceTimeS` | `number \| undefined` | no | Duration of silence to push when pushSilenceAfterStop is true. | | `pauseFrameProcessing` | `boolean \| undefined` | no | Whether to hold incoming frames after a turn until its audio contexts end. Needed by services whose audio doesn't carry its request's context ID. | | `maxConsecutiveZeroAudioContexts` | `number \| undefined` | no | How many consecutive TTS contexts may complete without producing any audio before the service is reported unable to do its job. Catches a provider that accepts requests and stays silent — an unknown voice ID, say — which no error ever surfaces. Every silent context reports an error the service can carry on from, whatever the limit is; on reaching the limit the service reports a permanent error instead, stops being given work, and the pipeline worker applies its `ProcessorUnusablePolicy`. Set to 0 to report silent contexts without ever writing the service off. | | `appendTrailingSpace` | `boolean \| undefined` | no | Whether to append a trailing space to text before sending to TTS. This helps prevent some TTS services from vocalizing trailing punctuation (e.g., "dot"). Only applied in sentence aggregation mode; when streaming tokens, the incoming text's own whitespace is preserved. | | `sampleRate` | `number \| null \| undefined` | no | Output sample rate for generated audio. | | `skipTextTypes` | `string[] \| null \| undefined` | no | List of text types that should not be spoken. | | `textTransforms` | `[string, TextTransform][] \| null \| undefined` | no | A list of TextTransforms to transform text just before sending it to TTS. Each TextTransform takes the aggregated text and its type, and returns the transformed text. To register, provide a list of tuples of `[textType \| "*", transformFunction]`. | | `transportDestination` | `string \| null \| undefined` | no | Destination for generated audio frames. | | `reuseContextIdWithinTurn` | `boolean \| undefined` | no | Whether the service should reuse context IDs within the same turn. | | `audioCache` | `TTSAudioCacheStore \| null \| undefined` | no | Where to keep synthesized audio for reuse (e.g. a shared `TTSAudioCache`). Text whose audio is in the cache is served from it, with no request to the provider. Lines get there through `warmAudioCache()`, and also from live synthesis when `cacheSynthesizedAudio` allows it. Only services that yield their audio from `runTts` (HTTP and streaming-RPC services, see `supportsAudioCache`) use it, and only in sentence aggregation mode. Defaults to null (no cache). pipedog only. | | `cacheSynthesizedAudio` | `boolean \| ((text: string) => boolean) \| undefined` | no | Whether audio synthesized live is stored in `audioCache`, or a function that decides per text. Defaults to false: only warmed lines are stored, since a generated reply can repeat what the user said. pipedog only. | | `audioCacheNamespace` | `string \| undefined` | no | Extra part of the cache key, for constructor options that change the audio but are not in `settings`. pipedog only. | ### Settings [#settings-2] | Setting | Type | Default | Description | | ---------------- | ------------------------------- | -------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | | `model` | `string` | `"gemini-3.1-flash-tts-preview"` | Gemini TTS model. This default works on both backends. | | `voice` | `string` | `"Kore"` | A Gemini voice (`GeminiTTSService.AVAILABLE_VOICES`) or a custom `voice_...` ID. | | `language` | `Language \| string` | `"en-US"` | Voice language. | | `prompt` | `string \| null` | `null` | Style instructions. See the backend notes below. | | `multiSpeaker` | `boolean` | `false` | Multi-speaker output (Cloud backend only). | | `speakerConfigs` | `GeminiSpeakerConfig[] \| null` | `null` | `{ speaker_alias, speaker_id? }` entries for multi-speaker mode. | | `speakingRate` | `number \| null` | `null` | Speaking rate, 0.25 to 2.0, sent in the streaming audio config (Cloud backend only, pipedog only). Out-of-range updates are ignored with a warning. | ### Usage [#usage-2] With the Gemini API backend: ```ts title="gemini-tts.ts" import { Language } from "@pipedog/core/transcriptions/language"; import { GeminiTTSService } from "@pipedog/google/tts"; // An apiKey selects the Gemini API (GenAI) backend. const tts = new GeminiTTSService({ apiKey: process.env.GOOGLE_API_KEY ?? "", settings: new GeminiTTSService.Settings({ model: "gemini-3.1-flash-tts-preview", voice: "Kore", language: Language.EN_US, }), }); ``` With the Cloud Text-to-Speech backend and a style prompt: ```ts // Service account credentials (and no apiKey) select Cloud Text-to-Speech. const styled = new GeminiTTSService({ credentialsPath: process.env.GOOGLE_APPLICATION_CREDENTIALS ?? "", settings: new GeminiTTSService.Settings({ voice: "Charon", prompt: "Say this in a calm, friendly tone.", }), }); ``` Multi-speaker (Cloud backend only): ```ts // Cloud backend only. Mark up the text with the speaker aliases. const dialogue = new GeminiTTSService({ credentialsPath: process.env.GOOGLE_APPLICATION_CREDENTIALS ?? "", settings: new GeminiTTSService.Settings({ multiSpeaker: true, speakerConfigs: [ { speaker_alias: "Host", speaker_id: "Kore" }, { speaker_alias: "Guest", speaker_id: "Charon" }, ], }), }); ``` ### Notes [#notes-2] * **Backend selection**: `useGenai` wins when set. Otherwise passing `apiKey` selects the Gemini API backend and leaving it out selects Cloud Text-to-Speech. With the Gemini API backend and no `apiKey`, the key falls back to `GOOGLE_API_KEY`. The env var alone never switches the backend. * **Gemini API backend**: no multi-speaker output (a warning is logged and one speaker is used). `prompt` works only with Gemini 3.8 TTS models, sent as a `speechMetadata.style` annotation; other models ignore it with a warning. * **Model names differ by backend.** Cloud Text-to-Speech takes `gemini-3.1-flash-tts-preview`, `gemini-2.5-flash-tts`, `gemini-2.5-flash-lite-preview-tts` and `gemini-2.5-pro-tts`. The Gemini API takes `gemini-3.8-flash-tts`, `gemini-3.8-flash-lite-tts`, `gemini-3.1-flash-tts-preview`, `gemini-2.5-flash-preview-tts` and `gemini-2.5-pro-preview-tts`. * Gemini 3.8 TTS models treat the text as a verbatim transcript. Put momentary vocal events inline as angle-bracket tags (``, ``, ``) and sustained style in `prompt`. * Gemini TTS outputs 24 kHz audio (`GeminiTTSService.GOOGLE_SAMPLE_RATE`). Any other sample rate logs a warning. * A voice that is neither in `AVAILABLE_VOICES` nor a `voice_...` ID logs a warning and is used anyway. ### Retrying blocked sentences [#retrying-blocked-sentences] Gemini TTS sometimes refuses a harmless sentence as a possible violation of Google's usage guidelines, at times after part of its audio has played, and the same request usually goes through on the next try. By default (as upstream) the sentence is lost and an `ErrorFrame` goes upstream. With `retryOnBlocked` (pipedog only), the service asks for the sentence again: ```ts const tts = new GeminiTTSService({ credentialsPath: process.env.GOOGLE_APPLICATION_CREDENTIALS, settings: new GeminiTTSSettings({ voice: "Kore", prompt: "Warm and calm." }), retryOnBlocked: { retries: 1, // the default; `retryOnBlocked: true` is this alone // Only when part of the sentence already played: it starts over with this text. restartText: (text) => `Sorry, ${text}`, // Still blocked after the retries: one last attempt with these settings. fallback: new GeminiTTSSettings({ prompt: null }), }, }); ``` * Only the usage-guidelines block is retried (`isGeminiTTSBlockedError()`): Cloud Text-to-Speech's INVALID_ARGUMENT that says so, the Gemini API's 400 that says so, or a Gemini API response blocked with no audio. Other errors fail the sentence as before. * Audio that already played can't be taken back, so a sentence blocked midway starts over from the beginning, as `restartText(text)` if given, instead of continuing from the middle. * `fallback` is either a `GeminiTTSSettings` delta used for that last attempt only (without the style prompt, or on the Cloud backend a Chirp 3 HD voice: `new GeminiTTSSettings({ model: null, prompt: null, voice: "en-US-Chirp3-HD-Kore" })`), or a function `(text, contextId) => AsyncIterable`, such as another TTS service's `runTts` at the same sample rate. * If every attempt is blocked, the sentence fails with one `ErrorFrame`, as without the option. Restarted and fallback audio is never stored in the audio cache. ## Sharing a client and streaming options [#sharing-a-client-and-streaming-options] These options are pipedog additions: * `sharedClient` (the three Cloud TTS services, and the Cloud backend of `GeminiTTSService`): use one client per process for the same endpoint (`location`), credentials and `clientOptions`, instead of one per service. Building a client costs a credential lookup, a TLS handshake and a token fetch on the first sentence; a server running one pipeline per call pays that once. Call `prewarm()` at start-up to pay it before the first call: it loads the SDK, opens the gRPC channel and fetches a token, and returns `false` (with a warning) instead of throwing if that fails, in which case the first sentence connects as usual. ```ts const ttsOptions = { location: "eu", sharedClient: true }; // At server start-up: a throwaway service warms the shared client. await new GeminiTTSService(ttsOptions).prewarm(); // Per call: every service uses that client. const tts = new GeminiTTSService({ ...ttsOptions, settings: new GeminiTTSSettings({ voice: "Kore" }) }); ``` It is off by default (each service builds its own client, as upstream) because a shared client puts every concurrent stream of the process on one gRPC channel. Services never close a client, so one call ending doesn't affect the others. `getSharedGoogleCloudTTSClient(options)` returns the same shared client. The Gemini API backend talks HTTP with an API key, so it has nothing to share or prewarm. * `client` (`GoogleTTSService` and the Cloud backend of `GeminiTTSService`): a `GoogleCloudTTSClient` to use instead of building one, for full control over sharing: ```ts const client = createGoogleCloudTTSClient({ apiEndpoint: "eu-texttospeech.googleapis.com" }); const tts = new GeminiTTSService({ client, settings: new GeminiTTSSettings({ voice: "Kore" }) }); ``` With `client`, the credentials, `location`, `clientOptions` and `sharedClient` are not used. * `bufferOnlyStartup` (`GoogleTTSService`, `GeminiTTSService`): upstream re-cuts the audio into `chunkSize` pieces (0.5 s) for the whole utterance, which holds back up to one piece throughout. With `bufferOnlyStartup: true`, only the start is buffered; after the first `chunkSize` bytes, audio goes on as it arrives (whole samples). Both streaming services treat a gRPC CANCELLED error, which is what an interrupted stream reports, as a quiet end of synthesis instead of an error. ## Pre-synthesized lines [#pre-synthesized-lines] All three services (and both backends of `GeminiTTSService`) work with the [TTS audio cache](https://polvallverdu.github.io/pipedog/docs/concepts/tts-audio-cache.md): give them an `audioCache` and call `warmAudioCache()` for fixed lines such as a greeting or a filler, and those sentences play from memory with no request to Google. ```ts const audioCache = new TTSAudioCache(); // one per process const tts = new GeminiTTSService({ location: "eu", audioCache }); // While an outbound call rings, before the pipeline starts: await tts.warmAudioCache("Hi! This is the clinic calling about your appointment.", { sampleRate: 24000, // the rate the service will run at }); ``` The cache key covers the text, the settings (model, voice, language, `prompt`, `speakingRate` and the rest), the sample rate, and what changes the audio outside the settings: the backend of `GeminiTTSService` and the `voiceCloningKey` of `GoogleTTSService` (hashed). A settings change therefore misses the cache instead of playing the old voice. Warming works before the pipeline starts, since the client is created with the service; pass `sampleRate` unless the service has one. ## Event handlers [#event-handlers] All three services inherit the `TTSService` events, for example `on_tts_request` (`contextId`, `text`), called for each text sent for synthesis. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/google/package.json`: ✅ means the compat smoke test passes on that runtime. Cloud Text-to-Speech streaming is gRPC only, so `GoogleTTSService`, `GoogleHttpTTSService` and the Cloud backend of `GeminiTTSService` use the Node SDK `@google-cloud/text-to-speech`. It is imported on the first request, so importing the module is safe everywhere, but these classes need Node or Bun. The SDK sits behind the `GoogleCloudTTSClient` interface; pass `client` (or override `_createClient()`) to supply another client. `clientOptions` (pipedog only) is merged into the SDK's `TextToSpeechClient` options. The Gemini API backend of `GeminiTTSService` uses `@google/genai` over `fetch` and runs on Bun, Node, Deno and Cloudflare Workers. `credentials` is parsed in the constructor, as upstream does. With `credentialsPath` or ADC, the SDK resolves credentials on the first request, so a missing credential surfaces as a synthesis error rather than as upstream's constructor error. The deprecated upstream `InputParams` classes, the `params`, `voiceId` and (Gemini) `model` constructor options, and the `GoogleStreamTTSSettings` alias are not ported. # Gradium > Text-to-speech with GradiumTTSService, over Gradium's WebSocket API. Source: https://polvallverdu.github.io/pipedog/docs/services/tts/gradium/ ## Overview [#overview] * **`GradiumTTSService`**: streams text over a WebSocket to Gradium and pushes 48 kHz audio as it arrives. Pipecat's documentation for the same service. `packages/gradium/src/tts.ts`. The text-to-speech WebSocket reference. ## Installation [#installation] ```bash bun add @pipedog/gradium ``` ```ts import { GradiumTTSService } from "@pipedog/gradium/tts"; ``` ## Prerequisites [#prerequisites] * A Gradium account and an API key from [Gradium](https://gradium.ai/). * Pass the key as `apiKey`. The examples read it from `GRADIUM_API_KEY`. ## GradiumTTSService [#gradiumttsservice] Connects to `wss://api.gradium.ai/api/speech/tts`, with the key in the `x-api-key` header. ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | Gradium API key for authentication. | | `url` | `string \| undefined` | no | Gradium websocket API endpoint. Defaults to "wss://api.gradium.ai/api/speech/tts". | | `jsonConfig` | `string \| null \| undefined` | no | Optional JSON configuration string for additional model settings. | | `settings` | `GradiumTTSSettings \| null \| undefined` | no | Runtime-updatable settings. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `reconnectBackoffMinWait` | `number \| undefined` | no | Minimum time, in seconds, to wait between reconnection attempts. | | `reconnectBackoffMaxWait` | `number \| undefined` | no | Maximum time, in seconds, to wait between reconnection attempts. | | `reconnectOnError` | `boolean \| undefined` | no | Whether to automatically reconnect on connection errors. | | `wsCloseTimeout` | `number \| undefined` | no | Maximum time, in seconds, to wait for the peer to acknowledge the websocket closing handshake before dropping the connection. Applied to connections opened through `_websocketConnect()`. Increase it for peers that need longer to complete a graceful close. | | `wsSendTimeout` | `number \| null \| undefined` | no | Maximum time, in seconds, a single send may wait on a peer that has stopped reading before the connection is dropped and the send throws `ConnectionClosedError`. Applied to connections opened through `_websocketConnect()`. Pass `null` to wait indefinitely. Defaults to `WS_SEND_TIMEOUT`. | | `textAggregationMode` | `TextAggregationMode \| null \| undefined` | no | How to aggregate incoming text before synthesis. `TextAggregationMode.SENTENCE` (default) buffers until sentence boundaries, `TextAggregationMode.TOKEN` streams tokens directly for lower latency. | | `pushTextFrames` | `boolean \| undefined` | no | Whether to push TextFrames and LLMFullResponseEndFrames. | | `pushStopFrames` | `boolean \| undefined` | no | Whether to automatically push TTSStoppedFrames. | | `pushStartFrame` | `boolean \| undefined` | no | Whether to automatically create audio contexts and push TTSStartedFrames. When true, the base class handles `createAudioContext` and yields `TTSStartedFrame` before each synthesis call, so `runTts` implementations do not need to. | | `stopFrameTimeoutS` | `number \| undefined` | no | Idle time before pushing TTSStoppedFrame when pushStopFrames is true. | | `pushSilenceAfterStop` | `boolean \| undefined` | no | Whether to push silence audio after TTSStoppedFrame. | | `silenceTimeS` | `number \| undefined` | no | Duration of silence to push when pushSilenceAfterStop is true. | | `pauseFrameProcessing` | `boolean \| undefined` | no | Whether to hold incoming frames after a turn until its audio contexts end. Needed by services whose audio doesn't carry its request's context ID. | | `maxConsecutiveZeroAudioContexts` | `number \| undefined` | no | How many consecutive TTS contexts may complete without producing any audio before the service is reported unable to do its job. Catches a provider that accepts requests and stays silent — an unknown voice ID, say — which no error ever surfaces. Every silent context reports an error the service can carry on from, whatever the limit is; on reaching the limit the service reports a permanent error instead, stops being given work, and the pipeline worker applies its `ProcessorUnusablePolicy`. Set to 0 to report silent contexts without ever writing the service off. | | `appendTrailingSpace` | `boolean \| undefined` | no | Whether to append a trailing space to text before sending to TTS. This helps prevent some TTS services from vocalizing trailing punctuation (e.g., "dot"). Only applied in sentence aggregation mode; when streaming tokens, the incoming text's own whitespace is preserved. | | `skipTextTypes` | `string[] \| null \| undefined` | no | List of text types that should not be spoken. | | `textTransforms` | `[string, TextTransform][] \| null \| undefined` | no | A list of TextTransforms to transform text just before sending it to TTS. Each TextTransform takes the aggregated text and its type, and returns the transformed text. To register, provide a list of tuples of `[textType \| "*", transformFunction]`. | | `transportDestination` | `string \| null \| undefined` | no | Destination for generated audio frames. | | `reuseContextIdWithinTurn` | `boolean \| undefined` | no | Whether the service should reuse context IDs within the same turn. | | `audioCache` | `TTSAudioCacheStore \| null \| undefined` | no | Where to keep synthesized audio for reuse (e.g. a shared `TTSAudioCache`). Text whose audio is in the cache is served from it, with no request to the provider. Lines get there through `warmAudioCache()`, and also from live synthesis when `cacheSynthesizedAudio` allows it. Only services that yield their audio from `runTts` (HTTP and streaming-RPC services, see `supportsAudioCache`) use it, and only in sentence aggregation mode. Defaults to null (no cache). pipedog only. | | `cacheSynthesizedAudio` | `boolean \| ((text: string) => boolean) \| undefined` | no | Whether audio synthesized live is stored in `audioCache`, or a function that decides per text. Defaults to false: only warmed lines are stored, since a generated reply can repeat what the user said. pipedog only. | | `audioCacheNamespace` | `string \| undefined` | no | Extra part of the cache key, for constructor options that change the audio but are not in `settings`. pipedog only. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new GradiumTTSService.Settings({...})` (`GradiumTTSSettings`). Any setting you leave out keeps the default below. | Setting | Type | Default | Description | | ---------- | ------------------ | -------------------- | -------------------------------------------- | | `model` | `string` | `"default"` | Speech model. | | `voice` | `string` | `"_6Aslh2DxfmnRLmP"` | Voice id. Changing it reconnects the socket. | | `language` | `Language \| null` | `null` | Language. | ### Usage [#usage] ```ts title="tts.ts" import { GradiumTTSService } from "@pipedog/gradium/tts"; const tts = new GradiumTTSService({ apiKey: process.env.GRADIUM_API_KEY ?? "", settings: new GradiumTTSService.Settings({ voice: "_6Aslh2DxfmnRLmP" }), }); ``` ### Notes [#notes] * The sample rate is fixed at 48000 Hz. * `jsonConfig` takes a JSON string of extra model settings, sent with the setup message. | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/gradium/package.json`: ✅ means the compat smoke test passes on that runtime. ## Runtime notes [#runtime-notes] * The WebSocket goes through the core `websocketConnect`, with headers on the handshake. * The deprecated `InputParams`, `params`, `voice_id` and `model` constructor options are not ported. * Tested against a local WebSocket server (upstream has no Gradium TTS test), not against Gradium itself. # Groq > Text-to-speech with GroqTTSService, over Groq's OpenAI-compatible speech API. Source: https://polvallverdu.github.io/pipedog/docs/services/tts/groq/ ## Overview [#overview] * **`GroqTTSService`**: posts each sentence to Groq's `/audio/speech` endpoint and pushes the WAV it returns as audio frames. Pipecat's documentation for the same service. `packages/groq/src/tts.ts`. Text-to-speech models and voices. ## Installation [#installation] ```bash bun add @pipedog/groq ``` ```ts import { GroqTTSService } from "@pipedog/groq/tts"; ``` ## Prerequisites [#prerequisites] * A Groq account and an API key from the [Groq console](https://console.groq.com/). * Pass the key as `apiKey`. The examples read it from `GROQ_API_KEY`. ## GroqTTSService [#groqttsservice] Posts to `https://api.groq.com/openai/v1/audio/speech`. ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | Groq API key for authentication. | | `outputFormat` | `GroqAudioFormat \| (string & {}) \| undefined` | no | Audio output format. Defaults to "wav". | | `baseUrl` | `string \| null \| undefined` | no | Base URL of Groq's OpenAI-compatible API. Defaults to `"https://api.groq.com/openai/v1"`. (Upstream: the SDK's default.) | | `settings` | `GroqTTSSettings \| null \| undefined` | no | Runtime-updatable settings; values override the built-in defaults. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `textAggregationMode` | `TextAggregationMode \| null \| undefined` | no | How to aggregate incoming text before synthesis. `TextAggregationMode.SENTENCE` (default) buffers until sentence boundaries, `TextAggregationMode.TOKEN` streams tokens directly for lower latency. | | `pushTextFrames` | `boolean \| undefined` | no | Whether to push TextFrames and LLMFullResponseEndFrames. | | `pushStopFrames` | `boolean \| undefined` | no | Whether to automatically push TTSStoppedFrames. | | `pushStartFrame` | `boolean \| undefined` | no | Whether to automatically create audio contexts and push TTSStartedFrames. When true, the base class handles `createAudioContext` and yields `TTSStartedFrame` before each synthesis call, so `runTts` implementations do not need to. | | `stopFrameTimeoutS` | `number \| undefined` | no | Idle time before pushing TTSStoppedFrame when pushStopFrames is true. | | `pushSilenceAfterStop` | `boolean \| undefined` | no | Whether to push silence audio after TTSStoppedFrame. | | `silenceTimeS` | `number \| undefined` | no | Duration of silence to push when pushSilenceAfterStop is true. | | `pauseFrameProcessing` | `boolean \| undefined` | no | Whether to hold incoming frames after a turn until its audio contexts end. Needed by services whose audio doesn't carry its request's context ID. | | `maxConsecutiveZeroAudioContexts` | `number \| undefined` | no | How many consecutive TTS contexts may complete without producing any audio before the service is reported unable to do its job. Catches a provider that accepts requests and stays silent — an unknown voice ID, say — which no error ever surfaces. Every silent context reports an error the service can carry on from, whatever the limit is; on reaching the limit the service reports a permanent error instead, stops being given work, and the pipeline worker applies its `ProcessorUnusablePolicy`. Set to 0 to report silent contexts without ever writing the service off. | | `appendTrailingSpace` | `boolean \| undefined` | no | Whether to append a trailing space to text before sending to TTS. This helps prevent some TTS services from vocalizing trailing punctuation (e.g., "dot"). Only applied in sentence aggregation mode; when streaming tokens, the incoming text's own whitespace is preserved. | | `sampleRate` | `number \| null \| undefined` | no | Output sample rate for generated audio. | | `skipTextTypes` | `string[] \| null \| undefined` | no | List of text types that should not be spoken. | | `textTransforms` | `[string, TextTransform][] \| null \| undefined` | no | A list of TextTransforms to transform text just before sending it to TTS. Each TextTransform takes the aggregated text and its type, and returns the transformed text. To register, provide a list of tuples of `[textType \| "*", transformFunction]`. | | `transportDestination` | `string \| null \| undefined` | no | Destination for generated audio frames. | | `reuseContextIdWithinTurn` | `boolean \| undefined` | no | Whether the service should reuse context IDs within the same turn. | | `audioCache` | `TTSAudioCacheStore \| null \| undefined` | no | Where to keep synthesized audio for reuse (e.g. a shared `TTSAudioCache`). Text whose audio is in the cache is served from it, with no request to the provider. Lines get there through `warmAudioCache()`, and also from live synthesis when `cacheSynthesizedAudio` allows it. Only services that yield their audio from `runTts` (HTTP and streaming-RPC services, see `supportsAudioCache`) use it, and only in sentence aggregation mode. Defaults to null (no cache). pipedog only. | | `cacheSynthesizedAudio` | `boolean \| ((text: string) => boolean) \| undefined` | no | Whether audio synthesized live is stored in `audioCache`, or a function that decides per text. Defaults to false: only warmed lines are stored, since a generated reply can repeat what the user said. pipedog only. | | `audioCacheNamespace` | `string \| undefined` | no | Extra part of the cache key, for constructor options that change the audio but are not in `settings`. pipedog only. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new GroqTTSService.Settings({...})` (`GroqTTSSettings`). Any setting you leave out keeps the default below. | Setting | Type | Default | Description | | ---------- | ---------------- | --------------------------------- | -------------------------------------- | | `model` | `string` | `"canopylabs/orpheus-v1-english"` | Speech model. | | `voice` | `string` | `"autumn"` | Voice. | | `language` | `string` | `"en"` | Language. | | `speed` | `number \| null` | `1.0` | Speed. Groq only supports 1.0 for now. | ### Usage [#usage] ```ts title="tts.ts" import { TTSUpdateSettingsFrame } from "@pipedog/core/frames/frames"; import { GroqTTSService } from "@pipedog/groq/tts"; const tts = new GroqTTSService({ apiKey: process.env.GROQ_API_KEY ?? "", settings: new GroqTTSService.Settings({ voice: "autumn" }), }); ``` Change settings mid-call with an `TTSUpdateSettingsFrame`: ```ts // Queue it on the worker, or push it from a processor upstream of the service. const update = new TTSUpdateSettingsFrame({ delta: new GroqTTSService.Settings({ voice: "diana" }), }); ``` ### Notes [#notes] * `outputFormat` is `"wav"` by default; the frames carry the WAV's own sample rate and channel count. * A failed request produces an `ErrorFrame` instead of audio. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/groq/package.json`: ✅ means the compat smoke test passes on that runtime. * Requests go through `fetch` (upstream uses the `groq` SDK). It works on Bun, Node, Deno and Cloudflare Workers. * The whole response is read and parsed as one WAV. Upstream parses each streamed chunk as a WAV. * The deprecated `InputParams` and the `params`, `model_name` and `voice_id` options are not ported. Use `settings`. * Tested against a local fake, not against Groq itself. # Hume > Text-to-speech with HumeTTSService, Hume's Octave voices over HTTP streaming, with word timestamps. Source: https://polvallverdu.github.io/pipedog/docs/services/tts/hume/ ## Overview [#overview] Hume's Octave models give speech acting directions and expressive control. pipedog ports Pipecat's **`HumeTTSService`**: it posts each request to Hume's `tts/stream/json` endpoint and reads the newline-delimited JSON stream back, emitting 48 kHz PCM audio and word-level timestamps. Pipecat's documentation for the same service. `packages/hume/src/tts.ts`. The streaming endpoint, voices and Octave versions. ## Installation [#installation] ```bash bun add @pipedog/hume ``` ```ts import { HumeTTSService } from "@pipedog/hume/tts"; ``` ## Prerequisites [#prerequisites] * A Hume account and an API key from the [Hume platform](https://platform.hume.ai). * A voice id, set as `voice`. Voice names are not supported, and without a voice the service reports an error. * Pass the key as `apiKey`, or set the `HUME_API_KEY` environment variable. ## HumeTTSService [#humettsservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string \| null \| undefined` | no | Hume API key. If omitted, reads the `HUME_API_KEY` environment variable. | | `fetch` | `{ (input: RequestInfo \| URL, init?: RequestInit): Promise; (input: string \| URL \| Request, init?: RequestInit): Promise; } \| null \| undefined` | no | `fetch` used for HTTP requests. Defaults to the global `fetch`. DIVERGENCE: replaces the Hume SDK's HTTP client. | | `baseUrl` | `string \| undefined` | no | Base URL of the Hume API. Defaults to "https://api.hume.ai". | | `settings` | `HumeTTSSettings \| null \| undefined` | no | Runtime-updatable settings. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `textAggregationMode` | `TextAggregationMode \| null \| undefined` | no | How to aggregate incoming text before synthesis. `TextAggregationMode.SENTENCE` (default) buffers until sentence boundaries, `TextAggregationMode.TOKEN` streams tokens directly for lower latency. | | `pushTextFrames` | `boolean \| undefined` | no | Whether to push TextFrames and LLMFullResponseEndFrames. | | `pushStopFrames` | `boolean \| undefined` | no | Whether to automatically push TTSStoppedFrames. | | `pushStartFrame` | `boolean \| undefined` | no | Whether to automatically create audio contexts and push TTSStartedFrames. When true, the base class handles `createAudioContext` and yields `TTSStartedFrame` before each synthesis call, so `runTts` implementations do not need to. | | `stopFrameTimeoutS` | `number \| undefined` | no | Idle time before pushing TTSStoppedFrame when pushStopFrames is true. | | `pushSilenceAfterStop` | `boolean \| undefined` | no | Whether to push silence audio after TTSStoppedFrame. | | `silenceTimeS` | `number \| undefined` | no | Duration of silence to push when pushSilenceAfterStop is true. | | `pauseFrameProcessing` | `boolean \| undefined` | no | Whether to hold incoming frames after a turn until its audio contexts end. Needed by services whose audio doesn't carry its request's context ID. | | `maxConsecutiveZeroAudioContexts` | `number \| undefined` | no | How many consecutive TTS contexts may complete without producing any audio before the service is reported unable to do its job. Catches a provider that accepts requests and stays silent — an unknown voice ID, say — which no error ever surfaces. Every silent context reports an error the service can carry on from, whatever the limit is; on reaching the limit the service reports a permanent error instead, stops being given work, and the pipeline worker applies its `ProcessorUnusablePolicy`. Set to 0 to report silent contexts without ever writing the service off. | | `appendTrailingSpace` | `boolean \| undefined` | no | Whether to append a trailing space to text before sending to TTS. This helps prevent some TTS services from vocalizing trailing punctuation (e.g., "dot"). Only applied in sentence aggregation mode; when streaming tokens, the incoming text's own whitespace is preserved. | | `sampleRate` | `number \| null \| undefined` | no | Output sample rate for generated audio. | | `skipTextTypes` | `string[] \| null \| undefined` | no | List of text types that should not be spoken. | | `textTransforms` | `[string, TextTransform][] \| null \| undefined` | no | A list of TextTransforms to transform text just before sending it to TTS. Each TextTransform takes the aggregated text and its type, and returns the transformed text. To register, provide a list of tuples of `[textType \| "*", transformFunction]`. | | `transportDestination` | `string \| null \| undefined` | no | Destination for generated audio frames. | | `reuseContextIdWithinTurn` | `boolean \| undefined` | no | Whether the service should reuse context IDs within the same turn. | | `audioCache` | `TTSAudioCacheStore \| null \| undefined` | no | Where to keep synthesized audio for reuse (e.g. a shared `TTSAudioCache`). Text whose audio is in the cache is served from it, with no request to the provider. Lines get there through `warmAudioCache()`, and also from live synthesis when `cacheSynthesizedAudio` allows it. Only services that yield their audio from `runTts` (HTTP and streaming-RPC services, see `supportsAudioCache`) use it, and only in sentence aggregation mode. Defaults to null (no cache). pipedog only. | | `cacheSynthesizedAudio` | `boolean \| ((text: string) => boolean) \| undefined` | no | Whether audio synthesized live is stored in `audioCache`, or a function that decides per text. Defaults to false: only warmed lines are stored, since a generated reply can repeat what the user said. pipedog only. | | `audioCacheNamespace` | `string \| undefined` | no | Extra part of the cache key, for constructor options that change the audio but are not in `settings`. pipedog only. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new HumeTTSService.Settings({...})`. | Setting | Type | Default | Description | | ----------------- | ---------------- | ------- | ----------------------------------------------------------------------------------- | | `voice` | `string \| null` | `null` | Hume voice id. Required to synthesize. | | `description` | `string \| null` | `null` | Natural-language acting directions, up to 100 characters. Selects Octave version 1. | | `speed` | `number \| null` | `null` | Speaking-rate multiplier, 0.5 to 2.0. | | `trailingSilence` | `number \| null` | `null` | Seconds of silence to add at the end, 0 to 5. | | `temperature` | `number \| null` | `null` | Sampling temperature. Hume's per-model default applies when unset. | ### Usage [#usage] ```ts title="tts.ts" import { TTSUpdateSettingsFrame } from "@pipedog/core/frames/frames"; import { HumeTTSService } from "@pipedog/hume/tts"; const tts = new HumeTTSService({ // Defaults to the HUME_API_KEY environment variable. apiKey: process.env.HUME_API_KEY, settings: new HumeTTSService.Settings({ // A Hume voice id. Voice names are not supported. voice: "your-voice-id", speed: 1.1, trailingSilence: 0.3, }), }); ``` Acting directions and temperature: ```ts // Acting directions switch Hume to Octave version 1. const acting = new HumeTTSService({ apiKey: process.env.HUME_API_KEY, settings: new HumeTTSService.Settings({ voice: "your-voice-id", description: "warm, unhurried", temperature: 0.7, }), }); ``` Change settings mid-call with a `TTSUpdateSettingsFrame`: ```ts // Queue it on the worker, or push it from a processor upstream of the service. // It applies from the next request. const update = new TTSUpdateSettingsFrame({ delta: new HumeTTSService.Settings({ voice: "another-voice-id", speed: 0.9 }), }); ``` ### Event handlers [#event-handlers] Inherited from `TTSService`: | Event | Description | | ---------------- | ----------------------------------------------- | | `on_tts_request` | A text chunk is about to be sent for synthesis. | ### Notes [#notes] * Hume streams at 48 kHz. Another `sampleRate` is accepted with a warning, and the service labels the frames with it without resampling. * Audio is buffered until it reaches a full chunk, to avoid glitches. * Instant mode is always on, and word timestamps are always requested. Octave version 2 is used, or version 1 when a `description` is set. * Word timestamps keep a running offset across the utterances of a turn. It resets on an interruption and when the context ends. ## Runtime notes [#runtime-notes] * The service calls Hume's HTTP endpoint with `fetch` (the global one unless you pass `fetch`) instead of Hume's SDK, so it runs on Bun, Node, Deno and Cloudflare Workers. The request and response formats follow the `hume` TypeScript SDK and Hume's API reference. * Requests carry `X-Hume-Client-Name: pipedog` and the pipedog version, where Pipecat sends its own name and version. * Not ported: the deprecated `InputParams`, `update_setting()` and the `voiceId` and `params` constructor options. Use `settings`. | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/hume/package.json`: ✅ means the compat smoke test passes on that runtime. # Text-to-speech > Turn the bot's text into audio frames. Source: https://polvallverdu.github.io/pipedog/docs/services/tts/ {/* Generated by scripts/generate-providers.ts from data/providers.ts. */} A text-to-speech service takes the bot's text frames and pushes audio frames, with word timestamps where the provider sends them. | Provider | Classes | Package | Status | | ------------------------------------------------------ | ----------------------------------------------------------------------------------- | ----------------------- | ----------- | | [OpenAI](https://polvallverdu.github.io/pipedog/docs/services/tts/openai.md) | `OpenAITTSService` | `@pipedog/openai` | Ported | | [Google](https://polvallverdu.github.io/pipedog/docs/services/tts/google.md) | `GoogleTTSService`, `GoogleHttpTTSService`, `GeminiTTSService` | `@pipedog/google` | Ported | | [Deepgram](https://polvallverdu.github.io/pipedog/docs/services/tts/deepgram.md) | `DeepgramTTSService`, `DeepgramHttpTTSService`, `DeepgramFluxTTSService` | `@pipedog/deepgram` | Ported | | [Cloudflare Workers AI](https://polvallverdu.github.io/pipedog/docs/services/tts/cloudflare.md) | `WorkersAIAuraTTSService`, `WorkersAIAuraHttpTTSService`, `WorkersAIMeloTTSService` | `@pipedog/cloudflare` | Ported | | [Speechmatics](https://polvallverdu.github.io/pipedog/docs/services/tts/speechmatics.md) | `SpeechmaticsTTSService` | `@pipedog/speechmatics` | Ported | | [Soniox](https://polvallverdu.github.io/pipedog/docs/services/tts/soniox.md) | `SonioxTTSService` | `@pipedog/soniox` | Ported | | [Cartesia](https://polvallverdu.github.io/pipedog/docs/services/tts/cartesia.md) | `CartesiaTTSService`, `CartesiaHttpTTSService` | `@pipedog/cartesia` | Ported | | [ElevenLabs](https://polvallverdu.github.io/pipedog/docs/services/tts/elevenlabs.md) | `ElevenLabsTTSService`, `ElevenLabsHttpTTSService`, `ElevenLabsDialogueTTSService` | `@pipedog/elevenlabs` | Ported | | [Groq](https://polvallverdu.github.io/pipedog/docs/services/tts/groq.md) | `GroqTTSService` | `@pipedog/groq` | Ported | | [Mistral](https://polvallverdu.github.io/pipedog/docs/services/tts/mistral.md) | `MistralTTSService` | `@pipedog/mistral` | Ported | | [Together AI](https://polvallverdu.github.io/pipedog/docs/services/tts/together.md) | `TogetherTTSService` | `@pipedog/together` | Ported | | [xAI](https://polvallverdu.github.io/pipedog/docs/services/tts/xai.md) | `XAITTSService`, `XAIHttpTTSService` | `@pipedog/xai` | Ported | | [Sarvam AI](https://polvallverdu.github.io/pipedog/docs/services/tts/sarvam.md) | `SarvamTTSService`, `SarvamHttpTTSService` | `@pipedog/sarvam` | Ported | | [Smallest AI](https://polvallverdu.github.io/pipedog/docs/services/tts/smallest.md) | `SmallestTTSService` | `@pipedog/smallest` | Ported | | [Gradium](https://polvallverdu.github.io/pipedog/docs/services/tts/gradium.md) | `GradiumTTSService` | `@pipedog/gradium` | Ported | | [Inworld](https://polvallverdu.github.io/pipedog/docs/services/tts/inworld.md) | `InworldTTSService`, `InworldHttpTTSService` | `@pipedog/inworld` | Ported | | [AWS Polly](https://polvallverdu.github.io/pipedog/docs/services/tts/aws.md) | `AWSPollyTTSService` | `@pipedog/aws` | Ported | | [Azure Speech](https://polvallverdu.github.io/pipedog/docs/services/tts/azure.md) | `AzureTTSService`, `AzureHttpTTSService` | `@pipedog/azure` | Ported | | [Kokoro (local)](https://polvallverdu.github.io/pipedog/docs/services/tts/kokoro.md) | `KokoroTTSService` | `@pipedog/kokoro` | Ported | | [Piper](https://polvallverdu.github.io/pipedog/docs/services/tts/piper.md) | `PiperTTSService`, `PiperHttpTTSService` | `@pipedog/piper` | Ported | | [Pocket TTS (local)](https://polvallverdu.github.io/pipedog/docs/services/tts/pocket-tts.md) | `PocketTTSService` | `@pipedog/pocket-tts` | Ported | | [Rime](https://polvallverdu.github.io/pipedog/docs/services/tts/rime.md) | `RimeTTSService`, `RimeHttpTTSService`, `RimeV1TTSService` | `@pipedog/rime` | Ported | | [Fish Audio](https://polvallverdu.github.io/pipedog/docs/services/tts/fish.md) | `FishAudioTTSService` | `@pipedog/fish` | Ported | | [Hume](https://polvallverdu.github.io/pipedog/docs/services/tts/hume.md) | `HumeTTSService` | `@pipedog/hume` | Ported | | [Neuphonic](https://polvallverdu.github.io/pipedog/docs/services/tts/neuphonic.md) | `NeuphonicTTSService`, `NeuphonicHttpTTSService` | `@pipedog/neuphonic` | Ported | | [Speechify](https://polvallverdu.github.io/pipedog/docs/services/tts/speechify.md) | `SpeechifyHttpTTSService` | `@pipedog/speechify` | Ported | | [MiniMax](https://polvallverdu.github.io/pipedog/docs/services/tts/minimax.md) | `MiniMaxHttpTTSService` | `@pipedog/minimax` | Ported | | [Resemble AI](https://polvallverdu.github.io/pipedog/docs/services/tts/resembleai.md) | `ResembleAITTSService` | `@pipedog/resembleai` | Ported | | [Camb.ai](https://polvallverdu.github.io/pipedog/docs/services/tts/camb.md) | `CambTTSService` | `@pipedog/camb` | Ported | | [Async](https://polvallverdu.github.io/pipedog/docs/services/tts/asyncai.md) | `AsyncAITTSService`, `AsyncAIHttpTTSService` | `@pipedog/asyncai` | Ported | | [Bland](https://polvallverdu.github.io/pipedog/docs/services/tts/bland.md) | `BlandTTSService`, `BlandHttpTTSService` | `@pipedog/bland` | Ported | | [AssemblyAI](https://polvallverdu.github.io/pipedog/docs/services/tts/assemblyai.md) | `AssemblyAITTSService` | `@pipedog/assemblyai` | Ported | | [NVIDIA Riva](https://polvallverdu.github.io/pipedog/docs/services/tts/nvidia.md) | `NvidiaTTSService` | | Coming soon | Pipecat's [text-to-speech reference](https://docs.pipecat.ai/api-reference/server/services/tts) lists every upstream provider. Those not in this table are on [Other upstream providers](https://polvallverdu.github.io/pipedog/docs/services/other-upstream.md). # Inworld > Text-to-speech with InworldTTSService (WebSocket) and InworldHttpTTSService (HTTP). Source: https://polvallverdu.github.io/pipedog/docs/services/tts/inworld/ ## Overview [#overview] pipedog ports both of Pipecat's Inworld text-to-speech services: * **`InworldTTSService`**: Inworld TTS over a bidirectional WebSocket, with several audio contexts on one connection and word-level timestamps. Lowest latency. * **`InworldHttpTTSService`**: Inworld TTS over HTTP, streaming (`/tts/v1/voice:stream`) or in one piece (`/tts/v1/voice`), also with word-level timestamps. Pipecat's documentation for the same services. `packages/inworld/src/tts.ts`. Inworld TTS models and voices. ## Installation [#installation] ```bash bun add @pipedog/inworld ``` ```ts import { InworldHttpTTSService, InworldTTSService } from "@pipedog/inworld/tts"; ``` ## Prerequisites [#prerequisites] * An Inworld account and an API key from the [Inworld portal](https://platform.inworld.ai). * Pass the key as `apiKey` (the Base64 key Inworld gives you). The examples read it from `INWORLD_API_KEY`. ## InworldTTSService [#inworldttsservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | Inworld API key. | | `url` | `string \| undefined` | no | URL of the Inworld WebSocket API. | | `encoding` | `string \| undefined` | no | Audio encoding format. Defaults to "PCM". | | `autoMode` | `boolean \| null \| undefined` | no | Whether to use auto mode. When enabled, the server controls flushing of buffered text. Defaults to true. | | `applyTextNormalization` | `string \| null \| undefined` | no | Whether to apply text normalization. | | `timestampTransportStrategy` | `InworldTimestampTransportStrategy \| null \| undefined` | no | Strategy for timestamp transport ("ASYNC" or "SYNC"). Defaults to "ASYNC". | | `settings` | `InworldTTSSettings \| null \| undefined` | no | Runtime-updatable settings. Its given fields override the defaults (model "inworld-tts-2", voice "Ashley"). | | `textAggregationMode` | `TextAggregationMode \| null \| undefined` | no | How to aggregate text before synthesis. | | `appendTrailingSpace` | `boolean \| undefined` | no | Whether to append a trailing space to text before sending to TTS. Defaults to true. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `reconnectBackoffMinWait` | `number \| undefined` | no | Minimum time, in seconds, to wait between reconnection attempts. | | `reconnectBackoffMaxWait` | `number \| undefined` | no | Maximum time, in seconds, to wait between reconnection attempts. | | `reconnectOnError` | `boolean \| undefined` | no | Whether to automatically reconnect on connection errors. | | `wsCloseTimeout` | `number \| undefined` | no | Maximum time, in seconds, to wait for the peer to acknowledge the websocket closing handshake before dropping the connection. Applied to connections opened through `_websocketConnect()`. Increase it for peers that need longer to complete a graceful close. | | `wsSendTimeout` | `number \| null \| undefined` | no | Maximum time, in seconds, a single send may wait on a peer that has stopped reading before the connection is dropped and the send throws `ConnectionClosedError`. Applied to connections opened through `_websocketConnect()`. Pass `null` to wait indefinitely. Defaults to `WS_SEND_TIMEOUT`. | | `pushTextFrames` | `boolean \| undefined` | no | Whether to push TextFrames and LLMFullResponseEndFrames. | | `pushStopFrames` | `boolean \| undefined` | no | Whether to automatically push TTSStoppedFrames. | | `pushStartFrame` | `boolean \| undefined` | no | Whether to automatically create audio contexts and push TTSStartedFrames. When true, the base class handles `createAudioContext` and yields `TTSStartedFrame` before each synthesis call, so `runTts` implementations do not need to. | | `stopFrameTimeoutS` | `number \| undefined` | no | Idle time before pushing TTSStoppedFrame when pushStopFrames is true. | | `pushSilenceAfterStop` | `boolean \| undefined` | no | Whether to push silence audio after TTSStoppedFrame. | | `silenceTimeS` | `number \| undefined` | no | Duration of silence to push when pushSilenceAfterStop is true. | | `pauseFrameProcessing` | `boolean \| undefined` | no | Whether to hold incoming frames after a turn until its audio contexts end. Needed by services whose audio doesn't carry its request's context ID. | | `maxConsecutiveZeroAudioContexts` | `number \| undefined` | no | How many consecutive TTS contexts may complete without producing any audio before the service is reported unable to do its job. Catches a provider that accepts requests and stays silent — an unknown voice ID, say — which no error ever surfaces. Every silent context reports an error the service can carry on from, whatever the limit is; on reaching the limit the service reports a permanent error instead, stops being given work, and the pipeline worker applies its `ProcessorUnusablePolicy`. Set to 0 to report silent contexts without ever writing the service off. | | `sampleRate` | `number \| null \| undefined` | no | Output sample rate for generated audio. | | `skipTextTypes` | `string[] \| null \| undefined` | no | List of text types that should not be spoken. | | `textTransforms` | `[string, TextTransform][] \| null \| undefined` | no | A list of TextTransforms to transform text just before sending it to TTS. Each TextTransform takes the aggregated text and its type, and returns the transformed text. To register, provide a list of tuples of `[textType \| "*", transformFunction]`. | | `transportDestination` | `string \| null \| undefined` | no | Destination for generated audio frames. | | `reuseContextIdWithinTurn` | `boolean \| undefined` | no | Whether the service should reuse context IDs within the same turn. | | `audioCache` | `TTSAudioCacheStore \| null \| undefined` | no | Where to keep synthesized audio for reuse (e.g. a shared `TTSAudioCache`). Text whose audio is in the cache is served from it, with no request to the provider. Lines get there through `warmAudioCache()`, and also from live synthesis when `cacheSynthesizedAudio` allows it. Only services that yield their audio from `runTts` (HTTP and streaming-RPC services, see `supportsAudioCache`) use it, and only in sentence aggregation mode. Defaults to null (no cache). pipedog only. | | `cacheSynthesizedAudio` | `boolean \| ((text: string) => boolean) \| undefined` | no | Whether audio synthesized live is stored in `audioCache`, or a function that decides per text. Defaults to false: only warmed lines are stored, since a generated reply can repeat what the user said. pipedog only. | | `audioCacheNamespace` | `string \| undefined` | no | Extra part of the cache key, for constructor options that change the audio but are not in `settings`. pipedog only. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new InworldTTSService.Settings({...})` (`InworldTTSSettings`). Any setting you leave out keeps the default below. `InworldHttpTTSService` uses the same settings class. | Setting | Type | Default | Description | | -------------- | ---------------------------------------------- | ----------------- | --------------------------------------------------- | | `voice` | `string` | `"Ashley"` | Inworld voice id. | | `model` | `string` | `"inworld-tts-2"` | TTS model. | | `language` | `Language \| null` | `null` | Language of the text. | | `speakingRate` | `number \| null` | `null` | Speaking rate. | | `temperature` | `number \| null` | `null` | Sampling temperature. | | `deliveryMode` | `"STABLE" \| "BALANCED" \| "CREATIVE" \| null` | `null` | Stability vs. expressiveness. Only `inworld-tts-2`. | ### Usage [#usage] ```ts title="tts.ts" import { TTSUpdateSettingsFrame } from "@pipedog/core/frames/frames"; import { Language } from "@pipedog/core/transcriptions/language"; import { InworldHttpTTSService, InworldTTSService } from "@pipedog/inworld/tts"; const tts = new InworldTTSService({ apiKey: process.env.INWORLD_API_KEY ?? "", settings: new InworldTTSService.Settings({ voice: "Ashley", model: "inworld-tts-2", language: Language.EN, deliveryMode: "BALANCED", }), }); ``` Change the voice mid-call with a `TTSUpdateSettingsFrame`: ```ts // Applies from the next context the service opens. const update = new TTSUpdateSettingsFrame({ delta: new InworldTTSService.Settings({ voice: "Dennis", temperature: 0.9 }), }); ``` ## InworldHttpTTSService [#inworldhttpttsservice] ### Configuration [#configuration-1] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | Inworld API key. | | `fetch` | `{ (input: RequestInfo \| URL, init?: RequestInit): Promise; (input: string \| URL \| Request, init?: RequestInit): Promise; } \| null \| undefined` | no | `fetch` used for HTTP requests. Defaults to the global `fetch`. DIVERGENCE: replaces upstream's required `aiohttp_session`. | | `streaming` | `boolean \| undefined` | no | Whether to use streaming mode. Defaults to true. | | `encoding` | `string \| undefined` | no | Audio encoding format. Defaults to "PCM". | | `timestampTransportStrategy` | `InworldTimestampTransportStrategy \| null \| undefined` | no | Strategy for timestamp transport ("ASYNC" or "SYNC"). Defaults to "ASYNC". | | `settings` | `InworldTTSSettings \| null \| undefined` | no | Runtime-updatable settings. Its given fields override the defaults (model "inworld-tts-2", voice "Ashley"). | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `textAggregationMode` | `TextAggregationMode \| null \| undefined` | no | How to aggregate incoming text before synthesis. `TextAggregationMode.SENTENCE` (default) buffers until sentence boundaries, `TextAggregationMode.TOKEN` streams tokens directly for lower latency. | | `pushTextFrames` | `boolean \| undefined` | no | Whether to push TextFrames and LLMFullResponseEndFrames. | | `pushStopFrames` | `boolean \| undefined` | no | Whether to automatically push TTSStoppedFrames. | | `pushStartFrame` | `boolean \| undefined` | no | Whether to automatically create audio contexts and push TTSStartedFrames. When true, the base class handles `createAudioContext` and yields `TTSStartedFrame` before each synthesis call, so `runTts` implementations do not need to. | | `stopFrameTimeoutS` | `number \| undefined` | no | Idle time before pushing TTSStoppedFrame when pushStopFrames is true. | | `pushSilenceAfterStop` | `boolean \| undefined` | no | Whether to push silence audio after TTSStoppedFrame. | | `silenceTimeS` | `number \| undefined` | no | Duration of silence to push when pushSilenceAfterStop is true. | | `pauseFrameProcessing` | `boolean \| undefined` | no | Whether to hold incoming frames after a turn until its audio contexts end. Needed by services whose audio doesn't carry its request's context ID. | | `maxConsecutiveZeroAudioContexts` | `number \| undefined` | no | How many consecutive TTS contexts may complete without producing any audio before the service is reported unable to do its job. Catches a provider that accepts requests and stays silent — an unknown voice ID, say — which no error ever surfaces. Every silent context reports an error the service can carry on from, whatever the limit is; on reaching the limit the service reports a permanent error instead, stops being given work, and the pipeline worker applies its `ProcessorUnusablePolicy`. Set to 0 to report silent contexts without ever writing the service off. | | `appendTrailingSpace` | `boolean \| undefined` | no | Whether to append a trailing space to text before sending to TTS. This helps prevent some TTS services from vocalizing trailing punctuation (e.g., "dot"). Only applied in sentence aggregation mode; when streaming tokens, the incoming text's own whitespace is preserved. | | `sampleRate` | `number \| null \| undefined` | no | Output sample rate for generated audio. | | `skipTextTypes` | `string[] \| null \| undefined` | no | List of text types that should not be spoken. | | `textTransforms` | `[string, TextTransform][] \| null \| undefined` | no | A list of TextTransforms to transform text just before sending it to TTS. Each TextTransform takes the aggregated text and its type, and returns the transformed text. To register, provide a list of tuples of `[textType \| "*", transformFunction]`. | | `transportDestination` | `string \| null \| undefined` | no | Destination for generated audio frames. | | `reuseContextIdWithinTurn` | `boolean \| undefined` | no | Whether the service should reuse context IDs within the same turn. | | `audioCache` | `TTSAudioCacheStore \| null \| undefined` | no | Where to keep synthesized audio for reuse (e.g. a shared `TTSAudioCache`). Text whose audio is in the cache is served from it, with no request to the provider. Lines get there through `warmAudioCache()`, and also from live synthesis when `cacheSynthesizedAudio` allows it. Only services that yield their audio from `runTts` (HTTP and streaming-RPC services, see `supportsAudioCache`) use it, and only in sentence aggregation mode. Defaults to null (no cache). pipedog only. | | `cacheSynthesizedAudio` | `boolean \| ((text: string) => boolean) \| undefined` | no | Whether audio synthesized live is stored in `audioCache`, or a function that decides per text. Defaults to false: only warmed lines are stored, since a generated reply can repeat what the user said. pipedog only. | | `audioCacheNamespace` | `string \| undefined` | no | Extra part of the cache key, for constructor options that change the audio but are not in `settings`. pipedog only. | ### Usage [#usage-1] ```ts title="tts-http.ts" const httpTts = new InworldHttpTTSService({ apiKey: process.env.INWORLD_API_KEY ?? "", // false posts to /tts/v1/voice and gets the audio in one piece. streaming: true, settings: new InworldHttpTTSService.Settings({ voice: "Ashley", speakingRate: 1.1 }), }); ``` ## Notes [#notes] * Both output PCM at the pipeline's sample rate and push `TTSTextFrame`s aligned to Inworld's word timestamps. When no timestamps arrive, the text is pushed as a whole. * `InworldTTSService` lets the server decide when to flush buffered text (`autoMode`, on by default) and closes a context on interruption. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/inworld/package.json`: ✅ means the compat smoke test passes on that runtime. * `InworldTTSService` opens its socket with `websocketConnect()` from core, with the key in the `Authorization` header (through `fetch` on Cloudflare Workers). `InworldHttpTTSService` uses the global `fetch`; pass `fetch` to use another one (upstream takes an aiohttp session). * Deprecated options are not ported: `voiceId`, `model` and `params` (`InputParams`) on the constructor, and `aggregateSentences`. The WebSocket buffer settings that only `params` could set always use their defaults. # Kokoro > Local text-to-speech with KokoroTTSService, running the Kokoro-82M ONNX model on your machine with eSpeak NG phonemization. No API key. Source: https://polvallverdu.github.io/pipedog/docs/services/tts/kokoro/ ## Overview [#overview] * **`KokoroTTSService`**: synthesizes each sentence locally with [Kokoro-82M](https://huggingface.co/hexgrad/Kokoro-82M), an 82M-parameter model that runs on the CPU, and streams 24 kHz audio resampled to the pipeline's rate. No account, no API key. * **`Kokoro`**: the engine behind it, a TypeScript reimplementation of the Python [kokoro-onnx](https://github.com/thewh1teagle/kokoro-onnx) package (0.6.1) on `@pipedog/onnx` and `@pipedog/espeak-ng`. Its audio is bit-identical to kokoro-onnx's for the int8 model. Kokoro is one of upstream's keyless local defaults for [evals](https://polvallverdu.github.io/pipedog/docs/evals.md): it speaks the simulated user's turns in audio scenarios (`service: kokoro`). Pipecat's documentation for the same service. `packages/kokoro/src/tts.ts` and the engine, `kokoro_onnx.ts`. The model card, with its voices and languages. ## Installation [#installation] ```bash bun add @pipedog/kokoro onnxruntime-node @echogarden/espeak-ng-emscripten ``` * `onnxruntime-node` runs the model. `onnxruntime-web@1.30.0` (WebAssembly) works too, slower. * `@echogarden/espeak-ng-emscripten` is eSpeak NG compiled to WebAssembly, for phonemization. It is GPL-3.0 (see [Licenses](#licenses)). Both are optional peer dependencies, loaded on first use. ```ts import { KokoroTTSService } from "@pipedog/kokoro/tts"; ``` ## What you need [#what-you-need] The same files upstream uses, from kokoro-onnx's `model-files-v1.0` GitHub release. They are downloaded once, when the pipeline starts, if they are missing: | File | Size | Default location | | ----------------------------- | ------- | -------------------------- | | `kokoro-v1.0.onnx` | 310 MiB | `/kokoro-v1.0.onnx` | | `voices-v1.0.bin` (54 voices) | 27 MiB | `/voices-v1.0.bin` | `` is `$PIPEDOG_CACHE_DIR/kokoro`, else `$XDG_CACHE_HOME/pipedog/kokoro`, else `~/.cache/pipedog/kokoro`. Pass `cacheDir` to use another directory. To use files you already have, pass `modelPath` and `voicesPath`. A path that exists is used as is; a missing one is downloaded there from `modelUrl` / `voicesUrl`. The same release has smaller variants of the model, such as `kokoro-v1.0.int8.onnx` (88 MiB): ```ts // Files you already have, or the smaller int8 model from the same release. const int8 = new KokoroTTSService({ modelPath: "/models/kokoro-v1.0.int8.onnx", // used as is if present... modelUrl: "https://github.com/thewh1teagle/kokoro-onnx/releases/download/model-files-v1.0/kokoro-v1.0.int8.onnx", // ...else downloaded from here voicesPath: "/models/voices-v1.0.bin", settings: new KokoroTTSService.Settings({ voice: "af_heart" }), }); ``` eSpeak NG adds about 25 MB (the WebAssembly build and its voice data), loaded into memory once per process. `@pipedog/espeak-ng` wraps it: `phonemize()` reproduces the Python `phonemizer` package that kokoro-onnx uses, so the same text gives the same phonemes. ## Runtimes [#runtimes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Bun | ✅ yes | onnxruntime-node (or onnxruntime-web) and @echogarden/espeak-ng-emscripten; the Kokoro model (310 MB, or 88 MB int8) and voices (27 MB), downloaded on first use. | | Node.js | ✅ yes | onnxruntime-node (or onnxruntime-web) and @echogarden/espeak-ng-emscripten; the Kokoro model (310 MB, or 88 MB int8) and voices (27 MB), downloaded on first use. | | Deno | ✅ yes | onnxruntime-node (or onnxruntime-web) and @echogarden/espeak-ng-emscripten; the Kokoro model (310 MB, or 88 MB int8) and voices (27 MB), downloaded on first use. | | Cloudflare Workers | ❌ no | onnxruntime-node (or onnxruntime-web) and @echogarden/espeak-ng-emscripten; the Kokoro model (310 MB, or 88 MB int8) and voices (27 MB), downloaded on first use. Not supported: the model and eSpeak NG data exceed Workers' memory and bundle limits. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/kokoro/package.json`: ✅ means the compat smoke test passes on that runtime. Cloudflare Workers is not supported: the model (88-310 MiB), the voices and eSpeak NG's data don't fit a Worker's 128 MB of memory or its 10 MB bundle limit. ## KokoroTTSService [#kokorottsservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `textAggregationMode` | `TextAggregationMode \| null \| undefined` | no | How to aggregate incoming text before synthesis. `TextAggregationMode.SENTENCE` (default) buffers until sentence boundaries, `TextAggregationMode.TOKEN` streams tokens directly for lower latency. | | `modelPath` | `string \| null \| undefined` | no | Path to the kokoro ONNX model file. Defaults to the auto-downloaded file. | | `voicesPath` | `string \| null \| undefined` | no | Path to the voices binary file. Defaults to the auto-downloaded file. | | `cacheDir` | `string \| null \| undefined` | no | Directory for the auto-downloaded files (default `kokoroCacheDir()`). Not upstream. | | `modelUrl` | `string \| null \| undefined` | no | Where to download a missing model from (default `KOKORO_MODEL_URL`). Not upstream. | | `voicesUrl` | `string \| null \| undefined` | no | Where to download a missing voices file from (default `KOKORO_VOICES_URL`). Not upstream. | | `fetch` | `((input: RequestInfo \| URL, init?: RequestInit) => Promise) \| null \| undefined` | no | `fetch` for downloads (default: the global `fetch`). Not upstream. | | `backend` | `OnnxBackend \| Promise \| null \| undefined` | no | The ONNX backend (default: `createOnnxBackend()`). Not upstream. | | `shareSession` | `boolean \| undefined` | no | Share the model session and voices across services (default `true`). Not upstream. | | `settings` | `KokoroTTSSettings \| null \| undefined` | no | Runtime-updatable settings. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `pushTextFrames` | `boolean \| undefined` | no | Whether to push TextFrames and LLMFullResponseEndFrames. | | `pushStopFrames` | `boolean \| undefined` | no | Whether to automatically push TTSStoppedFrames. | | `pushStartFrame` | `boolean \| undefined` | no | Whether to automatically create audio contexts and push TTSStartedFrames. When true, the base class handles `createAudioContext` and yields `TTSStartedFrame` before each synthesis call, so `runTts` implementations do not need to. | | `stopFrameTimeoutS` | `number \| undefined` | no | Idle time before pushing TTSStoppedFrame when pushStopFrames is true. | | `pushSilenceAfterStop` | `boolean \| undefined` | no | Whether to push silence audio after TTSStoppedFrame. | | `silenceTimeS` | `number \| undefined` | no | Duration of silence to push when pushSilenceAfterStop is true. | | `pauseFrameProcessing` | `boolean \| undefined` | no | Whether to hold incoming frames after a turn until its audio contexts end. Needed by services whose audio doesn't carry its request's context ID. | | `maxConsecutiveZeroAudioContexts` | `number \| undefined` | no | How many consecutive TTS contexts may complete without producing any audio before the service is reported unable to do its job. Catches a provider that accepts requests and stays silent — an unknown voice ID, say — which no error ever surfaces. Every silent context reports an error the service can carry on from, whatever the limit is; on reaching the limit the service reports a permanent error instead, stops being given work, and the pipeline worker applies its `ProcessorUnusablePolicy`. Set to 0 to report silent contexts without ever writing the service off. | | `appendTrailingSpace` | `boolean \| undefined` | no | Whether to append a trailing space to text before sending to TTS. This helps prevent some TTS services from vocalizing trailing punctuation (e.g., "dot"). Only applied in sentence aggregation mode; when streaming tokens, the incoming text's own whitespace is preserved. | | `sampleRate` | `number \| null \| undefined` | no | Output sample rate for generated audio. | | `skipTextTypes` | `string[] \| null \| undefined` | no | List of text types that should not be spoken. | | `textTransforms` | `[string, TextTransform][] \| null \| undefined` | no | A list of TextTransforms to transform text just before sending it to TTS. Each TextTransform takes the aggregated text and its type, and returns the transformed text. To register, provide a list of tuples of `[textType \| "*", transformFunction]`. | | `transportDestination` | `string \| null \| undefined` | no | Destination for generated audio frames. | | `reuseContextIdWithinTurn` | `boolean \| undefined` | no | Whether the service should reuse context IDs within the same turn. | | `audioCache` | `TTSAudioCacheStore \| null \| undefined` | no | Where to keep synthesized audio for reuse (e.g. a shared `TTSAudioCache`). Text whose audio is in the cache is served from it, with no request to the provider. Lines get there through `warmAudioCache()`, and also from live synthesis when `cacheSynthesizedAudio` allows it. Only services that yield their audio from `runTts` (HTTP and streaming-RPC services, see `supportsAudioCache`) use it, and only in sentence aggregation mode. Defaults to null (no cache). pipedog only. | | `cacheSynthesizedAudio` | `boolean \| ((text: string) => boolean) \| undefined` | no | Whether audio synthesized live is stored in `audioCache`, or a function that decides per text. Defaults to false: only warmed lines are stored, since a generated reply can repeat what the user said. pipedog only. | | `audioCacheNamespace` | `string \| undefined` | no | Extra part of the cache key, for constructor options that change the audio but are not in `settings`. pipedog only. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new KokoroTTSService.Settings({...})` (`KokoroTTSSettings`). Any setting you leave out keeps the default below. | Setting | Type | Default | Description | | ---------- | ---------- | -------------- | ----------------------------------------------------- | | `model` | `null` | `null` | Not used: the model is `modelPath`. | | `voice` | `string` | none, required | Voice from the voices file, for example `"af_heart"`. | | `language` | `Language` | `Language.EN` | Language, mapped to an espeak-ng code. | | `speed` | `number` | `1.0` | Speech rate multiplier, 0.5 to 2.0. | Languages: English (US and GB), Spanish, French, Hindi, Italian, Japanese, Portuguese (PT and BR) and Mandarin. Pick a voice for the language: `af_*` / `am_*` American English, `bf_*` / `bm_*` British, `ef_*` Spanish, `ff_*` French, `hf_*` Hindi, `if_*` Italian, `jf_*` Japanese, `pf_*` Portuguese, `zf_*` Mandarin. ### Usage [#usage] ```ts title="tts.ts" import { TTSUpdateSettingsFrame } from "@pipedog/core/frames/frames"; import { Language } from "@pipedog/core/transcriptions/language"; import { KokoroTTSService } from "@pipedog/kokoro/tts"; // Downloads kokoro-v1.0.onnx (310 MiB) and voices-v1.0.bin (27 MiB) on first start. const tts = new KokoroTTSService({ settings: new KokoroTTSService.Settings({ voice: "af_heart", language: Language.EN, speed: 1.0 }), }); ``` Change the voice, language or speed mid-call with a `TTSUpdateSettingsFrame`: ```ts // Queue it on the worker, or push it from a processor upstream of the service. const update = new TTSUpdateSettingsFrame({ delta: new KokoroTTSService.Settings({ voice: "bf_emma", language: Language.EN_GB, speed: 1.1 }), }); ``` The engine works on its own too: ```ts import { Kokoro } from "@pipedog/kokoro/kokoro_onnx"; const kokoro = await Kokoro.load("/models/kokoro-v1.0.onnx", "/models/voices-v1.0.bin"); for await (const [samples, sampleRate] of kokoro.createStream("Hello there!", { voice: "af_heart", lang: "en-us", })) { // samples: a Float32Array at sampleRate (24000 Hz), one chunk per phoneme batch } ``` ### Notes [#notes] * The model and voices load in `start()` (or on the first sentence). Services in one process share one model session and one copy of the voices; set `shareSession: false` to opt out. * A missing or blank voice is rejected by the constructor. A voice that isn't in the voices file is reported when the model loads. * Text is phonemized with espeak-ng, split into batches of at most 510 phonemes at sentence, clause and word boundaries, and the next batch is synthesized while the current one plays. * `backend` takes any `OnnxBackend`, for example one built on `onnxruntime-web`. ## Differences from upstream [#differences-from-upstream] * The model is downloaded and loaded in `start()`, not in the constructor, since loading is asynchronous. * Default cache directory: `~/.cache/pipedog/kokoro`, where upstream uses `~/.cache/pipecat/kokoro-onnx`. * `cacheDir`, `modelUrl`, `voicesUrl`, `fetch`, `backend` and `shareSession` are pipedog options. * The deprecated `voice_id` and `params` (`InputParams`) arguments are not ported. Use `settings`. * Not ported from kokoro-onnx: timed synthesis (`create_timed`), which needs a re-exported model. ## Licenses [#licenses] * `@pipedog/kokoro`: BSD 2-Clause. * Kokoro-82M model and voices: Apache-2.0 (hexgrad). The ONNX export and voice pack come from kokoro-onnx (MIT). * eSpeak NG and `@echogarden/espeak-ng-emscripten`: **GPL-3.0**. Distributing an application that includes them may subject it to the GPL's terms. Upstream is in the same position: kokoro-onnx ships espeak-ng through espeakng-loader. # MiniMax > Text-to-speech with MiniMax's T2A API: MiniMaxHttpTTSService streams PCM audio over HTTP. Source: https://polvallverdu.github.io/pipedog/docs/services/tts/minimax/ ## Overview [#overview] `MiniMaxHttpTTSService` posts each sentence to MiniMax's T2A (Text-to-Audio) endpoint with streaming on, and pushes the hex-encoded PCM audio of each streamed `data:` block as it arrives. Pipecat's documentation for the same service. `packages/minimax/src/tts.ts`. The T2A HTTP API reference. ## Installation [#installation] ```bash bun add @pipedog/minimax ``` ```ts import { MiniMaxHttpTTSService } from "@pipedog/minimax/tts"; ``` ## Prerequisites [#prerequisites] * A MiniMax account, an API key and your group id. * Pass them as `apiKey` and `groupId`. The examples read them from `MINIMAX_API_KEY` and `MINIMAX_GROUP_ID`. ## Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `sampleRate` | `number \| null \| undefined` | no | Output sample rate for generated audio. | | `skipTextTypes` | `string[] \| null \| undefined` | no | List of text types that should not be spoken. | | `textTransforms` | `[string, TextTransform][] \| null \| undefined` | no | A list of TextTransforms to transform text just before sending it to TTS. Each TextTransform takes the aggregated text and its type, and returns the transformed text. To register, provide a list of tuples of `[textType \| "*", transformFunction]`. | | `transportDestination` | `string \| null \| undefined` | no | Destination for generated audio frames. | | `apiKey` | `string` | yes | MiniMax API key for authentication. | | `baseUrl` | `string \| undefined` | no | API base URL, defaults to MiniMax's T2A endpoint. - Global: https://api.minimax.io/v1/t2a_v2 - Mainland China: https://api.minimaxi.chat/v1/t2a_v2 - Western United States: https://api-uw.minimax.io/v1/t2a_v2 | | `groupId` | `string` | yes | MiniMax Group ID to identify project. | | `fetch` | `{ (input: RequestInfo \| URL, init?: RequestInit): Promise; (input: string \| URL \| Request, init?: RequestInit): Promise; } \| null \| undefined` | no | `fetch` used for HTTP requests. Defaults to the global `fetch`. DIVERGENCE: upstream takes a required shared `aiohttp_session`. | | `stream` | `boolean \| undefined` | no | Whether to use streaming mode. Defaults to true. | | `settings` | `MiniMaxTTSSettings \| null \| undefined` | no | Runtime-updatable settings. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `textAggregationMode` | `TextAggregationMode \| null \| undefined` | no | How to aggregate incoming text before synthesis. `TextAggregationMode.SENTENCE` (default) buffers until sentence boundaries, `TextAggregationMode.TOKEN` streams tokens directly for lower latency. | | `pushTextFrames` | `boolean \| undefined` | no | Whether to push TextFrames and LLMFullResponseEndFrames. | | `pushStopFrames` | `boolean \| undefined` | no | Whether to automatically push TTSStoppedFrames. | | `pushStartFrame` | `boolean \| undefined` | no | Whether to automatically create audio contexts and push TTSStartedFrames. When true, the base class handles `createAudioContext` and yields `TTSStartedFrame` before each synthesis call, so `runTts` implementations do not need to. | | `stopFrameTimeoutS` | `number \| undefined` | no | Idle time before pushing TTSStoppedFrame when pushStopFrames is true. | | `pushSilenceAfterStop` | `boolean \| undefined` | no | Whether to push silence audio after TTSStoppedFrame. | | `silenceTimeS` | `number \| undefined` | no | Duration of silence to push when pushSilenceAfterStop is true. | | `pauseFrameProcessing` | `boolean \| undefined` | no | Whether to hold incoming frames after a turn until its audio contexts end. Needed by services whose audio doesn't carry its request's context ID. | | `maxConsecutiveZeroAudioContexts` | `number \| undefined` | no | How many consecutive TTS contexts may complete without producing any audio before the service is reported unable to do its job. Catches a provider that accepts requests and stays silent — an unknown voice ID, say — which no error ever surfaces. Every silent context reports an error the service can carry on from, whatever the limit is; on reaching the limit the service reports a permanent error instead, stops being given work, and the pipeline worker applies its `ProcessorUnusablePolicy`. Set to 0 to report silent contexts without ever writing the service off. | | `appendTrailingSpace` | `boolean \| undefined` | no | Whether to append a trailing space to text before sending to TTS. This helps prevent some TTS services from vocalizing trailing punctuation (e.g., "dot"). Only applied in sentence aggregation mode; when streaming tokens, the incoming text's own whitespace is preserved. | | `reuseContextIdWithinTurn` | `boolean \| undefined` | no | Whether the service should reuse context IDs within the same turn. | | `audioCache` | `TTSAudioCacheStore \| null \| undefined` | no | Where to keep synthesized audio for reuse (e.g. a shared `TTSAudioCache`). Text whose audio is in the cache is served from it, with no request to the provider. Lines get there through `warmAudioCache()`, and also from live synthesis when `cacheSynthesizedAudio` allows it. Only services that yield their audio from `runTts` (HTTP and streaming-RPC services, see `supportsAudioCache`) use it, and only in sentence aggregation mode. Defaults to null (no cache). pipedog only. | | `cacheSynthesizedAudio` | `boolean \| ((text: string) => boolean) \| undefined` | no | Whether audio synthesized live is stored in `audioCache`, or a function that decides per text. Defaults to false: only warmed lines are stored, since a generated reply can repeat what the user said. pipedog only. | | `audioCacheNamespace` | `string \| undefined` | no | Extra part of the cache key, for constructor options that change the audio but are not in `settings`. pipedog only. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new MiniMaxHttpTTSService.Settings({...})` (`MiniMaxTTSSettings`). Any setting you leave out keeps the default below. | Setting | Type | Default | Description | | ------------------- | ------------------ | -------------------- | ------------------------------------------------------------------------------------------------- | | `model` | `string` | `"speech-2.8-turbo"` | Speech model. | | `voice` | `string` | `"Calm_Woman"` | Voice id. | | `language` | `Language \| null` | `null` | Not sent. Use `languageBoost`. | | `speed` | `number \| null` | `1.0` | Speed, 0.5 to 2.0. | | `volume` | `number \| null` | `1.0` | Volume, above 0 up to 10. | | `pitch` | `number \| null` | `0` | Pitch, -12 to 12. | | `emotion` | `string \| null` | `null` | Emotional tone: `happy`, `sad`, `angry`, `fearful`, `disgusted`, `surprised`, `calm` or `fluent`. | | `textNormalization` | `boolean \| null` | `null` | Text normalization (Chinese and English). | | `latexRead` | `boolean \| null` | `null` | Read LaTeX formulas. | | `languageBoost` | `string \| null` | `null` | A language name to boost, such as `English`. `languageToMinimaxLanguage()` maps a `Language`. | ## Usage [#usage] ```ts title="tts.ts" import { TTSUpdateSettingsFrame } from "@pipedog/core/frames/frames"; import { MiniMaxHttpTTSService } from "@pipedog/minimax/tts"; const tts = new MiniMaxHttpTTSService({ apiKey: process.env.MINIMAX_API_KEY ?? "", groupId: process.env.MINIMAX_GROUP_ID ?? "", settings: new MiniMaxHttpTTSService.Settings({ model: "speech-2.8-turbo", voice: "Calm_Woman", emotion: "calm", }), }); ``` Change settings mid-call with a `TTSUpdateSettingsFrame`: ```ts // Queue it on the worker, or push it from a processor upstream of the service. const update = new TTSUpdateSettingsFrame({ delta: new MiniMaxHttpTTSService.Settings({ speed: 1.2 }), }); ``` ## Notes [#notes] * `baseUrl` picks the region: `https://api.minimax.io/v1/t2a_v2` (global, the default), `https://api.minimaxi.chat/v1/t2a_v2` (mainland China) or `https://api-uw.minimax.io/v1/t2a_v2` (western United States). The group id is added as `?GroupId=`. * The last streamed block repeats the whole utterance and carries `extra_info`. It is skipped, so no audio is played twice. * The audio is 16-bit PCM at the pipeline's sample rate, one channel. * `MiniMaxTTSSettings.fromMapping()` also accepts the older nested `voice_setting` dict (with `vol` for `volume`). ## Runtime notes [#runtime-notes] * It uses the global `fetch`, so it works on Bun, Node, Deno and Cloudflare Workers. Pass `fetch` to use another one (upstream takes an aiohttp session). * The deprecated `params` (`InputParams`), `model` and `voiceId` constructor options are not ported. Use `settings`. * Tested against a local server, not against MiniMax itself. | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/minimax/package.json`: ✅ means the compat smoke test passes on that runtime. # Mistral > Text-to-speech with MistralTTSService, Mistral's Voxtral TTS streamed over Server-Sent Events. Source: https://polvallverdu.github.io/pipedog/docs/services/tts/mistral/ ## Overview [#overview] * **`MistralTTSService`**: posts each sentence to Mistral's speech API with streaming on, converts the float32 PCM chunks to 16-bit, and resamples them from 24 kHz to the pipeline's rate. Pipecat's documentation for the same service. `packages/mistral/src/tts.ts`. Voxtral and the audio APIs. ## Installation [#installation] ```bash bun add @pipedog/mistral ``` ```ts import { MistralTTSService } from "@pipedog/mistral/tts"; ``` ## Prerequisites [#prerequisites] * A Mistral account and an API key from [Mistral](https://docs.mistral.ai/api/). * Pass the key as `apiKey`. The examples read it from `MISTRAL_API_KEY`. * A voice ID for `voice`: the default is `null`. ## MistralTTSService [#mistralttsservice] Posts to `https://api.mistral.ai/v1/audio/speech`. ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string \| null \| undefined` | no | Mistral API key for authentication. | | `baseUrl` | `string \| null \| undefined` | no | Base URL of Mistral's API. Defaults to `"https://api.mistral.ai/v1"`. | | `settings` | `MistralTTSSettings \| null \| undefined` | no | Runtime-updatable settings. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `textAggregationMode` | `TextAggregationMode \| null \| undefined` | no | How to aggregate incoming text before synthesis. `TextAggregationMode.SENTENCE` (default) buffers until sentence boundaries, `TextAggregationMode.TOKEN` streams tokens directly for lower latency. | | `pushTextFrames` | `boolean \| undefined` | no | Whether to push TextFrames and LLMFullResponseEndFrames. | | `pushStopFrames` | `boolean \| undefined` | no | Whether to automatically push TTSStoppedFrames. | | `pushStartFrame` | `boolean \| undefined` | no | Whether to automatically create audio contexts and push TTSStartedFrames. When true, the base class handles `createAudioContext` and yields `TTSStartedFrame` before each synthesis call, so `runTts` implementations do not need to. | | `stopFrameTimeoutS` | `number \| undefined` | no | Idle time before pushing TTSStoppedFrame when pushStopFrames is true. | | `pushSilenceAfterStop` | `boolean \| undefined` | no | Whether to push silence audio after TTSStoppedFrame. | | `silenceTimeS` | `number \| undefined` | no | Duration of silence to push when pushSilenceAfterStop is true. | | `pauseFrameProcessing` | `boolean \| undefined` | no | Whether to hold incoming frames after a turn until its audio contexts end. Needed by services whose audio doesn't carry its request's context ID. | | `maxConsecutiveZeroAudioContexts` | `number \| undefined` | no | How many consecutive TTS contexts may complete without producing any audio before the service is reported unable to do its job. Catches a provider that accepts requests and stays silent — an unknown voice ID, say — which no error ever surfaces. Every silent context reports an error the service can carry on from, whatever the limit is; on reaching the limit the service reports a permanent error instead, stops being given work, and the pipeline worker applies its `ProcessorUnusablePolicy`. Set to 0 to report silent contexts without ever writing the service off. | | `appendTrailingSpace` | `boolean \| undefined` | no | Whether to append a trailing space to text before sending to TTS. This helps prevent some TTS services from vocalizing trailing punctuation (e.g., "dot"). Only applied in sentence aggregation mode; when streaming tokens, the incoming text's own whitespace is preserved. | | `sampleRate` | `number \| null \| undefined` | no | Output sample rate for generated audio. | | `skipTextTypes` | `string[] \| null \| undefined` | no | List of text types that should not be spoken. | | `textTransforms` | `[string, TextTransform][] \| null \| undefined` | no | A list of TextTransforms to transform text just before sending it to TTS. Each TextTransform takes the aggregated text and its type, and returns the transformed text. To register, provide a list of tuples of `[textType \| "*", transformFunction]`. | | `transportDestination` | `string \| null \| undefined` | no | Destination for generated audio frames. | | `reuseContextIdWithinTurn` | `boolean \| undefined` | no | Whether the service should reuse context IDs within the same turn. | | `audioCache` | `TTSAudioCacheStore \| null \| undefined` | no | Where to keep synthesized audio for reuse (e.g. a shared `TTSAudioCache`). Text whose audio is in the cache is served from it, with no request to the provider. Lines get there through `warmAudioCache()`, and also from live synthesis when `cacheSynthesizedAudio` allows it. Only services that yield their audio from `runTts` (HTTP and streaming-RPC services, see `supportsAudioCache`) use it, and only in sentence aggregation mode. Defaults to null (no cache). pipedog only. | | `cacheSynthesizedAudio` | `boolean \| ((text: string) => boolean) \| undefined` | no | Whether audio synthesized live is stored in `audioCache`, or a function that decides per text. Defaults to false: only warmed lines are stored, since a generated reply can repeat what the user said. pipedog only. | | `audioCacheNamespace` | `string \| undefined` | no | Extra part of the cache key, for constructor options that change the audio but are not in `settings`. pipedog only. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new MistralTTSService.Settings({...})` (`MistralTTSSettings`). Any setting you leave out keeps the default below. | Setting | Type | Default | Description | | ---------- | ---------------- | ------------------------- | ----------------------------- | | `model` | `string` | `"voxtral-mini-tts-2603"` | Speech model. | | `voice` | `string \| null` | `null` | Voice ID, sent as `voice_id`. | | `language` | `null` | `null` | Not used by this service. | ### Usage [#usage] ```ts title="tts.ts" import { TTSUpdateSettingsFrame } from "@pipedog/core/frames/frames"; import { MistralTTSService } from "@pipedog/mistral/tts"; const tts = new MistralTTSService({ apiKey: process.env.MISTRAL_API_KEY ?? "", settings: new MistralTTSService.Settings({ voice: process.env.MISTRAL_VOICE_ID ?? "" }), }); ``` Change settings mid-call with an `TTSUpdateSettingsFrame`: ```ts // Queue it on the worker, or push it from a processor upstream of the service. const update = new TTSUpdateSettingsFrame({ delta: new MistralTTSService.Settings({ model: "voxtral-mini-tts-2603" }), }); ``` ### Notes [#notes] * Mistral returns 24 kHz audio, resampled to `sampleRate` when that differs. * A failed request produces an `ErrorFrame` instead of audio. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/mistral/package.json`: ✅ means the compat smoke test passes on that runtime. * Requests go through `fetch`, with the Server-Sent Events parsed in the service (upstream uses the `mistralai` SDK). It works on Bun, Node, Deno and Cloudflare Workers. * The request and events follow Mistral's SDK (`@mistralai/mistralai` 3.1.0, `audioSpeechComplete`) and the [streaming docs](https://docs.mistral.ai/studio/audio/text-to-speech/stream-speech): `speech.audio.delta` events with base64 `audio_data`, then `speech.audio.done`. The float32 layout of `"pcm"` audio comes from upstream Pipecat's conversion; Mistral's docs don't state it. * Tested against a local fake, not against Mistral itself. # Neuphonic > Text-to-speech with Neuphonic: NeuphonicTTSService over WebSocket and NeuphonicHttpTTSService over server-sent events. Source: https://polvallverdu.github.io/pipedog/docs/services/tts/neuphonic/ ## Overview [#overview] * **`NeuphonicTTSService`**: keeps one WebSocket open, sends each sentence followed by ``, and pushes the audio as it arrives. When the bot is interrupted while speaking, it reconnects (the service is an `InterruptibleTTSService`). * **`NeuphonicHttpTTSService`**: one HTTP request per sentence, read as server-sent events. Pipecat's documentation for the same service. `packages/neuphonic/src/tts.ts`. The Neuphonic documentation. ## Installation [#installation] ```bash bun add @pipedog/neuphonic ``` ```ts import { NeuphonicHttpTTSService, NeuphonicTTSService } from "@pipedog/neuphonic/tts"; ``` ## Prerequisites [#prerequisites] * A Neuphonic account and an API key. * Pass the key as `apiKey`. The examples read it from `NEUPHONIC_API_KEY`. ## NeuphonicTTSService [#neuphonicttsservice] Connects to `wss://api.neuphonic.com/speak/`. ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | Neuphonic API key for authentication. | | `url` | `string \| undefined` | no | WebSocket URL for the Neuphonic API. | | `encoding` | `string \| undefined` | no | Audio encoding format. Defaults to "pcm_linear". | | `settings` | `NeuphonicTTSSettings \| null \| undefined` | no | Runtime-updatable settings. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `reconnectBackoffMinWait` | `number \| undefined` | no | Minimum time, in seconds, to wait between reconnection attempts. | | `reconnectBackoffMaxWait` | `number \| undefined` | no | Maximum time, in seconds, to wait between reconnection attempts. | | `reconnectOnError` | `boolean \| undefined` | no | Whether to automatically reconnect on connection errors. | | `wsCloseTimeout` | `number \| undefined` | no | Maximum time, in seconds, to wait for the peer to acknowledge the websocket closing handshake before dropping the connection. Applied to connections opened through `_websocketConnect()`. Increase it for peers that need longer to complete a graceful close. | | `wsSendTimeout` | `number \| null \| undefined` | no | Maximum time, in seconds, a single send may wait on a peer that has stopped reading before the connection is dropped and the send throws `ConnectionClosedError`. Applied to connections opened through `_websocketConnect()`. Pass `null` to wait indefinitely. Defaults to `WS_SEND_TIMEOUT`. | | `textAggregationMode` | `TextAggregationMode \| null \| undefined` | no | How to aggregate incoming text before synthesis. `TextAggregationMode.SENTENCE` (default) buffers until sentence boundaries, `TextAggregationMode.TOKEN` streams tokens directly for lower latency. | | `pushTextFrames` | `boolean \| undefined` | no | Whether to push TextFrames and LLMFullResponseEndFrames. | | `pushStopFrames` | `boolean \| undefined` | no | Whether to automatically push TTSStoppedFrames. | | `pushStartFrame` | `boolean \| undefined` | no | Whether to automatically create audio contexts and push TTSStartedFrames. When true, the base class handles `createAudioContext` and yields `TTSStartedFrame` before each synthesis call, so `runTts` implementations do not need to. | | `stopFrameTimeoutS` | `number \| undefined` | no | Idle time before pushing TTSStoppedFrame when pushStopFrames is true. | | `pushSilenceAfterStop` | `boolean \| undefined` | no | Whether to push silence audio after TTSStoppedFrame. | | `silenceTimeS` | `number \| undefined` | no | Duration of silence to push when pushSilenceAfterStop is true. | | `pauseFrameProcessing` | `boolean \| undefined` | no | Whether to hold incoming frames after a turn until its audio contexts end. Needed by services whose audio doesn't carry its request's context ID. | | `maxConsecutiveZeroAudioContexts` | `number \| undefined` | no | How many consecutive TTS contexts may complete without producing any audio before the service is reported unable to do its job. Catches a provider that accepts requests and stays silent — an unknown voice ID, say — which no error ever surfaces. Every silent context reports an error the service can carry on from, whatever the limit is; on reaching the limit the service reports a permanent error instead, stops being given work, and the pipeline worker applies its `ProcessorUnusablePolicy`. Set to 0 to report silent contexts without ever writing the service off. | | `appendTrailingSpace` | `boolean \| undefined` | no | Whether to append a trailing space to text before sending to TTS. This helps prevent some TTS services from vocalizing trailing punctuation (e.g., "dot"). Only applied in sentence aggregation mode; when streaming tokens, the incoming text's own whitespace is preserved. | | `sampleRate` | `number \| null \| undefined` | no | Output sample rate for generated audio. | | `skipTextTypes` | `string[] \| null \| undefined` | no | List of text types that should not be spoken. | | `textTransforms` | `[string, TextTransform][] \| null \| undefined` | no | A list of TextTransforms to transform text just before sending it to TTS. Each TextTransform takes the aggregated text and its type, and returns the transformed text. To register, provide a list of tuples of `[textType \| "*", transformFunction]`. | | `transportDestination` | `string \| null \| undefined` | no | Destination for generated audio frames. | | `reuseContextIdWithinTurn` | `boolean \| undefined` | no | Whether the service should reuse context IDs within the same turn. | | `audioCache` | `TTSAudioCacheStore \| null \| undefined` | no | Where to keep synthesized audio for reuse (e.g. a shared `TTSAudioCache`). Text whose audio is in the cache is served from it, with no request to the provider. Lines get there through `warmAudioCache()`, and also from live synthesis when `cacheSynthesizedAudio` allows it. Only services that yield their audio from `runTts` (HTTP and streaming-RPC services, see `supportsAudioCache`) use it, and only in sentence aggregation mode. Defaults to null (no cache). pipedog only. | | `cacheSynthesizedAudio` | `boolean \| ((text: string) => boolean) \| undefined` | no | Whether audio synthesized live is stored in `audioCache`, or a function that decides per text. Defaults to false: only warmed lines are stored, since a generated reply can repeat what the user said. pipedog only. | | `audioCacheNamespace` | `string \| undefined` | no | Extra part of the cache key, for constructor options that change the audio but are not in `settings`. pipedog only. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new NeuphonicTTSService.Settings({...})` (`NeuphonicTTSSettings`). Any setting you leave out keeps the default below. A change reconnects the WebSocket, because the settings are query parameters of the connection. | Setting | Type | Default | Description | | ------------- | ------------------ | ------------- | --------------------------------------------------------------------- | | `voice` | `string \| null` | `null` | Voice id. | | `language` | `Language \| null` | `Language.EN` | Language, part of the connection's path. | | `speed` | `number` | `1.0` | Speed multiplier. | | `temperature` | `number \| null` | `null` | Randomness, 0.0 to 1.0. `null` leaves Neuphonic's default. | | `ipaKey` | `string \| null` | `null` | Regional variant for `` tags: `es-castilian` or `pt-br`. | ### Usage [#usage] ```ts title="tts.ts" import { TTSUpdateSettingsFrame } from "@pipedog/core/frames/frames"; import { Language } from "@pipedog/core/transcriptions/language"; import { NeuphonicHttpTTSService, NeuphonicTTSService } from "@pipedog/neuphonic/tts"; const tts = new NeuphonicTTSService({ apiKey: process.env.NEUPHONIC_API_KEY ?? "", settings: new NeuphonicTTSService.Settings({ voice: "your-voice-id", language: Language.EN, speed: 1.0, }), }); ``` Change settings mid-call with a `TTSUpdateSettingsFrame`: ```ts // Reconnects the WebSocket with the new settings. const update = new TTSUpdateSettingsFrame({ delta: new NeuphonicTTSService.Settings({ speed: 1.2 }), }); ``` ### Event handlers [#event-handlers] | Event | Description | | --------------------- | ------------------------ | | `on_connected` | The WebSocket connected. | | `on_disconnected` | The WebSocket closed. | | `on_connection_error` | The connection failed. | ## NeuphonicHttpTTSService [#neuphonichttpttsservice] Posts each sentence to `https://api.neuphonic.com/sse/speak/`. ### Configuration [#configuration-1] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | Neuphonic API key for authentication. | | `fetch` | `{ (input: RequestInfo \| URL, init?: RequestInit): Promise; (input: string \| URL \| Request, init?: RequestInit): Promise; } \| null \| undefined` | no | `fetch` used for HTTP requests. Defaults to the global `fetch`. DIVERGENCE: upstream takes a required shared `aiohttp_session`. | | `url` | `string \| undefined` | no | Base URL for the Neuphonic HTTP API. | | `encoding` | `string \| null \| undefined` | no | Audio encoding format. Defaults to "pcm_linear". | | `settings` | `NeuphonicTTSSettings \| null \| undefined` | no | Runtime-updatable settings. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `textAggregationMode` | `TextAggregationMode \| null \| undefined` | no | How to aggregate incoming text before synthesis. `TextAggregationMode.SENTENCE` (default) buffers until sentence boundaries, `TextAggregationMode.TOKEN` streams tokens directly for lower latency. | | `pushTextFrames` | `boolean \| undefined` | no | Whether to push TextFrames and LLMFullResponseEndFrames. | | `pushStopFrames` | `boolean \| undefined` | no | Whether to automatically push TTSStoppedFrames. | | `pushStartFrame` | `boolean \| undefined` | no | Whether to automatically create audio contexts and push TTSStartedFrames. When true, the base class handles `createAudioContext` and yields `TTSStartedFrame` before each synthesis call, so `runTts` implementations do not need to. | | `stopFrameTimeoutS` | `number \| undefined` | no | Idle time before pushing TTSStoppedFrame when pushStopFrames is true. | | `pushSilenceAfterStop` | `boolean \| undefined` | no | Whether to push silence audio after TTSStoppedFrame. | | `silenceTimeS` | `number \| undefined` | no | Duration of silence to push when pushSilenceAfterStop is true. | | `pauseFrameProcessing` | `boolean \| undefined` | no | Whether to hold incoming frames after a turn until its audio contexts end. Needed by services whose audio doesn't carry its request's context ID. | | `maxConsecutiveZeroAudioContexts` | `number \| undefined` | no | How many consecutive TTS contexts may complete without producing any audio before the service is reported unable to do its job. Catches a provider that accepts requests and stays silent — an unknown voice ID, say — which no error ever surfaces. Every silent context reports an error the service can carry on from, whatever the limit is; on reaching the limit the service reports a permanent error instead, stops being given work, and the pipeline worker applies its `ProcessorUnusablePolicy`. Set to 0 to report silent contexts without ever writing the service off. | | `appendTrailingSpace` | `boolean \| undefined` | no | Whether to append a trailing space to text before sending to TTS. This helps prevent some TTS services from vocalizing trailing punctuation (e.g., "dot"). Only applied in sentence aggregation mode; when streaming tokens, the incoming text's own whitespace is preserved. | | `sampleRate` | `number \| null \| undefined` | no | Output sample rate for generated audio. | | `skipTextTypes` | `string[] \| null \| undefined` | no | List of text types that should not be spoken. | | `textTransforms` | `[string, TextTransform][] \| null \| undefined` | no | A list of TextTransforms to transform text just before sending it to TTS. Each TextTransform takes the aggregated text and its type, and returns the transformed text. To register, provide a list of tuples of `[textType \| "*", transformFunction]`. | | `transportDestination` | `string \| null \| undefined` | no | Destination for generated audio frames. | | `reuseContextIdWithinTurn` | `boolean \| undefined` | no | Whether the service should reuse context IDs within the same turn. | | `audioCache` | `TTSAudioCacheStore \| null \| undefined` | no | Where to keep synthesized audio for reuse (e.g. a shared `TTSAudioCache`). Text whose audio is in the cache is served from it, with no request to the provider. Lines get there through `warmAudioCache()`, and also from live synthesis when `cacheSynthesizedAudio` allows it. Only services that yield their audio from `runTts` (HTTP and streaming-RPC services, see `supportsAudioCache`) use it, and only in sentence aggregation mode. Defaults to null (no cache). pipedog only. | | `cacheSynthesizedAudio` | `boolean \| ((text: string) => boolean) \| undefined` | no | Whether audio synthesized live is stored in `audioCache`, or a function that decides per text. Defaults to false: only warmed lines are stored, since a generated reply can repeat what the user said. pipedog only. | | `audioCacheNamespace` | `string \| undefined` | no | Extra part of the cache key, for constructor options that change the audio but are not in `settings`. pipedog only. | It takes the same settings as the WebSocket service. ### Usage [#usage-1] ```ts const httpTts = new NeuphonicHttpTTSService({ apiKey: process.env.NEUPHONIC_API_KEY ?? "", settings: new NeuphonicHttpTTSService.Settings({ voice: "your-voice-id" }), }); ``` ## Notes [#notes] * The sample rate defaults to 22050 Hz and the encoding to `pcm_linear`. * The WebSocket sends a keepalive (`{"text": ""}`) every 10 seconds. * An SSE event that carries `errors` is reported as an error frame, and the stream keeps going. ## Runtime notes [#runtime-notes] * The WebSocket opens through the core `websocketConnect`, with the key in an `x-api-key` header (through `fetch` on Cloudflare Workers). The HTTP service uses the global `fetch`; pass `fetch` to use another one (upstream takes an aiohttp session). Both work on Bun, Node, Deno and Cloudflare Workers. * Upstream talks to Neuphonic's API directly rather than through the `pyneuphonic` SDK, and so does this port. * The deprecated `params` (`InputParams`), `voiceId` and `aggregateSentences` options are not ported. * Upstream has no Neuphonic tests. Ours run against local fake servers, not against Neuphonic itself. | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/neuphonic/package.json`: ✅ means the compat smoke test passes on that runtime. # NVIDIA Riva > Speech synthesis from NVIDIA Riva. Not ported to pipedog yet. Source: https://polvallverdu.github.io/pipedog/docs/services/tts/nvidia/ {/* Generated by scripts/generate-providers.ts from data/providers.ts. */} NVIDIA Riva isn't ported to pipedog yet. It is available in Pipecat (Python), and its pipedog port will keep the same class names and options. [Ask for it, or upvote the request](https://github.com/polvallverdu/pipedog/issues?q=is%3Aissue+NVIDIA+Riva). ## What it is [#what-it-is] Speech synthesis from NVIDIA Riva. Upstream, it's this class: `NvidiaTTSService`, in [`services/nvidia/tts.py`](https://github.com/pipecat-ai/pipecat/blob/main/src/pipecat/services/nvidia/tts.py). Its pipedog package will be `@pipedog/nvidia`. * [Pipecat's NVIDIA Riva page](https://docs.pipecat.ai/api-reference/server/services/tts/nvidia) ## Get it ported [#get-it-ported] * [Search the issues](https://github.com/polvallverdu/pipedog/issues?q=is%3Aissue+NVIDIA+Riva) for a request, and upvote it with a reaction. * If there's none, [open a port request](https://github.com/polvallverdu/pipedog/issues/new?template=feature_request.yml\&title=Port+NVIDIA+Riva+%28NvidiaTTSService%29\&kind=Port+an+upstream+module\&upstream=src%2Fpipecat%2Fservices%2Fnvidia%2Ftts.py). * Or port it yourself: the [contributing guide](https://polvallverdu.github.io/pipedog/docs/contributing.md) explains how modules are ported, test by test. The [porting status](https://polvallverdu.github.io/pipedog/docs/contributing/status.md) lists what's done. Until then, see what is ported in [Text-to-speech](https://polvallverdu.github.io/pipedog/docs/services/tts.md). # OpenAI > Text-to-speech with OpenAITTSService, streaming 24 kHz PCM from OpenAI's speech API. Source: https://polvallverdu.github.io/pipedog/docs/services/tts/openai/ ## Overview [#overview] * **`OpenAITTSService`**: sends each sentence to OpenAI's speech API (`audio.speech`) and streams the reply back as raw 24 kHz PCM. Supports the `gpt-4o-mini-tts` instructions and speed controls. Pipecat's documentation for the same service. `packages/openai/src/tts.ts`. Models, voices and instructions. ## Installation [#installation] ```bash bun add @pipedog/openai ``` ```ts import { OpenAITTSService } from "@pipedog/openai/tts"; ``` ## Prerequisites [#prerequisites] * An OpenAI account and an API key from the [OpenAI dashboard](https://platform.openai.com/api-keys). * Pass the key as `apiKey`. If you leave it out, the OpenAI SDK reads `OPENAI_API_KEY`. The examples read it from `OPENAI_API_KEY`. ## OpenAITTSService [#openaittsservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string \| null \| undefined` | no | OpenAI API key for authentication. If null, uses environment variable. | | `baseUrl` | `string \| null \| undefined` | no | Custom base URL for OpenAI API. If null, uses default. | | `httpClient` | `{ (input: RequestInfo \| URL, init?: RequestInit): Promise; (input: string \| URL \| Request, init?: RequestInit): Promise; } \| null \| undefined` | no | Custom `fetch` for API requests, e.g. one with a longer request timeout. Defaults to null, which lets the SDK use its own. DIVERGENCE: upstream takes an httpx client here. | | `settings` | `OpenAITTSSettings \| null \| undefined` | no | Runtime-updatable settings. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `textAggregationMode` | `TextAggregationMode \| null \| undefined` | no | How to aggregate incoming text before synthesis. `TextAggregationMode.SENTENCE` (default) buffers until sentence boundaries, `TextAggregationMode.TOKEN` streams tokens directly for lower latency. | | `pushTextFrames` | `boolean \| undefined` | no | Whether to push TextFrames and LLMFullResponseEndFrames. | | `pushStopFrames` | `boolean \| undefined` | no | Whether to automatically push TTSStoppedFrames. | | `pushStartFrame` | `boolean \| undefined` | no | Whether to automatically create audio contexts and push TTSStartedFrames. When true, the base class handles `createAudioContext` and yields `TTSStartedFrame` before each synthesis call, so `runTts` implementations do not need to. | | `stopFrameTimeoutS` | `number \| undefined` | no | Idle time before pushing TTSStoppedFrame when pushStopFrames is true. | | `pushSilenceAfterStop` | `boolean \| undefined` | no | Whether to push silence audio after TTSStoppedFrame. | | `silenceTimeS` | `number \| undefined` | no | Duration of silence to push when pushSilenceAfterStop is true. | | `pauseFrameProcessing` | `boolean \| undefined` | no | Whether to hold incoming frames after a turn until its audio contexts end. Needed by services whose audio doesn't carry its request's context ID. | | `maxConsecutiveZeroAudioContexts` | `number \| undefined` | no | How many consecutive TTS contexts may complete without producing any audio before the service is reported unable to do its job. Catches a provider that accepts requests and stays silent — an unknown voice ID, say — which no error ever surfaces. Every silent context reports an error the service can carry on from, whatever the limit is; on reaching the limit the service reports a permanent error instead, stops being given work, and the pipeline worker applies its `ProcessorUnusablePolicy`. Set to 0 to report silent contexts without ever writing the service off. | | `appendTrailingSpace` | `boolean \| undefined` | no | Whether to append a trailing space to text before sending to TTS. This helps prevent some TTS services from vocalizing trailing punctuation (e.g., "dot"). Only applied in sentence aggregation mode; when streaming tokens, the incoming text's own whitespace is preserved. | | `sampleRate` | `number \| null \| undefined` | no | Output sample rate for generated audio. | | `skipTextTypes` | `string[] \| null \| undefined` | no | List of text types that should not be spoken. | | `textTransforms` | `[string, TextTransform][] \| null \| undefined` | no | A list of TextTransforms to transform text just before sending it to TTS. Each TextTransform takes the aggregated text and its type, and returns the transformed text. To register, provide a list of tuples of `[textType \| "*", transformFunction]`. | | `transportDestination` | `string \| null \| undefined` | no | Destination for generated audio frames. | | `reuseContextIdWithinTurn` | `boolean \| undefined` | no | Whether the service should reuse context IDs within the same turn. | | `audioCache` | `TTSAudioCacheStore \| null \| undefined` | no | Where to keep synthesized audio for reuse (e.g. a shared `TTSAudioCache`). Text whose audio is in the cache is served from it, with no request to the provider. Lines get there through `warmAudioCache()`, and also from live synthesis when `cacheSynthesizedAudio` allows it. Only services that yield their audio from `runTts` (HTTP and streaming-RPC services, see `supportsAudioCache`) use it, and only in sentence aggregation mode. Defaults to null (no cache). pipedog only. | | `cacheSynthesizedAudio` | `boolean \| ((text: string) => boolean) \| undefined` | no | Whether audio synthesized live is stored in `audioCache`, or a function that decides per text. Defaults to false: only warmed lines are stored, since a generated reply can repeat what the user said. pipedog only. | | `audioCacheNamespace` | `string \| undefined` | no | Extra part of the cache key, for constructor options that change the audio but are not in `settings`. pipedog only. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new OpenAITTSService.Settings({...})` (`OpenAITTSSettings`). Any setting you leave out keeps the default below. | Setting | Type | Default | Description | | -------------- | ---------------- | ------------------- | ------------------------------------------------------------------------------------------------------------------------ | | `model` | `string` | `"gpt-4o-mini-tts"` | Speech model. | | `voice` | `string` | `"alloy"` | One of `alloy`, `ash`, `ballad`, `cedar`, `coral`, `echo`, `fable`, `marin`, `nova`, `onyx`, `sage`, `shimmer`, `verse`. | | `language` | `null` | `null` | Not used by this service. | | `instructions` | `string \| null` | `null` | How the voice should speak (tone, pace, accent). | | `speed` | `number \| null` | `null` | Speed, 0.25 to 4.0. `null` leaves OpenAI's default (1.0). | ### Usage [#usage] ```ts title="tts.ts" import { TTSUpdateSettingsFrame } from "@pipedog/core/frames/frames"; import { OpenAITTSService } from "@pipedog/openai/tts"; const tts = new OpenAITTSService({ apiKey: process.env.OPENAI_API_KEY ?? "", sampleRate: 24000, settings: new OpenAITTSService.Settings({ model: "gpt-4o-mini-tts", voice: "marin", instructions: "Speak in a calm, friendly tone.", }), }); ``` Change the voice or speed mid-call with a `TTSUpdateSettingsFrame`: ```ts const update = new TTSUpdateSettingsFrame({ delta: new OpenAITTSService.Settings({ voice: "cedar", speed: 1.2 }), }); ``` ### Event handlers [#event-handlers] | Event | Description | | ---------------- | ---------------------------------------- | | `on_tts_request` | A synthesis request is about to be made. | This comes from the `TTSService` base class. ### Notes [#notes] * Requests `response_format: "pcm"` and emits `TTSAudioRawFrame`s (mono, `chunkSize` bytes each) between `TTSStartedFrame` and `TTSStoppedFrame`. * OpenAI only returns 24 kHz audio. Set `sampleRate: 24000` (or a 24 kHz pipeline output rate); any other rate logs a warning, and the frames are still labelled with that rate. * An unknown voice, or a request OpenAI rejects, produces an `ErrorFrame` instead of audio. * `httpClient` replaces the SDK's `fetch`, for example to set a longer request timeout. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/openai/package.json`: ✅ means the compat smoke test passes on that runtime. * The client is the `openai` JS SDK over `fetch`. The audio is read from the response body stream, so it works on Bun, Node, Deno and Cloudflare Workers. * `httpClient` takes a `fetch` function where upstream takes an httpx client. * The deprecated `OpenAITTSService.InputParams` and the `voice`, `model`, `instructions`, `speed` and `params` constructor options are not ported. Use `settings`. # Piper > Text-to-speech with Piper voices, run locally by PiperTTSService or through a Piper HTTP server by PiperHttpTTSService. Source: https://polvallverdu.github.io/pipedog/docs/services/tts/piper/ ## Overview [#overview] * **`PiperTTSService`**: runs a [Piper](https://github.com/OHF-Voice/piper1-gpl) voice on your machine, on the CPU, with no API key. Its engine (`@pipedog/piper/piper_voice`) is a TypeScript reimplementation of what upstream uses from the Python `piper-tts` package (1.8.0), on `@pipedog/onnx` and `@pipedog/espeak-ng`. * **`PiperHttpTTSService`**: posts each sentence to a [Piper HTTP server](https://github.com/OHF-Voice/piper1-gpl/blob/main/docs/API_HTTP.md) over `fetch`. It needs no model, no ONNX runtime and no eSpeak NG, and runs on Cloudflare Workers. Pipecat's documentation for the same services. `packages/piper/src/tts.ts` and the engine, `piper_voice.ts`. Every voice, by language and quality, with samples. ## Installation [#installation] ```bash bun add @pipedog/piper # for PiperTTSService (local voices) only: bun add onnxruntime-node @echogarden/espeak-ng-emscripten ``` * `onnxruntime-node` runs the voice model. `onnxruntime-web@1.30.0` (WebAssembly) works too, slower. * `@echogarden/espeak-ng-emscripten` is eSpeak NG compiled to WebAssembly, for phonemization. It is GPL-3.0 (see [Licenses](#licenses)). `PiperHttpTTSService` needs neither: the local engine is imported lazily, only by `PiperTTSService`. ```ts import { PiperHttpTTSService, PiperTTSService } from "@pipedog/piper/tts"; ``` ## What you need [#what-you-need] **`PiperTTSService`**: a Piper voice, `.onnx` and `.onnx.json` from Hugging Face [`rhasspy/piper-voices`](https://huggingface.co/rhasspy/piper-voices). It is downloaded once, when the pipeline starts, into `downloadDir` if it is missing, as `piper.download_voices` does. Sizes depend on the quality: | Quality | Sample rate | Size | | -------- | ----------- | --------- | | `x_low` | 16 kHz | 20-28 MB | | `low` | 16 kHz | 28-69 MB | | `medium` | 22.05 kHz | 62-81 MB | | `high` | 22.05 kHz | 63-136 MB | Audio is resampled to the pipeline's output rate. `downloadDir` defaults to `$PIPEDOG_CACHE_DIR/piper`, else `$XDG_CACHE_HOME/pipedog/piper`, else `~/.cache/pipedog/piper`. Put the two files there yourself to skip the download: ```ts // Voices already on disk: put .onnx and .onnx.json in downloadDir. const fromDisk = new PiperTTSService({ downloadDir: "/models/piper", settings: new PiperTTSService.Settings({ voice: "en_US-lessac-medium" }), }); ``` eSpeak NG adds about 25 MB (the WebAssembly build and its voice data), loaded into memory once per process. `@pipedog/espeak-ng` wraps it and reproduces piper-tts' clause-by-clause phonemization, so the phonemes match piper-tts 1.8.0's for the tested languages, except where piper-tts bundles newer espeak-ng data (`en-gb-x-rp`, `pt-br` and `it` differ slightly). **`PiperHttpTTSService`**: a running Piper HTTP server, for example: ```bash uv pip install "piper-tts[http]" uv run python -m piper.http_server -m en_US-ryan-high ``` ## Runtimes [#runtimes] | Runtime | Works | What it needs | | ------------------ | ----- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Bun | ✅ yes | PiperTTSService: onnxruntime-node (or onnxruntime-web) and @echogarden/espeak-ng-emscripten, plus a Piper voice (20-136 MB) downloaded on first use. PiperHttpTTSService: only fetch and a Piper HTTP server. | | Node.js | ✅ yes | PiperTTSService: onnxruntime-node (or onnxruntime-web) and @echogarden/espeak-ng-emscripten, plus a Piper voice (20-136 MB) downloaded on first use. PiperHttpTTSService: only fetch and a Piper HTTP server. | | Deno | ✅ yes | PiperTTSService: onnxruntime-node (or onnxruntime-web) and @echogarden/espeak-ng-emscripten, plus a Piper voice (20-136 MB) downloaded on first use. PiperHttpTTSService: only fetch and a Piper HTTP server. | | Cloudflare Workers | ✅ yes | PiperTTSService: onnxruntime-node (or onnxruntime-web) and @echogarden/espeak-ng-emscripten, plus a Piper voice (20-136 MB) downloaded on first use. PiperHttpTTSService: only fetch and a Piper HTTP server. PiperHttpTTSService only (PiperTTSService needs a filesystem and exceeds Workers' limits). | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/piper/package.json`: ✅ means the compat smoke test passes on that runtime. `PiperHttpTTSService` runs on Cloudflare Workers. `PiperTTSService` does not: it needs a filesystem for the voice cache, and the voice model (20-136 MB) and eSpeak NG's data don't fit a Worker's 128 MB of memory or its 10 MB bundle limit. ## PiperTTSService [#piperttsservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `downloadDir` | `string \| null \| undefined` | no | Directory for storing voice model files. Defaults to pipedog's cache directory (`piperCacheDir()`). | | `forceRedownload` | `boolean \| undefined` | no | Re-download the voice model even if it already exists. | | `useCuda` | `boolean \| undefined` | no | Use CUDA for GPU-accelerated inference. DIVERGENCE: accepted but has no effect; inference runs on the ONNX backend's default (CPU) execution provider. | | `fetch` | `((input: RequestInfo \| URL, init?: RequestInit) => Promise) \| null \| undefined` | no | `fetch` for downloads (default: the global `fetch`). Not upstream. | | `backend` | `OnnxBackend \| Promise \| null \| undefined` | no | The ONNX backend (default: `createOnnxBackend()`). Not upstream. | | `settings` | `PiperTTSSettings \| null \| undefined` | no | Runtime-updatable settings. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `textAggregationMode` | `TextAggregationMode \| null \| undefined` | no | How to aggregate incoming text before synthesis. `TextAggregationMode.SENTENCE` (default) buffers until sentence boundaries, `TextAggregationMode.TOKEN` streams tokens directly for lower latency. | | `pushTextFrames` | `boolean \| undefined` | no | Whether to push TextFrames and LLMFullResponseEndFrames. | | `pushStopFrames` | `boolean \| undefined` | no | Whether to automatically push TTSStoppedFrames. | | `pushStartFrame` | `boolean \| undefined` | no | Whether to automatically create audio contexts and push TTSStartedFrames. When true, the base class handles `createAudioContext` and yields `TTSStartedFrame` before each synthesis call, so `runTts` implementations do not need to. | | `stopFrameTimeoutS` | `number \| undefined` | no | Idle time before pushing TTSStoppedFrame when pushStopFrames is true. | | `pushSilenceAfterStop` | `boolean \| undefined` | no | Whether to push silence audio after TTSStoppedFrame. | | `silenceTimeS` | `number \| undefined` | no | Duration of silence to push when pushSilenceAfterStop is true. | | `pauseFrameProcessing` | `boolean \| undefined` | no | Whether to hold incoming frames after a turn until its audio contexts end. Needed by services whose audio doesn't carry its request's context ID. | | `maxConsecutiveZeroAudioContexts` | `number \| undefined` | no | How many consecutive TTS contexts may complete without producing any audio before the service is reported unable to do its job. Catches a provider that accepts requests and stays silent — an unknown voice ID, say — which no error ever surfaces. Every silent context reports an error the service can carry on from, whatever the limit is; on reaching the limit the service reports a permanent error instead, stops being given work, and the pipeline worker applies its `ProcessorUnusablePolicy`. Set to 0 to report silent contexts without ever writing the service off. | | `appendTrailingSpace` | `boolean \| undefined` | no | Whether to append a trailing space to text before sending to TTS. This helps prevent some TTS services from vocalizing trailing punctuation (e.g., "dot"). Only applied in sentence aggregation mode; when streaming tokens, the incoming text's own whitespace is preserved. | | `sampleRate` | `number \| null \| undefined` | no | Output sample rate for generated audio. | | `skipTextTypes` | `string[] \| null \| undefined` | no | List of text types that should not be spoken. | | `textTransforms` | `[string, TextTransform][] \| null \| undefined` | no | A list of TextTransforms to transform text just before sending it to TTS. Each TextTransform takes the aggregated text and its type, and returns the transformed text. To register, provide a list of tuples of `[textType \| "*", transformFunction]`. | | `transportDestination` | `string \| null \| undefined` | no | Destination for generated audio frames. | | `reuseContextIdWithinTurn` | `boolean \| undefined` | no | Whether the service should reuse context IDs within the same turn. | | `audioCache` | `TTSAudioCacheStore \| null \| undefined` | no | Where to keep synthesized audio for reuse (e.g. a shared `TTSAudioCache`). Text whose audio is in the cache is served from it, with no request to the provider. Lines get there through `warmAudioCache()`, and also from live synthesis when `cacheSynthesizedAudio` allows it. Only services that yield their audio from `runTts` (HTTP and streaming-RPC services, see `supportsAudioCache`) use it, and only in sentence aggregation mode. Defaults to null (no cache). pipedog only. | | `cacheSynthesizedAudio` | `boolean \| ((text: string) => boolean) \| undefined` | no | Whether audio synthesized live is stored in `audioCache`, or a function that decides per text. Defaults to false: only warmed lines are stored, since a generated reply can repeat what the user said. pipedog only. | | `audioCacheNamespace` | `string \| undefined` | no | Extra part of the cache key, for constructor options that change the audio but are not in `settings`. pipedog only. | ### Settings [#settings] Settings, passed as `settings: new PiperTTSService.Settings({...})` (`PiperTTSSettings`). | Setting | Type | Default | Description | | ---------- | -------- | -------------- | -------------------------------------------------- | | `model` | `null` | `null` | Not used. | | `voice` | `string` | none, required | Piper voice name, for example `"en_US-ryan-high"`. | | `language` | `null` | `null` | Not used: the voice sets the language. | The voice is loaded once. Settings changes are stored but not applied, with a warning, as upstream. ### Usage [#usage] ```ts title="tts.ts" import { PiperHttpTTSService, PiperTTSService } from "@pipedog/piper/tts"; // Downloads en_US-ryan-high.onnx and .onnx.json from rhasspy/piper-voices on first start. const tts = new PiperTTSService({ settings: new PiperTTSService.Settings({ voice: "en_US-ryan-high" }), }); ``` The engine works on its own too: ```ts import { downloadVoice, PiperVoice } from "@pipedog/piper/piper_voice"; await downloadVoice("en_US-lessac-medium", "/models"); const voice = await PiperVoice.load("/models/en_US-lessac-medium.onnx"); for await (const chunk of voice.synthesize("Hello world. How are you?")) { chunk.audioInt16Bytes; // one sentence, 16-bit PCM at voice.config.sampleRate } ``` ### Notes [#notes] * The voice is downloaded and loaded in `start()`. A failure is pushed as an error and retried on the first sentence. * Two services with the same voice in one process share one model session. * Each sentence is phonemized clause by clause, synthesized, and pushed as 16-bit PCM. * `forceRedownload: true` downloads the voice again even if it is on disk. * `useCuda` is accepted but has no effect: inference runs on the ONNX backend's default (CPU) execution provider. * Not supported: voices whose phoneme type isn't `espeak` or `text` (Chinese, Hebrew, Japanese and Thai voices need other G2P models), and Arabic diacritization. ## PiperHttpTTSService [#piperhttpttsservice] Posts `{ text, voice }` to `baseUrl` for each sentence and streams the WAV reply, without its header. ### Configuration [#configuration-1] | Option | Type | Required | Description | |---|---|---|---| | `baseUrl` | `string` | yes | Base URL for the Piper TTS HTTP server. | | `fetch` | `((input: RequestInfo \| URL, init?: RequestInit) => Promise) \| null \| undefined` | no | `fetch` used for HTTP requests. Defaults to the global `fetch`. DIVERGENCE: upstream takes a required `aiohttp_session`. | | `settings` | `PiperHttpTTSSettings \| null \| undefined` | no | Runtime-updatable settings. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `textAggregationMode` | `TextAggregationMode \| null \| undefined` | no | How to aggregate incoming text before synthesis. `TextAggregationMode.SENTENCE` (default) buffers until sentence boundaries, `TextAggregationMode.TOKEN` streams tokens directly for lower latency. | | `pushTextFrames` | `boolean \| undefined` | no | Whether to push TextFrames and LLMFullResponseEndFrames. | | `pushStopFrames` | `boolean \| undefined` | no | Whether to automatically push TTSStoppedFrames. | | `pushStartFrame` | `boolean \| undefined` | no | Whether to automatically create audio contexts and push TTSStartedFrames. When true, the base class handles `createAudioContext` and yields `TTSStartedFrame` before each synthesis call, so `runTts` implementations do not need to. | | `stopFrameTimeoutS` | `number \| undefined` | no | Idle time before pushing TTSStoppedFrame when pushStopFrames is true. | | `pushSilenceAfterStop` | `boolean \| undefined` | no | Whether to push silence audio after TTSStoppedFrame. | | `silenceTimeS` | `number \| undefined` | no | Duration of silence to push when pushSilenceAfterStop is true. | | `pauseFrameProcessing` | `boolean \| undefined` | no | Whether to hold incoming frames after a turn until its audio contexts end. Needed by services whose audio doesn't carry its request's context ID. | | `maxConsecutiveZeroAudioContexts` | `number \| undefined` | no | How many consecutive TTS contexts may complete without producing any audio before the service is reported unable to do its job. Catches a provider that accepts requests and stays silent — an unknown voice ID, say — which no error ever surfaces. Every silent context reports an error the service can carry on from, whatever the limit is; on reaching the limit the service reports a permanent error instead, stops being given work, and the pipeline worker applies its `ProcessorUnusablePolicy`. Set to 0 to report silent contexts without ever writing the service off. | | `appendTrailingSpace` | `boolean \| undefined` | no | Whether to append a trailing space to text before sending to TTS. This helps prevent some TTS services from vocalizing trailing punctuation (e.g., "dot"). Only applied in sentence aggregation mode; when streaming tokens, the incoming text's own whitespace is preserved. | | `sampleRate` | `number \| null \| undefined` | no | Output sample rate for generated audio. | | `skipTextTypes` | `string[] \| null \| undefined` | no | List of text types that should not be spoken. | | `textTransforms` | `[string, TextTransform][] \| null \| undefined` | no | A list of TextTransforms to transform text just before sending it to TTS. Each TextTransform takes the aggregated text and its type, and returns the transformed text. To register, provide a list of tuples of `[textType \| "*", transformFunction]`. | | `transportDestination` | `string \| null \| undefined` | no | Destination for generated audio frames. | | `reuseContextIdWithinTurn` | `boolean \| undefined` | no | Whether the service should reuse context IDs within the same turn. | | `audioCache` | `TTSAudioCacheStore \| null \| undefined` | no | Where to keep synthesized audio for reuse (e.g. a shared `TTSAudioCache`). Text whose audio is in the cache is served from it, with no request to the provider. Lines get there through `warmAudioCache()`, and also from live synthesis when `cacheSynthesizedAudio` allows it. Only services that yield their audio from `runTts` (HTTP and streaming-RPC services, see `supportsAudioCache`) use it, and only in sentence aggregation mode. Defaults to null (no cache). pipedog only. | | `cacheSynthesizedAudio` | `boolean \| ((text: string) => boolean) \| undefined` | no | Whether audio synthesized live is stored in `audioCache`, or a function that decides per text. Defaults to false: only warmed lines are stored, since a generated reply can repeat what the user said. pipedog only. | | `audioCacheNamespace` | `string \| undefined` | no | Extra part of the cache key, for constructor options that change the audio but are not in `settings`. pipedog only. | ### Settings [#settings-1] Settings, passed as `settings: new PiperHttpTTSService.Settings({...})` (`PiperHttpTTSSettings`). | Setting | Type | Default | Description | | ---------- | ---------------- | ------- | ---------------------------------------------------------------- | | `model` | `null` | `null` | Not used. | | `voice` | `string \| null` | `null` | Voice the server should use. `null` leaves the server's default. | | `language` | `null` | `null` | Not used. | ### Usage [#usage-1] ```ts // A Piper HTTP server: python -m piper.http_server -m en_US-ryan-high const remote = new PiperHttpTTSService({ baseUrl: "http://localhost:5000", settings: new PiperHttpTTSService.Settings({ voice: "en_US-ryan-high" }), }); ``` ### Notes [#notes-1] * A trailing slash on `baseUrl` is removed, with a warning. * A non-200 reply is pushed as an `ErrorFrame`. * `fetch` replaces the global `fetch`. ## Differences from upstream [#differences-from-upstream] * `PiperTTSService` downloads and loads the voice in `start()`, not in the constructor, since loading is asynchronous. A missing or blank voice is still rejected by the constructor. * Default download directory: `~/.cache/pipedog/piper`, where upstream uses `~/.cache/pipecat/piper`. * `useCuda` has no effect (above). `fetch` and `backend` are pipedog options. * `PiperHttpTTSService`: upstream's required `aiohttp_session` is an optional `fetch`. * The deprecated `voice_id` argument is not ported. Use `settings`. ## Licenses [#licenses] * `@pipedog/piper`: BSD 2-Clause. * Piper voices: each has its own license, in its `MODEL_CARD` on Hugging Face. * eSpeak NG and `@echogarden/espeak-ng-emscripten`, used by `PiperTTSService`: **GPL-3.0**. Upstream's `PiperTTSService` uses `piper-tts`, which is GPL-3.0 itself. To keep GPL code out of your application, use `PiperHttpTTSService` with a separately installed Piper server. # Pocket TTS > Local text-to-speech with PocketTTSService, running Kyutai's 100M-parameter Pocket TTS on ONNX Runtime, faster than real time on a CPU. No API key. Source: https://polvallverdu.github.io/pipedog/docs/services/tts/pocket-tts/ ## Overview [#overview] * **`PocketTTSService`**: synthesizes speech locally with Kyutai's [Pocket TTS](https://github.com/kyutai-labs/pocket-tts), a 100M-parameter streaming model that runs faster than real time on a CPU. No GPU, no account, no API key. Audio streams in 80 ms frames at 24 kHz. * **`TTSModel`** (`@pipedog/pocket-tts/pocket_tts`): the engine. Upstream wraps the `pocket_tts` Python package (PyTorch); this one runs the same model on ONNX Runtime through `@pipedog/onnx`, with the tokenizer, text chunking, flow-LM loop, LSD sampler and streaming Mimi decoding reimplemented in TypeScript. At `temp: 0` it produces the same audio as pocket-tts 2.1.0 to within 2e-4 per sample. At the default temperature the noise comes from a different generator, so the audio differs sample by sample, as two seeds would. Pipecat's documentation for the same service. `packages/pocket-tts/src/tts.ts` and the engine, `pocket_tts.ts`. Kyutai's repository, with the voices and the terms of use. ## Installation [#installation] ```bash bun add @pipedog/pocket-tts onnxruntime-node ``` `onnxruntime-node` is an optional peer of `@pipedog/onnx`, and it is required here: `onnxruntime-web` 1.30 can't read the graphs' empty state outputs. ```ts import { PocketTTSService } from "@pipedog/pocket-tts/tts"; ``` ## What you need [#what-you-need] Downloaded once, from Hugging Face, when the pipeline starts (or on the first utterance). No account or token is needed. | | | | -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Graphs | The ONNX export of Kyutai's weights, [`KevinAHM/pocket-tts-onnx`](https://huggingface.co/KevinAHM/pocket-tts-onnx) (pinned revision): `text_conditioner.onnx`, `flow_lm_main.onnx` (or `flow_lm_main_int8.onnx` with `quantize`), `flow_lm_flow.onnx`, `mimi_decoder.onnx`, `bundle.json` | | Tokenizer and voices | Kyutai's own files, from [`kyutai/pocket-tts-without-voice-cloning`](https://huggingface.co/kyutai/pocket-tts-without-voice-cloning), at the revisions pocket-tts 2.1.0 pins | | Sizes | 6-layer languages: 405 MB, or 180 MB with `quantize`. `french_24l`: 1.31 GB, or 400 MB with `quantize`. About 6 MB per voice | | Cache | `$PIPEDOG_CACHE_DIR/pocket-tts`, else `$XDG_CACHE_HOME/pipedog/pocket-tts`, else `~/.cache/pipedog/pocket-tts`, or `modelOptions.cacheDir`. Files go in `///` | To use files you already have, put them in one directory and pass `modelOptions: { modelDir }`. Anything missing there is downloaded: ```ts // Files you already have: bundle.json, the .onnx graphs, tokenizer.model and // embeddings/.safetensors in one directory. Anything missing is downloaded. const fromDisk = new PocketTTSService({ modelOptions: { modelDir: "/models/pocket-tts/english" }, settings: new PocketTTSService.Settings({ voice: "alba" }), }); ``` `modelOptions.hfToken` (default `$HF_TOKEN`) is sent to huggingface.co only, for private or gated files. ## Runtimes [#runtimes] | Runtime | Works | What it needs | | ------------------ | ----- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Bun | ✅ yes | onnxruntime-node (onnxruntime-web 1.30 can't read the graphs' empty state outputs) and a filesystem for the model cache: the model downloads on first use from Hugging Face (6-layer languages \~405 MB fp32, \~180 MB with quantize; french_24l \~1.3 GB; \~6 MB per voice). | | Node.js | ✅ yes | onnxruntime-node (onnxruntime-web 1.30 can't read the graphs' empty state outputs) and a filesystem for the model cache: the model downloads on first use from Hugging Face (6-layer languages \~405 MB fp32, \~180 MB with quantize; french_24l \~1.3 GB; \~6 MB per voice). | | Deno | ✅ yes | onnxruntime-node (onnxruntime-web 1.30 can't read the graphs' empty state outputs) and a filesystem for the model cache: the model downloads on first use from Hugging Face (6-layer languages \~405 MB fp32, \~180 MB with quantize; french_24l \~1.3 GB; \~6 MB per voice). | | Cloudflare Workers | ❌ no | onnxruntime-node (onnxruntime-web 1.30 can't read the graphs' empty state outputs) and a filesystem for the model cache: the model downloads on first use from Hugging Face (6-layer languages \~405 MB fp32, \~180 MB with quantize; french_24l \~1.3 GB; \~6 MB per voice). Not supported: the flow LM graph (76-1210 MB) and its 50 MB KV-cache state per step exceed Workers' memory and bundle limits. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/pocket-tts/package.json`: ✅ means the compat smoke test passes on that runtime. Cloudflare Workers is not supported: the flow-LM graph (76 MB to 1.2 GB) and its \~50 MB per-step KV-cache state don't fit a Worker's 128 MB of memory or its 10 MB bundle limit. On an Apple M4 Pro with Bun and `onnxruntime-node`, English, 2.24 s of audio takes 0.37 s (6x real time), with the first 80 ms chunk after about 45 ms. ## PocketTTSService [#pocketttsservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `temp` | `number \| null \| undefined` | no | Sampling temperature for audio generation. Defaults to the pocket-tts model default. | | `quantize` | `boolean \| undefined` | no | Quantize model weights for faster CPU inference (upstream: requires the `torchao` package; here: the int8 ONNX graph of the flow LM transformer). | | `settings` | `PocketTTSSettings \| null \| undefined` | no | Runtime-updatable settings. Defaults to the `"alba"` voice and English. | | `modelOptions` | `Omit \| null \| undefined` | no | Engine options passed to `TTSModel.loadModel()`: ONNX backend, model directory, cache directory, `fetch`, voice-cloning encoder, ... DIVERGENCE: pipedog-specific; pocket-tts downloads to its own cache. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `textAggregationMode` | `TextAggregationMode \| null \| undefined` | no | How to aggregate incoming text before synthesis. `TextAggregationMode.SENTENCE` (default) buffers until sentence boundaries, `TextAggregationMode.TOKEN` streams tokens directly for lower latency. | | `pushTextFrames` | `boolean \| undefined` | no | Whether to push TextFrames and LLMFullResponseEndFrames. | | `pushStopFrames` | `boolean \| undefined` | no | Whether to automatically push TTSStoppedFrames. | | `pushStartFrame` | `boolean \| undefined` | no | Whether to automatically create audio contexts and push TTSStartedFrames. When true, the base class handles `createAudioContext` and yields `TTSStartedFrame` before each synthesis call, so `runTts` implementations do not need to. | | `stopFrameTimeoutS` | `number \| undefined` | no | Idle time before pushing TTSStoppedFrame when pushStopFrames is true. | | `pushSilenceAfterStop` | `boolean \| undefined` | no | Whether to push silence audio after TTSStoppedFrame. | | `silenceTimeS` | `number \| undefined` | no | Duration of silence to push when pushSilenceAfterStop is true. | | `pauseFrameProcessing` | `boolean \| undefined` | no | Whether to hold incoming frames after a turn until its audio contexts end. Needed by services whose audio doesn't carry its request's context ID. | | `maxConsecutiveZeroAudioContexts` | `number \| undefined` | no | How many consecutive TTS contexts may complete without producing any audio before the service is reported unable to do its job. Catches a provider that accepts requests and stays silent — an unknown voice ID, say — which no error ever surfaces. Every silent context reports an error the service can carry on from, whatever the limit is; on reaching the limit the service reports a permanent error instead, stops being given work, and the pipeline worker applies its `ProcessorUnusablePolicy`. Set to 0 to report silent contexts without ever writing the service off. | | `appendTrailingSpace` | `boolean \| undefined` | no | Whether to append a trailing space to text before sending to TTS. This helps prevent some TTS services from vocalizing trailing punctuation (e.g., "dot"). Only applied in sentence aggregation mode; when streaming tokens, the incoming text's own whitespace is preserved. | | `sampleRate` | `number \| null \| undefined` | no | Output sample rate for generated audio. | | `skipTextTypes` | `string[] \| null \| undefined` | no | List of text types that should not be spoken. | | `textTransforms` | `[string, TextTransform][] \| null \| undefined` | no | A list of TextTransforms to transform text just before sending it to TTS. Each TextTransform takes the aggregated text and its type, and returns the transformed text. To register, provide a list of tuples of `[textType \| "*", transformFunction]`. | | `transportDestination` | `string \| null \| undefined` | no | Destination for generated audio frames. | | `reuseContextIdWithinTurn` | `boolean \| undefined` | no | Whether the service should reuse context IDs within the same turn. | | `audioCache` | `TTSAudioCacheStore \| null \| undefined` | no | Where to keep synthesized audio for reuse (e.g. a shared `TTSAudioCache`). Text whose audio is in the cache is served from it, with no request to the provider. Lines get there through `warmAudioCache()`, and also from live synthesis when `cacheSynthesizedAudio` allows it. Only services that yield their audio from `runTts` (HTTP and streaming-RPC services, see `supportsAudioCache`) use it, and only in sentence aggregation mode. Defaults to null (no cache). pipedog only. | | `cacheSynthesizedAudio` | `boolean \| ((text: string) => boolean) \| undefined` | no | Whether audio synthesized live is stored in `audioCache`, or a function that decides per text. Defaults to false: only warmed lines are stored, since a generated reply can repeat what the user said. pipedog only. | | `audioCacheNamespace` | `string \| undefined` | no | Extra part of the cache key, for constructor options that change the audio but are not in `settings`. pipedog only. | `modelOptions` (pipedog only) takes `TTSModel.loadModel`'s options: `backend`, `sessionOptions` (default `{ intraOpNumThreads: 4, interOpNumThreads: 1 }`), `shareSessions`, `modelDir`, `cacheDir`, `fetch`, `hfToken`, `voiceCloningModel`, `lsdDecodeSteps` (1), `noiseClamp` (`null`), `eosThreshold` (-4) and `random`. ### Settings [#settings] Runtime-updatable settings, passed as `settings: new PocketTTSService.Settings({...})` (`PocketTTSSettings`). Any setting you leave out keeps the default below. | Setting | Type | Default | Description | | ---------- | ---------- | ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `model` | `null` | `null` | Not used. | | `voice` | `string` | `"alba"` | A predefined voice, a `.safetensors` voice state, or a WAV file to clone (path, `https://` or `hf://` URL). Re-derived on the next utterance when it changes. | | `language` | `Language` | `Language.EN` | `EN`, `DE`, `ES`, `IT`, `PT` or `FR` (and regional variants), mapped to `english`, `german`, `spanish`, `italian`, `portuguese` and `french_24l`. Stored but not applied after construction, as upstream: the weights are loaded per language. | The 26 predefined voices: `alba`, `marius`, `javert`, `jean`, `fantine`, `cosette`, `eponine`, `azelma`, `anna`, `vera`, `charles`, `paul`, `george`, `mary`, `jane`, `michael`, `eve`, `bill_boerst`, `peter_yearsley`, `stuart_bell`, `caro_davy`, `giovanni`, `lola`, `juergen`, `rafael`, `estelle`. ### Usage [#usage] ```ts title="tts.ts" import { TTSUpdateSettingsFrame } from "@pipedog/core/frames/frames"; import { Language } from "@pipedog/core/transcriptions/language"; import { PocketTTSService } from "@pipedog/pocket-tts/tts"; // Downloads the English model (405 MB, or 180 MB with quantize) and the voice on first start. const tts = new PocketTTSService({ quantize: true, settings: new PocketTTSService.Settings({ voice: "alba", language: Language.EN }), }); ``` Change the voice mid-call with a `TTSUpdateSettingsFrame`: ```ts // Queue it on the worker, or push it from a processor upstream of the service. // The voice is re-derived on the next utterance; the language can't change at runtime. const update = new TTSUpdateSettingsFrame({ delta: new PocketTTSService.Settings({ voice: "marius" }), }); ``` The engine works on its own too: ```ts import { TTSModel } from "@pipedog/pocket-tts/pocket_tts"; const model = await TTSModel.loadModel({ language: "english", temp: 0.7 }); const voice = await model.getStateForAudioPrompt("alba"); for await (const chunk of model.generateAudioStream(voice, "Hello world.")) { // chunk: a Float32Array of 1920 samples at model.sampleRate (24000 Hz) } ``` ### Notes [#notes] * The constructor rejects a missing voice or language. The model and the voice state load in `start()` (or on the first utterance). * Voice cloning from a WAV file needs the Mimi encoder with Kyutai's speaker projection, which only exists in the **gated** `kyutai/pocket-tts` weights. It is not downloaded for you: accept Kyutai's terms, export the encoder to ONNX and pass it as `modelOptions.voiceCloningModel`. This path was not checked against upstream. * `TTSModel.loadModel` also accepts `german_24l`, `italian_24l`, `portuguese_24l` and `spanish_24l` (same export, not checked numerically). ## Differences from upstream [#differences-from-upstream] * The model and the voice state load in `start()`, not in the constructor. * Sampling noise comes from `Math.random` (or the `random` option), not torch's generator. * `quantize` uses ONNX Runtime's dynamic int8 quantization of the flow-LM transformer instead of torchao's, so quantized output is not bit-identical to upstream's. * The exported flow LM has a fixed 1000-position KV cache. A text chunk that would overflow it is cut short with a warning; pocket-tts' 50-token chunks with the predefined voices stay far below it. * Audio prompts are read from WAV only, and resampled to 24 kHz with soxr. * `TTSModel.loadModel` has no `config` argument (custom YAML model configs). * `modelOptions` is a pipedog option: pocket-tts downloads to its own cache. ## Licenses [#licenses] * `@pipedog/pocket-tts`: BSD 2-Clause. The engine reimplements parts of pocket-tts (MIT) and of SentencePiece's Unigram encoder (Apache-2.0). * Kyutai's Pocket TTS weights, tokenizers and voices, and the ONNX export: CC-BY-4.0 (attribution: Kyutai). They are downloaded at runtime, not shipped. Kyutai's terms forbid voice impersonation or cloning without consent, among other uses. # Resemble AI > Text-to-speech with Resemble AI: ResembleAITTSService streams audio and word timestamps over WebSocket. Source: https://polvallverdu.github.io/pipedog/docs/services/tts/resembleai/ ## Overview [#overview] `ResembleAITTSService` keeps one WebSocket open to Resemble AI's streaming endpoint and sends each sentence as a JSON request. Audio comes back as base64 JSON frames, which the service buffers (a one-second jitter buffer first, then 2208-byte frames) so playback survives Resemble's bursty delivery. Pipecat's documentation for the same service. `packages/resembleai/src/tts.ts`. The streaming WebSocket reference. ## Installation [#installation] ```bash bun add @pipedog/resembleai ``` ```ts import { ResembleAITTSService } from "@pipedog/resembleai/tts"; ``` ## Prerequisites [#prerequisites] * A Resemble AI account on a plan that includes the WebSocket API, an API key, and the UUID of a voice. * Pass the key as `apiKey`. The examples read it from `RESEMBLE_API_KEY`. ## Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | Resemble AI API key for authentication. | | `url` | `string \| undefined` | no | WebSocket URL for Resemble AI TTS API. | | `precision` | `string \| null \| undefined` | no | PCM bit depth (PCM_32, PCM_24, PCM_16, or MULAW). | | `outputFormat` | `string \| null \| undefined` | no | Audio format (wav or mp3). | | `projectUuid` | `string \| null \| undefined` | no | Project to attach the clips to. DIVERGENCE: not an upstream option. Resemble's WebSocket reference lists `project_uuid` as required, but upstream never sends it. It is sent only when this is set. | | `settings` | `ResembleAITTSSettings \| null \| undefined` | no | Runtime-updatable settings. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `reconnectBackoffMinWait` | `number \| undefined` | no | Minimum time, in seconds, to wait between reconnection attempts. | | `reconnectBackoffMaxWait` | `number \| undefined` | no | Maximum time, in seconds, to wait between reconnection attempts. | | `reconnectOnError` | `boolean \| undefined` | no | Whether to automatically reconnect on connection errors. | | `wsCloseTimeout` | `number \| undefined` | no | Maximum time, in seconds, to wait for the peer to acknowledge the websocket closing handshake before dropping the connection. Applied to connections opened through `_websocketConnect()`. Increase it for peers that need longer to complete a graceful close. | | `wsSendTimeout` | `number \| null \| undefined` | no | Maximum time, in seconds, a single send may wait on a peer that has stopped reading before the connection is dropped and the send throws `ConnectionClosedError`. Applied to connections opened through `_websocketConnect()`. Pass `null` to wait indefinitely. Defaults to `WS_SEND_TIMEOUT`. | | `textAggregationMode` | `TextAggregationMode \| null \| undefined` | no | How to aggregate incoming text before synthesis. `TextAggregationMode.SENTENCE` (default) buffers until sentence boundaries, `TextAggregationMode.TOKEN` streams tokens directly for lower latency. | | `pushTextFrames` | `boolean \| undefined` | no | Whether to push TextFrames and LLMFullResponseEndFrames. | | `pushStopFrames` | `boolean \| undefined` | no | Whether to automatically push TTSStoppedFrames. | | `pushStartFrame` | `boolean \| undefined` | no | Whether to automatically create audio contexts and push TTSStartedFrames. When true, the base class handles `createAudioContext` and yields `TTSStartedFrame` before each synthesis call, so `runTts` implementations do not need to. | | `stopFrameTimeoutS` | `number \| undefined` | no | Idle time before pushing TTSStoppedFrame when pushStopFrames is true. | | `pushSilenceAfterStop` | `boolean \| undefined` | no | Whether to push silence audio after TTSStoppedFrame. | | `silenceTimeS` | `number \| undefined` | no | Duration of silence to push when pushSilenceAfterStop is true. | | `pauseFrameProcessing` | `boolean \| undefined` | no | Whether to hold incoming frames after a turn until its audio contexts end. Needed by services whose audio doesn't carry its request's context ID. | | `maxConsecutiveZeroAudioContexts` | `number \| undefined` | no | How many consecutive TTS contexts may complete without producing any audio before the service is reported unable to do its job. Catches a provider that accepts requests and stays silent — an unknown voice ID, say — which no error ever surfaces. Every silent context reports an error the service can carry on from, whatever the limit is; on reaching the limit the service reports a permanent error instead, stops being given work, and the pipeline worker applies its `ProcessorUnusablePolicy`. Set to 0 to report silent contexts without ever writing the service off. | | `appendTrailingSpace` | `boolean \| undefined` | no | Whether to append a trailing space to text before sending to TTS. This helps prevent some TTS services from vocalizing trailing punctuation (e.g., "dot"). Only applied in sentence aggregation mode; when streaming tokens, the incoming text's own whitespace is preserved. | | `sampleRate` | `number \| null \| undefined` | no | Output sample rate for generated audio. | | `skipTextTypes` | `string[] \| null \| undefined` | no | List of text types that should not be spoken. | | `textTransforms` | `[string, TextTransform][] \| null \| undefined` | no | A list of TextTransforms to transform text just before sending it to TTS. Each TextTransform takes the aggregated text and its type, and returns the transformed text. To register, provide a list of tuples of `[textType \| "*", transformFunction]`. | | `transportDestination` | `string \| null \| undefined` | no | Destination for generated audio frames. | | `reuseContextIdWithinTurn` | `boolean \| undefined` | no | Whether the service should reuse context IDs within the same turn. | | `audioCache` | `TTSAudioCacheStore \| null \| undefined` | no | Where to keep synthesized audio for reuse (e.g. a shared `TTSAudioCache`). Text whose audio is in the cache is served from it, with no request to the provider. Lines get there through `warmAudioCache()`, and also from live synthesis when `cacheSynthesizedAudio` allows it. Only services that yield their audio from `runTts` (HTTP and streaming-RPC services, see `supportsAudioCache`) use it, and only in sentence aggregation mode. Defaults to null (no cache). pipedog only. | | `cacheSynthesizedAudio` | `boolean \| ((text: string) => boolean) \| undefined` | no | Whether audio synthesized live is stored in `audioCache`, or a function that decides per text. Defaults to false: only warmed lines are stored, since a generated reply can repeat what the user said. pipedog only. | | `audioCacheNamespace` | `string \| undefined` | no | Extra part of the cache key, for constructor options that change the audio but are not in `settings`. pipedog only. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new ResembleAITTSService.Settings({...})` (`ResembleAITTSSettings`). Any setting you leave out keeps the default below. | Setting | Type | Default | Description | | ---------- | ------------------ | ------- | --------------------------------- | | `voice` | `string \| null` | `null` | UUID of the voice (`voice_uuid`). | | `model` | `string \| null` | `null` | Not used. | | `language` | `Language \| null` | `null` | Not used. | ## Usage [#usage] ```ts title="tts.ts" import { ResembleAITTSService } from "@pipedog/resembleai/tts"; const tts = new ResembleAITTSService({ apiKey: process.env.RESEMBLE_API_KEY ?? "", // Resemble's WebSocket reference lists the project as required. projectUuid: process.env.RESEMBLE_PROJECT_UUID ?? null, settings: new ResembleAITTSService.Settings({ voice: "your-voice-uuid" }), }); ``` ### Event handlers [#event-handlers] | Event | Description | | --------------------- | ------------------------ | | `on_connected` | The WebSocket connected. | | `on_disconnected` | The WebSocket closed. | | `on_connection_error` | The connection failed. | ## Notes [#notes] * The sample rate defaults to 22050 Hz (8000, 16000, 22050, 32000 and 44100 work), `precision` to `PCM_16` and `outputFormat` to `wav`. Audio is requested without a WAV header. * Word timestamps come from Resemble's per-character timings, so each pushed "word" is one character. * A 401 or 403 error from Resemble closes the connection and opens a new one. * Resemble doesn't acknowledge the WebSocket closing handshake, so the service doesn't wait for one. * `projectUuid` is not an upstream option. Resemble's reference lists `project_uuid` as required, but upstream never sends it, so the service sends it only when you set it. ## Runtime notes [#runtime-notes] * The WebSocket opens through the core `websocketConnect`, with the key in an `Authorization` header (through `fetch` on Cloudflare Workers), so it works on Bun, Node, Deno and Cloudflare Workers. * The deprecated `voiceId` option is not ported. Use `settings`. * Tested against a local WebSocket server, not against Resemble AI itself. | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/resembleai/package.json`: ✅ means the compat smoke test passes on that runtime. # Rime > Text-to-speech with RimeTTSService (WebSocket, word timestamps), RimeHttpTTSService (HTTP) and RimeV1TTSService (WebSocket v1, protobuf). Source: https://polvallverdu.github.io/pipedog/docs/services/tts/rime/ ## Overview [#overview] Rime's Coda and Mist models turn text into speech. pipedog ports these Pipecat services: * **`RimeTTSService`**: Rime's JSON WebSocket API (`/ws3`), with word timestamps and one audio context per turn. * **`RimeHttpTTSService`**: Rime's HTTP API, streaming raw PCM, for when a WebSocket is not needed. * **`RimeV1TTSService`**: Rime's WebSocket v1 API, which carries protobuf envelopes (binary or proto3 JSON). Interruptions cancel a context without reconnecting, and Rime can accumulate streamed LLM tokens into sentences. Pipecat's documentation for the same services. `packages/rime/src/tts.ts` and `tts_v1.ts`. The WebSocket v1 protocol, voices and models. ## Installation [#installation] ```bash bun add @pipedog/rime ``` ```ts import { RimeHttpTTSService, RimeTTSService } from "@pipedog/rime/tts"; import { RimeV1TTSService } from "@pipedog/rime/tts_v1"; ``` ## Prerequisites [#prerequisites] * A Rime account and an API key from the [Rime dashboard](https://rime.ai). * A voice (`speaker`) for the model you use. There is no default voice. * Pass the key as `apiKey`. The examples read it from `RIME_API_KEY`. ## RimeTTSService [#rimettsservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `pushTextFrames` | `boolean \| undefined` | no | Whether to push TextFrames and LLMFullResponseEndFrames. | | `pushStopFrames` | `boolean \| undefined` | no | Whether to automatically push TTSStoppedFrames. | | `apiKey` | `string` | yes | Rime API key for authentication. | | `url` | `string \| undefined` | no | Rime websocket API endpoint. Defaults to "wss://users-ws.rime.ai/ws3". | | `settings` | `RimeTTSSettings \| null \| undefined` | no | Runtime-updatable settings (defaults: model "coda"). | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `reconnectBackoffMinWait` | `number \| undefined` | no | Minimum time, in seconds, to wait between reconnection attempts. | | `reconnectBackoffMaxWait` | `number \| undefined` | no | Maximum time, in seconds, to wait between reconnection attempts. | | `reconnectOnError` | `boolean \| undefined` | no | Whether to automatically reconnect on connection errors. | | `wsCloseTimeout` | `number \| undefined` | no | Maximum time, in seconds, to wait for the peer to acknowledge the websocket closing handshake before dropping the connection. Applied to connections opened through `_websocketConnect()`. Increase it for peers that need longer to complete a graceful close. | | `wsSendTimeout` | `number \| null \| undefined` | no | Maximum time, in seconds, a single send may wait on a peer that has stopped reading before the connection is dropped and the send throws `ConnectionClosedError`. Applied to connections opened through `_websocketConnect()`. Pass `null` to wait indefinitely. Defaults to `WS_SEND_TIMEOUT`. | | `textAggregationMode` | `TextAggregationMode \| null \| undefined` | no | How to aggregate incoming text before synthesis. `TextAggregationMode.SENTENCE` (default) buffers until sentence boundaries, `TextAggregationMode.TOKEN` streams tokens directly for lower latency. | | `pushStartFrame` | `boolean \| undefined` | no | Whether to automatically create audio contexts and push TTSStartedFrames. When true, the base class handles `createAudioContext` and yields `TTSStartedFrame` before each synthesis call, so `runTts` implementations do not need to. | | `stopFrameTimeoutS` | `number \| undefined` | no | Idle time before pushing TTSStoppedFrame when pushStopFrames is true. | | `pushSilenceAfterStop` | `boolean \| undefined` | no | Whether to push silence audio after TTSStoppedFrame. | | `silenceTimeS` | `number \| undefined` | no | Duration of silence to push when pushSilenceAfterStop is true. | | `pauseFrameProcessing` | `boolean \| undefined` | no | Whether to hold incoming frames after a turn until its audio contexts end. Needed by services whose audio doesn't carry its request's context ID. | | `maxConsecutiveZeroAudioContexts` | `number \| undefined` | no | How many consecutive TTS contexts may complete without producing any audio before the service is reported unable to do its job. Catches a provider that accepts requests and stays silent — an unknown voice ID, say — which no error ever surfaces. Every silent context reports an error the service can carry on from, whatever the limit is; on reaching the limit the service reports a permanent error instead, stops being given work, and the pipeline worker applies its `ProcessorUnusablePolicy`. Set to 0 to report silent contexts without ever writing the service off. | | `appendTrailingSpace` | `boolean \| undefined` | no | Whether to append a trailing space to text before sending to TTS. This helps prevent some TTS services from vocalizing trailing punctuation (e.g., "dot"). Only applied in sentence aggregation mode; when streaming tokens, the incoming text's own whitespace is preserved. | | `sampleRate` | `number \| null \| undefined` | no | Output sample rate for generated audio. | | `skipTextTypes` | `string[] \| null \| undefined` | no | List of text types that should not be spoken. | | `textTransforms` | `[string, TextTransform][] \| null \| undefined` | no | A list of TextTransforms to transform text just before sending it to TTS. Each TextTransform takes the aggregated text and its type, and returns the transformed text. To register, provide a list of tuples of `[textType \| "*", transformFunction]`. | | `transportDestination` | `string \| null \| undefined` | no | Destination for generated audio frames. | | `reuseContextIdWithinTurn` | `boolean \| undefined` | no | Whether the service should reuse context IDs within the same turn. | | `audioCache` | `TTSAudioCacheStore \| null \| undefined` | no | Where to keep synthesized audio for reuse (e.g. a shared `TTSAudioCache`). Text whose audio is in the cache is served from it, with no request to the provider. Lines get there through `warmAudioCache()`, and also from live synthesis when `cacheSynthesizedAudio` allows it. Only services that yield their audio from `runTts` (HTTP and streaming-RPC services, see `supportsAudioCache`) use it, and only in sentence aggregation mode. Defaults to null (no cache). pipedog only. | | `cacheSynthesizedAudio` | `boolean \| ((text: string) => boolean) \| undefined` | no | Whether audio synthesized live is stored in `audioCache`, or a function that decides per text. Defaults to false: only warmed lines are stored, since a generated reply can repeat what the user said. pipedog only. | | `audioCacheNamespace` | `string \| undefined` | no | Extra part of the cache key, for constructor options that change the audio but are not in `settings`. pipedog only. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new RimeTTSService.Settings({...})`. `RimeHttpTTSService` uses the same settings class. Each setting is only sent when set; the sampling settings go to `coda` and the `mistv2` ones to the other models. | Setting | Type | Default | Description | | ------------------------------------------------------------------------------------------------------ | ------------------ | -------- | ------------------------------------------------------------------------ | | `model` | `string` | `"coda"` | Rime model: `coda`, or a Mist model such as `mistv2`. | | `voice` | `string \| null` | `null` | Voice (`speaker`). Set one. | | `language` | `Language \| null` | `null` | Language. Sent as a Rime code such as `eng`. | | `segment` | `string \| null` | `null` | `"immediate"`, `"bySentence"` or `"never"`. | | `speedAlpha` | `number \| null` | `null` | Speech speed multiplier. | | `inlineSpeedAlpha` | `string \| null` | `null` | Inline speed markup. | | `repetitionPenalty`, `temperature`, `topP` | `number \| null` | `null` | Coda sampling (wire names `repetition_penalty`, `temperature`, `top_p`). | | `timeScaleFactor` | `number \| null` | `null` | Coda playback speed. Above 1.0 slows the audio down. | | `reduceLatency`, `pauseBetweenBrackets`, `phonemizeBetweenBrackets`, `noTextNormalization`, `saveOovs` | `boolean \| null` | `null` | Mist options. | ### Usage [#usage] ```ts title="tts.ts" import { TTSUpdateSettingsFrame } from "@pipedog/core/frames/frames"; import { RimeTTSService } from "@pipedog/rime/tts"; const tts = new RimeTTSService({ apiKey: process.env.RIME_API_KEY ?? "", settings: new RimeTTSService.Settings({ model: "coda", voice: "luna", timeScaleFactor: 1.0, }), }); ``` Text helpers for Rime's markup: ```ts // Text helpers for Rime's markup, for use in text transforms. const spelled = RimeTTSService.SPELL("A1B2"); const paused = `One moment.${RimeTTSService.PAUSE_TAG(0.5)} Thanks.`; // Custom pronunciation and inline speed apply to the next message sent. const pronounced = tts.PRONOUNCE("Say hello", "hello", "h'El0"); const slow = tts.INLINE_SPEED("slowly", 0.7); ``` Change settings mid-call with a `TTSUpdateSettingsFrame`: ```ts // Queue it on the worker, or push it from a processor upstream of the service. // Every setting of the WebSocket service is a URL parameter, so a change reconnects. const update = new TTSUpdateSettingsFrame({ delta: new RimeTTSService.Settings({ voice: "another-voice" }), }); ``` ### Event handlers [#event-handlers] Inherited from `TTSService`: | Event | Description | | --------------------- | ----------------------------------------------- | | `on_connected` | Connected to Rime. | | `on_disconnected` | Disconnected from Rime. | | `on_connection_error` | The connection failed. | | `on_tts_request` | A text chunk is about to be sent for synthesis. | ### Notes [#notes] * Every setting is a query parameter of the WebSocket URL, so any change reconnects. * The service does not push the LLM's text frames. It builds `TTSTextFrame`s from Rime's word timestamps. * Text inside `spell(...)` is never split across aggregates. * `PRONOUNCE()` and `INLINE_SPEED()` add fields to the next message only. ## RimeHttpTTSService [#rimehttpttsservice] ### Configuration [#configuration-1] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | Rime API key for authentication. | | `fetch` | `((input: RequestInfo \| URL, init?: RequestInit) => Promise) \| null \| undefined` | no | `fetch` used for HTTP requests. Defaults to the global `fetch`. DIVERGENCE: replaces upstream's required `aiohttp_session`. | | `settings` | `RimeTTSSettings \| null \| undefined` | no | Runtime-updatable settings (defaults: model "coda", language "eng"). | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `textAggregationMode` | `TextAggregationMode \| null \| undefined` | no | How to aggregate incoming text before synthesis. `TextAggregationMode.SENTENCE` (default) buffers until sentence boundaries, `TextAggregationMode.TOKEN` streams tokens directly for lower latency. | | `pushTextFrames` | `boolean \| undefined` | no | Whether to push TextFrames and LLMFullResponseEndFrames. | | `pushStopFrames` | `boolean \| undefined` | no | Whether to automatically push TTSStoppedFrames. | | `pushStartFrame` | `boolean \| undefined` | no | Whether to automatically create audio contexts and push TTSStartedFrames. When true, the base class handles `createAudioContext` and yields `TTSStartedFrame` before each synthesis call, so `runTts` implementations do not need to. | | `stopFrameTimeoutS` | `number \| undefined` | no | Idle time before pushing TTSStoppedFrame when pushStopFrames is true. | | `pushSilenceAfterStop` | `boolean \| undefined` | no | Whether to push silence audio after TTSStoppedFrame. | | `silenceTimeS` | `number \| undefined` | no | Duration of silence to push when pushSilenceAfterStop is true. | | `pauseFrameProcessing` | `boolean \| undefined` | no | Whether to hold incoming frames after a turn until its audio contexts end. Needed by services whose audio doesn't carry its request's context ID. | | `maxConsecutiveZeroAudioContexts` | `number \| undefined` | no | How many consecutive TTS contexts may complete without producing any audio before the service is reported unable to do its job. Catches a provider that accepts requests and stays silent — an unknown voice ID, say — which no error ever surfaces. Every silent context reports an error the service can carry on from, whatever the limit is; on reaching the limit the service reports a permanent error instead, stops being given work, and the pipeline worker applies its `ProcessorUnusablePolicy`. Set to 0 to report silent contexts without ever writing the service off. | | `appendTrailingSpace` | `boolean \| undefined` | no | Whether to append a trailing space to text before sending to TTS. This helps prevent some TTS services from vocalizing trailing punctuation (e.g., "dot"). Only applied in sentence aggregation mode; when streaming tokens, the incoming text's own whitespace is preserved. | | `sampleRate` | `number \| null \| undefined` | no | Output sample rate for generated audio. | | `skipTextTypes` | `string[] \| null \| undefined` | no | List of text types that should not be spoken. | | `textTransforms` | `[string, TextTransform][] \| null \| undefined` | no | A list of TextTransforms to transform text just before sending it to TTS. Each TextTransform takes the aggregated text and its type, and returns the transformed text. To register, provide a list of tuples of `[textType \| "*", transformFunction]`. | | `transportDestination` | `string \| null \| undefined` | no | Destination for generated audio frames. | | `reuseContextIdWithinTurn` | `boolean \| undefined` | no | Whether the service should reuse context IDs within the same turn. | | `audioCache` | `TTSAudioCacheStore \| null \| undefined` | no | Where to keep synthesized audio for reuse (e.g. a shared `TTSAudioCache`). Text whose audio is in the cache is served from it, with no request to the provider. Lines get there through `warmAudioCache()`, and also from live synthesis when `cacheSynthesizedAudio` allows it. Only services that yield their audio from `runTts` (HTTP and streaming-RPC services, see `supportsAudioCache`) use it, and only in sentence aggregation mode. Defaults to null (no cache). pipedog only. | | `cacheSynthesizedAudio` | `boolean \| ((text: string) => boolean) \| undefined` | no | Whether audio synthesized live is stored in `audioCache`, or a function that decides per text. Defaults to false: only warmed lines are stored, since a generated reply can repeat what the user said. pipedog only. | | `audioCacheNamespace` | `string \| undefined` | no | Extra part of the cache key, for constructor options that change the audio but are not in `settings`. pipedog only. | ### Usage [#usage-1] ```ts title="http.ts" import { RimeHttpTTSService } from "@pipedog/rime/tts"; const httpTts = new RimeHttpTTSService({ apiKey: process.env.RIME_API_KEY ?? "", settings: new RimeHttpTTSService.Settings({ model: "mistv2", voice: "luna", speedAlpha: 1.1 }), }); ``` ### Notes [#notes-1] * It posts to `https://users.rime.ai/v1/rime-tts` and streams the PCM body. There are no word timestamps. * The default `language` is `"eng"`. ## RimeV1TTSService [#rimev1ttsservice] ### Configuration [#configuration-2] | Option | Type | Required | Description | |---|---|---|---| | `pushTextFrames` | `boolean \| undefined` | no | Whether to push TextFrames and LLMFullResponseEndFrames. | | `pushStopFrames` | `boolean \| undefined` | no | Whether to automatically push TTSStoppedFrames. | | `apiKey` | `string` | yes | Rime API key for authentication. | | `websocketUrl` | `string \| null \| undefined` | no | Rime WebSocket v1 endpoint override. Defaults to the public endpoint for `settings.model`. Custom URLs can use any path. Models without a public default require an explicit URL. | | `websocketProtocol` | `WebSocketProtocol \| undefined` | no | Envelope encoding, "binary" or "json". | | `settings` | `RimeV1TTSSettings \| null \| undefined` | no | Settings with an explicit `model`. Other fields can be updated at runtime; changing `model` does not switch endpoints. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `reconnectBackoffMinWait` | `number \| undefined` | no | Minimum time, in seconds, to wait between reconnection attempts. | | `reconnectBackoffMaxWait` | `number \| undefined` | no | Maximum time, in seconds, to wait between reconnection attempts. | | `reconnectOnError` | `boolean \| undefined` | no | Whether to automatically reconnect on connection errors. | | `wsCloseTimeout` | `number \| undefined` | no | Maximum time, in seconds, to wait for the peer to acknowledge the websocket closing handshake before dropping the connection. Applied to connections opened through `_websocketConnect()`. Increase it for peers that need longer to complete a graceful close. | | `wsSendTimeout` | `number \| null \| undefined` | no | Maximum time, in seconds, a single send may wait on a peer that has stopped reading before the connection is dropped and the send throws `ConnectionClosedError`. Applied to connections opened through `_websocketConnect()`. Pass `null` to wait indefinitely. Defaults to `WS_SEND_TIMEOUT`. | | `textAggregationMode` | `TextAggregationMode \| null \| undefined` | no | How to aggregate incoming text before synthesis. `TextAggregationMode.SENTENCE` (default) buffers until sentence boundaries, `TextAggregationMode.TOKEN` streams tokens directly for lower latency. | | `pushStartFrame` | `boolean \| undefined` | no | Whether to automatically create audio contexts and push TTSStartedFrames. When true, the base class handles `createAudioContext` and yields `TTSStartedFrame` before each synthesis call, so `runTts` implementations do not need to. | | `stopFrameTimeoutS` | `number \| undefined` | no | Idle time before pushing TTSStoppedFrame when pushStopFrames is true. | | `pushSilenceAfterStop` | `boolean \| undefined` | no | Whether to push silence audio after TTSStoppedFrame. | | `silenceTimeS` | `number \| undefined` | no | Duration of silence to push when pushSilenceAfterStop is true. | | `pauseFrameProcessing` | `boolean \| undefined` | no | Whether to hold incoming frames after a turn until its audio contexts end. Needed by services whose audio doesn't carry its request's context ID. | | `maxConsecutiveZeroAudioContexts` | `number \| undefined` | no | How many consecutive TTS contexts may complete without producing any audio before the service is reported unable to do its job. Catches a provider that accepts requests and stays silent — an unknown voice ID, say — which no error ever surfaces. Every silent context reports an error the service can carry on from, whatever the limit is; on reaching the limit the service reports a permanent error instead, stops being given work, and the pipeline worker applies its `ProcessorUnusablePolicy`. Set to 0 to report silent contexts without ever writing the service off. | | `appendTrailingSpace` | `boolean \| undefined` | no | Whether to append a trailing space to text before sending to TTS. This helps prevent some TTS services from vocalizing trailing punctuation (e.g., "dot"). Only applied in sentence aggregation mode; when streaming tokens, the incoming text's own whitespace is preserved. | | `sampleRate` | `number \| null \| undefined` | no | Output sample rate for generated audio. | | `skipTextTypes` | `string[] \| null \| undefined` | no | List of text types that should not be spoken. | | `textTransforms` | `[string, TextTransform][] \| null \| undefined` | no | A list of TextTransforms to transform text just before sending it to TTS. Each TextTransform takes the aggregated text and its type, and returns the transformed text. To register, provide a list of tuples of `[textType \| "*", transformFunction]`. | | `transportDestination` | `string \| null \| undefined` | no | Destination for generated audio frames. | | `reuseContextIdWithinTurn` | `boolean \| undefined` | no | Whether the service should reuse context IDs within the same turn. | | `audioCache` | `TTSAudioCacheStore \| null \| undefined` | no | Where to keep synthesized audio for reuse (e.g. a shared `TTSAudioCache`). Text whose audio is in the cache is served from it, with no request to the provider. Lines get there through `warmAudioCache()`, and also from live synthesis when `cacheSynthesizedAudio` allows it. Only services that yield their audio from `runTts` (HTTP and streaming-RPC services, see `supportsAudioCache`) use it, and only in sentence aggregation mode. Defaults to null (no cache). pipedog only. | | `cacheSynthesizedAudio` | `boolean \| ((text: string) => boolean) \| undefined` | no | Whether audio synthesized live is stored in `audioCache`, or a function that decides per text. Defaults to false: only warmed lines are stored, since a generated reply can repeat what the user said. pipedog only. | | `audioCacheNamespace` | `string \| undefined` | no | Extra part of the cache key, for constructor options that change the audio but are not in `settings`. pipedog only. | ### Settings [#settings-1] Passed as `settings: new RimeV1TTSService.Settings({...})`. `model` is required at construction. Each context uses the settings in effect when it starts. A setting the model does not accept is left out with a warning. | Setting | Type | Description | | -------------------------------------------------------------- | ------------------ | ------------------------------------------------------------------------------------------------------------ | | `model` | `string` | Required. `"coda"` and `"mistv3"` select their public endpoints. Changing it later is reported, not applied. | | `voice` | `string \| null` | Voice (`speaker`). | | `language` | `Language \| null` | Sent as a BCP-47 tag such as `en`. | | `timeScaleFactor` | `number \| null` | Playback speed. Not for mistv2. | | `textLookaheadTokens` | `number \| null` | Tokens to collect before audio starts (Coda only). | | `pauseBetweenBrackets`, `phonemizeBetweenBrackets`, `saveOovs` | `boolean \| null` | Mist options. | ### Usage [#usage-2] ```ts title="v1.ts" import { TextAggregationMode } from "@pipedog/core/services/tts_service"; import { RimeV1TTSService } from "@pipedog/rime/tts_v1"; const v1 = new RimeV1TTSService({ apiKey: process.env.RIME_API_KEY ?? "", // "binary" (protobuf) or "json" (proto3 JSON). Binary is the default. websocketProtocol: "binary", // Stream LLM tokens and let Rime accumulate them into sentences. textAggregationMode: TextAggregationMode.TOKEN, // The model is required: "coda" and "mistv3" select Rime's public endpoints. settings: new RimeV1TTSService.Settings({ model: "coda", voice: "luna" }), }); ``` ### Notes [#notes-2] * Pass `websocketUrl` for a custom deployment. The model is never inferred from the URL. * Rime v1 sends a `ready` event before the first context. The service waits up to 10 seconds for it and reports a connection error otherwise. * A failed context is reported with the error's `ErrorCategory`, and the service stays usable for the next turn. * Rime v1 does not report word timestamps. ## Runtime notes [#runtime-notes] * The WebSocket services use the global `WebSocket` and send the key as an `Authorization` header on the handshake. That works on Bun, Node and Deno, and on Cloudflare Workers, where the socket is opened through `fetch`. * `RimeV1TTSService` reads the negotiated subprotocol (`rime.v1.binary` or `rime.v1.json`) from the socket and refuses a connection that did not accept it. * The protobuf messages come from Rime's own package, `@rimelabs/api`, with `@bufbuild/protobuf`. Both are plain JavaScript. * `RimeHttpTTSService` uses `fetch`. Upstream's required `aiohttp_session` is replaced by an optional `fetch` option. * Not ported: the deprecated `RimeNonJsonTTSService` and the `InputParams` classes, and the `voiceId`, `model`, `params` and `aggregateSentences` constructor options. * Whole-number floats, such as `PAUSE_TAG(0.5)`'s `500`, print without Python's `.0`. | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/rime/package.json`: ✅ means the compat smoke test passes on that runtime. # Sarvam AI > Text-to-speech for Indian languages: SarvamTTSService over WebSocket and SarvamHttpTTSService over HTTP. Source: https://polvallverdu.github.io/pipedog/docs/services/tts/sarvam/ ## Overview [#overview] * **`SarvamTTSService`**: streams text over Sarvam's WebSocket and pushes audio as it arrives. Interruptions reconnect the socket. * **`SarvamHttpTTSService`**: one HTTP request per sentence. Pipecat's documentation for the same service. `packages/sarvam/src/tts.ts`. The text-to-speech API reference. ## Installation [#installation] ```bash bun add @pipedog/sarvam ``` ```ts import { SarvamHttpTTSService, SarvamTTSService } from "@pipedog/sarvam/tts"; ``` ## Prerequisites [#prerequisites] * A Sarvam account and an API key from the [Sarvam dashboard](https://dashboard.sarvam.ai/). * Pass the key as `apiKey`. The examples read it from `SARVAM_API_KEY`. ## Models [#models] | Model | Pitch and loudness | Temperature | Pace | Sample rate | | --------------------------------------- | ------------------ | ----------- | ---------- | ------------------- | | `bulbul:v3` (default), `bulbul:v3-beta` | ignored | 0.01 to 1.0 | 0.5 to 2.0 | 24000 Hz by default | | `bulbul:v4-flash` | supported | ignored | 0.5 to 2.0 | 24000 Hz by default | `TTS_MODEL_CONFIGS` holds each model's speakers and limits, and `getSpeakersForModel(model)` lists its speakers. Preprocessing is always on for these models. Settings a model doesn't support are dropped or clamped when the service is constructed. ## SarvamTTSService [#sarvamttsservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | Sarvam API key for authenticating TTS requests. | | `url` | `string \| undefined` | no | WebSocket URL for the TTS backend (default production URL). | | `settings` | `SarvamTTSSettings \| null \| undefined` | no | Runtime-updatable settings. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `reconnectBackoffMinWait` | `number \| undefined` | no | Minimum time, in seconds, to wait between reconnection attempts. | | `reconnectBackoffMaxWait` | `number \| undefined` | no | Maximum time, in seconds, to wait between reconnection attempts. | | `reconnectOnError` | `boolean \| undefined` | no | Whether to automatically reconnect on connection errors. | | `wsCloseTimeout` | `number \| undefined` | no | Maximum time, in seconds, to wait for the peer to acknowledge the websocket closing handshake before dropping the connection. Applied to connections opened through `_websocketConnect()`. Increase it for peers that need longer to complete a graceful close. | | `wsSendTimeout` | `number \| null \| undefined` | no | Maximum time, in seconds, a single send may wait on a peer that has stopped reading before the connection is dropped and the send throws `ConnectionClosedError`. Applied to connections opened through `_websocketConnect()`. Pass `null` to wait indefinitely. Defaults to `WS_SEND_TIMEOUT`. | | `textAggregationMode` | `TextAggregationMode \| null \| undefined` | no | How to aggregate incoming text before synthesis. `TextAggregationMode.SENTENCE` (default) buffers until sentence boundaries, `TextAggregationMode.TOKEN` streams tokens directly for lower latency. | | `pushTextFrames` | `boolean \| undefined` | no | Whether to push TextFrames and LLMFullResponseEndFrames. | | `pushStopFrames` | `boolean \| undefined` | no | Whether to automatically push TTSStoppedFrames. | | `pushStartFrame` | `boolean \| undefined` | no | Whether to automatically create audio contexts and push TTSStartedFrames. When true, the base class handles `createAudioContext` and yields `TTSStartedFrame` before each synthesis call, so `runTts` implementations do not need to. | | `stopFrameTimeoutS` | `number \| undefined` | no | Idle time before pushing TTSStoppedFrame when pushStopFrames is true. | | `pushSilenceAfterStop` | `boolean \| undefined` | no | Whether to push silence audio after TTSStoppedFrame. | | `silenceTimeS` | `number \| undefined` | no | Duration of silence to push when pushSilenceAfterStop is true. | | `pauseFrameProcessing` | `boolean \| undefined` | no | Whether to hold incoming frames after a turn until its audio contexts end. Needed by services whose audio doesn't carry its request's context ID. | | `maxConsecutiveZeroAudioContexts` | `number \| undefined` | no | How many consecutive TTS contexts may complete without producing any audio before the service is reported unable to do its job. Catches a provider that accepts requests and stays silent — an unknown voice ID, say — which no error ever surfaces. Every silent context reports an error the service can carry on from, whatever the limit is; on reaching the limit the service reports a permanent error instead, stops being given work, and the pipeline worker applies its `ProcessorUnusablePolicy`. Set to 0 to report silent contexts without ever writing the service off. | | `appendTrailingSpace` | `boolean \| undefined` | no | Whether to append a trailing space to text before sending to TTS. This helps prevent some TTS services from vocalizing trailing punctuation (e.g., "dot"). Only applied in sentence aggregation mode; when streaming tokens, the incoming text's own whitespace is preserved. | | `sampleRate` | `number \| null \| undefined` | no | Output sample rate for generated audio. | | `skipTextTypes` | `string[] \| null \| undefined` | no | List of text types that should not be spoken. | | `textTransforms` | `[string, TextTransform][] \| null \| undefined` | no | A list of TextTransforms to transform text just before sending it to TTS. Each TextTransform takes the aggregated text and its type, and returns the transformed text. To register, provide a list of tuples of `[textType \| "*", transformFunction]`. | | `transportDestination` | `string \| null \| undefined` | no | Destination for generated audio frames. | | `reuseContextIdWithinTurn` | `boolean \| undefined` | no | Whether the service should reuse context IDs within the same turn. | | `audioCache` | `TTSAudioCacheStore \| null \| undefined` | no | Where to keep synthesized audio for reuse (e.g. a shared `TTSAudioCache`). Text whose audio is in the cache is served from it, with no request to the provider. Lines get there through `warmAudioCache()`, and also from live synthesis when `cacheSynthesizedAudio` allows it. Only services that yield their audio from `runTts` (HTTP and streaming-RPC services, see `supportsAudioCache`) use it, and only in sentence aggregation mode. Defaults to null (no cache). pipedog only. | | `cacheSynthesizedAudio` | `boolean \| ((text: string) => boolean) \| undefined` | no | Whether audio synthesized live is stored in `audioCache`, or a function that decides per text. Defaults to false: only warmed lines are stored, since a generated reply can repeat what the user said. pipedog only. | | `audioCacheNamespace` | `string \| undefined` | no | Extra part of the cache key, for constructor options that change the audio but are not in `settings`. pipedog only. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new SarvamTTSService.Settings({...})` (`SarvamTTSSettings`). Any setting you leave out keeps the default below. | Setting | Type | Default | Description | | --------------------- | ----------------- | ------------- | ------------------------------------------------------------------- | | `model` | `string` | `"bulbul:v3"` | TTS model. | | `voice` | `string` | `"shubh"` | Speaker. | | `language` | `Language` | `"en-IN"` | Language of the text. | | `enablePreprocessing` | `boolean \| null` | `false` | Text preprocessing. Always on for v3 and v4-flash. | | `pace` | `number \| null` | `1.0` | Speech pace, 0.5 to 2.0. | | `pitch` | `number \| null` | `null` | Pitch (`bulbul:v4-flash`). | | `loudness` | `number \| null` | `null` | Volume multiplier (`bulbul:v4-flash`). | | `temperature` | `number \| null` | `null` | Randomness, 0.01 to 1.0 (`bulbul:v3`; the server's default is 0.6). | | `minBufferSize` | `number \| null` | `50` | Characters to buffer before generating audio. | | `maxChunkLength` | `number \| null` | `150` | Largest chunk, in characters, processed at once. | ### Usage [#usage] ```ts title="tts.ts" import { TTSUpdateSettingsFrame } from "@pipedog/core/frames/frames"; import { Language } from "@pipedog/core/transcriptions/language"; import { SarvamHttpTTSService, SarvamTTSService } from "@pipedog/sarvam/tts"; const tts = new SarvamTTSService({ apiKey: process.env.SARVAM_API_KEY ?? "", settings: new SarvamTTSService.Settings({ model: "bulbul:v3", voice: "shubh", language: Language.HI_IN, }), }); ``` Change settings mid-call with a `TTSUpdateSettingsFrame`: ```ts // Queue it on the worker, or push it from a processor upstream of the service. const update = new TTSUpdateSettingsFrame({ delta: new SarvamTTSService.Settings({ pace: 1.2 }), }); ``` ### Notes [#notes] * The socket carries `config`, `text`, `flush` and `ping` messages. The API key and a `User-Agent` go in the handshake headers. * `sampleRate` can be 8000, 16000, 22050 or 24000; `null` uses the model's default. ## SarvamHttpTTSService [#sarvamhttpttsservice] ### Configuration [#configuration-1] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | Sarvam AI API subscription key. | | `baseUrl` | `string \| undefined` | no | Sarvam AI API base URL. Defaults to "https://api.sarvam.ai". | | `fetch` | `{ (input: RequestInfo \| URL, init?: RequestInit): Promise; (input: string \| URL \| Request, init?: RequestInit): Promise; } \| null \| undefined` | no | `fetch` used for HTTP requests. Defaults to the global `fetch`. DIVERGENCE: upstream takes a shared `aiohttp_session`. | | `settings` | `SarvamHttpTTSSettings \| null \| undefined` | no | Runtime-updatable settings. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `textAggregationMode` | `TextAggregationMode \| null \| undefined` | no | How to aggregate incoming text before synthesis. `TextAggregationMode.SENTENCE` (default) buffers until sentence boundaries, `TextAggregationMode.TOKEN` streams tokens directly for lower latency. | | `pushTextFrames` | `boolean \| undefined` | no | Whether to push TextFrames and LLMFullResponseEndFrames. | | `pushStopFrames` | `boolean \| undefined` | no | Whether to automatically push TTSStoppedFrames. | | `pushStartFrame` | `boolean \| undefined` | no | Whether to automatically create audio contexts and push TTSStartedFrames. When true, the base class handles `createAudioContext` and yields `TTSStartedFrame` before each synthesis call, so `runTts` implementations do not need to. | | `stopFrameTimeoutS` | `number \| undefined` | no | Idle time before pushing TTSStoppedFrame when pushStopFrames is true. | | `pushSilenceAfterStop` | `boolean \| undefined` | no | Whether to push silence audio after TTSStoppedFrame. | | `silenceTimeS` | `number \| undefined` | no | Duration of silence to push when pushSilenceAfterStop is true. | | `pauseFrameProcessing` | `boolean \| undefined` | no | Whether to hold incoming frames after a turn until its audio contexts end. Needed by services whose audio doesn't carry its request's context ID. | | `maxConsecutiveZeroAudioContexts` | `number \| undefined` | no | How many consecutive TTS contexts may complete without producing any audio before the service is reported unable to do its job. Catches a provider that accepts requests and stays silent — an unknown voice ID, say — which no error ever surfaces. Every silent context reports an error the service can carry on from, whatever the limit is; on reaching the limit the service reports a permanent error instead, stops being given work, and the pipeline worker applies its `ProcessorUnusablePolicy`. Set to 0 to report silent contexts without ever writing the service off. | | `appendTrailingSpace` | `boolean \| undefined` | no | Whether to append a trailing space to text before sending to TTS. This helps prevent some TTS services from vocalizing trailing punctuation (e.g., "dot"). Only applied in sentence aggregation mode; when streaming tokens, the incoming text's own whitespace is preserved. | | `sampleRate` | `number \| null \| undefined` | no | Output sample rate for generated audio. | | `skipTextTypes` | `string[] \| null \| undefined` | no | List of text types that should not be spoken. | | `textTransforms` | `[string, TextTransform][] \| null \| undefined` | no | A list of TextTransforms to transform text just before sending it to TTS. Each TextTransform takes the aggregated text and its type, and returns the transformed text. To register, provide a list of tuples of `[textType \| "*", transformFunction]`. | | `transportDestination` | `string \| null \| undefined` | no | Destination for generated audio frames. | | `reuseContextIdWithinTurn` | `boolean \| undefined` | no | Whether the service should reuse context IDs within the same turn. | | `audioCache` | `TTSAudioCacheStore \| null \| undefined` | no | Where to keep synthesized audio for reuse (e.g. a shared `TTSAudioCache`). Text whose audio is in the cache is served from it, with no request to the provider. Lines get there through `warmAudioCache()`, and also from live synthesis when `cacheSynthesizedAudio` allows it. Only services that yield their audio from `runTts` (HTTP and streaming-RPC services, see `supportsAudioCache`) use it, and only in sentence aggregation mode. Defaults to null (no cache). pipedog only. | | `cacheSynthesizedAudio` | `boolean \| ((text: string) => boolean) \| undefined` | no | Whether audio synthesized live is stored in `audioCache`, or a function that decides per text. Defaults to false: only warmed lines are stored, since a generated reply can repeat what the user said. pipedog only. | | `audioCacheNamespace` | `string \| undefined` | no | Extra part of the cache key, for constructor options that change the audio but are not in `settings`. pipedog only. | ### Settings [#settings-1] `settings: new SarvamHttpTTSService.Settings({...})` (`SarvamHttpTTSSettings`) takes the same fields as above except `minBufferSize` and `maxChunkLength`, with the same defaults. ### Usage [#usage-1] ```ts const http = new SarvamHttpTTSService({ apiKey: process.env.SARVAM_API_KEY ?? "", settings: new SarvamHttpTTSService.Settings({ voice: "shubh", pace: 1.1 }), }); ``` ### Notes [#notes-1] * `fetch` replaces the global `fetch` (upstream takes an aiohttp session). * `baseUrl` defaults to `https://api.sarvam.ai`. The WAV header of each reply is stripped before the audio is pushed. * `sampleRate` can also be 32000, 44100 or 48000 for `bulbul:v3`. | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/sarvam/package.json`: ✅ means the compat smoke test passes on that runtime. ## Runtime notes [#runtime-notes] * HTTP goes through `fetch` and the WebSocket through the core `websocketConnect`. Both work on Bun, Node, Deno and Cloudflare Workers. * `bulbul:v2`, the `InputParams` class and the `model`, `voice_id`, `params` and `aggregate_sentences` constructor options are deprecated upstream and not ported. * Tested against local servers, not against Sarvam itself. # Smallest AI > Realtime text-to-speech with SmallestTTSService, over Smallest AI's WebSocket API, with word timestamps. Source: https://polvallverdu.github.io/pipedog/docs/services/tts/smallest/ ## Overview [#overview] * **`SmallestTTSService`**: streams text over a WebSocket and pushes audio as it arrives. Fragments of one LLM turn share a continuation context, so the voice keeps its prosody across them. An interruption reconnects the socket. Pipecat's documentation for the same service. `packages/smallest/src/tts.ts`. The text-to-speech API reference. ## Installation [#installation] ```bash bun add @pipedog/smallest ``` ```ts import { SmallestTTSService } from "@pipedog/smallest/tts"; ``` ## Prerequisites [#prerequisites] * A Smallest AI account and an API key from [Smallest AI](https://smallest.ai/). * Pass the key as `apiKey`. The examples read it from `SMALLEST_API_KEY`. ## SmallestTTSService [#smallestttsservice] Connects to `wss://api.smallest.ai`, with the key as a Bearer token on the handshake. ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `textAggregationMode` | `TextAggregationMode \| null \| undefined` | no | How to aggregate incoming text before synthesis. `TextAggregationMode.SENTENCE` (default) buffers until sentence boundaries, `TextAggregationMode.TOKEN` streams tokens directly for lower latency. | | `apiKey` | `string` | yes | Smallest AI API key for authentication. | | `baseUrl` | `string \| undefined` | no | Base WebSocket URL for the Smallest API. Defaults to "wss://api.smallest.ai". | | `outputFormat` | `string \| undefined` | no | Audio format returned by the API. One of `pcm`, `mp3`, `wav`, `ulaw`, `alaw`. Defaults to `pcm`, which is what the pipeline expects internally. Fixed at init time. | | `wordTimestamps` | `boolean \| undefined` | no | Whether to request per-word timing events, enabled by default. When true, the server interleaves `word_timestamp` messages and the service emits aligned per-word `TTSTextFrame`s. Supported on base-queue English + Hindi voices (`meher`, `devansh`, `kartik`, `maithili`, `liam`, `avery`); other voices silently emit no word events, so leaving this on is safe regardless of voice. Fixed at init time because it determines whether text frames are produced from word timing or pushed whole. | | `maxBufferDelayMs` | `number \| null \| undefined` | no | Upper bound, in milliseconds (0-5000), on how long the server buffers a continuation fragment before speaking it. If null, the server's default (3000ms) applies. | | `settings` | `SmallestTTSSettings \| null \| undefined` | no | Runtime-updatable settings for the TTS service. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `reconnectBackoffMinWait` | `number \| undefined` | no | Minimum time, in seconds, to wait between reconnection attempts. | | `reconnectBackoffMaxWait` | `number \| undefined` | no | Maximum time, in seconds, to wait between reconnection attempts. | | `reconnectOnError` | `boolean \| undefined` | no | Whether to automatically reconnect on connection errors. | | `wsCloseTimeout` | `number \| undefined` | no | Maximum time, in seconds, to wait for the peer to acknowledge the websocket closing handshake before dropping the connection. Applied to connections opened through `_websocketConnect()`. Increase it for peers that need longer to complete a graceful close. | | `wsSendTimeout` | `number \| null \| undefined` | no | Maximum time, in seconds, a single send may wait on a peer that has stopped reading before the connection is dropped and the send throws `ConnectionClosedError`. Applied to connections opened through `_websocketConnect()`. Pass `null` to wait indefinitely. Defaults to `WS_SEND_TIMEOUT`. | | `pushTextFrames` | `boolean \| undefined` | no | Whether to push TextFrames and LLMFullResponseEndFrames. | | `pushStopFrames` | `boolean \| undefined` | no | Whether to automatically push TTSStoppedFrames. | | `pushStartFrame` | `boolean \| undefined` | no | Whether to automatically create audio contexts and push TTSStartedFrames. When true, the base class handles `createAudioContext` and yields `TTSStartedFrame` before each synthesis call, so `runTts` implementations do not need to. | | `stopFrameTimeoutS` | `number \| undefined` | no | Idle time before pushing TTSStoppedFrame when pushStopFrames is true. | | `pushSilenceAfterStop` | `boolean \| undefined` | no | Whether to push silence audio after TTSStoppedFrame. | | `silenceTimeS` | `number \| undefined` | no | Duration of silence to push when pushSilenceAfterStop is true. | | `pauseFrameProcessing` | `boolean \| undefined` | no | Whether to hold incoming frames after a turn until its audio contexts end. Needed by services whose audio doesn't carry its request's context ID. | | `maxConsecutiveZeroAudioContexts` | `number \| undefined` | no | How many consecutive TTS contexts may complete without producing any audio before the service is reported unable to do its job. Catches a provider that accepts requests and stays silent — an unknown voice ID, say — which no error ever surfaces. Every silent context reports an error the service can carry on from, whatever the limit is; on reaching the limit the service reports a permanent error instead, stops being given work, and the pipeline worker applies its `ProcessorUnusablePolicy`. Set to 0 to report silent contexts without ever writing the service off. | | `appendTrailingSpace` | `boolean \| undefined` | no | Whether to append a trailing space to text before sending to TTS. This helps prevent some TTS services from vocalizing trailing punctuation (e.g., "dot"). Only applied in sentence aggregation mode; when streaming tokens, the incoming text's own whitespace is preserved. | | `sampleRate` | `number \| null \| undefined` | no | Output sample rate for generated audio. | | `skipTextTypes` | `string[] \| null \| undefined` | no | List of text types that should not be spoken. | | `textTransforms` | `[string, TextTransform][] \| null \| undefined` | no | A list of TextTransforms to transform text just before sending it to TTS. Each TextTransform takes the aggregated text and its type, and returns the transformed text. To register, provide a list of tuples of `[textType \| "*", transformFunction]`. | | `transportDestination` | `string \| null \| undefined` | no | Destination for generated audio frames. | | `reuseContextIdWithinTurn` | `boolean \| undefined` | no | Whether the service should reuse context IDs within the same turn. | | `audioCache` | `TTSAudioCacheStore \| null \| undefined` | no | Where to keep synthesized audio for reuse (e.g. a shared `TTSAudioCache`). Text whose audio is in the cache is served from it, with no request to the provider. Lines get there through `warmAudioCache()`, and also from live synthesis when `cacheSynthesizedAudio` allows it. Only services that yield their audio from `runTts` (HTTP and streaming-RPC services, see `supportsAudioCache`) use it, and only in sentence aggregation mode. Defaults to null (no cache). pipedog only. | | `cacheSynthesizedAudio` | `boolean \| ((text: string) => boolean) \| undefined` | no | Whether audio synthesized live is stored in `audioCache`, or a function that decides per text. Defaults to false: only warmed lines are stored, since a generated reply can repeat what the user said. pipedog only. | | `audioCacheNamespace` | `string \| undefined` | no | Extra part of the cache key, for constructor options that change the audio but are not in `settings`. pipedog only. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new SmallestTTSService.Settings({...})` (`SmallestTTSSettings`). Any setting you leave out keeps the default below. | Setting | Type | Default | Description | | --------------- | -------------------------------------------------- | ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `model` | `"lightning_v3.1" \| "lightning_v3.1_pro"` | `"lightning_v3.1_pro"` | Speech model. | | `voice` | `string` | `"meher"` (`"sophia"` for `lightning_v3.1`) | Voice. | | `language` | `Language` | `Language.EN` | Language. | | `speed` | `number \| null` | `null` | Speed multiplier, 0.5 to 2.0. `null` uses the API default. | | `mathNotation` | `boolean \| null` | `null` | Read operators between digits as words ("2 + 2" as "two plus two"). | | `contentFilter` | `"reject" \| "flag" \| "redact" \| string \| null` | `null` | Check the text against a profanity list; `"reject"` fails the request, `"flag"` synthesizes and records the match, `"redact"` drops the matched term and speaks the rest. | | `dateFormat` | `"EU" \| "US" \| string \| null` | `null` | Read all-numeric dates day-first (`"EU"`) or month-first (`"US"`). | ### Usage [#usage] ```ts title="tts.ts" import { TTSUpdateSettingsFrame } from "@pipedog/core/frames/frames"; import { Language } from "@pipedog/core/transcriptions/language"; import { SmallestTTSService } from "@pipedog/smallest/tts"; const tts = new SmallestTTSService({ apiKey: process.env.SMALLEST_API_KEY ?? "", settings: new SmallestTTSService.Settings({ voice: "sophia", language: Language.EN }), }); ``` Change settings mid-call with a `TTSUpdateSettingsFrame`: ```ts // Queue it on the worker, or push it from a processor upstream of the service. const update = new TTSUpdateSettingsFrame({ delta: new SmallestTTSService.Settings({ speed: 1.1 }), }); ``` ### Notes [#notes] * `outputFormat` is `"pcm"` by default; it is fixed when the service is created. * `wordTimestamps` (on by default) makes the service push aligned per-word `TTSTextFrame`s. It works on `meher`, `devansh`, `kartik`, `maithili`, `liam` and `avery`; other voices send no word events, so leaving it on is safe. * `maxBufferDelayMs` (0 to 5000) bounds how long the server holds a continuation fragment. `null` keeps the server's 3000 ms. * The same package has Smallest AI's [LLM](https://polvallverdu.github.io/pipedog/docs/services/llm/smallest.md) and [speech-to-text](https://polvallverdu.github.io/pipedog/docs/services/stt/smallest.md) services. | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/smallest/package.json`: ✅ means the compat smoke test passes on that runtime. ## Runtime notes [#runtime-notes] * The WebSocket goes through the core `websocketConnect`, with headers on the handshake. * Tested against local WebSocket servers, as upstream's tests do, not against Smallest AI itself. # Soniox > Streaming text-to-speech with Soniox. Source: https://polvallverdu.github.io/pipedog/docs/services/tts/soniox/ ## Overview [#overview] `SonioxTTSService` streams text to Soniox's real-time TTS WebSocket and yields PCM audio, with keepalive and stream reuse across turns. Pipecat's documentation for the same service. `packages/soniox/src`. ## Installation [#installation] ```bash bun add @pipedog/soniox ``` ## Prerequisites [#prerequisites] * A [Soniox](https://soniox.com) account and API key, passed as `apiKey` (the examples read `SONIOX_API_KEY`). ## SonioxTTSService [#sonioxttsservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | Soniox API key for authentication. Create API keys at https://console.soniox.com. | | `url` | `string \| undefined` | no | WebSocket URL for the Soniox TTS endpoint. | | `audioFormat` | `string \| undefined` | no | Output audio format. Defaults to "pcm_s16le", which matches the downstream audio pipeline. | | `settings` | `SonioxTTSSettings \| null \| undefined` | no | Runtime-updatable settings. | | `textAggregationMode` | `TextAggregationMode \| null \| undefined` | no | How to aggregate incoming text before synthesis. Defaults to `TextAggregationMode.SENTENCE`. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `reconnectBackoffMinWait` | `number \| undefined` | no | Minimum time, in seconds, to wait between reconnection attempts. | | `reconnectBackoffMaxWait` | `number \| undefined` | no | Maximum time, in seconds, to wait between reconnection attempts. | | `reconnectOnError` | `boolean \| undefined` | no | Whether to automatically reconnect on connection errors. | | `wsCloseTimeout` | `number \| undefined` | no | Maximum time, in seconds, to wait for the peer to acknowledge the websocket closing handshake before dropping the connection. Applied to connections opened through `_websocketConnect()`. Increase it for peers that need longer to complete a graceful close. | | `wsSendTimeout` | `number \| null \| undefined` | no | Maximum time, in seconds, a single send may wait on a peer that has stopped reading before the connection is dropped and the send throws `ConnectionClosedError`. Applied to connections opened through `_websocketConnect()`. Pass `null` to wait indefinitely. Defaults to `WS_SEND_TIMEOUT`. | | `pushTextFrames` | `boolean \| undefined` | no | Whether to push TextFrames and LLMFullResponseEndFrames. | | `pushStopFrames` | `boolean \| undefined` | no | Whether to automatically push TTSStoppedFrames. | | `pushStartFrame` | `boolean \| undefined` | no | Whether to automatically create audio contexts and push TTSStartedFrames. When true, the base class handles `createAudioContext` and yields `TTSStartedFrame` before each synthesis call, so `runTts` implementations do not need to. | | `stopFrameTimeoutS` | `number \| undefined` | no | Idle time before pushing TTSStoppedFrame when pushStopFrames is true. | | `pushSilenceAfterStop` | `boolean \| undefined` | no | Whether to push silence audio after TTSStoppedFrame. | | `silenceTimeS` | `number \| undefined` | no | Duration of silence to push when pushSilenceAfterStop is true. | | `pauseFrameProcessing` | `boolean \| undefined` | no | Whether to hold incoming frames after a turn until its audio contexts end. Needed by services whose audio doesn't carry its request's context ID. | | `maxConsecutiveZeroAudioContexts` | `number \| undefined` | no | How many consecutive TTS contexts may complete without producing any audio before the service is reported unable to do its job. Catches a provider that accepts requests and stays silent — an unknown voice ID, say — which no error ever surfaces. Every silent context reports an error the service can carry on from, whatever the limit is; on reaching the limit the service reports a permanent error instead, stops being given work, and the pipeline worker applies its `ProcessorUnusablePolicy`. Set to 0 to report silent contexts without ever writing the service off. | | `appendTrailingSpace` | `boolean \| undefined` | no | Whether to append a trailing space to text before sending to TTS. This helps prevent some TTS services from vocalizing trailing punctuation (e.g., "dot"). Only applied in sentence aggregation mode; when streaming tokens, the incoming text's own whitespace is preserved. | | `sampleRate` | `number \| null \| undefined` | no | Output sample rate for generated audio. | | `skipTextTypes` | `string[] \| null \| undefined` | no | List of text types that should not be spoken. | | `textTransforms` | `[string, TextTransform][] \| null \| undefined` | no | A list of TextTransforms to transform text just before sending it to TTS. Each TextTransform takes the aggregated text and its type, and returns the transformed text. To register, provide a list of tuples of `[textType \| "*", transformFunction]`. | | `transportDestination` | `string \| null \| undefined` | no | Destination for generated audio frames. | | `reuseContextIdWithinTurn` | `boolean \| undefined` | no | Whether the service should reuse context IDs within the same turn. | | `audioCache` | `TTSAudioCacheStore \| null \| undefined` | no | Where to keep synthesized audio for reuse (e.g. a shared `TTSAudioCache`). Text whose audio is in the cache is served from it, with no request to the provider. Lines get there through `warmAudioCache()`, and also from live synthesis when `cacheSynthesizedAudio` allows it. Only services that yield their audio from `runTts` (HTTP and streaming-RPC services, see `supportsAudioCache`) use it, and only in sentence aggregation mode. Defaults to null (no cache). pipedog only. | | `cacheSynthesizedAudio` | `boolean \| ((text: string) => boolean) \| undefined` | no | Whether audio synthesized live is stored in `audioCache`, or a function that decides per text. Defaults to false: only warmed lines are stored, since a generated reply can repeat what the user said. pipedog only. | | `audioCacheNamespace` | `string \| undefined` | no | Extra part of the cache key, for constructor options that change the audio but are not in `settings`. pipedog only. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new SonioxTTSService.Settings({...})`. Settings you leave out keep their defaults. | Setting | Default | Description | | ------------------- | ------------- | ----------------------------- | | `model` | `"tts-rt-v2"` | Soniox TTS model. | | `voice` | `"Bryce"` | Voice. | | `language` | `Language.EN` | Language. | | `speed` | | Speaking speed. | | `reduceSilence` | | Not supported by every model. | | `clientReferenceId` | | Your own reference. | ### Usage [#usage] ```ts title="tts.ts" import { SonioxTTSService } from "@pipedog/soniox/tts"; const tts = new SonioxTTSService({ apiKey: process.env.SONIOX_API_KEY ?? "", settings: new SonioxTTSService.Settings({ voice: "Adrian", speed: 1.1 }), }); ``` ### Notes [#notes] * `sampleRate` must be one of `VALID_SAMPLE_RATES`. ## Runtimes [#runtimes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/soniox/package.json`: ✅ means the compat smoke test passes on that runtime. The service uses `fetch` and the global `WebSocket` (through core's `websocketConnect`, which sends the auth header on the handshake on Cloudflare Workers too). Pass no SDK; nothing is Node-only. # Speechify > Text-to-speech with Speechify: SpeechifyHttpTTSService streams audio and word timestamps over Server-Sent Events. Source: https://polvallverdu.github.io/pipedog/docs/services/tts/speechify/ ## Overview [#overview] `SpeechifyHttpTTSService` posts each sentence to Speechify's `/v1/audio/stream/with-timestamps` endpoint and reads the Server-Sent Events it answers with: PCM audio chunks and word-level speech marks. The marks become word timestamps, so the bot's `TTSTextFrame`s follow the audio. Pipecat's documentation for the same service. `packages/speechify/src/tts.ts`. The stream-with-timestamps API reference. ## Installation [#installation] ```bash bun add @pipedog/speechify ``` ```ts import { SpeechifyHttpTTSService } from "@pipedog/speechify/tts"; ``` ## Prerequisites [#prerequisites] * A Speechify API account and an API key. * Pass the key as `apiKey`. The examples read it from `SPEECHIFY_API_KEY`. ## Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | Speechify API key for authentication. | | `baseUrl` | `string \| undefined` | no | Base URL for the Speechify API. Defaults to "https://api.speechify.ai". | | `fetch` | `{ (input: RequestInfo \| URL, init?: RequestInit): Promise; (input: string \| URL \| Request, init?: RequestInit): Promise; } \| null \| undefined` | no | `fetch` used for HTTP requests. Defaults to the global `fetch`. DIVERGENCE: upstream takes a required shared `aiohttp_session`. | | `settings` | `SpeechifyTTSSettings \| null \| undefined` | no | Runtime-updatable settings. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `textAggregationMode` | `TextAggregationMode \| null \| undefined` | no | How to aggregate incoming text before synthesis. `TextAggregationMode.SENTENCE` (default) buffers until sentence boundaries, `TextAggregationMode.TOKEN` streams tokens directly for lower latency. | | `pushTextFrames` | `boolean \| undefined` | no | Whether to push TextFrames and LLMFullResponseEndFrames. | | `pushStopFrames` | `boolean \| undefined` | no | Whether to automatically push TTSStoppedFrames. | | `pushStartFrame` | `boolean \| undefined` | no | Whether to automatically create audio contexts and push TTSStartedFrames. When true, the base class handles `createAudioContext` and yields `TTSStartedFrame` before each synthesis call, so `runTts` implementations do not need to. | | `stopFrameTimeoutS` | `number \| undefined` | no | Idle time before pushing TTSStoppedFrame when pushStopFrames is true. | | `pushSilenceAfterStop` | `boolean \| undefined` | no | Whether to push silence audio after TTSStoppedFrame. | | `silenceTimeS` | `number \| undefined` | no | Duration of silence to push when pushSilenceAfterStop is true. | | `pauseFrameProcessing` | `boolean \| undefined` | no | Whether to hold incoming frames after a turn until its audio contexts end. Needed by services whose audio doesn't carry its request's context ID. | | `maxConsecutiveZeroAudioContexts` | `number \| undefined` | no | How many consecutive TTS contexts may complete without producing any audio before the service is reported unable to do its job. Catches a provider that accepts requests and stays silent — an unknown voice ID, say — which no error ever surfaces. Every silent context reports an error the service can carry on from, whatever the limit is; on reaching the limit the service reports a permanent error instead, stops being given work, and the pipeline worker applies its `ProcessorUnusablePolicy`. Set to 0 to report silent contexts without ever writing the service off. | | `appendTrailingSpace` | `boolean \| undefined` | no | Whether to append a trailing space to text before sending to TTS. This helps prevent some TTS services from vocalizing trailing punctuation (e.g., "dot"). Only applied in sentence aggregation mode; when streaming tokens, the incoming text's own whitespace is preserved. | | `sampleRate` | `number \| null \| undefined` | no | Output sample rate for generated audio. | | `skipTextTypes` | `string[] \| null \| undefined` | no | List of text types that should not be spoken. | | `textTransforms` | `[string, TextTransform][] \| null \| undefined` | no | A list of TextTransforms to transform text just before sending it to TTS. Each TextTransform takes the aggregated text and its type, and returns the transformed text. To register, provide a list of tuples of `[textType \| "*", transformFunction]`. | | `transportDestination` | `string \| null \| undefined` | no | Destination for generated audio frames. | | `reuseContextIdWithinTurn` | `boolean \| undefined` | no | Whether the service should reuse context IDs within the same turn. | | `audioCache` | `TTSAudioCacheStore \| null \| undefined` | no | Where to keep synthesized audio for reuse (e.g. a shared `TTSAudioCache`). Text whose audio is in the cache is served from it, with no request to the provider. Lines get there through `warmAudioCache()`, and also from live synthesis when `cacheSynthesizedAudio` allows it. Only services that yield their audio from `runTts` (HTTP and streaming-RPC services, see `supportsAudioCache`) use it, and only in sentence aggregation mode. Defaults to null (no cache). pipedog only. | | `cacheSynthesizedAudio` | `boolean \| ((text: string) => boolean) \| undefined` | no | Whether audio synthesized live is stored in `audioCache`, or a function that decides per text. Defaults to false: only warmed lines are stored, since a generated reply can repeat what the user said. pipedog only. | | `audioCacheNamespace` | `string \| undefined` | no | Extra part of the cache key, for constructor options that change the audio but are not in `settings`. pipedog only. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new SpeechifyHttpTTSService.Settings({...})` (`SpeechifyTTSSettings`). Any setting you leave out keeps the default below. | Setting | Type | Default | Description | | ----------------------- | ------------------ | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `model` | `string` | `"simba-3.5"` | Speech model, such as `simba-3.5` or `simba-3.0` (multilingual). | | `voice` | `string` | `"geffen_32"` | Voice id. | | `language` | `Language \| null` | `null` | Language, sent as a tag such as `en-US`. | | `loudnessNormalization` | `boolean \| null` | `null` | Normalize the audio's loudness. Adds latency. | | `textNormalization` | `boolean \| null` | `null` | Spell out numbers, dates and similar tokens. Adds latency. | | `safetyIdentifier` | `string \| null` | `null` | A stable, opaque id for the end user (1 to 64 characters), so Speechify can limit one user instead of the workspace. Never an email address. | | `pronunciations` | `Dict[] \| null` | `null` | How specific words are spoken (up to 200), by IPA phones or a spoken alias. Applied on `simba-3.5` in the languages Speechify lists for pronunciations, which excludes English. | ## Usage [#usage] ```ts title="tts.ts" import { TTSUpdateSettingsFrame } from "@pipedog/core/frames/frames"; import { SpeechifyHttpTTSService } from "@pipedog/speechify/tts"; const tts = new SpeechifyHttpTTSService({ apiKey: process.env.SPEECHIFY_API_KEY ?? "", settings: new SpeechifyHttpTTSService.Settings({ model: "simba-3.5", voice: "geffen_32", textNormalization: true, }), }); ``` Change settings mid-call with a `TTSUpdateSettingsFrame`: ```ts // Queue it on the worker, or push it from a processor upstream of the service. const update = new TTSUpdateSettingsFrame({ delta: new SpeechifyHttpTTSService.Settings({ voice: "another-voice-id" }), }); ``` ## Notes [#notes] * The sample rate must be one of Speechify's PCM rates (8000, 16000, 22050, 24000, 44100 or 48000). For any other pipeline rate the service asks for 24000 Hz, tags its audio frames with that rate, and the output transport resamples. * Requests pin Speechify's API version (`Speechify-Version: 2026-09-30`), and send `Speechify-Caller: pipecat` with the pipedog version. * Speechify marks the pieces of a hyphenated word separately. The service joins marks that touch in the text, so `text-to-speech` is one word timestamp. * A mid-stream `speech.error` event is reported as an error frame. A non-200 response is too. ## Runtime notes [#runtime-notes] * It uses the global `fetch` and reads the response as a stream, so it works on Bun, Node, Deno and Cloudflare Workers. Pass `fetch` to use another one (upstream takes an aiohttp session). * Tested against a local server that replays Speechify's event format, as upstream's tests do for the parsing, not against Speechify itself. | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/speechify/package.json`: ✅ means the compat smoke test passes on that runtime. # Speechmatics > Text-to-speech with Speechmatics. Source: https://polvallverdu.github.io/pipedog/docs/services/tts/speechmatics/ ## Overview [#overview] `SpeechmaticsTTSService` calls Speechmatics' HTTP TTS endpoint (`POST /generate/?output_format=pcm_16000`) and streams the PCM body as audio frames. The endpoint is documented as a preview. Pipecat's documentation for the same service. `packages/speechmatics/src`. ## Installation [#installation] ```bash bun add @pipedog/speechmatics ``` ## Prerequisites [#prerequisites] * A [Speechmatics](https://www.speechmatics.com) API key, passed as `apiKey`. ## SpeechmaticsTTSService [#speechmaticsttsservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | Speechmatics API key for authentication. | | `baseUrl` | `string \| undefined` | no | Base URL for Speechmatics TTS API. Defaults to "https://preview.tts.speechmatics.com". | | `fetch` | `((input: RequestInfo \| URL, init?: RequestInit) => Promise) \| undefined` | no | `fetch` used for HTTP requests. Defaults to the global `fetch`. | | `sampleRate` | `number \| null \| undefined` | no | Audio sample rate in Hz. Speechmatics only supports 16000. | | `settings` | `SpeechmaticsTTSSettings \| null \| undefined` | no | Runtime-updatable settings. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `textAggregationMode` | `TextAggregationMode \| null \| undefined` | no | How to aggregate incoming text before synthesis. `TextAggregationMode.SENTENCE` (default) buffers until sentence boundaries, `TextAggregationMode.TOKEN` streams tokens directly for lower latency. | | `pushTextFrames` | `boolean \| undefined` | no | Whether to push TextFrames and LLMFullResponseEndFrames. | | `pushStopFrames` | `boolean \| undefined` | no | Whether to automatically push TTSStoppedFrames. | | `pushStartFrame` | `boolean \| undefined` | no | Whether to automatically create audio contexts and push TTSStartedFrames. When true, the base class handles `createAudioContext` and yields `TTSStartedFrame` before each synthesis call, so `runTts` implementations do not need to. | | `stopFrameTimeoutS` | `number \| undefined` | no | Idle time before pushing TTSStoppedFrame when pushStopFrames is true. | | `pushSilenceAfterStop` | `boolean \| undefined` | no | Whether to push silence audio after TTSStoppedFrame. | | `silenceTimeS` | `number \| undefined` | no | Duration of silence to push when pushSilenceAfterStop is true. | | `pauseFrameProcessing` | `boolean \| undefined` | no | Whether to hold incoming frames after a turn until its audio contexts end. Needed by services whose audio doesn't carry its request's context ID. | | `maxConsecutiveZeroAudioContexts` | `number \| undefined` | no | How many consecutive TTS contexts may complete without producing any audio before the service is reported unable to do its job. Catches a provider that accepts requests and stays silent — an unknown voice ID, say — which no error ever surfaces. Every silent context reports an error the service can carry on from, whatever the limit is; on reaching the limit the service reports a permanent error instead, stops being given work, and the pipeline worker applies its `ProcessorUnusablePolicy`. Set to 0 to report silent contexts without ever writing the service off. | | `appendTrailingSpace` | `boolean \| undefined` | no | Whether to append a trailing space to text before sending to TTS. This helps prevent some TTS services from vocalizing trailing punctuation (e.g., "dot"). Only applied in sentence aggregation mode; when streaming tokens, the incoming text's own whitespace is preserved. | | `skipTextTypes` | `string[] \| null \| undefined` | no | List of text types that should not be spoken. | | `textTransforms` | `[string, TextTransform][] \| null \| undefined` | no | A list of TextTransforms to transform text just before sending it to TTS. Each TextTransform takes the aggregated text and its type, and returns the transformed text. To register, provide a list of tuples of `[textType \| "*", transformFunction]`. | | `transportDestination` | `string \| null \| undefined` | no | Destination for generated audio frames. | | `reuseContextIdWithinTurn` | `boolean \| undefined` | no | Whether the service should reuse context IDs within the same turn. | | `audioCache` | `TTSAudioCacheStore \| null \| undefined` | no | Where to keep synthesized audio for reuse (e.g. a shared `TTSAudioCache`). Text whose audio is in the cache is served from it, with no request to the provider. Lines get there through `warmAudioCache()`, and also from live synthesis when `cacheSynthesizedAudio` allows it. Only services that yield their audio from `runTts` (HTTP and streaming-RPC services, see `supportsAudioCache`) use it, and only in sentence aggregation mode. Defaults to null (no cache). pipedog only. | | `cacheSynthesizedAudio` | `boolean \| ((text: string) => boolean) \| undefined` | no | Whether audio synthesized live is stored in `audioCache`, or a function that decides per text. Defaults to false: only warmed lines are stored, since a generated reply can repeat what the user said. pipedog only. | | `audioCacheNamespace` | `string \| undefined` | no | Extra part of the cache key, for constructor options that change the audio but are not in `settings`. pipedog only. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new SpeechmaticsTTSService.Settings({...})`. Settings you leave out keep their defaults. | Setting | Default | Description | | ------------ | --------- | -------------------- | | `voice` | `"sarah"` | Voice. | | `maxRetries` | `5` | Retries on HTTP 503. | ### Usage [#usage] ```ts title="tts.ts" import { SpeechmaticsTTSService } from "@pipedog/speechmatics/tts"; const tts = new SpeechmaticsTTSService({ apiKey: process.env.SPEECHMATICS_API_KEY ?? "", settings: new SpeechmaticsTTSService.Settings({ voice: "sarah" }), }); ``` ### Notes [#notes] * `sampleRate` is 16000 only. Pass `fetch` to use your own HTTP client. ## Runtimes [#runtimes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/speechmatics/package.json`: ✅ means the compat smoke test passes on that runtime. The service uses `fetch` only. # Together AI > Text-to-speech with TogetherTTSService, streaming over Together AI's WebSocket API. Source: https://polvallverdu.github.io/pipedog/docs/services/tts/together/ ## Overview [#overview] * **`TogetherTTSService`**: streams text over a WebSocket to Together's speech endpoint and pushes 24 kHz PCM audio as it arrives. Pipecat's documentation for the same service. `packages/together/src/tts.ts`. Voices, models and the WebSocket speech API. ## Installation [#installation] ```bash bun add @pipedog/together ``` ```ts import { TogetherTTSService } from "@pipedog/together/tts"; ``` ## Prerequisites [#prerequisites] * A Together AI account and an API key from [Together AI](https://together.ai/). * Pass the key as `apiKey`. The examples read it from `TOGETHER_API_KEY`. ## TogetherTTSService [#togetherttsservice] Connects to `wss://api.together.ai/v1/audio/speech/websocket`. ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | Together AI API key for authentication. | | `url` | `string \| undefined` | no | WebSocket URL for Together AI TTS API. Defaults to "wss://api.together.ai/v1/audio/speech/websocket". | | `sampleRate` | `number \| null \| undefined` | no | Output sample rate for emitted PCM frames. Defaults to 24000; Together streams at 24 kHz and does not support other rates. | | `settings` | `TogetherTTSSettings \| null \| undefined` | no | Runtime-updatable settings for the TTS service. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `reconnectBackoffMinWait` | `number \| undefined` | no | Minimum time, in seconds, to wait between reconnection attempts. | | `reconnectBackoffMaxWait` | `number \| undefined` | no | Maximum time, in seconds, to wait between reconnection attempts. | | `reconnectOnError` | `boolean \| undefined` | no | Whether to automatically reconnect on connection errors. | | `wsCloseTimeout` | `number \| undefined` | no | Maximum time, in seconds, to wait for the peer to acknowledge the websocket closing handshake before dropping the connection. Applied to connections opened through `_websocketConnect()`. Increase it for peers that need longer to complete a graceful close. | | `wsSendTimeout` | `number \| null \| undefined` | no | Maximum time, in seconds, a single send may wait on a peer that has stopped reading before the connection is dropped and the send throws `ConnectionClosedError`. Applied to connections opened through `_websocketConnect()`. Pass `null` to wait indefinitely. Defaults to `WS_SEND_TIMEOUT`. | | `textAggregationMode` | `TextAggregationMode \| null \| undefined` | no | How to aggregate incoming text before synthesis. `TextAggregationMode.SENTENCE` (default) buffers until sentence boundaries, `TextAggregationMode.TOKEN` streams tokens directly for lower latency. | | `pushTextFrames` | `boolean \| undefined` | no | Whether to push TextFrames and LLMFullResponseEndFrames. | | `pushStopFrames` | `boolean \| undefined` | no | Whether to automatically push TTSStoppedFrames. | | `pushStartFrame` | `boolean \| undefined` | no | Whether to automatically create audio contexts and push TTSStartedFrames. When true, the base class handles `createAudioContext` and yields `TTSStartedFrame` before each synthesis call, so `runTts` implementations do not need to. | | `stopFrameTimeoutS` | `number \| undefined` | no | Idle time before pushing TTSStoppedFrame when pushStopFrames is true. | | `pushSilenceAfterStop` | `boolean \| undefined` | no | Whether to push silence audio after TTSStoppedFrame. | | `silenceTimeS` | `number \| undefined` | no | Duration of silence to push when pushSilenceAfterStop is true. | | `pauseFrameProcessing` | `boolean \| undefined` | no | Whether to hold incoming frames after a turn until its audio contexts end. Needed by services whose audio doesn't carry its request's context ID. | | `maxConsecutiveZeroAudioContexts` | `number \| undefined` | no | How many consecutive TTS contexts may complete without producing any audio before the service is reported unable to do its job. Catches a provider that accepts requests and stays silent — an unknown voice ID, say — which no error ever surfaces. Every silent context reports an error the service can carry on from, whatever the limit is; on reaching the limit the service reports a permanent error instead, stops being given work, and the pipeline worker applies its `ProcessorUnusablePolicy`. Set to 0 to report silent contexts without ever writing the service off. | | `appendTrailingSpace` | `boolean \| undefined` | no | Whether to append a trailing space to text before sending to TTS. This helps prevent some TTS services from vocalizing trailing punctuation (e.g., "dot"). Only applied in sentence aggregation mode; when streaming tokens, the incoming text's own whitespace is preserved. | | `skipTextTypes` | `string[] \| null \| undefined` | no | List of text types that should not be spoken. | | `textTransforms` | `[string, TextTransform][] \| null \| undefined` | no | A list of TextTransforms to transform text just before sending it to TTS. Each TextTransform takes the aggregated text and its type, and returns the transformed text. To register, provide a list of tuples of `[textType \| "*", transformFunction]`. | | `transportDestination` | `string \| null \| undefined` | no | Destination for generated audio frames. | | `reuseContextIdWithinTurn` | `boolean \| undefined` | no | Whether the service should reuse context IDs within the same turn. | | `audioCache` | `TTSAudioCacheStore \| null \| undefined` | no | Where to keep synthesized audio for reuse (e.g. a shared `TTSAudioCache`). Text whose audio is in the cache is served from it, with no request to the provider. Lines get there through `warmAudioCache()`, and also from live synthesis when `cacheSynthesizedAudio` allows it. Only services that yield their audio from `runTts` (HTTP and streaming-RPC services, see `supportsAudioCache`) use it, and only in sentence aggregation mode. Defaults to null (no cache). pipedog only. | | `cacheSynthesizedAudio` | `boolean \| ((text: string) => boolean) \| undefined` | no | Whether audio synthesized live is stored in `audioCache`, or a function that decides per text. Defaults to false: only warmed lines are stored, since a generated reply can repeat what the user said. pipedog only. | | `audioCacheNamespace` | `string \| undefined` | no | Extra part of the cache key, for constructor options that change the audio but are not in `settings`. pipedog only. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new TogetherTTSService.Settings({...})` (`TogetherTTSSettings`). Any setting you leave out keeps the default below. | Setting | Type | Default | Description | | ------------------ | --------------------------------- | ---------------------- | -------------------------------------------------------------------------------- | | `model` | `string` | `"hexgrad/Kokoro-82M"` | Speech model, sent in the URL. | | `voice` | `string` | `"af_heart"` | Voice, sent in the URL. | | `language` | `Language \| null` | `Language.EN` | Language, sent in the URL. | | `maxPartialLength` | `number \| null` | `null` | Maximum partial text length for streaming (`max_partial_length`). | | `extraParams` | `Record \| null` | `null` | Model-specific options sent in the session update, such as `pronunciation_dict`. | ### Usage [#usage] ```ts title="tts.ts" import { TTSUpdateSettingsFrame } from "@pipedog/core/frames/frames"; import { TogetherTTSService } from "@pipedog/together/tts"; const tts = new TogetherTTSService({ apiKey: process.env.TOGETHER_API_KEY ?? "", settings: new TogetherTTSService.Settings({ voice: "af_heart" }), }); ``` Change settings mid-call with an `TTSUpdateSettingsFrame`: ```ts // Queue it on the worker, or push it from a processor upstream of the service. const update = new TTSUpdateSettingsFrame({ delta: new TogetherTTSService.Settings({ voice: "af_bella" }), }); ``` ### Event handlers [#event-handlers] | Event | Description | | ----------------- | ------------------------ | | `on_connected` | The WebSocket connected. | | `on_disconnected` | The WebSocket closed. | ### Notes [#notes] * Together sends 24 kHz audio. Any other `sampleRate` logs a warning. * An interruption clears the text buffer (`input_text_buffer.clear`); the end of a turn commits it. * Any settings change reconnects, since the session is configured in the connection URL. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/together/package.json`: ✅ means the compat smoke test passes on that runtime. * The WebSocket goes through the core `websocketConnect`. It works on Bun, Node, Deno and Cloudflare Workers. * Tested against a local WebSocket fake, not against Together AI itself. # xAI > Text-to-speech with xAI: XAITTSService over WebSocket, with word timestamps, and XAIHttpTTSService over HTTP. Source: https://polvallverdu.github.io/pipedog/docs/services/tts/xai/ ## Overview [#overview] * **`XAITTSService`**: streams text over a WebSocket and pushes audio as it arrives, with word timestamps from xAI's character timings. * **`XAIHttpTTSService`**: one HTTP request per sentence. Pipecat's documentation for the same service. `packages/xai/src/tts.ts`. The voice API reference. ## Installation [#installation] ```bash bun add @pipedog/xai ``` ```ts import { XAITTSService, XAIHttpTTSService } from "@pipedog/xai/tts"; ``` ## Prerequisites [#prerequisites] * An xAI account and an API key from the [xAI console](https://console.x.ai/). * Pass the key as `apiKey`. The examples read it from `XAI_API_KEY`. ## XAITTSService [#xaittsservice] Connects to `wss://api.x.ai/v1/tts`. ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | xAI API key for authentication. | | `baseUrl` | `string \| undefined` | no | xAI TTS WebSocket endpoint. Defaults to "wss://api.x.ai/v1/tts". | | `codec` | `string \| undefined` | no | Output audio codec. One of `pcm`, `wav`, `mulaw`, `alaw`. Defaults to `pcm` so emitted `TTSAudioRawFrame` objects need no decoding downstream. | | `settings` | `XAIWebsocketTTSSettings \| null \| undefined` | no | Runtime-updatable settings. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `reconnectBackoffMinWait` | `number \| undefined` | no | Minimum time, in seconds, to wait between reconnection attempts. | | `reconnectBackoffMaxWait` | `number \| undefined` | no | Maximum time, in seconds, to wait between reconnection attempts. | | `reconnectOnError` | `boolean \| undefined` | no | Whether to automatically reconnect on connection errors. | | `wsCloseTimeout` | `number \| undefined` | no | Maximum time, in seconds, to wait for the peer to acknowledge the websocket closing handshake before dropping the connection. Applied to connections opened through `_websocketConnect()`. Increase it for peers that need longer to complete a graceful close. | | `wsSendTimeout` | `number \| null \| undefined` | no | Maximum time, in seconds, a single send may wait on a peer that has stopped reading before the connection is dropped and the send throws `ConnectionClosedError`. Applied to connections opened through `_websocketConnect()`. Pass `null` to wait indefinitely. Defaults to `WS_SEND_TIMEOUT`. | | `textAggregationMode` | `TextAggregationMode \| null \| undefined` | no | How to aggregate incoming text before synthesis. `TextAggregationMode.SENTENCE` (default) buffers until sentence boundaries, `TextAggregationMode.TOKEN` streams tokens directly for lower latency. | | `pushTextFrames` | `boolean \| undefined` | no | Whether to push TextFrames and LLMFullResponseEndFrames. | | `pushStopFrames` | `boolean \| undefined` | no | Whether to automatically push TTSStoppedFrames. | | `pushStartFrame` | `boolean \| undefined` | no | Whether to automatically create audio contexts and push TTSStartedFrames. When true, the base class handles `createAudioContext` and yields `TTSStartedFrame` before each synthesis call, so `runTts` implementations do not need to. | | `stopFrameTimeoutS` | `number \| undefined` | no | Idle time before pushing TTSStoppedFrame when pushStopFrames is true. | | `pushSilenceAfterStop` | `boolean \| undefined` | no | Whether to push silence audio after TTSStoppedFrame. | | `silenceTimeS` | `number \| undefined` | no | Duration of silence to push when pushSilenceAfterStop is true. | | `pauseFrameProcessing` | `boolean \| undefined` | no | Whether to hold incoming frames after a turn until its audio contexts end. Needed by services whose audio doesn't carry its request's context ID. | | `maxConsecutiveZeroAudioContexts` | `number \| undefined` | no | How many consecutive TTS contexts may complete without producing any audio before the service is reported unable to do its job. Catches a provider that accepts requests and stays silent — an unknown voice ID, say — which no error ever surfaces. Every silent context reports an error the service can carry on from, whatever the limit is; on reaching the limit the service reports a permanent error instead, stops being given work, and the pipeline worker applies its `ProcessorUnusablePolicy`. Set to 0 to report silent contexts without ever writing the service off. | | `appendTrailingSpace` | `boolean \| undefined` | no | Whether to append a trailing space to text before sending to TTS. This helps prevent some TTS services from vocalizing trailing punctuation (e.g., "dot"). Only applied in sentence aggregation mode; when streaming tokens, the incoming text's own whitespace is preserved. | | `sampleRate` | `number \| null \| undefined` | no | Output sample rate for generated audio. | | `skipTextTypes` | `string[] \| null \| undefined` | no | List of text types that should not be spoken. | | `textTransforms` | `[string, TextTransform][] \| null \| undefined` | no | A list of TextTransforms to transform text just before sending it to TTS. Each TextTransform takes the aggregated text and its type, and returns the transformed text. To register, provide a list of tuples of `[textType \| "*", transformFunction]`. | | `transportDestination` | `string \| null \| undefined` | no | Destination for generated audio frames. | | `reuseContextIdWithinTurn` | `boolean \| undefined` | no | Whether the service should reuse context IDs within the same turn. | | `audioCache` | `TTSAudioCacheStore \| null \| undefined` | no | Where to keep synthesized audio for reuse (e.g. a shared `TTSAudioCache`). Text whose audio is in the cache is served from it, with no request to the provider. Lines get there through `warmAudioCache()`, and also from live synthesis when `cacheSynthesizedAudio` allows it. Only services that yield their audio from `runTts` (HTTP and streaming-RPC services, see `supportsAudioCache`) use it, and only in sentence aggregation mode. Defaults to null (no cache). pipedog only. | | `cacheSynthesizedAudio` | `boolean \| ((text: string) => boolean) \| undefined` | no | Whether audio synthesized live is stored in `audioCache`, or a function that decides per text. Defaults to false: only warmed lines are stored, since a generated reply can repeat what the user said. pipedog only. | | `audioCacheNamespace` | `string \| undefined` | no | Extra part of the cache key, for constructor options that change the audio but are not in `settings`. pipedog only. | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new XAITTSService.Settings({...})` (`XAIWebsocketTTSSettings`). Any setting you leave out keeps the default below. | Setting | Type | Default | Description | | -------------------------- | ------------------ | ------------- | ------------------------------------------- | | `model` | `string \| null` | `null` | Speech model. `null` leaves xAI's default. | | `voice` | `string` | `"eve"` | Voice. | | `language` | `Language \| null` | `Language.EN` | Language; `null` sends `"auto"`. | | `speed` | `number \| null` | `null` | Speed, 0.7 to 1.5. | | `optimizeStreamingLatency` | `number \| null` | `null` | Latency optimization level: 0, 1 or 2. | | `textNormalization` | `boolean \| null` | `null` | Normalize the text before synthesis. | | `withTimestamps` | `boolean` | `true` | Word timestamps, pushed as `TTSTextFrame`s. | ### Usage [#usage] ```ts title="tts.ts" import { TTSUpdateSettingsFrame } from "@pipedog/core/frames/frames"; import { XAIHttpTTSService, XAITTSService } from "@pipedog/xai/tts"; const tts = new XAITTSService({ apiKey: process.env.XAI_API_KEY ?? "", settings: new XAITTSService.Settings({ voice: "eve" }), }); ``` Change settings mid-call with an `TTSUpdateSettingsFrame`: ```ts // Queue it on the worker, or push it from a processor upstream of the service. const update = new TTSUpdateSettingsFrame({ delta: new XAITTSService.Settings({ speed: 1.1 }), }); ``` ### Event handlers [#event-handlers] | Event | Description | | --------------------- | ------------------------ | | `on_connected` | The WebSocket connected. | | `on_disconnected` | The WebSocket closed. | | `on_connection_error` | The connection failed. | ### Notes [#notes] * `codec` is `"pcm"` by default; `wav`, `mulaw` and `alaw` also work. * The end of a turn sends `text.done`; an interruption sends `text.clear` and keeps the socket open. * Settings are query parameters of the connection, so any settings change reconnects. ## XAIHttpTTSService [#xaihttpttsservice] Posts each sentence to `https://api.x.ai/v1/tts`. ### Configuration [#configuration-1] | Option | Type | Required | Description | |---|---|---|---| | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `stopFrameTimeoutS` | `number \| undefined` | no | Idle time before pushing TTSStoppedFrame when pushStopFrames is true. | | `pushSilenceAfterStop` | `boolean \| undefined` | no | Whether to push silence audio after TTSStoppedFrame. | | `silenceTimeS` | `number \| undefined` | no | Duration of silence to push when pushSilenceAfterStop is true. | | `apiKey` | `string` | yes | xAI API key for authentication. | | `baseUrl` | `string \| undefined` | no | xAI TTS endpoint. Defaults to "https://api.x.ai/v1/tts". | | `encoding` | `string \| null \| undefined` | no | Output encoding format. Defaults to "pcm". | | `fetch` | `{ (input: RequestInfo \| URL, init?: RequestInit): Promise; (input: string \| URL \| Request, init?: RequestInit): Promise; } \| null \| undefined` | no | `fetch` used for HTTP requests. Defaults to the global `fetch`. DIVERGENCE: upstream takes an optional shared `aiohttp_session`. | | `settings` | `XAITTSSettings \| null \| undefined` | no | Runtime-updatable settings. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `textAggregationMode` | `TextAggregationMode \| null \| undefined` | no | How to aggregate incoming text before synthesis. `TextAggregationMode.SENTENCE` (default) buffers until sentence boundaries, `TextAggregationMode.TOKEN` streams tokens directly for lower latency. | | `pushTextFrames` | `boolean \| undefined` | no | Whether to push TextFrames and LLMFullResponseEndFrames. | | `pushStopFrames` | `boolean \| undefined` | no | Whether to automatically push TTSStoppedFrames. | | `pushStartFrame` | `boolean \| undefined` | no | Whether to automatically create audio contexts and push TTSStartedFrames. When true, the base class handles `createAudioContext` and yields `TTSStartedFrame` before each synthesis call, so `runTts` implementations do not need to. | | `pauseFrameProcessing` | `boolean \| undefined` | no | Whether to hold incoming frames after a turn until its audio contexts end. Needed by services whose audio doesn't carry its request's context ID. | | `maxConsecutiveZeroAudioContexts` | `number \| undefined` | no | How many consecutive TTS contexts may complete without producing any audio before the service is reported unable to do its job. Catches a provider that accepts requests and stays silent — an unknown voice ID, say — which no error ever surfaces. Every silent context reports an error the service can carry on from, whatever the limit is; on reaching the limit the service reports a permanent error instead, stops being given work, and the pipeline worker applies its `ProcessorUnusablePolicy`. Set to 0 to report silent contexts without ever writing the service off. | | `appendTrailingSpace` | `boolean \| undefined` | no | Whether to append a trailing space to text before sending to TTS. This helps prevent some TTS services from vocalizing trailing punctuation (e.g., "dot"). Only applied in sentence aggregation mode; when streaming tokens, the incoming text's own whitespace is preserved. | | `sampleRate` | `number \| null \| undefined` | no | Output sample rate for generated audio. | | `skipTextTypes` | `string[] \| null \| undefined` | no | List of text types that should not be spoken. | | `textTransforms` | `[string, TextTransform][] \| null \| undefined` | no | A list of TextTransforms to transform text just before sending it to TTS. Each TextTransform takes the aggregated text and its type, and returns the transformed text. To register, provide a list of tuples of `[textType \| "*", transformFunction]`. | | `transportDestination` | `string \| null \| undefined` | no | Destination for generated audio frames. | | `reuseContextIdWithinTurn` | `boolean \| undefined` | no | Whether the service should reuse context IDs within the same turn. | | `audioCache` | `TTSAudioCacheStore \| null \| undefined` | no | Where to keep synthesized audio for reuse (e.g. a shared `TTSAudioCache`). Text whose audio is in the cache is served from it, with no request to the provider. Lines get there through `warmAudioCache()`, and also from live synthesis when `cacheSynthesizedAudio` allows it. Only services that yield their audio from `runTts` (HTTP and streaming-RPC services, see `supportsAudioCache`) use it, and only in sentence aggregation mode. Defaults to null (no cache). pipedog only. | | `cacheSynthesizedAudio` | `boolean \| ((text: string) => boolean) \| undefined` | no | Whether audio synthesized live is stored in `audioCache`, or a function that decides per text. Defaults to false: only warmed lines are stored, since a generated reply can repeat what the user said. pipedog only. | | `audioCacheNamespace` | `string \| undefined` | no | Extra part of the cache key, for constructor options that change the audio but are not in `settings`. pipedog only. | ### Settings [#settings-1] Runtime-updatable settings, passed as `settings: new XAIHttpTTSService.Settings({...})` (`XAITTSSettings`). Any setting you leave out keeps the default below. | Setting | Type | Default | Description | | -------------------------- | ------------------ | ------------- | ------------------------------------------ | | `model` | `string \| null` | `null` | Speech model. `null` leaves xAI's default. | | `voice` | `string` | `"eve"` | Voice. | | `language` | `Language \| null` | `Language.EN` | Language; `null` sends `"auto"`. | | `speed` | `number \| null` | `null` | Speed, 0.7 to 1.5. | | `optimizeStreamingLatency` | `number \| null` | `null` | Latency optimization level: 0, 1 or 2. | | `textNormalization` | `boolean \| null` | `null` | Normalize the text before synthesis. | ### Usage [#usage-1] ```ts const xAIHttpTTSService = new XAIHttpTTSService({ apiKey: process.env.XAI_API_KEY ?? "", settings: new XAIHttpTTSService.Settings({ voice: "eve" }), }); ``` ### Notes [#notes-1] * `fetch` replaces the global `fetch` (upstream takes an aiohttp session). * `encoding` is `"pcm"` by default. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/xai/package.json`: ✅ means the compat smoke test passes on that runtime. * HTTP goes through `fetch` and the WebSocket through the core `websocketConnect`. Both work on Bun, Node, Deno and Cloudflare Workers. * Tested against local servers, as upstream's tests do, not against xAI itself. # HeyGen > HeyGen video avatars (LiveAvatar and Interactive Avatar) with HeyGenVideoService and HeyGenTransport. Source: https://polvallverdu.github.io/pipedog/docs/services/video/heygen/ ## Overview [#overview] HeyGen renders a talking avatar from your bot's speech. The session is controlled over HTTP and a WebSocket, and the avatar's video and audio travel through a LiveKit room. There are two ways to use it: * **`HeyGenVideoService`** (`@pipedog/heygen`): a pipeline service placed after your TTS. It sends the TTS audio to the avatar and pushes the avatar's video (`OutputImageRawFrame`) and audio (`SpeechOutputAudioRawFrame`) downstream, so any output transport can deliver them to the user. * **`HeyGenTransport`** (`@pipedog/transport-heygen`): the bot joins the avatar's LiveKit room as a participant. The user joins the same room and sees the avatar directly; the bot hears the user and sends its speech to the avatar. Both talk to **LiveAvatar** (`ServiceType.LIVE_AVATAR`, `api.liveavatar.com`) or to HeyGen's older **Interactive Avatar** streaming API (`ServiceType.INTERACTIVE_AVATAR`, `api.heygen.com`, the default for backwards compatibility). Pipecat's documentation for the same service. `video.ts`, `client.ts`, `api_liveavatar.ts` and `api_interactive_avatar.ts`, plus `packages/transport-heygen/src/transport.ts`. Sessions, avatars and API keys. ## Runtimes [#runtimes] | | Bun | Node ≥ 22 | Deno | Cloudflare Workers | | ------------------------------------------------------ | --- | --------- | ----------------- | ------------------ | | `LiveAvatarApi`, `HeyGenApi` (session HTTP APIs) | ✅ | ✅ | ✅ | not verified | | `HeyGenClient` control WebSocket | ✅ | ✅ | ✅ | not verified | | `HeyGenVideoService`, `HeyGenTransport` (avatar media) | ✅ | ✅ | ✅ (`--allow-all`) | ❌ | The session APIs use only `fetch` and the control channel only `WebSocket`, so they have no runtime requirements of their own, but the packages aren't tested on Workers. The media goes through LiveKit's native Node SDK, `@livekit/rtc-node`, which loads a native N-API addon. Workers can't load native code, so the service and the transport don't run there. Importing the packages works everywhere: the addon is only loaded when the client sets up. ## Installation [#installation] ```bash bun add @pipedog/heygen @livekit/rtc-node # and, for the transport: bun add @pipedog/transport-heygen ``` ```ts import { HeyGenVideoService } from "@pipedog/heygen/video"; import { HeyGenTransport, HeyGenParams } from "@pipedog/transport-heygen/transport"; import { LiveAvatarNewSessionRequest } from "@pipedog/heygen/api_liveavatar"; import { ServiceType } from "@pipedog/heygen/client"; ``` `@livekit/rtc-node` is an optional peer dependency: install it yourself (version 1.1.0 or newer in the 1.x line). It ships prebuilt binaries for these platforms only: | OS | Architectures | | ---------------- | ------------------------------------------------ | | macOS | arm64 (`darwin-arm64`), x64 (`darwin-x64`) | | Linux with glibc | x64 (`linux-x64-gnu`), arm64 (`linux-arm64-gnu`) | | Windows | x64 (`win32-x64-msvc`) | There is no build for musl (Alpine Linux): use a Debian- or Ubuntu-based image. Without the package, or on another platform, joining the room fails with an error that says how to install it. ## Prerequisites [#prerequisites] * A **LiveAvatar account** and its API key (for `ServiceType.LIVE_AVATAR`; see [LiveAvatar's docs](https://docs.liveavatar.com)), or a **HeyGen account** and API key for the Interactive Avatar API. Pass it as `apiKey`; the samples read `HEYGEN_LIVE_AVATAR_API_KEY`. * An avatar ID (`avatarId`). Without a session request, the client uses upstream's default public avatar for the chosen API. * No LiveKit server of your own: HeyGen hosts the room and returns its URL and tokens. To use your own LiveKit server or LiveKit Cloud instead, pass a `CustomSDKLiveKitConfig` as `livekitConfig`. * A long-running Bun, Node or Deno process on one of the platforms above. ## HeyGenVideoService [#heygenvideoservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string` | yes | HeyGen API key for authentication. | | `fetch` | `{ (input: RequestInfo \| URL, init?: RequestInit): Promise; (input: string \| URL \| Request, init?: RequestInit): Promise; } \| null \| undefined` | no | `fetch` used for API requests. Defaults to the global `fetch`. DIVERGENCE: replaces upstream's required aiohttp `session`. | | `sessionRequest` | `LiveAvatarNewSessionRequest \| NewSessionRequest \| null \| undefined` | no | Configuration for the HeyGen session. | | `serviceType` | `ServiceType \| null \| undefined` | no | Service type for the avatar session. | | `settings` | `HeyGenVideoSettings \| null \| undefined` | no | Runtime-updatable settings. HeyGen has no model concept, so this is primarily used for the `extra` dict. | | `clientOptions` | `Partial \| null \| undefined` | no | Extra `HeyGenClient` options (e.g. `websocketOptions`, `api`). pipedog addition. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | The LiveAvatar session request: | Option | Type | Required | Description | |---|---|---|---| | `mode` | `string \| undefined` | no | Session mode (default: "LITE"). | | `avatarId` | `string` | yes | Unique identifier for the avatar. | | `videoSettings` | `VideoSettings \| null \| undefined` | no | Video encoding settings (default: H264, high quality). | | `isSandbox` | `boolean \| null \| undefined` | no | Enable sandbox mode (default: false). | | `avatarPersona` | `AvatarPersona \| null \| undefined` | no | Avatar persona configuration. | | `livekitConfig` | `CustomSDKLiveKitConfig \| null \| undefined` | no | Custom LiveKit configuration. | | `maxSessionDuration` | `number \| null \| undefined` | no | Maximum session duration in seconds. The ceiling is set by the LiveAvatar subscription tier, which also supplies the default when this is unset. | ### Usage [#usage] ```ts title="bot.ts" import { Pipeline } from "@pipedog/core/pipeline/pipeline"; import { PipelineParams, PipelineWorker } from "@pipedog/core/pipeline/worker"; import { LiveAvatarNewSessionRequest, VideoEncoding, VideoSettings, } from "@pipedog/heygen/api_liveavatar"; import { ServiceType } from "@pipedog/heygen/client"; import { HeyGenVideoService } from "@pipedog/heygen/video"; const heygen = new HeyGenVideoService({ apiKey: process.env.HEYGEN_LIVE_AVATAR_API_KEY ?? "", serviceType: ServiceType.LIVE_AVATAR, sessionRequest: new LiveAvatarNewSessionRequest({ avatarId: "", videoSettings: new VideoSettings({ encoding: VideoEncoding.H264 }), }), }); // The avatar speaks the TTS audio; its video and audio come back as frames that // the output transport sends to the user. const worker = new PipelineWorker( new Pipeline([ transport.input(), stt, userAggregator, llm, tts, heygen, transport.output(), assistantAggregator, ]), { params: new PipelineParams({ audioInSampleRate: 16000, audioOutSampleRate: 24000 }) }, ); ``` ### Notes [#notes] * TTS audio is resampled to 24 kHz and sent as `agent.speak` messages: a 400 ms first chunk, then 1 s chunks. When no audio arrives for 0.35 s, the rest is flushed and `agent.speak_end` sent. * A `UserStartedSpeakingFrame` interrupts the avatar (`agent.interrupt`) and starts its listening animation; `UserStoppedSpeakingFrame` stops it. * The service joins the room with the user's token (`connectAsUser`), so HeyGen doesn't end the session after five minutes without a user. ## HeyGenTransport [#heygentransport] ### Configuration [#configuration-1] | Option | Type | Required | Description | |---|---|---|---| | `fetch` | `{ (input: RequestInfo \| URL, init?: RequestInit): Promise; (input: string \| URL \| Request, init?: RequestInit): Promise; } \| null \| undefined` | no | `fetch` used for API requests. Defaults to the global `fetch`. DIVERGENCE: replaces upstream's required aiohttp `session`. | | `apiKey` | `string` | yes | HeyGen API key for authentication. | | `params` | `HeyGenParams \| null \| undefined` | no | HeyGen-specific configuration parameters (default: new HeyGenParams()). | | `inputName` | `string \| null \| undefined` | no | Optional custom name for the input transport. | | `outputName` | `string \| null \| undefined` | no | Optional custom name for the output transport. | | `sessionRequest` | `LiveAvatarNewSessionRequest \| NewSessionRequest \| null \| undefined` | no | Configuration for the HeyGen session. | | `serviceType` | `ServiceType \| null \| undefined` | no | Service type for the avatar session. | | `clientOptions` | `Partial \| null \| undefined` | no | Extra `HeyGenClient` options (e.g. `websocketOptions`, `api`). pipedog addition. | `HeyGenParams` is a `TransportParams` with `audioInEnabled` and `audioOutEnabled` on by default. ### Events [#events] | Event | Arguments | When | | ------------------------ | -------------------------- | -------------------------------------------------------------------- | | `on_connected` | `(transport)` | The bot joined the avatar's room. | | `on_client_connected` | `(transport, participant)` | A participant other than the avatar joined. Their audio is captured. | | `on_client_disconnected` | `(transport, participant)` | That participant left. | ### Usage [#usage-1] ```ts title="bot.ts" import { HeyGenParams, HeyGenTransport } from "@pipedog/transport-heygen/transport"; const heygenTransport = new HeyGenTransport({ apiKey: process.env.HEYGEN_LIVE_AVATAR_API_KEY ?? "", serviceType: ServiceType.LIVE_AVATAR, sessionRequest: new LiveAvatarNewSessionRequest({ avatarId: "" }), params: new HeyGenParams({ audioInEnabled: true, audioOutEnabled: true }), }); heygenTransport.eventHandler("on_client_connected", async (_transport, participant) => { console.log(`user joined the avatar's room: ${participant}`); }); const avatarWorker = new PipelineWorker( new Pipeline([ heygenTransport.input(), stt, userAggregator, llm, tts, heygenTransport.output(), assistantAggregator, ]), ); ``` The session's room link (`https://meet.livekit.io/custom?...`) is logged at `info` level when the session starts; open it to join as the user. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ❌ no | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/heygen/package.json`: ✅ means the compat smoke test passes on that runtime. | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ❌ no | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/transport-heygen/package.json`: ✅ means the compat smoke test passes on that runtime. * The session APIs use `fetch`; pass `fetch` to replace the global one (upstream takes an aiohttp session). * The control WebSocket uses the global `WebSocket`. `clientOptions.websocketOptions` takes a `webSocketFactory` for runtimes that open sockets differently. * LiveAvatar's API documents `livekit_agent_token` and `ws_url` as optional. Upstream requires them; here a session without a WebSocket URL runs without avatar control (logged as a warning), and one without an agent token joins the room with the client token. * Callbacks given to `HeyGenClient.captureParticipantAudio` and `captureParticipantVideo` receive the frame fields (`audio`, `sampleRate`, `numChannels` / `image`, `size`, `format`) rather than an `AudioRawFrame` or `ImageRawFrame` instance. # Video avatars > Animated avatars that speak the bot's audio. Source: https://polvallverdu.github.io/pipedog/docs/services/video/ {/* Generated by scripts/generate-providers.ts from data/providers.ts. */} A video service animates an avatar that speaks the bot's audio, and sends its video to the user. | Provider | Classes | Package | Status | | ------------------------------------- | --------------------------------------- | ----------------- | ----------- | | [HeyGen](https://polvallverdu.github.io/pipedog/docs/services/video/heygen.md) | `HeyGenVideoService`, `HeyGenTransport` | `@pipedog/heygen` | Ported | | [Simli](https://polvallverdu.github.io/pipedog/docs/services/video/simli.md) | `SimliVideoService` | | Coming soon | | [Tavus](https://polvallverdu.github.io/pipedog/docs/services/video/tavus.md) | `TavusVideoService`, `TavusTransport` | | Blocked | Pipecat's [video avatars reference](https://docs.pipecat.ai/api-reference/server/services/video) lists every upstream provider. Those not in this table are on [Other upstream providers](https://polvallverdu.github.io/pipedog/docs/services/other-upstream.md). # Simli > Simli's realtime lip-synced avatars. Not ported to pipedog yet. Source: https://polvallverdu.github.io/pipedog/docs/services/video/simli/ {/* Generated by scripts/generate-providers.ts from data/providers.ts. */} Simli isn't ported to pipedog yet. It is available in Pipecat (Python), and its pipedog port will keep the same class names and options. [Ask for it, or upvote the request](https://github.com/polvallverdu/pipedog/issues?q=is%3Aissue+Simli). ## What it is [#what-it-is] Simli's realtime lip-synced avatars. Upstream, it's this class: `SimliVideoService`, in [`services/simli/video.py`](https://github.com/pipecat-ai/pipecat/blob/main/src/pipecat/services/simli/video.py). Its pipedog package will be `@pipedog/simli`. * [Pipecat's Simli page](https://docs.pipecat.ai/api-reference/server/services/video/simli) ## Get it ported [#get-it-ported] * [Search the issues](https://github.com/polvallverdu/pipedog/issues?q=is%3Aissue+Simli) for a request, and upvote it with a reaction. * If there's none, [open a port request](https://github.com/polvallverdu/pipedog/issues/new?template=feature_request.yml\&title=Port+Simli+%28SimliVideoService%29\&kind=Port+an+upstream+module\&upstream=src%2Fpipecat%2Fservices%2Fsimli%2Fvideo.py). * Or port it yourself: the [contributing guide](https://polvallverdu.github.io/pipedog/docs/contributing.md) explains how modules are ported, test by test. The [porting status](https://polvallverdu.github.io/pipedog/docs/contributing/status.md) lists what's done. Until then, see what is ported in [Video avatars](https://polvallverdu.github.io/pipedog/docs/services/video.md). # Tavus > Tavus' realtime video avatars. Blocked in pipedog: not ported yet. Source: https://polvallverdu.github.io/pipedog/docs/services/video/tavus/ {/* Generated by scripts/generate-providers.ts from data/providers.ts. */} Tavus is blocked: there is no viable JS path for it today. **Why.** Tavus avatars join a Daily room, so they need the Daily transport, which has no server-side JS path (see Daily). **What would unblock it.** Unblocked when Daily gets a server-side JS transport. The REST client for Tavus is portable on its own. Sources: [Pipecat's Tavus transport](https://docs.pipecat.ai/api-reference/server/services/transport/tavus). ## What it is [#what-it-is] Tavus' realtime video avatars. Upstream, it's these classes: `TavusVideoService`, `TavusTransport`, in [`services/tavus/video.py`](https://github.com/pipecat-ai/pipecat/blob/main/src/pipecat/services/tavus/video.py) and [`transports/tavus/transport.py`](https://github.com/pipecat-ai/pipecat/blob/main/src/pipecat/transports/tavus/transport.py). Its pipedog package will be `@pipedog/tavus`. * [Pipecat's Tavus page](https://docs.pipecat.ai/api-reference/server/services/video/tavus) ## Get it ported [#get-it-ported] * [Search the issues](https://github.com/polvallverdu/pipedog/issues?q=is%3Aissue+Tavus) for a request, and upvote it with a reaction. * If there's none, [open a port request](https://github.com/polvallverdu/pipedog/issues/new?template=feature_request.yml\&title=Port+Tavus+%28TavusVideoService%2C+TavusTransport%29\&kind=Port+an+upstream+module\&upstream=src%2Fpipecat%2Fservices%2Ftavus%2Fvideo.py%2C+src%2Fpipecat%2Ftransports%2Ftavus%2Ftransport.py). * Or port it yourself: the [contributing guide](https://polvallverdu.github.io/pipedog/docs/contributing.md) explains how modules are ported, test by test. The [porting status](https://polvallverdu.github.io/pipedog/docs/contributing/status.md) lists what's done. Until then, see what is ported in [Video avatars](https://polvallverdu.github.io/pipedog/docs/services/video.md). # Telephony extensions > IVRNavigator walks a phone menu with DTMF, and VoicemailDetector tells a person from a voicemail on outbound calls. Port of pipecat.extensions. Source: https://polvallverdu.github.io/pipedog/docs/telephony/extensions/ ## Overview [#overview] Two pieces of call logic that sit on top of any [phone serializer](https://polvallverdu.github.io/pipedog/docs/telephony.md), ported from Pipecat's `pipecat.extensions`: * **`IVRNavigator`** is a pipeline segment around your LLM. It classifies the other side as a person or an automated menu, and in a menu it has the LLM press keys (`1`) or speak until it reaches a goal you set. * **`VoicemailDetector`** tells a person from a voicemail greeting on an **outbound** call. Until it has decided, `TTSGate` holds the bot's speech, so a greeting is never talked over. Both are plain frame processors with no I/O of their own, so they run on Bun, Node, Deno and Cloudflare Workers. Pipecat's `pipecat.extensions`. `ivr_navigator.ts` and `voicemail_detector.ts`. ## Installation [#installation] ```bash bun add @pipedog/extensions ``` ```ts import { IVRNavigator, VoicemailDetector } from "@pipedog/extensions"; ``` ## Runtimes [#runtimes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/extensions/package.json`: ✅ means the compat smoke test passes on that runtime. ## IVR navigation [#ivr-navigation] ```ts const ivrNavigator = new IVRNavigator({ llm, // The goal. It is placed into the navigation instructions, so describe the // destination and anything the bot may say or enter along the way. ivrPrompt: "Reach the billing department and ask for a human agent.", // The IVR's prompts are read out slowly; wait longer before answering. ivrVadParams: new VADParams({ stopSecs: 2.0 }), }); const ivrContext = new LLMContext(); const ivrAggregators = new LLMContextAggregatorPair(ivrContext); const ivrPipeline = new Pipeline([ transport.input(), stt, new DTMFAggregator(), // the IVR's own keypad tones reach the LLM as text ivrAggregators.user(), ivrNavigator, // wraps the LLM and turns , and tags into frames tts, transport.output(), ivrAggregators.assistant(), ]); const ivrWorker = new PipelineWorker(ivrPipeline); // A person picked up: continue as a normal conversation. ivrNavigator.addEventHandler("on_conversation_detected", async (_processor, history) => { await ivrWorker.queueFrame( new LLMMessagesUpdateFrame({ messages: [{ role: "developer", content: "You are a helpful agent." }, ...history], runLlm: true, }), ); }); ivrNavigator.addEventHandler("on_ivr_status_changed", async (_processor, status) => { if (status === IVRStatus.COMPLETED) { // Transferred to a person: hand the call over to the conversation bot. } else if (status === IVRStatus.STUCK) { await ivrWorker.queueFrame(new EndWorkerFrame({ reason: "IVR navigation is stuck." })); } }); ``` How it works: 1. At `StartFrame` the `IVRProcessor` inside the navigator sets the **classifier prompt** on the LLM. The LLM answers `ivr` or `conversation` for what it hears. 2. `conversation` fires `on_conversation_detected` with the conversation history saved so far (the context without its first, system message). 3. `ivr` switches the context to the **navigation prompt** (your `ivrPrompt` inside the built-in instructions), with the history after it, runs the LLM, and switches the VAD to `ivrVadParams` (default `stopSecs: 2.0`) so the bot waits out the menu's pauses. 4. In navigation mode the LLM answers with tags that the processor strips from the text: | Tag | Effect | | ---------------------- | --------------------------------------------------------------------------------------------------------------------------- | | `1` | Pushes an `OutputDTMFUrgentFrame` for that key (`0`-`9`, `*`, `#`). Other values are logged and ignored. One tag per digit. | | `detected` | Back to navigation mode. Fires `on_ivr_status_changed`. | | `completed` | The goal is reached, usually a transfer. Fires `on_ivr_status_changed`. | | `stuck` | No way forward. Fires `on_ivr_status_changed` so you can end or retry. | | `wait` | The menu is incomplete; wait for more transcription. | Everything else is pushed on as `AggregatedTextFrame`s, which the TTS service speaks. The tags themselves are added to the context as `TextFrame`s with `skipTts`, so the LLM sees what it did. `on_ivr_status_changed` receives an `IVRStatus` (`"detected"`, `"completed"`, `"stuck"`, `"wait"`). `addEventHandler` on the navigator registers the two events on its `IVRProcessor`. ### With the phone serializers [#with-the-phone-serializers] * **Sending keys.** `OutputDTMFUrgentFrame` reaches the output transport, which plays the key as a tone in the audio stream when the transport has no native DTMF (the WebSocket transport has none). The Twilio, Telnyx, Plivo, Exotel, Vonage and Genesys serializers then send that audio like any other, so the menu hears the keys. This is checked end to end for Twilio in the package's tests. * **Hearing keys.** If the menu or a person presses keys, the serializers produce `InputDTMFFrame`s. Put a `DTMFAggregator` (`@pipedog/core/processors/aggregators/dtmf_aggregator`) between the STT service and the user aggregator, as above, and sequences such as `12#` arrive in the LLM context as `DTMF: 12#`. ### Options [#options] | Option | Type | Required | Description | |---|---|---|---| | `llm` | `LLMService>>` | yes | LLM service for text generation and decision making. | | `ivrPrompt` | `string` | yes | Navigation goal prompt integrated with IVR navigation instructions. | | `ivrVadParams` | `VADParams \| null \| undefined` | no | VAD parameters for IVR navigation mode. Defaults to `new VADParams({ stopSecs: 2.0 })`. | ## Voicemail detection [#voicemail-detection] ```ts const voicemail = new VoicemailDetector({ classifier: new JevClassifier({ apiKey: process.env.TYPESAFE_API_KEY ?? null }), decisionTimeout: 1.0, // seconds of caller silence before the verdict acts voicemailResponseDelay: 2.0, // seconds of silence after a voicemail verdict }); const context = new LLMContext(); const aggregators = new LLMContextAggregatorPair(context); const pipeline = new Pipeline([ transport.input(), stt, voicemail.detector(), // between the STT service and the user aggregator aggregators.user(), llm, tts, voicemail.gate(), // right after the TTS service transport.output(), aggregators.assistant(), ]); const worker = new PipelineWorker(pipeline); voicemail.addEventHandler("on_conversation_detected", async () => { console.log("A person answered."); }); voicemail.addEventHandler("on_voicemail_detected", async (processor) => { await processor.pushFrame( new TTSSpeakFrame({ text: "Hello, this is Alex calling about your appointment. Please call me back.", }), ); // The message is left, so the call is over. await processor.pushFrame(new EndWorkerFrame()); }); ``` The detector sits after the STT service. It passes every frame through, collects what the other side says, and after each transcription asks its **classifier** in the background whether the call reached a person or a voicemail. Once the caller has been quiet for `decisionTimeout`, the latest answer decides: * **Conversation.** The gate releases the speech it held, in order, and `on_conversation_detected` fires. * **Voicemail.** The held speech is dropped, the pipeline is interrupted, and nothing but lifecycle frames (system frames, `EndFrame`, `StopFrame`, worker frames) passes the detector from then on. `on_voicemail_detected` fires once the greeting has been quiet for `voicemailResponseDelay` seconds, and only once. Speak the message from the handler by pushing a `TTSSpeakFrame` through the processor it receives. A classifier that fails or does not answer in time counts as no answer. With no answer at all after the silence, the detector assumes a conversation. Any `BaseClassifier` works. `JevClassifier` is what upstream's example uses; an `LLMClassifier` needs a service that implements `runInference()`, so a realtime (speech-to-speech) LLM cannot back it. The classifier's `on_metrics` becomes a `MetricsFrame` in the pipeline. ### Options [#options-1] | Option | Type | Required | Description | |---|---|---|---| | `classifier` | `BaseClassifier` | yes | What decides between a person and a voicemail. It is asked a `choice` question with the transcript so far. | | `voicemailResponseDelay` | `number \| undefined` | no | Seconds of silence after a voicemail verdict before `on_voicemail_detected` fires, so the message is left after the greeting ends and the recording starts. Defaults to 2.0. | | `decisionTimeout` | `number \| undefined` | no | Seconds of silence after the caller stops speaking before the latest answer decides. A greeting resumes after its pauses while a person stays quiet, so a verdict on a fragment such as "hi, this is Sam" is not acted on until the caller has really stopped. Defaults to 1.0. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | Upstream's deprecated `llm` and `customSystemPrompt` parameters are not ported; pass a `classifier`. ## Testing without a phone [#testing-without-a-phone] Both classes are covered by ports of upstream's tests (`test_ivr_navigation.py`, `test_voicemail_detector.py`), which use `runTest` and a scripted classifier. The same approach works for your handlers: drive the processor with `LLMTextFrame`s or `TranscriptionFrame`s and read the frames and events back. Nothing here has been run against a live IVR or a real carrier's voicemail, so the prompts and classifier thresholds are upstream's, unverified by us. # How phone calls connect > How a phone call reaches a pipedog bot over a WebSocket media stream, and which serializer speaks each provider's protocol. Source: https://polvallverdu.github.io/pipedog/docs/telephony/ ## The path of a call [#the-path-of-a-call] A phone bot is a pipeline whose transport is a WebSocket that a phone provider opens to your server. The provider handles the phone network; your server handles the conversation. ```text caller ─► phone network ─► provider (Twilio, Telnyx, Plivo, Exotel, ...) │ 1. webhook: "a call came in" ──► your server answers with │ "stream it to wss://…/ws" │ 2. WebSocket to wss://…/ws ◄─► transport + serializer ▼ + your pipeline ``` 1. **The call arrives.** The provider calls your webhook (Twilio and Telnyx with TwiML/TeXML, Plivo with XML, Exotel and Genesys through a flow you configure in their console). The answer tells the provider to open a bidirectional media stream to your WebSocket URL, optionally with parameters such as the caller's number. 2. **The stream opens.** The provider connects to your WebSocket and sends a handshake: a `connected` message, then a `start` message with the **stream ID** and **call ID**. 3. **The conversation.** Audio flows both ways as JSON messages (base64 μ-law at 8 kHz for most providers) or binary frames (Vonage, Genesys). The provider's **serializer** turns them into `InputAudioRawFrame`s at your pipeline's rate, and your bot's audio back into the provider's format. Touch-tone keys become `InputDTMFFrame`s; an interruption tells the provider to drop buffered audio. 4. **The end.** When the caller hangs up, the provider closes the socket and the transport fires `on_client_disconnected`. When your bot ends the call (an `EndFrame`), serializers that support it hang up through the provider's REST API, so the caller isn't left on a silent line. All of this runs over a plain WebSocket, so the same bot runs on Bun, Node, Deno and Cloudflare Workers. Accept the socket with `websocketFetchHandler` from `@pipedog/transport-websocket`: Deno and Workers upgrade from fetch on their own, Bun upgrades with the server it passes to `fetch`, and `serveFetch()` serves the handler on Bun, Node and Deno. See [WebSocket](https://polvallverdu.github.io/pipedog/docs/transports/websocket.md). ## Serializers [#serializers] | Provider | Serializer | Audio | Hangs up on `EndFrame` | | ------------------------------------ | ---------------------------- | -------------------------- | ---------------------- | | [Twilio](https://polvallverdu.github.io/pipedog/docs/serializers/twilio.md) | `TwilioFrameSerializer` | μ-law 8 kHz, JSON | Yes (REST) | | [Telnyx](https://polvallverdu.github.io/pipedog/docs/serializers/telnyx.md) | `TelnyxFrameSerializer` | μ-law or A-law 8 kHz, JSON | Yes (REST) | | [Plivo](https://polvallverdu.github.io/pipedog/docs/serializers/plivo.md) | `PlivoFrameSerializer` | μ-law 8 kHz, JSON | Yes (REST) | | [Exotel](https://polvallverdu.github.io/pipedog/docs/serializers/exotel.md) | `ExotelFrameSerializer` | PCM 8 kHz, JSON | No | | [Vonage](https://polvallverdu.github.io/pipedog/docs/serializers/vonage.md) | `VonageFrameSerializer` | PCM 16 kHz, binary | No | | [Genesys](https://polvallverdu.github.io/pipedog/docs/serializers/genesys.md) | `GenesysAudioHookSerializer` | μ-law 8 kHz, binary | Sends `disconnect` | Set `addWavHeader: false` on `FastAPIWebsocketParams` for all of them. ## Reading the handshake [#reading-the-handshake] `@pipedog/runner` reads the handshake for you. `parseTelephonyWebsocket(websocket)` is upstream's `parse_telephony_websocket`: it reads the first two messages, detects Twilio, Telnyx, Plivo or Exotel, and returns `[transportType, callData]`. `callData.callId` is the provider's call identifier (Twilio's `callSid`, Telnyx's `call_control_id`, Plivo's `callId`, Exotel's `call_sid`), and `callData.fromNumber` and `callData.toNumber` are filled where the provider sends them (on Twilio, from the `from_number` and `to_number` stream parameters). `telephonyHandshake(websocket)` returns the same parse as a typed outcome, so a caller who hangs up while the call is still connecting is an ordinary result rather than an exception: ```ts title="telephony.ts" import type { FrameSerializer } from "@pipedog/core/serializers/base_serializer"; import type { CallData, TelnyxCallData } from "@pipedog/runner/types"; import { telephonyHandshake } from "@pipedog/runner/utils"; import { ExotelFrameSerializer } from "@pipedog/serializer-exotel/exotel"; import { PlivoFrameSerializer } from "@pipedog/serializer-plivo/plivo"; import { TelnyxFrameSerializer } from "@pipedog/serializer-telnyx/telnyx"; import { TwilioFrameSerializer } from "@pipedog/serializer-twilio/twilio"; import { websocketFetchHandler } from "@pipedog/transport-websocket/fetch"; const env = process.env; /** One serializer per provider, from the parsed handshake. */ function serializerFor(transportType: string, callData: CallData): FrameSerializer { const streamId = callData.streamId ?? ""; switch (transportType) { case "twilio": return new TwilioFrameSerializer({ streamSid: streamId, callSid: callData.callId, accountSid: env.TWILIO_ACCOUNT_SID, authToken: env.TWILIO_AUTH_TOKEN, }); case "telnyx": return new TelnyxFrameSerializer({ streamId, callControlId: callData.callId, outboundEncoding: (callData as TelnyxCallData).outboundEncoding ?? "PCMU", inboundEncoding: "PCMU", apiKey: env.TELNYX_API_KEY, }); case "plivo": return new PlivoFrameSerializer({ streamId, callId: callData.callId, authId: env.PLIVO_AUTH_ID, authToken: env.PLIVO_AUTH_TOKEN, }); default: return new ExotelFrameSerializer({ streamSid: streamId, callSid: callData.callId }); } } const handler = websocketFetchHandler(async (websocket) => { const handshake = await telephonyHandshake(websocket); if (handshake.status === "closed") return; // the caller hung up while connecting if (handshake.status === "unrecognized") throw new Error("not a phone provider"); const { transportType, callData } = handshake; console.log(`${transportType} call ${callData.callId} from ${callData.fromNumber}`); const serializer = serializerFor(transportType, callData); // new FastAPIWebsocketTransport(websocket, new FastAPIWebsocketParams({ serializer, ... })) }); ``` ```bash bun add @pipedog/runner ``` The same package is the [development runner](https://polvallverdu.github.io/pipedog/docs/runner.md): `main()` serves the webhook for Twilio (TwiML), Telnyx (TeXML) and Plivo (XML) and the `/ws` media stream for those and Exotel, and `createTransport()` builds the right serializer from the handshake, so a phone bot is one `bot(runnerArgs)` function. ## Next [#next] * [Twilio quickstart](https://polvallverdu.github.io/pipedog/docs/telephony/twilio-quickstart.md): a working phone bot from `examples/twilio-phone-bot`. * Testing without a phone: the example's tests drive the bot with a scripted fake Twilio that speaks the Media Streams protocol, against mock OpenAI APIs. # Twilio quickstart > Run a phone bot on Twilio Media Streams from examples/twilio-phone-bot, and test it without a phone. Source: https://polvallverdu.github.io/pipedog/docs/telephony/twilio-quickstart/ This guide runs [`examples/twilio-phone-bot`](https://github.com/polvallverdu/pipedog/tree/main/examples/twilio-phone-bot): call a Twilio number and talk to an assistant built on OpenAI transcription, chat and speech. Press `#` to end the call; the bot finishes its sentence and hangs up. ## What you need [#what-you-need] * [Bun](https://bun.sh) to install the repository and run the server (or [Deno](https://deno.com), with `bun run start:deno`). * An OpenAI API key. * A Twilio account and a phone number with voice. * A public HTTPS URL for your machine, e.g. from `ngrok http 8765`. ## Run it [#run-it] ```bash git clone https://github.com/polvallverdu/pipedog.git cd pipedog bun install cd examples/twilio-phone-bot OPENAI_API_KEY=sk-... \ TWILIO_ACCOUNT_SID=AC... TWILIO_AUTH_TOKEN=... \ PUBLIC_URL=https://your-tunnel.ngrok.app \ bun run start ``` Then, in the Twilio console, open your phone number and set **A call comes in** to **Webhook**, `https://your-tunnel.ngrok.app/twiml`, **HTTP POST**. Call the number. ## How it works [#how-it-works] The app is one fetch handler with two routes (`src/app.ts`): * **`POST /twiml`** is Twilio's voice webhook. It checks the `X-Twilio-Signature` with your auth token and answers with TwiML that connects the call to the media stream, passing the caller's and the dialed numbers as stream parameters: ```xml ``` * **`GET /ws`** is the Media Streams WebSocket, upgraded by `websocketFetchHandler`. Each connection is one call (`src/bot.ts`): 1. `telephonyHandshake` reads Twilio's `connected` and `start` messages: the stream SID, the call SID and the numbers. A caller who hangs up first ends the session here. 2. `TwilioFrameSerializer` is built from them, with the account SID and auth token so it can hang up the call. 3. `FastAPIWebsocketTransport` carries the call; the pipeline is OpenAI realtime transcription (with turn detection on OpenAI's side), `OpenAILLMService` and `OpenAITTSService`. The pipeline runs at 16 kHz in and 24 kHz out; the serializer converts to and from Twilio's 8 kHz μ-law. 4. A small processor ends the pipeline when the caller presses `#`; on the `EndFrame`, the serializer `POST`s `Status=completed` to Twilio's REST API. `src/server.ts` serves it on Bun (or Node) with `serveFetch`, which upgrades the media WebSocket with the runtime's own server; `src/deno.ts` serves it with `Deno.serve`. On Cloudflare Workers, serve `createPhoneApp(...)` from a Durable Object per call, as in `examples/workers-durable-voice-agent`. ## Test it without a phone [#test-it-without-a-phone] ```bash bun test examples/twilio-phone-bot ``` The tests need no API key and no Twilio account. A scripted fake Twilio (`test/fake_twilio.ts`) sends `connected` and `start`, waits for the greeting, then plays a recorded question as 20 ms μ-law `media` events in real time and waits for the answer. The OpenAI APIs are the mock from `examples/test-mocks`, and Twilio's REST API is a local stand-in that records the hang-up. The same call runs in process (through a stand-in for Workers' `WebSocketPair`), and on Bun and Deno over a real port. ## Next [#next] * [How phone calls connect](https://polvallverdu.github.io/pipedog/docs/telephony.md), and the other providers' serializers. * [Twilio serializer reference](https://polvallverdu.github.io/pipedog/docs/serializers/twilio.md): every option, regional hosts, and hang-up behavior. # Cloudflare Realtime > CloudflareRealtimeTransport connects a bot to the Cloudflare Realtime SFU over its WebSocket adapters, so users talk over WebRTC while the bot runs in a Worker, a Durable Object or any server. Source: https://polvallverdu.github.io/pipedog/docs/transports/cloudflare-realtime/ ## Overview [#overview] `@pipedog/transport-cloudflare-realtime` connects a bot to [Cloudflare Realtime](https://developers.cloudflare.com/realtime/sfu/), Cloudflare's WebRTC SFU (formerly Calls). The user talks over WebRTC from a browser or an app, with the SFU handling the network. The bot never speaks WebRTC: the SFU's [WebSocket adapters](https://developers.cloudflare.com/realtime/sfu/features/media-transport-adapters/websocket-adapter/) convert its tracks to and from PCM. That makes WebRTC users possible where the bot can't open UDP sockets, in particular [Cloudflare Workers and Durable Objects](https://polvallverdu.github.io/pipedog/docs/deployment/cloudflare-workers.md). This package is pipedog's own; Pipecat has no counterpart. It contains: * **`CloudflareRealtimeTransport`**: the transport. It takes the sockets the SFU dials, and the browser's messages socket, from your server. * **`CloudflareRealtimeAPI`**: a small client for the Realtime HTTPS API: sessions, tracks and WebSocket adapters. * **`CloudflareRealtimeCall`**: the API calls for one user talking to one bot, in order, keeping what it allocated so it can release it. * The adapter's packet codec (`/packet`). A Worker, a Durable Object per call, and a browser page. `transport.ts`, `api.ts` and `packet.ts`. Cloudflare's documentation for sessions, tracks and adapters. ## Installation [#installation] ```bash bun add @pipedog/transport-cloudflare-realtime ``` ```ts import { CloudflareRealtimeParams, CloudflareRealtimeTransport, } from "@pipedog/transport-cloudflare-realtime/transport"; import { CloudflareRealtimeAPI, CloudflareRealtimeCall } from "@pipedog/transport-cloudflare-realtime/api"; ``` ## Prerequisites [#prerequisites] * A Realtime SFU app: its App ID and API token (the app secret), from the Cloudflare dashboard (Realtime > SFU). Keep the secret on your backend. * A public `wss://` URL for the bot: the SFU dials it, and can't reach `localhost`. For local development, use a tunnel such as `cloudflared tunnel`. ## How it works [#how-it-works] ```text browser ──WebRTC (Opus)──► Realtime SFU ──WebSocket adapter (PCM)──► bot: input socket browser ◄──WebRTC (Opus)── Realtime SFU ◄──WebSocket adapter (PCM)── bot: output socket browser ◄────────────── WebSocket (RTVI JSON) ─────────────────────► bot: messages socket ``` * **The SFU dials the bot.** Each adapter connects one track to a WebSocket URL you choose, in one direction. An adapter with `location: "remote"` streams the user's microphone to the bot (the bot's **input**); one with `location: "local"` takes the bot's audio and publishes it as a new track (the bot's **output**). The SFU can't send headers, so authenticate them with a per-call token in the URL. * **Audio** on both adapters is 48 kHz stereo 16-bit PCM, one protobuf packet per WebSocket message, at most 32 KB. The transport converts to and from the pipeline's sample rates and channels, and paces the bot's audio in real time, so an interruption stops it at once. * **App messages** can't ride the adapters, and SFU DataChannels need a WebRTC peer on both ends, so RTVI goes over a third WebSocket that the browser opens to the bot: the **messages** socket. It carries the same JSON messages SmallWebRTC sends over its data channel, one per text frame. * **Reconnects.** The SFU reconnects a dropped input adapter for up to 15 seconds; the transport waits for it (`inputReconnectTimeout`). The output adapter doesn't reconnect. The order of a call, which `CloudflareRealtimeCall` follows: 1. The browser publishes its microphone: it sends an SDP offer; your backend creates a session, publishes the track (`publishUser`) and returns the SFU's answer. 2. Once the browser is connected, your backend creates both adapters (`attachBot`). The SFU dials both endpoints before the request returns, so the transport must already exist. 3. Your backend subscribes the browser's session to the bot's track (`subscribeUser`, or `connectBot` for steps 2 and 3) and returns the SFU's offer. The browser answers, and your backend submits the answer (`renegotiate`). 4. At the end, `close()` closes the adapters and force-closes the browser's tracks. What it can't close stays in `call.state`, plain JSON you can store and retry from. ## Configuration [#configuration] `new CloudflareRealtimeTransport(params?, options?)`. `params` is a `CloudflareRealtimeParams`, with the usual transport parameters (`audioInEnabled`, `audioOutEnabled`, sample rates, channels, filters) plus: | Option | Type | Required | Description | |---|---|---|---| | `audioOutEndSilenceSecs` | `number \| undefined` | no | How much silence (seconds) to send after an EndFrame (0 for no silence). | | `audioOutAutoSilence` | `boolean \| undefined` | no | Insert silence frames when the audio output queue is empty. When false, the transport will wait for audio data instead of inserting silence. | | `audioOutWriteTimeoutSecs` | `number \| undefined` | no | How long (seconds) a single write to the transport may take before the peer is considered gone. A client that stops reading leaves the write waiting with nothing to fail, so it would otherwise never return. | | `inputReconnectTimeout` | `number \| undefined` | no | Seconds to wait for the SFU to reconnect the input adapter after its socket drops, before the client counts as disconnected. The SFU retries the same endpoint for up to 15 seconds. Defaults to 15. | | `disconnectOnMessagesClose` | `boolean \| undefined` | no | Whether the messages socket closing counts as the client disconnecting. With a browser client it is the most reliable signal that the user left, since the SFU keeps the adapters open. Defaults to true. | | `audioOutEnabled` | `boolean \| undefined` | no | Enable audio output streaming. | | `audioOutSampleRate` | `number \| null \| undefined` | no | Output audio sample rate in Hz. | | `audioOutChannels` | `number \| undefined` | no | Number of output audio channels. | | `audioOutBitrate` | `number \| undefined` | no | Output audio bitrate in bits per second. | | `audioOut10msChunks` | `number \| undefined` | no | Number of 10ms chunks to buffer for output. | | `audioOutFilter` | `BaseAudioFilter \| null \| undefined` | no | Audio filter to apply to output audio of the default destination, before the mixer. Audio the transport generates itself (mixer audio, end silence, DTMF tones) is not filtered. | | `audioOutVolume` | `number \| undefined` | no | Volume of the output audio, as a multiplier on its samples, applied after the filter and before the mixer. A `VolumeFrame` changes it while the transport runs, and a `VolumeGainFrame` adjusts it for some audio. | | `audioOutMixer` | `AudioOutMixer \| null \| undefined` | no | Audio mixer instance or destination mapping. | | `audioOutDestinations` | `string[] \| undefined` | no | List of audio output destination identifiers. | | `audioInEnabled` | `boolean \| undefined` | no | Enable audio input streaming. | | `audioInSampleRate` | `number \| null \| undefined` | no | Input audio sample rate in Hz. | | `audioInChannels` | `number \| undefined` | no | Number of input audio channels. | | `audioInFilter` | `BaseAudioFilter \| null \| undefined` | no | Audio filter to apply to input audio. | | `audioInStreamOnStart` | `boolean \| undefined` | no | Start audio streaming immediately on transport start. | | `audioInPassthrough` | `boolean \| undefined` | no | Pass through input audio frames downstream. | | `videoInEnabled` | `boolean \| undefined` | no | Enable video input streaming. | | `videoInSources` | `Record \| undefined` | no | Video sources to capture from each user as they connect, keyed by source (`"camera"`, `"screenVideo"`, or a transport-specific custom source). Sources not listed here are captured only when the application asks for them, e.g. with the transport's `captureParticipantVideo()`. Requires `videoInEnabled`. Transports that don't support it log a warning and ignore it, so check the logs if video doesn't arrive as configured. | | `videoOutEnabled` | `boolean \| undefined` | no | Enable video output streaming. | | `videoOutIsLive` | `boolean \| undefined` | no | Enable real-time video output streaming. | | `videoOutWidth` | `number \| undefined` | no | Video output width in pixels. | | `videoOutHeight` | `number \| undefined` | no | Video output height in pixels. | | `videoOutFramerate` | `number \| undefined` | no | Video output frame rate in FPS. | | `videoOutColorFormat` | `string \| undefined` | no | Video output color format string. | | `videoOutCodec` | `string \| null \| undefined` | no | Preferred video codec for output (e.g., 'VP8', 'H264', 'H265'). | | `videoOutDestinations` | `string[] \| undefined` | no | List of video output destination identifiers. | `options`: | Option | Type | Required | Description | |---|---|---|---| | `clientId` | `string \| null \| undefined` | no | The client's id, as `getClientId` returns it. Defaults to a random UUID. | | `name` | `string \| null \| undefined` | no | Optional name for the transport. | | `inputName` | `string \| null \| undefined` | no | Optional name for the input processor. | | `outputName` | `string \| null \| undefined` | no | Optional name for the output processor. | `CloudflareRealtimeAPI` takes: | Option | Type | Required | Description | |---|---|---|---| | `appId` | `string` | yes | The Realtime SFU app ID. | | `appSecret` | `string` | yes | The app secret (the API token shown with the app). Keep it on the backend. | | `baseUrl` | `string \| undefined` | no | The API base URL. Defaults to `https://rtc.live.cloudflare.com/v1`. | | `fetch` | `FetchFn \| undefined` | no | The `fetch` to use. Defaults to the global `fetch`. | | `timeout` | `number \| undefined` | no | Seconds before a request times out. Defaults to 15. | ## Usage [#usage] The bot is an ordinary pipeline around the transport: ```ts title="bot.ts" import { Pipeline } from "@pipedog/core/pipeline/pipeline"; import { PipelineParams, PipelineWorker } from "@pipedog/core/pipeline/worker"; import { WorkerRunner } from "@pipedog/core/workers/runner"; import { CloudflareRealtimeAPI, CloudflareRealtimeCall, type SessionDescription, } from "@pipedog/transport-cloudflare-realtime/api"; import { CloudflareRealtimeParams, CloudflareRealtimeTransport, } from "@pipedog/transport-cloudflare-realtime/transport"; /** Starts a bot on `transport`; it runs until the user leaves. */ async function runBot(transport: CloudflareRealtimeTransport): Promise { const pipeline = new Pipeline([ transport.input(), // ... STT, LLM context aggregators, LLM, TTS transport.output(), ]); const worker = new PipelineWorker(pipeline, { // The transport converts to and from the SFU's 48 kHz stereo. params: new PipelineParams({ audioInSampleRate: 16000, audioOutSampleRate: 24000 }), }); const runner = new WorkerRunner({ handleSigint: false, handleSigterm: false }); transport.eventHandler("on_client_connected", async () => { console.log("the SFU attached both adapters"); }); transport.eventHandler("on_client_disconnected", async () => { await runner.cancel(); }); await runner.addWorkers(worker); await runner.run(); } ``` Your server does the signaling for the browser and hands the transport the sockets. As a Durable Object, or any fetch handler that keeps one object per call: ```ts title="voice_call.ts" /** One call, e.g. a Durable Object: signaling, the sockets the SFU dials, and the bot. */ class VoiceCall { readonly #env: Env; #call: CloudflareRealtimeCall | null = null; #transport: CloudflareRealtimeTransport | null = null; #token = crypto.randomUUID(); constructor(env: Env) { this.#env = env; } async fetch(request: Request): Promise { const url = new URL(request.url); const action = url.pathname.split("/").slice(4).join("/"); // /api/call// if (action === "offer") { // 1. The browser's microphone offer: publish it, and start the bot. const { sessionDescription, mid } = (await request.json()) as { sessionDescription: SessionDescription; mid: string; }; const api = new CloudflareRealtimeAPI({ appId: this.#env.CLOUDFLARE_REALTIME_APP_ID, appSecret: this.#env.CLOUDFLARE_REALTIME_APP_SECRET, }); this.#call = new CloudflareRealtimeCall(api); this.#transport = new CloudflareRealtimeTransport( new CloudflareRealtimeParams({ audioInEnabled: true, audioOutEnabled: true }), ); void runBot(this.#transport).finally(() => this.#call?.close()); const answer = await this.#call.publishUser(sessionDescription, mid); return Response.json({ sessionDescription: answer.sessionDescription }); } if (action === "connect" && this.#call) { // 2. The browser is connected: the SFU dials both endpoints during this call. const media = `wss://${url.host}${url.pathname.replace(/connect$/, "media")}`; const offer = await this.#call.connectBot({ input: `${media}/input?token=${this.#token}`, output: `${media}/output?token=${this.#token}`, }); return Response.json({ sessionDescription: offer.sessionDescription }); } if (action === "answer" && this.#call) { // 3. The browser's answer to the bot's track. const { sessionDescription } = (await request.json()) as { sessionDescription: SessionDescription; }; await this.#call.renegotiate(sessionDescription); return Response.json({ ok: true }); } if (this.#transport && (action === "media/input" || action === "media/output")) { // The SFU can't send headers: check the token from the URL. if (url.searchParams.get("token") !== this.#token) { return new Response("forbidden", { status: 403 }); } return this.#transport.acceptSocket(action === "media/input" ? "input" : "output", request); } if (this.#transport && action === "messages") { // The browser's RTVI messages. return this.#transport.acceptSocket("messages", request); } return new Response("not found", { status: 404 }); } } ``` `acceptSocket(role, request)` upgrades the request and attaches the socket. It upgrades from the request alone on Workers, Durable Objects and Deno; on Bun, pass `{ server }` and serve with `bunWebSocket`, and on Node serve with `serveFetch()`, as for the [WebSocket transport](https://polvallverdu.github.io/pipedog/docs/transports/websocket.md). With a socket you upgraded yourself, call `attachSocket(role, WebsocketConnection.fromWebSocket(ws))`. ### The browser [#the-browser] There's no Pipecat client SDK transport for Cloudflare Realtime yet, so the browser side is plain WebRTC: publish the microphone, play the bot's track, and speak RTVI over the messages socket: ```ts title="client.ts" /** The browser: publish the microphone, play the bot, and speak RTVI over the messages socket. */ async function call(base: string): Promise<{ pc: RTCPeerConnection; messages: WebSocket }> { const post = async (action: string, body: unknown = {}) => { const response = await fetch(`${base}/${action}`, { method: "POST", body: JSON.stringify(body), }); return (await response.json()) as { sessionDescription: RTCSessionDescriptionInit }; }; const gathered = (pc: RTCPeerConnection) => new Promise((resolve) => { if (pc.iceGatheringState === "complete") return resolve(); pc.addEventListener("icegatheringstatechange", () => { if (pc.iceGatheringState === "complete") resolve(); }); setTimeout(resolve, 2000); }); const mic = await navigator.mediaDevices.getUserMedia({ audio: true }); const pc = new RTCPeerConnection({ iceServers: [{ urls: "stun:stun.cloudflare.com:3478" }], bundlePolicy: "max-bundle", }); const audio = new Audio(); audio.autoplay = true; pc.addEventListener("track", (event) => { audio.srcObject = new MediaStream([event.track]); }); // 1. Publish the microphone. const transceiver = pc.addTransceiver(mic.getAudioTracks()[0]!, { direction: "sendonly" }); await pc.setLocalDescription(await pc.createOffer()); await gathered(pc); const published = await post("offer", { sessionDescription: pc.localDescription, mid: transceiver.mid, }); await pc.setRemoteDescription(published.sessionDescription); await new Promise((resolve) => { pc.addEventListener("connectionstatechange", () => { if (pc.connectionState === "connected") resolve(); }); }); // 2. The messages socket (RTVI), then the bot's track. const messages = new WebSocket(`${base.replace(/^http/, "ws")}/messages`); await new Promise((resolve) => messages.addEventListener("open", resolve, { once: true })); const subscription = await post("connect"); await pc.setRemoteDescription(subscription.sessionDescription); await pc.setLocalDescription(await pc.createAnswer()); await gathered(pc); await post("answer", { sessionDescription: pc.localDescription }); // 3. Ready: the bot can greet the user now. messages.send( JSON.stringify({ label: "rtvi-ai", type: "client-ready", id: crypto.randomUUID(), data: { version: "2.0.0", about: { library: "my-app" } }, }), ); return { pc, messages }; } ``` RTVI works as with any transport, since the bot's `RTVIProcessor` sends and receives through the transport's message frames: the bot answers `client-ready` with `bot-ready`, and sends transcriptions and bot output. A Pipecat client SDK transport would need to do the signaling above and carry RTVI over the messages socket. ### Event handlers [#event-handlers] | Event | Description | | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `on_client_connected` | Both adapter sockets are attached. Receives the client (`getClientId` gives its id). | | `on_client_disconnected` | The input adapter didn't reconnect within `inputReconnectTimeout`, the output adapter closed, or the messages socket closed (with `disconnectOnMessagesClose`, the default). Not when the transport itself closes. | | `on_app_message` | A JSON message arrived on the messages socket. Receives the message and the client's id. | ### Notes [#notes] * Greet the user when their browser can hear the bot: the bot's track only plays once the browser has answered the subscription offer. The example greets on RTVI's `on_client_ready`, which the browser triggers after `renegotiate`. * The transport doesn't end the call at the SFU: call `CloudflareRealtimeCall.close()` when the bot stops, or the SFU keeps redialing a closed input adapter for 15 seconds. * Input is mono at the pipeline's rate; with `audioInChannels: 2` and a 48 kHz pipeline it stays stereo. Set `audioOutSampleRate: 48000` to skip output resampling. * Audio that arrives before the pipeline starts is dropped; bot audio sent while the output adapter isn't attached is dropped. ## RealtimeKit [#realtimekit] RealtimeKit, Cloudflare's meeting SDK, is out of scope for now: its agent participant can't send or receive audio yet (it is "planned" in Cloudflare's capability matrix). This transport uses the SFU directly. ## Verification [#verification] The transport, the API helper and the example are tested against fakes built from Cloudflare's documentation: a fake SFU that dials the bot's endpoints as the adapters do, and a fake API. They haven't been run against the real Realtime service yet. A live API test runs when `CLOUDFLARE_REALTIME_APP_ID` and `CLOUDFLARE_REALTIME_APP_SECRET` are set. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/transport-cloudflare-realtime/package.json`: ✅ means the compat smoke test passes on that runtime. # Daily > WebRTC rooms on Daily, with telephony (PSTN and SIP) and recording. Blocked in pipedog: not ported yet. Source: https://polvallverdu.github.io/pipedog/docs/transports/daily/ {/* Generated by scripts/generate-providers.ts from data/providers.ts. */} Daily is blocked: there is no viable JS path for it today. **Why.** Upstream uses the native daily-python SDK. Daily publishes no headless or server-side JS SDK, and @daily-co/daily-js is browser-only. Tavus and LemonSlice run on Daily rooms, so they are blocked too. **What would unblock it.** Unblocked if Daily ships a Node or server SDK, or if someone builds a headless media path. The REST helpers for rooms and tokens are portable now, and a pull request for them would be accepted. Sources: [daily-js quickstart](https://docs.daily.co/docs/daily-js/quickstart). ## What it is [#what-it-is] WebRTC rooms on Daily, with telephony (PSTN and SIP) and recording. Upstream, it's this class: `DailyTransport`, in [`transports/daily/transport.py`](https://github.com/pipecat-ai/pipecat/blob/main/src/pipecat/transports/daily/transport.py). Its pipedog package will be `@pipedog/transport-daily`. * [Pipecat's Daily page](https://docs.pipecat.ai/api-reference/server/services/transport/daily) ## Get it ported [#get-it-ported] * [Search the issues](https://github.com/polvallverdu/pipedog/issues?q=is%3Aissue+Daily) for a request, and upvote it with a reaction. * If there's none, [open a port request](https://github.com/polvallverdu/pipedog/issues/new?template=feature_request.yml\&title=Port+Daily+%28DailyTransport%29\&kind=Port+an+upstream+module\&upstream=src%2Fpipecat%2Ftransports%2Fdaily%2Ftransport.py). * Or port it yourself: the [contributing guide](https://polvallverdu.github.io/pipedog/docs/contributing.md) explains how modules are ported, test by test. The [porting status](https://polvallverdu.github.io/pipedog/docs/contributing/status.md) lists what's done. Until then, see what is ported in [Transports](https://polvallverdu.github.io/pipedog/docs/transports.md). # Transports > Move audio, video and messages between the user and the pipeline. Source: https://polvallverdu.github.io/pipedog/docs/transports/ {/* Generated by scripts/generate-providers.ts from data/providers.ts. */} A transport moves audio, video and messages between the user and the pipeline: `transport.input()` goes first in a pipeline and `transport.output()` near the end. | Provider | Classes | Package | Status | | ----------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- | ---------------------------------------- | ------------ | | [WebSocket](https://polvallverdu.github.io/pipedog/docs/transports/websocket.md) | `FastAPIWebsocketTransport`, `SingleClientWebsocketServerTransport`, `WebsocketClientTransport`, `RTVIClientTransport` | `@pipedog/transport-websocket` | Ported | | [Cloudflare Realtime](https://polvallverdu.github.io/pipedog/docs/transports/cloudflare-realtime.md) | `CloudflareRealtimeTransport`, `CloudflareRealtimeCall` | `@pipedog/transport-cloudflare-realtime` | pipedog only | | [SmallWebRTC](https://polvallverdu.github.io/pipedog/docs/transports/smallwebrtc.md) | `SmallWebRTCTransport` | `@pipedog/transport-smallwebrtc` | Ported | | [LiveKit](https://polvallverdu.github.io/pipedog/docs/transports/livekit.md) | `LiveKitTransport` | `@pipedog/transport-livekit` | Ported | | [WhatsApp](https://polvallverdu.github.io/pipedog/docs/transports/whatsapp.md) | `WhatsAppClient`, `WhatsAppApi` | `@pipedog/transport-whatsapp` | Ported | | [Local audio](https://polvallverdu.github.io/pipedog/docs/transports/local-audio.md) | `LocalAudioTransport` | `@pipedog/transport-local` | Ported | | [Daily](https://polvallverdu.github.io/pipedog/docs/transports/daily.md) | `DailyTransport` | | Blocked | | [LemonSlice](https://polvallverdu.github.io/pipedog/docs/transports/lemonslice.md) | `LemonSliceTransport`, `LemonSliceApi` | | Blocked | | [MoQ](https://polvallverdu.github.io/pipedog/docs/transports/moq.md) | `MOQTransport`, `MOQParams` | | Deferred | | [Vonage Video](https://polvallverdu.github.io/pipedog/docs/transports/vonage.md) | `VonageVideoConnectorTransport` | | Blocked | Pipecat's [transports reference](https://docs.pipecat.ai/api-reference/server/services/transport) lists every upstream provider. Those not in this table are on [Other upstream providers](https://polvallverdu.github.io/pipedog/docs/services/other-upstream.md). # LemonSlice > LemonSlice's realtime video avatars, delivered through Daily rooms. Blocked in pipedog: not ported yet. Source: https://polvallverdu.github.io/pipedog/docs/transports/lemonslice/ {/* Generated by scripts/generate-providers.ts from data/providers.ts. */} LemonSlice is blocked: there is no viable JS path for it today. **Why.** LemonSliceTransport is built on the Daily transport (its params extend DailyParams), which has no server-side JS path (see Daily). **What would unblock it.** Unblocked when Daily gets a server-side JS transport. The REST client (LemonSliceApi) is portable on its own. Sources: [Pipecat's LemonSlice transport](https://docs.pipecat.ai/server/services/transport/lemonslice). ## What it is [#what-it-is] LemonSlice's realtime video avatars, delivered through Daily rooms. Upstream, it's these classes: `LemonSliceTransport`, `LemonSliceApi`, in [`transports/lemonslice/transport.py`](https://github.com/pipecat-ai/pipecat/blob/main/src/pipecat/transports/lemonslice/transport.py) and [`transports/lemonslice/api.py`](https://github.com/pipecat-ai/pipecat/blob/main/src/pipecat/transports/lemonslice/api.py). Its pipedog package will be `@pipedog/transport-lemonslice`. * [Pipecat's LemonSlice page](https://docs.pipecat.ai/server/services/transport/lemonslice) ## Get it ported [#get-it-ported] * [Search the issues](https://github.com/polvallverdu/pipedog/issues?q=is%3Aissue+LemonSlice) for a request, and upvote it with a reaction. * If there's none, [open a port request](https://github.com/polvallverdu/pipedog/issues/new?template=feature_request.yml\&title=Port+LemonSlice+%28LemonSliceTransport%2C+LemonSliceApi%29\&kind=Port+an+upstream+module\&upstream=src%2Fpipecat%2Ftransports%2Flemonslice%2Ftransport.py%2C+src%2Fpipecat%2Ftransports%2Flemonslice%2Fapi.py). * Or port it yourself: the [contributing guide](https://polvallverdu.github.io/pipedog/docs/contributing.md) explains how modules are ported, test by test. The [porting status](https://polvallverdu.github.io/pipedog/docs/contributing/status.md) lists what's done. Until then, see what is ported in [Transports](https://polvallverdu.github.io/pipedog/docs/transports.md). # LiveKit > WebRTC rooms on a LiveKit server or LiveKit Cloud, with LiveKitTransport, LiveKitRESTHelper and the runner's token helpers. Source: https://polvallverdu.github.io/pipedog/docs/transports/livekit/ ## Overview [#overview] `LiveKitTransport` joins a LiveKit room as a participant. Every other participant in the room (a browser with LiveKit's client SDK, a phone call through LiveKit SIP, another bot) can talk to the bot. LiveKit's server routes the media, so the bot needs no public UDP ports of its own. * **`LiveKitTransport`**: audio in and out (one `pipecat-audio` microphone track out, each participant's audio in), optional video out (one `pipecat-video` camera track) and video in (camera and screen share), data messages (RTVI travels over them), participant metadata, and SIP DTMF in and out. * **`LiveKitRESTHelper`**: LiveKit's room API over `fetch` (`deleteRoomByName`). * **`@pipedog/runner/livekit`**: token helpers (`generateToken`, `generateTokenWithAgent`, `generateSessionTokens`, `configure`) and the development runner's `-t livekit`. Pipecat's documentation for the same transport. `transport.ts`, `utils.ts` and `rtc.ts`. Servers, LiveKit Cloud, tokens and client SDKs. ## Runtimes [#runtimes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ❌ no | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/transport-livekit/package.json`: ✅ means the compat smoke test passes on that runtime. | | Bun | Node ≥ 22 | Deno | Cloudflare Workers | | ----------------------------------------------------------------- | --- | --------- | ----------------- | ------------------ | | `LiveKitTransport` (media) | ✅ | ✅ | ✅ (`--allow-all`) | ❌ | | `LiveKitRESTHelper`, `@pipedog/runner/livekit` (tokens, room API) | ✅ | ✅ | ✅ | not verified | The transport's media goes through LiveKit's Node SDK, `@livekit/rtc-node`, which loads a native N-API addon. The end-to-end test joins a real room on Bun 1.4, Node 24 and Deno 2.9 and checks data messages and audio both ways. Workers can't load native code, so the transport doesn't run there. Importing the package works everywhere: the addon is only loaded when the transport is set up. The token helpers sign with `livekit-server-sdk`, which is plain JavaScript on WebCrypto and `fetch`. ## Installation [#installation] ```bash bun add @pipedog/transport-livekit @livekit/rtc-node ``` ```ts import { LiveKitParams, LiveKitTransport } from "@pipedog/transport-livekit/transport"; import { LiveKitRESTHelper } from "@pipedog/transport-livekit/utils"; import { configure, generateSessionTokens } from "@pipedog/runner/livekit"; ``` `@livekit/rtc-node` is an optional peer dependency, so install it yourself (1.1.0 or newer in the 1.x line). It ships prebuilt binaries for these platforms only: | OS | Architectures | | ---------------- | ------------------------------------------------ | | macOS | arm64 (`darwin-arm64`), x64 (`darwin-x64`) | | Linux with glibc | x64 (`linux-x64-gnu`), arm64 (`linux-arm64-gnu`) | | Windows | x64 (`win32-x64-msvc`) | There is no build for musl (Alpine Linux), so use a Debian or Ubuntu based image. Without the package, or on another platform, setting up the transport fails with an error that says how to install it. ## Prerequisites [#prerequisites] * **A LiveKit server.** Either [LiveKit Cloud](https://cloud.livekit.io) or your own `livekit-server` (for local development: `docker run --rm -p 7880:7880 -p 7881:7881 -p 7882:7882/udp livekit/livekit-server --dev --node-ip 127.0.0.1`, whose key is `devkey` and secret `secret`). * **Its API key and secret**, to sign access tokens: `LIVEKIT_API_KEY` and `LIVEKIT_API_SECRET`. The server URL goes in `LIVEKIT_URL` (`wss://.livekit.cloud`, or `ws://localhost:7880`). * **A client** in the same room: LiveKit's [Agents Playground](https://agents-playground.livekit.io), [Meet](https://meet.livekit.io), or your own app with a LiveKit client SDK. * A long-running Bun, Node or Deno process on one of the platforms above. ## LiveKitTransport [#livekittransport] ### Configuration [#configuration] `new LiveKitTransport({ url, token, roomName, params?, inputName?, outputName? })`. The token is an access token whose identity is the bot's (see [Tokens](#tokens)). | Option | Type | Required | Description | |---|---|---|---| | `url` | `string` | yes | LiveKit server URL to connect to. | | `token` | `string` | yes | Authentication token for the room. | | `roomName` | `string` | yes | Name of the LiveKit room to join. | | `params` | `LiveKitParams \| null \| undefined` | no | Configuration parameters for the transport. | | `inputName` | `string \| null \| undefined` | no | Optional name for the input transport. | | `outputName` | `string \| null \| undefined` | no | Optional name for the output transport. | `LiveKitParams` is a `TransportParams` with three more fields: | Option | Type | Required | Description | |---|---|---|---| | `audioInUserTracks` | `boolean \| undefined` | no | Receive each participant's audio as its own stream of `UserAudioRawFrame`s tagged with their identity (the default). All of these streams enter the pipeline through the one input transport, so a pipeline that runs a single STT and VAD should only receive one of them. When false, the participants' audio is mixed into a single stream of `InputAudioRawFrame`s. | | `audioOutQueueSizeMs` | `number \| undefined` | no | Buffer size of the outgoing audio source, in milliseconds (LiveKit's default is 1000). | | `videoOutMaxBitrate` | `number \| null \| undefined` | no | Maximum bitrate of the published video track, in bits per second, capped at `videoOutFramerate`. LiveKit chooses the encoding from the track resolution when unset. | | `audioOutEnabled` | `boolean \| undefined` | no | Enable audio output streaming. | | `audioOutSampleRate` | `number \| null \| undefined` | no | Output audio sample rate in Hz. | | `audioOutChannels` | `number \| undefined` | no | Number of output audio channels. | | `audioOutBitrate` | `number \| undefined` | no | Output audio bitrate in bits per second. | | `audioOut10msChunks` | `number \| undefined` | no | Number of 10ms chunks to buffer for output. | | `audioOutFilter` | `BaseAudioFilter \| null \| undefined` | no | Audio filter to apply to output audio of the default destination, before the mixer. Audio the transport generates itself (mixer audio, end silence, DTMF tones) is not filtered. | | `audioOutVolume` | `number \| undefined` | no | Volume of the output audio, as a multiplier on its samples, applied after the filter and before the mixer. A `VolumeFrame` changes it while the transport runs, and a `VolumeGainFrame` adjusts it for some audio. | | `audioOutMixer` | `AudioOutMixer \| null \| undefined` | no | Audio mixer instance or destination mapping. | | `audioOutDestinations` | `string[] \| undefined` | no | List of audio output destination identifiers. | | `audioOutEndSilenceSecs` | `number \| undefined` | no | How much silence (seconds) to send after an EndFrame (0 for no silence). | | `audioOutAutoSilence` | `boolean \| undefined` | no | Insert silence frames when the audio output queue is empty. When false, the transport will wait for audio data instead of inserting silence. | | `audioOutWriteTimeoutSecs` | `number \| undefined` | no | How long (seconds) a single write to the transport may take before the peer is considered gone. A client that stops reading leaves the write waiting with nothing to fail, so it would otherwise never return. | | `audioInEnabled` | `boolean \| undefined` | no | Enable audio input streaming. | | `audioInSampleRate` | `number \| null \| undefined` | no | Input audio sample rate in Hz. | | `audioInChannels` | `number \| undefined` | no | Number of input audio channels. | | `audioInFilter` | `BaseAudioFilter \| null \| undefined` | no | Audio filter to apply to input audio. | | `audioInStreamOnStart` | `boolean \| undefined` | no | Start audio streaming immediately on transport start. | | `audioInPassthrough` | `boolean \| undefined` | no | Pass through input audio frames downstream. | | `videoInEnabled` | `boolean \| undefined` | no | Enable video input streaming. | | `videoInSources` | `Record \| undefined` | no | Video sources to capture from each user as they connect, keyed by source (`"camera"`, `"screenVideo"`, or a transport-specific custom source). Sources not listed here are captured only when the application asks for them, e.g. with the transport's `captureParticipantVideo()`. Requires `videoInEnabled`. Transports that don't support it log a warning and ignore it, so check the logs if video doesn't arrive as configured. | | `videoOutEnabled` | `boolean \| undefined` | no | Enable video output streaming. | | `videoOutIsLive` | `boolean \| undefined` | no | Enable real-time video output streaming. | | `videoOutWidth` | `number \| undefined` | no | Video output width in pixels. | | `videoOutHeight` | `number \| undefined` | no | Video output height in pixels. | | `videoOutFramerate` | `number \| undefined` | no | Video output frame rate in FPS. | | `videoOutColorFormat` | `string \| undefined` | no | Video output color format string. | | `videoOutCodec` | `string \| null \| undefined` | no | Preferred video codec for output (e.g., 'VP8', 'H264', 'H265'). | | `videoOutDestinations` | `string[] \| undefined` | no | List of video output destination identifiers. | ### Usage [#usage] ```ts title="bot.ts" import { Pipeline } from "@pipedog/core/pipeline/pipeline"; import { PipelineWorker } from "@pipedog/core/pipeline/worker"; import { WorkerRunner } from "@pipedog/core/workers/runner"; import { configure } from "@pipedog/runner/livekit"; import { LiveKitParams, LiveKitTransport } from "@pipedog/transport-livekit/transport"; // LIVEKIT_URL, LIVEKIT_API_KEY, LIVEKIT_API_SECRET and LIVEKIT_ROOM_NAME (or -r/--room). // configure() mints an agent token for the bot and logs one for a user. const [url, token, roomName] = await configure(); const transport = new LiveKitTransport({ url, token, roomName, params: new LiveKitParams({ audioInEnabled: true, audioOutEnabled: true }), }); const pipeline = new Pipeline([ transport.input(), // ... STT, LLM context aggregators, LLM, TTS transport.output(), ]); const worker = new PipelineWorker(pipeline); transport.eventHandler( "on_first_participant_joined", async (_t: unknown, participantId: string) => { console.log(`${participantId} joined`); }, ); transport.eventHandler("on_participant_disconnected", async () => { await worker.cancel(); }); const runner = new WorkerRunner(); await runner.addWorkers(worker); await runner.run(); ``` ### Event handlers [#event-handlers] Every `participantId` is the participant's LiveKit **identity**, the one set in its token, not its session id. | Event | Arguments | When | | ---------------------------------------------------------- | ---------------------------------------------------- | ------------------------------------------------------------------------------------- | | `on_connected` | `(transport)` | The bot joined the room. | | `on_disconnected` | `(transport)` | The bot left the room, or the server disconnected it. | | `on_before_disconnect` | `(transport)` | Sync, just before the bot leaves. | | `on_first_participant_joined` | `(transport, participantId)` | The first participant joined (or was already in the room). | | `on_participant_connected` | `(transport, participantId)` | A participant joined. | | `on_participant_disconnected` | `(transport, participantId)` | A participant left. | | `on_participant_left` | `(transport, participantId, reason)` | Same, with `"disconnected"`. | | `on_client_connected`, `on_client_disconnected` | `(transport, { id })` | Aliases with the shape other transports use. | | `on_audio_track_subscribed`, `on_audio_track_unsubscribed` | `(transport, participantId)` | A participant's audio started or stopped. | | `on_video_track_subscribed`, `on_video_track_unsubscribed` | `(transport, participantId)` | A participant's video started or stopped. | | `on_app_message` | `(transport, message, participantId)` | A JSON object arrived as a data message (RTVI uses this). | | `on_data_received` | `(transport, data, participantId)` | Any data message, as bytes. `participantId` is `null` for messages a server SDK sent. | | `on_dtmf_event` | `(transport, { tone, digit, code, participant_id })` | A SIP participant pressed a key. An `InputDTMFFrame` is pushed too. | | `on_active_speaker_changed` | `(transport, { id })` | The room's loudest speaker changed. | ### Methods [#methods] * `getParticipants()`, `getParticipantMetadata(participantId)`, `setMetadata(metadata)`. * `muteParticipant(participantId)` / `unmuteParticipant(participantId)`: stop or resume receiving a participant's audio (LiveKit doesn't let one participant mute another's microphone). * `sendMessage(message, participantId?)` / `sendMessageUrgent(...)`: a data message to everyone or to one participant. `OutputTransportMessageFrame`s in the pipeline go out the same way. * `captureParticipantVideo(participantId, framerate?, videoSource?, { onRequestOnly })`. ### Notes [#notes] * Audio comes in at 48 kHz from LiveKit and is resampled to the pipeline's rate. With `audioInUserTracks` (the default) each participant's audio is a stream of `UserAudioRawFrame`s with their identity; set it to `false` to mix everyone into one stream. * `OutputDTMFFrame` is sent as a native DTMF event (`publishDtmf`), one key per frame. * An `InterruptionFrame` clears the audio already queued in LiveKit's audio source, so the bot stops talking at once. * Connecting is retried 3 times, 4 seconds apart. ## Tokens [#tokens] The bot and each user need their own token for the room, signed with the API key and secret. `@pipedog/runner/livekit` signs them with `livekit-server-sdk`: * `generateTokenWithAgent(room, identity, key, secret)`: the bot's token (`agent: true`, so LiveKit's clients show it as an agent). * `generateToken(room, identity, key, secret)`: a user's token. * `generateSessionTokens(room, sessionId, key, secret)`: both, with identities suffixed by the session id so sessions sharing a room don't evict each other. * `configure()`: reads `LIVEKIT_URL`, `LIVEKIT_API_KEY`, `LIVEKIT_API_SECRET` and the room (`-r/--room` or `LIVEKIT_ROOM_NAME`), and returns `[url, botToken, roomName]`. It logs a user token you can paste into a client. All of them are async (signing uses WebCrypto). ## With the development runner [#with-the-development-runner] `bun bot.ts -t livekit` serves `POST /start`. Each call creates a session: it mints a bot token and a user token for `LIVEKIT_ROOM_NAME` (or a new `pipecat-xxxxxxxx` room), starts your `bot()` with `LiveKitRunnerArguments`, and answers `{ url, token, sessionId }` for the user. `createTransport` builds the transport from those arguments: ```ts title="bot.ts" import type { RunnerArguments } from "@pipedog/runner"; import { createTransport } from "@pipedog/runner/utils"; /** The bot the development runner starts for each session (`bun bot.ts -t livekit`). */ export async function bot(runnerArgs: RunnerArguments): Promise { const transport = await createTransport(runnerArgs, { livekit: () => new LiveKitParams({ audioInEnabled: true, audioOutEnabled: true }), }); // ... build the pipeline around transport.input() and transport.output() void transport; } ``` See [Runner](https://polvallverdu.github.io/pipedog/docs/runner.md). ## LiveKitRESTHelper [#livekitresthelper] `new LiveKitRESTHelper({ apiKey, apiSecret, apiUrl, fetch? })`. `deleteRoomByName(roomName)` deletes a room and disconnects everyone in it. `apiUrl` is the server's HTTP URL (`https://.livekit.cloud`). `fetch` replaces the global one (upstream takes an aiohttp session). # Local audio > The machine's microphone and speakers with LocalAudioTransport, through sox or ffmpeg, for developing a bot without a browser. No native addon. Source: https://polvallverdu.github.io/pipedog/docs/transports/local-audio/ ## Overview [#overview] `LocalAudioTransport` records from the machine's microphone and plays the bot's voice on its speakers: the quickest way to talk to a bot while you build it, with no browser, phone or media server. It is a port of Pipecat's `LocalAudioTransport`. Upstream uses PyAudio (PortAudio, native). pipedog has **no native addon**: it spawns [`sox`](https://sox.sourceforge.net/) or [`ffmpeg`](https://ffmpeg.org/) and pipes raw 16-bit PCM through the child's stdin and stdout. `transports/local/audio.py` in Pipecat. `audio.ts`, `audio_tools.ts` and `process.ts`. ## Installation [#installation] ```bash bun add @pipedog/transport-local ``` and one of the tools: | | macOS | Debian / Ubuntu | Windows | | ------ | --------------------- | ------------------------- | ---------------------------- | | sox | `brew install sox` | `sudo apt install sox` | `choco install sox.portable` | | ffmpeg | `brew install ffmpeg` | `sudo apt install ffmpeg` | `choco install ffmpeg` | With neither on the `PATH`, `setup()` throws `LocalAudioToolNotFoundError`, with the install commands in its message. `detectAudioTool()` does the same lookup on its own. ```ts import { LocalAudioTransport, LocalAudioTransportParams } from "@pipedog/transport-local"; ``` ## Runtimes [#runtimes] | Runtime | Works | What it needs | | ------------------ | ----- | ----------------------------------------------------------------------------------------------------------- | | Bun | ✅ yes | sox or ffmpeg on the PATH, and a machine with audio devices. | | Node.js | ✅ yes | sox or ffmpeg on the PATH, and a machine with audio devices. | | Deno | ✅ yes | sox or ffmpeg on the PATH, and a machine with audio devices. Run with --allow-run. | | Cloudflare Workers | ❌ no | sox or ffmpeg on the PATH, and a machine with audio devices. Not supported: Workers cannot start processes. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/transport-local/package.json`: ✅ means the compat smoke test passes on that runtime. Processes are started with `Bun.spawn`, `Deno.Command` or `node:child_process`, whichever runtime you are on. Cloudflare Workers can't start processes. ## Usage [#usage] ```ts import { Pipeline } from "@pipedog/core/pipeline/pipeline"; import { LocalAudioTransport, LocalAudioTransportParams } from "@pipedog/transport-local"; const transport = new LocalAudioTransport( new LocalAudioTransportParams({ audioInEnabled: true, audioOutEnabled: true }), ); const pipeline = new Pipeline([ transport.input(), stt, userAggregator, llm, tts, transport.output(), assistantAggregator, ]); ``` Wear headphones: with speakers, the microphone hears the bot, and the bot interrupts itself. ## Configuration [#configuration] `LocalAudioTransportParams` takes everything `TransportParams` does (sample rates, channels, VAD, filters, mixers), plus: | Option | Type | Default | Description | | -------------- | ----------------------------- | ----------------- | ------------------------------------------------- | | `tool` | `"sox" \| "ffmpeg" \| "auto"` | `"auto"` | The tool to run. `"auto"` picks sox, then ffmpeg. | | `inputDevice` | `string \| number \| null` | system default | The microphone. | | `outputDevice` | `string \| number \| null` | system default | The speakers. | | `spawner` | `ProcessSpawner` | the runtime's own | How to start processes, for tests. | ### Devices [#devices] PortAudio's device indices (upstream's `input_device_index`) don't exist here. Name a device in the tool's own terms: * **sox**: the device name for the platform's driver: CoreAudio on macOS, ALSA on Linux, WaveAudio on Windows. The default is the system's default device (`sox -d`, which is what `rec` and `play` run). * **ffmpeg**: on macOS an AVFoundation index or name for input and an AudioToolbox index for output; on Linux PulseAudio source and sink names (PipeWire provides them). On Windows input needs a DirectShow name, and ffmpeg has no playback device there, so use sox. ## How it works [#how-it-works] * **Input.** A recorder process writes PCM to stdout. The transport cuts it into 20 ms `InputAudioRawFrame`s and pushes them downstream. If the recorder dies, it pushes an error frame with the tool's stderr. * **Output.** Each `OutputAudioRawFrame` is written to the player's stdin, and the write then waits for the audio's duration, like the other non-blocking output transports. The player stays at most a chunk ahead of real time, so interruptions cut the speech quickly. * Both processes are stopped at cleanup and on cancel. ## Differences from upstream [#differences-from-upstream] * **sox or ffmpeg instead of PyAudio**, and `input_device_index` and `output_device_index` become `inputDevice` and `outputDevice`. * The recorder and player start in `start()`, where upstream opens the streams in `setup()`. * The desktop window transport, `transports/local/tk`, is not ported. * Not exercised in the project's tests: a real microphone and speakers. The tests run against a fake process spawner, and the compat check against a real child process. # MoQ > Media over QUIC (MoQ) relays. Deferred in pipedog: not ported yet. Source: https://polvallverdu.github.io/pipedog/docs/transports/moq/ {/* Generated by scripts/generate-providers.ts from data/providers.ts. */} MoQ is deferred: it is possible, but it isn't ready to port. **Why.** The protocol is still changing. The JS library, @moq/net, falls back to WebSocket on servers and only has experimental WebTransport. **What would unblock it.** Deferred until the spec and server-side JS support settle. Sources: [MoQ JS library](https://doc.moq.dev/js/). ## What it is [#what-it-is] Media over QUIC (MoQ) relays. Upstream, it's these classes: `MOQTransport`, `MOQParams`, in [`transports/moq/transport.py`](https://github.com/pipecat-ai/pipecat/blob/main/src/pipecat/transports/moq/transport.py). Its pipedog package will be `@pipedog/transport-moq`. * [Pipecat's MoQ page](https://docs.pipecat.ai/api-reference/server/services/transport/moq) ## Get it ported [#get-it-ported] * [Search the issues](https://github.com/polvallverdu/pipedog/issues?q=is%3Aissue+MoQ) for a request, and upvote it with a reaction. * If there's none, [open a port request](https://github.com/polvallverdu/pipedog/issues/new?template=feature_request.yml\&title=Port+MoQ+%28MOQTransport%2C+MOQParams%29\&kind=Port+an+upstream+module\&upstream=src%2Fpipecat%2Ftransports%2Fmoq%2Ftransport.py). * Or port it yourself: the [contributing guide](https://polvallverdu.github.io/pipedog/docs/contributing.md) explains how modules are ported, test by test. The [porting status](https://polvallverdu.github.io/pipedog/docs/contributing/status.md) lists what's done. Until then, see what is ported in [Transports](https://polvallverdu.github.io/pipedog/docs/transports.md). # SmallWebRTC > Peer-to-peer WebRTC with SmallWebRTCTransport, SmallWebRTCConnection, SmallWebRTCRequestHandler and the smallWebRTCOfferHandler signalling endpoint. Source: https://polvallverdu.github.io/pipedog/docs/transports/smallwebrtc/ ## Overview [#overview] SmallWebRTC connects one browser (or any WebRTC peer) straight to the bot, with no media server in between. Audio goes over RTP as Opus, and RTVI and app messages go over a data channel. It works with the Pipecat browser SDK's `SmallWebRTCTransport`. * **`SmallWebRTCTransport`**: the pipeline transport for one peer connection: audio in and out, optional camera and screen share, and app messages. * **`SmallWebRTCConnection`**: the peer connection itself (upstream on aiortc, here on [werift](https://github.com/shinyoshiaki/werift-webrtc)), with ICE servers, tracks, the data channel and renegotiation. * **`SmallWebRTCRequestHandler`**: answers SDP offers, creates or reuses connections and applies ICE candidates. Framework-agnostic. * **`smallWebRTCOfferHandler`**: the signalling endpoint (`POST` and `PATCH /api/offer`) as a fetch handler, `(Request) => Promise`. * **`nodeRequestListener`**: adapts a fetch handler to Node's `http.createServer()`. The [voice bot guide](https://polvallverdu.github.io/pipedog/docs/voice-bot.md) builds a complete SmallWebRTC voice agent, with a browser client. Pipecat's documentation for the same transport. `transport.ts`, `connection.ts`, `request_handler.ts` and `http.ts`. A runnable example in `examples/smallwebrtc-voice-bot`. ## Installation [#installation] ```bash bun add @pipedog/transport-smallwebrtc ``` ```ts import { SmallWebRTCTransport } from "@pipedog/transport-smallwebrtc/transport"; import { IceServer, SmallWebRTCConnection } from "@pipedog/transport-smallwebrtc/connection"; import { SmallWebRTCRequestHandler } from "@pipedog/transport-smallwebrtc/request_handler"; import { nodeRequestListener, smallWebRTCOfferHandler } from "@pipedog/transport-smallwebrtc/http"; ``` The package depends on `werift` (a pure TypeScript WebRTC stack: ICE, DTLS, SRTP, SCTP, RTP) and `libopus-wasm` (libopus compiled to WebAssembly). Neither needs native code. ## Prerequisites [#prerequisites] * A long-lived server process that can send and receive UDP. Serverless functions and Cloudflare Workers can't run the media. * A client: the Pipecat browser SDK with `@pipecat-ai/small-webrtc-transport`, pointed at your offer endpoint. * For peers behind restrictive NATs, a TURN server (see [ICE servers](#ice-servers)). ## SmallWebRTCTransport [#smallwebrtctransport] ### Configuration [#configuration] `new SmallWebRTCTransport(connection, params, options?)`. `connection` is the `SmallWebRTCConnection` from the request handler. `params` is a plain `TransportParams` (`audioInEnabled`, `audioOutEnabled`, `videoInEnabled`, `videoOutEnabled` and so on). `options`: | Option | Type | Required | Description | |---|---|---|---| | `inputName` | `string \| null \| undefined` | no | Optional name for the input processor. | | `outputName` | `string \| null \| undefined` | no | Optional name for the output processor. | ### Usage [#usage] Serve the offer endpoint and start a bot for each new connection: ```ts title="server.ts" import { createServer } from "node:http"; import { createTask } from "@pipedog/asyncio"; import { Pipeline } from "@pipedog/core/pipeline/pipeline"; import { PipelineWorker } from "@pipedog/core/pipeline/worker"; import { TransportParams } from "@pipedog/core/transports/base_transport"; import { WorkerRunner } from "@pipedog/core/workers/runner"; import { IceServer, type SmallWebRTCConnection } from "@pipedog/transport-smallwebrtc/connection"; import { nodeRequestListener, smallWebRTCOfferHandler } from "@pipedog/transport-smallwebrtc/http"; import { ConnectionMode, SmallWebRTCRequestHandler, } from "@pipedog/transport-smallwebrtc/request_handler"; import { SmallWebRTCTransport } from "@pipedog/transport-smallwebrtc/transport"; /** Runs one bot on a new peer connection. */ async function runBot(connection: SmallWebRTCConnection): Promise { const transport = new SmallWebRTCTransport( connection, new TransportParams({ audioInEnabled: true, audioOutEnabled: true }), ); const pipeline = new Pipeline([ transport.input(), // ... STT, LLM context aggregators, LLM, TTS transport.output(), ]); const worker = new PipelineWorker(pipeline); const runner = new WorkerRunner({ handleSigint: false }); transport.eventHandler("on_client_connected", async () => { console.log("client connected"); }); transport.eventHandler("on_client_disconnected", async () => { await runner.cancel(); }); await runner.addWorkers(worker); await runner.run(); } // POST /api/offer takes the browser's SDP offer and answers it; PATCH sends ICE candidates. const offer = smallWebRTCOfferHandler(new SmallWebRTCRequestHandler(), (connection) => { // Start the bot in the background: the answer is sent once this returns. createTask(() => runBot(connection), "voice-bot"); }); Bun.serve({ port: 7860, fetch(request) { if (new URL(request.url).pathname === "/api/offer") return offer(request); return new Response("not found", { status: 404 }); }, }); ``` On Node, wrap the same handler with `nodeRequestListener`: ```ts // On Node, serve the same handler with the http module instead of Bun.serve. createServer(nodeRequestListener(offer)).listen(7860); ``` App messages from the peer arrive on the data channel: ```ts async function runWithMessages(connection: SmallWebRTCConnection): Promise { const transport = new SmallWebRTCTransport( connection, new TransportParams({ audioInEnabled: true, audioOutEnabled: true }), ); transport.eventHandler("on_app_message", async (_transport, message: unknown, sender: string) => { console.log(`message from ${sender}:`, message); // Reply on the data channel. connection.sendAppMessage({ type: "ack" }); }); } ``` ### Event handlers [#event-handlers] | Event | Description | | ------------------------ | -------------------------------------------------------------------- | | `on_client_connected` | The peer connection is up. Receives the `SmallWebRTCConnection`. | | `on_client_disconnected` | The peer connection closed. Receives the `SmallWebRTCConnection`. | | `on_app_message` | A data channel message arrived. Receives the message and the sender. | ### Notes [#notes] * After `on_client_connected` runs, the input pushes a `ClientConnectedFrame`. App messages are also pushed into the pipeline as `InputTransportMessageFrame`s. * Video sources listed in `params.videoInSources` are captured before `on_client_connected` runs, so a handler that captures a source itself wins. `captureParticipantVideo()` captures the camera (`"camera"`) or the screen share (`"screenVideo"`), at 30 frames per second by default. * `getClientId(client)` returns the peer connection's `pcId`. * When the peer replaces the connection (`restart_pc`), the old track completes its pending writes instead of leaving them waiting. Upstream leaves them pending, so the output transport's write timeout fails the pipeline. ## SmallWebRTCConnection [#smallwebrtcconnection] You rarely build one yourself: `SmallWebRTCRequestHandler` creates one per offer, with the options from its `connectionOptions` and `iceServers`. ### Configuration [#configuration-1] | Option | Type | Required | Description | |---|---|---|---| | `iceServers` | `string[] \| RTCIceServer[] \| null \| undefined` | no | ICE servers as URLs or IceServer objects. | | `connectionTimeoutSecs` | `number \| undefined` | no | Timeout in seconds for connecting to the peer. | | `videoDecoderFactory` | `VideoDecoderFactory \| null \| undefined` | no | pipedog: decodes the peer's camera and screen share (none by default; see `video.ts`). | | `videoEncoderFactory` | `VideoEncoderFactory \| null \| undefined` | no | pipedog: encodes video sent to the peer (none by default; see `video.ts`). | | `opusCodec` | `OpusCodec \| undefined` | no | pipedog: the Opus implementation (default: libopus-wasm). | | `werift` | `Record \| undefined` | no | pipedog: extra werift `PeerConfig` fields (e.g. `icePortRange`). | ### Event handlers [#event-handlers-1] | Event | Description | | -------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | | `app-message` | A data channel message arrived. Messages that arrive before the client is connected are queued and delivered on connect. | | `track-started` | A remote track started. | | `track-ended` | A remote track ended. | | `track-status` | The peer turned a source on or off. | | `new`, `connecting`, `connected`, `disconnected`, `closed`, `failed` | Peer connection state changes. | ### Notes [#notes-1] * If the connection stays in the connecting state longer than `connectionTimeoutSecs` (default 60\), it is closed. * `sendAppMessage(message)` sends a JSON message on the data channel. * Audio is decoded straight to the pipeline's input rate and channel count when Opus supports it (8, 12, 16, 24 or 48 kHz). aiortc always decodes to 48 kHz stereo. * Opus decoding uses packet loss concealment and in-band FEC. Pass `opusCodec`, or call `setOpusCodec()`, to replace libopus-wasm (with a native binding, for example). * There is no built-in video codec. Without `videoDecoderFactory`, incoming video is negotiated but produces no frames; without `videoEncoderFactory`, outgoing video frames are dropped. A warning is logged once in each case. VP8 and H.264 depayloading and packetization are built in. * The SCTP data chunk payload is 1100 bytes, so data channel packets fit a 1280-byte MTU. Change it with the `PIPECAT_SCTP_MAX_CHUNK_SIZE` environment variable. ## SmallWebRTCRequestHandler [#smallwebrtcrequesthandler] ### Configuration [#configuration-2] | Option | Type | Required | Description | |---|---|---|---| | `iceServers` | `RTCIceServer[] \| null \| undefined` | no | List of ICE servers to use for WebRTC connections. | | `esp32Mode` | `boolean \| undefined` | no | If true, enables ESP32-specific SDP munging. | | `host` | `string \| null \| undefined` | no | Host address used for SDP munging in ESP32 mode. Ignored if `esp32Mode` is false. | | `connectionMode` | `ConnectionMode \| undefined` | no | SINGLE allows only one active connection, MULTIPLE allows several. | | `connectionOptions` | `Omit \| undefined` | no | pipedog: options for each new `SmallWebRTCConnection` (video codecs, Opus implementation, werift settings). `iceServers` comes from above. | ### ICE servers [#ice-servers] Without ICE servers, only host candidates are gathered, which works on a LAN or localhost. Add a STUN server to cross NATs, and a TURN server for networks that block direct UDP: ```ts title="ice.ts" // A TURN server, for peers behind NATs that block direct connections. const handler = new SmallWebRTCRequestHandler({ iceServers: [ new IceServer({ urls: "stun:stun.l.google.com:19302" }), new IceServer({ urls: "turn:turn.example.com:3478", username: process.env.TURN_USERNAME ?? "", credential: process.env.TURN_CREDENTIAL ?? "", }), ], // Accept one peer at a time. connectionMode: ConnectionMode.SINGLE, }); ``` `updateIceServers()` changes the list for connections created afterwards. ### Notes [#notes-2] * `ConnectionMode.MULTIPLE` (the default) accepts several peers. `ConnectionMode.SINGLE` keeps one: an offer with a different `pc_id`, or with none while a connection exists, gets HTTP 400. * An offer with the `pc_id` of an existing connection renegotiates it instead of creating a new one. * `esp32Mode` removes the SHA-384 and SHA-512 fingerprints ESP32 peers don't support and, with `host`, keeps only the ICE candidates that match it. ## smallWebRTCOfferHandler [#smallwebrtcofferhandler] `smallWebRTCOfferHandler(handler, onConnection)` serves one URL, such as `/api/offer`, with the same contract as Pipecat's development runner: | Method | Body | Response | | ------- | -------------------------------------------------------------- | ----------------------- | | `POST` | `{sdp, type, pc_id?, restart_pc?, request_data?}` | `{sdp, type, pc_id}` | | `PATCH` | `{pc_id, candidates: [{candidate, sdp_mid, sdp_mline_index}]}` | `{"status": "success"}` | `onConnection(connection, request)` runs for each new connection, with the parsed `SmallWebRTCRequest` (its `requestData` carries the client's custom data). Start the bot in the background and return: the answer is sent once it returns. Errors follow FastAPI: an `HTTPException` becomes its status with `{"detail": ...}`, a malformed body a 400, another method a 405 and anything else a 500. It is a standard fetch handler, so it mounts in `Bun.serve`, `Deno.serve`, Hono and most frameworks; see [Frameworks](https://polvallverdu.github.io/pipedog/docs/frameworks.md). On Node, `nodeRequestListener(fetchHandler)` turns it into a listener for `http.createServer()`. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ❌ no | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/transport-smallwebrtc/package.json`: ✅ means the compat smoke test passes on that runtime. | Bun | Node | Deno | Cloudflare Workers | | --- | ---- | ---- | ------------------ | | yes | yes | yes | no | WebRTC media runs over UDP. Bun, Node and Deno can open UDP sockets; Cloudflare Workers can't, so the package does not run there. Use the [WebSocket transport](https://polvallverdu.github.io/pipedog/docs/transports/websocket.md) on Workers. See [Runtimes](https://polvallverdu.github.io/pipedog/docs/guides/runtimes.md). The signalling endpoint is plain HTTP, but the connection it creates lives in the same process, so the offer endpoint and the media must run on the same long-lived server. Upstream encodes and decodes with PyAV (FFmpeg). Here Opus comes from libopus-wasm and video codecs are pluggable, as described above. werift's API wears an aiortc-shaped adapter (`rtc.ts`), so the ported code keeps upstream's structure. # Vonage Video > Vonage Video API sessions, through the Video Connector. Blocked in pipedog: not ported yet. Source: https://polvallverdu.github.io/pipedog/docs/transports/vonage/ {/* Generated by scripts/generate-providers.ts from data/providers.ts. */} Vonage Video is blocked: there is no viable JS path for it today. **Why.** Upstream uses a native, Python-only SDK. Vonage lists Node.js as coming soon. The Vonage voice (Audio Connector) serializer is a different thing and is already ported. **What would unblock it.** Unblocked when Vonage ships its Node SDK. Sources: [Vonage Video AI connectors](https://developer.vonage.com/en/video/ai-connectors/overview). ## What it is [#what-it-is] Vonage Video API sessions, through the Video Connector. Upstream, it's this class: `VonageVideoConnectorTransport`, in [`transports/vonage/video_connector.py`](https://github.com/pipecat-ai/pipecat/blob/main/src/pipecat/transports/vonage/video_connector.py). Its pipedog package will be `@pipedog/transport-vonage`. * [Pipecat's Vonage Video page](https://docs.pipecat.ai/api-reference/server/services/transport/vonage) ## Get it ported [#get-it-ported] * [Search the issues](https://github.com/polvallverdu/pipedog/issues?q=is%3Aissue+Vonage+Video) for a request, and upvote it with a reaction. * If there's none, [open a port request](https://github.com/polvallverdu/pipedog/issues/new?template=feature_request.yml\&title=Port+Vonage+Video+%28VonageVideoConnectorTransport%29\&kind=Port+an+upstream+module\&upstream=src%2Fpipecat%2Ftransports%2Fvonage%2Fvideo_connector.py). * Or port it yourself: the [contributing guide](https://polvallverdu.github.io/pipedog/docs/contributing.md) explains how modules are ported, test by test. The [porting status](https://polvallverdu.github.io/pipedog/docs/contributing/status.md) lists what's done. Until then, see what is ported in [Transports](https://polvallverdu.github.io/pipedog/docs/transports.md). # WebSocket > WebSocket transports with FastAPIWebsocketTransport (bring your own socket), SingleClientWebsocketServerTransport, WebsocketClientTransport and RTVIClientTransport, plus endpoint helpers for each runtime. Source: https://polvallverdu.github.io/pipedog/docs/transports/websocket/ ## Overview [#overview] `@pipedog/transport-websocket` carries frames over a WebSocket, encoded by a [serializer](https://polvallverdu.github.io/pipedog/docs/serializers/protobuf.md). It ports Pipecat's four WebSocket transports, and adds the helpers that accept a socket on each runtime: * **`FastAPIWebsocketTransport`**: runs a pipeline on a socket your server already accepted. Upstream takes the socket a FastAPI endpoint accepted; here it takes a `WebsocketConnection` from any runtime. This is the one to use for a web server, a Worker or telephony. * **`SingleClientWebsocketServerTransport`**: starts its own WebSocket server and serves one client at a time. For local development and single-session bots. * **`WebsocketClientTransport`**: connects out to a WebSocket server. * **`RTVIClientTransport`**: a `WebsocketClientTransport` that speaks RTVI as a client to a bot. * **`WebsocketConnection`**: the runtime-agnostic socket all four transports talk to. * Endpoint helpers that produce a `WebsocketConnection`: `websocketFetchHandler` (fetch handlers, with `upgradeWebSocket` and `bunWebSocket`), `serveFetch` (serve a fetch handler with WebSocket upgrades on Bun, Node and Deno), `websocketServe` (listen on a port) and `honoWebsocket` (Hono). For a browser voice client, WebRTC handles network loss and jitter better. See [SmallWebRTC](https://polvallverdu.github.io/pipedog/docs/transports/smallwebrtc.md). WebSockets fit server-to-server links, telephony and runtimes without UDP, such as Cloudflare Workers. Pipecat's page for `FastAPIWebsocketTransport`. Pipecat's page for the WebSocket server transport. `fastapi.ts`, `server.ts`, `client.ts`, `rtvi_client.ts`, `socket.ts`, `fetch.ts`, `serve.ts` and `hono.ts`. Running a voice agent in a Durable Object over a WebSocket. ## Installation [#installation] ```bash bun add @pipedog/transport-websocket ``` To use `serveFetch`, `websocketServe` or `SingleClientWebsocketServerTransport` on Node, also install the `ws` package (an optional peer dependency): ```bash npm install ws ``` ```ts import { FastAPIWebsocketParams, FastAPIWebsocketTransport, } from "@pipedog/transport-websocket/fastapi"; import { SingleClientWebsocketServerParams, SingleClientWebsocketServerTransport, } from "@pipedog/transport-websocket/server"; import { WebsocketClientParams, WebsocketClientTransport } from "@pipedog/transport-websocket/client"; import { RTVIClientTransport } from "@pipedog/transport-websocket/rtvi_client"; import { WebsocketConnection } from "@pipedog/transport-websocket/socket"; import { bunWebSocket, upgradeWebSocket, websocketFetchHandler } from "@pipedog/transport-websocket/fetch"; import { serveFetch, websocketServe } from "@pipedog/transport-websocket/serve"; import { honoWebsocket } from "@pipedog/transport-websocket/hono"; ``` The package depends on `@pipedog/serializer-protobuf` and `@pipedog/serializer-rtvi-client`, the default serializers of the two client transports. ## Prerequisites [#prerequisites] * A serializer that matches the other end: `ProtobufFrameSerializer` for Pipecat's WebSocket clients, a telephony serializer such as [Telnyx](https://polvallverdu.github.io/pipedog/docs/serializers/telnyx.md) for phone calls. * For `allowedOrigins`, the list of origins allowed to connect, or the `PIPECAT_ALLOWED_ORIGINS` environment variable (comma-separated). ## FastAPIWebsocketTransport [#fastapiwebsockettransport] ### Configuration [#configuration] `new FastAPIWebsocketTransport(websocket, params, options?)`. `websocket` is any object that implements the `FastAPIWebSocket` interface; a `WebsocketConnection` does. `params` is a `FastAPIWebsocketParams`: | Option | Type | Required | Description | |---|---|---|---| | `audioOutDestinations` | `string[] \| undefined` | no | List of audio output destination identifiers. | | `audioOutEndSilenceSecs` | `number \| undefined` | no | How much silence (seconds) to send after an EndFrame (0 for no silence). | | `audioOutAutoSilence` | `boolean \| undefined` | no | Insert silence frames when the audio output queue is empty. When false, the transport will wait for audio data instead of inserting silence. | | `audioOutWriteTimeoutSecs` | `number \| undefined` | no | How long (seconds) a single write to the transport may take before the peer is considered gone. A client that stops reading leaves the write waiting with nothing to fail, so it would otherwise never return. | | `audioInEnabled` | `boolean \| undefined` | no | Enable audio input streaming. | | `audioInSampleRate` | `number \| null \| undefined` | no | Input audio sample rate in Hz. | | `audioInChannels` | `number \| undefined` | no | Number of input audio channels. | | `addWavHeader` | `boolean \| undefined` | no | Whether to add WAV headers to audio frames. | | `serializer` | `FrameSerializer \| null \| undefined` | no | Frame serializer for encoding/decoding messages. | | `sessionTimeout` | `number \| null \| undefined` | no | Session timeout in seconds, null for no timeout. | | `fixedAudioPacketSize` | `number \| null \| undefined` | no | Optional fixed-size packetization for raw PCM audio payloads. Useful when the remote WebSocket media endpoint requires strict audio framing. | | `allowedOrigins` | `string[] \| undefined` | no | List of allowed origins. Empty list allows all origins. When set, connections with a missing or disallowed Origin header are rejected. Defaults to `PIPECAT_ALLOWED_ORIGINS` env var (comma-separated). | | `wsCloseTimeout` | `number \| undefined` | no | Maximum time, in seconds, to wait for the WebSocket close handshake during disconnect. See `FastAPIWebsocketParams`. | | `audioOutEnabled` | `boolean \| undefined` | no | Enable audio output streaming. | | `audioOutSampleRate` | `number \| null \| undefined` | no | Output audio sample rate in Hz. | | `audioOutChannels` | `number \| undefined` | no | Number of output audio channels. | | `audioOutBitrate` | `number \| undefined` | no | Output audio bitrate in bits per second. | | `audioOut10msChunks` | `number \| undefined` | no | Number of 10ms chunks to buffer for output. | | `audioOutFilter` | `BaseAudioFilter \| null \| undefined` | no | Audio filter to apply to output audio of the default destination, before the mixer. Audio the transport generates itself (mixer audio, end silence, DTMF tones) is not filtered. | | `audioOutVolume` | `number \| undefined` | no | Volume of the output audio, as a multiplier on its samples, applied after the filter and before the mixer. A `VolumeFrame` changes it while the transport runs, and a `VolumeGainFrame` adjusts it for some audio. | | `audioOutMixer` | `AudioOutMixer \| null \| undefined` | no | Audio mixer instance or destination mapping. | | `audioInFilter` | `BaseAudioFilter \| null \| undefined` | no | Audio filter to apply to input audio. | | `audioInStreamOnStart` | `boolean \| undefined` | no | Start audio streaming immediately on transport start. | | `audioInPassthrough` | `boolean \| undefined` | no | Pass through input audio frames downstream. | | `videoInEnabled` | `boolean \| undefined` | no | Enable video input streaming. | | `videoInSources` | `Record \| undefined` | no | Video sources to capture from each user as they connect, keyed by source (`"camera"`, `"screenVideo"`, or a transport-specific custom source). Sources not listed here are captured only when the application asks for them, e.g. with the transport's `captureParticipantVideo()`. Requires `videoInEnabled`. Transports that don't support it log a warning and ignore it, so check the logs if video doesn't arrive as configured. | | `videoOutEnabled` | `boolean \| undefined` | no | Enable video output streaming. | | `videoOutIsLive` | `boolean \| undefined` | no | Enable real-time video output streaming. | | `videoOutWidth` | `number \| undefined` | no | Video output width in pixels. | | `videoOutHeight` | `number \| undefined` | no | Video output height in pixels. | | `videoOutFramerate` | `number \| undefined` | no | Video output frame rate in FPS. | | `videoOutColorFormat` | `string \| undefined` | no | Video output color format string. | | `videoOutCodec` | `string \| null \| undefined` | no | Preferred video codec for output (e.g., 'VP8', 'H264', 'H265'). | | `videoOutDestinations` | `string[] \| undefined` | no | List of video output destination identifiers. | `options`: | Option | Type | Required | Description | |---|---|---|---| | `inputName` | `string \| null \| undefined` | no | Optional name for the input processor. | | `outputName` | `string \| null \| undefined` | no | Optional name for the output processor. | ### Usage [#usage] Build the transport inside a function that takes the accepted connection. The endpoint helpers below call it once per connection: ```ts title="bot.ts" import { createTask } from "@pipedog/asyncio"; import { Pipeline } from "@pipedog/core/pipeline/pipeline"; import { PipelineWorker } from "@pipedog/core/pipeline/worker"; import { WorkerRunner } from "@pipedog/core/workers/runner"; import { ProtobufFrameSerializer } from "@pipedog/serializer-protobuf/protobuf"; import { FastAPIWebsocketParams, FastAPIWebsocketTransport, } from "@pipedog/transport-websocket/fastapi"; import { websocketFetchHandler } from "@pipedog/transport-websocket/fetch"; import { honoWebsocket } from "@pipedog/transport-websocket/hono"; import { serveFetch, websocketServe } from "@pipedog/transport-websocket/serve"; import { WebsocketConnection } from "@pipedog/transport-websocket/socket"; /** Runs one bot on an accepted WebSocket connection. */ async function runBot(connection: WebsocketConnection): Promise { const transport = new FastAPIWebsocketTransport( connection, new FastAPIWebsocketParams({ audioInEnabled: true, audioOutEnabled: true, serializer: new ProtobufFrameSerializer(), sessionTimeout: 300, }), ); const pipeline = new Pipeline([ transport.input(), // ... STT, LLM context aggregators, LLM, TTS transport.output(), ]); const worker = new PipelineWorker(pipeline); const runner = new WorkerRunner({ handleSigint: false }); transport.eventHandler("on_client_connected", async () => { console.log("client connected"); }); transport.eventHandler("on_client_disconnected", async () => { await runner.cancel(); }); await runner.addWorkers(worker); await runner.run(); } ``` `websocketFetchHandler` turns it into a fetch handler, `(request, server?) => Promise`. It answers 426 to a request that is not a WebSocket upgrade. Cloudflare Workers, Durable Objects and Deno upgrade from the request alone: ```ts // Cloudflare Workers (and Durable Objects) and Deno upgrade the request from fetch. const handler = websocketFetchHandler(runBot); const workerEntry = { fetch(request: Request): Promise { return handler(request); }, }; ``` Bun upgrades with its server, so pass the `server` Bun gives `fetch` and serve with the `bunWebSocket` handlers. `serveFetch` serves a fetch handler with WebSocket upgrades on Bun, Node (with the `ws` package) and Deno, so one entry point runs on all three. `upgradeWebSocket(request, { server })`, from the same module, is the upgrade on its own, for your own handlers: ```ts // Bun, Node (with the `ws` package) and Deno: serve the same fetch handler, with upgrades. const fetchServer = await serveFetch(handler, { host: "0.0.0.0", port: 8765 }); // Later: stop listening. await fetchServer.close(); ``` ```ts title="Bun.serve" Bun.serve({ fetch: (request, server) => handler(request, server), websocket: bunWebSocket, // from @pipedog/transport-websocket/fetch }); ``` `websocketServe` listens on a host and port with the runtime's own server, on Bun, Node and Deno, and takes the connection handler directly: ```ts // Bun, Node (with the `ws` package) and Deno: listen on a port. const server = await websocketServe({ handler: runBot, host: "0.0.0.0", port: 8765, origins: ["https://app.example.com"], }); // Later: stop listening, close the open connections and wait for their handlers. await server.close(); ``` With Hono, `honoWebsocket` returns the events that Hono's `upgradeWebSocket()` expects, on any Hono adapter. Pass the upgrade request's headers so `allowedOrigins` can be checked: ```ts // Pass the result to Hono's upgradeWebSocket() (hono/bun, hono/deno, // hono/cloudflare-workers or @hono/node-ws). const honoEvents = (request: Request) => honoWebsocket(runBot, { headers: request.headers }); ``` ```ts title="app.ts" app.get( "/ws", upgradeWebSocket((c) => honoEvents(c.req.raw)), ); ``` A full Hono app, and the same bot mounted in Express, Next.js, SvelteKit and other frameworks, is in [`examples/framework-snippets`](https://github.com/polvallverdu/pipedog/tree/main/examples/framework-snippets). See also [Frameworks](https://polvallverdu.github.io/pipedog/docs/frameworks.md). ### Event handlers [#event-handlers] | Event | Description | | ------------------------ | ----------------------------------------------------------------------------------- | | `on_client_connected` | The input transport started. Receives the socket. | | `on_client_disconnected` | The peer closed the socket (not when the transport closes it). Receives the socket. | | `on_session_timeout` | `sessionTimeout` seconds passed since the transport started. Receives the socket. | ### Notes [#notes] * The constructor throws if `allowedOrigins` is non-empty and the socket's `Origin` header is missing or not listed. The caller closes the socket in that case; `websocketFetchHandler`, `websocketServe` and `honoWebsocket` log the error and close it for you. * When the input starts it pushes a `ClientConnectedFrame`, then reads messages and deserializes them with `serializer`. Without a serializer, incoming messages are dropped and nothing is sent. * Output audio is paced to real time. `fixedAudioPacketSize` cuts raw PCM into fixed-size packets, for media endpoints that need strict framing. `addWavHeader` wraps each chunk in a WAV header. * `wsCloseTimeout` (default 0.5 seconds) bounds how long a disconnect waits for the peer to acknowledge the close, so a dead peer does not stall shutdown. ## SingleClientWebsocketServerTransport [#singleclientwebsocketservertransport] ### Configuration [#configuration-1] `new SingleClientWebsocketServerTransport(params, options?)`. `params` is a `SingleClientWebsocketServerParams`: | Option | Type | Required | Description | |---|---|---|---| | `audioOutEnabled` | `boolean \| undefined` | no | Enable audio output streaming. | | `audioOutSampleRate` | `number \| null \| undefined` | no | Output audio sample rate in Hz. | | `audioOutChannels` | `number \| undefined` | no | Number of output audio channels. | | `audioOutBitrate` | `number \| undefined` | no | Output audio bitrate in bits per second. | | `audioOut10msChunks` | `number \| undefined` | no | Number of 10ms chunks to buffer for output. | | `addWavHeader` | `boolean \| undefined` | no | Whether to add WAV headers to audio frames. | | `serializer` | `FrameSerializer \| null \| undefined` | no | Frame serializer for message encoding/decoding. | | `sessionTimeout` | `number \| null \| undefined` | no | Timeout in seconds for client sessions. | | `allowedOrigins` | `string[] \| undefined` | no | List of allowed origins. Empty list allows all origins. When set, connections with a missing or disallowed Origin header are rejected. Defaults to `PIPECAT_ALLOWED_ORIGINS` env var (comma-separated). | | `audioOutFilter` | `BaseAudioFilter \| null \| undefined` | no | Audio filter to apply to output audio of the default destination, before the mixer. Audio the transport generates itself (mixer audio, end silence, DTMF tones) is not filtered. | | `audioOutVolume` | `number \| undefined` | no | Volume of the output audio, as a multiplier on its samples, applied after the filter and before the mixer. A `VolumeFrame` changes it while the transport runs, and a `VolumeGainFrame` adjusts it for some audio. | | `audioOutMixer` | `AudioOutMixer \| null \| undefined` | no | Audio mixer instance or destination mapping. | | `audioOutDestinations` | `string[] \| undefined` | no | List of audio output destination identifiers. | | `audioOutEndSilenceSecs` | `number \| undefined` | no | How much silence (seconds) to send after an EndFrame (0 for no silence). | | `audioOutAutoSilence` | `boolean \| undefined` | no | Insert silence frames when the audio output queue is empty. When false, the transport will wait for audio data instead of inserting silence. | | `audioOutWriteTimeoutSecs` | `number \| undefined` | no | How long (seconds) a single write to the transport may take before the peer is considered gone. A client that stops reading leaves the write waiting with nothing to fail, so it would otherwise never return. | | `audioInEnabled` | `boolean \| undefined` | no | Enable audio input streaming. | | `audioInSampleRate` | `number \| null \| undefined` | no | Input audio sample rate in Hz. | | `audioInChannels` | `number \| undefined` | no | Number of input audio channels. | | `audioInFilter` | `BaseAudioFilter \| null \| undefined` | no | Audio filter to apply to input audio. | | `audioInStreamOnStart` | `boolean \| undefined` | no | Start audio streaming immediately on transport start. | | `audioInPassthrough` | `boolean \| undefined` | no | Pass through input audio frames downstream. | | `videoInEnabled` | `boolean \| undefined` | no | Enable video input streaming. | | `videoInSources` | `Record \| undefined` | no | Video sources to capture from each user as they connect, keyed by source (`"camera"`, `"screenVideo"`, or a transport-specific custom source). Sources not listed here are captured only when the application asks for them, e.g. with the transport's `captureParticipantVideo()`. Requires `videoInEnabled`. Transports that don't support it log a warning and ignore it, so check the logs if video doesn't arrive as configured. | | `videoOutEnabled` | `boolean \| undefined` | no | Enable video output streaming. | | `videoOutIsLive` | `boolean \| undefined` | no | Enable real-time video output streaming. | | `videoOutWidth` | `number \| undefined` | no | Video output width in pixels. | | `videoOutHeight` | `number \| undefined` | no | Video output height in pixels. | | `videoOutFramerate` | `number \| undefined` | no | Video output frame rate in FPS. | | `videoOutColorFormat` | `string \| undefined` | no | Video output color format string. | | `videoOutCodec` | `string \| null \| undefined` | no | Preferred video codec for output (e.g., 'VP8', 'H264', 'H265'). | | `videoOutDestinations` | `string[] \| undefined` | no | List of video output destination identifiers. | `options`: | Option | Type | Required | Description | |---|---|---|---| | `host` | `string \| undefined` | no | Host address to bind the server to. Defaults to "localhost". | | `port` | `number \| undefined` | no | Port number to bind the server to. Defaults to 8765. | | `inputName` | `string \| null \| undefined` | no | Optional name for the input processor. | | `outputName` | `string \| null \| undefined` | no | Optional name for the output processor. | | `serve` | `WebsocketServe \| null \| undefined` | no | The function that starts the server. Defaults to `websocketServe` (the runtime's own server). | ### Usage [#usage-1] ```ts title="server.ts" import { Pipeline } from "@pipedog/core/pipeline/pipeline"; import { PipelineWorker } from "@pipedog/core/pipeline/worker"; import { WorkerRunner } from "@pipedog/core/workers/runner"; import { ProtobufFrameSerializer } from "@pipedog/serializer-protobuf/protobuf"; import { SingleClientWebsocketServerParams, SingleClientWebsocketServerTransport, } from "@pipedog/transport-websocket/server"; const transport = new SingleClientWebsocketServerTransport( new SingleClientWebsocketServerParams({ audioInEnabled: true, audioOutEnabled: true, serializer: new ProtobufFrameSerializer(), }), { host: "localhost", port: 8765 }, ); transport.eventHandler("on_websocket_ready", async () => { console.log("listening on ws://localhost:8765"); }); transport.eventHandler("on_client_connected", async () => { console.log("client connected"); }); const pipeline = new Pipeline([ transport.input(), // ... STT, LLM context aggregators, LLM, TTS transport.output(), ]); const runner = new WorkerRunner(); await runner.addWorkers(new PipelineWorker(pipeline)); await runner.run(); ``` ### Event handlers [#event-handlers-1] | Event | Description | | ------------------------ | ----------------------------------------------------------------------------------------------- | | `on_websocket_ready` | The server is listening. | | `on_client_connected` | A client connected. Receives the `WebsocketConnection`. | | `on_client_disconnected` | The client disconnected. Receives the `WebsocketConnection`. | | `on_session_timeout` | `sessionTimeout` seconds passed since the client connected. Receives the `WebsocketConnection`. | ### Notes [#notes-1] * One client at a time. While a client is connected, a new connection is closed with code 1013 and the existing client is kept. * The server starts when the pipeline starts, with `options.serve` (default `websocketServe`). Pass your own `WebsocketServe` function to listen some other way. * A non-empty `allowedOrigins` is passed to the server as `origins`; other origins get HTTP 403. * The server closes only after both the input and the output have finished, so output still being flushed after the `EndFrame` is not cut off. * The deprecated `WebsocketServer*` aliases are not ported. Use the `SingleClientWebsocketServer*` names. ## WebsocketClientTransport [#websocketclienttransport] ### Configuration [#configuration-2] `new WebsocketClientTransport(uri, params?)`. `params` is a `WebsocketClientParams`: | Option | Type | Required | Description | |---|---|---|---| | `audioOutEnabled` | `boolean \| undefined` | no | Enable audio output streaming. | | `audioOutSampleRate` | `number \| null \| undefined` | no | Output audio sample rate in Hz. | | `addWavHeader` | `boolean \| undefined` | no | Whether to add WAV headers to audio frames. | | `additionalHeaders` | `Record \| null \| undefined` | no | Extra headers for the WebSocket handshake. | | `serializer` | `FrameSerializer \| null \| undefined` | no | Frame serializer for encoding/decoding messages. | | `audioOutChannels` | `number \| undefined` | no | Number of output audio channels. | | `audioOutBitrate` | `number \| undefined` | no | Output audio bitrate in bits per second. | | `audioOut10msChunks` | `number \| undefined` | no | Number of 10ms chunks to buffer for output. | | `audioOutFilter` | `BaseAudioFilter \| null \| undefined` | no | Audio filter to apply to output audio of the default destination, before the mixer. Audio the transport generates itself (mixer audio, end silence, DTMF tones) is not filtered. | | `audioOutVolume` | `number \| undefined` | no | Volume of the output audio, as a multiplier on its samples, applied after the filter and before the mixer. A `VolumeFrame` changes it while the transport runs, and a `VolumeGainFrame` adjusts it for some audio. | | `audioOutMixer` | `AudioOutMixer \| null \| undefined` | no | Audio mixer instance or destination mapping. | | `audioOutDestinations` | `string[] \| undefined` | no | List of audio output destination identifiers. | | `audioOutEndSilenceSecs` | `number \| undefined` | no | How much silence (seconds) to send after an EndFrame (0 for no silence). | | `audioOutAutoSilence` | `boolean \| undefined` | no | Insert silence frames when the audio output queue is empty. When false, the transport will wait for audio data instead of inserting silence. | | `audioOutWriteTimeoutSecs` | `number \| undefined` | no | How long (seconds) a single write to the transport may take before the peer is considered gone. A client that stops reading leaves the write waiting with nothing to fail, so it would otherwise never return. | | `audioInEnabled` | `boolean \| undefined` | no | Enable audio input streaming. | | `audioInSampleRate` | `number \| null \| undefined` | no | Input audio sample rate in Hz. | | `audioInChannels` | `number \| undefined` | no | Number of input audio channels. | | `audioInFilter` | `BaseAudioFilter \| null \| undefined` | no | Audio filter to apply to input audio. | | `audioInStreamOnStart` | `boolean \| undefined` | no | Start audio streaming immediately on transport start. | | `audioInPassthrough` | `boolean \| undefined` | no | Pass through input audio frames downstream. | | `videoInEnabled` | `boolean \| undefined` | no | Enable video input streaming. | | `videoInSources` | `Record \| undefined` | no | Video sources to capture from each user as they connect, keyed by source (`"camera"`, `"screenVideo"`, or a transport-specific custom source). Sources not listed here are captured only when the application asks for them, e.g. with the transport's `captureParticipantVideo()`. Requires `videoInEnabled`. Transports that don't support it log a warning and ignore it, so check the logs if video doesn't arrive as configured. | | `videoOutEnabled` | `boolean \| undefined` | no | Enable video output streaming. | | `videoOutIsLive` | `boolean \| undefined` | no | Enable real-time video output streaming. | | `videoOutWidth` | `number \| undefined` | no | Video output width in pixels. | | `videoOutHeight` | `number \| undefined` | no | Video output height in pixels. | | `videoOutFramerate` | `number \| undefined` | no | Video output frame rate in FPS. | | `videoOutColorFormat` | `string \| undefined` | no | Video output color format string. | | `videoOutCodec` | `string \| null \| undefined` | no | Preferred video codec for output (e.g., 'VP8', 'H264', 'H265'). | | `videoOutDestinations` | `string[] \| undefined` | no | List of video output destination identifiers. | ### Usage [#usage-2] ```ts title="client.ts" import { Pipeline } from "@pipedog/core/pipeline/pipeline"; import { WebsocketClientParams, WebsocketClientTransport, } from "@pipedog/transport-websocket/client"; import { RTVIClientTransport } from "@pipedog/transport-websocket/rtvi_client"; // The serializer defaults to ProtobufFrameSerializer. const client = new WebsocketClientTransport( "ws://localhost:8765", new WebsocketClientParams({ audioInEnabled: true, audioOutEnabled: true, additionalHeaders: { Authorization: `Bearer ${process.env.BOT_TOKEN ?? ""}` }, }), ); client.eventHandler("on_connected", async () => { console.log("connected to the bot"); }); const pipeline = new Pipeline([client.input(), client.output()]); ``` ### Event handlers [#event-handlers-2] | Event | Description | | ----------------- | ------------------------------------------------------------ | | `on_connected` | Connected to the server. Receives the `WebsocketConnection`. | | `on_disconnected` | The connection closed. Receives the `WebsocketConnection`. | ### Notes [#notes-2] * `serializer` defaults to `ProtobufFrameSerializer`. `addWavHeader` defaults to `true` here, unlike the server transports. * The connection opens with `websocketConnect`, which uses the global `WebSocket`, and on Cloudflare Workers opens the socket through `fetch` so `additionalHeaders` are sent there too. Browsers can't send handshake headers. ## RTVIClientTransport [#rtviclienttransport] `new RTVIClientTransport(uri, params?)`, with the same `WebsocketClientParams`. It connects to a bot that runs an RTVI server transport, such as an eval harness, and exchanges frames with it. ### Usage [#usage-3] ```ts title="rtvi.ts" // Talks RTVI to a bot: sends client-ready on connect and waits for bot-ready. const rtvi = new RTVIClientTransport("ws://localhost:7860"); rtvi.eventHandler("on_bot_ready", async () => { console.log("bot ready:", rtvi.botReady); }); ``` ### Event handlers [#event-handlers-3] | Event | Description | | ----------------- | ------------------------------------------------------------- | | `on_connected` | Connected to the bot. | | `on_disconnected` | The connection closed. | | `on_bot_ready` | The bot answered the `client-ready` message with `bot-ready`. | ### Notes [#notes-3] * `serializer` defaults to [`RTVIClientSerializer`](https://polvallverdu.github.io/pipedog/docs/serializers/rtvi-client.md). * `addWavHeader` is always set to `false`: an RTVI audio message carries bare PCM. * On connect it sends an RTVI `client-ready` message. `botReady` turns `true` once `bot-ready` arrives. ## WebsocketConnection [#websocketconnection] The socket the transports talk to. It exposes both APIs the upstream transports use: the `websockets` one (`state`, `recv()`, `send()`, `close()`, async iteration) and the Starlette one (`clientState`, `applicationState`, `receive()`, `sendBytes()`, `sendText()`, `headers`). Build one in either of two ways: * `WebsocketConnection.fromWebSocket(ws, options)` wraps a standard `WebSocket`: the global client, the server side of a Cloudflare `WebSocketPair`, Deno's upgraded socket or a Node `ws` socket. * `new WebsocketConnection(sink, options)` wraps anything else that can send and close. Report what arrives with `feedMessage()` and `feedClose()`. This is what `bunWebSocket` does for you; to wire Bun's `ServerWebSocket` by hand, inside your own `Bun.serve` handlers: ```ts title="bun.ts" // Bun's ServerWebSocket reports its events to Bun.serve, so feed them to a // WebsocketConnection built on a sink. type SocketData = { connection: WebsocketConnection | null }; Bun.serve({ port: 3000, fetch(request, bunServer) { if (bunServer.upgrade(request, { data: { connection: null } })) return undefined; return new Response("not found", { status: 404 }); }, websocket: { open(ws) { const connection = new WebsocketConnection({ send: (message) => ws.send(message), close: (code, reason) => ws.close(code, reason), }); ws.data.connection = connection; createTask(() => runBot(connection), "bun-socket"); }, message(ws, message) { ws.data.connection?.feedMessage(message); }, close(ws, code, reason) { ws.data.connection?.feedClose(code, reason); }, }, }); ``` Options for both: | Option | Type | Required | Description | |---|---|---|---| | `headers` | `HeadersInit \| null \| undefined` | no | The handshake request's headers (server side). | | `remoteAddress` | `string \| null \| undefined` | no | The peer's address, for logging. | | `path` | `string \| null \| undefined` | no | The request path (server side). | | `state` | `State \| undefined` | no | The initial state. Defaults to `State.OPEN`. | | `closeTimeout` | `number \| undefined` | no | Seconds `close()` waits for the closing handshake. Defaults to 10. | Options for `websocketServe`: | Option | Type | Required | Description | |---|---|---|---| | `handler` | `(websocket: WebsocketConnection) => Promise` | yes | Runs for every accepted connection. | | `host` | `string` | yes | The host to listen on. | | `port` | `number` | yes | The port to listen on. | | `origins` | `string[] \| null \| undefined` | no | Accepted `Origin` header values; `null` accepts any. A connection whose origin isn't listed is rejected with HTTP 403. | ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | ----- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | ✅ yes | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/transport-websocket/package.json`: ✅ means the compat smoke test passes on that runtime. | | Bun | Node | Deno | Cloudflare Workers | | ------------------------------------------------- | ------------------------------------------------ | ----------------------- | ----------------------------- | --------------------- | | `FastAPIWebsocketTransport` | yes | yes | yes | yes | | `websocketFetchHandler`, `upgradeWebSocket` | yes (pass the server, serve with `bunWebSocket`) | yes (with `serveFetch`) | yes (`Deno.upgradeWebSocket`) | yes (`WebSocketPair`) | | `serveFetch` | yes (`Bun.serve`) | yes (`ws` peer) | yes (`Deno.serve`) | no | | `websocketServe` | yes (`Bun.serve`) | yes (`ws` peer) | yes (`Deno.serve`) | no | | `honoWebsocket` | yes | yes (`@hono/node-ws`) | yes | yes | | `SingleClientWebsocketServerTransport` | yes | yes (`ws` peer) | yes | no | | `WebsocketClientTransport`, `RTVIClientTransport` | yes | yes | yes | yes | * `websocketFetchHandler` needs the upgrading server on Bun (the second argument of `fetch`) and on Node (from `serveFetch`); without one it throws, naming the fix. * `websocketServe` and `serveFetch` throw on Cloudflare Workers, which can't listen on a port. Closing the server closes the open connections with code 1001 and waits for their handlers. * Both fetch and Hono helpers copy the request headers during the upgrade: on Deno the request is closed once upgraded, and reading it then throws. * Each connection's handler runs in its own task, and the connection closes when the handler returns or throws. Not ported yet: the `WebsocketServer*` deprecated aliases (by design) and the eval harness's FastAPI transport (`pipecat.evals.fastapi_transport`). # WhatsApp > WhatsApp Business voice calls with WhatsAppClient, over SmallWebRTC. Handles Meta's calling webhooks, signature checks, and accepting, rejecting and ending calls. Source: https://polvallverdu.github.io/pipedog/docs/transports/whatsapp/ ## Overview [#overview] A user calls your WhatsApp Business number. Meta's Cloud API sends your webhook a `connect` event with an SDP offer. `WhatsAppClient` verifies the request, answers the offer over a [SmallWebRTC](https://polvallverdu.github.io/pipedog/docs/transports/smallwebrtc.md) connection (it **pre-accepts**, then **accepts** the call through the Graph API), and hands you the connection. You run a pipeline on it with `SmallWebRTCTransport`, exactly as for a browser client. This is a port of Pipecat's `pipecat.transports.whatsapp`. * **`WhatsAppClient`**: handles webhooks and calls, and keeps track of the ongoing calls. * **`WhatsAppApi`**: the Graph API calls: answer (`pre_accept`, `accept`), `reject` and `terminate`. * **`whatsAppWebhookHandler`**: a fetch handler for the webhook endpoint (`GET` verification and `POST` events). New in pipedog; upstream leaves this to your web framework. * **`parseWebhookRequest`**, **`verifyWebhookRequest`**, **`verifyWebhookSignature`**: the pieces on their own. Pipecat's documentation for the same transport. `client.ts`, `api.ts` and `http.ts`. The WhatsApp Business Calling API: webhooks and call actions. ## Installation [#installation] ```bash bun add @pipedog/transport-whatsapp @pipedog/transport-smallwebrtc ``` ```ts import { WhatsAppClient, whatsAppWebhookHandler } from "@pipedog/transport-whatsapp"; ``` ## Runtimes [#runtimes] | Runtime | Works | What it needs | | ------------------ | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Bun | ✅ yes | A WhatsApp Business account with Calling enabled, an access token and a public HTTPS webhook URL. A long-lived server that can use UDP (SmallWebRTC runs on werift). | | Node.js | ✅ yes | A WhatsApp Business account with Calling enabled, an access token and a public HTTPS webhook URL. A long-lived server that can use UDP (SmallWebRTC runs on werift). | | Deno | ✅ yes | A WhatsApp Business account with Calling enabled, an access token and a public HTTPS webhook URL. A long-lived server that can use UDP (SmallWebRTC runs on werift). | | Cloudflare Workers | ⚠️ partly | A WhatsApp Business account with Calling enabled, an access token and a public HTTPS webhook URL. Webhook verification, signature checks and API calls only. There is no WebRTC, so calls cannot be answered. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/transport-whatsapp/package.json`: ✅ means the compat smoke test passes on that runtime. Answering calls needs WebRTC, which runs on werift and needs UDP sockets: Bun, Node and Deno only. On **Cloudflare Workers** only the webhook parts work: Meta's `GET` verification, the `X-Hub-Signature-256` check (HMAC through WebCrypto) and the Graph API calls (`rejectCallToWhatsapp`, `terminateCallToWhatsapp`). A `connect` event there fails with an error that says Workers have no WebRTC. The package's `./connection` export resolves to a stub under the `"workerd"` condition, so werift is never bundled into a Worker. ## Prerequisites [#prerequisites] * A WhatsApp Business phone number with **Calling** enabled, and a webhook subscribed to the `calls` field. See [Meta's calling docs](https://developers.facebook.com/docs/whatsapp/cloud-api/calling/user-initiated-calls). * A system-user **access token** (`whatsappToken`) and the **phone number ID** (`phoneNumberId`). * Your Meta **app secret** (`whatsappSecret`), so every webhook `POST` can be authenticated. * A **verify token** you choose, entered in the Meta dashboard, for the webhook's `GET` check. * A public HTTPS URL for the webhook, and a server that can use UDP for the media. ## Usage [#usage] ```ts import { TransportParams } from "@pipedog/core/transports/base_transport"; import { SmallWebRTCTransport } from "@pipedog/transport-smallwebrtc/transport"; import { WhatsAppClient, whatsAppWebhookHandler } from "@pipedog/transport-whatsapp"; const client = new WhatsAppClient({ whatsappToken: process.env.WHATSAPP_TOKEN!, phoneNumberId: process.env.WHATSAPP_PHONE_NUMBER_ID!, whatsappSecret: process.env.WHATSAPP_APP_SECRET, // checks X-Hub-Signature-256 }); const webhook = whatsAppWebhookHandler(client, { verifyToken: process.env.WHATSAPP_VERIFY_TOKEN!, connectionCallback: async (connection, call) => { const transport = new SmallWebRTCTransport( connection, new TransportParams({ audioInEnabled: true, audioOutEnabled: true }), ); void runBot(transport, call.from); // your pipeline }, }); Bun.serve({ port: 8080, routes: { "/webhook": { GET: webhook, POST: webhook } } }); ``` The callback gets the `SmallWebRTCConnection` and the connect event, with the caller's number (`call.from`) and the call ID (`call.id`). If it throws, the call stays up and the error is logged. ### With the runner [#with-the-runner] `@pipedog/runner` wires all of this for you, as upstream's runner does. Install `@pipedog/transport-whatsapp`, set `WHATSAPP_APP_SECRET`, `WHATSAPP_PHONE_NUMBER_ID`, `WHATSAPP_TOKEN` and `WHATSAPP_WEBHOOK_VERIFICATION_TOKEN`, and run: ```bash bun bot.ts -t webrtc --whatsapp ``` It serves `GET /whatsapp` (Meta's verification) and `POST /whatsapp` (call events). Each call starts your `bot(runnerArgs)` with `SmallWebRTCRunnerArguments`: build the transport with `createTransport()` as for any WebRTC session, and read the caller from `runnerArgs.body` (`from`, `id`). See the [runner](https://polvallverdu.github.io/pipedog/docs/runner/#whatsapp). ### Without the helper [#without-the-helper] ```ts // GET: Meta checks the endpoint const challenge = await client.handleVerifyWebhookRequest(params, verifyToken); // the number to send back // POST: an event. Pass the raw body: the signature is over the exact bytes. const raw = await request.text(); await client.handleWebhookRequest(parseWebhookRequest(JSON.parse(raw)), { connectionCallback, rawBody: raw, sha256Signature: request.headers.get("x-hub-signature-256"), }); ``` With a `whatsappSecret`, a request without a matching signature is rejected. Without one nothing is checked, as upstream does. ## Configuration [#configuration] `new WhatsAppClient(options)`: | Option | Type | Default | Description | | ------------------ | --------------------------------------- | ----------------------------------- | ----------------------------------------------- | | `whatsappToken` | `string` | required | Graph API access token. | | `phoneNumberId` | `string` | required | The business phone number ID. | | `whatsappSecret` | `string \| null` | `null` | The app secret, to validate webhook signatures. | | `iceServers` | `string[] \| IceServer[]` | Google's STUN server | ICE servers for the calls' connections. | | `fetch` | `typeof fetch` | the global `fetch` | Replaces upstream's aiohttp session. | | `baseUrl` | `string` | `https://graph.facebook.com/v23.0/` | The Graph API URL. | | `createConnection` | `(iceServers) => SmallWebRTCConnection` | a `SmallWebRTCConnection` | Create the call's connection yourself. | Methods: `handleWebhookRequest`, `handleVerifyWebhookRequest`, `terminateAllCalls`, and the setters `updateIceServers`, `updateWhatsappSecret`, `updateWhatsappToken` and `updateWhatsappPhoneNumberId`. `terminateAllCalls()` ends every active call through the API and disconnects its connection. ### How a call is answered [#how-a-call-is-answered] 1. The `connect` event arrives. The client creates a `SmallWebRTCConnection` and gives it the offer. 2. It takes the SDP answer and removes every DTLS fingerprint but SHA-256, the only one WhatsApp accepts. 3. It sends `pre_accept`, then `accept`, with that answer. Meta allows 30 to 60 seconds from the webhook. If either fails, the connection is disconnected and the error is raised. 4. It calls your `connectionCallback`. The call ends on a `terminate` event, or when the connection closes. ## Differences from upstream [#differences-from-upstream] * **`fetch` replaces `aiohttp.ClientSession`.** Every request goes through the `fetch` option. * **The models are interfaces with the wire's names**, validated by `parseWebhookRequest`: `call.from` is upstream's `from_`, and the others stay `sdp_type`, `start_time` and so on. It also accepts the optional contact fields Meta's current docs describe. * **The signature check is asynchronous**, because HMAC uses WebCrypto. * The deprecated one-argument `connectionCallback(connection)` form is not ported. * `updateWhatsappPhoneNumberId` also updates the calls URL, where upstream only stores the value. # Voice bot over SmallWebRTC > A voice assistant you talk to in the browser, with OpenAI speech-to-text, chat and text-to-speech. Source: https://polvallverdu.github.io/pipedog/docs/voice-bot/ This guide runs a voice assistant end to end: you speak in the browser, OpenAI transcribes it and writes a reply, and the bot speaks the reply back. It is the same bot as [`examples/smallwebrtc-voice-bot`](https://github.com/polvallverdu/pipedog/tree/main/examples/smallwebrtc-voice-bot) in the repository, and its test runs it in headless Chromium. The browser side uses the unchanged Pipecat client SDK, `@pipecat-ai/client-js` with `@pipecat-ai/small-webrtc-transport`. ### Get the example [#get-the-example] Clone the repository and install the workspace: ```bash git clone https://github.com/polvallverdu/pipedog.git cd pipedog bun install ``` ### Run it [#run-it] Set an OpenAI API key and start the server: ```bash OPENAI_API_KEY=sk-... bun run --cwd examples/smallwebrtc-voice-bot start ``` Open [http://localhost:7860](http://localhost:7860), click **Connect**, allow the microphone, and talk. The bot introduces itself, then answers your question. To use Google Gemini instead of OpenAI, set `VOICE_PROVIDER=google` and `GOOGLE_API_KEY`. The pipeline is the same, with Gemini Live for transcription, Gemini for the reply and Gemini speech synthesis for the voice. ```bash VOICE_PROVIDER=google GOOGLE_API_KEY=... bun run --cwd examples/smallwebrtc-voice-bot start ``` ### Read the pipeline [#read-the-pipeline] The whole bot is one function, `runVoiceBot`, which builds a pipeline for each connection: ```ts const transport = new SmallWebRTCTransport( connection, new TransportParams({ audioInEnabled: true, audioOutEnabled: true }), ); const stt = new OpenAISTTService({ apiKey, baseUrl }); const llm = new OpenAILLMService({ apiKey, baseUrl, settings: new OpenAILLMSettings({ systemInstruction: SYSTEM_INSTRUCTION }), }); const tts = new OpenAITTSService({ apiKey, baseUrl, sampleRate: 24000 }); const context = new LLMContext(); const aggregators = new LLMContextAggregatorPair(context, { userParams: new LLMUserAggregatorParams({ vadAnalyzer: new SileroVADAnalyzer(), userTurnStrategies: new UserTurnStrategies({ stop: [new TurnAnalyzerUserTurnStopStrategy({ turnAnalyzer: createLocalSmartTurnAnalyzerV3() })], }), }), }); const pipeline = new Pipeline([ transport.input(), stt, aggregators.user(), llm, tts, transport.output(), aggregators.assistant(), ]); ``` Audio from the browser arrives at 16 kHz. Silero VAD detects when the user starts speaking, and Smart Turn decides when they have finished, so the bot does not answer mid-sentence. The transcript goes to the chat model, and the reply goes to speech synthesis at 24 kHz, the rate OpenAI's speech endpoint returns. ### Check it with a scripted user [#check-it-with-a-scripted-user] The example's test plays a recorded question into a headless browser's microphone and checks that the bot answers it. OpenAI is replaced by a mock, so the test needs no key: ```bash bun test examples/smallwebrtc-voice-bot ``` The test needs Chromium for Playwright (`bunx playwright-core install chromium` in the example's directory). ## What the bot does on connect [#what-the-bot-does-on-connect] When the browser connects, the bot adds a developer message asking the model to introduce itself, and queues an `LLMRunFrame`. That first reply is the greeting. From then on, each user turn runs through the pipeline above. ## Configure it [#configure-it] | Setting | Where | Default | | ------------------------ | ------------------------------- | --------------------------------- | | Provider | `VOICE_PROVIDER` | `openai` (or `google`) | | OpenAI API key | `OPENAI_API_KEY` | required for OpenAI | | Google API key | `GOOGLE_API_KEY` | required for Google | | OpenAI-compatible server | `OPENAI_BASE_URL` | `https://api.openai.com/v1` | | Port | `PORT` | `7860` | | Chat model and prompt | `OpenAILLMSettings` in `bot.ts` | the service default, see the code | | Voice | `OpenAITTSService` settings | `alloy` | ## Tracing and alerts [#tracing-and-alerts] Both are off until you set them: | Variable | What it does | | ----------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `OTEL_EXPORTER_OTLP_ENDPOINT` | An OTLP/HTTP traces endpoint, such as `http://localhost:4318/v1/traces`. Each conversation is traced: its turns, and the speech-to-text, LLM and speech synthesis calls in them. | | `ERROR_WEBHOOK_URL` | Each pipeline error is POSTed there as JSON, with the conversation id, the processor and the error. The body has a `text` field, so a Slack-style incoming webhook takes it as is. | Alerts are sent once per distinct error, and at most five per conversation. Warnings and errors in the logs have credentials masked, so an error that quotes a URL with `?key=...` doesn't leak the key. The wiring is in the example's `src/observability.ts`. ## Reconnection [#reconnection] The browser SDK reconnects by itself. When the connection fails, or stays disconnected for five seconds, it opens a new peer connection for the same session (`restart_pc`). The bot keeps the pipeline and the conversation through it, so the call picks up where it was. This is how Pipecat behaves too. A session ends when the browser closes the connection, or when reconnecting gives up. ## Next steps [#next-steps] * [Frames](https://polvallverdu.github.io/pipedog/docs/concepts/frames.md) and [frame processors](https://polvallverdu.github.io/pipedog/docs/concepts/frame-processors.md) explain the pipeline's building blocks. * [Runtimes & compatibility](https://polvallverdu.github.io/pipedog/docs/guides/runtimes.md) lists where SmallWebRTC runs: Bun, Node and Deno, not Cloudflare Workers, which have no UDP.