AWS Nova Sonic

OverviewCopy Anchor Link

AWSNovaSonicLLMService streams the user's audio to Amazon Nova Sonic on Bedrock (InvokeModelWithBidirectionalStream) and returns spoken audio, transcripts and tool calls, so one service replaces the STT, LLM and TTS stages of a pipeline.

• AWSNovaSonicLLMService: the session, with AWSNovaSonicLLMSettings and AudioConfig.
• AWSNovaSonicLLMAdapter: converts the LLMContext history and tools to Nova Sonic's events.
• SessionContinuationParams: hands the conversation to a new session before Nova Sonic's session time limit, buffering audio so nothing is lost.

InstallationCopy Anchor Link

bun add @pipedog/aws

import { AudioConfig, AWSNovaSonicLLMService } from "@pipedog/aws/nova_sonic/llm";

The default transport needs no AWS SDK. For transport: "http2", also install @aws-sdk/client-bedrock-runtime (an optional peer dependency).

PrerequisitesCopy Anchor Link

• Access to Nova Sonic in the Bedrock console, in a region that offers it.

• An IAM identity allowed bedrock:InvokeModelWithBidirectionalStream.

• Keys, in this order of precedence:

  1. accessKeyId, secretAccessKey and sessionToken.
  2. The AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN environment variables.
  3. A credentialsProvider function, called for every new session.

  The region comes from region, else AWS_REGION, else us-east-1.

AWSNovaSonicLLMServiceCopy Anchor Link

ConfigurationCopy Anchor Link

SettingsCopy Anchor Link

Runtime-updatable settings, passed as settings: new AWSNovaSonicLLMService.Settings({...}).

AudioConfigCopy Anchor Link

Defaults: 16 kHz 16-bit mono in, 24 kHz 16-bit mono out.

UsageCopy Anchor Link

import { AudioConfig, AWSNovaSonicLLMService } from "@pipedog/aws/nova_sonic/llm";
import { SessionContinuationParams } from "@pipedog/aws/nova_sonic/session_continuation";

const llm = new AWSNovaSonicLLMService({
  // Or leave these out to read the AWS_* environment variables.
  accessKeyId: process.env.AWS_ACCESS_KEY_ID,
  secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
  region: "us-east-1",
  settings: new AWSNovaSonicLLMService.Settings({
    voice: "tiffany",
    systemInstruction: "You are a friendly voice assistant. Keep answers short.",
    endpointingSensitivity: "MEDIUM",
  }),
  audioConfig: new AudioConfig({ inputSampleRate: 16000, outputSampleRate: 24000 }),
  // Hand the conversation to a fresh session before Nova Sonic's time limit.
  sessionContinuation: new SessionContinuationParams({ enabled: true }),
});

To use HTTP/2 through the AWS SDK, as upstream does:

// Upstream's transport: HTTP/2 through the AWS SDK (Node and Bun only).
// Needs `@aws-sdk/client-bedrock-runtime`.
const http2Llm = new AWSNovaSonicLLMService({
  region: "us-east-1",
  transport: "http2",
});

NotesCopy Anchor Link

• Nova Sonic detects turns itself. User and bot transcripts are pushed as transcription and LLMTextFrames, and audio as TTSAudioRawFrames.
• Tools from the context (or tools) are sent when the session starts. Tool calls run through the usual function-call flow, and results go back as toolResult events.
• With sessionContinuation, a new session is opened ahead of the time limit (transitionThresholdSeconds) and takes over at the next turn boundary, with the conversation history and buffered user audio.
• The JSON events on the wire are upstream's, character for character. Float settings use JS formatting, so 1.0 is sent as 1.

Runtime notesCopy Anchor Link

Generated from pipecat.runtimes and pipecat.compat in packages/aws/package.json: ✅ means the compat smoke test passes on that runtime.

• The default transport is Bedrock's WebSocket endpoint for InvokeModelWithBidirectionalStream, the one the AWS SDK for JavaScript uses in browsers. pipedog presigns the URL and signs each event with SigV4 on WebCrypto, so it runs on Bun, Node, Deno and Cloudflare Workers.
• transport: "http2" loads @aws-sdk/client-bedrock-runtime on connect and uses Node's http2, as upstream does: Node and Bun only.
• clientFactory replaces the stream client entirely (for tests or another transport), and endpointUrl changes the host.
• Upstream requires the keys; here they are optional, with the environment and credentialsProvider as fallbacks.
• Not ported: the deprecated Params class and the model, voiceId, params and systemInstruction constructor options. Use settings and audioConfig.