# AWS Bedrock

> Foundation models on Amazon Bedrock (Claude, Nova, Llama and others) through AWSBedrockLLMService and the Converse API.

Source: https://polvallverdu.github.io/pipedog/docs/services/llm/aws/



## Overview [#overview]

`AWSBedrockLLMService` runs any Bedrock model that supports the Converse API. It streams text
with ConverseStream, calls functions, supports prompt caching and reasoning effort, and reports
token usage, including cache tokens.

* **`AWSBedrockLLMService`**: the LLM service, with `AWSBedrockLLMSettings` for runtime-updatable
  settings.
* **`AWSBedrockLLMAdapter`**: converts the universal `LLMContext` and tools to Converse messages,
  `system` blocks and `toolConfig`.

<Cards>
  <Card title="Upstream page" href="https://docs.pipecat.ai/api-reference/server/services/llm/aws">
    Pipecat's documentation for the same service.
  </Card>

  <Card title="Source" href="https://github.com/polvallverdu/pipedog/tree/main/packages/aws/src">
    `packages/aws/src/llm.ts` and `bedrock_adapter.ts`.
  </Card>

  <Card title="Bedrock docs" href="https://docs.aws.amazon.com/bedrock/latest/userguide/conversation-inference.html">
    The Converse API, models and prompt caching.
  </Card>
</Cards>

## Installation [#installation]

```bash
bun add @pipedog/aws
```

```ts
import { AWSBedrockLLMService } from "@pipedog/aws/llm";
```

No AWS SDK is needed: the service calls the Bedrock REST API with `fetch`, signed with SigV4.

## Prerequisites [#prerequisites]

* An AWS account with access to the model in the Bedrock console, in the region you use.
* An IAM identity allowed `bedrock:InvokeModel` and `bedrock:InvokeModelWithResponseStream`.
* Keys, in this order of precedence:

  1. `awsAccessKey`, `awsSecretKey` and `awsSessionToken`.
  2. The `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` and `AWS_SESSION_TOKEN` environment
     variables.
  3. A `credentialsProvider` function, called for every request.

  The region comes from `awsRegion`, else `AWS_REGION`, else `us-east-1`.

## AWSBedrockLLMService [#awsbedrockllmservice]

### Configuration [#configuration]

| Option | Type | Required | Description |
|---|---|---|---|
| `awsAccessKey` | `string \| null \| undefined` | no | AWS access key ID. If null, falls back to the `AWS_ACCESS_KEY_ID` environment variable. |
| `awsSecretKey` | `string \| null \| undefined` | no | AWS secret access key. Same fallback behaviour as `awsAccessKey`. |
| `awsSessionToken` | `string \| null \| undefined` | no | AWS session token for temporary credentials. |
| `awsRegion` | `string \| null \| undefined` | no | AWS region for the Bedrock service. |
| `settings` | `AWSBedrockLLMSettings \| null \| undefined` | no | Runtime-updatable settings for this service. Values given here override the service defaults. |
| `retryTimeoutSecs` | `number \| null \| undefined` | no | Request timeout in seconds for retry logic. Defaults to 5.0. |
| `retryOnTimeout` | `boolean \| null \| undefined` | no | Whether to retry the request once if it times out. Defaults to false. |
| `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. |
| `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. |
| `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. |
| `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. |
| `runInParallel` | `boolean \| undefined` | no | Whether to run function calls in parallel or sequentially. Defaults to true. |
| `groupParallelTools` | `boolean \| undefined` | no | Whether to group parallel function calls so the LLM is triggered exactly once after all calls in the batch complete. When false, each function call result triggers the LLM independently as it arrives. Defaults to true. |
| `functionCallTimeoutSecs` | `number \| null \| undefined` | no | Optional timeout in seconds for function calls. A call that runs past it is cancelled: its handler is cancelled (it gets a `CancelledError`), the call is settled as cancelled, and inference runs so the LLM can report that it didn't complete. |
| `fileResolver` | `FileResolver \| null \| undefined` | no | Resolves file URLs in the context that this provider can't fetch itself (see `FileResolver`). Pass one configured with a `fileStorage` to resolve uploaded-file URLs (`new FileResolver({ fileStorage: runnerArgs.fileStorage })` when a runner hosts the bot), and share one instance across services that may consume the same files (e.g. services switched between mid-session) so downloads are reused. Defaults to a resolver that handles `data:` and `http(s)` URLs only. |
| `credentialsProvider` | `AWSCredentialsProvider \| null \| undefined` | no | Keys to sign with when none are passed in or set in the environment. Called for every request or connection, so it can renew temporary credentials. |
| `fetch` | `FetchLike \| null \| undefined` | no | `fetch` used for HTTP calls (defaults to the global `fetch`). |
| `endpointUrl` | `string \| null \| undefined` | no | Override the service endpoint (e.g. a VPC endpoint or a local fake), `https://host[:port]`. |

### Settings [#settings]

Runtime-updatable settings, passed as `settings: new AWSBedrockLLMService.Settings({...})`. Any
setting you leave out keeps the default below. `null` means the field is not sent.

| Setting                        | Type                                | Default                      | Description                                                      |
| ------------------------------ | ----------------------------------- | ---------------------------- | ---------------------------------------------------------------- |
| `model`                        | `string`                            | `"us.amazon.nova-lite-v1:0"` | Bedrock model ID or inference profile.                           |
| `systemInstruction`            | `string \| null`                    | `null`                       | System prompt, sent as a `system` block.                         |
| `maxTokens`                    | `number \| null`                    | `null`                       | `inferenceConfig.maxTokens`.                                     |
| `temperature`                  | `number \| null`                    | `null`                       | `inferenceConfig.temperature`.                                   |
| `topP`                         | `number \| null`                    | `null`                       | `inferenceConfig.topP`.                                          |
| `stopSequences`                | `string[] \| null`                  | `null`                       | `inferenceConfig.stopSequences`.                                 |
| `latency`                      | `"standard" \| "optimized" \| null` | `null`                       | `performanceConfig.latency`.                                     |
| `effort`                       | `string \| null`                    | `null`                       | `outputConfig.effort` for reasoning models (`"low"` to `"max"`). |
| `enablePromptCaching`          | `boolean`                           | `false`                      | Add `cachePoint` markers to the system prompt and tools.         |
| `additionalModelRequestFields` | `Record<string, unknown>`           | `{}`                         | Model-specific fields, sent as `additionalModelRequestFields`.   |

### Usage [#usage]

```ts title="llm.ts"
import { AWSBedrockLLMService } from "@pipedog/aws/llm";
import { LLMUpdateSettingsFrame } from "@pipedog/core/frames/frames";

const llm = new AWSBedrockLLMService({
  // Or leave these out: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY,
  // AWS_SESSION_TOKEN and AWS_REGION are read from the environment.
  awsAccessKey: process.env.AWS_ACCESS_KEY_ID,
  awsSecretKey: process.env.AWS_SECRET_ACCESS_KEY,
  awsRegion: "us-east-1",
  settings: new AWSBedrockLLMService.Settings({
    model: "us.anthropic.claude-haiku-4-5-20251001-v1:0",
    systemInstruction: "You are a helpful voice assistant. Keep answers short.",
    maxTokens: 1024,
    enablePromptCaching: true,
  }),
});
```

With temporary credentials from a role, SSO or the instance metadata service, pass a provider.
`fromNodeProviderChain()` from `@aws-sdk/credential-providers` (Node only) caches and renews
them:

```ts
// Temporary credentials that renew themselves, e.g. from the AWS SDK's
// provider chain: the function is called for every request.
declare function fromNodeProviderChain(): () => Promise<{
  accessKeyId: string;
  secretAccessKey: string;
  sessionToken?: string;
}>;

const roleLlm = new AWSBedrockLLMService({
  awsRegion: "eu-west-1",
  credentialsProvider: fromNodeProviderChain(),
});
```

Change settings mid-call with an `LLMUpdateSettingsFrame`:

```ts
// Queue it on the worker, or push it from a processor upstream of the service.
const update = new LLMUpdateSettingsFrame({
  delta: new AWSBedrockLLMService.Settings({ temperature: 0.3, maxTokens: 512 }),
});
```

### Event handlers [#event-handlers]

Inherited from `LLMService`:

| Event                         | Description                                        |
| ----------------------------- | -------------------------------------------------- |
| `on_completion_timeout`       | A completion request timed out.                    |
| `on_function_calls_started`   | Function calls arrived and are about to run.       |
| `on_function_calls_cancelled` | One or more running function calls were cancelled. |

### Notes [#notes]

* Consumes `LLMContextFrame` and emits `LLMFullResponseStartFrame`, `LLMTextFrame`s and
  `LLMFullResponseEndFrame`.
* Reasoning the model returns in `reasoningContent` (Claude with extended thinking, Nova 2 with
  reasoning enabled) comes out as `LLMThoughtStartFrame`, `LLMThoughtTextFrame` and
  `LLMThoughtEndFrame`. Signed reasoning, such as Claude's, is kept in the context and sent back
  on later turns, so Claude keeps its reasoning across tool calls.
* Amazon Nova (v1) models write their reasoning into the answer, in `<thinking>...</thinking>`,
  before a tool call. A `<thinking>` block at the start of the response, or of any text between
  tool calls, is emitted as thought frames instead of `LLMTextFrame`s, so it isn't spoken.
* When the context has tool calls in its history but no tools are configured, the service sends
  a no-op tool, because Bedrock rejects tool messages without a `toolConfig`.
* For Claude models that reject assistant prefill, a trailing user message is added when the
  context ends with an assistant message.
* `runInference` uses Converse (not streaming). Converse has no reply schema, so a
  `responseSchema` is accepted but not enforced.
* With `retryOnTimeout: true`, the first attempt is limited to `retryTimeoutSecs` (5 by default).
  If it times out, the request is retried once with no timeout.
* Errors are `AWSServiceError`s with the AWS error code (`ThrottlingException`,
  `ValidationException`, …) and the HTTP status. An exception sent inside the event stream has
  status 0.

## Runtime notes [#runtime-notes]

| Runtime            | Works | What it needs |
| ------------------ | ----- | ------------- |
| Bun                | ✅ yes | Nothing.      |
| Node.js            | ✅ yes | Nothing.      |
| Deno               | ✅ yes | Nothing.      |
| Cloudflare Workers | ✅ yes | Nothing.      |

Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/aws/package.json`: ✅ means the compat smoke test passes on that runtime.

* Runs on Bun, Node, Deno and Cloudflare Workers: requests are signed with SigV4 on WebCrypto,
  and the `application/vnd.amazon.eventstream` response is decoded in the package.
* There is no default AWS credential chain (profiles, IMDS, ECS roles, SSO). Use the keys, the
  environment, or `credentialsProvider`.
* botocore's `client_config` is not ported, so there are no client-side retries of throttled or
  5xx requests and no 300 s timeouts. Pass your own `fetch` to add them.
* `endpointUrl` sends requests to another host (a VPC endpoint or a local fake); `fetch` replaces
  the global `fetch`.
* Not ported: the deprecated `AWSBedrockLLMService.InputParams` and the `model`, `params` and
  `stopSequences` constructor arguments. Use `settings`.