# AWS SageMaker (bidirectional streaming)

> SageMakerBidiClient, the HTTP/2 bidirectional streaming client for speech models hosted on Amazon SageMaker endpoints.

Source: https://polvallverdu.github.io/pipedog/docs/services/stt/aws-sagemaker/



## Overview [#overview]

`SageMakerBidiClient` streams data to and from a SageMaker real-time endpoint with
`InvokeEndpointWithBidirectionalStream`. Upstream, it is the transport under the SageMaker
variants of the Deepgram and NVIDIA speech services. Those services are not ported yet, but the
client is, so you can talk to a model on your own endpoint.

* `startSession()` opens the stream, retrying throttling and transient failures with jittered
  backoff (not 401 or 403).
* `sendAudioChunk()`, `sendText()`, `sendJson()` and `sendData()` send payload parts.
* `receiveResponse()` returns the next response event, or `null` once the stream has closed.
* `closeSession()` ends the input stream.
* `classifySagemakerBidiError()` sorts an error into an `ErrorCategory`, so a service can tell
  bad credentials from a busy endpoint.

<Cards>
  <Card title="Source" href="https://github.com/polvallverdu/pipedog/tree/main/packages/aws/src/sagemaker">
    `packages/aws/src/sagemaker/bidi_client.ts`.
  </Card>

  <Card title="SageMaker docs" href="https://docs.aws.amazon.com/sagemaker/latest/dg/realtime-endpoints-test-endpoints.html">
    Invoking real-time endpoints, including bidirectional streams.
  </Card>
</Cards>

## Installation [#installation]

```bash
bun add @pipedog/aws @aws-sdk/client-sagemaker-runtime-http2
```

```ts
import { SageMakerBidiClient } from "@pipedog/aws/sagemaker/bidi_client";
```

`@aws-sdk/client-sagemaker-runtime-http2` is an optional peer dependency, loaded on the first
`startSession()`. It is needed only by this client.

## Prerequisites [#prerequisites]

* A SageMaker endpoint serving a model that supports bidirectional streaming (for example
  Deepgram from the AWS Marketplace).
* An IAM identity allowed `sagemaker:InvokeEndpointWithBidirectionalStream`.
* Keys from `awsAccessKeyId`, `awsSecretAccessKey` and `awsSessionToken`, else the `AWS_*`
  environment variables, else a `credentialsProvider` function.

## SageMakerBidiClient [#sagemakerbidiclient]

### Configuration [#configuration]

| Option | Type | Required | Description |
|---|---|---|---|
| `endpointName` | `string` | yes | Name of the SageMaker endpoint to connect to. |
| `region` | `string` | yes | AWS region where the endpoint is deployed. |
| `modelInvocationPath` | `string \| null \| undefined` | no | API path for the model invocation (e.g., "v1/listen"). |
| `modelQueryString` | `string \| null \| undefined` | no | Query string parameters for the model (e.g., "model=nova-3"). |
| `maxConnectAttempts` | `number \| undefined` | no | How many times `startSession` attempts the session when SageMaker throttles it or fails transiently. The AWS SDK does not retry bidirectional streams itself. |
| `awsAccessKeyId` | `string \| null \| undefined` | no | AWS access key ID (pipedog addition; defaults to `AWS_ACCESS_KEY_ID`). |
| `awsSecretAccessKey` | `string \| null \| undefined` | no | AWS secret access key (pipedog addition; defaults to `AWS_SECRET_ACCESS_KEY`). |
| `awsSessionToken` | `string \| null \| undefined` | no | AWS session token (pipedog addition; defaults to `AWS_SESSION_TOKEN`). |
| `client` | `SageMakerRuntimeHTTP2ClientLike \| null \| undefined` | no | SDK client to use (pipedog addition; defaults to `SdkSageMakerRuntimeHTTP2Client`). |
| `credentialsProvider` | `AWSCredentialsProvider \| null \| undefined` | no | Keys to sign with when none are passed in or set in the environment. Called for every request or connection, so it can renew temporary credentials. |
| `fetch` | `FetchLike \| null \| undefined` | no | `fetch` used for HTTP calls (defaults to the global `fetch`). |
| `endpointUrl` | `string \| null \| undefined` | no | Override the service endpoint (e.g. a VPC endpoint or a local fake), `https://host[:port]`. |

### Usage [#usage]

```ts title="sagemaker.ts"
import { SageMakerBidiClient } from "@pipedog/aws/sagemaker/bidi_client";

const client = new SageMakerBidiClient({
  endpointName: "my-deepgram-endpoint",
  region: "us-east-2",
  modelInvocationPath: "v1/listen",
  modelQueryString: "model=nova-3&language=en",
});

async function transcribe(audio: Uint8Array): Promise<void> {
  await client.startSession();
  await client.sendAudioChunk(audio);
  await client.sendJson({ type: "CloseStream" });
  while (client.isActive) {
    const event = await client.receiveResponse();
    if (event === null) break;
    const bytes = event.PayloadPart?.Bytes;
    if (bytes) console.log(new TextDecoder().decode(bytes));
  }
  await client.closeSession();
}
```

### Notes [#notes]

* Requests go to `https://runtime.sagemaker.<region>.amazonaws.com:8443`, or `endpointUrl`.
* Events use the JS SDK's shapes: requests are `{ PayloadPart: { Bytes, DataType } }`, and
  `receiveResponse()` returns the SDK's `ResponseStreamEvent`.
* A failed start throws `SageMakerBidiSessionError`, with the AWS error ID, the HTTP status and
  the number of attempts.

## Runtime notes [#runtime-notes]

| Runtime            | Works | What it needs |
| ------------------ | ----- | ------------- |
| Bun                | ✅ yes | Nothing.      |
| Node.js            | ✅ yes | Nothing.      |
| Deno               | ✅ yes | Nothing.      |
| Cloudflare Workers | ✅ yes | Nothing.      |

Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/aws/package.json`: ✅ means the compat smoke test passes on that runtime.

* The API needs a full-duplex HTTP/2 stream, which `fetch` cannot do and Cloudflare Workers do
  not offer. The default client is the AWS SDK on Node's `http2`: it runs on Node and Bun. Deno is
  untested, and Workers are not supported.
* Importing the package never loads the SDK, so the rest of `@pipedog/aws` keeps working on
  Workers.
* Pass `client` (any object with `invokeEndpointWithBidirectionalStream(input)`) to use another
  transport, or a fake in tests.
* Not ported yet: the SageMaker services that use it (`DeepgramSageMakerSTTService`,
  `DeepgramSageMakerTTSService`, the Flux variants, `NvidiaSageMakerSTTService` and
  `NvidiaSageMakerTTSService`).