# Azure OpenAI

> Image generation through Azure OpenAI's REST API, with AzureImageGenServiceREST.

Source: https://polvallverdu.github.io/pipedog/docs/services/image/azure/



## Overview [#overview]

* **`AzureImageGenServiceREST`**: turns each `TextFrame` it receives into a prompt for Azure
  OpenAI's asynchronous image generation API. It submits the job, polls it once a second until
  it succeeds, downloads the image and pushes it as a `URLImageRawFrame` of RGB pixels.

<Cards>
  <Card title="Upstream page" href="https://docs.pipecat.ai/api-reference/server/services/image-generation/azure">
    Pipecat's documentation for the same service.
  </Card>

  <Card title="Source" href="https://github.com/polvallverdu/pipedog/tree/main/packages/azure/src">
    `packages/azure/src/image.ts`.
  </Card>

  <Card title="Azure docs" href="https://learn.microsoft.com/en-us/azure/ai-foundry/openai/how-to/dall-e">
    Image generation on Azure OpenAI.
  </Card>
</Cards>

## Installation [#installation]

```bash
bun add @pipedog/azure
```

```ts
import { AzureImageGenServiceREST } from "@pipedog/azure/image";
```

## Prerequisites [#prerequisites]

* An Azure OpenAI resource whose region offers image generation, and its API key. The examples
  read it from `AZURE_OPENAI_API_KEY`.
* The resource endpoint, ending in `/` (the service appends `openai/images/...` to it).

## AzureImageGenServiceREST [#azureimagegenservicerest]

### Configuration [#configuration]

| Option | Type | Required | Description |
|---|---|---|---|
| `apiKey` | `string` | yes | Azure OpenAI API key for authentication. |
| `endpoint` | `string` | yes | Azure OpenAI endpoint URL, ending in `/`. |
| `apiVersion` | `string \| undefined` | no | Azure API version string. Defaults to "2023-06-01-preview". |
| `fetch` | `((input: RequestInfo \| URL, init?: RequestInit) => Promise<Response>) \| null \| undefined` | no | `fetch` used for the requests. Defaults to the global `fetch`. DIVERGENCE: replaces upstream's required `aiohttp_session`. |
| `decodeImage` | `ImageDecoder \| null \| undefined` | no | Decodes the generated image into RGB pixels. Defaults to `decodePngToRgb`. DIVERGENCE: pipedog-only. Upstream decodes with Pillow. |
| `settings` | `AzureImageGenSettings \| null \| undefined` | no | Runtime-updatable settings. |
| `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. |
| `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. |
| `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. |
| `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. |

### Settings [#settings]

Runtime-updatable settings, passed as `settings: new AzureImageGenServiceREST.Settings({...})`
(`AzureImageGenSettings`). Any setting you leave out keeps the default below.

| Setting     | Type             | Default | Description                                                       |
| ----------- | ---------------- | ------- | ----------------------------------------------------------------- |
| `model`     | `string \| null` | `null`  | Not sent: the endpoint decides the model.                         |
| `imageSize` | `string \| null` | `null`  | Sent as `size` (for example `"1024x1024"`). `null` leaves it out. |

### Usage [#usage]

```ts title="image.ts"
import { AzureImageGenServiceREST } from "@pipedog/azure/image";

const imageGen = new AzureImageGenServiceREST({
  apiKey: process.env.AZURE_OPENAI_API_KEY ?? "",
  // The resource endpoint, with its trailing slash.
  endpoint: "https://my-resource.openai.azure.com/",
  settings: new AzureImageGenServiceREST.Settings({ imageSize: "1024x1024" }),
});
```

### Notes [#notes]

* The request is `POST {endpoint}openai/images/generations:submit?api-version=...` with the key
  in the `api-key` header and `{ prompt, n: 1, size }` as the body. The job's status is read
  from the `operation-location` URL the submission returns.
* Polling gives up after 119 attempts, about two minutes, with an `ErrorFrame`
  (`"Image generation timed out"`). A finished job without an image URL gives
  `"Image generation failed"`.
* A submission without an `operation-location` header (a 429, for example) throws, as upstream.
* `apiVersion` defaults to `"2023-06-01-preview"`, the version this asynchronous API was
  published under.

## Runtime notes [#runtime-notes]

| Runtime            | Works | What it needs |
| ------------------ | ----- | ------------- |
| Bun                | ✅ yes | Nothing.      |
| Node.js            | ✅ yes | Nothing.      |
| Deno               | ✅ yes | Nothing.      |
| Cloudflare Workers | ✅ yes | Nothing.      |

Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/azure/package.json`: ✅ means the compat smoke test passes on that runtime.

* All requests go through `fetch` (the global one, or the `fetch` option), which replaces
  upstream's required `aiohttp_session`. The service works on Bun, Node, Deno and Cloudflare
  Workers.
* The image is decoded to RGB with `decodePngToRgb` from `@pipedog/openai/image`, a pure JS PNG
  decoder, and the frame's `format` is always `"RGB"`. Upstream decodes with Pillow and reports
  Pillow's mode. Pass `decodeImage` to decode other formats.
* The deprecated `imageSize` and `model` constructor options are not ported. Use `settings`.