# Local audio

> The machine's microphone and speakers with LocalAudioTransport, through sox or ffmpeg, for developing a bot without a browser. No native addon.

Source: https://polvallverdu.github.io/pipedog/docs/transports/local-audio/



## Overview [#overview]

`LocalAudioTransport` records from the machine's microphone and plays the bot's voice on its speakers:
the quickest way to talk to a bot while you build it, with no browser, phone or media server. It is a
port of Pipecat's `LocalAudioTransport`. Upstream uses PyAudio (PortAudio, native). pipedog has **no
native addon**: it spawns [`sox`](https://sox.sourceforge.net/) or [`ffmpeg`](https://ffmpeg.org/)
and pipes raw 16-bit PCM through the child's stdin and stdout.

<Cards>
  <Card title="Upstream source" href="https://github.com/pipecat-ai/pipecat/blob/main/src/pipecat/transports/local/audio.py">
    `transports/local/audio.py` in Pipecat.
  </Card>

  <Card title="Source" href="https://github.com/polvallverdu/pipedog/tree/main/packages/transport-local/src">
    `audio.ts`, `audio_tools.ts` and `process.ts`.
  </Card>
</Cards>

## Installation [#installation]

```bash
bun add @pipedog/transport-local
```

and one of the tools:

|        | macOS                 | Debian / Ubuntu           | Windows                      |
| ------ | --------------------- | ------------------------- | ---------------------------- |
| sox    | `brew install sox`    | `sudo apt install sox`    | `choco install sox.portable` |
| ffmpeg | `brew install ffmpeg` | `sudo apt install ffmpeg` | `choco install ffmpeg`       |

With neither on the `PATH`, `setup()` throws `LocalAudioToolNotFoundError`, with the install commands
in its message. `detectAudioTool()` does the same lookup on its own.

```ts
import { LocalAudioTransport, LocalAudioTransportParams } from "@pipedog/transport-local";
```

## Runtimes [#runtimes]

| Runtime            | Works | What it needs                                                                                               |
| ------------------ | ----- | ----------------------------------------------------------------------------------------------------------- |
| Bun                | ✅ yes | sox or ffmpeg on the PATH, and a machine with audio devices.                                                |
| Node.js            | ✅ yes | sox or ffmpeg on the PATH, and a machine with audio devices.                                                |
| Deno               | ✅ yes | sox or ffmpeg on the PATH, and a machine with audio devices. Run with --allow-run.                          |
| Cloudflare Workers | ❌ no  | sox or ffmpeg on the PATH, and a machine with audio devices. Not supported: Workers cannot start processes. |

Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/transport-local/package.json`: ✅ means the compat smoke test passes on that runtime.

Processes are started with `Bun.spawn`, `Deno.Command` or `node:child_process`, whichever runtime
you are on. Cloudflare Workers can't start processes.

## Usage [#usage]

```ts
import { Pipeline } from "@pipedog/core/pipeline/pipeline";
import { LocalAudioTransport, LocalAudioTransportParams } from "@pipedog/transport-local";

const transport = new LocalAudioTransport(
  new LocalAudioTransportParams({ audioInEnabled: true, audioOutEnabled: true }),
);

const pipeline = new Pipeline([
  transport.input(),
  stt,
  userAggregator,
  llm,
  tts,
  transport.output(),
  assistantAggregator,
]);
```

Wear headphones: with speakers, the microphone hears the bot, and the bot interrupts itself.

## Configuration [#configuration]

`LocalAudioTransportParams` takes everything `TransportParams` does (sample rates, channels, VAD, filters,
mixers), plus:

| Option         | Type                          | Default           | Description                                       |
| -------------- | ----------------------------- | ----------------- | ------------------------------------------------- |
| `tool`         | `"sox" \| "ffmpeg" \| "auto"` | `"auto"`          | The tool to run. `"auto"` picks sox, then ffmpeg. |
| `inputDevice`  | `string \| number \| null`    | system default    | The microphone.                                   |
| `outputDevice` | `string \| number \| null`    | system default    | The speakers.                                     |
| `spawner`      | `ProcessSpawner`              | the runtime's own | How to start processes, for tests.                |

### Devices [#devices]

PortAudio's device indices (upstream's `input_device_index`) don't exist here. Name a device in the
tool's own terms:

* **sox**: the device name for the platform's driver: CoreAudio on macOS, ALSA on Linux, WaveAudio on
  Windows. The default is the system's default device (`sox -d`, which is what `rec` and `play` run).
* **ffmpeg**: on macOS an AVFoundation index or name for input and an AudioToolbox index for output;
  on Linux PulseAudio source and sink names (PipeWire provides them). On Windows input needs a
  DirectShow name, and ffmpeg has no playback device there, so use sox.

## How it works [#how-it-works]

* **Input.** A recorder process writes PCM to stdout. The transport cuts it into 20 ms
  `InputAudioRawFrame`s and pushes them downstream. If the recorder dies, it pushes an error frame with
  the tool's stderr.
* **Output.** Each `OutputAudioRawFrame` is written to the player's stdin, and the write then waits for
  the audio's duration, like the other non-blocking output transports. The player stays at most a chunk
  ahead of real time, so interruptions cut the speech quickly.
* Both processes are stopped at cleanup and on cancel.

## Differences from upstream [#differences-from-upstream]

* **sox or ffmpeg instead of PyAudio**, and `input_device_index` and `output_device_index` become
  `inputDevice` and `outputDevice`.
* The recorder and player start in `start()`, where upstream opens the streams in `setup()`.
* The desktop window transport, `transports/local/tk`, is not ported.
* Not exercised in the project's tests: a real microphone and speakers. The tests run against a fake
  process spawner, and the compat check against a real child process.