# NVIDIA > NvidiaLLMService streams chat completions from NVIDIA NIM, and NvidiaOmniLLMService hears the user's speech with Nemotron Omni. Source: https://polvallverdu.github.io/pipedog/docs/services/llm/nvidia/ ## Overview [#overview] NVIDIA NIM serves models such as Nemotron over an OpenAI-compatible API, on NVIDIA's cloud or on your own hardware. pipedog ports both of Pipecat's NVIDIA LLM services: * **`NvidiaLLMService`**: an [`OpenAILLMService`](https://polvallverdu.github.io/pipedog/docs/services/llm/openai.md) subclass with NIM's base URL and a default Nemotron model. It turns a model's reasoning into `LLMThought*Frame`s, from a `reasoning_content` field or a leading `` block, so TTS never reads it. * **`NvidiaOmniLLMService`**: Nemotron Omni, which takes speech, images and video as well as text, and replies with text. By default it hears the user directly: it transcribes and answers each spoken turn in one request, in place of an STT service. Upstream's NVIDIA package also has Riva services: `NvidiaSTTService`, `NvidiaSegmentedSTTService` and `NvidiaTTSService`, and SageMaker variants of them. They aren't ported yet. See [NVIDIA Riva speech-to-text](https://polvallverdu.github.io/pipedog/docs/services/stt/nvidia.md) and [NVIDIA Riva text-to-speech](https://polvallverdu.github.io/pipedog/docs/services/tts/nvidia.md). Use another TTS service with Omni. Pipecat's documentation for the same service. `packages/nvidia/src/llm.ts`, `omni/llm.ts` and `nvidia_omni_adapter.ts`. Models and the API reference. The Omni model card: supported inputs and deployment options. ## Installation [#installation] ```bash bun add @pipedog/nvidia ``` It depends on `@pipedog/openai`, which is installed with it. ```ts import { NvidiaLLMService } from "@pipedog/nvidia/llm"; import { NvidiaOmniLLMService } from "@pipedog/nvidia/omni/llm"; ``` ## Prerequisites [#prerequisites] * For NVIDIA's cloud endpoint, an API key from [build.nvidia.com](https://build.nvidia.com/), passed as `apiKey`. The examples read it from `NVIDIA_API_KEY`. The service logs a warning when it uses the cloud endpoint without a key. * A local NIM deployment needs no key: pass its address as `baseUrl`. ## NvidiaLLMService [#nvidiallmservice] ### Configuration [#configuration] | Option | Type | Required | Description | |---|---|---|---| | `apiKey` | `string \| null \| undefined` | no | NVIDIA API key for authentication. Required when using the cloud endpoint. Not needed for local NIM deployments. | | `baseUrl` | `string \| null \| undefined` | no | The base URL for NIM API. Defaults to NVIDIA's cloud endpoint. For local deployments, pass the local address (e.g. `http://localhost:8000/v1`). | | `settings` | `NvidiaLLMSettings \| null \| undefined` | no | Runtime-updatable settings. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `runInParallel` | `boolean \| undefined` | no | Whether to run function calls in parallel or sequentially. Defaults to true. | | `groupParallelTools` | `boolean \| undefined` | no | Whether to group parallel function calls so the LLM is triggered exactly once after all calls in the batch complete. When false, each function call result triggers the LLM independently as it arrives. Defaults to true. | | `functionCallTimeoutSecs` | `number \| null \| undefined` | no | Optional timeout in seconds for function calls. A call that runs past it is cancelled: its handler is cancelled (it gets a `CancelledError`), the call is settled as cancelled, and inference runs so the LLM can report that it didn't complete. | | `fileResolver` | `FileResolver \| null \| undefined` | no | Resolves file URLs in the context that this provider can't fetch itself (see `FileResolver`). Pass one configured with a `fileStorage` to resolve uploaded-file URLs (`new FileResolver({ fileStorage: runnerArgs.fileStorage })` when a runner hosts the bot), and share one instance across services that may consume the same files (e.g. services switched between mid-session) so downloads are reused. Defaults to a resolver that handles `data:` and `http(s)` URLs only. | | `organization` | `string \| null \| undefined` | no | OpenAI organization ID. | | `project` | `string \| null \| undefined` | no | OpenAI project ID. | | `defaultHeaders` | `Record \| null \| undefined` | no | Additional HTTP headers. | | `serviceTier` | `string \| null \| undefined` | no | Service tier to use (e.g., "auto", "flex", "priority"). | | `retryTimeoutSecs` | `number \| null \| undefined` | no | How long an inference may go without producing output before it is abandoned and re-issued, when `retryOnTimeout` is set. Defaults to 5.0 seconds. | | `retryOnTimeout` | `boolean \| null \| undefined` | no | Whether to re-issue the request once if the first attempt produces no output within `retryTimeoutSecs`. The retry is unbounded. | The other options (`organization`, `project`, `defaultHeaders`, `serviceTier`, `retryOnTimeout`, `retryTimeoutSecs`, function calling) are `OpenAILLMService`'s. See [OpenAI](https://polvallverdu.github.io/pipedog/docs/services/llm/openai/#configuration). | | | | ------------- | ---------------------------------------- | | Base URL | `https://integrate.api.nvidia.com/v1` | | Default model | `nvidia/nemotron-3.5-super-vl-120b-a12b` | ### Settings [#settings] Runtime-updatable settings, passed as `settings: new NvidiaLLMService.Settings({...})` (`NvidiaLLMSettings`). They are [OpenAI's settings](https://polvallverdu.github.io/pipedog/docs/services/llm/openai/#settings), with `model` defaulting to `"nvidia/nemotron-3.5-super-vl-120b-a12b"`. ### Usage [#usage] ```ts title="llm.ts" import { SileroVADAnalyzer } from "@pipedog/audio-silero"; import { CartesiaTTSService } from "@pipedog/cartesia/tts"; import { LLMUpdateSettingsFrame } from "@pipedog/core/frames/frames"; import { Pipeline } from "@pipedog/core/pipeline/pipeline"; import { LLMContext } from "@pipedog/core/processors/aggregators/llm_context"; import { LLMContextAggregatorPair, LLMUserAggregatorParams, type UserTurnMessageAddedMessage, } from "@pipedog/core/processors/aggregators/llm_response_universal"; import type { BaseTransport } from "@pipedog/core/transports/base_transport"; import { NvidiaLLMService } from "@pipedog/nvidia/llm"; import { NvidiaOmniLLMService } from "@pipedog/nvidia/omni/llm"; const llm = new NvidiaLLMService({ apiKey: process.env.NVIDIA_API_KEY ?? "", settings: new NvidiaLLMService.Settings({ model: "nvidia/nemotron-3.5-super-vl-120b-a12b", systemInstruction: "You are a helpful assistant in a voice conversation. Keep replies short.", }), }); ``` Point it at a NIM you run yourself: ```ts // A NIM running on your own machine takes no API key. const local = new NvidiaLLMService({ baseUrl: "http://localhost:8000/v1" }); ``` Change settings mid-call with an `LLMUpdateSettingsFrame`. The next completion uses them: ```ts // Queue it on the worker, or push it from a processor upstream of the service. const update = new LLMUpdateSettingsFrame({ delta: new NvidiaLLMService.Settings({ temperature: 0.4 }), }); ``` ### Notes [#notes] * Runs a completion for each `LLMContextFrame` and streams the reply as `LLMTextFrame`s, as `OpenAILLMService` does. Function calls, token usage metrics and `runInference` work the same way. * Reasoning sent in the delta's `reasoning_content` (or `reasoning`) field, as Nemotron Nano models do, is pushed as `LLMThoughtStartFrame`, `LLMThoughtTextFrame` and `LLMThoughtEndFrame`. * A reply that starts with a `...` block, as DeepSeek-R1 and some Nemotron models write, has the block pushed as thought frames and only the rest sent on as text. ## NvidiaOmniLLMService [#nvidiaomnillmservice] ### Configuration [#configuration-1] | Option | Type | Required | Description | |---|---|---|---| | `enableDirectMode` | `boolean \| undefined` | no | Process frames immediately instead of using internal queues. | | `metrics` | `FrameProcessorMetrics \| null \| undefined` | no | Optional metrics collector for this processor. | | `retryOnTimeout` | `boolean \| null \| undefined` | no | Whether to re-issue the request once if the first attempt produces no output within `retryTimeoutSecs`. The retry is unbounded. | | `apiKey` | `string \| null \| undefined` | no | NVIDIA API key. Required for the cloud endpoint, not for local deployments. | | `baseUrl` | `string \| null \| undefined` | no | OpenAI-compatible endpoint base URL. For local deployments, pass the local address (e.g. `http://localhost:8000/v1`). | | `audioInput` | `boolean \| undefined` | no | Whether Omni hears the user directly, with no STT service in the pipeline. `false` makes it a text LLM for a cascaded pipeline. Defaults to `true`. | | `context` | `LLMContext \| null \| undefined` | no | Context that audio turns are answered against until the first `LLMContextFrame` supplies one. | | `settings` | `NvidiaOmniLLMSettings \| null \| undefined` | no | Runtime-updatable settings. | | `requestTimeoutSecs` | `number \| undefined` | no | HTTP request timeout, in seconds. Defaults to 120. | | `name` | `string \| null \| undefined` | no | Custom name; defaults to `ClassName#count`. | | `taskManager` | `BaseTaskManager \| null \| undefined` | no | Task manager for handling tasks. | | `runInParallel` | `boolean \| undefined` | no | Whether to run function calls in parallel or sequentially. Defaults to true. | | `groupParallelTools` | `boolean \| undefined` | no | Whether to group parallel function calls so the LLM is triggered exactly once after all calls in the batch complete. When false, each function call result triggers the LLM independently as it arrives. Defaults to true. | | `functionCallTimeoutSecs` | `number \| null \| undefined` | no | Optional timeout in seconds for function calls. A call that runs past it is cancelled: its handler is cancelled (it gets a `CancelledError`), the call is settled as cancelled, and inference runs so the LLM can report that it didn't complete. | | `fileResolver` | `FileResolver \| null \| undefined` | no | Resolves file URLs in the context that this provider can't fetch itself (see `FileResolver`). Pass one configured with a `fileStorage` to resolve uploaded-file URLs (`new FileResolver({ fileStorage: runnerArgs.fileStorage })` when a runner hosts the bot), and share one instance across services that may consume the same files (e.g. services switched between mid-session) so downloads are reused. Defaults to a resolver that handles `data:` and `http(s)` URLs only. | | `organization` | `string \| null \| undefined` | no | OpenAI organization ID. | | `project` | `string \| null \| undefined` | no | OpenAI project ID. | | `defaultHeaders` | `Record \| null \| undefined` | no | Additional HTTP headers. | | `serviceTier` | `string \| null \| undefined` | no | Service tier to use (e.g., "auto", "flex", "priority"). | | `retryTimeoutSecs` | `number \| null \| undefined` | no | How long an inference may go without producing output before it is abandoned and re-issued, when `retryOnTimeout` is set. Defaults to 5.0 seconds. | | | | | --------------- | ----------------------------------------------- | | Base URL | `https://integrate.api.nvidia.com/v1` | | Default model | `nvidia/nemotron-3-nano-omni-30b-a3b-reasoning` | | Request timeout | 120 seconds (`requestTimeoutSecs`) | ### Settings [#settings-1] Passed as `settings: new NvidiaOmniLLMService.Settings({...})` (`NvidiaOmniLLMSettings`): the `NvidiaLLMService` settings, with these defaults and additions. | Setting | Type | Default | Description | | -------------------------- | ---------------- | ------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `model` | `string` | `"nvidia/nemotron-3-nano-omni-30b-a3b-reasoning"` | Model name. | | `temperature` | `number` | `0.6` | Sampling temperature. | | `topP` | `number` | `0.95` | Nucleus sampling. | | `audioResponseInstruction` | `string \| null` | `null` | Instruction sent with the user's speech in audio input mode. `null` uses `DEFAULT_AUDIO_RESPONSE_INSTRUCTION`, which asks for a `` and a `` section. A custom one should keep asking for both. | | `minUserAudioSecs` | `number` | `0.3` | Shortest buffered utterance that starts a turn. | | `preSpeechBufferSecs` | `number` | `0.2` | Audio kept from before the user started speaking, so the start of an utterance isn't clipped. | ### Usage [#usage-1] In audio input mode (the default), Omni takes the place of the STT service. The user aggregator still needs a VAD analyzer, to mark the turns: ```ts title="omni.ts" // Omni hears the user's speech itself, so there is no STT service. const omni = new NvidiaOmniLLMService({ apiKey: process.env.NVIDIA_API_KEY ?? "", settings: new NvidiaOmniLLMService.Settings({ systemInstruction: "You are a helpful assistant in a voice conversation. Keep replies short.", }), }); const tts = new CartesiaTTSService({ apiKey: process.env.CARTESIA_API_KEY ?? "" }); const context = new LLMContext(); const aggregators = new LLMContextAggregatorPair(context, { userParams: new LLMUserAggregatorParams({ vadAnalyzer: new SileroVADAnalyzer() }), }); const userAggregator = aggregators.user(); const pipeline = new Pipeline([ transport.input(), userAggregator, omni, // hears the user and answers tts, transport.output(), aggregators.assistant(), ]); // What the user said is known once Omni has transcribed it. userAggregator.eventHandler( "on_user_turn_message_added", async (_aggregator, message: UserTurnMessageAddedMessage) => { console.log("user said:", message.content); }, ); ``` With `audioInput: false`, Omni is a text LLM for a cascaded pipeline, with an STT service in front of it. Media placed in the context is sent in either mode: ```ts // With an STT service in the pipeline, Omni is a text LLM that also reads media in the context. const textOmni = new NvidiaOmniLLMService({ apiKey: process.env.NVIDIA_API_KEY ?? "", audioInput: false, }); ``` `runMultimodalInference()` runs one request outside the pipeline, for a worker that analyzes media. It returns the text, the reasoning and the finish reason, and can stream deltas to `onTextDelta` and `onReasoningDelta`: ```ts // Analyze a video outside the pipeline, with a cap on the reasoning. async function describeVideo(base64Mp4: string): Promise { const media = new LLMContext({ messages: [ await LLMContext.createFileMessage({ type: "bytes", format: "video/mp4", file: `data:video/mp4;base64,${base64Mp4}`, text: "Describe what happens in this clip.", }), ], }); const result = await textOmni.runMultimodalInference(media, { reasoningBudget: 512, maxTokens: 1024, }); return result.text; } ``` ### Notes [#notes-1] * In audio input mode the service announces itself as a realtime service, so the user aggregator works in realtime mode. It buffers the user's audio between turn boundaries and answers each spoken turn once it ends, after any turn still running. * Omni writes what the user said in a `` section. The service pushes it upstream as a `TranscriptionFrame`, which the user aggregator writes to the context, and sends only the `` section on to TTS. Read the user's text from `on_user_turn_message_added`, not `on_user_turn_stopped`. * The speech of a turn interrupted before Omni transcribed it is answered together with the user's next utterance. * Context frames that need an answer (tool results, developer messages, typed user input) still start a completion, so the `LLMRunFrame` greeting works in both modes. * Messages go through `NvidiaOmniLLMAdapter` (`@pipedog/nvidia/nvidia_omni_adapter`), which sends audio from `LLMContext.createAudioMessage()` and audio and video files from `LLMContext.createFileMessage()` as `audio_url` and `video_url` data URLs. "developer" messages are sent as "user" messages. * `pipedog init` offers `NvidiaLLMService`, not the Omni service, as upstream's CLI does. ## Runtime notes [#runtime-notes] | Runtime | Works | What it needs | | ------------------ | -------------- | ------------- | | Bun | ✅ yes | Nothing. | | Node.js | ✅ yes | Nothing. | | Deno | — not verified | Nothing. | | Cloudflare Workers | ✅ yes | Nothing. | Generated from `pipecat.runtimes` and `pipecat.compat` in `packages/nvidia/package.json`: ✅ means the compat smoke test passes on that runtime. * The client is the official `openai` JS SDK (through `@pipedog/openai`), which talks through `fetch`. * The deprecated `model` constructor option is not ported. Use `settings`. * Upstream keeps an Omni turn's audio in a Python `ContextVar`. pipedog keeps it per task instead, so tasks started from a turn don't inherit it. Only the turn's own request sends it, as upstream. * Unit-tested with mocked clients (the ported upstream tests), not against NVIDIA itself.