NVIDIA

OverviewCopy Anchor Link

NVIDIA NIM serves models such as Nemotron over an OpenAI-compatible API, on NVIDIA's cloud or on your own hardware. pipedog ports both of Pipecat's NVIDIA LLM services:

• NvidiaLLMService: an OpenAILLMService subclass with NIM's base URL and a default Nemotron model. It turns a model's reasoning into LLMThought*Frames, from a reasoning_content field or a leading <think> block, so TTS never reads it.
• NvidiaOmniLLMService: Nemotron Omni, which takes speech, images and video as well as text, and replies with text. By default it hears the user directly: it transcribes and answers each spoken turn in one request, in place of an STT service.

InstallationCopy Anchor Link

bun add @pipedog/nvidia

It depends on @pipedog/openai, which is installed with it.

import { NvidiaLLMService } from "@pipedog/nvidia/llm";
import { NvidiaOmniLLMService } from "@pipedog/nvidia/omni/llm";

PrerequisitesCopy Anchor Link

• For NVIDIA's cloud endpoint, an API key from build.nvidia.com, passed as apiKey. The examples read it from NVIDIA_API_KEY. The service logs a warning when it uses the cloud endpoint without a key.
• A local NIM deployment needs no key: pass its address as baseUrl.

NvidiaLLMServiceCopy Anchor Link

ConfigurationCopy Anchor Link

The other options (organization, project, defaultHeaders, serviceTier, retryOnTimeout, retryTimeoutSecs, function calling) are OpenAILLMService's. See OpenAI.

SettingsCopy Anchor Link

Runtime-updatable settings, passed as settings: new NvidiaLLMService.Settings({...}) (NvidiaLLMSettings). They are OpenAI's settings, with model defaulting to "nvidia/nemotron-3.5-super-vl-120b-a12b".

UsageCopy Anchor Link

import { SileroVADAnalyzer } from "@pipedog/audio-silero";
import { CartesiaTTSService } from "@pipedog/cartesia/tts";
import { LLMUpdateSettingsFrame } from "@pipedog/core/frames/frames";
import { Pipeline } from "@pipedog/core/pipeline/pipeline";
import { LLMContext } from "@pipedog/core/processors/aggregators/llm_context";
import {
  LLMContextAggregatorPair,
  LLMUserAggregatorParams,
  type UserTurnMessageAddedMessage,
} from "@pipedog/core/processors/aggregators/llm_response_universal";
import type { BaseTransport } from "@pipedog/core/transports/base_transport";
import { NvidiaLLMService } from "@pipedog/nvidia/llm";
import { NvidiaOmniLLMService } from "@pipedog/nvidia/omni/llm";

const llm = new NvidiaLLMService({
  apiKey: process.env.NVIDIA_API_KEY ?? "",
  settings: new NvidiaLLMService.Settings({
    model: "nvidia/nemotron-3.5-super-vl-120b-a12b",
    systemInstruction: "You are a helpful assistant in a voice conversation. Keep replies short.",
  }),
});

Point it at a NIM you run yourself:

// A NIM running on your own machine takes no API key.
const local = new NvidiaLLMService({ baseUrl: "http://localhost:8000/v1" });

Change settings mid-call with an LLMUpdateSettingsFrame. The next completion uses them:

// Queue it on the worker, or push it from a processor upstream of the service.
const update = new LLMUpdateSettingsFrame({
  delta: new NvidiaLLMService.Settings({ temperature: 0.4 }),
});

NotesCopy Anchor Link

• Runs a completion for each LLMContextFrame and streams the reply as LLMTextFrames, as OpenAILLMService does. Function calls, token usage metrics and runInference work the same way.
• Reasoning sent in the delta's reasoning_content (or reasoning) field, as Nemotron Nano models do, is pushed as LLMThoughtStartFrame, LLMThoughtTextFrame and LLMThoughtEndFrame.
• A reply that starts with a <think>...</think> block, as DeepSeek-R1 and some Nemotron models write, has the block pushed as thought frames and only the rest sent on as text.

NvidiaOmniLLMServiceCopy Anchor Link

ConfigurationCopy Anchor Link

SettingsCopy Anchor Link

Passed as settings: new NvidiaOmniLLMService.Settings({...}) (NvidiaOmniLLMSettings): the NvidiaLLMService settings, with these defaults and additions.

UsageCopy Anchor Link

In audio input mode (the default), Omni takes the place of the STT service. The user aggregator still needs a VAD analyzer, to mark the turns:

// Omni hears the user's speech itself, so there is no STT service.
const omni = new NvidiaOmniLLMService({
  apiKey: process.env.NVIDIA_API_KEY ?? "",
  settings: new NvidiaOmniLLMService.Settings({
    systemInstruction: "You are a helpful assistant in a voice conversation. Keep replies short.",
  }),
});
const tts = new CartesiaTTSService({ apiKey: process.env.CARTESIA_API_KEY ?? "" });

const context = new LLMContext();
const aggregators = new LLMContextAggregatorPair(context, {
  userParams: new LLMUserAggregatorParams({ vadAnalyzer: new SileroVADAnalyzer() }),
});
const userAggregator = aggregators.user();

const pipeline = new Pipeline([
  transport.input(),
  userAggregator,
  omni, // hears the user and answers
  tts,
  transport.output(),
  aggregators.assistant(),
]);

// What the user said is known once Omni has transcribed it.
userAggregator.eventHandler(
  "on_user_turn_message_added",
  async (_aggregator, message: UserTurnMessageAddedMessage) => {
    console.log("user said:", message.content);
  },
);

With audioInput: false, Omni is a text LLM for a cascaded pipeline, with an STT service in front of it. Media placed in the context is sent in either mode:

// With an STT service in the pipeline, Omni is a text LLM that also reads media in the context.
const textOmni = new NvidiaOmniLLMService({
  apiKey: process.env.NVIDIA_API_KEY ?? "",
  audioInput: false,
});

runMultimodalInference() runs one request outside the pipeline, for a worker that analyzes media. It returns the text, the reasoning and the finish reason, and can stream deltas to onTextDelta and onReasoningDelta:

// Analyze a video outside the pipeline, with a cap on the reasoning.
async function describeVideo(base64Mp4: string): Promise<string> {
  const media = new LLMContext({
    messages: [
      await LLMContext.createFileMessage({
        type: "bytes",
        format: "video/mp4",
        file: `data:video/mp4;base64,${base64Mp4}`,
        text: "Describe what happens in this clip.",
      }),
    ],
  });
  const result = await textOmni.runMultimodalInference(media, {
    reasoningBudget: 512,
    maxTokens: 1024,
  });
  return result.text;
}

NotesCopy Anchor Link

• In audio input mode the service announces itself as a realtime service, so the user aggregator works in realtime mode. It buffers the user's audio between turn boundaries and answers each spoken turn once it ends, after any turn still running.
• Omni writes what the user said in a <transcript> section. The service pushes it upstream as a TranscriptionFrame, which the user aggregator writes to the context, and sends only the <response> section on to TTS. Read the user's text from on_user_turn_message_added, not on_user_turn_stopped.
• The speech of a turn interrupted before Omni transcribed it is answered together with the user's next utterance.
• Context frames that need an answer (tool results, developer messages, typed user input) still start a completion, so the LLMRunFrame greeting works in both modes.
• Messages go through NvidiaOmniLLMAdapter (@pipedog/nvidia/nvidia_omni_adapter), which sends audio from LLMContext.createAudioMessage() and audio and video files from LLMContext.createFileMessage() as audio_url and video_url data URLs. "developer" messages are sent as "user" messages.
• pipedog init offers NvidiaLLMService, not the Omni service, as upstream's CLI does.

Runtime notesCopy Anchor Link

Generated from pipecat.runtimes and pipecat.compat in packages/nvidia/package.json: ✅ means the compat smoke test passes on that runtime.

• The client is the official openai JS SDK (through @pipedog/openai), which talks through fetch.
• The deprecated model constructor option is not ported. Use settings.
• Upstream keeps an Omni turn's audio in a Python ContextVar. pipedog keeps it per task instead, so tasks started from a turn don't inherit it. Only the turn's own request sends it, as upstream.
• Unit-tested with mocked clients (the ported upstream tests), not against NVIDIA itself.