Telephony extensions

OverviewCopy Anchor Link

Two pieces of call logic that sit on top of any phone serializer, ported from Pipecat's pipecat.extensions:

• IVRNavigator is a pipeline segment around your LLM. It classifies the other side as a person or an automated menu, and in a menu it has the LLM press keys (<dtmf>1</dtmf>) or speak until it reaches a goal you set.
• VoicemailDetector tells a person from a voicemail greeting on an outbound call. Until it has decided, TTSGate holds the bot's speech, so a greeting is never talked over.

Both are plain frame processors with no I/O of their own, so they run on Bun, Node, Deno and Cloudflare Workers.

InstallationCopy Anchor Link

bun add @pipedog/extensions

import { IVRNavigator, VoicemailDetector } from "@pipedog/extensions";

RuntimesCopy Anchor Link

Generated from pipecat.runtimes and pipecat.compat in packages/extensions/package.json: ✅ means the compat smoke test passes on that runtime.

IVR navigationCopy Anchor Link

const ivrNavigator = new IVRNavigator({
  llm,
  // The goal. It is placed into the navigation instructions, so describe the
  // destination and anything the bot may say or enter along the way.
  ivrPrompt: "Reach the billing department and ask for a human agent.",
  // The IVR's prompts are read out slowly; wait longer before answering.
  ivrVadParams: new VADParams({ stopSecs: 2.0 }),
});

const ivrContext = new LLMContext();
const ivrAggregators = new LLMContextAggregatorPair(ivrContext);

const ivrPipeline = new Pipeline([
  transport.input(),
  stt,
  new DTMFAggregator(), // the IVR's own keypad tones reach the LLM as text
  ivrAggregators.user(),
  ivrNavigator, // wraps the LLM and turns <dtmf>, <ivr> and <mode> tags into frames
  tts,
  transport.output(),
  ivrAggregators.assistant(),
]);
const ivrWorker = new PipelineWorker(ivrPipeline);

// A person picked up: continue as a normal conversation.
ivrNavigator.addEventHandler("on_conversation_detected", async (_processor, history) => {
  await ivrWorker.queueFrame(
    new LLMMessagesUpdateFrame({
      messages: [{ role: "developer", content: "You are a helpful agent." }, ...history],
      runLlm: true,
    }),
  );
});

ivrNavigator.addEventHandler("on_ivr_status_changed", async (_processor, status) => {
  if (status === IVRStatus.COMPLETED) {
    // Transferred to a person: hand the call over to the conversation bot.
  } else if (status === IVRStatus.STUCK) {
    await ivrWorker.queueFrame(new EndWorkerFrame({ reason: "IVR navigation is stuck." }));
  }
});

How it works:

1. At StartFrame the IVRProcessor inside the navigator sets the classifier prompt on the LLM. The LLM answers <mode>ivr</mode> or <mode>conversation</mode> for what it hears.

2. <mode>conversation</mode> fires on_conversation_detected with the conversation history saved so far (the context without its first, system message).

3. <mode>ivr</mode> switches the context to the navigation prompt (your ivrPrompt inside the built-in instructions), with the history after it, runs the LLM, and switches the VAD to ivrVadParams (default stopSecs: 2.0) so the bot waits out the menu's pauses.

4. In navigation mode the LLM answers with tags that the processor strips from the text:

   Tag	Effect
   <dtmf>1</dtmf>	Pushes an OutputDTMFUrgentFrame for that key (0-9, *, #). Other values are logged and ignored. One tag per digit.
   <ivr>detected</ivr>	Back to navigation mode. Fires on_ivr_status_changed.
   <ivr>completed</ivr>	The goal is reached, usually a transfer. Fires on_ivr_status_changed.
   <ivr>stuck</ivr>	No way forward. Fires on_ivr_status_changed so you can end or retry.
   <ivr>wait</ivr>	The menu is incomplete; wait for more transcription.

   Everything else is pushed on as AggregatedTextFrames, which the TTS service speaks. The tags themselves are added to the context as TextFrames with skipTts, so the LLM sees what it did.

on_ivr_status_changed receives an IVRStatus ("detected", "completed", "stuck", "wait"). addEventHandler on the navigator registers the two events on its IVRProcessor.

With the phone serializersCopy Anchor Link

• Sending keys. OutputDTMFUrgentFrame reaches the output transport, which plays the key as a tone in the audio stream when the transport has no native DTMF (the WebSocket transport has none). The Twilio, Telnyx, Plivo, Exotel, Vonage and Genesys serializers then send that audio like any other, so the menu hears the keys. This is checked end to end for Twilio in the package's tests.
• Hearing keys. If the menu or a person presses keys, the serializers produce InputDTMFFrames. Put a DTMFAggregator (@pipedog/core/processors/aggregators/dtmf_aggregator) between the STT service and the user aggregator, as above, and sequences such as 12# arrive in the LLM context as DTMF: 12#.

OptionsCopy Anchor Link

Voicemail detectionCopy Anchor Link

const voicemail = new VoicemailDetector({
  classifier: new JevClassifier({ apiKey: process.env.TYPESAFE_API_KEY ?? null }),
  decisionTimeout: 1.0, // seconds of caller silence before the verdict acts
  voicemailResponseDelay: 2.0, // seconds of silence after a voicemail verdict
});

const context = new LLMContext();
const aggregators = new LLMContextAggregatorPair(context);

const pipeline = new Pipeline([
  transport.input(),
  stt,
  voicemail.detector(), // between the STT service and the user aggregator
  aggregators.user(),
  llm,
  tts,
  voicemail.gate(), // right after the TTS service
  transport.output(),
  aggregators.assistant(),
]);
const worker = new PipelineWorker(pipeline);

voicemail.addEventHandler("on_conversation_detected", async () => {
  console.log("A person answered.");
});

voicemail.addEventHandler("on_voicemail_detected", async (processor) => {
  await processor.pushFrame(
    new TTSSpeakFrame({
      text: "Hello, this is Alex calling about your appointment. Please call me back.",
    }),
  );
  // The message is left, so the call is over.
  await processor.pushFrame(new EndWorkerFrame());
});

The detector sits after the STT service. It passes every frame through, collects what the other side says, and after each transcription asks its classifier in the background whether the call reached a person or a voicemail. Once the caller has been quiet for decisionTimeout, the latest answer decides:

• Conversation. The gate releases the speech it held, in order, and on_conversation_detected fires.
• Voicemail. The held speech is dropped, the pipeline is interrupted, and nothing but lifecycle frames (system frames, EndFrame, StopFrame, worker frames) passes the detector from then on. on_voicemail_detected fires once the greeting has been quiet for voicemailResponseDelay seconds, and only once. Speak the message from the handler by pushing a TTSSpeakFrame through the processor it receives.

A classifier that fails or does not answer in time counts as no answer. With no answer at all after the silence, the detector assumes a conversation.

Any BaseClassifier works. JevClassifier is what upstream's example uses; an LLMClassifier needs a service that implements runInference(), so a realtime (speech-to-speech) LLM cannot back it. The classifier's on_metrics becomes a MetricsFrame in the pipeline.

OptionsCopy Anchor Link

Upstream's deprecated llm and customSystemPrompt parameters are not ported; pass a classifier.

Testing without a phoneCopy Anchor Link

Both classes are covered by ports of upstream's tests (test_ivr_navigation.py, test_voicemail_detector.py), which use runTest and a scripted classifier. The same approach works for your handlers: drive the processor with LLMTextFrames or TranscriptionFrames and read the frames and events back. Nothing here has been run against a live IVR or a real carrier's voicemail, so the prompts and classifier thresholds are upstream's, unverified by us.