---
title: "Technologies — how a Dialogist AI voice agent answers"
description: "How a Dialogist AI voice agent answers phone calls in about a quarter of a second, keeps listening while it talks and finds answers in your documents."
canonical: "https://dialogist.cc/en/technology/"
locale: "en"
updatedAt: "2026-11-11"
---

# What is inside a Dialogist AI voice agent

Measured latency, turn-taking, knowledge and testing.

- **≈ 0.27 s** the agent’s own time, from the caller’s last word to its first sound
- **94%** of an agent’s first phrases come from the voice cache
- **176%** of rated capacity: the highest load tested, every call connected
- **48** languages the agent understands and speaks

## A quarter of a second

From the caller’s last word to the agent’s first sound, the agent itself needs about a quarter of a second.

**≈ 0.27 s** 232 ms to decide the caller has finished, then 35 ms to the first sound.

- **What we time.** The clock runs from the end of the caller’s speech to the first sound of the agent’s reply.
- **Deciding the caller has finished.** A silence timer is either slow or cuts people off. The agent waits 192 ms, then asks a small model whether the sentence is complete. The model answers in about 40 ms, so the turn closes at about 232 ms. A silence-only timer needs 512 ms.
- **Starting the reply.** Recognition has run since the caller began to speak, so the text is ready when the turn closes. With early start the language model gets the text while the agent is still inside the pause, and its first sentence is usually written by the time the turn closes. Synthesis then starts from the voice cache or from that first sentence.

### What is left after the decision

| Step | Standard | Early start |
|---|---:|---:|
| Transcription (final text after the pause) | 0.5 ms | 0.5 ms |
| First sentence from the language model | 111 ms | 3 ms |
| Voice, to the first sound | 6 ms | 31 ms |
| **Decision to first sound** | **117 ms** | **35 ms** |

Transcription takes 0.5 ms after the decision because recognition streams while the caller speaks and makes a trial pass during the pause. With early start the language model begins on that trial text inside the pause, so 3 ms of its work remain after the decision; in standard mode all 111 ms come after it. In the load test the early request left before the decision in 98% of replies, on average 225 ms ahead at 50 conversations and 508 ms ahead at 128. If the caller keeps talking, the agent drops the prepared reply and answers the whole phrase.

At 100 simultaneous calls on a Server Unit the same step takes about 0.4 s in standard mode and 0.25 s with early start. The Server Unit note has the full picture. [Server Unit under load](https://dialogist.cc/en/research/server-unit-under-load/)

- **232 ms** to decide the caller has finished (192 ms pause plus 40 ms model); a silence timer needs 512 ms
- **35 ms** from that decision to the first sound with early start; 117 ms without
- **≈ 0.3 s** given back per caller turn on real calls by the end-of-turn model, which closed 87% of turns early

Across 1,468 caller turns on real calls the end-of-turn model closed 87% of turns early, 95% of them on its first look.

## Where the milliseconds went

We measured each change on its own before it went in. Together they roughly doubled the number of calls a Server Unit carries at Standard grade.

- **Early start (89–235 ms saved).** The language model starts on the recognized text while the agent is still waiting out the pause. The saving grows with load: 89 ms with one call, 189 ms at 96 conversations, 235 ms at 128.
- **Dead air removed from speech (280 → 40 ms).** Synthesized sentences begin with silence; the median in our cache is 280 ms. The agent trims it to 40 ms as it plays the clip, and on the line a single call’s reply came about 200 ms sooner. The cache keeps the original.
- **End-of-turn model (≈ 0.3 s saved).** After a 192 ms pause a small model judges from intonation whether the thought is finished. On real calls it closed 87% of caller turns early and waited out 96% of mid-thought pauses. It can run in observation mode first and only log what it would have done.
- **Voice cache (94%).** The agent plays phrases it has said before from disk. The cache survives restarts, and out of hours the agent re-checks cached clips by recognizing them and regenerates any that fail.
- **First sentence jumps the queue.** Under load, the opening of one call’s reply never waits behind the tail of another’s. The agent synthesizes the next sentences while the first one plays.
- **One clock for all calls.** Media for every call runs on a single shared 20 ms tick, in place of a timer per call.
- **Cheaper audio frames (69 → 0.7 µs).** Handing a frame between the telephony stack and the agent takes under a microsecond instead of tens.

Before and after: the same stand, a 20-minute single call and 50 conversations, recruiting agent. Cache: 64,192 phrases in the load test. Frame handoff: timed in isolation on the telephony library.

## It keeps listening while it talks

The caller’s channel stays open. Voice detection and recognition run on it while the agent speaks, and the agent acts on what it hears.

- **Interruptions that mean something.** A caller can talk over the agent, and the agent stops. It detects an interruption from text or from sound. The acoustic mode has its own threshold and needs unbroken speech, so a keyboard, a slammed door or a cough does not cut the agent off. The agent ignores a short, low-confidence fragment over its own voice, which is what echo looks like.
- **The voice steps back.** When the caller starts talking, the agent’s voice dips and dulls: lower volume, fewer high frequencies. If the caller says something that counts, the agent stops. After an “uh-huh” or a cough it comes back, from the nearest syllable boundary, the start of the sentence or the exact point. Set per agent.
- **“Uh-huh” in the pauses.** Inside a long caller turn, the agent drops in short acknowledgements in its own voice. They are pre-rendered clips that a person has listened to and approved. They play in pauses, with a probability, a minimum gap, a cap per turn and no repeats, and never when the end-of-turn model thinks the caller has finished.
- **It says when it is not sure.** Recognition confidence goes into the agent’s prompt. The agent sees a shaky transcription marked as such and asks the caller to repeat instead of guessing a name or a number.
- **Silence handled politely.** A re-prompt, then a goodbye, timed from the true end of playback rather than from the end of synthesis.
- **A room around the voice.** Optional office or meeting-room background, mixed under the voice from the first second of the call.
- The agent generates the greeting during the ring-out delay and streams it sentence by sentence.
- Wideband G.722 where the other side supports it. Smoothed gain control lifted signal-to-noise from 23 to 28.7 dB on 11 labelled real calls.
- The agent’s own “uh-huh” can echo back down the line from the caller’s handset. Recognition removes it from the transcript, and the agent keeps acknowledgements out of the conversation history.

[Listening while talking: the full note](https://dialogist.cc/en/research/listening-while-talking/)

## Streaming voices, your names

Clone a voice or pick one from the catalogue, and teach it how to say your names.

- **Voice cloning.** A new voice from a 10 to 30 second sample and its transcript.
- **Pronunciation dictionaries.** Brand names, streets and jargon said the way you say them, with a preview.
- **One voice per agent.** From the catalogue or cloned, at the speed you set.
- **Streaming synthesis.** Sentence by sentence: the next sentence is ready before the first one finishes playing.

## Atlas: Knowledge Hub

Your documents, searched inside the latency budget.

Atlas is our knowledge hub. You connect an agent to one or several knowledge bases, and Atlas returns the passages that answer the question.

- **A search before every reply.** The fast search takes about 150 ms on the server. If it has not returned within 400 ms, the agent answers without it.
- **A deeper search on demand.** For hard questions the model asks for a fuller search itself. It takes 0.7 to 1.2 s, and the agent says a filler phrase while it waits, so the line does not go silent.
- **Hybrid retrieval.** Meaning-based and keyword search, fused, with a reranker in the deeper mode.
- **Any format.** PDF, Office files, tables, web pages and scans with OCR. Atlas re-crawls sources on a schedule.
- **You can see why.** In Prompt Studio every answer shows which knowledge bases the agent searched and what came back.

- **≈ 150 ms** fast search, server side
- **0.7–1.2 s** deeper search
- **400 ms** budget before the agent goes on without it

Atlas test domain: 144 to 156 ms server side in the fast mode; 636 to 1,090 ms in the deeper mode, 0.73 to 1.20 s wall time.

## Polygon

A test bench for voice agents. It simulates the callers, grades every conversation and shows where two versions of an agent start to differ. In the Pro plan it is Dialog simulation.

Scenario → Simulated caller → Conversation → Deterministic checks → Judge → Behavior probes → Comparison

### One score, eight criteria

A judge reads the whole conversation, the caller’s facts and the tool trace, and scores each criterion with evidence quoted from the transcript.

- Goal completion: 25
- Instruction adherence: 15
- Tool use quality: 15
- Factuality: 10
- Dialogue quality: 10
- Robustness: 10
- Safety compliance: 10
- Persona and tone: 5

0 to 100: pass from 75, warn from 50. Fails whatever the score: A forbidden phrase in the agent’s speech; A safety violation; An invalid tool call that reached the caller.

### Callers that push back

- **Simulated callers with a ground truth.** Six caller types (cooperative, busy, distrustful, confused, aggressive, talkative), four intended outcomes and five difficulty levels. Each caller carries its own facts, so the judge knows whether the agent reached the goal and whether it made anything up.
- **Generated from your agent.** Polygon writes the callers from the agent’s own prompt and regenerates them when the prompt changes.
- **A hard line on purpose.** Recognition noise (dropped words, digits read one by one, words that sound alike), slow or failing tools, callers who interrupt and wander off. Seeded, so a failing conversation replays turn for turn.

### A score that means something

- **Eight criteria, weighted for a phone agent.** Reaching the goal counts five times as much as tone. Scored 0 to 100, with pass, warn or fail.
- **Hard gates.** A forbidden phrase, a safety violation or an invalid tool call that reached the caller fails the conversation whatever its score.
- **A judge that does not guess.** The judge quotes evidence from the transcript for every score, repeats are combined by the median, and an unscored criterion counts as the minimum.

### Behavior, one question at a time

- **Probes.** Yes/no checks such as “did the agent ask again after a strange reply?”, reported with a 95% interval, so nobody reads “3 of 4” as 75%.
- **Deterministic metrics.** Computed on every transcript, with loop detection that ends a conversation going round in circles.

### Two versions, side by side

- **Scenario-by-version matrix.** Sorted by where the versions disagree most.
- **The divergence point.** The first turn where two conversations part ways, with the transcripts aligned side by side.
- **Parameter sweeps.** Over models and settings, one factor at a time, on a grid or at random, ranked by score, then stability, then speed.

### Before the first call, and after the last

- **Prompt analysis.** Before any conversation runs, Polygon reads the prompt and highlights overloaded instructions, buried rules, conflicts, ambiguity and repetition.
- **Real calls.** Import production transcripts and grade them with the same rubric and probes, to see whether the new version does better on last month’s calls.
- **Every version kept.** Prompts, callers, tool stubs, rubrics and probes are immutable versions you can compare.

- **8** scored criteria
- **3** hard gates
- **6** caller types
- **5** difficulty levels

[Included in Pro as Dialog simulation](https://dialogist.cc/en/pricing/) · [How we grade a conversation](https://dialogist.cc/en/research/how-we-grade-a-conversation/)

## Our own infrastructure

Our own and open-weight models on servers we run in the EU, or on yours. No caller audio or text goes to a third-party AI provider.

- **Language model:** An open-weight language model on our own GPUs
- **Speech recognition:** Oasis Speech, our in-house recognition service; it hears 48 languages ([Oasis Speech 2](https://dialogist.cc/en/research/oasis-speech-2/))
- **Speech synthesis:** Streaming neural voices, stock or cloned from your sample
- **Turn-taking:** Voice activity detection and an end-of-turn model
- **Knowledge:** Atlas, our knowledge hub, with embeddings and a reranker
- **Testing:** Polygon, our conversation test bench ([How we grade a conversation](https://dialogist.cc/en/research/how-we-grade-a-conversation/))
- **Training:** Cadence, the workbench we train Oasis Speech in. The public repository is a fork of our production system, under the MIT license ([Cadence](https://dialogist.cc/en/research/cadence/))
- **Telephony:** SIP with G.722 and G.711, on Incoplax telephony

On-premise, the whole stack installs on your servers with Docker, offline if it has to be. A Server Unit carries 100 simultaneous calls and has been load-tested to 176% of that. [Server Unit under load](https://dialogist.cc/en/research/server-unit-under-load/)

## 100 calls per Server Unit, load-tested to 176%

A Server Unit carries the whole voice stack for up to 100 simultaneous calls. We stepped the load up from one call to 176% of rated capacity and watched reply time, the share of turns answered and the calls themselves. Then we optimized the stack for high load and ran the test again: Standard-grade capacity roughly doubled, and at the top step every call connected and none was dropped.

Load bands: Low, Moderate, High, Peak, Surge; the plan (100 calls) is the end of High.

Measured on one Server Unit. [Server Unit under load](https://dialogist.cc/en/research/server-unit-under-load/)

## By the numbers

Measured on the load-test stand with a recruiting agent, unless noted.

- **≈ 0.27 s** the agent’s own time from the caller’s last word to its first sound
- **35 ms** from the decision that the caller has finished to the first sound, with early start
- **96%** of turns have the recognized text ready within 10 ms of that decision
- **≈ 100 ms** to the first sound of a freshly synthesized sentence (median)
- **≈ 1 ms** to play a phrase that is already in the voice cache
- **5–8×** faster than real time: how quickly the voice is synthesized
- **94%** of first phrases come from the voice cache
- **98%** of replies start the language model before the pause ends
- **> 99%** of caller turns answered at 176% of rated capacity
- **≈ 150 ms** for a fast knowledge search on the server

Recognition readiness: 96 to 97% of 58,000 turns across all loads. Voice speed: median 5.0 to 6.5× across load levels, cache misses only. First sound of a fresh sentence: median 76 to 106 ms across load levels.

---

Contact: hello@dialogist.cc