Polygon is the test bench behind “Dialog simulation” in the Pro plan. It builds a simulated caller, scores each conversation on eight criteria, applies hard gates, runs behavior probes and finds the first turn where two versions of an agent part ways. The Polygon block on the Technologies page gives the short version.
This is a method note. It describes the protocol and reports no measured results. The method is written up in internal papers, with the rubric versioned (v1 and v2).
In short: eight scored criteria, weighted for a phone agent; three hard gates that fail a conversation whatever its score; six simulated caller types, each with a fixed set of facts; four tiers of prompt analysis, from cheap rules to a judge. Recognition noise and sampling use fixed seeds, so a failing run replays turn for turn.
The problem
A voice agent can answer every single question correctly and still fail the call. It forgets the goal, thanks the caller and hangs up, makes a promise it cannot keep, or repeats “Could you say that again?” four times. Grading answers one by one does not see any of that, so we grade the whole conversation.
A second problem is comparison. When we change a prompt, a model or a sampling parameter, we need to know whether the agent got better, on which kind of caller, and at which turn the behavior first changed. A single average cannot say that.
The pipeline
scenario → simulated caller → conversation with tool stubs and recognition noise → deterministic checks → judge → behavior probes → comparison
Each stage is described below. We keep the stages separate so that a result can be traced to the stage that produced it.
Simulated callers with a ground truth
The agent is tested against a model playing a persona. A persona has:
- A type: cooperative, busy, distrustful, confused, aggressive or talkative.
- An intended outcome: success, refusal, call back or transfer.
- A difficulty: 1 to 5.
- A ground truth: the facts the caller knows. The judge uses it to decide whether the agent reached the goal and whether it invented anything.
Polygon generates callers from the agent’s own prompt, one persona per request, so a single failure does not take the whole batch with it. Each caller is tied to the version of the prompt it was generated from. When the prompt changes, Polygon marks the callers stale and regenerates them.
The callers speak like callers on a phone: short sentences, interruptions, sidetracks.
Recognition noise on purpose
An agent that is flawless on clean text can break on what recognition delivers on a real call. Polygon distorts the caller’s text before the agent sees it, with one intensity dial from 0 to 1:
| Distortion | Example |
|---|---|
| Dropped words | a word disappears |
| Numbers read digit by digit | “fifty” becomes “five zero” |
| Words that sound alike | in Russian, «код» (code) becomes «кот» (cat) |
| Fillers at the start | “uh”, “hmm”, “hello?” |
Polygon seeds the distortion from the conversation and the turn number, so a conversation that failed can be replayed turn for turn.
Tool stubs with faults
Agents call tools: calendar, CRM, transfer. Polygon replaces them with stubs that return fixed answers, templated answers or answers by rules, or keep state across calls. A stub can add latency and inject errors, so the agent is tested against a slow or failing CRM as well as a healthy one.
The rubric and the composite score
A judge model reads the whole conversation, the caller’s ground truth and the tool trace. It scores eight criteria from 1 to 5 and quotes evidence from the transcript for each.
| Criterion | Weight |
|---|---|
| Goal completion | 25 |
| Instruction adherence | 15 |
| Tool use quality | 15 |
| Factuality | 10 |
| Dialogue quality | 10 |
| Robustness | 10 |
| Safety compliance | 10 |
| Persona and tone | 5 |
The composite is the weighted mean of the normalised scores, from 0 to 100: composite = Σ weight × (score − 1) / 4. All ones give 0, all fives give 100, all threes give 50. A conversation passes at 75 and above and gets a warning from 50. The weights say what a phone agent is for: reaching the goal counts five times as much as tone.
Four rules keep the score honest:
- A missing score is the minimum, never the average. An incomplete judgment cannot flatter an agent.
- An unreadable score is not a 1. When the judge returns a score in a form Polygon cannot read, it counts as “not scored” and triggers nothing, so it cannot fail a conversation by accident. The result carries a flag, so nobody mistakes a partial composite for a complete one.
- Repeats take the median. The judge can run k times; Polygon uses the median of each criterion, and for an even k it averages the two middle values.
- Scores compare only inside one rubric version. Changing weights or criteria means a new version, and the comparison screen refuses to put two versions on one chart.
Hard gates
Some failures must not be averaged away. Whatever the composite, the verdict is fail when:
- a forbidden phrase appears in the agent’s speech;
- the safety criterion scores 1 (and was scored);
- a tool call with invalid arguments produced a reply to the caller. Rubric v2 added this gate; v1 conversations keep their original verdict.
Behavior probes
The rubric says how good a conversation was overall. A probe asks one yes-or-no question about one behavior: did the agent ask again after a strange reply, did it give the reason for a refusal, did it make a promise it could not keep.
- Answers: yes, no or not applicable. Polygon leaves not-applicable answers out of the denominator and never counts them as zero.
- Polarity: for a wanted behavior yes is good, for a guardrail no is good. The reported number is conformance, the share of applicable conversations in which the agent did the right thing.
- A separate pass: probes run with their own prompt and cannot change the composite or the verdict. Asking for a calibrated 1-to-5 score and for a binary fact in the same call anchors both.
- Intervals: every rate carries a Wilson 95% interval, so “3 of 4” shows as a wide range around 75%.
- Fixed wording: Polygon compares a probe across runs only while its wording is unchanged (it checks a content hash), so a rephrased question cannot shift a trend unnoticed.
- Cheap to add: we add probes freely as hypotheses about behavior. The ones that prove important move into the rubric or become a gate.
Comparing two versions
The comparison screen takes two or more configurations (a model, a prompt version, a parameter profile) and shows:
- A scorecard per configuration.
- A matrix of scenarios, ordered by spread: the difference between the best and the worst configuration on that scenario. The scenarios where the versions disagree most come first.
- The divergence point: the first turn at which two conversations part ways. Polygon aligns the conversations turn by turn and takes the first turn where the tool calls differ, where one ends or transfers the call and the other does not, where the wording moves apart (Jaccard similarity of word bigrams below 0.5), or where the speakers differ.
- A written analysis of systematic differences, produced from a deterministic summary of the scorecards.
When one configuration has no result for a scenario, Polygon marks the scenario incomparable instead of filling it with zeros.
Reading the prompt before running it
Polygon reads the agent’s prompt in four tiers, cheapest first: deterministic rules, then the surprisal of each phrase under a language model, then embeddings, then a judge. The output is a score from 0 to 1 per axis, higher being worse, with highlights in the editor.
| Axis | What it catches |
|---|---|
| Overload | too many simultaneous constraints |
| Attention dilution | key rules lost in the middle of a long prompt; related rules scattered |
| Conflict | rules that exclude or water down each other |
| Ambiguity | wording with more than one reading |
| Redundancy | the same rule stated several ways |
A prompt can be fragile and still pass every test, and this check finds the weak spots before a caller runs into them.
Parameter sweeps and loops
- Sweeps change one parameter at a time, or run a full grid or random points, over a short scenario set. Each point becomes an immutable profile version. A sweep can stop early if the first point is already below a floor, and it refuses to start a point that would exceed a token cap. The leaderboard sorts by mean score, then by stability, then by speed.
- Loop detection ends a conversation early when two of the agent’s last replies overlap by 90% or more, and marks it. A one-word reply such as “Yes.” counts, which a bigram-only check would have missed.
Real calls, as they happened
Polygon can import production transcripts and grade them without replaying them. A real conversation has no simulated caller and no ground truth, so the checks that depend on those become not applicable. The rest (deterministic metrics, the judge on the transcript, probes) runs as for a simulated conversation. The result is a normal run that can sit in the same comparison with synthetic ones, and it tells us whether a new prompt does better on last month’s calls.
Reading the results
- Judge settings. The judge runs at a temperature near zero with structured output. Repeats with a median make it steadier.
- The criteria behind the composite. The composite is one number for a ranking; the eight criteria behind it say what to fix.
- Realistic callers. Polygon generates callers from the agent’s prompt, and we sample them by hand from time to time to keep them realistic.
- Real calls. Importing production transcripts tests the same agent against the calls it had.