“Yes”, “Thursday”, “no, the other one”. Most of what a caller says to an agent is a few words, and a few words are the hardest thing for speech recognition to get right. This note explains why, and describes the short-reply path we built into Oasis Speech 2. The per-language results are in Oasis Speech 2: recognition in 48 languages.
We measured in the 48 languages Oasis Speech recognizes, in phone quality: each turn clean and through the G.711 phone codec, plus recordings of real people making short spoken requests. Version 1 and version 2 of Oasis Speech ran on the same audio.
Oasis Speech 2 lowers the error rate on one-word replies by 49% (median across the 48 languages), and all 48 languages improved on short replies, each change statistically significant. A short reply is recognized in about 150 ms at the median (95th percentile about 210 ms), and on call-center audio the calibration error of the confidence score fell by 74%.
A phone call is made of short answers
Listen to a service call from the caller’s side. “Yes.” “Tuesday.” “No, the other number.” “Petrović.” “Four four one seven.” The agent asks, and the caller answers in one to three words.
Those words carry the call. A misheard “yes” books the wrong slot. A misheard surname creates a second customer record. A misheard digit sends the reminder to the wrong account.
Teams usually build, tune and test speech recognition on sentences. On a sentence, a conventional recognizer gets about two characters in a hundred wrong. On a single word cut out of the same speech, a conventional recognizer can get one character in two wrong. Short answers are the hardest case for recognition and the most common one on a call.
Why short is hard
Six effects stack up on a short reply. Each is small on a sentence and large on a word.
The context deficit
Recognition leans on context. Inside a sentence the recognizer predicts each word from its neighbors, and the neighbors rule out most of the wrong candidates. A word on its own has no neighbors. The only evidence is the sound.
The evidence budget
A one-word reply lasts 300 to 600 ms. A click on the line, a breath or a codec artifact covers a far larger share of it than of a five-second sentence, and there is nothing around it to average the damage away.
Fragile edges
The first and last sounds of a turn are the easiest to lose: the speaker starts softly, voice detection opens a fraction late, the last consonant fades. In a sentence those are two sounds out of fifty. In “yes” they are two out of three.
The narrow band
A phone line carries roughly 300 to 3,400 Hz. The consonants that tell many words apart (s, f, sh and th) carry much of their energy above that. In a sentence, context restores them. In a single word, nothing does.
The pull of silence
A recognizer trained on fluent speech wants to produce words. Given a breath, a cough or a few hundred milliseconds of line noise, it tends to return a short, plausible word: “yes”, “okay”, “hello”. Short turns contain much more of this kind of audio than sentences do.
The unforgiving metric
One wrong letter in “yes” is a third of the word. The same mistakes that cost a sentence one point of error cost a short reply thirty.
A path of its own
Oasis Speech 2 treats a short turn as a problem of its own. When a turn is shorter than a few seconds, it takes a dedicated short-reply path that sentences never reach. The path does four things a sentence recognizer does not need to do.
It weighs several readings before it answers
The short-reply path keeps several competing readings instead of committing to the first transcription it finds. It chooses between them with the evidence that matters for a short answer: how well each one agrees with the sound, how plausible it is as something a person says in reply, and whether its length fits the length of the speech.
It knows how long a reply should be
On short audio a recognizer tends to drop a word or invent one. A length check, fitted per language, holds the answer to what the duration of the speech can contain.
It keeps silence silent
A dedicated non-speech gate looks at the acoustics of the turn. For a cough, a click or a background voice it returns an empty result instead of an invented “yes”. On real recordings of short replies the gate stayed out of the way: it blanked at most 0.2% of genuine short answers.
It says how sure it is
Every result carries a confidence score, calibrated so that 0.8 means about four words in five of that turn are right, for a one-word reply and for a sentence alike. On call-center audio the calibration error fell by 74%. The agent uses the score: below a threshold it asks the caller to repeat instead of guessing a name or a digit.
Tuned for each language
We fit the path separately for each of the 48 languages, on short replies treated as their own category rather than as a by-product of sentences. A language’s settings ship only if the gain is statistically certain: a paired bootstrap interval of the change that lies wholly below zero.
The path keeps recognition fast. A short reply comes back in about 150 ms at the median and about 210 ms at the 95th percentile.
How we measured
- 48 languages, measured in phone quality.
- Both lines. We measure each turn twice: as recorded, and after passing through G.711, the codec of an ordinary phone line. Figures are the average of the two.
- Short turns cut like replies. From each sentence we take one long turn and two short ones of one to three words. We cut a short turn at a natural pause where possible, keep 80 ms of the silence around it, and pad it with the recording’s own silence, never with the neighboring words.
- Reply-like turns. A subset of short turns with silence on both sides and no lone filler word: the closest thing to a caller’s answer.
- Version 1 against version 2 on the same audio, cut and padded the same way.
- Statistics. Medians across languages with 95% intervals that resample both the languages and the sentences in each. A language counts as improved only if its change stays significant after Holm’s correction across all 48.
Results
Oasis Speech 2 lowers the error rate on short replies, compared with version 1. Median across 48 languages.
| Short replies | Error rate reduction | Languages improved |
|---|---|---|
| One word | 49% | 48 / 48 |
| Two words | 42% | 48 / 48 |
| Three words | 41% | 48 / 48 |
| All short turns | 44% | 48 / 48 |
| Reply-like turns | 43% | 36 / 47, none worse |
Eight languages up close
| Language | Error rate reduction |
|---|---|
| Serbian | 51% |
| Bosnian | 49% |
| Croatian | 44% |
| Italian | 40% |
| German | 36% |
| Spanish | 32% |
| Portuguese | 31% |
| French | 27% |
On real short recordings
We also measured on recordings of real people making short spoken requests, one to three words long, in their own voices. Oasis Speech 2 lowers the error rate on them, through the G.711 phone codec:
| Language | Error rate reduction |
|---|---|
| Spanish | 27% |
| German | 41% |
| Dutch | 30% |
| Turkish | 33% |
| Portuguese | 31% |
| Polish | 29% |
| French | 40% |
| Hungarian | 52% |
| Arabic | 34% |
Version 2 beat version 1 in every language here, on clean audio and through the phone codec alike. These are the short-turn figures closest to what a caller sounds like. On clean audio the error rate on Spanish short replies is 2.9% and on German 4.0% (lower is better).
What the agent does with it
The agent builds on what recognition returns:
- It reads back what matters. The agent reads names, numbers, amounts and dates back to the caller before it records them.
- It asks again instead of guessing. When the confidence score is low, the agent asks the caller to repeat, and it never fills in a digit it did not hear.
- We test it on bad recognition on purpose. Polygon feeds agents distorted text, dropped words, digits read one by one and words that sound alike, to check that the agent copes when recognition does not. How we grade a conversation describes the method.
The Languages page grades each of the 48 languages the agent hears and speaks.