Oasis Speech is our in-house speech recognition. Version 2 is an architectural change to the model, made for accuracy and quality of recognition on phone calls. We compared it with version 1 on the same recordings in all 48 languages it recognizes, on full sentences and on short replies of one to three words.
Every language is measured in phone quality: we scored each turn as recorded and after the G.711 phone codec. Short replies are cut at natural pauses. The details are in the last section.
On short replies, Oasis Speech 2 lowers the error rate by 44% (median across the 48 languages), and all 48 languages improved. A short reply comes back in about 150 ms at the median and about 210 ms at the 95th percentile.
Why short replies
A phone call is mostly short answers: “Yes.” “Tuesday.” “No, the other number.” A surname spelled out. Recognizers are usually built and tested on sentences, and a one-word turn is the case they handle worst. A single word gives the recognizer no context to lean on, and one wrong character is a large share of the word.
The agent pays for the mistake. A wrong “yes” or a misheard surname becomes a wrong booking or a confused customer. So we measure twice: on full sentences, where most recognizers do well, and on replies of one to three words, where the difference shows. The shortest answers are the hardest explains the short-reply problem in detail.
What version 2 is
Version 1 recognized each turn in one pass.
Version 2 is an architectural change to the model, built around one goal: accuracy and quality of recognition on phone calls. We designed it against what a call sounds like: short answers, narrowband audio, background noise, a caller who starts mid-word.
What a caller and an agent notice:
- Fewer errors on short replies, in all 48 languages.
- Silence stays empty. Oasis Speech returns an empty result for a cough, a line click or a voice in the background, and does not invent a word for it.
- A confidence score that means something. A score of 0.8 means about four words in five of that turn are right, for a one-word reply and for a sentence alike. The agent uses it to ask again instead of guessing.
- Tuning per language. We tune each language on its own data, and the results ship with the service.
- Real-time speed. A short turn comes back in about 150 ms at the median and about 210 ms at the 95th percentile.
Results across 48 languages
Oasis Speech 2 lowers the error rate on short replies, compared with version 1. Medians across the 48 languages; short replies are one to three words. Each turn is scored clean and through G.711, the phone codec, and the tables average the two.
| Reply length | Error rate reduction |
|---|---|
| One word | 49% |
| Two words | 42% |
| Three words | 41% |
| All short replies | 44% |
All 48 languages improve on short replies, and the change is significant in all 48 after correcting for multiple comparisons. The shorter the reply, the larger the gain.
On full sentences, where recognizers are already strong, the median error rate is 2.17% (lower is better):
| Metric | Version 1 | Version 2 | Error rate reduction |
|---|---|---|---|
| Error rate, sentences | 2.29% | 2.17% | 5% |
Sentences were already strong, so most of the gain lands where callers need it: in the first words of an answer.
Eight languages up close
Short replies and sentences, compared with version 1.
| Language | Short replies, reduction | Sentences, v1 → v2 |
|---|---|---|
| Serbian | 51% | 1.45% → 1.41% |
| Croatian | 44% | 2.97% → 2.75% |
| Bosnian | 49% | 1.91% → 1.65% |
| German | 36% | 1.53% → 1.26% |
| French | 27% | 3.45% → 2.55% |
| Spanish | 32% | 1.12% → 1.10% |
| Italian | 40% | 0.90% → 0.83% |
| Portuguese | 31% | 1.59% → 1.43% |
Central Europe, the Nordics and Benelux
Eleven more languages from the same measurements.
| Language | Short replies, reduction | Sentences, v1 → v2 |
|---|---|---|
| Polish | 42% | 2.20% → 1.89% |
| Czech | 41% | 2.12% → 1.96% |
| Slovak | 40% | 1.53% → 1.39% |
| Hungarian | 44% | 2.72% → 2.53% |
| Romanian | 36% | 2.44% → 2.21% |
| Bulgarian | 52% | 2.41% → 2.37% |
| Swedish | 39% | 3.11% → 2.83% |
| Norwegian Bokmål | 39% | 2.03% → 1.99% |
| Danish | 32% | 3.64% → 3.58% |
| Finnish | 47% | 1.34% → 1.36% |
| Dutch | 30% | 2.15% → 1.78% |
The largest gains
| Language | Error rate reduction |
|---|---|
| Tajik | 71% |
| Filipino | 66% |
| Estonian | 63% |
| Arabic | 59% |
| Macedonian | 57% |
All 48 languages
| Language | Error rate reduction |
|---|---|
| Arabic | 59% |
| Armenian | 43% |
| Azerbaijani | 43% |
| Belarusian | 39% |
| Bosnian | 49% |
| Bulgarian | 52% |
| Catalan | 30% |
| Croatian | 44% |
| Czech | 41% |
| Danish | 32% |
| Dutch | 30% |
| English | 38% |
| Estonian | 63% |
| Filipino | 66% |
| Finnish | 47% |
| French | 27% |
| Galician | 22% |
| Georgian | 42% |
| German | 36% |
| Greek | 41% |
| Hebrew | 42% |
| Hungarian | 44% |
| Icelandic | 39% |
| Indonesian | 42% |
| Irish | 22% |
| Italian | 40% |
| Kazakh | 44% |
| Kyrgyz | 47% |
| Latvian | 38% |
| Lithuanian | 45% |
| Luxembourgish | 41% |
| Macedonian | 57% |
| Maltese | 36% |
| Norwegian Bokmål | 39% |
| Polish | 42% |
| Portuguese | 31% |
| Romanian | 36% |
| Russian | 34% |
| Serbian | 51% |
| Slovak | 40% |
| Slovenian | 54% |
| Spanish | 32% |
| Swedish | 39% |
| Tajik | 71% |
| Turkish | 40% |
| Ukrainian | 44% |
| Uzbek | 42% |
| Welsh | 41% |
Real calls
On call-center audio, the calibration error of the confidence score fell from 0.193 to 0.051, a 74% reduction.
How the numbers were made
- Recordings. Recordings in each language, scored clean and through the G.711 phone codec. The tables average the two.
- Turns. We use each sentence whole, and we cut it into short replies of one to three words at natural pauses: silence of at least 150 ms on both sides, keeping 80 ms of it. A short reply cut this way sounds like a reply instead of a clipped fragment.
- Statistics. Changes carry 95% intervals paired over sentences. The claim that all 48 languages improve uses a Holm correction across languages.
The Languages page grades every language from these measurements, from A++ down to B+. 37 of the 48 hold A++.