A public, reproducible evaluation of Amharic text to speech across 100 prompts drawn from open datasets.
Every measure from this run, with the leading system in each row highlighted. Where another system leads, its cell is highlighted rather than left out.
| Measure | Addis Voice 2 | Azure Ameha | Gemini 3.1 Flash | gpt-4o-mini-tts | ElevenLabs v3 | Meta MMS |
|---|---|---|---|---|---|---|
| Composite scoreAmharic, 100 clips, blind, 1 to 5, higher is better | 4.25 | 3.49 | 3.84 | 2.55 | 2.00 | 3.03 |
| Character error rate87 prompts, ASR judge, lower is better | 4.44% | 3.03% | 2.89% | 18.74% | 111.65% | 7.86% |
| Word error rate87 prompts, ASR judge, lower is better | 13.87% | 11.16% | 11.02% | 31.34% | 147.86% | 22.89% |
| NaturalnessAmharic, blind listening axis, higher is better | 4.01 | 2.54 | 3.90 | 1.20 | 1.00 | 3.09 |
| Intelligibility, rated by earAmharic, blind listening axis, higher is better | 4.80 | 4.49 | 3.96 | 3.47 | 1.23 | 2.84 |
| Voice qualityAmharic, blind listening axis, higher is better | 3.94 | 3.44 | 3.67 | 2.98 | 3.77 | 3.17 |
| Latency, p50Amharic, median request time, not adjusted for clip length, lower is better | 5.40s | 1.05s | 8.94s | 3.80s | 9.97s | 7.01s |
Azure ships two Amharic voices. The stronger is shown.
A speech system has to be understood and it has to sound like a person. We publish both rankings side by side because they measure different properties of speech quality.
Scored by running every clip through a speech recognition judge and measuring how far the transcript drifts from the prompt. Addis Voice 2 records 4.44% CER and 13.87% WER. Gemini 3.1 Flash TTS and Azure Ameha Neural score lower error on this proxy, and their confidence intervals overlap ours.
Read the method and the full tableIn a blind pass over the same clips, Addis Voice 2 rates highest of every system tested on naturalness 4.01, intelligibility 4.80 and voice quality 3.94, each out of 5.
Read the method and the full tableRecognition error measures transcript accuracy, while the listening evaluation measures perceived speech quality.
Systems without documented Amharic synthesis support are included as unsupported-language conditions and should be interpreted in that context.
| System | Model | Voice | Condition |
|---|---|---|---|
| Addis Voice 2Production standard checkpoint. | addis-voice-2 | am-yohanes-calm | Documented language support |
| Azure Ameha NeuralMicrosoft ships two documented Amharic neural voices. | am-ET-AmehaNeural | am-ET-AmehaNeural | Documented language support |
| Azure Mekdes Neural | am-ET-MekdesNeural | am-ET-MekdesNeural | Documented language support |
| Gemini 3.1 Flash TTSPublic preview. Google documents Amharic support for Gemini-TTS. | gemini-3.1-flash-tts-preview | Kore | Documented language support |
| OpenAI gpt-4o-mini-tts | gpt-4o-mini-tts | coral | Unsupported language condition |
| ElevenLabs v3ElevenLabs documents Amharic for transcription, not for synthesis. | eleven_v3 | 21m00Tcm4TlvDq8ikWAM | Unsupported language condition |
| Meta MMS | facebook/mms-tts-amh | single-speaker | Open research model |
| Human reference recordingThe original recording each prompt was taken from. Field audio with variable level and microphone quality, included as a reference point for the listening axes. | Original dataset audio | Dataset speakers | Human recording |
Addis Voice 2 in this run is the production standard checkpoint, with reference voice am-yohanes-calm.
The prompts are public test-split rows from three open datasets, pinned to exact repository revisions. Nothing was written for this benchmark, nothing was hand picked, and the selection was drawn once with a fixed seed.
| Source | Repository | Split | Licence | Rows | Pinned revision |
|---|---|---|---|---|---|
| FLEURS | google/fleurs | test | CC BY 4.0 | 34 | 70bb2e84b976b7e960aa89f1c648e09c59f894dd |
| Waxal | google/WaxalNLP | test | CC BY-SA 4.0 | 33 | e91442a8989b291485cb6f8a56c0a8743d36036a |
| Horn-ASR | LesanAI/Horn-ASR | test | CC BY-SA 4.0 | 33 | a2a8ab2f09e715256a1b43db572d6abedd1a8129 |
Balanced on purpose. Short prompts give a recogniser almost no context to recover from an error, and long prompts are where repetition and truncation failures appear.
WER/CER was measured with Chirp 3, a third-party recogniser, so that no system in this benchmark is scored by its own vendor. The pipeline is recogniser-agnostic and can be re-run with Addis AI STT or our open Amharic models; across the recognisers we tested the ordering held, so the scores track the audio rather than the judge. Read them as a recognition-based proxy for intelligibility.
Text is Unicode normalised, punctuation is stripped, equivalent Amharic characters are folded, and whitespace is collapsed before the edit distance is taken. CER is measured over the normalised text with spaces removed; WER over whitespace-split tokens.
Lower is better. Each row is the same 87 prompts put through the same judge. Confidence intervals are deterministic utterance-bootstrap intervals at 95%. Confidence intervals show uncertainty around each estimate; overlapping intervals should be interpreted cautiously.
Primary CER/WER uses 87 non-numeric prompts to avoid digit and spoken-form normalisation ambiguity. The full 100-prompt scores are shown separately in the strict full set column.
Amharic, 87 prompts, lower is better
Amharic, 87 prompts, lower is better
| System | n | CER | 95% CI | WER | 95% CI | Strict full set CER / WER |
|---|---|---|---|---|---|---|
| Gemini 3.1 Flash TTS | 87 | 2.89% | 2.17 – 3.73 | 11.02% | 8.94 – 13.27 | 3.90% / 12.48% |
| Azure Ameha Neural | 87 | 3.03% | 2.25 – 3.88 | 11.16% | 9.08 – 13.27 | 4.18% / 12.93% |
| Azure Mekdes Neural | 87 | 4.15% | 3.11 – 5.16 | 13.01% | 10.78 – 15.29 | 5.37% / 14.94% |
| Addis Voice 2 | 87 | 4.44% | 3.46 – 5.54 | 13.87% | 11.57 – 16.21 | 5.76% / 15.80% |
| Meta MMS | 87 | 7.86% | 6.71 – 9.19 | 22.89% | 20.27 – 25.55 | 8.97% / 24.96% |
| OpenAI gpt-4o-mini-tts | 87 | 18.74% | 13.31 – 25.23 | 31.34% | 26.06 – 37.02 | 19.07% / 31.77% |
| ElevenLabs v3 | 87 | 111.65% | 98.55 – 128.99 | 147.86% | 130.97 – 168.92 | 112.72% / 148.81% |
Amharic, lower is better
Amharic, lower is better
| By source | n | CER | WER |
|---|---|---|---|
| FLEURS | 34 | 6.71% | 16.61% |
| Horn-ASR | 33 | 7.47% | 19.03% |
| Waxal | 33 | 3.42% | 11.78% |
| By length | n | CER | WER |
|---|---|---|---|
| Short | 33 | 8.06% | 18.75% |
| Medium | 34 | 5.47% | 14.79% |
| Long | 33 | 5.14% | 15.36% |
| Held out of the primary table | n | CER | WER |
|---|---|---|---|
| Numeric prompts | 13 | 14.94% | 28.18% |
| Slice | Rows | Character errors / reference characters | CER | Word errors / reference words | WER |
|---|---|---|---|---|---|
| Primary, digits excluded | 87 | 362 / 8,152 | 4.44% | 292 / 2,106 | 13.87% |
| Strict full set | 100 | 537 / 9,323 | 5.76% | 385 / 2,436 | 15.80% |
| Numeric prompts only | 13 | 175 / 1,171 | 14.94% | 93 / 330 | 28.18% |
Percentages are micro-averages of raw edit counts, not the mean of per-utterance percentages. The two differ, and publishing the counts lets you check which one we used.
Every clip was rated by native Amharic speakers on naturalness, intelligibility and voice quality on a 1 to 5 scale. Provider identities were hidden behind rotating labels during evaluation and revealed only after scoring was complete. Composite is the mean of the three axes.
Amharic, 100 clips per system, 1 to 5, higher is better
| System | Composite | n | Naturalness | Intelligibility | Voice quality |
|---|---|---|---|---|---|
| Addis Voice 2 | 4.25 | 100 | 4.01 | 4.80 | 3.94 |
| Gemini 3.1 Flash TTS | 3.84 | 100 | 3.90 | 3.96 | 3.67 |
| Azure Ameha Neural | 3.49 | 100 | 2.54 | 4.49 | 3.44 |
| Meta MMS | 3.03 | 100 | 3.09 | 2.84 | 3.17 |
| OpenAI gpt-4o-mini-tts | 2.55 | 100 | 1.20 | 3.47 | 2.98 |
| ElevenLabs v3 | 2.00 | 100 | 1.00 | 1.23 | 3.77 |
Numbers about speech are only worth what the audio behind them is worth. These are the archived benchmark outputs, unedited, with the exact transcript the recognition judge returned and the score that transcript produced.
The lowest-numbered medium-length Amharic prompt from each of the three sources, selected before the audio was reviewed.
ሁሉም ሰው በኅብረተሰብ ውስጥ ይሳተፋል እና የትራንስፖርት ስርዓቶችን ይጠቀማል ሁሉም ሰው ማለት ይቻላል ስለ መጓጓዣ ስርዓቶች ቅሬታ ያቀርባል
ሁሉም ሰው በህብረተሰብ ውስጥ ይሳተፋል እና የትራንስፖርት ስርዓቶችን ይጠቀማል ሁሉም ሰው ማለት ይቻላል ስለ መጓጓዣ ስርዓቶች ቅርሬታ ያቀርባል
ሁሉም ሰው በሕረብተሰብ ውስጥ ይሳተፋል እና የትራንስፖርት ስርዓቶችን ይጠቀማል፣ ሁሉም ሰው ማለት ይቻላል ስለ መጓጓዣ ስርዓቶች ቅሬታ ያቀርባል።
ሁሉም ሰው በህብረተሰብ ውስጥ ይሳተፋልና የትራንስፖርት ስርዓቶችን ይጠቀማል ሁሉም ሰው ማለት ይቻላል ስለ መጓጓዣ ስርዓቶች ቅሬታ ያቀርባል
ሁሉም ሰው በህብረተሰብ ውስጥ ይሳተፋልና የትራንስፖርት ስርዓቶችን ይጠቀማል ሁሉም ሰው ማለት ይቻላል ስለ መጓጓዣ ስርዓቶች ቅሬታ ያቀርባል
ሁሉም ሰው በህብረተሰብ ውስጥ ይሳተፋልና የትራንስፖርት ስርዓቶችን ይጠቀማል። ሁሉም ሰው ማለት ይቻላል ስለ መጓጓዣ ስርዓቶች ቅሬታ ያቀርባል።
ሁሉም ሰው በህብረተሰብ ውስጥ ይሳተፋል እና የትራንስፖርት ስርዓቶችን ይጠቀማል ሁሉም ሰው ማለት ይቻላል ስለ መጓጓዣ ስርዓቶች ቅሬታ ያቀርባል
ሁሉም ሰው በህብረተሰብ ውስጥ ይሳተፋል እና የጠራን ስፖርት ስርዓቶችን ይጠቀማል ሁሉም ሰው ማለት ይቻላል ስለ መጓጓዣ ስርዓቶች ቀሬታ ያቀርባል
ሳይሞ ብሪግን አወት ሶአርት ቱዋይ ኢት ኢን ቬካቭ ፓቡ ሳይቭ ዶር ሲንግ ኡን ሱጋ ዲ ቻይ ሳይሞ ፋይቭ ሱጃይ ሴ አማቴቶ ሳይቭ ዶር ሲንግ ኡን ቱል ፊ ዳብ ቻይ
Clips are loudness-normalised to a common level and transcoded to mono 24 kHz MP3 for playback. Scoring ran on canonical 16 kHz mono PCM WAV files. Human reference clips are redistributed under the licence of their source dataset, CC BY 4.0 for FLEURS and CC BY-SA 4.0 for Waxal and Horn-ASR, with attribution in the dataset table above.
Every system was measured through its public API from a single location, so these are raw request times. Clip length differs between systems and is not adjusted for: Addis Voice 2 generated 31.25 minutes of audio across the 100 prompts against Azure Ameha Neural's 18.68. The RTF column divides time taken by audio produced and is the like-for-like comparison.
p50 with the tail out to p95, lower is better
| System | n | p50 | p95 | Mean RTF | Failures | Serving |
|---|---|---|---|---|---|---|
| Addis Voice 2 | 100 | 5.401s | 10.827s | 0.444 | 0 | Vendor API |
| Azure Ameha Neural | 100 | 1.046s | 1.871s | 0.106 | 0 | Vendor API |
| Azure Mekdes Neural | 100 | 1.045s | 1.752s | 0.110 | 0 | Vendor API |
| OpenAI gpt-4o-mini-tts | 100 | 3.798s | 8.238s | 0.268 | 0 | Vendor API |
| Meta MMS | 100 | 7.014s | 18.444s | 0.576 | 0 | Open weights, run locally |
| Gemini 3.1 Flash TTS | 100 | 8.941s | 17.537s | 0.700 | 0 | Vendor API |
| ElevenLabs v3 | 100 | 9.970s | 21.591s | 0.891 | 0 | Vendor API |
| Provider | Billing unit | Rate | Per generated audio minute | Recorded in this run |
|---|---|---|---|---|
| Addis Voice 2 | Generated audio minute | USD 0.031 per minute | USD 0.0309 | USD 0.96 for 31.25 generated minutes |
| Azure Ameha Neural | Characters of input | USD 15.00 per 1M characters | USD 0.0095 | USD 0.18 for 11,889 input characters, 18.68 minutes |
| Azure Mekdes Neural | Characters of input | USD 15.00 per 1M characters | USD 0.0100 | USD 0.18 for 11,889 input characters, 17.92 minutes |
| Gemini 3.1 Flash TTS | Output audio tokens | USD 20.00 per 1M output tokens | USD 0.0300 | USD 0.69 for 34,695 output tokens, 23.13 minutes |
| OpenAI gpt-4o-mini-tts | Generated audio minute | USD 0.015 per minute | USD 0.0150 | USD 0.44 for 29.57 generated minutes |
| ElevenLabs v3 | Characters of input | USD 0.10 per 1,000 characters | USD 0.0575 | USD 1.19 for 11,889 input characters, 20.68 minutes |
| Meta MMS | Open weights, run locally | No vendor rate | No vendor rate | 26.50 minutes, infrastructure cost only |
Every row covers the same 100 Amharic prompts and is computed from the generated audio and input text of this run. Providers bill in different units, so the columns are not directly comparable. The Gemini figure covers output audio only.
The prompts, every generated clip, the judge transcripts, the scorer and the checksums are published as an open dataset. Running score.py rebuilds the CER/WER tables and figures from the published judge transcripts.
100 rows. Source dataset, pinned repository revision, upstream row id, split, licence, reference text, normalised text, length stratum, character count, human audio hash and duration.
800 rows. System, raw and normalised transcript, character and word edit counts, reference lengths, CER, WER, judge confidence, plus the audio path and SHA-256 of the exact file that produced the row.
800 clips. One per prompt per system, plus the original human recordings, with a metadata index listing each clip against its transcript, CER and WER.
The scorer, which rebuilds the CER/WER tables and figures from the published judge transcripts with no arguments, and a SHA-256 for every shipped file.
800 judge rows, 800 audio clips, the scorer and a checksum for every file are published at addisai/amharic-tts-benchmark. Questions to contact@addisassistant.com.
The clips above are 100 rows of a public dataset. Your workload is not. Run Addis Voice 2 against your own Amharic content and check the result yourself.