A public, reproducible evaluation of realtime Amharic speech recognition on held-out natural and read speech, scoring both accuracy and how fast text reaches the screen.
Addis Scribe Streaming is the most accurate live system in this run and the fastest by a wide margin. An offline-only model, Hohe, is more accurate on natural speech, and we publish that alongside the rest.
On 100 held-out WAXAL clips, streaming in 0.32 s steps, Addis Scribe records 28.4% WER and 13.1% CER. Google Chirp 3 records 32.5% WER streaming and 29.2% WER when given the whole file at once.
Read the full accuracy tablesFirst words appear after 1.56 s and the transcript is final 30 ms after speech ends. Google Chirp 3 streaming takes 6.67 s and 1,411 ms on the same clips.
Read how latency was measuredHohe, an open wav2vec2-BERT model, records 25.4% WER on the same WAXAL clips, about 3 points ahead of Addis Scribe. It needs the full recording before it can answer, so it cannot run on live audio.
Read the limitationsStreaming here means the model reads each new piece of audio once, keeps what it learned in a cache, and updates the text without re-reading earlier audio.
| System | Architecture | Live audio | How it handles live audio |
|---|---|---|---|
| Addis Scribe Streaming | Cache-aware FastConformer, hybrid RNNT + CTC, 0.6B | Native streaming | Each 0.32 s (or 1.12 s) chunk is read once and cached, so cost and delay stay flat however long someone talks. |
| Google Chirp 3 (am-ET) | Cloud API | Native (cloud API) | Server-side streaming over the network. Also tested as a batch request on the whole file. |
| Shook (Whisper medium) | Whisper medium, Addis AI's previous open ASR model | Not supported, run live | Whisper transcribes complete audio. Run live with the WhisperLiveKit method: once per second it re-transcribes all audio so far, so delay grows with length. |
| Hohe (wav2vec2-BERT) | wav2vec2-BERT, third-party open model | Offline only | Needs the whole recording before it can answer. Scored for accuracy only. |
Shook is Addis AI's previous open speech recognition model. Hohe is a third-party open model, scored for accuracy only because it cannot run on live audio.
Both test sets are public and neither was used in training. WAXAL test, FLEURS and the evaluation speakers were held out of every training stage.
| Test set | Repository | Split | What it measures | Selection |
|---|---|---|---|---|
| WAXAL | google/WaxalNLP | amh test | Natural speech with human transcripts | 100 clips, 22 speakers, 29 minutes, 2 to 30 s each, drawn at random with a fixed seed |
| FLEURS | google/fleurs | am_et test | Read sentences | First 150 clips for streaming, all 516 offline |
Every system is scored with the same text normaliser: Unicode NFC, punctuation and symbols removed, whitespace collapsed. WER is word error rate and CER is character error rate, both lower is better. Addis Scribe outputs text without punctuation, so removing it from every system keeps the comparison even.
Natural conversational speech is harder than read sentences, so WAXAL error rates run higher than FLEURS for every system.
100 held-out clips, word error rate, lower is better
First 150 test clips, word error rate, lower is better
| System | Mode | WER | CER | Final text after speech ends | First words |
|---|---|---|---|---|---|
| Addis Scribe Streaming | Streaming, 0.32 s chunks | 28.4% | 13.1% | 30 ms | 1.56 s |
| Addis Scribe Streaming | Streaming, 1.12 s chunks | 27.3% | 12.4% | 34 ms | 2.21 s |
| Google Chirp 3 (am-ET) | Streaming, realtime | 32.5% | 16.5% | 1,411 ms | 6.67 s |
| Google Chirp 3 (am-ET) | Batch, whole file | 29.2% | 13.5% | n/a | n/a |
| Shook (Whisper medium) | Live, WhisperLiveKit method (1 s updates) | 35.0% | 16.6% | 10,839 ms | 3.66 s |
| Shook (Whisper medium) | Offline | 35.0% | 16.6% | n/a | n/a |
| Hohe (wav2vec2-BERT) | Offline only | 25.4% | 11.2% | n/a | n/a |
Bold marks the best value in each column across every mode, including offline ones. The 1.12 s chunk setting trades about 0.65 s of extra delay for 1.1 points lower WER.
| System | Mode | Clips | WER | CER |
|---|---|---|---|---|
| Addis Scribe Streaming | Streaming, 0.32 s chunks | First 150 | 19.8% | 8.2% |
| Hohe (wav2vec2-BERT) | Offline | First 150 | 20.6% | 7.9% |
| Previous Addis AI streaming model | Streaming, 0.32 s chunks | First 150 | 35.0% | 17.0% |
| Addis Scribe Streaming | Offline | All 516 | 20.6% | 7.7% |
| Hohe (wav2vec2-BERT) | Offline | All 516 | 20.2% | 7.3% |
On the first 150 clips Addis Scribe streaming has the lowest WER and Hohe the lowest CER. On all 516 clips offline, Hohe leads on both, by 0.4 points of WER and CER.
Audio is fed at real-time speed, as from a live microphone. We report two numbers: how long until the first words appear, and how long after the speaker stops until the transcript is final.
WAXAL, 100 clips, lower is better
WAXAL, 100 clips, lower is better
Addis Scribe and Shook were measured on one NVIDIA A100 with no network hop. Google's figures include the network round trip to its EU endpoint from the benchmark machine, so part of its delay is transport rather than the model. Shook's bar is cut at 2 s; its delay grows with utterance length because every update re-transcribes all audio so far.
Every per-clip transcript, every latency measurement and the scoring script are published next to the model. Running python score.py rebuilds the summary exactly.
The 100 WAXAL Amharic test clips: ids, speakers, durations and human reference transcripts.
Every system's transcript and latency for every clip, in every mode tested.
The scored summary behind every number on this page.
The scorer, including the shared text normaliser applied to every system.
Published at addisai/addis-scribe-streaming/benchmark. Questions to contact@addisassistant.com.
Addis Scribe Streaming is published on Hugging Face with the checkpoint, a streaming helper and the benchmark records.