Researchers from Hugging Face and Hume AI report that several top-scoring speech-to-text models reproduce the wording of standard test transcripts even when the audio says something different, numbers have been silenced, or two spellings sound identical. The behaviour largely disappears on freshly recorded audio from the same settings, suggesting the models recognise which test they are being given. A "Benchmark fitting" tab measuring this has been added to the Open ASR Leaderboard.
What changed
Speech recognition models were ranked mainly by word error rate on public test sets like VoxPopuli and LibriSpeech.
What it unlocks
Checking whether a speech-to-text model's benchmark score reflects real transcription skill, using a new "Benchmark fitting" tab on the Open ASR Leaderboard and open-sourced test scripts.
- 11 open-source models tested
- 18-30% reproduce wrong reference
- 30-40% recover silenced numbers
- ~90% top spelling-switch accuracy
Sources