# Open ASR Leaderboard: Towards Reproducible and Transparent Multilingual and Long-Form Speech Recognition Evaluation

arXiv: https://arxiv.org/abs/2510.06961 · HTML: https://mlsanity.com/paper/2510.06961/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 38 references that are in the map.
- First posted on arXiv 8 Oct 2025.

## Builds on (most cited first, 30 of 30)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language](https://mlsanity.com/paper/2202.03555/) (2022)
- [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018)
- [Datasets: A Community Library for Natural Language Processing](https://mlsanity.com/paper/2109.02846/) (2021)
- [MLS: A Large-Scale Multilingual Dataset for Speech Research](https://mlsanity.com/paper/2012.03411/) (2020)
- [FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech](https://mlsanity.com/paper/2205.12446/) (2022)
- [VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation](https://mlsanity.com/paper/2101.00390/) (2021)
- [GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audio](https://mlsanity.com/paper/2106.06909/) (2021)
- [NeMo: a toolkit for building AI applications using Neural Modules](https://mlsanity.com/paper/1909.09577/) (2019)
- [TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptation](https://mlsanity.com/paper/1805.04699/) (2018)
- [Fast Conformer with Linearly Scalable Attention for Efficient Speech Recognition](https://mlsanity.com/paper/2305.05084/) (2023)
- [Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo Labelling](https://mlsanity.com/paper/2311.00430/) (2023)
- [SPGISpeech: 5,000 hours of transcribed financial audio for fully formatted end-to-end speech recognition](https://mlsanity.com/paper/2104.02014/) (2021)
- [CoVoST 2 and Massively Multilingual Speech-to-Text Translation](https://mlsanity.com/paper/2007.10310/) (2020)
- [Earnings-21: A Practical Benchmark for ASR in the Wild](https://mlsanity.com/paper/2104.11348/) (2021)
- [Efficient Sequence Transduction by Jointly Predicting Tokens and Durations](https://mlsanity.com/paper/2304.06795/) (2023)
- [Open-Source Conversational AI with SpeechBrain 1.0](https://mlsanity.com/paper/2407.00463/) (2024)
- [Earnings-22: A Practical Benchmark for Accents in the Wild](https://mlsanity.com/paper/2203.15591/) (2022)
- [CrisperWhisper: Accurate Timestamps on Verbatim Speech Transcriptions](https://mlsanity.com/paper/2408.16589/) (2024)
- [OWSM v4: Improving Open Whisper-Style Speech Models via Data Scaling and Cleaning](https://mlsanity.com/paper/2506.00338/) (2025)
- [Streaming Sequence-to-Sequence Learning with Delayed Streams Modeling](https://mlsanity.com/paper/2509.08753/) (2025)
- [Moonshine: Speech Recognition for Live Transcription and Voice Commands](https://mlsanity.com/paper/2410.15608/) (2024)
- [Investigating End-to-End ASR Architectures for Long Form Audio Transcription](https://mlsanity.com/paper/2309.09950/) (2023)
- [Evaluation of LLMs in Speech is Often Flawed: Test Set Contamination in Large Language Models for Speech Recognition](https://mlsanity.com/paper/2505.22251/) (2025)
- [LiteASR: Efficient Automatic Speech Recognition with Low-Rank Approximation](https://mlsanity.com/paper/2502.20583/) (2025)
- [SpeechColab Leaderboard: An Open-Source Platform for Automatic Speech Recognition Evaluation](https://mlsanity.com/paper/2403.08196/) (2024)

## Built on by (most cited first, 23 of 23)

- [Nemotron 3 Nano Omni: Efficient and Open Multimodal Intelligence](https://mlsanity.com/paper/2604.24954/) (2026)
- [NemotronLabs VoiceChat: An Open Full-duplex Speech-to-Speech Model with Tool Calling Capabilities](https://mlsanity.com/paper/2609.21967/) (2026)
- [RW-Voice-EQ Bench: A Real World Benchmark for Evaluating Voice AI Systems](https://mlsanity.com/paper/2607.14846/) (2026)
- [A frontend-backend architecture for tool calls in full-duplex speech models](https://mlsanity.com/paper/2609.19334/) (2026)
- [Exploring the potential and limitations of Model Merging for Multi-Domain Adaptation in ASR](https://mlsanity.com/paper/2603.05354/) (2026)
- [Reducing the Offline-Streaming Gap for Unified ASR Transducer with Consistency Regularization](https://mlsanity.com/paper/2604.19079/) (2026)
- [FastSLM: Hierarchical Temporal Abstraction for Efficient Long-Form Speech Adaptation](https://mlsanity.com/paper/2601.06199/) (2026)
- [AppTek Call-Center Dialogues: A Multi-Accent Long-Form Benchmark for English ASR](https://mlsanity.com/paper/2604.27543/) (2026)
- [FSA-GRPO: Teaching Auditory LLMs to Use Few-Shot Demonstrations](https://mlsanity.com/paper/2606.02615/) (2026)
- [TRADE: Transducer-Augmented Decoder for Speech LLM](https://mlsanity.com/paper/2606.08486/) (2026)
- [MLLP-VRAIN UPV system for the IWSLT 2026 Simultaneous Speech Translation task](https://mlsanity.com/paper/2606.17255/) (2026)
- [HALAS: A Human-Annotated Dataset of Hallucinations of Modern ASR Systems](https://mlsanity.com/paper/2606.23048/) (2026)
- [Position: AI Leaderboards Are Underserving the Global South: A Case Study from India](https://mlsanity.com/paper/2608.18117/) (2026)
- [Towards Quantifying Benchmark Optimization in ASR Models](https://mlsanity.com/paper/2608.19936/) (2026)
- [Domain-Adaptive ASR for Telephony AI Agents: Fine-tuning Canary Flash Models for Enterprise Contact Center Applications](https://mlsanity.com/paper/2608.24916/) (2026)
- [Formal, Executable and Explainable Runtime Monitoring of Spoken Air Traffic Control Operational Procedures](https://mlsanity.com/paper/2608.25926/) (2026)
- [Listen to the Latents: Self-Correcting Speech Recognition in Large Audio Language Models Through Hidden-State Interactions](https://mlsanity.com/paper/2609.02940/) (2026)
- [MedWER: A Reproducible, Model-Free Evaluation Protocol for Medical Speech Recognition](https://mlsanity.com/paper/2609.05728/) (2026)
- [Whisper-Based Speech Transcription from Videos Across Multiple Languages for Cross-Cultural Understanding](https://mlsanity.com/paper/2609.11772/) (2026)
- [Look Less, Hear Better: Jointly Rewarded GRPO for Streaming ASR](https://mlsanity.com/paper/2609.18333/) (2026)
- [In-Context Adaptation of Encoder-Decoder Models in Speech Recognition](https://mlsanity.com/paper/2609.33865/) (2026)
- [Benchmarking Automatic Speech Recognition Tools for Iberian Languages](https://mlsanity.com/paper/2609.36920/) (2026)
- [CAFE: Confining Feature Distillation to Language-Routed Adapters for Compact Multilingual Automatic Speech Recognition](https://mlsanity.com/paper/2610.19134/) (2026)

## Abstract (arXiv)

We present the Open ASR Leaderboard, a reproducible benchmarking platform with community contributions from academia and industry. It compares 86 open-source and proprietary systems across 12 datasets, with English short- and long-form and multilingual short-form tracks. We standardize word error rate (WER) and inverse real-time factor (RTFx) evaluation for consistent accuracy-efficiency comparisons across model architectures and toolkits (e.g., ESPNet, NeMo, SpeechBrain, Transformers). We observe that Conformer-based encoders paired with transformer-based decoders achieve the best average WER, while connectionist temporal classification (CTC) and token-and-duration transducer (TDT) decoders offer superior RTFx, making them better suited for long-form and batched processing. All code and dataset loaders are open-sourced to support transparent, extensible evaluation. We present our evaluation methodology to facilitate community-driven benchmarking in ASR and other tasks.