mlsanityData to 14 Oct 2026

Open ASR Leaderboard: Towards Reproducible and Transparent Multilingual and Long-Form Speech Recognition Evaluation

arXiv 2510.06961 ↗First posted 8 Oct 2025cs.CL cs.AI cs.SDcited by 23

Not on the map. Why?

Abstract (arXiv)

We present the Open ASR Leaderboard, a reproducible benchmarking platform with community contributions from academia and industry. It compares 86 open-source and proprietary systems across 12 datasets, with English short- and long-form and multilingual short-form tracks. We standardize word error rate (WER) and inverse real-time factor (RTFx) evaluation for consistent accuracy-efficiency comparisons across model architectures and toolkits (e.g., ESPNet, NeMo, SpeechBrain, Transformers). We observe that Conformer-based encoders paired with transformer-based decoders achieve the best average WER, while connectionist temporal classification (CTC) and token-and-duration transducer (TDT) decoders offer superior RTFx, making them better suited for long-form and batched processing. All code and dataset loaders are open-sourced to support transparent, extensible evaluation. We present our evaluation methodology to facilitate community-driven benchmarking in ASR and other tasks.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Builds on 30 of its 38 references that are in the map. First posted on arXiv 8 Oct 2025.

Builds on

30 of 38 references.

References · 30

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  4. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  5. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  6. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  7. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 558
  8. 2021Datasets: A Community Library for Natural Language Processingcited by 530
  9. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 476
  10. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 469
  11. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 463
  12. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 399
18 more
  1. 2019NeMo: a toolkit for building AI applications using Neural Modulescited by 235
  2. 2018TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptationcited by 226
  3. 2023Fast Conformer with Linearly Scalable Attention for Efficient Speech Recognitioncited by 124
  4. 2023Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo Labellingcited by 90
  5. 2021SPGISpeech: 5,000 hours of transcribed financial audio for fully formatted end-to-end speech recognitioncited by 78
  6. 2020CoVoST 2 and Massively Multilingual Speech-to-Text Translationcited by 74
  7. 2021Earnings-21: A Practical Benchmark for ASR in the Wildcited by 62
  8. 2023Efficient Sequence Transduction by Jointly Predicting Tokens and Durationscited by 58
  9. 2024Open-Source Conversational AI with SpeechBrain 1.0cited by 56
  10. 2022Earnings-22: A Practical Benchmark for Accents in the Wildcited by 50
  11. 2024CrisperWhisper: Accurate Timestamps on Verbatim Speech Transcriptionscited by 34
  12. 2025OWSM v4: Improving Open Whisper-Style Speech Models via Data Scaling and Cleaningcited by 27
  13. 2025Streaming Sequence-to-Sequence Learning with Delayed Streams Modelingcited by 27
  14. 2024Moonshine: Speech Recognition for Live Transcription and Voice Commandscited by 19
  15. 2023Investigating End-to-End ASR Architectures for Long Form Audio Transcriptioncited by 11
  16. 2025Evaluation of LLMs in Speech is Often Flawed: Test Set Contamination in Large Language Models for Speech Recognitioncited by 9
  17. 2025LiteASR: Efficient Automatic Speech Recognition with Low-Rank Approximationcited by 7
  18. 2024SpeechColab Leaderboard: An Open-Source Platform for Automatic Speech Recognition Evaluationcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 23.

23 papers.

Sort:

By area:

  1. 2026Nemotron 3 Nano Omni: Efficient and Open Multimodal Intelligencecited by 30
  2. 2026NemotronLabs VoiceChat: An Open Full-duplex Speech-to-Speech Model with Tool Calling Capabilitiescited by 4
  3. 2026RW-Voice-EQ Bench: A Real World Benchmark for Evaluating Voice AI Systemscited by 2
  4. 2026A frontend-backend architecture for tool calls in full-duplex speech modelscited by 2
  5. 2026Exploring the potential and limitations of Model Merging for Multi-Domain Adaptation in ASRcited by 1
  6. 2026Reducing the Offline-Streaming Gap for Unified ASR Transducer with Consistency Regularizationcited by 1
  7. 2026FastSLM: Hierarchical Temporal Abstraction for Efficient Long-Form Speech Adaptationcited by 0
  8. 2026AppTek Call-Center Dialogues: A Multi-Accent Long-Form Benchmark for English ASRcited by 0
  9. 2026FSA-GRPO: Teaching Auditory LLMs to Use Few-Shot Demonstrationscited by 0
  10. 2026TRADE: Transducer-Augmented Decoder for Speech LLMcited by 0
  11. 2026MLLP-VRAIN UPV system for the IWSLT 2026 Simultaneous Speech Translation taskcited by 0
  12. 2026HALAS: A Human-Annotated Dataset of Hallucinations of Modern ASR Systemscited by 0
11 more
  1. 2026Position: AI Leaderboards Are Underserving the Global South: A Case Study from Indiacited by 0
  2. 2026Towards Quantifying Benchmark Optimization in ASR Modelscited by 0
  3. 2026Domain-Adaptive ASR for Telephony AI Agents: Fine-tuning Canary Flash Models for Enterprise Contact Center Applicationscited by 0
  4. 2026Formal, Executable and Explainable Runtime Monitoring of Spoken Air Traffic Control Operational Procedurescited by 0
  5. 2026Listen to the Latents: Self-Correcting Speech Recognition in Large Audio Language Models Through Hidden-State Interactionscited by 0
  6. 2026MedWER: A Reproducible, Model-Free Evaluation Protocol for Medical Speech Recognitioncited by 0
  7. 2026Whisper-Based Speech Transcription from Videos Across Multiple Languages for Cross-Cultural Understandingcited by 0
  8. 2026Look Less, Hear Better: Jointly Rewarded GRPO for Streaming ASRcited by 0
  9. 2026In-Context Adaptation of Encoder-Decoder Models in Speech Recognitioncited by 0
  10. 2026Benchmarking Automatic Speech Recognition Tools for Iberian Languagescited by 0
  11. 2026CAFE: Confining Feature Distillation to Language-Routed Adapters for Compact Multilingual Automatic Speech Recognitioncited by 0