mlsanityData to 14 Oct 2026

Evaluation of LLMs in Speech is Often Flawed: Test Set Contamination in Large Language Models for Speech Recognition

arXiv 2505.22251 ↗First posted 28 May 2025eess.AS cs.CLcited by 9

Not on the map. Why?

Abstract (arXiv)

Recent work suggests that large language models (LLMs) can improve performance of speech tasks compared to existing systems. To support their claims, results on LibriSpeech and Common Voice are often quoted. However, this work finds that a substantial amount of the LibriSpeech and Common Voice evaluation sets appear in public LLM pretraining corpora. This calls into question the reliability of findings drawn from these two datasets. To measure contamination impact, LLMs trained with/without contamination are compared. A contaminated LLM is more likely to generate test sentences it has seen during training. Then, speech recognisers based on LLMs are compared. They show only subtle error rate differences if the LLM is contaminated, but assign significantly higher probabilities to transcriptions seen during LLM training. Results show that LLM outputs can be biased by tiny amounts of data contamination, highlighting the importance of evaluating LLM-based speech systems with held-out data.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 33 of its 43 references that are in the map. First posted on arXiv 28 May 2025.

Builds on

33 of 43 references.

References · 33

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  5. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  6. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  7. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  8. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  9. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  10. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 864
  11. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  12. 2024OLMo: Accelerating the Science of Language Modelscited by 613
21 more
  1. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 483
  2. 2021SpeechBrain: A General-Purpose Speech Toolkitcited by 406
  3. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 348
  4. 2022Data Contamination: From Memorization to Exploitationcited by 236
  5. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 229
  6. 2024RedPajama: an Open Dataset for Training Large Language Modelscited by 182
  7. 2023The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Datasetcited by 148
  8. 2023On decoder-only architecture for speech-to-text and large language model integrationcited by 145
  9. 2023Prompting Large Language Models with Speech Recognition Abilitiescited by 140
  10. 2024WavLLM: Towards Robust and Adaptive Speech Large Language Modelcited by 109
  11. 2024An Embarrassingly Simple Approach for LLM with Strong ASR Capacitycited by 88
  12. 2023Connecting Speech Encoder and Large Language Model for ASRcited by 80
  13. 2023Can Generative Large Language Models Perform ASR Error Correction?cited by 57
  14. 2024Open-Source Conversational AI with SpeechBrain 1.0cited by 56
  15. 2023COSMIC: Data Efficient Instruction-tuning For Speech In-Context Learningcited by 31
  16. 2024CTC-Assisted LLM-Based Contextual ASRcited by 19
  17. 2024Comparing Discrete and Continuous Space LLMs for Speech Recognitioncited by 15
  18. 2025LLM360 K2: Building a 65B 360-Open-Source Large Language Model from Scratchcited by 15
  19. 2024Efficient Streaming LLM for Speech Recognitioncited by 12
  20. 2024SALSA: Speedy ASR-LLM Synchronous Aggregationcited by 8
  21. 2025Delayed Fusion: Integrating Large Language Models into First-Pass Decoding in End-to-end Speech Recognitioncited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 7.

9 papers.

  1. 2025Open ASR Leaderboard: Towards Reproducible and Transparent Multilingual and Long-Form Speech Recognition Evaluationcited by 23
  2. 2026AQUA-Bench: Beyond Finding Answers to Knowing When There Are None in Audio Question Answeringcited by 6
  3. 2026Walking Through Uncertainty: An Empirical Study of Uncertainty Estimation for Audio-Aware Large Language Modelscited by 4
  4. 2025Data-Centric Lessons To Improve Speech-Language Pretrainingcited by 3
  5. 2026Subtitle-Aligned Fine-Tuning of Whisper for Swiss German ASR: Benchmark Contamination, Convention Mismatch, and an Honest Baseline at 25.6% WER (13.8% cWER)cited by 1
  6. 2026Modeling Overlapped Speech with Shufflescited by 0
  7. 2026Who Spoke What When? Evaluating Spoken Language Models for Conversational ASR with Semantic and Overlap-Aware Metricscited by 0
  8. 2026Towards Quantifying Benchmark Optimization in ASR Modelscited by 0
  9. 2026Personalized Korean Lipreading as Visual Speech Recognition: Transfer, Census and Adaptation on OLKAVScited by 0