# Lost in Transcription, Found in Distribution Shift: Demystifying Hallucination in Speech Foundation Models

arXiv: https://arxiv.org/abs/2502.12414 · HTML: https://mlsanity.com/paper/2502.12414/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 78 references that are in the map.
- First posted on arXiv 18 Feb 2025.

## Builds on (most cited first, 30 of 42)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs](https://mlsanity.com/paper/2306.13063/) (2023)
- [Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback](https://mlsanity.com/paper/2305.14975/) (2023)
- [Prometheus: Inducing Fine-grained Evaluation Capability in Language Models](https://mlsanity.com/paper/2310.08491/) (2023)
- [Towards Out-Of-Distribution Generalization: A Survey](https://mlsanity.com/paper/2108.13624/) (2021)
- [Hallucination of Multimodal Large Language Models: A Survey](https://mlsanity.com/paper/2404.18930/) (2024)
- [ToTTo: A Controlled Table-To-Text Generation Dataset](https://mlsanity.com/paper/2004.14373/) (2020)
- [Central Moment Discrepancy (CMD) for Domain-Invariant Representation Learning](https://mlsanity.com/paper/1702.08811/) (2017)
- [LLM Lies: Hallucinations are not Bugs, but Features as Adversarial Examples](https://mlsanity.com/paper/2310.01469/) (2023)
- [Fast Conformer with Linearly Scalable Attention for Efficient Speech Recognition](https://mlsanity.com/paper/2305.05084/) (2023)
- [Do I Know This Entity? Knowledge Awareness and Hallucinations in Language Models](https://mlsanity.com/paper/2411.14257/) (2024)
- [SONAR: Sentence-Level Multimodal and Language-Agnostic Representations](https://mlsanity.com/paper/2308.11466/) (2023)
- [A Comprehensive Survey of Hallucination in Large Language, Image, Video and Audio Foundation Models](https://mlsanity.com/paper/2405.09589/) (2024)
- [Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo Labelling](https://mlsanity.com/paper/2311.00430/) (2023)
- [Controlled Hallucinations: Learning to Generate Faithfully from Noisy Data](https://mlsanity.com/paper/2010.05873/) (2020)
- [On Verbalized Confidence Scores for LLMs](https://mlsanity.com/paper/2412.14737/) (2024)
- [SLUE: New Benchmark Tasks for Spoken Language Understanding Evaluation on Natural Speech](https://mlsanity.com/paper/2111.10367/) (2021)
- [Earnings-21: A Practical Benchmark for ASR in the Wild](https://mlsanity.com/paper/2104.11348/) (2021)
- [Efficient Sequence Transduction by Jointly Predicting Tokens and Durations](https://mlsanity.com/paper/2304.06795/) (2023)
- [Careless Whisper: Speech-to-Text Hallucination Harms](https://mlsanity.com/paper/2402.08021/) (2024)
- [A Survey on Large Language Model Hallucination via a Creativity Perspective](https://mlsanity.com/paper/2402.06647/) (2024)
- [Investigation of Whisper ASR Hallucinations Induced by Non-Speech Audio](https://mlsanity.com/paper/2501.11378/) (2025)
- [GLOBE: A High-quality English Corpus with Global Accents for Zero-shot Speaker Adaptive Text-to-Speech](https://mlsanity.com/paper/2406.14875/) (2024)
- [Hallucinations in Neural Automatic Speech Recognition: Identifying Errors and Hallucinatory Models](https://mlsanity.com/paper/2401.01572/) (2024)
- [Domain Divergences: a Survey and Empirical Analysis](https://mlsanity.com/paper/2010.12198/) (2020)
- [N-Shot Benchmarking of Whisper on Diverse Arabic Speech Recognition](https://mlsanity.com/paper/2306.02902/) (2023)
- [Lost in Transcription: Identifying and Quantifying the Accuracy Biases of Automatic Speech Recognition Systems Against Disfluent Speech](https://mlsanity.com/paper/2405.06150/) (2024)

## Built on by (most cited first, 18 of 18)

- [Hallucination Benchmark for Speech Foundation Models](https://mlsanity.com/paper/2510.16567/) (2025)
- [Do LLM Decoders Listen Fairly? Benchmarking How Language Model Priors Shape Bias in Speech Recognition](https://mlsanity.com/paper/2604.21276/) (2026)
- [Whisper-CD: Accurate Long-Form Speech Recognition using Multi-Negative Contrastive Decoding](https://mlsanity.com/paper/2603.06193/) (2026)
- [PSRB: A Comprehensive Benchmark for Evaluating Persian ASR Systems](https://mlsanity.com/paper/2505.21230/) (2025)
- [ArtiFree: Detecting and Reducing Generative Artifacts in Diffusion-based Speech Enhancement](https://mlsanity.com/paper/2509.19495/) (2025)
- [From Dispersion to Attraction: Spectral Dynamics of Hallucination Across Whisper Model Scales](https://mlsanity.com/paper/2604.08591/) (2026)
- [Said Aloud, Read Different: Cross-Modal Instability in Multimodal Models](https://mlsanity.com/paper/2608.27135/) (2026)
- [WAXAL-NET: Finetuned Edge ASR Across 19 African Languages](https://mlsanity.com/paper/2606.02375/) (2026)
- [From Text Metrics to Model Internals: A Study of Whisper ASR Hallucination Detection](https://mlsanity.com/paper/2606.23060/) (2026)
- [SPEARBench: A Benchmark for Naturalness Evaluation in Streaming Speech-to-Speech Language Models](https://mlsanity.com/paper/2607.05365/) (2026)
- [Lost in Speech: Trilingual Spoken Hallucination Detection Across Audio and Transcripts](https://mlsanity.com/paper/2608.24707/) (2026)
- [Out of the Box, into the Clinic? Evaluating State-of-the-Art ASR for Clinical Applications for Older Adults](https://mlsanity.com/paper/2508.08684/) (2025)
- [Back to Basics: Revisiting ASR in the Age of Voice Agents](https://mlsanity.com/paper/2603.25727/) (2026)
- [Detecting Hallucinations in SpeechLLMs at Inference Time Using Attention Maps](https://mlsanity.com/paper/2604.19565/) (2026)
- [ERRORQUAKE: Heavy-Tailed Error Severity Distributions in Open-Weight Large Language Models](https://mlsanity.com/paper/2606.05170/) (2026)
- [HALAS: A Human-Annotated Dataset of Hallucinations of Modern ASR Systems](https://mlsanity.com/paper/2606.23048/) (2026)
- [The Anatomy of an ASR Hallucination](https://mlsanity.com/paper/2609.04404/) (2026)
- [AURA: Uncertainty-Routed Activation Editing for Acoustic Grounding in Speech Foundation Models](https://mlsanity.com/paper/2609.23979/) (2026)

## Abstract (arXiv)

Speech foundation models trained at a massive scale, both in terms of model and data size, result in robust systems capable of performing multiple speech tasks, including automatic speech recognition (ASR). These models transcend language and domain barriers, yet effectively measuring their performance remains a challenge. Traditional metrics like word error rate (WER) and character error rate (CER) are commonly used to evaluate ASR performance but often fail to reflect transcription quality in critical contexts, particularly when detecting fabricated outputs. This phenomenon, known as hallucination, is especially concerning in high-stakes domains such as healthcare, legal, and aviation, where errors can have severe consequences. In our work, we address this gap by investigating hallucination in ASR models. We examine how factors such as distribution shifts, model size, and model architecture influence the hallucination error rate (HER), a metric we introduce to quantify hallucinations. Our analysis of over 20 ASR models reveals numinsights~key insights: (1) High WERs can mask low hallucination rates, while low WERs may conceal dangerous hallucinations. (2) Synthetic noise, both adversarial and common perturbations like white noise, pitch shift, and time stretching, increase HER. (3) Distribution shift correlates strongly with HER (α = 0.91). Our findings highlight the importance of incorporating HER alongside traditional metrics like WER to better assess ASR model performance, particularly in high-stakes domains.