# Hallucinations in Neural Automatic Speech Recognition: Identifying Errors and Hallucinatory Models

arXiv: https://arxiv.org/abs/2401.01572 · HTML: https://mlsanity.com/paper/2401.01572/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 27 references that are in the map.
- First posted on arXiv 3 Jan 2024.

## Builds on (most cited first, 11 of 11)

- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Does Learning Require Memorization? A Short Tale about a Long Tail](https://mlsanity.com/paper/1906.05271/) (2019)
- [Six Challenges for Neural Machine Translation](https://mlsanity.com/paper/1706.03872/) (2017)
- [Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labels](https://mlsanity.com/paper/2303.14307/) (2023)
- [How Bad Are Artifacts?: Analyzing the Impact of Speech Enhancement Errors on ASR](https://mlsanity.com/paper/2201.06685/) (2022)
- [Hallucination of speech recognition errors with sequence to sequence learning](https://mlsanity.com/paper/2103.12258/) (2021)
- [Learning from Past Mistakes: Improving Automatic Speech Recognition Output via Noisy-Clean Phrase Context Modeling](https://mlsanity.com/paper/1802.02607/) (2018)
- [Topic Model Robustness to Automatic Speech Recognition Errors in Podcast Transcripts](https://mlsanity.com/paper/2109.12306/) (2021)

## Built on by (most cited first, 27 of 27)

- [Careless Whisper: Speech-to-Text Hallucination Harms](https://mlsanity.com/paper/2402.08021/) (2024)
- [Investigation of Whisper ASR Hallucinations Induced by Non-Speech Audio](https://mlsanity.com/paper/2501.11378/) (2025)
- [Lost in Transcription, Found in Distribution Shift: Demystifying Hallucination in Speech Foundation Models](https://mlsanity.com/paper/2502.12414/) (2025)
- [Beyond Transcription: Mechanistic Interpretability in ASR](https://mlsanity.com/paper/2508.15882/) (2025)
- [Calm-Whisper: Reduce Whisper Hallucination On Non-Speech By Calming Crazy Heads Down](https://mlsanity.com/paper/2505.12969/) (2025)
- [Estimating the Completeness of Discrete Speech Units](https://mlsanity.com/paper/2409.06109/) (2024)
- [Hallucination Benchmark for Speech Foundation Models](https://mlsanity.com/paper/2510.16567/) (2025)
- [HalluAudio: A Comprehensive Benchmark for Hallucination Detection in Large Audio-Language Models](https://mlsanity.com/paper/2604.19300/) (2026)
- [AHA: Aligning Large Audio-Language Models for Reasoning Hallucinations via Counterfactual Hard Negatives](https://mlsanity.com/paper/2512.24052/) (2025)
- [Do LLM Decoders Listen Fairly? Benchmarking How Language Model Priors Shape Bias in Speech Recognition](https://mlsanity.com/paper/2604.21276/) (2026)
- [Addressing Pitfalls in Auditing Practices of Automatic Speech Recognition Technologies: A Case Study of People with Aphasia](https://mlsanity.com/paper/2506.08846/) (2025)
- [PSRB: A Comprehensive Benchmark for Evaluating Persian ASR Systems](https://mlsanity.com/paper/2505.21230/) (2025)
- [SpeechIQ: Speech-Agentic Intelligence Quotient Across Cognitive Levels in Voice Understanding by Large Language Models](https://mlsanity.com/paper/2507.19361/) (2025)
- [Supplementary Resources and Analysis for Automatic Speech Recognition Systems Trained on the Loquacious Dataset](https://mlsanity.com/paper/2512.17915/) (2025)
- [Identifying and typifying demographic unfairness in phoneme-level embeddings of self-supervised speech recognition models](https://mlsanity.com/paper/2604.22631/) (2026)
- [Whisper Hallucination Detection and Mitigation via Hidden Representation Steering and Sparse AutoEncoders](https://mlsanity.com/paper/2606.07473/) (2026)
- [Said Aloud, Read Different: Cross-Modal Instability in Multimodal Models](https://mlsanity.com/paper/2608.27135/) (2026)
- [LearnerVoice: A Dataset of Non-Native English Learners' Spontaneous Speech](https://mlsanity.com/paper/2407.04280/) (2024)
- [Survey of Design Paradigms for Social Robots](https://mlsanity.com/paper/2407.20556/) (2024)
- [From Text Metrics to Model Internals: A Study of Whisper ASR Hallucination Detection](https://mlsanity.com/paper/2606.23060/) (2026)
- [Lost in Speech: Trilingual Spoken Hallucination Detection Across Audio and Transcripts](https://mlsanity.com/paper/2608.24707/) (2026)
- [Whispy: Adapting STT Whisper Models to Real-Time Environments](https://mlsanity.com/paper/2405.03484/) (2024)
- [Scalable Music Cover Retrieval Using Lyrics-Aligned Audio Embeddings](https://mlsanity.com/paper/2601.11262/) (2026)
- [When Demonstrations Fail: Diagnosing the Limits of In-Context Learning in Large Audio-Language Models with Progressive Cue Removal](https://mlsanity.com/paper/2603.20433/) (2026)
- [Back to Basics: Revisiting ASR in the Age of Voice Agents](https://mlsanity.com/paper/2603.25727/) (2026)
- [Detecting Hallucinations in SpeechLLMs at Inference Time Using Attention Maps](https://mlsanity.com/paper/2604.19565/) (2026)
- [HALAS: A Human-Annotated Dataset of Hallucinations of Modern ASR Systems](https://mlsanity.com/paper/2606.23048/) (2026)

## Abstract (arXiv)

Hallucinations are a type of output error produced by deep neural networks. While this has been studied in natural language processing, they have not been researched previously in automatic speech recognition. Here, we define hallucinations in ASR as transcriptions generated by a model that are semantically unrelated to the source utterance, yet still fluent and coherent. The similarity of hallucinations to probable natural language outputs of the model creates a danger of deception and impacts the credibility of the system. We show that commonly used metrics, such as word error rates, cannot differentiate between hallucinatory and non-hallucinatory models. To address this, we propose a perturbation-based method for assessing the susceptibility of an automatic speech recognition (ASR) model to hallucination at test time, which does not require access to the training dataset. We demonstrate that this method helps to distinguish between hallucinatory and non-hallucinatory models that have similar baseline word error rates. We further explore the relationship between the types of ASR errors and the types of dataset noise to determine what types of noise are most likely to create hallucinatory outputs. We devise a framework for identifying hallucinations by analysing their semantic connection with the ground truth and their fluency. Finally, we discover how to induce hallucinations with a random noise injection to the utterance.