# Investigation of Whisper ASR Hallucinations Induced by Non-Speech Audio

arXiv: https://arxiv.org/abs/2501.11378 · HTML: https://mlsanity.com/paper/2501.11378/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 25 references that are in the map.
- First posted on arXiv 20 Jan 2025.

## Builds on (most cited first, 15 of 15)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [MUSAN: A Music, Speech, and Noise Corpus](https://mlsanity.com/paper/1510.08484/) (2015)
- [FSD50K: An Open Dataset of Human-Labeled Sound Events](https://mlsanity.com/paper/2010.00475/) (2020)
- [Analyzing and Mitigating Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2310.00754/) (2023)
- [The Curious Case of Hallucinations in Neural Machine Translation](https://mlsanity.com/paper/2104.06683/) (2021)
- [Large Language Models can Strategically Deceive their Users when Put Under Pressure](https://mlsanity.com/paper/2311.07590/) (2023)
- [DCCRN: Deep Complex Convolution Recurrent Network for Phase-Aware Speech Enhancement](https://mlsanity.com/paper/2008.00264/) (2020)
- [A Survey of Resource-efficient LLM and Multimodal Foundation Models](https://mlsanity.com/paper/2401.08092/) (2024)
- [Careless Whisper: Speech-to-Text Hallucination Harms](https://mlsanity.com/paper/2402.08021/) (2024)
- [Confidence Estimation for Attention-based Sequence-to-sequence Models for Speech Recognition](https://mlsanity.com/paper/2010.11428/) (2020)
- [Hallucinations in Neural Automatic Speech Recognition: Identifying Errors and Hallucinatory Models](https://mlsanity.com/paper/2401.01572/) (2024)
- [Hallucination of speech recognition errors with sequence to sequence learning](https://mlsanity.com/paper/2103.12258/) (2021)

## Built on by (most cited first, 30 of 31)

- [Lost in Transcription, Found in Distribution Shift: Demystifying Hallucination in Speech Foundation Models](https://mlsanity.com/paper/2502.12414/) (2025)
- [Calm-Whisper: Reduce Whisper Hallucination On Non-Speech By Calming Crazy Heads Down](https://mlsanity.com/paper/2505.12969/) (2025)
- [Hallucination Benchmark for Speech Foundation Models](https://mlsanity.com/paper/2510.16567/) (2025)
- [AudioSAE: Towards Understanding of Audio-Processing Models with Sparse AutoEncoders](https://mlsanity.com/paper/2602.05027/) (2026)
- [MECAT: A Multi-Experts Constructed Benchmark for Fine-Grained Audio Understanding Tasks](https://mlsanity.com/paper/2507.23511/) (2025)
- [Swivuriso: The South African Next Voices Multilingual Speech Dataset](https://mlsanity.com/paper/2512.02201/) (2025)
- [Whisper-CD: Accurate Long-Form Speech Recognition using Multi-Negative Contrastive Decoding](https://mlsanity.com/paper/2603.06193/) (2026)
- [Quantization for OpenAI's Whisper Models: A Comparative Analysis](https://mlsanity.com/paper/2503.09905/) (2025)
- [PSRB: A Comprehensive Benchmark for Evaluating Persian ASR Systems](https://mlsanity.com/paper/2505.21230/) (2025)
- [CAM~OES: A Comprehensive Automatic Speech Recognition Benchmark for European Portuguese](https://mlsanity.com/paper/2508.19721/) (2025)
- [From Dispersion to Attraction: Spectral Dynamics of Hallucination Across Whisper Model Scales](https://mlsanity.com/paper/2604.08591/) (2026)
- [Identifying and typifying demographic unfairness in phoneme-level embeddings of self-supervised speech recognition models](https://mlsanity.com/paper/2604.22631/) (2026)
- [Whisper Hallucination Detection and Mitigation via Hidden Representation Steering and Sparse AutoEncoders](https://mlsanity.com/paper/2606.07473/) (2026)
- [Improving French Synthetic Speech Quality via SSML Prosody Control](https://mlsanity.com/paper/2508.17494/) (2025)
- [Exploring the potential and limitations of Model Merging for Multi-Domain Adaptation in ASR](https://mlsanity.com/paper/2603.05354/) (2026)
- [ADAPTS: Agentic Decomposition for Automated Protocol-agnostic Tracking of Symptoms](https://mlsanity.com/paper/2605.03212/) (2026)
- [WAXAL-NET: Finetuned Edge ASR Across 19 African Languages](https://mlsanity.com/paper/2606.02375/) (2026)
- [From Text Metrics to Model Internals: A Study of Whisper ASR Hallucination Detection](https://mlsanity.com/paper/2606.23060/) (2026)
- [SPEARBench: A Benchmark for Naturalness Evaluation in Streaming Speech-to-Speech Language Models](https://mlsanity.com/paper/2607.05365/) (2026)
- [Out of the Box, into the Clinic? Evaluating State-of-the-Art ASR for Clinical Applications for Older Adults](https://mlsanity.com/paper/2508.08684/) (2025)
- [Scalable Music Cover Retrieval Using Lyrics-Aligned Audio Embeddings](https://mlsanity.com/paper/2601.11262/) (2026)
- [WhisperAlign: Word-Boundary-Aware ASR and WhisperX-Anchored Pyannote Diarization for Long-Form Bengali Speech](https://mlsanity.com/paper/2603.04809/) (2026)
- [Tagarela - A Portuguese speech dataset from podcasts](https://mlsanity.com/paper/2603.15326/) (2026)
- [Sommelier: Scalable Open Multi-turn Audio Pre-processing for Full-duplex Speech Language Models](https://mlsanity.com/paper/2603.25750/) (2026)
- [HALAS: A Human-Annotated Dataset of Hallucinations of Modern ASR Systems](https://mlsanity.com/paper/2606.23048/) (2026)
- [Development and Feasibility Evaluation of an Edge AI as Medical Device System for Breast Cancer Multidisciplinary Team Meetings](https://mlsanity.com/paper/2608.22108/) (2026)
- [SURE-Voice: A Front-End Baseline for Speech-Evidence Filtering in Speech LLMs](https://mlsanity.com/paper/2608.27783/) (2026)
- [Reducing Hallucinated Transcripts in Whisper via Hallucination Space Projection](https://mlsanity.com/paper/2609.04561/) (2026)
- [AURA: Uncertainty-Routed Activation Editing for Acoustic Grounding in Speech Foundation Models](https://mlsanity.com/paper/2609.23979/) (2026)
- [Quieter Than the Room: Representation Drift and Task Robustness in Speech Encoders](https://mlsanity.com/paper/2609.27195/) (2026)

## Abstract (arXiv)

Hallucinations of deep neural models are amongst key challenges in automatic speech recognition (ASR). In this paper, we investigate hallucinations of the Whisper ASR model induced by non-speech audio segments present during inference. By inducting hallucinations with various types of sounds, we show that there exists a set of hallucinations that appear frequently. We then study hallucinations caused by the augmentation of speech with such sounds. Finally, we describe the creation of a bag of hallucinations (BoH) that allows to remove the effect of hallucinations through the post-processing of text transcriptions. The results of our experiments show that such post-processing is capable of reducing word error rate (WER) and acts as a good safeguard against problematic hallucinations.