mlsanityData to 7 Oct 2026

Investigation of Whisper ASR Hallucinations Induced by Non-Speech Audio

arXiv 2501.11378 ↗First posted 20 Jan 2025cs.SD cs.AI eess.AScited by 31

Not on the map. Why?

Abstract (arXiv)

Hallucinations of deep neural models are amongst key challenges in automatic speech recognition (ASR). In this paper, we investigate hallucinations of the Whisper ASR model induced by non-speech audio segments present during inference. By inducting hallucinations with various types of sounds, we show that there exists a set of hallucinations that appear frequently. We then study hallucinations caused by the augmentation of speech with such sounds. Finally, we describe the creation of a bag of hallucinations (BoH) that allows to remove the effect of hallucinations through the post-processing of text transcriptions. The results of our experiments show that such post-processing is capable of reducing word error rate (WER) and acts as a good safeguard against problematic hallucinations.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Builds on 15 of its 25 references that are in the map. First posted on arXiv 20 Jan 2025.

Builds on

15 of 25 references.

References · 15

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  3. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  4. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  5. 2015MUSAN: A Music, Speech, and Noise Corpuscited by 400
  6. 2020FSD50K: An Open Dataset of Human-Labeled Sound Eventscited by 339
  7. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Modelscited by 314
  8. 2021The Curious Case of Hallucinations in Neural Machine Translationcited by 161
  9. 2020DCCRN: Deep Complex Convolution Recurrent Network for Phase-Aware Speech Enhancementcited by 133
  10. 2023Large Language Models can Strategically Deceive their Users when Put Under Pressurecited by 132
  11. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
  12. 2024Careless Whisper: Speech-to-Text Hallucination Harmscited by 53
3 more
  1. 2020Confidence Estimation for Attention-based Sequence-to-sequence Models for Speech Recognitioncited by 35
  2. 2024Hallucinations in Neural Automatic Speech Recognition: Identifying Errors and Hallucinatory Modelscited by 26
  3. 2021Hallucination of speech recognition errors with sequence to sequence learningcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 10; 2026: 21.

31 papers.

Sort:

By area:

  1. 2025Lost in Transcription, Found in Distribution Shift: Demystifying Hallucination in Speech Foundation Modelscited by 17
  2. 2025Calm-Whisper: Reduce Whisper Hallucination On Non-Speech By Calming Crazy Heads Downcited by 12
  3. 2026AudioSAE: Towards Understanding of Audio-Processing Models with Sparse AutoEncoderscited by 8
  4. 2025Hallucination Benchmark for Speech Foundation Modelscited by 7
  5. 2025MECAT: A Multi-Experts Constructed Benchmark for Fine-Grained Audio Understanding Taskscited by 6
  6. 2025Swivuriso: The South African Next Voices Multilingual Speech Datasetcited by 3
  7. 2026Whisper-CD: Accurate Long-Form Speech Recognition using Multi-Negative Contrastive Decodingcited by 3
  8. 2025Quantization for OpenAI's Whisper Models: A Comparative Analysiscited by 2
  9. 2025PSRB: A Comprehensive Benchmark for Evaluating Persian ASR Systemscited by 2
  10. 2025CAM~OES: A Comprehensive Automatic Speech Recognition Benchmark for European Portuguesecited by 2
  11. 2026From Dispersion to Attraction: Spectral Dynamics of Hallucination Across Whisper Model Scalescited by 2
  12. 2026Whisper Hallucination Detection and Mitigation via Hidden Representation Steering and Sparse AutoEncoderscited by 2
19 more
  1. 2025Improving French Synthetic Speech Quality via SSML Prosody Controlcited by 1
  2. 2026Identifying and typifying demographic unfairness in phoneme-level embeddings of self-supervised speech recognition modelscited by 1
  3. 2026ADAPTS: Agentic Decomposition for Automated Protocol-agnostic Tracking of Symptomscited by 1
  4. 2026From Text Metrics to Model Internals: A Study of Whisper ASR Hallucination Detectioncited by 1
  5. 2026SPEARBench: A Benchmark for Naturalness Evaluation in Streaming Speech-to-Speech Language Modelscited by 1
  6. 2025Out of the Box, into the Clinic? Evaluating State-of-the-Art ASR for Clinical Applications for Older Adultscited by 0
  7. 2026Scalable Music Cover Retrieval Using Lyrics-Aligned Audio Embeddingscited by 0
  8. 2026WhisperAlign: Word-Boundary-Aware ASR and WhisperX-Anchored Pyannote Diarization for Long-Form Bengali Speechcited by 0
  9. 2026Exploring the potential and limitations of Model Merging for Multi-Domain Adaptation in ASRcited by 0
  10. 2026Tagarela - A Portuguese speech dataset from podcastscited by 0
  11. 2026Sommelier: Scalable Open Multi-turn Audio Pre-processing for Full-duplex Speech Language Modelscited by 0
  12. 2026WAXAL-NET: Finetuned Edge ASR Across 19 African Languagescited by 0
  13. 2026HALAS: A Human-Annotated Dataset of Hallucinations of Modern ASR Systemscited by 0
  14. 2026Development and Feasibility Evaluation of an Edge AI as Medical Device System for Breast Cancer Multidisciplinary Team Meetingscited by 0
  15. 2026SURE-Voice: A Front-End Baseline for Speech-Evidence Filtering in Speech LLMscited by 0
  16. 2026Reducing Hallucinated Transcripts in Whisper via Hallucination Space Projectioncited by 0
  17. 2026AURA: Uncertainty-Routed Activation Editing for Acoustic Grounding in Speech Foundation Modelscited by 0
  18. 2026Quieter Than the Room: Representation Drift and Task Robustness in Speech Encoderscited by 0
  19. 2026How to Reduce Whisper Hallucinationcited by 0