mlsanityData to 14 Oct 2026

Calm-Whisper: Reduce Whisper Hallucination On Non-Speech By Calming Crazy Heads Down

arXiv 2505.12969 ↗First posted 19 May 2025cs.CLcited by 12

Not on the map. Why?

Abstract (arXiv)

OpenAI's Whisper has achieved significant success in Automatic Speech Recognition. However, it has consistently been found to exhibit hallucination issues, particularly in non-speech segments, which limits its broader application in complex industrial settings. In this paper, we introduce a novel method to reduce Whisper's hallucination on non-speech segments without using any pre- or post-possessing techniques. Specifically, we benchmark the contribution of each self-attentional head in the Whisper-large-v3 decoder to the hallucination problem by performing a head-wise mask. Our findings reveal that only 3 of the 20 heads account for over 75% of the hallucinations on the UrbanSound dataset. We then fine-tune these three crazy heads using a collection of non-speech data. The results show that our best fine-tuned model, namely Calm-Whisper, achieves over 80% reduction in non-speech hallucination with only less than 0.1% WER degradation on LibriSpeech test-clean and test-other.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 12 of its 23 references that are in the map. First posted on arXiv 19 May 2025.

Builds on

12 of 23 references.

References · 12

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  2. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  3. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  4. 2015MUSAN: A Music, Speech, and Noise Corpuscited by 425
  5. 2023WhisperX: Time-Accurate Speech Transcription of Long-Form Audiocited by 326
  6. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Modelscited by 316
  7. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 232
  8. 2021The Curious Case of Hallucinations in Neural Machine Translationcited by 161
  9. 2023Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo Labellingcited by 90
  10. 2024Careless Whisper: Speech-to-Text Hallucination Harmscited by 53
  11. 2025Investigation of Whisper ASR Hallucinations Induced by Non-Speech Audiocited by 31
  12. 2024Hallucinations in Neural Automatic Speech Recognition: Identifying Errors and Hallucinatory Modelscited by 27

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 12.

12 papers.

  1. 2026Whisper Hallucination Detection and Mitigation via Hidden Representation Steering and Sparse AutoEncoderscited by 2
  2. 2026Subtitle-Aligned Fine-Tuning of Whisper for Swiss German ASR: Benchmark Contamination, Convention Mismatch, and an Honest Baseline at 25.6% WER (13.8% cWER)cited by 1
  3. 2026From Text Metrics to Model Internals: A Study of Whisper ASR Hallucination Detectioncited by 1
  4. 2026SPEARBench: A Benchmark for Naturalness Evaluation in Streaming Speech-to-Speech Language Modelscited by 1
  5. 2026Scalable Music Cover Retrieval Using Lyrics-Aligned Audio Embeddingscited by 0
  6. 2026Detecting Hallucinations in SpeechLLMs at Inference Time Using Attention Mapscited by 0
  7. 2026HALAS: A Human-Annotated Dataset of Hallucinations of Modern ASR Systemscited by 0
  8. 2026The Null Token Knows: Reducing Message-Free Hallucination in ASR and NMTcited by 0
  9. 2026The Anatomy of an ASR Hallucinationcited by 0
  10. 2026Reducing Hallucinated Transcripts in Whisper via Hallucination Space Projectioncited by 0
  11. 2026AURA: Uncertainty-Routed Activation Editing for Acoustic Grounding in Speech Foundation Modelscited by 0
  12. 2026AnchorPrompt: Self-Distilled Soft Prompts for Robust Audio-Language Modelscited by 0