mlsanityData to 14 Oct 2026

Whisper Hallucination Detection and Mitigation via Hidden Representation Steering and Sparse AutoEncoders

arXiv 2606.07473 ↗First posted 5 Jun 2026cs.SD cs.AIcited by 2

On the map: Mechanistic interpretability → Hallucination detection from model internals

Abstract (arXiv)

Whisper, a widely adopted ASR model, is known to suffer from hallucinations - coherent transcriptions generated for non-speech audio entirely disconnected from the input. We investigate whether hallucinations can be detected and mitigated through Whisper's internal representations. We extract audio encoder activations and evaluate two representation spaces: raw Whisper activations and Sparse AutoEncoder (SAE) latents. We show that both spaces encode linearly separable hallucination-related information, with discriminative power concentrated in a sparse feature subset and increasing toward deeper encoder layers. We propose two steering strategies: activation-space steering and SAE latent-space steering. SAE-based steering reduces hallucination rate from 72.63% to 14.11% for Whisper small and from 86.88% to 27.33% for Whisper large-v3 on the full non-speech test set, with small WER degradation on speech data, approaching the performance of fine-tuning-based methods.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Hallucination detection from model internals, one of 13 directions in Mechanistic interpretability. Builds on 29 of its 39 references that are in the map. First posted on arXiv 5 Jun 2026.

Builds on

29 of 39 references.

References · 29

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  2. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  3. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
  4. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  5. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
  6. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
  7. 2023Steering Language Models With Activation Engineeringcited by 922
  8. 2025Qwen3-Omni Technical Reportcited by 497
  9. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 469
  10. 2015MUSAN: A Music, Speech, and Noise Corpuscited by 425
  11. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 394
  12. 2017AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baselinecited by 373
17 more
  1. 2020FSD50K: An Open Dataset of Human-Labeled Sound Eventscited by 342
  2. 2018Streaming End-to-end Speech Recognition For Mobile Devicescited by 234
  3. 2025Kimi-Audio Technical Reportcited by 222
  4. 2024Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generationcited by 204
  5. 2019WHAM!: Extending Speech Separation to Noisy Environmentscited by 156
  6. 2025Step-Audio 2 Technical Reportcited by 105
  7. 2019Recognizing long-form speech using streaming end-to-end modelscited by 77
  8. 2025Sparse Autoencoders Learn Monosemantic Features in Vision-Language Modelscited by 71
  9. 2024Sparse autoencoders reveal selective remapping of visual concepts during adaptationcited by 68
  10. 2025Investigation of Whisper ASR Hallucinations Induced by Non-Speech Audiocited by 31
  11. 2024Hallucinations in Neural Automatic Speech Recognition: Identifying Errors and Hallucinatory Modelscited by 27
  12. 2025CASteer: Cross-Attention Steering for Controllable Concept Erasurecited by 16
  13. 2026VIBEVOICE-ASR Technical Reportcited by 16
  14. 2025Feature-Level Insights into Artificial Text Detection with Sparse Autoencoderscited by 15
  15. 2025Discovering and Steering Interpretable Concepts in Large Generative Music Modelscited by 13
  16. 2025Calm-Whisper: Reduce Whisper Hallucination On Non-Speech By Calming Crazy Heads Downcited by 12
  17. 2025GigaAM: Efficient Self-Supervised Learner for Speech Recognitioncited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Interpreting and Steering a Text-to-Speech Language Model with Sparse Autoencoderscited by 1
  2. 2026SPEARBench: A Benchmark for Naturalness Evaluation in Streaming Speech-to-Speech Language Modelscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Listen Like a Teacher: Mitigating Whisper Hallucinations using Adaptive Layer Attention and Knowledge Distillation
  2. 2026HalluSAE: Detecting Hallucinations in Large Language Models via Sparse Auto-Encoders
  3. 2026HalluAudio: A Comprehensive Benchmark for Hallucination Detection in Large Audio-Language Models
  4. 2026Mitigating Entangled Steering in Large Vision-Language Models for Hallucination Reduction
  5. 2025Steering LVLMs via Sparse Autoencoder for Hallucination Mitigation
  6. 2026RAGognizer: Hallucination-Aware Fine-Tuning via Detection Head Integration
  7. 2026Steer Where It Matters: Token-Level Visual-Sensitivity Steering for LVLMs Hallucination Mitigation
  8. 2025SAFE: A Sparse Autoencoder-Based Framework for Robust Query Enrichment and Hallucination Mitigation in LLMs