# Reducing Hallucinated Transcripts in Whisper via Hallucination Space Projection

arXiv: https://arxiv.org/abs/2609.04561 · HTML: https://mlsanity.com/paper/2609.04561/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 13 references that are in the map.
- First posted on arXiv 3 Sep 2026.

## Builds on (most cited first, 8 of 8)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [FSD50K: An Open Dataset of Human-Labeled Sound Events](https://mlsanity.com/paper/2010.00475/) (2020)
- [WhisperX: Time-Accurate Speech Transcription of Long-Form Audio](https://mlsanity.com/paper/2303.00747/) (2023)
- [Careless Whisper: Speech-to-Text Hallucination Harms](https://mlsanity.com/paper/2402.08021/) (2024)
- [Nullu: Mitigating Object Hallucinations in Large Vision-Language Models via HalluSpace Projection](https://mlsanity.com/paper/2412.13817/) (2024)
- [Investigation of Whisper ASR Hallucinations Induced by Non-Speech Audio](https://mlsanity.com/paper/2501.11378/) (2025)
- [Calm-Whisper: Reduce Whisper Hallucination On Non-Speech By Calming Crazy Heads Down](https://mlsanity.com/paper/2505.12969/) (2025)
- [Fighting Hallucinations with Counterfactuals: Diffusion-Guided Perturbations for LVLM Hallucination Suppression](https://mlsanity.com/paper/2603.10470/) (2026)

## Abstract (arXiv)

Whisper is a widely used foundation model for automatic speech recognition (ASR), but its generative decoder can produce fluent hallucinated transcripts for inputs containing little or no speech. We propose a training-free, inference-time method to reduce these hallucinations using low-rank projection of decoder activations. A compact hallucination-associated subspace is estimated from non-speech calibration data, and decoder hidden states are projected away from this subspace during inference. We evaluate two variants: always-on, which applies projection to all inputs, and gated, which applies it only when Whisper predicts that an input is likely non-speech. Across non-speech benchmarks, always-on projection reduces average hallucination rate (HR) from 31.31% to 2.44%, a 92.21% relative reduction, while gated projection reduces HR to 3.74%, an 88.05% relative reduction, with lower false rejection of genuine speech. On LibriSpeech, gated projection increases absolute word error rate (WER) by 0.33-4.39 percentage points and yields false-rejection rates (FRR) of 0.41--9.97% across model and split settings. These results show that low-rank activation projection can substantially suppress Whisper hallucinations without retraining, while providing a controllable trade-off between hallucination suppression and speech recognition performance.