mlsanityData to 14 Oct 2026

Silence is Golden: Mitigating Hallucinations in Large Audio-Language Models via Layer-Weighted Vector Steering

arXiv 2510.12851 ↗First posted 14 Oct 2025cs.SD cs.LG eess.AScited by 6

Not on the map. Why?

Abstract (arXiv)

Large Audio-Language Models (LALMs) excel in Audio QA but often suffer from hallucinations ungrounded in the audio. To our knowledge, we are the first to propose applying vector steering to the audio domain to mitigate this. Unlike text-based steering, our silence-anchored contrastive approach steers the model away from hallucinations by contrasting active audio against a silent baseline. Probing internal states reveals a strong correlation between specific layer representations and output correctness. Leveraging this, we introduce Layer-Weighted Vector Steering (LWVS), a training-free intervention that increases steering strength at influential layers. On the Audio Hallucination QA dataset, LWVS significantly outperforms baselines, boosting Recall on the Gemma model by 15.6% (53.4% to 69.0%). Crucially, MMAU benchmark tests confirm LWVS preserves and even enhances general audio understanding, achieving an 8% relative accuracy increase on the Qwen model (54.8% to 59.2%).

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 26 of its 28 references that are in the map. First posted on arXiv 14 Oct 2025.

Builds on

26 of 28 references.

References · 26

  1. 2025Gemma 3 Technical Reportcited by 1,824
  2. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  3. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  4. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  5. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  6. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  7. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 609
  8. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 568
  9. 2024Qwen2-Audio Technical Reportcited by 560
  10. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 529
  11. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 506
  12. 2021Generative Spoken Language Modeling from Raw Audiocited by 340
14 more
  1. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 298
  2. 2024MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmarkcited by 288
  3. 2023Pengi: An Audio Language Model for Audio Taskscited by 220
  4. 2025Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Modelscited by 219
  5. 2019An Unsupervised Autoregressive Model for Speech Representation Learningcited by 186
  6. 2024LLaMA-Omni: Seamless Speech Interaction with Large Language Modelscited by 173
  7. 2025Step-Audio 2 Technical Reportcited by 105
  8. 2025The Hidden Life of Tokens: Reducing Hallucination of Large Vision-Language Models via Visual Information Steeringcited by 66
  9. 2024Style Vectors for Steering Generative Large Language Modelcited by 56
  10. 2024Understanding Sounds, Missing the Questions: The Challenge of Object Hallucination in Large Audio-Language Modelscited by 31
  11. 2025Removal of Hallucination on Hallucination: Debate-Augmented RAGcited by 29
  12. 2025Reducing Object Hallucination in Large Audio-Language Models via Audio-Aware Decodingcited by 24
  13. 2025TASTE: Text-Aligned Speech Tokenization and Embedding for Spoken Language Modelingcited by 16
  14. 2024On the Audio Hallucinations in Large Audio-Video Language Modelscited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026Are Audio-Language Models Listening? Audio-Specialist Heads for Adaptive Audio Steeringcited by 4
  2. 2026Nudging Hidden States: Training-Free Model Steering for Chain-of-Thought Reasoning in Large Audio-Language Modelscited by 2
  3. 2026Steering Where to Listen: Instruction-Based Activation Steering Redirects Temporal Attention in Large Audio-Language Modelscited by 1
  4. 2026Linear Script Representations in Speech Foundation Models Enable Zero-Shot Transliterationcited by 0
  5. 2026Encoder-Side Neuron Identification and Amplification for Acoustic Perception in Large Audio-Language Modelscited by 0
  6. 2026AnchorPrompt: Self-Distilled Soft Prompts for Robust Audio-Language Modelscited by 0