mlsanityData to 7 Oct 2026

Don't Let the Video Speak: Audio-Contrastive Preference Optimization for Audio-Visual Language Models

arXiv 2604.14129 ↗First posted 15 Apr 2026cs.CVcited by 4

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

also in 1 more
Abstract (arXiv)

While Audio-Visual Language Models (AVLMs) have achieved remarkable progress over recent years, their reliability is bottlenecked by cross-modal hallucination. A particularly pervasive manifestation is video-driven audio hallucination: models routinely exploit visual shortcuts to hallucinate expected sounds, discarding true auditory evidence. To counteract this deeply ingrained visual dominance, we propose Audio-Contrastive Preference Optimization (ACPO). This dual-axis preference learning framework introduces an output-contrastive objective to penalize visual descriptions masquerading as audio facts, alongside an input-contrastive objective that swaps audio tracks to explicitly penalize generation invariant to the true auditory signal. Extensive experiments demonstrate that ACPO establishes highly faithful audio grounding and mitigates audio hallucination.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 52 of its 57 references that are in the map. First posted on arXiv 15 Apr 2026.

Builds on

52 of 57 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025MRFD: Multi-Region Fusion Decoding with Self-Consistency for Mitigating Hallucinations in LVLMs14 Aug
  2. 2025AVCD: Mitigating Hallucinations in Audio-Visual Large Language Models through Contrastive Decoding27 May
  3. 2025Generate, but Verify: Reducing Hallucination in Vision-Language Models with Retrospective Resampling17 Apr
  4. 2024V-DPO: Mitigating Hallucination in Large Vision Language Models via Vision-Guided Direct Preference Optimization5 Nov
  5. 2024The Curse of Multi-Modalities: Evaluating Hallucinations of Large Multimodal Models across Language, Visual, and Audio16 Oct
  6. 2024Prescribing the Right Remedy: Mitigating Hallucinations in Large Vision-Language Models via Targeted Instruction Tuning16 Apr
  7. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  8. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization28 Nov
  9. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models23 Oct

Other references · 43

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2017Proximal Policy Optimization Algorithmscited by 14,441
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  8. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  9. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  10. 2023Visual Instruction Tuningcited by 7,792
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  12. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
31 more
  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  3. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  5. 2020Learning to summarize from human feedbackcited by 2,562
  6. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  7. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  8. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  9. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  10. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  11. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,127
  12. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  13. 2025Qwen2.5-Omni Technical Reportcited by 704
  14. 2018Object Hallucination in Image Captioningcited by 661
  15. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 641
  16. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 527
  17. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  18. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 378
  19. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 330
  20. 2022Learning to Answer Questions in Dynamic Audio-Visual Scenarioscited by 188
  21. 2023VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Datasetcited by 168
  22. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 138
  23. 2024AVHBench: A Cross-Modal Hallucination Benchmark for Audio-Visual Large Language Modelscited by 50
  24. 2025video-SALMONN 2: Caption-Enhanced Audio-Visual Large Language Modelscited by 39
  25. 2025Seeing the Arrow of Time in Large Multimodal Modelscited by 28
  26. 2024On the Audio Hallucinations in Large Audio-Video Language Modelscited by 15
  27. 2025When Thinking Drifts: Evidential Grounding for Robust Video Reasoningcited by 15
  28. 2025Vid2Coach: Transforming How-To Videos into Task Assistantscited by 12
  29. 2024Progress-Aware Video Frame Captioningcited by 10
  30. 2025OmniDPO: A Preference Optimization Framework to Address Omni-Modal Hallucinationcited by 8
  31. 2025Audo-Sight: Enabling Ambient Interaction For Blind And Visually Impaired Individualscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026When Vision Speaks for Soundcited by 2
  2. 2026Omni-Streaming Thinkingcited by 1
  3. 2026Seeing What Should Be Heard: Diagnosing and Repairing Cross-Modal Shortcuts in Omni-Modal LLMscited by 0
  4. 2026Devils in Question Relay: Source-Conditioned Relay Steering to Mitigate Hallucinations in Audio-visual Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Clearer Sight, Fewer Lies: Oriented Pickup Preference Optimization for Multimodal Hallucination Mitigation
  2. 2026MoD-DPO: Towards Mitigating Cross-modal Hallucinations in Omni LLMs using Modality Decoupled Preference Optimization
  3. 2025Mitigating Hallucination Through Theory-Consistent Symmetric Multimodal Preference Optimization
  4. 2026Grounded Post-Training with Hard Examples for Reducing Hallucination in Multimodal Large Language Models
  5. 2024Mitigating Hallucination in Multimodal Large Language Model via Hallucination-targeted Direct Preference Optimization
  6. 2026Relevant Evidence Decoding for Audio-Visual Hallucination Mitigation
  7. 2026P2-DPO: Grounding Hallucination in Perceptual Processing via Calibration Direct Preference Optimization
  8. 2026Context Blindness in DPO: Mitigating Object Hallucination in MLLMs via Context-Calibrated Preference Optimization