mlsanityData to 14 Oct 2026

OmniDPO: A Preference Optimization Framework to Address Omni-Modal Hallucination

arXiv 2509.00723 ↗First posted 31 Aug 2025cs.AI cs.MMcited by 9

On the map: LLM alignment / RLHF / preference optimisation → Diffusion and multimodal model alignment

Abstract (arXiv)

Recently, Omni-modal large language models (OLLMs) have sparked a new wave of research, achieving impressive results in tasks such as audio-video understanding and real-time environment perception. However, hallucination issues still persist. Similar to the bimodal setting, the priors from the text modality tend to dominate, leading OLLMs to rely more heavily on textual cues while neglecting visual and audio information. In addition, fully multimodal scenarios introduce new challenges. Most existing models align visual or auditory modalities with text independently during training, while ignoring the intrinsic correlations between video and its corresponding audio. This oversight results in hallucinations when reasoning requires interpreting hidden audio cues embedded in video content. To address these challenges, we propose OmniDPO, a preference-alignment framework designed to mitigate hallucinations in OLLMs. Specifically, OmniDPO incorporates two strategies: (1) constructing text-preference sample pairs to enhance the model's understanding of audio-video interactions; and (2) constructing multimodal-preference sample pairs to strengthen the model's attention to visual and auditory information. By tackling both challenges, OmniDPO effectively improves multimodal grounding and reduces hallucination. Experiments conducted on two OLLMs demonstrate that OmniDPO not only effectively mitigates multimodal hallucinations but also significantly enhances the models' reasoning capabilities across modalities. All code and datasets will be released upon paper acceptance.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in Diffusion and multimodal model alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 70 of its 84 references that are in the map. First posted on arXiv 31 Aug 2025.

Builds on

70 of 84 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Task Preference Optimization: Improving Multimodal Large Language Models with Vision Task Alignment26 Dec
  2. 2024V-DPO: Mitigating Hallucination in Large Vision Language Models via Vision-Guided Direct Preference Optimization5 Nov
  3. 2024mDPO: Conditional Preference Optimization for Multimodal Large Language Models17 Jun
  4. 2024Multi-Modal Hallucination Control by Visual Information Grounding20 Mar
  5. 2024Aligning Modalities in Vision Large Language Models via Preference Fine-tuning18 Feb
  6. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization28 Nov
  7. 2023Aligning Large Multimodal Models with Factually Augmented RLHF25 Sep
  8. 2023Detecting and Preventing Hallucinations in Large Vision Language Models11 Aug

Other references · 62

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  6. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  7. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  8. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  9. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  10. 2023A Survey on Multimodal Large Language Modelscited by 909
  11. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 819
  12. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 688
38 more
  1. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 561
  2. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 478
  3. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 445
  4. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 388
  5. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 376
  6. 2024A Survey on Hallucination in Large Vision-Language Modelscited by 357
  7. 2023FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarioscited by 251
  8. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Modelscited by 226
  9. 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decodingcited by 203
  10. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMscited by 179
  11. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decodingcited by 174
  12. 2023VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Datasetcited by 169
  13. 2021The Curious Case of Hallucinations in Neural Machine Translationcited by 161
  14. 2024Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similaritiescited by 159
  15. 2023HalluciDoctor: Mitigating Hallucinatory Toxicity in Visual Instruction Datacited by 126
  16. 2024IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decodingcited by 118
  17. 2024xGen-MM (BLIP-3): A Family of Open Large Multimodal Modelscited by 116
  18. 2019Towards Causal VQA: Revealing and Reducing Spurious Correlations by Invariant and Covariant Semantic Editingcited by 108
  19. 2024A Comprehensive Survey of Hallucination in Large Language, Image, Video and Audio Foundation Modelscited by 100
  20. 2024Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Modelscited by 94
  21. 2025Ola: Pushing the Frontiers of Omni-Modal Language Modelcited by 83
  22. 2024Look Twice Before You Answer: Memory-Space Visual Retracing for Hallucination Mitigation in Multimodal Large Language Modelscited by 72
  23. 2024Visual Hallucinations of Multi-modal Large Language Modelscited by 68
  24. 2024Reducing Hallucinations in Vision-Language Models via Latent Space Steeringcited by 59
  25. 2024AVHBench: A Cross-Modal Hallucination Benchmark for Audio-Visual Large Language Modelscited by 56
  26. 2024The Curse of Multi-Modalities: Evaluating Hallucinations of Large Multimodal Models across Language, Visual, and Audiocited by 53
  27. 2024Logical Closed Loop: Uncovering Object Hallucinations in Large Vision-Language Modelscited by 47
  28. 2024Mitigating Object Hallucination in MLLMs via Data-augmented Phrase-level Alignmentcited by 42
  29. 2024OmniBind: Large-scale Omni Multimodal Representation via Binding Spacescited by 39
  30. 2024CODE: Contrasting Self-generated Description to Combat Hallucination in Large Multi-modal Modelscited by 38
  31. 2025PerturboLLaVA: Reducing Multimodal Hallucinations with Perturbative Visual Trainingcited by 38
  32. 2023Re-Reading Improves Reasoning in Large Language Modelscited by 37
  33. 2024Debiasing Multimodal Large Language Models via Penalization of Language Priorscited by 37
  34. 2024Alleviating Hallucinations in Large Vision-Language Models through Hallucination-Induced Optimizationcited by 34
  35. 2025Generate, but Verify: Reducing Hallucination in Vision-Language Models with Retrospective Resamplingcited by 33
  36. 2024Investigating and Mitigating the Multimodal Hallucination Snowballing in Large Vision-Language Modelscited by 31
  37. 2024ICT: Image-Object Cross-Level Trusted Intervention for Mitigating Object Hallucination in Large Vision-Language Modelscited by 30
  38. 2024Look, Compare, Decide: Alleviating Hallucination in Large Vision-Language Models via Multi-View Multi-Path Reasoningcited by 23

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 9.

9 papers.

  1. 2026MoD-DPO: Towards Mitigating Cross-modal Hallucinations in Omni LLMs using Modality Decoupled Preference Optimizationcited by 6
  2. 2026Don't Let the Video Speak: Audio-Contrastive Preference Optimization for Audio-Visual Language Modelscited by 6
  3. 2026AVERE: Improving Audiovisual Emotion Reasoning with Preference Optimizationcited by 4
  4. 2026Separate First, Fuse Later: Mitigating Cross-Modal Interference in Audio-Visual LLMs Reasoning with Modality-Specific Chain-of-Thoughtcited by 2
  5. 2026TLive-Omni: An Omni-Modal Understanding Model for E-Commerce Live Streamingcited by 2
  6. 2026Chain of Modality: From Static Fusion to Dynamic Orchestration in Omni-MLLMscited by 1
  7. 2026Seeing What Should Be Heard: Diagnosing and Repairing Cross-Modal Shortcuts in Omni-Modal LLMscited by 0
  8. 2026Devils in Question Relay: Source-Conditioned Relay Steering to Mitigate Hallucinations in Audio-visual Large Language Modelscited by 0
  9. 2026Mitigating Cross-Modal Hallucinations by Correcting Modality Misrouting in Omnimodal LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026OmniHallu: Unified Hallucination Detection for Cross-Modal Comprehension and Generation in Multimodal Large Language Models
  2. 2025Mitigating Hallucinations in Large Vision-Language Models via Entity-Centric Multimodal Preference Optimization
  3. 2024Mitigating Hallucination in Multimodal Large Language Model via Hallucination-targeted Direct Preference Optimization
  4. 2025TARS: MinMax Token-Adaptive Preference Strategy for Hallucination Reduction in MLLMs
  5. 2026Clearer Sight, Fewer Lies: Oriented Pickup Preference Optimization for Multimodal Hallucination Mitigation
  6. 2025OViP: Online Vision-Language Preference Learning for VLM Hallucination
  7. 2024Cracking the Code of Hallucination in LVLMs with Vision-aware Head Divergence
  8. 2025Mitigating Hallucination Through Theory-Consistent Symmetric Multimodal Preference Optimization