mlsanityData to 7 Oct 2026

CHiP: Cross-modal Hierarchical Direct Preference Optimization for Multimodal LLMs

arXiv 2501.16629 ↗First posted 28 Jan 2025cs.CL cs.CVcited by 31

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

also in 1 more
Abstract (arXiv)

Multimodal Large Language Models (MLLMs) still struggle with hallucinations despite their impressive capabilities. Recent studies have attempted to mitigate this by applying Direct Preference Optimization (DPO) to multimodal scenarios using preference pairs from text-based responses. However, our analysis of representation distributions reveals that multimodal DPO struggles to align image and text representations and to distinguish between hallucinated and non-hallucinated descriptions. To address these challenges, in this work, we propose a Cross-modal Hierarchical Direct Preference Optimization (CHiP) to address these limitations. We introduce a visual preference optimization module within the DPO framework, enabling MLLMs to learn from both textual and visual preferences simultaneously. Furthermore, we propose a hierarchical textual preference optimization module that allows the model to capture preferences at multiple granular levels, including response, segment, and token levels. We evaluate CHiP through both quantitative and qualitative analyses, with results across multiple benchmarks demonstrating its effectiveness in reducing hallucinations. On the Object HalBench dataset, CHiP outperforms DPO in hallucination reduction, achieving improvements of 52.7% and 55.5% relative points based on the base model Muffin and LLaVA models, respectively. We make all our datasets and code publicly available: https://github.com/LVUGAI/CHiP.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 19 of its 49 references that are in the map. First posted on arXiv 28 Jan 2025.

Builds on

19 of 49 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Reflective Instruction Tuning: Mitigating Hallucinations in Large Vision-Language Models16 Jul
  2. 2024Mitigating Object Hallucination in MLLMs via Data-augmented Phrase-level Alignment28 May
  3. 2024Hallucination of Multimodal Large Language Models: A Survey29 Apr
  4. 2024A Survey on Hallucination in Large Vision-Language Models1 Feb
  5. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation13 Nov
  6. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Models24 Oct
  7. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun

Other references · 12

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  3. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  4. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  5. 2018Object Hallucination in Image Captioningcited by 661
  6. 2024RLAIF-V: Open-Source AI Feedback Leads to Super GPT-4V Trustworthinesscited by 129
  7. 2023Silkie: Preference Distillation for Large Visual Language Modelscited by 128
  8. 2024Efficient Multimodal Learning from Data-centric Perspectivecited by 120
  9. 2024mDPO: Conditional Preference Optimization for Multimodal Large Language Modelscited by 92
  10. 2024Enhancing Large Vision Language Models with Self-Training on Image Comprehensioncited by 70
  11. 2024Correlated Proxies: A New Definition and Improved Mitigation for Reward Hackingcited by 50
  12. 2023Reformulating Vision-Language Foundation Models and Datasets Towards Universal Multimodal Assistantscited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 16; 2026: 15.

31 papers.

Sort:

By area:

  1. 2025A Survey of Direct Preference Optimizationcited by 35
  2. 2025Hierarchical Fine-grained Preference Optimization for Physically Plausible Video Generationcited by 20
  3. 2025PaMi-VDPO: Mitigating Video Hallucinations by Prompt-Aware Multi-Instance Video Preference Learningcited by 15
  4. 2025Symmetrical Visual Contrastive Optimization: Aligning Vision-Language Models with Minimal Contrastive Imagescited by 14
  5. 2025VistaDPO: Video Hierarchical Spatial-Temporal Direct Preference Optimization for Large Video Modelscited by 14
  6. 2025Mitigating Hallucination Through Theory-Consistent Symmetric Multimodal Preference Optimizationcited by 14
  7. 2025Aligning Multimodal LLM with Human Preference: A Surveycited by 12
  8. 2026Look Carefully: Adaptive Visual Reinforcements in Multimodal Large Language Models for Hallucination Mitigationcited by 7
  9. 2025SuperWriter: Reflection-Driven Long-Form Generation with Large Language Modelscited by 5
  10. 2025Importance Sampling for Multi-Negative Multimodal Direct Preference Optimizationcited by 5
  11. 2025RL makes MLLMs see better than SFTcited by 3
  12. 2025See Less, See Right: Bi-directional Perceptual Shaping For Multimodal Reasoningcited by 3
19 more
  1. 2025OViP: Online Vision-Language Preference Learning for VLM Hallucinationcited by 2
  2. 2025Zooming from Context to Cue: Hierarchical Preference Optimization for Multi-Image MLLMscited by 2
  3. 2026FINER: MLLMs Hallucinate under Fine-grained Negative Queriescited by 2
  4. 2026Mitigating Multimodal Hallucination via Phase-wise Self-rewardcited by 2
  5. 2026Same Semantics, Different Paths: Self-Improving Alignment for Vision-Text Compressioncited by 2
  6. 2025Mitigating Visual Hallucinations via Semantic Curriculum Preference Optimization in MLLMscited by 1
  7. 2026Enhancing Video Representations with Spatiotemporal-Semantic Residual to Mitigate Hallucinations in Video Large Multimodal Modelscited by 1
  8. 2026LongVPO: From Anchored Cues to Self-Reasoning for Long-Form Video Preference Optimizationcited by 1
  9. 2026Clearer Sight, Fewer Lies: Oriented Pickup Preference Optimization for Multimodal Hallucination Mitigationcited by 1
  10. 2026No Place to Hide: Benchmarking Video Hallucination with Background-Controlled Pairscited by 1
  11. 2025TARS: MinMax Token-Adaptive Preference Strategy for Hallucination Reduction in MLLMscited by 0
  12. 2025MCM-DPO: Multifaceted Cross-Modal Direct Preference Optimization for Alt-text Generationcited by 0
  13. 2026Countering the Over-Reliance Trap: Mitigating Object Hallucination for LVLMs via a Self-Validation Frameworkcited by 0
  14. 2026IRIS: Implicit Reward-Guided Internal Sifting for Mitigating Multimodal Hallucinationcited by 0
  15. 2026Language Bias in LVLMs: From In-Depth Analysis to Simple and Effective Mitigationcited by 0
  16. 2026Learning from Fine-Grained Visual Discrepancies: Mitigating Multimodal Hallucinations via In-Context Visual Contrastive Optimizationcited by 0
  17. 2026S2COPE: Self-Supervised Concept Discovery via Preference Learningcited by 0
  18. 2026PEA-DPO: Perception-Enhanced Alignment Direct Preference Optimization for MLLMs Alignmentcited by 0
  19. 2026Hallucination Mitigation for Large Vision-Language Models via Implicit Feature Stabilizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization
  2. 2025Mitigating Hallucinations in Large Vision-Language Models via Entity-Centric Multimodal Preference Optimization
  3. 2024Mitigating Hallucination in Multimodal Large Language Model via Hallucination-targeted Direct Preference Optimization
  4. 2026Grounded Post-Training with Hard Examples for Reducing Hallucination in Multimodal Large Language Models
  5. 2026Context Blindness in DPO: Mitigating Object Hallucination in MLLMs via Context-Calibrated Preference Optimization
  6. 2025Mitigating Hallucinations in Multimodal LLMs via Object-aware Preference Optimization
  7. 2024Automated Multi-level Preference for MLLMs
  8. 2026DA-DPO: Cost-efficient Difficulty-aware Preference Optimization for Reducing MLLM Hallucinations