# CHiP: Cross-modal Hierarchical Direct Preference Optimization for Multimodal LLMs

arXiv: https://arxiv.org/abs/2501.16629 · HTML: https://mlsanity.com/paper/2501.16629/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 19 of its 49 references that are in the map.
- First posted on arXiv 28 Jan 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Hallucination mitigation in LVLMs](https://mlsanity.com/mllm/hallucination-mitigation-in-lvlms/)
- LLM alignment / RLHF / preference optimisation / [Diffusion and multimodal model alignment](https://mlsanity.com/alignment/diffusion-and-multimodal-model-alignment/)

## Builds on (most cited first, 19 of 19)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [Aligning Large Multimodal Models with Factually Augmented RLHF](https://mlsanity.com/paper/2309.14525/) (2023)
- [Object Hallucination in Image Captioning](https://mlsanity.com/paper/1809.02156/) (2018)
- [Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning](https://mlsanity.com/paper/2306.14565/) (2023)
- [Hallucination of Multimodal Large Language Models: A Survey](https://mlsanity.com/paper/2404.18930/) (2024)
- [A Survey on Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2402.00253/) (2024)
- [AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation](https://mlsanity.com/paper/2311.07397/) (2023)
- [Woodpecker: Hallucination Correction for Multimodal Large Language Models](https://mlsanity.com/paper/2310.16045/) (2023)
- [RLAIF-V: Open-Source AI Feedback Leads to Super GPT-4V Trustworthiness](https://mlsanity.com/paper/2405.17220/) (2024)
- [Silkie: Preference Distillation for Large Visual Language Models](https://mlsanity.com/paper/2312.10665/) (2023)
- [Efficient Multimodal Learning from Data-centric Perspective](https://mlsanity.com/paper/2402.11530/) (2024)
- [mDPO: Conditional Preference Optimization for Multimodal Large Language Models](https://mlsanity.com/paper/2406.11839/) (2024)
- [Enhancing Large Vision Language Models with Self-Training on Image Comprehension](https://mlsanity.com/paper/2405.19716/) (2024)
- [Correlated Proxies: A New Definition and Improved Mitigation for Reward Hacking](https://mlsanity.com/paper/2403.03185/) (2024)
- [Mitigating Object Hallucination in MLLMs via Data-augmented Phrase-level Alignment](https://mlsanity.com/paper/2405.18654/) (2024)
- [Reformulating Vision-Language Foundation Models and Datasets Towards Universal Multimodal Assistants](https://mlsanity.com/paper/2310.00653/) (2023)
- [Reflective Instruction Tuning: Mitigating Hallucinations in Large Vision-Language Models](https://mlsanity.com/paper/2407.11422/) (2024)

## Built on by (most cited first, 30 of 31)

- [A Survey of Direct Preference Optimization](https://mlsanity.com/paper/2503.11701/) (2025)
- [Hierarchical Fine-grained Preference Optimization for Physically Plausible Video Generation](https://mlsanity.com/paper/2508.10858/) (2025)
- [PaMi-VDPO: Mitigating Video Hallucinations by Prompt-Aware Multi-Instance Video Preference Learning](https://mlsanity.com/paper/2504.05810/) (2025)
- [VistaDPO: Video Hierarchical Spatial-Temporal Direct Preference Optimization for Large Video Models](https://mlsanity.com/paper/2504.13122/) (2025)
- [Symmetrical Visual Contrastive Optimization: Aligning Vision-Language Models with Minimal Contrastive Images](https://mlsanity.com/paper/2502.13928/) (2025)
- [Mitigating Hallucination Through Theory-Consistent Symmetric Multimodal Preference Optimization](https://mlsanity.com/paper/2506.11712/) (2025)
- [Aligning Multimodal LLM with Human Preference: A Survey](https://mlsanity.com/paper/2503.14504/) (2025)
- [Look Carefully: Adaptive Visual Reinforcements in Multimodal Large Language Models for Hallucination Mitigation](https://mlsanity.com/paper/2602.24041/) (2026)
- [SuperWriter: Reflection-Driven Long-Form Generation with Large Language Models](https://mlsanity.com/paper/2506.04180/) (2025)
- [Importance Sampling for Multi-Negative Multimodal Direct Preference Optimization](https://mlsanity.com/paper/2509.25717/) (2025)
- [FINER: MLLMs Hallucinate under Fine-grained Negative Queries](https://mlsanity.com/paper/2603.17662/) (2026)
- [RL makes MLLMs see better than SFT](https://mlsanity.com/paper/2510.16333/) (2025)
- [See Less, See Right: Bi-directional Perceptual Shaping For Multimodal Reasoning](https://mlsanity.com/paper/2512.22120/) (2025)
- [OViP: Online Vision-Language Preference Learning for VLM Hallucination](https://mlsanity.com/paper/2505.15963/) (2025)
- [Zooming from Context to Cue: Hierarchical Preference Optimization for Multi-Image MLLMs](https://mlsanity.com/paper/2505.22396/) (2025)
- [Mitigating Multimodal Hallucination via Phase-wise Self-reward](https://mlsanity.com/paper/2604.17982/) (2026)
- [No Place to Hide: Benchmarking Video Hallucination with Background-Controlled Pairs](https://mlsanity.com/paper/2606.31933/) (2026)
- [Same Semantics, Different Paths: Self-Improving Alignment for Vision-Text Compression](https://mlsanity.com/paper/2608.02109/) (2026)
- [Mitigating Visual Hallucinations via Semantic Curriculum Preference Optimization in MLLMs](https://mlsanity.com/paper/2509.24491/) (2025)
- [Enhancing Video Representations with Spatiotemporal-Semantic Residual to Mitigate Hallucinations in Video Large Multimodal Models](https://mlsanity.com/paper/2601.22574/) (2026)
- [LongVPO: From Anchored Cues to Self-Reasoning for Long-Form Video Preference Optimization](https://mlsanity.com/paper/2602.02341/) (2026)
- [Clearer Sight, Fewer Lies: Oriented Pickup Preference Optimization for Multimodal Hallucination Mitigation](https://mlsanity.com/paper/2606.29805/) (2026)
- [TARS: MinMax Token-Adaptive Preference Strategy for Hallucination Reduction in MLLMs](https://mlsanity.com/paper/2507.21584/) (2025)
- [MCM-DPO: Multifaceted Cross-Modal Direct Preference Optimization for Alt-text Generation](https://mlsanity.com/paper/2510.00647/) (2025)
- [Countering the Over-Reliance Trap: Mitigating Object Hallucination for LVLMs via a Self-Validation Framework](https://mlsanity.com/paper/2601.22451/) (2026)
- [IRIS: Implicit Reward-Guided Internal Sifting for Mitigating Multimodal Hallucination](https://mlsanity.com/paper/2602.01769/) (2026)
- [Language Bias in LVLMs: From In-Depth Analysis to Simple and Effective Mitigation](https://mlsanity.com/paper/2605.25036/) (2026)
- [Learning from Fine-Grained Visual Discrepancies: Mitigating Multimodal Hallucinations via In-Context Visual Contrastive Optimization](https://mlsanity.com/paper/2605.31312/) (2026)
- [S2COPE: Self-Supervised Concept Discovery via Preference Learning](https://mlsanity.com/paper/2606.14586/) (2026)
- [PEA-DPO: Perception-Enhanced Alignment Direct Preference Optimization for MLLMs Alignment](https://mlsanity.com/paper/2608.19598/) (2026)

## Abstract (arXiv)

Multimodal Large Language Models (MLLMs) still struggle with hallucinations despite their impressive capabilities. Recent studies have attempted to mitigate this by applying Direct Preference Optimization (DPO) to multimodal scenarios using preference pairs from text-based responses. However, our analysis of representation distributions reveals that multimodal DPO struggles to align image and text representations and to distinguish between hallucinated and non-hallucinated descriptions. To address these challenges, in this work, we propose a Cross-modal Hierarchical Direct Preference Optimization (CHiP) to address these limitations. We introduce a visual preference optimization module within the DPO framework, enabling MLLMs to learn from both textual and visual preferences simultaneously. Furthermore, we propose a hierarchical textual preference optimization module that allows the model to capture preferences at multiple granular levels, including response, segment, and token levels. We evaluate CHiP through both quantitative and qualitative analyses, with results across multiple benchmarks demonstrating its effectiveness in reducing hallucinations. On the Object HalBench dataset, CHiP outperforms DPO in hallucination reduction, achieving improvements of 52.7% and 55.5% relative points based on the base model Muffin and LLaVA models, respectively. We make all our datasets and code publicly available: https://github.com/LVUGAI/CHiP.