mlsanityData to 14 Oct 2026

Zooming from Context to Cue: Hierarchical Preference Optimization for Multi-Image MLLMs

arXiv 2505.22396 ↗First posted 28 May 2025cs.CVcited by 2

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 1 more
Abstract (arXiv)

Multi-modal Large Language Models (MLLMs) excel at single-image tasks but struggle with multi-image understanding due to cross-modal misalignment, leading to hallucinations (context omission, conflation, and misinterpretation). Existing methods using Direct Preference Optimization (DPO) constrain optimization to a solitary image reference within the input sequence, neglecting holistic context modeling. We propose Context-to-Cue Direct Preference Optimization (CcDPO), a multi-level preference optimization framework that enhances per-image perception in multi-image settings by zooming into visual clues -- from sequential context to local details. It features: (i) Context-Level Optimization : Re-evaluates cognitive biases underlying MLLMs' multi-image context comprehension and integrates a spectrum of low-cost global sequence preferences for bias mitigation. (ii) Needle-Level Optimization : Directs attention to fine-grained visual details through region-targeted visual prompts and multimodal preference supervision. To support scalable optimization, we also construct MultiScope-42k, an automatically generated dataset with high-quality multi-level preference pairs. Experiments show that CcDPO significantly reduces hallucinations and yields consistent performance gains across general single- and multi-image tasks.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 76 of its 83 references that are in the map. First posted on arXiv 28 May 2025.

Builds on

76 of 83 references.

Closest ancestors · 23

Same direction, newest first

  1. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling6 Dec
  2. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization15 Nov
  3. 2024MIA-DPO: Multi-Image Augmented Direct Preference Optimization For Large Vision-Language Models23 Oct
  4. 2024Modality-Fair Preference Optimization for Trustworthy MLLM Alignment20 Oct
  5. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  6. 2024LLaVA-OneVision: Easy Visual Task Transfer6 Aug
  7. 2024MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Models5 Aug
  8. 2024ControlMLLM: Training-Free Visual Prompt Learning for Multimodal Large Language Models31 Jul
  9. 2024MIBench: Evaluating Multimodal Large Language Models over Multiple Images21 Jul
  10. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models10 Jul
  11. 2024MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and Instruction-Tuning Dataset for LVLMs17 Jun
  12. 2024MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding13 Jun
11 more
  1. 2024Needle In A Multimodal Haystack11 Jun
  2. 2024MANTIS: Interleaved Multi-Image Instruction Tuning2 May
  3. 2024BLINK: Multimodal Large Language Models Can See but Not Perceive18 Apr
  4. 2024Direct Preference Optimization of Video Large Multimodal Models from Language Model Reward1 Apr
  5. 2024Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Want29 Mar
  6. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
  7. 2023Generative Multimodal Models are In-Context Learners20 Dec
  8. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  9. 2023Aligning Large Multimodal Models with Factually Augmented RLHF25 Sep
  10. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  11. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents21 Jun

Other references · 53

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2023Segment Anythingcited by 8,234
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2023Visual Instruction Tuningcited by 7,869
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  9. 2024SAM 2: Segment Anything in Images and Videoscited by 2,863
  10. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  11. 2015VQA: Visual Question Answeringcited by 2,506
  12. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
38 more
  1. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  2. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  3. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  4. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,109
  5. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
  6. 2016A Diagram Is Worth A Dozen Imagescited by 804
  7. 2018Object Hallucination in Image Captioningcited by 669
  8. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 603
  9. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 583
  10. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 565
  11. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 532
  12. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 476
  13. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 459
  14. 2025Seed1.5-VL Technical Reportcited by 373
  15. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 356
  16. 2024What matters when building vision-language models?cited by 326
  17. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluationcited by 300
  18. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 291
  19. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMscited by 218
  20. 2023Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Textcited by 208
  21. 2024Aligning Modalities in Vision Large Language Models via Preference Fine-tuningcited by 198
  22. 2021CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Modelscited by 183
  23. 2024LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inferencecited by 95
  24. 2025Tarsier2: Advancing Large Vision-Language Models from Detailed Video Description to Comprehensive Video Understandingcited by 81
  25. 2024V-DPO: Mitigating Hallucination in Large Vision Language Models via Vision-Guided Direct Preference Optimizationcited by 67
  26. 2023Large Language Models are Visual Reasoning Coordinatorscited by 58
  27. 2024Benchmarking Multi-Image Understanding in Vision and Language Models: Perception, Knowledge, Reasoning, and Multi-Hop Reasoningcited by 54
  28. 2024CLIP-DPO: Vision-Language Models as a Source of Preference for Fixing Hallucinations in LVLMscited by 49
  29. 2024CounterCurate: Enhancing Physical and Semantic Visio-Linguistic Compositional Reasoning via Counterfactual Examplescited by 35
  30. 2025CHiP: Cross-modal Hierarchical Direct Preference Optimization for Multimodal LLMscited by 31
  31. 2024ReMI: A Dataset for Reasoning with Multiple Imagescited by 30
  32. 2024Automated Multi-level Preference for MLLMscited by 29
  33. 2025Identifying and Mitigating Position Bias of Multi-image Vision-Language Modelscited by 25
  34. 2025CMMCoT: Enhancing Complex Multi-Image Comprehension via Multi-Modal Chain-of-Thought and Memory Augmentationcited by 23
  35. 2024FineCops-Ref: A new Dataset and Task for Fine-Grained Compositional Referring Expression Comprehensioncited by 20
  36. 2024Fine-Grained Verifiers: Preference Modeling as Next-token Prediction in Vision-Language Alignmentcited by 19
  37. 2025VistaDPO: Video Hierarchical Spatial-Temporal Direct Preference Optimization for Large Video Modelscited by 17
  38. 2024Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findingscited by 15

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Flow caching for autoregressive video generationcited by 25
  2. 2026Looking Back and Forth: Cross-Image Attention Calibration and Attentive Preference Learning for Multi-Image Hallucination Mitigationcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2025Zoomer: Adaptive Image Focus Optimization for Black-box MLLM
  2. 2024Task Preference Optimization: Improving Multimodal Large Language Models with Vision Task Alignment
  3. 2026Zooming without Zooming: Region-to-Image Distillation for Fine-Grained Multimodal Perception
  4. 2026CoVFT: Context-aware Visual Fine-tuning for Multimodal Large Language Models
  5. 2025M3PO: Multimodal-Model-Guided Preference Optimization for Visual Instruction Following
  6. 2026S2H-DPO: Hardness-Aware Preference Optimization for Vision-Language Models
  7. 2026Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Models
  8. 2026LOCUS: Local Visual Cue Search for Enhancing Fine-Grained Perception in Multimodal Large Language Models