mlsanityData to 7 Oct 2026

MIA-DPO: Multi-Image Augmented Direct Preference Optimization For Large Vision-Language Models

arXiv 2410.17637 ↗First posted 23 Oct 2024cs.CV cs.AIcited by 34

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 1 more
Abstract (arXiv)

Visual preference alignment involves training Large Vision-Language Models (LVLMs) to predict human preferences between visual inputs. This is typically achieved by using labeled datasets of chosen/rejected pairs and employing optimization algorithms like direct preference optimization (DPO). Existing visual alignment methods, primarily designed for single-image scenarios, struggle to effectively handle the complexity of multi-image tasks due to the scarcity of diverse training data and the high cost of annotating chosen/rejected pairs. We present Multi-Image Augmented Direct Preference Optimization (MIA-DPO), a visual preference alignment approach that effectively handles multi-image inputs. MIA-DPO mitigates the scarcity of diverse multi-image training data by extending single-image data with unrelated images arranged in grid collages or pic-in-pic formats, significantly reducing the costs associated with multi-image data annotations. Our observation reveals that attention values of LVLMs vary considerably across different images. We use attention values to identify and filter out rejected responses the model may have mistakenly focused on. Our attention-aware selection for constructing the chosen/rejected pairs without relying on (i) human annotation, (ii) extra data, and (iii) external models or APIs. MIA-DPO is compatible with various architectures and outperforms existing methods on five multi-image benchmarks, achieving an average performance boost of 3.0% on LLaVA-v1.5 and 4.3% on the recent InternLM-XC2.5. Moreover, MIA-DPO has a minimal effect on the model's ability to understand single images.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 50 of its 55 references that are in the map. First posted on arXiv 23 Oct 2024.

Builds on

50 of 55 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  2. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models10 Jul
  3. 2024MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and Instruction-Tuning Dataset for LVLMs17 Jun
  4. 2024MINT-1T: Scaling Open-Source Multimodal Data by 10x: A Multimodal Dataset with One Trillion Tokens17 Jun
  5. 2024MileBench: Benchmarking MLLMs in Long Context29 Apr
  6. 2023Generative Multimodal Models are In-Context Learners20 Dec
  7. 2023VILA: On Pre-training for Visual Language Models12 Dec
  8. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  9. 2023Aligning Large Multimodal Models with Factually Augmented RLHF25 Sep
  10. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  11. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  12. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents21 Jun
1 more
  1. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May

Other references · 37

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2024The Llama 3 Herd of Modelscited by 14,192
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  10. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  11. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  12. 2019Towards VQA Models That Can Readcited by 1,696
25 more
  1. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  2. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  3. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  4. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  5. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  6. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  7. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  8. 2016A Diagram Is Worth A Dozen Imagescited by 793
  9. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  10. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  11. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 466
  12. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  13. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  14. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 404
  15. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
  16. 2024What matters when building vision-language models?cited by 321
  17. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 285
  18. 2024Iterative Reasoning Preference Optimizationcited by 244
  19. 2024A Survey on Data Selection for Language Modelscited by 230
  20. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimizationcited by 223
  21. 2024Aligning Modalities in Vision Large Language Models via Preference Fine-tuningcited by 197
  22. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  23. 2024MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizationscited by 174
  24. 2024Tarsier: Recipes for Training and Evaluating Large Video Description Modelscited by 140
  25. 2024CLIP-DPO: Vision-Language Models as a Source of Preference for Fixing Hallucinations in LVLMscited by 49

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 19; 2026: 13.

34 papers.

Sort:

By area:

  1. 2025Visual-RFT: Visual Reinforcement Fine-Tuningcited by 501
  2. 2025InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Modelcited by 73
  3. 2025Visual Agentic Reinforcement Fine-Tuningcited by 53
  4. 2025CapRL: Stimulating Dense Image Caption Capabilities via Reinforcement Learningcited by 36
  5. 2025CMMCoT: Enhancing Complex Multi-Image Comprehension via Multi-Modal Chain-of-Thought and Memory Augmentationcited by 22
  6. 2024X-Prompt: Towards Universal In-Context Image Generation in Auto-Regressive Vision Language Foundation Modelscited by 15
  7. 2025ScaleCap: Inference-Time Scalable Image Captioning via Dual-Modality Debiasingcited by 15
  8. 2025GRE Suite: Geo-localization Inference via Fine-Tuned Vision-Language Models and Enhanced Reasoning Chainscited by 14
  9. 2025Mitigating Hallucination Through Theory-Consistent Symmetric Multimodal Preference Optimizationcited by 14
  10. 2025Aligning Multimodal LLM with Human Preference: A Surveycited by 12
  11. 2025Omni-Reward: Towards Generalist Omni-Modal Reward Modeling with Free-Form Preferencescited by 11
  12. 2024SymDPO: Boosting In-Context Learning of Large Multimodal Models with Symbol Demonstration Direct Preference Optimizationcited by 10
22 more
  1. 2025STAR-Bench: Probing Deep Spatio-Temporal Reasoning as Audio 4D Intelligencecited by 10
  2. 2025SPARK: Synergistic Policy And Reward Co-Evolving Frameworkcited by 9
  3. 2026Visual Self-Refine: A Pixel-Guided Paradigm for Accurate Chart Parsingcited by 5
  4. 2026Visual Preference Optimization with Rubric Rewardscited by 5
  5. 2026HP-Edit: A Human-Preference Post-Training Framework for Image Editingcited by 4
  6. 2025VisChainBench: A Benchmark for Multi-Turn, Multi-Image Visual Reasoning Beyond Language Priorscited by 3
  7. 2026Advancing Multimodal Judge Models through a Capability-Oriented Benchmark and MCTS-Driven Data Generationcited by 3
  8. 2025DPO Kernels: A Semantically-Aware, Kernel-Enhanced, and Divergence-Rich Paradigm for Direct Preference Optimizationcited by 2
  9. 2025Zooming from Context to Cue: Hierarchical Preference Optimization for Multi-Image MLLMscited by 2
  10. 2025Direct Diffusion Score Preference Optimization via Stepwise Contrastive Policy-Pair Supervisioncited by 2
  11. 2026Looking Back and Forth: Cross-Image Attention Calibration and Attentive Preference Learning for Multi-Image Hallucination Mitigationcited by 2
  12. 2026When Prompts Override Vision: Prompt-Induced Hallucinations in LVLMscited by 2
  13. 2025Instruction-Oriented Preference Alignment for Enhancing Multi-Modal Comprehension Capability of MLLMscited by 1
  14. 2025LeanPO: Lean Preference Optimization for Likelihood Alignment in Video-LLMscited by 1
  15. 2025Epistemic-aware Vision-Language Foundation Model for Fetal Ultrasound Interpretationcited by 1
  16. 2026LongVPO: From Anchored Cues to Self-Reasoning for Long-Form Video Preference Optimizationcited by 1
  17. 2026No Place to Hide: Benchmarking Video Hallucination with Background-Controlled Pairscited by 1
  18. 2026Schr"oMind: Mitigating Hallucinations in Multimodal Large Language Models via Solving the Schr"odinger Bridge Problemcited by 0
  19. 2026Scalpel: Fine-Grained Alignment of Attention Activation Manifolds via Mixture Gaussian Bridges to Mitigate Multimodal Hallucinationcited by 0
  20. 2026Uncertainty-Aware Exploratory Direct Preference Optimization for Multimodal Large Language Modelscited by 0
  21. 2026Advancing Creative Physical Intelligence in Large Multimodal Modelscited by 0
  22. 2026Beyond the Current Observation: Evaluating Multimodal Large Language Models in Controllable Non-Markov Gamescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Re-Align: Aligning Vision Language Models via Retrieval-Augmented Direct Preference Optimization
  2. 2024Self-Supervised Visual Preference Alignment
  3. 2026PEA-DPO: Perception-Enhanced Alignment Direct Preference Optimization for MLLMs Alignment
  4. 2025AdaViP: Aligning Multi-modal LLMs via Adaptive Vision-enhanced Preference Optimization
  5. 2026S2H-DPO: Hardness-Aware Preference Optimization for Vision-Language Models
  6. 2026ACPO: Counteracting Likelihood Displacement in Vision-Language Alignment with Asymmetric Constraints
  7. 2024Task Preference Optimization: Improving Multimodal Large Language Models with Vision Task Alignment
  8. 2024Fine-Grained Verifiers: Preference Modeling as Next-token Prediction in Vision-Language Alignment