mlsanityData to 14 Oct 2026

Instruction-Oriented Preference Alignment for Enhancing Multi-Modal Comprehension Capability of MLLMs

arXiv 2503.20309 ↗First posted 26 Mar 2025cs.CVcited by 1

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 1 more
Abstract (arXiv)

Preference alignment has emerged as an effective strategy to enhance the performance of Multimodal Large Language Models (MLLMs) following supervised fine-tuning. While existing preference alignment methods predominantly target hallucination factors, they overlook the factors essential for multi-modal comprehension capabilities, often narrowing their improvements on hallucination mitigation. To bridge this gap, we propose Instruction-oriented Preference Alignment (IPA), a scalable framework designed to automatically construct alignment preferences grounded in instruction fulfillment efficacy. Our method involves an automated preference construction coupled with a dedicated verification process that identifies instruction-oriented factors, avoiding significant variability in response representations. Additionally, IPA incorporates a progressive preference collection pipeline, further recalling challenging samples through model self-evolution and reference-guided refinement. Experiments conducted on Qwen2VL-7B demonstrate IPA's effectiveness across multiple benchmarks, including hallucination evaluation, visual question answering, and text understanding tasks, highlighting its capability to enhance general comprehension.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 60 of its 75 references that are in the map. First posted on arXiv 26 Mar 2025.

Builds on

60 of 75 references.

Closest ancestors · 15

Same direction, newest first

  1. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling6 Dec
  2. 2024Efficient Self-Improvement in Multimodal Large Language Models: A Model-Level Judge-Free Approach26 Nov
  3. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization15 Nov
  4. 2024Infinity-MM: Scaling Multimodal Performance with Large-Scale and High-Quality Instruction Data24 Oct
  5. 2024MIA-DPO: Multi-Image Augmented Direct Preference Optimization For Large Vision-Language Models23 Oct
  6. 2024LLaVA-Critic: Learning to Evaluate Multimodal Models3 Oct
  7. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phone3 Aug
  8. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models10 Jul
  9. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites25 Apr
  10. 2024Vision-Flan: Scaling Human-Labeled Tasks in Visual Instruction Tuning18 Feb
  11. 2023Silkie: Preference Distillation for Large Visual Language Models17 Dec
  12. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
3 more
  1. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  2. 2023Aligning Large Multimodal Models with Factually Augmented RLHF25 Sep
  3. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun

Other references · 45

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  6. 2025Qwen2.5-VL Technical Reportcited by 5,587
  7. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  8. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  9. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  10. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  11. 2019Towards VQA Models That Can Readcited by 1,718
  12. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
33 more
  1. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  2. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  3. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  4. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
  5. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  6. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 688
  7. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
  8. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 625
  9. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 610
  10. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 476
  11. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 463
  12. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 455
  13. 2021InfographicVQAcited by 437
  14. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 412
  15. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 400
  16. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 367
  17. 2018DVQA: Understanding Data Visualizations via Question Answeringcited by 340
  18. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 274
  19. 2023Detecting and Preventing Hallucinations in Large Vision Language Modelscited by 256
  20. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimizationcited by 225
  21. 2024Agent Q: Advanced Reasoning and Learning for Autonomous AI Agentscited by 208
  22. 2024ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Modelscited by 205
  23. 2024Aligning Modalities in Vision Large Language Models via Preference Fine-tuningcited by 198
  24. 2024RLAIF-V: Open-Source AI Feedback Leads to Super GPT-4V Trustworthinesscited by 131
  25. 2023HalluciDoctor: Mitigating Hallucinatory Toxicity in Visual Instruction Datacited by 126
  26. 2023Deductive Verification of Chain-of-Thought Reasoningcited by 124
  27. 2024M3CoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-Thoughtcited by 121
  28. 2023LLM-Eval: Unified Multi-Dimensional Automatic Evaluation for Open-Domain Conversations with Large Language Modelscited by 101
  29. 2024Speculative RAG: Enhancing Retrieval Augmented Generation through Draftingcited by 77
  30. 2023CritiqueLLM: Towards an Informative Critique Generation Model for Evaluation of Large Language Model Generationcited by 66
  31. 2024VLFeedback: A Large-Scale AI Feedback Dataset for Large Vision-Language Models Alignmentcited by 58
  32. 2024CLIP-DPO: Vision-Language Models as a Source of Preference for Fixing Hallucinations in LVLMscited by 49
  33. 2024Systematic Reward Gap Optimization for Mitigating VLM Hallucinationscited by 7

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Visual Alignment of Medical Vision-Language Models for Grounded Radiology Report Generationcited by 5

Similar, not cited

Close in wording, no citation either way

  1. 2024Align2LLaVA: Cascaded Human and Large Language Model Preference Alignment for Multi-modal Instruction Curation
  2. 2024Understanding Alignment in Multimodal LLMs: A Comprehensive Study
  3. 2024Multi-modal Preference Alignment Remedies Degradation of Visual Instruction Tuning on Language Models
  4. 2024Modality-Fair Preference Optimization for Trustworthy MLLM Alignment
  5. 2024A Systematic Examination of Preference Learning through the Lens of Instruction-Following
  6. 2025P-Aligner: Enabling Pre-Alignment of Language Models via Principled Instruction Synthesis
  7. 2025Improving Alignment in LVLMs with Debiased Self-Judgment
  8. 2025Mitigating Hallucinations in Large Vision-Language Models via Entity-Centric Multimodal Preference Optimization