mlsanityData to 14 Oct 2026

PEA-DPO: Perception-Enhanced Alignment Direct Preference Optimization for MLLMs Alignment

arXiv 2608.19598 ↗First posted 20 Aug 2026cs.CV cs.AI cs.CLcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Diffusion and multimodal model alignment

Abstract (arXiv)

Direct Preference Optimization (DPO) has emerged as an effective approach for aligning large language models (LLMs) with human preferences. However, its adaptation to multimodal settings remains unexplored. Through representational analysis, we identify a key limitation in multimodal preference optimization, which we term visual insensitivity: models often fail to distinguish between images and those with critical visual context removed. Our theoretical analysis further uncovers two manifestations of this problem, namely Across-Image Insensitivity and Within-Image Insensitivity. To address these challenges, we propose Perception-Enhanced Alignment DPO (PEA-DPO), a framework for multimodal LLMs alignment, which explicitly leverages visual preference signals to overcome visual insensitivity. We further provide a theoretical analysis demonstrating that PEA-DPO provably mitigates both failure modes. Empirical results demonstrate that PEA-DPO enhances sensitivity to visual context while preserving the language modeling capacity of the base model. Evaluations across three hallucination benchmarks using MLLMs of varying scales show that PEA-DPO effectively mitigates visual insensitivity, achieves stronger multimodal alignment, and substantially reduces hallucinations.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Diffusion and multimodal model alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 28 of its 46 references that are in the map. First posted on arXiv 20 Aug 2026.

Builds on

28 of 46 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025Mitigating Hallucination Through Theory-Consistent Symmetric Multimodal Preference Optimization13 Jun
  2. 2025LPOI: Listwise Preference Optimization for Vision Language Models27 May
  3. 2025CHiP: Cross-modal Hierarchical Direct Preference Optimization for Multimodal LLMs28 Jan
  4. 2025Mitigating Hallucinations in Large Vision-Language Models via DPO: On-Policy Data Hold the Key16 Jan
  5. 2024Understanding Alignment in Multimodal LLMs: A Comprehensive Study2 Jul
  6. 2024Detecting and Mitigating Hallucination in Large Vision Language Models via Fine-Grained AI Feedback22 Apr
  7. 2023Silkie: Preference Distillation for Large Visual Language Models17 Dec
  8. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback1 Dec
  9. 2023Aligning Large Multimodal Models with Factually Augmented RLHF25 Sep

Other references · 19

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  3. 2025Qwen2.5-VL Technical Reportcited by 5,587
  4. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  5. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  6. 2016A Diagram Is Worth A Dozen Imagescited by 804
  7. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 688
  8. 2018Object Hallucination in Image Captioningcited by 669
  9. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 478
  10. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluationcited by 300
  11. 2024Enhancing Large Vision Language Models with Self-Training on Image Comprehensioncited by 70
  12. 2024WPO: Enhancing RLHF with Weighted Preference Optimizationcited by 54
7 more
  1. 2024Enhancing Zero-Shot Vision Models by Label-Free Prompt Distribution Learning and Bias Correctingcited by 26
  2. 2025Enhancing CLIP Robustness via Cross-Modality Alignmentcited by 15
  3. 2025Res-Bench: Benchmarking the Robustness of Multimodal Large Language Models to Dynamic Resolution Inputcited by 9
  4. 2025AdPO: Enhancing the Adversarial Robustness of Large Vision-Language Models with Preference Optimizationcited by 8
  5. 2026Look Carefully: Adaptive Visual Reinforcements in Multimodal Large Language Models for Hallucination Mitigationcited by 7
  6. 2026Mitigating Hallucinations in Large Vision-Language Models without Performance Degradationcited by 3
  7. 2026Robustifying Vision-Language Models via Test-Time Prompt Adaptationcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025AdaViP: Aligning Multi-modal LLMs via Adaptive Vision-enhanced Preference Optimization
  2. 2024mDPO: Conditional Preference Optimization for Multimodal Large Language Models
  3. 2025Re-Align: Aligning Vision Language Models via Retrieval-Augmented Direct Preference Optimization
  4. 2026ACPO: Counteracting Likelihood Displacement in Vision-Language Alignment with Asymmetric Constraints
  5. 2024MIA-DPO: Multi-Image Augmented Direct Preference Optimization For Large Vision-Language Models
  6. 2025PerPO: Perceptual Preference Optimization via Discriminative Rewarding
  7. 2025Aligning Multimodal LLM with Human Preference: A Survey
  8. 2026LVRPO: Language-Visual Alignment with GRPO for Multimodal Understanding and Generation