mlsanityData to 7 Oct 2026

PerPO: Perceptual Preference Optimization via Discriminative Rewarding

arXiv 2502.04371 ↗First posted 5 Feb 2025cs.AI cs.CL cs.LGcited by 9

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 1 more
Abstract (arXiv)

This paper presents Perceptual Preference Optimization (PerPO), a perception alignment method aimed at addressing the visual discrimination challenges in generative pre-trained multimodal large language models (MLLMs). To align MLLMs with human visual perception process, PerPO employs discriminative rewarding to gather diverse negative samples, followed by listwise preference optimization to rank them.By utilizing the reward as a quantitative margin for ranking, our method effectively bridges generative preference optimization and discriminative empirical risk minimization. PerPO significantly enhances MLLMs' visual discrimination capabilities while maintaining their generative strengths, mitigates image-unconditional reward hacking, and ensures consistent performance across visual tasks. This work marks a crucial step towards more perceptually aligned and versatile MLLMs. We also hope that PerPO will encourage the community to rethink MLLM alignment strategies.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 23 of its 86 references that are in the map. First posted on arXiv 5 Feb 2025.

Builds on

23 of 86 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Understanding Alignment in Multimodal LLMs: A Comprehensive Study2 Jul
  2. 2024mDPO: Conditional Preference Optimization for Multimodal Large Language Models17 Jun
  3. 2024Self-Supervised Visual Preference Alignment16 Apr
  4. 2023Merlin:Empowering Multimodal LLMs with Foresight Minds30 Nov
  5. 2023Aligning Large Multimodal Models with Factually Augmented RLHF25 Sep

Other references · 18

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  5. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  6. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  7. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  8. 2017Focal Loss for Dense Object Detectioncited by 749
  9. 2024KTO: Model Alignment as Prospect Theoretic Optimizationcited by 256
  10. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positivecited by 239
  11. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimizationcited by 223
  12. 2024General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Modelcited by 174
6 more
  1. 2024LiPO: Listwise Preference Optimization through Learning-to-Rankcited by 94
  2. 2023Vary: Scaling up the Vision Vocabulary for Large Vision-Language Modelscited by 84
  3. 2024A Survey on Multilingual Large Language Models: Corpora, Alignment, and Biascited by 82
  4. 2024Focus Anywhere for Fine-grained Multi-page Document Understandingcited by 62
  5. 2024sDPO: Don't Use Your Data All at Oncecited by 46
  6. 2024Slow Perception: Let's Perceive Geometric Figures Step-by-stepcited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 3.

9 papers.

  1. 2025Perception-R1: Pioneering Perception Policy with Reinforcement Learningcited by 116
  2. 2025Open Vision Reasoner: Transferring Linguistic Cognitive Behavior for Visual Reasoningcited by 34
  3. 2025Unhackable Temporal Rewarding for Scalable Video MLLMscited by 33
  4. 2025Perception in Reflectioncited by 13
  5. 2025Reinforcement Learning Meets Masked Generative Models: Mask-GRPO for Text-to-Image Generationcited by 12
  6. 2025Thinking by Doing: Building Efficient World Model Reasoning in LLMs via Multi-turn Interactioncited by 2
  7. 2026Multimodal Reward Hacking in Reinforcement Learningcited by 2
  8. 2026From Imitation to Intuition: Intrinsic Reasoning for Open-Instance Video Classificationcited by 0
  9. 2026From Structure to Synergy: A Survey of Vision-Language Perception Paradigm Evolution in Multimodal Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026PEA-DPO: Perception-Enhanced Alignment Direct Preference Optimization for MLLMs Alignment
  2. 2025Seeing What Matters: Visual Preference Policy Optimization for Visual Generation
  3. 2024MIA-DPO: Multi-Image Augmented Direct Preference Optimization For Large Vision-Language Models
  4. 2026ACPO: Counteracting Likelihood Displacement in Vision-Language Alignment with Asymmetric Constraints
  5. 2024Task Preference Optimization: Improving Multimodal Large Language Models with Vision Task Alignment
  6. 2026LVRPO: Language-Visual Alignment with GRPO for Multimodal Understanding and Generation
  7. 2025AdaViP: Aligning Multi-modal LLMs via Adaptive Vision-enhanced Preference Optimization
  8. 2025Aligning Large Vision-Language Models by Deep Reinforcement Learning and Direct Preference Optimization