mlsanityData to 7 Oct 2026

Visual Preference Optimization with Rubric Rewards

arXiv 2604.13029 ↗First posted 14 Apr 2026cs.CV cs.AIcited by 5

On the map: LLM alignment / RLHF / preference optimisation → Diffusion and multimodal model alignment

Abstract (arXiv)

The effectiveness of Direct Preference Optimization (DPO) depends on preference data that reflect the quality differences that matter in multimodal tasks. Existing pipelines often rely on off-policy perturbations or coarse outcome-based signals, which are not well suited to fine-grained visual reasoning. We propose rDPO, a preference optimization framework based on instance-specific rubrics. For each image-instruction pair, we create a checklist-style rubric of essential and additional criteria to score responses from any possible policies. The instruction-rubric pool is built offline and reused during the construction of on-policy data. On public reward modeling benchmarks, rubric-based prompting massively improves a 30B-A3B judge and brings it close to GPT-5.4. On public downstream benchmarks, rubric-based filtering raises the macro average to 82.69, whereas outcome-based filtering drops it to 75.82 from 81.14. When evaluating scalability on a comprehensive benchmark, rDPO achieves 61.01, markedly outperforming the style-constrained baseline (52.36) and surpassing the 59.48 base model. Together, these results show that visual preference optimization benefits from combining on-policy data construction with instance-specific criterion-level feedback.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Diffusion and multimodal model alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 64 of its 71 references that are in the map. First posted on arXiv 14 Apr 2026.

Builds on

64 of 71 references.

Closest ancestors · 25

Same direction, newest first

  1. 2026Omni-RRM: Advancing Omni Reward Modeling via Automatic Rubric-Grounded Preference Synthesis31 Jan
  2. 2025Mitigating Hallucination Through Theory-Consistent Symmetric Multimodal Preference Optimization13 Jun
  3. 2025OViP: Online Vision-Language Preference Learning for VLM Hallucination21 May
  4. 2025Skywork-VL Reward: An Effective Reward Model for Multimodal Understanding and Reasoning12 May
  5. 2025Benchmarking Multimodal CoT Reward Model Stepwise by Visual Program9 Apr
  6. 2025From Captions to Rewards (CAREVL): Leveraging Large Language Model Experts for Enhanced Reward Modeling in Large Vision-Language Models8 Mar
  7. 2025Multimodal RewardBench: Holistic Evaluation of Reward Models for Vision Language Models20 Feb
  8. 2025Re-Align: Aligning Vision Language Models via Retrieval-Augmented Direct Preference Optimization18 Feb
  9. 2025MM-RLHF: The Next Step Forward in Multimodal LLM Alignment14 Feb
  10. 2025InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Model21 Jan
  11. 2025Mitigating Hallucinations in Large Vision-Language Models via DPO: On-Policy Data Hold the Key16 Jan
  12. 2024VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Models26 Nov
13 more
  1. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization15 Nov
  2. 2024V-DPO: Mitigating Hallucination in Large Vision Language Models via Vision-Guided Direct Preference Optimization5 Nov
  3. 2024MIA-DPO: Multi-Image Augmented Direct Preference Optimization For Large Vision-Language Models23 Oct
  4. 2024Modality-Fair Preference Optimization for Trustworthy MLLM Alignment20 Oct
  5. 2024mDPO: Conditional Preference Optimization for Multimodal Large Language Models17 Jun
  6. 2024RLAIF-V: Open-Source AI Feedback Leads to Super GPT-4V Trustworthiness27 May
  7. 2024Enhancing Visual-Language Modality Alignment in Large Vision Language Models via Self-Improvement24 May
  8. 2024Self-Supervised Visual Preference Alignment16 Apr
  9. 2024Strengthening Multimodal Large Language Model with Bootstrapped Preference Optimization13 Mar
  10. 2024Aligning Modalities in Vision Large Language Models via Preference Fine-tuning18 Feb
  11. 2023Silkie: Preference Distillation for Large Visual Language Models17 Dec
  12. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback1 Dec
  13. 2023Aligning Large Multimodal Models with Factually Augmented RLHF25 Sep

Other references · 39

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2025Qwen2.5-VL Technical Reportcited by 5,418
  4. 2025Qwen3-VL Technical Reportcited by 2,534
  5. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  6. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  7. 2016A Diagram Is Worth A Dozen Imagescited by 793
  8. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 385
  9. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 362
  10. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 328
  11. 2025PaperBench: Evaluating AI's Ability to Replicate AI Researchcited by 286
  12. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positivecited by 239
27 more
  1. 2024Disentangling Length from Quality in Direct Preference Optimizationcited by 194
  2. 2024LLaVA-Critic: Learning to Evaluate Multimodal Modelscited by 132
  3. 2024Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluationcited by 123
  4. 2024M3CoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-Thoughtcited by 120
  5. 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignmentcited by 118
  6. 2025Checklists Are Better Than Reward Models For Aligning Language Modelscited by 115
  7. 2025Reinforcement Learning with Rubric Anchorscited by 108
  8. 2025MiMo-VL Technical Reportcited by 106
  9. 2024LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Textscited by 81
  10. 2024WildVision: Evaluating Vision-Language Models in the Wild with Human Preferencescited by 76
  11. 2024Enhancing Large Vision Language Models with Self-Training on Image Comprehensioncited by 70
  12. 2025Kwai Keye-VL 1.5 Technical Reportcited by 68
  13. 2025Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoningcited by 56
  14. 2024Critic-V: VLM Critics Help Catch VLM Errors in Multimodal Reasoningcited by 50
  15. 2026Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Trainingcited by 45
  16. 2024Probing Visual Language Priors in VLMscited by 44
  17. 2026RubricHub: A Comprehensive and Highly Discriminative Rubric Dataset via Automated Coarse-to-Fine Generationcited by 42
  18. 2025Online Rubrics Elicitation from Pairwise Comparisonscited by 41
  19. 2025Judge Anything: MLLM as a Judge Across Any Modalitycited by 33
  20. 2024TextHawk2: A Large Vision-Language Model Excels in Bilingual OCR and Grounding with 16x Fewer Tokenscited by 32
  21. 2025PRBench: Large-Scale Expert Rubrics for Evaluating High-Stakes Professional Reasoningcited by 29
  22. 2024VisionArena: 230K Real World User-VLM Conversations with Preference Labelscited by 22
  23. 2024CARMO: Dynamic Criteria Generation for Context-Aware Reward Modellingcited by 21
  24. 2025ProfBench: Multi-Domain Rubrics requiring Professional Knowledge to Answer and Judgecited by 21
  25. 2025Multi-Crit: Benchmarking Multimodal Judges on Pluralistic Criteria-Followingcited by 14
  26. 2024Efficient Self-Improvement in Multimodal Large Language Models: A Model-Level Judge-Free Approachcited by 9
  27. 2025Optimizing LVLMs with On-Policy Data for Effective Hallucination Mitigationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026From Holistic Evaluation to Structured Criteria: Rubrics Across the Evolving LLM Landscapecited by 4
  2. 2026Reinforcement Learning with Robust Rubric Rewardscited by 2
  3. 2026Reinforcement Learning with Evolving Rubrics as Rewards for Audio Reasoningcited by 1
  4. 2026A Survey on Rubric-Guided Reinforcement Learning for Language Modelscited by 1
  5. 2026SciForma: Structure-Faithful Generation of Scientific Diagramscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Auto-Rubric as Reward: From Implicit Preferences to Explicit Multimodal Generative Criteria
  2. 2025Seeing What Matters: Visual Preference Policy Optimization for Visual Generation
  3. 2026V-Rubrics: Visual Faithfulness via Rubric-Based Reinforcement Learning
  4. 2026Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Tasks
  5. 2025Test-Time Reasoning Through Visual Human Preferences with VLMs and Soft Rewards
  6. 2026ViPO: Visual Preference Optimization at Scale
  7. 2024VisionReward: Fine-Grained Multi-Dimensional Human Preference Learning for Image and Video Generation
  8. 2025Delving into RL for Image Generation with CoT: A Study on DPO vs. GRPO