mlsanityData to 7 Oct 2026

DeltaRubric: Generative Multimodal Reward Modeling via Joint Planning and Verification

arXiv 2605.09269 ↗First posted 10 May 2026cs.CL cs.CVcited by 2

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

also in 1 more
Abstract (arXiv)

Aligning Multimodal Large Language Models (MLLMs) requires reliable reward models, yet existing single-step evaluators can suffer from lazy judging, exploiting language priors over fine-grained visual verification. While rubric-based evaluation mitigates these biases in text-only settings, extending it to multimodal tasks is bottlenecked by the complexity of visual reasoning. The critical differences between responses often depend on instance-specific visual details. Robust evaluation requires dynamically synthesizing rubrics that isolate spatial and factual discrepancies. To address this, we introduce DeltaRubric, an approach that reformulates multimodal preference evaluation as a plan-and-execute process within a single MLLM. DeltaRubric operates in two steps: acting first as a textitDisagreement Planner, the model generates a neutral, instance-specific verification checklist. Transitioning into a textitChecklist Verifier, it executes these self-generated checks against the image and question to produce the final grounded judgment. We formulate DeltaRubric as a multi-role reinforcement learning problem, jointly optimizing planning and verification capabilities. Validated on Qwen3-VL 4B and 8B Instruct models, DeltaRubric achieves solid empirical gains. For instance, On VL-RewardBench, it improves base model overall accuracy by +22.6 (4B) and +18.8 (8B) points, largely outperforming standard no-rubric baselines. The results demonstrate that decomposing evaluation into structured, verifiable steps leads to more reliable and generalizable multimodal reward modeling.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 49 of its 53 references that are in the map. First posted on arXiv 10 May 2026.

Builds on

49 of 53 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models25 Sep

Other references · 48

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  8. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  9. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  10. 2025Qwen3-VL Technical Reportcited by 2,534
  11. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  12. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
36 more
  1. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 877
  2. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  3. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  4. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  5. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 468
  6. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 404
  7. 2022Self-critiquing models for assisting human evaluatorscited by 372
  8. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 362
  9. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 328
  10. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 285
  11. 2024ReFT: Reasoning with Reinforced Fine-Tuningcited by 236
  12. 2025VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interactioncited by 191
  13. 2024LLaVA-Critic: Learning to Evaluate Multimodal Modelscited by 132
  14. 2024RLAIF-V: Open-Source AI Feedback Leads to Super GPT-4V Trustworthinesscited by 129
  15. 2024NVLM: Open Frontier-Class Multimodal LLMscited by 124
  16. 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignmentcited by 118
  17. 2025Checklists Are Better Than Reward Models For Aligning Language Modelscited by 115
  18. 2025Reinforcement Learning with Rubric Anchorscited by 108
  19. 2025DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Researchcited by 98
  20. 2025Self-Rewarding Vision-Language Model via Reasoning Decompositioncited by 95
  21. 2025Unified Multimodal Chain-of-Thought Reward Model through Reinforcement Fine-Tuningcited by 86
  22. 2024VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Modelscited by 85
  23. 2025MM-RLHF: The Next Step Forward in Multimodal LLM Alignmentcited by 79
  24. 2025R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement Learningcited by 65
  25. 2025Parallel-R1: Towards Parallel Thinking via Reinforcement Learningcited by 59
  26. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  27. 2025Reinforcement Learning from Human Feedbackcited by 53
  28. 2026Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Trainingcited by 45
  29. 2025CDE: Curiosity-Driven Exploration for Efficient Reinforcement Learning in Large Language Modelscited by 28
  30. 2026Reinforcing Chain-of-Thought Reasoning with Self-Evolving Rubricscited by 23
  31. 2024Benchmarking Large Multimodal Models against Common Corruptionscited by 22
  32. 2025BaseReward: A Strong Baseline for Multimodal Reward Modelcited by 15
  33. 2025ARM-Thinker: Reinforcing Multimodal Generative Reward Models with Agentic Tool Use and Visual Reasoningcited by 13
  34. 2025AutoRubric: Rubric-Based Generative Rewards for Faithful Multimodal Reasoningcited by 11
  35. 2025Stable and Efficient Single-Rollout RL for Multimodal Reasoningcited by 8
  36. 2026MSRL: Scaling Generative Multimodal Reward Modeling via Multi-Stage Reinforcement Learningcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026From Holistic Evaluation to Structured Criteria: Rubrics Across the Evolving LLM Landscapecited by 4
  2. 2026Align Then Reason: A Multimodal Lip-Sync Judge for Dubbingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Multimodal RewardBench: Holistic Evaluation of Reward Models for Vision Language Models
  2. 2025Multimodal RewardBench 2: Evaluating Omni Reward Models for Interleaved Text and Image
  3. 2025StructVRM: Aligning Multimodal Reasoning with Structured and Verifiable Reward Models
  4. 2025Flex-Judge: Text-Only Reasoning Unleashes Zero-Shot Multimodal Evaluators
  5. 2026Reinforcing Multimodal Reasoning via Token-Level Perception-Grounded Advantage Estimation
  6. 2026MJ1: Multimodal Judgment via Grounded Verification
  7. 2025Self-Rewarded Multimodal Coherent Reasoning Across Diverse Visual Domains
  8. 2026DeltaV: Thinking with Visual State Updates in Unified Large Multimodal Models