Reinforcement Learning with Robust Rubric Rewards
On the map: LLM reasoning → Reinforcement learning for reasoning
Abstract (arXiv)
While Reinforcement Learning with Verifiable Rewards (RLVR) is effective for deterministically checkable tasks, many vision-language tasks are partially verifiable, demanding multi-criteria supervision (e.g., perceptual details, reasoning steps, and constraints). Rubrics provide a natural interface for this fine-grained supervision, but their effectiveness depends on the execution accuracy during online RL. We propose Reinforcement Learning with Robust Rubric Rewards (RLR3), extending RLVR from task-level verification to criterion-level verification. RLR3 routes instance-specific rubrics through two execution paths: an LLM-as-an-extractor paired with a deterministic verifier, or an LLM-as-a-Judge for non-verifiable criteria. To ensure faithful scoring, RLR3 introduce a minimal exposure strategy that masks ground truths from extractors and images from judges. Furthermore, RLR3 employs hierarchical aggregation to prioritize essential criteria over additional criteria, and mitigates score saturation within rollout groups. Evaluated on Qwen3-VL-30B-A3B across 15 benchmarks, RLR3 consistently outperforms RLVR, yielding a 4.7-point improvement over the base model and exceeding the official instruct-to-thinking model gap. Controlled audits confirm our deterministic verification and minimal exposure significantly reduce exploitable false positives.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 36 of its 41 references that are in the map. First posted on arXiv 28 May 2026.
Builds on
36 of 41 references.
Closest ancestors · 2
Same direction, newest first
- 2025Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoning23 Aug
- 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
Other references · 34
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
- 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
- 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
- 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
- 2024PaliGemma: A versatile 3B VLM for transfercited by 679
- 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 573
- 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Datasetcited by 516
- 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Modelcited by 515
- 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
- 2026Kimi K2.5: Visual Agentic Intelligencecited by 449
- 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 446
- 2021InfographicVQAcited by 430
22 more
- 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 385
- 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 328
- 2025VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learningcited by 315
- 2025PaperBench: Evaluating AI's Ability to Replicate AI Researchcited by 286
- 2024We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?cited by 281
- 2024DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Modelscited by 143
- 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignmentcited by 118
- 2025Checklists Are Better Than Reward Models For Aligning Language Modelscited by 115
- 2025Reinforcement Learning with Rubric Anchorscited by 108
- 2024LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Textscited by 81
- 2025SimpleVQA: Multimodal Factuality Evaluation for Multimodal Large Language Modelscited by 61
- 2026Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Trainingcited by 45
- 2026RubricHub: A Comprehensive and Highly Discriminative Rubric Dataset via Automated Coarse-to-Fine Generationcited by 42
- 2025Online Rubrics Elicitation from Pairwise Comparisonscited by 41
- 2025OpenMMReasoner: Pushing the Frontiers for Multimodal Reasoning with an Open and General Recipecited by 37
- 2025Judge Anything: MLLM as a Judge Across Any Modalitycited by 33
- 2025PRBench: Large-Scale Expert Rubrics for Evaluating High-Stakes Professional Reasoningcited by 29
- 2025ProfBench: Multi-Domain Rubrics requiring Professional Knowledge to Answer and Judgecited by 21
- 2025Multi-Crit: Benchmarking Multimodal Judges on Pluralistic Criteria-Followingcited by 14
- 2026Omni-RRM: Advancing Omni Reward Modeling via Automatic Rubric-Grounded Preference Synthesiscited by 7
- 2026DeepVision-103K: A Visually Diverse, Broad-Coverage, and Verifiable Mathematical Dataset for Multimodal Reasoningcited by 5
- 2026Visual Preference Optimization with Rubric Rewardscited by 5
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2026Step-wise Rubric Rewards for LLM Reasoning
- 2026Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVR
- 2026Rationale Matters: Learning Transferable Rubrics via Proxy-Guided Critique for VLM Reward Models
- 2026Small Language Models as Judges for Rubric-Based Reinforcement Learning
- 2026Alternating Reinforcement Learning with Contextual Rubric Rewards: Beyond the Scalarization Strategy
- 2026Rubric-Grounded RL: Structured Judge Rewards for Generalizable Reasoning
- 2026Open Rubric System: Scaling Reinforcement Learning with Pairwise Adaptive Rubric
- 2026Reward Hacking in Rubric-Based Reinforcement Learning