mlsanityData to 14 Oct 2026

Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains

arXiv 2507.17746 ↗First posted 23 Jul 2025cs.LG cs.AI cs.CLcited by 366

On the map: Deep reinforcement learning methods → RL training of language and agent models

Abstract (arXiv)

Reinforcement Learning with Verifiable Rewards (RLVR) has proven effective for complex reasoning tasks with clear correctness signals such as math and coding. However, extending it to real-world reasoning tasks is challenging, as evaluation depends on nuanced, multi-criteria judgments rather than binary correctness. Instance-specific rubrics have recently been used in evaluation benchmarks to capture such judgments, but their potential as reward signals for on-policy post-training remains underexplored. We introduce Rubrics as Rewards (RaR), an on-policy reinforcement learning method that extends RLVR beyond verifiable domains by using rubric-based feedback. Across both medical and science domains, we evaluate multiple strategies for aggregating rubric feedback into rewards. The best RaR variant achieves relative improvements of up to 31% on HealthBench and 7% on GPQA-Diamond over popular LLM-as-judge baselines that rely on direct Likert-based rewards. These results demonstrate that RaR-trained policies adapt well to diverse evaluation formats, performing strongly on both rubric-based and multiple-choice tasks. Moreover, we find that using rubrics as structured reward signals yields better alignment for smaller judges and reduces performance variance across judge scales.

Key facts. Cited by 366 papers in the map (data through 14 Oct 2026). Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods. Builds on 35 of its 42 references that are in the map. First posted on arXiv 23 Jul 2025.

Builds on

35 of 42 references.

References · 35

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  5. 2024GPT-4o System Cardcited by 4,588
  6. 2023Let's Verify Step by Stepcited by 3,789
  7. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  8. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  9. 2025Process Reinforcement through Implicit Rewardscited by 504
  10. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 446
  11. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 429
  12. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 291
23 more
  1. 2024HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMscited by 265
  2. 2023The False Promise of Imitating Proprietary LLMscited by 263
  3. 2025Learning to Reason without External Rewardscited by 210
  4. 2025MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMscited by 180
  5. 2025Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domainscited by 156
  6. 2025RM-R1: Reward Modeling as Reasoningcited by 154
  7. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewardscited by 148
  8. 2024ODIN: Disentangled Reward Mitigates Hacking in RLHFcited by 136
  9. 2025General-Reasoner: Advancing LLM Reasoning Across All Domainscited by 135
  10. 2025Process Reward Models That Thinkcited by 118
  11. 2024LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Textscited by 87
  12. 2025J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learningcited by 75
  13. 2025Reward Reasoning Modelcited by 47
  14. 2024Improving Reward Models with Synthetic Critiquescited by 44
  15. 2025Med-RLVR: Emerging Medical Reasoning from a 3B base model via reinforcement Learningcited by 40
  16. 2025SCP-116K: A High-Quality Problem-Solution Dataset and a Generalized Pipeline for Automated Extraction in the Higher Education Science Domaincited by 31
  17. 2025Rubric Is All You Need: Enhancing LLM-based Code Evaluation With Question-Specific Rubricscited by 29
  18. 2025QA-LIGN: Aligning LLMs through Constitutionally Decomposed QAcited by 28
  19. 2025ExpertLongBench: Benchmarking Language Models on Expert-Level Long-Form Generation Tasks with Structured Checklistscited by 24
  20. 2025R3: Robust Rubric-Agnostic Reward Modelscited by 23
  21. 2025Enhancing Reasoning through Process Supervision with Monte Carlo Tree Searchcited by 14
  22. 2025Configurable Preference Tuning with Rubric-Guided Synthetic Datacited by 8
  23. 2025Toward Evaluative Thinking: Meta Policy Optimization with Evolving Reward Modelscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 56; 2026: 310.

366 papers.

Sort:

By area:

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  2. 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignmentcited by 132
  3. 2025Reinforcement Learning with Rubric Anchorscited by 113
  4. 2025On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weightingcited by 91
  5. 2025On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Modelscited by 69
  6. 2025Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoningcited by 63
  7. 2026Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Taskscited by 62
  8. 2026Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Trainingcited by 54
  9. 2025Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Trainingcited by 53
  10. 2026RubricHub: A Comprehensive and Highly Discriminative Rubric Dataset via Automated Coarse-to-Fine Generationcited by 52
  11. 2025AdvancedIF: Rubric-Based Benchmarking and Reinforcement Learning for Advancing LLM Instruction Followingcited by 51
  12. 2025Online Rubrics Elicitation from Pairwise Comparisonscited by 46
38 more
  1. 2025Seed-Prover 1.5: Mastering Undergraduate-Level Theorem Proving via Learning from Experiencecited by 44
  2. 2025CapRL: Stimulating Dense Image Caption Capabilities via Reinforcement Learningcited by 38
  3. 2025A Survey of Process Reward Models: From Outcome Signals to Process Supervisions for Large Language Modelscited by 38
  4. 2026Gaia2: Benchmarking LLM Agents on Dynamic and Asynchronous Environmentscited by 33
  5. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 33
  6. 2025PRBench: Large-Scale Expert Rubrics for Evaluating High-Stakes Professional Reasoningcited by 32
  7. 2025Beyond Seeing: Evaluating Multimodal LLMs on Tool-Enabled Image Perception, Transformation, and Reasoningcited by 29
  8. 2026RubricBench: Aligning Model-Generated Rubrics with Human Standardscited by 28
  9. 2026Aligning Language Models from User Interactionscited by 28
  10. 2026Qwen-AgentWorld: Language World Models for General Agentscited by 28
  11. 2026Open Rubric System: Scaling Reinforcement Learning with Pairwise Adaptive Rubriccited by 27
  12. 2026Reinforcing Chain-of-Thought Reasoning with Self-Evolving Rubricscited by 26
  13. 2026Reward Hacking in Rubric-Based Reinforcement Learningcited by 26
  14. 2026Agentic Rubrics as Contextual Verifiers for SWE Agentscited by 25
  15. 2025ProfBench: Multi-Domain Rubrics requiring Professional Knowledge to Answer and Judgecited by 24
  16. 2025MoReBench: Evaluating Procedural and Pluralistic Moral Reasoning in Language Models, More than Outcomescited by 23
  17. 2026Inference-Time Scaling of Verification: Self-Evolving Deep Research Agents via Test-Time Rubric-Guided Verificationcited by 22
  18. 2025Goal Alignment in LLM-Based User Simulators for Conversational AIcited by 21
  19. 2025Reward and Guidance through Rubrics: Promoting Exploration to Improve Multi-Domain Reasoningcited by 21
  20. 2026Learning to Self-Verify Makes Language Models Better Reasonerscited by 21
  21. 2026Odysseys: Benchmarking Web Agents on Realistic Long Horizon Taskscited by 21
  22. 2026RubricEval: A Rubric-Level Meta-Evaluation Benchmark for LLM Judges in Instruction Followingcited by 20
  23. 2026WebGym: Scaling Training Environments for Visual Web Agents with Realistic Taskscited by 19
  24. 2026From Context to Skills: Can Language Models Learn from Context Skillfully?cited by 19
  25. 2025Compute as Teacher: Turning Inference Compute Into Reference-Free Supervisioncited by 18
  26. 2026Learning Query-Specific Rubrics from Human Preferences for DeepResearch Report Generationcited by 18
  27. 2026Intelligent AI Delegationcited by 18
  28. 2026Reproducing, Analyzing, and Detecting Reward Hacking in Rubric-Based Reinforcement Learningcited by 16
  29. 2026Rubric-Guided Self-Distillation: Post-Training Without Rubric Verifierscited by 15
  30. 2026Chaining the Evidence: Robust Reinforcement Learning for Deep Search Agents with Citation-Aware Rubric Rewardscited by 14
  31. 2026CM2: Reinforcement Learning with Checklist Rewards for Multi-Turn and Multi-Step Agentic Tool Usecited by 14
  32. 2026Step-wise Rubric Rewards for LLM Reasoningcited by 14
  33. 2025AutoSCORE: Enhancing Automated Scoring with Multi-Agent Large Language Models via Structured Component Recognitioncited by 13
  34. 2025Environment Scaling for Interactive Agentic Experience Collection: A Surveycited by 13
  35. 2026Examining Reasoning LLMs-as-Judges in Non-Verifiable LLM Post-Trainingcited by 13
  36. 2026VisBrowse-Bench: Benchmarking Visual-Native Search for Multimodal Browsing Agentscited by 13
  37. 2026An Imperfect Verifier is Good Enough: Learning with Noisy Rewardscited by 13
  38. 2026EvoLM: Self-Evolving Language Models through Co-Evolved Discriminative Rubricscited by 13

50 most cited of 366 All 366 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026ComplexConstraints and Beyond: Expert Rubrics for RLVR
  2. 2026A Survey on Rubric-Guided Reinforcement Learning for Language Models
  3. 2026Rubrics to Tokens: Bridging Response-level Rubrics and Token-level Rewards in Instruction Following Tasks
  4. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs
  5. 2025Direct Reasoning Optimization: Token-Level Reasoning Reflectivity Meets Rubric Gates for Unverifiable Tasks
  6. 2025RLPR: Extrapolating RLVR to General Domains without Verifiers
  7. 2025Position: The Hidden Costs and Measurement Gaps of Reinforcement Learning with Verifiable Rewards
  8. 2025Auto-Rubric: Learning From Implicit Weights to Explicit Rubrics for Reward Modeling