mlsanityData to 14 Oct 2026

Reinforcement Learning with Rubric Anchors

arXiv 2508.12790 ↗First posted 18 Aug 2025cs.AI cs.CL cs.LGcited by 113

On the map: Deep reinforcement learning methods → RL training of language and agent models

Abstract (arXiv)

Reinforcement Learning from Verifiable Rewards (RLVR) has emerged as a powerful paradigm for enhancing Large Language Models (LLMs), exemplified by the success of OpenAI's o-series. In RLVR, rewards are derived from verifiable signals-such as passing unit tests in code generation or matching correct answers in mathematical reasoning. While effective, this requirement largely confines RLVR to domains with automatically checkable outcomes. To overcome this, we extend the RLVR paradigm to open-ended tasks by integrating rubric-based rewards, where carefully designed rubrics serve as structured, model-interpretable criteria for automatic scoring of subjective outputs. We construct, to our knowledge, the largest rubric reward system to date, with over 10,000 rubrics from humans, LLMs, or a hybrid human-LLM collaboration. Implementing rubric-based RL is challenging; we tackle these issues with a clear framework and present an open-sourced Qwen-30B-A3B model with notable gains: 1) With only 5K+ samples, our system improves by +5.2% on open-ended benchmarks (especially humanities), outperforming a 671B DeepSeek-V3 model by +2.4%, while preserving general and reasoning abilities. 2) Our method provides fine-grained stylistic control, using rubrics as anchors to mitigate the "AI-like" tone and produce more human-like, expressive responses. We share key lessons in rubric construction, data selection, and training, and discuss limitations and future releases.

Key facts. Cited by 113 papers in the map (data through 14 Oct 2026). Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods. Builds on 29 of its 35 references that are in the map. First posted on arXiv 18 Aug 2025.

Builds on

29 of 35 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains23 Jul

Other references · 28

  1. 2025Qwen3 Technical Reportcited by 8,820
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  6. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  7. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  8. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  9. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  10. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  11. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,210
  12. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
16 more
  1. 2018CoQA: A Conversational Question Answering Challengecited by 683
  2. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 429
  3. 2022Self-critiquing models for assisting human evaluatorscited by 380
  4. 2024Deliberative Alignment: Reasoning Enables Safer Language Modelscited by 308
  5. 2025Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learningcited by 262
  6. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoningcited by 250
  7. 2025Checklists Are Better Than Reward Models For Aligning Language Modelscited by 128
  8. 2024Rule Based Rewards for Language Model Safetycited by 108
  9. 2025WritingBench: A Comprehensive Benchmark for Generative Writingcited by 106
  10. 2023EQ-Bench: An Emotional Intelligence Benchmark for Large Language Modelscited by 80
  11. 2023COLLIE: Systematic Construction of Constrained Text Generation Taskscited by 70
  12. 2023SALMON: Self-Alignment with Instructable Reward Modelscited by 55
  13. 2025How Many Instructions Can LLMs Follow at Once?cited by 20
  14. 2025Safety Reasoning with Guidelinescited by 15
  15. 2018A Simple and Effective Approach to the Story Cloze Testcited by 10
  16. 2025Omni-Thinker: Scaling Multi-Task RL in LLMs with Hybrid Reward and Task Schedulingcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 22; 2026: 91.

113 papers.

Sort:

By area:

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  2. 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignmentcited by 132
  3. 2025Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoningcited by 63
  4. 2026Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Taskscited by 62
  5. 2025Baichuan-M2: Scaling Medical Capability with Large Verifier Systemcited by 59
  6. 2026Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Trainingcited by 54
  7. 2026Agent-World: Scaling Real-World Environment Synthesis for Evolving General Agent Intelligencecited by 54
  8. 2025Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Trainingcited by 53
  9. 2025AdvancedIF: Rubric-Based Benchmarking and Reinforcement Learning for Advancing LLM Instruction Followingcited by 51
  10. 2025PRBench: Large-Scale Expert Rubrics for Evaluating High-Stakes Professional Reasoningcited by 32
  11. 2026Open Rubric System: Scaling Reinforcement Learning with Pairwise Adaptive Rubriccited by 27
  12. 2026Reinforcing Chain-of-Thought Reasoning with Self-Evolving Rubricscited by 26
38 more
  1. 2026Reward Hacking in Rubric-Based Reinforcement Learningcited by 26
  2. 2025Learning to Optimize Multi-Objective Alignment Through Dynamic Reward Weightingcited by 23
  3. 2026Inference-Time Scaling of Verification: Self-Evolving Deep Research Agents via Test-Time Rubric-Guided Verificationcited by 22
  4. 2025Hybrid Reinforcement: When Reward Is Sparse, It's Better to Be Densecited by 20
  5. 2026WebGym: Scaling Training Environments for Visual Web Agents with Realistic Taskscited by 19
  6. 2026Learning Query-Specific Rubrics from Human Preferences for DeepResearch Report Generationcited by 18
  7. 2025DeepTravel: An End-to-End Agentic Reinforcement Learning Framework for Autonomous Travel Planning Agentscited by 17
  8. 2026ArenaRL: Scaling RL for Open-Ended Agents via Tournament-based Relative Rankingcited by 17
  9. 2026Reproducing, Analyzing, and Detecting Reward Hacking in Rubric-Based Reinforcement Learningcited by 16
  10. 2026Rubric-based On-policy Distillationcited by 13
  11. 2026Rethinking Reward Supervision: Rubric-Conditioned Self-Distillationcited by 13
  12. 2025Self-Rewarding Rubric-Based Reinforcement Learning for Open-Ended Reasoningcited by 11
  13. 2025AutoRubric: Rubric-Based Generative Rewards for Faithful Multimodal Reasoningcited by 11
  14. 2026Baichuan-M3: Modeling Clinical Inquiry for Reliable Medical Decision-Makingcited by 11
  15. 2026Self-Preference Bias in Rubric-Based Evaluation of Large Language Modelscited by 11
  16. 2026EvoRubric: Self-Evolving Rubric-Driven RL for Open-Ended Generationcited by 11
  17. 2026Quantifying the Utility of User Simulators for Building Collaborative LLM Assistantscited by 10
  18. 2026Rubric Dropout: A Simple Way to Mitigate Reward Hacking in Rubric-as-Reward RLcited by 9
  19. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 8
  20. 2025Reinforcement Learning for Tool-Integrated Interleaved Thinking towards Cross-Domain Generalizationcited by 7
  21. 2025RubricRL: Simple Generalizable Rewards for Text-to-Image Generationcited by 7
  22. 2025TableGPT-R1: Advancing Tabular Reasoning Through Reinforcement Learningcited by 7
  23. 2026SCOPE: Self-Play via Co-Evolving Policies for Open-Ended Taskscited by 7
  24. 2026BigFinanceBench: A Workflow-Grounded Benchmark for Financial-Research Agentscited by 7
  25. 2026TRCA: Transition-wise Rubric Credit Assignment for Long-horizon LLM Agentscited by 7
  26. 2025ClinTutor-R1: Advancing Scalable and Robust One-to-Many Alignment in Clinical Socratic Educationcited by 6
  27. 2026Grad2Reward: From Sparse Judgment to Dense Rewards for Improving Open-Ended LLM Reasoningcited by 6
  28. 2026SibylSense: Adaptive Rubric Learning via Memory Tuning and Adversarial Probingcited by 6
  29. 2026Alternating Reinforcement Learning with Contextual Rubric Rewards: Beyond the Scalarization Strategycited by 6
  30. 2026Lens: Rethinking Training Efficiency for Foundational Text-to-Image Modelscited by 6
  31. 2026DeepInnovator: Triggering the Innovative Capabilities of LLMscited by 5
  32. 2026RuCL: Stratified Rubric-Based Curriculum Learning for Multimodal Large Language Model Reasoningcited by 5
  33. 2026Verbalizing LLM's Higher-order Uncertainty via Imprecise Probabilitiescited by 5
  34. 2026Visual Preference Optimization with Rubric Rewardscited by 5
  35. 2026Bootstrapping Post-training Signals for Open-ended Tasks via Rubric-based Self-play on Pre-training Textcited by 5
  36. 2026JudgmentBench: Comparing Rubric and Preference Evaluation for Quality Assessmentcited by 5
  37. 2026Scaling Large Reasoning Models beyond Human Supervision: A Path toward Superintelligencecited by 5
  38. 2025Automated Refinement of Essay Scoring Rubrics for Language Models via Reflect-and-Revisecited by 4

50 most cited of 113 All 113 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Step-wise Rubric Rewards for LLM Reasoning
  2. 2026Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVR
  3. 2026Small Language Models as Judges for Rubric-Based Reinforcement Learning
  4. 2025Reward and Guidance through Rubrics: Promoting Exploration to Improve Multi-Domain Reasoning
  5. 2026ARES: Automated Rubric Synthesis for Scalable LLM Reinforcement Learning
  6. 2025Extending RLVR to Open-Ended Tasks via Verifiable Multiple-Choice Reformulation
  7. 2025Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domains
  8. 2026Rationale Matters: Learning Transferable Rubrics via Proxy-Guided Critique for VLM Reward Models