mlsanityData to 13 Oct 2026

Reward and Guidance through Rubrics: Promoting Exploration to Improve Multi-Domain Reasoning

arXiv 2511.12344 ↗First posted 15 Nov 2025cs.AIcited by 19

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Recent advances in reinforcement learning (RL) have significantly improved the complex reasoning capabilities of large language models (LLMs). Despite these successes, existing methods mainly focus on single-domain RL (e.g., mathematics) with verifiable rewards (RLVR), and their reliance on purely online RL frameworks restricts the exploration space, thereby limiting reasoning performance. In this paper, we address these limitations by leveraging rubrics to provide both fine-grained reward signals and offline guidance. We propose RGR-GRPO (Reward and Guidance through Rubrics), a rubric-driven RL framework for multi-domain reasoning. RGR-GRPO enables LLMs to receive dense and informative rewards while exploring a larger solution space during GRPO training. Extensive experiments across 14 benchmarks spanning multiple domains demonstrate that RGR-GRPO consistently outperforms RL methods that rely solely on alternative reward schemes or offline guidance. Compared with verifiable online RL baseline, RGR-GRPO achieves average improvements of +7.0%, +5.4%, +8.4%, and +6.6% on mathematics, physics, chemistry, and general reasoning tasks, respectively. Notably, RGR-GRPO maintains stable entropy fluctuations during off-policy training and achieves superior pass@k performance, reflecting sustained exploration and effective breakthrough beyond existing performance bottlenecks.

Key facts. Cited by 19 papers in the map (data through 13 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 40 of its 52 references that are in the map. First posted on arXiv 15 Nov 2025.

Builds on

40 of 52 references.

Closest ancestors · 18

Same direction, newest first

  1. 2025Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoning23 Aug
  2. 2025RL-PLUS: Countering Capability Boundary Collapse of LLMs in Reinforcement Learning with Hybrid-policy Optimization31 Jul
  3. 2025First Return, Entropy-Eliciting Explore9 Jul
  4. 2025A Simple "Motivation" Can Enhance Reinforcement Finetuning of Large Reasoning Models23 Jun
  5. 2025Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective17 Jun
  6. 2025Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedback3 Jun
  7. 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models30 May
  8. 2025General-Reasoner: Advancing LLM Reasoning Across All Domains20 May
  9. 2025Learning to Reason under Off-Policy Guidance21 Apr
  10. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  11. 2025Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining10 Apr
  12. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
6 more
  1. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  2. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  3. 2025Process Reinforcement through Implicit Rewards3 Feb
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  5. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  6. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 22

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,091
  2. 2025Qwen3 Technical Reportcited by 8,510
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,876
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,270
  5. 2024OpenAI o1 System Cardcited by 1,233
  6. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 404
  7. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 345
  8. 2023SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Modelscited by 250
  9. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hackingcited by 196
  10. 2025Checklists Are Better Than Reward Models For Aligning Language Modelscited by 116
  11. 2025The Invisible Leash: Why RLVR May or May Not Escape Its Origincited by 95
  12. 2025Reward Shaping to Mitigate Reward Hacking in RLHFcited by 88
10 more
  1. 2024TICKing All the Boxes: Generated Checklists Improve LLM Evaluation and Generationcited by 65
  2. 2025MegaScience: Pushing the Frontiers of Post-Training Datasets for Science Reasoningcited by 58
  3. 2025e3: Learning to Explore Enables Extrapolation of Test-Time Compute for LLMscited by 57
  4. 2025Code to Think, Think to Code: A Survey on Code-Enhanced Reasoning and Reasoning-Driven Code Intelligence in LLMscited by 47
  5. 2024Are large language models superhuman chemists?cited by 34
  6. 2024CS-Bench: A Comprehensive Benchmark for Large Language Models towards Computer Science Masterycited by 30
  7. 2025Compute as Teacher: Turning Inference Compute Into Reference-Free Supervisioncited by 18
  8. 2025ThinkTuning: Instilling Cognitive Reflections without Distillationcited by 9
  9. 2025Rubrik's Cube: Testing a New Rubric for Evaluating Explanations on the CUBE datasetcited by 8
  10. 2025Datasheets Aren't Enough: DataRubrics for Automated Quality Metrics and Accountabilitycited by 6

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2026: 19.

19 papers.

Sort:

By area:

  1. 2026Open Rubric System: Scaling Reinforcement Learning with Pairwise Adaptive Rubriccited by 27
  2. 2026Step-wise Rubric Rewards for LLM Reasoningcited by 12
  3. 2026Rethinking Reward Supervision: Rubric-Conditioned Self-Distillationcited by 10
  4. 2026What if Agents Could Imagine? Reinforcing Open-Vocabulary HOI Comprehension through Generationcited by 7
  5. 2026Gated Differentiable Working Memory for Long-Context Language Modelingcited by 6
  6. 2026Grad2Reward: From Sparse Judgment to Dense Rewards for Improving Open-Ended LLM Reasoningcited by 5
  7. 2026Bootstrapping Post-training Signals for Open-ended Tasks via Rubric-based Self-play on Pre-training Textcited by 5
  8. 2026From Holistic Evaluation to Structured Criteria: Rubrics Across the Evolving LLM Landscapecited by 4
  9. 2026SAW: Stage-Aware Dynamic Weighting for Multi-Objective Reinforcement Learning in Large Language Modelscited by 3
  10. 2026Reinforcement Learning with Evolving Rubrics as Rewards for Audio Reasoningcited by 3
  11. 2026Reward Design for Physical Reasoning in Vision-Language Modelscited by 2
  12. 2026HE-SNR: Uncovering Latent Logic via Entropy for Guiding Mid-Training on SWE-benchcited by 1
7 more
  1. 2026IndusAgent: Reinforcing Open-Vocabulary Industrial Anomaly Detection with Agentic Toolscited by 1
  2. 2026RISE-RL: Rubric-Informed Selective Exploration for Open-Ended Reinforcement Learningcited by 1
  3. 2026Emerging from Ground: Addressing Intent Deviation in Tool-Using Agents via Deriving Real Calls into Virtual Trajectoriescited by 0
  4. 2026APTER: Adaptive Post-Training with Expert-Grounded Rubricscited by 0
  5. 2026RULER: Instance-aware Rubric Rewards for SVG Generationcited by 0
  6. 2026From Judgment Quality to Downstream Utility: Rethinking LLM-as-a-Judge for Open-Ended Taskscited by 0
  7. 2026VepAgent: Bridging Causal-Transition via Tool-Augmented Reinforcement Learning for Video Event Predictioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Reinforcement Learning with Rubric Anchors
  2. 2026Rubric-Grounded RL: Structured Judge Rewards for Generalizable Reasoning
  3. 2025RLPR: Extrapolating RLVR to General Domains without Verifiers
  4. 2025RM-R1: Reward Modeling as Reasoning
  5. 2026DARL: Encouraging Diverse Answers for General Reasoning without Verifiers
  6. 2026To Mix or To Merge: Toward Multi-Domain Reinforcement Learning for Large Language Models
  7. 2025Can One Domain Help Others? A Data-Centric Study on Multi-Domain Reasoning via Reinforcement Learning
  8. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs