mlsanityData to 7 Oct 2026

Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoning

arXiv 2508.16949 ↗First posted 23 Aug 2025cs.LG cs.AIcited by 56

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Recent advances in Large Language Models (LLMs) have underscored the potential of Reinforcement Learning (RL) to facilitate the emergence of reasoning capabilities. Despite the encouraging results, a fundamental dilemma persists as RL improvement relies on learning from high-quality samples, yet the exploration for such samples remains bounded by the inherent limitations of LLMs. This, in effect, creates an undesirable cycle in which what cannot be explored cannot be learned. In this work, we propose Rubric-Scaffolded Reinforcement Learning (RuscaRL), a novel instructional scaffolding framework designed to break the exploration bottleneck for open-ended reasoning. Specifically, RuscaRL introduces checklist-style rubrics as (1) explicit scaffolding for exploration during rollout generation, where different rubrics are provided as external guidance within task instructions to steer diverse high-quality responses. This guidance is gradually decayed over time, encouraging the model to internalize the underlying reasoning patterns; (2) verifiable rewards for exploitation during model training, where we can obtain robust LLM-as-a-Judge scores using rubrics as references, enabling effective RL on open-ended reasoning tasks. Extensive experiments demonstrate the superiority of the proposed RuscaRL across various benchmarks, effectively expanding reasoning boundaries under the Best-of-N evaluation. Our code is available at https://github.com/IANNXANG/RuscaRL.

Key facts. Cited by 56 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 87 of its 114 references that are in the map. First posted on arXiv 23 Aug 2025.

Builds on

87 of 114 references.

Closest ancestors · 16

Same direction, newest first

  1. 2025RL-PLUS: Countering Capability Boundary Collapse of LLMs in Reinforcement Learning with Hybrid-policy Optimization31 Jul
  2. 2025Revisiting LLM Reasoning via Information Bottleneck24 Jul
  3. 2025First Return, Entropy-Eliciting Explore9 Jul
  4. 2025A Simple "Motivation" Can Enhance Reinforcement Finetuning of Large Reasoning Models23 Jun
  5. 2025Reasoning with Exploration: An Entropy Perspective17 Jun
  6. 2025Consistent Paths Lead to Truth: Self-Rewarding Reinforcement Learning for LLM Reasoning10 Jun
  7. 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models30 May
  8. 2025SeRL: Self-Play Reinforcement Learning for Large Language Models with Limited Data25 May
  9. 2025TTRL: Test-Time Reinforcement Learning22 Apr
  10. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  11. 2025Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining10 Apr
  12. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
4 more
  1. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov
  2. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Search6 Jun
  3. 2023Let's Verify Step by Step31 May
  4. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 71

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  7. 2025Qwen3 Technical Reportcited by 8,195
  8. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  9. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  10. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  11. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  12. 2024Qwen2.5 Technical Reportcited by 4,713
38 more
  1. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  2. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  3. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  4. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  5. 2020Learning to summarize from human feedbackcited by 2,562
  6. 2023A Survey of Large Language Modelscited by 2,537
  7. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  8. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  9. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  10. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  11. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  12. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  13. 2024A Survey on LLM-as-a-Judgecited by 1,384
  14. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  15. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,175
  16. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  17. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  18. 2022PAL: Program-aided Language Modelscited by 635
  19. 2022Towards Reasoning in Large Language Models: A Surveycited by 593
  20. 2022MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answeringcited by 525
  21. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 484
  22. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 389
  23. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 385
  24. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 342
  25. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 328
  26. 2025DeepSeek-Prover-V2: Advancing Formal Mathematical Reasoning via Reinforcement Learning for Subgoal Decompositioncited by 270
  27. 2023Foundation Models in Robotics: Applications, Challenges, and the Futurecited by 212
  28. 2023HuatuoGPT, towards Taming Language Model to Be a Doctorcited by 211
  29. 2023Large Language Models for Software Engineering: Survey and Open Problemscited by 178
  30. 2025Generalizing Verifiable Instruction Followingcited by 178
  31. 2024LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMscited by 142
  32. 2023Large Language Models in Law: A Surveycited by 128
  33. 2024Meta-Prompting: Enhancing Language Models with Task-Agnostic Scaffoldingcited by 115
  34. 2025Checklists Are Better Than Reward Models For Aligning Language Modelscited by 115
  35. 2025HealthGPT: A Medical Large Vision-Language Model for Unifying Comprehension and Generation via Heterogeneous Knowledge Adaptationcited by 112
  36. 2023LawBench: Benchmarking Legal Knowledge of Large Language Modelscited by 111
  37. 2025Reinforcement Learning with Rubric Anchorscited by 108
  38. 2025Seed-Prover: Deep and Broad Reasoning for Automated Theorem Provingcited by 96

50 most cited of 71 All 71 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 48.

56 papers.

Sort:

By area:

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  2. 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignmentcited by 118
  3. 2026Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Taskscited by 55
  4. 2025AdvancedIF: Rubric-Based Benchmarking and Reinforcement Learning for Advancing LLM Instruction Followingcited by 46
  5. 2026Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Trainingcited by 45
  6. 2026RubricHub: A Comprehensive and Highly Discriminative Rubric Dataset via Automated Coarse-to-Fine Generationcited by 42
  7. 2026Open Rubric System: Scaling Reinforcement Learning with Pairwise Adaptive Rubriccited by 26
  8. 2026Reinforcing Chain-of-Thought Reasoning with Self-Evolving Rubricscited by 23
  9. 2025Reward and Guidance through Rubrics: Promoting Exploration to Improve Multi-Domain Reasoningcited by 18
  10. 2026Reward Hacking in Rubric-Based Reinforcement Learningcited by 18
  11. 2025Language Models Can Learn from Verbal Feedback Without Scalar Rewardscited by 17
  12. 2025Quantile Advantage Estimation: Stabilizing RLVR for LLM Reasoningcited by 11
38 more
  1. 2026An Imperfect Verifier is Good Enough: Learning with Noisy Rewardscited by 11
  2. 2026Reproducing, Analyzing, and Detecting Reward Hacking in Rubric-Based Reinforcement Learningcited by 9
  3. 2026Rubrics to Tokens: Bridging Response-level Rubrics and Token-level Rewards in Instruction Following Taskscited by 8
  4. 2025Reinforcement Learning for Tool-Integrated Interleaved Thinking towards Cross-Domain Generalizationcited by 7
  5. 2026Omni-RRM: Advancing Omni Reward Modeling via Automatic Rubric-Grounded Preference Synthesiscited by 7
  6. 2025FAPO: Flawed-Aware Policy Optimization for Efficient and Reliable Reasoningcited by 6
  7. 2026Grad2Reward: From Sparse Judgment to Dense Rewards for Improving Open-Ended LLM Reasoningcited by 5
  8. 2026RuCL: Stratified Rubric-Based Curriculum Learning for Multimodal Large Language Model Reasoningcited by 5
  9. 2026Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learningcited by 5
  10. 2026Visual Preference Optimization with Rubric Rewardscited by 5
  11. 2026Bootstrapping Post-training Signals for Open-ended Tasks via Rubric-based Self-play on Pre-training Textcited by 5
  12. 2026Alternating Reinforcement Learning with Contextual Rubric Rewards: Beyond the Scalarization Strategycited by 4
  13. 2026From Holistic Evaluation to Structured Criteria: Rubrics Across the Evolving LLM Landscapecited by 4
  14. 2026CriPO: Enhancing Rubric-based RL via Self-Distillationcited by 4
  15. 2026Beyond Alignment: Expanding Reasoning Capacity via Manifold-Reshaping Policy Optimizationcited by 3
  16. 2026Rubric-Grounded RL: Structured Judge Rewards for Generalizable Reasoningcited by 3
  17. 2026Rubric-Guided Process Reward for Stepwise Model Routingcited by 2
  18. 2026Reinforcement Learning with Robust Rubric Rewardscited by 2
  19. 2026Deep Research as Rubric for Reinforcement Learningcited by 2
  20. 2026ARBOR: Online Process Rewards via a Reusable Rubric Buffer for Search Agentscited by 2
  21. 2026Rubric Dropout: A Simple Way to Mitigate Reward Hacking in Rubric-as-Reward RLcited by 2
  22. 2026TRCA: Transition-wise Rubric Credit Assignment for Long-horizon LLM Agentscited by 2
  23. 2026Scaling Large Reasoning Models beyond Human Supervision: A Path toward Superintelligencecited by 2
  24. 2026Experience is the Best Teacher: Motivating Effective Exploration in Reinforcement Learning for LLMscited by 1
  25. 2026Vocabulary Dropout for Curriculum Diversity in LLM Co-Evolutioncited by 1
  26. 2026Mid-Training with Self-Generated Data Improves Reinforcement Learning in Language Modelscited by 1
  27. 2026QUBRIC: Co-Designing Queries and Rubrics for RL Beyond Verifiable Rewardscited by 1
  28. 2026Instruction-Conditioned Exploration for Reinforcement Learning with Self-Distillation to an Unconditioned Policycited by 1
  29. 2026Reinforcement Learning with Evolving Rubrics as Rewards for Audio Reasoningcited by 1
  30. 2026RISE-RL: Rubric-Informed Selective Exploration for Open-Ended Reinforcement Learningcited by 1
  31. 2026Decoupled Physical Modeling and Execution for Physics Reasoningcited by 1
  32. 2026DiPO: Disentangled Perplexity Policy Optimization for Fine-grained Exploration-Exploitation Trade-Offcited by 0
  33. 2026Metacognition as Reward: Reinforcing LLM Reasoning via Knowledge and Regulation Signalscited by 0
  34. 2026Teach-to-Reason: Competition-Guided Reasoning with a Self-Improving Teachercited by 0
  35. 2026RubricReviewer: From Direct Critique to Objective and Comprehensive Rubric-Driven Peer Reviewcited by 0
  36. 2026PRISM: Priority-aware Rubric Internalization via Structured Multimodal Data Synthesiscited by 0
  37. 2026When Metrics Reward the Worst Translations: Internalizing Cultural Reasoning for Social Media Translation Evaluationcited by 0
  38. 2026RLVR2: Reinforcement Learning with Verifiable Rubric-based Rankingcited by 0

50 most cited of 56 All 56 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025From Trial-and-Error to Improvement: A Systematic Analysis of LLM Exploration Mechanisms in RLVR
  2. 2026Think-with-Rubrics: From External Evaluator to Internal Reasoning Guidance
  3. 2025Navigate the Unknown: Enhancing LLM Reasoning with Intrinsic Motivation Guided Exploration
  4. 2025Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective
  5. 2026Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learning
  6. 2025Improving RL Exploration for LLM Reasoning through Retrospective Replay
  7. 2025Can LLMs Guide Their Own Exploration? Gradient-Guided Reinforcement Learning for LLM Reasoning
  8. 2025Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM Reasoning via Autoregressive Search