mlsanityData to 14 Oct 2026

TemplateRL: Structured Template-Guided Reinforcement Learning for LLM Reasoning

arXiv 2505.15692 ↗First posted 21 May 2025cs.CL cs.LGcited by 28

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning (RL) has emerged as an effective paradigm for enhancing model reasoning. However, existing RL methods like GRPO typically rely on unstructured self-sampling to fit scalar rewards, often producing inefficient rollouts that fail to capture transferable problem-solving strategies. To address this limitation, we propose **TemplateRL**, a structured template-guided RL framework that augments policy optimization with explicit template guidance. Our approach first constructs a problem-solving template library via MCTS on a small seed set, then seamlessly integrates this high-level structured guidance into RL training. By guiding rollout generation to align with proven template structures, TemplateRL significantly improves high-quality trajectory hit rates while reducing ineffective exploration. This structure-guided design steers the policy toward validated strategic patterns, stabilizing training dynamics, and enhancing RL sampling efficiency. Notably, the explicit template library is interpretable, editable, and supports online updates-enabling continuous updates during both training and inference. Extensive experiments demonstrate that TemplateRL outperforms GRPO by 99% on AIME and 41% on AMC, with superior stability on weak models and remarkable cross-domain generalization, highlighting its potential for broader tasks.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 31 of its 66 references that are in the map. First posted on arXiv 21 May 2025.

Builds on

31 of 66 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Example29 Apr
  2. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  3. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  4. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  5. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  6. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs3 Mar

Other references · 25

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  3. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  4. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  5. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  6. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  7. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  8. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  9. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  10. 2023Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Modelscited by 574
  11. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 558
  12. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Datasetcited by 526
13 more
  1. 2021Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoningcited by 383
  2. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 354
  3. 2025rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinkingcited by 313
  4. 2021Mastering Atari Games with Limited Datacited by 237
  5. 2023Evaluating the Performance of Large Language Models on GAOKAO Benchmarkcited by 186
  6. 2017Equivalence Between Policy Gradients and Soft Q-Learningcited by 173
  7. 2024BALROG: Benchmarking Agentic LLM and VLM Reasoning On Gamescited by 146
  8. 2024MathScale: Scaling Instruction Tuning for Mathematical Reasoningcited by 144
  9. 2024O1 Replication Journey: A Strategic Progress Report -- Part 1cited by 143
  10. 2024Mutual Reasoning Makes Smaller LLMs Stronger Problem-Solverscited by 138
  11. 2025ReasonFlux: Hierarchical LLM Reasoning via Scaling Thought Templatescited by 63
  12. 2024Beyond Examples: High-level Automated Reasoning Paradigm in In-Context Learning via MCTScited by 54
  13. 2024Retrieval-style In-Context Learning for Few-shot Hierarchical Text Classificationcited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 16; 2026: 12.

28 papers.

Sort:

By area:

  1. 2025SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoningcited by 84
  2. 2025RL-PLUS: Countering Capability Boundary Collapse of LLMs in Reinforcement Learning with Hybrid-policy Optimizationcited by 35
  3. 2025MUR: Momentum Uncertainty guided Reasoning for Large Language Modelscited by 17
  4. 2025The Debate on RLVR Reasoning Capability Boundary: Shrinkage, Expansion, or Both? A Two-Stage Dynamic Viewcited by 13
  5. 2025ADHint: Adaptive Hints with Difficulty Priors for Reinforcement Learningcited by 12
  6. 2026Atlas: Orchestrating Heterogeneous Models and Tools for Multi-Domain Complex Reasoningcited by 11
  7. 2026Spark: Strategic Policy-Aware Exploration via Dynamic Branching for Long-Horizon Agentic Learningcited by 11
  8. 2026SFT-then-RL Outperforms Mixed-Policy Methods for LLM Reasoningcited by 9
  9. 2026KnowRL: Boosting LLM Reasoning via Reinforcement Learning with Minimal-Sufficient Knowledge Guidancecited by 8
  10. 2025Metacognitive Self-Correction for Multi-Agent System via Prototype-Guided Next-Execution Reconstructioncited by 7
  11. 2026Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learningcited by 6
  12. 2025AMFT: Aligning LLM Reasoners by Meta-Learning the Optimal Imitation-Exploration Balancecited by 5
16 more
  1. 2026TACO: Tool-Augmented Credit Optimization for Agentic Tool Usecited by 5
  2. 2025FloorPlan-DeepSeek (FPDS): A multimodal approach to floorplan generation using vector-based next room predictioncited by 4
  3. 2025Two-Stage Regularization-Based Structured Pruning for LLMscited by 3
  4. 2025Harnessing Negative Signals: Reinforcement Distillation from Teacher Data for LLM Reasoningcited by 3
  5. 2025More Than One Teacher: Adaptive Multi-Guidance Policy Optimization for Diverse Explorationcited by 3
  6. 2025Toward Honest Language Models for Deductive Reasoningcited by 3
  7. 2026Zone of Proximal Policy Optimization: Teacher in Prompts, Not Gradientscited by 3
  8. 2025ArchiLense: A Framework for Quantitative Analysis of Architectural Styles Based on Vision Large Language Modelscited by 2
  9. 2025Reasoning Pattern Matters: Learning to Reason without Human Rationalescited by 2
  10. 2026From Reasoning Chains to Verifiable Subproblems: Curriculum Reinforcement Learning Enables Credit Assignment for LLM Reasoningcited by 2
  11. 2026Controllable Exploration in Hybrid-Policy RLVR for Multi-Modal Reasoningcited by 1
  12. 2026Implicit Hierarchical GRPO: Decoupling Tool Invocation from Execution for Tool-Integrated Mathematical Reasoningcited by 1
  13. 2025UrbanSense:A Framework for Quantitative Analysis of Urban Streetscapes leveraging Vision Large Language Modelscited by 0
  14. 2025From Imitation to Discrimination: Toward A Generalized Curriculum Advantage Mechanism Enhancing Cross-Domain Reasoning Taskscited by 0
  15. 2026Decoupled Visual Processing: Efficient Multimodal Adaptation via Modality-Specific Transformer Substitutioncited by 0
  16. 2026Learning Perception to Action Rules through Reinforcement Learning for Theory-of-Mind Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learning
  2. 2025Can LLMs Guide Their Own Exploration? Gradient-Guided Reinforcement Learning for LLM Reasoning
  3. 2025Part I: Tricks or Traps? A Deep Dive into RL for LLM Reasoning
  4. 2026Trajectory-Aware Policy Optimization: Closing the State Distribution Gap in Long-Horizon LLM RL
  5. 2025Plan Then Action:High-Level Planning Guidance Reinforcement Learning for LLM Reasoning
  6. 2026Growing and Transferring Pathways for LLM Reinforcement Learning
  7. 2025Slow-Fast Policy Optimization: Reposition-Before-Update for LLM Reasoning
  8. 2025Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection