mlsanityData to 7 Oct 2026

Plan Then Action:High-Level Planning Guidance Reinforcement Learning for LLM Reasoning

arXiv 2510.01833 ↗First posted 2 Oct 2025cs.AI cs.CLcited by 17

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Large language models (LLMs) demonstrate strong reasoning abilities via Chain-of-Thought (CoT), but their token-level generation encourages local decisions and lacks global planning, often leading to redundant or inaccurate reasoning. Existing methods, such as tree-based search and reinforcement learning (RL), attempt to address this issue but incur high computational costs and still struggle to produce reliable reasoning trajectories. To address these challenges, we propose Plan-Then-Action Enhanced Reasoning with Group Relative Policy Optimization (PTA-GRPO), a two-stage framework designed to jointly improve high-level planning and fine-grained CoT reasoning. Specifically, in the first stage, a given LLM is responsible for summarizing CoT reasoning into compact high-level guidance, which is then leveraged for supervised fine-tuning. Then, we introduce a guidance-aware reinforcement learning method that jointly optimizes the final output and the quality of guidance, enhancing reasoning effectiveness. We evaluate PTA-GRPO on ten reasoning benchmarks across mathematics and natural sciences, using five diverse base models spanning multiple data modalities. The results show that PTA-GRPO consistently delivers significant improvements across models and tasks, demonstrating strong effectiveness and generalization.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 48 of its 55 references that are in the map. First posted on arXiv 2 Oct 2025.

Builds on

48 of 55 references.

Closest ancestors · 15

Same direction, newest first

  1. 2025TreePO: Bridging the Gap of Policy Optimization and Efficacy and Inference Efficiency with Heuristic Tree-based Modeling24 Aug
  2. 2025SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimization18 May
  3. 2025Group-in-Group Policy Optimization for LLM Agent Training16 May
  4. 2025DSADF: Thinking Fast and Slow for Decision Making13 May
  5. 2025SRPO: A Cross-Domain Implementation of Large-Scale Reinforcement Learning on LLM19 Apr
  6. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  7. 2025DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning15 Apr
  8. 2025Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivization8 Apr
  9. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  10. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  11. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  12. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs3 Mar
3 more
  1. 2025Process Reinforcement through Implicit Rewards3 Feb
  2. 2024CPL: Critical Plan Step Learning Boosts LLM Generalization in Reasoning Tasks13 Sep
  3. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 33

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  5. 2025Qwen3 Technical Reportcited by 8,195
  6. 2025Qwen2.5-VL Technical Reportcited by 5,418
  7. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  8. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  9. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  10. 2024A Survey on Large Language Models for Code Generationcited by 646
  11. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 559
  12. 2023Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Modelscited by 550
21 more
  1. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
  2. 2025VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learningcited by 315
  3. 2025MM-Eureka: Exploring the Frontiers of Multimodal Reasoning with Rule-based Reinforcement Learningcited by 294
  4. 2023Alphazero-like Tree-Search can Guide Large Language Model Decoding and Trainingcited by 291
  5. 2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Modelscited by 291
  6. 2025OpenThoughts: Data Recipes for Reasoning Modelscited by 287
  7. 2025Plan-and-Act: Improving Planning of Agents for Long-Horizon Taskscited by 190
  8. 2025Seed1.5-Thinking: Advancing Superb Reasoning Models with Reinforcement Learningcited by 159
  9. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 131
  10. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 108
  11. 2024LLaMA-Berry: Pairwise Optimization for O1-like Olympiad-Level Mathematical Reasoningcited by 90
  12. 2025ReMA: Learning to Meta-think for LLMs with Multi-Agent Reinforcement Learningcited by 78
  13. 2025ARPO:End-to-End Policy Optimization for GUI Agents with Experience Replaycited by 75
  14. 2023MathChat: Converse to Tackle Challenging Math Problems with LLM Agentscited by 67
  15. 2025SRPO: Enhancing Multimodal LLM Reasoning via Reflection-Aware Reinforcement Learningcited by 53
  16. 2025Demystifying Reasoning Dynamics with Mutual Information: Thinking Tokens are Information Peaks in LLM Reasoningcited by 50
  17. 2023Improving Large Language Model Fine-tuning for Solving Math Problemscited by 48
  18. 2025Rethinking Reflection in Pre-Trainingcited by 43
  19. 2024LiteSearch: Efficacious Tree Search for LLMcited by 30
  20. 2025Full-Step-DPO: Self-Supervised Preference Optimization with Step-wise Rewards for Mathematical Reasoningcited by 14
  21. 2023An Extensible Multimodal Multi-task Object Dataset with Materialscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 16.

17 papers.

Sort:

By area:

  1. 2025CostBench: Evaluating Multi-Turn Cost-Optimal Planning and Adaptation in Dynamic Environments for LLM Tool-Use Agentscited by 32
  2. 2026PALM: Progress-Aware Policy Learning via Affordance Reasoning for Long-Horizon Robotic Manipulationcited by 20
  3. 2026Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learningcited by 8
  4. 2026ERPO: Token-Level Entropy-Regulated Policy Optimization for Large Reasoning Modelscited by 6
  5. 2026One Refiner to Unlock Them All: Inference-Time Reasoning Elicitation via Reinforcement Query Refinementcited by 6
  6. 2026Learning Transferable Temporal Primitives for Video Reasoning via Synthetic Videoscited by 2
  7. 2026Learning Rollout from Sampling:An R1-Style Tokenized Traffic Simulation Modelcited by 1
  8. 2026Latent Thought Credit: Multi-Answer Credit Assignment for Latent Reasoningcited by 1
  9. 2026SRVAU-R1: Enhancing Video Anomaly Understanding via Reflection-Aware Learningcited by 0
  10. 2026Mitigating Distribution Sharpening in Math RLVR via Distribution-Aligned Hint Synthesis and Backward Hint Annealingcited by 0
  11. 2026CRPO: Character-centric Group Relative Policy Optimization for Role-aware Reasoning in Role-playing Agentscited by 0
  12. 2026Knowing What to Solve Before How: Preplan Empowered LLM Mathematical Reasoningcited by 0
5 more
  1. 2026DAIN: Dynamic Agent-Based Interaction Network for Efficient and Collaborative Multimodal Reasoningcited by 0
  2. 2026FedCausal-Dyn: A Causal-Dynamic Paradigm for Federated Learning under Dynamic Feature Driftcited by 0
  3. 2026OpenClaw and Ollama in Agentic AI: Toward Fully Autonomous and Scalable AI Agent Systemscited by 0
  4. 2026Graph-Based Modeling of Financial Volatility Dynamicscited by 0
  5. 2026From Outcomes to Strategies: Learning Strategy Utility for Mathematical Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Policy Guided Tree Search for Enhanced LLM Reasoning
  2. 2025Rectifying LLM Thought from Lens of Optimization
  3. 2025G2RPO-A: Guided Group Relative Policy Optimization with Adaptive Guidance
  4. 2025Scaf-GRPO: Scaffolded Group Relative Policy Optimization for Enhancing LLM Reasoning
  5. 2025Walk Before You Run! Concise LLM Reasoning via Reinforcement Learning
  6. 2024Chasing Progress, Not Perfection: Revisiting Strategies for End-to-End LLM Plan Generation
  7. 2025Can LLMs Guide Their Own Exploration? Gradient-Guided Reinforcement Learning for LLM Reasoning
  8. 2025Slow-Fast Policy Optimization: Reposition-Before-Update for LLM Reasoning