mlsanityData to 7 Oct 2026

Learning Planning-based Reasoning by Trajectories Collection and Process Reward Synthesizing

arXiv 2402.00658 ↗First posted 1 Feb 2024cs.AI cs.CLcited by 49

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Large Language Models (LLMs) have demonstrated significant potential in handling complex reasoning tasks through step-by-step rationale generation. However, recent studies have raised concerns regarding the hallucination and flaws in their reasoning process. Substantial efforts are being made to improve the reliability and faithfulness of the generated rationales. Some approaches model reasoning as planning, while others focus on annotating for process supervision. Nevertheless, the planning-based search process often results in high latency due to the frequent assessment of intermediate reasoning states and the extensive exploration space. Additionally, supervising the reasoning process with human annotation is costly and challenging to scale for LLM training. To address these issues, in this paper, we propose a framework to learn planning-based reasoning through Direct Preference Optimization (DPO) on collected trajectories, which are ranked according to synthesized process rewards. Our results on challenging logical reasoning benchmarks demonstrate the effectiveness of our learning framework, showing that our 7B model can surpass the strong counterparts like GPT-3.5-Turbo.

Key facts. Cited by 49 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 28 of its 44 references that are in the map. First posted on arXiv 1 Feb 2024.

Builds on

28 of 44 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations14 Dec
  2. 2023Let's Verify Step by Step31 May
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  4. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec
  5. 2022Solving math word problems with process- and outcome-based feedback25 Nov
  6. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 22

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  6. 2023Qwen Technical Reportcited by 3,273
  7. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  8. 2024Mixtral of Expertscited by 1,785
  9. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  10. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  11. 2023LIMA: Less Is More for Alignmentcited by 1,080
  12. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
10 more
  1. 2023Reasoning with Language Model is Planning with World Modelcited by 846
  2. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  3. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 668
  4. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  5. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  6. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 417
  7. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 325
  8. 2023SeaEval for Multilingual Foundation Models: From Cross-Lingual Alignment to Cultural Reasoningcited by 94
  9. 2022ALERT: Adapting Language Models to Reasoning Taskscited by 17
  10. 2023Exploring Self-supervised Logic-enhanced Training for Large Language Modelscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 13; 2025: 29; 2026: 7.

49 papers.

Sort:

By area:

  1. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Searchcited by 338
  2. 2024Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMscited by 94
  3. 2025STAIR: Improving Safety Alignment with Introspective Reasoningcited by 72
  4. 2025Self-rewarding correction for mathematical reasoningcited by 67
  5. 2024Building Math Agents with Multi-Turn Iterative Preference Learningcited by 66
  6. 2025Dynamic Parallel Tree Search for Efficient LLM Reasoningcited by 42
  7. 2024Preference Optimization for Reasoning with Pseudo Feedbackcited by 38
  8. 2024Entropy-Regularized Process Reward Modelcited by 38
  9. 2025Large Language Models for Planning: A Comprehensive and Systematic Surveycited by 38
  10. 2025STeCa: Step-level Trajectory Calibration for LLM Agent Learningcited by 37
  11. 2024Step-Controlled DPO: Leveraging Stepwise Error for Enhanced Mathematical Reasoningcited by 35
  12. 2024Self-Explore: Enhancing Mathematical Reasoning in Language Models with Fine-grained Rewardscited by 31
37 more
  1. 2024Process Supervision-Guided Policy Optimization for Code Generationcited by 25
  2. 2025Thinking Machines: A Survey of LLM based Reasoning Strategiescited by 25
  3. 2025Demystifying Domain-adaptive Post-training for Financial LLMscited by 24
  4. 2025Genius: A Generalizable and Purely Unsupervised Self-Training Framework For Advanced Reasoningcited by 24
  5. 2024Can We Further Elicit Reasoning in LLMs? Critic-Guided Planning with Retrieval-Augmentation for Solving Challenging Taskscited by 22
  6. 2024Reasoning Through Execution: Unifying Process and Outcome Rewards for Code Generationcited by 21
  7. 2024TPO: Aligning Large Language Models with Multi-branch & Multi-step Preference Treescited by 19
  8. 2025DiagnosisArena: Benchmarking Diagnostic Reasoning for Large Language Modelscited by 19
  9. 2025BLEUBERI: BLEU is a surprisingly effective reward for instruction followingcited by 17
  10. 2025Beyond the Last Answer: Your Reasoning Trace Uncovers More than You Thinkcited by 16
  11. 2025PlanGEN: A Multi-Agent Framework for Generating Planning and Reasoning Trajectories for Complex Problem Solvingcited by 15
  12. 2025Full-Step-DPO: Self-Supervised Preference Optimization with Step-wise Rewards for Mathematical Reasoningcited by 14
  13. 2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewardscited by 14
  14. 2025A Survey on Feedback-based Multi-step Reasoning for Large Language Models on Mathematicscited by 11
  15. 2025Self-Guided Process Reward Optimization with Redefined Step-wise Advantage for Process Reinforcement Learningcited by 10
  16. 2024LASP: Surveying the State-of-the-Art in Large Language Model-Assisted AI Planningcited by 9
  17. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 7
  18. 2025Elicit and Enhance: Advancing Multimodal Reasoning in Medical Scenarioscited by 6
  19. 2025PLAN-TUNING: Post-Training Language Models to Learn Step-by-Step Planning for Complex Problem Solvingcited by 6
  20. 2025EvoMem: Improving Multi-Agent Planning with Dual-Evolving Memorycited by 6
  21. 2025Mind the Generation Process: Fine-Grained Confidence Estimation During LLM Generationcited by 5
  22. 2025Putting on the Thinking Hats: A Survey on Chain of Thought Fine-tuning from the Perspective of Human Reasoning Mechanismcited by 4
  23. 2024Mars-PO: Multi-Agent Reasoning System Preference Optimizationcited by 3
  24. 2026PEARL: Self-Evolving Assistant for Time Management with Reinforcement Learningcited by 3
  25. 2025Local Look-Ahead Guidance via Verifier-in-the-Loop for Automated Theorem Provingcited by 2
  26. 2025MARGE: Improving Math Reasoning for LLMs with Guided Explorationcited by 2
  27. 2025MCTS-EP: Empowering Embodied Planning with Online Preference Optimizationcited by 2
  28. 2026Towards AGI A Pragmatic Approach Towards Self Evolving Agentcited by 2
  29. 2025CuDIP: Enhancing Theorem Proving in LLMs via Curriculum Learning-based Direct Preference Optimizationcited by 1
  30. 2025Future Policy Approximation for Offline Reinforcement Learning in LLM Reasoningcited by 1
  31. 2025iCLP: Large Language Model Reasoning with Implicit Cognition Latent Planningcited by 1
  32. 2026CARE-RFT: Confidence-Anchored Reinforcement Finetuning for Reliable Reasoning in Large Language Modelscited by 1
  33. 2026Beyond Entropy: Correctness-Aware Advantage Shaping via Contrastive Policy Optimizationcited by 1
  34. 2026Decoupled Physical Modeling and Execution for Physics Reasoningcited by 1
  35. 2025SWI: Speaking with Intent in Large Language Modelscited by 0
  36. 2026V-tableR1: Process-Supervised Multimodal Table Reasoning with Critic-Guided Policy Optimizationcited by 0
  37. 2026Reinforcement Learning without Ground-Truth Solutions can Improve LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Process Reward Models Meet Planning: Generating Precise and Scalable Datasets for Step-Level Rewards
  2. 2024Plan of Thoughts: Heuristic-Guided Problem Solving with Large Language Models
  3. 2024Chasing Progress, Not Perfection: Revisiting Strategies for End-to-End LLM Plan Generation
  4. 2025Plan Then Action:High-Level Planning Guidance Reinforcement Learning for LLM Reasoning
  5. 2023Let's reward step by step: Step-Level reward model as the Navigators for Reasoning
  6. 2026Knowledge Model Prompting Increases LLM Performance on Planning Tasks
  7. 2024Exploring and Benchmarking the Planning Capabilities of Large Language Models
  8. 2025Can LLM-Reasoning Models Replace Classical Planning? A Benchmark Study