mlsanityData to 14 Oct 2026

Reasoning Paths Optimization: Learning to Reason and Explore From Diverse Paths

arXiv 2410.10858 ↗First posted 7 Oct 2024cs.CL cs.AI cs.LGcited by 6

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Advanced models such as OpenAI o1 exhibit impressive problem-solving capabilities through step-by-step reasoning. However, they may still falter on more complex problems, making errors that disrupt their reasoning paths. We attribute this to the expansive solution space, where each step has the risk of diverging into mistakes. To enhance language model reasoning, we introduce a specialized training framework called Reasoning Paths Optimization (RPO), which enables learning to reason and explore from diverse paths. Our approach encourages favorable branches at each reasoning step while penalizing unfavorable ones, enhancing the model's overall problem-solving performance. Reasoning Paths Optimization does not rely on large-scale human-annotated rationales or outputs from closed-source models, making it scalable and data-efficient. We focus on multi-step reasoning tasks, such as math word problems and science-based exam questions. The experiments demonstrate that our framework significantly enhances the reasoning performance of large language models, with up to 3.1% and 4.3% improvement on GSM8K and MMLU (STEM) respectively. Our data and code can be found at https://reasoning-paths.github.io.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 21 of its 63 references that are in the map. First posted on arXiv 7 Oct 2024.

Builds on

21 of 63 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023PathFinder: Guided Search over Multi-Step Reasoning Paths8 Dec

Other references · 20

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2021Program Synthesis with Large Language Modelscited by 3,637
  6. 2023Mistral 7Bcited by 3,164
  7. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  8. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  9. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 928
  10. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  11. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 562
  12. 2023Llemma: An Open Language Model For Mathematicscited by 388
8 more
  1. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 379
  2. 2024Iterative Reasoning Preference Optimizationcited by 246
  3. 2023Secrets of RLHF in Large Language Models Part I: PPOcited by 229
  4. 2023Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Futurecited by 184
  5. 2024Evaluation of OpenAI o1: Opportunities and Challenges of AGIcited by 127
  6. 2023Boosting Language Models Reasoning with Chain-of-Knowledge Promptingcited by 77
  7. 2024Multi-step Problem Solving Through a Verifier: An Empirical Analysis on Model-induced Process Supervisioncited by 52
  8. 2023Flacuna: Unleashing the Problem Solving Power of Vicuna using FLAN Fine-Tuningcited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 3.

6 papers.

  1. 2025The Danger of Overthinking: Examining the Reasoning-Action Dilemma in Agentic Taskscited by 152
  2. 2025SciReplicate-Bench: Benchmarking LLMs in Agent-driven Algorithmic Reproduction from Research Paperscited by 43
  3. 2026FROST: Filtering Reasoning Outliers with Attention for Efficient Reasoningcited by 5
  4. 2025Putting on the Thinking Hats: A Survey on Chain of Thought Fine-tuning from the Perspective of Human Reasoning Mechanismcited by 4
  5. 2026Beyond Entropy: Correctness-Aware Advantage Shaping via Contrastive Policy Optimizationcited by 1
  6. 2026It Takes 8 Tokens: Weak-to-Strong Off-Policy RL via Auxiliary Branchescited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2023DialCoT Meets PPO: Decomposing and Exploring Reasoning Paths in Smaller Language Models
  2. 2026Neural Chain-of-Thought Search: Searching the Optimal Reasoning Path to Enhance Large Language Models
  3. 2024A Comparative Study on Reasoning Patterns of OpenAI's o1 Model
  4. 2024Multi-Step Reasoning with Large Language Models, a Survey
  5. 2025Reasoning Path Divergence: A New Metric and Curation Strategy to Unlock LLM Diverse Thinking
  6. 2025Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models
  7. 2024Understanding Reasoning Ability of Language Models From the Perspective of Reasoning Paths Aggregation
  8. 2025Step-Tagging: Toward controlling the generation of Language Reasoning Models through step monitoring