mlsanityData to 14 Oct 2026

Self-Explore: Enhancing Mathematical Reasoning in Language Models with Fine-grained Rewards

arXiv 2404.10346 ↗First posted 16 Apr 2024cs.CLcited by 31

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Training on large amounts of rationales (i.e., CoT Fine-tuning) is effective at improving the reasoning capabilities of large language models (LLMs). However, acquiring human-authored rationales or augmenting rationales from proprietary models is costly and not scalable. In this paper, we study the problem of whether LLMs could self-improve their reasoning capabilities. To this end, we propose Self-Explore, where the LLM is tasked to explore the first wrong step (i.e., the first pit) within the rationale and use such signals as fine-grained rewards for further improvement. On the GSM8K and MATH test set, Self-Explore achieves 11.57% and 2.89% improvement on average across three LLMs compared to supervised fine-tuning (SFT). Our code is available at https://github.com/hbin0701/Self-Explore.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 57 of its 60 references that are in the map. First posted on arXiv 16 Apr 2024.

Builds on

57 of 60 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024V-STaR: Training Verifiers for Self-Taught Reasoners9 Feb
  2. 2023Self-Evaluation Guided Beam Search for Reasoning1 May
  3. 2022Large Language Models Can Self-Improve20 Oct
  4. 2022STaR: Bootstrapping Reasoning With Reasoning28 Mar

Other references · 53

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  8. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  9. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  10. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  11. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  12. 2023Let's Verify Step by Stepcited by 3,789
38 more
  1. 2023Mistral 7Bcited by 3,164
  2. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  3. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,076
  4. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 928
  5. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  6. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 872
  7. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  8. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
  9. 2024Self-Rewarding Language Modelscited by 681
  10. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 625
  11. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 562
  12. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  13. 2023Zephyr: Direct Distillation of LM Alignmentcited by 506
  14. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 500
  15. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 467
  16. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 427
  17. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 396
  18. 2023Llemma: An Open Language Model For Mathematicscited by 388
  19. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 379
  20. 2024KTO: Model Alignment as Prospect Theoretic Optimizationcited by 264
  21. 2023The False Promise of Imitating Proprietary LLMscited by 263
  22. 2023Specializing Smaller Language Models towards Multi-Step Reasoningcited by 251
  23. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positivecited by 241
  24. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2cited by 229
  25. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 201
  26. 2021Does Knowledge Distillation Really Work?cited by 190
  27. 2024Teaching Large Language Models to Reason with Reinforcement Learningcited by 160
  28. 2023Orca 2: Teaching Small Language Models How to Reasoncited by 159
  29. 2024Orca-Math: Unlocking the potential of SLMs in Grade School Mathcited by 151
  30. 2024OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Datasetcited by 143
  31. 2022Explanations from Large Language Models Make Small Reasoners Bettercited by 124
  32. 2024Common 7B Language Models Already Possess Strong Math Capabilitiescited by 109
  33. 2023The CoT Collection: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuningcited by 108
  34. 2024GLoRe: When, Where, and How to Improve LLM Reasoning via Global and Local Refinementscited by 103
  35. 2023OVM, Outcome-supervised Value Models for Planning in Mathematical Reasoningcited by 85
  36. 2023TinyGSM: achieving >80% on GSM8k with small language modelscited by 70
  37. 2023Don't throw away your value model! Generating more preferable text with Value-Guided Monte-Carlo Tree Search decodingcited by 60
  38. 2024Multi-step Problem Solving Through a Verifier: An Empirical Analysis on Model-induced Process Supervisioncited by 52

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 21; 2026: 4.

31 papers.

Sort:

By area:

  1. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRscited by 377
  2. 2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Modelscited by 293
  3. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  4. 2025SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Taskscited by 118
  5. 2025Concise Reasoning via Reinforcement Learningcited by 79
  6. 20251.4 Million Open-Source Distilled Reasoning Dataset to Empower Large Language Model Trainingcited by 49
  7. 2025Ignore the KL Penalty! Boosting Exploration on Critical Tokens to Enhance RL Fine-Tuningcited by 34
  8. 2025Topology of Reasoning: Understanding Large Reasoning Models through Reasoning Graph Propertiescited by 29
  9. 2025Exploiting Tree Structure for Credit Assignment in RL Training of LLMscited by 27
  10. 2025Thinking Machines: A Survey of LLM based Reasoning Strategiescited by 25
  11. 2025GRPO-λ: Credit Assignment improves LLM Reasoningcited by 23
  12. 2024The Role of Deductive and Inductive Reasoning in Large Language Modelscited by 20
19 more
  1. 2025Stepwise Guided Policy Optimization: Coloring your Incorrect Reasoning in GRPOcited by 19
  2. 2025Generative AI Act II: Test Time Scaling Drives Cognition Engineeringcited by 15
  3. 2025Don't Pass@k: A Bayesian Framework for Large Language Model Evaluationcited by 15
  4. 2025Enhancing Reasoning through Process Supervision with Monte Carlo Tree Searchcited by 14
  5. 2025MPBench: A Comprehensive Multimodal Reasoning Benchmark for Process Errors Identificationcited by 13
  6. 2024CPL: Critical Plan Step Learning Boosts LLM Generalization in Reasoning Taskscited by 12
  7. 2025ClinicalGPT-R1: Pushing reasoning capability of generalist disease diagnosis with large language modelcited by 7
  8. 2024RATIONALYST: Mining Implicit Rationales for Process Supervision of Reasoningcited by 6
  9. 2024Self-Training Meets Consistency: Improving LLMs' Reasoning with Consistency-Driven Rationale Evaluationcited by 6
  10. 2025A Short Survey on Small Reasoning Models: Training, Inference, Applications and Research Directionscited by 4
  11. 2025AQuilt: Weaving Logic and Self-Inspection into Low-Cost, High-Relevance Data Synthesis for Specialist LLMscited by 4
  12. 2026Self-Verified Distillation: Your Language Model Is Secretly Its Own Synthetic Data Pipelinecited by 2
  13. 2024Large Language Models Are Self-Taught Reasoners: Enhancing LLM Applications via Tailored Problem-Solving Demonstrationscited by 1
  14. 2025p2-TQA: A Process-based Preference Learning Framework for Self-Improving Table Question Answering Modelscited by 1
  15. 2025Datarus-R1: An Adaptive Multi-Step Reasoning LLM for Automated Data Analysiscited by 0
  16. 2025TORSO: Template-Oriented Reasoning Towards General Taskscited by 0
  17. 2026From Implicit to Explicit: Token-Efficient Logical Supervision for Mathematical Reasoning in LLMscited by 0
  18. 2026CoTZero: Annotation-Free Human-Like Vision Reasoning via Hierarchical Synthetic CoTcited by 0
  19. 2026When Tokenization is Secretly Output Supervisioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MARGE: Improving Math Reasoning for LLMs with Guided Exploration
  2. 2024MindStar: Enhancing Math Reasoning in Pre-trained LLMs at Inference Time
  3. 2023Teaching Language Models to Self-Improve through Interactive Demonstrations
  4. 2024ChatGLM-Math: Improving Math Problem-Solving in Large Language Models with a Self-Critique Pipeline
  5. 2025A Model Can Help Itself: Reward-Free Self-Training for LLM Reasoning
  6. 2025Self-Training Elicits Concise Reasoning in Large Language Models
  7. 2024Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizing
  8. 2024Navigating the Labyrinth: Evaluating LLMs' Ability to Reason About Search Problems