mlsanityData to 14 Oct 2026

Learning From Mistakes Makes LLM Better Reasoner

arXiv 2310.20689 ↗First posted 31 Oct 2023cs.CL cs.AIcited by 101

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

Large language models (LLMs) recently exhibited remarkable reasoning capabilities on solving math problems. To further improve their reasoning capabilities, this work explores whether LLMs can LEarn from MistAkes (LEMA), akin to the human learning process. Consider a human student who failed to solve a math problem, he will learn from what mistake he has made and how to correct it. Mimicking this error-driven learning process, LEMA incorporates mistake-correction data pairs during fine-tuning LLMs. Specifically, we first collect inaccurate reasoning paths from various LLMs, and then employ GPT-4 as a ''corrector'' to identify the mistake step, explain the reason for the mistake, correct the mistake and generate the final answer. In addition, we apply a correction-centric evolution strategy that effectively expands the question set for generating correction data. Experiments across various LLMs and reasoning tasks show that LEMA effectively improves CoT-alone fine-tuning. Our further ablations shed light on the non-homogeneous effectiveness between CoT data and correction data. These results suggest a significant potential for LLMs to improve through learning from their mistakes. Our code, models and prompts are publicly available at https://github.com/microsoft/LEMA.

Key facts. Cited by 101 papers in the map (data through 14 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 49 of its 59 references that are in the map. First posted on arXiv 31 Oct 2023.

Builds on

49 of 59 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023MuggleMath: Assessing the Impact of Query and Response Augmentation on Math Reasoning9 Oct
  2. 2023MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoning5 Oct
  3. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct18 Aug
  4. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Models3 Aug
  5. 2022Language Models are Multilingual Chain-of-Thought Reasoners6 Oct

Other references · 44

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  8. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  10. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  12. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
32 more
  1. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  2. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  3. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,407
  4. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  5. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 878
  6. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 743
  7. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
  8. 2023Baichuan 2: Open Large-scale Language Modelscited by 670
  9. 2022Large Language Models Can Self-Improvecited by 664
  10. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  11. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 488
  12. 2023Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoningcited by 391
  13. 2022Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoningcited by 334
  14. 2022Large Language Models Are Reasoning Teacherscited by 329
  15. 2021A Diverse Corpus for Evaluating and Developing English Math Word Problem Solverscited by 318
  16. 2022Teaching Small Language Models to Reasoncited by 284
  17. 2022Quark: Controllable Text Generation with Reinforced Unlearningcited by 237
  18. 2022Making Large Language Models Better Reasoners with Step-Aware Verifiercited by 233
  19. 2022Faithful Reasoning Using Large Language Modelscited by 130
  20. 2022Explanations from Large Language Models Make Small Reasoners Bettercited by 124
  21. 2023Can Large Language Models Really Improve by Self-critiquing Their Own Plans?cited by 108
  22. 2023GPT-4 Doesn't Know It's Wrong: An Analysis of Iterative Prompting for Reasoning Problemscited by 89
  23. 2023Guiding Language Model Reasoning with Planning Tokenscited by 58
  24. 2023Improving Large Language Model Fine-tuning for Solving Math Problemscited by 49
  25. 2023Boosting Logical Reasoning in Large Language Models through a New Framework: The Graph of Thoughtcited by 41
  26. 2022Input-Tuning: Adapting Unfamiliar Inputs to Frozen Pretrained Modelscited by 34
  27. 2023Let's Synthesize Step by Step: Iterative Dataset Synthesis with Large Language Models by Extrapolating Errors from Small Modelscited by 30
  28. 2023Let GPT be a Math Tutor: Teaching Math Word Problem Solvers with Customized Exercise Generationcited by 29
  29. 2023A Comparative Study between Full-Parameter and LoRA-based Fine-Tuning on Chinese Instruction Data for Instruction Following Large Language Modelcited by 19
  30. 2023Democratizing Reasoning Ability: Tailored Learning from Large Language Modelcited by 13
  31. 2023Leveraging Training Data in Few-Shot Prompting for Numerical Reasoningcited by 11
  32. 2023Exploring the Impact of Model Scaling on Parameter-Efficient Tuningcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 44; 2025: 38; 2026: 16.

101 papers.

Sort:

By area:

  1. 2024Understanding the planning of LLM agents: A surveycited by 421
  2. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 384
  3. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 320
  4. 2024Large Language Models for Mathematical Reasoning: Progresses and Challengescited by 284
  5. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 260
  6. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positivecited by 241
  7. 2024V-STaR: Training Verifiers for Self-Taught Reasonerscited by 217
  8. 2024Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervisioncited by 96
  9. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 93
  10. 2024Small Language Models Need Strong Verifiers to Self-Correct Reasoningcited by 86
  11. 2025T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scalingcited by 77
  12. 2024Can LLMs Learn from Previous Mistakes? Investigating LLMs' Errors to Boost for Reasoningcited by 69
38 more
  1. 2024Vision-Language Models Can Self-Improve Reasoning via Reflectioncited by 69
  2. 2024Key-Point-Driven Data Synthesis with its Enhancement on Mathematical Reasoningcited by 58
  3. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
  4. 2024Internal Consistency and Self-Feedback in Large Language Models: A Surveycited by 50
  5. 2025Sample, Scrutinize and Scale: Effective Inference-Time Search by Scaling Verificationcited by 50
  6. 2024Evaluating Mathematical Reasoning of Large Language Models: A Focus on Error Identification and Correctioncited by 45
  7. 2024Think Twice Before Trusting: Self-Detection for Large Language Models through Comprehensive Answer Reflectioncited by 41
  8. 2024A Survey on Data Synthesis and Augmentation for Large Language Modelscited by 41
  9. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
  10. 2024Weak-to-Strong Reasoningcited by 33
  11. 2025The First Few Tokens Are All You Need: An Efficient and Effective Unsupervised Prefix Fine-Tuning Method for Reasoning Modelscited by 31
  12. 2024SMoA: Improving Multi-agent Large Language Models with Sparse Mixture-of-Agentscited by 30
  13. 2025Step Back to Leap Forward: Self-Backtracking for Boosting Reasoning of Language Modelscited by 27
  14. 2024Self-Training with Direct Preference Optimization Improves Chain-of-Thought Reasoningcited by 26
  15. 2024Skywork-Math: Data Scaling Laws for Mathematical Reasoning in Large Language Models -- The Story Goes Oncited by 24
  16. 2024How Numerical Precision Affects Arithmetical Reasoning Capabilities of LLMscited by 24
  17. 2023Temporal Knowledge Question Answering via Abstract Reasoning Inductioncited by 23
  18. 2024Can LLMs Solve longer Math Word Problems Better?cited by 22
  19. 2025LEMMA: Learning from Errors for MatheMatical Advancement in LLMscited by 21
  20. 2026Large Language Model Reasoning Failurescited by 21
  21. 2025CRITICTOOL: Evaluating Self-Critique Capabilities of Large Language Models in Tool-Calling Error Scenarioscited by 19
  22. 2024Learning From Correctness Without Prompting Makes LLM Efficient Reasonercited by 15
  23. 2024Learn Beyond The Answer: Training Language Models with Reflection for Mathematical Reasoningcited by 15
  24. 2024Beyond Imitation: Learning Key Reasoning Steps from Dual Chain-of-Thoughts in Reasoning Distillationcited by 14
  25. 2024BEATS: Optimizing LLM Mathematical Capabilities with BackVerify and Adaptive Disambiguate based Efficient Tree Searchcited by 13
  26. 2024Embedding Self-Correction as an Inherent Ability in Large Language Models for Enhanced Mathematical Reasoningcited by 13
  27. 2024A Survey on Human-Centric LLMscited by 13
  28. 2025No Need for Explanations: LLMs can implicitly learn from mistakes in-contextcited by 13
  29. 2024Reason from Fallacy: Enhancing Large Language Models' Logical Reasoning through Logical Fallacy Understandingcited by 12
  30. 2024Teaching-Assistant-in-the-Loop: Improving Knowledge Distillation from Imperfect Teacher Models in Low-Budget Scenarioscited by 12
  31. 2024Toward Adaptive Reasoning in Large Language Models with Thought Rollbackcited by 11
  32. 2025A Survey on Feedback-based Multi-step Reasoning for Large Language Models on Mathematicscited by 11
  33. 2025Can Vision-Language Models Evaluate Handwritten Math?cited by 10
  34. 2025A Comprehensive Survey on Imbalanced Data Learningcited by 10
  35. 2024DAFNet: Dynamic Auxiliary Fusion for Sequential Model Editing in Large Language Modelscited by 9
  36. 2024E2CL: Exploration-based Error Correction Learning for Embodied Agentscited by 9
  37. 2024Cross-Refine: Improving Natural Language Explanation Generation by Learning in Tandemcited by 9
  38. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 8

50 most cited of 101 All 101 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Exposing the Achilles' Heel: Evaluating LLMs Ability to Handle Mistakes in Mathematical Reasoning
  2. 2023LLMs cannot find reasoning errors, but can correct them given the error location
  3. 2026Improving Mathematical Reasoning Capabilities in Large Language Models via Reasoning Process Error Classification
  4. 2025Unraveling Misinformation Propagation in LLM Reasoning
  5. 2025Learning to Make MISTAKEs: Modeling Incorrect Student Thinking And Key Errors
  6. 2025Can LLMs Correct Themselves? A Benchmark of Self-Correction in LLMs
  7. 2023Novice Learner and Expert Tutor: Evaluating Math Reasoning Abilities of Large Language Models with Misconceptions
  8. 2025SMRC: Aligning Large Language Models with Student Reasoning for Mathematical Error Correction