mlsanityData to 7 Oct 2026

Recursive Introspection: Teaching Language Model Agents How to Self-Improve

arXiv 2407.18219 ↗First posted 25 Jul 2024cs.LG cs.AI cs.CLcited by 162

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 2 more
Abstract (arXiv)

A central piece in enabling intelligent agentic behavior in foundation models is to make them capable of introspecting upon their behavior, reasoning, and correcting their mistakes as more computation or interaction is available. Even the strongest proprietary large language models (LLMs) do not quite exhibit the ability of continually improving their responses sequentially, even in scenarios where they are explicitly told that they are making a mistake. In this paper, we develop RISE: Recursive IntroSpEction, an approach for fine-tuning LLMs to introduce this capability, despite prior work hypothesizing that this capability may not be possible to attain. Our approach prescribes an iterative fine-tuning procedure, which attempts to teach the model how to alter its response after having executed previously unsuccessful attempts to solve a hard test-time problem, with optionally additional environment feedback. RISE poses fine-tuning for a single-turn prompt as solving a multi-turn Markov decision process (MDP), where the initial state is the prompt. Inspired by principles in online imitation learning and reinforcement learning, we propose strategies for multi-turn data collection and training so as to imbue an LLM with the capability to recursively detect and correct its previous mistakes in subsequent iterations. Our experiments show that RISE enables Llama2, Llama3, and Mistral models to improve themselves with more turns on math reasoning tasks, outperforming several single-turn strategies given an equal amount of inference-time computation. We also find that RISE scales well, often attaining larger benefits with more capable models. Our analysis shows that RISE makes meaningful improvements to responses to arrive at the correct solution for challenging prompts, without disrupting one-turn abilities as a result of expressing more complex distributions.

Key facts. Cited by 162 papers in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 56 of its 66 references that are in the map. First posted on arXiv 25 Jul 2024.

Builds on

56 of 66 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models6 Oct
  2. 2023Teaching Large Language Models to Self-Debug11 Apr

Other references · 54

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  4. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  7. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  8. 2023Let's Verify Step by Stepcited by 3,598
  9. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  10. 2023Mistral 7Bcited by 3,065
  11. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  12. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
38 more
  1. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  2. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  3. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  4. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  5. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  6. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 895
  7. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  8. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  9. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  10. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  11. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  12. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  13. 2024Self-Rewarding Language Modelscited by 654
  14. 2022PAL: Program-aided Language Modelscited by 635
  15. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  16. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  17. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  18. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 478
  19. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  20. 2023LeanDojo: Theorem Proving with Retrieval-Augmented Language Modelscited by 414
  21. 2023AgentTuning: Enabling Generalized Agent Abilities for LLMscited by 319
  22. 2021How to Train Your Energy-Based Modelscited by 271
  23. 2022Generating Sequences by Learning to Self-Correctcited by 271
  24. 2023Auto-GPT for Online Decision Making: Benchmarks and Additional Opinionscited by 250
  25. 2022Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matterscited by 221
  26. 2024V-STaR: Training Verifiers for Self-Taught Reasonerscited by 214
  27. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RLcited by 212
  28. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 191
  29. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 176
  30. 2024OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Datasetcited by 143
  31. 2024Stream of Search (SoS): Learning to Search in Languagecited by 141
  32. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 137
  33. 2024Common 7B Language Models Already Possess Strong Math Capabilitiescited by 108
  34. 2023Branch-Solve-Merge Improves Large Language Model Evaluation and Generationcited by 104
  35. 2024GLoRe: When, Where, and How to Improve LLM Reasoning via Global and Local Refinementscited by 101
  36. 2024Beyond A*: Better Planning with Transformers via Search Dynamics Bootstrappingcited by 89
  37. 2023Contrastive Chain-of-Thought Promptingcited by 52
  38. 2022TEMPERA: Test-Time Prompting via Reinforcement Learningcited by 50

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 15; 2025: 96; 2026: 51.

162 papers.

Sort:

By area:

  1. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 405
  2. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRscited by 373
  3. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  4. 2025The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoningcited by 160
  5. 2024O1 Replication Journey: A Strategic Progress Report -- Part 1cited by 140
  6. 2025Optimizing Test-Time Compute via Meta Reinforcement Fine-Tuningcited by 121
  7. 2025Towards System 2 Reasoning in LLMs: Learning How to Think With Meta Chain-of-Thoughtcited by 108
  8. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 108
  9. 2026AutoSkill: Experience-Driven Lifelong Learning via Skill Self-Evolutioncited by 105
  10. 2025Inducing Programmatic Skills for Agentic Taskscited by 103
  11. 2024Self-Improvement in Language Models: The Sharpening Mechanismcited by 102
  12. 2025Maximizing Confidence Alone Improves Reasoningcited by 97
38 more
  1. 2025Scaling Test-Time Compute Without Verification or RL is Suboptimalcited by 82
  2. 2025MAPoRL: Multi-Agent Post-Co-Training for Collaborative Large Language Models with Reinforcement Learningcited by 80
  3. 2024MALT: Improving Reasoning with Multi-Agent LLM Trainingcited by 71
  4. 2024The Death of Schema Linking? Text-to-SQL in the Age of Well-Reasoned Language Modelscited by 67
  5. 2025Self-rewarding correction for mathematical reasoningcited by 67
  6. 2025Are Reasoning Models More Prone to Hallucination?cited by 62
  7. 2025Multi-Agent Verification: Scaling Test-Time Compute with Multiple Verifierscited by 58
  8. 2025A Self-Improving Coding Agentcited by 58
  9. 2025e3: Learning to Explore Enables Extrapolation of Test-Time Compute for LLMscited by 56
  10. 2025SETS: Leveraging Self-Verification and Self-Correction for Improved Test-Time Scalingcited by 54
  11. 2025From Reflection to Perfection: Scaling Inference-Time Optimization for Text-to-Image Diffusion Models via Reflection Tuningcited by 53
  12. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
  13. 2025Sample, Scrutinize and Scale: Effective Inference-Time Search by Scaling Verificationcited by 49
  14. 2025Self-Questioning Language Modelscited by 47
  15. 2025A Survey on the Optimization of Large Language Model-based Agentscited by 45
  16. 2025Reflect-DiT: Inference-Time Scaling for Text-to-Image Diffusion Transformers via In-Context Reflectioncited by 42
  17. 2025ChemAgent: Self-updating Library in Large Language Models Improves Chemical Reasoningcited by 41
  18. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
  19. 2026POPE: Learning to Reason on Hard Problems via Privileged On-Policy Explorationcited by 39
  20. 2025S2R: Teaching LLMs to Self-verify and Self-correct via Reinforcement Learningcited by 37
  21. 2025STeCa: Step-level Trajectory Calibration for LLM Agent Learningcited by 37
  22. 2025ReVISE: Learning to Refine at Test-Time via Intrinsic Self-Verificationcited by 35
  23. 2025Measuring Agents in Productioncited by 35
  24. 2025Thinking vs. Doing: Agents that Reason by Scaling Test-Time Interactioncited by 34
  25. 2025Reflect, Retry, Reward: Self-Improving LLMs via Reinforcement Learningcited by 32
  26. 2025From Language to Action: A Review of Large Language Models as Autonomous Agents and Tool Userscited by 32
  27. 2025Self-Improving Transformers Overcome Easy-to-Hard and Length Generalization Challengescited by 30
  28. 2025Think Smarter not Harder: Adaptive Reasoning with Inference Aware Optimizationcited by 29
  29. 2024TS-Reasoner: Domain-Oriented Time Series Inference Agents for Reasoning and Automated Analysiscited by 28
  30. 2025PAG: Multi-Turn Reinforced LLM Self-Correction with Policy as Generative Verifiercited by 28
  31. 2024G"odel Agent: A Self-Referential Agent Framework for Recursive Self-Improvementcited by 27
  32. 2025Multi-Turn Code Generation Through Single-Step Rewardscited by 27
  33. 2025Scaling Offline RL via Efficient and Expressive Shortcut Modelscited by 27
  34. 2025To Backtrack or Not to Backtrack: When Sequential Search Limits Model Reasoningcited by 24
  35. 2025Heterogeneous Swarms: Jointly Optimizing Model Roles and Weights for Multi-LLM Systemscited by 22
  36. 2025On the Emergence of Thinking in LLMs I: Searching for the Right Intuitioncited by 22
  37. 2025Reasoning as an Adaptive Defense for Safetycited by 22
  38. 2025A Survey on LLM Test-Time Compute via Search: Tasks, LLM Profiling, Search Algorithms, and Relevant Frameworkscited by 21

50 most cited of 162 All 162 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Shockingly Simple Self-retrospection Improves Agentic Models Without RL
  2. 2026Test-time Recursive Thinking: Self-Improvement without External Feedback
  3. 2025Debate, Reflect, and Distill: Multi-Agent Feedback with Tree-Structured Preference Optimization for Efficient Language Model Enhancement
  4. 2024MetaReflection: Learning Instructions for Language Agents using Past Reflections
  5. 2025Agent-R: Training Language Model Agents to Reflect via Iterative Self-Training
  6. 2026Self-Evaluating Recursive Agents
  7. 2023Shattering the Agent-Environment Interface for Fine-Tuning Inclusive Language Models
  8. 2025The Path of Self-Evolving Large Language Models: Achieving Data-Efficient Learning via Intrinsic Feedback