mlsanityData to 14 Oct 2026

ReST meets ReAct: Self-Improvement for Multi-Step Reasoning LLM Agent

arXiv 2312.10003 ↗First posted 15 Dec 2023cs.CLcited by 81

On the map: LLM reasoning → Knowledge- and tool-augmented reasoning

also in 2 more
Abstract (arXiv)

Answering complex natural language questions often necessitates multi-step reasoning and integrating external information. Several systems have combined knowledge retrieval with a large language model (LLM) to answer such questions. These systems, however, suffer from various failure cases, and we cannot directly train them end-to-end to fix such failures, as interaction with external knowledge is non-differentiable. To address these deficiencies, we define a ReAct-style LLM agent with the ability to reason and act upon external knowledge. We further refine the agent through a ReST-like method that iteratively trains on previous trajectories, employing growing-batch reinforcement learning with AI feedback for continuous self-improvement and self-distillation. Starting from a prompted large model and after just two iterations of the algorithm, we can produce a fine-tuned small model that achieves comparable performance on challenging compositional question-answering benchmarks with two orders of magnitude fewer parameters.

Key facts. Cited by 81 papers in the map (data through 14 Oct 2026). Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning. Builds on 12 of its 28 references that are in the map. First posted on arXiv 15 Dec 2023.

Builds on

12 of 28 references.

References · 12

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  3. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  4. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  5. 2023PaLM 2 Technical Reportcited by 1,162
  6. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,075
  7. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
  8. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 832
  9. 2019ELI5: Long Form Question Answeringcited by 575
  10. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 500
  11. 2022Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLPcited by 318
  12. 2021Hurdles to Progress in Long-form Question Answeringcited by 167

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 30; 2025: 29; 2026: 22.

81 papers.

Sort:

By area:

  1. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 376
  2. 2024Trial and Error: Exploration-Based Trajectory Optimization for LLM Agentscited by 197
  3. 2025From LLM Reasoning to Autonomous AI Agents: A Comprehensive Reviewcited by 149
  4. 2025Deep Research Agents: A Systematic Examination And Roadmapcited by 128
  5. 2024AgentGym: Evolving Large Language Model-based Agents across Diverse Environmentscited by 120
  6. 2024Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMscited by 96
  7. 2025Learn-by-interact: A Data-Centric Framework for Self-Adaptive Agents in Realistic Environmentscited by 95
  8. 2024A Survey on Self-Evolution of Large Language Modelscited by 91
  9. 2024Functional Benchmarks for Robust Evaluation of Reasoning Performance, and the Reasoning Gapcited by 79
  10. 2025AgentPRM: Process Reward Models for LLM Agents via Step-Wise Promise and Progresscited by 77
  11. 2025WebEvolver: Enhancing Web Agent Self-Improvement with Coevolving World Modelcited by 61
  12. 2024Large Language Models Can Self-Improve At Web Agent Taskscited by 55
38 more
  1. 2024Offline Reinforcement Learning for LLM Multi-Step Reasoningcited by 54
  2. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 50
  3. 2025A Survey on the Optimization of Large Language Model-based Agentscited by 46
  4. 2024Optima: Optimizing Effectiveness and Efficiency for LLM-Based Multi-Agent Systemcited by 43
  5. 2025UProp: Investigating the Uncertainty Propagation of LLMs in Multi-Step Agentic Decision-Makingcited by 42
  6. 2024LLM-as-a-Judge & Reward Model: What They Can and Cannot Docited by 32
  7. 2024DynaSaur: Large Language Agents Beyond Predefined Actionscited by 32
  8. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  9. 2025ATLaS: Agent Tuning via Learning Critical Stepscited by 31
  10. 2024Learning to Plan for Retrieval-Augmented Large Language Models from Knowledge Graphscited by 24
  11. 2024CodeIt: Self-Improving Language Models with Prioritized Hindsight Replaycited by 23
  12. 2024ReAct Meets ActRe: When Language Agents Enjoy Training Data Autonomycited by 23
  13. 2024DataGen: Unified Synthetic Dataset Generation via Large Language Modelscited by 23
  14. 2025A Comprehensive Survey on Reinforcement Learning-based Agentic Search: Foundations, Roles, Optimizations, Evaluations, and Applicationscited by 22
  15. 2025The Lighthouse of Language: Enhancing LLM Agents via Critique-Guided Improvementcited by 20
  16. 2024On the Brittle Foundations of ReAct Prompting for Agentic Large Language Modelscited by 17
  17. 2025WebCoT: Enhancing Web Agent Reasoning by Reconstructing Chain-of-Thought in Reflection, Branching, and Rollbackcited by 17
  18. 2024Husky: A Unified, Open-Source Language Agent for Multi-Step Reasoningcited by 16
  19. 2024Automatic Curriculum Expert Iteration for Reliable LLM Reasoningcited by 16
  20. 2026Toward Efficient Agents: Memory, Tool learning, and Planningcited by 16
  21. 2025Truly Self-Improving Agents Require Intrinsic Metacognitive Learningcited by 15
  22. 2026Multi-Turn On-Policy Distillation with Prefix Replaycited by 15
  23. 2026Agentic Artificial Intelligence (AI): Architectures, Taxonomies, and Evaluation of Large Language Model Agentscited by 14
  24. 2025LiveTradeBench: Seeking Real-World Alpha with Large Language Modelscited by 13
  25. 2025Position: The Real Barrier to LLM Agent Usability is Agentic ROIcited by 12
  26. 2024Can LLMs Understand Computer Networks? Towards a Virtual System Administratorcited by 11
  27. 2024WONDERBREAD: A Benchmark for Evaluating Multimodal Foundation Models on Business Process Management Taskscited by 11
  28. 2026OVD: On-policy Verbal Distillationcited by 11
  29. 2025Debt Collection Negotiations with Large Language Models: An Evaluation System and Optimizing Decision Making with Multi-Agentcited by 10
  30. 2025Generalizability of Large Language Model-Based Agents: A Comprehensive Surveycited by 10
  31. 2024Professional Agents -- Evolving Large Language Models into Autonomous Experts with Human-Level Competenciescited by 9
  32. 2024Enabling Weak LLMs to Judge Response Reliability via Meta Rankingcited by 9
  33. 2025Exploring Expert Failures Improves LLM Agent Tuningcited by 9
  34. 2025Beyond Markovian: Reflective Exploration via Bayes-Adaptive RL for LLM Reasoningcited by 9
  35. 2024From Text to Life: On the Reciprocal Relationship between Artificial Life and Large Language Modelscited by 8
  36. 2025LLM Applications: Current Paradigms and the Next Frontiercited by 8
  37. 2025Dynamic Risk Assessments for Offensive Cybersecurity Agentscited by 8
  38. 2026Quantifying Trust: Financial Risk Management for Trustworthy AI Agentscited by 8

50 most cited of 81 All 81 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025EvolveR: Self-Evolving LLM Agents through an Experience-Driven Lifecycle
  2. 2026SAGE: Multi-Agent Self-Evolution for LLM Reasoning
  3. 2025Reinforce LLM Reasoning through Multi-Agent Reflection
  4. 2025KnowCoder-A1: Incentivizing Agentic Reasoning Capability with Outcome Supervision for KBQA
  5. 2025Reinforcement Fine-Tuning for Reasoning towards Multi-Step Multi-Source Search in Large Language Models
  6. 2026PRISMA: Reinforcement Learning Guided Two-Stage Policy Optimization in Multi-Agent Architecture for Open-Domain Multi-Hop Question Answering
  7. 2025Agent-Based Detection and Resolution of Incompleteness and Ambiguity in Interactions with Large Language Models
  8. 2025Knowledge-Aware Iterative Retrieval for Multi-Agent Systems