mlsanityData to 7 Oct 2026

Shockingly Simple Self-retrospection Improves Agentic Models Without RL

arXiv 2609.35741 ↗First posted 28 Sep 2026cs.AI cs.CLcited by 0

On the map: LLM agents and tool use → Reinforcement learning for LLM agents also fits: Agent skill learning and evolution

Abstract (arXiv)

People learn not only by repeating successful actions, but also by recounting and explaining their experiences, revising their understanding to guide future behavior. Can a language-model agent improve its future actions by training only on explanations of its own experience? We investigate this question by studying Retrospection-Only Fine-Tuning (ROFT), a minimal online procedure designed to isolate the effect of explanation-only training on subsequent behavior. The agent attempts a task, observes available feedback, generates a retrospective explanation, and is fine-tuned with a next-token prediction loss on the explanation tokens alone. The procedure uses neither an external teacher nor a reward-based policy update. In software-engineering experiments with Qwen3.5-4B, ROFT is trained on problems with mixed successful and unsuccessful base-model attempts. On held-out SWE-bench Verified and Pro, it reaches 49.2% and 26.8% solve rates after 20 updates without using a verifier, compared with GRPO's 48.0% and 25.3% after 40 updates in the evaluated runs, and makes faster early progress in training time and sampled attempts. It also learns to solve individual tasks on which all 64 sampled base-model attempts failed, showing that learning can begin without any initially successful trajectories. Behavioral analyses find that ROFT indirectly assigns credit to actions, encouraging good actions and discouraging incorrect ones. Moreover, prompting retrospections to emphasize more direct solutions yields shorter subsequent attempts even without an explicit length penalty. Together, these findings show that learning to explain can also improve learning to do, establishing self-generated retrospections as useful training targets and motivating further study of explanation-to-action transfer.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 40 of its 44 references that are in the map. First posted on arXiv 28 Sep 2026.

Builds on

40 of 44 references.

Closest ancestors · 4

Same direction, newest first

  1. 2026HERO: Hindsight-Enhanced Reflection from Environment Observations for Agentic Self-Distillation10 Jun
  2. 2023AvalonBench: Evaluating LLMs Playing the Game of Avalon8 Oct
  3. 2023Retroformer: Retrospective Large Language Agents with Policy Gradient Optimization4 Aug
  4. 2023Reflexion: Language Agents with Verbal Reinforcement Learning20 Mar

Other references · 36

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  3. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  4. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  5. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  6. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  7. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  8. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 814
  9. 2025L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learningcited by 381
  10. 2026Reinforcement Learning via Self-Distillationcited by 285
  11. 2025SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?cited by 254
  12. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoningcited by 234
24 more
  1. 2026On-Policy Context Distillation for Language Modelscited by 161
  2. 2022Learning by Distilling Contextcited by 128
  3. 2025SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agentscited by 127
  4. 2023Training Language Models with Language Feedback at Scalecited by 125
  5. 2025Agent Learning via Early Experiencecited by 79
  6. 2026Privileged Information Distillation for Language Modelscited by 79
  7. 2023Improving Code Generation by Training with Natural Language Feedbackcited by 75
  8. 2025Critique Fine-Tuning: Learning to Critique is More Effective than Learning to Imitatecited by 61
  9. 2026Expanding the Capabilities of Reinforcement Learning via Text Feedbackcited by 49
  10. 2016Dialogue Learning With Human-In-The-Loopcited by 46
  11. 2026Embarrassingly Simple Self-Distillation Improves Code Generationcited by 36
  12. 2026Rethinking the Trust Region in LLM Reinforcement Learningcited by 32
  13. 2026Online Experiential Learning for Language Modelscited by 24
  14. 2026Experiential Reinforcement Learningcited by 21
  15. 2025On the Effect of Sampling Diversity in Scaling LLM Inferencecited by 18
  16. 2024Strategist: Self-improvement of LLM Decision Making via Bi-Level Tree Searchcited by 16
  17. 2024Scattered Forest Search: Smarter Code Space Exploration with LLMscited by 10
  18. 2025DISC: Dynamic Decomposition Improves LLM Inference Scalingcited by 9
  19. 2026Actor-Curator: Co-adaptive Curriculum Learning via Policy-Improvement Bandits for RL Post-Trainingcited by 7
  20. 2026Learning with Rare Success but Rich Feedback via Reflection-Enhanced Self-Distillationcited by 6
  21. 2026Teaching Large Reasoning Models Effective Reflectioncited by 3
  22. 2026Procedural Memory Distillation: Online Reflection for Self-Improving Language Modelscited by 3
  23. 2026FrogNano: Training a 4B Coding Agent via Online Task Synthesiscited by 3
  24. 2024PIANIST: Learning Partially Observable World Models with LLMs for Multi-Agent Decision Makingcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Recursive Introspection: Teaching Language Model Agents How to Self-Improve
  2. 2026RetroAgent: From Solving to Evolving via Retrospective Dual Intrinsic Feedback
  3. 2025Agent-R: Training Language Model Agents to Reflect via Iterative Self-Training
  4. 2026SkillAdaptor: Self-Adapting Skills for LLM Agents from Trajectories
  5. 2026Thinking Outside the Box: Can Language Models Rely on External Guidance Selectively?
  6. 2025AgentRefine: Enhancing Agent Generalization through Refinement Tuning
  7. 2025Retrospex: Language Agent Meets Offline Reinforcement Learning Critic
  8. 2026Self-Designed Evaluators and Warm Memory for Long-Horizon Agents