# Shockingly Simple Self-retrospection Improves Agentic Models Without RL

arXiv: https://arxiv.org/abs/2609.35741 · HTML: https://mlsanity.com/paper/2609.35741/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use.
- Builds on 40 of its 44 references that are in the map.
- First posted on arXiv 28 Sep 2026.

## Placed in

- LLM agents and tool use / [Reinforcement learning for LLM agents](https://mlsanity.com/agents/reinforcement-learning-for-llm-agents/)

## Builds on (most cited first, 30 of 40)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [A General Language Assistant as a Laboratory for Alignment](https://mlsanity.com/paper/2112.00861/) (2021)
- [Tulu 3: Pushing Frontiers in Open Language Model Post-Training](https://mlsanity.com/paper/2411.15124/) (2024)
- [STaR: Bootstrapping Reasoning With Reasoning](https://mlsanity.com/paper/2203.14465/) (2022)
- [ExpeL: LLM Agents Are Experiential Learners](https://mlsanity.com/paper/2308.10144/) (2023)
- [L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learning](https://mlsanity.com/paper/2503.04697/) (2025)
- [Reinforcement Learning via Self-Distillation](https://mlsanity.com/paper/2601.20802/) (2026)
- [SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?](https://mlsanity.com/paper/2509.16941/) (2025)
- [AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning](https://mlsanity.com/paper/2505.24298/) (2025)
- [On-Policy Context Distillation for Language Models](https://mlsanity.com/paper/2602.12275/) (2026)
- [SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents](https://mlsanity.com/paper/2505.20411/) (2025)
- [Learning by Distilling Context](https://mlsanity.com/paper/2209.15189/) (2022)
- [Training Language Models with Language Feedback at Scale](https://mlsanity.com/paper/2303.16755/) (2023)
- [Retroformer: Retrospective Large Language Agents with Policy Gradient Optimization](https://mlsanity.com/paper/2308.02151/) (2023)
- [Privileged Information Distillation for Language Models](https://mlsanity.com/paper/2602.04942/) (2026)
- [Agent Learning via Early Experience](https://mlsanity.com/paper/2510.08558/) (2025)
- [Improving Code Generation by Training with Natural Language Feedback](https://mlsanity.com/paper/2303.16749/) (2023)
- [Critique Fine-Tuning: Learning to Critique is More Effective than Learning to Imitate](https://mlsanity.com/paper/2501.17703/) (2025)
- [Expanding the Capabilities of Reinforcement Learning via Text Feedback](https://mlsanity.com/paper/2602.02482/) (2026)
- [AvalonBench: Evaluating LLMs Playing the Game of Avalon](https://mlsanity.com/paper/2310.05036/) (2023)
- [Dialogue Learning With Human-In-The-Loop](https://mlsanity.com/paper/1611.09823/) (2016)
- [Embarrassingly Simple Self-Distillation Improves Code Generation](https://mlsanity.com/paper/2604.01193/) (2026)
- [Rethinking the Trust Region in LLM Reinforcement Learning](https://mlsanity.com/paper/2602.04879/) (2026)
- [Online Experiential Learning for Language Models](https://mlsanity.com/paper/2603.16856/) (2026)
- [Experiential Reinforcement Learning](https://mlsanity.com/paper/2602.13949/) (2026)
- [On the Effect of Sampling Diversity in Scaling LLM Inference](https://mlsanity.com/paper/2502.11027/) (2025)

## Abstract (arXiv)

People learn not only by repeating successful actions, but also by recounting and explaining their experiences, revising their understanding to guide future behavior. Can a language-model agent improve its future actions by training only on explanations of its own experience? We investigate this question by studying Retrospection-Only Fine-Tuning (ROFT), a minimal online procedure designed to isolate the effect of explanation-only training on subsequent behavior. The agent attempts a task, observes available feedback, generates a retrospective explanation, and is fine-tuned with a next-token prediction loss on the explanation tokens alone. The procedure uses neither an external teacher nor a reward-based policy update. In software-engineering experiments with Qwen3.5-4B, ROFT is trained on problems with mixed successful and unsuccessful base-model attempts. On held-out SWE-bench Verified and Pro, it reaches 49.2% and 26.8% solve rates after 20 updates without using a verifier, compared with GRPO's 48.0% and 25.3% after 40 updates in the evaluated runs, and makes faster early progress in training time and sampled attempts. It also learns to solve individual tasks on which all 64 sampled base-model attempts failed, showing that learning can begin without any initially successful trajectories. Behavioral analyses find that ROFT indirectly assigns credit to actions, encouraging good actions and discouraging incorrect ones. Moreover, prompting retrospections to emphasize more direct solutions yields shorter subsequent attempts even without an explicit length penalty. Together, these findings show that learning to explain can also improve learning to do, establishing self-generated retrospections as useful training targets and motivating further study of explanation-to-action transfer.