# RLTL;DR: Self-improvement by Internalizing Self-generated Feedback

arXiv: https://arxiv.org/abs/2609.37633 · HTML: https://mlsanity.com/paper/2609.37633/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 37 references that are in the map.
- First posted on arXiv 29 Sep 2026.

## Builds on (most cited first, 30 of 31)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [Understanding R1-Zero-Like Training: A Critical Perspective](https://mlsanity.com/paper/2503.20783/) (2025)
- [A General Language Assistant as a Laboratory for Alignment](https://mlsanity.com/paper/2112.00861/) (2021)
- [Tulu 3: Pushing Frontiers in Open Language Model Post-Training](https://mlsanity.com/paper/2411.15124/) (2024)
- [Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?](https://mlsanity.com/paper/2504.13837/) (2025)
- [Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes](https://mlsanity.com/paper/2305.02301/) (2023)
- [AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents](https://mlsanity.com/paper/2407.18901/) (2024)
- [Reinforcement Learning via Self-Distillation](https://mlsanity.com/paper/2601.20802/) (2026)
- [RL's Razor: Why Online Reinforcement Learning Forgets Less](https://mlsanity.com/paper/2509.04259/) (2025)
- [Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models](https://mlsanity.com/paper/2508.10751/) (2025)
- [MemRL: Self-Evolving Agents via Runtime Reinforcement Learning on Episodic Memory](https://mlsanity.com/paper/2601.03192/) (2026)
- [SKILL0: In-Context Agentic Reinforcement Learning for Skill Internalization](https://mlsanity.com/paper/2604.02268/) (2026)
- [SPICE: Self-Play In Corpus Environments Improves Reasoning](https://mlsanity.com/paper/2510.24684/) (2025)
- [STP: Self-play LLM Theorem Provers with Iterative Conjecturing and Proving](https://mlsanity.com/paper/2502.00212/) (2025)
- [LeetCodeDataset: A Temporal Dataset for Robust Evaluation and Efficient Training of Code LLMs](https://mlsanity.com/paper/2504.14655/) (2025)
- [Expanding the Capabilities of Reinforcement Learning via Text Feedback](https://mlsanity.com/paper/2602.02482/) (2026)
- [Scaf-GRPO: Scaffolded Group Relative Policy Optimization for Enhancing LLM Reasoning](https://mlsanity.com/paper/2510.19807/) (2025)
- [SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning](https://mlsanity.com/paper/2607.14777/) (2026)
- [Nudging the Boundaries of LLM Reasoning](https://mlsanity.com/paper/2509.25666/) (2025)
- [Trained on Tokens, Calibrated on Concepts: The Emergence of Semantic Calibration in LLMs](https://mlsanity.com/paper/2511.04869/) (2025)
- [ECHO: Terminal Agents Learn World Models for Free](https://mlsanity.com/paper/2605.24517/) (2026)
- [Policy and World Modeling Co-Training for Language Agents](https://mlsanity.com/paper/2606.02388/) (2026)
- [WikiSkill: Compiling Agent Experience into Persistent Knowledge for Skill Evolution](https://mlsanity.com/paper/2608.27454/) (2026)
- [Expanding LLM Agent Boundaries with Strategy-Guided Exploration](https://mlsanity.com/paper/2603.02045/) (2026)
- [Off-Context GRPO: Learning to Reason on Hard Problems using Privileged Information](https://mlsanity.com/paper/2607.19313/) (2026)
- [Contextual Drag: How Errors in the Context Affect LLM Reasoning](https://mlsanity.com/paper/2602.04288/) (2026)
- [Rethinking Continual Experience Internalization for Self-Evolving LLM Agents](https://mlsanity.com/paper/2606.04703/) (2026)
- [When Context Returns: Toward Robust Internalization in On-Policy Distillation](https://mlsanity.com/paper/2606.11627/) (2026)
- [Programming by Backprop: An Instruction is Worth 100 Examples When Finetuning LLMs](https://mlsanity.com/paper/2506.18777/) (2025)

## Built on by (most cited first, 1 of 1)

- [ECG-RSI: Recursive Self-Improvement through Targeted Electrocardiogram Waveform Evolution](https://mlsanity.com/paper/2610.17718/) (2026)

## Abstract (arXiv)

The common paradigm of reinforcement learning with verifiable rewards (RLVR) is to let agents make multiple attempts at a task, and optimize towards the successful ones. This becomes problematic in the realms of self-improvement, where tasks are so difficult that the agent has a low or even no chance of success, and where there are no teacher models or example solutions to distill from. In this paper, we introduce RLTL;DR. After each failed attempt, we show the policy the verifier outputs and let it write its own feedback, in the form of a single TL;DR insight. The next rollout is conditioned on all previous insights, and we sequentially sample rollouts until a solution is found. Moreover, we enable backpropagation on the in-context insights to internalize a direct task to insight mapping. On challenging tool-calling and coding datasets (filtered to Pass@128=0), standard GRPO training of a Qwen 3.5 9B Thinking policy stays flat at a Pass@1 of 0% to 1%. RLTL;DR breaks through this learning barrier, achieving a Pass@1 of 14-31% with insights in context during training and, crucially, 12-13% when no insight is in context at eval time. We identify that the key is the task to insight internalization. To study this further, we reduce our approach to SFTL;DR, training only on (task, insight) tuples, without showing or backpropagating on any rollouts. Training on only 4k of these tuples recovers almost the full performance of RLTL;DR and classical SFT on full rollouts. This demonstrates a promising compacted training paradigm of the form "on this sort of task, keep this sort of thing in mind", which we hope to inspire future research on.