mlsanityData to 14 Oct 2026

RLTL;DR: Self-improvement by Internalizing Self-generated Feedback

arXiv 2609.37633 ↗First posted 29 Sep 2026cs.LG cs.AI cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

The common paradigm of reinforcement learning with verifiable rewards (RLVR) is to let agents make multiple attempts at a task, and optimize towards the successful ones. This becomes problematic in the realms of self-improvement, where tasks are so difficult that the agent has a low or even no chance of success, and where there are no teacher models or example solutions to distill from. In this paper, we introduce RLTL;DR. After each failed attempt, we show the policy the verifier outputs and let it write its own feedback, in the form of a single TL;DR insight. The next rollout is conditioned on all previous insights, and we sequentially sample rollouts until a solution is found. Moreover, we enable backpropagation on the in-context insights to internalize a direct task to insight mapping. On challenging tool-calling and coding datasets (filtered to Pass@128=0), standard GRPO training of a Qwen 3.5 9B Thinking policy stays flat at a Pass@1 of 0% to 1%. RLTL;DR breaks through this learning barrier, achieving a Pass@1 of 14-31% with insights in context during training and, crucially, 12-13% when no insight is in context at eval time. We identify that the key is the task to insight internalization. To study this further, we reduce our approach to SFTL;DR, training only on (task, insight) tuples, without showing or backpropagating on any rollouts. Training on only 4k of these tuples recovers almost the full performance of RLTL;DR and classical SFT on full rollouts. This demonstrates a promising compacted training paradigm of the form "on this sort of task, keep this sort of thing in mind", which we hope to inspire future research on.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 31 of its 37 references that are in the map. First posted on arXiv 29 Sep 2026.

Builds on

31 of 37 references.

References · 31

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  3. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,345
  4. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  5. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  6. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 982
  7. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  8. 2024AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agentscited by 401
  9. 2026Reinforcement Learning via Self-Distillationcited by 332
  10. 2025RL's Razor: Why Online Reinforcement Learning Forgets Lesscited by 176
  11. 2025Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Modelscited by 138
  12. 2026MemRL: Self-Evolving Agents via Runtime Reinforcement Learning on Episodic Memorycited by 129
19 more
  1. 2026SKILL0: In-Context Agentic Reinforcement Learning for Skill Internalizationcited by 94
  2. 2025SPICE: Self-Play In Corpus Environments Improves Reasoningcited by 83
  3. 2025STP: Self-play LLM Theorem Provers with Iterative Conjecturing and Provingcited by 79
  4. 2025LeetCodeDataset: A Temporal Dataset for Robust Evaluation and Efficient Training of Code LLMscited by 70
  5. 2026Expanding the Capabilities of Reinforcement Learning via Text Feedbackcited by 53
  6. 2025Scaf-GRPO: Scaffolded Group Relative Policy Optimization for Enhancing LLM Reasoningcited by 52
  7. 2026SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learningcited by 32
  8. 2025Nudging the Boundaries of LLM Reasoningcited by 26
  9. 2025Trained on Tokens, Calibrated on Concepts: The Emergence of Semantic Calibration in LLMscited by 19
  10. 2026ECHO: Terminal Agents Learn World Models for Freecited by 19
  11. 2026Policy and World Modeling Co-Training for Language Agentscited by 15
  12. 2026WikiSkill: Compiling Agent Experience into Persistent Knowledge for Skill Evolutioncited by 12
  13. 2026Expanding LLM Agent Boundaries with Strategy-Guided Explorationcited by 9
  14. 2026Off-Context GRPO: Learning to Reason on Hard Problems using Privileged Informationcited by 7
  15. 2026Contextual Drag: How Errors in the Context Affect LLM Reasoningcited by 4
  16. 2026Rethinking Continual Experience Internalization for Self-Evolving LLM Agentscited by 4
  17. 2026When Context Returns: Toward Robust Internalization in On-Policy Distillationcited by 3
  18. 2025Programming by Backprop: An Instruction is Worth 100 Examples When Finetuning LLMscited by 2
  19. 2026CORE: Contrastive Reflection Enables Rapid Improvements in Reasoningcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026ECG-RSI: Recursive Self-Improvement through Targeted Electrocardiogram Waveform Evolutioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026From RLVR to RLSVR: Task Transformation Induces Self-Verifiable Rewards for Open-Ended LLM Self-Improvement
  2. 2025RLSR: Reinforcement Learning from Self Reward
  3. 2025Extending RLVR to Open-Ended Tasks via Verifiable Multiple-Choice Reformulation
  4. 2026Verifier-Induced Support Reshaping in On-Policy Optimization
  5. 2025Trust, But Verify: A Self-Verification Approach to Reinforcement Learning with Verifiable Rewards
  6. 2025ConfClip: Confidence-Weighted and Clipped Reward for Reinforcement Learning in LLMs
  7. 2025SERL: Self-Examining Reinforcement Learning on Open-Domain
  8. 2026MulFeRL: Enhancing Reinforcement Learning with Verbal Feedback in a Multi-turn Loop