# Sample-Efficient Learning from Agent Experience

arXiv: https://arxiv.org/abs/2607.21051 · HTML: https://mlsanity.com/paper/2607.21051/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 65 references that are in the map.
- First posted on arXiv 23 Jul 2026.

## Builds on (most cited first, 21 of 21)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [A General Language Assistant as a Laboratory for Alignment](https://mlsanity.com/paper/2112.00861/) (2021)
- [In-context Learning and Induction Heads](https://mlsanity.com/paper/2209.11895/) (2022)
- [Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes](https://mlsanity.com/paper/2305.02301/) (2023)
- [RL2: Fast Reinforcement Learning via Slow Reinforcement Learning](https://mlsanity.com/paper/1611.02779/) (2016)
- [Learning to reinforcement learn](https://mlsanity.com/paper/1611.05763/) (2016)
- [Reinforcement Learning via Self-Distillation](https://mlsanity.com/paper/2601.20802/) (2026)
- [On-Policy Context Distillation for Language Models](https://mlsanity.com/paper/2602.12275/) (2026)
- [Learning by Distilling Context](https://mlsanity.com/paper/2209.15189/) (2022)
- [Privileged Information Distillation for Language Models](https://mlsanity.com/paper/2602.04942/) (2026)
- [Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents](https://mlsanity.com/paper/2604.10674/) (2026)
- [Doc-to-LoRA: Learning to Instantly Internalize Contexts](https://mlsanity.com/paper/2602.15902/) (2026)
- [In-context Learning Distillation: Transferring Few-shot Learning Ability of Pre-trained Language Models](https://mlsanity.com/paper/2212.10670/) (2022)
- [Online Experiential Learning for Language Models](https://mlsanity.com/paper/2603.16856/) (2026)
- [EdgeBench: Unveiling Scaling Laws of Learning from Real-World Environments](https://mlsanity.com/paper/2607.05155/) (2026)
- [TALES: Text Adventure Learning Environment Suite](https://mlsanity.com/paper/2504.14128/) (2025)
- [Context Distillation as Latent Memory Management](https://mlsanity.com/paper/2605.28889/) (2026)
- [In-Context Learning Distillation for Efficient Few-Shot Fine-Tuning](https://mlsanity.com/paper/2412.13243/) (2024)

## Built on by (most cited first, 3 of 3)

- [ReSAIL: Mitigating Collapse in Iterative Agent Self-Distillation](https://mlsanity.com/paper/2609.39306/) (2026)
- [Not All Experience Belongs in the Weights: Component Routing for Self-Improving GUI Agents](https://mlsanity.com/paper/2610.01787/) (2026)
- [Hindsight Structured Advice Distillation: Steering Executor Agents with Advisors Trained on Their Failures and Task Characteristics](https://mlsanity.com/paper/2610.13000/) (2026)

## Abstract (arXiv)

Real-world agent learning is often constrained by costly environment interactions, such as running time-consuming experiments or obtaining human feedback. In-context learning offers a highly sample-efficient way for agents to learn from their own interaction histories, but its gains disappear once that experience is removed from the context. Separately, context distillation provides a mechanism for internalizing contextual information into model weights. However, applying it to agents' interaction histories without sacrificing environment sample efficiency remains underexplored. We term this problem Experience Distillation and develop an implementation that requires no further environment interaction beyond the collected experience. Experiments on 749 curated software-engineering tasks and six text-adventure games show that it retains at least 64.8% of the gains from in-context learning across both domains, whereas direct supervised fine-tuning on the collected experience recovers only 3.8%. Compared with classical reinforcement-learning baselines, in-context learning from trial-and-error experience followed by Experience Distillation matches their performance with at least (9.6×) fewer environment samples.