# Online Experiential Learning for Language Models

arXiv: https://arxiv.org/abs/2603.16856 · HTML: https://mlsanity.com/paper/2603.16856/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 18 references that are in the map.
- First posted on arXiv 17 Mar 2026.

## Builds on (most cited first, 15 of 15)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [Instruction-Following Evaluation for Large Language Models](https://mlsanity.com/paper/2311.07911/) (2023)
- [A General Language Assistant as a Laboratory for Alignment](https://mlsanity.com/paper/2112.00861/) (2021)
- [On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes](https://mlsanity.com/paper/2306.13649/) (2023)
- [ExpeL: LLM Agents Are Experiential Learners](https://mlsanity.com/paper/2308.10144/) (2023)
- [On-Policy Context Distillation for Language Models](https://mlsanity.com/paper/2602.12275/) (2026)
- [RL's Razor: Why Online Reinforcement Learning Forgets Less](https://mlsanity.com/paper/2509.04259/) (2025)
- [MiniLLM: On-Policy Distillation of Large Language Models](https://mlsanity.com/paper/2306.08543/) (2023)
- [Retaining by Doing: The Role of On-Policy Data in Mitigating Forgetting](https://mlsanity.com/paper/2510.18874/) (2025)
- [Agent Learning via Early Experience](https://mlsanity.com/paper/2510.08558/) (2025)
- [InfiniteICL: Breaking the Limit of Context Window Size via Long Short-term Memory Transformation](https://mlsanity.com/paper/2504.01707/) (2025)
- [Cogito, Ergo Ludo: An Agent that Learns to Play by Reasoning and Planning](https://mlsanity.com/paper/2509.25052/) (2025)

## Built on by (most cited first, 28 of 28)

- [Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe](https://mlsanity.com/paper/2604.13016/) (2026)
- [A Survey of On-Policy Distillation for Large Language Models](https://mlsanity.com/paper/2604.00626/) (2026)
- [Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents](https://mlsanity.com/paper/2604.10674/) (2026)
- [Skill1: Unified Evolution of Skill-Augmented Agents via Reinforcement Learning](https://mlsanity.com/paper/2605.06130/) (2026)
- [Multi-Turn On-Policy Distillation with Prefix Replay](https://mlsanity.com/paper/2607.04763/) (2026)
- [OPRD: On-Policy Representation Distillation](https://mlsanity.com/paper/2606.06021/) (2026)
- [Training with Harnesses: On-Policy Harness Self-Distillation for Complex Reasoning](https://mlsanity.com/paper/2605.08741/) (2026)
- [Tailoring Teaching to Aptitude: Direction-Adaptive Self-Distillation for LLM Reasoning](https://mlsanity.com/paper/2605.22263/) (2026)
- [Learning from Language Feedback via Variational Policy Distillation](https://mlsanity.com/paper/2605.15113/) (2026)
- [A Formula-Driven Survey and Research Agenda for On-Policy Distillation](https://mlsanity.com/paper/2606.22793/) (2026)
- [Procedural Memory Distillation: Online Reflection for Self-Improving Language Models](https://mlsanity.com/paper/2607.01480/) (2026)
- [Rethinking Continual Experience Internalization for Self-Evolving LLM Agents](https://mlsanity.com/paper/2606.04703/) (2026)
- [LLM-as-a-Coach: Experiential Learning for Non-Verifiable Tasks](https://mlsanity.com/paper/2607.18110/) (2026)
- [AgentStream: How Well Do Self-Evolving LLM Agents Perform Under Streaming Tasks?](https://mlsanity.com/paper/2608.00155/) (2026)
- [Sample-Efficient Learning from Agent Experience](https://mlsanity.com/paper/2607.21051/) (2026)
- [Language Models Need Sleep: Learning to Self-Modify and Consolidate Memories](https://mlsanity.com/paper/2606.03979/) (2026)
- [Evidence Over Plans: Online Trajectory Verification for Skill Distillation](https://mlsanity.com/paper/2605.09192/) (2026)
- [A Predictive Law for On-Policy Self-Distillation From World Feedback](https://mlsanity.com/paper/2605.30070/) (2026)
- [Flux-OPD: On-Policy Distillation with Evolving Contexts](https://mlsanity.com/paper/2607.28022/) (2026)
- [ORACLE: Anticipating Scams from Partial Trajectories in Streaming App Usage](https://mlsanity.com/paper/2605.16363/) (2026)
- [Echo: Learning from Experience Data via User-Driven Refinement](https://mlsanity.com/paper/2605.21984/) (2026)
- [AdaStop: Cost-Aware Early Stopping for DNN Test Selection](https://mlsanity.com/paper/2607.05461/) (2026)
- [Few-Shot Demonstrations Elicit the Use of In-Context World Representations in LLMs](https://mlsanity.com/paper/2609.24352/) (2026)
- [Shockingly Simple Self-retrospection Improves Agentic Models Without RL](https://mlsanity.com/paper/2609.35741/) (2026)
- [ReSAIL: Mitigating Collapse in Iterative Agent Self-Distillation](https://mlsanity.com/paper/2609.39306/) (2026)
- [Flash-OPD: Fast On-Policy Distillation](https://mlsanity.com/paper/2610.06105/) (2026)
- [Environmental Feedback Modeling Matters: Rethinking Feedback Treatment in Agentic Hindsight Self-Distillation](https://mlsanity.com/paper/2610.11384/) (2026)
- [CUSE: Fine-Grained Multimodal Unlearning with Counterfactual Guidance and On-Policy Self-Distillation](https://mlsanity.com/paper/2610.14763/) (2026)

## Abstract (arXiv)

The prevailing paradigm for improving large language models relies on offline training with human annotations or simulated environments, leaving the rich experience accumulated during real-world deployment entirely unexploited. We propose Online Experiential Learning (OEL), a framework that enables language models to continuously improve from their own deployment experience. OEL operates in two stages: first, transferable experiential knowledge is extracted and accumulated from interaction trajectories collected on the user side; second, this knowledge is consolidated into model parameters via on-policy context distillation, requiring no access to the user-side environment. The two stages are iterated to form an online learning loop, where the improved model collects higher-quality trajectories that yield richer experiential knowledge for subsequent rounds. We evaluate OEL on text-based game environments across multiple model scales and both thinking and non-thinking variants. OEL achieves consistent improvements over successive iterations, enhancing both task accuracy and token efficiency while preserving out-of-distribution performance. Our analysis further shows that extracted experiential knowledge is significantly more effective than raw trajectories, and that on-policy consistency between the knowledge source and the policy model is critical for effective learning.