# Experiential Reinforcement Learning

arXiv: https://arxiv.org/abs/2602.13949 · HTML: https://mlsanity.com/paper/2602.13949/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods.
- Builds on 17 of its 21 references that are in the map.
- First posted on arXiv 15 Feb 2026.

## Placed in

- Deep reinforcement learning methods / [RL training of language and agent models](https://mlsanity.com/deeprl/rl-training-of-language-and-agent-models/)

## Builds on (most cited first, 17 of 17)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [STaR: Bootstrapping Reasoning With Reasoning](https://mlsanity.com/paper/2203.14465/) (2022)
- [On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes](https://mlsanity.com/paper/2306.13649/) (2023)
- [Efficient Reinforcement Finetuning via Adaptive Curriculum Learning](https://mlsanity.com/paper/2504.05520/) (2025)
- [Expanding the Capabilities of Reinforcement Learning via Text Feedback](https://mlsanity.com/paper/2602.02482/) (2026)
- [The Hallucination Tax of Reinforcement Finetuning](https://mlsanity.com/paper/2505.13988/) (2025)
- [CoAct-1: Computer-using Multi-Agent System with Coding Actions](https://mlsanity.com/paper/2508.03923/) (2025)
- [WildFeedback: Aligning LLMs With In-situ User Interactions And Feedback](https://mlsanity.com/paper/2408.15549/) (2024)
- [Safer-Instruct: Aligning Language Models with Automated Preference Data](https://mlsanity.com/paper/2311.08685/) (2023)
- [One Model, All Roles: Multi-Turn, Multi-Agent Self-Play Reinforcement Learning for Conversational Social Intelligence](https://mlsanity.com/paper/2602.03109/) (2026)

## Built on by (most cited first, 22 of 22)

- [OpenClaw-RL: Train Any Agent Simply by Talking](https://mlsanity.com/paper/2603.10165/) (2026)
- [Aligning Language Models from User Interactions](https://mlsanity.com/paper/2603.12273/) (2026)
- [OdysSim: Building Foundation Models for Human Behavior Simulation](https://mlsanity.com/paper/2606.14199/) (2026)
- [Learning from Your Own Mistakes: Constructing Learnable Micro-Reflective Trajectories for Self-Distillation](https://mlsanity.com/paper/2606.18844/) (2026)
- [Beyond Scalar Rewards: Dense Feedback for LLM Policy Synthesis in Sequential Social Dilemmas](https://mlsanity.com/paper/2603.19453/) (2026)
- [GUI Agents with Reinforcement Learning: Toward Digital Inhabitants](https://mlsanity.com/paper/2604.27955/) (2026)
- [Agent Reinforcement Learning via Pivotal-Aware Self-Feedback Retry](https://mlsanity.com/paper/2607.03702/) (2026)
- [Policy Improvement Reinforcement Learning](https://mlsanity.com/paper/2604.00860/) (2026)
- [Rollout Pass-Rate Control: Steering Binary-Reward RL Toward Its Most Informative Regime](https://mlsanity.com/paper/2605.05112/) (2026)
- [CORE: Contrastive Reflection Enables Rapid Improvements in Reasoning](https://mlsanity.com/paper/2605.28742/) (2026)
- [A Predictive Law for On-Policy Self-Distillation From World Feedback](https://mlsanity.com/paper/2605.30070/) (2026)
- [MetaResearcher: Scaling Deep Research via Self-Reflective Reinforcement Learning in Adversarial Virtual Environments](https://mlsanity.com/paper/2606.19893/) (2026)
- [Procedural Graphs: Self-Evolving Execution Structures for LLM Agents](https://mlsanity.com/paper/2609.09153/) (2026)
- [Beyond Attention: True Adaptive World Models via Spherical Kernel Operator](https://mlsanity.com/paper/2603.13263/) (2026)
- [Prompting Policies for Multi-step Reasoning and Tool-Use in Black-box LLMs with Iterative Distillation of Experience](https://mlsanity.com/paper/2605.14443/) (2026)
- [BUS: Brain-Inspired Unsupervised Self-Reflection via Backward Prediction for Multimodal Reasoning](https://mlsanity.com/paper/2607.07361/) (2026)
- [Taming CoT Obfuscation in VLMs: From Mechanistic Evidence to Activation Enforcement](https://mlsanity.com/paper/2609.24243/) (2026)
- [Few-Shot Demonstrations Elicit the Use of In-Context World Representations in LLMs](https://mlsanity.com/paper/2609.24352/) (2026)
- [Reinforcement Learning with Decomposed Subtasks](https://mlsanity.com/paper/2609.27035/) (2026)
- [Shockingly Simple Self-retrospection Improves Agentic Models Without RL](https://mlsanity.com/paper/2609.35741/) (2026)
- [Test-Time Adaptation of Reasoning Strategies with Bayesian Nonparametric Memory](https://mlsanity.com/paper/2610.06516/) (2026)
- [MIMESIS: Learning User Simulators as Training Environments for Interactive Agents](https://mlsanity.com/paper/2610.09484/) (2026)

## Abstract (arXiv)

Reinforcement learning has become the central approach for language models (LMs) to learn from environmental reward or feedback. In practice, the environmental feedback is usually sparse and delayed. Learning from such signals is challenging, as LMs must implicitly infer how observed failures should translate into behavioral changes for future iterations. We introduce Experiential Reinforcement Learning (ERL), a training paradigm that embeds an explicit experience-reflection-consolidation loop into the reinforcement learning process. Given a task, the model generates an initial attempt, receives environmental feedback, and produces a reflection that guides a refined second attempt, whose success is reinforced and internalized into the base policy. This process converts feedback into structured behavioral revision, improving exploration and stabilizing optimization while preserving gains at deployment without additional inference cost. Across sparse-reward control environments and agentic reasoning benchmarks, ERL consistently improves learning efficiency and final performance over strong reinforcement learning baselines, achieving gains of up to +81% in complex multi-step environments and up to +11% in tool-using reasoning tasks. These results suggest that integrating explicit self-reflection into policy training provides a practical mechanism for transforming feedback into durable behavioral improvement.