# Learning From Failures: Efficient Reinforcement Learning Control with Episodic Memory

arXiv: https://arxiv.org/abs/2603.07110 · HTML: https://mlsanity.com/paper/2603.07110/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 22 of its 30 references that are in the map.
- First posted on arXiv 7 Mar 2026.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 22 of 22)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Learning agile and dynamic motor skills for legged robots](https://mlsanity.com/paper/1901.08652/) (2019)
- [Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learning](https://mlsanity.com/paper/2410.21845/) (2024)
- [Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes](https://mlsanity.com/paper/2408.03539/) (2024)
- [Recovery RL: Safe Reinforcement Learning with Learned Recovery Zones](https://mlsanity.com/paper/2010.15920/) (2020)
- [Learning to Walk via Deep Reinforcement Learning](https://mlsanity.com/paper/1812.11103/) (2018)
- [Neural Episodic Control](https://mlsanity.com/paper/1703.01988/) (2017)
- [Model-Free Episodic Control](https://mlsanity.com/paper/1606.04460/) (2016)
- [CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicity](https://mlsanity.com/paper/1902.05605/) (2019)
- [DreamWaQ: Learning Robust Quadrupedal Locomotion With Implicit Terrain Imagination via Deep Reinforcement Learning](https://mlsanity.com/paper/2301.10602/) (2023)
- [Episodic Memory Deep Q-Networks](https://mlsanity.com/paper/1805.07603/) (2018)
- [Learning Robotic Manipulation through Visual Planning and Acting](https://mlsanity.com/paper/1905.04411/) (2019)
- [Learning Quadruped Locomotion Using Differentiable Simulation](https://mlsanity.com/paper/2403.14864/) (2024)
- [Generalizable Episodic Memory for Deep Reinforcement Learning](https://mlsanity.com/paper/2103.06469/) (2021)
- [When to Ask for Help: Proactive Interventions in Autonomous Reinforcement Learning](https://mlsanity.com/paper/2210.10765/) (2022)
- [Safe Reinforcement Learning of Dynamic High-Dimensional Robotic Tasks: Navigation, Manipulation, Interaction](https://mlsanity.com/paper/2209.13308/) (2022)
- [Asynchronous Episodic Deep Deterministic Policy Gradient: Towards Continuous Control in Computationally Complex Environments](https://mlsanity.com/paper/1903.00827/) (2019)
- [Solving Continuous Control with Episodic Memory](https://mlsanity.com/paper/2106.08832/) (2021)
- [Efficient Episodic Memory Utilization of Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2403.01112/) (2024)

## Abstract (arXiv)

Reinforcement learning has achieved remarkable success in robot learning. However, under challenging exploration and contact-rich dynamics, early-stage training is frequently dominated by premature terminations such as collisions and falls. As a result, learning is overwhelmed by short-horizon, low-return trajectories, which hinder convergence and limit long-horizon exploration. To alleviate this issue, we propose a technique called Failure Episodic Memory Alert (FEMA). FEMA explicitly stores short-horizon failure experiences through an episodic memory module. During interactions, it retrieves similar failure experiences and prevents the robot from recurrently relapsing into unstable states, guiding the policy toward long-horizon trajectories with greater long-term value. FEMA can be combined easily with model-free reinforcement learning algorithms, and yields a substantial sample-efficiency improvement of 33.11% on MuJoCo tasks across several classical RL algorithms. Furthermore, integrating FEMA into a parallelized PPO training pipeline demonstrates its effectiveness on a real-world bipedal robot task.