# ALER: Adaptive Learnable Experience Rewriting for Reinforcement Learning

arXiv: https://arxiv.org/abs/2610.00592 · HTML: https://mlsanity.com/paper/2610.00592/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 37 of its 45 references that are in the map.
- First posted on arXiv 30 Sep 2026.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 30 of 37)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [DeepMind Control Suite](https://mlsanity.com/paper/1801.00690/) (2018)
- [Simplified State Space Layers for Sequence Modeling](https://mlsanity.com/paper/2208.04933/) (2022)
- [Neural Turing Machines](https://mlsanity.com/paper/1410.5401/) (2014)
- [Leveraging Procedural Generation to Benchmark Reinforcement Learning](https://mlsanity.com/paper/1912.01588/) (2019)
- [Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Tasks](https://mlsanity.com/paper/2306.13831/) (2023)
- [Stabilizing Transformers for Reinforcement Learning](https://mlsanity.com/paper/1910.06764/) (2019)
- [Neural Episodic Control](https://mlsanity.com/paper/1703.01988/) (2017)
- [Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPs](https://mlsanity.com/paper/2110.05038/) (2021)
- [Structured State Space Models for In-Context Reinforcement Learning](https://mlsanity.com/paper/2303.03982/) (2023)
- [Memory-based control with recurrent neural networks](https://mlsanity.com/paper/1512.04455/) (2015)
- [Memory-Augmented Reinforcement Learning for Image-Goal Navigation](https://mlsanity.com/paper/2101.05181/) (2021)
- [When Do Transformers Shine in RL? Decoupling Memory from Credit Assignment](https://mlsanity.com/paper/2307.03864/) (2023)
- [POPGym: Benchmarking Partially Observable Reinforcement Learning](https://mlsanity.com/paper/2303.01859/) (2023)
- [Unsupervised Predictive Memory in a Goal-Directed Agent](https://mlsanity.com/paper/1803.10760/) (2018)
- [Obstacle Tower: A Generalization Challenge in Vision, Control, and Planning](https://mlsanity.com/paper/1902.01378/) (2019)
- [Evaluating Long-Term Memory in 3D Mazes](https://mlsanity.com/paper/2210.13383/) (2022)
- [Towards mental time travel: a hierarchical memory for reinforcement learning agents](https://mlsanity.com/paper/2105.14039/) (2021)
- [Generalization of Reinforcement Learners with Working and Episodic Memory](https://mlsanity.com/paper/1910.13406/) (2019)
- [Deep Transformer Q-Networks for Partially Observable Reinforcement Learning](https://mlsanity.com/paper/2206.01078/) (2022)
- [Memory, Benchmark & Robots: A Benchmark for Solving Complex Tasks with Reinforcement Learning](https://mlsanity.com/paper/2502.10550/) (2025)
- [Recurrent Action Transformer with Memory](https://mlsanity.com/paper/2306.09459/) (2023)
- [Memory Gym: Towards Endless Tasks to Benchmark Memory Capabilities of Agents](https://mlsanity.com/paper/2309.17207/) (2023)
- [Memory Augmented Neural Networks with Wormhole Connections](https://mlsanity.com/paper/1701.08718/) (2017)
- [Remember When It Matters: Proactive Memory Agent for Long-Horizon Agents](https://mlsanity.com/paper/2607.08716/) (2026)
- [Reinforcement Learning with Fast and Forgetful Memory](https://mlsanity.com/paper/2310.04128/) (2023)
- [Stable Hadamard Memory: Revitalizing Memory-Augmented Agents for Reinforcement Learning](https://mlsanity.com/paper/2410.10132/) (2024)

## Abstract (arXiv)

In partially observable reinforcement learning (RL), a later observation can make stored information obsolete or change what it implies for the next decision. Memory architectures and benchmarks for RL mostly test retention, the ability to keep information unchanged until it is needed. We formalize two further requirements. Rewriting sets the decision-relevant content to a value independent of the old one, and experience fusion transforms the old content by a rule that a later observation specifies. For tasks built from such updates, we count the memory states that a solution needs, and several baselines reach their lowest success rates on compositions that need more states. We introduce ALER (Adaptive Learnable Experience Rewriting), an agent that pairs an LSTM with a slot memory. An independently addressed Gumbel-Softmax write that concentrates its weight on one slot overwrites that slot, and a learned gate fuses the retrieved content with the recurrent state before the policy and value heads. We also introduce Rune-Mazes, three environments in which rune observations invert, cancel, reset, or repeat updates of a hidden cue under vector and pixel observations. Against seven baselines, ALER reaches a success rate of at least 0.82 in all sixteen Endless T-Maze configurations and at least 0.99 on all five Rune T-Maze compositions, and it has the highest mean success rate on four-branch Rune Multi-Corridor with an Invert rune. On pixel-based Rune MiniGrid Memory, it has a higher mean success rate than PPO-LSTM in eight of ten configurations. Project page: https://quartz-admirer.github.io/ALER-Adaptive-Learnable-Experience-Rewriting/.