mlsanityData to 14 Oct 2026

Re:Frame -- Retrieving Experience From Associative Memory

arXiv 2508.19344 ↗First posted 26 Aug 2025cs.LG cs.AIcited by 1

On the map: Offline RL and imitation learning from data → Offline RL algorithms

also in 1 more
Abstract (arXiv)

Offline reinforcement learning (RL) often deals with suboptimal data when collecting large expert datasets is unavailable or impractical. This limitation makes it difficult for agents to generalize and achieve high performance, as they must learn primarily from imperfect or inconsistent trajectories. A central challenge is therefore how to best leverage scarce expert demonstrations alongside abundant but lower-quality data. We demonstrate that incorporating even a tiny amount of expert experience can substantially improve RL agent performance. We introduce Re:Frame (Retrieving Experience From Associative Memory), a plug-in module that augments a standard offline RL policy (e.g., Decision Transformer) with a small external Associative Memory Buffer (AMB) populated by expert trajectories drawn from a separate dataset. During training on low-quality data, the policy learns to retrieve expert data from the Associative Memory Buffer (AMB) via content-based associations and integrate them into decision-making; the same AMB is queried at evaluation. This requires no environment interaction and no modifications to the backbone architecture. On D4RL MuJoCo tasks, using as few as 60 expert trajectories (0.1% of a 6000-trajectory dataset), Re:Frame consistently improves over a strong Decision Transformer baseline in three of four settings, with gains up to +10.7 normalized points. These results show that Re:Frame offers a simple and data-efficient way to inject scarce expert knowledge and substantially improve offline RL from low-quality datasets.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 20 of its 41 references that are in the map. First posted on arXiv 26 Aug 2025.

Builds on

20 of 41 references.

Closest ancestors · 9

Same direction, newest first

  1. 2023Guided Data Augmentation for Offline Reinforcement Learning and Imitation Learning27 Oct
  2. 2023Hundreds Guide Millions: Adaptive Offline Reinforcement Learning with Expert Guidance4 Sep
  3. 2023Optimal Transport for Offline Imitation Learning24 Mar
  4. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  5. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  6. 2020PLAS: Latent Action Space for Offline Reinforcement Learning14 Nov
  7. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  8. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  9. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 11

  1. 2017Billion-scale similarity search with GPUscited by 2,497
  2. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  3. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  4. 2014Neural Turing Machinescited by 644
  5. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
  6. 2020Learning Associative Inference Using Fast Weight Memorycited by 51
  7. 2020Self-Attentive Associative Memorycited by 33
  8. 2023Recurrent Action Transformer with Memorycited by 17
  9. 2024Associative Recurrent Memory Transformercited by 15
  10. 2024Unraveling the Complexity of Memory in RL Agents: an Approach for Classification and Evaluationcited by 5
  11. 2022Associative Memory Based Experience Replay for Deep Reinforcement Learningcited by 2

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025ELMUR: External Layer Memory with Update/Rewrite for Long-Horizon RL Problemscited by 6

Similar, not cited

Close in wording, no citation either way

  1. 2023Using Offline Data to Speed Up Reinforcement Learning in Procedurally Generated Environments
  2. 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning
  3. 2021Representation Matters: Offline Pretraining for Sequential Decision Making
  4. 2025Efficient Reinforcement Learning by Guiding World Models with Non-Curated Data
  5. 2026Complementary RL: Towards Efficient Experience-Driven Agent Learning
  6. 2025Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Models
  7. 2022Bridging the Gap Between Offline and Online Reinforcement Learning Evaluation Methodologies
  8. 2023Offline Retraining for Online RL: Decoupled Policy Learning to Mitigate Exploration Bias