mlsanityData to 14 Oct 2026

Large-Scale Retrieval for Reinforcement Learning

arXiv 2206.05314 ↗First posted 10 Jun 2022cs.LG cs.AIcited by 24

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Effective decision making involves flexibly relating past experiences and relevant contextual information to a novel situation. In deep reinforcement learning (RL), the dominant paradigm is for an agent to amortise information that helps decision making into its network weights via gradient descent on training losses. Here, we pursue an alternative approach in which agents can utilise large-scale context sensitive database lookups to support their parametric computations. This allows agents to directly learn in an end-to-end manner to utilise relevant information to inform their outputs. In addition, new information can be attended to by the agent, without retraining, by simply augmenting the retrieval dataset. We study this approach for offline RL in 9x9 Go, a challenging game for which the vast combinatorial state space privileges generalisation over direct matching to past experiences. We leverage fast, approximate nearest neighbor techniques in order to retrieve relevant data from a set of tens of millions of expert demonstration states. Attending to this information provides a significant boost to prediction accuracy and game-play performance over simply using these demonstrations as training trajectories, providing a compelling demonstration of the value of large-scale retrieval in offline RL agents.

Key facts. Cited by 24 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 27 of its 55 references that are in the map. First posted on arXiv 10 Jun 2022.

Builds on

27 of 55 references.

Closest ancestors · 3

Same direction, newest first

  1. 2021Online and Offline Reinforcement Learning by Planning with a Learned Model13 Apr
  2. 2020DeepAveragers: Offline Reinforcement Learning by Solving Derived Non-Parametric MDPs18 Oct
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May

Other references · 24

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2015Trust Region Policy Optimizationcited by 2,580
  3. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  4. 2016Identity Mappings in Deep Residual Networkscited by 1,965
  5. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  6. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  7. 2022A Generalist Agentcited by 808
  8. 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 618
  9. 2019Causal Confusion in Imitation Learningcited by 288
  10. 2019Stabilizing Transformers for Reinforcement Learningcited by 270
  11. 2017Neural Episodic Controlcited by 151
  12. 2016Model-Free Episodic Controlcited by 102
12 more
  1. 2021VLM: Task-agnostic Video-Language Model Pre-training for Video Understandingcited by 102
  2. 2018Playing hard exploration games by watching YouTubecited by 99
  3. 2016Control of Memory, Active Perception, and Action in Minecraftcited by 61
  4. 2018Unsupervised Predictive Memory in a Goal-Directed Agentcited by 55
  5. 2022Retrieval-Augmented Reinforcement Learningcited by 53
  6. 2018Episodic Memory Deep Q-Networkscited by 42
  7. 2021Towards mental time travel: a hierarchical memory for reinforcement learning agentscited by 40
  8. 2018Memory-based Parameter Adaptationcited by 35
  9. 2018Been There, Done That: Meta-Learning with Episodic Recallcited by 26
  10. 2020Rapid Task-Solving in Novel Environmentscited by 23
  11. 2014Practical Kernel-Based Reinforcement Learningcited by 14
  12. 2018Fast deep reinforcement learning using online adjustments from the pastcited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 10; 2024: 4; 2025: 4; 2026: 4.

24 papers.

Sort:

By area:

  1. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 876
  2. 2023Policy Regularization with Dataset Constraint for Offline Reinforcement Learningcited by 34
  3. 2024Retrieval-Augmented Decision Transformer: External Memory for In-context RLcited by 21
  4. 2025Latent learning: episodic memory complements parametric learning by enabling flexible reuse of experiencescited by 15
  5. 2024Retrieval-Guided Reinforcement Learning for Boolean Circuit Minimizationcited by 10
  6. 2023Offline Reinforcement Learning with Imbalanced Datasetscited by 6
  7. 2024Memory-Enhanced Neural Solvers for Routing Problemscited by 4
  8. 2023Learning, Fast and Slow: A Goal-Directed Memory-Based Approach for Dynamic Environmentscited by 3
  9. 2025Universal Retrieval for Multimodal Trajectory Modelingcited by 3
  10. 2023Using Memory-Based Learning to Solve Tasks with State-Action Constraintscited by 2
  11. 2023K-Nearest-Neighbor Resampling for Off-Policy Evaluation in Stochastic Controlcited by 2
  12. 2025RANa: Retrieval-Augmented Navigationcited by 2
12 more
  1. 2022ALCAP: Alignment-Augmented Music Captionercited by 1
  2. 2022Contrastive Distillation Is a Sample-Efficient Self-Supervised Loss Policy for Transfer Learningcited by 1
  3. 2023Heterogeneous Knowledge for Augmented Modular Reinforcement Learningcited by 1
  4. 2024Enhancing Sample Efficiency and Exploration in Reinforcement Learning through the Integration of Diffusion Models and Proximal Policy Optimizationcited by 1
  5. 2023Linear Alignment of Vision-language Models for Image Captioningcited by 0
  6. 2023Heterogenous Memory Augmented Neural Networkscited by 0
  7. 2023Augmenting Unsupervised Reinforcement Learning with Self-Referencecited by 0
  8. 2025Generalizable Agent Modeling for Agent Collaboration-Competition Adaptation with Multi-Retrieval and Dynamic Generationcited by 0
  9. 2026Tactile Memory with Soft Robot: Robust Object Insertion via Masked Encoding and Soft Wristcited by 0
  10. 2026Textual Planning with Explicit Latent Transitionscited by 0
  11. 2026ASH: Agents that Self-Hone in Long-Horizon Worldscited by 0
  12. 2026Vector Search As Nearest Neighbor Matching: RAG-based Policy Learning in Causal Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025RAD: Retrieval High-quality Demonstrations to Enhance Decision-making
  2. 2024REGENT: A Retrieval-Augmented Generalist Agent That Can Act In-Context in New Environments
  3. 2025Re:Frame -- Retrieving Experience From Associative Memory
  4. 2023Using Offline Data to Speed Up Reinforcement Learning in Procedurally Generated Environments
  5. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems
  6. 2026Beyond Reward Engineering: A Data Recipe for Long-Context Reinforcement Learning
  7. 2024Efficient Exploration in Deep Reinforcement Learning: A Novel Bayesian Actor-Critic Algorithm
  8. 2025DeepSearch: Overcome the Bottleneck of Reinforcement Learning with Verifiable Rewards via Monte Carlo Tree Search