mlsanityData to 7 Oct 2026

Search on the Replay Buffer: Bridging Planning and Reinforcement Learning

arXiv 1906.05253 ↗First posted 12 Jun 2019cs.AI cs.LG cs.ROcited by 188

Not on the map. Why?

Cited by 28 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The history of learning for control has been an exciting back and forth between two broad classes of algorithms: planning and reinforcement learning. Planning algorithms effectively reason over long horizons, but assume access to a local policy and distance metric over collision-free paths. Reinforcement learning excels at learning policies and the relative values of states, but fails to plan over long horizons. Despite the successes of each method in various domains, tasks that require reasoning over long horizons with limited feedback and high-dimensional observations remain exceedingly challenging for both planning and reinforcement learning algorithms. Frustratingly, these sorts of tasks are potentially the most useful, as they are simple to design (a human only need to provide an example goal state) and avoid reward shaping, which can bias the agent towards finding a sub-optimal solution. We introduce a general control algorithm that combines the strengths of planning and reinforcement learning to effectively solve these tasks. Our aim is to decompose the task of reaching a distant goal state into a sequence of easier tasks, each of which corresponds to reaching a subgoal. Planning algorithms can automatically find these waypoints, but only if provided with suitable abstractions of the environment -- namely, a graph consisting of nodes and edges. Our main insight is that this graph can be constructed via reinforcement learning, where a goal-conditioned value function provides edge weights, and nodes are taken to be previously seen observations in a replay buffer. Using graph search over our replay buffer, we can automatically generate this sequence of subgoals, even in image-based environments. Our algorithm, search on the replay buffer (SoRB), enables agents to solve sparse reward tasks over one hundred steps, and generalizes substantially better than standard RL algorithms.

Key facts. Cited by 188 papers in the map (data through 7 Oct 2026). First posted on arXiv 12 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 36; 2022: 30; 2023: 35; 2024: 24; 2025: 30; 2026: 32.

188 papers.

Sort:

By area:

  1. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  2. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  3. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  4. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
  5. 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 216
  6. 2021RvS: What is Essential for Offline RL via Supervised Learning?cited by 184
  7. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 169
  8. 2023Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planningcited by 161
  9. 2022Goal-Conditioned Reinforcement Learning: Problems and Solutionscited by 145
  10. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 133
  11. 2023Large Language Models for Robotics: A Surveycited by 132
  12. 2021Discovering and Achieving Goals via World Modelscited by 131
38 more
  1. 2021Goal-Conditioned Reinforcement Learning with Imagined Subgoalscited by 129
  2. 2021Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skillscited by 125
  3. 2021Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotationcited by 117
  4. 2025Reward Shaping to Mitigate Reward Hacking in RLHFcited by 88
  5. 2020COG: Connecting New Skills to Past Experience with Offline Reinforcement Learningcited by 83
  6. 2023METRA: Scalable Unsupervised RL with Metric-Aware Abstractioncited by 80
  7. 2021Augmenting Reinforcement Learning with Behavior Primitives for Diverse Manipulation Taskscited by 75
  8. 2024Mobility VLA: Multimodal Instruction Navigation with Long-Context VLMs and Topological Graphscited by 72
  9. 2021A Survey on Interpretable Reinforcement Learningcited by 70
  10. 2024ReMEmbR: Building and Reasoning Over Long-Horizon Spatio-Temporal Memory for Robot Navigationcited by 67
  11. 2023Grounded Decoding: Guiding Text Generation with Grounded Models for Embodied Agentscited by 66
  12. 2021Compositional Reinforcement Learning from Logical Specificationscited by 65
  13. 2022A Survey on Model-based Reinforcement Learningcited by 65
  14. 2021Memory-Augmented Reinforcement Learning for Image-Goal Navigationcited by 64
  15. 2021No RL, No Simulation: Learning to Navigate without Navigatingcited by 63
  16. 2025Horizon Reduction Makes RL Scalablecited by 55
  17. 2022Retrieval-Augmented Reinforcement Learningcited by 53
  18. 2024LoTa-Bench: Benchmarking Language-oriented Task Planners for Embodied Agentscited by 51
  19. 2022Hierarchical Planning Through Goal-Conditioned Offline Reinforcement Learningcited by 48
  20. 2023Tree-Planner: Efficient Close-loop Task Planning with Large Language Modelscited by 48
  21. 2021Landmark-Guided Subgoal Generation in Hierarchical Reinforcement Learningcited by 46
  22. 2022Contextualize Me -- The Case for Context in Reinforcement Learningcited by 41
  23. 2023Decomposing the Enigma: Subgoal-based Demonstration Learning for Formal Theorem Provingcited by 37
  24. 2023Contrastive Difference Predictive Codingcited by 35
  25. 2022Planning to Practice: Efficient Online Fine-Tuning by Composing Goals in Latent Spacecited by 31
  26. 2021Model-Based Reinforcement Learning via Latent-Space Collocationcited by 27
  27. 2021TempoRL: Learning When to Actcited by 25
  28. 2021Value Function Spaces: Skill-Centric State Abstractions for Long-Horizon Reasoningcited by 25
  29. 2021Differentiable Spatial Planning using Transformerscited by 25
  30. 2021The Logical Options Frameworkcited by 24
  31. 2022Generalization with Lossy Affordances: Leveraging Broad Offline Data for Learning Visuomotor Taskscited by 24
  32. 2021Successor Feature Landmarks for Long-Horizon Goal-Conditioned Reinforcement Learningcited by 23
  33. 2023Universal Visual Decomposer: Long-Horizon Manipulation Made Easycited by 23
  34. 2025Option-aware Temporally Abstracted Value for Offline Goal-Conditioned Reinforcement Learningcited by 22
  35. 2022Generating Executable Action Plans with Environmentally-Aware Language Modelscited by 21
  36. 2024Stitching Sub-Trajectories with Conditional Diffusion Model for Goal-Conditioned Offline RLcited by 20
  37. 2024A Single Goal is All You Need: Skills and Exploration Emerge from Contrastive RL without Rewards, Demonstrations, or Subgoalscited by 20
  38. 2021Solving Compositional Reinforcement Learning Problems via Task Reductioncited by 19

50 most cited of 188 All 188 (JSON)