mlsanityData to 14 Oct 2026

Offline Imitation Learning Through Graph Search and Retrieval

arXiv 2407.15403 ↗First posted 22 Jul 2024cs.RO cs.AI cs.LGcited by 14

On the map: Offline RL and imitation learning from data → Robot manipulation imitation learning

Abstract (arXiv)

Imitation learning is a powerful machine learning algorithm for a robot to acquire manipulation skills. Nevertheless, many real-world manipulation tasks involve precise and dexterous robot-object interactions, which make it difficult for humans to collect high-quality expert demonstrations. As a result, a robot has to learn skills from suboptimal demonstrations and unstructured interactions, which remains a key challenge. Existing works typically use offline deep reinforcement learning (RL) to solve this challenge, but in practice these algorithms are unstable and fragile due to the deadly triad issue. To overcome this problem, we propose GSR, a simple yet effective algorithm that learns from suboptimal demonstrations through Graph Search and Retrieval. We first use pretrained representation to organize the interaction experience into a graph and perform a graph search to calculate the values of different behaviors. Then, we apply a retrieval-based procedure to identify the best behavior (actions) on each state and use behavior cloning to learn that behavior. We evaluate our method in both simulation and real-world robotic manipulation tasks with complex visual inputs, covering various precise and dexterous manipulation skills with objects of different physical properties. GSR can achieve a 10% to 30% higher success rate and over 30% higher proficiency compared to baselines. Our project page is at https://zhaohengyin.github.io/gsr.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 41 of its 50 references that are in the map. First posted on arXiv 22 Jul 2024.

Builds on

41 of 50 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023On the Effectiveness of Retrieval, Alignment, and Replay in Manipulation19 Dec
  2. 2023Learning to Discern: Imitating Heterogeneous Human Demonstrations with Preference and Representation Learning22 Oct
  3. 2023GELLO: A General, Low-Cost, and Intuitive Teleoperation Framework for Robot Manipulators22 Sep
  4. 2023FurnitureBench: Reproducible Real-World Benchmark for Long-Horizon Complex Manipulation22 May
  5. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware23 Apr
  6. 2023Behavior Retrieval: Few-Shot Imitation Learning by Querying Unlabeled Datasets18 Apr
  7. 2021The Surprising Effectiveness of Representation Learning for Visual Imitation2 Dec

Other references · 34

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2020Denoising Diffusion Implicit Modelscited by 9,781
  4. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  5. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  6. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
  7. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  8. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  9. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  10. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  11. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  12. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
22 more
  1. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  2. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  3. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 725
  4. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 694
  5. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  6. 2019Behavior Regularized Offline Reinforcement Learningcited by 662
  7. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
  8. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 551
  9. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 260
  10. 2023Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?cited by 218
  11. 2018Semi-parametric Topological Memory for Navigationcited by 209
  12. 2019Search on the Replay Buffer: Bridging Planning and Reinforcement Learningcited by 194
  13. 2022Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrationscited by 89
  14. 2022Few-Shot Preference Learning for Human-in-the-Loop RLcited by 87
  15. 2020Learning from Suboptimal Demonstration via Self-Supervised Reward Regressioncited by 84
  16. 2020World Model as a Graph: Learning Latent Landmarks for Planningcited by 69
  17. 2022Imitation Learning by Estimating Expertise of Demonstratorscited by 50
  18. 2021Breaking the Deadly Triad with a Target Networkcited by 47
  19. 2021Learning Multimodal Rewards from Rankingscited by 41
  20. 2020Sparse Graphical Memory for Robust Planningcited by 31
  21. 2022Discriminator-Guided Model-Based Offline Imitation Learningcited by 17
  22. 2022Value Memory Graph: A Graph-Structured World Model for Offline Reinforcement Learningcited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 6; 2026: 6.

14 papers.

Sort:

By area:

  1. 2024STRAP: Robot Sub-Trajectory Retrieval for Augmented Policy Learningcited by 44
  2. 2025Object-centric 3D Motion Field for Robot Learning from Human Videoscited by 14
  3. 2025Test-Time Graph Search for Goal-Conditioned Reinforcement Learningcited by 8
  4. 2025COLLAGE: Adaptive Fusion-based Retrieval for Augmented Policy Learningcited by 5
  5. 2025LLM-Guided Reinforcement Learning: Addressing Training Bottlenecks through Policy Modulationcited by 2
  6. 2024ReMoBot: Retrieval-Based Few-Shot Imitation Learning for Mobile Manipulation with Vision Foundation Modelscited by 1
  7. 2025Robust Offline Imitation Learning Through State-level Trajectory Stitchingcited by 1
  8. 2026ReGIL: Retrieval-Guided Imitation Learning from a Single Demonstrationcited by 1
  9. 2026Retrieve in Time, Correct in Frequencycited by 1
  10. 2025TReF-6: Inferring Task-Relevant Frames from a Single Demonstration for One-Shot Skill Generalizationcited by 0
  11. 2026Mana: Dexterous Manipulation of Articulated Toolscited by 0
  12. 2026MinInter: Minimizing Trajectory Interpolation During Data Augmentation for Imitation Learningcited by 0
2 more
  1. 2026NEEDLEWORK: Offline Rewriting of Robot Data with Verified Local Stitchescited by 0
  2. 2026Ultrasound Operator Guidance Using World Modeling and Retrieval Based Action Planningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2020Learning to Generalize Across Long-Horizon Tasks from Human Demonstrations
  2. 2022Learning and Retrieval from Prior Data for Skill-based Imitation Learning
  3. 2026Human-to-Robot Interaction: Learning from Video Demonstration for Robot Imitation
  4. 2026Difference-Aware Retrieval Policies for Imitation Learning
  5. 2021Efficient and Interpretable Robot Manipulation with Graph Neural Networks
  6. 2025Robotic Manipulation via Imitation Learning: Taxonomy, Evolution, Benchmark, and Challenges
  7. 2026Coherent Off-Policy Improvement of Large Behavior Models with Learned Rewards
  8. 2024Generalized Robot Learning Framework