# Offline Imitation Learning Through Graph Search and Retrieval

arXiv: https://arxiv.org/abs/2407.15403 · HTML: https://mlsanity.com/paper/2407.15403/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 41 of its 50 references that are in the map.
- First posted on arXiv 22 Jul 2024.

## Placed in

- Offline RL and imitation learning from data / [Robot manipulation imitation learning](https://mlsanity.com/offlinerl/robot-manipulation-imitation-learning/)

## Builds on (most cited first, 30 of 41)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware](https://mlsanity.com/paper/2304.13705/) (2023)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [Offline Reinforcement Learning with Implicit Q-Learning](https://mlsanity.com/paper/2110.06169/) (2021)
- [A Minimalist Approach to Offline Reinforcement Learning](https://mlsanity.com/paper/2106.06860/) (2021)
- [Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction](https://mlsanity.com/paper/1906.00949/) (2019)
- [Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning](https://mlsanity.com/paper/1910.00177/) (2019)
- [What Matters in Learning from Offline Human Demonstrations for Robot Manipulation](https://mlsanity.com/paper/2108.03298/) (2021)
- [R3M: A Universal Visual Representation for Robot Manipulation](https://mlsanity.com/paper/2203.12601/) (2022)
- [AWAC: Accelerating Online Reinforcement Learning with Offline Datasets](https://mlsanity.com/paper/2006.09359/) (2020)
- [Behavior Regularized Offline Reinforcement Learning](https://mlsanity.com/paper/1911.11361/) (2019)
- [Offline Reinforcement Learning as One Big Sequence Modeling Problem](https://mlsanity.com/paper/2106.02039/) (2021)
- [Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation](https://mlsanity.com/paper/2209.05451/) (2022)
- [Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations](https://mlsanity.com/paper/1904.06387/) (2019)
- [Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?](https://mlsanity.com/paper/2303.18240/) (2023)
- [Semi-parametric Topological Memory for Navigation](https://mlsanity.com/paper/1803.00653/) (2018)
- [Search on the Replay Buffer: Bridging Planning and Reinforcement Learning](https://mlsanity.com/paper/1906.05253/) (2019)
- [GELLO: A General, Low-Cost, and Intuitive Teleoperation Framework for Robot Manipulators](https://mlsanity.com/paper/2309.13037/) (2023)
- [The Surprising Effectiveness of Representation Learning for Visual Imitation](https://mlsanity.com/paper/2112.01511/) (2021)
- [FurnitureBench: Reproducible Real-World Benchmark for Long-Horizon Complex Manipulation](https://mlsanity.com/paper/2305.12821/) (2023)
- [Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrations](https://mlsanity.com/paper/2207.10050/) (2022)
- [Few-Shot Preference Learning for Human-in-the-Loop RL](https://mlsanity.com/paper/2212.03363/) (2022)

## Built on by (most cited first, 14 of 14)

- [STRAP: Robot Sub-Trajectory Retrieval for Augmented Policy Learning](https://mlsanity.com/paper/2412.15182/) (2024)
- [Object-centric 3D Motion Field for Robot Learning from Human Videos](https://mlsanity.com/paper/2506.04227/) (2025)
- [Test-Time Graph Search for Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2510.07257/) (2025)
- [COLLAGE: Adaptive Fusion-based Retrieval for Augmented Policy Learning](https://mlsanity.com/paper/2508.01131/) (2025)
- [LLM-Guided Reinforcement Learning: Addressing Training Bottlenecks through Policy Modulation](https://mlsanity.com/paper/2505.20671/) (2025)
- [ReMoBot: Retrieval-Based Few-Shot Imitation Learning for Mobile Manipulation with Vision Foundation Models](https://mlsanity.com/paper/2408.15919/) (2024)
- [Robust Offline Imitation Learning Through State-level Trajectory Stitching](https://mlsanity.com/paper/2503.22524/) (2025)
- [ReGIL: Retrieval-Guided Imitation Learning from a Single Demonstration](https://mlsanity.com/paper/2606.09381/) (2026)
- [Retrieve in Time, Correct in Frequency](https://mlsanity.com/paper/2608.04527/) (2026)
- [TReF-6: Inferring Task-Relevant Frames from a Single Demonstration for One-Shot Skill Generalization](https://mlsanity.com/paper/2509.00310/) (2025)
- [Mana: Dexterous Manipulation of Articulated Tools](https://mlsanity.com/paper/2606.13677/) (2026)
- [MinInter: Minimizing Trajectory Interpolation During Data Augmentation for Imitation Learning](https://mlsanity.com/paper/2606.24078/) (2026)
- [NEEDLEWORK: Offline Rewriting of Robot Data with Verified Local Stitches](https://mlsanity.com/paper/2610.02339/) (2026)
- [Ultrasound Operator Guidance Using World Modeling and Retrieval Based Action Planning](https://mlsanity.com/paper/2610.06008/) (2026)

## Abstract (arXiv)

Imitation learning is a powerful machine learning algorithm for a robot to acquire manipulation skills. Nevertheless, many real-world manipulation tasks involve precise and dexterous robot-object interactions, which make it difficult for humans to collect high-quality expert demonstrations. As a result, a robot has to learn skills from suboptimal demonstrations and unstructured interactions, which remains a key challenge. Existing works typically use offline deep reinforcement learning (RL) to solve this challenge, but in practice these algorithms are unstable and fragile due to the deadly triad issue. To overcome this problem, we propose GSR, a simple yet effective algorithm that learns from suboptimal demonstrations through Graph Search and Retrieval. We first use pretrained representation to organize the interaction experience into a graph and perform a graph search to calculate the values of different behaviors. Then, we apply a retrieval-based procedure to identify the best behavior (actions) on each state and use behavior cloning to learn that behavior. We evaluate our method in both simulation and real-world robotic manipulation tasks with complex visual inputs, covering various precise and dexterous manipulation skills with objects of different physical properties. GSR can achieve a 10% to 30% higher success rate and over 30% higher proficiency compared to baselines. Our project page is at https://zhaohengyin.github.io/gsr.