mlsanityData to 7 Oct 2026

DINOBot: Robot Manipulation via Retrieval and Alignment with Vision Foundation Models

arXiv 2402.13181 ↗First posted 20 Feb 2024cs.RO cs.LGcited by 43

On the map: Offline RL and imitation learning from data → Robot manipulation imitation learning

Abstract (arXiv)

We propose DINOBot, a novel imitation learning framework for robot manipulation, which leverages the image-level and pixel-level capabilities of features extracted from Vision Transformers trained with DINO. When interacting with a novel object, DINOBot first uses these features to retrieve the most visually similar object experienced during human demonstrations, and then uses this object to align its end-effector with the novel object to enable effective interaction. Through a series of real-world experiments on everyday tasks, we show that exploiting both the image-level and pixel-level properties of vision foundation models enables unprecedented learning efficiency and generalisation. Videos and code are available at https://www.robot-learning.uk/dinobot.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 36 of its 47 references that are in the map. First posted on arXiv 20 Feb 2024.

Builds on

36 of 47 references.

Closest ancestors · 10

Same direction, newest first

  1. 2023On the Effectiveness of Retrieval, Alignment, and Replay in Manipulation19 Dec
  2. 2023Few-Shot In-Context Imitation Learning via Implicit Graph Alignment18 Oct
  3. 2023One-Shot Imitation Learning: A Pose Estimation Perspective18 Oct
  4. 2023Behavior Retrieval: Few-Shot Imitation Learning by Querying Unlabeled Datasets18 Apr
  5. 2023Teach a Robot to FISH: Versatile Imitation from One Minute of Demonstrations2 Mar
  6. 2022Demonstrate Once, Imitate Immediately (DOME): Learning Visual Servoing for One-Shot Imitation Learning6 Apr
  7. 2021The Surprising Effectiveness of Representation Learning for Visual Imitation2 Dec
  8. 2021Learning Multi-Stage Tasks with One Demonstration via Self-Replay14 Nov
  9. 2021Coarse-to-Fine Imitation Learning: Robot Manipulation from a Single Demonstration13 May
  10. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning25 Oct

Other references · 26

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  5. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  6. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,385
  7. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  8. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  9. 2021Deep ViT Features as Dense Visual Descriptorscited by 327
  10. 2023TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinementcited by 281
  11. 2022Real-World Robot Learning with Masked Visual Pre-trainingcited by 241
  12. 2022Masked World Models for Visual Controlcited by 169
14 more
  1. 2019kPAM: KeyPoint Affordances for Category-Level Robotic Manipulationcited by 152
  2. 2018Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulationcited by 140
  3. 2021RRL: Resnet as representation for Reinforcement Learningcited by 89
  4. 2023RoboTAP: Tracking Arbitrary Points for Few-Shot Visual Imitationcited by 88
  5. 2024Robo-ABC: Affordance Generalization Beyond Categories via Semantic Correspondence for Robot Manipulationcited by 69
  6. 2022SE(3)-Equivariant Relational Rearrangement with Neural Descriptor Fieldscited by 51
  7. 2018Q-learning with Nearest Neighborscited by 37
  8. 2023D3Fields: Dynamic 3D Descriptor Fields for Zero-Shot Generalizable Rearrangementcited by 35
  9. 2022One-Shot Transfer of Affordance Regions? AffCorrs!cited by 31
  10. 2020S3K: Self-Supervised Semantic Keypoints for Robotic Manipulation via Multi-View Consistencycited by 28
  11. 2023SparseDFF: Sparse-View Feature Distillation for One-Shot Dexterous Manipulationcited by 22
  12. 2020FlowControl: Optical Flow Based Visual Servoingcited by 15
  13. 2023You Only Look at One: Category-Level Object Representations for Pose Estimation From a Single Examplecited by 8
  14. 2022Conditional Visual Servoing for Multi-Step Taskscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 15; 2025: 16; 2026: 12.

43 papers.

Sort:

By area:

  1. 2024ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulationcited by 305
  2. 2024Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulationcited by 142
  3. 2024RAM: Retrieval-Based Affordance Transfer for Generalizable Zero-Shot Robotic Manipulationcited by 55
  4. 2024Vision-based Manipulation from Single Human Video with Open-World Object Graphscited by 46
  5. 2024STRAP: Robot Sub-Trajectory Retrieval for Augmented Policy Learningcited by 42
  6. 2024R+X: Retrieval and Execution from Everyday Human Videoscited by 36
  7. 2024Learning Precise Affordances from Egocentric Videos for Robotic Manipulationcited by 34
  8. 2024FlowRetrieval: Flow-Guided Data Retrieval for Few-Shot Imitation Learningcited by 32
  9. 2025ZeroMimic: Distilling Robotic Manipulation Skills from Web Videoscited by 31
  10. 2024DITTO: Demonstration Imitation by Trajectory Transformationcited by 28
  11. 2024GenDP: 3D Semantic Fields for Category-Level Generalizable Diffusion Policycited by 25
  12. 2025SKIL: Semantic Keypoint Imitation Learning for Generalizable Data-efficient Manipulationcited by 21
31 more
  1. 2025Robust Finetuning of Vision-Language-Action Robot Policies via Parameter Mergingcited by 19
  2. 2024GLOVER: Generalizable Open-Vocabulary Affordance Reasoning for Task-Oriented Graspingcited by 15
  3. 2025GLOVER++: Unleashing the Potential of Affordance Learning from Human Behaviors for Robotic Manipulationcited by 14
  4. 2025MimicFunc: Imitating Tool Manipulation from a Single Human Video via Functional Correspondencecited by 11
  5. 2025Learning Interactive World Model for Object-Centric Reinforcement Learningcited by 11
  6. 2024Learning from 10 Demos: Generalisable and Sample-Efficient Policy Learning with Oriented Affordance Framescited by 9
  7. 2025FUNCTO: Function-Centric One-Shot Imitation Learning for Tool Manipulationcited by 8
  8. 2025CodeDiffuser: Attention-Enhanced Diffusion Policy via VLM-Generated Code for Instruction Ambiguitycited by 8
  9. 2026Uni-Skill: Building Self-Evolving Skill Repository for Generalizable Robotic Manipulationcited by 8
  10. 2025One-Shot Dual-Arm Imitation Learningcited by 7
  11. 2025COLLAGE: Adaptive Fusion-based Retrieval for Augmented Policy Learningcited by 5
  12. 2026DeFM: Learning Foundation Representations from Depth for Roboticscited by 3
  13. 2025Weakly-Supervised Learning of Dense Functional Correspondencescited by 2
  14. 2025Observer-Actor: Active Vision Imitation Learning with Sparse-View Gaussian Splattingcited by 2
  15. 2024Enabling robots to follow abstract instructions and complete complex dynamic taskscited by 1
  16. 2024ReMoBot: Retrieval-Based Few-Shot Imitation Learning for Mobile Manipulation with Vision Foundation Modelscited by 1
  17. 2025Large Pre-Trained Models for Bimanual Manipulation in 3Dcited by 1
  18. 2026Instant-Fold: In-Context Imitation Learning for Deformable Object Manipulationcited by 1
  19. 2026ReGIL: Retrieval-Guided Imitation Learning from a Single Demonstrationcited by 1
  20. 2024Semi-Supervised Neural Processes for Articulated Object Interactionscited by 0
  21. 2025GIFT: Geometry-Induced Functional Transfer for Category-level Object Manipulationcited by 0
  22. 2025RoboEye: Enhancing 2D Robotic Object Identification with Selective 3D Geometric Keypoint Matchingcited by 0
  23. 2025Annotation-Free One-Shot Imitation Learning for Multi-Step Manipulation Taskscited by 0
  24. 2026See and Switch: Vision-Based Branching for Interactive Robot-Skill Programmingcited by 0
  25. 2026Easy-IIL: Reducing Human Operational Burden in Interactive Imitation Learning via Assistant Expertscited by 0
  26. 2026Spatially Conditioned Diffusion Policy: Learning Precise and Robust Manipulation with a Single RGB Cameracited by 0
  27. 2026Retrieve in Time, Correct in Frequencycited by 0
  28. 2026RoboTok: A Scalable Data Engine for Internet Demonstration Video Retrieval and Dexterous Manipulation Learningcited by 0
  29. 2026Beyond End-Task Success: How to Audit Visual Experience Retrieval in Roboticscited by 0
  30. 2026POIL: Point-based One-Shot Imitation Learning with Stable Dynamical Systemscited by 0
  31. 2026Encore: Few-Shot Agentic Discovery of Manipulation Strategiescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Robotic Manipulation via Imitation Learning: Taxonomy, Evolution, Benchmark, and Challenges
  2. 2026Human-to-Robot Interaction: Learning from Video Demonstration for Robot Imitation
  3. 2021Towards More Generalizable One-shot Visual Imitation Learning
  4. 2025RoboManipBaselines: A Unified Framework for Imitation Learning in Robotic Manipulation across Real and Simulation Environments
  5. 2021DexMV: Imitation Learning for Dexterous Manipulation from Human Videos
  6. 2026SparTa: Sparse Graphical Task Models from a Handful of Demonstrations
  7. 2025Knowledge-Driven Imitation Learning: Enabling Generalization Across Diverse Conditions
  8. 2025Interactive Imitation Learning for Dexterous Robotic Manipulation: Challenges and Perspectives -- A Survey