mlsanityData to 7 Oct 2026

R+X: Retrieval and Execution from Everyday Human Videos

arXiv 2407.12957 ↗First posted 17 Jul 2024cs.RO cs.LGcited by 36

Not on the map. Why?

Abstract (arXiv)

We present R+X, a framework which enables robots to learn skills from long, unlabelled, first-person videos of humans performing everyday tasks. Given a language command from a human, R+X first retrieves short video clips containing relevant behaviour, and then executes the skill by conditioning an in-context imitation learning method (KAT) on this behaviour. By leveraging a Vision Language Model (VLM) for retrieval, R+X does not require any manual annotation of the videos, and by leveraging in-context learning for execution, robots can perform commanded skills immediately, without requiring a period of training on the retrieved videos. Experiments studying a range of everyday household tasks show that R+X succeeds at translating unlabelled human videos into robust robot skills, and that R+X outperforms several recent alternative methods. Videos and code are available at https://www.robot-learning.uk/r-plus-x.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Builds on 45 of its 56 references that are in the map. First posted on arXiv 17 Jul 2024.

Builds on

45 of 56 references.

References · 45

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2023GPT-4 Technical Reportcited by 18,132
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  7. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  8. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  9. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  10. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  11. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  12. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
33 more
  1. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  2. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  3. 2015ORB-SLAM: a Versatile and Accurate Monocular SLAM Systemcited by 983
  4. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  5. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
  6. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  7. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  8. 2021Image Segmentation Using Text and Image Promptscited by 509
  9. 2023Gaussian Splatting SLAMcited by 410
  10. 2021Deep ViT Features as Dense Visual Descriptorscited by 327
  11. 2023Reconstructing Hands in 3D with Transformerscited by 298
  12. 2023TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinementcited by 281
  13. 2023Affordances from Human Videos as a Versatile Representation for Roboticscited by 221
  14. 2023MimicPlay: Long-Horizon Imitation Learning by Watching Human Playcited by 214
  15. 2023Large Language Models as General Pattern Machinescited by 196
  16. 2022Embodied Hands: Modeling and Capturing Hands and Bodies Togethercited by 179
  17. 2022Human-to-Robot Imitation in the Wildcited by 157
  18. 2024DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulationcited by 149
  19. 2021Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videoscited by 116
  20. 2024Track2Act: Predicting Point Tracks from Internet Videos enables Generalizable Robot Manipulationcited by 116
  21. 2024SERL: A Software Suite for Sample-Efficient Robotic Reinforcement Learningcited by 113
  22. 2021Coarse-to-Fine Imitation Learning: Robot Manipulation from a Single Demonstrationcited by 92
  23. 2024General Flow as Foundation Affordance for Scalable Robot Learningcited by 80
  24. 2024Keypoint Action Tokens Enable In-Context Imitation Learning in Roboticscited by 71
  25. 2023Language Models as Zero-Shot Trajectory Generatorscited by 59
  26. 2023Behavior Retrieval: Few-Shot Imitation Learning by Querying Unlabeled Datasetscited by 56
  27. 2024Vid2Robot: End-to-end Video-conditioned Policy Learning with Cross-Attention Transformerscited by 47
  28. 2024Vision-based Manipulation from Single Human Video with Open-World Object Graphscited by 46
  29. 2024DINOBot: Robot Manipulation via Retrieval and Alignment with Vision Foundation Modelscited by 43
  30. 2024Retrieval-Augmented Embodied Agentscited by 30
  31. 2024DITTO: Demonstration Imitation by Trajectory Transformationcited by 28
  32. 2024ScrewMimic: Bimanual Imitation from Human Videos with Screw Space Projectioncited by 16
  33. 2023On the Effectiveness of Retrieval, Alignment, and Replay in Manipulationcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 17; 2026: 13.

36 papers.

Sort:

By area:

  1. 2024EgoMimic: Scaling Imitation Learning via Egocentric Videocited by 160
  2. 2024Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulationcited by 142
  3. 2025Motion Tracks: A Unified Representation for Human-Robot Transfer in Few-Shot Imitation Learningcited by 57
  4. 2025VidBot: Learning Generalizable 3D Actions from In-the-Wild 2D Human Videos for Zero-Shot Robotic Manipulationcited by 46
  5. 2024STRAP: Robot Sub-Trajectory Retrieval for Augmented Policy Learningcited by 42
  6. 2024Instant Policy: In-Context Imitation Learning via Graph Diffusioncited by 38
  7. 2025ZeroMimic: Distilling Robotic Manipulation Skills from Web Videoscited by 31
  8. 2025BEHAVIOR Robot Suite: Streamlining Real-World Whole-Body Manipulation for Everyday Household Activitiescited by 29
  9. 2025EgoZero: Robot Learning from Smart Glassescited by 28
  10. 2025SPIDER: Scalable Physics-Informed Dexterous Retargetingcited by 25
  11. 2025RICL: Adding In-Context Adaptability to Pre-Trained Vision-Language-Action Modelscited by 19
  12. 2025Robot Learning from a Physical World Modelcited by 18
24 more
  1. 2025AirExo-2: Scaling up Generalizable Robotic Imitation Learning with Low-Cost Exoskeletonscited by 17
  2. 2025DemoDiffusion: One-Shot Human Imitation using pre-trained Diffusion Policycited by 16
  3. 2024MILES: Making Imitation Learning Easy with Self-Supervisioncited by 9
  4. 2025Human2LocoMan: Learning Versatile Quadrupedal Manipulation with Human Pretrainingcited by 8
  5. 2024On-Robot Reinforcement Learning with Goal-Contrastive Rewardscited by 7
  6. 2025Chain-of-Modality: Learning Manipulation Programs from Multimodal Human Videos with Vision-Language-Modelscited by 6
  7. 2025COLLAGE: Adaptive Fusion-based Retrieval for Augmented Policy Learningcited by 5
  8. 2026SkillVLA: Tackling Combinatorial Diversity in Dual-Arm Manipulation via Skill Reusecited by 5
  9. 2025Scalable Strategies for Continual Learning with Replaycited by 3
  10. 2026Scaling Single Human Demonstrations for Imitation Learning using Generative Foundational Modelscited by 3
  11. 2026Retrieve, Don't Retrain: Extending Vision Language Action Models to New Tasks at Test Timecited by 3
  12. 2025Knowledge-Driven Imitation Learning: Enabling Generalization Across Diverse Conditionscited by 2
  13. 2026Robot Self-Improvement via Human-Video Dynamics Modelscited by 2
  14. 2026SiMDex: Mining Similar Egocentric Videos for Cross-Embodiment Dexterous Manipulationcited by 2
  15. 2026Imitating What Works: Simulation-Filtered Modular Policy Learning from Human Videoscited by 1
  16. 2025Imitation Learning with Precisely Labeled Human Demonstrationscited by 0
  17. 2025EgoLoc: A Generalizable Solution for Temporal Interaction Localization in Egocentric Videoscited by 0
  18. 2026Tactile Memory with Soft Robot: Robust Object Insertion via Masked Encoding and Soft Wristcited by 0
  19. 2026A Hierarchical Spatiotemporal Action Tokenizer for In-Context Imitation Learning in Roboticscited by 0
  20. 2026VLAff: Vision-Language-Affordance Model for Unified Actionable Affordancescited by 0
  21. 2026RoboTok: A Scalable Data Engine for Internet Demonstration Video Retrieval and Dexterous Manipulation Learningcited by 0
  22. 20263DWay: Generalizing Robot Manipulation via 3D Consistent Waypointscited by 0
  23. 2026Beyond End-Task Success: How to Audit Visual Experience Retrieval in Roboticscited by 0
  24. 2026InfiniHand: Streaming World-Space Hand Motion Estimation from Egocentric Videocited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Human-to-Robot Interaction: Learning from Video Demonstration for Robot Imitation
  2. 2026Monkey See, Can Monkey Do? A Benchmark for Evaluating Robot Skill Learning by Observation ↗
  3. 2025Emergence of Human to Robot Transfer in Vision-Language-Action Models
  4. 2024Video-Language Critic: Transferable Reward Functions for Language-Conditioned Robotics
  5. 2026HumanX: Toward Agile and Generalizable Humanoid Interaction Skills from Human Videos
  6. 2025Masquerade: Learning from In-the-wild Human Videos using Data-Editing
  7. 2023RH20T: A Comprehensive Robotic Dataset for Learning Diverse Skills in One-Shot
  8. 2023XSkill: Cross Embodiment Skill Discovery