mlsanityData to 7 Oct 2026

Vid2Robot: End-to-end Video-conditioned Policy Learning with Cross-Attention Transformers

arXiv 2403.12943 ↗First posted 19 Mar 2024cs.RO cs.AIcited by 47

Not on the map. Why?

Abstract (arXiv)

Large-scale multi-task robotic manipulation systems often rely on text to specify the task. In this work, we explore whether a robot can learn by observing humans. To do so, the robot must understand a person's intent and perform the inferred task despite differences in the embodiments and environments. We introduce Vid2Robot, an end-to-end video-conditioned policy that takes human videos demonstrating manipulation tasks as input and produces robot actions. Our model is trained with a large dataset of prompt video-robot trajectory pairs to learn unified representations of human and robot actions from videos. Vid2Robot uses cross-attention transformer layers between video features and the current robot state to produce the actions and perform the same task as shown in the video. We use auxiliary contrastive losses to align the prompt and robot video representations for better policies. We evaluate Vid2Robot on real-world robots and observe over 20% improvement over BC-Z when using human prompt videos. Further, we also show cross-object motion transfer ability that enables video-conditioned policies to transfer a motion observed on one object in the prompt video to another object in the robot's own environment. Videos available at https://vid2robot.github.io

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Builds on 47 of its 55 references that are in the map. First posted on arXiv 19 Mar 2024.

Builds on

47 of 55 references.

References · 47

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  3. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  4. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  5. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  6. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  7. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  8. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  9. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  10. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  11. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  12. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
35 more
  1. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
  2. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  3. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  4. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  5. 2023Learning Universal Policies via Text-Guided Video Generationcited by 506
  6. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  7. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 384
  8. 2018Visual Reinforcement Learning with Imagined Goalscited by 248
  9. 2022Masked Visual Pre-training for Motor Controlcited by 246
  10. 2017One-Shot Imitation Learningcited by 232
  11. 2023Affordances from Human Videos as a Versatile Representation for Roboticscited by 221
  12. 2023MimicPlay: Long-Horizon Imitation Learning by Watching Human Playcited by 214
  13. 2017Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translationcited by 167
  14. 2019Goal-conditioned Imitation Learningcited by 159
  15. 2022Human-to-Robot Imitation in the Wildcited by 157
  16. 2023Open-World Object Manipulation using Pre-trained Vision-Language Modelscited by 155
  17. 2019Temporal Cycle-Consistency Learningcited by 151
  18. 2023Structured World Models from Human Videoscited by 144
  19. 2021Goal-Conditioned Reinforcement Learning with Imagined Subgoalscited by 129
  20. 2023RoboCat: A Self-Improving Generalist Agent for Robotic Manipulationcited by 108
  21. 2019AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videoscited by 105
  22. 2023HomeRobot: Open-Vocabulary Mobile Manipulationcited by 104
  23. 2021XIRL: Cross-embodiment Inverse Reinforcement Learningcited by 101
  24. 2020Reinforcement Learning with Videos: Combining Offline Observations with Interactioncited by 80
  25. 2020Transformers for One-Shot Visual Imitationcited by 77
  26. 2023XSkill: Cross Embodiment Skill Discoverycited by 74
  27. 2019Third-Person Visual Imitation Learning via Decoupled Hierarchical Controllercited by 73
  28. 2023MUTEX: Learning Unified Policies from Multimodal Task Specificationscited by 70
  29. 2023Navigating to Objects Specified by Imagescited by 50
  30. 2021Towards More Generalizable One-shot Visual Imitation Learningcited by 48
  31. 2021Forecasting Action through Contact Representations from First Person Videocited by 46
  32. 2022Transformers are Adaptable Task Plannerscited by 15
  33. 2019Online Object Representations with Contrastive Learningcited by 13
  34. 2020Learning Object Manipulation Skills via Approximate State Estimation from Real Videoscited by 12
  35. 2020Goal-Conditioned End-to-End Visuomotor Control for Versatile Skill Primitivescited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 15; 2025: 16; 2026: 16.

47 papers.

Sort:

By area:

  1. 2024EgoMimic: Scaling Imitation Learning via Egocentric Videocited by 160
  2. 2024Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulationcited by 142
  3. 2024In-Context Imitation Learning via Next-Token Predictioncited by 65
  4. 2025Motion Tracks: A Unified Representation for Human-Robot Transfer in Few-Shot Imitation Learningcited by 57
  5. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  6. 2024RT-Affordance: Affordances are Versatile Intermediate Representations for Robot Manipulationcited by 40
  7. 2024Instant Policy: In-Context Imitation Learning via Graph Diffusioncited by 38
  8. 2024R+X: Retrieval and Execution from Everyday Human Videoscited by 36
  9. 2024A Survey of Embodied Learning for Object-Centric Robotic Manipulationcited by 34
  10. 2025UniSkill: Imitating Human Videos via Cross-Embodiment Skill Representationscited by 30
  11. 2025X-Sim: Cross-Embodiment Learning via Real-to-Sim-to-Realcited by 25
  12. 2024Mitigating the Human-Robot Domain Discrepancy in Visual Pre-training for Robotic Manipulationcited by 22
35 more
  1. 2025WristWorld: Generating Wrist-Views via 4D World Models for Robotic Manipulationcited by 21
  2. 2025Robot Learning from a Physical World Modelcited by 18
  3. 2025DemoDiffusion: One-Shot Human Imitation using pre-trained Diffusion Policycited by 16
  4. 2026HumanEgo: Zero-Shot Robot Learning from Minutes of Human Egocentric Videoscited by 14
  5. 2024Synthesizing Programmatic Reinforcement Learning Policies with Large Language Model Guided Searchcited by 13
  6. 2025ManualVLA: A Unified VLA Model for Chain-of-Thought Manual Generation and Robotic Manipulationcited by 13
  7. 2024One-Shot Imitation under Mismatched Executioncited by 11
  8. 2024VIEW: Visual Imitation Learning with Waypointscited by 10
  9. 2024GROOT-2: Weakly Supervised Multi-Modal Instruction Following Agentscited by 9
  10. 2026Dexterous Manipulation Policies from RGB Human Videos via 3D Hand-Object Trajectory Reconstructioncited by 8
  11. 2026Zero-WAM: In-Context World-Action Modeling from Human Videos for Open-Ended Task Generalizationcited by 8
  12. 2024Generative AI Agents in Autonomous Machines: A Safety Perspectivecited by 7
  13. 2025Imitation Learning from a Single Temporally Misaligned Videocited by 7
  14. 2025What Changed and What Could Have Changed? State-Change Counterfactuals for Procedure-Aware Video Representation Learningcited by 6
  15. 2025Chain-of-Modality: Learning Manipulation Programs from Multimodal Human Videos with Vision-Language-Modelscited by 6
  16. 2025X-Diffusion: Training Diffusion Policies on Cross-Embodiment Human Demonstrationscited by 6
  17. 2026WAM-TTT: Steering World-Action Models by Watching Human Play at Test Timecited by 6
  18. 2024HANDI: Hand-Centric Text-and-Image Conditioned Video Generationcited by 5
  19. 2025ViSA-Flow: Accelerating Robot Skill Learning via Large-Scale Video Semantic Action Flowcited by 5
  20. 2026Retrieve, Don't Retrain: Extending Vision Language Action Models to New Tasks at Test Timecited by 3
  21. 2025Slot-Level Robotic Placement via Visual Imitation from Single Human Videocited by 2
  22. 2025Trajectory Conditioned Cross-embodiment Skill Transfercited by 2
  23. 2025DRAW2ACT: Turning Depth-Encoded Trajectories into Robotic Demonstration Videoscited by 2
  24. 2026EasyMimic: A Low-Cost Framework for Robot Imitation Learning from Human Videoscited by 2
  25. 2026SeeTraceAct: Visibility-Aware Latent Planning from Cross-Embodiment Demonstration Videoscited by 2
  26. 2026In-Context World Modeling for Robotic Controlcited by 2
  27. 2025RoboSwap: A GAN-driven Video Diffusion Framework For Unsupervised Robot Arm Swappingcited by 1
  28. 2026RADAR: Closed-Loop Robotic Data Generation via Semantic Planning and Autonomous Causal Environment Resetcited by 1
  29. 2026Interact3D: Compositional 3D Generation of Interactive Objectscited by 1
  30. 2026SynthICL: Scalable In-context Imitation Learning with Synthetic Datacited by 0
  31. 2026The Imitator Game: Benchmarking Robot Imitative Ability Beyond Action Predictioncited by 0
  32. 2026PonderPounce: A Pretrained MLLM as an Episode Context Engine for Robot Controlcited by 0
  33. 2026Track, Articulate, Act: Generating Articulation from Casual Human Videoscited by 0
  34. 2026INSPECT: Learning Robot View Selection from Assistant Usecited by 0
  35. 2026KnowDemo: Knowledge-Guided Robot Demonstration Generation from Human Videoscited by 0