mlsanityData to 7 Oct 2026

Motion Tracks: A Unified Representation for Human-Robot Transfer in Few-Shot Imitation Learning

arXiv 2501.06994 ↗First posted 13 Jan 2025cs.RO cs.AI cs.LGcited by 57

On the map: Offline RL and imitation learning from data → Robot manipulation imitation learning

Abstract (arXiv)

Teaching robots to autonomously complete everyday tasks remains a challenge. Imitation Learning (IL) is a powerful approach that imbues robots with skills via demonstrations, but is limited by the labor-intensive process of collecting teleoperated robot data. Human videos offer a scalable alternative, but it remains difficult to directly train IL policies from them due to the lack of robot action labels. To address this, we propose to represent actions as short-horizon 2D trajectories on an image. These actions, or motion tracks, capture the predicted direction of motion for either human hands or robot end-effectors. We instantiate an IL policy called Motion Track Policy (MT-pi) which receives image observations and outputs motion tracks as actions. By leveraging this unified, cross-embodiment action space, MT-pi completes tasks with high success given just minutes of human video and limited additional robot demonstrations. At test time, we predict motion tracks from two camera views, recovering 6DoF trajectories via multi-view synthesis. MT-pi achieves an average success rate of 86.5% across 4 real-world tasks, outperforming state-of-the-art IL baselines which do not leverage human data or our action space by 40%, and generalizes to scenarios seen only in human videos. Code and videos are available on our website https://portal-cornell.github.io/motion_track_policy/.

Key facts. Cited by 57 papers in the map (data through 7 Oct 2026). Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 58 of its 72 references that are in the map. First posted on arXiv 13 Jan 2025.

Builds on

58 of 72 references.

Closest ancestors · 15

Same direction, newest first

  1. 2024FlowRetrieval: Flow-Guided Data Retrieval for Few-Shot Imitation Learning29 Aug
  2. 2024Bunny-VisionPro: Real-Time Bimanual Dexterous Teleoperation for Imitation Learning3 Jul
  3. 2024EquiBot: SIM(3)-Equivariant Diffusion Policy for Generalizable and Data Efficient Learning1 Jul
  4. 2024VIEW: Visual Imitation Learning with Waypoints27 Apr
  5. 20243D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representations6 Mar
  6. 2024Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperation4 Jan
  7. 2023Any-point Trajectory Modeling for Policy Learning28 Dec
  8. 2023GELLO: A General, Low-Cost, and Intuitive Teleoperation Framework for Robot Manipulators22 Sep
  9. 2023XSkill: Cross Embodiment Skill Discovery19 Jul
  10. 2023Decomposing the Generalization Gap in Imitation Learning for Visual Robotic Manipulation7 Jul
  11. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware23 Apr
  12. 2023MimicPlay: Long-Horizon Imitation Learning by Watching Human Play24 Feb
3 more
  1. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training30 Sep
  2. 2021XIRL: Cross-embodiment Inverse Reinforcement Learning7 Jun
  3. 2019Third-Person Visual Imitation Learning via Decoupled Hierarchical Controller21 Nov

Other references · 43

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  3. 2024SAM 2: Segment Anything in Images and Videoscited by 2,784
  4. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  5. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  6. 2014Unsupervised Domain Adaptation by Backpropagationcited by 1,906
  7. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  8. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  9. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  10. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
  11. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  12. 2024DROID: A Large-Scale In-The-Wild Robot Manipulation Datasetcited by 801
31 more
  1. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  2. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  3. 2019MediaPipe: A Framework for Building Perception Pipelinescited by 572
  4. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  5. 2018Behavioral Cloning from Observationcited by 482
  6. 2024Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robotscited by 373
  7. 2023Humans in 4D: Reconstructing and Tracking Humans with Transformerscited by 344
  8. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 338
  9. 2023Reconstructing Hands in 3D with Transformerscited by 298
  10. 2017One-Shot Imitation Learningcited by 232
  11. 2023RVT: Robotic View Transformer for 3D Object Manipulationcited by 224
  12. 2023Language-Driven Representation Learning for Roboticscited by 163
  13. 2023Learning to Act from Actionless Videos through Dense Correspondencescited by 159
  14. 2023Open-World Object Manipulation using Pre-trained Vision-Language Modelscited by 155
  15. 2024Flow as the Cross-Domain Manipulation Interfacecited by 126
  16. 2019AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videoscited by 105
  17. 2023Language Embedded Radiance Fields for Zero-Shot Task-Oriented Graspingcited by 91
  18. 2023RoboTAP: Tracking Arbitrary Points for Few-Shot Visual Imitationcited by 88
  19. 2024Manipulate-Anything: Automating Real-World Robots using Vision-Language Modelscited by 82
  20. 2024General Flow as Foundation Affordance for Scalable Robot Learningcited by 80
  21. 2020Human Mesh Recovery from Multiple Shotscited by 52
  22. 2023Towards Generalizable Zero-Shot Manipulation via Translating Human Interaction Planscited by 51
  23. 2024Vid2Robot: End-to-end Video-conditioned Policy Learning with Cross-Attention Transformerscited by 47
  24. 2024Vision-based Manipulation from Single Human Video with Open-World Object Graphscited by 46
  25. 2023An Unbiased Look at Datasets for Visuo-Motor Pre-Trainingcited by 41
  26. 2023KITE: Keypoint-Conditioned Policies for Semantic Manipulationcited by 39
  27. 2023EquivAct: SIM(3)-Equivariant Visuomotor Policies beyond Rigid Object Manipulationcited by 37
  28. 2024R+X: Retrieval and Execution from Everyday Human Videoscited by 36
  29. 2023SparseDFF: Sparse-View Feature Distillation for One-Shot Dexterous Manipulationcited by 22
  30. 2024Language-Model-Assisted Bi-Level Programming for Reward Learning from Internet Videoscited by 3
  31. 2024Learning Wheelchair Tennis Navigation from Broadcast Videos with Domain Knowledge Transfer and Diffusion Motion Planningcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 34; 2026: 23.

57 papers.

Sort:

By area:

  1. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,167
  2. 2025EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Videocited by 191
  3. 2025Humanoid Policy ~ Human Policycited by 65
  4. 2026EgoVerse: An Egocentric Human Dataset for Robot Learning from Around the Worldcited by 49
  5. 2025Robotic Manipulation by Imitating Generated Videos Without Physical Demonstrationscited by 33
  6. 2025ZeroMimic: Distilling Robotic Manipulation Skills from Web Videoscited by 31
  7. 2025UniSkill: Imitating Human Videos via Cross-Embodiment Skill Representationscited by 30
  8. 2025Learning Long-Context Diffusion Policies via Past-Token Predictioncited by 28
  9. 2025EgoBridge: Domain Adaptation for Generalizable Imitation from Egocentric Human Datacited by 27
  10. 2025X-Sim: Cross-Embodiment Learning via Real-to-Sim-to-Realcited by 25
  11. 2025SPIDER: Scalable Physics-Informed Dexterous Retargetingcited by 25
  12. 2025Robot Learning from a Physical World Modelcited by 18
38 more
  1. 2026LAP: Language-Action Pre-Training Enables Zero-shot Cross-Embodiment Transfercited by 17
  2. 2026EgoWAM: World Action Models Beyond Pixels with In-the-Wild Egocentric Human Datacited by 17
  3. 2025Diffusion Trajectory-guided Policy for Long-horizon Robot Manipulationcited by 15
  4. 2025Pixel Motion as Universal Representation for Robot Controlcited by 15
  5. 2025Object-centric 3D Motion Field for Robot Learning from Human Videoscited by 13
  6. 2025OpenEgo: A Large-Scale Multimodal Egocentric Dataset for Dexterous Manipulationcited by 12
  7. 2025EC-Flow: Enabling Versatile Robotic Manipulation from Action-Unlabeled Videos via Embodiment-Centric Flowcited by 10
  8. 2025Scaffolding Dexterous Manipulation with Vision-Language Modelscited by 7
  9. 2025LatBot: Distilling Universal Latent Actions for Vision-Language-Action Modelscited by 7
  10. 2026Future Optical Flow Prediction Improves Robot Control & Video Generationcited by 7
  11. 2025What Changed and What Could Have Changed? State-Change Counterfactuals for Procedure-Aware Video Representation Learningcited by 6
  12. 2025X-Diffusion: Training Diffusion Policies on Cross-Embodiment Human Demonstrationscited by 6
  13. 2025OSMO: Open-Source Tactile Glove for Human-to-Robot Skill Transfercited by 6
  14. 2025ViSA-Flow: Accelerating Robot Skill Learning via Large-Scale Video Semantic Action Flowcited by 5
  15. 2026Cross-Hand Latent Representation for Vision-Language-Action Modelscited by 5
  16. 2025Provably Correct Automata Embeddings for Optimal Automata-Conditioned Reinforcement Learningcited by 4
  17. 2025Uni-Hand: Universal Hand Motion Forecasting in Egocentric Viewscited by 4
  18. 2026AugVLA-3D: Depth-Driven Feature Augmentation for Vision-Language-Action Modelscited by 3
  19. 2026Scaling Single Human Demonstrations for Imitation Learning using Generative Foundational Modelscited by 3
  20. 2025Learning Predictive Visuomotor Coordinationcited by 2
  21. 2025Traj2Action: A Co-Denoising Framework for Trajectory-Guided Human-to-Robot Skill Transfercited by 2
  22. 2025Trajectory Conditioned Cross-embodiment Skill Transfercited by 2
  23. 2025Towards Physics-informed Spatial Intelligence with Human Priors: An Autonomous Driving Pilot Studycited by 2
  24. 2026VISTA: Enhancing Visual Conditioning via Track-Following Preference Optimization in Vision-Language-Action Modelscited by 2
  25. 2026EasyMimic: A Low-Cost Framework for Robot Imitation Learning from Human Videoscited by 2
  26. 2026Tether: Autonomous Functional Play with Correspondence-Driven Trajectory Warpingcited by 2
  27. 2026GHOST: Hierarchical Sub-Goal Policies for Generalizing Robot Manipulationcited by 2
  28. 2026LUCID: Learning Embodiment-Agnostic Intent Models from Unstructured Human Videos for Scalable Dexterous Robot Skill Acquisitioncited by 2
  29. 2025ToolTipNet: A Segmentation-Driven Deep Learning Baseline for Surgical Instrument Tip Detectioncited by 1
  30. 2025HiBerNAC: Hierarchical Brain-inspired Robotic Neural Agent Collective for Disentangling Complex Manipulationcited by 1
  31. 2025Atomic Action Slicing: Planner-Aligned Options for Generalist VLA Agentscited by 1
  32. 2026Imitating What Works: Simulation-Filtered Modular Policy Learning from Human Videoscited by 1
  33. 2026MotionForesight: Re-purposing Video Models for Future 3D Scene-Flow Predictioncited by 1
  34. 2025Imitation Learning with Precisely Labeled Human Demonstrationscited by 0
  35. 2025EgoVIS@CVPR: What Changed and What Could Have Changed? State-Change Counterfactuals for Procedure-Aware Video Representation Learningcited by 0
  36. 2025EgoLoc: A Generalizable Solution for Temporal Interaction Localization in Egocentric Videoscited by 0
  37. 2026FSAG: Enhancing Human-to-Dexterous-Hand Finger-Specific Affordance Grounding via Diffusion Modelscited by 0
  38. 2026Flow-Enabled Generalization to Human Demonstrations in Few-Shot Imitation Learningcited by 0

50 most cited of 57 All 57 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Human-to-Robot Interaction: Learning from Video Demonstration for Robot Imitation
  2. 2021Coarse-to-Fine Imitation Learning: Robot Manipulation from a Single Demonstration
  3. 2021The Surprising Effectiveness of Representation Learning for Visual Imitation
  4. 2024Robotic Imitation of Human Actions
  5. 2024VITAL: Interactive Few-Shot Imitation Learning via Visual Human-in-the-Loop Corrections
  6. 2026Multi-Camera View Scaling for Data-Efficient Robot Imitation Learning
  7. 2026CORE: Common Outcome Regularities from Action-Free Visual Demonstrations for Robot Manipulation
  8. 2025OSVI-WM: One-Shot Visual Imitation for Unseen Tasks using World-Model-Guided Trajectory Generation