mlsanityData to 14 Oct 2026

Track2Act: Predicting Point Tracks from Internet Videos enables Generalizable Robot Manipulation

arXiv 2405.01527 ↗First posted 2 May 2024cs.RO cs.CVcited by 124

On the map: Deep reinforcement learning methods → RL for robot locomotion and manipulation

Abstract (arXiv)

We seek to learn a generalizable goal-conditioned policy that enables zero-shot robot manipulation: interacting with unseen objects in novel scenes without test-time adaptation. While typical approaches rely on a large amount of demonstration data for such generalization, we propose an approach that leverages web videos to predict plausible interaction plans and learns a task-agnostic transformation to obtain robot actions in the real world. Our framework,Track2Act predicts tracks of how points in an image should move in future time-steps based on a goal, and can be trained with diverse videos on the web including those of humans and robots manipulating everyday objects. We use these 2D track predictions to infer a sequence of rigid transforms of the object to be manipulated, and obtain robot end-effector poses that can be executed in an open-loop manner. We then refine this open-loop plan by predicting residual actions through a closed loop policy trained with a few embodiment-specific demonstrations. We show that this approach of combining scalably learned track prediction with a residual policy requiring minimal in-domain robot-specific data enables diverse generalizable robot manipulation, and present a wide array of real-world robot manipulation results across unseen tasks, objects, and scenes. https://homangab.github.io/track2act/

Key facts. Cited by 124 papers in the map (data through 14 Oct 2026). Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods. Builds on 68 of its 75 references that are in the map. First posted on arXiv 2 May 2024.

Builds on

68 of 75 references.

References · 68

  1. 2022Scalable Diffusion Models with Transformerscited by 6,054
  2. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  3. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  4. 2017The Kinetics Human Action Video Datasetcited by 1,909
  5. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,568
  6. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  7. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,103
  8. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 862
  9. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 795
  10. 2017PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenescited by 775
  11. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 694
  12. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 657
38 more
  1. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 523
  2. 2023Learning Universal Policies via Text-Guided Video Generationcited by 522
  3. 2023CoTracker: It is Better to Track Togethercited by 522
  4. 2023Learning Interactive Real-World Simulatorscited by 458
  5. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 395
  6. 2019Learning joint reconstruction of hands and manipulated objectscited by 378
  7. 2023Photorealistic Video Generation with Diffusion Modelscited by 306
  8. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  9. 2022Unifying Flow, Stereo and Depth Estimationcited by 275
  10. 2022TAP-Vid: A Benchmark for Tracking Any Point in a Videocited by 263
  11. 2022Masked Visual Pre-training for Motor Controlcited by 248
  12. 2017SSD-6D: Making RGB-based 3D detection and 6D pose estimation great againcited by 243
  13. 2017Learning to Estimate 3D Hand Pose from Single RGB Imagescited by 234
  14. 2017One-Shot Visual Imitation Learning via Meta-Learningcited by 228
  15. 2023Affordances from Human Videos as a Versatile Representation for Roboticscited by 226
  16. 2017BB8: A Scalable, Accurate, Robust to Partial Occlusion Method for Predicting the 3D Poses of Challenging Objects without Using Depthcited by 225
  17. 2023MimicPlay: Long-Horizon Imitation Learning by Watching Human Playcited by 223
  18. 2023Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?cited by 218
  19. 2023Any-point Trajectory Modeling for Policy Learningcited by 215
  20. 2021DexMV: Imitation Learning for Dexterous Manipulation from Human Videoscited by 206
  21. 2018RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitationcited by 204
  22. 2019PVN3D: A Deep Point-wise 3D Keypoints Voting Network for 6DoF Pose Estimationcited by 204
  23. 20193D Hand Shape and Pose from Images in the Wildcited by 190
  24. 2021Where2Act: From Pixels to Actions for Articulated 3D Objectscited by 182
  25. 20193D Hand Shape and Pose Estimation from a Single RGB Imagecited by 174
  26. 2023RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunkingcited by 169
  27. 2023Learning to Act from Actionless Videos through Dense Correspondencescited by 164
  28. 2022Human-to-Robot Imitation in the Wildcited by 161
  29. 2022The Unsurprising Effectiveness of Pre-Trained Vision Models for Controlcited by 155
  30. 2020Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wildcited by 146
  31. 2021Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Timecited by 141
  32. 2018Grounded Human-Object Interaction Hotspots from Videocited by 129
  33. 2019Pushing the Envelope for RGB-based Dense 3D Hand Pose Estimation via Neural Renderingcited by 115
  34. 2018Hand Pose Estimation via Latent 2.5D Heatmap Regressioncited by 111
  35. 2020Visual Imitation Made Easycited by 111
  36. 2022Joint Hand Motion and Interaction Hotspots Prediction from Egocentric Videoscited by 111
  37. 2022VideoDex: Learning Dexterity from Internet Videoscited by 106
  38. 2019AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videoscited by 105

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 48; 2026: 70.

124 papers.

Sort:

By area:

  1. 2024TraceVLA: Visual Trace Prompting Enhances Spatial-Temporal Awareness for Generalist Robotic Policiescited by 304
  2. 2025Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasetscited by 240
  3. 2024EgoMimic: Scaling Imitation Learning via Egocentric Videocited by 175
  4. 2026π0.7: a Steerable Generalist Robotic Foundation Model with Emergent Capabilitiescited by 138
  5. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 84
  6. 2026PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulationcited by 82
  7. 2025Scalable Vision-Language-Action Model Pretraining for Robotic Manipulation with Real-Life Human Activity Videoscited by 70
  8. 2025TAPIP3D: Tracking Any Point in Persistent 3D Geometrycited by 69
  9. 2026EgoVerse: An Egocentric Human Dataset for Robot Learning from Around the Worldcited by 57
  10. 20254D-VLA: Spatiotemporal Vision-Language-Action Pretraining with Cross-Scene Calibrationcited by 56
  11. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 49
  12. 2024FLIP: Flow-Centric Generative Planning as General-Purpose Manipulation World Modelcited by 42
38 more
  1. 2025Emergence of Human to Robot Transfer in Vision-Language-Action Modelscited by 42
  2. 2025Diffusion Models for Robotic Manipulation: A Surveycited by 41
  3. 2025NovaFlow: Zero-Shot Manipulation via Actionable Flow from Generated Videoscited by 40
  4. 2024R+X: Retrieval and Execution from Everyday Human Videoscited by 38
  5. 2025Embodied-R1: Reinforced Embodied Reasoning for General Robotic Manipulationcited by 38
  6. 20253DFlowAction: Learning Cross-Embodiment Manipulation from 3D Flow World Modelcited by 37
  7. 2025From Seeing to Doing: Bridging Reasoning and Decision for Robotic Manipulationcited by 35
  8. 2025Trace Anything: Representing Any Video in 4D via Trajectory Fieldscited by 33
  9. 2025SPIDER: Scalable Physics-Informed Dexterous Retargetingcited by 32
  10. 2025Compliant Residual DAgger: Improving Real-World Contact-Rich Manipulation with Human Correctionscited by 29
  11. 2025EgoBridge: Domain Adaptation for Generalizable Imitation from Egocentric Human Datacited by 29
  12. 2025Pre-training Auto-regressive Robotic Models with 4D Representationscited by 27
  13. 2025ViPRA: Video Prediction for Robot Actionscited by 26
  14. 2025In-N-On: Scaling Egocentric Manipulation with in-the-wild and on-task Datacited by 26
  15. 2025Dream2Flow: Bridging Video Generation and Open-World Manipulation with 3D Object Flowcited by 25
  16. 2025Dexterous Manipulation through Imitation Learning: A Surveycited by 21
  17. 2025Robot Learning from a Physical World Modelcited by 20
  18. 2025Act2Goal: From World Model To General Goal-conditioned Policycited by 20
  19. 2025DemoDiffusion: One-Shot Human Imitation using pre-trained Diffusion Policycited by 19
  20. 2025Dexterity from Smart Lenses: Multi-Fingered Robot Manipulation with In-the-Wild Human Demonstrationscited by 19
  21. 2024The One RING: a Robotic Indoor Navigation Generalistcited by 16
  22. 2026HumanEgo: Zero-Shot Robot Learning from Minutes of Human Egocentric Videoscited by 16
  23. 2024HandsOnVLM: Vision-Language Models for Hand-Object Interaction Predictioncited by 15
  24. 2025TraceGen: World Modeling in 3D Trace Space Enables Learning from Cross-Embodiment Videoscited by 15
  25. 2025Mitty: Diffusion-based Human-to-Robot Video Generationcited by 14
  26. 2026MVP-LAM: Learning Action-Centric Latent Action via Cross-Viewpoint Reconstructioncited by 14
  27. 2026Dex4D: Task-Agnostic Point Track Policy for Zero-Shot Dexterous Manipulationcited by 14
  28. 2025Mantis: A Versatile Vision-Language-Action Model with Disentangled Visual Foresightcited by 13
  29. 2026Do as I Do: Dexterous Manipulation Data from Everyday Human Videoscited by 13
  30. 2025L4P: Towards Unified Low-Level 4D Vision Perceptioncited by 12
  31. 2025X-Humanoid: Robotize Human Videos to Generate Humanoid Videos at Scalecited by 12
  32. 2025EC-Flow: Enabling Versatile Robotic Manipulation from Action-Unlabeled Videos via Embodiment-Centric Flowcited by 11
  33. 2025What Happens Next? Anticipating Future Motion by Generating Point Trajectoriescited by 10
  34. 2025MV-UMI: A Scalable Multi-View Interface for Cross-Embodiment Learningcited by 9
  35. 2025H2R-Grounder: A Paired-Data-Free Paradigm for Translating Human Interaction Videos into Physically Grounded Robot Videoscited by 9
  36. 2025Flowing from Reasoning to Motion: Learning 3D Hand Trajectory Prediction from Egocentric Human Interaction Videoscited by 8
  37. 2026NovaPlan: Zero-Shot Long-Horizon Manipulation via Closed-Loop Video Language Planningcited by 8
  38. 2026SpatialVAM:Spatial-Aware Multi-View Video Diffusion as a Data-Efficient Robot Policycited by 8

50 most cited of 124 All 124 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Video Prediction Policy 2: Predict Better, Act Better
  2. 2025Motion Tracks: A Unified Representation for Human-Robot Transfer in Few-Shot Imitation Learning
  3. 2025ZeroMimic: Distilling Robotic Manipulation Skills from Web Videos
  4. 2025Generalist Robot Manipulation beyond Action Labeled Data
  5. 2023Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Models
  6. 2026GE-Act 2.0: Pretraining and Scaling a World-Action Model for Robotic Manipulation
  7. 2026Say, Dream, and Act: Learning Video World Models for Instruction-Driven Robot Manipulation
  8. 2026Zero-WAM: In-Context World-Action Modeling from Human Videos for Open-Ended Task Generalization