mlsanityData to 7 Oct 2026

Any-point Trajectory Modeling for Policy Learning

arXiv 2401.00025 ↗First posted 28 Dec 2023cs.RO cs.CVcited by 207

On the map: Offline RL and imitation learning from data → Robot manipulation imitation learning

Abstract (arXiv)

Learning from demonstration is a powerful method for teaching robots new skills, and having more demonstration data often improves policy learning. However, the high cost of collecting demonstration data is a significant bottleneck. Videos, as a rich data source, contain knowledge of behaviors, physics, and semantics, but extracting control-specific information from them is challenging due to the lack of action labels. In this work, we introduce a novel framework, Any-point Trajectory Modeling (ATM), that utilizes video demonstrations by pre-training a trajectory model to predict future trajectories of arbitrary points within a video frame. Once trained, these trajectories provide detailed control guidance, enabling the learning of robust visuomotor policies with minimal action-labeled data. Across over 130 language-conditioned tasks we evaluated in both simulation and the real world, ATM outperforms strong video pre-training baselines by 80% on average. Furthermore, we show effective transfer learning of manipulation skills from human videos and videos from a different robot morphology. Visualizations and code are available at: urlhttps://xingyu-lin.github.io/atm.

Key facts. Cited by 207 papers in the map (data through 7 Oct 2026). Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 52 of its 62 references that are in the map. First posted on arXiv 28 Dec 2023.

Builds on

52 of 62 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023GELLO: A General, Low-Cost, and Intuitive Teleoperation Framework for Robot Manipulators22 Sep
  2. 2023BridgeData V2: A Dataset for Robot Learning at Scale24 Aug
  3. 2023MimicPlay: Long-Horizon Imitation Learning by Watching Human Play24 Feb
  4. 2022ToolFlowNet: Robotic Manipulation with Tools via Predicting Tool Flow from Point Clouds16 Nov
  5. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training30 Sep

Other references · 47

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023Segment Anythingcited by 8,158
  4. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  5. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  6. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  7. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,338
  8. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  9. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
  10. 2022A Generalist Agentcited by 802
  11. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  12. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
35 more
  1. 2023CoTracker: It is Better to Track Togethercited by 516
  2. 2023Learning Universal Policies via Text-Guided Video Generationcited by 506
  3. 2018Behavioral Cloning from Observationcited by 482
  4. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 453
  5. 2023Learning Interactive Real-World Simulatorscited by 440
  6. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 338
  7. 2019Causal Confusion in Imitation Learningcited by 286
  8. 2023TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinementcited by 281
  9. 2023PointOdyssey: A Large-Scale Synthetic Dataset for Long-Term Point Trackingcited by 274
  10. 2023Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Modelscited by 264
  11. 2022TAP-Vid: A Benchmark for Tracking Any Point in a Videocited by 258
  12. 2023Affordances from Human Videos as a Versatile Representation for Roboticscited by 221
  13. 2022Particle Video Revisited: Tracking Through Occlusions Using Point Trajectoriescited by 215
  14. 2023Tracking Everything Everywhere All at Oncecited by 190
  15. 2023RH20T: A Comprehensive Robotic Dataset for Learning Diverse Skills in One-Shotcited by 173
  16. 2023Learning to Act from Actionless Videos through Dense Correspondencescited by 159
  17. 2023Structured World Models from Human Videoscited by 144
  18. 2023RT-Trajectory: Robotic Task Generalization via Hindsight Trajectory Sketchescited by 131
  19. 2023Learning to Act without Actionscited by 116
  20. 20213D Neural Scene Representations for Visuomotor Controlcited by 110
  21. 2022Reinforcement Learning with Action-Free Pre-Training from Videoscited by 107
  22. 2022VideoDex: Learning Dexterity from Internet Videoscited by 101
  23. 2023Language Embedded Radiance Fields for Zero-Shot Task-Oriented Graspingcited by 91
  24. 2023RoboTAP: Tracking Arbitrary Points for Few-Shot Visual Imitationcited by 88
  25. 2021Learning by Watching: Physical Imitation of Manipulation Skills from Human Videoscited by 87
  26. 2023Video Prediction Models as Rewards for Reinforcement Learningcited by 76
  27. 2020Fighting Copycat Agents in Behavioral Cloning from Observation Historiescited by 70
  28. 2020Keypoints into the Future: Self-Supervised Correspondence in Model-Based Reinforcement Learningcited by 69
  29. 2019KETO: Learning Keypoint Representations for Tool Manipulationcited by 68
  30. 2021Learning Visible Connectivity Dynamics for Cloth Smoothingcited by 61
  31. 2023Zero-Shot Robot Manipulation from Passive Human Videoscited by 49
  32. 2022IFOR: Iterative Flow Minimization for Robotic Object Rearrangementcited by 42
  33. 2019Learning Predictive Models From Observation and Interactioncited by 41
  34. 2023SpawnNet: Learning Generalizable Visuomotor Skills from Pre-trained Networkscited by 24
  35. 2022Fighting Fire with Fire: Avoiding DNN Shortcuts through Primingcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 33; 2025: 91; 2026: 83.

207 papers.

Sort:

By area:

  1. 2025CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Modelscited by 440
  2. 2025UniVLA: Learning to Act Anywhere with Task-centric Latent Actionscited by 386
  3. 2024ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulationcited by 305
  4. 2024Latent Action Pretraining from Videoscited by 302
  5. 2024TraceVLA: Visual Trace Prompting Enhances Spatial-Temporal Awareness for Generalist Robotic Policiescited by 298
  6. 2024GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulationcited by 290
  7. 2025Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasetscited by 218
  8. 2025DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledgecited by 203
  9. 2025DreamGen: Unlocking Generalization in Robot Learning through Video World Modelscited by 168
  10. 2024EgoMimic: Scaling Imitation Learning via Egocentric Videocited by 160
  11. 2024Data Scaling Laws in Imitation Learning for Robotic Manipulationcited by 144
  12. 2024Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulationcited by 142
38 more
  1. 2026π0.7: a Steerable Generalist Robotic Foundation Model with Emergent Capabilitiescited by 127
  2. 2024Flow as the Cross-Domain Manipulation Interfacecited by 126
  3. 2025HAMSTER: Hierarchical Action Models For Open-World Robot Manipulationcited by 125
  4. 2024Track2Act: Predicting Point Tracks from Internet Videos enables Generalizable Robot Manipulationcited by 116
  5. 2024BootsTAP: Bootstrapped Training for Tracking-Any-Pointcited by 115
  6. 2025GR-3 Technical Reportcited by 110
  7. 2025FLARE: Robot Learning with Implicit World Modelingcited by 105
  8. 2025DexGraspVLA: A Vision-Language-Action Framework Towards General Dexterous Graspingcited by 87
  9. 2024Video as the New Language for Real-World Decision Makingcited by 86
  10. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 82
  11. 2024General Flow as Foundation Affordance for Scalable Robot Learningcited by 80
  12. 2025Universal Actions for Enhanced Embodied Foundation Modelscited by 80
  13. 2026PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulationcited by 74
  14. 2024TAPVid-3D: A Benchmark for Tracking Any Point in 3Dcited by 69
  15. 2026Being-H0.7: A Latent World-Action Model from Egocentric Videoscited by 68
  16. 2025VQ-VLA: Improving Vision-Language-Action Models via Scaling Vector-Quantized Action Tokenizerscited by 67
  17. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Surveycited by 66
  18. 2025Humanoid Policy ~ Human Policycited by 65
  19. 2025WorldEval: World Model as Real-World Robot Policies Evaluatorcited by 64
  20. 2025Scalable Vision-Language-Action Model Pretraining for Robotic Manipulation with Real-Life Human Activity Videoscited by 63
  21. 2026Being-H0.5: Scaling Human-Centric Robot Learning for Cross-Embodiment Generalizationcited by 61
  22. 2025Motion Tracks: A Unified Representation for Human-Robot Transfer in Few-Shot Imitation Learningcited by 57
  23. 20254D-VLA: Spatiotemporal Vision-Language-Action Pretraining with Cross-Scene Calibrationcited by 55
  24. 2024Robot See Robot Do: Imitating Articulated Object Manipulation with Monocular 4D Reconstructioncited by 51
  25. 2025RoboGround: Robotic Manipulation with Grounded Vision-Language Priorscited by 50
  26. 2025ChatVLA-2: Vision-Language-Action Model with Open-World Embodied Reasoning from Pretrained Knowledgecited by 49
  27. 2024Vision-based Manipulation from Single Human Video with Open-World Object Graphscited by 46
  28. 2024FLIP: Flow-Centric Generative Planning as General-Purpose Manipulation World Modelcited by 41
  29. 2025H-RDT: Human Manipulation Enhanced Bimanual Robotic Manipulationcited by 41
  30. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  31. 2024RT-Affordance: Affordances are Versatile Intermediate Representations for Robot Manipulationcited by 40
  32. 2024Hand-Object Interaction Pretraining from Videoscited by 38
  33. 2025CoMo: Learning Continuous Latent Motion from Internet Videos for Scalable Robot Learningcited by 37
  34. 20253DFlowAction: Learning Cross-Embodiment Manipulation from 3D Flow World Modelcited by 36
  35. 2025Embodied-R1: Reinforced Embodied Reasoning for General Robotic Manipulationcited by 36
  36. 2025TAPNext: Tracking Any Point (TAP) as Next Token Predictioncited by 35
  37. 2024From LLMs to Actions: Latent Codes as Bridges in Hierarchical Robot Controlcited by 34
  38. 2025From Seeing to Doing: Bridging Reasoning and Decision for Robotic Manipulationcited by 34

50 most cited of 207 All 207 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Latent Policy Steering with Embodiment-Agnostic Pretrained World Models
  2. 2024Learning from 10 Demos: Generalisable and Sample-Efficient Policy Learning with Oriented Affordance Frames
  3. 2025Scalable Policy Evaluation with Video World Models
  4. 2026Latent Policy Steering: An Efficient and Flexible Framework for Cross-Embodiment Transfer ↗
  5. 2024Manipulate-Anything: Automating Real-World Robots using Vision-Language Models
  6. 2026DeVA: Decoupled Video-Action Model with physical guidance for robot policy learning
  7. 2024Dreamitate: Real-World Visuomotor Policy Learning via Video Generation
  8. 2026CORE: Common Outcome Regularities from Action-Free Visual Demonstrations for Robot Manipulation