mlsanityData to 7 Oct 2026

RoboTAP: Tracking Arbitrary Points for Few-Shot Visual Imitation

arXiv 2308.15975 ↗First posted 30 Aug 2023cs.RO cs.AI cs.CVcited by 88

Not on the map. Why?

Abstract (arXiv)

For robots to be useful outside labs and specialized factories we need a way to teach them new useful behaviors quickly. Current approaches lack either the generality to onboard new tasks without task-specific engineering, or else lack the data-efficiency to do so in an amount of time that enables practical use. In this work we explore dense tracking as a representational vehicle to allow faster and more general learning from demonstration. Our approach utilizes Track-Any-Point (TAP) models to isolate the relevant motion in a demonstration, and parameterize a low-level controller to reproduce this motion across changes in the scene configuration. We show this results in robust robot policies that can solve complex object-arrangement tasks such as shape-matching, stacking, and even full path-following tasks such as applying glue and sticking objects together, all from demonstrations that can be collected in minutes.

Key facts. Cited by 88 papers in the map (data through 7 Oct 2026). Builds on 30 of its 49 references that are in the map. First posted on arXiv 30 Aug 2023.

Builds on

30 of 49 references.

References · 30

  1. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  2. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  3. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  4. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  5. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  6. 2020Object-Centric Learning with Slot Attentioncited by 865
  7. 2017PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenescited by 768
  8. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  9. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  10. 2023TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinementcited by 281
  11. 2022TAP-Vid: A Benchmark for Tracking Any Point in a Videocited by 258
  12. 2022Particle Video Revisited: Tracking Through Occlusions Using Point Trajectoriescited by 215
18 more
  1. 2023Tracking Everything Everywhere All at Oncecited by 190
  2. 2019kPAM: KeyPoint Affordances for Category-Level Robotic Manipulationcited by 152
  3. 2019Temporal Cycle-Consistency Learningcited by 151
  4. 2018Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulationcited by 140
  5. 2021XIRL: Cross-embodiment Inverse Reinforcement Learningcited by 101
  6. 2019Self-supervised 6D Object Pose Estimation for Robot Manipulationcited by 100
  7. 2019Making Sense of Vision and Touch: Learning Multimodal Representations for Contact-Rich Taskscited by 80
  8. 2020Model-Based Inverse Reinforcement Learning from Visual Demonstrationscited by 52
  9. 2021Learning to See before Learning to Act: Visual Pre-training for Manipulationcited by 50
  10. 2020S3K: Self-Supervised Semantic Keypoints for Robotic Manipulation via Multi-View Consistencycited by 28
  11. 2021Robust Multi-Modal Policies for Industrial Assembly via Reinforcement Learning and Demonstrations: A Large-Scale Studycited by 24
  12. 2022TexPose: Neural Texture Learning for Self-Supervised 6D Object Pose Estimationcited by 23
  13. 2022Sim-to-Real 6D Object Pose Estimation via Iterative Self-training for Robotic Bin Pickingcited by 20
  14. 2021Wish you were here: Hindsight Goal Selection for long-horizon dexterous manipulationcited by 13
  15. 2020Efficient video annotation with visual interpolation and frame selection guidancecited by 10
  16. 2021Research Challenges and Progress in Robotic Grasping and Manipulation Competitionscited by 9
  17. 2020Real-Time Deep Learning Approach to Visual Servo Control and Grasp Detection for Autonomous Robotic Manipulationcited by 4
  18. 2021Few-Shot Keypoint Detection as Task Adaptation via Latent Embeddingscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 27; 2025: 34; 2026: 23.

88 papers.

Sort:

By area:

  1. 2024ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulationcited by 305
  2. 2024TraceVLA: Visual Trace Prompting Enhances Spatial-Temporal Awareness for Generalist Robotic Policiescited by 298
  3. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  4. 2023Any-point Trajectory Modeling for Policy Learningcited by 207
  5. 2026π0.7: a Steerable Generalist Robotic Foundation Model with Emergent Capabilitiescited by 127
  6. 2024Flow as the Cross-Domain Manipulation Interfacecited by 126
  7. 2024Track2Act: Predicting Point Tracks from Internet Videos enables Generalizable Robot Manipulationcited by 116
  8. 2024BootsTAP: Bootstrapped Training for Tracking-Any-Pointcited by 115
  9. 2024General Flow as Foundation Affordance for Scalable Robot Learningcited by 80
  10. 2024Local All-Pair Correspondence for Point Trackingcited by 77
  11. 2024TAPVid-3D: A Benchmark for Tracking Any Point in 3Dcited by 69
  12. 2025TAPIP3D: Tracking Any Point in Persistent 3D Geometrycited by 67
38 more
  1. 2025Motion Tracks: A Unified Representation for Human-Robot Transfer in Few-Shot Imitation Learningcited by 57
  2. 2024RAM: Retrieval-Based Affordance Transfer for Generalizable Zero-Shot Robotic Manipulationcited by 55
  3. 2025AllTracker: Efficient Dense Point Tracking at High Resolutioncited by 54
  4. 2025RoboGround: Robotic Manipulation with Grounded Vision-Language Priorscited by 50
  5. 2024Vision-based Manipulation from Single Human Video with Open-World Object Graphscited by 46
  6. 2024DINOBot: Robot Manipulation via Retrieval and Alignment with Vision Foundation Modelscited by 43
  7. 2023DeformGS: Scene Flow in Highly Deformable Scenes for Deformable Object Manipulationcited by 42
  8. 2024Dynamic 3D Gaussian Tracking for Graph-Based Neural Dynamics Modelingcited by 42
  9. 2025TAPNext: Tracking Any Point (TAP) as Next Token Predictioncited by 35
  10. 2025Robotic Manipulation by Imitating Generated Videos Without Physical Demonstrationscited by 33
  11. 2024FlowRetrieval: Flow-Guided Data Retrieval for Few-Shot Imitation Learningcited by 32
  12. 2024DITTO: Demonstration Imitation by Trajectory Transformationcited by 28
  13. 2023DriveTrack: A Benchmark for Long-Range Point Tracking in Real-World Videoscited by 24
  14. 2025UAD: Unsupervised Affordance Distillation for Generalization in Robotic Manipulationcited by 23
  15. 2025SKIL: Semantic Keypoint Imitation Learning for Generalizable Data-efficient Manipulationcited by 21
  16. 2024Generative Image as Action Modelscited by 18
  17. 2024TAPTRv2: Attention-based Position Update Improves Tracking Any Pointcited by 18
  18. 2025AMPLIFY: Actionless Motion Priors for Robot Learning from Videoscited by 18
  19. 2025Multi-View 3D Point Trackingcited by 17
  20. 2025Exploring Temporally-Aware Features for Point Trackingcited by 15
  21. 2024Can Visual Foundation Models Achieve Long-term Point Tracking?cited by 14
  22. 2025Object-centric 3D Motion Field for Robot Learning from Human Videoscited by 13
  23. 2025TraceGen: World Modeling in 3D Trace Space Enables Learning from Cross-Embodiment Videoscited by 13
  24. 2026CoWTracker: Tracking by Warping instead of Correlationcited by 12
  25. 2023Zero-Shot Open-Vocabulary Tracking with Large Pre-Trained Modelscited by 11
  26. 2025Spatial Traces: Enhancing VLA Models with Spatial-Temporal Understandingcited by 11
  27. 2024Self-Supervised Any-Point Tracking by Contrastive Random Walkscited by 10
  28. 2024EgoPoints: Advancing Point Tracking for Egocentric Videoscited by 10
  29. 2026Track4World: Feedforward World-centric Dense 3D Tracking of All Pixelscited by 10
  30. 2024ETAP: Event-based Tracking of Any Pointcited by 9
  31. 2024TAPTRv3: Spatial and Temporal Context Foster Robust Tracking of Any Point in Long Videocited by 8
  32. 2024MFTIQ: Multi-Flow Tracker with Independent Matching Quality Estimationcited by 7
  33. 2025One-Shot Dual-Arm Imitation Learningcited by 7
  34. 2024TieBot: Learning to Knot a Tie from Visual Demonstration through a Real-to-Sim-to-Real Approachcited by 5
  35. 2026Video Understanding: From Geometry and Semantics to Unified Modelscited by 5
  36. 2026MegaFlow: Zero-Shot Large Displacement Optical Flowcited by 5
  37. 2026TAPNext++: What's Next for Tracking Any Point (TAP)?cited by 5
  38. 2026Beyond Data Scaling: Representation-Centric Continued Pre-training for Vision-Language-Action Modelscited by 5

50 most cited of 88 All 88 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2021Coarse-to-Fine Imitation Learning: Robot Manipulation from a Single Demonstration
  2. 2024SPOT: SE(3) Pose Trajectory Diffusion for Object-Centric Manipulation
  3. 2026Multi-Camera View Scaling for Data-Efficient Robot Imitation Learning
  4. 2026SparTa: Sparse Graphical Task Models from a Handful of Demonstrations
  5. 2024VITAL: Interactive Few-Shot Imitation Learning via Visual Human-in-the-Loop Corrections
  6. 2021Towards More Generalizable One-shot Visual Imitation Learning
  7. 2024Learning from 10 Demos: Generalisable and Sample-Efficient Policy Learning with Oriented Affordance Frames
  8. 2023Teach a Robot to FISH: Versatile Imitation from One Minute of Demonstrations