mlsanityData to 14 Oct 2026

TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinement

arXiv 2306.08637 ↗First posted 14 Jun 2023cs.CVcited by 287

Not on the map. Why?

Abstract (arXiv)

We present a novel model for Tracking Any Point (TAP) that effectively tracks any queried point on any physical surface throughout a video sequence. Our approach employs two stages: (1) a matching stage, which independently locates a suitable candidate point match for the query point on every other frame, and (2) a refinement stage, which updates both the trajectory and query features based on local correlations. The resulting model surpasses all baseline methods by a significant margin on the TAP-Vid benchmark, as demonstrated by an approximate 20% absolute average Jaccard (AJ) improvement on DAVIS. Our model facilitates fast inference on long and high-resolution video sequences. On a modern GPU, our implementation has the capacity to track points faster than real-time, and can be flexibly extended to higher-resolution videos. Given the high-quality trajectories extracted from a large dataset, we demonstrate a proof-of-concept diffusion model which generates trajectories from static images, enabling plausible animations. Visualizations, source code, and pretrained models can be found on our project webpage.

Key facts. Cited by 287 papers in the map (data through 14 Oct 2026). Builds on 50 of its 90 references that are in the map. First posted on arXiv 14 Jun 2023.

Builds on

50 of 90 references.

References · 50

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  3. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  4. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  5. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  6. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  7. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  8. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,415
  9. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,333
  10. 2022Video Diffusion Modelscited by 2,305
  11. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  12. 2017SuperPoint: Self-Supervised Interest Point Detection and Descriptioncited by 1,320
38 more
  1. 2015FlowNet: Learning Optical Flow with Convolutional Networkscited by 1,059
  2. 2017PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volumecited by 968
  3. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 957
  4. 2015A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimationcited by 881
  5. 2016FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networkscited by 784
  6. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 774
  7. 2018MegaDepth: Learning Single-View Depth Prediction from Internet Photoscited by 737
  8. 2018DensePose: Dense Human Pose Estimation In The Wildcited by 575
  9. 2016Optical Flow Estimation using a Spatial Pyramid Networkcited by 424
  10. 2022Kubric: A scalable dataset generatorcited by 381
  11. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 360
  12. 2021GMFlow: Learning Optical Flow via Global Matchingcited by 327
  13. 2017DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agentscited by 310
  14. 2016Learning from Simulated and Unsupervised Images through Adversarial Trainingcited by 288
  15. 2019FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Imagescited by 271
  16. 2017HPatches: A benchmark and evaluation of handcrafted and learned local descriptorscited by 264
  17. 2022TAP-Vid: A Benchmark for Tracking Any Point in a Videocited by 263
  18. 2022Particle Video Revisited: Tracking Through Occlusions Using Point Trajectoriescited by 217
  19. 2021COTR: Correspondence Transformer for Matching Across Imagescited by 183
  20. 2014Deep Learning Face Representation by Joint Identification-Verificationcited by 162
  21. 2018LF-Net: Learning Local Features from Imagescited by 151
  22. 2019PRNet: Self-Supervised Learning for Partial-to-Partial Registrationcited by 146
  23. 2015Human Pose Estimation with Iterative Error Feedbackcited by 121
  24. 2019Learning the Depths of Moving People by Watching Frozen Peoplecited by 120
  25. 2022ZebraPose: Coarse to Fine Surface Encoding for 6DoF Object Pose Estimationcited by 116
  26. 2018Unsupervised Learning of Object Landmarks through Conditional Image Generationcited by 114
  27. 2021AutoFlow: Learning a Better Training Set for Optical Flowcited by 102
  28. 2018Hierarchical Discrete Distribution Decomposition for Match Density Estimationcited by 93
  29. 2017SfM-Net: Learning of Structure and Motion from Videocited by 89
  30. 2021Rethinking Self-supervised Correspondence Learning: A Video Frame-level Similarity Perspectivecited by 81
  31. 2019Sim2real transfer learning for 3D human pose estimation: motion to the rescuecited by 76
  32. 2018Unsupervised Discovery of Object Landmarks as Structural Representationscited by 73
  33. 2020Keypoints into the Future: Self-Supervised Correspondence in Model-Based Reinforcement Learningcited by 70
  34. 2020MANTRA: Memory Augmented Networks for Multiple Trajectory Predictioncited by 69
  35. 2017Unsupervised learning of object landmarks by factorized spatial embeddingscited by 64
  36. 2022Coupled Iterative Refinement for 6D Multi-Object Pose Estimationcited by 48
  37. 2021Monocular 3D Reconstruction of Interacting Hands via Collision-Aware Factorized Refinementscited by 27
  38. 2021TSM: Temporal Shift Module for Efficient and Scalable Video Understanding on Edge Devicecited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 17; 2024: 78; 2025: 104; 2026: 88.

287 papers.

Sort:

By area:

  1. 2025VGGT: Visual Geometry Grounded Transformercited by 1,539
  2. 2023Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesiscited by 737
  3. 2023CoTracker: It is Better to Track Togethercited by 522
  4. 2024ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulationcited by 315
  5. 20243D Reconstruction with Spatial Memorycited by 277
  6. 2024Shape of Motion: 4D Reconstruction from a Single Videocited by 250
  7. 2023Visual Geometry Grounded Deep Structure From Motioncited by 227
  8. 2023Any-point Trajectory Modeling for Policy Learningcited by 215
  9. 2024SpatialTracker: Tracking Any 2D Pixels in 3D Spacecited by 181
  10. 2024Motion Prompting: Controlling Video Generation with Motion Trajectoriescited by 178
  11. 2024CAT4D: Create Anything in 4D with Multi-View Video Diffusion Modelscited by 161
  12. 2024Flow as the Cross-Domain Manipulation Interfacecited by 132
38 more
  1. 2025HAMSTER: Hierarchical Action Models For Open-World Robot Manipulationcited by 130
  2. 2024CoTracker3: Simpler and Better Point Tracking by Pseudo-Labelling Real Videoscited by 128
  3. 2024BootsTAP: Bootstrapped Training for Tracking-Any-Pointcited by 120
  4. 2023RoboTAP: Tracking Arbitrary Points for Few-Shot Visual Imitationcited by 90
  5. 2025St4RTrack: Simultaneous 4D Reconstruction and Tracking in the Worldcited by 89
  6. 2025PhysTwin: Physics-Informed Reconstruction and Simulation of Deformable Objects from Videoscited by 87
  7. 2025SpatialTrackerV2: 3D Point Tracking Made Easycited by 87
  8. 2023Segment Anything Meets Point Trackingcited by 81
  9. 2024Local All-Pair Correspondence for Point Trackingcited by 80
  10. 2024TAPVid-3D: A Benchmark for Tracking Any Point in 3Dcited by 72
  11. 2023VideoSwap: Customized Video Subject Swapping with Interactive Semantic Point Correspondencecited by 69
  12. 2025TAPIP3D: Tracking Any Point in Persistent 3D Geometrycited by 69
  13. 2024DELTA: Dense Efficient Long-range 3D Tracking for any videocited by 65
  14. 2024DINO-Tracker: Taming DINO for Self-Supervised Point Tracking in a Single Videocited by 64
  15. 2023Dense Optical Tracking: Connecting the Dotscited by 61
  16. 2024FlowMap: High-Quality Camera Poses, Intrinsics, and Depth via Gradient Descentcited by 61
  17. 2025AllTracker: Efficient Dense Point Tracking at High Resolutioncited by 58
  18. 2024LEAP-VO: Long-term Effective Any Point Tracking for Visual Odometrycited by 56
  19. 2025Efficiently Reconstructing Dynamic Scenes One D4RT at a Timecited by 54
  20. 2024GenXD: Generating Any 3D and 4D Scenescited by 50
  21. 20254DGT: Learning a 4D Gaussian Transformer Using Real-World Monocular Videoscited by 49
  22. 2024TAPTR: Tracking Any Point with Transformers as Detectioncited by 45
  23. 2024DINOBot: Robot Manipulation via Retrieval and Alignment with Vision Foundation Modelscited by 44
  24. 2024Track4Gen: Teaching Video Diffusion Models to Track Points Improves Video Generationcited by 44
  25. 2024Dynamic 3D Gaussian Tracking for Graph-Based Neural Dynamics Modelingcited by 43
  26. 2023DeformGS: Scene Flow in Highly Deformable Scenes for Deformable Object Manipulationcited by 42
  27. 2025GS-DiT: Advancing Video Generation with Pseudo 4D Gaussian Fields through Efficient Dense 3D Point Trackingcited by 39
  28. 2024R+X: Retrieval and Execution from Everyday Human Videoscited by 38
  29. 2025TAPNext: Tracking Any Point (TAP) as Next Token Predictioncited by 37
  30. 2023Tracking and Mapping in Medical Computer Vision: A Reviewcited by 36
  31. 2025MoVieS: Motion-Aware 4D Dynamic View Synthesis in One Secondcited by 35
  32. 2025Trace Anything: Representing Any Video in 4D via Trajectory Fieldscited by 33
  33. 2024Splatter a Video: Video Gaussian Representation for Versatile Processingcited by 31
  34. 2025Emergent Temporal Correspondences from Video Diffusion Transformerscited by 31
  35. 2024Scaling 4D Representationscited by 30
  36. 20244D Gaussian Splatting in the Wild with Uncertainty-Aware Regularizationcited by 28
  37. 2024PreF3R: Pose-Free Feed-Forward 3D Gaussian Splatting from Variable-length Image Sequencecited by 28
  38. 2023Pseudo-Generalized Dynamic View Synthesis from a Videocited by 27

50 most cited of 287 All 287 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Generative Point Tracking with Flow Matching
  2. 2026Track2View: 4D-Consistent Camera-Controlled Video Generation via Paired 3D Point Tracks
  3. 2026Glance Forcing: Making Bidirectional Diffusion Autoregressive at a Glance
  4. 2025Online Long-term Point Tracking in the Foundation Model Era
  5. 20263D Point Tracking with State Space Models
  6. 2026Go-with-the-Track: Video Compositing and Motion Control with Point Tracking
  7. 2025RELIC: Interactive Video World Model with Long-Horizon Memory
  8. 2023TrackDiffusion: Tracklet-Conditioned Video Generation via Diffusion Models