mlsanityData to 14 Oct 2026

TAP-Vid: A Benchmark for Tracking Any Point in a Video

arXiv 2211.03726 ↗First posted 7 Nov 2022cs.CV stat.MLcited by 263

Not on the map. Why?

Abstract (arXiv)

Generic motion understanding from video involves not only tracking objects, but also perceiving how their surfaces deform and move. This information is useful to make inferences about 3D shape, physical properties and object interactions. While the problem of tracking arbitrary physical points on surfaces over longer video clips has received some attention, no dataset or benchmark for evaluation existed, until now. In this paper, we first formalize the problem, naming it tracking any point (TAP). We introduce a companion benchmark, TAP-Vid, which is composed of both real-world videos with accurate human annotations of point tracks, and synthetic videos with perfect ground-truth point tracks. Central to the construction of our benchmark is a novel semi-automatic crowdsourced pipeline which uses optical flow estimates to compensate for easier, short-term motion like camera shake, allowing annotators to focus on harder sections of video. We validate our pipeline on synthetic data and propose a simple end-to-end point tracking model TAP-Net, showing that it outperforms all prior methods on our benchmark when trained on synthetic data.

Key facts. Cited by 263 papers in the map (data through 14 Oct 2026). Builds on 52 of its 98 references that are in the map. First posted on arXiv 7 Nov 2022.

Builds on

52 of 98 references.

References · 52

  1. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  2. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  3. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,415
  4. 2017SuperPoint: Self-Supervised Interest Point Detection and Descriptioncited by 1,320
  5. 2015FlowNet: Learning Optical Flow with Convolutional Networkscited by 1,059
  6. 2017PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volumecited by 968
  7. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 957
  8. 2015A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimationcited by 881
  9. 2015End-to-End Training of Deep Visuomotor Policiescited by 875
  10. 2016FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networkscited by 784
  11. 2018MegaDepth: Learning Single-View Depth Prediction from Internet Photoscited by 737
  12. 2018DensePose: Dense Human Pose Estimation In The Wildcited by 575
38 more
  1. 2020HOTA: A Higher Order Metric for Evaluating Multi-Object Trackingcited by 552
  2. 2016MOT16: A Benchmark for Multi-Object Trackingcited by 532
  3. 2018GOT-10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wildcited by 475
  4. 2016Optical Flow Estimation using a Spatial Pyramid Networkcited by 424
  5. 2018YouTube-VOS: A Large-Scale Video Object Segmentation Benchmarkcited by 383
  6. 2022Kubric: A scalable dataset generatorcited by 381
  7. 2016Learning from Simulated and Unsupervised Images through Adversarial Trainingcited by 288
  8. 2019Learning Correspondence from the Cycle-Consistency of Timecited by 269
  9. 2017HPatches: A benchmark and evaluation of handcrafted and learned local descriptorscited by 264
  10. 2020Space-Time Correspondence as a Contrastive Random Walkcited by 239
  11. 2021COTR: Correspondence Transformer for Matching Across Imagescited by 183
  12. 2020Whole-Body Human Pose Estimation in the Wildcited by 178
  13. 2017PoseTrack: A Benchmark for Human Pose Estimation and Trackingcited by 173
  14. 2018Tracking Emerges by Colorizing Videoscited by 166
  15. 2014Deep Learning Face Representation by Joint Identification-Verificationcited by 162
  16. 2019kPAM: KeyPoint Affordances for Category-Level Robotic Manipulationcited by 153
  17. 2015Stereo Matching by Training a Convolutional Neural Network to Compare Image Patchescited by 151
  18. 2018LF-Net: Learning Local Features from Imagescited by 151
  19. 2020TAO: A Large-Scale Benchmark for Tracking Any Objectcited by 147
  20. 2018Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulationcited by 142
  21. 2021Unidentified Video Objects: A Benchmark for Dense, Open-World Segmentationcited by 133
  22. 2018Efficient Interactive Annotation of Segmentation Datasets with Polygon-RNN++cited by 120
  23. 2019Learning the Depths of Moving People by Watching Frozen Peoplecited by 120
  24. 2018Unsupervised Learning of Object Landmarks through Conditional Image Generationcited by 114
  25. 2017YouTube-BoundingBoxes: A Large High-Precision Human-Annotated Data Set for Object Detection in Videocited by 109
  26. 2020MAST: A Memory-Augmented Self-supervised Trackercited by 109
  27. 2019Self-Supervised Correspondence in Visuomotor Policy Learningcited by 107
  28. 2021AutoFlow: Learning a Better Training Set for Optical Flowcited by 102
  29. 2017SfM-Net: Learning of Structure and Motion from Videocited by 89
  30. 2018Creatures great and SMAL: Recovering the shape and motion of animals from videocited by 84
  31. 2021Rethinking Self-supervised Correspondence Learning: A Video Frame-level Similarity Perspectivecited by 81
  32. 2019Joint-task Self-supervised Learning for Temporal Correspondencecited by 80
  33. 2018Unsupervised Discovery of Object Landmarks as Structural Representationscited by 73
  34. 2021Beyond Pick-and-Place: Tackling Robotic Stacking of Diverse Shapescited by 71
  35. 2020Keypoints into the Future: Self-Supervised Correspondence in Model-Based Reinforcement Learningcited by 70
  36. 2017Unsupervised learning of object landmarks by factorized spatial embeddingscited by 64
  37. 2021Unsupervised Object Learning via Common Fatecited by 29
  38. 2020S3K: Self-Supervised Semantic Keypoints for Robotic Manipulation via Multi-View Consistencycited by 28

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 30; 2024: 69; 2025: 99; 2026: 65.

263 papers.

Sort:

By area:

  1. 2025VGGT: Visual Geometry Grounded Transformercited by 1,539
  2. 2023Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesiscited by 737
  3. 2023CoTracker: It is Better to Track Togethercited by 522
  4. 2025Fast3R: Towards 3D Reconstruction of 1000+ Images in One Forward Passcited by 367
  5. 2023Perception Test: A Diagnostic Benchmark for Multimodal Video Modelscited by 314
  6. 2023TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinementcited by 287
  7. 2023PointOdyssey: A Large-Scale Synthetic Dataset for Long-Term Point Trackingcited by 276
  8. 2024Shape of Motion: 4D Reconstruction from a Single Videocited by 250
  9. 2023Visual Geometry Grounded Deep Structure From Motioncited by 227
  10. 2023Any-point Trajectory Modeling for Policy Learningcited by 215
  11. 2024Motion-I2V: Consistent and Controllable Image-to-Video Generation with Explicit Motion Modelingcited by 211
  12. 2023Tracking Everything Everywhere All at Oncecited by 193
38 more
  1. 2024SpatialTracker: Tracking Any 2D Pixels in 3D Spacecited by 181
  2. 2024Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulationcited by 153
  3. 2025Magma: A Foundation Model for Multimodal AI Agentscited by 139
  4. 2024CoTracker3: Simpler and Better Point Tracking by Pseudo-Labelling Real Videoscited by 128
  5. 2024Track2Act: Predicting Point Tracks from Internet Videos enables Generalizable Robot Manipulationcited by 124
  6. 2024BootsTAP: Bootstrapped Training for Tracking-Any-Pointcited by 120
  7. 2025Aether: Geometric-Aware Unified World Modelingcited by 97
  8. 2023RoboTAP: Tracking Arbitrary Points for Few-Shot Visual Imitationcited by 90
  9. 2024Stereo4D: Learning How Things Move in 3D from Internet Stereo Videoscited by 89
  10. 2025St4RTrack: Simultaneous 4D Reconstruction and Tracking in the Worldcited by 89
  11. 2025SpatialTrackerV2: 3D Point Tracking Made Easycited by 87
  12. 2023VideoFlow: Exploiting Temporal Cues for Multi-frame Optical Flow Estimationcited by 82
  13. 2023Segment Anything Meets Point Trackingcited by 81
  14. 2024Local All-Pair Correspondence for Point Trackingcited by 80
  15. 2024TAPVid-3D: A Benchmark for Tracking Any Point in 3Dcited by 72
  16. 2023VideoSwap: Customized Video Subject Swapping with Interactive Semantic Point Correspondencecited by 69
  17. 2025TAPIP3D: Tracking Any Point in Persistent 3D Geometrycited by 69
  18. 2024MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Modelscited by 67
  19. 2024DELTA: Dense Efficient Long-range 3D Tracking for any videocited by 65
  20. 2024DINO-Tracker: Taming DINO for Self-Supervised Point Tracking in a Single Videocited by 64
  21. 2024DreamScene4D: Dynamic Multi-Object Scene Generation from Monocular Videoscited by 63
  22. 2023MFT: Long-Term Tracking of Every Pixelcited by 62
  23. 2023Dense Optical Tracking: Connecting the Dotscited by 61
  24. 2025AllTracker: Efficient Dense Point Tracking at High Resolutioncited by 58
  25. 2024LEAP-VO: Long-term Effective Any Point Tracking for Visual Odometrycited by 56
  26. 2025Efficiently Reconstructing Dynamic Scenes One D4RT at a Timecited by 54
  27. 2024TAPTR: Tracking Any Point with Transformers as Detectioncited by 45
  28. 2024Track4Gen: Teaching Video Diffusion Models to Track Points Improves Video Generationcited by 44
  29. 2024Dynamic 3D Gaussian Tracking for Graph-Based Neural Dynamics Modelingcited by 43
  30. 2025ATI: Any Trajectory Instruction for Controllable Video Generationcited by 43
  31. 2023DeformGS: Scene Flow in Highly Deformable Scenes for Deformable Object Manipulationcited by 42
  32. 2023DragVideo: Interactive Drag-style Video Editingcited by 40
  33. 2023Fairy: Fast Parallelized Instruction-Guided Video-to-Video Synthesiscited by 39
  34. 2025GS-DiT: Advancing Video Generation with Pseudo 4D Gaussian Fields through Efficient Dense 3D Point Trackingcited by 39
  35. 2025TAPNext: Tracking Any Point (TAP) as Next Token Predictioncited by 37
  36. 2023Tracking and Mapping in Medical Computer Vision: A Reviewcited by 36
  37. 2025Trace Anything: Representing Any Video in 4D via Trajectory Fieldscited by 33
  38. 2024Splatter a Video: Video Gaussian Representation for Versatile Processingcited by 31

50 most cited of 263 All 263 (JSON)