mlsanityData to 7 Oct 2026

SpatialTracker: Tracking Any 2D Pixels in 3D Space

arXiv 2404.04319 ↗First posted 5 Apr 2024cs.CVcited by 178

Not on the map. Why?

Cited by 22 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recovering dense and long-range pixel motion in videos is a challenging problem. Part of the difficulty arises from the 3D-to-2D projection process, leading to occlusions and discontinuities in the 2D motion domain. While 2D motion can be intricate, we posit that the underlying 3D motion can often be simple and low-dimensional. In this work, we propose to estimate point trajectories in 3D space to mitigate the issues caused by image projection. Our method, named SpatialTracker, lifts 2D pixels to 3D using monocular depth estimators, represents the 3D content of each frame efficiently using a triplane representation, and performs iterative updates using a transformer to estimate 3D trajectories. Tracking in 3D allows us to leverage as-rigid-as-possible (ARAP) constraints while simultaneously learning a rigidity embedding that clusters pixels into different rigid parts. Extensive evaluation shows that our approach achieves state-of-the-art tracking performance both qualitatively and quantitatively, particularly in challenging scenarios such as out-of-plane rotation.

Key facts. Cited by 178 papers in the map (data through 7 Oct 2026). Builds on 42 of its 82 references that are in the map. First posted on arXiv 5 Apr 2024.

Builds on

42 of 82 references.

References · 42

  1. 2017Attention Is All You Needcited by 59,932
  2. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  3. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,385
  4. 2021Vision Transformers for Dense Predictioncited by 1,789
  5. 2021Efficient Geometry-aware 3D Generative Adversarial Networkscited by 1,204
  6. 2015FlowNet: Learning Optical Flow with Convolutional Networkscited by 1,054
  7. 2017PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volumecited by 966
  8. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 942
  9. 2015A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimationcited by 878
  10. 2016FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networkscited by 780
  11. 2020Convolutional Occupancy Networkscited by 760
  12. 2023ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depthcited by 646
30 more
  1. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  2. 2023CoTracker: It is Better to Track Togethercited by 516
  3. 2022Kubric: A scalable dataset generatorcited by 373
  4. 2023TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinementcited by 281
  5. 2020Softmax Splatting for Video Frame Interpolationcited by 276
  6. 2023PointOdyssey: A Large-Scale Synthetic Dataset for Long-Term Point Trackingcited by 274
  7. 2022FlowFormer: A Transformer Architecture for Optical Flowcited by 273
  8. 2022TAP-Vid: A Benchmark for Tracking Any Point in a Videocited by 258
  9. 2018FlowNet3D: Learning Scene Flow in 3D Point Cloudscited by 241
  10. 2022Particle Video Revisited: Tracking Through Occlusions Using Point Trajectoriescited by 215
  11. 2023Tracking Everything Everywhere All at Oncecited by 190
  12. 2023MiDaS v3.1 -- A Model Zoo for Robust Monocular Relative Depth Estimationcited by 173
  13. 2021Layered Neural Atlases for Consistent Video Editingcited by 137
  14. 2020RAFT-3D: Scene Flow using Rigid-Motion Embeddingscited by 121
  15. 2023FlowFormer++: Masked Cost Volume Autoencoding for Pretraining Optical Flow Estimationcited by 114
  16. 2019HPLFlowNet: Hierarchical Permutohedral Lattice FlowNet for Scene Flow Estimation on Large-scale Point Cloudscited by 108
  17. 2018Creatures great and SMAL: Recovering the shape and motion of animals from videocited by 84
  18. 2021Beyond Pick-and-Place: Tackling Robotic Stacking of Diverse Shapescited by 70
  19. 2017Accurate Optical Flow via Direct Cost Volume Processingcited by 66
  20. 2020Neural Deformation Graphs for Globally-consistent Non-rigid Reconstructioncited by 66
  21. 2022Global Matching with Overlapping Attention for Optical Flow Estimationcited by 64
  22. 2022Deformable Sprites for Unsupervised Video Decompositioncited by 61
  23. 2023MFT: Long-Term Tracking of Every Pixelcited by 61
  24. 2022CaDeX: Learning Canonical Deformation Coordinate Space for Dynamic Surface Representation via Neural Homeomorphismcited by 56
  25. 2019FlowNet3D++: Geometric Losses For Deep Scene Flow Estimationcited by 55
  26. 2019Deep Rigid Instance Scene Flowcited by 54
  27. 2018A Fusion Approach for Multi-Frame Optical Flow Estimationcited by 37
  28. 2022OcclusionFusion: Occlusion-aware Motion Estimation for Real-time Dynamic 3D Reconstructioncited by 27
  29. 2016Optical Flow with Semantic Segmentation and Localized Layerscited by 21
  30. 2022ALTO: Alternating Latent Topologies for Implicit 3D Reconstructioncited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 30; 2025: 86; 2026: 62.

178 papers.

Sort:

By area:

  1. 2025VGGT: Visual Geometry Grounded Transformercited by 1,475
  2. 2024ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulationcited by 305
  3. 2025ReCamMaster: Camera-Controlled Generative Rendering from A Single Videocited by 278
  4. 20243D Reconstruction with Spatial Memorycited by 269
  5. 2024Shape of Motion: 4D Reconstruction from a Single Videocited by 247
  6. 2025Diffusion as Shader: 3D-aware Video Diffusion for Versatile Video Generation Controlcited by 184
  7. 2024MoSca: Dynamic Gaussian Fusion from Casual Videos via 4D Motion Scaffoldscited by 177
  8. 2024Lotus: Diffusion-based Visual Foundation Model for High-quality Dense Predictioncited by 163
  9. 2025St4RTrack: Simultaneous 4D Reconstruction and Tracking in the Worldcited by 88
  10. 2025SpatialTrackerV2: 3D Point Tracking Made Easycited by 84
  11. 2024Align3R: Aligned Monocular Depth Estimation for Dynamic Videoscited by 79
  12. 2026PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulationcited by 74
38 more
  1. 2025CineMaster: A 3D-Aware and Controllable Framework for Cinematic Text-to-Video Generationcited by 73
  2. 2025MotionStream: Real-Time Video Generation with Interactive Motion Controlscited by 71
  3. 2024TAPVid-3D: A Benchmark for Tracking Any Point in 3Dcited by 69
  4. 2025TAPIP3D: Tracking Any Point in Persistent 3D Geometrycited by 67
  5. 2025Geo4D: Leveraging Video Generators for Geometric 4D Scene Reconstructioncited by 66
  6. 2024DELTA: Dense Efficient Long-range 3D Tracking for any videocited by 62
  7. 2025AllTracker: Efficient Dense Point Tracking at High Resolutioncited by 54
  8. 2025Efficiently Reconstructing Dynamic Scenes One D4RT at a Timecited by 46
  9. 2025Dynamic Point Maps: A Versatile Representation for Dynamic 3D Reconstructioncited by 43
  10. 2024Dynamic 3D Gaussian Tracking for Graph-Based Neural Dynamics Modelingcited by 42
  11. 2024Track4Gen: Teaching Video Diffusion Models to Track Points Improves Video Generationcited by 42
  12. 2024I2VControl-Camera: Precise Video Camera Control with Adjustable Motion Strengthcited by 38
  13. 2025RoboScape: Physics-informed Embodied World Modelcited by 38
  14. 2025GS-DiT: Advancing Video Generation with Pseudo 4D Gaussian Fields through Efficient Dense 3D Point Trackingcited by 37
  15. 2025GeometryCrafter: Consistent Geometry Estimation for Open-world Videos with Diffusion Priorscited by 36
  16. 2025TAPNext: Tracking Any Point (TAP) as Next Token Predictioncited by 35
  17. 2025Any4D: Unified Feed-Forward Metric 4D Reconstructioncited by 32
  18. 2024Splatter a Video: Video Gaussian Representation for Versatile Processingcited by 31
  19. 2025MoVieS: Motion-Aware 4D Dynamic View Synthesis in One Secondcited by 31
  20. 2025Trace Anything: Representing Any Video in 4D via Trajectory Fieldscited by 31
  21. 2025Pre-training Auto-regressive Robotic Models with 4D Representationscited by 27
  22. 2024PreF3R: Pose-Free Feed-Forward 3D Gaussian Splatting from Variable-length Image Sequencecited by 26
  23. 2025POMATO: Marrying Pointmap Matching with Temporal Motion for Dynamic 3D Reconstructioncited by 24
  24. 2025Back on Track: Bundle Adjustment for Dynamic Scene Reconstructioncited by 23
  25. 2025Perception-as-Control: Fine-grained Controllable Image Animation with 3D-aware Motion Representationcited by 22
  26. 2025LongVie: Multimodal-Guided Controllable Ultra-Long Video Generationcited by 21
  27. 2025Generative Video Motion Editing with 3D Point Trackscited by 21
  28. 2024DynOMo: Online Point Tracking by Dynamic Online Monocular Gaussian Reconstructioncited by 20
  29. 2025Zero-Shot Monocular Scene Flow Estimation in the Wildcited by 20
  30. 2024I2VControl: Disentangled and Unified Video Motion Synthesis Controlcited by 19
  31. 2024TAPTRv2: Attention-based Position Update Improves Tracking Any Pointcited by 18
  32. 2024ObjCtrl-2.5D: Training-free Object Control with Camera Posescited by 18
  33. 2025AMPLIFY: Actionless Motion Priors for Robot Learning from Videoscited by 18
  34. 2025FlowDreamer: A RGB-D World Model with Flow-based Motion Representations for Robot Manipulationcited by 17
  35. 2025Multi-View 3D Point Trackingcited by 17
  36. 20264RC: 4D Reconstruction via Conditional Querying Anytime and Anywherecited by 17
  37. 2024MotionStone: Decoupled Motion Intensity Modulation with Diffusion Transformer for Image-to-Video Generationcited by 16
  38. 2024Fast Encoder-Based 3D from Casual Videos via Point Track Processingcited by 14

50 most cited of 178 All 178 (JSON)