mlsanityData to 7 Oct 2026

Tracking Everything Everywhere All at Once

arXiv 2306.05422 ↗First posted 8 Jun 2023cs.CVcited by 190

Not on the map. Why?

Abstract (arXiv)

We present a new test-time optimization method for estimating dense and long-range motion from a video sequence. Prior optical flow or particle video tracking algorithms typically operate within limited temporal windows, struggling to track through occlusions and maintain global consistency of estimated motion trajectories. We propose a complete and globally consistent motion representation, dubbed OmniMotion, that allows for accurate, full-length motion estimation of every pixel in a video. OmniMotion represents a video using a quasi-3D canonical volume and performs pixel-wise tracking via bijections between local and canonical space. This representation allows us to ensure global consistency, track through occlusions, and model any combination of camera and object motion. Extensive evaluations on the TAP-Vid benchmark and real-world footage show that our approach outperforms prior state-of-the-art methods by a large margin both quantitatively and qualitatively. See our project page for more results: http://omnimotion.github.io/

Key facts. Cited by 190 papers in the map (data through 7 Oct 2026). Builds on 50 of its 86 references that are in the map. First posted on arXiv 8 Jun 2023.

Builds on

50 of 86 references.

References · 50

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  3. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  4. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  5. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,385
  6. 2020Implicit Neural Representations with Periodic Activation Functionscited by 2,214
  7. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,170
  8. 2016Density estimation using Real NVPcited by 2,029
  9. 2021Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fieldscited by 2,013
  10. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,484
  11. 2017SuperPoint: Self-Supervised Interest Point Detection and Descriptioncited by 1,313
  12. 2020Nerfies: Deformable Neural Radiance Fieldscited by 1,194
38 more
  1. 2015FlowNet: Learning Optical Flow with Convolutional Networkscited by 1,054
  2. 2021LoFTR: Detector-Free Local Feature Matching with Transformerscited by 974
  3. 2017PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volumecited by 966
  4. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 942
  5. 2016FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networkscited by 780
  6. 2020Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenescited by 750
  7. 2018MegaDepth: Learning Single-View Depth Prediction from Internet Photoscited by 731
  8. 2022Block-NeRF: Scalable Large Scene Neural View Synthesiscited by 720
  9. 2020Space-time Neural Irradiance Fields for Free-Viewpoint Videocited by 414
  10. 2022Kubric: A scalable dataset generatorcited by 373
  11. 2019D2-Net: A Trainable CNN for Joint Detection and Description of Local Featurescited by 324
  12. 2019Learning Correspondence from the Cycle-Consistency of Timecited by 269
  13. 2022TAP-Vid: A Benchmark for Tracking Any Point in a Videocited by 258
  14. 2020Space-Time Correspondence as a Contrastive Random Walkcited by 239
  15. 2022DynIBaR: Neural Dynamic Image-Based Renderingcited by 224
  16. 2022Particle Video Revisited: Tracking Through Occlusions Using Point Trajectoriescited by 215
  17. 2022Monocular Dynamic View Synthesis: A Reality Checkcited by 199
  18. 2021COTR: Correspondence Transformer for Matching Across Imagescited by 182
  19. 2018Neighbourhood Consensus Networkscited by 173
  20. 2018Tracking Emerges by Colorizing Videoscited by 166
  21. 2019GLU-Net: Global-Local Universal Network for Dense Flow and Correspondencescited by 139
  22. 2021Layered Neural Atlases for Consistent Video Editingcited by 137
  23. 2021LASR: Learning Articulated Shape Reconstruction from a Monocular Videocited by 115
  24. 2021Learning Accurate Dense Correspondences and When to Trust Themcited by 113
  25. 2020Learning Feature Descriptors using Camera Pose Supervisioncited by 104
  26. 2021Neural Parts: Learning Expressive 3D Shape Abstractions with Invertible Neural Networkscited by 101
  27. 2016Learning Dense Correspondence via 3D-guided Cycle Consistencycited by 91
  28. 2022ParticleSfM: Exploiting Dense Point Trajectories for Localizing Moving Cameras in the Wildcited by 90
  29. 2021Rethinking Self-supervised Correspondence Learning: A Video Frame-level Similarity Perspectivecited by 81
  30. 2019Joint-task Self-supervised Learning for Temporal Correspondencecited by 80
  31. 2021Beyond Pick-and-Place: Tackling Robotic Stacking of Diverse Shapescited by 70
  32. 2022Deformable Sprites for Unsupervised Video Decompositioncited by 61
  33. 2023MFT: Long-Term Tracking of Every Pixelcited by 61
  34. 2022Neural Surface Reconstruction of Dynamic Scenes with Monocular RGB-D Cameracited by 57
  35. 2022CaDeX: Learning Canonical Deformation Coordinate Space for Dynamic Surface Representation via Neural Homeomorphismcited by 56
  36. 2018A Fusion Approach for Multi-Frame Optical Flow Estimationcited by 37
  37. 2022Learning Pixel Trajectories with Multiscale Contrastive Random Walkscited by 36
  38. 2022CASA: Category-agnostic Skeletal Animal Reconstructioncited by 35

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 24; 2024: 68; 2025: 66; 2026: 32.

190 papers.

Sort:

By area:

  1. 2023Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesiscited by 731
  2. 2023CoTracker: It is Better to Track Togethercited by 516
  3. 2023DrivingGaussian: Composite Gaussian Splatting for Surrounding Dynamic Autonomous Driving Scenescited by 359
  4. 2024ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulationcited by 305
  5. 2024Shape of Motion: 4D Reconstruction from a Single Videocited by 247
  6. 2023EmerNeRF: Emergent Spatial-Temporal Scene Decomposition via Self-Supervisioncited by 216
  7. 2023Any-point Trajectory Modeling for Policy Learningcited by 207
  8. 2024SpatialTracker: Tracking Any 2D Pixels in 3D Spacecited by 178
  9. 20243DGStream: On-the-Fly Training of 3D Gaussians for Efficient Streaming of Photo-Realistic Free-Viewpoint Videoscited by 156
  10. 2025HAMSTER: Hierarchical Action Models For Open-World Robot Manipulationcited by 125
  11. 2023DynMF: Neural Motion Factorization for Real-time Dynamic View Synthesis with 3D Gaussian Splattingcited by 124
  12. 20234DGen: Grounded 4D Content Generation with Spatial-temporal Consistencycited by 117
38 more
  1. 2024BootsTAP: Bootstrapped Training for Tracking-Any-Pointcited by 115
  2. 2023RoboTAP: Tracking Arbitrary Points for Few-Shot Visual Imitationcited by 88
  3. 2025SpatialTrackerV2: 3D Point Tracking Made Easycited by 84
  4. 2023Segment Anything Meets Point Trackingcited by 81
  5. 2024Local All-Pair Correspondence for Point Trackingcited by 77
  6. 2023VideoSwap: Customized Video Subject Swapping with Interactive Semantic Point Correspondencecited by 69
  7. 2024TAPVid-3D: A Benchmark for Tracking Any Point in 3Dcited by 69
  8. 2025TAPIP3D: Tracking Any Point in Persistent 3D Geometrycited by 67
  9. 2024DINO-Tracker: Taming DINO for Self-Supervised Point Tracking in a Single Videocited by 64
  10. 2024MotionGS: Exploring Explicit Motion Guidance for Deformable 3D Gaussian Splattingcited by 62
  11. 2024DELTA: Dense Efficient Long-range 3D Tracking for any videocited by 62
  12. 2023Dense Optical Tracking: Connecting the Dotscited by 58
  13. 2024LEAP-VO: Long-term Effective Any Point Tracking for Visual Odometrycited by 54
  14. 2024MoDGS: Dynamic Gaussian Splatting from Casually-captured Monocular Videos with Depth Priorscited by 51
  15. 2024Towards A Better Metric for Text-to-Video Generationcited by 50
  16. 2024Matching Anything by Segmenting Anythingcited by 49
  17. 2024TAPTR: Tracking Any Point with Transformers as Detectioncited by 43
  18. 2023DeformGS: Scene Flow in Highly Deformable Scenes for Deformable Object Manipulationcited by 42
  19. 2024Dynamic 3D Gaussian Tracking for Graph-Based Neural Dynamics Modelingcited by 42
  20. 2024Track4Gen: Teaching Video Diffusion Models to Track Points Improves Video Generationcited by 42
  21. 2023DragVideo: Interactive Drag-style Video Editingcited by 40
  22. 2023MAS: Multi-view Ancestral Sampling for 3D motion generation using 2D diffusioncited by 37
  23. 2025GS-DiT: Advancing Video Generation with Pseudo 4D Gaussian Fields through Efficient Dense 3D Point Trackingcited by 37
  24. 2023Tracking and Mapping in Medical Computer Vision: A Reviewcited by 36
  25. 2025TAPNext: Tracking Any Point (TAP) as Next Token Predictioncited by 35
  26. 2024FaceXFormer: A Unified Transformer for Facial Analysiscited by 34
  27. 2024Splatter a Video: Video Gaussian Representation for Versatile Processingcited by 31
  28. 2025MoVieS: Motion-Aware 4D Dynamic View Synthesis in One Secondcited by 31
  29. 2025Trace Anything: Representing Any Video in 4D via Trajectory Fieldscited by 31
  30. 2024Recent Trends in 3D Reconstruction of General Non-Rigid Scenescited by 28
  31. 2023ResFields: Residual Neural Fields for Spatiotemporal Signalscited by 26
  32. 2025Dynamic Camera Poses and Where to Find Themcited by 26
  33. 2025Scaling up self-supervised learning for improved surgical foundation modelscited by 25
  34. 2024GauStudio: A Modular Framework for 3D Gaussian Splatting and Beyondcited by 24
  35. 2023Emergence of Segmentation with Minimalistic White-Box Transformerscited by 23
  36. 2024NetTrack: Tracking Highly Dynamic Objects with a Netcited by 22
  37. 2024Towards Generalizable Multi-Object Trackingcited by 22
  38. 2025Generative Video Motion Editing with 3D Point Trackscited by 21

50 most cited of 190 All 190 (JSON)