# Dense Optical Tracking: Connecting the Dots

arXiv: https://arxiv.org/abs/2312.00786 · HTML: https://mlsanity.com/paper/2312.00786/

## Key facts

- Cited by 61 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 76 references that are in the map.
- First posted on arXiv 1 Dec 2023.

## Builds on (most cited first, 30 of 44)

- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [SuperPoint: Self-Supervised Interest Point Detection and Description](https://mlsanity.com/paper/1712.07629/) (2017)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation](https://mlsanity.com/paper/1512.02134/) (2015)
- [FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks](https://mlsanity.com/paper/1612.01925/) (2016)
- [Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis](https://mlsanity.com/paper/2308.09713/) (2023)
- [Perceiver IO: A General Architecture for Structured Inputs & Outputs](https://mlsanity.com/paper/2107.14795/) (2021)
- [LightGlue: Local Feature Matching at Light Speed](https://mlsanity.com/paper/2306.13643/) (2023)
- [CoTracker: It is Better to Track Together](https://mlsanity.com/paper/2307.07635/) (2023)
- [Optical Flow Estimation using a Spatial Pyramid Network](https://mlsanity.com/paper/1611.00850/) (2016)
- [Kubric: A scalable dataset generator](https://mlsanity.com/paper/2203.03570/) (2022)
- [DVC: An End-to-end Deep Video Compression Framework](https://mlsanity.com/paper/1812.00101/) (2018)
- [GMFlow: Learning Optical Flow via Global Matching](https://mlsanity.com/paper/2111.13680/) (2021)
- [DISK: Learning local features with policy gradient](https://mlsanity.com/paper/2006.13566/) (2020)
- [TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinement](https://mlsanity.com/paper/2306.08637/) (2023)
- [PointOdyssey: A Large-Scale Synthetic Dataset for Long-Term Point Tracking](https://mlsanity.com/paper/2307.15055/) (2023)
- [FlowFormer: A Transformer Architecture for Optical Flow](https://mlsanity.com/paper/2203.16194/) (2022)
- [TAP-Vid: A Benchmark for Tracking Any Point in a Video](https://mlsanity.com/paper/2211.03726/) (2022)
- [Learning to Estimate Hidden Motions with Global Motion Aggregation](https://mlsanity.com/paper/2104.02409/) (2021)
- [Particle Video Revisited: Tracking Through Occlusions Using Point Trajectories](https://mlsanity.com/paper/2204.04153/) (2022)
- [Real-Time Intermediate Flow Estimation for Video Frame Interpolation](https://mlsanity.com/paper/2011.06294/) (2020)
- [ProPainter: Improving Propagation and Transformer for Video Inpainting](https://mlsanity.com/paper/2309.03897/) (2023)
- [Tracking Everything Everywhere All at Once](https://mlsanity.com/paper/2306.05422/) (2023)
- [AMT: All-Pairs Multi-Field Transforms for Efficient Frame Interpolation](https://mlsanity.com/paper/2304.09790/) (2023)
- [Self-supervised Video Object Segmentation by Motion Grouping](https://mlsanity.com/paper/2104.07658/) (2021)
- [Layered Neural Atlases for Consistent Video Editing](https://mlsanity.com/paper/2109.11418/) (2021)
- [ALIKED: A Lighter Keypoint and Descriptor Extraction Network via Deformable Transformation](https://mlsanity.com/paper/2304.03608/) (2023)

## Built on by (most cited first, 30 of 61)

- [VGGT: Visual Geometry Grounded Transformer](https://mlsanity.com/paper/2503.11651/) (2025)
- [Motion-I2V: Consistent and Controllable Image-to-Video Generation with Explicit Motion Modeling](https://mlsanity.com/paper/2401.15977/) (2024)
- [T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generation](https://mlsanity.com/paper/2407.14505/) (2024)
- [Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulation](https://mlsanity.com/paper/2409.16283/) (2024)
- [CoTracker3: Simpler and Better Point Tracking by Pseudo-Labelling Real Videos](https://mlsanity.com/paper/2410.11831/) (2024)
- [BootsTAP: Bootstrapped Training for Tracking-Any-Point](https://mlsanity.com/paper/2402.00847/) (2024)
- [Local All-Pair Correspondence for Point Tracking](https://mlsanity.com/paper/2407.15420/) (2024)
- [TAPIP3D: Tracking Any Point in Persistent 3D Geometry](https://mlsanity.com/paper/2504.14717/) (2025)
- [DELTA: Dense Efficient Long-range 3D Tracking for any video](https://mlsanity.com/paper/2410.24211/) (2024)
- [AllTracker: Efficient Dense Point Tracking at High Resolution](https://mlsanity.com/paper/2506.07310/) (2025)
- [Efficiently Reconstructing Dynamic Scenes One D4RT at a Time](https://mlsanity.com/paper/2512.08924/) (2025)
- [GS-DiT: Advancing Video Generation with Pseudo 4D Gaussian Fields through Efficient Dense 3D Point Tracking](https://mlsanity.com/paper/2501.02690/) (2025)
- [MotionPro: A Precise Motion Controller for Image-to-Video Generation](https://mlsanity.com/paper/2505.20287/) (2025)
- [Uni4D: Unifying Visual Foundation Models for 4D Modeling from a Single Video](https://mlsanity.com/paper/2503.21761/) (2025)
- [Back on Track: Bundle Adjustment for Dynamic Scene Reconstruction](https://mlsanity.com/paper/2504.14516/) (2025)
- [TAPTRv2: Attention-based Position Update Improves Tracking Any Point](https://mlsanity.com/paper/2407.16291/) (2024)
- [FlowDreamer: A RGB-D World Model with Flow-based Motion Representations for Robot Manipulation](https://mlsanity.com/paper/2505.10075/) (2025)
- [BlobGEN-Vid: Compositional Text-to-Video Generation with Blob Video Representations](https://mlsanity.com/paper/2501.07647/) (2025)
- [CoWTracker: Tracking by Warping instead of Correlation](https://mlsanity.com/paper/2602.04877/) (2026)
- [Trajectory-aligned Space-time Tokens for Few-shot Action Recognition](https://mlsanity.com/paper/2407.18249/) (2024)
- [DynaGSLAM: Real-Time Gaussian-Splatting SLAM for Online Rendering, Tracking, Motion Predictions of Moving Objects in Dynamic Scenes](https://mlsanity.com/paper/2503.11979/) (2025)
- [DELTAv2: Accelerating Dense 3D Tracking](https://mlsanity.com/paper/2508.01170/) (2025)
- [PiTe: Pixel-Temporal Alignment for Large Video-Language Model](https://mlsanity.com/paper/2409.07239/) (2024)
- [MFTIQ: Multi-Flow Tracker with Independent Matching Quality Estimation](https://mlsanity.com/paper/2411.09551/) (2024)
- [Trokens: Semantic-Aware Relational Trajectory Tokens for Few-Shot Action Recognition](https://mlsanity.com/paper/2508.03695/) (2025)
- [TrackingWorld: World-centric Monocular 3D Tracking of Almost All Pixels](https://mlsanity.com/paper/2512.08358/) (2025)
- [DyST-XL: Dynamic Layout Planning and Content Control for Compositional Text-to-Video Generation](https://mlsanity.com/paper/2504.15032/) (2025)
- [MonoMobility: Zero-Shot 3D Mobility Analysis from Monocular Videos](https://mlsanity.com/paper/2505.11868/) (2025)
- [Video Understanding: From Geometry and Semantics to Unified Models](https://mlsanity.com/paper/2603.17840/) (2026)
- [Online Dense Point Tracking with Streaming Memory](https://mlsanity.com/paper/2503.06471/) (2025)

## Abstract (arXiv)

Recent approaches to point tracking are able to recover the trajectory of any scene point through a large portion of a video despite the presence of occlusions. They are, however, too slow in practice to track every point observed in a single frame in a reasonable amount of time. This paper introduces DOT, a novel, simple and efficient method for solving this problem. It first extracts a small set of tracks from key regions at motion boundaries using an off-the-shelf point tracking algorithm. Given source and target frames, DOT then computes rough initial estimates of a dense flow field and visibility mask through nearest-neighbor interpolation, before refining them using a learnable optical flow estimator that explicitly handles occlusions and can be trained on synthetic data with ground-truth correspondences. We show that DOT is significantly more accurate than current optical flow techniques, outperforms sophisticated "universal" trackers like OmniMotion, and is on par with, or better than, the best point tracking algorithms like CoTracker while being at least two orders of magnitude faster. Quantitative and qualitative experiments with synthetic and real videos validate the promise of the proposed approach. Code, data, and videos showcasing the capabilities of our approach are available in the project webpage: https://16lemoing.github.io/dot .