# DATAP-SfM: Dynamic-Aware Tracking Any Point for Robust Structure from Motion in the Wild

arXiv: https://arxiv.org/abs/2411.13291 · HTML: https://mlsanity.com/paper/2411.13291/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 60 references that are in the map.
- First posted on arXiv 20 Nov 2024.

## Builds on (most cited first, 30 of 34)

- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [SuperGlue: Learning Feature Matching with Graph Neural Networks](https://mlsanity.com/paper/1911.11763/) (2019)
- [SuperPoint: Self-Supervised Interest Point Detection and Description](https://mlsanity.com/paper/1712.07629/) (2017)
- [Depth Map Prediction from a Single Image using a Multi-Scale Deep Network](https://mlsanity.com/paper/1406.2283/) (2014)
- [ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual-Inertial and Multi-Map SLAM](https://mlsanity.com/paper/2007.11898/) (2020)
- [ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras](https://mlsanity.com/paper/1610.06475/) (2016)
- [FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks](https://mlsanity.com/paper/1612.01925/) (2016)
- [MegaDepth: Learning Single-View Depth Prediction from Internet Photos](https://mlsanity.com/paper/1804.00607/) (2018)
- [DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras](https://mlsanity.com/paper/2108.10869/) (2021)
- [Direct Sparse Odometry](https://mlsanity.com/paper/1607.02565/) (2016)
- [Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture](https://mlsanity.com/paper/1411.4734/) (2014)
- [OneFormer: One Transformer to Rule Universal Image Segmentation](https://mlsanity.com/paper/2211.06220/) (2022)
- [GMFlow: Learning Optical Flow via Global Matching](https://mlsanity.com/paper/2111.13680/) (2021)
- [TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinement](https://mlsanity.com/paper/2306.08637/) (2023)
- [Consistent Video Depth Estimation](https://mlsanity.com/paper/2004.15021/) (2020)
- [Particle Video Revisited: Tracking Through Occlusions Using Point Trajectories](https://mlsanity.com/paper/2204.04153/) (2022)
- [D3VO: Deep Depth, Deep Pose and Deep Uncertainty for Monocular Visual Odometry](https://mlsanity.com/paper/2003.01060/) (2020)
- [Tracking Everything Everywhere All at Once](https://mlsanity.com/paper/2306.05422/) (2023)
- [Robust Consistent Video Depth Estimation](https://mlsanity.com/paper/2012.05901/) (2020)
- [DeepVO: Towards End-to-End Visual Odometry with Deep Recurrent Convolutional Neural Networks](https://mlsanity.com/paper/1709.08429/) (2017)
- [MiDaS v3.1 -- A Model Zoo for Robust Monocular Relative Depth Estimation](https://mlsanity.com/paper/2307.14460/) (2023)
- [Learning Two-View Correspondences and Geometry Using Order-Aware Network](https://mlsanity.com/paper/1908.04964/) (2019)
- [Self-supervised Video Object Segmentation by Motion Grouping](https://mlsanity.com/paper/2104.07658/) (2021)
- [DynaSLAM: Tracking, Mapping and Inpainting in Dynamic Scenes](https://mlsanity.com/paper/1806.05620/) (2018)
- [TartanVO: A Generalizable Learning-based VO](https://mlsanity.com/paper/2011.00359/) (2020)
- [ParticleSfM: Exploiting Dense Point Trajectories for Localizing Moving Cameras in the Wild](https://mlsanity.com/paper/2207.09137/) (2022)
- [The Emergence of Objectness: Learning Zero-Shot Segmentation from Videos](https://mlsanity.com/paper/2111.06394/) (2021)

## Built on by (most cited first, 4 of 4)

- [ViPE: Video Pose Engine for 3D Geometric Perception](https://mlsanity.com/paper/2508.10934/) (2025)
- [Dynamic Camera Poses and Where to Find Them](https://mlsanity.com/paper/2504.17788/) (2025)
- [CityGS-X: A Scalable Architecture for Efficient and Geometrically Accurate Large-Scale Scene Reconstruction](https://mlsanity.com/paper/2503.23044/) (2025)
- [Can Video Diffusion Model Reconstruct 4D Geometry?](https://mlsanity.com/paper/2503.21082/) (2025)

## Abstract (arXiv)

This paper proposes a concise, elegant, and robust pipeline to estimate smooth camera trajectories and obtain dense point clouds for casual videos in the wild. Traditional frameworks, such as ParticleSfM~citezhao2022particlesfm, address this problem by sequentially computing the optical flow between adjacent frames to obtain point trajectories. They then remove dynamic trajectories through motion segmentation and perform global bundle adjustment. However, the process of estimating optical flow between two adjacent frames and chaining the matches can introduce cumulative errors. Additionally, motion segmentation combined with single-view depth estimation often faces challenges related to scale ambiguity. To tackle these challenges, we propose a dynamic-aware tracking any point (DATAP) method that leverages consistent video depth and point tracking. Specifically, our DATAP addresses these issues by estimating dense point tracking across the video sequence and predicting the visibility and dynamics of each point. By incorporating the consistent video depth prior, the performance of motion segmentation is enhanced. With the integration of DATAP, it becomes possible to estimate and optimize all camera poses simultaneously by performing global bundle adjustments for point tracking classified as static and visible, rather than relying on incremental camera registration. Extensive experiments on dynamic sequences, e.g., Sintel and TUM RGBD dynamic sequences, and on the wild video, e.g., DAVIS, demonstrate that the proposed method achieves state-of-the-art performance in terms of camera pose estimation even in complex dynamic challenge scenes.