# TrackEverything: Long Horizon Dense Tracking via De-Duplicating 3D Scene Representations

arXiv: https://arxiv.org/abs/2609.30222 · HTML: https://mlsanity.com/paper/2609.30222/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 31 of its 34 references that are in the map.
- First posted on arXiv 24 Sep 2026.

## Builds on (most cited first, 30 of 31)

- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [DUSt3R: Geometric 3D Vision Made Easy](https://mlsanity.com/paper/2312.14132/) (2023)
- [Vision Transformer Adapter for Dense Predictions](https://mlsanity.com/paper/2205.08534/) (2022)
- [π3: Permutation-Equivariant Visual Geometry Learning](https://mlsanity.com/paper/2507.13347/) (2025)
- [TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinement](https://mlsanity.com/paper/2306.08637/) (2023)
- [3D Diffuser Actor: Policy Diffusion with 3D Scene Representations](https://mlsanity.com/paper/2402.10885/) (2024)
- [PointOdyssey: A Large-Scale Synthetic Dataset for Long-Term Point Tracking](https://mlsanity.com/paper/2307.15055/) (2023)
- [TAP-Vid: A Benchmark for Tracking Any Point in a Video](https://mlsanity.com/paper/2211.03726/) (2022)
- [Particle Video Revisited: Tracking Through Occlusions Using Point Trajectories](https://mlsanity.com/paper/2204.04153/) (2022)
- [Tracking Everything Everywhere All at Once](https://mlsanity.com/paper/2306.05422/) (2023)
- [SpatialTracker: Tracking Any 2D Pixels in 3D Space](https://mlsanity.com/paper/2404.04319/) (2024)
- [Aria Digital Twin: A New Benchmark Dataset for Egocentric 3D Machine Perception](https://mlsanity.com/paper/2306.06362/) (2023)
- [DynamicStereo: Consistent Dynamic Depth from Stereo Videos](https://mlsanity.com/paper/2305.02296/) (2023)
- [π0.7: a Steerable Generalist Robotic Foundation Model with Emergent Capabilities](https://mlsanity.com/paper/2604.15483/) (2026)
- [CoTracker3: Simpler and Better Point Tracking by Pseudo-Labelling Real Videos](https://mlsanity.com/paper/2410.11831/) (2024)
- [St4RTrack: Simultaneous 4D Reconstruction and Tracking in the World](https://mlsanity.com/paper/2504.13152/) (2025)
- [TAPVid-3D: A Benchmark for Tracking Any Point in 3D](https://mlsanity.com/paper/2407.05921/) (2024)
- [TAPIP3D: Tracking Any Point in Persistent 3D Geometry](https://mlsanity.com/paper/2504.14717/) (2025)
- [DELTA: Dense Efficient Long-range 3D Tracking for any video](https://mlsanity.com/paper/2410.24211/) (2024)
- [AllTracker: Efficient Dense Point Tracking at High Resolution](https://mlsanity.com/paper/2506.07310/) (2025)
- [Efficiently Reconstructing Dynamic Scenes One D4RT at a Time](https://mlsanity.com/paper/2512.08924/) (2025)
- [Dynamic Point Maps: A Versatile Representation for Dynamic 3D Reconstruction](https://mlsanity.com/paper/2503.16318/) (2025)
- [Any4D: Unified Feed-Forward Metric 4D Reconstruction](https://mlsanity.com/paper/2512.10935/) (2025)
- [WAFT: Warping-Alone Field Transforms for Optical Flow](https://mlsanity.com/paper/2506.21526/) (2025)
- [V-DPM: 4D Video Reconstruction with Dynamic Point Maps](https://mlsanity.com/paper/2601.09499/) (2026)
- [Trace Anything: Representing Any Video in 4D via Trajectory Fields](https://mlsanity.com/paper/2510.13802/) (2025)
- [Learning 3D Persistent Embodied World Models](https://mlsanity.com/paper/2505.05495/) (2025)
- [DriveTrack: A Benchmark for Long-Range Point Tracking in Real-World Videos](https://mlsanity.com/paper/2312.09523/) (2023)
- [Unifying 2D and 3D Vision-Language Understanding](https://mlsanity.com/paper/2503.10745/) (2025)
- [CoWTracker: Tracking by Warping instead of Correlation](https://mlsanity.com/paper/2602.04877/) (2026)

## Abstract (arXiv)

Existing point tracking models face a fundamental tradeoff: they can either track a sparse set of query points over long horizons, or track all points across only short clips. We introduce TrackEverything, a 3D point tracker that breaks this trade-off by representing videos as persistent 3D scene tracks in world coordinates. Grounded in the insight that videos are 2D projections of an underlying 3D world, TrackEverything decouples model complexity from video duration, allowing it to scale with unique physical scene geometry instead. Our approach introduces three key innovations. First, we employ a voxelization-based de-duplication mechanism at sliding-window boundaries to merge co-located tracks, preventing repeated observations of the same surface from redundantly accumulating. Second, we decompose tracking into an endpoint refiner that predicts each point's destination and static-versus-dynamic classification, followed by a lightweight trajectory refiner that decodes dense trajectories exclusively for dynamic points. Third, we propose 3D WAFT, replacing memory-prohibitive 4D correlation volumes with efficient feature sampling in the scene cloud. To the best of our knowledge, TrackEverything is the first 3D tracker capable of tracking all visible points across videos exceeding 1000 frames within 40 GB of GPU memory. On TAPVid-3D, TrackEverything outperforms all open-source all-frame dense 3D trackers by more than 20% APD on short clips, while remaining competitive with state-of-the-art sparse trackers on long sequences, despite tracking far more points.