# TrackCraft3R: Repurposing Video Diffusion Transformers for Dense 3D Tracking arXiv: https://arxiv.org/abs/2605.12587 ยท HTML: https://mlsanity.com/paper/2605.12587/ ## Key facts - Cited by 5 papers in the map (data through 14 Oct 2026). - Placed in Video generation, one of 14 directions in Diffusion models. - Builds on 81 of its 87 references that are in the map. - First posted on arXiv 12 May 2026. ## Placed in - Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/) ## Builds on (most cited first, 30 of 81) - [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022) - [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021) - [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017) - [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025) - [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023) - [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019) - [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024) - [HunyuanVideo: A Systematic Framework For Large Video Generative Models](https://mlsanity.com/paper/2412.03603/) (2024) - [DUSt3R: Geometric 3D Vision Made Easy](https://mlsanity.com/paper/2312.14132/) (2023) - [Objaverse: A Universe of Annotated 3D Objects](https://mlsanity.com/paper/2212.08051/) (2022) - [VGGT: Visual Geometry Grounded Transformer](https://mlsanity.com/paper/2503.11651/) (2025) - [Depth Anything V2](https://mlsanity.com/paper/2406.09414/) (2024) - [Grounding Image Matching in 3D with MASt3R](https://mlsanity.com/paper/2406.09756/) (2024) - [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017) - [ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenes](https://mlsanity.com/paper/2308.11417/) (2023) - [MegaDepth: Learning Single-View Depth Prediction from Internet Photos](https://mlsanity.com/paper/1804.00607/) (2018) - [Depth Anything 3: Recovering the Visual Space from Any Views](https://mlsanity.com/paper/2511.10647/) (2025) - [Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction](https://mlsanity.com/paper/2109.00512/) (2021) - [Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding](https://mlsanity.com/paper/2011.02523/) (2020) - [DL3DV-10K: A Large-Scale Scene Dataset for Deep Learning-based 3D Vision](https://mlsanity.com/paper/2312.16256/) (2023) - [CoTracker: It is Better to Track Together](https://mlsanity.com/paper/2307.07635/) (2023) - [Habitat 2.0: Training Home Assistants to Rearrange their Habitat](https://mlsanity.com/paper/2106.14405/) (2021) - [TartanAir: A Dataset to Push the Limits of Visual SLAM](https://mlsanity.com/paper/2003.14338/) (2020) - [BlendedMVS: A Large-scale Dataset for Generalized Multi-view Stereo Networks](https://mlsanity.com/paper/1911.10127/) (2019) - [DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors](https://mlsanity.com/paper/2310.12190/) (2023) - [Emergent Correspondence from Image Diffusion](https://mlsanity.com/paper/2306.03881/) (2023) - [Kubric: A scalable dataset generator](https://mlsanity.com/paper/2203.03570/) (2022) - [Fast3R: Towards 3D Reconstruction of 1000+ Images in One Forward Pass](https://mlsanity.com/paper/2501.13928/) (2025) - [Virtual KITTI 2](https://mlsanity.com/paper/2001.10773/) (2020) - [Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation](https://mlsanity.com/paper/2312.02145/) (2023) ## Built on by (most cited first, 5 of 5) - [MotionForesight: Re-purposing Video Models for Future 3D Scene-Flow Prediction](https://mlsanity.com/paper/2607.16192/) (2026) - [Track, Articulate, Act: Generating Articulation from Casual Human Videos](https://mlsanity.com/paper/2609.19119/) (2026) - [3D Point Tracking with State Space Models](https://mlsanity.com/paper/2609.34035/) (2026) - [CASTER: Formalizing Zero-Shot Manipulation from Synthetic Videos as Constrained Optimization](https://mlsanity.com/paper/2610.12567/) (2026) - [Cepheid: Decoupling Scale from Structure for Cross-Family Motion Completion](https://mlsanity.com/paper/2610.18871/) (2026) ## Abstract (arXiv) Dense 3D tracking from monocular video is fundamental to dynamic scene understanding. While recent 3D foundation models provide reliable per-frame geometry, recovering object motion in this geometry remains challenging and benefits from strong motion priors learned from real-world videos. Existing 3D trackers either follow iterative paradigms trained from scratch on synthetic data or fine-tune 3D reconstruction models learned from static multi-view images, both lacking real-world motion priors. Pre-trained video diffusion transformers (video DiTs) offer rich spatio-temporal priors from internet-scale videos, making them a promising foundation for 3D tracking. However, their frame-anchored formulation, which generates each frame's content, is fundamentally mismatched with reference-anchored dense 3D tracking, which must follow the same physical points from a reference frame across time. We present TrackCraft3R, the first method to repurpose a video DiT as a feed-forward dense 3D tracker. Given a monocular video and its frame-anchored reconstruction pointmap, TrackCraft3R predicts a reference-anchored tracking pointmap that follows every pixel of the first frame across time in a single forward pass, along with its visibility. We achieve this through two designs: (i) a dual-latent representation that uses per-frame geometry latents and reference-anchored track latents as dense queries, and (ii) temporal RoPE alignment, which specifies the target timestamp of each track latent. Together, these designs convert the per-frame generative paradigm of video DiTs into a reference-anchored tracking formulation with LoRA fine-tuning. TrackCraft3R achieves state-of-the-art performance on standard sparse and dense 3D tracking benchmarks, while running 1.3x faster and using 4.6x less peak memory than the strongest prior method. We further demonstrate robustness to large motions and long videos.