# Efficiently Reconstructing Dynamic Scenes One D4RT at a Time

arXiv: https://arxiv.org/abs/2512.08924 · HTML: https://mlsanity.com/paper/2512.08924/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 62 references that are in the map.
- First posted on arXiv 9 Dec 2025.

## Builds on (most cited first, 30 of 48)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [DUSt3R: Geometric 3D Vision Made Easy](https://mlsanity.com/paper/2312.14132/) (2023)
- [VGGT: Visual Geometry Grounded Transformer](https://mlsanity.com/paper/2503.11651/) (2025)
- [Depth Map Prediction from a Single Image using a Multi-Scale Deep Network](https://mlsanity.com/paper/1406.2283/) (2014)
- [ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenes](https://mlsanity.com/paper/2308.11417/) (2023)
- [Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction](https://mlsanity.com/paper/2109.00512/) (2021)
- [Continuous 3D Perception Model with Persistent State](https://mlsanity.com/paper/2501.12387/) (2025)
- [VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking](https://mlsanity.com/paper/2303.16727/) (2023)
- [TartanAir: A Dataset to Push the Limits of Visual SLAM](https://mlsanity.com/paper/2003.14338/) (2020)
- [BlendedMVS: A Large-scale Dataset for Generalized Multi-view Stereo Networks](https://mlsanity.com/paper/1911.10127/) (2019)
- [MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motion](https://mlsanity.com/paper/2410.03825/) (2024)
- [Kubric: A scalable dataset generator](https://mlsanity.com/paper/2203.03570/) (2022)
- [Fast3R: Towards 3D Reconstruction of 1000+ Images in One Forward Pass](https://mlsanity.com/paper/2501.13928/) (2025)
- [Virtual Worlds as Proxy for Multi-Object Tracking Analysis](https://mlsanity.com/paper/1605.06457/) (2016)
- [π3: Permutation-Equivariant Visual Geometry Learning](https://mlsanity.com/paper/2507.13347/) (2025)
- [MapAnything: Universal Feed-Forward Metric 3D Reconstruction](https://mlsanity.com/paper/2509.13414/) (2025)
- [TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinement](https://mlsanity.com/paper/2306.08637/) (2023)
- [DeepMVS: Learning Multi-view Stereopsis](https://mlsanity.com/paper/1804.00650/) (2018)
- [PointOdyssey: A Large-Scale Synthetic Dataset for Long-Term Point Tracking](https://mlsanity.com/paper/2307.15055/) (2023)
- [TAP-Vid: A Benchmark for Tracking Any Point in a Video](https://mlsanity.com/paper/2211.03726/) (2022)
- [MegaSaM: Accurate, Fast, and Robust Structure and Motion from Casual Dynamic Videos](https://mlsanity.com/paper/2412.04463/) (2024)
- [Scene Representation Transformer: Geometry-Free Novel View Synthesis Through Set-Latent Scene Representations](https://mlsanity.com/paper/2111.13152/) (2021)
- [Particle Video Revisited: Tracking Through Occlusions Using Point Trajectories](https://mlsanity.com/paper/2204.04153/) (2022)
- [ReFusion: 3D Reconstruction in Dynamic Environments for RGB-D Cameras Exploiting Residuals](https://mlsanity.com/paper/1905.02082/) (2019)

## Built on by (most cited first, 30 of 54)

- [Image Generators are Generalist Vision Learners](https://mlsanity.com/paper/2604.20329/) (2026)
- [VGGT-Omega](https://mlsanity.com/paper/2605.15195/) (2026)
- [4RC: 4D Reconstruction via Conditional Querying Anytime and Anywhere](https://mlsanity.com/paper/2602.10094/) (2026)
- [Video Generation Models are General-Purpose Vision Learners](https://mlsanity.com/paper/2607.09024/) (2026)
- [In Pursuit of Pixel Supervision for Visual Pre-training](https://mlsanity.com/paper/2512.15715/) (2025)
- [Track4World: Feedforward World-centric Dense 3D Tracking of All Pixels](https://mlsanity.com/paper/2603.02573/) (2026)
- [Flow4R: Unifying 4D Reconstruction and Tracking with Scene Flow](https://mlsanity.com/paper/2602.14021/) (2026)
- [RecGen3D: Reconstruction-Guided 3D Generation in a Shared Canonical Space](https://mlsanity.com/paper/2604.01479/) (2026)
- [SM4RT: Learning Structured Motion Geometry for 4D Reconstruction](https://mlsanity.com/paper/2607.22534/) (2026)
- [OmniStream: Mastering Perception, Reconstruction and Action in Continuous Streams](https://mlsanity.com/paper/2603.12265/) (2026)
- [Syn4D: A Multiview Synthetic 4D Dataset](https://mlsanity.com/paper/2605.05207/) (2026)
- [OmniX: Any-view and Any-time 4D Reconstruction via Feed-forward Trajectory Fields](https://mlsanity.com/paper/2607.10840/) (2026)
- [SynthVerse: A Large-Scale Diverse Synthetic Dataset for Point Tracking](https://mlsanity.com/paper/2602.04441/) (2026)
- [PAS3R: Pose-Adaptive Streaming 3D Reconstruction for Long Video Sequences](https://mlsanity.com/paper/2603.21436/) (2026)
- [LSRM: High-Fidelity Object-Centric Reconstruction via Scaled Context Windows](https://mlsanity.com/paper/2604.05182/) (2026)
- [Lift4D: Harmonizing Single-View 3D Estimation for 4D Reconstruction In-the-Wild](https://mlsanity.com/paper/2606.23688/) (2026)
- [4DGS-WAM: Bridging Past and Future with an Object-Centric World Action Model based on 4D Gaussian Splatting](https://mlsanity.com/paper/2608.25956/) (2026)
- [Surflo: Consistent 3D Surface Flow Model with Global State](https://mlsanity.com/paper/2606.13644/) (2026)
- [Gen4U: Unifying Video Generation and Understanding via Diffusion](https://mlsanity.com/paper/2607.06856/) (2026)
- [SeeSE3: Emergence of 3D Space in Vision Features](https://mlsanity.com/paper/2607.14228/) (2026)
- [UniQuery4R: Unified 4D Scene Reconstruction from a Single Query](https://mlsanity.com/paper/2608.17283/) (2026)
- [TAPVid-MV: A Benchmark for Tracking Any Point in 3D Across Multiple Views](https://mlsanity.com/paper/2609.01899/) (2026)
- [Point4D: Long-range 4D Motion Reconstruction](https://mlsanity.com/paper/2609.09145/) (2026)
- [AgentSTAR: Agentic Shape Tracking and Reconstruction from Monocular Videos](https://mlsanity.com/paper/2609.24487/) (2026)
- [ReFlow: Self-correction Motion Learning for Dynamic Scene Reconstruction](https://mlsanity.com/paper/2604.01561/) (2026)
- [QueST: Persistent Queries as Semantic Monitors for Drift Suppression in Long-Horizon Tracking](https://mlsanity.com/paper/2605.09513/) (2026)
- [LongDPM: Overlap-Aware 4D Reconstruction from Long Monocular Videos](https://mlsanity.com/paper/2605.17303/) (2026)
- [DynaTok: Token-Based 4D Reconstruction from Partial Point Clouds](https://mlsanity.com/paper/2606.12189/) (2026)
- [HAT-4D: Lifting Monocular Video for 4D Multi-Object Interactions via Human-Agent Collaboration](https://mlsanity.com/paper/2606.28215/) (2026)
- [Geometry Beats Estimated Depth: RGB-Only Multi-Camera 3D Tracking under Sim2Real](https://mlsanity.com/paper/2608.07579/) (2026)

## Abstract (arXiv)

Understanding and reconstructing the complex geometry and motion of dynamic scenes from video remains a formidable challenge in computer vision. This paper introduces D4RT, a simple yet powerful feedforward model designed to efficiently solve this task. D4RT utilizes a unified transformer architecture to jointly infer depth, spatio-temporal correspondence, and full camera parameters from a single video. Its core innovation is a novel querying mechanism that sidesteps the heavy computation of dense, per-frame decoding and the complexity of managing multiple, task-specific decoders. Our decoding interface allows the model to independently and flexibly probe the 3D position of any point in space and time. The result is a lightweight and highly scalable method that enables remarkably efficient training and inference. We demonstrate that our approach sets a new state of the art, outperforming previous methods across a wide spectrum of 4D reconstruction tasks. We refer to the project webpage for animated results: https://d4rt-paper.github.io/.