# R3D3: Dense 3D Reconstruction of Dynamic Scenes from Multiple Cameras

arXiv: https://arxiv.org/abs/2308.14713 · HTML: https://mlsanity.com/paper/2308.14713/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 66 references that are in the map.
- First posted on arXiv 28 Aug 2023.

## Builds on (most cited first, 30 of 46)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [ORB-SLAM: a Versatile and Accurate Monocular SLAM System](https://mlsanity.com/paper/1502.00956/) (2015)
- [ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual-Inertial and Multi-Map SLAM](https://mlsanity.com/paper/2007.11898/) (2020)
- [Unsupervised Monocular Depth Estimation with Left-Right Consistency](https://mlsanity.com/paper/1609.03677/) (2016)
- [ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras](https://mlsanity.com/paper/1610.06475/) (2016)
- [MVSNet: Depth Inference for Unstructured Multi-view Stereo](https://mlsanity.com/paper/1804.02505/) (2018)
- [Unsupervised Learning of Depth and Ego-Motion from Video](https://mlsanity.com/paper/1704.07813/) (2017)
- [DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras](https://mlsanity.com/paper/2108.10869/) (2021)
- [Direct Sparse Odometry](https://mlsanity.com/paper/1607.02565/) (2016)
- [3D Packing for Self-Supervised Monocular Depth Estimation](https://mlsanity.com/paper/1905.02693/) (2019)
- [Cascade Cost Volume for High-Resolution Multi-View Stereo and Stereo Matching](https://mlsanity.com/paper/1912.06378/) (2019)
- [Virtual KITTI 2](https://mlsanity.com/paper/2001.10773/) (2020)
- [GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose](https://mlsanity.com/paper/1803.02276/) (2018)
- [DeepMVS: Learning Multi-view Stereopsis](https://mlsanity.com/paper/1804.00650/) (2018)
- [Recurrent MVSNet for High-resolution Multi-view Stereo Depth Inference](https://mlsanity.com/paper/1902.10556/) (2019)
- [D3VO: Deep Depth, Deep Pose and Deep Uncertainty for Monocular Visual Odometry](https://mlsanity.com/paper/2003.01060/) (2020)
- [Learning a Multi-View Stereo Machine](https://mlsanity.com/paper/1708.05375/) (2017)
- [The Temporal Opportunist: Self-Supervised Multi-Frame Monocular Depth](https://mlsanity.com/paper/2104.14540/) (2021)
- [DeepV2D: Video to Depth with Differentiable Structure from Motion](https://mlsanity.com/paper/1812.04605/) (2018)
- [Kimera: an Open-Source Library for Real-Time Metric-Semantic Localization and Mapping](https://mlsanity.com/paper/1910.02490/) (2019)
- [BA-Net: Dense Bundle Adjustment Network](https://mlsanity.com/paper/1806.04807/) (2018)
- [Unsupervised Learning of Monocular Depth Estimation and Visual Odometry with Deep Feature Reconstruction](https://mlsanity.com/paper/1803.03893/) (2018)
- [Learning Depth from Monocular Videos using Direct Methods](https://mlsanity.com/paper/1712.00175/) (2017)
- [CNN-SLAM: Real-time dense monocular SLAM with learned depth prediction](https://mlsanity.com/paper/1704.03489/) (2017)

## Built on by (most cited first, 30 of 39)

- [DUSt3R: Geometric 3D Vision Made Easy](https://mlsanity.com/paper/2312.14132/) (2023)
- [DrivingGaussian: Composite Gaussian Splatting for Surrounding Dynamic Autonomous Driving Scenes](https://mlsanity.com/paper/2312.07920/) (2023)
- [MV-DUSt3R+: Single-Stage Scene Reconstruction from Sparse Views In 2 Seconds](https://mlsanity.com/paper/2412.06974/) (2024)
- [SelfOcc: Self-Supervised Vision-Based 3D Occupancy Prediction](https://mlsanity.com/paper/2311.12754/) (2023)
- [GaussianOcc: Fully Self-supervised and Efficient 3D Occupancy Estimation with Gaussian Splatting](https://mlsanity.com/paper/2408.11447/) (2024)
- [OccNeRF: Advancing 3D Occupancy Prediction in LiDAR-Free Environments](https://mlsanity.com/paper/2312.09243/) (2023)
- [Dynamic 3D Gaussian Fields for Urban Areas](https://mlsanity.com/paper/2406.03175/) (2024)
- [Multi-Level Neural Scene Graphs for Dynamic Urban Environments](https://mlsanity.com/paper/2404.00168/) (2024)
- [DVGT: Driving Visual Geometry Transformer](https://mlsanity.com/paper/2512.16919/) (2025)
- [DROID-Splat: Combining end-to-end SLAM with 3D Gaussian Splatting](https://mlsanity.com/paper/2411.17660/) (2024)
- [InverseMatrixVT3D: An Efficient Projection Matrix-Based Approach for 3D Occupancy Prediction](https://mlsanity.com/paper/2401.12422/) (2024)
- [Know Your Neighbors: Improving Single-View Reconstruction via Spatial Vision-Language Reasoning](https://mlsanity.com/paper/2404.03658/) (2024)
- [SurroundSDF: Implicit 3D Scene Understanding Based on Signed Distance Field](https://mlsanity.com/paper/2403.14366/) (2024)
- [M2Depth: Self-supervised Two-Frame Multi-camera Metric Depth Estimation](https://mlsanity.com/paper/2405.02004/) (2024)
- [ViPOcc: Leveraging Visual Priors from Vision Foundation Models for Single-View 3D Occupancy Prediction](https://mlsanity.com/paper/2412.11210/) (2024)
- [Towards Cross-View-Consistent Self-Supervised Surround Depth Estimation](https://mlsanity.com/paper/2407.04041/) (2024)
- [RCDN: Towards Robust Camera-Insensitivity Collaborative Perception via Dynamic Feature-based 3D Neural Modeling](https://mlsanity.com/paper/2405.16868/) (2024)
- [Towards Scale-Aware Full Surround Monodepth with Transformers](https://mlsanity.com/paper/2407.10406/) (2024)
- [Geometry-Grounded Gaussian Splatting](https://mlsanity.com/paper/2601.17835/) (2026)
- [TruckDrive: Long-Range Autonomous Highway Driving Dataset](https://mlsanity.com/paper/2603.02413/) (2026)
- [SS3DM: Benchmarking Street-View Surface Reconstruction with a Synthetic 3D Mesh Dataset](https://mlsanity.com/paper/2410.21739/) (2024)
- [Inverse++: Vision-Centric 3D Semantic Occupancy Prediction Assisted with 3D Object Detection](https://mlsanity.com/paper/2504.04732/) (2025)
- [DrivingGaussian++: Towards Realistic Reconstruction and Editable Simulation for Surrounding Dynamic Driving Scenes](https://mlsanity.com/paper/2508.20965/) (2025)
- [MGCA-Net: Multi-Graph Contextual Attention Network for Two-View Correspondence Learning](https://mlsanity.com/paper/2512.23369/) (2025)
- [Self-Supervised Geometry-Guided Initialization for Robust Monocular Visual Odometry](https://mlsanity.com/paper/2406.00929/) (2024)
- [Semi-SMD: Semi-Supervised Metric Depth Estimation via Surrounding Cameras for Autonomous Driving](https://mlsanity.com/paper/2503.19713/) (2025)
- [CylinderDepth: Cylindrical Spatial Attention for Multi-View Consistent Self-Supervised Surround Depth Estimation](https://mlsanity.com/paper/2511.16428/) (2025)
- [Dense Dynamic Scene Reconstruction and Camera Pose Estimation from Multi-View Videos](https://mlsanity.com/paper/2603.12064/) (2026)
- [A Pose-only Geometric Constraint for Multi-Camera Pose Adjustment](https://mlsanity.com/paper/2604.23704/) (2026)
- [SurroundNEXO: Ego-Centric Metric Bridging for Spatially Consistent Geometry in Autonomous Driving](https://mlsanity.com/paper/2606.16960/) (2026)

## Abstract (arXiv)

Dense 3D reconstruction and ego-motion estimation are key challenges in autonomous driving and robotics. Compared to the complex, multi-modal systems deployed today, multi-camera systems provide a simpler, low-cost alternative. However, camera-based 3D reconstruction of complex dynamic scenes has proven extremely difficult, as existing solutions often produce incomplete or incoherent results. We propose R3D3, a multi-camera system for dense 3D reconstruction and ego-motion estimation. Our approach iterates between geometric estimation that exploits spatial-temporal information from multiple cameras, and monocular depth refinement. We integrate multi-camera feature correlation and dense bundle adjustment operators that yield robust geometric depth and pose estimates. To improve reconstruction where geometric depth is unreliable, e.g. for moving objects or low-textured regions, we introduce learnable scene priors via a depth refinement network. We show that this design enables a dense, consistent 3D reconstruction of challenging, dynamic outdoor environments. Consequently, we achieve state-of-the-art dense depth prediction on the DDAD and NuScenes benchmarks.