# Towards Accurate Reconstruction of 3D Scene Shape from A Single Monocular Image

arXiv: https://arxiv.org/abs/2208.13241 · HTML: https://mlsanity.com/paper/2208.13241/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Builds on 57 of its 86 references that are in the map.
- First posted on arXiv 28 Aug 2022.

## Builds on (most cited first, 30 of 57)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [Depth Map Prediction from a Single Image using a Multi-Scale Deep Network](https://mlsanity.com/paper/1406.2283/) (2014)
- [PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization](https://mlsanity.com/paper/1905.05172/) (2019)
- [MegaDepth: Learning Single-View Depth Prediction from Internet Photos](https://mlsanity.com/paper/1804.00607/) (2018)
- [Deep Ordinal Regression Network for Monocular Depth Estimation](https://mlsanity.com/paper/1806.02446/) (2018)
- [Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images](https://mlsanity.com/paper/1804.01654/) (2018)
- [Taskonomy: Disentangling Task Transfer Learning](https://mlsanity.com/paper/1804.08328/) (2018)
- [Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling](https://mlsanity.com/paper/1610.07584/) (2016)
- [PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization](https://mlsanity.com/paper/2004.00452/) (2020)
- [Joint 2D-3D-Semantic Data for Indoor Scene Understanding](https://mlsanity.com/paper/1702.01105/) (2017)
- [Point-Voxel CNN for Efficient 3D Deep Learning](https://mlsanity.com/paper/1907.03739/) (2019)
- [GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose](https://mlsanity.com/paper/1803.02276/) (2018)
- [Sparsity Invariant CNNs](https://mlsanity.com/paper/1708.06500/) (2017)
- [Generating 3D faces using Convolutional Mesh Autoencoders](https://mlsanity.com/paper/1807.10267/) (2018)
- [What Do Single-view 3D Reconstruction Networks Learn?](https://mlsanity.com/paper/1905.03678/) (2019)
- [Unsupervised Scale-consistent Depth and Ego-motion Learning from Monocular Video](https://mlsanity.com/paper/1908.10553/) (2019)
- [DIODE: A Dense Indoor and Outdoor DEpth Dataset](https://mlsanity.com/paper/1908.00463/) (2019)
- [Enforcing geometric constraints of virtual normal for depth prediction](https://mlsanity.com/paper/1907.12209/) (2019)
- [3D Photography using Context-aware Layered Depth Inpainting](https://mlsanity.com/paper/2004.04727/) (2020)
- [DeMoN: Depth and Motion Network for Learning Monocular Stereo](https://mlsanity.com/paper/1612.02401/) (2016)
- [Non-Local Spatial Propagation Network for Depth Completion](https://mlsanity.com/paper/2007.10042/) (2020)
- [Single-Image Depth Perception in the Wild](https://mlsanity.com/paper/1604.03901/) (2016)
- [Learning Depth from Single Monocular Images Using Deep Convolutional Neural Fields](https://mlsanity.com/paper/1502.07411/) (2015)

## Built on by (most cited first, 30 of 42)

- [DUSt3R: Geometric 3D Vision Made Easy](https://mlsanity.com/paper/2312.14132/) (2023)
- [Metric3Dv2: A Versatile Monocular Geometric Foundation Model for Zero-shot Metric Depth and Surface Normal Estimation](https://mlsanity.com/paper/2404.15506/) (2024)
- [Metric3D: Towards Zero-shot Metric 3D Prediction from A Single Image](https://mlsanity.com/paper/2307.10984/) (2023)
- [MoGe: Unlocking Accurate Monocular Geometry Estimation for Open-Domain Images with Optimal Training Supervision](https://mlsanity.com/paper/2410.19115/) (2024)
- [DepthSplat: Connecting Gaussian Splatting and Depth](https://mlsanity.com/paper/2410.13862/) (2024)
- [MoGe-2: Accurate Monocular Geometry with Metric Scale and Sharp Details](https://mlsanity.com/paper/2507.02546/) (2025)
- [NoPe-NeRF: Optimising Neural Radiance Field with No Pose Prior](https://mlsanity.com/paper/2212.07388/) (2022)
- [ZeroNVS: Zero-Shot 360-Degree View Synthesis from a Single Image](https://mlsanity.com/paper/2310.17994/) (2023)
- [Pow3R: Empowering Unconstrained 3D Reconstruction with Camera and Scene Priors](https://mlsanity.com/paper/2503.17316/) (2025)
- [Hierarchical Normalization for Robust Monocular Depth Estimation](https://mlsanity.com/paper/2210.09670/) (2022)
- [GeometryCrafter: Consistent Geometry Estimation for Open-world Videos with Diffusion Priors](https://mlsanity.com/paper/2504.01016/) (2025)
- [SCADE: NeRFs from Space Carving with Ambiguity-Aware Depth Estimates](https://mlsanity.com/paper/2303.13582/) (2023)
- [Adaptive Fusion of Single-View and Multi-View Depth for Autonomous Driving](https://mlsanity.com/paper/2403.07535/) (2024)
- [Learning to Fuse Monocular and Multi-view Cues for Multi-frame Depth Estimation in Dynamic Scenes](https://mlsanity.com/paper/2304.08993/) (2023)
- [MonSter++: Unified Stereo Matching, Multi-view Stereo, and Real-time Stereo with Monodepth Priors](https://mlsanity.com/paper/2501.08643/) (2025)
- [FrozenRecon: Pose-free 3D Scene Reconstruction with Frozen Depth Models](https://mlsanity.com/paper/2308.05733/) (2023)
- [Know Your Neighbors: Improving Single-View Reconstruction via Spatial Vision-Language Reasoning](https://mlsanity.com/paper/2404.03658/) (2024)
- [Depth Anything at Any Condition](https://mlsanity.com/paper/2507.01634/) (2025)
- [Depth-Driven Geometric Prompt Learning for Laparoscopic Liver Landmark Detection](https://mlsanity.com/paper/2406.17858/) (2024)
- [Depth Awakens: A Depth-perceptual Attention Fusion Network for RGB-D Camouflaged Object Detection](https://mlsanity.com/paper/2405.05614/) (2024)
- [Boost 3D Reconstruction using Diffusion-based Monocular Camera Calibration](https://mlsanity.com/paper/2411.17240/) (2024)
- [QuartDepth: Post-Training Quantization for Real-Time Depth Estimation on the Edge](https://mlsanity.com/paper/2503.16709/) (2025)
- [The Fourth Monocular Depth Estimation Challenge](https://mlsanity.com/paper/2504.17787/) (2025)
- [E3D-Bench: A Benchmark for End-to-End 3D Geometric Foundation Models](https://mlsanity.com/paper/2506.01933/) (2025)
- [Robust Geometry-Preserving Depth Estimation Using Differentiable Rendering](https://mlsanity.com/paper/2309.09724/) (2023)
- [GRIN: Zero-Shot Metric Depth with Pixel-Level Diffusion](https://mlsanity.com/paper/2409.09896/) (2024)
- [Scale Propagation Network for Generalizable Depth Completion](https://mlsanity.com/paper/2410.18408/) (2024)
- [Always Clear Depth: Robust Monocular Depth Estimation under Adverse Weather](https://mlsanity.com/paper/2505.12199/) (2025)
- [Towards In-the-wild 3D Plane Reconstruction from a Single Image](https://mlsanity.com/paper/2506.02493/) (2025)
- [RIDERS: Radar-Infrared Depth Estimation for Robust Sensing](https://mlsanity.com/paper/2402.02067/) (2024)

## Abstract (arXiv)

Despite significant progress made in the past few years, challenges remain for depth estimation using a single monocular image. First, it is nontrivial to train a metric-depth prediction model that can generalize well to diverse scenes mainly due to limited training data. Thus, researchers have built large-scale relative depth datasets that are much easier to collect. However, existing relative depth estimation models often fail to recover accurate 3D scene shapes due to the unknown depth shift caused by training with the relative depth data. We tackle this problem here and attempt to estimate accurate scene shapes by training on large-scale relative depth data, and estimating the depth shift. To do so, we propose a two-stage framework that first predicts depth up to an unknown scale and shift from a single monocular image, and then exploits 3D point cloud data to predict the depth shift and the camera's focal length that allow us to recover 3D scene shapes. As the two modules are trained separately, we do not need strictly paired training data. In addition, we propose an image-level normalized regression loss and a normal-based geometry loss to improve training with relative depth annotation. We test our depth model on nine unseen datasets and achieve state-of-the-art performance on zero-shot evaluation. Code is available at: https://git.io/Depth