# CoReNet: Coherent 3D scene reconstruction from a single RGB image

arXiv: https://arxiv.org/abs/2004.12989 · HTML: https://mlsanity.com/paper/2004.12989/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Apr 2020.

## Built on by (most cited first, 30 of 45)

- [MonoScene: Monocular 3D Semantic Scene Completion](https://mlsanity.com/paper/2112.00726/) (2021)
- [VoxFormer: Sparse Voxel Transformer for Camera-based 3D Semantic Scene Completion](https://mlsanity.com/paper/2302.12251/) (2023)
- [SAM 3D: 3Dfy Anything in Images](https://mlsanity.com/paper/2511.16624/) (2025)
- [Mix3D: Out-of-Context Data Augmentation for 3D Scenes](https://mlsanity.com/paper/2110.02210/) (2021)
- [3D Semantic Scene Completion: a Survey](https://mlsanity.com/paper/2103.07466/) (2021)
- [Panoptic 3D Scene Reconstruction From a Single RGB Image](https://mlsanity.com/paper/2111.02444/) (2021)
- [NAVI: Category-Agnostic Image Collections with High-Quality 3D Shape and Pose Annotations](https://mlsanity.com/paper/2306.09109/) (2023)
- [SceneRF: Self-Supervised Monocular 3D Scene Reconstruction with Radiance Fields](https://mlsanity.com/paper/2212.02501/) (2022)
- [Indoor Scene Generation from a Collection of Semantic-Segmented Depth Images](https://mlsanity.com/paper/2108.09022/) (2021)
- [Learning 3D Scene Priors with 2D Supervision](https://mlsanity.com/paper/2211.14157/) (2022)
- [Pose2Room: Understanding 3D Scenes from Human Activities](https://mlsanity.com/paper/2112.03030/) (2021)
- [Voxel-based 3D Detection and Reconstruction of Multiple Objects from a Single Image](https://mlsanity.com/paper/2111.03098/) (2021)
- [BUOL: A Bottom-Up Framework with Occupancy-aware Lifting for Panoptic 3D Scene Reconstruction From A Single Image](https://mlsanity.com/paper/2306.00965/) (2023)
- [CoCoNets: Continuous Contrastive 3D Scene Representations](https://mlsanity.com/paper/2104.03851/) (2021)
- [RayTran: 3D pose estimation and shape reconstruction of multiple objects from videos with ray-traced transformers](https://mlsanity.com/paper/2203.13296/) (2022)
- [SketchSampler: Sketch-based 3D Reconstruction via View-dependent Depth Sampling](https://mlsanity.com/paper/2208.06880/) (2022)
- [CAD-Estate: Large-scale CAD Model Annotation in RGB Videos](https://mlsanity.com/paper/2306.09011/) (2023)
- [Shape from Blur: Recovering Textured 3D Shape and Motion of Fast Moving Objects](https://mlsanity.com/paper/2106.08762/) (2021)
- [Single-view 3D Mesh Reconstruction for Seen and Unseen Categories](https://mlsanity.com/paper/2208.02676/) (2022)
- [3D-RE-GEN: 3D Reconstruction of Indoor Scenes with a Generative Framework](https://mlsanity.com/paper/2512.17459/) (2025)
- [Coherent 3D Scene Diffusion From a Single RGB Image](https://mlsanity.com/paper/2412.10294/) (2024)
- [Neural Rendering in a Room: Amodal 3D Understanding and Free-Viewpoint Rendering for the Closed Scene Composed of Pre-Captured Objects](https://mlsanity.com/paper/2205.02714/) (2022)
- [Ray-ONet: Efficient 3D Reconstruction From A Single RGB Image](https://mlsanity.com/paper/2107.01899/) (2021)
- [Estimating Generic 3D Room Structures from 2D Annotations](https://mlsanity.com/paper/2306.09077/) (2023)
- [REST3D: Reconstructing Physically Stable 3D Scenes from a Single Image](https://mlsanity.com/paper/2605.30338/) (2026)
- [Pose Adaptive Dual Mixup for Few-Shot Single-View 3D Reconstruction](https://mlsanity.com/paper/2112.12484/) (2021)
- [SimuScene: Simulation-Ready Compositional 3D Scene Reconstruction from a Single Image](https://mlsanity.com/paper/2606.03994/) (2026)
- [Modern Augmented Reality: Applications, Trends, and Future Directions](https://mlsanity.com/paper/2202.09450/) (2022)
- [VPFusion: Joint 3D Volume and Pixel-Aligned Feature Fusion for Single and Multi-view 3D Reconstruction](https://mlsanity.com/paper/2203.07553/) (2022)
- [RIC: Rotate-Inpaint-Complete for Generalizable Scene Reconstruction](https://mlsanity.com/paper/2307.11932/) (2023)

## Abstract (arXiv)

Advances in deep learning techniques have allowed recent work to reconstruct the shape of a single object given only one RBG image as input. Building on common encoder-decoder architectures for this task, we propose three extensions: (1) ray-traced skip connections that propagate local 2D information to the output 3D volume in a physically correct manner; (2) a hybrid 3D volume representation that enables building translation equivariant models, while at the same time encoding fine object details without an excessive memory footprint; (3) a reconstruction loss tailored to capture overall object geometry. Furthermore, we adapt our model to address the harder task of reconstructing multiple objects from a single image. We reconstruct all objects jointly in one pass, producing a coherent reconstruction, where all objects live in a single consistent 3D coordinate frame relative to the camera and they do not intersect in 3D space. We also handle occlusions and resolve them by hallucinating the missing object parts in the 3D volume. We validate the impact of our contributions experimentally both on synthetic data from ShapeNet as well as real images from Pix3D. Our method improves over the state-of-the-art single-object methods on both datasets. Finally, we evaluate performance quantitatively on multiple object reconstruction with synthetic scenes assembled from ShapeNet objects.