# Deep Reinforcement Learning of Volume-guided Progressive View Inpainting for 3D Point Scene Completion from a Single Depth Image

arXiv: https://arxiv.org/abs/1903.04019 · HTML: https://mlsanity.com/paper/1903.04019/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Mar 2019.

## Built on by (most cited first, 15 of 15)

- [DiffuScene: Denoising Diffusion Models for Generative Indoor Scene Synthesis](https://mlsanity.com/paper/2303.14207/) (2023)
- [3D Semantic Scene Completion: a Survey](https://mlsanity.com/paper/2103.07466/) (2021)
- [SemCity: Semantic Scene Generation with Triplane Diffusion](https://mlsanity.com/paper/2403.07773/) (2024)
- [RGBD2: Generative Scene Synthesis via Incremental View Inpainting using RGBD Diffusion Models](https://mlsanity.com/paper/2212.05993/) (2022)
- [Sketching without Worrying: Noise-Tolerant Sketch-Based Image Retrieval](https://mlsanity.com/paper/2203.14817/) (2022)
- [Point Scene Understanding via Disentangled Instance Mesh Reconstruction](https://mlsanity.com/paper/2203.16832/) (2022)
- [SSR-2D: Semantic 3D Scene Reconstruction from 2D Images](https://mlsanity.com/paper/2302.03640/) (2023)
- [Point Cloud Scene Completion with Joint Color and Semantic Estimation from Single RGB-D Image](https://mlsanity.com/paper/2210.05891/) (2022)
- [Learning the Next Best View for 3D Point Clouds via Topological Features](https://mlsanity.com/paper/2103.02789/) (2021)
- [Explore Contextual Information for 3D Scene Graph Generation](https://mlsanity.com/paper/2210.06240/) (2022)
- [Touch2Shape: Touch-Conditioned 3D Diffusion for Shape Exploration and Reconstruction](https://mlsanity.com/paper/2505.13091/) (2025)
- [Learning to Predict Scene-Level Implicit 3D from Posed RGBD Data](https://mlsanity.com/paper/2306.08671/) (2023)
- [Shape Anchor Guided Holistic Indoor Scene Understanding](https://mlsanity.com/paper/2309.11133/) (2023)
- [Towards Practicality of Sketch-Based Visual Understanding](https://mlsanity.com/paper/2210.15146/) (2022)
- [AIDOVECL: AI-generated Dataset of Outpainted Vehicles for Eye-level Classification and Localization](https://mlsanity.com/paper/2410.24116/) (2024)

## Abstract (arXiv)

We present a deep reinforcement learning method of progressive view inpainting for 3D point scene completion under volume guidance, achieving high-quality scene reconstruction from only a single depth image with severe occlusion. Our approach is end-to-end, consisting of three modules: 3D scene volume reconstruction, 2D depth map inpainting, and multi-view selection for completion. Given a single depth image, our method first goes through the 3D volume branch to obtain a volumetric scene reconstruction as a guide to the next view inpainting step, which attempts to make up the missing information; the third step involves projecting the volume under the same view of the input, concatenating them to complete the current view depth, and integrating all depth into the point cloud. Since the occluded areas are unavailable, we resort to a deep Q-Network to glance around and pick the next best view for large hole completion progressively until a scene is adequately reconstructed while guaranteeing validity. All steps are learned jointly to achieve robust and consistent results. We perform qualitative and quantitative evaluations with extensive experiments on the SUNCG data, obtaining better results than the state of the art.