# Neural Rendering in a Room: Amodal 3D Understanding and Free-Viewpoint Rendering for the Closed Scene Composed of Pre-Captured Objects arXiv: https://arxiv.org/abs/2205.02714 ยท HTML: https://mlsanity.com/paper/2205.02714/ ## Key facts - Cited by 7 papers in the map (data through 14 Oct 2026). - Builds on 60 of its 85 references that are in the map. - First posted on arXiv 5 May 2022. ## Builds on (most cited first, 30 of 60) - [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022) - [NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction](https://mlsanity.com/paper/2106.10689/) (2021) - [Plenoxels: Radiance Fields without Neural Networks](https://mlsanity.com/paper/2112.05131/) (2021) - [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020) - [SuperGlue: Learning Feature Matching with Graph Neural Networks](https://mlsanity.com/paper/1911.11763/) (2019) - [Deep High-Resolution Representation Learning for Human Pose Estimation](https://mlsanity.com/paper/1902.09212/) (2019) - [NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections](https://mlsanity.com/paper/2008.02268/) (2020) - [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018) - [EfficientDet: Scalable and Efficient Object Detection](https://mlsanity.com/paper/1911.09070/) (2019) - [Volume Rendering of Neural Implicit Surfaces](https://mlsanity.com/paper/2106.12052/) (2021) - [NetVLAD: CNN architecture for weakly supervised place recognition](https://mlsanity.com/paper/1511.07247/) (2015) - [Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations](https://mlsanity.com/paper/1906.01618/) (2019) - [Implicit Geometric Regularization for Learning Shapes](https://mlsanity.com/paper/2002.10099/) (2020) - [IBRNet: Learning Multi-View Image-Based Rendering](https://mlsanity.com/paper/2102.13090/) (2021) - [Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision](https://mlsanity.com/paper/1912.07372/) (2019) - [KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs](https://mlsanity.com/paper/2103.13744/) (2021) - [Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelines](https://mlsanity.com/paper/1905.00889/) (2019) - [Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images](https://mlsanity.com/paper/1804.01654/) (2018) - [UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction](https://mlsanity.com/paper/2104.10078/) (2021) - [From Coarse to Fine: Robust Hierarchical Localization at Large Scale](https://mlsanity.com/paper/1812.03506/) (2018) - [AtlasNet: A Papier-Mach'e Approach to Learning 3D Surface Generation](https://mlsanity.com/paper/1802.05384/) (2018) - [FastNeRF: High-Fidelity Neural Rendering at 200FPS](https://mlsanity.com/paper/2103.10380/) (2021) - [Semantic Scene Completion from a Single Depth Image](https://mlsanity.com/paper/1611.08974/) (2016) - [Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning](https://mlsanity.com/paper/1904.01786/) (2019) - [DeepVoxels: Learning Persistent 3D Feature Embeddings](https://mlsanity.com/paper/1812.01024/) (2018) - [INeRF: Inverting Neural Radiance Fields for Pose Estimation](https://mlsanity.com/paper/2012.05877/) (2020) - [Neural Scene Graphs for Dynamic Scenes](https://mlsanity.com/paper/2011.10379/) (2020) - [Single-View View Synthesis with Multiplane Images](https://mlsanity.com/paper/2004.11364/) (2020) - [Free View Synthesis](https://mlsanity.com/paper/2008.05511/) (2020) - [Learning Object-Compositional Neural Radiance Field for Editable Scene Rendering](https://mlsanity.com/paper/2109.01847/) (2021) ## Built on by (most cited first, 7 of 7) - [SINE: Semantic-driven Image-based NeRF Editing with Prior-guided Editing Field](https://mlsanity.com/paper/2303.13277/) (2023) - [Coin3D: Controllable and Interactive 3D Assets Generation with Proxy-Guided Conditioning](https://mlsanity.com/paper/2405.08054/) (2024) - [Mirror-NeRF: Learning Neural Radiance Fields for Mirrors with Whitted-Style Ray Tracing](https://mlsanity.com/paper/2308.03280/) (2023) - [Factorized and Controllable Neural Re-Rendering of Outdoor Scene for Photo Extrapolation](https://mlsanity.com/paper/2207.06899/) (2022) - [Pano2Room: Novel View Synthesis from a Single Indoor Panorama](https://mlsanity.com/paper/2408.11413/) (2024) - [Casual Indoor HDR Radiance Capture from Omnidirectional Images](https://mlsanity.com/paper/2208.07903/) (2022) - [AssetField: Assets Mining and Reconfiguration in Ground Feature Plane Representation](https://mlsanity.com/paper/2303.13953/) (2023) ## Abstract (arXiv) We, as human beings, can understand and picture a familiar scene from arbitrary viewpoints given a single image, whereas this is still a grand challenge for computers. We hereby present a novel solution to mimic such human perception capability based on a new paradigm of amodal 3D scene understanding with neural rendering for a closed scene. Specifically, we first learn the prior knowledge of the objects in a closed scene via an offline stage, which facilitates an online stage to understand the room with unseen furniture arrangement. During the online stage, given a panoramic image of the scene in different layouts, we utilize a holistic neural-rendering-based optimization framework to efficiently estimate the correct 3D scene layout and deliver realistic free-viewpoint rendering. In order to handle the domain gap between the offline and online stage, our method exploits compositional neural rendering techniques for data augmentation in the offline training. The experiments on both synthetic and real datasets demonstrate that our two-stage design achieves robust 3D scene understanding and outperforms competing methods by a large margin, and we also show that our realistic free-viewpoint rendering enables various applications, including scene touring and editing. Code and data are available on the project webpage: https://zju3dv.github.io/nr_in_a_room/.