# Neural Groundplans: Persistent Neural Scene Representations from a Single Image

arXiv: https://arxiv.org/abs/2207.11232 · HTML: https://mlsanity.com/paper/2207.11232/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 64 of its 79 references that are in the map.
- First posted on arXiv 22 Jul 2022.

## Builds on (most cited first, 30 of 64)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields](https://mlsanity.com/paper/2111.12077/) (2021)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [TensoRF: Tensorial Radiance Fields](https://mlsanity.com/paper/2203.09517/) (2022)
- [Efficient Geometry-aware 3D Generative Adversarial Networks](https://mlsanity.com/paper/2112.07945/) (2021)
- [Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D](https://mlsanity.com/paper/2008.05711/) (2020)
- [Object-Centric Learning with Slot Attention](https://mlsanity.com/paper/2006.15055/) (2020)
- [Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations](https://mlsanity.com/paper/1906.01618/) (2019)
- [GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields](https://mlsanity.com/paper/2011.12100/) (2020)
- [Convolutional Occupancy Networks](https://mlsanity.com/paper/2003.04618/) (2020)
- [Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision](https://mlsanity.com/paper/1912.07372/) (2019)
- [DeepVoxels: Learning Persistent 3D Feature Embeddings](https://mlsanity.com/paper/1812.01024/) (2018)
- [MONet: Unsupervised Scene Decomposition and Representation](https://mlsanity.com/paper/1901.11390/) (2019)
- [Advances in Neural Rendering](https://mlsanity.com/paper/2111.05849/) (2021)
- [MonoScene: Monocular 3D Semantic Scene Completion](https://mlsanity.com/paper/2112.00726/) (2021)
- [Multi-Object Representation Learning with Iterative Variational Inference](https://mlsanity.com/paper/1903.00450/) (2019)
- [Light Field Networks: Neural Scene Representations with Single-Evaluation Rendering](https://mlsanity.com/paper/2106.02634/) (2021)
- [Neural Scene Graphs for Dynamic Scenes](https://mlsanity.com/paper/2011.10379/) (2020)
- [Conditional Object-Centric Learning from Video](https://mlsanity.com/paper/2111.12594/) (2021)
- [FIERY: Future Instance Prediction in Bird's-Eye View from Surround Monocular Cameras](https://mlsanity.com/paper/2104.10490/) (2021)
- [DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks](https://mlsanity.com/paper/2103.03231/) (2021)
- [Learning Object-Compositional Neural Radiance Field for Editable Scene Rendering](https://mlsanity.com/paper/2109.01847/) (2021)
- [GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations](https://mlsanity.com/paper/1907.13052/) (2019)
- [Scene Representation Transformer: Geometry-Free Novel View Synthesis Through Set-Latent Scene Representations](https://mlsanity.com/paper/2111.13152/) (2021)
- [Attend, Infer, Repeat: Fast Scene Understanding with Generative Models](https://mlsanity.com/paper/1603.08575/) (2016)
- [Orthographic Feature Transform for Monocular 3D Object Detection](https://mlsanity.com/paper/1811.08188/) (2018)
- [SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decomposition](https://mlsanity.com/paper/2001.02407/) (2020)
- [GRF: Learning a General Radiance Field for 3D Representation and Rendering](https://mlsanity.com/paper/2010.04595/) (2020)
- [CodeNeRF: Disentangled Neural Radiance Fields for Object Categories](https://mlsanity.com/paper/2109.01750/) (2021)

## Built on by (most cited first, 17 of 17)

- [pixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstruction](https://mlsanity.com/paper/2312.12337/) (2023)
- [EmerNeRF: Emergent Spatial-Temporal Scene Decomposition via Self-Supervision](https://mlsanity.com/paper/2311.02077/) (2023)
- [SUDS: Scalable Urban Dynamic Scenes](https://mlsanity.com/paper/2303.14536/) (2023)
- [Diffusion with Forward Models: Solving Stochastic Inverse Problems Without Direct Supervision](https://mlsanity.com/paper/2306.11719/) (2023)
- [Deepfake Generation and Detection: A Benchmark and Survey](https://mlsanity.com/paper/2403.17881/) (2024)
- [Contrastive Lift: 3D Object Instance Segmentation by Slow-Fast Contrastive Fusion](https://mlsanity.com/paper/2306.04633/) (2023)
- [EPIC Fields: Marrying 3D Geometry and Video Understanding](https://mlsanity.com/paper/2306.08731/) (2023)
- [FlowCam: Training Generalizable 3D Radiance Fields without Camera Poses via Pixel-Aligned Scene Flow](https://mlsanity.com/paper/2306.00180/) (2023)
- [Neural Fields in Robotics: A Survey](https://mlsanity.com/paper/2410.20220/) (2024)
- [RoDUS: Robust Decomposition of Static and Dynamic Elements in Urban Scenes](https://mlsanity.com/paper/2403.09419/) (2024)
- [Semantically-aware Neural Radiance Fields for Visual Scene Understanding: A Comprehensive Review](https://mlsanity.com/paper/2402.11141/) (2024)
- [Boosting Self-Supervision for Single-View Scene Completion via Knowledge Distillation](https://mlsanity.com/paper/2404.07933/) (2024)
- [Planar Gaussian Splatting](https://mlsanity.com/paper/2412.01931/) (2024)
- [ClusteringSDF: Self-Organized Neural Implicit Surfaces for 3D Decomposition](https://mlsanity.com/paper/2403.14619/) (2024)
- [6Img-to-3D: Few-Image Large-Scale Outdoor Driving Scene Reconstruction](https://mlsanity.com/paper/2404.12378/) (2024)
- [Efficient, Self-Supervised Human Pose Estimation with Inductive Prior Tuning](https://mlsanity.com/paper/2311.02815/) (2023)
- [Layered Motion Fusion: Lifting Motion Segmentation to 3D in Egocentric Videos](https://mlsanity.com/paper/2506.05546/) (2025)

## Abstract (arXiv)

We present a method to map 2D image observations of a scene to a persistent 3D scene representation, enabling novel view synthesis and disentangled representation of the movable and immovable components of the scene. Motivated by the bird's-eye-view (BEV) representation commonly used in vision and robotics, we propose conditional neural groundplans, ground-aligned 2D feature grids, as persistent and memory-efficient scene representations. Our method is trained self-supervised from unlabeled multi-view observations using differentiable rendering, and learns to complete geometry and appearance of occluded regions. In addition, we show that we can leverage multi-view videos at training time to learn to separately reconstruct static and movable components of the scene from a single image at test time. The ability to separately reconstruct movable objects enables a variety of downstream tasks using simple heuristics, such as extraction of object-centric 3D representations, novel view synthesis, instance-level segmentation, 3D bounding box prediction, and scene editing. This highlights the value of neural groundplans as a backbone for efficient 3D scene understanding models.