# Simple and Effective Synthesis of Indoor 3D Scenes

arXiv: https://arxiv.org/abs/2204.02960 · HTML: https://mlsanity.com/paper/2204.02960/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 60 of its 84 references that are in the map.
- First posted on arXiv 6 Apr 2022.

## Builds on (most cited first, 30 of 60)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections](https://mlsanity.com/paper/2008.02268/) (2020)
- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments](https://mlsanity.com/paper/1711.07280/) (2017)
- [End to End Learning for Self-Driving Cars](https://mlsanity.com/paper/1604.07316/) (2016)
- [ORB-SLAM: a Versatile and Accurate Monocular SLAM System](https://mlsanity.com/paper/1502.00956/) (2015)
- [MegaDepth: Learning Single-View Depth Prediction from Internet Photos](https://mlsanity.com/paper/1804.00607/) (2018)
- [Block-NeRF: Scalable Large Scene Neural View Synthesis](https://mlsanity.com/paper/2202.05263/) (2022)
- [On Evaluation of Embodied Navigation Agents](https://mlsanity.com/paper/1807.06757/) (2018)
- [Image Inpainting for Irregular Holes Using Partial Convolutions](https://mlsanity.com/paper/1804.07723/) (2018)
- [End-to-end Driving via Conditional Imitation Learning](https://mlsanity.com/paper/1710.02410/) (2017)
- [Gibson Env: Real-World Perception for Embodied Agents](https://mlsanity.com/paper/1808.10654/) (2018)
- [Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis](https://mlsanity.com/paper/2104.00677/) (2021)
- [PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications](https://mlsanity.com/paper/1701.05517/) (2017)
- [SynSin: End-to-end View Synthesis from a Single Image](https://mlsanity.com/paper/1912.08804/) (2019)
- [A Recurrent Vision-and-Language BERT for Navigation](https://mlsanity.com/paper/2011.13922/) (2020)

## Built on by (most cited first, 30 of 33)

- [ProcTHOR: Large-Scale Embodied AI Using Procedural Generation](https://mlsanity.com/paper/2206.06994/) (2022)
- [BEVBert: Multimodal Map Pre-training for Language-guided Navigation](https://mlsanity.com/paper/2212.04385/) (2022)
- [Consistent View Synthesis with Pose-Guided Diffusion Models](https://mlsanity.com/paper/2303.17598/) (2023)
- [Scaling Data Generation in Vision-and-Language Navigation](https://mlsanity.com/paper/2307.15644/) (2023)
- [Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Models](https://mlsanity.com/paper/2407.07035/) (2024)
- [PanoGen: Text-Conditioned Panoramic Environment Generation for Vision-and-Language Navigation](https://mlsanity.com/paper/2305.19195/) (2023)
- [Long-Term Photometric Consistent Novel View Synthesis with Diffusion Models](https://mlsanity.com/paper/2304.10700/) (2023)
- [Set-the-Scene: Global-Local Training for Generating Controllable NeRF Scenes](https://mlsanity.com/paper/2303.13450/) (2023)
- [GenWarp: Single Image to Novel Views with Semantic-Preserving Generative Warping](https://mlsanity.com/paper/2405.17251/) (2024)
- [MultiDiff: Consistent Novel View Synthesis from a Single Image](https://mlsanity.com/paper/2406.18524/) (2024)
- [A New Path: Scaling Vision-and-Language Navigation with Synthetic Instructions and Imitation Learning](https://mlsanity.com/paper/2210.03112/) (2022)
- [MindJourney: Test-Time Scaling with World Models for Spatial Reasoning](https://mlsanity.com/paper/2507.12508/) (2025)
- [World-in-World: World Models in a Closed-Loop World](https://mlsanity.com/paper/2510.18135/) (2025)
- [Learning Navigational Visual Representations with Semantic Map Supervision](https://mlsanity.com/paper/2307.12335/) (2023)
- [3D Scene Generation: A Survey](https://mlsanity.com/paper/2505.05474/) (2025)
- [EvoWorld: Evolving Panoramic World Generation with Explicit 3D Memory](https://mlsanity.com/paper/2510.01183/) (2025)
- [NavRAG: Generating User Demand Instructions for Embodied Navigation through Retrieval-Augmented LLM](https://mlsanity.com/paper/2502.11142/) (2025)
- [Forest2Seq: Revitalizing Order Prior for Sequential Indoor Scene Synthesis](https://mlsanity.com/paper/2407.05388/) (2024)
- [Bridging Implicit and Explicit Geometric Transformation for Single-Image View Synthesis](https://mlsanity.com/paper/2209.07105/) (2022)
- [ShowRoom3D: Text to High-Quality 3D Room Generation Using 3D Priors](https://mlsanity.com/paper/2312.13324/) (2023)
- [PanoGen++: Domain-Adapted Text-Guided Panoramic Environment Generation for Vision-and-Language Navigation](https://mlsanity.com/paper/2503.09938/) (2025)
- [Generating Vision-Language Navigation Instructions Incorporated Fine-Grained Alignment Annotations](https://mlsanity.com/paper/2506.08566/) (2025)
- [HORIZON: High-Resolution Semantically Controlled Panorama Synthesis](https://mlsanity.com/paper/2210.04522/) (2022)
- [BlendScape: Enabling End-User Customization of Video-Conferencing Environments through Generative AI](https://mlsanity.com/paper/2403.13947/) (2024)
- [Constructing a 3D Scene from a Single Image](https://mlsanity.com/paper/2505.15765/) (2025)
- [World-Consistent Data Generation for Vision-and-Language Navigation](https://mlsanity.com/paper/2412.06413/) (2024)
- [Into the Unknown: Generating Geospatial Descriptions for New Environments](https://mlsanity.com/paper/2406.19967/) (2024)
- [Image2Sim: Scaling Embodied Navigation via Generative Neural Simulator](https://mlsanity.com/paper/2607.05765/) (2026)
- [Pre-Trained Masked Image Model for Mobile Robot Navigation](https://mlsanity.com/paper/2310.07021/) (2023)
- [Real-Time Position-Aware View Synthesis from Single-View Input](https://mlsanity.com/paper/2412.14005/) (2024)

## Abstract (arXiv)

We study the problem of synthesizing immersive 3D indoor scenes from one or more images. Our aim is to generate high-resolution images and videos from novel viewpoints, including viewpoints that extrapolate far beyond the input images while maintaining 3D consistency. Existing approaches are highly complex, with many separately trained stages and components. We propose a simple alternative: an image-to-image GAN that maps directly from reprojections of incomplete point clouds to full high-resolution RGB-D images. On the Matterport3D and RealEstate10K datasets, our approach significantly outperforms prior work when evaluated by humans, as well as on FID scores. Further, we show that our model is useful for generative data augmentation. A vision-and-language navigation (VLN) agent trained with trajectories spatially-perturbed by our model improves success rate by up to 1.5% over a state of the art baseline on the R2R benchmark. Our code will be made available to facilitate generative data augmentation and applications to downstream robotics and embodied AI tasks.