mlsanityData to 14 Oct 2026

Simple and Effective Synthesis of Indoor 3D Scenes

arXiv 2204.02960 ↗First posted 6 Apr 2022cs.CV cs.AI cs.LGcited by 33

Not on the map. Why?

Abstract (arXiv)

We study the problem of synthesizing immersive 3D indoor scenes from one or more images. Our aim is to generate high-resolution images and videos from novel viewpoints, including viewpoints that extrapolate far beyond the input images while maintaining 3D consistency. Existing approaches are highly complex, with many separately trained stages and components. We propose a simple alternative: an image-to-image GAN that maps directly from reprojections of incomplete point clouds to full high-resolution RGB-D images. On the Matterport3D and RealEstate10K datasets, our approach significantly outperforms prior work when evaluated by humans, as well as on FID scores. Further, we show that our model is useful for generative data augmentation. A vision-and-language navigation (VLN) agent trained with trajectories spatially-perturbed by our model improves success rate by up to 1.5% over a state of the art baseline on the R2R benchmark. Our code will be made available to facilitate generative data augmentation and applications to downstream robotics and embodied AI tasks.

Key facts. Cited by 33 papers in the map (data through 14 Oct 2026). Builds on 60 of its 84 references that are in the map. First posted on arXiv 6 Apr 2022.

Builds on

60 of 84 references.

References · 60

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  5. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  6. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,527
  7. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,694
  8. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  9. 2021Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fieldscited by 1,964
  10. 2021Vision Transformers for Dense Predictioncited by 1,803
  11. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,584
  12. 2018World Modelscited by 1,513
38 more
  1. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,387
  2. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,376
  3. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,299
  4. 2020NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collectionscited by 1,232
  5. 2019Habitat: A Platform for Embodied AI Researchcited by 1,191
  6. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,027
  7. 2016End to End Learning for Self-Driving Carscited by 1,009
  8. 2015ORB-SLAM: a Versatile and Accurate Monocular SLAM Systemcited by 988
  9. 2018MegaDepth: Learning Single-View Depth Prediction from Internet Photoscited by 737
  10. 2022Block-NeRF: Scalable Large Scene Neural View Synthesiscited by 724
  11. 2018On Evaluation of Embodied Navigation Agentscited by 547
  12. 2018Image Inpainting for Irregular Holes Using Partial Convolutionscited by 505
  13. 2017End-to-end Driving via Conditional Imitation Learningcited by 485
  14. 2018Gibson Env: Real-World Perception for Embodied Agentscited by 481
  15. 2021Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesiscited by 464
  16. 2017PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modificationscited by 417
  17. 2019SynSin: End-to-end View Synthesis from a Single Imagecited by 364
  18. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 302
  19. 2016Deep Visual Foresight for Planning Robot Motioncited by 285
  20. 2017Data Augmentation Generative Adversarial Networkscited by 278
  21. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 271
  22. 2020Single-View View Synthesis with Multiplane Imagescited by 263
  23. 2019DeepView: View Synthesis with Learned Gradient Descentcited by 253
  24. 2018Stochastic Video Generation with a Learned Priorcited by 245
  25. 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 244
  26. 2020Infinite Nature: Perpetual View Generation of Natural Scenes from a Single Imagecited by 243
  27. 2017Geometric GANcited by 233
  28. 2020Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-trainingcited by 231
  29. 2018Stereo Magnification: Learning View Synthesis using Multiplane Imagescited by 211
  30. 20203D Photography using Context-aware Layered Depth Inpaintingcited by 206
  31. 2017Learning a Multi-View Stereo Machinecited by 194
  32. 2019Pushing the Boundaries of View Extrapolation with Multiplane Imagescited by 191
  33. 2015DeepStereo: Learning to Predict New Views from the World's Imagerycited by 185
  34. 2021Pathdreamer: A World Model for Indoor Navigationcited by 118
  35. 2020Language and Visual Entity Relationship Graph for Agent Navigationcited by 100
  36. 2021Geometry-Free View Synthesis: Transformers and no 3D Priorscited by 100
  37. 2021PixelSynth: Generating a 3D-Consistent Experience from a Single Imagecited by 82
  38. 2019Tactical Rewind: Self-Correction via Backtracking in Vision-and-Language Navigationcited by 81

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 5; 2023: 8; 2024: 8; 2025: 9; 2026: 3.

33 papers.

Sort:

By area:

  1. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 380
  2. 2022BEVBert: Multimodal Map Pre-training for Language-guided Navigationcited by 122
  3. 2023Consistent View Synthesis with Pose-Guided Diffusion Modelscited by 117
  4. 2023Scaling Data Generation in Vision-and-Language Navigationcited by 117
  5. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  6. 2023PanoGen: Text-Conditioned Panoramic Environment Generation for Vision-and-Language Navigationcited by 77
  7. 2023Long-Term Photometric Consistent Novel View Synthesis with Diffusion Modelscited by 76
  8. 2023Set-the-Scene: Global-Local Training for Generating Controllable NeRF Scenescited by 75
  9. 2024GenWarp: Single Image to Novel Views with Semantic-Preserving Generative Warpingcited by 73
  10. 2024MultiDiff: Consistent Novel View Synthesis from a Single Imagecited by 68
  11. 2022A New Path: Scaling Vision-and-Language Navigation with Synthetic Instructions and Imitation Learningcited by 60
  12. 2025MindJourney: Test-Time Scaling with World Models for Spatial Reasoningcited by 50
21 more
  1. 2025World-in-World: World Models in a Closed-Loop Worldcited by 47
  2. 2023Learning Navigational Visual Representations with Semantic Map Supervisioncited by 43
  3. 20253D Scene Generation: A Surveycited by 24
  4. 2025EvoWorld: Evolving Panoramic World Generation with Explicit 3D Memorycited by 20
  5. 2025NavRAG: Generating User Demand Instructions for Embodied Navigation through Retrieval-Augmented LLMcited by 19
  6. 2024Forest2Seq: Revitalizing Order Prior for Sequential Indoor Scene Synthesiscited by 12
  7. 2022Bridging Implicit and Explicit Geometric Transformation for Single-Image View Synthesiscited by 11
  8. 2023ShowRoom3D: Text to High-Quality 3D Room Generation Using 3D Priorscited by 7
  9. 2025PanoGen++: Domain-Adapted Text-Guided Panoramic Environment Generation for Vision-and-Language Navigationcited by 6
  10. 2025Generating Vision-Language Navigation Instructions Incorporated Fine-Grained Alignment Annotationscited by 5
  11. 2022HORIZON: High-Resolution Semantically Controlled Panorama Synthesiscited by 3
  12. 2024BlendScape: Enabling End-User Customization of Video-Conferencing Environments through Generative AIcited by 3
  13. 2025Constructing a 3D Scene from a Single Imagecited by 3
  14. 2024World-Consistent Data Generation for Vision-and-Language Navigationcited by 2
  15. 2024Into the Unknown: Generating Geospatial Descriptions for New Environmentscited by 1
  16. 2026Image2Sim: Scaling Embodied Navigation via Generative Neural Simulatorcited by 1
  17. 2023Pre-Trained Masked Image Model for Mobile Robot Navigationcited by 0
  18. 2024Real-Time Position-Aware View Synthesis from Single-View Inputcited by 0
  19. 2025HierRelTriple: Guiding Indoor Layout Generation with Hierarchical Relationship Triplet Lossescited by 0
  20. 2026Advancing DialNav through Automatic Embodied Dialog Augmentationcited by 0
  21. 2026Assessing the Operational Impact of Poisoning Attacks over Augmented 3D Point Cloud Public Datasets for Connected and Autonomous Vehiclescited by 0