mlsanityData to 7 Oct 2026

GAUDI: A Neural Architect for Immersive 3D Scene Generation

arXiv 2207.13751 ↗First posted 27 Jul 2022cs.CV cs.GR cs.LGcited by 136

Not on the map. Why?

Cited by 57 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce GAUDI, a generative model capable of capturing the distribution of complex and realistic 3D scenes that can be rendered immersively from a moving camera. We tackle this challenging problem with a scalable yet powerful approach, where we first optimize a latent representation that disentangles radiance fields and camera poses. This latent representation is then used to learn a generative model that enables both unconditional and conditional generation of 3D scenes. Our model generalizes previous works that focus on single objects by removing the assumption that the camera pose distribution can be shared across samples. We show that GAUDI obtains state-of-the-art performance in the unconditional generative setting across multiple datasets and allows for conditional generation of 3D scenes given conditioning variables like sparse image observations or text that describes the scene.

Key facts. Cited by 136 papers in the map (data through 7 Oct 2026). Builds on 67 of its 81 references that are in the map. First posted on arXiv 27 Jul 2022.

Builds on

67 of 81 references.

References · 67

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2020Denoising Diffusion Implicit Modelscited by 9,655
  8. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  9. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  10. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  11. 2017Neural Discrete Representation Learningcited by 4,650
  12. 2021Zero-Shot Text-to-Image Generationcited by 4,276
38 more
  1. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  2. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  3. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  4. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  5. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,682
  6. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  7. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,037
  8. 2021Plenoxels: Radiance Fields without Neural Networkscited by 1,607
  9. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,574
  10. 2018World Modelscited by 1,444
  11. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  12. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  13. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
  14. 2021Efficient Geometry-aware 3D Generative Adversarial Networkscited by 1,204
  15. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  16. 2020Neural Sparse Voxel Fieldscited by 959
  17. 2019The Replica Dataset: A Digital Replica of Indoor Spacescited by 885
  18. 2020GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fieldscited by 784
  19. 2020Convolutional Occupancy Networkscited by 760
  20. 2020pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesiscited by 708
  21. 2020GRAF: Generative Radiance Fields for 3D-Aware Image Synthesiscited by 688
  22. 2021KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPscited by 672
  23. 2021Score-based Generative Modeling in Latent Spacecited by 634
  24. 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
  25. 2021StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesiscited by 477
  26. 2021ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Datacited by 442
  27. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 429
  28. 2021Text2Mesh: Text-Driven Neural Stylization for Meshescited by 332
  29. 2020The Creation and Detection of Deepfakes: A Surveycited by 246
  30. 2021Scene Representation Transformer: Geometry-Free Novel View Synthesis Through Set-Latent Scene Representationscited by 222
  31. 2021ATISS: Autoregressive Transformers for Indoor Scene Synthesiscited by 195
  32. 2022From data to functa: Your data point is a function and you can treat it like onecited by 183
  33. 2022Learning Fast Samplers for Diffusion Models by Differentiating Through Sample Qualitycited by 181
  34. 2016ViZDoom: A Doom-based AI Research Platform for Visual Reinforcement Learningcited by 178
  35. 2017Optimizing the Latent Space of Generative Networkscited by 157
  36. 2020SceneFormer: Indoor Scene Generation with Transformerscited by 154
  37. 2021CIPS-3D: A 3D-Aware Generator of GANs Based on Conditionally-Independent Pixel Synthesiscited by 149
  38. 2021Unconstrained Scene Generation with Locally Conditioned Radiance Fieldscited by 130

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 15; 2023: 55; 2024: 36; 2025: 21; 2026: 9.

136 papers.

Sort:

By area:

  1. 2022Point-E: A System for Generating 3D Point Clouds from Complex Promptscited by 694
  2. 2022Latent-NeRF for Shape-Guided Generation of 3D Shapes and Texturescited by 506
  3. 2023Shap-E: Generating Conditional 3D Implicit Functionscited by 379
  4. 2022Rodin: A Generative Model for Sculpting 3D Digital Avatars Using Diffusioncited by 309
  5. 20223D Neural Field Generation using Triplane Diffusioncited by 274
  6. 2023Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Modelscited by 245
  7. 2022DiffRF: Rendering-Guided 3D Radiance Field Diffusioncited by 204
  8. 2024WonderWorld: Interactive 3D Scene Generation from a Single Imagecited by 197
  9. 2023Single-Stage Diffusion NeRF: A Unified Approach to 3D Generation and Reconstructioncited by 180
  10. 2023NerfDiff: Single-image View Synthesis with NeRF-guided Distillation from 3D-aware Diffusioncited by 179
  11. 2023Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Modelscited by 169
  12. 2022RenderDiffusion: Image Diffusion for 3D Reconstruction, Inpainting and Generationcited by 162
38 more
  1. 2023SceneScape: Text-Driven Consistent Scene Generationcited by 160
  2. 2023DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Priorcited by 157
  3. 2023WonderJourney: Going from Anywhere to Everywherecited by 157
  4. 2022NeRDi: Single-View NeRF Synthesis with Language-Guided Diffusion as General Image Priorscited by 156
  5. 2023HyperDiffusion: Generating Implicit Neural Fields with Weight-Space Diffusioncited by 152
  6. 2023Tri-MipRF: Tri-Mip Representation for Efficient Anti-Aliasing Neural Radiance Fieldscited by 142
  7. 2023DiffTalk: Crafting Diffusion Models for Generalized Audio-Driven Portraits Animationcited by 136
  8. 2023State of the Art on Diffusion Models for Visual Computingcited by 126
  9. 2023Consistent View Synthesis with Pose-Guided Diffusion Modelscited by 117
  10. 2023HoloDiffusion: Training a 3D Diffusion Model using 2D Imagescited by 106
  11. 2023ATT3D: Amortized Text-to-3D Object Synthesiscited by 87
  12. 2024I-Design: Personalized LLM Interior Designercited by 87
  13. 2023TexFusion: Synthesizing 3D Textures with Text-Guided Image Diffusion Modelscited by 86
  14. 2022BeLFusion: Latent Diffusion for Behavior-Driven Human Motion Predictioncited by 82
  15. 2023AnyHome: Open-Vocabulary Generation of Structured and Textured 3D Homescited by 81
  16. 2023Score-based Diffusion Models in Function Spacecited by 80
  17. 2023NeuralField-LDM: Scene Generation with Hierarchical Latent Diffusion Modelscited by 76
  18. 2023Long-Term Photometric Consistent Novel View Synthesis with Diffusion Modelscited by 76
  19. 2023CityDreamer: Compositional Generative Model of Unbounded 3D Citiescited by 75
  20. 2023SceneDreamer: Unbounded 3D Scene Generation from 2D Image Collectionscited by 72
  21. 2024MagicDrive3D: Controllable 3D Generation for Any-View Rendering in Street Scenescited by 70
  22. 20223D-LDM: Neural Implicit 3D Shape Generation with Latent Diffusion Modelscited by 67
  23. 2024MultiDiff: Consistent Novel View Synthesis from a Single Imagecited by 67
  24. 2024EchoScene: Indoor Scene Generation via Information Echo over Scene Graph Diffusioncited by 66
  25. 2024Advances in 3D Generation: A Surveycited by 62
  26. 2023Generative AI meets 3D: A Survey on Text-to-3D in AIGC Eracited by 61
  27. 2023Ctrl-Room: Controllable Text-to-3D Room Meshes Generation with Layout Constraintscited by 59
  28. 2023CC3D: Layout-Conditioned Generation of Compositional 3D Scenescited by 58
  29. 20223DDesigner: Towards Photorealistic 3D Object Generation and Editing with Text-guided Diffusion Modelscited by 57
  30. 2025Bolt3D: Generating 3D Scenes in Secondscited by 57
  31. 2023Spatial Functa: Scaling Functa to ImageNet Classification and Generationcited by 55
  32. 2023ControlRoom3D: Room Generation using Semantic Proxy Roomscited by 54
  33. 2023RoomDreamer: Text-Driven 3D Indoor Scene Synthesis with Coherent Geometry and Texturecited by 53
  34. 2023NeO 360: Neural Fields for Sparse View Synthesis of Outdoor Scenescited by 53
  35. 2022DiffDreamer: Towards Consistent Unsupervised Single-view Scene Extrapolation with Conditional Diffusion Modelscited by 51
  36. 2023AutoDecoding Latent 3D Diffusion Modelscited by 50
  37. 2022RGBD2: Generative Scene Synthesis via Incremental View Inpainting using RGBD Diffusion Modelscited by 45
  38. 2024Generative Gaussian Splatting for Unbounded 3D City Generationcited by 43

50 most cited of 136 All 136 (JSON)