mlsanityData to 7 Oct 2026

NeRF-VAE: A Geometry Aware 3D Scene Generative Model

arXiv 2104.00587 ↗First posted 1 Apr 2021stat.ML cs.LGcited by 111

Not on the map. Why?

Abstract (arXiv)

We propose NeRF-VAE, a 3D scene generative model that incorporates geometric structure via NeRF and differentiable volume rendering. In contrast to NeRF, our model takes into account shared structure across scenes, and is able to infer the structure of a novel scene -- without the need to re-train -- using amortized inference. NeRF-VAE's explicit 3D rendering process further contrasts previous generative models with convolution-based rendering which lacks geometric structure. Our model is a VAE that learns a distribution over radiance fields by conditioning them on a latent scene representation. We show that, once trained, NeRF-VAE is able to infer and render geometrically-consistent scenes from previously unseen 3D environments using very few input images. We further demonstrate that NeRF-VAE generalizes well to out-of-distribution cameras, while convolutional models do not. Finally, we introduce and study an attention-based conditioning mechanism of NeRF-VAE's decoder, which improves model performance.

Key facts. Cited by 111 papers in the map (data through 7 Oct 2026). Builds on 33 of its 55 references that are in the map. First posted on arXiv 1 Apr 2021.

Builds on

33 of 55 references.

References · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2017Squeeze-and-Excitation Networkscited by 4,763
  4. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  5. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,037
  6. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  7. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  8. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,574
  9. 2020D-NeRF: Neural Radiance Fields for Dynamic Scenescited by 1,459
  10. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  11. 2020NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collectionscited by 1,230
  12. 2020Nerfies: Deformable Neural Radiance Fieldscited by 1,194
21 more
  1. 2019Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representationscited by 864
  2. 2020GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fieldscited by 784
  3. 2020Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenescited by 750
  4. 2020GRAF: Generative Radiance Fields for 3D-Aware Image Synthesiscited by 688
  5. 2019Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelinescited by 672
  6. 2020NVAE: A Deep Hierarchical Variational Autoencodercited by 635
  7. 2020Space-time Neural Irradiance Fields for Free-Viewpoint Videocited by 414
  8. 2021Bottleneck Transformers for Visual Recognitioncited by 364
  9. 2016A Learned Representation For Artistic Stylecited by 340
  10. 2020Neural Radiance Flow for 4D View Synthesis and Video Processingcited by 270
  11. 2020Learned Initializations for Optimizing Coordinate-Based Neural Representationscited by 240
  12. 2019GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representationscited by 227
  13. 2018Stereo Magnification: Learning View Synthesis using Multiplane Imagescited by 206
  14. 2020BlockGAN: Learning 3D Object-aware Scene Representations from Unlabelled Imagescited by 166
  15. 2018Inference Suboptimality in Variational Autoencoderscited by 131
  16. 2016BundleFusion: Real-time Globally Consistent 3D Reconstruction using On-the-fly Surface Re-integrationcited by 116
  17. 2018Iterative Amortized Inferencecited by 93
  18. 2018Semi-Amortized Variational Autoencoderscited by 83
  19. 2017Learning to Synthesize a 4D RGBD Light Field from a Single Imagecited by 60
  20. 2019HoloGAN: Unsupervised learning of 3D representations from natural imagescited by 59
  21. 2020Unsupervised object-centric video generation and decomposition in 3Dcited by 31

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 14; 2022: 29; 2023: 34; 2024: 24; 2025: 6; 2026: 4.

111 papers.

Sort:

By area:

  1. 2021Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstructioncited by 961
  2. 2022Point-E: A System for Generating 3D Point Clouds from Complex Promptscited by 694
  3. 2021Neural Fields in Visual Computing and Beyondcited by 539
  4. 2023Shap-E: Generating Conditional 3D Implicit Functionscited by 379
  5. 2021Advances in Neural Renderingcited by 365
  6. 2021Light Field Networks: Neural Scene Representations with Single-Evaluation Renderingcited by 330
  7. 2021StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2cited by 300
  8. 20223D Neural Field Generation using Triplane Diffusioncited by 274
  9. 2021Scene Representation Transformer: Geometry-Free Novel View Synthesis Through Set-Latent Scene Representationscited by 222
  10. 2023MeshDiffusion: Score-based Generative 3D Mesh Modelingcited by 198
  11. 2022Generating Videos with Dynamics-aware Implicit Generative Adversarial Networkscited by 186
  12. 2022From data to functa: Your data point is a function and you can treat it like onecited by 183
38 more
  1. 2022RenderDiffusion: Image Diffusion for 3D Reconstruction, Inpainting and Generationcited by 162
  2. 2022ESLAM: Efficient Dense SLAM System Based on Hybrid Representation of Signed Distance Fieldscited by 158
  3. 2022NeRF: Neural Radiance Field in 3D Vision: A Comprehensive Review (Updated Post-Gaussian Splatting)cited by 156
  4. 2022NeRFPlayer: A Streamable Dynamic Scene Representation with Decomposed Neural Radiance Fieldscited by 151
  5. 2022GAUDI: A Neural Architect for Immersive 3D Scene Generationcited by 136
  6. 2022ActiveNeRF: Learning where to See with Uncertainty Estimationcited by 124
  7. 2021NeSF: Neural Semantic Fields for Generalizable Semantic Segmentation of 3D Scenescited by 119
  8. 2023Diffusion with Forward Models: Solving Stochastic Inverse Problems Without Direct Supervisioncited by 112
  9. 2022EpiGRAF: Rethinking training of 3D GANscited by 108
  10. 2023Compositional 3D Scene Generation using Locally Conditioned Diffusioncited by 101
  11. 2021Unsupervised Discovery of Object Radiance Fieldscited by 100
  12. 2021ClevrTex: A Texture-Rich Benchmark for Unsupervised Multi-Object Segmentationcited by 93
  13. 2024LN3DIFF++: Scalable Latent Neural Fields Diffusion for Speedy 3D Generationcited by 91
  14. 2022Object Scene Representation Transformercited by 90
  15. 2021Decomposing 3D Scenes into Objects via Unsupervised Volume Segmentationcited by 88
  16. 2021Stochastic Neural Radiance Fields: Quantifying Uncertainty in Implicit 3D Representationscited by 80
  17. 2024NVS-Solver: Video Diffusion Model as Zero-Shot Novel View Synthesizercited by 79
  18. 2021Learning to Stylize Novel Viewscited by 76
  19. 2023SODA: Bottleneck Diffusion Models for Representation Learningcited by 76
  20. 2022Conditional-Flow NeRF: Accurate 3D Modelling with Reliable Uncertainty Quantificationcited by 71
  21. 2021SIMONe: View-Invariant, Temporally-Abstracted Object Representations via Unsupervised Video Decompositioncited by 69
  22. 2022Physical Adversarial Attack meets Computer Vision: A Decade Surveycited by 66
  23. 2024Advances in 3D Generation: A Surveycited by 62
  24. 2023NeO 360: Neural Fields for Sparse View Synthesis of Outdoor Scenescited by 53
  25. 2022Compressing Explicit Voxel Grid Representations: fast NeRFs become also smallcited by 48
  26. 2023NeRF-Supervised Deep Stereocited by 46
  27. 2022V4d: voxel for 4d novel view synthesiscited by 42
  28. 2023ReLight My NeRF: A Dataset for Novel View Synthesis and Relighting of Real World Objectscited by 42
  29. 2022RUST: Latent Neural Scene Representations from Unposed Imagerycited by 41
  30. 2024SAR3D: Autoregressive 3D Object Generation and Understanding via Multi-scale 3D VQVAEcited by 38
  31. 2022Transframer: Arbitrary Frame Prediction with Generative Modelscited by 37
  32. 2023SHAP-EDITOR: Instruction-guided Latent 3D Editing in Secondscited by 36
  33. 2024GSEdit: Efficient Text-Guided Editing of 3D Objects via Gaussian Splattingcited by 36
  34. 2021NeRF in detail: Learning to sample for view synthesiscited by 35
  35. 2024A survey of synthetic data augmentation methods in computer visioncited by 34
  36. 2023VQ3D: Learning a 3D-Aware Generative Model on ImageNetcited by 32
  37. 2022Neural Fourier Filter Bankcited by 29
  38. 2023Physics-Informed Computer Vision: A Review and Perspectivescited by 28

50 most cited of 111 All 111 (JSON)