mlsanityData to 14 Oct 2026

Solving Inverse Problems with NerfGANs

arXiv 2112.09061 ↗First posted 16 Dec 2021cs.CV cs.AI cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

We introduce a novel framework for solving inverse problems using NeRF-style generative models. We are interested in the problem of 3-D scene reconstruction given a single 2-D image and known camera parameters. We show that naively optimizing the latent space leads to artifacts and poor novel view rendering. We attribute this problem to volume obstructions that are clear in the 3-D geometry and become visible in the renderings of novel views. We propose a novel radiance field regularization method to obtain better 3-D surfaces and improved novel views given single view observations. Our method naturally extends to general inverse problems including inpainting where one observes only partially a single view. We experimentally evaluate our method, achieving visual improvements and performance boosts over the baselines in a wide range of tasks. Our method achieves 30-40% MSE reduction and 15-25% reduction in LPIPS loss compared to the previous state of the art.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 41 of its 47 references that are in the map. First posted on arXiv 16 Dec 2021.

Builds on

41 of 47 references.

References · 41

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  4. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  5. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  6. 2014Deep Learning Face Attributes in the Wildcited by 3,693
  7. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,588
  8. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  9. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,626
  10. 2020Implicit Neural Representations with Periodic Activation Functionscited by 2,238
  11. 2021Alias-Free Generative Adversarial Networkscited by 1,184
  12. 2020Neural Sparse Voxel Fieldscited by 959
29 more
  1. 2020GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fieldscited by 785
  2. 2020Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenescited by 751
  3. 2020pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesiscited by 708
  4. 2020GRAF: Generative Radiance Fields for 3D-Aware Image Synthesiscited by 689
  5. 2021KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPscited by 673
  6. 2021FastNeRF: High-Fidelity Neural Rendering at 200FPScited by 511
  7. 2021StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesiscited by 477
  8. 2020Space-time Neural Irradiance Fields for Free-Viewpoint Videocited by 414
  9. 2020PULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Modelscited by 358
  10. 2017Compressed Sensing using Generative Modelscited by 339
  11. 2020Deep Learning Techniques for Inverse Problems in Imagingcited by 291
  12. 2021Robust Compressed Sensing MRI with Deep Generative Priorscited by 286
  13. 20193D Morphable Face Models -- Past, Present and Futurecited by 270
  14. 2020Neural Radiance Flow for 4D View Synthesis and Video Processingcited by 270
  15. 2021DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networkscited by 232
  16. 2015Coordinate Descent Algorithmscited by 219
  17. 2020AutoInt: Automatic Integration for Fast Neural Volume Renderingcited by 204
  18. 2020DeRF: Decomposed Radiance Fieldscited by 163
  19. 2021GNeRF: GAN-based Neural Radiance Field without Posed Cameracited by 153
  20. 2021CIPS-3D: A 3D-Aware Generator of GANs Based on Conditionally-Independent Pixel Synthesiscited by 149
  21. 2018Visual Object Networks: Image Generation with Disentangled 3D Representationcited by 124
  22. 2018Phase Retrieval Under a Generative Priorcited by 73
  23. 2021A Shading-Guided Generative Implicit Model for Shape-Accurate 3D-Aware Image Synthesiscited by 72
  24. 2021Intermediate Layer Optimization for Inverse Problems using Deep Generative Modelscited by 70
  25. 2020SMYRF: Efficient Attention using Asymmetric Clusteringcited by 43
  26. 2018RenderNet: A deep convolutional network for differentiable rendering from 3D shapescited by 42
  27. 2021Fairness for Image Generation with Uncertain Sensitive Attributescited by 31
  28. 2019Your Local GAN: Designing Two Dimensional Local Attention Mechanisms for Generative Modelscited by 21
  29. 2021Constrained Instance and Class Reweighting for Robust Learning under Label Noisecited by 12

Built on by

Cited by year

Citing papers per year: 2022: 4; 2023: 2; 2024: 1.

7 papers.

  1. 20223D GAN Inversion for Controllable Portrait Image Animationcited by 53
  2. 20223D GAN Inversion with Pose Optimizationcited by 44
  3. 20233D-aware Blending with Generative NeRFscited by 13
  4. 2022Score-Guided Intermediate Layer Optimization: Fast Langevin Mixing for Inverse Problemscited by 11
  5. 2022Implicit Neural Representation for Mesh-Free Inverse Obstacle Scatteringcited by 9
  6. 2024Deep Learning for Accelerated and Robust MRI Reconstruction: a Reviewcited by 9
  7. 2023Neural Radiance Fields: Past, Present, and Futurecited by 8