mlsanityData to 7 Oct 2026

AE-NeRF: Auto-Encoding Neural Radiance Fields for 3D-Aware Object Manipulation

arXiv 2204.13426 ↗First posted 28 Apr 2022cs.CVcited by 4

Not on the map. Why?

Abstract (arXiv)

We propose a novel framework for 3D-aware object manipulation, called Auto-Encoding Neural Radiance Fields (AE-NeRF). Our model, which is formulated in an auto-encoder architecture, extracts disentangled 3D attributes such as 3D shape, appearance, and camera pose from an image, and a high-quality image is rendered from the attributes through disentangled generative Neural Radiance Fields (NeRF). To improve the disentanglement ability, we present two losses, global-local attribute consistency loss defined between input and output, and swapped-attribute classification loss. Since training such auto-encoding networks from scratch without ground-truth shape and appearance information is non-trivial, we present a stage-wise training scheme, which dramatically helps to boost the performance. We conduct experiments to demonstrate the effectiveness of the proposed model over the latest methods and provide extensive ablation studies.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 71 of its 81 references that are in the map. First posted on arXiv 28 Apr 2022.

Builds on

71 of 81 references.

References · 71

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  5. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  6. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  7. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  8. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  9. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,678
  10. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  11. 2017Improved Training of Wasserstein GANscited by 2,223
  12. 2020Implicit Neural Representations with Periodic Activation Functionscited by 2,214
38 more
  1. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,037
  2. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,574
  3. 2020D-NeRF: Neural Radiance Fields for Dynamic Scenescited by 1,459
  4. 2020NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collectionscited by 1,230
  5. 2021Efficient Geometry-aware 3D Generative Adversarial Networkscited by 1,204
  6. 2020Nerfies: Deformable Neural Radiance Fieldscited by 1,194
  7. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,186
  8. 2017End-to-end Recovery of Human Shape and Posecited by 883
  9. 2019Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representationscited by 864
  10. 2018Multimodal Unsupervised Image-to-Image Translationcited by 795
  11. 2020GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fieldscited by 784
  12. 2020Convolutional Occupancy Networkscited by 760
  13. 2019Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervisioncited by 699
  14. 2020GRAF: Generative Radiance Fields for 3D-Aware Image Synthesiscited by 688
  15. 2018Pixel2Mesh: Generating 3D Mesh Models from Single RGB Imagescited by 649
  16. 2016Conditional Image Synthesis With Auxiliary Classifier GANscited by 562
  17. 2018AtlasNet: A Papier-Mach'e Approach to Learning 3D Surface Generationcited by 539
  18. 2021StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesiscited by 477
  19. 2021Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesiscited by 464
  20. 2017Neural 3D Mesh Renderercited by 437
  21. 2019Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoningcited by 420
  22. 2020In-Domain GAN Inversion for Real Image Editingcited by 420
  23. 2021In-Place Scene Labelling and Understanding with Implicit Scene Representationcited by 411
  24. 2018DeepVoxels: Learning Persistent 3D Feature Embeddingscited by 401
  25. 2021CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fieldscited by 369
  26. 2021GAN Inversion: A Surveycited by 366
  27. 2018Learning Category-Specific Mesh Reconstruction from Image Collectionscited by 317
  28. 2018Diverse Image-to-Image Translation via Disentangled Representationscited by 296
  29. 2019DRIT++: Diverse Image-to-Image Translation via Disentangled Representationscited by 244
  30. 2020Infinite Nature: Perpetual View Generation of Natural Scenes from a Single Imagecited by 239
  31. 2020Free View Synthesiscited by 231
  32. 2021Editing Conditional Radiance Fieldscited by 222
  33. 2019Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderercited by 221
  34. 2021GRAM: Generative Radiance Manifolds for 3D-Aware Image Generationcited by 216
  35. 2020Swapping Autoencoder for Deep Image Manipulationcited by 215
  36. 2019Texture Fields: Learning Texture Representations in Function Spacecited by 210
  37. 2021Stereo Radiance Fields (SRF): Learning View Synthesis for Sparse Views of Novel Scenescited by 190
  38. 2020Editing in Style: Uncovering the Local Semantics of GANscited by 183

50 most cited of 71 All 71 (JSON)

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 1; 2024: 1.

4 papers.

  1. 2022DM-NeRF: 3D Scene Geometry Decomposition and Manipulation from 2D Imagescited by 69
  2. 2022PaletteNeRF: Palette-based Appearance Editing of Neural Radiance Fieldscited by 43
  3. 2024Semantically-aware Neural Radiance Fields for Visual Scene Understanding: A Comprehensive Reviewcited by 9
  4. 2023BlendFields: Few-Shot Example-Driven Facial Modelingcited by 6