mlsanityData to 14 Oct 2026

Sem2NeRF: Converting Single-View Semantic Masks to Neural Radiance Fields

arXiv 2203.10821 ↗First posted 21 Mar 2022cs.CV cs.AI cs.GRcited by 29

Not on the map. Why?

Abstract (arXiv)

Image translation and manipulation have gain increasing attention along with the rapid development of deep generative models. Although existing approaches have brought impressive results, they mainly operated in 2D space. In light of recent advances in NeRF-based 3D-aware generative models, we introduce a new task, Semantic-to-NeRF translation, that aims to reconstruct a 3D scene modelled by NeRF, conditioned on one single-view semantic mask as input. To kick-off this novel task, we propose the Sem2NeRF framework. In particular, Sem2NeRF addresses the highly challenging task by encoding the semantic mask into the latent code that controls the 3D scene representation of a pre-trained decoder. To further improve the accuracy of the mapping, we integrate a new region-aware learning strategy into the design of both the encoder and the decoder. We verify the efficacy of the proposed Sem2NeRF and demonstrate that it outperforms several strong baselines on two benchmark datasets. Code and video are available at https://donydchen.github.io/sem2nerf/

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Builds on 63 of its 74 references that are in the map. First posted on arXiv 21 Mar 2022.

Builds on

63 of 74 references.

References · 63

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  6. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  7. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  8. 2022Generative Adversarial Networkscited by 5,627
  9. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  10. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,527
  11. 2014Deep Learning Face Attributes in the Wildcited by 3,693
  12. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,588
38 more
  1. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,241
  2. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,988
  3. 2016Improved Techniques for Training GANscited by 2,970
  4. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,694
  5. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,333
  6. 2020Implicit Neural Representations with Periodic Activation Functionscited by 2,238
  7. 2021Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fieldscited by 1,964
  8. 2020D-NeRF: Neural Radiance Fields for Dynamic Scenescited by 1,465
  9. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,299
  10. 2021Efficient Geometry-aware 3D Generative Adversarial Networkscited by 1,205
  11. 2017High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANscited by 1,179
  12. 2019StarGAN v2: Diverse Image Synthesis for Multiple Domainscited by 1,098
  13. 2018Learning Implicit Fields for Generative Shape Modelingcited by 1,060
  14. 2021Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstructioncited by 964
  15. 2020Neural Sparse Voxel Fieldscited by 959
  16. 2020GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fieldscited by 785
  17. 2021Diffusion Probabilistic Models for 3D Point Cloud Generationcited by 735
  18. 2020Encoding in Style: a StyleGAN Encoder for Image-to-Image Translationcited by 727
  19. 2020pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesiscited by 708
  20. 2020GRAF: Generative Radiance Fields for 3D-Aware Image Synthesiscited by 689
  21. 2018Which Training Methods for GANs do actually Converge?cited by 679
  22. 2019MaskGAN: Towards Diverse and Interactive Facial Image Manipulationcited by 628
  23. 2016Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modelingcited by 558
  24. 2021StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesiscited by 477
  25. 2021In-Place Scene Labelling and Understanding with Implicit Scene Representationcited by 415
  26. 2019Making Convolutional Networks Shift-Invariant Againcited by 391
  27. 2020INeRF: Inverting Neural Radiance Fields for Pose Estimationcited by 382
  28. 2020StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generationcited by 332
  29. 2018Learning Category-Specific Mesh Reconstruction from Image Collectionscited by 317
  30. 2017Toward Multimodal Image-to-Image Translationcited by 296
  31. 2021Mixture of Volumetric Primitives for Efficient Neural Renderingcited by 277
  32. 2021DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effortcited by 264
  33. 2021HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fieldscited by 214
  34. 2018Stereo Magnification: Learning View Synthesis using Multiplane Imagescited by 211
  35. 2020Editing in Style: Uncovering the Local Semantics of GANscited by 183
  36. 2019Pluralistic Image Completioncited by 162
  37. 2021EditGAN: High-Precision Semantic Image Editingcited by 156
  38. 2021CIPS-3D: A 3D-Aware Generator of GANs Based on Conditionally-Independent Pixel Synthesiscited by 149

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 11; 2023: 10; 2024: 5; 2025: 2; 2026: 1.

29 papers.

Sort:

By area:

  1. 2022CelebV-HQ: A Large-Scale Video Facial Attributes Datasetcited by 250
  2. 2022NeRF: Neural Radiance Field in 3D Vision: A Comprehensive Review (Updated Post-Gaussian Splatting)cited by 158
  3. 2022Next3D: Generative Neural Texture Rasterization for 3D-Aware Head Avatarscited by 124
  4. 2024MVSplat360: Feed-Forward 360 Scene Synthesis from Sparse Viewscited by 116
  5. 2022Object-Compositional Neural Implicit Surfacescited by 74
  6. 2022Controllable 3D Face Synthesis with Conditional Generative Occupancy Fieldscited by 37
  7. 2022NeRFFaceEditing: Disentangled Face Editing in Neural Radiance Fieldscited by 33
  8. 20233D-aware Conditional Image Synthesiscited by 24
  9. 2022A Survey on Deep Generative 3D-aware Image Synthesiscited by 16
  10. 2023Semantic 3D-aware Portrait Synthesis and Manipulation Based on Compositional Neural Radiance Fieldcited by 10
  11. 2022Training and Tuning Generative Neural Radiance Fields for Attribute-Conditional 3D-Aware Face Generationcited by 8
  12. 20223D-Aware Encoding for Style-based Neural Radiance Fieldscited by 5
17 more
  1. 2022LatentSwap3D: Semantic Edits on 3D Image GANscited by 5
  2. 20234D-Editor: Interactive Object-level Editing in Dynamic Neural Radiance Fields via Semantic Distillationcited by 5
  3. 20233D-CLFusion: Fast Text-to-3D Rendering with Contrastive Latent Diffusioncited by 4
  4. 2023MaTe3D: Mask-guided Text-based 3D-aware Portrait Editingcited by 3
  5. 2024SemanticHuman-HD: High-Resolution Semantic Disentangled 3D Human Generationcited by 3
  6. 2023Make Your Brief Stroke Real and Stereoscopic: 3D-Aware Simplified Sketch to Portrait Generationcited by 2
  7. 2023Painting 3D Nature in 2D: View Synthesis of Natural Scenes from a Single Semantic Maskcited by 2
  8. 20243D-SSGAN: Lifting 2D Semantics for 3D-Aware Compositional Portrait Synthesiscited by 2
  9. 2024InfoNorm: Mutual Information Shaping of Normals for Sparse-View Reconstructioncited by 2
  10. 2023MM-NeRF: Multimodal-Guided 3D Multi-Style Transfer of Neural Radiance Fieldcited by 1
  11. 2025FFaceNeRF: Few-shot Face Editing in Neural Radiance Fieldscited by 1
  12. 2022CGOF++: Controllable 3D Face Synthesis with Conditional Generative Occupancy Fieldscited by 0
  13. 2023Freestyle 3D-Aware Portrait Synthesis Based on Compositional Generative Priorscited by 0
  14. 2023BluNF: Blueprint Neural Fieldcited by 0
  15. 2024Pomo3D: 3D-Aware Portrait Accessorizing and Morecited by 0
  16. 2025Cooperative Perception: A Resource-Efficient Framework for Multi-Drone 3D Scene Reconstruction Using Federated Diffusion and NeRFcited by 0
  17. 2026Learning Spectral and Polarimetric Clues for One-to-Multimodal Novel View Synthesiscited by 0