mlsanityData to 7 Oct 2026

Nerflets: Local Radiance Fields for Efficient Structure-Aware 3D Scene Representation from 2D Supervision

arXiv 2303.03361 ↗First posted 6 Mar 2023cs.CV cs.GRcited by 44

Not on the map. Why?

Abstract (arXiv)

We address efficient and structure-aware 3D scene representation from images. Nerflets are our key contribution -- a set of local neural radiance fields that together represent a scene. Each nerflet maintains its own spatial position, orientation, and extent, within which it contributes to panoptic, density, and radiance reconstructions. By leveraging only photometric and inferred panoptic image supervision, we can directly and jointly optimize the parameters of a set of nerflets so as to form a decomposed representation of the scene, where each object instance is represented by a group of nerflets. During experiments with indoor and outdoor environments, we find that nerflets: (1) fit and approximate the scene more efficiently than traditional global NeRFs, (2) allow the extraction of panoptic and photometric renderings from arbitrary views, and (3) enable tasks rare for NeRFs, such as 3D panoptic segmentation and interactive editing.

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). Builds on 54 of its 63 references that are in the map. First posted on arXiv 6 Mar 2023.

Builds on

54 of 63 references.

References · 54

  1. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  4. 2017Mask R-CNNcited by 7,259
  5. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  6. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  7. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  8. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  9. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
  10. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  11. 2016Pyramid Scene Parsing Networkcited by 2,472
  12. 2021Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fieldscited by 2,013
38 more
  1. 2021Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fieldscited by 1,957
  2. 2015SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentationcited by 1,483
  3. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
  4. 2019KPConv: Flexible and Deformable Convolution for Point Cloudscited by 1,202
  5. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,201
  6. 2021Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstructioncited by 961
  7. 2020Neural Sparse Voxel Fieldscited by 959
  8. 2019PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detectioncited by 890
  9. 2020NeRF++: Analyzing and Improving Neural Radiance Fieldscited by 857
  10. 2018Panoptic Segmentationcited by 796
  11. 2021KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPscited by 672
  12. 2021KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3Dcited by 570
  13. 2016Semantic Scene Completion from a Single Depth Imagecited by 468
  14. 2020ResNeSt: Split-Attention Networkscited by 445
  15. 2021In-Place Scene Labelling and Understanding with Implicit Scene Representationcited by 411
  16. 2021Mega-NeRF: Scalable Construction of Large-Scale NeRFs for Virtual Fly-Throughscited by 396
  17. 2022Decomposing NeRF for Editing via Feature Field Distillationcited by 370
  18. 2019Panoptic-DeepLabcited by 362
  19. 2016OctNet: Learning Deep 3D Representations at High Resolutionscited by 340
  20. 2020Neural Scene Graphs for Dynamic Scenescited by 298
  21. 20193D Scene Graph: A Structure for Unified Semantics, 3D Space, and Cameracited by 280
  22. 2021Mixture of Volumetric Primitives for Efficient Neural Renderingcited by 277
  23. 2022Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representationcited by 243
  24. 2019Learning Shape Templates with Structured Implicit Functionscited by 242
  25. 2020Free View Synthesiscited by 231
  26. 2021Learning Object-Compositional Neural Radiance Field for Editable Scene Renderingcited by 228
  27. 2021Point-Based Neural Rendering with Per-View Optimizationcited by 188
  28. 2019Learning Object Bounding Boxes for 3D Instance Segmentation on Point Cloudscited by 156
  29. 2016SemanticFusion: Dense 3D Semantic Mapping with Convolutional Neural Networkscited by 134
  30. 2021NeSF: Neural Semantic Fields for Generalizable Semantic Segmentation of 3D Scenescited by 119
  31. 20183DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentationcited by 118
  32. 2020Virtual Multi-view Fusion for 3D Semantic Segmentationcited by 99
  33. 2022NeRFusion: Fusing Radiance Fields for Large-Scale Scene Reconstructioncited by 99
  34. 2019PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Thingscited by 97
  35. 2015Sparse 3D convolutional neural networkscited by 71
  36. 2018MeshCNN: A Network with an Edgecited by 69
  37. 2022DM-NeRF: 3D Scene Geometry Decomposition and Manipulation from 2D Imagescited by 69
  38. 2019JSIS3D: Joint Semantic-Instance Segmentation of 3D Point Clouds with Multi-Task Pointwise Networks and Multi-Value Conditional Random Fieldscited by 62

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 16; 2024: 18; 2025: 6; 2026: 4.

44 papers.

Sort:

By area:

  1. 2023Gaussian Grouping: Segment and Edit Anything in 3D Scenescited by 387
  2. 2024Street Gaussians: Modeling Dynamic Urban Scenes with Gaussian Splattingcited by 218
  3. 2023Language Embedded 3D Gaussians for Open-Vocabulary Scene Understandingcited by 205
  4. 2024HUGSIM: A Real-Time, Photo-Realistic and Closed-Loop Simulator for Autonomous Drivingcited by 84
  5. 2023Set-the-Scene: Global-Local Training for Generating Controllable NeRF Scenescited by 75
  6. 2023SSCBench: A Large-Scale 3D Semantic Scene Completion Benchmark for Autonomous Drivingcited by 72
  7. 2023GAvatar: Animatable 3D Gaussian Avatars with Implicit Mesh Learningcited by 63
  8. 2023EPIC Fields: Marrying 3D Geometry and Video Understandingcited by 49
  9. 2023Flexible Techniques for Differentiable Rendering with 3D Gaussianscited by 28
  10. 2023Blended-NeRF: Zero-Shot Object Generation and Blending in Existing Neural Radiance Fieldscited by 23
  11. 2023Gaussian3Diff: 3D Gaussian Diffusion for 3D Full Head Synthesis and Editingcited by 22
  12. 2023Total-Recon: Deformable Scene Reconstruction for Embodied View Synthesiscited by 20
32 more
  1. 2024DC-Gaussian: Improving 3D Gaussian Splatting for Reflective Dash Cam Videoscited by 16
  2. 2024Neural Directional Encoding for Efficient and Accurate View-Dependent Appearance Modelingcited by 15
  3. 2024RoDUS: Robust Decomposition of Static and Dynamic Elements in Urban Scenescited by 14
  4. 2024LidaRF: Delving into Lidar for Neural Radiance Field on Street Scenescited by 14
  5. 2023PointNeRF++: A multi-scale, point-based Neural Radiance Fieldcited by 13
  6. 2024Lightning NeRF: Efficient Hybrid Scene Representation for Autonomous Drivingcited by 12
  7. 2024Neural Radiance Field-based Visual Rendering: A Comprehensive Reviewcited by 12
  8. 2025CF3: Compact and Fast 3D Feature Fieldscited by 11
  9. 2023PanopticNeRF-360: Panoramic 3D-to-2D Label Transfer in Urban Scenescited by 9
  10. 2024Semantically-aware Neural Radiance Fields for Visual Scene Understanding: A Comprehensive Reviewcited by 9
  11. 2024ConDense: Consistent 2D/3D Pre-training for Dense and Sparse Features from Multi-View Imagescited by 9
  12. 2023PlaNeRF: SVD Unsupervised 3D Plane Regularization for NeRF Large-Scale Scene Reconstructioncited by 8
  13. 2024UrbanCAD: Towards Highly Controllable and Photorealistic 3D Vehicles for Urban Scene Simulationcited by 7
  14. 2023Factored Neural Representation for Scene Understandingcited by 4
  15. 20243D StreetUnveiler with Semantic-aware 2DGS -- a simple baselinecited by 4
  16. 2024ProvNeRF: Modeling per Point Provenance in NeRFs as a Stochastic Fieldcited by 3
  17. 2024Neural Graph Map: Dense Mapping with Efficient Loop Closure Integrationcited by 3
  18. 2024MomentsNeRF: Leveraging Orthogonal Moments for Few-Shot Neural Renderingcited by 2
  19. 2025NeRFs are Mirror Detectors: Using Structural Similarity for Multi-View Mirror Scene Reconstruction with 3D Surface Primitivescited by 2
  20. 2023Self-supervised novel 2D view synthesis of large-scale scenes with efficient multi-scale voxel carvingcited by 1
  21. 2024InfoGaussian: Structure-Aware Dynamic Gaussians through Lightweight Information Shapingcited by 1
  22. 2023Survey on Controlable Image Synthesis with Deep Learningcited by 0
  23. 2024Methods and strategies for improving the novel view synthesis quality of neural radiation fieldcited by 0
  24. 2024LeC2O-NeRF: Learning Continuous and Compact Large-Scale Occupancy for Urban Scenescited by 0
  25. 2025A self-supervised cyclic neural-analytic approach for novel view synthesis and 3D reconstructioncited by 0
  26. 2025Layered Motion Fusion: Lifting Motion Segmentation to 3D in Egocentric Videoscited by 0
  27. 2025Embodied Spatial Intelligence: from Implicit Scene Modeling to Spatial Reasoningcited by 0
  28. 2025Taming the Light: Illumination-Invariant Semantic 3DGS-SLAMcited by 0
  29. 2026LR-SGS: Robust LiDAR-Reflectance-Guided Salient Gaussian Splatting for Self-Driving Scene Reconstructioncited by 0
  30. 2026Recent Advances and Trends in Learning-based 3D Representationscited by 0
  31. 2026Relation-Centric Open-Vocabulary 3D Gaussian Segmentationcited by 0
  32. 2026Efficient Human-Contact Representation for Human-Scene Interactioncited by 0