mlsanityData to 14 Oct 2026

In-Place Scene Labelling and Understanding with Implicit Scene Representation

arXiv 2103.15875 ↗First posted 29 Mar 2021cs.CVcited by 415

Not on the map. Why?

Abstract (arXiv)

Semantic labelling is highly correlated with geometry and radiance reconstruction, as scene entities with similar shape and appearance are more likely to come from similar classes. Recent implicit neural reconstruction techniques are appealing as they do not require prior training data, but the same fully self-supervised approach is not possible for semantics because labels are human-defined properties. We extend neural radiance fields (NeRF) to jointly encode semantics with appearance and geometry, so that complete and accurate 2D semantic labels can be achieved using a small amount of in-place annotations specific to the scene. The intrinsic multi-view consistency and smoothness of NeRF benefit semantics by enabling sparse labels to efficiently propagate. We show the benefit of this approach when labels are either sparse or very noisy in room-scale scenes. We demonstrate its advantageous properties in various interesting applications such as an efficient scene labelling tool, novel semantic view synthesis, label denoising, super-resolution, label interpolation and multi-view semantic label fusion in visual semantic mapping systems.

Key facts. Cited by 415 papers in the map (data through 14 Oct 2026). Builds on 28 of its 38 references that are in the map. First posted on arXiv 29 Mar 2021.

Builds on

28 of 38 references.

References · 28

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,223
  5. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  6. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,689
  7. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,200
  8. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,041
  9. 2020NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collectionscited by 1,232
  10. 2019Habitat: A Platform for Embodied AI Researchcited by 1,191
  11. 2019The Replica Dataset: A Digital Replica of Indoor Spacescited by 904
  12. 2019Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representationscited by 864
16 more
  1. 2020NeRF++: Analyzing and Improving Neural Radiance Fieldscited by 861
  2. 2020Convolutional Occupancy Networkscited by 761
  3. 2018Taskonomy: Disentangling Task Transfer Learningcited by 634
  4. 2018End-to-End Multi-Task Learning with Attentioncited by 534
  5. 2020NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesiscited by 496
  6. 2014Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecturecited by 483
  7. 2019DIST: Rendering Deep Implicit Signed Distance Function with Differentiable Sphere Tracingcited by 180
  8. 2018CodeSLAM - Learning a Compact, Optimisable Representation for Dense Visual SLAMcited by 152
  9. 2016SemanticFusion: Dense 3D Semantic Mapping with Convolutional Neural Networkscited by 135
  10. 2018Fusion++: Volumetric Object-Level SLAMcited by 98
  11. 2019PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Thingscited by 98
  12. 2018MaskFusion: Real-Time Recognition, Tracking and Reconstruction of Multiple Moving Objectscited by 95
  13. 2019SceneCode: Monocular Dense Semantic Reconstruction using Learned Encoded Scene Representationscited by 37
  14. 2016Meaningful Maps With Object-Oriented Semantic Mappingcited by 31
  15. 2020Semantic Implicit Neural Scene Representations With Semi-Supervised Trainingcited by 9
  16. 2018Fast and Accurate Semantic Mapping through Geometric-based Incremental Segmentationcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 15; 2022: 45; 2023: 115; 2024: 120; 2025: 86; 2026: 34.

415 papers.

Sort:

By area:

  1. 2023ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenescited by 764
  2. 2023HexPlane: A Fast Representation for Dynamic Scenescited by 587
  3. 2023LERF: Language Embedded Radiance Fieldscited by 543
  4. 2021Neural Fields in Visual Computing and Beyondcited by 540
  5. 2023LangSplat: 3D Language Gaussian Splattingcited by 415
  6. 2023Gaussian Grouping: Segment and Edit Anything in 3D Scenescited by 393
  7. 2022Kubric: A scalable dataset generatorcited by 381
  8. 2022Decomposing NeRF for Editing via Feature Field Distillationcited by 371
  9. 2021Advances in Neural Renderingcited by 365
  10. 2023Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fieldscited by 364
  11. 2021NeRF in the Dark: High Dynamic Range View Synthesis from Noisy Raw Imagescited by 336
  12. 2023GaussianEditor: Swift and Controllable 3D Editing with Gaussian Splattingcited by 335
38 more
  1. 2021Light Field Networks: Neural Scene Representations with Single-Evaluation Renderingcited by 332
  2. 2021Urban Radiance Fieldscited by 263
  3. 2022Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representationcited by 243
  4. 2021Learning Object-Compositional Neural Radiance Field for Editable Scene Renderingcited by 228
  5. 2023Language Embedded 3D Gaussians for Open-Vocabulary Scene Understandingcited by 208
  6. 2022Neural Feature Fusion Fields: 3D Distillation of Self-Supervised 2D Image Representationscited by 196
  7. 2022Panoptic Lifting for 3D Scene Understanding with Neural Fieldscited by 174
  8. 2023MARS: An Instance-aware, Modular and Realistic Simulator for Autonomous Drivingcited by 170
  9. 2023Segment Any 3D Gaussianscited by 161
  10. 2022NeRF: Neural Radiance Field in 3D Vision: A Comprehensive Review (Updated Post-Gaussian Splatting)cited by 158
  11. 2022Panoptic NeRF: 3D-to-2D Label Transfer for Panoptic Urban Scene Segmentationcited by 152
  12. 2023SUDS: Scalable Urban Dynamic Scenescited by 151
  13. 2022SPIn-NeRF: Multiview Segmentation and Perceptual Inpainting with Neural Radiance Fieldscited by 144
  14. 2024HUGS: Holistic Urban 3D Scene Understanding via Gaussian Splattingcited by 142
  15. 2023Weakly Supervised 3D Open-vocabulary Segmentationcited by 135
  16. 2022Neural 3D Scene Reconstruction with the Manhattan-world Assumptioncited by 123
  17. 2021NeSF: Neural Semantic Fields for Generalizable Semantic Segmentation of 3D Scenescited by 119
  18. 2022Semantic-Aware Implicit Neural Audio-Driven Video Portrait Generationcited by 119
  19. 2022PartSLIP: Low-Shot Part Segmentation for 3D Point Clouds via Pretrained Image-Language Modelscited by 118
  20. 2021FENeRF: Face Editing in Neural Radiance Fieldscited by 112
  21. 2024GARField: Group Anything with Radiance Fieldscited by 109
  22. 2023Language Embedded Radiance Fields for Zero-Shot Task-Oriented Graspingcited by 91
  23. 2024HUGSIM: A Real-Time, Photo-Realistic and Closed-Loop Simulator for Autonomous Drivingcited by 90
  24. 2024FreeSplat: Generalizable 3D Gaussian Splatting Towards Free-View Synthesis of Indoor Scenescited by 89
  25. 2024Semantic Gaussians: Open-Vocabulary Scene Understanding with 3D Gaussian Splattingcited by 82
  26. 2023StyleRF: Zero-shot 3D Style Transfer of Neural Radiance Fieldscited by 79
  27. 2021Learning to Stylize Novel Viewscited by 76
  28. 2023OmniSeg3D: Omniversal 3D Segmentation via Hierarchical Contrastive Learningcited by 75
  29. 2022Object-Compositional Neural Implicit Surfacescited by 74
  30. 2023UniPAD: A Universal Pre-training Paradigm for Autonomous Drivingcited by 73
  31. 2024Large Spatial Model: End-to-end Unposed Images to Semantic 3Dcited by 73
  32. 2022NeRF-SOS: Any-View Self-supervised Object Segmentation on Complex Scenescited by 71
  33. 2023Learning Neural Volumetric Representations of Dynamic Humans in Minutescited by 70
  34. 2022DM-NeRF: 3D Scene Geometry Decomposition and Manipulation from 2D Imagescited by 69
  35. 2024OpenNeRF: Open Set 3D Neural Scene Segmentation with Pixel-Wise Features and Rendered Novel Viewscited by 69
  36. 2023Contrastive Lift: 3D Object Instance Segmentation by Slow-Fast Contrastive Fusioncited by 64
  37. 2024Advances in 3D Generation: A Surveycited by 63
  38. 2023SATR: Zero-Shot Semantic Segmentation of 3D Shapescited by 62

50 most cited of 415 All 415 (JSON)