mlsanityData to 7 Oct 2026

Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representation

arXiv 2205.04334 ↗First posted 9 May 2022cs.CVcited by 243

Not on the map. Why?

Abstract (arXiv)

We present Panoptic Neural Fields (PNF), an object-aware neural scene representation that decomposes a scene into a set of objects (things) and background (stuff). Each object is represented by an oriented 3D bounding box and a multi-layer perceptron (MLP) that takes position, direction, and time and outputs density and radiance. The background stuff is represented by a similar MLP that additionally outputs semantic labels. Each object MLPs are instance-specific and thus can be smaller and faster than previous object-aware approaches, while still leveraging category-specific priors incorporated via meta-learned initialization. Our model builds a panoptic radiance field representation of any scene from just color images. We use off-the-shelf algorithms to predict camera poses, object tracks, and 2D image semantic segmentations. Then we jointly optimize the MLP weights and bounding box parameters using analysis-by-synthesis with self-supervision from color images and pseudo-supervision from predicted semantic segmentations. During experiments with real-world dynamic scenes, we find that our model can be used effectively for several tasks like novel view synthesis, 2D panoptic segmentation, 3D scene editing, and multiview depth prediction.

Key facts. Cited by 243 papers in the map (data through 7 Oct 2026). Builds on 54 of its 70 references that are in the map. First posted on arXiv 9 May 2022.

Builds on

54 of 70 references.

References · 54

  1. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  3. 2017Mask R-CNNcited by 7,259
  4. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  5. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  6. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  7. 2015Fast R-CNNcited by 2,930
  8. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  9. 2016Pyramid Scene Parsing Networkcited by 2,472
  10. 2021Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fieldscited by 1,957
  11. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,574
  12. 2020D-NeRF: Neural Radiance Fields for Dynamic Scenescited by 1,459
38 more
  1. 2020Nerfies: Deformable Neural Radiance Fieldscited by 1,194
  2. 2018On First-Order Meta-Learning Algorithmscited by 1,003
  3. 2015ORB-SLAM: a Versatile and Accurate Monocular SLAM Systemcited by 983
  4. 2019Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representationscited by 864
  5. 2020NeRF++: Analyzing and Improving Neural Radiance Fieldscited by 857
  6. 2018Panoptic Segmentationcited by 796
  7. 2020Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenescited by 750
  8. 2020pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesiscited by 708
  9. 2020GRAF: Generative Radiance Fields for 3D-Aware Image Synthesiscited by 688
  10. 2021BARF: Bundle-Adjusting Neural Radiance Fieldscited by 641
  11. 2020Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humanscited by 634
  12. 2021KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3Dcited by 570
  13. 2021Neural Fields in Visual Computing and Beyondcited by 539
  14. 2020Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstructioncited by 465
  15. 2020Space-time Neural Irradiance Fields for Free-Viewpoint Videocited by 414
  16. 2021In-Place Scene Labelling and Understanding with Implicit Scene Representationcited by 411
  17. 2021Advances in Neural Renderingcited by 365
  18. 2019Panoptic-DeepLabcited by 362
  19. 2020Virtual KITTI 2cited by 345
  20. 2016Virtual Worlds as Proxy for Multi-Object Tracking Analysiscited by 340
  21. 2020Neural Scene Graphs for Dynamic Scenescited by 298
  22. 2020Neural Radiance Flow for 4D View Synthesis and Video Processingcited by 270
  23. 2019Improving Federated Learning Personalization via Model Agnostic Meta Learningcited by 264
  24. 2021Is Pseudo-Lidar needed for Monocular 3D Object detection?cited by 248
  25. 2020Learned Initializations for Optimizing Coordinate-Based Neural Representationscited by 240
  26. 2020Free View Synthesiscited by 231
  27. 2021Learning Object-Compositional Neural Radiance Field for Editable Scene Renderingcited by 228
  28. 2020Atlas: End-to-End 3D Scene Reconstruction from Posed Imagescited by 206
  29. 2021Point-Based Neural Rendering with Per-View Optimizationcited by 188
  30. 2021CodeNeRF: Disentangled Neural Radiance Fields for Object Categoriescited by 183
  31. 2017Computer Vision for Autonomous Vehicles: Problems, Datasets and State of the Artcited by 182
  32. 2019Kimera: an Open-Source Library for Real-Time Metric-Semantic Localization and Mappingcited by 181
  33. 2020Total3DUnderstanding: Joint Layout, Object Pose and Mesh Reconstruction for Indoor Scenes from a Single Imagecited by 170
  34. 2022Panoptic NeRF: 3D-to-2D Label Transfer for Panoptic Urban Scene Segmentationcited by 152
  35. 20203D Dynamic Scene Graphs: Actionable Spatial Perception with Places, Objects, and Humanscited by 141
  36. 2019PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Thingscited by 97
  37. 2020Object-Centric Neural Scene Renderingcited by 90
  38. 2020An Analysis of SVD for Deep Rotation Estimationcited by 85

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 22; 2023: 88; 2024: 85; 2025: 32; 2026: 16.

243 papers.

Sort:

By area:

  1. 2023ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenescited by 747
  2. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 485
  3. 2023Gaussian Grouping: Segment and Edit Anything in 3D Scenescited by 387
  4. 2022Decomposing NeRF for Editing via Feature Field Distillationcited by 370
  5. 2023DrivingGaussian: Composite Gaussian Splatting for Surrounding Dynamic Autonomous Driving Scenescited by 359
  6. 2023UniSim: A Neural Closed-Loop Sensor Simulatorcited by 294
  7. 2024Street Gaussians: Modeling Dynamic Urban Scenes with Gaussian Splattingcited by 218
  8. 2023Language Embedded 3D Gaussians for Open-Vocabulary Scene Understandingcited by 205
  9. 2022DiffRF: Rendering-Guided 3D Radiance Field Diffusioncited by 204
  10. 2022Neural Feature Fusion Fields: 3D Distillation of Self-Supervised 2D Image Representationscited by 195
  11. 2022Panoptic Lifting for 3D Scene Understanding with Neural Fieldscited by 171
  12. 2023MARS: An Instance-aware, Modular and Realistic Simulator for Autonomous Drivingcited by 170
38 more
  1. 2024OmniRe: Omni Urban Scene Reconstructioncited by 159
  2. 2022NeRF: Neural Radiance Field in 3D Vision: A Comprehensive Review (Updated Post-Gaussian Splatting)cited by 156
  3. 2023SUDS: Scalable Urban Dynamic Scenescited by 151
  4. 2023NeuRAD: Neural Rendering for Autonomous Drivingcited by 146
  5. 2023Periodic Vibration Gaussian: Dynamic Urban Scene Reconstruction and Real-time Renderingcited by 143
  6. 2024HUGS: Holistic Urban 3D Scene Understanding via Gaussian Splattingcited by 141
  7. 2022D2NeRF: Self-Supervised Decoupling of Dynamic and Static Objects from a Monocular Videocited by 136
  8. 2024MV-DUSt3R+: Single-Stage Scene Reconstruction from Sparse Views In 2 Secondscited by 136
  9. 2024DriveDreamer4D: World Models Are Effective Data Machines for 4D Driving Scene Representationcited by 123
  10. 2023StreetSurf: Extending Multi-view Implicit Surface Reconstruction to Street Viewscited by 120
  11. 2023Language Embedded Radiance Fields for Zero-Shot Task-Oriented Graspingcited by 91
  12. 2024Editable Scene Simulation for Autonomous Driving via Collaborative LLM-Agentscited by 91
  13. 2024ReconDreamer: Crafting World Models for Driving Scene Reconstruction via Online Restorationcited by 89
  14. 2023Set-the-Scene: Global-Local Training for Generating Controllable NeRF Scenescited by 75
  15. 2023UniPAD: A Universal Pre-training Paradigm for Autonomous Drivingcited by 73
  16. 2022NeRF-SOS: Any-View Self-supervised Object Segmentation on Complex Scenescited by 71
  17. 2022DM-NeRF: 3D Scene Geometry Decomposition and Manipulation from 2D Imagescited by 69
  18. 2024OpenNeRF: Open Set 3D Neural Scene Segmentation with Pixel-Wise Features and Rendered Novel Viewscited by 68
  19. 2024SplatAD: Real-Time Lidar and Camera Rendering with 3D Gaussian Splatting for Autonomous Drivingcited by 66
  20. 2023SATR: Zero-Shot Semantic Segmentation of 3D Shapescited by 62
  21. 2023Contrastive Lift: 3D Object Instance Segmentation by Slow-Fast Contrastive Fusioncited by 62
  22. 2022GO-Surf: Neural Feature Grid Optimization for Fast, High-Fidelity RGB-D Surface Reconstructioncited by 61
  23. 2023FreeMask: Synthetic Images with Dense Annotations Make Stronger Segmentation Modelscited by 61
  24. 2023NeO 360: Neural Fields for Sparse View Synthesis of Outdoor Scenescited by 53
  25. 2023NeRF-Det: Learning Geometry-Aware Volumetric Representation for Multi-View 3D Object Detectioncited by 51
  26. 2022IntrinsicNeRF: Learning Intrinsic Neural Radiance Fields for Editable Novel View Synthesiscited by 50
  27. 2023EPIC Fields: Marrying 3D Geometry and Video Understandingcited by 49
  28. 2024Co-Occ: Coupling Explicit Feature Fusion with Volume Rendering Regularization for Multi-Modal 3D Semantic Occupancy Predictioncited by 46
  29. 2023Nerflets: Local Radiance Fields for Efficient Structure-Aware 3D Scene Representation from 2D Supervisioncited by 44
  30. 2023NeRF-LiDAR: Generating Realistic LiDAR Point Clouds with Neural Radiance Fieldscited by 44
  31. 2024Gaga: Group Any Gaussians via 3D-aware Memory Bankcited by 40
  32. 2023Instance Neural Radiance Fieldcited by 39
  33. 2023ObjectSDF++: Improved Object-Compositional Neural Implicit Surfacescited by 38
  34. 2022Feature-Realistic Neural Fusion for Real-Time, Open Set Scene Understandingcited by 37
  35. 2023Preface: A Data-driven Volumetric Prior for Few-shot Ultra High-resolution Face Synthesiscited by 37
  36. 2025ReconDreamer++: Harmonizing Generative and Reconstructive Models for Driving Scene Representationcited by 36
  37. 2024Dynamic 3D Gaussian Fields for Urban Areascited by 35
  38. 2023Zero-Shot 3D Shape Correspondencecited by 33

50 most cited of 243 All 243 (JSON)