mlsanityData to 7 Oct 2026

NeRF-RPN: A general framework for object detection in NeRFs

arXiv 2211.11646 ↗First posted 21 Nov 2022cs.CVcited by 46

Not on the map. Why?

Abstract (arXiv)

This paper presents the first significant object detection framework, NeRF-RPN, which directly operates on NeRF. Given a pre-trained NeRF model, NeRF-RPN aims to detect all bounding boxes of objects in a scene. By exploiting a novel voxel representation that incorporates multi-scale 3D neural volumetric features, we demonstrate it is possible to regress the 3D bounding boxes of objects in NeRF directly without rendering the NeRF at any viewpoint. NeRF-RPN is a general framework and can be applied to detect objects without class labels. We experimented NeRF-RPN with various backbone architectures, RPN head designs and loss functions. All of them can be trained in an end-to-end manner to estimate high quality 3D bounding boxes. To facilitate future research in object detection for NeRF, we built a new benchmark dataset which consists of both synthetic and real-world data with careful labeling and clean up. Code and dataset are available at https://github.com/lyclyc52/NeRF_RPN.

Key facts. Cited by 46 papers in the map (data through 7 Oct 2026). Builds on 63 of its 74 references that are in the map. First posted on arXiv 21 Nov 2022.

Builds on

63 of 74 references.

References · 63

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  6. 2020End-to-End Object Detection with Transformerscited by 7,384
  7. 2017Mask R-CNNcited by 7,259
  8. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  9. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  10. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
  11. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  12. 2015SSD: Single Shot MultiBox Detectorcited by 3,125
38 more
  1. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  2. 2015Fast R-CNNcited by 2,930
  3. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,160
  4. 2019FCOS: Fully Convolutional One-Stage Object Detectioncited by 1,573
  5. 2017VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detectioncited by 1,323
  6. 2022TensoRF: Tensorial Radiance Fieldscited by 1,275
  7. 2019Objects as Pointscited by 1,172
  8. 2018PointRCNN: 3D Object Proposal Generation and Detection from Point Cloudcited by 939
  9. 2021PlenOctrees for Real-time Rendering of Neural Radiance Fieldscited by 904
  10. 2018CornerNet: Detecting Objects as Paired Keypointscited by 763
  11. 2017Focal Loss for Dense Object Detectioncited by 749
  12. 20173D Semantic Segmentation with Submanifold Sparse Convolutional Networkscited by 746
  13. 2014Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognitioncited by 719
  14. 2021DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queriescited by 655
  15. 2019CenterNet: Keypoint Triplets for Object Detectioncited by 654
  16. 2019Deep Hough Voting for 3D Object Detection in Point Cloudscited by 648
  17. 2017Frustum PointNets for 3D Object Detection from RGB-D Datacited by 640
  18. 2020Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understandingcited by 593
  19. 2022PETR: Position Embedding Transformation for Multi-View 3D Object Detectioncited by 550
  20. 2018Pseudo-LiDAR from Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Drivingcited by 455
  21. 2019Distance-IoU Loss: Faster and Better Learning for Bounding Box Regressioncited by 454
  22. 2021FCOS3D: Fully Convolutional One-Stage Monocular 3D Object Detectioncited by 444
  23. 20203D-FRONT: 3D Furnished Rooms with layOuts and semaNTicscited by 375
  24. 2021An End-to-End Transformer Model for 3D Object Detectioncited by 329
  25. 2021Dense Depth Priors for Neural Radiance Fields from Sparse Input Viewscited by 301
  26. 2014One weird trick for parallelizing convolutional neural networkscited by 264
  27. 2019M3D-RPN: Monocular 3D Region Proposal Network for Object Detectioncited by 251
  28. 2021Voxel Transformer for 3D Object Detectioncited by 247
  29. 2021PV-RCNN++: Point-Voxel Feature Set Abstraction With Local Vector Representation for 3D Object Detectioncited by 232
  30. 2021Oriented R-CNN for Object Detectioncited by 230
  31. 2019Disentangling Monocular 3D Object Detectioncited by 215
  32. 2021Group-Free 3D Object Detection via Transformerscited by 215
  33. 2018Orthographic Feature Transform for Monocular 3D Object Detectioncited by 201
  34. 20203D Object Detection with Pointformercited by 194
  35. 2019Pseudo-LiDAR++: Accurate Depth for 3D Object Detection in Autonomous Drivingcited by 188
  36. 2021ImVoxelNet: Image to Voxels Projection for Monocular and Multi-View General-Purpose 3D Object Detectioncited by 157
  37. 2019Bottom-up Object Detection by Grouping Extreme and Center Pointscited by 156
  38. 2017Receptive Field Block Net for Accurate and Fast Object Detectioncited by 140

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 13; 2024: 21; 2025: 10; 2026: 2.

46 papers.

Sort:

By area:

  1. 2023NeRF-Det: Learning Geometry-Aware Volumetric Representation for Multi-View 3D Object Detectioncited by 51
  2. 2023Instance Neural Radiance Fieldcited by 39
  3. 2024The NeRFect Match: Exploring NeRF Features for Visual Localizationcited by 32
  4. 2024DDN-SLAM: Real-time Dense Dynamic Neural Implicit SLAMcited by 22
  5. 2024Neural Fields in Robotics: A Surveycited by 21
  6. 2023GP-NeRF: Generalized Perception NeRF for Context-Aware 3D Scene Understandingcited by 20
  7. 2023SANeRF-HQ: Segment Anything for NeRF in High Qualitycited by 20
  8. 2024NeRF-MAE: Masked AutoEncoders for Self-Supervised 3D Representation Learning for Neural Radiance Fieldscited by 20
  9. 2024NeRFs in Robotics: A Surveycited by 16
  10. 2024RelationField: Relate Anything in Radiance Fieldscited by 16
  11. 2024DaReNeRF: Direction-aware Representation for Dynamic Scenescited by 15
  12. 2024ChatCam: Empowering Camera Control through Conversational AIcited by 15
34 more
  1. 2025Tackling View-Dependent Semantics in 3D Language Gaussian Splattingcited by 15
  2. 2023Pixel-Aligned Recurrent Queries for Multi-View 3D Object Detectioncited by 14
  3. 20243DGS-DET: Empower 3D Gaussian Splatting with Boundary Guidance and Box-Focused Sampling for Indoor 3D Object Detectioncited by 13
  4. 2024Neural Radiance Field-based Visual Rendering: A Comprehensive Reviewcited by 12
  5. 2025Neural Radiance Fields for the Real World: A Surveycited by 12
  6. 2025FreeSplat++: Generalizable 3D Gaussian Splatting for Efficient Indoor Scene Reconstructioncited by 12
  7. 2023Scene-Generalizable Interactive Segmentation of Radiance Fieldscited by 10
  8. 2024Semantically-aware Neural Radiance Fields for Visual Scene Understanding: A Comprehensive Reviewcited by 9
  9. 2024Finding Waldo: Towards Efficient Exploration of NeRF Scene Spacescited by 8
  10. 2023HarmonicNeRF: Geometry-Informed Synthetic View Augmentation for 3D Scene Reconstruction in Driving Scenarioscited by 7
  11. 2025TransDiff: Diffusion-Based Method for Manipulating Transparent Objects Using a Single RGB-D Imagecited by 7
  12. 2023Clean-NeRF: Reformulating NeRF to account for View-Dependent Observationscited by 6
  13. 2023Teaching Robots to Build Simulations of Themselvescited by 6
  14. 2024VSRD: Instance-Aware Volumetric Silhouette Rendering for Weakly Supervised 3D Object Detectioncited by 6
  15. 2024Connecting NeRFs, Images, and Textcited by 6
  16. 2024LLaNA: Large Language and NeRF Assistantcited by 6
  17. 2024Gaussian-Det: Learning Closed-Surface Gaussians for 3D Object Detectioncited by 6
  18. 2026VGGT-Det: Mining VGGT Internal Priors for Sensor-Geometry-Free Multi-View Indoor 3D Object Detectioncited by 6
  19. 2025GO-N3RDet: Geometry Optimized NeRF-enhanced 3D Object Detectorcited by 5
  20. 2024NeRF-Det++: Incorporating Semantic Cues and Perspective-aware Depth Supervision for Indoor Multi-View 3D Detectioncited by 4
  21. 2023Registering Neural Radiance Fields as 3D Density Imagescited by 2
  22. 2024NeRF-DetS: Enhanced Adaptive Spatial-wise Sampling and View-wise Fusion Strategies for NeRF-based Indoor Multi-view 3D Object Detectioncited by 2
  23. 2025UnMix-NeRF: Spectral Unmixing Meets Neural Radiance Fieldscited by 2
  24. 2023uSF: Learning Neural Semantic Field with Uncertaintycited by 1
  25. 2023Hyper-VolTran: Fast and Generalizable One-Shot Image to 3D Object Structure via HyperNetworkscited by 1
  26. 2024KRONC: Keypoint-based Robust Camera Optimization for 3D Car Reconstructioncited by 1
  27. 2024TeX-NeRF: Neural Radiance Fields from Pseudo-TeX Visioncited by 1
  28. 2026Scene Grounding In the Wildcited by 1
  29. 2023InceptionHuman: Controllable Prompt-to-NeRF for Photorealistic 3D Human Generationcited by 0
  30. 2024DaRePlane: Direction-aware Representations for Dynamic Scene Reconstructioncited by 0
  31. 2025Weight Space Representation Learning on Diverse NeRF Architecturescited by 0
  32. 2025Scaling LLaNA: Advancing NeRF-Language Understanding Through Large-Scale Trainingcited by 0
  33. 2025AegisRF: Adversarial Perturbations Guided with Sensitivity for Protecting Intellectual Property of Neural Radiance Fieldscited by 0
  34. 2025VSRD++: Autolabeling for 3D Object Detection via Instance-Aware Volumetric Silhouette Renderingcited by 0