mlsanityData to 7 Oct 2026

NeRF-Loc: Transformer-Based Object Localization Within Neural Radiance Fields

arXiv 2209.12068 ↗First posted 24 Sep 2022cs.CV cs.ROcited by 7

Not on the map. Why?

Abstract (arXiv)

Neural Radiance Fields (NeRFs) have become a widely-applied scene representation technique in recent years, showing advantages for robot navigation and manipulation tasks. To further advance the utility of NeRFs for robotics, we propose a transformer-based framework, NeRF-Loc, to extract 3D bounding boxes of objects in NeRF scenes. NeRF-Loc takes a pre-trained NeRF model and camera view as input and produces labeled, oriented 3D bounding boxes of objects as output. Using current NeRF training tools, a robot can train a NeRF environment model in real-time and, using our algorithm, identify 3D bounding boxes of objects of interest within the NeRF for downstream navigation or manipulation tasks. Concretely, we design a pair of paralleled transformer encoder branches, namely the coarse stream and the fine stream, to encode both the context and details of target objects. The encoded features are then fused together with attention layers to alleviate ambiguities for accurate object localization. We have compared our method with conventional RGB(-D) based methods that take rendered RGB images and depths from NeRFs as inputs. Our method is better than the baselines.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 39 of its 49 references that are in the map. First posted on arXiv 24 Sep 2022.

Builds on

39 of 49 references.

References · 39

  1. 2020End-to-End Object Detection with Transformerscited by 7,384
  2. 2017Mask R-CNNcited by 7,259
  3. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,574
  4. 2019Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regressioncited by 1,361
  5. 2018PointRCNN: 3D Object Proposal Generation and Detection from Point Cloudcited by 939
  6. 2020GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fieldscited by 784
  7. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  8. 2017PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenescited by 768
  9. 2016Multi-View 3D Object Detection Network for Autonomous Drivingcited by 721
  10. 2022Block-NeRF: Scalable Large Scene Neural View Synthesiscited by 720
  11. 2021DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queriescited by 655
  12. 2017Frustum PointNets for 3D Object Detection from RGB-D Datacited by 640
27 more
  1. 2021NICE-SLAM: Neural Implicit Scalable Encoding for SLAMcited by 586
  2. 2021iMAP: Implicit Mapping and Positioning in Real-Timecited by 476
  3. 2021In-Place Scene Labelling and Understanding with Implicit Scene Representationcited by 411
  4. 2020INeRF: Inverting Neural Radiance Fields for Pose Estimationcited by 381
  5. 2019DenseFusion: 6D Object Pose Estimation by Iterative Dense Fusioncited by 352
  6. 2021An End-to-End Transformer Model for 3D Object Detectioncited by 329
  7. 2019PVN3D: A Deep Point-wise 3D Keypoints Voting Network for 6DoF Pose Estimationcited by 204
  8. 2019Cross-view Semantic Segmentation for Sensing Surroundingscited by 173
  9. 2020Objectron: A Large Scale Dataset of Object-Centric Videos in the Wild with Pose Annotationscited by 155
  10. 2021Episodic Transformer for Vision-and-Language Navigationcited by 155
  11. 2021Vision-Only Robot Navigation in a Neural Radiance Worldcited by 153
  12. 2021FFB6D: A Full Flow Bidirectional Fusion Network for 6D Pose Estimationcited by 144
  13. 2021Dex-NeRF: Using a Neural Radiance Field to Grasp Transparent Objectscited by 119
  14. 2021LENS: Localization enhanced by NeRF synthesiscited by 116
  15. 20213D Neural Scene Representations for Visuomotor Controlcited by 110
  16. 2019GS3D: An Efficient 3D Object Detection Framework for Autonomous Drivingcited by 106
  17. 2022GPV-Pose: Category-level Object Pose Estimation via Geometry-guided Point-wise Votingcited by 94
  18. 2021ShaRF: Shape-conditioned Radiance Fields from a Single Viewcited by 92
  19. 2022NeRF-Supervision: Learning Dense Object Descriptors from Neural Radiance Fieldscited by 86
  20. 2020STaR: Self-supervised Tracking and Reconstruction of Rigid Objects in Motion with Neural Renderingcited by 76
  21. 2022Loc-NeRF: Monte Carlo Localization using Neural Radiance Fieldscited by 69
  22. 20163D Object Proposals using Stereo Imagery for Accurate Object Class Detectioncited by 58
  23. 2022NeRF-Pose: A First-Reconstruct-Then-Regress Approach for Weakly-supervised 6D Object Pose Estimationcited by 57
  24. 2020MoreFusion: Multi-object Reasoning for 6D Pose Estimation from Volumetric Fusioncited by 46
  25. 2021PlaTe: Visually-Grounded Planning with Transformers in Procedural Taskscited by 45
  26. 2020SegVoxelNet: Exploring Semantic Context and Depth-aware Features for 3D Vehicle Detection from Point Cloudcited by 16
  27. 2022LocATe: End-to-end Localization of Actions in 3D with Transformerscited by 8

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 2; 2024: 2; 2025: 2.

7 papers.

  1. 2024Neural Fields in Robotics: A Surveycited by 21
  2. 2024Implicit-Zoo: A Large-Scale Dataset of Neural Implicit Functions for 2D Images and 3D Scenescited by 12
  3. 2023DGNR: Density-Guided Neural Point Rendering of Large Driving Scenescited by 9
  4. 2025NeuraLoc: Visual Localization in Neural Implicit Map with Dual Complementary Featurescited by 5
  5. 2023Aria-NeRF: Multimodal Egocentric View Synthesiscited by 4
  6. 2025NeRFs are Mirror Detectors: Using Structural Similarity for Multi-View Mirror Scene Reconstruction with 3D Surface Primitivescited by 2
  7. 2022Efficient 3D Reconstruction, Streaming and Visualization of Static and Dynamic Scene Parts for Multi-client Live-telepresence in Large-scale Environmentscited by 0