# AutoNeRF: Training Implicit Scene Representations with Autonomous Agents arXiv: https://arxiv.org/abs/2304.11241 ยท HTML: https://mlsanity.com/paper/2304.11241/ ## Key facts - Cited by 10 papers in the map (data through 14 Oct 2026). - Builds on 53 of its 64 references that are in the map. - First posted on arXiv 21 Apr 2023. ## Builds on (most cited first, 30 of 53) - [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014) - [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014) - [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017) - [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018) - [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017) - [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022) - [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019) - [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018) - [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020) - [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019) - [Learning Implicit Fields for Generative Shape Modeling](https://mlsanity.com/paper/1812.02822/) (2018) - [Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments](https://mlsanity.com/paper/1711.07280/) (2017) - [A Point Set Generation Network for 3D Object Reconstruction from a Single Image](https://mlsanity.com/paper/1612.00603/) (2016) - [Block-NeRF: Scalable Large Scene Neural View Synthesis](https://mlsanity.com/paper/2202.05263/) (2022) - [Nerfstudio: A Modular Framework for Neural Radiance Field Development](https://mlsanity.com/paper/2302.04264/) (2023) - [NICE-SLAM: Neural Implicit Scalable Encoding for SLAM](https://mlsanity.com/paper/2112.12130/) (2021) - [On Evaluation of Embodied Navigation Agents](https://mlsanity.com/paper/1807.06757/) (2018) - [Neural Fields in Visual Computing and Beyond](https://mlsanity.com/paper/2111.11426/) (2021) - [AtlasNet: A Papier-Mach'e Approach to Learning 3D Surface Generation](https://mlsanity.com/paper/1802.05384/) (2018) - [FastNeRF: High-Fidelity Neural Rendering at 200FPS](https://mlsanity.com/paper/2103.10380/) (2021) - [Habitat 2.0: Training Home Assistants to Rearrange their Habitat](https://mlsanity.com/paper/2106.14405/) (2021) - [Gibson Env: Real-World Perception for Embodied Agents](https://mlsanity.com/paper/1808.10654/) (2018) - [iMAP: Implicit Mapping and Positioning in Real-Time](https://mlsanity.com/paper/2103.12352/) (2021) - [Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environments](https://mlsanity.com/paper/2004.02857/) (2020) - [Object Goal Navigation using Goal-Oriented Semantic Exploration](https://mlsanity.com/paper/2007.00643/) (2020) - [In-Place Scene Labelling and Understanding with Implicit Scene Representation](https://mlsanity.com/paper/2103.15875/) (2021) - [INeRF: Inverting Neural Radiance Fields for Pose Estimation](https://mlsanity.com/paper/2012.05877/) (2020) - [Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning](https://mlsanity.com/paper/1609.05143/) (2016) - [Learning to Explore using Active Neural SLAM](https://mlsanity.com/paper/2004.05155/) (2020) - [Cognitive Mapping and Planning for Visual Navigation](https://mlsanity.com/paper/1702.03920/) (2017) ## Built on by (most cited first, 10 of 10) - [Neural Fields in Robotics: A Survey](https://mlsanity.com/paper/2410.20220/) (2024) - [Selective Visual Representations Improve Convergence and Generalization for Embodied AI](https://mlsanity.com/paper/2311.04193/) (2023) - [NeRFs in Robotics: A Survey](https://mlsanity.com/paper/2405.01333/) (2024) - [EmbodiedSplat: Personalized Real-to-Sim-to-Real Navigation with Gaussian Splats from a Mobile Device](https://mlsanity.com/paper/2509.17430/) (2025) - [ExploreGS: Explorable 3D Scene Reconstruction with Virtual Camera Samplings and Diffusion Priors](https://mlsanity.com/paper/2508.06014/) (2025) - [Language-enhanced RNR-Map: Querying Renderable Neural Radiance Field maps with natural language](https://mlsanity.com/paper/2308.08854/) (2023) - [Understanding while Exploring: Semantics-driven Active Mapping](https://mlsanity.com/paper/2506.00225/) (2025) - [Fast and Generalizable NeRF Architecture Selection for Satellite Scene Reconstruction](https://mlsanity.com/paper/2603.18306/) (2026) - [Embodied Active Learning under Limited Annotation and Navigation Budget for Object Detection](https://mlsanity.com/paper/2607.15974/) (2026) - [FillGS: Filling Observation Gaps in 4D Gaussian Splatting via Viewpoint-Time Selection and Generative Refinement](https://mlsanity.com/paper/2607.29284/) (2026) ## Abstract (arXiv) Implicit representations such as Neural Radiance Fields (NeRF) have been shown to be very effective at novel view synthesis. However, these models typically require manual and careful human data collection for training. In this paper, we present AutoNeRF, a method to collect data required to train NeRFs using autonomous embodied agents. Our method allows an agent to explore an unseen environment efficiently and use the experience to build an implicit map representation autonomously. We compare the impact of different exploration strategies including handcrafted frontier-based exploration, end-to-end and modular approaches composed of trained high-level planners and classical low-level path followers. We train these models with different reward functions tailored to this problem and evaluate the quality of the learned representations on four different downstream tasks: classical viewpoint rendering, map reconstruction, planning, and pose refinement. Empirical results show that NeRFs can be trained on actively collected data using just a single episode of experience in an unseen environment, and can be used for several downstream robotic tasks, and that modular trained exploration models outperform other classical and end-to-end baselines. Finally, we show that AutoNeRF can reconstruct large-scale scenes, and is thus a useful tool to perform scene-specific adaptation as the produced 3D environment models can be loaded into a simulator to fine-tune a policy of interest.