# Renderable Neural Radiance Map for Visual Navigation

arXiv: https://arxiv.org/abs/2303.00304 · HTML: https://mlsanity.com/paper/2303.00304/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 59 references that are in the map.
- First posted on arXiv 1 Mar 2023.

## Builds on (most cited first, 30 of 42)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [SuperGlue: Learning Feature Matching with Graph Neural Networks](https://mlsanity.com/paper/1911.11763/) (2019)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](https://mlsanity.com/paper/2103.17249/) (2021)
- [BARF: Bundle-Adjusting Neural Radiance Fields](https://mlsanity.com/paper/2104.06405/) (2021)
- [NICE-SLAM: Neural Implicit Scalable Encoding for SLAM](https://mlsanity.com/paper/2112.12130/) (2021)
- [On Evaluation of Embodied Navigation Agents](https://mlsanity.com/paper/1807.06757/) (2018)
- [Gibson Env: Real-World Perception for Embodied Agents](https://mlsanity.com/paper/1808.10654/) (2018)
- [iMAP: Implicit Mapping and Positioning in Real-Time](https://mlsanity.com/paper/2103.12352/) (2021)
- [Object Goal Navigation using Goal-Oriented Semantic Exploration](https://mlsanity.com/paper/2007.00643/) (2020)
- [DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Frames](https://mlsanity.com/paper/1911.00357/) (2019)
- [Learning to Explore using Active Neural SLAM](https://mlsanity.com/paper/2004.05155/) (2020)
- [A Survey of Embodied AI: From Simulators to Research Tasks](https://mlsanity.com/paper/2103.04918/) (2021)
- [CLIP-Forge: Towards Zero-Shot Text-to-Shape Generation](https://mlsanity.com/paper/2110.02624/) (2021)
- [CLIP2Video: Mastering Video-Text Retrieval via Image CLIP](https://mlsanity.com/paper/2106.11097/) (2021)
- [Cognitive Mapping and Planning for Visual Navigation](https://mlsanity.com/paper/1702.03920/) (2017)
- [CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoders](https://mlsanity.com/paper/2106.14843/) (2021)
- [Vision-Only Robot Navigation in a Neural Radiance World](https://mlsanity.com/paper/2110.00168/) (2021)
- [Unconstrained Scene Generation with Locally Conditioned Radiance Fields](https://mlsanity.com/paper/2104.00670/) (2021)
- [Dex-NeRF: Using a Neural Radiance Field to Grasp Transparent Objects](https://mlsanity.com/paper/2110.14217/) (2021)
- [Learning Exploration Policies for Navigation](https://mlsanity.com/paper/1903.01959/) (2019)
- [3D Neural Scene Representations for Visuomotor Control](https://mlsanity.com/paper/2107.04004/) (2021)
- [Occupancy Anticipation for Efficient Exploration and Navigation](https://mlsanity.com/paper/2008.09285/) (2020)
- [Image Generators with Conditionally-Independent Pixel Synthesis](https://mlsanity.com/paper/2011.13775/) (2020)
- [MultiON: Benchmarking Semantic Map Memory using Multi-Object Navigation](https://mlsanity.com/paper/2012.03912/) (2020)
- [Neural Map: Structured Memory for Deep Reinforcement Learning](https://mlsanity.com/paper/1702.08360/) (2017)
- [Offline Visual Representation Learning for Embodied Navigation](https://mlsanity.com/paper/2204.13226/) (2022)
- [Loc-NeRF: Monte Carlo Localization using Neural Radiance Fields](https://mlsanity.com/paper/2209.09050/) (2022)

## Built on by (most cited first, 30 of 51)

- [NVS-Solver: Video Diffusion Model as Zero-Shot Novel View Synthesizer](https://mlsanity.com/paper/2405.15364/) (2024)
- [Lookahead Exploration with Neural Radiance Representation for Continuous Vision-Language Navigation](https://mlsanity.com/paper/2404.01943/) (2024)
- [Volumetric Environment Representation for Vision-Language Navigation](https://mlsanity.com/paper/2403.14158/) (2024)
- [VoroNav: Voronoi-based Zero-shot Object Navigation with Large Language Model](https://mlsanity.com/paper/2401.02695/) (2024)
- [UniGoal: Towards Universal Zero-shot Goal-oriented Navigation](https://mlsanity.com/paper/2503.10630/) (2025)
- [See and Think: Embodied Agent in Virtual Environment](https://mlsanity.com/paper/2311.15209/) (2023)
- [GaussNav: Gaussian Splatting for Visual Navigation](https://mlsanity.com/paper/2403.11625/) (2024)
- [CROSSFIRE: Camera Relocalization On Self-Supervised Features from an Implicit Representation](https://mlsanity.com/paper/2303.04869/) (2023)
- [What Is The Best 3D Scene Representation for Robotics? From Geometric to Foundation Models](https://mlsanity.com/paper/2512.03422/) (2025)
- [Hierarchical Auto-Organizing System for Open-Ended Multi-Agent Navigation](https://mlsanity.com/paper/2403.08282/) (2024)
- [General Place Recognition Survey: Towards Real-World Autonomy](https://mlsanity.com/paper/2405.04812/) (2024)
- [g3D-LF: Generalizable 3D-Language Feature Fields for Embodied Tasks](https://mlsanity.com/paper/2411.17030/) (2024)
- [Navigation Instruction Generation with BEV Perception and Large Language Models](https://mlsanity.com/paper/2407.15087/) (2024)
- [Neural Fields in Robotics: A Survey](https://mlsanity.com/paper/2410.20220/) (2024)
- [Do We Really Need a Complex Agent System? Distill Embodied Agent into a Single Model](https://mlsanity.com/paper/2404.04619/) (2024)
- [OVER-NAV: Elevating Iterative Vision-and-Language Navigation with Open-Vocabulary Detection and StructurEd Representation](https://mlsanity.com/paper/2403.17334/) (2024)
- [LangSurf: Language-Embedded Surface Gaussians for 3D Scene Understanding](https://mlsanity.com/paper/2412.17635/) (2024)
- [NeRFs in Robotics: A Survey](https://mlsanity.com/paper/2405.01333/) (2024)
- [VDG: Vision-Only Dynamic Gaussian for Driving Simulation](https://mlsanity.com/paper/2406.18198/) (2024)
- [Instance-aware Exploration-Verification-Exploitation for Instance ImageGoal Navigation](https://mlsanity.com/paper/2402.17587/) (2024)
- [MSGNav: Unleashing the Power of Multi-modal 3D Scene Graph for Zero-Shot Embodied Navigation](https://mlsanity.com/paper/2511.10376/) (2025)
- [Fed3DGS: Scalable 3D Gaussian Splatting with Federated Learning](https://mlsanity.com/paper/2403.11460/) (2024)
- [Neural Radiance Fields for the Real World: A Survey](https://mlsanity.com/paper/2501.13104/) (2025)
- [REGNav: Room Expert Guided Image-Goal Navigation](https://mlsanity.com/paper/2502.10785/) (2025)
- [IGL-Nav: Incremental 3D Gaussian Localization for Image-goal Navigation](https://mlsanity.com/paper/2508.00823/) (2025)
- [Navigating Beyond Instructions: Vision-and-Language Navigation in Obstructed Environments](https://mlsanity.com/paper/2407.21452/) (2024)
- [Federated Learning for Large-Scale Scene Modeling with Neural Radiance Fields](https://mlsanity.com/paper/2309.06030/) (2023)
- [CAMON: Cooperative Agents for Multi-Object Navigation with LLM-based Conversations](https://mlsanity.com/paper/2407.00632/) (2024)
- [CodecNeRF: Toward Fast Encoding and Decoding, Compact, and High-quality Novel-view Synthesis](https://mlsanity.com/paper/2404.04913/) (2024)
- [Language-enhanced RNR-Map: Querying Renderable Neural Radiance Field maps with natural language](https://mlsanity.com/paper/2308.08854/) (2023)

## Abstract (arXiv)

We propose a novel type of map for visual navigation, a renderable neural radiance map (RNR-Map), which is designed to contain the overall visual information of a 3D environment. The RNR-Map has a grid form and consists of latent codes at each pixel. These latent codes are embedded from image observations, and can be converted to the neural radiance field which enables image rendering given a camera pose. The recorded latent codes implicitly contain visual information about the environment, which makes the RNR-Map visually descriptive. This visual information in RNR-Map can be a useful guideline for visual localization and navigation. We develop localization and navigation frameworks that can effectively utilize the RNR-Map. We evaluate the proposed frameworks on camera tracking, visual localization, and image-goal navigation. Experimental results show that the RNR-Map-based localization framework can find the target location based on a single query image with fast speed and competitive accuracy compared to other baselines. Also, this localization framework is robust to environmental changes, and even finds the most visually similar places when a query image from a different environment is given. The proposed navigation framework outperforms the existing image-goal navigation methods in difficult scenarios, under odometry and actuation noises. The navigation framework shows 65.7% success rate in curved scenarios of the NRNS dataset, which is an improvement of 18.6% over the current state-of-the-art. Project page: https://rllab-snu.github.io/projects/RNR-Map/