# Multi-Object Navigation with dynamically learned neural implicit representations

arXiv: https://arxiv.org/abs/2210.05129 · HTML: https://mlsanity.com/paper/2210.05129/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 51 of its 71 references that are in the map.
- First posted on arXiv 11 Oct 2022.

## Builds on (most cited first, 30 of 51)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [Learning Implicit Fields for Generative Shape Modeling](https://mlsanity.com/paper/1812.02822/) (2018)
- [HyperNetworks](https://mlsanity.com/paper/1609.09106/) (2016)
- [A Point Set Generation Network for 3D Object Reconstruction from a Single Image](https://mlsanity.com/paper/1612.00603/) (2016)
- [A Generalist Agent](https://mlsanity.com/paper/2205.06175/) (2022)
- [An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks](https://mlsanity.com/paper/1312.6211/) (2013)
- [NICE-SLAM: Neural Implicit Scalable Encoding for SLAM](https://mlsanity.com/paper/2112.12130/) (2021)
- [On Evaluation of Embodied Navigation Agents](https://mlsanity.com/paper/1807.06757/) (2018)
- [Neural Fields in Visual Computing and Beyond](https://mlsanity.com/paper/2111.11426/) (2021)
- [AtlasNet: A Papier-Mach'e Approach to Learning 3D Surface Generation](https://mlsanity.com/paper/1802.05384/) (2018)
- [iMAP: Implicit Mapping and Positioning in Real-Time](https://mlsanity.com/paper/2103.12352/) (2021)
- [Object Goal Navigation using Goal-Oriented Semantic Exploration](https://mlsanity.com/paper/2007.00643/) (2020)
- [In-Place Scene Labelling and Understanding with Implicit Scene Representation](https://mlsanity.com/paper/2103.15875/) (2021)
- [Reinforcement Learning with Unsupervised Auxiliary Tasks](https://mlsanity.com/paper/1611.05397/) (2016)
- [Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning](https://mlsanity.com/paper/1609.05143/) (2016)
- [Learning to Explore using Active Neural SLAM](https://mlsanity.com/paper/2004.05155/) (2020)
- [Learning to Navigate in Complex Environments](https://mlsanity.com/paper/1611.03673/) (2016)
- [Neural Topological SLAM for Visual Navigation](https://mlsanity.com/paper/2005.12256/) (2020)
- [Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigation](https://mlsanity.com/paper/2202.11742/) (2022)
- [Episodic Transformer for Vision-and-Language Navigation](https://mlsanity.com/paper/2105.06453/) (2021)
- [Vision-Only Robot Navigation in a Neural Radiance World](https://mlsanity.com/paper/2110.00168/) (2021)
- [Rapid Exploration for Open-World Navigation with Latent Goal Models](https://mlsanity.com/paper/2104.05859/) (2021)
- [Predicting trends in the quality of state-of-the-art neural networks without access to training or testing data](https://mlsanity.com/paper/2002.06716/) (2020)

## Built on by (most cited first, 21 of 21)

- [SayNav: Grounding Large Language Models for Dynamic Planning to Navigation in New Environments](https://mlsanity.com/paper/2309.04077/) (2023)
- [GaussNav: Gaussian Splatting for Visual Navigation](https://mlsanity.com/paper/2403.11625/) (2024)
- [Learning Generalizable Feature Fields for Mobile Manipulation](https://mlsanity.com/paper/2403.07563/) (2024)
- [End-to-End (Instance)-Image Goal Navigation through Correspondence as an Emergent Phenomenon](https://mlsanity.com/paper/2309.16634/) (2023)
- [NeRFs in Robotics: A Survey](https://mlsanity.com/paper/2405.01333/) (2024)
- [One Map to Find Them All: Real-time Open-Vocabulary Mapping for Zero-shot Multi-Object Navigation](https://mlsanity.com/paper/2409.11764/) (2024)
- [Neural Radiance Fields for the Real World: A Survey](https://mlsanity.com/paper/2501.13104/) (2025)
- [MOPA: Modular Object Navigation with PointGoal Agents](https://mlsanity.com/paper/2304.03696/) (2023)
- [AutoNeRF: Training Implicit Scene Representations with Autonomous Agents](https://mlsanity.com/paper/2304.11241/) (2023)
- [CAMON: Cooperative Agents for Multi-Object Navigation with LLM-based Conversations](https://mlsanity.com/paper/2407.00632/) (2024)
- [Learning with a Mole: Transferable latent spatial representations for navigation without reconstruction](https://mlsanity.com/paper/2306.03857/) (2023)
- [Learning to navigate efficiently and precisely in real environments](https://mlsanity.com/paper/2401.14349/) (2024)
- [CARFF: Conditional Auto-encoded Radiance Field for 3D Scene Forecasting](https://mlsanity.com/paper/2401.18075/) (2024)
- [Reasoning in visual navigation of end-to-end trained agents: a dynamical systems approach](https://mlsanity.com/paper/2503.08306/) (2025)
- [Hierarchical Scoring with 3D Gaussian Splatting for Instance Image-Goal Navigation](https://mlsanity.com/paper/2506.07338/) (2025)
- [RANa: Retrieval-Augmented Navigation](https://mlsanity.com/paper/2504.03524/) (2025)
- [What does really matter in image goal navigation?](https://mlsanity.com/paper/2507.01667/) (2025)
- [Point2Act: Efficient 3D Distillation of Multimodal LLMs for Zero-Shot Context-Aware Grasping](https://mlsanity.com/paper/2508.03099/) (2025)
- [M3DMap: Object-aware Multimodal 3D Mapping for Dynamic Environments](https://mlsanity.com/paper/2508.17044/) (2025)
- [DIV-Nav: Open-Vocabulary Spatial Relationships for Multi-Object Navigation](https://mlsanity.com/paper/2510.16518/) (2025)
- [CoReLIN: Constraint-based Reasoning for Zero-shot Lifelong Interactive Navigation](https://mlsanity.com/paper/2602.20055/) (2026)

## Abstract (arXiv)

Understanding and mapping a new environment are core abilities of any autonomously navigating agent. While classical robotics usually estimates maps in a stand-alone manner with SLAM variants, which maintain a topological or metric representation, end-to-end learning of navigation keeps some form of memory in a neural network. Networks are typically imbued with inductive biases, which can range from vectorial representations to birds-eye metric tensors or topological structures. In this work, we propose to structure neural networks with two neural implicit representations, which are learned dynamically during each episode and map the content of the scene: (i) the Semantic Finder predicts the position of a previously seen queried object; (ii) the Occupancy and Exploration Implicit Representation encapsulates information about explored area and obstacles, and is queried with a novel global read mechanism which directly maps from function space to a usable embedding space. Both representations are leveraged by an agent trained with Reinforcement Learning (RL) and learned online during each episode. We evaluate the agent on Multi-Object Navigation and show the high impact of using neural implicit representations as a memory source.