# Continual Neural Mapping: Learning An Implicit Scene Representation from Sequential Observations

arXiv: https://arxiv.org/abs/2108.05851 · HTML: https://mlsanity.com/paper/2108.05851/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 73 references that are in the map.
- First posted on arXiv 12 Aug 2021.

## Builds on (most cited first, 30 of 49)

- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017)
- [Experience Replay for Continual Learning](https://mlsanity.com/paper/1811.11682/) (2018)
- [Learning Implicit Fields for Generative Shape Modeling](https://mlsanity.com/paper/1812.02822/) (2018)
- [Efficient Lifelong Learning with A-GEM](https://mlsanity.com/paper/1812.00420/) (2018)
- [PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning](https://mlsanity.com/paper/1711.05769/) (2017)
- [Convolutional Occupancy Networks](https://mlsanity.com/paper/2003.04618/) (2020)
- [Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence](https://mlsanity.com/paper/1801.10112/) (2018)
- [Open3D: A Modern Library for 3D Data Processing](https://mlsanity.com/paper/1801.09847/) (2018)
- [Implicit Geometric Regularization for Learning Shapes](https://mlsanity.com/paper/2002.10099/) (2020)
- [Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision](https://mlsanity.com/paper/1912.07372/) (2019)
- [Lifelong Learning with Dynamically Expandable Networks](https://mlsanity.com/paper/1708.01547/) (2017)
- [GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis](https://mlsanity.com/paper/2007.02442/) (2020)
- [Overcoming catastrophic forgetting with hard attention to the task](https://mlsanity.com/paper/1801.01423/) (2018)
- [Three scenarios for continual learning](https://mlsanity.com/paper/1904.07734/) (2019)
- [Local Implicit Grid Representations for 3D Scenes](https://mlsanity.com/paper/2003.08981/) (2020)
- [Expert Gate: Lifelong Learning with a Network of Experts](https://mlsanity.com/paper/1611.06194/) (2016)
- [Deep Local Shapes: Learning Local SDF Priors for Detailed 3D Reconstruction](https://mlsanity.com/paper/2003.10983/) (2020)
- [Learning Shape Templates with Structured Implicit Functions](https://mlsanity.com/paper/1904.06447/) (2019)
- [BSP-Net: Generating Compact Meshes via Binary Space Partitioning](https://mlsanity.com/paper/1911.06971/) (2019)
- [Atlas: End-to-End 3D Scene Reconstruction from Posed Images](https://mlsanity.com/paper/2003.10432/) (2020)

## Built on by (most cited first, 30 of 30)

- [NICE-SLAM: Neural Implicit Scalable Encoding for SLAM](https://mlsanity.com/paper/2112.12130/) (2021)
- [Neural Fields in Visual Computing and Beyond](https://mlsanity.com/paper/2111.11426/) (2021)
- [Point-SLAM: Dense Neural Point Cloud-based SLAM](https://mlsanity.com/paper/2304.04278/) (2023)
- [ESLAM: Efficient Dense SLAM System Based on Hybrid Representation of Signed Distance Fields](https://mlsanity.com/paper/2211.11704/) (2022)
- [Gaussian-SLAM: Photo-realistic Dense SLAM with Gaussian Splatting](https://mlsanity.com/paper/2312.10070/) (2023)
- [iSDF: Real-Time Neural Signed Distance Fields for Robot Perception](https://mlsanity.com/paper/2204.02296/) (2022)
- [POCO: Point Convolution for Surface Reconstruction](https://mlsanity.com/paper/2201.01831/) (2022)
- [Active Neural Mapping](https://mlsanity.com/paper/2308.16246/) (2023)
- [Active Implicit Object Reconstruction using Uncertainty-guided Next-Best-View Optimization](https://mlsanity.com/paper/2303.16739/) (2023)
- [NARUTO: Neural Active Reconstruction from Uncertain Target Observations](https://mlsanity.com/paper/2402.18771/) (2024)
- [Continual Learning in Medical Image Analysis: A Comprehensive Review of Recent Advancements and Future Prospects](https://mlsanity.com/paper/2312.17004/) (2023)
- [UncLe-SLAM: Uncertainty Learning for Dense Neural SLAM](https://mlsanity.com/paper/2306.11048/) (2023)
- [IR-MCL: Implicit Representation-Based Online Global Localization](https://mlsanity.com/paper/2210.03113/) (2022)
- [NextBestPath: Efficient 3D Mapping of Unseen Environments](https://mlsanity.com/paper/2502.05378/) (2025)
- [Active Neural Mapping at Scale](https://mlsanity.com/paper/2409.20276/) (2024)
- [City-scale Incremental Neural Mapping with Three-layer Sampling and Panoptic Representation](https://mlsanity.com/paper/2209.14072/) (2022)
- [Learning Online Multi-Sensor Depth Fusion](https://mlsanity.com/paper/2204.03353/) (2022)
- [Depth Reconstruction with Neural Signed Distance Fields in Structured Light Systems](https://mlsanity.com/paper/2405.12006/) (2024)
- [Learn to Memorize and to Forget: A Continual Learning Perspective of Dynamic SLAM](https://mlsanity.com/paper/2407.13338/) (2024)
- [UNIKD: UNcertainty-filtered Incremental Knowledge Distillation for Neural Implicit Representation](https://mlsanity.com/paper/2212.10950/) (2022)
- [HIO-SDF: Hierarchical Incremental Online Signed Distance Fields](https://mlsanity.com/paper/2310.09463/) (2023)
- [Continually Learn to Map Visual Concepts to Large Language Models in Resource-constrained Environments](https://mlsanity.com/paper/2407.08279/) (2024)
- [Meta-Continual Learning of Neural Fields](https://mlsanity.com/paper/2504.05806/) (2025)
- [A Fusion of Variational Distribution Priors and Saliency Map Replay for Continual 3D Reconstruction](https://mlsanity.com/paper/2308.08812/) (2023)
- [AiSDF: Structure-aware Neural Signed Distance Fields in Indoor Scenes](https://mlsanity.com/paper/2403.01861/) (2024)
- [LGSDF: Continual Global Learning of Signed Distance Fields Aided by Local Updating](https://mlsanity.com/paper/2404.05187/) (2024)
- [Decomposition of Neural Discrete Representations for Large-Scale 3D Mapping](https://mlsanity.com/paper/2407.15554/) (2024)
- [CCNDF: Curvature Constrained Neural Distance Fields from 3D LiDAR Sequences](https://mlsanity.com/paper/2412.15909/) (2024)
- [Matching-Free Depth Recovery from Structured Light](https://mlsanity.com/paper/2501.07113/) (2025)
- [Continual Learning as a Multiphase Moving-Boundary Problem](https://mlsanity.com/paper/2606.01863/) (2026)

## Abstract (arXiv)

Recent advances have enabled a single neural network to serve as an implicit scene representation, establishing the mapping function between spatial coordinates and scene properties. In this paper, we make a further step towards continual learning of the implicit scene representation directly from sequential observations, namely Continual Neural Mapping. The proposed problem setting bridges the gap between batch-trained implicit neural representations and commonly used streaming data in robotics and vision communities. We introduce an experience replay approach to tackle an exemplary task of continual neural mapping: approximating a continuous signed distance function (SDF) from sequential depth images as a scene geometry representation. We show for the first time that a single network can represent scene geometry over time continually without catastrophic forgetting, while achieving promising trade-offs between accuracy and efficiency.