# SCARF: Scalable Continual Learning Framework for Memory-efficient Multiple Neural Radiance Fields arXiv: https://arxiv.org/abs/2409.04482 ยท HTML: https://mlsanity.com/paper/2409.04482/ ## Key facts - Cited by 8 papers in the map (data through 14 Oct 2026). - Builds on 48 of its 61 references that are in the map. - First posted on arXiv 6 Sep 2024. ## Builds on (most cited first, 30 of 48) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015) - [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014) - [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021) - [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017) - [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016) - [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022) - [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016) - [Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields](https://mlsanity.com/paper/2111.12077/) (2021) - [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016) - [NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections](https://mlsanity.com/paper/2008.02268/) (2020) - [HyperNetworks](https://mlsanity.com/paper/1609.09106/) (2016) - [Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction](https://mlsanity.com/paper/2111.11215/) (2021) - [Neural Sparse Voxel Fields](https://mlsanity.com/paper/2007.11571/) (2020) - [PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning](https://mlsanity.com/paper/1711.05769/) (2017) - [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023) - [MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo](https://mlsanity.com/paper/2103.15595/) (2021) - [IBRNet: Learning Multi-View Image-Based Rendering](https://mlsanity.com/paper/2102.13090/) (2021) - [Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelines](https://mlsanity.com/paper/1905.00889/) (2019) - [iMAP: Implicit Mapping and Positioning in Real-Time](https://mlsanity.com/paper/2103.12352/) (2021) - [MobileNeRF: Exploiting the Polygon Rasterization Pipeline for Efficient Neural Field Rendering on Mobile Architectures](https://mlsanity.com/paper/2208.00277/) (2022) - [Learned Initializations for Optimizing Coordinate-Based Neural Representations](https://mlsanity.com/paper/2012.02189/) (2020) - [BungeeNeRF: Progressive Neural Radiance Field for Extreme Multi-scale Scene Rendering](https://mlsanity.com/paper/2112.05504/) (2021) - [Learning Object-Compositional Neural Radiance Field for Editable Scene Rendering](https://mlsanity.com/paper/2109.01847/) (2021) - [Accelerating Sparse Deep Neural Networks](https://mlsanity.com/paper/2104.08378/) (2021) - [MetaSDF: Meta-learning Signed Distance Functions](https://mlsanity.com/paper/2006.09662/) (2020) - [AutoInt: Automatic Integration for Fast Neural Volume Rendering](https://mlsanity.com/paper/2012.01714/) (2020) - [Adversarial Continual Learning](https://mlsanity.com/paper/2003.09553/) (2020) - [Generative Feature Replay For Class-Incremental Learning](https://mlsanity.com/paper/2004.09199/) (2020) - [SINE: Semantic-driven Image-based NeRF Editing with Prior-guided Editing Field](https://mlsanity.com/paper/2303.13277/) (2023) ## Built on by (most cited first, 8 of 8) - [iVR-GS: Inverse Volume Rendering for Explorable Visualization via Editable 3D Gaussian Splatting](https://mlsanity.com/paper/2504.17954/) (2025) - [CL-Splats: Continual Learning of Gaussian Splatting with Local Optimization](https://mlsanity.com/paper/2506.21117/) (2025) - [WildSeg3D: Segment Any 3D Objects in the Wild from 2D Images](https://mlsanity.com/paper/2503.08407/) (2025) - [Implicit neural representations for accurate estimation of the standard model of white matter](https://mlsanity.com/paper/2506.15762/) (2025) - [Tile and Slide : A New Framework for Scaling NeRF from Local to Global 3D Earth Observation](https://mlsanity.com/paper/2507.01631/) (2025) - [Learning Two-factor Representation for Magnetic Resonance Image Super-resolution](https://mlsanity.com/paper/2409.09731/) (2024) - [Taking Language Embedded 3D Gaussian Splatting into the Wild](https://mlsanity.com/paper/2507.19830/) (2025) - [Incremental Multi-Scene Modeling via Continual Neural Graphics Primitives](https://mlsanity.com/paper/2411.19903/) (2024) ## Abstract (arXiv) This paper introduces a novel continual learning framework for synthesising novel views of multiple scenes, learning multiple 3D scenes incrementally, and updating the network parameters only with the training data of the upcoming new scene. We build on Neural Radiance Fields (NeRF), which uses multi-layer perceptron to model the density and radiance field of a scene as the implicit function. While NeRF and its extensions have shown a powerful capability of rendering photo-realistic novel views in a single 3D scene, managing these growing 3D NeRF assets efficiently is a new scientific problem. Very few works focus on the efficient representation or continuous learning capability of multiple scenes, which is crucial for the practical applications of NeRF. To achieve these goals, our key idea is to represent multiple scenes as the linear combination of a cross-scene weight matrix and a set of scene-specific weight matrices generated from a global parameter generator. Furthermore, we propose an uncertain surface knowledge distillation strategy to transfer the radiance field knowledge of previous scenes to the new model. Representing multiple 3D scenes with such weight matrices significantly reduces memory requirements. At the same time, the uncertain surface distillation strategy greatly overcomes the catastrophic forgetting problem and maintains the photo-realistic rendering quality of previous scenes. Experiments show that the proposed approach achieves state-of-the-art rendering quality of continual learning NeRF on NeRF-Synthetic, LLFF, and TanksAndTemples datasets while preserving extra low storage cost.