# Self-Improving Semantic Perception for Indoor Localisation

arXiv: https://arxiv.org/abs/2105.01595 · HTML: https://mlsanity.com/paper/2105.01595/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 79 references that are in the map.
- First posted on arXiv 4 May 2021.

## Builds on (most cited first, 30 of 47)

- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [Continual Learning with Deep Generative Replay](https://mlsanity.com/paper/1705.08690/) (2017)
- [Colorful Image Colorization](https://mlsanity.com/paper/1603.08511/) (2016)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [Progress & Compress: A scalable framework for continual learning](https://mlsanity.com/paper/1805.06370/) (2018)
- [A Survey of Autonomous Driving: Common Practices and Emerging Technologies](https://mlsanity.com/paper/1906.05113/) (2019)
- [Incremental Learning of Object Detectors without Catastrophic Forgetting](https://mlsanity.com/paper/1708.06977/) (2017)
- [Modeling the Background for Incremental Learning in Semantic Segmentation](https://mlsanity.com/paper/2002.00718/) (2020)
- [Kimera: an Open-Source Library for Real-Time Metric-Semantic Localization and Mapping](https://mlsanity.com/paper/1910.02490/) (2019)
- [PLOP: Learning without Forgetting for Continual Semantic Segmentation](https://mlsanity.com/paper/2011.11390/) (2020)
- [FickleNet: Weakly and Semi-supervised Semantic Image Segmentation using Stochastic Inference](https://mlsanity.com/paper/1902.10421/) (2019)
- [Learning to Walk via Deep Reinforcement Learning](https://mlsanity.com/paper/1812.11103/) (2018)
- [Incremental Learning Techniques for Semantic Segmentation](https://mlsanity.com/paper/1907.13372/) (2019)
- [Self-supervised Learning with Geometric Constraints in Monocular Video: Connecting Flow, Depth, and Camera](https://mlsanity.com/paper/1907.05820/) (2019)
- [Memory Efficient Experience Replay for Streaming Learning](https://mlsanity.com/paper/1809.05922/) (2018)
- [Fast-SCNN: Fast Semantic Segmentation Network](https://mlsanity.com/paper/1902.04502/) (2019)
- [The Fishyscapes Benchmark: Measuring Blind Spots in Semantic Segmentation](https://mlsanity.com/paper/1904.03215/) (2019)
- [A Deeper Look at Experience Replay](https://mlsanity.com/paper/1712.01275/) (2017)
- [Learning to See by Moving](https://mlsanity.com/paper/1505.01596/) (2015)
- [Fusion++: Volumetric Object-Level SLAM](https://mlsanity.com/paper/1808.08378/) (2018)
- [Continual Learning for Robotics: Definition, Framework, Learning Strategies, Opportunities and Challenges](https://mlsanity.com/paper/1907.00182/) (2019)

## Built on by (most cited first, 5 of 5)

- [BioSLAM: A Bio-inspired Lifelong Memory System for General Place Recognition](https://mlsanity.com/paper/2208.14543/) (2022)
- [Mitigating Object Dependencies: Improving Point Cloud Self-Supervised Learning through Object Exchange](https://mlsanity.com/paper/2404.07504/) (2024)
- [SCIM: Simultaneous Clustering, Inference, and Mapping for Open-World Semantic Scene Understanding](https://mlsanity.com/paper/2206.10670/) (2022)
- [LTS-NET: End-to-end Unsupervised Learning of Long-Term 3D Stable objects](https://mlsanity.com/paper/2301.03426/) (2023)
- [BIM-Constrained Optimization for Accurate Localization and Deviation Correction in Construction Monitoring](https://mlsanity.com/paper/2504.17693/) (2025)

## Abstract (arXiv)

We propose a novel robotic system that can improve its perception during deployment. Contrary to the established approach of learning semantics from large datasets and deploying fixed models, we propose a framework in which semantic models are continuously updated on the robot to adapt to the deployment environments. By combining continual learning with self-supervision, our robotic system learns online during deployment without external supervision. We conduct real-world experiments with robots localising in 3D floorplans. Our experiments show how the robot's semantic perception improves during deployment and how this translates into improved localisation, even across drastically different environments. We further study the risk of catastrophic forgetting that such a continuous learning setting poses. We find memory replay an effective measure to reduce forgetting and show how the robotic system can improve even when switching between different environments. On average, our system improves by 60% in segmentation and 10% in localisation accuracy compared to deployment of a fixed model, and it maintains this improvement while adapting to further environments.