# SCIM: Simultaneous Clustering, Inference, and Mapping for Open-World Semantic Scene Understanding

arXiv: https://arxiv.org/abs/2206.10670 · HTML: https://mlsanity.com/paper/2206.10670/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 32 references that are in the map.
- First posted on arXiv 21 Jun 2022.

## Builds on (most cited first, 21 of 21)

- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [Towards Open World Object Detection](https://mlsanity.com/paper/2103.02603/) (2021)
- [Unsupervised Semantic Segmentation by Distilling Feature Correspondences](https://mlsanity.com/paper/2203.08414/) (2022)
- [PLOP: Learning without Forgetting for Continual Semantic Segmentation](https://mlsanity.com/paper/2011.11390/) (2020)
- [The Perfect Match: 3D Point Cloud Matching with Smoothed Densities](https://mlsanity.com/paper/1811.06879/) (2018)
- [Incremental Learning Techniques for Semantic Segmentation](https://mlsanity.com/paper/1907.13372/) (2019)
- [SemanticFusion: Dense 3D Semantic Mapping with Convolutional Neural Networks](https://mlsanity.com/paper/1609.05130/) (2016)
- [The Fishyscapes Benchmark: Measuring Blind Spots in Semantic Segmentation](https://mlsanity.com/paper/1904.03215/) (2019)
- [Volumetric Instance-Aware Semantic Mapping and 3D Object Discovery](https://mlsanity.com/paper/1903.00268/) (2019)
- [Standardized Max Logits: A Simple yet Effective Approach for Identifying Unexpected Road Obstacles in Urban-Scene Segmentation](https://mlsanity.com/paper/2107.11264/) (2021)
- [Semantics for Robotic Mapping, Perception and Interaction: A Survey](https://mlsanity.com/paper/2101.00443/) (2021)
- [Deep Metric Learning for Open World Semantic Segmentation](https://mlsanity.com/paper/2108.04562/) (2021)
- [SEAL: Self-supervised Embodied Active Learning using Exploration and 3D Consistency](https://mlsanity.com/paper/2112.01001/) (2021)
- [Panoptic Multi-TSDFs: a Flexible Representation for Online Multi-resolution Volumetric Mapping and Long-term Dynamic Scene Consistency](https://mlsanity.com/paper/2109.10165/) (2021)
- [Self-Training for Class-Incremental Semantic Segmentation](https://mlsanity.com/paper/2012.03362/) (2020)
- [Deep Continuous Clustering](https://mlsanity.com/paper/1803.01449/) (2018)
- [Incremental Class Discovery for Semantic Segmentation with RGBD Sensing](https://mlsanity.com/paper/1907.10008/) (2019)
- [Towards Unsupervised Open World Semantic Segmentation](https://mlsanity.com/paper/2201.01073/) (2022)
- [Embodied Active Domain Adaptation for Semantic Segmentation via Informative Path Planning](https://mlsanity.com/paper/2203.00549/) (2022)
- [Self-Improving Semantic Perception for Indoor Localisation](https://mlsanity.com/paper/2105.01595/) (2021)

## Built on by (most cited first, 4 of 4)

- [LOSS-SLAM: Lightweight Open-Set Semantic Simultaneous Localization and Mapping](https://mlsanity.com/paper/2404.04377/) (2024)
- [DynaVINS++: Robust Visual-Inertial State Estimator in Dynamic Environments by Adaptive Truncated Least Squares and Stable State Recovery](https://mlsanity.com/paper/2410.15373/) (2024)
- [NeurNCD: Novel Class Discovery via Implicit Neural Representation](https://mlsanity.com/paper/2506.06412/) (2025)
- [Trinity: Unifying Class-Agnostic Terrain and Semantic Segmentation for Unstructured Outdoor Environments by Leveraging Synthetic Data](https://mlsanity.com/paper/2605.27644/) (2026)

## Abstract (arXiv)

In order to operate in human environments, a robot's semantic perception has to overcome open-world challenges such as novel objects and domain gaps. Autonomous deployment to such environments therefore requires robots to update their knowledge and learn without supervision. We investigate how a robot can autonomously discover novel semantic classes and improve accuracy on known classes when exploring an unknown environment. To this end, we develop a general framework for mapping and clustering that we then use to generate a self-supervised learning signal to update a semantic segmentation model. In particular, we show how clustering parameters can be optimized during deployment and that fusion of multiple observation modalities improves novel object discovery compared to prior work. Models, data, and implementations can be found at https://github.com/hermannsblum/scim