# Open-World Semantic Segmentation Including Class Similarity

arXiv: https://arxiv.org/abs/2403.07532 · HTML: https://mlsanity.com/paper/2403.07532/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 65 of its 80 references that are in the map.
- First posted on arXiv 12 Mar 2024.

## Builds on (most cited first, 30 of 65)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks](https://mlsanity.com/paper/1706.02690/) (2017)
- [Energy-based Out-of-distribution Detection](https://mlsanity.com/paper/2010.03759/) (2020)
- [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018)
- [Panoptic Segmentation](https://mlsanity.com/paper/1801.00868/) (2018)
- [Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images](https://mlsanity.com/paper/1412.1897/) (2014)
- [Towards Total Recall in Industrial Anomaly Detection](https://mlsanity.com/paper/2106.08265/) (2021)
- [Panoptic Feature Pyramid Networks](https://mlsanity.com/paper/1901.02446/) (2019)
- [RegionCLIP: Region-based Language-Image Pretraining](https://mlsanity.com/paper/2112.09106/) (2021)
- [Towards Open Set Deep Networks](https://mlsanity.com/paper/1511.06233/) (2015)
- [DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting](https://mlsanity.com/paper/2112.01518/) (2021)
- [DRAEM -- A discriminatively trained reconstruction embedding for surface anomaly detection](https://mlsanity.com/paper/2108.07610/) (2021)
- [InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions](https://mlsanity.com/paper/2211.05778/) (2022)
- [Scaling Out-of-Distribution Detection for Real-World Settings](https://mlsanity.com/paper/1911.11132/) (2019)
- [Future Frame Prediction for Anomaly Detection -- A New Baseline](https://mlsanity.com/paper/1712.09867/) (2017)
- [Uninformed Students: Student-Teacher Anomaly Detection with Discriminative Latent Embeddings](https://mlsanity.com/paper/1911.02357/) (2019)
- [Open-Set Recognition: a Good Closed-Set Classifier is All You Need?](https://mlsanity.com/paper/2110.06207/) (2021)

## Built on by (most cited first, 24 of 24)

- [Prior2Former -- Evidential Modeling of Mask Transformers for Assumption-Free Open-World Panoptic Segmentation](https://mlsanity.com/paper/2504.04841/) (2025)
- [PraNet-V2: Dual-Supervised Reverse Attention for Medical Image Segmentation](https://mlsanity.com/paper/2504.10986/) (2025)
- [LOC: A General Language-Guided Framework for Open-Set 3D Occupancy Prediction](https://mlsanity.com/paper/2510.22141/) (2025)
- [Multi-Scale Foreground-Background Confidence for Out-of-Distribution Segmentation](https://mlsanity.com/paper/2412.16990/) (2024)
- [Open-World Panoptic Segmentation](https://mlsanity.com/paper/2412.12740/) (2024)
- [A Novel Decomposed Feature-Oriented Framework for Open-Set Semantic Segmentation on LiDAR Data](https://mlsanity.com/paper/2503.11097/) (2025)
- [Open-set Anomaly Segmentation in Complex Scenarios](https://mlsanity.com/paper/2504.19706/) (2025)
- [Out-of-Distribution Segmentation via Wasserstein-Based Evidential Uncertainty](https://mlsanity.com/paper/2512.11373/) (2025)
- [Indexing Multimodal Language Models for Large-scale Image Retrieval](https://mlsanity.com/paper/2604.13268/) (2026)
- [OW-Rep: Open World Object Detection with Instance Representation Learning](https://mlsanity.com/paper/2409.16073/) (2024)
- [Objectomaly: Objectness-Aware Refinement for OoD Segmentation with Structural Consistency and Boundary Precision](https://mlsanity.com/paper/2507.07460/) (2025)
- [Towards Open World Detection: A Survey](https://mlsanity.com/paper/2508.16527/) (2025)
- [MVT: Mask-Grounded Vision-Language Models for Taxonomy-Aligned Land-Cover Tagging](https://mlsanity.com/paper/2509.18693/) (2025)
- [A Machine Learning Perspective on Automated Driving Corner Cases](https://mlsanity.com/paper/2510.10653/) (2025)
- [Language-Guided Open-World Anomaly Segmentation](https://mlsanity.com/paper/2512.01427/) (2025)
- [Learning to Identify Out-of-Distribution Objects for 3D LiDAR Anomaly Segmentation](https://mlsanity.com/paper/2604.23604/) (2026)
- [Trinity: Unifying Class-Agnostic Terrain and Semantic Segmentation for Unstructured Outdoor Environments by Leveraging Synthetic Data](https://mlsanity.com/paper/2605.27644/) (2026)
- [LARAD: Layout-Aware Road Anomaly Detection via Spatial-Logic Reasoning](https://mlsanity.com/paper/2607.12858/) (2026)
- [Still image and spatial-temporal tomato data enabling detection, segmentation, tracking, and video-instance segmentation using strong and weak labels](https://mlsanity.com/paper/2607.14934/) (2026)
- [Open-World Semantic Segmentation with Sensitivity Modeling](https://mlsanity.com/paper/2608.08308/) (2026)
- [Combining Foundation Model Confidence and Monocular Depth for Training-Free Out-of-Distribution Segmentation](https://mlsanity.com/paper/2609.22896/) (2026)
- [SAFe: Segment-guided Aggregation of Feature Densities for Anomaly-aware Segmentation](https://mlsanity.com/paper/2609.24204/) (2026)
- [Energy Ring: Stabilizing Feature Energy for Reliable Test-Time Discovery](https://mlsanity.com/paper/2610.14525/) (2026)
- [Relational Semantic Grounding for Test-Time Discovery](https://mlsanity.com/paper/2610.14721/) (2026)

## Abstract (arXiv)

Interpreting camera data is key for autonomously acting systems, such as autonomous vehicles. Vision systems that operate in real-world environments must be able to understand their surroundings and need the ability to deal with novel situations. This paper tackles open-world semantic segmentation, i.e., the variant of interpreting image data in which objects occur that have not been seen during training. We propose a novel approach that performs accurate closed-world semantic segmentation and, at the same time, can identify new categories without requiring any additional training data. Our approach additionally provides a similarity measure for every newly discovered class in an image to a known category, which can be useful information in downstream tasks such as planning or mapping. Through extensive experiments, we show that our model achieves state-of-the-art results on classes known from training data as well as for anomaly segmentation and can distinguish between different unknown classes.