# CSL: Class-Agnostic Structure-Constrained Learning for Segmentation Including the Unseen

arXiv: https://arxiv.org/abs/2312.05538 · HTML: https://mlsanity.com/paper/2312.05538/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 52 references that are in the map.
- First posted on arXiv 9 Dec 2023.

## Builds on (most cited first, 30 of 43)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks](https://mlsanity.com/paper/1706.02690/) (2017)
- [Semantic Understanding of Scenes through the ADE20K Dataset](https://mlsanity.com/paper/1608.05442/) (2016)
- [Deep Anomaly Detection with Outlier Exposure](https://mlsanity.com/paper/1812.04606/) (2018)
- [ADVENT: Adversarial Entropy Minimization for Domain Adaptation in Semantic Segmentation](https://mlsanity.com/paper/1811.12833/) (2018)
- [Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Ugly](https://mlsanity.com/paper/1707.00600/) (2017)
- [Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samples](https://mlsanity.com/paper/1711.09325/) (2017)
- [DAFormer: Improving Network Architectures and Training Strategies for Domain-Adaptive Semantic Segmentation](https://mlsanity.com/paper/2111.14887/) (2021)
- [Characterizing Adversarial Subspaces Using Local Intrinsic Dimensionality](https://mlsanity.com/paper/1801.02613/) (2018)
- [Zero-Shot Semantic Segmentation](https://mlsanity.com/paper/1906.00817/) (2019)
- [Weakly-supervised Video Anomaly Detection with Robust Temporal Feature Magnitude Learning](https://mlsanity.com/paper/2101.10030/) (2021)
- [A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Model](https://mlsanity.com/paper/2112.14757/) (2021)
- [MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretraining](https://mlsanity.com/paper/2208.12262/) (2022)
- [Mask2Former for Video Instance Segmentation](https://mlsanity.com/paper/2112.10764/) (2021)
- [SegmentMeIfYouCan: A Benchmark for Anomaly Segmentation](https://mlsanity.com/paper/2104.14812/) (2021)
- [Evaluating Bayesian Deep Learning Methods for Semantic Segmentation](https://mlsanity.com/paper/1811.12709/) (2018)
- [Detecting the Unexpected via Image Resynthesis](https://mlsanity.com/paper/1904.07595/) (2019)
- [Entropy Maximization and Meta Classification for Out-Of-Distribution Detection in Semantic Segmentation](https://mlsanity.com/paper/2012.06575/) (2020)
- [Lost and Found: Detecting Small Road Hazards for Self-Driving Vehicles](https://mlsanity.com/paper/1609.04653/) (2016)
- [Pixel-wise Anomaly Detection in Complex Driving Scenes](https://mlsanity.com/paper/2103.05445/) (2021)
- [Synscapes: A Photorealistic Synthetic Dataset for Street Scene Parsing](https://mlsanity.com/paper/1810.08705/) (2018)
- [Synthesize then Compare: Detecting Failures and Anomalies for Semantic Segmentation](https://mlsanity.com/paper/2003.08440/) (2020)
- [Pixel-wise Energy-biased Abstention Learning for Anomaly Segmentation on Complex Urban Driving Scenes](https://mlsanity.com/paper/2111.12264/) (2021)

## Built on by (most cited first, 15 of 15)

- [PixOOD: Pixel-Level Out-of-Distribution Detection](https://mlsanity.com/paper/2405.19882/) (2024)
- [MagicPose4D: Crafting Articulated Models with Appearance and Motion Control](https://mlsanity.com/paper/2405.14017/) (2024)
- [Out-of-Distribution Segmentation in Autonomous Driving: Problems and State of the Art](https://mlsanity.com/paper/2503.08695/) (2025)
- [S3O: A Dual-Phase Approach for Reconstructing Dynamic Shape and Skeleton of Articulated Objects from Single Monocular Video](https://mlsanity.com/paper/2405.12607/) (2024)
- [A Likelihood Ratio-Based Approach to Segmenting Unknown Objects](https://mlsanity.com/paper/2409.06424/) (2024)
- [Prior2Former -- Evidential Modeling of Mask Transformers for Assumption-Free Open-World Panoptic Segmentation](https://mlsanity.com/paper/2504.04841/) (2025)
- [Improving Multi-label Recognition using Class Co-Occurrence Probabilities](https://mlsanity.com/paper/2404.16193/) (2024)
- [DUDA: Distilled Unsupervised Domain Adaptation for Lightweight Semantic Segmentation](https://mlsanity.com/paper/2504.09814/) (2025)
- [Uncertainty-Aware Likelihood Ratio Estimation for Pixel-Wise Out-of-Distribution Detection](https://mlsanity.com/paper/2508.00587/) (2025)
- [Benchmarking Vision Foundation Models for Input Monitoring in Autonomous Driving](https://mlsanity.com/paper/2501.08083/) (2025)
- [Panoramic Out-of-Distribution Segmentation](https://mlsanity.com/paper/2505.03539/) (2025)
- [From Pixel to Mask: A Survey of Out-of-Distribution Segmentation](https://mlsanity.com/paper/2508.10309/) (2025)
- [Bridge the Gap Between Visual and Linguistic Comprehension for Generalized Zero-shot Semantic Segmentation](https://mlsanity.com/paper/2503.23806/) (2025)
- [Objectomaly: Objectness-Aware Refinement for OoD Segmentation with Structural Consistency and Boundary Precision](https://mlsanity.com/paper/2507.07460/) (2025)
- [Combining Foundation Model Confidence and Monocular Depth for Training-Free Out-of-Distribution Segmentation](https://mlsanity.com/paper/2609.22896/) (2026)

## Abstract (arXiv)

Addressing Out-Of-Distribution (OOD) Segmentation and Zero-Shot Semantic Segmentation (ZS3) is challenging, necessitating segmenting unseen classes. Existing strategies adapt the class-agnostic Mask2Former (CA-M2F) tailored to specific tasks. However, these methods cater to singular tasks, demand training from scratch, and we demonstrate certain deficiencies in CA-M2F, which affect performance. We propose the Class-Agnostic Structure-Constrained Learning (CSL), a plug-in framework that can integrate with existing methods, thereby embedding structural constraints and achieving performance gain, including the unseen, specifically OOD, ZS3, and domain adaptation (DA) tasks. There are two schemes for CSL to integrate with existing methods (1) by distilling knowledge from a base teacher network, enforcing constraints across training and inference phrases, or (2) by leveraging established models to obtain per-pixel distributions without retraining, appending constraints during the inference phase. We propose soft assignment and mask split methodologies that enhance OOD object segmentation. Empirical evaluations demonstrate CSL's prowess in boosting the performance of existing algorithms spanning OOD segmentation, ZS3, and DA segmentation, consistently transcending the state-of-art across all three tasks.