# Semantic Image Segmentation: Two Decades of Research

arXiv: https://arxiv.org/abs/2302.06378 · HTML: https://mlsanity.com/paper/2302.06378/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 339 of its 514 references that are in the map.
- First posted on arXiv 13 Feb 2023.

## Builds on (most cited first, 30 of 339)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)

## Built on by (most cited first, 27 of 27)

- [All for One, and One for All: UrbanSyn Dataset, the third Musketeer of Synthetic Driving Scenes](https://mlsanity.com/paper/2312.12176/) (2023)
- [SatSynth: Augmenting Image-Mask Pairs through Diffusion Models for Aerial Semantic Segmentation](https://mlsanity.com/paper/2403.16605/) (2024)
- [One-shot Unsupervised Domain Adaptation with Personalized Diffusion Models](https://mlsanity.com/paper/2303.18080/) (2023)
- [Image Segmentation in Foundation Model Era: A Survey](https://mlsanity.com/paper/2408.12957/) (2024)
- [Reliability in Semantic Segmentation: Are We on the Right Track?](https://mlsanity.com/paper/2303.11298/) (2023)
- [Open-Vocabulary Camouflaged Object Segmentation](https://mlsanity.com/paper/2311.11241/) (2023)
- [FreeSeg-Diff: Training-Free Open-Vocabulary Segmentation with Diffusion Models](https://mlsanity.com/paper/2403.20105/) (2024)
- [Contrast, Stylize and Adapt: Unsupervised Contrastive Learning Framework for Domain Adaptive Semantic Segmentation](https://mlsanity.com/paper/2306.09098/) (2023)
- [RaSP: Relation-aware Semantic Prior for Weakly Supervised Incremental Segmentation](https://mlsanity.com/paper/2305.19879/) (2023)
- [Synthetic Data Generation Framework, Dataset, and Efficient Deep Model for Pedestrian Intention Prediction](https://mlsanity.com/paper/2401.06757/) (2024)
- [A Simple and Generalist Approach for Panoptic Segmentation](https://mlsanity.com/paper/2408.16504/) (2024)
- [SAMIC: Segment Anything with In-Context Spatial Prompt Engineering](https://mlsanity.com/paper/2412.11998/) (2024)
- [Leveraging LLMs and attention-mechanism for automatic annotation of historical maps](https://mlsanity.com/paper/2504.11050/) (2025)
- [GeloVec: Higher Dimensional Geometric Smoothing for Coherent Visual Feature Extraction in Image Segmentation](https://mlsanity.com/paper/2505.01057/) (2025)
- [Weatherproofing Retrieval for Localization with Generative AI and Geometric Consistency](https://mlsanity.com/paper/2402.09237/) (2024)
- [Divide, Ensemble and Conquer: The Last Mile on Unsupervised Domain Adaptation for Semantic Segmentation](https://mlsanity.com/paper/2406.18809/) (2024)
- [Optimizing Dense Visual Predictions Through Multi-Task Coherence and Prioritization](https://mlsanity.com/paper/2412.03179/) (2024)
- [Dedicated Inference Engine and Binary-Weight Neural Networks for Lightweight Instance Segmentation](https://mlsanity.com/paper/2501.01841/) (2025)
- [Automated Thoracolumbar Stump Rib Detection and Analysis in a Large CT Cohort](https://mlsanity.com/paper/2505.05004/) (2025)
- [FedS2R: One-Shot Federated Domain Generalization for Synthetic-to-Real Semantic Segmentation in Autonomous Driving](https://mlsanity.com/paper/2507.19881/) (2025)
- [FS-SAM2: Adapting Segment Anything Model 2 for Few-Shot Semantic Segmentation via Low-Rank Adaptation](https://mlsanity.com/paper/2509.12105/) (2025)
- [SegFly: A Dataset and 2D-3D-2D Paradigm for Aerial RGB-Thermal Semantic Segmentation at Scale](https://mlsanity.com/paper/2603.17920/) (2026)
- [Make me an Expert: Distilling from Generalist Black-Box Models into Specialized Models for Semantic Segmentation](https://mlsanity.com/paper/2509.00509/) (2025)
- [Auto Quantum Machine Learning for Multisource Classification](https://mlsanity.com/paper/2602.18642/) (2026)
- [Trinity: Unifying Class-Agnostic Terrain and Semantic Segmentation for Unstructured Outdoor Environments by Leveraging Synthetic Data](https://mlsanity.com/paper/2605.27644/) (2026)
- [CoordFormer: Give Me Any Coordinates and I Will Give You Labels](https://mlsanity.com/paper/2609.08660/) (2026)
- [Why DiT Models Underperform as Representation Learners without Long Skip Connections](https://mlsanity.com/paper/2610.13344/) (2026)

## Abstract (arXiv)

Semantic image segmentation (SiS) plays a fundamental role in a broad variety of computer vision applications, providing key information for the global understanding of an image. This survey is an effort to summarize two decades of research in the field of SiS, where we propose a literature review of solutions starting from early historical methods followed by an overview of more recent deep learning methods including the latest trend of using transformers. We complement the review by discussing particular cases of the weak supervision and side machine learning techniques that can be used to improve the semantic segmentation such as curriculum, incremental or self-supervised learning. State-of-the-art SiS models rely on a large amount of annotated samples, which are more expensive to obtain than labels for tasks such as image classification. Since unlabeled data is instead significantly cheaper to obtain, it is not surprising that Unsupervised Domain Adaptation (UDA) reached a broad success within the semantic segmentation community. Therefore, a second core contribution of this book is to summarize five years of a rapidly growing field, Domain Adaptation for Semantic Image Segmentation (DASiS) which embraces the importance of semantic segmentation itself and a critical need of adapting segmentation models to new environments. In addition to providing a comprehensive survey on DASiS techniques, we unveil also newer trends such as multi-domain learning, domain generalization, domain incremental learning, test-time adaptation and source-free domain adaptation. Finally, we conclude this survey by describing datasets and benchmarks most widely used in SiS and DASiS and briefly discuss related tasks such as instance and panoptic image segmentation, as well as applications such as medical image segmentation.