mlsanityData to 14 Oct 2026

Semantic Segmentation using Vision Transformers: A survey

arXiv 2305.03273 ↗First posted 5 May 2023cs.CV cs.AI cs.LGcited by 73

Not on the map. Why?

Abstract (arXiv)

Semantic segmentation has a broad range of applications in a variety of domains including land coverage analysis, autonomous driving, and medical image analysis. Convolutional neural networks (CNN) and Vision Transformers (ViTs) provide the architecture models for semantic segmentation. Even though ViTs have proven success in image classification, they cannot be directly applied to dense prediction tasks such as image segmentation and object detection since ViT is not a general purpose backbone due to its patch partitioning scheme. In this survey, we discuss some of the different ViT architectures that can be used for semantic segmentation and how their evolution managed the above-stated challenge. The rise of ViT and its performance with a high success rate motivated the community to slowly replace the traditional convolutional neural networks in various computer vision tasks. This survey aims to review and compare the performances of ViT architectures designed for semantic segmentation using benchmarking datasets. This will be worthwhile for the community to yield knowledge regarding the implementations carried out in semantic segmentation and to discover more efficient methodologies using ViTs.

Key facts. Cited by 73 papers in the map (data through 14 Oct 2026). Builds on 60 of its 83 references that are in the map. First posted on arXiv 5 May 2023.

Builds on

60 of 83 references.

References · 60

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  6. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  7. 2016Feature Pyramid Networks for Object Detectioncited by 5,036
  8. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,859
  9. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  10. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  11. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
  12. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,940
38 more
  1. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,351
  2. 2016V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentationcited by 2,292
  3. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
  4. 2021Vision Transformers for Dense Predictioncited by 1,803
  5. 2021TransUNet: Transformers Make Strong Encoders for Medical Image Segmentationcited by 1,558
  6. 2018UNet++: A Nested U-Net Architecture for Medical Image Segmentationcited by 1,525
  7. 2018Attention U-Net: Learning Where to Look for the Pancreascited by 1,435
  8. 2019Deep High-Resolution Representation Learning for Visual Recognitioncited by 1,420
  9. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,369
  10. 2021Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentationcited by 1,222
  11. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,152
  12. 2021Transformers in Vision: A Surveycited by 1,013
  13. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  14. 2016Playing for Data: Ground Truth from Computer Gamescited by 818
  15. 2021CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classificationcited by 806
  16. 2021PVT v2: Improved Baselines with Pyramid Vision Transformercited by 751
  17. 2020Self-supervised Learning: Generative or Contrastivecited by 694
  18. 2020A Survey on Visual Transformercited by 654
  19. 2021Twins: Revisiting the Design of Spatial Attention in Vision Transformerscited by 565
  20. 2017Skin Lesion Analysis Toward Melanoma Detection: A Challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), Hosted by the International Skin Imaging Collaboration (ISIC)cited by 549
  21. 2019Stand-Alone Self-Attention in Vision Modelscited by 542
  22. 2019The Liver Tumor Segmentation Benchmark (LiTS)cited by 509
  23. 2020End-to-End Video Instance Segmentation with Transformerscited by 443
  24. 2021Conditional Positional Encodings for Vision Transformerscited by 406
  25. 2016Virtual Worlds as Proxy for Multi-Object Tracking Analysiscited by 341
  26. 2020UNet 3+: A Full-Scale Connected UNet for Medical Image Segmentationcited by 332
  27. 2022How Do Vision Transformers Work?cited by 307
  28. 2018DeepGlobe 2018: A Challenge to Parse the Earth through Satellite Imagescited by 277
  29. 2019ResUNet-a: a deep learning framework for semantic segmentation of remotely sensed datacited by 261
  30. 2021LoveDA: A Remote Sensing Land-Cover Dataset for Domain Adaptive Semantic Segmentationcited by 261
  31. 2019CE-Net: Context Encoder Network for 2D Medical Image Segmentationcited by 220
  32. 2020A survey of loss functions for semantic segmentationcited by 217
  33. 2019Local Relation Networks for Image Recognitioncited by 210
  34. 2016Deep Feature Flow for Video Recognitioncited by 186
  35. 2017Computer Vision for Autonomous Vehicles: Problems, Datasets and State of the Artcited by 182
  36. 2018IDD: A Dataset for Exploring Problems of Autonomous Navigation in Unconstrained Environmentscited by 163
  37. 2021A Survey of Visual Transformerscited by 117
  38. 2019A Survey of Deep Learning Applications to Autonomous Vehicle Controlcited by 106

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 9; 2024: 23; 2025: 30; 2026: 11.

73 papers.

Sort:

By area:

  1. 2024SelfSplat: Pose-Free and 3D Prior-Free Generalizable 3D Gaussian Splattingcited by 42
  2. 2023Assessment of a new GeoAI foundation model for flood inundation mappingcited by 14
  3. 2023Open-Vocabulary Camouflaged Object Segmentationcited by 14
  4. 2024SwinMTL: A Shared Architecture for Simultaneous Depth Estimation and Semantic Segmentation from Monocular Camera Imagescited by 13
  5. 2023A Survey on Deep Learning for Polyp Segmentation: Techniques, Challenges and Future Trendscited by 12
  6. 2024PRANCE: Joint Token-Optimization and Structural Channel-Pruning for Adaptive ViT Inferencecited by 10
  7. 2024A citizen science toolkit to collect human perceptions of urban environments using open street view imagescited by 9
  8. 2024A Survey of the Self Supervised Learning Mechanisms for Vision Transformerscited by 9
  9. 2024CLFT: Camera-LiDAR Fusion Transformer for Semantic Segmentation in Autonomous Drivingcited by 7
  10. 2024Adaptive Patching for High-resolution Image Segmentation with Transformerscited by 6
  11. 2024Variational Autoencoder for Anomaly Detection: A Comparative Studycited by 6
  12. 2025TransMamba: Fast Universal Architecture Adaption from Transformers to Mambacited by 6
38 more
  1. 2025Isolated Channel Vision Transformers: From Single-Channel Pretraining to Multi-Channel Finetuningcited by 5
  2. 2025Segmenting Objectiveness and Task-awareness Unknown Region for Autonomous Drivingcited by 5
  3. 2025AVA-Bench: Atomic Visual Ability Benchmark for Vision Foundation Modelscited by 5
  4. 2023A Comprehensive Survey of Document-level Relation Extraction (2016-2023)cited by 4
  5. 2024An Efficient Rehearsal Scheme for Catastrophic Forgetting Mitigation during Multi-stage Fine-tuningcited by 4
  6. 2025Janus: Collaborative Vision Transformer Under Dynamic Network Environmentcited by 4
  7. 2025CLICv2: Image Complexity Representation via Content Invariance Contrastive Learningcited by 4
  8. 2025SemiDAViL: Semi-supervised Domain Adaptation with Vision-Language Guidance for Semantic Segmentationcited by 4
  9. 2026Sapiens2cited by 4
  10. 2023gcDLSeg: Integrating Graph-cut into Deep Learning for Binary Semantic Segmentationcited by 3
  11. 2024Token Pruning using a Lightweight Background Aware Vision Transformercited by 3
  12. 2025LL-ViT: Edge Deployable Vision Transformers with Look Up Table Neuronscited by 3
  13. 2023Not All Pixels Are Equal: Learning Pixel Hardness for Semantic Segmentationcited by 2
  14. 2024Independently Keypoint Learning for Small Object Semantic Correspondencecited by 2
  15. 2024SOOD-ImageNet: a Large-Scale Dataset for Semantic Out-Of-Distribution Image Classification and Semantic Segmentationcited by 2
  16. 2025Enhancing kelp forest detection in remote sensing images using crowdsourced labels with Mixed Vision Transformers and ConvNeXt segmentation modelscited by 2
  17. 2025Context-Aware Semantic Segmentation: Enhancing Pixel-Level Understanding with Large Language Models for Advanced Vision Applicationscited by 2
  18. 2025WoundAmbit: Bridging State-of-the-Art Semantic Segmentation and Real-World Wound Carecited by 2
  19. 2025GeloVec: Higher Dimensional Geometric Smoothing for Coherent Visual Feature Extraction in Image Segmentationcited by 2
  20. 2025Image Recognition with Online Lightweight Vision Transformer: A Surveycited by 2
  21. 2025Enhancing Quantization-Aware Training on Edge Devices via Relative Entropy Coreset Selection and Cascaded Layer Correctioncited by 2
  22. 2023Toward Flare-Free Images: A Surveycited by 1
  23. 2024Adaptive Point Transformercited by 1
  24. 2024Influence of Water Droplet Contamination for Transparency Segmentationcited by 1
  25. 2024Using Interleaved Ensemble Unlearning to Keep Backdoors at Bay for Finetuning Vision Transformerscited by 1
  26. 2024ULTra: Unveiling Latent Token Interpretability in Transformer-Based Understanding and Segmentationcited by 1
  27. 2024AFANet: Adaptive Frequency-Aware Network for Weakly-Supervised Few-Shot Semantic Segmentationcited by 1
  28. 2025Task-based Loss Functions in Computer Vision: A Comprehensive Reviewcited by 1
  29. 2025Masked Autoencoder Self Pre-Training for Defect Detection in Microelectronicscited by 1
  30. 2025WoundAIssist: Development and Evaluation of an AI-Based Mobile Application for Remote Chronic Wound Care in Elderly Patientscited by 1
  31. 2025Multimodal SAM-adapter for Semantic Segmentationcited by 1
  32. 2025Multi-Scale High-Resolution Logarithmic Grapher Module for Efficient Vision GNNscited by 1
  33. 2025Assessing the value of Geo-Foundational Models for Flood Inundation Mapping: Benchmarking models for Sentinel-1, Sentinel-2, and Planetscope for end-userscited by 1
  34. 2023Scalable Label-efficient Footpath Network Generation Using Remote Sensing Data and Self-supervised Learningcited by 0
  35. 2023BenthIQ: a Transformer-Based Benthic Classification Model for Coral Restorationcited by 0
  36. 2024Towards In-Vehicle Multi-Task Facial Attribute Recognition: Investigating Synthetic Data and Vision Foundation Modelscited by 0
  37. 2024Deep Learning-Based Segmentation of Tumors in PET/CT Volumes: Benchmark of Different Architectures and Training Strategiescited by 0
  38. 2024Leveraging Knowledge Distillation for Lightweight Skin Cancer Classification: Balancing Accuracy and Computational Efficiencycited by 0

50 most cited of 73 All 73 (JSON)