mlsanityData to 14 Oct 2026

Transformer-Based Visual Segmentation: A Survey

arXiv 2304.09854 ↗First posted 19 Apr 2023cs.CVcited by 124

Not on the map. Why?

Abstract (arXiv)

Visual segmentation seeks to partition images, video frames, or point clouds into multiple segments or groups. This technique has numerous real-world applications, such as autonomous driving, image editing, robot sensing, and medical analysis. Over the past decade, deep learning-based methods have made remarkable strides in this area. Recently, transformers, a type of neural network based on self-attention originally designed for natural language processing, have considerably surpassed previous convolutional or recurrent approaches in various vision processing tasks. Specifically, vision transformers offer robust, unified, and even simpler solutions for various segmentation tasks. This survey provides a thorough overview of transformer-based visual segmentation, summarizing recent advancements. We first review the background, encompassing problem definitions, datasets, and prior convolutional methods. Next, we summarize a meta-architecture that unifies all recent transformer-based approaches. Based on this meta-architecture, we examine various method designs, including modifications to the meta-architecture and associated applications. We also present several closely related settings, including 3D point cloud segmentation, foundation model tuning, domain-aware segmentation, efficient segmentation, and medical segmentation. Additionally, we compile and re-evaluate the reviewed methods on several well-established datasets. Finally, we identify open challenges in this field and propose directions for future research. The project page can be found at https://github.com/lxtGH/Awesome-Segmentation-With-Transformer. We will also continually monitor developments in this rapidly evolving field.

Key facts. Cited by 124 papers in the map (data through 14 Oct 2026). Builds on 336 of its 419 references that are in the map. First posted on arXiv 19 Apr 2023.

Builds on

336 of 419 references.

References · 336

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  5. 2020Language Models are Few-Shot Learnerscited by 32,419
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  7. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  8. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  9. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  10. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  11. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  12. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
38 more
  1. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  2. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  3. 2023Segment Anythingcited by 8,234
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  5. 2020End-to-End Object Detection with Transformerscited by 7,414
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  7. 2017Mask R-CNNcited by 7,277
  8. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  9. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,248
  10. 2016Feature Pyramid Networks for Object Detectioncited by 5,036
  11. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,859
  12. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  13. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  14. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  15. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  16. 2022A ConvNet for the 2020scited by 3,579
  17. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
  18. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,241
  19. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,223
  20. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,940
  21. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  22. 2016Pyramid Scene Parsing Networkcited by 2,479
  23. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  24. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,351
  25. 2017Non-local Neural Networkscited by 2,293
  26. 2016V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentationcited by 2,292
  27. 2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,161
  28. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
  29. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
  30. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  31. 2021TransUNet: Transformers Make Strong Encoders for Medical Image Segmentationcited by 1,558
  32. 2020Point Transformercited by 1,470
  33. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  34. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,446
  35. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,433
  36. 2019MMDetection: Open MMLab Detection Toolbox and Benchmarkcited by 1,385
  37. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,369
  38. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322

50 most cited of 336 All 336 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 29; 2024: 42; 2025: 40; 2026: 13.

124 papers.

Sort:

By area:

  1. 2023GRES: Generalized Referring Expression Segmentationcited by 254
  2. 2023MeViS: A Large-scale Benchmark for Video Segmentation with Motion Expressionscited by 228
  3. 2023Towards Open Vocabulary Learning: A Surveycited by 166
  4. 2024OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understandingcited by 127
  5. 2024MambaAD: Exploring State Space Models for Multi-class Unsupervised Anomaly Detectioncited by 113
  6. 2024SIDA: Social Media Image Deepfake Detection, Localization and Explanation with Large Multimodal Modelcited by 108
  7. 2023The Segment Anything Model (SAM) for Remote Sensing Applications: From Zero to One Shotcited by 83
  8. 2023One-for-All: Bridge the Gap Between Heterogeneous Architectures in Knowledge Distillationcited by 78
  9. 2024Open-Vocabulary SAM: Segment and Recognize Twenty-thousand Classes Interactivelycited by 75
  10. 2024OMG-Seg: Is One Model Good Enough For All Segmentation?cited by 75
  11. 2024Point Cloud Mamba: Point Cloud Learning via State Space Modelcited by 66
  12. 2025MOSEv2: A More Challenging Dataset for Video Object Segmentation in Complex Scenescited by 62
38 more
  1. 2024Decoupling Static and Hierarchical Motion Perception for Referring Video Segmentationcited by 55
  2. 2024A Survey of Multimodal-Guided Image Editing with Text-to-Image Diffusion Modelscited by 52
  3. 2023CTVIS: Consistent Training for Online Video Instance Segmentationcited by 51
  4. 2024Vitron: A Unified Pixel-level Vision LLM for Understanding, Generating, Segmenting, Editingcited by 49
  5. 2023RenAIssance: A Survey into AI Text-to-Image Generation in the Era of Large Modelcited by 47
  6. 2023Exploring Plain ViT Reconstruction for Multi-class Unsupervised Anomaly Detectioncited by 47
  7. 2023TaskExpert: Dynamically Assembling Multi-Task Representations with Memorial Mixture-of-Expertscited by 45
  8. 2024Towards Semantic Equivalence of Tokenization in Multimodal LLMcited by 45
  9. 2023GREC: Generalized Referring Expression Comprehensioncited by 42
  10. 2023Change Detection Methods for Remote Sensing in the Last Decade: A Comprehensive Reviewcited by 41
  11. 2023Multi-Modal Mutual Attention and Iterative Interaction for Referring Image Segmentationcited by 36
  12. 2024Image Segmentation in Foundation Model Era: A Surveycited by 35
  13. 2025MeViS: A Multi-Modal Dataset for Referring Motion Expression Video Segmentationcited by 35
  14. 2024Self-Calibrated CLIP for Training-Free Open-Vocabulary Segmentationcited by 34
  15. 2024Towards Language-Driven Video Inpainting via Multimodal Large Language Modelscited by 33
  16. 2025X-SAM: From Segment Anything to Any Segmentationcited by 30
  17. 2023Primitive Generation and Semantic-related Alignment for Universal Zero-Shot Segmentationcited by 28
  18. 2023Prototype Adaption and Projection for Few- and Zero-shot 3D Point Cloud Semantic Segmentationcited by 26
  19. 2024RefMask3D: Language-Guided Transformer for 3D Referring Segmentationcited by 25
  20. 2023TransFace++: Rethinking the Face Recognition Paradigm with a Focus on Accuracy, Efficiency, and Securitycited by 22
  21. 2024CapHuman: Capture Your Moments in Parallel Universescited by 22
  22. 20244D Panoptic Scene Graph Generationcited by 22
  23. 2025Multimodal Referring Segmentation: A Surveycited by 22
  24. 2024Textual Query-Driven Mask Transformer for Domain Generalized Segmentationcited by 21
  25. 2024SemFlow: Binding Semantic Segmentation and Image Synthesis via Rectified Flowcited by 20
  26. 2025SegDINO: An Efficient Design for Medical and Natural Image Segmentation with DINO-V3cited by 20
  27. 2023SegGen: Supercharging Segmentation Models with Text2Mask and Mask2Img Synthesiscited by 18
  28. 2024No Time to Train: Empowering Non-Parametric Networks for Few-shot 3D Scene Segmentationcited by 18
  29. 2024Mamba or RWKV: Exploring High-Quality and High-Efficiency Segment Anything Modelcited by 18
  30. 2023Semantic-Promoted Debiasing and Background Disambiguation for Zero-Shot Instance Segmentationcited by 16
  31. 2025DenseWorld-1M: Towards Detailed Dense Grounded Caption in the Real Worldcited by 16
  32. 2024BA-SAM: Scalable Bias-Mode Attention Mask for Segment Anything Modelcited by 15
  33. 2024Customize Segment Anything Model for Multi-Modal Semantic Segmentation with Mixture of LoRA Expertscited by 15
  34. 2024UnSeg: One Universal Unlearnable Example Generator is Enough against All Image Segmentationcited by 14
  35. 2025Circle-RoPE: Cone-like Decoupled Rotary Positional Embedding for Large Vision-Language Modelscited by 14
  36. 2024Explore In-Context Segmentation via Latent Diffusion Modelscited by 12
  37. 2024PVUW 2024 Challenge on Complex Video Understanding: Methods and Resultscited by 12
  38. 2025Hierarchical Alignment-enhanced Adaptive Grounding Network for Generalized Referring Expression Comprehensioncited by 12

50 most cited of 124 All 124 (JSON)