mlsanityData to 14 Oct 2026

SegViT: Semantic Segmentation with Plain Vision Transformers

arXiv 2210.05844 ↗First posted 12 Oct 2022cs.CVcited by 106

Not on the map. Why?

Abstract (arXiv)

We explore the capability of plain Vision Transformers (ViTs) for semantic segmentation and propose the SegVit. Previous ViT-based segmentation networks usually learn a pixel-level representation from the output of the ViT. Differently, we make use of the fundamental component -- attention mechanism, to generate masks for semantic segmentation. Specifically, we propose the Attention-to-Mask (ATM) module, in which the similarity maps between a set of learnable class tokens and the spatial feature maps are transferred to the segmentation masks. Experiments show that our proposed SegVit using the ATM module outperforms its counterparts using the plain ViT backbone on the ADE20K dataset and achieves new state-of-the-art performance on COCO-Stuff-10K and PASCAL-Context datasets. Furthermore, to reduce the computational cost of the ViT backbone, we propose query-based down-sampling (QD) and query-based up-sampling (QU) to build a Shrunk structure. With the proposed Shrunk structure, the model can save up to 40% computations while maintaining competitive performance.

Key facts. Cited by 106 papers in the map (data through 14 Oct 2026). Builds on 40 of its 49 references that are in the map. First posted on arXiv 12 Oct 2022.

Builds on

40 of 49 references.

References · 40

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  4. 2020End-to-End Object Detection with Transformerscited by 7,414
  5. 2016Feature Pyramid Networks for Object Detectioncited by 5,036
  6. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,859
  7. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,241
  8. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,223
  9. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,940
  10. 2016Pyramid Scene Parsing Networkcited by 2,479
  11. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,351
  12. 2016V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentationcited by 2,292
28 more
  1. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
  2. 2021Vision Transformers for Dense Predictioncited by 1,803
  3. 2018UNet++: A Nested U-Net Architecture for Medical Image Segmentationcited by 1,525
  4. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,369
  5. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,271
  6. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,152
  7. 2018Dual Attention Network for Scene Segmentationcited by 1,003
  8. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 864
  9. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  10. 2019Segmentation Transformer: Object-Contextual Representations for Semantic Segmentationcited by 576
  11. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 569
  12. 2021Twins: Revisiting the Design of Spatial Attention in Vision Transformerscited by 565
  13. 2016RefineNet: Multi-Path Refinement Networks for High-Resolution Semantic Segmentationcited by 465
  14. 2018Context Encoding for Semantic Segmentationcited by 290
  15. 2021K-Net: Towards Unified Image Segmentationcited by 212
  16. 2019High-Resolution Representations for Labeling Pixels and Regionscited by 211
  17. 2019Expectation-Maximization Attention Networks for Semantic Segmentationcited by 147
  18. 2020Context Prior for Scene Segmentationcited by 121
  19. 2021P2T: Pyramid Pooling Transformer for Scene Understandingcited by 84
  20. 2019GFF: Gated Fully Fusion for Semantic Segmentationcited by 57
  21. 2021Mining Contextual Information Beyond Image for Semantic Segmentationcited by 48
  22. 2020Spatial Pyramid Based Graph Reasoning for Semantic Segmentationcited by 47
  23. 2021ISNet: Integrate Image-Level and Semantic-Level Context for Semantic Segmentationcited by 40
  24. 2022StructToken : Rethinking Semantic Segmentation with Structural Priorcited by 30
  25. 2020Tensor Low-Rank Reconstruction for Semantic Segmentationcited by 29
  26. 2020EfficientFCN: Holistically-guided Decoding for Semantic Segmentationcited by 15
  27. 2020GINet: Graph Interaction Network for Scene Parsingcited by 13
  28. 2021Dynamic Neural Representational Decoders for High-Resolution Semantic Segmentationcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 3; 2023: 30; 2024: 28; 2025: 32; 2026: 13.

106 papers.

Sort:

By area:

  1. 2022ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentationcited by 178
  2. 2023CLIPSelf: Vision Transformer Distills Itself for Open-Vocabulary Dense Predictioncited by 118
  3. 2023Delivering Arbitrary-Modal Semantic Segmentationcited by 117
  4. 2023Region-Aware Pretraining for Open-Vocabulary Object Detection with Vision Transformerscited by 91
  5. 2023Stronger, Fewer, & Superior: Harnessing Vision Foundation Models for Domain Generalized Semantic Segmentationcited by 83
  6. 2025Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Groundingcited by 83
  7. 2022ViTPose++: Vision Transformer for Generic Body Pose Estimationcited by 65
  8. 2023Dynamic Token Pruning in Plain Vision Transformers for Semantic Segmentationcited by 62
  9. 2023A Survey on Open-Vocabulary Detection and Segmentation: Past, Present, and Futurecited by 61
  10. 2024LaSagnA: Language-based Segmentation Assistant for Complex Queriescited by 55
  11. 2023Exploring Plain ViT Reconstruction for Multi-class Unsupervised Anomaly Detectioncited by 47
  12. 2023Conflict-Based Cross-View Consistency for Semi-Supervised Semantic Segmentationcited by 35
38 more
  1. 2023Contrastive Feature Masking Open-Vocabulary Vision Transformercited by 31
  2. 2024ALGM: Adaptive Local-then-Global Token Merging for Efficient Semantic Segmentation with Plain Vision Transformerscited by 28
  3. 2023SegViTv2: Exploring Efficient and Continual Semantic Segmentation with Plain Vision Transformerscited by 25
  4. 2024Context-Guided Spatial Feature Reconstruction for Efficient Semantic Segmentationcited by 25
  5. 2025USP: Unified Self-Supervised Pretraining for Image Generation and Understandingcited by 24
  6. 2024Parameter-Efficient Fine-Tuning for Pre-Trained Vision Models: A Survey and Benchmarkcited by 23
  7. 2023InvPT++: Inverted Pyramid Multi-Task Transformer for Visual Scene Understandingcited by 22
  8. 2023SCTNet: Single-Branch CNN with Transformer Semantic Information for Real-Time Segmentationcited by 22
  9. 2024Centering the Value of Every Modality: Towards Efficient and Resilient Modality-agnostic Semantic Segmentationcited by 22
  10. 2024Multi-Layer Transformers Gradient Can be Approximated in Almost Linear Timecited by 22
  11. 2024Context-Aware Interaction Network for RGB-T Semantic Segmentationcited by 21
  12. 2024Textual Query-Driven Mask Transformer for Domain Generalized Segmentationcited by 21
  13. 2026Generation Enhances Understanding in Unified Multimodal Models via Multi-Representation Generationcited by 20
  14. 2023Scattering Vision Transformer: Spectral Mixing Matterscited by 17
  15. 2024Cascade-CLIP: Cascaded Vision-Language Embeddings Alignment for Zero-Shot Semantic Segmentationcited by 17
  16. 2025Hypergraph Vision Transformers: Images are More than Nodes, More than Edgescited by 16
  17. 2023BPKD: Boundary Privileged Knowledge Distillation For Semantic Segmentationcited by 15
  18. 2023TagCLIP: Improving Discrimination Ability of Open-Vocabulary Semantic Segmentationcited by 14
  19. 2023Win-Win: Training High-Resolution Vision Transformers from Two Windowscited by 12
  20. 2024Open-Vocabulary Segmentation with Unpaired Mask-Text Supervisioncited by 12
  21. 2024MAGIC++: Efficient and Resilient Modality-Agnostic Semantic Segmentation via Hierarchical Modality Selectioncited by 12
  22. 2024CatLIP: CLIP-level Visual Recognition Accuracy with 2.7x Faster Pre-training on Web-scale Image-Text Datacited by 10
  23. 2024LookHere: Vision Transformers with Directed Attention Generalize and Extrapolatecited by 10
  24. 2025Frequency-Dynamic Attention Modulation for Dense Predictioncited by 10
  25. 2023Stitched ViTs are Flexible Vision Backbonescited by 8
  26. 2023APTv2: Benchmarking Animal Pose Estimation and Tracking with a Large-scale Dataset and Beyondcited by 8
  27. 2024Memorize What Matters: Emergent Scene Decomposition from Multitraversecited by 8
  28. 2024RGB2Point: 3D Point Cloud Generation from Single RGB Imagescited by 7
  29. 2025Dynamic Dictionary Learning for Remote Sensing Image Segmentationcited by 7
  30. 2024Language-Driven Visual Consensus for Zero-Shot Semantic Segmentationcited by 6
  31. 2024MLIP: Efficient Multi-Perspective Language-Image Pretraining with Exhaustive Data Utilizationcited by 6
  32. 2024CathAction: A Benchmark for Endovascular Intervention Understandingcited by 6
  33. 2023Multi-Scale And Token Mergence: Make Your ViT More Efficientcited by 5
  34. 2025FocusTrack: A Self-Adaptive Local Sampling Algorithm for Efficient Anti-UAV Trackingcited by 5
  35. 2025Revisiting Efficient Semantic Segmentation: Learning Offsets for Better Spatial and Class Feature Alignmentcited by 5
  36. 2022Representation Separation for Semantic Segmentation with Vision Transformerscited by 4
  37. 2023ShiftNAS: Improving One-shot NAS via Probability Shiftcited by 4
  38. 2024SSA-Seg: Semantic and Spatial Adaptive Pixel-level Classifier for Semantic Segmentationcited by 4

50 most cited of 106 All 106 (JSON)