mlsanityData to 7 Oct 2026

Per-Pixel Classification is Not All You Need for Semantic Segmentation

arXiv 2107.06278 ↗First posted 13 Jul 2021cs.CVcited by 1,148

Not on the map. Why?

Cited by 48 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Modern approaches typically formulate semantic segmentation as a per-pixel classification task, while instance-level segmentation is handled with an alternative mask classification. Our key insight: mask classification is sufficiently general to solve both semantic- and instance-level segmentation tasks in a unified manner using the exact same model, loss, and training procedure. Following this observation, we propose MaskFormer, a simple mask classification model which predicts a set of binary masks, each associated with a single global class label prediction. Overall, the proposed mask classification-based method simplifies the landscape of effective approaches to semantic and panoptic segmentation tasks and shows excellent empirical results. In particular, we observe that MaskFormer outperforms per-pixel classification baselines when the number of classes is large. Our mask classification-based method outperforms both current state-of-the-art semantic (55.6 mIoU on ADE20K) and panoptic segmentation (52.7 PQ on COCO) models.

Key facts. Cited by 1,148 papers in the map (data through 7 Oct 2026). Builds on 38 of its 56 references that are in the map. First posted on arXiv 13 Jul 2021.

Builds on

38 of 56 references.

References · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  7. 2020End-to-End Object Detection with Transformerscited by 7,384
  8. 2017Mask R-CNNcited by 7,259
  9. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  10. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  11. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  12. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
26 more
  1. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,210
  2. 2016Pyramid Scene Parsing Networkcited by 2,472
  3. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,320
  4. 2017Non-local Neural Networkscited by 2,290
  5. 2016V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentationcited by 2,284
  6. 2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,154
  7. 2018Group Normalizationcited by 1,514
  8. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
  9. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,265
  10. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,132
  11. 2018Dual Attention Network for Scene Segmentationcited by 1,001
  12. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
  13. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  14. 2018Panoptic Segmentationcited by 796
  15. 2017Focal Loss for Dense Object Detectioncited by 749
  16. 2018CCNet: Criss-Cross Attention for Semantic Segmentationcited by 717
  17. 2019Segmentation Transformer: Object-Contextual Representations for Semantic Segmentationcited by 573
  18. 2019Panoptic-DeepLabcited by 362
  19. 2020MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformerscited by 355
  20. 2020Conditional Convolutions for Instance Segmentationcited by 314
  21. 2020Hierarchical Multi-Scale Attention for Semantic Segmentationcited by 183
  22. 2014Simultaneous Detection and Segmentationcited by 158
  23. 2014Convolutional Feature Masking for Joint Object and Stuff Segmentationcited by 41
  24. 2020Bidirectional Graph Reasoning Network for Panoptic Segmentationcited by 26
  25. 2019SPGNet: Semantic Prediction Guidance for Scene Parsingcited by 21
  26. 2020Auto-Panoptic: Cooperative Multi-Component Architecture Search for Panoptic Segmentationcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 33; 2022: 190; 2023: 342; 2024: 283; 2025: 187; 2026: 113.

1,148 papers.

Sort:

By area:

  1. 2023Segment Anythingcited by 8,158
  2. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  3. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
  4. 2022Planning-oriented Autonomous Drivingcited by 1,068
  5. 2024Grounded SAM: Assembling Open-World Models for Diverse Visual Taskscited by 914
  6. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 770
  7. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  8. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 560
  9. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  10. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 471
  11. 2021Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysiscited by 418
  12. 2022Point Transformer V2: Grouped Vector Attention and Partition-based Poolingcited by 384
38 more
  1. 2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Modelscited by 371
  2. 2022Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentationcited by 341
  3. 2022OneFormer: One Transformer to Rule Universal Image Segmentationcited by 327
  4. 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 310
  5. 2022ActionFormer: Localizing Moments of Actions with Transformerscited by 308
  6. 2023VoxFormer: Sparse Voxel Transformer for Camera-based 3D Semantic Scene Completioncited by 306
  7. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 302
  8. 2022Mask3D: Mask Transformer for 3D Semantic Instance Segmentationcited by 299
  9. 2023Side Adapter Network for Open-Vocabulary Semantic Segmentationcited by 296
  10. 2023OccFormer: Dual-path Transformer for Vision-based 3D Semantic Occupancy Predictioncited by 287
  11. 2023Fast Segment Anythingcited by 286
  12. 2023Unleashing Text-to-Image Diffusion Models for Visual Perceptioncited by 266
  13. 2022SegNeXt: Rethinking Convolutional Attention Design for Semantic Segmentationcited by 252
  14. 2025Seg-Zero: Reasoning-Chain Guided Segmentation via Cognitive Reinforcementcited by 234
  15. 2023Break-A-Scene: Extracting Multiple Concepts from a Single Imagecited by 230
  16. 2022CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformerscited by 229
  17. 2021A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Modelcited by 223
  18. 2023CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentationcited by 215
  19. 2022UniFormer: Unifying Convolution and Self-attention for Visual Recognitioncited by 206
  20. 2023Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIPcited by 199
  21. 2021Decoupling Zero-Shot Semantic Segmentationcited by 198
  22. 2022Focal Modulation Networkscited by 186
  23. 2021Adaptive Token Sampling For Efficient Vision Transformerscited by 181
  24. 2022Language as Queries for Referring Video Object Segmentationcited by 180
  25. 2023RSPrompter: Learning to Prompt for Remote Sensing Instance Segmentation based on Visual Foundation Modelcited by 180
  26. 2022ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentationcited by 178
  27. 2022Panoptic Lifting for 3D Scene Understanding with Neural Fieldscited by 171
  28. 2023FLatten Transformer: Vision Transformer using Focused Linear Attentioncited by 170
  29. 2021Mask2Former for Video Instance Segmentationcited by 169
  30. 2023SCLIP: Rethinking Self-Attention for Dense Vision-Language Inferencecited by 169
  31. 2021Multimodal Virtual Point 3D Detectioncited by 166
  32. 2022BinsFormer: Revisiting Adaptive Bins for Monocular Depth Estimationcited by 166
  33. 2023Query-Dependent Video Representation for Moment Retrieval and Highlight Detectioncited by 166
  34. 2021SPViT: Enabling Faster Vision Transformers via Soft Token Pruningcited by 165
  35. 2023Towards Open Vocabulary Learning: A Surveycited by 165
  36. 2022SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic Segmentationcited by 162
  37. 2024ControlNet++: Improving Conditional Controls with Efficient Consistency Feedbackcited by 153
  38. 2022A Unified Sequence Interface for Vision Taskscited by 152

50 most cited of 1,148 All 1,148 (JSON)