mlsanityData to 7 Oct 2026

CMT-DeepLab: Clustering Mask Transformers for Panoptic Segmentation

arXiv 2206.08948 ↗First posted 17 Jun 2022cs.CVcited by 60

Not on the map. Why?

Abstract (arXiv)

We propose Clustering Mask Transformer (CMT-DeepLab), a transformer-based framework for panoptic segmentation designed around clustering. It rethinks the existing transformer architectures used in segmentation and detection; CMT-DeepLab considers the object queries as cluster centers, which fill the role of grouping the pixels when applied to segmentation. The clustering is computed with an alternating procedure, by first assigning pixels to the clusters by their feature affinity, and then updating the cluster centers and pixel features. Together, these operations comprise the Clustering Mask Transformer (CMT) layer, which produces cross-attention that is denser and more consistent with the final segmentation task. CMT-DeepLab improves the performance over prior art significantly by 4.4% PQ, achieving a new state-of-the-art of 55.7% PQ on the COCO test-dev set.

Key facts. Cited by 60 papers in the map (data through 7 Oct 2026). Builds on 86 of its 102 references that are in the map. First posted on arXiv 17 Jun 2022.

Builds on

86 of 102 references.

References · 86

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  8. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  9. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  10. 2020End-to-End Object Detection with Transformerscited by 7,384
  11. 2017Mask R-CNNcited by 7,259
  12. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
38 more
  1. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  2. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  3. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  4. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  5. 2020Longformer: The Long-Document Transformercited by 3,362
  6. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
  7. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,210
  8. 2020Supervised Contrastive Learningcited by 2,803
  9. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  10. 2015Learning Deep Features for Discriminative Localizationcited by 2,430
  11. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,320
  12. 2017Non-local Neural Networkscited by 2,290
  13. 2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,154
  14. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  15. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  16. 2020Reformer: The Efficient Transformercited by 1,658
  17. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  18. 2017Deformable Convolutional Networkscited by 1,481
  19. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  20. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,345
  21. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,148
  22. 2018Self-Attention with Relative Position Representationscited by 1,112
  23. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,024
  24. 2018Dual Attention Network for Scene Segmentationcited by 1,001
  25. 2016Deep Networks with Stochastic Depthcited by 973
  26. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  27. 2018Image Transformercited by 770
  28. 2018CCNet: Criss-Cross Attention for Semantic Segmentationcited by 717
  29. 2019Panoptic Feature Pyramid Networkscited by 644
  30. 2014Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFscited by 552
  31. 2019Stand-Alone Self-Attention in Vision Modelscited by 542
  32. 2018Efficient Attention: Attention with Linear Complexitiescited by 366
  33. 2019Attention Augmented Convolutional Networkscited by 363
  34. 2019Panoptic-DeepLabcited by 362
  35. 2020MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformerscited by 355
  36. 2019Axial Attention in Multidimensional Transformerscited by 350
  37. 2020Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentationcited by 326
  38. 2016Dynamic Filter Networkscited by 318

50 most cited of 86 All 86 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 7; 2023: 33; 2024: 13; 2025: 2; 2026: 5.

60 papers.

Sort:

By area:

  1. 2022OneFormer: One Transformer to Rule Universal Image Segmentationcited by 327
  2. 2024An Image is Worth 32 Tokens for Reconstruction and Generationcited by 295
  3. 2023Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIPcited by 199
  4. 2022DETRs with Hybrid Matchingcited by 149
  5. 2024OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understandingcited by 126
  6. 2023Transformer-Based Visual Segmentation: A Surveycited by 124
  7. 2022A Generalist Framework for Panoptic Segmentation of Images and Videoscited by 99
  8. 2023Contrastive Lift: 3D Object Instance Segmentation by Slow-Fast Contrastive Fusioncited by 62
  9. 2023Rank-DETR for High Quality Object Detectioncited by 49
  10. 2022MOAT: Alternating Mobile Convolution and Attention Brings Strong Vision Modelscited by 47
  11. 2023Image as Set of Pointscited by 43
  12. 2024ViTamin: Designing Scalable Vision Models in the Vision-Language Eracited by 39
38 more
  1. 2023FastInst: A Simple Query-Based Model for Real-Time Instance Segmentationcited by 32
  2. 2023CLUSTSEG: Clustering for Universal Segmentationcited by 32
  3. 2023HGFormer: Hierarchical Grouping Transformer for Domain Generalized Semantic Segmentationcited by 31
  4. 2023PhenoBench -- A Large Dataset and Benchmarks for Semantic Image Interpretation in the Agricultural Domaincited by 31
  5. 2023Learning Content-enhanced Mask Transformer for Domain Generalized Urban-Scene Segmentationcited by 29
  6. 2023VST++: Efficient and Stronger Visual Saliency Transformercited by 28
  7. 2023Video-kMaX: A Simple Unified Approach for Online and Near-Online Video Panoptic Segmentationcited by 24
  8. 2024COCONut: Modernizing COCO Segmentationcited by 24
  9. 2023Point2Mask: Point-supervised Panoptic Segmentation via Optimal Transportcited by 23
  10. 20233D TransUNet: Advancing Medical Image Segmentation through Vision Transformerscited by 20
  11. 2023DFormer: Diffusion-guided Transformer for Universal Image Segmentationcited by 19
  12. 2023ReMaX: Relaxing for Better Training on Efficient Panoptic Segmentationcited by 18
  13. 2023Devil is in the Queries: Advancing Mask Transformers for Real-world Medical Image Segmentation and Out-of-Distribution Localizationcited by 16
  14. 2022Panoramic Panoptic Segmentation: Insights Into Surrounding Parsing for Mobile Agents via Unsupervised Contrastive Learningcited by 14
  15. 2023PolyMaX: General Dense Prediction with Mask Transformercited by 14
  16. 2024SPFormer: Enhancing Vision Transformer with Superpixel Representationcited by 13
  17. 2022kMaX-DeepLab: k-means Mask Transformercited by 12
  18. 2023Jaccard Metric Losses: Optimizing the Jaccard Index with Soft Labelscited by 12
  19. 2023Superpixel Transformers for Efficient Semantic Segmentationcited by 12
  20. 2024Open-Vocabulary Segmentation with Unpaired Mask-Text Supervisioncited by 12
  21. 2023Compositor: Bottom-up Clustering and Compositing for Robust Part and Object Segmentationcited by 11
  22. 2024Clustering Propagation for Universal Medical Image Segmentationcited by 11
  23. 2023Towards Open-Ended Visual Recognition with Large Language Modelcited by 8
  24. 2023ClusterFormer: Clustering As A Universal Visual Learnercited by 6
  25. 2023Revisiting the Encoding of Satellite Image Time Seriescited by 5
  26. 2024ProMotion: Prototypes As Motion Learnerscited by 5
  27. 2023MaskConver: Revisiting Pure Convolution Model for Panoptic Segmentationcited by 4
  28. 2024CompetitorFormer: Competitor Transformer for 3D Instance Segmentationcited by 4
  29. 2026Beyond Accuracy: Benchmarking Cross-Task Consistency in Unified Multimodal Modelscited by 4
  30. 2023A Simple Video Segmenter by Tracking Objects Along Axial Trajectoriescited by 3
  31. 2022A Review of Modern Approaches for Coronary Angiography Imaging Analysiscited by 2
  32. 2023Cluster-Induced Mask Transformers for Effective Opportunistic Gastric Cancer Screening on Non-contrast CT Scanscited by 2
  33. 2026Mitigating Objectness Bias and Region-to-Text Misalignment for Open-Vocabulary Panoptic Segmentationcited by 2
  34. 2023Uncertainty estimation in Deep Learning for Panoptic segmentationcited by 1
  35. 2023SE-shapelets: Semi-supervised Clustering of Time Series Using Representative Shapeletscited by 1
  36. 2023RT-K-Net: Revisiting K-Net for Real-Time Panoptic Segmentationcited by 1
  37. 2024Semantic Line Combination Detectorcited by 1
  38. 2026SPEGC: Continual Test-Time Adaptation via Semantic-Prompt-Enhanced Graph Clustering for Medical Image Segmentationcited by 1

50 most cited of 60 All 60 (JSON)