mlsanityData to 7 Oct 2026

MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers

arXiv 2012.00759 ↗First posted 1 Dec 2020cs.CVcited by 355

Not on the map. Why?

Abstract (arXiv)

We present MaX-DeepLab, the first end-to-end model for panoptic segmentation. Our approach simplifies the current pipeline that depends heavily on surrogate sub-tasks and hand-designed components, such as box detection, non-maximum suppression, thing-stuff merging, etc. Although these sub-tasks are tackled by area experts, they fail to comprehensively solve the target task. By contrast, our MaX-DeepLab directly predicts class-labeled masks with a mask transformer, and is trained with a panoptic quality inspired loss via bipartite matching. Our mask transformer employs a dual-path architecture that introduces a global memory path in addition to a CNN path, allowing direct communication with any CNN layers. As a result, MaX-DeepLab shows a significant 7.1% PQ gain in the box-free regime on the challenging COCO dataset, closing the gap between box-based and box-free methods for the first time. A small variant of MaX-DeepLab improves 3.0% PQ over DETR with similar parameters and M-Adds. Furthermore, MaX-DeepLab, without test time augmentation, achieves new state-of-the-art 51.3% PQ on COCO test-dev set. Code is available at https://github.com/google-research/deeplab2.

Key facts. Cited by 355 papers in the map (data through 7 Oct 2026). First posted on arXiv 1 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 72; 2022: 88; 2023: 107; 2024: 54; 2025: 23; 2026: 11.

355 papers.

Sort:

By area:

  1. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  2. 2022Visual Prompt Tuningcited by 1,674
  3. 2021ViViT: A Video Vision Transformercited by 1,610
  4. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,148
  5. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  6. 2021CvT: Introducing Convolutions to Vision Transformerscited by 946
  7. 2021Multiscale Vision Transformerscited by 896
  8. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  9. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 647
  10. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  11. 2023Segment Everything Everywhere All at Oncecited by 546
  12. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 471
38 more
  1. 2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Modelscited by 371
  2. 2021Learning Spatio-Temporal Transformer for Visual Trackingcited by 348
  3. 2022Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentationcited by 341
  4. 2022OneFormer: One Transformer to Rule Universal Image Segmentationcited by 327
  5. 2021Medical Transformer: Gated Axial-Attention for Medical Image Segmentationcited by 319
  6. 2021Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformerscited by 316
  7. 2024An Image is Worth 32 Tokens for Reconstruction and Generationcited by 295
  8. 2022Generalized Decoding for Pixel, Image, and Languagecited by 292
  9. 2021Focal Self-attention for Local-Global Interactions in Vision Transformerscited by 286
  10. 2021Voxel Transformer for 3D Object Detectioncited by 247
  11. 2022CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformerscited by 229
  12. 2021K-Net: Towards Unified Image Segmentationcited by 211
  13. 2021Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encodingcited by 207
  14. 2023Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIPcited by 199
  15. 2022Focal Modulation Networkscited by 186
  16. 2021Chasing Sparsity in Vision Transformers: An End-to-End Explorationcited by 181
  17. 2021Visual Saliency Transformercited by 175
  18. 2021PTQ4ViT: Post-training quantization for vision transformers with twin uniform quantizationcited by 173
  19. 2023SCLIP: Rethinking Self-Attention for Dense Vision-Language Inferencecited by 169
  20. 2021ASFormer: Transformer for Action Segmentationcited by 166
  21. 2023Towards Open Vocabulary Learning: A Surveycited by 165
  22. 2021Instances as Queriescited by 157
  23. 2022Masked Discrimination for Self-Supervised Learning on Point Cloudscited by 155
  24. 2021Efficient Self-supervised Vision Transformers for Representation Learningcited by 149
  25. 2022DETRs with Hybrid Matchingcited by 149
  26. 2022More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsitycited by 142
  27. 2022MonoViT: Self-Supervised Monocular Depth Estimation with a Vision Transformercited by 136
  28. 2021End-to-End Referring Video Object Segmentation with Multimodal Transformerscited by 135
  29. 2021TransFG: A Transformer Architecture for Fine-grained Recognitioncited by 131
  30. 2023Diffuse, Attend, and Segment: Unsupervised Zero-Shot Segmentation using Stable Diffusioncited by 131
  31. 2024OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understandingcited by 126
  32. 2023Transformer-Based Visual Segmentation: A Surveycited by 124
  33. 2021MPViT: Multi-Path Vision Transformer for Dense Predictioncited by 122
  34. 2021A Survey of Visual Transformerscited by 117
  35. 2022Revisiting Weakly Supervised Pre-Training of Visual Perception Modelscited by 116
  36. 2022ReSTR: Convolution-free Referring Image Segmentation Using Transformerscited by 112
  37. 2021Panoptic SegFormer: Delving Deeper into Panoptic Segmentation with Transformerscited by 109
  38. 2021Video Instance Segmentation using Inter-Frame Communication Transformerscited by 105

50 most cited of 355 All 355 (JSON)