mlsanityData to 14 Oct 2026

Vision Transformers Are Good Mask Auto-Labelers

arXiv 2301.03992 ↗First posted 10 Jan 2023cs.CV cs.LG cs.MMcited by 14

Not on the map. Why?

Abstract (arXiv)

We propose Mask Auto-Labeler (MAL), a high-quality Transformer-based mask auto-labeling framework for instance segmentation using only box annotations. MAL takes box-cropped images as inputs and conditionally generates their mask pseudo-labels.We show that Vision Transformers are good mask auto-labelers. Our method significantly reduces the gap between auto-labeling and human annotation regarding mask quality. Instance segmentation models trained using the MAL-generated masks can nearly match the performance of their fully-supervised counterparts, retaining up to 97.4% performance of fully supervised models. The best model achieves 44.1% mAP on COCO instance segmentation (test-dev 2017), outperforming state-of-the-art box-supervised methods by significant margins. Qualitative results indicate that masks produced by MAL are, in some cases, even better than human annotations.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 54 of its 63 references that are in the map. First posted on arXiv 10 Jan 2023.

Builds on

54 of 63 references.

References · 54

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  6. 2020End-to-End Object Detection with Transformerscited by 7,414
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  8. 2017Mask R-CNNcited by 7,277
  9. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  10. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  11. 2016Feature Pyramid Networks for Object Detectioncited by 5,036
  12. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,859
38 more
  1. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  2. 2022A ConvNet for the 2020scited by 3,579
  3. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
  4. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,546
  5. 2015Learning Deep Features for Discriminative Localizationcited by 2,436
  6. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  7. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
  8. 2019MMDetection: Open MMLab Detection Toolbox and Benchmarkcited by 1,385
  9. 2017Cascade R-CNN: Delving into High Quality Object Detectioncited by 1,276
  10. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,228
  11. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,217
  12. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,152
  13. 2021Going deeper with Image Transformerscited by 764
  14. 2021PVT v2: Improved Baselines with Pyramid Vision Transformercited by 751
  15. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 724
  16. 2017Generalised Dice overlap as a deep learning loss function for highly unbalanced segmentationscited by 620
  17. 2018The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scalecited by 544
  18. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 511
  19. 2019Hybrid Task Cascade for Instance Segmentationcited by 481
  20. 2019YOLACT: Real-time Instance Segmentationcited by 443
  21. 2022DeiT III: Revenge of the ViTcited by 419
  22. 2020MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformerscited by 355
  23. 2022Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentationcited by 344
  24. 2018Learning Pixel-level Semantic Affinity with Image-level Supervision for Weakly Supervised Semantic Segmentationcited by 321
  25. 2020Conditional Convolutions for Instance Segmentationcited by 314
  26. 2021DeepViT: Towards Deeper Vision Transformercited by 289
  27. 2021AdaViT: Adaptive Vision Transformers for Efficient Image Recognitioncited by 228
  28. 2019PolarMask: Single Shot Instance Segmentation with Polar Representationcited by 188
  29. 2021Mask2Former for Video Instance Segmentationcited by 171
  30. 2020BoxInst: High-Performance Instance Segmentation with Box Annotationscited by 151
  31. 2020Mining Cross-Image Semantics for Weakly Supervised Semantic Segmentationcited by 144
  32. 2022Understanding The Robustness in Vision Transformerscited by 144
  33. 2016Fully Convolutional Instance-aware Semantic Segmentationcited by 123
  34. 2015Instance-aware Semantic Segmentation via Multi-task Network Cascadescited by 121
  35. 2021Panoptic SegFormer: Delving Deeper into Panoptic Segmentation with Transformerscited by 109
  36. 2021DiscoBox: Weakly Supervised Instance Segmentation and Semantic Correspondence from Box Supervisioncited by 59
  37. 2022Box-supervised Instance Segmentation with Level Set Evolutioncited by 41
  38. 2021SOLO: A Simple Framework for Instance Segmentationcited by 32

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 7; 2024: 6; 2025: 1.

14 papers.

Sort:

By area:

  1. 2023A Simple Framework for Open-Vocabulary Segmentation and Detectioncited by 198
  2. 2023Transformer-Based Visual Segmentation: A Surveycited by 124
  3. 2023BEVNeXt: Reviving Dense BEV Frameworks for 3D Object Detectioncited by 35
  4. 2023DaTaSeg: Taming a Universal Multi-Dataset Multi-Task Segmentation Modelcited by 31
  5. 2023Label-efficient Segmentation via Affinity Propagationcited by 7
  6. 2024SPAMming Labels: Efficient Annotations for the Trackers of Tomorrowcited by 7
  7. 2024UVCPNet: A UAV-Vehicle Collaborative Perception Network for 3D Object Detectioncited by 7
  8. 2023GaPro: Box-Supervised 3D Point Cloud Instance Segmentation Using Gaussian Processes as Pseudo Labelerscited by 6
  9. 2024MonoBox: Tightness-free Box-supervised Polyp Segmentation using Monotonicity Constraintcited by 5
  10. 2024Detecting Wildfires on UAVs with Real-time Segmentation Trained by Larger Teacher Modelscited by 5
  11. 2024Extreme Point Supervised Instance Segmentationcited by 3
  12. 2025Advancing Visual Large Language Model for Multi-granular Versatile Perceptioncited by 3
2 more
  1. 2024EchoDFKD: Data-Free Knowledge Distillation for Cardiac Ultrasound Segmentation using Synthetic Datacited by 1
  2. 2023BoxVIS: Video Instance Segmentation with Box Annotationscited by 0