mlsanityData to 7 Oct 2026

A Simple and Generalist Approach for Panoptic Segmentation

arXiv 2408.16504 ↗First posted 29 Aug 2024cs.CVcited by 3

Not on the map. Why?

Abstract (arXiv)

Panoptic segmentation is an important computer vision task, where the current state-of-the-art solutions require specialized components to perform well. We propose a simple generalist framework based on a deep encoder - shallow decoder architecture with per-pixel prediction. Essentially fine-tuning a massively pretrained image model with minimal additional components. Naively this method does not yield good results. We show that this is due to imbalance during training and propose a novel method for reducing it - centroid regression in the space of spectral positional embeddings. Our method achieves panoptic quality (PQ) of 55.1 on the challenging MS-COCO dataset, state-of-the-art performance among generalist methods.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 54 of its 73 references that are in the map. First posted on arXiv 29 Aug 2024.

Builds on

54 of 73 references.

References · 54

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  6. 2023GPT-4 Technical Reportcited by 18,132
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  8. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  9. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  10. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  11. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  12. 2020End-to-End Object Detection with Transformerscited by 7,384
38 more
  1. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  2. 2017Mask R-CNNcited by 7,259
  3. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  4. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  5. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  6. 2017Neural Discrete Representation Learningcited by 4,650
  7. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  8. 2022A ConvNet for the 2020scited by 3,545
  9. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  10. 2024Depth Anything V2cited by 1,385
  11. 2024Depth Anything: Unleashing the Power of Large-Scale Unlabeled Datacited by 1,346
  12. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,345
  13. 2017Cascade R-CNN: Delving into High Quality Object Detectioncited by 1,275
  14. 2014Depth Map Prediction from a Single Image using a Multi-Scale Deep Networkcited by 1,264
  15. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
  16. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,148
  17. 2018Panoptic Segmentationcited by 796
  18. 2020Image Segmentation Using Deep Learning: A Surveycited by 659
  19. 2017Generalised Dice overlap as a deep learning loss function for highly unbalanced segmentationscited by 614
  20. 2021Neural Fields in Visual Computing and Beyondcited by 539
  21. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  22. 2019Panoptic-DeepLabcited by 362
  23. 2022Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentationcited by 341
  24. 2022OneFormer: One Transformer to Rule Universal Image Segmentationcited by 327
  25. 2022Images Speak in Images: A Generalist Painter for In-Context Visual Learningcited by 317
  26. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  27. 2019SOLO: Segmenting Objects by Locationscited by 289
  28. 2023MiDaS v3.1 -- A Model Zoo for Robust Monocular Relative Depth Estimationcited by 173
  29. 2018Improving Semantic Segmentation via Video Propagation and Label Relaxationcited by 134
  30. 2020EfficientPS: Efficient Panoptic Segmentationcited by 111
  31. 2022A Generalist Framework for Panoptic Segmentation of Images and Videoscited by 99
  32. 2020A Survey on Instance Segmentation: State of the artcited by 85
  33. 2022UViM: A Unified Modeling Approach for Vision with Learned Guiding Codescited by 77
  34. 2019Explicit Shape Encoding for Real-Time Instance Segmentationcited by 41
  35. 2023Semantic Image Segmentation: Two Decades of Researchcited by 26
  36. 2024A Simple Latent Diffusion Approach for Panoptic Segmentation and Mask Inpaintingcited by 18
  37. 2022kMaX-DeepLab: k-means Mask Transformercited by 12
  38. 2019Learning Fully Dense Neural Networks for Image Semantic Segmentationcited by 9

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025Advancing Semantic Future Prediction through Multimodal Visual Sequence Transformerscited by 7
  2. 2026Accelerated Decoding of Centroid Positional Encoding for Instance Segmentationcited by 0
  3. 2026AHMAD: Adaptive Hybrid Multi-task Vision Learning with Assisted Distillation for Keypoint Detectioncited by 0