mlsanityData to 7 Oct 2026

OneFormer: One Transformer to Rule Universal Image Segmentation

arXiv 2211.06220 ↗First posted 10 Nov 2022cs.CVcited by 327

Not on the map. Why?

Abstract (arXiv)

Universal Image Segmentation is not a new concept. Past attempts to unify image segmentation in the last decades include scene parsing, panoptic segmentation, and, more recently, new panoptic architectures. However, such panoptic architectures do not truly unify image segmentation because they need to be trained individually on the semantic, instance, or panoptic segmentation to achieve the best performance. Ideally, a truly universal framework should be trained only once and achieve SOTA performance across all three image segmentation tasks. To that end, we propose OneFormer, a universal image segmentation framework that unifies segmentation with a multi-task train-once design. We first propose a task-conditioned joint training strategy that enables training on ground truths of each domain (semantic, instance, and panoptic segmentation) within a single multi-task training process. Secondly, we introduce a task token to condition our model on the task at hand, making our model task-dynamic to support multi-task training and inference. Thirdly, we propose using a query-text contrastive loss during training to establish better inter-task and inter-class distinctions. Notably, our single OneFormer model outperforms specialized Mask2Former models across all three segmentation tasks on ADE20k, CityScapes, and COCO, despite the latter being trained on each of the three tasks individually with three times the resources. With new ConvNeXt and DiNAT backbones, we observe even more performance improvement. We believe OneFormer is a significant step towards making image segmentation more universal and accessible. To support further research, we open-source our code and models at https://github.com/SHI-Labs/OneFormer

Key facts. Cited by 327 papers in the map (data through 7 Oct 2026). Builds on 53 of its 63 references that are in the map. First posted on arXiv 10 Nov 2022.

Builds on

53 of 63 references.

References · 53

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2020End-to-End Object Detection with Transformerscited by 7,384
  7. 2017Mask R-CNNcited by 7,259
  8. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  9. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  10. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  11. 2022A ConvNet for the 2020scited by 3,545
  12. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
38 more
  1. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
  2. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  3. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  4. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  5. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
  6. 2017Cascade R-CNN: Delving into High Quality Object Detectioncited by 1,275
  7. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,265
  8. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
  9. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,148
  10. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  11. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  12. 2018Panoptic Segmentationcited by 796
  13. 2018CCNet: Criss-Cross Attention for Semantic Segmentationcited by 717
  14. 2019Panoptic Feature Pyramid Networkscited by 644
  15. 2020Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentationcited by 595
  16. 2022DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETRcited by 569
  17. 2022Vision Transformer Adapter for Dense Predictionscited by 564
  18. 2014Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFscited by 552
  19. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
  20. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  21. 2022DN-DETR: Accelerate DETR Training by Introducing Query DeNoisingcited by 486
  22. 2019Hybrid Task Cascade for Instance Segmentationcited by 479
  23. 2019Panoptic-DeepLabcited by 362
  24. 2020MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformerscited by 355
  25. 2022Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentationcited by 341
  26. 2020Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentationcited by 326
  27. 2022Neighborhood Attention Transformercited by 235
  28. 2019High-Resolution Representations for Labeling Pixels and Regionscited by 211
  29. 2021K-Net: Towards Unified Image Segmentationcited by 211
  30. 2014Simultaneous Detection and Segmentationcited by 158
  31. 2020EfficientPS: Efficient Panoptic Segmentationcited by 111
  32. 2021Panoptic SegFormer: Delving Deeper into Panoptic Segmentation with Transformerscited by 109
  33. 2022Contrastive Learning Rivals Masked Image Modeling in Fine-tuning via Feature Distillationcited by 93
  34. 2021FaPN: Feature-aligned Pyramid Network for Dense Image Predictioncited by 81
  35. 2022Dilated Neighborhood Attention Transformercited by 64
  36. 2022CMT-DeepLab: Clustering Mask Transformers for Panoptic Segmentationcited by 60
  37. 2014Convolutional Feature Masking for Joint Object and Stuff Segmentationcited by 41
  38. 2021Polarized Self-Attention: Towards High-quality Pixel-wise Regressioncited by 39

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 74; 2024: 101; 2025: 81; 2026: 69.

327 papers.

Sort:

By area:

  1. 2023Segment Anythingcited by 8,158
  2. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  3. 2024Depth Anything V2cited by 1,385
  4. 2024Depth Anything: Unleashing the Power of Large-Scale Unlabeled Datacited by 1,346
  5. 2023Segment Everything Everywhere All at Oncecited by 546
  6. 2023Forget-Me-Not: Learning to Forget in Text-to-Image Diffusion Modelscited by 297
  7. 2022Versatile Diffusion: Text, Images and Variations All in One Diffusion Modelcited by 199
  8. 2023Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIPcited by 199
  9. 2023A Simple Framework for Open-Vocabulary Segmentation and Detectioncited by 197
  10. 2024WonderWorld: Interactive 3D Scene Generation from a Single Imagecited by 197
  11. 2023Semantic-SAM: Segment and Recognize Anything at Any Granularitycited by 185
  12. 2025GAIA-2: A Controllable Multi-View Generative World Model for Autonomous Drivingcited by 178
38 more
  1. 2022SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic Segmentationcited by 162
  2. 2023WonderJourney: Going from Anywhere to Everywherecited by 157
  3. 2023DDP: Diffusion Model for Dense Visual Predictioncited by 148
  4. 2024Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation with Multimodal Generative Pretrainingcited by 148
  5. 2024OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understandingcited by 126
  6. 2023Transformer-Based Visual Segmentation: A Surveycited by 124
  7. 2024PSALM: Pixelwise SegmentAtion with Large Multi-Modal Modelcited by 100
  8. 2023InternGPT: Solving Vision-Centric Tasks by Interacting with ChatGPT Beyond Languagecited by 96
  9. 2023OneFormer3D: One Transformer for Unified Point Cloud Segmentationcited by 95
  10. 2024MM-Interleaved: Interleaved Image-Text Generative Modeling via Multi-modal Feature Synchronizercited by 80
  11. 2023ProRes: Exploring Degradation-aware Visual Prompt for Universal Image Restorationcited by 77
  12. 2023Review of Large Vision Models and Visual Prompt Engineeringcited by 77
  13. 2024OMG-Seg: Is One Model Good Enough For All Segmentation?cited by 73
  14. 2024How NeRFs and 3D Gaussian Splatting are Reshaping SLAM: a Surveycited by 72
  15. 20253D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformercited by 67
  16. 2023AVSegFormer: Audio-Visual Segmentation with Transformercited by 66
  17. 2023A Comprehensive Survey on Segment Anything Model for Vision and Beyondcited by 64
  18. 2023Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Modelscited by 62
  19. 2025Your ViT is Secretly an Image Segmentation Modelcited by 61
  20. 2024Gen3DSR: Generalizable 3D Scene Reconstruction via Divide and Conquer from a Single Viewcited by 59
  21. 2024CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Expertscited by 56
  22. 2024LayerPano3D: Layered 3D Panorama for Hyper-Immersive Scene Generationcited by 53
  23. 2023Visual In-Context Promptingcited by 52
  24. 2024Frequency-Spatial Entanglement Learning for Camouflaged Object Detectioncited by 46
  25. 2023VCoder: Versatile Vision Encoders for Multimodal Large Language Modelscited by 42
  26. 2023Matte Anything: Interactive Natural Image Matting with Segment Anything Modelscited by 39
  27. 2025DiST-4D: Disentangled Spatiotemporal Diffusion with Metric Depth for 4D Driving Scene Generationcited by 39
  28. 2024VistaDream: Sampling multiview consistent images for single-view scene reconstructioncited by 38
  29. 2025DICEPTION: A Generalist Diffusion Model for Visual Perceptual Taskscited by 37
  30. 2024MoAI: Mixture of All Intelligence for Large Language and Vision Modelscited by 36
  31. 2024AllSpark: Reborn Labeled Features from Unlabeled in Transformer for Semi-Supervised Semantic Segmentationcited by 35
  32. 2024Image Segmentation in Foundation Model Era: A Surveycited by 35
  33. 2023Unsupervised Universal Image Segmentationcited by 34
  34. 2024CAFuser: Condition-Aware Multimodal Fusion for Robust Semantic Perception of Driving Scenescited by 33
  35. 2024HyperSeg: Towards Universal Visual Segmentation with Large Language Modelcited by 33
  36. 2024PixWizard: Versatile Image-to-Image Visual Assistant with Open-Language Instructionscited by 31
  37. 2023DaTaSeg: Taming a Universal Multi-Dataset Multi-Task Segmentation Modelcited by 30
  38. 2023Matting Anythingcited by 30

50 most cited of 327 All 327 (JSON)