# OMG-Seg: Is One Model Good Enough For All Segmentation?

arXiv: https://arxiv.org/abs/2401.10229 · HTML: https://mlsanity.com/paper/2401.10229/

## Key facts

- Cited by 75 papers in the map (data through 14 Oct 2026).
- Builds on 96 of its 112 references that are in the map.
- First posted on arXiv 18 Jan 2024.

## Builds on (most cited first, 30 of 96)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [MMDetection: Open MMLab Detection Toolbox and Benchmark](https://mlsanity.com/paper/1906.07155/) (2019)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [Kosmos-2: Grounding Multimodal Large Language Models to the World](https://mlsanity.com/paper/2306.14824/) (2023)
- [Semantic Understanding of Scenes through the ADE20K Dataset](https://mlsanity.com/paper/1608.05442/) (2016)
- [Open-vocabulary Object Detection via Vision and Language Knowledge Distillation](https://mlsanity.com/paper/2104.13921/) (2021)
- [Generation and Comprehension of Unambiguous Object Descriptions](https://mlsanity.com/paper/1511.02283/) (2015)
- [Panoptic Segmentation](https://mlsanity.com/paper/1801.00868/) (2018)
- [Language-driven Semantic Segmentation](https://mlsanity.com/paper/2201.03546/) (2022)
- [EVA: Exploring the Limits of Masked Visual Representation Learning at Scale](https://mlsanity.com/paper/2211.07636/) (2022)
- [Image Segmentation Using Deep Learning: A Survey](https://mlsanity.com/paper/2001.05566/) (2020)
- [Panoptic Feature Pyramid Networks](https://mlsanity.com/paper/1901.02446/) (2019)
- [Detecting Twenty-thousand Classes using Image-level Supervision](https://mlsanity.com/paper/2201.02605/) (2022)
- [Vision Transformer Adapter for Dense Predictions](https://mlsanity.com/paper/2205.08534/) (2022)

## Built on by (most cited first, 30 of 75)

- [Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videos](https://mlsanity.com/paper/2501.04001/) (2025)
- [OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understanding](https://mlsanity.com/paper/2406.19389/) (2024)
- [PSALM: Pixelwise SegmentAtion with Large Multi-Modal Model](https://mlsanity.com/paper/2403.14598/) (2024)
- [SAM2-UNet: Segment Anything 2 Makes Strong Encoder for Natural and Medical Image Segmentation](https://mlsanity.com/paper/2408.08870/) (2024)
- [Your ViT is Secretly an Image Segmentation Model](https://mlsanity.com/paper/2503.19108/) (2025)
- [MOSEv2: A More Challenging Dataset for Video Object Segmentation in Complex Scenes](https://mlsanity.com/paper/2508.05630/) (2025)
- [Vitron: A Unified Pixel-level Vision LLM for Understanding, Generating, Segmenting, Editing](https://mlsanity.com/paper/2412.19806/) (2024)
- [Towards Semantic Equivalence of Tokenization in Multimodal LLM](https://mlsanity.com/paper/2406.05127/) (2024)
- [Graphic Design with Large Multimodal Model](https://mlsanity.com/paper/2404.14368/) (2024)
- [Image Segmentation in Foundation Model Era: A Survey](https://mlsanity.com/paper/2408.12957/) (2024)
- [FaceXFormer: A Unified Transformer for Facial Analysis](https://mlsanity.com/paper/2403.12960/) (2024)
- [Self-Calibrated CLIP for Training-Free Open-Vocabulary Segmentation](https://mlsanity.com/paper/2411.15869/) (2024)
- [HyperSeg: Towards Universal Visual Segmentation with Large Language Model](https://mlsanity.com/paper/2411.17606/) (2024)
- [X-SAM: From Segment Anything to Any Segmentation](https://mlsanity.com/paper/2508.04655/) (2025)
- [ObjectRelator: Enabling Cross-View Object Relation Understanding Across Ego-Centric and Exo-Centric Perspectives](https://mlsanity.com/paper/2411.19083/) (2024)
- [From Perception to Cognition: A Survey of Vision-Language Interactive Reasoning in Multimodal Large Language Models](https://mlsanity.com/paper/2509.25373/) (2025)
- [Stepping Stones: A Progressive Training Strategy for Audio-Visual Semantic Segmentation](https://mlsanity.com/paper/2407.11820/) (2024)
- [CapHuman: Capture Your Moments in Parallel Universes](https://mlsanity.com/paper/2402.00627/) (2024)
- [MG-LLaVA: Towards Multi-Granularity Visual Instruction Tuning](https://mlsanity.com/paper/2406.17770/) (2024)
- [Pixel-SAIL: Single Transformer For Pixel-Grounded Understanding](https://mlsanity.com/paper/2504.10465/) (2025)
- [Multimodal Referring Segmentation: A Survey](https://mlsanity.com/paper/2508.00265/) (2025)
- [DGMamba: Domain Generalization via Generalized State Space Model](https://mlsanity.com/paper/2404.07794/) (2024)
- [SemFlow: Binding Semantic Segmentation and Image Synthesis via Rectified Flow](https://mlsanity.com/paper/2405.20282/) (2024)
- [LoRACLR: Contrastive Adaptation for Customization of Diffusion Models](https://mlsanity.com/paper/2412.09622/) (2024)
- [HiMTok: Learning Hierarchical Mask Tokens for Image Segmentation with Large Multimodal Model](https://mlsanity.com/paper/2503.13026/) (2025)
- [PanSt3R: Multi-view Consistent Panoptic Segmentation](https://mlsanity.com/paper/2506.21348/) (2025)
- [Are They the Same? Exploring Visual Correspondence Shortcomings of Multimodal LLMs](https://mlsanity.com/paper/2501.04670/) (2025)
- [DenseWorld-1M: Towards Detailed Dense Grounded Caption in the Real World](https://mlsanity.com/paper/2506.24102/) (2025)
- [A Unified Framework for 3D Scene Understanding](https://mlsanity.com/paper/2407.03263/) (2024)
- [Explore In-Context Segmentation via Latent Diffusion Models](https://mlsanity.com/paper/2403.09616/) (2024)

## Abstract (arXiv)

In this work, we address various segmentation tasks, each traditionally tackled by distinct or partially unified models. We propose OMG-Seg, One Model that is Good enough to efficiently and effectively handle all the segmentation tasks, including image semantic, instance, and panoptic segmentation, as well as their video counterparts, open vocabulary settings, prompt-driven, interactive segmentation like SAM, and video object segmentation. To our knowledge, this is the first model to handle all these tasks in one model and achieve satisfactory performance. We show that OMG-Seg, a transformer-based encoder-decoder architecture with task-specific queries and outputs, can support over ten distinct segmentation tasks and yet significantly reduce computational and parameter overhead across various tasks and datasets. We rigorously evaluate the inter-task influences and correlations during co-training. Code and models are available at https://github.com/lxtGH/OMG-Seg.