mlsanityData to 7 Oct 2026

SegGPT: Segmenting Everything In Context

arXiv 2304.03284 ↗First posted 6 Apr 2023cs.CVcited by 193

Not on the map. Why?

Abstract (arXiv)

We present SegGPT, a generalist model for segmenting everything in context. We unify various segmentation tasks into a generalist in-context learning framework that accommodates different kinds of segmentation data by transforming them into the same format of images. The training of SegGPT is formulated as an in-context coloring problem with random color mapping for each data sample. The objective is to accomplish diverse tasks according to the context, rather than relying on specific colors. After training, SegGPT can perform arbitrary segmentation tasks in images or videos via in-context inference, such as object instance, stuff, part, contour, and text. SegGPT is evaluated on a broad range of tasks, including few-shot semantic segmentation, video object segmentation, semantic segmentation, and panoptic segmentation. Our results show strong capabilities in segmenting in-domain and out-of-domain targets, either qualitatively or quantitatively.

Key facts. Cited by 193 papers in the map (data through 7 Oct 2026). Builds on 47 of its 57 references that are in the map. First posted on arXiv 6 Apr 2023.

Builds on

47 of 57 references.

References · 47

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2020End-to-End Object Detection with Transformerscited by 7,384
  7. 2017Mask R-CNNcited by 7,259
  8. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  9. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  10. 2015Fast R-CNNcited by 2,930
  11. 2016Pyramid Scene Parsing Networkcited by 2,472
  12. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
35 more
  1. 2021Vision Transformers for Dense Predictioncited by 1,789
  2. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  3. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 942
  4. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  5. 2019Panoptic Feature Pyramid Networkscited by 644
  6. 2016RefineNet: Multi-Path Refinement Networks for High-Resolution Semantic Segmentationcited by 463
  7. 2019YOLACT: Real-time Instance Segmentationcited by 441
  8. 2019Video Object Segmentation using Space-Time Memory Networkscited by 436
  9. 2022XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Modelcited by 436
  10. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  11. 2018YouTube-VOS: A Large-Scale Video Object Segmentation Benchmarkcited by 381
  12. 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 345
  13. 2022Images Speak in Images: A Generalist Painter for In-Context Visual Learningcited by 317
  14. 2021LoveDA: A Remote Sensing Land-Cover Dataset for Domain Adaptive Semantic Segmentationcited by 259
  15. 2022Visual Prompting via Image Inpaintingcited by 248
  16. 2023MOSE: A New Dataset for Video Object Segmentation in Complex Scenescited by 232
  17. 2019iSAID: A Large-scale Dataset for Instance Segmentation in Aerial Imagescited by 195
  18. 2021Hypercorrelation Squeeze for Few-Shot Segmentationcited by 193
  19. 2016Deep Interactive Object Selectioncited by 160
  20. 2022A Unified Sequence Interface for Vision Taskscited by 152
  21. 2023PACO: Parts and Attributes of Common Objectscited by 146
  22. 2021Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Taskscited by 124
  23. 2017Semantic Instance Segmentation with a Discriminative Loss Functioncited by 118
  24. 2022Cost Aggregation with 4D Convolutional Swin Transformer for Few-Shot Segmentationcited by 114
  25. 2020Video Object Segmentation with Adaptive Feature Bank and Uncertain-Region Refinementcited by 101
  26. 2022Self-Support Few-Shot Semantic Segmentationcited by 96
  27. 2018Look into Person: Joint Body Parsing & Pose Estimation Network and A New Benchmarkcited by 79
  28. 2022UViM: A Unified Modeling Approach for Vision with Learned Guiding Codescited by 77
  29. 2018A Generative Appearance Model for End-to-end Video Object Segmentationcited by 66
  30. 2022Feature-Proxy Transformer for Few-Shot Segmentationcited by 58
  31. 2018Iteratively Trained Interactive Segmentationcited by 52
  32. 2022Recurrent Dynamic Embedding for Video Object Segmentationcited by 52
  33. 2021SOLO: A Simple Framework for Instance Segmentationcited by 32
  34. 2022SWEM: Towards Real-Time Video Object Segmentation with Sequential Weighted Expectation-Maximizationcited by 26
  35. 2022Doubly Deformable Aggregation of Covariance Matrices for Few-shot Segmentationcited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 65; 2024: 71; 2025: 38; 2026: 19.

193 papers.

Sort:

By area:

  1. 2023Segment Everything Everywhere All at Oncecited by 546
  2. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  3. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  4. 2023Segment Anything in High Qualitycited by 373
  5. 2023Customized Segment Anything Model for Medical Image Segmentationcited by 277
  6. 2023Personalize Segment Anything Model with One Shotcited by 250
  7. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  8. 2023Semantic-SAM: Segment and Recognize Anything at Any Granularitycited by 185
  9. 2023RSPrompter: Learning to Prompt for Remote Sensing Instance Segmentation based on Visual Foundation Modelcited by 180
  10. 2024DragAnything: Motion Control for Anything using Entity Representationcited by 159
  11. 2024Medical SAM 2: Segment medical images as video via Segment Anything Model 2cited by 151
  12. 2023Matcher: Segment Anything with One Shot Using All-Purpose Feature Matchingcited by 142
38 more
  1. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 136
  2. 2024OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understandingcited by 126
  3. 2023Vision Language Models in Autonomous Driving: A Survey and Outlookcited by 124
  4. 2023Segment Any Point Cloud Sequences by Distilling Vision Foundation Modelscited by 123
  5. 2023SAM-Med2Dcited by 123
  6. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  7. 2023In-Context Learning Unlocked for Diffusion Modelscited by 103
  8. 2023SAM-Med3D: Towards General-purpose Segmentation Models for Volumetric Medical Imagescited by 101
  9. 2024PSALM: Pixelwise SegmentAtion with Large Multi-Modal Modelcited by 100
  10. 2023UniverSeg: Universal Medical Image Segmentationcited by 90
  11. 2024Toward Generalist Anomaly Detection via In-context Residual Learning with Few-shot Sample Promptscited by 88
  12. 2023Fine-Grained Visual Promptingcited by 81
  13. 2023Segment Anything Meets Point Trackingcited by 81
  14. 2023Review of Large Vision Models and Visual Prompt Engineeringcited by 77
  15. 2024VRP-SAM: SAM with Visual Reference Promptcited by 75
  16. 2024Open-Vocabulary SAM: Segment and Recognize Twenty-thousand Classes Interactivelycited by 74
  17. 2024OMG-Seg: Is One Model Good Enough For All Segmentation?cited by 73
  18. 20233DSAM-adapter: Holistic adaptation of SAM from 2D to 3D for promptable tumor segmentationcited by 70
  19. 2023A Comprehensive Survey on Segment Anything Model for Vision and Beyondcited by 64
  20. 2023ImageBrush: Learning Visual In-Context Instructions for Exemplar-Based Image Manipulationcited by 59
  21. 2023Adapting Segment Anything Model for Change Detection in HR Remote Sensing Imagescited by 56
  22. 2023SA-Med2D-20M Dataset: Segment Anything in 2D Medical Imaging with 20 Million maskscited by 54
  23. 2023Segment Anything Model (SAM) Enhanced Pseudo Labels for Weakly Supervised Semantic Segmentationcited by 53
  24. 2023Visual In-Context Promptingcited by 52
  25. 2023Going Denser with Open-Vocabulary Part Segmentationcited by 51
  26. 2023LLaVA-Interactive: An All-in-One Demo for Image Chat, Segmentation, Generation and Editingcited by 47
  27. 2023Exploring Plain ViT Reconstruction for Multi-class Unsupervised Anomaly Detectioncited by 47
  28. 2023SAMPro3D: Locating SAM Prompts in 3D for Zero-Shot Instance Segmentationcited by 44
  29. 2023Change Detection Methods for Remote Sensing in the Last Decade: A Comprehensive Reviewcited by 41
  30. 2024LVOS: A Benchmark for Large-scale Long-term Video Object Segmentationcited by 41
  31. 2023Explore In-Context Learning for 3D Point Cloud Understandingcited by 40
  32. 2024Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perceptioncited by 39
  33. 2023One-Prompt to Segment All Medical Imagescited by 38
  34. 2025DICEPTION: A Generalist Diffusion Model for Visual Perceptual Taskscited by 37
  35. 2023SLiMe: Segment Like Mecited by 34
  36. 2023SAM3D: Segment Anything Model in Volumetric Medical Imagescited by 34
  37. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  38. 2024On the Out-Of-Distribution Generalization of Multimodal Large Language Modelscited by 30

50 most cited of 193 All 193 (JSON)