mlsanityData to 7 Oct 2026

Segment Everything Everywhere All at Once

arXiv 2304.06718 ↗First posted 13 Apr 2023cs.CVcited by 546

Not on the map. Why?

Cited by 80 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this work, we present SEEM, a promptable and interactive model for segmenting everything everywhere all at once in an image, as shown in Fig.1. In SEEM, we propose a novel decoding mechanism that enables diverse prompting for all types of segmentation tasks, aiming at a universal segmentation interface that behaves like large language models (LLMs). More specifically, SEEM is designed with four desiderata: i) Versatility. We introduce a new visual prompt to unify different spatial queries including points, boxes, scribbles and masks, which can further generalize to a different referring image; ii) Compositionality. We learn a joint visual-semantic space between text and visual prompts, which facilitates the dynamic composition of two prompt types required for various segmentation tasks; iii) Interactivity. We further incorporate learnable memory prompts into the decoder to retain segmentation history through mask-guided cross-attention from decoder to image features; and iv) Semantic-awareness. We use a text encoder to encode text queries and mask labels into the same semantic space for open-vocabulary segmentation. We conduct a comprehensive empirical study to validate the effectiveness of SEEM across diverse segmentation tasks. Notably, our single SEEM model achieves competitive performance across interactive segmentation, generic segmentation, referring segmentation, and video object segmentation on 9 datasets with minimum 1/100 supervision. Furthermore, SEEM showcases a remarkable capacity for generalization to novel prompts or their combinations, rendering it a readily universal image segmentation interface.

Key facts. Cited by 546 papers in the map (data through 7 Oct 2026). Builds on 76 of its 90 references that are in the map. First posted on arXiv 13 Apr 2023.

Builds on

76 of 90 references.

References · 76

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  8. 2020End-to-End Object Detection with Transformerscited by 7,384
  9. 2017Mask R-CNNcited by 7,259
  10. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  11. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  12. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
38 more
  1. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  2. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  3. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  4. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  5. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
  6. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  7. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  8. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  9. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  10. 2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,154
  11. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  12. 2016Modeling Context in Referring Expressionscited by 1,054
  13. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 942
  14. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  15. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  16. 2018Panoptic Segmentationcited by 796
  17. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 774
  18. 2022Language-driven Semantic Segmentationcited by 711
  19. 2020Image Segmentation Using Deep Learning: A Surveycited by 659
  20. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  21. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
  22. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 530
  23. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
  24. 2019Object Detection in 20 Years: A Surveycited by 528
  25. 2019YOLACT: Real-time Instance Segmentationcited by 441
  26. 2022XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Modelcited by 436
  27. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  28. 2018YouTube-VOS: A Large-Scale Video Object Segmentation Benchmarkcited by 381
  29. 2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Modelscited by 371
  30. 2020MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformerscited by 355
  31. 2021LAVT: Language-Aware Vision Transformer for Referring Image Segmentationcited by 347
  32. 2022Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentationcited by 341
  33. 2022OneFormer: One Transformer to Rule Universal Image Segmentationcited by 327
  34. 2022Images Speak in Images: A Generalist Painter for In-Context Visual Learningcited by 317
  35. 2023Side Adapter Network for Open-Vocabulary Semantic Segmentationcited by 296
  36. 2022Generalized Decoding for Pixel, Image, and Languagecited by 292
  37. 2018Fast Online Object Tracking and Segmentation: A Unifying Approachcited by 272
  38. 2019Cross-Modal Self-Attention Network for Referring Image Segmentationcited by 263

50 most cited of 76 All 76 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 104; 2024: 182; 2025: 178; 2026: 82.

546 papers.

Sort:

By area:

  1. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  2. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 770
  3. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  4. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  5. 2023Segment Anything in High Qualitycited by 373
  6. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 291
  7. 2023Personalize Segment Anything Model with One Shotcited by 250
  8. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  9. 2023Segment Anything in Medical Imagescited by 226
  10. 2023Tracking Anything with Decoupled Video Segmentationcited by 197
  11. 2023LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agentscited by 188
  12. 2023Semantic-SAM: Segment and Recognize Anything at Any Granularitycited by 185
38 more
  1. 2023RSPrompter: Learning to Prompt for Remote Sensing Instance Segmentation based on Visual Foundation Modelcited by 180
  2. 2023SCLIP: Rethinking Self-Attention for Dense Vision-Language Inferencecited by 169
  3. 2023Segment Any 3D Gaussianscited by 158
  4. 2023GSVA: Generalized Segmentation via Multimodal Large Language Modelscited by 153
  5. 2024SpatialBot: Precise Spatial Understanding with Vision Language Modelscited by 146
  6. 2023Matcher: Segment Anything with One Shot Using All-Purpose Feature Matchingcited by 142
  7. 2025Magma: A Foundation Model for Multimodal AI Agentscited by 138
  8. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 136
  9. 2024Aria-UI: Visual Grounding for GUI Instructionscited by 136
  10. 2024MOKA: Open-World Robotic Manipulation through Mark-Based Visual Promptingcited by 126
  11. 2023Transformer-Based Visual Segmentation: A Surveycited by 124
  12. 2023Segment Any Point Cloud Sequences by Distilling Vision Foundation Modelscited by 123
  13. 2023SAM-Med2Dcited by 123
  14. 2024VISA: Reasoning Video Object Segmentation via Large Language Modelscited by 123
  15. 2024Groma: Localized Visual Tokenization for Grounding Multimodal Large Language Modelscited by 118
  16. 2023SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial Understandingcited by 111
  17. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  18. 2024BiomedParse: a biomedical foundation model for image parsing of everything everywhere all at oncecited by 107
  19. 2023SAM-Med3D: Towards General-purpose Segmentation Models for Volumetric Medical Imagescited by 101
  20. 2024PSALM: Pixelwise SegmentAtion with Large Multi-Modal Modelcited by 100
  21. 2024One Token to Seg Them All: Language Instructed Reasoning Segmentation in Videoscited by 99
  22. 2024Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Wantcited by 98
  23. 2024Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Modelscited by 97
  24. 2023MA-SAM: Modality-agnostic SAM Adaptation for 3D Medical Image Segmentationcited by 82
  25. 2024RandAR: Decoder-only Autoregressive Visual Generation in Random Orderscited by 82
  26. 2023SAM on Medical Images: A Comprehensive Study on Three Prompt Modescited by 81
  27. 2023Review of Large Vision Models and Visual Prompt Engineeringcited by 77
  28. 2024Open-Vocabulary SAM: Segment and Recognize Twenty-thousand Classes Interactivelycited by 74
  29. 2024OMG-Seg: Is One Model Good Enough For All Segmentation?cited by 73
  30. 20243D-Mem: 3D Scene Memory for Embodied Exploration and Reasoningcited by 73
  31. 2023General Object Foundation Model for Images and Videos at Scalecited by 71
  32. 20233DSAM-adapter: Holistic adaptation of SAM from 2D to 3D for promptable tumor segmentationcited by 70
  33. 2024Inducing High Energy-Latency of Large Vision-Language Models with Verbose Imagescited by 69
  34. 2024SAM4MLLM: Enhance Multi-Modal Large Language Model for Referring Expression Segmentationcited by 68
  35. 2025Detect Anything via Next Point Predictioncited by 66
  36. 2023A Comprehensive Survey on Segment Anything Model for Vision and Beyondcited by 64
  37. 2023Segment Anything in 3D with Radiance Fieldscited by 61
  38. 2024CoMat: Aligning Text-to-Image Diffusion Model with Image-to-Text Concept Matchingcited by 60

50 most cited of 546 All 546 (JSON)