mlsanityData to 7 Oct 2026

Semantic-SAM: Segment and Recognize Anything at Any Granularity

arXiv 2307.04767 ↗First posted 10 Jul 2023cs.CVcited by 185

Not on the map. Why?

Cited by 22 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this paper, we introduce Semantic-SAM, a universal image segmentation model to enable segment and recognize anything at any desired granularity. Our model offers two key advantages: semantic-awareness and granularity-abundance. To achieve semantic-awareness, we consolidate multiple datasets across three granularities and introduce decoupled classification for objects and parts. This allows our model to capture rich semantic information. For the multi-granularity capability, we propose a multi-choice learning scheme during training, enabling each click to generate masks at multiple levels that correspond to multiple ground-truth masks. Notably, this work represents the first attempt to jointly train a model on SA-1B, generic, and part segmentation datasets. Experimental results and visualizations demonstrate that our model successfully achieves semantic-awareness and granularity-abundance. Furthermore, combining SA-1B training with other segmentation tasks, such as panoptic and part segmentation, leads to performance improvements. We will provide code and a demo for further exploration and evaluation.

Key facts. Cited by 185 papers in the map (data through 7 Oct 2026). Builds on 52 of its 69 references that are in the map. First posted on arXiv 10 Jul 2023.

Builds on

52 of 69 references.

References · 52

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  7. 2020End-to-End Object Detection with Transformerscited by 7,384
  8. 2017Mask R-CNNcited by 7,259
  9. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  10. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  11. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  12. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
38 more
  1. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  2. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
  3. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  4. 2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,154
  5. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  6. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  7. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  8. 2018Panoptic Segmentationcited by 796
  9. 2022Language-driven Semantic Segmentationcited by 711
  10. 2020Image Segmentation Using Deep Learning: A Surveycited by 659
  11. 2023Segment Everything Everywhere All at Oncecited by 546
  12. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 530
  13. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
  14. 2019Object Detection in 20 Years: A Surveycited by 528
  15. 2019YOLACT: Real-time Instance Segmentationcited by 441
  16. 2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Modelscited by 371
  17. 2022Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentationcited by 341
  18. 2022OneFormer: One Transformer to Rule Universal Image Segmentationcited by 327
  19. 2022Generalized Decoding for Pixel, Image, and Languagecited by 292
  20. 2014Detect What You Can: Detecting and Representing Objects using Holistic Models and Body Partscited by 230
  21. 2023A Simple Framework for Open-Vocabulary Segmentation and Detectioncited by 197
  22. 2022Unified Contrastive Learning in Image-Text-Label Spacecited by 194
  23. 2023SegGPT: Segmenting Everything In Contextcited by 193
  24. 2016Deep Interactive Object Selectioncited by 160
  25. 2023PACO: Parts and Attributes of Common Objectscited by 146
  26. 2021Panoptic SegFormer: Delving Deeper into Panoptic Segmentation with Transformerscited by 109
  27. 2022SimpleClick: Interactive Image Segmentation with Simple Vision Transformerscited by 109
  28. 2021PartImageNet: A Large, High-Quality Dataset of Partscited by 107
  29. 2022FocalClick: Towards Practical Interactive Image Segmentationcited by 101
  30. 2018ApolloCar3D: A Large 3D Car Instance Understanding Benchmark for Autonomous Drivingcited by 59
  31. 2020Fashionpedia: Ontology, Segmentation, and an Attribute Localization Datasetcited by 57
  32. 2021Part-aware Panoptic Segmentationcited by 56
  33. 2018Parsing R-CNN for Instance-Level Human Analysiscited by 53
  34. 2023Going Denser with Open-Vocabulary Part Segmentationcited by 51
  35. 2023MP-Former: Mask-Piloted Transformer for Image Segmentationcited by 39
  36. 2022Panoptic-PartFormer: Learning a Unified Model for Panoptic Part Segmentationcited by 38
  37. 2019Learning Semantic Neural Tree for Human Parsingcited by 29
  38. 2017Holistic, Instance-Level Human Parsingcited by 19

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 37; 2024: 78; 2025: 47; 2026: 23.

185 papers.

Sort:

By area:

  1. 2024Grounded SAM: Assembling Open-World Models for Diverse Visual Taskscited by 914
  2. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 291
  3. 2024Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Modelscited by 260
  4. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  5. 2023Tracking Anything with Decoupled Video Segmentationcited by 197
  6. 2023LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agentscited by 188
  7. 2023Osprey: Pixel Understanding with Visual Instruction Tuningcited by 158
  8. 2025Magma: A Foundation Model for Multimodal AI Agentscited by 138
  9. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 136
  10. 2024Aria-UI: Visual Grounding for GUI Instructionscited by 136
  11. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
  12. 2024MOKA: Open-World Robotic Manipulation through Mark-Based Visual Promptingcited by 126
38 more
  1. 2023SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial Understandingcited by 111
  2. 2024Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Wantcited by 98
  3. 2024Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Modelscited by 97
  4. 2024T-Rex2: Towards Generic Object Detection via Text-Visual Prompt Synergycited by 94
  5. 2023SAI3D: Segment Any Instance in 3D Scenescited by 90
  6. 2025ReFocus: Visual Editing as a Chain of Thought for Structured Image Understandingcited by 81
  7. 2024SAM2Long: Enhancing SAM 2 for Long Video Segmentation with a Training-Free Memory Treecited by 77
  8. 2024Open-Vocabulary SAM: Segment and Recognize Twenty-thousand Classes Interactivelycited by 74
  9. 2024OMG-Seg: Is One Model Good Enough For All Segmentation?cited by 73
  10. 2023General Object Foundation Model for Images and Videos at Scalecited by 71
  11. 2023LISA++: An Improved Baseline for Reasoning Segmentation with Large Language Modelcited by 70
  12. 2023A Survey on Open-Vocabulary Detection and Segmentation: Past, Present, and Futurecited by 59
  13. 2024Code-as-Monitor: Constraint-aware Visual Programming for Reactive and Proactive Robotic Failure Detectioncited by 59
  14. 2024Scaffolding Coordinates to Promote Vision-Language Coordination in Large Multi-Modal Modelscited by 56
  15. 2023Open-Fusion: Real-time Open-Vocabulary 3D Mapping and Queryable Scene Representationcited by 54
  16. 2023Visual In-Context Promptingcited by 52
  17. 2026Ultralytics YOLO26: Unified Real-Time End-to-End Vision Modelscited by 47
  18. 2023DVIS++: Improved Decoupled Framework for Universal Video Segmentationcited by 46
  19. 2023SAMPro3D: Locating SAM Prompts in 3D for Zero-Shot Instance Segmentationcited by 44
  20. 2023MobileSAMv2: Faster Segment Anything to Everythingcited by 41
  21. 2024EmbodiedSAM: Online Segment Any 3D Thing in Real Timecited by 40
  22. 2024Chat2SVG: Vector Graphics Generation with Large Language Models and Image Diffusion Modelscited by 36
  23. 2024Image Segmentation in Foundation Model Era: A Surveycited by 35
  24. 2024Bridge the Points: Graph-based Few-shot Segment Anything Semanticallycited by 35
  25. 2023Collaborating Foundation Models for Domain Generalized Semantic Segmentationcited by 33
  26. 2024List Items One by One: A New Data Source and Learning Paradigm for Multimodal LLMscited by 32
  27. 2024PTQ4SAM: Post-Training Quantization for Segment Anythingcited by 32
  28. 2024DarkSAM: Fooling Segment Anything Model to Segment Nothingcited by 32
  29. 2023Tokenize Anything via Promptingcited by 31
  30. 2024Search3D: Hierarchical Open-Vocabulary 3D Segmentationcited by 31
  31. 2024Hi-SAM: Marrying Segment Anything Model for Hierarchical Text Segmentationcited by 30
  32. 2023Universal Segmentation at Arbitrary Granularity with Language Instructioncited by 29
  33. 2024EarthMarker: A Visual Prompting Multi-modal Large Language Model for Remote Sensingcited by 29
  34. 2025Rethinking End-to-End 2D to 3D Scene Segmentation in Gaussian Splattingcited by 29
  35. 2024CAT-SAM: Conditional Tuning for Few-Shot Adaptation of Segment Anything Modelcited by 27
  36. 2024Segment Anything without Supervisioncited by 27
  37. 2024UniVS: Unified and Universal Video Segmentation with Prompts as Queriescited by 26
  38. 2023Cooperation Does Matter: Exploring Multi-Order Bilateral Relations for Audio-Visual Segmentationcited by 25

50 most cited of 185 All 185 (JSON)