mlsanityData to 14 Oct 2026

Open-Vocabulary SAM: Segment and Recognize Twenty-thousand Classes Interactively

arXiv 2401.02955 ↗First posted 5 Jan 2024cs.CVcited by 75

Not on the map. Why?

Abstract (arXiv)

The CLIP and Segment Anything Model (SAM) are remarkable vision foundation models (VFMs). SAM excels in segmentation tasks across diverse domains, whereas CLIP is renowned for its zero-shot recognition capabilities. This paper presents an in-depth exploration of integrating these two models into a unified framework. Specifically, we introduce the Open-Vocabulary SAM, a SAM-inspired model designed for simultaneous interactive segmentation and recognition, leveraging two unique knowledge transfer modules: SAM2CLIP and CLIP2SAM. The former adapts SAM's knowledge into the CLIP via distillation and learnable transformer adapters, while the latter transfers CLIP knowledge into SAM, enhancing its recognition capabilities. Extensive experiments on various datasets and detectors show the effectiveness of Open-Vocabulary SAM in both segmentation and recognition tasks, significantly outperforming the na"ive baselines of simply combining SAM and CLIP. Furthermore, aided with image classification data training, our method can segment and recognize approximately 22,000 classes.

Key facts. Cited by 75 papers in the map (data through 14 Oct 2026). Builds on 85 of its 93 references that are in the map. First posted on arXiv 5 Jan 2024.

Builds on

85 of 93 references.

References · 85

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  8. 2023Segment Anythingcited by 8,234
  9. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  10. 2017Mask R-CNNcited by 7,277
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  12. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
38 more
  1. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  2. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  3. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  4. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,351
  5. 2016V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentationcited by 2,292
  6. 2021Learning to Prompt for Vision-Language Modelscited by 2,281
  7. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  8. 2019MMDetection: Open MMLab Detection Toolbox and Benchmarkcited by 1,385
  9. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  10. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,217
  11. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  12. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,026
  13. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 906
  14. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 864
  15. 2018Panoptic Segmentationcited by 802
  16. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 798
  17. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 724
  18. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 653
  19. 2021Learning To Retrieve Prompts for In-Context Learningcited by 648
  20. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 614
  21. 2021Extract Free Dense Labels from CLIPcited by 583
  22. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 577
  23. 2022Vision Transformer Adapter for Dense Predictionscited by 565
  24. 2023Segment Everything Everywhere All at Oncecited by 548
  25. 2023Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentationcited by 490
  26. 2020Open-Vocabulary Object Detection Using Captionscited by 427
  27. 2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Modelscited by 375
  28. 2023Faster Segment Anything: Towards Lightweight SAM for Mobile Applicationscited by 352
  29. 2022Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentationcited by 344
  30. 2022Images Speak in Images: A Generalist Painter for In-Context Visual Learningcited by 318
  31. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  32. 2023Recognize Anything: A Strong Image Tagging Modelcited by 309
  33. 2023Side Adapter Network for Open-Vocabulary Semantic Segmentationcited by 300
  34. 2022Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuningcited by 294
  35. 2023Fast Segment Anythingcited by 290
  36. 2022Visual Prompting via Image Inpaintingcited by 250
  37. 2023Personalize Segment Anything Model with One Shotcited by 250
  38. 2021A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Modelcited by 223

50 most cited of 85 All 85 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 32; 2025: 29; 2026: 14.

75 papers.

Sort:

By area:

  1. 2024OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understandingcited by 127
  2. 2024ProxyCLIP: Proxy Attention Improves CLIP for Open-Vocabulary Segmentationcited by 111
  3. 2024OMG-Seg: Is One Model Good Enough For All Segmentation?cited by 75
  4. 2024Gen3DSR: Generalizable 3D Scene Reconstruction via Divide and Conquer from a Single Viewcited by 61
  5. 2025UAVs Meet LLMs: Overviews and Perspectives Toward Agentic Low-Altitude Mobilitycited by 49
  6. 2024Image Segmentation in Foundation Model Era: A Surveycited by 35
  7. 2024Bridge the Points: Graph-based Few-shot Segment Anything Semanticallycited by 35
  8. 2024Self-Calibrated CLIP for Training-Free Open-Vocabulary Segmentationcited by 34
  9. 2024Towards Language-Driven Video Inpainting via Multimodal Large Language Modelscited by 33
  10. 2025DeCLIP: Decoupled Learning for Open-Vocabulary Dense Perceptioncited by 33
  11. 2025X-SAM: From Segment Anything to Any Segmentationcited by 30
  12. 2024UNIC: Universal Classification Models via Multi-teacher Distillationcited by 27
38 more
  1. 2024PointSAM: Pointly-Supervised Segment Anything Model for Remote Sensing Imagescited by 24
  2. 2024MG-LLaVA: Towards Multi-Granularity Visual Instruction Tuningcited by 22
  3. 2025Pixel-SAIL: Single Transformer For Pixel-Grounded Understandingcited by 22
  4. 2024SemFlow: Binding Semantic Segmentation and Image Synthesis via Rectified Flowcited by 20
  5. 2024Mamba or RWKV: Exploring High-Quality and High-Efficiency Segment Anything Modelcited by 18
  6. 2024Training-Free Open-Ended Object Detection and Segmentation via Attention as Promptscited by 16
  7. 2025DenseWorld-1M: Towards Detailed Dense Grounded Caption in the Real Worldcited by 16
  8. 2025DC-SAM: In-Context Segment Anything in Images and Videos via Dual Consistencycited by 15
  9. 2024Explore In-Context Segmentation via Latent Diffusion Modelscited by 12
  10. 2025Not All Data Are Unlearned Equallycited by 12
  11. 2025CRISP-SAM2: SAM2 with Cross-Modal Interaction and Semantic Prompting for Multi-Organ Segmentationcited by 12
  12. 2025Plug-in Feedback Self-adaptive Attention in CLIP for Training-free Open-Vocabulary Segmentationcited by 12
  13. 2024Generative Active Learning for Long-tailed Instance Segmentationcited by 10
  14. 2025Online Language Splattingcited by 10
  15. 2025Generalized Decoupled Learning for Enhancing Open-Vocabulary Dense Perceptioncited by 10
  16. 2025Unveiling the Potential of Segment Anything Model 2 for RGB-Thermal Semantic Segmentation with Language Guidancecited by 9
  17. 2024Open-Vocabulary SAM3D: Towards Training-free Open-Vocabulary 3D Scene Understandingcited by 7
  18. 2024FrozenSeg: Harmonizing Frozen Foundation Models for Open-Vocabulary Segmentationcited by 7
  19. 2025UniVCD: A New Method for Unsupervised Change Detection in the Open-Vocabulary Eracited by 7
  20. 2024RMP-SAM: Towards Real-Time Multi-Purpose Segment Anythingcited by 6
  21. 2024Exploring Simple Open-Vocabulary Semantic Segmentationcited by 6
  22. 2024Tuning-free Universally-Supervised Semantic Segmentationcited by 6
  23. 2024Adapting Segment Anything Model to Multi-modal Salient Object Detection with Semantic Feature Fusion Guidancecited by 6
  24. 2025Zero-Shot 4D Lidar Panoptic Segmentationcited by 6
  25. 2024Bayesian Active Learning for Semantic Segmentationcited by 5
  26. 2024Quantifying the Limits of Segmentation Foundation Models: Modeling Challenges in Segmenting Tree-Like and Low-Contrast Objectscited by 5
  27. 2024High-Quality Mask Tuning Matters for Open-Vocabulary Segmentationcited by 5
  28. 2024Boosting Few-Shot Semantic Segmentation Via Segment Anything Modelcited by 4
  29. 2024EchoONE: Segmenting Multiple echocardiography Planes in One Modelcited by 4
  30. 2025SAM-MI: A Mask-Injected Framework for Enhancing Open-Vocabulary Semantic Segmentation with SAMcited by 4
  31. 2025DC-TTA: Divide-and-Conquer Framework for Test-Time Adaptation of Interactive Segmentationcited by 3
  32. 2025Zero-shot Hierarchical Plant Segmentation via Foundation Segmentation Models and Text-to-image Attentioncited by 3
  33. 2026Segment Any Events with Languagecited by 3
  34. 2024Segment Any Object Model (SAOM): Real-to-Simulation Fine-Tuning Strategy for Multi-Class Multi-Instance Segmentationcited by 2
  35. 2025MSPL: Multi-Step Pseudo-Labeling for Open-Vocabulary Object Detectioncited by 2
  36. 2025Foundation Model Priors Enhance Object Focus in Feature Space for Source-Free Object Detectioncited by 2
  37. 2026SSP-SAM: SAM with Semantic-Spatial Prompt for Referring Expression Segmentationcited by 2
  38. 2025Benchmarking Feature Upsampling Methods for Vision Foundation Models using Interactive Segmentationcited by 1

50 most cited of 75 All 75 (JSON)