Taming SAM3 in the Wild: A Concept Bank for Open-Vocabulary Segmentation
Not on the map. Why?
Abstract (arXiv)
The recent introduction of textttSAM3 has revolutionized Open-Vocabulary Segmentation (OVS) through textitpromptable concept segmentation, which grounds pixel predictions in flexible concept prompts. However, this reliance on pre-defined concepts makes the model vulnerable: when visual distributions shift (textitdata drift) or conditional label distributions evolve (textitconcept drift) in the target domain, the alignment between visual evidence and prompts breaks down. In this work, we present textscConceptBank, a parameter-free calibration framework to restore this alignment on the fly. Instead of adhering to static prompts, we construct a dataset-specific concept bank from the target statistics. Our approach (textiti) anchors target-domain evidence via class-wise visual prototypes, (textitii) mines representative supports to suppress outliers under data drift, and (textitiii) fuses candidate concepts to rectify concept drift. We demonstrate that textscConceptBank effectively adapts textttSAM3 to distribution drifts, including challenging natural-scene and remote-sensing scenarios, establishing a new baseline for robustness and efficiency in OVS. Code and model are available at https://github.com/pgsmall/ConceptBank.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 72 of its 86 references that are in the map. First posted on arXiv 6 Feb 2026.
Builds on
72 of 86 references.
References · 72
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
- 2014Microsoft COCO: Common Objects in Contextcited by 17,498
- 2015Distilling the Knowledge in a Neural Networkcited by 10,076
- 2023Segment Anythingcited by 8,158
- 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
- 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
- 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
- 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
- 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
- 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
38 more
- 2024SAM 2: Segment Anything in Images and Videoscited by 2,784
- 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
- 2025DINOv3cited by 1,297
- 2021Grounded Language-Image Pre-trainingcited by 1,147
- 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
- 2023Vision Transformers Need Registerscited by 844
- 2025SAM 3: Segment Anything with Conceptscited by 835
- 2022Language-driven Semantic Segmentationcited by 711
- 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
- 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 613
- 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 612
- 2021Extract Free Dense Labels from CLIPcited by 580
- 2023Segment Everything Everywhere All at Oncecited by 546
- 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
- 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 471
- 2020Learning under Concept Drift: A Reviewcited by 470
- 2023RemoteCLIP: A Vision Language Foundation Model for Remote Sensingcited by 357
- 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classificationcited by 355
- 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 315
- 2023Side Adapter Network for Open-Vocabulary Semantic Segmentationcited by 296
- 2022Generalized Decoding for Pixel, Image, and Languagecited by 292
- 2022What does a platypus look like? Generating customized prompts for zero-shot image classificationcited by 263
- 2021LoveDA: A Remote Sensing Land-Cover Dataset for Domain Adaptive Semantic Segmentationcited by 259
- 2019Zero-Shot Semantic Segmentationcited by 245
- 2021A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Modelcited by 223
- 2023CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentationcited by 215
- 2019iSAID: A Large-scale Dataset for Instance Segmentation in Aerial Imagescited by 195
- 2023Cut and Learn for Unsupervised Object Detection and Instance Segmentationcited by 186
- 2023SCLIP: Rethinking Self-Attention for Dense Vision-Language Inferencecited by 169
- 2020Single-Stage Semantic Segmentation from Image Labelscited by 167
- 2022Self-Supervised Transformers for Unsupervised Object Discovery using Normalized Cutcited by 166
- 2022SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic Segmentationcited by 162
- 2022Learning to Generate Text-grounded Mask for Open-world Semantic Segmentation from Only Image-Text Pairscited by 130
- 2022Open Vocabulary Semantic Segmentation with Patch Aligned Contrastive Learningcited by 115
- 2024ClearCLIP: Decomposing CLIP Representations for Dense Vision-Language Inferencecited by 112
- 2023SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial Understandingcited by 111
- 2024ProxyCLIP: Proxy Attention Improves CLIP for Open-Vocabulary Segmentationcited by 110
- 2023SED: A Simple Encoder-Decoder for Open-Vocabulary Semantic Segmentationcited by 95
50 most cited of 72 All 72 (JSON)
Built on by
Citing papers per year: 2026: 3.
3 papers.