mlsanityData to 14 Oct 2026

CLIP as RNN: Segment Countless Visual Concepts without Training Endeavor

arXiv 2312.07661 ↗First posted 12 Dec 2023cs.CV cs.CL cs.LGcited by 69

Not on the map. Why?

Abstract (arXiv)

Existing open-vocabulary image segmentation methods require a fine-tuning step on mask labels and/or image-text datasets. Mask labels are labor-intensive, which limits the number of categories in segmentation datasets. Consequently, the vocabulary capacity of pre-trained VLMs is severely reduced after fine-tuning. However, without fine-tuning, VLMs trained under weak image-text supervision tend to make suboptimal mask predictions. To alleviate these issues, we introduce a novel recurrent framework that progressively filters out irrelevant texts and enhances mask quality without training efforts. The recurrent unit is a two-stage segmenter built upon a frozen VLM. Thus, our model retains the VLM's broad vocabulary space and equips it with segmentation ability. Experiments show that our method outperforms not only the training-free counterparts, but also those fine-tuned with millions of data samples, and sets the new state-of-the-art records for both zero-shot semantic and referring segmentation. Concretely, we improve the current record by 28.8, 16.0, and 6.9 mIoU on Pascal VOC, COCO Object, and Pascal Context.

Key facts. Cited by 69 papers in the map (data through 14 Oct 2026). Builds on 70 of its 84 references that are in the map. First posted on arXiv 12 Dec 2023.

Builds on

70 of 84 references.

References · 70

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  5. 2023Segment Anythingcited by 8,234
  6. 2020End-to-End Object Detection with Transformerscited by 7,414
  7. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  8. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  9. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  10. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  11. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,223
  12. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
38 more
  1. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,351
  2. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,162
  3. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  4. 2017Cascade R-CNN: Delving into High Quality Object Detectioncited by 1,276
  5. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  6. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,152
  7. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  8. 2016Modeling Context in Referring Expressionscited by 1,063
  9. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 906
  10. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 904
  11. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 864
  12. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 819
  13. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 798
  14. 2022Language-driven Semantic Segmentationcited by 717
  15. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 614
  16. 2021Extract Free Dense Labels from CLIPcited by 583
  17. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 577
  18. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 565
  19. 2014Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFscited by 554
  20. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 531
  21. 2021Image Segmentation Using Text and Image Promptscited by 512
  22. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 473
  23. 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 463
  24. 2023Segment Anything in High Qualitycited by 376
  25. 2020MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformerscited by 355
  26. 2021LAVT: Language-Aware Vision Transformer for Referring Image Segmentationcited by 348
  27. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  28. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 305
  29. 2022GLIPv2: Unifying Localization and Vision-Language Understandingcited by 275
  30. 2023GRES: Generalized Referring Expression Segmentationcited by 254
  31. 2016Modeling Context Between Objects for Referring Expression Understandingcited by 251
  32. 2019Zero-Shot Semantic Segmentationcited by 245
  33. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMscited by 218
  34. 2015Conditional Random Fields as Recurrent Neural Networkscited by 214
  35. 2023Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIPcited by 201
  36. 2023A Simple Framework for Open-Vocabulary Segmentation and Detectioncited by 198
  37. 2020Single-Stage Semantic Segmentation from Image Labelscited by 167
  38. 2022SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic Segmentationcited by 164

50 most cited of 70 All 70 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 23; 2025: 24; 2026: 22.

69 papers.

Sort:

By area:

  1. 2024ClearCLIP: Decomposing CLIP Representations for Dense Vision-Language Inferencecited by 113
  2. 2025ConceptAttention: Diffusion Transformers Learn Highly Interpretable Featurescited by 52
  3. 2024Talking to DINO: Bridging Self-Supervised Vision Backbones with Language for Open-Vocabulary Segmentationcited by 44
  4. 2024Image Segmentation in Foundation Model Era: A Surveycited by 35
  5. 2024CorrCLIP: Reconstructing Patch Correlations in CLIP for Open-Vocabulary Semantic Segmentationcited by 34
  6. 2024Self-Calibrated CLIP for Training-Free Open-Vocabulary Segmentationcited by 34
  7. 2024ResCLIP: Residual Attention for Training-free Dense Vision-language Inferencecited by 29
  8. 2024Learning Visual Grounding from Generative Vision and Language Modelcited by 27
  9. 2025IteRPrimE: Zero-shot Referring Image Segmentation with Iterative Grad-CAM Refinement and Primary Word Emphasiscited by 26
  10. 2024SegEarth-OV: Towards Training-Free Open-Vocabulary Segmentation for Remote Sensing Imagescited by 25
  11. 2024Distilling Spectral Graph for Object-Context Aware Open-Vocabulary Semantic Segmentationcited by 24
  12. 2025Multimodal Referring Segmentation: A Surveycited by 22
38 more
  1. 2024kNN-CLIP: Retrieval Enables Training-Free Segmentation on Continually Expanding Large Vocabulariescited by 20
  2. 2024Cascade-CLIP: Cascaded Vision-Language Embeddings Alignment for Zero-Shot Semantic Segmentationcited by 17
  3. 2024CLIPer: Hierarchically Improving Spatial Representation of CLIP for Open-Vocabulary Semantic Segmentationcited by 17
  4. 2025Exploring the Underwater World Segmentation without Extra Trainingcited by 16
  5. 2025SegEarth-OV3: Exploring SAM 3 for Open-Vocabulary Semantic Segmentation in Remote Sensing Imagescited by 16
  6. 2025Hybrid Global-Local Representation with Augmented Spatial Guidance for Zero-Shot Referring Image Segmentationcited by 14
  7. 2024High-Quality Entity Segmentation and Groundingcited by 13
  8. 2024As Firm As Their Foundations: Can open-sourced foundation models be used to create adversarial examples for downstream tasks?cited by 11
  9. 2024iSeg: An Iterative Refinement-based Framework for Training-free Segmentationcited by 10
  10. 2025Online Language Splattingcited by 10
  11. 2025ReME: A Data-Centric Framework for Training-Free Open-Vocabulary Segmentationcited by 9
  12. 2025Training-Free Class Purification for Open-Vocabulary Semantic Segmentationcited by 9
  13. 2025Annotation-Free Open-Vocabulary Segmentation for Remote-Sensing Imagescited by 8
  14. 2024ITACLIP: Boosting Training-Free Semantic Segmentation with Image, Text, and Architectural Enhancementscited by 7
  15. 2025RealSyn: An Effective and Scalable Multimodal Interleaved Document Transformation Paradigmcited by 7
  16. 2024DreamBeast: Distilling 3D Fantastical Animals with Part-Aware Knowledge Transfercited by 5
  17. 2024High-Quality Mask Tuning Matters for Open-Vocabulary Segmentationcited by 5
  18. 2025Target Refocusing via Attention Redistribution for Open-Vocabulary Semantic Segmentation: An Explainability Perspectivecited by 5
  19. 2025LGD: Leveraging Generative Descriptions for Zero-Shot Referring Image Segmentationcited by 4
  20. 2025RESAnything: Attribute Prompting for Arbitrary Referring Segmentationcited by 4
  21. 2025Segment Anyword: Mask Prompt Inversion for Open-Set Grounded Segmentationcited by 4
  22. 2025Evolving, Not Training: Zero-Shot Reasoning Segmentation via Evolutionary Promptingcited by 4
  23. 2026Taming SAM3 in the Wild: A Concept Bank for Open-Vocabulary Segmentationcited by 4
  24. 2026Open-Vocabulary Domain Generalization in Urban-Scene Segmentationcited by 4
  25. 2024A Simple Framework for Open-Vocabulary Zero-Shot Segmentationcited by 3
  26. 2026PEARL: Geometry Aligns Semantics for Training-Free Open-Vocabulary Semantic Segmentationcited by 3
  27. 2024Annotation Free Semantic Segmentation with Vision Foundation Modelscited by 2
  28. 2025SCHNet: SAM Marries CLIP for Human Parsingcited by 2
  29. 2025RefAM: Attention Magnets for Zero-Shot Referral Segmentationcited by 2
  30. 2025NERVE: Neighbourhood & Entropy-guided Random-walk for training free open-Vocabulary sEgmentationcited by 2
  31. 2026Enabling Training-Free Text-Based Remote Sensing Segmentationcited by 2
  32. 2024Gotta Hear Them All: Towards Sound Source Aware Audio Generationcited by 1
  33. 2026Tarot-SAM3: Training-free SAM3 for Any Referring Expression Segmentationcited by 1
  34. 2026DINO Soars: DINOv3 for Open-Vocabulary Semantic Segmentation of Remote Sensing Imagerycited by 1
  35. 2026Early Semantic Grounding in Image Editing Models for Zero-Shot Referring Image Segmentationcited by 1
  36. 2026AnchorDiff: Training-Free Concept Grounding for MM-DiTs via Anchor-Based Graph Propagationcited by 1
  37. 2026DINOde: Continuous Vision-Text Alignment for Open-Vocabulary Semantic Segmentationcited by 1
  38. 2024OVS Meets Continual Learning: Towards Sustainable Open-Vocabulary Segmentationcited by 0

50 most cited of 69 All 69 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Adapting Vision-Language Model with Fine-grained Semantics for Open-Vocabulary Segmentation
  2. 2023Emergent Open-Vocabulary Semantic Segmentation from Off-the-shelf Vision-Language Models
  3. 2021A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Model
  4. 2022Open-vocabulary Semantic Segmentation with Frozen Vision-Language Models
  5. 2023Global Knowledge Calibration for Fast Open-Vocabulary Segmentation
  6. 2026Retrieve and Segment: Are a Few Examples Enough to Bridge the Supervision Gap in Open-Vocabulary Segmentation?
  7. 2024Effective SAM Combination for Open-Vocabulary Semantic Segmentation
  8. 2024Mask-Adapter: The Devil is in the Masks for Open-Vocabulary Segmentation