mlsanityData to 14 Oct 2026

ITACLIP: Boosting Training-Free Semantic Segmentation with Image, Text, and Architectural Enhancements

arXiv 2411.12044 ↗First posted 18 Nov 2024cs.CVcited by 7

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Recent advances in foundational Vision Language Models (VLMs) have reshaped the evaluation paradigm in computer vision tasks. These foundational models, especially CLIP, have accelerated research in open-vocabulary computer vision tasks, including Open-Vocabulary Semantic Segmentation (OVSS). Although the initial results are promising, the dense prediction capabilities of VLMs still require further improvement. In this study, we enhance the semantic segmentation performance of CLIP by introducing new modules and modifications: 1) architectural changes in the last layer of ViT and the incorporation of attention maps from the middle layers with the last layer, 2) Image Engineering: applying data augmentations to enrich input image representations, and 3) using Large Language Models (LLMs) to generate definitions and synonyms for each class name to leverage CLIP's open-vocabulary capabilities. Our training-free method, ITACLIP, outperforms current state-of-the-art approaches on segmentation benchmarks such as COCO-Stuff, COCO-Object, Pascal Context, and Pascal VOC. Our code is available at https://github.com/m-arda-aydn/ITACLIP.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 61 of its 70 references that are in the map. First posted on arXiv 18 Nov 2024.

Builds on

61 of 70 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024ClearCLIP: Decomposing CLIP Representations for Dense Vision-Language Inference17 Jul
  2. 2024Explore the Potential of CLIP for Training-Free Open Vocabulary Semantic Segmentation11 Jul
  3. 2024Cascade-CLIP: Cascaded Vision-Language Embeddings Alignment for Zero-Shot Semantic Segmentation2 Jun
  4. 2023Learning Mask-aware CLIP Representations for Zero-Shot Segmentation30 Sep
  5. 2021A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Model29 Dec

Other references · 56

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  5. 2020Language Models are Few-Shot Learnerscited by 32,419
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  8. 2024The Llama 3 Herd of Modelscited by 14,715
  9. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  10. 2023Segment Anythingcited by 8,234
  11. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  12. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
38 more
  1. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  2. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  3. 2015Learning Deep Features for Discriminative Localizationcited by 2,436
  4. 2018Deep contextualized word representationscited by 2,290
  5. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  6. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  7. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 906
  8. 2020Object-Centric Learning with Slot Attentioncited by 876
  9. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 864
  10. 2022Language-driven Semantic Segmentationcited by 717
  11. 2021Extract Free Dense Labels from CLIPcited by 583
  12. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 565
  13. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 531
  14. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 473
  15. 2021MetaICL: Learning to Learn In Contextcited by 454
  16. 2022Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Modelcited by 317
  17. 2023Segment Anything Model for Medical Image Analysis: an Experimental Studycited by 291
  18. 2020Self-Supervised MultiModal Versatile Networkscited by 276
  19. 2023Demystifying CLIP Datacited by 270
  20. 2023Improving CLIP Training with Language Rewritescited by 217
  21. 2023Interpreting CLIP's Image Representation via Text-Based Decompositioncited by 196
  22. 2022ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentationcited by 178
  23. 2023SCLIP: Rethinking Self-Attention for Dense Vision-Language Inferencecited by 170
  24. 2020Single-Stage Semantic Segmentation from Image Labelscited by 167
  25. 2022SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic Segmentationcited by 164
  26. 2023Segment Anything Model (SAM) for Digital Pathology: Assess Zero-shot Segmentation on Whole Slide Imagingcited by 161
  27. 2022Learning to Generate Text-grounded Mask for Open-world Semantic Segmentation from Only Image-Text Pairscited by 130
  28. 2022CLIP is Also an Efficient Segmenter: A Text-Driven Approach for Weakly Supervised Semantic Segmentationcited by 118
  29. 2022ReCo: Retrieve and Co-segment for Zero-shot Transfercited by 117
  30. 2023Learning Open-vocabulary Semantic Segmentation Models From Natural Language Supervisioncited by 99
  31. 2023Grounding Everything: Emerging Localization Properties in Vision-Language Transformerscited by 91
  32. 2023GALIP: Generative Adversarial CLIPs for Text-to-Image Synthesiscited by 85
  33. 2022Perceptual Grouping in Contrastive Vision-Language Modelscited by 82
  34. 2023Open-vocabulary Object Segmentation with Diffusion Modelscited by 82
  35. 2024Pay Attention to Your Neighbours: Training-Free Open-Vocabulary Semantic Segmentationcited by 77
  36. 2023CLIP as RNN: Segment Countless Visual Concepts without Training Endeavorcited by 69
  37. 2022CLIP-GEN: Language-Free Training of a Text-to-Image Generator with CLIPcited by 60
  38. 2023Going Denser with Open-Vocabulary Part Segmentationcited by 51

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 5.

7 papers.

  1. 2025Exploring the Underwater World Segmentation without Extra Trainingcited by 16
  2. 2026ACE-LoRA: Graph-Attentive Context Enhancement for Parameter-Efficient Adaptation of Medical Vision-Language Modelscited by 1
  3. 2025A Survey on Training-free Open-Vocabulary Semantic Segmentationcited by 0
  4. 2026SPAR: Single-Pass Any-Resolution ViT for Open-vocabulary Segmentationcited by 0
  5. 2026[CLS] is Not Enough: Multi-Label Recognition via Patch-Level Inference and Adaptive Aggregationcited by 0
  6. 2026Perceptual Anchoring: Prototype-Guided Text Calibration for Training-free Open-Vocabulary Semantic Segmentationcited by 0
  7. 2026Text-to-seed generation: Training-free open-vocabulary seeded semantic segmentation via re-purposing diffusion as text-guided seed generatorcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards Open-Vocabulary Semantic Segmentation Without Semantic Labels
  2. 2024CLIPer: Hierarchically Improving Spatial Representation of CLIP for Open-Vocabulary Semantic Segmentation
  3. 2023Global Knowledge Calibration for Fast Open-Vocabulary Segmentation
  4. 2024Self-Calibrated CLIP for Training-Free Open-Vocabulary Segmentation
  5. 2025InfoCLIP: Bridging Vision-Language Pretraining and Open-Vocabulary Semantic Segmentation via Information-Theoretic Alignment Transfer
  6. 2023Emergent Open-Vocabulary Semantic Segmentation from Off-the-shelf Vision-Language Models
  7. 2022Open-vocabulary Semantic Segmentation with Frozen Vision-Language Models
  8. 2024LMSeg: Unleashing the Power of Large-Scale Models for Open-Vocabulary Semantic Segmentation