mlsanityData to 14 Oct 2026

Learning to Generate Text-grounded Mask for Open-world Semantic Segmentation from Only Image-Text Pairs

arXiv 2212.00785 ↗First posted 1 Dec 2022cs.CVcited by 130

Not on the map. Why?

Cited by 32 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We tackle open-world semantic segmentation, which aims at learning to segment arbitrary visual concepts in images, by using only image-text pairs without dense annotations. Existing open-world segmentation methods have shown impressive advances by employing contrastive learning (CL) to learn diverse visual concepts and transferring the learned image-level understanding to the segmentation task. However, these CL-based methods suffer from a train-test discrepancy, since it only considers image-text alignment during training, whereas segmentation requires region-text alignment during testing. In this paper, we proposed a novel Text-grounded Contrastive Learning (TCL) framework that enables a model to directly learn region-text alignment. Our method generates a segmentation mask for a given text, extracts text-grounded image embedding from the masked region, and aligns it with text embedding via TCL. By learning region-text alignment directly, our framework encourages a model to directly improve the quality of generated segmentation masks. In addition, for a rigorous and fair comparison, we present a unified evaluation protocol with widely used 8 semantic segmentation datasets. TCL achieves state-of-the-art zero-shot segmentation performances with large margins in all datasets. Code is available at https://github.com/kakaobrain/tcl.

Key facts. Cited by 130 papers in the map (data through 14 Oct 2026). Builds on 26 of its 41 references that are in the map. First posted on arXiv 1 Dec 2022.

Builds on

26 of 41 references.

References · 26

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  4. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  5. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,940
  6. 2015Fast R-CNNcited by 2,933
  7. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
  8. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  9. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,332
  10. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  11. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  12. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,152
14 more
  1. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  2. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 906
  3. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 864
  4. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 819
  5. 2022Language-driven Semantic Segmentationcited by 717
  6. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 616
  7. 2021Extract Free Dense Labels from CLIPcited by 583
  8. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 565
  9. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 531
  10. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 473
  11. 2020Single-Stage Semantic Segmentation from Image Labelscited by 167
  12. 2022Domain Generalization by Mutual-Information Regularization with Pre-trained Modelscited by 133
  13. 2022ReCo: Retrieve and Co-segment for Zero-shot Transfercited by 117
  14. 2022Open-world Semantic Segmentation via Contrasting and Clustering Vision-Language Embeddingcited by 67

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 20; 2024: 44; 2025: 42; 2026: 24.

130 papers.

Sort:

By area:

  1. 2023SCLIP: Rethinking Self-Attention for Dense Vision-Language Inferencecited by 170
  2. 2023Towards Open Vocabulary Learning: A Surveycited by 166
  3. 2024ClearCLIP: Decomposing CLIP Representations for Dense Vision-Language Inferencecited by 113
  4. 2023SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial Understandingcited by 111
  5. 2023CLIP-DINOiser: Teaching CLIP a few DINO tricks for open-vocabulary semantic segmentationcited by 96
  6. 2024DINOv2 Meets Text: A Unified Framework for Image- and Pixel-Level Vision-Language Alignmentcited by 79
  7. 2024Pay Attention to Your Neighbours: Training-Free Open-Vocabulary Semantic Segmentationcited by 77
  8. 2023Diffusion Model is Secretly a Training-free Open Vocabulary Semantic Segmentercited by 76
  9. 2023CLIP as RNN: Segment Countless Visual Concepts without Training Endeavorcited by 69
  10. 2023SILC: Improving Vision Language Pretraining with Self-Distillationcited by 66
  11. 2023Diffusion Models for Open-Vocabulary Segmentationcited by 64
  12. 2023SATR: Zero-Shot Semantic Segmentation of 3D Shapescited by 62
38 more
  1. 2023A Survey on Open-Vocabulary Detection and Segmentation: Past, Present, and Futurecited by 61
  2. 2024Explore the Potential of CLIP for Training-Free Open Vocabulary Semantic Segmentationcited by 61
  3. 2024Training-Free Open-Vocabulary Segmentation with Offline Diffusion-Augmented Prototype Generationcited by 44
  4. 2024Talking to DINO: Bridging Self-Supervised Vision Backbones with Language for Open-Vocabulary Segmentationcited by 44
  5. 2024EmerDiff: Emerging Pixel-level Semantic Knowledge in Diffusion Modelscited by 43
  6. 2023SemiVL: Semi-Supervised Semantic Segmentation with Vision-Language Guidancecited by 41
  7. 2023CLIP-DIY: CLIP Dense Inference Yields Open-Vocabulary Semantic Segmentation For-Freecited by 39
  8. 2024Harnessing Vision Foundation Models for High-Performance, Training-Free Open Vocabulary Segmentationcited by 39
  9. 2024In Defense of Lazy Visual Grounding for Open-Vocabulary Semantic Segmentationcited by 38
  10. 2024CorrCLIP: Reconstructing Patch Correlations in CLIP for Open-Vocabulary Semantic Segmentationcited by 34
  11. 2024Self-Calibrated CLIP for Training-Free Open-Vocabulary Segmentationcited by 34
  12. 2025DeCLIP: Decoupled Learning for Open-Vocabulary Dense Perceptioncited by 33
  13. 2024ResCLIP: Residual Attention for Training-free Dense Vision-language Inferencecited by 29
  14. 2023Uncovering Prototypical Knowledge for Weakly Open-Vocabulary Semantic Segmentationcited by 27
  15. 2024DiffCut: Catalyzing Zero-Shot Semantic Segmentation with Diffusion Features and Recursive Normalized Cutcited by 26
  16. 2024SegEarth-OV: Towards Training-Free Open-Vocabulary Segmentation for Remote Sensing Imagescited by 25
  17. 2024Distilling Spectral Graph for Object-Context Aware Open-Vocabulary Semantic Segmentationcited by 24
  18. 2023Multi-Modal Prototypes for Open-World Semantic Segmentationcited by 22
  19. 2023Rewrite Caption Semantics: Bridging Semantic Gaps for Language-Supervised Semantic Segmentationcited by 21
  20. 2024Image-Text Co-Decomposition for Text-Supervised Semantic Segmentationcited by 21
  21. 2025LPOSS: Label Propagation Over Patches and Pixels for Open-vocabulary Semantic Segmentationcited by 21
  22. 2024Leveraging Hallucinations to Reduce Manual Prompt Dependency in Promptable Segmentationcited by 20
  23. 2024Towards Open-Vocabulary Semantic Segmentation Without Semantic Labelscited by 19
  24. 2024Cascade-CLIP: Cascaded Vision-Language Embeddings Alignment for Zero-Shot Semantic Segmentationcited by 17
  25. 2024CLIPer: Hierarchically Improving Spatial Representation of CLIP for Open-Vocabulary Semantic Segmentationcited by 17
  26. 2025SegEarth-OV3: Exploring SAM 3 for Open-Vocabulary Semantic Segmentation in Remote Sensing Imagescited by 16
  27. 2023Towards Training-free Open-world Segmentation via Image Prompt Foundation Modelscited by 15
  28. 2023Emergent Open-Vocabulary Semantic Segmentation from Off-the-shelf Vision-Language Modelscited by 15
  29. 2024Image-to-Image Matching via Foundation Models: A New Perspective for Open-Vocabulary Semantic Segmentationcited by 15
  30. 2024DetailCLIP: Detail-Oriented CLIP for Fine-Grained Taskscited by 15
  31. 2024COSMOS: Cross-Modality Self-Distillation for Vision Language Pre-trainingcited by 15
  32. 2023Can CLIP Help Sound Source Localization?cited by 14
  33. 2024GLASS: Guided Latent Slot Diffusion for Object-Centric Learningcited by 14
  34. 2026TIPSv2: Advancing Vision-Language Pretraining with Enhanced Patch-Text Alignmentcited by 13
  35. 2024Open-Vocabulary Segmentation with Unpaired Mask-Text Supervisioncited by 12
  36. 2024FreeSeg-Diff: Training-Free Open-Vocabulary Segmentation with Diffusion Modelscited by 12
  37. 2025Plug-in Feedback Self-adaptive Attention in CLIP for Training-free Open-Vocabulary Segmentationcited by 12
  38. 2024As Firm As Their Foundations: Can open-sourced foundation models be used to create adversarial examples for downstream tasks?cited by 11

50 most cited of 130 All 130 (JSON)