mlsanityData to 14 Oct 2026

CRIS: CLIP-Driven Referring Image Segmentation

arXiv 2111.15174 ↗First posted 30 Nov 2021cs.CVcited by 353

Not on the map. Why?

Cited by 72 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Referring image segmentation aims to segment a referent via a natural linguistic expression.Due to the distinct data properties between text and image, it is challenging for a network to well align text and pixel-level features. Existing approaches use pretrained models to facilitate learning, yet separately transfer the language/vision knowledge from pretrained models, ignoring the multi-modal corresponding information. Inspired by the recent advance in Contrastive Language-Image Pretraining (CLIP), in this paper, we propose an end-to-end CLIP-Driven Referring Image Segmentation framework (CRIS). To transfer the multi-modal knowledge effectively, CRIS resorts to vision-language decoding and contrastive learning for achieving the text-to-pixel alignment. More specifically, we design a vision-language decoder to propagate fine-grained semantic information from textual representations to each pixel-level activation, which promotes consistency between the two modalities. In addition, we present text-to-pixel contrastive learning to explicitly enforce the text feature similar to the related pixel-level features and dissimilar to the irrelevances. The experimental results on three benchmark datasets demonstrate that our proposed framework significantly outperforms the state-of-the-art performance without any post-processing. The code will be released.

Key facts. Cited by 353 papers in the map (data through 14 Oct 2026). Builds on 39 of its 48 references that are in the map. First posted on arXiv 30 Nov 2021.

Builds on

39 of 48 references.

References · 39

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  7. 2020End-to-End Object Detection with Transformerscited by 7,414
  8. 2017Mask R-CNNcited by 7,277
  9. 2016Layer Normalizationcited by 5,669
  10. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,859
  11. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  12. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
27 more
  1. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
  2. 2018Dual Attention Network for Scene Segmentationcited by 1,003
  3. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 991
  4. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
  5. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 651
  6. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 647
  7. 2020Dense Contrastive Learning for Self-Supervised Visual Pre-Trainingcited by 512
  8. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 507
  9. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 431
  10. 2018An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solutioncited by 307
  11. 2019Cross-Modal Self-Attention Network for Referring Image Segmentationcited by 263
  12. 2016Segmentation from Natural Language Expressionscited by 257
  13. 2016Modeling Context Between Objects for Referring Expression Understandingcited by 251
  14. 2021Vision-Language Transformer and Query Generation for Referring Segmentationcited by 240
  15. 2021CLIP2Video: Mastering Video-Text Retrieval via Image CLIPcited by 225
  16. 2020Multi-task Collaborative Network for Joint Referring Expression Comprehension and Segmentationcited by 204
  17. 2020Unsupervised Learning of Dense Visual Representationscited by 147
  18. 2018Learning to Assemble Neural Module Tree Networks for Visual Groundingcited by 140
  19. 2020Referring Image Segmentation via Cross-Modal Progressive Comprehensioncited by 131
  20. 2017Recurrent Multimodal Interaction for Referring Image Segmentationcited by 125
  21. 2021Encoder Fusion Network with Co-Attention Embedding for Referring Image Segmentationcited by 111
  22. 2020Linguistic Structure Guided Context Modeling for Referring Image Segmentationcited by 102
  23. 2021Locate then Segment: A Strong Pipeline for Referring Image Segmentationcited by 98
  24. 2021CLIP4Caption: CLIP for Video Captioncited by 88
  25. 2018Dynamic Multimodal Instance Segmentation guided by natural language queriescited by 83
  26. 2019Referring Expression Object Segmentation with Caption-Aware Consistencycited by 46
  27. 2020GINet: Graph Interaction Network for Scene Parsingcited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 18; 2023: 117; 2024: 94; 2025: 70; 2026: 53.

353 papers.

Sort:

By area:

  1. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 776
  2. 2023LERF: Language Embedded Radiance Fieldscited by 543
  3. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 455
  4. 2022Decomposing NeRF for Editing via Feature Field Distillationcited by 371
  5. 2023Side Adapter Network for Open-Vocabulary Semantic Segmentationcited by 300
  6. 2023GRES: Generalized Referring Expression Segmentationcited by 254
  7. 2023MeViS: A Large-scale Benchmark for Video Segmentation with Motion Expressionscited by 228
  8. 2023CLIP-Driven Universal Model for Organ Segmentation and Tumor Detectioncited by 224
  9. 2023PixelLM: Pixel Reasoning with Large Multimodal Modelcited by 222
  10. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMscited by 218
  11. 2022ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentationcited by 178
  12. 2023GSVA: Generalized Segmentation via Multimodal Large Language Modelscited by 154
38 more
  1. 2022Diffusion-based Image Translation using Disentangled Style and Content Representationcited by 149
  2. 2023PolyFormer: Referring Image Segmentation as Sequential Polygon Generationcited by 148
  3. 2023Weakly Supervised 3D Open-vocabulary Segmentationcited by 135
  4. 2024VISA: Reasoning Video Object Segmentation via Large Language Modelscited by 127
  5. 2023Transformer-Based Visual Segmentation: A Surveycited by 124
  6. 2022Selective-Supervised Contrastive Learning with Noisy Labelscited by 112
  7. 2022VLT: Vision-Language Transformer and Query Generation for Referring Segmentationcited by 111
  8. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 110
  9. 2023Contextual Object Detection with Multimodal Large Language Modelscited by 100
  10. 2024One Token to Seg Them All: Language Instructed Reasoning Segmentation in Videoscited by 100
  11. 2024PromptAD: Learning Prompts with only Normal Samples for Few-Shot Anomaly Detectioncited by 98
  12. 2024FMGS: Foundation Model Embedded 3D Gaussian Splatting for Holistic 3D Scene Understandingcited by 86
  13. 2023Not All Features Matter: Enhancing Few-shot CLIP with Adaptive Prior Refinementcited by 84
  14. 2023CLIP-Count: Towards Text-Guided Zero-Shot Object Countingcited by 83
  15. 2025Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Groundingcited by 83
  16. 2023NExT-Chat: An LMM for Chat, Detection and Segmentationcited by 80
  17. 2022CLIP-Driven Fine-grained Text-Image Person Re-identificationcited by 79
  18. 2023Rotated Multi-Scale Interaction Network for Referring Remote Sensing Image Segmentationcited by 76
  19. 2025AA-CLIP: Enhancing Zero-shot Anomaly Detection via Anomaly-Aware CLIPcited by 73
  20. 2024GROUNDHOG: Grounding Large Language Models to Holistic Segmentationcited by 70
  21. 2024SAM4MLLM: Enhance Multi-Modal Large Language Model for Referring Expression Segmentationcited by 69
  22. 2023Beyond One-to-One: Rethinking the Referring Image Segmentationcited by 68
  23. 2024EVF-SAM: Early Vision-Language Fusion for Text-Prompted Segment Anything Modelcited by 65
  24. 2024Perceive-IR: Learning to Perceive Degradation Better for All-in-One Image Restorationcited by 62
  25. 2023Zero-shot Referring Image Segmentation with Global-Local Context Featurescited by 61
  26. 2023Bridging Vision and Language Encoders: Parameter-Efficient Tuning for Referring Image Segmentationcited by 61
  27. 2023RRSIS: Referring Remote Sensing Image Segmentationcited by 59
  28. 2023SOC: Semantic-Assisted Object Cluster for Referring Video Object Segmentationcited by 57
  29. 2024A Comprehensive Survey of Foundation Models in Medicinecited by 56
  30. 2023Open-VCLIP: Transforming CLIP to an Open-vocabulary Video Model via Interpolated Weight Optimizationcited by 55
  31. 2024Decoupling Static and Hierarchical Motion Perception for Referring Video Segmentationcited by 55
  32. 2024LaSagnA: Language-based Segmentation Assistant for Complex Queriescited by 55
  33. 2023Referred by Multi-Modality: A Unified Temporal Transformer for Video Object Segmentationcited by 54
  34. 2023Towards Label-free Scene Understanding by Vision Foundation Modelscited by 54
  35. 2023Hierarchical Open-vocabulary Universal Image Segmentationcited by 54
  36. 2023Contrastive Grouping with Transformer for Referring Image Segmentationcited by 54
  37. 2023Jack of All Tasks, Master of Many: Designing General-purpose Coarse-to-Fine Vision-Language Modelcited by 53
  38. 2024LLM-Seg: Bridging Image Segmentation and Large Language Model Reasoningcited by 52

50 most cited of 353 All 353 (JSON)