mlsanityData to 7 Oct 2026

Extending CLIP's Image-Text Alignment to Referring Image Segmentation

arXiv 2306.08498 ↗First posted 14 Jun 2023cs.CVcited by 19

Not on the map. Why?

Abstract (arXiv)

Referring Image Segmentation (RIS) is a cross-modal task that aims to segment an instance described by a natural language expression. Recent methods leverage large-scale pretrained unimodal models as backbones along with fusion techniques for joint reasoning across modalities. However, the inherent cross-modal nature of RIS raises questions about the effectiveness of unimodal backbones. We propose RISCLIP, a novel framework that effectively leverages the cross-modal nature of CLIP for RIS. Observing CLIP's inherent alignment between image and text features, we capitalize on this starting point and introduce simple but strong modules that enhance unimodal feature extraction and leverage rich alignment knowledge in CLIP's image-text shared-embedding space. RISCLIP exhibits outstanding results on all three major RIS benchmarks and also outperforms previous CLIP-based methods, demonstrating the efficacy of our strategy in extending CLIP's image-text alignment to RIS.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 54 of its 79 references that are in the map. First posted on arXiv 14 Jun 2023.

Builds on

54 of 79 references.

References · 54

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  8. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  10. 2016Layer Normalizationcited by 5,579
  11. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  12. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
38 more
  1. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  2. 2016Wide Residual Networkscited by 3,057
  3. 2018YOLOv3: An Incremental Improvementcited by 2,652
  4. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  5. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  6. 2016V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentationcited by 2,284
  7. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  8. 2016Modeling Context in Referring Expressionscited by 1,054
  9. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  10. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  11. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 714
  12. 2021ImageNet-21K Pretraining for the Massescited by 607
  13. 2021Extract Free Dense Labels from CLIPcited by 580
  14. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  15. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 427
  16. 2021CRIS: CLIP-Driven Referring Image Segmentationcited by 353
  17. 2021LAVT: Language-Aware Vision Transformer for Referring Image Segmentationcited by 347
  18. 2022Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Modelcited by 317
  19. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 304
  20. 2023Side Adapter Network for Open-Vocabulary Semantic Segmentationcited by 296
  21. 2019Cross-Modal Self-Attention Network for Referring Image Segmentationcited by 263
  22. 2016Segmentation from Natural Language Expressionscited by 257
  23. 2016Modeling Context Between Objects for Referring Expression Understandingcited by 250
  24. 2021Vision-Language Transformer and Query Generation for Referring Segmentationcited by 239
  25. 2020Multi-task Collaborative Network for Joint Referring Expression Comprehension and Segmentationcited by 204
  26. 2022SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic Segmentationcited by 162
  27. 2023PolyFormer: Referring Image Segmentation as Sequential Polygon Generationcited by 147
  28. 2022Bridging the Gap between Object and Image-level Representations for Open-Vocabulary Detectioncited by 145
  29. 2021Referring Transformer: A One-step Approach to Multi-task Visual Groundingcited by 143
  30. 2020Referring Image Segmentation via Cross-Modal Progressive Comprehensioncited by 131
  31. 2017Recurrent Multimodal Interaction for Referring Image Segmentationcited by 125
  32. 2021Encoder Fusion Network with Co-Attention Embedding for Referring Image Segmentationcited by 111
  33. 2022VLT: Vision-Language Transformer and Query Generation for Referring Segmentationcited by 110
  34. 2020Linguistic Structure Guided Context Modeling for Referring Image Segmentationcited by 102
  35. 2021Cross-Modal Progressive Comprehension for Referring Segmentationcited by 102
  36. 2021Locate then Segment: A Strong Pipeline for Referring Image Segmentationcited by 98
  37. 2018Dynamic Multimodal Instance Segmentation guided by natural language queriescited by 83
  38. 2023Zero-shot Referring Image Segmentation with Global-Local Context Featurescited by 61

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 6; 2026: 6.

19 papers.

Sort:

By area:

  1. 2025Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Groundingcited by 78
  2. 2024Toward Visual Grounding: A Surveycited by 52
  3. 2024Image Segmentation in Foundation Model Era: A Surveycited by 35
  4. 2024HiVG: Hierarchical Multimodal Fine-grained Modulation for Visual Groundingcited by 31
  5. 2024OneRef: Unified One-tower Expression Grounding and Segmentation with Mask Referring Modelingcited by 28
  6. 2025Multimodal Referring Segmentation: A Surveycited by 22
  7. 2024Pseudo-RIS: Distinctive Pseudo-supervision Generation for Referring Image Segmentationcited by 13
  8. 2024CoHD: A Counting-Aware Hierarchical Decoding Framework for Generalized Referring Expression Segmentationcited by 10
  9. 2025Latent Expression Generation for Referring Image Segmentation and Groundingcited by 6
  10. 2025DGTRSD & DGTRS-CLIP: A Dual-Granularity Remote Sensing Image-Text Dataset and Vision Language Foundation Model for Alignmentcited by 4
  11. 2025SaFiRe: Saccade-Fixation Reiteration with Mamba for Referring Image Segmentationcited by 3
  12. 2026GenMask: Adapting DiT for Segmentation via Direct Mask Generationcited by 3
7 more
  1. 2024A Simple Baseline with Single-encoder for Referring Image Segmentationcited by 2
  2. 2026Guiding Diffusion-based Reconstruction with Contrastive Signals for Balanced Visual Representationcited by 2
  3. 2026SSP-SAM: SAM with Semantic-Spatial Prompt for Referring Expression Segmentationcited by 2
  4. 2025Holistic Order Prediction in Natural Scenescited by 1
  5. 2026Spatio-Semantic Expert Routing Architecture with Mixture-of-Experts for Referring Image Segmentationcited by 0
  6. 2026DiCoR: Decoupled Referent Disambiguation and Contour Recalibration for Efficient Referring Remote Sensing Image Segmentationcited by 0
  7. 2026DeCo: Efficient Decouple-to-Couple Learning for Multi-Task Visual Groundingcited by 0