mlsanityData to 7 Oct 2026

Zero-shot Referring Image Segmentation with Global-Local Context Features

arXiv 2303.17811 ↗First posted 31 Mar 2023cs.CV cs.AI cs.CLcited by 61

Not on the map. Why?

Abstract (arXiv)

Referring image segmentation (RIS) aims to find a segmentation mask given a referring expression grounded to a region of the input image. Collecting labelled datasets for this task, however, is notoriously costly and labor-intensive. To overcome this issue, we propose a simple yet effective zero-shot referring image segmentation method by leveraging the pre-trained cross-modal knowledge from CLIP. In order to obtain segmentation masks grounded to the input text, we propose a mask-guided visual encoder that captures global and local contextual information of an input image. By utilizing instance masks obtained from off-the-shelf mask proposal techniques, our method is able to segment fine-detailed Istance-level groundings. We also introduce a global-local text encoder where the global feature captures complex sentence-level semantics of the entire input expression while the local feature focuses on the target noun phrase extracted by a dependency parser. In our experiments, the proposed method outperforms several zero-shot baselines of the task and even the weakly supervised referring expression segmentation method with substantial margins. Our code is available at https://github.com/Seonghoon-Yu/Zero-shot-RIS.

Key facts. Cited by 61 papers in the map (data through 7 Oct 2026). Builds on 56 of its 66 references that are in the map. First posted on arXiv 31 Mar 2023.

Builds on

56 of 66 references.

References · 56

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  7. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  9. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  10. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  11. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  12. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
38 more
  1. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  2. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  3. 2021Extract Free Dense Labels from CLIPcited by 580
  4. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 560
  5. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  6. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 530
  7. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 471
  8. 2021CRIS: CLIP-Driven Referring Image Segmentationcited by 353
  9. 2021LAVT: Language-Aware Vision Transformer for Referring Image Segmentationcited by 347
  10. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  11. 2021ActionCLIP: A New Paradigm for Video Action Recognitioncited by 317
  12. 2022Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Modelcited by 317
  13. 2023Side Adapter Network for Open-Vocabulary Semantic Segmentationcited by 296
  14. 2019Cross-Modal Self-Attention Network for Referring Image Segmentationcited by 263
  15. 2016Segmentation from Natural Language Expressionscited by 257
  16. 2016Modeling Context Between Objects for Referring Expression Understandingcited by 250
  17. 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239
  18. 2021Vision-Language Transformer and Query Generation for Referring Segmentationcited by 239
  19. 2021A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Modelcited by 223
  20. 2021Decoupling Zero-Shot Semantic Segmentationcited by 198
  21. 2023Cut and Learn for Unsupervised Object Detection and Instance Segmentationcited by 186
  22. 2022ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentationcited by 178
  23. 2022SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic Segmentationcited by 162
  24. 2022F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Modelscited by 149
  25. 2023PolyFormer: Referring Image Segmentation as Sequential Polygon Generationcited by 147
  26. 2022Bridging the Gap between Object and Image-level Representations for Open-Vocabulary Detectioncited by 145
  27. 2017Recurrent Multimodal Interaction for Referring Image Segmentationcited by 125
  28. 2022ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehensioncited by 119
  29. 2020PhraseCut: Language-based Image Segmentation in the Wildcited by 116
  30. 2022ReSTR: Convolution-free Referring Image Segmentation Using Transformerscited by 112
  31. 2021Encoder Fusion Network with Co-Attention Embedding for Referring Image Segmentationcited by 111
  32. 2022FreeSOLO: Learning to Segment Objects without Annotationscited by 102
  33. 2021Locate then Segment: A Strong Pipeline for Referring Image Segmentationcited by 98
  34. 2022Learning Object-Language Alignments for Open-Vocabulary Object Detectioncited by 95
  35. 2021Contrastive Embedding for Generalized Zero-Shot Learningcited by 94
  36. 2021TransZero: Attribute-guided Transformer for Zero-Shot Learningcited by 53
  37. 2022Towards Robust Referring Image Segmentationcited by 49
  38. 2022CoupAlign: Coupling Word-Pixel with Sentence-Mask Alignments for Referring Image Segmentationcited by 28

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 11; 2024: 16; 2025: 20; 2026: 14.

61 papers.

Sort:

By area:

  1. 2023Grounding Everything: Emerging Localization Properties in Vision-Language Transformerscited by 89
  2. 2025Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Groundingcited by 78
  3. 2023CLIP as RNN: Segment Countless Visual Concepts without Training Endeavorcited by 69
  4. 2024Decoupling Static and Hierarchical Motion Perception for Referring Video Segmentationcited by 55
  5. 2024Diffusion Feedback Helps CLIP See Bettercited by 37
  6. 2024Image Segmentation in Foundation Model Era: A Surveycited by 35
  7. 2023Ref-Diff: Zero-shot Referring Image Segmentation with Generative Modelscited by 33
  8. 2023Text Promptable Surgical Instrument Segmentation with Vision-Language Modelscited by 27
  9. 2024Learning Visual Grounding from Generative Vision and Language Modelcited by 27
  10. 2025IteRPrimE: Zero-shot Referring Image Segmentation with Iterative Grad-CAM Refinement and Primary Word Emphasiscited by 26
  11. 2025Multimodal Referring Segmentation: A Surveycited by 22
  12. 2023Text Augmented Spatial-aware Zero-shot Referring Image Segmentationcited by 21
38 more
  1. 2023Zero-shot Referring Expression Comprehension via Structural Similarity Between Images and Captionscited by 21
  2. 2023Unveiling Parts Beyond Objects:Towards Finer-Granularity Referring Expression Segmentationcited by 21
  3. 2023Extending CLIP's Image-Text Alignment to Referring Image Segmentationcited by 19
  4. 2023VGDiffZero: Text-to-image Diffusion Models Can Be Zero-shot Visual Grounderscited by 18
  5. 2025Hybrid Global-Local Representation with Augmented Spatial Guidance for Zero-Shot Referring Image Segmentationcited by 14
  6. 2024Exploring Phrase-Level Grounding with Text-to-Image Diffusion Modelcited by 13
  7. 2024Pseudo-RIS: Distinctive Pseudo-supervision Generation for Referring Image Segmentationcited by 13
  8. 2024INTRA: Interaction Relationship-aware Weakly Supervised Affordance Groundingcited by 12
  9. 2024RESMatch: Referring Expression Segmentation in a Semi-Supervised Mannercited by 11
  10. 2024VLSM-Adapter: Finetuning Vision-Language Segmentation Efficiently with Lightweight Blockscited by 11
  11. 2025AdvPaint: Protecting Images from Inpainting Manipulation via Adversarial Attention Disruptioncited by 11
  12. 2024Toward Interactive Regional Understanding in Vision-Large Language Modelscited by 9
  13. 2024Emergent Visual Grounding in Large Multimodal Models Without Grounding Supervisioncited by 8
  14. 2024Addressing Text Embedding Leakage in Diffusion-based Image Editingcited by 7
  15. 2025Latent Expression Generation for Referring Image Segmentation and Groundingcited by 6
  16. 2025Target Refocusing via Attention Redistribution for Open-Vocabulary Semantic Segmentation: An Explainability Perspectivecited by 5
  17. 2025SketchFusion: Learning Universal Sketch Features through Fusing Foundation Modelscited by 4
  18. 2025LGD: Leveraging Generative Descriptions for Zero-Shot Referring Image Segmentationcited by 4
  19. 2025RESAnything: Attribute Prompting for Arbitrary Referring Segmentationcited by 4
  20. 2025Segment Anyword: Mask Prompt Inversion for Open-Set Grounded Segmentationcited by 4
  21. 2025Evolving, Not Training: Zero-Shot Reasoning Segmentation via Evolutionary Promptingcited by 4
  22. 2023Segment, Select, Correct: A Framework for Weakly-Supervised Referring Segmentationcited by 3
  23. 2024PVLR: Prompt-driven Visual-Linguistic Representation Learning for Multi-Label Image Recognitioncited by 2
  24. 2024MaskInversion: Localized Embeddings via Optimization of Explainability Mapscited by 2
  25. 2024A Simple Baseline with Single-encoder for Referring Image Segmentationcited by 2
  26. 2025Toward Rich Video Human-Motion2D Generationcited by 2
  27. 2025Novel Category Discovery with X-Agent Attention for Open-Vocabulary Semantic Segmentationcited by 2
  28. 2025RefAM: Attention Magnets for Zero-Shot Referral Segmentationcited by 2
  29. 2024Spatial Semantic Recurrent Mining for Referring Image Segmentationcited by 1
  30. 2025Towards Agentic AI for Multimodal-Guided Video Object Segmentationcited by 1
  31. 2025Improvise, Adapt, Overcome -- Telescopic Adapters for Efficient Fine-tuning of Vision Language Models in Medical Imagingcited by 1
  32. 2026DISC: Dense Integrated Semantic Context for Large-Scale Open-Set Semantic Mappingcited by 1
  33. 2026RealVLG-R1: A Large-Scale Real-World Visual-Language Grounding Benchmark for Robotic Perception and Manipulationcited by 1
  34. 2026Early Semantic Grounding in Image Editing Models for Zero-Shot Referring Image Segmentationcited by 1
  35. 2023Towards Generalizable Referring Image Segmentation via Target Prompt and Visual Coherencecited by 0
  36. 2025Image Embedding Sampling Method for Diverse Captioningcited by 0
  37. 2025DGL-RSIS: Decoupling Global Spatial Context and Local Class Semantics for Training-Free Remote Sensing Image Segmentationcited by 0
  38. 2025Temporal Prompting Matters: Rethinking Referring Video Object Segmentationcited by 0

50 most cited of 61 All 61 (JSON)