# IteRPrimE: Zero-shot Referring Image Segmentation with Iterative Grad-CAM Refinement and Primary Word Emphasis

arXiv: https://arxiv.org/abs/2503.00936 · HTML: https://mlsanity.com/paper/2503.00936/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 51 references that are in the map.
- First posted on arXiv 2 Mar 2025.

## Builds on (most cited first, 30 of 35)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [Generation and Comprehension of Unambiguous Object Descriptions](https://mlsanity.com/paper/1511.02283/) (2015)
- [Extract Free Dense Labels from CLIP](https://mlsanity.com/paper/2112.01071/) (2021)
- [Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP](https://mlsanity.com/paper/2210.04150/) (2022)
- [GroupViT: Semantic Segmentation Emerges from Text Supervision](https://mlsanity.com/paper/2202.11094/) (2022)
- [CRIS: CLIP-Driven Referring Image Segmentation](https://mlsanity.com/paper/2111.15174/) (2021)
- [LAVT: Language-Aware Vision Transformer for Referring Image Segmentation](https://mlsanity.com/paper/2112.02244/) (2021)
- [Modeling Context Between Objects for Referring Expression Understanding](https://mlsanity.com/paper/1608.00525/) (2016)
- [Vision-Language Transformer and Query Generation for Referring Segmentation](https://mlsanity.com/paper/2108.05565/) (2021)
- [MeViS: A Large-scale Benchmark for Video Segmentation with Motion Expressions](https://mlsanity.com/paper/2308.08544/) (2023)
- [mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connections](https://mlsanity.com/paper/2205.12005/) (2022)
- [mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Video](https://mlsanity.com/paper/2302.00402/) (2023)
- [ReCo: Retrieve and Co-segment for Zero-shot Transfer](https://mlsanity.com/paper/2206.07045/) (2022)
- [PhraseCut: Language-based Image Segmentation in the Wild](https://mlsanity.com/paper/2008.01187/) (2020)
- [ReSTR: Convolution-free Referring Image Segmentation Using Transformers](https://mlsanity.com/paper/2203.16768/) (2022)
- [FreeSOLO: Learning to Segment Objects without Annotations](https://mlsanity.com/paper/2202.12181/) (2022)
- [Locate then Segment: A Strong Pipeline for Referring Image Segmentation](https://mlsanity.com/paper/2103.16284/) (2021)
- [CLEVR-Ref+: Diagnosing Visual Reasoning with Referring Expressions](https://mlsanity.com/paper/1901.00850/) (2019)
- [CLIP as RNN: Segment Countless Visual Concepts without Training Endeavor](https://mlsanity.com/paper/2312.07661/) (2023)
- [Zero-shot Referring Image Segmentation with Global-Local Context Features](https://mlsanity.com/paper/2303.17811/) (2023)
- [BridgeTower: Building Bridges Between Encoders in Vision-Language Representation Learning](https://mlsanity.com/paper/2206.08657/) (2022)
- [Open-Vocabulary Segmentation with Semantic-Assisted Calibration](https://mlsanity.com/paper/2312.04089/) (2023)
- [Global Knowledge Calibration for Fast Open-Vocabulary Segmentation](https://mlsanity.com/paper/2303.09181/) (2023)
- [Ref-Diff: Zero-shot Referring Image Segmentation with Generative Models](https://mlsanity.com/paper/2308.16777/) (2023)

## Built on by (most cited first, 26 of 26)

- [Flash-VStream: Efficient Real-Time Understanding for Long Video Streams](https://mlsanity.com/paper/2506.23825/) (2025)
- [Multimodal Referring Segmentation: A Survey](https://mlsanity.com/paper/2508.00265/) (2025)
- [SAM2-LOVE: Segment Anything Model 2 in Language-aided Audio-Visual Scenes](https://mlsanity.com/paper/2506.01558/) (2025)
- [Intra- and Cross-Scale Feature Interaction Network for Referring Remote Sensing Image Segmentation](https://mlsanity.com/paper/2506.03583/) (2025)
- [VG-Refiner: Towards Tool-Refined Referring Grounded Reasoning via Agentic Reinforcement Learning](https://mlsanity.com/paper/2512.06373/) (2025)
- [Stepping Out of Similar Semantic Space for Open-Vocabulary Segmentation](https://mlsanity.com/paper/2506.16058/) (2025)
- [Evolving, Not Training: Zero-Shot Reasoning Segmentation via Evolutionary Prompting](https://mlsanity.com/paper/2512.24702/) (2025)
- [Learning Cross-View Object Correspondence via Cycle-Consistent Mask Prediction](https://mlsanity.com/paper/2602.18996/) (2026)
- [RefAM: Attention Magnets for Zero-Shot Referral Segmentation](https://mlsanity.com/paper/2509.22650/) (2025)
- [DiSa: Saliency-Aware Foreground-Background Disentangled Framework for Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2601.20064/) (2026)
- [Segment Anything with Motion, Geometry, and Semantic Adaptation for Complex Nonlinear Visual Object Tracking](https://mlsanity.com/paper/2605.22538/) (2026)
- [Uncertainty-Aware World Model for Aerial Image-Goal Navigation](https://mlsanity.com/paper/2608.05597/) (2026)
- [RIS-LAD: A Benchmark and Model for Referring Low-Altitude Drone Image Segmentation](https://mlsanity.com/paper/2507.20920/) (2025)
- [Layover or Direct Flight: Rethinking Audio-Guided Image Segmentation](https://mlsanity.com/paper/2511.22025/) (2025)
- [3D-DRES: Detailed 3D Referring Expression Segmentation](https://mlsanity.com/paper/2603.02896/) (2026)
- [FCL-COD: Weakly Supervised Camouflaged Object Detection with Frequency-aware and Contrastive Learning](https://mlsanity.com/paper/2603.22969/) (2026)
- [Tarot-SAM3: Training-free SAM3 for Any Referring Expression Segmentation](https://mlsanity.com/paper/2604.07916/) (2026)
- [Early Semantic Grounding in Image Editing Models for Zero-Shot Referring Image Segmentation](https://mlsanity.com/paper/2605.13122/) (2026)
- [Unlocking Diffusion Hierarchies: Adaptive Timestep Selection for Zero-Shot Segmentation](https://mlsanity.com/paper/2606.15590/) (2026)
- [DGL-RSIS: Decoupling Global Spatial Context and Local Class Semantics for Training-Free Remote Sensing Image Segmentation](https://mlsanity.com/paper/2509.00598/) (2025)
- [Heuristic-inspired Reasoning Priors Facilitate Data-Efficient Referring Object Detection](https://mlsanity.com/paper/2603.24166/) (2026)
- [Channel Attention-Guided Cross-Modal Knowledge Distillation for Referring Image Segmentation](https://mlsanity.com/paper/2604.16806/) (2026)
- [PinPoint: Prompting with Informative Interior Points](https://mlsanity.com/paper/2605.26689/) (2026)
- [Learning to Label: A Reinforced Self-Evolving Framework for Semi-supervised Referring Expression Segmentation](https://mlsanity.com/paper/2605.28239/) (2026)
- [MG-RWKV: Multi-Grained Context-Aware RWKV for Temporal Forgery Localization](https://mlsanity.com/paper/2607.00902/) (2026)
- [Grad-CAM for Vision Transformers: A Systematic Taxonomy and Audit of Methodological Ambiguity in Explainable AI](https://mlsanity.com/paper/2608.05258/) (2026)

## Abstract (arXiv)

Zero-shot Referring Image Segmentation (RIS) identifies the instance mask that best aligns with a specified referring expression without training and fine-tuning, significantly reducing the labor-intensive annotation process. Despite achieving commendable results, previous CLIP-based models have a critical drawback: the models exhibit a notable reduction in their capacity to discern relative spatial relationships of objects. This is because they generate all possible masks on an image and evaluate each masked region for similarity to the given expression, often resulting in decreased sensitivity to direct positional clues in text inputs. Moreover, most methods have weak abilities to manage relationships between primary words and their contexts, causing confusion and reduced accuracy in identifying the correct target region. To address these challenges, we propose IteRPrimE (Iterative Grad-CAM Refinement and Primary word Emphasis), which leverages a saliency heatmap through Grad-CAM from a Vision-Language Pre-trained (VLP) model for image-text matching. An iterative Grad-CAM refinement strategy is introduced to progressively enhance the model's focus on the target region and overcome positional insensitivity, creating a self-correcting effect. Additionally, we design the Primary Word Emphasis module to help the model handle complex semantic relations, enhancing its ability to attend to the intended object. Extensive experiments conducted on the RefCOCO/+/g, and PhraseCut benchmarks demonstrate that IteRPrimE outperforms previous state-of-the-art zero-shot methods, particularly excelling in out-of-domain scenarios.