# Referring Image Segmentation Using Text Supervision

arXiv: https://arxiv.org/abs/2308.14575 · HTML: https://mlsanity.com/paper/2308.14575/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 64 of its 81 references that are in the map.
- First posted on arXiv 28 Aug 2023.

## Builds on (most cited first, 30 of 64)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Conditional Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2203.05557/) (2022)
- [Modeling Context in Referring Expressions](https://mlsanity.com/paper/1608.00272/) (2016)
- [CLIP-Adapter: Better Vision-Language Models with Feature Adapters](https://mlsanity.com/paper/2110.04544/) (2021)
- [Generation and Comprehension of Unambiguous Object Descriptions](https://mlsanity.com/paper/1511.02283/) (2015)
- [Focal Loss for Dense Object Detection](https://mlsanity.com/paper/1708.02002/) (2017)
- [Transformer Interpretability Beyond Attention Visualization](https://mlsanity.com/paper/2012.09838/) (2020)
- [Extract Free Dense Labels from CLIP](https://mlsanity.com/paper/2112.01071/) (2021)
- [DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting](https://mlsanity.com/paper/2112.01518/) (2021)
- [GroupViT: Semantic Segmentation Emerges from Text Supervision](https://mlsanity.com/paper/2202.11094/) (2022)
- [MAttNet: Modular Attention Network for Referring Expression Comprehension](https://mlsanity.com/paper/1801.08186/) (2018)
- [CRIS: CLIP-Driven Referring Image Segmentation](https://mlsanity.com/paper/2111.15174/) (2021)
- [LAVT: Language-Aware Vision Transformer for Referring Image Segmentation](https://mlsanity.com/paper/2112.02244/) (2021)
- [Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation](https://mlsanity.com/paper/1811.10092/) (2018)
- [Cross-Modal Self-Attention Network for Referring Image Segmentation](https://mlsanity.com/paper/1904.04745/) (2019)
- [Segmentation from Natural Language Expressions](https://mlsanity.com/paper/1603.06180/) (2016)
- [Modeling Context Between Objects for Referring Expression Understanding](https://mlsanity.com/paper/1608.00525/) (2016)
- [Weakly Supervised Learning of Instance Segmentation with Inter-pixel Relations](https://mlsanity.com/paper/1904.05044/) (2019)
- [Vision-Language Transformer and Query Generation for Referring Segmentation](https://mlsanity.com/paper/2108.05565/) (2021)
- [BoxSup: Exploiting Bounding Boxes to Supervise Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1503.01640/) (2015)
- [Multi-task Collaborative Network for Joint Referring Expression Comprehension and Segmentation](https://mlsanity.com/paper/2003.08813/) (2020)
- [Single-Stage Semantic Segmentation from Image Labels](https://mlsanity.com/paper/2005.08104/) (2020)

## Built on by (most cited first, 30 of 35)

- [LLaFS: When Large Language Models Meet Few-Shot Segmentation](https://mlsanity.com/paper/2311.16926/) (2023)
- [Image Segmentation in Foundation Model Era: A Survey](https://mlsanity.com/paper/2408.12957/) (2024)
- [Harnessing Earnings Reports for Stock Predictions: A QLoRA-Enhanced LLM Approach](https://mlsanity.com/paper/2408.06634/) (2024)
- [Recasting Regional Lighting for Shadow Removal](https://mlsanity.com/paper/2402.00341/) (2024)
- [Curriculum Point Prompting for Weakly-Supervised Referring Image Segmentation](https://mlsanity.com/paper/2404.11998/) (2024)
- [Multimodal Referring Segmentation: A Survey](https://mlsanity.com/paper/2508.00265/) (2025)
- [Pseudo-RIS: Distinctive Pseudo-supervision Generation for Referring Image Segmentation](https://mlsanity.com/paper/2407.07412/) (2024)
- [Compositional Image Decomposition with Diffusion Models](https://mlsanity.com/paper/2406.19298/) (2024)
- [Intra- and Cross-Scale Feature Interaction Network for Referring Remote Sensing Image Segmentation](https://mlsanity.com/paper/2506.03583/) (2025)
- [RESMatch: Referring Expression Segmentation in a Semi-Supervised Manner](https://mlsanity.com/paper/2402.05589/) (2024)
- [RG-SAN: Rule-Guided Spatial Awareness Network for End-to-End 3D Referring Expression Segmentation](https://mlsanity.com/paper/2412.02402/) (2024)
- [Boosting Weakly-Supervised Referring Image Segmentation via Progressive Comprehension](https://mlsanity.com/paper/2410.01544/) (2024)
- [Toward Interactive Regional Understanding in Vision-Large Language Models](https://mlsanity.com/paper/2403.18260/) (2024)
- [WeakMCN: Multi-task Collaborative Network for Weakly Supervised Referring Expression Comprehension and Segmentation](https://mlsanity.com/paper/2505.18686/) (2025)
- [Unleashing the Potential of Multimodal LLMs for Zero-Shot Spatio-Temporal Video Grounding](https://mlsanity.com/paper/2509.15178/) (2025)
- [HiFi-CS: Towards Open Vocabulary Visual Grounding For Robotic Grasping Using Vision-Language Models](https://mlsanity.com/paper/2409.10419/) (2024)
- [Cognition Transferring and Decoupling for Text-supervised Egocentric Semantic Segmentation](https://mlsanity.com/paper/2410.01341/) (2024)
- [DSV-LFS: Unifying LLM-Driven Semantic Cues with Visual Features for Robust Few-Shot Segmentation](https://mlsanity.com/paper/2503.04006/) (2025)
- [Segmentation as A Plug-and-Play Capability for Frozen Multimodal LLMs](https://mlsanity.com/paper/2510.16785/) (2025)
- [Segment, Select, Correct: A Framework for Weakly-Supervised Referring Segmentation](https://mlsanity.com/paper/2310.13479/) (2023)
- [MIP-GAF: A MLLM-annotated Benchmark for Most Important Person Localization and Group Context Understanding](https://mlsanity.com/paper/2409.06224/) (2024)
- [MaskRIS: Semantic Distortion-aware Data Augmentation for Referring Image Segmentation](https://mlsanity.com/paper/2411.19067/) (2024)
- [RefAM: Attention Magnets for Zero-Shot Referral Segmentation](https://mlsanity.com/paper/2509.22650/) (2025)
- [Think Before You Drive: World Model-Inspired Multimodal Grounding for Autonomous Vehicles](https://mlsanity.com/paper/2512.03454/) (2025)
- [Spatial Semantic Recurrent Mining for Referring Image Segmentation](https://mlsanity.com/paper/2405.09006/) (2024)
- [EmoSEM: Segment and Explain Emotion Stimuli in Visual Art](https://mlsanity.com/paper/2504.14658/) (2025)
- [Deformable Attentive Visual Enhancement for Referring Segmentation Using Vision-Language Model](https://mlsanity.com/paper/2505.19242/) (2025)
- [Prototypical Progressive Alignment and Reweighting for Generalizable Semantic Segmentation](https://mlsanity.com/paper/2507.11955/) (2025)
- [GuideMe: Multi-Domain Task Guidance and Intervention in Streaming Video](https://mlsanity.com/paper/2607.02991/) (2026)
- [The ART of Composition: Attention-Regularized Training for Compositional Visual Grounding](https://mlsanity.com/paper/2412.08110/) (2024)

## Abstract (arXiv)

Existing Referring Image Segmentation (RIS) methods typically require expensive pixel-level or box-level annotations for supervision. In this paper, we observe that the referring texts used in RIS already provide sufficient information to localize the target object. Hence, we propose a novel weakly-supervised RIS framework to formulate the target localization problem as a classification process to differentiate between positive and negative text expressions. While the referring text expressions for an image are used as positive expressions, the referring text expressions from other images can be used as negative expressions for this image. Our framework has three main novelties. First, we propose a bilateral prompt method to facilitate the classification process, by harmonizing the domain discrepancy between visual and linguistic features. Second, we propose a calibration method to reduce noisy background information and improve the correctness of the response maps for target object localization. Third, we propose a positive response map selection strategy to generate high-quality pseudo-labels from the enhanced response maps, for training a segmentation network for RIS inference. For evaluation, we propose a new metric to measure localization accuracy. Experiments on four benchmarks show that our framework achieves promising performances to existing fully-supervised RIS methods while outperforming state-of-the-art weakly-supervised methods adapted from related areas. Code is available at https://github.com/fawnliu/TRIS.