# Locate then Segment: A Strong Pipeline for Referring Image Segmentation

arXiv: https://arxiv.org/abs/2103.16284 · HTML: https://mlsanity.com/paper/2103.16284/

## Key facts

- Cited by 98 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 38 references that are in the map.
- First posted on arXiv 30 Mar 2021.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation](https://mlsanity.com/paper/1511.00561/) (2015)
- [Generation and Comprehension of Unambiguous Object Descriptions](https://mlsanity.com/paper/1511.02283/) (2015)
- [Fully-Convolutional Siamese Networks for Object Tracking](https://mlsanity.com/paper/1606.09549/) (2016)
- [RefineNet: Multi-Path Refinement Networks for High-Resolution Semantic Segmentation](https://mlsanity.com/paper/1611.06612/) (2016)
- [PointRend: Image Segmentation as Rendering](https://mlsanity.com/paper/1912.08193/) (2019)
- [MAttNet: Modular Attention Network for Referring Expression Comprehension](https://mlsanity.com/paper/1801.08186/) (2018)
- [Residual Attention Network for Image Classification](https://mlsanity.com/paper/1704.06904/) (2017)
- [Attention Augmented Convolutional Networks](https://mlsanity.com/paper/1904.09925/) (2019)
- [SOLO: Segmenting Objects by Locations](https://mlsanity.com/paper/1912.04488/) (2019)
- [Cross-Modal Self-Attention Network for Referring Image Segmentation](https://mlsanity.com/paper/1904.04745/) (2019)
- [Segmentation from Natural Language Expressions](https://mlsanity.com/paper/1603.06180/) (2016)
- [Modeling Context Between Objects for Referring Expression Understanding](https://mlsanity.com/paper/1608.00525/) (2016)
- [Multi-task Collaborative Network for Joint Referring Expression Comprehension and Segmentation](https://mlsanity.com/paper/2003.08813/) (2020)
- [Learning to Assemble Neural Module Tree Networks for Visual Grounding](https://mlsanity.com/paper/1812.03299/) (2018)
- [Referring Image Segmentation via Cross-Modal Progressive Comprehension](https://mlsanity.com/paper/2010.00514/) (2020)
- [Recurrent Multimodal Interaction for Referring Image Segmentation](https://mlsanity.com/paper/1703.07939/) (2017)
- [Modeling Relationships in Referential Expressions with Compositional Modular Networks](https://mlsanity.com/paper/1611.09978/) (2016)
- [Linguistic Structure Guided Context Modeling for Referring Image Segmentation](https://mlsanity.com/paper/2010.00515/) (2020)
- [Dynamic Multimodal Instance Segmentation guided by natural language queries](https://mlsanity.com/paper/1807.02257/) (2018)
- [Referring Expression Object Segmentation with Caption-Aware Consistency](https://mlsanity.com/paper/1910.04748/) (2019)

## Built on by (most cited first, 30 of 98)

- [CRIS: CLIP-Driven Referring Image Segmentation](https://mlsanity.com/paper/2111.15174/) (2021)
- [LAVT: Language-Aware Vision Transformer for Referring Image Segmentation](https://mlsanity.com/paper/2112.02244/) (2021)
- [Unleashing Text-to-Image Diffusion Models for Visual Perception](https://mlsanity.com/paper/2303.02153/) (2023)
- [GRES: Generalized Referring Expression Segmentation](https://mlsanity.com/paper/2306.00968/) (2023)
- [MeViS: A Large-scale Benchmark for Video Segmentation with Motion Expressions](https://mlsanity.com/paper/2308.08544/) (2023)
- [Universal Instance Perception as Object Discovery and Retrieval](https://mlsanity.com/paper/2303.06674/) (2023)
- [GSVA: Generalized Segmentation via Multimodal Large Language Models](https://mlsanity.com/paper/2312.10103/) (2023)
- [PolyFormer: Referring Image Segmentation as Sequential Polygon Generation](https://mlsanity.com/paper/2302.07387/) (2023)
- [Referring Transformer: A One-step Approach to Multi-task Visual Grounding](https://mlsanity.com/paper/2106.03089/) (2021)
- [SeqTR: A Simple yet Universal Network for Visual Grounding](https://mlsanity.com/paper/2203.16265/) (2022)
- [ReSTR: Convolution-free Referring Image Segmentation Using Transformers](https://mlsanity.com/paper/2203.16768/) (2022)
- [VLT: Vision-Language Transformer and Query Generation for Referring Segmentation](https://mlsanity.com/paper/2210.15871/) (2022)
- [Rotated Multi-Scale Interaction Network for Referring Remote Sensing Image Segmentation](https://mlsanity.com/paper/2312.12470/) (2023)
- [SAM4MLLM: Enhance Multi-Modal Large Language Model for Referring Expression Segmentation](https://mlsanity.com/paper/2409.10542/) (2024)
- [Beyond One-to-One: Rethinking the Referring Image Segmentation](https://mlsanity.com/paper/2308.13853/) (2023)
- [Zero-shot Referring Image Segmentation with Global-Local Context Features](https://mlsanity.com/paper/2303.17811/) (2023)
- [RRSIS: Referring Remote Sensing Image Segmentation](https://mlsanity.com/paper/2306.08625/) (2023)
- [LaSagnA: Language-based Segmentation Assistant for Complex Queries](https://mlsanity.com/paper/2404.08506/) (2024)
- [Hierarchical Open-vocabulary Universal Image Segmentation](https://mlsanity.com/paper/2307.00764/) (2023)
- [Contrastive Grouping with Transformer for Referring Image Segmentation](https://mlsanity.com/paper/2309.01017/) (2023)
- [Jack of All Tasks, Master of Many: Designing General-purpose Coarse-to-Fine Vision-Language Model](https://mlsanity.com/paper/2312.12423/) (2023)
- [Language-Bridged Spatial-Temporal Interaction for Referring Video Object Segmentation](https://mlsanity.com/paper/2206.03789/) (2022)
- [Towards Robust Referring Image Segmentation](https://mlsanity.com/paper/2209.09554/) (2022)
- [ClipSAM: CLIP and SAM Collaboration for Zero-Shot Anomaly Segmentation](https://mlsanity.com/paper/2401.12665/) (2024)
- [ReMamber: Referring Image Segmentation with Mamba Twister](https://mlsanity.com/paper/2403.17839/) (2024)
- [Multi-Modal Mutual Attention and Iterative Interaction for Referring Image Segmentation](https://mlsanity.com/paper/2305.15302/) (2023)
- [Language Adaptive Weight Generation for Multi-task Visual Grounding](https://mlsanity.com/paper/2306.04652/) (2023)
- [Referring Image Segmentation Using Text Supervision](https://mlsanity.com/paper/2308.14575/) (2023)
- [MeViS: A Multi-Modal Dataset for Referring Motion Expression Video Segmentation](https://mlsanity.com/paper/2512.10945/) (2025)
- [Universal Segmentation at Arbitrary Granularity with Language Instruction](https://mlsanity.com/paper/2312.01623/) (2023)

## Abstract (arXiv)

Referring image segmentation aims to segment the objects referred by a natural language expression. Previous methods usually focus on designing an implicit and recurrent feature interaction mechanism to fuse the visual-linguistic features to directly generate the final segmentation mask without explicitly modeling the localization information of the referent instances. To tackle these problems, we view this task from another perspective by decoupling it into a "Locate-Then-Segment" (LTS) scheme. Given a language expression, people generally first perform attention to the corresponding target image regions, then generate a fine segmentation mask about the object based on its context. The LTS first extracts and fuses both visual and textual features to get a cross-modal representation, then applies a cross-model interaction on the visual-textual features to locate the referred object with position prior, and finally generates the segmentation result with a light-weight segmentation network. Our LTS is simple but surprisingly effective. On three popular benchmark datasets, the LTS outperforms all the previous state-of-the-art methods by a large margin (e.g., +3.2% on RefCOCO+ and +3.4% on RefCOCOg). In addition, our model is more interpretable with explicitly locating the object, which is also proved by visualization experiments. We believe this framework is promising to serve as a strong baseline for referring image segmentation.