Data to 7 Oct 2026

Segmentation from Natural Language Expressions

arXiv 1603.06180 ↗First posted 20 Mar 2016cs.CVcited by 257

Not on the map. Why?

Cited by 22 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this paper we approach the novel problem of segmenting an image based on a natural language expression. This is different from traditional semantic segmentation over a predefined set of semantic classes, as e.g., the phrase "two men sitting on the right bench" requires segmenting only the two people on the right bench and no one standing or sitting on another bench. Previous approaches suitable for this task were limited to a fixed set of categories and/or rectangular regions. To produce pixelwise segmentation for the language expression, we propose an end-to-end trainable recurrent and convolutional network model that jointly learns to process visual and linguistic information. In our model, a recurrent LSTM network is used to encode the referential expression into a vector representation, and a fully convolutional network is used to a extract a spatial feature map from the image and output a spatial response map for the target object. We demonstrate on a benchmark dataset that our model can produce quality segmentation output from the natural language expression, and outperforms baseline methods by a large margin.

Key facts. Cited by 257 papers in the map (data through 7 Oct 2026). First posted on arXiv 20 Mar 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 1; 2019: 0; 2020: 0; 2021: 30; 2022: 26; 2023: 68; 2024: 46; 2025: 49; 2026: 37.

257 papers.

Sort:

By area:

2023Segment Everything Everywhere All at Oncecited by 546
2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 530
2021Image Segmentation Using Text and Image Promptscited by 509
2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 471
2021CRIS: CLIP-Driven Referring Image Segmentationcited by 353
2021LAVT: Language-Aware Vision Transformer for Referring Image Segmentationcited by 347
2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 304
2022Generalized Decoding for Pixel, Image, and Languagecited by 292
2023GRES: Generalized Referring Expression Segmentationcited by 252
2021Vision-Language Transformer and Query Generation for Referring Segmentationcited by 239
2023MeViS: A Large-scale Benchmark for Video Segmentation with Motion Expressionscited by 225
2022Language as Queries for Referring Video Object Segmentationcited by 180

38 more

2021Source-Free Domain Adaptation for Semantic Segmentationcited by 168
2023GSVA: Generalized Segmentation via Multimodal Large Language Modelscited by 153
2023PolyFormer: Referring Image Segmentation as Sequential Polygon Generationcited by 147
2021Referring Transformer: A One-step Approach to Multi-task Visual Groundingcited by 143
2021End-to-End Referring Video Object Segmentation with Multimodal Transformerscited by 135
2022ReSTR: Convolution-free Referring Image Segmentation Using Transformerscited by 112
2021Encoder Fusion Network with Co-Attention Embedding for Referring Image Segmentationcited by 111
2022VLT: Vision-Language Transformer and Query Generation for Referring Segmentationcited by 110
2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
2021Cross-Modal Progressive Comprehension for Referring Segmentationcited by 102
2021Locate then Segment: A Strong Pipeline for Referring Image Segmentationcited by 98
2025Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Groundingcited by 78
2023Rotated Multi-Scale Interaction Network for Referring Remote Sensing Image Segmentationcited by 76
2023General Object Foundation Model for Images and Videos at Scalecited by 71
2023A Comprehensive Survey on Applications of Transformers for Deep Learning Taskscited by 70
2023Beyond One-to-One: Rethinking the Referring Image Segmentationcited by 68
2024EVF-SAM: Early Vision-Language Fusion for Text-Prompted Segment Anything Modelcited by 63
2023OnlineRefer: A Simple Online Baseline for Referring Video Object Segmentationcited by 62
2023Zero-shot Referring Image Segmentation with Global-Local Context Featurescited by 61
2023RRSIS: Referring Remote Sensing Image Segmentationcited by 59
2023SOC: Semantic-Assisted Object Cluster for Referring Video Object Segmentationcited by 57
2024Decoupling Static and Hierarchical Motion Perception for Referring Video Segmentationcited by 55
2022Unsupervised Semantic Segmentation with Self-supervised Object-centric Representationscited by 54
2023Referred by Multi-Modality: A Unified Temporal Transformer for Video Object Segmentationcited by 54
2023Contrastive Grouping with Transformer for Referring Image Segmentationcited by 54
2023Hierarchical Open-vocabulary Universal Image Segmentationcited by 53
2021D3Net: A Unified Speaker-Listener Architecture for 3D Dense Captioning and Visual Groundingcited by 52
2024Toward Visual Grounding: A Surveycited by 52
2022Language-Bridged Spatial-Temporal Interaction for Referring Video Object Segmentationcited by 50
2022Towards Robust Referring Video Object Segmentation with Cyclic Relational Consensuscited by 49
2022Towards Robust Referring Image Segmentationcited by 49
2024ClipSAM: CLIP and SAM Collaboration for Zero-Shot Anomaly Segmentationcited by 49
2023Spectrum-guided Multi-granularity Referring Video Object Segmentationcited by 48
2022Instance-Specific Feature Propagation for Referring Segmentationcited by 45
2022Local-Global Context Aware Transformer for Language-Guided Video Segmentationcited by 41
2022Text2Pos: Text-to-Point-Cloud Cross-Modal Localizationcited by 40
2023See, Say, and Segment: Teaching LMMs to Overcome False Premisescited by 40
2023Mask Grounding for Referring Image Segmentationcited by 40

50 most cited of 257 All 257 (JSON)