mlsanityData to 7 Oct 2026

Vision-Language Transformer and Query Generation for Referring Segmentation

arXiv 2108.05565 ↗First posted 12 Aug 2021cs.CVcited by 239

Not on the map. Why?

Cited by 34 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this work, we address the challenging task of referring segmentation. The query expression in referring segmentation typically indicates the target object by describing its relationship with others. Therefore, to find the target one among all instances in the image, the model must have a holistic understanding of the whole image. To achieve this, we reformulate referring segmentation as a direct attention problem: finding the region in the image where the query language expression is most attended to. We introduce transformer and multi-head attention to build a network with an encoder-decoder attention mechanism architecture that "queries" the given image with the language expression. Furthermore, we propose a Query Generation Module, which produces multiple sets of queries with different attention weights that represent the diversified comprehensions of the language expression from different aspects. At the same time, to find the best way from these diversified comprehensions based on visual clues, we further propose a Query Balance Module to adaptively select the output features of these queries for a better mask generation. Without bells and whistles, our approach is light-weight and achieves new state-of-the-art performance consistently on three referring segmentation datasets, RefCOCO, RefCOCO+, and G-Ref. Our code is available at https://github.com/henghuiding/Vision-Language-Transformer.

Key facts. Cited by 239 papers in the map (data through 7 Oct 2026). Builds on 24 of its 33 references that are in the map. First posted on arXiv 12 Aug 2021.

Builds on

24 of 33 references.

References · 24

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020End-to-End Object Detection with Transformerscited by 7,384
  5. 2017Mask R-CNNcited by 7,259
  6. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  7. 2017Non-local Neural Networkscited by 2,290
  8. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
  9. 2016Modeling Context in Referring Expressionscited by 1,054
  10. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  11. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 427
  12. 2019Cross-Modal Self-Attention Network for Referring Image Segmentationcited by 263
12 more
  1. 2016Segmentation from Natural Language Expressionscited by 257
  2. 2016Modeling Context Between Objects for Referring Expression Understandingcited by 250
  3. 2020Multi-task Collaborative Network for Joint Referring Expression Comprehension and Segmentationcited by 204
  4. 2020Referring Image Segmentation via Cross-Modal Progressive Comprehensioncited by 131
  5. 2017Recurrent Multimodal Interaction for Referring Image Segmentationcited by 125
  6. 2019Boundary-Aware Feature Propagation for Scene Segmentationcited by 107
  7. 2016Modeling Relationships in Referential Expressions with Compositional Modular Networkscited by 106
  8. 2020Linguistic Structure Guided Context Modeling for Referring Image Segmentationcited by 102
  9. 2018Dynamic Multimodal Instance Segmentation guided by natural language queriescited by 83
  10. 2019Semantic Correlation Promoted Shape-Variant Context for Segmentationcited by 61
  11. 2017Comprehension-guided referring expressionscited by 52
  12. 2019Dynamic Evaluation of Transformer Language Modelscited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 23; 2023: 70; 2024: 57; 2025: 55; 2026: 29.

239 papers.

Sort:

By area:

  1. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 770
  2. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 471
  3. 2021CRIS: CLIP-Driven Referring Image Segmentationcited by 353
  4. 2021LAVT: Language-Aware Vision Transformer for Referring Image Segmentationcited by 347
  5. 2023Unleashing Text-to-Image Diffusion Models for Visual Perceptioncited by 266
  6. 2023GRES: Generalized Referring Expression Segmentationcited by 252
  7. 2023MOSE: A New Dataset for Video Object Segmentation in Complex Scenescited by 232
  8. 2023MeViS: A Large-scale Benchmark for Video Segmentation with Motion Expressionscited by 225
  9. 2023PixelLM: Pixel Reasoning with Large Multimodal Modelcited by 220
  10. 2023Universal Instance Perception as Object Discovery and Retrievalcited by 190
  11. 2022Language as Queries for Referring Video Object Segmentationcited by 180
  12. 2023Towards Open Vocabulary Learning: A Surveycited by 165
38 more
  1. 2023GSVA: Generalized Segmentation via Multimodal Large Language Modelscited by 153
  2. 2023PolyFormer: Referring Image Segmentation as Sequential Polygon Generationcited by 147
  3. 2022SeqTR: A Simple yet Universal Network for Visual Groundingcited by 138
  4. 2023Transformer-Based Visual Segmentation: A Surveycited by 124
  5. 2024VISA: Reasoning Video Object Segmentation via Large Language Modelscited by 123
  6. 2022ReSTR: Convolution-free Referring Image Segmentation Using Transformerscited by 112
  7. 2022LViT: Language meets Vision Transformer in Medical Image Segmentationcited by 110
  8. 2022VLT: Vision-Language Transformer and Query Generation for Referring Segmentationcited by 110
  9. 2023Contextual Object Detection with Multimodal Large Language Modelscited by 100
  10. 2024PSALM: Pixelwise SegmentAtion with Large Multi-Modal Modelcited by 100
  11. 2024One Token to Seg Them All: Language Instructed Reasoning Segmentation in Videoscited by 99
  12. 2023NExT-Chat: An LMM for Chat, Detection and Segmentationcited by 79
  13. 2023Rotated Multi-Scale Interaction Network for Referring Remote Sensing Image Segmentationcited by 76
  14. 2024Unifying 3D Vision-Language Understanding via Promptable Queriescited by 70
  15. 2023Beyond One-to-One: Rethinking the Referring Image Segmentationcited by 68
  16. 2024SAM4MLLM: Enhance Multi-Modal Large Language Model for Referring Expression Segmentationcited by 68
  17. 2024EVF-SAM: Early Vision-Language Fusion for Text-Prompted Segment Anything Modelcited by 63
  18. 2023Zero-shot Referring Image Segmentation with Global-Local Context Featurescited by 61
  19. 2023PerceptionGPT: Effectively Fusing Visual Perception into LLMcited by 59
  20. 2025MOSEv2: A More Challenging Dataset for Video Object Segmentation in Complex Scenescited by 59
  21. 2023SOC: Semantic-Assisted Object Cluster for Referring Video Object Segmentationcited by 57
  22. 2024DoraemonGPT: Toward Understanding Dynamic Scenes with Large Language Models (Exemplified as A Video Agent)cited by 56
  23. 2024Decoupling Static and Hierarchical Motion Perception for Referring Video Segmentationcited by 55
  24. 2024LaSagnA: Language-based Segmentation Assistant for Complex Queriescited by 55
  25. 2023Referred by Multi-Modality: A Unified Temporal Transformer for Video Object Segmentationcited by 54
  26. 2023Contrastive Grouping with Transformer for Referring Image Segmentationcited by 54
  27. 2023Hierarchical Open-vocabulary Universal Image Segmentationcited by 53
  28. 2024Toward Visual Grounding: A Surveycited by 52
  29. 2022Towards Robust Referring Video Object Segmentation with Cyclic Relational Consensuscited by 49
  30. 2022Towards Robust Referring Image Segmentationcited by 49
  31. 2022Instance-Specific Feature Propagation for Referring Segmentationcited by 45
  32. 2022Local-Global Context Aware Transformer for Language-Guided Video Segmentationcited by 41
  33. 2022Feature Aggregation and Propagation Network for Camouflaged Object Detectioncited by 40
  34. 2023See, Say, and Segment: Teaching LMMs to Overcome False Premisescited by 40
  35. 2023Mask Grounding for Referring Image Segmentationcited by 40
  36. 2023GREC: Generalized Referring Expression Comprehensioncited by 39
  37. 2024Correlation Matching Transformation Transformers for UHD Image Restorationcited by 39
  38. 2024ReMamber: Referring Image Segmentation with Mamba Twistercited by 38

50 most cited of 239 All 239 (JSON)