mlsanityData to 7 Oct 2026

LAVT: Language-Aware Vision Transformer for Referring Image Segmentation

arXiv 2112.02244 ↗First posted 4 Dec 2021cs.CV cs.CLcited by 347

Not on the map. Why?

Cited by 55 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Referring image segmentation is a fundamental vision-language task that aims to segment out an object referred to by a natural language expression from an image. One of the key challenges behind this task is leveraging the referring expression for highlighting relevant positions in the image. A paradigm for tackling this problem is to leverage a powerful vision-language ("cross-modal") decoder to fuse features independently extracted from a vision encoder and a language encoder. Recent methods have made remarkable advancements in this paradigm by exploiting Transformers as cross-modal decoders, concurrent to the Transformer's overwhelming success in many other vision-language tasks. Adopting a different approach in this work, we show that significantly better cross-modal alignments can be achieved through the early fusion of linguistic and visual features in intermediate layers of a vision Transformer encoder network. By conducting cross-modal feature fusion in the visual feature encoding stage, we can leverage the well-proven correlation modeling power of a Transformer encoder for excavating helpful multi-modal context. This way, accurate segmentation results are readily harvested with a light-weight mask predictor. Without bells and whistles, our method surpasses the previous state-of-the-art methods on RefCOCO, RefCOCO+, and G-Ref by large margins.

Key facts. Cited by 347 papers in the map (data through 7 Oct 2026). Builds on 52 of its 67 references that are in the map. First posted on arXiv 4 Dec 2021.

Builds on

52 of 67 references.

References · 52

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  9. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  10. 2020End-to-End Object Detection with Transformerscited by 7,384
  11. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  12. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
38 more
  1. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  2. 2017Squeeze-and-Excitation Networkscited by 4,763
  3. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  4. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  5. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  6. 2018YOLOv3: An Incremental Improvementcited by 2,652
  7. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  8. 2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,154
  9. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  10. 2021ViViT: A Video Vision Transformercited by 1,610
  11. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
  12. 2016Instance Normalization: The Missing Ingredient for Fast Stylizationcited by 1,187
  13. 2021Video Swin Transformercited by 1,090
  14. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  15. 2016Modeling Context in Referring Expressionscited by 1,054
  16. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  17. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  18. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  19. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 530
  20. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  21. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 427
  22. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 304
  23. 2019Cross-Modal Self-Attention Network for Referring Image Segmentationcited by 263
  24. 2016Segmentation from Natural Language Expressionscited by 257
  25. 2016Modeling Context Between Objects for Referring Expression Understandingcited by 250
  26. 2021Vision-Language Transformer and Query Generation for Referring Segmentationcited by 239
  27. 2020Multi-task Collaborative Network for Joint Referring Expression Comprehension and Segmentationcited by 204
  28. 2021UniT: Multimodal Multitask Learning with a Unified Transformercited by 195
  29. 2020Collaborative Video Object Segmentation by Foreground-Background Integrationcited by 142
  30. 2020Referring Image Segmentation via Cross-Modal Progressive Comprehensioncited by 131
  31. 2017Recurrent Multimodal Interaction for Referring Image Segmentationcited by 125
  32. 2021Encoder Fusion Network with Co-Attention Embedding for Referring Image Segmentationcited by 111
  33. 2020Linguistic Structure Guided Context Modeling for Referring Image Segmentationcited by 102
  34. 2021Cross-Modal Progressive Comprehension for Referring Segmentationcited by 102
  35. 2021Locate then Segment: A Strong Pipeline for Referring Image Segmentationcited by 98
  36. 2021Mitigating Language-Dependent Ethnic Bias in BERTcited by 87
  37. 2018Dynamic Multimodal Instance Segmentation guided by natural language queriescited by 83
  38. 2017Language-Based Image Editing with Recurrent Attentive Modelscited by 53

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 13; 2023: 84; 2024: 85; 2025: 90; 2026: 75.

347 papers.

Sort:

By area:

  1. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 770
  2. 2023Segment Everything Everywhere All at Oncecited by 546
  3. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 447
  4. 2022Generalized Decoding for Pixel, Image, and Languagecited by 292
  5. 2023Unleashing Text-to-Image Diffusion Models for Visual Perceptioncited by 266
  6. 2023GRES: Generalized Referring Expression Segmentationcited by 252
  7. 2025Seg-Zero: Reasoning-Chain Guided Segmentation via Cognitive Reinforcementcited by 234
  8. 2023Uni3D: Exploring Unified 3D Representation at Scalecited by 231
  9. 2023MeViS: A Large-scale Benchmark for Video Segmentation with Motion Expressionscited by 225
  10. 2023PixelLM: Pixel Reasoning with Large Multimodal Modelcited by 220
  11. 2023Universal Instance Perception as Object Discovery and Retrievalcited by 190
  12. 2023Towards Open Vocabulary Learning: A Surveycited by 165
38 more
  1. 2023InstructDiffusion: A Generalist Modeling Interface for Vision Taskscited by 162
  2. 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videoscited by 154
  3. 2023GSVA: Generalized Segmentation via Multimodal Large Language Modelscited by 153
  4. 2022DETRs with Hybrid Matchingcited by 149
  5. 2023PolyFormer: Referring Image Segmentation as Sequential Polygon Generationcited by 147
  6. 2024OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understandingcited by 126
  7. 2023Transformer-Based Visual Segmentation: A Surveycited by 124
  8. 2024VISA: Reasoning Video Object Segmentation via Large Language Modelscited by 123
  9. 2022LViT: Language meets Vision Transformer in Medical Image Segmentationcited by 110
  10. 2022VLT: Vision-Language Transformer and Query Generation for Referring Segmentationcited by 110
  11. 2023Contextual Object Detection with Multimodal Large Language Modelscited by 100
  12. 2024One Token to Seg Them All: Language Instructed Reasoning Segmentation in Videoscited by 99
  13. 2025Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Groundingcited by 78
  14. 2023Rotated Multi-Scale Interaction Network for Referring Remote Sensing Image Segmentationcited by 76
  15. 2024GROUNDHOG: Grounding Large Language Models to Holistic Segmentationcited by 70
  16. 2023CLIP as RNN: Segment Countless Visual Concepts without Training Endeavorcited by 69
  17. 2023Beyond One-to-One: Rethinking the Referring Image Segmentationcited by 68
  18. 2024SAM4MLLM: Enhance Multi-Modal Large Language Model for Referring Expression Segmentationcited by 68
  19. 2024EVF-SAM: Early Vision-Language Fusion for Text-Prompted Segment Anything Modelcited by 63
  20. 2023Zero-shot Referring Image Segmentation with Global-Local Context Featurescited by 61
  21. 2023RRSIS: Referring Remote Sensing Image Segmentationcited by 59
  22. 2023PerceptionGPT: Effectively Fusing Visual Perception into LLMcited by 59
  23. 2024Text4Seg: Reimagining Image Segmentation as Text Generationcited by 59
  24. 2023SOC: Semantic-Assisted Object Cluster for Referring Video Object Segmentationcited by 57
  25. 2024Decoupling Static and Hierarchical Motion Perception for Referring Video Segmentationcited by 55
  26. 2024LaSagnA: Language-based Segmentation Assistant for Complex Queriescited by 55
  27. 2023Referred by Multi-Modality: A Unified Temporal Transformer for Video Object Segmentationcited by 54
  28. 2023Contrastive Grouping with Transformer for Referring Image Segmentationcited by 54
  29. 2023Hierarchical Open-vocabulary Universal Image Segmentationcited by 53
  30. 2024Visual Agents as Fast and Slow Thinkerscited by 53
  31. 2023Jack of All Tasks, Master of Many: Designing General-purpose Coarse-to-Fine Vision-Language Modelcited by 52
  32. 2024Toward Visual Grounding: A Surveycited by 52
  33. 2025VisionReasoner: Unified Reasoning-Integrated Visual Perception via Reinforcement Learningcited by 50
  34. 2022Towards Robust Referring Video Object Segmentation with Cyclic Relational Consensuscited by 49
  35. 2022Towards Robust Referring Image Segmentationcited by 49
  36. 2024Vitron: A Unified Pixel-level Vision LLM for Understanding, Generating, Segmenting, Editingcited by 49
  37. 2023Spectrum-guided Multi-granularity Referring Video Object Segmentationcited by 48
  38. 2025KV-Edit: Training-Free Image Editing for Precise Background Preservationcited by 47

50 most cited of 347 All 347 (JSON)