mlsanityData to 7 Oct 2026

Referring Transformer: A One-step Approach to Multi-task Visual Grounding

arXiv 2106.03089 ↗First posted 6 Jun 2021cs.CVcited by 143

Not on the map. Why?

Abstract (arXiv)

As an important step towards visual reasoning, visual grounding (e.g., phrase localization, referring expression comprehension/segmentation) has been widely explored Previous approaches to referring expression comprehension (REC) or segmentation (RES) either suffer from limited performance, due to a two-stage setup, or require the designing of complex task-specific one-stage architectures. In this paper, we propose a simple one-stage multi-task framework for visual grounding tasks. Specifically, we leverage a transformer architecture, where two modalities are fused in a visual-lingual encoder. In the decoder, the model learns to generate contextualized lingual queries which are then decoded and used to directly regress the bounding box and produce a segmentation mask for the corresponding referred regions. With this simple but highly contextualized model, we outperform state-of-the-arts methods by a large margin on both REC and RES tasks. We also show that a simple pre-training schedule (on an external dataset) further improves the performance. Extensive experiments and ablations illustrate that our model benefits greatly from contextualized information and multi-task training.

Key facts. Cited by 143 papers in the map (data through 7 Oct 2026). Builds on 50 of its 61 references that are in the map. First posted on arXiv 6 Jun 2021.

Builds on

50 of 61 references.

References · 50

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  6. 2020End-to-End Object Detection with Transformerscited by 7,384
  7. 2017Mask R-CNNcited by 7,259
  8. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  9. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  10. 2018YOLOv3: An Incremental Improvementcited by 2,652
  11. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  12. 2016V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentationcited by 2,284
38 more
  1. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  2. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  3. 2019Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regressioncited by 1,361
  4. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  5. 2016Modeling Context in Referring Expressionscited by 1,054
  6. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  7. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  8. 2019CenterNet: Keypoint Triplets for Object Detectioncited by 654
  9. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 427
  10. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  11. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  12. 2021TransVG: End-to-End Visual Grounding with Transformerscited by 269
  13. 2019Cross-Modal Self-Attention Network for Referring Image Segmentationcited by 263
  14. 2016Segmentation from Natural Language Expressionscited by 257
  15. 2016Modeling Context Between Objects for Referring Expression Understandingcited by 250
  16. 2020ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graphcited by 211
  17. 2020Multi-task Collaborative Network for Joint Referring Expression Comprehension and Segmentationcited by 204
  18. 2019A Fast and Accurate One-Stage Approach to Visual Groundingcited by 194
  19. 2021Fast Convergence of DETR with Spatially Modulated Co-Attentioncited by 176
  20. 2020Rethinking Transformer-based Set Prediction for Object Detectioncited by 156
  21. 2018Learning to Assemble Neural Module Tree Networks for Visual Groundingcited by 139
  22. 2020Improving One-stage Visual Grounding by Recursive Sub-query Constructioncited by 137
  23. 2020Referring Image Segmentation via Cross-Modal Progressive Comprehensioncited by 131
  24. 2017Learning Two-Branch Neural Networks for Image-Text Matching Taskscited by 119
  25. 2019A Real-Time Cross-modality Correlation Filtering Method for Referring Expression Comprehensioncited by 111
  26. 2016Modeling Relationships in Referential Expressions with Compositional Modular Networkscited by 106
  27. 2019Learning to Compose and Reason with Language Tree Structures for Visual Groundingcited by 98
  28. 2021Locate then Segment: A Strong Pipeline for Referring Image Segmentationcited by 98
  29. 2019Improving Referring Expression Grounding with Cross-modal Attention-guided Erasingcited by 88
  30. 2018Dynamic Multimodal Instance Segmentation guided by natural language queriescited by 83
  31. 2019Zero-Shot Grounding of Objects from Natural Language Queriescited by 79
  32. 2018Real-Time Referring Expression Comprehension by Single-Stage Grounding Networkcited by 56
  33. 2017Conditional Image-Text Embedding Networkscited by 50
  34. 2018Rethinking Diversified and Discriminative Proposal Generation for Visual Groundingcited by 49
  35. 2019Referring Expression Object Segmentation with Caption-Aware Consistencycited by 46
  36. 2021Visual Grounding with Transformerscited by 42
  37. 2017Query-guided Regression Network with Context Policy for Phrase Groundingcited by 34
  38. 2019Neural Sequential Phrase Grounding (SeqGROUND)cited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 19; 2023: 49; 2024: 35; 2025: 27; 2026: 11.

143 papers.

Sort:

By area:

  1. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  2. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMscited by 215
  3. 2023Universal Instance Perception as Object Discovery and Retrievalcited by 190
  4. 2023PolyFormer: Referring Image Segmentation as Sequential Polygon Generationcited by 147
  5. 2023Multi3DRefer: Grounding Text Description to Multiple 3D Objectscited by 142
  6. 2022SeqTR: A Simple yet Universal Network for Visual Groundingcited by 138
  7. 2023PaLI-3 Vision Language Models: Smaller, Faster, Strongercited by 125
  8. 2022RSVG: Exploring Data and Models for Visual Grounding on Remote Sensing Datacited by 121
  9. 2021A Survey of Visual Transformerscited by 117
  10. 2023Contextual Object Detection with Multimodal Large Language Modelscited by 100
  11. 2023Referring Multi-Object Trackingcited by 91
  12. 2023Rotated Multi-Scale Interaction Network for Referring Remote Sensing Image Segmentationcited by 76
38 more
  1. 2024EVF-SAM: Early Vision-Language Fusion for Text-Prompted Segment Anything Modelcited by 63
  2. 2023Hierarchical Open-vocabulary Universal Image Segmentationcited by 53
  3. 2023Jack of All Tasks, Master of Many: Designing General-purpose Coarse-to-Fine Vision-Language Modelcited by 52
  4. 2024Toward Visual Grounding: A Surveycited by 52
  5. 2025VisionReasoner: Unified Reasoning-Integrated Visual Perception via Reinforcement Learningcited by 50
  6. 2022Towards Robust Referring Video Object Segmentation with Cyclic Relational Consensuscited by 49
  7. 2023Spectrum-guided Multi-granularity Referring Video Object Segmentationcited by 48
  8. 2022Local-Global Context Aware Transformer for Language-Guided Video Segmentationcited by 41
  9. 2022TransVG++: End-to-End Visual Grounding with Language Conditioned Vision Transformercited by 41
  10. 2023Described Object Detection: Liberating Object Detection with Flexible Expressionscited by 40
  11. 2022Hierarchical Local-Global Transformer for Temporal Sentence Groundingcited by 38
  12. 2025RemoteSAM: Towards Segment Anything for Earth Observationcited by 38
  13. 2023A Systematic Evaluation of GPT-4V's Multimodal Capability for Medical Image Analysiscited by 37
  14. 2024SimVG: A Simple Framework for Visual Grounding with Decoupled Multi-modal Fusioncited by 36
  15. 2023CLIP-VG: Self-paced Curriculum Adapting of CLIP for Visual Groundingcited by 35
  16. 2023Language Adaptive Weight Generation for Multi-task Visual Groundingcited by 35
  17. 2023Referring Image Segmentation Using Text Supervisioncited by 35
  18. 2024LocCa: Visual Pretraining with Location-aware Captionerscited by 35
  19. 2021INVIGORATE: Interactive Visual Grounding and Grasping in Cluttercited by 32
  20. 2023Medical Phrase Grounding with Region-Phrase Context Contrastive Alignmentcited by 32
  21. 2024HiVG: Hierarchical Multimodal Fine-grained Modulation for Visual Groundingcited by 31
  22. 2022Dynamic MDETR: A Dynamic Multimodal Transformer Decoder for Visual Groundingcited by 29
  23. 2023Lenna: Language Enhanced Reasoning Detection Assistantcited by 29
  24. 2023Parallel Vertex Diffusion for Unified Visual Groundingcited by 28
  25. 2024OneRef: Unified One-tower Expression Grounding and Segmentation with Mask Referring Modelingcited by 28
  26. 2022Multi-Attention Network for Compressed Video Referring Object Segmentationcited by 27
  27. 2024Exploring Pre-trained Text-to-Video Diffusion Models for Referring Video Object Segmentationcited by 27
  28. 20233DRP-Net: 3D Relative Position-aware Network for 3D Visual Groundingcited by 25
  29. 2024RefMask3D: Language-Guided Transformer for 3D Referring Segmentationcited by 25
  30. 2022DQ-DETR: Dual Query Detection Transformer for Phrase Extraction and Groundingcited by 24
  31. 2023RefSAM: Efficiently Adapting Segmenting Anything Model for Referring Video Object Segmentationcited by 24
  32. 2023Context Disentangling and Prototype Inheriting for Robust Visual Groundingcited by 23
  33. 2024Adversarial Robustness for Visual Grounding of Multimodal Large Language Modelscited by 23
  34. 2023Language-guided Robot Grasping: CLIP-based Referring Grasp Synthesis in Cluttercited by 22
  35. 2023Mono3DVG: 3D Visual Grounding in Monocular Imagescited by 22
  36. 2023UniRef++: Segment Every Reference Object in Spatial and Temporal Spacescited by 22
  37. 2025Multimodal Referring Segmentation: A Surveycited by 22
  38. 2023RefEgo: Referring Expression Comprehension Dataset from First-Person Perception of Ego4Dcited by 21

50 most cited of 143 All 143 (JSON)