mlsanityData to 14 Oct 2026

Visual Grounding with Transformers

arXiv 2105.04281 ↗First posted 10 May 2021cs.CVcited by 42

Not on the map. Why?

Abstract (arXiv)

In this paper, we propose a transformer based approach for visual grounding. Unlike previous proposal-and-rank frameworks that rely heavily on pretrained object detectors or proposal-free frameworks that upgrade an off-the-shelf one-stage detector by fusing textual embeddings, our approach is built on top of a transformer encoder-decoder and is independent of any pretrained detectors or word embedding models. Termed VGTR -- Visual Grounding with TRansformers, our approach is designed to learn semantic-discriminative visual features under the guidance of the textual description without harming their location ability. This information flow enables our VGTR to have a strong capability in capturing context-level semantics of both vision and language modalities, rendering us to aggregate accurate visual clues implied by the description to locate the interested object instance. Experiments show that our method outperforms state-of-the-art proposal-free approaches by a considerable margin on five benchmarks while maintaining fast inference speed.

Key facts. Cited by 42 papers in the map (data through 14 Oct 2026). Builds on 49 of its 55 references that are in the map. First posted on arXiv 10 May 2021.

Builds on

49 of 55 references.

References · 49

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  7. 2020End-to-End Object Detection with Transformerscited by 7,414
  8. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
  9. 2018YOLOv3: An Incremental Improvementcited by 2,659
  10. 2017Deformable Convolutional Networkscited by 1,488
  11. 2015Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcastingcited by 1,436
  12. 2019Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regressioncited by 1,368
37 more
  1. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  2. 2016Modeling Context in Referring Expressionscited by 1,063
  3. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
  4. 2020On Layer Normalization in the Transformer Architecturecited by 922
  5. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 904
  6. 2018Image Transformercited by 772
  7. 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 645
  8. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 431
  9. 2019Attention Augmented Convolutional Networkscited by 363
  10. 2020MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformerscited by 355
  11. 2016Modeling Context Between Objects for Referring Expression Understandingcited by 251
  12. 2016Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Groundingcited by 243
  13. 2019A Fast and Accurate One-Stage Approach to Visual Groundingcited by 195
  14. 2020Rethinking Transformer-based Set Prediction for Object Detectioncited by 156
  15. 2018Learning to Assemble Neural Module Tree Networks for Visual Groundingcited by 140
  16. 2020Improving One-stage Visual Grounding by Recursive Sub-query Constructioncited by 138
  17. 2015Natural Language Object Retrievalcited by 124
  18. 2015Grounding of Textual Phrases in Images by Reconstructioncited by 112
  19. 2019A Real-Time Cross-modality Correlation Filtering Method for Referring Expression Comprehensioncited by 111
  20. 2020End-to-end Lane Shape Prediction with Transformerscited by 111
  21. 2018Neighbourhood Watch: Referring Expression Comprehension via Language-guided Graph Attention Networkscited by 102
  22. 2019Learning to Compose and Reason with Language Tree Structures for Visual Groundingcited by 98
  23. 2019Improving Referring Expression Grounding with Cross-modal Attention-guided Erasingcited by 88
  24. 2017Grounding Referring Expressions in Images by Variational Contextcited by 82
  25. 2019Zero-Shot Grounding of Objects from Natural Language Queriescited by 79
  26. 2020Referring Expression Comprehension: A Survey of Methods and Datasetscited by 79
  27. 2016A Joint Speaker-Listener-Reinforcer Model for Referring Expressionscited by 74
  28. 2016CNN Image Retrieval Learns from BoW: Unsupervised Fine-Tuning with Hard Examplescited by 58
  29. 2018Real-Time Referring Expression Comprehension by Single-Stage Grounding Networkcited by 56
  30. 2017Conditional Image-Text Embedding Networkscited by 50
  31. 2020Cops-Ref: A new Dataset and Task on Compositional Referring Expression Comprehensioncited by 50
  32. 2020Ref-NMS: Breaking Proposal Bottlenecks in Two-Stage Referring Expression Groundingcited by 50
  33. 2018Rethinking Diversified and Discriminative Proposal Generation for Visual Groundingcited by 49
  34. 2020Attention-Based Transformers for Instance Segmentation of Cells in Microstructurescited by 38
  35. 2017Query-guided Regression Network with Context Policy for Phrase Groundingcited by 34
  36. 2019Relationship-Embedded Representation Learning for Grounding Referring Expressionscited by 25
  37. 2018Interpretable and Globally Optimal Prediction for Textual Grounding using Image Conceptscited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 3; 2022: 13; 2023: 9; 2024: 10; 2025: 4; 2026: 3.

42 papers.

Sort:

By area:

  1. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  2. 2021Referring Transformer: A One-step Approach to Multi-task Visual Groundingcited by 143
  3. 2022RSVG: Exploring Data and Models for Visual Grounding on Remote Sensing Datacited by 123
  4. 2021A Survey of Visual Transformerscited by 117
  5. 2022TransVG++: End-to-End Visual Grounding with Language Conditioned Vision Transformercited by 42
  6. 2022Local-Global Context Aware Transformer for Language-Guided Video Segmentationcited by 41
  7. 2023A Systematic Evaluation of GPT-4V's Multimodal Capability for Medical Image Analysiscited by 37
  8. 2024SimVG: A Simple Framework for Visual Grounding with Decoupled Multi-modal Fusioncited by 36
  9. 2023CLIP-VG: Self-paced Curriculum Adapting of CLIP for Visual Groundingcited by 35
  10. 2023Medical Phrase Grounding with Region-Phrase Context Contrastive Alignmentcited by 32
  11. 2022Dynamic MDETR: A Dynamic Multimodal Transformer Decoder for Visual Groundingcited by 29
  12. 2022DQ-DETR: Dual Query Detection Transformer for Phrase Extraction and Groundingcited by 24
30 more
  1. 2023Context Disentangling and Prototype Inheriting for Robust Visual Groundingcited by 23
  2. 2024ChEX: Interactive Localization and Region Description in Chest X-rayscited by 19
  3. 2022YORO -- Lightweight End to End Visual Groundingcited by 17
  4. 2024MedViLaM: A multimodal large language model with advanced generalizability and explainability for medical data understanding and generationcited by 17
  5. 2022Understanding Embodied Reference with Touch-Line Transformercited by 16
  6. 2023Learning A Multi-Task Transformer Via Unified And Customized Instruction Tuning For Chest Radiograph Interpretationcited by 14
  7. 2024WaterVG: Waterway Visual Grounding based on Text-Guided Vision and mmWave Radarcited by 13
  8. 2024Uncertainty-aware Medical Diagnostic Phrase Identification and Groundingcited by 13
  9. 2023Visual Dependency Transformers: Dependency Tree Emerges from Reversed Attentioncited by 12
  10. 2024M2IST: Multi-Modal Interactive Side-Tuning for Efficient Referring Expression Comprehensioncited by 11
  11. 2022Self-paced Multi-grained Cross-modal Interaction Modeling for Referring Expression Comprehensioncited by 8
  12. 2023Enhancing Visual Grounding and Generalization: A Multi-Task Cycle Training Approach for Vision-Language Modelscited by 8
  13. 2025SwimVG: Step-wise Multimodal Fusion and Adaption for Visual Groundingcited by 8
  14. 2026A unified multi-task framework enables interpretable chest radiograph analysiscited by 6
  15. 2024DARA: Domain- and Relation-aware Adapters Make Parameter-efficient Tuning for Visual Groundingcited by 5
  16. 2024SeCG: Semantic-Enhanced 3D Visual Grounding via Cross-modal Graph Attentioncited by 4
  17. 2021Predicting Physical World Destinations for Commands Given to Self-Driving Carscited by 3
  18. 2023Champion Solution for the WSDM2023 Toloka VQA Challengecited by 3
  19. 2022Sim-To-Real Transfer of Visual Grounding for Human-Aided Ambiguity Resolutioncited by 2
  20. 2024Phrase Decoupling Cross-Modal Hierarchical Matching and Progressive Position Correction for Visual Groundingcited by 2
  21. 2024Referencing Where to Focus: Improving VisualGrounding with Referential Querycited by 2
  22. 2022Enhancing Interpretability and Interactivity in Robot Manipulation: A Neurosymbolic Approachcited by 1
  23. 2025A Coarse-to-Fine Approach to Multi-Modality 3D Occupancy Groundingcited by 1
  24. 2025Synthetic Object Compositions for Scalable and Accurate Learning in Detection, Segmentation, and Groundingcited by 1
  25. 2026Enhancing Medical Visual Grounding via Knowledge-guided Spatial Promptscited by 1
  26. 2022Differentiated Relevances Embedding for Group-based Referring Expression Comprehensioncited by 0
  27. 2022Attention Mechanism based Cognition-level Scene Understandingcited by 0
  28. 2022Bear the Query in Mind: Visual Grounding with Query-conditioned Convolutioncited by 0
  29. 2025Prototype-Aware Multimodal Alignment for Open-Vocabulary Visual Groundingcited by 0
  30. 2026Heuristic-inspired Reasoning Priors Facilitate Data-Efficient Referring Object Detectioncited by 0