mlsanityData to 7 Oct 2026

TransVG: End-to-End Visual Grounding with Transformers

arXiv 2104.08541 ↗First posted 17 Apr 2021cs.CVcited by 269

Not on the map. Why?

Cited by 45 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this paper, we present a neat yet effective transformer-based framework for visual grounding, namely TransVG, to address the task of grounding a language query to the corresponding region onto an image. The state-of-the-art methods, including two-stage or one-stage ones, rely on a complex module with manually-designed mechanisms to perform the query reasoning and multi-modal fusion. However, the involvement of certain mechanisms in fusion module design, such as query decomposition and image scene graph, makes the models easily overfit to datasets with specific scenarios, and limits the plenitudinous interaction between the visual-linguistic context. To avoid this caveat, we propose to establish the multi-modal correspondence by leveraging transformers, and empirically show that the complex fusion modules e.g., modular attention network, dynamic graph, and multi-modal tree) can be replaced by a simple stack of transformer encoder layers with higher performance. Moreover, we re-formulate the visual grounding as a direct coordinates regression problem and avoid making predictions out of a set of candidates i.e., region proposals or anchor boxes). Extensive experiments are conducted on five widely used datasets, and a series of state-of-the-art records are set by our TransVG. We build the benchmark of transformer-based visual grounding framework and make the code available at urlhttps://github.com/djiajunustc/TransVG.

Key facts. Cited by 269 papers in the map (data through 7 Oct 2026). Builds on 54 of its 68 references that are in the map. First posted on arXiv 17 Apr 2021.

Builds on

54 of 68 references.

References · 54

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2020End-to-End Object Detection with Transformerscited by 7,384
  7. 2016Layer Normalizationcited by 5,579
  8. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  9. 2018YOLOv3: An Incremental Improvementcited by 2,652
  10. 2015VQA: Visual Question Answeringcited by 2,489
  11. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  12. 2019Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regressioncited by 1,361
38 more
  1. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  2. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  3. 2016Modeling Context in Referring Expressionscited by 1,054
  4. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  5. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  6. 2020Pre-Trained Image Processing Transformercited by 891
  7. 2018Image Transformercited by 770
  8. 2018Universal Transformerscited by 613
  9. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 427
  10. 2020Learning Texture Transformer Network for Image Super-Resolutioncited by 346
  11. 2015Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networkscited by 305
  12. 2016Modeling Context Between Objects for Referring Expression Understandingcited by 250
  13. 2020X-Linear Attention Networks for Image Captioningcited by 198
  14. 2019A Fast and Accurate One-Stage Approach to Visual Groundingcited by 194
  15. 2020Learning Joint Spatial-Temporal Transformations for Video Inpaintingcited by 162
  16. 2015Learning Deep Structure-Preserving Image-Text Embeddingscited by 139
  17. 2018Learning to Assemble Neural Module Tree Networks for Visual Groundingcited by 139
  18. 2020Improving One-stage Visual Grounding by Recursive Sub-query Constructioncited by 137
  19. 2015Natural Language Object Retrievalcited by 124
  20. 2017Learning Two-Branch Neural Networks for Image-Text Matching Taskscited by 119
  21. 2019A Real-Time Cross-modality Correlation Filtering Method for Referring Expression Comprehensioncited by 111
  22. 2019Dynamic Graph Attention for Referring Expression Comprehensioncited by 111
  23. 2021Causal Attention for Vision-Language Taskscited by 111
  24. 2016Modeling Relationships in Referential Expressions with Compositional Modular Networkscited by 106
  25. 2020Object Relational Graph with Teacher-Recommended Learning for Video Captioningcited by 106
  26. 2018Neighbourhood Watch: Referring Expression Comprehension via Language-guided Graph Attention Networkscited by 102
  27. 2019Learning to Compose and Reason with Language Tree Structures for Visual Groundingcited by 98
  28. 2020TAP: Text-Aware Pre-training for Text-VQA and Text-Captioncited by 95
  29. 2019Improving Referring Expression Grounding with Cross-modal Attention-guided Erasingcited by 88
  30. 2019Transformer-based Acoustic Modeling for Hybrid Speech Recognitioncited by 86
  31. 2017Grounding Referring Expressions in Images by Variational Contextcited by 82
  32. 2019Zero-Shot Grounding of Objects from Natural Language Queriescited by 79
  33. 2020Streaming automatic speech recognition with the transformer modelcited by 73
  34. 2020Graph-Structured Referring Expression Reasoning in The Wildcited by 58
  35. 2017Parallel Attention: A Unified Framework for Visual Object Discovery through Dialogs and Queriescited by 56
  36. 2018Real-Time Referring Expression Comprehension by Single-Stage Grounding Networkcited by 56
  37. 2017Conditional Image-Text Embedding Networkscited by 50
  38. 2018Rethinking Diversified and Discriminative Proposal Generation for Visual Groundingcited by 49

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 10; 2022: 46; 2023: 66; 2024: 52; 2025: 62; 2026: 33.

269 papers.

Sort:

By area:

  1. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  2. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  3. 2023Universal Instance Perception as Object Discovery and Retrievalcited by 190
  4. 2024EarthGPT: A Universal Multi-modal Large Language Model for Multi-sensor Image Comprehension in Remote Sensing Domaincited by 181
  5. 2021SPViT: Enabling Faster Vision Transformers via Soft Token Pruningcited by 165
  6. 2023Towards Open Vocabulary Learning: A Surveycited by 165
  7. 2021Referring Transformer: A One-step Approach to Multi-task Visual Groundingcited by 143
  8. 2023Multi3DRefer: Grounding Text Description to Multiple 3D Objectscited by 142
  9. 2024MAIRA-2: Grounded Radiology Report Generationcited by 140
  10. 2022SeqTR: A Simple yet Universal Network for Visual Groundingcited by 138
  11. 2023GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigationcited by 137
  12. 2021Bottom Up Top Down Detection Transformers for Language Grounding in Images and Point Cloudscited by 128
38 more
  1. 2023Chat-3D: Data-efficiently Tuning Large Language Model for Universal Dialogue of 3D Scenescited by 125
  2. 2021SAT: 2D Semantics Assisted Training for 3D Visual Groundingcited by 124
  3. 2021UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modelingcited by 122
  4. 2022RSVG: Exploring Data and Models for Visual Grounding on Remote Sensing Datacited by 121
  5. 2021A Survey of Visual Transformerscited by 117
  6. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  7. 2022VLT: Vision-Language Transformer and Query Generation for Referring Segmentationcited by 110
  8. 20223D-SPS: Single-Stage 3D Visual Grounding via Referred Point Progressive Selectioncited by 109
  9. 2022EDA: Explicit Text-Decoupling and Dense Alignment for 3D Visual Groundingcited by 98
  10. 2022TubeDETR: Spatio-Temporal Video Grounding with Transformerscited by 97
  11. 2023Vision-Language Models in Remote Sensing: Current Progress and Future Trendscited by 93
  12. 2025UniVG-R1: Reasoning Guided Universal Visual Grounding with Reinforcement Learningcited by 83
  13. 2023NExT-Chat: An LMM for Chat, Detection and Segmentationcited by 79
  14. 2022Improving Visual Grounding with Visual-Linguistic Verification and Iterative Reasoningcited by 69
  15. 2023AVSegFormer: Audio-Visual Segmentation with Transformercited by 66
  16. 2023ChatSpot: Bootstrapping Multimodal LLMs via Precise Referring Instruction Tuningcited by 65
  17. 2023Joint Visual Grounding and Tracking with Natural Language Specificationcited by 59
  18. 2022Towards Open-vocabulary Scene Graph Generation with Prompt-based Finetuningcited by 54
  19. 2022Shifting More Attention to Visual Backbone: Query-modulated Refinement Networks for End-to-End Visual Groundingcited by 52
  20. 2024Toward Visual Grounding: A Surveycited by 52
  21. 2025VisionReasoner: Unified Reasoning-Integrated Visual Perception via Reinforcement Learningcited by 50
  22. 2024Small Language Model Meets with Reinforced Vision Vocabularycited by 43
  23. 2022Local-Global Context Aware Transformer for Language-Guided Video Segmentationcited by 41
  24. 2022TransVG++: End-to-End Visual Grounding with Language Conditioned Vision Transformercited by 41
  25. 2022Look Around and Refer: 2D Synthetic Semantics Knowledge Distillation for 3D Visual Groundingcited by 41
  26. 2023Described Object Detection: Liberating Object Detection with Flexible Expressionscited by 40
  27. 2023GREC: Generalized Referring Expression Comprehensioncited by 39
  28. 2022Temporal Sentence Grounding in Videos: A Survey and Future Directionscited by 37
  29. 2023A Systematic Evaluation of GPT-4V's Multimodal Capability for Medical Image Analysiscited by 37
  30. 2023CoTDet: Affordance Knowledge Prompting for Task Driven Object Detectioncited by 36
  31. 2024SimVG: A Simple Framework for Visual Grounding with Decoupled Multi-modal Fusioncited by 36
  32. 2022Exploring Optical-Flow-Guided Motion and Detection-Based Appearance for Temporal Sentence Groundingcited by 35
  33. 2022Compound Prototype Matching for Few-shot Action Recognitioncited by 35
  34. 2023CLIP-VG: Self-paced Curriculum Adapting of CLIP for Visual Groundingcited by 35
  35. 2023Language Adaptive Weight Generation for Multi-task Visual Groundingcited by 35
  36. 2022Pseudo-Q: Generating Pseudo Language Queries for Visual Groundingcited by 33
  37. 2021INVIGORATE: Interactive Visual Grounding and Grasping in Cluttercited by 32
  38. 2023Medical Phrase Grounding with Region-Phrase Context Contrastive Alignmentcited by 32

50 most cited of 269 All 269 (JSON)