mlsanityData to 7 Oct 2026

Improving Visual Grounding by Encouraging Consistent Gradient-based Explanations

arXiv 2206.15462 ↗First posted 30 Jun 2022cs.CV cs.CL cs.LGcited by 24

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

We propose a margin-based loss for tuning joint vision-language models so that their gradient-based explanations are consistent with region-level annotations provided by humans for relatively smaller grounding datasets. We refer to this objective as Attention Mask Consistency (AMC) and demonstrate that it produces superior visual grounding results than previous methods that rely on using vision-language models to score the outputs of object detectors. Particularly, a model trained with AMC on top of standard vision-language modeling objectives obtains a state-of-the-art accuracy of 86.49% in the Flickr30k visual grounding benchmark, an absolute improvement of 5.38% when compared to the best previous model trained under the same level of supervision. Our approach also performs exceedingly well on established benchmarks for referring expression comprehension where it obtains 80.34% accuracy in the easy test of RefCOCO+, and 64.55% in the difficult split. AMC is effective, easy to implement, and is general as it can be adopted by any vision-language model, and can use any type of region annotations.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 47 of its 64 references that are in the map. First posted on arXiv 30 Jun 2022.

Builds on

47 of 64 references.

References · 47

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  7. 2017Mask R-CNNcited by 7,259
  8. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  9. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  10. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  11. 2015Learning Deep Features for Discriminative Localizationcited by 2,430
  12. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
35 more
  1. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  2. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  3. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  4. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  5. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  6. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  7. 2021Grounded Language-Image Pre-trainingcited by 1,147
  8. 2016Modeling Context in Referring Expressionscited by 1,054
  9. 2017Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networkscited by 936
  10. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 427
  11. 2017Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraintscited by 420
  12. 2016Top-down Neural Attention by Excitation Backpropcited by 318
  13. 2021Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformerscited by 316
  14. 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
  15. 2015Cross Modal Distillation for Supervision Transfercited by 173
  16. 2019Taking a HINT: Leveraging Explanations to Make Vision and Language Models More Groundedcited by 127
  17. 2015Grounding of Textual Phrases in Images by Reconstructioncited by 112
  18. 2020Contrastive Learning for Weakly Supervised Phrase Groundingcited by 102
  19. 2021Learning to Predict Visual Attributes in the Wildcited by 95
  20. 2020CASTing Your Model: Learning to Localize Improves Self-Supervised Representationscited by 63
  21. 2019Align2Ground: Weakly Supervised Phrase Grounding Guided by Image-Caption Alignmentcited by 51
  22. 2020Improving Weakly Supervised Visual Grounding by Contrastive Knowledge Distillationcited by 51
  23. 2017Weakly-supervised Visual Grounding of Phrases with Linguistic Structurescited by 40
  24. 2021Relation-aware Instance Refinement for Weakly Supervised Visual Groundingcited by 35
  25. 2018Knowledge Aided Consistency for Weakly Supervised Phrase Groundingcited by 32
  26. 2018Multi-level Multimodal Common Semantic Space for Image-Phrase Groundingcited by 32
  27. 2019Phrase Localization Without Paired Training Examplescited by 24
  28. 2021Consistent Explanations by Contrastive Learningcited by 18
  29. 2019Modularized Textual Grounding for Counterfactual Resiliencecited by 17
  30. 2021Detector-Free Weakly Supervised Grounding by Separationcited by 16
  31. 2018Interpretable and Globally Optimal Prediction for Textual Grounding using Image Conceptscited by 15
  32. 2018Unsupervised Textual Grounding: Linking Words to Image Conceptscited by 13
  33. 2022What is Where by Looking: Weakly-Supervised Open-World Phrase-Grounding without Text Inputscited by 11
  34. 2018Learning Unsupervised Visual Grounding Through Semantic Self-Supervisioncited by 10
  35. 2020MAF: Multimodal Alignment Framework for Weakly-Supervised Phrase Groundingcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 9; 2025: 6; 2026: 2.

24 papers.

Sort:

By area:

  1. 2023OT-Attack: Enhancing Adversarial Transferability of Vision-Language Models via Optimal Transport Optimizationcited by 36
  2. 2023Improved Visual Grounding through Self-Consistent Explanationscited by 17
  3. 2023Studying How to Efficiently and Effectively Guide Models with Explanationscited by 16
  4. 2023InstructDET: Diversifying Referring Object Detection with Generalized Instructionscited by 16
  5. 2023Localizing Active Objects from Egocentric Vision with Symbolic World Knowledgecited by 8
  6. 2025LangGas: Introducing Language in Selective Zero-Shot Background Subtraction for Semi-Transparent Gas Leak Detection with a New Datasetcited by 6
  7. 2024Learning from Synthetic Data for Visual Groundingcited by 4
  8. 2025mRAG: Elucidating the Design Space of Multi-modal Retrieval-Augmented Generationcited by 4
  9. 2024Q-GroundCAM: Quantifying Grounding in Vision Language Models via GradCAMcited by 3
  10. 2025TRACE: Textual Relevance Augmentation and Contextual Encoding for Multimodal Hate Detectioncited by 3
  11. 2024Phrase Decoupling Cross-Modal Hierarchical Matching and Progressive Position Correction for Visual Groundingcited by 2
  12. 2025AIM: Amending Inherent Interpretability via Self-Supervised Maskingcited by 2
12 more
  1. 2024Beyond Image-Text Matching: Verb Understanding in Multimodal Transformers Using Guided Maskingcited by 1
  2. 2024VQA Training Sets are Self-play Environments for Generating Few-shot Poolscited by 1
  3. 2024Paint Outside the Box: Synthesizing and Selecting Training Data for Visual Groundingcited by 1
  4. 2025Compositional Image-Text Matching and Retrieval by Grounding Entitiescited by 1
  5. 2025MagiC: Evaluating Multimodal Cognition Toward Grounded Visual Reasoningcited by 1
  6. 2023Dissecting RGB-D Learning for Improved Multi-modal Fusioncited by 0
  7. 2023Augment the Pairs: Semantics-Preserving Image-Caption Pair Augmentation for Grounding-Based Vision and Language Modelscited by 0
  8. 2024Adversarial Testing for Visual Grounding via Image-Aware Property Reductioncited by 0
  9. 2024Make Graph-based Referring Expression Comprehension Great Again through Expression-guided Dynamic Gating and Regressioncited by 0
  10. 2024Feedback-based Modal Mutual Search for Attacking Vision-Language Pre-training Modelscited by 0
  11. 2026Consistent Evidence, Robust Recognition: Faithful Attribution Regularization under Geometric Transformationscited by 0
  12. 2026Grad-CAM for Vision Transformers: A Systematic Taxonomy and Audit of Methodological Ambiguity in Explainable AIcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Grounding
  2. 2025Connecting the Dots: Training-Free Visual Grounding via Agentic Reasoning
  3. 2026Beyond Static Cropping: Layer-Adaptive Visual Localization and Decoding Enhancement
  4. 2026TempoGround: State-Aware Streaming Visual Grounding with Vision-Language Models
  5. 2023Advancing Visual Grounding with Scene Knowledge: Benchmark and Method
  6. 2024The ART of Composition: Attention-Regularized Training for Compositional Visual Grounding
  7. 2024MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMs
  8. 2024GROUNDHOG: Grounding Large Language Models to Holistic Segmentation