mlsanityData to 7 Oct 2026

Taking a HINT: Leveraging Explanations to Make Vision and Language Models More Grounded

arXiv 1902.03751 ↗First posted 11 Feb 2019cs.CVcited by 127

Not on the map. Why?

Abstract (arXiv)

Many vision and language models suffer from poor visual grounding - often falling back on easy-to-learn language priors rather than basing their decisions on visual concepts in the image. In this work, we propose a generic approach called Human Importance-aware Network Tuning (HINT) that effectively leverages human demonstrations to improve visual grounding. HINT encourages deep networks to be sensitive to the same input regions as humans. Our approach optimizes the alignment between human attention maps and gradient-based network importances - ensuring that models learn not just to look at but rather rely on visual concepts that humans found relevant for a task when making predictions. We apply HINT to Visual Question Answering and Image Captioning tasks, outperforming top approaches on splits that penalize over-reliance on language priors (VQA-CP and robust captioning) using human attention demonstrations for just 6% of the training data.

Key facts. Cited by 127 papers in the map (data through 7 Oct 2026). First posted on arXiv 11 Feb 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 29; 2022: 32; 2023: 24; 2024: 15; 2025: 12; 2026: 15.

127 papers.

Sort:

By area:

  1. 2022From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AIcited by 299
  2. 2021EnD: Entangling and Disentangling deep representations for bias correctioncited by 105
  3. 2021Interpretable Image Classification with Differentiable Prototypes Assignmentcited by 97
  4. 2021Towards Interpreting and Mitigating Shortcut Learning Behavior of NLU Modelscited by 78
  5. 2021When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Datacited by 75
  6. 2022Concept-level Debugging of Part-Prototype Networkscited by 52
  7. 2022Leveraging Explanations in Interactive Machine Learning: An Overviewcited by 52
  8. 2021Interactive Label Cleaning with Example-based Explanationscited by 48
  9. 2022Beyond Explaining: Opportunities and Challenges of XAI-Based Model Improvementcited by 48
  10. 2021A Review on Explainability in Multimodal Deep Neural Netscited by 46
  11. 2022SwapMix: Diagnosing and Regularizing the Over-Reliance on Visual Context in Visual Question Answeringcited by 38
  12. 2021Greedy Gradient Ensemble for Robust Visual Question Answeringcited by 34
38 more
  1. 2023ProtoSeg: Interpretable Semantic Segmentation with Prototypical Partscited by 33
  2. 2022Language Prior Is Not the Only Shortcut: A Benchmark for Shortcut Learning in VQAcited by 30
  3. 2025Language Guided Concept Bottleneck Models for Interpretable Continual Learningcited by 28
  4. 2021Panoptic Narrative Groundingcited by 27
  5. 2023Interpretability Benchmark for Evaluating Spatial Misalignment of Prototypical Parts Explanationscited by 27
  6. 2022Predicting is not Understanding: Recognizing and Addressing Underspecification in Machine Learningcited by 26
  7. 2021Introspective Distillation for Robust Question Answeringcited by 25
  8. 2023Multimodal Explainable Artificial Intelligence: A Comprehensive Review of Methodological Advances and Future Research Directionscited by 25
  9. 2021Separating Skills and Concepts for Novel Visual Question Answeringcited by 24
  10. 2021Interactive Disentanglement: Learning Concepts by Interacting with their Prototype Representationscited by 24
  11. 2022Improving Visual Grounding by Encouraging Consistent Gradient-based Explanationscited by 24
  12. 2023ICICLE: Interpretable Class Incremental Continual Learningcited by 24
  13. 2023Robust Visual Question Answering: Datasets, Methods, and Future Challengescited by 23
  14. 2021ProtoMIL: Multiple Instance Learning with Prototypical Parts for Whole-Slide Image Classificationcited by 22
  15. 2021Counterfactual Samples Synthesizing and Training for Robust Visual Question Answeringcited by 22
  16. 2021Multimodal Integration of Human-Like Attention in Visual Question Answeringcited by 20
  17. 2022On Modality Bias Recognition and Reductioncited by 20
  18. 2021AdaVQA: Overcoming Language Priors with Adapted Margin Cosine Losscited by 18
  19. 2022Visual Perturbation-aware Collaborative Learning for Overcoming the Language Prior Problemcited by 18
  20. 2021LPF: A Language-Prior Feedback Objective Function for De-biased Visual Question Answeringcited by 17
  21. 2021Check It Again: Progressive Visual Question Answering via Visual Entailmentcited by 17
  22. 2023Learning by Self-Explainingcited by 17
  23. 2022Multimodal Conversational AI: A Survey of Datasets and Approachescited by 16
  24. 2022Overcoming Language Priors in Visual Question Answering via Distinguishing Superficially Similar Instancescited by 16
  25. 2023Studying How to Efficiently and Effectively Guide Models with Explanationscited by 16
  26. 2021Are Bias Mitigation Techniques for Deep Learning Effective?cited by 14
  27. 2021Explainable Machine Learning with Prior Knowledge: An Overviewcited by 14
  28. 2021Discovering the Unknown Knowns: Turning Implicit Knowledge in the Dataset into Explicit Training Examples for Visual Question Answeringcited by 14
  29. 2022VisFIS: Visual Feature Importance Supervision with Right-for-the-Right-Reason Objectivescited by 12
  30. 2022Generative Bias for Robust Visual Question Answeringcited by 12
  31. 2022Joint Answering and Explanation for Visual Commonsense Reasoningcited by 11
  32. 2024Unveiling and Mitigating Bias in Audio Visual Segmentationcited by 11
  33. 2021Detecting Spurious Correlations with Sanity Tests for Artificial Intelligence Guided Radiology Systemscited by 10
  34. 2021X-GGM: Graph Generative Modeling for Out-of-Distribution Generalization in Visual Question Answeringcited by 10
  35. 2022Prophet Attention: Predicting Attention with Future Attention for Image Captioningcited by 10
  36. 2021Answer Questions with Right Image Regions: A Visual Attention Regularization Approachcited by 9
  37. 2022Towards Robust Visual Question Answering: Making the Most of Biased Samples via Contrastive Learningcited by 9
  38. 2024Uncovering the Full Potential of Visual Grounding Methods in VQAcited by 9

50 most cited of 127 All 127 (JSON)