mlsanityData to 7 Oct 2026

Hierarchical Question-Image Co-Attention for Visual Question Answering

arXiv 1606.00061 ↗First posted 31 May 2016cs.CV cs.CLcited by 278

Not on the map. Why?

Abstract (arXiv)

A number of recent works have proposed attention models for Visual Question Answering (VQA) that generate spatial maps highlighting image regions relevant to answering the question. In this paper, we argue that in addition to modeling "where to look" or visual attention, it is equally important to model "what words to listen to" or question attention. We present a novel co-attention model for VQA that jointly reasons about image and question attention. In addition, our model reasons about the question (and consequently the image via the co-attention mechanism) in a hierarchical fashion via a novel 1-dimensional convolution neural networks (CNN). Our model improves the state-of-the-art on the VQA dataset from 60.3% to 60.5%, and from 61.6% to 63.3% on the COCO-QA dataset. By using ResNet, the performance is further improved to 62.1% for VQA and 65.4% for COCO-QA.

Key facts. Cited by 278 papers in the map (data through 7 Oct 2026). First posted on arXiv 31 May 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 80; 2022: 67; 2023: 58; 2024: 46; 2025: 20; 2026: 7.

278 papers.

Sort:

By area:

  1. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  2. 2023NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenariocited by 258
  3. 2024Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalizationcited by 203
  4. 2022Learning to Answer Questions in Dynamic Audio-Visual Scenarioscited by 188
  5. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learningcited by 187
  6. 2022Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learningcited by 159
  7. 2021AGQA: A Benchmark for Compositional Spatio-Temporal Reasoningcited by 120
  8. 2021Counterfactual Attention Learning for Fine-Grained Visual Categorization and Re-identificationcited by 114
  9. 2021Causal Attention for Vision-Language Taskscited by 111
  10. 2021Probing Image-Language Transformers for Verb Understandingcited by 109
  11. 2021Learning to Predict Visual Attributes in the Wildcited by 95
  12. 2021Medical Visual Question Answering: A Surveycited by 93
38 more
  1. 2021TransRefer3D: Entity-and-Relation Aware Transformer for Fine-Grained 3D Visual Groundingcited by 86
  2. 2022Video Question Answering: Datasets, Algorithms and Challengescited by 84
  3. 2022ARTEMIS: Attention-based Retrieval with Text-Explicit Matching and Implicit Similaritycited by 82
  4. 2021Explainability of deep vision-based autonomous driving systems: Review and challengescited by 80
  5. 2021Multi-Modal Answer Validation for Knowledge-Based VQAcited by 78
  6. 2022Changer: Feature Interaction is What You Need for Change Detectioncited by 78
  7. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Surveycited by 74
  8. 2021Vision-Language Navigation with Random Environmental Mixupcited by 70
  9. 2023Learning to Model the World with Languagecited by 66
  10. 2023Align and Attend: Multimodal Summarization with Dual Contrastive Lossescited by 65
  11. 2022A General Survey on Attention Mechanisms in Deep Learningcited by 64
  12. 2021Decoupling the Role of Data, Attention, and Losses in Multimodal Transformerscited by 63
  13. 2021Mind Your Outliers! Investigating the Negative Impact of Outliers on Active Learning for Visual Question Answeringcited by 62
  14. 2024CAT: Enhancing Multimodal Large Language Model to Answer Questions in Dynamic Audio-Visual Scenarioscited by 53
  15. 2023FedMultimodal: A Benchmark For Multimodal Federated Learningcited by 51
  16. 2021Attention, please! A survey of Neural Attention Models in Deep Learningcited by 50
  17. 2021ExplainaBoard: An Explainable Leaderboard for NLPcited by 50
  18. 2022Learning Multi-View Aggregation In the Wild for Large-Scale 3D Semantic Segmentationcited by 47
  19. 2022GAMMA Challenge:Glaucoma grAding from Multi-Modality imAgescited by 45
  20. 2024Exploring Diverse Methods in Visual Question Answeringcited by 45
  21. 2021VX2TEXT: End-to-End Learning of Video-Based Text Generation From Multimodal Inputscited by 42
  22. 2021Adaptive Proposal Generation Network for Temporal Sentence Localization in Videoscited by 42
  23. 20213D Question Answeringcited by 41
  24. 2022RGB-T Semantic Segmentation with Location, Activation, and Sharpeningcited by 39
  25. 2021Zero-shot Visual Question Answering using Knowledge Graphcited by 37
  26. 2021Diagnosing Vision-and-Language Navigation: What Really Matterscited by 36
  27. 2021Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directionscited by 35
  28. 2021Dual-Key Multimodal Backdoors for Visual Question Answeringcited by 35
  29. 2022Exploring Optical-Flow-Guided Motion and Detection-Based Appearance for Temporal Sentence Groundingcited by 35
  30. 2021Multimodal Dialogue Response Generationcited by 32
  31. 2021Local Interpretations for Explainable Natural Language Processing: A Surveycited by 31
  32. 2021Progressively Guide to Attend: An Iterative Alignment Framework for Temporal Sentence Groundingcited by 31
  33. 2023GPT-4 Enhanced Multimodal Grounding for Autonomous Driving: Leveraging Cross-Modal Attention with Large Language Modelscited by 31
  34. 2023NuScenes-MQA: Integrated Evaluation of Captions and QA for Autonomous Driving Datasets using Markup Annotationscited by 31
  35. 2022Learning to Answer Visual Questions from Web Videoscited by 30
  36. 2022Attention Mechanism in Neural Networks: Where it Comes and Where it Goescited by 29
  37. 2023Progressive Spatio-temporal Perception for Audio-Visual Question Answeringcited by 29
  38. 2021Attentive Neural Controlled Differential Equations for Time-series Classification and Forecastingcited by 28

50 most cited of 278 All 278 (JSON)