mlsanityData to 14 Oct 2026

Heterogeneous Memory Enhanced Multimodal Attention Model for Video Question Answering

arXiv 1904.04357 ↗First posted 8 Apr 2019cs.CVcited by 146

Not on the map. Why?

Abstract (arXiv)

In this paper, we propose a novel end-to-end trainable Video Question Answering (VideoQA) framework with three major components: 1) a new heterogeneous memory which can effectively learn global context information from appearance and motion features; 2) a redesigned question memory which helps understand the complex semantics of question and highlights queried subjects; and 3) a new multimodal fusion layer which performs multi-step reasoning by attending to relevant visual and textual hints with self-updated attention. Our VideoQA model firstly generates the global context-aware visual and textual features respectively by interacting current inputs with memory contents. After that, it makes the attentional fusion of the multimodal visual and textual representations to infer the correct answer. Multiple cycles of reasoning can be made to iteratively refine attention weights of the multimodal data and improve the final representation of the QA pair. Experimental results demonstrate our approach achieves state-of-the-art performance on four VideoQA benchmark datasets.

Key facts. Cited by 146 papers in the map (data through 14 Oct 2026). First posted on arXiv 8 Apr 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 31; 2022: 35; 2023: 30; 2024: 26; 2025: 16; 2026: 8.

146 papers.

Sort:

By area:

  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  2. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 744
  3. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  4. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  5. 2021Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptionscited by 226
  6. 2022Learning to Answer Questions in Dynamic Audio-Visual Scenarioscited by 191
  7. 2021VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modelingcited by 188
  8. 2022All in One: Exploring Unified Video-Language Pre-trainingcited by 171
  9. 2021Align and Prompt: Video-and-Language Pre-training with Entity Promptscited by 149
  10. 2023EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbonecited by 130
  11. 2021AGQA: A Benchmark for Compositional Spatio-Temporal Reasoningcited by 120
  12. 2022EgoTaskQA: Understanding Human Tasks in Egocentric Videoscited by 115
38 more
  1. 2021MeGA-CDA: Memory Guided Attention for Category-Aware Unsupervised Domain Adaptive Object Detectioncited by 106
  2. 2025EgoLife: Towards Egocentric Life Assistantcited by 91
  3. 2021SUTD-TrafficQA: A Question Answering Benchmark and an Efficient Network for Video Reasoning over Traffic Eventscited by 84
  4. 2022Video Question Answering: Datasets, Algorithms and Challengescited by 84
  5. 2021Divide-and-Assemble: Learning Block-wise Memory for Unsupervised Anomaly Detectioncited by 83
  6. 2022An Empirical Study of End-to-End Video-Language Transformers with Masked Visual Modelingcited by 74
  7. 2021Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoningcited by 68
  8. 2021Video as Conditional Graph Hierarchy for Multi-Granular Question Answeringcited by 68
  9. 2022Invariant Grounding for Video Question Answeringcited by 68
  10. 2021Enriching Local and Global Contexts for Temporal Action Localizationcited by 64
  11. 2022Video Graph Transformer for Video Question Answeringcited by 63
  12. 2024DoraemonGPT: Toward Understanding Dynamic Scenes with Large Language Models (Exemplified as A Video Agent)cited by 58
  13. 2022Cross-Modal Causal Relational Reasoning for Event-Level Visual Question Answeringcited by 57
  14. 2021Bridge to Answer: Structure-aware Graph Interaction Network for Video Question Answeringcited by 56
  15. 2021Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Languagecited by 55
  16. 2022MIST: Multi-modal Iterative Spatial-Temporal Transformer for Long-form Video Question Answeringcited by 55
  17. 2024CAT: Enhancing Multimodal Large Language Model to Answer Questions in Dynamic Audio-Visual Scenarioscited by 53
  18. 2022From Representation to Reasoning: Towards both Evidence and Commonsense Reasoning for Video Question-Answeringcited by 51
  19. 2021Temporal Memory Attention for Video Semantic Segmentationcited by 50
  20. 2021HVPR: Hybrid Voxel-Point Representation for Single-stage 3D Object Detectioncited by 47
  21. 2022Clover: Towards A Unified Video-Language Alignment and Fusion Modelcited by 41
  22. 2024Sports-QA: A Large-Scale Video Question Answering Benchmark for Complex and Professional Sportscited by 41
  23. 2023Zero-Shot Video Question Answering with Procedural Programscited by 37
  24. 2024Video-Language Understanding: A Survey from Model Architecture, Model Training, and Data Perspectivescited by 37
  25. 2021DualVGR: A Dual-Visual Graph Reasoning Unit for Video Question Answeringcited by 35
  26. 2021SQUID: Deep Feature In-Painting for Unsupervised Anomaly Detectioncited by 34
  27. 2023Contrastive Video Question Answering via Video Graph Transformercited by 32
  28. 2022(2.5+1)D Spatio-Temporal Scene Graphs for Video Question Answeringcited by 30
  29. 2022Learning to Answer Visual Questions from Web Videoscited by 30
  30. 2021Hierarchical Object-oriented Spatio-Temporal Reasoning for Video Question Answeringcited by 29
  31. 2021A Survey on Temporal Sentence Grounding in Videoscited by 29
  32. 2023Progressive Spatio-temporal Perception for Audio-Visual Question Answeringcited by 29
  33. 2021Attend What You Need: Motion-Appearance Synergistic Networks for Video Question Answeringcited by 26
  34. 2023ViLA: Efficient Video-Language Alignment for Video Question Answeringcited by 23
  35. 2022Equivariant and Invariant Grounding for Video Question Answeringcited by 22
  36. 2023Tem-adapter: Adapting Image-Text Pretraining for Video Question Answercited by 22
  37. 2022SimVTP: Simple Video Text Pre-training with Masked Autoencoderscited by 21
  38. 2023Discovering Spatio-Temporal Rationales for Video Question Answeringcited by 21

50 most cited of 146 All 146 (JSON)