mlsanityData to 7 Oct 2026

Video Question Answering: Datasets, Algorithms and Challenges

arXiv 2203.01225 ↗First posted 2 Mar 2022cs.CVcited by 84

Not on the map. Why?

Abstract (arXiv)

Video Question Answering (VideoQA) aims to answer natural language questions according to the given videos. It has earned increasing attention with recent research trends in joint vision and language understanding. Yet, compared with ImageQA, VideoQA is largely underexplored and progresses slowly. Although different algorithms have continually been proposed and shown success on different VideoQA datasets, we find that there lacks a meaningful survey to categorize them, which seriously impedes its advancements. This paper thus provides a clear taxonomy and comprehensive analyses to VideoQA, focusing on the datasets, algorithms, and unique challenges. We then point out the research trend of studying beyond factoid QA to inference QA towards the cognition of video contents, Finally, we conclude some promising directions for future exploration.

Key facts. Cited by 84 papers in the map (data through 7 Oct 2026). Builds on 73 of its 128 references that are in the map. First posted on arXiv 2 Mar 2022.

Builds on

73 of 128 references.

References · 73

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  6. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  7. 2015VQA: Visual Question Answeringcited by 2,489
  8. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  9. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  10. 2021Video Swin Transformercited by 1,090
  11. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  12. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
38 more
  1. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  2. 2019CLEVRER: CoLlision Events for Video REpresentation and Reasoningcited by 475
  3. 2018TVQA: Localized, Compositional Video Question Answeringcited by 463
  4. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  5. 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
  6. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  7. 2015MovieQA: Understanding Stories in Movies through Question-Answeringcited by 304
  8. 2018Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understandingcited by 304
  9. 2016Hierarchical Question-Image Co-Attention for Visual Question Answeringcited by 278
  10. 2020ActBERT: Learning Global-Local Video-Text Representationscited by 267
  11. 2016SCA-CNN: Spatial and Channel-wise Attention in Convolutional Networks for Image Captioningcited by 219
  12. 2021VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modelingcited by 187
  13. 2020Hierarchical Conditional Relation Networks for Video Question Answeringcited by 163
  14. 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
  15. 2019TVQA+: Spatio-Temporal Grounding for Video Question Answeringcited by 153
  16. 2019Heterogeneous Memory Enhanced Multimodal Attention Model for Video Question Answeringcited by 146
  17. 2021AGQA: A Benchmark for Compositional Spatio-Temporal Reasoningcited by 120
  18. 2019Learning to Compose and Reason with Language Tree Structures for Visual Groundingcited by 98
  19. 2018Motion-Appearance Co-Memory Networks for Video Question Answeringcited by 95
  20. 2021Pano-AVQA: Grounded Audio-Visual Question Answering on 360^circ Videoscited by 86
  21. 2021SUTD-TrafficQA: A Question Answering Benchmark and an Efficient Network for Video Reasoning over Traffic Eventscited by 84
  22. 2021Deconfounded Video Moment Retrieval with Causal Interventioncited by 84
  23. 2020Location-aware Graph Convolutional Networks for Video Question Answeringcited by 80
  24. 2021Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Modelscited by 74
  25. 2021Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoningcited by 68
  26. 2021Video as Conditional Graph Hierarchy for Multi-Granular Question Answeringcited by 68
  27. 2017DeepStory: Video Story QA by Deep Embedded Memory Networkscited by 67
  28. 2021Human-Adversarial Visual Question Answeringcited by 57
  29. 2021Bridge to Answer: Structure-aware Graph Interaction Network for Video Question Answeringcited by 56
  30. 2016Leveraging Video Descriptions to Learn Video Question Answeringcited by 55
  31. 2021Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Languagecited by 55
  32. 2019KnowIT VQA: Answering Knowledge-Based Questions about Videoscited by 53
  33. 2020Look Before you Speak: Visually Contextualized Utterancescited by 53
  34. 2020DramaQA: Character-Centered Video Story Understanding with Hierarchical QAcited by 45
  35. 2016MarioQA: Answering Questions by Watching Gameplay Videoscited by 42
  36. 2016A dataset and exploration of models for understanding video data through fill-in-the-blank question-answeringcited by 41
  37. 2020Modality Shifting Attention Network for Multi-modal Video Question Answeringcited by 38
  38. 2020Video2Commonsense: Generating Commonsense Descriptions to Enrich Video Captioningcited by 36

50 most cited of 73 All 73 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 20; 2024: 27; 2025: 21; 2026: 12.

84 papers.

Sort:

By area:

  1. 2023Can I Trust Your Answer? Visually Grounded Video Question Answeringcited by 133
  2. 2024MoReVQA: Exploring Modular Reasoning Models for Video Question Answeringcited by 74
  3. 2022Video Graph Transformer for Video Question Answeringcited by 63
  4. 2025LION-FS: Fast & Slow Video-Language Thinker as Online Video Assistantcited by 56
  5. 2024Embodied-RAG: General Non-parametric Embodied Memory for Retrieval and Generationcited by 44
  6. 2024Video-Language Understanding: A Survey from Model Architecture, Model Training, and Data Perspectivescited by 37
  7. 2024VideoCoT: A Video Chain-of-Thought Dataset with Active Annotation Toolcited by 35
  8. 2025CausalVQA: A Physically Grounded Causal Reasoning Benchmark for Video Modelscited by 35
  9. 2023Contrastive Video Question Answering via Video Graph Transformercited by 32
  10. 2024VideoQA in the Era of LLMs: An Empirical Studycited by 29
  11. 2023Retrieving-to-Answer: Zero-Shot Video Question Answering with Frozen Large Language Modelscited by 27
  12. 2023Connecting Vision and Language with Video Localized Narrativescited by 24
38 more
  1. 2023From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunitiescited by 24
  2. 2024Neptune: The Long Orbit to Benchmarking Long Video Understandingcited by 24
  3. 2022Equivariant and Invariant Grounding for Video Question Answeringcited by 22
  4. 2023Discovering Spatio-Temporal Rationales for Video Question Answeringcited by 21
  5. 2024Assessing Modality Bias in Video Question Answering Benchmarks with Multimodal Large Language Modelscited by 21
  6. 2025EgoCross: Benchmarking Multimodal Large Language Models for Cross-Domain Egocentric Video Question Answeringcited by 19
  7. 2025A Survey on Video Temporal Grounding with Multimodal Large Language Modelcited by 19
  8. 2023VaQuitA: Enhancing Alignment in LLM-Assisted Video Understandingcited by 17
  9. 2025Time Blindness: Why Video-Language Models Can't See What Humans Can?cited by 17
  10. 2023Learning Situation Hyper-Graphs for Video Question Answeringcited by 15
  11. 2024CausalChaos! Dataset for Comprehensive Causal Action Question Answering Over Longer Causal Chains Grounded in Dynamic Visual Scenescited by 13
  12. 2024TV-TREES: Multimodal Entailment Trees for Neuro-Symbolic Video Reasoningcited by 12
  13. 2024Can 3D Vision-Language Models Truly Understand Natural Language?cited by 12
  14. 2025Vid2Coach: Transforming How-To Videos into Task Assistantscited by 12
  15. 2024Step Differences in Instructional Videocited by 11
  16. 2024Question-Answering Dense Video Eventscited by 11
  17. 2025MAGNET: A Multi-agent Framework for Finding Audio-Visual Needles by Reasoning over Multi-Video Haystackscited by 11
  18. 2024Grounding Partially-Defined Events in Multimodal Datacited by 10
  19. 2022WildQA: In-the-Wild Video Question Answeringcited by 9
  20. 2023ChatGPT-Like Large-Scale Foundation Models for Prognostics and Health Management: A Survey and Roadmapscited by 8
  21. 2023Dissecting Multimodality in VideoQA Transformer Models by Impairing Modality Fusioncited by 8
  22. 2023Benchmarks for Physical Reasoning AIcited by 8
  23. 2024Question-Instructed Visual Descriptions for Zero-Shot Video Question Answeringcited by 8
  24. 2025VidEvent: A Large Dataset for Understanding Dynamic Evolution of Events in Videoscited by 8
  25. 2023Redundancy-aware Transformer for Video Question Answeringcited by 7
  26. 2025AURA: A Fine-Grained Benchmark and Decomposed Metric for Audio-Visual Reasoningcited by 7
  27. 2022MRTNet: Multi-Resolution Temporal Network for Video Sentence Groundingcited by 6
  28. 2023TG-VQA: Ternary Game of Video Question Answeringcited by 6
  29. 2024Navigating the Landscape of Hint Generation Research: From the Past to the Futurecited by 6
  30. 2024DocTabQA: Answering Questions from Long Documents Using Tablescited by 6
  31. 2023Learning to Summarize and Answer Questions about a Virtual Robot's Past Actionscited by 5
  32. 2023Generating Context-Aware Natural Answers for Questions in 3D Scenescited by 5
  33. 2024BDIQA: A New Dataset for Video Question Answering to Explore Cognitive Reasoning through Theory of Mindcited by 5
  34. 2025ViQAgent: Zero-Shot Video Question Answering via Agent with Open-Vocabulary Grounding Validationcited by 5
  35. 2025Gather and Trace: Rethinking Video TextVQA from an Instance-oriented Perspectivecited by 5
  36. 2024Episodic Memory Verbalization using Hierarchical Representations of Life-Long Robot Experiencecited by 4
  37. 2024Hierarchical Banzhaf Interaction for General Video-Language Representation Learningcited by 4
  38. 2025Mitigating Easy Option Bias in Multiple-Choice Question Answeringcited by 4

50 most cited of 84 All 84 (JSON)