mlsanityData to 7 Oct 2026

Leveraging Video Descriptions to Learn Video Question Answering

arXiv 1611.04021 ↗First posted 12 Nov 2016cs.CV cs.AI cs.MMcited by 55

Not on the map. Why?

Abstract (arXiv)

We propose a scalable approach to learn video-based question answering (QA): answer a "free-form natural language question" about a video content. Our approach automatically harvests a large number of videos and descriptions freely available online. Then, a large number of candidate QA pairs are automatically generated from descriptions rather than manually annotated. Next, we use these candidate QA pairs to train a number of video-based QA methods extended fromMN (Sukhbaatar et al. 2015), VQA (Antol et al. 2015), SA (Yao et al. 2015), SS (Venugopalan et al. 2015). In order to handle non-perfect candidate QA pairs, we propose a self-paced learning procedure to iteratively identify them and mitigate their effects in training. Finally, we evaluate performance on manually generated video-based QA pairs. The results show that our self-paced learning procedure is effective, and the extended SS model outperforms various baselines.

Key facts. Cited by 55 papers in the map (data through 7 Oct 2026). First posted on arXiv 12 Nov 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 16; 2022: 12; 2023: 11; 2024: 12; 2025: 3; 2026: 1.

55 papers.

Sort:

By area:

  1. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  2. 2021AGQA: A Benchmark for Compositional Spatio-Temporal Reasoningcited by 120
  3. 2022Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questionscited by 116
  4. 2022EgoTaskQA: Understanding Human Tasks in Egocentric Videoscited by 112
  5. 2021Pano-AVQA: Grounded Audio-Visual Question Answering on 360^circ Videoscited by 86
  6. 2021SUTD-TrafficQA: A Question Answering Benchmark and an Efficient Network for Video Reasoning over Traffic Eventscited by 84
  7. 2022Video Question Answering: Datasets, Algorithms and Challengescited by 84
  8. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Surveycited by 74
  9. 2021Video as Conditional Graph Hierarchy for Multi-Granular Question Answeringcited by 68
  10. 2022Invariant Grounding for Video Question Answeringcited by 68
  11. 2024Too Many Frames, Not All Useful: Efficient Strategies for Long-Form Video QAcited by 68
  12. 2021Visual Semantic Role Labeling for Video Understandingcited by 60
38 more
  1. 2021On Semantic Similarity in Video Retrievalcited by 45
  2. 2024MMWorld: Towards Multi-discipline Multi-faceted World Model Evaluation in Videoscited by 43
  3. 2024Video-Language Understanding: A Survey from Model Architecture, Model Training, and Data Perspectivescited by 37
  4. 2022Learning to Answer Visual Questions from Web Videoscited by 30
  5. 2022M^4I: Multi-modal Models Membership Inferencecited by 30
  6. 2023A Review of Deep Learning for Video Captioningcited by 28
  7. 2024Understanding Long Videos with Multimodal Language Modelscited by 25
  8. 2023Connecting Vision and Language with Video Localized Narrativescited by 24
  9. 2023From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunitiescited by 24
  10. 2024TraveLER: A Modular Multi-LMM Agent Framework for Video Question-Answeringcited by 24
  11. 2022Equivariant and Invariant Grounding for Video Question Answeringcited by 22
  12. 2023Summarize the Past to Predict the Future: Natural Language Descriptions of Context Boost Multimodal Object Interaction Anticipationcited by 21
  13. 2021Question Answering Survey: Directions, Challenges, Datasets, Evaluation Matricescited by 17
  14. 2022Learning to Retrieve Videos by Asking Questionscited by 17
  15. 2023Learning Situation Hyper-Graphs for Video Question Answeringcited by 15
  16. 2023ANetQA: A Large-scale Benchmark for Fine-grained Compositional Reasoning over Untrimmed Videoscited by 15
  17. 2021Video Question Answering with Phrases via Semantic Rolescited by 13
  18. 2024CausalChaos! Dataset for Comprehensive Causal Action Question Answering Over Longer Causal Chains Grounded in Dynamic Visual Scenescited by 13
  19. 2021Recent Advances in Video Question Answering: A Review of Datasets and Methodscited by 12
  20. 2024Progress-Aware Video Frame Captioningcited by 10
  21. 2022WildQA: In-the-Wild Video Question Answeringcited by 9
  22. 20223MASSIV: Multilingual, Multimodal and Multi-Aspect dataset of Social Media Short Videoscited by 8
  23. 2023Long Story Short: a Summarize-then-Search Method for Long Video Question Answeringcited by 7
  24. 2025An LMM for Efficient Video Understanding via Reinforced Compression of Video Cubescited by 7
  25. 2021End-to-End Video Question-Answer Generation with Generator-Pretester Networkcited by 6
  26. 2023Language Models are Causal Knowledge Extractors for Zero-shot Video Question Answeringcited by 6
  27. 2023Learning to Summarize and Answer Questions about a Virtual Robot's Past Actionscited by 5
  28. 2024Prompting Video-Language Foundation Models with Domain-specific Fine-grained Heuristics for Video Question Answeringcited by 5
  29. 2021iReason: Multimodal Commonsense Reasoning using Videos and Natural Language with Interpretabilitycited by 4
  30. 2022NEWSKVQA: Knowledge-Aware News Video Question Answeringcited by 4
  31. 2021Relation-aware Hierarchical Attention Framework for Video Question Answeringcited by 3
  32. 2021TrUMAn: Trope Understanding in Movies and Animationscited by 3
  33. 2023VideoOFA: Two-Stage Pre-Training for Video-to-Text Generationcited by 3
  34. 2024Multi-granularity Contrastive Cross-modal Collaborative Generation for End-to-End Long-term Video Question Answeringcited by 3
  35. 2021Simple Dialogue System with AUDITEDcited by 2
  36. 2022Learning Fine-Grained Visual Understanding for Video Question Answering via Decoupling Spatial-Temporal Modelingcited by 2
  37. 2021Narration Generation for Cartoon Videoscited by 1
  38. 2024Investigating Video Reasoning Capability of Large Language Models with Tropes in Moviescited by 1

50 most cited of 55 All 55 (JSON)