mlsanityData to 14 Oct 2026

Just Ask: Learning to Answer Questions from Millions of Narrated Videos

arXiv 2012.00451 ↗First posted 1 Dec 2020cs.CV cs.CL cs.LGcited by 248

Not on the map. Why?

Cited by 49 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent methods for visual question answering rely on large-scale annotated datasets. Manual annotation of questions and answers for videos, however, is tedious, expensive and prevents scalability. In this work, we propose to avoid manual annotation and generate a large-scale training dataset for video question answering making use of automatic cross-modal supervision. We leverage a question generation transformer trained on text data and use it to generate question-answer pairs from transcribed video narrations. Given narrated videos, we then automatically generate the HowToVQA69M dataset with 69M video-question-answer triplets. To handle the open vocabulary of diverse answers in this dataset, we propose a training procedure based on a contrastive loss between a video-question multi-modal transformer and an answer transformer. We introduce the zero-shot VideoQA task and show excellent results, in particular for rare answers. Furthermore, we demonstrate our method to significantly outperform the state of the art on MSRVTT-QA, MSVD-QA, ActivityNet-QA and How2QA. Finally, for a detailed evaluation we introduce iVQA, a new VideoQA dataset with reduced language biases and high-quality redundant manual annotations. Our code, datasets and trained models are available at https://antoyang.github.io/just-ask.html.

Key facts. Cited by 248 papers in the map (data through 14 Oct 2026). First posted on arXiv 1 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 10; 2022: 49; 2023: 87; 2024: 68; 2025: 29; 2026: 5.

248 papers.

Sort:

By area:

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  2. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  3. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  4. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,109
  5. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 771
  6. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 680
  7. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 615
  8. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 537
  9. 2022Multimodal Learning with Transformers: A Surveycited by 400
  10. 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understandingcited by 356
  11. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 346
  12. 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agentcited by 332
38 more
  1. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  2. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  3. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  4. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understandingcited by 229
  5. 2024CogVLM2: Visual Language Models for Image and Video Understandingcited by 217
  6. 2024STAR: A Benchmark for Situated Reasoning in Real-World Videoscited by 213
  7. 2024NVILA: Efficient Frontier Visual Language Modelscited by 209
  8. 2023Self-Chained Image-Language Model for Video Localization and Question Answeringcited by 203
  9. 2023Unmasked Teacher: Towards Training-Efficient Video Foundation Modelscited by 199
  10. 2021VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modelingcited by 188
  11. 2022mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connectionscited by 187
  12. 2023OneLLM: One Framework to Align All Modalities with Languagecited by 187
  13. 2023Making LLaMA SEE and Draw with SEED Tokenizercited by 185
  14. 2022MERLOT Reserve: Neural Script Knowledge through Vision and Language and Soundcited by 184
  15. 2023A Simple LLM Framework for Long-Range Video Question-Answeringcited by 178
  16. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learningcited by 172
  17. 2022All in One: Exploring Unified Video-Language Pre-trainingcited by 171
  18. 2023VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Datasetcited by 169
  19. 2023mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Videocited by 154
  20. 2021Align and Prompt: Video-and-Language Pre-training with Entity Promptscited by 149
  21. 2022OmniVL:One Foundation Model for Image-Language and Video-Language Taskscited by 143
  22. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 140
  23. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 140
  24. 2023Can I Trust Your Answer? Visually Grounded Video Question Answeringcited by 135
  25. 2024Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Inputcited by 134
  26. 2022Revealing Single Frame Bias for Video-and-Language Learningcited by 133
  27. 2023M3IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuningcited by 126
  28. 2023BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questionscited by 125
  29. 2024CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmarkcited by 120
  30. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  31. 2022End-to-end Generative Pretraining for Multimodal Video Captioningcited by 116
  32. 2022EgoTaskQA: Understanding Human Tasks in Egocentric Videoscited by 115
  33. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 110
  34. 2023Mixture of Cluster-conditional LoRA Experts for Vision-language Instruction Tuningcited by 106
  35. 2022TubeDETR: Spatio-Temporal Video Grounding with Transformerscited by 98
  36. 2023AssistGPT: A General Multi-modal Assistant that can Plan, Execute, Inspect, and Learncited by 97
  37. 2022VindLU: A Recipe for Effective Video-and-Language Pretrainingcited by 87
  38. 2022Bridging Video-text Retrieval with Multiple Choice Questionscited by 79

50 most cited of 248 All 248 (JSON)