mlsanityData to 14 Oct 2026

TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering

arXiv 1704.04497 ↗First posted 14 Apr 2017cs.CVcited by 378

Not on the map. Why?

Cited by 99 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Vision and language understanding has emerged as a subject undergoing intense study in Artificial Intelligence. Among many tasks in this line of research, visual question answering (VQA) has been one of the most successful ones, where the goal is to learn a model that understands visual content at region-level details and finds their associations with pairs of questions and answers in the natural language form. Despite the rapid progress in the past few years, most existing work in VQA have focused primarily on images. In this paper, we focus on extending VQA to the video domain and contribute to the literature in three important ways. First, we propose three new tasks designed specifically for video VQA, which require spatio-temporal reasoning from videos to answer questions correctly. Next, we introduce a new large-scale dataset for video VQA named TGIF-QA that extends existing VQA work with our new tasks. Finally, we propose a dual-LSTM based approach with both spatial and temporal attention, and show its effectiveness over conventional VQA techniques through empirical evaluations.

Key facts. Cited by 378 papers in the map (data through 14 Oct 2026). First posted on arXiv 14 Apr 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 46; 2022: 52; 2023: 59; 2024: 86; 2025: 84; 2026: 51.

378 papers.

Sort:

By area:

  1. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  2. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  3. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,109
  4. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  5. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 744
  6. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 680
  7. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 642
  8. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 547
  9. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 537
  10. 2024Long Context Transfer from Language to Visioncited by 515
  11. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  12. 2024LVBench: An Extreme Long Video Understanding Benchmarkcited by 410
38 more
  1. 2024SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inferencecited by 391
  2. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 371
  3. 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 363
  4. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 326
  5. 2024VisionZip: Longer is Better but Not Necessary in Vision Language Modelscited by 313
  6. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  7. 2022X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrievalcited by 280
  8. 2023NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenariocited by 262
  9. 2024VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videoscited by 251
  10. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  11. 2021Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptionscited by 226
  12. 2024CogVLM2: Visual Language Models for Image and Video Understandingcited by 217
  13. 2024STAR: A Benchmark for Situated Reasoning in Real-World Videoscited by 213
  14. 2023Video Understanding with Large Language Models: A Surveycited by 206
  15. 2022Learning to Answer Questions in Dynamic Audio-Visual Scenarioscited by 191
  16. 2021VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modelingcited by 188
  17. 2024MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understandingcited by 182
  18. 2022All in One: Exploring Unified Video-Language Pre-trainingcited by 171
  19. 2022Revisiting the "Video" in Video-Language Understandingcited by 169
  20. 2023VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Datasetcited by 169
  21. 2024Beyond Text-Visual Attention: Exploiting Visual Cues for Effective Token Pruning in VLMscited by 157
  22. 2025Stop Looking for Important Tokens in Multimodal Language Models: Duplication Matters Morecited by 155
  23. 2023mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Videocited by 154
  24. 2021Align and Prompt: Video-and-Language Pre-training with Entity Promptscited by 149
  25. 2024Direct Preference Optimization of Video Large Multimodal Models from Language Model Rewardcited by 147
  26. 2024StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understandingcited by 144
  27. 2025OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video Understanding?cited by 142
  28. 2024Tarsier: Recipes for Training and Evaluating Large Video Description Modelscited by 141
  29. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 140
  30. 2024Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inferencecited by 139
  31. 2025MMVU: Measuring Expert-Level Multi-Discipline Video Understandingcited by 137
  32. 2023Can I Trust Your Answer? Visually Grounded Video Question Answeringcited by 135
  33. 2024Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Inputcited by 134
  34. 2022Revealing Single Frame Bias for Video-and-Language Learningcited by 133
  35. 2021Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Surveycited by 130
  36. 2021AGQA: A Benchmark for Compositional Spatio-Temporal Reasoningcited by 120
  37. 2024MiniGPT4-Video: Advancing Multimodal LLMs for Video Understanding with Interleaved Visual-Textual Tokenscited by 118
  38. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117

50 most cited of 378 All 378 (JSON)