mlsanityData to 7 Oct 2026

AGQA: A Benchmark for Compositional Spatio-Temporal Reasoning

arXiv 2103.16002 ↗First posted 30 Mar 2021cs.CV cs.CLcited by 120

Not on the map. Why?

Cited by 34 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Visual events are a composition of temporal actions involving actors spatially interacting with objects. When developing computer vision models that can reason about compositional spatio-temporal events, we need benchmarks that can analyze progress and uncover shortcomings. Existing video question answering benchmarks are useful, but they often conflate multiple sources of error into one accuracy metric and have strong biases that models can exploit, making it difficult to pinpoint model weaknesses. We present Action Genome Question Answering (AGQA), a new benchmark for compositional spatio-temporal reasoning. AGQA contains 192M unbalanced question answer pairs for 9.6K videos. We also provide a balanced subset of 3.9M question answer pairs, 3 orders of magnitude larger than existing benchmarks, that minimizes bias by balancing the answer distributions and types of question structures. Although human evaluators marked 86.02% of our question-answer pairs as correct, the best model achieves only 47.74% accuracy. In addition, AGQA introduces multiple training/test splits to test for various reasoning abilities, including generalization to novel compositions, to indirect references, and to more compositional steps. Using AGQA, we evaluate modern visual reasoning systems, demonstrating that the best models barely perform better than non-visual baselines exploiting linguistic biases and that none of the existing models generalize to novel compositions unseen during training.

Key facts. Cited by 120 papers in the map (data through 7 Oct 2026). Builds on 41 of its 65 references that are in the map. First posted on arXiv 30 Mar 2021.

Builds on

41 of 65 references.

References · 41

  1. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  2. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  3. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  4. 2015VQA: Visual Question Answeringcited by 2,489
  5. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  6. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  7. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  8. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  9. 2016Hollywood in Homes: Crowdsourcing Data Collection for Activity Understandingcited by 535
  10. 2019CLEVRER: CoLlision Events for Video REpresentation and Reasoningcited by 475
  11. 2018TVQA: Localized, Compositional Video Question Answeringcited by 463
  12. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
29 more
  1. 2019The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervisioncited by 369
  2. 2014Memory Networkscited by 354
  3. 2016Visual Relationship Detection with Language Priorscited by 333
  4. 2018Image Generation from Scene Graphscited by 324
  5. 2015Visual7W: Grounded Question Answering in Imagescited by 315
  6. 2015MovieQA: Understanding Stories in Movies through Question-Answeringcited by 304
  7. 2019Measuring Compositional Generalization: A Comprehensive Method on Realistic Datacited by 280
  8. 2016Hierarchical Question-Image Co-Attention for Visual Question Answeringcited by 278
  9. 2019Action Genome: Actions as Composition of Spatio-temporal Scene Graphscited by 249
  10. 2018Compositional Attention Networks for Machine Reasoningcited by 183
  11. 2020Hierarchical Conditional Relation Networks for Video Question Answeringcited by 163
  12. 2019REPAIR: Removing Representation Bias by Dataset Resamplingcited by 157
  13. 2019TVQA+: Spatio-Temporal Grounding for Video Question Answeringcited by 153
  14. 2020Counterfactual Samples Synthesizing for Robust Visual Question Answeringcited by 153
  15. 2019Heterogeneous Memory Enhanced Multimodal Attention Model for Video Question Answeringcited by 146
  16. 2017Inferring and Executing Programs for Visual Reasoningcited by 130
  17. 2017Learning to Reason: End-to-End Module Networks for Visual Question Answeringcited by 125
  18. 2019Specifying Object Attributes and Relations in Interactive Scene Generationcited by 76
  19. 2017DeepStory: Video Story QA by Deep Embedded Memory Networkscited by 67
  20. 2016Leveraging Video Descriptions to Learn Video Question Answeringcited by 55
  21. 2016Semantic Compositional Networks for Visual Captioningcited by 50
  22. 2019Compositional Video Predictioncited by 43
  23. 2016MarioQA: Answering Questions by Watching Gameplay Videoscited by 42
  24. 2016A dataset and exploration of models for understanding video data through fill-in-the-blank question-answeringcited by 41
  25. 2019Scene Graph Prediction with Limited Labelscited by 32
  26. 2018Referring Relationshipscited by 29
  27. 2017Teaching Compositionality to CNNscited by 18
  28. 2020What Gives the Answer Away? Question Answering Bias Analysis on Video QA Datasetscited by 13
  29. 2018VQA with no questions-answers trainingcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 16; 2023: 20; 2024: 37; 2025: 27; 2026: 19.

120 papers.

Sort:

By area:

  1. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  2. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 550
  3. 2024Long Context Transfer from Language to Visioncited by 502
  4. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 416
  5. 2022SQA3D: Situated Question Answering in 3D Scenescited by 303
  6. 2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionalitycited by 238
  7. 2024STAR: A Benchmark for Situated Reasoning in Real-World Videoscited by 213
  8. 2024MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understandingcited by 180
  9. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?cited by 177
  10. 2024Direct Preference Optimization of Video Large Multimodal Models from Language Model Rewardcited by 145
  11. 2022EgoTaskQA: Understanding Human Tasks in Egocentric Videoscited by 112
  12. 2022Video Question Answering: Datasets, Algorithms and Challengescited by 84
38 more
  1. 2024MoReVQA: Exploring Modular Reasoning Models for Video Question Answeringcited by 74
  2. 2022ComPhy: Compositional Physical Reasoning of Objects and Events from Videoscited by 61
  3. 2022Compositional Temporal Grounding with Structured Variational Cross-Graph Correspondence Learningcited by 59
  4. 2025LLaVA-ST: A Multimodal Large Language Model for Fine-Grained Spatial-Temporal Understandingcited by 57
  5. 2022MIST: Multi-modal Iterative Spatial-Temporal Transformer for Long-form Video Question Answeringcited by 55
  6. 2023COLA: A Benchmark for Compositional Text-to-image Retrievalcited by 53
  7. 2023Test of Time: Instilling Video-Language Models with a Sense of Timecited by 51
  8. 2023Benchmarks for Automated Commonsense Reasoning: A Surveycited by 40
  9. 2023AutoEval-Video: An Automatic Benchmark for Assessing Large Vision Language Models in Open-Ended Video Question Answeringcited by 40
  10. 2024Enhancing Video-LLM Reasoning via Agent-of-Thoughts Distillationcited by 36
  11. 2023FunQA: Towards Surprising Video Comprehensioncited by 35
  12. 2024Needle In A Video Haystack: A Scalable Synthetic Evaluator for Video MLLMscited by 35
  13. 2022Causal Reasoning Meets Visual Representation Learning: A Prospective Studycited by 24
  14. 2024Video-STaR: Self-Training Enables Video Instruction Tuning with Any Supervisioncited by 24
  15. 2024Neptune: The Long Orbit to Benchmarking Long Video Understandingcited by 24
  16. 2025V-ReasonBench: Toward Unified Reasoning Benchmark Suite for Video Generation Modelscited by 22
  17. 2024Coarse Correspondences Boost Spatial-Temporal Reasoning in Multimodal Language Modelcited by 21
  18. 2024The Hard Positive Truth about Vision-Language Compositionalitycited by 21
  19. 2024ProVision: Programmatically Scaling Vision-centric Instruction Data for Multimodal Language Modelscited by 21
  20. 2025Unleashing Hour-Scale Video Training for Long Video-Language Understandingcited by 21
  21. 2024Towards Event-oriented Long Video Understandingcited by 20
  22. 2021COMPOSER: Compositional Reasoning of Group Activity in Videos with Keypoint-Only Modalitycited by 19
  23. 2022Locate before Answering: Answer Guided Question Localization for Video Question Answeringcited by 19
  24. 2023Variational Cross-Graph Reasoning and Adaptive Structured Semantics Learning for Compositional Temporal Groundingcited by 19
  25. 2024DAM: Dynamic Adapter Merging for Continual Video QA Learningcited by 18
  26. 2024What to Say and When to Say it: Live Fitness Coaching as a Testbed for Situated Interactioncited by 18
  27. 2024VideoAutoArena: An Automated Arena for Evaluating Large Multimodal Models in Video Analysis through User Simulationcited by 18
  28. 2024Iterated Learning Improves Compositionality in Large Vision-Language Modelscited by 17
  29. 2024WorldQA: Multimodal World Knowledge in Videos through Long-Chain Reasoningcited by 16
  30. 2022Measuring Compositional Consistency for Video Question Answeringcited by 15
  31. 2023Learning Situation Hyper-Graphs for Video Question Answeringcited by 15
  32. 2023ANetQA: A Large-scale Benchmark for Fine-grained Compositional Reasoning over Untrimmed Videoscited by 15
  33. 2024Efficient Temporal Extrapolation of Multimodal Large Language Models with Temporal Grounding Bridgecited by 14
  34. 2026VideoSeek: Long-Horizon Video Agent with Tool-Guided Seekingcited by 14
  35. 2022CRIPP-VQA: Counterfactual Reasoning about Implicit Physical Properties via Video Question Answeringcited by 13
  36. 2024CausalChaos! Dataset for Comprehensive Causal Action Question Answering Over Longer Causal Chains Grounded in Dynamic Visual Scenescited by 13
  37. 2023SViTT: Temporal Learning of Sparse Video-Text Transformerscited by 11
  38. 2023CLEVRER-Humans: Describing Physical and Causal Events the Human Waycited by 11

50 most cited of 120 All 120 (JSON)