mlsanityData to 7 Oct 2026

NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actions

arXiv 2105.08276 ↗First posted 18 May 2021cs.CV cs.AIcited by 735

Not on the map. Why?

Cited by 269 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce NExT-QA, a rigorously designed video question answering (VideoQA) benchmark to advance video understanding from describing to explaining the temporal actions. Based on the dataset, we set up multi-choice and open-ended QA tasks targeting causal action reasoning, temporal action reasoning, and common scene comprehension. Through extensive analysis of baselines and established VideoQA techniques, we find that top-performing methods excel at shallow scene descriptions but are weak in causal and temporal action reasoning. Furthermore, the models that are effective on multi-choice QA, when adapted to open-ended QA, still struggle in generalizing the answers. This raises doubt on the ability of these models to reason and highlights possibilities for improvement. With detailed results for different question types and heuristic observations for future works, we hope NExT-QA will guide the next generation of VQA research to go beyond superficial scene description towards a deeper understanding of videos. (The dataset and related resources are available at https://github.com/doc-doc/NExT-QA.git)

Key facts. Cited by 735 papers in the map (data through 7 Oct 2026). Builds on 43 of its 69 references that are in the map. First posted on arXiv 18 May 2021.

Builds on

43 of 69 references.

References · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  6. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  7. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  8. 2015VQA: Visual Question Answeringcited by 2,489
  9. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  10. 2017The Kinetics Human Action Video Datasetcited by 1,894
  11. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  12. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,067
31 more
  1. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 852
  2. 2017Dense-Captioning Events in Videoscited by 789
  3. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  4. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  5. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  6. 2017Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?cited by 551
  7. 2016Temporal Convolutional Networks for Action Segmentation and Detectioncited by 504
  8. 2019CLEVRER: CoLlision Events for Video REpresentation and Reasoningcited by 475
  9. 2018TVQA: Localized, Compositional Video Question Answeringcited by 463
  10. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  11. 2017Temporal Relational Reasoning in Videoscited by 324
  12. 2015MovieQA: Understanding Stories in Movies through Question-Answeringcited by 304
  13. 2016Hierarchical Question-Image Co-Attention for Visual Question Answeringcited by 278
  14. 2020ActBERT: Learning Global-Local Video-Text Representationscited by 267
  15. 2015YFCC100M: The New Data in Multimedia Researchcited by 234
  16. 2018End-to-End Dense Video Captioning with Masked Transformercited by 214
  17. 2020Hierarchical Conditional Relation Networks for Video Question Answeringcited by 163
  18. 2015Sequence to Sequence -- Video to Textcited by 157
  19. 2019Heterogeneous Memory Enhanced Multimodal Attention Model for Video Question Answeringcited by 146
  20. 2016TGIF: A New Dataset and Benchmark on Animated GIF Descriptioncited by 124
  21. 2014A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain Inputcited by 112
  22. 2018Motion-Appearance Co-Memory Networks for Video Question Answeringcited by 95
  23. 2015Describing Videos by Exploiting Temporal Structurecited by 88
  24. 2020Location-aware Graph Convolutional Networks for Video Question Answeringcited by 80
  25. 2019Joint Reasoning for Temporal and Causal Relationscited by 60
  26. 2016Leveraging Video Descriptions to Learn Video Question Answeringcited by 55
  27. 2016A dataset and exploration of models for understanding video data through fill-in-the-blank question-answeringcited by 41
  28. 2017Video Question Answering via Attribute-Augmented Attention Network Learningcited by 39
  29. 2020Video2Commonsense: Generating Commonsense Descriptions to Enrich Video Captioningcited by 36
  30. 2020On Modality Bias in the TVQA Datasetcited by 24
  31. 2020Visual Relation Grounding in Videoscited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 24; 2023: 69; 2024: 175; 2025: 279; 2026: 186.

735 papers.

Sort:

By area:

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  3. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  4. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  5. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  6. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  7. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understandingcited by 644
  8. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  9. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 571
  10. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 550
  11. 2024Long Context Transfer from Language to Visioncited by 502
  12. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
38 more
  1. 2024LVBench: An Extreme Long Video Understanding Benchmarkcited by 402
  2. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
  3. 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 352
  4. 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understandingcited by 341
  5. 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agentcited by 323
  6. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 320
  7. 2023Perception Test: A Diagnostic Benchmark for Multimodal Video Modelscited by 310
  8. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 303
  9. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 285
  10. 2024MLVU: Benchmarking Multi-task Long Video Understandingcited by 275
  11. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 254
  12. 2024VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videoscited by 242
  13. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  14. 2024VideoAgent: A Memory-augmented Multimodal Agent for Video Understandingcited by 224
  15. 2024CogVLM2: Visual Language Models for Image and Video Understandingcited by 214
  16. 2024MANTIS: Interleaved Multi-Image Instruction Tuningcited by 212
  17. 2026Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Rankingcited by 211
  18. 2024Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understandingcited by 210
  19. 2023Video Understanding with Large Language Models: A Surveycited by 203
  20. 2023Self-Chained Image-Language Model for Video Localization and Question Answeringcited by 200
  21. 2022Learning to Answer Questions in Dynamic Audio-Visual Scenarioscited by 188
  22. 2023OneLLM: One Framework to Align All Modalities with Languagecited by 185
  23. 2023Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Futurecited by 184
  24. 2023Making LLaMA SEE and Draw with SEED Tokenizercited by 184
  25. 2025DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Modelscited by 177
  26. 2023A Simple LLM Framework for Long-Range Video Question-Answeringcited by 176
  27. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learningcited by 171
  28. 2022Revisiting the "Video" in Video-Language Understandingcited by 167
  29. 2024Streaming Long Video Understanding with Large Language Modelscited by 164
  30. 2024Flash-VStream: Memory-Based Real-Time Understanding for Long Video Streamscited by 160
  31. 2024Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolutioncited by 157
  32. 2023Emu: Generative Pretraining in Multimodalitycited by 156
  33. 2024Video-of-Thought: Step-by-Step Video Reasoning from Perception to Cognitioncited by 151
  34. 2024Direct Preference Optimization of Video Large Multimodal Models from Language Model Rewardcited by 145
  35. 2024Tarsier: Recipes for Training and Evaluating Large Video Description Modelscited by 140
  36. 2025Magma: A Foundation Model for Multimodal AI Agentscited by 138
  37. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 137
  38. 2025OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video Understanding?cited by 134

50 most cited of 735 All 735 (JSON)