mlsanityData to 14 Oct 2026

EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding

arXiv 2308.09126 ↗First posted 17 Aug 2023cs.CV cs.AI cs.CLcited by 680

Not on the map. Why?

Cited by 274 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce EgoSchema, a very long-form video question-answering dataset, and benchmark to evaluate long video understanding capabilities of modern vision and language systems. Derived from Ego4D, EgoSchema consists of over 5000 human curated multiple choice question answer pairs, spanning over 250 hours of real video data, covering a very broad range of natural human activity and behavior. For each question, EgoSchema requires the correct answer to be selected between five given options based on a three-minute-long video clip. While some prior works have proposed video datasets with long clip lengths, we posit that merely the length of the video clip does not truly capture the temporal difficulty of the video task that is being considered. To remedy this, we introduce temporal certificate sets, a general notion for capturing the intrinsic temporal understanding length associated with a broad range of video understanding tasks & datasets. Based on this metric, we find EgoSchema to have intrinsic temporal lengths over 5.7x longer than the second closest dataset and 10x to 100x longer than any other video understanding dataset. Further, our evaluation of several current state-of-the-art video and language models shows them to be severely lacking in long-term video understanding capabilities. Even models with several billions of parameters achieve QA accuracy less than 33% (random is 20%) on the EgoSchema multi-choice question answering task, while humans achieve about 76% accuracy. We posit that name, with its long intrinsic temporal structures and diverse complexity, would serve as a valuable evaluation probe for developing effective long-term video understanding systems in the future. Data and Zero-shot model evaluation code are open-sourced for both public and commercial use under the Ego4D license at http://egoschema.github.io

Key facts. Cited by 680 papers in the map (data through 14 Oct 2026). Builds on 48 of its 102 references that are in the map. First posted on arXiv 17 Aug 2023.

Builds on

48 of 102 references.

References · 48

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2017Mask R-CNNcited by 7,277
  4. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  5. 2015VQA: Visual Question Answeringcited by 2,506
  6. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  7. 2018Datasheets for Datasetscited by 1,386
  8. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,103
  9. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  10. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
  11. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 744
  12. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 561
36 more
  1. 2019CLEVRER: CoLlision Events for Video REpresentation and Reasoningcited by 484
  2. 2018TVQA: Localized, Compositional Video Question Answeringcited by 467
  3. 2017AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actionscited by 449
  4. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 447
  5. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  6. 2016YouTube-8M: A Large-Scale Video Classification Benchmarkcited by 324
  7. 2015MovieQA: Understanding Stories in Movies through Question-Answeringcited by 304
  8. 2020Just Ask: Learning to Answer Questions from Millions of Narrated Videoscited by 248
  9. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  10. 2020MovieNet: A Holistic Dataset for Movie Understandingcited by 206
  11. 2022Learning to Answer Questions in Dynamic Audio-Visual Scenarioscited by 191
  12. 2022Data Cards: Purposeful and Transparent Dataset Documentation for Responsible AIcited by 171
  13. 2019TVQA+: Spatio-Temporal Grounding for Video Question Answeringcited by 154
  14. 2021Towards Long-Form Video Understandingcited by 146
  15. 2021AGQA: A Benchmark for Compositional Spatio-Temporal Reasoningcited by 120
  16. 2021MAD: A Scalable Dataset for Language Grounding in Videos from Movie Audio Descriptionscited by 108
  17. 2021Pano-AVQA: Grounded Audio-Visual Question Answering on 360^circ Videoscited by 86
  18. 2021SUTD-TrafficQA: A Question Answering Benchmark and an Efficient Network for Video Reasoning over Traffic Eventscited by 84
  19. 2022Temporal Alignment Networks for Long-term Videocited by 84
  20. 2022An Empirical Study of End-to-End Video-Language Transformers with Masked Visual Modelingcited by 74
  21. 2017MovieGraphs: Towards Understanding Human-Centric Situations from Videoscited by 67
  22. 2021Generic Event Boundary Detection: A Benchmark for Event Segmentationcited by 65
  23. 2018Online Model Distillation for Efficient Video Inferencecited by 60
  24. 2019KnowIT VQA: Answering Knowledge-Based Questions about Videoscited by 53
  25. 2020DramaQA: Character-Centered Video Story Understanding with Hierarchical QAcited by 45
  26. 2016MarioQA: Answering Questions by Watching Gameplay Videoscited by 42
  27. 2017Video Question Answering via Attribute-Augmented Attention Network Learningcited by 39
  28. 2020Video2Commonsense: Generating Commonsense Descriptions to Enrich Video Captioningcited by 36
  29. 2019A Graph-Based Framework to Bridge Movies and Synopsescited by 31
  30. 2017iVQA: Inverse Visual Question Answeringcited by 22
  31. 2021Video Question Answering with Phrases via Semantic Rolescited by 13
  32. 2021FIBER: Fill-in-the-Blanks as a Challenging Video Understanding Evaluation Frameworkcited by 12
  33. 2022AGQA 2.0: An Updated Benchmark for Compositional Spatio-Temporal Reasoningcited by 10
  34. 2021Transferring Domain-Agnostic Knowledge in Video Question Answeringcited by 7
  35. 2020Video Question Answering on Screencast Tutorialscited by 6
  36. 2022NEWSKVQA: Knowledge-Aware News Video Question Answeringcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 12; 2024: 128; 2025: 275; 2026: 265.

680 papers.

Sort:

By area:

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  2. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  3. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  4. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,109
  5. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 771
  6. 2025Qwen2.5-Omni Technical Reportcited by 736
  7. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understandingcited by 665
  8. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 611
  9. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 583
  10. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 518
  11. 2024LVBench: An Extreme Long Video Understanding Benchmarkcited by 410
  12. 2025Kimi-VL Technical Reportcited by 388
38 more
  1. 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understandingcited by 356
  2. 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agentcited by 332
  3. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 326
  4. 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 317
  5. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 309
  6. 2024MLVU: Benchmarking Multi-task Long Video Understandingcited by 292
  7. 2024VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videoscited by 251
  8. 2024VideoAgent: A Memory-augmented Multimodal Agent for Video Understandingcited by 229
  9. 2024VideoChat-Flash: Hierarchical Compression for Long-Context Video Modelingcited by 220
  10. 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modelingcited by 211
  11. 2023Video Understanding with Large Language Models: A Surveycited by 206
  12. 2025DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Modelscited by 181
  13. 2023A Simple LLM Framework for Long-Range Video Question-Answeringcited by 178
  14. 2024Flash-VStream: Memory-Based Real-Time Understanding for Long Video Streamscited by 169
  15. 2024Streaming Long Video Understanding with Large Language Modelscited by 167
  16. 2025LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Tokencited by 166
  17. 2024LongVLM: Efficient Long Video Understanding via Large Language Modelscited by 147
  18. 2024Vision language models are blind: Failing to translate detailed visual features into wordscited by 147
  19. 2025Time-R1: Post-Training Large Vision Language Model for Temporal Video Groundingcited by 146
  20. 2025OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video Understanding?cited by 142
  21. 2024Tarsier: Recipes for Training and Evaluating Large Video Description Modelscited by 141
  22. 2024Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Inputcited by 134
  23. 2024MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scalecited by 127
  24. 2025Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reactioncited by 127
  25. 2024SlowFast-LLaVA: A Strong Training-Free Baseline for Video Large Language Modelscited by 125
  26. 2025Streaming Video Question-Answering with In-context Video KV-Cache Retrievalcited by 117
  27. 2025VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documentscited by 116
  28. 2025MiMo-VL Technical Reportcited by 115
  29. 2024CinePile: A Long Video Question Answering Dataset and Benchmarkcited by 114
  30. 2024HourVideo: 1-Hour Video-Language Understandingcited by 113
  31. 2025Baichuan-Omni-1.5 Technical Reportcited by 111
  32. 2025Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalitiescited by 110
  33. 2024Video Mamba Suite: State Space Model as a Versatile Alternative for Video Understandingcited by 108
  34. 2025VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Modelscited by 107
  35. 2024TimeSuite: Improving MLLMs for Long Video Understanding via Grounded Tuningcited by 106
  36. 2025HD-EPIC: A Highly-Detailed Egocentric Video Datasetcited by 104
  37. 2024Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenizationcited by 100
  38. 2025Deep Video Discovery: Agentic Search with Tool Use for Long-form Video Understandingcited by 98

50 most cited of 680 All 680 (JSON)