mlsanityData to 14 Oct 2026

HourVideo: 1-Hour Video-Language Understanding

arXiv 2411.04998 ↗First posted 7 Nov 2024cs.CV cs.AI cs.LGcited by 113

Not on the map. Why?

Cited by 50 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present HourVideo, a benchmark dataset for hour-long video-language understanding. Our dataset consists of a novel task suite comprising summarization, perception (recall, tracking), visual reasoning (spatial, temporal, predictive, causal, counterfactual), and navigation (room-to-room, object retrieval) tasks. HourVideo includes 500 manually curated egocentric videos from the Ego4D dataset, spanning durations of 20 to 120 minutes, and features 12,976 high-quality, five-way multiple-choice questions. Benchmarking results reveal that multimodal models, including GPT-4 and LLaVA-NeXT, achieve marginal improvements over random chance. In stark contrast, human experts significantly outperform the state-of-the-art long-context multimodal model, Gemini Pro 1.5 (85.0% vs. 37.3%), highlighting a substantial gap in multimodal capabilities. Our benchmark, evaluation toolkit, prompts, and documentation are available at https://hourvideo.stanford.edu

Key facts. Cited by 113 papers in the map (data through 14 Oct 2026). Builds on 91 of its 111 references that are in the map. First posted on arXiv 7 Nov 2024.

Builds on

91 of 111 references.

References · 91

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  6. 2023Visual Instruction Tuningcited by 7,869
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  10. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  11. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  12. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
38 more
  1. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  2. 2015VQA: Visual Question Answeringcited by 2,506
  3. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  4. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  5. 2017The Kinetics Human Action Video Datasetcited by 1,909
  6. 2019Towards VQA Models That Can Readcited by 1,718
  7. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  8. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  9. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  10. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  11. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  12. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  13. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  14. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,109
  15. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,103
  16. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  17. 2023A Survey on Multimodal Large Language Modelscited by 909
  18. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  19. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 744
  20. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 680
  21. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 619
  22. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 586
  23. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 565
  24. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 561
  25. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 542
  26. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  27. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 497
  28. 2018TVQA: Localized, Compositional Video Question Answeringcited by 467
  29. 2022XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Modelcited by 439
  30. 2019Video Object Segmentation using Space-Time Memory Networkscited by 438
  31. 2019A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmarkcited by 425
  32. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understandingcited by 406
  33. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 393
  34. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  35. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 371
  36. 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 363
  37. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  38. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 322

50 most cited of 91 All 91 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 68; 2026: 42.

113 papers.

Sort:

By area:

  1. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 611
  2. 2024VideoChat-Flash: Hierarchical Compression for Long-Context Video Modelingcited by 220
  3. 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modelingcited by 211
  4. 2025WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMscited by 155
  5. 2025OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video Understanding?cited by 142
  6. 2025OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Modelscited by 114
  7. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Modelscited by 109
  8. 2025Seeing, Listening, Remembering, and Reasoning: A Multimodal Agent with Long-Term Memorycited by 97
  9. 2025SpatialLadder: Progressive Training for Spatial Reasoning in Vision-Language Modelscited by 90
  10. 2025VideoRAG: Retrieval-Augmented Generation with Extreme Long-Context Videoscited by 77
  11. 2024VidHalluc: Evaluating Temporal Hallucinations in Multimodal Large Language Models for Video Understandingcited by 68
  12. 2025V-STaR: Benchmarking Video-LLMs on Video Spatio-Temporal Reasoningcited by 68
38 more
  1. 2025Ego-R1: Chain-of-Tool-Thought for Ultra-Long Egocentric Video Reasoningcited by 68
  2. 2025Eagle 2.5: Boosting Long-Context Post-Training for Frontier Vision-Language Modelscited by 64
  3. 2025WorldMM: Dynamic Multimodal Memory Agent for Long Video Reasoningcited by 53
  4. 2025EgoTextVQA: Towards Egocentric Scene-Text Aware Video Question Answeringcited by 44
  5. 2025Generative Frame Sampler for Long Video Understandingcited by 37
  6. 2025UrbanVideo-Bench: Benchmarking Vision-Language Models on Embodied Intelligence with Video Data in Urban Spacescited by 36
  7. 2025Vamba: Understanding Hour-Long Videos with Hybrid Mamba-Transformerscited by 36
  8. 2025CausalVQA: A Physically Grounded Causal Reasoning Benchmark for Video Modelscited by 35
  9. 2025CityEQA: A Hierarchical LLM Agent on Embodied Question Answering Benchmark in City Spacecited by 32
  10. 2025MINERVA: Evaluating Complex Video Reasoningcited by 32
  11. 2025Active Video Perception: Iterative Evidence Seeking for Agentic Long Video Understandingcited by 29
  12. 2025Video-SafetyBench: A Benchmark for Safety Evaluation of Video LVLMscited by 28
  13. 2025Spatial Understanding from Videos: Structured Prompts Meet Simulation Datacited by 28
  14. 2025Exploring the Effect of Reinforcement Learning on Video Understanding: Insights from SEED-Bench-R1cited by 25
  15. 2025Unleashing Hour-Scale Video Training for Long Video-Language Understandingcited by 23
  16. 2025EgoExoBench: A Benchmark for First- and Third-person View Video Understanding in MLLMscited by 23
  17. 2025ST-Think: How Multimodal Large Language Models Reason About 4D Worlds from Ego-Centric Videoscited by 21
  18. 2025VideoEval-Pro: Robust and Realistic Long Video Understanding Evaluationcited by 21
  19. 2025Does Your Vision-Language Model Get Lost in the Long Video Sampling Dilemma?cited by 20
  20. 2025EgoNight: Towards Egocentric Vision Understanding at Night with a Challenging Benchmarkcited by 19
  21. 2025ViTCoT: Video-Text Interleaved Chain-of-Thought for Boosting Video Understanding in Large Language Modelscited by 18
  22. 2025HSSBench: Benchmarking Humanities and Social Sciences Ability for Multimodal Large Language Modelscited by 17
  23. 2026VideoSeek: Long-Horizon Video Agent with Tool-Guided Seekingcited by 17
  24. 2025Seeing from Another Perspective: Evaluating Multi-View Understanding in MLLMscited by 16
  25. 2025VideoMultiAgents: A Multi-Agent Framework for Video Question Answeringcited by 16
  26. 2025X-LeBench: A Benchmark for Extremely Long Egocentric Video Understandingcited by 15
  27. 2025Less Is More, but Where? Dynamic Token Compression via LLM-Guided Keyframe Priorcited by 15
  28. 2025RSCC: A Large-Scale Remote Sensing Change Caption Dataset for Disaster Eventscited by 14
  29. 2025EgoNormia: Benchmarking Physical Social Norm Understandingcited by 13
  30. 2025SurgVidLM: Towards Multi-grained Surgical Video Understanding with Large Language Modelcited by 13
  31. 2025Towards Online Multi-Modal Social Interaction Understandingcited by 12
  32. 2025Exploring Diffusion Transformer Designs via Graftingcited by 12
  33. 2025V2P-Bench: Evaluating Video-Language Understanding with Visual Prompts for Better Human-Model Interactioncited by 11
  34. 2025Video SimpleQA: Towards Factuality Evaluation in Large Video Language Modelscited by 11
  35. 2025VidText: Towards Comprehensive Evaluation for Video Text Understandingcited by 11
  36. 2025Fine-grained Spatiotemporal Grounding on Egocentric Videoscited by 11
  37. 2025Threading Keyframe with Narratives: MLLMs as Strong Long Video Comprehenderscited by 10
  38. 2025Video-EM: Event-Centric Episodic Memory for Long-Form Video Understandingcited by 10

50 most cited of 113 All 113 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding
  2. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding
  3. 2024TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Models
  4. 2024From Seconds to Hours: Reviewing MultiModal Large Language Models on Comprehensive Long Video Understanding
  5. 2025OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs
  6. 2024Lost in Time: A New Temporal Benchmark for VideoLLMs
  7. 2026Meta-Video: Efficient Standardized Evaluation for Video Understanding
  8. 2025HumanVideo-MME: Benchmarking MLLMs for Human-Centric Video Understanding