mlsanityData to 7 Oct 2026

LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding

arXiv 2407.15754 ↗First posted 22 Jul 2024cs.CV cs.CL cs.LGcited by 644

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

Large multimodal models (LMMs) are processing increasingly longer and richer inputs. Albeit the progress, few public benchmark is available to measure such development. To mitigate this gap, we introduce LongVideoBench, a question-answering benchmark that features video-language interleaved inputs up to an hour long. Our benchmark includes 3,763 varying-length web-collected videos with their subtitles across diverse themes, designed to comprehensively evaluate LMMs on long-term multimodal understanding. To achieve this, we interpret the primary challenge as to accurately retrieve and reason over detailed multimodal information from long inputs. As such, we formulate a novel video question-answering task termed referring reasoning. Specifically, as part of the question, it contains a referring query that references related video contexts, called referred context. The model is then required to reason over relevant video details from the referred context. Following the paradigm of referring reasoning, we curate 6,678 human-annotated multiple-choice questions in 17 fine-grained categories, establishing one of the most comprehensive benchmarks for long-form video understanding. Evaluations suggest that the LongVideoBench presents significant challenges even for the most advanced proprietary models (e.g. GPT-4o, Gemini-1.5-Pro, GPT-4-Turbo), while their open-source counterparts show an even larger performance gap. In addition, our results indicate that model performance on the benchmark improves only when they are capable of processing more frames, positioning LongVideoBench as a valuable benchmark for evaluating future-generation long-context LMMs.

Key facts. Cited by 644 papers in the map (data through 7 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 20 of its 30 references that are in the map. First posted on arXiv 22 Jul 2024.

Builds on

20 of 30 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understanding8 Apr
  2. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models28 Nov
  3. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding5 Jun

Other references · 17

  1. 2023Visual Instruction Tuningcited by 7,792
  2. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  5. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  6. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  7. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  8. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  9. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  10. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  11. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  12. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
5 more
  1. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  2. 2023Q-Align: Teaching LMMs for Visual Scoring via Discrete Text-Defined Levelscited by 489
  3. 2024World Model on Million-Length Video And Language With Blockwise RingAttentioncited by 196
  4. 2024Ada-LEval: Evaluating long-context LLMs with length-adaptable benchmarkscited by 32
  5. 2023LvBench: A Benchmark for Long-form Video Understanding with Versatile Multi-modal Question Answeringcited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 46; 2025: 252; 2026: 346.

644 papers.

Sort:

By area:

  1. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  2. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  3. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  4. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 550
  5. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  6. 2026Kimi K2.5: Visual Agentic Intelligencecited by 449
  7. 2025Kimi-VL Technical Reportcited by 381
  8. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 303
  9. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 285
  10. 2025VideoChat-R1: Enhancing Spatio-Temporal Perception via Reinforcement Fine-Tuningcited by 248
  11. 2024VideoChat-Flash: Hierarchical Compression for Long-Context Video Modelingcited by 212
  12. 2024Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understandingcited by 210
38 more
  1. 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modelingcited by 203
  2. 2025Unified Reward Model for Multimodal Understanding and Generationcited by 194
  3. 2025MiniCPM-V 4.5: Cooking Efficient MLLMs via Architecture, Data, and Training Recipecited by 174
  4. 2024Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolutioncited by 157
  5. 2025Adaptive Keyframe Sampling for Long Video Understandingcited by 153
  6. 2025SpaceR: Reinforcing MLLMs in Video Spatial Reasoningcited by 152
  7. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 137
  8. 2024Aria: An Open Multimodal Native Mixture-of-Experts Modelcited by 136
  9. 2024Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehensioncited by 134
  10. 2025OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video Understanding?cited by 134
  11. 2024Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Inputcited by 132
  12. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  13. 2025Video-Holmes: Can MLLM Think Like Holmes for Complex Video Reasoning?cited by 107
  14. 2025Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalitiescited by 101
  15. 2024LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via a Hybrid Architecturecited by 100
  16. 2025Deep Video Discovery: Agentic Search with Tool Use for Long-form Video Understandingcited by 94
  17. 2025Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMscited by 92
  18. 2025TimeChat-Online: 80% Visual Tokens are Naturally Redundant in Streaming Videoscited by 90
  19. 2025Seeing, Listening, Remembering, and Reasoning: A Multimodal Agent with Long-Term Memorycited by 88
  20. 2024Apollo: An Exploration of Video Understanding in Large Multimodal Modelscited by 87
  21. 2025M-LLM Based Video Frame Selection for Efficient Video Understandingcited by 87
  22. 2025EgoLife: Towards Egocentric Life Assistantcited by 86
  23. 2025Scaling RL to Long Videoscited by 83
  24. 2024CG-Bench: Clue-grounded Question Answering Benchmark for Long Video Understandingcited by 82
  25. 2025Tarsier2: Advancing Large Vision-Language Models from Detailed Video Description to Comprehensive Video Understandingcited by 80
  26. 2025Ola: Pushing the Frontiers of Omni-Modal Language Modelcited by 79
  27. 2025VideoRAG: Retrieval-Augmented Generation with Extreme Long-Context Videoscited by 76
  28. 2025STI-Bench: Are MLLMs Ready for Precise Spatial-Temporal World Understanding?cited by 74
  29. 2025BOLT: Boost Large Vision-Language Model Without Training for Long-form Video Understandingcited by 72
  30. 2025Q-Frame: Query-aware Frame Selection and Multi-Resolution Adaptation for Video-LLMscited by 72
  31. 2024TimeMarker: A Versatile Video-LLM for Long and Short Video Understanding with Superior Temporal Localization Abilitycited by 69
  32. 2025MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Modelscited by 69
  33. 2025Kwai Keye-VL 1.5 Technical Reportcited by 68
  34. 2025V-STaR: Benchmarking Video-LLMs on Video Spatio-Temporal Reasoningcited by 66
  35. 2024VidHalluc: Evaluating Temporal Hallucinations in Multimodal Large Language Models for Video Understandingcited by 65
  36. 2026MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interactioncited by 64
  37. 2025VideoRoPE: What Makes for Good Video Rotary Position Embedding?cited by 63
  38. 2025Eagle 2.5: Boosting Long-Context Post-Training for Frontier Vision-Language Modelscited by 62

50 most cited of 644 All 644 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025TextVidBench: A Benchmark for Long Video Scene Text Understanding
  2. 2024Lost in Time: A New Temporal Benchmark for VideoLLMs
  3. 2025An Empirical Study for Representations of Videos in Video Question Answering via MLLMs
  4. 2025SVBench: A Benchmark with Temporal Multi-Turn Dialogues for Streaming Video Understanding
  5. 2023A Simple LLM Framework for Long-Range Video Question-Answering
  6. 2024TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Models
  7. 2025POVQA: Preference-Optimized Video Question Answering with Rationales for Data Efficiency
  8. 2026Beyond Frame Selection: Rethinking Long-Video Understanding with MLLMs