mlsanityData to 7 Oct 2026

Video-Bench: A Comprehensive Benchmark and Toolkit for Evaluating Video-based Large Language Models

arXiv 2311.16103 ↗First posted 27 Nov 2023cs.CV cs.AIcited by 107

Not on the map. Why?

Cited by 46 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Video-based large language models (Video-LLMs) have been recently introduced, targeting both fundamental improvements in perception and comprehension, and a diverse range of user inquiries. In pursuit of the ultimate goal of achieving artificial general intelligence, a truly intelligent Video-LLM model should not only see and understand the surroundings, but also possess human-level commonsense, and make well-informed decisions for the users. To guide the development of such a model, the establishment of a robust and comprehensive evaluation system becomes crucial. To this end, this paper proposes textitVideo-Bench, a new comprehensive benchmark along with a toolkit specifically designed for evaluating Video-LLMs. The benchmark comprises 10 meticulously crafted tasks, evaluating the capabilities of Video-LLMs across three distinct levels: Video-exclusive Understanding, Prior Knowledge-based Question-Answering, and Comprehension and Decision-making. In addition, we introduce an automatic toolkit tailored to process model outputs for various tasks, facilitating the calculation of metrics and generating convenient final scores. We evaluate 8 representative Video-LLMs using textitVideo-Bench. The findings reveal that current Video-LLMs still fall considerably short of achieving human-like comprehension and analysis of real-world videos, offering valuable insights for future research directions. The benchmark and toolkit are available at: urlhttps://github.com/PKU-YuanGroup/Video-Bench.

Key facts. Cited by 107 papers in the map (data through 7 Oct 2026). Builds on 45 of its 60 references that are in the map. First posted on arXiv 27 Nov 2023.

Builds on

45 of 60 references.

References · 45

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  11. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  12. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
33 more
  1. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  2. 2017The Kinetics Human Action Video Datasetcited by 1,894
  3. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  4. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  5. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  6. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  7. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  8. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  9. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  10. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  11. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  12. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
  13. 2018Real-world Anomaly Detection in Surveillance Videoscited by 570
  14. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  15. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  16. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  17. 2018TVQA: Localized, Compositional Video Question Answeringcited by 463
  18. 2017AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actionscited by 449
  19. 2019Video Instance Segmentationcited by 391
  20. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  21. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 368
  22. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 330
  23. 2015MovieQA: Understanding Stories in Movies through Question-Answeringcited by 304
  24. 2022SQA3D: Situated Question Answering in 3D Scenescited by 303
  25. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 231
  26. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Modelscited by 217
  27. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integrationcited by 168
  28. 2022OmniVL:One Foundation Model for Image-Language and Video-Language Taskscited by 143
  29. 2022ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Modelscited by 141
  30. 2015MOTChallenge 2015: Towards a Benchmark for Multi-Target Trackingcited by 133
  31. 2023X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languagescited by 127
  32. 2023ChatVideo: A Tracklet-centric Multimodal and Versatile Video Understanding Systemcited by 65
  33. 2021FineAction: A Fine-Grained Video Dataset for Temporal Action Localizationcited by 60

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 35; 2025: 48; 2026: 23.

107 papers.

Sort:

By area:

  1. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  2. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 580
  3. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
  4. 2024We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?cited by 281
  5. 2024MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmarkcited by 281
  6. 2024MLVU: Benchmarking Multi-task Long Video Understandingcited by 275
  7. 2023Video Understanding with Large Language Models: A Surveycited by 203
  8. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  9. 2025WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMscited by 147
  10. 2025MMVU: Measuring Expert-Level Multi-Discipline Video Understandingcited by 131
  11. 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Modelscited by 130
  12. 2024CinePile: A Long Video Question Answering Dataset and Benchmarkcited by 112
38 more
  1. 2024HourVideo: 1-Hour Video-Language Understandingcited by 110
  2. 2024MotionLLM: Understanding Human Behaviors from Human Motions and Videoscited by 89
  3. 2024CG-Bench: Clue-grounded Question Answering Benchmark for Long Video Understandingcited by 82
  4. 2024A Survey on Benchmarks of Multimodal Large Language Modelscited by 81
  5. 2024VideoVista: A Versatile Benchmark for Video Understanding and Reasoningcited by 72
  6. 2024TOMATO: Assessing Visual Temporal Reasoning Capabilities in Multimodal Foundation Modelscited by 71
  7. 2024E.T. Bench: Towards Open-Ended Event-Level Video-Language Understandingcited by 68
  8. 2024InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactionscited by 48
  9. 2025OmniMMI: A Comprehensive Multi-modal Interaction Benchmark in Streaming Video Contextscited by 48
  10. 2024MMWorld: Towards Multi-discipline Multi-faceted World Model Evaluation in Videoscited by 43
  11. 2025VLM4D: Towards Spatiotemporal Awareness in Vision Language Modelscited by 43
  12. 2024Align Anything: Training All-Modality Models to Follow Instructions with Language Feedbackcited by 38
  13. 2024Temporal Reasoning Transfer from Text to Videocited by 36
  14. 2024Needle In A Video Haystack: A Scalable Synthetic Evaluator for Video MLLMscited by 35
  15. 2024How Good is my Video LMM? Complex Video Reasoning and Robustness Evaluation Suite for Video-LMMscited by 30
  16. 2024VidEgoThink: Assessing Egocentric Video Understanding Capabilities for Embodied AIcited by 30
  17. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  18. 2024Neptune: The Long Orbit to Benchmarking Long Video Understandingcited by 24
  19. 2025VRBench: A Benchmark for Multi-Step Reasoning in Long Narrative Videoscited by 24
  20. 2024ProteinGPT: Multimodal LLM for Protein Property Prediction and Structure Understandingcited by 23
  21. 2024PhysGame: Uncovering Physical Commonsense Violations in Gameplay Videoscited by 21
  22. 2026DrivingGen: A Comprehensive Benchmark for Generative Video World Models in Autonomous Drivingcited by 20
  23. 2024VidHal: Benchmarking Temporal Hallucinations in Vision LLMscited by 19
  24. 2025AVTrustBench: Assessing and Enhancing Reliability and Robustness in Audio-Visual LLMscited by 19
  25. 2025video-SALMONN-o1: Reasoning-enhanced Audio-visual Large Language Modelcited by 19
  26. 2025AgroBench: Vision-Language Model Benchmark in Agriculturecited by 19
  27. 2025See, Hear, and Understand: Benchmarking Audiovisual Human Speech Understanding in Multimodal Large Language Modelscited by 19
  28. 2024VidComposition: Can MLLMs Analyze Compositions in Compiled Videos?cited by 18
  29. 2024On the Consistency of Video Large Language Models in Temporal Comprehensioncited by 18
  30. 2025SIV-Bench: A Video Benchmark for Social Interaction Understanding and Reasoningcited by 18
  31. 2025Impossible Videoscited by 16
  32. 2025SAVVY: Spatial Awareness via Audio-Visual LLMs through Seeing and Hearingcited by 14
  33. 2025MCIF: Multimodal Crosslingual Instruction-Following Benchmark from Scientific Talkscited by 14
  34. 2025CVBench: Benchmarking Cross-Video Synergies for Complex Multimodal Reasoningcited by 14
  35. 2024OmnixR: Evaluating Omni-modality Language Models on Reasoning across Modalitiescited by 13
  36. 2025AdsQA: Towards Advertisement Video Understandingcited by 12
  37. 2025VideoComp: Advancing Fine-Grained Compositional and Temporal Alignment in Video-Text Modelscited by 11
  38. 2025MAGNET: A Multi-agent Framework for Finding Audio-Visual Needles by Reasoning over Multi-Video Haystackscited by 11

50 most cited of 107 All 107 (JSON)