mlsanityData to 14 Oct 2026

VidComposition: Can MLLMs Analyze Compositions in Compiled Videos?

arXiv 2411.10979 ↗First posted 17 Nov 2024cs.CV cs.AIcited by 18

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

The advancement of Multimodal Large Language Models (MLLMs) has enabled significant progress in multimodal understanding, expanding their capacity to analyze video content. However, existing evaluation benchmarks for MLLMs primarily focus on abstract video comprehension, lacking a detailed assessment of their ability to understand video compositions, the nuanced interpretation of how visual elements combine and interact within highly compiled video contexts. We introduce VidComposition, a new benchmark specifically designed to evaluate the video composition understanding capabilities of MLLMs using carefully curated compiled videos and cinematic-level annotations. VidComposition includes 982 videos with 1706 multiple-choice questions, covering various compositional aspects such as camera movement, angle, shot size, narrative structure, character actions and emotions, etc. Our comprehensive evaluation of 33 open-source and proprietary MLLMs reveals a significant performance gap between human and model capabilities. This highlights the limitations of current MLLMs in understanding complex, compiled video compositions and offers insights into areas for further improvement. The leaderboard and evaluation code are available at https://yunlong10.github.io/VidComposition/

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 58 of its 68 references that are in the map. First posted on arXiv 17 Nov 2024.

Builds on

58 of 68 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via a Hybrid Architecture4 Sep
  2. 2024Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Input28 Aug
  3. 2024Long Context Transfer from Language to Vision24 Jun
  4. 2024LVBench: An Extreme Long Video Understanding Benchmark12 Jun
  5. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis31 May
  6. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmark28 Nov
  7. 2023AutoEval-Video: An Automatic Benchmark for Assessing Large Vision Language Models in Open-Ended Video Question Answering25 Nov
  8. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projection16 Nov

Other references · 50

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  5. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  6. 2015VQA: Visual Question Answeringcited by 2,506
  7. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  8. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  9. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  10. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  11. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  12. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
38 more
  1. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  2. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  3. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  4. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
  5. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  6. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  7. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 771
  8. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 744
  9. 2023VILA: On Pre-training for Visual Language Modelscited by 692
  10. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 610
  11. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 583
  12. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 561
  13. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 542
  14. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 529
  15. 2018TVQA: Localized, Compositional Video Question Answeringcited by 467
  16. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 414
  17. 2024InternLM2 Technical Reportcited by 397
  18. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  19. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 371
  20. 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 363
  21. 2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionalitycited by 243
  22. 2023Video Understanding with Large Language Models: A Surveycited by 206
  23. 2024Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolutioncited by 159
  24. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 140
  25. 2024DOCCI: Descriptions of Connected and Contrasting Imagescited by 132
  26. 2023Video-Bench: A Comprehensive Benchmark and Toolkit for Evaluating Video-based Large Language Modelscited by 107
  27. 2023Caption Anything: Interactive Image Description with Diverse Multimodal Controlscited by 99
  28. 2021VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluationcited by 95
  29. 2023AV-NeRF: Learning Neural Fields for Real-World Audio-Visual Scene Synthesiscited by 49
  30. 2024V2Xum-LLM: Cross-Modal Video Summarization with Temporal Prompt Instruction Tuningcited by 46
  31. 2024PromptFix: You Prompt and We Fix the Photocited by 40
  32. 2024MMCOMPOSITION: Revisiting the Compositionality of Pre-trained Vision-Language Modelscited by 31
  33. 2021UniCon: Unified Context Network for Robust Active Speaker Detectioncited by 28
  34. 2024An Empirical Analysis on Large Language Models in Debate Evaluationcited by 20
  35. 2024Language-Guided Joint Audio-Visual Editing via One-Shot Adaptationcited by 19
  36. 2024EAGLE: Egocentric AGgregated Language-video Enginecited by 16
  37. 2024Do More Details Always Introduce More Hallucinations in LVLM-based Image Captioning?cited by 6
  38. 2024Revisiting Multi-Modal LLM Evaluationcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 10; 2026: 6.

18 papers.

Sort:

By area:

  1. 2025Towards Understanding Camera Motions in Any Videocited by 64
  2. 2024FINECAPTION: Compositional Image Captioning Focusing on Wherever You Want at Any Granularitycited by 27
  3. 2025VERIFY: A Benchmark of Visual Explanation and Reasoning for Investigating Multimodal Reasoning Fidelitycited by 23
  4. 2024KinMo: Kinematic-aware Human Motion Understanding and Generationcited by 21
  5. 2025Generative AI for Cel-Animation: A Surveycited by 17
  6. 2025MMIG-Bench: Towards Comprehensive and Explainable Evaluation of Multi-Modal Image Generation Modelscited by 17
  7. 2025Shot-by-Shot: Film-Grammar-Aware Training-Free Audio Description Generationcited by 13
  8. 2025Caption Anything in Video: Fine-grained Object-centric Captioning via Spatiotemporal Multimodal Promptingcited by 10
  9. 2026ChartNet: A Million-Scale, High-Quality Multimodal Dataset for Robust Chart Understandingcited by 8
  10. 2025Video-R4: Reinforcing Text-Rich Video Reasoning with Visual Ruminationcited by 6
  11. 2026BVB: Benchmarking Agentic Video Understanding via Programmatic Reconstruction in Blendercited by 6
  12. 2025MMPerspective: Do MLLMs Understand Perspective? A Comprehensive Benchmark for Perspective Perception, Reasoning, and Robustnesscited by 5
6 more
  1. 2025What Happens When: Learning Temporal Orders of Events in Videoscited by 3
  2. 2026Geometry-Guided Camera Motion Understanding in VideoLLMscited by 3
  3. 2025DAVE: A VLM Vision Encoder for Document Understanding and Web Agentscited by 2
  4. 2026Natural Language Camera Movement Understandingcited by 1
  5. 2026Temporally Grounded Compositional Camera Motion Understanding via Geometric Knowledge Distillationcited by 1
  6. 2026TAKE 85: Testing Audiovisual filmmaKer's intEnt across 85 Hours of Filmcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Evaluation of MLLM-Agnostic Plug-and-Play Keyframe Selection Methods for Long Video Understanding
  2. 2026Benchmarking Visual State Tracking in Multimodal Video Understanding
  3. 2026HAVEN: Hierarchically Aligned Multimodal Benchmark for Unified Video Understanding
  4. 2026Multimodal Large Language Model-Enabled Video Translation: A Role-Oriented Survey
  5. 2025RO-Bench: Large-scale robustness evaluation of MLLMs with text-driven counterfactual videos
  6. 2024Exploring the Design Space of Visual Context Representation in Video MLLMs
  7. 2024MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding
  8. 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modeling