mlsanityData to 14 Oct 2026

MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models

arXiv 2501.02955 ↗First posted 6 Jan 2025cs.CVcited by 70

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

In recent years, vision language models (VLMs) have made significant advancements in video understanding. However, a crucial capability - fine-grained motion comprehension - remains under-explored in current benchmarks. To address this gap, we propose MotionBench, a comprehensive evaluation benchmark designed to assess the fine-grained motion comprehension of video understanding models. MotionBench evaluates models' motion-level perception through six primary categories of motion-oriented question types and includes data collected from diverse sources, ensuring a broad representation of real-world video content. Experimental results reveal that existing VLMs perform poorly in understanding fine-grained motions. To enhance VLM's ability to perceive fine-grained motion within a limited sequence length of LLM, we conduct extensive experiments reviewing VLM architectures optimized for video feature compression and propose a novel and efficient Through-Encoder (TE) Fusion method. Experiments show that higher frame rate inputs and TE Fusion yield improvements in motion understanding, yet there is still substantial room for enhancement. Our benchmark aims to guide and motivate the development of more capable video understanding models, emphasizing the importance of fine-grained motion comprehension. Project page: https://motion-bench.github.io .

Key facts. Cited by 70 papers in the map (data through 14 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 43 of its 58 references that are in the map. First posted on arXiv 6 Jan 2025.

Builds on

43 of 58 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Input28 Aug
  2. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding22 Jul
  3. 2024Towards Event-oriented Long Video Understanding20 Jun
  4. 2024VideoVista: A Versatile Benchmark for Video Understanding and Reasoning17 Jun
  5. 2024LVBench: An Extreme Long Video Understanding Benchmark12 Jun
  6. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis31 May
  7. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioning25 Apr
  8. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmark28 Nov
  9. 2023AutoEval-Video: An Automatic Benchmark for Assessing Large Vision Language Models in Open-Ended Video Question Answering25 Nov
  10. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding5 Jun

Other references · 33

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023Visual Instruction Tuningcited by 7,869
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  4. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  6. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  7. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  8. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  9. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  10. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,103
  11. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
  12. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
21 more
  1. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  2. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 744
  3. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  4. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 680
  5. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 583
  6. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 561
  7. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 542
  8. 2019CLEVRER: CoLlision Events for Video REpresentation and Reasoningcited by 484
  9. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 393
  10. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  11. 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 363
  12. 2020Just Ask: Learning to Answer Questions from Millions of Narrated Videoscited by 248
  13. 2024CogVLM2: Visual Language Models for Image and Video Understandingcited by 217
  14. 2024Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolutioncited by 159
  15. 2024Aria: An Open Multimodal Native Mixture-of-Experts Modelcited by 138
  16. 2024ST-LLM: Large Language Models Are Effective Temporal Learnerscited by 126
  17. 2022UniFormerV2: Spatiotemporal Learning by Arming Image ViTs with Video UniFormercited by 116
  18. 2024An Image Grid Can Be Worth a Video: Zero-shot Video Question Answering Using a VLMcited by 97
  19. 2022A Dataset for Medical Instructional Video Classification and Question Answeringcited by 26
  20. 2023HA-ViD: A Human Assembly Video Dataset for Comprehensive Assembly Knowledge Understandingcited by 18
  21. 2023SportsSloMo: A New Benchmark and Baselines for Human-centric Video Frame Interpolationcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 20; 2026: 50.

70 papers.

Sort:

By area:

  1. 2026Kimi K2.5: Visual Agentic Intelligencecited by 501
  2. 2025Towards Understanding Camera Motions in Any Videocited by 64
  3. 2026Video-MME-v2: Towards the Next Stage in Benchmarks for Comprehensive Video Understandingcited by 55
  4. 2026Seed1.8 Model Card: Towards Generalized Real-World Agencycited by 50
  5. 2025VideoHallu: Evaluating and Mitigating Multi-modal Hallucinations on Synthetic Video Understandingcited by 32
  6. 2025Improving LLM Video Understanding with 16 Frames Per Secondcited by 26
  7. 2025FAVOR-Bench: A Comprehensive Benchmark for Fine-Grained Video Motion Understandingcited by 24
  8. 2025TRAVL: A Recipe for Making Video-Language Models Better Judges of Physics Implausibilitycited by 18
  9. 2026MolmoPoint: Better Pointing for VLMs with Grounding Tokenscited by 15
  10. 2026JoyAI-VL-Interaction: Real-Time Vision-Language Interaction Intelligencecited by 14
  11. 2025MotionSight: Boosting Fine-Grained Motion Understanding in Multimodal LLMscited by 11
  12. 2025Map the Flow: Revealing Hidden Pathways of Information in VideoLLMscited by 10
38 more
  1. 2026VideoChat3: Fully Open Video MLLM for Efficient and Generalist Video Understandingcited by 10
  2. 2026CamReasoner: Reinforcing Camera Movement Understanding via Structured Spatial Reasoningcited by 9
  3. 2026MA-Bench: Towards Fine-grained Micro-Action Understandingcited by 9
  4. 2026SAMA: Factorized Semantic Anchoring and Motion Alignment for Instruction-Guided Video Editingcited by 8
  5. 2026CapRL++: Unified Reinforcement Learning with Verifiable Rewards for Dense Image and Video Captioningcited by 8
  6. 2026AdaCodec: A Predictive Visual Code for Video MLLMscited by 7
  7. 2025FaVChat: Hierarchical Prompt-Query Guided Facial Video Understanding with Data-Efficient GRPOcited by 6
  8. 2025MOVE: Motion-Guided Few-Shot Video Object Segmentationcited by 6
  9. 2025DeepSport: A Multimodal Large Language Model for Comprehensive Sports Video Reasoning via Agentic Reinforcement Learningcited by 5
  10. 2025VideoPerceiver: Enhancing Fine-Grained Temporal Perception in Video Multimodal Large Language Modelscited by 5
  11. 2026Moment-Video: Diagnosing Temporal Fidelity of Video MLLMs on Momentary Visual Eventscited by 5
  12. 2025From My View to Yours: Learning Egocentric Cues from Exocentric Video using Privileged Egocentric Supervisioncited by 4
  13. 2026Motion-o: Trajectory-Grounded Video Reasoningcited by 4
  14. 2026Watch, Remember, Reason: Human-View Video Understanding with MLLMscited by 4
  15. 2025Learning to Refuse: Refusal-Aware Reinforcement Fine-Tuning for Hard-Irrelevant Queries in Video Temporal Groundingcited by 3
  16. 2026EventFlash: Towards Efficient MLLMs for Event-Based Visioncited by 3
  17. 2026Can Vision-Language Models Solve the Shell Game?cited by 3
  18. 2026HumanScore: Benchmarking Human Motions in Generated Videoscited by 3
  19. 2026Learning from the Near Future: Temporal Self-Distillation for RLVRcited by 3
  20. 2026PhyMotion: Structured 3D Motion Reward for Physics-Grounded Human Video Generationcited by 3
  21. 2025MOAT: Evaluating LMMs for Capability Integration and Instruction Groundingcited by 2
  22. 2025The Escalator Problem: Identifying Implicit Motion Blindness in AI for Accessibilitycited by 2
  23. 2025HumanPCR: Probing MLLM Capabilities in Diverse Human-Centric Scenescited by 2
  24. 2025Video Parallel Scaling: Aggregating Diverse Frame Subsets for VideoLLMscited by 2
  25. 2026It's a Matter of Time: Three Lessons on Long-Term Motion for Perceptioncited by 2
  26. 2026KPM-Bench: A Kinematic Parsing Motion Benchmark for Fine-grained Motion-centric Video Understandingcited by 2
  27. 2026U-Mind: A Unified Framework for Real-Time Multimodal Interaction with Audiovisual Generationcited by 2
  28. 2026Which Way Did It Move? Diagnosing and Overcoming Directional Motion Blindness in Video-LLMscited by 2
  29. 2026EgoPolice: A Benchmark for Egocentric Video Understanding in High-Stakes Police Body-Worn Camera Footagecited by 2
  30. 2025EventBench: Towards Comprehensive Benchmarking of Event-based MLLMscited by 1
  31. 2026Stress Tests REVEAL Fragile Temporal and Visual Grounding in Video-Language Modelscited by 1
  32. 2026Unified Spatio-Temporal Token Scoring for Efficient Video VLMscited by 1
  33. 2026EasyVideoR1: Easier RL for Video Understandingcited by 1
  34. 2026Exploring High-Order Self-Similarity for Video Understandingcited by 1
  35. 2026PolarVSR: A Unified Framework and Benchmark for Continuous Space-Time Polarization Video Reconstructioncited by 1
  36. 2026Learning Spatiotemporal Sensitivity in Video LLMs via Counterfactual Reinforcement Learningcited by 1
  37. 2026Learning to Solve, Forgetting to Retain: Correct-Set Turnover in RLVRcited by 1
  38. 2026MotionAtlas: Detailed Region Captioning for Motion-Centric Videoscited by 1

50 most cited of 70 All 70 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026FineBench: Benchmarking and Enhancing Vision-Language Models for Fine-grained Human Activity Understanding
  2. 2024MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding
  3. 2026MVPBench: A Multi-Video Perception Evaluation Benchmark for Multi-Modal Video Understanding
  4. 2026NextMotionQA: Benchmarking and Judging Human Motion Understanding with Vision-Language Models
  5. 2024TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Models
  6. 2026MotionBlind: Probing the Illusion of Motion Understanding in Video-LLMs
  7. 2026FastBench: Can Streaming VLMs Perceive High-Dynamic Real-World Streams?
  8. 2024Fewer Tokens and Fewer Videos: Extending Video Understanding Abilities in Large Vision-Language Models