# MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models

arXiv: https://arxiv.org/abs/2501.02955 · HTML: https://mlsanity.com/paper/2501.02955/

## Key facts

- Cited by 70 papers in the map (data through 14 Oct 2026).
- Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 43 of its 58 references that are in the map.
- First posted on arXiv 6 Jan 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Video understanding with MLLMs](https://mlsanity.com/mllm/video-understanding-with-mllms/)

## Builds on (most cited first, 30 of 43)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis](https://mlsanity.com/paper/2405.21075/) (2024)
- [ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools](https://mlsanity.com/paper/2406.12793/) (2024)
- [MVBench: A Comprehensive Multi-modal Video Understanding Benchmark](https://mlsanity.com/paper/2311.17005/) (2023)
- [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017)
- [MiniCPM-V: A GPT-4V Level MLLM on Your Phone](https://mlsanity.com/paper/2408.01800/) (2024)
- [Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models](https://mlsanity.com/paper/2306.05424/) (2023)
- [VideoChat: Chat-Centric Video Understanding](https://mlsanity.com/paper/2305.06355/) (2023)
- [NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actions](https://mlsanity.com/paper/2105.08276/) (2021)
- [CogVLM: Visual Expert for Pretrained Language Models](https://mlsanity.com/paper/2311.03079/) (2023)
- [EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding](https://mlsanity.com/paper/2308.09126/) (2023)
- [LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding](https://mlsanity.com/paper/2407.15754/) (2024)
- [LLaVA-Video: Video Instruction Tuning With Synthetic Data](https://mlsanity.com/paper/2410.02713/) (2024)
- [ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answering](https://mlsanity.com/paper/1906.02467/) (2019)
- [MovieChat: From Dense Token to Sparse Memory for Long Video Understanding](https://mlsanity.com/paper/2307.16449/) (2023)
- [CLEVRER: CoLlision Events for Video REpresentation and Reasoning](https://mlsanity.com/paper/1910.01442/) (2019)
- [LVBench: An Extreme Long Video Understanding Benchmark](https://mlsanity.com/paper/2406.08035/) (2024)
- [Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers](https://mlsanity.com/paper/2402.19479/) (2024)
- [TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering](https://mlsanity.com/paper/1704.04497/) (2017)
- [TempCompass: Do Video LLMs Really Understand Videos?](https://mlsanity.com/paper/2403.00476/) (2024)
- [PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioning](https://mlsanity.com/paper/2404.16994/) (2024)
- [Just Ask: Learning to Answer Questions from Millions of Narrated Videos](https://mlsanity.com/paper/2012.00451/) (2020)

## Built on by (most cited first, 30 of 70)

- [Kimi K2.5: Visual Agentic Intelligence](https://mlsanity.com/paper/2602.02276/) (2026)
- [Towards Understanding Camera Motions in Any Video](https://mlsanity.com/paper/2504.15376/) (2025)
- [Video-MME-v2: Towards the Next Stage in Benchmarks for Comprehensive Video Understanding](https://mlsanity.com/paper/2604.05015/) (2026)
- [Seed1.8 Model Card: Towards Generalized Real-World Agency](https://mlsanity.com/paper/2603.20633/) (2026)
- [VideoHallu: Evaluating and Mitigating Multi-modal Hallucinations on Synthetic Video Understanding](https://mlsanity.com/paper/2505.01481/) (2025)
- [Improving LLM Video Understanding with 16 Frames Per Second](https://mlsanity.com/paper/2503.13956/) (2025)
- [FAVOR-Bench: A Comprehensive Benchmark for Fine-Grained Video Motion Understanding](https://mlsanity.com/paper/2503.14935/) (2025)
- [TRAVL: A Recipe for Making Video-Language Models Better Judges of Physics Implausibility](https://mlsanity.com/paper/2510.07550/) (2025)
- [MolmoPoint: Better Pointing for VLMs with Grounding Tokens](https://mlsanity.com/paper/2603.28069/) (2026)
- [JoyAI-VL-Interaction: Real-Time Vision-Language Interaction Intelligence](https://mlsanity.com/paper/2606.14777/) (2026)
- [MotionSight: Boosting Fine-Grained Motion Understanding in Multimodal LLMs](https://mlsanity.com/paper/2506.01674/) (2025)
- [Map the Flow: Revealing Hidden Pathways of Information in VideoLLMs](https://mlsanity.com/paper/2510.13251/) (2025)
- [VideoChat3: Fully Open Video MLLM for Efficient and Generalist Video Understanding](https://mlsanity.com/paper/2607.14935/) (2026)
- [CamReasoner: Reinforcing Camera Movement Understanding via Structured Spatial Reasoning](https://mlsanity.com/paper/2602.00181/) (2026)
- [MA-Bench: Towards Fine-grained Micro-Action Understanding](https://mlsanity.com/paper/2603.26586/) (2026)
- [SAMA: Factorized Semantic Anchoring and Motion Alignment for Instruction-Guided Video Editing](https://mlsanity.com/paper/2603.19228/) (2026)
- [CapRL++: Unified Reinforcement Learning with Verifiable Rewards for Dense Image and Video Captioning](https://mlsanity.com/paper/2606.09393/) (2026)
- [AdaCodec: A Predictive Visual Code for Video MLLMs](https://mlsanity.com/paper/2606.02569/) (2026)
- [FaVChat: Hierarchical Prompt-Query Guided Facial Video Understanding with Data-Efficient GRPO](https://mlsanity.com/paper/2503.09158/) (2025)
- [MOVE: Motion-Guided Few-Shot Video Object Segmentation](https://mlsanity.com/paper/2507.22061/) (2025)
- [DeepSport: A Multimodal Large Language Model for Comprehensive Sports Video Reasoning via Agentic Reinforcement Learning](https://mlsanity.com/paper/2511.12908/) (2025)
- [VideoPerceiver: Enhancing Fine-Grained Temporal Perception in Video Multimodal Large Language Models](https://mlsanity.com/paper/2511.18823/) (2025)
- [Moment-Video: Diagnosing Temporal Fidelity of Video MLLMs on Momentary Visual Events](https://mlsanity.com/paper/2606.02522/) (2026)
- [From My View to Yours: Learning Egocentric Cues from Exocentric Video using Privileged Egocentric Supervision](https://mlsanity.com/paper/2501.05711/) (2025)
- [Motion-o: Trajectory-Grounded Video Reasoning](https://mlsanity.com/paper/2603.18856/) (2026)
- [Watch, Remember, Reason: Human-View Video Understanding with MLLMs](https://mlsanity.com/paper/2606.07433/) (2026)
- [Learning to Refuse: Refusal-Aware Reinforcement Fine-Tuning for Hard-Irrelevant Queries in Video Temporal Grounding](https://mlsanity.com/paper/2511.23151/) (2025)
- [EventFlash: Towards Efficient MLLMs for Event-Based Vision](https://mlsanity.com/paper/2602.03230/) (2026)
- [Can Vision-Language Models Solve the Shell Game?](https://mlsanity.com/paper/2603.08436/) (2026)
- [HumanScore: Benchmarking Human Motions in Generated Videos](https://mlsanity.com/paper/2604.20157/) (2026)

## Abstract (arXiv)

In recent years, vision language models (VLMs) have made significant advancements in video understanding. However, a crucial capability - fine-grained motion comprehension - remains under-explored in current benchmarks. To address this gap, we propose MotionBench, a comprehensive evaluation benchmark designed to assess the fine-grained motion comprehension of video understanding models. MotionBench evaluates models' motion-level perception through six primary categories of motion-oriented question types and includes data collected from diverse sources, ensuring a broad representation of real-world video content. Experimental results reveal that existing VLMs perform poorly in understanding fine-grained motions. To enhance VLM's ability to perceive fine-grained motion within a limited sequence length of LLM, we conduct extensive experiments reviewing VLM architectures optimized for video feature compression and propose a novel and efficient Through-Encoder (TE) Fusion method. Experiments show that higher frame rate inputs and TE Fusion yield improvements in motion understanding, yet there is still substantial room for enhancement. Our benchmark aims to guide and motivate the development of more capable video understanding models, emphasizing the importance of fine-grained motion comprehension. Project page: https://motion-bench.github.io .