# Q-Bench-Video: Benchmarking the Video Quality Understanding of LMMs

arXiv: https://arxiv.org/abs/2409.20063 · HTML: https://mlsanity.com/paper/2409.20063/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 46 of its 65 references that are in the map.
- First posted on arXiv 30 Sep 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Video understanding with MLLMs](https://mlsanity.com/mllm/video-understanding-with-mllms/)

## Builds on (most cited first, 30 of 46)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models](https://mlsanity.com/paper/2306.13394/) (2023)
- [Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis](https://mlsanity.com/paper/2405.21075/) (2024)
- [Video-LLaVA: Learning United Visual Representation by Alignment Before Projection](https://mlsanity.com/paper/2311.10122/) (2023)
- [OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge](https://mlsanity.com/paper/1906.00067/) (2019)
- [MVBench: A Comprehensive Multi-modal Video Understanding Benchmark](https://mlsanity.com/paper/2311.17005/) (2023)
- [How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites](https://mlsanity.com/paper/2404.16821/) (2024)
- [mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality](https://mlsanity.com/paper/2304.14178/) (2023)
- [EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding](https://mlsanity.com/paper/2308.09126/) (2023)
- [LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding](https://mlsanity.com/paper/2407.15754/) (2024)
- [mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration](https://mlsanity.com/paper/2311.04257/) (2023)
- [Q-Align: Teaching LMMs for Visual Scoring via Discrete Text-Defined Levels](https://mlsanity.com/paper/2312.17090/) (2023)
- [PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioning](https://mlsanity.com/paper/2404.16994/) (2024)
- [Exploring Video Quality Assessment on User Generated Contents from Aesthetic and Technical Perspectives](https://mlsanity.com/paper/2211.04894/) (2022)
- [InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Composition](https://mlsanity.com/paper/2309.15112/) (2023)
- [mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models](https://mlsanity.com/paper/2408.04840/) (2024)
- [Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Vision](https://mlsanity.com/paper/2309.14181/) (2023)
- [VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generation](https://mlsanity.com/paper/2406.15252/) (2024)
- [MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding](https://mlsanity.com/paper/2406.14515/) (2024)
- [FAST-VQA: Efficient End-to-end Video Quality Assessment with Fragment Sampling](https://mlsanity.com/paper/2207.02595/) (2022)
- [Quality Assessment of In-the-Wild Videos](https://mlsanity.com/paper/1908.00375/) (2019)
- [Patch-VQ: 'Patching Up' the Video Quality Problem](https://mlsanity.com/paper/2011.13544/) (2020)
- [A Deep Learning based No-reference Quality Assessment Model for UGC Videos](https://mlsanity.com/paper/2204.14047/) (2022)
- [ST-LLM: Large Language Models Are Effective Temporal Learners](https://mlsanity.com/paper/2404.00308/) (2024)
- [Blindly Assess Quality of In-the-Wild Videos via Quality-aware Pre-training and Motion Perception](https://mlsanity.com/paper/2108.08505/) (2021)
- [VILA: Learning Image Aesthetics from User Comments with Vision-Language Pretraining](https://mlsanity.com/paper/2303.14302/) (2023)

## Built on by (most cited first, 30 of 39)

- [Q-Insight: Understanding Image Quality via Visual Reinforcement Learning](https://mlsanity.com/paper/2503.22679/) (2025)
- [ECBench: Can Multi-modal Foundation Models Understand the Egocentric World? A Holistic Embodied Cognition Benchmark](https://mlsanity.com/paper/2501.05031/) (2025)
- [VQA2: Visual Question Answering for Video Quality Assessment](https://mlsanity.com/paper/2411.03795/) (2024)
- [Q-Ponder: A Unified Training Pipeline for Reasoning-based Visual Quality Assessment](https://mlsanity.com/paper/2506.05384/) (2025)
- [MEMO-Bench: A Multiple Benchmark for Text-to-Image and Multimodal Large Language Models on Human Emotion Analysis](https://mlsanity.com/paper/2411.11235/) (2024)
- [VQAThinker: Exploring Generalizable and Explainable Video Quality Assessment via Reinforcement Learning](https://mlsanity.com/paper/2508.06051/) (2025)
- [Generative World Renderer](https://mlsanity.com/paper/2604.02329/) (2026)
- [RadarQA: Multi-modal Quality Analysis of Weather Radar Forecasts](https://mlsanity.com/paper/2508.12291/) (2025)
- [UVE: Are MLLMs Unified Evaluators for AI-Generated Videos?](https://mlsanity.com/paper/2503.09949/) (2025)
- [NTIRE 2025 XGC Quality Assessment Challenge: Methods and Results](https://mlsanity.com/paper/2506.02875/) (2025)
- [Scaling-up Perceptual Video Quality Assessment](https://mlsanity.com/paper/2505.22543/) (2025)
- [UARE: A Unified Vision-Language Model for Image Quality Assessment, Restoration, and Enhancement](https://mlsanity.com/paper/2512.06750/) (2025)
- [Q-Save: Towards Scoring and Attribution for Generated Video Evaluation](https://mlsanity.com/paper/2511.18825/) (2025)
- [PuzzleCraft: Exploration-Aware Curriculum Learning for Puzzle-Based RLVR in VLMs](https://mlsanity.com/paper/2512.14944/) (2025)
- [AVIS: Adaptive Test-Time Scaling for Vision-Language Models](https://mlsanity.com/paper/2606.11576/) (2026)
- [Looking Beyond the Obvious: A Survey on Abstract Concept Recognition for Video Understanding](https://mlsanity.com/paper/2508.20765/) (2025)
- [OBIFormer: A Fast Attentive Denoising Framework for Oracle Bone Inscriptions](https://mlsanity.com/paper/2504.13524/) (2025)
- [Q-Router: Agentic Video Quality Assessment with Expert Model Routing and Artifact Localization](https://mlsanity.com/paper/2510.08789/) (2025)
- [QualiRAG: Retrieval-Augmented Generation for Visual Quality Understanding](https://mlsanity.com/paper/2601.18195/) (2026)
- [Q-Bench-Portrait: Benchmarking Multimodal Large Language Models on Portrait Image Quality Perception](https://mlsanity.com/paper/2601.18346/) (2026)
- [Audit After Segmentation: Reference-Free Mask Quality Assessment for Language-Referred Audio-Visual Segmentation](https://mlsanity.com/paper/2602.03892/) (2026)
- [Context and Pixel Aware Large Language Model for Video Quality Assessment](https://mlsanity.com/paper/2505.16025/) (2025)
- [Who is a Better Player: LLM against LLM](https://mlsanity.com/paper/2508.04720/) (2025)
- [Exploring Instruction Data Quality for Explainable Image Quality Assessment](https://mlsanity.com/paper/2510.03880/) (2025)
- [VITAL: Vision-Encoder-centered Pre-training for LMMs in Visual Quality Assessment](https://mlsanity.com/paper/2511.17962/) (2025)
- [VideoAesBench: Benchmarking the Video Aesthetics Perception Capabilities of Large Multimodal Models](https://mlsanity.com/paper/2601.21915/) (2026)
- [Seeing Beyond 8bits: Subjective and Objective Quality Assessment of HDR-UGC Videos](https://mlsanity.com/paper/2603.00938/) (2026)
- [DPC-VQA: Decoupling Quality Perception and Residual Calibration for Video Quality Assessment](https://mlsanity.com/paper/2604.12813/) (2026)
- [LoViF 2026 The First Challenge on Holistic Quality Assessment for 4D World Model (PhyScore)](https://mlsanity.com/paper/2605.05187/) (2026)
- [Beyond Trial-and-Error: Agentic Optimization for Image-to-Video Adherence](https://mlsanity.com/paper/2608.12290/) (2026)

## Abstract (arXiv)

With the rising interest in research on Large Multi-modal Models (LMMs) for video understanding, many studies have emphasized general video comprehension capabilities, neglecting the systematic exploration into video quality understanding. To address this oversight, we introduce Q-Bench-Video in this paper, a new benchmark specifically designed to evaluate LMMs' proficiency in discerning video quality. a) To ensure video source diversity, Q-Bench-Video encompasses videos from natural scenes, AI-generated Content (AIGC), and Computer Graphics (CG). b) Building on the traditional multiple-choice questions format with the Yes-or-No and What-How categories, we include Open-ended questions to better evaluate complex scenarios. Additionally, we incorporate the video pair quality comparison question to enhance comprehensiveness. c) Beyond the traditional Technical, Aesthetic, and Temporal distortions, we have expanded our evaluation aspects to include the dimension of AIGC distortions, which addresses the increasing demand for video generation. Finally, we collect a total of 2,378 question-answer pairs and test them on 12 open-source & 5 proprietary LMMs. Our findings indicate that while LMMs have a foundational understanding of video quality, their performance remains incomplete and imprecise, with a notable discrepancy compared to human performance. Through Q-Bench-Video, we seek to catalyze community interest, stimulate further research, and unlock the untapped potential of LMMs to close the gap in video quality understanding.