# LMM-VQA: Advancing Video Quality Assessment with Large Multimodal Models

arXiv: https://arxiv.org/abs/2408.14008 · HTML: https://mlsanity.com/paper/2408.14008/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Placed in Image editing and quality assessment, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 61 of its 88 references that are in the map.
- First posted on arXiv 26 Aug 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Image editing and quality assessment](https://mlsanity.com/mllm/image-editing-and-quality-assessment/)

## Builds on (most cited first, 30 of 61)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [Video-LLaVA: Learning United Visual Representation by Alignment Before Projection](https://mlsanity.com/paper/2311.10122/) (2023)
- [Perceiver: General Perception with Iterative Attention](https://mlsanity.com/paper/2103.03206/) (2021)
- [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021)
- [Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models](https://mlsanity.com/paper/2306.05424/) (2023)
- [VideoChat: Chat-Centric Video Understanding](https://mlsanity.com/paper/2305.06355/) (2023)
- [Image Quality Assessment: Unifying Structure and Texture Similarity](https://mlsanity.com/paper/2004.07728/) (2020)
- [LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention](https://mlsanity.com/paper/2303.16199/) (2023)
- [VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs](https://mlsanity.com/paper/2406.07476/) (2024)
- [VILA: On Pre-training for Visual Language Models](https://mlsanity.com/paper/2312.07533/) (2023)
- [LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models](https://mlsanity.com/paper/2407.07895/) (2024)
- [mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration](https://mlsanity.com/paper/2311.04257/) (2023)
- [LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models](https://mlsanity.com/paper/2311.17043/) (2023)
- [Q-Align: Teaching LMMs for Visual Scoring via Discrete Text-Defined Levels](https://mlsanity.com/paper/2312.17090/) (2023)
- [KonIQ-10k: An ecologically valid database for deep learning of blind image quality assessment](https://mlsanity.com/paper/1910.06180/) (2019)
- [Exploring Video Quality Assessment on User Generated Contents from Aesthetic and Technical Perspectives](https://mlsanity.com/paper/2211.04894/) (2022)
- [InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Composition](https://mlsanity.com/paper/2309.15112/) (2023)

## Built on by (most cited first, 28 of 28)

- [Q-Bench-Video: Benchmarking the Video Quality Understanding of LMMs](https://mlsanity.com/paper/2409.20063/) (2024)
- [FineVQ: Fine-Grained User Generated Content Video Quality Assessment](https://mlsanity.com/paper/2412.19238/) (2024)
- [CVBench: Benchmarking Cross-Video Synergies for Complex Multimodal Reasoning](https://mlsanity.com/paper/2508.19542/) (2025)
- [Video Quality Assessment: A Comprehensive Survey](https://mlsanity.com/paper/2412.04508/) (2024)
- [AGHI-QA: A Subjective-Aligned Dataset and Metric for AI-Generated Human Images](https://mlsanity.com/paper/2504.21308/) (2025)
- [VQQA: An Agentic Approach for Video Evaluation and Quality Improvement](https://mlsanity.com/paper/2603.12310/) (2026)
- [RadarQA: Multi-modal Quality Analysis of Weather Radar Forecasts](https://mlsanity.com/paper/2508.12291/) (2025)
- [Generalizable Video Quality Assessment via Weak-to-Strong Learning](https://mlsanity.com/paper/2505.03631/) (2025)
- [Scaling-up Perceptual Video Quality Assessment](https://mlsanity.com/paper/2505.22543/) (2025)
- [An Ensemble Approach to Short-form Video Quality Assessment Using Multimodal LLM](https://mlsanity.com/paper/2412.18060/) (2024)
- [Engagement Prediction of Short Videos with Large Multimodal Models](https://mlsanity.com/paper/2508.02516/) (2025)
- [Efficient Face Image Quality Assessment via Self-training and Knowledge Distillation](https://mlsanity.com/paper/2507.15709/) (2025)
- [DHQA-4D: Perceptual Quality Assessment of Dynamic 4D Digital Human](https://mlsanity.com/paper/2510.03874/) (2025)
- [Towards Holistic Visual Quality Assessment of AI-Generated Videos: A LLM-Based Multi-Dimensional Evaluation Model](https://mlsanity.com/paper/2506.04715/) (2025)
- [CAMP-VQA: Caption-Embedded Multimodal Perception for No-Reference Quality Assessment of Compressed Video](https://mlsanity.com/paper/2511.07290/) (2025)
- [Q-Bench-Portrait: Benchmarking Multimodal Large Language Models on Portrait Image Quality Perception](https://mlsanity.com/paper/2601.18346/) (2026)
- [Audit After Segmentation: Reference-Free Mask Quality Assessment for Language-Referred Audio-Visual Segmentation](https://mlsanity.com/paper/2602.03892/) (2026)
- [Context and Pixel Aware Large Language Model for Video Quality Assessment](https://mlsanity.com/paper/2505.16025/) (2025)
- [Exploring Instruction Data Quality for Explainable Image Quality Assessment](https://mlsanity.com/paper/2510.03880/) (2025)
- [VideoAesBench: Benchmarking the Video Aesthetics Perception Capabilities of Large Multimodal Models](https://mlsanity.com/paper/2601.21915/) (2026)
- [GLIMPSE : Real-Time Text Recognition and Contextual Understanding for VQA in Wearables](https://mlsanity.com/paper/2602.13479/) (2026)
- [Score2Instruct: Scaling Up Video Quality-Centric Instructions via Automated Dimension Scoring](https://mlsanity.com/paper/2506.21011/) (2025)
- [FDIM: A Feature-distance-based Generic Video Quality Metric for Versatile Codecs](https://mlsanity.com/paper/2604.24123/) (2026)
- [FGSVQA: Frequency-Guided Short-form Video Quality Assessment](https://mlsanity.com/paper/2605.20016/) (2026)
- [Community-Aware Assessment of Social Textual Engagement and Resonance: A Human-Centric Perspective on User-Generated Content Evaluation](https://mlsanity.com/paper/2606.01897/) (2026)
- [LEIQ-Assessor: Multi-dimensional Quality Assessment of Low-light Enhanced Images via Multi-task Learning](https://mlsanity.com/paper/2606.29752/) (2026)
- [LLaVA-Assessor: Building the Foundation LMM For Visual Quality Assessment](https://mlsanity.com/paper/2609.26205/) (2026)
- [Dyna3: VLM-Guided Training-Free 4D Reconstruction via Depth Foundation Models](https://mlsanity.com/paper/2610.01286/) (2026)

## Abstract (arXiv)

The explosive growth of videos on streaming media platforms has underscored the urgent need for effective video quality assessment (VQA) algorithms to monitor and perceptually optimize the quality of streaming videos. However, VQA remains an extremely challenging task due to the diverse video content and the complex spatial and temporal distortions, thus necessitating more advanced methods to address these issues. Nowadays, large multimodal models (LMMs), such as GPT-4V, have exhibited strong capabilities for various visual understanding tasks, motivating us to leverage the powerful multimodal representation ability of LMMs to solve the VQA task. Therefore, we propose the first Large Multi-Modal Video Quality Assessment (LMM-VQA) model, which introduces a novel spatiotemporal visual modeling strategy for quality-aware feature extraction. Specifically, we first reformulate the quality regression problem into a question and answering (Q&A) task and construct Q&A prompts for VQA instruction tuning. Then, we design a spatiotemporal vision encoder to extract spatial and temporal features to represent the quality characteristics of videos, which are subsequently mapped into the language space by the spatiotemporal projector for modality alignment. Finally, the aligned visual tokens and the quality-inquired text tokens are aggregated as inputs for the large language model (LLM) to generate the quality score and level. Extensive experiments demonstrate that LMM-VQA achieves state-of-the-art performance across five VQA benchmarks, exhibiting an average improvement of 5% in generalization ability over existing methods. Furthermore, due to the advanced design of the spatiotemporal encoder and projector, LMM-VQA also performs exceptionally well on general video understanding tasks, further validating its effectiveness. Our code will be released at https://github.com/Sueqk/LMM-VQA.