# GAIA: Rethinking Action Quality Assessment for AI-Generated Videos

arXiv: https://arxiv.org/abs/2406.06087 · HTML: https://mlsanity.com/paper/2406.06087/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 78 of its 122 references that are in the map.
- First posted on arXiv 10 Jun 2024.

## Builds on (most cited first, 30 of 78)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [VBench: Comprehensive Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2311.17982/) (2023)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021)
- [AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning](https://mlsanity.com/paper/2307.04725/) (2023)
- [Towards Accurate Generative Models of Video: A New Metric & Challenges](https://mlsanity.com/paper/1812.01717/) (2018)
- [ShareGPT4V: Improving Large Multi-Modal Models with Better Captions](https://mlsanity.com/paper/2311.12793/) (2023)
- [CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers](https://mlsanity.com/paper/2205.15868/) (2022)
- [Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators](https://mlsanity.com/paper/2303.13439/) (2023)
- [ModelScope Text-to-Video Technical Report](https://mlsanity.com/paper/2308.06571/) (2023)
- [Structure and Content-Guided Video Synthesis with Diffusion Models](https://mlsanity.com/paper/2302.03011/) (2023)
- [VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models](https://mlsanity.com/paper/2401.09047/) (2024)
- [VideoCrafter1: Open Diffusion Models for High-Quality Video Generation](https://mlsanity.com/paper/2310.19512/) (2023)

## Built on by (most cited first, 30 of 30)

- [Quality Assessment in the Era of Large Models: A Survey](https://mlsanity.com/paper/2409.00031/) (2024)
- [AIGV-Assessor: Benchmarking and Evaluating the Perceptual Quality of Text-to-Video Generation with LMM](https://mlsanity.com/paper/2411.17221/) (2024)
- [OBI-Bench: Can LMMs Aid in Study of Ancient Script on Oracle Bones?](https://mlsanity.com/paper/2412.01175/) (2024)
- [A Comprehensive Survey of Action Quality Assessment: Method and Benchmark](https://mlsanity.com/paper/2412.11149/) (2024)
- [Video Quality Assessment: A Comprehensive Survey](https://mlsanity.com/paper/2412.04508/) (2024)
- [Benchmarking Multi-dimensional AIGC Video Quality Assessment: A Dataset and Unified Model](https://mlsanity.com/paper/2407.21408/) (2024)
- [LOVE: Benchmarking and Evaluating Text-to-Video Generation and Video-to-Text Interpretation](https://mlsanity.com/paper/2505.12098/) (2025)
- [ETVA: Evaluation of Text-to-Video Alignment via Fine-grained Question Generation and Answering](https://mlsanity.com/paper/2503.16867/) (2025)
- [Just Noticeable Difference for Large Multimodal Models](https://mlsanity.com/paper/2507.00490/) (2025)
- [A Decade of Action Quality Assessment: Largest Systematic Survey of Trends, Challenges, and Future Directions](https://mlsanity.com/paper/2502.02817/) (2025)
- [LMME3DHF: Benchmarking and Evaluating Multimodal 3D Human Face Generation with LMMs](https://mlsanity.com/paper/2504.20466/) (2025)
- [DFBench: Benchmarking Deepfake Image Detection Capability of Large Multimodal Models](https://mlsanity.com/paper/2506.03007/) (2025)
- [T2VEval: Benchmark Dataset and Objective Evaluation Method for T2V-generated Videos](https://mlsanity.com/paper/2501.08545/) (2025)
- [Towards All-in-One Medical Image Re-Identification](https://mlsanity.com/paper/2503.08173/) (2025)
- [VisioMath: Benchmarking Figure-based Mathematical Reasoning in LMMs](https://mlsanity.com/paper/2506.06727/) (2025)
- [Can Large Models Fool the Eye? A New Turing Test for Biological Animation](https://mlsanity.com/paper/2508.06072/) (2025)
- [SPIRAL: Self-Evolving Action-Conditioned Video Generation via Reflective Planning Agents](https://mlsanity.com/paper/2603.08403/) (2026)
- [Medical Manifestation-Aware De-Identification](https://mlsanity.com/paper/2412.10804/) (2024)
- [Engagement Prediction of Short Videos with Large Multimodal Models](https://mlsanity.com/paper/2508.02516/) (2025)
- [Q-Save: Towards Scoring and Attribution for Generated Video Evaluation](https://mlsanity.com/paper/2511.18825/) (2025)
- [Towards Holistic Visual Quality Assessment of AI-Generated Videos: A LLM-Based Multi-Dimensional Evaluation Model](https://mlsanity.com/paper/2506.04715/) (2025)
- [Learning Long-Range Action Representation by Two-Stream Mamba Pyramid Network for Figure Skating Assessment](https://mlsanity.com/paper/2508.16291/) (2025)
- [GeneVA: A Dataset of Human Annotations for Generative Text to Video Artifacts](https://mlsanity.com/paper/2509.08818/) (2025)
- [FLEX: A Largescale Multimodal, Multiview Dataset for Learning Structured Representations for Fitness Action Quality Assessment](https://mlsanity.com/paper/2506.03198/) (2025)
- [ROFI: A Deep Learning-Based Ophthalmic Sign-Preserving and Reversible Patient Face Anonymizer](https://mlsanity.com/paper/2510.11073/) (2025)
- [VITAL: Vision-Encoder-centered Pre-training for LMMs in Visual Quality Assessment](https://mlsanity.com/paper/2511.17962/) (2025)
- [EduVQA: Towards Concept-Aware Assessment of Educational AI-Generated Videos](https://mlsanity.com/paper/2603.03066/) (2026)
- [MyoMechanix: Biomechanically-Grounded Compositional Skilled Activity Understanding and Coaching](https://mlsanity.com/paper/2608.26094/) (2026)
- [LLaVA-Assessor: Building the Foundation LMM For Visual Quality Assessment](https://mlsanity.com/paper/2609.26205/) (2026)
- [REMEDY: How Far Is Video Generation from Medical Education World Models?](https://mlsanity.com/paper/2609.32460/) (2026)

## Abstract (arXiv)

Assessing action quality is both imperative and challenging due to its significant impact on the quality of AI-generated videos, further complicated by the inherently ambiguous nature of actions within AI-generated video (AIGV). Current action quality assessment (AQA) algorithms predominantly focus on actions from real specific scenarios and are pre-trained with normative action features, thus rendering them inapplicable in AIGVs. To address these problems, we construct GAIA, a Generic AI-generated Action dataset, by conducting a large-scale subjective evaluation from a novel causal reasoning-based perspective, resulting in 971,244 ratings among 9,180 video-action pairs. Based on GAIA, we evaluate a suite of popular text-to-video (T2V) models on their ability to generate visually rational actions, revealing their pros and cons on different categories of actions. We also extend GAIA as a testbed to benchmark the AQA capacity of existing automatic evaluation methods. Results show that traditional AQA methods, action-related metrics in recent T2V benchmarks, and mainstream video quality methods perform poorly with an average SRCC of 0.454, 0.191, and 0.519, respectively, indicating a sizable gap between current models and human action perception patterns in AIGVs. Our findings underscore the significance of action quality as a unique perspective for studying AIGVs and can catalyze progress towards methods with enhanced capacities for AQA in AIGVs.