# Video-Bench: Human-Aligned Video Generation Benchmark

arXiv: https://arxiv.org/abs/2504.04907 · HTML: https://mlsanity.com/paper/2504.04907/

## Key facts

- Cited by 57 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion and multimodal model alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 57 of its 71 references that are in the map.
- First posted on arXiv 7 Apr 2025.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Diffusion and multimodal model alignment](https://mlsanity.com/alignment/diffusion-and-multimodal-model-alignment/)

## Builds on (most cited first, 30 of 57)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [VBench: Comprehensive Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2311.17982/) (2023)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [Towards Accurate Generative Models of Video: A New Metric & Challenges](https://mlsanity.com/paper/1812.01717/) (2018)
- [MUSIQ: Multi-scale Image Quality Transformer](https://mlsanity.com/paper/2108.05997/) (2021)
- [CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers](https://mlsanity.com/paper/2205.15868/) (2022)
- [Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation](https://mlsanity.com/paper/2212.11565/) (2022)
- [Vector Quantized Diffusion Model for Text-to-Image Synthesis](https://mlsanity.com/paper/2111.14822/) (2021)
- [Genie: Generative Interactive Environments](https://mlsanity.com/paper/2402.15391/) (2024)
- [ModelScope Text-to-Video Technical Report](https://mlsanity.com/paper/2308.06571/) (2023)
- [VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models](https://mlsanity.com/paper/2401.09047/) (2024)
- [InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation](https://mlsanity.com/paper/2307.06942/) (2023)
- [OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation](https://mlsanity.com/paper/2311.17911/) (2023)
- [TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering](https://mlsanity.com/paper/2303.11897/) (2023)
- [Latent Video Diffusion Models for High-Fidelity Long Video Generation](https://mlsanity.com/paper/2211.13221/) (2022)
- [MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark](https://mlsanity.com/paper/2402.04788/) (2024)
- [EvalCrafter: Benchmarking and Evaluating Large Video Generation Models](https://mlsanity.com/paper/2310.11440/) (2023)
- [Fake News Detection on Social Media: A Data Mining Perspective](https://mlsanity.com/paper/1708.01967/) (2017)
- [VideoFusion: Decomposed Diffusion Models for High-Quality Video Generation](https://mlsanity.com/paper/2303.08320/) (2023)

## Built on by (most cited first, 30 of 57)

- [Rethinking Video Generation Model for the Embodied World](https://mlsanity.com/paper/2601.15282/) (2026)
- [Can World Simulators Reason? Gen-ViRe: A Generative Visual Reasoning Benchmark](https://mlsanity.com/paper/2511.13853/) (2025)
- [MBench: A Comprehensive Benchmark on Memory Capability for Video World Models](https://mlsanity.com/paper/2606.00793/) (2026)
- [UI2V-Bench: An Understanding-based Image-to-video Generation Benchmark](https://mlsanity.com/paper/2509.24427/) (2025)
- [LPM 1.0: Video-based Character Performance Model](https://mlsanity.com/paper/2604.07823/) (2026)
- [MSAVBench: Towards Comprehensive and Reliable Evaluation of Multi-Shot Audio-Video Generation](https://mlsanity.com/paper/2605.20183/) (2026)
- [SCAIL-2: Unifying Controlled Character Animation with End-to-End In-Context Conditioning](https://mlsanity.com/paper/2606.10804/) (2026)
- [LoCoT2V-Bench: Benchmarking Long-Form and Complex Text-to-Video Generation](https://mlsanity.com/paper/2510.26412/) (2025)
- [Benchmarking Scientific Understanding and Reasoning for Video Generation using VideoScience-Bench](https://mlsanity.com/paper/2512.02942/) (2025)
- [VideoWeaver: Evaluating and Evolving Skills for Agentic Long Video Generation](https://mlsanity.com/paper/2606.08091/) (2026)
- [ViBES: A Conversational Agent with Behaviorally-Intelligent 3D Virtual Body](https://mlsanity.com/paper/2512.14234/) (2025)
- [EntityBench: Towards Entity-Consistent Long-Range Multi-Shot Video Generation](https://mlsanity.com/paper/2605.15199/) (2026)
- [ReCA: Multi-Shot Long Video Extrapolation via Recursive Context Allocation](https://mlsanity.com/paper/2605.26525/) (2026)
- [ViStoryBench: Comprehensive Benchmark Suite for Story Visualization](https://mlsanity.com/paper/2505.24862/) (2025)
- [DreamActor-M2: Universal Character Image Animation via Spatiotemporal In-Context Learning](https://mlsanity.com/paper/2601.21716/) (2026)
- [SPIRAL: Self-Evolving Action-Conditioned Video Generation via Reflective Planning Agents](https://mlsanity.com/paper/2603.08403/) (2026)
- [Video Models Start to Solve Chess, Maze, Sudoku, Mental Rotation, and Raven' Matrices](https://mlsanity.com/paper/2512.05969/) (2025)
- [Learning Visual Feature-Based World Models via Residual Latent Action](https://mlsanity.com/paper/2605.07079/) (2026)
- [MTAVG-Bench 2.0: Diagnosing Failure Modes of Cinematic Expressiveness in Multi-Talker Audio-Video Generation](https://mlsanity.com/paper/2605.28035/) (2026)
- [DirectorBench: Diagnosing Long-Form Video Generation with Personalized Multi-Agent Evaluation](https://mlsanity.com/paper/2605.30090/) (2026)
- [HarnessEval-W: Agentifying the Evaluation of Visual Worlds](https://mlsanity.com/paper/2608.16859/) (2026)
- [From Inpainting to Editing: Unlocking Robust Mask-Free Visual Dubbing via Generative Bootstrapping](https://mlsanity.com/paper/2512.25066/) (2025)
- [Omni-Judge: Can Omni-LLMs Serve as Human-Aligned Judges for Text-Conditioned Audio-Video Generation?](https://mlsanity.com/paper/2602.01623/) (2026)
- [HumanScore: Benchmarking Human Motions in Generated Videos](https://mlsanity.com/paper/2604.20157/) (2026)
- [The Lipreading Gap: Do VSR Models Perceive Visual Speech Like Human Lipreaders?](https://mlsanity.com/paper/2606.07435/) (2026)
- [VideoVerse: Does Your T2V Generator Have World Model Capability to Synthesize Videos?](https://mlsanity.com/paper/2510.08398/) (2025)
- [Are Multimodal LLMs Ready for Surveillance? A Reality Check on Zero-Shot Anomaly Detection in the Wild](https://mlsanity.com/paper/2603.04727/) (2026)
- [How Far Are Video Models from True Multimodal Reasoning?](https://mlsanity.com/paper/2604.19193/) (2026)
- [Risk-Aware Reranking for Agentic Tool Retrieval](https://mlsanity.com/paper/2608.22751/) (2026)
- [Trimming the Long-Tail of Visual World Modeling Evaluation](https://mlsanity.com/paper/2606.24256/) (2026)

## Abstract (arXiv)

Video generation assessment is essential for ensuring that generative models produce visually realistic, high-quality videos while aligning with human expectations. Current video generation benchmarks fall into two main categories: traditional benchmarks, which use metrics and embeddings to evaluate generated video quality across multiple dimensions but often lack alignment with human judgments; and large language model (LLM)-based benchmarks, though capable of human-like reasoning, are constrained by a limited understanding of video quality metrics and cross-modal consistency. To address these challenges and establish a benchmark that better aligns with human preferences, this paper introduces Video-Bench, a comprehensive benchmark featuring a rich prompt suite and extensive evaluation dimensions. This benchmark represents the first attempt to systematically leverage MLLMs across all dimensions relevant to video generation assessment in generative models. By incorporating few-shot scoring and chain-of-query techniques, Video-Bench provides a structured, scalable approach to generated video evaluation. Experiments on advanced models including Sora demonstrate that Video-Bench achieves superior alignment with human preferences across all dimensions. Moreover, in instances where our framework's assessments diverge from human evaluations, it consistently offers more objective and accurate insights, suggesting an even greater potential advantage over traditional human judgment.