mlsanityData to 7 Oct 2026

VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generation

arXiv 2406.15252 ↗First posted 21 Jun 2024cs.CV cs.AIcited by 190

Not on the map. Why?

Cited by 51 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The recent years have witnessed great advances in video generation. However, the development of automatic video metrics is lagging significantly behind. None of the existing metric is able to provide reliable scores over generated videos. The main barrier is the lack of large-scale human-annotated dataset. In this paper, we release VideoFeedback, the first large-scale dataset containing human-provided multi-aspect score over 37.6K synthesized videos from 11 existing video generative models. We train VideoScore (initialized from Mantis) based on VideoFeedback to enable automatic video quality assessment. Experiments show that the Spearman correlation between VideoScore and humans can reach 77.1 on VideoFeedback-test, beating the prior best metrics by about 50 points. Further result on other held-out EvalCrafter, GenAI-Bench, and VBench show that VideoScore has consistently much higher correlation with human judges than other metrics. Due to these results, we believe VideoScore can serve as a great proxy for human raters to (1) rate different video models to track progress (2) simulate fine-grained human feedback in Reinforcement Learning with Human Feedback (RLHF) to improve current video generation models.

Key facts. Cited by 190 papers in the map (data through 7 Oct 2026). Builds on 47 of its 72 references that are in the map. First posted on arXiv 21 Jun 2024.

Builds on

47 of 72 references.

References · 47

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  6. 2016Improved Techniques for Training GANscited by 2,938
  7. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  8. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  9. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  10. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  11. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
  12. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
35 more
  1. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
  2. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  3. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  4. 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 767
  5. 2017Localizing Moments in Video with Natural Languagecited by 660
  6. 2023ModelScope Text-to-Video Technical Reportcited by 651
  7. 2023Structure and Content-Guided Video Synthesis with Diffusion Modelscited by 589
  8. 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Modelscited by 568
  9. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generationcited by 530
  10. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Modelscited by 463
  11. 2024Lumiere: A Space-Time Diffusion Model for Video Generationcited by 450
  12. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewardscited by 431
  13. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 389
  14. 2022Latent Video Diffusion Models for High-Fidelity Long Video Generationcited by 385
  15. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
  16. 2023LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Modelscited by 349
  17. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  18. 2022Exploring Video Quality Assessment on User Generated Contents from Aesthetic and Technical Perspectivescited by 314
  19. 2023Photorealistic Video Generation with Diffusion Modelscited by 304
  20. 2022X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrievalcited by 279
  21. 2023Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioningcited by 254
  22. 2024StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Textcited by 242
  23. 2023SEINE: Short-to-Long Video Diffusion Model for Generative Transition and Predictioncited by 231
  24. 2024VideoPhy: Evaluating Physical Commonsense for Video Generationcited by 215
  25. 2024MANTIS: Interleaved Multi-Image Instruction Tuningcited by 212
  26. 2023VIEScore: Towards Explainable Metrics for Conditional Image Synthesis Evaluationcited by 196
  27. 2022FAST-VQA: Efficient End-to-end Video Quality Assessment with Fragment Samplingcited by 179
  28. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewardscited by 145
  29. 2023Latent-Shift: Latent Diffusion with Temporal Shift for Efficient Text-to-Video Generationcited by 137
  30. 2024VidProM: A Million-scale Real Prompt-Gallery Dataset for Text-to-Video Diffusion Modelscited by 124
  31. 2023FETV: A Benchmark for Fine-Grained Evaluation of Open-Domain Text-to-Video Generationcited by 119
  32. 2024T2V-Turbo: Breaking the Quality Bottleneck of Video Consistency Model with Mixed Reward Feedbackcited by 84
  33. 2024GenAI Arena: An Open Evaluation Platform for Generative Modelscited by 81
  34. 2024Large-scale Reinforcement Learning for Diffusion Modelscited by 78
  35. 2024Towards A Better Metric for Text-to-Video Generationcited by 50

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 18; 2025: 96; 2026: 76.

190 papers.

Sort:

By area:

  1. 2025DanceGRPO: Unleashing GRPO on Visual Generationcited by 388
  2. 2025Improving Video Generation with Human Feedbackcited by 267
  3. 2025Unified Reward Model for Multimodal Understanding and Generationcited by 194
  4. 2024Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generationcited by 167
  5. 2025Do generative video models understand physical principles?cited by 151
  6. 2024VisionReward: Fine-Grained Multi-Dimensional Human Preference Learning for Image and Video Generationcited by 148
  7. 2025VideoPhy-2: A Challenging Action-Centric Physical Commonsense Evaluation in Video Generationcited by 141
  8. 2025WorldModelBench: Judging Video Generation Models As World Modelscited by 116
  9. 2025Unified Multimodal Chain-of-Thought Reward Model through Reinforcement Fine-Tuningcited by 86
  10. 2025OpenS2V-Nexus: A Detailed Benchmark and Million-Scale Dataset for Subject-to-Video Generationcited by 73
  11. 2024T2V-Turbo-v2: Enhancing Video Generation Model Post-Training through Data, Reward, and Conditional Guidance Designcited by 54
  12. 2024LiFT: Leveraging Human Feedback for Text-to-Video Model Alignmentcited by 51
38 more
  1. 2025VMBench: A Benchmark for Perception-Aligned Video Motion Generationcited by 48
  2. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challengescited by 46
  3. 2024Improving Dynamic Object Interactions in Text-to-Video Generation with AI Feedbackcited by 45
  4. 2025PhysAnimator: Physics-Guided Generative Cartoon Animationcited by 40
  5. 2025Inference-Time Text-to-Video Alignment with Diffusion Latent Beam Searchcited by 40
  6. 2024Q-Bench-Video: Benchmarking the Video Quality Understanding of LMMscited by 39
  7. 2024Mind the Time: Temporally-Controlled Multi-Event Video Generationcited by 39
  8. 2025Video-T1: Test-Time Scaling for Video Generationcited by 39
  9. 2024OnlineVPO: Align Video Diffusion Model with Online Video-Centric Preference Optimizationcited by 38
  10. 2025Scaling Image and Video Generation via Test-Time Evolutionary Searchcited by 38
  11. 2025VideoScore2: Think before You Score in Generative Video Evaluationcited by 37
  12. 2025"PhyWorldBench": A Comprehensive Evaluation of Physical Realism in Text-to-Video Modelscited by 34
  13. 2024Sharp Analysis for KL-Regularized Contextual Bandits and RLHFcited by 32
  14. 2025VideoHallu: Evaluating and Mitigating Multi-modal Hallucinations on Synthetic Video Understandingcited by 32
  15. 2025DenseDPO: Fine-Grained Temporal Preference Optimization for Video Diffusion Modelscited by 32
  16. 20244Real-Video: Learning Generalizable Photo-Realistic 4D Video Diffusioncited by 27
  17. 2025VQ-Insight: Teaching VLMs for AI-Generated Video Quality Understanding via Progressive Visual Reinforcement Learningcited by 26
  18. 2025VISTA: A Test-Time Self-Improving Video Generation Agentcited by 26
  19. 2024DOLLAR: Few-Step Video Generation via Distillation and Latent Reward Optimizationcited by 25
  20. 2024VideoAgent: Self-Improving Video Generationcited by 24
  21. 2025Generative Physical AI in Vision: A Surveycited by 24
  22. 20253D Scene Generation: A Surveycited by 24
  23. 2024Prompt-A-Video: Prompt Your Video Diffusion Model via Preference-Aligned LLMcited by 22
  24. 2025VidCapBench: A Comprehensive Benchmark of Video Captioning for Controllable Text-to-Video Generationcited by 22
  25. 2025TheoremExplainAgent: Towards Video-based Multimodal Explanations for LLM Theorem Understandingcited by 20
  26. 2025T2AV-Compass: Towards Unified Evaluation for Text-to-Audio-Video Generationcited by 19
  27. 2025HumanAesExpert: Advancing a Multi-Modality Foundation Model for Human Image Aesthetic Assessmentcited by 17
  28. 2024VQA2: Visual Question Answering for Video Quality Assessmentcited by 15
  29. 2025InfLVG: Reinforce Inference-Time Consistent Long Video Generation with GRPOcited by 15
  30. 2025Video Generation Models Are Good Latent Reward Modelscited by 15
  31. 2025MJ-VIDEO: Fine-Grained Benchmarking and Rewarding Video Preferences in Video Generationcited by 14
  32. 2025Dual-IPO: Dual-Iterative Preference Optimization for Text-to-Video Generationcited by 12
  33. 2024SPAgent: Adaptive Task Decomposition and Model Selection for General Video Generation and Editingcited by 11
  34. 2025What Is That Talk About? A Video-to-Text Summarization Dataset for Scientific Presentationscited by 11
  35. 2025Next Token Is Enough: Realistic Image Quality and Aesthetic Scoring with Multimodal Large Language Modelcited by 11
  36. 2025Integrating Reinforcement Learning with Visual Generative Models: Foundations and Advancescited by 11
  37. 2025VR-Thinker: Boosting Video Reward Models through Thinking-with-Image Reasoningcited by 11
  38. 2025Omni-Reward: Towards Generalist Omni-Modal Reward Modeling with Free-Form Preferencescited by 11

50 most cited of 190 All 190 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Video-Bench: Human-Aligned Video Generation Benchmark
  2. 2025World Consistency Score: A Unified Metric for Video Generation Quality
  3. 2025Score2Instruct: Scaling Up Video Quality-Centric Instructions via Automated Dimension Scoring
  4. 2026VideoArgus: Agentic Rubric-Grounded Unified Evaluation for Video Generation and Editing
  5. 2026VGA-BenchV2: An Expanded Unified Benchmark and Multi-Model Framework for Evaluating Video Aesthetics and Generation Quality
  6. 2026Multi-Dimensional Quality Assessment for AI-Generated Human-Centric Videos: Dataset and Model
  7. 2026UniVBench: Towards Unified Evaluation for Video Foundation Models
  8. 2026KeyFrame-Compass: Towards Comprehensive Evaluation of Keyframe-Conditioned Video Generation