mlsanityData to 7 Oct 2026

EvalCrafter: Benchmarking and Evaluating Large Video Generation Models

arXiv 2310.11440 ↗First posted 17 Oct 2023cs.CVcited by 332

Not on the map. Why?

Cited by 70 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The vision and language generative models have been overgrown in recent years. For video generation, various open-sourced models and public-available services have been developed to generate high-quality videos. However, these methods often use a few metrics, e.g., FVD or IS, to evaluate the performance. We argue that it is hard to judge the large conditional generative models from the simple metrics since these models are often trained on very large datasets with multi-aspect abilities. Thus, we propose a novel framework and pipeline for exhaustively evaluating the performance of the generated videos. Our approach involves generating a diverse and comprehensive list of 700 prompts for text-to-video generation, which is based on an analysis of real-world user data and generated with the assistance of a large language model. Then, we evaluate the state-of-the-art video generative models on our carefully designed benchmark, in terms of visual qualities, content qualities, motion qualities, and text-video alignment with 17 well-selected objective metrics. To obtain the final leaderboard of the models, we further fit a series of coefficients to align the objective metrics to the users' opinions. Based on the proposed human alignment method, our final score shows a higher correlation than simply averaging the metrics, showing the effectiveness of the proposed evaluation method.

Key facts. Cited by 332 papers in the map (data through 7 Oct 2026). Builds on 53 of its 73 references that are in the map. First posted on arXiv 17 Oct 2023.

Builds on

53 of 73 references.

References · 53

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  9. 2023Segment Anythingcited by 8,158
  10. 2020End-to-End Object Detection with Transformerscited by 7,384
  11. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  12. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
38 more
  1. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  2. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  3. 2014Going Deeper with Convolutionscited by 4,231
  4. 2016Improved Techniques for Training GANscited by 2,938
  5. 2023A Survey of Large Language Modelscited by 2,537
  6. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,385
  7. 2022Video Diffusion Modelscited by 2,280
  8. 2017The Kinetics Human Action Video Datasetcited by 1,894
  9. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  10. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  11. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  12. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
  13. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  14. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
  15. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  16. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  17. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  18. 2023ModelScope Text-to-Video Technical Reportcited by 651
  19. 2023Structure and Content-Guided Video Synthesis with Diffusion Modelscited by 589
  20. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 516
  21. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Modelscited by 463
  22. 2023FateZero: Fusing Attentions for Zero-shot Text-based Video Editingcited by 462
  23. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 416
  24. 2022Latent Video Diffusion Models for High-Fidelity Long Video Generationcited by 385
  25. 2022SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animationcited by 380
  26. 2023LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Modelscited by 349
  27. 2022Exploring Video Quality Assessment on User Generated Contents from Aesthetic and Technical Perspectivescited by 314
  28. 2021StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2cited by 300
  29. 2023Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generationcited by 300
  30. 2018Learning Blind Video Temporal Consistencycited by 247
  31. 2023A Survey on Video Diffusion Modelscited by 203
  32. 2023Segment and Track Anythingcited by 193
  33. 2022Generating Videos with Dynamics-aware Implicit Generative Adversarial Networkscited by 186
  34. 2023On the Tool Manipulation Capability of Open-source Large Language Modelscited by 122
  35. 2019ICDAR 2019 Competition on Large-scale Street View Text with Partial Labeling -- RRC-LSVTcited by 121
  36. 2023HRS-Bench: Holistic, Reliable and Scalable Benchmark for Text-to-Image Modelscited by 94
  37. 2020Blind Video Temporal Consistency via Deep Video Priorcited by 76
  38. 2020Unified Quality Assessment of In-the-Wild Videos with Mixed Datasets Trainingcited by 72

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 9; 2024: 83; 2025: 109; 2026: 131.

332 papers.

Sort:

By area:

  1. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  2. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 867
  3. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 479
  4. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 465
  5. 2024Pyramidal Flow Matching for Efficient Video Generative Modelingcited by 354
  6. 2024OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generationcited by 332
  7. 2025VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulnesscited by 289
  8. 2025Improving Video Generation with Human Feedbackcited by 267
  9. 2024VideoPhy: Evaluating Physical Commonsense for Video Generationcited by 215
  10. 2024How Far is Video Generation from World Model: A Physical Law Perspectivecited by 204
  11. 2025Unified Reward Model for Multimodal Understanding and Generationcited by 194
  12. 2024T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generationcited by 177
38 more
  1. 2023FreeNoise: Tuning-Free Longer Video Diffusion via Noise Reschedulingcited by 173
  2. 2024Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generationcited by 167
  3. 2024SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Accelerationcited by 158
  4. 2024Identity-Preserving Text-to-Video Generation by Frequency Decompositioncited by 155
  5. 2024GenAD: Generalized Predictive Model for Autonomous Drivingcited by 150
  6. 2024MiraData: A Large-Scale Video Dataset with Long Durations and Structured Captionscited by 144
  7. 2025WorldScore: A Unified Evaluation Benchmark for World Generationcited by 143
  8. 2025VideoPhy-2: A Challenging Action-Centric Physical Commonsense Evaluation in Video Generationcited by 141
  9. 2024GenAI-Bench: Evaluating and Improving Compositional Text-to-Visual Generationcited by 130
  10. 2024VidProM: A Million-scale Real Prompt-Gallery Dataset for Text-to-Video Diffusion Modelscited by 124
  11. 2025WorldModelBench: Judging Video Generation Models As World Modelscited by 116
  12. 2024SageAttention2: Efficient Attention with Thorough Outlier Smoothing and Per-thread INT4 Quantizationcited by 113
  13. 2025SpargeAttention: Accurate and Training-free Sparse Attention Accelerating Any Model Inferencecited by 113
  14. 2024Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyondcited by 111
  15. 2024ViDiT-Q: Efficient and Accurate Quantization of Diffusion Transformers for Image and Video Generationcited by 110
  16. 2024Snap Video: Scaled Spatiotemporal Transformers for Text-to-Video Synthesiscited by 105
  17. 2024VideoDPO: Omni-Preference Alignment for Video Diffusion Generationcited by 102
  18. 2024Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenizationcited by 98
  19. 2024DeMamba: AI-Generated Video Detection on Million-Scale GenVideo Benchmarkcited by 85
  20. 2024T2V-Turbo: Breaking the Quality Bottleneck of Video Consistency Model with Mixed Reward Feedbackcited by 84
  21. 2024GenAI Arena: An Open Evaluation Platform for Generative Modelscited by 81
  22. 2024ChronoMagic-Bench: A Benchmark for Metamorphic Evaluation of Text-to-Time-lapse Video Generationcited by 74
  23. 2024DiTCtrl: Exploring Attention Control in Multi-Modal Diffusion Transformer for Tuning-Free Multi-Prompt Longer Video Generationcited by 73
  24. 2025OpenS2V-Nexus: A Detailed Benchmark and Million-Scale Dataset for Subject-to-Video Generationcited by 73
  25. 2023I2V-Adapter: A General Image-to-Video Adapter for Diffusion Modelscited by 70
  26. 2023VideoSwap: Customized Video Subject Swapping with Interactive Semantic Point Correspondencecited by 69
  27. 2024Allegro: Open the Black Box of Commercial-Level Video Generation Modelcited by 68
  28. 2026WorldArena: A Unified Benchmark for Evaluating Perception and Functional Utility of Embodied World Modelscited by 68
  29. 2024Evaluation of Text-to-Video Generation Models: A Dynamics Perspectivecited by 66
  30. 2024T2VSafetyBench: Evaluating the Safety of Text-to-Video Generative Modelscited by 66
  31. 2025SageAttention3: Microscaling FP4 Attention for Inference and An Exploration of 8-Bit Trainingcited by 63
  32. 2023GPT4Motion: Scripting Physical Motions in Text-to-Video Generation via Blender-Oriented GPT Planningcited by 61
  33. 2025STAR: Spatial-Temporal Augmentation with Text-to-Video Models for Real-World Video Super-Resolutioncited by 56
  34. 2024FreeTraj: Tuning-Free Trajectory Control in Video Diffusion Modelscited by 55
  35. 2024IDOL: Instant Photorealistic 3D Human Creation from a Single Imagecited by 55
  36. 2025SLA: Beyond Sparsity in Diffusion Transformers via Fine-Tunable Sparse-Linear Attentioncited by 55
  37. 2024Sora as a World Model? A Complete Survey on Text-to-Video Generationcited by 53
  38. 2025Video-Bench: Human-Aligned Video Generation Benchmarkcited by 53

50 most cited of 332 All 332 (JSON)