mlsanityData to 7 Oct 2026

FETV: A Benchmark for Fine-Grained Evaluation of Open-Domain Text-to-Video Generation

arXiv 2311.01813 ↗First posted 3 Nov 2023cs.CVcited by 119

Not on the map. Why?

Abstract (arXiv)

Recently, open-domain text-to-video (T2V) generation models have made remarkable progress. However, the promising results are mainly shown by the qualitative cases of generated videos, while the quantitative evaluation of T2V models still faces two critical problems. Firstly, existing studies lack fine-grained evaluation of T2V models on different categories of text prompts. Although some benchmarks have categorized the prompts, their categorization either only focuses on a single aspect or fails to consider the temporal information in video generation. Secondly, it is unclear whether the automatic evaluation metrics are consistent with human standards. To address these problems, we propose FETV, a benchmark for Fine-grained Evaluation of Text-to-Video generation. FETV is multi-aspect, categorizing the prompts based on three orthogonal aspects: the major content, the attributes to control and the prompt complexity. FETV is also temporal-aware, which introduces several temporal categories tailored for video generation. Based on FETV, we conduct comprehensive manual evaluations of four representative T2V models, revealing their pros and cons on different categories of prompts from different aspects. We also extend FETV as a testbed to evaluate the reliability of automatic T2V metrics. The multi-aspect categorization of FETV enables fine-grained analysis of the metrics' reliability in different scenarios. We find that existing automatic metrics (e.g., CLIPScore and FVD) correlate poorly with human evaluation. To address this problem, we explore several solutions to improve CLIPScore and FVD, and develop two automatic metrics that exhibit significant higher correlation with humans than existing metrics. Benchmark page: https://github.com/llyx97/FETV.

Key facts. Cited by 119 papers in the map (data through 7 Oct 2026). Builds on 17 of its 51 references that are in the map. First posted on arXiv 3 Nov 2023.

Builds on

17 of 51 references.

References · 17

  1. 2017The Kinetics Human Action Video Datasetcited by 1,894
  2. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  3. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  4. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  5. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 958
  6. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generatorscited by 666
  7. 2023ModelScope Text-to-Video Technical Reportcited by 651
  8. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  9. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
  10. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Modelscited by 463
  11. 2022Latent Video Diffusion Models for High-Fidelity Long Video Generationcited by 385
  12. 2018A Short Note about Kinetics-600cited by 331
5 more
  1. 2023VideoFusion: Decomposed Diffusion Models for High-Quality Video Generationcited by 330
  2. 2021GODIVA: Generating Open-DomaIn Videos from nAtural Descriptionscited by 254
  3. 2022Diffusion Probabilistic Modeling for Video Generationcited by 237
  4. 2023Dreamix: Video Diffusion Models are General Video Editorscited by 189
  5. 2020A Short Note on the Kinetics-700-2020 Human Action Datasetcited by 89

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 37; 2025: 41; 2026: 38.

119 papers.

Sort:

By area:

  1. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  2. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understandingcited by 399
  3. 2025VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulnesscited by 289
  4. 2024VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generationcited by 190
  5. 2024T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generationcited by 177
  6. 2024Identity-Preserving Text-to-Video Generation by Frequency Decompositioncited by 155
  7. 2025WorldScore: A Unified Evaluation Benchmark for World Generationcited by 143
  8. 2024VidProM: A Million-scale Real Prompt-Gallery Dataset for Text-to-Video Diffusion Modelscited by 124
  9. 2024Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyondcited by 111
  10. 2024VideoDPO: Omni-Preference Alignment for Video Diffusion Generationcited by 102
  11. 2024ChronoMagic-Bench: A Benchmark for Metamorphic Evaluation of Text-to-Time-lapse Video Generationcited by 74
  12. 2025OpenS2V-Nexus: A Detailed Benchmark and Million-Scale Dataset for Subject-to-Video Generationcited by 73
38 more
  1. 2024Allegro: Open the Black Box of Commercial-Level Video Generation Modelcited by 68
  2. 2024Evaluation of Text-to-Video Generation Models: A Dynamics Perspectivecited by 66
  3. 2024T2VSafetyBench: Evaluating the Safety of Text-to-Video Generative Modelscited by 66
  4. 2023VITATECS: A Diagnostic Dataset for Temporal Concept Understanding of Video-Language Modelscited by 55
  5. 2024Sora as a World Model? A Complete Survey on Text-to-Video Generationcited by 53
  6. 2025Video-Bench: Human-Aligned Video Generation Benchmarkcited by 53
  7. 2024Towards A Better Metric for Text-to-Video Generationcited by 50
  8. 2025VMBench: A Benchmark for Perception-Aligned Video Motion Generationcited by 48
  9. 2024TC-Bench: Benchmarking Temporal Compositionality in Text-to-Video and Image-to-Video Generationcited by 42
  10. 2024VBench++: Comprehensive and Versatile Benchmark Suite for Video Generative Modelscited by 42
  11. 2025Are Video Models Ready as Zero-Shot Reasoners? An Empirical Study with the MME-CoF Benchmarkcited by 41
  12. 2024Subjective-Aligned Dataset and Metric for Text-to-Video Quality Assessmentcited by 39
  13. 2025T2VPhysBench: A First-Principles Benchmark for Physical Consistency in Text-to-Video Generationcited by 35
  14. 2024GAIA: Rethinking Action Quality Assessment for AI-Generated Videoscited by 30
  15. 2025PAI-Bench: A Comprehensive Benchmark For Physical AIcited by 30
  16. 2026Rethinking Video Generation Model for the Embodied Worldcited by 30
  17. 2024Is Your World Simulator a Good Story Presenter? A Consecutive Events-Based Benchmark for Future Long Video Generationcited by 28
  18. 2024AIGCBench: Comprehensive Evaluation of Image-to-Video Content Generated by AIcited by 27
  19. 2024Detecting AI-Generated Video via Frame Consistencycited by 26
  20. 2024Quality Assessment in the Era of Large Models: A Surveycited by 26
  21. 2024AIGV-Assessor: Benchmarking and Evaluating the Perceptual Quality of Text-to-Video Generation with LMMcited by 24
  22. 2025VideoGen-Eval: Agent-based System for Video Generation Evaluationcited by 23
  23. 2024Warped Diffusion: Solving Video Inverse Problems with Image Diffusion Modelscited by 21
  24. 2024NTIRE 2024 Quality Assessment of AI-Generated Content Challengecited by 20
  25. 2025Can World Simulators Reason? Gen-ViRe: A Generative Visual Reasoning Benchmarkcited by 19
  26. 2025T2VWorldBench: A Benchmark for Evaluating World Knowledge in Text-to-Video Generationcited by 18
  27. 2024K-Sort Arena: Efficient and Reliable Benchmarking for Generative Models via K-wise Human Preferencescited by 16
  28. 2024Video Quality Assessment: A Comprehensive Surveycited by 15
  29. 2025BlobGEN-Vid: Compositional Text-to-Video Generation with Blob Video Representationscited by 15
  30. 2026MBench: A Comprehensive Benchmark on Memory Capability for Video World Modelscited by 15
  31. 2024Benchmarking Multi-dimensional AIGC Video Quality Assessment: A Dataset and Unified Modelcited by 13
  32. 2025DanceTogether! Identity-Preserving Multi-Person Interactive Video Generationcited by 12
  33. 2025RULER-Bench: Probing Rule-based Reasoning Abilities of Next-level Video Generation Models for Vision Foundation Intelligencecited by 12
  34. 2024GT23D-Bench: A Comprehensive General Text-to-3D Generation Benchmarkcited by 11
  35. 2024OccScene: Semantic Occupancy-based Cross-task Mutual Learning for 3D Scene Generationcited by 10
  36. 2025Can You Count to Nine? A Human Evaluation Benchmark for Counting Limits in Modern Text-to-Video Modelscited by 10
  37. 2025Direct Motion Models for Assessing Generated Videoscited by 10
  38. 2025LOVE: Benchmarking and Evaluating Text-to-Video Generation and Video-to-Text Interpretationcited by 10

50 most cited of 119 All 119 (JSON)