# Are Synthetic Videos Useful? A Benchmark for Retrieval-Centric Evaluation of Synthetic Videos

arXiv: https://arxiv.org/abs/2507.02316 · HTML: https://mlsanity.com/paper/2507.02316/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 40 references that are in the map.
- First posted on arXiv 3 Jul 2025.

## Builds on (most cited first, 24 of 24)

- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [VBench: Comprehensive Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2311.17982/) (2023)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [Towards Accurate Generative Models of Video: A New Metric & Challenges](https://mlsanity.com/paper/1812.01717/) (2018)
- [CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers](https://mlsanity.com/paper/2205.15868/) (2022)
- [WorldSimBench: Towards Video Generation Models as World Simulators](https://mlsanity.com/paper/2410.18072/) (2024)
- [Cosmos World Foundation Model Platform for Physical AI](https://mlsanity.com/paper/2501.03575/) (2025)
- [Playing for Data: Ground Truth from Computer Games](https://mlsanity.com/paper/1608.02192/) (2016)
- [CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval](https://mlsanity.com/paper/2104.08860/) (2021)
- [TEMOS: Generating diverse human motions from textual descriptions](https://mlsanity.com/paper/2204.14109/) (2022)
- [Phenaki: Variable Length Video Generation From Open Domain Textual Description](https://mlsanity.com/paper/2210.02399/) (2022)
- [Learning from Synthetic Humans](https://mlsanity.com/paper/1701.01370/) (2017)
- [EvalCrafter: Benchmarking and Evaluating Large Video Generation Models](https://mlsanity.com/paper/2310.11440/) (2023)
- [InternVideo2: Scaling Foundation Models for Multimodal Video Understanding](https://mlsanity.com/paper/2403.15377/) (2024)
- [A Joint Sequence Fusion Model for Video Question Answering and Retrieval](https://mlsanity.com/paper/1808.02559/) (2018)
- [X-Pool: Cross-Modal Language-Video Attention for Text-Video Retrieval](https://mlsanity.com/paper/2203.15086/) (2022)
- [TS2-Net: Token Shift and Selection Transformer for Text-Video Retrieval](https://mlsanity.com/paper/2207.07852/) (2022)
- [DiffusionRet: Generative Text-Video Retrieval with Diffusion Model](https://mlsanity.com/paper/2303.09867/) (2023)
- [Prompt Switch: Efficient CLIP Adaptation for Text-Video Retrieval](https://mlsanity.com/paper/2308.07648/) (2023)
- [Continual Text-to-Video Retrieval with Frame Fusion and Task-Aware Routing](https://mlsanity.com/paper/2503.10111/) (2025)
- [MV-Adapter: Multimodal Video Transfer Learning for Video Text Retrieval](https://mlsanity.com/paper/2301.07868/) (2023)
- [Binary Subspace Coding for Query-by-Image Video Retrieval](https://mlsanity.com/paper/1612.01657/) (2016)

## Built on by (most cited first, 5 of 5)

- [World Simulation with Video Foundation Models for Physical AI](https://mlsanity.com/paper/2511.00062/) (2025)
- [Quantifying and Narrowing the Unknown: Interactive Text-to-Video Retrieval via Uncertainty Minimization](https://mlsanity.com/paper/2507.15504/) (2025)
- [Distributed Zero-Shot Learning for Visual Recognition](https://mlsanity.com/paper/2511.08170/) (2025)
- [Accelerating Multimodal Large Language Models with Prior-Corrected Token Reduction](https://mlsanity.com/paper/2606.24156/) (2026)
- [Spectral Evolution-Guided Token Pruning in Multimodal Large Language Models](https://mlsanity.com/paper/2606.24165/) (2026)

## Abstract (arXiv)

Text-to-video (T2V) synthesis has advanced rapidly, yet current evaluation metrics primarily capture visual quality and temporal consistency, offering limited insight into how synthetic videos perform in downstream tasks such as text-to-video retrieval (TVR). In this work, we introduce SynTVA, a new dataset and benchmark designed to evaluate the utility of synthetic videos for building retrieval models. Based on 800 diverse user queries derived from MSRVTT training split, we generate synthetic videos using state-of-the-art T2V models and annotate each video-text pair along four key semantic alignment dimensions: Object & Scene, Action, Attribute, and Prompt Fidelity. Our evaluation framework correlates general video quality assessment (VQA) metrics with these alignment scores, and examines their predictive power for downstream TVR performance. To explore pathways of scaling up, we further develop an Auto-Evaluator to estimate alignment quality from existing metrics. Beyond benchmarking, our results show that SynTVA is a valuable asset for dataset augmentation, enabling the selection of high-utility synthetic samples that measurably improve TVR outcomes. Project page and dataset can be found at https://jasoncodemaker.github.io/SynTVA/.