# From Sora What We Can See: A Survey of Text-to-Video Generation

arXiv: https://arxiv.org/abs/2405.10674 · HTML: https://mlsanity.com/paper/2405.10674/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 122 of its 153 references that are in the map.
- First posted on arXiv 17 May 2024.

## Builds on (most cited first, 30 of 122)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework](https://mlsanity.com/paper/2308.00352/) (2023)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation](https://mlsanity.com/paper/1606.06650/) (2016)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning](https://mlsanity.com/paper/2307.04725/) (2023)
- [Towards Accurate Generative Models of Video: A New Metric & Challenges](https://mlsanity.com/paper/1812.01717/) (2018)
- [Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval](https://mlsanity.com/paper/2104.00650/) (2021)
- [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017)
- [CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers](https://mlsanity.com/paper/2205.15868/) (2022)
- [Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation](https://mlsanity.com/paper/2212.11565/) (2022)

## Built on by (most cited first, 27 of 27)

- [VMBench: A Benchmark for Perception-Aligned Video Motion Generation](https://mlsanity.com/paper/2503.10076/) (2025)
- [The Dawn of Video Generation: Preliminary Explorations with SORA-like Models](https://mlsanity.com/paper/2410.05227/) (2024)
- [Exploring the Evolution of Physics Cognition in Video Generation: A Survey](https://mlsanity.com/paper/2503.21765/) (2025)
- [HumanDreamer: Generating Controllable Human-Motion Videos via Decoupled Generation](https://mlsanity.com/paper/2503.24026/) (2025)
- [A Survey of Sustainability in Large Language Models: Applications, Economics, and Challenges](https://mlsanity.com/paper/2412.04782/) (2024)
- [T2VShield: Model-Agnostic Jailbreak Defense for Text-to-Video Models](https://mlsanity.com/paper/2504.15512/) (2025)
- [Simulating the Visual World with Artificial Intelligence: A Roadmap](https://mlsanity.com/paper/2511.08585/) (2025)
- [Benchmarking Multi-dimensional AIGC Video Quality Assessment: A Dataset and Unified Model](https://mlsanity.com/paper/2407.21408/) (2024)
- [DreamCinema: Cinematic Transfer with Free Camera and 3D Character](https://mlsanity.com/paper/2408.12601/) (2024)
- [Video Is Worth a Thousand Images: Exploring the Latest Trends in Long Video Generation](https://mlsanity.com/paper/2412.18688/) (2024)
- [Seeing World Dynamics in a Nutshell](https://mlsanity.com/paper/2502.03465/) (2025)
- [QuantSparse: Comprehensively Compressing Video Diffusion Transformer with Model Quantization and Attention Sparsification](https://mlsanity.com/paper/2509.23681/) (2025)
- [VDC-Agent: When Video Detailed Captioners Evolve Themselves via Agentic Self-Reflection](https://mlsanity.com/paper/2511.19436/) (2025)
- [VidLeaks: Membership Inference Attacks Against Text-to-Video Models](https://mlsanity.com/paper/2601.11210/) (2026)
- [A Survey: Spatiotemporal Consistency in Video Generation](https://mlsanity.com/paper/2502.17863/) (2025)
- [Bridging Text and Video Generation: A Survey](https://mlsanity.com/paper/2510.04999/) (2025)
- [SSG-Dit: A Spatial Signal Guided Framework for Controllable Video Generation](https://mlsanity.com/paper/2508.17062/) (2025)
- [SeqBench: Benchmarking Sequential Narrative Generation in Text-to-Video Models](https://mlsanity.com/paper/2510.13042/) (2025)
- [PipeDiT: Accelerating Diffusion Transformers in Video Generation with Task Pipelining and Model Decoupling](https://mlsanity.com/paper/2511.12056/) (2025)
- [When Cultures Move: Measuring and Improving Multicultural Text-to-Video Generation](https://mlsanity.com/paper/2605.16716/) (2026)
- [Beyond Trial-and-Error: Agentic Optimization for Image-to-Video Adherence](https://mlsanity.com/paper/2608.12290/) (2026)
- [Scene Co-pilot: Procedural Text to Video Generation with Human in the Loop](https://mlsanity.com/paper/2411.18644/) (2024)
- [VACT: A Video Automatic Causal Testing System and a Benchmark](https://mlsanity.com/paper/2503.06163/) (2025)
- [Speaking images. A novel framework for the automated self-description of artworks](https://mlsanity.com/paper/2506.05368/) (2025)
- [VC-Bench: Pioneering the Video Connecting Benchmark with a Dataset and Evaluation Metrics](https://mlsanity.com/paper/2601.19236/) (2026)
- [TDMM-LM: Bridging Facial Understanding and Animation via Language Models](https://mlsanity.com/paper/2603.16936/) (2026)
- [LAION-BVD: A 10-Million-Hour Open Video Dataset for Multimodal Pre-training](https://mlsanity.com/paper/2608.24845/) (2026)

## Abstract (arXiv)

With impressive achievements made, artificial intelligence is on the path forward to artificial general intelligence. Sora, developed by OpenAI, which is capable of minute-level world-simulative abilities can be considered as a milestone on this developmental path. However, despite its notable successes, Sora still encounters various obstacles that need to be resolved. In this survey, we embark from the perspective of disassembling Sora in text-to-video generation, and conducting a comprehensive review of literature, trying to answer the question, textitFrom Sora What We Can See. Specifically, after basic preliminaries regarding the general algorithms are introduced, the literature is categorized from three mutually perpendicular dimensions: evolutionary generators, excellent pursuit, and realistic panorama. Subsequently, the widely used datasets and metrics are organized in detail. Last but more importantly, we identify several challenges and open problems in this domain and propose potential future directions for research and development.