mlsanityData to 7 Oct 2026

From Sora What We Can See: A Survey of Text-to-Video Generation

arXiv 2405.10674 ↗First posted 17 May 2024cs.CV cs.AIcited by 26

Not on the map. Why?

Abstract (arXiv)

With impressive achievements made, artificial intelligence is on the path forward to artificial general intelligence. Sora, developed by OpenAI, which is capable of minute-level world-simulative abilities can be considered as a milestone on this developmental path. However, despite its notable successes, Sora still encounters various obstacles that need to be resolved. In this survey, we embark from the perspective of disassembling Sora in text-to-video generation, and conducting a comprehensive review of literature, trying to answer the question, textitFrom Sora What We Can See. Specifically, after basic preliminaries regarding the general algorithms are introduced, the literature is categorized from three mutually perpendicular dimensions: evolutionary generators, excellent pursuit, and realistic panorama. Subsequently, the widely used datasets and metrics are organized in detail. Last but more importantly, we identify several challenges and open problems in this domain and propose potential future directions for research and development.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Builds on 122 of its 153 references that are in the map. First posted on arXiv 17 May 2024.

Builds on

122 of 153 references.

References · 122

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2020Denoising Diffusion Implicit Modelscited by 9,655
  7. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  8. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
  9. 2022Scalable Diffusion Models with Transformerscited by 5,931
  10. 2022Generative Adversarial Networkscited by 5,603
  11. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  12. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
38 more
  1. 2017Neural Discrete Representation Learningcited by 4,650
  2. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  3. 2016Improved Techniques for Training GANscited by 2,938
  4. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  5. 2022Video Diffusion Modelscited by 2,280
  6. 2020Implicit Neural Representations with Periodic Activation Functionscited by 2,214
  7. 2017The Kinetics Human Action Video Datasetcited by 1,894
  8. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  9. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,761
  10. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  11. 20163D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotationcited by 1,471
  12. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  13. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
  14. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
  15. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  16. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  17. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  18. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 958
  19. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  20. 2017Dense-Captioning Events in Videoscited by 789
  21. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  22. 2022MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Modelcited by 768
  23. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  24. 2024Genie: Generative Interactive Environmentscited by 706
  25. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generatorscited by 666
  26. 2017Localizing Moments in Video with Natural Languagecited by 660
  27. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651
  28. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
  29. 2017Generative Adversarial Networks: An Overviewcited by 588
  30. 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Modelscited by 568
  31. 2024Latte: Latent Diffusion Transformer for Video Generationcited by 536
  32. 2016Hollywood in Homes: Crowdsourcing Data Collection for Activity Understandingcited by 535
  33. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generationcited by 530
  34. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 488
  35. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
  36. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 467
  37. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Modelscited by 463
  38. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priorscited by 463

50 most cited of 122 All 122 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 15; 2026: 5.

26 papers.

Sort:

By area:

  1. 2025VMBench: A Benchmark for Perception-Aligned Video Motion Generationcited by 48
  2. 2025Exploring the Evolution of Physics Cognition in Video Generation: A Surveycited by 31
  3. 2024The Dawn of Video Generation: Preliminary Explorations with SORA-like Modelscited by 30
  4. 2025HumanDreamer: Generating Controllable Human-Motion Videos via Decoupled Generationcited by 19
  5. 2024A Survey of Sustainability in Large Language Models: Applications, Economics, and Challengescited by 18
  6. 2025T2VShield: Model-Agnostic Jailbreak Defense for Text-to-Video Modelscited by 17
  7. 2025Simulating the Visual World with Artificial Intelligence: A Roadmapcited by 15
  8. 2024Benchmarking Multi-dimensional AIGC Video Quality Assessment: A Dataset and Unified Modelcited by 13
  9. 2024DreamCinema: Cinematic Transfer with Free Camera and 3D Charactercited by 9
  10. 2024Video Is Worth a Thousand Images: Exploring the Latest Trends in Long Video Generationcited by 7
  11. 2025Seeing World Dynamics in a Nutshellcited by 7
  12. 2025QuantSparse: Comprehensively Compressing Video Diffusion Transformer with Model Quantization and Attention Sparsificationcited by 5
14 more
  1. 2025VDC-Agent: When Video Detailed Captioners Evolve Themselves via Agentic Self-Reflectioncited by 4
  2. 2026VidLeaks: Membership Inference Attacks Against Text-to-Video Modelscited by 4
  3. 2025A Survey: Spatiotemporal Consistency in Video Generationcited by 3
  4. 2025Bridging Text and Video Generation: A Surveycited by 2
  5. 2025SeqBench: Benchmarking Sequential Narrative Generation in Text-to-Video Modelscited by 1
  6. 2025PipeDiT: Accelerating Diffusion Transformers in Video Generation with Task Pipelining and Model Decouplingcited by 1
  7. 2026When Cultures Move: Measuring and Improving Multicultural Text-to-Video Generationcited by 1
  8. 2026Beyond Trial-and-Error: Agentic Optimization for Image-to-Video Adherencecited by 1
  9. 2024Scene Co-pilot: Procedural Text to Video Generation with Human in the Loopcited by 0
  10. 2025VACT: A Video Automatic Causal Testing System and a Benchmarkcited by 0
  11. 2025Speaking images. A novel framework for the automated self-description of artworkscited by 0
  12. 2025SSG-Dit: A Spatial Signal Guided Framework for Controllable Video Generationcited by 0
  13. 2026VC-Bench: Pioneering the Video Connecting Benchmark with a Dataset and Evaluation Metricscited by 0
  14. 2026TDMM-LM: Bridging Facial Understanding and Animation via Language Modelscited by 0