mlsanityData to 14 Oct 2026

DirecT2V: Large Language Models are Frame-Level Directors for Zero-Shot Text-to-Video Generation

arXiv 2305.14330 ↗First posted 23 May 2023cs.CV cs.AI cs.CLcited by 49

Not on the map. Why?

Cited by 23 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In the paradigm of AI-generated content (AIGC), there has been increasing attention to transferring knowledge from pre-trained text-to-image (T2I) models to text-to-video (T2V) generation. Despite their effectiveness, these frameworks face challenges in maintaining consistent narratives and handling shifts in scene composition or object placement from a single abstract user prompt. Exploring the ability of large language models (LLMs) to generate time-dependent, frame-by-frame prompts, this paper introduces a new framework, dubbed DirecT2V. DirecT2V leverages instruction-tuned LLMs as directors, enabling the inclusion of time-varying content and facilitating consistent video generation. To maintain temporal consistency and prevent mapping the value to a different object, we equip a diffusion model with a novel value mapping method and dual-softmax filtering, which do not require any additional training. The experimental results validate the effectiveness of our framework in producing visually coherent and storyful videos from abstract user prompts, successfully addressing the challenges of zero-shot video generation.

Key facts. Cited by 49 papers in the map (data through 14 Oct 2026). Builds on 61 of its 95 references that are in the map. First posted on arXiv 23 May 2023.

Builds on

61 of 95 references.

References · 61

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  5. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  7. 2022Training language models to follow instructions with human feedbackcited by 15,971
  8. 2020Denoising Diffusion Implicit Modelscited by 9,781
  9. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  10. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  11. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  12. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
38 more
  1. 2022Classifier-Free Diffusion Guidancecited by 5,847
  2. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  3. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  4. 2017Deep reinforcement learning from human preferencescited by 3,912
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  6. 2021Program Synthesis with Large Language Modelscited by 3,637
  7. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  8. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,045
  9. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
  10. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  11. 2020Learning to summarize from human feedbackcited by 2,588
  12. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
  13. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,415
  14. 2022Video Diffusion Modelscited by 2,305
  15. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  16. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
  17. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  18. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,476
  19. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,290
  20. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,163
  21. 2023PaLM 2 Technical Reportcited by 1,162
  22. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,045
  23. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 985
  24. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 961
  25. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 781
  26. 2022Pseudo Numerical Methods for Diffusion Models on Manifoldscited by 700
  27. 2023Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creationcited by 687
  28. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generatorscited by 670
  29. 2022Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generationcited by 597
  30. 2023Structure and Content-Guided Video Synthesis with Diffusion Modelscited by 590
  31. 2022Latent-NeRF for Shape-Guided Generation of 3D Shapes and Texturescited by 506
  32. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 484
  33. 2023FateZero: Fusing Attentions for Zero-shot Text-based Video Editingcited by 466
  34. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Modelscited by 465
  35. 2023VideoFusion: Decomposed Diffusion Models for High-Quality Video Generationcited by 331
  36. 2022CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformerscited by 307
  37. 2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
  38. 2022Pre-Trained Language Models for Interactive Decision-Makingcited by 221

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 13; 2024: 15; 2025: 11; 2026: 10.

49 papers.

Sort:

By area:

  1. 2023Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioningcited by 255
  2. 2024StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generationcited by 247
  3. 2023A Survey on Video Diffusion Modelscited by 206
  4. 2023DreamVideo: Composing Your Dream Videos with Customized Subject and Motioncited by 188
  5. 2026Cosmos 3: Omnimodal World Models for Physical AIcited by 165
  6. 2024Ca2-VDM: Efficient Autoregressive Video Diffusion Model with Causal Generation and Cache Sharingcited by 74
  7. 2023GPT4Motion: Scripting Physical Motions in Text-to-Video Generation via Blender-Oriented GPT Planningcited by 61
  8. 2023LAMP: Learn A Motion Pattern for Few-Shot-Based Video Generationcited by 59
  9. 2024Smoothed Energy Guidance: Guiding Diffusion Models with Reduced Energy Curvature of Attentioncited by 59
  10. 2024Sora as a World Model? A Complete Survey on Text-to-Video Generationcited by 53
  11. 2023Hierarchical Spatio-temporal Decoupling for Text-to-Video Generationcited by 50
  12. 2023MEVG: Multi-event Video Generation with Text-to-Video Modelscited by 40
37 more
  1. 2024ViD-GPT: Introducing GPT-style Autoregressive Generation in Video Diffusion Modelscited by 36
  2. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  3. 2023MicroCinema: A Divide-and-Conquer Approach for Text-to-Video Generationcited by 26
  4. 2023Plan, Posture and Go: Towards Open-World Text-to-Motion Generationcited by 26
  5. 2024TALC: Time-Aligned Captions for Multi-Scene Text-to-Video Generationcited by 25
  6. 2023FlowZero: Zero-Shot Text-to-Video Synthesis with LLM-Driven Dynamic Scene Syntaxcited by 19
  7. 2024Compositional 3D-aware Video Generation with LLM Directorcited by 18
  8. 2026PackForcing: Short Video Training Suffices for Long Video Sampling and Long Context Inferencecited by 17
  9. 2024AesopAgent: Agent-driven Evolutionary System on Story-to-Video Productioncited by 13
  10. 20253D Scene Prompting for Scene-Consistent Camera-Controllable Video Generationcited by 13
  11. 2025MotionAgent: Fine-grained Controllable Video Generation via Motion Field Agentcited by 11
  12. 2025Spatial Policy: Guiding Visuomotor Robotic Manipulation with Spatial-Aware Modeling and Reasoningcited by 10
  13. 2024Multi-modal Generative AI: Multi-modal LLMs, Diffusions, and the Unificationcited by 8
  14. 2023MotionZero:Exploiting Motion Priors for Zero-shot Text-to-Video Generationcited by 7
  15. 2024Video Is Worth a Thousand Images: Exploring the Latest Trends in Long Video Generationcited by 7
  16. 2025LoCoT2V-Bench: Benchmarking Long-Form and Complex Text-to-Video Generationcited by 7
  17. 2026Evolution of Video Generative Foundationscited by 7
  18. 2026PhysVid: Physics Aware Local Conditioning for Generative Video Modelscited by 6
  19. 2026ReCA: Multi-Shot Long Video Extrapolation via Recursive Context Allocationcited by 6
  20. 2025GenTune: Toward Traceable Prompts to Improve Controllability of Image Refinement in Environment Designcited by 5
  21. 2023POS: A Prompts Optimization Suite for Augmenting Text-to-Video Generationcited by 4
  22. 2024Perturb-and-Revise: Flexible 3D Editing with Generative Trajectoriescited by 3
  23. 2024ModelGrow: Continual Text-to-Video Pre-training with Model Expansion and Language Understanding Enhancementcited by 3
  24. 2025RISE-T2V: Rephrasing and Injecting Semantics with LLM for Expansive Text-to-Video Generationcited by 3
  25. 2025ACT-R: Adaptive Camera Trajectories for Single View 3D Reconstructioncited by 2
  26. 2025MultiCOIN: Multi-Modal COntrollable Video INbetweeningcited by 2
  27. 2025Lights, Camera, Consistency: A Multistage Pipeline for Character-Stable AI Video Storiescited by 2
  28. 2026COMIC: Agentic Sketch Comedy Generationcited by 2
  29. 2023Multi-View Unsupervised Image Generation with Cross Attention Guidancecited by 1
  30. 2024Intelligent Director: An Automatic Framework for Dynamic Visual Composition using ChatGPTcited by 1
  31. 2024AutoDirector: Online Auto-scheduling Agents for Multi-sensory Compositioncited by 1
  32. 2026Training-free Motion Factorization for Compositional Video Generationcited by 1
  33. 2025Exploring Conditions for Diffusion models in Robotic Controlcited by 0
  34. 2025DiffusionBrowser: Interactive Diffusion Previews via Multi-Branch Decoderscited by 0
  35. 2026MLLM-Guided Semantic Correction for Text-to-Video Generationcited by 0
  36. 2026No Corners Cut: State-Grounded Transitions for Mid-Stream Prompt Switches in Video Generationcited by 0
  37. 2026Any Time, Any Space: A World-State-Centric Framework for Agentic Video Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Anchored Video Generation: Decoupling Scene Construction and Temporal Synthesis in Text-to-Video Diffusion Models
  2. 2024T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generation
  3. 2025CINEMA: Coherent Multi-Subject Video Generation via MLLM-Based Guidance
  4. 2023Free-Bloom: Zero-Shot Text-to-Video Generator with LLM Director and LDM Animator
  5. 2025TV2TV: A Unified Framework for Interleaved Language and Video Generation
  6. 2025The Best of Both Worlds: Integrating Language Models and Diffusion Models for Video Generation
  7. 2025AI Powered High Quality Text to Video Generation with Enhanced Temporal Consistency
  8. 2023FusionFrames: Efficient Architectural Aspects for Text-to-Video Generation Pipeline