mlsanityData to 14 Oct 2026

Free-Bloom: Zero-Shot Text-to-Video Generator with LLM Director and LDM Animator

arXiv 2309.14494 ↗First posted 25 Sep 2023cs.CVcited by 74

Not on the map. Why?

Cited by 26 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Text-to-video is a rapidly growing research area that aims to generate a semantic, identical, and temporal coherence sequence of frames that accurately align with the input text prompt. This study focuses on zero-shot text-to-video generation considering the data- and cost-efficient. To generate a semantic-coherent video, exhibiting a rich portrayal of temporal semantics such as the whole process of flower blooming rather than a set of "moving images", we propose a novel Free-Bloom pipeline that harnesses large language models (LLMs) as the director to generate a semantic-coherence prompt sequence, while pre-trained latent diffusion models (LDMs) as the animator to generate the high fidelity frames. Furthermore, to ensure temporal and identical coherence while maintaining semantic coherence, we propose a series of annotative modifications to adapting LDMs in the reverse process, including joint noise sampling, step-aware attention shift, and dual-path interpolation. Without any video data and training requirements, Free-Bloom generates vivid and high-quality videos, awe-inspiring in generating complex scenes with semantic meaningful frame sequences. In addition, Free-Bloom is naturally compatible with LDMs-based extensions.

Key facts. Cited by 74 papers in the map (data through 14 Oct 2026). Builds on 51 of its 60 references that are in the map. First posted on arXiv 25 Sep 2023.

Builds on

51 of 60 references.

References · 51

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  7. 2022Training language models to follow instructions with human feedbackcited by 15,971
  8. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  9. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  10. 2020Denoising Diffusion Implicit Modelscited by 9,781
  11. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  12. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
38 more
  1. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  2. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  3. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  4. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  5. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
  6. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  7. 2019Language Models as Knowledge Bases?cited by 1,936
  8. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
  9. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  10. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  11. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,476
  12. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,351
  13. 2018ESRGAN: Enhanced Super-Resolution Generative Adversarial Networkscited by 1,254
  14. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Modelscited by 1,179
  15. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  16. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 985
  17. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 961
  18. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 719
  19. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generatorscited by 670
  20. 2017Video Enhancement with Task-Oriented Flowcited by 619
  21. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 484
  22. 2023FateZero: Fusing Attentions for Zero-shot Text-based Video Editingcited by 466
  23. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Modelscited by 465
  24. 2023VideoFusion: Decomposed Diffusion Models for High-Quality Video Generationcited by 331
  25. 2023Pix2Video: Video Editing using Image Diffusioncited by 320
  26. 2023Video-P2P: Video Editing with Cross-attention Controlcited by 314
  27. 2021GODIVA: Generating Open-DomaIn Videos from nAtural Descriptionscited by 256
  28. 2021Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptionscited by 226
  29. 2021Design Guidelines for Prompt Engineering Text-to-Image Generative Modelscited by 218
  30. 2023AMT: All-Pairs Multi-Field Transforms for Efficient Frame Interpolationcited by 180
  31. 2023Zero-Shot Video Editing Using Off-The-Shelf Image Diffusion Modelscited by 137
  32. 2023Latent-Shift: Latent Diffusion with Temporal Shift for Efficient Text-to-Video Generationcited by 137
  33. 2023Edit-A-Video: Single Video Editing with Object-Aware Consistencycited by 67
  34. 2022Investigating Prompt Engineering in Diffusion Modelscited by 58
  35. 2022A Taxonomy of Prompt Modifiers for Text-To-Image Generationcited by 55
  36. 2023CoTDet: Affordance Knowledge Prompting for Task Driven Object Detectioncited by 36
  37. 2023LoGoPrompt: Synthetic Text Images Can Be Good Visual Prompts for Vision-Language Modelscited by 29
  38. 2023Zero-shot Generation of Coherent Storybook from Plain Text Story using Diffusion Modelscited by 26

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 14; 2024: 25; 2025: 27; 2026: 8.

74 papers.

Sort:

By area:

  1. 2023A Survey on Video Diffusion Modelscited by 206
  2. 2023DreamVideo: Composing Your Dream Videos with Customized Subject and Motioncited by 188
  3. 2023WonderJourney: Going from Anywhere to Everywherecited by 158
  4. 2023DDCoT: Duty-Distinct Chain-of-Thought Prompting for Multimodal Reasoning in Language Modelscited by 128
  5. 2024Large Language Models for Robotics: Opportunities, Challenges, and Perspectivescited by 89
  6. 2023PEEKABOO: Interactive Video Generation via Masked-Diffusioncited by 76
  7. 2024Vlogger: Make Your Dream A Vlogcited by 70
  8. 2024Image-of-Thought Prompting for Visual Reasoning Refinement in Multimodal Large Language Modelscited by 65
  9. 2023GPT4Motion: Scripting Physical Motions in Text-to-Video Generation via Blender-Oriented GPT Planningcited by 61
  10. 2025Hunyuan-GameCraft-2: Instruction-following Interactive Game World Modelcited by 61
  11. 2023LAMP: Learn A Motion Pattern for Few-Shot-Based Video Generationcited by 59
  12. 2024Smoothed Energy Guidance: Guiding Diffusion Models with Reduced Energy Curvature of Attentioncited by 59
38 more
  1. 2024Sora as a World Model? A Complete Survey on Text-to-Video Generationcited by 53
  2. 2023A Recipe for Scaling up Text-to-Video Generation with Text-free Videoscited by 51
  3. 2024MagicTime: Time-lapse Video Generation Models as Metamorphic Simulatorscited by 51
  4. 2023Hierarchical Spatio-temporal Decoupling for Text-to-Video Generationcited by 50
  5. 2024UniEdit: A Unified Tuning-Free Framework for Video Motion and Appearance Editingcited by 48
  6. 2026Infinite Worlds with Versatile Interactionscited by 48
  7. 2024TC-Bench: Benchmarking Temporal Compositionality in Text-to-Video and Image-to-Video Generationcited by 43
  8. 2023MEVG: Multi-event Video Generation with Text-to-Video Modelscited by 40
  9. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 36
  10. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  11. 2024TALC: Time-Aligned Captions for Multi-Scene Text-to-Video Generationcited by 25
  12. 2024AID: Adapting Image2Video Diffusion Models for Instruction-guided Video Predictioncited by 21
  13. 2024DisenStudio: Customized Multi-subject Text-to-Video Generation with Disentangled Spatial Controlcited by 20
  14. 2023FlowZero: Zero-Shot Text-to-Video Synthesis with LLM-Driven Dynamic Scene Syntaxcited by 19
  15. 2024Compositional 3D-aware Video Generation with LLM Directorcited by 18
  16. 2025FreeLong++: Training-Free Long Video Generation via Multi-band SpectralFusioncited by 17
  17. 2023VideoDreamer: Customized Multi-Subject Text-to-Video Generation with Disen-Mix Finetuning on Language-Video Foundation Modelscited by 16
  18. 2023EVP: Enhanced Visual Perception using Inverse Multi-Attentive Feature Refinement and Regularized Image-Text Alignmentcited by 14
  19. 2025Rethinking Query-based Transformer for Continual Image Segmentationcited by 14
  20. 2024Adaptive Pre-training Data Detection for Large Language Models via Surprising Tokenscited by 11
  21. 2025MotionAgent: Fine-grained Controllable Video Generation via Motion Field Agentcited by 11
  22. 2024The devil is in the object boundary: towards annotation-free instance segmentation using Foundation Modelscited by 10
  23. 2025Why LVLMs Are More Prone to Hallucinations in Longer Responses: The Role of Contextcited by 10
  24. 2024MDSGen: Fast and Efficient Masked Diffusion Temporal-Aware Transformers for Open-Domain Sound Generationcited by 9
  25. 2024Optical Flow Representation Alignment Mamba Diffusion Model for Medical Video Generationcited by 8
  26. 2023MotionZero:Exploiting Motion Priors for Zero-shot Text-to-Video Generationcited by 7
  27. 2024Video Is Worth a Thousand Images: Exploring the Latest Trends in Long Video Generationcited by 7
  28. 2026Evolution of Video Generative Foundationscited by 7
  29. 2024Llama Learns to Direct: DirectorLLM for Human-Centric Video Generationcited by 6
  30. 2025Intervene-All-Paths: Unified Mitigation of LVLM Hallucinations across Alignment Formatscited by 6
  31. 2024Rethinking Human Evaluation Protocol for Text-to-Video Models: Enhancing Reliability,Reproducibility, and Practicalitycited by 4
  32. 2025Separate Motion from Appearance: Customizing Motion via Customizing Text-to-Video Diffusion Modelscited by 4
  33. 2025PersonaVlog: Personalized Multimodal Vlog Generation with Multi-Agent Collaboration and Iterative Self-Correctioncited by 4
  34. 2025Sim-DETR: Unlock DETR for Temporal Sentence Groundingcited by 4
  35. 2025Closed-Loop Transfer for Weakly-supervised Affordance Groundingcited by 4
  36. 2024Text Prompting for Multi-Concept Video Customization by Autoregressive Generationcited by 3
  37. 2024Perturb-and-Revise: Flexible 3D Editing with Generative Trajectoriescited by 3
  38. 2025No More Sibling Rivalry: Debiasing Human-Object Interaction Detectioncited by 3

50 most cited of 74 All 74 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Gen-L-Video: Multi-Text to Long Video Generation via Temporal Co-Denoising
  2. 2023ControlVideo: Training-free Controllable Text-to-Video Generation
  3. 2023DirecT2V: Large Language Models are Frame-Level Directors for Zero-Shot Text-to-Video Generation
  4. 2026UniVid: Pyramid Diffusion Model for High Quality Video Generation
  5. 2024I4VGen: Image as Free Stepping Stone for Text-to-Video Generation
  6. 2025Anchored Video Generation: Decoupling Scene Construction and Temporal Synthesis in Text-to-Video Diffusion Models
  7. 2025Training-free Guidance in Text-to-Video Generation via Multimodal Planning and Structured Noise Initialization
  8. 2025The Best of Both Worlds: Integrating Language Models and Diffusion Models for Video Generation