mlsanityData to 14 Oct 2026

VideoTetris: Towards Compositional Text-to-Video Generation

arXiv 2406.04277 ↗First posted 6 Jun 2024cs.CVcited by 51

On the map: Diffusion models → Video generation

Abstract (arXiv)

Diffusion models have demonstrated great success in text-to-video (T2V) generation. However, existing methods may face challenges when handling complex (long) video generation scenarios that involve multiple objects or dynamic changes in object numbers. To address these limitations, we propose VideoTetris, a novel framework that enables compositional T2V generation. Specifically, we propose spatio-temporal compositional diffusion to precisely follow complex textual semantics by manipulating and composing the attention maps of denoising networks spatially and temporally. Moreover, we propose an enhanced video data preprocessing to enhance the training data regarding motion dynamics and prompt understanding, equipped with a new reference frame attention mechanism to improve the consistency of auto-regressive video generation. Extensive experiments demonstrate that our VideoTetris achieves impressive qualitative and quantitative results in compositional T2V generation. Code is available at: https://github.com/YangLing0818/VideoTetris

Key facts. Cited by 51 papers in the map (data through 14 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 50 of its 78 references that are in the map. First posted on arXiv 6 Jun 2024.

Builds on

50 of 78 references.

Closest ancestors · 15

Same direction, newest first

  1. 2024StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text21 Mar
  2. 2024Lumiere: A Space-Time Diffusion Model for Video Generation23 Jan
  3. 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models17 Jan
  4. 2024Latte: Latent Diffusion Transformer for Video Generation5 Jan
  5. 2023Photorealistic Video Generation with Diffusion Models11 Dec
  6. 2023MEVG: Multi-event Video Generation with Text-to-Video Models7 Dec
  7. 2023ARTboldsymbol·V: Auto-Regressive Text-to-Video Generation with Diffusion Models30 Nov
  8. 2023SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Models28 Nov
  9. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets25 Nov
  10. 2023FreeNoise: Tuning-Free Longer Video Diffusion via Noise Rescheduling23 Oct
  11. 2023LLM-grounded Video Diffusion Models29 Sep
  12. 2023ModelScope Text-to-Video Technical Report12 Aug
3 more
  1. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning10 Jul
  2. 2023Gen-L-Video: Multi-Text to Long Video Generation via Temporal Co-Denoising29 May
  3. 2022Video Diffusion Models7 Apr

Other references · 35

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  7. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  10. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,415
  11. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  12. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,464
23 more
  1. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  2. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,234
  3. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,114
  4. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  5. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 781
  6. 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Modelscited by 702
  7. 2022Compositional Visual Generation with Composable Diffusion Modelscited by 624
  8. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 497
  9. 2024InstantID: Zero-shot Identity-Preserving Generation in Secondscited by 436
  10. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 393
  11. 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesiscited by 388
  12. 2023Training-Free Layout Control with Cross-Attention Guidancecited by 313
  13. 2023BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusioncited by 282
  14. 2023T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-image Generationcited by 260
  15. 2023LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Modelscited by 243
  16. 2024Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMscited by 224
  17. 2023VideoDirectorGPT: Consistent Multi-scene Video Generation via LLM-Guided Planningcited by 141
  18. 2021Simple multi-dataset detectioncited by 127
  19. 2023LayoutLLM-T2I: Eliciting Layout Guidance from LLM for Text-to-Image Generationcited by 92
  20. 2023Compositional Text-to-Image Synthesis with Attention Map Control of Diffusion Modelscited by 83
  21. 2024CogView3: Finer and Faster Text-to-Image Generation via Relay Diffusioncited by 72
  22. 2024Vlogger: Make Your Dream A Vlogcited by 70
  23. 2023A Picture is Worth a Thousand Words: Principled Recaptioning Improves Image Generationcited by 41

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 9; 2025: 32; 2026: 10.

51 papers.

Sort:

By area:

  1. 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 250
  2. 2024T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generationcited by 185
  3. 2024FreeLong: Training-Free Long Video Generation with SpectralBlend Temporal Attentioncited by 89
  4. 2024Progressive Autoregressive Video Diffusion Modelscited by 68
  5. 2025StreamDiT: Real-Time Streaming Text-to-Video Generationcited by 48
  6. 2025Captain Cinema: Towards Short Movie Generationcited by 48
  7. 2024ARLON: Boosting Diffusion Transformers with Autoregressive Models for Long Video Generationcited by 41
  8. 2024The Dawn of Video Generation: Preliminary Explorations with SORA-like Modelscited by 31
  9. 2024HumanEdit: A High-Quality Human-Rewarded Dataset for Instruction-based Image Editingcited by 29
  10. 2025Video-SafetyBench: A Benchmark for Safety Evaluation of Video LVLMscited by 28
  11. 2025Training-free Diffusion Acceleration with Bottleneck Samplingcited by 25
  12. 2025NarrLV: Towards a Comprehensive Narrative-Centric Evaluation for Long Video Generationcited by 22
38 more
  1. 2024Trans4D: Realistic Geometry-Aware Transition for Compositional Text-to-4D Synthesiscited by 19
  2. 2025VideoAuteur: Towards Long Narrative Video Generationcited by 18
  3. 2025HermesFlow: Seamlessly Closing the Gap in Multimodal Understanding and Generationcited by 18
  4. 2025T2VShield: Model-Agnostic Jailbreak Defense for Text-to-Video Modelscited by 18
  5. 2025MoGA: Mixture-of-Groups Attention for End-to-End Long Video Generationcited by 18
  6. 2025FreeLong++: Training-Free Long Video Generation via Multi-band SpectralFusioncited by 17
  7. 2025BlobGEN-Vid: Compositional Text-to-Video Generation with Blob Video Representationscited by 15
  8. 2025Token Perturbation Guidance for Diffusion Modelscited by 15
  9. 2025Simulating the Visual World with Artificial Intelligence: A Roadmapcited by 15
  10. 2024Self-Correcting Text-to-Video Generation with Misalignment Detection and Localized Refinementcited by 14
  11. 2025LongDiff: Training-Free Long Video Generation in One Gocited by 13
  12. 2025MotionAgent: Fine-grained Controllable Video Generation via Motion Field Agentcited by 11
  13. 2025Ouroboros-Diffusion: Exploring Consistent Content Generation in Tuning-free Long Video Diffusioncited by 10
  14. 2025Turning Internal Gap into Self-Improvement: Promoting the Generation-Understanding Unification in MLLMscited by 10
  15. 2025VCapsBench: A Large-scale Fine-grained Benchmark for Video Caption Quality Evaluationcited by 9
  16. 2025AutoMV: An Automatic Multi-Agent System for Music Video Generationcited by 8
  17. 2024Video Is Worth a Thousand Images: Exploring the Latest Trends in Long Video Generationcited by 7
  18. 2025Tuning-Free Multi-Event Long Video Generation via Synchronized Coupled Samplingcited by 6
  19. 2025Preacher: Paper-to-Video Agentic Systemcited by 6
  20. 2026PhysVid: Physics Aware Local Conditioning for Generative Video Modelscited by 6
  21. 2025Multi-Object Sketch Animation by Scene Decomposition and Motion Planningcited by 4
  22. 2025CI-VID: A Coherent Interleaved Text-Video Datasetcited by 4
  23. 2025TTOM: Test-Time Optimization and Memorization for Compositional Video Generationcited by 4
  24. 2026TS-Attn: Temporal-wise Separable Attention for Multi-Event Video Generationcited by 4
  25. 2025Seeing Clearly, Forgetting Deeply: Revisiting Fine-Tuned Video Generators for Driving Simulationcited by 3
  26. 2025Revealing Multimodal Causality with Large Language Modelscited by 3
  27. 2026OmniLottie: Generating Vector Animations via Parameterized Lottie Tokenscited by 3
  28. 2025Bridging Text and Video Generation: A Surveycited by 2
  29. 2026GTA: Advancing Image-to-3D World Generation via Geometry Then Appearance Video Diffusioncited by 2
  30. 2025From Prompt to Progression: Taming Video Diffusion Models for Seamless Attribute Transitioncited by 1
  31. 2025Adaptive Begin-of-Video Tokens for Autoregressive Video Diffusion Modelscited by 1
  32. 2025Anchored Video Generation: Decoupling Scene Construction and Temporal Synthesis in Text-to-Video Diffusion Modelscited by 1
  33. 2026TokenTrim: Inference-Time Token Pruning for Autoregressive Long Video Generationcited by 1
  34. 2026Training-free Motion Factorization for Compositional Video Generationcited by 1
  35. 2026NoisEasier: Test-Time Noise Optimization for Text-to-Video Generationcited by 1
  36. 2025Comp-Attn: Present-and-Align Attention for Compositional Video Generationcited by 0
  37. 2026Compositional Video Generation via Inference-Time Guidancecited by 0
  38. 2026SafeCA: Safe Cross-Attention Localization and Regulation for Text-to-Video Jailbreak Defensecited by 0

50 most cited of 51 All 51 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Compositional Video Generation as Flow Equalization
  2. 2023VideoGen: A Reference-Guided Latent Diffusion Approach for High Definition Text-to-Video Generation
  3. 2024xGen-VideoSyn-1: High-fidelity Text-to-Video Synthesis with Compressed Representations
  4. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators
  5. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation
  6. 2025RepVideo: Rethinking Cross-Layer Representation for Video Generation
  7. 2026Image-to-Video Diffusion: From Foundations to Open Frontiers
  8. 2025Training-free Guidance in Text-to-Video Generation via Multimodal Planning and Structured Noise Initialization