mlsanityData to 7 Oct 2026

Imagine This! Scripts to Compositions to Videos

arXiv 1804.03608 ↗First posted 10 Apr 2018cs.CV cs.CL cs.IRcited by 54

Not on the map. Why?

Abstract (arXiv)

Imagining a scene described in natural language with realistic layout and appearance of entities is the ultimate test of spatial, visual, and semantic world knowledge. Towards this goal, we present the Composition, Retrieval, and Fusion Network (CRAFT), a model capable of learning this knowledge from video-caption data and applying it while generating videos from novel captions. CRAFT explicitly predicts a temporal-layout of mentioned entities (characters and objects), retrieves spatio-temporal entity segments from a video database and fuses them to generate scene videos. Our contributions include sequential training of components of CRAFT while jointly modeling layout and appearances, and losses that encourage learning compositional representations for retrieval. We evaluate CRAFT on semantic fidelity to caption, composition consistency, and visual quality. CRAFT outperforms direct pixel generation approaches and generalizes well to unseen captions and to unseen video databases with no text annotations. We demonstrate CRAFT on FLINTSTONES, a new richly annotated video-caption dataset with over 25000 videos. For a glimpse of videos generated by CRAFT, see https://youtu.be/688Vv86n0z8.

Key facts. Cited by 54 papers in the map (data through 7 Oct 2026). First posted on arXiv 10 Apr 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 11; 2023: 15; 2024: 16; 2025: 8; 2026: 1.

54 papers.

Sort:

By area:

  1. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  2. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  3. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  4. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 958
  5. 2022Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformercited by 244
  6. 2023Gen-L-Video: Multi-Text to Long Video Generation via Temporal Co-Denoisingcited by 135
  7. 2024MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scalecited by 125
  8. 2022MaskViT: Masked Visual Pre-Training for Video Predictioncited by 112
  9. 2023CelebV-Text: A Large-Scale Facial Text-Video Datasetcited by 103
  10. 2023SimDA: Simple Diffusion Adapter for Efficient Video Generationcited by 101
  11. 2025One-Minute Video Generation with Test-Time Trainingcited by 100
  12. 2023A Complete Survey on Generative AI (AIGC): Is ChatGPT from GPT-4 to GPT-5 All You Need?cited by 84
38 more
  1. 2024MM-Interleaved: Interleaved Image-Text Generative Modeling via Multi-modal Feature Synchronizercited by 80
  2. 2022Make-A-Story: Visual Memory Conditioned Consistent Story Generationcited by 76
  3. 2023Intelligent Grimm -- Open-ended Visual Storytelling via Latent Diffusion Modelscited by 76
  4. 2023Fine-tuning Multimodal LLMs to Follow Zero-shot Demonstrative Instructionscited by 76
  5. 2022StoryDALL-E: Adapting Pretrained Text-to-Image Transformers for Story Continuationcited by 74
  6. 2023ARTboldsymbol·V: Auto-Regressive Text-to-Video Generation with Diffusion Modelscited by 74
  7. 2024SEED-Story: Multimodal Long Story Generation with Large Language Modelcited by 70
  8. 2022Synthesizing Coherent Story with Auto-Regressive Latent Diffusion Modelscited by 69
  9. 2021Integrating Visuospatial, Linguistic and Commonsense Structure into Story Visualizationcited by 51
  10. 2024DreamStory: Open-Domain Story Visualization by LLM-Guided Multi-Subject Consistent Diffusioncited by 46
  11. 2021Improving Generation and Evaluation of Visual Stories via Semantic Consistencycited by 43
  12. 2022Character-Centric Story Visualization via Visual Planning and Token Alignmentcited by 32
  13. 2022Tell Me What Happened: Unifying Text-guided Video Completion via Multimodal Masked Video Generationcited by 32
  14. 2024From Sora What We Can See: A Survey of Text-to-Video Generationcited by 26
  15. 2023StoryGPT-V: Large Language Models as Consistent Story Visualizerscited by 25
  16. 2022Word-Level Fine-Grained Story Visualizationcited by 22
  17. 2023Improved Visual Story Generation with Adaptive Context Modelingcited by 22
  18. 2022MUGEN: A Playground for Video-Audio-Text Multimodal Understanding and GENerationcited by 20
  19. 2024StoryImager: A Unified and Efficient Framework for Coherent Story Visualization and Completioncited by 19
  20. 2024TheaterGen: Character Management with LLM for Consistent Multi-turn Image Generationcited by 18
  21. 2024What Makes a Good Story and How Can We Measure It? A Comprehensive Survey of Story Evaluationcited by 17
  22. 2025VideoAuteur: Towards Long Narrative Video Generationcited by 17
  23. 2024DiffSensei: Bridging Multi-Modal LLMs and Diffusion Models for Customized Manga Generationcited by 16
  24. 2024Story-Iter: A Training-free Iterative Paradigm for Long Story Visualizationcited by 12
  25. 2025ManipDreamer: Boosting Robotic Manipulation World Model with Action Tree and Visual Guidancecited by 11
  26. 2023Causal-Story: Local Causal Attention Utilizing Parameter-Efficient Tuning For Visual Story Synthesiscited by 9
  27. 2024Evolving Storytelling: Benchmarks and Methods for New Character Customization with Diffusion Modelscited by 9
  28. 2024StoryWeaver: A Unified World Model for Knowledge-Enhanced Story Character Customizationcited by 9
  29. 2024ContextualStory: Consistent Visual Storytelling with Spatially-Enhanced and Storyline Contextcited by 8
  30. 2023Story Visualization by Online Text Augmentation with Context Memorycited by 7
  31. 2023Make-A-Storyboard: A General Framework for Storyboard with Disentangled and Merged Controlcited by 7
  32. 2024Generating Coherent Sequences of Visual Illustrations for Real-World Manual Taskscited by 6
  33. 2023SAVE: Spectral-Shift-Aware Adaptation of Image Diffusion Models for Text-driven Video Editingcited by 5
  34. 2025Breaking Language Barriers in Visual Language Models via Multilingual Textual Regularizationcited by 4
  35. 2025Burn After Reading: Do Multimodal Large Language Models Truly Capture Order of Events in Image Sequences?cited by 4
  36. 2023Learning Data-Driven Vector-Quantized Degradation Model for Animation Video Super-Resolutioncited by 3
  37. 2024Improving Multi-Subject Consistency in Open-Domain Image Generation with Isolation and Reposition Attentioncited by 3
  38. 2025Top-Down Compression: Revisit Efficient Vision Token Projection for Visual Instruction Tuningcited by 3

50 most cited of 54 All 54 (JSON)