mlsanityData to 7 Oct 2026

SEED-Story: Multimodal Long Story Generation with Large Language Model

arXiv 2407.08683 ↗First posted 11 Jul 2024cs.CVcited by 70

Not on the map. Why?

Abstract (arXiv)

With the remarkable advancements in image generation and open-form text generation, the creation of interleaved image-text content has become an increasingly intriguing field. Multimodal story generation, characterized by producing narrative texts and vivid images in an interleaved manner, has emerged as a valuable and practical task with broad applications. However, this task poses significant challenges, as it necessitates the comprehension of the complex interplay between texts and images, and the ability to generate long sequences of coherent, contextually relevant texts and visuals. In this work, we propose SEED-Story, a novel method that leverages a Multimodal Large Language Model (MLLM) to generate extended multimodal stories. Our model, built upon the powerful comprehension capability of MLLM, predicts text tokens as well as visual tokens, which are subsequently processed with an adapted visual de-tokenizer to produce images with consistent characters and styles. We further propose multimodal attention sink mechanism to enable the generation of stories with up to 25 sequences (only 10 for training) in a highly efficient autoregressive manner. Additionally, we present a large-scale and high-resolution dataset named StoryStream for training our model and quantitatively evaluating the task of multimodal story generation in various aspects.

Key facts. Cited by 70 papers in the map (data through 7 Oct 2026). Builds on 52 of its 62 references that are in the map. First posted on arXiv 11 Jul 2024.

Builds on

52 of 62 references.

References · 52

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  11. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  12. 2020Longformer: The Long-Document Transformercited by 3,362
38 more
  1. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  2. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  3. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  4. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  5. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  6. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  7. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  8. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesiscited by 854
  9. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  10. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 466
  11. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  12. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 361
  13. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 328
  14. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  15. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  16. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 296
  17. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 283
  18. 2023Generating Images with Multimodal Language Modelscited by 259
  19. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 232
  20. 2023Making LLaMA SEE and Draw with SEED Tokenizercited by 184
  21. 2016Visual Storytellingcited by 176
  22. 2023Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuningcited by 168
  23. 2023Emu: Generative Pretraining in Multimodalitycited by 156
  24. 2023Planting a SEED of Vision in Large Language Modelcited by 140
  25. 2018StoryGAN: A Sequential Conditional GAN for Story Visualizationcited by 137
  26. 2024Training-Free Consistent Text-to-Image Generationcited by 123
  27. 2024MM-Interleaved: Interleaved Image-Text Generative Modeling via Multi-modal Feature Synchronizercited by 80
  28. 2022Make-A-Story: Visual Memory Conditioned Consistent Story Generationcited by 76
  29. 2023Intelligent Grimm -- Open-ended Visual Storytelling via Latent Diffusion Modelscited by 76
  30. 2022StoryDALL-E: Adapting Pretrained Text-to-Image Transformers for Story Continuationcited by 74
  31. 2023TaleCrafter: Interactive Story Visualization with Multiple Characterscited by 71
  32. 2022Synthesizing Coherent Story with Auto-Regressive Latent Diffusion Modelscited by 69
  33. 2023The Chosen One: Consistent Characters in Text-to-Image Diffusion Modelscited by 62
  34. 2018Imagine This! Scripts to Compositions to Videoscited by 54
  35. 2021Integrating Visuospatial, Linguistic and Commonsense Structure into Story Visualizationcited by 51
  36. 2021Improving Generation and Evaluation of Visual Stories via Semantic Consistencycited by 43
  37. 2023VL-GPT: A Generative Pre-trained Transformer for Vision and Language Understanding and Generationcited by 41
  38. 2023Jointly Training Large Autoregressive Multimodal Modelscited by 32

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 18; 2025: 35; 2026: 17.

70 papers.

Sort:

By area:

  1. 2025LongLive: Real-time Interactive Long Video Generationcited by 237
  2. 2024When Attention Sink Emerges in Language Models: An Empirical Viewcited by 200
  3. 2024In-Context LoRA for Diffusion Transformerscited by 155
  4. 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videoscited by 154
  5. 2024VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained Video Reasoning via Core Frame Selectioncited by 64
  6. 2025Captain Cinema: Towards Short Movie Generationcited by 48
  7. 2024Orthus: Autoregressive Interleaved Image-Text Generation with Modality-Specific Headscited by 44
  8. 2025SCORE: Story Coherence and Retrieval Enhancement for AI Narrativescited by 42
  9. 2024Autoregressive Models in Vision: A Surveycited by 39
  10. 2025StoryMem: Multi-shot Long Video Storytelling with Memorycited by 35
  11. 2024OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generationcited by 30
  12. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
38 more
  1. 2024Trans4D: Realistic Geometry-Aware Transition for Compositional Text-to-4D Synthesiscited by 19
  2. 2025VideoAuteur: Towards Long Narrative Video Generationcited by 17
  3. 2025Generative AI for Cel-Animation: A Surveycited by 17
  4. 2024DiffSensei: Bridging Multi-Modal LLMs and Diffusion Models for Customized Manga Generationcited by 16
  5. 2025Story2Board: A Training-Free Approach for Expressive Storyboard Generationcited by 16
  6. 2025Unified Dense Prediction of Video Diffusioncited by 13
  7. 2025MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Modelscited by 13
  8. 2024Story-Iter: A Training-free Iterative Paradigm for Long Story Visualizationcited by 12
  9. 2025Pushing the Limits of Safety: A Technical Report on the ATLAS Challenge 2025cited by 11
  10. 2024Safe + Safe = Unsafe? Exploring How Safe Images Can Be Exploited to Jailbreak Large Vision-Language Modelscited by 9
  11. 2024Divot: Diffusion Powers Video Tokenizer for Comprehension and Generationcited by 8
  12. 2025AnimeShooter: A Multi-Shot Animation Dataset for Reference-Guided Video Generationcited by 8
  13. 2026The Script is All You Need: An Agentic Framework for Long-Horizon Dialogue-to-Cinematic Video Generationcited by 8
  14. 2024IDEA-Bench: How Far are Generative Models from Professional Designing?cited by 7
  15. 2025AnimeGamer: Infinite Anime Life Simulation with Next Game State Predictioncited by 7
  16. 2025SceneDecorator: Towards Scene-Oriented Story Generation with Scene Planning and Scene Consistencycited by 7
  17. 2025Rethinking Causal Mask Attention for Vision-Language Inferencecited by 6
  18. 2025Lay2Story: Extending Diffusion Transformers for Layout-Togglable Story Generationcited by 6
  19. 2024SkyScript-100M: 1,000,000,000 Pairs of Scripts and Shooting Scripts for Short Dramacited by 5
  20. 2024Olympus: A Universal Task Router for Computer Vision Taskscited by 4
  21. 2024ChatDiT: A Training-Free Baseline for Task-Agnostic Free-Form Chatting with Diffusion Transformerscited by 4
  22. 2025UniMoD: Efficient Unified Multimodal Transformers with Mixture-of-Depthscited by 4
  23. 2025VisAgent: Narrative-Preserving Story Visualization Frameworkcited by 4
  24. 2025Storybooth: Training-free Multi-Subject Consistency for Improved Visual Storytellingcited by 4
  25. 2025CineVerse: Consistent Keyframe Synthesis for Cinematic Scene Compositioncited by 4
  26. 2025SpotEdit: Evaluating Visually-Guided Image Editing Methodscited by 4
  27. 2025DynaKV: Enabling Accurate and Efficient Long-Sequence LLM Decoding on Smartphonescited by 4
  28. 2024ACDC: Autoregressive Coherent Multimodal Generation using Diffusion Correctioncited by 3
  29. 2025A High-Quality Dataset and Reliable Evaluation for Interleaved Image-Text Generationcited by 3
  30. 2025ViSTA: Visual Storytelling using Multi-modal Adapters for Text-to-Image Diffusion Modelscited by 3
  31. 2026Frayed RoPE and Long Inputs: A Geometric Perspectivecited by 3
  32. 2026CANVAS: Continuity-Aware Narratives via Visual Agentic Storyboardingcited by 3
  33. 2025Character-Centered Dialogue Generation from Scene-Level Promptscited by 2
  34. 2025ViStoryBench: Comprehensive Benchmark Suite for Story Visualizationcited by 2
  35. 2025Bridging Your Imagination with Audio-Video Generation via a Unified Directorcited by 2
  36. 2026Mind-of-Director: Multi-modal Agent-Driven Film Previsualization via Collaborative Decision-Makingcited by 2
  37. 2025CoDi: Subject-Consistent and Pose-Diverse Text-to-Image Generationcited by 1
  38. 2025Aether Weaver: Multimodal Affective Narrative Co-Generation with Dynamic Scene Graphscited by 1

50 most cited of 70 All 70 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Improving Visual Storytelling with Multimodal Large Language Models
  2. 2023Zero-shot Generation of Coherent Storybook from Plain Text Story using Diffusion Models
  3. 2024CoMM: A Coherent Interleaved Image-Text Dataset for Multimodal Understanding and Generation
  4. 2024The Revolution of Multimodal Large Language Models: A Survey
  5. 2023MiniGPT-5: Interleaved Vision-and-Language Generation via Generative Vokens
  6. 2023OpenLEAF: Open-Domain Interleaved Image-Text Generation and Evaluation
  7. 2025One-Prompt-One-Story: Free-Lunch Consistent Text-to-Image Generation Using a Single Prompt
  8. 2025Beyond Words: Advancing Long-Text Image Generation via Multimodal Autoregressive Models