# Let Storytelling Tell Vivid Stories: An Expressive and Fluent Multimodal Storyteller

arXiv: https://arxiv.org/abs/2403.07301 · HTML: https://mlsanity.com/paper/2403.07301/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 26 references that are in the map.
- First posted on arXiv 12 Mar 2024.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2308.06721/) (2023)
- [Multimodal Few-Shot Learning with Frozen Language Models](https://mlsanity.com/paper/2106.13884/) (2021)
- [NExT-GPT: Any-to-Any Multimodal LLM](https://mlsanity.com/paper/2309.05519/) (2023)
- [Visual Storytelling](https://mlsanity.com/paper/1604.03968/) (2016)
- [Vokenization: Improving Language Understanding with Contextualized, Visual-Grounded Supervision](https://mlsanity.com/paper/2010.06775/) (2020)
- [No Metrics Are Perfect: Adversarial Reward Learning for Visual Storytelling](https://mlsanity.com/paper/1804.09160/) (2018)
- [What Makes A Good Story? Designing Composite Rewards for Visual Storytelling](https://mlsanity.com/paper/1909.05316/) (2019)
- [Knowledge-Enriched Visual Storytelling](https://mlsanity.com/paper/1912.01496/) (2019)
- [GLAC Net: GLocal Attention Cascading Networks for Multi-image Cued Story Generation](https://mlsanity.com/paper/1805.10973/) (2018)
- [Commonsense Knowledge Aware Concept Selection For Diverse and Informative Visual Storytelling](https://mlsanity.com/paper/2102.02963/) (2021)
- [Multimodal Storytelling via Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1712.01455/) (2017)

## Built on by (most cited first, 4 of 4)

- [A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks](https://mlsanity.com/paper/2411.06284/) (2024)
- [What Makes a Good Story and How Can We Measure It? A Comprehensive Survey of Story Evaluation](https://mlsanity.com/paper/2408.14622/) (2024)
- [Character-Centered Dialogue Generation from Scene-Level Prompts](https://mlsanity.com/paper/2505.16819/) (2025)
- [FlexMUSE: Multimodal Unification and Semantics Enhancement Framework with Flexible interaction for Creative Writing](https://mlsanity.com/paper/2508.16230/) (2025)

## Abstract (arXiv)

Storytelling aims to generate reasonable and vivid narratives based on an ordered image stream. The fidelity to the image story theme and the divergence of story plots attract readers to keep reading. Previous works iteratively improved the alignment of multiple modalities but ultimately resulted in the generation of simplistic storylines for image streams. In this work, we propose a new pipeline, termed LLaMS, to generate multimodal human-level stories that are embodied in expressiveness and consistency. Specifically, by fully exploiting the commonsense knowledge within the LLM, we first employ a sequence data auto-enhancement strategy to enhance factual content expression and leverage a textual reasoning architecture for expressive story generation and prediction. Secondly, we propose SQ-Adatpter module for story illustration generation which can maintain sequence consistency. Numerical results are conducted through human evaluation to verify the superiority of proposed LLaMS. Evaluations show that LLaMS achieves state-of-the-art storytelling performance and 86% correlation and 100% consistency win rate as compared with previous SOTA methods. Furthermore, ablation experiments are conducted to verify the effectiveness of proposed sequence data enhancement and SQ-Adapter.