# Multimodal Event Transformer for Image-guided Story Ending Generation

arXiv: https://arxiv.org/abs/2301.11357 · HTML: https://mlsanity.com/paper/2301.11357/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 38 references that are in the map.
- First posted on arXiv 26 Jan 2023.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Modeling Relational Data with Graph Convolutional Networks](https://mlsanity.com/paper/1703.06103/) (2017)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [Scene Graph Generation by Iterative Message Passing](https://mlsanity.com/paper/1701.02426/) (2017)
- [Neural Motifs: Scene Graph Parsing with Global Context](https://mlsanity.com/paper/1711.06640/) (2017)
- [Plan-And-Write: Towards Better Automatic Storytelling](https://mlsanity.com/paper/1811.05701/) (2018)
- [Movie Description](https://mlsanity.com/paper/1605.03705/) (2016)
- [Visual Storytelling](https://mlsanity.com/paper/1604.03968/) (2016)
- [Strategies for Structuring Story Generation](https://mlsanity.com/paper/1902.01109/) (2019)
- [A Knowledge-Enhanced Pretraining Model for Commonsense Story Generation](https://mlsanity.com/paper/2001.05139/) (2020)
- [ASER: A Large-scale Eventuality Knowledge Graph](https://mlsanity.com/paper/1905.00270/) (2019)
- [Story Ending Generation with Incremental Encoding and Commonsense Knowledge](https://mlsanity.com/paper/1808.10113/) (2018)
- [No Metrics Are Perfect: Adversarial Reward Learning for Visual Storytelling](https://mlsanity.com/paper/1804.09160/) (2018)
- [ClarET: Pre-training a Correlation-Aware Context-To-Event Transformer for Event-Centric Generation and Classification](https://mlsanity.com/paper/2203.02225/) (2022)
- [EventBERT: A Pre-Trained Model for Event Correlation Reasoning](https://mlsanity.com/paper/2110.06533/) (2021)
- [Story Realization: Expanding Plot Events into Sentences](https://mlsanity.com/paper/1909.03480/) (2019)
- [Commonsense Knowledge Aware Concept Selection For Diverse and Informative Visual Storytelling](https://mlsanity.com/paper/2102.02963/) (2021)
- [Hide-and-Tell: Learning to Bridge Photo Streams for Visual Storytelling](https://mlsanity.com/paper/2002.00774/) (2020)

## Built on by (most cited first, 21 of 21)

- [MemoryMamba: Memory-Augmented State Space Model for Defect Recognition](https://mlsanity.com/paper/2405.03673/) (2024)
- [Real-Time Pill Identification for the Visually Impaired Using Deep Learning](https://mlsanity.com/paper/2405.05983/) (2024)
- [A Character-Centric Creative Story Generation via Imagination](https://mlsanity.com/paper/2409.16667/) (2024)
- [Exploiting Diffusion Prior for Out-of-Distribution Detection](https://mlsanity.com/paper/2406.11105/) (2024)
- [Topic-Selective Graph Network for Topic-Focused Summarization](https://mlsanity.com/paper/2302.13106/) (2023)
- [ContextualLVLM-Agent: A Holistic Framework for Multi-Turn Visually-Grounded Dialogue and Complex Instruction Following](https://mlsanity.com/paper/2508.15164/) (2025)
- [Knowledge-Augmented Language Models Interpreting Structured Chest X-Ray Findings](https://mlsanity.com/paper/2505.01711/) (2025)
- [Leveraging Language Models for Emotion and Behavior Analysis in Education](https://mlsanity.com/paper/2408.06874/) (2024)
- [Coal Mining Question Answering with LLMs](https://mlsanity.com/paper/2410.02959/) (2024)
- [Evidence-Driven Marker Extraction for Social Media Suicide Risk Detection](https://mlsanity.com/paper/2502.18823/) (2025)
- [Generating Medically-Informed Explanations for Depression Detection using LLMs](https://mlsanity.com/paper/2503.14671/) (2025)
- [Hierarchical Contextual Grounding LVLM: Enhancing Fine-Grained Visual-Language Understanding with Robust Grounding](https://mlsanity.com/paper/2508.16974/) (2025)
- [Consultation on Industrial Machine Faults with Large language Models](https://mlsanity.com/paper/2410.03223/) (2024)
- [Exploring Large Vision-Language Models for Robust and Efficient Industrial Anomaly Detection](https://mlsanity.com/paper/2412.00890/) (2024)
- [Bridging Vision and Language: Modeling Causality and Temporality in Video Narratives](https://mlsanity.com/paper/2412.10720/) (2024)
- [Optimizing Vision-Language Interactions Through Decoder-Only Models](https://mlsanity.com/paper/2412.10758/) (2024)
- [Hierarchical Vision-Language Alignment for Text-to-Image Generation via Diffusion Models](https://mlsanity.com/paper/2501.00917/) (2025)
- [Enhancing Contrastive Demonstration Selection with Semantic Diversity for Robust In-Context Machine Translation](https://mlsanity.com/paper/2504.09305/) (2025)
- [Contextual Candor: Enhancing LLM Trustworthiness Through Hierarchical Unanswerability Detection](https://mlsanity.com/paper/2506.01104/) (2025)
- [Unlocking Compositional Control: Self-Supervision for LVLM-Based Image Generation](https://mlsanity.com/paper/2507.04151/) (2025)
- [When Does External Guidance Help LLM Reasoning? A Bias-Variance Theory of Guidance-Augmented GRPO](https://mlsanity.com/paper/2610.06861/) (2026)

## Abstract (arXiv)

Image-guided story ending generation (IgSEG) is to generate a story ending based on given story plots and ending image. Existing methods focus on cross-modal feature fusion but overlook reasoning and mining implicit information from story plots and ending image. To tackle this drawback, we propose a multimodal event transformer, an event-based reasoning framework for IgSEG. Specifically, we construct visual and semantic event graphs from story plots and ending image, and leverage event-based reasoning to reason and mine implicit information in a single modality. Next, we connect visual and semantic event graphs and utilize cross-modal fusion to integrate different-modality features. In addition, we propose a multimodal injector to adaptive pass essential information to decoder. Besides, we present an incoherence detection to enhance the understanding context of a story plot and the robustness of graph modeling for our model. Experimental results show that our method achieves state-of-the-art performance for the image-guided story ending generation.