mlsanityData to 14 Oct 2026

Multimodal Event Transformer for Image-guided Story Ending Generation

arXiv 2301.11357 ↗First posted 26 Jan 2023cs.CV cs.CLcited by 21

Not on the map. Why?

Abstract (arXiv)

Image-guided story ending generation (IgSEG) is to generate a story ending based on given story plots and ending image. Existing methods focus on cross-modal feature fusion but overlook reasoning and mining implicit information from story plots and ending image. To tackle this drawback, we propose a multimodal event transformer, an event-based reasoning framework for IgSEG. Specifically, we construct visual and semantic event graphs from story plots and ending image, and leverage event-based reasoning to reason and mine implicit information in a single modality. Next, we connect visual and semantic event graphs and utilize cross-modal fusion to integrate different-modality features. In addition, we propose a multimodal injector to adaptive pass essential information to decoder. Besides, we present an incoherence detection to enhance the understanding context of a story plot and the robustness of graph modeling for our model. Experimental results show that our method achieves state-of-the-art performance for the image-guided story ending generation.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Builds on 24 of its 38 references that are in the map. First posted on arXiv 26 Jan 2023.

Builds on

24 of 38 references.

References · 24

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  6. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  7. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  8. 2017Modeling Relational Data with Graph Convolutional Networkscited by 1,694
  9. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,026
  10. 2017Scene Graph Generation by Iterative Message Passingcited by 462
  11. 2017Neural Motifs: Scene Graph Parsing with Global Contextcited by 414
  12. 2018Plan-And-Write: Towards Better Automatic Storytellingcited by 234
12 more
  1. 2016Movie Descriptioncited by 181
  2. 2016Visual Storytellingcited by 176
  3. 2019Strategies for Structuring Story Generationcited by 124
  4. 2020A Knowledge-Enhanced Pretraining Model for Commonsense Story Generationcited by 116
  5. 2019ASER: A Large-scale Eventuality Knowledge Graphcited by 89
  6. 2018Story Ending Generation with Incremental Encoding and Commonsense Knowledgecited by 60
  7. 2018No Metrics Are Perfect: Adversarial Reward Learning for Visual Storytellingcited by 46
  8. 2022ClarET: Pre-training a Correlation-Aware Context-To-Event Transformer for Event-Centric Generation and Classificationcited by 45
  9. 2021EventBERT: A Pre-Trained Model for Event Correlation Reasoningcited by 40
  10. 2019Story Realization: Expanding Plot Events into Sentencescited by 35
  11. 2021Commonsense Knowledge Aware Concept Selection For Diverse and Informative Visual Storytellingcited by 22
  12. 2020Hide-and-Tell: Learning to Bridge Photo Streams for Visual Storytellingcited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 10; 2025: 9; 2026: 1.

21 papers.

Sort:

By area:

  1. 2024MemoryMamba: Memory-Augmented State Space Model for Defect Recognitioncited by 24
  2. 2024Real-Time Pill Identification for the Visually Impaired Using Deep Learningcited by 21
  3. 2024A Character-Centric Creative Story Generation via Imaginationcited by 12
  4. 2024Exploiting Diffusion Prior for Out-of-Distribution Detectioncited by 7
  5. 2023Topic-Selective Graph Network for Topic-Focused Summarizationcited by 4
  6. 2025ContextualLVLM-Agent: A Holistic Framework for Multi-Turn Visually-Grounded Dialogue and Complex Instruction Followingcited by 3
  7. 2025Knowledge-Augmented Language Models Interpreting Structured Chest X-Ray Findingscited by 2
  8. 2024Leveraging Language Models for Emotion and Behavior Analysis in Educationcited by 1
  9. 2024Coal Mining Question Answering with LLMscited by 1
  10. 2025Evidence-Driven Marker Extraction for Social Media Suicide Risk Detectioncited by 1
  11. 2025Generating Medically-Informed Explanations for Depression Detection using LLMscited by 1
  12. 2025Hierarchical Contextual Grounding LVLM: Enhancing Fine-Grained Visual-Language Understanding with Robust Groundingcited by 1
9 more
  1. 2024Consultation on Industrial Machine Faults with Large language Modelscited by 0
  2. 2024Exploring Large Vision-Language Models for Robust and Efficient Industrial Anomaly Detectioncited by 0
  3. 2024Bridging Vision and Language: Modeling Causality and Temporality in Video Narrativescited by 0
  4. 2024Optimizing Vision-Language Interactions Through Decoder-Only Modelscited by 0
  5. 2025Hierarchical Vision-Language Alignment for Text-to-Image Generation via Diffusion Modelscited by 0
  6. 2025Enhancing Contrastive Demonstration Selection with Semantic Diversity for Robust In-Context Machine Translationcited by 0
  7. 2025Contextual Candor: Enhancing LLM Trustworthiness Through Hierarchical Unanswerability Detectioncited by 0
  8. 2025Unlocking Compositional Control: Self-Supervision for LVLM-Based Image Generationcited by 0
  9. 2026When Does External Guidance Help LLM Reasoning? A Bias-Variance Theory of Guidance-Augmented GRPOcited by 0