# ViNTER: Image Narrative Generation with Emotion-Arc-Aware Transformer

arXiv: https://arxiv.org/abs/2202.07305 · HTML: https://mlsanity.com/paper/2202.07305/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 51 references that are in the map.
- First posted on arXiv 15 Feb 2022.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [VL-BERT: Pre-training of Generic Visual-Linguistic Representations](https://mlsanity.com/paper/1908.08530/) (2019)
- [Plug and Play Language Models: A Simple Approach to Controlled Text Generation](https://mlsanity.com/paper/1912.02164/) (2019)
- [DenseCap: Fully Convolutional Localization Networks for Dense Captioning](https://mlsanity.com/paper/1511.07571/) (2015)
- [Visual Storytelling](https://mlsanity.com/paper/1604.03968/) (2016)
- [CAN: Creative Adversarial Networks, Generating "Art" by Learning About Styles and Deviating from Style Norms](https://mlsanity.com/paper/1706.07068/) (2017)
- [MEGATRON-CNTRL: Controllable Story Generation with External Knowledge Using Large-Scale Language Models](https://mlsanity.com/paper/2010.00840/) (2020)
- [The emotional arcs of stories are dominated by six basic shapes](https://mlsanity.com/paper/1606.07772/) (2016)
- [Modeling Protagonist Emotions for Emotion-Aware Storytelling](https://mlsanity.com/paper/2010.06822/) (2020)
- [Beyond Narrative Description: Generating Poetry from Images by Multi-Adversarial Training](https://mlsanity.com/paper/1804.08473/) (2018)
- ["My Way of Telling a Story": Persona based Grounded Story Generation](https://mlsanity.com/paper/1906.06401/) (2019)
- [A Survey on Sentiment and Emotion Analysis for Computational Literary Studies](https://mlsanity.com/paper/1808.03137/) (2018)
- [An Analysis of Emotion Communication Channels in Fan Fiction: Towards Emotional Storytelling](https://mlsanity.com/paper/1906.02402/) (2019)
- [Audio-Visual Sentiment Analysis for Learning Emotional Arcs in Movies](https://mlsanity.com/paper/1712.02896/) (2017)
- [Customized Image Narrative Generation via Interactive Visual Question Generation and Answering](https://mlsanity.com/paper/1805.00460/) (2018)

## Built on by (most cited first, 4 of 4)

- [Creative Data Generation: A Review Focusing on Text and Poetry](https://mlsanity.com/paper/2305.08493/) (2023)
- ["We are at the mercy of others' opinion": Supporting Blind People in Recreational Window Shopping with AI-infused Technology](https://mlsanity.com/paper/2405.06611/) (2024)
- [Text-Only Training for Visual Storytelling](https://mlsanity.com/paper/2308.08881/) (2023)
- [SPECTRUM: Semantic Processing and Emotion-informed video-Captioning Through Retrieval and Understanding Modalities](https://mlsanity.com/paper/2411.01975/) (2024)

## Abstract (arXiv)

Image narrative generation is a task to create a story from an image with a subjective viewpoint. Given the importance of the subjective feelings of writers, readers, and characters in storytelling, an image narrative generation method should consider human emotion. In this study, we propose a novel method of image narrative generation called ViNTER (Visual Narrative Transformer with Emotion arc Representation), which takes "emotion arc" as input to capture a sequence of emotional changes. Since emotion arcs represent the trajectory of emotional change, it is expected that we can include detailed information about the emotional changes in the story to the model. We present experimental results of both automatic and manual evaluations on the Image Narrative dataset and demonstrate the effectiveness of the proposed approach.