# Ordered Attention for Coherent Visual Storytelling

arXiv: https://arxiv.org/abs/2108.02180 · HTML: https://mlsanity.com/paper/2108.02180/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 51 references that are in the map.
- First posted on arXiv 4 Aug 2021.

## Builds on (most cited first, 30 of 35)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks](https://mlsanity.com/paper/2004.06165/) (2020)
- [Show and Tell: A Neural Image Caption Generator](https://mlsanity.com/paper/1411.4555/) (2014)
- [Bilinear Attention Networks](https://mlsanity.com/paper/1805.07932/) (2018)
- [Hierarchical Question-Image Co-Attention for Visual Question Answering](https://mlsanity.com/paper/1606.00061/) (2016)
- [Stacked Attention Networks for Image Question Answering](https://mlsanity.com/paper/1511.02274/) (2015)
- [Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding](https://mlsanity.com/paper/1606.01847/) (2016)
- [X-Linear Attention Networks for Image Captioning](https://mlsanity.com/paper/2003.14080/) (2020)
- [Visual Storytelling](https://mlsanity.com/paper/1604.03968/) (2016)
- [MUTAN: Multimodal Tucker Fusion for Visual Question Answering](https://mlsanity.com/paper/1705.06676/) (2017)
- [Dynamic Fusion with Intra- and Inter- Modality Attention Flow for Visual Question Answering](https://mlsanity.com/paper/1812.05252/) (2018)
- [Beyond Bilinear: Generalized Multimodal Factorized High-order Pooling for Visual Question Answering](https://mlsanity.com/paper/1708.03619/) (2017)
- [Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering](https://mlsanity.com/paper/1511.05234/) (2015)
- [Normalized and Geometry-Aware Self-Attention Network for Image Captioning](https://mlsanity.com/paper/2003.08897/) (2020)
- [No Metrics Are Perfect: Adversarial Reward Learning for Visual Storytelling](https://mlsanity.com/paper/1804.09160/) (2018)
- [Factor Graph Attention](https://mlsanity.com/paper/1904.05880/) (2019)
- [Knowledge-Enriched Visual Storytelling](https://mlsanity.com/paper/1912.01496/) (2019)
- [GLAC Net: GLocal Attention Cascading Networks for Multi-image Cued Story Generation](https://mlsanity.com/paper/1805.10973/) (2018)
- [High-Order Attention Models for Visual Question Answering](https://mlsanity.com/paper/1711.04323/) (2017)
- [Hierarchically Structured Reinforcement Learning for Topically Coherent Visual Story Generation](https://mlsanity.com/paper/1805.08191/) (2018)

## Built on by (most cited first, 6 of 6)

- [DataNarrative: Automated Data-Driven Storytelling with Visualizations and Texts](https://mlsanity.com/paper/2408.05346/) (2024)
- [Diffusion-Based Visual Art Creation: A Survey and New Perspectives](https://mlsanity.com/paper/2408.12128/) (2024)
- [Semantic Alignment for Multimodal Large Language Models](https://mlsanity.com/paper/2408.12867/) (2024)
- [Text-Only Training for Visual Storytelling](https://mlsanity.com/paper/2308.08881/) (2023)
- [Describing Sets of Images with Textual-PCA](https://mlsanity.com/paper/2210.12112/) (2022)
- [TARN-VIST: Topic Aware Reinforcement Network for Visual Storytelling](https://mlsanity.com/paper/2403.11550/) (2024)

## Abstract (arXiv)

We address the problem of visual storytelling, i.e., generating a story for a given sequence of images. While each sentence of the story should describe a corresponding image, a coherent story also needs to be consistent and relate to both future and past images. To achieve this we develop ordered image attention (OIA). OIA models interactions between the sentence-corresponding image and important regions in other images of the sequence. To highlight the important objects, a message-passing-like algorithm collects representations of those objects in an order-aware manner. To generate the story's sentences, we then highlight important image attention vectors with an Image-Sentence Attention (ISA). Further, to alleviate common linguistic mistakes like repetitiveness, we introduce an adaptive prior. The obtained results improve the METEOR score on the VIST dataset by 1%. In addition, an extensive human study verifies coherency improvements and shows that OIA and ISA generated stories are more focused, shareable, and image-grounded.