# Sort Story: Sorting Jumbled Images and Captions into Stories

arXiv: https://arxiv.org/abs/1606.07493 · HTML: https://mlsanity.com/paper/1606.07493/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Jun 2016.

## Built on by (most cited first, 12 of 12)

- [MERLOT: Multimodal Neural Script Knowledge Models](https://mlsanity.com/paper/2106.02636/) (2021)
- [Housekeep: Tidying Virtual Households using Commonsense Reasoning](https://mlsanity.com/paper/2205.10712/) (2022)
- [Understanding Multimodal Procedural Knowledge by Sequencing Multimodal Instructional Manuals](https://mlsanity.com/paper/2110.08486/) (2021)
- [An Overview Of Temporal Commonsense Reasoning and Acquisition](https://mlsanity.com/paper/2308.00002/) (2023)
- [Synopses of Movie Narratives: a Video-Language Dataset for Story Understanding](https://mlsanity.com/paper/2203.05711/) (2022)
- [Positional Diffusion: Ordering Unordered Sets with Diffusion Probabilistic Models](https://mlsanity.com/paper/2303.11120/) (2023)
- [Using BERT Encoding and Sentence-Level Language Model for Sentence Ordering](https://mlsanity.com/paper/2108.10986/) (2021)
- [Improving Graph-based Sentence Ordering with Iteratively Predicted Pairwise Orderings](https://mlsanity.com/paper/2110.06446/) (2021)
- [Learning Fine-Grained Visual Understanding for Video Question Answering via Decoupling Spatial-Temporal Modeling](https://mlsanity.com/paper/2210.03941/) (2022)
- [A New Sentence Ordering Method Using BERT Pretrained Model](https://mlsanity.com/paper/2108.11994/) (2021)
- [Local and Global Context-Based Pairwise Models for Sentence Ordering](https://mlsanity.com/paper/2110.04291/) (2021)
- [Pruned Graph Neural Network for Short Story Ordering](https://mlsanity.com/paper/2203.06778/) (2022)

## Abstract (arXiv)

Temporal common sense has applications in AI tasks such as QA, multi-document summarization, and human-AI communication. We propose the task of sequencing -- given a jumbled set of aligned image-caption pairs that belong to a story, the task is to sort them such that the output sequence forms a coherent story. We present multiple approaches, via unary (position) and pairwise (order) predictions, and their ensemble-based combinations, achieving strong results on this task. We use both text-based and image-based features, which depict complementary improvements. Using qualitative examples, we demonstrate that our models have learnt interesting aspects of temporal common sense.