# The Amazing Mysteries of the Gutter: Drawing Inferences Between Panels in Comic Book Narratives

arXiv: https://arxiv.org/abs/1611.05118 · HTML: https://mlsanity.com/paper/1611.05118/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Nov 2016.

## Built on by (most cited first, 30 of 42)

- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [WebSRC: A Dataset for Web-Based Structural Reading Comprehension](https://mlsanity.com/paper/2101.09465/) (2021)
- [Fine-tuning Multimodal LLMs to Follow Zero-shot Demonstrative Instructions](https://mlsanity.com/paper/2308.04152/) (2023)
- [Underspecification in Scene Description-to-Depiction Tasks](https://mlsanity.com/paper/2210.05815/) (2022)
- [CoMix: A Comprehensive Benchmark for Multi-Task Comic Understanding](https://mlsanity.com/paper/2407.03550/) (2024)
- [COO: Comic Onomatopoeia Dataset for Recognizing Arbitrary or Truncated Texts](https://mlsanity.com/paper/2207.04675/) (2022)
- [Unsupervised Manga Character Re-identification via Face-body and Spatial-temporal Associated Clustering](https://mlsanity.com/paper/2204.04621/) (2022)
- [JDocQA: Japanese Document Question Answering Dataset for Generative Language Models](https://mlsanity.com/paper/2403.19454/) (2024)
- [OASIS: Online Sample Selection for Continual Visual Instruction Tuning](https://mlsanity.com/paper/2506.02011/) (2025)
- [Synopses of Movie Narratives: a Video-Language Dataset for Story Understanding](https://mlsanity.com/paper/2203.05711/) (2022)
- [Identity-Aware Semi-Supervised Learning for Comic Character Re-Identification](https://mlsanity.com/paper/2308.09096/) (2023)
- [MangaUB: A Manga Understanding Benchmark for Large Multimodal Models](https://mlsanity.com/paper/2407.19034/) (2024)
- [One missing piece in Vision and Language: A Survey on Comics Understanding](https://mlsanity.com/paper/2409.09502/) (2024)
- [Comics Datasets Framework: Mix of Comics datasets for detection benchmarking](https://mlsanity.com/paper/2407.03540/) (2024)
- [MangaVQA and MangaLMM: A Benchmark and Specialized Model for Multimodal Manga Understanding](https://mlsanity.com/paper/2505.20298/) (2025)
- [A Challenging Benchmark of Anime Style Recognition](https://mlsanity.com/paper/2204.14034/) (2022)
- [Video Panels for Long Video Understanding](https://mlsanity.com/paper/2509.23724/) (2025)
- [Multimodal Transformer for Comics Text-Cloze](https://mlsanity.com/paper/2403.03719/) (2024)
- [ComicsPAP: understanding comic strips by picking the correct panel](https://mlsanity.com/paper/2503.08561/) (2025)
- [From Panels to Prose: Generating Literary Narratives from Comics](https://mlsanity.com/paper/2503.23344/) (2025)
- [Gaze Heads: How VLMs Look at What They Describe](https://mlsanity.com/paper/2606.14703/) (2026)
- [A Comprehensive Gold Standard and Benchmark for Comics Text Detection and Recognition](https://mlsanity.com/paper/2212.14674/) (2022)
- [A Customizable Generator for Comic-Style Visual Narrative](https://mlsanity.com/paper/2401.02863/) (2023)
- [Breaking Language Barriers in Visual Language Models via Multilingual Textual Regularization](https://mlsanity.com/paper/2503.22577/) (2025)
- [KTRL+F: Knowledge-Augmented In-Document Search](https://mlsanity.com/paper/2311.08329/) (2023)
- [ComiCap: A VLMs pipeline for dense captioning of Comic Panels](https://mlsanity.com/paper/2409.16159/) (2024)
- [Re:Verse -- Can Your VLM Read a Manga?](https://mlsanity.com/paper/2508.08508/) (2025)
- [Towards Solving Multimodal Comprehension](https://mlsanity.com/paper/2104.10139/) (2021)
- [Modeling Temporal-Modal Entity Graph for Procedural Multimodal Machine Comprehension](https://mlsanity.com/paper/2204.02566/) (2022)
- [Toward accessible comics for blind and low vision readers](https://mlsanity.com/paper/2407.08248/) (2024)

## Abstract (arXiv)

Visual narrative is often a combination of explicit information and judicious omissions, relying on the viewer to supply missing details. In comics, most movements in time and space are hidden in the "gutters" between panels. To follow the story, readers logically connect panels together by inferring unseen actions through a process called "closure". While computers can now describe what is explicitly depicted in natural images, in this paper we examine whether they can understand the closure-driven narratives conveyed by stylized artwork and dialogue in comic book panels. We construct a dataset, COMICS, that consists of over 1.2 million panels (120 GB) paired with automatic textbox transcriptions. An in-depth analysis of COMICS demonstrates that neither text nor image alone can tell a comic book story, so a computer must understand both modalities to keep up with the plot. We introduce three cloze-style tasks that ask models to predict narrative and character-centric aspects of a panel given n preceding panels as context. Various deep neural architectures underperform human baselines on these tasks, suggesting that COMICS contains fundamental challenges for both vision and language.