# Semantic Similarity is a Spurious Measure of Comic Understanding: Lessons Learned from Hallucinations in a Benchmarking Experiment

arXiv: https://arxiv.org/abs/2603.01950 · HTML: https://mlsanity.com/paper/2603.01950/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 20 references that are in the map.
- First posted on arXiv 2 Mar 2026.

## Builds on (most cited first, 11 of 11)

- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [MiniCPM-V: A GPT-4V Level MLLM on Your Phone](https://mlsanity.com/paper/2408.01800/) (2024)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [Object Hallucination in Image Captioning](https://mlsanity.com/paper/1809.02156/) (2018)
- [Sketch-based Manga Retrieval using Manga109 Dataset](https://mlsanity.com/paper/1510.04389/) (2015)
- [PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understanding](https://mlsanity.com/paper/2501.16411/) (2025)
- [CoMix: A Comprehensive Benchmark for Multi-Task Comic Understanding](https://mlsanity.com/paper/2407.03550/) (2024)
- [A survey of comics research in computer science](https://mlsanity.com/paper/1804.05490/) (2018)
- [Comics for Everyone: Generating Accessible Text Descriptions for Comic Strips](https://mlsanity.com/paper/2310.00698/) (2023)
- [ComiCap: A VLMs pipeline for dense captioning of Comic Panels](https://mlsanity.com/paper/2409.16159/) (2024)

## Abstract (arXiv)

A system that enables blind or visually impaired users to access comics/manga would introduce a new medium of storytelling to this community. However, no such system currently exists. Generative vision-language models (VLMs) have shown promise in describing images and understanding comics, but most research on comic understanding is limited to panel-level analysis. To fully support blind and visually impaired users, greater attention must be paid to page-level understanding and interpretation. In this work, we present a preliminary benchmark of VLM performance on comic interpretation tasks. We identify and categorize hallucinations that emerge during this process, organizing them into generalized object-hallucination taxonomies. We conclude with guidance on future research, emphasizing hallucination mitigation and improved data curation for comic interpretation.