# A Computational Approach to Visual Metonymy

arXiv: https://arxiv.org/abs/2601.17706 · HTML: https://mlsanity.com/paper/2601.17706/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 24 of its 62 references that are in the map.
- First posted on arXiv 25 Jan 2026.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 24 of 24)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [gpt-oss-120b & gpt-oss-20b Model Card](https://mlsanity.com/paper/2508.10925/) (2025)
- [OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge](https://mlsanity.com/paper/1906.00067/) (2019)
- [ConceptNet 5.5: An Open Multilingual Graph of General Knowledge](https://mlsanity.com/paper/1612.03975/) (2016)
- [Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling](https://mlsanity.com/paper/2501.17811/) (2025)
- [From Recognition to Cognition: Visual Commonsense Reasoning](https://mlsanity.com/paper/1811.10830/) (2018)
- [VisDA: The Visual Domain Adaptation Challenge](https://mlsanity.com/paper/1710.06924/) (2017)
- [Benchmarking Vision Language Models for Cultural Understanding](https://mlsanity.com/paper/2407.10920/) (2024)
- [I Spy a Metaphor: Large Language Models and Diffusion Models Co-Create Visual Metaphors](https://mlsanity.com/paper/2305.14724/) (2023)
- [MetaCLUE: Towards Comprehensive Visual Metaphors Research](https://mlsanity.com/paper/2212.09898/) (2022)
- [IRFL: Image Recognition of Figurative Language](https://mlsanity.com/paper/2303.15445/) (2023)
- [Understanding Figurative Meaning through Explainable Visual Entailment](https://mlsanity.com/paper/2405.01474/) (2024)
- [MMDocBench: Benchmarking Large Vision-Language Models for Fine-Grained Visual Document Understanding](https://mlsanity.com/paper/2410.21311/) (2024)
- [Text-to-Image Generation for Abstract Concepts](https://mlsanity.com/paper/2309.14623/) (2023)
- [MVP-Bench: Can Large Vision--Language Models Conduct Multi-level Visual Perception Like Humans?](https://mlsanity.com/paper/2410.04345/) (2024)
- [Unveiling the Invisible: Captioning Videos with Metaphors](https://mlsanity.com/paper/2406.04886/) (2024)
- [ConMeC: A Dataset for Metonymy Resolution with Common Nouns](https://mlsanity.com/paper/2502.06087/) (2025)
- [Seeing the Abstract: Translating the Abstract Language for Vision Language Models](https://mlsanity.com/paper/2505.03242/) (2025)
- [Unveiling the Mystery of Visual Attributes of Concrete and Abstract Concepts: Variability, Nearest Neighbors, and Challenging Categories](https://mlsanity.com/paper/2410.11657/) (2024)
- [Hummus: A Dataset of Humorous Multimodal Metaphor Use](https://mlsanity.com/paper/2504.02983/) (2025)

## Built on by (most cited first, 3 of 3)

- [MetFuse: Figurative Fusion between Metonymy and Metaphor](https://mlsanity.com/paper/2604.12919/) (2026)
- [Exploring Concreteness Through a Figurative Lens](https://mlsanity.com/paper/2604.18296/) (2026)
- [Metonymic Circuits for Abstract Concept Grounding in Vision Transformers](https://mlsanity.com/paper/2610.06928/) (2026)

## Abstract (arXiv)

Images often communicate more than they literally depict: a set of tools can suggest an occupation and a cultural artifact can suggest a tradition. This kind of indirect visual reference, known as visual metonymy, invites viewers to recover a target concept via associated cues rather than explicit depiction. In this work, we present the first computational investigation of visual metonymy. We introduce a novel pipeline grounded in semiotic theory that leverages large language models and text-to-image models to generate metonymic visual representations. Using this framework, we construct ViMET, the first visual metonymy dataset comprising 2,000 multiple-choice questions to evaluate the cognitive reasoning abilities in multimodal language models. Experimental results on our dataset reveal a significant gap between human performance (86.9%) and state-of-the-art vision-language models (65.9%), highlighting limitations in machines' ability to interpret indirect visual references. Our dataset is publicly available at: https://github.com/cincynlp/ViMET.