# MetaCLUE: Towards Comprehensive Visual Metaphors Research

arXiv: https://arxiv.org/abs/2212.09898 · HTML: https://mlsanity.com/paper/2212.09898/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 61 references that are in the map.
- First posted on arXiv 19 Dec 2022.

## Builds on (most cited first, 20 of 20)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [Generation and Comprehension of Unambiguous Object Descriptions](https://mlsanity.com/paper/1511.02283/) (2015)
- [SPICE: Semantic Propositional Image Caption Evaluation](https://mlsanity.com/paper/1607.08822/) (2016)
- [PaLI: A Jointly-Scaled Multilingual Language-Image Model](https://mlsanity.com/paper/2209.06794/) (2022)
- [MAttNet: Modular Attention Network for Referring Expression Comprehension](https://mlsanity.com/paper/1801.08186/) (2018)
- [ArtEmis: Affective Language for Visual Art](https://mlsanity.com/paper/2101.07396/) (2021)
- [Automatic Understanding of Image and Video Advertisements](https://mlsanity.com/paper/1707.03067/) (2017)
- [MERMAID: Metaphor Generation with Symbolism and Discriminative Decoding](https://mlsanity.com/paper/2103.06779/) (2021)
- [MelBERT: Metaphor Detection via Contextualized Late Interaction using Metaphorical Identification Theories](https://mlsanity.com/paper/2104.13615/) (2021)
- [Metaphor Generation with Conceptual Mappings](https://mlsanity.com/paper/2106.01228/) (2021)
- [ADVISE: Symbolism and External Knowledge for Decoding Advertisements](https://mlsanity.com/paper/1711.06666/) (2017)
- [Adapting CLIP For Phrase Localization Without Further Training](https://mlsanity.com/paper/2204.03647/) (2022)
- [Generating Conceptual Metaphors from Proposition Stores](https://mlsanity.com/paper/1409.7619/) (2014)

## Built on by (most cited first, 28 of 28)

- [I Spy a Metaphor: Large Language Models and Diffusion Models Co-Create Visual Metaphors](https://mlsanity.com/paper/2305.14724/) (2023)
- [Understanding Figurative Meaning through Explainable Visual Entailment](https://mlsanity.com/paper/2405.01474/) (2024)
- [Text-to-Image Generation for Abstract Concepts](https://mlsanity.com/paper/2309.14623/) (2023)
- [AdsQA: Towards Advertisement Video Understanding](https://mlsanity.com/paper/2509.08621/) (2025)
- [Learning to Follow Object-Centric Image Editing Instructions Faithfully](https://mlsanity.com/paper/2310.19145/) (2023)
- [Creating a Lens of Chinese Culture: A Multimodal Dataset for Chinese Pun Rebus Art Understanding](https://mlsanity.com/paper/2406.10318/) (2024)
- [Cultural Bias Matters: A Cross-Cultural Benchmark Dataset and Sentiment-Enriched Model for Understanding Multimodal Metaphors](https://mlsanity.com/paper/2506.06987/) (2025)
- [KAFA: Rethinking Image Ad Understanding with Knowledge-Augmented Feature Adaptation of Vision-Language Models](https://mlsanity.com/paper/2305.18373/) (2023)
- [Unveiling the Invisible: Captioning Videos with Metaphors](https://mlsanity.com/paper/2406.04886/) (2024)
- [Probing and Inducing Combinational Creativity in Vision-Language Models](https://mlsanity.com/paper/2504.13120/) (2025)
- [Benchmarking VLMs' Reasoning About Persuasive Atypical Images](https://mlsanity.com/paper/2409.10719/) (2024)
- [The Mind's Eye: A Multi-Faceted Reward Framework for Guiding Visual Metaphor Generation](https://mlsanity.com/paper/2508.18569/) (2025)
- [Beyond Pixels: Visual Metaphor Transfer via Schema-Driven Agentic Reasoning](https://mlsanity.com/paper/2602.01335/) (2026)
- [ViPE: Visualise Pretty-much Everything](https://mlsanity.com/paper/2310.10543/) (2023)
- [Impressions: Understanding Visual Semiotics and Aesthetic Impact](https://mlsanity.com/paper/2310.17887/) (2023)
- [Concept Drift Guided LayerNorm Tuning for Efficient Multimodal Metaphor Identification](https://mlsanity.com/paper/2505.11237/) (2025)
- [Looking Beyond the Obvious: A Survey on Abstract Concept Recognition for Video Understanding](https://mlsanity.com/paper/2508.20765/) (2025)
- [A Computational Approach to Visual Metonymy](https://mlsanity.com/paper/2601.17706/) (2026)
- [NYK-MS: A Well-annotated Multi-modal Metaphor and Sarcasm Understanding Benchmark on Cartoon-Caption Dataset](https://mlsanity.com/paper/2409.01037/) (2024)
- [Enhancing multimodal analogical reasoning with Logic Augmented Generation](https://mlsanity.com/paper/2504.11190/) (2025)
- [Meanings are like Onions: a Layered Approach to Metaphor Processing](https://mlsanity.com/paper/2507.10354/) (2025)
- [Seeing is Believing? Evaluating Vision-Language Model Susceptibility in Agent-to-Agent Multimodal Persuasion](https://mlsanity.com/paper/2510.22768/) (2025)
- [Beyond Input-Output: Rethinking Creativity through Design-by-Analogy in Human-AI Collaboration](https://mlsanity.com/paper/2602.09423/) (2026)
- [Rhetorical Text-to-Image Generation via Two-layer Diffusion Policy Optimization](https://mlsanity.com/paper/2505.22792/) (2025)
- [Do Vision-Language Models Understand Visual Persuasiveness? A Diagnosis via Visual Persuasive Factors](https://mlsanity.com/paper/2511.17036/) (2025)
- [Sense it with your eyes: Sensation Generation and Understanding for Advertisements](https://mlsanity.com/paper/2607.25314/) (2026)
- [Blending Concepts: Benchmarking Visual Metaphor Generation in Text-to-Image Models](https://mlsanity.com/paper/2609.02502/) (2026)
- [PRISM: A Category-Theoretic Framework for Measuring and Refining Multimodal Analogies](https://mlsanity.com/paper/2610.01383/) (2026)

## Abstract (arXiv)

Creativity is an indispensable part of human cognition and also an inherent part of how we make sense of the world. Metaphorical abstraction is fundamental in communicating creative ideas through nuanced relationships between abstract concepts such as feelings. While computer vision benchmarks and approaches predominantly focus on understanding and generating literal interpretations of images, metaphorical comprehension of images remains relatively unexplored. Towards this goal, we introduce MetaCLUE, a set of vision tasks on visual metaphor. We also collect high-quality and rich metaphor annotations (abstract objects, concepts, relationships along with their corresponding object boxes) as there do not exist any datasets that facilitate the evaluation of these tasks. We perform a comprehensive analysis of state-of-the-art models in vision and language based on our annotations, highlighting strengths and weaknesses of current approaches in visual metaphor Classification, Localization, Understanding (retrieval, question answering, captioning) and gEneration (text-to-image synthesis) tasks. We hope this work provides a concrete step towards developing AI systems with human-like creative capabilities.