mlsanityData to 7 Oct 2026

Understanding Figurative Meaning through Explainable Visual Entailment

arXiv 2405.01474 ↗First posted 2 May 2024cs.CL cs.AI cs.CVcited by 19

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Large Vision-Language Models (VLMs) have demonstrated strong capabilities in tasks requiring a fine-grained understanding of literal meaning in images and text, such as visual question-answering or visual entailment. However, there has been little exploration of the capabilities of these models when presented with images and captions containing figurative meaning, such as metaphors or humor. To close this gap, we propose a new task framing the figurative meaning understanding problem as an explainable visual entailment task, where the model has to predict whether the image (premise) entails a caption (hypothesis) and justify the predicted label with a textual explanation. The figurative phenomena can be present in the image, in the caption, or both. Using a human-AI collaboration approach, we build the accompanying expert-verified dataset V-FLUTE, containing 6,027 {image, caption, label, explanation} instances spanning five diverse figurative phenomena: metaphors, similes, idioms, sarcasm, and humor. Through automatic evaluation, we find that VLMs struggle to generalize from literal to figurative meaning, particularly when it is present in images. Further, we identify common types of errors in VLM reasoning (hallucination and incomplete or unsound reasoning) across classes of models via human evaluation.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 37 of its 65 references that are in the map. First posted on arXiv 2 May 2024.

Builds on

37 of 65 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Compositional Chain-of-Thought Prompting for Large Multimodal Models27 Nov
  2. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan

Other references · 35

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  5. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  6. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  7. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  8. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  9. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  10. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  11. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  12. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
23 more
  1. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  2. 2018Annotation Artifacts in Natural Language Inference Datacited by 656
  3. 2018e-SNLI: Natural Language Inference with Natural Language Explanationscited by 468
  4. 2018Hypothesis Only Baselines in Natural Language Inferencecited by 291
  5. 2019Visual Entailment: A Novel Task for Fine-Grained Image Understandingcited by 250
  6. 2022WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creationcited by 200
  7. 2021Reframing Human-AI Collaboration for Generating Free-Text Explanationscited by 125
  8. 2024Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessmentcited by 117
  9. 2023Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Taskscited by 102
  10. 2022Do Androids Laugh at Electric Sheep? Humor "Understanding" Benchmarks from The New Yorker Caption Contestcited by 98
  11. 2021The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generationcited by 96
  12. 2021Learning Compact Metrics for MTcited by 93
  13. 2021Competency Problems: On Finding and Removing Artifacts in Language Datacited by 91
  14. 2021e-ViL: A Dataset and Benchmark for Natural Language Explanations in Vision-Language Taskscited by 84
  15. 2024PUB: A Pragmatics Understanding Benchmark for Assessing LLMs' Pragmatics Capabilitiescited by 58
  16. 2023MemeCap: A Dataset for Captioning and Interpreting Memescited by 46
  17. 2019Integrating Text and Image: Determining Multimodal Document Intent in Instagram Postscited by 37
  18. 2023I Spy a Metaphor: Large Language Models and Diffusion Models Co-Create Visual Metaphorscited by 31
  19. 2022MetaCLUE: Towards Comprehensive Visual Metaphors Researchcited by 28
  20. 2021Nice perfume. How long did you marinate in it? Multimodal Sarcasm Explanationcited by 25
  21. 2023IRFL: Image Recognition of Figurative Languagecited by 23
  22. 2023Sociocultural Norm Similarities and Differences via Situational Alignment and Explainable Textual Entailmentcited by 22
  23. 2024SOCIALITE-LLAMA: An Instruction-Tuned Model for Social Scientific Taskscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 13.

19 papers.

Sort:

By area:

  1. 2025BottleHumor: Self-Informed Humor Explanation using the Information Bottleneck Principlecited by 6
  2. 2025The Mind's Eye: A Multi-Faceted Reward Framework for Guiding Visual Metaphor Generationcited by 4
  3. 2025VideoNorms: Benchmarking Cultural Awareness of Video Language Modelscited by 4
  4. 2025Looking Beyond the Obvious: A Survey on Abstract Concept Recognition for Video Understandingcited by 3
  5. 2026When Meaning Isn't Literal: Exploring Idiomatic Meaning Across Languages and Modalitiescited by 3
  6. 2025Large Vision-Language Models for Knowledge-Grounded Data Annotation of Memescited by 2
  7. 2025CPJ: Explainable Agricultural Pest Diagnosis via Caption-Prompt-Judge with LLM-Judged Refinementcited by 2
  8. 2026A Computational Approach to Visual Metonymycited by 2
  9. 2026I Came, I Saw, I Explained: Benchmarking Multimodal LLMs on Figurative Meaning in Memescited by 2
  10. 2026A Parallel Cross-Lingual Benchmark for Multimodal Idiomaticity Understandingcited by 1
  11. 2026Culture-Aware Humorous Captioning: Multimodal Humor Generation across Cultural Contextscited by 0
  12. 2026DRInQ: Evaluating Conversational Implicature with Controlled Context Variationcited by 0
7 more
  1. 2026MetaphorVU: Towards Metaphorical Video Understandingcited by 0
  2. 2026Computational Humor with Multimodal LLMs: Methods, Datasets, Evaluation, and Challengescited by 0
  3. 2026Towards High-Level Semantic Intelligencecited by 0
  4. 2026Simile Understanding in Text-to-Image Models: An Evaluation Frameworkcited by 0
  5. 2026CaRGo-T: Causal Reasoning Graph-of-Thought improves Multimodal Humor Comprehensioncited by 0
  6. 2026Presence Is Not Faithfulness: Figurative Vehicle Intrusion in Text-to-Image Generationcited by 0
  7. 2026It's Not What the Image Shows: Irrelevant Context Destabilises VLM Judges Without Informing Themcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Reasoning Beyond Literal: Cross-style Multimodal Reasoning for Figurative Language Understanding
  2. 2024Can visual language models resolve textual ambiguity with visual cues? Let visual puns tell you!
  3. 2025InsightVision: A Comprehensive, Multi-Level Chinese-based Benchmark for Evaluating Implicit Visual Semantics in Large Vision Language Models
  4. 2025Visual Puns from Idioms: An Iterative LLM-T2IM-MLLM Framework
  5. 2026More Than Meets the Eye: Measuring the Semiotic Gap in Vision-Language Models via Semantic Anchorage
  6. 2022Testing the Ability of Language Models to Interpret Figurative Language
  7. 2026Blending Concepts: Benchmarking Visual Metaphor Generation in Text-to-Image Models
  8. 2026"I See What You Did There": Can Large Vision-Language Models Understand Multimodal Puns?