mlsanityData to 7 Oct 2026

IRFL: Image Recognition of Figurative Language

arXiv 2303.15445 ↗First posted 27 Mar 2023cs.CL cs.AI cs.CVcited by 23

Not on the map. Why?

Abstract (arXiv)

Figures of speech such as metaphors, similes, and idioms are integral parts of human communication. They are ubiquitous in many forms of discourse, allowing people to convey complex, abstract ideas and evoke emotion. As figurative forms are often conveyed through multiple modalities (e.g., both text and images), understanding multimodal figurative language is an important AI challenge, weaving together profound vision, language, commonsense and cultural knowledge. In this work, we develop the Image Recognition of Figurative Language (IRFL) dataset. We leverage human annotation and an automatic pipeline we created to generate a multimodal dataset, and introduce two novel tasks as a benchmark for multimodal figurative language understanding. We experimented with state-of-the-art vision and language models and found that the best (22%) performed substantially worse than humans (97%). We release our dataset, benchmark, and code, in hopes of driving the development of models that can better understand figurative language.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 31 of its 73 references that are in the map. First posted on arXiv 27 Mar 2023.

Builds on

31 of 73 references.

References · 31

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  5. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  7. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  8. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  9. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  10. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  11. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  12. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
19 more
  1. 2018Gender Bias in Coreference Resolutioncited by 438
  2. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  3. 2022End-to-End Multimodal Fact-Checking and Explanation Generation: A Challenging Dataset and Modelscited by 105
  4. 2021Abstraction and Analogy-Making in Artificial Intelligencecited by 103
  5. 2023Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Imagescited by 79
  6. 2019Do Neural Language Representations Learn Physical Commonsense?cited by 62
  7. 2021ExplaGraphs: An Explanation Graph Generation Task for Structured Commonsense Reasoningcited by 58
  8. 2016Detecting Sarcasm in Multimodal Social Platformscited by 43
  9. 2022Abstract Visual Reasoning with Tangram Shapescited by 43
  10. 2021It's not Rocket Science : Interpreting Figurative Language in Narrativescited by 39
  11. 2020Detecting Hate Speech in Multi-modal Memescited by 38
  12. 2021Metaphor Generation with Conceptual Mappingscited by 32
  13. 2022Metaphors in Pre-Trained Language Models: Probing and Generalization Across Datasets and Languagescited by 30
  14. 2019Still a Pain in the Neck: Evaluating Text Representations on Lexical Compositioncited by 27
  15. 2020Identifying Depressive Symptoms from Tweets: Figurative Language Enabled Multitask Learning Frameworkcited by 25
  16. 2022VASR: Visual Analogies of Situation Recognitioncited by 22
  17. 2022WinoGAViL: Gamified Association Benchmark to Challenge Vision-and-Language Modelscited by 20
  18. 2019Neural Simile Recognition with Cyclic Multitask Learning and Local Attentioncited by 18
  19. 2022Can Pre-trained Language Models Interpret Similes as Smart as Human?cited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 5; 2025: 5; 2026: 7.

23 papers.

Sort:

By area:

  1. 2023VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Usecited by 100
  2. 2023Factorized Contrastive Learning: Going Beyond Multi-view Redundancycited by 73
  3. 2023MemeCap: A Dataset for Captioning and Interpreting Memescited by 46
  4. 2023I Spy a Metaphor: Large Language Models and Diffusion Models Co-Create Visual Metaphorscited by 31
  5. 2024HEMM: Holistic Evaluation of Multimodal Foundation Modelscited by 21
  6. 2023Multimodal Learning Without Labeled Multimodal Data: Guarantees and Applicationscited by 20
  7. 2024Understanding Figurative Meaning through Explainable Visual Entailmentcited by 19
  8. 2025Jawaher: A Multidialectal Dataset of Arabic Proverbs for LLM Benchmarkingcited by 18
  9. 2025LLLMs: A Data-Driven Survey of Evolving Research on Limitations of Large Language Modelscited by 9
  10. 2024Unveiling the Invisible: Captioning Videos with Metaphorscited by 7
  11. 2024Can We Predict Performance of Large Models across Vision-Language Tasks?cited by 6
  12. 2023ViPE: Visualise Pretty-much Everythingcited by 3
11 more
  1. 2025SemEval-2025 Task 1: AdMIRe -- Advancing Multimodal Idiomaticity Representationcited by 3
  2. 2025Concept Drift Guided LayerNorm Tuning for Efficient Multimodal Metaphor Identificationcited by 3
  3. 2025MINT: Multimodal Instruction Tuning with Multimodal Interaction Groupingcited by 2
  4. 2026A Computational Approach to Visual Metonymycited by 2
  5. 2026A Parallel Cross-Lingual Benchmark for Multimodal Idiomaticity Understandingcited by 1
  6. 2024Foundations of Multisensory Artificial Intelligencecited by 0
  7. 2026Reasoning Beyond Literal: Cross-style Multimodal Reasoning for Figurative Language Understandingcited by 0
  8. 2026When Meaning Travels: A Granular Lens on Hybrid-MoE's Role in Idiomatic Understanding for Language Modelscited by 0
  9. 2026RePercENT: Scaling Disentangled Representation Learning Beyond Two Modalitiescited by 0
  10. 2026Simile Understanding in Text-to-Image Models: An Evaluation Frameworkcited by 0
  11. 2026It's Not What the Image Shows: Irrelevant Context Destabilises VLM Judges Without Informing Themcited by 0