mlsanityData to 7 Oct 2026

FLUTE: Figurative Language Understanding through Textual Explanations

arXiv 2205.12404 ↗First posted 24 May 2022cs.CLcited by 78

Not on the map. Why?

Abstract (arXiv)

Figurative language understanding has been recently framed as a recognizing textual entailment (RTE) task (a.k.a. natural language inference, or NLI). However, similar to classical RTE/NLI datasets, the current benchmarks suffer from spurious correlations and annotation artifacts. To tackle this problem, work on NLI has built explanation-based datasets such as e-SNLI, allowing us to probe whether language models are right for the right reasons.Yet no such data exists for figurative language, making it harder to assess genuine understanding of such expressions. To address this issue, we release FLUTE, a dataset of 9,000 figurative NLI instances with explanations, spanning four categories: Sarcasm, Simile, Metaphor, and Idioms. We collect the data through a model-in-the-loop framework based on GPT-3, crowd workers, and expert annotators. We show how utilizing GPT-3 in conjunction with human annotators (novices and experts) can aid in scaling up the creation of datasets even for such complex linguistic phenomena as figurative language. The baseline performance of the T5 model fine-tuned on FLUTE shows that our dataset can bring us a step closer to developing models that understand figurative language through textual explanations.

Key facts. Cited by 78 papers in the map (data through 7 Oct 2026). Builds on 30 of its 58 references that are in the map. First posted on arXiv 24 May 2022.

Builds on

30 of 58 references.

References · 30

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  4. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  6. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  7. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,400
  8. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  9. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  10. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  11. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  12. 2018e-SNLI: Natural Language Inference with Natural Language Explanationscited by 468
18 more
  1. 2021Persistent Anti-Muslim Bias in Large Language Modelscited by 376
  2. 2019Explain Yourself! Leveraging Language Models for Commonsense Reasoningcited by 367
  3. 2018Hypothesis Only Baselines in Natural Language Inferencecited by 291
  4. 2020Experience Grounds Languagecited by 241
  5. 2021Societal Biases in Language Generation: Progress and Challengescited by 201
  6. 2022WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creationcited by 200
  7. 2020Measuring Association Between Labels and Free-Text Rationalescited by 159
  8. 2020WT5?! Training Text-to-Text Models to Explain their Predictionscited by 148
  9. 2018Breaking NLI Systems with Sentences that Require Simple Lexical Inferencescited by 134
  10. 2021Reframing Human-AI Collaboration for Generating Free-Text Explanationscited by 125
  11. 2021e-ViL: A Dataset and Benchmark for Natural Language Explanations in Vision-Language Taskscited by 84
  12. 2022Testing the Ability of Language Models to Interpret Figurative Languagecited by 67
  13. 2021It's not Rocket Science : Interpreting Figurative Language in Narrativescited by 39
  14. 2022When did you become so smart, oh wise one?! Sarcasm Explanation in Multi-modal Multi-party Dialoguescited by 32
  15. 2021Figurative Language in Recognizing Textual Entailmentcited by 26
  16. 2021Investigating Robustness of Dialog Models to Popular Figurative Language Constructscited by 19
  17. 2021Few-Shot Out-of-Domain Transfer Learning of Natural Language Explanations in a Label-Abundant Setupcited by 16
  18. 2022Neuro-symbolic Natural Logic with Introspective Revision for Natural Language Inferencecited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 17; 2024: 22; 2025: 19; 2026: 19.

78 papers.

Sort:

By area:

  1. 2023Can Large Language Models Transform Computational Social Science?cited by 245
  2. 2024PUB: A Pragmatics Understanding Benchmark for Assessing LLMs' Pragmatics Capabilitiescited by 58
  3. 2023Multi-lingual and Multi-cultural Figurative Language Understandingcited by 47
  4. 2023Using Imperfect Surrogates for Downstream Inference: Design-based Supervised Learning for Social Science Applications of Large Language Modelscited by 37
  5. 2023Are Multilingual LLMs Culturally-Diverse Reasoners? An Investigation into Multicultural Proverbs and Sayingscited by 32
  6. 2023I Spy a Metaphor: Large Language Models and Diffusion Models Co-Create Visual Metaphorscited by 31
  7. 2024Metaphor Understanding Challenge Dataset for LLMscited by 30
  8. 2024AI for social science and social science of AI: A Surveycited by 29
  9. 2023Sociocultural Norm Similarities and Differences via Situational Alignment and Explainable Textual Entailmentcited by 22
  10. 2025Jawaher: A Multidialectal Dataset of Arabic Proverbs for LLM Benchmarkingcited by 18
  11. 2025Fann or Flop: A Multigenre, Multiera Benchmark for Arabic Poetry Understanding in LLMscited by 15
  12. 2023Learning to Follow Object-Centric Image Editing Instructions Faithfullycited by 12
38 more
  1. 2023Can Language Models Laugh at YouTube Short-form Videos?cited by 11
  2. 2024SoMeLVLM: A Large Vision Language Model for Social Media Processingcited by 11
  3. 2024Sign of the Times: Evaluating the use of Large Language Models for Idiomaticity Detectioncited by 11
  4. 2024A Hard Nut to Crack: Idiom Detection with Conversational Large Language Modelscited by 11
  5. 2025The Stochastic Parrot on LLM's Shoulder: A Summative Assessment of Physical Concept Understandingcited by 11
  6. 2025Position: Text Embeddings Should Capture Implicit Semantics, Not Just Surface Meaningcited by 10
  7. 2024Explicit, Implicit, and Scattered: Revisiting Event Extraction to Capture Complex Argumentscited by 9
  8. 2022Just-DREAM-about-it: Figurative Language Understanding with DREAM-FLUTEcited by 8
  9. 2023NeuroComparatives: Neuro-Symbolic Distillation of Comparative Knowledgecited by 8
  10. 2023Representation Of Lexical Stylistic Features In Language Models' Embedding Spacecited by 8
  11. 2025Metaphor and Large Language Models: When Surface Features Matter More than Deep Understandingcited by 8
  12. 2026Evaluating Decision Models for Text Annotation in Computational Social Sciencecited by 8
  13. 2024SOCIALITE-LLAMA: An Instruction-Tuned Model for Social Scientific Taskscited by 7
  14. 2024Creativity in AI: Progresses and Challengescited by 7
  15. 2024Generating Harder Cross-document Event Coreference Resolution Datasets using Metaphoric Paraphrasingcited by 6
  16. 2025Mitigating Biases in Language Models via Bias Unlearningcited by 6
  17. 2023ICLEF: In-Context Learning with Expert Feedback for Explainable Style Transfercited by 5
  18. 2024Generalizable Sarcasm Detection Is Just Around The Corner, Of Course!cited by 5
  19. 2024Meta4XNLI: A Crosslingual Parallel Corpus for Metaphor Detection and Interpretationcited by 5
  20. 2024VAGUE: Visual Contexts Clarify Ambiguous Expressionscited by 5
  21. 2025KRISTEVA: Close Reading as a Novel Task for Benchmarking Interpretive Reasoningcited by 4
  22. 2025Isolating Culture Neurons in Multilingual Large Language Modelscited by 4
  23. 2025VideoNorms: Benchmarking Cultural Awareness of Video Language Modelscited by 4
  24. 2023FEED PETs: Further Experimentation and Expansion on the Disambiguation of Potentially Euphemistic Termscited by 3
  25. 2023ViPE: Visualise Pretty-much Everythingcited by 3
  26. 2023That was the last straw, we need more: Are Translation Systems Sensitive to Disambiguating Context?cited by 3
  27. 2024Finding Challenging Metaphors that Confuse Pretrained Language Modelscited by 3
  28. 2025SemEval-2025 Task 1: AdMIRe -- Advancing Multimodal Idiomaticity Representationcited by 3
  29. 2026When Meaning Isn't Literal: Exploring Idiomatic Meaning Across Languages and Modalitiescited by 3
  30. 2024Towards a Fully Interpretable and More Scalable RSA Model for Metaphor Understandingcited by 2
  31. 2024ExpressivityBench: Can LLMs Communicate Implicitly?cited by 2
  32. 2025Hummus: A Dataset of Humorous Multimodal Metaphor Usecited by 2
  33. 2025A Survey of Idiom Datasets for Psycholinguistic and Computational Researchcited by 2
  34. 2025NLP Datasets for Idiom and Figurative Language Taskscited by 2
  35. 2026FFE-Hallu:Hallucinations in Fixed Figurative Expressions:Benchmark of Idioms and Proverbs in the Persian Languagecited by 2
  36. 2026When Words Don't Mean What They Say: Figurative Understanding in Bengali Idiomscited by 2
  37. 2026I Came, I Saw, I Explained: Benchmarking Multimodal LLMs on Figurative Meaning in Memescited by 2
  38. 2023Evaluating Large Language Models' Ability Using a Psychiatric Screening Tool Based on Metaphor and Sarcasm Scenarioscited by 1

50 most cited of 78 All 78 (JSON)