# Evaluating Large Language Models' Ability Using a Psychiatric Screening Tool Based on Metaphor and Sarcasm Scenarios

arXiv: https://arxiv.org/abs/2309.10744 · HTML: https://mlsanity.com/paper/2309.10744/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 79 references that are in the map.
- First posted on arXiv 19 Sep 2023.

## Builds on (most cited first, 16 of 16)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Large Language Models Can Be Easily Distracted by Irrelevant Context](https://mlsanity.com/paper/2302.00093/) (2023)
- [How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection](https://mlsanity.com/paper/2301.07597/) (2023)
- [Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks](https://mlsanity.com/paper/2302.08399/) (2023)
- [FLUTE: Figurative Language Understanding through Textual Explanations](https://mlsanity.com/paper/2205.12404/) (2022)
- [Testing the Ability of Language Models to Interpret Figurative Language](https://mlsanity.com/paper/2204.12632/) (2022)
- [Evaluating Theory of Mind in Question Answering](https://mlsanity.com/paper/1808.09352/) (2018)
- [Do Large Language Models know what humans know?](https://mlsanity.com/paper/2209.01515/) (2022)
- [Language acquisition: do children and language models follow similar learning stages?](https://mlsanity.com/paper/2306.03586/) (2023)
- [Metaphors in Pre-Trained Language Models: Probing and Generalization Across Datasets and Languages](https://mlsanity.com/paper/2203.14139/) (2022)
- [A Transformer-based approach to Irony and Sarcasm detection](https://mlsanity.com/paper/1911.10401/) (2019)
- [Large Language Model Displays Emergent Ability to Interpret Novel Literary Metaphors](https://mlsanity.com/paper/2308.01497/) (2023)
- [Sarcasm Detection: A Comparative Study](https://mlsanity.com/paper/2107.02276/) (2021)

## Built on by (most cited first, 2 of 2)

- [Large Language Models for Subjective Language Understanding: A Survey](https://mlsanity.com/paper/2508.07959/) (2025)
- [MoPrO-SD: Modular Prompt Optimization for Stance Detection](https://mlsanity.com/paper/2610.13386/) (2026)

## Abstract (arXiv)

Metaphors and sarcasm are precious fruits of our highly evolved social communication skills. However, children with the condition then known as Asperger syndrome are known to have difficulties in comprehending sarcasm, even if they possess adequate verbal IQs for understanding metaphors. Accordingly, researchers had employed a screening test that assesses metaphor and sarcasm comprehension to distinguish Asperger syndrome from other conditions with similar external behaviors (e.g., attention-deficit/hyperactivity disorder). This study employs a standardized test to evaluate recent large language models' (LLMs) understanding of nuanced human communication. The results indicate improved metaphor comprehension with increased model parameters; however, no similar improvement was observed for sarcasm comprehension. Considering that a human's ability to grasp sarcasm has been associated with the amygdala, a pivotal cerebral region for emotional learning, a distinctive strategy for training LLMs would be imperative to imbue them with the ability in a cognitively grounded manner.